Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper presents an algorithm to train neural networks combining sparsity and adversarial training. 
On the sparsity inducing regularization, the paper proposes using proximal methods. 

On the positive side:
- The goal is of interest when it comes to real-world applications.  
- There is an interesting analysis of the weights and how pruning / AT affects them. 


On the negative side:

- In my opinion, the introduction messes unstructured and structured. Weight sparsification is associated with structured while I do believe is unstructured. 


- Related work does not seem very comprehensive. The paper claims novelty on using RGSM to improve performance. How is this different from the formulation used in "Learning the Number of Neurons in Deep Networks, Alvarez and Salzmann NIPS 2016".



Experimental settings:
- One thing that got my attention is the threshold for pruning weights (1e-15). I think that is not a fair value. There are related works suggesting no loss in accuracy if the threshold is ~1e-5 (y. Sparse convolutional neural networks CVPR2015). I do believe the numbers would change drastically. 

- The comparison between ADMM and the proximal is unfair if using that threshold. The proximal has an implicit thresholding Eq. 6. 

- What is the goal for the ensembles of the small networks? where are the numbers?


Minor stuff:
- In sparsity and robustness, there are some works missing (as a typo). 
- On the method, there are two parts where I am confused. What is the aim for including all-around Eq. 4? The same with the theoretical guarantees. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary: This paper presents an observation that Feynman-Kac formalism principled ResNet ensemble [1] yields sparser network weights compared to those of standard Resnet.  Based on this observation, the authors combine the ensemble model with Lagrangian based sparsification methods to obtain sparse and robust models. 

Pros:
- Obtaining sparse and robust networks is an important and challenging problem that could be of interest to a large audience.
- The paper provides an interesting observation that EnResNet [1] yields sparser network weights compared to standard ResNet, and further leverage it to obtain sparse and robust models. 

Cons:
- The paper has limited novelty. It has already been shown in the original EnResNet paper [1] that EnResNet is more robust to adversarial attacks. Thus the only additional contribution of this paper is the observation that EnResnet is more sparse than standard ResNet, and combination of EnResnet with sparsification methods.  
- The paper is not well justified on why one should use sparsifying techniques such as RVSM, RGSM with the Feynman-Kac formula principled EnResnet. The authors state that this enables sparsity to meet robustness; however, in all experimental results, the robustness actually decreases with the increase in sparsity which is opposite to the claims made in the paper. 
- The authors only report robustness on white-box gradient-based attacks. Thus it is not clear whether the method will generalize to black-box/gradient-free attack approaches such as NAttack [2].

Minor comments:

- Why didn’t you report the accuracy on the clean examples? This is important in showing that the method generalizes well to clean examples while maintaining robustness.
- Why use only 20 iterations to evaluate the attack? Will the model maintain its robustness with the increase in the iteration and epsilon? 
- Figures are not referenced anywhere in the text.
- It would be better to put the results on CIFAR100 in the main paper rather than in the appendix.
- Page 2, “lower cases”  -> “lower case”.
- Page 2, “Related Works” -> “Related work”, it’s better to have it as a separate section.

Overall, while the paper provides an interesting observation, it has limited contributions due to lack of novelty. Further, inadequate experimental validation makes it difficult to see if the claims made in the paper are actually true. Thus I believe that this paper requires substantial improvements in order to be accepted to top-tier publication venues such as ICLR.

References:
[1] Bao et al. 2018, https://arxiv.org/abs/1811.10745 
[2] Yadong et al. 2019, https://arxiv.org/abs/1905.00441
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper focuses on the study of sparse neural architectures and efficient DNN compression algorithms for the robust and accurate deep learning. The authors apply relaxed augmented Lagrangian based sparsification algorithms to perform both unstructured and channel pruning for the AT DNNs. By combing with the sparsity merit of EnResNet, their sparsification algorithms further boost the sparsity limit of the AT DNNs, leading to much better robustness and accuracy. Their approach demonstrates superior natural and robust accuracies under several benchmark attacks.

Strengths:
1 The authors practically apply the RVSM/RGSM algorithms to the AT using robust PGD training, resulting in significantly enhanced sparsity of DNNs, and achieving promising robust accuracies against various adversarial attacks.   
2 The authors provide the theoretical analysis showing the convergence of the RVSM algorithm.

Weaknesses:
1 The main contribution of paper is to adapt the RVSM/RGSM algorithms to the AT of DNNs to sparsify the deep model. However, both RVSM and RGSM have already been used as sparsification algorithms for DNNs, and the Feynman-Kac formalism principled DNNs has also been investigated for the purpose of sparsification, thus the contribution is largely reduced and the novelty appear to be limited.

2 For the experiments, the effectiveness of RVSM/RGSM are verified on the variants of ResNet and EnResNet, and there are insufficient comparisons with other cutting-edge compression/sparsification methods to show the advantage of the proposed method.

Question:
1 The author claim that ADMM produce a lot of small weights that cannot be regarded as zero since the norm is large than 1e-15. Is the ‘1e-15’ threshold too small to regard a value as zero?
review_title: Official Blind Review #3

Meta-review: The paper is rejected based on unanimous reviews.
Decision: Reject