Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: 1. The paper proposes GDWS with an error optimization strategy to replace regular convolution for enhancing both throughput (FPS) and adversarila robustness.
2. Experimental results on CIFAR-10, SVHN, and ImageNet validate that the replacing a CNN's regular conv with GDWS can achieve significantly higher FPS and preserve adversarial robustness.
main_review: Pros.
1. Motivation, improving throughput and adversarial robustness, is clearly written and valuable.
2. The proposed method is implementable. Code is provided.
3. The paper formally provides theoretical analysis for error approximation of the proposed GDWS.
4. The proposed GDWS is directly applied to a pretrained model, i.e., no retraining is required.
5. Experimental results are convincing and promising.

Cons.
1. Comparison between the proposed GDWS and traditional DWS is not fair. In Section 3.1, the authors argues advantages of GDWS over DWS, e.g., higher throughput and better robustness. I agree that a GDWS-based CNN have such advantages compared to a DWS-based CNN, as shown in Experiments (Section 4). However, I think the advantages of a GDWS-based CNN come from two aspects: error optimization as shown in Algorithm 3 and replacing regular conv with GDWS, but not the formulation of GDWS itself. Compared to the formulation of DWS, GDWS tends to have more kernels which enlarges parameter space but also increases computation burden, as explained in Section 3.1. If all DWS in a CNN can be optimized by an algorithm similar with Algorithm 3, I think a DWS-based CNN can also have improved robustness. Furthermore, current experiments "ResNet vs. ResNet+GDWS" are not best examples to show advantage on higher throughput (FPS), since ResNet adopts regular conv but not DWS. I think the following experiment is more appropriate:  "MobileNet-V1 vs. Replacing MobileNet-V1 with GDWS".
2. The proposed method indirectly but not directly improve adversarial robustness. As shown in Table 5, adversarial robustness is indeed enhanced. However, main techniques of the proposed method, as shown in Section 3.3 and Section 3.4, work on minimizing errors (i.e., accuracy loss) brought by using GDWS but not considering adversarial perturbations. So I think enhancing  adversarial robustness is a by-product of using GDWS but not an unique advantage (or direct target) of using GDWS. 
limitations_and_societal_impact: Limitations
1) The paper does not introduce limitations of the proposed method.
2) As mentioned in Cons. 2), I think a potential limitation is that GWDS cannot significantly improve throughput and adversarial robustness if a mobile CNN adopts DWS or similar techniques already.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a post-training transformation of a pre-trained 2D convolutional layer into a general depth-wise separable convolution operation. The optimization primarily focuses on minimizing error and limiting computation complexity. The paper claims to achieve a higher robustness (i.e., defend adversarial input attack) and higher frame per second than existing methods.
main_review: i) Originality: The proposed method is a novel technique in particular for two reasons:
First, the idea of post training transformation of the Convolution operation to a depth-wise separable convolution certainly carries significant weight for hardware application of these models (e.g., speed-up). Second, the method is well-designed to minimize the error after GDWS so that both baseline accuracy and accuracy under attack remains fairly stable.  The authors have missed to cite relevant dynamic pruning and dynamic architecture works. I think dynamic neural networks would have shown more competitive performance in terms of efficiency.

ii) Clarity: The paper is well-written and easy to follow.  

iii) Significance: The results presented in table 1 & 2 clearly validate that the proposed method improves the FPS in comparison to the prior methods.  While the computation gain is impressive, the lower inference accuracy and in particular the A_{rob} raises a few concerns regarding the initial claim of the authors.

First, looking at the method presented in the paper, the first question is why not directly use a MobileNet-v2 model? The authors did answer this question in line 129-131 and stated that [7] already showed the in-efficacy of mobilenet-v2 against PGD attack. However, when I open reference [7] and check table 7 [7], it shows mobilenet-v2 is more robust than both ResNet-18 and ResNet-50 models. On the contrary ,  applying GDWS on any ResNet model would cause slight accuracy degradation against PGD. Can the author clarify this issue? In addition the comparison with Robnet in table 2 is not clear.  Looks like RobNet has many variants, certainly the one chosen for table 2 is not performing well.  

Second,  In general based on my first point, I would emphasize that the robustness claim of this work is weak. In all cases, the method seems to improve FPS with slight degradation of A_{nat} and A_{rob}. Hence, the contribution is mainly coming from the efficiency side. I would suggest the authors modify the robustness claim  of line 53: “achieve higher robustness and higher FPS than existing robust complexity reduction”. I think the correct claim is in line 240 “without additional training and without compromising on robust accuracy”. I think the strength of the method is it does not require any training and does not compromise robust accuracy much. However, in comparison to ADMM [42] it is showing inferior performance under attack.

Third, I am just curious to know if this method is applicable during training? Since you are applying to PGD, all the steps should be differentiable and hence the method should be applicable for the training phase.  What happens if you try to fine-tune the model after post-training GDWS? Or if you train it from scratch would that be similar to mobilenet training? What is the FPS of MobileNet-v2? 

limitations_and_societal_impact: I do not see any negative social impact of this work
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed a method named  Generalized Depthwise-Separable (GDWS) to improve the throughput on real-life hardware while simultaneously preserving robustness. The advantage of GDWS is that does not need any additional training.  This paper has demonstrated the effectiveness of GDWS via extensive experiments on CIFAR-10, SVHN, and ImageNet datasets.


main_review: - Originality: The article clearly describes the necessity and relevance of the research.
- Quality: For the proposed method, I don't see any flaws. The experiment is relatively complete, and could potentially support their claims.
- Clarity: This paper is well written and easy to follow.
- Significance: From the experimental results, the proposed GDWS has a significant increase in network throughput and the decline in accuracy is within an acceptable range. 


limitations_and_societal_impact: I have some suggestions that may help the author to further improve the quality of the article.
1.The author may consider adding some verifications for tasks that rely more heavily on throughput, such as real-time target detection, real-time depth estimation. 
2.The authors need to describe the experimental environment in more detail, such as the CUDA version and the PyTorch version. Because different versions of the experimental environment will have a certain impact on training speed and inference speed.
3. The description of "W_D has the following property" could change to"the property of W_D is shown in Fig.3" which may be more clear.
review_title: null

Meta-review: This paper proposes a new approach for converting convolutions into depth-wise separable convolutions after training that offers advantages in terms of both throughput and adversarial robustness. The reviewers all agree that the proposed method is clever and the demonstrated performance boost is significant.  The adversarial robustness claims, while not a major focus of the reviewers, adds a nice bonus benefit for the method.  


Decision: Accept (Spotlight)