Generalized Depthwise-Separable Convolutions for Adversarially Robust and Efficient Neural Networks
Authors: Hassan Dbouk
Naresh Shanbhag
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: adversarial robustness
efficient inference
generalized depthwise-seprarable
convolutions
neural networks

Abstract: Despite their tremendous successes, convolutional neural networks (CNNs) incur high computational/storage costs and are vulnerable to adversarial perturbations. Recent works on robust model compression address these challenges by combining model compression techniques with adversarial training. But these methods are unable to improve throughput (frames-per-second) on real-life hardware while simultaneously preserving robustness to adversarial perturbations. To overcome this problem, we propose the method of Generalized Depthwise-Separable (GDWS) convolution - an efficient, universal, post-training approximation of a standard 2D convolution. GDWS dramatically improves the throughput of a standard pre-trained network on real-life hardware while preserving its robustness. Lastly, GDWS is scalable to large problem sizes since it operates on pre-trained models and doesn't require any additional training. We establish the optimality of GDWS as a 2D convolution approximator and present exact algorithms for constructing optimal GDWS convolutions under complexity and error constraints. We demonstrate the effectiveness of GDWS via extensive experiments on CIFAR-10, SVHN, and ImageNet datasets. Our code can be found at https://github.com/hsndbk4/GDWS.

Generalized Depthwise-Separable Convolutions for Adversarially Robust and Efficient Neural Networks
Hassan Dbouk Naresh Shanbhag
Department of Electrical and Computer Engineering
University of Illinois Urbana-Char Chammpagnn
Urbana, IL 61801
Abstract
Despite their tremendous successes, convolutional neural networks (CNNs) incur high computaaiionalstorrag costs and vulnerable to adversarial perturbations. Recent works on robust model compression address these challenges by combining model compression techniques with adversarial training. But these methods are unable to improve throughput (framee-per-second) on real-life hardware while simultaneously preserving robustness adversarial perturbations. To overcome this problem, we propose the method of Generalized Septhwise-Separable (GDWS) convolution an efficient, universal, post-training approximation of a standard 2D convolution. GDWS dramatically improves the throughput of a standard pre-trained network on real-life hardware while preserving its robustness. Lastly, GDWS is scalable to large problem sizes since it operates on pre-trained models and doesn't 't require any additional training. We establish the optimality of GDWS as a 2D convolution approximator and present exact algorithms for constructing optimal GDWS convolutions under complexity and error constraints. We demonstrate the effectiveness of GDWS via extensive experiments on CIFAR-10, SVHN, and ImageNet datasets. Our code can found https: //gitttubbb com hsndbk4// GDWS.
Introduction
Nearly decade of research after the release of AlexNet [18] in 2012, convolutional neural networks (CNNs) have unequivocally established themselves as the de facto classification algorithm for various machine learning tasks []1, 38, 4J. The tremendous success of CNNs is often attributed to their unrivaled ability to extract correlations from large volumes of data, allowing them to surpass human level accuracy on some tasks such image classification [11].
Today, the deployment of CNNs in safety-critical Edge applications is hindered due to their high computational costs [11, 30, 31] and their vulnerability to adversarial samples [37, 5, 16]. Tradi- tionally, those two problems have been addressed in isolation. Recently, very few bodies of works [19, 35, 42, 6, 34, 7] have addressed the daunting task of designing both efficient and robust CNNs. majority of these methods focus on model compression, i.e. reducing the storage requirements of CNNs. None have demonstrated their real-time benefits in hardware. For instance, Fig. la shows recent robust pruning works HYDRA [34] and ADMM [42] achieve high compression ratios (up to 97x) but either fail to achieve high throughput measured in frames-per-seconte (FPS) or compromise significantly on robustness. Furthermore, the overreliance of current robust complexity reduction techniques adversarial training (AT) [45, 21] increases their training time significantly (Fig. lb). This prohibits their application to complex ImageNet scale problems with stronger attack models, such as union of norm-bounded perturbations [22]. Thus, there is critical need for methods to
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

AT
ADMM
HYDRA
HYDRA GDWS
GDWS
pretrain
54
x0
3000
retrain
Robust Accuracy [%)
52
(1 ()))
x
Total Time [min]
GDWS
(2
50
(10%)
(19
48
(3.5 x)
2000
46
1500
44
x3
(19 x)
1000
42
(97-X)
+4
40
(8x)
40
100
120 140
Frames Per Second (a)
ADMM HYDRA
(b)
Figure 1: Performance existing robust pruning works (HYDRA [34] and ADMM [42]) and the proposed GDWS with VGG-16 on CIFAR-10, captured by: (a) robust accuracy against too -bundedd perturbations vs frames-per-second measured on an NVIDIA Jetson Xavier, and (b) total time required to implement these methods measured single NVIDIA 1080 Ti GPU. To ensure fair comparison, the same AT baseline (obtained from [34]) is used for all methods. The compression ratio of each method, highlighted in parenthesis, is with respect to the AT baseline.
design deep nets that are both adversarially robust and achieve high throughput when mapped real hardware.
To address this need, we propose Generalized Depthwise-Separable (GDWS) convolutions, universal post-training approximation of a standard 2D convolution that dramatically improves the real hardware FPS of pre-trained networks (Fig. 1a) while preserving their robust accuracy. Interestingly, we find GDWS applied un-pruned robust networks simultaneously achieves higher FPS and higher robustness than robust pruned models obtained from current methods. This in spite of GDWS's compression ratio being smaller than those obtained from robust pruning methods. Furthermore, GDWS easily scales to large problem sizes since it operates pre-trained models and doesn't 't require any additional training.
Our contributions:
We propose GDWS, novel convolutional structure that can be seamlessly mapped onto off-the-shelf hardware and accelerate pre-trained CNNs significantly while maintaining robust accuracy.
2. We show that the error-optimal and complexity-optine GDWS approximations of any pre-trained standard 2D convolution be obtained via greedy polynomial time algorithms, thus eliminating the need for any expensive training.
3. We apply GDWS to variety of networks on CIFAR-10, SVHN, and ImageNet to simultane- ously achieve higher robustness and higher FPS than existing robust complexity reduction techniques, while incurring no extra training cost.
4. We demonstrate the versatility of GDWS by using it to design efficient CNNs that are robust union of (Loc L2, é1) perturbation models. To the best of our knowledge, this the first work that proposes efficient and robust networks to the union of norm-bounded perturbation models.
2 Background and Related Work
The problem of designing efficient and robust CNNs, though crucial for safety-critical Edge applications, not yet well understood. Very few recent works have addressed this problem [19, 35, 42, 6, 34, 7]. We cluster prior works into the following categories:

---

Quantization Reducing the complexity of CNNs via model quantization in the absence of any
adversary is a well studied problem in the deep learning literature [30, 31, 2, 44, 14, 26, 3]. The role of quantization on adversarial robustness was studied in Defensive Quantization (DQ) [19] where was observed that conventional post-training fixed-point quantization makes networks more vulnerable to adversarial perturbations than their full-precisios counterparts. EMPIR [35] also leverages extreme model quantization (up to 2-bits) to build an ensemble of efficient and robust networks. However, [40] broke EMPIR by constructing attacks that fully leverage the model structure, i... adaptive attacks. In contrast, GDWS is an orthogonal complexity reduction technique that preserves the base model's 's adversarial robustness and can be applied in conjunction with model quantization.
Pruning The goal of pruning is to compress neural networks by zeroing out unimportant weights [10, 8, 46, 41J. The structured pruning method [42] combines the alternating direction method of multipliers (ADMM) [46] for parameter pruning within the AT framework [21] to design pruned and robust networks. The flexibility of ADMM enables it to achieve high FPS on Jetson (as seen in Fig. Ia) but suffers from significant drop in robustness. ATMC [6] augments the ADMM framework [42] with model quantization and matrix factorization to further boost the compression ratio. On the other hand, unstructured pruning methods such as HYDRA [34] prunes models via important score optimization [25]. However, HYDRA's high pruning ratios (> 90%) doesn't 't translate into real-time FPS improvements on off-the-shels hardware and often requires custom hardware design to fully leverage their capabilities [9]. GDWS is complementary to unstructured pruning methods, c.g., when applied to HYDRA. GDWS boosts the achievable FPS and achieves much higher robustness at iso-FPS when compared to structured (filter) pruning ADMM
Neural Architecture Search Resource-efficien CNNs can be designed by exploiting design in- tuitions such as depthwise separable (DWS) convolutions [12, 32, 13, 47, 15, 38]. While neural architecture search (NAS) [48, 27] automates the process, it requires massive compute resources, e.g., thousands of GPU hours for a single network. Differentiable NAS [20] and one-shot NAS [I] drastically reduce the cost of this search. In [7]. a one-shot NAS framework [I] is combined with the AT framework [21] ] to search for robust network architectures, called RobNets. RobNets achieve slightly higher robustness than existing networks with less storage requirements. In this work, show that applying GDWS to existing architectures, c.g., WideResNet-28. achieves significantly higher FPS than RobNet, at iso-robustness and model size.
3 Generalized Depthvise-Separable Convolutions
In this section, we introduce GDWS convolutions and develop error-optimal and cmmplexity-optinnee GDWS approximations of standard 2D convolution. These optimal approximations are then employed to construct GDWS networks from any pre-trained robust CNN built from standard 2D convolutions.
Notation: A (C, M) standard 2D convolution operates on an input feature map X E Xx via M filters (also referred to as kernels or output channels) each consisting of C' channels each of dimension K K generate an output feature map R M x xW"
2D Convolution as Matrix Multiplication: The M filters can be viewed as vectors {w; } E RCK* obtained by vectorizing the K2 elements within channel and then across the C channels. The resulting weight matrix W E RMxCK² constructed by stacking these filter vectors, i.e., W = ///////...|ww wrr)".
From operational viewpoint, the matrix W can be used to compute the 2D convolution via Matrix Multiplication (MM) with the input matrix X (xX) RCK3 H''',
Y= WX (X)
(1)
where V is an unrolling operator that generates all H'W" input feature map slices and stacks them in matrix format. The resultant output matrix Y E xMMxH' W' can be reshaped via the operator & to retrieve Y = (YY). The computational complexity of (1) in terms of multipy--ccumuulte (MAC) operations is given by:
H'W'MCK'
(2)
The reshaping operators & and v/ are only for notational convenience and are computation-free
3

---

input feature map
GDW filters
output feature map
9c
: x
91
92
G = Eg
H'
K
91
W
K
W
Figure 2: The (C, K, g) generalized depthwise (GDW) convolution operation. A standard depthwise
(DW) convolution is obtained by setting 9c Ve [C].
3.1 GDWS Formulation
Definition: A (C, K, g, M) GDWS convolution is parameterized by the channel distribution vector g _Φ in addition to the parameters (C, K, MM) of standard 2D convolution. GDWS convolution is composed of a (C, K, g) Generalized Depthwise (GDW) convolution and a (G, 1, standard pointwise (PW) convolution where
Σ 9c with c [C]!.
A (C, K, g) GDW convolutional layer (Fig. 2) operates on an input feature map X E RC by convolving the el channel with Ge 24 depthwise K K filters to produce total of G intermediate output channels each of size H' x W" The (G, 1, .M) PW layer operates on the intermediate output feature map by convolving with M filters of size 1, thus producing the output feature map Y IR x xW"
Relation to DWS: Setting 9. = 1 Vc E [C] reduces the GDWS convolution to the standard DWS convolution popularized by [12]. Thus, GDWS generalizes DWS by allowing for more than one (ge 2 1) depthwise filters per channel. This simple generalization relaxes DWS's highly constrained structure enabling accurate approximations of the 2D convolution. Thus, GDWS when applied to pre-trained models preserves its original behavior and therefore its natural and robust accuracy. Furthermore, GDWS achieves high throughput since it exploits the same hardware features that enable networks with DWS to be implemented efficiently. One might ask: why not use DWS on pre-trained models? Doing so will result in very high approximation errors. In fact, in Section 4.2, we show that applying GDWS to pre-trained complex network such as ResNet- 18 achieves better robust accuracy than MobileNet trained from scratch, while achieving similar FPS.
GDWS Complexity: The total number of MAC operations required by GDWS convolutions is:
H'W' E (K2 + +)) H'W'G(K3 + M)
Thus, replacing standard 2D convolutions with GDWS convolutions results a complexity reduction by factor of G(K*+M) CK*M
3.2 Properties of GDWS Convolutions
We present properties of the GDWS weight matrix W that will be vital for developing the optimal approximation procedures.
Property 1. The weight matrix W M of a (C, K, M) GDWS convolution be expressed as:
W = W,WD
(4)
where Wp E MxG and Wp E R the weight matrices of the PW and GDW convolutions, respectively.
Property implies that any GDWS convolution has equivalent 2D convolution whose weight matrix is the product of Wp and Wp. where Wp is a regular convolution weight matrix with K = and Wp u has the following property:
we the notation C] 21,2,.., C) for brevity.

---

GDW filters
GDW weight matrix Wp
Wi
Wp
W
Vi.c
Var
Vant
x 9c
C
0
G
V12
Va,2
Ya,.
X 9z
Vel
0
0
K
x
Yeal
K²
C
Figure 3: The weight matrix representation of a GDW convolution operation. All the Vi.j vectors row vectors with K" elements.
Property 2. The weight matrix WD E BGxOK² of a (C, K.) GDW convolution has a block- diagonal structure. Specifically, WD a concatenation of C sub-matrices where each sub-matrix WD. E RGxK² has at most ge non-zero consecutive rows, starting at row index 1 + >k=1 9k as shown in Fig. 3.
This structure is due to the fact that input channels are convolved independently with at most ge depthwise filters per channel. Finally, let W [W] WW|/W/|... /W2l..|Wo of C' sub-matrices, then combining Properties & 2 establishes the following lemma:
be represented the concatenation
Lemma 1. The weight matrix of a (C, K, M) GDWS convolution can be expressed as the con- catenation of C sub-matrices W where rank rank(W.) (..) S min(ge: K2) Ve [C].
The reason for this is that each sub-matrix W. can be expressed as the sum of 9c rank matrices of size M x K2, A detailed proof of Lemma can be found in the Appendix. A major implication of Lemma 1 is that any 2D standard convolution is equivalent to GDWS convolution with g. K Ve.. Hence, the rest of this paper we will assume Je K22 when we approximate convolutions with GDWS.
3.3 Optimal GDWS Approximation Methods
We wish to approximate a standard 2D convolution with weight matrix W RMCCK* with GDWS convolution with weight matrix Q M to minimize the weighted approximation error defined as:
C Σ
e(W, Q)
ao W. Qel
(5)
where ||-|/r denotes the Frobenius norm of matrix and a E RCC is a vector of positive weights.
Setting 0₂ = Ve simplifies (5) to the Frobenius norm of the error matrix W Q. Furthermore,
from (3), one can upper bound the complexity of the GDWS approximation Q via an upper bound on
G= Σ ge where the ge are obtained from Lemma
Based on the GDWS properties and Lemma 1, we state the following error-optimal approximation theorem:
Theorem 1. Given (C, K, M) standard 2D convolution with weight matrix W, the (C, K, M) GDWS approximation with weight matrix W that minimizes the error (5) subject to Σg. G S (for some & E Z44, can be obtained in polynomial time via Algorithm 1.
That is:
W arg min e (W, Q)
Q: GST
can be solved any weight error vector a E RC in polynomial time. While Theorem shows that the optimal GDWS approximation under complexity constraint can be solved efficiently, a similar result can be obtained for the reverse setting shown next.
TTypcca CNNs satisfy K² M.M.g. e.g., K and M 16
5

---

Theorem 2. Given a (C, K, M) standard 2D convolution with weight matrix W, the gC, GDWS approximation with weight matrix W that minimizes the complexity in (3) subject to e(W, Q, S 3 (for some 2 0), can be constructed polynomial time via Algorithm 2.
That is:
W
arg min Q: <(W Q8 c=1
Σ 9c
can be solved for any weight error vector E R4 in polynomial time. Proof' of Theorems 1 & can be found in the Appendix.
Algorithm 1: (MEGO) Minimum Error Complexity-constrained GDWS Optimal Approximation
Algorithm 2: (LEGO) Least Complex Error- constrained GDWS Optimal Approximation
Input: A (C, K, M) convolution W, weight error vector a, and constraint 3 2 0. Output: A (C, K, g, M) GDWS
Input: A (C, M) convolution W, weight error vector a, and constraint E 2. Output: (C, K, g, M) GDWS
convolution W. satisfying B.
convolution W, satisfying
1 Compute SVDs of W = E C
Eg 5% Compute SVDs of - - Tiooul,oVi 1.c
2 Initialize 9r = Te 0,
d' = arg min, a.
Tr.er = and
3 while b + h < do
2 Initialize g 0
while Ege < do
4
f b b+ and ge f Ye
c' = max, 0
9,+ 11 De < re
5
d' arg min, ag gaie
11 9c >1
Je' ge' +1
6
= a
Compute W. via truncated SVD of W. with rank 92: W. = Ene Ti Cic U.c i.
7
Compute Wa via truncated SVD of W. with rank 9e: W. E Us
7 Construct W W Wel
8 Construct W [Wi]..[Wcl
3.4 Constructing GDWS Networks
When confronted with a CNN with L convolutional layers, the question arises: How to assign resources (in terms of complexity) amongst the L layers such that the robustness of the CNN is minimally compromised? To answer this question, we compute per-layer weight error vectors ou, such that the computed error in (5) weighs how different sub-matrices affect the final output of the CNN.
Let R + N be pre-trained CNN for an N-way classification problem with L convolutional
layers parameterizd by weight matrices w()) E K2 The CNN operates on D-dimensional input vector x to produce a vector z f(x) of soft outputs or logits. Denote by n. E [N] the predicted class label associated with and define 8z,j = 2j Zn, to be the soft output differences Vj E [N]] \ }nz}.
Inspired by [30, 31J, we propose simple yet effective method for computing the per-layer weight error vectors as follows:
N
ad uF
(c.0
0c.11
E
Σ
V Vc E [Ci]
(8)
M,K7
28
where ((o)) æJ E MRMIIK is the derivative of 82. w.r.t. the sub-matrix w!!) The expectation is taken
the input vector Equation (8) can be thought of the expected noise gain from particular
channel in a particular layer to the network output required to flip its decision. The Appendix provides detailed rationale underlying (8).
Computation of (8) can be simplified by obtaining an estimate of the mean over a small batch of inputs sampled from the training set and by leveraging software frameworks such as Py Torch

---

[24] that automatically take care of computing D D Algorithm 3 summarizes the steps re- quired to approximate any pre-trained CNN with an equivalent CNN utilizing GDWS convo- lutions. Unless specified otherwise, all the results in this paper obtained via Algorithm 3.
Algorithm 3: Constructing GDWS networks
Input: CNN f with convolutional layers (W(), {ai} computed via (8), and constraint
3 2 0.
Output: CNN f with GDWS convolutions (WO)
=
// Initialize f
2 for E {l,..., L. L} do
3
on, Wi( = LEGO W ()
// solve via Algorithm 2
Decompose W into GDW and PW convolutions via Property 1 and Lemma
Replace the 1hh convolution layer in with GDW and PW convolutions
Experiments
4.1 Evaluation Setup
measure the throughput in FPS by mapping the networks onto an NVIDIA Jetson Xavier via native PyTTorch [24] commands. We experiment with VGG-16 [36], ResNet-18 [11], ResNet-50, and WideResNet-28-44 [43] network architectures, and report both natural accuracy (Anaa) and robust accuracy (AAbb)). Following standard procedure, report Arob against too bounded perturbations generated via PGD [21] with standard attack strengths: = 8/255 with PGD-100 for both CIFAR-10 [17] and SVHN [23] datasets, and 4/255 with PGD-50 for the ImageNet [29] dataset. Section 4.3 studies union of multiple perturbation models (fo,, l2, &1). In the absence of publicly released pre- trained models, utilizes early stopping to avoid robust over-fitting. Details on the training/evaluation setup can be found in the Appendix.
establish strong baselines using AT [21] following the approach of [28] which
Table Comparison between RobNet [7] and GDWS on the CIFAR-10 dataset. GDWS is applied to standard pre-traines models.
Models
Auat [%]]
Avb [%
Size [MB]
FPS
RobNet [7]
82.72
52.23
20.8
5
ResNet-50
84.21
53.05
89.7
16
+ GDWS (/B = 0.001)
83.72
52.94
81.9
37
WRN-28-4
84,00
$1.80
22.3
17
GDWS (B3 10-5)
83.27
51.70
18.9
65
ResNet-18
82.41
$1.55
42.6
28
GDWS (B 0.005)
81.17
50.98
29.1
104
VGG-16
77.49
48.92
56.2
36
GDWS (ff 0.25)
77.17
49.56
28.7
129
4.2 Results
Ablation Study: We first show the effectiveness of GDWS on the CIFAR-10 datasets using four network architectures. Table summarizes Ans and Arub well as FPS and model size. It clear that GDWS networks preserve robustness as both Aust and Abo are always within ~1% of their respective baselines. The striking conclusion is that in spite of GDWS offering modest reductions in model size, it drastically improves the FPS of the base network across diverse architectures. For instance, a ResNet-18 utilizing GDWS convolutions is able (>250% improvement) without additional training and without compromising on robust accuracy. In the Appendix, we explore the benefits of applying GDWS using both Algorithms & provide more detailed results on CIFAR-10 10 and show that similar gains are observed with SVHN dataset.
104 FPS compared to the baseline's 28 FPS
For CIFAR-10 10 and SVHN, the standard pre-activation version of ResNets.
7

---

GDWS vs, RobNet: In Table I, we also compare GDWS networks (obtained from standard networks) with a publicly available pre-trained RobNet model, the robust network architecture designed via the NAS framework in [7]. Note that RobNet utilizes DWS convolutions which precludes the use of GDWS. However, despite the efficiency of DWS convolutions RobNet, its irregular cell structure leads to extremely poor mapping on the Jetson as seen by its low 5 FPS. For reference, standard WideResNeet28-44 (WRN-28-4) runs at 17 FPS with similar robustness and model size. Applying GDWS to the WideResNet-288- further increases the throughput to 65 FPS which is 1200% improvement compared to RobNet while maintaining robustness. This further supports our assertion that model compression alone does not lead to enhanced performance real hardware.
GDWS vs. Lightweight Networks: A natural question that might arise from this work: why not train lightweight networks utilizing DWS convolutions from scratch instead of approximating pre-traind complex net- works with GDWS? In Table 2, we compare the performance of GDWS networks (obtained from Table I) vs.
Table 2: Comparison between GDWS and lightweight net- works on the CIFAR-10 dataset. The GDWS numbers from Table 1.
Models
Anat [%]
Arob [%)]
Size [MB]
FPS
ResNet-18 GDWS
81.17
50.98
29.1
104
GDWS
77.17
49.56
28.7
129
MobileNetV1
79.92
49.08
12.3
125
standard lightweight networks: Mo-
MobileNetV2
79.59
48.55
8.5
70
bileNetV1 [12], MobileNetV2 [32],
ResNet-18 (DWS)
80.12
48.52
5.5
120
and ResNet-20 [11], as well as
ResNet-20
74.82
47.00
6.4
125
DWS-version of the standard ResNet-
18 trained from seratch on the CIFAR-
10 dataset. We find that applying GDWS to a pre-trained complex network such as ResNet-18 18
achieves better Anat and Anb than all lightweight networks, while achieving DWS-like FPS and requiring no extra training despite offering modest reductions in model size. The only benefit of using lightweight networks is the much smaller model size compared GDWS networks.
GDWS vs. Structured Pruning: In Table 3, we compare GDWS with the robust structured pruning method ADMM [42] on CIFAR-1O, using two networks: VGG-16 and ResNet-18 18. Due to the lack of publicly available pre-trained models, we use their re- leased code reproduce both the AT baselines, and the corresponding pruned models at differrntt pruning ra- tios. The nature of structured pruming allows ADMM pruned networks (p > 50%) to achieve both high compres- sion ratios and significant improve- ment in FPS over their un-pruned baselines but at the expense of robust- ness and accuracy, For instance, a ResNet-18 with 75% of its channels pruned results in massive 7% (4%) drop Anst (Arah) compared to the baseline even though achieves a 160% improvement in FPS. In contrast, post-training application of GDWS to the same ResNet-18 18 baseline results in massive 275% improvement in FPS while preserving both Anat and Arob within 1% of their baseline values. Thus, despite achieving modest compression ratios compared to ADMM, GDWS achieves comparable improvements in FPS without compromising robustness.
Table 3: Comparison between ADMM [42] and GDWS using VGG-16 and ResNet- on CIFAR-10.
Models
Anat [%]
Anb [%]
Size [MB]
FPS
VGG-16 (AT from (42))
77.45
45.78
56.2
GDWS (8 0.5)
76.40
46.28
38.8
119
VGG-16 (p = 25%)
77.88
43.80
31.6
VGG-16 (p 50%)
75.33
42.93
14.0
113
VGG-16 (p 75%)
70.39
41.07
3.5
174
ResNet-18 (AT from [42))
80.65
47.05
42.6
GDWS (f 0.75)
79.13
46.15
30.4
105
ResNet-18 (p = 25%)
81.61
42.67
32.1
R (p 50%)
79.42
42.23
21.7
60
18 (p 75%)
74.62
43.23
11.2
74
We compare GDWS with HYDRA [34] which is an unstructured robust pruning method, on both CIFAR-10 10 and ImageNet datasets. We use the publicly released HYDRA models as well as their AT baselines, and apply GDWS to both the un un-pruned and pruned models. Table 4 summarizes the robustness and FPS of HYDRA and GDWS networks on CIFAR- 10. HYDRA pruned models have arbitrarily sparse weight matrices that cannot be leveraged by thf- shelf hardware platforms immediately. Instead, we rely on the extremely high sparsity (99%) of these matrices to emulate channel pruning whereby channels are discarded only if all filter weights are zero
GDWS vs. Unstructured Pruning:

---

This explains why. despite their high compression ratios, HYDRA models do not achieve significant improvements in FPS compared to their baselines.
For instance, a 99% HYDRA pruned WideResNet model achieves
Table 4: Comparison between HYDRA [34] and GDWS
110e~~000x compression ratio and im- proves the FPS from 17 28, but suf- fers from a large ~ 10% drop in both Ana and Ayab- In contrast, GDWS applied to the same un un-pruned base- line preserves robustness and achieves significantly better throughput of 68 FPS, even though the model size re- duction is negligible. Interestingly, we find that applying GDWS directly to HYDRA pruned models results in net- works with high compression ratios with no robustness degradation and massive improvements in FPS com- pared to the pruned bascline. For ample. applying GDWS to the same 99% HYDRA pruned WideResNet achieves a ~20 compression ratio and improves the throughput from 28 FPS to 68 FPS while preserving Anat and Aro of the pruned baseline. This synergy between HYDRA and GDWS is due to the fact that highly sparse convolution weight matrices more likely to have low-rank and sparse sub-matrices. This implies that, using Lemma sparse convolutions can be transformed to sparse GDWS versions with negligible approximation error. We explore this synergy in detail in the Appendix. Table shows that GDWS benefits also show up in ImageNet using ResNet-50.
using VGG-16 and WRN-28-4 on CIFAR-IO.
Models
Anot [%]
Arb [%]]
Size MRR
FPS
VGG-16 (AT from (34])
82.72
51.93
58.4
GDWS 0.5)
82.53
50.96
50.6
102
VGG-16 (p = 90%)
80.54
49.44
5.9
GDWS 0.1)
80.47
49.52
31.5
VGG-16 (p 95%)
78.91
48.74 74
3.0
+ GDWS 0.1)
78.71
48.53
18.3
VGG-16 (p = 99%)
73.16
41.74
0.6
GDWS 0.02)
72.75
41.56
136
WRN-28-4 from [34]
85.35
$7.23
22.3
GDWS (3
84.17
55.87
20.5
WRN-28-4 (p 90%)
83.69
55.20
2.3
GDWS 0.125)
83.38
54.79
11.9
WRN-28-4 (p 95%)
82.68
54.18
1.1
GDWS 0.005)
82.59
54.22
7.2
WRN-28-4 (p 99%)
47.21
0.2
GDWS 0.0025)
75.36
47.04
1.2
Table 5: Comparison between HYDRA [34] and GDWS using ResNet-50 on ImageNet.
Models
top-1 5 Ami [%]
top-1 5 Anb [%]
Size [MB]
FPS
ResNet-50 (AT from (34))
60.25 / 82.39
/1.41/6..1
97.5
15
GDWS (B 50)
58.04 / 80,56
30.22 / 58.48
86.2
19
ResNett-0 (p 95%)
44,60 / 70.12
19.53 / 44.28
5.1
15
+ GDWS (/ = 0.5)
43.91 / 69.46
19.27 / /43.58
12.6
19
ResNect50 (p 99%)
27.68 / 52.55
11.32 / 28.83
1.2
17
+ GDWS (Ø 0.5)
26.27 / 50.90
10.92 27.55
2.9
25
4.3 Defending against Union of Perturbation Models
Recent work has shown that adversarial training with a single perturbation model leads to classifiers vulnerable to the union of (Loo, l2, bbuuudde perturbations [33, 39, 22]. The method of multi steepest descent (MSD) [22] achieves state-of-the-e union robust accuracy (A44, against the union of (Exc, &,, &1 bounded perturbations. We demonstrate the versatility of GDWS by applying it to a publicly available [22] robust premMeeD- ResNet-18 model on CIFAR- 10. Following the setup in [22], all attacks were run on a subset of the first 1000 test images with 10 random restarts with the following attack configurations: 6n 0.03 with PGD-100. é2 = 0.5 with PGD-500. and é = 12 with PGD-100. Table 6 shows that applying GDWS with 3 0.01 to the pre-trained ResNet-18 incurs a negligible (<~1%) drop Anat and AU while improving the throughput from 28 FPS to 101 FPS (> 250% improvement).

---

Table 6: Benefits of GDWS when evaluated against union of perturbation models on CIFAR-10. A is the fraction of test images that are simultancously resistant to all perturbation models.
Models
Ams [%]]
Anih [%6
A/bb
[%6]
A [%]]
A [%]]
FPS
ResNet- 18 (AT from [22])
81.74
47.50
53.60
66.10
46.10
28
GDWS (3 = 0.0025)
81.67
47.60
53.60
66.00
46,30
87
GDWS (3 0.005)
81.43
47.30
52.60
65.60
45.70
GDWS (3- 0.01)
81.10
47.20
52.20
65.00
45.20
101
5 Discussion
We have established that the proposed GDWS convolutions are universal and efficient approximations of standard 2D convolutions that are able to accelerate any pre-trained CNN utilizing standard 2D convolution while preserving its accuracy and robustness. This facilitates the deployment of CNNs in safety critical edge applications where real-time decision making is crucial and robustness cannot be compromised. One limitation of this work is that GDWS alone does not achieve high compression ratios compared to pruning. Combining unstructured pruning with GDWS alleviates this problem to some extent. Furthermore, GDWS cannot be applied to CNNs utilizing DWS convolutions, such as RobNet for instance. An interesting question is explore the possibility of training GDWS-structune strucured networks from scratch. Another possible direction is fine-tuning post GDWS approximation to recover robustness, which we explore in the Appendix.
In summary, a GDWS approximated network inherits all the properties, e.g., accuracy, robustness, compression and others, of the baseline CNN while significantly enhancing its throughput (FPS) on real hardware. Therefore, the socictal impact of GDWS approximated networks are also inherited from those of the baseline CNNs.
Acknowledgments
This work was supported by the Center for Brain-Inspired Computing (C-BRIC) and the Artificial Intelligence Hardware (AIHW) program funded by the Semiconductor Research Corporation (SRC) and the Defense Advanced Research Projects Agency (DARPA).
References
[1] Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Under- standing and simplifying one-shot architecture search. In International Conference on Machine Learning, pages 550-559. PMLR, 2018.
[2] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: Parameterized clipping activation for quantized neural networks. arXiv preprint /00.::8805.00000 2018.
[3] Hassan Dbouk, Hetul Sanghvi, Mahesh Mehendale, and Naresh Shanbhag. DBQ: A differ- entiable branch quantizer for lightweight deep neural networks. In European Conference on Computer Vision, pages 90-106. Springer, 2020.
[4] Ross Girshick. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
[5] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint 1xXiv:1412.6572 2014.
[6] Shupeng Gui, Haotao Wang, Chen Yu, Haichuan Yang, Zhangyang Wang, and Ji Liu. Model compression with adversarial robustness: A unified optimization framework. Advances in Neural Information Processing Systems, 2019.
[7] Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu, and Dahua Lin. When nas meets robustness: In search of robust architectures against adversarial attacks. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 6-40,, 2020.
10

---

[8] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs, In Advances Neural Information Processing Systems (NIPS), 2016.
[9] Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark Horowitz, and William Dally. EIE: Efficient inference engine compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):243-254, 2016.
[10] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.0014: 2015.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[12] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprini 174vv:1704.04661, 2017.
[13] Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Q Weinberger. Condensenet: An efficient densenet using learned group convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2752-2761, 2018.
[14] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. In Advances in neural information processing systems, pages 4107-4115, 2016.
[15] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 0.5 MB model size. arXiv preprint arXiv: 1602 07360, 2016.
[16] Andrew Ilyas, Shibani Santurkar, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
[17] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[18] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[19] Lin, Chuang Gan, and Song Han. Defensive quantization: When efficiency meets robustness. In International Conference on Learning Representations, 2019.
[20] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2018
[21] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
[22] Pratyush Maini, Eric Wong, and Zico Kolter. Adversarial robustness against the union of multiple perturbation models. In International Conference on Machine Learning, pages 6640- 6650. PMLR, 2020.
[23] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NeurIPS Workshop Deep Learning and Unsupervised Feature Learning, 2011.
[24] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.

---

[25] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's hidden in a randomly weighted neural network? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 11893-11902, 2020.
[26] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-net: Ima- genet classification using binary convolutional neural networks. In European Conference on Computer Vision, pages 525- 542.. Springer, 2016.
[27] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4780-4789, 2019.
[28] Leslic Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pages 8093-8104 8004. PMLR, 2020.
[29] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International jouraiaaa of computer vision, 1 115(3):211-252 2015.
[30] Charbel Sakr, Yongjune Kim, and Naresh Shanbhag. Analytical guarantees numerical precision of deep neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 3007-3016. JMLR. org, 2017.
[31] Charbel Sakr and Naresh Shanbhag. An analytical method to determine minimum per-layer precision of deep neural networks. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1090-1090 IEEE, 2018
[32] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018
[33] Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adver- sarially robust neural network model on MNIST. In International Conference on Learning Representations, 2019.
[34] Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana. HYDRA: Pruning adversarially robust neural networks. Advances in Neural Information Processing Systems (NeurIPS), 7, 2020.
[35] Sanchari Sen, Balaraman Ravindran, and Anand Raghunathan. EMPIR: Ensembles of mixed precision deep networks for increased robustness against adversarial attacks. In International Conference on Learning Representations, 2019.
[36] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1559.1556, 2014.
[37] Christian Szegedy, Wojeiech Zaremba. Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.619 2013.
[38] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning, pages 6105-6114 14. PMIR, 2019.
[39] Florian Tramèr and Dan Boneh. Adversarial training and robustness for multiple perturbations. In Conference on Neural Information Processing Systems (NeurIPS), 2019.
[40] Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances Neural Information Processing Systems, 33, 2020.
[41] Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. Designing energy-efficient convolutional neural networks using energy- -waar pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5687-5695, 2017.
12

---

[42] Shaokai Ye, Kaidi Xu, Sijia Liu, Hao Cheng, Jan-Henrik Lambrechts, Huan Zbang, Aojun Zhou, Kaisheng Ma, Yanzhi Wang, and Xue Lin. Adversarial robustness vs. model compression, or both? In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 111-120, 2019.
[43] Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146. 2016.
Wide residual networks. arXiv preprint
[44] Dongging Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. LQ-nets: Learned quantiza- tion for highly accurate and compact deep neural networks. In Proceedings of the European Conference Computer Vision (ECCV), pages 365-382, 2018.
[45] Hongyang Zhang, Yaodong Yu, Jiantao Jíao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pages 7472-7482 PMLR, 2019.
[46] Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic DNN weight pruning framework using alternating direction method of multipliers. In Proceedings of the European Conference on Computer Vision (ECCV), pages 184-199, 2018.
[47] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6848 6856, 2018.
[48] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018.
13


Reviews:
[{"review_id": "grdRGO6N0rS", "reviewer": "Reviewer_uhXk", "summary": "1. The paper proposes GDWS with an error optimization strategy to replace regular convolution for enhancing both throughput (FPS) and adversarila robustness.\n2. Experimental results on CIFAR-10, SVHN, and ImageNet validate that the replacing a CNN's regular conv with GDWS can achieve significantly higher FPS and preserve adversarial robustness.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros.\n1. Motivation, improving throughput and adversarial robustness, is clearly written and valuable.\n2. The proposed method is implementable. Code is provided.\n3. The paper formally provides theoretical analysis for error approximation of the proposed GDWS.\n4. The proposed GDWS is directly applied to a pretrained model, i.e., no retraining is required.\n5. Experimental results are convincing and promising.\n\nCons.\n1. Comparison between the proposed GDWS and traditional DWS is not fair. In Section 3.1, the authors argues advantages of GDWS over DWS, e.g., higher throughput and better robustness. I agree that a GDWS-based CNN have such advantages compared to a DWS-based CNN, as shown in Experiments (Section 4). However, I think the advantages of a GDWS-based CNN come from two aspects: error optimization as shown in Algorithm 3 and replacing regular conv with GDWS, but not the formulation of GDWS itself. Compared to the formulation of DWS, GDWS tends to have more kernels which enlarges parameter space but also increases computation burden, as explained in Section 3.1. If all DWS in a CNN can be optimized by an algorithm similar with Algorithm 3, I think a DWS-based CNN can also have improved robustness. Furthermore, current experiments \"ResNet vs. ResNet+GDWS\" are not best examples to show advantage on higher throughput (FPS), since ResNet adopts regular conv but not DWS. I think the following experiment is more appropriate:  \"MobileNet-V1 vs. Replacing MobileNet-V1 with GDWS\".\n2. The proposed method indirectly but not directly improve adversarial robustness. As shown in Table 5, adversarial robustness is indeed enhanced. However, main techniques of the proposed method, as shown in Section 3.3 and Section 3.4, work on minimizing errors (i.e., accuracy loss) brought by using GDWS but not considering adversarial perturbations. So I think enhancing  adversarial robustness is a by-product of using GDWS but not an unique advantage (or direct target) of using GDWS. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations\n1) The paper does not introduce limitations of the proposed method.\n2) As mentioned in Cons. 2), I think a potential limitation is that GWDS cannot significantly improve throughput and adversarial robustness if a mobile CNN adopts DWS or similar techniques already.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "dXxDjlyBBgq", "reviewer": "Reviewer_t8ZX", "summary": "The paper proposes a post-training transformation of a pre-trained 2D convolutional layer into a general depth-wise separable convolution operation. The optimization primarily focuses on minimizing error and limiting computation complexity. The paper claims to achieve a higher robustness (i.e., defend adversarial input attack) and higher frame per second than existing methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "i) Originality: The proposed method is a novel technique in particular for two reasons:\nFirst, the idea of post training transformation of the Convolution operation to a depth-wise separable convolution certainly carries significant weight for hardware application of these models (e.g., speed-up). Second, the method is well-designed to minimize the error after GDWS so that both baseline accuracy and accuracy under attack remains fairly stable.  The authors have missed to cite relevant dynamic pruning and dynamic architecture works. I think dynamic neural networks would have shown more competitive performance in terms of efficiency.\n\nii) Clarity: The paper is well-written and easy to follow.  \n\niii) Significance: The results presented in table 1 & 2 clearly validate that the proposed method improves the FPS in comparison to the prior methods.  While the computation gain is impressive, the lower inference accuracy and in particular the A_{rob} raises a few concerns regarding the initial claim of the authors.\n\nFirst, looking at the method presented in the paper, the first question is why not directly use a MobileNet-v2 model? The authors did answer this question in line 129-131 and stated that [7] already showed the in-efficacy of mobilenet-v2 against PGD attack. However, when I open reference [7] and check table 7 [7], it shows mobilenet-v2 is more robust than both ResNet-18 and ResNet-50 models. On the contrary ,  applying GDWS on any ResNet model would cause slight accuracy degradation against PGD. Can the author clarify this issue? In addition the comparison with Robnet in table 2 is not clear.  Looks like RobNet has many variants, certainly the one chosen for table 2 is not performing well.  \n\nSecond,  In general based on my first point, I would emphasize that the robustness claim of this work is weak. In all cases, the method seems to improve FPS with slight degradation of A_{nat} and A_{rob}. Hence, the contribution is mainly coming from the efficiency side. I would suggest the authors modify the robustness claim  of line 53: “achieve higher robustness and higher FPS than existing robust complexity reduction”. I think the correct claim is in line 240 “without additional training and without compromising on robust accuracy”. I think the strength of the method is it does not require any training and does not compromise robust accuracy much. However, in comparison to ADMM [42] it is showing inferior performance under attack.\n\nThird, I am just curious to know if this method is applicable during training? Since you are applying to PGD, all the steps should be differentiable and hence the method should be applicable for the training phase.  What happens if you try to fine-tune the model after post-training GDWS? Or if you train it from scratch would that be similar to mobilenet training? What is the FPS of MobileNet-v2? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any negative social impact of this work", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see any ethical issue concerning this work.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "3IjTO37DfUK", "reviewer": "Reviewer_7tWD", "summary": "This paper proposed a method named  Generalized Depthwise-Separable (GDWS) to improve the throughput on real-life hardware while simultaneously preserving robustness. The advantage of GDWS is that does not need any additional training.  This paper has demonstrated the effectiveness of GDWS via extensive experiments on CIFAR-10, SVHN, and ImageNet datasets.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Originality: The article clearly describes the necessity and relevance of the research.\n- Quality: For the proposed method, I don't see any flaws. The experiment is relatively complete, and could potentially support their claims.\n- Clarity: This paper is well written and easy to follow.\n- Significance: From the experimental results, the proposed GDWS has a significant increase in network throughput and the decline in accuracy is within an acceptable range. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have some suggestions that may help the author to further improve the quality of the article.\n1.The author may consider adding some verifications for tasks that rely more heavily on throughput, such as real-time target detection, real-time depth estimation. \n2.The authors need to describe the experimental environment in more detail, such as the CUDA version and the PyTorch version. Because different versions of the experimental environment will have a certain impact on training speed and inference speed.\n3. The description of \"W_D has the following property\" could change to\"the property of W_D is shown in Fig.3\" which may be more clear.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach for converting convolutions into depth-wise separable convolutions after training that offers advantages in terms of both throughput and adversarial robustness. The reviewers all agree that the proposed method is clever and the demonstrated performance boost is significant.  The adversarial robustness claims, while not a major focus of the reviewers, adds a nice bonus benefit for the method.  


Decision: Accept (Spotlight)