Generalized Depthwise-Separable Convolutions for Adversarially Robust and Efficient Neural Networks
Authors: Hassan Dbouk
Naresh Shanbhag
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: adversarial robustness
efficient inference
generalized depthwise-seprarable
convolutions
neural networks

Abstract: Despite their tremendous successes, convolutional neural networks (CNNs) incur high computational/storage costs and are vulnerable to adversarial perturbations. Recent works on robust model compression address these challenges by combining model compression techniques with adversarial training. But these methods are unable to improve throughput (frames-per-second) on real-life hardware while simultaneously preserving robustness to adversarial perturbations. To overcome this problem, we propose the method of Generalized Depthwise-Separable (GDWS) convolution - an efficient, universal, post-training approximation of a standard 2D convolution. GDWS dramatically improves the throughput of a standard pre-trained network on real-life hardware while preserving its robustness. Lastly, GDWS is scalable to large problem sizes since it operates on pre-trained models and doesn't require any additional training. We establish the optimality of GDWS as a 2D convolution approximator and present exact algorithms for constructing optimal GDWS convolutions under complexity and error constraints. We demonstrate the effectiveness of GDWS via extensive experiments on CIFAR-10, SVHN, and ImageNet datasets. Our code can be found at https://github.com/hsndbk4/GDWS.

Generalized Depthwise-Separable Convolutions for Adversarially Robust and Efficient Neural Networks
Hassan Dbouk Naresh Shanbhag
Department of Electrical and Computer Engineering
University of Illinois Urbana-Char Chammpagnn
Urbana, IL 61801
Abstract
Despite their tremendous successes, convolutional neural networks (CNNs) incur high computaaiionalstorrag costs and vulnerable to adversarial perturbations. Recent works on robust model compression address these challenges by combining model compression techniques with adversarial training. But these methods are unable to improve throughput (framee-per-second) on real-life hardware while simultaneously preserving robustness adversarial perturbations. To overcome this problem, we propose the method of Generalized Septhwise-Separable (GDWS) convolution an efficient, universal, post-training approximation of a standard 2D convolution. GDWS dramatically improves the throughput of a standard pre-trained network on real-life hardware while preserving its robustness. Lastly, GDWS is scalable to large problem sizes since it operates on pre-trained models and doesn't 't require any additional training. We establish the optimality of GDWS as a 2D convolution approximator and present exact algorithms for constructing optimal GDWS convolutions under complexity and error constraints. We demonstrate the effectiveness of GDWS via extensive experiments on CIFAR-10, SVHN, and ImageNet datasets. Our code can found https: //gitttubbb com hsndbk4// GDWS.
Introduction
Nearly decade of research after the release of AlexNet [18] in 2012, convolutional neural networks (CNNs) have unequivocally established themselves as the de facto classification algorithm for various machine learning tasks []1, 38, 4J. The tremendous success of CNNs is often attributed to their unrivaled ability to extract correlations from large volumes of data, allowing them to surpass human level accuracy on some tasks such image classification [11].
Today, the deployment of CNNs in safety-critical Edge applications is hindered due to their high computational costs [11, 30, 31] and their vulnerability to adversarial samples [37, 5, 16]. Tradi- tionally, those two problems have been addressed in isolation. Recently, very few bodies of works [19, 35, 42, 6, 34, 7] have addressed the daunting task of designing both efficient and robust CNNs. majority of these methods focus on model compression, i.e. reducing the storage requirements of CNNs. None have demonstrated their real-time benefits in hardware. For instance, Fig. la shows recent robust pruning works HYDRA [34] and ADMM [42] achieve high compression ratios (up to 97x) but either fail to achieve high throughput measured in frames-per-seconte (FPS) or compromise significantly on robustness. Furthermore, the overreliance of current robust complexity reduction techniques adversarial training (AT) [45, 21] increases their training time significantly (Fig. lb). This prohibits their application to complex ImageNet scale problems with stronger attack models, such as union of norm-bounded perturbations [22]. Thus, there is critical need for methods to
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

AT
ADMM
HYDRA
HYDRA GDWS
GDWS
pretrain
54
x0
3000
retrain
Robust Accuracy [%)
52
(1 ()))
x
Total Time [min]
GDWS
(2
50
(10%)
(19
48
(3.5 x)
2000
46
1500
44
x3
(19 x)
1000
42
(97-X)
+4
40
(8x)
40
100
120 140
Frames Per Second (a)
ADMM HYDRA
(b)
Figure 1: Performance existing robust pruning works (HYDRA [34] and ADMM [42]) and the proposed GDWS with VGG-16 on CIFAR-10, captured by: (a) robust accuracy against too -bundedd perturbations vs frames-per-second measured on an NVIDIA Jetson Xavier, and (b) total time required to implement these methods measured single NVIDIA 1080 Ti GPU. To ensure fair comparison, the same AT baseline (obtained from [34]) is used for all methods. The compression ratio of each method, highlighted in parenthesis, is with respect to the AT baseline.
design deep nets that are both adversarially robust and achieve high throughput when mapped real hardware.
To address this need, we propose Generalized Depthwise-Separable (GDWS) convolutions, universal post-training approximation of a standard 2D convolution that dramatically improves the real hardware FPS of pre-trained networks (Fig. 1a) while preserving their robust accuracy. Interestingly, we find GDWS applied un-pruned robust networks simultaneously achieves higher FPS and higher robustness than robust pruned models obtained from current methods. This in spite of GDWS's compression ratio being smaller than those obtained from robust pruning methods. Furthermore, GDWS easily scales to large problem sizes since it operates pre-trained models and doesn't 't require any additional training.
Our contributions:
We propose GDWS, novel convolutional structure that can be seamlessly mapped onto off-the-shelf hardware and accelerate pre-trained CNNs significantly while maintaining robust accuracy.
2. We show that the error-optimal and complexity-optine GDWS approximations of any pre-trained standard 2D convolution be obtained via greedy polynomial time algorithms, thus eliminating the need for any expensive training.
3. We apply GDWS to variety of networks on CIFAR-10, SVHN, and ImageNet to simultane- ously achieve higher robustness and higher FPS than existing robust complexity reduction techniques, while incurring no extra training cost.
4. We demonstrate the versatility of GDWS by using it to design efficient CNNs that are robust union of (Loc L2, é1) perturbation models. To the best of our knowledge, this the first work that proposes efficient and robust networks to the union of norm-bounded perturbation models.
2 Background and Related Work
The problem of designing efficient and robust CNNs, though crucial for safety-critical Edge applications, not yet well understood. Very few recent works have addressed this problem [19, 35, 42, 6, 34, 7]. We cluster prior works into the following categories:

---

Quantization Reducing the complexity of CNNs via model quantization in the absence of any
adversary is a well studied problem in the deep learning literature [30, 31, 2, 44, 14, 26, 3]. The role of quantization on adversarial robustness was studied in Defensive Quantization (DQ) [19] where was observed that conventional post-training fixed-point quantization makes networks more vulnerable to adversarial perturbations than their full-precisios counterparts. EMPIR [35] also leverages extreme model quantization (up to 2-bits) to build an ensemble of efficient and robust networks. However, [40] broke EMPIR by constructing attacks that fully leverage the model structure, i... adaptive attacks. In contrast, GDWS is an orthogonal complexity reduction technique that preserves the base model's 's adversarial robustness and can be applied in conjunction with model quantization.
Pruning The goal of pruning is to compress neural networks by zeroing out unimportant weights [10, 8, 46, 41J. The structured pruning method [42] combines the alternating direction method of multipliers (ADMM) [46] for parameter pruning within the AT framework [21] to design pruned and robust networks. The flexibility of ADMM enables it to achieve high FPS on Jetson (as seen in Fig. Ia) but suffers from significant drop in robustness. ATMC [6] augments the ADMM framework [42] with model quantization and matrix factorization to further boost the compression ratio. On the other hand, unstructured pruning methods such as HYDRA [34] prunes models via important score optimization [25]. However, HYDRA's high pruning ratios (> 90%) doesn't 't translate into real-time FPS improvements on off-the-shels hardware and often requires custom hardware design to fully leverage their capabilities [9]. GDWS is complementary to unstructured pruning methods, c.g., when applied to HYDRA. GDWS boosts the achievable FPS and achieves much higher robustness at iso-FPS when compared to structured (filter) pruning ADMM
Neural Architecture Search Resource-efficien CNNs can be designed by exploiting design in- tuitions such as depthwise separable (DWS) convolutions [12, 32, 13, 47, 15, 38]. While neural architecture search (NAS) [48, 27] automates the process, it requires massive compute resources, e.g., thousands of GPU hours for a single network. Differentiable NAS [20] and one-shot NAS [I] drastically reduce the cost of this search. In [7]. a one-shot NAS framework [I] is combined with the AT framework [21] ] to search for robust network architectures, called RobNets. RobNets achieve slightly higher robustness than existing networks with less storage requirements. In this work, show that applying GDWS to existing architectures, c.g., WideResNet-28. achieves significantly higher FPS than RobNet, at iso-robustness and model size.
3 Generalized Depthvise-Separable Convolutions
In this section, we introduce GDWS convolutions and develop error-optimal and cmmplexity-optinnee GDWS approximations of standard 2D convolution. These optimal approximations are then employed to construct GDWS networks from any pre-trained robust CNN built from standard 2D convolutions.
Notation: A (C, M) standard 2D convolution operates on an input feature map X E Xx via M filters (also referred to as kernels or output channels) each consisting of C' channels each of dimension K K generate an output feature map R M x xW"
2D Convolution as Matrix Multiplication: The M filters can be viewed as vectors {w; } E RCK* obtained by vectorizing the K2 elements within channel and then across the C channels. The resulting weight matrix W E RMxCK² constructed by stacking these filter vectors, i.e., W = ///////...|ww wrr)".
From operational viewpoint, the matrix W can be used to compute the 2D convolution via Matrix Multiplication (MM) with the input matrix X (xX) RCK3 H''',
Y= WX (X)
(1)
where V is an unrolling operator that generates all H'W" input feature map slices and stacks them in matrix format. The resultant output matrix Y E xMMxH' W' can be reshaped via the operator & to retrieve Y = (YY). The computational complexity of (1) in terms of multipy--ccumuulte (MAC) operations is given by:
H'W'MCK'
(2)
The reshaping operators & and v/ are only for notational convenience and are computation-free
3

---

input feature map
GDW filters
output feature map
9c
: x
91
92
H
G = Eg
H'
K
91
W
K
W
Figure 2: The (C, K, g) generalized depthwise (GDW) convolution operation. A standard depthwise
(DW) convolution is obtained by setting 9c Ve [C].
3.1 GDWS Formulation
Definition: A (C, K, g, M) GDWS convolution is parameterized by the channel distribution vector g _Φ in addition to the parameters (C, K, MM) of standard 2D convolution. GDWS convolution is composed of a (C, K, g) Generalized Depthwise (GDW) convolution and a (G, 1, standard pointwise (PW) convolution where
Σ 9c with c [C]!.
A (C, K, g) GDW convolutional layer (Fig. 2) operates on an input feature map X E RC by convolving the el channel with Ge 24 depthwise K K filters to produce total of G intermediate output channels each of size H' x W" The (G, 1, .M) PW layer operates on the intermediate output feature map by convolving with M filters of size 1, thus producing the output feature map Y IR x xW"
Relation to DWS: Setting 9. = 1 Vc E [C] reduces the GDWS convolution to the standard DWS convolution popularized by [12]. Thus, GDWS generalizes DWS by allowing for more than one (ge 2 1) depthwise filters per channel. This simple generalization relaxes DWS's highly constrained structure enabling accurate approximations of the 2D convolution. Thus, GDWS when applied to pre-trained models preserves its original behavior and therefore its natural and robust accuracy. Furthermore, GDWS achieves high throughput since it exploits the same hardware features that enable networks with DWS to be implemented efficiently. One might ask: why not use DWS on pre-trained models? Doing so will result in very high approximation errors. In fact, in Section 4.2, we show that applying GDWS to pre-trained complex network such as ResNet- 18 achieves better robust accuracy than MobileNet trained from scratch, while achieving similar FPS.
GDWS Complexity: The total number of MAC operations required by GDWS convolutions is:
H'W' E (K2 + +)) H'W'G(K3 + M)
Thus, replacing standard 2D convolutions with GDWS convolutions results a complexity reduction by factor of G(K*+M) CK*M
3.2 Properties of GDWS Convolutions
We present properties of the GDWS weight matrix W that will be vital for developing the optimal approximation procedures.
Property 1. The weight matrix W M of a (C, K, M) GDWS convolution be expressed as:
W = W,WD
(4)
where Wp E MxG and Wp E R the weight matrices of the PW and GDW convolutions, respectively.
Property implies that any GDWS convolution has equivalent 2D convolution whose weight matrix is the product of Wp and Wp. where Wp is a regular convolution weight matrix with K = and Wp u has the following property:
we the notation C] 21,2,.., C) for brevity.

---

GDW filters
GDW weight matrix Wp
Wi
Wp
W
Vi.c
Var
Vant
x 9c
C
0
G
V12
Va,2
Ya,.
X 9z
Vel
0
0
K
x
Yeal
K²
C
Figure 3: The weight matrix representation of a GDW convolution operation. All the Vi.j vectors row vectors with K" elements.
Property 2. The weight matrix WD E BGxOK² of a (C, K.) GDW convolution has a block- diagonal structure. Specifically, WD a concatenation of C sub-matrices where each sub-matrix WD. E RGxK² has at most ge non-zero consecutive rows, starting at row index 1 + >k=1 9k as shown in Fig. 3.
This structure is due to the fact that input channels are convolved independently with at most ge depthwise filters per channel. Finally, let W [W] WW|/W/|... /W2l..|Wo of C' sub-matrices, then combining Properties & 2 establishes the following lemma:
be represented the concatenation
Lemma 1. The weight matrix of a (C, K, M) GDWS convolution can be expressed as the con- catenation of C sub-matrices W where rank rank(W.) (..) S min(ge: K2) Ve [C].
The reason for this is that each sub-matrix W. can be expressed as the sum of 9c rank matrices of size M x K2, A detailed proof of Lemma can be found in the Appendix. A major implication of Lemma 1 is that any 2D standard convolution is equivalent to GDWS convolution with g. K Ve.. Hence, the rest of this paper we will assume Je K22 when we approximate convolutions with GDWS.
3.3 Optimal GDWS Approximation Methods
We wish to approximate a standard 2D convolution with weight matrix W RMCCK* with GDWS convolution with weight matrix Q M to minimize the weighted approximation error defined as:
C Σ
e(W, Q)
ao W. Qol
(5)
where ||-|/r denotes the Frobenius norm of matrix and a E RCC is a vector of positive weights.
Setting 0₂ = Ve simplifies (5) to the Frobenius norm of the error matrix W Q. Furthermore,
from (3), one can upper bound the complexity of the GDWS approximation Q via an upper bound on
G= Σ ge where the ge are obtained from Lemma
Based on the GDWS properties and Lemma 1, we state the following error-optimal approximation theorem:
Theorem 1. Given (C, K, M) standard 2D convolution with weight matrix W, the (C, K, M) GDWS approximation with weight matrix W that minimizes the error (5) subject to Eg. G S (for some & E Z44, can be obtained in polynomial time via Algorithm 1.
That is:
W arg min e (W, Q)
Q: GST
can be solved any weight error vector a E RC in polynomial time. While Theorem shows that the optimal GDWS approximation under complexity constraint can be solved efficiently, a similar result can be obtained for the reverse setting shown next.
TTypcca CNNs satisfy K² M.M.g. e.g., K and M 16
5

---

Theorem 2. Given a (C, K, M) standard 2D convolution with weight matrix W, the gC, GDWS approximation with weight matrix W that minimizes the complexity in (3) subject to e(W, Q, S 3 (for some 2 0), can be constructed polynomial time via Algorithm 2.
That is:
W
arg min Q: <(W Q8 c=1
Σ 9c
can be solved for any weight error vector E R4 in polynomial time. Proof' of Theorems 1 & can be found in the Appendix.
Algorithm 1: (MEGO) Minimum Error Complexity-constrained GDWS Optimal Approximation
Algorithm 2: (LEGO) Least Complex Error- constrained GDWS Optimal Approximation
Input: A (C, K, M) convolution W, weight error vector a, and constraint 3 2 0. Output: A (C, K, g, M) GDWS
Input: A (C, M) convolution W, weight error vector a, and constraint E 2. Output: (C, K, g, M) GDWS
convolution W. satisfying B.
convolution W, satisfying
1 Compute SVDs of W = E C
Eg 5% Compute SVDs of -
2 Initialize 9r = Te 0,
d' = arg min, a. Tr.er = and
- Tiooul,oVi 1.c
3 while b + h < do
2 Initialize g 0
while Ege < do
4
f b b+ and ge f Ye
c' = max, 0
9,+ 11 De < re
5
d' arg min ag gaie
11 9c >1
Je' ge' +1
6
= a
Compute W. via truncated SVD of W. with rank 92: W. = Ene Ti Cic U.c i.
7
Compute Wa via truncated SVD of W. with rank 9e: W. E Us
7 Construct W W Wel
8 Construct W [Wi]..[Wcl
3.4 Constructing GDWS Networks
When confronted with a CNN with L convolutional layers, the question arises: How to assign resources (in terms of complexity) amongst the L layers such that the robustness of the CNN is minimally compromised? To answer this question, we compute per-layer weight error vectors ou, such that the computed error in (5) weighs how different sub-matrices affect the final output of the CNN.
Let R + N be pre-trained CNN for an N-way classification problem with L convolutional
layers parameterizd by weight matrices w()) E K2 The CNN operates on D-dimensional input vector x to produce a vector z f(x) of soft outputs or logits. Denote by n. E [N] the predicted class label associated with and define 8z,j = 2j Zn, to be the soft output differences Vj E [N]] \ }nz}.
Inspired by [30, 31J, we propose simple yet effective method for computing the per-layer weight error vectors as follows:
N
ad uF
(c.0
0c.11
E
Σ
V Vc E [Ci]
(8)
M,K7
28
where ((o)) æJ E MRMIIK is the derivative of 82. w.r.t. the sub-matrix w!!) The expectation is taken
the input vector Equation (8) can be thought of the expected noise gain from particular
channel in a particular layer to the network output required to flip its decision. The Appendix provides detailed rationale underlying (8).
Computation of (8) can be simplified by obtaining an estimate of the mean over a small batch of inputs sampled from the training set and by leveraging software frameworks such as Py Torch

---

[24] that automatically take care of computing D D Algorithm 3 summarizes the steps re- quired to approximate any pre-trained CNN with an equivalent CNN utilizing GDWS convo- lutions. Unless specified otherwise, all the results in this paper obtained via Algorithm 3.
Algorithm 3: Constructing GDWS networks
Input: CNN f with convolutional layers (W(), {ai} computed via (8), and constraint
3 2 0.
Output: CNN f with GDWS convolutions (WO)
=
// Initialize f
2 for E {l,..., L. L} do
3
on, Wi( = LEGO W ()
// solve via Algorithm 2
Decompose W into GDW and PW convolutions via Property 1 and Lemma
Replace the 1hh convolution layer in with GDW and PW convolutions
Experiments
4.1 Evaluation Setup
measure the throughput in FPS by mapping the networks onto an NVIDIA Jetson Xavier via native PyTTorch [24] commands. We experiment with VGG-16 [36], ResNet-18 [11], ResNet-50, and WideResNet-28-44 [43] network architectures, and report both natural accuracy (Anaa) and robust accuracy (AAbb)). Following standard procedure, report Arob against too bounded perturbations generated via PGD [21] with standard attack strengths: = 8/255 with PGD-100 for both CIFAR-10 [17] and SVHN [23] datasets, and 4/255 with PGD-50 for the ImageNet [29] dataset. Section 4.3 studies union of multiple perturbation models (fo,, l2, &1). In the absence of publicly released pre- trained models, utilizes early stopping to avoid robust over-fitting. Details on the training/evaluation setup can be found in the Appendix.
establish strong baselines using AT [21] following the approach of [28] which
Table Comparison between RobNet [7] and GDWS on the CIFAR-10 dataset. GDWS is applied to standard pre-traines models.
Models
Auat [%]]
Avb [%
Size [MB]
FPS
RobNet [7]
82.72
52.23
20.8
5
ResNet-50
84.21
53.05
89.7
16
+ GDWS (/B = 0.001)
83.72
52.94
81.9
37
WRN-28-4
84,00
$1.80
22.3
17
GDWS (B3 10-5)
83.27
51.70
18.9
65
ResNet-18
82.41
$1.55
42.6
28
GDWS (B 0.005)
81.17
50.98
29.1
104
VGG-16
77.49
48.92
56.2
36
GDWS (ff 0.25)
77.17
49.56
28.7
129
4.2 Results
Ablation Study: We first show the effectiveness of GDWS on the CIFAR-10 datasets using four network architectures. Table summarizes Ans and Arub well as FPS and model size. It clear that GDWS networks preserve robustness as both Aust and Abo are always within ~1% of their respective baselines. The striking conclusion is that in spite of GDWS offering modest reductions in model size, it drastically improves the FPS of the base network across diverse architectures. For instance, a ResNet-18 utilizing GDWS convolutions is able (>250% improvement) without additional training and without compromising on robust accuracy. In the Appendix, we explore the benefits of applying GDWS using both Algorithms & provide more detailed results on CIFAR-10 10 and show that similar gains are observed with SVHN dataset.
104 FPS compared to the baseline's 28 FPS
For CIFAR-10 10 and SVHN, the standard pre-activation version of ResNets.
7

---

GDWS vs, RobNet: In Table I, we also compare GDWS networks (obtained from standard networks) with a publicly available pre-trained RobNet model, the robust network architecture designed via the NAS framework in [7]. Note that RobNet utilizes DWS convolutions which precludes the use of GDWS. However, despite the efficiency of DWS convolutions RobNet, its irregular cell structure leads to extremely poor mapping on the Jetson as seen by its low 5 FPS. For reference, standard WideResNeet28-44 (WRN-28-4) runs at 17 FPS with similar robustness and model size. Applying GDWS to the WideResNet-288- further increases the throughput to 65 FPS which is 1200% improvement compared to RobNet while maintaining robustness. This further supports our assertion that model compression alone does not lead to enhanced performance real hardware.
GDWS vs. Lightweight Networks: A natural question that might arise from this work: why not train lightweight networks utilizing DWS convolutions from scratch instead of approximating pre-traind complex net- works with GDWS? In Table 2, we compare the performance of GDWS networks (obtained from Table I) vs.
Table 2: Comparison between GDWS and lightweight net- works on the CIFAR-10 dataset. The GDWS numbers from Table 1.
Models
Anat [%]
Arob [%)]
Size [MB]
FPS
ResNet-18 GDWS
81.17
50.98
29.1
104
GDWS
77.17
49.56
28.7
129
MobileNetV1
79.92
49.08
12.3
125
standard lightweight networks: Mo-
MobileNetV2
79.59
48.55
8.5
70
bileNetV1 [12], MobileNetV2 [32],
ResNet-18 (DWS)
80.12
48.52
5.5
120
and ResNet-20 [11], as well as
ResNet-20
74.82
47.00
6.4
125
DWS-version of the standard ResNet-
18 trained from seratch on the CIFAR-
10 dataset. We find that applying GDWS to a pre-trained complex network such as ResNet-18 18
achieves better Anat and Anb than all lightweight networks, while achieving DWS-like FPS and requiring no extra training despite offering modest reductions in model size. The only benefit of using lightweight networks is the much smaller model size compared GDWS networks.
GDWS vs. Structured Pruning: In Table 3, we compare GDWS with the robust structured pruning method ADMM [42] on CIFAR-1O, using two networks: VGG-16 and ResNet-18 18. Due to the lack of publicly available pre-trained models, we use their re- leased code reproduce both the AT baselines, and the corresponding pruned models at differrntt pruning ra- tios. The nature of structured pruming allows ADMM pruned networks (p > 50%) to achieve both high compres- sion ratios and significant improve- ment in FPS over their un-pruned baselines but at the expense of robust- ness and accuracy, For instance, a ResNet-18 with 75% of its channels pruned results in massive 7% (4%) drop Anst (Arah) compared to the baseline even though achieves a 160% improvement in FPS. In contrast, post-training application of GDWS to the same ResNet-18 18 baseline results in massive 275% improvement in FPS while preserving both Anat and Arob within 1% of their baseline values. Thus, despite achieving modest compression ratios compared to ADMM, GDWS achieves comparable improvements in FPS without compromising robustness.
Table 3: Comparison between ADMM [42] and GDWS using VGG-16 and ResNet- on CIFAR-10.
Models
Anat [%]
Anb [%]
Size [MB]
FPS
VGG-16 (AT from (42))
77.45
45.78
56.2
GDWS (8 0.5)
76.40
46.28
38.8
119
VGG-16 (p = 25%)
77.88
43.80
31.6
VGG-16 (p 50%)
75.33
42.93
14.0
113
VGG-16 (p 75%)
70.39
41.07
3.5
174
ResNet-18 (AT from [42))
80.65
47.05
42.6
GDWS (f 0.75)
79.13
46.15
30.4
105
ResNet-18 (p = 25%)
81.61
42.67
32.1
R (p 50%)
79.42
42.23
21.7
60
18 (p 75%)
74.62
43.23
11.2
74
We compare GDWS with HYDRA [34] which is an unstructured robust pruning method, on both CIFAR-10 10 and ImageNet datasets. We use the publicly released HYDRA models as well as their AT baselines, and apply GDWS to both the un un-pruned and pruned models. Table 4 summarizes the robustness and FPS of HYDRA and GDWS networks on CIFAR- 10. HYDRA pruned models have arbitrarily sparse weight matrices that cannot be leveraged by thf- shelf hardware platforms immediately. Instead, we rely on the extremely high sparsity (99%) of these matrices to emulate channel pruning whereby channels are discarded only if all filter weights are zero
GDWS vs. Unstructured Pruning:

---

This explains why. despite their high compression ratios, HYDRA models do not achieve significant improvements in FPS compared to their baselines.
For instance, a 99% HYDRA pruned WideResNet model achieves
Table 4: Comparison between HYDRA [34] and GDWS
110e~~000x compression ratio and im- proves the FPS from 17 28, but suf- fers from a large ~ 10% drop in both Ana and Ayab- In contrast, GDWS applied to the same un un-pruned base- line preserves robustness and achieves significantly better throughput of 68 FPS, even though the model size re- duction is negligible. Interestingly, we find that applying GDWS directly to HYDRA pruned models results in net- works with high compression ratios with no robustness degradation and massive improvements in FPS com- pared to the pruned bascline. For ample. applying GDWS to the same 99% HYDRA pruned WideResNet achieves a ~20 compression ratio and improves the throughput from 28 FPS to 68 FPS while preserving Anat and Aro of the pruned baseline. This synergy between HYDRA and GDWS is due to the fact that highly sparse convolution weight matrices more likely to have low-rank and sparse sub-matrices. This implies that, using Lemma sparse convolutions can be transformed to sparse GDWS versions with negligible approximation error. We explore this synergy in detail in the Appendix. Table shows that GDWS benefits also show up in ImageNet using ResNet-50.
using VGG-16 and WRN-28-4 on CIFAR-IO.
Models
Anot [%]
Arb [%]]
Size MRR
FPS
VGG-16 (AT from (34])
82.72
51.93
58.4
GDWS 0.5)
82.53
50.96
50.6
102
VGG-16 (p = 90%)
80.54
49.44
5.9
GDWS 0.1)
80.47
49.52
31.5
VGG-16 (p 95%)
78.91
48.74 74
3.0
+ GDWS 0.1)
78.71
48.53
18.3
VGG-16 (p = 99%)
73.16
41.74
0.6
GDWS 0.02)
72.75
41.56
136
WRN-28-4 from [34]
85.35
$7.23
22.3
GDWS (3
84.17
55.87
20.5
WRN-28-4 (p 90%)
83.69
55.20
2.3
GDWS 0.125)
83.38
54.79
11.9
WRN-28-4 (p 95%)
82.68
54.18
1.1
GDWS 0.005)
82.59
54.22
7.2
WRN-28-4 (p 99%)
47.21
0.2
GDWS 0.0025)
75.36
47.04
1.2
Table 5: Comparison between HYDRA [34] and GDWS using ResNet-50 on ImageNet.
Models
top-1 5 Ami [%]
top-1 5 Anb [%]
Size [MB]
FPS
ResNet-50 (AT from (34))
60.25 / 82.39
/1.11/6..1
97.5
15
GDWS (B 50)
58.04 / 80,56
30.22 / 58.48
86.2
19
ResNett-0 (p 95%)
44,60 / 70.12
19.53 / 44.28
5.1
15
+ GDWS (/ = 0.5)
43.91 / 69.46
19.27 / /43.58
12.6
19
ResNect50 (p 99%)
27.68 / 52.55
11.32 / 28.83
1.2
17
+ GDWS (Ø 0.5)
26.27 / 50.90
10.92 27.55
2.9
25
4.3 Defending against Union of Perturbation Models
Recent work has shown that adversarial training with a single perturbation model leads to classifiers vulnerable to the union of (Loo, l2, bbuuudde perturbations [33, 39, 22]. The method of multi steepest descent (MSD) [22] achieves state-of-the-e union robust accuracy (A44, against the union of (Exc, &,, &1 bounded perturbations. We demonstrate the versatility of GDWS by applying it to a publicly available [22] robust premMeeD- ResNet-18 model on CIFAR- 10. Following the setup in [22], all attacks were run on a subset of the first 1000 test images with 10 random restarts with the following attack configurations: 6n 0.03 with PGD-100. é2 = 0.5 with PGD-500. and é = 12 with PGD-100. Table 6 shows that applying GDWS with 3 0.01 to the pre-trained ResNet-18 incurs a negligible (<~1%) drop Anat and AU while improving the throughput from 28 FPS to 101 FPS (> 250% improvement).

---

Table 6: Benefits of GDWS when evaluated against union of perturbation models on CIFAR-10. A is the fraction of test images that are simultancously resistant to all perturbation models.
Models
Ams [%]]
Anih [%6
A/bb
[%6]
A [%]]
A [%]]
FPS
ResNet- 18 (AT from [22])
81.74
47.50
53.60
66.10
46.10
28
GDWS (3 = 0.0025)
81.67
47.60
53.60
66.00
46,30
87
GDWS (3 0.005)
81.43
47.30
52.60
65.60
45.70
GDWS (3- 0.01)
81.10
47.20
52.20
65.00
45.20
101
5 Discussion
We have established that the proposed GDWS convolutions are universal and efficient approximations of standard 2D convolutions that are able to accelerate any pre-trained CNN utilizing standard 2D convolution while preserving its accuracy and robustness. This facilitates the deployment of CNNs in safety critical edge applications where real-time decision making is crucial and robustness cannot be compromised. One limitation of this work is that GDWS alone does not achieve high compression ratios compared to pruning. Combining unstructured pruning with GDWS alleviates this problem to some extent. Furthermore, GDWS cannot be applied to CNNs utilizing DWS convolutions, such as RobNet for instance. An interesting question is explore the possibility of training GDWS-structune strucured networks from scratch. Another possible direction is fine-tuning post GDWS approximation to recover robustness, which we explore in the Appendix.
In summary, a GDWS approximated network inherits all the properties, e.g., accuracy, robustness, compression and others, of the baseline CNN while significantly enhancing its throughput (FPS) on real hardware. Therefore, the socictal impact of GDWS approximated networks are also inherited from those of the baseline CNNs.
Acknowledgments
This work was supported by the Center for Brain-Inspired Computing (C-BRIC) and the Artificial Intelligence Hardware (AIHW) program funded by the Semiconductor Research Corporation (SRC) and the Defense Advanced Research Projects Agency (DARPA).
References
[1] Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Under- standing and simplifying one-shot architecture search. In International Conference on Machine Learning, pages 550-559. PMLR, 2018.
[2] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: Parameterized clipping activation for quantized neural networks. arXiv preprint /00.::8805.00000 2018.
[3] Hassan Dbouk, Hetul Sanghvi, Mahesh Mehendale, and Naresh Shanbhag. DBQ: A differ- entiable branch quantizer for lightweight deep neural networks. In European Conference on Computer Vision, pages 90-106. Springer, 2020.
[4] Ross Girshick. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
[5] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint 1xXiv:1412.6572 2014.
[6] Shupeng Gui, Haotao Wang, Chen Yu, Haichuan Yang, Zhangyang Wang, and Ji Liu. Model compression with adversarial robustness: A unified optimization framework. Advances in Neural Information Processing Systems, 2019.
[7] Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu, and Dahua Lin. When nas meets robustness: In search of robust architectures against adversarial attacks. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 6-40,, 2020.
10

---

[8] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs, In Advances Neural Information Processing Systems (NIPS), 2016.
[9] Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark Horowitz, and William Dally. EIE: Efficient inference engine compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):243-254, 2016.
[10] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.0014: 2015.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[12] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprini 1704 2017.
[13] Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Q Weinberger. Condensenet: An efficient densenet using learned group convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2752-2761, 2018.
[14] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. In Advances in neural information processing systems, pages 4107-4115, 2016.
[15] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 0.5 MB model size. arXiv preprint arXiv: 1602 07360, 2016.
[16] Andrew Ilyas, Shibani Santurkar, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
[17] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[18] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[19] Lin, Chuang Gan, and Song Han. Defensive quantization: When efficiency meets robustness. In International Conference on Learning Representations, 2019.
[20] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2018
[21] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
[22] Pratyush Maini, Eric Wong, and Zico Kolter. Adversarial robustness against the union of multiple perturbation models. In International Conference on Machine Learning, pages 6640- 6650. PMLR, 2020.
[23] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NeurIPS Workshop Deep Learning and Unsupervised Feature Learning, 2011.
[24] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.

---

[25] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's hidden in a randomly weighted neural network? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 11893-11902, 2020.
[26] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-net: Ima- genet classification using binary convolutional neural networks. In European Conference on Computer Vision, pages 525- 542.. Springer, 2016.
[27] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4780-4789, 2019.
[28] Leslic Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pages 8093-8104 8004. PMLR, 2020.
[29] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International jouraiaaa of computer vision, 1 115(3):211-252 2015.
[30] Charbel Sakr, Yongjune Kim, and Naresh Shanbhag. Analytical guarantees numerical precision of deep neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 3007-3016. JMLR. org, 2017.
[31] Charbel Sakr and Naresh Shanbhag. An analytical method to determine minimum per-layer precision of deep neural networks. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1090-1090 IEEE, 2018
[32] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018
[33] Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adver- sarially robust neural network model on MNIST. In International Conference on Learning Representations, 2019.
[34] Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana. HYDRA: Pruning adversarially robust neural networks. Advances in Neural Information Processing Systems (NeurIPS), 7, 2020.
[35] Sanchari Sen, Balaraman Ravindran, and Anand Raghunathan. EMPIR: Ensembles of mixed precision deep networks for increased robustness against adversarial attacks. In International Conference on Learning Representations, 2019.
[36] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1559.155,, 2014.
[37] Christian Szegedy, Wojeiech Zaremba. Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.619 2013.
[38] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning, pages 6105-6114 14. PMIR, 2019.
[39] Florian Tramèr and Dan Boneh. Adversarial training and robustness for multiple perturbations. In Conference on Neural Information Processing Systems (NeurIPS), 2019.
[40] Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances Neural Information Processing Systems, 33, 2020.
[41] Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. Designing energy-efficient convolutional neural networks using energy- -waar pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5687-5695, 2017.
12

---

[42] Shaokai Ye, Kaidi Xu, Sijia Liu, Hao Cheng, Jan-Henrik Lambrechts, Huan Zbang, Aojun Zhou, Kaisheng Ma, Yanzhi Wang, and Xue Lin. Adversarial robustness vs. model compression, or both? In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 111-120, 2019.
[43] Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146. 2016.
Wide residual networks. arXiv preprint
[44] Dongging Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. LQ-nets: Learned quantiza- tion for highly accurate and compact deep neural networks. In Proceedings of the European Conference Computer Vision (ECCV), pages 365-382, 2018.
[45] Hongyang Zhang, Yaodong Yu, Jiantao Jíao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pages 7472-7482 PMLR, 2019.
[46] Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic DNN weight pruning framework using alternating direction method of multipliers. In Proceedings of the European Conference on Computer Vision (ECCV), pages 184-199, 2018.
[47] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6848 6856, 2018.
[48] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018.
13


Reviews:
[{"review_id": "qJ-Do4hSkLs", "reviewer": "Reviewer_n5ng", "summary": "This work proposes a scalable and universal post-training approximation of standard 2D convolutions, namely Generalized Depthwise-Separable (GDWS) convolutions, to improve the throughput (FPS) of adversarial robust neural networks (NNs) on real-life hardware. Experiments are performed on different datasets, including CIFAR-10, SVHN, and ImageNet, with different network architectures, including VGG-16 and several ResNet variants.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The GDWS convolution improves the throughout of adversarial robust NNs, which is always ignored by existing pruning-based compression methods. The method directly operates on pre-trained models without additional training, which makes it more efficient and easier to scale up than pruning methods. Experiments are performed on four different network architectures and is also compared to a NAS method and several structured and unstructured pruning methods.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are some limitations:\n\n1. In Table 3, GDWS uses a fixed setting for each architecture. With VGG-16, the FPS of GDWS is outperformed by ADMM with $p=75\\%$. Although GDWS has better accuracy than ADMM ($p=75\\%$), I still wondering if the FPS is further improved, whether GDWS can still outperform ADMM.\n\n2. In Table 4, when GDWS is applied on a highly compressed network, the parameter size is increased, which is an undesired behavior.\n\nA minor issue: Table 5 should be with section 4.2 but is misplaced in section 4.3.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "YG26-iYATw", "reviewer": "Reviewer_c4Yo", "summary": "The paper proposes a generalization of depthwise-separable convolutions (called GDWS) where each channel can have multiple depthwise filters (instead of just one) that are subsequently combined using a pointwise convolution. Two algorithms are proposed to minimize the error/complexity of mapping general convolutions to GDWS as a post-processing step after training. Results show small degradation on natural and robust accuracy while increasing the FPS substantially.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written (easy to follow) and considers an important topic. The idea of generalizing depthwise-separable convolutions as proposed in the paper seems novel to me and seems to substantially increase FPS while only having a small negative impact on accuracy.\n\nThe paper provides an analysis showing that a general convolution can be written as a GDWS convolution where each channel has K^2 depthwise filters. This is a good justification for approximating general convolutions using fewer depthwise filters, but using more than a single filter as in conventional depthwise-separable convolutions.\n\nI wonder why the proposed method is suitable for adversarial robustness in particular. Is it because the method maintains the adversarial robustness of a pre-trained neural network due to the good approximation quality? Or is there something special to GDWS that is particularly well suited to achieve adversarial robustness? At least I have not figured this out after reading the paper and some insights are required. It does not appear that the proposed method is specifically tailored to improving adversarial robustness, but that this is rather a byproduct that is empirically evaluated.\n\nAs mentioned in Section 5, I would have liked to see some experiments where GDWS is used to train from scratch or, at least, to apply some fine-tuning after applying Algorithm 3 to regain some of the lost accuracy. This is a standard practice of methods that approximate building blocks of pre-trained methods using cheaper building blocks (e.g., pruning). Did the authors try this? If yes: Does it help? Does it harm the adversarial robustness?\n\nHow much do the results regarding throughput depend on the hardware and the implementation? Since the increased throughput seems to be a major selling point of the proposed method, this also needs to be evaluated on some other non-GPU/TPU hardware without massive parallelism. Furthermore, I have the feeling that the comparison with unstructured pruning is unfair. Unstructured pruning can benefit a lot if suitable sparse tensor structures are used. However, in the proposed setting it seems that only those filters are removed that only contain zero entries, and the rest is treated as a dense tensor. I believe that a network with 95% sparsity can have way more throughput than one without any sparsity, but the reported results have the same FPS.\n\nI would have liked to see some experiments that show how the individual layers are affected by the compression, e.g., how does the average g_c look like for individual layers? For instance, is there some difference between input/output layers and intermediate layers? From the reported FPS it is difficult to tell where the increased throughput stems from. Maybe you can also provide some intuition why the FPS is affected more severely than the size. It would also be interesting to see how the accuracy behaves for different g_c / different approximation errors.\n\nJust to be sure: Is the \"natural accuracy\" the accuracy on the standard test dataset (without adversarial perturbations)? In this case, can you comment on the rather weak natural accuracies on the given datasets (e.g., approx 83% is rather weak on Cifar-10).\n\nminor:\n- Figure 1a: What are the numbers in brackets?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Sufficiently addressed (limitations: see above)", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "MGEgrbPfN5s", "reviewer": "Reviewer_PpQM", "summary": "The authors propose Generalized Depthwise-Separable (GDWS) convolutions for processing images for the purpose of getting high throughput in efficiency and robustness. This is carried out by introducing channel distribution vectors in the standard 2D CNNs. The efficiency is managed by means of the singular value decomposition of the matrix in the convolutional operations after vectorizations. Experiments show that the method provides similar results for some benchmark data. ", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem of getting efficiency and robustness is important for applying 2-D CNNs to dealing with images. The idea of channel distribution vectors to get efficiency is interesting. However, this idea has not been fully explored in the paper. The reviewer has the following concerns:\n\n1. Activation. In the proposed method, only the output layer involves an activation function. The GDWS does not involve any nonlinear activation functions in the hidden layers. So the role of depth and the effect of deep learning cannot be seen. \n\n2. Vectorization. The technique of vectorization is often used for dealing with image data. But it breaks the structures of images and hidden layers, and the role of convolutions is not reflected in GDWS. \n\n3. The experiment results do not demonstrate significant improvements of GDWS in improving efficiency and robustness of the standard CNNs. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors admit the limitations of their work in not achieving high compression ratios and being unable to be applied to CNNs utilizing depthwise-separable convolutions. This is described in the session of discussion. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "4UXhG-znI5Y", "reviewer": "Reviewer_Sfha", "summary": "This paper proposes the method of Generalized Depthwise-Separable (GDWS) convolution for robust model compression. It tries to improve frames-per-second on real-life hardware while simultaneously preserving robustness to adversarial perturbations. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ The paper is clearly written with sound theoretical proof. \n+ Good design of experiments with different models for comparison. \n\nWeaknesses:\n- The results look good in general, but for Table 2, RobNet results look weird. As in the original RobNet paper, it is much stronger than Resnet in terms of perturbations. But in Table 2, it seems that RobNet underperforms ResNet baseline. I am wondering if the same backbone are used for RobNet? Is it a fair comparison?\n- Only PGD are present in the experiments. I am wondering how the proposed method performs under different types of attacks. Then the robustness of the proposed method would be better evaluated.  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Taken at face value, this paper seems to have a lot of variance in the scores, but this is driven mostly by the low score given by reviewer PpQM. The authors have noted that PpQM's review does not make sense, I have read it and confirmed that it does not make sense, and in fact another reviewer has confirmed to me that it does not make sense. That makes it especially strange that they have chosen to respond to the rebuttal that their score is unchanged, but at any rate I think it's safe to ignore their review. 

With that out of the way, the scores are 6, 6, and 7.
I found the review of c4Yo (a 6) most informative, and appreciated the discussion that followed. 
I will recommend acceptance in this instance
Decision: Accept (Spotlight)