Generalized Depthwise-Separable Convolutions for Adversarially Robust and Efficient Neural Networks

1 Introduction . Nearly a decade of research after the release of AlexNet [ 18 ] in 2012 , convolutional neural networks ( CNNs ) have unequivocally established themselves as the de facto classification algorithm for various machine learning tasks [ 11 , 38 , 4 ] . The tremendous success of CNNs is often attributed to their unrivaled ability to extract correlations from large volumes of data , allowing them to surpass human level accuracy on some tasks such as image classification [ 11 ] . Today , the deployment of CNNs in safety-critical Edge applications is hindered due to their high computational costs [ 11 , 30 , 31 ] and their vulnerability to adversarial samples [ 37 , 5 , 16 ] . Traditionally , those two problems have been addressed in isolation . Recently , very few bodies of works [ 19 , 35 , 42 , 6 , 34 , 7 ] have addressed the daunting task of designing both efficient and robust CNNs . A majority of these methods focus on model compression , i.e . reducing the storage requirements of CNNs . None have demonstrated their real-time benefits in hardware . For instance , Fig . 1a shows recent robust pruning works HYDRA [ 34 ] and ADMM [ 42 ] achieve high compression ratios ( up to 97× ) but either fail to achieve high throughput measured in frames-per-second ( FPS ) or compromise significantly on robustness . Furthermore , the overreliance of current robust complexity reduction techniques on adversarial training ( AT ) [ 45 , 21 ] increases their training time significantly ( Fig . 1b ) . This prohibits their application to complex ImageNet scale problems with stronger attack models , such as union of norm-bounded perturbations [ 22 ] . Thus , there is critical need for methods to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . design deep nets that are both adversarially robust and achieve high throughput when mapped to real hardware . To address this need , we propose Generalized Depthwise-Separable ( GDWS ) convolutions , a universal post-training approximation of a standard 2D convolution that dramatically improves the real hardware FPS of pre-trained networks ( Fig . 1a ) while preserving their robust accuracy . Interestingly , we find GDWS applied to un-pruned robust networks simultaneously achieves higher FPS and higher robustness than robust pruned models obtained from current methods . This in spite of GDWS ’ s compression ratio being smaller than those obtained from robust pruning methods . Furthermore , GDWS easily scales to large problem sizes since it operates on pre-trained models and doesn ’ t require any additional training . Our contributions : 1 . We propose GDWS , a novel convolutional structure that can be seamlessly mapped onto off-the-shelf hardware and accelerate pre-trained CNNs significantly while maintaining robust accuracy . 2 . We show that the error-optimal and complexity-optimal GDWS approximations of any pre-trained standard 2D convolution can be obtained via greedy polynomial time algorithms , thus eliminating the need for any expensive training . 3 . We apply GDWS to a variety of networks on CIFAR-10 , SVHN , and ImageNet to simultaneously achieve higher robustness and higher FPS than existing robust complexity reduction techniques , while incurring no extra training cost . 4 . We demonstrate the versatility of GDWS by using it to design efficient CNNs that are robust to union of ( ` ∞ , ` 2 , ` 1 ) perturbation models . To the best of our knowledge , this is the first work that proposes efficient and robust networks to the union of norm-bounded perturbation models . 2 Background and Related Work . The problem of designing efficient and robust CNNs , though crucial for safety-critical Edge applications , is not yet well understood . Very few recent works have addressed this problem [ 19 , 35 , 42 , 6 , 34 , 7 ] . We cluster prior works into the following categories : Quantization Reducing the complexity of CNNs via model quantization in the absence of any adversary is a well studied problem in the deep learning literature [ 30 , 31 , 2 , 44 , 14 , 26 , 3 ] . The role of quantization on adversarial robustness was studied in Defensive Quantization ( DQ ) [ 19 ] where it was observed that conventional post-training fixed-point quantization makes networks more vulnerable to adversarial perturbations than their full-precision counterparts . EMPIR [ 35 ] also leverages extreme model quantization ( up to 2-bits ) to build an ensemble of efficient and robust networks . However , [ 40 ] broke EMPIR by constructing attacks that fully leverage the model structure , i.e. , adaptive attacks . In contrast , GDWS is an orthogonal complexity reduction technique that preserves the base model ’ s adversarial robustness and can be applied in conjunction with model quantization . Pruning The goal of pruning is to compress neural networks by zeroing out unimportant weights [ 10 , 8 , 46 , 41 ] . The structured pruning method in [ 42 ] combines the alternating direction method of multipliers ( ADMM ) [ 46 ] for parameter pruning within the AT framework [ 21 ] to design pruned and robust networks . The flexibility of ADMM enables it to achieve a high FPS on Jetson ( as seen in Fig . 1a ) but suffers from a significant drop in robustness . ATMC [ 6 ] augments the ADMM framework [ 42 ] with model quantization and matrix factorization to further boost the compression ratio . On the other hand , unstructured pruning methods such as HYDRA [ 34 ] prunes models via important score optimization [ 25 ] . However , HYDRA ’ s high pruning ratios ( > 90 % ) doesn ’ t translate into real-time FPS improvements on off-the-shelf hardware and often requires custom hardware design to fully leverage their capabilities [ 9 ] . GDWS is complementary to unstructured pruning methods , e.g. , when applied to HYDRA , GDWS boosts the achievable FPS and achieves much higher robustness at iso-FPS when compared to structured ( filter ) pruning ADMM . Neural Architecture Search Resource-efficient CNNs can be designed by exploiting design intuitions such as depthwise separable ( DWS ) convolutions [ 12 , 32 , 13 , 47 , 15 , 38 ] . While neural architecture search ( NAS ) [ 48 , 27 ] automates the process , it requires massive compute resources , e.g. , thousands of GPU hours for a single network . Differentiable NAS [ 20 ] and one-shot NAS [ 1 ] drastically reduce the cost of this search . In [ 7 ] , a one-shot NAS framework [ 1 ] is combined with the AT framework [ 21 ] to search for robust network architectures , called RobNets . RobNets achieve slightly higher robustness than existing networks with less storage requirements . In this work , we show that applying GDWS to existing architectures , e.g. , WideResNet-28-4 , achieves significantly higher FPS than RobNet , at iso-robustness and model size . 3 Generalized Depthwise-Separable Convolutions . In this section , we introduce GDWS convolutions and develop error-optimal and complexity-optimal GDWS approximations of standard 2D convolution . These optimal approximations are then employed to construct GDWS networks from any pre-trained robust CNN built from standard 2D convolutions . Notation : A ( C , K , M ) standard 2D convolution operates on an input feature map X ∈ RC×H×W via M filters ( also referred to as kernels or output channels ) each consisting of C channels each of dimension K ×K to generate an output feature map Y ∈ RM×H′×W ′ . 2D Convolution as Matrix Multiplication : The M filters can be viewed as vectors { wi } Mi=1 ∈ RCK2 obtained by vectorizing the K2 elements within a channel and then across the C channels . The resulting weight matrix W ∈ RM×CK2 is constructed by stacking these filter vectors , i.e. , W = [ w1|w2| ... |wM ] T. From an operational viewpoint , the matrix W can be used to compute the 2D convolution via Matrix Multiplication ( MM ) with the input matrix X = Ψ ( X ) ∈ RCK2×H′W ′ : Y = WX = WΨ ( X ) ( 1 ) where Ψ is an unrolling operator that generates all H ′W ′ input feature map slices and stacks them in matrix format . The resultant output matrix Y ∈ RM×H′W ′ can be reshaped via the operator Φ to retrieve Y = Φ ( Y ) . The computational complexity of ( 1 ) in terms of multiply-accumulate ( MAC ) operations is given by : H ′W ′MCK2 ( 2 ) The reshaping operators Φ and Ψ are only for notational convenience and are computation-free . 3.1 GDWS Formulation . Definition : A ( C , K , g , M ) GDWS convolution is parameterized by the channel distribution vector g ∈ ZC+ in addition to the parameters ( C , K , M ) of a standard 2D convolution . A GDWS convolution is composed of a ( C , K , g ) Generalized Depthwise ( GDW ) convolution and a ( G , 1 , M ) standard pointwise ( PW ) convolution where G = ∑ gc with c ∈ [ C ] 1 . A ( C , K , g ) GDW convolutional layer ( Fig . 2 ) operates on an input feature map X ∈ RC×H×W by convolving the cth channel with gc ∈ Z+ depthwiseK×K filters to produce a total ofG intermediate output channels each of size H ′ ×W ′ . The ( G , 1 , M ) PW layer operates on the intermediate output feature map by convolving it with M filters of size 1 × 1 , thus producing the output feature map Y ∈ RM×H′×W ′ . Relation to DWS : Setting gc = 1 ∀c ∈ [ C ] reduces the GDWS convolution to the standard DWS convolution popularized by [ 12 ] . Thus , GDWS generalizes DWS by allowing for more than one ( gc ≥ 1 ) depthwise filters per channel . This simple generalization relaxes DWS ’ s highly constrained structure enabling accurate approximations of the 2D convolution . Thus , GDWS when applied to pre-trained models preserves its original behavior and therefore its natural and robust accuracy . Furthermore , GDWS achieves high throughput since it exploits the same hardware features that enable networks with DWS to be implemented efficiently . One might ask : why not use DWS on pre-trained models ? Doing so will result in very high approximation errors . In fact , in Section 4.2 , we show that applying GDWS to a pre-trained complex network such as ResNet-18 achieves better robust accuracy than MobileNet trained from scratch , while achieving similar FPS . GDWS Complexity : The total number of MAC operations required by GDWS convolutions is : H ′W ′ ( C∑ c=1 gc ( K 2 +M ) ) = H ′W ′G ( K2 +M ) ( 3 ) Thus , replacing standard 2D convolutions with GDWS convolutions results in a complexity reduction by a factor of G ( K 2+M ) CK2M . 3.2 Properties of GDWS Convolutions . We present properties of the GDWS weight matrix W that will be vital for developing the optimal approximation procedures . Property 1 . The weight matrix W ∈ RM×CK2 of a ( C , K , g , M ) GDWS convolution can be expressed as : W = WPW D ( 4 ) where WP ∈ RM×G and WD ∈ RG×CK 2 are the weight matrices of the PW and GDW convolutions , respectively . Property 1 implies that any GDWS convolution has an equivalent 2D convolution whose weight matrix is the product of WP and WD , where WP is a regular convolution weight matrix with K = 1 and WD has the following property : 1we use the notation [ C ] = { 1 , 2 , ... , C } for brevity . Property 2 . The weight matrix WD ∈ RG×CK 2 of a ( C , K , g ) GDW convolution has a blockdiagonal structure . Specifically , WD is a concatenation of C sub-matrices where each sub-matrix WD , c ∈ RG×K 2 has at most gc non-zero consecutive rows , starting at row index 1 + ∑c−1 k=1 gk as shown in Fig . 3 . This structure is due to the fact that input channels are convolved independently with at most gc depthwise filters per channel . Finally , let W = [ W1|W2| ... |WC ] be represented as the concatenation of C sub-matrices , then combining Properties 1 & 2 establishes the following lemma : Lemma 1 . The weight matrix of a ( C , K , g , M ) GDWS convolution can be expressed as the concatenation of C sub-matrices Wc where rank ( Wc ) ≤ min ( gc , K2 ) ∀c ∈ [ C ] . The reason for this is that each sub-matrix Wc can be expressed as the sum of gc rank 1 matrices of size M ×K2 . A detailed proof of Lemma 1 can be found in the Appendix . A major implication of Lemma 1 is that any 2D standard convolution is equivalent to a GDWS convolution with gc = K2 ∀c2 . Hence , in the rest of this paper we will assume gc ≤ K2 when we approximate 2D convolutions with GDWS .

Summary: The paper proposes a generalization of depthwise-separable convolutions (called GDWS) where each channel can have multiple depthwise filters (instead of just one) that are subsequently combined using a pointwise convolution. Two algorithms are proposed to minimize the error/complexity of mapping general convolutions to GDWS as a post-processing step after training. Results show small degradation on natural and robust accuracy while increasing the FPS substantially.