Generalized Depthwise-Separable Convolutions for Adversarially Robust and Efficient Neural Networks

1 Introduction . Nearly a decade of research after the release of AlexNet [ 18 ] in 2012 , convolutional neural networks ( CNNs ) have unequivocally established themselves as the de facto classification algorithm for various machine learning tasks [ 11 , 38 , 4 ] . The tremendous success of CNNs is often attributed to their unrivaled ability to extract correlations from large volumes of data , allowing them to surpass human level accuracy on some tasks such as image classification [ 11 ] . Today , the deployment of CNNs in safety-critical Edge applications is hindered due to their high computational costs [ 11 , 30 , 31 ] and their vulnerability to adversarial samples [ 37 , 5 , 16 ] . Traditionally , those two problems have been addressed in isolation . Recently , very few bodies of works [ 19 , 35 , 42 , 6 , 34 , 7 ] have addressed the daunting task of designing both efficient and robust CNNs . A majority of these methods focus on model compression , i.e . reducing the storage requirements of CNNs . None have demonstrated their real-time benefits in hardware . For instance , Fig . 1a shows recent robust pruning works HYDRA [ 34 ] and ADMM [ 42 ] achieve high compression ratios ( up to 97× ) but either fail to achieve high throughput measured in frames-per-second ( FPS ) or compromise significantly on robustness . Furthermore , the overreliance of current robust complexity reduction techniques on adversarial training ( AT ) [ 45 , 21 ] increases their training time significantly ( Fig . 1b ) . This prohibits their application to complex ImageNet scale problems with stronger attack models , such as union of norm-bounded perturbations [ 22 ] . Thus , there is critical need for methods to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . design deep nets that are both adversarially robust and achieve high throughput when mapped to real hardware . To address this need , we propose Generalized Depthwise-Separable ( GDWS ) convolutions , a universal post-training approximation of a standard 2D convolution that dramatically improves the real hardware FPS of pre-trained networks ( Fig . 1a ) while preserving their robust accuracy . Interestingly , we find GDWS applied to un-pruned robust networks simultaneously achieves higher FPS and higher robustness than robust pruned models obtained from current methods . This in spite of GDWS ’ s compression ratio being smaller than those obtained from robust pruning methods . Furthermore , GDWS easily scales to large problem sizes since it operates on pre-trained models and doesn ’ t require any additional training . Our contributions : 1 . We propose GDWS , a novel convolutional structure that can be seamlessly mapped onto off-the-shelf hardware and accelerate pre-trained CNNs significantly while maintaining robust accuracy . 2 . We show that the error-optimal and complexity-optimal GDWS approximations of any pre-trained standard 2D convolution can be obtained via greedy polynomial time algorithms , thus eliminating the need for any expensive training . 3 . We apply GDWS to a variety of networks on CIFAR-10 , SVHN , and ImageNet to simultaneously achieve higher robustness and higher FPS than existing robust complexity reduction techniques , while incurring no extra training cost . 4 . We demonstrate the versatility of GDWS by using it to design efficient CNNs that are robust to union of ( ` ∞ , ` 2 , ` 1 ) perturbation models . To the best of our knowledge , this is the first work that proposes efficient and robust networks to the union of norm-bounded perturbation models . 2 Background and Related Work . The problem of designing efficient and robust CNNs , though crucial for safety-critical Edge applications , is not yet well understood . Very few recent works have addressed this problem [ 19 , 35 , 42 , 6 , 34 , 7 ] . We cluster prior works into the following categories : Quantization Reducing the complexity of CNNs via model quantization in the absence of any adversary is a well studied problem in the deep learning literature [ 30 , 31 , 2 , 44 , 14 , 26 , 3 ] . The role of quantization on adversarial robustness was studied in Defensive Quantization ( DQ ) [ 19 ] where it was observed that conventional post-training fixed-point quantization makes networks more vulnerable to adversarial perturbations than their full-precision counterparts . EMPIR [ 35 ] also leverages extreme model quantization ( up to 2-bits ) to build an ensemble of efficient and robust networks . However , [ 40 ] broke EMPIR by constructing attacks that fully leverage the model structure , i.e. , adaptive attacks . In contrast , GDWS is an orthogonal complexity reduction technique that preserves the base model ’ s adversarial robustness and can be applied in conjunction with model quantization . Pruning The goal of pruning is to compress neural networks by zeroing out unimportant weights [ 10 , 8 , 46 , 41 ] . The structured pruning method in [ 42 ] combines the alternating direction method of multipliers ( ADMM ) [ 46 ] for parameter pruning within the AT framework [ 21 ] to design pruned and robust networks . The flexibility of ADMM enables it to achieve a high FPS on Jetson ( as seen in Fig . 1a ) but suffers from a significant drop in robustness . ATMC [ 6 ] augments the ADMM framework [ 42 ] with model quantization and matrix factorization to further boost the compression ratio . On the other hand , unstructured pruning methods such as HYDRA [ 34 ] prunes models via important score optimization [ 25 ] . However , HYDRA ’ s high pruning ratios ( > 90 % ) doesn ’ t translate into real-time FPS improvements on off-the-shelf hardware and often requires custom hardware design to fully leverage their capabilities [ 9 ] . GDWS is complementary to unstructured pruning methods , e.g. , when applied to HYDRA , GDWS boosts the achievable FPS and achieves much higher robustness at iso-FPS when compared to structured ( filter ) pruning ADMM . Neural Architecture Search Resource-efficient CNNs can be designed by exploiting design intuitions such as depthwise separable ( DWS ) convolutions [ 12 , 32 , 13 , 47 , 15 , 38 ] . While neural architecture search ( NAS ) [ 48 , 27 ] automates the process , it requires massive compute resources , e.g. , thousands of GPU hours for a single network . Differentiable NAS [ 20 ] and one-shot NAS [ 1 ] drastically reduce the cost of this search . In [ 7 ] , a one-shot NAS framework [ 1 ] is combined with the AT framework [ 21 ] to search for robust network architectures , called RobNets . RobNets achieve slightly higher robustness than existing networks with less storage requirements . In this work , we show that applying GDWS to existing architectures , e.g. , WideResNet-28-4 , achieves significantly higher FPS than RobNet , at iso-robustness and model size . 3 Generalized Depthwise-Separable Convolutions . In this section , we introduce GDWS convolutions and develop error-optimal and complexity-optimal GDWS approximations of standard 2D convolution . These optimal approximations are then employed to construct GDWS networks from any pre-trained robust CNN built from standard 2D convolutions . Notation : A ( C , K , M ) standard 2D convolution operates on an input feature map X ∈ RC×H×W via M filters ( also referred to as kernels or output channels ) each consisting of C channels each of dimension K ×K to generate an output feature map Y ∈ RM×H′×W ′ . 2D Convolution as Matrix Multiplication : The M filters can be viewed as vectors { wi } Mi=1 ∈ RCK2 obtained by vectorizing the K2 elements within a channel and then across the C channels . The resulting weight matrix W ∈ RM×CK2 is constructed by stacking these filter vectors , i.e. , W = [ w1|w2| ... |wM ] T. From an operational viewpoint , the matrix W can be used to compute the 2D convolution via Matrix Multiplication ( MM ) with the input matrix X = Ψ ( X ) ∈ RCK2×H′W ′ : Y = WX = WΨ ( X ) ( 1 ) where Ψ is an unrolling operator that generates all H ′W ′ input feature map slices and stacks them in matrix format . The resultant output matrix Y ∈ RM×H′W ′ can be reshaped via the operator Φ to retrieve Y = Φ ( Y ) . The computational complexity of ( 1 ) in terms of multiply-accumulate ( MAC ) operations is given by : H ′W ′MCK2 ( 2 ) The reshaping operators Φ and Ψ are only for notational convenience and are computation-free . 3.1 GDWS Formulation . Definition : A ( C , K , g , M ) GDWS convolution is parameterized by the channel distribution vector g ∈ ZC+ in addition to the parameters ( C , K , M ) of a standard 2D convolution . A GDWS convolution is composed of a ( C , K , g ) Generalized Depthwise ( GDW ) convolution and a ( G , 1 , M ) standard pointwise ( PW ) convolution where G = ∑ gc with c ∈ [ C ] 1 . A ( C , K , g ) GDW convolutional layer ( Fig . 2 ) operates on an input feature map X ∈ RC×H×W by convolving the cth channel with gc ∈ Z+ depthwiseK×K filters to produce a total ofG intermediate output channels each of size H ′ ×W ′ . The ( G , 1 , M ) PW layer operates on the intermediate output feature map by convolving it with M filters of size 1 × 1 , thus producing the output feature map Y ∈ RM×H′×W ′ . Relation to DWS : Setting gc = 1 ∀c ∈ [ C ] reduces the GDWS convolution to the standard DWS convolution popularized by [ 12 ] . Thus , GDWS generalizes DWS by allowing for more than one ( gc ≥ 1 ) depthwise filters per channel . This simple generalization relaxes DWS ’ s highly constrained structure enabling accurate approximations of the 2D convolution . Thus , GDWS when applied to pre-trained models preserves its original behavior and therefore its natural and robust accuracy . Furthermore , GDWS achieves high throughput since it exploits the same hardware features that enable networks with DWS to be implemented efficiently . One might ask : why not use DWS on pre-trained models ? Doing so will result in very high approximation errors . In fact , in Section 4.2 , we show that applying GDWS to a pre-trained complex network such as ResNet-18 achieves better robust accuracy than MobileNet trained from scratch , while achieving similar FPS . GDWS Complexity : The total number of MAC operations required by GDWS convolutions is : H ′W ′ ( C∑ c=1 gc ( K 2 +M ) ) = H ′W ′G ( K2 +M ) ( 3 ) Thus , replacing standard 2D convolutions with GDWS convolutions results in a complexity reduction by a factor of G ( K 2+M ) CK2M . 3.2 Properties of GDWS Convolutions . We present properties of the GDWS weight matrix W that will be vital for developing the optimal approximation procedures . Property 1 . The weight matrix W ∈ RM×CK2 of a ( C , K , g , M ) GDWS convolution can be expressed as : W = WPW D ( 4 ) where WP ∈ RM×G and WD ∈ RG×CK 2 are the weight matrices of the PW and GDW convolutions , respectively . Property 1 implies that any GDWS convolution has an equivalent 2D convolution whose weight matrix is the product of WP and WD , where WP is a regular convolution weight matrix with K = 1 and WD has the following property : 1we use the notation [ C ] = { 1 , 2 , ... , C } for brevity . Property 2 . The weight matrix WD ∈ RG×CK 2 of a ( C , K , g ) GDW convolution has a blockdiagonal structure . Specifically , WD is a concatenation of C sub-matrices where each sub-matrix WD , c ∈ RG×K 2 has at most gc non-zero consecutive rows , starting at row index 1 + ∑c−1 k=1 gk as shown in Fig . 3 . This structure is due to the fact that input channels are convolved independently with at most gc depthwise filters per channel . Finally , let W = [ W1|W2| ... |WC ] be represented as the concatenation of C sub-matrices , then combining Properties 1 & 2 establishes the following lemma : Lemma 1 . The weight matrix of a ( C , K , g , M ) GDWS convolution can be expressed as the concatenation of C sub-matrices Wc where rank ( Wc ) ≤ min ( gc , K2 ) ∀c ∈ [ C ] . The reason for this is that each sub-matrix Wc can be expressed as the sum of gc rank 1 matrices of size M ×K2 . A detailed proof of Lemma 1 can be found in the Appendix . A major implication of Lemma 1 is that any 2D standard convolution is equivalent to a GDWS convolution with gc = K2 ∀c2 . Hence , in the rest of this paper we will assume gc ≤ K2 when we approximate 2D convolutions with GDWS .

Summary: The authors propose Generalized Depthwise-Separable (GDWS) convolutions for processing images for the purpose of getting high throughput in efficiency and robustness. This is carried out by introducing channel distribution vectors in the standard 2D CNNs. The efficiency is managed by means of the singular value decomposition of the matrix in the convolutional operations after vectorizations. Experiments show that the method provides similar results for some benchmark data.