Spatial-Winograd Pruning Enabling Sparse Winograd Convolution

1 INTRODUCTION . Deep convolutional neural networks ( CNNs ) have been ubiquitously utilized in various application domains . However , their performance comes at the cost of a significant amount of computation which keeps growing over time . As an example , for the ImageNet challenge ( Russakovsky et al. , 2015 ) , Krizhevsky et al . ( 2012 ) proposed AlexNet which requires more than 1.1 × 109 multiplications . Later , in 2016 , the ResNet-152 model ( He et al. , 2016 ) increased the computation cost to 11.3 × 109 multiplications . This high computation cost limits the deployment of larger and deeper CNN models . There are two primary methods to reduce the required computation of CNN models : pruning techniques and Winograd/FFT convolution . Pruning removes redundant weight parameters , inducing sparsity into the network . On the other hand , Winograd convolution ( Lavin & Gray , 2016 ) and FFT convolution ( Mathieu et al. , 2013 ) transform the computation into different domains . The convolution operations can then be replaced by element-wise multiplications . For the typical convolution kernel size of 3 × 3 , Winograd convolution can achieve more than twofold speedup over highly optimized spatial convolution algorithms , and typically requires fewer flops than FFT-based approaches ( Li et al. , 2017 ) . Therefore , in this paper , we focus on the Winograd convolution . The pruning techniques and Winograd convolution are not directly compatible with each other . Sparse weight matrices , which are generated by pruning , lose most of the sparsity after the Winograd transformation from the spatial ( original ) domain to the Winograd domain . The remaining sparsity is much lower than what we need for improving computation performance . To increase the Winograd-domain sparsity , Li et al . ( 2017 ) propose to perform pruning and retraining directly on Winograd-domain weights . However , it requires using an extremely small learning rate , e.g. , 200x smaller for AlexNet , in retraining and is difficult to be applied to deep networks . Besides , Winograd-ReLU pruning ( Liu et al. , 2018 ) moves ReLU function into the Winograd domain , which helps increase Winograd-domain sparsity but requires changes in the network structure . In this paper , to further improve the sparsity of Winograd-domain weights without changing the network structure , we propose a new pruning method , spatial-Winograd pruning . It includes two parts : spatial structured pruning and Winograd direct pruning . In spatial structured pruning , we prune the spatial-domain weights in a structured way , in which the structures are designed to transfer the spatial-domain sparsity into the Winograd domain efficiently . After spatial structured pruning , weights of the pruned layers will be converted to and kept in the Winograd domain . Then , for Winograd direct pruning , we perform pruning and retraining entirely in the Winograd domain to improve the sparsity further . This paper makes the following contributions : • We propose a new pruning method , spatial-Winograd pruning . Without changing the network structure , it can achieve higher sparsity in Winograd-domain weights compared with previous methods . • As the first part of spatial-Winograd pruning , we provide a structured pruning method to transfer the spatial-domain sparsity into the Winograd domain efficiently . It can help avoid Winograd-domain retraining in this part and accelerate the pruning process . • In the second part , to perform pruning directly in the Winograd domain , we present a new approach to measuring the importance of each Winograd-domain weight based on its impact on output activations . Also , we propose to use an importance factor matrix to adjust the gradients of Winograd-domain weights , which makes it much faster to retrain deep networks directly in the Winograd domain without changing the network structure . 2 PRELIMINARY AND RELATED WORK . Winograd convolution ( Lavin & Gray , 2016 ) is a typical algorithm to reduce the arithmetic complexity of CNNs . It transforms the computation into the Winograd domain , and the convolution operations can then be replaced by element-wise multiplications . We call the domain , in which the conventional convolution operation is executed , to be the spatial domain . The basic block of Winograd convolution works on a 2D input tile , I , with a size of m ×m and a 2D weight filter , W , with a size of n × n. In this case , the 2D output tile generated , O , will have a size of ( m − n + 1 ) × ( m − n + 1 ) . For a typical convolutional layer , the input feature maps are first disassembled into input tiles and , after the Winograd convolution , the output tiles will be reassembled into the output feature maps . Figure 1a shows how conventional Winograd convolution works . As the first step , the weight filter W and the input tile I are converted into the Winograd domain using the predefined matrices G and B. Element-wise multiplication is then applied to the Winograd-domain weight filter , GWG > , and input tile , B > I B , to generate the Winograd-domain output tile with a size of m ×m . In the last step , the output tile is converted back into the spatial domain with another predefined matrix A . With as the Hadamard product ( element-wise multiplication ) , the entire process can be written as O = A > [ ( GWG > ) ( B > I B ) ] A ( 1 ) The transform/inverse-transform matrices A , B and G are only determined by m and n. These matrices contain many repeating elements and applying them requires only few multiplications . In this case , considering only the element-wise multiplication between GWG > and B > I B , the Winograd convolution can reduce the number of multiplications from ( m− n+ 1 ) 2n2 to m2 . In addition to Winograd convolution , pruning is also a well-explored method to reduce CNN computation . Han et al . ( 2015b ; a ) propose to perform pruning and retraining iteratively , which can help reduce the computation by up to 5× . To fully utilize the sparsity incurred by pruning to accelerate CNN computation , Wen et al . ( 2016 ) and Yu et al . ( 2017 ) prune networks in a structured way : weights are clustered into groups with hardware-friendly structures and then get pruned in groups . However , Winograd convolution is not directly compatible with conventional pruning algorithms . The transformation GWG > fills in the zeros in the sparse weight filters generated by pruning . There have been several research attempts to solve this problem . Liu & Turakhia ( 2016 ) propose to directly mask out Winograd-domain weights and use backpropagation to train the spatial-domain weights . However , compared with spatial-domain weights , Winograd-domain weights are in a higher-dimensional space . Directly setting Winograd-domain weights to zero will cause an inconsistency between the spatial domain and the Winograd domain . This inconsistency will lead to a significant accuracy loss or a low sparsity on networks , e.g. , AlexNet , for large datasets ( Li et al. , 2017 ) . To address the inconsistency between the spatial and Winograd domain , Li et al . ( 2017 ) propose the sparse Winograd convolution . Figure . 1b shows how it works . Weight values are stored in the Winograd domain instead of the spatial domain . Both pruning and retraining are applied directly to Winograd-domain weights . This native pruning algorithm achieves > 90 % sparsity on AlexNet ( Krizhevsky et al. , 2012 ) but can not provide a high sparsity for deep networks ( Liu et al. , 2018 ) . Also , direct retraining in the Winograd domain requires an extremely small learning rate , e.g. , 200x smaller for AlexNet , which makes the retraining much slower . Based on sparse Winograd convolution , Liu et al . ( 2018 ) introduce the Winograd-ReLU pruning . It moves the ReLU function from the spatial domain into the Winograd domain . In this case , the computation of Winograd convolution becomes O = A > [ ( GWG > ) ReLU ( B > I B ) ] A ( 2 ) The Winograd-domain inputs also become sparse , which helps further reduce the required computation . Besides , a higher sparsity in Winograd-domain weights can be achieved . However , with weight filters being sparse , it is challenging to utilize both the weight and input sparsity for CNN acceleration on general-purpose processors due to more irregularity in the access pattern and control flow . Also , Winograd-ReLU pruning can not be applied to conventional CNN models since the new computation in Equation 2 does not correspond to the original convolution operation . It requires changing the network structure and retraining the network from scratch . 3 SPATIAL-WINOGRAD PRUNING . In this paper , to achieve a high Winograd-domain weight sparsity on deep CNN models without changing network structures , we propose the spatial-Winograd pruning . As shown in Figure 2 , it consists of two parts : spatial structured pruning and Winograd direct pruning . In spatial structured pruning , spatial-domain weights are pruned in a structured way and then retrained to regain the original accuracy . After spatial structured pruning , the weights of the pruned model will be transferred into and kept in the Winograd domain . The Winograd direct pruning then performs pruning and retraining directly onto the weights in the Winograd domain . The pruning and retraining steps in both spatial structured pruning and Winograd direct pruning will be iteratively executed until we achieve the desired sparsity or the produced model loses much accuracy . 3.1 SPATIAL STRUCTURED PRUNING . The first part of the spatial-Winograd pruning is spatial structured pruning . Spatial-domain weights which affect the same Winograd-domain weight are clustered into the same group . Less important weight groups are removed , and the pruned network will be retrained to regain the accuracy . This structured pruning method can help transfer more spatial-domain sparsity into the Winograd domain . Spatial Pruning For each spatial-domain filter W , we need to generate a mask M spatial to indicate the redundant weights . Assuming W has a size of n× n and the Winograd-domain filter Q is m×m , we have Q = GWG > . Each element of the Winograd-domain filter , Qi , j , is the weighted sum of the spatial-domain weights : Qi , j = ∑ 06u , v6n−1 ( Si , j , u , v ·Wu , v ) 0 6 i , j 6 m− 1 ( 3 ) where S is a 4D tensor containing the weight coefficients of the spatial-domain weights and is only determined by m and n. Details about the calculation of S can be found in Appendix A.1 . For each Winograd-domain weight Qi , j , we can create a set Di , j containing the spatial-domain weights which affect the value of Qi , j . Di , j is defined as Di , j = { Wu , v ∣∣ Si , j , u , v 6= 0 , 0 6 u , v 6 n− 1 } ( 4 ) In this case , for each weight group Di , j , we use a function h ( Di , j ) to measure its importance . In this paper , we use the maximum norm function as h h ( Di , j ) = max ( { |Wu , v| ∣∣Wu , v ∈ Di , j } ) ( 5 ) With a specific threshold tspatial , if h ( Di , j ) < tspatial , then Di , j is considered as redundant and all weights included need to be removed . In this case , the corresponding Qi , j will be fixed to 0 and also removed . The set of redundant weights for entire W is the union of all redundant Di , j and can be calculated as D = ⋃ 06i , j6m−1 , h ( Di , j ) < tspatial Di , j ( 6 ) Here we define Di , j in a structured way based on the relation between spatial-domain weights and Winograd-domain weights . It helps transfer as much spatial-domain sparsity into Winograd-domain sparsity as possible . The mask matrix M spatial can be generated by M spatialu , v = { 0 Wu , v ∈ D 1 Wu , v /∈ D 0 6 u , v 6 n− 1 ( 7 ) Spatial Retraining After spatial pruning , we can perform the spatial retraining with conventional training algorithms , e.g. , stochastic gradient descent ( SGD ) . The removed weights are fixed to 0 by applying W = W M spatial after each training iteration . is element-wise multiplication . The steps of spatial pruning and spatial retraining will be iteratively performed until the retrained model loses much accuracy . The threshold tspatial is gradually increased to incur more sparsity into the network . In spatial structured pruning , both pruning and retraining steps are performed in the spatial domain . It helps avoid the Winograd-domain retraining to accelerate the pruning process but , at the same time , incurs high Winograd-domain sparsity .

Summary: The paper proposes a technique (well, two) to prune convolutional layers to reduce the required amount of computation when  the convolutions are done using the winograd algorithm. Winograd convolutions first transform the image and the filter, apply a multiplication in the transformed space, and then retransform the image back to the intended image space. The transformation of the filter, however, means that sparsity in the regular domain does not translate to sparsity in the winograd domain.