RED : Looking for Redundancies for Data-FreeStructured Compression of Deep Neural Networks

1 Introduction . Modern Deep Neural Networks ( DNNs ) have become the mainstream approach in machine learning in general and in computer vision in particular , with CNNs achieving outstanding performance on various tasks such as object classification ( He et al. , 2016 ) , detection ( He et al. , 2017 ) or segmentation ( Chen et al. , 2017 ) . However , DNNs usually reach high requirements in terms of computational runtime . This prevents most state-of-the-art models to be deployed , most notably on edge devices . To address this shortcoming , a number of approaches for DNN compression have been proposed over the past few years . Architecture compression constitutes a convenient and popular way to address this runtime limitation , involving pruning as well as tensor decomposition techniques ( Cheng et al. , 2017 ) . It consists in either removing connections , i.e . an unstructured way or suppressing or reordering specific channels or filters i.e . a structured fashion . Although the former usually removes more weights than the latter ( Park et al. , 2020 ) , unstructured compression has the drawback to produce sparse weight matrices , which require dedicated hardware or libraries ( Han et al. , 2016 ) for real-case runtime improvements . Furthermore , these methods can also be divided in data-driven vs. data-free methods . While data-free methods are far more convenient for privacy concerns , as some data may be confidential ( e.g . health data or military ) , they are still significantly outperformed by data-driven methods . Hence , despite recent work ( Kim et al. , 2020 ; Tanaka et al. , 2020 ) , data-free architecture compression remains a challenging and promising domain with room for improvements . In this paper , we propose RED , a novel data-free structured compression framework . First , RED leverages a novel adaptive scalar hashing of the layer-wise weight distributions to introduce redundancies in DNNs . In particular , we show that this hashing allows to introduce vector redundancies ( i.e . neurons that perform the same operation ) as well as tensor redundancies ( i.e . low-rank flattened convolution kernels ) . These redundancies can respectively be exploited by applying similarity-based neuron 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . merging , as well as a novel uneven depthwise separation of convolutional layers . To sum it up , our contributions are : • An adaptive scalar weight hashing technique based on local extrema search of the weight distribution density , that introduces redundancies among neurons without significantly altering the predictive function . • A method for exploiting redundancies at the vector level with similarity-based neuron merging , and at the tensor level , with an uneven depthwise separation of convolutional layers that factors spatially redundant components . • We introduce RED , a portable method for data-free structured DNN compression that significantly outperforms state-of-the-art data-free methods and often rivals existing datadriven approaches . 2 Related Work . Most architecture compression methods rely on an underlying approximation of the predictive function to later perform pruning , wether it can is unstructured or structured ( as stated in Renda et al . ( 2020 ) ) , data-driven or data-free , magnitude-based or similarity-based . Predictive function approximation : Perhaps one of the most studied such approximation is quantization ( Nagel et al. , 2019 ; Meller et al. , 2019 ; Zhao et al. , 2019 ) . Quantization consists in mapping DNN weights to a finite , regular grid of values . It generally aims at reducing the inference time by coding this restricted set of values with fewer bytes ( e.g . float16 , int8 quantization ) , although it is generally non-adaptive to the weight distribution . Hashing constitutes another intuitive approach for predictive function approximation . Most hashing algorithms are extensions or variants of k-means ( Lloyd , 1982 ) which requires a prior on the studied distribution to determine a fitting value for the number of clusters . This step can also be formulated as a learning problem ( Wang et al. , 2019 ; Stock et al. , 2020 ) , but requires data as a consequence . In this work we propose a data-free , prior-free and adaptive hashing of the scalar weight distribution , which introduces redundancies in DNNs . Structured Pruning : On the one hand , unstructured approaches ( Frankle & Carbin , 2018 ; Lin et al. , 2020b ; Park et al. , 2020 ; Lee et al. , 2020 ) consist in removing individual weights : hence , these methods rely on sparse matrices for implementation , which require dedicated hardware to fully exploit the reduction at inference time . On the other hand , the so-called structured approaches ( Liebenwein et al. , 2020 ; Li et al. , 2017 ; He et al. , 2018 ; Luo et al. , 2017 ) aim at removing specific filters , channels or neurons . Although the latter usually results in less impressive raw pruning ratios as compared to the former , they allow significant runtime reduction without using dedicated hardware . Data-free Pruning : Most pruning methods can be classified as data-driven as they involve , to some extent , the use of a training database . Lee et al . ( 2020 ) uses the drift of DNN weights from their initial values during training to select and replace irrelevant weights . The Hrank method ( Lin et al. , 2020a ) consists in removing low-rank feature maps , based on the observation that the latter usually contain less relevant information . Lin et al . ( 2020b ) extend the single layer magnitude-based weight pruning to a simultaneous multi-layer optimization , in order to better preserve the representation power during training . Other approaches , such as ( Liebenwein et al. , 2020 ; Meng et al. , 2020 ) , train an over-parameterized model and apply an absolute magnitude-based pruning scheme which removes a number of channels or neurons but generally causes accuracy drop . To address this problem , most of these methods usually fine-tune this pruned model for enhanced performance ( Liu et al. , 2018 ; Gale et al. , 2019 ; Frankle & Carbin , 2018 ) . Nevertheless , there exists a number of so-called data-free approaches which do not require any data or fine-tuning of the pruned network , however usually resulting in lower pruning ratios . For instance , Tanaka et al . ( 2020 ) is a data-free pruning method with lower performances but still addresses the layer-collapse issue ( where all the weights in a layer are set to 0 ) by preserving the total synaptic saliency scores . Similarity-based Pruning : All the aforementioned methods remove connections during training via more or less adaptive or learnable thresholds under which DNN weights are pruned . Hence , such paradigm constitutes an absolute pruning heuristic . However , relative methods based on comparison between neurons or feature maps have also been proposed in the literature . For instance , Ayinde et al . ( 2019 ) use a graph-based-group-average technique to define proximity between feature maps in order to prune filters . Recently , in Kim et al . ( 2020 ) , similarity-based data-free pruning is applied by merging neurons . The authors decompose the weight tensor into new weights and a scaling matrix to fold into the next layer . However , in this approach , merging is performed at the tensor-level , rather than using a fine-grained pairwise similarity . Srinivas & Babu ( 2015 ) , conversely , propose to merge together neurons of each layer based on pairwise vector similarity between them . However , their approach doesn ’ t exploit scalar weight approximation , nor does it take into account tensor-level redundancies , resulting in low pruning ratios . In this work , we introduce redundancies in DNNs by performing scalar hashing of the layer-wise weight distributions . In particular , we show that this hashing allows to introduce vector redundancies ( i.e . neurons that perform the same operation ) as well as tensor redundancies ( i.e . low-rank flattened convolution kernels ) . These redundancies can respectively be exploited by applying similarity-based neuron merging in the same vein as Srinivas & Babu ( 2015 ) , as well as a novel uneven depthwise separation of convolutional layers . 3 Introducing Redundancies via Adaptive Weights Hashing . Let ’ s consider a DNN f with L layers f = fL · · · f1 . Each layer indexed by l is defined by parameters W l for a nl-dimensional output . As DNN weights take values in R , the probability for two values to be equal is almost surely zero : as a consequence , the probability to have redundancies in DNNs ( e.g . two neurons that perform the same operation , i.e . that have the same weight vector ) is also zero , thus limiting the possible simplification of the predictive function f . To deal with this , we propose to simplify f by first hashing the scalar weight distribution . As illustrated on Figure 1 , we approximate the density function of the weights distribution for each layer fl using Kernel Density Estimation ( KDE ) : dl : ! 7 ! 1 nl ⇥ nl 1 l X w2W l K ✓ ! w l ◆ ( 1 ) where K is the density of a Gaussian kernel with bandwidth l. Then , we estimate the local minima K l = ( m l k ) k and maxima K + l = ( M l k ) k of dl by computing its values over a discrete grid with range [ min { w 2 W l } ; max { w 2 W l } ] . Because the KDE provides a continuous density function , the intermediate value theorem guarantees that |K+l | + 1 = |K l | . We can thus partition R in |K+l | intervals with boundaries defined by the local minima and assign the value of the local maximum to all parameters within the corresponding intervals . Assuming the ( mlk ) k and ( M l k ) k sorted , with ml0 = 1 and ml|K l | 1 = +1 , for every weight w in W l there exists k such that w 2 ⇥ mlk ; m l k+1 ⇥ and w̃ = M lk . This defines the hashed layer f̃ l with weights W̃ l. Note that this method is adaptive and has no prior on the weight distribution contrary to k-means . In practice , we find that DNN weights concentrate around a limited number of local modes : hence , the proposed adaptive hashing dramatically reduces the number of different values that the weights can take , and introduces redundancies both at the vector and tensor level . Optionally , to conveniently enable further compression , we define ( global and per-layer ) contrast hyperparameters ⌧ = 1L PL l=1 ⌧ l. These hyperparameters allow to define a distance threshold ( relative to each layer ’ s weight values range ) under which two modes are collapsed to the dominant one , further increasing redundancies . We show in the experiments that hashing with ⌧ = 0 do not significantly alter the predictive function , thus setting ⌧ > 0 allows to find suitable trade-off between runtime acceleration and accuracy . 4 Exploiting Redundancies in DNNs . As the hashed weights W̃ l take values in a finite set , redundancies are much more likely to occur both at the vector and tensor level . Vector Redundancies : First , let ’ s consider the case of a two-layers fully-connected neural network f with an element-wise activation function and no biases , i.e . f̃ : z 7 ! W̃ 2 ( W̃ 1z ) with W̃ 1 2 K+1 n1⇥n0 and W̃ 2 2 K+2 n2⇥n1 the hashed parameters . Let ’ s also assume that we have n̄1 < n1 distinct neurons , i.e . n̄1 distinct rows in W̃ 1 . Let W̄ 1 be the sub-matrix of W̃ 1 containing all the distinct rows of W̃ 1 only once and W̄ 2 the matrix such that all columns from W̃2 that were applied to identical neurons of W̃ 1 are summed . Then , for each output dimension i we have : ⇣ f̃ ( z ) ⌘ i = n1X j W̃ 2i , j n1X k W̃ 1j , kzk ! = n̄1X j W̄ 2i , j n̄1X k W̄ 1j , kzk ! = ⇣ f̄ ( z ) ⌘ i ( 2 ) In other words , we can merge identical neurons ( rows of W̃ 1 ) and sum the corresponding weights in W̃ 2 without altering the ( hashed ) layers outputs , as illustrated in Figure 2 on a simple case . This process can straightforwardly extended to neural networks with L layers by repeating this process from the first to last layer . Furthermore , it can be adapted to : • Layers with bias by considering W̃ 0 = ( W̃ b̃ ) and z0 = ( zT 1 ) T • Convolutional layers , by using a rewriting of the kernel , following Ma & Lu ( 2017 ) • Batch-Normalization layers , by folding them like in Nagel et al . ( 2019 ) • Skip Connections , where each output channel is computed using the corresponding weights in W̃ l and W̃ l+k . Merge is performed on the two added layers simultaneously by considering the concatenation W̃ l W̃ l+k . In the case of depthwise convolutions ( Sandler et al. , 2018 ; Howard et al. , 2019 ) , each filter is applied to a distinct input . As a consequence , two filters intrinsically can not perform identical operations and thus can not be merged . In practice , we find that after hashing , such exact neuron merging already allows to remove significant number of parameters . However , to obtain larger pruning ratios , this merging can be relaxed : for a layer l , we sort the set of pairwise distances between neurons or filters ( e.g . as defined by the Euclidean distance between their weight vectors ) and merge the ↵l % closest neurons by taking their average weight values . We thus define ( global and layer-wise ) hyperparameters ↵ = 1L PL l=1 ↵ l. Tensor Redundancies : The proposed hashing method also induces tensor-level redundancies in regular 2D convolutional layers : to handle these redundancies we propose a novel uneven depthwise separation method . Let ’ s consider a kernel W̄ 2 K+w⇥h⇥nin⇥nout for a layer l after either the hashing or neuron merging steps ( as merging and the proposed uneven depthwise separation steps can be applied in any order , see Appendix A.1 ) . W̄ can be expressed as a depthwise separable convolution if there exists D 2 Rw⇥h⇥nin⇥1 and P 2 R1⇥1⇥nin⇥nout the respective weights of depthwise and pointwise convolution such that : 8y , x , i , j , W̄ [ y , x , i , j ] = D [ y , x , i,1 ] P [ 1,1 , i , j ] ( 3 ) This condition can be expressed in terms of matrix ranks , as illustrated on Figure 3 . We define the matrix Wi as the restrictions of filters to input channel i , flattened and concatenated along the output channel axis : Wi = 0 B @ W̄ [ 1,1 , i,1 ] . . . W̄ [ w , h , i,1 ] ... . . . ... W̄ [ 1,1 , i , nout ] . . . W̄ [ w , h , i , nout ] 1 CA ( 4 ) Thus , any convolutional layer can be converted to a depthwise separable convolution layer if we have ri = 1 for all i . In such case a basis B1i = D [ : , : ,i,1 ] of the rows of the matrix Wi can trivially be retrieved by considering the first non-zero row and the whole layer can be transformed to a depthwise convolutional layer . In practice , we find that it happens often after hashing . Otherwise , for all i such that ri 6= 1 we denote ( Bki ) a basis of the rows of Wi , with k 2 J1 ; riK . Then , for every j 2 J1 ; noutK there exists µki 2 R and k 2 J1 ; ryK such that W̄ [ : , : ,i , j ] = µ j iB k i . Thus , in this case , each output channel j gives rise to a number of basis kernels ( Bki ) k=1 ... ri , and the corresponding subsequent point-wise convolutions with coefficients µji = P [ 1,1 , i , j ] , that depends of the rank of its concatenated flattened filter restriction matrices . Hence , we call this layer an uneven depthwise convolution with kernels defined as the basis kernels Bki . To sum it up , the proposed RED method consists in three steps : an adaptive scalar hashing step ( with hyperparameter ⌧ ) , followed by pruning via similarity-based neuron merging ( with hyperparameter ↵ ) , and an uneven depthwise separation step ( summarized as algorithms in Section A.2 of the supplementary material ) . As it will be shown in the experiments RED significantly outperforms other data-free architecture compression methods and often matches data-driven methods .

Summary: The author propose a data free pruning pipeline to compress a given deep neural network. The author propose to first apply adaptive weight hashing to force the weights tensors to be similar and then merge weight tensors based on a calculated similarity score. The proposed methods use kernel density function for hashing and do not require training data for hashing the weights tensors. The proposed pipeline (RED) is verified on CIFAR-10 and ImageNet.