RED : Looking for Redundancies for Data-FreeStructured Compression of Deep Neural Networks
Authors: Edouard YVINEC
Arnaud Dapogny
Matthieu Cord
Kevin Bailly
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep learning
computer vision
pruning
data-free
hashing
compression
acceleration

Abstract:  Deep Neural Networks (DNNs) are ubiquitous in today's computer vision landscape, despite involving considerable computational costs. The mainstream approaches for runtime acceleration consist in pruning connections (unstructured pruning) or, better, filters (structured pruning), both often requiring data to retrain the model. 
 In this paper, we present RED, a data-free, unified approach to tackle structured pruning. First, we propose a novel adaptive hashing of the scalar DNN weight distribution densities to increase the number of identical neurons represented by their weight vectors. Second, we prune the network by merging redundant neurons based on their relative similarities, as defined by their distance. Third, we propose a novel uneven depthwise separation technique to further prune convolutional layers. We demonstrate through a large variety of benchmarks that RED largely outperforms other data-free pruning methods, often reaching performance similar to unconstrained, data-driven methods.

RED Looking for Redundancies for Data-Free Structured Compression of Deep Neural Networks
Edouard Yvinec Arnaud Dapogny², Matthieu Cord Kevin Bailly Sorbonne Université CNRS, ISIR, f-75005, 4 Place Jussieu 75005 Paris, France Datakalab², 114 boulevard Malesherbes, 75017 Paris, France ey@datakalab.com
Abstract
Deep Neural Networks (DNNs) are ubiquitous in today's 's computer vision land- scape, despite involving considerable computational costs. The mainstream ap- proaches for runtime acceleration consist in pruning connections (unstructured pruning) or, better, filters (structured pruning), both often requiring data train the model. In this paper, we present RED, a data-free structured, unified approach to tackle structured pruning. First, we propose novel adaptive hashing of the scalar DNN weight distribution densities to increase the number of identical neurons represented by their weight vectors. Second, we prune the network by merging redundant neurons based on their relative similarities, defined by their distance. Third, we propose novel uneven depthwise separation technique further prune convolutional layers. We demonstrate through large variety of benchmarks that RED largely outperforms other data-free pruning methods, often reaching performance similar to unconstrained, data-driven methods.
Introduction
Modern Deep Neural Networks (DNNs) have become the mainstream approach in machine learning in general and in computer vision in particular, with CNNs achieving outstanding performance various tasks such as object classification (He et al. 2016). detection (He et al., 2017) or segmentation (Chen al. 2017). However, DNNs usually reach high requirements in terms of computational runtime. This prevents most state-of-the-arr models to be deployed, most notably on edge devices. To address this shortcoming, a number of approaches for DNN compression have been proposed over the past few years. Architecture compression constitutes a convenient and popular way to address this runtime limitation, involving pruning as well as tensor decomposition techniques (Cheng et al.. (017)). It consists in either removing connections, i.e. unstructured way or suppressing reordering specific channels or filters i.e. structured fashion. Although the former usually removes more weights than the latter (Park et al. 2020), unstructured compression has the drawback to produce sparse weight matrices, which require dedicated hardware or libraries (Han et all 2016 real-case runtime improvements. Furthermore, these methods can also be divided data-driven vs. data-free methods. While data-free methods are far more convenient for privacy concerns, as some data may be confidential (e.g. health data or military), they still significantly outperformed by data-driven methods. Hence, despite recent work (Kim et al. 2020; Tanaka et al. 2020), data-free architecture compression remains challenging and promising domain with room for improvements. In this paper, we propose RED, a novel data-free structured compression framework. First, RED leverages novel adaptive scalar hashing of the layer- wise weight distributions introduce redundancies in DNNs. In particular, we show that this hashing allows to introduce vector redundancies (i.e. neurons that perform the same operation) well tensor redundancies (i.e. low-rank flattened convolution kernels). These redundancies can respectively be exploited by applying similarity-band neuron
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

merging, as well as a novel uneven depthwise separation of convolutional layers. To sum it up, our contributions are:
An adaptive scalar weight hashing technique based on local extrema search of the weight distribution density, that introduces redundancies among neurons without significantly altering the predictive function.
A method for exploiting redundancies at the vector level with similarity- neuron merging, and at the tensor level, with uneven depthwise separation of convolutional layers that factors spatially redundant components.
We introduce RED, portable method for data-free structured DNN compression that significantly outperforms state-of-the-art data-free methods and often rivals existing data- driven approaches.
2 Related Work
Most architecture compression methods rely on an underlying approximation of the predictive function to later perform pruning, wether it can is unstructured or structured (as stated in Renda et al. 22200). data-driven or data-free, magnitude-basee based or similarity-base.
Predictive function approximation: Perhaps one of the most studied such approximation is quan- tization (Nagel et al. 2019| [eller et al. 2019 Zhao et al. 2019). Quantization consists in mapping DNN weights finite, regular grid of values. It generally aims reducing the inference time by coding this restricted set of values with fewer bytes (e.g. float16, int8 quantization), although is generally non-adaptive the weight distribution. Hashing constitutes another intuitive approach for predictive function approximation. Most hashing algorithms are extensions or variants of k-means [Looyd] 1982) which requires a prior on the studied distribution to determine a fitting value for the number of clusters. This step can also be formulated as a learning problem (Wang et al. 2019 Stock al. 2020), but requires data as a consequence. In this work we propose a data-free, prior-free and adaptive hashing of the scalar weight distribution, which introduces redundancies in DNNs.
Structured Pruning: On the one hand, unstructured approaches (Frankle & Carbin 2018 Lin
et al., 20205; Park et al. 2020) Lee et al., 2020) consist in removing individual weights: hence, these methods rely on sparse matrices for implementation, which require dedicated hardware to fully exploit the reduction at inference time. On the other hand, the so-called structured approaches (Liebenwein et al., 2020 Li al. 2017 He et al., 2018 Luo al. 2017) aim at removing specific filters, channels or neurons. Although the latter usually results in less impressive raw pruning ratios as compared to the former, they allow significant runtime reduction without using dedicated hardware.
Most pruning methods can be classified as data-driven as they involve, to some extent, the use of training database. [ee et al. (220)) uses the drift of DNN weights from their initial values during training to select and replace irrelevant weights. The Hrank method (Lin et al., 2020a) consists removing low-rank feature maps, based on the observation that the latter usually contain less relevant information. Lin et al. (020b) extend the single layer magnitude- pruning to a simultaneous multi-layer optimization, in order to better preserve the representation power during training. Other approaches, such as (Liebenwein et al. 2020: Meng et al. 2020), train an over- weer-paaaeeteriee model and apply an absolute magnitude- based pruning scheme which removes a number of channels or neurons but generally causes accuracy drop. To address this problem, most of these methods usually fine-tune this pruned model for enhanced performance (Liu et al. 2018 Gale et al.] 2019 Frankle & Carbin 2018). Nevertheless, there exists number of so-called data-free approaches which do not require any data or fine-tuning of the pruned network, however usually resulting in lower pruning ratios. For instance, Tanaka et al. (2000) is data-free pruning method with lower performances but still addresses the layer-collapse issue (where all the weights in layer are set to 0) by preserving the total synaptic saliency scores.
Data-free Pruning:
weight
Similarity- -ased Pruning: All the aforementioned methods remove connections during training via more or less adaptive or learnable thresholds under which DNN weights are pruned. Hence, such paradigm constitutes an absolute pruning heuristic. However, relative methods based on comparison between neurons or feature maps have also been proposed in the literature. For instance, Ayinde

---

M2
M2
M
d'
M
ma
M
W
MO m
m4
5
m
m22
my
m
Figure 1: Illustration of the proposed adaptive scalar weight hashing, First (left), we find the estimate of the density function d' associated to the weights values W Second (right), we find the local extrema (mi))ek and (MA of d'. Then we assign the new values W E Ky K.t"ixm--1
soreeeddrou--avveee average technique define proximity between feature maps in order to prune filters. Recently, in Kim et al. (2020), similarity- data-free pruning is applied by merging neurons. The authors decompose the weight tensor into new weights and scaling matrix to fold into the next layer. However, in this approach, merging is performed at the tensor-level, rather than using fine-grained pairwise similarity. Srinivas & Babu (2015). conversely, propose to merge together neurons of each layer based on pairwise vector similarity between them. However, their approach doesn't exploit scalar weight approximation, nor does take into account tensor-level redundancies, resulting in low pruning ratios.
al. (019) use
In this work, we introduce redundancies DNNs by performing scalar hashing of the layer-wise weight distributions. In particular, show that this hashing allows to introduce vector redundancies (i.e. neurons that perform the same operation) as well as tensor redundancies (i.e. low-rank flattened convolution kemelss.. These redundancies respectively be exploited by applying similarity-baase based neuron merging in the same vein as Srinivas & Babu (2015), as well as a novel uneven depthwise separation of convolutional layers.
3 Introducing Redundancies via Adaptive Weights Hashing
Let's consider a DNN f with L layers = fL ... fi Each layer indexed by I is defined by parameters W for ny-dimensional output. As DNN weights take values R, the probability for two values to be equal is almost surely zero: consequence, the probability to have redundancies DNNs (e.g. two neurons that perform the same operation, i.e. that have the same weight vector) is also zero, thus limiting the possible simplification of the predictive function f. To deal with this, we propose to simplify by first hashing the scalar weight distribution. As illustrated on Figure [ we approximate the density function of the weights distribution for each layer fi using Kernel Density Estimation (KDE):
d' H.ww Ti 1
Σ WEW
K
A
where K is the density of Gaussian kernel with bandwidth A, Then, we estimate the local minima KJ- = (m/), and maxima K,* (M²) of d; by computing its values over discrete grid with range mmin(w tion, the intermediate value theorem guarantees that | + |- We can thus partition IR in Kitl intervals with boundaries defined by the local minima and assign the value of the local maximum to all parameters within the corresponding intervals. Assuming the (m/) & and (M)) sorted, with m/ ooo and m |i, |-1 +oe, for every weight w in W' there exists k such that
max E W.). Because the KDE provides continuous density func-
w E [m]: m and i = M This defines the hashed layer J with weights W Note that this method adaptive and has no prior on the weight distribution contrary to k-means. In practice, we find that DNN weights concentrate around a limited number of local modes: hence, the proposed adaptive hashing dramatically reduces the number of different values that the weights can take, and introduces redundancies both at the vector and tensor level. Optionally, to conveniently en- able further compression, define (global and per-layer) contrast hyperparameters = E
3

---

to mengeed
W + W W
Wasl
W W
Wi W
Waz!
Figure 2: Neuron merging in the case of fully-com mully-connected layer I with weights W Similar colors indicate equal weight values, e.g. W W The pruned network weights W are obtained by merging the first 2 neurons of layer I and simply summing the corresponding weights in layer I + 1.
These hyperparameters allow to define distance threshold (relative to each layer's weight values range) under which two modes are collapsed to the dominant one, further increasing redundancies. We show in the experiments that hashing with т = 0 do not significantly alter the predictive function, thus setting 7 > 0 allows to find suitable trade-off between runtime acceleration and accuracy.
4 Exploiting Redundancies in DNNs
As the hashed weights Wi take values in a finite set, redundancies are much more likely to occur both at the vector and tensor level.
First, let's consider the case of two-layers fully-connected neural network with an element-wise activation function a and no biases, i.e. f + W2o(WW12) with W E Ktt 7 and W22 E K2 K the hashed parameters. Let's also assume that we have ū1 < distinct neurons, i.e. ūı distinct rows in W Let W be the sub-matrix of W containing all the distinct rows of W only once and W the matrix such that all columns from W2 that were applied to identical neurons of W1 are summed. Then, for each output dimension i we have:
Vector Redundancies:
R2
D1 Σ
81
a1
(f()) - Σ W
W J.
Σ W2
Σ
W W3J
(f(2),
In other words, we can merge identical neurons (rows of W and sum the corresponding weights on simple case. This
in 2 without altering the (hashed) layers outputs, as illustrated in Figure 2 process can straightforwardly extended to neural networks with layers by repeating this process from the first to last layer. Furthermore, it can be adapted to:
Layers with bias by considering W" = (W b) and z'' == 1)T
Convolutional layers, by using a rewriting of the kernel, following Ma Lu 2017)
Batch-Normalization layers, by folding them like in Nagel al. (2019)
Skip Connections, where each output channel is computed using the corresponding weights in Wi and W Merge is performed on the two added layers simultaneously by considering the concatenation (w> W=++)).
In the case of depthwise convolutions (Sndlerr et al. 2018 Howard et al. 2019), each filter is applied distinct input. As consequence, two filters intrinsically cannot perform identical operations and thus cannot be merged.
In practice, we find that after hashing, such exact neuron merging already allows to remove significant number of parameters. However, to obtain larger pruning ratios, this merging can be relaxed: for a layer 1, we sort the set of pairwise distances between neurons or filters (e.g. as defined by the Fuclidean distance between their weight vectors) and merge the 1% closest neurons by taking their average weight values. We thus define (global and layer-wise) hyperparameters = E'
Tensor Redundancies:
The proposed hashing method also induces tensor-level redundancies in regular 2D convolutional layers: to handle these redundancies we propose novel uneven depthwise

---

Hashed Conv
grouped by Input channel
Separalion extractinn on flattened Kernels
Depthwise Separable Conv
W
20
P
extraction norm
B7 1
20
basis extraction
B
Figure 3: Tensor-level Simplification via of our uneven depthwise separation method for a layer with 2 imput and output channels. Restrictions of filters to each input channel are flattened and concatenated along the output channel axis, to form matrices 20, If these matrices have rank rú we extract basis B/ (for j in [1;r of the rows of each 23, and normalize it so that the last element is equal to 1. Then we deduce the weights of the pointwise kernels P from the (B() ) and the W.
for layer after either the hashing or neuron merging steps (as merging and the proposed uneven depthwise separation steps can be applicd in any order, see Appendix A.1). 1). can be expressed as a depthwise separable convolution if there exists D 1= x1 and P E RR 21x1xX% X the respective weights of depthwise and pointwise convolution such that
separation method. Let's consider kernel W E K+ w x
Vy, 1,
Wiaaa D Da PI,t,al
(3)
This condition be expressed in terms of matrix ranks, illustrated Figure 3
We define the
matrix 20, as the restrictions of filters to input channel i, flattened and concatenated along the output channel axis:
W WILLAJ
20,
(4)
W (mnl
Thus, any convolutional layer can be converted to depthwise separable convolution layer if we have r; = for all i. In such case basis B!- D iJ of the rows of the matrix 21, can trivially be retrieved by considering the first non-zero row and the whole layer can be transformed to a depthwise convolutional layer. In practice, we find that happens often after hashing. Otherwise, for all such that r; * we denote (23))) basis of the rows of 21, with 1 E [1;r:] Then, for every j E [1; noull n Thus, in this case, each output there exists 1 E R and [[:ry] such that W [[s.))) channel j gives rise number of basis kernels (B k= and the corresponding subsequent point-wise convolutions with coefficients μr; = PI...i.)) that depends of the rank of its concatenated flattened filter restriction matrices. Hence, call this layer uneven depthwise convolution with kernels defined as the basis kernels B...
To sum up, the proposed RED method consists in three steps: adaptive scalar hashing step (with hyperparameter r), followed by pruning via similarity- neuron merging (with hyperparameter a), and uneven depthwise separation step (summarized algorithms in Section A.2 of the supplementary material). As it will be shown the experiments RED significantly outperforms other data-free architecture compression methods and often matches data-driven methods.
Experiments
5.1 Experimental setup
Datasets and baselines: we evaluate our models on the two de facto standard datasets for architec- ture compression, iee. CIFAR-10 ((rrihhvssy et al. 2009), under the MIT License) and ImageNet (Deng al., 2009), under the BSD-3 license). We use the standard evaluation metric of removed parameters as well as removed FLOPs. We apply our approach on ResNet ((He et al. 2016). ResNet 20-56-1 10 and 164 with respective number of parameters 270k, 852k, 1.7M and 2.6M, and accuracies 42.48, 93.46, 93.81 and 94.54 on CIFAR-10 10 and ResNet 50 with 25M parameters and 76. 6..7 accuracy on ImageNet) as well as ReseeR (Zaoorykko & Komodakis) 2016). Wide ResNet architectures defined by their number of layers well as their wideness multiplier: evaluate Wide
5

---

2 10
- 2
= 10
density amplitude
42
density amplitude
-i= 50
-1.0
1.0
1.0
rescaled weight values ranges Figure 4: Weight distributions for several layers of a ResNet-56 on CIFAR-10 (left) and ImageNet (right) reveal that weights concentrate around multiple modes that can be captured via hashing.
rescaled weight values ranges
ResNet 16-8 (with 1 11.0M and 95. 2 accuracy on CIFAR-I0), 22-2 (with 1.5 and 94.1 accuracy), 28 (28-2, 28-4, 28-8 and 28-10 with 1.9M, 7.4M, 29.8M and 36.5M parameters and 94.3, 94.8, 95.4 and 95.8 accuracies) and 40-4 (with 8.9M parameters and 95.0 accuracy).
The proposed adaptive hashing is implemented using Scikit-learn python library, with bandwidth A set as the median of the differences between consecutive weight values per layer (see Appendix 10 and the contrast hyperparameter T set to O, by default. Merging and depthwise separation are implemented using Numpy: for relaxed merging, and unless stated otherwise, hyperparameter a is set to the highest value that fully preserves the model accuracy (noted a²). We apply per block strategy for setting the layer-wise (a') and constant strategy for the (r' T). Different strategy for setting the layer-wise (a') and (r²) are discussed in Section A.3 of the supplementary material. We ran our experiments on a Intel(R) Core(TM) i7-7820X CPU. The proposed hashing method processing time depends on the model's size: ranging from 45 to 413 seconds for ResNets on CIFAR-10 10 and up to 21 hours for a ResNet-50 on ImageNet. This step could however be accelerated by layer-wise parallelization as discussed in Appendix A.9 Pruning, on the other hand, is much faster, taking seconds for ResNet-20 on CIFAR-10 and 15 minutes for a ResNet-50 on ImageNet.
Implementation details:
5.2 Hashing empirically induces vector and tensor-level redundancies
First, Figure displays the weight distributions for several layers for ResNet-56 trained on CIFAR-10 and ResNet-50 on ImageNet. We observe that the weights concentrate around multiple dominant modes, with the number modes and their relative proximity being variable from one layer to another: this motivates the design of the proposed adaptive hashing method, where each weight can be assigned to its corresponding mode, with optional collapse of adjacent modes.
In order to measure the impact of the change in the predictive function f from the scalar hashing considered the average етror induced by hashing in logits [x) f=))]]. This error is compared to the distribution of the differences between the top1 and top2 logits. An error smaller than the aforementioned difference implies that the modifications due to hashing can not change the prediction made by the DNN. For instance, we observe on ResNet 56 on CIFAR-10 10 an average error of 0.75 while the average difference between top 1 ad top 2 logits is 10.43 in the baseline model with standard deviation of 6.29. Furthermore, we observe similar values on other considered networks (for details see Appendix A.4), which empirically validates the efficiency of the hashing mechanism preserving the accuracy.
Figure also shows the evolution, during training and for each layer, of the removed parameter ratio (obtained with hashing and either merging, relaxed merging and RED, which consists in applying relaxed merging plus uneven depthwise separation). Initially, for the first epochs, there are no vector redundancies (left and central plots) and few tensor redundancies (right plot), due to the initialization scheme (e.g. Glorot or He) specifically designed to avoid vector redundancies. However, we observe a rapid convergence towards high numbers of both vector and tensor redundant weight distributions after only 10 epochs. In fact, we show in Section A. of the supplementary material that the effectiveness of RED is robust to the choice of the initialization scheme. Additionally, we show in Section A.5. of the supplementary material that we observe the same phenomenon even in the presence of training methods that aims at promoting diversity among neurons (c.g. dropout).

---

Merging
Lossy Relaxation
RED
compresson
compression
comp
D.6
04
82
4
epochs
is epochs
d 4
epochs
10
dn
10
layers
layers
layers
Figure 5: 3D plot of the removed parameters ratio as a function of the layer's depth and the training epoch for ResNet-56 ResNet 50 CIFAR-10. 10. Regardless of the initialization schemes designed to avoid redundancies, the hashed networks present large numbers of vector and tensor redundancies, that can be removed with minimal impact the accuracy.
Table 1: Ablation results in terms of % removed parameters compared the base model.
Hashing
/
/
Merge
0 a
a
Q
0
a
a
a
Depthwise Separation
/
X
X
ResNet 20
0.00
18.58
18.58
25.18
41.03
65.05
ResNet 56
0.00
61.19
61.19
58,45
77.68
84.52
ResNet 110
0.00
75.29
75.29
62.41
84.43
89.64
ResNet 164
0.00
78.61
78.61
62.73
88.87
91.06
Wide ResNet 16-8
0.00
31.08
31.08
19.67
38.67
51.92
Wide ResNet 22-2
0.00
51.17
51.17
13.27
63.67
64.98
Wide ResNet 28-2
0.00
49.19
49.19
11.46
61.20
64.19
Wide ResNet 28-4
0.00
41.79
41.79
20.99
51.99
56.07
Wide ResNet 28-8
0.00
33.58
33.58
19.78
41.78
52.87
Wide ResNet 28-10
0.00
47.25
47.25
25.59
58.79
60.80
Wide ResNet 40-4
0.00
49.67
49.67
43.37
61.80
70.35
As such, after hashing, merging exact redundancies removes 0 50% of the weights, depending on the layer, with more emphasis the last layers. Relaxed merging (which looks after similar but necessarily equal neurons) removes up to 60% of the layers' weights without changing the network accuracy. Furthermore, we can remove lot of parameters (40 85%) with uneven depthwise separation among all the layers.
5.3 Ablation study
Table presents results in term of % removed parameters for each step in RED for both ResNet and Wide ResNet architectures on CIFAR-10. First, we observe that without hashing, the % removed parameters is significantly lower for every model and pruning method combination thereof. For instance, on ResNet 20, Merge (a = 0) with hashing (column 4) outperforms Merge (a a*) without hashing. Furthermore, it can be seen by comparing the first two columns that depthwise separation does not add much without hashing: therefore, we argue that scalar hashing is critical to introduce vector and tensor-level redundancies without accuracy loss.
As such, hashing merging (a 0) already achieves good results without altering the predictive function (up to 62. 62.7% on deeper networks, e.g. ResNet-166)) Furthermore, setting a² and using depthwise separation allows to reach higher removed parameters rates, e.g. up to 90% on ResNet- 110 and 164, without witnessing any accuracy drop. The experiments on thinner the network the higher the removed parameters rates, with Wide ResNet 40-4 and 22-2 having highest removed parameters ratio and 16-8 and 28-8 having the lowest. This is natural since, from combinatorial standpoint, assuming similar prior, the chance to find redundancies (strict or approximate) reduces as the channel depth increases. Last but not least, these results are very stable, observe standard deviations between 0.09 0.84 for the different models over 10 runs
show that the

---

Restiet
Resivet
ResNet 110
ResNet 164
Compression
60
02
0.5
650
0.0
1.0
03
0.2
0.5 09
0.4
60
02
010
1.5
022
28
12
Figure 6: Accuracy drop (%, red) and compression (in term of % removed parameters, blue) vs. values of a for ResNet 20, 56, 1 110 and 164 on CIFAR-10. We can remove high number of similar neurons without impacting the accuracy, particularly for deeper networks.
Resnet-20
Resnet-56
Resnet-110
000
90
12
87
96
95
25
30
35
45
50
55
15
25
35
65
65
15
25
romaning paramottrss
remaining parameters
56 remaining parameters
Figure 7: Comparison between RED and state-of-the-are methods on Cifar-10, in terms of % accuracy (measured as percentage of the base model accuracy, the higher the better) and % remaining param- eters (the lower the better). Each method classified either as data-free (red) or data-driven (blue) and structured (triangle) or unstructured (rectangle). For each network, RED performs significantly better than other data-free methods, and often as well as data-driven or unstructured methods.
of retraining and application of RED.
Figure 0 shows the compression rate, and accuracy drop for relaxed merging function of parameter a. For ResNet 20 we observe an initial low-slope phase for which merging neurons have lower influence the accuracy. For ResNet 56 and deeper, we observe initial plateau where merging together large number of similar neurons (up to 50% and 80% for ResNet 56 and 110- respectively) does not impact the accuracy at all. This further illustrates that the hashed DNN (and particularly deeper networks) weights present structural redundancies, and that similarity- -aased analysis can dramatically increase their efficiency. Note that the theoretical pruning factor of the merging and tensor decomposition steps are discussed in Section A.6 of the supplementary material. Also, report the GPU and FLOPs performance of RED in Section A.7. Finally, hashing also affects memory footprint as evaluated in Appendix A.8.
5.4 Comparison with state-of-the-art approaches
Figure draws a comparison between RED and ofate-of-the-art architecture compression methods, in terms of % remaining parameters and % accuracy w.r.t the base model. For comparison purposes, we report results on the most popular architectures, i.e. ResNet 20, 56 and 110. Each of these approaches is either classified as:
Comparisons on CIFAR-10:
Data-driven (blue): methods that rely on fine-tuning or retraining protocols requiring labelled data, such as LDI (Lee et al. 2020). DPF (Lin et al.] 2020b). PFP (Liebenwein et al. 2020).
FT (Li et al., 2017). SoftNet (He et al., 2018), Lottery (Frankle Carbin 2018). PoReg
Zhuang et al. 2020), PFF (Meng et al. 2020). OS (Renda et al.) 2020) and SCOP (Tang
et al.] 2020).
Data-free (red): methods that do not use data such as Merge (Kim et al. 2020), SynFlow
Tanaka et al. 2020), or methods that generate synthetic data for fine-tuning such as Dream
Yin et al. 2020).
Another classification of these methods lies on the type of pruning: we distinguish unstructured meth- ods (depicted by square) from structured methods (triangle). Generally speaking, RED performs significantly better than all the others satte-of-the-ar data-free methods: it outperforms its closest contenders, SynFlow (Tanaka et al. 2020) on ResNet 20, as it removes 15% more parameters without accuracy drop. When compared to the most similar method (Kim al. 2020), the merging step alone (with 0) already achieves 18% higher removed parameters ratio on ResNet 56. Furthermore, the complete RED approach significantly widens the gap and reach 44% higher pruning ratio on ResNet 56. Furthermore, merging and RED also allows higher removed parameters ratios on Wide
8

---

Table 2: Comparison with state-of-the-art methods in term of % removed parameters/FLOPS and
(top-1, top-5) accuracy on a ResNet-50 trained on ImageNet. Gray cells highlight data-free methods.
Model
%removed parameters
removed FLOPs
Top-1% 1%
Top-5%
Baseline
76.2
92.9
GAL-0.5
16.9
43.0
72.0
90.9
Dream
20.0
37.0
73.3
SSS-32
27.1
31.1
74.2
91.9
Hrank1.58
36.7
43.7
75.0
92.3
RED 7=0
39.6
42.7
76.1
92.9
GAL-1
42.5
61.4
69.9
89.8
SCOP
42.8
45.3
76.0
92.8
RED 7=0.05
42.1
44.5
75.3
92.1
Hrank
46.0
62.1
72.0
91.0
RED 7=0.10
47.3
47.9
74.1
91.1
Dreamço
50.0
71.0
60.7
RED 7=0.15
54.7
55.0
71.1
90.7
RED 7=0.20
56.9
57.3
67.9
90.3
Hrank
67.7
76.0
69.1
89.6
ResNet 40-4 (5.6% and 30%, respectively), due finer pairwise similarity modeling well as handling structural redundancies through uneven depthwise separation. RED also outperforms the best structured, data-driven method, PFP (Liebenwein et al. 2020) by achieving similar compression ratios on the three networks without any accuracy drop. Last but not least, RED, despite being data-free approach. is competitive with recent data-driven and unstructured methods such as DPF in et al. 2020b), Lottery (Frankle & Carbin 2018) or LDI (Lee et al. 2020).
Comparisons on ImageNet: Table 2 shows a comparison between RED and other state-of-the.
art structured pruning approaches ImageNet. These methods allow find different levels of compromise between the removed parameter ratio (as indicated by the % remaining parameters) and the accuracy drop.In order find such trade-off with RED, we vary the contrast hyperparameter r in the hashing step: by merging together close modes of the weight distribution, we introduce more redundancies, allowing to further compress the network the expense of accuracy. Although this initial hashing step may cause the network accuracy to drop, both redundant neuron merging and uneven depthwise separation induce negligible loss, as echoed by the previous experiments. Thus, RED allows to remove between 39.6% (42.7% FLOPs) of a ResNet-50 with only 0.1% top-1 accuracy drop, and (with 7 0.20) 56.9% 77% FLOPs) with 8.3% top-1 accuracy drop. Thus, RED appears as a more efficient pruning algorithm than data-free free Dream (Yin et al., 2020), allowing remove large numbers of parameters with minimal accuracy drop. Furthermore, RED is once again very competitive with recent data-driven methods such as Hrank (Lin et al. 2020a). SCOP (Tang et al. 2020), SSS (Huang & Wang 2018) or GAL (Lin et al. 2019). While some recent data-driven, unstructured approaches achieved higher levels of compression on this benchmark Evci et al. (2020), these results show the potential RED as an efficient, portable and privacy compliant data-free, structured pruning method. In addition to that, tested RED more challenging backbones in Appendix 12
6 Conclusion
In this paper, we proposed RED,, novel data-free structured architecture compression method. First, RED uses novel adaptive scalar hashing of the weight distributions to introduce redundancies in DNNs under the form of vector redundancies well as tensor redundancies. These redundancies can be exploited with similarity- bassed neuron merging, as well as novel uneven depthwise separation scheme for convolutional layers, respectively. Furthermore, we demonstrated through thorough experiments involving several architectures and databases, that RED significantly outperforms other data-free, structured pruning methods and often matches recent state-of-the data-driven pruning techniques, at very minimal expanse in term of accuracy drop.

---

The whole approach is fully data-free and easy-to-use. On the one hand, DNN acceleration leads either less energy consumption inference or higher accuracy on a given device. On the other hand, the advantage of data-free methods is the compliance with data-privacy laws, particularly for applications where data access sensible and the need to share such data may infringe privacy laws or make partnerships with external actors difficult.
shall nevertheless be empathized that RED could theoretically be used in conjunction with other DNN compression techniques such as existing data-free quantization techniques. Furthermore, sparse pruning schemes, e.g. magnitude-base based pruning methods could be considered further reduce the computational runtime, given appropriate hardware. Last but not least, RED could quite straightfor wardly be applied to any existing off-the-shelf computer vision model where runtime optimization is a concern, such as, for instance, object detection or semantic segmentation architectures.
References
Ayinde, B. O. et al. Redundant feature pruning for accelerated inference in deep neural networks. Neural Networks, 11::148-158,
2019. 2019.
Chen, L.-C. et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. TPAMI, pp. 834-848, 2017.
Cheng. Y., Wang. D.. al. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09250 2017.
Deng, J., Dong, W., et al. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.
Evei, U., Gale, T., Menick, J., Castro, P. S., and Elsen, F. Rigging the lottery: Making all tickets winners. ICMI., 2020.
pp.
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, 2018.
Gale, T., Elsen, E., and Hooker, S. The state of sparsity in deep neural networks. arXiv preprint arXiv: 190.009544 2019.
Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. AISTATS, pp. 249-256, 2010.
Han, S. et al. Eie: efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, pp. 243-254, 2016.
He, K., Zhang, X., al. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. ICCV, pp. 1026-1033 2015.
He, K., Zhang, X., et al. Deep residual learning for image recognition. CVPR, pp. 770-778,001 2016.
He, K., Gkioxari, G., et al. Mask r-cnn. ICCV, pp. 2961-2969, 2017.
He, Y., Kang, G., et al. Soft filter pruning for accelcrating deep convolutional neural networks. IJCAI, pp. 2234-2240, 2018
Howard, A. et al. Searching for mobilenetv3. ICCV, pp. 1314-1324, 2019.
Huang, Z. and Wang, N. Data-driven sparse structure selection for deep neural networks. ECCV, pp. 304-320, 2018.
Kim, W., Kim, S., et al. Neuron merging: Compensating for pruned neurons. NeurIPS, 2020.
Krizhevsky, A., Hinton, G., et Learning multiple layers features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Lee, N., Ajanthan, T., et al. signal propagation perspective for pruning neural networks at initialization. ICLR. 2020.
Li, H. et al. Pruning filters efficient convnets. ICLR, 2017.
Liebenwein, L.,, Baykal, C., et al. Provable filter pruning for eflicient neural networks. ICLR, 2020.
Lin, M., Ji. R., al. Hrank: Filter pruning using high-rank feature map. CVPR, pp. 1529 1538, 2020a.
10

---

Lin. S.. Ji. R., et al. Towards optimal structured can pruning via generative adversarial learning. CVPR, pp. 2790-2799, 2019.
Lin, T., Stich, S. U., et al. Dynamic model pruning with feedback. ICLR, 2020h.
Liu, Z., Sun, M., et al. Rethinking the value of network pruning. ICLR, 2018.
Llayd, S. Least squares quantization pem. Trans. Inf. Theory, 1892)7129-137, 1982.
Luo, J.-.., Wu, J., and Lin, w. Thinet: filter level pruning method for deep neural network compression. 2017.
ICCV, pp.
Ma, Lu. An equivalence fully connected layer convolutional layer. arXiv preprint 17120022/2 2017.
Meller, E., Finkelstein. A., Almog, U., Grobman, Same, same but different: Recovering neural network quantization error through weight factorization. ICML, pp. 4486 4495, 2019.
Meng, F., Cheng, H., Pruning filter filter. NeurIPS, 2020.
Nagel, M., Baalen, M. et al. Data-free quantization through weight equalization and bias correction. ICCV, pp. 1325- 1334, 2019.
Park, S., Lee, J., et al. Lookahead: far-sighted alternative of magnitude-based pruning. ICLR, 2020.
Renda, A., Frankle, J., and Carbin, M. Comparing rewinding and fine-tuning in neural network pruning. ICLR, 2020.
Sandler, M., Howard, A., al. Mobilenetv2: Inverted residuals and linear bottlenecks. CVPR, pp. 4510-4520, 2018.
Srinivas, S. and Babu, R. V. Data-free parameter pruning for deep neural networks. BMVC, 2015.
Srivastava, N., Hinton, G., et al. Dropout: a simple way to prevent neural networks from overfitting. JMLR, pp- 1929-1958, 2014.
Stock, P. et al. And the bit goes down: Revisiting the quantization of neural networks. ICLR, 2020.
Tanaka, H., Kunin, D., et al. Pruning neural networks without any data by iteratively conserving synaptic flow.
NewrlPS, 2020.
Tang, Y., Wang. Y., al. Scop: Scientific control for reliable neural network pruming. NewrIPS, 2020.
Wang. K., Liu, Z., et Haq: Hardware-aware automated quantization with mixed precision. CVPR, pp-
8612-8620, 2019.
Yin, H., Molchanov, P., al. Dreaming to distill: Data-free knowledge transfer via deepinversion. CVPR, pp-
8715 8724, 2020.
Zagoruyko, and Komodakis, Wide residual networks. BMVC, pp. 87.1-87.12, 2016.
Zhao, R., Hu, Y., Dotzel, J., De Sa, C., and Zhang. Z. Improving neural network quantization without retraining using outlier channel splitting. ICML, pp. 7543-7552, 2019.
Zhuang. T., Zhang, Z., al. Neuron-level structured pruning using polarization regularizer. NewrIPS. 2020.


Reviews:
[{"review_id": "Z4o4RUOqWyv", "reviewer": "Reviewer_xhzB", "summary": "This paper introduce a data-free compression methods by combing three strategies: adaptive scalar hashing, merging similar neurons and uneven depth-wise separation method to prune conv layers, obtaining SOTA data-free compression performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Novelty is pretty good. The observations of redundancies in DNN are pretty novel. \n2. The comparisons conducted on Imagenet are not very strong. There are many published data-driven compression papers. The listed paper's performances are not real SOTA. However, this method is data-free, I will buy it.  It's better to show more comparisons on other benchmark models (for example, MobilenetV2) on Imagenet. \n3. As mentioned in paper, this method can incorporate with other compression methods together for further compressions. \n\nI am curious about:  What if we apply three strategies in loop iteratively? Can it improve the performance?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please provide Societal Impact discussion.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "NNDxp2mykvt", "reviewer": "Reviewer_tP4V", "summary": "The author propose a data free pruning pipeline to compress a given deep neural network. The author propose to first apply adaptive weight hashing to force the weights tensors to be similar and then merge weight tensors based on a calculated similarity score. The proposed methods use kernel density function for hashing and do not require training data for hashing the weights tensors. The proposed pipeline (RED) is verified on CIFAR-10 and ImageNet.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n1. The proposed pipeline (Hashing + Merging) is interesting and the author has demonstrated the effectiveness of hashing on introducing redundancies on the weights tensor.\n\n2. Data-free compression is also worth to explore for its efficiency on computation resources.\n\nWeakness\n1. The writing of the paper need further refinement. For an example, table 1 in the paper seems to be incomplete as there is a missing of the column headers. I cannot figure out the meaning of those results based on the current version.\n\n2. The impact of hashing on the model accuracy is not clear. While the hashing introduce tensor level redundancies, will it hurts the model representation capability? \n\n3. How is the proposed methods affected to different initialization methods? This is mentioned in the caption of Figure 5 but the impacts are not clear as there is not experiment results on this point. I thought this is worth to explore.\n\n4. The proposed method is mainly verified on large models like ResNet. It is not clear if the proposed method will work on more efficient backbones such as MobileNet [1], EfficientNet [2] and MobileNeXt [3].\n\nOverall, the proposed method is interesting to explore. However, the current presentation and ablation experiments do not convince me on the effectiveness of the proposed RED pipeline. Besides, the difference between the proposed hashing methods and the quantization algorithm (as there are a lot of well performing quantization algorithms) is also worth to explore.\n\nI am willing to incease my score if the author could address mu concerns sufficiently.\n\n[1] Sandler, Mark, et al. \"Mobilenetv2: Inverted residuals and linear bottlenecks.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[2] Tan, Mingxing, and Quoc Le. \"Efficientnet: Rethinking model scaling for convolutional neural networks.\" International Conference on Machine Learning. PMLR, 2019.\n\n[3] Zhou, Daquan, et al. \"Rethinking bottleneck structure for efficient mobile network design.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer International Publishing, 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author do not provide social impact of the paper", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "9u2VHrYHaJe", "reviewer": "Reviewer_HdEN", "summary": "### Summary\nThis paper proposes a three-step compression of neural networks without fine-tuning, which comprises weight hashing, merging of similar weights, and depthwise separation via basis extraction. The compression is filter-based, data-free, and with minimal accuracy degradation.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\nThe combined approach appears to be effective and competitive against related works which includes some recent fine-grained and data-driven methods.\n\nThe reviewer believes that the method proposed is interesting with the use of density estimation for redundancy discovering, despite some presentations issues (as discussed below).\n\nThe experiments and ablation analyses appear to be sufficient to justify the claims in the paper.\n\n## Weaknesses\n* It is not immediately clear why hashing weights helps with identifying redundancy. Why not simply merge neurons with small $\\ell^2$ distances? Perhaps providing a formal definition of how $\\tau$ is used would clear up the confusion.\n* It is also not clear how the KDE bandwidth $\\Delta_l$ is determined. The hashing process is highly dependent on this.\n* The proposed hashing method may have a scalability problem (“ranging from 45 to 413 seconds for ResNets on CIFAR-10 and up to 21 hours for a ResNet-50 on ImageNet”, line 202). It appears to not extrapolate well to larger models. What about ResNet-$N$ ($N \\geq 101$)? Could you explain why this is the case or provide a discussion on its complexity?\n* Minor issues:\n\t* Line 112 discreet -> discrete\n\t* Line 123 defines the contrast hyperparameter $\\tau$, but provides no definition of how it is used.\n\t* Line 109: grammar: “for a each layer”.\n* The combined method appear to be overall too complex, the reviewer would appreciate if the work will be open-source in the future.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Sufficiently discussed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I agree with the two of the reviewers that this contribution is novel and interesting. While the empirical evidence presented here could (and was) improved (in the rebuttals), I think that a sufficiently revised version of the paper (as suggested in the rebuttals) can be accepted for a poster at NeuriPS.
Decision: Accept (Poster)