Pruning with Output Error Minimization for Producing Efficient Neural Networks
Authors: Koji Kamma
Toshikazu Wada
Venue: Submitted to ICLR 2023 (2023)
Keywords: pruning
weighted least squares method
convolutional neural networks
compression

Abstract: DNNs are known to have excessive parameters and thus are computationally expensive, which poses a challenge for implementations in various applications. Structured pruning is a technique of compressing a trained DNN model by removing redundant neurons (or channels). How well a pruned model maintains its accuracy depends on two factors. The first is compression ratio optimization, in other words, how many neurons are reduced in each layer. The other is layer-wise optimization, in other words, which neurons to be preserved in each layer. In this paper, we propose Pruning with Output Error Minimization (POEM), a layer-wise pruning method that conducts pruning and then performs reconstruction to compensate the error caused by pruning. The strength of POEM lies in its reconstruction using the weighted least squares method so as to minimize the output error of the activation function, while the previous methods minimize the error of the value before applying the activation function. The experiments with well-known DNN models and a large scale image recognition dataset show that POEM is better than the previous methods in maintaining the accuracy of those models.

Under review as a conference paper at ICLR 2023
PRUNING WITH OUTPUT ERROR MINIMIZATION FOR PRODUCING EFFICIENT NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep Neural Networks (DNNs) are dominant in the field of machine learning. However, because DNN models have large computational complexity, implemen- tation with resource- limitedd equipment is challenging. Therefore, techniques of compressing DNN models without degrading their accuracy is desired. Pruning is such technique to remove redundant neurons (or channels). In this paper, we present Pruning with Output Error Minimization (POEM), a method that performs not only pruning but also reconstruction to compensate the error caused by prun- ing. The strength of POEM lies in its reconstruction to minimize the output error of the activation function, while the previous methods minimize the error of the value before applying the activation function. The experiments with well-known DNN models (VGG-16, ResNet-18, MobileNet) and image recognition datasets (ImageNet, CUB-200-2011 were conducted. The results show that POEM sig- nificantly outperformed the previous methods maintaining the accuracy of the compressed models.
INTRODUCTION
Nowadays, Deep Neural Networks (DNNs) are dominant in the field of machine learning. The demand for DNNs is increasing in various applications. However, DNNs are known to be over- parameterized and require large computational cost. This makes them computationally slow, power- consuming, and difficult to be implemented in resource-limited equipment.
Therefore, there is a need for the techniques to create efficient DNN models by compressing large models while maintaining the accuracy. Pruning is one such technique remove redundant weights from trained DNN models. Pruning methods can be divided into two groups: unstructured pruning and structured pruning. The former removes weight parameters in order to make the weight tensor sparse. Since the shape of the weight tensor remains the same, the compressed model should be implemented using hardware and libraries that can perform calculations only on non-zero weights. The latter removes neurons (or channels) in order to make the shape of the weight tensor smaller. Therefore, the effect of compression can be achieved by implementing the compressed model using general hardware and libraries. In this paper, we focus on structured pruning.
How well a pruned model maintains its accuracy depends on two factors. The first is compression ratio optimization, in other words, how many neurons are reduced in each layer. The other is layer- wise optimization, in other words, which neurons to be preserved each layer.
In recent years, there is a growing awareness that the value of pruning lies in the search for an efficient sub-architecture out of large redundant architecture. This is due to the research results showing that DNN model with the pruned architecture trained from scratch can achieve at least as good accuracy as the pruned and fine-tuned model (Liu et al. 2019). For this reason, the recent trend is to focus compression ratio optimization problem.
However, does it mean that the layer-wise optimization is no more important? It is reasonable to claim that combining compression ratio optimization method with better layer-wise optimization method should result in more effective pruning. Therefore, layer-wise optimization still important and worth investigating.
In this paper, we propose pruning method named Pruning with Output Error Minimization (POEM) that performs layer-wise optimization. The strength of POEM lies in its reconstruction using the

---

Under review as a conference paper at ICLR 2023
Weighted Least Squares (WLS) method to minimize the output error of the activation function, while the previous methods (Luo et al. 2017 He et al., 2017: Dong et al. 2017: Kamma & Wada 202I) minimize the eπror of the value before applying the activation function. For example, since the ReI.U function rounds a negative value to zero, the error on a negative element need not be compensated (unless it turns positive due to the error). POEM can perform reconstruction only for the positive elements, while the previous methods perform reconstruction for all elements includ- ing negative ones. For this reason, POEM is superior to the previous methods in maintaining the accuracy of the pruned model. To the best of our knowledge, POEM is the first method to perform reconstruction based on the output error of the activation function.
For verifying POEM, we conducted experiments on ImageNet (Deng et al., 2009). large-scale image classification dataset, and well-known DNN models for image classification, such as VGG- 16 (Simonyan & Zisserman, 2015), ResNet-18 (He et al. 2016), and MobileNet (Howard et al.) 2017]. The results show that POEM can prevent the output error of the activation function better than the previous methods (He et al. 2017 Kamma & Wada 2021), and improve the accuracy both before and after fine-tuning. We also confirmed that the accuracy of the compressed model can be further improved by combining POEM and the compression ratio optimization methods (He et al., 2018b Kamma et al. [022] Li et al. (202).
The rest of this paper is structured as follows. In Sec. we introduce related works. In Sec. explain our proposed method. In Sec. we show experimental results to verify the effectiveness of POEM. In Sec. [5] we conclude discussions in this paper.
2 RELATED WORKS
DNN compression methods can be divided into four groups: structured pruning, unstructured prun- ing (or sparsification), tensor decomposition, and quantization. Structured pruning is to make the shape of the weight tensor smaller by removing redundant neurons (or channels) (Molchanov et al.) 2017 He a 201 8a 2017 Luo al.) 2017 Kamma & Wada [021] Jiang et al.) 2018). The ad- vantage of these methods is that the effect of compression can be obtained without special hardware or libraries. Unstructured pruning is to make the weight tensor sparse without changing its shape
(LeCun et al. 1990 Liu et al., [013] Han et al.] 2016 Lee et al.] 2019). The effect of unstructured pruning can be obtained by implementing the compressed model using hardware or libraries that can perform computation only for the non-zero elements of the weight tensor. The methods based on tensor decomposition replace a large weight tensor by the product of multiple smaller weight tensors (Xue et al.) 2013 Kim et al. 2019: Denton et al.) 2014). These methods effectively reduce the number of parameters and the computational complexity, although the compressed model gets extra layers incurring
additional computational overhead. Quantization is reduce the memory and complexity requirements of a model by discretizing the weights (Courbariaux et al. 20155] Liu et al.,
2022 Li et al. |021 Wei et al. 2022). A quantized model needs be implemented on low-bit computation equipment.
In this paper, we focus on structured pruning because of the following benefits: the structured prun- ing methods can compress the model without incurring computational overhead; the compressed model can be implemented without special hardware or libraries.
For developing an effective pruning method, 2 problems should be addressed. One is the problem of compression ratio optimization, and the other is layer-wise optimization. Some pruning methods address both of these problems in a single framework, while others handle each problem separately.
The compression ratio optimization is to configure the number of pruned neurons in each layer. AutoMI. Model Compression (AMC) uses reinforcement learning to tune compression ratios so that the accuracy of the model maximized posing constraint on FL.OPs (the number of floating point multiplications), or the FLOPs are minimized posing a constraint on accuracy (He et al.) 2018b). Pruning Ratio Optimizer (PRO) tunes compression ratios by alternately performing layer selection and pruning based the output error of the final layer (Kamma al.] 2022). RandomPruning performs random search in the search space of compression ratios (Li et al. 2022). These methods can be combined with any layer- wise optimization methods.
The layer-wise optimization is to select which neurons to be preserved in each layer. A lot of neuron selection criteria have been investigated, such as the ones based the norm of outgoing
2

---

Under review as a conference paper at ICLR 2023
weights (He al. 2014), the derivative information of the loss function (Molchanov et al. 2017
the geometric median of incoming weights (He et al. 2019). the output difference final layer of It known that performing reconstruction to tune the remaining weights is effective for compensating the error caused by pruning. Channel Pruning (CP) (He al. 2017). ThiNet (Luo al.
the model (Yu et al. 2018 Luo & Wu 2020), and
2017), and Reconstruction Error Aware Pruning (REAP) (Kamma& & Wada (021) use the Least Squares (LS) method for reconstruction. Layer-wise Optimal Brain Surgeon (LWOBS) (Dong al. 2017) performs reconstruction based on the second order derivative information of the layer-wise Mean Squared Error (MSE). With these methods, high compression ratio can be achieved even without fine-tuning.
The compression ratio optimization and the layer-wise optimization can be performed simultane- ously in a single pruning framework. For example, the gradient information of loss function used to compare the importance of neurons in the whole model (Molchanov et al. 2017). Some methods perform fine-tuning with regularization terms to achieve compression (Liu al., [217]]
Ding et al. 2019 Guo et al. 202Ib. Although these methods have advantage in maintaining accuracy while performing pruning, the pruning process itself is time-consuming.
The proposed method in this paper, POEM, is to perform layer-wise optimization. The strength of POEM lies in its strategy for neuron selection and reconstruction based on the output error of the activation function (rather than the error before applying the activation function). It should be noted that Jiang et al. (2018) proposed a pruning method that performs reconstruction based on the activation function outputs, however, their neuron selection is based on heuristic criteria, whereas we purposively select the neurons to minimize the output error based on sound formalization.
3 PRUNING WITH OUTPUT ERROR MINIMIZATION
In this section, we introduce the proposed method POEM. POEM is characterized by its neuron (channel) selection and reconstruction strategies based the output error of the activation function. We define notations in Sec. 3.I, explain the reconstruction strategy of the previous methods (Luo et al. 2017: He et al. 2017 Kamma & Wada (02T) and that of POEM in Sec. 3.2 our neuron selec-
tion strategy in Sec. [.3 and the algorithm for efficient reconstruction in Sec. 3.4 Although POEM
can be applied to both fully connected layers and convolutional layers, we make explanation
using a fully connected layer for simplicity.
3.1 PREPARATIONS
Let n be the number of input samples (e.g. images), and c be the numbers of neurons in the current and the following layers, X E IRaxa be the outputs in the current layer, WALL E Raxe be the weight matrix, f be the activation function (e.g. ReLUU,, and f' be its first derivative. The outputs in the following layer can be obtained by calculating f(Y), where Y ALLL By applying pruning, we remove the columns of X ALL and the rows of ALLL corresponding to the pruned neurons, and get X E R and W R6be, where denotes the number of preserved neurons.
For matrix M and index i, M. and M,, denote the i-th column and the i-th row of M, M. denotes a matrix composed of all but the i-th columns of M, and M-j,; denotes a matrix composed of all but the i-th rows of M. For a set of indices A, M. and Ms,: denote matrices that have the columns and the rows of M corresponding to A. For a vector v, e; denotes its i-th element and Us. denotes its elements corresponding A.
We also define the operators: + denotes the assignment, \ denotes the set difference, o denotes Hadamard product, and || || denotes Frobenius norm. For vector v. diag(v) denotes diagonal matrix having the elements of on the diagonal entries. For matrix M, M2 M o M.
3.2 OUTPUT ERROR MINIMIZATION WITH THE WEIGHTED LEAST SQUARES METHOO
The previous pruning methods perform reconstruction so as to minimize the error of Y. This can be formalized as typical LS problem:
W arg min Y - X 2
(1)
W
3

---

Under review as a conference paper at ICLR 2023
f(y)
f6)
f
f6) f 8y)
Figure 1: Illustrations of activation functions (left: ReLU, right: Sigmoid). In flat zones of these functions, the errors are suppressed. Therefore, is more important to perform fitting to the elements in nontfllt zones.
However, this does not minimize the output error of the activation function. For example, since ReLU function rounds a negative value to zero, the error of a negative element Y is canceled by ReLU, unless that element turns positive due the error. Therefore, it is better perform reconstruction so as to minimize the error only for the positive elements of Y. To generalize, as illustrated Figure [ the eтror of an element suppressed if that element is in a flat zone of the activation function. On the other hand, the error of an element in a non-flat zone is not suppressed. For minimizing the output errоr of the activation function, is more important reduce the error for the elements in non-flat zones.
This idea can be formalized as WLS problem. Our reconstruction strategy is minimize the output error of the activation function:
W arg min (I((() ((X))
w
Because it difficult to solve Eq. 2 directly, we perform relaxation. By performing Taylor expan- sion for each element of f(Y). we get
f(Y +OY) f(Y) (Y) oY + gYY,8Y)
where g(Y, ,Y) denotes the second and the higher order terms. The reconstruction error for Y can be written as XW- Y. We substitute öY XW and omit g:
f(XW) ~ f(Y) )- (Y) (Y - XW).
(4)
By using this approximation, Eq. ( can be replaced by a WLS problem:
W arg min (Y) (Y XW)
(5)
w
For each element of Y, the emor is weighted by the corresponding element of f'(Y). Therefore, the error is suppressed for the elements in flat zones of f.
3.3 NEURON SELECTION CRITERIA
When we conduct DNN compression with POEM, we first select and prune the neurons, and then perform reconstruction. We already explained how to perform reconstruction in Sec. 3.2 In this subsection, we explain how to select the neurons to be pruned.
Let A fI.... a} be subset of column indices of ALL corresponding the preserved neurons.
Since we perform reconstruction by solving Eq. ( we should optimize A so as to minimize the reconstruction error:
A
arg min min
['(Y) (Y WALL
X WALLW!L
subject to 1
2
(6)
where p denotes the target compression ratio. Since Eq. 66 a difficult problem, we solve it in greedy fashion. Assume that have already pruned several neurons and got X out of XLL Then, we should select the neuron to be pruned next. This is, other words, to select the k-th column of X so to minimize the error:
k. = arg min
min o W_k.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Memory-efficient algorithm for a WLS problem
Input: X, Y, f', number of sampled columns s, number of iterations M
Output: W
1: Initialize the weight: W < arg minw Y XW
2: for m 1...... M do
3: Sample Thrh (the column subset X) so that
T0
S holds
4: for = 1,... e do
5: Perform regression: w² < arg min (V)()))) o ((Y,) XW a. w)
6: Update the weight: WBm Wmm, w
7: end for
8: end for
In order to solve Eq. we should solve the following problem for all k.
k, = arg min (Y) (Y - -
(
W-E,:
Since solving Eq. (8 for all & time-consuming, we once omit fYYY) in order to apply an efficient
algorithm (Kamma & Wada (021). Then, we have
W k.s arg min - - W.e.
(9)
Now, we can solve Eq. by using W k
k = arg min ( (Y - X - 1I.
(10)
We prune the - neuron, and continue pruning with the remaining neurons until the target com- pression ratio is achieved.
3.4 AN EFFICIENT ALGORITIM FOR THE WEIGIITED LEAST SQUARES PROBLEM
In this subsection, we show how to implement the solution for a WLS problem. We show an algo- rithm to solve it efficiently, while explain its details in Appendix B.3
For LS problem Eq. T we have an one-shot solution:
W T X
(11)
On the other hand, we do not have such an one-shot solution for a WLS problem. In order solve Eq. (5) regression should be performed for each column of Y separately:
W
arg min || (Y.,) (Y. XW. wll
(12)
W.,
Let G = -digg(f(XX.))) The solution of Eq.
12 is given by
W = (X ( x) G
(13)
We parallelize this for each j to obtain the solution efficiently. Although, implementing Eq. is would require a large memory space for computing (X GO² xx)
13 as
Algorithm describes the solution for Eq. (12) with limited memory space. This algorithm is to approach the solution asymptotically by updating some of the elements of W in turn. To do this, we sample some columns from X and use them for regression. See Appendix B.3 for more details.
3.5 LIMITATION
POEM is supposed to be used for layers with fixed weights, i.c. convolutional layers and fully connected layers. Therefore, layers with dynamic weights, such as (Ramachandran et al. 2019), are not supported.
5

---

Under review as a conference paper at ICLR 2023
Benchmarking channel selection methods and reconstruction methods. We pruned 32 channels out of 64 channels in "Convl-1" layer of VGG-16, and compared the MSEs in the follow- ing layer. Each cell contains the MSE of the outputs of the activation function (after f) or of the values before applying the activation function (before f).
Table 1:
No rec.
b) LS
c) WLS (ours)
Select. Select.\Rec.
after f
before f
after f
before f
after f
before f
LI
0.2731
0.8614
0.0379
0.0953
0.0244
0.1750
2) L2
0.2183
0.7685
0.0141
0.0341
0.0116
0.0660
3) Lasso
0.1711
0.5537
0.0117
0.0251
0.0095
0.0460
4) GM
0.2749
0.9114
0.0550
0.1443
0.0432
0.4143
5) Error (LS)
0,2110
0.6582
0.0067
0.0143
0,0060
0.0215
6) Error (WLS)
0.2277
0.7226
0,0062
0,0136
0.0055
0.0200
4 EXPERIMENTS
In this section, we show the experiment setups and results. We conducted experiments with VGG- 16 (Simonyan & Zisserman, 2015). ResNet-18 (He et al.] (2166) and MobileNet (Howard et al.) 2017). We mainly used ImageNet (Deng et al., 2009) dataset for evaluation, while a fine-grained classification dataset CUB-200-201 (Wah al.
2011 was also used.
4.1 DATASET AND AUGMENTATION
ImageNet dataset contains 1. 1.28M training images and 50K validation images (Deng et al.) 2009). Since POEM is data-dependent method, we randomly sampled 5K training images for pruning. For all images, we resized as make the shorter side be 256. For training images, we applied 224 x 224 random crop and random horizontal flip. For validation images, we applied 224 224 center crop. CUB-200-2011 dataset contains 6K training images and 5.7K test images. For pruning with POEM, we used all training images. The augmentation settings were the same with those for ImageNet dataset.
4.2 PRUNING SETUPS
We performed pruning in convolutional layers of VGG-16, ResNet- 18, and MobileNet. Since POEM performs only layer-wise optimization, the compression ratio optimization should be performed ex- ternally. We conducted experiments two scenarios: with and without compression ratio optimiza- tion. For configurating compression ratios, we used three methods, AutoML Model Compression (AMC) (He al. 2018b), Pruning Ratio Optimizer (PRO) (Kamma al. 2022), and RandomPrun- ing (Li et al. 2022). For AMC, we used the official source code published by the authors of (He) al. 2018b). We implemented the other 2 methods on our own.
4.3
TRAINING SETUPS
In the experiments with ImageNet dataset, fine-tuned VGG-16 and ResNet-18 for 25 epochs, and MobileNet for 150 epochs after pruning, with Stochastic Gradient Descent (SGD) optimizer. For VGG-16 16, the mini-batch size was set 128, the learning rate was set to 0.0005, and the weight decay was set to 0.0001. For ResNet-18 18, the mini-batch size was set to 256, the learning rate was set to 0.001, and the weight decay was set to 0.0001. For MobileNet, the mini-batch size was set to 256, the learning rate was set to 0,0005, and the weight decay was set to 0,00004. For only MobileNet, we applied cosine annealing. The momentum was set to 0.9 for all models.
In the experiments with CUB-200-201 dataset, we first applied domain adaptation to the pre-trained ResNet- 18 model. We initialized the fully connected layer of the ResNet- 18 model and conducted training for 100 epochs with SGD optimizer. The learning rate set to 0.01, and divide by 10 at 50 epoch. The mani-batch size was set to 64, the weight decay was set to 0.001, and the momentum was set to 0.9. After pruning, fine-tuned the pruned model using the same settings.

---

Under review as a conference paper at ICLR 2023
Table 2: The results of pruning with uniform compression ratios. The baseline accuracy of VGG-
16 was 0.70272 (top-1) and 0.8946 (top-5) and the accuracy of ResNet-18 was 0.69758 (top-1) 1) and 0.89076 (top-5).
Acc. w/o fine-tune
Acc. fine-tuned
Model
Method
FLOPs
Top-1
Top-5
Top-1
Top-5
CP-A
-75.8%
17.33
35.042
65.754
86.868
REAP-A
-75.8%
27.926
50.688
66.642
87.466
POEM-A
-75.8%
51.492
77.472
67.268
87.68
VGG-16
CP-B
-75.7%
39.254
70.212
67.938
88. 182
REAP-B
-75.7%
52.368
77.452
68.844
88.716
POEM-B
-75.7%
59.412
82.446
69.044
88.892
CP
-50.4%
27.136
52.894
62.0
84.182
ResNet-18
REAP
-50.4%
32.186
58.326
62.59
84.694
POEM
-50.4%
42.376
69.546
62,812
84.8
4.4 LAYER-WISS ANALYSIS
We performed pruning "Conv1-1" layer of 16,-16, and calculated the MSEs of the outputs of the activation function and the MSEs of the values before applying the activation function. Note that we can decouple and combine the channel selection methods and the reconstruction methods. For example, we could select channels based on LI norm of the output weights (He et al. 2014) and perform POEM's reconstruction. We used various combinations of channel selection methods and reconstruction methods. For channel selection, we used the ones based on 1) LI norm of outgoing weights (He al. 2014), , 2) L2 norm of outgoing weights. 3) Lasso regression (He et al.) 2017), 4) Geometric Median (GM) of incoming weights (He et al. 2019). 5) LS error (Kamma & Wada 2021 and 6) WLS error (ours). For reconstruction, used a) reconstruction, b) LS (He al.
2017
Kamma & Wada, 2021), and c) WLS (ours).
Table shows the results. Comparing b) and c), see that our reconstruction method better than the previous method in preventing the output error of the activation function. Comparing the row of 6) and the other rows, we can see that our channel selection method also outperformed the other channel selection methods except for the case of not performing reconstruction. These results show the superiority of POEM in both channel selection and reconstruction.
An interesting observation is that the output error of the activation function was the lowest when we used our WLS-based reconstruction, while we suffered relatively large errors of the values before applying the activation function. By performing WLS, we can reduce the errors of the output ele- ments in non-flat zones of the activation function, which increases the errors of the elements in flat zones. However, as already mentioned, the error in a flat zone is suppressed and thus is not harmful.
worth noting that the error can be significantly reduced even with the previous reconstruction method, compared to the case of not performing reconstruction. This indicates that reconstruction crucial for pruning. No matter which channel selection method is chosen, we should perform reconstruction.
4.5 PRUNING WITH FIXED COMPRESSION RATIOS
We conducted experiments with V6G-16 and ResNet-18. We performed pruning to compress their convolutional layers with uniform compression ratios, and then conducted fine-tuning. In this sec- tion, we compare POEM with two previous methods, CP (He et al.) (017) and REAP (Kamma & Wada 2021). Both CP and REAP are the pruning methods performing reconstruction but based on the error before applying the activation function. For VGG-16, it known that the last two convo- lutional layers not redundant, and these layers are often excluded from the compression targets (Luo et al. 2017 He et al. 2017). We performed compression in two cases. One is to compress all convolutional layers (Case A), and the other is to compress all but those two layers (Case B).
The results are shown in Table 2. We can see that POEM outperformed the other methods con- sistently. The performance difference significant especially before fine-tuning. For example,
7

---

Under review as a conference paper at ICLR 2023
Table The results of pruning using AMC. The baseline accuracy of MobileNet 0.71144 (top-
1) and 0.89842 (top-S). The baseline accuracy of VGG-16 was 0.70272 (top-1) and 0.8946 (top-5)
and that of ResNet-18 was 0.69758 (top-1) and 0.89076 (top-5).
Acc. w/o fine-tune
Acc. fine-tuned
Comp. ratio
Model
Method
FLOPs
Top-1
Top-5
Top-1
Top-5
LI&LS
-50.0%
43.686
69.13
70.382
89.398
POEM
-0000%
51.984
76.648
70.488
89.492
AMC
MobileNet
LI&LS
-70.0%
3.146
11.12
67.57
87.772
POEM
-70.0%
17.476
38.74
68.104
87.992
LI&LS
-75.7%
28.356
51.396
67.134
87.804
VGG-16
REAP
-75.4%
57.196
80.846
69.09
88.95
POEM
-75.7%
61.956
84.18
69.484
89.02
PRO
LI&LS
-50.3%
42.86
70.372
64.544
86.124
ResNet-18
REAP
-0.44%
47.648
73.964
64.874
86.33
POEM
-50.0%
52.306
77.428
65.316
86.218
LI&LS
-75.7%
67,438
87.854
VGG-16
REAP
-75.6%
68.78
88.614
POEM
-74.6%
69.296
88.92
RandomPruning
LT&LS
-50.5%
63.606
85.386
ResNet-18
REAP -50.7%
63.738
85,43
POEM
-49.2%
64.286
85.686
in Case of VGG-16, POEM outperformed REAP by more than 23% margin in top-1 accuracy. Although fine-tuning reduced the accuracy gaps, the model compressed with POEM was still better than the ones compressed with the previous methods.
For VGG-16, the performance gap of POEM and the other methods was more significant in Case A than in Case B. In Case A, we performed pruning on non-redundant layers, while those layers were excluded from compression targets in Case This indicates the relative advantage of POEM is greater when we conduct compression for less redundant layers.
4.6 PRUNING WITH COMPRESSION RATIO OPTIMIZATION METHODS
We evaluated performance of POEM combined with three compression ratio optimization methods, AMC (He al. 2018b), PRO (Kamma al. 2022), and RandomPruning et 2022). For AMC, LI norm-based channel selection and LS-based reconstruction are used as default. We refer this default setting as "LI&LS". We evaluated the pruning performance with LI&LS and with POEM. For PRO and RandomPruning we used REAP as well.
Table 3 shows the results. With AMC, POEM outperformed LI&LS by 14.3% in top-1 and 27.6% in top-5 accuracy at 70% FLOPs reduction. After fine-tuning, POEM still outperforms LI&LS by 0.5% and 0.2% in top-1 and top-5 accuracy.
With PRO, POEM outperformed the other two methods by considerable margin before fine-tuning. After fine-tuning, although REAP was better than POEM marginally only in top-5 accuracy of ResNet- 18, POEM was the best for the rest.
For RandomPruning, we show only the accuracy after fine-tuning, since fine-tuning is already con- ducted in the process of compression ratio optimization. For both VGG-16 and ResNet-18, POEM outperformed the second best method REAP by the margin of 0.5% and 0.2% in top-1 and top-5 accuracy.
The difference of POEM and LI&LS for MobileNet was larger at higher compression ratio. After significant amount of fine-tuning (150 epochs), we still see clear accuracy gap. This indicates that the performance superiority of POEM becomes more significant at higher compression ratio.
Another remarkable observation is that the pruning performance and PRO. The result for VGG-16 was worse than even some results with the fixed compression ratios. This is most likely because PRO is type of method that narrows down the search space
poor when used LI&LS
8

---

Under review as a conference paper at ICLR 2023
Table 4: The results with CUB-220-2011 dataset. The baseline accuracy of ResNet-18 was 0.7571
(top-1) and 0.9283 (top-5).
Acc. w/o fine-tune
Acc. fine-tuned
Model
Method
FLOPs
Top-1
Top-5
Top-1
Top-5
CP
-50.4%
62.35
85.57
70.84
90.24
REAP
-50.4%
64.70
86.71
71.91
90.54
POEM
-50.4%
69.58
89.86
72.33
90.81
Scratch
-50.4%
62.94
84.84
ResNet- 18
CP -75.3%
20.27
43.66
65.82
88.09
REAP
-75.3%
21.67
45.65
66.79
88.64
POEM
-75.3%
47.91
76.38
68.08
89.21
Scratch
-75.3%
58.14
83.12
of compression ratios sequentially. Once the model accuracy severely damaged, it is no more possible to find good solution. On the other hand, we could find much better solutions for both VGG-16 and ResNet-18 when we used POEM and PRO. Thus, combining a compression ratio optimization method with a better layer-wise optimization method results in more effective pruning.
4.7 EXPERIMENTS WITH A FINE-GRAINED CLASSIFCATION DATASET
We conducted experiments with ResNet-18 18 model and CUB-200-200 200-20001 dataset. We first fine-tuned the pre-trained model using CUB-200-201 dataset, then performed pruning with the uniform com- pression ratios in all convolutional layers. We also evaluated the performance of the models with the pruned architectures but trained from scratch. For the trraieed-from-scrate models, we conducted training for 400 epochs. The learning rate was set 0.1, and was divided by 10 200 and 300 epochs.
The results are shown in Table. 4 We can see that POEM was better than the other methods consistently. At 75% FLOPs reduction ratio, POEM outperformed REAP by 26.2% and by 30.7% in top-1 and top-5 accuracy, respectively. After fine-tuning, POEM still outperformed REAP by approximately 1.2% and 0.5% in top-1 and top-5 accuracy.
worth noting that the performance of trained-from-scratte models were much worse than the pruned and fine-tuned models. In the case of training them from scratch, it is difficult to generalize for such small dataset. These results let us rethink the value pruning. Nowadays, it is believed that the value of pruning lies in the search for efficient architecture rather than the weight parameters obtained by pruming (and fine-tuning), based on the research results in (Liu et al. (219). Although this is true for the large scale datasets, such ImageNet, CIFAR-10 10 (Krizhevsky is not the case with small scale datasets with which training from scratch easily falls into bad local minima. In many real scenarios, the model needs to be trained with limited amount of data due to poor availability of the data, annotation cost, and other reasons. In such cases, is an effective approach use the pruned and fine-tuned models, rather than the trained-frie sereedd-fro----e crom-sccrach models.
al.), and so on,
CONCLUSION
In this paper, we presented Pruning with Output Error Minimization (POEM), the method to conduct pruning and perform reconstruction to minimize the output error of the activation function rather than the error of the value before applying the activation function. In experiments using well-known DNN models for image recognition, VGG-16, ResNet-18, and MobileNet, proposed method can perform compression with smaller errors than the previous methods. We also confirmed that the proposed method, combined with the compression ratio optimization methods, enables more effective compression. In Appendices, we show additional information for experiment reproducibility and implementation.
confirmed that the

---

Under review as a conference paper at ICLR 2023
REFERENCES
M. Courbariaux, Y. Bengio, and J. David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Proceedings of Conference on Neural Information Processing Systems (NIPS), pp. 3123-3131, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fee. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
doi: 10.1 109/CVPR.2009 202.04444.0..8....
Emily I. Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Xin Dong, Shangyu Chen, and Simno Pan. Learning to prune deep neural networks via layer- wise optimal brain surgeon. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Yi Guo, Huan Yuan, Jianchao Tan, Zhangyang Wang. Sen Yang, and Liu. Gdp: Stabilized neural network pruning via gates with differentiable polarization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5239-5250, October 2021.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016.
K. He, X. X.Znang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
T. He, Y. Fan, Y. Qian, T. Tan, and K. Yu. Reshaping deep neural network for fast decoding by node-prumee pruning. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 245-249, 2014.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI'8, pp.
AAAI Press, 20. ISBN 9780999241127.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruming via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In The IEEE International Conference on Computer Vision (ICCV) Oct 2017.
Yihui He, Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference Computer Vision (ECCV), September 2018b.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Effficintt convolutional neural networks for mobile vision applications, 2017. URL htt.t::tt/taaiiiii org/ abss////.. 04861
Chunhui Jiang, Guiying Li, Chao Qian, and Ke Tang. Efficient dnn neuron pruning by minimizing layer-wise nonlinear reconstruction error. In Proceedings of the Twenty-Sevent International Joint Conference Artificial Intelligence. IJCAI-18, pp. 2994-330.. International Joint Con- ferences on Artificial Intelligence Organization, 7 2018. doi: 10.24963/jcaa.20188/333 URL
221.5////aaa.211//3111111 https://doi www./oorgoo......o 249634496332 jiaii 2018/318
10

---

Under review as a conference paper at ICLR 2023
Koji Kamma and Toshikazu Wada. Reap: A method for pruning convolutional neural networks with performance preservation. IEICE Transactions on Information and Systems, E104.D:194-202 01 2021. doi: 10.1587/taassnn2222000000000
Koji Kamma, Sarimu Inoue, and Toshikazu Wada. Pruning ratio optimization with layer-wise prun- ing method for accelerating convolutional neural networks. IEICE Transactions on Information and Systems, 1.1.5:::6119999 01 2022. doi: 10. 2087/traninnf.22111130
Hyeji Kim, Muhammad Umar Karim Khan, and Chong-Min Kyung. Efficient neural network com- pression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- fion (CVPR), June 2019.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 (canadian institute for advanced re- search). BRLhhtp:/// wwww cs www.ooootooo..o.ooooooooo....... edu/-kriz/cifaz .html
Y. LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In Proceedings of Conference on Neural Information Processing Systems (NIPS), pp. 50855598-555 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: SINGLE-SHOT NETWORK
PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference on Learn- ing Representations, 2019. htt.lttttss//peeeeeee/e
net/ formm Fttt:/orrmmm111122a/aaaaa
Yawei Li, Kamil Adamczewski, Wen Li, Shuhang Gu. Radu Timofte, and Lue Van Gool. Revisit- ing random channel pruning for neural network compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 191-201, June 2022.
Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. [BRECQ]: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations, 2021. URL.tttps //tpn///eee/..eeeeeeeeeee net cttt/forummicco wowdddddwwwwdddP0
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolu- tional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2015.
Zechun Liu, Kwang-Ting Cheng, Dong Huang, Eric P. Xing, and Zhiqiang Shen. Nonuniform- to-uniform quantization: Towards accurate quantization via generalized straight-through estima- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4942-4952, June 2022.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In ICCV, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruming. In ICLR, 2019.
Hao Luo and Jianxin Wu. Neural network pruning with residual-connections and limited-data. In The IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1458- 1467, June 2020.
Jian Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In The IEEE International Conference Computer Vision (ICCV), pp. 5068-5076, 10 2017. doi: 10. 1..11099
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Repre- sentations, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner. Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp-

---

Under review as a conference paper at ICLR 2023
8024 8035. Curran Associates, Inc., 2019. URL http:///pppos..aaar//cc///a/// wat.eaaaaaaarrraaa-eee-eeeeaaaaa -iiianiinngibrrrr....mmm ibrary.
pdf
Prajit Ramachandran, Niki Parmar. Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens.
Stand-alone self-attention in vision models. In H. Wallach, Н. Larochelle, A. Beygelzimer,
F. d'Alché-Buu Buc, E. Fox, and Gamett (eds.). Advances Neural Information Processing Sys-
tems, volume 32. Curran Associates, Inc.. 2019. URL httpttts////rrcced.igg..... neurips......pppp......... cC/ paaper/2019/ee 0ar.3330666.44440000000000a0a000
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2015. URL http://ablp. uni-trier: wwwieerddedb/cooo htt.e///ccoooocc//////////////// htt.:eee1111...1m1...mmm.mmm...m Wiioonlyimm
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011 2011-001, Califoria Institute of Technology, 2011.
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. QDrop: Randomly dropping quantization for extremely low-bit post-training quantization. In International Confer- ence on Learning Representations, 2022. URL ySQH0oDyp
net/ fore/fformmidd
J. Xue, J. Li, and Y. Gong. Restructuring of deep neural network acoustic models with singular value decomposition. Proceedings of the Anmual Conference of the International Speech Com- munication Association, INTERSPEECH, pp. 2365-2369, 2013.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S. Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
12

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL INFORMATION FOR EXPERIMENT REPRODUCIBILITY
A.1 WHERE WE GOT THE PRE-TRAIE TRE-TRIINED MODELS
In the experiments, used off-the-shee models. The trained weights for VGG-16, ResNet-18, and MobileNet can be downloaded via the following URLs.
VGG Lag.
httpf////eeaallloo/ooo///o/////d maggnet..thhh...
A.2 PRUNING FOR BRANCHED PATHS OF RESNET
For the layers having skip connections of the ResNet model, cannot perform pruning is, because the number of channels should be the same in the both ends of skip connections. We have several ways to avoid this problem. The first is perform pruning only for the layers without skip connections. The second is to add a layer that only samples the channels to be preserved (He et al., 2017), as shown in Fig. The third is conduct pruning at the both ends of the skip connection (Luo & Wu 2020). We took the the second one.
unpruned
pruned
cXcx3x3
sampling
.cxbx3x3
cXcx3x3
eXcx3x3
+
Figure 2: Instead of removing the channels in the fist layer, we sample channels be preserved.
B ADDITIONAL INFORMATION ON IMPLEMENTATION OF POEM
B. HOW TO EXTEND TO CONVOLUTIONAL LAYERS
In most major DNN frameworks, such Pytorch, feature map in the convolutional layer is panded to be a matrix by using "im2col" function (Paszke et al. 2019). With the expanded matrices, the convolutional operation can be replaced by simple dot product, such that Y XW. There- fore, POEM can be applied to the convolutional layers in the similar with to the fully connected layers. The only difference is that with the expanded feature map, each channel corresponds to sev- eral columns of the matrix. Thus, for convolutional layers, we need to remove several columns of X at each time.
B.2 PRUNING FOR WHOLE MODEL
Once we conduct pruning on layer, the outputs in the subsequent layers are more less affected. Assume that we have conducted pruming in a previous layer and are now pruning in the current layer. The original outputs in the current layer, X, has become X' * X. We use the outputs X', whereas we still use the original Y for neuron selection and reconstruction so that the original performance of the model can be the best preserved.
13

---

Under review as a conference paper at ICLR 2023
regression
S
S
Wi
2nd
S
- W
W
and
S
t -- (St + S2w2)
S1 S2
W
:
m&th
S
Siii
Wm
(=1,",m-1
Figure 3: The concept of Algorithm
B.3 DETAILS OF ALGORITIIM
1
Algorithm aims to solve Eq. 12 with a limited memory space. Let us simplify the notations:
S G Gxtt= G.Y., and W.j. Then, Eq. 12) can be rewritten as typical LS problem:
u² = arg min || Sull?.
(14)
"
The solution is given by
u" = ST STt.
(15)
However, Eq. would require large memory space for computing (STS)-
To solve Eq. 14p with a limited memory space, we sample a small number of columns from S perform regression, as shown in Fig. Let S1, S2. ... be the matrices having a few columns sampled from S. We first perform regression with S and t:
wi arg min || Siw.
(16)
The eтror for is given by - S₁ l. Next, we perform regression so as to fill this error using Sy:
wi arg min [[(t S(wi) S2w2ll
(17)
In this way, we continue to perform regression as to fill the error for t:
wn = arg min Wi
Σ
S.n Wm
(18)
i= L......-a0 mm-1
Once we finish all of the S-s, we continue from S₁ again. After enough cycles, the error for t will be minimized. The approximate solution for Eq. (14) can be calculated from the -s-s.
This algorithm is memory-efficient. The solution of Eq. 18 is given by
"., = Sm) ST
t
Σ
S.w
(19)
i' 1,10
m-11
Since S..) is small matrix, calculating it does not require large memory. Therefore, this algorithm is suitable for parallel calculation.
In Algorithm I we have two hyper parameters: is the number of sampled columns, and M is how many times we perform column sampling and regression. In our experiments, we set = 9, and M 5b, where b is the number of preserved channels.
14

---

Under review as a conference paper at ICLR 2023
Table 5: Benchmarking channel selection methods and reconstruction methods. We pruned 50% of channels in "Conv2-1", "Conv3-1", "Conv4-1" layers of VGG-16, and compared the MSEs in the following layer. Each cell contains the output eπror of f(Y). and the value in parenthesis the error of Y (the error before applying the activation function).
Conv2-1
a) No rec.
b) LS
c) WLS (ours)
Select. Rec.
after f
before f
after
before f
after f
before f
L)
1.2695
5.8902
0.2700
0.8461
0.2570
1.05677
2) L2
1.2433
5.9156
0.2651
0.8305
0,2516
1.0331
3) Lasso
1.3555
4.8980
0.2670
0.8175
0,2519
1.0692
4) GM
1.2001
5.7369
0.2538
0.7958
0.2422
0.9789
5) Error (LS)
1.2060
5.2416
0.2242
0.7000
0.2143
0.8743
6) Error (WLS)
1.1688
5.4151
0.2168
0.6827
0.2048
0.8634
Conv3-1
a) No rec.
b) LS
c) WLS (ours)
Select. Rec.
after f
before f
after
before f
after f
before f
L1
4.0288
21.332
1.0621
3.6876
0.9978
4.5270
2) L2
4.0986
21.388
1.0588
3.6779
0.9959
4,5660
3) Lasso
4.5589
18.946
1.1959
4.1310
1.0997
5.1430
4) GM
4.1844
21.933
1.0842
3.7897
1.0179
4.7850
5) Error (LS)
3.9163
21.916
0.9818
3,4051
0.9260
4.1682
6) Error (WLS)
3.7945
21.145
0.9688
3.3444
0.9094
4.1231
Conv4-1
a) No rec.
bS
c) WLS (ours)
Select. Rec.
after f
before f
after
before f
after f
before f
LI
3.3865
38.488
1.2485
7.6413
1.1966
13.866
L2
3,4854
40.214
1.2656
7.7990
1,2154
14.327
3) Lasso
3.5152
34.330
1.3719
8.2762
1.3325
14.780
GM
3.3631
43.098
1.2504
7.7296
1.1997
13.040
5) Error (LS)
2.9017
35.803
1.1540
6.8973
1.0948
11.518
6) Error (WLS)
2.9430
36.686
1.1132
6.7026
1.0527
10.885
B.4 AN EFFICIENT ALGORITHM FOR SOLVING EQ. 9
implemented the algorithm proposed in (Kamma & Wada 202I) to solve Eq. ! for all efficiently. Here, define additional notations: norm(v) denotes the L2 norm operator for a vector v, and M denotes a Moore-Penrose pseudo inverse of a matrix M.
We assume X ALL and W ALL for simplicity. The motivation for solving Eq. 9 is to obtain the following t for all k.
(20)
We can compute z directly without actually computing W k. To do so, we first compute the following.
X = T
(21)
Let D diag(d), where norm x.)). We calculate the following.
R= x=
(22)
Then, & can be computed
t R. WE.
(23)
15

---

Under review as a conference paper at ICLR 2023
Table 6:
The MSEs in Conv2-1, Conv3-1, Conv4-1 after pruning 50% of the chennels in Conv 1-1.
Conv2-1
No
b) LS
c) WLS (ours)
Select. Ree.
after f
before f
after f
before f
after f
before f
L1
0.7564
2.4049
0.1624
0.4946
0.1041
0.3181
2) L2
0.7395
2.4725
0.0593
0.1789
0.0444
0.1350
3) Lasso
0.6858
2.2340
0.0586
0.1959
0.0382
0.1201
4) GM
0.7189
2.2748
0.2121
0.6496
0.1555
0.4798
5) Error (LS)
0.7177
2.4643
0.0374
0.1167
0.0293
0.0904
6) Error (WLS)
0.8078
2.9315
0.0338
0.1022
0.0279
0.0845
Conv3-1
No rec.
b) LS
c) WLS (ours)
Select. Rec.
after f
before f
after f
before f
after f
before f
LI
1.9539
8.0325
0.3933
1.4787
0,2551
0.8983
2) L2
1.9751
8.3912
0.1518
0,5384
0.1141
0.3973
Lasso
1.9891
8.2181
0.1649
0.6534
0.1071
0.3892
4) GM
1.8400
8.0139
0.5241
1.9235
0.3897
1.3814
5) Error (LS)
1.9232
8.5490
0.0987
0.3580
0.0777
0.2733
6) Error (WLS)
2.1078
10.0640
0.0850
0.2967
0.0704
0.2430
Conv4-1
No rec.
b) LS
C) WLS (ours)
Select. Rec.
after /
before /
after f
before f
after /
before /
L)LI
1.5469
10.8463
0.2801
1.8480
0.1752
1.1079
2) L.2
1.4809
11.1189
0.1097
0.6932
0.0811
0.5015
Lasso
1.5120
11.8430
0.1208
0.8532
0,0790
0.5054
4) GM
1.4223
10.7641
0.3933
2.5382
0.2857
1,7862
Error (LS)
1.3722
11.2451
0.0680
0.4377
0.0540
0.3383
6) Error (WLS)
1.5128
15,2813
0,0564
0.3459
0.0468
0.2847
C EXTRA RESULTS
C.I EXTRA LAYER-WISE ANALYSIS
We conducted layer- wise analyses several layers other than Conv1-1 layer. Table shows the results. Similarly with Table | our WLS-based reconstruction and neuron selection was better than the other approaches.
C.2 IMPACT OF PRUNING LAYER ON THE SUBSEQUENT LAYERS
We pruned 50% of the channels in Convl-1 (the first convolutional layer) of VGG-16. and calculated the MSES in several subsequent layers (Conv2-1, Conv3-1, Conv4-1), Table the results. We that the proposed channel selection method and the reconstruction method better than the others. This trend consistent with Table The trend is that the smaller the error in Convl-1 (where pruning performed) was, the smaller the errors in the subsequent layers were.
C.3 IMPORTANCE CHANNEL SELECTION
We performed pruning in all convolutional layers of VGG-16 to benchmark the channel selection criteria. For channel selection, we used all criteria that are listed in Table [ (LI, L2, Lasso, GM, Error (LS). Error (WLS));; we combined these channel selection options with our WLS-based reconstruction. The compression ratios were set to 50% in all target layers.
Table shows the results. The accuracy after pruning without fine-tuning is reported. With channel selection criteria (Error (WLS)), the accuracy was the best preserved. These results show the importance of our channel selection criteria well reconstruction.
16

---

Under review as a conference paper at ICLR 2023
Table 7: The comparison of channel selection criteria combined with WL.S-based reconstruc- tion. We conducted pruning with fixed compression ratios (50% in one case, 60% in another case) in all layers. The accuracy before fine-tuning is reported in this table. The baseline accuracy of VGG-16 16 was 0.70272 (top-1) 1) and 0.8946 (top-5).
w/o fine-tune
Model
Select criteria
FLOPs
Top-1
Top-5
LI
-73.5%
41.31
68.85
L2
-73.5%
45.62
72.62
Lasso
-73.5%
43.35
70.82
GM
-73.5%
48.38
74.75
Error (LS)
-73.5%
50.19
76.38
Error (WLS)
-73.5%
51.20
77.04
VGG-16
LI
-82.6%
23.68
48.24
L2
-82.6%
29.05
55.92
Lasso
82.6%
27.03
52.63
GM
-82.6%
25.36
50.77
Error (LSS
-82.6%
37.63
64.77
Error (WLS)
-82.6%
39.13
66.82
Table 8: The results pruning with uniform compression ratios. The accuracy after 100 epochs of fine-tuning is reported in this table. The baseline accuracy of VGG-16 was 0.70272 (top- 1) and 0.8946 (top-5) and the accuracy ResNet- 18 was 0.69758 (top-1) 1) and 0.89076 (top-5).
Acc. fine-tuned
Model
Method
FLOPs
Top-1
Top-5
CP-A
-75.8%
68.89
89.08
REAP-A
-75.8%
0.6945
0.8928
POEM-A
-75.8%
0.6998
0.8941
VGG-16
CP-B
-75.7%
70.47
89.59
REAP-B
-75.7%
70.68
89.85
POEM-B
-75.7%
71.01
89.96
CP
-50.4%
64.15
85.66
ResNet- 18
REAP
-50.4%
64.46
85,89
POEM
-50.4%
64.77
85.97
C.4 EXTRA FINE-TUNING ON VGG-16 AND RESNET-18
We conducted longer fine- mne-tuning (100 epochs) for the pruned VGG-16 and ResNet- 18 models. The results are shown in Table & (with uniform compression ratios) and Table optimization). Except for the top-5 error of ResNet-18 with RandomPruning, POEM outperformed the previous methods. These results show the trend: the higher the accuracy was preserved during the pruning procedure, the higher the accuracy becomes after fine-tuning.
(with (with compression ratio
C.5 RESULTS WITH RESNET-50 (WITH FIXED PRUNING RATIOS)
We evaluated POEM with larger model ResNet-50. We conducted pruning in 32 layers that do have skip connections. The pruning ratios were set uniformly over all target layers and were tuned so to make the FLOPs reduction ratio approximately 50%.
The results are shown in Table 10 Before fine-tuning, POEM outperformed the second best method REAP significantly: the top- accuracy was 59.75% (POEM) and 44.60% (REAP). After fine- tuning, POEM still outperformed REAP in top-1 accuracy, although the top-5 accuracy of CP was slightly better than POEM slightly. To confirm the winner for ResNet-50, more experiments are necessary with higher pruning ratios, and with pruning ratio optimizations.
17

---

Under review as a conference paper at ICLR 2023
Table The results of pruning with pruning ratio optimization. The accuracy after 100 epochs of fine-tuning is reported in this table. The baseline accuracy of VGG-16 16 was 0.70272 (top-1) 1) and 0.8946 (top-5) and that of ResNet-18 was 0.69758 (top-1) and 0.89076 (top-5).
Acc. fine-tuned
Comp. ratio
Model
Method
FLOPs
Top-1
Top-5
LI&LS
-75.7%
70.01
89.46
VGG-16
REAP
-75.4%
70.95
89.97
POEM
-75.7%
71.19
90.00
PRO
LT&LS
-50.3%
66.00
86.89
ResNet-18
REAP
-50.4%
65.81
86.90
POEM
-50.0%
66.27
87.08
LI&LS
-75.7%
70.00
89.37
VGG-16
REAP
-75.6%
70.58
89.73
POEM
-74.6%
70.99
89.94
RandomPruning
LI&LS -0.5%
65.44
86.77
ResNet-18
REAP
-00.7%
65.47
86.51
POEM
49.2%
65.68
86.62
Table 10: The results of pruning for ResNet-50 with uniform compression ratios.
The baseline
accuracy of ResNet-50 was 76.12 (top-1) and 92.86 (top-5).
Acc. w/o fine-tune
Acc. fine-tuned
Model
Method
FLOPs
Top-1
Top-5
Top-1
Top-5
CP
-49.9%
42.74
68.55
73.31
91.61
ResNer-50
REAP
-49.9%
44.60
70.55
73.50
91.42
POEM
-49.9%
59.75
83.68
73.73
91.58
18


Reviews:
[{"review_id": "UZckqyR5z4c", "reviewer": "Reviewer_fHTY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose Pruning with Output Error Minimization (POEM) technique to improve model accuracy after structured pruning. In order to include activation function in the objective function to be optimized, the authors suggest the weighted least squares method while the authors claim the previous methods minimize the error of the value before applying the activation function. One example is ReLU function when negative inputs are converted into zeros such that such highly non-linear functions need to be included during selecting neurons to be pruned. Throughout various CNN models, POEM significantly outperforms the previous methods when fune-tuning is not employed. Even after fine-tuning, POEM usually yields better top-1 and top-5 accuracy.", "summary_of_the_review": "POEM idea itself is quite interesting and intuitive while this reviewer believes that the underlying principles (that can include activation functions in the objective function) have already been considered in other model compression techniques (especially for post-training quantization, e.g., BRECQ). Also, this reviewer is wondering why structured pruning has been investigated in the manuscript while POEM can be extended to other model compression ideas (experimental results are not very impressive after fine-tuning).", "strength_and_weaknesses": "*Strength\n- The need to include activation function is clear (especially as shown in Figure 1)\n- Simplified reconstruction strategy is intuitive and computationally reasonable.\n- WLS can be combined with various pruning criteria and pruning ratio decision methods.\n\n*Weakness\n- Even though model accuracy without fine-tuning is impressive by POEM, after fine-tuning, the impact of POEM seems to be marginal.\n- Experiments are limited to old models. What about the chances to apply POEM to Transformers? Is the method specific to vision tasks?\n- Limitations of the proposed techniques are not clearly described.\n- There are numerous structured pruning methods recently published. Comparison with those methods are not provided (such as SNIP?)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BrICW-2nt_Z", "reviewer": "Reviewer_z5BU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new pruning algorithm Pruning with Output Error Minimization (POEM), which focus on the problem of \"picking which neurons for pruning\", and \"how to reconstruct the optimal value for unpruned weights\".\n\nThe paper's main claim is using the value after activation, as the targets and output, to minimize their difference before and after pruning. \nFor the proposed method, pruned channel selection and reconstruction are based on this optimization criterion.\n\nThe experiments show that using POEM's channel selection and reconstruction, DNNs can have smaller reconstruction error and better classification accuracy.", "summary_of_the_review": "The paper proposed a pruning channel selection method and show that it is better than some other channel selection methods.\nThe experiments comparison is not clear on two issues: 1) it's not clear how is the advantage of proposed method if better fine-tuning is conducted; 2) training from scratch baseline in this paper gives different conclusion to an existing reference.\n\n=================================================\n\nThanks for the authors' response and additional experiments results. From the new results I think the gap between different pruning methods is relatively smaller.\nI think it will be a good point that the proposed method is more important on datasets with insufficient samples per classes. However, in the current version of this paper, this point was not proposed as the main contribution. I would suggest the authors to dive deep on this point and show more results in terms of this observation.\nThus I keep the previous score for the paper.", "strength_and_weaknesses": "Strength:\n- This paper start from an important observation for the problem of how to select the channels for pruning, and how to adjust the values of unpruned weights after pruning.\n- The proposed formulation looks solid and reasonable: the author considers both computation complexity and algorithm accuracy in the formulation and algorithm derivation.\n- From the experiments, the proposed algorithm show its effectiveness in reducing the output activation L2 error, and also show that ImageNet classification accuracy is better than compared methods, with or without finetuning.\n\nWeaknesses\n- On the pruning criterion, although it makes more sense to use the layer's output after activation function than before activation function. However, why not directly use the final output as the error can be accumulated to the final layer and minimizing each layer's output will be sub-optimal. In addition, directly optimizing the final metric: accuracy or loss should be a better choice. E.g., in LeGR[1], the pruning metric is directly set as the accuracy.\n- In the experiments, pruning before finetuning is significantly worse than after finetuning (e.g., table 3). It means that finetuning is very important to pruning methods. However, in the experiments, fine-tuning is only performed for 25 epochs, and the effect of different fine-tuning hyperparameters are not comprehensively studied. I wonder if better fine-tuning is conducted (e.g., better lr schedule and longer training epochs, using knowledge distillation etc), the benefits of better channel selection and weights reconstruction is less or more.\n- In [2], experiments show that smaller network trained from scratch with similar or even better accuracy than channel pruning. While in this paper, table 4, training scratch has significantly worse accuracy than any pruning methods. Why the experiments in this paper cannot repro what was observed in [2]?\n\n[1] Chin, Ting-Wu, et al. \"Towards efficient model compression via learned global ranking.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[2] Liu, Zhuang, et al. \"Rethinking the value of network pruning.\" arXiv preprint arXiv:1810.05270 (2018).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AupobF-zHF", "reviewer": "Reviewer_9TdV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a layer-wise pruning method by minimizing the reconstruction errors of nonlinear outputs. Unlike the previous methods that compute the errors before the nonlinear function, the proposed method does it after the nonlinearity and focuses on reducing the errors in the non-saturating regions. The authors formulate their idea as a weighted least squares method and also introduce an efficient way to solve it. The proposed approach is validated with relatively shallow networks.", "summary_of_the_review": "I think the main idea is simple and intuitive, but the empirical results are quite weak and some claims should be better supported. I thus find it difficult to argue for acceptance of the work.", "strength_and_weaknesses": "Strength\n- The idea that considers the reconstruction errors of post-activation values is simple yet well-motivated.\n- The authors validate the proposed layer-wise optimization under various experimental setups (e.g., at fixed compression ratios, combined with recent compression ratio optimization methods). Furthermore, the results before as well as after finetuning are presented to show the effectiveness of the method.\n- The paper is easy to follow and fairly clearly written. I think most members of the community would be able to easily dissect it without requiring substantial prior knowledge.\n\nWeaknesses \n- I am not sure whether good reconstruction of the features leads to (or is actually related to) good final accuracy after finetuning. One may think that because finetuning significantly recovers the performance, how well the features are reconstructed may have less impact. I would recommend the authors to show lower MSEs in Table 1 result in better accuracies after finetuning. Furthermore, I was wondering if the result trend in Table 1 could be also observable in higher layers and/or other networks.\n- I am not sure whether omitting f’(Y) when deriving (9) from (8) is okay. I think a key element of the proposed method is applying the derivative of nonlinearity to a typical LS problem of (1), changing it into the weighted LS of (5). However, the omission of f’(Y) in (9) looks similar to the typical LS setup of (1) and may be unreasonable according to the authors’ claim in Section 3.1. Could the authors provide why this approximation is valid?\n- The main idea of this paper looks very closer to that of [Jiang, IJCAI’18]: “Taking the widely used activation function ReLU r(x) = max(0,x) as an example, the Euclidean distance can be large between two negative input values, but would become 0 after activation. Thus, nonlinear reconstruction error (NRE), which computes the Euclidean distance between nonlinear activation values of the unpruned model and those of the pruned model, can be a more reasonable metric than LRE when performing layer-wise pruning.” What are the differences and merits of the proposed approach over [Jiang, IJCAI’18]?\n    - Jiang et al., Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error, IJCAI'18\n- I think the experiments are conducted on relatively shallow networks (i.e., VGG-16, ResNet-18). Recently, many pruning algorithms have been tested on ResNet-50 and/or ResNet-101 [Luo et al., ICCV’17; Liu et al., ICCV’19; He et al, CVPR’19; Li et al., CVPR’22]. I would recommend the authors to present additional results with deeper depth (at least, ResNet-50) to demonstrate the scalability and effectiveness of the proposed method. \n    - Luo et al., ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression, ICCV’17\n    - Liu et al., MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning, ICCV'19\n    - He et al., Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration, CVPR’19\n    - Li et al., Revisiting Random Channel Pruning for Neural Network Compression, CVPR’22\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6uHbztX0aUj", "reviewer": "Reviewer_E1Gd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a layer-wise pruning method that conducts pruning and performs reconstruction to minimize the output error caused by pruning. In reconstruction they minimize the output error of the activation function, while the previous methods minimize the error of the value before applying the activation function. Their experimental results show improvement over existing methods.", "summary_of_the_review": "The paper proposed to minimize the output error after applying the activation function. This approach is very similar to knowledge transfer at the feature level. The paper needs to clarify the similarity and mention how different they are. Also the experimental results need to be improved to show the impact of the channel selection over the existing method. \n", "strength_and_weaknesses": "Strength:\n1- The paper motivates minimizing the output error of the activation function by illustrating that it is more important to reduce the error for the elements in non-flat zones. In flat zones of these functions, the errors are suppressed. \n\n2- In neuron selection they use a greedy approach to pick the best neuron next.\n\n3- Since the regression should be performed for each column separately they showed that this can be done by  parallelizing this for each j to obtain the solution efficiently\n\nWeakness:\n1- This approach looks similar to the distillation methods where the goal is to minimize the distance between the output distribution of the teacher and student model where the knowledge transfer is on feature level. The paper needs to clarify how different they are.\n\n2- Table 1 shows the layer wise analysis of the channel selection and reconstruction methods for Conv1-1. Even though this results sounds interesting I am wondering how applying this approach in one layer would affect other layers? Also for the result in table 2 does the proposed approach apply all at once (independently) on all layers or it is an iterative approach where one is conditional on the previous pruning. \n\n3- In both scenarios above I am wondering how the channel selection approach would affect the final result. It is true that it is shown in table 1 proposed channel selection and reconstruction is best (for one layer) but apparently the reconstruction approach improves the result independent of the channel selection. So I am wondering if you use the proposed reconstruction method but use (L1, L2, Lasso, …) for the channel selection for all the layers then what table 2 looks like? This experiment would show the importance of proposed channel selection.\n\n4- The paper mentions that POEM is superior to the previous methods in maintaining the accuracy of the pruned model since it performs reconstruction only for the positive elements, while the previous methods perform reconstruction for all elements including negative ones. I am not sure if this is a correct statement given that this method is data dependent.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject