AN OPERATOR NORM BASED PASSIVE FILTER PRUNING METHOD FOR EFFICIENT CNNS
Authors: Arshdeep Singh
Yunpeng Li
Mark D Plumbley
Venue: Submitted to ICLR 2023 (2023)
Keywords: Convolutional neural network
filter pruning
VGGish
DCASE
MNIST

Abstract: Convolutional neural networks (CNNs) have shown state-of-the-art performance in various applications. However, CNNs are resource-hungry due to their requirement of high computational complexity and memory storage. Recent efforts toward achieving  computational efficiency in CNNs involve filter pruning methods that eliminate some of the filters in CNNs based on the "importance" of the filters. Existing passive filter pruning methods typically use the entry-wise norm of the filters to quantify filter importance, without considering how well the filter contributes in producing the node output. Under situations where the large number of filters are to be pruned from the network, the entry-wise norm methods always select high entry-wise norm filters as important, and ignore the diversity learned by the other filters that may result in degradation in the performance.  To address this, we present a passive filter pruning method where the filters are pruned based on their contribution in producing output by implicitly considering the operator norm of the filters. The computational cost and memory requirement is reduced significantly by eliminating filters and their corresponding feature maps from the network. Accuracy similar to the original network is recovered by fine-tuning the pruned network. The proposed pruning method gives similar or better performance and recovers accuracy faster during the fine-tuning process than the entry-wise norm-based pruning methods. The efficacy of the proposed pruning method is evaluated on audio scene classification (e.g. TAU Urban Acoustic Scenes 2020) and image classification (MNIST handwritten digit classification). 

Under review as a conference paper at ICLR 2023
AN OPERATOR NORM BASED PASSIVE FILTER PRUNING METHOD FOR EFFICIENT CNNS
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional neural networks (CNNs) have shown ofate-of-the-ar performance in various applications. However, CNNs are resource-hungry due to their require- ment of high computational complexity and memory storage. Recent efforts to- ward achieving computational efficiency CNNs involve filter pruning methods that eliminate some of the filters in CNNs based the "importance" of the fil- ters. Existing passive filter pruning methods typically use the entry-wise norm of the filters quantify filter importance, without considering how well the fil- ter contributes in producing the node output. Under high pruning ratio where the large number of filters to be pruned from the network, the entry- norm methods always select high entry-wise norm filters as important, and ignore the diversity learned by the other filters that may result in degradation in the perfor- mance. To address this, we present a passive filter pruning method where the filters are pruned based on their contribution in producing output by implicitly considering the operator norm of the filters. The computational cost and memory requirement reduced significantly by eliminating filters and their corresponding feature maps from the network. Accuracy similar to the original network is re- covered by fine-tuning the pruned network. The proposed pruning method gives similar or better performance than the entry-wise norm-based pruning methods at various pruning ratios. The efficacy the proposed pruning method is evaluated audio scene classification (e.g. TAU Urban Acoustic Scenes 2020) and image classification (MNIST handwritten digit classification, CIFAR-10).
INTRODUCTION
Convolutional neural networks (CNNs) have shown great success and exhibit state-of-the-arr per- formance when compared to traditional hand-crafted methods in many domains (Gu et al. (018)). ). Even though CNNs are highly efficient in solving non-linear complex tasks (Denton et al. 2014
it may be challenging to deploy large-size CNNs resource- conuurcce-cnttrieeeed devices such as mobile phones or internet of things (IoT) devices, owing to high computational costs during inference and the memory requirement for CNNs (Simonyan & Zisserman (2015); Krizhevsky et al. (2012)). Thus, the issue of reducing the size and the computational cost of CNNs has drawn a sigmificant amount of attention in the research community.
Recent efforts toward reducing the computational complexity of CNNs involve pruning methods where a set of parameters, such as weights or filters, are eliminated from the CNNs. These prun- ing methods are motivated by the existence of redundant parameters (Denil et al., (2013): Livni al. (2014)) in CNNs that only yield extra computations without contributing much in performance (Frankle & Carbin (201)). For example, [ii et al. (0017) found that 64% of the parameters, con- tributing approximately 34% of the computation time, are redundant. Eliminating such redundant parameters from CNNs provides small CNNs that perform similar to the original CNNs while re- ducing the computations and the memory requirement compared to the original CNNs.
While eliminating weights from unpruned CNN may result in a highly sparse network with few parameters, a pruned network obtained by eliminating individual weights is unstructured and may not be straightforward pruned networks limited due to random connections despite high sparsity (Luo et al. (2017). Moreover, the unstructured sparse networks not be supported by off-the-shelf libraries and
more efficiently. The practical acceleration in the unstructured sparse

---

Under review as a conference paper at ICLR 2023
Input (X)
Ouigut (X)
F:X-Y
Stretches X maximally by [|]
UEW?
x E
Maxiimally stretched
01
Y (er (mr (oruaw))
approximation
Figure 1: A geometrical view of output produced by a convolution operation, where input feature maps X in Iº² are transformed to output feature maps Y in 2² using transformation matrix F. F decomposed to left singular matrix (U), a right singular matrix (W) and a diagonal matrix (E) using a singular value decomposition method. U and W are orthogonal matrices that cause rotation in the input, and σ1 and 02 are singular values that scale the input. F stretches X maximally by ||||| 0i which is an operator norm of F.
quire specialised software or hardware for speed-up (Wen et al. (2016); : Han et al. (0166). To ad- dress this unstructured pruning problem, several filter pruning methods have been proposed which eliminates whole filters, resulting in structured pruned network (Luo et al. (2018) that does not require additional resources for speed-up.
In these structured filter pruning methods, the "importance" of filter, used to decide if filter is retained or eliminated, measured using either active or passive methods. Active filter pruning methods involve dataset to compute the importance of filters. For example, (Luo & Wu (017)) 7); Polyak & Wolf 2015); Hu et al. 2016); Lin et al. (2020); Yeom et al. (202)) use feature maps which are the outputs generated from CNN filters corresponding to set of examples, and apply metrics such as entropy, variance, average rank of feature maps and the average percentage zeros on the feature maps to quantify the importance of the filters. Other methods including (Liu et al. (217)) ; Lin et al. (2019)) compute important set of filters during training process by optimizing a soft mask associated with each feature map using regularization. However, these methods are time- consuming and use extra memory resources to obtain feature maps. On the other hand, passive filter pruning methods only use the parameters of the filters without involving any dataset or optimizing process to compute the importance of the filters. Therefore, the passive filter pruning methods are less time-consuming and require less memory resources identifying important set of filters. In par- ticular, when there exists already a pre-trained network to be pruned, identifying pruned set of filters using optimization process (Liu et al. 20117) Lin et al. 2019)) would be heavily computationally expensive compared to that of the passive filter pruning methods.
typical passive filter pruning method uses an entry-wise norm the filters to measure their im- portance. For example, this might be 1 lo-ormm sum of absolute values of each entry in the filter or an 12-norm square root of the squared sum of each entry in the filter. Li et al. (017)) eliminates filters having smaller entry-wise 1n-mo or 12-norm as measured from the origin, and finds that eliminating filters based on the entry-wise 12-norm of the filters gives similar performance to that of the entry-wise ( (ooom.. He al. (2019) eliminates the filters with smaller (2-norm as measured from the geometric median of all filters. Both the previous methods assume that a filter with smaller entry-wise norm less informative less important, without considering how significantly filter contribute in producing output. For example, an illustration of the contribution by the filters in pro- ducing output is shown Figure where filters produces an output Y by maximally stretching the input X by largest singular value 0 that represents an operator norm of the F. However, the entry-wise norm methods not consider any input-output relationship information and rely on each entry of the filter while computing the filter importance.
To illustrate the above further, we pictorially show in Figure a that two filters F' and F having same entry-wise norm contribute differently and produce different output due to different operator norm of the each filter shown in Figure b). Hence such filters should be given different importance. Moreover, when a few number of filters have to be retained in the CNN to yield very small pruned CNN, selecting filters with only high entry-wise norm may ignore the smaller norm filters that may also contribute significantly in producing output (Ye et al. (2018). This may degrade the accuracy of
2

---

Under review as a conference paper at ICLR 2023
0.1
F
0.1
F²
F
Entry wise I corm (Bormallaed)
0.94
1
Operator
Y' = x'
Y X'
Y X2
(6) Outpats produced by applying these different fllters
Figure 2: (a) An illustration of output produced in the convolution layer by three CNN filters, F1, F2 and F$ after applying a convolution operation on E 1R2210000 (b) shows the entry-wise 4n-ormm and the operator norm of the three filters.
the pruned network significantly due relying on the filters having high entry-wise norm, however low significance in producing output. For example as shown in Figure bb), F² has 0 = 3, and stretches X relatively larger than that of the F' and the F3, However, the entry-wise norm of the F2 is the smallest among all three filters. Therefore F² shall be ignored by the entry-wise norm methods despite a relatively high contribution than other filters.
To select the important filters based their contribution in producing output, we propose novel passive filter pruning framework by considering filters in a convolution layer as an operator that transform input feature maps output feature maps. To compute the importance of filters, we use the operator norm of the filters, which represents the maximum contribution of the filters in producing output rather than relying the entry-wise norm of the filters. Utilising all filters in convolutional layer, we use a Rank-1 approximation of the filters to obtain the maximally stretched direction associated with the operator norm in which the input gets stretched maximally by all filters. filter in the convolutional layer is deemed important based on how well aligned along the maximally stretched direction represented by all filters in that convolutional layer.
The proposed passive filter pruning method is evaluated on pre-trained CNNs such as VGGish (Her- shey al. (2017)) and DCASE 2021 Task 1A baseline (Martín-Morató al. (2021) designed for audio scene classification (ASC), VGG-16 network (Liu & Deng (015))and ResNet-50 (He et al. 2016 designed for image classification problem.
2
METHODOLOGY
Notation: Consider a CNN having L convolutional layers with indexes E .............. A feature map in a convolutional layer denotes an output produced by a CNN filter. the 1th convo- lution layer, let denote the number of input channels with each feature map of size h; x wy. Let X of size (Mt-1 f1-1 X w1-1) and Y of size (nı X h x wi), denote the feature map matrices, produced by stacking all the respective feature maps in the (1 1)th and the (th layer respectively. A 5th feature map in the 1hh layer produced by a convolution operation using a jth filter F's of size (n--1 X k; k)) having R1-1 2D kernels each of (k₃ k2) on X. All filters in the 1th convolution layer constitute a kernel tensor K, = [F'.1 F Fl.ri of size (n X ny x kj x kt). A pictorial illustration of two intermediate layers in the CNN is described in Figure

---

Under review as a conference paper at ICLR 2023
filters
Eiters
pirld
Fli pla.. plln
R4-1
channels
chounels
K2
Ki+1 of size
(Ri-1
X
k)
Y
RJ 7i-1 Raya Rio1
(1+ lager Figure 3: An illustration of the intermediate layer structure and filter pruning in CNN. In the 1 layer, there are n, filters each having (k₃ x ki) size and n4-1 channels. The ny filters produce ny feature maps. Pruning the jth filter, F in the 1th convolution layer results in elimination of the feature map produced by the pruned filter and corresponding channel the (1 + 1)th layer.
Lager
Mager
v
Filter
2 channel specificc matrix
SVD
Filer
V
viu
Channol-eise maximally
Ro-cchaneel up ppiifii mateix
stretched direction
V
SVD
Y
Compute directions amociiate with an opecator noem each chaanel speetfic matrix
T. Sibees each having Ri-1 channols in e convloutional layer
singular value decotapaiiion (SVD)
Figure 4: An illustration obtain maximally stretched direction for each n channels across all filters in the 1t convolutional layer.
2.1 COMPUTING FILTER IMPORTANCE IN THE (TH CONVOLUTION LAYER
Given kernel tensor K, for the 1th convolution layer, aim is compute the importance of each of the ny filters. Without loss of generality, we transform the kernel tensor K₁ to K" of size (ni-1 X ny kz") by vectrozing the 2D kernels each of (kz kj) kj".
To identify importance of each filter, we use K" to obtain a maximally stretched direction corre- sponding the operator norm of the filters along which the filters in the given convolutional layer stretches the input maximally. Since the output produced by the th channel of a filter depends only on the th channel of the input in CNNs, the maximally stretched direction is obtained for each channel of the filters independently.
channel-specific matrix V- of size (nz x kz") constructed by taking nh channel of filters. V_ denotes the learned weights of the et channel across the filters. Next, singular value de- composition (SVD) is performed on V. to compute rank-1 approximation of the d channel as V. ~ Ij w|. Here, a denotes the maximum singular value that affects the corresponding chan- nel of the input maximally and is equivalent to the operator norm of the Ve. u, the first left singular vector, and Wa is the first right singular vector. A row of V. normalized to unit norm is denoted as Vmax and is considered as a maximally stretched direction for the ch channel. Vmax provides reference for measuring the significance the et channel of the filter.

---

Under review as a conference paper at ICLR 2023
Algorithm Filter Importance Calculation
input : 1 of size (ri=1 X nı kt kernel matrix in (th layer.
output: norme # normalized importance scores of filters, in 1th layer.
Initialization: C₁ = I, Score [ J, importance score of filters.
# Obtaining channel-wise maximally stretched direction associated with operator norm...
for <- 1 to 1 do
=K= : # Take e channel of size (n4 x ktt) from all the filters,
2 SVD( V.) # Perform SVD on Vo. (u)
#Take first left (u,) and first right (w1) singular vector associated with 01, take any row of un u.w] and normalized 1 and append the normalized row Ct.
end for
Importance-score calculation
-- to ny do F = K' j. # Take 1th filter.
F Score "ecoreaapeed((trcc(((""" CC)))) #Compute importance.
end
[cooe]]
Score norm
= max(n?) # Normalized importance.
return Score_norm
After obtaining Vinax corresponding to the et channel, we obtain maximally stretched directions for other channels. Finally, the Vmmax of all channels are stacked together yield C₁ of size (k₂" n for given convolutional layer. A pictorial representation to obtain C is shown in Figure 4
Given C₁, the importance for the jth filter is computed as, Trace (F)) The filters are ranked as per their importance with relatively high importance score indicates a high contribution of the filter in producing output. Algorithm [ssmmarizes the process compute the importance of various filters in given convolutional layer.
After ranking the filters based on their importance for various convolutional layers, few unimportant filters are eliminated based on user-defined pruning ratio for various convolutional layers of CNN, and a pruned network is obtained. Pruning a filter in the 7th convolution layer also eliminates the feature map produced by the pruned filter and the related kernel or channel in the next layer as shown Figure hence the computations is reduced in both the 1th layer and the (I 1)th layer. In the end, fine-tuning of the pruned network is performed to regain the performance loss due elimination of filters from the original network.
3 EXPERIMENTS
We evaluate the proposed pruning method on CNNs designed for audio scene classification (ASC) and image classification. A brief summary of the unpruned CNNs used for experimentation is shown in F and is described below.
Unpruned CNNs: We use two different unpruned networks for ASC, (a) VGGish Net and (b) DCASE21_Ne We also use (c) VGG-16 network for image classification.
(a) VGGish_Nes The VGGish_ Net is built using publicly available pre-trained VGGish net- work (Hershey et al. (2017) followed by a dense and a classification layer. We train the VGGish on TUT Urban Acoustic Scenes 2018 development (we denote it DCASE- 18) training dataset (Mesaros et al. (2018) to classify 10 different audio scenes using Adam optimizer with cross-entropy loss function for 200 epochs. The input to the VGGish is a log melspectrogram of size (96 x 64) computed corresponding to 960ms audio segment for whole 10s audio scene. The VGGish Net has 55,361,162 (approximately 55M) pa- rameters, requiring 903M caltiply-accumulatt operations (MACs during inference cor- responding to an audio clip of 960ms and gives 64.69% accuracy on 10s audio scene for
'To compute MACs, use publicly available "nessi.py" script available httpt:t chub.com
A1 bertoAnci etto/NeSsi
5

---

Under review as a conference paper at ICLR 2023
Parameter Count: 55M MACs: 901M
Unpraned accuracy:
Pre-trained VGGish layer
Acoustic seenc
(a) VGGish
960ms audio
dataee
log-mel spectrogram 64x336
segment
Parameter Count: 46246
Unpraned accuracy: 48.58%
MACs: 287M
Pre trained DCASE 2021 Taskl baseline
Acoustic scene classification DCASE- ditinec
(b) DCASE21 Net
10s audio
clip
log-mel spectrogram
R 40:000
Parameter Count 15M
Unpruned accuracy: 99 49%
(c) VGG-16
Handwritten Digit
VGG-16 network
Ceasiiicaion
Image
2xx2xx
Figure 5: Unpruned CNNs used for experimentation; (a) VGGish (b) DCASE21 .Net and (c) VGG.Net.
DCASE-18 development validation dataset. The VGGish has six convolution layers (termed as C1 to C6).
(b) DCASE21 Net: DCASE21 Net is publicly available pre-trained network designed for DCASE 2021 Task 1A that is trained using TAU Urban Acoustic Scenes 2020 Mobile de- velopment dataset (we denote "DCASE-20") to classify 10 different audio scenes (Martín- Morató et al. (2021)). The input to the network is a log-med meg-messecttrrraa of size (40 x 500) corresponding to a 10s audio clip. The trained network has 46246 parameters, re- quiring approximately 287M MACs during inference corresponding to 10-second-length audio clip and gives 48. 48.58% accuracy on the DCASE-20 development validation dataset. DCASE21 Net consists of three convolutional layers (termed C1 to C3) and one fully connected layer.
(c) VGG-16: We use VGG-16 network (Liu & Deng 20155 )) for handwritten digit classifi- cation using MNIST dataset" LeCun LeCun et al. (2100). We train the VGG-16 16 from scratch for 200 epochs on the MNIST traiiing dataset. The VGG-16 consists of convolutional layer (termed as C1 to C13) and 2 dense layers. The VGG-16 has 15M parameters and requires
329M MACs per inference and the trained VGG-16 gives 99.499.4499 accuracy for the MNIST testing dataset.
Apart from unpruned networks (a), (b) and (c), evaluate the prposed method VGG-16 16 and ResNet-50 (He et al. 2016)) networks using CIFAR-10 dataset (Krizhevsky & Hinton 2009 and an initial analysis ResNet-50 using CIFAR-10 dataset. Please refer Appendix for detailed analy-
sis.
Fine-tuning of the pruned network: After computing importance of the filters using Algorithm across various convolutional layers, we eliminate percentage of unimportant filters from various convolutional layer, where p {25, 50, 75, 90) denotes a pruning ratio. Once the pruned network is obtained, we re-train the network with similar conditions such as same optimizer as used in the training of the unpruned networks (a), (b) and (c) for 100 epochs. The codes for the proposed pruning framework, various pruned and unpruned networks can be found at the link
'The MNIST dataset is downloaded using Keras API which has pre-defined training and testing of grayscale images. Each grayscale image of size (28 28) is converted into three channels by depthwise stacking grayscale image and then reshaping to (32 x 32 3) that used as an input to VGG-16.
Shtpps: pttttss://annyymaat ananymonnn.uuuu. 4open. scceene////cccccccccccccc / /peeator norm pass i ve__________ Fil_iiitee 1ter_
Pruni mouning-1200

---

Under review as a conference paper at ICLR 2023
25% pruning ratio
50% pruning ratio
"Green" color: Accuracy
75% pruning ratio
90% pruning ratio
"Red" color: MACs/aaamters
80.
66.7
80
Faramters (N)
8
80
40.0
Atsurery
8
40.0
8
No
65.6
C
No
C3 c
Pruning
Pruncd layers
Pruning
Pruned layers
(a) VGGish Net on DCASE-18 dataset
100
100
80
1%6
Parambers (%0
00
2 196)
30.0
MACS
16
15.0
20
8
No
C1
C2
C2
C1
C1
C5
C2
c12
C2
C1
Pruning
Prumed layers
Pruning
Pruned layers
(b) DCASE21 Net on DCASE-20 dataset
100.0
00.0
100
100
80
ALcURaR (%)
naramders (%a)
80
8
ACCUS aayy (%)
(ACX(1))
60
15
8
%
No CII
No CII C9 C5 C CL
Pruning
Pruned layers
Pruning
Pruned layers
(c) VGG-16 MNIST dataset
Figure 6: Accuracy, MACs and parameters across different pruned networks for (a) VGGishNNet (b) DCASE21_Ne and (c) VGG-16 network, when different subsets of convolutional layers are pruned at different pruning ratios. Here, "CA_B" as "pruned layers" means that convolution layers from A B are pruned.
Other methods for comparison: We compare the proposed operator norm based pruning method with that of the entry-wise norm based methods, (a) In-norm method that climinates filters with smaller entry-wise 4ı-ormm (Li et al. (2017)) and (b) geometric median (GM) method that climi- nates filters with smaller 12-norm as measured from the geometric median of all filters (He et al.) 2219)). We also compare the proposed pruning method with the existing active filter pruning meth- ods including HRank (Lin et al. 2020)) ), Energy-aware pruning (Yeom et al. (202T L1 L1-slimming
(Liu et al. (0177). and GAL-0.1 (Lin al. 2019)) in Appendix: Table
4 RESULTS
We analyse accuracy, the number MACs and the number of parameters the pruned networks obtained after pruning various subsets of convolution layers at different pruning ratios from the unpruned network.
VGGish Net and DCASE21 Neet on ASC: As shown in Figure (a.b), the accuracy obtained using the various pruned networks is similar to that of the unpruned networks at p = 25%. For VG- Get, the number of MACs reduced by 40 percentage points and the parameters by 25 percentage points, when all convolutional layers (C1 CI to C6) are pruned at = 25% shown in Figure 6 a). For DCASE21 Net, both the MACs and the parameters reduced by 40
reduced

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of accuracy, the MACs and the parameters among the (a) unpruned, (b) the pruned network obtained using the proposed pruning method and the same pruned network obtained in (b), however, trained from scratch.
Network
Accuracy (%)
MACs
Reduced MACs
Paramters
Reduced paramters
VGGish_Noe Net
64.69
903M
55M
(a) Pruned VGGish Net
64.30
253M
72%
26.82M
52%
(b) Pruned "GGish.Net-scree
61.40
DCASE21NNG (a) Pruned DCASE21 Net
48.58
286M
46246 27906
48.18
164M
43%
40%
(b) Pruned DCASE21 Net-scratch
46.80
VGGNet-MNIST
99.49
329M
15M 0.18M
(a) Pruned BUGGNe--MNIST
99.10
3.33M
99%
99%
(b) Pruned VGGNot-MNIST-ssracch
98.97
percentage points when all convolutional layers (CI.3: CI to C3) are pruned at p - 25% shown in Figure bb).
p = 50%, the accuracy drop across various pruned networks is less than percentage points compared that of the unpruned network for both VGGish and DCASE21 as shown in Figure B/a b). For VGtish_Net, the number of MACs are reduced by 75 percentage points and the parameters are reduced by 55 percentage points, when all convolutional layers (CI 66; CI to C6) are pruned at = 50% shown in Figure ((... For DCASE21 Net, both the MACs and the parameters reduced by 75 percentage points when all convolutional layers (C1 3: CI to C3) are pruned at 50% shown in Figure b(b).
At p 75%, the accuracy drop across various pruned networks is less than 5 percentage points and 10 percentage points compared to that of the unpruned network for both VGGish_Nee and DCASE2I_Ne respectively as shown in Figure Gia, b). The accuracy of the pruned networks de- grades further at 90% and the drop in accuracy for VGGish Net and DCASE21 Net is 10 percentage points and 20 percentage points respectively, when all layer are pruned p = 90%. On the other hand, both the MACs and the parameters are reduced significantly by more than 75 percentage points when all convolutional layers of VGGish Net and DCASE21 Net are pruned at p (75%, 90%} as shown in Figure 6(a, b).
In general, the MACs, the parameters and the accuracy decrease when various convolutional layers are pruned from 25% to 90% pruning ratio. The accuracy of the pruned DCASE211 _Net reduces significantly from 0.2 percentage points to 20 percentage points compared to that of the unpruned network, when all layers are pruned with 25% to 90% pruning ratio. This might be due to the smaller network size of the DCASE21 1 Net, where eliminating large number of parameters at high pruning ratio results in fidttfiitting problem due to insufficient parameters.
VGG-16 for image classification For VGG-16, the accuracy of the various pruned networks is reduced by less than 0.5% percentage points compared to that of the unpruned VGG-16 shown in Figure c)). The number of MACs and the parameters are reduced from approximately 40 percent- age points 99 percentage points when pruning ratio across all convolutional layers (C1_13: CI to C13) varies from 25% to 90% respectively.
Next, Table D compares the accuracy, MACs and parameters in the unpruned network with that of (a) the pruned network having important filters obtained using the proposed pruning method as an initialisation before fine-tuning process, (b) the same pruned network as obtained in (a) and trained from scratch with random initialization. For comparison, we choose the pruned network in (a) which gives maximum drop of 0.5 percentage point in accuracy compared to that of the unpruned network. For various networks, the parameters and the MACs are reduced significantly with a marginal drop in accuracy compared to that of the unpruned network. The accuracy of (a) the pruned network obtained from the proposed pruning method is better than (b) the same pruned network trained from scratch. This shows that is beneficial to train a large network, and then perform pruning to reduce the parameters rather than training the same size pruned network from scratch. For experimental analysis of VGG-16 network on CIFAR-10 10 and ResNet-50 network on CIFAR- 10, please refer Appendix and Appendix A.2.2 respectively.
8

---

Under review as a conference paper at ICLR 2023
Unpruned
Proposed Operator torm based pouning
Eitry-wise 1y-orrm based pruning
witry-wise gecmetric median (GM) pruning
Accuracy
8 85 8
Accumenyy
Accuraccy
22
to 15
90
Pruming ratio (p)
Pruning ratio (p)
Pruming ratio (p)
VGGish Net DCASE-18
DCASE221 Net DCASB-20
(c) VGG-16 MNIST
Figure 7: Accuracy obtained using the pruned network when all layers are pruned at different prun-
ing ratio using entry-wise I--rr method, geometrical-median (GM) method and the proposed operator norm pruning method across various networks.
Comparison with other methods: Figure 7 compares the accuracy of the proposed operator norm pruning method with that of the entry-wise I I-nrr and the geometrical median (GM) based meth- ods, when all layers in the unpruned network are pruned at different pruning ratios. For VGGish_Ne and DCASE21 _Net, the accuracy obtained using the proposed operator norm based pruning method better than that of the entry-wise norm methods particularly when a large number of filters (p = +9%% are pruned from the network as shown in Figure a b). Moreover, the pruned net- work obtained using the proposed operator norm based pruning method recovers faster regain the accuracy during the fine-tuning process compared to that of the other pruning methods shown in Appendix: Figure X
a, b) at different pruning ratios. For VGG-16, the accuracy obtained using the proposed operator norm pruning method is similar to the other methods as shown in Figure 7 c) and recovers faster compared to that of the other pruning methods as shown in Appendix: Figure c) at different pruning ratios.
For VGG- on CIFAR-IO, we find that the accuracy obtained using the proposed pruning method slightly better than that of the entry-wise norm methods various pruning ratio (Appendix: Figure I0.. and the convergence speed is also more or less similar to the entry-wise norm methods (Ap- pendix: Figure T.. Similar analysis is also observed for ResNer-50 network CIFAR-10 dataset (Appendix: Figure 15p. This suggests that the proposed pruning method performs similar to or better than the existing entry-wise norm based pruning methods across different CNNs designed for audio and image classification, hence showing better generalization. This shows that selecting important filters based on the operator norm of the filters in producing output advantageous compared to considering entry-wise norm of the filters due to similar or better performance across various CNNs.
In comparison the active filter pruning methods including HRank (Lin et al. (020 ). Energy- aware pruning (Yeom et al. (22Th), LL Ll-slimming (Liu et al.) (217)), and GAL-0.1 Lin et al. (019) as given in Appendix: Table 3] the proposed pruning method reduces similar number of parameters and can achieve accuracy within R 1 percentage point as given by the active filter pruning methods, with an advantage of no requirement of dataset, feature maps, regularization techniques or knowl- edge distillation techniques while identifying pruned set of filters.
CONCLUSION
This paper presents passive filter pruning method which reduces the computational complexity and the memory of the unprumed CNN significantly at marginal drop in accuracy. The proposed pruning method reduces similar number of parameters and MACs at a maximum drop in accuracy of less than 1 percentage points compared to the existing active filter pruning methods without involving any dataset or regularization methods while eliminating filters from the unpruned network. We find that the proposed pruning method gives similar or better accuracy at various pruning ratio compared to that of the existing entry-wise norm methods across different CNNs designed in audio and image domains, hence showing the generalization ability across different domains as well. In future, we would like improve the performance of the pruned network by incorporating data augmentation techniques to improve the fine-tuning process.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Misha Denil, Babak Shakibi, Laurent Dinh, M.A. Ranzato, and Nando De Freitas. Predicting param- eters in deep learning. Advances Neural Information Processing Systems, pp. 2158-2166, 2013.
Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Informa- tion Processing Systems, pp. 1269-1277, 2014.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural
networks. In International Conference on Learning Representations, 2019.
Yonatan Geifman. Pre-trained vgg- "gg-16 on cifar-10, online: https:///gtttbb.co/////////////e wttp://tttt...oooeeeeeeeeo cifar-vqq
Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, Jianfei Cai, et al. Recent advances in convolutional neural networks. In Pattern Recognition, volume 77, pp. 354-377. Elsevier, 2018.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Eie: Efficient inference engine on compressed deep neural network. In ACM SIGARCH Computer Architecture News, volume 44, pp. 243-254 254.. ACM New York, NY, USA, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340-4344 2019.
Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson. CNN Architectures for large-scale audio classification. In International Con- ference Acoustics, Speech and Signal Processing (ICASSP). 2017.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neu- ron pruning approach towards efficient deep architectures. In arXiv preprint /rXiv::607.0000 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. In Technical Report. Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. AT&T Labs [Online]. Available: http:/yann.lecum.com www..//....//ex///missstpo 2, 2010.
Hao Li, Asim Kaday, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient ConvNets. In International Conference on Learning Representations, 2017.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. HRank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF Conference Computer Vision and Pateern Recognition, pp.
2020.
Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann. Towards optimal structured enn pruning via generative adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2790-2799, 2019.
10

---

Under review as a conference paper at ICLR 2023
Shuying Liu and Weihong Deng. Very deep convolutional neural network based image classification using small training sample size. In 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pp. 730-734. IEEE, 2015.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2734,2744,2 2017.
Roi Livni, Shai Shalev-Shwartz, and Ohad Shamir. On the computational efficiency of training neural networks. In Advances in Neural Information Processing Systems, pp. 855-863, 2014.
Jian-Hao Luo and Jianxin Wu. An entropy- -basod pruning method for CNN compression. In arXiv preprint vrxxiv:706.0579/1 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. ThiNet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5058-5066, 2017.
Jian-Hao Luo, Hao Zhang, Hong- Yu Zhou, Chen-Wei Xie, Jianxin Wu, and Weiyao Lin. ThiNet: pruning CNN filters for thinner net. IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 41, pp. 2525-2538, 2018.
Irene
Toni Heittola, Annamaria Mesaros, and Tuomas Virtanen. Low-complexity acoustic scene classification for multi-device audio: Analysis of DCASE 2021 Challenge systems. DCASE Workshop, 2021.
Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. A mulli--dviie dataset for urban acoustic scene classification. In Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop, pp. 9-13, November 2018
Adam Polyak and Lior Wolf. Channel-level acceleration of deep face representations. In IEEE Access, volume 3, pp.
2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances Neural Information Processing Systems, pp. 2074-2082. 2016.
Jianbo Ye. Xin Lu, Zhe Lin, and James Z Wang. Rethinking the smaller- wew-nooooss--iooomattiii informatioe assumption channel pruning of convolution layers. In International Conference on Learning Representations, 2018.
Seul-Ki Yeom, Kyung-Hwan Shim, and Jee-Hyun Hwang. Toward compact deep neural networks via energy-aware pruning. arXiv preprint arXiv:2103.1088. 2021.

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
Recovered maximum accuracy using the proposed operator based pruning
Proposed openitor norm pruning
Recovered maximum accuracy using the enity-wase I -norm based pruning
Entry-wise 1 n-morm based pruning
Recovered maximum accurncy using the entry-wise goometric median (GM) pruning
Entry-wisc GM pruning
25%
50%
75%
p 90%
Accuracy
8
Accunrys
Accuracyy9)
x x
8
100
40
103
10
20
160
Epochs
Epochs
Epochs
Fpochs
(a) VGGish on DCASE-18
-5%
P
p=
p 90%
X 8 x
8 8
Accurecy(9)
Accuracynte
Acturecy
B
Accurasy 8 H
18
16
20
00
100
80
40
100
Epochs
Epochs
Epochs
Epochs
(b) DCASE21 ont on DCASE-20
p 25%
50%
75%
90%
Aucurucy())
Rccurcyy8y
Accormty(N)
Accuracy150
30
20
401
80
180
20
41
60
100
20
80
180
20
60
150
Epochs
Epachs
Epochs
Epochs
VGG-16 MNIST
Figure 8: Convergence plots showing regain in the accuracy during fine-tuning of the pruned net- work when the pruned network is obtained by pruning all convolutional layers different pruning ratios.
A.1 EXPERIMENTAL SETUP AND ANALYSIS: VGG-16 ON CIFAR-10 10
A.1.1 EXPERIMENTAL SETUP
Unpruned network: We use a publicly available pre-trained VGG-16 network trained on CIFAR- unpruned network. The unpruned architecture is based on the VGG- 16 (Simonyan & Zisserman 2015)) with adaptation to CIFAR- 10 dataset based on (Liu & Deng 2015). The VGG-16 has 5M parameters and requires 329M MACs per inference and the pre- trained VGG-16 gives 93.58% accuracy for the CIFAR-10 testing dataset. The unpruned network has 13 convolutional layer, denoted as C1 to C13.
10 dataset (Geifman)
To perform fine-tuning of the pruned network, we re-train the pruned network for 100 epochs with similar conditions such as same optimizer as used training the unpruned network (Geifman).
Other methods used for comparison: For comparison, we use passive filter pruning methods such
as entry-wise Ii-orrm (Li et al. (2017) and entry-wise geometric median (GM) (He et al.. (2019) )-
We also compare the proposed pruning method with the existing active filter pruning methods in- cluding HRank (Lin et al. (2020)) Energy-aware pruning (Yeom et al. (202T)), Ll-slimming (Liu)
al. (2017)), and GAL-0.1 (Lin et al. 2019)). A brief overview of the active filter pruning methods used for comparison given below,
HRank: This method opts three steps obtain pruned network. 1) A set of feature maps generated for a given filter using a set examples. 2) Rank of the feature maps is computed and an average rank computed across feature maps of various examples is used criterion quantify
12

---

Under review as a conference paper at ICLR 2023
filter importance. 3) The filters with low average rank are eliminated and a fine-tuning procedure is opted to compensate the performance loss.
This method uses set of input data to generate feature maps for given convolu- tional layer and then compute energy of each feature map by computing nuclear norm (sum of all singular values) of each feature map. A feature map with low energy is pruned and then fine-tuning of the pruned network is performed.
Energy-aware:
L1-slimming: This method uses three steps to obtain prumed network: (1) Trains the underly- ing unpruned network with channel-level sparsity regularisation, (2) Eliminate the channels (feature maps and corresponding filters) having small value of a scaling coefficient (7) in batch normalisa- tion layer, and (3) Fine-tune the pruned network.
GAL-0.1: This method applies knowledge distillation train the pruned network with 4- regularisation on the soft mask associated with channels (feature maps) to mimic the unpruned network by aligning their output using generative adversarial learning (GAL) from the unpruned network to the pruned network. Subsequently, the pruned network is fine-tuned after eliminating the unimportant feature maps and corresponding filters.
A.1.2 PERFORMANCE ANALYSIS
25% pruning ratio
50% pruning ratio
"Green" color: Accuracy "Red" color: MACs/Parmmters
75% pruning ratio
90% pruning ratio
100
100
80
Accuracy 199)
Paramters (%)
80
88
Accuracy 1961
MACs190
40
20-
30
C1L
67,13
13,33
C1_33
69,13
C5
Pruned layers
Pruned layers
(a) VGG-16 on CIFAR-10 dataset
Figure 9: Accuracy, MACs and parameters across different pruned networks for VGG-16 network trained on CIFAR-10, 10, when different subsets of convolutional layers are pruned at different pruning ratios. Here, "CA B" as "pruned layers" means that convolution layers from A to B are pruned.
Figure 9 shows the number of MACs and the number of parameters in the pruned network obtained after pruning various subsets of convolution layers at different pruning ratios from the unpruned network. By varying p from 25% to 70% across convolution layers from C5 to C13, the number of parameters are reduced from 40% to 90%, and the MACs are reduced from 25% to 70% with an accuracy drop from 0.25 to 3 percentage points respectively. Pruning all convolutional layers = 25% results in approximately 45% reduction in noth parameters and MACs at marginal drop accuracy compared to that of the unpruned network. On the other hand, pruning all convolu- tional layers at p 90% reduces parameters and MACs significantly, however accuracy drop is approximately 20 percentage points compared the unpruned network. Therefore, the pruning ra- tio can be chosen according to the requirement whether the underlying resources (computation and parameters) are primarily important or the accuracy.
Next, Table 2 compares the accuracy, MACs and parameters in the unpruned network with that of (a) the pruned network having important filters obtained using the proposed pruning method as an initialisation before fine-tuning process, (b) the same pruned network as obtained in (a) and trained from scratch with random initialization. For comparison, we choose the pruned network in (a) which gives maximum drop of approximately 0.5 percentage point in accuracy compared to that of the unprumed network. The parameters and the MACs are reduced by 34% and 78% respectively with marginal drop in accuracy compared to that of the unpruned network. The accuracy of (a) the pruned network obtained from the proposed pruning method is better than (b) the same pruned network trained from seratch.
13

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of accuracy, the MACs and the parameters among the (a) unpruned, (b) the pruned network obtained using the proposed pruning method and the same pruned network obtained (b), however, trained from scratch.
Network
Accuracy (%)
MAC'
Reduced MACs
Parameters
Reduced purumeters
VGG-16 CIFARIO
93.56
329M
15M
(a) Pruned VGG-16 16 CIFARIO
93.00
217M
34%
3.29M
78%
(b) Pruned VGG-16 on CIFARIO-scratc
86.02
Comparison with entry-wise norm methods: Figure T0 and Figure 11 compares accuracy and convergence during fine-tuning of the pruned network among the proposed pruning method, entry- wise ln-rrmm and entry-wise GM pruning methods at different pruning ratio. In general, we find that the accuracy obtained using the proposed pruning method is similar or marginally better (0.5 to percentage points) than that of the entry-wise norm methods particularly at high pruning ratio as shown in Figure 10 with similar convergence speed (= 0 to 20 epochs) in recovering the accuracy during fine- mee-ttning as shown in Figure
95
Unprumed
Accuracy(80)
Proposed Operator norm based pruning
Entry-wise -norm based pruning
80
Entry-wise geometric modian (GM) based pruning
75
70
Pruning ratio (p)
Accuracy obtained using the pruned network when each layer of VGG-16 network on CIFAR-10 dataset is pruned at pruning ratio using entry-wise 4-norm method, geometrical- median (GM) method and the proposed operator norm pruning method.
Figure 10:
Recovered maximum accuracy using the proposed operator norm based pruning
Proposed operntor based pruning Entry-wise Iy-norm based pruning
Recovered maximum accuracy using the entry-wise l -nomm based pruning
Recovered maximum accuracy using the entry-wise geometric median (GM) pruning
Entry-wise GM pruning
25%
p
75%
90%
Accuracy
Accuracy(1t)
8
40
30
20
100
40
80
20
90
80
100
(8) VGG-16
Figure 11:
Convergence plots showing regain in the accuracy during fine-tuning of the pruned VGG-16 network on CIFAR-10 when the pruned network is obtained by pruning each convolutional layers at p pruning ratios.
Comparison with existing active filter pruning methods: Table 3 compares the proposed pruning method with existing active filter pruning methods. In general, pruning filters from the unpruned network using only filters result in drop in accuracy approximately percentage points compared to pruning filters by involving a dataset, using feature maps, regularization or knowledge distillation (KD) at similar reduction in parameters.
In comparison to feature map based pruning methods such as HRank and Energy- aware as given in Table 3 we find that the proposed filter based pruning method can achieve similar performance compared to that of feature map based pruning method, however the proposed method requires more number of parameters and MACs.
14

---

Under review as a conference paper at ICLR 2023
Table 3: Comparison of accuracy, the MACs and the parameters among different existing methods for VGG-16 on CIFAR-10 dataset. Please note that the unpruned network used for each comparison methods has different training procedure and unpruned accuracy. Therefore, we compare the pa- rameters, MACs and accuracy obtained from the pruned network obtained from each method. Here "FT" is fine-tuning, "KD" is knowledge distillation.
Method
Data Pruning
KD
Accurucy (%)
Parameters
MACs
HRank (Lin 2020
X
93.43
2.50M
146M
Ours (p
90%, C7.13)
X
92.00
1.37M
159M
Ours 75%. C7_13)
X
92.46
2.18M
172M
Ours (p 75%. C9 C9.13)
X
X
93.25
4.18M
227M
Ours (p 50%, C9_13)
X
93.50
6.60M
251M
Energy-aware pruning Yoom et 2021
X
93.48
2.86M
104.67M
Ours (p = 75%, C7.13]
X
92.46
2.18M
172M
Ours (p 50%, C9.13)
X
93.50
6.60M
251M
L1-shimming [Läu
2017)
x 93.80
2.30M
391M
Ours (p
50%6,
X
93.50
6.60M
251M
GAL-0.1
Lin al. 2019)
/ 90.78
2,67M
17189M
GAL-0011
/
93.42
2.67M
17189M
ours = 75%, C7 13)
X
92.46
2.18M
172.29M
In comparison L1-slimming method as given in Table the proposed pruning method results in similar accuracy and requires 1.6 times less MACs, however with 3 times more requirement of parameters. The LI-slimming method prunes filter based on y parameters the batch normalisa- tion layer and hence, solely dependent on batch normalisation layer. Moreover, the LI-Slimming method requires training of the unpruned network identify which feature maps need to be pruned. Assuming that there exists already pre-trained network which is to be pruned, the Ll-slimming method would require extra computations to obtain the pruned network. On the other hand, the pro- posed pruning method can directly use pre-trained filters to obtain the pruned network. Although, the proposed pruning method result times more parameters that require more memory size, yet the quantization (float32 to INT8) of the pruned network can be performed to reduce the pruned network size by times at marginal (< 0.15 percentage point) drop in accuracy. Therefore, the proposed pruning method is advantageous over the L1-slimming pruning method providing similar accuracy at reduced MACs.
The GAL-0.1 (without fine-tuning (FT)) method is closely related to the proposed pruning method, where knowledge distillation step to train the pruned network can be considered as equivalent to the fine-tuning of the pruned network in the proposed pruning method. In contrast GAL-0.1 without FT, the proposed pruning method gives better performance as given in Table 3 However, fine-tuning the pruned network obtained using (GAL-0.1 without FT) results in approximately percentage points better performance compared to that of the proposed pruning method.
The proposed pruning method simple and advantageous in terms of obtaining pruned network using filters only compared to involving dataset based pruning methods which results in 1 percentage points better accuracy, however with relatively more efforts, resources (memory, extra parameters such as learning soft mask) compared to the proposed pruning method.
15

---

Under review as a conference paper at ICLR 2023
A.2 EXPERIMENTAL SETUP AND ANALYSIS: RESNET-50 ON CIFAR-10
A.2.1 EXPERIMENTAL SETUP
We perform preliminary analysis on ResNet-50 using CIFAR-10 10 dataset. We use a pre-trained ResNet-50 with ImageNet (He et al. (016) weights followed by a global average pooling, fully connected layer and classification layer train unpruned network for CIFAR- 10. The architecture used for experiments is shown in Figure 12 consisting of various stages and different blocks. The unpruned network trained for 300 epochs with stochastic gradient descent (SGD) optimizer and cross entropy loss function. The unpruned network has 24M parameters, 99M MACs andd gives 83.377 33.37% accuracy on CIFAR-I 10 test dataset with (32 x 32 3) shaped input.
Unpruned network:
Stage?
Stage 2
Stage 3
Stage 4
Stage 5
(32 x X3)
Input
Conv2D
ReLU
Max pooling
Conv Block
Block
Conv Block
Block
Canv Block
Block
Conv Block
Block
Pooling
Average
Dense
Classifization
oulpul
BN
(a) ResNet-50 archilecture classifier
ReLU
ReLU
F(x)
F(w) + a(x)
F(x)
F(#)
Conv2D
Conv2D 1) BN
BN
branch
G(*)
Conv2D BN ReLU
Conv2D 3)+ BN + ReLU
Conv2D x 1)
residual branch
branch
BN
Conv2D BN ReLU
Conv2D BN ReLU
branch
(b) Identity (ID) Block
(c) Convolutional (Conv) Block
(a) Architecture to classify CIFAR-10 dataset using ResNet-50 (stage 1 to stage 5) which comprises of (b) identity blocks and (c) convolutional blocks, followed by global average pooling layer, dense and a classification layer. "BN" denotes batch normalization. "ReLU" denotes recitified linear unit activation function and Conv2D is convolutional layer with 2D filters.
Figure 12:
Pruning layers in ResNet-50: In the current preliminary analysis for ResNet-50,, we consider con- volutional layers having (3 3) filters for pruming since they contain more parameters and computa- tions compared to that of the convolutional layers with (1 1) filters. The proposed pruning method can be applied directly to the convolutional layers in the residual branch as well. For this, the same number of filters from the convolutional layer in the residual branch should be pruned as that of the convolutional layer from the main "branch c" as shown in Figure c2) This is to ensure the same dimensionality while performing addition operation between the main branch output F Fzx) and the residual branch output G(z).
We analyse performance of the pruned network when various convolutional layers with (3 x 3) filters are pruned from (a) stage to stage 5, (b) stage 2 to stage and (c) stage to stage 5 at different pruning ratios.
16

---

Under review as a conference paper at ICLR 2023
A.2.2 PERFORMANCE ANALYSIS
Figure 13 shows accuracy obtained at different pruning ratio after pruning convolutional layers from different stages in ResNet-50. Pruning convolutional layers from stage 1 to stage 5 result in more reduction in accuracy compared to that of stage 3 to stage 5 at various pruning ratios. Pruning convolutional layers from stage to stage result in accuracy drop from -0.15 to 2.5 percentage points, reduces parameters from 17% to 60% and reduces MACs from 14% to 54%, when p is varied from 25% to 90% as showing in Figure 14
Accuracy (%6)
75
70-
Stage
85
Stage
Stage
50
50
90
Pruning ratio (%)
Figure 13:
Accuracy when each convolutional layer having (3 x 3) filters from various stages in ResNet-50 are pruned at different pruning ratio.
Accuracy
x
Parameters
Unpruned accuracy
Paramtter MACs (9)
100
23.4
Accuracy 9%5
60
00.0
40
40.0
20.0
75
0.0
Pruning Ratio (%)
Figure 14: Accuracy, MACs and parameters obtained after pruning each convolutional layers hav- ing (3 3) filters from stage 3 to stage 5 at different pruning ratio.
Next, Table 4 compares the accuracy, MACs and parameters in the unpruned network with that of (a) the pruned network having important filters obtained using the proposed pruning method as an initialisation before fine-tuning process, (b) the same pruned network as obtained in (a) and trained from scratch with random initialization. For comparison, we choose the pruned network in (a) which gives a maximum drop of approximately 0.5 percentage point in accuracy compared to that of the unpruned network. The parameters and the MACs are reduced by 33% and 30% respectively with a marginal drop in accuracy compared to that of the unpruned network. The accuracy of (a) the pruned network obtained from the proposed pruning method is significantly better than (b) the same pruned network trained from scratch. This suggests that it is useful to obtain smaller size network from relatively large-size network compared to obtaining similar small architecture trained from scratch.
Table 4: Comparison of accuracy, the MACs and the parameters among the (a) unpruned, (b) the pruned network obtained using the proposed pruning method and the same pruned network obtained in (b), however, trained from scratch.
Network
Accuracy (%)
MACs Reduced MACs Parameters
Reduced parameters
RcsNet-50 on CIFARIO
83.37
99M
24M
(a) Pruned 50 on CIFARIO
83.08
70M
30%
16M
33%
(b) Pruned ResNet-50 on CIFAR 10-scratch
66.45
17

---

Under review as a conference paper at ICLR 2023
Recovered maximum accuracy using the proposed operator norm based pruning
Proposed operator norm based pruning
Recovered maximum uccuricy using the entry-wise based pruning
Entry-wise 4-mnom based pruning
Recovered maximum accuracy using the entry-wise geometric median (GMM pruning
Entry-wise GM pruning
25%
50%
75%
90%
ArturaryrSu 12
Accuracy
42
80
30
60
100
100
80
Epochs
Epochs
Epochs
Epochs
(+) ResNet-50 on CIFAR-10
Figure 15: Convergence plot during fine-tuning process of pruned ResNet-50 (stage3 to stage 5) on CIFAR-10 at different pruning ratio.
In comparison to existing passive filter pruning methods such as 4nomr and geometric median (GM), the proposed pruning method gives similar performance (# 0.5 percentage points) and recov- ers accuracy at similar epochs (#: 20) at different pruning ratio as shown in Figure T.5
18


Reviews:
[{"review_id": "tlf-NkmSO0", "reviewer": "Reviewer_qjdr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a weight-based pruning criterion that considers the contribution of the filters in yielding outputs. The importance score is computed using the SVD on the flattened filters and their rank-1 approximation. The experiments are conducted on vanilla convolutional neural networks without residual connections.", "summary_of_the_review": "The novelty of this work is incremental, while the experimental results are very weak and not impressive.", "strength_and_weaknesses": "Strength\n- The proposed method that computes the contribution of filters in producing output is intuitive.\n- The experiments for audio-scene classification, which have been less explored in pruning papers, look interesting.  \n- Figures 1–4 are very helpful for understanding the proposed method; especially, Figure 2 clearly describes the differences between the norm-based filter pruning and the proposed method.\n- The ideas in the paper are sufficiently simple for people to build on.\n\nWeakness\n- The experimental validation is very weak. For visual classification, the authors performed VGG-on-MNIST experiments solely, which cannot fully support the superiority and applicability of the proposed method. In contrast, recent works (e.g., Molchanov [2019]., Lin [2020], Sui [2021]) always include the experiments with ResNets on ImageNet as well as CIFAR to show the merits of their methods. I would highly recommend the authors to add empirical support on ImageNet and CIFAR with various architectures including ResNets. In particular, how can the proposed method be applied/extended for residual blocks?\n- Furthermore, the baselines seem very weak: only the comparison to L1-norm and geometric median pruning (which are classical methods nowadays) was conducted. It would be helpful to include additional recent baselines (e.g., L1 Slimming (Liu et al., 2017), Polarization Regularization (Zhuang et al., 2020), Deep Hoyer (Yang et al., 2019)) to demonstrate the superiority of this work. \n- I was not able to identify a clear superiority of the proposed method over the baselines, especially in the VGG-MNIST experiments. In addition, in the DCASE21-DCASE-20 case in Figure 7(b), the results of the operator norm method and those of the GM-based one look very similar at the pruning ratio of 25% and 50%; I think the performance drop at the large pruning ratio (75%, 90%) may be unacceptable and thus the comparison at these ratios may be useless.\n- As the authors describe the merits of structured pruning over unstructured pruning include actual inference speedup, it would be good to measure the latency on high-end and/or edge GPUs.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ar2ypWEq3w", "reviewer": "Reviewer_u6V5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a passive filter pruning method worked on scene classification and image classification tasks. In detail, the authors aim to compute filter importance with the norm of each convolution layer and leverage singular value decomposition to compute a rank-1 approximation of the target channel. The authors provide experiments with TAU Urban Acoustic Scenes 2020 and MNIST datasets.", "summary_of_the_review": "In this paper, the authors propose a passive filter pruning method worked on scene classification and image classification tasks. In detail, the authors aim to compute filter importance with the norm of each convolution layer and leverage singular value decomposition to compute a rank-1 approximation of the target channel. The authors provide experiments with TAU Urban Acoustic Scenes 2020 and MNIST datasets. The authors should clearly explain the novelty of this paper and conduct experiments with imagenet dataset.", "strength_and_weaknesses": "Strength:\n1. This paper is easy to read.\n\n2. The authors propose a norm-based filter pruning method.\n\n3. The authors conduct experiments on scene and image classification tasks.\n\nWeaknesses:\n1. The novelty of this paper is limited. This paper is based on the norm of channels, and there have been many similar novel papers in recent years.\n\n2. The authors should give experiments with ImageNet dataset.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_ltgh3956Y8", "reviewer": "Reviewer_mkSs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, it is stated that the norm of a convolution filter is insufficient as a criterion for judging how much the filter affects the output of the layer. Instead, the authors state that the operator norm, which indicates how much the filter affects the output during operation, is a good\ncriterion and can be obtained through SVD.", "summary_of_the_review": "Although it is a well-written paper, the evidence supporting the proposed technique needs to be clearer. In addition, it is necessary to compare and analyze the performance through additional experiments (refer to the weakness section).", "strength_and_weaknesses": "Strength\n- The paper is written clearly and easy to understand.\n\nWeaknesses\n-  As claimed in the paper, in order to judge the influence of the filter only from the maximally stretched direction point of view and ignore all other directions, it seems the largest singular value in SVD should be much larger than other singular values. That is, in Figure 1, 에서 Y=FX=(σ_1 u_1 w_1^T )X+(σ_2 u_2 w_2^T )X≈(σ_1 u_1 w_1^T )X seems to hold, but there is no explanation for this.\n- This paper proposes a passive filter pruning method, but there is no explanation or comparison about the advantages compared to the active method.\n- In the image classification experiment, the MNIST dataset and the VGG16 model alone are insufficient to guarantee the performance of the proposed method.\n- In Figure 6, the criteria for determining the pruned layers are not specified, and I am not sure what the meaning of comparing only some layers is pruned.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject