## 3 Pruning Methodology
### Pruning Algorithm  
As stated earlier, at each pruning iteration, after selecting the set of unimportant filters using filter important estimation criteria, the pruning is carried out from the particular convolution layer in which network loss is minimal. This means we use a greedy search approach to find the optimal set of filters at each pruning iteration, which results in variable rate pruning for each layer. During pruning, in each pruning iteration, the loss estimation using the full training dataset is computationally expensive, therefore randomly sampled small subset of the training dataset is used for this purpose. Given a dataset \(\mathcal{D}\), \(\mathcal{D}^{\prime}\) is the randomly sampled small subset of this dataset. We denote \(\mathcal{W}^{\prime(i)}\) as the filter of the \(i_{th}\) layer after pruning, and \(N^{\prime}_{i+1}\), the number of filters after pruning. Now, the estimation of accuracy loss during each pruning step is denoted as \(Loss(\mathcal{D}^{\prime}|\mathcal{W}^{\prime})\). In each pruning iteration, we temporarily prune each layer and select the layer with minimal accuracy loss. The number of filters to be pruned in each step is calculated as \(N_{i+1}\times\alpha_{s}\), where \(\alpha_{s}\) is the step pruning rate. In order to avoid excessive pruning from the single convolutional layer, we limit the maximum pruning rate of a layer as \(\alpha_{max}\). For example, if \(\alpha_{max}=70\%\), at max \(N_{i+1}\times 70\%\) filters can be pruned from \(i_{th}\) convolutional layer. The filter magnitude and filter similarity score are two different criteria for measuring filter importance. In our paper, we take advantage of both criteria to get optimal accuracy of the model after pruning. We design the pruning algorithm in such a way that we can input the partition rate for desired pruning ratio into two criteria. The ultimate goal of network pruning is to reduce the FLOPs in the network. Therefore instead of filter pruning rate, our algorithm directly takes the FLOPs reduction rate (\(\mathcal{P}\)) as the desired pruning rate and pruning is carried out iteratively until the desired FLOPs reduction rate is achieved. If \(f_{flop}(\mathcal{W}^{\prime})\) is the total FLOPs of pruned network \(\mathcal{W}^{\prime}\), the FLOPs reduction rate over the original network is calculated as  
\[\mathcal{P}^{\prime}=1-\frac{f_{flop}(\mathcal{W}^{\prime})}{f_{flop}( \mathcal{W})} \tag{6}\]  
At first, unimportant, low-magnitude filters are pruned using norm-based criteria (e.g., \(l_{1}\)-norm), and in the next step, redundant filters are pruned using similarity-based criteria (e.g., Euclidean similarity). The reason behind pruning low-magnitude filters at first is that the similarity criterion is able to remove redundant filters even if their magnitude is large. If the pruning weight for magnitude criteria is \(w_{mag}\) among two criteria, a total of \(\mathcal{P}\times w_{mag}\) FLOPs reduction is achieved using magnitude criteria, and the remaining \(\mathcal{P}\times(1-w_{mag})\) FLOPs reduction is achieved using similarity criteria. Therefore according to the value of \(w_{mag}\) we can decide what portion of filters are removed using filter norm magnitude criteria or filter similarity criteria to achieve the target pruning rate. For example, if we set \(w_{mag}=0.5\), almost the same number of filters will be removed using each criterion. The detailed pruning steps of the proposed approach are summarized in Algorithm 1. Given the training dataset (\(\mathcal{D}\)) and pretrained model (\(\mathcal{W}\)), the algorithm iteratively prunes a certain percentage of filters from each selected layer until the target FLOPs reduction rate is achieved. As we partition pruning among two filter selection criteria, if the current FLOPs reduction rate is in the range from 0 to \(\mathcal{P}\times w_{mag}\) (Line 5), magnitude based pruning (Line 6-11) is used.