## 4 Algorithms
### Greedy Layerwise Channel Pruning Using Nonparametric Statistics  
When viewing the input data \(\mathbf{x}\) as a random variable, neural networks are sometimes interpreted as Markov chains, where every hidden layer \(i\) defines the conditional probability \(p(\mathbf{h}_{i}|\mathbf{h}_{i-1})\)[21, 36]. Such a formulation motivates our observations that the effectiveness of neuron importance measurements relying on \(\mathbf{h}_{i}\) is dependent on the stability of input activation distribution \(\mathbf{h}_{i-1}\). In Section 3, we show that iteratively increasing the sparsity of the input activations of a given layer results in a monotonic decorrelation between the rank orderings of its output activations when using sparse versus dense input activations. Thus, we hypothesize that by allowing a given layer to adjust to these abrupt shifts in its input activation distribution \(\mathbf{h}_{i-1}\)_before_ pruning \(\mathbf{h}_{i}\), we can increase the effectiveness of the heuristics used to rank order neurons by importance. As such, we design an iterative channel pruning algorithm that greedily traverses the topology of a feedforward neural network, starting from the first hidden layer \(\mathbf{h}_{1}\) and ending at the final hidden layer \(\mathbf{h}_{L}\). Our layerwise iterative channel pruning algorithm is summarized in Algorithm 1. ```
1\(s:=\) sparsity target
2\(t_{p}:=\) number of evaluation steps for each layer
3\(\Delta_{p}:=\) mask update frequency
4 Output:\(\{\mathbf{m}_{i}\}_{i=1}^{L}\)
5\(\{\mathbf{m}_{i}\}_{i=1}^{L}\leftarrow\) initializeMassIsoOne()
6\(\{\mathbf{m}_{i}\}_{i=1}^{L}\leftarrow\) initializeImportanceEstimatesToZero()
7 for\(i\gets 1\) to \(\mathbf{l}\)do
8\(t\gets 1\) to \(t_{p}\)do
9 networkForwardPass()
10 for\(c\gets 1\) to \(C_{i}\)do
11\(\mu_{c}\leftarrow(\mu_{i,c}\cdot(t-1)+\|\mathbf{h}_{i,c}\|_{1})/t\)
12 end for
13 networkBackwardPass()
14
15 every\(\Delta_{p}\)
16 for\(c\gets 1\) to \(C_{i}\)do
17 if\(\mu_{i,c}<\) quantile\((\mu_{i},s)\)then
18\(m_{i,c}\gets 0\)
19
20 end if
21
22 end for
23
24 end for
```  
**Algorithm 1**Our proposed layerwise channel pruning algorithm, using per-channel \(\ell_{1}\)-norm of activations to measure importance. All channel masks \(\{\mathbf{m}_{i}\}_{i=1}^{L}\) are initialized to \(1\) and all importance measurements \(\{\mathbf{\mu}_{i}\}_{i=1}^{L}\) are initialized to \(0\). We update learned weights of all layers in the network \(\{\mathbf{w}_{i}\}_{i=1}^{L}\) at every step using backpropagation, but only update the mask \(\mathbf{m}_{i}\) for layer \(i\) at step \(i\). We first initialize the values of all masks \(\{\mathbf{m}_{i}\}_{i=1}^{L}\) to \(1\) and the values of all importance measurements \(\{\mathbf{m}_{i}\}_{i=1}^{L}\) to \(0\). Note that the mask for a given neuron \(m_{i,c}\) is a binary value that prunes the neuron when \(m_{i,c}=0\), and \(\mu_{i,c}\) is a non-negative value where \(\mu_{i,1}>\mu_{i,2}\) is interpreted as neuron one being more important than neuron two in layer \(i\). When pruning layer \(i\), we estimate the importance of each neuron \(c\) using the moving average of the \(\ell_{1}\)-norm of activations over channel \(c\) from \(\mathbf{h}_{i}\) over \(t_{p}\) steps. We evaluate our pruning mask every \(\Delta_{p}\) steps, where \(\Delta_{p}\leq t_{p}\), at which point we prune neurons according to the \(\mu_{i}\) using a pre-defined sparsity target \(s_{T}\). The entire network is pruned to the given target sparsity once all \(L\) layers are processed. As is standard practice, we do not prune visible activations (i.e., the input and output activations \(\mathbf{x}\) and \(\mathbf{y}\), respectively). In practice, we apply this procedure on a pre-trained network and continue to fine-tune our structured subnetwork for \(T\) more epochs after pruning. As we greedily increase the sparsity of the network layer-by-layer, we refer to this iterative pruning schedule as \"layerwise\" channel pruning.