## 5 Experiments
### Evaluating Pruning Schedules and Importance Measures  
Here, we evaluate the importance of our layerwise iterative pruning schedule by comparing it to three alternative pruning schedules:  
1. **Training from scratch. ** Prior work has demonstrated that, in some cases, there is no need to implement a pruning schedule because pre-defined structured subnetwork architectures can be trained from scratch to match or surpass the performance of the original larger network [38]. As such, we evaluate our pruning algorithm against this baseline. 2. **One-shot. ** We compare against the common \"prune then fine-tune\" strategy [20], where we train a fully connected baseline, and then prune the converged model to our target sparsity in one step before fine-tuning to heal the network. 3. **Stepwise. ** Unlike one-shot pruning schedules, which jump to the target sparsity in one step, stepwise pruning schedules iteratively increase the sparsity in the network over many steps throughout training. We benchmark against the state-of-the-art iterative pruning schedule proposed by [PERSON] and [PERSON] [23]. In Table 1, we report the performance of each of these algorithms with target sparsity levels of \(s_{T}=~{}50\%\) and \(s_{T}=~{}75\%\) across our discriminative and generative tasks. Each entry is the averaged result over three independent runs. We use the same learning rate schedule and optimizer for each experiment to ensure an even comparison. We use top-1 accuracy to evaluate image classification models, mean intersection-over-union (mIOU) to evaluate semantic segmentation models, and Frechet inception distance (FID) [35] to measure the difference between real images and images generated from our CycleGAN. Based on these metrics, we observe that our layerwise channel pruning algorithm performs stronger at more extreme levels of sparsity in a majority of cases. Next, we compare the performance of data-driven and data-free neuron importance measures when used to guide our greedy layerwise channel pruning algorithm. In Table 2, we compare the performance of our algorithm when using the \(\ell_{1}\)-norm of the channel activations against using the \(\ell_{1}\)-norm of the channel weights. Although prior work had experimented with using the \(\ell_{0}\)-norm as a data-driven importance measure [20], we do not compare against this as not all layers in our baselines are followed by a ReLU. We observe that using the \(\ell_{1}\)-norm of the output activation channels yields superior results than using the \(\ell_{1}\)-norm of channel weights in a majority of cases. Furthermore, we also observe that using the \(\ell_{1}\)-norm of channel weights to guide our greedy layerwise pruning algorithm often yields better results than other pruning schedules reported in Table 1.  
Figure 6: (a) Input, (b) Output (Baseline), (c) Output (P50%), (d) Output (P75%), (e) Output (P50%, W8A8), (f) Output (P75%, W8A8). We provide examples of images generated from CycleGAN given the same input (left). Here, P50% and P75% denote 50% and 75% channel pruning, respectively. We use “W8A8” to denote that the weights and activations have both been quantized to 8 bits. The results reported in Tables 1 and 2 suggest that we can increase the effectiveness of nonparametric data-driven pruning criteria by only pruning a given layer _after_ its preceding layer has been fully pruned, which supports our motivating hypothesis. Our experiments in Section 3 show that the rank ordering of channels for a given layer \(i\) is strongly impacted by the sparsity of the input activations \(\mathbf{h}_{i-1}\) to that layer. Equation-based pruning schedules such as [23] gradually introduce sparsity throughout the network with each step, as shown in Figure 5. Thus, at time \(t\), they approximate the neuron importance with respect to the target sparsity \(s_{T}\) using the neuron importance with respect to the current sparsity \(s_{t}\), where \(s_{t}\leq s_{T}\) and \(t\leq T\).