# Activation-Based Pruning of Neural Networks
## 5 Results and Discussion  
### Global Activation-Based Pruning  
Figures 2 and 3 demonstrate the training and validation accuracy for, Neurons and Neurons & Weights. The target pruning percentages were \(80\%\) and \(85\%\) with the number of pruning cycles specified as \(5\) and \(10\). Subsequently, Figures 4 and 5 demonstrate the training and validation accuracy for pruning percentages of \(80\%\) and \(85\%\) providing a different perspective of the same test run. The drop in accuracy at certain epochs in Figure 4 was due to the affect of pruning. Eventually, the network trains and recovers the loss in accuracy. Figure 2 illustrates that a higher percentage of pruning can be achieved by increasing the number of pruning cycles, particularly when employing importance score to Neurons. When the total pruning was specified as \(85\%\), we achieved \(79.25\%\) pruning in \(10\) pruning cycles and \(74.64\%\) pruning in \(5\) pruning cycles. We did not see evidence for a similar result when the importance score was Neurons & Weights. Figure 4 illustrates that using Neurons as the importance score resulted in a pronounced decrease in accuracy compared to Neurons & Weights. For the same number of pruning cycles, the pruning percentage attained for Neurons & Weights exceeded that achieved for Neurons. For \(10\) pruning cycles, the left figure achieved \(77.59\%\) pruning for Neurons & Weights and \(74.59\%\) for Neurons. We see the same results in the right-hand figure where total pruning specified was \(85\%\).  
Figure 4: Training accuracy for different pruning percentages: (**a**) 80%; (**b**) 85%.  
Figure 5: Validation accuracy for different pruning percentages: (**a**) 80%; (**b**) 85%.  
Figure 3: Validation accuracy. (**a**) Neurons. (**b**) Neurons & Weights.  
Figure 2: Training accuracy. (**a**) Neurons. (**b**) Neurons & Weights.  
Table 2 demonstrates the training and pruning of models with different sets of hyperparameters. The first row illustrates the results of training the network without any pruning. All subsequent rows specify the details of training and pruning the network based on the pruning number, pruning type, and target pruning percentage. The epoch column demonstrates that it took more epochs to prune and train the network compared to training the network without any pruning. After every subsequent pruning cycle, a greater number of epochs was required to recover the network's lost accuracy. All pruned models achieved training accuracy comparable to the standard model for top-1% accuracy and perform better in terms of validation and test accuracy. For top-5% accuracy, all pruned models demonstrated better results, except G1 and G6, which corresponded to the importance score of Neurons. For the same number of pruning and target prune percentages, we observed better results for the importance score of Neurons & Weights. Overall, G2, G3, G5, G7, and G8 performed better than the standard run in terms of pruning and accuracy achieved. However, the top-three results were G5, G7, and G8, which achieved considerable pruning and accuracy results. This demonstrated that the probability of achieving improved performance increased with more pruning iterations.  
While activation-based pruning prolonged the network training duration, it consistently led to better validation and test accuracy in almost all scenarios. Pruning resulted in a loss of accuracy, which was recovered by retraining the network. The pruned model can be used for prediction in place of the original model. As discussed in Section 3.2, a limitation of global activation-based pruning is its memory- and time-intensive nature, which may lead to poor scalability for larger networks.