# Activation-Based Pruning of Neural Networks
## 5 Results and Discussion
### Activation-vs. Magnitude-Based Pruning  
In this section, we will conduct a comprehensive comparison between activation-based and magnitude-based pruning across various evaluation metrics. We will first analyze the impact of these pruning techniques on training, validation, and test accuracy. Furthermore, we will assess the computational efficiency by comparing the FLOPS requirements after network pruning. Additionally, we will examine the test results obtained after retraining the pruned network. Lastly, we will evaluate the rank of each hidden layer both before and after retraining the pruned networks. Table 4 demonstrates that activation-based pruning was able to generate equivalent results to magnitude-based pruning. Activation-based pruning yielded better training accuracy compared to magnitude-based pruning, while demonstrating slightly lower validation accuracy and equivalent test accuracy. Activation-based pruning achieved network pruning comparable to magnitude-based pruning while preserving equivalent accuracy. In Tables 5-7, model M represents magnitude-based pruning, while the other entries correspond to activation-based pruning. Within activation-based pruning, models G1 to G8 underwent global pruning, while models L1 to L4 underwent local pruning. Table 5 demonstrates the floating point operation per second (FLOPS) for each model. The explanation for the model symbols can be obtained from Tables 2 and 3. We demonstrate the FLOPS using kFLOPS, denoted as \(1/1000\)th of FLOPS, commonly referred to as KiloFLOPS. For a fully connected feedforward network, the number of floating-point operations considering only non-zero weights was computed as \(\sum_{l=1}^{l}(2\times W_{\text{nz}}^{i})\), where \(W_{\text{nz}}^{i}\) is the number of non-zero weights of the \(i\)-th layer and \(l\) is the total number of hidden layers of the network. We multiplied \(W_{\text{nz}}^{i}\) by 2 inside the summation, as each weight will be involved in one multiplication and one addition with the input. The values G4, G8, L1 to L4 in the Pct. Reduction column signify better performance relative to magnitude-based pruning. Local activation-based pruning outperformed global activation-based pruning and magnitude-based pruning, in reducing the computational cost of the network, as assessed through the FLOPS computation. Table 6 demonstrates the training, validation, and testing accuracy achieved after retraining the pruned network. The training accuracy for models G1 to G8 was comparable to model \(M\). However, the training accuracy of models L1 to L4 was lower than that of model M. The validation and test accuracy of models G1 to G8 and L1 to L4 was comparable to model \(M\). For top-1% accuracy, the validation and test accuracy for model \(M\) were much higher than for the other models. For top-5% accuracy, models G3, G4, L2, L3, and L4 performed much better than model \(M\). Our observations indicated that, in certain instances, magnitude-based pruning may outperform activation-based pruning. This disparity in performance is primarily attributed to the fact that retraining a network \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Activation Type** & **Pruning Percentage** & **Training Accuracy** & **Validation Accuracy** & **Test Accuracy** \\ \hline Magnitude & & 80.00 & 91.79 & 92.35 & 88.72 \\ Activation-based & Neurons & 79.88 & 93.12 & 88.02 & 87.56 \\ Activation-based & Neurons \& Weights & 79.88 & 93.29 & 87.92 & 87.11 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Test result after pruning.