# Activation-Based Pruning of Neural Networks
## 5 Results and Discussion
### Activation-vs. Magnitude-Based Pruning
\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline
**No.** & **Pruning Type** & **Epochs** & \begin{tabular}{c} **Pruning** \\ **Achieved** \\ \end{tabular} & \begin{tabular}{c} **Training** \\ **Accuracy** \\ \end{tabular} & \begin{tabular}{c} **Validation** \\ **Accuracy** \\ \end{tabular} & \
\begin{tabular}{c} **Test** \\ **Accuracy** \\ \end{tabular} &
\begin{tabular}{c} **Top 1\% Accuracy** \\ \end{tabular} & \multicolumn{3}{c}{**Top 5\% Accuracy**} \\ \hline  & & & & & & **Training** & **Validation** & **Test** & **Training** & **Validation** & **Test** \\ \hline S & Standard & 46 & NA & 95.64 & 85.68 & 84.35 & 10.19 & 8.17 & 8.22 & 69.10 & 68.72 & 69.40 \\ L1 & Neurons & 88 & 79.88 & 93.12 & 88.02 & 87.56 & 10.22 & 11.08 & 10.83 & 64.33 & 63.37 & 64.09 \\ L2 & Neurons & 88 & 84.85 & 92.29 & 86.73 & 86.79 & 10.18 & 12.12 & 11.99 & 72.81 & 73.55 & 74.67 \\ L3 & Neurons \& Weights & 75 & 79.88 & 93.29 & 87.92 & 87.11 & 10.19 & 7.73 & 8.01 & 86.36 & 84.92 & 85.29 \\ L4 & Neurons \& Weights & 73 & 84.87 & 92.83 & 87.62 & 86.94 & 10.23 & 12.10 & 12.10 & 74.26 & 75.92 & 75.64 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Local activation-based pruning. subjected to magnitude-based pruning tends to utilize the full spectrum of network weights. In contrast, networks pruned using the activation-based method strive to preserve the low-rank structure of the hidden layers. A more-detailed explanation of this phenomenon is provided in the following paragraph. \begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline
**Model** & **Pruning Type** & **Epochs** & \begin{tabular}{c} **Training** \\ **Accuracy** \\ \end{tabular} & \
\begin{tabular}{c} **Validation** \\ **Accuracy** \\ \end{tabular} &
\begin{tabular}{c} **Test** \\ **Accuracy** \\ \end{tabular} & \multicolumn{3}{c}{**Top 1\% Accuracy**} & \multicolumn{3}{c}{**Top 5\% Accuracy**} \\ \hline  & & & & & & **Training** & **Validation** & **Test** & **Training** & **Validation** & **Test** \\ \hline M & Magnitude & 9 & 95.01 & 87.37 & 87.01 & 10.23 & 14.10 & 14.09 & 66.62 & 74.13 & 74.85 \\ G1 & Neuron & 21 & 95.24 & 86.55 & 85.99 & 10.10 & 7.57 & 7.82 & 70.90 & 62.45 & 62.52 \\ G2 & Neuron & 24 & 95.34 & 86.97 & 86.08 & 10.10 & 7.58 & 7.73 & 71.26 & 70.43 & 70.68 \\ G3 & Neuron & 24 & 95.29 & 88.73 & 88.50 & 10.18 & 10.17 & 10.51 & 79.96 & 84.55 & 84.22 \\ G4 & Neuron & 23 & 94.69 & 87.78 & 87.41 & 10.13 & 9.28 & 9.93 & 76.62 & 78.15 & 78.40 \\ G5 & Neuron \& Weights & 93.72 & 87.65 & 86.94 & 10.11 & 9.15 & 9.23 & 82.72 & 73.02 & 73.29 \\ G6 & Neuron \& Weights & 17 & 95.22 & 86.35 & 85.79 & 10.15 & 7.70 & 8.23 & 67.68 & 66.60 & 66.59 \\ G7 & Neuron \& Weights & 7 & 93.65 & 88.38 & 87.94 & 10.14 & 11.17 & 11.24 & 76.49 & 78.18 & 77.92 \\ G8 & Neuron \& Weights & 25 & 95.97 & 89.42 & 88.87 & 10.17 & 9.00 & 9.28 & 77.17 & 73.78 & 74.41 \\ L1 & Neuron & 29 & 94.73 & 87.05 & 86.30 & 10.20 & 9.32 & 9.20 & 63.53 & 61.12 & 61.67 \\ L2 & Neuron & 31 & 93.67 & 86.93 & 86.54 & 10.10 & 8.73 & 8.88 & 74.43 & 75.52 & 76.16 \\ L3 & Neuron \& Weights & 5 & 92.39 & 84.57 & 84.25 & 10.13 & 10.10 & 10.39 & 87.58 & 85.90 & 85.86 \\ L4 & Neuron \& Weights & 20 & 93.95 & 86.67 & 85.99 & 10.16 & 11.82 & 11.56 & 74.90 & 79.88 & 79.78 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Test results after retraining pruned networks.