# Activation-Based Pruning of Neural Networks
## 5 Results and Discussion
### Activation-vs. Magnitude-Based Pruning
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline
**Model** & **Pruning Type** & **Percentage of Pruning** & \multicolumn{3}{c}{**Rank of Pruned Network**} & \multicolumn{3}{c}{**Rank after Retraining Pruned Network**} \\ \hline  & & \
\begin{tabular}{c} **Before** \\ **Retraining** \\ \end{tabular} &
\begin{tabular}{c} **After** \\ **Retraining** \\ \end{tabular} & **Layer 1** & **Layer 2** & **Layer 3** & **Layer 4** & **Layer 1** & **Layer 2** & **Layer 3** & **Layer 4** \\ \hline M & Magnitude & 80 & 0.11 & 300 & 200 & 100 & 50 & 300 & 200 & 100 & 50 \\ G1 & Neuron & 70.25 & 43.57 & 78 & 80 & 40 & 21 & 170 & 132 & 66 & 34 \\ G2 & Neuron & 74.64 & 40.74 & 60 & 68 & 56 & 29 & 184 & 114 & 72 & 37 \\ G3 & Neuron & 74.59 & 37.16 & 61 & 74 & 50 & 25 & 199 & 113 & 66 & 37 \\ G4 & Neuron & 79.25 & 41.02 & 48 & 64 & 38 & 26 & 185 & 122 & 59 & 43 \\ G5 & Neuron \& Weights & 70.25 & 20.37 & 225 & 163 & 74 & 36 & 244 & 167 & 77 & 36 \\ G6 & Neuron \& Weights & 85 & 24.36 & 207 & 143 & 83 & 38 & 237 & 150 & 84 & 40 \\ G7 & Neuron \& Weights & 77.59 & 17.39 & 243 & 161 & 79 & 36 & 257 & 163 & 80 & 36 \\ G8 & Neuron \& Weights & 82.44 & 25.19 & 205 & 156 & 84 & 43 & 227 & 163 & 87 & 46 \\ L1 & Neuron & 79.88 & 50.43 & 60 & 40 & 20 & 10 & 151 & 130 & 50 & 32 \\ L2 & Neuron & 84.85 & 58.72 & 45 & 30 & 15 & 8 & 132 & 86 & 42 & 26 \\ L3 & Neuron \& Weights & 79.88 & 22.76 & 148 & 98 & 46 & 26 & 235 & 160 & 74 & 39 \\ L4 & Neuron \& Weights & 84.87 & 26.67 & 151 & 80 & 45 & 24 & 225 & 166 & 84 & 38 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Ranks before and after retraining pruned networks. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Pruning Type** & **\(\approx\)kFLOPS** & **Pct. Reduction** \\ \hline M & 129 & 79.87 \\ \hline G1 & 191 & 70.25 \\ G2 & 163 & 74.64 \\ G3 & 191 & 70.25 \\ G4 & 96 & 85.00 \\ G5 & 163 & 74.59 \\ G6 & 133 & 79.25 \\ G7 & 144 & 77.59 \\ G8 & 113 & 82.44 \\ \hline L1 & 129 & 79.88 \\ L2 & 97 & 84. In Table 7, we demonstrate the pruning percentage before and after the retraining of pruned networks. We specify the rank of each hidden layer before and after retraining. The percentage of pruning of model \(M\) before training was 80%, which implies that 80% of the matrix weights were initialized to a 0 value. Hence, the matrix of weights for model \(M\) before training was sparse. The column Rank Before Training illustrates the rank of each hidden layer of model \(M\) to be full rank. Networks pruned using activation-based pruning and, subsequently, retrained produced weight matrices that were sparse and low rank. Model \(M\) recovered a training accuracy close to 95%, but utilized the complete network weights, resulting in the pruning percentage being 0.11%. This is illustrated under the heading Percentage of Pruning with sub-heading After Training. However, retraining networks pruned with activation-based pruning was able to achieve comparable training accuracy while maintaining a certain level of sparsity and low-rank approximation for each hidden layer. The networks that were pruned using activation-based pruning and, subsequently, retrained contained 20-50% of the weights with zero value. This demonstrated that, if we trained a network pruned using activation-based pruning, it will try to maintain sparsity and a low-rank structure for each hidden layer in the network. We observed that, after retraining, pruning based on Neurons maintained a larger percentage of sparsity and a lower rank for each hidden layer compared to pruning based on Neurons & Weights. Figure 6 illustrates the heatmap representing the weight matrix of the first hidden layer of the network. The darker regions represent the non-zero weight, and lighter regions represent zero weight values, respectively. The figure contains two sets of heatmaps. The set on the left represents weight matrices for pruned networks, and the set on the right represents weight matrices for networks retrained after pruning. Figures 7-8 in the Appendix B correspond to heatmaps for the remaining hidden layers of the network.