# Activation-Based Pruning of Neural Networks
## 5 Results and Discussion
### Analysis of Singular Value Changes in Each Layer  
The experimental setup was similar to the setup outlined in Section 4.1. The final training accuracy was set at 98%; the total pruning to be achieved was set to 80%; the total number of pruning cycles was 10; the pruning was initiated when the network reached 70% training accuracy. At every stage of pruning, the singular values of the weight matrix were calculated before and after pruning. Figure 7 demonstrates the percentage variation in the singular values of the weight matrix, comparing values before the first pruning cycle to those after the completion of the final pruning cycle. Figures 11-18 demonstrate the percentage variation in the singular values for each pruning cycle.  
We observed that the singular values were reduced by a weighted amount. We also observed that, during the initial phase of pruning, the singular values with smaller magnitudes experienced a greater reduction in value than singular values with larger magnitudes. This corresponds to the notion that activation-based pruning results in removing the basis vectors that contain relatively little information about the data. As pruning progresses, the basis vectors that hold the least amount of information are removed successively. After a while, we are left with basis vectors that contain equally important amounts of information. At this stage, the rate of reduction in the magnitude of singular values shifts from a weighted reduction to a uniform reduction. This is equivalent to stating that a reduction  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Model** & **Test Accuracy** & **Top-1\% Accuracy** & **Top-5\% Accuracy** \\ \hline S & 84.35 & 8.22 & 69.40 \\ \hline \hline \end{tabular}
\end{table}
Table 13: PCA analysis: test results  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Model** & **Low-Rank Approximation** & **Using PCA** & \multicolumn{3}{c}{**Activation-Based Pruning**} \\ \hline  & **Test** & **Top-1\%** & **Top 5\%** & **Test** & **Top 1\%** & **Top 5\%** \\  & **Accuracy** & **Accuracy** & **Accuracy** & **Accuracy** & **Accuracy** & **Accuracy** \\ \hline G1 & 81.79 & 8.85 & 63.73 & 87.90 & 10.16 & 65.69 \\ G2 & 80.47 & 8.57 & 61.93 & 87.48 & 9.54 & 71.67 \\ G3 & 84.75 & 8.24 & 68.75 & 87.12 & 9.66 & 81.83 \\ G4 & 84.50 & 8.02 & 68.44 & 88.19 & 9.46 & 66.77 \\ G5 & 78.01 & 9.24 & 65.08 & 87.83 & 9.57 & 80.24 \\ G6 & 74.67 & 6.41 & 53.91 & 86.69 & 9.69 & 64.99 \\ G7 & 84.52 & 8.26 & 69.05 & 88.07 & 12.30 & 79.23 \\ G8 & 84.35 & 8.34 & 69.34 & 88.67 & 11.58 & 79.31 \\ \hline \hline \end{tabular}
\end{table}
Table 12: Test results of the fully trained model.  
Figure 7: Ratio of singular values before and after pruning.  
in the magnitude of the singular values shifts from a weighted nuclear norm minimization (WNNM) to nuclear norm minimization (NNM). [PERSON] et al. (2018) demonstrated that WNNM is better than NNM for low-rank matrix approximation problems. [PERSON] et al. (2018) demonstrated that NNM and WNNM are equivalent to L1-norm and weighted L1-norm minimization, respectively. Inspired by the superiority of weighted L1-norm minimization for sparse coding, they explained that WNNM is more effective than NNM for low-rank approximation. Their analysis provides a theoretical justification for WNNM's empirical effectiveness over NNM. To the best of our knowledge, our work is the first of its kind to analyze the effect of pruning on the singular values of hidden layers and provides empirical data to support the hypothesis that activation-based pruning results in a weighted nuclear norm minimization of the hidden layers. We hypothesize that, in a feedforward neural network using SGD as the optimization algorithm and ReLU as the activation function, activation-based pruning is equivalent to introducing the weighted nuclear norm as a regularization parameter to the original cost function.