# On the Redundancy in the Rank of Neural Network Parameters and Its Controllability
## 5 Experiments
### Results  
We use accuracy as our evaluation metric, which is the standard metric in this dataset. Accuracy is defined as the percentage of correct predictions and can be acquired by dividing the number of correctly classified test samples with the total number of test samples. Compression rate is the fraction of parameters removed, compared to the initial number of parameters. To test the statistical significance between different configurations, we train each model 20 times with same hyperparameters but with different random seeds and report the mean and standard deviation. The change in accuracy and training loss over the course of training is plotted in Figure 1. We also plot the moving average of 20 epochs to highlight the trend. As can be seen, the proposed method achieves the best accuracy of the baseline model, plotted with a dashed line, in just around 280 epochs. This is about 7.1\(\times\) speedup in terms of number of training steps. Even after this point, the accuracy of this method keeps increasing and reaches to a significantly higher point. The cross entropy loss of the proposed method decreases faster to a lower point compared to the baseline, while also exhibiting less deviation. These results suggest that the proposed method makes the training process more effective and efficient. In Table 1, the result of an ablation study is summarized. To account for the noise from mini-batched training, all numbers are the average of the last 5 epochs. Please note that we do not show the result of minimizing \(\mathcal{L}_{RaRe}\) alone, since it would only reduce the pairwise sine value of a single pair of vectors and had almost no effect on the model. When Frobenius Normalization is applied to the baseline model, there is a little improvement in accuracy and decrease in standard deviation. However, the difference is not statistically significant with \(p>0.1\). On the other hand, with both Frobenius normalization and dynamic low-rank approximation, we see an average of 2.09% relative error reduction with \(p<0.001\). The proposed method is also most reliable in achieving the accuracy, as indicated by the lowest standard deviation. This is more encouraging given the fact that our model has 30.65% less trainable parameters. There are three hyperparameters that affect the compression--\(\tau_{cos}\), \(\tau_{enc}\), and \(\lambda_{1}\). Overall, we find that \(\lambda_{1}\) is more suitable at controlling the compression rate than \(\tau_{cos}\) or \(\tau_{enc}\), since the effect of the latter two hyperparameters is less predictable as shown in Figure 2. For this reason, we fix the values of these two hyperparameters and control the model \begin{table}
\begin{tabular}{c c c c c c} \hline  & **Acc. (\%)** & \(p\)**-Value** & **Comp. (\%)** & **\# Param.** & **FLOPs** \\ \hline Baseline & 90.45 \(\pm\) 0.21 & N/A & 0.00 & 1.40 M & 2.79 M \\ +FN & 90.51 \(\pm\) 0.18 & 1.99 \(\times\) 10\({}^{-1}\) & 0.00 & 1.40 M & 2.79 M \\ +DLRA (\(\lambda_{1}\) = 20.0) & 90.65 \(\pm\) 0.15 & 3.72 \(\times\) 10\({}^{-4}\) & 30.65 \(\pm\) 0.97 & 0.97 \(\pm\) 0.01 M & 2.09 \(\pm\) 0.03 M \\ \hline \end{tabular}
\end{table} 
Table 1: Ablation results, on the Fashion-MNIST test set after training for 2000 epochs. Acc = accuracy, Comp = compression rate, # Param = number of trainable parameters, FLOPs = number of floating point operations for a single inference step, FN = Frobenius normalization, DLRA = dynamic low-rank approximation. \(P\)-values are calculated against the baseline model. Figure 1: (**a**) Test set accuracy and (**b**) training loss over the course of training. We also plot the moving average of 20 epochs. compression using \(\lambda_{1}\) in the following experiments. It is worth noting that all combinations of \(\tau_{cos}\) and \(\tau_{euc}\) that we have tested still outperformed the baseline model. To have a better understanding on the effect of compression on the accuracy, the proposed model is trained with different values of \(\lambda_{1}\) and the result is plotted in Figures 3 and 4.