# Fast Normalization for Bilinear Pooling via Eigenvalue Regularization
## 4 Experiments
### Computational Complexity Analysis  
Our approach can be considered as a variant of knowledge distillation. This is because RegCov employs \(\bar{D}_{e}\) and \(\bar{I}_{e}\) after the first training stage to guide a lighter network at the second stage. Hence, in this section, we use VGG-16 backbone on the Birds dataset and compare our approach with other knowledge distillation approaches, as shown in Table 7. The approach of logits [58] and Features [61] refer to, respectively, using prediction logits and normalized bilinear representation of the network at the first stage for knowledge distillation. It can be seen that the two approaches require more training time and more parameters. This is because they should run the teacher network every training iteration. In addition, they cannot transfer exact knowledge about the eigenvalues in the bilinear representation matrix. This limits its effect of the normalization and finally results in lower accuracy. SVD calculates explicitly the eigenvalues and consumes much more training time than ours due to its poor efficiency on the GPU.  
Furthermore, at the first training stage, we selected Newton Iteration as explicit square root normalization. However, as discussed in Section 3.4, it would be important to implement a more efficient approach to compute the matrix square root. To this end, we utilized the official implementation of a state-of-the-art method, MPA [51] and conducted experiments on the Birds dataset [25] using the VGG-16 network. As shown in Table 8, MPA demonstrates faster computational speeds than the Newton Iteration method during the training phase. The results are consistent with the findings in ref. [51]. However, MPA provides lower classification accuracy in Table 8. We found that this is attributed to the reduced accuracy of the square-rooted matrix computed by MPA, particularly when the matrix dimension is large (please see Appendix A for details). Given that the matrix dimension in our approach RegCov is at least 256, and considering the inefficiency of SVD variants shown in Table 8, we retain Newton Iteration.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Acc.** & \(\lambda_{m}\) & **0** & \(\mathbf{1\times 10^{-6}}\) & \(\mathbf{1\times 10^{-5}}\) & \(\mathbf{1\times 10^{-4}}\) & **Linear** \\ \hline
0 & 73.7 & 74.6 & 74.8 & 74.0 & 74.4 \\ \(1\times 10^{2}\) & 73.7 & 74.6 & 73.9 & 74.2 & 75.2 \\ \(1\times 10^{3}\) & 73.6 & 74.5 & 74.0 & 74.1 & 75.2 \\ Linear & 73.5 & 74.5 & 75.1 & 74.1 & **75.3** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Top-1 validation accuracy (%) of RegCov using ResNet-18 on Imagenet1K with respect to loss weights \(\lambda_{i}\) and \(\lambda_{d}\).  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Methods** & **T.S. (ms)** & **\# Params (M)** & **Acc. (\%)** \\ \hline SVD & 1087.8 & 268.5 & 86.8 \\ Logits [58] & 733.4 & 537.1 & 86.4 \\ Features [61] & 731.9 & 327.4 & 86.4 \\ RegCov & 561.1 & 268.5 & 86.8 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Comparison of our method RegCov with knowledge distillation approaches in terms of training speed (T.S.) for one mini-batch (32 images) on an RTX-3090 GPU, model parameters (# params) and accuracy (Acc.).