# Fast Normalization for Bilinear Pooling via Eigenvalue Regularization
## 2 Related Work
### Structure-Wise Normalization  
Bilinear pooling results in a Symmetric Positive Definite (SPD) matrix whose space forms a Riemannian manifold. Due to lying in such a non-Euclidean space, the result is sub-optimal to the next linear classifier, for example, a fully-connected layer in the deep network. The Log-Euclidean Riemannian Metric (LERM) [43; 44] is proposed to measure the distance between two SPD matrices. Then, this method is implemented into deep networks to normalize the output of bilinear pooling and achieves success with linear classifiers [18; 19; 45]. More precisely, these works apply the logarithm to each eigenvalue in the SPD matrix. Grassmann Pooling (GP) [46] transforms the CNN feature matrix into a set of its singular vectors. Although these works improve the effectiveness, the normalization process involves Singular Value Decomposition (SVD), which is not efficient when using GPUs and consequently slows down the network. Hence, to achieve efficient matrix normalization, [PERSON] and [PERSON] [15] suggested normalizing bilinear representation into its square-rooted matrix with [PERSON]'s iteration, where only matrix-matrix multiplications occur. Thus, this method is adapted to GPUs and speeds up the normalization. iSQRT-Cov [20] extends the idea of [PERSON]'s iteration to back-propagation and makes the normalization more efficient when training a network. Since then, a series of works have applied [PERSON]'s iteration to batch whitening [47; 48; 49] and have combined the fast square root normalization with compact bilinear pooling [9; 17; 30; 50]. For example, PSMR [9] proposes a pseudo-square-rooted matrix, which can be obtained by normalizing the CNN feature matrix before bilinear pooling. One advantage of this matrix is its compatibility with most existing compact bilinear pooling methods. This method also improves the efficiency of square root normalization when the number of feature vectors is smaller than the feature dimension. [PERSON] et al. [51] proposed two methods, called Matrix TaylorPolynomial (MTP) and Matrix Pade Approximant (MPA), to calculate square-rooted matrix. This work shows that, beyond [PERSON]'s iteration, there is still room to improve the speed of square root normalization. Besides square root normalization, a few other types of normalization are proposed. RUN [23] is an efficient matrix normalization that normalizes the largest eigenvalue in the bilinear representation. DeepKSPD [52] generalizes square root normalization to a case where eigenvalues are rooted with a learnable parameter instead of a fixed value of 0.5. ReDro [53] randomly groups feature channels and normalizes bilinear representation in each group. As the feature dimension is reduced, the normalization is more efficient. [PERSON] et al. [54] implemented democratic aggregation and weight each local feature so that their contribution to bilinear representation is equalized. iSICE [8] is a recent normalization that mitigates the confounding effect in bilinear representation. [PERSON] et al. [55] proposed Riemannian classifiers that enable accurate classification on the geometry of SPD manifolds. [PERSON] et al. [56] proposed a Minimum Description Length (MDL) estimator to detect the number of superimposed signals, demonstrating how eigenvalue regularization can enhance robustness in high-dimensional spaces. Different from the above works, our approach directly produces normalized bilinear representation, and any explicit normalization process is not involved for inference. This characteristic makes the bilinear pooling equipped with our approach run more efficiently. The most similar work to ours is DropCov [10], where for each training sample, a network branch estimates channel dropout rate and randomly zeroes feature channels with a probability. During inference, the branch is omitted and no normalization is performed. Compared with DropCov, this paper proposes two regularization terms and adds them to the loss function so no additional network structure is needed for our method.