# Fast Normalization for Bilinear Pooling via Eigenvalue Regularization
## 1 Introduction  
Over the past decade, deep networks have achieved remarkable success in the field of computer vision. Extracting local features on an image and then pooling them into a global representation has become one of the most promising types of network architectures [1, 2, 3]. As one of the most common pooling methods, bilinear pooling aggregates local features into a matrix where correlations between different features are preserved. With their higher-order information, the enriched representations have been used in various tasks [4, 5, 6, 7], especially in visual recognition [8, 9, 10, 11, 12]. Among bilinear pooling approaches, normalizing bilinear representation is almost applied, and is considered as an indispensable step to further boost performance [13]. Existing normalization approaches can be roughly categorized into element-wise normalization and structure-wise normalization. Element-wise normalization was first proposed to suppress the burstiness of bag-of-words features [14], and was then found to be effective in bilinear pooling as well. For each element in bilinear representation, the approach in [13] normalizes it by its square root and then divides the result by its Frobenius norm. This kind of normalization is simple yet effective and, thus, is widely used in bilinear pooling approaches [9, 13, 15, 16, 17]. Structure-wise normalization considers the fact that bilinear representation is a Symmetric Positive Definite matrix whose space forms a Riemannian manifold. Thus, it is clearly sub-optimal to send the representation to a linear classifier. The representation should be normalized by mapping the manifold into an Euclidean space. In particular, the O2P method [18] applies the matrix logarithm function as the normalization which can improve performance. In the following, DeepO2P [19] proposes back-propagation of the matrix logarithm function and achieves end-to-end learning of deep networks. However, computing matrix logarithm and its gradient are less efficient on the GPU. To accelerate the normalization, other works [15; 20; 21] suggest the square root of an SPD matrix which can also complete the linear mapping [22]. More importantly, the square root process can be approximated by [PERSON]'s iteration [15] where only basic matrix-matrix multiplications are involved. Therefore, the normalization with [PERSON]'s iteration is GPU-supported, and its efficiency has made itself one of the major normalization approaches to bilinear pooling. Although [PERSON]'s iteration provides strong theoretical support and good performance [15; 22], its sequential matrix multiplications still result in a high computational complexity of \(O(D^{3})\), where \(D\) is feature dimension and is always large (\(D=2048\) for ResNet-50 network [1]) for modern deep network architectures. Thus, more efficient structure-wise methods are further studied. For example, Run [23] only normalizes the largest eigenvalue and simplifies the computation into matrix-vector multiplications. PSMR [9] can reduce computational complexity into \(O(n^{3})\). Here, \(N\) is the number of local features and can be smaller than the feature dimension in most cases. iSICE [8] and iSQRT [20] adopt a convolutional layer to reduce feature dimension before bilinear pooling. Although great efforts are made, the aforementioned studies make a trade-off between the efficiency and recognition accuracy of the network. Recently, a pioneering work [10; 24] analyzed the effect of the square root normalization and, for the first time, showed that the normalization reaches a good balance between feature decorrelation and information preservation. According to this finding, its method, called DropCov, achieves normalization by applying an adaptive channel dropout to the features before bilinear pooling. Therefore, compared with the square root normalization, this method only has a linear complexity of \(O(D)\) and is free of inference, largely improving the efficiency while keeping good recognition accuracy.