# Grouping Bilinear Pooling for Fine-Grained Image Classification
## 2 Analysis of Bilinear Pooling  
In BCNN [15], in order to get the full bilinear representation \(Z\) of the input image, CNN is used to extract the higher-order feature representation \(X\) of the input image, \(X\in\mathbb{R}^{C\times H\times W}\), where \(C\) is the number of feature layers, \(H\) and \(W\) are the height and the width of feature layers, respectively. For each feature layer, there are \(H\times W\) different locations. The architecture of BCNN is illustrated in Figure 1.  
Here, we define local descriptor \(x_{i}^{T}=\left[x_{i}^{1},x_{i}^{2},\ldots,x_{i}^{C}\right]\in\mathbb{R}^{C}(i \in[1,\ldots,HW])\), \(x_{i}^{n}\) represents the value of \(i\)th position on the \(nth\) feature layer. Besides, for convenience, considering \(x_{i}\) as a vector the dimension of which is \(C\). The bilinear representation \(Z\) is as follows:  
\[Z=\frac{1}{HW}\sum_{i}^{HW}x_{i}x_{i}^{T}=\frac{1}{HW}\left[ \begin{array}{ cccc}\sum_{i}^{HW}x_{i}^{1}x_{i}^{1}&\ldots&\sum_{i}^{HW}x_{i}^{1}x_{i}^{C}\\ &\ldots&\\ \sum_{i}^{HW}x_{i}^{1}x_{i}^{C}&\ldots&\sum_{i}^{HW}x_{i}^{C}x_{i}^{C}.\end{array}  \right] \tag{1}\]Vectorizing \(Z\), \(vector(Z)\in\mathbb{R}^{C^{2}}\). Assuming \(C\) is 512, \(vector(Z)\) will be up to 250 K dimensions. The high dimensions of bilinear representation result in high computation and storage costs. The representation \(Z\) is used for classification after passing through the full-connection layer,  
\[Output=ZW_{C}+b=\frac{1}{HW}\Bigg{(}\sum_{i}^{HW}x_{i}x_{i}^{T}\Bigg{)}W_{C}+b_ {C}, \tag{2}\]  
where \(W_{C}\in\mathbb{R}^{C^{2}\times N}\) is the weight matrix of the full-connection layer, \(b_{C}\in\mathbb{R}^{k}\), \(Output\)\(\in\mathbb{R}^{N}\), \(N\) is the number of categories. In general, \(C^{2}\gg N\), the rank of \(W_{C}\) is as follows:  
\[rank(W_{C})\leq\min\Big{(}C^{2},N\Big{)}=N. \tag{3}\]  
Vectorizing the \(i\)th feature layer: \(f_{i}^{T}=\big{[}x_{1}^{i},x_{2}^{i},\ldots,x_{HW}^{i}\big{]}\in\mathbb{R}^{ HW}\), \(F^{T}=[f_{1},f_{2},\ldots,f_{C}]\)\(\in\mathbb{R}^{C}\),  
\[Z=\frac{1}{HW}\Bigg{[} \begin{array}{ccc}f_{1}^{T}f_{1}&\ldots&f_{1}^{T}f_{c} \\ &\ldots&\\ f_{1}^{T}f_{c}&\ldots&f_{c}^{T}f_{c}\end{array} \Bigg{]}=\frac{1}{HW}FF^{T}. \tag{4}\]  
\[Output=ZW_{C}+b=\frac{1}{HW}\Big{(}FF^{T}\Big{)}W_{C}+b_{C}. \tag{5}\]  
In [17], RM [21] is used to sample the feature layers, then bilinear pooling is carried out. In fact, the representation obtained by [17] is the recombination of part of the element in \(Z\). The representation obtained by [18], using the low-dimensional approximation of Hadamard product, is the elements on the diagonal of \(Z\). This low-rank approximation actually abandons the vast majority of information of \(Z\). The loss of information is inevitable, although the dimensions of representation are reduced. \(X\) contains \(C\) different feature layers \(f\); the bilinear representation \(Z\) in Equation (1) is a symmetric matrix. The elements on the diagonal of \(Z\) are the dot product sum of the corresponding positions of the feature layer itself. The scalar obtained by the point-wise product can be regarded as the pixel-level self-correlation of the feature layer to some extent. Similarly, the elements on the non-diagonal of \(Z\) are the dot product sum of the corresponding positions of different feature layers, which can be regarded as the cross-correlation between feature layers. Since the bilinear representation obtained by bilinear pooling is a correlation representation with extremely high dimensions between high-order feature layers (increasing with the square of the number of feature layers), it will greatly increase the parameters to be learned by the full-connection layers even if there is only a single-layer full-connection layer. Furthermore, nearly half of the calculations in the symmetric matrices \(Z\) are repeated; obviously, it greatly reduces computational efficiency and results in redundancy of the model. Figure 1: The architecture of Bilinear CNN. \(X\) is the high-order features extracted by CNN.