# Bilinear CNN Model for Fine-Grained Classification Based on Subcategory-Similarity Measurement
## 3 Materials and Methods  
### Weakly Supervised Localization  
Previous studies have shown that second-order representation is more effective than first-order features (such as VLAD and HOG) in image processing. [PERSON] and [PERSON] [27] proposed a bilinear model to model the change of two-factor variations, such as the \"style\" and \"content\" of the image. With the development of deep learning, [PERSON] et al. introduced the bilinear CNN model for image classification. A bilinear model can be regarded as a quaternion: \(B=(f_{A},f_{B},P,C)\), \(f_{A}\) and \(f_{B}\) are two feature functions corresponding to CNN stream \(A\) and \(B\) in bilinear CNN. \(P\) denotes the pooling function and \(C\) represents the classification function. B-CNN does not require additional annotations  
Figure 2: Schematic view of overall structure. except for image labels. It is thought that one of the networks is used for detecting part while the other is for extracting local features. However, the problem is that the background of the original image brings noise to feature extraction, so a localization method based on a B-CNN model is proposed to locate the main object and remove the background under the weakly supervised setting. The entire framework is shown in Figure 3.  
In the forward-propagation phase of CNN, the features extracted by different convolutional layers are different. Using \"feature map\" to indicate different convolution results of a channel; \"activations\" is used to represent the feature maps of all channels in a single convolution layer; and \"descriptor\" represents the d-dimensional component vector of activations [28]. Since the classical VGG-16 convolution neural network can extract the initial feature of the input image after a large amount of image training, the popular pretrained VGG-16 model was selected as the feature-extraction model. The VGG-16 architecture is shown in Figure 4. The blue layer represents the pooling layer while the white layer represents the convolution and activation layer, \"_pools_\" refers to the activations of the max-pooling layer after the last convolution layer. For input image \(I\) of size \(H\times W\), each layer outputs \(d\) two-dimensional feature maps of \(h\times w\) marked as \(F=\{F_{n}\}\left(n=1,\dots,d\right)\), \(\{F_{n}\}\) is the \(n\)-th feature map with the size of \(h\times w\) in the corresponding convolution channel. \(F\) can also be regarded as having \(h\times w\) positions, and each position \((i,j)\) contains one d-dimensional component vector \(x_{(i,j)}eR^{d}(i\epsilon\{1,\dots,h\},je\{1,\dots,w\})\). Since the semantics of activated regions are very different even on the same channel, it is unrealistic to only locate the object through a single-feature map. However, if feature maps with multiple channels are activated in the same region, this region may be judged as an object rather than the background. The _pool_s activation is added through depth direction to obtain the two-dimensional matrix \(F=\sum_{n=1}^{d}\{F_{n}\}\). Calculating the average value \(\tilde{f}\) of all positions in \(F\), and taking \(\tilde{f}\) as the threshold to determine which position localizes the object, position \((i,j)\) is likely to be the position of the object if the activation response of \((i,j)\) is higher than the threshold. Define a mask map \(M\) whose size is the same as \(F\):  
\[M_{i,j}= \begin{cases}1&\text{if }F_{i,j}>\tilde{f}\\ 0&\text{otherwise}\end{cases}  \tag{1}\]  
The position in \(M\) whose corresponding value is \(1\) is likely to be the location of the main object. Because the features extracted from different convolutional layers are different, the lower convolution layer of convolution network is more inclined to learn low-level features, while the higher convolution layer acquires high-level semantic features. By fusing different masks from different convolutional layers, high-level semantic information and low-level local features can be simultaneously preserved. Figure 3: Schematic diagram of weakly supervised localization.