# Bilinear CNN Model for Fine-Grained Classification Based on Subcategory-Similarity Measurement
## 3 Materials and Methods
### Classification Based on Subcategory-Similarity Measurement
#### 3.2.2 Jointly Learned Loss Function  
As shown in Figure 5, a triplet set \(T=\left\{I_{i}^{o},I_{i}^{p},I_{i}^{n}\right\}_{i}^{t}\) consists of positive pairs and negative pairs, where \(I_{i}^{o}\) is the reference image from a specific subcategory, \(I_{i}^{p}\) and \(I_{i}^{o}\) belong to the same subcategory while \(I_{i}^{n}\) and \(I_{i}^{o}\) belong to the different subcategories. Let \(f(x)\) denotes the network's feature representation of image \(x\), for \(I_{i}^{o}\), we expect its distance from any \(I_{i}^{n}\) of different subcategory is larger than \(I_{i}^{p}\) within the same subcategory by a certain margin \(\lambda_{1}\):  
\[\parallel f(I_{i}^{o})-f(I_{i}^{p})\parallel_{2}^{2}+\lambda_{1}\leq\parallel f (I_{i}^{o})-f(I_{i}^{n})\parallel_{2}^{2}\quad\forall(f(I_{i}^{o}),f(I_{i}^{p }),f(I_{i}^{n}))\epsilon T \tag{4}\]  
where \(\lambda_{1}>0\) is a predefined hyperparameter representing the minimum margin between matched and mismatched pairs, then the corresponding triplet loss is expressed as:  
\[L=\frac{1}{N}\sum_{i=1}^{N}[\parallel f(I_{i}^{o})-f(I_{i}^{p})\parallel_{2} ^{2}-\parallel f(I_{i}^{o})-f(I_{i}^{n})\parallel_{2}^{2}+\lambda_{1}]_{+} \tag{5}\]  
\(N\) donates the number of training images for the triplets.  
According to the CNN based fine-grained image-classification algorithms, traditional classification constraints, such as softmax loss, are usually used to classify, so different subcategories can be accurately distinguished. However, intraclass variance is not preserved, and such differences are essential for finding instances that are visually and semantically similar. In order to reduce intraclass difference, a new constraint condition is added to the triplet-loss function to constrain the distance between positive pairs \(\{I_{i}^{o},I_{i}^{p}\}\) in the same subcategory less than \(\lambda_{2}\):  
\[\parallel f(I_{i}^{o})-f(I_{i}^{p})\parallel_{2}^{2}\leq\lambda_{2} \tag{6}\]  
Figure 5: Schematic diagram of triplet-loss learning.  
Then the improved triplet-loss function is defined as:  
\[L_{triplet}=\beta[\parallel f(I_{i}^{o})-f(I_{i}^{p})\parallel_{2}^{2}-\lambda_{ 2}]_{+}+\frac{1}{N}\sum_{i=1}^{N}[\parallel f(I_{i}^{o})-f(I_{i}^{p})\parallel_ {2}^{2}-\parallel f(I_{i}^{o})-f(I_{i}^{n})\parallel_{2}^{2}+\lambda_{1}]_{+} \tag{7}\]  
where \(\lambda_{2}>0\) is a predefined hyperparameter representing the maximum distance within a subcategory. The improved triplet-loss function can not only enlarge the distance between different subcategories, but also restrict the distance between samples of the same subcategory. Compared with the traditional triplet-loss function, the improved triplet-loss function is easier to overfit, and convergence speed becomes slower due to input increase. In order to improve convergence speed, triplet loss and softmax loss are jointly optimized. In view of the fact that traditional softmax loss treats all subcategories equally, in order to further enhance the ability to distinguish similar subcategories, softmax loss is improved to make the model more punishable for misclassification. We obtain the misclassification probability \(P_{i}\) according to the fuzzying similarity matrix \(S\):  
\[P_{i}=\sum_{j=1,j\
eq j}^{k}S_{i,j} \tag{8}\]  
Then, the weighted softmax loss is defined as:  
\[L_{softmax}=\frac{1}{N}\sum_{i=1}^{N}-P_{i}\times log(f_{s}(i)) \tag{9}\]  
where \(f_{s}(i)\) is the return value of the image being classified as a subcategory labeled \(L_{i}\). The features of the triplets are transmitted to the triplet-loss layer to compute similarity loss, as well as forwarded to the softmax-loss layer to compute the classification error. Finally, we combine these two kinds of losses by a weighted combination:  
\[L=\alpha L_{softmax}+(1-\alpha)L_{triplet} \tag{10}\]  
where \(\alpha\) is the weight that controls the trade-off between two kinds of losses.