# M2 Former: Multiscale Patch Selection for Fine-Grained Visual Recognition
## 4 Experiments
### Further Analysis
#### 4.5.2 Contributions from MSPS  
In the earlier section, we pointed out that SSPS is suboptimal because it is difficult to deal with scale changes. We now show that our proposed MSPS can consistently improve performance on different object scales. First, we need to classify a given set of objects according to their scale. Following COCO [92], we categorize all objects into large, medium, and small-sized objects according to their bounding box size. We compute the size of the bounding box using the given box coordinates and sort them in ascending order. Then, we compute the quartiles for sorted bounding box sizes. Finally, we classify an object as a small-sized (ob\({}_{s}\)) object if its bounding box size is less than the first quartile, as a large-sized object (ob\({}_{l}\)) if its bounding box size is greater than the third quartile, and as a medium-sized (ob\({}_{m}\)) object otherwise. Specifically, 25% of the objects are small, 50% are medium, and 25% are large. Example images for each category are shown in Figure 4.  
We train five model variants with different MSPS stages: M2 Former\({}_{none}\), M2 Former\({}_{4}\), M2 Former\({}_{3,4}\), M2 Former\({}_{2,3,4}\), M2 Former\({}_{full}\). It should be noted that M2 Former\({}_{none}\) is exactly the same as the MViTv2-B baseline. For comparison, we trained TransFG [18], which conducts SSPS at the last encoder block. The results are presented in Table 8. First, TransFG obtained a total top-1 accuracy of 91.3%, which obtained accuracies of 91.7% for ob\({}_{l}\), 91.7% for ob\({}_{m}\), and 90.1% for ob\({}_{s}\). The performance of ob\({}_{s}\) seems to lag behind ob\({}_{l}\) and ob\({}_{m}\). M2 Former\({}_{none}\) obtains a total top-1 accuracy of 91.6%, which outperforms TransFG by 0.3% higher. This means that we can achieve satisfactory results simply by using MS-ViT. Additionally, the improvement was prominent in ob\({}_{l}\) and ob\({}_{s}\) (0.6% and 0.4% higher, respectively), indicating that multiscale features are important in mitigating scale variability. \begin{table}
\begin{tabular}{c c} \hline
**Method** & **Accuracy (\%)** \\ \hline global pool & 91.5 \\ simple attach & 91.9 \\ CTT w/1-MLP & 92.1 \\ CTT w/2-MLP & 92.4 \\ \hline \end{tabular}
\end{table} 
Table 7: Comparison of different CTT methods on CUB. When we add MSPS to stage-4 (M2 Former\({}_{4}\)), we obtain a top-1 accuracy of 91.5%, which is 0.2% higher than the baseline. M2 Former\({}_{4}\) is almost identical to SSPS, as it selects only single-scale patches in a single stage. However, M2 Former\({}_{4}\) obtained a lower total top-1 accuracy than M2 Former\({}_{\text{none}}\), and it generally results in lower accuracy at all scales. This is also in contrast to previous findings [18], where performance was improved when patch selection was conducted at the last block. However, when adding the MSPS in stage-3 (M2 Former\({}_{3,4}\)), we obtained a top-1 accuracy of 91.9%, which outperforms the SSPS baseline by 0.6%. The improvements were found at all object scales but were noticeable at obj, (1.2% higher compared with the SSPS baseline). This means that salient patches from stage-3 (along with selected patches from stage-4) provide the definitive cue for large objects. We can see this improvement even when adding MSPS in stage-2 (M2 Former\({}_{2,3,4}\)). In particular, adding MSPS in stage-2 leads to improvements for obj\({}_{m}\) and obj\({}_{s}\). Compared with the SSPS baseline, it is 0.7% higher for obj\({}_{m}\), 1.3% higher for obj\({}_{s}\), and 1.0% higher for total accuracy. This indicates that the finer-grained object features extracted at stage-2 enhance representations for small/medium-sized objects. Figure 4: Example images for different scale objects. Following COCO [92], we classify objects into three scales, large, medium, and small, according to their bounding box size. The first row shows example images belonging to the large category (obj\({}_{l}\)). The second row shows example images belonging to the medium category (obj\({}_{m}\)).