# M2 Former: Multiscale Patch Selection for Fine-Grained Visual Recognition
## 3 Our Method
### Multiscale Patch Selection  
Single-Scale Patch Selection (SSPS) has limited representations due to its fixed receptive field. Therefore, we propose MultiScale Patch Selection (MSPS) that extends SSPS to multiscale. With multiple receptive fields, the proposed MSPS encourages rich representations of objects from deep semantic information to fine-grained details. We design MSPS based on the MViT backbone. Specifically, we select salient patches from the intermediate feature maps produced at each stage of MViT.  
Given the patch sequence \(\widetilde{X}_{i}\in\mathbb{R}^{\overline{l}_{i}\times c_{i}}\), we start by detaching the CLS token and reshaping it into 2D feature maps to \(X_{i}\in\mathbb{R}^{h_{i}\times w_{i}\times c_{i}}\). And then we group \(r\times r\) neighboring patches, reshaping \(X_{i}\) into \(\hat{X}_{i}\in\mathbb{R}^{r^{2}\times\frac{h_{i}w_{i}}{r^{2}}\times c_{i}}\). This means \(\frac{h_{i}w_{i}}{r^{2}}\) neighboring patch groups are generated. Afterwards, we apply a per-group average to merge patch groups, producing \(\widetilde{X}_{i}\in\mathbb{R}^{\overline{l}_{i}\times c_{i}}\), where \(\overline{l}_{i}=\frac{h_{i}w_{i}}{r^{2}}\). We set \(r=2\) to merge patches within a \(2\times 2\) local region. This merging process removes the redundancies of neighboring patches, which forces MSPS to search for salient patches in wider areas of the image.  
Now, we produce a score map \(S_{i}\in\mathbb{R}^{\overline{l}_{i}}\) using a predefined scoring function \(\mathcal{S}(\cdot)\). Then, patches with top-\(k\) scores are selected from \(\widehat{X}_{i}\),  
\[P_{i}=\text{MSPS}\big{(}\widehat{X}_{i};S_{i},k_{i}\big{)}\,, \tag{1}\]  
where \(P_{i}\in\mathbb{R}^{k_{i}\times c_{i}}\). We set \(k\) differently for each stage to consider hierarchical representations. Since the high-resolution feature maps of the lower stage capture the detailed shape of the object with a small receptive field, we set \(k\) to be large so that enough patches are selected to sufficiently represent the details of the object. On the other hand, low-resolution feature maps of the higher stage capture the semantic information of objects with a large receptive field, so small \(k\) is sufficient to represent the overall semantics.  
For patch selection, we decided how to define the scoring function \(\mathcal{S}\). Attention roll-out [29] has been mainly used as a scoring function for SSPS [18; 20]. Attention roll-out aggregates the attention weights of the Transformer blocks through successive matrix multiplications, and the patch selection module selects the most salient patches based on the aggregated attention weights. However, since we use MS-ViT as the backbone, we cannot use attention roll-out because the size of attention weights is different for each stage,even each block. Instead, we propose a simple scoring function based on mean activation, where the score for the \(j\)-th patch of \(\hat{X}_{i}\) is calculated by  
\[S_{i}^{j}=\mathcal{S}(\hat{X}_{i})=\frac{1}{c_{i}}\sum_{c=1}^{c_{i}}\hat{X}_{i}^ {j}(c), \tag{2}\]  
where \(c\) is the channel index \(c\in\{1,2,\ldots,c_{i}\}\). Mean activation measures how strongly the channels in each patch are activated on average. After computing the score map, our MSPS conducts patch selection based on it. This is implemented through top-\(k\) and gather operations. We extract \(k_{i}\) patch indices with the highest scores from the \(S_{i}\) through the top-\(k\) operation, and patches corresponding to the patch indices \(I_{i}\) are selected from \(\hat{X}_{i}\),  
\[I_{i}=\texttt{topLRIndex}(S_{i};k_{i}), \tag{3}\] \[P_{i}=\texttt{gather}(\hat{X}_{i};I_{i}),\]  
where \(I_{i}\in\mathbb{N}^{k_{i}}\), and \(P_{i}\in\mathbb{R}^{k_{i}\times c_{i}}\).