# M2 Former: Multiscale Patch Selection for Fine-Grained Visual Recognition
## 3 Our Method
### Class Token Transfer  
Through MSPS, we can extract salient patches from each stage, \(\mathbf{P}=\{P_{1},P_{2},P_{3},P_{4}\}\). In Section 3.2, the \(\mathsf{CLS}\) token is detached from the patch sequence before MSPS at each stage. The simplest way to reflect the selected multiscale patches in the model decisions is to concatenate the detached \(\mathsf{CLS}\) token \(\mathsf{CLS}_{i}\) with the \(P_{i}\) again and feed it into a few additional ViT blocks, consisting of multihead self-attention (MSA) and feed-forward networks (FFN):  
\[\widetilde{P}_{i}=\texttt{concat}(P_{i},\mathsf{CLS}_{i}), \tag{4}\] \[\widetilde{O}_{i}=\text{FFN}(\text{MSA}(\widetilde{P}_{i})),\]  
where \(\widetilde{P}_{i}\), \(\widetilde{O}_{i}\in\mathbb{R}^{\widetilde{k}_{i}\times c_{i}}\), and \(\widetilde{k}_{i}=k_{i}+1\). Finally, predictions for each stage are computed by extracting the \(\mathsf{CLS}_{i}\) from \(\widetilde{O}_{i}\) and connecting the linear classifier. It should be noted that the \(\mathsf{CLS}_{i}\) is shared by all stages: the set of \(\mathsf{CLS}_{i}\) is derived from the global \(\mathsf{CLS}\) token and it is detached with different dimensions at each stage. This means that the stage-specific multiscale information is shared through \(\mathsf{CLS}_{i}\). However, the current sharing method may cause inconsistency between stage features because the detached \(\mathsf{CLS}_{i}\) does not equally utilize the representational power of the network. For example, \(\mathsf{CLS}_{1}\) is detached right after stage-1 and it will always lag behind \(\mathsf{CLS}_{4}\), which utilizes representations of all stages.  
To this end, we introduce a Class Token Transfer (CTT) strategy that aggregates multiscale information more effectively. The core idea is to use the \(\mathsf{CLS}\) token transferred from the global \(\mathsf{CLS}\) token \(\mathsf{CLS}_{g}\) rather than using the detached \(\mathsf{CLS}_{i}\) at each stage. It should be noted that \(\mathsf{CLS}_{g}\) is equal to \(\mathsf{CLS}_{4}\), so \(\mathsf{CLS}_{g}\in\mathbb{R}^{c_{4}}\). We transfer the \(\mathsf{CLS}_{g}\) according to the dimension of each stage through a projection layer consisting of two linear layers along with Batch Normalization (BN) and ReLU activation:  
\[\overline{\mathsf{CLS}}_{i}=W_{i}^{1}\Big{(}\text{ReLU}\Big{(}\text{BN} \Big{(}W_{i}^{0}\mathsf{CLS}_{g}\Big{)}\Big{)}\Big{)}, \tag{5}\]  
where \(W_{i}^{0}\in\mathbb{R}^{2c_{i}\times c_{4}}\), \(W_{i}^{1}\in\mathbb{R}^{c_{i}\times 2c_{i}}\) are the weight matrices, and \(\overline{\mathsf{CLS}}_{i}\) is the transferred \(\mathsf{CLS}_{g}\) in stage \(i\in\{1,2,3\}\). Now, (4) is reformulated as  
\[\widetilde{P}_{i}=\texttt{concat}(P_{i},\overline{\mathsf{CLS}}_{i }), \tag{6}\] \[\widetilde{O}_{i}=\text{FFN}(\text{MSA}(\widetilde{P}_{i})).\]  
Compared with conventional approaches,CTT guarantees consistency between stage features as it uses \(\mathsf{CLS}\) tokens with the same representational power. Each stage encodes stage-specific patch information into a globally updated \(\mathsf{CLS}\) token.CTT is similar to the top-down pathway [35; 39]: it combines high-level representations of objects with multiscale representations of lower layers to generate richer network-level representations.