# Wild Mushroom Classification Based on Improved MobileViT Deep Learning
## 2 Materials and Methods
### M-ViT Model
#### 2.3.4 Multidimension Attention  
The multi-scale attention module decomposes the spatial axis into two sparse forms, local (block attention) and global (grid attention), to solve the quadratic complexity caused by global self-attention. Its structure is shown in Figure 8.  
1. Block Attention: For the input feature map \(X\in R^{H\times W\times C}\), it is transformed into a shape tensor \(\left(\frac{H}{P}\times\frac{W}{P},P\times P,C\right)\) to represent the division into non-overlapping windows, where the size of each window is \(P\times P\), and finally, the RelAttention computation is performed in windows. First, we define the block (\(\cdot\)) operation to convert the input feature graph into a non-overlapping window of \(P\times P\) size, and then we define unblock (\(\cdot\)) to do the opposite. The Block-SA Block in Figure 6 is to divide the Block into Windows and then performs the RelAttention operation, as in the following equations: \[Block:(H,W,C)\rightarrow\left(\frac{H}{P}\times P,\frac{W}{P}\times P,C\right) \rightarrow\left(\frac{HW}{P^{2}},P^{2},C\right)\] (4) \[RelAttention(Q,K,V)=softmax(\frac{QK^{T}}{(\sqrt{d})}V\] (5) where \(Q\), \(K\), and \(V\) represent query, key, and value matrices, respectively, and \(d\) is the dim of \(Q\) and \(K\). \[Block-SA(x)=(x+Unblock(RelAttention(Block(DWConv(LN(x)))))))\] (6) \[x=x+gMLP(LN(x))\] (7)
2. Grid Attention: Grid attention uses Grid(\(\cdot\)) to convert input features into \(G\times G\) uniform grid to grid the input tensor as \(\left(G\times G,\frac{H}{G}\times\frac{W}{G},C\right)\) at which point the window of adaptive size is obtained \(\frac{H}{G}\times\frac{W}{G}\), and finally on \(G\times G\) The RelAttention calculation  
Figure 8: The MDA Block. Figure 7: The gMLP Block. is performed on the grid attention. Unlike the Block operation, we need additional transposes to place the grid dimension on the assumed spatial axis. Additionally, the Ungrid () is defined to perform the inverse operation to return it to 2D space, as in the follwoing equations:  
\[\text{Grid}:(H,W,C)\rightarrow\left(G\times\frac{H}{G},G\times\frac{W}{G},C \right)\rightarrow\left(G^{2},\frac{HW}{G^{2}},C\right)\rightarrow\left(\frac {HW}{G^{2}},G^{2},C\right) \tag{8}\]  
\[\text{Grid-}SA(x)=(x+UnGrid(RelAttention(Grid(DWConv(LN(x)))))) \tag{9}\]  
\[x=x+gMLP(LN(x)) \tag{10}\]
3. Global Attention: For the input feature map \(X\in R^{H\times W\times C}\), we first operate on the input with \(DWConv\) and \(1\times 1\) convolution to obtain \(X_{L}\in R^{H\times W\times d}\). DW convolution is used to learn the local and channel spatial information to prevent the loss of spatial information of channels, and \(1\times 1\) convolution is used to project the input features into the high-dimensional space. A visual transformer (ViT) with multi-headed self-attentiveness is used for modeling to obtain longer-distance relations. However, ViT has many parameters and weak optimization capability because ViT lacks inductive bias. To enable the Transformer (ViT) to learn a global representation with spatial inductive bias, first, the \(X_{L}\) is expanded into N non-overlapping patches \(X_{U}\in R^{H\times N\times d}\), where \(P=wh\), \(N=\frac{HW}{P}\) is the number of patches and \(h\leq n\), \(w\leq n\) is the height and width of the patches. Each pixel in the patches \(P\in\{1,\cdots,P\}\) is modeled by Transformer to get \(X_{G}\in R^{P\times N\times d}\), as in the following equations:  
\[Block:(H,W,C)\rightarrow\left(\frac{H}{P}\times P,\frac{W}{P}\times P,C\right) \rightarrow\left(\frac{HW}{P^{2}},P^{2},C\right)\]0  
Unlike ViT, which loses the spatial order of pixels, global attention does not lose either the patch order or the spatial order of pixels within each patch. As shown above, first collapsing the \(X_{G}\in R^{P\times N\times d}\) to obtain \(X_{F}\in R^{H\times W\times d}\) and then using a \(1\times 1\) convolution to \(X_{G}\) adjustment to obtain the \(C\) dimensional features.