# Optimizing Mobile Vision Transformers for Land Cover Classification
## 3 Materials and Methods  
### Vision Transformers  
The architecture of ViT models can be seen in Figure 1. The ViT model reshapes the input image tensor into a sequence of flattened patches with dimensions \(3P\times N\). The dimension \(3P\) is obtained by multiplying the height and width of the pixels in the patches to produce \(P\). Then, \(P\) is multiplied by the number of channels in the input image tensor to produce \(3P\). The dimension \(N\) represents the number of patches. The sequence of flattened patches is then projected onto a fixed dimensional space with dimensions \(d\times N\). The dimension \(d\) represents the size of the fixed dimensional space, while the dimension \(N\) represents the number of patches. Finally, a stack of \(L\) transformer blocks is used to learn long-range dependencies and global attention. The dimension \(L\) represents the number of transformer blocks utilized for this purpose. The mobile-friendly vision transformer or MViT model [15] contains MViT blocks that use convolution layers to generate local representations of the input tensor. Also, this block contains a transformer block with an MHSA mechanism that is used to generate global representations with spatial inductive biases that are fused with the local representations to preserve the benefits of transformers and CNNs. The overall architecture of the model is displayed in Figure 2a. The architecture starts with a striped \(3\times 3\) convolutional layer. This layer is followed by four MobileNetv2 (MV2 blocks). The second and fourth blocks both use a stride of 2. These MV2 blocks are quite narrow and shallow, which means they do not significantly factor into the training parameter count. This is because their main responsibility is downsampling. These blocks are followed by an MViT block, which utilizes two transformer blocks that are represented by \(L=2\). Also, the spatial dimensions of the feature maps are often multiples of 2. As a result, the height and width dimensions of the feature maps, represented by \(h\) and \(w\), respectively, are set to 2 at all spatial levels. Another MV2 block, with a stride of 2, is used before an MViT block with 4 transformer blocks. A dimension of 2 for the height and width spatial dimensions of the feature maps is applied. Then, an MV2 block with a stride of 2 is applied before the final MViT block, which has 3 transformer blocks and a dimension of 2 for the height and width spatial dimensions of the feature maps. Finally, a \(1\times 1\) convolutional layer and a global average pooling operation for spatial data are applied to produce the logits or the output of the last layer inside the model. Also, the output spatial dimensions of the model get smaller as the model gets closer to generating the logits. The output spatial dimensions used are \(128\times 128\), \(64\times 64\), \(32\times 32\), \(16\times 16\), \(8\times 8\), and \(1\times 1\). Figure 1: Architectureof the vision transformer model. The architecture of one MViT block is displayed in Figure 2b. The MViT block receives an input tensor with the dimensions \(C\), \(H\), and \(W\), which represent the channels, height, and width of the input tensor. Then, an \(n\times n\) convolutional layer and a \(1\times 1\) convolutional layer are applied to the tensor to encode local spatial information and project the tensor to a high-dimensional space, respectively. As a result, this produces a modified tensor with the dimensions \(d\), \(H\), and \(W\), where \(d\) is the size of the fixed dimensional space, while the dimensions \(H\) and \(W\) represent the height and width of the input tensor. After this, the modified tensor is unfolded into non-overlapping flattened patches with the dimensions \(d\), \(N\), and \(P\), which represent the size of the fixed dimensional space, the number of patches, and the product of the height and width of the pixels in the patches, respectively.