# Optimizing Mobile Vision Transformers for Land Cover Classification
## 3 Materials and Methods
### Proposed Lightweight Transformer Modifications  
The first variant tested in our research is called MViT-Depth. To create this variant, a \(1\times 1\) convolution layer in the local representations section of the MViT block was removed. An average pooling layer that features a pool_size of \(2\times 2\) was added. A \(1\times 1\) upsampling layer was introduced in between the folded global feature map; this layer contains the global features extracted by the transformer block and the concatenate layer, which combines the folded global features with the local features extracted using the \(N\times N\) convolution layer and the average pooling layer. We further modified the MViT block inside the model by replacing the first \(N\times N\) convolution layer, which takes in the input tensor to generate local representations, with an \(N\times N\) depthwise separable convolution layer. When compared to the ViT model, MViT-Depth has 20,509,607 fewer parameters than ViT on average (94.62% decrease). When compared to the MViT model, MViT-Depth had exactly 145,200 fewer parameters than MViT across all four datasets (11.06% decrease). The architecture of the MViT-Depth block is shown in Figure 3. This block receives an input tensor with the dimensions \(C\), \(H\), and \(W\). Then, the model modifies the local representations section by using a \(n\times n\) separable convolutional layer instead of a regular convolutional layer as well as an average pooling layer with a pool size of \(2\times 2\) to encode local spatial information and project the tensor to a high-dimensional space, respectively. This produces a modified tensor with the dimensions \(d\), \(H\), and \(W\) that passes through the transformer layers as explained in the previous MViT-Avg section. The new sequence of flattened patches is folded and projected to a high-dimensional space to make a tensor with dimensions \(d\), \(H\), and \(W\). Then, a \(1\times 1\) convolutional layer is applied to the tensor to project it to a low-dimensional space with dimensions \(C\), \(H\), and \(W\). The tensor is then fed into an upsampling layer that helps balance the class labels followed by concatenation with the input tensor to produce a new tensor with dimensions \(2C\), \(H\), and \(W\). Finally, an \(n\times n\) convolutional layer is applied to fuse the concatenated features and generate an output tensor with the same dimensions \(C\), \(H\), and \(W\). In the MViT-combined variant, a custom ShuffleNet block replaces the final MViT block, as shown in Figure 2a, to create a new model, as shown in Figure 4a. This ShuffleNet block sets the default kernel size of its layers to be \(1\times 1\), and it sets the default number of filters to be 320. The custom ShuffleNet block can be seen in Figure 4b. This block consists of two pathways of layers that are fused to generate the output  
Figure 3: Architecture of the proposed MViT-Depth transformer block, with changes in red. receive an input tensor with dimensions \(C\), \(H\), and \(W\). The first pathway of layers is called \(A\), and it starts by feeding the input tensor into a \(3\times 3\) depthwise convolution layer. Then, the output from that initial layer is processed by a batch normalization layer as well as a \(1\times 1\) convolution layer. Finally, the pathway ends with the output being processed by another batch normalization and a ReLU layer. The second pathway of layers is labeled with \(B\), and it starts by feeding the input tensor into a \(1\times 1\) convolution layer. The output from the initial layer is processed by a batch normalization layer and a ReLU Layer. Then, the new output is processed by a \(3\times 3\) depthwise convolution layer as well as another \(1\times 1\) convolution layer. The activation map is then processed by another batch normalization layer and another ReLU layer. To fuse the two pathways, a concatenate layer is utilized. The fusion results in an output tensor with dimensions \(C\), \(H\), and \(W\).