# Optimizing Mobile Vision Transformers for Land Cover Classification
## 5 Discussion  
This work introduced variants of the benchmark MViT model and presented their training and evaluation outcomes on four GIS datasets: AID, EuroSAT, UC-Merced, and WHU-RS19. The results highlighted that MViT-Combined outperformed both the benchmark MViT and ViT architectures, achieving this with a significantly reduced number of parameters. While the SWIN models, including SWIN-ConvBlock and SWIN-ShuffleNet, did not surpass the MViT variants or the ViT in terms of performance, they demonstrated a commendable balance between parameter efficiency and accuracy, contributing a valuable perspective on model optimization. As a result, we can reason that our methods retained the benefits of CNNs and transformers while replacing some of the expensive deep-learning computational layers and boosting accuracy and reducing the number of training parameters. The activation maps from these datasets can be seen in Figure 11. It is worth noting that, along with the successful variants, we also explored other modifications to CNN architectures that did not have a similar outcome. Five of the unsuccessful MViT variants involved removing entire MViT blocks from the overall architecture of the benchmark model in an attempt to decrease the number of training parameters. All of these variants had their respective architectures configured on the UC-Merced dataset and a 50-50 train-test split. One of these variants had a modified structure that removed the first MViT block from the overall MViT architecture. The removal resulted in the variant having 1,112,725 training parameters, which meant the number of training parameters was reduced by 15.24% relative to the benchmark MViT architecture. However, this variant could not maintain the benchmark's performance, as it achieved an accuracy of 78.00%, which was 9.14% lower than the 87.14% accuracy achieved by MViT. In addition, we created another variant that removed the last MViT block from the overall MViT architecture. The removal resulted in the variant having 687,605 training parameters, which meant the number of training parameters was reduced by 47.62% relative to the benchmark. Unfortunately, the variant attained a testing accuracy of 83.24%, which was lower than the benchmark's accuracy. The other three out of the five variants for which we removed entire blocks from the overall architecture of the benchmark model focused on eliminating some of the inverted residual blocks. One of these variants had a modified architecture for which we removed the first inverted residual MV2 block from the overall benchmark architecture. This removal resulted in the variant having 1,309,501 training parameters, which meant the number of training parameters was reduced by 0.25%. However, the variant could not achieve the benchmark's performance, as it attained an accuracy of 84.19%, which was 2.95% lower than the benchmark's accuracy. In addition, we also created another variant that removed the second inverted residual MV2 block from the overall benchmark architecture. This  
Figure 10: Training (**top row**) and testing (**bottom row**) losses on WHU-RS19 dataset across multiple splits. elimination resulted in the variant having 1,309,501 training parameters which meant the number of training parameters was reduced by 0.25%. Unfortunately, the variant also could not achieve the benchmark's accuracy as it attained an accuracy of 84.19% which was 2.95% lower than the benchmark's performance on the same dataset and train-test split. Our final unsuccessful variant featured changes that were similar to the more fine-tuned and subtle manipulations made on the benchmark MViT model to generate the three successful variants. This variant was also configured on the UC-Merced dataset and a 50-50 train-test split before being compared to the original MViT benchmark configured on the same dataset and train-test split. It had a modified structure that replaced the last N \(\times\) N convolutional layer in the MViT block's fusion section with an N \(\times\) N depthwise separable convolution layer.