# Optimizing Mobile Vision Transformers for Land Cover Classification
## 2 Previous Work  
MViT has been applied for various purposes in fields where computer science is heavily applied, as demonstrated in [17]. In this research, the authors create a drone detection algorithm that has a lightweight MViTv1 backbone feature extractor and multi-scale attention feature fusion network (CA-PANNet). The backbone allows the algorithm to fully extract local and global features due to its combined CNN and transformer architecture, which helps the network extract target location information with high accuracy while having a lesser number of parameters relative to other methods. Another application was outlined in [18]; the authors create an automatic diabetic retinopathy (DR) grading framework that has a ResNet101 (CNN) backbone and a custom MViT-Plus backbone to extract local and global information that is fused to assign DR grades (none, mild, moderate, or severe) for 2D fundus images. The custom MViT-Plus backbone was made during the research and was made using depthwise separable convolutions as well as MViT-Plus blocks to replace the transformer block with a lightweight transformer block. As a result, the model can obtain results quickly with lower costs relative to ViT models while achieving better results when compared to MViT and other models such as Resnext101 [19] and Se_resnet101 [20]. Although work applying the relatively lightweight but powerful MViT to different fields such as drones and healthcare is important, this research highlighted the fusion of ResNets with MobileNets, which increases model complexity. ResNets are, to some extent, costlier replacements to options like pooling, MobileNet [21], and ShuffleNet [22]. Additionally, the model was trained to predict only five classes compared to the extensive number of land cover classes in geospatial datasets with textual variation. Two noteworthy contributions in this area are the studies by [PERSON] et al. (2023) [23] and [PERSON] et al. [24]. [PERSON] et al. introduced LTNet. This novel model fuses CNNs with transformers, focusing on efficient scene analysis in remote sensing. LTNet incorporates a multi-level group convolution module and the LightFormer block to effectively balance local and long-range dependencies. Its efficiency is marked by reduced parameters, leading to enhanced performance with shorter training durations, which is a critical aspect in rapid-deployment scenarios. Conversely, [PERSON] et al. 's work presented MLDANets. This model is designed for swift change detection and employs a unique attention aggregation and flexible sampling strategy. The MLDANets model excels at capturing the intricate details necessary for analyzing images captured at different times and showcases computational efficiency and strategic multi-level information coordination. Another work, proposed by [PERSON] et al. [25], developed a spatial-channel-feature-preserving ViT (SCViT) model that added a progressive aggregation (PA) strategy capable of combining neighboring tokens that overlap so that spatial information can be retained. The process was able to increase land cover classification accuracy. However, the number of parameters used for training was significantly large. For example, the SCViT-L variant utilized more than 40 million trainable parameters. With a vision transformer at its core, the model lacks the benefits of convolutional layers, so these complex iterative additions may hold the key to exploring simpler channel shuffling operations from lightweight CNN models used in the past. A list of notable works in the domain of reducing deep learning trainable parameters in geospatial applications is presented in Table 1. While these results are promising compared to the baseline architectures, the number of training parameters is still significantly higher compared to the modifications proposed in this research. Moreover, usage of pre-trained models freezes the majority of trainable parameters, rendering them unusable.