# Optimizing Mobile Vision Transformers for Land Cover Classification
###### Abstract
However, these works do not address an MViT model that combines CNNs and transformers. As a result, experiments do not support how computationally expensive CNN layers are and if modifying them would provide a better outcome. There are significant drawbacks of ViT, such as the exorbitant amount of training parameters as well as the high computational costs, which include the excessive number of images are required to train the model. Although transformers can be applied for image analysis, CNNs are generally the dominant model for most aspects of computer vision. CNN models are typically more compact and resource-efficient, while transformers are usually large: requiring a significant number of graphics processing units (GPUs) for training. However, transformers can get contextual understanding and global dependencies from images using self-attention, unlike CNNs, which typically use local operations that are restricted to small parts of images. This research demonstrates that a combination of CNNs and transformers can be optimized to extract both local and global contextual information for image classification. To achieve this objective, we propose modifications to the mobile vision transformer (MViT) [15] model, as it closely relates to a blend of CNN and vision transformers. Our findings demonstrate that higher classification performance can be achieved even by using lightweight convolutional variants, but only if they are used strategically in the entire architecture. Motivated by our previous work on ShuffleNet optimization [16], this research further solidifies our approach to building lightweight deep learning models without reducing accuracy. Building upon these intricate models, our research opts for a simpler, less parameter-intensive architecture. This strategic simplicity aims to explore the potential of efficient training while maintaining accuracy in remote sensing image analysis. Our focus is on practicality and applicability, particularly in GIS scenarios, where resource efficiency is a key concern. As we progress to the specific methods in the following sections, our emphasis on the practical advantages of reduced complexity remains a cornerstone of our approach. In summary, here are our main contributions in this paper:  
* Model variants reduce the original MViT's training parameters by replacing expensive CNN modules with a combination of average pooling, depthwise separable convolution, and ShuffleNet blocks. Our models retain the benefits of CNN and transformers and use them to boost performance on geospatial datasets without some of the other unnecessary costs. * The usage of convolution layers combined with the self-attention layers of transformers inside the MViT variants provides better performance across all geospatial datasets when compared with the standalone ViT model, which uses 95% more parameters than the MViT variants. * We test our proposed architectures on four geospatial datasets; we generate several models and conform to testing standards as presented in the literature. The trained models are also made available on GitHub ([[https://github.com/rahulgomes19/gistransformer](https://github.com/rahulgomes19/gistransformer)]([https://github.com/rahulgomes19/gistransformer](https://github.com/rahulgomes19/gistransformer)) (accessed on 23 September 2023)) for benchmarking and research purposes.