# A Self-Supervised Detail-Sensitive ViT-Based Model for COVID-19 X-ray Image Diagnosis: SDViT
## 2 Related Work  
The Transformer [3] method was originally designed for natural language processing (NLP). With the efforts of scientists and researchers from different fields, the Transformer model has been widely applied to various tasks and produced outstanding results. Multiple variations have been designed to improve performance on different tasks, including Bert [4], ALBERT [5], and RoBERTa [6]. Transformer-based models have also been applied to computer vision tasks, which require unique designs for specific tasks. By making adjustments to the way in which data are handled, the Transformer model can be applied to extended fields, for example, DETR [7] for object identification and SETR [8] for semantic segmentation. The success of the Transformer model in these tasks demonstrates that it is applicable to image classification tasks. It can efficiently capture the crucial semantic information in the image and combine it to classify the target. With the growing demand for medical care, efforts are being made to effectively diagnose issues from medical images by fusing computer vision technologies with medical services. ViT [9] and DeiT [10] can be used for medical diagnosis to solve the most fundamental and pertinent image classification problems in computer vision. In order to further improve the accuracy of diagnosis in the medical field, deep learning technologies can be used for technological innovation in areas such as tumor detection [11], medical image semantic segmentation [12], and other model tasks [13]. In 2021, MSRA proposed the Swin Transformer [14] model. It significantly improves the outcomes of the ViT family for classification, segmentation, and detection tasks. However, due to its arithmetical power limitations, window-based self-attention is used to reduce computational consumption. This reflects an issue with ViT, which consumes a lot of computational resources even though the Transformer structure is very efficient. Additional model optimization is necessary in order to lower the computational consumption. With the wide spread of COVID-19, it is essential to create an appropriate recognition model. In [15], a novel attention-based deep learning model was proposed using an attention module with VGG-16. This approach was able to capture the spatial relationship between the regions of interest (ROI) in CXR images. A novel deep-learning model was defined for the classification process to specifically perform fine-tuning. The experimental results indicated that this method is efficient in both classification accuracy and training parameters. In [16], CovidDetNet, a novel deep learning-based approach, was proposed for the automatic and reliable diagnosis of COVID-19 using chest radiographs. It has ten learnable layers, nine of which are convolutional layers, along with a fully-connected layer. The novelty of this model is that it uses two activation functions (the ReLu activation function and the Leaky Relu activation function) and two normalization operations (batch normalization and cross channel normalization). Through experiments on a standard COVID-19 radiography database, the model was shown to be superior to existing models, and outperformed the baseline models. However, in the field of medical imaging there is no pneumonia classification model specifically created for X-ray images. Because COVID-19 is a very recent disease, the research on it has not been not systematic. In addition, the related datasets are limited, which necessitates exploring the intrinsic linkages of the data as much as possible in order to obtain an accurate diagnosis. Table 1 highlights the contributions along with the pros and cons of prior studies.