## 1 Introduction
This is particularly noticeable when dealing with images that contain substantial objects or intricate relationships between remote regions. The Vision Transformer (ViT) [18] addresses this problem by leveraging the transformer architecture's attention mechanism, enabling efficient global information capture. ViT divides an image into fixed-size, non-overlapping patches. These patches are linearly projected by multiplying them with a learnable weight matrix and their positions are embedded with them. Finally, they are treated as a sequence. It utilizes the self-attention mechanism to identify long-distance dependencies, allowing for interactions between patches. ViT is now highly competitive in a variety of image classification tasks [19; 20]. ViT, however, still has trouble managing high-resolution images effectively. When applied directly to large images, the self-attention mechanism's quadratic complexity, with respect to the input sequence length, becomes a bottleneck and hinders scalability. The Swin Transformer [21] is useful in this situation. The Swin Transformer uses a hierarchical structure of self-attention mechanisms to handle high-resolution images. The image is split into non-overlapping windows and is processed hierarchically across several layers. As attention is calculated within local windows rather than across the entire sequence, this lessens the quadratic complexity issue. Swin Transformer outperforms some state-of-the-art architectures in various image recognition tasks. Therefore, it is currently dominating the field of computer vision. The rising complexity and resource requirements of contemporary deep learning models give rise to the need for knowledge distillation [22; 23]. These models are difficult to deploy on resource-restricted devices and in situations with constrained processing power as their sizes and computational requirements increase. By transferring the information and insights of a larger teacher model to a smaller student model through knowledge distillation, this problem is resolved [24]. This allows the student model to attain substantial performance while being more computationally efficient. This knowledge compression improves model interpretability, energy efficiency, and robustness while providing a realistic way to regularize model training. It also enables deployment on edge devices, mobile platforms, and low-resource contexts [25]. In order to address the need for accurate diabetic retinopathy classification using a lightweight image classifier, this article presents a knowledge distillation-based approach. The proposed solution involves a series of image preprocessing tasks to enhance the image quality and amplify the visibility of the interested areas of the retinal image. Subsequently, a robust teacher model is constructed by fusing Swin Transformer and ResNet152V2, which is leveraged to train the student model. The use of Swin Transformer and ResNet152V2 in the teacher model for knowledge distillation is supported by an amalgamation of architectural efficacy and task-specific considerations. Swin Transformer, a relatively recent approach in computer vision, has remarkable abilities; it extracts both the global and local context from images. It is highly suited for tasks like image categorization due to its hierarchical structure and self-attention mechanism. ResNet152V2, a deep form of the ResNet architecture, on the other hand, is known for its dependability and toughness in a variety of computer vision tasks. Combining these diverse architectures improves the process of knowledge transfer by offering the student model access to a wider range of knowledge. The student model is composed of Xception and a modified convolutional block attention module (CBAM). An empirical study of the portable model on the APTOS and IDRiD datasets attests to its effectiveness. In summary, this research paper has the following major contributions. * The research involves various image preprocessing techniques to enhance image quality and highlight key areas of interest in the image.