# A Study on Small-Scale Snake Image Classification Based on Improved SimCLR
## 2 Materials and Methods
### Related Network
#### 2.3.2 Improvements to the Feature Encoding Module  
Transformer is a deep neural network with a self-attention mechanism proposed by the Google team [28], and it has a wide range of applications in Natural Language Processing (NLP). In 2020, Transformer was introduced into computer vision [18], and ViT demonstrated the image classification task. Swin Transformer is an improvement of the ViT series [29], which improves the problem of high computation and poor fine-grainedness of ViT networks. The model under discussion is a visual Transformer model based on a Window Self-Attention mechanism. The sliding window mechanism is adopted to capture both local and global features. This results in a low computational complexity and an improvement in the efficiency of feature extraction. In this study, we adopted the Swin Transformer as the feature extractor in the contrast learning framework to enhance the model's capacity to characterize images of exotic snakes. The structure of Swin Transformer is illustrated in the Figure 4, which is divided into four phases. Initially, the Patch partition divides the image into equal-sized image blocks, and then the image is flattened in the channel direction. Linear Embedding embeds the image blocks and changes them linearly, and then the deep features of the image are gradually extracted through several Block, Block, and Patch Merging operations [30]. We used Swin Transformer as the backbone network and removed the final classification layer to obtain a 1024-dimensional global feature vector. We then designed a projection head for mapping high-dimensional features onto a low-dimensional contrast space to optimize the contrast loss computation, which consists of a two-layer fully connected network. 1. Multiple self-attention mechanisms based on shift windows  
The window-based multihead self-attention module (W-MSA) and the sliding window multihead self-attention module (SW-MSA) are the core parts of the Swin Transformer block. The specific structure is shown in the Figure 5:  
Among them, \(z^{t}\) and \(z^{t}\) represent the output features of the (S) W-MSA module and the MLP module of the t-th block, respectively, where LN stands for layer norm layer, and MLP stands for multilayer perceptron. The LN module normalizes the input features, ensuring training stability and faster convergence. The W-MSA module segments the input feature map into fixed-size, non-overlapping windows and applies self-attention independently within each window. This local attention design significantly reduces the computational complexity from quadratic to linear concerning image size while still allowing the model to capture rich local dependencies. Figure 4: Swin Transformer flow chart. Figure 5: Swin Transformer block. To compensate for the limited receptive field caused by the non-overlapping nature of W-MSA, the SW-MSA module introduces a window shift mechanism. Specifically, it shifts the window partitions by a fixed offset before applying attention. This causes each token to interact with new neighbors across window boundaries in alternating layers, enabling cross-window connections and allowing information to propagate beyond local regions. Overall, the alternating use of W-MSA and SW-MSA modules allows Swin Transformer to model local and global relationships in visual data efficiently, balancing computational efficiency with representational power. 2. Projection head  
Within the framework of contrastive learning, the projection head is pivotal in establishing a connection between the backbone network and the contrastive loss. This paper adopts a non-linear projection structure comprising two fully connected layers. This is shown in the Figure 6 below. Firstly, the 1024-dimensional features extracted by the Swin Transformer backbone network are compressed to 512 dimensions through a linear transformation layer. This dimensionality reduction helps eliminate redundant information and improves computational efficiency.