# Research on State Diagnosis Methods of UHF Partial Discharge Sensors Based on Improved ViT
## 2 Introduction of State Diagnostic Methods Based on ViT Models  
The ViT model first cuts the input image into multiple patch blocks and then transforms each block into a fixed-length vector, which is then fed into the Transformer for processing. It is worth noting that the accuracy of the model increases as the number of divided patches increases. but the training time will also increase significantly. Based on the findings in the literature [32], 16 \(\times\) 16 patches were chosen. The subsequent encoding process follows the same pattern as the original Transformer. Meanwhile, a special token is added to the input sequence to classify the image. The output corresponding to this token is the final predicted category output. According to the Figure 1, a ViT block can be operated according to the following steps: The image is divided into 1 to 9 patches. After feature extraction through a combination of LayerNorm and ResNet layers, the extracted features are fused with positional information to form feature data corresponding to the numbers 0 through 9. Finally, the data is processed through a Transformer encoder followed by a fully connected layer to produce the model's classification result. The Vision Transformer (ViT) constitutes a significant departure from traditional image-processing methods used in convolutional neural networks (CNNs), offering a novel  
Figure 1: Transformer Model Architecture. and effective approach to visual tasks. Instead of employing convolutions, which extract features through local receptive fields, ViT begins by dividing the input image into fixed-size, non-overlapping patches, typically measuring 16 \(\times\) 16 pixels. These patches are then flattened into vectors, which are subsequently linearly embedded into a higher-dimensional space. Each of these vectors, or \"tokens\", represents a specific part of the image and serves a similar function to words in natural language processing (NLP) tasks, acting as individual units of information within the model. The collection of these patch tokens forms the input sequence for the Transformer model. At the initial stage, the image is split into small patches, each of which undergoes a transformation to create a low-dimensional vector. This step parallels traditional feature extraction in image processing, whereby essential features are distilled from the raw pixel data. However, unlike CNNs, which naturally capture the spatial relationships between nearby pixels due to their local receptive fields, ViT models require additional mechanisms to maintain positional information. To address this, ViT introduces position embeddings--learnable vectors added to each patch embedding. These position embeddings allow the model to retain the spatial arrangement of patches, ensuring that the image's structure is preserved even when processed as a sequence of patches. To achieve patch embedding, ViT applies a linear layer (such as a fully connected layer) to each patch, transforming it into a higher-dimensional vector. This transformation is critical for encoding the visual information into a format that the Transformer can process. In addition to the patch embeddings, the position encodings are added, ensuring that the model captures both the content and the location of each patch. Once these embeddings are computed, they are concatenated into a single sequence, which is then passed into the Transformer encoder. The Transformer encoder, a key component of ViT, consists of multiple layers, each containing a self-attention mechanism and a feed-forward network (FFN). These layers are stacked, and each includes residual connections and layer normalization to stabilize training and facilitate better learning. As the sequence of patch tokens moves through the encoder, the self-attention mechanism allows the model to consider all token relationships simultaneously, making it capable of learning long-range dependencies and complex contextual information across the entire image.