# Research on State Diagnosis Methods of UHF Partial Discharge Sensors Based on Improved ViT
## 2 Introduction of State Diagnostic Methods Based on ViT Models
At the core of the ViT model is the multi-head self-attention mechanism, which allows the model to compute attention weights between all pairs of input tokens, thus capturing both local and global contextual information. Unlike CNNs, which are limited by their fixed receptive fields and can only process local regions at a time, self-attention offers ViT greater flexibility. It enables the model to capture dependencies between image patches, regardless of their distance, allowing it to recognize and understand long-range relationships that CNNs might miss. This makes ViT particularly well-suited for tasks that require a broader understanding of image structure. For image classification tasks, once the Transformer encoder has processed the sequence of patch tokens, the output of a special token, which is typically appended to the front of the patch sequence, is used. This token is designed to capture a global representation of the entire image. Its final state is fed into a classification head, which is a fully connected layer that maps the [CLS] token's representation to a probability distribution over the possible categories. This output provides the final classification result for the input image. \[\mathbf{z}_{0}=[x_{class};x_{p}^{1}E;x_{p}^{2}E;\ldots;x_{p}^{N}E]+E_{pos},E \in R^{(P^{2}\cdot C)\times D},E_{pos}\in R^{(N+1)\times D} \tag{1}\]  
\[z_{\partial}^{\prime}=MSA(LN(z_{\partial-1}))+z_{\partial-1},\partial=1\ldots L \tag{2}\]  
\[z_{\partial}=MLP(LN(z_{0}^{\prime}))+z_{\partial}^{\prime},\partial=1\ldots L \tag{3}\]  
\[y=LN(z_{0}^{\prime}) \tag{4}\]where \(z_{i}\), (\(i=1\dots L\)) represents the output of the model layers, \(i\)\(z^{\prime}\) represents the intermediate results of the model, \(x_{p}^{i}\), (\(i=1\dots N\)) represents the patch of each image, \(i\), \(x_{class}\) represents the types of the result, \(E\) represents one of the function modules in the model, \(E_{pos}\) represents the module for positional information transformation, \(LN\) corresponds to the averaging operation, \(MLP\) is the fully connected layer function, and \(MSA\) is a module of the Multiple Attention Mechanism. One of the most significant findings from the original ViT research is that ViT models outperform CNN when trained on extremely large datasets, such as JFT-300M (a dataset containing approximately 300 million images). This suggests that CNN's inductive biases, such as spatial locality and translation invariance, are less essential when massive amounts of training data are available. In these cases, the flexibility of the self-attention mechanism allows ViT models to learn richer, more expressive representations of the image data. ViT models have demonstrated excellent performance in transfer learning scenarios, where models are pretrained on large datasets and then fine-tuned on smaller datasets. In such cases, ViT has often surpassed CNN in benchmarks such as ImageNet, achieving better generalization across new tasks. It is worth noting that The Vision Transformer represents a paradigm shift in computer vision, trading the localized feature extraction of CNNs for a global, attention-based approach. While ViT shows impressive results, especially on large datasets, it complements rather than replaces CNNs, with hybrid architectures often combining the strengths of both for specific tasks. Despite its impressive performance on large datasets, ViT has some limitations, primarily in terms of computational cost. The self-attention mechanism has a quadratic complexity of O(N\({}^{2}\)), where CNN is the number of patches in the input image. This complexity makes ViT computationally expensive, especially for high-resolution images where CNN becomes large. In conclusion, the Vision Transformer offers a powerful new paradigm for image processing by leveraging the strengths of self-attention mechanisms, allowing for richer, more flexible representations of visual data.