# Research on State Diagnosis Methods of UHF Partial Discharge Sensors Based on Improved ViT
## 3 Methodological Improvements Based on the ViT Model  
One of the significant challenges associated with the Vision Transformer (ViT) model is its fixed approach to classifying input images into image patches of a specified resolution. This method poses a particular problem when dealing with image data collected from various sensors, which often operate at different resolutions. Uniformly compressing these diverse images into a standardized P*P resolution can significantly distort the inherent features and details present in the original images. As a result, this misalignment may adversely impact the final classification outcomes produced by the ViT model, leading to reduced accuracy and reliability in tasks that depend on precise image analysis.  
Moreover, another critical drawback of the ViT model, particularly in the context of state diagnosis, lies in its reliance on global self-attention mechanisms inherent to both the standard Transformer architecture and its adaptations for image classification. In these models, the relationships between markers--essentially the key elements or features identified within the images--are computed against all other markers. This global computation results in a quadratic complexity relative to the number of markers involved. Such a scaling issue renders the ViT model less efficient and suitable for various vision problems, particularly those that necessitate handling large sets of markers for intensive prediction tasks or require the representation of high-resolution images.  
To address these pressing issues, this paper presents enhancements to the ViT model. These improvements aim to optimize the processing of input images with varying resolutions and to refine the attention mechanism, thereby reducing the computational complexity associated with global self-attention. Therefore, FFN-based training was chosen. In transformers, the Feed-Forward Network (FFN) within each block is crucial for processinginformation post-attention. It is worth noting that FFNs are computationally efficient and straightforward, consisting of two linear transformations with non-linearity in between. This simplicity ensures fast training and inference. FFNs are a core component of transformer architectures like BERT, GPT, and the original ViT, providing strong evidence of their effectiveness. The structure of FFNs complements the attention mechanism by providing localized feature transformation, effectively balancing global and local information. In addition, Recurrent Neural Networks, Gated Mechanisms, Convolutional FFNs and Advanced Alternatives are all alternative training methods. However, RNNs are unsuitable due to vanishing gradient issues and inefficiency in processing long sequences. They are less efficient and overkill for vision tasks where spatial hierarchies dominate. Moreover, they will reintroduce locality biases, defeating the purpose of transformers' global view. They can improve specificity but add complexity and computational overhead, which is often unnecessary for general vision tasks. By implementing these modifications, we anticipate achieving more accurate classifications while maintaining efficiency in processing, ultimately enhancing the model's applicability to a wider range of vision-related challenges.