## 3 The Proposed Method
### Specific Details of the Fault Diagnosis Method (ISCV-ViT)
#### 3.2.3 Transformer Blocks  
In ViT, feature extraction is carried out by the serially stacked transformer blocks, and the classification task is performed with the features corresponding to the class token at the end. The core architecture of each transformer block in ViT is composed of MSA and multilayer perceptron (MLP) modules connected alternately. The Transformer block includes layer normalization, MSA, a dropout layer, a residual connector, and MLP. MSA is the most critical component and has been described in detail in Section 2.2. The specific details of the MLP module are described as follows.  
The MLP module is mainly composed of three parts: fully connected (FC) layers, a Gaussian error linear unit (GELU) function, and a dropout layer. It is worth noting that the shapes of the input and output are consistent before and after the transformer encoder. This is because there are two fully connected layers in the MLP. The first FC layer will quadruple the input dimension (\(4d_{model}\)), and the second FC layer will restore the original feature dimension (\(d_{model}\)). Different from the vanilla Transformer, ViT uses GELU as the activation function. Compared with the rectified linear unit (ReLU) function, it can retain not only the probability of the activation parameter value but also the dependence on input. Then, the data passes through the dropout layer, which can effectively alleviate overfitting and achieve regularization to some extent. The GELU activation function is defined as follows:  
\[\text{GELU}(x)=x\Phi(x)=x\cdot\frac{1}{2}\Big{[}1+erf\Big{(}x/\sqrt{2}\Big{)} \Big{]} \tag{8}\]  
The result can be estimated as follows:  
\[0.5x\Big{(}1+tanh\Big{[}\sqrt{2/\pi}\Big{(}x+0.044715x^{3}\Big{)}\Big{]}\Big{)} \tag{9}\]  
where \(\Phi(x)\) is the standard normal distribution function and \(erf(\cdot)\) represents the Gaussian error function.  
Finally, ViT realizes data transfer between the MSA module and MLP module through residual connections. In addition, residual connections play a great role in the transfer of features from the bottom to the top and effectively avoid the disappearance of the gradient.  
For this step, it can be seen from the previous section that the one-dimensional input embeddings \(z_{0}=\big{[}z_{0}^{0}\); \(z_{0}^{1}\); \(\cdots\); \(z_{N}^{N}\big{]}\) are formed by patch embeddings, position embeddings, and learnable embeddings. Then, they are fed into the Transformer encoder, and the calculation process is shown in Equations (10)-(12). Finally, the learnable class tokens \(z_{L}^{0}\) are extracted as the overall feature, which is input into the MLP classifier to implement the fault classification task. The MLP classifier is a simple neural network composed of an FC layer. Significantly, the base 16 version of ViT is chosen as the basic model according to the influence of the number of attention heads on the behavior of the model. The structure and main hyperparameters of the proposed ISCV-ViT are shown in Table 1. The data information flowchart of the ISCV-ViT model is shown in Figure 5.  
\[z_{l}^{\prime}=\text{MSA}(LN(z_{l-1}))+z_{l-1},l=1,\cdots,L \tag{10}\]  
\[z_{l}=\text{MLP}(LN\big{(}z_{l}^{\prime}\big{)})+z_{l}^{\prime},l=1,\cdots,L \tag{11}\]  
\[y=LN\Big{(}z_{L}^{0}\Big{)} \tag{12}\]  
where \(z_{l}^{\prime}\) is the operation result of the \(l\)th MSA module, \(z_{l}\) represents the output after one MSA and MLP operation in the \(l\)th Transformer block, \(L\) is the number of Transformer blocks that can be optionally stacked, and \(y\) is the output of the Transformer encoder.