## 3 Bearing Fault Recognition Based on SSL-VCST Method
### Architecture of VCST Network  
Here, a basic classification network, VCST, is constructed by combining a CNN with a Swin Transformer (Swin-T) based on variational attention. The structure of the VCST network is shown in Figure 4.  
The convolutional network module is used as a primary feature extractor to extract and integrate local features and global information from input images through convolutional layers, GELU activation functions, and batch normalization layers. Feature images output from the convolutional network module are input to Swin-T, and feature extraction and representation learning are carried out through four layers of Stage. Each Stage consists of a Patch Merging module and a Swin-T Block. The Patch Merging module is used to reduce dimensionality and reorganize image features. The Swin-T Block is responsible for multi-scale and multi-location abstraction as well as the encoding of image features. The structures of two consecutive Swin-T Blocks are shown in Figure 5. The first block consists of a Multilayer Perceptron (MLP), a Window Multihead Self-Attention Mechanism (W-MSA), and a Layer Normalization (LN) Block. The MLP is used for nonlinear transformations and mappings of input features, W-MSA is used to capture local and global feature relationships, and LN is used to normalize the outputs of each layer. The second module replaces the W-MSA in the first block with a Moving Window Multihead Self-Attention (SW-MSA), while the remaining parts remain unchanged. W-MSA calculates self-attention separately by dividing an image into different windows. Feature maps are uniformly divided into small windows via window segmentation. Then, pixels are moved using a window configuration different from the previous layer's window configuration. Residual computation is applied after each block, and successive Swin-T blocks are computed as follows:  
\[ \begin{split}&\hat{z}^{l}=W-MSA\Big{(}LN\Big{(}z^{l-1}\Big{)} \Big{)}+z^{l-1}\\ & z^{l}=MLP\Big{(}LN\Big{(}\hat{z}^{l}\Big{)}+\hat{z}^{l}\\ &\hat{z}^{l+1}=SW-MSA\Big{(}LN\Big{(}z^{l}\Big{)}\Big{)}+z^{l}\\ & z^{l+1}=MLP\Big{(}LN\Big{(}z^{l+1}\Big{)}\Big{)}+z^{l+1}\end{split}  \tag{13}\]  
where \(\hat{z}\) denotes the (S)W-MSA output characteristics of block \(l\), and \(z^{l}\) denotes the MLP output characteristics of block \(l\). Figure 4: Network structure of VCST. In the encoding process of Swin-T, let the length of input encoding be S and the dimension of single embedding space be \(D\). The encoding can be represented as an \(S\times D\) matrix. By multiplying by three weight matrices \(Wq\), \(Wk\), and \(Wv\) of size \(D\times d\), three coding matrices \(Q\), \(K\), and \(V\) of size \(S\times d\) are obtained. The self-attention mechanism is calculated as follows:  
\[Attention(Q,K,V)=softmax\bigg{(}\frac{QK^{T}}{\sqrt{d}}+B\bigg{)}V \tag{14}\]  
where \(d\) is the dimension of Q/K along each axis in the range of relative positions \([-M+1,\)\(M-1]\) parameterizing a smaller size bias matrix \(\hat{B}\in R^{(2M-1)\times(2M-1)}\), and B is taken from \(\hat{B}\). A variational attention mechanism is introduced after the attention matrix of Equation (14) to continue the computation. First, random zero padding is applied to the attention matrix using a dropout operation with a probability of 0.1, i.e., each element has a probability of 10% being zeroed during training. The masking of attention weights introduces randomness to reduce overfitting. Then, a normalization operation is performed on the attention matrix to ensure that the sum of attention weights is 1 so that it obeys a probability distribution.