## 2 Materials and Methods
### Methods
#### 2.2.1 Overview
\[X_{gaf}=H\textit{swish}\Big{(}BN\Big{(}Conv_{3\times 3}\Big{(}X_{gaf}\Big{)} \Big{)}\Big{)} \tag{1}\]4

\[X_{gaf}=H\textit{swish}\Big{(}BN\Big{(}Conv_{3\times 3}\Big{(}X_{gaf}\Big{)} \Big{)}\Big{)} \tag{1}\]5The outputs of all heads were concatenated and processed through a linear transformation layer to generate the final output. The complete structure diagram is shown in \[X_{gaf}=H\textit{swish}\Big{(}BN\Big{(}Conv_{3\times 3}\Big{(}X_{gaf}\Big{)} \Big{)}\Big{)} \tag{1}\]6

### Residual with Zero initialization

The proposed framework uses the ReZero mechanism [49] to address the gradient vanishing and explosion problems at different layers. Traditional residual connections propagate information by directly adding the input x to the output after nonlinear transformation. In contrast, the ReZero method introduces a learnable scaling factor, allowing the network to rely solely on the input \(x\) during the early stages of training, thus avoiding excessive interference from initial residual information. The residual connection formula for ReZero is as follows: for an input x, the output after processing by a sublayer is denoted as \(F(x)\). \[X_{gaf}=H\textit{swish}\Big{(}BN\Big{(}Conv_{3\times 3}\Big{(}X_{gaf}\Big{)} \Big{)}\Big{)} \tag{1}\]7

\(\alpha\) is a learnable scaling factor, initially set to 0 and gradually optimized during training. X is the input tensor, and \(F(\cdot)\) represents a layer. This method is particularly suitable for deep neural networks, especially in the early stages of the network, as it effectively improves training stability and convergence speed. ## 3 Results

### Training Environment and Parameter Settings

The experimental platform ran on the Windows 10 operating system, equipped with 64 GB of RAM and an RTX 3090 GPU. The code was written in the Python 3.12 programming language and implemented using the PyTorch 2.6 framework, diagnosing gearbox faults in the MCC5 and HUST gearbox datasets. During training, the CrossEntropyLoss loss function and AdamW optimizer were used, and the specific configuration of key hyperparameters is detailed in \[X_{gaf}=H\textit{swish}\Big{(}BN\Big{(}Conv_{3\times 3}\Big{(}X_{gaf}\Big{)} \Big{)}\Big{)} \tag{1}\]8 \begin{table}
\begin{tabular}{c c c} \hline - & **Parameter** & **Setting** \\ \hline \multirow{6}{*}{Data preprocessing} & & [i, i + 6], \\  & & i \(\in\) {0, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, \\  & Sample split & 48, 52]; \\  & & [i, i + 6], \\  & & i \(\in\) {0, 10, 20, 30, 40, 50]} \\  & Window size for PAA dimensionality reduction & 150 \\  & Number of signal channels & 8 \\  & Output GASE/GADF image resolution & 512 \\  & Shape of GAF (merges all channels of GASP/GADF) & (16, 512, 512) \\  & Patch size & 16 \\ \hline \multirow{6}{*}{Network} & Kernel size of depthwise convolution & \(3\times 3\), \(5\times 5\) \\  & Kernel number of depthwise convolution & 32 \\  & Number of attention heads & [1, 2, 3, 4] \\ \cline{1-1}  & Numerical stability parameters for LayerNorm & \(1\times 10^{-12}\) \\ \cline{1-1}  & The number of hidden layers in the transformer module & 6 \\ \cline{1-1}  & FFN hidden size & 2048 \\ \hline \multirow{6}{*}{Others} & Dropout & 0.1 \\  & Loss Function & CrossEntropyLoss \\ \cline{1-1}  & Batch size & 16 \\ \cline{1-1}  & Epochs & 256 \\ \cline{1-1}  & Learning rate & \(2\times 10^{-6}\) \\ \cline{1-1}  & Optimizer & AdamW \\ \hline \end{tabular}
\end{table} 
Table 2: Detailed configuration of key hyperparameters. Figure 8: Optimized self-attention model structure. ### Evaluation Criteria

In the experiments, several common evaluation metrics were selected to comprehensively assess the performance of the proposed model in gearbox bearing fault diagnosis, including accuracy, precision, recall, and F1-score. Accuracy was used to measure the proportion of correctly predicted fault categories. Precision evaluated the proportion of actual faults among all samples predicted as faults. Recall reflected the proportion of successfully identified faults in all actual fault samples.