## 3 Methods
### Multimodal Fusion for Decision-Making  
STFT image samples and time series samples are fed into RGCNN and 1 DCNN, respectively. Both neural networks are trained simultaneously, and the model parameters are optimized to obtain a basic classifier for rolling bearing fault diagnosis. Comprehensive fault diagnosis results can be achieved through multimodal decision fusion, which is implemented by integrating multiple different deep learning models.  
The output features from the two networks are integrated using a weighted fusion strategy. We employ an adaptive weight adjustment mechanism to dynamically adjust the weights based on the importance of different modal features. This strategy ensures that the features from each modality are appropriately considered in the final classification decision.  
The two feature vectors, F1 and F2, come from the two networks. We aim to obtain a fused feature vector F through a weighted fusion strategy. We can express F as follows.  
\[\mathrm{F}=\alpha\mathrm{F1}+(1-\alpha)\mathrm{F2} \tag{11}\]  
\(\alpha\) is a weight that represents the importance of the first feature vector F1 in the fused feature vector F. To enable the adaptive adjustment of \(\alpha\), we can implement an adaptive weight adjustment mechanism. This mechanism dynamically adjusts \(\alpha\) based on the importance of different modal features.  
Assume we have a loss function L that represents the performance of the fused feature vector F in the classification task. We can optimize the weight \(\alpha\) using gradient descent to minimize the loss function L. Specifically, we can update \(\alpha\) through gradient updating as follows.  
\[\alpha=\alpha-\lambda\partial L/\partial\alpha \tag{12}\]  
\begin{table}
\begin{tabular}{c c c c c c c} \hline
**Layer** & **Type** & **Kernel** & **Channel** & **Stride** & **Pading** & **OUTPUT** \\ \hline
1 & INPUT & - & - & - & - & 2048 \(\times\) 1 \\ \hline
2 & Conv & 32 \(\times\) 1 & 32 & 1 & Yes & 2048 \(\times\) 32 \\ \hline
3 & Conv & 1 \(\times\) 1 & 32 & - & Yes & 2048 \(\times\) 32 \\ \hline
4 & AVP & 32 \(\times\) 1 & - & - & - & 2048 \(\times\) 32 \\ \hline
5 & Conv & 16 \(\times\) 1 & 32 & 2 & Yes & 1024 \(\times\) 32 \\ \hline
6 & Conv & 1 \(\times\) 1 & 32 & - & Yes & 1024 \(\times\) 32 \\ \hline
7 & AVP & 32 \(\times\) 1 & - & - & - & 1024 \(\times\) 32 \\ \hline
8 & Conv & 9 \(\times\) 1 & 64 & 2 & Yes & 512 \(\times\) 64 \\ \hline
9 & Conv & 1 \(\times\) 1 & 64 & - & Yes & 512 \(\times\) 64 \\ \hline
10 & AVP & 32 \(\times\) 1 & - & - & - & 512 \(\times\) 64 \\ \hline
11 & Conv & 6 \(\times\) 1 & 64 & 2 & Yes & 256 \(\times\) 64 \\ \hline
12 & Conv & 1 \(\times\) 1 & 64 & - & Yes & 256 \(\times\) 64 \\ \hline
13 & AVP & 32 \(\times\) 1 & - & - & - & 256 \(\times\) 64 \\ \hline
14 & Conv & 3 \(\times\) 1 & 128 & 4 & Yes & 64 \(\times\) 128 \\ \hline
15 & Conv & 1 \(\times\) 1 & 128 & - & Yes & 64 \(\times\) 128 \\ \hline
16 & AVP & 32 \(\times\) 1 & - & - & - & 64 \(\times\) 128 \\ \hline
17 & Conv & 3 \(\times\) 1 & 128 & 4 & Yes & 16 \(\times\) 128 \\ \hline \multicolumn{6}{c}{Global AVP} \\ \hline \multicolumn{6}{c}{SoftMax} \\ \end{tabular}
\end{table}
Table 1: Network configuration of The 1 DCNN architecture.  
\(\lambda\) is the learning rate, which is used to control the rate at which the weights are updated. In this way, \(\alpha\) can be automatically adjusted according to the changes in the loss function L, ensuring that the features of each modality receive appropriate attention in the final classification decision.  
\[\mathrm{P(x)}=\mathrm{argmax}(\alpha\mathrm{F1}+(1-\alpha)\mathrm{F2})/2 \tag{13}\]