# Continuous Wavelet Transform and CNN for Fault Detection in a Helical Gearbox
## 5 Experiments and Discussions
### Experiments with the Proposed Defect Detection Models
Precision, recall, their harmonic mean, and F1 score [32] are metrics used to evaluate the model at the level of classes. The confidence interval for accuracy, \(CI=ma\pm 1.96\ \sigma/\sqrt{n}\)[33], at the 95% confidence level of the mean accuracy (\(ma\)) over \(n=10\) runs, with \(\sigma\) being the standard deviation, helps in verifying the model's stability. Tables 3-6 contain the global confusion matrices and the calculated performance metrics for all four defect detection models. We observe small confidence intervals (CI) for accuracy in all four models, so all are stable models. As expected, the best-performing model is Mcwt_xyz, with a mean accuracy of 99.91%, followed by Mcwt_y, Mcwt_z, and Mcwt_x, with mean accuracies of 99.54%, 99.15%, and 99.03%, respectively. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multirow{2}{*}{**Model Mcwt\_y**} & \multicolumn{5}{c}{**Predicted class**} \\ \cline{3-7}  & & **D1** & **D2** & **D3** & **D4** & **Recall (\%)** \\ \hline \multirow{4}{*}{**C**} & **D1** & 3878 & 9 & 0 & 3 & 99.69 \\ \cline{2-7}  & **D2** & 6 & 3643 & 9 & 2 & 99.54 \\ \cline{1-1} \cline{2-7}  & **D3** & 0 & 8 & 3644 & 8 & 99.56 \\ \cline{1-1} \cline{2-7}  & **D4** & 5 & 4 & 14 & 3637 & 99.37 \\ \hline \multirow{4}{*}{**C**} & **Precision (\%)** & 99.72 & 99.43 & 99.37 & 99.64 & **CI accuracy (\%)** \\ \cline{1-1} \cline{2-7}  & **F1 score (\%)** & 99.70 & 99.48 & 99.47 & 99.51 & **99.54 \(\pm\) 0.13** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Model Mcwt_y: global confusion matrix and the performance metrics for 10 runs. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multirow{2}{*}{**Model Mcwt\_x**} & \multicolumn{5}{c}{**Predicted class**} \\ \cline{3-6}  & & **D1** & **D2** & **D3** & **D4** & **Recall (\%)** \\ \hline \multirow{4}{*}{**C**} & **D1** & 3852 & 33 & 4 & 1 & 99.02 \\ \cline{2-6}  & **D2** & 13 & 3643 & 0 & 4 & 99.54 \\ \cline{1-1} \cline{2-6}  & **D3** & 1 & 1 & 3629 & 29 & 99.15 \\ \cline{1-1} \cline{2-6}  & **D4** & 3 & 2 & 53 & 3602 & 98.42 \\ \hline \multirow{4}{*}{**C**} & **Precision (\%)** & 99.56 & 99.02 & 98.45 & 99.06 & **CI accuracy (\%)** \\ \cline{1-1} \cline{2-6}  & **F1 score (\%)** & 99.29 & 99.28 & 98.80 & 98.74 & **99.03 \(\pm\) 0.18** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Model Mcwt_x global confusion matrix and the performance metrics for 10 runs. In Table 3 (Mcwt_x model) and Table 5 (Mcwt_z model), we see confusion in prediction between the normal state (class D2) and the low-oil level state (class D1). The structural defects (D3 and D4 classes) are confused by the model Mcwt_x (Table 3), while the misclassified instances are distributed more uniformly among all classes by the model Mcwt_y (Table 4). The F1 scores at the level of classes in all four models, from Tables 3-6, are plotted comparatively using the bar charts in Figures 10 and 11. As expected, Figure 10 shows that the best-performing model for all classes is Mcwt_xyz, with F1 scores between 99.85% and 99.96%. Using the information acquired on a single axis by the triaxial accelerometer, we observe the following:  
* Classes D1 and D2 are best detected by the Mcwt_y model (with F1 scores of 99.70% and 99.48%, respectively), followed by Mcwt_x (99.29% and 99.28%, respectively), and Mcwt_z (F1 score = 98.73% for both classes);
* Classes D3 and D4, corresponding to the structural defects, are best identified by the Mcwt_z model (F1 score is 99.54% and 99.62%, respectively), followed by Mcwt_y (F1 score is 99.47% and 99.51%, respectively), and Mcwt_x (F1 score is 98.80% and 98.84%, respectively). In Figure 11, we notice that the model Mcwt_xyz detects all four defects with almost the same performance (F1 scores between 99.85% and 99.96%). The Mcwt_y model follows the same pattern, achieving very close F1 scores for the classes D2 (99.48%), D3 (99.47%), and D4 (99.51%), but a slightly higher score (99.7%) for D1. The signal measured on the _X_-axis distinguishes better (+0.5% in terms of F1 score) the classes D1 and D2 compared to classes D3 and D4.