# Acoustic-Based Engine Fault Diagnosis Using WPT, PCA and Bayesian Optimization
## 4 Results
### Discussion  
The objective of this research was to determine the faulty engine signal in real-time with acceptable classification accuracy for different fault conditions. Table 8 shows the model evaluation metrics for the three models using the equations in Table 7.  
Figure 14: Model 3—confusion matrix for AFR variation data.  
Figure 12: Model 3—confusion matrix for engine misfire data.  
Figure 13: Model 3—confusion matrix for ignition timing variation data.  
The bar chart in Figure 15 shows the improvement in accuracy and F1 score when using Bayesian optimization over the standard model for all three fault conditions. In engine misfire condition, all three models show the same performance, as classification is apparent in a simple fault scenario, but for the other two more complex faults there is a 5% increase in accuracy by implementation of Bayesian optimized model, and a 3-4% increase in accuracy by implementation of PCA with Bayesian optimized model, compared to the standard model. On the other hand, there is a less significant change in the accuracy and F1 score when comparing Model 2 and Model 3, even though the feature dimensionality has been considerably reduced by using PCA in Model 3. This effect can be demonstrated by comparing their evaluation time.  
The comparison the total training and testing time between Model 2 and Model 3 in Figure 16 shows that the PCA has helped significantly lower the evaluation time by \(\sim\)20% compared to the Model 2 due to its reduced feature dimensionality. Standard model is not considered in this comparison since it requires manual effort and time to implement the best fit model, whereas the other two models are automated processes which require minimal human intervention.  
\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Model 1**} & \multicolumn{3}{c}{**Model 2**} & \multicolumn{3}{c}{**Model 3**} \\ \cline{2-10}  & **EM 1** & **IGV 2** & **AFRV 3** & **EM** & **IGV** & **AFRV** & **EM** & **IGV** & **AFRV** \\ \hline Accuracy & 100 & 80 & 68 & 100 & 85 & 73 & 100 & 83 & 72 \\ Precision & 100 & 83 & 96 & 100 & 86 & 100 & 100 & 88 & 100 \\ Sensitivity & 100 & 88 & 55 & 100 & 93 & 60 & 100 & 88 & 58 \\ Specificity & 100 & 65 & 95 & 100 & 70 & 100 & 100 & 75 & 100 \\ F1 score & 100 & 85 & 70 & 100 & 89 & 75 & 100 & 88 & 73 \\ \hline \hline \end{tabular}  
* Engine misfire; 2 ignition timing variation; 3 AFR variation.  
\end{table}
Table 8: Model evaluation metrics (%).  
Figure 15: Accuracy and F1 score of the three models with three fault conditions. * Model 1, standard classification; * Model 2, Bayesian optimization; * Model 3, PCA with Bayesian optimization.  
Figure 17 shows that using PCA reduces the testing time for all fault cases by 8-19%, when compared to Model 1 and Model 2, as well due to the fact that, PCA has reduced the dimensionality of the feature matrix.  
In summary, PCA with Bayesian optimized model is a promising and highly effective methodology with better evaluation time and classification accuracy compared to the standard model for different fault settings.  
Figure 16: Evaluation time comparison for the models with three fault conditions. * Model 2, Bayesian optimization; * Model 3, PCA with Bayesian optimization. ** Computations were performed on a 2.7 GHz Intel dual Core i5 MacBook pro and programming was done in MATLAB R20.  
Figure 17: Testing time comparison for the models with three fault conditions. * Model 1, standard classification; * Model 2, Bayesian optimization; * Model 3, PCA with Bayesian optimization. ** Computations were performed on a 2.7 GHz Intel dual Core i5 MacBook pro and programming was done in MATLAB R20.