# Intelligent Multi-Fault Diagnosis for a Simplified Aircraft Fuel System
## 3 Methodology
### Evaluating the Interpretability of the Selected ML Algorithms
The performance of the selected ML algorithms was evaluated using the following metrics: multiclass confusion matrix, accuracy, Matthews correlation coefficient (MCC), kappa coefficient, and F-1 score. Before finalising each algorithm's configuration, including model structure and hyperparameters, preliminary evaluations were performed using an independent test set comprising 10% of the total dataset, instead of cross-validation. This approach was adopted due to the ample data generated by the simulation model for ML. The entire dataset was randomly divided into three subsets--training, testing, and validation--ensuring that each subset had a similar distribution and included all 32 classes. Figure 9 highlights the test set using the yellow color and illustrates the quantity of data for each class within it. A straightforward approach was then employed in the MATLAB R2024a environment to tune only one to two hyperparameters that significantly impacted algorithm performance based on accuracy with the test data. All other parameters were kept at their default values. For the logistic regression, no hyperparameters were identified as substantially affecting performance. In the case of the SVM, the choice of kernel functions and solvers significantly influenced performance, with MATLAB offering four kernel options and three solver options, as summarised in Figure 10a. The key hyperparameters for decision trees (DTs) and artificial neural networks (ANNs) were the number of decision nodes and neurons in the hidden layers, respectively. Figure 9: Distribution of training, testing, and validation datasets. Finally, based on the classification errors under the different hyperparameter settings shown in Figure 10, a one-vs-one strategy was implemented for the SVM. This configuration used a 4 th-order polynomial kernel, 496 sub-learners, and an ISDA solver, which yielded the lowest classification error. Additionally, when achieving the highest test accuracy, the ANN featured a three-layer network with 11 input nodes, a hidden layer of 16 nodes, and 32 output nodes. The decision tree (DT) utilised a CART structure with 1100 splits (decision nodes). Subsequently, the optimised models were evaluated on the remaining validation dataset, representing 20% of the total data. Their performance was assessed using the four evaluation metrics previously mentioned. The results of this validation process are summarised in Table 5.  
All four evaluation metrics indicate that the ANN outperformed the other models, correctly classifying 90% of the samples, while the logistic regression (LR) showed the weakest performance. Figure 11 provides a detailed view of the F-1 score distribution across classes for the four selected ML methods, with the F-1 score being the only metric capable of offering class-level detail. The average F-1 scores in Table 5 correspond to the values in Figure 11. The ANN achieved the highest F-1 scores in nearly all classes except  
Figure 10: ML results on the test data: (**a**) SVM testing results; (**b**) DT testing results; (**c**) ANN testing results. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Accuracy** & **MCC** & **Kappa** & **F-1** \\ \hline LR & 0.7926 & 0.7826 & 0.781 & 0.7535 \\ DT & 0.8382 & 0.8303 & 0.83 & 0.7874 \\ SVM & 0.8395 & 0.8319 & 0.8308 & 0.8195 \\ ANN & 0.9008 & 0.8959 & 0.8958 & 0.879 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Performance on the validation dataset. for Class 12 and 17. Conversely, the LR exhibited the lowest F-1 scores in most classes, with only seven exceptions. The F-1 scores of the SVM and DT generally fell between those of the ANN and LR, although the DT's average score was notably lower than that of the SVM due to higher variability across classes. Specifically, the DT's performance was particularly poor in certain classes, including Classes 11, 15, 16, 30, and 31. Figure 12 presents the confusion matrix for SVM results on the validation dataset. In this matrix, rows correspond to the actual classes, while columns represent the SVM's predicted classes.