## 5 Results and Discussion
### Model Comparison  
Accuracy is a good parameter to consider for judging the performance of a system as a whole, but it fails to capture details, such as false-positive predictions or the inability to identify faults that are present. Details such as these are especially important in the context of frequency and criticality of failures that the reasoner is being used to identify. Here it can be seen that among the univariate models, the reasoner employing the TSF model is the most accurate, with 99.3% accuracy. This is followed by the LSTM model providing 85.3% and, lastly, the k-NN model with 72.3%. Contrary to the univariate models, the k-NN multivariate model is the most accurate of the three models with 36.7% accuracy, followed by the TSF and LSTM with 34.3% and 30.7%, respectively. Accuracy is an effective indicator of performance when the distribution chosen for the dataset for testing is symmetric. For this experiment, the test data are programmed such that it is not always symmetric so as to depict real-life scenarios. Therefore, it will not be appropriate to consider accuracy as a sole indicator of a reasoner performance. Table 13 displays the comparison in model accuracy from the experiment. Another parameter to consider is precision, which in the experiment gives an idea of the ratio of correctly identified OC faults to the total number of OC faults predicted by the model. It can be observed that again, the TSF univariate model provides the highest precision, followed by the LSTM and k-NN models. Among the multivariate models, the LSTM model was unable to identify any faults and the k-NN multivariate was able to achieve a precision of 46.7%. The higher precision of the TSF univariate model is an indicator that it had produced the lowest false positives among the models compared in this experiment. Table 14 show the performance parameters of the OC fault classification. The recall rate for classifying OC informs the observer of the number of faults that the classifier was able to identify among the total number of OC faults introduced to it. The TSF univariate model has the highest recall rate showcasing the ability to identify all the relevant cases it was shown. The next best value for this metric is showcased by a k-NN univariate model with a recall rate of 83.1%, followed by an LSTM single feature \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **Univariate** & & & **Multivariate** & \\ \cline{2-6}  & **LSTM** & **TSF** & **k-NN** & **LSTM** & **TSF** & **k-NN** \\ \hline Accuracy & 85.3\% & 99.3\% & 72.3\% & 30.7\% & 34.3\% & 36.7\% \\ \hline \hline \end{tabular}
\end{table} 
Table 13: ML Model Accuracy Comparison. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Average Accuracy** & & **36.7\%** \\ \hline \multicolumn{3}{c}{**Average Precision**} & **Average Recall** & **Average F1-Score** \\ \hline OC & 46.7\% & 46.7\% & 46.7\% \\ IOC & 20.0\% & 28.6\% & 23.5\% \\ Jamming & 20.0\% & 12.5\% & 15.4\% \\ \hline \hline \end{tabular}
\end{table} 
Table 12: k-NN Multivariate Performance. model with 71.7%, k-NN multivariate with 46.7%, TSF multivariate with 24.7%, and LSTM multi-feature with no recalling ability. It is worth noting that even though the recall rate is good for the k-NN univariate model, the precision rate is around 60%, indicating that it was able to identify a large number of OC faults at the cost of incorrectly classifying some other faults as OC. F1-score is a measure that gives equal importance to both precision and recall. TSF univariate has the highest score with 98.9%, and the LSTM univariate comes in second with 79.4%. The F1-score for the k-NN univariate model can be said to be a decent 70.8%. Similarly, for the classification of IOC, both TSF and k-NN univariate models offer 100% precision implying no false-positive cases were recorded. The next best precision is offered by LSTM univariate model with 92.8% precision, followed by TSF, k-NN and LSTM multivariate models with 27.0%, 20.0%, and 10.7%, respectively.