# Improving Deep Learning Anomaly Diagnostics with a Physics-Based Simulation Model
## 4 Validation with the Long Short-Term Memory Autoencoder
### Comparative-Validation Results
The negative-precision (neg) and recall (neg) scores in the Diagnostics validation are low; however, the relatively high overall balance presented by the F1-score suggests that the diagnostic process can successfully classify the machine states. The performance results obtained by the Simulation validation are the highest overall, which is as expected due to the lack of randomness or noise in the signal. The division between normal and anomalistic behavior was separated with the +25% dynamic threshold limit in correlation with the simulated torque x6, which works in favor of the absence of noise characteristics typical for real-life signals. In particular, the Recall (neg) score's increase between Class. 1 and Class. 2 is conspicuous and shows a significant positive difference in the model's failure-class identification capability. The classification process for the Comparative validation was iterated multiple times due to the nature of the algorithm's function to randomly select the training and test sets. The results present an average score received from the iterations. The model's classification capability improvement is evident from the quantified measures presented in Table 11. In addition to the aforementioned scoring metrics, the AUC has become increasingly popular for measuring the performance of binary classifiers [37]. The graphical results of the LSTM classification shown in Figure 10 also include the numeric measure of the AUC. Notably, the quantified confusion matrix results are not directly comparable between the data sets Figure 10a,c due to the differences in data set sizes, yet a significant relative reduction in the false-positive class and the distinct increase in the true-negative class is observable. The receiver operating characteristics (ROC) curve visualizes the function of the true-positive rate and false-positive rate against the discrimination threshold between zero and one [38]. This discrimination threshold line is presented in 45-degree red diagonal line in Figure 10b,d. The AUC values closer to 1 emphasize better classification capability, whereas results at 0.5 indicate the model's poor capability to classify the data points [39]. By observing the AUC score, the improvement from the Diagnostics validation (AUC = 0.774) (Figure 10b) to the Comparative validation (AUC = 0.928) (Figure 10d) is generally comparable due to inclusion of DS2 data; thus, showing the model's above average increase in classification capability with the help of simulated (SN and SA) x6 torque data. To summarize, the quantified and visualized results demonstrate improvement in the model's classification capability following the inclusion of the simulation data. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Validation** & **Accuracy** & **Precision (pos)** & **Precision (neg)** & **Recall (pos)** & **Recall (neg)** & **F1-Score** \\ \hline Diagnostics & & & & & & \\ validation (Class. 1) & 0.799 & 0.955 & 0.090 & 0.827 & 0.308 & 0.886 \\ Simulation validation (Class. 2) & & & & & & \\ Comparative validation (Class. 3) & & & & & \\ \(\Sigma\) & 0.845 & 0.990 & 0.170 & 0.847 & 0.789 & 0.913 \\ \(\Sigma\) & \(\frac{TP+TN}{TP+FP+TN+FN}\) & \(\frac{TP}{TP+FP}\) & \(\frac{TN}{TP+FN}\) & \(\frac{TN}{TN+FP}\) & \(\frac{2\times Precision\times Recall}{Precision+Recall}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Validation-performance results using the Class. 1, 2, and 3 classifications containing the summation of the results.