# Improving Deep Learning Anomaly Diagnostics with a Physics-Based Simulation Model
## 4 Validation with the Long Short-Term Memory Autoencoder
### Comparative-Validation Results  
The Comparative validation of the data sets is performed to evaluate how the simulated data affects the model's classification capability. To assess the change, the validation is performed with the same LSTM architecture used in the diagnostics validation and Simulation validation sections, where the data sets 2 and 3 were individually tested. The term comparative validation is used to describe a process of verifying the accuracy of two or more models, processes, or systems against each other by validating the effects of the compared methods [33]. In this study, the Comparative validation stage concatenates the previously validated DS2 output and DS3 output data sets into a DS4 data set to evaluate the simulated data influence on the classification score. To determine the status of origin for such classification testing, the LSTM algorithm's hidden states are reset to exclude learning from the previous validation rounds, as the dimension of the hidden states is modeled to retain information from past events [34]. Table 10 summarizes the data points and their states in the DS4 data set after the data concatenation process. In data set DS4 Equation (7) the variables \(y\) and \(x1\ldots x4\) are positioned in congruent column positions as per the previous validation rounds; however, the collection of torque excitation datapoints is conjugated to \(\{x5_{n,2},x6_{n,3}\}\). This is done to establish a single column library for the final classification task, including excitation values from both the real and simulated signals, containing all designated states of RN, RA, SN, and SA. All the row-specific data in the DS4 data set remain synchronized, yet the order of the individual data rows is shuffled to randomize the data points used for the LSTM training and testing. The Comparative validation data set DS4 is formed as presented in Equation (7). \[\text{DS4}= \begin{bmatrix}y_{1,1.2}&x1_{1,2.3}&x2_{1,2.3}&x3_{1,2.3}&x4_{1,2.3}& \{x5_{1,2},x6_{1,3}\}\\ \ldots&\ldots&\ldots&\ldots&\ldots\\ y_{22825,1.2}&x1_{22825,2.3}&x2_{22825,2.3}&x3_{22825,2.3}&x4_{22825,2.3}&\{x5_{ 1335,2},x6_{11470,3}\}\end{bmatrix}  \tag{7}\]  
The Comparative validation results together with more comprehensive results from the Diagnostics validation and Simulation validation are presented in Table 11. Metrics to measure the binary classification method success are generally recognized to include accuracy, precision, recall, F1-score, and area under the receiver operating characteristics curve (AUC) [13, 32, 35, 36]. Accuracy is the number of correctly classified samples from all samples present in the test set. Precision measures the proportion of correctly identified events relative to all events identified, while recall measures the proportion of correctly identified events relative to all events that should have been identified [14]. Similar to the \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Total Data Points**} & \multirow{2}{*}{**(\%)**} & \multicolumn{2}{c}{**Normal State**} & \multirow{2}{*}{**(\%)**} & \multicolumn{2}{c}{**Anomaly State**} & \multirow{2}{*}{**(\%)**} \\  & & **(RN + SN)** & & & **(RA + SA)** & \\ \hline
22,825 & 100.0 & 18,766 & 82.2 & 4059 & 17.8 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: The number of normal and anomaly states in the DS4 data set. \begin{table}
\begin{tabular}{c c c c} \hline \hline TP & 1895 & 15 & FP \\ \hline FN & 302 & 81 & TN \\ \hline \hline \end{tabular}
\end{table} 
Table 9: The simulation validation (Class. 2) confusion matrix. other scoring metrics, the F1-score (also known as the F-measure) ranges from zero to one, but it is considered a more comprehensive measure to evaluate the model's performance due to its harmonic mean of recall and precision metrics. The precision and recall values are given both in positive and negative correlations due to the interest in observing both the normal operation as well as the anomaly classification capability of the algorithm.