# Novelty Detection with Autoencoders for System Health Monitoring in Industrial Environments
## 5 Experimental Evaluation
### Effectiveness and Robustness of the AEs in Novelty Detection
#### 5.1.1 Evaluating the Novelty Detection Task in Normal Operating Conditions  
Description. In this section, we evaluate only the first three operating conditions of the dataset, which refer to \"normal\" statuses of the machinery, where no anomaly is generated. We experimented with six scenarios (see Table 2), generated through all possible permutations of the conditions. In each scenario, the operating conditions described in the dataset are alternatively evaluated as part of the known set (i.e., the past operating condition) or the novel set. Scenario S12, for example, considers C3 as a known status, and C1 and C2 as novel conditions to detect. Two experiments were performed. In the first, the ability of the AE models is evaluated by detecting the novel state. We trained the AE models with the time series representing the known sets and we evaluated them with the ones in the novel sets. The training set constitutes 10 min of the known sets specified in the scenarios randomly selected. The test set is composed of the remaining data (from the known and novel sets). Column Novel in Table 3 shows the results of this experiment in terms of F1 score by considering a window size equal to 200 (100 s). In the second experiment, the ability of AEs to detect already existing operating conditions is evaluated. The input conditions belonging to the known sets are in this case also used to evaluate the approach. The training and test sets are built as for the previous experiment. Column Known in Table 3 shows the results of these experiments in terms of F1 score. Table 4 shows the F1 score obtained by averaging the results on the known and novel sets, thus providing an overall evaluation of the behavior of the approach. \begin{table}
\begin{tabular}{c c c} \hline
**Scenario** & **Known Set** & **Novel Set** \\ \hline S12 & C3 & C1, C2 \\ S13 & C2 & C1, C3 \\ S23 & C1 & C2, C3 \\ S1 & C2, C3 & C1 \\ S2 & C1, C3 & C2 \\ S3 & C1, C2 & C3 \\ \hline \end{tabular}
\end{table} 
Table 2: Scenarios used for the evaluation of the novelty detection task with no anomaly. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Scenario**} & \multicolumn{2}{c}{**FC**} & \multicolumn{2}{c}{**CNN**} & \multicolumn{2}{c}{**LSTM**} & \multicolumn{2}{c}{**BiLSTM**} & \multicolumn{2}{c}{**PCA**} & \multicolumn{2}{c}{**SVM**} & \multicolumn{2}{c}{**Cluster**} \\ \cline{2-13}  & **Known** & **Novel** & **Known** & **Novel** & **Known** & **Novel** & **Known** & **Novel** & **Known** & **Novel** & **Known** & **Novel** \\ \hline S12 & 99.22 & 99.83 & 99.29 & 99.84 & 99.34 & 99.85 & 99.60 & 99.91 & 5.90 & 90.21 & 15.86 & 0 & 66.57 & 94.69 \\ S13 & 100 & 100 & 100 & 100 & 100 & 100 & 99.98 & 99.98 & 8.36 & 65.56 & 32.82 & 40.80 & 75.07 & 82.02 \\ S23 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 0 & 87.57 & 80.07 & 95.49 & 91.90 & 97.92 \\ S1 & 100 & 100 & 100 & 100 & 100 & 100 & 99.81 & 99.47 & 69.77 & 61.21 & 48.76 & 0 & 99.64 & 99.02 \\ S2 & 100 & 100 & 98.74 & 99.08 & 100 & 100 & 99.89 & 99.92 & 86.31 & 92.03 & 39.68 & 22.47 & 100 & 100 \\ S3 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 68.95 & 55.09 & 62.24 & 51.47 & 100 & 100 \\ \hline Average & 99.87 & 99.97 & 99.67 & 99.82 & 99.89 & 99.98 & 99.88 & 99.88 & 39.88 & 75.28 & 46.57 & 35.04 & 88.86 & 95.61 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Evaluation of the effectiveness in normal operating conditions: F1 Score on the Known and Novel sets. The AE models are compared with three state-of-the-art approaches. Firstly, the Principal Component Analysis (PCA) is frequently used in exploratory data analysis because it reveals the inner structure of the data and explains the variance in the data. PCA looks for correlations among the variables and determines the combination of values that best captures differences in outcomes. For anomaly detection, each new input is analyzed, and the anomaly detection algorithm computes its projection on the eigenvectors, together with a normalized reconstruction error.