# Novelty Detection with Autoencoders for System Health Monitoring in Industrial Environments
## 5 Experimental Evaluation
### Effectiveness and Robustness of the AEs in Novelty Detection
#### 5.1.4 Lesson Learned  
The experiments demonstrate that AEs can effectively address novelty detection tasks, as they achieve the highest F1 scores in all scenarios outperforming the baselines. The experiments also demonstrate that AE is a robust technology: the results are not largely affected by the task addressed (know status or novel status), by the implementation \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline
**Scenario** & **FC** & **CNN** & **LSTM** & **BiLSTM** & **PCA** & **SVM** & **Cluster** \\ \hline S123 & 64.67 & 100 & 60.26 & 98.15 & 56.70 & 20.59 & 3.19 \\ S124 & 100 & 99.88 & 99.93 & 100 & 48.39 & 7.47 & 80.82 \\ S134 & 99.10 & 94.53 & 99.38 & 99.66 & 38.42 & 39.49 & 79.48 \\ S234 & 100 & 100 & 100 & 100 & 44.22 & 87.95 & 94.99 \\ S12 & 95.07 & 99.29 & 98.38 & 98.69 & 63.93 & 1.22 & 29.33 \\ S13 & 100 & 100 & 100 & 100 & 100 & 45.44 & 37.32 \\ S14 & 94.51 & 91.99 & 97.42 & 98.11 & 68.15 & 30.21 & 99.43 \\ S23 & 45.29 & 99.52 & 97.93 & 97.71 & 76.25 & 94.84 & 19.66 \\ S24 & 99.96 & 98.31 & 99.61 & 99.77 & 89.56 & 35.83 & 100 \\ S34 & 92.73 & 91.12 & 91.91 & 90.65 & 65.30 & 60.20 & 97.80 \\ S1 & 100 & 100 & 100 & 100 & 87.99 & 29.17 & 51.67 \\ S2 & 99.11 & 85.46 & 97.29 & 97.43 & 92.30 & 28.83 & 30.38 \\ S3 & 100 & 100 & 100 & 99.63 & 43.75 & 57.39 & 43.88 \\ S4 & 69.73 & 86.53 & 88.99 & 75.51 & 61.84 & 40.50 & 53.22 \\ \hline Average & 90.01 & 96.19 & 95.08 & 96.81 & 66.91 & 41.37 & 58.66 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Evaluation of the effectiveness in operating conditions with anomalies: average F1 score. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Window**} & \multicolumn{2}{c}{**FC**} & \multicolumn{2}{c}{**CNN**} & \multicolumn{2}{c}{**LSTM**} & \multicolumn{2}{c}{**BiLSTM**} & \multicolumn{2}{c}{**PCA**} & \multicolumn{2}{c}{**SVM**} & \multicolumn{2}{c}{**Cluster**} \\ \cline{2-13}  & **Avg** & **Std** & **Avg** & **Std** & **Avg** & **Std** & **Avg** & **Std** & **Avg** & **Std** & **Avg** & **Std** \\ \hline
40 & 99.48 & 0.62 & 99.37 & 0.65 & 98.88 & 1.19 & 94.89 & 9.19 & 90.12 & 6.56 & 40.83 & 25.14 & 97.31 & 2.54 \\
80 & 99.71 & 0.49 & 99.63 & 0.45 & 99.56 & 0.32 & 99.16 & 0.75 & 76.67 & 8.72 & 39.82 & 25.36 & 96.15 & 4.40 \\
120 & 99.99 & 0.02 & 99.99 & 0.02 & 99.86 & 0.27 & 99.29 & 0.95 & 67.24 & 12.07 & 40.25 & 25.78 & 94.95 & 5.76 \\
200 & 99.92 & 0.18 & 99.75 & 0.41 & 99.93 & 0.15 & 99.88 & 0.14 & 57.58 & 17.26 & 40.80 & 25.58 & 92.24 & 9.13 \\
240 & 100 & 0 & 99.86 & 0.3 & 100 & 0 & 99.61 & 0.59 & 55.39 & 16.71 & 41.36 & 25.38 & 90.76 & 11 \\
360 & 100 & 0 & 100 & 0 & 100 & 0 & 99.94 & 0.13 & 52.96 & 14.58 & 45.10 & 26.62 & 87.54 & 14.8 \\ \hline Average & 99.85 & 0.22 & 99.77 & 0.31 & 99.71 & 0.32 & 98.80 & 1.96 & 66.66 & 12.65 & 41.36 & 25.64 & 93.16 & 7.94 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Average F1 score and standard deviation for each window size in normal operating conditions. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c} \hline \hline
**Window** & \multicolumn{2}{c}{**FC**} & \multicolumn{2}{c}{**CNN**} & \multicolumn{2}{c}{**LSTM**} & \multicolumn{2}{c}{**BiLSTM**} & \multicolumn{2}{c}{**PCA**} & \multicolumn{2}{c}{**SVM**} & \multicolumn{2}{c}{**Cluster**} \\ \cline{2-13}  & **Avg** & **Std** & **Avg** & **Std** & **Avg** & **Std** & **Avg** & **Std** & **Avg** & **Std** & **Avg** & **Std** \\ \hline
40 & 78.86 & 25.41 & 82.15 & 21.20 & 69.89 & 29.53 & 67.33 & 29.66 & 59.31 & 30.62 & 41.75 & 25.57 & 60.57 & 33.50 \\
80 & 85.66 & 19.10 & 88.51 & 13.05 & 82.23 & 23.60 & 80.14 & 27.10 & 62.67 & 23.91 & 41.33 & 25.63 & 60.89 & 32.77 \\
12  
(four deep learning techniques are evaluated), the quality of the dataset (scenarios with and without anomalies have been tested), and the dimension of the signals (we evaluated the approach varying the window size).