# Novelty Detection with Autoencoders for System Health Monitoring in Industrial Environments
## 5 Experimental Evaluation
### Effectiveness and Robustness of the AEs in Novelty Detection
#### 5.1.3 Window Size Invariance  
Description. In this experiment, we evaluate the capability of the AE approaches of being invariant to the window size. Six window sizes are selected (ranging from 40 records per window-20 s to 360 records-180 s) and the same experiment for detecting the known and novel sets as in Sections 5.1.1 and 5.1.2 was performed.  
Discussion. The results of the experiments show that AE-based approaches outperform the competing approaches independent of the window size. Nevertheless, they suffer from the noise that slightly decreases the average F1 score and increases the standard deviation. In detail, if we consider AEs and normal operating conditions (Table 8), the larger the window sizes, the better the results obtained. The same does not happen for the baseline, where the performance of SVM is window size invariant, and the other approaches decrease with the increase in the size. Moreover, the AE models offer low values for the standard deviation, thus showing that they do not depend on the data analyzed. The same does not happen for the baseline where the standard deviation assumes values greater than 25%.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Scenario**} & \multicolumn{2}{c}{**FC**} & \multicolumn{2}{c}{**CNN**} & \multicolumn{2}{c}{**LSTM**} & \multicolumn{2}{c}{**BiLSTM**} & \multicolumn{2}{c}{**PCA**} & \multicolumn{2}{c}{**SVM**} & \multicolumn{2}{c}{**Cluster**} \\ \cline{2-19}  & **Known** & **Novel** & **Known** & **Novel** & **Known** & **Novel** & **Known** & **Novel** & **Known** & **Novel** & **Known** & **Novel** \\ \hline S123 & 35.19 & 94.14 & 100 & 100 & 28.62 & 91.90 & 96.41 & 99.89 & 23.94 & 89.45 & 0 & 41.19 & 5.68 & 0.70 \\ S124 & 100 & 100 & 99.79 & 99.96 & 99.88 & 99.97 & 100 & 100 & 5.90 & 90.88 & 14.94 & 0 & 66.57 & 95.07 \\ S134 & 99.09 & 99.10 & 94.72 & 94.34 & 99.38 & 99.93 & 99.66 & 99.67 & 8.36 & 68.48 & 32.55 & 46.44 & 75.07 & 83.89 \\ S234 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 0 & 88.44 & 80.07 & 95.83 & 91.90 & 98.08 \\ S12 & 92.08 & 98.06 & 98.88 & 99.70 & 97.44 & 99.32 & 97.92 & 99.46 & 52.50 & 75.36 & 1.44 & 0.99 & 37.28 & 21.38 \\ S13 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 47.33 & 43.56 & 70.42 & 4.21 \\ S14 & 96.78 & 92.24 & 95.46 & 88.52 & 98.43 & 96.42 & 98.83 & 97.38 & 69.77 & 66.53 & 48.03 & 12.40 & 99.64 & 99.22 \\ S23 & 47.25 & 43.33 & 99.27 & 99.76 & 96.90 & 98.95 & 96.57 & 98.84 & 69.28 & 83.23 & 92.02 & 97.66 & 39.32 & 0 \\ S24 & 99.95 & 99.97 & 97.97 & 98.65 & 99.53 & 99.69 & 99.73 & 99.82 & 86.31 & 92.82 & 39.49 & 32.16 & 100 & 100 \\ S34 & 96.38 & 89.07 & 95.68 & 86.56 & 96.02 & 87.8 & 95.36 & 85.94 & 68.95 & 61.65 & 62.24 & 58.15 & 98.82 & 96.77 \\ S1 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 100 & 92.53 & 83.46 & 58.34 & 0 & 86.28 & 17.07 \\ S2 & 99.00 & 99.21 & 85.72 & 85.20 & 97.01 & 97.56 & 97.11 & 97.76 & 91.88 & 92.72 & 38.81 & 18.85 & 60.75 & 0 \\ S3 & 100 & 99.99 & 100 & 100 & 100 & 100 & 99.84 & 99.42 & 87.49 & 0 & 63.64 & 51.13 & 87.76 & 0 \\ S4 & 95.51 & 43.96 & 97.84 & 75.23 & 98.14 & 79.83 & 94.19 & 56.83 & 89.88 & 33.79 & 60.69 & 20.31 & 96.68 & 9.75 \\ \hline Average & 90.09 & 89.93 & 97.52 & 94.85 & 93.67 & 96.49 & 98.26 & 95.36 & 60.49 & 73.34 & 45.69 & 37.05 & 72.58 & 44.72 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Evaluation of the effectiveness in normal conditions with anomalies: F1 Score on the Known and Novel sets.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Scenario** & **Known Set** & **Novel Set** \\ \hline S123 & C4 & C1, C2, C3 \\ S124 & C3 & C1, C2, C4 \\ S134 & C2 & C1, C3, C4 \\ S234 & C1 & C2, C3, C4 \\ S12 & C3, C4 \\ S13 & C2, C4 \\ S14 & C2, C3 \\ S23 & C1, C4 \\ S24 & C1, C3 \\ S34 & C1, C2 \\ S1 & C2, C3, C4 \\ S2 & C1, C3, C4 \\ S3 & C1, C2, C4 \\ S4 & C1, C2, C3 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Scenarios used for the evaluation of the novelty detection task with anomalies.