# Novelty Detection with Autoencoders for System Health Monitoring in Industrial Environments
## 5 Experimental Evaluation
### Evaluation Localization Task  
Description. The experiment evaluates the capability of the AE architecture to reconstruct the multivariate time series, using the reconstruction error for each feature of the time series, i.e., for each sensor of the machinery. This allows us not only to identify the presence of a never-seen-before operating condition, but also to identify the most anomalous signal that distinguishes the status. This evaluation allows us to understand if the experimented AE models can separate the features of the multivariate series and to identify the most anomalous signal with respect to the normality learned during training. The sensor that measures the strongest variation for each status has been identified in the laboratory. Table 15 provides a simple analysis of the signals for each operating condition. In particular, it shows the minimal distance from each signal in each operating condition to the other signals in the other conditions. The higher the distance, the easier the task of recognizing the most meaningful feature. The difference between the maximum and the minimum distance roughly shows how easy the task is to perform. According to the distance, the identification of the most relevant signal is easier for condition C1 (and then C2) than for the other conditions. The experiment consists of the selection of the signal that maximizes the reconstruction error. Table 10 shows the results in terms of precision at the first, third, and fifth levels.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Features** & **C1** & **C2** & **C3** & **C4** \\ \hline Acceleration X1 & 0.009515 & 0.002341 & 0.002009 & 0.002009 \\ Acceleration Y1 & 0.012629 & 0.006512 & 0.003413 & 0.003413 \\ Acceleration Z1 & 0.005493 & 0.009277 & 0.005493 & 0.009277 \\ Acceleration X2 & 0.008993 & 0.002243 & 0.002243 & 0.006444 \\ Acceleration Y2 & 0.000181 & 0.000181 & 0.012083 & 0.005866 \\ Acceleration Z2 & 0.009027 & 0.005133 & 0.004702 & 0.004702 \\ Acceleration X3 & 0.015048 & 0.003005 & 0.003005 & 0.004736 \\ Acceleration Y3 & 0.003957 & 0.003795 & 0.015512 & 0.003795 \\ Acceleration Z3 & 0.024635 & 0.001038 & 0.001339 & 0.001038 \\ \hline Avg & 0.009942 & 0.003725 & 0.005533 & 0.004587 \\ Std & 0.007089 & 0.002850 & 0.004936 & 0.002459 \\ Min & 0.000181 & 0.000181 & 0.001339 & 0.001038 \\ Max & 0.024635 & 0.009277 & 0.015512 & 0.009277 \\ Diff & 0.024455 & 0.009096 & 0.014174 & 0.008238 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Input signals on the operating conditions.  
Discussion. Table 16 shows a large dependence of the results from the DL model used and that only in a few configurations (FC for condition C1, FC and LSTM for C3) is the approach able to identify as the first result the most relevant signal. A different situation occurs if we consider the first three results (p3): for all operating conditions, there is almost an implementation that obtains a high precision level (greater than 0.9). Moreover, as we were expecting, the performance on C4 and on C2 was lower than on the other conditions. In particular, on C4, only the CNN-based AE obtained a high precision score. This is due to the particular implementation of CNNs that base (through the padding and the kernel) their learning approach on all the signals simultaneously.  
Lesson Learned  
The experiments show that AEs allow users to find the signals that have the largest impacts on the predictions, thus providing model explanations and allowing localization of the fault when it happens for the first time (novel condition).