## LSTMCaps Autoencoder Network
### Experimental Results
#### 2.6.4 SKAB Anomaly Benchmark  
The SKAB anomaly detection benchmark [41] is a public benchmark available online used for offline outlier detection and changepoint detection testing. The benchmark consists of 35 subsets of data from a water circulation system, which contain 8 features, each from different sensors in the system. The test is conducted by looping through each subset, training the neural network on a slice of clean data from the subset, and then testing it on labelled anomalies that were simulated with the test rig. The metric used to gauge the performance of the outlier detection is the \(F_{1}\) score (Equation (12)), and the NAB Changepoint Metric [42] was used to measure the changepoint detection ability of each model. Figure 8(a) illustrates a subset of data from the benchmark, and Figure 8(b) shows the plot for the anomalies in the data. This experiment aims to compare the anomaly detection and changepoint detection performance of popular and state-of-the-art unsupervised anomaly detection methods with the proposed NN model. A selection of NNs and ML-based fault detection methods were chosen to compare on the benchmark with minimal hyperparameter optimisation applied. During model testing it was found that there were different hyperparameter values that could be used to optimise the proposed model for the outlier detection and changepoint  
Figure 9: Visualisation of the SKAB Dataset. detection tasks, respectively. This meant that hyperparameters optimal for a good \(F_{1}\) score would not necessarily perform as well on the NAB score. To demonstrate this, the hyperparameters of the LSTMCaps NN were optimised for each score separately in order to achieve the maximum score for each metric. The different hyperparameter settings used are shown in Table 10. Furthermore, Z-score normalisation (Equation (16)) was used in this case as it was found to improve the performance for the proposed method. The same testing procedure utilised in the SKAB benchmark's GitHub repo [41] for each model already tested on the benchmark was used to test the proposed model architecture. The model was trained with 100 epochs on a subset from each dataset with early stopping set at a patience of 20, and then tested on the remainder of the dataset. The \(F_{1}\) scores and NAB scores achieved for each dataset are averaged, which gives the final score of the benchmark. Each model compared was also tested on the same hardware for a fair comparison. To gain a better understanding of the effectiveness of each method tested as a hybrid solution for both outlier and changepoint detection, a scaled average of both the \(F_{1}\) and NAB metrics was calculated to represent the overall accuracy of the model over both outlier detection and changepoint detection tasks. This was done by scaling the NAB score between 0 and 1, then averaging it with the \(F_{1}\) score. Equation (17) was used to calculate this score. The results in Table 11 depict the average outlier detection score, the changepoint detection score, and the scaled average score over five test iterations, respectively, and the results in Table 12 detail the best score achieved in a single test iteration over the outlier, changepoint, and scaled average scores, respectively. A scatter plot of the average \(F_{1}\) scores against NAB scores for each model tested is illustrated in Figure 10. The ANOVA of the overall accuracies of each model is calculated in Table 13.  
\[Overall\ Accuracy=\frac{F_{1}+\frac{NAB}{100}}{2} \tag{17}\]