## LSTMCaps Autoencoder Network
### Experimental Results
#### 2.6.3 Drone Dataset Outlier Resilient Anomaly Detection  
This experiment aims to explore the resilience of each model variant in Figure 5 with outliers in the training data. To prepare the training data, a total of 200 anomalous points were randomly inserted into the training set and the same experiment, as previously outlined in Section 2.6.2, was conducted. Figure 8 illustrates a visualisation of the training set with anomalies, Table 7 shows the best and average training results for each model variant, and Table 8 shows the anomaly detection scores for each model variant. The ANOVA source table for the \(F_{1}\) scores is depicted in Table 9.  
The results in Table 8 show that all the tested models have resilience to \"non-ideal\" training data that contains anomalies. In fact, results show a marginal improvement in performance on average for every model tested in comparison to the results in Table 5. As found in the previous experiment, the proposed model, Design D, still outperforms all models tested. The \(p\)-value of 0.0085 calculated from the ANOVA source table in Table 9 suggests that the means of the experiment are significantly statistically different, and unlikely to be a result of randomness. There are various factors that contribute to this resilience. One aspect to consider is the anomalies that occur as a result of faults will most likely be different to the anomalies that appear during \"healthy\" condition, and thus such anomalies from the latter category can be learned during the training of the model without significant impact on the anomaly detection ability as they could be considered as part of the \"healthy\" condition. Another aspect considered during model training was the use of the Huber loss function, which uses a combination of Mean Absolute Error (MAE) and Mean Squared Error (MSE) depending on the magnitude of the loss value. This results in a model that is more robust to outliers \begin{table}
\begin{tabular}{l l l l l l l l} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Score**} & \multicolumn{1}{c}{**Final**} & \multicolumn{1}{c}{**Final**} & \multirow{2}{*}{**Training Time (s)**} & \multirow{2}{*}{**MSE**} & \multirow{2}{*}{**\% Overfitting**} & \multicolumn{1}{c}{**\% Val Loss Improvement**} \\  & & & **Training Loss** & & & & & **from Non-Caps** \\ \hline \multirow{3}{*}{Design A} & Average over 5 runs & 0.00626 & 0.00667 & 35.89 & 10.61 & 6.12 & \\  & Best over 5 runs & 0.00614 & 0.00653 & 36.53 & 10.19 & 5.98 & N/A–Non-Caps Version \\  & Standard Deviation & 0.00024 & 0.00035 & 0.41 & 0.71 & 0.00035 & \\ \hline \multirow{3}{*}{**Design B**} & \multicolumn{1}{c}{**Average over 5 runs**} & \multicolumn{1}{c}{**0.00188**} & \multicolumn{1}{c}{**0.00155**} & \multicolumn{1}{c}{**49.77**} & \multicolumn{1}{c}{**2.08**} & \multicolumn{1}{c}{\(-\)**21.24**} & \multicolumn{1}{c}{**3.31**} \\  & \multicolumn{1}{c}{**Best over 5 runs**} & \multicolumn{1}{c}{**0.00194**} & \multicolumn{1}{c}{**0.00158**} & \multicolumn{1}{c}{**49.62**} & \multicolumn{1}{c}{**2.09**} & \multicolumn{1}{c}{\(-\)**22.59**} & \multicolumn{1}{c}{**3.14**} \\  & \multicolumn{1}{c}{**Standard Deviation**} & \multicolumn{1}{c}{**0.00004**} & \multicolumn{1}{c}{**0.00003**} & \multicolumn{1}{c}{**0.23**} & \multicolumn{1}{c}{**0.04**} & \multicolumn{1}{c}{**0.00003**} & \\ \hline \multirow{3}{*}{Design C} & Average over 5 runs & 0.00575 & 0.00620 & 56.52 & 10.08 & 7.28 & \\  & Best over 5 runs & 0.00610 & 0.00709 & 56.09 & 12.41 & 14.01 & N/A–Non-Caps Version \\ \cline{1-1}  & Standard Deviation & 0.00052 & 0.00071 & 0.83 & 1.13 & 0.00071 & \\ \hline \multirow{3}{*}{Design D (proposed)} & Average over 5 runs & 0.00162 & 0.00161 & 143.34 & 2.14 & \(-\)0.69 & 2.84 \\  & Best over 5 runs & 0.00168 & 0.00172 & 142.59 & 2.32 & 2.52 & 3.12 \\ \cline{1-1}  & Standard Deviation & 0.00007 & 0.00007 & 0.76 & 0.084 & 0.00007 & \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Results for training for each NN model in Figure 5 using hyperparameters from Table 3.