## LSTMCaps Autoencoder Network
### Experimental Results
#### 2.6.2 Drone Dataset Anomaly Detection
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Score**} & **Final** & **Final** & \multirow{2}{*}{**Training Time (s)**} & **MSE** & \multirow{2}{*}{**\% Overfitting**} & \multicolumn{2}{c}{**\% Val Loss Improvement**} \\  & & **Training Loss** & **Validation Loss** & & & & **from Non-Caps** \\ \hline \multirow{3}{*}{Design A} & Average over 5 runs & 0.00591 & 0.00639 & 37.75 & 10.37 & 7.45 & \multirow{3}{*}{N/A - Non-Caps} \\  & Best over 5 runs & 0.00589 & 0.00614 & 43.42 & 10.10 & 4.09 & \\  & Standard Deviation & 0.00034 & 0.00046 & 3.22 & 0.85 & 0.04953 & \\ \hline \multirow{3}{*}{**Design B**} & **Average over 5 runs** & **0.00158** & **0.00157** & **50.44** & **2.12** & \(-\)**0.27** & **3.07** \\  & **Best over 5 runs** & **0.00164** & **0.00159** & **50.71** & **2.15** & \(-\)**3.15** & **2.87** \\  & **Standard Deviation** & **0.00006** & **0.00008** & **0.58** & **0.10** & **0.00005** & \\ \hline \multirow{3}{*}{Design C} & Average over 5 runs & 0.00575 & 0.00620 & 56.52 & 10.08 & 7.28 & \multirow{3}{*}{N/A - Non-Caps} \\  & Best over 5 runs & 0.00610 & 0.00709 & 56.09 & 12.41 & 14.01 & \\ \cline{1-1}  & Standard Deviation & 0.00034 & 0.00074 & 0.83 & 1.89 & 0.00034 & \\ \hline \multirow{3}{*}{Design D} & Average over 5 runs & 0.00162 & 0.00161 & 143.34 & 2.14 & \(-\)0.69 & 2.84 \\ (proposed) & Best over 5 runs & 0.00168 & 0.00172 & 142.59 & 2.32 & 2.52 & 3.12 \\ \cline{1-1}  & Standard Deviation & 0.00005 & 0.00006 & 1.63 & 0.11 & 0.00005 & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results for training for each NN model in Figure 5 using the hyperparameters from Table 3.  
Figure 6: Training plot comparison for best models in Table 4. (**a**) Design A; (**b**) Design B; (**c**) Design C; (**d**) Design D (proposed). The results in Table 4 show that the proposed additions result in an overall better training performance. The addition of the Capsule layer to both the multi-channel and single channel variant of the model architecture shows a clear improvement of the training and validation losses over the same number of training epoch in comparison to the variants without Capsules. Furthermore, whilst all architectures did not significantly overfit on the training data, the architecture variants with Capsules exhibited no difference between the training and validation losses and in some cases, the validation loss was marginally below the training loss, which indicates strong generalisation ability when using Capsules. The training plots in Figure 6 also show that the architecture variants with Capsules converge to low loss values significantly faster than the variants without Capsules, which means that without a fixed number of training epochs, the Capsule based models will require less training epochs to converge to the same loss values as the model variants without capsules. One drawback that can be observed however is the longer training times for the variants using Capsules, which is expected as Capsule-based Networks require more complex calculations than traditional NN architectures due to the dynamic routing algorithm. However, due to the faster convergence, reducing the number of epochs on the model variants with Capsules is feasible and will reduce the training time to a period comparable with the variants without capsules. The anomaly detection results in Table 5 show that both the inclusion of Capsules and a multi-channel input architecture result in stronger anomaly detection performance. Both variants with multi-channel inputs, Designs B and D, are able to perform more confident predictions, as shown by the MAE thresholds for each feature in the data. Due to this, the model variants with Capsules also achieve better \(F_{1}\) scores than the single channel variants.