## LSTMCaps Autoencoder Network
### Experimental Results
#### 2.6.2 Drone Dataset Anomaly Detection  
The drone dataset was acquired from previous work conducted on Virtual Sensing fault detection [38]. The datasets consists of 3 subsets of data sampled at 150 Hz: 2 subsets of data are taken from a fully functional drone, where 1 subset is sampled for a duration of 600 s, giving 90,000 samples, and the other for 30 s, giving 4500 samples. These subsets will be used for the training and validation sets, respectively. The third subset is acquired from a drone with faulty controls and is recorded for a duration of 30 s, similarly giving 4500 samples. This subset will be used as the test data. From this dataset, 3 features are \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Source** & **Degrees of Freedom (DP)** & **Sum of Squares (SS)** & **Mean Square (MS)** & **F-Ratio** & \(p\)**-Value** \\ \hline Between Groups & \(DF_{b}=k-1\) & \(SSB=\sum_{n}(\bar{X}_{j}-\bar{X})^{2}\) & \(MSB=\frac{SBS}{D_{n}}\) & \(F=\frac{SBS}{BSB}\) & Right tail \(F(DF_{b},DF_{b})\) \\ Error & \(DF_{b}=N-k\) & \(SSE=\sum\Sigma(X-\bar{X}_{j})^{2}\) & \(MSE=\frac{SBS}{D_{n}}\) & & \\ Total & \(DF_{b}=N-1\) & \(SST=\sum\Sigma(X-\bar{X})^{2}\) & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: One way ANOVA source table, courtesy of [37], where: X = individual observation, \(\bar{X}_{j}=\) sample mean of jth group, \(\bar{X}\) = overall sample mean, k = number of groups, N = number of observations per group. being used. Feature 1 represents the pitch, feature 2 represents the roll, and feature 3 represents the throttle of the drone. These features overall represent the input signals to the drone, so anomaly detection on these signals will serve the purpose of detecting any abnormal control issues on the drone. More details on the acquisition of this data can be found in Ref. [38]. A visualisation of the training data, the validation data, and the test data with various anomalies outlined is provided in Figure 4. As the data is unlabelled, the anomalies were manually labelled so that a measure of the anomaly detection performance of each NN model could be attained. The metric used for this is the \(F_{1}\) score, which is defined in Equation (12) [39]. \[F_{1}=\frac{TP}{TP+\frac{1}{2}(FP+FN)} \tag{12}\]  
The \(F_{1}\) score can also be expressed in terms of Precision, Equation (13), and Recall, Equation (14). The \(F_{1}\) score expressed in these terms is described in Equation (15). Figure 4: Visualisation of the drone dataset. \[Precision=\frac{TP}{TP+FP} \tag{13}\]  
\[Recall=\frac{TP}{TP+FN} \tag{14}\]  
\[F_{1}=2\cdot\frac{precision\times recall}{precision+recall} \tag{15}\]  
where \(TN\) = True positive, \(TN\) = True negative, \(FP\) = False positive and \(FN\) = False negative. For this dataset, it was found to be advantageous to normalise the data before using the Z-Score normalisation to improve the performance of the models. This operation transforms the mean of the sample to 0 and the standard deviation to 1. This is accomplished using Equation (16). \[z=\frac{(X-\mu)}{\sigma} \tag{16}\]  
This experiment aims to explore the training capability of the proposed NN architecture (Design D) by making a comparison with a non-hybridised single channel LSTM NN model (Design A), a single channel hybridised LSTMCaps NN model (Design B) and a multi-channel non-hybridised LSTM NN model (Design C). A visualisation of the models is provided in Figure 5.  
Since the trainable parameters of the proposed model, Design D and the multi-channel LSTM, Design C, will scale with the number of features in the data, the size of each model was adjusted so that all networks being trained have a similar number of parameters. This is done to show the difference in performance of each NN model variant regardless of the number of parameters, which will result in a fairer comparison of the variants. The trainable parameters for each model variant are shown in Table 2. The models were optimised for the drone dataset using the \(F_{1}\) score as the target variable to be optimised.