# Stream: Spatial Trajectory Enhanced Attention Mechanism for Abnormal UAV Trajectory Detection
## 4 Evaluation
### Performance Evaluation of STEAM  
As described in Section 3, we demonstrate through experiments how suitable STEAM is for UAV trajectory data compared to conventional methods. To represent the performance of STEAM only, we implement a simple neural network model using STEAM as shown in Figure 6c. Then, we represent the utility of STEAM using CNN models that are the same as Figure 6c but without STEAM as a comparator. In addition, for the comparison group, the model with CBAM applied as shown in Figure 6a is implemented under similar conditions, and the ViT model is implemented as shown in Figure 6b. Since CBAM is a performance-like model with SE, comparison with SE is omitted [49]. We quantitatively evaluate the performance for each model shown as Table 1, and each metric is as follows [50]:  
* Loss: This metric represents the error between the predicted and actual values of the data set. A smaller loss value means that the prediction value of the model is closer to the actual data. * Accuracy: Accuracy measures the percentage of correct predictions made by the model among all predictions. High accuracy means that the model can correctly identify or classify data points. * Precision: Precision evaluates the model's ability to accurately identify only the data points involved. High precision indicates that the model is likely to be accurate when predicting a class or outcome. * Recall: Recall, also known as sensitivity, is the percentage at which the model predicted the correct answer among those that are actually correct. We propose that it can be an important indicator when it should not be mispredicted in practice such as UAV identification. Figure 7 displays the overall results for both test and verification data. To evaluate the performance of our study on limited UAV identification data, we trained the model using only 50% of the entire dataset and devoted the remaining 50% to validation. Experimental results highlight the effectiveness of STEAM in processing UAV trajectory data. STEAM achieved a remarkable accuracy of 97.92% and an impressive validation accuracy of 96.88% during training. Specifically, STEAM achieved a low loss of 0.1698, indicating its effectiveness in closely predicting the true values [51]. At the initial epoch, STEAM's validation accuracy began at a modest rate, aligning closely with the training accuracy, which suggests consistency in model learning. As the epochs progressed, a steady increase in validation accuracy was observed, indicating the model's improving ability to generalize. This demonstrates that STEAM has emphasized its ability to successfully capture most of the relevant functions in the data and identify them well in the invisible data. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Loss** & **Accuracy** & **Precision** & **Recall** \\ \hline \multicolumn{5}{c}{Training Performance} \\ \hline CNN & 1.7753 & 0.3125 & 1.0000 & 0.0417 \\ CBAM [44] & 0.0568 & 0.9924 & 1.0000 & 1.0000 \\ STEAM & 0.1698 & 0.9792 & 1.0000 & 0.9375 \\ ViT [46] & 2.7292 & 0.0303 & 0.0000 & 0.0000 \\ \hline \multicolumn{5}{c}{Validation Performance} \\ \hline CNN & 1.5938 & 0.5052 & 1.0000 & 0.0625 \\ CBAM [44] & 0.1941 & 0.8333 & 0.9886 & 0.9010 \\ STEAM & 0.1890 & 0.9688 & 0.9831 & 0.9115 \\ ViT [46] & 2.5221 & 0.0833 & 0.0000 & 0.0000 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of model performances on UAV trajectory data. Figure 6: Performance evaluation of STEAM by comparing various attention mechanisms on UAV trajectory images. (**a**) Structure of CBAM model; (**b**) structure of ViT model; (**c**) structure of our STEAM model. In comparison, CNN models without STEAM layers exhibited a fluctuating performance, with a training accuracy starting at around 31% and a validation accuracy at approximately 50%. This fluctuation was particularly noticeable between epochs 2 and 4, where a divergence between training and validation accuracy indicated potential overfitting to the training data.