# A Review of Anomaly Detection in Spacecraft Telemetry Data
## 4 Datasets and Performance Evaluation Indices
### Evaluation Indices  
The implementation of different methods provides the opportunity to compare the obtained results. In order to quantify the anomaly detection performance, the vast majority of papers, especially those which adopt ML and DL models [7; 20; 21; 22; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 68], make use of confusion matrices and visualizations. A confusion matrix displays the number of true or false positives (TPs/FPs) and the respective true or false negatives (TNs/FNs). A TP is registered if the detected anomalous sequence overlaps with a ground truth anomalous sequence. If multiple detected anomalies overlap with one labeled sequence, only one TP is recorded. A FN is recorded if no predicted sequences overlap with the ground truth anomalous sequence. On the other hand, a FP is registered if a detected anomaly sequence does not overlap with a true anomaly sequence. The number of correctly detected anomalies is determined by the sum of TPs or TNs, while the set of incorrect anomalies is obtained from the total number of FPs or FNs.  
Accuracy, precision, recall, \(F_{1}\) score, true positive ratio (TPR), false positive ratio (FPR), false alarm rate, receiver operating characteristic (ROC), area under curve (AUC) and prediction error (RMSE, MQE...), among others, were employed in the literature as metrics to evaluate the performance of the anomaly detection models.  
Some papers, such as in [6], consider that the accuracy metric can be a misleading measure when the data are highly imbalanced. In this case, it is recommended to conduct model evaluation in terms of precision, recall, \(F_{0.5}\) and \(F_{1}\) scores. A high precision means that the model has a low false positive rate, while a high recall means that a low number of true anomalies are undetected.  
\(F_{1}\) and \(F_{0.5}\) are evaluation metrics that assess the overall effectiveness of an anomaly detection model by integrating both recall and precision. The \(F_{1}\) score equally weights precision and recall, providing a balanced measure of a model's performance. It is especially useful when both precision and recall are equally important in the anomaly detection problem. The \(F_{0.5}\) score places more emphasis on precision than recall, making it more suitable for situations where reducing false positives is a priority. \(F_{1}\) and \(F_{0.5}\) are defined as follows:  
\[F_{1}\;score=\frac{2\times Precision\times Recall}{Precision+Recall} \tag{1}\]  
\[F_{0.5}\;score=\frac{1.25\times Precision\times Recall}{0.25\times Precision+ Recall} \tag{2}\]  
The TPR determines the rate of anomalies correctly identified by a model, while the FPR measures the proportion of normal instances incorrectly reported as anomalies (false alarms).  
The thresholding methods use simple performance metrics such as the false positive rate and accuracy. In addition to anomaly detection performance metrics, some studies [9; 22; 32; 47] have accorded importance to the computation time.