# A Review of Anomaly Detection in Spacecraft Telemetry Data
## 4 Datasets and Performance Evaluation Indices
### Datasets  
The performance of the algorithms and methods introduced in the previous section was demonstrated on several different datasets. Several anomaly detection methods was tested on synthetic [45] or simulated spacecraft telemetry datasets [37; 48; 54], while over 80% of anomaly detection application research in spacecraft was done using real public or anonymized datasets [7; 8; 9; 20; 22; 24; 26; 27; 28; 29; 32; 36; 44; 47]. In simulated datasets, the anomalous data are produced by injecting fault into the ordinary data. For example, in [54], point and contextual anomalies were injected into the real data to evaluate the efficiency of TCN method. In [37], [PERSON] et al. inserted fault \begin{table}
\begin{tabular}{c c c} \hline
**Approach** & **Precision (\%)** & **Recall (\%)** \\ \hline LSTM [20] & 87.5 & 80.0 \\ LSTM + transfer & 79.4 & 83.5 \\ learning [22] & 94.74 & 94.17 \\ Attention TCN [7] & 80.76 & 58.87 \\ ResNet [8] & 69.31 & 58.09 \\ ELMs [27] & 76.4 & 77.9 \\ SVM [38] & 84.1 & 85.7 \\ Bi-Transformer [32] & 85.4.0 & 72.4 \\ TranAD [31] & 85.40 & **99.99** \\ GCN [30] & **94.86** & 99.59 \\ GNU [56] & 89.2 & 89.6 \\ GRU + VAE [60] & 81.41 & 94.46 \\ LSTM + GAN [60] & 90.28 & 92.45 \\ LSTM + multi-scale & 87.37 & 86.54 \\ strategy [55] & 87.37 & 86.54 \\ \hline \end{tabular}
\end{table} 
Table 2: Performance metrics of different AD methods on NASA SMAP/MSL datasets. \begin{table}
\begin{tabular}{c c c c} \hline
**Approach** & **Computational** & **Training Time (s)** & **Inference Time (s)** \\ \hline LSTM + transfer & — & 71.45 & — \\ learning [22] & — & 71.45 & — \\  & Virtual machine & & \\ ResNet [8] & equipped with 8 & 135 & — \\  & CPU cores: Intel & & \\  & Xeon Platinum 8260 & & \\ FCN [8] & CPU @ 2.40 GHz & 555 & — \\  & and 16 GB of RAM & & \\ Bi-Transformer [32] & — & 109.75 & — \\ TranAD [31] & — & 0.66 & — \\  & Huawei G5500 & & \\ GRU + VAE [60] & Linux server with & — & _0.013345_ \\  & NVIDIA Tesla V100 & & \\ LSTM+GAN [60] & GPU & — & _0.01085_ \\ \hline \end{tabular}
\end{table} 
Table 3: Computational time of AD methods on NASA SMAP/MSL datasets. Values in italics present the run time for each sample, while the rest correspond to the average training/inference time for each channel/parameter. data into the normal data of a control subsystem of an actual in-orbit satellite, using the Matlab/Simulink environment, to detect and identify anomaly type in telemetry data. Numerous real datasets were designed and used to evaluate the state-of-the-art anomaly detection methods. These datasets, collected from different spacecraft telemetry systems, mainly include the following:  
* ESA datasets: ESA benchmark datasets such as Sentinel-1 [36] and OPS-SAT [26; 40] were tested with several numbers of ML and DL algorithms for anomaly detection. The ESA Sentinel-1 dataset contains data from 10 channels, while OPS-SAT includes data from 9 channels. In [9], another ESA database, collected from the XMM-Newton orbit, was explored to detect anomalous behavior in space operations using OOL alarms. The XMM-Newton dataset contains around 2000 housekeeping telemetry parameters. * NASA SMAP/MSL datasets: The Soil Moisture Active Passive (SMAP) satellite and Mars Science Laboratory (MSL) rover data are the most commonly used telemetry datasets [3; 7; 8; 20; 29; 30; 32; 64] in spacecraft anomaly detection due to dataset availability. The SMAP and MSL are very different missions. The SMAP collects information about the moisture level within the first five centimeters of soil to obtain a better understanding of the carbon and water cycle, while the MSL was designed to study the potential habitability in the Red Planet in the past and present environments. The two data missions were published by the NASA Jet Propulsion Laboratory. They contain 81 channels of telemetry data. The SMAP/MSL data consist of 496,444 telemetry values associated with 105 anomaly sequences. Anomaly measurements present only 16.12% of the total samples. They contain about 75387 anomalous samples.