SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: AUC score in the three real-world datasets. The higher AUC score represents the better performance, and the best AUC scores are highlighted in bold.
Method	PSM	SMD	SWaT	Average
Sampling	0.8791	0.9611	0.5348	0.7917
COPOD	0.8526	0.9230	0.7396	0.8384
ECOD	0.8394	0.9220	0.7532	0.8382
OCSVM	0.8708	0.6789	0.5379	0.6959
PCA	0.8617	0.9671	0.5324	0.7871
kNN	0.8640	0.8935	0.5806	0.7793
CBLOF	0.8681	0.9694	0.5378	0.7918
HBOS	0.8150	0.7393	0.8085	0.7876
IForest	0.8892	0.9218	0.7238	0.8450
LODA	0.8619	0.9180	0.6780	0.8193
VAE	0.8583	0.9674	0.5325	0.7861
DeepSVDD	0.8100	0.9187	0.5063	0.7450
LSTM-AE	0.8894	0.9698	0.6255	0.8283
MTAD-GAT	0.9093	0.9443	0.6386	0.8307
TFAD	0.8185	0.9386	0.6966	0.8179
Anomaly Transformer	0.7074	0.7150	0.6638	0.6954
Adversary	0.7199	0.5374	0.6487	0.6353
Our	0.9223	0.9759	0.8554	0.9179
D Additional detection results
In order to avoid the influence of SPOT parameters and thresholds on the results, we compare the methods from the perspective of the original anomaly scores. If an anomaly lasts too long, more weight will be assigned, resulting in an inaccurate metric. Thus, we aggregate each anomaly event. An anomaly event after aggregation is equivalent to only one timestamp, labeled as an anomaly, and scored as the maximum of the original anomaly range.
We use Area Under Curve (AUC) to evaluate the anomaly scores of each method. The value of AUC ranges between 0.5 and 1, while the closer to 1.0, the better the method. The results of the experiments are shown in Table 5.  $\mathrm{D}^3\mathrm{R}$  significantly outperforms existing methods, with a  $9\%$  average relative improvement over other SOTA methods. Specifically, we achieve  $1.30\%$ $(0.9093\rightarrow 0.9223)$ ,  $0.61\%$ $(0.9698\rightarrow 0.9759)$ , and  $4.69\%$ $(0.8085\rightarrow 0.8554)$  absolute improvement over the best SOTA performance on PSM, SMD and SWaT datasets, respectively. This experiment proves the superiority of  $\mathrm{D}^3\mathrm{R}$  once again.
E Anomaly score visualizations
We visualize the anomaly scores of the partial baseline and  $\mathrm{D}^3\mathrm{R}$  for a more intuitive comparison, as shown in Figure 6.  $\mathrm{D}^3\mathrm{R}$  consistently provided the most distinguishable anomaly scores in all three real-world datasets. In the PSM dataset, the vertical drift is insignificant. In this case, some of the baselines (MTAD-GAT) can achieve correct detection results, but there are still missed detections (Anomaly Transformer) and false detections (VAE). In the SMD and SWaT datasets, the vertical drift of the data is significant. The anomaly scores of baselines in the normal region are already far from the training data. The unstable normal data is likely to be mistakenly detected as an anomaly. Once the anomaly occurs, even if the anomaly scores of the baselines improve, they are not significant enough due to the large base.
F Run Times
We comprehensively compare training time, inference time, and model size for neural network-based models on the SMD dataset to validate the practicality of  $\mathrm{D}^3\mathrm{R}$  in the production environment. The outcomes are presented in Table 6.
Figure 6: Visualization of the anomaly scores in the three real-world datasets. The red area represents the ground truth of the anomaly event. In the first row, we visualize the raw data. In the remaining six rows, we visualize the anomaly scores provided by different methods. The moment with a higher anomaly score is more likely to be identified as an anomaly. Green boxes bound the successful cases. We normalize the data according to the statistical characteristics of the training set. Test denotes the actual value of the testing set after the normalization, while Train denotes the average value of the training set after the normalization, which is always 0.
Table 6: Run times in the SMD. The lower values represent the better performance. The best results are highlighted in bold, and the worst results are underlined.
Method	Training Time (s)	Inference Time (s)	Model Size (MB)
VAE	157.91	30.90	0.02
DeepSVDD	60.70	12.61	0.01
LSTM-AE	283.61	72.82	0.04
MTAD-GAT	188.52	60.23	1.20
TFAD	315.39	38.54	1.04
Anomaly Transformer	422.43	94.36	28.15
Our	399.32	104.12	109.35
Both the training set and the test set of SMD have approximately 16 days of data. Statistically, the training time for these models remains below 10 minutes, which is acceptable for real-world deployment and maintenance. The utilization of attention and its variants in the backbone network of our model leads to its larger size compared to previous algorithms. Thanks to the highly parallelized nature of the attention mechanism, both the training time and inference time of our model remain competitive. The inference process for 16 days of data necessitates a mere 104.12 seconds, satisfying the online, real-time detection criteria. Compared to the substantial enhancement (11%) in detection accuracy that we have achieved, the model size of 109.35MB remains affordable within the context of the expanding hardware resources of the present era. Additionally, there has been substantial recent work (such as Flowformer) on transformer linearization, which may aid in reducing the burden of our model. We plan to explore this aspect in our future research.
Recently, Deep Learning has been increasingly used in anomaly detection. It improves safety and prevents potential risks and financial losses by detecting anomalies in healthcare, industrial manufacturing, and autonomous driving. However, since real-time data is constantly changing, data characteristics can change drastically as time accumulates. Due to the obsolescence of the training data, our models cannot adapt to this situation and may provide incorrect judgments. In the future, we consider designing algorithms that adopt online updates to address this shortcoming.
The computational cost of our model is large due to the Transformer architecture used for the backbone network. In future work, we plan to replace it with a more lightweight and efficient base structure, such as dilated convolutions and graph neural networks. Furthermore, due to the shortcomings of the point-adjustment evaluation metric, in order to prove the superiority of  $\mathrm{D}^3\mathrm{R}$ , we have to compare various other metrics and provide rich visualizations. We plan to design a more intuitive and effective anomaly detection evaluation metric in future work.