SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For SWaT, we use SWaT_Dataset_Access_v0.csv and SWaT_Dataset_Normal_v1.csv from the folder SWaT.A1 & A2_Dec 2015 (manually converted from *.x1sx). We corrected some original flaws in the dataset (e.g. redundant blank spaces in some labels), and set the 5th and 10th columns to all 0. For WADI, we use the 2017 year dataset. Columns with excessive NaNs (more than half of the entire length) are deleted. Other NaNs are forward-filled. After deleting all the columns with excessive NaNs, the 86th column is further set to all 0. For PSM, we forward-fill all NaNs. For MSL and SMD, two additional blank channels are added to make the number of channels divisible by the number of heads. For trimSyn, we separated the dataset into training  $(t\in \{0,\dots ,9999\})$  and testing  $(t\in \{10000,\dots ,19999\})$  data (cf. [24]). For the testing data, we extracted segments within the time intervals  $t\in \{10000,\dots ,11719\} \cup \{11900,\dots ,12849\} \cup \{14630,\dots ,17699\} \cup \{17880,\dots ,18529\} \cup \{18710,\dots ,19999\}$  and concatenated them to form a single-entity test dataset. We also inserted additional segment IDs using one-hot encoding to enable segment identification. This process resulted in a test dataset with a single anomaly segment  $(t\in \{12670,\dots ,12849\})$  and a corresponding anomaly rate of  $2.34\%$ . In the case of the training data, we duplicated the original segment five times, concatenated them, and added the necessary segment IDs.
Table 4: Implementation details. (c) stands for the combined training method (cf. section 4.2)
Parameter\Dataset	SWaT	WADI	PSM	MSL	MSL (c)	SMAP	SMAP (c)	SMD	SMD (c)	trimSyn
Preprocess
Downsample	10	10	10	1	1	1	1	2	2	1
Normalization	Minmax	Minmax	Minmax	Minmax	Minmax	Minmax	Minmax	Minmax	Minmax	Minmax
Stride	10	10	10	10	10	10	10	10	10	10
W(for Mpt)	100	100	100	100	100	50	50	50	50	50
W0(for Mseq)	100	100	100	50	50	50	50	50	50	50
δ	20	20	20	6	6	6	6	6	6	6
Model architecture
# of heads	9	14	5	11	12	5	10	8	11	5
Dlat	10	10	10	10	10	10	10	10	10	4
ff_mult	4	4	4	4	4	4	4	4	4	4
Nperf	4	4	4	4	4	4	4	4	4	4
Nenc	8	8	8	8	8	8	8	8	8	4
Induced anomaly score
Gate function	soft	soft	soft	soft	soft	soft	soft	soft	soft	soft
d	16	16	64	128	32	64	32	16	256	256
Ratio of Ntrnfor θN	99.85%	99.85%	99.85%	99.85%	97.5%	99.85%	99.85%	99.85%	99.85%	99.85%
Training
Learn rate	10-4	10-4	10-4	10-4	10-4	10-4	10-4	10-4	10-4	10-4
Optimizer	Adam	Adam	Adam	Adam	Adam	Adam	Adam	Adam	Adam	Adam
Batch size	64	64	64	64	64	64	64	64	64	64
Training epochs	100	100	100	100	100	100	100	100	100	25
D The Point-adjusted Best F1 Score
Analogous to  $\mathrm{F1}^*$ , the point-adjusted best F1 score  $(\mathrm{F1}_{\mathrm{PA}}^{*})$  corresponds to  $\mathrm{F1}^*$  calculated after point-adjustment. Table 5 shows the  $\mathrm{F1}^*$  and  $\mathrm{F1}_{\mathrm{PA}}^{*}$  of different algorithms, including NPSR, applied to several datasets. We did not show the results for the trimSyn dataset with point-adjustment. However, by applying NPSR and the same setting as without using point-adjust, we can achieve  $\mathrm{F1}_{\mathrm{PA}}^{*} = 1$  within the first few epochs. This indicates the point with the highest anomaly score lies within the single anomaly segment.
The results suggest that  $\mathrm{F1}_{\mathrm{PA}}^{*}$  may not be reliable - on the SWaT, WADI, PSM, and MSL datasets, simple heuristic approaches (e.g. using the mean squared value of an input time point as the anomaly score) outperform all deep learning methods when evaluated using  $\mathrm{F1}_{\mathrm{PA}}^{*}$ . Moreover, optimizing on  $\mathrm{F1}_{\mathrm{PA}}^{*}$  does not necessarily guarantee a higher  $\mathrm{F1}^{*}$ . NPSR is optimized on  $\mathrm{F1}^{*}$  by tuning the algorithm-specific parameters (e.g.  $d$ ) and general parameters. To additionally optimize on  $\mathrm{F1}_{\mathrm{PA}}^{*}$ , we simply added spikes to the induced anomaly score ( $\hat{A}_{\text{spike}}(\cdot)$ ) with value  $\infty$  for some fixed interval  $s$ . The results show that  $\hat{A}_{\text{spike}}(\cdot)$  can also achieve competitive  $\mathrm{F1}_{\mathrm{PA}}^{*}$  values.
Table 5: Point-adjusted best F1 score  $(\mathrm{F1}_{\mathrm{PA}}^{*})$  and best F1 score  $(\mathrm{F1}^{*})$  results on several datasets, with bold text denoting the highest and underlined text denoting the second highest value. The deep learning methods are sorted with older methods at the top and newer ones at the bottom.
Dataset	SWaT	WADI	PSM	MSL	SMAP	SMD
Metric	F1*PA	F1*	F1*PA	F1*	F1*PA	F1*	F1*PA	F1*	F1*PA	F1*	F1*PA	F1*
Simple Heuristics [11, 30, 31]	0.969	0.789	0.965	0.353	0.985	0.509	0.965	0.239	0.961	0.229	0.934	0.494
DAGMM [26]	0.853	0.750	0.209	0.121	0.761	0.483	0.701	0.199	0.712	0.333	0.723	0.238
LSTM-VAE [22]	0.805	0.776	0.380	0.227	0.809	0.455	0.854	0.212	0.756	0.235	0.808	0.435
MSCRED [24]	0.807	0.757	0.374	0.046	0.626	0.556	0.936	0.250	0.866	0.170	0.841	0.382
OmniAnomaly [9]	0.866	0.782	0.417	0.223	0.664	0.452	0.901	0.207	0.854	0.227	0.962	0.474
MAD-GAN [23]	0.815	0.770	0.556	0.370	0.658	0.471	0.917	0.267	0.865	0.175	0.915	0.220
MTAD-GAT [27]	0.860	0.784	0.602	0.437	0.780	0.571	0.908	0.275	0.901	0.296	0.908	0.400
USAD [28]	0.846	0.792	0.430	0.233	0.725	0.479	0.911	0.211	0.819	0.228	0.946	0.426
THOC [18]	0.881	0.612	0.506	0.130	0.895	-	0.937	0.190	0.952	0.240	0.541	0.168
UAE [11]	0.869	0.453	0.957	0.354	0.936	0.427	0.920	0.451	0.896	0.390	0.972	0.435
GDN [12]	0.935	0.810	0.855	0.570	0.923	0.552	0.903	0.217	0.708	0.252	0.716	0.529
GTA [41]	0.910	0.761	0.84	0.531	0.855	0.542	0.911	0.218	0.904	0.231	0.919	0.351
Anomaly Transformer [40]	0.941	0.019	0.714	0.015	0.979	0.022	0.936	0.021	0.967	0.019	0.923	0.021
TranAD [25]	0.815	0.669	0.495	0.415	0.882	0.649	0.949	0.251	0.892	0.247	0.961	0.310
NPSR (combined)	-	-	-	-	-	-	0.960	0.261	0.978	0.511	0.850	0.252
NPSR	0.953	0.839	0.938	0.642	0.957	0.648	-	0.551	-	0.437	-	0.535
E Source of Data
Table 6 shows the data sources used to produce Table 5, as well as the sources for section 4.3. The reference number is followed by a number between 1 and 3, where 1 indicates that the data comes from the original work, 2 indicates that the data comes from reproduced values of another literature, and 3 indicates that we have reproduced the values using public repositories..
Table 6: Data sources for algorithms and datasets. **Reproduced by using the squared value of channel 1 as  $A(\cdot)$ . †Reproduced by using the squared value of channel 9 as  $A(\cdot)$ .
Dataset	SWaT	WADI	PSM	MSL	SMAP	SMD	trimSyn
Metric	F1*PA	F1*	F1*PA	F1*	F1*PA	F1*	F1*PA	F1*	F1*PA	F1*	F1*PA	F1*	F1*
Simple Heuristic	[30] - 1	[30] - 1	[30] - 1	[30] - 1	[30] - 1	[31] - 1	**	[31] - 1	[30] - 1	[31] - 1	[30] - 1	[31] - 1	[30] - 1	†
DAGMM	[30] - 2	[25] - 3	[30] - 2	[30] - 2	[25] - 3	[25] - 3	[30] - 2	[30] - 2	[30] - 2	[30] - 2	[30] - 2	[30] - 2	[30] - 2	[25] - 3
LSTM-VAE	[28] - 2	[28] - 2	[28] - 2	[28] - 2	[40] - 2	[45] - 3	[28] - 2	[30] - 2	[28] - 2	[30] - 2	[28] - 2	[30] - 2	[30] - 2	[45] - 3
MSCRED	[25] - 2	[25] - 3	[25] - 2	[25] - 3	[25] - 3	[25] - 3	[25] - 2	[25] - 3	[25] - 2	[25] - 3	[25] - 2	[25] - 3	[25] - 3	[25] - 3
OmniAnomaly	[30] - 2	[30] - 2	[30] - 2	[30] - 2	[25] - 3	[25] - 3	[9] - 1	[30] - 2	[9] - 1	[30] - 2	[9] - 1	[30] - 2	[30] - 2	[25] - 3
MAD-GAN	[25] - 3	[23] - 1	[25] - 3	[23] - 1	[25] - 3	[25] - 3	[25] - 2	[25] - 3	[25] - 2	[25] - 3	[25] - 2	[25] - 3	[25] - 3	[25] - 3
MTAD-GAT	[46] - 2	[27] - 3	[46] - 2	[27] - 3	[27] - 3	[27] - 3	[27] - 1	[27] - 3	[27] - 1	[27] - 3	[46] - 2	[27] - 3	[25] - 3	[25] - 3
USAD	[28] - 1	[28] - 1	[28] - 1	[28] - 1	[25] - 3	[25] - 3	[28] - 1	[30] - 2	[28] - 1	[30] - 2	[28] - 1	[30] - 2	[30] - 2	[25] - 3
THOC	[18] - 1	[30] - 2	[30] - 2	[30] - 2	[40] - 2	-	[18] - 1	[30] - 2	[18] - 1	[30] - 2	[30] - 2	[30] - 2	[30] - 2	-
UAE	[11] - 1	[11] - 1	[11] - 1	[11] - 1	[11] - 3	[11] - 3	[11] - 1	[11] - 1	[11] - 1	[11] - 1	[11] - 1	[11] - 1	[11] - 1	[11] - 3
GDN	[30] - 2	[12] - 1	[30] - 2	[12] - 1	[12] - 3	[12] - 3	[30] - 2	[30] - 2	[30] - 2	[30] - 2	[30] - 2	[30] - 2	[25] - 3	[25] - 3
GTA	[41] - 1	[41] - 3	[41] - 1	[41] - 3	[41] - 3	[41] - 3	[41] - 1	[41] - 3	[41] - 1	[41] - 3	[41] - 3	[41] - 3	[41] - 3	[41] - 3
AnomalyTransformer	[40] - 1	[40] - 3	[40] - 3	[40] - 3	[40] - 1	[40] - 3	[40] - 1	[40] - 3	[40] - 1	[40] - 3	[40] - 1	[40] - 3	[40] - 3	[40] - 3
TranAD	[25] - 1	[25] - 3	[25] - 1	[25] - 3	[25] - 3	[25] - 3	[25] - 1	[25] - 3	[25] - 1	[25] - 3	[25] - 1	[25] - 3	[25] - 3	[25] - 3
F Model and Parameter Selection Heuristics
We believe that the most important idea in model selection is to achieve a balance between effectively utilizing both point and sequence-based models and carefully selecting the appropriate parameters. As an extreme example, we optimize our algorithm on the Mackey-Glass anomaly benchmark (MGAB) [47], which is a univariate time series dataset, and find that sequence-based models can significantly outperform point-based models when using their reconstruction errors as  $A(\cdot)$ . As in Fig. 6, we see that  $\mathcal{M}_{seq}$  can correctly identify the anomalies, whereas  $\mathcal{M}_{pt}$  did not learn at all. This is reasonable, as point-based models only consider a single time point. In general, the lower the dimensionality of the dataset, the harder it gets for a point-based model to learn statistically meaningful representations. Moreover, looking at the high-dimensional datasets reported in Table 3, the induced anomaly score remains important for some datasets: For the MSL dataset,  $F1^*$  improves 0.1 compared to only using point-based reconstruction. In Fig. 7b, when using a soft gate function,  $F1^*$  improves 0.047 compared to using the point-based anomaly score. Since the amount of improvement depends on the statistical structure of the test data, it is still useful to consider using both  $\mathcal{M}_{seq}$  and  $\mathcal{M}_{pt}$  in general.
The difficulty in choosing the parameters for unsupervised time series anomaly detection stems from the absence of anomalies in the training dataset. The selection of soft or hard gate functions, the value for  $d$ , and the ratio for  $\theta_N$  largely depend on how we presume the anomaly will occur based on domain knowledge. For instance,  $d$  controls the distance that anomaly scores may propagate. This value should be higher if we presume the average anomaly length is long and vice versa. If
Figure 6: (a) best F1 scores and (b) AUCs using point and sequence-based models as the anomaly score on the Mackey-Glass anomaly benchmark.
the anomaly is expected to occur abruptly and significantly, there would be a clear gap between the distribution of nominality scores for normal and anomaly data (Fig. 8a). In this case, a hard gate function should be chosen as it allows anomaly scores to propagate through time points without reduction, as long as an anomaly time point has a nominality score lower than  $\theta_N$ . Conversely, if the anomaly occurs progressively, the distribution of nominality scores is likely to overlap (Fig. 8b). Here, a soft gate function will be more appropriate to prevent excessive accumulation of anomaly scores on normal time points, reducing false-positives. A dataset could contain both abrupt and progressive anomalies. However, based on Table 3, it is evident that using a soft gate function generally yields better performance compared to a hard gate function. This suggests that the distribution of nominality scores is predominantly overlapped, which is also evident in Fig. 4 and Fig. 7a.
Figure 7: Training results on the WADI dataset. Nominality score vs (a) anomaly score from point-based reconstruction, (b) best F1 score  $(d = 16)$ , (c) false positive rate  $(d = 16)$ , and (d) best F1 score  $(d = 1)$  using different  $\theta_N$ .
Figure 8: Illustration of different distributional relationships between the nominality scores of normal and anomaly data. (a) No overlap with a paired threshold and hard gate function. (b) Overlapped with threshold and soft gate function.
The detection of anomalies in time series data can minimize downtime and avert financial losses. Utilizing real-time monitoring of system conditions, anomaly detection techniques for time series data can automatically detect deviations from the expected system behavior, thereby avoiding potential risks and financial harm. This has the potential to reduce the need for manual monitoring of faults and to expedite decision-making processes. Additionally, it can promote the sustainability of AI by preventing energy wastage and system malfunction.
Despite exhibiting competitive performance against other models, the proposed NPSR algorithm has a few limitations. The point-based model used in training does not incorporate temporal information, which makes it challenging to effectively reconstruct low-dimensional datasets. This issue is particularly challenging for univariate time series since raw inputs would not work for point-based models. One possible solution to this problem is to increase dimensionality by aggregating multiple time points. However, the effectiveness of this approach is yet to be confirmed.
Another limitation of NPSR is the absence of an automatic threshold  $(\theta_{a})$ -finding method, which makes it difficult to determine a suitable threshold when deploying the model. To address this issue, one can define a target false positive rate and estimate the threshold that achieves this target rate using the validation set since only normal data is needed. Similarly, estimating the optimal values for  $\theta_{N}$ ,  $d$ , and selecting the model used for calculating  $A(\cdot)$  will be an important future work.
Within Fig. 5b of the main text, a false negative instance was identified in the temporal range between  $t = 14800$  and  $t = 14900$  when employing the induced anomaly score. According to the WADI dataset, this anomaly spans approximately 14.26 minutes and is characterized as "Damage 1 MV 001 and raw water pump to drain Elevated Reservoir tank." Notably, our analysis suggests that when assessing individual time points, the model  $\mathcal{M}_{pt}$  encounters difficulty in recognizing this anomaly. Conversely, the model  $\mathcal{M}_{seq}$  excels in identifying time-dependent relationships, making it more effective in capturing such contextual anomalies. The observed disparity in anomaly detection implies that this section comprises a relatively higher proportion of contextual anomalies than point anomalies. Consequently, when using  $\mathcal{M}_{seq}$ , we achieve a higher anomaly score. However, our current approach utilizes the reconstruction error of  $\mathcal{M}_{pt}$  as the basis for the anomaly score calculation, thus neglecting the effectiveness of the reconstruction error generated by  $\mathcal{M}_{seq}$ . Consequently, the induced anomaly score fails to surpass the predefined threshold. In light of these findings, an essential avenue for future research is to investigate methods for selecting the model to be used in the computation of  $A(\cdot)$ . This undertaking holds promise for enhancing the overall performance and accuracy of anomaly detection in time series data.