Nominality Score Conditioned Time Series Anomaly Detection by Point/Sequential Reconstruction
Authors: Chih-Yu Lai
Fan-Keng Sun
Zhengqi Gao
Jeffrey Lang
Duane S Boning
Venue: NeurIPS 2023 poster (2023)
Keywords: time series
anomaly detection
point anomalies
contextual anomalies
nominality score
induced anomaly score
TL;DR: Our time series anomaly detection approach combines point and sequence-based models to derive a nominality score, and furthermore, an induced anomaly score, which is proven superior under some conditions and outperforms most state-of-the-art models.

Abstract: Time series anomaly detection is challenging due to the complexity and variety of patterns that can occur. One major difficulty arises from modeling time-dependent relationships to find contextual anomalies while maintaining detection accuracy for point anomalies. In this paper, we propose a framework for unsupervised time series anomaly detection that utilizes point-based and sequence-based reconstruction models. The point-based model attempts to quantify point anomalies, and the sequence-based model attempts to quantify both point and contextual anomalies. Under the formulation that the observed time point is a two-stage deviated value from a nominal time point, we introduce a nominality score calculated from the ratio of a combined value of the reconstruction errors. We derive an induced anomaly score by further integrating the nominality score and anomaly score, then theoretically prove the superiority of the induced anomaly score over the original anomaly score under certain conditions. Extensive studies conducted on several public datasets show that the proposed framework outperforms most state-of-the-art baselines for time series anomaly detection.

Nominality Score Conditioned Time Series Anomaly Detection by Point/Sequential Reconstruction
Chih-Yu La Department of EECS, MIT Cambridge, MA 02139 Chihyul@mite @du
Fan-Keng Sun Department of EECS, MIT Cambridge, MA 02139
Zhengqi Gao
Department of EECS, MIT Cambridge, MA 02139 zhengqi edu
fankeng@mit edu
Jeffrey H. Lang
Duane S. Boning
Department EECS, MIT Cambridge, MA 02139
Department of EECS, MIT Cambridge, MA 02139 boningomtl. mit. edu
lang@mit edu
Abstract
Time series anomaly detection is challenging due the complexity and variety of patterns that can occur. One major difficulty arises from modeling time-dependent relationships to find contextual anomalies while maintaining detection accuracy for point anomalies. In this paper. we propose a framework for unsupervised time series anomaly detection that utilizes point-based and sequence-based reconstruction models. The point-based model attempts to quantify point anomalies, and the sequence-basee baaeed model attempts quantify both point and contextual anomalies. Under the formulation that the observed time point is a two-stage deviated value from a nominal time point, we introduce a nominality score calculated from the ratio of combined value of the reconstruction errors. We derive induced anomaly score by further integrating the nominality score and anomaly score, then theoretically prove the superiority of the induced anomaly score over the original anomaly score under certain conditions. Extensive studies conducted on several public datasets show that the proposed framework outperforms most state-of-the-art baselines for time series anomaly detection.
Introduction
Time series anomaly detection involves identifying unusual patterns or events in sequence of data collected over time [T]. This technique is crucial in fields such as finance (2), healthcare [], manufacturing [4. transportation [5). and more (6). comprehensive evaluation of different techniques can be found in [7]. Time series anomaly detection using unsupervised learning approaches favored by recent studies due the fact that they don't require labeled data and have better detection of unseen anomalies [ which is also the approach used in this work. Most unsupervised time series anomaly detection methods involve calculating an anomaly score each time point and then comparing this score to some threshold. For calculating this score, we can categorize different methods into three main groups: Reconstruction-bant time series data and comparing the reconstructed data with the actual data 99, 10 [T]. Prediction- based methods involve predicting the next value in the time serics and comparing it with the actual value 2 13 [4]. Dissimilarity-bart based methods measure the distance between the value obtained from the model and the distribution or cluster the aggregated data [1] [6 17
methods involve reconstructing the original
20). There
can also be hybrid techniques where multiple methods are applied.
"ode available https:/ggiiua.ccom//dddww.dddddd
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

While the classification approaches for the types of anomalies differ in the literature 71, we will focus on two classes: point anomalies and contextual anomalies. Point anomalies refer individual data points that significantly deviate from the expected behavior of the time series and can be detected by observing the data a single time point. Contextual anomalies, on the other hand, refer to data points that deviate from the expected behavior of the time series in specific context or condition. These anomalies cannot be detected by observing the data at single time point and can only be detected by observing the contextual information. As result, point anomalies can be detected using any general anomaly detection technique that does not require temporal information. Such models are called point-based models in this work. However, detecting contextual anomalies requires model that can learn temporal information. Such models require a sequence of input, hence they are termed sequence- based models in this work. A time point may contain both point and contextual anomalies. Obviously, contextual anomalies are harder to detect. An important trade-off arises when a model tries to learn dependent relationships for detecting contextual anomalies but loses precision or accuracy in finding point anomalies. This trade-off becomes more significant in high-dimensional data, where modeling the temporal relationship is difficltt (2].
Our intuition comes from the observation that state-of-the methods using sequence-based recon- struction models encounter the point-contextual detection trade-off, resulting in noisy reconstruction results and suboptimal performance. As an alternative, we start by experimenting with point- based reconstruction methods, which exhibit lower variance they do not require the modeling of time- dependent relationships. Despite the absence of temporal information, we find that the corresponding anomaly score for point-based reconstruction model can already yield competitive performance. To further bridge the gap between point-based and sequence- score that be calculated from their outputs and derive an induced anomaly score based on the nominality score and the original anomaly score. We find that the induced anomaly score can be superior the original anomaly score. To provide theoretical proof of our findings, we frame the reconstruction process as means to fix the anomalies and identify underlying nominal time points and prove that the induced anomaly score can always perform better or just as well as the original anomaly score under certain conditions. We also conduct experiments on several single and multi-entity datasets and demonstrate that our proposed method surpasses the performance of recent state-of of-the-art techniques. We coin our method Nominality score conditioned time series anomaly detection by Point/Sequential Reconstruction (NPSR).
models, we introduce nominality
2 Related Works
Reconstruction- based techniques have seen a diversity of approaches over the years. The simplest reconstruction technique involves training a separate model sequentially for each channel using UAE
[1. One type of improvement focuses on network architecture, with notable examples like LSTM-
VAE 22), MAD-GAN (23), MSCRED [24]. Amniln [9] and TranAD 25 Additionally, hybrid architectures like DAGMM 26 and MTAD-GAT (27 have been proposed. Another type aims to improve the anomaly score instead of using the original reconstruction error. Designing this anomaly score involves a considcrable amount of art, given the high diversity of methods for calculating it across studies. For instance, USAD uses two weighted reconstruction errors (28), OmniAnomaly employs the "reconstruction probability" as an alternative anomaly score (9. MTAD- GAT combines forecasting error and reconstruction probability (27]. and TranAD uses integrated reconstruction error and discriminator loss as the anomaly score (25). In the context of network architecture, method utilizes straightforward performer-based structure without incorporating any specialized components. Based on our insights into the point-sequential detection tradeoff, our approach stands out by integration of point-based and sequence-based reconstruction errors for competitive performance.
3 Methods
3.1 Problem Formulation
Let X {x1,..., } denote multivariate time series with x, E RR., where T the time length
and D is the dimensionality or number of channels. There exists a corresponding set of labels y = {yi,..., "r......: Be E (0. indicating whether the time point normal (μı = 0) or anomalous (y/+ 1). For given X, the goal is to yield anomaly scores for all time points {ai,... .... (,), a, E R and
corresponding threshold 0. such that the predicted labels ý {ū1..., ..., ÿT }. where ÿe A 1 20. match

---

y as much as possible. To quantify how matched y and is, or how good a is for potentially yielding a good y, there are several performance metries that takes either a or y into account [2T 29 30 3T.
This work mainly focuses on the best F1 score (F1*) without point-adjust, also known as the point- wise Fl score, which is defined as the maximum possible Fl score considering all thresholds. (A complete derivation for F1 is covered in Appendix
3.2 Nominal Time Series and Two-stage Deviation
We denote the observed data as X = (x..-.., x,. where x E RDD Assume that for each x", there
exists corresponding underlying nominal time series data X* {xt, ...,...' .,. that comes from nominal time- dependent process x = ff" (t): N R.. The corresponding total deviation at (Ax)) is defined Ax! & x x We denote X as the set of all possible x for all i E (1,..., 7). Ax can be separated into two additive factors Ax; and Ax,, such that, by definition, X5 X; + Axt and x = =x + Axf. We define Axÿ as the in-distribution deviation, where xj X and Ax, as the ou--of-distribution deviation, which is non-zero if and only if x & X Ax can be a means for quantifying the point anomaly, and Axξ can be means for quantifying the contextual anomaly. This is reasonable, since no matter how large Ax; is, we still have X E X", i.e., the in in-distribution deviated value xf is still in the set of all possible nominal time point data, and cannot be detected using a point- based model. On the other hand, it is possible that having learned X', point- based model can negate the deviated value caused by Ax.. Fig. [a) gives an illustration of the relationships between the variables at time i. We clarify this using the example below.
Assume we obtain dataset from the streaming data of 2D position sensor, where x;, x x R22 and we have learned that the nominal time series is the circular movement of a point around the origin with some angular velocity w and radius r, where Rmin I r S Rmaz. Accordingly, we can deduce that X = { vi² [R? I x² + y² S R } and x = [r cos wt r rin wt]" One possible cause (among many others) of contextual anomalies might be due to unexpected change in angular velocity. For instance, failures in the system might lead to a slowdown the circular movement between ( and 22, i.e., Axt = [r(cos w't - cos wt) r(sinw't sin w for / E (4,,..,,/2) and Ax; = 0 elsewhere. Moreover, noisy measurements of individual time points may induce point anomalies in the observed time series, i.e., Ax, = [ Wy.t] where Wa,t or Wy, is nonzero such that x & X for some t. Fig. [)()) gives an illustration of the above example. The black dots are time points where x x² (no anomalies). The blue dots are time points exhibiting slowdown (contextual anomalies). The red dots are time points with noisy measurements (point anomalies). The purple dots are time points with both slowdown and noisy measurements (point and contextual anomalies). The green dots are time points with noisy measurements but x E X", so are still contextual anomalies since their deviations cannot be detected by observing a single time point.
3.3 The Nominality Score
Now we conceptualize the Nominality Score N-)). Analogous to the anomaly score, N(.) indicates how normal a time point is. A nominality score N(-) is appropriate if for every possible 0N > (tN(t) > n |y, = 0) > (t) > 0x Wt = 1) for all t E {1, T), i.e., the portion of normal points that has nominality score larger than ON strictly larger than the portion of anomaly points
(a)
(b)
(c) Ar
Ax
T
T
3T
X
r
4
2
Ax
x
&pt.t
Rmin
100
R
0
X
X+
x
T
T
37
T
4
2
4
Figure 1: (a) Relationships between variables, (b) observed time series on 2D plane, and (c) radial and angular displacement time from nominal time series (x²) for the 2D position sensor example.
3

---

that has a nominality score larger than NN. There are many ways to define N(-). In this study, we define (t) as the ratio of the squared L2-norm between Axp and Ax!.
Axx Ax
| 1 | + Ax
x X
x
(())
(I)
We provide an example as to when N-) will be appropriate. In this derivation, we add an n and a in the subscript to denote variables that are only associated with normal and anomaly points, respectively, Consider toy dataset, where Axi,, Axt, Axi,a and Ax have been defined:
Axi.. ~ NO0 ID), AxT ~ ~NO TD), Axi ~ NO0, TD), Ax ~ N(O, d²Ip)
(2)
According to (.. we have
A 1
2N.(1) = A i.
F(D,))
Ax Ax
|
(1+ ((+a2)Na(1)> a1+
F(D,D)
A + Ax Ax
where F is the F-distribution with D and D degrees of freedom. Fig. 2 illustrates the probability density function of (-) and Na(-) for different D and a. If appropriate nominality score, since
then N(.) becomes an
((N(() > 0 Ut 0) f f(z; D, D)dx >
L (1402)0N
f(x; D, D)da (P((()) ON We 1)
228N
where [(;; D.D) D) the probability density function of the F-distribution with degrees of freedom D and D. Indeed, it is reasonable to assume that Ax, has larger variance than Ax," "
3.4 The Induced Anomaly Score
Having N()) defined, we now propose method for integrating any given NC) and anomaly score A(-) yield an induced anomaly score A(-), and show some instances where the performance will improve over using A()) or a smoothed A(-). Consider a dataset that contains subsequence anomalies. For two near time points and it is natural assume that the possibility of f being anomalous is affected by т. We quantify this effect as A(t; r), which the induced anomaly score t due to т. By summing over range of around I, we get the induced anomaly score
min(T,++d)
A(t)
A(tT))
(6)
where d is the induction length. Furthermore, we define A(t; r) as a gated value of A(T), which is controlled by the nominality score from to t:
max(t-
1)
A(())a (N (1)..++)))).
1>T
A(; & A(T)
II
90 (N(())
A(T)
t= (7)
k=min(++1.))
A(T)geN (N (N)...9x((((()) l <<T
1.0
No. M,(a=1)
(b)
N.. N,
N.(a-0.5)
N.la-0.5)
1N, D) (D=2)
N.(o=0.8)
(W, D. (D=100)
1.5
N.la=0.8)
N,la=121
N.la=1.)
N,(0=15)
N.la=1.5)
0.6
2.0
N,(a=2)
N,lo=2)
0.5
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
No N2
No Na
Figure 2: The probability density function for N₂, and N₂ of the toy dataset (a) D and (b) D = 100.

---

where the gate function 90 (N) is some transformation function of N conditioned on threshold N A reasonable assumption that gow (NV is a -on-enccaasing function of N, i.e. V N' implies Jo (N) I gow (N'). Indeed, if (k) is large, then time point k is likely normal point, and any two points 11, 12, where 1 < < 12, are unlikely be in the same anomaly subsequence, hence A(4;; (2) and A(t2;tg) should small. Explicitly, we can use A(t) = A(t: gox) and A(t; r) = A(t; T, 9ox) to denote that these values are conditioned on Iey. Overall, A(-) can be thought of as some (unnormalized) weighted smoothed value of A(:), where the weights are the product of gaw (N)) across some range. We consider the following two cases:
Claim Using soft gate function,
N
go (N) - max
(8)
ON
If there exists 0 such that ((t) 2 01 for all normal points (yi = 0), then (11(A(; go, go:): ):y) 2
F1" (A()):)), i.e., the best F1 score using the induced anomaly score with 9o, the gate function greater or equal to the best F1 score using the original anomaly score.
Proof
For any normal time point bn, we have
min(T, to +d)
mi, +d)
A(tn:g0) T_
Σ
Σ
= (9)
==max(1,fa -d-
The equality in the middle arises from the fact that go, (N(tn) = 0, according to and the assumptions. However, for any anomaly point ta have
min(T.to+d)
A(laige))
Σ
Ala5T,90.) A(aa)
(10)
This is because (ta.) might be lower than 01, and hence go, (Nt(. 2 0. By potentially having a higher A(t) than A(t for anomaly points, we get potentially higher F1
Such 0, indeed exists in real applications (e.g. the minimum nominality score among all normal points 4,) However, targeting this value barely leads to any improvement for F1* in practice. This because anomaly points generally fewer than normal points, resulting in barely any anomaly points to having N (ta) < 0,. Nevertheless, we have shown that the soft gate function along with the induced anomaly score can provably yield equal better F1* under some threshold.
Claim Using a hard gate function,
J0 (N)= A 1NC0N
(11)
If d = 1, and there exist two thresholds: (i) 02 such that N(L) < 02 for all anomaly time points (y, 1) (ii) 0xx = oo; then F1" (AA(( 9o,):y) F1FFF14400 (AA:(((:99))))) y),, i.c., the best F1 score using the induced anomaly score with go, as gate function greater or equal the best score using induced anomaly score with go., as the gate function.
Proof For any anomaly time point tor have min(T,t++1)
Á(La
Σ
A(7)
Alla 1)1 + A(ta) + A(ta + 1)1,,T
(12)
where the first equality arises from the fact that 90, (N((t) according to 11 and the assump- tions. For any normal time point t,. we have
min(T +1)
A(Inigo.)
Σ
A(tni S A(t. 1)1,,>1 + A(t,) + Altn 1)I,, (13)
7=
since ((,)) might be greater than 02 and hence 9o, (N().) < 1. However, we have
min(T.++1)
Á(t; go..)
Σ
A(7) = A(t 1)1 1 + A(t) + A(t 1)1,er
(14)
(=
5

---

regardless of normal or anomaly points since (t) < 8x for any t. Therefore, since A(ta: (e,) = A(ta; g0 and A(tni ge,) I A(tni 90 we get a potentially higher F1 when using 0g compared to using 20..
Á(; (o0) can viewed as the smoothed value (or shifted simple moving average) over A()) with a period of 2d + 1. This averaging method is common among other studies [1 12
Claim 2 implies that by conditioning on (-) and caleulating A(t), the performance can be improved over using a simple smoothing value of A(.). . In practice, we can relax the constraint of d, and use other gated functions to yield more flexible architecture. Note that the appropriateness of nominality score is critical for this to work.
32
33
3.5 Point-based Reconstruction Models
Consider some model Mp that reconstructs each time point t point- wise: Xpt.e _ Mpd (x²). Using Mpt, a method for yielding the anomaly score is by using the point- point-based reconstruction equan-s error, defined as Apr = {apt,1, Apt, where dpt, - p| x One concern for using this kind of anomaly score is that we are not taking into account any time-dependent relationships for deriving apt, so simply using Mpt and ape can barely be classified as a time series anomaly detection approach. Surprisingly, however, we find that simple realization of Mpt can already achieve impressive results for F1 (4.4))on4444)
Since Mpt learns capture the distribution of all normal point data, we assume that &pt, X or is very close. Moreover, since &pt. can offset point-anomalies, we assume x² x &pt,ts and implement a point-based reconstruction model to calculate xpt, in practice.
X
(XL
(&
xf &pt,t= Mpx(x))
(15)
Mpp can be any model that has the ability to reconstruct X point-wise. To our surprise, the best performance can be achieved by using a simple Performer-basee autoencoder ((34)) that actually has the potential to discover temporal information. Despite having such a possibility, it only learned to reconstruct point-by-point during training. We demonstrated this fact by shuffling the input time points and observing that the result will be the same after reordering the output sequence. One possible explanation is that during training, it is a lot easier to individually reconstruct single time points than to find complex time-dependent relationships; and since the Performer-based autoencoder tries to optimize over batch of time points, this reduces the effect of overfitting and allows the model to better generalize to unseen data. However, the exact reason for this remains an open question. For the rest of the study, we will use Mpt to refer to the Performer-baut autoencoder model. Details for the architecture of Mpp are shown in Appendix B.I
3.6 Sequence-based Reconstruction Models
Contrary to xf. x² should not only be in X² but also obey the time-dependent relationships. Therefore, it is necessary that the model (Maeg) for approximating x; takes a sequence of time points as input.
X X = (x) x
Msag (
(16)
How close (X)) approximates X* depends on the amount of training data and the model capacity. In practice, for computational reasons, only a section of x is input and reconstructed at a time. We found that given a subsequence X = {xx, ...... #01 x..
as input for reconstruction, a model tends to simply reconstruct individual points and do not take temporal information into account (as discussed in section 3.5). Therefore. we use a Performer based stacked encoder as Meeg. which predicts the middle 8 points from its surrounding 2> points to foree the learning of dimeddependent relationships.
We concatenate all the predicted time points output by Mang to construct X", For the rest of the study, we will use Maeg refer to the Performer-based stacked encoder model. Details for the architecture of Manq are shown in Appendix B.2 Fig. 3 Figives an illustration of the architecture for Mps. Maeg. and the overall scheme. By obtaining X and X" we can calculate N--) and select some A(-) for calculating Á(.). The algorithm for evaluating trained Myt and M seg using the soft gate function shown in Algorithm 1.

---

(b)
(c)
Mpt
8
true labels y
predicted tabels 9:
LA(e)aRa
Performer
Lin
6x?
L2
ALC
Ate)
Perf
Linear
nominality (C))
Induce Bou
Linear
Linear
X
d
dx?
Performer
Performer
L.2
5200 2920
05
Mtt
Mseq
M seq
Pa
L2
||ax!"||
Senominatoo
Figure 3: (a) Performer-based autoencoder Mps. (b) Performer-based stacked encoder Mang. and (c) main scheme for NPSR. GELUs are used the activation function for each layer.
Algorithm 1 NPSR F1* Evaluation (soft gate function)
function NPSR Mpd. Mang, X [xf. x = 1r.
Construct X {89. ... X4) with &3 t (xp)
Construct X = [xi. x7] -- Maag (X9)
Construct A(-) with A(t) || -x211
section 3.5
Construct N-) with (() ||g -
- 11
Construct ge x (N()) with gew (N(()) max(0, N(t/ONN
Construct A(s with A(4:T -A(()) -
(
gon (N()))
miin(Te++d) Construct A(-) with A(t) + 2-=mux(1,_a)) A(t;T)
return F1 maxe, (1(5((()))... 0.))))
Experiments
4.1 Datasets
We evaluate NPSR on the following datasets:
SWaT (Secure Water Treatment) [35: The SWaT dataset is collected over 11 days from scaled- down water treatment testbed with 51 sensors. During the last days, 41 anomalies were injected using diverse attack methods, while only normal data were generated during the first days.
WADI (WAter DIstribution testbed) [36]: The WADI dataset is acquired from reduced city water distribution system with 123 sensors and actuators operating for 16 days. The first 14 days contain only normal data, while the remaining two days have 15 anomaly segments.
PSM (Pooled Server Metrics) 37 The PSM dataset is collected internally from multiple applica- tion server nodes at eBay. There are 13 weeks of training data and 8 weeks of testing data.
The MSL and SMAP datasets are public datasets collected by NASA, containing telemetry anomaly data derived from the Incident Surprise Anomaly (ISA) reports of spacecraft monitoring systems. The datasets have 55 and 25 dimensions respectively. The training set contains unlabeled anomalies.
MSL (Mars Science Laboratory) and SMAP (Soil Moisture Active Passive) B8 13
SMD (Server Machine Dataset) 2 The SMD is collected from a large internet company, compris- ing 5 weeks of data from 28 server machines with 38 sensors. The first 5 days contain only normal data, and anomalies are injected intermittently for the last 5 days.
trimSyn (Trimmed Synthetic Dataset) (24]: The original synthetic dataset was generated using trigonometric functions and Gaussian noises. We obtained the dataset from [39] and trimmed the test dataset such that only one segment of anomaly present.
The statistics for the datasets are summarized in Table For multi-entity datasets, corresponds to the number of train/test time points summed over entities, and the anomaly rate calculated from the ratio between the sum of all anomaly points and sum of all test points.
7

---

Table 1: Datasets used in this study before preprocess.
Dataset
Entities
Dims
Train #
Test #
Anomaly Rate (%)
SWaT
495000
449919
12.14
WADI
123
1209601
172801
5.71
PSM
25
132481
87841
27.76
MSL
27
55
58317
73729
10.48
SMAP
140825
444035
12.83
SMD
28
38
708405
708420
4.16
trimSyn
35
10000
7680
2.34
4.2 Baselines
We evaluate the performance of NPSR against several deep learning algorithms and simple heuristics using F1*. Due the exhaustive nature of optimizing for all datasets and algorithms, we follow three-step approach to populate Table 2 Firstly, we reference values from the original paper, and if unavailable, we search for the highest reported values among other publications. Finally, if no reported values are found, we modify and run publicly available code. We cannot find any reported F1* of the PSM dataset using THOC or any publicly available code, hence leaving the value blank, For multi-entity datasets (MSL, SMAP, and SMD), we compare the performance using two methods (I) combining all entities and training them together; (2) training each entity separately and averaging the results. Moreover, some literature attempt to find threshold (0,) and then calculate the performance conditioned on it [24] 40). Since 0a can simply be one-value parameter, we assume that other studies have already optimized this value, and regard their reported F1 as F1*. More information on the sources of data can be found in Appendix
4.3 Main Results
In Table 2] report the results for F1" several datasets. Detailed preprocessing steps and training settings are reported Appendix NPSR almost consistently outperforms other algorithms, only being slightly inferior to TranAD the PSM dataset. NPSR makes use of Mpt precisely capture point anomalies with low false-positive rates (given the best threshold). Moreover, it acquires the ability to detect contextual anomalies by incorporating Maeq through the calculation of A(-), without compromising its capability of detecting point anomalies. We observe that recent studies do not necessarily have higher F1* scores than older ones. Interestingly, simple heuristics can even perform fairly well F1*. with NPSR being the only algorithm consistently outperforming them.
C
Table 2: Best F1 score (F1*) results on several datasets, with bold text denoting the highest and underlined denoting the second highest value. The deep learning methods are sorted with older methods at the top and newer ones at the bottom.
Algorithm Dataset
SWaT
WADI
PSM
MSL
SMAP
SMD
trimSyn
Simple Heuristic
30
31
0.789
0.353
0.509
0.239
0.229
0.494
0.093
DAGMM
26
0.750
0.121
0.483
0.199
0.333
0,238
0.326
LSTM- VAE
0.776
0.227
0.455
0.212
0.235
0.435
0.061
MSCRED 24
0.757
0.046
0,556
0.250
0.170
0,382
0,340
OmniAnomaly
9
0.782
0.223
0.452
0.207
0.227
0.474
0.314
MAD-GAN
23
0.770
0.370
0.471
0.267
0.175
0.220
0.331
MTAD-GAT
27
0.784
0.437
0.571
0.275
0.296
0.400
0.372
USAD 23
0.792
0.233
0.479
0.211
0.228
0.426
0.326
THOC
0.612
0.130
0.190
0.240
0.168
UAE
0.453
0.354
0,427
0.451
0.390
0.435
0.094
GDN
0.810
0.570
0,552
0.217
0.252
0,529
0.284
GTA
41
0.761
0.531
0.542
0.218
0.231
0.351
0.256
Anomaly Transformer
40
0.220
0.108
0.434
0.191
0.227
0.080
0.049
TranAD
25
0.669
0.415
0.649
0.251
0.247
0.310
0.282
NPSR (combined)
0.261
0.511
0.227
NPSR
0.839
0.642
0.648
0.551
0.505
0.535
0.481

---

We suggest using these simple heuristic values as strong baseline for future studies to compare against. In light of recent publications highlighting the limitations of using the point-adjusted Fl score (1I] 30 31 42) and yet the large amount of work still using it, we also report the results using point-adjustment in Appendix D
For multi-entity datasets, we observe that the standard method (training one point-based and sequence- based model per entity) outperforms the combined method for MSL and SMD datasets. This is not surprising, given that entity-to-entity variations might be large. However, for the SMAP dataset, we observe that the combined method performs better. We attribute such results to the fact that the SMAP spacecraft are routine, hence the resulting telemetry between entities can have similar underlying distributions. This contributes to additive learning from the increased training data [13]
4.4 Ablation Study
The ablation study conducted this section sheds light on several aspects of the proposed method. In Table we compare the performance of five methods that yield different anomaly scores (either A(-) or A(-)). For the first two methods, the reconstruction errors of Mpt and Meeg are used, respectively. Since Mnr mostly performs better than Mac.. we use the point-based reconstruction error as A(-) to calculate A(-) for the last three methods. The third method corresponds to an unnormalized simple smoothed value of A(-) (cf. section 3.4). The fourth and fifth methods use gate functions (II) and (8) respectively, but the same ON that corresponds to the 98.5 percentile of the nominality score from the training data (Nern). This method for setting 0x works well enough and can be applied across different datasets. Illustrations of the distribution of nominality scores for the SWaT and WADI datasets, along with the 98.5% threshold value are shown in Fig. 4 We can observe that the distributions are close appropriate (ef. section 3.3). The results for the last three methods are averaged over induction lengths d = 1, 4, 8, 16, 32, 64, 128, and 256. Each entity is trained separately for multi-entity datasets and the results are pooled together.
Firstly, we observe that Mrr outperforms May and achieves competitive results on its own, despite not modeling the time-dependent relationships. Secondly, by smoothing A(-) (third row), the AUC and F1" are increased for most datasets. This means smoothing is generally an effective method for improving performance. Moreover, experiments show that the soft gate function along with an appropriate d performed the best on average in terms of F This suggests that the distribution of nominality scores is predominantly overlapped, and soft gate function will be more appropriate to prevent excessive accumulation of anomaly scores on normal time points, reducing false- sille--ositives (cf. Appendix F.. This method will also have generally stable AUC and F1 (low o.)) across wide range of d. This makes sense when time point 7 is farther away from time point i, more gate function outputs are multiplied onto A(T), hence A(l, - O. However, the results also suggest that the best choice of gate function and f may depend on the specific dataset at hand.
4.5 Detection Trade-off Between Point and Contextual Anomalies
We elaborate on the trade-off between detecting point and contextual anomalies and relate them with the performance of Mst and Mseg. may seem intuitive that finding temporal information in
Table 3: AUC and F1* for different methods and datasets, with bold text denoting the highest and underlined text denoting the second highest value. The mean (1a)) and standard deviation (a.)) of the performance metrics evaluated across 1, 1,2, 4, 8, 16, 32, 64, 128, 256 are shown.
Dataset
SWaT
WADI
PSM
MSL.
SMAP
SMD
trimSyn
Method
AUC
F1'
AUC
F1
AUC
AUC
F1'
AUC
F1
AUC
F1'
AUC
F1
MMr (||) x
0.908
0.839
0.819
0.629
0.790
0.626
0.640
0.366
0.647
0.329
0.820
0.485
0.721
0.100
Non <|
x
0.899
0.755
0.843
0.559
0.766
0.576
0.621
0.351
0.611
0.292
0.820
0.482
0.832
0.345
M₁r Hard
11
0.912
0.813
0.827
0.630
0.775
0.621
0.708
0.451
0.665
0.389
0.835
0.492
0.785
0,144
(0tr 20)
0.005
0.034
0.007
0.037
0.023
0.020
0.032
0.038
0.010
0.036
0.025
0.052
0.037
0.021
My, Hard
11
Mic
0.912
0.820
0.844
0.625
0.779
0.624
0.718
0.467
0.659
0.386
0.833
0.495
0.791
0.292
(0x 98.5% Nem.
0.005
0.024
0.007
0.023
0.017
0.015
0.041
0.051
0.012
0.034
0.024
0.050
0.069
0,121
M₂p Soft 8
Mic
0.909
0.832
0.856
0.639
0.804
0.636
0.698
0.465
0.656
0.388
0.840
0.525
0.862
0.434
(4x 08.5%
0.000
0.001
0.011
0.008
0.005
0.004
0.031
0.061
0.005
0.039
0.003
0.011
0.063
0.099

---

SWaT
WADI
normal anomelyy Non
normal
Requency
10a
Eequency
10"
anomely
98. Non
10a
10'
0
1
0
30
15
20
25
30
Figure 4: Histograms of the nominality scores for the SWaT and WADI dataset.
time series would lead to better performance. However, the modeling complexity increases with the number of time points being considered. This leads to the difficulty of focusing on the reconstruction of single time points. Fig. 5(a) shows A(:) calculated using either Mpt or Maeg. and A(-) calculated by NPSR using the WADI dataset. Given the fact that point is predicted anomalous if A(t) 2 Oa. we see that Maey has higher false positive rate due to the spikes. In comparison, Mp more accurately detects anomalies in a point-wise fashion. This highlights the superiority of Mn over Maeg for this dataset. Furthermore, the induced anomaly score calculated by NPSR has very low false positive rates, and can sometimes even learn to identify anomalies that are not detected by Mpt Maeg (Fig. ()), the anomaly subsequence at t 15200 and the subsequence to its left, pointed by the black arrows). This suggests that A() superior to the original A(-) for this dataset. However, Mpt might not always perform superior to M,sq.. False negatives can be visualized between t 14800 and
14900, where Mpt struggles to recognize the anomaly but Masy effectively detects anomalous time-dependent relationships. This suggests that the anomaly segment contains relatively more contextual than point anomalies. Since the reconstruction error of Mne is used as A()). we lose the advantage of effectively utilizing the reconstruction error of Maeq This results in
A-) not high enough to reach 0g within this segment. An important future direction would be to explore how to appropriately select A(.) among multiple models (cf. Appendix F.
(8)
bast--aseddddee model
Mpp
(b)
Mpt
point based
true labels
true labels
seq based model best threshold
Mseq seq
Mseq
seq based model best threshold true labels
true labels
Whh
NPSR true lebrels
Mpt + Mseq
Mpt + Msed
NPSR true lubels
Arc
2000
4000
5000
8000
10000
12000
14000
16000
14600
14800
15000
15200
15400
15600
Figure 5: (a) Anomaly scores using Mpt. Mseg and NPSR (soft gate function, 0yy = 99.85%%.r... and d 16), and the true labels of the WADI dataset. (b) Magnification for t ( {14500, -... 15600).
Conclusion
In conclusion, we introduce an improved framework for unsupervised time series anomaly detection. We specify the relationships between point and contextual anomalies and derive the nominality score and induced anomaly score to provide theory- algorithm with provable superiority. NPSR captures both point and contextual anomalies, resulting in high combined precision and recall. Our results show that NPSR exhibits high performance, is widely applicable, and has a relatively straightforward training process. It has the potential to decrease labor needs for fault monitoring and correspondingly accelerates decision making and can also contribute to Al sustainability by preventing energy waste or system failure.
10

---

6 Acknowledgement
We thank Piyush Desai, Raphael Schutz, and Nikhil Deshmukh from Turntide Technologies for their helpful discussions and insights.
References
[1] Ane Blázquez-Garvia, Angel Conde, Usue Mori, and Josc A Lozano. A review outlicr/anomaly detection in time series data, ACM Computing Surveys (CSUR), 543) 2021.
[2] Stéphane Crépey. Noureddine Lehdili, Nisrine Madhar, and Maud Thomas. Anomaly detection in financial time series by principal component analysis and neural networks. Algorithms, 15(10)3385, 2022.
[3] João Pereira and Margarida Silveira. Learning representations from healthcare time series data for unsupervised anomaly detection. 20191EER International Conference on Big Data and Smart Computing (BigComp), pages 1 1-7, 2019.
[4] Tiankai Chen al. Anomaly detection in semiconductor manufacturing through time series forecasting using neural networks. PhD thesis, Massachusetts Institute of Technology. 2018
[5] Mohammad Bawaneh and Vilmos Simon. Anomaly detection in smart city traffic based on time series analysis. In 2019 International Conference on Software, Telecommanicationns and Computer Networks (SofiCOM), pages 1-6, 2019.
[6] Kukjin Choi, Jihun Yi, Changhwa Park, and Sungroh Yoon. Deep learning for anomaly detection in time-series data: Review, analysis, and guidelines. IEEE Access, 9:120033-20066 2021.
[7] Sebastian Schmidl, Phillip Wenig, and Thorsten Papenbrock. Anomaly detection in time series: A comprehensive evaluation. Proc. VLDB Endow., 15(9):1779-117 1797, may 2022.
[8] Yan Zhao, Liwei Deng, Xuanhao Chen, Chenjuan Guo, Bin Yang. Tung Kieu, Feiteng Huang, Torben Bach Pedersen, Kai Zheng, and Christian S Jensen. A comparative study unsupervised anomaly detection for time series: Experiments and analysis. arXiv preprint arXiv:2209, 04635, 2022.
[9] Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of 25th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining, KDD '19, page 2828-2837, New York, NY, USA, 2019. Association for Computing Machinery.
[10] Zhihan Li, Youjian Zhao, Jiaqi Han, Ya Su, Rui Jiao, Xidao Wen, and Dan Pei. Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. In Proceedings the 27th ACM SIGKDD Conference on Knowledge Discovery; Data Mining, KDD 221. page 3220-3230, New York, NY, USA. 2021. Association Computing Machinery.
[11] Astha Garg, Wenyu Zhang, Jules Samaran, Ramasamy Savitha, and Chuan-Sheng Foo. An evaluation of anomaly detection and diagnosis multivariate time series. IEEE Transactions on Neural Networks and Learning Systems, 33(6):2508 2517, jun 2022.
[12] Ailin Deng and Bryan Hooi. Graph neural network- based anomaly detection in multivariate time series. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):4027-4035 May 2021.
[13] Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using Istms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining, KDD '18, page 387-395, New York, NY, USA, 2018. Association for Computing Machinery.
[14] Nan Ding, HaoXuan Ma, Huanbo Gao, YanHua Ma, and GuoZhen Tan. Real-time anomaly detection based on long short-term memory and gaussian mixture model. Computers & Electrical Engineering, 19:106458, 2019.
[15] Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng. and Jörg Sander. Lof: Identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, SIGMOD '00, page 93-104, New York, NY, USA, 2000. Association for Computing Machinery.
[16] Bernhard Schölkopf, John C. Platt, John C. Shawe Shawe-Taylor Alex Smola, and Robert C. Williamson. Estimating the support of nigh-dimensional distribution. Neural Computation, 13(7):1443-1445 jul 2001.
11

---

[17] Bo Liu, Yanshan Xiao, Longbing Cao. Zhifeng Hao, and Feiqi Deng. Svdd-based outlier detection on uncertain data. Knowledge and information systems, 34:597-618, 2013.
[18] Lifeng Shen, Zhuocong Li, and James Kwok. Timeseries anomaly detection using temporal hierarchical one-class network. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Newral Information Processing Systems, volume 33, pages 13016-13026. Curran Associates, Inc., 2020.
[19] Haibin Cheng, Pang-Ning Tan, Christopher Potter, and Steven Klooster. A robust graph- baapd algorithu for detection and characterization of anomalics in noisy multivariate time series. In 2008 IEEE International Conference on Data Mining Workshops, pages 349-358, 2008.
[20] Fan-Keng Sun, Chris Lang, and Duane Boning, Adjusting for autocorrelated errors in neural networks for time series. Advances in Neural Information Processing Systems, 34:29806-29819, 2021.
[21] Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. Multivariate probabilistic time series forecasting via conditioned normalizing flows. arXiv preprint arXiv:2002.66100 2020.
[22] Dachyung Park. Yuuna Hoshi. and Charles C Kemp. A multimodal anomaly detector for robot-assisted feeding using lstm-based variational autoencoder. IEEE Robotics and Automation Letters, 3(3):1544- 1551, 2018.
[23] Dan Li, Dacheng Chen, Baihong Jin, Lei Shi, Jonathan Goh, and See-Kiong Ng. Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks. In Artificial Neural Networks and Machine Learning-ICANN 2019: Text and Time Series: 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17-19, 2019, Proceedings, Part IV, pages 703-716. Springer, 2019.
[24] Chuxu Zhang, Dongjin Song, Yuncong Chen, Xinyang Feng, Cristian Lumezanu, Wei Cheng, Jingchao Ni. Bo Zong, Haifeng Chen, and Nitesh V Chawla. A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. In Proceedings the AAAI conference artificial intelligence, volume 33, pages 1409-1416, 2019.
[25] Shreshth Tuli, Giuliano Casale, and Nicholas R Jennings. Tranad: Deep transformer networks for anomaly detection in multivariate time series data, arXiv preprint viXiv:220100228. 2022.
[26] Bo Zong. Qi Song. Martin Renqiang Min, Wei Cheng. Cristian Lumezanu, Dacki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations, 2018.
[27] Hang Zhao, Yujing Wang, Juanyong Duan, Congrui Huang, Defu Cao, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. Multivariate time-series anomaly detection via graph attention network. In 2020 IEEE International Conference on Data Mining (ICDM), pages 841-850. IEEE, 2020.
[28] Julien Audibert, Pietro Michiardi, Frédéric Guyard, Sébastien Marti, and Maria A. Zuluaga. Usad: Unsupervised anomaly detection on multivariate time series. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining, KDD '20, page 3395-3404 New York. NY, USA, 2020. Association for Computing Machinery.
[29] Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, Jie Chen, Zhaogang Wang, and Honglin Qiao. Unsupervised anomaly detection via variational auto-encoder for scasonal kpis in web applications. In Proceedings of the 2018 World Wide Web Conference, WWW '18 page 187-196, Republic and Canton of Geneva, CHE 2018. International World Wide Web Conferences Steering Committee.
[30] Siwon Kim, Kukjin Choi, Hyun-Soo Choi, Byunghan Lee, and Sungroh Yoon. Towards a rigorous evaluation of time-series anomaly detection, 2022.
[31] Keval Doshi, Shatha Abudalou, and Yasin Yilmaz. Reward once, penalize once: Rectifying time series anomaly detection. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2022.
[32] Siwei Guan, Binjie Zhao, Zhekang Dong, Mingyu Gao, and Zhiwei He. Gtad: Graph and temporal neural network for multivariate time series anomaly detection. Entropy, 24(6), 2022.
[33] Enyan Dai and Jie Chen. ccaph-augmented normalizing flows for anomaly detection of multiple time series. arxiv preprint 27x 2022.
12

---

[34] Krzysztof Choromanski. Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos. Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.144 14794, 2020.
[35] Jonathan Goh, Sridhar Adepu, Khurum Nazit Junejo, and Aditya Mathur. A dataset support research in design of secure water treatment systems. Grigore Havarneanu, Roberto Setola, Hypatia Nassopoulos. and Stephen Wolthusen, editors, Critical Information Infrastructures Security, pages 88 99, Cham, 2017. Springer International Publishing.
[36] Chuadhry Mujeeb Ahmed, Venkata Reddy Palleti, and Aditya P. Mathur. Wadi: water distribution testbed for research in the design of secure cyber physical systems. In Proceedings of the 3rd International Workshop Cyber- Sayberl Systems for Smart Water Networks, CySWATER '17, page 25-28, 22, New York.
NY, USA, 2017. Association for Computing Machinery.
[37] Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate series anomaly detection and localization. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery; Data Mining, KDD 21, page 2485-2494, New York, NY, USA, 2021. Association for Computing Machinery.
[38] Dara Entckhabi, Eni G. Njoku, Peggy E. O'Neill, Kent H. Kellogg, Wade T. Crow, Wendy N. Edelstein, Jared K. Entin, Shawn D. Goodman, Thomas J. Jackson, Joel Johnson, John Kimball, Jeffrey R. Piepmeier, Randal D. Koster, Neil Martin, Kyle C. McDonald, Mahta Moghaddam, Susan Moran, Rolf Reichle, J. C. Shi, Michael W. Spencer, Samuel W. Thurman, Leung Tsang, and Jakob Van Zyl. The soil moisture active passive (smap) mission. Proceedings of the IEEE, 98(5)704-716, 2010.
[39] Sadaf Tafazoli and Eamonn Keogh. Matrix Profile XXVIII: Discovering Multi--imeessionl Time Series Anomalies with <italic> K</italic> of Nitalic><<ttali>>> Anomaly Detection<<upp>></sppp. pages 685-
693.
[40] Jiehui Xu, Haixu Wu, Jianmin Wang. and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. arXiv preprint arXiv:2110.02642 2021.
[41] Zekai Chen, Dingshuo Chen, Xiao Zhang, Zixuan Yuan, Xiuzhen Cheng. Learning graph structures with transformer for multivariate time-series anomaly detection in iot. IEEE Internet of Things Journal, (12))9179-9188) 2021.
[42] Dennis Wagner. Tobias Michels, Florian C.F. Schulz, Maja Rudolph, and Marius Kloft. TimeseAD: Benchmarking deep time-series anomaly detection, 2023.
[43] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.044 0,40066 2020.
[44] Dan Hendryycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.000 2016.
[45] TimyadNyda Github dimyadnyda/variationa. -atell--s----uuta autoencoder: Lstm variational auto-encoder for time series anomaly detection and features extraction.
[46] Jun Zhan, Siqi Wang, Xiandong Ma, Chengkun Wu, Canqun Yang, Detian Zeng, and Shilin Wang. Stgat- mad: Spatial-temporal graph attention network for multivariate time series anomaly detection. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3568-3572, 3572, 2022.
[47] Markus Thill, Wolfgang Konen, and Thomas Bäck. Markus/hiil/MGAB, The Mackey-Glan Anomaly Benchmark, April 2020,
13

---

A The Best F1 Score
This section describes the method for calculating the best Fl score (F11) from a set of anomaly scores {ai,..., ar) and a set of labels y = (V1,..., Vr Firstly, given a and some arbitrary threshold On we can calculate { where f). _ 20. Secondly, is used to calculate TP, FP, and FN, which corresponds the sets of time points for true positives, false positives, and false negatives.
TP -
{ 1, - 1}, FP _ {4] 1, 0}. FN (1 0, yt =
(17)
Thirdly, calculate the precision (P) and recall (R), and then calculate the FI score, which is the harmonic mean between R and P.
2PR P+R
(TP)) (TTP) + (FP) R 2
n(TP) n(TP) + (FN)
F1
PE
(18)
Finally, we calculate F1 by using the threshold that yields the highest FI.
F1 (1; max F1 (f 0a);y)
(19)
B Model Architecture
Performers (an improved variant of Transformers) are competitive in terms of execution speed compared with other Transformer variants [4 43), hence we use them the basic building block of our models. For both Mpr and Maeg. we use linear layer with input and output dimensions equal to as the token embedding layer, fixed positional embedding layer at the beginning, feature redraw interval of 1, and tanh activation function immediately before the output. GELUs [44 are used as the activation layer for all linear layers. We do not change any other predefined activation layer inside Performers.
B.1 Performer-based autoencoder
For the Performer-basee autoencoder Mye. the input with the shape (batch, W, D) is passed through a plain Performer with N 5fper layers after the positional embedding step, where W represents the input window size. This is followed by linear encoding layer that transforms the dimensionality from D to Diat. resulting in the shape (batch, W, Diat) for the latent variables. In the case of Mpt. there is no compression along the time domain. The latent variables then pass through another linear decoding layer that transforms the dimensionality from Diat back to D, followed by another Performer with Sper layers.
B.2 Performer-based stacked encoder
For the Performer-baid basedd stacked encoder Mssg the input with shape (batch, Wo. D) is passed through a plain Performer with one layer after the positional embedding step. followed by linear encoding layer that transforms the window size from Wy to W1. where Wo = 27 (cf. section 3.6). done along the time domain for Maeg. The
should be noted that unlike Mpt, compression one-layer Performer and linear layer are stacked Newe times, where in the i-th stack, the window size compressed from W, to W. WNene is equal to target output window size 8. For both Mnt and Mang. we optimize Np Diat, W, Nene: W, for i E (O.... Nene), and ó for the best performance. Note that Maeg isn't capable of reconstructing the first and last y time points due to its architecture, hence we discard the first and last points reconstructed by Mpt that rest of the time points have exactly two reconstructed values corresponding to using Meeg and Mpt. respectively.
14

---

C Data Preprocessing and Training Details
Table shows the hyperparameters used for implementing NPSR on the experimented datasets. To ensure fair comparison, the same preprocessing method is applied to all algorithms for the same dataset. The search for hyperparameters is done manually, starting from some reasonable value (e.g. learning rate of 10-4) The authors believe that there is still room for improvement by fine-tuning these hyperparameters. To speed up training, we load all training inputs and outputs, and testing inputs onto the GPU before training. We use a local GPU, which can be either GeForce RTX 3070 (8GB), 3080 (12GB) or 3090 (24GB). For an individual experiment using a single dataset and training method, the training time ranges from approximately 2 minutes to 12 hours. For single-entity datasets and multi-entity datasets that use the combined training method, we run the experiments for at least times and confirm that the results are stable given different random seeds. For multi-entity datasets with entities trained individually, the results are averaged across all entities. Generally, datasets with single entities train faster than those with multiple entities. There are some additional remarks regarding the preprocessing of the datasets.
shows
For SWaT, we use SWaT_ _aaritDDtttt____ee Borttt_Attck_ktt v0. cav and SWaT _atts_tateet_tta Loarsstt_oomaa____ v1 csv from the folder SWaT. A1 A2_Dec 2015 (manually converted from * xls).. We corrected some original flaws in the dataset (e.g. redundant blank spaces in some labels), and set the 5th and 10th columns to all O. For WADI, we use the 2017 year dataset. Columns with excessive NaNs (more than half of the entire length) are deleted. Other NaNs are forward-filled After deleting all the columns with excessive NaNs, the 86th column further set to all For PSM, we forward -ill all NaNs. For MSL and SMD, two additional blank channels are added to make the number of channels divisible by the number of heads. For trimSyn, we separated the dataset into training (t E (O.... 9999}) ) and testing E {10000, .... 19999)) data (ef. (24)). For the testing data, we extracted segments within the time intervals E { {10000, 11719] (11900, 12849) (14630, 17699) } U [17880, 18529] U {18710, .... 19999) and concatenated them form single-entity test dataset. We also inserted additional segment IDs using one-hot encoding to enable segment identification. This process resulted in test dataset with a single anomaly segment anomaly rate of 2.34%. In the case of the training data, we duplicated the original segment five times, concatenated them, and added the necessary segment IDs.
(12670, .... 12849)) and a corresponding
Table 4: Implementation details. (c) stands for the combined training method (cf. section 4.2
Parameter
Dataset
SWal
WADI
MSL
MSL
SMAP
SMAP (c)
SMD
SMD
trimSyn
Preprozess
Downsample
10
2
Nornalization
Minmax
Minmax
Minmax
Minmax
Minmax
Minmax
Minmax
Minmax
Minmax
Minmax
Stride
10
10
10
W (for M,r))
100
100
100
100
100
50
50
50
50
50
Wo (for M.ney)
100
100
100
50
50
50
20
20
20
6
6
Model architecture
of heads
8
Dus
10
10
10
10
10
10
10
10
10
mult
Nger
4
4
4
4
4
Nene
Induced anomaly
Gate function
solt
sult
soft
soft
suft
soll
soft
soll
soll
soll
d
16
16
64
128
32
64
32
16
256
256
Ratio of New for 0x
90.85%
99. 85%
99.85%
99.85%
97.5%
99.85%
99.85%
99.85%
99.85%
99.85%
Training
Learn rate
10-4
10-4
10-1
10-1
10-1
10-1
10-1
10-1
10-1
10-1
Optimizer
Adam
Adam
Adam
Adam
Adam
Adam
Adam
Adam
Adam
Adam
Batch size
64
64
64
64
64
64
64
64
64
64
Training epochs
100
100
100
100
100
100
100
100
100
15

---

D The Point-adjusted Best F1 Score
Analogous to F1", the point-adjusted best Fl score (F1pA) corresponds F1 calculated after point- adjustment. Table 5able5]sshn shows the F1" and FI of different algorithms, including NPSR, applied to several datasets. We did not show the results for the trimSyn dataset with point-adjustment However, by applying NPSR and the same setting as without using point-adjust, we can achieve FlPA = 1 within the first few epochs. This indicates the point with the highest anomaly score lies within the single anomaly segment.
The results suggest that FI may not be reliable the SWaT, WADI, PSM, and MSI. datasets, simple heuristic approaches (e.g. using the mean squared value of an input time point as the anomaly score) outperform all deep learning methods when evaluated using FIPA- Moreover, optimizing on FL does not necessarily guarantee a higher FI*. NPSR is optimized on F1 by tuning the algorithm- specinii parameters (e.g. d) and general parameters. To additionally optimize on FI we simply added spikes to the induced anomaly score (Aepske()) (-)) with value oo for some fixed interval s. The results show that Å Aspike() (.) can also achieve competitive FIPA PA values.
Table 5: Point-adjusted best FI score (F1pA) and best Fl score (F1*) results on several datasets, with bold text denoting the highest and underlined text denoting the second highest value. The deep learning methods are sorted with older methods at the top and newer ones at the bottom.
Dataset
SWaT
WADI
PSM
MSL.
SMAP
SMD
Metric
F1pr
F1
F1BA
F1
FI
F1
FI
F1*
F1
F1
F1øx
F11
Simple Heuristics
0.969
0.789
0.965
0.353
0.985
0.509
0.965
0.239
0.961
0.229
0.934
0.494
11:0 31
DAGMM 26
0.853
0.750
0.209
0.121
0.761
0.483
0.701
0.199
0.712
0.333
0.723
0.238
LSTM-VAE
0.776
0.227
MSCRED 24
0.807
0.757
0.374
0.046
0.626
0.556
0.936
0.250
0.866
0.170
0.841
0,382
Omominnomaly
0.782
MAD-GAN 23
0,815
0.770
0.556
0.370
0.658
0.471
0.917
0.267
0.865
0.175
0,915
0.220
MTAD-GAT 2
0.860
0.784
0.602
0.437
0.780
0.571
0.908
0.901
0.296
0.908
0.400
USAD 28
0,846
0.792
0.430
0.233
0.725
0.479
0.911
0.211
0.819
0.228
0,946
0.426
THOC 18
0.881
0.612
0.506
0.130
0.895
0.937
0.190
0.952
0.240
0.541
0.168
UAE 11
0.869
0.453
0.957
0.354
0.936
0.427
0.920
0.451
0.896
0.390
0.972
0.435
GDN 12
0.935
0.810
0.855
0,570
0.923
0.552
0.903
0.217
0.708
0.252
0.716
0.529
GTA
41
0.910
0.761
0.84
0.531
0.855
0.542
0.911
0.218
0.904
0.231
0.919
0.351
Anomaly Trunsformer
0.941
0.019
0.714
0.015
0.979
0.022
0.936
0.021
0.967
0.019
0.923
0.021
A0
TranAD 25
0.815
0.669
0.495
0.415
0.882
0.649
0.949
0.251
0.892
0.247
0.961
0.310
NPSR (combined)
0.960
0.261
0.978
0.511
0.850
0.252
NPSR
0.953
0.839
0.938
0.642
0.957
0.648
0.551
0.437
0.535
16

---

E
Source of Data
Table shows the data sources used to produce Table 5 as well as the sources for section 4.3 The reference number is followed by number between I and 3, where I indicates that the data comes from the original work, 2 indicates that the data comes from reproduced values of another literature, and 3 indicates that we have reproduced the values using public repositories.
Table 6: Data sources for algorithms and datasets. *#eeroduced by using the squared value of channel I as A()) iReproduced by using the squared value of channel 9 A(-).
Dorsset
SWiT
WADI
MSI.
SMAP
SMD
trimSyn
Metric
Fij.
Figh
Fljs
Fij.
Fils
Fij.
F1"
Simple Heuristic
30
1
-
30
31
-1
80
31
-1
30
31 -1
30
DAGMM
30
-2
23
30
2
30
-2
23
-3
25
-3
30
-2
100
-2
100
-2
30
-2
K0
-2
30
-2
P5
-3
LSTM-VAB
28
-2
-2
78
78
-2
-0
85
-3
28
-2
50
13
-2
30
18
-2
30
-2
45
-
MSCRED
25
25
25
25
25
25
25
OmmiAnonaly
MAD-GAN
25
3
25
1
25
25
-2
25
-2
25
-3
25
-2
15
-3
25
GAT
30
2
16
27
221
3
221
23
1
50
-
22
3
25
-3
USAD
25
28
25
23
30
30
23
30
P5
THOC
UAE
11
11
GDN
50
2
30
[2]
12
12
2D
-
80
-2
100
-2
R0
2
25
GTA
41
1
41
91
41
1
-3
51
-3
41
Anomaly Fransr
40
40
10
40
33
40
60
400
TrnnD)
XX
F Model and Parameter Selection Heuristics
We believe that the most important idea in model selection is to achieve a balance between effectively utilizing both point and sequence- based models and carefully selecting the appropriate parameters. As an extreme example, we optimize our algorithm on the Mackey-Glass anomaly benchmark (MGAB) [7]. which is a univariate time series dataset, and find that sequence-based models can significantly outperform point- based models when using their reconstruction errors as A(.). As in Fig. we see that Mseg can correctly identify the anomalies, whereas Mee did not learn at all. This is reasonable, as point-based models only consider a single time point. In general, the lower the dimensionality of the dataset, the harder it gets for point-based model to learn statistically meaningful representations. Moreover, looking at the high-dimensional datasets reported Table 3 the induced anomaly score remains important for some datasets: For the MSL dataset, F1 improves 0.1 compared to only using point-based reconstruction.
Fig. 7b when using a soft gate function, F1* improves 0.047 compared to using the point- point-based anomaly score. Since the amount of improvement depends on the statistical structure of the test data, it is still useful to consider using both Maeg and Mpr in general.
The difficulty in choosing the parameters for unsupervised time series anomaly detection stems from the absence of anomalies in the training dataset. The selection of soft or hard gate functions, the value for d, and the ratio for 0x largely depend on how we presume the anomaly will occur based on domain knowledge. For instance, d controls the distance that anomaly scores may propagate. This value should be higher if we presume the average anomaly length is long and vice versa. If
(a)
(b)
MGAB NPSR
MGAB NPSR
0.5
0.9
0.8
0.3
M₁t
0.7
M₁e
Mara
Mase
0.6
0.1
80
100
100
opoch
epoch
Figure 6: (a) best Fl scores and (b) AUCs using point and sequence-based models as the anomaly score on the Mackey- Glass anomaly benchmark.
17

---

the anomaly is expected to occur abruptly and significantly, there would be clear gap between the distribution of nominality scores for normal and anomaly data (Fig. Ba). this case, hard gate function should be chosen as it allows anomaly scores to propagate through time points without reduction, as long as an anomaly time point has nominality score lower than 0x. Conversely, if the anomaly occurs progressively, the distribution of nominality scores is likely to overlap (Fig. B)). Here, a soft gate function will be more appropriate to prevent excessive accumulation of anomaly scores on normal time points, reducing false-positives. A dataset could contain both abrupt and progressive anomalies. However, based on 3 it is evident that using soft gate function generally yields better performance compared hard gate function. This suggests that the distribution of nominality scores is predominantly overlapped, which is also evident Fig. and Fig.
(a)
bb
(c)
(d)
Formal
Ancomall
Ontire
Noninally
10-1
hard
hard
32R
sdft
hard (0011
poit
port
10
0625
0400
0675
6:01
0.22
0G3 0GS 8:65 best (d:1)
Anomaly score (pasd)
Fest (8-16)
APA (d:10
Figure 7: Training results on the WADI dataset. Nominality score vs (a) anomaly score from point- based reconstruction, (b) best F1 score (d 16), (c) false positive rate (d 16), and (d) best F1 score (d = 1) using different ON.
(a)
hard func.
soft gate func.
nominality
nominality score
normal data
normal
threshold
anomaly
anomaly
data
data
Figure 8: Illustration of different distributional relationships between the nominality scores of normal and anomaly data. (a) No overlap with a paired threshold and hard gate function. (b) Overlapped with threshold and soft gate function.
G Broader Impacts
The detection of anomalies in time series data can minimize downtime and avert financial losses. Utilizing real-time monitoring of system conditions, anomaly detection techniques for time series data can automatically detect deviations from the expected system behavior, thereby avoiding potential risks and financial harm. This has the potential to reduce the need for manual monitoring of faults and to expedite decision-making processes. Additionally, it can promote the sustainability of AT by preventing energy wastage and system malfunction.
18

---

H
Limitations
Despite exhibiting competitive performance against other models, the proposed NPSR algorithm has few limitations. The point-based model used in training does not incorporate temporal informa- tion, which makes it challenging to effectively reconstruct oow-dimeniional datasets. This issue particularly challenging for univariate time series since raw inputs would not work for point- based models. One possible solution to this problem to increase dimensionality by aggregating multiple time points. However, the effectiveness of this approach is yet to be confirmed.
Another limitation of NPSR the absence of an automatic threshold (0,) )-indiigg method, which makes it difficult to determine suitable threshold when deploying the model. To address this issue, one can define a target false positive rate and estimate the threshold that achieves this target rate using the validation set since only normal data is needed. Similarly, estimating the optimal values for ON. d, and selecting the model used for calculating A(-) will be an important future work.
Within Fig. 5p of the main text, false negative instance was identified the temporal range between t = 14800 and dataset, this anomaly spans approximately 14.26 minutes and characterized as "Damage MV 001 and raw water pump to drain Elevated Reservoir tank." Notably, our analysis suggests that when assessing individual time points, the model Mg encounters difficulty in recognizing this anomaly. Conversely, the model Maeg excels in identifying time-dependent relationships, making more effective in capturing such contextual anomalies. The observed disparity in anomaly detection implies that this section comprises a relatively higher proportion of contextual anomalies than point anomalies. Consequently, when using Mseg we achieve higher anomaly score. However, our current approach utilizes the reconstruction error of Mgs as the basis for the anomaly score calculation, thus neglecting the effectiveness of the reconstruction eтror generated by Mseg. Consequently, the induced anomaly score fails to surpass the predefined threshold. In light of these findings, an essential avenue for future research is to investigate methods for selecting the model to be used in the computation of A(-). This undertaking holds promise for enhancing the overall performance and accuracy of anomaly detection in time series data.
14900 when employing the induced anomaly score. According to the WADI
19


Reviews:
[{"review_id": "lxo8RJmV9s", "reviewer": "Reviewer_m3iW", "summary": "Basically, this paper aims to consider anomalousness of data points both from a point perspective, which is independent of the temporal relationships in the data, and contextual perspective, which reflects temporal relationships in the data. The paper derives an induced anomaly score that considers both.", "questions": "At the start of section 2.2, $\\mathbf{x}_^0$ is part of the observed dataset but is also defined to be $\\mathbf{x}_t^c + \\Delta\\mathbf{x}_t^p$. This cannot be correct. Did you mean to define $\\delta\\mathbf{x}_t^p$?", "limitations": "No limitations are described on the work. Societal impact material is not relevant here. However, a description of future work should be provided based on any patterns in the errors that the presented algorithm makes.\n\n---\n\nPost rebuttal addition:\nThanks to the authors for pointing out the discussions on limitations in the supplementary material. However, such material is critical to understanding the nature of the contribution, and so needs to be in the main paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper addresses an important problem of considering both point and contextual anomalies. The paper provides good theoretical background for their work and convincing experimental results, showing particular anomalies that their method finds that are not found by methods that look for either point-based or contextual anomalies.", "weaknesses": "The datasets on which testing is performed have rather high anomaly rates. The authors should experiment with some datasets with much lower anomaly rates, perhaps by leaving out some anomalies in the datasets that they use.\n\nPost rebuttal comment:\nThanks to the authors for the responses. While the anomaly rates that the authors use are worth testing, it is quite common to use lower anomaly rates, and such lower rates would constitute a better test of the performance of the new algorithm. With regard to removing anomalies from datasets, one can remove individual time series that are identified as anomalous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere appreciation to Reviewer *m3iW* for conducting a thorough review and offering valuable insights. The raised points will be thoughtfully considered and integrated into the revised version.\n\n#### 1. The datasets on which testing is performed have rather high anomaly rates. The authors should experiment with some datasets with much lower anomaly rates, perhaps by leaving out some anomalies in the datasets that they use.\n> In the context of other time series anomaly detection literature, the datasets utilized in our work are widely used, and the anomaly rates are similar. The following contains a list of literature using the datasets:\n- SWaT – [10, 11, 12, 18, 23, 24, 26, 33, 37, 39, 40, 41] Supplementary [20]\n- WADI – [10, 11, 12, 23, 37, 39, 40, 41] Supplementary [20]\n- PSM – [24, 33]\n- MSL – [9, 11, 18, 23, 24, 25, 33, 38, 39, 40, 41]\n- SMAP – [9, 11, 18, 23, 24, 25, 33, 38, 39, 40, 41]\n- SMD – [9, 10, 11, 23, 24, 25, 33, 39, 41] Supplementary [20]\n\n> In considering removal of some anomalies from existing datasets, it is unclear how to appropriately do that – i.e., results may depend strongly on how missing values are imputed, how to remove sequence related anomalies particularly in multivariate cases, etc. We can note as future work, extensions to explore how detection rates extend to very rare anomaly rate scenarios.\n#### 2. At the start of section 2.2, **x**$^0$ is part of the observed dataset but is also defined to be **x**$^c_t$ + Δ**x**$^p_t$. This cannot be correct. Did you mean to define Δ**x**$^p_t$?\n\n> Yes, we are defining Δ**x**$^p_t$ and expressing that by definition (≜), **x**$^0$ equals **x**$^c_t$ + Δ**x**$^p_t$. We will update the equation and include it in the revision.\n\n#### 3. No limitations are described on the work.\n\n> We respectfully point out that we have presented the limitations in **Supplementary G**.\n\n#### 4. A description of future work should be provided based on any patterns in the errors that the presented algorithm makes.\n\n> In the revised version, we will expand the limitations section in the supplementary to include relevant discussions. \nFor instance, in (main text) **Figure 4 (b)**, there exists a false negative between $t=14800$ and $t=14900$ when using the induced anomaly score. According to the WADI dataset, this is a 14.26-minute anomaly described as “Damage 1 MV 001 and raw water pump to drain Elevated Reservoir tank.” When examining a single time point, $M_{pt}$ struggles to recognize the anomaly. However, $M_{seq}$ can detect anomalous time-dependent relationships between the time points. This observation indicates that the section contains relatively more contextual anomalies than point anomalies, resulting in a higher anomaly score using $M_{seq}$. Since we use the reconstruction error of $M_{pt}$ as $A(\\cdot)$, we lose the advantage of effectively utilizing the reconstruction error of $M_{seq}$, leading to an induced anomaly score that does not reach the threshold.\nAn important future direction would be to explore how to select the model used for calculating $A(\\cdot)$.\n"}, {"review_id": "KRLi5f9VnD", "reviewer": "Reviewer_EBhj", "summary": "This paper proposed a novel idea to handle the time-series anomaly detection problem by calculating the \"nominality score\"(concerns on this name are in the question part and I'll keep using this name in the following review) and the induced anomaly score. And the F1 score can be mathematically proved to improve using the proposed scores. The authors then constructed the point-based and sequence-based reconstruction models to estimate the anomaly score and experiments on several datasets demonstrate the superior performances.", "questions": "1. I wonder whether the word \"nominality\" is a misuse, although it is presented across the whole paper. The antonym to the word \"anomaly\" is \"normality\", and \"nominality\" is the noun of \"nominal\" which means \"existing in name only; far below the real value or cost\", and I think it's not related to the topic at all.\n2. How is the backbone of the model (i.e., the performer-based autoencoder) chosen? Will that cause an unfair comparison between the proposed method and the baselines due to the neural model's approximation ability?\n3. The supplementary file mentions how a best threshold is chosen, but in 3.4 ablation study the threshold is set to 98.5% of the nominality score from the training data. Why and how will the choice of different threshold influence the performance (i.e., whether the method is sensitive to the choice of hyperparameters?)", "limitations": "No apparent limitations and negative societal impact observed by the reviewer.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method is novel and provides a new angle to address the anomaly detection problem. Without directly considering how \"abnormal\" a point is, this work start with thinking what is \"normal\" and induce anomality score based on the nominality score. \n2. The authors provided both theoretical and experimental supports for the proposed methods which are reasonable.\n3. The writing of the paper is clear and easy to follow.\n", "weaknesses": "1. A major concern is on the expectation of improvement by using $\\hat A(\\cdot ;g_{\\theta_2})$. While the Claim 1 and Claim 2 as well as the proofs gives a guarantee that using this method won't result in a worse F1 score, it says nothing about the gap with/without the method. Intuitively, the expected improvement may be related to $\\alpha$ in Eq. (2), $d$, the function, the number of data, and other factors. The actual case could be that even if the $\\alpha$ is larger than 1 by a margin (say \\alpha=1.5, 2, or 5), the expected number of $N(t)\\leq \\theta$ for abnormal points are still a small portion of the whole dataset, making the proposed method useless in practical use. The analysis of a bound or expectation is missing.\n2. The correctness of the method relies on several hypotheses, typically: 1) distribution of abnormal points $\\Delta x_{t,a}^p$ has a larger variance than that of normal points $\\Delta x_{t, n}^p$. This is used to guarantee that nominality score of normal points are larger than abnormal points; 2) the threshold is well selected so that the nominality score of all the normal points (with $y_t=0$) passes the threshold, while a significant number of abnormal points cannot pass the threshold. While the hypotheses are reasonable theoretically, I wonder whether they still hold when the scores are estimated and calculated based on the outputs from neural networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our heartfelt gratitude to Reviewer *EBhj* for conducting a comprehensive review and providing valuable insights. We will take into account the raised points and incorporate them appropriately in the revised version.\n\n#### 1. The analysis of a bound or expectation of improvement for the induced anomaly score is missing.\n\n> To compute the bound or expectation for this scenario, certain statistical assumptions need to be made regarding the time series, as demonstrated in the toy example provided in **Section 2.3**. In the context of unsupervised anomaly detection, we can only observe normal data in the training set, which restricts our ability to model the distribution of normal data. Any deviation from this distribution may be labeled as an anomaly. Due to the lack of constraints on anomalies, establishing a universal bound on the potential improvement becomes highly challenging.\n#### 2. While the hypotheses are reasonable theoretically, I wonder whether they still hold when the scores are estimated and calculated based on the outputs from neural networks.\n\n> The hypotheses will be influenced by the underlying statistical structure of the time series and our choice of threshold, but not by the neural network approximations. By using **(1)**, **(8)**, **(11)**,   Δ**x**$^c_t$, Δ**x**$^p_t$, and $\\theta_N$, we can calculate the gate function outputs $g_{\\theta_N}(N(\\cdot))$, thereby inducing any $A(\\cdot)$ and computing a superior $\\hat{A}(\\cdot)$ (provided that $\\theta_N$ satisfies the condition in either **Claim 1** or **2**).\nWe approximate Δ**x**$^c_t$, Δ**x**$^p_t$, using neural networks, but the accuracy of these approximations will not affect the claims since there always exists some $\\theta_N$ that satisfies the conditions (but it will certainly affect the performance).\nThe toy example presented in **Section 2.3** serves as a case study to verify the appropriateness of the nominality score defined in **(1)**. We do not assume that real-world datasets have an i.i.d. normal distribution.\n\n#### 3. Misuse of the word “nominality”\n> We use the term **nominal** due to its usage in terms of measurement (*From a philosophical viewpoint, nominal value represents an accepted condition, which is a goal or an approximation, as opposed to the real value, which is always present.*). We did not use the term **normality** due to its related meaning with the normal distribution (*In statistics, **normality tests** are used to determine if a data set is well-modeled by a normal distribution and to compute how likely it is for a random variable underlying the data set to be normally distributed.*) (Source: Wikipedia)\n\n#### 4. How is the backbone of the model (i.e., the performer-based autoencoder) chosen? Will that cause an unfair comparison between the proposed method and the baselines due to the neural model's approximation ability?\n\n> We opt for performer-based models due to their high efficiency compared to the original transformer model [27]. Changing the architecture is also a viable option. Our approach remains feasible as long as we have both a point-based and sequence-based reconstruction model.\nWe do not anticipate that our model will lead to unfair comparisons. Similar to this work, most reconstruction-based baselines [11, 32, 36, 39, Zhou et al.] that utilize neural networks propose a combined workflow from designing the model architecture to calculating the anomaly score. As long as we maintain consistency in data preprocessing and metrics, our comparison method is standardized.\n\n- Bin Zhou, Shenghua Liu, Bryan Hooi, Xueqi Cheng, and Jing Ye. 2019. *BeatGAN: anomalous rhythm detection using adversarially generated time series*. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI'19). AAAI Press, 4433–4439.\n\n#### 5. The supplementary file mentions how a best threshold is chosen, but in 3.4 ablation study the threshold is set to 98.5% of the nominality score from the training data. Why and how will the choice of different threshold influence the performance (i.e., whether the method is sensitive to the choice of hyperparameters?)\n\n> To recap, there are two thresholds: $\\theta_a$ and $\\theta_N$. $\\theta_a$ is determined automatically by **Supplementary (3)**. $\\theta_N$ is a parameter for $g_{\\theta_N}(N(\\cdot))$,  when using either a soft **(8)** or hard **(11)** gate function. **Rebuttal Figure 3 (a)** shows the nominality scores vs anomaly scores (from point-based reconstruction errors) for the WADI dataset using the parameters shown in **Supplementary Table 1**. Here, we compare the best F1 scores and corresponding false positive rates using soft/hard gate functions at different $\\theta_N$. We see that our method is sensitive to both $\\theta_N$ and the choice of gate function. If $\\theta_N$ is set too high, it will result in an excessive accumulation of anomaly scores on normal time points, leading to an increase in false positives (**rebuttal Figure 3 (c)**). Conversely, if $\\theta_N$ is set too low, no anomaly scores will propagate through time points, and the induced anomaly score will approach the original anomaly score (**rebuttal Figure 3 (b)**). The value of 98.5% for $\\theta_N$ is an empirical choice that performs well across most datasets."}, {"review_id": "Q30rMKQIEN", "reviewer": "Reviewer_gJSP", "summary": "This paper proposes an unsupervised time series anomaly detection algorithm called NPSR (Nominality score conditioned TSAD by Point/Sequential Reconstruction), which combines both point-based and sequence-based reconstruction models. Specifically, it proposes a nominality score which is the ratio of a contextual deviation (or in-distribution deviation) to the total deviation (which is assumed to be the sum of the in-distribution deviation and the out-of-distribution deviation). The contextual deviation and the in-distribution deviation is computed by using the sequence-based and point-based reconstruction models. Based on the nominality score and anomaly score (computed using point-based reconstruction model) computed in the neighborhood, the induced anomaly score is further proposed by considering the temporal relationship. Some theoretical results of the proposed algorithm are provided. Experiments on several benchmark time series anomaly detection datasets are performed to demonstrate the performance of the proposed algorithm in comparison with several state-of-the-art algorithms.", "questions": "1.\tIn Table 2, the performance of several algorithms, e.g., Anomaly Transformer, is significantly lower than those reported in the literature. More justification is required. In Figure 4(b), the anomalies on the right of t=14800 are not detected by $M_{pt}+M_{seq}$, but detected by $M_{seq}$, resulting in false negative. Can you provide some explanations?    \n2.\tThe proposed algorithm is an unsupervised algorithm. Thus, how to adapt the algorithm on different datasets remains not clear to me (besides the threshold used in adjusting the F1).", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper studies an important and interesting problem, i.e., how to detect anomalies in time series data without label data. \n2.\tThe proposed algorithm combines both point-based and sequence-based reconstruction models, which achieves quite good performance on several time series anomaly detection benchmark datasets. \n3.\tSome theoretical results are provided for the proposed algorithm.\n4.\tThe paper is generally well written and the presentation is clear.", "weaknesses": "1.\tOverall, this paper proposes a heuristic-based unsupervised time series anomaly detection algorithm. The overall pipeline in Algorithm looks Ok to me, but from experiments it seems the most important part is the point-based reconstruction. In other words, the rest of the proposed algorithm may be simplified. The ablation studies in Table 3 also partially confirms it.\n2.\tFlawed experiments. The F1 scores reported in Table 2 for other algorithms, e.g., Anomaly Transformer, are not consistent with results reported in the literature, such as [33]. More justifications are required. \n3.\tThe theoretical results look sound, but may not be useful in practice. In other words, can you justify the value of these results in real-world time series anomaly detection applications?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere appreciation to Reviewer *gJSP* for their thorough review and valuable insights. The points raised will be duly addressed in the revised version.\n\n#### 1. From experiments it seems the most important part is the point-based reconstruction. In other words, the rest of the proposed algorithm may be simplified.\n\n> Instead of focusing solely on point-based reconstruction, we believe that the most important idea is to achieve a balance between effectively utilizing both point and sequence-based models and carefully selecting the appropriate parameters.\nAs an extreme example, we optimize our algorithm on the Mackey-Glass anomaly benchmark (MGAB), which is a univariate time series dataset, and find that sequential-based models can significantly outperform point-based models when using their reconstruction errors as $A(\\cdot)$. As in **rebuttal Figure 1**, we see that sequence-based models can correctly identify the anomalies, whereas point-based models did not learn at all. This is reasonable, as point-based models only consider a single time point. In general, the lower the dimensionality of the dataset, the harder it gets for a point-based model to learn statistically meaningful representations.\nMoreover, looking at the high-dimensional datasets reported in (main text) **Table 3**, the induced anomaly score remains important for some datasets: For the MSL dataset, F$1^*$ improves 0.1 (F$1^*=0.467$) compared to using the point-based reconstruction (F$1^*=0.366$). In **rebuttal Figure 3**, F$1^*$ improves 0.047 (F$1^*=0.673$) compared to the point-based anomaly score (F$1^*=0.626$).\nSince the amount of improvement depends on the statistical structure of the test data, it is still useful to consider using both sequential-based and point-based reconstruction in general.\nWe will include relevant additional discussions in the revision.\n\n#### 2. Flawed experiments. The F1 scores reported in Table 2 for other algorithms, e.g., Anomaly Transformer, are not consistent with results reported in the literature, such as [33]. In Table 2, the performance of several algorithms, e.g., Anomaly Transformer, is significantly lower than those reported in the literature. More justifications are required.\n\n> In (main text) **Table 2**, we present the best F1 scores *without point-adjustment (PA)*, while [33] reports the scores *with PA*. PA involves adjusting predictions based on the true labels. However, as this method has been shown to be a flawed approach that can overestimate performance in previous studies [11, 23, 24, 34], we do not endorse its use.\nNevertheless, we have included both PA and non-PA data in **Supplementary D**. Our results demonstrate that even simple heuristic methods can appear to achieve state-of-the-art performance when using PA. We see that our method still achieves competitive performance when using PA, but again, the community is moving away from this flawed metric/approach.\n\n#### 3. The theoretical results look sound, but may not be useful in practice. In other words, can you justify the value of these results in real-world time series anomaly detection applications?\n\n> Let's consider (main text) **Figure 3**. If we set $\\theta_N=2$ for SWaT (or $\\theta_N=10$ for WADI), which ensures that no anomaly points have a nominality score higher than $\\theta_N$, and set $d=1$, then according to **Claim 2**, the performance using the induced anomaly score will surpass that of using the original anomaly score. This shows that if we set a proper $\\theta_N$, we can always get a better or equal performance using the induced anomaly score. We can also see from **rebuttal Figure 3 (d)** that **Claim 2** always hold when $d=1$ and $\\theta_N$ is set such that no anomaly points have a nominality score above $\\theta_N$.\nAt present, we are using a certain quantile of the nominality score of the training data (e.g., 98.5%) to set $\\theta_N$. Estimating the optimal values for $\\theta_N$ and $d$ will be an important future work.\n\n#### 4. In Figure 4 (b), the anomalies on the right of t = 14800 are not detected by $M_{pt}+M_{seq}$, but detected by $M_{seq}$, resulting in false negative. Can you provide some explanations?\n\n> The anomaly section between $t=14800$ and $14900$ contains relatively more contextual than point anomalies. This anomaly is described as *Damage 1 MV 001 and raw water pump to drain Elevated Reservoir tank.*\nWhen solely considering a time point, $M_{pt}$ struggles to recognize the anomaly. However, $M_{seq}$ can effectively detect anomalous time-dependent relationships, leading to a higher anomaly score using $M_{seq}$. Since the reconstruction error of $M_{pt}$ is used as $A(\\cdot)$ in (main text) **Figure 4**, we lose the advantage of effectively utilizing that of $M_{seq}$. This results in an induced anomaly score that is not high enough to reach the threshold. An important future direction would be to explore how to select $A(\\cdot)$ among multiple models.\n\n#### 5. The proposed algorithm is an unsupervised algorithm. Thus, how to adapt the algorithm on different datasets remains not clear to me (besides the threshold used in adjusting the F1).\n> The difficulty of choosing these parameters is due to the absence of anomalies in the training data and the real-time nature of the algorithm. However, leveraging domain knowledge allows us to make assumptions about anomalies. The selection of parameters, including the choice between soft or hard gate functions, the value for $d$ and the ratio for $\\theta_N$, mostly depends on how we presume the anomaly will occur. For instance, $d$ controls the distance that anomaly scores may propagate. This value should be higher if we presume the average anomaly length is long and vice versa.\nDiscussions regarding how to choose a soft or hard gate function can be found in our rebuttal replies to reviewer *UPqe* (*Question 1*). We did not discuss it here due to character limit.\nWe will include relevant discussions in the revision.\n"}, {"review_id": "5hrcTDHbMK", "reviewer": "Reviewer_UPqe", "summary": "TSAD techniques are targeted towards detecting either point or contextual anomalies, but often struggle to adequately capture both simultaneously. In this work, the authors propose a novel reconstruction-based AD technique that introduces the notion of a nominality score and a subsequent induced anomaly score. The method achieves a better trade-off between detecting point and contextual anomalies than competing techniques, and outperforms them over a variety of benchmark multivariate datasets. ", "questions": "I would appreciate if the authors could respond to the concerns listed in the *Weaknesses* section, and either defend their approach, or indicate how they can improve on the existing submission. \n\nAlthough I find the problem investigated in the paper to be interesting and well-motivated, I don’t think that the submission is ready for publication in its current state, and believe it would strongly benefit from another revision/round of reviews.\n", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The problem statement is interesting and relevant, and I appreciate the use of visualisations such as in Figure 1, that convey the difficulty of the task.\n- I appreciated that the model has several components that could be tweaked, such as the choice of soft or hard gate function, and the ablation study does a good job in highlighting their individual contributions. I would however have liked to see more discussion or synthetic examples that demonstrate whether there are certain properties of the monitored data that can guide the selection (beyond empirical comparison).", "weaknesses": "- I found the presentation and clarity to be quite poor overall, as a result of which the contributions are sometimes difficult to follow properly.  Section 2.2 introduces a lot of similar notations that are difficult to keep track of – a figure could be very helpful here for conveying the differences between the various time series and deviations.\n- The authors rely on results reported in earlier papers as performance measures for several competing techniques. While I appreciate that it may be time-consuming to re-implement and re-run all experiments, I do worry about potential inconsistencies in the experimental set-up that may involuntarily skew the comparison. The authors mention how if the results from the original paper are unavailable, *“we search for the highest reported values among other publications.”* I can see this as introducing several inconsistencies, especially since the source of the result isn’t reported alongside the performance figures presented in the table.\n- I would have liked there to be a dedicated *Related Work* section in the main paper in order to better understand how the work fits alongside other reconstruction-based techniques.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere gratitude to Reviewer *UPqe* for their detailed review and valuable insights. We will address the raised points in the revised version accordingly.\n\n#### 1. More discussion or synthetic examples that demonstrate whether there are certain properties of the monitored data that can guide the selection (beyond empirical comparison).\n\n> The difficulty in choosing these parameters stems from the absence of anomalies in the training dataset. The selection of soft or hard gate functions, the value for $d$, and the ratio for $\\theta_N$ largely depend on how we presume the anomaly will occur based on domain knowledge.\nFor example, if the anomaly is expected to occur abruptly and significantly, there would be a clear gap between the distribution of nominality scores for normal and anomaly data (**rebuttal Figure 2 (a)**). In this case, a hard gate function should be chosen as it allows anomaly scores to propagate through time points without reduction, as long as an anomaly time point has a nominality score lower than $\\theta_N$. Conversely, if the anomaly occurs progressively, the distribution of nominality scores is likely to overlap (**rebuttal Figure 2 (b)**). Here, a soft gate function will be more appropriate to prevent excessive accumulation of anomaly scores on normal time points, reducing false-positives.\nA dataset could contain both abrupt and progressive anomalies. However, based on (main text) **Table 3**, it is evident that using a soft gate function generally yields better performance compared to a hard gate function. This suggests that the distribution of nominality scores is predominantly overlapped, which is also evident in (main text) **Figure 3** and **rebuttal Figure 3**.\nAnother heuristic for selecting whether to use the point or seq-based reconstruction error as $A(\\cdot)$ is by considering the dataset's dimensionality. In an experiment conducted on a univariate dataset, we demonstrated that a sequence-based model outperforms a point-based model significantly (**rebuttal Figure 1**). This is because when dimensionality reduces, extracting meaningful statistical structure from a single time point becomes more challenging.\nWe will include relevant additional discussions in the revision.\n\n#### 2. Section 2.2 introduces a lot of similar notations that are difficult to keep track of – a figure could be very helpful here for conveying the differences between the various time series and deviations.\n\n> (main text) **Figure 2 (a)** illustrates the relationships for the introduced time series notations, including **x**$^0_t$, **x**$^p_t$, **x**$^c_t$, Δ**x**$^0_t$, … etc.. These notations are placed beside (main text) **Figure 2 (b)** to accommodate page limit requirements. In the revised version, we will reorganize the figures to arrange them more closely to **Section 2.2**.\n\n#### 3. The source of the result isn’t reported alongside the performance figures presented in the table. Potential inconsistencies in the experimental set-up that may involuntarily skew the comparison. \n\n> The source of data is reported in **Supplementary E**, where we indicate how we obtain the data values and reference the literature where the values are reported. We are aware of the inconsistencies arising from different sources. Therefore, we ensure that we compare against the best baseline values reported. We also acknowledge other time series anomaly detection literature that utilize a similar data populating method [11, 18, 23].\n\n#### 4. Dedicated Related Work section is required in the main paper to better understand how the work fits alongside other reconstruction-based techniques.\n\n> We will include a section for related work in the revised version similar to the following:\nReconstruction-based techniques have seen a diversity of approaches over the years. The simplest reconstruction technique involves training a separate model sequentially for each channel using UAE [11]. One type of improvement focuses on network architecture, with notable examples like LSTM-VAE [36], MAD-GAN [37], MSCRED [32], OmniAnomaly [9], and TranAD [41]. Additionally, hybrid architectures like DAGMM [35] and MTAD-GAT [38] have been proposed. Another type aims to improve the anomaly score instead of using the original reconstruction error. Designing this anomaly score involves a considerable amount of art, given the high diversity of methods for calculating it across studies. For instance, USAD uses two weighted reconstruction errors [39], OmniAnomaly employs the \"reconstruction probability\" as an alternative anomaly score [9], MTAD-GAT combines forecasting error and reconstruction probability [38], and TranAD uses an integrated reconstruction error and discriminator loss as the anomaly score [41].\nIn the context of network architecture, our method utilizes a straightforward performer-based encoder-decoder structure without incorporating any specialized components. Based on our insight into the point-sequential detection tradeoff, our approach stands out due to the introduction of nominality and induced anomaly scores. This integration enables us to seamlessly combine point-based and sequence-based reconstruction errors, resulting in competitive performance."}]

Author rebuttal:
We sincerely express our gratitude to all the reviewers for their thorough reviews and valuable insights.

A significant hurdle in time series anomaly detection involves effectively modeling time-dependent relationships and detect both contextual and point anomalies accurately. To tackle this issue, we propose an unsupervised framework – *NPSR* – that utilizes both point and sequence-based reconstruction models. We introduce a *nominality score* derived from the ratio of a combined value of the reconstruction errors, and a subsequent derivation of an *induced anomaly score*. We provide theoretical evidence supporting the superiority of the induced anomaly score under specific conditions. Extensive experiments on various public datasets demonstrate that NPSR outperforms most state-of-the-art baselines.

Overall, the reviews provide constructive feedback and praise the paper's novelty and relevance. They also raise concerns regarding experimental comparisons, clarity of presentation, and the need for addressing certain theoretical aspects of the proposed method. 

We diligently addressed all the concerns raised by providing ample evidence and the requested results. The raised points will be thoughtfully considered and integrated into the revised version.

Here is the summary of the revisions:
1. **Parameter Selection (Reviewers UPqe, gJSP, EBhj)**: We have thoroughly discussed heuristics for selecting the gate function (**rebuttal Figure 2**), $d$, and $\theta_N$. Additionally, we have provided two additional experimental results (**rebuttal Figure 1** and **3**) to support our viewpoints.
2. **Theoretical Justifications (Reviewers gJSP, EBhj, m3iW)**: We have presented an example where, according to **Claim 2**, the induced anomaly score demonstrates provable superior performance compared to the anomaly score (**rebuttal Figure 3**). Furthermore, we have clarified the conditions under which these hypotheses hold.
3. **Algorithm Justifications (Reviewers gJSP, EBhj)**: To underscore the importance of utilizing both point and sequence-based models, we have included an additional example (**rebuttal Figure 1**). Moreover, we have discussed the fairness of our algorithm in comparison to other studies.
4. **Missing Sections/Figures (Reviewers UPqe, m3iW)**: In response to the feedback, we will add the Related Works section to the main text, and we will extend the Limitations section in the supplementary material to include future works that can mitigate errors made by the presented algorithm.
5. **Flawed Experiments (Reviewer gJSP)**: We have identified a bug with calculating $\theta_a$ for Anomaly Transformer. The bug has been fixed, and the results have been updated. The results for other algorithms remain consistent. Please refer to the **updated Table 2** in the global response. We wish to emphasize that we ***do not use*** the *point-adjustment* method (a method for adjusting predictions according to true labels), which is used in some other literature.
6. **Experiment Results Explanation (Reviewer gJSP)**: In the revision, we have extended our discussion about the false negatives appearing in (main text) **Figure 4 (b)** to provide a more comprehensive explanation.

Additionally, we have noticed that some important information is present in the supplementary material but not in the main text. In response to this, we will rearrange certain aspects in the revision to ensure all pertinent information is appropriately included in the main text.

We are enthusiastic about addressing any further questions or inquiries from the reviewers to ensure the continued improvement and refinement of our work.


Meta-review: This paper addresses a problem of unsupervised time series anomaly detection, presenting a method which leverages both point-based and sequence-based reconstruction. A nominality score is introduced as the ratio of in-distribution deviation of the total deviation and the induced anomaly score is presented. The problem itself is timely and interesting. All reviewers feel that the paper has interesting contributions with some theoretical results as well. The authors did a good job in resolving most of reviewers' concerns in their rebuttal,
leading to the overall score raised. It was suggested to include experiments on datasets with lower than 4% anomalies. I believe that the authors can handle this in their final version.
Decision: Accept (poster)