Title: Nominality Score Conditioned Time Series Anomaly Detection by Point/Sequential Reconstruction
Venue: NeurIPS 2023
Accepted: yes
Keywords: time series

anomaly detection

point anomalies

contextual anomalies

nominality score

induced anomaly score
Abstract: Time series anomaly detection is challenging due to the complexity and variety of patterns that can occur. One major difficulty arises from modeling time-dependent relationships to find contextual anomalies while maintaining detection accuracy for point anomalies. In this paper, we propose a framework for unsupervised time series anomaly detection that utilizes point-based and sequence-based reconstruction models. The point-based model attempts to quantify point anomalies, and the sequence-based model attempts to quantify both point and contextual anomalies. Under the formulation that the observed time point is a two-stage deviated value from a nominal time point, we introduce a nominality score calculated from the ratio of a combined value of the reconstruction errors. We derive an induced anomaly score by further integrating the nominality score and anomaly score, then theoretically prove the superiority of the induced anomaly score over the original anomaly score under certain conditions. Extensive studies conducted on several public datasets show that the proposed framework outperforms most state-of-the-art baselines for time series anomaly detection.

Nominality Score Conditioned Time Series Anomaly Detection by Point/Sequential Reconstruction

Abstract

Time series anomaly detection is challenging due to the complexity and variety of patterns that can occur. One major difficulty arises from modeling time-dependent relationships to find contextual anomalies while maintaining detection accuracy for point anomalies. In this paper, we propose a framework for unsupervised time series anomaly detection that utilizes point-based and sequence-based reconstruction models. The point-based model attempts to quantify point anomalies, and the sequence-based model attempts to quantify both point and contextual anomalies. Under the formulation that the observed time point is a two-stage deviated value from a nominal time point, we introduce a nominality score calculated from the ratio of a combined value of the reconstruction errors. We derive an induced anomaly score by further integrating the nominality score and anomaly score, then theoretically prove the superiority of the induced anomaly score over the original anomaly score under certain conditions. Extensive studies conducted on several public datasets show that the proposed framework outperforms most state-of-the-art baselines for time series anomaly detection.

1 Introduction

Time series anomaly detection involves identifying unusual patterns or events in a sequence of data collected over time [1]. This technique is crucial in fields such as finance [2], healthcare [3], manufacturing [4], transportation [5], and more [6]. A comprehensive evaluation of different techniques can be found in [7]. Time series anomaly detection using unsupervised learning approaches is favored by recent studies due to the fact that they don't require labeled data and have better detection of unseen anomalies [8], which is also the approach used in this work. Most unsupervised time series anomaly detection methods involve calculating an anomaly score at each time point and then comparing this score to some threshold. For calculating this score, we can categorize different methods into three main groups: Reconstruction-based methods involve reconstructing the original time series data and comparing the reconstructed data with the actual data [9, 10, 11]. Predictionbased methods involve predicting the next value in the time series and comparing it with the actual value [12, 13, 14]. Dissimilarity-based methods measure the distance between the value obtained from the model and the distribution or cluster of the aggregated data [15, 16, 17, 18, 19, 20]. There can also be hybrid techniques where multiple methods are applied.

While the classification approaches for the types of anomalies differ in the literature [6, 7], we will focus on two classes: point anomalies and contextual anomalies. Point anomalies refer to individual data points that significantly deviate from the expected behavior of the time series and can be detected by observing the data at a single time point. Contextual anomalies, on the other hand, refer to data points that deviate from the expected behavior of the time series in a specific context or condition. These anomalies cannot be detected by observing the data at a single time point and can only be detected by observing the contextual information. As a result, point anomalies can be detected using any general anomaly detection technique that does not require temporal information. Such models are called point-based models in this work. However, detecting contextual anomalies requires a model that can learn temporal information. Such models require a sequence of input, hence they are termed sequence-based models in this work. A time point may contain both point and contextual anomalies. Obviously, contextual anomalies are harder to detect. An important trade-off arises when a model tries to learn time-dependent relationships for detecting contextual anomalies but loses precision or accuracy in finding point anomalies. This trade-off becomes more significant in high-dimensional data, where modeling the temporal relationship is difficult [21].

Our intuition comes from the observation that state-of-the-art methods using sequence-based reconstruction models encounter the point-contextual detection trade-off, resulting in noisy reconstruction results and suboptimal performance. As an alternative, we start by experimenting with point-based reconstruction methods, which exhibit a lower variance as they do not require the modeling of timedependent relationships. Despite the absence of temporal information, we find that the corresponding anomaly score for a point-based reconstruction model can already yield a competitive performance. To further bridge the gap between point-based and sequence-based models, we introduce a nominality score that can be calculated from their outputs and derive an induced anomaly score based on the nominality score and the original anomaly score. We find that the induced anomaly score can be superior to the original anomaly score. To provide theoretical proof of our findings, we frame the reconstruction process as a means to fix the anomalies and identify underlying nominal time points and prove that the induced anomaly score can always perform better or just as well as the original anomaly score under certain conditions. We also conduct experiments on several single and multi-entity datasets and demonstrate that our proposed method surpasses the performance of recent state-of-the-art techniques. We coin our method Nominality score conditioned time series anomaly detection by Point/Sequential Reconstruction (NPSR).

2 Related Works

Reconstruction-based techniques have seen a diversity of approaches over the years. The simplest reconstruction technique involves training a separate model sequentially for each channel using UAE [11]. One type of improvement focuses on network architecture, with notable examples like LSTMVAE [22], MAD-GAN [23], MSCRED [24], OmniAnomaly [9], and TranAD [25]. Additionally, hybrid architectures like DAGMM [26] and MTAD-GAT [27] have been proposed. Another type aims to improve the anomaly score instead of using the original reconstruction error. Designing this anomaly score involves a considerable amount of art, given the high diversity of methods for calculating it across studies. For instance, USAD uses two weighted reconstruction errors [28], OmniAnomaly employs the "reconstruction probability" as an alternative anomaly score [9], MTADGAT combines forecasting error and reconstruction probability [27], and TranAD uses an integrated reconstruction error and discriminator loss as the anomaly score [25]. In the context of network architecture, our method utilizes a straightforward performer-based structure without incorporating any specialized components. Based on our insights into the point-sequential detection tradeoff, our approach stands out by integration of point-based and sequence-based reconstruction errors for competitive performance.

3 Methods

3.1 Problem Formulation

Let X = { x 1 , ..., x T } denote a multivariate time series with x t ∈ R D , where T is the time length and D is the dimensionality or number of channels. There exists a corresponding set of labels y = { y 1 , ..., y T } , y t ∈ { 0 , 1 } indicating whether the time point is normal ( y t = 0 ) or anomalous ( y t = 1 ). For a given X , the goal is to yield anomaly scores for all time points a = { a 1 , ..., a T } , a t ∈ R and a corresponding threshold θ a such that the predicted labels ˆ y = { ˆ y 1 , ..., ˆ y T } , where ˆ y t ≜ ✶ a t ≥ θ a , match

y as much as possible. To quantify how matched ˆ y and y is, or how good a is for potentially yielding a good ˆ y , there are several performance metrics that takes either a or ˆ y into account [11, 12, 29, 30, 31]. This work mainly focuses on the best F1 score ( F1 ∗ ) without point-adjust, also known as the pointwise F1 score, which is defined as the maximum possible F1 score considering all thresholds. (A complete derivation for F1 ∗ is covered in Appendix A.)

3.2 Nominal Time Series and Two-stage Deviation

We denote the observed data as X 0 = { x 0 1 , ..., x 0 T } , where x 0 t ∈ R D . Assume that for each X 0 , there exists a corresponding underlying nominal time series data X ∗ = { x ∗ 1 , ..., x ∗ T } that comes from a nominal time-dependent process x ∗ t = f ∗ ( t ) : N → R D . The corresponding total deviation at t ( ∆ x 0 t ) is defined as ∆ x 0 t ≜ x 0 t -x ∗ t . We denote X ∗ as the set of all possible x ∗ t for all t ∈ { 1 , ..., T } . ∆ x 0 t can be separated into two additive factors ∆ x c t and ∆ x p t , such that, by definition, x c t = x ∗ t +∆ x c t and x 0 t = x c t +∆ x p t . We define ∆ x c t as the in-distribution deviation , where x c t ∈ X ∗ ; and ∆ x p t as the out-of-distribution deviation , which is non-zero if and only if x 0 t / ∈ X ∗ . ∆ x p t can be a means for quantifying the point anomaly, and ∆ x c t can be a means for quantifying the contextual anomaly. This is reasonable, since no matter how large ∆ x c t is, we still have x c t ∈ X ∗ , i.e., the in-distribution deviated value x c t is still in the set of all possible nominal time point data, and cannot be detected using a point-based model. On the other hand, it is possible that having learned X ∗ , a point-based model can negate the deviated value caused by ∆ x p t . Fig. 1(a) gives an illustration of the relationships between the variables at time t . We clarify this using the example below.

Assume we obtain a dataset from the streaming data of a 2D position sensor, where x ∗ t , x c t , x 0 t ∈ R 2 , and we have learned that the nominal time series is the circular movement of a point around the origin with some angular velocity ω and radius r , where R min ≤ r ≤ R max . Accordingly, we can deduce that X ∗ = { [ x y ] T | R 2 min ≤ x 2 + y 2 ≤ R 2 max } and x ∗ t = [ r cos ωt r sin ωt ] T . One possible cause (among many others) of contextual anomalies might be due to an unexpected change in angular velocity. For instance, failures in the system might lead to a slowdown of the circular movement between t 1 and t 2 , i.e., ∆ x c t = [ r (cos ω ′ t -cos ωt ) r (sin ω ′ t -sin ωt )] T for t ∈ { t 1 , ..., t 2 } and ∆ x c t = 0 elsewhere. Moreover, noisy measurements of individual time points may induce point anomalies in the observed time series, i.e., ∆ x p t = [ w x,t w y,t ] T where w x,t or w y,t is nonzero such that x 0 t / ∈ X ∗ for some t . Fig. 1(b)(c) gives an illustration of the above example. The black dots are time points where x 0 t = x ∗ t (no anomalies). The blue dots are time points exhibiting a slowdown (contextual anomalies). The red dots are time points with noisy measurements (point anomalies). The purple dots are time points with both slowdown and noisy measurements (point and contextual anomalies). The green dots are time points with noisy measurements but x 0 t ∈ X ∗ , so are still contextual anomalies since their deviations cannot be detected by observing a single time point.

3.3 The Nominality Score

Now we conceptualize the Nominality Score N ( · ) . Analogous to the anomaly score, N ( · ) indicates how normal a time point is. A nominality score N ( · ) is appropriate if for every possible θ N > 0 , P ( N ( t ) > θ N | y t = 0) > P ( N ( t ) > θ N | y t = 1) for all t ∈ { 1 , ..., T } , i.e., the portion of normal points that has a nominality score larger than θ N is strictly larger than the portion of anomaly points

(a)

(b)

(c) ↑△r

0

T

T

3T

T

x

*

4

2

4

x

&pt,t

Rmin

Rmax

↑△θ

0

X*

x

*

*

xt

*

T

T

3T

T

4

2

4

that has a nominality score larger than θ N . There are many ways to define N ( · ) . In this study, we define N ( t ) as the ratio of the squared L2-norm between ∆ x c t and ∆ x 0 t .

We provide an example as to when N ( · ) will be appropriate. In this derivation, we add an n and a in the subscript to denote variables that are only associated with normal and anomaly points, respectively. Consider a toy dataset, where ∆ x c t,n , ∆ x p t,n , ∆ x c t,a , and ∆ x p t,a have been defined:

where F is the F-distribution with D and D degrees of freedom. Fig. 2 illustrates the probability density function of N n ( · ) and N a ( · ) for different D and α . If α > 1 , then N ( · ) becomes an appropriate nominality score, since

where f ( · ; D,D ) is the probability density function of the F-distribution with degrees of freedom D and D . Indeed, it is reasonable to assume that ∆ x p t,a has a larger variance than ∆ x p t,n .

3.4 The Induced Anomaly Score

Having N ( · ) defined, we now propose a method for integrating any given N ( · ) and anomaly score A ( · ) to yield an induced anomaly score ˆ A ( · ) , and show some instances where the performance will improve over using A ( · ) or a smoothed A ( · ) . Consider a dataset that contains subsequence anomalies. For two near time points t and τ , it is natural to assume that the possibility of t being anomalous is affected by τ . We quantify this effect as A ( t ; τ ) , which is the induced anomaly score at t due to τ . By summing over a range of τ around t , we get the induced anomaly score at t :

where d is the induction length. Furthermore, we define A ( t ; τ ) as a gated value of A ( τ ) , which is controlled by the nominality score from τ to t :

(a)

1.0

Nn, Na(a=1)

(b)

2.0

Nn, Na(a=1)

Na(a=0.5)

Na(a=0.5)

20.8

Na(a=0.8)

(001

1.5

Na(a=0.8)

II

Na(a=1.2)

1

II

Na(a=1.2)

D

0.6

Na(a=1.5)

D

1.0

Na(a=1.5)

D

Na(a=2)

Na(a=2)

D

f(N,

M

0.5

0.4

0.0

0.0

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Nn or Na

Nn or Na

where the gate function g θ N ( N ) is some transformation function of N conditioned on a threshold θ N . A reasonable assumption is that g θ N ( N ) is a non-increasing function of N , i.e., N > N ′ implies g θ N ( N ) ≤ g θ N ( N ′ ) . Indeed, if N ( k ) is large, then time point k is likely a normal point, and any two points t 1 , t 2 , where t 1 < k < t 2 , are unlikely to be in the same anomaly subsequence, hence A ( t 1 ; t 2 ) and A ( t 2 ; t 1 ) should be small. Explicitly, we can use ˆ A ( t ) = ˆ A ( t ; g θ N ) and A ( t ; τ ) = A ( t ; τ, g θ N ) to denote that these values are conditioned on g θ N . Overall, ˆ A ( · ) can be thought of as some (unnormalized) weighted smoothed value of A ( · ) , where the weights are the product of g θ N ( N ( · )) across some range. We consider the following two cases:

Claim 1 Using a soft gate function,

If there exists θ 1 such that N ( t ) ≥ θ 1 for all normal points ( y t = 0 ), then F1 ∗ ( ˆ A ( · ; g θ 1 ); y ) ≥ F1 ∗ ( A ( · ); y ) , i.e., the best F1 score using the induced anomaly score with g θ 1 as the gate function is greater or equal to the best F1 score using the original anomaly score.

Proof 1 For any normal time point t n , we have

The equality in the middle arises from the fact that g θ 1 ( N ( t n )) = 0 , according to (8) and the assumptions. However, for any anomaly point t a , we have

This is because N ( t a ) might be lower than θ 1 , and hence g θ 1 ( N ( t a )) ≥ 0 . By potentially having a higher ˆ A ( t ) than A ( t ) for anomaly points, we get a potentially higher F1 ∗ .

Such a θ 1 indeed exists in real applications (e.g. the minimum nominality score among all normal points t n ). However, targeting this value barely leads to any improvement for F1 ∗ in practice. This is because anomaly points are generally fewer than normal points, resulting in barely any anomaly points t a having N ( t a ) < θ 1 . Nevertheless, we have shown that the soft gate function along with the induced anomaly score can provably yield equal or better F1 ∗ under some threshold.

Claim 2 Using a hard gate function, g θ N ( N ) ≜ ✶ N<θ N (11) If d = 1 , and there exist two thresholds: (i) θ 2 such that N ( t ) < θ 2 for all anomaly time points ( y t = 1 ) (ii) θ ∞ = ∞ ; then F1 ∗ ( ˆ A ( · ; g θ 2 ); y ) ≥ F1 ∗ ( ˆ A ( · ; g θ ∞ ); y ) , i.e., the best F1 score using the induced anomaly score with g θ 2 as the gate function is greater or equal to the best F1 score using the induced anomaly score with g θ ∞ as the gate function.

Proof 2 For any anomaly time point t a , we have

where the first equality arises from the fact that g θ 2 ( N ( t a )) = 1 , according to (11) and the assumptions. For any normal time point t n , we have

regardless of normal or anomaly points since N ( t ) < θ ∞ for any t . Therefore, since ˆ A ( t a ; g θ 2 ) = ˆ A ( t a ; g θ ∞ ) and ˆ A ( t n ; g θ 2 ) ≤ ˆ A ( t n ; g θ ∞ ) , we get a potentially higher F1 ∗ when using θ 2 compared to using θ ∞ .

ˆ A ( · ; g θ ∞ ) can be viewed as the smoothed value (or shifted simple moving average) over A ( · ) with a period of 2 d +1 . This averaging method is common among other studies [11, 12, 32, 33]. Claim 2 implies that by conditioning on N ( · ) and calculating ˆ A ( t ) , the performance can be improved over using a simple smoothing value of A ( · ) . In practice, we can relax the constraint of d , and use other gated functions to yield a more flexible architecture. Note that the appropriateness of a nominality score is critical for this to work.

3.5 Point-based Reconstruction Models

Consider some model M pt that reconstructs each time point t point-wise: ˆ x pt,t ≜ M pt ( x 0 t ) . Using M pt , a method for yielding the anomaly score is by using the point-based reconstruction mean-squared error, defined as a pt = { a pt, 1 , ..., a pt,T } , where a pt,t ≜ ∥ ˆ x pt,t -x 0 t ∥ 2 2 . One concern for using this kind of anomaly score is that we are not taking into account any time-dependent relationships for deriving a pt , so simply using M pt and a pt can barely be classified as a time series anomaly detection approach. Surprisingly, however, we find that a simple realization of M pt can already achieve impressive results for F1 ∗ (section 4.4).

Since M pt learns to capture the distribution of all normal point data, we assume that ˆ x pt,t ∈ X ∗ or is very close. Moreover, since ˆ x pt,t can offset point-anomalies, we assume x c t ≈ ˆ x pt,t , and implement a point-based reconstruction model to calculate ˆ x pt,t in practice.

M pt can be any model that has the ability to reconstruct X 0 point-wise. To our surprise, the best performance can be achieved by using a simple Performer-based autoencoder ([34]) that actually has the potential to discover temporal information. Despite having such a possibility, it only learned to reconstruct point-by-point during training. We demonstrated this fact by shuffling the input time points and observing that the result will be the same after reordering the output sequence. One possible explanation is that during training, it is a lot easier to individually reconstruct single time points than to find complex time-dependent relationships; and since the Performer-based autoencoder tries to optimize over a batch of time points, this reduces the effect of overfitting and allows the model to better generalize to unseen data. However, the exact reason for this remains an open question. For the rest of the study, we will use M pt to refer to the Performer-based autoencoder model. Details for the architecture of M pt are shown in Appendix B.1.

3.6 Sequence-based Reconstruction Models

Contrary to x c t , x ∗ t should not only be in X ∗ but also obey the time-dependent relationships. Therefore, it is necessary that the model ( M seq ) for approximating x ∗ t takes a sequence of time points as input.

How close M seq ( X 0 ) approximates X ∗ depends on the amount of training data and the model capacity. In practice, for computational reasons, only a section of X 0 is input and reconstructed at a time. We found that given a subsequence X 0 ab = { x 0 a , ..., x 0 b } as input for reconstruction, a model tends to simply reconstruct individual points and do not take temporal information into account (as discussed in section 3.5). Therefore, we use a Performer-based stacked encoder as M seq , which predicts the middle δ points from its surrounding 2 γ points to force the learning of time-dependent relationships. We concatenate all the predicted time points output by M seq to construct ˆ X ∗ . For the rest of the study, we will use M seq to refer to the Performer-based stacked encoder model. Details for the architecture of M seq are shown in Appendix B.2. Fig. 3 gives an illustration of the architecture for M pt , M seq , and the overall scheme. By obtaining ˆ X c and ˆ X ∗ , we can calculate N ( · ) and select some A ( · ) for calculating ˆ A ( · ) . The algorithm for evaluating a trained M pt and M seq using the soft gate function is shown in Algorithm 1 .

a

(b)

(c)

Mpt

truelabelsyt

Performer

ab

Linear

1A(t)≥0a

predicted

labelsyt

Performer

a'=a+

Linear

b'=b-

Performer

3

L2

A(t)

A(t)

Perf

inear:

nominality

了

xC

Induce

scoreN(t)

Linear

Linear

90N

d

x

Performer

Performer

L2

→△x2→

Perf

D

Q6

dw

t

ab

Mseq

不2

Mseq

Performer

Linear

L2

→/x12

denominator

4 Experiments

4.1 Datasets

We evaluate NPSR on the following datasets:

SWaT (Secure Water Treatment) [35]: The SWaT dataset is collected over 11 days from a scaleddown water treatment testbed with 51 sensors. During the last 4 days, 41 anomalies were injected using diverse attack methods, while only normal data were generated during the first 7 days.

WADI (WAter DIstribution testbed) [36]: The WADI dataset is acquired from a reduced city water distribution system with 123 sensors and actuators operating for 16 days. The first 14 days contain only normal data, while the remaining two days have 15 anomaly segments.

PSM (Pooled Server Metrics) [37]: The PSM dataset is collected internally from multiple application server nodes at eBay. There are 13 weeks of training data and 8 weeks of testing data.

MSL (Mars Science Laboratory) and SMAP (Soil Moisture Active Passive) [38, 13]: The MSL and SMAP datasets are public datasets collected by NASA, containing telemetry anomaly data derived from the Incident Surprise Anomaly (ISA) reports of spacecraft monitoring systems. The datasets have 55 and 25 dimensions respectively. The training set contains unlabeled anomalies.

SMD (Server Machine Dataset) [9]: The SMD is collected from a large internet company, comprising 5 weeks of data from 28 server machines with 38 sensors. The first 5 days contain only normal data, and anomalies are injected intermittently for the last 5 days.

trimSyn (Trimmed Synthetic Dataset) [24]: The original synthetic dataset was generated using trigonometric functions and Gaussian noises. We obtained the dataset from [39] and trimmed the test dataset such that only one segment of anomaly is present.

The statistics for the datasets are summarized in Table 1. For multi-entity datasets, Train#/Test# corresponds to the number of train/test time points summed over all entities, and the anomaly rate is calculated from the ratio between the sum of all anomaly points and sum of all test points.

4.2 Baselines

We evaluate the performance of NPSR against several deep learning algorithms and simple heuristics using F1 ∗ . Due to the exhaustive nature of optimizing for all datasets and algorithms, we follow a three-step approach to populate Table 2. Firstly, we reference values from the original paper, and if unavailable, we search for the highest reported values among other publications. Finally, if no reported values are found, we modify and run publicly available code. We cannot find any reported F1 ∗ of the PSM dataset using THOC or any publicly available code, hence leaving the value blank. For multi-entity datasets (MSL, SMAP, and SMD), we compare the performance using two methods - (1) combining all entities and training them together; (2) training each entity separately and averaging the results. Moreover, some literature attempt to find a threshold ( θ a ) and then calculate the performance conditioned on it [24, 40]. Since θ a can simply be a one-value parameter, we assume that other studies have already optimized this value, and regard their reported F1 as F1 ∗ . More information on the sources of data can be found in Appendix E.

4.3 Main Results

In Table 2, we report the results for F1 ∗ on several datasets. Detailed preprocessing steps and training settings are reported in Appendix C. NPSR almost consistently outperforms other algorithms, only being slightly inferior to TranAD on the PSM dataset. NPSR makes use of M pt to precisely capture point anomalies with low false-positive rates (given the best threshold). Moreover, it acquires the ability to detect contextual anomalies by incorporating M seq through the calculation of ˆ A ( · ) , without compromising its capability of detecting point anomalies. We observe that recent studies do not necessarily have higher F1 ∗ scores than older ones. Interestingly, simple heuristics can even perform fairly well on F1 ∗ , with NPSR being the only algorithm consistently outperforming them.

We suggest using these simple heuristic values as a strong baseline for future studies to compare against. In light of recent publications highlighting the limitations of using the point-adjusted F1 score ([11, 30, 31, 42]), and yet the large amount of work still using it, we also report the results using point-adjustment in Appendix D.

For multi-entity datasets, we observe that the standard method (training one point-based and sequencebased model per entity) outperforms the combined method for MSL and SMD datasets. This is not surprising, given that entity-to-entity variations might be large. However, for the SMAP dataset, we observe that the combined method performs better. We attribute such results to the fact that the SMAP spacecraft are routine, hence the resulting telemetry between entities can have similar underlying distributions. This contributes to additive learning from the increased training data [13].

4.4 Ablation Study

The ablation study conducted in this section sheds light on several aspects of the proposed method. In Table 3, we compare the performance of five methods that yield different anomaly scores (either A ( · ) or ˆ A ( · ) ). For the first two methods, the reconstruction errors of M pt and M seq are used, respectively. Since M pt mostly performs better than M seq , we use the point-based reconstruction error as A ( · ) to calculate ˆ A ( · ) for the last three methods. The third method corresponds to an unnormalized simple smoothed value of A ( · ) (cf. section 3.4). The fourth and fifth methods use gate functions (11) and (8), respectively, but the same θ N that corresponds to the 98.5 percentile of the nominality score from the training data ( N trn ). This method for setting θ N works well enough and can be applied across different datasets. Illustrations of the distribution of nominality scores for the SWaT and WADI datasets, along with the 98.5% threshold value are shown in Fig. 4. We can observe that the distributions are close to appropriate (cf. section 3.3). The results for the last three methods are averaged over induction lengths d = 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , and 256 . Each entity is trained separately for multi-entity datasets and the results are pooled together.

Firstly, we observe that M pt outperforms M seq and achieves competitive results on its own, despite not modeling the time-dependent relationships. Secondly, by smoothing A ( · ) (third row), the AUC and F1 ∗ are increased for most datasets. This means smoothing is generally an effective method for improving performance. Moreover, our experiments show that the soft gate function along with an appropriate θ N performed the best on average in terms of F1 ∗ . This suggests that the distribution of nominality scores is predominantly overlapped, and a soft gate function will be more appropriate to prevent excessive accumulation of anomaly scores on normal time points, reducing false-positives (cf. Appendix F). This method will also have a generally stable AUC and F1 ∗ (low σ d ) across a wide range of d . This makes sense - when time point τ is farther away from time point t , more gate function outputs are multiplied onto A ( τ ) , hence A ( t, τ ) → 0 . However, the results also suggest that the best choice of gate function and θ N may depend on the specific dataset at hand.

4.5 Detection Trade-off Between Point and Contextual Anomalies

We elaborate on the trade-off between detecting point and contextual anomalies and relate them with the performance of M pt and M seq . It may seem intuitive that finding temporal information in

SWaT

WADI

105

train

105

train

normal

normal

frequency

anomaly

frequency

anomaly

103

98.5% Ntrn

103

98.5% Ntrn

101

101

0

1

2

3

0

5

10

15

20

25

30

N

N

time series would lead to better performance. However, the modeling complexity increases with the number of time points being considered. This leads to the difficulty of focusing on the reconstruction of single time points. Fig. 5(a) shows A ( · ) calculated using either M pt or M seq , and ˆ A ( · ) calculated by NPSR using the WADI dataset. Given the fact that point t is predicted anomalous if A ( t ) ≥ θ a , we can see that M seq has a higher false positive rate due to the spikes. In comparison, M pt more accurately detects anomalies in a point-wise fashion. This highlights the superiority of M pt over M seq for this dataset. Furthermore, the induced anomaly score calculated by NPSR has very low false positive rates, and can sometimes even learn to identify anomalies that are not detected by M pt or M seq (Fig. 5(b), the anomaly subsequence at t = 15200 and the subsequence to its left, pointed by the black arrows). This suggests that ˆ A ( · ) is superior to the original A ( · ) for this dataset. However, M pt might not always perform superior to M seq . False negatives can be visualized between t = 14800 and t = 14900 , where M pt struggles to recognize the anomaly but M seq effectively detects anomalous time-dependent relationships. This suggests that the anomaly segment contains relatively more contextual than point anomalies. Since the reconstruction error of M pt is used as A ( · ) , we lose the advantage of effectively utilizing the reconstruction error of M seq . This results in ˆ A ( · ) not high enough to reach θ a within this segment. An important future direction would be to explore how to appropriately select A ( · ) among multiple models (cf. Appendix F).

(a)

point-based model

Mpt

(b)

Mpt

point-based model

best threshold

best threshold

true labels

truelabels

MULN

seq-based model

seq-based model

best threshold

best threshold

true labels

true labels

NPSR

Mpt + Mseq

Mpt + Msed

NPSR

best threshold

best threshold

true labels

true labels

0

2000

4000

6000

8000

10000

12000

14000

16000

14600

14800

15000

15200

15400

15600

t

t

5 Conclusion

In conclusion, we introduce an improved framework for unsupervised time series anomaly detection. We specify the relationships between point and contextual anomalies and derive the nominality score and induced anomaly score to provide a theory-based algorithm with provable superiority. NPSR captures both point and contextual anomalies, resulting in a high combined precision and recall. Our results show that NPSR exhibits high performance, is widely applicable, and has a relatively straightforward training process. It has the potential to decrease labor needs for fault monitoring and correspondingly accelerates decision making and can also contribute to AI sustainability by preventing energy waste or system failure.

References

Ane Blázquez-García, Angel Conde, Usue Mori, and Jose A Lozano. A review on outlier/anomaly detection in time series data. ACM Computing Surveys (CSUR) , 54(3):1-33, 2021.

Stéphane Crépey, Noureddine Lehdili, Nisrine Madhar, and Maud Thomas. Anomaly detection in financial time series by principal component analysis and neural networks. Algorithms , 15(10):385, 2022.

João Pereira and Margarida Silveira. Learning representations from healthcare time series data for unsupervised anomaly detection. In 2019 IEEE International Conference on Big Data and Smart Computing (BigComp) , pages 1-7, 2019.

Tiankai Chen et al. Anomaly detection in semiconductor manufacturing through time series forecasting using neural networks . PhD thesis, Massachusetts Institute of Technology, 2018.

Mohammad Bawaneh and Vilmos Simon. Anomaly detection in smart city traffic based on time series analysis. In 2019 International Conference on Software, Telecommunications and Computer Networks (SoftCOM) , pages 1-6, 2019.

Kukjin Choi, Jihun Yi, Changhwa Park, and Sungroh Yoon. Deep learning for anomaly detection in time-series data: Review, analysis, and guidelines. IEEE Access , 9:120043-120065, 2021.

Sebastian Schmidl, Phillip Wenig, and Thorsten Papenbrock. Anomaly detection in time series: A comprehensive evaluation. Proc. VLDB Endow. , 15(9):1779-1797, may 2022.

Yan Zhao, Liwei Deng, Xuanhao Chen, Chenjuan Guo, Bin Yang, Tung Kieu, Feiteng Huang, Torben Bach Pedersen, Kai Zheng, and Christian S Jensen. A comparative study on unsupervised anomaly detection for time series: Experiments and analysis. arXiv preprint arXiv:2209.04635 , 2022.

Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining , KDD '19, page 2828-2837, New York, NY, USA, 2019. Association for Computing Machinery.

Zhihan Li, Youjian Zhao, Jiaqi Han, Ya Su, Rui Jiao, Xidao Wen, and Dan Pei. Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery; Data Mining , KDD '21, page 3220-3230, New York, NY, USA, 2021. Association for Computing Machinery.

Astha Garg, Wenyu Zhang, Jules Samaran, Ramasamy Savitha, and Chuan-Sheng Foo. An evaluation of anomaly detection and diagnosis in multivariate time series. IEEE Transactions on Neural Networks and Learning Systems , 33(6):2508-2517, jun 2022.

Ailin Deng and Bryan Hooi. Graph neural network-based anomaly detection in multivariate time series. Proceedings of the AAAI Conference on Artificial Intelligence , 35(5):4027-4035, May 2021.

Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining , KDD '18, page 387-395, New York, NY, USA, 2018. Association for Computing Machinery.

Nan Ding, HaoXuan Ma, Huanbo Gao, YanHua Ma, and GuoZhen Tan. Real-time anomaly detection based on long short-term memory and gaussian mixture model. Computers & Electrical Engineering , 79:106458, 2019.

Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. Lof: Identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data , SIGMOD '00, page 93-104, New York, NY, USA, 2000. Association for Computing Machinery.

Bernhard Schölkopf, John C. Platt, John C. Shawe-Taylor, Alex J. Smola, and Robert C. Williamson. Estimating the support of a high-dimensional distribution. Neural Computation , 13(7):1443-1471, jul 2001.

Bo Liu, Yanshan Xiao, Longbing Cao, Zhifeng Hao, and Feiqi Deng. Svdd-based outlier detection on uncertain data. Knowledge and information systems , 34:597-618, 2013.

Lifeng Shen, Zhuocong Li, and James Kwok. Timeseries anomaly detection using temporal hierarchical one-class network. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 13016-13026. Curran Associates, Inc., 2020.

Haibin Cheng, Pang-Ning Tan, Christopher Potter, and Steven Klooster. A robust graph-based algorithm for detection and characterization of anomalies in noisy multivariate time series. In 2008 IEEE International Conference on Data Mining Workshops , pages 349-358, 2008.

Fan-Keng Sun, Chris Lang, and Duane Boning. Adjusting for autocorrelated errors in neural networks for time series. Advances in Neural Information Processing Systems , 34:29806-29819, 2021.

Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. Multivariate probabilistic time series forecasting via conditioned normalizing flows. arXiv preprint arXiv:2002.06103 , 2020.

Daehyung Park, Yuuna Hoshi, and Charles C Kemp. A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder. IEEE Robotics and Automation Letters , 3(3):15441551, 2018.

Dan Li, Dacheng Chen, Baihong Jin, Lei Shi, Jonathan Goh, and See-Kiong Ng. Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks. In Artificial Neural Networks and Machine Learning-ICANN 2019: Text and Time Series: 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17-19, 2019, Proceedings, Part IV , pages 703-716. Springer, 2019.

Chuxu Zhang, Dongjin Song, Yuncong Chen, Xinyang Feng, Cristian Lumezanu, Wei Cheng, Jingchao Ni, Bo Zong, Haifeng Chen, and Nitesh V Chawla. A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 1409-1416, 2019.

Shreshth Tuli, Giuliano Casale, and Nicholas R Jennings. Tranad: Deep transformer networks for anomaly detection in multivariate time series data. arXiv preprint arXiv:2201.07284 , 2022.

Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations , 2018.

Hang Zhao, Yujing Wang, Juanyong Duan, Congrui Huang, Defu Cao, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. Multivariate time-series anomaly detection via graph attention network. In 2020 IEEE International Conference on Data Mining (ICDM) , pages 841-850. IEEE, 2020.

Julien Audibert, Pietro Michiardi, Frédéric Guyard, Sébastien Marti, and Maria A. Zuluaga. Usad: Unsupervised anomaly detection on multivariate time series. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining , KDD '20, page 3395-3404, New York, NY, USA, 2020. Association for Computing Machinery.

Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, Jie Chen, Zhaogang Wang, and Honglin Qiao. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In Proceedings of the 2018 World Wide Web Conference , WWW '18, page 187-196, Republic and Canton of Geneva, CHE, 2018. International World Wide Web Conferences Steering Committee.

Siwon Kim, Kukjin Choi, Hyun-Soo Choi, Byunghan Lee, and Sungroh Yoon. Towards a rigorous evaluation of time-series anomaly detection, 2022.

Keval Doshi, Shatha Abudalou, and Yasin Yilmaz. Reward once, penalize once: Rectifying time series anomaly detection. In 2022 International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2022.

Siwei Guan, Binjie Zhao, Zhekang Dong, Mingyu Gao, and Zhiwei He. Gtad: Graph and temporal neural network for multivariate time series anomaly detection. Entropy , 24(6), 2022.

Enyan Dai and Jie Chen. Graph-augmented normalizing flows for anomaly detection of multiple time series. arXiv preprint arXiv:2202.07857 , 2022.

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 , 2020.

Jonathan Goh, Sridhar Adepu, Khurum Nazir Junejo, and Aditya Mathur. A dataset to support research in the design of secure water treatment systems. In Grigore Havarneanu, Roberto Setola, Hypatia Nassopoulos, and Stephen Wolthusen, editors, Critical Information Infrastructures Security , pages 88-99, Cham, 2017. Springer International Publishing.

Chuadhry Mujeeb Ahmed, Venkata Reddy Palleti, and Aditya P. Mathur. Wadi: A water distribution testbed for research in the design of secure cyber physical systems. In Proceedings of the 3rd International Workshop on Cyber-Physical Systems for Smart Water Networks , CySWATER '17, page 25-28, New York, NY, USA, 2017. Association for Computing Machinery.

Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery; Data Mining , KDD '21, page 2485-2494, New York, NY, USA, 2021. Association for Computing Machinery.

Dara Entekhabi, Eni G. Njoku, Peggy E. O'Neill, Kent H. Kellogg, Wade T. Crow, Wendy N. Edelstein, Jared K. Entin, Shawn D. Goodman, Thomas J. Jackson, Joel Johnson, John Kimball, Jeffrey R. Piepmeier, Randal D. Koster, Neil Martin, Kyle C. McDonald, Mahta Moghaddam, Susan Moran, Rolf Reichle, J. C. Shi, Michael W. Spencer, Samuel W. Thurman, Leung Tsang, and Jakob Van Zyl. The soil moisture active passive (smap) mission. Proceedings of the IEEE , 98(5):704-716, 2010.

Sadaf Tafazoli and Eamonn Keogh. Matrix Profile XXVIII: Discovering Multi-Dimensional Time Series Anomalies with <italic>K</italic> of <italic>N</italic> Anomaly Detection<sup>†</sup> , pages 685693.

Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. arXiv preprint arXiv:2110.02642 , 2021.

Zekai Chen, Dingshuo Chen, Xiao Zhang, Zixuan Yuan, and Xiuzhen Cheng. Learning graph structures with transformer for multivariate time-series anomaly detection in iot. IEEE Internet of Things Journal , 9(12):9179-9189, 2021.

Dennis Wagner, Tobias Michels, Florian C.F. Schulz, Maja Rudolph, and Marius Kloft. TimeseAD: Benchmarking deep time-series anomaly detection, 2023.

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006 , 2020.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

TimyadNyda. Github - timyadnyda/variational-lstm-autoencoder: Lstm variational auto-encoder for time series anomaly detection and features extraction.

Jun Zhan, Siqi Wang, Xiandong Ma, Chengkun Wu, Canqun Yang, Detian Zeng, and Shilin Wang. Stgatmad : Spatial-temporal graph attention network for multivariate time series anomaly detection. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3568-3572, 2022.

Markus Thill, Wolfgang Konen, and Thomas Bäck. MarkusThill/MGAB: The Mackey-Glass Anomaly Benchmark, April 2020.

A The Best F1 Score

This section describes the method for calculating the best F1 score ( F1 ∗ ) from a set of anomaly scores a = { a 1 , ..., a T } and a set of labels y = { y 1 , ..., y T } . Firstly, given a and some arbitrary threshold θ a , we can calculate ˆ y = { ˆ y 1 , ... ˆ y T } , where ˆ y t ≜ ✶ a t ≥ θ a . Secondly, ˆ y is used to calculate TP , FP , and FN , which corresponds to the sets of time points for true positives, false positives, and false negatives.

Thirdly, we calculate the precision ( P ) and recall ( R ), and then calculate the F1 score, which is the harmonic mean between R and P .

Finally, we calculate F1 ∗ by using the threshold that yields the highest F1 .

B Model Architecture

Performers (an improved variant of Transformers) are competitive in terms of execution speed compared with other Transformer variants [34, 43], hence we use them as the basic building block of our models. For both M pt and M seq , we use a linear layer with input and output dimensions equal to D as the token embedding layer, a fixed positional embedding layer at the beginning, a feature redraw interval of 1, and a tanh activation function immediately before the output. GELUs [44] are used as the activation layer for all linear layers. We do not change any other predefined activation layer inside Performers.

B.1 Performer-based autoencoder

For the Performer-based autoencoder M pt , the input with the shape ( batch, W, D ) is passed through a plain Performer with N perf layers after the positional embedding step, where W represents the input window size. This is followed by a linear encoding layer that transforms the dimensionality from D to D lat , resulting in the shape ( batch, W, D lat ) for the latent variables. In the case of M pt , there is no compression along the time domain. The latent variables then pass through another linear decoding layer that transforms the dimensionality from D lat back to D , followed by another Performer with N perf layers.

B.2 Performer-based stacked encoder

For the Performer-based stacked encoder M seq , the input with shape ( batch, W 0 , D ) is passed through a plain Performer with one layer after the positional embedding step, followed by a linear encoding layer that transforms the window size from W 0 to W 1 , where W 0 = 2 γ (cf. section 3.6). It should be noted that unlike M pt , compression is done along the time domain for M seq . The one-layer Performer and linear layer are stacked N enc times, where in the i -th stack, the window size is compressed from W i -1 to W i . W N enc is equal to the target output window size δ . For both M pt and M seq , we optimize N perf , D lat , W , N enc , W i for i ∈ { 0 , ..., N enc } , and δ for the best performance. Note that M seq isn't capable of reconstructing the first and last γ time points due to its architecture, hence we discard the first and last γ points reconstructed by M pt so that rest of the time points have exactly two reconstructed values corresponding to using M seq and M pt , respectively.

C Data Preprocessing and Training Details

Table 4 shows the hyperparameters used for implementing NPSR on the experimented datasets. To ensure fair comparison, the same preprocessing method is applied to all algorithms for the same dataset. The search for hyperparameters is done manually, starting from some reasonable value (e.g. a learning rate of 10 -4 ). The authors believe that there is still room for improvement by fine-tuning these hyperparameters. To speed up training, we load all training inputs and outputs, and testing inputs onto the GPU before training. We use a local GPU, which can be either GeForce RTX 3070 (8GB), 3080 (12GB) or 3090 (24GB). For an individual experiment using a single dataset and training method, the training time ranges from approximately 2 minutes to 12 hours. For single-entity datasets and multi-entity datasets that use the combined training method, we run the experiments for at least 3 times and confirm that the results are stable given different random seeds. For multi-entity datasets with entities trained individually, the results are averaged across all entities. Generally, datasets with single entities train faster than those with multiple entities. There are some additional remarks regarding the preprocessing of the datasets.

For SWaT, we use SWaT_Dataset_Attack_v0.csv and SWaT_Dataset_Normal_v1.csv from the folder SWaT.A1 & A2_Dec 2015 (manually converted from *.xlsx ). We corrected some original flaws in the dataset (e.g. redundant blank spaces in some labels), and set the 5th and 10th columns to all 0. For WADI, we use the 2017 year dataset. Columns with excessive NaNs (more than half of the entire length) are deleted. Other NaNs are forward-filled. After deleting all the columns with excessive NaNs, the 86th column is further set to all 0. For PSM, we forward-fill all NaNs. For MSL and SMD, two additional blank channels are added to make the number of channels divisible by the number of heads. For trimSyn, we separated the dataset into training ( t ∈ { 0 , ..., 9999 } ) and testing ( t ∈ { 10000 , ..., 19999 } ) data (cf. [24]). For the testing data, we extracted segments within the time intervals t ∈ { 10000 , ..., 11719 }∪{ 11900 , ..., 12849 }∪{ 14630 , ..., 17699 }∪{ 17880 , ..., 18529 }∪ { 18710 , ..., 19999 } and concatenated them to form a single-entity test dataset. We also inserted additional segment IDs using one-hot encoding to enable segment identification. This process resulted in a test dataset with a single anomaly segment ( t ∈ { 12670 , ..., 12849 } ) and a corresponding anomaly rate of 2 . 34% . In the case of the training data, we duplicated the original segment five times, concatenated them, and added the necessary segment IDs.

D The Point-adjusted Best F1 Score

Analogous to F1 ∗ , the point-adjusted best F1 score ( F1 ∗ PA ) corresponds to F1 ∗ calculated after pointadjustment. Table 5 shows the F1 ∗ and F1 ∗ PA of different algorithms, including NPSR, applied to several datasets. We did not show the results for the trimSyn dataset with point-adjustment. However, by applying NPSR and the same setting as without using point-adjust, we can achieve F1 ∗ PA = 1 within the first few epochs. This indicates the point with the highest anomaly score lies within the single anomaly segment.

The results suggest that F1 ∗ PA may not be reliable - on the SWaT, WADI, PSM, and MSL datasets, simple heuristic approaches (e.g. using the mean squared value of an input time point as the anomaly score) outperform all deep learning methods when evaluated using F1 ∗ PA . Moreover, optimizing on F1 ∗ PA does not necessarily guarantee a higher F1 ∗ . NPSR is optimized on F1 ∗ by tuning the algorithm-specific parameters (e.g. d ) and general parameters. To additionally optimize on F1 ∗ PA , we simply added spikes to the induced anomaly score ( ˆ A spike ( · ) ) with value ∞ for some fixed interval s . The results show that ˆ A spike ( · ) can also achieve competitive F1 ∗ PA values.

E Source of Data

Table 6 shows the data sources used to produce Table 5, as well as the sources for section 4.3. The reference number is followed by a number between 1 and 3, where 1 indicates that the data comes from the original work, 2 indicates that the data comes from reproduced values of another literature, and 3 indicates that we have reproduced the values using public repositories..

F Model and Parameter Selection Heuristics

We believe that the most important idea in model selection is to achieve a balance between effectively utilizing both point and sequence-based models and carefully selecting the appropriate parameters. As an extreme example, we optimize our algorithm on the Mackey-Glass anomaly benchmark (MGAB) [47], which is a univariate time series dataset, and find that sequence-based models can significantly outperform point-based models when using their reconstruction errors as A ( · ) . As in Fig. 6, we see that M seq can correctly identify the anomalies, whereas M pt did not learn at all. This is reasonable, as point-based models only consider a single time point. In general, the lower the dimensionality of the dataset, the harder it gets for a point-based model to learn statistically meaningful representations. Moreover, looking at the high-dimensional datasets reported in Table 3, the induced anomaly score remains important for some datasets: For the MSL dataset, F1 ∗ improves 0.1 compared to only using point-based reconstruction. In Fig. 7b, when using a soft gate function, F1 ∗ improves 0.047 compared to using the point-based anomaly score. Since the amount of improvement depends on the statistical structure of the test data, it is still useful to consider using both M seq and M pt in general.

The difficulty in choosing the parameters for unsupervised time series anomaly detection stems from the absence of anomalies in the training dataset. The selection of soft or hard gate functions, the value for d , and the ratio for θ N largely depend on how we presume the anomaly will occur based on domain knowledge. For instance, d controls the distance that anomaly scores may propagate. This value should be higher if we presume the average anomaly length is long and vice versa. If

(a)

(b)

MGAB-NPSR

MGAB-NPSR

0.5

0.9

0.4

0.8

score

0.3

Mpt

Mseq

0.7

Mseq

Mpt

0.2

0.6

0.1

0.5

0.0

0

20

40

60

80

100

0

20

40

60

80

100

the anomaly is expected to occur abruptly and significantly, there would be a clear gap between the distribution of nominality scores for normal and anomaly data (Fig. 8a). In this case, a hard gate function should be chosen as it allows anomaly scores to propagate through time points without reduction, as long as an anomaly time point has a nominality score lower than θ N . Conversely, if the anomaly occurs progressively, the distribution of nominality scores is likely to overlap (Fig. 8b). Here, a soft gate function will be more appropriate to prevent excessive accumulation of anomaly scores on normal time points, reducing false-positives. A dataset could contain both abrupt and progressive anomalies. However, based on Table 3, it is evident that using a soft gate function generally yields better performance compared to a hard gate function. This suggests that the distribution of nominality scores is predominantly overlapped, which is also evident in Fig. 4 and Fig. 7a.

(a)

(b)

(c)

(d)

101

Normal

Anomaly

98.5% 0Ntrn

Nominality score

100

10-1

hard

hard

soft

soft

hard

point

point

point

10-3

10-2

10-1

0.625

0.650

0.675

0.01

0.02

0.63 0.64 0.65

Anomaly score (point-based)

best F1 (d=16)

FPR (d=16)

best F1 (d=1)

(a)

hard gate

(b)

soft gate

func.

func.

nominality

nominality

score

score

个

normal

data

normal

threshold

data

anomaly

anomaly

data

data

1

0

1

0

G Broader Impacts

The detection of anomalies in time series data can minimize downtime and avert financial losses. Utilizing real-time monitoring of system conditions, anomaly detection techniques for time series data can automatically detect deviations from the expected system behavior, thereby avoiding potential risks and financial harm. This has the potential to reduce the need for manual monitoring of faults and to expedite decision-making processes. Additionally, it can promote the sustainability of AI by preventing energy wastage and system malfunction.

H Limitations

Despite exhibiting competitive performance against other models, the proposed NPSR algorithm has a few limitations. The point-based model used in training does not incorporate temporal information, which makes it challenging to effectively reconstruct low-dimensional datasets. This issue is particularly challenging for univariate time series since raw inputs would not work for point-based models. One possible solution to this problem is to increase dimensionality by aggregating multiple time points. However, the effectiveness of this approach is yet to be confirmed.

Another limitation of NPSR is the absence of an automatic threshold ( θ a )-finding method, which makes it difficult to determine a suitable threshold when deploying the model. To address this issue, one can define a target false positive rate and estimate the threshold that achieves this target rate using the validation set since only normal data is needed. Similarly, estimating the optimal values for θ N , d , and selecting the model used for calculating A ( · ) will be an important future work.

Within Fig. 5b of the main text, a false negative instance was identified in the temporal range between t = 14800 and t = 14900 when employing the induced anomaly score. According to the WADI dataset, this anomaly spans approximately 14.26 minutes and is characterized as "Damage 1 MV 001 and raw water pump to drain Elevated Reservoir tank." Notably, our analysis suggests that when assessing individual time points, the model M pt encounters difficulty in recognizing this anomaly. Conversely, the model M seq excels in identifying time-dependent relationships, making it more effective in capturing such contextual anomalies. The observed disparity in anomaly detection implies that this section comprises a relatively higher proportion of contextual anomalies than point anomalies. Consequently, when using M seq , we achieve a higher anomaly score. However, our current approach utilizes the reconstruction error of M pt as the basis for the anomaly score calculation, thus neglecting the effectiveness of the reconstruction error generated by M seq . Consequently, the induced anomaly score fails to surpass the predefined threshold. In light of these findings, an essential avenue for future research is to investigate methods for selecting the model to be used in the computation of A ( · ) . This undertaking holds promise for enhancing the overall performance and accuracy of anomaly detection in time series data.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTSAD techniques are targeted towards detecting either point or contextual anomalies, but often struggle to adequately capture both simultaneously. In this work, the authors propose a novel reconstruction-based AD technique that introduces the notion of a nominality score and a subsequent induced anomaly score. The method achieves a better trade-off between detecting point and contextual anomalies than competing techniques, and outperforms them over a variety of benchmark multivariate datasets.\n\nSTRENGTHS:\n- The problem statement is interesting and relevant, and I appreciate the use of visualisations such as in Figure 1, that convey the difficulty of the task.\n- I appreciated that the model has several components that could be tweaked, such as the choice of soft or hard gate function, and the ablation study does a good job in highlighting their individual contributions. I would however have liked to see more discussion or synthetic examples that demonstrate whether there are certain properties of the monitored data that can guide the selection (beyond empirical comparison).\n\nWEAKNESSES:\n- I found the presentation and clarity to be quite poor overall, as a result of which the contributions are sometimes difficult to follow properly.  Section 2.2 introduces a lot of similar notations that are difficult to keep track of – a figure could be very helpful here for conveying the differences between the various time series and deviations.\n- The authors rely on results reported in earlier papers as performance measures for several competing techniques. While I appreciate that it may be time-consuming to re-implement and re-run all experiments, I do worry about potential inconsistencies in the experimental set-up that may involuntarily skew the comparison. The authors mention how if the results from the original paper are unavailable, *“we search for the highest reported values among other publications.”* I can see this as introducing several inconsistencies, especially since the source of the result isn’t reported alongside the performance figures presented in the table.\n- I would have liked there to be a dedicated *Related Work* section in the main paper in order to better understand how the work fits alongside other reconstruction-based techniques.\n\nQUESTIONS:\nI would appreciate if the authors could respond to the concerns listed in the *Weaknesses* section, and either defend their approach, or indicate how they can improve on the existing submission. \n\nAlthough I find the problem investigated in the paper to be interesting and well-motivated, I don’t think that the submission is ready for publication in its current state, and believe it would strongly benefit from another revision/round of reviews.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an unsupervised time series anomaly detection algorithm called NPSR (Nominality score conditioned TSAD by Point/Sequential Reconstruction), which combines both point-based and sequence-based reconstruction models. Specifically, it proposes a nominality score which is the ratio of a contextual deviation (or in-distribution deviation) to the total deviation (which is assumed to be the sum of the in-distribution deviation and the out-of-distribution deviation). The contextual deviation and the in-distribution deviation is computed by using the sequence-based and point-based reconstruction models. Based on the nominality score and anomaly score (computed using point-based reconstruction model) computed in the neighborhood, the induced anomaly score is further proposed by considering the temporal relationship. Some theoretical results of the proposed algorithm are provided. Experiments on several benchmark time series anomaly detection datasets are performed to demonstrate the performance of the proposed algorithm in comparison with several state-of-the-art algorithms.\n\nSTRENGTHS:\n1.\tThis paper studies an important and interesting problem, i.e., how to detect anomalies in time series data without label data. \n2.\tThe proposed algorithm combines both point-based and sequence-based reconstruction models, which achieves quite good performance on several time series anomaly detection benchmark datasets. \n3.\tSome theoretical results are provided for the proposed algorithm.\n4.\tThe paper is generally well written and the presentation is clear.\n\nWEAKNESSES:\n1.\tOverall, this paper proposes a heuristic-based unsupervised time series anomaly detection algorithm. The overall pipeline in Algorithm looks Ok to me, but from experiments it seems the most important part is the point-based reconstruction. In other words, the rest of the proposed algorithm may be simplified. The ablation studies in Table 3 also partially confirms it.\n2.\tFlawed experiments. The F1 scores reported in Table 2 for other algorithms, e.g., Anomaly Transformer, are not consistent with results reported in the literature, such as [33]. More justifications are required. \n3.\tThe theoretical results look sound, but may not be useful in practice. In other words, can you justify the value of these results in real-world time series anomaly detection applications?\n\nQUESTIONS:\n1.\tIn Table 2, the performance of several algorithms, e.g., Anomaly Transformer, is significantly lower than those reported in the literature. More justification is required. In Figure 4(b), the anomalies on the right of t=14800 are not detected by $M_{pt}+M_{seq}$, but detected by $M_{seq}$, resulting in false negative. Can you provide some explanations?    \n2.\tThe proposed algorithm is an unsupervised algorithm. Thus, how to adapt the algorithm on different datasets remains not clear to me (besides the threshold used in adjusting the F1).\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel idea to handle the time-series anomaly detection problem by calculating the \"nominality score\"(concerns on this name are in the question part and I'll keep using this name in the following review) and the induced anomaly score. And the F1 score can be mathematically proved to improve using the proposed scores. The authors then constructed the point-based and sequence-based reconstruction models to estimate the anomaly score and experiments on several datasets demonstrate the superior performances.\n\nSTRENGTHS:\n1. The proposed method is novel and provides a new angle to address the anomaly detection problem. Without directly considering how \"abnormal\" a point is, this work start with thinking what is \"normal\" and induce anomality score based on the nominality score. \n2. The authors provided both theoretical and experimental supports for the proposed methods which are reasonable.\n3. The writing of the paper is clear and easy to follow.\n\nWEAKNESSES:\n1. A major concern is on the expectation of improvement by using $\\hat A(\\cdot ;g_{\\theta_2})$. While the Claim 1 and Claim 2 as well as the proofs gives a guarantee that using this method won't result in a worse F1 score, it says nothing about the gap with/without the method. Intuitively, the expected improvement may be related to $\\alpha$ in Eq. (2), $d$, the function, the number of data, and other factors. The actual case could be that even if the $\\alpha$ is larger than 1 by a margin (say \\alpha=1.5, 2, or 5), the expected number of $N(t)\\leq \\theta$ for abnormal points are still a small portion of the whole dataset, making the proposed method useless in practical use. The analysis of a bound or expectation is missing.\n2. The correctness of the method relies on several hypotheses, typically: 1) distribution of abnormal points $\\Delta x_{t,a}^p$ has a larger variance than that of normal points $\\Delta x_{t, n}^p$. This is used to guarantee that nominality score of normal points are larger than abnormal points; 2) the threshold is well selected so that the nominality score of all the normal points (with $y_t=0$) passes the threshold, while a significant number of abnormal points cannot pass the threshold. While the hypotheses are reasonable theoretically, I wonder whether they still hold when the scores are estimated and calculated based on the outputs from neural networks.\n\nQUESTIONS:\n1. I wonder whether the word \"nominality\" is a misuse, although it is presented across the whole paper. The antonym to the word \"anomaly\" is \"normality\", and \"nominality\" is the noun of \"nominal\" which means \"existing in name only; far below the real value or cost\", and I think it's not related to the topic at all.\n2. How is the backbone of the model (i.e., the performer-based autoencoder) chosen? Will that cause an unfair comparison between the proposed method and the baselines due to the neural model's approximation ability?\n3. The supplementary file mentions how a best threshold is chosen, but in 3.4 ablation study the threshold is set to 98.5% of the nominality score from the training data. Why and how will the choice of different threshold influence the performance (i.e., whether the method is sensitive to the choice of hyperparameters?)\n\nLIMITATIONS:\nNo apparent limitations and negative societal impact observed by the reviewer.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBasically, this paper aims to consider anomalousness of data points both from a point perspective, which is independent of the temporal relationships in the data, and contextual perspective, which reflects temporal relationships in the data. The paper derives an induced anomaly score that considers both.\n\nSTRENGTHS:\nThis paper addresses an important problem of considering both point and contextual anomalies. The paper provides good theoretical background for their work and convincing experimental results, showing particular anomalies that their method finds that are not found by methods that look for either point-based or contextual anomalies.\n\nWEAKNESSES:\nThe datasets on which testing is performed have rather high anomaly rates. The authors should experiment with some datasets with much lower anomaly rates, perhaps by leaving out some anomalies in the datasets that they use.\n\nPost rebuttal comment:\nThanks to the authors for the responses. While the anomaly rates that the authors use are worth testing, it is quite common to use lower anomaly rates, and such lower rates would constitute a better test of the performance of the new algorithm. With regard to removing anomalies from datasets, one can remove individual time series that are identified as anomalous.\n\nQUESTIONS:\nAt the start of section 2.2, $\\mathbf{x}_^0$ is part of the observed dataset but is also defined to be $\\mathbf{x}_t^c + \\Delta\\mathbf{x}_t^p$. This cannot be correct. Did you mean to define $\\delta\\mathbf{x}_t^p$?\n\nLIMITATIONS:\nNo limitations are described on the work. Societal impact material is not relevant here. However, a description of future work should be provided based on any patterns in the errors that the presented algorithm makes.\n\n---\n\nPost rebuttal addition:\nThanks to the authors for pointing out the discussions on limitations in the supplementary material. However, such material is critical to understanding the nature of the contribution, and so needs to be in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}