Title: SARAD: Spatial Association-Aware Anomaly Detection and Diagnosis for Multivariate Time Series
Venue: NeurIPS 2024
Accepted: yes
Keywords: multivariate time series

anomaly detection

anomaly diagnosis

spatial associations
Abstract: Anomaly detection in time series data is fundamental to the design, deployment, and evaluation of industrial control systems. Temporal modeling has been the natural focus of anomaly detection approaches for time series data. However, the focus on temporal modeling can obscure or dilute the spatial information that can be used to capture complex interactions in multivariate time series. In this paper, we propose SARAD, an approach that leverages spatial information beyond data autoencoding errors to improve the detection and diagnosis of anomalies. SARAD trains a Transformer to learn the spatial associations, the pairwise inter-feature relationships which ubiquitously characterize such feedback-controlled systems. As new associations form and old ones dissolve, SARAD applies subseries division to capture their changes over time. Anomalies exhibit association descending patterns, a key phenomenon we exclusively observe and attribute to the disruptive nature of anomalies detaching anomalous features from others. To exploit the phenomenon and yet dismiss non-anomalous descent, SARAD performs anomaly detection via autoencoding in the association space. We present experimental results to demonstrate that SARAD achieves state-of-the-art performance, providing robust anomaly detection and a nuanced understanding of anomalous events.

SARAD: Spatial Association-Aware Anomaly Detection and Diagnosis for Multivariate Time Series

Abstract

Anomaly detection in time series data is fundamental to the design, deployment, and evaluation of industrial control systems. Temporal modeling has been the natural focus of anomaly detection approaches for time series data. However, the focus on temporal modeling can obscure or dilute the spatial information that can be used to capture complex interactions in multivariate time series. In this paper, we propose SARAD, an approach that leverages spatial information beyond data autoencoding errors to improve the detection and diagnosis of anomalies. SARAD trains a Transformer to learn the spatial associations, the pairwise inter-feature relationships which ubiquitously characterize such feedback-controlled systems. As new associations form and old ones dissolve, SARAD applies subseries division to capture their changes over time. Anomalies exhibit association descending patterns, a key phenomenon we exclusively observe and attribute to the disruptive nature of anomalies detaching anomalous features from others. To exploit the phenomenon and yet dismiss non-anomalous descent, SARAD performs anomaly detection via autoencoding in the association space. We present experimental results to demonstrate that SARAD achieves state-of-the-art performance, providing robust anomaly detection and a nuanced understanding of anomalous events.

1 Introduction

Time series anomaly detection is critical for industrial automation (Rieth et al., 2018), intrusion detection (Mathur and Tippenhauer, 2016), and healthcare sensing (Goldberger et al., 2000). Anomaly detection in these contexts is typically treated as an unsupervised learning problem, owing to the novelty of anomalies and the scarcity of labeled anomalies.

Temporal modeling is the mainstream basis of current time series anomaly detectors. By learning the dependencies between discrete time steps, temporal modeling can pinpoint the time spans of anomalies. During anomalies, unseen and peculiar temporal dependence patterns degrade data autoencoding (Wang et al., 2023b) or autoregression (Zhao et al., 2020) performance, thereby enabling detection. Alternatively, irregular temporal representations can be driven to breach learned enclosing

Ligang He ∗

Matthew Leeke

time range: [68964, 69639]

feature #12

feature #15

(a) Raw time series before, during, and after an anomaly p i .

#12

#15

features

#12

#15

features

range: [0.0032, 0.1190]

(b) Pre-

p

i

's

A

.

#12

#15

features

#12

#15

features

range: [0.0032, 0.1190]

(c) In-

p

i

's

A

.

#12

#15

features

#12

#15

features

range: [0.0032, 0.1190]

(d) Post-

p

i

's

A

.

#12

#15

features

#12

#15

features

range: [0.0000, 0.0398]

(e) Pre-reduction.

#12

#15

features

#12

#15

features

range: [0.0000, 0.0398]

(f) Post-reduction.

Figure 1: Spatial associations captured by Transformer on a service monitoring benchmark. 1a shows the raw time series right before, during, and after an anomaly (colored in red). Association mapping A L h by final L -th layer's MHSA are averaged across heads to derive A before (1b), during (1c), and after (1d) the anomaly. Darker cells have larger values. Anomalous features (#12 and #15) are highlighted with red bounding boxes. The reduction-only changes from before or after the anomaly to during the anomaly are shown in 1e and 1f, i.e., ReLU ( A pre -A in ) and ReLU ( A post -A in ) . The anomaly leads to association reductions on anomalous features, prominently column-wise on A .

hyperspheres (Shen et al., 2020), resulting in high anomaly scores that enable detection. Despite its temporal precision in anomaly detection, temporal modeling either assumes feature independence or combines variables of diverse physical nature, the former simplifying the modeling and the latter mitigating the multicollinearity issue. Such assumptions lead to either omission or dilution of spatial information crucial to anomaly detection. Specifically, it overlooks the long-time-range spatial associations, the relationships between various features which ubiquitously characterize normal behaviors of multivariate time series. Where anomaly detection pinpoints the temporal locations of an anomaly, anomaly diagnosis identifies the spatial locations, i.e., the anomalous feature set, of an anomaly. Temporal methods also restrict diagnostic capabilities, as the lack of spatial information mismatches autoencoding-based or autoregression-based anomaly criterion, which de facto measures temporal novelty, with its objective of capturing spatial novelty.

Furthermore, time series anomalies frequently dissolve spatial associations, motivating anomaly detection in the association space. Using an vanilla Transformer (Vaswani et al., 2017), we investigate the changes in spatial associations throughout anomalies. Applied on transposed time windows (the spatial dimension comes before the temporal), an encoder-only Transformer is trained to minimize reconstruction errors on unlabeled N -variate time series and, by doing so, learns to model the multivariate series spatially via the Multi-Head Self-Attention (MHSA) illustrated in Figure 2. MHSA at each stacked l -th layer computes an intermediate association mapping A l h ∈ R N × N per h -th head, mapping back input X to produce attention scores. The last layer's mapping A L h thus effectively captures the contributions of k -th feature to the reconstruction of j -th feature at each location ( j, k ) , not least for its architectural proximity to the reconstructed output. Recent research (Liu et al., 2024) also highlights the important role MHSA plays in capturing the inter-feature associative relationships when applied on the multi-variate dimension. As new associations emerge and old ones dissolve over time, Figure 1 shows the association changes on a real-world benchmark. We observe

Linear

Linear

Softmax

Linear

Linear

Transpose

Dot Product

Dot Product

that anomalies exhibit reductions for anomalous features, a phenomenon we herein coin as Spatial

Association Reduction (SAR) . The rationale is that anomalies either originate from or result in dissolution of pre-existing associations, detaching anomalous features from their non-anomalous counterparts. Additionally, we make the observation that SAR is most prominent column-wise on A L h , since each j -th column characterizes the dropouts of j from associating with other, mostly non-anomalous, features. Due to lack of explicit spatial information, temporal modeling is inadequate for exploiting SAR. More examples are given in Appendix C.

From a spatial modeling perspective, we propose SARAD to leverage spatial information and to exploit SAR for enabling robust time series anomaly detection and diagnosis. For quantifying anomalous spatial novelty in the data space, we train a Transformer on transposed time windows as an autoencoder. To capture the spatial association progression, the reduction-only changes of associations over time, the data reconstruction divides the input window by time into two halves to be processed in parallel. Consequently, the progression is the non-negative backward difference of the intermediate association mappings via MHSA. Subseries division circumvents memory storage of latest association mappings and enables time window shuffling during training, which reduces order bias, enhances generalization, and prevents catastrophic forgetting. For quantifying anomalous reduction novelty, we train a Multi-Layer Perceptron (MLP) as an antuencoder on progression in the association space. Whereas progression encompass all association reduction, autoencoding rules out those not caused by anomalies. The reconstruction errors via the data module measure data-only anomalous deviation from expected system behaviors and falter when such deviations are not prominent, e.g., at the start of an anomaly. The reconstruction errors via the progression module are sensitive to change in spatial associations, thus complementing the former. We develop a joint anomaly detection criterion that combines both. Experiments show SARAD delivers stateof-the-art detection and diagnosis performance with architectural elegance. Code is available at https://github.com/daidahao/SARAD/ . We summarize our contributions as follows.

We reveal and extract spatial association descending patterns of time series anomalies with a bespoke Transformer and subseries division. The former learns the pairwise inter-feature associations via autoencoding in the data space and the latter enables shuffled autoencoding training and memory-less progression aggregation.

We propose progression autoencoding to quantify anomalous descent in the association space and a joint detection criterion in both data and association spaces, which complement each other.

Experimentally, SARAD performs state-of-the-art anomaly detection and diagnosis on multivariate time series and ablation studies support our design choices.

2 Related Work

Influenced by the dominance of temporal modeling in time series forecasting (Wang et al., 2023a; Zhang et al., 2023; Wu et al., 2021), temporal modeling is also prevalent in time series anomaly detection. Recurrent neural networks such as LSTM (Hochreiter and Schmidhuber, 1997) have innate capabilities for handling sequential data. These approaches use hidden states for past input memorization, enabling detection (Li et al., 2019; Malhotra et al., 2015) and diagnosis (Qian et al., 2021). Transformer (Vaswani et al., 2017) network is widely adopted (Fan et al., 2023; Xu et al., 2022) approach that is commonly applied to model temporal associations between different time points using its attention mechanism. Linear regression (Zeng et al., 2023) and MLP (Wang et al., 2024; Audibert et al., 2020) directly model temporal dependencies. TranAD (Tuli et al., 2022) replaces the MLP in Audibert et al. (2020) with a Transformer, making the detection criterion more robust through its adversarial training paradigm. Temporal modeling, however, is restricted by the exceptionally small receptive field in time and adversely impacted by the timestamp misalignment across features. In the context of anomaly detection, temporal modeling helps capture anomalous temporal associations (Xu et al., 2022; Yang et al., 2023), but offers limited detection capabilities in absence of spatial information. In a diagnostic context, temporal detectors mismatch anomaly criterion of temporal novelty with spatial interpretation.

Spatial associations characterize the multivariate time series commonly found in such supervisory systems for industrial control. The relationships range from strongly correlated, e.g., due to spatial proximity, to fully independent, e.g., due to mechanical disconnection. For forecasting, iTransformer (Liu et al., 2024) applies Transformer on the transposed time series to enable direct spatial

modeling. Crossformer (Zhang and Yan, 2023) screens the time series through custom Two-Stage Attention layers for more efficient spatial modeling. In terms of detection, GDN (Deng and Hooi, 2021) learns a directed graph of features for the prediction of last time points, whose errors serve as anomaly scores. GDN is partially limited by a mismatch between its singe-timestamp prediction target and the prevalent range-wise anomalies as well as unstable Top-K node selection during training. InterFusion (Li et al., 2021) learns compressed spatial and temporal dependencies, using a hierarchical Variational Auto-Encoder (Kingma and Welling, 2014) to reconstruct the series. Neither inspects temporal changes in associations throughout anomalies.

On another front, Isolation Forest (IF) models build a binary decision tree ensemble by partitioning either the data space (Liu et al., 2008) or the deep embedding space (Xu et al., 2023) formed by randomized neural networks. They are constrained by the lack of temporal and spatial (in the former case) or spatial (in the latter case) information, and their anomaly scores are not reflective of the degrees of anomalies.

We emphasize anomalous association descending patterns towards better time series detection and diagnosis. Different from previous work, we explicitly utilize the reduction in spatial associations over time during an anomaly, an insight we derived from the cyber-physical defense space. Dynamic watermarking (Satchidanandan and Kumar, 2017) and similar defense techniques (Dai et al., 2023) overlay actuation with private signals to reveal attacks resulting in correlational breakdowns. While their approaches are intrusive and actively alter system behaviors, our detector remains non-intrusive, passively monitors the spatial associations, and is applicable to any supervisory system.

We refer to spatiality in this work as the multi-dimensional vector nature inherent to multivariate time series data. The terminology is also used in literature on time series related tasks (Gangopadhyay et al., 2021; Zheng et al., 2023). We note that spatiality may carry different meanings in other AI contexts, such as geographic positions or characteristics on Earth. We differentiate those meanings from our definition of spatiality, which traces its root to the spatial distribution of sensors and actuators in control systems where time series are routinely collected.

3 Method

The problems of anomaly detection and diagnosis are specified as follows.

Anomaly Detection Given a N -feature time series T = { x 1 , · · · , x N } where x n ∈ R T is of the same length T , the objective is to predict the anomaly label y t ∈ { 0 , 1 } at each timestamp t .

Anomaly Diagnosis Given the same time series T , the objective is to predict the diagnosis label g t ⊆ [ N ] , the set of anomalous features at each timestamp t .

3.1 Overview

SARAD comprises two sequential modules; a Transformer for time series data reconstruction and a MLP network for spatial progression reconstruction. Table 1 decomposes the system framework of SARAD. The Transformer temporally divides by 2 and reconstructs the input time series to learn pairwise inter-feature associations and to enable order-free memory-efficient progression aggregation. The MLP reconstructs the aggregated progression to quantify anomalous association reduction while dismissing non-anomalous reduction. Towards robust anomaly detection, reconstruction errors from the two modules jointly serve as a criterion, sensitive to data-only anomalous deviation and anomalous association reduction.

3.2 Data Reconstruction

In light of restricted capabilities of temporal detectors, here we adapt Transformer to spatially reconstruct the series data. The data module contains two components, Subseries Split & Merge and Subseries Reconstruction, shown in the first and second columns in Table 1. The former wraps around the second by temporally splitting a multivariate input series in half at its beginning and temporally merging at its end. Subseries division enables capturing of spatial progression within a single time window. Without the former, the model must store in memory the last association mappings at each step and keep to the time ordering during training, which is prone to overfitting and

Data Reconstruction

Progression Recon.

Subseries Split & Merge

Subseries Recon.

Architecture

-

Encoder-only Transformer

MLP

Diagram

Subseries Split:

Subseries Merge:

Time

Feature

Embedding

MLP

Linear

Encoder

MHSA

ReLU

Aggregate

MLP

Training Objective

min

||

ˆ

X

-

X

||

2

2

min

||

ˆ

S

-

S

||

2

2

Detection Criterion

||

ˆ

X

-

X

||

2

2

||

ˆ

S

-

S

||

2

2

Diagnosis Criterion

||

ˆ

X

(

j,

·

)

-

X

(

j,

·

)

||

2

2

||

ˆ

S

(

·

,j

)

-

S

(

·

,j

)

||

2

2

catastrophic forgetting. The latter is an encoder-only Transformer composed of an embedding layer, a L -layer attention-based encoder, and finally a linear projection layer. Encoder-only Transformers are commonly found in Transformer-based detectors (Kang and Kang, 2024; Kim et al., 2023) due to its simplicity and the length uniformity of the target output, i.e., the reconstructed series. Ours exclusively models spatial associations, unconventional to the aforenamed detectors and most temporal forecasters (Nie et al., 2023; Zhou et al., 2022; Liu et al., 2022) and yet more aligned with recent spatial-aware forecasters (Liu et al., 2024; Zhang and Yan, 2023). At each encoding layer, MHSA computes pairwise association mappings, a representation of inter-feature dependencies which ubiquitously characterize the multivariate time series and are crucial to anomaly detection.

Subseries Split and Merge We suppose the input series is a time window X ∈ R 2 W × N of length 2 W , where 2 is for the convenience of a temporal split. Before reconstruction, X is split into two half multivariate subseries of equal temporal length: X = { X 1 ∈ R W × N , X 2 ∈ R W × N } . After subseries reconstruction, the two reconstructed subseries are concatenated to form the full reconstructed ˆ X = { ˆ X 1 ∈ R W × N , ˆ X 2 ∈ R W × N } .

Embedding To lead subseries reconstruction, each X i is embedded as X 0 i = E i + M , wherein E i = Linear ( X T i ) ∈ R N × D and M = { m i ∈ R D | i ∈ [ N ] } is a learnable feature-level embedding.

Spatial-Aware Encoding A stack of L Transformer encoding layers is used to encode the series in the D -length attention space. Each layer is stacked with MHSA and MLP with residual connections:

where X l -1 i , Z l i , X l i ∈ R N × D are l -1 layer's output, l -th layer's hidden state and output respectively and LN ( · ) is Layer Normalization (Ba et al., 2016). It is spatial-aware because MHSA explicitly learns a pairwise inter-feature association mapping to exchange information in between feature

representations. Notably within the MHSA with H heads as shown in Figure 2, each ( j, k ) -th element on the association mapping A h ∈ R N × N of its h -th head computes how much j -th feature's attention scores should originate from the k -th feature's key. From a broader perspective of data reconstruction, it is the quantification of the residual impact of k -th feature's originals on the j -th feature's reconstructions. We refactor MHSA implementation to enable parallel encoding of subseries.

Linear Projection Output from the last encoding layer X L i is linearly projected and transposed to derive the reconstructed subseries ˆ X i ∈ R W × N .

3.3 Spatial Progression Reconstruction

To exploit SAR caused by anomalies, the module first extracts and aggregates the spatial progression, the non-negative backward difference of the association mappings via MHSA. In line with general anomaly detectors (Aggarwal, 2013), the module conducts autoencoding in the association space to quantify anomalous SAR and to dismiss non-anomalous SAR. Anomalous SAR occurs when, say, a compromised sensor's readings are no longer correlating with its spatially adjacent or mechanically related counterparts.

Association Progression We define association progression S l h ∈ R N × N at the h -th attention head in the l -th layer to be the non-negative backward difference in association mappings { A l 1 ,h , A l 2 ,h } :

where ReLU ( · ) passes through only non-negative values and outputs zeros otherwise.

Progression Aggregation To center the detection on association dropouts, we aggregate the column sums of progression S l h from all attention heads in the final L -th layer to form S ∈ R H × N :

We recall from the data module, each k -th column in A l i,h quantifies the impact of k -th feature on all features' reconstruction. Taking the sum per each k -th column, we measure with S the dropout rates of k from participating all features' reconstruction. As we have observed in Section 1, SAR at the column level is indicative of time series anomalies, more so than at the row level. The last layer's progression is focused not least for its proximity to the final reconstructed output, whereafter no more information is exchanged between features. Liu et al. (2024) manifests that final layer's mappings resemble closely with the inter-feature correlations of the target, in our case, the reconstructed.

Autoencoding With S flattened as an one-dimensional vector, a 2-layer MLP is trained to output the reconstructed and reshaped ˆ S ∈ R H × N , synchronously with the data module training.

3.4 Joint Training and Anomaly Detection

Training Objective We train an end-to-end model with a joint minimization objective:

where L R is the data reconstruction loss, L S the progression reconstruction loss, and λ L S a weight hyper-parameter. Gradients are stopped from flowing into S to prevent updates to the data module and collapses in association representation. We are training two anomaly detectors simultaneously, one working in the original data space, the other in the spatial progression space.

Anomaly Detection Criterion For an input series X , the anomaly score s is a scalar defined to be:

where r is the data reconstruction error, p the progression reconstruction error, µ r , µ p the means of r, p on the validation set, and σ r , σ p the standard deviation of r, p . The criterion takes into account the normalized errors in the data space and the progression space, each of which quantifies the anomalous magnitude in respective spaces and complment the other.

Anomaly Diagnosis Criterion For an input series X and its j -th feature, its anomaly score s j is a scalar defined to be:

where r j is feature j 's data reconstruction error. The criterion is sensitive to spatial novelty.

4 Experiments

SARAD is compared against state-of-the-art detectors on real-world benchmarks for detection and diagnosis, the latter only when diagnostic labels are available.

4.1 Experimental Setup

Datasets We evaluate on four real-world datasets collected under industrial control and service monitoring settings. These dataset are: 1) Server Machine Dataset ( SMD ) (Su et al., 2019b,a), 2) Pooled Server Metrics ( PSM ) dataset (Abdulaal et al., 2021a,b) 3) Secure Water Treatment ( SWaT ) dataset (Mathur and Tippenhauer, 2016; iTrust, 2023), and 4) Hardware-In-the-Loop-based Augmented ICS ( HAI ) dataset (Shin et al., 2021b,a). All training sets contain only unlabeled data and the test sets contain data with anomaly labels. Anomalies range from service outages to external cyber-physical attacks. We summarize the statistics of the datasets in Table 2. Descriptions of each dataset are detailed in Appendix E.

Detection Metrics Real-world benchmarks are rife with range-wise anomalies spanning consecutive time points (Wagner et al., 2023). We use the range-based metrics proposed in (Paparrizos et al., 2022). Compared against their point-based counterparts, they provide robustness to labeling delay and scoring noises as well as performant detector separability and series consistency. We compute the threshold-independent AUC-ROC and AUC-PR scores to be rid of thresholding impact and fully parameter-free Volume Under the Surface (VUS) AUC-ROC and AUC-PR scores. Full details are discussed in Appendix I.

Diagnosis Metrics Consistent with previous works (Tuli et al., 2022; Zhao et al., 2020), we use common metrics such as Hit Rate (HR) (Su et al., 2019b) and Normalized Discounted Cumulative Gain (NDCG) (Järvelin and Kekäläinen, 2002) where diagnosis labels are available. At the range level, we measure the I nter p retation S core (IPS) initially proposed in Li et al. (2021) and here expanded to fit the P % parameterization. Full details are discussed in Appendix J.

Baselines We compare SARAD against state-of-the-art anomaly detection baselines, including Isolation Forest-based IF (Liu et al., 2008), Deep IF (DIF) (Xu et al., 2023); MLP-based USAD (Audibert et al., 2020); graph-based GDN (Deng and Hooi, 2021); LSTM-based MAD-GAN (Li et al., 2019); CNN-based DiffAD (Xiao et al., 2023); and Transformer-based TranAD (Tuli et al., 2022), ATF-UAD (Fan et al., 2023), AT (Xu et al., 2022), DCdetector (Yang et al., 2023). Noticeably, GDN employs explicit spatial modeling in its graph construction although spatial associations are not directly involved in anoamly scoring. MAD-GAN emphasizes on anomaly detection within cyber-physical systems. All baselines are trained using official implementations where available and recommended hyperparameters from respectively papers are used.

4.2 Results

Anomaly Detection Table 3 shows the anomaly detection performance in metrics defined in Section 4.1. It demonstrates that, despite its architectural elegance, SARAD either outperforms all baselines by significant margins on the threshold-independent VUS-ROC scores (SMD: +15 . 51 % MAD-GAN, HAI: +9 . 92 %DiffAD) or performs on par with current best detectors (PSM: -1 . 36 % GDN, SWaT: -0 . 36 %DIF). IF scrutinizes the distributional shifts of anomalies with random data partitions and delivers consistent performance across datasets. DIF extends IF into randomized deep representation spaces and archives decent improvements due to more flexible partitions and temporally local information extraction via dilated convolutions. Temporal modeling methods such as DiffAD, ATF-UAD, and AT rely solely or heavily on reconstruction errors and when the errors do not correspond the the underlying anomalies their performance plummet. Adversarial training in USAD and MAD-GAN amplifies reconstruction errors of anomalies to mitigate but not eliminate such issues and thus suffer less performance drops. In contrast, our SARAD additionally accounts for the SAR frequent with anomalies and independent of data distributional shifts, thus outperforming all. SARAD also overpasses GDN, which despite its explicit spatial modeling adopts prediction errors as its sole detection criterion, limiting its performance. SARAD's top performance on SWaT and HAI underlines its ability to unravel complex spatial associations even in complex large-scale systems. Standard deviations of Table 3 are reported in Appendix K.

Anomaly Diagnosis Table 4 shows the anomaly diagnosis performance in metrics defined in Section 4.1. DiffAD uses a subset of SMD features and thus is discarded from comparisons for fairness. SARAD outperforms baselines on the point-based HR @150% (SMD: +26 . 67 %TranAD, SWaT: +5 . 10 %USAD, HAI: +3 . 81% GDN) and NDCG @150% (SMD: +27 . 97 %TranAD, SWaT: +4 . 76 % GDN, HAI: +2 . 93 % GDN). SARAD also outperforms on the range-based IPS @150% on most datasets (SMD: 43 . 19 %TranAD, SWaT: 33 . 43 %TranAD). Unlike SMD which performs forensic diagnosis to label anomalous features, SWaT and HAI label only the origins of cyberattacks as diagnosis labels. Consequentially, attack origins sometimes might not behave anomalously, e.g., attacks had failed, or the full set of anomalous features were not identified, thus diminishing the performance numbers on SWaT and HAI. SARAD generally outperforms detectors underpinned by temporal modeling due to its sensitivity to spatial associative changes. SARAD also outperforms spatial detectors such as GDN whose prediction errors limit its temporal scope to a single time point. Standard deviations of Table 4 are reported in Appendix L.

Visualization Figure 3 visualizes a real-world anomaly example via SARAD. Our detector captures the significant SAR caused by the anomalous features. The loose reconstruction of the progression raises the progression-based score p and, in turn, the joint detection score s . Taking a broader view of the series in Fig. 3h, SAR significantly raises the scores at the start of the anomaly, even when the data-based errors r are small. SARAD exploits SAR to achieve more robust anomaly detection.

(i)

SMD

SWaT

HAI

Method

HR

@

P

%

ND

@

P

%

IPS

@

P

%

HR

@

P

%

ND

@

P

%

IPS

@

P

%

HR

@

P

%

ND

@

P

%

IPS

@

P

%

P

100

150

100

150

100

150

100

150

100

150

100

150

100

150

100

150

100

150

TranAD

33.33 45.26 34.71 41.81 22.21 33.25 4.82

6.36 4.82

5.76 17.47 20.91 4.08

6.27 3.98

5.33

12.67 21.78

ATF-UAD 27.80 40.94 26.67 34.46 17.34 26.16 1.85

3.19 1.83

2.65

5.45

8.28 2.96

5.22 3.08

4.48

5.04

8.59

USAD

27.03 39.74 25.83 33.44 18.31 26.07 4.39

7.73 4.39

6.48 12.83 27.17 3.78

5.83 3.76

5.01 12.37 16.22

GDN

28.67 41.57 28.62 36.27 21.18 30.76 5.99

7.21 6.13

6.89 14.04 21.72 5.45

8.29 5.50

7.23

7.41 12.07

DiffAD

N/A

N/A

N/A

N/A

N/A

N/A 1.82

2.90 1.81

2.47

5.45 12.63 3.32

4.79 3.41

4.32 12.00 17.26

Ours

56.73 71.93 60.79 69.78 61.38 76.44 9.57 12.83 9.61 11.65 35.45 54.34 6.45 12.10 6.69 10.16

7.48 14.07

#1

#9

#10

#12

#13

#14

#15

time range: [15798, 15949]

(a)

features

range: [0.0006, 0.2200]

(b)

features

range: [0.0006, 0.2200]

(c)

heads

range:

[0.0000, 1.5716]

(d)

heads

range:

[0.0000, 1.5716]

(e)

(f)

(g) (h)

r

j

pj

sj

日

Figure 3: Visualization of applying SARAD for detection on SMD. 3a shows the raw time series right before and during an anomaly p i (colored in red). An input time window for SARAD is bounded in the black box. 3b and 3c show the average association mapping ¯ A L via final L -th layer's MHSA. 3d shows the aggregated progression S according to Eq. 3. 3e is its reconstruction. 3f, 3g, 3h show the scores p , r , and joint s according to Eq. 5 per feature. 3i shows the anomaly scores for 3a's segment. Anomalous features (#1, #9, #10, #12, #13, #14 ,and #15) are highlighted with red bounding boxes.

Complexity and Time Overheads SARAD incurs 32 mins for training and and 0.39 ms for inference per sample on HAI, the largest dataset, falling far below the data collection time and sampling frequency. Those numbers are comparable with baselines and detailed in Appendix N.

4.3 Ablation Studies

Spatial Progression Reconstruction To evaluate the effectiveness of the progression module, we perform ablation studies on its submodules in Table 5. Standard deviations are reported in Appendix K. Removing the ReLU, i.e., to capture both association increases and reductions, in progression loses the focus on asscoation reduction and impairs the detection performance. Replacing the column sum operation in aggregation with the row sum which characterizes the disconnection of anomalous features from others and is shown to be less effectiveness than the column sum representing the drop out rates. Fully concatenating without sum operation dilutes the reduction patterns and significantly hurts the detection, at a cost of complexity. For the detection submodule, using the progression directly instead of the reconstruction errors registers reductions as anomalies directly and underperforms except on SMD due to its inability to rule out normal reduction patterns.

Choice of Detection Criterion Table 6 compares the detection performance using Eq. 5 (Joint), using only data-based r (DR), and using only progression-based p (SPR). While the data reconstruction is a robust criterion of anomalousness, SARAD embeds the spatial information into the joint criterion and outperforms either single criterion overall. Standard deviations are reported in Appendix K.

Additional ablation studies on the choice of diagnosis criterion are detailed in Appendix D.

5 Conclusion

In this work, we propose SARAD for time series anomaly detection and diagnosis. The approach effectively exploits the spatial association descending patterns of anomalies. Data reconstruction with Transformer guides learning of spatial associations from data and captured as progression, while progression reconstruction quantifies the anomalous association descent and complements the insensitivity of the former to spatial disassociation during anomalies. SARAD experimentally demonstrates state-of-the-art detection and diagnosis performance and foreshadows the power of spatial modeling for related time series tasks.

Acknowledgments and Disclosure of Funding

Calculations were performed using the Sulis Tier 2 HPC platform hosted by the Scientific Computing Research Technology Platform at the University of Warwick. Sulis is funded by EPSRC Grant EP/T022108/1 and the HPC Midlands+ consortium. The authors acknowledge the use of the Batch Compute System in Department of Computer Science at the University of Warwick, and associated support services, in the completion of this work.

References

Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. 2021a. Practical Approach to Asynchronous Multivariate Time Series Anomaly Detection and Localization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (Virtual Event, Singapore) (KDD '21) . Association for Computing Machinery, New York, NY, USA, 2485-2494. https://doi.org/10.1145/3447548.3467174

Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. 2021b. RANSynCoders. https:// github.com/icsdataset/hai .

Charu C. Aggarwal. 2013. Outlier Analysis . Springer. http://dx.doi.org/10.1007/ 978-1-4614-6396-2

Julien Audibert, Pietro Michiardi, Frédéric Guyard, Sébastien Marti, and Maria A. Zuluaga. 2020. USAD: UnSupervised Anomaly Detection on Multivariate Time Series. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Virtual Event, CA, USA) (KDD '20) . Association for Computing Machinery, New York, NY, USA, 3395-3404. https://doi.org/10.1145/3394486.3403392

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for HyperParameter Optimization. In Advances in Neural Information Processing Systems , J. ShaweTaylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (Eds.), Vol. 24. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2011/file/ 86e8f7ab32cfd12577bc2619bc635690-Paper.pdf

Zhihao Dai, Ligang He, Shuang-Hua Yang, and Matthew Leeke. 2023. Revealing Ongoing Sensor Attacks in Industrial Control System Via Setpoint Modification. In 2023 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech) . 0191-0199. https://doi.org/10.1109/ DASC/PiCom/CBDCom/Cy59711.2023.10361334

Ailin Deng and Bryan Hooi. 2021. Graph Neural Network-Based Anomaly Detection in Multivariate Time Series. Proceedings of the AAAI Conference on Artificial Intelligence 35, 5 (May 2021), 4027-4035. https://doi.org/10.1609/aaai.v35i5.16523

Jin Fan, Zehao Wang, Huifeng Wu, Danfeng Sun, Jia Wu, and Xin Lu. 2023. An Adversarial Time-Frequency Reconstruction Network for Unsupervised Anomaly Detection. Neural Networks 168 (2023), 44-56. https://doi.org/10.1016/j.neunet.2023.09.018

Tryambak Gangopadhyay, Sin Yong Tan, Zhanhong Jiang, Rui Meng, and Soumik Sarkar. 2021. Spatiotemporal Attention for Multivariate Time Series Prediction and Interpretation. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . 3560-3564. https://doi.org/10.1109/ICASSP39728.2021.9413914

Astha Garg, Wenyu Zhang, Jules Samaran, Ramasamy Savitha, and Chuan-Sheng Foo. 2022. An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series. IEEE Transactions on Neural Networks and Learning Systems 33, 6 (2022), 2508-2517. https://doi.org/10. 1109/TNNLS.2021.3105827

Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. 2000. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. circulation 101, 23 (2000), e215-e220.

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735

iTrust. 2023. Datasets. https://itrust.sutd.edu.sg/itrust-labs_datasets/ .

Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACMTrans. Inf. Syst. 20, 4 (oct 2002), 422-446. https://doi.org/10.1145/582415.582418

Hyeongwon Kang and Pilsung Kang. 2024. Transformer-based multivariate time series anomaly detection using inter-variable attention mechanism. Knowledge-Based Systems 290 (2024), 111507. https://doi.org/10.1016/j.knosys.2024.111507

Jina Kim, Hyeongwon Kang, and Pilsung Kang. 2023. Time-series anomaly detection with stacked Transformer representations and 1D convolutional network. Engineering Applications of Artificial Intelligence 120 (2023), 105964. https://doi.org/10.1016/j.engappai.2023.105964

Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations .

Diederik P Kingma and Max Welling. 2014. Auto-encoding variational bayes. In International Conference on Learning Representations .

Dan Li, Dacheng Chen, Baihong Jin, Lei Shi, Jonathan Goh, and See-Kiong Ng. 2019. MAD-GAN: Multivariate Anomaly Detection for Time Series Data with Generative Adversarial Networks. In Artificial Neural Networks and Machine Learning - ICANN 2019: Text and Time Series , Igor V. Tetko, Vˇ era K˚ urková, Pavel Karpov, and Fabian Theis (Eds.). Springer International Publishing, Cham, 703-716.

Zhihan Li, Youjian Zhao, Jiaqi Han, Ya Su, Rui Jiao, Xidao Wen, and Dan Pei. 2021. Multivariate Time Series Anomaly Detection and Interpretation using Hierarchical Inter-Metric and Temporal Embedding. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (Virtual Event, Singapore) (KDD '21) . Association for Computing Machinery, New York, NY, USA, 3220-3230. https://doi.org/10.1145/3447548.3467075

Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008. Isolation Forest. In 2008 Eighth IEEE International Conference on Data Mining . IEEE, Pisa, Italy, 413-422. https://doi.org/10. 1109/ICDM.2008.17

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. 2024. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations . https://openreview.net/ forum?id=JePfAI8fah

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. 2022. Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting. In Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 9881-9893. https://proceedings.neurips.cc/paper_files/ paper/2022/file/4054556fcaa934b0bf76da52cf4f92cb-Paper-Conference.pdf

Pankaj Malhotra, Lovekesh Vig, Gautam Shroff, Puneet Agarwal, et al. 2015. Long Short Term Memory Networks for Anomaly Detection in Time Series.. In Esann , Vol. 2015. 89.

Aditya P. Mathur and Nils Ole Tippenhauer. 2016. SWaT: a water treatment testbed for research and training on ICS security. In 2016 International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater) . IEEE, Vienna, Austria, 31-36. https://doi.org/10.1109/ CySWater.2016.7469060

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In The Eleventh International Conference on Learning Representations . https://openreview.net/forum?id=Jbdc0vTOcol

John Paparrizos, Paul Boniol, Themis Palpanas, Ruey S Tsay, Aaron Elmore, and Michael J Franklin. 2022. Volume Under the Surface: A New Accuracy Evaluation Measure for Time-Series Anomaly Detection. Proceedings of the VLDB Endowment 15, 11 (2022), 2774-2787.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, HighPerformance Deep Learning Library. In Advances in Neural Information Processing Systems , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2019/ file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf

Kai Qian, Jie Jiang, Yulong Ding, and Shuang-Hua Yang. 2021. DLGEA: a deep learning guided evolutionary algorithm for water contamination source identification. Neural Comput. Appl. 33, 18 (sep 2021), 11889-11903. https://doi.org/10.1007/s00521-021-05894-y

Cory A. Rieth, Ben D. Amsel, Randy Tran, and Maia B. Cook. 2018. Issues and Advances in Anomaly Detection Evaluation for Joint Human-Automated Systems. In Advances in Human Factors in Robots and Unmanned Systems , Jessie Chen (Ed.). Springer International Publishing, Cham, 52-63.

Bharadwaj Satchidanandan and P. R. Kumar. 2017. Dynamic Watermarking: Active Defense of Networked Cyber-Physical Systems. Proc. IEEE 105, 2 (2017), 219-240. https://doi.org/ 10.1109/JPROC.2016.2575064

Lifeng Shen, Zhuocong Li, and James Kwok. 2020. Timeseries Anomaly Detection using Temporal Hierarchical One-Class Network. In Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 13016-13026. https://proceedings.neurips.cc/paper_files/paper/ 2020/file/97e401a02082021fd24957f852e0e475-Paper.pdf

Hyeok-Ki Shin, Woomyo Lee, Seungoh Choi, Jeong-Han Yun, and Byung-Gi Min. 2021a. HAI security datasets. https://github.com/icsdataset/hai .

Hyeok-Ki Shin, Woomyo Lee, Jeong-Han Yun, and Byung-Gi Min. 2021b. Two ICS Security Datasets and Anomaly Detection Contest on the HIL-Based Augmented ICS Testbed. In Cyber Security Experimentation and Test Workshop (Virtual, CA, USA) (CSET '21) . Association for Computing Machinery, New York, NY, USA, 36-40. https://doi.org/10.1145/3474718.3474719

Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. 2019a. OmniAnomaly. https://github.com/NetManAIOps/OmniAnomaly .

Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. 2019b. Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Anchorage, AK, USA) (KDD '19) . Association for Computing Machinery, New York, NY, USA, 2828-2837. https://doi.org/10.1145/3292500.3330672

Nesime Tatbul, Tae Jun Lee, Stan Zdonik, Mejbah Alam, and Justin Gottschlich. 2018. Precision and Recall for Time Series. In Advances in Neural Information Processing Systems , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2018/ file/8f468c873a32bb0619eaeb2050ba45d1-Paper.pdf

Shreshth Tuli, Giuliano Casale, and Nicholas R. Jennings. 2022. TranAD: deep transformer networks for anomaly detection in multivariate time series data. Proc. VLDB Endow. 15, 6 (feb 2022), 1201-1214. https://doi.org/10.14778/3514061.3514067

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems , I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/ paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

Dennis Wagner, Tobias Michels, Florian C.F. Schulz, Arjun Nair, Maja Rudolph, and Marius Kloft. 2023. TimeSeAD: Benchmarking Deep Multivariate Time-Series Anomaly Detection. Transactions on Machine Learning Research (2023). https://openreview.net/forum?id=iMmsCI0JsS

Chengsen Wang, Zirui Zhuang, Qi Qi, Jingyu Wang, Xingyu Wang, Haifeng Sun, and Jianxin Liao. 2023b. Drift doesn't Matter: Dynamic Decomposition with Diffusion Reconstruction for Unstable Multivariate Time Series Anomaly Detection. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 10758-10774. https://proceedings.neurips.cc/paper_files/ paper/2023/file/22f5d8e689d2a011cd8ead552ed59052-Paper-Conference.pdf

Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. 2023a. MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting. In The Eleventh International Conference on Learning Representations . https://openreview.net/forum? id=zt53IDUR1U

Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and JUN ZHOU. 2024. TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting. In The Twelfth International Conference on Learning Representations . https://openreview. net/forum?id=7oLshfEIC2

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. In Advances in Neural Information Processing Systems , M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 22419-22430. https://proceedings.neurips.cc/paper_files/paper/2021/file/ bcc0d400288793e8bdcd7c19a8ac0c2b-Paper.pdf

Chunjing Xiao, Zehua Gou, Wenxin Tai, Kunpeng Zhang, and Fan Zhou. 2023. Imputation-based Time-Series Anomaly Detection with Conditional Weight-Incremental Diffusion Models. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Long Beach, CA, USA) (KDD '23) . Association for Computing Machinery, New York, NY, USA, 2742-2751. https://doi.org/10.1145/3580305.3599391

Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, Jie Chen, Zhaogang Wang, and Honglin Qiao. 2018. Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications. In Proceedings of the 2018 World Wide Web Conference (Lyon, France) (WWW '18) . International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 187-196. https: //doi.org/10.1145/3178876.3185996

Hongzuo Xu, Guansong Pang, Yijie Wang, and Yongjun Wang. 2023. Deep Isolation Forest for Anomaly Detection. IEEE Transactions on Knowledge and Data Engineering 35, 12 (2023), 12591-12604. https://doi.org/10.1109/TKDE.2023.3270293

Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. 2022. Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy. In International Conference on Learning Representations . https://openreview.net/forum?id=LzQQ89U1qm_

Omry Yadan. 2019. Hydra - A framework for elegantly configuring complex applications. Github. https://github.com/facebookresearch/hydra

Yiyuan Yang, Chaoli Zhang, Tian Zhou, Qingsong Wen, and Liang Sun. 2023. DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (<conf-loc>, <city>Long Beach</city>, <state>CA</state>, <country>USA</country>, </conf-loc>) (KDD '23) . Association for Computing Machinery, New York, NY, USA, 3033-3045. https://doi. org/10.1145/3580305.3599295

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023. Are Transformers Effective for Time Series Forecasting? 37 (Jun. 2023), 11121-11128. https://doi.org/10.1609/aaai.v37i9.26317

Chaoli Zhang, Tian Zhou, Qingsong Wen, and Liang Sun. 2022. TFAD: A Decomposition Time Series Anomaly Detection Architecture with Time-Frequency Analysis. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (Atlanta, GA, USA) (CIKM '22) . Association for Computing Machinery, New York, NY, USA, 2497-2507. https://doi.org/10.1145/3511808.3557470

Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao, Karan Goel, and Christopher Re. 2023. Effectively Modeling Time Series with Simple Discrete State Spaces. In The Eleventh International Conference on Learning Representations . https://openreview.net/forum? id=2EpjkjzdCAa

Yunhao Zhang and Junchi Yan. 2023. Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting. In The Eleventh International Conference on Learning Representations . https://openreview.net/forum?id=vSVLM2j9eie

Hang Zhao, Yujing Wang, Juanyong Duan, Congrui Huang, Defu Cao, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. 2020. Multivariate Time-Series Anomaly Detection via Graph Attention Network. In 2020 IEEE International Conference on Data Mining (ICDM) . 841-850. https://doi.org/10.1109/ICDM50108.2020.00093

Yu Zheng, Huan Yee Koh, Ming Jin, Lianhua Chi, Khoa T. Phan, Shirui Pan, Yi-Ping Phoebe Chen, and Wei Xiang. 2023. Correlation-Aware Spatial-Temporal Graph Learning for Multivariate Time-Series Anomaly Detection. IEEE Transactions on Neural Networks and Learning Systems (2023), 1-15. https://doi.org/10.1109/TNNLS.2023.3325667

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162) , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu,

Tian Zhou, Peisong Niu, xue wang, Liang Sun, and Rong Jin. 2023. One Fits All: Power General Time Series Analysis by Pretrained LM. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 43322-43355. https://proceedings.neurips.cc/paper_files/paper/ 2023/file/86c17de05579cde52025f9984e6e2ebb-Paper-Conference.pdf

A Broader Impacts

The broader impact of the work presented rests in the increasingly pervasive nature of industrial control systems, intrusion detection systems, and remote monitoring solutions in healthcare contexts, all of which commonly utilize some form of anomaly detection. Further to the time series analysis that is commonplace, the work presented in this paper demonstrates how Transformer can be used to learn the spatial associations that ubiquitously characterize these feedback-controlled systems, supplementing time series analysis to provide state-of-the-art performance. As such, the work presented has broad applicability, whilst explicitly targeting automated industrial control systems.

B Limitations

While the model size scales linearly with the number of features, the time complexity of SARAD is quadratic with respect to the features. SARAD could incur significant training and inference overheads when the supervisory system is extensively large. We caution that the overheads of the largest dataset in our experiments fall well below data collection overhead and sampling frequency (see Appendix N). To scale, we will explore hierarchical time series anomaly detection via clustering. Another limitation of this work is the scarcity of forensically labeled datasets like SMD for anomaly diagnosis, not least due to the intensive labor and domain knowledge implied. To address that gap, we will explore publicly available audit and operational time series data for sources.

feature #1

feature #9

feature #10

feature #12

feature #13

feature #14

feature #15

time range: [15303, 16941]

#1

#9

#10

#12

#13

#14

#15

features

#1

#9

#10

#12

#13

#14

#15

features

range: [0.0035, 0.1441]

#1

#9

#10

#12

#13

#14

#15

features

#1

#9

#10

#12

#13

#14

#15

features

range: [0.0035, 0.1441]

#1

#9

#10

#12

#13

#14

#15

features

#1

#9

#10

#12

#13

#14

#15

features

range: [0.0035, 0.1441]

#1

#9

#10

#12

#13

#14

#15

features

#1

#9

#10

#12

#13

#14

#15

features

range: [0.0000, 0.0367]

#1

#9

#10

#12

#13

#14

#15

features

#1

#9

#10

#12

#13

#14

#15

features

range: [0.0000, 0.0367]

feature #12

feature #15

time range: [92799, 93210]

#12

#15

features

#12

#15

features

range: [0.0014, 0.1178]

(b) Pre-

p

's

A

.

#12

#15

features

#12

#15

features

range: [0.0014, 0.1178]

(c) In-

p

's

A

.

#12

#15

features

#12

#15

features

range: [0.0014, 0.1178]

(d) Post-

p

's

A

.

#12

#15

features

#12

#15

features

range: [0.0000, 0.0415]

(e) Pre-reduction.

#12

#15

features

#12

#15

features

range: [0.0000, 0.0415]

(f) Post-reduction.

i

i

i

C Examples of Spatial Association Reduction

As mentioned in Section 1, herein we provide more real-world examples of Spatial Association Reduction (SAR) exhibited by time series anomalies. Figures 4, 5, 6, 7 showcase the spatial associations captured within Transformer via MHSA. Subfigures in each aforementioned figure show, in that order, (a) the raw time series right before, during, and after an anomaly p i ∈ P (colored in red), (b) association mapping A L h output by final L -th layer's MHSA are averaged across heads to derive average association A before p i , (c) A L h during p i , (d) A L h after p i wherein brighter cells have smaller values and anomalous features are highlighted with red bounding boxes, (e) the reduction-only changes from before the anomaly to during the anomaly, i.e., ReLU ( A pre -A in ) , and finally (f) the reduction-only changes from after the anomaly to during the anomaly, i.e., ReLU ( A post -A in ) .

feature #38

time range: [115748, 116933]

(a) Raw time series before, during, and after an anomaly p i .

#38

features

#38

features

range: [0.0044, 0.3045]

(b) Pre-

p

i

's

A

.

#38

features

#38

features

range: [0.0044, 0.3045]

(c) In-

p

i

's

A

.

#38

features

#38

features

range: [0.0044, 0.3045]

(d) Post-

p

i

's

A

.

#38

features

#38

features

range: [0.0000, 0.0136]

(e) Pre-reduction.

#38

features

#38

features

range: [0.0000, 0.0136]

(f) Post-reduction.

feature #4

time range: [371378, 371681]

(a) Raw time series before, during, and after an anomaly p i .

#4

features

#4

features

range: [0.0013, 0.3153]

(b) Pre-

p

i

's

A

.

#4

features

#4

features

range: [0.0013, 0.3153]

(c) In-

p

i

's

A

.

#4

features

#4

features

range: [0.0013, 0.3153]

(d) Post-

p

i

's

A

.

#4

features

#4

features

range: [0.0000, 0.0132]

(e) Pre-reduction.

#4

features

#4

features

range: [0.0000, 0.0132]

(f) Post-reduction.

D Choice of Diagnosis Criterion

Concerning the rationality of data-only diagnosis criterion in Eq. 6, we consider an alternate joint diagnosis criterion in line with the detection criterion in Eq. 5.

where r j is feature j 's data reconstruction error, p j its progression reconstruction error, µ r j , µ p j the means of r j , p j on the validation set, and σ r j , σ p j the standard deviation of r j , p j . Table 7 compares the diagnosis performance using only r j (SARAD), using only p j (SPR), and using Eq. 7 (Joint). Unlike in anomaly detection, the great discrepancy between r j and p j more than often degrades the performance of the joint criterion. SARAD uses r j only which produces suboptimal and yet reliable performance in the longer anomalous horizons. Standard deviations are reported in Appendix L.

E Datasets

We include four real-world datasets collected under industrial control and IT service monitoring settings for evaluations. Anomalies range from IT service outages to external cyber-physical attacks against control systems.

E.1 Dataset Descriptions

Server Machine Dataset (SMD) (Su et al., 2019b,a) is a server metric dataset from a large-scale IT company. Engineers annotated anomalous events in the second half of the data with indicator-level attributions.

Pooled Server Metrics (PSM) dataset (Abdulaal et al., 2021a,b) captures key performance indicators of servers on an online shopping platform. Website engineers annotated anomalous events for data in the last eight weeks.

Secure Water Treatment (SWaT) dataset (Mathur and Tippenhauer, 2016; iTrust, 2023) contains sensor readings and actuator status on a minuscule real-world water treatment system during a six-day normal operational period. A knowledgeable attacker performed 36 cyber-physical attacks during a five-day attack period and they are labelled as anomalous accordingly.

HIL-based Augmented ICS (HAI) dataset (Shin et al., 2021b,a) records measurements and control actions within a Hardware-In-the-Loop (HIL) dual power (steam-turbine and hydropower) generation testbed during its two-week operation. Both single-point primitive and multi-point combined attacks are performed on the testbed to emulate a threat actor with cyber-physical capacities. We use the 21.03 version of HAI.

All training sets contain only unlabeled data and the test sets contain data with anomaly labels. Statistics of the datasets are given in Table 2.

E.2 Lengths of Anomalies

We further characterize the detection datasets by the lengths of the anomalous events. Figure 8 shows the empirical cumulative distribution function of the anomalous lengths. SWaT has the longest median length of 447 among the four datasets considered, followed by HAI (162), SMD (11), and lastly PSM (5). The very short lengths on SMD and PSM benefit temporal detectors which tend to embed a single or few time points (see Appendix M), whereas SARAD adopts a half time window embedding strategy. The catch is that SARAD can learn spatial relationships with temporal aggregated information per feature, while temporal detectors could not, bringing about benefits of performing anomaly detection in the spatial association space. A more scalable approach for temporal aggregation is to be explored in the future, though variable window sizes or subseries splits might be implied.

E.3 Lengths of Diagnosis Labels

Figure 8 shows the empirical cumulative distribution function of the diagnosis label lengths. Whereas SMD forensically labels features which deviate from their normal behavioral patterns as anomalous, SWaT and HAI only label the points of attacks as anomalous as the their creators have advanced knowledge of such attacks. The latter labeling strategy results in incomplete sets of anomalous features and diminishes the diagnosis performance of all models, as evidence in Table 4 in Section 4.2.

10 1

10 2

10 3

Length of Anomaly

0%

20%

40%

60%

80%

100%

Proportion

4

25%

11

50%

72

75%

min = 2

max = 3161

std = 238.42

(a) SMD.

10 0

10 1

10 2

10 3

Length of Anomaly

0%

20%

40%

60%

80%

100%

Proportion

2

25%

5

50%

88

75%

min = 1

max = 8861

std = 1226.88

(b) PSM.

10 3

10 4

Length of Anomaly

0%

20%

40%

60%

80%

100%

Proportion

332

25%

447

50%

712

75%

min = 101

max = 35900

std = 5982.59

(c) SWaT.

50

100

150

200

250

300

350

400

Length of Anomaly

0%

20%

40%

60%

80%

100%

Proportion

107

25%

162

50%

256

75%

min = 17

std = 95.13

(d) HAI.

5

10

15

20

25

30

Length of Anomaly

0%

20%

40%

60%

80%

100%

Proportion

4

25%

8

50%

18

75%

min = 1

std = 7.88

(a) SMD.

1.00

1.25

1.50

1.75

2.00

2.25

2.50

2.75

3.00

Length of Anomaly

0%

20%

40%

60%

80%

100%

Proportion

min = 1

std = 0.51

(b) SWaT.

1.00

1.25

1.50

1.75

2.00

2.25

2.50

2.75

3.00

Length of Anomaly

0%

20%

40%

60%

80%

100%

Proportion

2

50%

2

75%

min = 1

std = 0.72

(c) HAI.

F Implementation Details

We implement SARAD in Python using pyTorch library (Paszke et al., 2019) and Hydra framework (Yadan, 2019). All experiments are run on a single NVIDIA A10 (24GB) GPU. Adam optimizer (Kingma and Ba, 2015) is used and learning rate is halved every epoch for 3 epochs to prevent over-fitting. The time window size is 2 W = 100 . The data reconstruction module has H = 8 attention heads per layer with attention length D = 512 and hidden length D FF = 2048 . For hyperparameter tuning, training set is temporally partitioned into 80% for training and 20% for validation. On each dataset we first perform TPE sampling (Bergstra et al., 2011) for number of encoding layers L ∈ { 3 , 5 } and learning rate ∈ [10 -4 , 10 -2 ] to derive the best data reconstruction loss L R on the validation set. The progression module by default has hidden length of D P = 64 . We then perform TPE sampling to search weight λ L S ∈ [10 -2 , 10 2 ] for the progression reconstruction loss L S on the validation set.

G Open-accessed Code and Data

During the review period, code is anonymized and openly available at https://github.com/ daidahao/SARAD/ with specific instructions and scripts to reproduce experimental results. All data used in our experiments can be openly accessed from public repositories or requested via original authors' websites. Full links are provided in Appendix E.

10

20

50

100

200

400

60

70

80

90

VUS-ROC (%)

SMD

PSM

SWaT

HAI

(a) Window size.

1

2

3

4

5

60

65

70

75

80

85

90

95

VUS-ROC (%)

SMD

PSM

SWaT

HAI

(b) Training epochs.

64

128

256

512

1024

2048

60

70

80

90

VUS-ROC (%)

SMD

PSM

SWaT

HAI

(c) Attention length

D

.

8

16

32

64

128

256

60

65

70

75

80

85

90

95

VUS-ROC (%)

SMD

PSM

SWaT

HAI

(d) Progression hidden length

D

P

.

H Hyperparameter Sensitivity

We examine the hyperparameter sensitivity of SARAD's detection performance. Concretely, we consider the effects of the sliding window size (default is 100 ), the number of training epochs ( 3 ), the attention length of Transformer encoding layers D ( 512 ), and the hidden length of the progression reconstruction module D P ( 64 ). Figure 10 and 11 present the results. On the window size, a sliding window too small confines temporal modeling to small temporal receptive fields and contains anomaly detection in the association space. However, a sliding window too large incurs higher computational costs, although unlike temporal modeling the costs here are linear. On datasets with shorter anomalous lengths such as SMD and PSM, the anomalous patterns are diluted even further, resulting in performance degradation.

On the number of training epochs, fewer epochs lead to model underfitting, and yet overfitting is largely prevented with more epochs due to the aggressive learning rate halving per epoch. On the attention length D , a larger Transformer is prone to overfitting with visible performance drops on SMD and PSM as D passes the default 512 , both of whose monitored systems are smaller in scale. On the hidden length D P , a more complex progression anomaly detector does not adversely impact the performance, suggesting that the association space is less prone to detection overfitting than the data space.

10

20

50

100

200

400

10

20

30

40

50

60

70

VUS-PR (%)

SMD

PSM

SWaT

HAI

(a) Window size.

1

2

3

4

5

10

20

30

40

50

60

70

VUS-PR (%)

SMD

PSM

SWaT

HAI

(b) Training epochs.

64

128

256

512

1024

2048

10

20

30

40

50

60

70

VUS-PR (%)

SMD

PSM

SWaT

HAI

(c) Attention length

D

.

8

16

32

64

128

256

20

30

40

50

60

70

VUS-PR (%)

SMD

PSM

SWaT

HAI

(d) Progression hidden length

D

P

.

I Detection Metrics

Conventional metrics such as precision, recall, F1 and the threshold-independent Area Under the Curve (AUC) scores are commonly point-based, i.e., predicted labels are scored individually by time points (Zhou et al., 2023; Zhang et al., 2022; Xu et al., 2018). Real-world benchmarks are rife with range-wise anomalies spanning consecutive time points (Wagner et al., 2023). Point-based metrics are generally ill-suited for evaluating detection performance due to the continuous-discrete conversion and the series-label misalignment (labeling anomalies precisely is hard) (Garg et al., 2022; Tatbul et al., 2018). Here, we use the range-based metrics proposed in (Paparrizos et al., 2022). Compared against their point-based counterparts, they provide robustness to labeling delay and scoring noises as well as performant detector separability and series consistency.

Given a set of anomalous ranges P = { p i = ( s i , e i ) } wherein each anomaly p i starts at timestamp s i and ends at e i , we enclose each range with uniform l/ 2 -length preceding and succeeding buffers. Given the anomaly label y t ∈ { 0 , 1 } at each timestamp t , we derive a new soft label ˜ y t ∈ [0 , 1] as per the minimum temporal distance of t to any anomaly p i ∈ P :

where l is the buffer length, normally set to the median segment length in P . Within the buffers, ˜ y t monotonically increases from √ 2 / 2 to 1 as the distance decreases. With the new soft label series ˜ Y = { ˜ y t } , we define the True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN) accordingly:

We then compute the threshold-independent AUC for the Receiver Operating Characteristic (AUCROC), i.e., TP rate vs FP rate, and the Precision-Recall (AUC-PR) curves respectively to be rid of thresholding impact. Fully parameter-free Volume Under the Surface (VUS) scores for AUC-ROC (VUS-ROC) and AUC-PR (VUS-PR) are also computed under different buffer lengths ˆ l ∈ [0 , 2 l ] .

J Diagnosis Metrics

In line with previous works (Tuli et al., 2022; Zhao et al., 2020), we use common metrics such as Hit Rate (HR) (Su et al., 2019b) and Normalized Discounted Cumulative Gain (NDCG) (Järvelin and Kekäläinen, 2002) to measure performance where diagnosis labels are available. Given a set of anomalous features G i ⊆ [ N ] at an anomalous timestamp t ∈ p i ∈ P as a diagnosis ground-truth and the set of top k -ranked features Γ t @ P % according to Eq. 5 where k = ⌈| G i | × P % ⌉ , say k = 5 when | G i | = 3 and P = 150 , the HR at P % ( P ≥ 100 ) features is the overlap ratio between the two:

In information retrieval, DCG measures the cumulative utility of retrieved documents by their ranking order up to a certain position. NDCG normalizes the DCG by the maximum possible DCG. They are parameterized by P % to determine the location in our evaluation and calculated as follows.

where r j ∈ { 0 , 1 } is the relevance value of the j -th element and, in this case, the membership of Γ t @ P % 's j -th feature in G i . NDCG has a value strictly between 0 and 1 .

At the range level, we measure the I nter p retation Score (IPS) initially proposed in Li et al. (2021) and here expanded to fit the P % parameterization. For each anomalous range p i , the IPS score is:

where Ω i @ P % is the top k -ranked features according to max t ∈ p i s j,t , the j -th feature's maximum anomaly score during p i and k = ⌈| G i | × P % ⌉ . It is the HR equivalence at the range level as per the highest anomalous scores per feature.

K Standard Deviations of Detection Performance

Table 8 reports the standard deviations of anomaly detetion performance as reported in Table 3 in Section 4.2.

Table 9 reports the standard deviations of anomaly diagnosis performance as reported in Table 5 in Section 4.3.

Table 10 reports the standard deviations of anomaly detection performance as reported in Table 6 in Section 4.3.

L Standard Deviations of Diagnosis Diagnosis

Table 11 reports the standard deviations of anomaly diagnosis performance as reported in Table 4 in Section 4.2.

Table 12 reports the standard deviations of anomaly diagnosis performance as reported in Table 7 in Appendix D.

M Point-based Evaluations

In addition to the model evaluations using range-based metrics such as VUS-ROC and VUS-PR in Sections 4.2 and 4.3, we conduct point-based evaluations herein using point-based metrics exclusively. Table 13 reports report the Precision, Recall, and F1 scores under the threshold where the method of interest achieves the best F1 score. To mitigate the impact of thresholding protocol, Table 13 also reports the threshold-independent AUC-ROC and AUC-PR scores. Table 14 reports the standard deviations.

SARAD achieves state-of-the-art performance on most datasets except PSM. The enlarged temporal receptive field of SARAD (half input window) contributes to its underperformance by point-based metrics, especially on datasets where anomalous ranges are short (see Table 2), when compared against most others' receptive filed of single or few time points (typical 1D Conv kernel size is 3). We again underline that most real-world anomalies are continuous and point-based metrics are mismatched for such anomalies (see Appendix I for discussion).

N Model Complexity and Overheads

We study and compare the complexity and time overheads of all baselines and SARAD. Concretely, we evaluate the model complexity by the number of parameters being used and the total training time as well as the inference time per sample. All experiments on time overheads are performed on a compute node with AMD EPYC 7443 (48 cores, 96 threads) CPU, NVIDIA A10 (24GB) GPU, and 512 GB RAM.

Table 15 reports the total training time (in minutes), the inference time per sampling point, and the number of network parameters used (where applicable) of all baselines on the main datasets. The size of the training set T , the number of features N , and the sampling period T are also reported per dataset for easy reference. SARAD, while not the fastest nor the lightest model, incurs moderate time overheads and model complexity.

We note that even SWaT, the smallest dataset in terms of actual clock time, spans approximately 6 days for collection. This is far exceeding most detectors' training time besides MAD-GAN and levigates concerns for training overheads for them. All detectors also incur an inference per sample time several magnitudes below the smallest sampling frequency of 1 second, guaranteeing real-time deployment of all detectors once trained.

O Baselines

We trained all baselines using official implementations where available and recommended hyperparameters from respectively papers are used. Some baselines, such as DiffAD and AT, have dataset-specific hyperparameters and here we adopted them as well. The open-accessed URLs of the baselines used are listed as followed.

IF (ICDM'08) (Liu et al., 2008): https://github.com/xuhongzuo/deep-iforest .

DIF (TKDE'23) (Xu et al., 2023): https://github.com/xuhongzuo/deep-iforest .

TranAD (VLDB'22) (Tuli et al., 2022): https://github.com/imperial-qore/ TranAD .

ATF-UAD (NN'23) (Fan et al., 2023): https://github.com/wzhSteve/ATF-UAD .

AT (ICLR'22) (Xu et al., 2022): https://github.com/thuml/Anomaly-Transformer .

DCdetector (KDD'23) (Yang et al., 2023): https://github.com/DAMO-DI-ML/ KDD2023-DCdetector .

USAD (KDD'20) (Audibert et al., 2020): https://github.com/manigalati/usad .

GDN (AAAI'21) (Deng and Hooi, 2021): https://github.com/d-ailin/GDN .

MAD-GAN (ICANN'19) (Li et al., 2019): https://github.com/LiDan456/MAD-GANs . Official implementation was migrated to pyTorch for uniform environmental set-up. See our codebase for details.

DiffAD (KDD'23) (Xiao et al., 2023): https://github.com/ChunjingXiao/DiffAD .

10

2

10

1

10 0

10 1

Data Reconstruction Error (

r

)

10

3

10

2

10

1

Progression Recon. Error (

p

)

Anomalous

Normal

(a) SMD.

10

2

10

1

10 0

Data Reconstruction Error (

r

)

10

3

10

2

Progression Recon. Error (

p

)

Anomalous

Normal

(b) PSM.

10

2

10

1

Data Reconstruction Error (

r

)

10

3

10

2

10

1

10 0

Progression Recon. Error (

p

)

Anomalous

Normal

(c) SWaT.

10

3

10

2

10

1

10 0

10 1

10 2

Data Reconstruction Error (

r

)

10

3

10

2

10

1

Progression Recon. Error (

p

)

Anomalous

Normal

(d) HAI.

P Joint Detection Criterion

Figure 12 visualizes the two components of the joint detection criterion s in Eq. 5, i.e., the data reconstruction error r and the progression reconstruction error p . Balanced resampling is applied here. Recall that s is the sum of normalized r and p . Most anomalous samples (input series) either has high r or high p , and oftentimes both. The former measures the magnitude of anomalousness in the data space, the latter in the spatial association space. The basis underpins the formalization of the joint detection criterion.

10

6

10

5

10

4

10

3

10

2

10

1

10 0

10 1

10 2

Anomaly Score

0.0

0.2

0.4

0.6

Density

Normal

Anomaly

(a) SMD.

10

4

10

3

10

2

10

1

10 0

10 1

10 2

Anomaly Score

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Density

Normal

Anomaly

(b) PSM.

10

5

10

3

10

1

10 1

10 3

Anomaly Score

0.0

0.5

1.0

1.5

2.0

2.5

Density

Normal

Anomaly

(c) SWaT.

10

4

10

2

10 0

10 2

10 4

Anomaly Score

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Density

Normal

Anomaly

(d) HAI.

Q Distributions of Anomaly Scores

Figure 13 the distributions of the joint anomaly scores. On SMD and PSM, the scores of the normal and anomalous samples (input series) overlap more heavily than on SWaT and HAI. The observations here highlight the difficulty of anomaly detection in the service monitoring space, more so than in industrial control where measurements and actuation result from well-defined control logics. The hardness is evident is Table 3 of detection performance and 4 of diagnosis performance.

SMD

PSM

SWaT

HAI

TPR

↑

FPR

↓

(@TPR)

TPR

↑

FPR

↓

(@TPR)

TPR

↑

FPR

↓

(@TPR)

TPR

↑

FPR

↓

(@TPR)

Method

1%

5%

10%

90%

95%

99%

1%

5%

10%

90%

95%

99%

1%

5%

10%

90%

95%

99%

1%

5%

10%

90%

95%

99%

IF

0.29

3.02

7.22 78.49 86.46 93.63 0.50 3.50 11.09 80.94 89.64 94.26 32.88 56.45 65.77 48.48 56.06 64.54

4.15 18.14 28.67 57.31 61.96 68.40

DIF

1.56

8.43 15.17 79.37 87.22 92.33 0.23 1.77

4.63 78.51 83.80 91.86 48.07

63.25 69.63 41.18

50.41 59.62 31.32 41.90 49.40 48.56 54.21 58.81

TranAD

0.86

4.25

8.64 87.86 91.53 95.21 1.15 2.58

7.62 75.94 87.46 96.24

0.12

0.57

1.61 75.78 81.11 85.72 21.39 31.48 37.79 53.60 58.50 66.99

ATF-UAD

0.04

1.04

4.42 90.65 93.75 96.61 0.43 2.05

5.67 82.90 89.02 95.95

0.23

1.02

2.16 69.66 75.93 81.20 16.93 24.21 31.12 60.42 65.08 69.25

AT

1.47

3.78

3.94 99.77 99.77 99.77 0.56 2.73

5.43 99.77 99.77 99.77

1.43

4.07

7.68 98.46 98.46 98.46

4.37 11.61 17.51 95.60 96.91 99.11

DCdetector

0.32

3.40

8.10 97.47 98.43 99.07 0.25 1.88

4.32 99.85 99.85 99.85

0.91

2.19

2.19 98.48 98.48 98.48

N/A

N/A

N/A

N/A

N/A

N/A

USAD

0.38

3.48

7.55 84.13 89.58 94.09 0.55 1.85

5.13 87.22 90.07 92.05 33.13 45.56 52.37 60.13 70.99 74.63 17.82 25.89 33.17 57.63 61.71 70.61

GDN

1.75 12.03 22.41 70.54 77.88 85.00 0.50 3.12 11.77

68.10 74.60 84.05

1.34

1.91

2.57 41.72 50.87 59.36 35.91 48.13 55.29 44.44 50.43 55.77

MAD-GAN

2.66

15.92 24.81 78.00 84.93 90.21

3.28 8.89 15.61

79.78 85.25 93.30 22.43 60.97 67.95 46.78 59.60 70.56 42.69 52.38 58.18 47.93 54.24 59.33

DiffAD

0.55

7.74 16.10 81.87 88.12 93.06 0.58 3.83

8.22 86.78 93.18 98.33

0.51

3.01

5.88 98.46 98.46 98.46

9.31 27.85 48.94 29.57 34.39 37.69

Ours

1.44

16.42 36.41 48.19 57.97 68.38

0.71 5.85

9.60 82.33 90.10 96.65

53.87

60.11 68.83 42.85

50.24 59.05 58.26 83.03 88.82 11.13 19.76 27.01

R Detection Performance Under Thresholds

Effectiveness of each anomaly detector is influenced by the selection of thresholds. Previously, we reported the range-based and point-based threshold-independent performance metrics. Here, we study the influence of threshold selection on the detection performance. Table 16 reports the range-based True Positive Rates (TPRs) under thresholds set by pre-defined False Positive Rates (FPRs)as well as the FPRs under different thresholds set by pre-defined TPRs. Table 17 reports the standard deviations.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims clearly reflect the paper's contributions and scope of time series anomaly detection in the abstract and Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Appendix B.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclose full details of our experiments in Appendixes F and O.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open access to the data and code in Appendix G.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We disclose essential experimental settings in Section 4.1 and full details in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report standard deviations of the experiments in Appendices K and L. Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We disclose information on compute resources for experiments in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts in Appendix A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no risks of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite and properly credit all datasets, baselines, and libraries used in this paper (see Appendices E and O for details).

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve IRB approvals and equivalent approvals/reviews.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript proposes a spatial association reduction method for anomaly detection and diagnosis (SARAD) where anomalous features have low association values in reconstructing the multivariate time-series. The association values are derived from an attention between the features and exhibit changes with time which enables the detection of anomalous features and the time period of anomalies. SARAD is built on two models: a transformer for data reconstruction and an MLP for the reconstruction of spatial progression. The last anomaly score is a combination of both reconstruction scores.\n\nSTRENGTHS:\n- The proposed methods can be useful to detect the source of anomalies in multivariate time-series data.\n- The manuscript provides a sufficient evaluation.\n\nWEAKNESSES:\n- In my opinion a main issue regarding the anomaly diagnosis should be addressed. For example, in Fig. 1 there are three features (9, 13 and 16) correlated with 12 and 15. An additional example is from Fig. 5 where features 13 and 16 are highly correlated with 12 and 15. What distinguishes the detected features from their correlated features?\n- There was no discussion about the identity shortcut problem. Such a problem is typical for reconstruction-based methods where the model learns to reconstruct anomalies and thus it becomes harder to separate them from the normal ones. Maybe I missed it in the manuscript, but can you elaborate on this issue? How do you address this issue in your method?\n- The Point-based evaluation on PSM dataset indicates some weaknesses regarding the sensitivity to the window size and short-term anomalies (see Table 13). The clustering for SMD and PSM is also not of the same quality compared to other datasets (see Figs. 12 and 13).\n- The model size/parameters of the proposed method exceed other baselines by big margines (except for DiffAD).\n\nQUESTIONS:\n- How is the threshold defined from the anomaly scores to detect anomalies? \n- Can you please provide a brief explanation why progression-based $p$ (SPR) is less effective for SWaT?\n\nMinor:\n- I find it a bit confusing to call it spatial association for time-series processing. Maybe it is better to call it channel/feature/variable.\n- It is clearer if you visualize or indicate the ground truth for anomaly diagnosis alongside the detected features i.e., in Figs. 1, 3, and 4-7.\n- Lines 84-85: Can you please make it clearer which spaces you are refereeing to? I think you mean data space and spatial progression space.\n\nLIMITATIONS:\nI think the issues about anomaly diagnosis and the identity shortcut should be discussed (see weaknesses).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-30"}

comments: SUMMARY:
In this paper, the authors aim to address the problem of anomaly detection and diagnosis for time series data. The authors consider that the existing methods may obscure or dilute the spatial information and the interaction between different variates, they propose the SARAD model to capture these interactions with the transformer. The authors evaluate the proposed method on several datasets and achieve good performance.

STRENGTHS:
N.A.

WEAKNESSES:
However, there are some problems to solve  
1.	The authors claim that existing methods may assume feature independence or combine variables of a diverse physical nature. Indeed, the recent methods for time series analysis usually employ independent channel assumptions. However, several methods like the conventional RNN and the graph-based methods do not assume that the features are independent. Therefore, the motivation of this paper might not be convincing.  
2.	Moreover, Figure 1 might be not clear to explain the motivation of the paper. Specifically, Figure 1(a) is not informative and Figure 1(b)(c)(d) are almost the same.  
3.	The contribution of the proposed method is limited. There are other methods that use transformers as the backbone networks like [1]. It is suggested that the authors should provide a detailed discussion and compare with it.  

[1] Xu, Jiehui, et al. "Anomaly transformer: Time series anomaly detection with association discrepancy." arXiv preprint arXiv:2110.02642 (2021).

QUESTIONS:
Please refer to weaknesses

LIMITATIONS:
Please refer to weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

comments: SUMMARY:
In this paper, the authors focus on incorporating spatial information for time series anomaly detection and diagnosis. The proposed algorithm, SARAD, employs a transformer-based data reconstruction approach to capture inter-feature associations. By analyzing changes in these associations over time, the algorithm identifies anomalies under the assumption that anomalous features cause a reduction in the perceived association.

STRENGTHS:
The idea of utilizing spatial association descending patterns for time series anomaly detection and diagnosis seems interesting. 
The authors proposed new criteria for calculating anomaly scores. 
Detailed experimental analysis has been performed to validate the proposed method.

WEAKNESSES:
The related work section lacks a detailed discussion of all the baseline methods used for comparison in result analysis. For a more comprehensive understanding and to highlight the contributions of the paperwork, it would be better to incorporate those methods and discuss their limitations.

The third paragraph in the Introduction section is not easy to follow.  Providing more background information and reorganizing would be recommended.

QUESTIONS:
I'm interested in understanding how the SARAD method distinguishes reductions in spatial associations that indicate anomalies from those that could be considered normal variations. Please clarify the types of anomalies the SAR phenomenon is effective for.

The authors could have enhanced their study by comparing SARAD with existing methods (e.g., InterFusion), which integrate both spatial and temporal information for anomaly detection

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper proposes a deep learning based anomaly detection method for multi-variate time series data. The proposed method has two components in a single neural network. First component is a traditional auto encoder approach in which the input time series is broken into two parts (along time) and the model is trained to use the first part to generate the second part, and identify anomalies as a difference between the predicted and true output. The second component trains a transformer to generate the associations across the different features (akin to cross-correlation) and then train an MLP to predict the associations and identify anomalies if the predict associations are different than the observed associations. Thus, the method captures a broader range of temporal anomalies. \n\nExperimental results on a variety of benchmark data sets and comparisons with state of art methods demonstrate that the proposed method (SARAD) is able to identify anomalies that might not be apparent to approaches that do not consider the associations.\n\nSTRENGTHS:\nPaper is well-described. The idea of using association seems novel, though I have seen papers using cross-correlation to understand the multi-variate effect. \n\nResults are promising. The approach is compute intensive (scaling quadratic with the number of features), but authors do acknowledge the limitation and outline possible approaches to work around it.\n\nWEAKNESSES:\nA terminology issue - spatial often refers to data with positional coordinates (geographic or xy on a domain) which indicates spatial relationships between observations. On top of that, each observation would be represented as a multi-dimensional feature vector. In this paper, the authors use the term spatial to refer to the multi-dimensional feature vector representation of data. I found that very confusing. I think it might be better to fix the terminology to improve readability of the paper.\n\nQUESTIONS:\nAre there any types of anomalies that would still not be captured by this method?\n\nSimilar to above, what kind of false positives could be captured by adding the association part?\n\nLIMITATIONS:\nThe paper has addressed the limitations - I have suggested a couple above. Paper does not have a direct societal impact so the authors have a reasonable response to that.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-22"}