SARAD: Spatial Association-Aware Anomaly Detection and Diagnosis for Multivariate Time Series
Authors: Zhihao Dai
Ligang He
Shuanghua Yang
Matthew Leeke
Venue: NeurIPS 2024 poster (2024)
Keywords: multivariate time series
anomaly detection
anomaly diagnosis
spatial associations
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: An approach that leverages spatial information to improve the detection and diagnosis of time series anomalies.

Abstract: Anomaly detection in time series data is fundamental to the design, deployment, and evaluation of industrial control systems. Temporal modeling has been the natural focus of anomaly detection approaches for time series data. However, the focus on temporal modeling can obscure or dilute the spatial information that can be used to capture complex interactions in multivariate time series. In this paper, we propose SARAD, an approach that leverages spatial information beyond data autoencoding errors to improve the detection and diagnosis of anomalies. SARAD trains a Transformer to learn the spatial associations, the pairwise inter-feature relationships which ubiquitously characterize such feedback-controlled systems. As new associations form and old ones dissolve, SARAD applies subseries division to capture their changes over time. Anomalies exhibit association descending patterns, a key phenomenon we exclusively observe and attribute to the disruptive nature of anomalies detaching anomalous features from others. To exploit the phenomenon and yet dismiss non-anomalous descent, SARAD performs anomaly detection via autoencoding in the association space. We present experimental results to demonstrate that SARAD achieves state-of-the-art performance, providing robust anomaly detection and a nuanced understanding of anomalous events.

SARAD: Spatial Association-Aware Anomaly Detection and Diagnosis for Multivariate Time Series
Zhihao Dai
Ligang He* Department of Computer Science University of Warwick Coventry, UK 1 ligang. www.whwwwwww...... uk
Department of Computer Science University of Warwick Coventry, UK
zhihao.co .ataa..daiie wwa.addaaaarrrr Srai@warwiche ck. ac. uk
Shuang-Hun Yang
Matthew Leeke
Department of Computer Science
School of Computer Science University of Birmingham
University of Reading Reading, UK shuang-hit hua. ywg.eraggr....ngg....mmmgo.mg... ac. uk
Birmingham, UK
m. w.leeke@bham... ac uk
Abstract
Anomaly detection in time series data fundamental to the design, deployment, and evaluation of industrial control systems. Temporal modeling has been the natural focus of anomaly detection approaches for time series data. However, the focus temporal modeling obseure dilute the spatial information that be used to capture complex interactions in multivariate time series. In this paper, we propose SARAD, approach that leverages spatial information beyond data autoencoding errors to improve the detection and diagnosis of anomalies. SARAD trains a Transformer learn the spatial associations, the pairwise inter-feature relationships which ubiquitously characterize such feedback-comtrolled systems. As new associations form and old ones dissolve, SARAD applies subseries division capture their changes over time. Anomalies exhibit association descending patterns, key phenomenon we exclusively observe and attribute to the disruptive nature of anomalies detaching anomalous features from others. To exploit the phenomenon and yet dismiss non-anomalous descent, SARAD performs anomaly detection via autoencoding in the association space. We present experimental results to demonstrate that SARAD achieves state-of-the-art performance, providing robust anomaly detection and nuanced understanding of anomalous events.
Introduction
Time series anomaly detection is critical for industrial automation (Rieth et al. 2018). intrusion detection (Mathur and Tippenhauer 22166), and healtheare sensing (Goldberger et al. 2000). Anomaly detection these contexts typically treated as an unsupervised learning problem, owing to the novelty of anomalies and the scarcity of labeled anomalies.
Temporal modeling is the mainstream basis of current time series anomaly detectors. By learning the dependencies between discrete time steps, temporal modeling can pinpoint the time spans of anomalies. During anomalies, unseen and peculiar temporal dependence patterns degrade data autoencoding (Wang et al. 2023b) or autoregression (Zhao et al. (020)) performance, thereby enabling detection. Alternatiely, irregular temporal representations can driven to breach learned enclosing
"Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

19324 305
(a) Raw time series before, during, and after an anomaly Pi.
range: 16,0322 11941
DN 1A 3.11900
mage: 0032, 6.2900
magge 10,0000, 6064)
TaTON 1.0300)
412
A12
features
Features
features
featuuees
(b) Pre-p,'s A.
(c) In-p.'s A. (d) Post-p,'s A. (e) Pre-reduction. (f) Post-reduction.
Figure I: Spatial associations captured by Transformer service monitoring benchmark, Ia shows the raw time series right before, during, and after an anomaly (colored in red). Association mapping A by final L-th ayeers MHSA are averaged across heads to derive A before (Ib). during Icp. and after (Id) the anomaly. Darker cells have larger values. Anomalous features (#I2 and #15) highlighted with red bounding boxes. The reduction-onle changes from before or after the anomaly to during the anomaly are shown in Ic] and If i.e., ReLU (Are Ain) and ReLU(App A. The anomaly leads to association reductions on anomalous features, prominently column-wise on A.
hyperspheres (Shen et al. 2020), resulting in high anomaly scores that enable detection. Despite its temporal precision in anomaly detection, temporal modeling either assumes feature independence or combines variables of diverse physical nature, the former simplifying the modeling and the latter mitigating the multicollinearity issue. Such assumptions lead to either omission or dilution of spatial information crucial to anomaly detection. Specifically, it overlooks the long- long-time-rann time-range spatial associations, the relationships between various features which ubiquitously characterize normal behaviors of multivariate time series. Where anomaly detection pinpoints the temporal locations of an anomaly, anomaly diagnosis identifies the spatial locations, i.e., the anomalous feature set, of an anomaly. Temporal methods also restrict diagnostic capabilities, the lack of spatial information mismatches autoencoding-bart based or autoregression-batt -ased anomaly criterion, which de facto measures temporal novelty, with its objective of capturing spatial novelty.
Furthermore, time series anomalies frequently dissolve spatial associa- tions, motivating anomaly detection in the association space. Using vanilla Transformer (Vaswani et al., (017)). we investigate the changes in spatial associations throughout anomalies. Applied transposed time windows (the spatial dimension comes before the temporal), an encoder-only Transformer is trained minimize reconstruction errors on unlabeled N-variate time series and, by doing so, learns to model the multivariate series spatially via the Multi-Head Self-Attention (MHSA) illustrated in Figure MHSA at each stacked (-th layer computes an inter- mediate association mapping A,, E N per /-th head, mapping back input X' to produce attention scores. The last layer's 's mapping A thus effectively captures the contributions of k-th feature the reconstruction of j-th feature at each location (j. k), not least for its architectural prox- imity to the reconstructed output. Recent research (Liu et al.) (202)) also highlights the important role MHSA plays in capturing the inter-feature associative relationships when applied on the multi-variate dimension. As new associations emerge and old ones dissolve over time, Figure I shows the association changes on real-world benchmark. We observe that anomalies exhibit reductions for anomalous features, phenomenon we herein coin Spatial
Linear
Dot Product
A
Seftror
Tranepose
Linear
Linear
Linerr
Figure MHSA.

---

Association Reduction (SAR). The rationale is that anomalies either originate from or result in dissolution of pre-existing associations, detaching anomalous features from their -on-aoomal counterparts. Additionally, we make the observation that SAR is most prominent column-wise on A since each j-th column characterizes the dropouts of from associating with other, mostly non-anomalous, features. Due to lack of explicit spatial information, temporal modeling is inadequate for exploiting SAR. More examples are given in Appendix C
From spatial modeling perspective, propose SARAD to leverage spatial information and to exploit SAR for enabling robust time series anomaly detection and diagnosis. For quantifying anomalous spatial novelty in the data space, we train a Transformer on transposed time windows as an autoencoder. To capture the spatial association progression, the reduction-only changes of associations over time, the data reconstruction divides the input window by time into two halves to be processed in parallel. Consequently, the progression is the non-negative backward difference of the intermediate association mappings via MHSA. Subseries division cireumvents memory storage of latest association mappings and enables time window shuffling during training, which reduces order bias, enhances generalization, and prevents catastrophic forgetting. For quantifying anomalous reduction novelty, we train Multi-Layer Perceptron (MLP) as an antuencoder on progression in the association space. Whereas progression encompass all association reduction, autoencoding rules out those not caused by anomalies. The reconstruction errors via the data module measure data-only anomalous deviation from expected system behaviors and falter when such deviations not prominent, c.g., at the start of an anomaly. The reconstruction errors via the progression module are sensitive to change in spatial associations, thus complementing the former. We develop joint anomaly detection criterion that combines both. Experiments show SARAD delivers state- of-the-art detection and diagnosis performance with architectural elegance. Code is available at https: bttpss//gg //i wtw.ttoooccoeoeeeeeetoootttttttt /a////ddddaaaaaaaaaa
We summarize our contributions as follows.
We reveal and extract spatial association descending patterns of time series anomalies with a bespoke Transformer and subseries division. The former learns the pairwise inter-feature associations via autoencoding in the data space and the latter enables shuffled autoencoding training and memory- progression aggregation.
We propose progression autoencoding to quantify anomalous descent in the association space and a joint detection criterion in both data and association spaces, which complement each other.
Experimentally, SARAD performs state-of-the-ar anomaly detection and diagnosis multivariate time series and ablation studies support our design choices.
2 Related Work
Influenced by the dominance of temporal modeling in time series forecasting (Wang et al. 2023a Zhang et al. [023] Wu et al. 2021). temporal modeling is also prevalent in time series anomaly detection. Recurrent neural networks such as LSTM (Hochreiter and Schmidhuber 1997) have
innate capabilities for handling sequential data. These approaches use hidden states for past input memorization, enabling detection (Li et al. 2019: Malhotra et al. 2015) and diagnosis QQaa et al. 2021). Transformer /Vaswani et al. 2017) network is widely adopted (Fan et al.] 2023] Xu et al. 2022) approach that is commonly applied to model temporal associations between different time
points using its attention mechanism. Linear regression (Zeng et al. (203) and MLP (Wang et al., 2024, Audibert et al.] 2020) directly model temporal dependencies. TranAD (Tuli et al.) 2022) replaces the ML. in Audibert et al. (2020) with a Transformer, making the detection criterion more robust through its adversarial training paradigm. Temporal modeling. however, is restricted by the exceptionally small receptive field in time and adversely impacted by the timestamp misalignment across features. In the context of anomaly detection, temporal modeling helps capture anomalous temporal associations (Xu et al. 2022 Yang et al., (023), ) but offers limited detection capabilities in absence of spatial information. a diagnostic context, temporal detectors mismatch anomaly criterion of temporal novelty with spatial interpretation.
Spatial associations characterize the multivariate time series commonly found in such supervisory systems for industrial control. The relationships range from strongly correlated, e.g., due to spatial proximity, to fully independent, e-g., due to mechanical disconnection. For forecasting, iTTrans- former (Luu et al. 2024) applics Transformer on the transposed time series to enable direct spatial
3

---

modeling. Crossformer (Zhang and Yan 2023) screens the time series through custom Two-Stage Attention layers for more efficient spatial modeling. In terms of detection, GDN (Deng and Hooi 2021) learns directed graph of features for the prediction of last time points, whose errors serve as anomaly scores. GDN is partially limited by a mismatch between its singe-timestamp prediction target and the prevalent range-wise anomalies
well as unstable Top-K node selection during training. InterFusion (Li et al. 2021) learns compressed spatial and temporal dependencies, using a hierarchical Variational Auto-Encodee (Kingma and Welling 2014) to reconstruct the series. Neither inspects temporal changes in associations throughout anomalies.
On another front, Isolation Forest (IF) models build binary decision tree ensemble by partitioning either the data space (Liu al. 2008) or the deep embedding space (Xu et al. (2023) formed by randomized neural networks. They are constrained by the lack of temporal and spatial (in the former case) or spatial (in the latter case) information, and their anomaly scores are not reflective of the degrees of anomalies.
We emphasize anomalous association descending patterns towards better time series detection and diagnosis. Different from previous work, we explicitly utilize the reduction in spatial associations over time during an anomaly, an insight we derived from the cyber-physical defense space. Dynamic watermarking (Satchidanandan and Kumar (017) and similar defense techniques (Dai et al. 2023) overlay actuation with private signals reveal attacks resulting in correlational breakdowns. While their approaches are intrusive and actively alter system behaviors, our detector remains non- onn-inruusive, passively monitors the spatial associations, and is applicable to any supervisory system.
We refer to spatiality in this work as the multi-dimensional vector nature inherent to multivariate time series data. The terminology is also used in literature time series related tasks (Gangopadhyay iet al. 2021; Zheng et al., (023). We note that spatiality may carry different meanings in other AI contexts, such as geographic positions or characteristics on Earth. We differentiate those meanings from our definition of spatiality, which traces its root to the spatial distribution of sensors and actuators in control systems where time series are routinely collected.
3 Method
The problems of anomaly detection and diagnosis are specified as follows.
Anomaly Detection Given N-feature time series T = {a:1 ... , ,NN)) where x² RT is of the same length T, the objective is to predict the anomaly label y/. E (0,1) 1) at each timestamp t.
Anomaly Diagnosis Given the same time series T, the objective is to predict the diagnosis label gr S [N], the set of anomalous features at each timestamp t.
3.1
Overview
SARAD comprises two sequential modules; Transformer for time series data reconstruction and MLP network for spatial progression reconstruction. [aable[]decoomosee the system framework of SARAD. The Transformer temporally divides by 2 and reconstructs the input time series to learn pairwise inter-feature associations and to enable order-free memory-efficient progression aggregation. The MLP reconstructs the aggregated progression to quantify anomalous association reduction while dismissing non-anomalous reduction. Towards robust anomaly detection, reconstruction errors from the two modules jointly serve a criterion, sensitive to data-only anomalous deviation and anomalous association reduction.
3.2 Data Reconstruction
In light of restricted capabilities of temporal detectors, here we adapt Transformer to spatially reconstruct the series data. The data module contains two components, Subseries Split & Merge and Subseries Reconstruction, shown in the first and second columns in Table The former wraps around the second by temporally splitting a multivariate input series in half at its beginning and temporally merging at its end. Subseries division enables capturing of spatial progression within a single time window. Without the former, the model must store in memory the last association mappings at each step and keep to the time ordering during training, which is prone to overfitting and

---

Table 1: SARAD is a composition of two modules and three components: data reconstruction (subseries split and merge, and subseries reconstruction) and spatial progression reconstruction.
Data Reconstruction
Progression Recon.
Subseries Split Merge
Subseries Recon.
Architecture
Encoder-onl onlyy Transformer
MLP
X
Subseries Split:
Lineur
X1
X;
Encoder
x.
MLP
$
MLP
Aggregate
Diagram
Subseries Merge:
(A
(8
ReLU
x:
{A
{
Embedding
Time
Training Ohjective
min| || X X|1 || X
min ||$ S12
Detection Criterion Diagnosis Criterion
||S S S ((.) S
(1-) - X (1-1
catastrophic forgetting. The latter is an encoder-only Transformer composed of an embedding layer, L-layer attention-based encoder, and finally a linear projection layer. Encoder-only Transformers are commonly found in Transformer-basee detectors (Kang and Kang 2024 Kim et al. (023) due its simplicity and the length uniformity of the target output, i.e., the reconstructed series. Ours exclusively models spatial associations, unconventional to the aforenamed detectors and most temporal forecasters (Nie al. 2023 Zhou et al., 2022; Liu et al.) (022) and yet more aligned with recent spatial-awart forecasters (Liu al. 20244 Zhang and Yan (023). At each encoding layer, MHSA computes pairwise association mappings, representation of inter-feature dependencies which ubiquitously characterize the multivariate time series and are crucial to anomaly detection.
Subseries Split and Merge We suppose the input series a time window X 2 of length 2W, where is for the convenience of temporal split. Before reconstruction, X is split into two half multivariate subseries equal temporal length: X = {X, E W xN X₂ E R& xN After subseries reconstruction, the two reconstructed subseries are concatenated to form the full reconstructed X (X1 E R RWXN X2 E NNN
Embedding To lead subseries reconstruction, each X; is embedded as x. E, + M, wherein E, Linear( (?T) E N and M = {m, E R E [N]} is learnable feature-lever embedding.
Spatial-Aware Encoding stack of L Transformer encoding layers is used to encode the series in the D-length attention space. Each layer is stacked with MHSA and MLP with residual connections:
z LN(MMSSA
x
x LN(MLP(Z)) + 2
(1)
where X z!_ x E NxD are - layer's output, 7-th layer's hidden state and output respectively and (.N Layer Normalization (Ba et al. 2016). is apati because MHSA explicitly learns pairwise inter-fcature association mapping to exchange information in between feature
5

---

representations. Notably within the MHSA with H heads as shown in Figure each (j, k)-th element on the association mapping A. E RNNN of its h-th head computes how much j-th feature's 's attention scores should originate from the k-th feature's key. From a broader perspective of data reconstruction, it the quantification of the residual impact of k-th feature's originals on the j-th feature's reconstructions. We refactor MHSA implementation to enable parallel encoding of subseries.
Linear Projection Output from the last encoding layer X² is linearly projected and transposed to derive the reconstructed subseries X; E
3.3 Spatial Progression Reconstruction
To exploit SAR caused by anomalies, the module first extracts and aggregates the spatial progression, the non-negative backward difference the association mappings via MHSA. In line with general anomaly detectors (Aggarwal 2013), the module conducts autoencoding in the association space to quantify anomalous SAR and to dismiss non-anomalous SAR. Anomalous SAR occurs when, say, a compromised sensor's readings no longer correlating with its spatially adjacent or mechanically related counterparts.
Association Progression We define association progression S E N at the /-th attention head in the 1-th layer to be the non-negative backward difference in association mappings {A), A2,n
S = ReLU -,2,))
(2)
where ReLU(-)) passes through only non-negative values and outputs zeros otherwise.
Progression Aggregation To center the detection association dropouts, we aggregate the column sums of progression 5) from all attention heads in the final L-th layer to form S IRNN
N
S=
2 S sk. h E [ E [N]}
(3)
We recall from the data module, each k-th column in A quantifies the impact of k-th feature on all features' reconstruction. Taking the sum per each k-th column, we measure with S the dropout rates of k from participating all features' reconstruction. As have observed in Section [ SAR the column level is indicative of time series anomalies, more so than at the row level. The last layer's 's progression is focused not least for its proximity to the final reconstructed output, whereafter more information is exchanged between features. Liu et al. (2024) manifests that final layer's 's mappings resemble closely with the inter-feature correlations of the target, in case, the reconstructed.
Autoencoding With S flattened as an one-dimensional vector, a 2-layer MLP is trained to output the reconstructed and reshaped $ H synchronously with the data module training.
3.4 Joint Training and Anomaly Detection
Training Objective We train an end-to-end model with a joint minimization objective:
LR= X x Ls $
S 2 I. IR At. Ls
where La is the data reconstruction loss. Ls the progression reconstruction loss, and ÀLS weight hyper-parameter. Gradients are stopped from flowing into S to prevent updates to the data module and collapses in association representation. We training anomaly detectors simultaneously, one working in the original data space, the other in the spatial progression space.
Anomaly Detection Criterion
For an input series X, the anomaly score is scalar defined to be: 22 p | S (r pt)/o, + (p - -1pp///pppppp
r= ||
X
(5)
where r the data reconstruction error, p the progression reconstruction error, jrr; Mp the means of T,P on the validation set, and or, op the standard deviation of p. The criterion takes into account the normalized errors in the data space and the progression space, each of which quantifies the anomalous magnitude in respective spaces and complment the other.

---

Table 2: Statistics of the main datasets.
Training
Test
Anomalies
Lengths
Sampling
Dataset
Features
Set
Set
Count
Ratio
min
med
max
Period
SMD
38
708,405
708,420
327
4.16%
2
11
3,161
min
PSM
25
132,481
87,841
71
27.73%
5
8,861
min
SWaT
51
496,800
449.919
34
12.02%
101
447
35,900
sec
HAI
79
921,603
402,005
50
2.23%
17
162.5
422
sec
Anomaly Diagnosis Criterion For an input series X and its j-th feature, its anomaly score 8y is scalar defined to be:
- - || (.)) - X 2
(6)
where ry is feature j's data reconstruction error. The criterion is sensitive to spatial novelty.
Experiments
SARAD is compared against state-of-te sfute-of-the-an art detectors on real-world benchmarks for detection and diagnosis, the latter only when diagnostic labels are available.
4.1 Experimental Setup
Datasets We evaluate on four real-world datasets collected under industrial control and service monitoring settings. These dataset 1) Server Machine Dataset (SMD) (Su et al. 2019b a).
2) Pooled Server Metrics (PSM) dataset (Abdulaal et al. 2021a 3) Secure Water Treatment (SWaT) dataset (Mathur and Tippenhauer 2016 Trust 2023). and 4) Hardware-In wwwwaar-In-thh-Leee -oop--asdd Augmented ICS (HAI) dataset (Shin et al. (2021]aa All traning sets contain only unlabeled data and the test sets contain data with anomaly labels. Anomalies range from service outages external cyber-physical attacks. We summarize the statistics of the datasets in Table Descriptions of each dataset are detailed in Appendix E
Detection Metrics Real-world benchmarks are rife with range- wise anomalies spanning consecutive time points (Wagner et al.] 2023). We use the range-based metrics proposed in (Paparrizos et al.) 2022). Compared against their point-based counterparts, they provide robustness to labeling delay and scoring noises as well as performant detector separability and series consistency. We compute the Aarshhod-iindppnnde parameter-fir Volume Under the Surface (VUS) AUC-ROC and AUC-PR scores. Full details are discussed in Appendix
AUC-ROC and AUC-PR scores to be rid of thresholding impact and fully
Diagnosis Metrics Consistent with previous works (Tuli et al., 2022 Zhao et al.) 2020). we use common metrics such as Hit Rate (HR) (Su et al. 2019b) and Normalized Discounted Cumulative Gain (NDCG) (Järvelin and Kekäläinen, 2002) where diagnosis labels are available. At the range level, we measure Interpretation Score (IPS) initially proposed in [i al. (202T) and here expanded to fit the P% parameterization. Full details discussed in Appendix
Baselines We compare SARAD against state-of-the-arr anomaly detection baselines. including Isolation Forest-based IF (Liu al. 2008), Deep IF (DIF) (Xu et al. 2023); MLP-based USAD (Au- dibert et al., 2020y; graph- grash-based based GDN (Deng and Hooi 2021): L.T L-TM-baaed MAD-GAV GD-GAN (Li et al.
2019); CNN-based DiffAD (iao al. 2003): and Transformer--asse TranAD (Tuli al. 2022). ATF-UAD (Fan et al. 2023). AT (Xu et al. 2022), DCdetector (Yang et al., 2023). Noticeably, GDN employs explicit spatial modeling its graph construction although spatial associations are not directly involved in anoamly scoring. MAD-GAN emphasizes on anomaly detection within cyber-physical systems. All baselines are trained using official implementations where available and recommended hyperparameters from respectively papers used.
7

---

Table 3: Anomaly detection performance. Shreshold-independdee AUC-ROC and AUC-PR metrics and fully parameter- free percentages from five random seeds. The best values are in bold and the second best underlined.
and VUS-PR metrics are reported. All values are average
SMD
PSM
SWaT
HAI
Method
Agoc
ApR
V VRoc
VPR|
AROC
APR
VROC
Vps|
Agoc
ApR
VRoс
VPR Agoc
App
VROG
VpR
IF
53.81
7.27
53.56
7.25
58.08
41.51
57.99 41.48
86.11
66.52
84,39
72.57 10.03
71,65 9.81
DIF
60.27
10.30
59.84
10.23
52.00
36.61 51.88 36.55
89,38
73.19 87.88 70.54
82.10 35.86 81.13
34.16
TranAD
46.86
5.92
46.54
5.88
50,20 35.22 49.47 35.19
47.78
17.65 47.13 17.57
75.60 25.80 75.06
25.41
ATF-UAD
43.41
4,98 43.10
4,97
46.44 33.20
46.03 33.18
55.18
20.66
54.35
20.58
70.56
22.47 69.81
22.06
AT
50.01
5.42
49.97
37.66
26.49 36.82 26.47
46.77
12.95 46.45
12.79.7.44
47.41
5.85
47.24 5.85
DCdetector
49.47
4,51 49.10
4.50
45.94
24.76
46.01 24.82
50.80
14.47 50.76 14.37
N/A
N/A
N/A
N/A
USAD
50.20
6.93
50.01
6.91
42.45 33.23 42.30 33.20
80.36
60.06 78.53 57.33
72.59
23.27
71.84
22.73
GDN
66.37
9.40
66.07
9.34
63.51 40.66
63.13 40.53
79.30
28.12 78.79
28.17
84.79
35.82
84.03 35.05
MAD-GAN
64.35
9.77
64.16
9.74
57.50
40.08 57.37 40.03
86.51
61.95 86.
62.03
84.92
49.06
84.09 48.14
DifFAD
58.71
7.22
58.40
7.19
51.60
32
51.02 32.01
27.02
9.22
26.45
9.21
86.96
21.95 86.25 21.74
Ours
79.97
15.09
79.67
15.02
61.87
41.06
61.77
41.01
88.
72.90
87.52
70.68
96.87
67.78
96.17 64.70
4.2 Results
Anomaly Detection Table 3 shows the anomaly detection performance in metrics defined in Section 4.1 It demonstrates that, despite its architectural elegance, SARAD either outperforms all VUS-ROC scores (SMD: 51%
baselines by significant margins on the
MAD-GAN, HAI: +9.92% DiffAD) or performs on par with current best detectors (PSM: 1.36%
GDN, SWaT: -0.36% DIF). IF scrutinizes the distributional shifts of anomalies with random data partitions and delivers consistent performance across datasets. DIF extends IF into randomized deep representation spaces and archives decent improvements due to more flexible partitions and temporally local information extraction via dilated convolutions. Temporal modeling methods such as DiffAD, ATF-UAD, and AT rely solely or heavily on reconstruction errors and when the errors do not correspond the the underlying anomalies their performance plummet. Adversarial training in USAD and MAD-GAN amplifies reconstruction errors of anomalies to mitigate but not eliminate such issues and thus suffer less performance drops. In contrast, our SARAD additionally accounts for the SAR frequent with anomalies and independent of data distributional shifts, thus outperforming all. SARAD also overpasses GDN, which despite its explicit spatial modeling adopts prediction errors as its sole detection criterion, limiting its performance. SARAD's top performance on SWaT and HAI underlines its ability to unravel complex spatial associations even in complex large-scale systems. Standard deviations of Table 3 are reported in Appendix K
Anomaly Diagnosis Table 4 shows the anomaly diagnosis performance in metrics defined in Section 4.1 DiffAD uses subset of SMD features and thus is discarded from comparisons for fairness. SARAD outperforms baselines on the point-based @10% (SMD: +26.67% TranAD), SWaT: 10% USAD, HAI: 3.81% GDN) and NDCG&150% (SMD: 27.97% TranAD, SWaT: +4.76% GDN, HAI: +2.93% GDN), SARAD also outperforms on the range-based IPS 150% on most datasets (SMD: 43.19% TranAD, SWaT: 33.43% TranAD). Unlike SMD which performs forensic diagnosis to label anomalous features, SWaT and HAI label only the origins of cyberattacks as diagnosis labels. Consequentiallyy attack origins sometimes might not behave anomalously, e.g.. attacks had failed, or the full set of anomalous features were not identified, thus diminishing the performance numbers on SWaT and HAI SARAD generally outperforms detectors underpinned by temporal modeling due to its sensitivity to spatial associative changes. SARAD also outperforms spatial detectors such as GDN whose prediction errors limit its temporal scope to a single time point. Standard deviations of Table 4 are reported in Appendix L
Visualization Figure Segure visualizes real-world anomaly example via SARAD. Our detector captures the significant SAR caused by the anomalous features. The loose reconstruction of the progression raises the pngressioo-bbase score p and, in turn, the joint detection score 8. Taking a broader view of the series in Fig.3h SAR significantly raises the scores at the start of the anomaly, even when the data-based errors are small. SARAD exploits SAR to achieve more robust anomaly detection.
8

---

Table 4: Anomaly diagnosis performance. Point- HR@ NDCG@ P%, and range-based
IPS P% are reported. All values are average percentages from five random seeds.
SMD
SWaT
HAI
Method
HRaP%
ND8
P%
IPS0/%
HR 07%
ND0/%
IPS
HR
P%
ND
P%
IPS
P%
P
100
150
100
150
100
150
100
150
100
150
100
150
100
150
100
150
100
150
TranAD
33.33
45.26
34.71
41.81
22.21
33.25
4.82
6.36
4.82
5.76
17.47
20.91
4.08
6.27
3.98
5.33
12.67
21.78
ATE-UAD
27.80
40.94 26.67 34.46
17.34
26.16
1.85
3.19
1.83
2.65
5.45
8.28
2.96
5.22
3.08
4.48
5.04
8.59
USAD
27.03
39.74 25.83
33.44
18.31
26.07
4.39
7.73
4.39
6.48
12.83
27.17
3.78
5.83
3.76
5.01
12.37
16.22
GDN
28.67
41.57
28.62
36.27
21,18
30.76
5.99
7.21
6.13
6.89
14.04
21.72
5.45
8.29
5.50
7.23
7.41
12.07
DifAA
N/A
NIA
N/A
N/A
N/A
N/A
1.82
2.90
1.81
2.47
5,45
12.63
3.32
4.79
3.41
4.32
12.00
17.26
Ours
56.73
71.33
60.79
69.78
61.38
76.44
9.57
12.83
9.61
11.65
35,45
54.34
6.45
12.10
6.69
10.16
7.48
14.07
ranger 115720, 259403
(0.0006, 0.2200)
10.0006, 0.2200)
features
features
heads
heads
(a)
(b)
(C)
(d)
(e)
(h)
Figure 3: Visualization of applying SARAD for detection on SMD. Ba shows the raw time series right before and during an anomaly p: (colored in red). An input time window for SARAD is bounded in
the black box. 3b and 30 show the average association mapping A via final L-th layer's MHSA. 3d
shows the aggregated progression according to Eq. 3 Be its reconstruction. 31 3g 3h show the
scores and joint according to Eq. 5 per feature. 3i shows the anomaly scores for Bal s segment.
Anomalous features (#1, #9, #10, #12, #I3, #14 ,and #I5) are highlighted with red bounding boxes.
Complexity and Time Overheads SARAD incurs 32 mins for training and and 0.39 ms for inference per sample on HAI, the largest dataset, falling far below the data collection time and sampling frequency. Those numbers are comparable with baselines and detailed in Appendix
4.3 Ablation Studies
Spatial Progression Reconstruction To evaluate the effectiveness of the progression module, we perform ablation studies on its submodules in Table Standard deviations are reported in Appendix Removing the ReLU, i.e., to capture both association increases and reductions, in progression loses the focus on asscoation reduction and impairs the detection performance. Replacing the column sum operation in aggregation with the row sum which characterizes the disconnection of anomalous features from others and is shown be less effectiveness than the column sum representing the drop out rates. Fully concatenating without sum operation dilutes the reduction patterns and significantly hurts the detection, at cost of complexity. For the detection submodule, using the progression directly instead of the reconstruction errors registers reductions as anomalies directly and underperforms except on SMD due to its inability to nule out normal reduction patterns.
Choice of Detection Criterion Table o the detection performance using Eq. (Joint), us- ing only tttt-bbaedd r (DR), and using only progression- based p (SPR). While the data reconstruction robust criterion of anomalousness, SARAD embeds the spatial information into the joint criterion and outperforms either single criterion overall. Standard deviations are reported in Appendix K
Additional ablation studies the choice of diagnosis criterion are detailed in Appendix D

---

Table 5: Anomaly detection performance under progression reconstruction changes.
SMD
PSM
SWaT
HAI
Submodule
[Change
A
App
VROC
VPR
ABoc
Apg.
V
VRoC
V PR.
ARoC
Apg.
VROC
VPR
Agoc
App.
VROC
Vpr
Ours
79.97
15.09 79.67 15.0266187 41.06 61.77 41.01 88.29 72,90 87.52 70.68 96.87 67.78 96.17 64,70
Prog. (F4
no Rel.U
75.43
12.38 75.16 12.35
60.35 40.32 60.21 40.24
87.48 .88 86.75 66.04
95.59 64.81 94,93 61.87
Aggr. (Eq
sum
79.47
15.87 79.17 15.82
63.36
42,41 63.20 42.33
88.19 20.12 87.40
96.56
67.26 95.80 61.21
Aggr. (Eq
no sum
57.31
6.09
56,98
6.08
47.50
33,92 47.34
33.86
86,
69.69 85 26
67.71
91.63
57.49 90,70 55.01
Detection
S directly
80.42
15.79
80.14
15.73
60.59
39.71
60.49
39.63 87.33 69.38
86, 67.39
95.72 64.95
94.98 61.91
Table Anomaly detection performance under different choices of detection criterion.
SMD
PSM
SWaT
HAI
Method Criter. ABoC
ArR
VROG
VPR
Aroc:
APR
VROG
VpR
AROC
APR
VROG
VpR
Aaoce
APR
VROG:
VpR
Ours
both 79.97 15.09 79.67 15.02 61.87
41,06 61.77 41.01 88.29 72.90
87.52 70,68
96.87 67.78
96.17 64.70
DR
ronly 73.21 13.53 72.72
13.44
39.99
61.45 39.93 N6.76 68.66
85.15 67.13
96.28
66.38
95.56 63.33
SPR
polly 79.71
15.44
79.39
15.36
62.41
41.47 62.20
41.36 65.87
20.10
65.31
19.53
95.54
61.73
94.77 59.07
5
Conclusion
In this work, we propose SARAD for time series anomaly detection and diagnosis. The approach effectively exploits the spatial association descending patterns of anomalies. Data reconstruction with Transformer guides learning of spatial associations from data and captured as progression, while progression reconstruction quantifies the anomalous association descent and complements the insensitivity of the former spatial disassociation during anomalies. SARAD experimentally demonstrates state-of-the-ar detection and diagnosis performance and foreshadows the power of spatial modeling for related time series tasks.
Acknowledgments and Disclosure of Funding
Calculations were performed using the Sulis Tier 2 HPC platform hosted by the Scientific Computing Research Technology Platform at the University of Warwick. Sulis is funded by EPSRC Grant EP/T022108/1 and the HPC Midlands+ consortium. The authors acknowledge the use of the Batch Compute System in Department of Computer Science at the University of Warwick, and associated support services, in the completion of this work.
References
Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. 2021a. Practical Approach Asyn- chronous Multivariate Time Series Anomaly Detection and Localization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining (Virtual Event, Sin- gapore) (KDD '21). Association for Computing Machinery, New York, NY, USA, 2485-2494 https://doi: fstpp://doi.org/111 org/ 10. 1011/114414455 3467174
Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. 2021b. RANSynCoders. https:// caaub..co/iccsaaee (tchuu..com/icssaaaaettt//
Charu C. Aggarwal. 2013.
Outlier Analysis.
Springer.
do.dxx.oo.... 1007
978-1-4614- 6396-2
Julien Audibert, Pietro Michiardi, Frédéric Guyard, Sébastien Marti, and Maria A. Zuluaga. 2020. USAD: UnSupervised Anomaly Detection on Multivariate Time Series. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining (Virtual Event, CA, USA) (KDD '20). Association for Computing Machinery, New York, NY, USA, 3395-3404. 1tt.:.0..111//// 11tg///0.11//339448666 .3003392
https://doi
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).
10

---

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for Hyper- Parameter Optimization. In Advances in Neural Information Processing Systems, Shawe- Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (Eds.), Vol. 24. Curran Associates, Inc. https: https://proceedingss www.aeedddnnnnn..iss..s.. newrips wwwpccpppppppeeeeeeee httppppaaeeeeee//aeeaaaaaaaaeeee (tilee/paperr2011/////
ppf
Zhihao Dai. Ligang He, Shuang- Hua Yang, and Matthew Lecke. 2023. Revealing Ongoing Sen- sor Attacks in Industrial Control System Via Setpoint Modification, In 2023 IEFE Inil Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Comput- ing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology CataPP/CCoooCCCCCC/////rerssseee Congress CCDCoo/CCy9911111111 Cy59711 2023.....02... 10361334
0191-0199, https: wtl.d/ii.og/1.1.1. 1109/
DASC/PiCom/ http:/PPicoo//DDoomm//
Ailin Deng and Bryan Hooi. 2021. Graph Neural Network- Network-Based Anomaly Detection in Multivariate Time Series. Proceedings of the AAAI Conference on Artificial Intelligence 35, (May 2021),
4027-4035. httpp:///oi. org/ 10. 1609/ /aaaaaaii..... aaai.v3515. 16523
Jin Fan, Zehao Wang, Huifeng Wu, Danfeng Sun, Jia Wu, and Xin Lu. 2023. An Adversarial Fime-FFr Reconstruction Network for Unsupervised Anomaly Detection. Neural Networks 168 (2023), 44-56. https:///oli org/ 10. 1016/ nwwnet. 2023. .9..199011111 018
Tryambak Gangopadhyay, Sin Yong Tan, Zhanhong Jiang. Rui Meng, and Soumik Sarkar. 2021. Spatiotemporal Attention for Multivariate Time Series Prediction and Interpretation. In ICASSP 2021 202/ / IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 3560-3564
R.110//CASS3999200 2021 9413914
Astha Garg. Wenyu Zhang, Jules Samaran, Ramasamy Savitha, and Chuan Chuan-Sheng Foo. 2022. An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series. IEEE Transactions on Neural Networks and Learning Systems 33, 6 (2022),
https://doi org/10
1109/TNNLS 2021.310582 3105827
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng. and H Eugene Stanley. 2000. PhysioBank,
and PhysioNet: components of a new research resource for complex physiologic signals. circulation 101, 23 (2000),
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short Memory. Neural Computation 9,
8 (1997), 1735-1780. httts:///oii. 10. www.:1111cen.o.. 1997 9. 1735
iTrust. 2023. Datasets.
d
Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated basd of IR techniques. ACM Trans. Inf. Syst. 20, 4 (oet 2002), 422 446. hhttps://doi. 0111/5552211 582418
Hycongwon Kang and Pilsung Kang. 2024. Transformer-based multivariate time series anomaly detection using varialle attention mechanism. Knowledge-Bass Systems 290 (2024), 111507.
https://doi. org/10. 1016/j www.ys.....sy................... 2024. 111507
Jina Kim, Hyeongwon Kang, and Pilsung Kang. 2023. Time-series anomaly detection with stacked Transformer representations and ID convolutional network. Engineering Applications of Artificial Intelligence 120 (2023), 105964.
engappai. 2023. 105964
Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Interna- tional Conference on Learning Representations.
Diederik P Kingma and Max Welling. 2014. Auto-encoding variational bayes. In International Conference on Learning Representations.
Dan Li, Dacheng Chen, Baihong Jin, Lei Shi, Jonathan Goh, and See-Kiong Ng. 2019. MAD-GAN: Multivariate Anomaly Detection for Time Series Data with Generative Adversarial Networks. In Artificial Neural Networks and Machine Learning ICANN 2019: Text and Time Series, Igor V. Tetko, Vera Kūrková, Pavel Karpov, and Fabian Theis (Eds.). Springer International Publishing, Cham, 703-716.
11

---

Zhihan Li, Youjian Zhao, Jiaqi Han, Ya Su, Rui Jiao, Xidao Wen, and Dan Pei. 2021. Multivariate Time Series Anomaly Detection and Interpretation using Hicrarchical Inter-Metric and Temporal Embedding. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (Virtual Event, Singapore) (Kg)))rr)(KDD) (KDD '21). Association for Computing Machinery, New York, NY, USA, 3220-3230. lattps://don org/ 10. 1145/3447544 3467075
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008. Isolation Forest. In 2008 Eighth IEEE International Conference on Data Mining. IEEE, Pisa, Italy, 413-422. httpss//ddi. http///ii.og////////
1109/ICDM. 2008.00000 17
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. 2024. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations. httpp:://penrrevier net/
forum/id=JePFAA8Bal
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. 2022. Non-stationary Transformers: Ex- ploring the Stationarity in Time Series Forecasting. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.). Vol. 35. Curran Associates, Inc., 9881-9892 https://proceeddnggs. wwt.:eewwwrr....ppppppeeeoee _files/
pdf
Pankaj Malhotra, Lovekesh Vig, Gautam Shroff, Puneet Agarwal, et al. 2015. Long Short Term Memory Networks for Anomaly Detection in Time Series.. In Esann, Vol. 2015. 89.
Aditya P. Mathur and Nils Ole Tippenhauer. 2016. SWaT: a water treatment testbed for research and training on ICS security. In 2016 International Workshop on Cyber-physical Systems for Smart Water Nenworks (CySWater). IEEE, Vienna, Austria, 31-36. https::/doo org/ 10. 1109/ CySWater 2016. 7469060
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. A Time Series is Worth 64 Words: Long-term Forecasting with Trannformers. In The Eleventh International Confer- ence on Learning Representations. attps:///penrrvveew.ee
John Paparrizos, Paul Boniol, Themis Palpanas, Ruey S Tsay, Aaron Elmore, and Michael J Franklin. 2022. Volume Under the Surface: New Accuracy Evaluation Measure for Time-Series Anomaly Detection. Proceedings of the VLDB Endowment 15, 11 (2022), 2774-2787
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Ed- ward Yang. Zachary Vetioo, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High- Performance Deep Learning Library. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer,, F. d' d'Alché-Bou Buc, E. Fox, and R. Garnett (Eds.). Vol. 32. Curran Associates, Inc. https:
pdf
Kai Qian, Jie Jiang, Yulong Ding, and Shuang-Huu Hua Yang. 2021. DLGEA: deep learning guided evolutionary algorithm for water contamination source identification. Neural Comput. Appl. 33, 18 (sep 2021), 11889-13 11903. https://doi htt:///oo.org//// 10. 00000//00021100 121000021--21--000 021-0094--1
Cory A. Rieth, Ben D. Amsel, Randy Tran, and Maia B. Cook. 2018. Issues and Advances in Anomaly Detection Evaluation for Joint Automatee Systems. In Advances Human Factors in Robots and Unmanned Systems, Jessie Chen (Ed.). Springer International Publishing, Cham, 52-63.
Bharadwaj Satchidanandan and P. R. Kumar. 2017. Dynamic Watermarking: Active Defense of Networked Sybber-Physica Systems. Proc. IEEE 105, 2 (2017), 219-240. httpps://doi org/ 10. 10111999JPRCC 2016. 2575064
Lifeng Shen, Zhuocong Li, and James Kwok. 2020. Timeseries Anomaly Detection using Tem- poral Hierarchical One-Class Network. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran As- sociates, Inc., 13016-13026. http:::/proceeding. www.eeeerrpp.... wwwuuips.ccpppppr... hctppeeerpppppleeeeeeeeeeeeeseee firerrfilleepaaeeee / h2000//2112227000000000000000310
12

---

Hyeok-Ki Ki Shin, Woomyo Lee, Seungoh Choi, Jeong-Han Yun, and Byung- Gi Min. 2021a. HAI
security datasets. https: httppss//gtaabb.ccmm//sssaattttt
Hyeok-Ki Shin, Woomyo Lee, Jeong-Han Yun, and Byung- Min. 2021b. Two ICS Security Datasets and Anomaly Detection Contest on the HIL--Based Augmented ICS Testbed. In Cyber Security Experimentation and Test Workshop (Virtual, CA. USA) (CSET '21). Association for Computing Machinery, New York, NY, USA, 40..
httpss//ddo org/ 1145/1444444
Ya Su. Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. 2019a. OmniAnomaly. https:// www.ttt...o//eeeaaaaaaaaaaaaa 0m Anooaly
Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. 2019b. Robust Anomaly De- tection for Multivariate Time Series through Stochastic Recurrent Neural Network. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining (Anchorage, AK, USA) (KDD '19). Association for Computing Machinery, New York, NY, USA, 2828-2837.
https: hoops://doi..ooo 11t......1111111111 33306722
Nesime Tatbul, Tae Jun Lee, Stan Zdonik, Mejbah Alam, and Justin Gottschlich. 2018. Precision and Recall for Time Series. In Advances in Neural Information Processing Systems, S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Curran Associates, Inc. 7f119556666e77222200000000000002
and R. Garnett (Eds.), Vol. 31. mewerddddiinnnnnniippp neurips. ppapperfilles/ppppr/2001 pdf
Shreshth Tuli, Giuliano Casale, and Nicholas R. Jennings. 2022. TranAD: deep transformer networks for anomaly detection in multivariate time series data. Proc. VLDB Endow. 15, 6 (feb 2022),
1201-1214. httpss//doi..om
35140677
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances Neural Information Processing Systems, Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https: /ttp::es//poocddiingg wttppporreennnnnn....ip.pppp.ppp saper_files/pee httppp/r..07/////ffff22666666111
neurips cc/ pdf
Dennis Wagner, Tobias Michels, Florian C.F. Schulz, Arjun Nair, Maja Rudolph, and Marius Kloft. 2023. TimeSeAD: Benchmarking Deep Multivariate Semee-Sriessae Anomaly Detection. Transactions on Machine Learning Research (2023). https://openrrevie: http://ppnrevvieeeeeeerrrrrrrrip
Chengsen Wang. Zirui Zhuang. Qi Qi, Jingyu Wang, Xingyu Wang, Haifeng Sun, and Jianxin Liao. 2023b. Drift doesn't Matter: Dynamic Decomposition with Diffusion Reconstruction for Unstable Multivariate Time Series Anomaly Detection. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 10758-10774, httpps:/pproceeeingg..ot neurips. ctweiiiss.ccppppeeee cc/paper files/ hptp:/22222f2222266eec00o...ee22
www.errreerrerreeeeeeeeeeeeeeeee pdf
Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. 2023a. MICN:
Multi-scale Local and Global Context Modeling for Long-term Series Forecasting. In The Eleventh International Conference on Learning Representations.
www.:neeffoorro
id=2t53IDUR1U
Shiyu Wang. Haixu WU, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and JUN ZHOU. 2024. TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting. In The Twelfth International Conference on Learning Representations. httpss///peereviee net het/forrm?id=77oshhee
Haixu Wu, Jiehui Xu, Jianmin Wang. and Mingsheng Long. 2021. Autoformer: De- composition Transformers with Auto-Correlation for Long-Term Series Forecasting, In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 22419-22430. https:// http://rrrceediiggeeeeerrppaaee bccodooossssseencccccaaaaao222- neurips. cc/ ltce/appr__fiiie www.fflllefppaeeee fiiles/paper/222//fteel
.dd
13

---

Chunjing Xiao, Zehua Gou, Wenxin Tai, Kunpeng Zhang, and Fan Zhou. 2023. Imputation based Time-Series Anomaly Detection with Conditional Weight-Ineremental Diffusion Models. In Pro- ceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Long Beach, CA, USA) (KDD '23). Association for Computing Machinery, New York, NY, USA, 2742-2751, https: wttp://diii.oo wtgg///i..og//////// 10. 1145/3580305 3599391
Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, Jie Chen, Zhaogang Wang, and Honglin Qiao. 2018. Unsupervised Anomaly Detection via Variational Auto-Encoder for Scasonal KPIs in Web Applications. In Proceedings of the 2018 World Wide Web Conference (Lyon, France) ( (WWW '18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 187-196. https
//doi.org/10. 10e...1115/188881 3185999
Hongzuo Xu, Guansong Pang, Yijie Wang, and Yongjun Wang. 2023. Deep Isolation Forest for Anomaly Detection. IEEE Transactions on Knowledge and Data Engineering 35, 12 (2023),
12591-12204. hattps://di..rrs 10. 1109/TKDE.. 2023. 3270293
Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. 2022. Anomaly Transformer: Time
Series Anomaly Detection with Association Discrepancy. In International Conference on Learning Representations. http:://ppperreviieereettfooorrp
Omry Yadan. 2019, Hydra A framework for elegantly configuring complex applications. Github. http:///gggttoocccccccceeeeeeeee
Yiyuan Yang, Chaoli Zhang, Tian Zhou, Qingsong Wen, and Liang Sun. 2023. DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (<conf-loc>, <city>Long Beach</city>, <state>C<</state> 223). Association for Computing Machinery. New York, NY, USA, org 11410..8114133330001 3599295
</conf-loc>) (KDD https://doi
Ailing Zeng, Muxi Chen, Lei Zhang. and Qiang Xu. 2023, Are Transformers Effective for Time Series Forecasting? 37 (Jun. 2023), 11121-11128. https://doi.org:
1609/ /a/9///aa....// vv7i9. 26317
Chaoli Zhang, Tian Zhou, Qingsong Wen, and Liang Sun. 2022. TEAD: Decomposition Time Series Anomaly Detection Architecture with Time-Frequency Analysis. In Proceedings of the 31st ACM International Conference on Information Knowledge Management (Atlanta, GA, USA) (CIKM '22), Association for Computing Machinery, New York, NY, USA, 2497-2507. https://dot /ttp://///oo..//1.... 1145/3511808. 1511///5518888.57777700 3557470
Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao. Karan Goel, and Christopher Rc. 2023. Effectively Modeling Time Series with Simple Discrete State Spaces. In The Eleventh International Conference on Learning Representations. https://openrevier. httpe/wwwwwwwee.eeeeeee net/forum?
id id=2Epjes 2EpjkjzcCkk
Yunhao Zhang and Junchi Yan. 2023. Crossformer: Transformer Utilizing Cross-Dimension De- pendency for Multivariate Time Series Forecasting. In The Eleventh International Conference on Learning Representations. pottps://opennee
Hang Zhao, Yujing Wang, Juanyong Duan, Congrui Huang, Defu Cao, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. 2020. Multivariate Time-Series Anomaly Detection via Graph Attention Network. In 2020 IEEE International Conference on Data Mining (ICDM). 841-850. https:///oi.. org/ 10. 1109/ ICDM50108 2020. 00093
Yu Zheng, Huan Yee Koh, Ming Jin, Lianhua Chi, Khoa T. Phan, Shirui Pan, Yi-Ping Phoebe Chen, and Wei Xiang. 2023. Pnrelltion-Awart Spatial-Temporr Graph Learning for Multivariate Time-Series Anomaly Detection. IEEE Transactions Neural Networks and Learning Systems (2023), 1-15. https://aoi.ooe 10. 1109/TNNLS, 2023. 3325667
Tian Zhou, Ziqing Ma. Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu,
14

---

and Sivan Sabato (Eds.). PMLR, 77268-27286.
https://proceedings mlr press/v162/
zhou22g. html
Tian Zhou, Peisong Niu, xue wang, Liang Sun, and Rong Jin. 2023, One Fits All: Power General Time Series Analysis by Pretrained LM. In Advances in Neural Information Processing Systems, A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.). Vol. 36. Curran Associates, Inc., 43322 43355. h
neurdd cctpaper_ffiiaaeceece siapperrffllee/eeeeeeeeeeeeeeeer / 20211110066611066777700277776662 ceatperrCoffeeece..dde
15

---

A Broader Impacts
The broader impact of the work presented rests the increasingly pervasive nature of industrial control systems, intrusion detection systems, and remote monitoring solutions in healthcare contexts, all of which commonly utilize some form of anomaly detection. Further to the time series analysis that is commonplace, the work presented in this paper demonstrates how Transformer can be used to learn the spatial associations that ubiquitously characterize these feedback-controlle supplementing time series analysis to provide state-of-the-art performance. As such, the work presented has broad applicability, whilst explicitly targeting automated industrial control systems.
systems,
B Limitations
While the model size scales linearly with the number of features, the time complexity of SARAD quadratic with respect to the features. SARAD could incur significant training and inference overheads when the supervisory system is extensively large. We caution that the overheads of the largest dataset in our experiments fall well below data collection overhead and sampling frequency (see Appendix N). To scale, will explore hierarchical time series anomaly detection via clustering. Another limitation of this work is the scarcity of forensically labeled datasets like SMD for anomaly diagnosis, least due to the intensive labor and domain knowledge implied. To address that gap, we will explore publicly available audit and operational time series data for sources.
16

---

13X0 10M0
J 4P 29%
(a) Raw time series before, during, and after an anomaly pi.
reope: 10 0035, 0.14431
FRTOK (5,0055, 0.36411
Tange (6 10,0 0.14433
Pange: 00000 0.33671
range: (6.0000 (.0367
restures
1csture
features
features
features
Features
(b) Pre-p,'s A.
(c) In-p,'s 's A.
(d) Post-p,'s 's A. (e) Pre-reduction. (f) Post-reduction.
Figure 4: Spatial associations captured by Transformer SMD (Su et al. 2019b
1275S M210
(a) Raw time series before, during, and after an anomaly Pi.
renpee 10 0,04,0000 0.1370)
FRTOX (5,033A (C.0534
rage: (0 COL0 6.117))
range: 0026, 0.3455)
Fange: (0.0000, 8.6415)
restures
(catuncs
fostures
fealunce
Features
features
features
Features
Seatures
(b) Pre-p,'s A. (c) In-p,'s 's A.
(d) Post-p,'s 's A. (e) Pre-reduction. (f) Post-reduction.
Figure 5: Spatial associations captured by Transformer SMD (Su et al. 2019b).
C Examples of Spatial Association Reduction
As mentioned in Section 1 herein we provide more real-world examples of Spatial Association Reduction (SAR) exhibited by time series anomalies. Figures 4 5 6 showcase the spatial associations captured within Transformer via MHSA. Subfigures in each aforementioned figure show, in that order, (a) the raw time series right before, during, and after an anomaly p. E P (colored in red), (b) association mapping A output by final L-th layer's 's MHSA are averaged across heads to derive average association A before ps. (c) A during Pt. (d) A after pj wherein brighter cells have smaller values and anomalous features are highlighted with red bounding boxes, (e) the reduction-only changes from before the anomaly to during the anomaly, i.c., ReLU(App Am), and finally (f) the reduction-only changes from after the anomaly to during the anomaly, i.e., ReL.U( (Apost
Am))
17

---

313704 (1991)
(a) Raw time series before, during, and after an anomaly Pi.
eutpee 10 0016, 3045)
range L5.0044, 5, 1.9655
range: (0 0.3045)
ENTOP: 100000 0.813)
rarge: (6.000) 5.0134)
Testures
fnatune
reatures
features
fcatures
Features
(b) Pre-p,'s A.
(c) In-p,'s 's A.
(d) Post-p, A. (e) Pre-reduction. (f) Post-reduction.
Figure 6: Spatial associations captured by Transformer on SWaT (Mathur and Tippenhauer 2016).
276 narg 177370 1770370337000 3760831
(a) Raw time series before, during, and after an anomaly pi.
wrop: 10.0033, 0.33
mage: 0000333. o 03533)
magg:: 1C. 0.0033, 33153]
rarge: 0000, 0.3132)
mmge: (0 01302
Teatures
featurss
featuras
festures
features
features
features
features
features
(b) Pre-p,'s 's
(c) In-p.'s 's A.
(d) Post-p,'s A. (e) Pre-reduction. (f) Post-reduction.
Figure 7: Spatial associations captured by Transformer on SWaT (Mathur and Tippenhauer 2016
18

---

Table 7: Anomaly diagnosis performance under different choices of diagnosis criterion.
SMD
SWaT
HAI
Method
Criter.
HR 8%
NDRP%
1P
HR
ND8
IP%
HRRP%
ND
IPS8/%
100
150
Ours
rs only
56.73 71.93 60.79 69.78
61.38
76.44
9.57
12.83
9.61
11.65
35.45
54.34
6.45
12.10
6.69
10.16
7.48
14.07
SPR only
42,97 57.33 46.32
54.85
52.01
64.82
14.32
17.18
14.40
16.18
20.10
37.37
5.71
8.50
5.86
7,58
9.70
17.14
Joint
both
48.91 53.12 60.59 56.56
70.06
2.60
3.50
2.66
16.16
20.40
15.42
10.74
13.92
12.59
16.52
D Choice of Diagnosis Criterion
Concerning the rationality of data-only diagnosis criterion in Eq.
6 we consider an alternate joint
diagnosis criterion in line with the detection criterion in Eq. 5
rЛ (.))) x.>1l P; = (.{,)) - (. 8, - - Mr, -r, Mr,)/o,, +
- -pp,)// (7)
where r, feature j's data reconstruction error, Pj its progression reconstruction error, Pr, Mp, the
means of ry Py the validation set, and Ø,,, Jp, the standard deviation of T. Table compares the diagnosis performance using only ry (SARAD), using only Pj (SPR), and using Eq. (Joint). Unlike in anomaly detection, the great discrepancy between rj and p5 more than often degrades the performance of the joint criterion. SARAD uses ry only which produces suboptimal and yet reliable performance in the longer anomalous horizons. Standard deviations are reported in Appendix L
19

---

E
Datasets
We include four real-world datasets collected under industrial control and IT service monitoring settings for evaluations. Anomalies range from ITT service outages to external cyber-physical attacks against control systems.
E.1 Dataset Descriptions
1. Server Machine Dataset (SMD) (Su et al. 2019b is a server metric dataset from a large-scale IT company. Engineers annotated anomalous events in the second half of the data with indicator-level attributions.
2. Pooled Server Metrics (PSM) dataset (Abdulaal et al. 2021a by captures key performance indicators of servers an online shopping platform. Website engineers annotated anomalous events for data in the last eight weeks.
3. Secure Water Treatment (SWaT) dataset (Mathur and Tippenhauer 2016 iTrst 2023) contains sensor readings and actuator status on minuscule world water treatment system during six-day normal operational period. knowledgeable attacker performed 36 cyber-physical attacks during a five-day attack period and they are labelled as anomalous accordingly.
4. HIL-based Augmented ICS (HAT) dataset (Shin al. 2021b records measurements and control actions within hydropower) generation testbed during its two-week operation. Both single-point primitive and multi-point combined attacks are performed on the testbed to emulate a threat actor with cyber-physical capacities. We use the 21.03 version of HAI.
In-rr---I-teh-ooop (HIL) dual power (stcam-turbine and
All training sets contain only unlabeled data and the sets contain data with anomaly labels. Statistics of the datasets are given in Table
E.2 Lengths of Anomalies
We further characterize the detection datasets by the lengths of the anomalous events. Figure shows the empirical cumulative distribution function of the anomalous lengths. SWaT has the longest median length of 447 among the four datasets considered, followed by HAI (162), SMD (11), and lastly PSM (5). The very short lengths on SMD and PSM benefit temporal detectors which tend embed single few time points (see Appendix M). whereas SARAD adopts a half time window embedding strategy. The catch is that SARAD can learn spatial relationships with temporal aggregated information per feature, while temporal detectors could not, bringing about benefits of performing anomaly detection in the spatial association space. A more scalable approach for temporal aggregation is be explored in the future, though variable window sizes or subseries splits might be implied.
E.3
Lengths of Diagnosis Labels
Figure 8 shows the empirical cumulative distribution function of the diagnosis label lengths. Whereas SMD forensically labels features which deviate from their normal behavioral patterns as anomalous, SWaT and HAT only label the points of attacks anomalous
the their creators have advanced knowledge of such attacks. The latter labeling strategy results in incomplete sets of anomalous features and diminishes the diagnosis performance of all models, as evidence in Table Section
20

---

100%
100%
80%
72%
80%
75%
Proportion
min 2
Proportion
min max 8961
60%
max = 3161
60%
50%
std - 238.42
50%
std - 1226.88
40%
40%
253
25%
20%
20%
0%
0%
10
102
10
10a
10
10
10*
Length of Anomaly
Length of Anomaly
(a) SMD.
(b) PSM.
80%
75%
80%
75%
Proportion
Proportion
60%
60%
5.%
50%
40%
1447
40%
162
25%
25%
20%
1332
20%
100
150
250
300
350
400
Length of Anomaly
Length of Anomaly
(c) SWaT.
(d) HAI.
Figure 8: Empirical distribution function of the lengths of anomalies.
100%
80%
Proportion
min 1 34 std 7.88
60%
50%
118
40%
25%
0%
10
15
2.0
30
Length of Anomaly
(a) SMD.
100%
100%
80%
72%
Proportion
min max 3
Proportion
min max
60%
60%
3
std 0.51
50%
std 0.72
40%
40%
20%
20%
0%
0%
1.00
1.25
1.50
1.75
2.00
2.25
2.50
2.75
3.00
1.00
1.25
1.50
2.25 Length of Anomaly
1.75
2.00
2.50
2.75
3.00
Length of Anomaly
(b) SWaT.
(c) HAI.
Figure 9: Empirical distribution function of the lengths of diagnosis labels.
21

---

Implementation Details
We implement SARAD in Python using pyTorch library Paszke et al. 2019) and Hydra frame- work (Yadan 2019). All experiments are run on single NVIDIA ATO (24GB) GPU. Adam optimizer (Kingma and Ba 2015) is used and learning rate is halved every epoch for 3 epochs to prevent over-fitiigg The time window size is 2W 100. The data reconstruction module has H = 8 attention heads per layer with attention length D 512 and hidden length DFF 2048. For hyperparameter tuning, training set is temporally partitioned into 80% for training and 20% for validation. each dataset we first perform TPE sampling (Bergstra et al. 201T) for number of encoding layers L {3, 5) and learning rate [10-1 10 to derive the best data reconstruction loss Lg on the validation set. The progression module by default has hidden length of Dp = 64. We then perform TPE sampling to search weight ÀLS E [10 2, 10 for the progression reconstruction loss Lg on the validation set.
G Open-accessed Code and Data
During the review period, code is anonymized and openly available https://github. com/ daidahao/ Soldahao/SARAAe /SAADD/ with specific instructions and scripts to reproduce experimental results. All data used in our experiments can be openly accessed from public repositories or requested via original authors' websites. Full links are provided in Appendix
22

---

95
90
90
VUS-ROC (%)
SMD
SMD
80
PSM
80
PSM
SWaT
MUS
SWaT
HAI
HAI
70
60
60
10
20
50
100
200
400
2
(a) Window size.
(b) Training epochs.
90
90
VUS-ROC (%)
SMD
PSM
VUS-ROC
SWaT
75
HAI
SMD
PSM
65
SWaT
60
HAI
60
64
128
256
512
1024
2048
16
32
64
128
256
(c) Attention length D.
(d) Progression hidden length Dp.
Figure 10: Hyperparameter Sensitivity of detection performance in VUS-ROC scores.
H Hyperparameter Sensitivity
We examine the hyperparameter sensitivity of SARAD's detection performance. Concretely, we consider the effects of the sliding window size (default is 100). the number of training epochs (3), the attention length of Transformer encoding layers D (512), and the hidden length of the progression reconstruction module Dr (64). Figure TO and T1 present the results. On the window size, sliding window too small confines temporal modeling to small temporal receptive fields and contains anomaly detection in the association space. However, sliding window too large incurs higher computational costs, although unlike temporal modeling the costs here are linear. On datasets with shorter anomalous lengths such as SMD and PSM, the anomalous patterns are diluted even further, resulting in performance degradation.
On the number of training epochs, fewer epochs lead to model underfitting, and yet overfitting is largely prevented with more epochs due to the aggressive learning rate halving per epoch. On the attention length D, a larger Transformer is prone overfitting with visible performance drops on SMD and PSM as D passes the default 512, both of whose monitored systems are smaller in scale. On the hidden length Dp, more complex progression anomaly detector does not adversely impact the performance, suggesting that the association space is less prone to detection overfitting than the data space.
23

---

70
70
60
VUS-PR (%)
JUS-PR (%)
50
40
40
30
SMD
PSM
PSM
20
SWaT
SWaT
HAI
HAI
10
100
200
400
2
(a) Window size.
(b) Training epochs.
70
SMD
70
PSM
60
SWaT
60
JUS-PR (%)
HAI
VUS-PR (%)
50
50
40
30
30
PSM
20
SWaT
10
HAI
64
128
256
512
1024
2048
32
64
128
256
(c) Attention length D.
(d) Progression hidden length Dp.
Figure 11: Hyperparameter Sensitivity of detection performance in VUS-PR scores.
24

---

I
Detection Metrics
Conventional metries such as precision, recall, F1 and the threshold-indee sinhhoodddiddeeddeent Area Under the Curve (AUC) scores are commonly boint-based, i.e., predicted labels are scored individually by time points (Zoou et al. 2023 Zhang et al. 2022 Xu et al.] 2018). Real-world benchmarks are rife with range- Wise anomalies spanning consecutive time points (Wagner et al.) (023). Point-based metrics are generally ill-suited for evaluating detection performance due to the continuous-discrete conversion and the series-label misalignment (labeling anomalies precisely is hard) (Garg et al. 2022: Tatbul
al. 2018). Here, we use the range-based metrics proposed in (Paparrizos et al. 2022). ). Compared
against their point-based counterparts, they provide robustness to labeling delay and scoring noises as well as performant detector separability and series consistency.
Given a set of anomalous ranges p = {p: = (st, e wherein each anomaly starts at timestamp 8; and ends at c;. we enclose each range with uniform 1/2-length preceding and succeeding buffers. Given the anomaly label Ye E {0, } at each timestamp f. we derive new soft label ft E [0, 1] as per
the minimum temporal distance to any anomaly P: E P::
8
t/,
Ep. E P,te [s; 1/2, si)
01
4l/l,
Epr E P,LE (ei, + 1/2)
(8)
Ut
otherwise
where is the buffer length, normally to the median segment length in P. Within the buffers, Ve monotonically increases from v2/2 to I as the distance decreases. With the new soft label series
y = } we define the True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN) accordingly:
TP= 5" 8, FP (1 (1- 5 TN (1 5) (1- FN (1- (9)
We then compute the fireshold-indepencee ROC), i.e., TP rate vs FP rate, and the Precision-Recat Recall (AUC-PR) curves respectively to be rid of thresholding impact. Fully parameter-fire Volume Under the Surface (VUS) scores for AUC-ROC (VUS-ROC) and AUC-PR (VUS-PR) also computed under different buffer lengths i E 00 21).
AUC for the Receiver Operating Characteristic (AUC-
25

---

Diagnosis Metrics
In line with previous works Tuli et 2022 Zhao et 2020). we use common metrics such as Hit Rate (HR) (Su et al., 2019b) and Normalized Discounted Cumulative Gain (NDCG) (ärvelin and Kekäläinen 2002) measure performance where diagnosis labels are available. Given set of anomalous features G; C [M] at an anomalous timestamp t p; E P as a diagnosis ground-truth
and the set of top k-ranked features T P% according Eq. 5 where = [|G,] P% 1, say
when [G]] 3 and P = 150, the HR at P% (P 2 100) features is the overlap ratio between the two:
G; 01,00%
HR, a P%
(10)
G
In information retrieval, DCG measures the cumulative utility of retrieved documents by their ranking order up to certain position. NDCG normalizes the DCG by the maximum possible DCG. They parameterized by 5% to determine the location in our evaluation and calculated as follows.
DCG, 0P%
rj lo 5=1
IDCG, Σ
NDCG, 8P% +
DCG, 0P% IDCG,
(11)
where E {0, 1} is the relevance value of the j-th element and, in this case, the membership of T,0P%'s j-th feature in G.. NDCG has value strictly between and 1.
At the range level, we measure the Interpretation Score (IPS) initially proposed in Li al. (2021) and here expanded fit the P% parameterization. For each anomalous range p;, the IPS score
GG (G,N 0%%
IPS PP%
(12)
G
where Ω2 @ @.%%%%%.. is the top k-ranked features according maXtep; Sy.t. the j-th feature's maximum anomaly score during p4 and = [|G,] x P%]. the HR equivalence at the range level as per the highest anomalous scores per feature.
26

---

Table 8: Standard deviations of anomaly detection performance in Table Standard deviations of crreshoddindependeen AUC-ROC and AUC-PR metrics and fully parameter-free VUS- AIS-ROC and VUS- -RR metrics reported. All values are percentages.
SMD
PSM
SWaT
HAI
Method
AROC
ApR
VROC
PR
AROC
ApR
VROC
VpR
AROC
ApR
ROC
VPR
Agoc
ApR
V ROC
PR
IF
0.45 0.11
0.40
0.11
1.15 1.11
1.13
1.10
1.42
2.49
1.56
2.58
1.30
0.94
1.25
0.79
DIF
0.81 0.16
0.76
0.15
0.94
0.64
1.00
0.64
0.26
0.73
0.33
0.90
1.65
3.31
1.64
3.18
TranAD
0.21 0.11
0.23
0.10
0.66
0.29
0.62
0.28
12.15
5.30
11.92
5.30
8.89
9.83
8.83
9.61
ATF-UAD
2.23 0.12
2.19
0.12
4.19
1.75
4.33
1.74
12.06
11.92
6.27
10.83
12.11
10.77
11.86
AT
0.39 0.50
0.39
0.48
3.05
0.87
3.22
0.86
3.51
1.43
3.61
1.43
5,50
1.96
5.24
1.90
DCdetector
0.39 0.12
0.39
0.12
1.72
0.13
1.57
0.13
0.06
0.23
0.06
0.24
N/A
N/A
N/A
N/A
USAD
1.43 0,48
1.44 0,48
0.40
0.21
0.40
0.21
6,41
7.42
6.43
6.77
7,41
8.02
7.17
7.79
GDN
0.81 1.56
0.78 1.55
2.58
1.26
2.39
1.20
1.35
3.21
1.17
3.14
1.63
4.62
1.66
4.46
MAD-GAN
2.00
0.50
1.99
0.50
4.65
4.08
4.71
4.09
4.15
4.09
4.17
4.13
2.05
1.76
1.99
1.72
DifFAD
0.40 0.17
0.31
0.18
1.15
0.42
0.89
0.40
0.26
0.09
0.30
0.12
0.60
0.47
0.56
0.47
Ours
0.98 0.75
1.01 0.76
1.07
0.69
1.09
0.69
0.70
1.34
0.60
1.05
0.47
1.08
0.56
1.11
Table Standard deviations of anomaly detection performance Table prreshold-ineepente AUC-ROC and AUC-PR metrics and fully parameter-free VUS-ROC and VUS-PR metrics are reported. All values are percentages.
Standard deviations of
SMD
PSM
SWaT
HAI
Submodule
CCannge
Agac APR
VaOc
VPR
Asoc
APR
VROG
Vpa ARoC
APR
VROG
Vpr
AROC
Arm
VBOC
Vpr
Ours
0.98 0.75
1.01 0.76
1.07 0.69
1.09 0.69
0.70 1.34
0.60
1.05
0.47 1.08
0.56 1.11
Prog. (Eq
ReLU
1.00 0.45
1.00 0.45
0.76
0.61
0.73 0.61
0.52 5.17
0.51 4.87
0.19 0.33
0.17 0.33
Aggr. (Eq
sum
0.87 0.61
0.88
0.62
1.30
1.30
1,29 1.28
1.39
3.62
1.40
3.61
0.22
0.67
0.21 0.60
Aggr. (Eq
6.34 1.21
6.37
1.20
0.71
0.21
0.69
0.21
0.48
0.98
0.48
1.00
0.51
0.42
0.56 0.39
Delection
S directly
0.70 0.51
0.69
0.50
0.85
0.42
0.85
0.42
1.17 1.20
1.12 1.12
1.74 4.39
1.81 4.18
K
Standard Deviations of Detection Performance
Table 8 reports the standard deviations of anomaly detetion performance as reported in Table 3
Section 4.2
Table 9 reports the standard deviations of anomaly diagnosis performance as reported in Table 5 in
Section 4.3
Table 10 reports the standard deviations of anomaly detection performance reported in Table
Section 4.3
Table 10: Standard deviations of anomaly detection performance in Table Standard deviations of threshold-independeen AUC-ROC and AUC-PR metrics and fully parameter-free VUS-ROX and VUS-PR metrics are reported. All values are percentages.
SMD
PSM
SWaT
HAI
Method Criter.
AROC APR
VRoC VPR ARoc APR VROC VPR AROC
APR
VROC
VPR
ARoc
APR
VROG
VeR
Ours
both
0.98 0.75
1.01 0.76
1.07 0.69
1.09 0.69
0.70 1.34
0.60
1.05
0.47 1.08
0.56 1.11
DR
only
0.82 0.19
0.56 0.15
1.76 1.00
1.76 1.01
1.07 1.52
1.11
1.72
0.44 0.86
0.46 0.8T
SPR
p only
0.99 0.83
0.85
0.39 0.18
0.42 0.20
1.61 0.67
0.63
1.10 3.30
1.21 3.32
27

---

Table 11: Standard deviations of anomaly diagnosis performance in Table 4 Standard deviations of point-based HR@P%, NDCG@P% and range-based IPS P% are reported. All values are percentages.
SMD
SWaT
HAI
Method
HR@P%
ND@P%
IP5@P%
HR@P%
ID
IPS@P%
HR@P% ND@
1%
P
P
100
150
100
150
100
150
100
150
100
150
100
150
100
150
100
150
100
150
TranAD
0.08
0.29
0.22
0.20
0.25
0.69
1.10
2.50
1.13
1.99
4.25
7.37
0.84
2.16
0.54
1.36
1.47
3.07
ATF-UAD
1.62
1.68
2.60
2.39
2.67
2.73
1.19
2.45
1.17
1.92
4.42
5.81
2.37
4.39 2.68
3.93
5.06 6.52
USAD
0.37
0.81
0.90
0.50
0.77
1.54
1.75
3.06
0.52
0.27 0.72
0.56
2.04
GDN
1.41
1.27
1.63 1.57 1.08
1.20
1.36
1.33
1.36
1.32
2.51
3.15
2.08
2.59
2.38
2.70
2.37
6.06
DiffAD
N/A N/A
N/A N/A
N/A
N/A
0.09
0.16
0.09
0.12
3.18
3.11
0.37
0,52
0.35
0.44
0.67
1.45
Ours
0.46 0.46
0.56 0.58 1.61 0.96
0.98 1.68
1.03
1.45
4.20
2.52
0.64
0.96
0.74
0.88
1.71
1.57
Table 12: Standard deviations of anomaly diagnosis performance in Table 7 Standard deviations of point-based HR@P%, NDCG@ P%, and range-base IPS P% are reported. All values are percentages.
SMD
SWaT
HAI
Method
HR@P%
ND@
P%
IPS@ P%
HR@P%
ND@
IPS@
HR P%
NDØ
P%
PP
P
100
150 100
150
100
150
100
150
100
150
100
150
100
150
100 150
100
150
Ours
0.46
0.46
0.56
0.58
1.61
0.96
0.98
1.68
1.03
1.45
4.20
2.52
0.664
0.96
0.74
0.88
1.71 1.57
SPR
1.38
1.58
1,28
1.36
2.05
2.55
21.26
24.13
21.31
23.10
8.55
10.41
0.95
1.35
0.99
1.20
2.94
4.36
Joint
1.63
1,27
1.60
1.39
1.73
2.46
0.20
0.92
0.16
0.62
2.83
5.36
0.63
1.20
0.65
0.94
1.72
1.94
L
Standard Deviations of Diagnosis Diagnosis
Table
II
reports the standard deviations anomaly diagnosis performance as reported Table 4in in
Section
Table 12 reports the standard deviations of anomaly diagnosis performance as reported Table in
Appendix D
28

---

Table 13: Anomaly detection based performance. Threshold-dependeee Precision, Recall, F1 scores and preshold--iddpeedeee AUC-ROC and AUC-PR scores are reported. All values are average percentages from five random seeds. The best values are in bold and the second best underlined.
SMD
PSM
SWaT
HAI
Method
Agoc
App.
Agoc
Asoc
Amoc
Aps
IF
15.22
18.53
16.61
66.26
12.67
99.12
76.75
51.50
71.00
45.61
99.54
59.06
74.13
84
72.86
12.87
22.00
16.
70.68
8.20
DIF
29.62
21.67
24.87
69.20
16.47
34.17
93.14
49.98
65.00
40.68
96.79
75.01
87.37
75.63
62.67
39.
48.59
79.71
30.62
18.54
62.09
74.01
31.03
ATE-UAD
6.07
24.06
9.49
52.23
4.71
30.50
91.67
48.56
57.20
38.35
90.09
73.37
40.54
62.47
17.89
73.25
25.37
37.18
68.62
26.67
AT
4,16
100.00
7.98
49.97
4.55
27.73
100
43.42
45.79
26.21
12.02
100.00
21.
42.43
10.23
6.86
8.88
7.60
35.82
4.40
19.12
11.82
N/A
NIA
USAD
14.15
22.20
17.12
63.14
10:08
50.04
68
45.95
$7.96
42.64
95.99
62
75.30
83.26
72.59
69:20
28,
39.54
71.35 29.18
GDN
16.47
26.84
17.87
65.30
9.62
59.08
84.15
53.25
69.09
42.18
38.04
72.12 49.53
76
24,10
98
45.
52.64
82,39 39.43
63.07
89.5
19.02
DiFAD
10.51
19.89
13.75
60.34
7.67
27.76
100.00
43.45
55.09
33.04
12.02
100.00
21.46
18.71
7.27
46.76
16.53
24.37
80.96 20.74
Ours
17.06
41.26
24
79.82
15.10
41.55
38.60
48.60
65.42
43.28
96.20
66.92
78,92
86.91
74,81
65.42
62,63
63.99
93.40
49.22
Table 14: Standard deviations of anomaly detection point- based performance in Table 14 Standard deviations of prreshold-dpendde Precision, Recall, FI scores and burehhld-independeee AUC-ROC and AUC-PR scores are reported. All values are percentages.
SMD
PSM
SwaT
HAI
Method
P
R
Fl Asoc
Am
P
R
Fl
ABoc
AeR
R
Fl
Azoc
Ane
P
R
FI
Asoc
Arn
IF
1.16
2.02
0.47
0.72
|.50
1.50
3.06
9.03
1.54
1.12
1.70
0.22
0.11
0.03
0.91
0.86
1.43
2.09
0.59
0.67
0.90
DIF
4.06
0.98
0.98
0.50
1.05
1.59
1.06
1.65
1.52
2.38
1.87
0.34
0.56
1.91
3.22
2.62
1.15
TranAD
5.98
3.31
0.46
0.28
0.50
0.50
0.33
0.62
0.30
0.62
0.74
12.41
16.00
12.29
19.93
6.80
14.08
11.13
6.63
7.58
8.08
2.70
9.45
12.34
15.50
9.99
AT
0.00
0.00
0.00
0.44
0.43
0.00
0.00
0.00
1.95
1.01
0.00
0.00
0.00
5.39
1.35
3.33
3.13
3.18
5.81
1.59
DCdetector
0.00
0.78
0.13
0.00
0.00 0.00
0.32
0.15
0.00
N/A
N/A
USAD
2.10
1.91
1.17
2.68
1.20
0.44
0.43 0.50
0.61
0.27
3,30
1.41
0.70
2.64
1.25
13.69
6.17
4.12
4.73
5.44
GDN
7.60
12.09
3.46
0.58
2.19
2.50
11.27
4.26
2.87
2.06
1.75
1.49
1.19
1.50
1.56
14.18
11.11
5.65
1.60
4.23
MAD-GAN
2.46
2.16
1.08
1.98
0.52
2.31
5.94
2.08
4.54
5,54
0.91
0.93
0.60
2.44
2.22
0.86
0,90
0.87
1.42
1.27
DHEAD
0.38
0.61
0.28
0.34
0.39
0.00
0.01
0.00
0.74
0.45
0.00
0.00
0.00
0.20
0.03
5.14
0.87
1.24
0.77
0.47
Ours
4.22
0.51
L65
0.65
0.79
1.09
0.38
0.23
0.60
0.62
M Point-based Evaluations
In addition to the model evaluations using range-based metrics such as VUS-ROC and VUS-PR in Sections 4.2 and 4.3 we conduct point- evaluations herein using point-based metrics exclusively. Table [3 reports report the Precision, Recall, and Fl scores under the threshold where the method of interest achieves the best F1 score. To mitigate the impact of thresholding protocol, Table 13 also reports the greeshold-indepeneenn AUC-ROC and AUC-PR scores. Table 14 reports the standard deviations.
SARAD achieves state-of-the-art performance on most datasets except PSM. The enlarged temporal receptive field of SARAD (half input window) contributes to its underperformance by point-based metrics, especially on datasets where anomalous ranges are short (see Table 2). when compared against most others' receptive filed of single few time points (typical ID Conv kernel size is 3). We again underline that most real-world anomalies are continuous and point- based metrics are mismatched for such anomalies (see Appendix for discussion).
29

---

Table 15: Model complexity and overheads. The total training time (in minutes), the inference time per sample (in milliseconds), and the total number of parameters (where applicable) are reported. The best values are in bold and the second best underlined.
SMD
PSM
SWaT T = 197K
HAI
T = 708K
T = 132K
T = 922K
N 38
N 25
N = 51
N 79
I = min
I min
I
T 1
Train.
Infer. (ms)
Param,
Train.
Infer.
Param.
Train. (mins)
Infer. (ms)
Param.
Train. (mins)
Infer. (ms)
Param.
Method
(mins)
(mins)
(ms)
IF
0.01
0.01
N/A
0.00
0.01
N/A
0.01
0.01
N/A
0.03
0.01
N/A
DIF
10.13
1.66
874K
1.91
1.52
853K
7.67
1.68
895K
15.47
1.87
940K
TranAD
6.80
0.04
127K
0.96
0.03
57K
6.06
0.05
226K
17.15
0.07
531K
ATF-UAD
14.20
0.06
414K
1.90
408K
12.77
0.06
421K
16.42
0.06
436K
AT
0.74
0.00
867K
9.25
0.00
4.80M
36.53
0.00
910K
60.57
0.00
4.91M
DCdetector
102.16
0.01
867K
25.72
0.03
895K
214.56
0.02
910K
Out of Memory
USAD
4.34
0.01
803K
0.86
0.01
441K
3.74
0.01
1.26M
9.34
0.01
2.54M
GDN
36,04
0.44
9.64
0.34
19.65
0.46
65.69
0.68
6K
MAD-GAN
42.56
0.40
268K
45,06
0.46
261K
1416,26
0.40
274K
165.01
0.47
289K
DifFAD
11.93
1.49
38.85M
4.94
7.91
38.85M
15.32
1.69
38.85M
42,87
2.80
38.85M
Ours
1.47
0.11
9.57M
2.32
0.12 15.85M
10.87
0.16
9.59M
31.97
0.39
15.94M
N
Model Complexity and Overheads
We study and compare the complexity and time overheads of all baselines and SARAD. Concretely, we evaluate the model complexity by the number of parameters being used and the total training time as well as the inference time per sample. All experiments time overheads are performed on compute node with AMD EPYC 7443 (48 cores, 96 threads) CPU, NVIDIA A10 (24GB) GPU, and 512 GB RAM.
Table 15 reports the total training time (in minutes), the inference time per sampling point, and the number of network parameters used (where applicable) of all baselines the main datasets. The size of the training set T, the number of features N, and the sampling period I are also reported per dataset for easy reference. SARAD, while not the fastest nor the lightest model, incurs moderate time overheads and model complexity.
We note that even SWaT, the smallest dataset in terms of actual clock time, spans approximately days for collection. This is far exceeding most detectors' training time besides MAD-GAN and levigates concerns for training overheads for them. All detectors also ineur inference per sample time several magnitudes below the smallest sampling frequency of second, guaranteeing real-tin time deployment of all detectors once trained.
30

---

Baselines
We trained all baselines using official implementations where available and recommended hyper- parameters from respectively papers are used. Some baselines, such as DiffAD and AT, have -ataset-specific hyperparameters and here we adopted them as well. The open-accessed URLs of the baselines used are listed as followed.
IF (ICDM'08) (Liu et al.
2008 :
http://gttaabbooouuuouoggggggggg
DIF (TKDE'23) (Xu et al.
2023 https://gttttuucco///uuuuuuggggo
TranAD (VLDB'22)
(Tuli et al.
2022): httpss//ggtaub.coommmmmppaaa1111
TranAD
ATF-UAD (NN'23) Fan et al.
2023)
httpes/ggthub..cm///btteer///11
AT (ICLR Xu et al.
2022): http:///gtttuuaoco///mmm////oyyt
DCdetector (KDD''23
Yang et al.
2023
http:///gthhbb.ccm//D0000.11L11
SDb2023-DDCdttcci
USAD (KDD'20) (Audibert et al.
2020
http::/ggthhb..commmniiigglaa//a
GDN (AAAI21)) (Deng and Hooi
2021
attps///tthub..cm//aa111//////
MAD-GAN (ICANN' 19) (Li al. Official implementation was migrated to Torch for uniform envrrnnnalliiionnttttnallll set-uppptttaaauuuuuttptat See our codebase for details.
2019:
DiffAD (KDD'2) 23)
(Xiao al.
2023
31

---

Progression Recon. Error (p)
Progression Recon. Error (p)
Anomalous Normal
Anomalous Normal
10-1
10-4
10-1
10-1
10:1
10
10
10
Data Reconstruction Error (r)
Data Reconstruction Error (r)
(a) SMD.
(b) PSM.
Progression Recon. Error (p)
10f
Progression Recon. Error (p)
Anomalous Normal
Anomalous
10
10
10
10-1
10-1
10
10
10
10²
Data Reconstruction Error (r)
Data Reconstruction Error (?)
(c) SWaT.
(d) HAI.
Figure 12: Joint detection criterion data reconstruction error and progression reconstruction p.
P Joint Detection Criterion
Figure 12 visualizes the two components of the joint detection criterion in Eq. i.c., the data reconstruction error r and the progression reconstruction error p. Balanced resampling is applied here. Recall that is the sum of normalized and p. Most anomalous samples (input series) either has high r or high p. and oftentimes both. The former measures the magnitude of anomalousness in the data space, the latter in the spatial association space. The basis underpins the formalization of the joint detection criterion.
32

---

Normal
1.75
Normal
AMAA
Anomaly
1.50
Anomaly
1.25
Density
Density
1.00
0.75
0.50
0.25
10a
100
Anomaly Score
Anomaly Score
(a) SMD.
(b) PSM.
Normal
Normal
Anamaly
Anomaly
1.0
Density
Density
0.6
1.0
0.2
10
16
10a
10
10
10a
10'
10*
Anomaly Score
Anomaly Score
(c) SWaT.
(d) HAI.
Figure 13: Distributions of anomalies scores.
Distributions of Anomaly Scores
Figure 13 the distributions the joint anomaly scores. On SMD and PSM. the scores the normal and anomalous samples (input series) overlap more heavily than SWaT and HAL The observations here highlight the difficulty anomaly detection in the service monitoring space, more so than in industrial control where measurements and actuation result from well-defined control logics. The hardness evident Table of detection performance and of diagnosis performance.
33

---

Table 16: Anomaly detection performance under different thresholds. TPRs under thresholds set by pre-defined FPRs E f1%, 5%, } are reported. Higher TPRs are better. Similarly, FPRs under thresholds set by pre-defined TPRs (90%, 95%, 99%) are reported. Lower FPRs are better. All values are average percentages from five random seeds. The best values are in bold and the second best underlined.
SMD
PSM
SWar
HAI
TPP8 (FF)
FPR]
(STTPR)
TPRT
(MFPR)
PPRC
(FR)
TPK" (OFFFR)
FPRL (OTTPR)
TPR1 (MPEP))
FPR,
( (WFPP))
Method
15t
5%
10%
909
955
999
156
5%
10%
90%
05%
091
195g
5%
1098
0556
1998
1%
556
10%
9056
95%6
99%
IF
0.20
3,02
7.22
78.49
56.46
91.63
1..50
3.50
11.09
50.94
80.64
94.26
$2.88
$6.45
65.77
48.4
56.06
64.54
4.15
18.18
28.67
57
61.96
68.40
DIF
1,56
8.43
15.17
79.37
57.22
92.33
0.23
1.77
4,83
78.51
$3.80
91.86
48.07
63.25
69.63
41.18
50.41
59
31.32
41
49.40
18.56
54.21
58.81
ThanA
01.86
4.25
8.64
57.86
91.53
95
115
2.58
7.42
75.94
87.40
96.24
0.12
0.57
1.61
75.78
81.11
85.72
21.39
31
37.79
53,50
58.50
06.99
ATF-IAA
0.04
1.04
4.42
00.65
91
96
0.4
2.05
5.67
83.00
89.02
95.95
0.25
1.02
2.16
99
75
81:30
16
24.21
11.12
60
65.08
69.25
1.57
3.78
3.94
79.77
99.77
99.77
0.56
2.73
5:43
99.77
99.77
99.77
1.45
4.07
7.68
98466
95.46
4.33
11.61
17.51
95
96.91
99.11
DCdenector
0.32
3.40
5.10
97.47
98.43
99
0.25
1.68
4.32
99.65
99.65
99.85
0.91
2.19
29
98
98
95.48
NIA
N1A
NIA
NIA
N/A
N/A
USAD
3.48
0.55
1.85
1.75
79
55.77
DiTTD
0.55
0.58
99.10
0.51
29.57
34.39
37:69
Ours
1.44 16.42 36.41
48.19
57.97
(0.71
5.85
9.50
82.33
90.
96.6
53.87
60.1
68.8
42.85
50.34
59.86
58.26
83.03
88.82 11.13 19.76 27.01
Table 17: Standard deviations of anomaly detection performance under different thresholds in Table 16 Standard deviations of TPRs under thresholds set by pre-defined FPRs E f1%, 5%, 10%)
are reported. Similarly, standard deviations of FPRs under thresholds set by pre-defined TPRs
(90%, 95%, 99%} are reported. All values are percentages.
SMD
PSM
SWAT
HAI
TPR1 (OSFPR)
FPRL (ATTPR)
(0FFPR)
FPR.
TPRT (BFFPR)
FPRL (#TTPR)
TPR1
(OPTRR)
Method
1%:
5%
10%
(00%
95%
993
15
10%
(00%
95%
5%
10%
900S
905
1%
10%
905%
95%
99%
IF
0.08
0.52
0.46
1.34
1.2
0.91
0.00
0.84
1.00
3.07
1.03
2.23
4.29
4.61
2.39
2.68
2,88
2.74
0.51
1.04
1.11
2.00
1.46
2.98
DIF
0.48 0:90
1.44
L45
1.48
0.16
0.66
0.38
1.73
1.91
4.64
2.05
2.18
1.48
0.78
0.84
2.91
3.61
3.00
2.37
2.74
2.17
ATF-UAD
0.04 0.91
1.66
2.12
1.87
1.70
0.30
0.97
2,09
7.55
7.66
2.80
0.17
0.21
6.07
5.91
3.89
10.48
13.55
16.69
6.00
4.69
4.35
AT
0.06
0.51
0.67
0.00
0.00
0.00
0.05
0.53
0.91
0.00
0.00
0.00
0.62
0.81
1.31
0.00
0.00
0.00
2.24
4.63
5.58
3.32
2.06
0.00
DCJetector
0.04 0.61
1.39
5.15
3.00
0.03
0.42
1.01
0.00
0.00
0.02
0.60
0.60
0.00
0.00
NIA
NAA
N/A
NIA
NIA
N/A
1.26
1.51
4.85
2.13
0.42
2.87
1:25
1.72
5.96
4.28
MAD-GAN
1.23 1.54
1.30
4.22
4.57
3.55
1.74
2.58
3.12
6.60
6.30
2.91
19.27
5.15
6.39
15.81
11.37
11.88
1.92
2.97
2.88
4.87
1.66
4.20
DiTAD
0.11
0.75
0.52
1.18
0.74
0.009
0.28
0.26
0.48
0.24
0.07
0.32
0.34
0.01
0.00
0.00
0.48
2.76
0.99
1,88
2.66
2.54
037
0.35
0.97
3.57
R Detection Performance Under Thresholds
Effectiveness of each anomaly detector is influenced by the selection of thresholds. Previously, we reported the range-based and point-based threshold-indee -nahheddddddeeeddennt performance metrics. Here, we study the influence of threshold selection on the detection performance. Table T6] reports the range-based True Positive Rates (TPRs) under thresholds set by pre-defined False Positive Rates (FPRs)as well as the FPRs under different thresholds set by pre-defined TPRs. Table I reports the standard deviations.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: The main claims clearly reflect the paper's contributions and scope of time series anomaly detection in the abstract and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
35

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We disclose full details of our experiments in Appendixes and O
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
36

---

Answer: [Yes]
Justification:
We provide open access to the data and code in Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com http///bbb////iiiio/oeee/eaaaaaa swiniiiiicypyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer:
[Yes]
Justification: We disclose essential experimental settings in Section 4.1 and full details in Appendix [F]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report standard deviations of the experiments Appendices and
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
37

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We disclose information on compute resources for experiments in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
38

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no risks of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite and properly credit all datasets, baselines, and libraries used this paper (see Appendices E Ond for details).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
39

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve IRB approvals and equivalent approvals/reviews.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "E6oQUAccNK", "reviewer": "Reviewer_7FD6", "summary": "Paper proposes a deep learning based anomaly detection method for multi-variate time series data. The proposed method has two components in a single neural network. First component is a traditional auto encoder approach in which the input time series is broken into two parts (along time) and the model is trained to use the first part to generate the second part, and identify anomalies as a difference between the predicted and true output. The second component trains a transformer to generate the associations across the different features (akin to cross-correlation) and then train an MLP to predict the associations and identify anomalies if the predict associations are different than the observed associations. Thus, the method captures a broader range of temporal anomalies. \n\nExperimental results on a variety of benchmark data sets and comparisons with state of art methods demonstrate that the proposed method (SARAD) is able to identify anomalies that might not be apparent to approaches that do not consider the associations.", "questions": "Are there any types of anomalies that would still not be captured by this method?\n\nSimilar to above, what kind of false positives could be captured by adding the association part?", "limitations": "The paper has addressed the limitations - I have suggested a couple above. Paper does not have a direct societal impact so the authors have a reasonable response to that.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Paper is well-described. The idea of using association seems novel, though I have seen papers using cross-correlation to understand the multi-variate effect. \n\nResults are promising. The approach is compute intensive (scaling quadratic with the number of features), but authors do acknowledge the limitation and outline possible approaches to work around it.", "weaknesses": "A terminology issue - spatial often refers to data with positional coordinates (geographic or xy on a domain) which indicates spatial relationships between observations. On top of that, each observation would be represented as a multi-dimensional feature vector. In this paper, the authors use the term spatial to refer to the multi-dimensional feature vector representation of data. I found that very confusing. I think it might be better to fix the terminology to improve readability of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank all reviewers for their detailed and constructive reviews. Here are our responses:\n\n**Weakness 1:**\n\n**Response:**\nApologies for causing the confusion. We note that spatiality has different connotations in some AI literature, e.g., geographic locations or characteristics on Earth. We will mention such definitions in other related literature and with respect to that clarifications on our usage of spatiality in this work to refer to the multi-dimensional feature vector of time series data. This terminology is also used in some related literature [1][2], i.e., time series prediction and anomaly detection.\n\n[1] Tryambak Gangopadhyay, Sin Yong Tan, Zhanhong Jiang, Rui Meng, and Soumik Sarkar. 2021. Spatiotemporal Attention for Multivariate Time Series Prediction and Interpretation. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 3560–3564.\n\n[2] Yu Zheng, Huan Yee Koh, Ming Jin, Lianhua Chi, Khoa T. Phan, Shirui Pan, Yi-Ping Phoebe Chen, and Wei Xiang. 2023. Correlation-Aware Spatial–Temporal Graph Learning for Multivariate Time-Series Anomaly Detection. IEEE Transactions on Neural Networks and Learning Systems (2023), 1–15.\n\n**Question 1:**\n\n**Response:**\nVery long-range anomalies are hard to capture even by our method. We note that on PSM and SWaT, there are a few very long-range anomalies lasting several thousand time points (the longest one is 35,900 as per Table 2). Over the long course of such anomalies, the anomalous features re-correlate with non-anomalous features, or the correlational breakdowns at the beginning of anomalies cease in the view of divided subseries, as the physical process regains stability. Table 6 shows the impact of those very long-range anomalies on SWaT performance when the score is entirely based on spatial associations (SPR). Without the cues from changes in spatial associations, our method relies on data reconstruction to detect such anomalies. In some cases, the system returned to normal operational states, leading to lower reconstruction errors and causing the anomalies to evade detection.\n\n**Q2:**\nFalse positives can occur when the correlational relationships between features are altered but not due to anomalies. Examples include situations in which the monitored system changes operational modes unexpectedly. On SWaT, we observe a few false positives which are suspected to be during changes of operational modes. However, due to a lack of operational scheduling details and setpoint (desired conditions) data during test set collection in the SWaT technical document, we were not able to verify that. On HAI, there is an operation task scheduler that periodically adjusts the setpoint values within the legal ranges to simulate benign scenarios. As these adjustments are periodical, there are fewer false positives on HAI due to mode changes. Table 16 shows the elevated false positive rates for SWaT when compared to HAI. Real-world possibilities of unplanned operational mode or setpoint changes are low.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kl445vER4d", "reviewer": "Reviewer_RKBC", "summary": "In this paper, the authors focus on incorporating spatial information for time series anomaly detection and diagnosis. The proposed algorithm, SARAD, employs a transformer-based data reconstruction approach to capture inter-feature associations. By analyzing changes in these associations over time, the algorithm identifies anomalies under the assumption that anomalous features cause a reduction in the perceived association.", "questions": "I'm interested in understanding how the SARAD method distinguishes reductions in spatial associations that indicate anomalies from those that could be considered normal variations. Please clarify the types of anomalies the SAR phenomenon is effective for.\n\nThe authors could have enhanced their study by comparing SARAD with existing methods (e.g., InterFusion), which integrate both spatial and temporal information for anomaly detection", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea of utilizing spatial association descending patterns for time series anomaly detection and diagnosis seems interesting. \nThe authors proposed new criteria for calculating anomaly scores. \nDetailed experimental analysis has been performed to validate the proposed method.", "weaknesses": "The related work section lacks a detailed discussion of all the baseline methods used for comparison in result analysis. For a more comprehensive understanding and to highlight the contributions of the paperwork, it would be better to incorporate those methods and discuss their limitations.\n\nThe third paragraph in the Introduction section is not easy to follow.  Providing more background information and reorganizing would be recommended.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank all reviewers for their detailed and constructive reviews. Here are our responses:\n\n**Weakness 1:**\n\n**Response:**\n\nApologies for not including all baselines being compared in the related work section. Lines 89-102 introduced and discussed the limitations of several baselines including MAD-GAN (Li et al., 2019), ATF-UAD (Fan et al., 2023), USAD (Audibert et al., 2020), AT (Xu et al., 2022), and DCdetector (Yang et al., 2023). Lines 108-110 briefly mentioned GDN (Deng and Hooi, 2021) as a graph-based prediction approach for detection. \n\nWe did not discuss IF and DIF baselines as they are based on neither temporal nor spatial modeling. That was our oversight. We will include a summary and limitations of IF and DIF in the final version. They work by building a binary decision tree ensemble that partitions either the data space (IF) or the deep embedding spaces (DIF). They are limited by the lack of temporal or spatial information and their anomaly scores are not reflective of the magnitude of the anomalies.\n\nWe will also include TranAD, which replaces the MLP in USAD with Transformer, and its adversarial training paradigm makes the reconstruction errors more robust. However, it shares the temporal modeling nature with other Transformer-based detectors and is limited by the small receptive field to capture long-range inter-feature correlation and handle timestamp misalignment.\n\nWith regards to GDN, we will also highlight its limitations beyond the lack of temporal changes of spatial associations mentioned, including the mismatch between its single-point prediction target and the range-wise anomalies and unstable Top-K node selection during training.\n\n**W2:**\nWe will reorganize the third paragraph in the final version. Specifically, we will discuss the role of MHSA inside the Transformer and explain why the intermediate association mapping via MHSA captures the correlational relationships between features under our setting.\n\n**Question 1:**\n\n**Response:**\nSAR is effective whenever anomalies originate from or lead to the dissolution of pre-existing associations, detaching anomalous features from their non-anomalous counterparts (Lines 56-58). Such associations are common in industrial control systems and many other monitoring systems, in which sensors routinely collect measurements from different locations of interconnected subprocesses. Examples of SAR can be found in Figs. 1, 3, 4, 5, 6, and 7 where during the anomalies anomalous features experienced sudden unusual spikes, significant deviation from normal values, or some forms of correlational breakdowns with others. For example, in Fig. 7 a water pump P-101 (feature #4) was turned off without correlations with others causing an anomaly.\n\nTo distinguish between anomalous SAR from normal variations, SARAD uses an MLP autoencoder on reduction in the association space (Lines 70-73). The autoencoder learns the patterns of normal variations by minimizing their reconstruction errors. It could dismiss normal SAR during inference, compared to using reductions directly (Lines 286-288 in Ablation Studies). In addition, SARAD’s data autoencoder produces higher scores for anomalies, complementing the shortcomings of the association autoencoder.\n\nLines 113-119 stated that we derived our SAR insights from the cyber-physical defense literature, where time series anomaly detection is extensively applied (SWaT and HAI are both cyber-physical systems). There are dynamic watermarking approaches (Satchidanandan and Kumar, 2017 and Dai et al. 2023) that overlay actuation with randomized signals to exploit correlational breakdowns (SAR in some way) of attacks (anomalies) for detection. While they are intrusive defense approaches, our method is non-intrusive and yet still exploits SAR.\n\n**Q2:**\nWe ran into some compatibility issues when we tried to set up InterFusion during the rebuttal period, i.e., InterFusion’s CUDA and TensorFlow versions are so old that our machines no longer support them. Unfortunately, we couldn’t sort out the problem by the rebuttal deadline. We will try our best to resolve the issues and provide InterFusion experimental results in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qRVKEzwbj7", "reviewer": "Reviewer_BjR8", "summary": "In this paper, the authors aim to address the problem of anomaly detection and diagnosis for time series data. The authors consider that the existing methods may obscure or dilute the spatial information and the interaction between different variates, they propose the SARAD model to capture these interactions with the transformer. The authors evaluate the proposed method on several datasets and achieve good performance.", "questions": "Please refer to weaknesses", "limitations": "Please refer to weaknesses", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "N.A.", "weaknesses": "However, there are some problems to solve  \n1.\tThe authors claim that existing methods may assume feature independence or combine variables of a diverse physical nature. Indeed, the recent methods for time series analysis usually employ independent channel assumptions. However, several methods like the conventional RNN and the graph-based methods do not assume that the features are independent. Therefore, the motivation of this paper might not be convincing.  \n2.\tMoreover, Figure 1 might be not clear to explain the motivation of the paper. Specifically, Figure 1(a) is not informative and Figure 1(b)(c)(d) are almost the same.  \n3.\tThe contribution of the proposed method is limited. There are other methods that use transformers as the backbone networks like [1]. It is suggested that the authors should provide a detailed discussion and compare with it.  \n\n[1] Xu, Jiehui, et al. \"Anomaly transformer: Time series anomaly detection with association discrepancy.\" arXiv preprint arXiv:2110.02642 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank all reviewers for their detailed and constructive reviews. Here are our responses:\n\n**Weakness1:**\n\n**Response:** \nWe agree that some temporal modeling methods, such as RNN, do not make feature independence assumptions. However, Lines 28-34 stated that temporal methods including RNNs which do not assume feature independence combine variables/features of diverse physical nature. That leads to the dilution of spatial information, which could be crucial to detection. Our related work also covered RNN-based methods such as LSTM and MAD-GAN (Lines 91-94), the latter being one of our baselines. We highlighted their limitations regarding small receptive fields in time and the impact of timestamp misalignment across features (Lines 97-99).\n\nOur motivation for this paper is expressed in Lines 32-38: temporal methods ignore the spatial associations that characterize multivariate time series patterns. They also restrict diagnostic capabilities caused by a mismatch between temporal novelty and capturing spatial novelty. Our proposed SARAD aims to leverage spatial information and exploit the spatial association descending patterns common with time series anomalies.\n\n**W2:** Figure 1 investigates the Spatial Association Reduction (SAR) phenomenon by showing the changes in spatial associations throughout a time series anomaly.\n\nFig. 1a shows the raw time series of all feature channels, with the anomalous features (#12 and #15) highlighted in red and marked with dashed lines. The intention of 1a is to show a full picture of the raw data before, during, and after the anomaly of interest. We were unable to insert proper space between features due to the page limit, which may cause readability issues without digital zoom. Based on the reviews, we can replace 1a with a selected feature set to improve readability.\n\nFigures 1(b)(c)(d) show the average association mappings before, during, and after the anomaly. While they might look similar, the differences between them are shown in Figs 1(e)(f) which display elevated association reductions on the anomalous features (#12 and #15). Apologies for the color scheme as it does not have enough contrast. To improve readability, we will increase the contrast for Figs. 1(b)(c)(d).\n\n**W3:** We discussed existing detectors of the Transformer backbone such as Anomaly Transformer and DCdetector in Lines 94-102 and 105-108. Furthermore, we compared against Transformer-based detectors including TranAD, Anomaly Transformer, DCdetector, and ATF-UAD in Tables 3, 4, 13, and 16.\n\nConcretely, for the cited Anomaly Transformer, we underlined its temporal modeling capabilities underpinned by the self-attention mechanism and yet restricted by the exceptionally small receptive field (typical 1D Conv kernel size is 3 for embedding) and cross-feature timestamp misalignment issues. For anomaly diagnosis, temporal methods such as Anomaly Transformer also mismatch the anomaly criterion of temporal novelty with spatial interpretation.\n\nWe emphasize our contributions in this paper in Lines 80-87. Previous Transformer-based detectors fall short of spatial modeling. In both detection and diagnosis contexts, they ignore the spatial associations that ubiquitously characterize multivariate time series patterns. Specifically, compared with previous detectors using a Transformer backbone, our contributions are as follows.\n\n- Our method explicitly models spatial associations with self-attention modules to address the aforementioned receptive field and timestamp misalignment issues.\n- We design subseries division and refactor MHSA implementation accordingly to enable data shuffling during training and prevent memory storage of last association mappings. \n- We propose autoencoding in the spatial association space to exploit the association descending patterns common with anomalies and complement data autoencoding much less sensitive to spatial novelty.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fv1s4QBpem", "reviewer": "Reviewer_KV18", "summary": "This manuscript proposes a spatial association reduction method for anomaly detection and diagnosis (SARAD) where anomalous features have low association values in reconstructing the multivariate time-series. The association values are derived from an attention between the features and exhibit changes with time which enables the detection of anomalous features and the time period of anomalies. SARAD is built on two models: a transformer for data reconstruction and an MLP for the reconstruction of spatial progression. The last anomaly score is a combination of both reconstruction scores.", "questions": "- How is the threshold defined from the anomaly scores to detect anomalies? \n- Can you please provide a brief explanation why progression-based $p$ (SPR) is less effective for SWaT?\n\nMinor:\n- I find it a bit confusing to call it spatial association for time-series processing. Maybe it is better to call it channel/feature/variable.\n- It is clearer if you visualize or indicate the ground truth for anomaly diagnosis alongside the detected features i.e., in Figs. 1, 3, and 4-7.\n- Lines 84-85: Can you please make it clearer which spaces you are refereeing to? I think you mean data space and spatial progression space.", "limitations": "I think the issues about anomaly diagnosis and the identity shortcut should be discussed (see weaknesses).", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed methods can be useful to detect the source of anomalies in multivariate time-series data.\n- The manuscript provides a sufficient evaluation.", "weaknesses": "- In my opinion a main issue regarding the anomaly diagnosis should be addressed. For example, in Fig. 1 there are three features (9, 13 and 16) correlated with 12 and 15. An additional example is from Fig. 5 where features 13 and 16 are highly correlated with 12 and 15. What distinguishes the detected features from their correlated features?\n- There was no discussion about the identity shortcut problem. Such a problem is typical for reconstruction-based methods where the model learns to reconstruct anomalies and thus it becomes harder to separate them from the normal ones. Maybe I missed it in the manuscript, but can you elaborate on this issue? How do you address this issue in your method?\n- The Point-based evaluation on PSM dataset indicates some weaknesses regarding the sensitivity to the window size and short-term anomalies (see Table 13). The clustering for SMD and PSM is also not of the same quality compared to other datasets (see Figs. 12 and 13).\n- The model size/parameters of the proposed method exceed other baselines by big margines (except for DiffAD).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank all reviewers for their detailed and constructive reviews. Here are our responses:\n\n**Weakness1:**\n\n**Response:**\nWhat distinguishes features 12 and 15 from features 9, 13, and 16 in Fig. 1 is that the spatial association reduction only occurs on the former features. Figs. 1(e)(f) shows that column-wise association reductions are significant (in darker cells) on features 12 and 15, but not on 9, 13, and 16. The reductions in Fig. 1e(f) are computed as the negative-only changes of the Transformer’s intermediate spatial association mappings from before (after) the anomaly to during the anomaly. Similarly, in Fig. 5, spatial association reduction occurs on features 12 and 15, but not on 13 and 16.\n\n**W2:** \nIdentity shortcut happens when the data reconstructor learns to copy the anomalous input directly to the output. We considered two kinds of techniques to address the problem. On the data side, we could inject noises into the training input such that the reconstruction-based method does not know the true values. The added noise prevents the method from overfitting anomalies in the training set and learning to copy the input. On the model architecture side, we could apply self-attention masks within the MHSA module. In our case of SARAD, it prevents each feature’s representation from directly reading itself during attention computation. \n\nWe initially implemented both but dropped them early in development because we did not run into the identity shortcut problem across all datasets. We also observed that, even in cases where the errors approach zeros (<1e-4), anomalies still lead to high anomaly scores relative to normal time points.\n\n**W3:**\nWe acknowledged the weakness in Lines 688-693 on page 28. PSM contains many short-term anomalies (Fig. 8), making it hard to detect for all methods. In all our evaluations, we applied a unified time window length (of 100) to all datasets to offer a fair comparison with all baselines, as the majority of them originally or officially have done so as well. \n\nFig. 10 and 11 show that with a smaller window size (of 10 or 20), SARAD achieves better performance than the default size. In deployment, the window length could be set based on the seasonal patterns, the sampling frequency, and (if available) previous anomaly history to maximize performance. We also note that nearly all methods are limited by the choice of window size.\n\nSMD and PSM are harder to detect anomalies upon compared to other datasets, leading to lower quality clustering in Figs. 12 and 13. The hardness is due to the existence of many short-term anomalies. PSM and SMD almost always have lower best performance scores than other datasets in Tables 3, 13, and 16.\n\n**W4:**\nIn Lines 527-535, we acknowledged the limitations of the model size and complexity in regard to the number of features. The model size is modest for modern GPUs as we were training it on a three-year-old NVIDIA A10 GPU.\n\nTable 15 in Appendix N shows that, while our model size exceeds some other baselines, the time overheads are smaller than lightweight models such as GDN and MAD-GAN and comparable with many others as well, Anomaly Transformer and TranAD among them. In some cases, such as the DCdetector, the small model size does not translate into low time overheads due to its excessive contrastive learning pipeline. \n\nWe also note that the inference times of our model are several magnitudes below all datasets’ sampling periods, guaranteeing real-time deployment. Also, the training times are several magnitudes below all datasets’ collection time, incurring little to no delay for model/project development. \n\nFinally, our backbone Transformer is known for fast parallel processing and we refactored MHSA implementation to enable parallel subseries processing (Line 168). Our design choice of the near vanilla Transformer and MLP architectures makes possible existing optimization techniques and a stripped-down loss function prunes excessive operations during loss computation.\n\n**Question1:**\n\n**Response:**\nWhile our main evaluation metrics are threshold-independent AUC scores, our method supports any threshold selection protocol. A common strategy would be to measure all anomaly scores $\\boldsymbol{s}$ on the training/validation set and set the threshold to be $mean(\\boldsymbol{s}) + 2 \\cdot std(\\boldsymbol{s})$.\n\n**Q2:**\nSWaT contains several extensively long-lasting anomalies, the longest being 35,900, during which most previously correlated features maintain their correlation due to the limited cyber-attack range (single attack point and single attack subprocess). While the SPR is indeed sensitive to changes in correlational relationships (similar observations are made in Fig. 3 example), the progression-based score decreases in the long anomaly span as the physical process regains stability. In addition, capturing SPR during a long-range anomaly is difficult as we are differentiating the association mappings between consecutive subseries within a fixed-length time window. The data-based score, however, captures the elevated or dropped steady-state measurements during an anomaly and thus complements its progression counterpart. In the end, we adopted a joint detection criterion that utilizes both.\n\n**Minor1:**\n\n**Response:**\nWe note that spatiality has different connotations in some AI literature, e.g., geographic location or characteristics on Earth. We will mention such definitions in other related literature and with respect to that clarifications on our usage of the phrase “spatial association” in this work.\n\n**M2:**\nGround truths for diagnosis labels are highlighted in red bounding boxes in Figs. 1, 3, 4, and 7. We will append the necessary descriptions.\n\n**M3:**\nApologies for the unclear mentions of spaces. Yes, we are referring to both.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors proposed algorithm, SARAD, employs a transformer-based data reconstruction approach to capture inter-feature associations. By analyzing changes in these associations over time, the algorithm identifies anomalies. The strength of the paper lies in novel formulation, new anomaly score, and a Detailed experimental analysis. The weakness lies in lack of motivation, lack of clarity about contribution, and missing benchmarks. Most of them have been addressed in rebuttal. All reviewers are consensus about the acceptance of the paper after rebuttal.  Congratulations to the authors! I am requesting the author to carefully incorporate the suggestion provided by the review in the camera-ready version.
Decision: Accept (poster)