Drift doesn't Matter: Dynamic Decomposition with Diffusion Reconstruction for Unstable Multivariate Time Series Anomaly Detection
Authors: Chengsen Wang
Zirui Zhuang
Qi Qi
Jingyu Wang
Xingyu Wang
Haifeng Sun
Jianxin Liao
Venue: NeurIPS 2023 poster (2023)
Keywords: Anomaly Detection
Time Series
Diffusion
Transformer
TL;DR: This paper proposes a method for long-period unstable multivariate time series anomaly detection in the real world.

Abstract: Many unsupervised methods have recently been proposed for multivariate time series anomaly detection. However, existing works mainly focus on stable data yet often omit the drift generated from non-stationary environments, which may lead to numerous false alarms. We propose **D**ynamic **D**ecomposition with **D**iffusion **R**econstruction (D$^3$R), a novel anomaly detection network for real-world unstable data to fill the gap. D$^3$R tackles the drift via decomposition and reconstruction. In the decomposition procedure, we utilize data-time mix-attention to dynamically decompose long-period multivariate time series, overcoming the limitation of the local sliding window. The information bottleneck is critical yet difficult to determine in the reconstruction procedure. To avoid retraining once the bottleneck changes, we control it externally by noise diffusion and directly reconstruct the polluted data. The whole model can be trained end-to-end. Extensive experiments on various real-world datasets demonstrate that D$^3$R significantly outperforms existing methods, with a 11% average relative improvement over the previous SOTA models.

Drift doesn't Matter: Dynamic Decomposition with Diffusion Reconstruction for Unstable Multivariate Time Series Anomaly Detection
Chengsen Wang" Zirui Zhuang Qi Qil Jingyu Wang Xingyu Wang Haifeng Sun Jianxin Liao
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China {eswang, zhuangzirui, qiqi8266, (anngingyu)@buptt edw.. {wangxingyu, hfsun, liaojx) edu cn
Abstract
Many unsupervised methods have recently been proposed for multivariate time series anomaly detection. However, existing works mainly focus on stable data yet often omit the drift generated from non-stationary environments, which may lead numerous false alarms. We propose Dynamic Decomposition with Diffusion Reconstruction (D²R), novel anomaly detection network for real-world unstable data fill the gap. D²R tackles the drift via decomposition and reconstruction. In the decomposition procedure, we utilize data-time mix- mix-aatention to dynamically decompose long-period multivariate time series, overcoming the limitation of the local sliding window. The information bottleneck is critical yet difficult to determine in the reconstruction procedure. To avoid retraining once the bottleneck changes, control it externally by noise diffusion and directly reconstruct the polluted data. The whole model can be trained end-to-end. Extensive experiments various real-world datasets demonstrate that D²R significantly outperforms existing methods, with a 11% average relative improvement over the previous SOTA models. Code is available at http:///gthubccoo///rrrsiiiiggit
Introduction
Due to the rarity of anomalies, unsupervised anomaly detection of multivariate time series is an essential area in data mining and industrial applications. Reconstruction based models are commonly used in unsupervised anomaly detection. The reconstruction error is small for normal series while large for abnormal series. Based on this principle, most anomalies can be detected without the label. In the real world, the temporal patterns typically change time they are generated from non-stationary environments. For example, the growth the popularity of a service would cause customer metrics (e.g.. request count) drift upwards over time. Ignoring these factors would cause a deterioration in the performance of the anomaly detector. Despite the enormous advancements of previous research, most focus on stable data. Figure illustrates how the anomaly scores provided by existing approaches tend to increase in the red area when the data is unstable, leading to false alarms.
To handle the unstable data, attempt decompose it into stable component and trend component, focusing more on the stable part in the reconstruction procedure. However, the trend part cannot be ignored completely, the drift may be anomalous. There still two challenges in this
"Equal contribution.
t Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) SMD
(b) SwaT
Value
Data Tread
Value
Data Trend
32
Score
21
MTAD-GAT
D'R
Suore
MIAD-GAT
D
10-
-1
S00U
10000
15000
20000
25000
500
1000
1500
2000
2,500
Time
Time
Figure I: Anomaly scores of existing methods SMD and SWaT datasets. All moments are normal. Vertical drift occurs in the red area. The moment with a higher anomaly score is more likely to be identified as anomaly.
context: Challenge I: Limitation of the decomposition for long-perion time series. Most classical decomposition algorithms [2] [1 27] static, making it hard to be applied in the real world, where the data updated in real time. With deep learning, some dynamic algorithms have been proposed; however, they mainly based averaging 29 or Fourier Transform (28 26] within local sliding window. Fundamentally, they can not apply to the data whose period is larger than the size of the sliding window. Challenge 2: High training cost of adjusting the information bottleneck in the reconstruction procedure. The information bottleneck critical for reconstruction- and adjusting it is difficult. If it is too small, normal data will be poorly reconstructed; if it is too large, abnormal data will be successfully reconstructed. Previous methods most rely on an internal information bottleneck, such as the latent space size. As the information bottleneck is an attribute of the model itself, any change to the bottleneck necessitates retraining the model.
models,
After analyzing numerous data series, systematically organize the distribution shift into two categories. Vertical drift refers to statistical characteristics such as mean changing over time, while horizontal drift refers values shifting left right at similar times in various periods. In response the above challenges, we propose Dynamic Decomposition with Diffusion Reconstruction (D*R) for long-period unstable multivariate time series anomaly detection. To solve Challenge 1, utilize timestamps external information to overcome the limitations of the local sliding window, Specifically, the data-time mix-attention and offset subtraction are utilized solve vertical and horizontal drift, respectively. Moreover, we introduced a disturbance strategy in training to increase the robustness of the model. To solve Challenge we propose a new method named noise diffusion to control the information bottleneck externally. The diffusion (6) provides a new view of the information bottleneck. treating the noise bottleneck and the unpolluted information the bottleneck is no longer an attribute of the model itself, the different sizes can be set during revision without retraining the model.
condition. Because
The contribution of our paper is summarised as follows:
A novel dynamic decomposition method for long-period multivariate time series is proposed. It effectively utilizes external information to overcome the limitations of the local sliding window.
A new approach to control information bottleneck externally by noise diffusion is also proposed. It avoids the high training cost of adjusting the information bottleneck.
Based on the findings for the unstable data when taking unsupervised anomaly detection, we propose a novel anomaly detection network called D²R. The DoR achieves the new SOTA results on various real-world datasets and significantly outperforms baselines on unstable datasets.
2
Related work
As significant real- world problem, unsupervised anomaly detection of multivariate time series has received much attention. According to the criterion for anomaly detection, the paradigms roughly fall

---

Next block
Next block
Remove Disturbance
Out
Hiata
gesz
Reconstruction
Sub
Out
1x
Norm
Diffusion
Add & Norm
Food Forward
Ta
Nime
Food Forward
Offset Subtraction
Cal & Norm
Add Norm
Temporal
Spatial Nx
Decomposition Blocks
Higtt
Hu
OrdAttcation
Data Encoder
Time Encoder
MixAttention
X
Xtim
Embed
Embed X
Embed
Preprocess Disturb
Ta
Atim
H
time
Figure 2: The architecture of D*R mainly consists of two modules: dynamic decomposition and diffusion reconstruction. The detailed architecture of the decomposition blocks is shown on the right panel. The detailed architecture of the reconstruction backbone network is shown on the left panel.
into the categories of probability-based linear transformation-basedd proximity-based outlier-based, and neural network-based approaches.
As for the probability- based approach, multivariate cumulative distribution, providing anomaly scores based on the probability of occurrence of the sample. the linear transformation-basee method, [20] [2] begin by mapping the multivariate data before determining the boundary from the mapping space. Proximity-based algorithms, such
are modeled by statistical probability function on
as [8 5), seek to cluster data based on similarity and then calculate intra-cluster and inter-cluster distances. In the outlier-bane based approach, [14 16] compare the outliers degree of the testing samples with the training samples to identify the anomaly.
Most of the approaches mentioned above do not consider the temporal continuity of series. In recent years, neural network-based methods have become increasingly significant. Current neural network based approaches can be divided into prediction- based and acconstruction-based Although prediction- based methods [2] & are effective modeling for the next timestamp, they are susceptible
to interference from historical information. Reconstruction based approaches [32] [T] do a great
job capturing the distribution across the whole series. However, they are sensitive to the size of the information bottleneck.
Furthermore, existing works mainly foeus stable data, and their performance may suffer signifi- cantly in the real world, where the drift is frequent.
Method
input multivariate time series anomaly detection is denoted by X E Raak where n the length of timestamps, and & is the number of variables. The task is to produce output vector y IR", where y E 10,1) denotes whether the 7t timestamp is an anomaly.
3.1 Overview
The overall architecture of D is shown in Figure The dynamic decomposition module first models the data and timestamp features by the data encoder and time encoder. Next, uses stacked decomposition blocks to extract the stable component. Finally, we get the trend component by offset subtraction. The diffusion reconstruction module utilizes noise diffusion to construct information bottleneck externally and then directly reconstructs the polluted data by the backbone network. Reconstruction error is the anomaly score. order to model both temporal dependency and dimension dependency, the data encoder and the reconstruction backbone network both consist of stacked spatial-

---

temporal transformer blocks. To increase the robustness of the model, we also propose a disturbance strategy during training.
3.2 Data preprocessing
We perform timestamp hard embedding, labeled stable component construction, and disturbance strategy for the input. Timestamp hard embedding is applied to the training and testing sets, while labeled stable component construction and disturbance strategy are applied to the training set only.
Timestamp hard embedding To make better use of timestamps in anomaly detection, we hard- code the timestamps of [ into an embedding Xime E R2x6 like [34,22) [2]], with each dimension representing minute of the hour, hour of the day, day of the week, day of the month, and month of the year.
Labeled stable component construction We extract the trend T E R by moving average, then the labeled stable component stable components prevents our model from being disturbed when the training data is unstable.
T, E R obtained. The construction of the labeled
Disturbance strategy To increase the robustness of the model, add vertical drift d 1 sampled from a p-p.p] uniform distribution to each variable of the training data. The final input of dynamic decomposition module is X = X + d, Xa E Baxk
3.3 Dynamic decomposition
The dynamic decomposition module consists of data encoder, time encoder, stacked decomposition blocks, and an offset subtraction. The data encoder is implemented based on the spatial-tempona transformer block, which captures the temporal and dimension dependency. The output of data encoder Ham E d where dmmadel is hidden state dimension the model. The time encoder consists only of the temporal transformer block, which models the temporal correlation of timestamps to obtain Hime axxdmt Data-time mix-attention constitutes the subject of stacked decomposition
blocks, which are used to extract the stable component B Finally, to solve the challenges of horizontal drift, we obtain the trend component T E Ra*k through offset subtraction.
Spatial-temporal transformer block The architecture of the spatial-temporal transformer block shown the solid line box on the left panel of Figure 2 Assuming the input of 4 layer is H' xdd with timestamps and dmodel dimensions. In the temporal transformer, obtain ammporl E xd by directly applying multi-head self-attention (MSA) to n
temporal relationship H
vectors of size dmodet:
H emporal LayerNorm (H' + MSA H,,
(I)
H imporal LayerNorm H Hemprrra FeedForward
H ((iaopora)
where Emporrl is intermediate variable. LayerNorm(.) denotes layer normalization as widely
adopted in [25] 34 29), FeedForward(()) denotes multi-layer feedforward network, MSA(Q, K, V)
denotes the multi-head self-attention (25) layer where Q, K, V serve as queries, keys and values. In
the spatial transformer, obtain dimension relationship H Haputial E n by applying MSA to
dmodel vectors of size
H spatial LayerNorm
(HH') + MSA ((H'), (H') (H))))
(2)
(H (ii )"
FeedForward ((ik
H spatial LayerNorm
where Fr spatial is intermediate variable. (.)T denotes the transposition of matrices. Finally, obtain
the input of + tt layer H 1 R guods by:
H LayerNorm (H (Homporal ① H Happaiaa)
H' +1 LayerNorm (FeedForward (i)
(3)

---

where +1 E Rux22dust is intermediate variable, and @ represents concatenation.
Data-time mix-attention The original self-attention A K, V) [25] only models the data information and ignores the role of timestamps. We define the data-time mix-attention as:
QaKj + Q.K. V
Am Qq: Ka, Qu K, Vi) Softmax
vdz
where Softmax(-) is conducted row by row like [25 queries, data keys of dk dimension mapped from Hatta respectively. Q Kr Vr E Raxd are time queries, time keys and time values of dy dimension mapped from Hume- respectively. D*R learns to autonomously merge data and time information by mapping them into attention space.
29
Qa, Ka E Raxd are length-n data
Decomposition block The structure of stacked decomposition blocks shown on the right side of Figure 2 Assuming the input of t layer is H and H lime St 1 by: H Am H H Hama: H H time?
obtain the sub-stable components
H Hime)
data:
timer
H LayerNorm (H' Ao H.,
S LayerNorm
H FeedForward
(H)
S' Out ($)
where H', H E B intermediate variables. Out(-) is single linear layer. Then can obtain the input I 1t layer H H St and H
lime H Hime- Finally, the sub-stable components S' of all stacked decomposition blocks are summed to obtain the stable component Š of the series.
Offset subtraction Although the same time slots in different periods usually have similar fluctua- tions, is not strictly one-to-one correspondence. Here, we employ a simple method to handle the horizontal drift, considering the vector minus the vector b, with maximum horizontal offset d:
M (a,,b) b) Min (a, bi-d. a b;+a)
where a, and b, are the tth element in a and b, respectively. Min(.) means taking the minimum value. Based on the offset subtraction, we obtain the predicted trend component as Ta = MM $).
3.4 Diffusion reconstruction
The diffusion reconstruction module mainly consists of noise diffusion and reconstruction backbone network. The noise diffusion is used to construct an information bottleneck externally by polluting the input data with noise, and the backbone network used reconstruct the polluted data directly.
Noise diffusion Given the original data Zo, the polluted data 31, T2, FT are obtained by adding T step Gaussian noise. Each moment f in the noise addition process is only relevant to the - moment. The hyperparameter 3 controls the ratio of added noise. We add trend retaining to the DDPM so that the model can concentrate the more crucial stable components. The data x; after t steps of noise pollution is:
21 = Vorio VI- a. + +1 (1- v
where o I as a;
Br, 2, (0, 1) the noise, and is the retained information. The proof is given in Appendix A. So we can obtain the noisy data:
X
0 X + vT a (1 T.
(8)
whereX gank Z ~ N0 I), and Z E n
5

---

Table 1: Statistics of the datasets, A smaller ADF test statistic indicates a more stationary dataset.
Training Size
Testing Size
Series Number
Attacks Number
Anomaly Durations
Anomaly Rate
ADF Test Statistic
Frequency
PSM
132481
87841
25
73
1~8861
0.2776
minute
-9.2314
SMD
23688
23689
33
30
3~3161
0.1565
minute
-4.0947
SWaT
6840
7500
33
3~-599
0.1263
minute
-2.9442
As shown in the left panel of Figure 2 the backbone network consists of stacked spatial-temporal transformer blocks. The extracted trend Ta, the noisy data X/3, and the timestamp Xtime are encoded by the embedding layer first. To make the network more focused on the stable part without completely ignoring the trend part, we make the hidden vector:
Backbone network
H embed(X; X') embed (Ta) embed (Xiime)
whereH E Ruxdouke embed(.) is the embedding network. Then, we model the temporal and Transformer blocks. After that, we sum the outputs of the stacked blocks with Tu and send the result into the output layer. Finally, we directly obtain the reconstructed result X, of X2 instead of the predicted noise.
dimension relationship by feeding into the stacked
3.5 Joint optimization
As described in the previous section, the dynamic decomposition and diffusion reconstruction are interconnected. The dynamic decomposition module learns the inherent features of long-period unstable multivariate time series, i.e., the stable component. We use the Mean Square Error (MSE) between S and S the loss of this module. The diffusion reconstruction module directly recon- structs the data polluted by noise diffusion. It is worth noting that the output of this module the reconstruction of X.. Because the drift d is added inside D²R, and the external who calculates the loss is only aware of X, we obtain the reconstruction of X by X = X d, X E Raxk Similar to the dynamic decomposition module, the MSE of X and X is used directly as the loss of this module.
During the training process, D²R needs be trained end-to-end, so the loss function is defined as the sum of the two optimization objectives:
1 Loss
( + (X.ss X1.s)
(10)
nk
i=1==
where St.j. Si.j X X, i represent the labeled stable component, the predicted stable component, the original input and the reconstructed output of the jth variable at the 7t timestamp, respectively.
3.6 Model inference
The online detection (inference) does not need labeled stable component construction and disturbance strategy in the data preprocessing stage. The anomaly score for the current timestamp consists only of the MSE of the original input and the reconstructed output. Then, we run the SPOT [23] streaming algorithm based on extreme value theory, to label each timestamp.
4 Experiments
4.1 Experimental settings
Datasets We evaluate DRR extensively on three real-world datasets: PSM (Pooled Server Metrics) [.. SMD (Server Machine Dataset) [24], and SWaT (Secure Water Treatment) [15] For each dataset, we only preserve continuous variables. That is why we do not employ MSL (Mars Science Laboratory rover) and SMAP (Soil Moisture Active Passive satellite) [8] which are discrete except for the first dimension. The normal data is divided into training data (80%) and validation data (20%). Anomaly only present in the testing data. More descriptions of the datasets are shown in Table Further details about the datasets are available in Appendix B.I.

---

Table 2: Results in the three real- real-world datasets. The higher values for all metrics represent the better performance, and the best F1 scores are highlighted in bold.
PSM Recall
SMD
SWaT
Average
Method
Precision
F1
Precision
Recall
F1
Precision
Recall
F1
Fl
Sampling
0.8439
0,5165
0.6408
0.7453
0.3144
0.4422
0.6062
0.8466
0.7065
0.5965
COPOD
0,7602
0.3175
0.4479
0.6676
0.1366
0.2268
0,9876
0.1180
0.2108
0.2951
ECOD
0.7460
0.3384
0.4656
0.7398
0.1615
0.2651
0,9761
0.1151
0.2059
0.3122
OCSVM
0.8761
0.4744
0.6155
0.0000
0.0000
0.0000
0.6196
0.7558
0.6810
0.4321
PCA
0,9220
0.3771
0.5353
0,8388
0,4019
0,5434
0.6358
0.7218
0.6761
0,5849
KNN
0.5317
1.0000
0.6943
0. 0.6988
0.3368
0.4546
0,0000
0.0000
0.0000
0,3830
CBLOF
0,5990
0,9845
0.7449
0. 168667
0.3352
0,4834
0,6308
0.7091
0,6677
0.6320
HBOS
1,0000
0.0654
0.1228
0.5628
0.8007
0.6610
0,5771
018044
0.6722
01,4X33
Forest
1,0000
0.0335
0.0648
1 1,0000
0.0937
0.1713
0.6127
0.6280
0.6203
0,2855
LODA
0,9266
0,4017
0.5605
0. 0.5902
0.6618
0.6240
0.6117
0.7014
0,6535
0.6126
VAE
0.6221
0.8772
0.7280
0,8209
0,4349
0.5686
0.6355
0.7218
0.6759
0.6575
VDD
0.7405
0.5064
0.6015
0.6498
0.6477
0.6488
0.5911
0,9353
0.7244
0.6582
LSTM-AE
0.7511
0,7586
0.7548
0.8496
0.4349
0.5753
0.6018
0.7219
0.6564
0.6622
MTAD-GAT
0,7990
0.6014
0,6863
0,8590
0,6769
0.7571
0.6590
0.7751
0,7123
0.7186
TFAD
0.7914
0.7163
0.7520
0.5632
0.9783
0.7149
0.6038
0.8196
0.6953
0.7207
Anomaly Transfomer
0.5201
0.8504
0.6455
1.0000
0,0319
0.0619
0,5541
0.5994
0.5759
0.4278
Adversary
0,5351
0.8971
0.6703
0.5135
0,9663
0.6706
0,5410
0.7531
0,6297
0.6569
Our
0.6294
0,9619
0.7609
0,7715
0,9926
0.8682
0.7206
0.8529
0,7812
0,8034
Baselines We extensively compare our model with 15 baselines, including the probability-basee
methods: Sampling, COPÓD [12], ECOD 13] the linear transformation-bee bbased methods: OCSVM
20 PCA [2]; the proximity- -basd methods: KNN [8] CBLOF (5), HBOS [4; the outlier-based
methods: IForest [14] LODÁ [I6]: the neural network- methods: VAE 11, DeepSVDD (19),
Lstm-AE [21 MTAD-GAT 33 TFAD (1. Anomaly Transformer (30]. Additionally, we set up an adversary algorithm, which is implemented simply. A timestamp will be marked as abnormal if its sequential id is divisible by n, else it will be marked as normal. In our experiment, n is set as 40. This adversary algorithm works as timed detector and offers no information about the location of the anomaly. All baselines are based on our runs, using the identical hardware. We employ official or open-source implementations published in GitHub and follow the configurations recommended in their papers. Further details concerning the Baselines are available in Appendix B.2.
Metries For performance evaluation, we use precision, recall, and Fl score. The classical metrics are reasonable for tasks with sample granularity but are not applicable for continuous time series, where anomalies are frequently continuous.
Most previous work [30 33 3 21 employ the point adjustment method: If a point in a contigu- ous anomalous segment is detected correctly, all anomalies in the same segment are also con- sidered to have been correctly detected. How- ever, point adjustment is unreasonable, as [10) pointed out, and creates the illusion of progress. Assuming that ground truth and the predicted event are shown in Figure 3 although the real predicted event is just a timed detection, it still achieves an F1 score of 0.91 after point adjust- ment. The point adjustment may increase TP and decrease FN dramatically [10]. We test all basclines on all datasets with point adjustment metrics. The detailed experiment results can be found in Appendix C. Experimental results show that the average F1 score of the adversary algorithm surprisingly outperforms all baselines, even though it can provide no anomaly location information. This phenomenon arises because point adjustment unfairly assigns higher weights to long anomaly events. Furthermore, this algorithm does not consider the adjacency of time series 77. To address the aforementioned challenges, we employ an Fl score based on affiliation 77] This score takes into account the average directed distance between predicted anomaly events and ground truth events to calculate precision, as well as the average directed distance between ground truth events and predicted anomaly events to determine recall.
Value
Value
Adjusted
Real
7
12
24
36
60
Time
Figure 3: Example of point adjustment strategy. The red area represents the ground truth of the anomaly event. The precision/recall/f1 before ad- justment (Real) is 0.20/0.05/0.0 08, while after ad- justment (Adjusted) is 140/..0///.9.
7

---

(b)
0.90
SMD
0,85
Score
4000
8000
12000
15000
20000
0.80
Swar
Stable
0.75
2000
4mosel
Time
Parameter
Figure 4: Results of effectiveness analyses. (a) is the visualization in the dynamic decomposition. (b)
is the statistic results of sensitivity analyses in the diffusion reconstruction module.
Detailed implementation of R can be found in Appendix B.3. All experiments are repeated 5 times, and we report the average for each metric.
4.2 Detection Results
On all three real-world datasets, shown in Table 2 D²R outperforms the adversary algorithm and achieves the best Fl performance, confirming its eifectiveness and superiority, Specifically, we achieve 0.61% (0.7548 +0. 7609). 11.11% (0.7571-b (7)2),).. and 5.68% 047244-00.7822 absolute improvement over the previous SOTA methods on PSM, SMD, and SWaT datasets, respectively.
The statistical and machine learning methods frequently perform poorly in generalization because they do not account for time series continuity. They may perform well on a partial dataset while performing poorly even failing another (e.g., OCSVM and CBLOF). In opposition, neural network-based methods usually perform more balanced over various datasets. Furthermore, much of the previous works were evaluated by point adjustment, leading false boom. The metrics based on affiliation provide a more objective and reasonable evaluation of various methods, although their scores will drop.
Noticeably, D significantly outperforms the other methods on the SMD and SWaT datasets, which typically characterized by high nonstationarity, suggesting the limitation of previous work on unstable real-world data. In our model, the dynamic decomposition and diffusion reconstruction modules complement each other. The former dynamically eliminates the unstable interference from the original data. The latter models the series from more crucial components. These designs tackle the shortcomings of previous work and maintain excellent robustness in complex real-world data.
In addition to the affiliation evaluation metric, we compare D with other baselines in all datasets using the AUC score. The detailed experiment results can be found in Appendix D. For a more intuitive comparison, we visualize the anomaly scores of the D and partial baselines, as shown in Appendix E. The quantitative and qualitative experimental results all show that D outperforms other baselines. Moreover, to validate the practicality D the actual production environment, compare time across different neural network-based algorithms in the SMD and present summary in Appendix F. Both the training time and inference time our model are acceptable.
4.3 Effectiveness verifications
Dynamic decomposition module We propose a dynamic decomposition module to tackle Chal- lenge 1. To more intuitively verify the performance of the dynamic decomposition module, we provide visualization the decomposition results on highly unstable datasets (SMD and SWaT). Figure 44)) shows that model robustly extracts the fundamental stable components, although real-world multivariate data is long-period and complex.
Diffusion reconstruction module We propose a diffusion reconstruction module to tackle Chal- lenge 2. To verify the superiority of controlling information bottleneck externally, we replace the entire diffusion reconstruction module with the VAE module and perform sensitivity analyses for the VAE module and diffusion reconstruction module. The hyperparameter that significantly af-
8

---

Table 3: Results of ablation studies. Fl scores are reported, with higher values meaning better performance. The best scores are highlighted bold.
dataset
D
w/o temporal
w/o spatial
w/o
w/o
w/o offset
w/o disturbance
w/o diffusion
w/o
time
data
trend
PSM
0.7609
0.7280
0.7571
0.7074
0.7374
0.7420
0.7508
0.7347
0.7592
SMD
0.8682
0.8169
0.8173
0.7403
0.8114
0.8446
0.6947
0.8392
0.8006
SWaT
0.7812
0.7031
0.7274
0.7563
0.6720
0.7205
0.7293
0.7293
0.7109
Average
0.8034
0.7493
0.7673
0.7347
0.7403
0.7690
0.7249
0.7677
0.7569
fects the performance of the VAE module is the latent space size z. The hyperparameters that may affect the performance of the diffusion reconstruction module are the hidden state dimension dmodet- the noise ratio 8. and the pollution step T. We scale each hyperparameter by factor of {0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0}. the statistic results for the SMD are shown in Figure bb.. As well as allowing adjusting the information bottleneck without retraining, the diffusion reconstruction module is significantly more effective (i.e.. the F1 score is higher) and more robust (i.e., the FI score more concentrated) than the VAE module. During the hyperparameter changing, the Fl score span of the VAE module is 8.96% (..496-x088322) while the maximum span of the diffusion reconstruction module (3) 3.30% 0..811)..
4.4 Ablation studies
In order to verify the effectiveness and necessity of our designs, we perform ablation studies on both dynamic decomposition and diffusion reconstruction modules of D*R, respectively.
Dynamic decomposition module In this module, our main designs are spatial-temporal transformer, data-time mix-attention, offset subtraction, and disturbance strategy. Their ablation results shown in Table 3 The w/o temporal, w/o spatial, w/o time, w/o data, w/o offset, and w/o disturbance represent the variants of D removing temporal transformer, spatial transformer, time attention, data attention, offset subtraction, and disturbance strategy, respectively. The temporal transformer and spatial transformer bring an average absolute improvement of 5.41% (0.7493->0 -0.80344 and 3.61% % 00.663--0.8034 respectively. Compared with the spatial transformer, the temporal transformer is more critical. The spatial transformer performs much worse when the datasets have smaller series number (PSM and SWaT). When the series number is large (SMD), the dimension dependency is more prosperous, and the spatial transformer performs better. Moreover, time attention and data attention in data-time mix-attention bring great absolute promotion of 6.87% 77400->0.80$3 While data attention better suited to short-period data (SWaT), time attention is better suited to long-period data (PSM and SMD). Offset subtraction can further improve model with 3.27% (0.7690->0) 0.8004). Finally, the disturbance strategy brings a significant improvement of 7.85% (0.7249 +0.8034), especially for the SMD and SWaT datasets with high nonstationary characteristics.
and 6.31 %
Diffusion reconstruction module This module uses noise diffusion to control the information bottleneck externally. To verify the superiority noise diffusion, we replace the entire diffusion reconstruction module with the VAE module, and the ablation results are shown in Table 3 (w/o diffusion). As well as allowing adjustment the information bottleneck without retraining, noise diffusion also provides 3.57% (0.7677->0.sk will discuss more advantages of the diffusion reconstruction module. An additional innovation of this module is trend retaining, and the ablation results are shown in Table 3 (w/o retaining). After removing trend retaining, the performance of D decreases significantly, especially in the highly nonstationary SMD dataset 00.868220.8000 and SWaT dataset (0.7812 (0.7122-0.7109) Trend retaining enables the model to focus on the critical stable components, avoiding being distracted by irrelevant information.
absolute improvement. The following section
4.5 Hy Hyperparameter analyses
The hyperparameter that significantly affects the performance of the VAE module the latent space size The hyperparameters that may affect the performance of DOR are the added drift

---

P
0.87
0.92
0.92
Score
0.83
0.88
0..88
0.79
0.84
0.84
model
B
0.92
0.92
01.92
Soore
0.88
0.88
0.88
0.80
0.80
0.80
0.2
0.4
0.6
0.8
1.0
1.2 Factor
1.4
1.8
2.0
0.2 0.4
0.6
0.8
1.0 1.2 Factor
1.4
1.6
1.8
0.2 0.4
0.6
0.8
1. 1.2
1.4
1.8
Factor
Figure 5: Results of the sensitivity analysis. The higher Fl Score represents the better performance. The dark line represents the mean of 5 experiments, and the light area represents the range.
boundary p. the maximum horizontal offset d, the hidden state dimension dmnddel the noise ra- tio 3, and the pollution step T. To analyze their influence on anomaly detection, we perform hyperparameter sensitivity analysis in the SMD. We scale each hyperparameter by factor of (0.2, 00.2,0.4,00 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0), and the results are shown in Figure
Anomaly detection methods based reconstruction require constructing the information bottleneck to reconstruct normal data and inhibit anomaly effectively. While traditional autoencoders control the information bottleneck from inside through the latent space size, approach controls the information bottleneck from outside through noise diffusion. As the experimental results show, with the bottleneck going from strict to loose (latent space size noise ratio 3. pollution step T going from small to large), the performance the model goes from up to down. If the information bottleneck tight, the normal data cannot be reconstructed, and the precision is low. If the information bottleneck is too loose, the anomaly can be reconstructed well, and the recall is low. On the one hand, approach control the information bottleneck externally is more robust (smaller range of variation). On the other hand, as the information bottleneck is not an attribute of the model itself, we can try different bottleneck sizes when inference without retraining the model again.
Since is no longer an information bottleneck, D is not sensitive to the hidden state dimension dmodel- A larger hidden space tends to bring better results but also greater computational cost. As for the added drift boundary and maximum horizontal offset d, the impact on our model becomes small once they exceed certain threshold. Overall, D²R is robust all hyperparameters test.
5 Conclusion
This paper proposes Dynamic Decomposition with Diffusion Reconstruction (D²R) for long-period unstable multivariate time series anomaly detection cover the overlook in previous work. We first decompose the long-period unstable multivariate time series dynamically and then directly reconstruct the data polluted by noise diffusion. Extensive experiments prove that R significantly outperforms existing works. The method we proposed to break the limitation of the local sliding window is also meaningful for other long-period multivariate time series analysis tasks, such as prediction and imputation. Meanwhile, the approach to controlling information bottleneck externally can also be used for anomaly detection other modal data, such as picture, video, and log data.
Acknowledgments and Disclosure of Funding
This work supported in part by the National Natural Science Foundation of China under Grants (62101064, 62171057, 62201072, 6207 1067), in part by the Ministry of Education and China Mobile Joint Fund (MCM20200202, MCM201 80101), in part by the Beijing University of Posts and Telecommunications Mobile Research Institute Joint Innovation Center.
10

---

References
[1] Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2485-2499 2021.
[2] Robert В Cleveland, William S Cleveland, Jean E McRac, and Irma Terpenning. Stl: A seasonal-trend decomposition. Journal of Official Statistics, 6:3-73, 1990.
[3] Ailin Deng and Bryan Hooi. Graph neural network- based anomaly detection multivariate time series. In AAAI Conference Artificial Intelligence, pages 4027-4035, 2021.
[4] Markus Goldstein and Andreas Dengel. Histogram- outlier score (hbos): A fast unsuper- vised anomaly detection algorithm. German Conference on Artificial Intelligence, 1:59-63, 2012.
[5] Zengyou He, Xiaofei Xu, and Shengchun Deng. Puttern Recognition Letters, 24:1641-1650, 2003.
Discovering cluster-based local outliers.
[6] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Neural Information Processing Systems, pages 6840-6851, 2020.
[7] Alexis Huet, José Manuel Navarro, and Dario Rossi. Local evaluation of time series anomaly detection algorithms. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 635-645, 2022.
[8] Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Söderström. Detecting spacecraft anomalies using Istms and nonparametric dynamic thresholding. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 387-395, 2018.
[9] Tung Kieu, Bin Yang, and Christian S. Jensen. Outlier detection for multidimensional time series using deep neural networks. In IEEE International Conference on Mobile Data Management, pages 125-134, 2018.
[10] Siwon Kim, Kukjin Choi, Hyun-Soo Choi, Byunghan Lee, and Sungroh Yoon. Towards a rigorous evaluation of time-series anomaly detection. In AAAI Conference on Artificial Intelligence, pages 7194-7201, 2022.
[11] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. Conference Learning Representations, 2014. 2014.
International
[12] Zheng Li, Yue Zhao, Nicola Botta, Cezar lonescu, and Xiyang Hu. Copod: Copula-based outlier detection. In IEEE International Conference on Data Mining, pages 1118-1123, 2020.
[13] Zheng Li, Yue Zhao, Xiyang Hu, Nicola Botta, Cezar Ionescu, and George H. Chen. ECOD: unsupervised outlier detection using empirical cumulative distribution functions. arXiv, 2201 2201.00382, 2022.
[14] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou Isolation forest. In IEEE International Conference on Data Mining, pages 413 422,,22,44 2008.
[15] Aditya P. Mathur and Nils Ole Tippenhauer. Swat: A water treatment testbed for research and training on ies security. In CySWater, pages 31-36, 2016.
[16] Tomás Pevný. Loda: Lightweight on-line detector of anomalies. Machine Learning, 102: 275-304, 2016.
[17] Jinwen Qiu, S. Rao Jammalamadaka, and Ning Ning. Multivariate bayesian structural time series model. Journal of Machine Learning Research, 19:68:1-68:33 2018
[18] Sridhar Ramaswamy, Rajeev Rastogi, and Kyuseok Shim. Efficient algorithms for mining outliers from large data sets. In ACM SIGMOD International Conference on Management of Data, pages 427-438, 2000.

---

[19] Lukas Ruff, Nico Görnitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Robert A. Vandermeulen, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4303-4402, 2018.
[20] Bernhard Schölkopf, John C. Platt, John Shawe-Taylor, Alexander J. Smola, and Robert C. Williamson. Estimating the support of a high-dimensional distribution. Neural Computation, 13:1443-1471, 2001. 2001.
[21] Lifeng Shen, Zhuocong Li, and James T. Kwok. Timeseries anomaly detection using temporal hierarchical one-class network. In Neural Information Processing Systems, 2020.
[22] Meiling Shyu, Shuching Chen, Kanoksri Sarinnapakorn, and Liwu Chang. A novel anomaly detection scheme based on principal component classifier. Technical report, Miami Univ Coral Gables Fl Dept of Electrical and Computer Engineering, 2003.
[23] Alban Siffer, Pierre-Alain Fouque, Alexandre Termier, and Christine Largouët. Anomaly detection in streams with extreme value theory. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1067-1075, 2017.
[24] Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2828 2837, 2019.
[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, pages 1908, 2017.
[26] Zhiyuan Wang, Xovee Xu, Weifeng Zhang. Goee Trajcevski, Ting Zhong, and Fan Zhou. Learning latent seasonal-trend representations for time series forecasting. In Neural Information Processing Systems, 2022.
[27] Qingsong Wen, Jingkun Gao, Xiaomin Song, Liang Sun, Huan Xu, and Shenghuo Zhu. Robust- stl: A robust seasonal-trens decomposition algorithm for long time series. In AAAI Conference on Artificial Intelligence, pages 1409-5416, 2019.
[28] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In International Conference on Learning Representations, 2022.
[29] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Neural Information 2021.
Processing Systems, pages
[30] Jiehui Xu, Haixu Wa, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. In International Conference on Learning Representations, 2022.
[31] Chaoli Zhang, Tian Zhou, Qingsong Wen, and Liang Sun. TFAD: A decomposition time series anomaly detection architecture with time-frequency analysis. In ACM International Conference on Information & Knowledge Management, pages 2497-2507, 2022.
[32] Weiqi Zhang, Chen Zhang, and Fugee Tsung. Grelen: Multivariate time series anomaly
detection from the perspective of graph relational learning. In International Joint Conference on Artificial Intelligence, pages 2390-2397, 2022.
[33] Hang Zhao, Yujing Wang, Juanyong Duan, Congrui Huang, Defu Cao, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. Multivariate time-series anomaly detection via graph attention network. In IEEE International Conference on Data Mining, pages 841-850, 2020.
[34] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI Conference on Artificial Intelligence. pages 11 11106-111 1111,, 2021.
12

---

A Noise diffusion proofs
In this section, we explain the proof of Equation 7 in our paper.
Suppose 2 ~ (0, 1) is the Gaussian noise to be added, & controls the amount of noise added each step, B, and b is the trend information be retained. The 2t after steps of noise pollution is:
Voi (II-1 - b) + vI 042 + b 1 + vI ae + (1 / The Gaussian distribution, as we know, has the following properties:
(11)
Suppose [N, (²), then ax +b~ Nap + 6, d²²o²).
Suppose N (M2, oz), (Ny, y are independent random variables, then x + yy N( My, o + to)))
Based on these properties, the noise diffusion process can be deduced further:
34 aau (ft-1 b) + v1 Ct +
aa 01 (1r b) + v - a1-12 + b b) + vI 0+ +
(12)
- + a104-12 + 10
Repeating the above process, we can directly infer from as follows:
2, Vārzo
a (1 vā.) b
(13)
where -I a and & ~ ().
B
Detailed experimental settings
B.1
Datasets
The datasets can be downloaded from the following:
PSM:
SMD:
SWaT:
B.2
Baselines
All baselines are based on our runs, using the identical hardware. We employ official or open-source implementations published in GitHub and follow the configurations recommended in their papers.
The baselines can be downloaded from the following:
Sampling: https//ggihubb.omm/zzzaaa222220
h https///gthhbbccm///////////////
COPOD:
ECOD: ECOD: http:h/://gghhbboooo//y/////////
PCA: http://s//ghllbbcco//////o//////
kNN: httpss//gihhubbco////yyooo/////
CBLOF: h https:/ggihaabccoom////hoo//////
HBOS: https://gthubb.oomm//yha////y/y
IForest: http::/ggihub.commyyyaaaa///////
LODA: https//githbbbcom///yhho///////
VAE:
https//gihlubbcom///zloooo//////
13

---

Table 4: Quantitative results after point adjustment in the three real- world datasets. The higher values for all metrics represent the better performance, and the best F1 scores are highlighted in bold.
Method
PSM Recall
SMD
SWaT Recall
Average Fl
Precision
F1
Precision
Recall
F1
Precision
Sampling
0.9986
0,8571
0.9224
0.9958
0,9690
0,9822
0,1583
0.9271
0.2705
0.7250
COPOD
0.9795
0,8572
0.9143
0.9975
0.8538
0.9201
0,9953
0.6695
0,8005
0.8783
ECOD
0.9565
0.8632
0.9075
0.9645
0.8573
0.9078
0,9954
0.6832
0.8103
0.8752
OCSVM
0,9990
0,8115
0.8955
0.0000
0.0000
0.0000
0.1596
0.9113
0.2716
0,3890
PCA
0.9996
0.7777
0.8748
0.9964
0.9720
0.9840
0.1645
0.9029
0.2783
0.7124
KNN
0.2776
1,0000
0.4345
0,9950
0.9676
0.9811
0,0000
0.0000
0.0000
0.4719
CBLOF
0,4301
0,9996
0.6014
0.9970
0,9698
0.9832
0,36
0.9029
0.2769
0.6205
HBOS
1,0000
0.5957
0.7466
0.3745
0,9922
0.5437
0,3932
01,8965
0,5467
116613
FFrret
1,0000
0.2204
0.3612
1 1,0000
0.8560
0.9224
0.4204
0.8141
0.5545
0.6127
LODA
0.9975
0.8465
0.9158
0 0.8452
0,9803
0.9077
0.2502
0.8912
0.3907
0.7381
VAE
0,6020
0,9486
0.7365
0.9953
0,9720
0.9835
0,1646
0,9029
0.2784
016661
VDD
0.6919
0.8779
0.7739
0.9575
0,9714
0.9644
0,1465
0,9842
0.2551
0.6645
LSTM-AE
0.7280
0,9288
0.8162
0.9961
0,9720
0.9839
0.1614
0,9029
0.2739
0.6913
MTAD-GAT
0,9589
0.8987
0.9279
0.9940
0,9857
0,9898
0.1745
0.9197
0.2934
0.7370
TFAD
0.8819
0.9851
0.9306
0.9991
0.9492
0.9735
0,2559
0,9387
0.4022
0.7688
Anomaly Transfomer
0.9022
0,9879
0.9431
1,0000
0.8525
0.9204
0.9076
0.7159
0.8005
0,8880
Adversary
0.9384
0.9921
0,9645
0.8774
0.9649
0.9191
0,8309
0.8300
0.8304
0.9047
DeepSVDD: DeepSVDDD http:///hhhhuu.cmm///yy//o/2o2b
LSTM-AE: httpa///h/tppgsssiiocccoo/ooonna Autbeee
TFAD: http://ghhhlcccooo////oooo////at
Anomaly Transformer: http::/gghhhaacccmm/mmmm////oona
B.3 Implementation
In our experiment, the sliding window has a fixed size 64 for all datasets. We set the dimension of hidden states 512. the dimension of attention 64, the number of attention heads as 8. the number of network layers as 2, and the dropout as 0.6. We use grid search to obtain the best SPOT parameters for each dataset and record the results with the highest F1 scores. The 3 of the diffusion model changes from 0.0001 to 0.02 in 1000 steps like DDPM without adjustment. We add noise with 500 steps (selected from {100, 300, 500, 700, 900)) as external information bottleneck when the noise diffusion. The boundary of added drift in the disturbance strategy is 10 (selected from (1, 2.5, 5, 10, 20)). The max offset of offset subtraction is 30 (selected from (5, 10, 20, 30, 40)). We use the Adam optimizer with an initial learning rate of 10-4, The training process is early stopped within 8 epochs with batch size of 8. The implementation of D is carried out using Python 3.9.13 and PyTorch 1.1..0. All experiments are performed on Ubuntu Server with a I 12th Gen Intel(R) Core(TM) 19-129005 @ 3.60GHz processor and NVIDIA GeForce RTX 3090 Graphics Card.
C Detection results with point adjusted
We test all baselines on all datasets with point-adjusted evaluation metrics, and the detailed quantitative results are shown in Table | Since the data processing and hardware are not the same as those in the original paper, the results of the baselines in our experiments slightly different from those in the original paper. Nevertheless, the errors are still within a reasonable range.
The point adjustment method is unreasonable. Despite the no helpful information provided by the adversary algorithm, its average performance still exceeds other SOTA methods by 2% of the relative Fl score. Specifically, the adversary algorithm achieves an absolute improvement of 2. 14% (0.9431-+0.964)) and 2.01 % 10.8103-0.880 +0.8304 on the PSM and SWaT datasets, respectively. Only on the SMD dataset the adversary algorithm is slightly weaker than the best baseline (0.9898 *0.9111. If we further adjust the n in the adversary algorithm, it can also outperform all the baselines in the SMD dataset.
14

---

Table 5: AUC score in the three real-world datasets, The higher AUC score represents the better performance, and the best AUC scores are highlighted in bold.
Method
PSM
SMD
SWaT
Average
Sampling
0.8791
0,9611
0.5348
0.7917
COPOD
0.8526
0.9230
0.7396
0.8384
ECOD
0.8394
0,9220
0.7532
0.8382
OCSVM
0.8708
0.6789
0.5379
0.6959
PCA
0.8617
0.9671
0.5324
0.7871
KNN
0.8640
0,8935
0.5806
0.7793
CBLOF
0.8681
0.9694
0.5378
0.7918
HBOS
0.8150
0.7393
0.8085
0. 0.7876
IForest
0.8892
0.9218
0.7238
0.8450
LODA
0.8619
0.9180
0.6780
0.8193
VAE
0.8583
0.9674
0.5325
0.7861
DeepSVDD
0.8100
0.9187
0.5063
0.7450
LSTM-AE
0.8894
0.9698
0.6255
0.8283
MTAD-GAT
0.9093
0.9443
0.6386
0.8307
TFAD
0.8185
0.9386
0.6966
0.8179
Anomaly Transformer
0.7074
0.7150
0.6638
0.6954
Adversary
0.7199
0.5374
0.6487
0.6353
Our
0.9223
0.9759
0.8554
0.9179
D Additional detection results
In order to avoid the influence of SPOT parameters and thresholds on the results, we compare the methods from the perspective of the original anomaly scores. If anomaly lasts too long, more weight will be assigned, resulting in an inaccurate metric. Thus, we aggregate each anomaly event. An anomaly event after aggregation is equivalent to only one timestamp, labeled an anomaly, and scored as the maximum of the original anomaly range.
We use Area Under Curve (AUC) to evaluate the anomaly scores of each method. The value of AUC ranges between 0.5 and 1. while the closer to 1.0, the better the method. The results of the experiments shown in Table DOR significantly outperforms existing methods, with 9% average relative improvement over other SOTA methods. Specifically, we achieve 1.30% (0.9093->0.922 0.61% % (0.9698 +0.9759), and 4.69% performance on PSM, SMD and SWaT datasets, respectively. This experiment proves the superiority of D once again.
absolute improvement over the best SOTA
E Anomaly score visualizations
visualize anomaly scores the partial baseline and D²RR a more intuitive comparison, as shown in Figure 6 D²R consistently provided the most distinguishable anomaly scores in all three real-world datasets. the PSM dataset, the vertical drift is insignificant. In this case, some of the baselines (MTAD-GAT) (Anomaly Transformer) and false detections (VAE). In the SMD and SWaT datasets, the vertical drift of the data is significant. The anomaly scores of baselines in the normal region already far from the training data. The unstable normal data is likely to be mistakenly detected as an anomaly. Once the anomaly occurs, even if the anomaly scores of the baselines improve, they not significant enough due to the large base.
achieve correct detection results, but there still missed detections
F
Run Times
We comprehensively compare training time, inference time, and model size for neural network-based models on the SMD dataset validate the practicality of D&R the production environment, The outcomes presented in Table 6
15

---

PSM
SMD
SWaT
2.1
Train Test
15
Tran
Train Test
Test
0.7
2
0.0
6-
12
6
Train
Traic Test
Test
VAB
Train
m
Train
Train Test
Test
MMW Mww.
DeepS
Train Test
0-
1.8
0.6
Train
Train Test
Test
LstmAB
1.2
Train
0.6
1.5
Train
Train Test
MTAD-GAT
Test
1.0
Train Test
mmnum
0.5
0.2
0.0
0.0
0-
1.5
24
Train
Train Tost
Transformer
Test
Anomaly
1.0
16
Train
0.5
5.4
Train
Train
Train
3.6
1.8-
1.4
3
0.0
0.0
0
Time
Time
Figure Visualization of the anomaly scores the three real-world datasets. The red area represents the ground truth of the anomaly event. In the first row, visualize the raw data. In the remaining six rows, we visualize the anomaly scores provided by different methods. The moment with higher anomaly score is more likely to be identified as anomaly. Green boxes bound the successful cases. We normalize the data according to the statistical characteristics of the training set. Test denotes the actual value of the testing after the normalization, while Train denotes the average value of the training set after the normalization, which is always 0.
16

---

Table 6: Run times in the SMD. The lower values represent the better performance. The best results are highlighted in bold, and the worst results are underlined.
Method
Training Time (s)
Inference Time (s)
Model Size (MB)
VAE DeepSVDD LSTM-AE
157.91 91
30,90
0.02
60.70
12.61
0.01
283.61
72.82
0.04
MTAD-GAN
188.52
60.23
1.20
TFAD
315.39
38.54
1.04
Anomaly Transformer
422.43
94.36
28.15
Our
399.32
104.12
109.35
Both the training set and the test set of SMD have approximately 16 days of data. Statistically, the training time for these models remains below 10 minutes, which is acceptable for real-world deployment and maintenance. The utilization of attention and its variants in the backbone network of our model leads to its larger size compared to previous algorithms. Thanks to the highly parallelized nature of the attention mechanism, both the training time and inference time of our model remain competitive. The inference process for 16 days of data necessitates mere 104.12 12 seconds, satisfying the online, real-time detection criteria. Compared to the substantial enhancement (11% (1%) in detection accuracy that we have achieved, the model size of 109.35MB remains affordable within the context of the expanding hardware resources of the present era. Additionally, there has been substantial recent work (such as Flowformer) on transformer linearization, which may aid in reducing the burden of our model. We plan to explore this aspect in our future research.
G Broader Impacts
Recently, Deep Learning has been increasingly used in anomaly detection. It improves safety and prevents potential risks and financial losses by detecting anomalies in healtheare, industrial manufacturing, and autonomous driving. However, since real-time data is constantly changing, data characteristics can change drastically as time accumulates. Due to the obsolescence of the training data, our models cannot adapt to this situation and may provide incorrect judgments. In the future, we consider designing algorithms that adopt online updates to address this shorteoming.
H Limitations
The computational cost of our model is large due the Transformer architecture used for the backbone network. In future work, we plan to replace it with a more lightweight and efficient base structure, such as dilated convolutions and graph neural networks. Furthermore, due to the shortcomings of the point-adjustment evaluation metric, in order to prove the superiority of D we have to compare various other metrics and provide rich visualizations. We plan design more intuitive and effective anomaly detection evaluation metric in future work.
17


Reviews:
[{"review_id": "DcVllbkcra", "reviewer": "Reviewer_njrU", "summary": "Current unsupervised methods for multivariate time series anomaly detection often overlook drift from non-stationary environments, leading to false alarms. To address this, this paper presents Dynamic Decomposition with Diffusion Reconstruction (D3R), a new anomaly detection network for unstable data. D3R decomposes and reconstructs drift, using data-time mix-attention for dynamic decomposition and noise diffusion to manage the information bottleneck in reconstruction. This end-to-end trainable model outperforms existing methods, showing a 12% average improvement over previous top-performing models.", "questions": "The authors can refer to the weakness listed above.", "limitations": "The authors do acknowledge some minor limitations in their work. However, the primary limitation that is yet to be addressed pertains to the types of anomaly patterns their model cannot detect. ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The outcomes of the experiments seem to be promising.\n\n2. The problem addressed is intriguing and holds substantial value.\n", "weaknesses": "1. The challenge suggested by the authors is debatable. They assert that \"classical decomposition algorithms cannot be applied to data with a period larger than the size of the sliding window.\" In my view, this problem seems relatively simple to tackle by merely extending the length of the sliding window. Additionally, the authors seem to continue to depend on the moving average method to ascertain the trend and the labeled stable components. Therefore, they need to clarify how their technique specifically addresses this problem.\n\n2. The rationale for creating dynamic decomposition remains vague. The sole beneficial intermediate outcome yielded by dynamic decomposition is T_d, recognized as the predicted trend component. Nevertheless, it seems that there is a more direct and feasible approach to extract the trend component, much like the authors have done during the data preprocessing phase. The necessity of introducing this intricate trend component, T_d, in place of using the standard trend component, demands clarification.\n\n3. The concern of dynamically modifying the information bottleneck doesn't seem to be a common occurrence in real-life scenarios. I struggle to envision a situation where it would be necessary to adjust the information bottleneck during the inference phase. The authors need to elaborate on the practical value of employing the diffusion model if they aim to emphasize their contribution to reducing the cost of modifying the information bottleneck. Furthermore, offering a principle to guide the dynamic setting of these hyperparameters could make their method more persuasive.\n\n4. No experiments have been conducted to substantiate their method's superiority concerning reducing the high training cost associated with adjusting the information bottleneck.\n\n5. There is insufficient engagement with previous research. The ensuing paper also examines the impact of the trend component in anomaly detection, but the authors seemingly fail to appropriately cite this paper or compare their method with it.\n[1] Zhang, Chaoli, et al. \"TFAD: A decomposition time series anomaly detection architecture with time-frequency analysis.\" Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022.\n\n\n\n\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We will answer the questions one by one.\n\n**Q1: Necessity and details of dynamic decomposition.**\n\n**Q1.1: Why not over extend the length of the sliding window?**\n\nBelow, we shall expound the reasonableness of Challenge 1 (Line 29 in the paper). Expanding the length of the sliding window excessively has constrained its adaptability due to the expensive computational resources and huge memory consumption. Taking the SMD dataset with a period of 1440 as an example:\n\n- Expensive computational resources: extending the length of the sliding window results in a corresponding explosion of model parameters. Many previous algorithms, such as MTAD-GAT [1], utilize a sliding window of 128 or less. If we extend the length to 1440 or larger, the model parameters of the input layer alone increase by a factor of 10x.\n\n- Huge memory consumption: when the model is deployed online in the real world, the incoming data stream must be cached for a full sliding window before being fed into the model. This results in a 10x increase in memory consumption.\n\n**Q1.2: Why we can break the limitations of local window?**\n\nAs two reviewers have question about this question, we provide a consistent response in **Q2** of the \"global\" response. Thanks.\n\n**Q2: Necessity of the predicted trend component.**\n\nThe core of this question is the same as **Q1**. Obtaining the trend component directly using the moving average, as done in the data preprocessing phase, is more direct. However, the method necessitates a sliding window length several times larger than the period. Otherwise, the \"standard trend component\" merely represents local smoothing of the original series, lacking genuine trend extraction. In the case of long-period time series, employing such a large sliding window presents many problems.\n\n**Q3: Necessity of dynamically modifying information bottleneck and the  principle of hyperparameters setting.**\n\nBelow, we shall expound the reasonableness of Challenge 2 (Line 34 in the paper). Firstly, we express regret for confusion arising from our imprecise statement (line 52 in the paper). The term \"inference\" would be more appropriately substituted with \"revision\". In truth, the requirement to adjust information bottlenecks not only exists but is also common in real world:\n\n- Training: when deploying the anomaly detection model on a novel scenario, guaranteeing the availability of the initial information bottleneck size proves challenging. Consequently, it becomes necessary to make multiple hyperparameter adjustments to revise the model.\n- Inference: in a real industry environment, temporal patterns are likely to change over time. The information bottleneck size set during training may not always yield satisfactory performance during inference. Hence, it becomes necessary to adjust it to revise the model.\n\nUnlike previous methods that require retraining the model when adjusting the information bottleneck, our approach can flexibly modify the information bottleneck without the need for retraining. Consequently, our method substantially reduces the training cost associated with information bottleneck modifications, even approaching close to zero cost.\n\nRegarding the principles of setting hyperparameters, a thorough analysis and discussion have already provided in Appendix F.\n\n**Q4: Verify the superiority of the model in reducing training cost.**\n\nOur algorithm inherently possesses the capacity to reduce the high training cost associated with adjusting the information bottleneck. This superiority stems from the fundamental design logic of our method.\n\nSpecifically, in contrast to previous methods, our approach can flexibly adjust the bottleneck without the need for retraining. In other words, our method incurs close to zero cost solely for adjusting information bottleneck. Furthermore, as shown in Figure 4(b) in the paper, our algorithm exhibits superior performance and greater insensitivity to parameters. This advantage is evident as our algorithm achieves excellent performance with fewer adjustments. The negligible cost of a single adjustment and the limited number of adjustments required both substantiate the superiority of our design.\n\n**Q5: Insufficient investigation of previous research.**\n\nApologies for not being able to provide a comprehensive comparison of previous research. In the revised version, we will include relevant analyses about TFAD [3].\n\nTFAD notes the significance of trend in anomaly detection. However, for series decomposition, they employ the method based on HP filter during both training and inference. This static method lacks applicability for real-world scenarios where data is updated in real-time. Furthermore, they use the distance between the context window and suspect window as the anomaly score. This method heavily relies on the assumption that the context window must be normal, which is challenging to ensure in complex real world.\n\nWe conduct comparison experiments between our method and TFAD using the official open-source code. The experimental results are summarized in the table below.  The detailed version is available in the \"global\" response (Table 1 in the attached PDF). As analyzed in the previous paragraph, our method achieves the best experimental results.\n\n| Method |  PSM (F1)  |  SMD (F1)  | SWaT (F1)  | Average (F1) |\n| :----: | :--------: | :--------: | :--------: | :----------: |\n|  TFAD  |   0.7520   |   0.7149   |   0.6953   |    0.7207    |\n|  Ours  | **0.7609** | **0.8682** | **0.7812** |  **0.8034**  |\n\n**References:**\n\n[1] Hang Zhao, et al. “Multivariate time-series anomaly detection via graph attention network.”, In *ICDM*, 2020.\n\n[2] Chaoli Zhang, et al. “TFAD: A decomposition time series anomaly detection architecture with time-frequency analysis.”, In *CIKM*, 2022. "}, {"review_id": "NTYjnDcKkP", "reviewer": "Reviewer_crri", "summary": "This paper tackles the problem that existing works omit the drift generated from non-stationary environments by focusing on stable data, which may lead to numerous false alarms. \nAs a solution, they propose an end-to-end anomaly detection network for real-world unstable data, named Dynamic Decomposition with Diffusion Reconstruction (D3R). \nIn the decomposition stage, they dynamically decompose long-period multivariate time series by utilizing data-time mix-attention to overcome the limitation of local sliding window. \nIn the reconstruction stage, they control the information bottleneck extenerlly by noise diction and directly reconstruct the polluted data. \nThey evaluate on three real-world datasets (PSN, SMD, SWaT), and achieve the best performance compared to existing unsupervised anomaly detection methods.\n", "questions": ".", "limitations": ".", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Well-motivated and soundness: The problem that existing methods overlook the anomaly score in unstable data. Their proposed dynamic decomposition for long-period multivariate time series and diffusion reconstruction for controlling information bottleneck is sound.\n\nAblation study and Analysis: In sections 4.3 and 4.4, they analyze their proposed dynamic decomposition module and diffusion reconstruction module. This helps to understand the effect of the proposed modules.\n\nWell-structured paper: Their paper is well-organized and easy to read. ", "weaknesses": "Computational cost: there is no comparison of computational cost (inference time, # of params, Gflops ..) compared to existing models. For check the efficiency, they should report the comparison of computational cost.\n\nMinor comments: \nRepeated citation [11] and [12] of references \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments and insightful suggestions. Please find our response below.\n\n**Q1: Experiments on computational cost.**\n\nAs three reviewers have posed a similar question, we provide a consistent response in **Q1** of the \"global\" response. Thanks.\n\n**Q2: Repetitive citation in references.**\n\nThanks for pointing out our error, we will correct it in the revised paper."}, {"review_id": "IV8ZiwTi0f", "reviewer": "Reviewer_KWMB", "summary": "The paper presents a Transformer-based model called Dynamic Decomposition with Diffusion Reconstruction for Anomaly Detection in unstable multivariate time series. The authors addressed two challenges: the limitation of decomposition for long-period time series and high training cost for adjusting the information bottleneck in the reconstruction procedure. They introduced a novel dynamic decomposition method and a noise diffusion method to enable effective utilization of external information, overcome the limitations of local sliding windows, and avoid the high training cost associated with adjusting the information bottleneck. The proposed model achieves state-of-the-art results on different real-world datasets and outperforms baselines significantly on unstable datasets.", "questions": "see weaknesses", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  the authors presenting a clear exposition of their research motivation and proposed solution.\n2. The authors' methods effectively tackle two crucial issues: limitations of decomposition for long-period time series and high training cost of adjusting the information bottleneck.\n3. The authors conducted experiments on diverse datasets, and their proposed method demonstrated excellent performance.", "weaknesses": "1. The authors used the mean of five runs as the final results in their experiments but did not provide the standard deviation of the results.\n\n2. While the model achieved the best F-score performance due to higher recall values, its precision values were comparatively lower than other models. The authors should explain this observation.\n\n3. Further explanations for Figure 1 are recommended to clarify why other methods failed in this specific scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments and insightful suggestions. Please find our response below.\n\n**Q1: Standard deviation of the experimental results.**\n\nDue to spatial limitations, we exclusively present the mean of the results from the five runs in the paper, as it offers a more representative depiction of the method's performance. The detailed version of the primary experimental results (Table 2 in the paper) is available in the \"global\" response (Table 1 in the attached PDF).\n\n**Q2: Precision values of the model are comparatively lower than others.**\n\nThe primary reason for this phenomenon is that our selection criterion for the grid search of the SPOT parameters is based on the F1 score (line 52 in the Appendix). In other words, we prioritize parameters that yield the highest F1 score. Precision and recall are significantly impacted by the threshold in unsupervised anomaly detection method, where a high threshold results in relatively high precision, and a low threshold leads to relatively high recall. Therefore, we primarily focus on the comprehensive performance of the model, i.e., the F1 score. Additionally, to mitigate the influence of SPOT parameters and thresholds, we also employ AUC to evaluate the raw anomaly scores (Table 2 in the Appendix), and our model demonstrates superior performance in this evaluation as well.\n\n**Q3: Further explanation of Figure 1.**\n\nThe core of the unsupervised anomaly detection method is to acquire knowledge of normal temporal patterns from the training data. Moments in the test data that significantly deviate from these established patterns are classified as anomalies. As the data is collected from a non-stationary real-world environment, the temporal patterns may drift over time. The flat drift is not typically an anomaly. Previous methods have often overlooked this aspect, resulting in misclassifying moments with drift as anomalies. For these methods, moments with drift indeed deviates from the established incomplete temporal patterns."}, {"review_id": "Nc8CRk4Hur", "reviewer": "Reviewer_xTrQ", "summary": "To overcome the temporal drift issues in unstable time series data, this work proposes an anomaly detection method, $D^3R$. By considering the temporal continuity of series and relieving the constraints of information bottleneck, $D^3R$ realizes the dynamic decomposition and the noise-diffusion-based series reconstruction. Extensive experiments on real-world time series datasets demonstrate the superiority of $D^3R$ w.r.t. anomaly detection. ", "questions": "1. As shown in experiments, the performance improvement of $D^3R$ on PSM is less impressive than other datasets. You highlight that $D^3R$'s performance improvement is more significant on the time series data with high nonstationarity. Does this mean that the application of $D^3R$ might be limited to the highly unstable time series data? ", "limitations": "Potential applicability issues should be elaborated more. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This work is well motivated by tackling the overlooked issues in existing time series anomaly detection works. \n2. The proposed $D^3R$ approach employs decomposition and reconstruction for time series anomaly detection, which is technically sound. \n3. This draft is well organized, and the presentation is clear. ", "weaknesses": "1. Time series anomaly detection is a well-studied problem. More datasets are expected to conduct experiments to validate the proposed method's effectiveness, such as the Yahoo dataset (Webscope dataset ydata-labeled-time-series-anomalies-v1 0, 2015). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments and insightful suggestions. Please find our response below.\n\n**Q1: Reason for the dataset selection.**\n\nThanks for your suggestions. The datasets utilized in our research encompasses both server (PSM, SMD) and water treatment (SWaT) scenarios. Additionally, stable (PSM) and unstable (SMD, SWaT) conditions are included. These datasets can already offer a comprehensive validation of our model's superiority. While the Yahoo dataset is acknowledged as a classic anomaly detection dataset, it consists solely of univariate time series, which may not exactly match with our scenario.\n\n**Q2: Applicability of the model.**\n\nBased on the experimental results (Table 2 in the paper), our model demonstrates more substantial improvements when applied to unstable time series data. However, it is equally well-suited for stable data, such as PSM, where we achieve the best performance despite not observing a significant improvement. Technically, stable data undergoes the dynamic decomposition module, producing a predicted trend component $\\hat{\\mathbf{T}}_\\text{d}$ that approximates a constant. This component does not adversely affect the subsequent diffusion reconstruction module, as it still allows the module to robustly control the information bottleneck from external sources and effectively leverage our model's strengths."}, {"review_id": "6J4PleQ1yD", "reviewer": "Reviewer_WuHJ", "summary": "The authors propose a new model called D3R (Decomposition with Diffusion Reconstruction) for anomaly detection in multivariate time series data. The proposed model applies a dynamic decomposition method to separate the stable components and trends in time series data. This method can effectively separate long-period stable components as well. Additionally, it utilizes a noise diffusion model to control the information bottleneck from an external perspective. Through this approach, the proposed model allows for controlling the data restoration performance, which varies for each dataset, from the outside without iterative exploration processes.", "questions": "- Using a box plot to show the results in Figure 4(b) is not appropriate. Each hyperparameter decreases to one-fifth of the standard value and can increase up to twice the standard value, varying according to the values determined by the authors. It would be better to utilize Figure 2 in Appendix F instead.\n\n- In Table 1, what do the terms 'Series Number' and 'Attacks Number' refer to?\n", "limitations": "- The authors mentioned in Limitations section that the proposed model has a large computational cost (Appendix line 114). It would be beneficial if they provide specific information on the amount of GPU-hours required for the experiments.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors clearly defined the problem they wanted to solve in time series anomaly detection and demonstrated instances of the phenomenon occurring in real data. The proposed method is experimentally well-supported in its ability to solve the given problem.\n\n- The proposed data-time mix-attention and the dynamic decomposition method are ingenious.\n", "weaknesses": "- The proposed method sets the ground truth for the trend as a moving average (line 106), which seems that it may still not be free from the local window issue (Challenge 1, line 33) in long-term component decomposition.\n\n- In the authors' comparative experiments (Table 2), all comparison models used default hyperparameters (Appendix C.2.), while the proposed model explored combinations that yielded the best performance for each dataset (Appendix C.3.). Although the proposed method appears to be robust to hyperparameter variations, such comparisons may not be fair. It would be more appropriate to apply optimal hyperparameters to the comparison models or apply a single hyperparameter for all data in the proposed model, and then compare the results.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments and insightful suggestions. Please find our response below.\n\n**Q1: Details of dynamic decomposition.**\n\nAs two reviewers have question about why our method breaks the limitations of the local window, even though we also used the moving average to obtain the trend component, we provide a consistent response in **Q2** of the \"global\" response. Thanks.\n\n**Q2: Misunderstanding of hyperparameter setting.**\n\nApologies for the confusion in the Appendix C.3. Your concern pertains to the boundary of added drift in the disturbance strategy (line 56 in the Appendix) and the max offset of offset subtraction (line 57 in the Appendix). The original text actually describes the specific way in which the hyperparameters were set, i.e., the grid search, and its search range. In fact, we do use a single hyperparameter for all datasets, as evident in our submitted code (line 33, 34 in main.py). Specifically, the boundary of added drift in the disturbance strategy is set at 10, and the max offset of offset subtraction is set to 30.\n\n**Q3: Reasons for using box plots in Figure 4(b).**\n\nThanks for your suggestions. Figure 4(b) in the paper is designed to demonstrate the range of variation in the performance of the diffusion reconstruction module and the VAE module, using the same scaling range of hyperparameters. The purpose is to verify the robustness of our design. In comparison to the line plot (Figure 2 in the Appendix), which effectively demonstrates the trend of changes, the box plot is better suited for intuitively illustrating the distribution of the data.\n\n**Q4: Explanations of terms in Table 1.**\n\n\"Series Number\" refers to the number of variables ($k$ in line 85 in the paper) in the multivariate time series. \"Attacks Number\" refers to the number of anomaly segments present in the test set. In the real world, anomalies are usually continuous and appear in the form of segments. For instance, in the PSM dataset, the test set consists of data $\\mathbf{X} \\in \\mathbb{R}^{87481 \\times 25}$, where 87481 represents the length of timestamps, and 25 represents the number of variables. Among the 87481 timestamps, there are 73 anomalies. The anomalies vary in duration, with the shortest anomaly lasting for only 1 timestamp, and the longest anomaly spanning across 8861 timestamps.\n\n|      | Testing Size | Series Number | Attacks Number | Anomaly Durations |\n| :--: | :----------: | :-----------: | :------------: | :---------------: |\n| PSM  |    87481     |      25       |       73       |    1$\\sim$8861    |\n\n**Q5: Experiments on computational cost.**\n\nAs three reviewers have posed a similar question, we provide a consistent response in **Q1** of the \"global\" response. Thanks."}, {"review_id": "92Q15nj7V2", "reviewer": "Reviewer_JUP1", "summary": "The paper introduces D3R, a novel anomaly detection network for unstable multivariate time series data. D3R addresses the issue of drift by dynamically decomposing the data and reconstructing it using noise diffusion. Experimental results show that D3R outperforms existing methods, with a 12% average relative improvement. The proposed approach has implications for other tasks involving long-period multivariate time series analysis and can be applied to anomaly detection in different types of data. Overall, the paper presents an innovative solution to improve anomaly detection in unstable time series data.", "questions": "NA", "limitations": "The time complexity is not discussed. As real-world time series datasets can be large-scale and complex, it is important to address scalability and computational complexity.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tNovel Approach: The paper proposes a novel anomaly detection network, D3R, specifically designed for unstable multivariate time series data. Incorporating dynamic decomposition, noise diffusion, and end-to-end training sets it apart from existing methods.\n2.\tAddressing Drift: The paper addresses the drift generated from non-stationary environments, often overlooked in previous works. D3R aims to reduce false alarms and improve anomaly detection accuracy by tackling drift through decomposition and reconstruction.\n3.\tExperimental Results: The paper presents extensive experiments on real-world datasets, demonstrating that D3R outperforms existing methods with a notable 12% average relative improvement. This provides strong empirical evidence for the effectiveness of the proposed approach.\n", "weaknesses": "The time complexity is not discussed. As real-world time series datasets can be large-scale and complex, it is important to address scalability and computational complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments and insightful suggestions. Please find our response below.\n\n**Q1: Experiments on computational cost.**\n\nAs three reviewers have posed a similar question, we provide a consistent response in **Q1** of the \"global\" response. Thanks."}]

Author rebuttal:
**Q1: Experiments on computational cost.**

As real-world time series datasets can be large-scale and complex, we supplement the measures of training time, inference time, and model size for deep learning-based models on the SMD dataset. The experimental results are presented in the subsequent table.

Statistically, the training time for these models remains below 10 minutes, which is acceptable for real-world deployment and maintenance. The utilization of attention and its variants in the backbone network of our model leads to its larger size compared to previous algorithms. Thanks to the highly parallelized nature of the attention mechanism, both the training time and inference time of our model remain competitive. Additionally, it is worth mentioning that there has been substantial recent work [1] on transformer linearization, which may aid in reducing the burden of our model. We plan to explore this aspect in our future research.

|       Method        | Training Time (s) | Inference Time (s) | Model Size (MB) |
| :-----------------: | :---------------: | :----------------: | :-------------: |
|         VAE         |      157.91       |       30.90        |      0.02       |
|      DeepSVDD       |     **60.70**     |     **12.61**      |    **0.01**     |
|       LSTM-AE       |      283.61       |       72.82        |      0.04       |
|      MTAD-GAT       |      188.52       |       60.23        |      1.20       |
| Anomaly Transformer |      422.43       |       94.36        |      28.15      |
|        TFAD         |      315.39       |       38.54        |      1.04       |
|        Ours         |      399.32       |       104.12       |     109.35      |

**Q2: Details of dynamic decomposition.**

The moving average method is utilized **solely in the data preprocessing stage during training** to extract the trend and stable components. At this stage, the training data has not undergone slicing based on sliding windows, thus avoiding any limitations of local sliding window. The specific decomposition algorithm employed at this stage is not a concern, as our primary objective is to generate a labeled stable component for model training. We can also use other more advanced decomposition algorithms, such as STL [2] or HP filter [3], to obtain more precise label.

With a labeled stable component, our model can start training. Essentially, the core of the dynamic decomposition module is **learning a mapping function from timestamps to the stable component**. During training, the model learns this mapping function iteratively. During testing, it can **directly map the current input series to its stable component with this function**. It is worth noting that we no longer require the labeled stable component during testing. With the help of mapping rather than moving averages within localized windows, our model allows for precise decomposition of time series within smaller sliding window (or even single point).

**References:**

[1] Haixu Wu, et al. “Flowformer: Linearizing transformers with conservation flows.” In *ICML*, 2022.

[2] Cleveland Robert, et al. “STL: A seasonal-trend decomposition procedure based on loess.” In *Journal of official statistics*, 1990.

[3] Robert J Hodrick, et al. “Postwar US business cycles: an empirical investigation.” In *Journal of Money, credit, and Banking*, 1997.

Meta-review: The paper proposes the Decomposition with Diffusion Reconstruction (D3R) for anomaly detection in multivariate time series data. They decompose the time series into the trend and stable components, to capture long periods. They then use a reconstruction component involving a diffusion model based on stacked spatio-temporal transformer blocks. Experiments show convincing improvements over a good range of baselines on real world anomaly datasets. 

There reviews contain 5 positive reviews (77766) and 1 negative review (3). On the positive side, reviewers noted that the method is well-motivated, novel, well-written, and shows good performance. On the negative side, the reviewer raises a number of weaknesses, including motivation (the challenge / significance of long periods, trends, and dynamically modifying the information bottleneck being debatable) and comparison to previous research (both in terms of experiments and discussion). In the authors' response, they give some specific examples to justify the motivation, and provide additional results showing computation time (comparable to other transformer-based approaches), and provide an additional baseline, TFAD. Overall, in my view the weaknesses have been reasonably addressed, and considering the strengths of the paper and near consensus between reviewers / AC, I recommend accepting the paper.

Authors are requested to note the suggested improvements / action items arising from discussion with the reviewers, such as adding the TFAD discussion / analyses and discussion of long periods to the final manuscript, and other improvements (e.g. minor writing fixes) arising from reviewer discussion.
Decision: Accept (poster)