Title: Drift doesn't Matter: Dynamic Decomposition with Diffusion Reconstruction for Unstable Multivariate Time Series Anomaly Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: Anomaly Detection

Time Series

Diffusion

Transformer
Abstract: Many unsupervised methods have recently been proposed for multivariate time series anomaly detection. However, existing works mainly focus on stable data yet often omit the drift generated from non-stationary environments, which may lead to numerous false alarms. We propose **D**ynamic **D**ecomposition with **D**iffusion **R**econstruction (D$^3$R), a novel anomaly detection network for real-world unstable data to fill the gap. D$^3$R tackles the drift via decomposition and reconstruction. In the decomposition procedure, we utilize data-time mix-attention to dynamically decompose long-period multivariate time series, overcoming the limitation of the local sliding window. The information bottleneck is critical yet difficult to determine in the reconstruction procedure. To avoid retraining once the bottleneck changes, we control it externally by noise diffusion and directly reconstruct the polluted data. The whole model can be trained end-to-end. Extensive experiments on various real-world datasets demonstrate that D$^3$R significantly outperforms existing methods, with a 11% average relative improvement over the previous SOTA models.

Drift doesn't Matter: Dynamic Decomposition with Diffusion Reconstruction for Unstable Multivariate Time Series Anomaly Detection

Abstract

Many unsupervised methods have recently been proposed for multivariate time series anomaly detection. However, existing works mainly focus on stable data yet often omit the drift generated from non-stationary environments, which may lead to numerous false alarms. We propose D ynamic D ecomposition with D iffusion R econstruction (D 3 R), a novel anomaly detection network for real-world unstable data to fill the gap. D 3 R tackles the drift via decomposition and reconstruction. In the decomposition procedure, we utilize data-time mix-attention to dynamically decompose long-period multivariate time series, overcoming the limitation of the local sliding window. The information bottleneck is critical yet difficult to determine in the reconstruction procedure. To avoid retraining once the bottleneck changes, we control it externally by noise diffusion and directly reconstruct the polluted data. The whole model can be trained end-to-end. Extensive experiments on various real-world datasets demonstrate that D 3 R significantly outperforms existing methods, with a 11% average relative improvement over the previous SOTA models. Code is available at https://github.com/ForestsKing/D3R.

1 Introduction

Due to the rarity of anomalies, unsupervised anomaly detection of multivariate time series is an essential area in data mining and industrial applications. Reconstruction-based models are commonly used in unsupervised anomaly detection. The reconstruction error is small for normal series while large for abnormal series. Based on this principle, most anomalies can be detected without the label. In the real world, the temporal patterns typically change over time as they are generated from non-stationary environments. For example, the growth in the popularity of a service would cause customer metrics (e.g., request count) to drift upwards over time. Ignoring these factors would cause a deterioration in the performance of the anomaly detector. Despite the enormous advancements of previous research, most focus on stable data. Figure 1 illustrates how the anomaly scores provided by existing approaches tend to increase in the red area when the data is unstable, leading to false alarms.

To handle the unstable data, we attempt to decompose it into a stable component and a trend component, focusing more on the stable part in the reconstruction procedure. However, the trend part cannot be ignored completely, as the drift may be anomalous. There are still two challenges in this

-2

3

8

Value

(a) SMD

Data

Trend

-1

4

9

Value

(b) SWaT

Data

Trend

0

5000

10000

15000

20000

25000

Time

-1

10

21

32

Score

LstmAE

MTAD-GAT

D

3

R

0

500

1000

1500

2000

2500

Time

-1

4

9

14

Score

LstmAE

MTAD-GAT

D

3

R

context: Challenge 1 : Limitation of the decomposition for long-period time series. Most classical decomposition algorithms [2, 17, 27] are static, making it hard to be applied in the real world, where the data is updated in real time. With deep learning, some dynamic algorithms have been proposed; however, they are mainly based on averaging [29] or Fourier Transform [28, 26] within a local sliding window. Fundamentally, they can not apply to the data whose period is larger than the size of the sliding window. Challenge 2 : High training cost of adjusting the information bottleneck in the reconstruction procedure. The information bottleneck is critical for reconstruction-based models, and adjusting it is difficult. If it is too small, normal data will be poorly reconstructed; if it is too large, abnormal data will be successfully reconstructed. Previous methods most rely on an internal information bottleneck, such as the latent space size. As the information bottleneck is an attribute of the model itself, any change to the bottleneck necessitates retraining the model.

After analyzing numerous data series, we systematically organize the distribution shift into two categories. Vertical drift refers to statistical characteristics such as mean changing over time, while horizontal drift refers to values shifting left or right at similar times in various periods. In response to the above challenges, we propose D ynamic D ecomposition with D iffusion R econstruction (D 3 R) for long-period unstable multivariate time series anomaly detection. To solve Challenge 1, we utilize timestamps as external information to overcome the limitations of the local sliding window. Specifically, the data-time mix-attention and offset subtraction are utilized to solve vertical and horizontal drift, respectively. Moreover, we introduced a disturbance strategy in training to increase the robustness of the model. To solve Challenge 2, we propose a new method named noise diffusion to control the information bottleneck externally. The diffusion [6] provides a new view of the information bottleneck, treating the noise as a bottleneck and the unpolluted information as a condition. Because the bottleneck is no longer an attribute of the model itself, the different sizes can be set during revision without retraining the model.

The contribution of our paper is summarised as follows:

Anovel dynamic decomposition method for long-period multivariate time series is proposed. It effectively utilizes external information to overcome the limitations of the local sliding window.

A new approach to control information bottleneck externally by noise diffusion is also proposed. It avoids the high training cost of adjusting the information bottleneck.

Based on the findings for the unstable data when taking unsupervised anomaly detection, we propose a novel anomaly detection network called D 3 R. The D 3 R achieves the new SOTA results on various real-world datasets and significantly outperforms baselines on unstable datasets.

2 Related work

As a significant real-world problem, unsupervised anomaly detection of multivariate time series has received much attention. According to the criterion for anomaly detection, the paradigms roughly fall

𝐒 "

Reconstruction

Decomposition Blocks

Preprocess & Disturb

Data Encoder

Diffusion

Offset Subtraction

𝐗 $

!

𝐗

𝐗

!

𝐓 $

!

𝐗

!

𝐗 $

𝐗′

!

Time Encoder

𝐗

"#$%

𝐗

"#$%

Add & Norm

Add & Norm

Sub

Feed Forward

Next block

Out

S "

𝐇

!&"&

'()

N×

OrdAttention

MixAttention

𝐒

𝐇

"#$%

𝐇

!&"&

Embed

Embed

Embed

+

Out

Cat & Norm

+

+

+

-

Temporal

Spatial

𝐓 $

!

𝐗′

!

𝐗

"#$%

𝐗 $

!

N×

Norm

Feed Forward

+

Next block

𝐇

"#$%

'()

𝐇

"#$%

'

𝐇

!&"&

'

into the categories of probability-based, linear transformation-based, proximity-based, outlier-based, and neural network-based approaches.

As for the probability-based approach, [12, 13] are modeled by a statistical probability function on a multivariate cumulative distribution, providing anomaly scores based on the probability of occurrence of the sample. In the linear transformation-based method, [20, 22] begin by mapping the multivariate data before determining the boundary from the mapping space. Proximity-based algorithms, such as [18, 5], seek to cluster data based on similarity and then calculate intra-cluster and inter-cluster distances. In the outlier-based approach, [14, 16] compare the outliers degree of the testing samples with the training samples to identify the anomaly.

Most of the approaches mentioned above do not consider the temporal continuity of series. In recent years, neural network-based methods have become increasingly significant. Current neural network-based approaches can be divided into prediction-based and reconstruction-based. Although prediction-based methods [21, 8] are effective in modeling for the next timestamp, they are susceptible to interference from historical information. Reconstruction-based approaches [32, 11] do a great job capturing the distribution across the whole series. However, they are sensitive to the size of the information bottleneck.

Furthermore, existing works mainly focus on stable data, and their performance may suffer significantly in the real world, where the drift is frequent.

3 Method

An input of multivariate time series anomaly detection is denoted by X ∈ R n × k , where n is the length of timestamps, and k is the number of variables. The task is to produce an output vector y ∈ R n , where y i ∈ { 0 , 1 } denotes whether the i th timestamp is an anomaly.

3.1 Overview

The overall architecture of D 3 R is shown in Figure 2. The dynamic decomposition module first models the data and timestamp features by the data encoder and time encoder. Next, it uses stacked decomposition blocks to extract the stable component. Finally, we get the trend component by offset subtraction. The diffusion reconstruction module utilizes noise diffusion to construct information bottleneck externally and then directly reconstructs the polluted data by the backbone network. Reconstruction error is the anomaly score. In order to model both temporal dependency and dimension dependency, the data encoder and the reconstruction backbone network both consist of stacked spatial-

temporal transformer blocks. To increase the robustness of the model, we also propose a disturbance strategy during training.

3.2 Data preprocessing

We perform timestamp hard embedding, labeled stable component construction, and disturbance strategy for the input. Timestamp hard embedding is applied to the training and testing sets, while labeled stable component construction and disturbance strategy are applied to the training set only.

Timestamp hard embedding To make better use of timestamps in anomaly detection, we hardcode the timestamps of R n × 1 into an embedding X time ∈ R n × 5 like [34, 29], with each dimension representing minute of the hour, hour of the day, day of the week, day of the month, and month of the year.

Labeled stable component construction We extract the trend T ∈ R n × k by moving average, then the labeled stable component S = X -T , S ∈ R n × k is obtained. The construction of the labeled stable components prevents our model from being disturbed when the training data is unstable.

Disturbance strategy To increase the robustness of the model, we add a vertical drift d ∈ R k sampled from a [ -p, p ] uniform distribution to each variable of the training data. The final input of dynamic decomposition module is X d = X + d , X d ∈ R n × k .

3.3 Dynamic decomposition

The dynamic decomposition module consists of a data encoder, a time encoder, stacked decomposition blocks, and an offset subtraction. The data encoder is implemented based on the spatial-temporal transformer block, which captures the temporal and dimension dependency. The output of data encoder is H data ∈ R n × d model , where d model is hidden state dimension in the model. The time encoder consists only of the temporal transformer block, which models the temporal correlation of timestamps to obtain H time ∈ R n × d model . Data-time mix-attention constitutes the subject of stacked decomposition blocks, which are used to extract the stable component ˆ S ∈ R n × k . Finally, to solve the challenges of horizontal drift, we obtain the trend component ˆ T d ∈ R n × k through offset subtraction.

Spatial-temporal transformer block The architecture of the spatial-temporal transformer block is shown in the solid line box on the left panel of Figure 2. Assuming the input of l th layer is H l ∈ R n × d model with n timestamps and d model dimensions. In the temporal transformer, we obtain temporal relationship H l temporal ∈ R n × d model by directly applying multi-head self-attention (MSA) to n vectors of size d model:

where ˆ H l temporal is intermediate variable. LayerNorm( · ) denotes layer normalization as widely adopted in [25, 34, 29], FeedForward( · ) denotes a multi-layer feedforward network, MSA( Q , K , V ) denotes the multi-head self-attention [25] layer where Q , K , V serve as queries, keys and values. In the spatial transformer, we obtain dimension relationship H l spatial ∈ R n × d model by applying MSA to d model vectors of size n :

where ˆ H l spatial is intermediate variable. ( · ) T denotes the transposition of matrices. Finally, we obtain the input of l +1 th layer H l +1 ∈ R n × d model by:

where ˆ H l +1 ∈ R n × 2 d model is intermediate variable, and ⊕ represents concatenation.

Data-time mix-attention The original self-attention A o ( Q , K , V ) [25] only models the data information and ignores the role of timestamps. We define the data-time mix-attention as:

where Softmax( · ) is conducted row by row like [25, 34, 29], Q d , K d ∈ R n × d k are length-n data queries, data keys of d k dimension mapped from H data, respectively. Q t , K t , V t ∈ R n × d k are time queries, time keys and time values of d k dimension mapped from H time, respectively. D 3 R learns to autonomously merge data and time information by mapping them into attention space.

Decomposition block The structure of stacked decomposition blocks is shown on the right side of Figure 2. Assuming the input of l th layer is H l data and H l time , we can obtain the sub-stable components ˆ S l ∈ R n × k by:

where ˜ H l , ˆ H l , ˜ S l ∈ R n × d model are intermediate variables. Out( · ) is a single linear layer. Then we can obtain the input of l +1 th layer H l +1 data = H l data -˜ S l and H l +1 time = H l time . Finally, the sub-stable components ˆ S l of all stacked decomposition blocks are summed to obtain the stable component ˆ S of the series.

Offset subtraction Although the same time slots in different periods usually have similar fluctuations, it is not strictly one-to-one correspondence. Here, we employ a simple method to handle the horizontal drift, considering the vector a minus the vector b , with a maximum horizontal offset d :

where a i and b i are the i th element in a and b , respectively. Min( · ) means taking the minimum value. Based on the offset subtraction, we obtain the predicted trend component as ˆ T d = M ( X d , ˆ S ) .

3.4 Diffusion reconstruction

The diffusion reconstruction module mainly consists of a noise diffusion and a reconstruction backbone network. The noise diffusion is used to construct an information bottleneck externally by polluting the input data with noise, and the backbone network is used to reconstruct the polluted data directly.

Noise diffusion Given the original data x 0 , the polluted data x 1 , x 2 , · · · , x T are obtained by adding T step Gaussian noise. Each moment t in the noise addition process is only relevant to the t -1 moment. The hyperparameter β controls the ratio of added noise. We add trend retaining to the DDPM so that the model can concentrate on the more crucial stable components. The data x t after t steps of noise pollution is:

where ¯ α t = ∏ t i =1 α i , α i = 1 -β i , ¯ z t ∼ N (0 , 1) is the noise, and b is the retained information. The proof is given in Appendix A. So we can obtain the noisy data:

where X ′ d ∈ R n × k , ¯ Z ∼ N (0 , I ) , and ¯ Z ∈ R n × k .

Backbone network As shown in the left panel of Figure 2, the backbone network consists of stacked spatial-temporal transformer blocks. The extracted trend ˆ T d, the noisy data X ′ d , and the timestamp X time are encoded by the embedding layer first. To make the network more focused on the stable part without completely ignoring the trend part, we make the hidden vector:

where H ∈ R n × d model , embed( · ) is the embedding network. Then, we model the temporal and dimension relationship by feeding H into the stacked spatial-temporal Transformer blocks. After that, we sum the outputs of the stacked blocks with ˆ T d and send the result into the output layer. Finally, we directly obtain the reconstructed result ˆ X d of X d instead of the predicted noise.

3.5 Joint optimization

As described in the previous section, the dynamic decomposition and diffusion reconstruction are interconnected. The dynamic decomposition module learns the inherent features of long-period unstable multivariate time series, i.e., the stable component. We use the Mean Square Error (MSE) between S and ˆ S as the loss of this module. The diffusion reconstruction module directly reconstructs the data polluted by noise diffusion. It is worth noting that the output of this module is the reconstruction of X d. Because the drift d is added inside D 3 R, and the external who calculates the loss is only aware of X , we obtain the reconstruction of X by ˆ X = ˆ X d -d , ˆ X ∈ R n × k . Similar to the dynamic decomposition module, the MSE of X and ˆ X is used directly as the loss of this module.

During the training process, D 3 R needs to be trained end-to-end, so the loss function is defined as the sum of the two optimization objectives:

where S i,j , ˆ S i,j , X i,j , ˆ X i,j represent the labeled stable component, the predicted stable component, the original input and the reconstructed output of the j th variable at the i th timestamp, respectively.

3.6 Model inference

The online detection (inference) does not need labeled stable component construction and disturbance strategy in the data preprocessing stage. The anomaly score for the current timestamp consists only of the MSE of the original input X and the reconstructed output. Then, we run the SPOT [23], a streaming algorithm based on extreme value theory, to label each timestamp.

4 Experiments

4.1 Experimental settings

Datasets We evaluate D 3 R extensively on three real-world datasets: PSM (Pooled Server Metrics) [1], SMD (Server Machine Dataset) [24], and SWaT (Secure Water Treatment) [15]. For each dataset, we only preserve continuous variables. That is why we do not employ MSL (Mars Science Laboratory rover) and SMAP (Soil Moisture Active Passive satellite) [8], which are discrete except for the first dimension. The normal data is divided into training data (80%) and validation data (20%). Anomaly is only present in the testing data. More descriptions of the datasets are shown in Table 1. Further details about the datasets are available in Appendix B.1.

Baselines We extensively compare our model with 15 baselines, including the probability-based methods: Sampling, COPOD [12], ECOD [13]; the linear transformation-based methods: OCSVM [20], PCA [22]; the proximity-based methods: kNN [18], CBLOF [5], HBOS [4]; the outlier-based methods: IForest [14], LODA [16]; the neural network-based methods: VAE [11], DeepSVDD [19], Lstm-AE [9], MTAD-GAT [33], TFAD [31], Anomaly Transformer [30]. Additionally, we set up an adversary algorithm, which is implemented simply. A timestamp will be marked as abnormal if its sequential id is divisible by n , else it will be marked as normal. In our experiment, n is set as 40. This adversary algorithm works as a timed detector and offers no information about the location of the anomaly. All baselines are based on our runs, using the identical hardware. We employ official or open-source implementations published in GitHub and follow the configurations recommended in their papers. Further details concerning the Baselines are available in Appendix B.2.

Metrics For performance evaluation, we use precision, recall, and F1 score. The classical metrics are reasonable for tasks with sample granularity but are not applicable for continuous time series, where anomalies are frequently continuous. Most previous work [30, 33, 3, 21] employ the point adjustment method: If a point in a contiguous anomalous segment is detected correctly, all anomalies in the same segment are also considered to have been correctly detected. However, point adjustment is unreasonable, as [10] pointed out, and creates the illusion of progress. Assuming that ground truth and the predicted event are shown in Figure 3, although the real predicted event is just a timed detection, it still achieves an F1 score of 0.91 after point adjustment. The point adjustment may increase TP and decrease FN dramatically [10]. We test all baselines on all datasets with point adjustment

0

12

24

36

48

60

Time

-3

2

7

12

17

Value

Value

Adjusted

Real

metrics. The detailed experiment results can be found in Appendix C. Experimental results show that the average F1 score of the adversary algorithm surprisingly outperforms all baselines, even though it can provide no anomaly location information. This phenomenon arises because point adjustment unfairly assigns higher weights to long anomaly events. Furthermore, this algorithm does not consider the adjacency of time series [7]. To address the aforementioned challenges, we employ an F1 score based on affiliation [7]. This score takes into account the average directed distance between predicted anomaly events and ground truth events to calculate precision, as well as the average directed distance between ground truth events and predicted anomaly events to determine recall.

0

4000

8000

12000

16000

20000

-1

1

3

5

SMD

(a)

0

400

800

1200

1600

2000

Time

-1

2

5

8

SWaT

Data

Stable

z

d

model

T

Parameter

0.75

0.80

0.85

0.90

(b)

Detailed implementation of D 3 R can be found in Appendix B.3. All experiments are repeated 5 times, and we report the average for each metric.

4.2 Detection Results

On all three real-world datasets, as shown in Table 2, D 3 R outperforms the adversary algorithm and achieves the best F1 performance, confirming its effectiveness and superiority. Specifically, we achieve 0.61% (0.7548 → 0.7609), 11.11% (0.7571 → 0.8682), and 5.68% (0.7244 → 0.7812) absolute improvement over the previous SOTA methods on PSM, SMD, and SWaT datasets, respectively.

The statistical and machine learning methods frequently perform poorly in generalization because they do not account for time series continuity. They may perform well on a partial dataset while performing poorly or even failing on another (e.g., OCSVM and CBLOF). In opposition, neural network-based methods usually perform more balanced over various datasets. Furthermore, much of the previous works were evaluated by point adjustment, leading to a false boom. The metrics based on affiliation provide a more objective and reasonable evaluation of various methods, although their scores will drop.

Noticeably, D 3 R significantly outperforms the other methods on the SMD and SWaT datasets, which are typically characterized by high nonstationarity, suggesting the limitation of previous work on unstable real-world data. In our model, the dynamic decomposition and diffusion reconstruction modules complement each other. The former dynamically eliminates the unstable interference from the original data. The latter models the series from more crucial components. These designs tackle the shortcomings of previous work and maintain excellent robustness in complex real-world data.

In addition to the affiliation evaluation metric, we compare D 3 R with other baselines in all datasets using the AUC score. The detailed experiment results can be found in Appendix D. For a more intuitive comparison, we visualize the anomaly scores of the D 3 R and partial baselines, as shown in Appendix E. The quantitative and qualitative experimental results all show that D 3 R outperforms other baselines. Moreover, to validate the practicality of D 3 R in the actual production environment, we compare run time across different neural network-based algorithms in the SMD and present a summary in Appendix F. Both the training time and inference time of our model are acceptable.

4.3 Effectiveness verifications

Dynamic decomposition module We propose a dynamic decomposition module to tackle Challenge 1. To more intuitively verify the performance of the dynamic decomposition module, we provide visualization of the decomposition results on highly unstable datasets (SMD and SWaT). Figure 4(a) shows that our model robustly extracts the fundamental stable components, although real-world multivariate data is long-period and complex.

Diffusion reconstruction module We propose a diffusion reconstruction module to tackle Challenge 2. To verify the superiority of controlling information bottleneck externally, we replace the entire diffusion reconstruction module with the VAE module and perform sensitivity analyses for the VAE module and diffusion reconstruction module. The hyperparameter that significantly af-

fects the performance of the VAE module is the latent space size z . The hyperparameters that may affect the performance of the diffusion reconstruction module are the hidden state dimension d model, the noise ratio β , and the pollution step T . We scale each hyperparameter by a factor of { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 , 1 . 2 , 1 . 4 , 1 . 6 , 1 . 8 , 2 . 0 } , the statistic results for the SMD are shown in Figure 4(b). As well as allowing adjusting the information bottleneck without retraining, the diffusion reconstruction module is significantly more effective (i.e., the F1 score is higher) and more robust (i.e., the F1 score is more concentrated) than the V AE module. During the hyperparameter changing, the F1 score span of the V AE module is 8.96% (0.7496 → 0.8392), while the maximum span of the diffusion reconstruction module ( β ) is 3.30% (0.8381 → 0.8711).

4.4 Ablation studies

In order to verify the effectiveness and necessity of our designs, we perform ablation studies on both dynamic decomposition and diffusion reconstruction modules of D 3 R, respectively.

Dynamic decomposition module In this module, our main designs are spatial-temporal transformer, data-time mix-attention, offset subtraction, and disturbance strategy. Their ablation results are shown in Table 3. The w/o temporal, w/o spatial, w/o time, w/o data, w/o offset, and w/o disturbance represent the variants of D 3 R removing temporal transformer, spatial transformer, time attention, data attention, offset subtraction, and disturbance strategy, respectively. The temporal transformer and spatial transformer bring an average absolute improvement of 5.41% (0.7493 → 0.8034) and 3.61% (0.7673 → 0.8034), respectively. Compared with the spatial transformer, the temporal transformer is more critical. The spatial transformer performs much worse when the datasets have smaller series number (PSM and SWaT). When the series number is large (SMD), the dimension dependency is more prosperous, and the spatial transformer performs better. Moreover, time attention and data attention in data-time mix-attention bring a great absolute promotion of 6.87% (0.7347 → 0.8034) and 6.31% (0.7403 → 0.8034). While data attention is better suited to short-period data (SWaT), time attention is better suited to long-period data (PSM and SMD). Offset subtraction can further improve our model with 3.27% (0.7690 → 0.8034). Finally, the disturbance strategy brings a significant improvement of 7.85% (0.7249 → 0.8034), especially for the SMD and SWaT datasets with high nonstationary characteristics.

Diffusion reconstruction module This module uses noise diffusion to control the information bottleneck externally. To verify the superiority of noise diffusion, we replace the entire diffusion reconstruction module with the VAE module, and the ablation results are shown in Table 3 (w/o diffusion). As well as allowing adjustment of the information bottleneck without retraining, noise diffusion also provides a 3.57% (0.7677 → 0.8034) absolute improvement. The following section will discuss more advantages of the diffusion reconstruction module. An additional innovation of this module is trend retaining, and the ablation results are shown in Table 3 (w/o retaining). After removing trend retaining, the performance of D 3 R decreases significantly, especially in the highly nonstationary SMD dataset (0.8682 → 0.8006) and SWaT dataset (0.7812 → 0.7109). Trend retaining enables the model to focus on the critical stable components, avoiding being distracted by irrelevant information.

4.5 Hyperparameter analyses

The hyperparameter that significantly affects the performance of the VAE module is the latent space size z . The hyperparameters that may affect the performance of D 3 R are the added drift

0.75

0.79

0.83

0.87

z

0.80

0.84

0.88

0.92

p

0.80

0.84

0.88

0.92

d

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Factor

0.80

0.84

0.88

0.92

d

model

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Factor

0.80

0.84

0.88

0.92

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Factor

0.80

0.84

0.88

0.92

T

boundary p , the maximum horizontal offset d , the hidden state dimension d model, the noise ratio β , and the pollution step T . To analyze their influence on anomaly detection, we perform hyperparameter sensitivity analysis in the SMD. We scale each hyperparameter by a factor of { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 , 1 . 2 , 1 . 4 , 1 . 6 , 1 . 8 , 2 . 0 } , and the results are shown in Figure 5.

Anomaly detection methods based on reconstruction require constructing the information bottleneck to reconstruct normal data and inhibit anomaly effectively. While traditional autoencoders control the information bottleneck from inside through the latent space size, our approach controls the information bottleneck from outside through noise diffusion. As the experimental results show, with the bottleneck going from strict to loose (latent space size z , noise ratio β , pollution step T going from small to large), the performance of the model goes from up to down. If the information bottleneck is too tight, the normal data cannot be reconstructed, and the precision is low. If the information bottleneck is too loose, the anomaly can be reconstructed well, and the recall is low. On the one hand, our approach to control the information bottleneck externally is more robust (smaller range of variation). On the other hand, as the information bottleneck is not an attribute of the model itself, we can try different bottleneck sizes when inference without retraining the model again.

Since it is no longer an information bottleneck, D 3 R is not sensitive to the hidden state dimension d model. A larger hidden space tends to bring better results but also a greater computational cost. As for the added drift boundary p and maximum horizontal offset d , the impact on our model becomes small once they exceed a certain threshold. Overall, D 3 R is robust to all hyperparameters we test.

5 Conclusion

This paper proposes D ynamic D ecomposition with D iffusion R econstruction (D 3 R) for long-period unstable multivariate time series anomaly detection to cover the overlook in previous work. We first decompose the long-period unstable multivariate time series dynamically and then directly reconstruct the data polluted by noise diffusion. Extensive experiments prove that D 3 R significantly outperforms existing works. The method we proposed to break the limitation of the local sliding window is also meaningful for other long-period multivariate time series analysis tasks, such as prediction and imputation. Meanwhile, the approach to controlling information bottleneck externally can also be used for anomaly detection of other modal data, such as picture, video, and log data.

Acknowledgments and Disclosure of Funding

This work was supported in part by the National Natural Science Foundation of China under Grants (62101064, 62171057, 62201072, 62071067), in part by the Ministry of Education and China Mobile Joint Fund (MCM20200202, MCM20180101), in part by the Beijing University of Posts and Telecommunications-China Mobile Research Institute Joint Innovation Center.

References

Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2485-2494, 2021.

Robert B Cleveland, William S Cleveland, Jean E McRae, and Irma Terpenning. Stl: A seasonal-trend decomposition. Journal of Official Statistics , 6:3-73, 1990.

Ailin Deng and Bryan Hooi. Graph neural network-based anomaly detection in multivariate time series. In AAAI Conference on Artificial Intelligence , pages 4027-4035, 2021.

Markus Goldstein and Andreas Dengel. Histogram-based outlier score (hbos): A fast unsupervised anomaly detection algorithm. German Conference on Artificial Intelligence , 1:59-63, 2012.

Zengyou He, Xiaofei Xu, and Shengchun Deng. Discovering cluster-based local outliers. Pattern Recognition Letters , 24:1641-1650, 2003.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Neural Information Processing Systems , pages 6840-6851, 2020.

Alexis Huet, José Manuel Navarro, and Dario Rossi. Local evaluation of time series anomaly detection algorithms. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 635-645, 2022.

Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Söderström. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 387-395, 2018.

Tung Kieu, Bin Yang, and Christian S. Jensen. Outlier detection for multidimensional time series using deep neural networks. In IEEE International Conference on Mobile Data Management , pages 125-134, 2018.

Siwon Kim, Kukjin Choi, Hyun-Soo Choi, Byunghan Lee, and Sungroh Yoon. Towards a rigorous evaluation of time-series anomaly detection. In AAAI Conference on Artificial Intelligence , pages 7194-7201, 2022.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations , 2014.

Zheng Li, Yue Zhao, Nicola Botta, Cezar Ionescu, and Xiyang Hu. Copod: Copula-based outlier detection. In IEEE International Conference on Data Mining , pages 1118-1123, 2020.

Zheng Li, Yue Zhao, Xiyang Hu, Nicola Botta, Cezar Ionescu, and George H. Chen. ECOD: unsupervised outlier detection using empirical cumulative distribution functions. arXiv , 2201.00382, 2022.

Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In IEEE International Conference on Data Mining , pages 413-422, 2008.

Aditya P. Mathur and Nils Ole Tippenhauer. Swat: A water treatment testbed for research and training on ics security. In CySWater , pages 31-36, 2016.

Tomás Pevný. Loda: Lightweight on-line detector of anomalies. Machine Learning , 102: 275-304, 2016.

Jinwen Qiu, S. Rao Jammalamadaka, and Ning Ning. Multivariate bayesian structural time series model. Journal of Machine Learning Research , 19:68:1-68:33, 2018.

Sridhar Ramaswamy, Rajeev Rastogi, and Kyuseok Shim. Efficient algorithms for mining outliers from large data sets. In ACM SIGMOD International Conference on Management of Data , pages 427-438, 2000.

Lukas Ruff, Nico Görnitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Robert A. Vandermeulen, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International Conference on Machine Learning , pages 4393-4402, 2018.

Bernhard Schölkopf, John C. Platt, John Shawe-Taylor, Alexander J. Smola, and Robert C. Williamson. Estimating the support of a high-dimensional distribution. Neural Computation , 13:1443-1471, 2001.

Lifeng Shen, Zhuocong Li, and James T. Kwok. Timeseries anomaly detection using temporal hierarchical one-class network. In Neural Information Processing Systems , 2020.

Meiling Shyu, Shuching Chen, Kanoksri Sarinnapakorn, and Liwu Chang. A novel anomaly detection scheme based on principal component classifier. Technical report, Miami Univ Coral Gables Fl Dept of Electrical and Computer Engineering, 2003.

Alban Siffer, Pierre-Alain Fouque, Alexandre Termier, and Christine Largouët. Anomaly detection in streams with extreme value theory. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1067-1075, 2017.

Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2828-2837, 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems , pages 5998-6008, 2017.

Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. Learning latent seasonal-trend representations for time series forecasting. In Neural Information Processing Systems , 2022.

Qingsong Wen, Jingkun Gao, Xiaomin Song, Liang Sun, Huan Xu, and Shenghuo Zhu. Robuststl: A robust seasonal-trend decomposition algorithm for long time series. In AAAI Conference on Artificial Intelligence , pages 5409-5416, 2019.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In International Conference on Learning Representations , 2022.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Neural Information Processing Systems , pages 22419-22430, 2021.

Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. In International Conference on Learning Representations , 2022.

Chaoli Zhang, Tian Zhou, Qingsong Wen, and Liang Sun. TFAD: A decomposition time series anomaly detection architecture with time-frequency analysis. In ACM International Conference on Information & Knowledge Management , pages 2497-2507, 2022.

Weiqi Zhang, Chen Zhang, and Fugee Tsung. Grelen: Multivariate time series anomaly detection from the perspective of graph relational learning. In International Joint Conference on Artificial Intelligence , pages 2390-2397, 2022.

Hang Zhao, Yujing Wang, Juanyong Duan, Congrui Huang, Defu Cao, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. Multivariate time-series anomaly detection via graph attention network. In IEEE International Conference on Data Mining , pages 841-850, 2020.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI Conference on Artificial Intelligence , pages 11106-11115, 2021.

A Noise diffusion proofs

In this section, we explain the proof of Equation 7 in our paper.

Suppose z ∼ N (0 , 1) is the Gaussian noise to be added, β controls the amount of noise added each step, α = 1 -β , and b is the trend information to be retained. The x t after t steps of noise pollution is: √

The Gaussian distribution, as we know, has the following properties:

Suppose x ∼ N ( µ, σ 2 ) , then ax + b ∼ N ( aµ + b, a 2 σ 2 ) .

Suppose x ∼ N ( µ x , σ 2 x ) , y ∼ N ( µ y , σ 2 y ) , x, y are independent random variables, then x + y ∼ N ( µ x + µ y , σ 2 x + σ 2 y ) .

Based on these properties, the noise diffusion process can be deduced further:

Repeating the above process, we can directly infer x t from x 0 as follows:

where ¯ α t = ∏ t i =1 α i and ¯ z t ∼ N (0 , 1) .

B Detailed experimental settings

B.1 Datasets

The datasets can be downloaded from the following:

PSM: https://github.com/eBay/RANSynCoders/tree/main/data.

SMD: https://github.com/NetManAIOps/OmniAnomaly/tree/master/ServerMachineDataset.

SWaT: https://itrust.sutd.edu.sg/testbeds/secure-water-treatment-swat/.

B.2 Baselines

All baselines are based on our runs, using the identical hardware. We employ official or open-source implementations published in GitHub and follow the configurations recommended in their papers. The baselines can be downloaded from the following:

Sampling: https://github.com/yzhao062/pyod.

COPOD: https://github.com/yzhao062/pyod.

ECOD: https://github.com/yzhao062/pyod.

OCSVM: https://github.com/yzhao062/pyod.

PCA: https://github.com/yzhao062/pyod.

kNN: https://github.com/yzhao062/pyod.

CBLOF: https://github.com/yzhao062/pyod.

HBOS: https://github.com/yzhao062/pyod.

IForest: https://github.com/yzhao062/pyod.

LODA: https://github.com/yzhao062/pyod.

VAE: https://github.com/yzhao062/pyod.

DeepSVDD: https://github.com/yzhao062/pyod.

LSTM-AE: https://github.com/matanle51/LSTM_AutoEncoder.

MTAD-GAT: https://github.com/ML4ITS/mtad-gat-pytorch.

TFAD: https://github.com/DAMO-DI-ML/CIKM22-TFAD.

Anomaly Transformer: https://github.com/thuml/Anomaly-Transformer.

B.3 Implementation

In our experiment, the sliding window has a fixed size 64 for all datasets. We set the dimension of hidden states as 512, the dimension of attention as 64, the number of attention heads as 8, the number of network layers as 2, and the dropout as 0.6. We use grid search to obtain the best SPOT parameters for each dataset and record the results with the highest F1 scores. The β of the diffusion model changes from 0.0001 to 0.02 in 1000 steps like DDPM without adjustment. We add noise with 500 steps (selected from { 100 , 300 , 500 , 700 , 900 } ) as an external information bottleneck when the noise diffusion. The boundary of added drift in the disturbance strategy is 10 (selected from { 1 , 2 . 5 , 5 , 10 , 20 } ). The max offset of offset subtraction is 30 (selected from { 5 , 10 , 20 , 30 , 40 } ). We use the Adam optimizer with an initial learning rate of 10 -4 . The training process is early stopped within 8 epochs with a batch size of 8. The implementation of D 3 R is carried out using Python 3.9.13 and PyTorch 1.11.0. All experiments are performed on a Ubuntu Server with a 12th Gen Intel(R) Core(TM) i9-12900K @ 3.60GHz processor and an NVIDIA GeForce RTX 3090 Graphics Card.

C Detection results with point adjusted

Wetest all baselines on all datasets with point-adjusted evaluation metrics, and the detailed quantitative results are shown in Table 4. Since the data processing and hardware are not the same as those in the original paper, the results of the baselines in our experiments are slightly different from those in the original paper. Nevertheless, the errors are still within a reasonable range.

The point adjustment method is unreasonable. Despite the no helpful information provided by the adversary algorithm, its average performance still exceeds other SOTA methods by 2% of the relative F1 score. Specifically, the adversary algorithm achieves an absolute improvement of 2.14% (0.9431 → 0.9645) and 2.01% (0.8103 → 0.8304) on the PSM and SWaT datasets, respectively. Only on the SMD dataset the adversary algorithm is slightly weaker than the best baseline (0.9898 → 0.9191). If we further adjust the n in the adversary algorithm, it can also outperform all the baselines in the SMD dataset.

D Additional detection results

In order to avoid the influence of SPOT parameters and thresholds on the results, we compare the methods from the perspective of the original anomaly scores. If an anomaly lasts too long, more weight will be assigned, resulting in an inaccurate metric. Thus, we aggregate each anomaly event. An anomaly event after aggregation is equivalent to only one timestamp, labeled as an anomaly, and scored as the maximum of the original anomaly range.

We use Area Under Curve (AUC) to evaluate the anomaly scores of each method. The value of AUC ranges between 0.5 and 1, while the closer to 1.0, the better the method. The results of the experiments are shown in Table 5. D 3 Rsignificantly outperforms existing methods, with a 9% average relative improvement over other SOTA methods. Specifically, we achieve 1.30% (0.9093 → 0.9223), 0.61% (0.9698 → 0.9759), and 4.69% (0.8085 → 0.8554) absolute improvement over the best SOTA performance on PSM, SMD and SWaT datasets, respectively. This experiment proves the superiority of D 3 R once again.

E Anomaly score visualizations

We visualize the anomaly scores of the partial baseline and D 3 R for a more intuitive comparison, as shown in Figure 6. D 3 R consistently provided the most distinguishable anomaly scores in all three real-world datasets. In the PSM dataset, the vertical drift is insignificant. In this case, some of the baselines (MTAD-GAT) can achieve correct detection results, but there are still missed detections (Anomaly Transformer) and false detections (VAE). In the SMD and SWaT datasets, the vertical drift of the data is significant. The anomaly scores of baselines in the normal region are already far from the training data. The unstable normal data is likely to be mistakenly detected as an anomaly. Once the anomaly occurs, even if the anomaly scores of the baselines improve, they are not significant enough due to the large base.

F Run Times

We comprehensively compare training time, inference time, and model size for neural network-based models on the SMD dataset to validate the practicality of D 3 R in the production environment. The outcomes are presented in Table 6.

0.0

0.7

1.4

2.1

Data

PSM

Train

Test

0

5

10

15

SMD

Train

Test

0

2

4

6

SWaT

Train

Test

0

2

4

6

VAE

Train

Test

0

4

8

12

Train

Test

0

2

4

6

Train

Test

0

1

2

3

DeepSVDD

Train

Test

0

1

2

3

Train

Test

0

6

12

18

Train

Test

0.0

0.6

1.2

1.8

LstmAE

Train

Test

0.0

0.2

0.4

0.6

Train

Test

0

14

28

42

Train

Test

0.0

0.5

1.0

1.5

MTAD-GAT

Train

Test

0.0

0.2

0.4

0.6

Train

Test

0

15

30

45

Train

Test

0.0

0.5

1.0

1.5

Anomaly

Transformer

Train

Test

0

2

4

6

Train

Test

0

8

16

24

Train

Test

0

40

80

120

160

200

Time

0.0

1.8

3.6

5.4

D

3

R

Train

Test

0

20

40

60

80

100

Time

0.0

1.4

2.8

4.2

Train

Test

0

10

20

30

40

50

Time

0

3

6

9

Train

Test

Both the training set and the test set of SMD have approximately 16 days of data. Statistically, the training time for these models remains below 10 minutes, which is acceptable for real-world deployment and maintenance. The utilization of attention and its variants in the backbone network of our model leads to its larger size compared to previous algorithms. Thanks to the highly parallelized nature of the attention mechanism, both the training time and inference time of our model remain competitive. The inference process for 16 days of data necessitates a mere 104.12 seconds, satisfying the online, real-time detection criteria. Compared to the substantial enhancement (11%) in detection accuracy that we have achieved, the model size of 109.35MB remains affordable within the context of the expanding hardware resources of the present era. Additionally, there has been substantial recent work (such as Flowformer) on transformer linearization, which may aid in reducing the burden of our model. We plan to explore this aspect in our future research.

G Broader Impacts

Recently, Deep Learning has been increasingly used in anomaly detection. It improves safety and prevents potential risks and financial losses by detecting anomalies in healthcare, industrial manufacturing, and autonomous driving. However, since real-time data is constantly changing, data characteristics can change drastically as time accumulates. Due to the obsolescence of the training data, our models cannot adapt to this situation and may provide incorrect judgments. In the future, we consider designing algorithms that adopt online updates to address this shortcoming.

H Limitations

The computational cost of our model is large due to the Transformer architecture used for the backbone network. In future work, we plan to replace it with a more lightweight and efficient base structure, such as dilated convolutions and graph neural networks. Furthermore, due to the shortcomings of the point-adjustment evaluation metric, in order to prove the superiority of D 3 R, we have to compare various other metrics and provide rich visualizations. We plan to design a more intuitive and effective anomaly detection evaluation metric in future work.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces D3R, a novel anomaly detection network for unstable multivariate time series data. D3R addresses the issue of drift by dynamically decomposing the data and reconstructing it using noise diffusion. Experimental results show that D3R outperforms existing methods, with a 12% average relative improvement. The proposed approach has implications for other tasks involving long-period multivariate time series analysis and can be applied to anomaly detection in different types of data. Overall, the paper presents an innovative solution to improve anomaly detection in unstable time series data.\n\nSTRENGTHS:\n1.\tNovel Approach: The paper proposes a novel anomaly detection network, D3R, specifically designed for unstable multivariate time series data. Incorporating dynamic decomposition, noise diffusion, and end-to-end training sets it apart from existing methods.\n2.\tAddressing Drift: The paper addresses the drift generated from non-stationary environments, often overlooked in previous works. D3R aims to reduce false alarms and improve anomaly detection accuracy by tackling drift through decomposition and reconstruction.\n3.\tExperimental Results: The paper presents extensive experiments on real-world datasets, demonstrating that D3R outperforms existing methods with a notable 12% average relative improvement. This provides strong empirical evidence for the effectiveness of the proposed approach.\n\nWEAKNESSES:\nThe time complexity is not discussed. As real-world time series datasets can be large-scale and complex, it is important to address scalability and computational complexity.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nThe time complexity is not discussed. As real-world time series datasets can be large-scale and complex, it is important to address scalability and computational complexity.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new model called D3R (Decomposition with Diffusion Reconstruction) for anomaly detection in multivariate time series data. The proposed model applies a dynamic decomposition method to separate the stable components and trends in time series data. This method can effectively separate long-period stable components as well. Additionally, it utilizes a noise diffusion model to control the information bottleneck from an external perspective. Through this approach, the proposed model allows for controlling the data restoration performance, which varies for each dataset, from the outside without iterative exploration processes.\n\nSTRENGTHS:\n- The authors clearly defined the problem they wanted to solve in time series anomaly detection and demonstrated instances of the phenomenon occurring in real data. The proposed method is experimentally well-supported in its ability to solve the given problem.\n\n- The proposed data-time mix-attention and the dynamic decomposition method are ingenious.\n\nWEAKNESSES:\n- The proposed method sets the ground truth for the trend as a moving average (line 106), which seems that it may still not be free from the local window issue (Challenge 1, line 33) in long-term component decomposition.\n\n- In the authors' comparative experiments (Table 2), all comparison models used default hyperparameters (Appendix C.2.), while the proposed model explored combinations that yielded the best performance for each dataset (Appendix C.3.). Although the proposed method appears to be robust to hyperparameter variations, such comparisons may not be fair. It would be more appropriate to apply optimal hyperparameters to the comparison models or apply a single hyperparameter for all data in the proposed model, and then compare the results.\n\nQUESTIONS:\n- Using a box plot to show the results in Figure 4(b) is not appropriate. Each hyperparameter decreases to one-fifth of the standard value and can increase up to twice the standard value, varying according to the values determined by the authors. It would be better to utilize Figure 2 in Appendix F instead.\n\n- In Table 1, what do the terms 'Series Number' and 'Attacks Number' refer to?\n\nLIMITATIONS:\n- The authors mentioned in Limitations section that the proposed model has a large computational cost (Appendix line 114). It would be beneficial if they provide specific information on the amount of GPU-hours required for the experiments.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
To overcome the temporal drift issues in unstable time series data, this work proposes an anomaly detection method, $D^3R$. By considering the temporal continuity of series and relieving the constraints of information bottleneck, $D^3R$ realizes the dynamic decomposition and the noise-diffusion-based series reconstruction. Extensive experiments on real-world time series datasets demonstrate the superiority of $D^3R$ w.r.t. anomaly detection.

STRENGTHS:
1. This work is well motivated by tackling the overlooked issues in existing time series anomaly detection works. 
2. The proposed $D^3R$ approach employs decomposition and reconstruction for time series anomaly detection, which is technically sound. 
3. This draft is well organized, and the presentation is clear.

WEAKNESSES:
1. Time series anomaly detection is a well-studied problem. More datasets are expected to conduct experiments to validate the proposed method's effectiveness, such as the Yahoo dataset (Webscope dataset ydata-labeled-time-series-anomalies-v1 0, 2015).

QUESTIONS:
1. As shown in experiments, the performance improvement of $D^3R$ on PSM is less impressive than other datasets. You highlight that $D^3R$'s performance improvement is more significant on the time series data with high nonstationarity. Does this mean that the application of $D^3R$ might be limited to the highly unstable time series data?

LIMITATIONS:
Potential applicability issues should be elaborated more.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The paper presents a Transformer-based model called Dynamic Decomposition with Diffusion Reconstruction for Anomaly Detection in unstable multivariate time series. The authors addressed two challenges: the limitation of decomposition for long-period time series and high training cost for adjusting the information bottleneck in the reconstruction procedure. They introduced a novel dynamic decomposition method and a noise diffusion method to enable effective utilization of external information, overcome the limitations of local sliding windows, and avoid the high training cost associated with adjusting the information bottleneck. The proposed model achieves state-of-the-art results on different real-world datasets and outperforms baselines significantly on unstable datasets.

STRENGTHS:
1.  the authors presenting a clear exposition of their research motivation and proposed solution.
2. The authors' methods effectively tackle two crucial issues: limitations of decomposition for long-period time series and high training cost of adjusting the information bottleneck.
3. The authors conducted experiments on diverse datasets, and their proposed method demonstrated excellent performance.

WEAKNESSES:
1. The authors used the mean of five runs as the final results in their experiments but did not provide the standard deviation of the results.

2. While the model achieved the best F-score performance due to higher recall values, its precision values were comparatively lower than other models. The authors should explain this observation.

3. Further explanations for Figure 1 are recommended to clarify why other methods failed in this specific scenario.

QUESTIONS:
see weaknesses

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-16

comments: SUMMARY:
This paper tackles the problem that existing works omit the drift generated from non-stationary environments by focusing on stable data, which may lead to numerous false alarms. 
As a solution, they propose an end-to-end anomaly detection network for real-world unstable data, named Dynamic Decomposition with Diffusion Reconstruction (D3R). 
In the decomposition stage, they dynamically decompose long-period multivariate time series by utilizing data-time mix-attention to overcome the limitation of local sliding window. 
In the reconstruction stage, they control the information bottleneck extenerlly by noise diction and directly reconstruct the polluted data. 
They evaluate on three real-world datasets (PSN, SMD, SWaT), and achieve the best performance compared to existing unsupervised anomaly detection methods.

STRENGTHS:
Well-motivated and soundness: The problem that existing methods overlook the anomaly score in unstable data. Their proposed dynamic decomposition for long-period multivariate time series and diffusion reconstruction for controlling information bottleneck is sound.

Ablation study and Analysis: In sections 4.3 and 4.4, they analyze their proposed dynamic decomposition module and diffusion reconstruction module. This helps to understand the effect of the proposed modules.

Well-structured paper: Their paper is well-organized and easy to read.

WEAKNESSES:
Computational cost: there is no comparison of computational cost (inference time, # of params, Gflops ..) compared to existing models. For check the efficiency, they should report the comparison of computational cost.

Minor comments: 
Repeated citation [11] and [12] of references

QUESTIONS:
.

LIMITATIONS:
.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCurrent unsupervised methods for multivariate time series anomaly detection often overlook drift from non-stationary environments, leading to false alarms. To address this, this paper presents Dynamic Decomposition with Diffusion Reconstruction (D3R), a new anomaly detection network for unstable data. D3R decomposes and reconstructs drift, using data-time mix-attention for dynamic decomposition and noise diffusion to manage the information bottleneck in reconstruction. This end-to-end trainable model outperforms existing methods, showing a 12% average improvement over previous top-performing models.\n\nSTRENGTHS:\n1. The outcomes of the experiments seem to be promising.\n\n2. The problem addressed is intriguing and holds substantial value.\n\nWEAKNESSES:\n1. The challenge suggested by the authors is debatable. They assert that \"classical decomposition algorithms cannot be applied to data with a period larger than the size of the sliding window.\" In my view, this problem seems relatively simple to tackle by merely extending the length of the sliding window. Additionally, the authors seem to continue to depend on the moving average method to ascertain the trend and the labeled stable components. Therefore, they need to clarify how their technique specifically addresses this problem.\n\n2. The rationale for creating dynamic decomposition remains vague. The sole beneficial intermediate outcome yielded by dynamic decomposition is T_d, recognized as the predicted trend component. Nevertheless, it seems that there is a more direct and feasible approach to extract the trend component, much like the authors have done during the data preprocessing phase. The necessity of introducing this intricate trend component, T_d, in place of using the standard trend component, demands clarification.\n\n3. The concern of dynamically modifying the information bottleneck doesn't seem to be a common occurrence in real-life scenarios. I struggle to envision a situation where it would be necessary to adjust the information bottleneck during the inference phase. The authors need to elaborate on the practical value of employing the diffusion model if they aim to emphasize their contribution to reducing the cost of modifying the information bottleneck. Furthermore, offering a principle to guide the dynamic setting of these hyperparameters could make their method more persuasive.\n\n4. No experiments have been conducted to substantiate their method's superiority concerning reducing the high training cost associated with adjusting the information bottleneck.\n\n5. There is insufficient engagement with previous research. The ensuing paper also examines the impact of the trend component in anomaly detection, but the authors seemingly fail to appropriately cite this paper or compare their method with it.\n[1] Zhang, Chaoli, et al. \"TFAD: A decomposition time series anomaly detection architecture with time-frequency analysis.\" Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022.\n\nQUESTIONS:\nThe authors can refer to the weakness listed above.\n\nLIMITATIONS:\nThe authors do acknowledge some minor limitations in their work. However, the primary limitation that is yet to be addressed pertains to the types of anomaly patterns their model cannot detect.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-24"}