GenAD: General Representations of Multivariate Time Series for Anomaly Detection

Abstract: Anomaly Detection(AD) for multivariate time series is an active area in machine learning, with critical applications in Information Technology system management, Spacecraft Health monitoring, Multi-Robot Systems detection, etc.. However, due to complex correlations and various temporal patterns of large-scale multivariate time series, a general unsupervised anomaly detection model with higher F1-score and Timeliness remains a challenging task. In this paper, We propose a General representations of multivariate time series for Anomaly Detection(GenAD). First, we apply Time-Series Attention to represent the various temporal patterns of each time series. Second, we employ Multi-Correlation Attention to represent the complex correlations of multivariate time series. With the above innovations, GenAD improves F1-scores of AD by 0.3% to 5% over state-of-the-art model in public datasets, while detecting anomalies more rapidly in anomaly segments. Moreover, we propose a general pre-training algorithm on large-scale multivariate time series, which can be easily transferred to a specific AD tasks with only a few fine-tuning steps. Extensive experiments show that GenAD is able to outperform state-of-the-art model with only 10% of the training data. 

Reviews and author responses:

Dear reviewer,
We appreciate the time and effort you put into this paper, Your reviews are really helpful, and we revised the paper carefully.

1.	"In practice, the overall status of an entity is more concerned..." This sentence is really hard to understand without special background, and we have modified the expression in the revised version, which can be seen in line 7 of chapter 1. There are two kinds of anomaly detection directions, which are detecting anomalies at the entity-level with multivariate time series, and detecting anomalies at the metric-level with uni-variate time series. We are more concerned about whether the equipment is abnormal, and the equipment is composed of multivariate time series. There is less equipment with only one single series, so we focus on detecting anomalies at the entity-level with multivariate time series. 

2.	"Note that, with OCSVM, time series or sequence-oriented kernels can be used to enable it to be used with time series.". We have read the paper you have mentioned carefully, and OCSVM with the kernel can be used for time series. We have also revised the expression in chapter2 and cited the method. However, the OCSVM may perform better for short-term abnormalities, while the performance may be attenuated for long-term abnormalities.

3.	"In section 3, it says that GenAD sets the duration of the training portion of the time series to be four times the duration of the testing portion. How was this chosen? What different splits were tested?" As there is no network architecture in the first version, resulting in a difficult understanding of model architecture and model training methods. GenAD is a model based on self-supervised learning, and learns the complex correlations and various temporal patterns of multivariate time series in (0, T), then determines whether there are anomaly segments at certain time steps after T. Therefore, all multivariate time series in the time of (0, T) is the training data for the model, and GenAD represents various characteristics of time series by self-supervised learning. Meanwhile, the test data is the series after time T, which can be any length. However, in the absence of model architecture, the expression above is not intuitive. We have added section 3.2 to elaborate on the model's architecture separately, and the architecture is shown in Figure 1. According to Figure 1, (0, T) is divided into 5 parts (Ta, Tb, Tc, Td, Te) with the same length of each part, and the multivariate time series of Te (the last 20% of T) is used for Multi-Correlation Attention training, Ta~Tb (the first 80% of T) is used for Time-Series Attention training. 

4.	"The proposition is not stated in a rigorous enough way to be able to tell what is being proven in the proof that is below it.". Yes, the proposition is indeed not specific and it did not clearly express the content that needs to be proved. In the revised version, we do not mention the proposition, but first give the clear formula for Time-Series Attention (in Eq(2)), and for each temporal pattern, we give the detailed formula derivation for each capability based on the formula of Time-Series Attention (in Eq(3~8)).

5.	" Is a different 20% chosen with every time series or for a given problem? How do the results vary depending on the dimensions chosen?". This review is in-depth because it involves the design of the model architecture and the design details above are all not stated in the paper. First, we explain the purpose of the MASK. We have added the method and purpose of MASK in the first paragraph of section 3.2 (around Figure 1). By Mask method, the model will be forced to learn the complex Inter-dependencies and various Intra-dependencies after a sufficient number of training epochs. Then, we explain why we choose 20%. As when the Pearson Correlation of two time series is greater than 0.8, we believe that there is an extremely strong correlation between the two series, which means that one of the series can be reconstructed by the transformation of the other series. So we calculated the Pearson Correlation between 38 series in SMD datasets and found that for each series, there were 4.2 surrounding series with the Pearson Correlation greater than 0.8 on average. Therefore, choosing 20% here means to reconstruct 1 series with 4 series in each training epoch. Finally, we explain how the results vary if we choose other values. If the ratio of Mask selection is increased, Multi-Correlation Attention will learn less information with less input of the model, which lowers the speed of model training. If the ratio of Mask selection is reduced, for example, modified to 10%, then each time series can be learned for reconstruction after at least 10 times in model training. While for 20%, each series can be learned after 5 times model training. So reduction in the ratio will increase the number of training epochs. Of course, 20% is not fixed and can be changed according to the actual datasets.


Dear reviewer,
Thank you for the time and effort on the paper. Your reviews are really helpful, and we revised the paper carefully.

1. "The authors have to clarify their architecture and the computation that are made before proposing proofs".  Yes, the lack of architecture does make the paper more difficult to understand. And without the architecture, the elaboration of the model details in Section 3 lacks systematicity and logicality. More importantly, it is difficult for the readers to implement the model according to Section 3. Although considering the limitation of paper length in the first version, we put the model architecture in the appendix, but the architecture and the paper were separated, and the architecture was not organically combined with the implementation of Chapter 3. In the revised version, we have added section 3.2 to elaborate on the model's architecture separately, and the architecture is shown in Figure 1. Section 3.2 explains the model architecture systematically, including three parts: input data preprocessing, model architecture, and model training methods. Preprocessing of the input series explains how the model preprocesses the input series. The network architecture of GenAD consists of four parts, which are position encoding, 4 hidden layers with Time-Series Attention and Multi-Correlation Attention in each layer, linear layer, and loss function. Model training employs Adam Optimizer to minimize the MSE loss, and forces the model to learn temporal patterns and correlations of multivariate time series to minimize loss after a sufficient number of training epochs.

2. "f1 -> f4 are not formalized". As f1 -> f4 were Time-Series Attention, Multi-Correlation Attention, and the fusion of Time-Series Attention and Multi-Correlation Attention, the formulas of f1 -> f4 were not clearly stated. However, as you mentioned, the formulas of Time-Series Attention and Multi-Correlation Attention were also not stated, which made it difficult for readers to implement code independently even if they understood the role of each part. In response to this review, because the model architecture has been provided, we will not mention f1 -> f4 here, but give a detailed representation of Time-Series Attention and Multi-Correlation Attention. The formula of Time-Series Attention is shown in Eq.(2). Time-Series Attention means that GenAD learns the various temporal patterns of X in Ta, Tb, Tc, and Td, then reconstructs X in time Te. The formula of Multi-Correlation Attention is shown in Eq.(9). Multi-Correlation Attention means that GenAD learns the correlations of multivariate time series in Te and reconstructs the time series by a weighted sum of surrounding time series.

3. "the link between function f and FFT should be clarified". As you said, the relationship between FFT and f was not clarified. At the same time, the paper paid more attention to proving that Time-Series Attention based on FFT theory could capture Periodicity of time series, causing readers to put more energy on FFT, thus weakening the various capabilities of Time-Series Attention for time series. In response to the above problems, we have completely redesigned the content of Chapter 3.2.1, and emphasize that one of the core ideas of GenAD is Time-Series Attention. Time-Series Attention can not only capture Periodicity, but more importantly, it can capture various timing characteristics such as Sequentiality, Trend, Delay, etc. Periodicity is only one of the four capabilities of Time-Series Attention. Chapter 3.2.1 also gives the detailed formula derivation for each capability. For Periodicity, we no longer emphasize FFT, but directly capture Periodicity through Time-Series Attention, and FFT only appears as the basis of the formula.

4. "notation on x and X are not consistent in equation (1) to (8)". Yes, in equation (1), X represents the multivariate time series of a machine or systems, but in equation  (3) to (8), X represents the frequency domain of time series. We have unified the formula expression of the whole paper, including not only x, X, but also the parameters used in T1, T, t, k, and so on. 

4. "The variable ranking on precision and recall depending on the model would push to an AUC metrics. Do the authors carry out some experiments in that direction?". We think the F1 score may be a better evaluation metirc in this scenario. We have conducted some experiments and found that data at many time steps have been perfectly reconstructed (i.e., the reconstruction error is 0). In addition, as described in the paper, we adopt a point-adjust approach to calculate the evaluation metrics, that is, if any observation in the anomaly segment is detected, it is considered that the entire segment is correctly detected. Accordingly, no matter how the threshold is adjusted (or even set to 0), FPR will fluctuate in a small range (due to the high value of TN), resulting in a small value of AUC. 


Dear reviewer,
Great thanks to you for the time and effort you expand into this paper. Your reviews are really helpful and we have revised the paper carefully according to your suggestions.

1. "It seems that a big benefit from the attention mechanisms comes from working in the frequency domain.". In Chapter 3, most of the space is used for proving that Time-Series Attention can realize the function of FFT, and then capture the Periodicity of time series. However, too much emphasis on this ability has led to the weakening of other innovations, and also made readers feel that most of the benefits come from FFT. But even if the abilities of Time-Series Attention for capturing Trend and Delay are not considered, the FFT-filter may only have a better benefit for the single timing series, and cannot capture the complex correlations(dynamic, non-linear and coupling correlations, etc.) among multivariate time series that are not identically distributed, while these correlations have a great contribution to signal reconstruction. In response to the focus deviation of the paper, we first give the network architecture which can be seen in Figure1. Then we have completely redesigned Chapter 3.2.1 and revised 3.2.2. For Periodicity, we no longer emphasize FFT, but directly capture Periodicity through Time-Series Attention, while FFT only appears as the basis of the formula (in Eq(8)). Besides, GenAD is a general model, which is particularly important in real-world scenarios. Another innovative point is the ability to detect anomalies rapidly in anomaly segments. 

2. "experiments are required to understand how the method performs with and without transformation to the frequency domain". We have added some experiments in Appendix G to explore the performance of the model without Time-Series Attention(including Periodicity representation based on the function of FFT). We have observed that due to the loss of some temporal information, the performance of GenAD decreases but overall it is still better than the baseline model. So Time-Series Attention is not the the sole reason for the improvement.

3. "Many formulas have issues. Most references need spacing between previous work. Terms are mentioned (ripple effect) before introduced, etc.". We have carefully checked the formula, and there are some issues, especially the parameters of some formulas are ambiguous. We have unified the formula of the whole paper. Besides, when the ripple effect first appears, we have cited references. And for some format issues, we have made corresponding adjustments.

4. "In the MSL dataset there is no improvement. In the MSL dataset there is no improvement. In the SMD datasets, which is split into 3 categories, there is improvement, but as mentioned, it's not clear which part of the method is responsible for that. is the Fourier-smoothing ? the attention mechanism?". SMD contains three types of machines, a total of 28 machines (training set size is 708405, test set size is 708420). In addition to the SMD and MSL data sets, we have also carried out experiments in the wireless data of real-world scenarios, and achieved good results. However, we are sorry that these data can not be disclosed, so only SMD and MSL data sets are included in the paper.

5. "Previous methods can take advantage of current attention mechanisms. Even though they might not have been proposed with them, and integration is needed to understand if the newly proposed attention mechanisms are substantially better than existing approaches". Previous methods for anomaly detection rarely consider the attention mechanism when optimizing, for the following reasons. First of all, attention-based models are more complicated, which results in more training time. Then the attention mechanism loses sequentiality and is not very friendly to time series. Besides, the attention mechanism performs better in capturing the correlations between the time series and their surrounding series, but the ability to capture temporal patterns of the time series is weaker. In response to the above problems, we first proposed generality for GenAD. With the empowerment of pre-training, only 10\% of the training data is used to represent the characteristics of each device, which brings a significant reduction in training time. What's more, for capturing temporal patterns, Time-series Attention can not only allow for parallelization, but also capture various temporal patterns simultaneously such as Trend, Delay, and complex Periodicity, etc. More importantly, there is a lack of general representations for multivariate time series in previous methods. For specific tasks of time series, such as anomaly detection, time series prediction, root cause location, etc., it is necessary to design task-specific architecture. However, GenAD realizes the general representations, which can be fine-tuned for specific tasks. 


This paper proposes to better represent relationships between different variables within time series and use the result to perform anomaly detection. The experiments show reasonable results. However, the algorithm needs to be presented in a cohesive way to be able to understand what is actually being done. Various steps are presented, but they are not tied together in a way that a reader can implement the algorithm. A detailed algorithm explanation and pseudocode would give all the details needed, including some of what are referenced below.

1. The introduction has a sentence "In practice, the overall status of an entity is more concerned about than each constituent metric..." I don't understand the meaning of this sentence.
2. Section 2. Note that, with OCSVM, time series or sequence-oriented kernels can be used to enable it to be used with time series. E.g., Das, et. al., "Multiple Kernel Learning for Heterogeneous Anomaly Detection: Algorithm and Aviation Safety Case Study," KDD 2010.
3. In section 3, it says that GenAD sets the duration of the training portion of the time series to be four times the duration of the testing portion. How was this chosen? What different splits were tested?
4. Section 3.2.1: The proposition is not stated in a rigorous enough way to be able to tell what is being proven in the proof that is below it.
5. In section 3.3, it says that GenAD selects a random 20% of the dimensions and masks them with fixed random series. Is a different 20% chosen with every time series or for a given problem? How do the results vary depending on the dimensions chosen?

The paper presents GenAD, a DNN architecture for multivariate time-series anomaly detection. The key differences in comparison to existing approaches are two attention mechanisms tailored to capture patterns within time series and correlations between time series. Experimental results on two types of datasets show the effectiveness of the proposed architecture. In addition, the paper studies a pre-training method to transfer models to new applications. 

Pros

- Tackles a timely problem as multi-variate time series are becoming prevalent
- The architecture is simple yet effective
- The pre-trained methodology can improve the practicality of such architectures

Cons

- Technical novelty is limited with attention mechanisms proposed to be extensions of current approaches
- The writting needs substantial improvement. Math formulas have issues
- Evaluation is weak - essentially one of the two datasets show improvement - in the other dataset the difference is marginally better

Details:

- The novelty is low. It seems that a big benefit from the attention mechanisms comes from working in the frequency domain. However, this is something that all other methods can take advantage of. The FFT-filter essentially smooths the data, which might be the sole reason for the improvement. The integration of such smoothing in the remaining methods is necessary to understand the true effect of the proposed attention mechanisms.

-  Similarly as above, experiments are required to understand how the method performs with and without transformation to the frequency domain. 

- Many formulas have issues. Most references need spacing between previous work. Writting is not clear, terms are mentioned (ripple effect) before introduced etc.

- The evaluation seems preliminary. In the MSL dataset there is no improvement. In the SMD datasets, which is split into 3 categories, there is improvement, but as mentioned, it's not clear which part of the method is responsible for that. is the Fourier-smoothing ? the attention mechanism?

- Previous methods can take advantage of current attention mechanisms. Even though they might not have been proposed with them, an integration is needed to understand if the newly proposed attention mechanisms are substantially better than existing approaches

The authors present an architecture to detect anomalies in multi-variate time series. To represent this complex input, they rely on a multi-correlation attention. Their approach compute correlation while taking into account periodicity of the signal through FFT.
After describing their approach, the authors experiment sucessfully on 2 datasets.

Section 3 should be clarified. The authors have to clarified their architecture and the computation that are made before proposing proofs. For me, the architecture is not clear.
f1 -> f4 are not formalized, the link between function f anf FFT should be clarified.

I think that notation on x and X are not consistent in equation (1) to (8)

The variable ranking on precision and recall depending on the model would push to an AUC metrics. Do the authors carry out some experiments in that direction?

Experiments on latency are very interesting

Meta-review: The paper proposes an architecture of a model for multiple correlated time series that has application in anomaly detection.  The main idea is to use attention both along time to capture trends, seasonality, etc and along series types to capture their correlation.  Training loss attempts to reconstruct masked ranges of values, and thus the discrepancy between predicted and actual values can be used to flag anomalies. 
While the ideas proposed in the paper are useful and lead to a well-engineered model while combining current architectural elements to best suit their task, the paper fails to impress as a research contribution.   The writing requires improvement, and the experiments on two datasets are not entirely convincing.  The authors have realized these limitations, as can be seen in the author feedback.  However, the changes entailed are too extensive to  revise the ratings across all reviewers.
I hope these will help the authors submit a better next version to another conference.

Decision: withdrawn-rejected-submissions