Towards zero shot multivariate time series anomaly detection - A Realistic Evaluation

Authors: Ayan Banerjee, Sandeep Gupta
Primary area: learning on time series and dynamical systems
Keywords: multidimensional time series anomaly detection, zero shot, model recovery, model conformance

Abstract:
A long line of multivariate timeseries anomaly detection (MTAD) approaches use performance enhancement techniques that are not feasible in practical scenarios. In specific, a) point adjustment technique is employed which uses ground truth to forcefully convert false negatives to true positives and inflates precision to unrealistic proportions, and b) significant data leakage is introduced where anomaly score threshold is determined using the test data and test labels. In this paper, we show the real world performance of existing MTAD techniques when point adjustment and threshold learning on test data is disabled. Moreover, we show that anomalies introduced in real world benchmark datasets result in significant distribution shift between normal and anomalous data, and when point adjustment and threshold learning are used even untrained deterministic methods can perform on par or even beat baseline techniques. We then introduce six synthetic benchmark examples derived from real world systems, where anomalous data and normal data have statistically insignificant distribution shift. We propose, sparse model identification enhanced anomaly detection (SPIE-AD), a model recovery and conformance based zero shot MTAD approach that outperforms state of art MTAD techniques on three real world benchmark datasets without using point adjustment and threshold learning on test data. We evaluate state-of-art MTAD and SPIE-AD on the novel synthetic benchmarks. SPIE-AD outperforms state-of-art MTAD techniques on both standard and novel benchmarks.

Decision: reject
Reviewer scores: 5, 8, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730115689017

thread_type:
official_review

<thread_object>
creator: reviewer_awo8

<content>
summary:
This paper introduces zero shot multivariate time series anomaly detection that can avoid point adjust step and data leakage in hyperparameter tuning. The proposed method is a sparse model recovery built upon ODE. A neural network is also used to handle the robustness. The proposed method could finally produce an interval based on the kth smallest value for labeling.

strengths:
1. The proposed method can outperform existing methods by a large margin without point adjustment.
2. The hyperparameter of the proposed method can be tuned only with training data.
3. Build new benchmarks for better evaluating the U2 anomaly.

weaknesses:
1. The writing needs a lot of improvement.  Below are some examples:
  + It is hard to follow the main content of the proposed method. 
  + There are limited details about how the proposed benchmark can reflect the claimed properties. 
  + Why can the claimed difficulty not be handled by some heuristics or advanced tech? For example, A1 could also be addressed by extreme value theory for dynamic threshold [1]. A2 could be mitigated by some better measurements to mitigate the effect of noisy labeling [2]. 

2. The experiments are not rigious, which decreases the confidence of the experiments. 
  + The compared methods are too weak. In the real-world dataset evaluation, the compared method achieves a 0 F1 score. This is hard to trust. Why not provide heuristics like quantile for thresholding or extreme value theory for dynamic threshold? 
  + Recent progress [2] in evaluating AD has proposed several threshold-free measurements. It seems the bad performance of the compared method largely results from only using F1 score, which requires a threshold.
  + There is zero empirical analysis for the proposed benchmark. 
  + In Table 3, AT+ is the best but is not marked as bold in F8Slow. 

[1] Anomaly detection in streams with extreme value theory. KDD 2017.
[2] Volume under the surface: a new accuracy evaluation measure for time-series anomaly detection. VLDB 2022.

questions:
Please address the concerns in the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730271293114

thread_type:
official_review

<thread_object>
creator: reviewer_szlj

<content>
summary:
The paper proposes a sparse model identification enhanced anomaly detection (SPIE-AD) model which is a recovery and conformance-based zero shot MTAD approach. It disables point adjustment and thresholding methods. The backbone of SPIE-AD are the two fundamental theoretical contributions of this paper: a) robust sparse non-linear dynamical model recovery from real-world multi-variate data using neural architectures with automated differentiation and b) statistical conformance-based model robustness interval extraction (CRIE) algorithm that can identify statistically relevant difference in recovered models.

strengths:
The proposed model using dynamic modeling is novel. And paper is mathematically sound and well-written. Performance on synthetic data is good.

weaknesses:
U2 scenario is nonrealistic as real world data is mixed with unknown noise. It is hard to get training data cleaned up in real-world data. It will be better to discuss how author would handle noisy or imperfect training data in real-world applications.

What is a contribution against the Autoencoder-based method? Can you please compare SPIE-AD approach to autoencoder-based methods like USAD, highlighting key differences and potential advantages in terms of performance, computational efficiency, or interpretability.

Also, validation data should not be used as training data. Is there any real-world application in your mind behind this? 

What happens if you use separate validation data? It would be better to conduct an additional experiment using a separate validation set that is distinct from both the training and test sets and kindly report how this affects the results. 

The author models robustness as a reconstruction error. Why the dynamic model is called sparse and robust in not clear. Please provide a more detailed explanation of the definition of "sparse" and "robust" in the context of their dynamic model. Can you please explain how sparsity is enforced in the model structure and how robustness is quantified beyond reconstruction error?  Some experimental evidence on the usefulness of sparsity and robustness for detecting anomalies would be nice to have.


Table 4 shows that the proposed model is performing worse than many benchmarks. Please provide a more in-depth analysis of these results, discussing potential reasons for underperformance.

Some analytical results about where the proposed method fails would be good.  Please provide a detailed error analysis, identifying specific types of anomalies, datasets, or scenarios where your method underperforms. Please also discuss potential reasons for these failures and propose potential improvements or future work to address these limitations.

questions:
As above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730595560922

thread_type:
official_review

<thread_object>
creator: reviewer_t8kh

<content>
summary:
This paper proposes a multi-variant time series anomaly detection (MTAD) method that differs from existing approaches by learning the relationships between points on a curve. Instead, it judges whether a certain time series is anomalous by comparing the changes in model parameters \( w_i \) fitted from different data. This method no longer relies on labels from the validation set to tune the model and can achieve good results without using point adjustment.

strengths:
1. The paper points out that both point adjustment and reliance on validation labels are significant issues.
2. The application of the SMR method to anomaly detection in the paper is innovative.
3. The paper contributes a synthetic evaluation dataset.

weaknesses:
1. More related methods should be compared and discussed.
2. The writing part of the paper requires some improvement; the introduction is too long.
3. The references are not placed in parentheses, which affects readability.
4. The placement of related work in the middle of the paper, together with the baseline in the evaluation, is not intuitive. Related work should be an independent section as it discusses relevant papers in this research field.

questions:
1. The paper uses relatively simple baselines. Some methods with better performance, such as TFAD[1], OFA[2], and FITS[3], are not compared. It is uncertain whether the method in this paper still has certain advantages under zero-shot evaluation methods and the u2 dataset.
2. The paper only mentions the SMD dataset, but there is a benchmark[4] that even focuses on zero-shot and point adjustment alternatives. How does the paper's method perform on this benchmark?

[1] TFAD: A decomposition time series anomaly detection architecture with time-frequency analysis

[2] Zhou T, Niu P, Sun L, et al. One fits all: Power general time series analysis by pretrained lm[J]. Advances in neural information processing systems, 2023, 36: 43322-43355.

[3] FITS: Modeling time series with 10k parameters, in The Twelfth International Conference on Learning Representations, 2024.

[4] Si, Haotian, et al. "Timeseriesbench: An industrial-grade benchmark for time series anomaly detection models." arxiv.org/pdf/2402.10802 (ISSRE 2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648015048

thread_type:
official_review

<thread_object>
creator: reviewer_v6ne

<content>
summary:
The paper introduces SPIE-AD, a zero-shot multivariate time series anomaly detection approach that aims at overcoming the prevalent information leakage from threshold selection prevalent in the literature. The key innovation is combining sparse model identification with conformance testing to detect anomalies without requiring anomaly examples during training. The authors also contribute new synthetic benchmarks and demonstrate issues with current evaluation practices.

strengths:
The paper tackles an important problem, pervasive in the anomaly detection literature that puts into question the validity of a lot of published results in real-work scenarios. The paper introduces a novel technical approach combining model identification with conformance testing and performed a comprehensive empirical evaluation including with the release of new benchmarks. The theoretical foundations of the paper appear sound.

weaknesses:
The paper contains limited discussion of computational complexity of the proposed methods. This is an important weakness given that the proposed method is not straightforward. Similarly, the authors could better explain parameter sensitivity and failure cases. 
The issue with information leakage from calibration of threshold on the training set has been identified in previous works, the related literature section is missing a discussion of this. 

The empirical analysis is limited. It relies mostly on the synthetic datasets proposed by the authors. The quality and informativeness of the real-world datasets they use have been questioned, rightfully so in this reviewer's opinion. UCR offers a large collection on time-series anomaly datasets that are well documented. The authors should use them. Similarly, the authors should include simple, algorithmic baselines to their comparison and include computational cost as a relevant dimension.

questions:
What is the computational overhead of SPIE-AD compared to existing methods?
How sensitive is the method to the choice of window size?
Are there specific types of anomalies where the method might fail?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684380670

thread_type:
official_review

<thread_object>
creator: reviewer_fj9q

<content>
summary:
This paper proposes a zero-shot time series anomaly detection method called SPIE-AD, which leverages sparse model recovery (SMR) technology to detect anomalies by monitoring changes in model parameters. SPIE-AD makes two main modifications: it does not rely on manually labeled labels for hyperparameter tuning in the validation set, and it abandons the controversial point adjustment evaluation method that has been heavily relied upon. The evaluation results are explicit, showing that SPIE-AD significantly outperforms existing methods on the U2 benchmark.

strengths:
1. The method proposed in the paper for anomaly detection through model recovery is innovative, as traditional methods that model normal patterns based on data and then compare them have high dependencies on the quantity and quality of data.
2. The challenge of consistent distribution between anomalous and normal data in the U2 benchmark presented in this paper, which is difficult to distinguish, has not been addressed in many works, making it somewhat innovative.

weaknesses:
1. The meanings of H and P in Figure 3 are not indicated.
2. Lines 6 and 9 in Algorithm 2 are reversed.
3. On line 84, it is stated that existing anomaly detection methods are based on the assumption of anomaly-free data, which is invalid. The biggest difference between anomaly detection tasks and time series tasks is that anomaly detection tasks do not assume that training data is free of anomalies, which is why there are many reconstruction methods have been proposed.
4. The relationship between zero-shot and A1, A2, A3 needs to be clarified. Are they parallel, or do A1, A2, and A3 make the zero-shot objective face greater challenges?

questions:
1. Some very classic MTS evaluation datasets, besides SMD, have not been evaluated, such as Yahoo[1].
2. Some very classic MTS baselines and methods have not been compared, such as OmniAnomaly[2].
3. To achieve good zero-shot performance, some time series foundation models proposed through time series analysis have been proposed, which can do both time series forecasting and anomaly detection. These methods should also be compared, such as OFA's sec 4.2[3], and some related work mentioned by OFA.
4. A very similar evaluation work, TimeseriesBench[4], has not been mentioned. TimeseriesBench discusses zero-shot and point adjustment comparisons. How would SPIE-AD perform on TimeseriesBench?

[1] Y. Research, “A benchmark dataset for time series anomaly detection.” https://yahooresearch.tumblr.com/post/114590420346/a-benchmark-dataset-for-time-series-anomaly, 2015.
[2] Su, Ya, et al. "Robust anomaly detection for multivariate time series through stochastic recurrent neural network." Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2019.
[3] Zhou, Tian, et al. "One fits all: Power general time series analysis by pretrained lm." Advances in neural information processing systems 36 (2023): 43322-43355.
[4] Si, Haotian, et al. "Timeseriesbench: An industrial-grade benchmark for time series anomaly detection models." arXiv preprint arXiv:2402.10802 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732479851146

thread_type:
public_comment

<thread_object>
creator: ~eamonn_keogh1

<content>
title:
You test on SMD SMAP MSL. However it is increasingly understood that it is meaningless to test on these datasets

comment:
You test on SMD SMAP MSL. However it is increasingly understood that it is meaningless to test on these datasets, because the ground truth is mislabeled, they are trivial, and have other flaws. See [a]

Perhaps I misunderstand, but isn't the Matrix Profile TSAD algorithm zero shot?  (and zero parameter?) 
https://www.youtube.com/watch?v=vH4MzuaBeOQ&ab_channel=EamonnKeogh 

[a]
https://www.dropbox.com/scl/fi/x6ie264xrfkl0nbdw1vtb/Irrational-Exuberance_why_most_TSAD_is_wrong.pdf?rlkey=16frcr2lo6ip5o6uf18qwdeud&dl=0
and
https://www.dropbox.com/scl/fi/cwduv5idkwx9ci328nfpy/Problems-with-Time-Series-Anomaly-Detection.pdf?rlkey=d9mnqw4tuayyjsplu0u1t7ugg&dl=0
</content>
</thread_object>

</thread>
<thread>
creation date:
1733160216802

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of changes

comment:
Dear Reviewers and Area Chair,

Based on the review comments we have made the following changes in our paper:

1) Added real world example of U2 scenarios and the properties of datasets induced by U2 scenarios.

2) Clarified the problem definition of anomaly detection in Figure 1

3) Added three new anomaly detection baselines OFA, TFAD and FITS

4) Added new real world benchmark dataset from UCR that has 250 real world timeseries with anomalies

5) Added computational complexity of SPIE-AD and compared the execution time of baselines with SPIE-AD

6) Added a new experiment to determine sensitivity of SPIE-AD to window size Figure 5

7) Expanded on the limitations of the approach in a limitation section.

After the revision there has been limited interaction with reviewers. Please let us know if you have any more feedback after the revisions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The work aims to address a suspicious information leakage issue in multivariate time series anomaly detection, where anomaly detectors require validation/test data with anomaly samples to determine an effective threshold for the binary labeling. It proposes an approach based on sparse model recovery to tackle the issue.

As pointed out by the reviewers, the strengths of the work include: 1) the sparse model recovery-based method is new or theoretically sound (fJ9Q, V6nE, T8KH, sZLJ, awo8), 2) the proposed evaluation protocol is interesting and sounds more practical (fJ9Q, V6nE, T8KH, awo8), 3) the creation of synthetic evaluation datasets for avoiding the information leakage problem (T8KH)

The reviewers also identify a number of weaknesses: 1) some major paper clarity issues (fJ9Q, T8KH, sZLJ, awo8), 2) missing of comparison to some baselines, relevant benchmarks (fJ9Q, V6nE, T8KH, sZLJ, awo8), 3) the proposed method is not straightforward and its time complexity analysis is missing (V6nE), 4) lack of empirical justification on real-world datasets (V6nE), 4) the studied setting is unrealistic to some extent due to possible noise/anomaly contamination in real-life data (sZLJ), 5) lack of failure case analysis (sZLJ)