Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a residual-style architecture namely STRIC for multivariate time series (TS) forecasting and anomaly detection, by first decomposing TS to trend, seasonality, and irregular components as residual, then extracting features from the residual using TCN, and finally introducing a likelihood ratio estimation method. Experimental results show that STRIC can provide interpretability for TS forecasting and outperform some existing anomaly detection models. 
main_review: Strengths: The regularization method introduced in Section 4 facilitates avoiding model overfitting. 

Weaknesses:

1. Experimental results are not convincing. The chosen baselines are not state-of-the-art solutions for TS anomaly detection, please refer to [1] for details. The benchmark dataset used for comparison should also be changed. For example, as pointed out in [2], Yahoo dataset is not preferred for comparing AD solutions. 

2. TS decomposition itself is a well-studied research problem and it is essential to compare the proposed architecture with existing techniques (e.g., [3]) to obtain the residual. 

3. The paper is difficult to follow. Firstly, the overall workflow for anomaly detection is not clear, and it is difficult to see the connections between Section 4 and Section 5. Secondly, the model is not described clearly. For example, what is the non-linear module? Is it the TCN block shown in the figure without a title? Thirdly, there does not exist any statement on how the anomaly detector determines a particular time point or subsequence to be an anomaly. 

4. The paper is not focused and swings between prediction, anomaly detection, and even text embeddings. 


References: 

[1] Lai, Kwei-Herng, et al. "Revisiting Time Series Outlier Detection: Definitions and Benchmarks." (2021).

[2] Wu, Renjie, and Eamonn Keogh. "Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress." IEEE Transactions on Knowledge and Data Engineering (2021).

[3] Wen, Qingsong, et al. "RobustSTL: A robust seasonal-trend decomposition algorithm for long time series." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.
summary_of_the_review: While some of the proposed techniques seem to be interesting, the paper is very difficult to follow and the experimental results are not convincing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to boost the performance of deep neural networks (DNNs) for time series applications by focusing on the characteristics of interpretable forecasting and anomaly detection which are important for real-world time series data. The authors propose an end-to-end trainable DNN architecture which is composed of stacked residual blocks to separate signal components including slow trends, quasi-periodicity, and linear dynamics, followed by a temporal convolutional network (TCN) to model other components. Although previous studies showed that conventional simple linear models often outperformed DNN models on typical time series benchmarks that require robustness and interpretability, this paper shows that the proposed DNN model outperforms state-of-the-art robust statistical methods in some datasets, kind of demonstrating the best of both worlds.

main_review: Strengths:

* The network architecture (Figure 1) is carefully designed to incorporate useful ideas from both conventional statistical time series models and DNN models to allow the proposed model to get the best of both worlds, including its interpretability and flexibility.

* Although the regularization method for the TCN is inspired by the automatic relevance determination (ARD) method proposed more than two decades ago for (sparse) Bayesian learning of neural networks, its use here is novel as far as I know.

* The nonparametric extension of the classical CUMSUM algorithm for anomaly detection does not require knowledge of the distributions of the two windows of data points, relaxing a major restriction of the original CUMSUM.

Weaknesses:

* Presentation of the experimental results (Table 1) can be improved. For example, some baseline models (e.g., ARIMA) are in fact univariate time series forecasting models. I suppose they handle each dimension of a multivariate time series separately. It would be clearer in the presentation to separate the models into two groups depending on whether they are really multivariate models. In fact, there also exist conventional multivariate time series forecasting models (e.g., VAR). Shouldn’t some of them be included as well? For clarity, it would help to show the dimensionality of each dataset as well, perhaps using a table to summarize the major characteristics (including dimensionality) of all the datasets used.

* While the nonparametric extension of CUMSUM is good, the estimation error can be high (also with high variance) if the windows are not long enough, yet having too long windows may miss some small, subtle anomalies. This is a dilemma. A more detailed analysis is needed.

* The experimental study is a bit limited. It is not clear whether the superior performance of the proposed method shown in Table 1 also holds for a wider range of real-world multivariate time series datasets.

Some general comments:

* You may consider how the proposed automatic complexity determination scheme is related to the attention mechanism.

* For qualitative evaluation, it would help to demonstrate the efficacy of the automatic complexity determination scheme by visualizing the “relevant past” selected automatically.

* In Figure 1, other than the captions of the sub-figures, there should also be a caption for the figure as well. At least it should be shown as “Figure 1” explicitly.

* There are some minor language errors in the paper. It should be proofread carefully to correct them.
summary_of_the_review: This is a good work which tries to integrate ideas from both conventional time series forecasting models and more recently DNN models. There are some novel ideas which are interesting and are worth studying further. Nevertheless, as mentioned above, the significance of the proposed model for multivariate time series anomaly detection is unknown due to the limitations of the empirical study.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: ·	An interpretable stacked residual framework is proposed for time series anomaly detection with interpretation on seasonality and trend of the target time series.

·	A novel regularization term is proposed to optimize network parameters with Bayesian Optimization

·	An extension of CUMSUM algorithm is proposed to detect outliers without knowledge of pre- and post-distributions of the data.

main_review: ·	Strength:

o	The motivation of each components are clearly justified.

o	The empirical result on univariate time series data outperforms most of the baselines.

o	To the best of my knowledge, the Bayesian learning formulation and CUMSUM algorithm with subspace likelihood ratio estimation are novel.

·	Weakness:

o	Empirical evaluation is only conducted on univariate time series data, making it not convincing that it is capable of dealing with multivariate time series data (which is very common in many application scenarios). Add some evaluations on multivariate time series may help on eliminate the concern.

o	The selected datasets has been shown as flawed benchmark [1] and can be easily solved with few lines of rules. Conducting experiment on these datasets making it questionable about the performance of the proposed algorithm. Either adopting new datasets to evaluate the algorithm or involving the one-line rule as baselines may be helpful to eliminate the concern.

o	The proposed residual framework assume that the input time series is the addition of trend, seasonality and residual series. However, it is very likely that the three components are not linearly dependent to the resulting time series. Providing more justifications on using addition to model aggregate the information may be helpful to eliminate the concern.

o	There is no empirical evaluation of the interpretable components. The author only show the result of decomposition without illustrating what makes it helpful for interpreting outlier detection result (which is one of the main contributions of this paper). Adding more justification with outlier annotations on Figure 2 may be helpful for readers to understand the contribution in empirical analysis.

[1] Renjie Wu, Eamon Keogh, Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress, IEEE Transactions on Knowledge and Data Engineering, 2021

summary_of_the_review: ·	To summarize, this paper proposed a novel interpretable outlier detection algorithm with stack residual blocks, regularized model complexity selection and novel CUMSUM detection algorithm. The proposed method is technically sound. However, the selecting datasets are solely in univariate setting and are shown to be flawed in previous study, which makes the empirical evaluation less credible. In addition, technical justification and empirical analysis of residual framework is missing, making it hard to evaluate the contribution of interpretable components. I believe some more modifications will certainly improve the quality of paper to meet the ICLR standard. Therefore, I vote for marginally below the acceptance threshold.
review_title: null

Meta-review: This paper studies the important problem of time series anomaly detection using deep neural networks (DNNs). Unlike many other DNN models, it focuses on incorporating in its model architecture interpretable components that are inspired by previous studies based on both conventional statistical methods and more recent DNN models.

While the paper has merits as pointed out by the reviewers (esp. TtBt), a number of concerns have also been raised, including the choice of datasets (e.g., by reviewers rnBY and zX4p). We appreciate the authors’ effort by adding some preliminary results of further experiments, but addressing all the concerns thoroughly will need a lot more work to get a scholarly paper that is more ready for publication. We believe this work has potential to be accepted for publication in a reputable venue if the concerns are thoroughly addressed after substantial revision.
Decision: Reject