Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies time series outlier detection. The authors propose to compare recurrent and static outlier detection methodologies on multiple synthetic and real-world temporal datasets. Additionally, they present an extension to the LSTM outlier detection using an attention-based approach. The analysis suggests that non-recurrent methodologies should be prefered as they perform marginally better while requiring fewer parameters and training. Only in the context of complex temporal relations, recurrent approaches should be used.
main_review: The work presents numerous experimental results. However, the related literature should be further developed to get a better sense of the different methods existing. Additionally, some experimental clarification would help the reader understand: are the authors using a Monte Carlo cross-validation? Is the contamination repeated in each fold? How is the contamination performed?

Moreover, the results need statistical comparisons to understand if the distributions between recurrent and static models are significantly different (KS test) and also if the ROC are different (DeLong test). The current figures do not show clear performance gain from any method. 

The final message should also be clarified: which method should be used in which context? Do some approaches perform better when lower dimensionality, high colinearities, long time patterns? Is one approach to be prefered if static methods are used. It would also be important to integrate an 'outlier time series' comparison to have a full overview of the use case of the different approaches.

Finally, the proposed approaches do not seem to offer any gain, it might be more interesting to develop on which methods perform better in which context.
summary_of_the_review: The authors present a substantial experimental work that offers interesting insights on time series outlier detection. However, the paper needs further statistical analysis to better understand the significance of these results. Additional analyses would also help to understand which techniques to use in different contexts.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper compares non-RNN and RNN methods in the outlier detection context, and concludes that non-RNN approach is suited to point outliers while RNN approach is suited to collective outliers. 
main_review: There are many existing non-RNN methods for time series outlier detection which can be found in e.g.

Manish Gupta, Jing Gao, Charu C. Aggarwal, Jiawei Han:
Outlier Detection for Temporal Data: A Survey. IEEE Trans. Knowl. Data Eng. 26(9): 2250-2267 (2014)

Those methods are for both point and collective outliers and should be included in the evaluation study.

The conclusion is drawn on 3 real-world data sets, one of which (Movement data) seems not publicly available. I find this not sufficient. For comprehensiveness, I would recommend to use existing relevant data sets from UCI Repository and optionally add artificial outliers.




summary_of_the_review: The technical contribution is incremental. Relevant existing non-RNN methods for outlier detection on time series are not included in the study.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a comparative study of the performance of non-recurrent models and deep recurrent models for time series outlier detection. Seven models are evaluated on multiple synthetic and real-world time series datasets. 
main_review: The strengths of this paper are as follows:
- Time series outlier detection is an important problem as it can be applied to many real-world applications
- The research question 'do we need recurrent neural networks for time series outlier detection?' is of interest to the community
- A series of empirical results is presented to answer the question

The weaknesses include:
- The work is a rather simple comparative study, performing a few relevant models on some existing datasets without in-depth analysis of the experimental settings and empirical results. As a result, the empirical findings are not convincing. Some major issues here include: 1) there are a number of recent LSTM/RNN-based models that are specifically designed for time series outlier detection, but they are not used in the study; only basic LSTM/RNN models are used instead. 2) the way of constructing or injecting the outliers (as well as the types of outlier) can largely affect the performance of each model, but this is largely ignored in the study. 3) the analysis of the empirical results is so shallow that it cannot provide any insight into the underlying reasons of the obtained results.
- The crucial concept -- data complexity of time series data in outlier detection -- is not properly defined and quantified. Without a clear definition of data complexity, it is invalid to state whether or not more complex models are needed. The authors claim about the data complexity in terms of the presence of some complex outliers, such as collective outliers, but detailed analysis of this complexity through some ablation studies  is missing. There are many other complexity factors that should be considered, for example, how is the temporal dependence presented in each dataset? does the length of the temporal dependence vary significantly within each dataset? etc.
- It is unclear whether the used models are capable of detecting collective outliers or not. If the answer is yes to some models, how?
- No technical novelty is identified. It is a purely comparative study with only some shallow results. The work may be largely improved by including more datasets with natural real-life outliers and performing controlled experiments to justify some specific questions rather than some high-level general questions.
summary_of_the_review: Overall, the work seems to be a technical report that contains some preliminary results but has many key issues to be further addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper evaluates a set of methods for time-series anomaly detection, including RNNs among the considered techniques. The evaluation on a set of datasets shows that RNNs perform better than the non-RNN architectures.
main_review: Strengths:

1) Anomaly detection in time series is an important problem

Weaknesses:

1) The evaluation seems incomplete. 

Details:

1) The title/abstract makes you believe that will read some other kind of paper but this is essentially an evaluation paper of anomaly detection methods. Nothing is wrong with that, this is just far from complete. There are dozens of methods to consider. 

2) Not sufficient datasets to make such claims. Not sufficient analysis between anomaly types. Many questions of this form: are CNNs better? Are subsequence methods better? Are traditional vs modern approaches better? ....
summary_of_the_review: Preliminary results of an evaluation work for time-series anomaly detection. Motivation has to be reconsidered.
review_title: null

Meta-review: This paper has been reviewed by four experts. Their independent evaluations were consistent, all recommended rejection. I agree with that assessment as this paper is not ready for publication at ICLR in its current form. The reviewers have provided the authors with ample constructive feedback and the authors have been encouraged to consider this feedback if they choose to continue the work on this topic.
Decision: Reject