When Complexity Is Good: Do We Need Recurrent Deep Learning For Time Series Outlier Detection?
Authors: Alexander Capstick
Samaneh Kouchaki
Mazdak Ghajari
David J. Sharp
Payam M. Barnaghi
Venue: ICLR 2022 Submitted (2022)
Keywords: Outlier Detection
Time Series
Deep Learning
Recurrent Neural Networks

Abstract: Outlier detection is a critical part of understanding a dataset and extracting results. Outlier detection is used in different domains for various reasons; including detecting stolen credit cards, spikes of energy usage, web attacks, or in-home activity monitoring. Within this paper, we look at when it is appropriate to apply recurrent deep learning methods for time series outlier detection versus non-recurrent methods. Recurrent deep learning methods have a larger capacity for learning complex representations in time series data. We apply these methods to various synthetic and real-world datasets, including a dataset containing information about the in-home movement of people living with dementia in a clinical study cross-referenced with their recorded unplanned hospital admissions and infection episodes. We also introduce two new outlier detection methods, that can be useful in detecting contextual outliers in time series data where complex temporal relationships and local variations in the time series are important.

Under review as a conference paper at ICLR 2022
WHEN COMPLEXITY Is GOOD: Do WE NEED RECUR- RENT DEEP LEARNING FOR TIME SERIES OUTLIER DETECTION?
Anonymous authors Paper under double-blind review
ABSTRACT
Outlier detection a critical part of understanding a dataset and extracting results. Outlier detection is used in different domains for various reasons; including de- tecting stolen credit cards, spikes of energy usage, web attacks, or in-home activity monitoring. Within this paper, we look at when it is appropriate to apply recur- rent deep learning methods for time series outlier detection versus non-recurrent methods. Recurrent deep learning methods have larger capacity for learning complex representations in time series data. We apply these methods to vari- ous synthetic and real-world datasets, including dataset containing information about the in-home movement of people living with dementia in clinical study crose-referenced with their recorded unplanned hospital admissions and infection episodes. We also introduce two new outlier detection methods, that can be use- ful in detecting contextual outliers in time series data where complex temporal relationships and local variations in the time series are important.
INTRODUCTION
Outlier detection is the process of identifying unusual data points, or collections of data points within a dataset. Within different contexts, these outliers can take various forms, for example:
Credit card fraud detection: Unusual spending.
Internet traffic monitoring: Unusual surges in traffic or security monitoring.
Dementia care: Unusual patterns of activity, unusual sleep/wake patterns night.
This paper will focus specifically on time series outliers. These are outliers that appear in sequential data, which which (lázzuBzz-arrraree al. (202T) argued could take three main forms:
Point outliers: These are single data points in time that are unusual from the rest of the dataset when considering other values in the time series points locally. Point outliers could have global unusual values, or contextually (in our case, temporally) unusual values.
Subsequence/Collectiee outliers: These are consecutive points whose behaviour col- lective is unusual, but not necessarily if considered separately.
Outlier time series: These are instances in which an entire time series (for example, variable in multivariate dataset) can be considered unusual.
In this paper, we focus on point and subsequence outliers and discuss when and how recurrent deep learning techniques could be used for this type of outliers compared to non-recurrent techniques. This involves testing several algorithms on multiple datasets containing different outlier types. We also introduce two new outlier detection methods by modifying the Long Short Term Memory Re- current Neural Network (LSTM-RNT) (Hochreiter & Schmidhuber 1997) outlier detection tech- nique presented in Bontemps et al. (2016); Singh (2017). We change the underlying forecasting model and instead of an LSTM-RNN we train a transtormer model (Vaswani et al. 2017) and sim- ilarly, use its prediction loss the basis for the outlier score calculation. We also removed the

---

datasets for which there is no Under review as a conference paper at ICLR 2022
focus collective outliers (as is in the LSTM-RNN outlier detection model), so that model was applicable to both point and collective outliers.
We continue this paper by introducing some of the preliminary concepts that will be required to design the outlier detection settings, as well as explaining the difference between recurrent and non-recurrent based outlier detection algorithms. Next, this paper discusses the related works and describes how our study extends the existing work. We introduce the methods and models that will be used answer the questions of this paper, followed by the results and discussion on their significance.
This paper's aim is to answer the following question:
"When detecting outliers in time series data, appropriate use recurrent neural networks and how do we design models for this purpose?"
Our motivation for answering this question is to enable better understanding of which outlier detection techniques would be most usefully applied to our healthcare monitoring "real-world" time series dataset, containing movement information collected form the homes of people living with dementia (more information on Section 2.I.. We also wish to understand how useful recurrent based algorithms
when points in the dataset contain complex temporal relationships with each other. For example, we wish to find outliers that are not only locally and globally unusual in their values, but where they are also unusual given some sequence that they exist in.
1.1 PRELIMINARY CONCEPTS
WHAT IS RECURRENT NEURAL NETWORK?
Within this paper, will refer two different types of outlier detection methods. Those that use non-recurrent techniques and those that rely on recurrent deep learning models.
Recurrent neural network approaches are characterised by their use of sequences in the time series data to detect outliers. Classical non-recurrent algorithms assume no temporal relationship between data points and calculate outlier scores on point-by-point basis, whilst recurrent neural network algorithms use temporal relationships when detecting outliers. In theory, this difference should make recurrent neural network models superior to non-recurrent models when detecting outliers in complicated time series data. This is not necessarily the case however, since many outliers will hold unusual enough values such that they can be detected by non-recurrent techniques. In addition, extra capacity for learning complicated relationships in the data brings larger numbers of hyper- parameters to tune, which can cause difficulty in training robust models. This tuning becomes issue when applying complex techniques outlier ground truth to measure model performance.
OUTLIERS IN A DATASET
As discussed in Section M outliers take three main forms when viewed in the context time series data. The number of outliers in dataset is defined proportion of the whole dataset and is called the contamination. To contaminate a dataset is to add outliers to that dataset. When measuring the performance of outlier detection techniques, we will harmonic mean of precision and recall, defined as the F1 score (see Appendix A.1
1.2 RELATED WORK
There has been long history of developing outlier detection methods and exploring different algo- rithms for time series outlier detection (Lai et al., 2021bb Blázquez-Garce et al. (021). The existing works propose various methods to identify different types outliers that are present in time series datasets. Lai et al. (2021b) show comparisons between different algorithms for outlier detection and evaluate their performance on both synthetic and real-worl world datasets. However, Lai et al.) (2021b) do not directly compare recurrent and non-recurrent based outlier detection algorithms.
Elsayed et al. (202T) aimed to address a similar question in relation to time series forecasting, in which they argued that carefully configured Gradient Boosted Regression Trees (GBRT) model
2

---

Under review as a conference paper at ICLR 2022
(Friedman 2001) could compete and even sometimes out-perform many complex deep learning models for time series forecasting. Pang et al. (202I) studied the role of deep learning in outlier detection, in which they acknowledge that complex models would be useful for complex anomalies (those with complicated spatial or temporal dependencies), although it is not as widely explored as evaluating the performance of complex models for detecting point anomalies. Both of these studies, however, do not directly compare recurrent and non-recurrent based algorithms.
We aim extend the scope Lai al. (2021b)3s 's work by specifically investigating whether recur- rent neural networks are useful for outlier detection compared to non-recurrent methods. An answer to this question would guide us in our search for a reliable outlier detection method for application on our remote monitoring time series dataset, related to dementia care (described in Section (2.1) We are aware that our dataset will contain complex temporal relationships between data points and we wish for an outlier detection algorithm to be proficient at finding all of these types of outliers.
2 METHODS AND MODELS
In this section, we discuss different datasets used in our experiments and describe how the datasets are generated or collected. Next, we discuss which existing recurrent and non-recurrent outlier detection models we have chosen. We also introduce two new outlier detection methods that rely on transformer models (Vaswani et al. 2017).
2.1 DATASET
To evaluate the models, we have used the TODS python package ! and benchmarks (Lai et al. 202 which contain several synthetic and real-world datasets for time series outlier detection.
The synthetic datasets were generated using sinusoidal waves with various frequencies and ampli- tudes. These datasets are multivariate and always consist of 5 variables. Contamination functions are used to contaminate this dataset with 5 different objectives as outlined in Lai et al. (2021b). These contamination objectives are aimed at producing point outliers or collective outliers. We test the algorithms in this paper on 15 different datasets, with different combinations of types of contam- ination. A more thorough explanation of the way that these datasets are generated is given in Lai ctt al. (021b).
The TODS package also readily makes available two real-world datasets (see Appendix A.2. for more information), labelled with outliers relating to the context. Those two datasets are:
SWAN-SF: SWAN- SF was collected by the Harvard Dataverse and contains space weather data (Angryk et al. 2020a). This dataset has contamination factor of 23.8%.
GECCO:
This is dataset collected by SPOTSeven Lab for data challenge in 2018 and contains water quality information. This dataset has a contamination factor of 1.2%.
These datasets were processed by Lai et al. (2021b) before being made available for outlier detec- tion. The code that processes the data is readily available in the TODS package (see Section 6 for information on reproducibility).
Alongside these datasets, we also tested different models to in this paper as the Movement dataset). This data contains movement information, collected from over 140 people living with dementia in their real home settings for a period of over 2 years (from February 2019 to June 2021). Each feature in this dataset represents the number of times sensor was triggered in 3 hour period in a location of the home. To simplify this, we will think of the sensor triggers as a visit to a room containing the sensor. The dataset contains 5 of these locations, which represent the hallway, living room, bedroom, bathroom and kitchen. These 3 hour time windows were then concatenated to form points that cover 24 hour periods and contain 40 features. This means that each of these features represents frequency of visits to location in a particular 3 hour window. The outliers were labelled as points in the data in which the person living with dementia was admitted to hospital or had an infection. The labelling is done by a clinical
dementia care dataset (referred
'https://ithh..com//// http:///hhh..commmaaamm///////es https::/wwww.pposeveee.
3

---

Under review as a conference paper at ICLR 2022
monitoring team who have been part of the study and interact with the participants based the observations and measurements collected by the in-home monitoring technologies integrated into a clinical dashboard. Only the people living with dementia who had at least one labelled outlier, were included in the final dataset. The data was labelled as an outlier 4 days preceding and proceeding the infection and hospital admission episodes. Data points in which all of the feature values were and there was no hospital admíssion or infection were then removed. This leaves the contamination factor of the dataset 2.3%. Graphs showing the distributions of values in the dataset be viewed in Appendix A.2.2
After applying the methods to the dataset, we split the data by each source household containing a person living with dementia) and tested the algorithms on each of these. We removed any sources in which there were less than 5 outliers labelled. We will refer to this dataset as the separated Movement data.
The datasets chosen span wide range of contamination values, allowing the algorithms to be eval- uated over a range of scenarios. They also contain a mix of ratios of point outliers to collective outliers. See Appendix A.2 for more information.
We assumed that all of the datasets either have point outliers or collective outliers. Point outliers are defined by a single outlier on own, whilst collective outliers are defined by consecutive outlier points in time window.
2.2 MODELS
To test what extent recurrent neural networks are more useful detecting outliers within time series data, we have chosen three classical outlier detection methods as well as four recurrent neural networks set up in an outlier detection framework. All of these methods were chosen because their ability to be useful for sequential outlier detection. The majority of the models were implemented using the TODS framework (Lai et al. 2021a), where the pipelines were built by Lai al. (202Tb). This was requirement for us when deciding which algorithms to evaluate we required them to be readily available.
EVALUATED METHODS
The methods include unsupervised techniques and recurrent models. The performance of the meth- ods based on reconstruction errors or the ability separate unexpected patterns. These models use the error between the reconstructed or predicted values and the true values to identify the outliers.
more descriptive list given Appendix A.3
Stationary Methods:
OCSVM: One-class support vector machines (OCSVMs) (Schölkopf et al. 1999).
Isolation Forest: (Liu et al. 2009).
Auto-Encoder: (Sakurada & Yairi 2014
RNNs:
Autoregression: (Roussceuw & Leroy 2005).
GBRT: Gradient boosting regression trees (GBRTs) (Friedman 2001
Elsayed al.
2021).
LSTM-RNN: Long short term memory recurrent neural networks (LSTM-RNNss (Hochreiter & Schmidhuber 1997 Bontemps et al. 2016 Singh 2017).
Self-A Attention: Descriptions of this paper's proposed methods in the following section.
PROPOSED METHODS
To get better understanding of how recurrent neural networks perform several time series datasets, we introduce two new methods based on the transformer model and self-attention mecha- nisms (Vaswani al. 2017).

---

Under review as a conference paper at ICLR 2022
1. Transformer Encoder:
This model uses positional encoding, self-attention and linear lay- ers to calculate the next time point values based on sequence of previous values. The loss between the predicted time point and the correct time point is used to calculate outlier score using a Gaussian distribution.
2. Transformer Encoder-Decoder:: This model uses self-attention encoder to encode sequence, before using self-attention decoder predict the next a time points. The algorithm then calculates the mean loss over each of the predictions and uses a Gaussian distribution to calculate outlier scores. Within our experiments, we x at
Both of the above-mentioned models are a modification of the Long Short Term Memory Recurrent Neural Network (LSTM-RH) (NNN outlier detection method. Instead of using LSTM-RNN (Hochre- iter & Schmidhuber 1997 Bontemps et al. 2016 Singh (017)) as the underlying time series fore- casting model, we transformer models Vaswani al. (017) of two different architectures. A diagram the proposed architecture given in Appendix A.3 Unlike the LSTM-RNN outlier detection method, do focus on collective outliers, and so remove the sliding window which requires collective outliers to be prioritised (see Bontemps et al. (0016) for details).
One of the main draw-backs of LSTM-RNN is the way that it processes sequences. LSTM-RNNS process sequences one point at time. This can cause problems when making predictions, since the model has attempt to extract meaning from an input that was passed through the Long Short Term Memory (LSTM) block as many steps previously as the length of the sequence. Also, because of the way that they sequentially handle inputs, LSTM-RNNY can not parallelised when calculating an output. Transformers aim to tackle this issue, along with introducing the elff-attention unit /Vaswani jet al., 2017). Transformers take entire sequences at once, allowing for all parts of the sequence to be understood by the model. This allows for further temporal relationships to be built. Moreover, because of this, transformers can be parallelised when training. and the encoding section can be par- allelised when making predictions. We believe that transformer based outlier detection technique would be a useful addition when attempting to find outliers in a dataset in which points can have complicated temporal dependencies within sequence.
2.3 EVALUATION
To evaluate the effectiveness of our outlier detection methods the datasets, will mainly use the Fl score, which is defined as a harmonic mean of precision and recall. We will also look at the receiver operating characteristic (ROC) curve to understand the models performance against random classifier.
We have tested each of the methods understand whether Recurrent Neural Network (RNN) meth- ods are required for outlier detection and if so. in which scenarios are they helpful. To answer this question, we have selected several datasets (both synthetic and real-world) that contain different types of outliers and span wide range of contamination values.
We use contamination values of 5%, 10%, 15%, 20% and 25% throughout this paper. These are the values at which the outlier detection algorithm will be evaluated, however, they are not the values of the true numbers of outliers within the dataset.
In addition, the datasets vary in size and complexity (in terms of their underlying mechanics), al- lowing us test the different individual algorithm's performance on collective outliers and point outliers that are global or contextual. We especially interested in evaluating the methods on the Movement data, since this dataset contains complicated temporal relationships that may describe the outliers.
We will run each of the experiments 5 times, with various A.3) to make sure the results reliable. Since in each experiment the model is detecting outliers on a single dataset, we train each of the forecasting models on the same data that we predict outliers
(described in Appendix

---

Under review as a conference paper at ICLR 2022
3 RESULTS
We start by looking the results of the methods applied the synthetic datasets supplied within the TODS package (Lai et al. (021a). Figure Ia saoss the maximum F1 score achieved on each of the synthetic datasets by either non-recurrent or recurrent based outlier detection algorithm. These experiments show that non-recurrent based algorithms perform better (in terms of their FI scores) than recurrent models when applied to synthetic datasets, but only marginally. fact, could be argued that both types of methods produced similar results on these datasets.
The greatest F1 score was 90% and was produced by the One Class Support Vector Machine (OCSVM) model. The median maximum FI score (averaged over the datasets) achieved by current methods was 61% compared to the non-recurrent value of 63%. Figure Ib shows the num- ber of times each algorithm non-recurrent based algorithms performed better than the recurrent models on datasets that were dominated by point outliers, however both performed similarly collective outliers.
the best performance
given dataset. This indicates that the
Distribution Of Maximum Score For Each Synthetic Dataset
Frequancy Of Miximum Prodicting Synthllic Data
Mebrada
Nen-fecument
Frequency
Frequency
Goadeed Sociit Segessoe
Therbore Droo Colodd
One Oast
06
85%
456
65%
85%
1975
F1 Score
(a)
(b)
Figure 1: Figure shows a histogram of the F1 scores that resulted from applying the methods to the different synthetic datasets 5 times (with the best performing contamination level for each model shown). Here, the maximum Fl score for each dataset is plotted for both the recurrent based methods and the non-recurrent based methods. Figure 1b shows the number of times a particular algorithm produced the best F1 score on synthetic dataset. Auto-Endoer, GBRT, Transformer Encoder and Transformer Encoder-De Encoder-Decoder Decoder never showed the best performance on dataset.
When applied to the real-world data supplied within the TODS package LLai al. 2021a), the non- recurrent outlier detection methods performed better. Figures 2a and 2hlshow the maximum Fl score for each the methods applied to the real-world data. In both cases, the lowest performing non- recurrent method performed better than the highest performing recurrent method. When considering the SWAN-SF dataset in particular (Figure 2b). the non-recurrent methods had a greater advantage over the recurrent methods, with the best performing non-recurrent algorithm achieving an Fl score of 57.7% compared to the best performing recurrent algorithm result of 39.2%.
However, the strong advantage the non-recurrent methods had the real-world data is not present when considering the performance of the outlier detection algorithms on the Movement data. The Fl scores and receiver operating characteristic (ROC) curve can be observed Figure 3 Here, the recurrent based algorithms and non-recurrent based algorithms performed similarly. The best result was obtained by a small margin, by the OCSVM model, which achieved a score of 5.77% compared to the best performing recurrent algorithm (Transformer ROC shown Figure 3b| displays the true positive rate against the false positive rate and shows that both types of outlier detection algorithms performed better than random guessing (area of > 0.5 suggests better performance than random guessing), however neither performed confidently better than the other. Isolation Forest achieved the largest under the receiver operating characteristic curve AUC-ROC) with a value of 0.6, which wasn't significantly larger than the greatest AUC-ROC value for the recurrent based algorithms (with value of 0.59, achieved by GBRT).
result of 5.58%. The

---

Under review as a conference paper at ICLR 2022
Quality
Maximum Score On SWAN-SF Data
40%
Han- Roc-Rou Preatet
Racaner
12%
16%
40%
Score
F1 Soore
2016
30%
25%
106
One bok
Ado
Long then ferm Merrod Phe
5%
Tramsbre Chooce Deooce
Trashirel
Thealoomet
Figure 2: Figure 2a shows the average F1 score of the best performing contamination level for
different methods when applied to the Water Quality dataset. Figure (2b shows the average F1 score of the best performing contamination level for the different methods when applied to the SWAN-SF dataset. Both of these datasets were discussed in Section 2.1
Maximum
Score On Movement Data
ROC For Methods On Movement Data
1.8
Mchoos
Non Rucumant
0%
Hecument
F1 Score
0%
04
Trua Positive
2%
0.4
Nemoss
0%
Ore Cac 5000
Grstert Blodting Represton
Long Shor Twen deencey 2AN
Trundomm Encodar
Treaatree Errccde Decode
louador Fomet Jansa ( (.80)
Ona Class BVM (ana =: 0.59)
AnnzEn paax - 0.59)
Auto Regrosshvo farai 0.57;
Grodiott Buceting Rogressor (arco 0.39)
Long Short Torn Memory RNN (ara 56)
Transforer Ercacer jana 0.59)
Transforer Eroucer Decade (scc 0.50)
850
False Positive Rate
(a)
(b)
Figure 3: Figure 3a shows the average Fl scores of each model over runs for the best performing contamination level the Movement data. Figure 3bb shows the ROC curve for these experiments in which the average F1 score was highest over the contamination levels.
The methods were then tested on the Movement data, split by source (each household containing a person living with dementia (PLWD)). Figure 4a shows the area under the ROC curve for each of the best performing recurrent and non-recurrent based algorithms when applied to the separated Movement data. These experiments show that the recurrent models performed significantly better than the non-recurrent methods (in terms of their AUC-ROC values) when applied to each of the households individually. The recurrent based algorithms achieved an average AUC-ROC of 73% compared to the average AUC-ROC of 60% achieved by the non-recurrent methods. We also that all of the recurrent models (that achieved the maximum score on a given household's 's data) achieved an AUC-ROC value of larger than 50%. which is not the case with the non-recurrent based methods. This suggests that the recurrent models were on average always better than a random classifier, but the same true for the non-recurrent methods.

---

Under review as a conference paper at ICLR 2022
Table shows the average maximum FI score each dataset by either recurrent non-recurren based algorithm. We can see here, that the recurrent models were a significant improvement over the non-recurrent methods.
Table 1: Average maximum Fl score on each household's data, achieved by either recurrent or non-recurrent method.
Methods
Average FI Score
Non-Recurrent
12.9%
Recurrent
19.1%
Figure 4b shows the number of times each model scored the largest Fl score on household with a PLWD. Here, it is clear that the recurrent based algorithms have an advantage, with the method producing the largest FI score most often being the LSTM-RNN based outlier detection algorithm.
See Appendix A.4 for small discussion on the effect of sypee-parameters,
These results show that recurrent based outlier detection algorithms make a significant improvement over the non non-recurrent based algorithms when applied to the separated Movement data.
Macimum Scores When Predictng PLW
Distribution Of AUC-ROC For Each PLWD Dataset
Metrods
Noe-Bucarent
Recumet
Frequency
AUC
(a)
(b)
Figure 4: Figure 4a Shows a histogram of values of the area under the ROC curve for each of the best performing recurrent and non-recurrent methods on the separated Movement data. Figure shows the number of times each model obtained the best Fl score when applied to the separated Movement data. For both figures, each data point represents result household's data.
DISCUSSION
From Section 3 we can see that when applied the synthetic data, there were little differences between the performance of the two types of methods where the datasets were dominated by collec- tive outliers. However, when point outliers were also considered, it was clear that the non-recurrent based outlier detection methods performed better. They also had the advantage of having fewer hyper-parameters to tune.
In addition, when applied to the two real-world datasets supplied in the TODS package (Lai et al.) 2021a), non-recurrent based outlier detection methods perform better by achieving higher Fi scores. In the real-world tests on both datasets, the poorest performing non-recurrent methods still scored larger maximum Fl score than the highest performing recurrent model.
On these experiments alone, it could be argued the benefit of applying recurrent models to outlier de- tection is not significant, since they are more complicated, have larger numbers of papper-rrameters, and perform less well than the non-recurrent methods. Although, it is worth noting that with more experiments and changes to the designs the models; such as different hyper parameters, architec- ture changes, different training pipelines, the recurrent based methods could offer improvements.
8

---

Under review as a conference paper at ICLR 2022
However, the results of the methods applied to the Movement Data were interesting, since both out- lier detection methods struggled to produce meaningful results. No algorithm produced an Fl score larger than 6% and the largest AUC-ROC was 60%. This is likely due to the way the dataset is con- structed, since it is a concatenation of the movement data from several different households. Since each household's behaviour will be different, outliers caused by given household will be disguised by another household's s behaviour, unless the point is considered unusual behaviour when consider- ing the entire cohort. This dataset noisy and so the struggle performance is understandable.
On the other hand, when the algorithms were applied to the separated Movement data (another real- world dataset), the recurrent methods performed significantly better. The main differences between the Water Quality and SWAN-SF datasets, and the Movement data is the complexity of the outliers. Outliers in the SWAN-SF and Water Quality datasets are categorised by sudden changes in the series (ngryy et al., 2020b). Outliers in the Movement data are due to an individual's 's daily activities and behaviour and complex temporal relationships which could be subject changes due to various factors such as seasonal changes, wellness and daily events. Figure 4 shows that the recurrent methods had a significant advantage in this scenario. This is likely because in this case, the recurrent based models were able to learn the individual behavioural signatures of the household they were searching for outliers within, enabling them to learn the complex temporal relationships that might make a point outlier.
These experiments suggest that if outliers are hidden in complex temporal relationships, then recur- rent based outlier detection algorithms can help classify them more accurately.
5 CONCLUSION
This paper focused on improving on the work done by Lai et al. (2021b) by asking similar question to Elsayed et al.. (202I). The goal was to understand whether recurrent based outlier detection al- gorithms were significantly better performing than non-recurrent outlier detection based algorithms and in which contexts they can add value. This paper has shown that recurrent based outlier detection algorithms are important when outliers depend complex temporal relationships hidden within the data, as was the case with the separated Movement data. However, when searching for outliers that not defined by these subtle relationships, non-recurrent methods can perform significantly better and are easier to implement.
The advantages of recurrent based methods should be carefully considered before being employed, since the scenarios in which they add value can be difficult to define; for example, how does a user know whether their data has temporal related outliers within their dataset? The models also come with more dyper-paramen attempting to use non-recurrent based methods to see whether the results are satisfactory before introducing more complexity in the form of recurrent neural networks.
tuning and less explain-ability in their results. It is worth first
The recurrent methods for online outlier detection can be trained on a previous subset of the data, but allowed to continue to make predictions as data streams in. This means that the deep learning models could continually learn and to update their underlying forecasting models as the data or environment changes (e.g. to accommodate for seasonal and contextual changes).
would also be useful to understand how to know whether a particular dataset will have complex, non-linear temporal relationships within the data, since it is not obvious when this is the case and how might affect the detection of outliers. This highlights the importance of exploratory analysis in the datasets before designing the models or deciding the dyper-paramedt
selection and training.
would also be interesting to evaluate the transformer based outlier detection models with a sliding window, similar to the LSTM-RNN method as this might improve performance collective outlier detection (where outliers are grouped together) and possibly the Movement data. However, this should be designed carefully avoid overfitting.
In conclusion, the results here demonstrate when recurrent models could be effectively applied to time series outlier detection. We also show that Movement data, transformer based outlier detection models provide an effective way to detect complex temporal dependent outliers and make suggestions for further improvements.

---

Under review as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
These experiments are designed to reproducible and easy to extend. Please see the IPython notebook titled "Testing Recurrent Von-Recurrent", available in the supplementary material. This notebook provides instructions on how to install the TODS package as well as the code required to the experiments. The code itself saves all of the outlier scores for every test, allowing the user to quickly load and study them after running the algorithms.
In addition, this paper provides descriptions of the different algorithms in Appendix A.3 as well as clear details of hyper-parameters used. We also provide clear definitions of the metrics used and how they are calculated. We do not however, give access to the Movement data, since this is sensitive dataset that forms part of wider study within our centre. To this end, we do provide details on some of the statistics of the dataset (that can be viewed in Appendix (...2 as well as the preprocessing done (see Section 2.
REFERENCES
Rafal Angryk, Petrus Martens, Berkay Aydin, Dustin Kempton, Sushant Mahajan, Sunitha Basodi, Azim Ahmadzadeh, Xumin Cai, Soukaina Filali Boubrahimi, Shah Muhammad Hamdi, Micheal Schuh, and Manolis Georgoulis. SWAN-SF, 2020a. URL https://dci. org 10. 7910/ DVN/EBCFKM
Rafal A. Angryk, Petrus C. Martens, Berkay Aydin, Dustin Kempton, Sushant S. Mahajan, Sunitha Basodi, Azim Ahmadzadeh, Xumin Cai, Soukaina Filali Boubrahimi, Shah Muhammad Hamdi, Michael A. Schuh, and Manolis K. Georgoulis. Multivariate time series dataset for space weather data analytics. Scientific Data, 7(1):227, 2020b. doi: 10. 101.3///41157-00000544444 URLhttps.
//doi. 1038 / 020 0548-1
Ane Blázquez-García Angel Conde. Usue Mori, and Jose A. Lozano. A review outlier/anomaly detection in time series data. ACM Comput. Surv. 54(3), April 2021. ISSN 0360-0300, doi:
10.1145/3444690. htt.l:tts:///a/ http:///oooo..o/...ooo/oo 1145/3444690
Loic Bontemps, Van Loi Cao. James McDermott, and Nhien-An Le-Khac. Collective Anomaly
Detection Based on Long Short- Memory Recurrent Neural Networks, pp. 141-152. Springer International Publishing, 2016. doi: 10.1007/978-3-319-2200
URL http://dxx doi.
oxg/10.1007/973 0780000-7778800 - 48057-2
Shereen Elsayed, Daniela Thyssens, Ahmed Rashed, Lars Thiemd,, and Hadi Samer Jomaa. Do we really need deep learning models for time series forecasting?, 2021.
Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5):1189 1232, 2001. doi: 10. 0o.2214/a0/10013233551 URL lntpps://doi.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory, Neural Computation, (8):1735-1780, 11 1997. ISSN 0899-7667. doi: 0..1662/ccc..1197.9887777 URL https:
//doi. 10. /0///////cc.... 1997. 8....735
Kwei-Herng Lai, Daochen Zha, Guanchu Wang, Junjie Xu, Yue Zhao, Devesh Kumar, Yile Chen, Purav Zumkhawaka, Minyang Wan, Diego Martinez, and Xia Hu. Tods: An automated time series outlier detection system, 2021a.
Kwei-Herng Lai, Daochen Zha, Junjie Xu, Yue Zhao, Guanchu Wang, and Xia Hu. Revisiting time series outlier detection: Definitions and benchmarks. In Thirty-fifth Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021b. URL ht
Fei Tony Liu, Kai Ting, and Zhi-Hua Zhou. Isolation forest. pp. 413 422, 01 2009. doi: 10.1109/ ICDM.2008.17.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM Comput. Surv., 54(2), March 2021. ISSN 0360-0300. doi: 10.1145/3439950. URL https:::: //dot 1tte///11111/3399100
10

---

Under review as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vitito Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http: /paperss.ppppppp.ppppppppp wttp://rssseurissccccsssoooo 9015
ibrary.
pdf
P.J. Rousseeuw and A.M. Leroy. Robust Regression and Outlier Detection. Wiley Series in Prob- ability and Statistics. Wiley, 2005. ISBN 9780471725374 URL httpt:ttt///oooooo gotpl/oooooooool....ooooooog.... google.
co. uo.uk/Bin / ?id 735-MWC
Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimen- sionality reduction. In Proceedings of the MLSDA 2014 2nd Workshop on Machine Learn-
ing for Sensory Data Analysis, MLSDA' 14, pp. 4-11, New York, NY. USA, 2014. Associa- tion for Computing Machinery. ISBN 9781450331593. doi: 10.1145/2689746.26897744 URL
https: .rrgorrg/oo //lg/ 1241145//22 11445/2689746.2697772 2689747
Bernhard Schölkopf, Robert Williamson, Alex Smola, John Shawe-Taylor, and John Platt. Support vector method for novelty detection. In Proceedings of the 12th International Conference Neural Information Processing Systems, NIPS' 99, pp. 382-588, Cambridge, MA, USA, 1999.
MIT Press.
Akash Singh. Anomaly detection for temporal data using long short-term memory (lstm). Master's thesis, KTH, School of Information and Communication Technology (ICT), 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N you need. U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gareet (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL / neurieersss pcspccc/pppeer/01//f1e... pdf
Gomez, ukasz Kaiser, and Illia Polosukhin. Attention
Guyon,
A APPENDIX
A.1 METRICS
F1 SCORE
Precision is defined as:
True Positives
Precision
True Positives + False Positives
And is interpreted as the proportion of predicted positives that were actually positive.
Recall is defined
True Positives
Recall True Positives + False Negatives
This is the proportion actual positives that were correctly classified as positive.
Finally, The Fl Score is defined as:
Recall Recall Precision And is interpreted as the harmonic mean of the recall and precision. The larger the Fl score, the better classifier is at classifying positives. The F1 score can take values between 0 and 1.
11

---

Under review as a conference paper at ICLR 2022
Dataset Contamination
Proportion of Point Outliers
Deteeet
Synth
Synthol ell
Frequency
Frequency
0%
10%
20%
50%
40%
52%
0%
20%
40%
60%
80%
100%
Contamination
Proportion Of Point Outlers
(a)
(b)
Figure 5: Figure 5a shows the contamination of the synthetic data, the real-world data and the Movement data. Figure [5]sshws shows the proportion point outliers that are each of these datasets. We assume all of the datasets either have point outliers collective outliers.
COFFFICIENT VARIATION
The coefficient of variation is defined as:
Gp Standard Deviation
Mean
This can take any positive value, with larger value representing a higher variation in the series being tested.
A.2 DATA
histogram showing the distribution of contamination values for the different datasets is given in Figure 5a Similarly, the proportion of point outliers to collective outliers is shown in Figure 5b
A.2.1 TODS PACKAGE REAL WORLD DATA
The TODS package contains four real world datasets, and scripts to load them. Unfortunately. when testing, were only able to successfully perform testing two of these. We believe, along with our own datasets, this is enough to understand the performance of the models.
A.2.2 MOVEMENT DATA
Figure shows the distribution the values the Movement data.
Figure shows the average values in the Movement data, split by the time of day. Each data point that we calculate an outlier score for, is made by flattening an array like this into single vector.
A.3 OUTLIER DETECTION ALGORITHMS
Here, give brief description of each of the outlier detection algorithms that are tested within this paper:
Stationary Methods:
OCSVM: One-Class Support Vector Machines (OCSVMs) (Schölkopf et al. 1999) are unsupervised outlier detection techniques which calculate a binary function that captures where the majority of the data lives. leaving proportion, P. of the data outside of the positive region of the functions output. This proportion, P corresponds the contamination of the outliers in dataset.
12

---

Under review as a conference paper at ICLR 2022
Analysed Portion Of The Movement Data
Valies
12 18
20 Features
Analysed Portion The Movement Data Split By Label
Data Type
Infers
Octiers
Values
Features
(b)
Figure 6: Box plot showing the distributions of values in the Movement data after processing. Figure shows the distribution of the whole dataset, whereas Figure obl shows the distribution split by their label.
Average Frequencies Of Visits To Locations During The Day
Faalay
Kitohon
1.0
1.1
15.8
28.0
28.4
25.0
192
Lourge
1.9
1.8
7.6
9.5
5.2
5.0
4.7
4.8
3.3
2.9
4.5
Bedroom
Hour Of Day
Figure 7: This shows the average frequencies of visits each location over the entire cohort, sepa- rated by time.
Isolation Forest: Isolation Forest (Liu et al. 2009) is an unsupervised outlier detec- tion technique that is built upon the concepts of random forest models. After building
13

---

Under review as a conference paper at ICLR 2022
tree used to segment dataset, point is considered more of an outlier than another it takes less branches in a tree to separate that point.
Auto-Encoder: Using Auto-Encoders for calculating outlier scores (Sakurada & Yairi 2014) relies on mapping points latent space that is used to reconstruct the data. Outliers are assumed to have more reconstructive loss than inliers, and this loss used to score points as anomalies.
RNNs
Autoregression: Autoregressive models assume that each point is linearly correlated to the previous few points (Rousseeuw & Leroy] 2005). In this way, this algorithm can be considered an RNN outlier detection method.
GBRT:
Gradient Boosting Regression Trees (GBRTs) (Friedman) 2001 Elsayed et al.
2021) maps time series sequences into single time points by concatenating them along the feature axis. Regression is then used to predict the next time point and the loss is used derive an outlier score.
LSTM-RNN: Long Short Term Memory Recurrent Neural Networks (LSTM-RNNG
take similar approach to Autoregressive models. They assume that points can be predicted from the previous few points using an LSTM-RNN. The loss between the predicted time point and the actual time point are used to calculate the outlier score (Bontemps et al. 2016 Singh 2017). In this calculation the LSTM-RNN outlier detection method employs a sliding window to check the density of outlier scores and detect collective outliers.
Description was given the Section 2.2 This model was built using the python package Pytorch (Paszke et al., 2019).
:elf-Attention:
For each of the models, we tested various hyper-parametern and display the results of the best per- forming model over 5 repeated tests. These hyper-parameters mostly align with the parameters chosen Lai et al. (021b).
Stationary Methods:
OCSVM: We tried both the subsequence (with window size of 10) and standard im- plementations of the model provided by the TODS framework.
Isolation Forest: We also tried both the subsequence (with window size of 10) and standard implementations of the model provided by the TODS package.
Auto-Encoder: We used feed forward neural networks with sequential layer sizes of (1,4,1), (1,4,8,4,1), (5.32,16, 122,1,116 16.32,5), (22,16,6,,2))
(,16,,2,16,4,1). These were all tested with dropout rate of 0.2, batch size of 32 and trained for 50 epochs. We also tested this method using the subsequence implementation in the TODS package.
RNNs
Autoregression: We tested step size of 1 and window size of 5 and 10.
GBRT: We tested a step size of and window sizes of 3, and 10. We also tested this method using the subsequence implementation in the TODS package on each of the different window sizes.
LSTM-RNN: We tested 1, 2, 5 and 10 layers. For the single layer LSTM-RNN, we tested hidden vector size of 1. For the layer sizes of 2, 5 and 10, we tested hidden vector sizes of 32 and 64. All other parameters were left as the default within the TODS package.
Self-Attention: For both the encoder and the encoder-decoder model, we tested mod- els with 2 and encoder and decoder layers as well with feed-forward network dimensions of 32 and 64. These models were trained for 20 epochs, with a batch size of 30. The encoder only model used 5 data points to forecast the next data point, whereas the encoder-decoder model used 5 data points to forecast the next 3 data points.
A photo of the architecture implemented in the Celf-Attention Encoder-Decoder model is given in
Figure 8
14

---

Under review as a conference paper at ICLR 2022
Trpu: Data
Nolcona Breeding
Transfymer Broader
Peciioall Ereading
Doccdir
Dis riuut
Dutter Scoee
(N, 5)
Forecatted 1N, %
Figure 8: Self-Attention Encoder-Decoder outlier detection architecture.
A.4 EFFECT OF HYPER-PARAMETE PARAMETERS
When testing the algorithms the separated Movement data, also calculated the median efficient of variation (see Appendix A.I) for the Fl scores over each of the hyper parameter tests. This tells how the Fl scores varied when different hyper-parameters were given. Table shows the results. Interestingly, the largest variation in FI score comes from the OCSVM model, and the lowest from the Transformer Encoder-Decodee suggesting that good choices of hyper-parameter are needed for both recurrent and non-recurrent based outlier detection algorithms.
Table 2: Average coefficient of variation for the F1 scores produced by the different model hyper- parameters when applied to the separated Movement data. The smaller, less the hyper parameter choices effected the results.
Methods
Average Cy
One Class SVM Isolation Forest
0.43
0.37
Auto-Encoder
0.33
Reto-Reereesive
0.24
Gradient Boosting Regression
0.27
Long Short Term Memory RNN
0.31
Transformer Encoder
0.26
Transformer Encoder Decoder 0.23
Although these interesting statistics, we were able to all reasonable hyper-parameters and there no way of saying how comparable hyper-parameter changes were across models. For example, changing the OCSVM method from its default settings to subsequence method is not necessarily comparable to changing the layer sizes in Transformer Encoder-Decoder model.
15


Reviews:
[{"review_id": "seApix77Uk_", "reviewer": "Reviewer_FAw1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of this paper are as follows:\n- Time series outlier detection is an important problem as it can be applied to many real-world applications\n- The research question 'do we need recurrent neural networks for time series outlier detection?' is of interest to the community\n- A series of empirical results is presented to answer the question\n\nThe weaknesses include:\n- The work is a rather simple comparative study, performing a few relevant models on some existing datasets without in-depth analysis of the experimental settings and empirical results. As a result, the empirical findings are not convincing. Some major issues here include: 1) there are a number of recent LSTM/RNN-based models that are specifically designed for time series outlier detection, but they are not used in the study; only basic LSTM/RNN models are used instead. 2) the way of constructing or injecting the outliers (as well as the types of outlier) can largely affect the performance of each model, but this is largely ignored in the study. 3) the analysis of the empirical results is so shallow that it cannot provide any insight into the underlying reasons of the obtained results.\n- The crucial concept -- data complexity of time series data in outlier detection -- is not properly defined and quantified. Without a clear definition of data complexity, it is invalid to state whether or not more complex models are needed. The authors claim about the data complexity in terms of the presence of some complex outliers, such as collective outliers, but detailed analysis of this complexity through some ablation studies  is missing. There are many other complexity factors that should be considered, for example, how is the temporal dependence presented in each dataset? does the length of the temporal dependence vary significantly within each dataset? etc.\n- It is unclear whether the used models are capable of detecting collective outliers or not. If the answer is yes to some models, how?\n- No technical novelty is identified. It is a purely comparative study with only some shallow results. The work may be largely improved by including more datasets with natural real-life outliers and performing controlled experiments to justify some specific questions rather than some high-level general questions.", "summary_of_the_paper": "This paper presents a comparative study of the performance of non-recurrent models and deep recurrent models for time series outlier detection. Seven models are evaluated on multiple synthetic and real-world time series datasets. ", "summary_of_the_review": "Overall, the work seems to be a technical report that contains some preliminary results but has many key issues to be further addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "liSYJyoblvq", "reviewer": "Reviewer_aJuJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are many existing non-RNN methods for time series outlier detection which can be found in e.g.\n\nManish Gupta, Jing Gao, Charu C. Aggarwal, Jiawei Han:\nOutlier Detection for Temporal Data: A Survey. IEEE Trans. Knowl. Data Eng. 26(9): 2250-2267 (2014)\n\nThose methods are for both point and collective outliers and should be included in the evaluation study.\n\nThe conclusion is drawn on 3 real-world data sets, one of which (Movement data) seems not publicly available. I find this not sufficient. For comprehensiveness, I would recommend to use existing relevant data sets from UCI Repository and optionally add artificial outliers.\n\n\n\n", "summary_of_the_paper": "The paper compares non-RNN and RNN methods in the outlier detection context, and concludes that non-RNN approach is suited to point outliers while RNN approach is suited to collective outliers. ", "summary_of_the_review": "The technical contribution is incremental. Relevant existing non-RNN methods for outlier detection on time series are not included in the study.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kXwi4MlXryr", "reviewer": "Reviewer_HFSk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work presents numerous experimental results. However, the related literature should be further developed to get a better sense of the different methods existing. Additionally, some experimental clarification would help the reader understand: are the authors using a Monte Carlo cross-validation? Is the contamination repeated in each fold? How is the contamination performed?\n\nMoreover, the results need statistical comparisons to understand if the distributions between recurrent and static models are significantly different (KS test) and also if the ROC are different (DeLong test). The current figures do not show clear performance gain from any method. \n\nThe final message should also be clarified: which method should be used in which context? Do some approaches perform better when lower dimensionality, high colinearities, long time patterns? Is one approach to be prefered if static methods are used. It would also be important to integrate an 'outlier time series' comparison to have a full overview of the use case of the different approaches.\n\nFinally, the proposed approaches do not seem to offer any gain, it might be more interesting to develop on which methods perform better in which context.", "summary_of_the_paper": "This paper studies time series outlier detection. The authors propose to compare recurrent and static outlier detection methodologies on multiple synthetic and real-world temporal datasets. Additionally, they present an extension to the LSTM outlier detection using an attention-based approach. The analysis suggests that non-recurrent methodologies should be prefered as they perform marginally better while requiring fewer parameters and training. Only in the context of complex temporal relations, recurrent approaches should be used.", "summary_of_the_review": "The authors present a substantial experimental work that offers interesting insights on time series outlier detection. However, the paper needs further statistical analysis to better understand the significance of these results. Additional analyses would also help to understand which techniques to use in different contexts.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HVdolXcvl3M", "reviewer": "Reviewer_cSV4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1) Anomaly detection in time series is an important problem\n\nWeaknesses:\n\n1) The evaluation seems incomplete. \n\nDetails:\n\n1) The title/abstract makes you believe that will read some other kind of paper but this is essentially an evaluation paper of anomaly detection methods. Nothing is wrong with that, this is just far from complete. There are dozens of methods to consider. \n\n2) Not sufficient datasets to make such claims. Not sufficient analysis between anomaly types. Many questions of this form: are CNNs better? Are subsequence methods better? Are traditional vs modern approaches better? ....", "summary_of_the_paper": "The paper evaluates a set of methods for time-series anomaly detection, including RNNs among the considered techniques. The evaluation on a set of datasets shows that RNNs perform better than the non-RNN architectures.", "summary_of_the_review": "Preliminary results of an evaluation work for time-series anomaly detection. Motivation has to be reconsidered.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has been reviewed by four experts. Their independent evaluations were consistent, all recommended rejection. I agree with that assessment as this paper is not ready for publication at ICLR in its current form. The reviewers have provided the authors with ample constructive feedback and the authors have been encouraged to consider this feedback if they choose to continue the work on this topic.
Decision: Reject