QUANTILE-LSTM: A ROBUST LSTM FOR ANOMALY DETECTION
Authors: Snehanshu Saha
Soma Dhavala
Jyotirmoy Sarkar
Preyank Bhavesh Mota
Santonu Sarkar
Venue: Submitted to ICLR 2023 (2023)
Keywords: Anomaly
Quantile
LSTM
Activation Function

Abstract: Anomalies refer to departure of systems and devices from their normal behaviour in standard operating conditions. An anomaly in an industrial device can indicate an upcoming failure, often in the temporal direction. In this paper, we make two contributions: 1) we estimate conditional quantiles, and consider three different ways to define anomalies based on the estimated quantiles and 2) use a new
learnable activation function in the popular Long Short Term Memory (LSTM) architecture to model temporal long-range dependency. In particular, we propose Parametrized Elliot Function (Parametric Elliot Function (PEF)) as an activation function inside LSTM, which saturates lately compared to sigmoid and tanh. The proposed algorithms are compared with other well known anomaly detection algorithms, such as Isolation Forest (iForest), Elliptic Envelope, Autoencoder,and modern Deep Learning models such as Deep Autoencoding Gaussian Mixture Model (DAGMM), Generative Adversarial Networks (GAN) etc. The algorithms are evaluated in terms of various performance metrics, such as precision and recall. The algorithms are experimented on multiple industrial timeseries datasets such as Yahoo, AWS, GE, and machine sensor. We have found the LSTM based quantile algorithms are very effective and outperformed the existing algorithms in identifying the anomalies.


Under review as a conference paper at ICLR 2023
QUANTILE-LSTM: A ROBUST LSTM FOR ANOMALY DETECTION IN TIME SERIES DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Anomalies refer to the departure of systems and devices from their normal be- haviour in standard operating conditions. An anomaly in an industrial device can indicate an upcoming failure, often in the temporal direction. In this paper. we contribute to: 1) multiple novel L.STM architectures, q-I.TTM by immers- ing quantile techniques for anomaly detection. 2) new learnable, parameterized activation function Parameterized Elliot Function (PEF) inside LSTM, which sat- urates late compared to its nonparameterized siblings including the sigmoid and tanh, to model temporal long-range dependency. The proposed algorithms are compared with other in terms of performance metries, such as Recall, Precision and Fl-score. Exten- sive experiments on multiple industrial timeseries datasets (Yahoo, AWS, GE, and machine sensors, Numenta and VLDB Benchmark data) and non-time series data show evidence of effectiveness and superior performance of LSLM---ass quantile techniques in identifying anomalies.
anomaly detection algorithms and are evaluated
INTRODUCTION
Anomalies indicate a departure of a system from its normal behaviour. In Industrial systems, they often lead failures. By definition, anomalies rare events. As result, from a Machine Learn- ing standpoint, collecting and classifying anomalies pose significant challenges. For example, when anomaly detection is posed a classification problem, it leads extreme class imbalance (data paucity problem). Mooales-Foreet & Bassetto (2019) have applied semi-supervised neural net- work, combination of an autoencoder and LSTM, detect anomalies in the industrial dataset to mitigate the data paucity problem. [Sperl et al. (2000) also tried to address the data imbalance is- sue of anomaly detection and applied semi sumi-sueervieed method to inspect large amounts of data for anomalies. However, these approaches do not address the problem completely since they still require some labeled data. Our proposed approach is to train models on a normal dataset and device some post-processing techniques to detect anomalies. It implies that the model tries to capture the normal behavior of the industrial device. Hence, no expensive dataset labeling is required. Similar approaches were tried in the past. Autoencoder-based family of models uses some form of thresh- olds to detect anomalies. For example, [Sakurada Axaupae.,Sakurada & Yairi (2014); Jinwon Ch (2015) mostly relied on reconstruction errors. The reconstruction error be considered reconstruction error of a datapoint higher than a threshold, then the datapoint is declared as an anomaly. However, the threshold value can be specific to the domain and the model, and deciding the threshold on the reconstruction error can be cumbersome.
anomaly score. If the
MOTIVATION AND CONTRIBUTION
Unlike the above, our proposed quantile- thresholds applied the quantile-LSTTM are generic and not specific to the domain or dataset. We have introduced multiple versions of the LSTMMbased anomaly detector in this paper, namely (i) quantile-LSTM (ii) iqr-LSTM and (iii) Median-LSTM. All the LSTM versions are based estimating the quantiles instead of the mean behaviour of an industrial device. For example, the median is 50% quantile. Our contributions are three-fold:
(1) Introduction of Quantiles in design of muantile-based LSTM techniques and their application in anomaly identification.
(2) Proposal of the Parameterized Elliot as 'flexible-fomm, adaptive, learnable' activation function

---

Under review as a conference paper at ICLR 2023
in LSTM, where the parameter is learnt from the dataset. We have shown empirically that the mod- ified LSTM architecture with PEF performed better than the Elliot Function (EF) and showed that such behavior might be attributed to the slower saturation rate of PEF. PEF contributes to improved performance in anomaly detection in comparison to its non-parameterized siblings.
(3) Evidence of superior performance of the proposed Long Short Term Memory networks (LSTM) methods over state-of thet (SoTA) deep learning and non-deep learning algorithms across multi- ple Industrial and Non-industrial data sets including Numenta Anomaly Benchmark and the VLDB anomaly benchmark (Appendix, Table 7 9 and (T).
There are three key pieces to modelling anomalies: type of time-series we need to work with; model the temporal dependency and post-process the forecasts to flag that forecast as an anomaly. Given the nature of anomalies, it is obvious they should model the departure normality or the tail behaviour. Quantities are the natural statistical quantities to consider in this respect. The temporal modeling of time-series models is some sort of dynamical systems, including the classical statistical models like ARMA and its variants. LSTMs are the most popular versions of the non- man-paamettrc non-linear dynamical models. One could technically swap LSTMs with any other sequence architectures suit- able for the problem. The added advantage LSTMs brings is the multiplicative gates which help prevent vanishing gradients. This is coupled with the introduction of Parameterized Elliot as activa- tion function (PEF) which shifts the saturation. A classifier to flag anomalies is also comparator, either learnt via supervised task or is based on reasonable heuristics. For the former, we need labels which we assume do not have in large numbers in reality, For the latter, there is no option but to default to some heuristics. But thankfully, with a non non-parametric, non non-linear dynamical system such as Q-L. modelling the quantities, even fixed, deterministic comparators turn out to be adaptive comparators. Therefore, we can consider our contribution as setting this template and making cer- tain sensible choices in each of the three important puzzles of this template. The rest of the paper is organized as follows, The proposal and discussion of various LSTM-based algorithms are presented in section 2 Section describes the LSTM structure and introduces the PEF. This section also ex- plains the intuition behind choosing parameterized version of the AF and better variability due to it. Experimental results are presented in section 4 ddisuusssss discusses relevant literature in anomaly detection. We conclude the paper in section 6
ANOMALY DETECTION WITH QUANTILE LSTMS
Quantiles are used as a robust alternative to classical conditional means in Econometrics and Statis- tics, as they can capture the uncertainty in a prediction and model tail behaviours (Koenker) 2005). The additional benefit lies in quantiles making very few distributional assumptions. It also shown by Tambwekar et al. (202)) that quantiles aid in explainability as they can be used to obtain several univariate summary statisties that can be directly applied existing explanation tools. This served as the motivation behind adopting the idea of quantiles from classification to anomaly detec- tion, as quantiles capture tail behavior succinetly. It is well known that quantiles minimize check loss (Horowitz) 1992). which is a generalized version of Mean Absolute Error (MAE) arising from medians rather than means. It also known that medians are often preferred to means in robust settings, particularly in skewed and heavy-tailed data. Thus, in time series data, where LSTM ar- chitecture has shown beneficial, L.TTM architecture is coupled with the idea of quantiles to capture anomalies (outliers). It is to be noted that this method is applied to univariate time series data only, and the method agnostic to data distribution (see Table ). As the empirical results exhibit, the distributional variance does not impact the prediction quality.
Before we discuss quantile- based anomaly detection, we describe the data structure and processing setup, with some notations. Let us consider I;, = 1,2, 2, ... be the time-series training datapoints. We consider T3 = {z, i = k.--.. . + t} be the set of datapoints, and let Tg be split into disjoint windows with each window of integer size m
and Tj (T..... )h Here, Ty = {Ik+m(j- .... -1) Let (D) be the sample quantile of the datapoints in the set D. The training data consists of, for every Tk. Xx.- (Q- (T))), 1,... as predictors with yk.r = QT(T++1) 1). sample quantile at a future time-step, as the label or response. Let Dr.r be the predicted value by an LST model.
2

---

Under review as a conference paper at ICLR 2023
2.1 VARIOUS QUANTILE-LSTM, ALGORITHMS
A general recipe we are proposing to detect anomalies is to: (i) estimate quantile Q (TR+++11) with r E (0, 1) and (ii) define a statistic that measures the outlier-ness of the data, given the observation 51+1+1- Instead of using global thresholds, thresholds are adaptive i.e. they change at every time- point depending on quantiles.
2.1.1 QUANTILE-LSTM
As the name suggests, in quantile-LSTM, we forecast two quantiles Soow and Dhigh to detect the anomalies present in a dataset. We assume the next quantile values of the time period after sliding the time period by one position are dependent on the quantile values of the current time period.
Time Pwad
The Wadeet ket fight)
The Wilddod Joond Hincc
Thee Windon YL rapap
(a) Anomaly detection process using quantile-I.S LSSM
Peine
(22,0.1
(b) Anomaly detection process using median-LSTI
Figure 1: Sigmoid function has been applied recurrent function, which is applied the outcome of the forget gate (ft = aWW * [hg-1. + by)) as well as input gate (ir = a(W; [/--1, 2x] + bz)). PEF decides the information store cell & PEF(W, [ be).
It is further expected that, nominal range of the data can be gleaned from Hoo and Thigh- Using these Now and ghigh values of the current time windows, we can forecast Now and qhigh values of the next time period after sliding by one position. Here, it is required to build two LSTM models, one for Now (LSTM,tow) and another for Thigh (LSTMghigh). ). Let's take the hypothetical dataset as training set from Figure Ta It has three time windows from time period x2 Zg. Table I defines the three time windows of the time period I ... the time window.
.y By and the corresponding fows thigh values against
TW
Hoow
(high
I1, F2, 43
X1 Qiow (T))
X1.high Qhigh (TH)
F4, 175, T6
22, Qtow (TP)
X2 Qhigh (TT)
I7, T8,
X Qiow (T$)
Xa.high Qhigh (TH)
Table 1: The first time period and its corresponding time windows
The size of the inputs to the LSTM depends the number of time windows and one output. Since three time windows have been considered for a time period in this example, both the LSTM models will have three inputs and one output. For example, the LST predicting the lower quantile,
3

---

Under review as a conference paper at ICLR 2023
would have X1,low, X2,oow Xo,l its puts and V1.tow its output, for one time time-period. A total of n - + +1 instances will be available for training the LSTM models assuming no missing values.
After building the LSTM models, for each time period it predicts the corresponding quantile value and slides one position to the next time period on the test dataset. Quantile-LSTM, applics a fol- lowing anomaly identification approach. If the observed value I*++++1 falls outside of the predicted (Now (high), then the observation will be declared as an anomaly. For example, the observed value 3io will be detected as an anomaly if 2110 ÿ) low or 210 > high- Figure Ta illustrates the anomaly identification technique of the quantile-LSTM, on a hypothetical test dataset.
2.1.2 IQR-LSTM
IQR-LSTM is special case of quantile-LSTTM where Now is 0.25 and (high the 0.75 quantile. In addition, another LSTM model predicts median go.5 as well. Effectively, at every time index k, three predictions are made 0k.0025, 2 0k.0.5. 9k.0 0k.0.75- Based on this, we define the Inter Quartile Range (IQR) Vk.00.5 2k.0.25. Using IQR, the following rule identifies anomaly when 214 0k.0.5 + a(Uk.0.75 Úk,0.25) or D++k+1 < 0k,0.s a(0k,0.75 (k,0.25)
2.1.3 MEDIAN-LSTM
Median-LSTM, unlike quantile-LSTM, does not identify the range of the normal datapoints; rather, based on a single LSTM, distance between the observed value and predicted median (I++*+1 (k.o.s) is computed, as depicted Figure lb and running statistics are computed on this derived data stream. The training set preparation is similar to quantile-LSTM.
To detect the anomalics, Median- LSTM uses an implicit adaptive threshold. It is not reasonable to have a single threshold value for the entire time series dataset when dataset exhibits seasonality and trends. We introduce some notations to make description concrete. Adopting the same conventions introduced before, define dę 21+8+11 - Q0.5
1,2..... n- and partition the difference series into s sets of size t each, i.e., D = Dp. 1, 1,...; where D, = {d; i = (s - 1)t + 1,...,st) st}. After computing the differences and standard deviation (op) are calculated for the individual time period Dp- As a result, Up and op will differ from one time period to another time period. Median-LSTM detects the anomalies
the entire dataset, for every window Dp. mean (1p)
using upper threshold and lower threshold parameters of a particular time period Dp and they are computed follows:
Tollower = Hp + Wopi Tp hiigher Up WOp
anomaly can be flagged for dz E Tр when either d, > highigh or d: < T powwrr Now, what should be the probable value for w If we consider w: = 2, it means that any datapoint beyond two standard deviations away from the mean on either side will be considered as an anomaly. is based on the intuition that differences of the normal datapoints should be close to the mean value, whereas the anomalous differences will be far from the mean value. Hence 95.45% datapoints are within two standard deviations distance from the mean value. It is imperative to consider there is a higher probability of the anomalies falling into the 4.55% datapoints. We can consider within three standard deviations. However, it may miss
2 since
3 too where 99.7% datapoints the border anomalies, which are relatively close to the normal datapoints and only can detect the prominent anomalies. Therefore we have used w 2 across the experiments (See Appendix Kor the characteristics of the proposed methods).
2.2 PROBABILITY BOUND
In this subsection, we analyze different datasets by computing the probability of occurrence of anomalies using the quantile approach. We have considered 0.1, 0.25, 0.75, 0.9, and 0.95 quantiles and computed the probability of anomalies beyond these values, as shown in Table 5 of Appendix A The multivariate datasets are not considered since every feature may follow different quantile threshold. Hence it is not possible to derive a single quantile threshold for all the features. It is evident from Table of Appendix A that the probability of datapoint being anomaly is high if the datapoint's 's quantile value is either higher than 0.9 or lower than 0.1. However, if we increase the threshold to 0.95, the probability becomes 0 across the datasets. This emphasizes that higher quantile threshold does not detect anomalies. It required to identify the appropriate threshold value, and it is apparent from the table that most of the anomalies are nearby 0.9 and 0.1 quantile
4

---

Under review as a conference paper at ICLR 2023
values. Table also demonstrates the different nature of the anomalies present in the datasets. For instance, the anomalies of Yahoo Dataset, to Yahoo Dataset; are present nearby the quantile value 0.9, whereas the anomalies in Yahoo Dataset, to Yahoo Dataset, are close to both quantile values 0.9 and 0.1. Therefore, it is possible to detect anomalies by two extreme quantile values. We can consider these extreme quantile values as higher and lower quantile thresholds and derive lemma. We provide a proof in the Appendix [ The lemma entails the fact that anomalies are trapped out- side the high and low quantile threshold values. The bound is independent of data distribution as quantiles assume nominal distributional characteristics.
3 LSTM WITH PARAMETERIZED ELLIOT ACTIVATION (PEF)
We introduce the novel parameterized Elliot activation function, which is the parameterized version of the Elliot activation function. The suitability of parameterized Elliot function, PEF, conceived as an adaptive variant of usual activation wherein we modify the LSTM architecture by replacing the activation function of the LSTM gates with PEF. We save 'parameter tuning' efforts by learning the parameter values from backpropagation. Additionally, the cost of saturation of standard activation functions impedes training and prediction, which is an important barrier to overcome. We expect PEF to have lower rate of saturation in LSTM compared to other activation functions such as tanh, sigmoid, etc. To the best of our knowledge, insights on 'learning' the parameters of an AF are not available in literature except for the standard smoothness or saturation properties AFs are supposed to possess. is, therefore, worthwhile to investigate the possibilities of learning an AF within a framework architecture that uses the inherent patterns and variances from data.
A single LSTM block is composed of four major components: an input gate, a forget gate, an output gate, and a cell state. We have applied the parameterized Elliot Function (PEF) as activation and introduced a parameter a, which controls the shape of the Elliot function, represented by
f(x) 1 1+ [x]
with the first order derivative of PEF as: f'(x) = ([z|+1)2- The a in equation is learnt during the back-proe pook-propagation like other weight parameters of the LSTM model. There are multiple reasons to implement the PEF instead of other activation functions. Some of the reasons and salient features of the PEF are 1. The function is equal to O, and the derivatives are also equal to at the origin. 2. The function's ' derivative also saturates as the |x| increases. However, the saturation rate is less than other activation functions, such as tangent. 3. One of the major benefits of the PEF is that it further decreases the rate of saturation in comparison to the non- non-parameterized Elliot function.
Parameterized Elliot Function (PEF): One of the major benefits of the Paramterized Elliot func- tion is that further decreases the rate of saturation in comparison to the non- non-parane parameterize Elliot function. We have introduced parameter o, which controls the shape of the Elliot function. After the introduction of the PEF, the hidden state equation is as follows:h, = O,a,,.EEE (.FF,PFF(CC)).. (C2). By chain O₂ * Elliot(C.). After each iteration, the ag is updated by gradient descent BJ
rule, DJ DJ aJ
Ba. dac
She
ac
(n+1) a. + * da.
(See Appendix C for back propagation of LSTM with PEF).
Intuition behind Our Hypothesis: Glorot & Bengio (2010) hypothesized that in neural networks, the logistic layer output softmax(b+WD) might initially rely more on the biases b and hence push the activation value h towards O, thus resulting in error gradients of smaller values. They referred to this as the saturation property of neural networks. This results in slower training and prevents the gradients from propagating backward until the layers close to the input learns. This saturation property is observed in the sigmoid. The sigmoid non-symmetric around zero and obtains smaller error gradients when the sigmoid outputs a value close to O. Similarly, tanh in all layers tends to saturate towards 1, which leads to layer saturation. All the layers attain a partieular value, which is detrimental to the propagation of gradients. However, this issue of attaining saturation would be less pronounced in cases where two different activation functions are used. Since each activation function behaves differently in terms of gradients, i.e., sigmoid outputs are in the range [0. and the gradients are minimum at the maximum and minimum values of the function. Tanh, on the other hand, has minimum gradients at -1 and I and reaches its maximum at O. Therefore, even if the layers begin to saturate to a common value, some of the layers would escape the saturation regime of their activations and would still be able learn essential features. As outcome, this might
5

---

Under review as a conference paper at ICLR 2023
Derivatives Nunctions
00-8
0.5
-0.25
-0.75
finst layer
-1.00
sccand
thind
-1.25
10
20
23.0
1.5
5.5
-25
2.5
30
7.5
200
Epochs
(b) LSTM values for 4 layers and 50 epochs using PEF as activation function using AWS2.
(a) Derivatives comparisons of various activation functions.
1--5
fest, Jayer
90
therd
second, Jayer
Sourth
-3.0
-1.5
firtt layer wecnd Jayer
sird Jayer
Jayer
10
20
50
Number
Epochs
(c) LSTM values for layers and 50 cpochs using Sigmoid as activation function using AWS2.
(d) LSTM values for 4 layers and 50 epochs using Tanh as activation function using AWS2.
Dustect
rrindal
Dirted,
AWIS
Daiaeedyas
1W6X Daisuntyn
F316
03
Valora Theaahh
141
01
Yaloo Datazel
L770
031
Yahoo Datasets
788
05
Yahoo Darseets
1886
0.
Taloo Littaet
1888
03
Yaloo Dateets
1725
0.
Yaloo Dossees
130
02
(e) The final values learn each dataset. We can see the final value is different for different datasets.
T.bl: Diflaae valkes
lictriiingg
Figure 2: The Fig shows the slow saturation rate as well as behavioral comparison of the different layers of LSTM model after the introduction of PEF with other activation functions. It also shows the final value of the learned parameter a on various datasets.
result in fewer instances of vanishing gradients. This assumption would mean that networks with two different activations would learn faster and converge faster to a minima, and the same premise supported by Convergence study (details in section V). As demonstrated by Glorot and Bengio, if the saturation ratio of layers is less pronounced, it leads to better results in terms of accuracy. A standard neural network with N layers is given by h' oW'-' + b) and s' h'- "-''''' + b. Here / is the output of the first hidden layer, a is a non-linear activation function, and b is the 0Cost Now, we find He f' (s(sss WL. @Cost, 8Cest o ds;
bias. We compute the gradients as 0Cast
k. 0.ttr: ZWL m.m
must ensure that the variances the variances of these values. As the network propagates,
equal to keep the information flowing. Essentially, when V(I,"), Varlh!] = Varlh J. it ensures that forward propagation does not saturate, and when V(I, Varl (CCstt = Var @Cost) it ensures Oz!
ayi"
that backward propagation flows at constant rate. Now, what remains is to caleulate these variance values. An elaborate example has been given in the Appendix D
PEF saturation: The derivative of the PEF represented by: o FF2. While the derivatives
of the sigmoid and tanh are dependent on x, PEF is dependent both and Even if (F

---

Under review as a conference paper at ICLR 2023
saturates, the learned parameter will help the PEF escape saturation. The derivatives of the sig- moid, tanh saturate when x' >> 5 or x < -5. However, is not true with PEF as evident from fig 2a As empirical evidence, the layer values for every epoch of the model captured using various activation functions like PEF, sigmoid and tanh. It is observed that, after about 10 epochs, the val- ues of the layers becomes more less constant for sigmoid and tanh (fig and fig 2d). indicating their values have already saturated whereas for PEF, variation can be seen till it reaches 50 epochs (fig 2b). This shows that comparison to sigmoid and tanh as activation functions, PEF escapes saturation due to its learned parameter a. The parameter in PEF changes its value as the model trains over the training dataset while using PEF as the activation function. Since it self training parameter, it returns different values for different datasets at the end of training. These values have been documented table and plotted fig 2e Table demonstrates the variations in a values across multiple datasets as these values get updated.
The outcome of such an approach saves the overhead of tuning the model and also opens up avenues for discovering essential features of not-so-popular AFs. The inherent idea is to consider a 'fixed- form' activation and parameterize it. The parameter is 'learned' via the backpropagation step of the LSTM network such that the shape of the activation, determined by the parameter, is learned from data. Thus, if the dataset changes, so does the final form of the activation. In our case, the fixed form activation is the Elliot Activation function.
EXPERIMENT
In this section, we have evaluated the performance of the quantile-LSTM techniques on multiple datasets. We have identified multiple baseline methods, such as Isolation Forest (iForest), Elliptic envelope, Autoencoder and several deep learning based approaches for comparison purposes (See section 5 for more details on baseline methods).
4.1 DATASETS
The dataset properties have been shown in table A total of 29 datasets, including real industrial datasets and synthetic datasets. have been considered in the experiments. The industrial datasets include Yahoo webscope (Yahoo! 2019), AWS cloudwatch (Lavin Ahmad (015)). GE, etc. There are couple of datasets with either one few anomalies, such as AWS; S1, AW S2. We have injected anomalies in AWS, Yahoo, and GE datasets to produce synthetic data for fair comparison purposes. The datasets are univariate, unimodal or binodal and follow mostly Weibull, Gamma and Log normal distribution. The highest anomaly percentage is 1.47 (GE Datasety), whereas AWS Dataset, has reported the lowest percentage of anomaly i.e. 0.08 (For more details see Table 6 of Appendix, section
4.2 RESULTS
Table demonstrates the performance comparison of various LSTM techniques. Precision and Recall, two performance metrics, are shown in the table. The Median-LSTM, has achieved Recall in most datasets (10 out of 15 datasets). In comparison to existing benchmarks, LSTM methods are SOTA on most of the datasets in terms of Recall. For comparison purposes, we have first compared the Recall. If the Recall is the same for two different methods, then have compared the Precision. The method which has a higher Recall and Precision will be considered as better performer. In AWS datasets, most of the techniques have achieved the highest Recall apart from DAGMM and DevNet. DevNet needs minimum two anomalies hence it is not applicable for AWSI and AWS2. However, as per Precision, iqr-L.TTM has performed better than other methods. In the case of GE1, DevNet has produced better result, whereas quantile based LSTM techniques has outperformed others on GE2. Median-LSTM has demonstrated better result in Ambient temperature. In the case of Yahoo datasets, Median-LSTM has achieved the highest Recall on four datasets; however, quantile- LSTM and LqT-LSTM have produced better results on several datasets. For example, Median-LSTM and Lgr-LSTM both achieved Recall 1 on Yahoo,. However, if we compare the Precision, iqr-LSTM has shown better results. is evident from the table 3 that all these LSTM versions are performing very well on these industrial datasets.
7

---

Under review as a conference paper at ICLR 2023
Patect
Anardd
IPLSTMM
Vctas
Askessate:
GAS
Dolka
iFeed
Endaae
Pacike
Acall
Prodica
Roadi
Pretatin
Real
Pedkae
Racll
Prstatan
Prodase
81181
0031
Asblir
6X0
00099
032
885
LH1
B10
035
0201
699
1001
666
0
28
166
0.0
0054
Viiel
85%
10085
8186
0186
0.75
8335
0235
Yewe?
B196
6.3
151
1383
1354
198
8.49
9.3
1085
1100
B167
Viled
8063
07
0143
04
01
0
0
3061
511
031
36
8000
Yalee
633
0.30
222e
022
837
23
6375
800
834
noer
Table 3: Performance comparison of various quantile LSTM techniques with other state of the art algorithms.
[006)
0:111
0.55
066
A511
0.71
0.17
0.56
661
1:50
691
333
60030
3541
0.1
6.6
0:13
03315
14
0565
0.5
0:55
0006
6364
0328
0300244
2355
516
0.2
0.01
6:36
561
0.5
666
0822
0052
D16
0.12
0.1
0.51
001
0.11
28
0.3
966
0107
6m099
035
0001
0399
4.65
0.5
031
Yiloo
18
037
656
0355
0.58
622
0.0
0.2
Table 4: Performance comparison of various quantile LSTM techniques on synthetic datasets with other state of the algorithms.
Table shows comparison with other baseline algorithms on multiple synthetic datasets. As in the previous table, Recall and Precision have been shown as performance metrics. As per these metries, quantile- based approaches have outperformed iForest and other deep learning based algo- rithms on out of 13 datasets. If consider the Precision alone, the quantile LSTM based tech- niques have demonstrated better performance on 10 synthetic datasets. Our methods outperformed Tree (MGTTree) and Uni-variate Multii-Generaionss Tree (UVMGTree) (Sarkar et al., (022)) as well, in terms of Recall (See Table S of Appendix G There are multiple reasons for the better performance demonstrated by the quantile based LSTM approaches. First is the efficacy of the LSTM, which is well documented. Median-LSTM has de- tected the anomalies for each time period utilizing mean and standard deviation. It also has helped to capture the trend and seasonality. quantile-LSTM do not have any predefined threshold, which has improved their performance. Additionally, the flexibility the parameter of the activation helped in isolating the anomalies. This is evident from Fig 2e which represents the variation in a values of the PEF function across the datasets. a has been initialized to 1.5 for all the datasets. We have also experimented the algorithms on another well known benchmark dataset VLDB Paparrizos et al. (022) (See Appendix M) and observed superior performance median- LSTM over other benchmark methods including LSTM autoencoders where reconstruction loss in the autoencoder is used conjunction with the recurrent structure to flag anomalies Srivastava et al.
tree based anomaly identifiers
determining the shape
2015).
5 RELATED WORK
Literature on quantile based anomaly detection (Tambuwal & Neagu 2021 Solovjovs et al. 2021)
including quantile based Autoencoder approach (Ryu et al. 2022) suggested diversifying the
8

---

Under review as a conference paper at ICLR 2023
source of anomaly score by considering uncertainty term along with reconstruction error in anomaly score computation. However, there is no discussion on setting the appropriate threshold on the anomaly scores. Well known supervised machine learning approaches such as Linear Support Vec-
tor Machines (SVM). Random Forest (RF) and Random Survival Forest (RSF) (Voronov et al. [0188 Verma et al.) 2017) have been explored for fault diagnosis and the life-time prediction of industrial systems. Popular unsupervised approaches such as Anomaly Detection Forest (Sternby et al., 2020),
and k-means based Isolation Forest (Karezmarek et al. 2020) try to isolate the anomalies from the normal dataset. These methods like Anomaly Detection Forest do not require labeled data requires a training phase with a subsample of the dataset under consideration. A wrong selection of the training subsample can cause too many false alarms. Recently, Deep Learning (DL) models based on auto-encoders, long-short term memory (Erfani et al.) [016]] Zhhou & Paffenroth (017) are in- creasingly gaining attention for anomaly detection. Y'n et al. (2020) have proposed an integrated model of Convolutional Neural Network (CNN) and LSTM based auto-encoder for Yahoo Web- scope time-series anomaly detection. For reasons unknown, Yin et al. (2020) have taken only one Yahoo Webscope data to demonstrate their approach's 's efficacy. TA stacked LSTM (Malhotra et al. normal dataset. Hierarchical Temporal Memory (HTM) method has been applied recently on sequential streamed data and compared with other time series forecasting models (Osegi, 202T The authors in (Suuavv itt al. (0188) have performed online timeseries anomaly detection using deep RNN. The incremental retraining of the neural network allows to adopt concept drift across multiple datasets. There are various works (Morales-Frrcee
2013) is used for time series anomaly prediction, and the network is trained
& Bassetto 2019 Sperl et al., 2020), which attempt to address the data imbalance issue of the anomaly datasets since anomalies are very rare and occur occasionally. Hence they propose semi-super approaches but cannot avoid the expensive dataset labeling. Some approaches (Zong et al., 2018) apply predefined thresholds, such as fixed percentile value to detect the anomalies. However, a fixed threshold value may not be equally effective on different domain datasets. Deep Autoencoding Gaussian Mixture Model (DAGMM) is an unsupervised DL based anomaly detection algorithm (Zong et al. 2018), where it utilizes deep autoencoder to gener- ate a oow-dimesiional representation and reconstruction error for each input datapoint and is further fed into Gaussian Mixture Model (GMM). Deviation (eevee))) (Pang et al. 2019) is novel method that harnesses anomaly scoring networks, Z-score based deviation loss and Gaussian prior together to increase efficiency for anomaly detection.
6 DISCUSSION AND CONCLUSION
In this paper, we have proposed multiple versions of the SoTA anomaly detection algorithms along with a forecasting based LSTM method. We have demonstrated that combining the quantile tech- nique with LSTM can be successfully implemented to detect anomalies in industrial and non- industrial datasets (See Table of Appendix D without label availability for training. It has been shown that the performance of the bascline methods is sensitive to the predefined thresholds (See Appendix D. whereas quantile based thresholds are generic. We have also exploited the parameter- ized Elliot activation function and shown anomaly distribution against quantile values, which helps in deciding the quantile anomaly threshold. The design of a flexible form activation, i.e., PEF, also helps in accommodating variance in the unseen data as the shape of the activation is learned from data. PEF, as seen in Table Appendix H captures anomalies better than vanilla Elliot. The statis- tical significance of the performance is highlighted in Table T6] The quantile thresholds are generic and will not differ for different datasets. The proposed techniques have addressed the data imbalance issue and expensive training dataset labeling in anomaly detection. These methods are useful where data is abundant. Traditional deep learning based methods use classical conditional means and as- sume random normal distributions as the underlying structure of data. These assumptions make the methods vulnerable to capturing the uncertainty in prediction and make them incapable of modeling tail behaviours. Quantile in LSTM (for time series data) is robust alternative that we leveraged in isolating anomalies successfully. The distribution-agee aagnosicc behavior of Quantiles tuned out to be a useful tool in modeling tail behavior and detecting anomalies. The proposed methods have a few drawbacks 1. quantile based LSTM techniques are applicable only on univariate datasets. 2. A few of the methods such quantile-LSSTM iqr-LSTM have dependency on multiple thresholds.
9

---

Under review as a conference paper at ICLR 2023
7 ETHICS STATEMENT
The present paper discusses various anomaly detection techniques and mostly performed the exper- iments on publicly available datasets for reproducibility purposes. The paper empirically studies various behaviors of (mostly existing) algorithms on machine/systems data. Since our experiment do not involve human or animal subjects, it is unlikely to introduce any ethical or societal concerns.
REPRODUCIBILITY
We have performed the experiments mostly on the publicly available datasets (except for the GE dataset). We have given the links of the datasets in the main text. Code of the quan- tile based LSTM techniques are available https//amoonooosopopassseeeeeeea D840/README.mdd
REFERENCES
S.M. Erfani, S. Rajasegarar, S. Karunasekera, and C. Leckie. High-dimensional and large-scale anomaly detection using linear one-class svm with deep learning. Pattern Recognit, 58:121-134, 2016.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.). Proceedings the Thirteenth Inter- national Conference on Artificial Intelligence and Statistics, volume of Proceedings of Ma- chine Learning Research, pp- 249-256, Chia Laguna Resort, Sardinia, Italy, 2010. PMLR. URL wwwp://wwedddn..........oo. ngs. m press /v9 orot htmml
L. Horowitz. A smoothed maximum score estimator for the binary response model. Economet- rica, 6003):005-531, 1992. ISSN 00129682, 14680262. URL http::htp//wwtt jstor org/ stable/2951582
A. Jinwon and S. Ch. Variational autoencoder based anomaly detection using reconstruction proba- bility. In Special Lecture on IE, pp. 1-18, 2015.
Karczmarek, A. Kiersztyn, W. Pedrycz, and E. Al. K-means-based isolation forest, Knowledge- Based Systems, 195, 2020.
Roger Koenker. Quantile Regression. Cambridge University Press, 2005.
A. Lavin and S. Ahmad. [dataset] the numenta anomaly benchmark. huttps://githui com/ Inmeen a WWB/tteeeeaatea seassteerr//attaa 2015. accessed on 15/1/2019].
T. F. Liu, M. K. Ting, and Z. H. Zhou. Isolation forest. In 2008 Eighth IEEE International Confer- ence on Data Mining, pp. 273-280, December 2008. doi: 10. 1119/ TCDMM 201 6. 0046
P. Malhotra, I.. Vig, G.G. Shroff, and P. Agarwal. Long short term memory networks for anomaly detection time series. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, pp. 89-94, 2015.
A. Morales-Forero and S. Bassetto. Case study: A semi-supervised methodology for anomaly de- tection and diagnosis. In 2019 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), pp. 1031-1037, 2019.
T. Nakae. Dagmm tf implementation, 2018. URL accessed on 8/2/2022.
E. N. Osegi. Using the hierarchical temporal memory spatial pooler for short-term forecasting of electrical load time series. Applied Computing and Informatics, 7(2):264 278, 2021. ISSN 2634- 1964. doi: http:///do..org/100 1010
10

---

Under review as a conference paper at ICLR 2023
Guansong Pang, Chunhua Shen, and Anton van den Hengel. Deep anomaly detection with deviation networks. In Proceedings of the 25th ACM SICKDD international conference on knowledge discovery & data mining, pp.
2019.
J. Paparrizos, Y. Kang. P. Boniol, R. S. Tsay, and M. Franklin. Tsb-uad: an end-to-end benchmark suite for univariate time-series anomaly detection. In Proceedings of the VLDB Endowment, pp. 1697-1711. ACM, 2022.
Ryu, Yim, J. Seo, Y. Yu, and H. Seo. Quantile autoencoder with abnormality accumulation for anomaly detection of multivariate sensor data. IEEE Access, 10:70428-70439 2022. doi: 10. 10.1109//CSESS.. 202.3188426...
M.
Sakurada and T. Yairi. Anomaly detection using autoencoders with nonlinear dimensionality reduction. In Proceedings of the MLSDA 2014 2nd Workshop on Machine Learning for Sensory
Data Analysis MLSDA 14, pp. 1031-1037, 2014. doi: 0..114//268774226887777
A. Sankar and
E.
Harper.
Anomaly
detection
in network data
using gans, 2021.
URL htttss//jovian..a ai/iann.a/gnn /ttp:///boiaaaaaaattts////////
/thevos/ accessed on
anomaly- detection in-inn--eeen network data www-uuuiinnaaaee 10/2/2022.
gans- 98fc4
J. Sarkar, S. Saha, and S. Sarkar. Efficient anomaly identification in temporal and non-temporal industrial data using tree based approaches. Appl Intell, 2022. doi: https://doi.orr//000 0.1007/ $10489-022 000499022--394000
S. Saurav, P. Malhotra, V. T. N. Gugulothu, L. Vig, P. Agarwal, and G. Shroff. Online anomaly detection with concept drift adaptation using recurrent neural networks. In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data, ser. CoDS- COMAD, pp. 78-87, 2018.
S. Solovjovs, E. Nortey, R. Pometsey, L. Asiedu, S. Iddi, and F. Mettle. Anomaly detection in health insurance claims using bayesian quantile regression. International Journal of Mathematics and Mathematical Sciences, 2021 (475), 2021. doi: 10.1155/2021/666760
P. Sperl, J. P. Schulze, and K. B. ottinger. A3: Activation anomaly analysis. CoRR, 2020. doi: MrXiv:0003.0180111 URL /ttxp://arxiv. wwtpw//.wwrgppp//....ooom http:/220000.001.pppo.o 01801.
N. Srivastava, Mansimov, and R. Salakhutdinov, Unsupervised learning of video representations using Istms. In ICML'15: Proceedings of the 32nd International Conference International Conference on Machine Learning, pp. 843-852 ACM, 2015.
J. Sternby, E. Thormarker, and M. Liljenstam. Anomaly detection forest. In ECAI, 2020.
A. I. Tambuwal and D. Neagu. Deep quantile regression for unsupervised anomaly detection in time-series. SN COMPUT. SCI., 2(475), 2021. doi: 10. 10.1077/42999-00--086-44
A. Tambwekar, A. Maiya, Dhavala, and S. Saha. Estimation and applications of quantiles in deep binary classification. IEEE Transactions on Artificial Intelligence, 3(2):275-286, 2022. doi: 10. 10.1109/AA.20221311155
K. N. Verma, R. Sevakula, and R. Thinukovalluru, Pattern analysis framework with graphical in- dices for condition-baset monitoring. JEEE Transactions on Reliability, 6664):1085-1100) 2017. doi: 10. 1w19/11.1... 2017. 2729465
S. Voronov, E. Frisk, and M. Krysander. Data-driven battery lifetime prediction and confidence estimation for heavy-duty trucks. IEEE Transactions Reliability, 67(2):623-639, 2018. doi:
20.1109//RR.013 2018.28038888
Yahoo!
sandbox. yahoo. com/ catalog. wwwppppppttt.. thppdatttype adia d-70d-0000 2019. accessed on 15/1/2019).
[dataset] computing systems data.
C. Yin, J. Wang S. Zhang, and N. N. Xiong. Anomaly detection based convolutional recurrent autoencoder for iot time series. IEEE Transactions on Systems, Man, and Cybernetics: Systems, pp. 1-11, 2020. doi: 10. 111199tttste 2020 2968811

---

Under review as a conference paper at ICLR 2023
C. Zhou and R. C. Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining KDD, 2017. doi: 15.1145/3097993.3000
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In In- ternational Conference on Learning Representations, 2018. URL bttts::/ het/Forrum31d33311D000
12

---

Under review as a conference paper at ICLR 2023
A
PROBABILITY BOUND
Dataset
P(g 0.95)
P(E 0.9)
P(E 0.75)
P(F 0.25)
PIF 0.10)
AWS Dutaset,
0.01
0.004
Datasetz
Dataset;
0.007
0.0032
Yahoo Dataset,
0
0.014
0.005
Yahoo Dataseta
0
0.105
0.062
0
Yahoo Datasety
0
0.103
0.076
Yahoo Dataset,
0
0.014
0.0055
0
Yahoo Dataset,
0.043
0.016
Yahoo Dataseta
0.028
0.011
Yahoo Dataset,
0.047
0.018
0.0069
Dutasety
0.011
0.004
0.041
Yahoo Dutaseto
0
0.017
0.0069
0.011
0.029
Sensor Dataset,
0
0.0344
0.0135
Sensor Datasety
0
0
0.013
0.033
GE Dataset;
0
0.003
0.002
GE Dataseta
0.05
0.042
AWS Datascisyn;
0.08
0.035
Datasetsyng
Datasetsyns
0.1
0.1
Yahon Dataselsyn;
0.074
0.034
Yahoo DatasetsyD2
0
0.21
0.15
0
Yahoo Dataselsyng
0
0.13
0.11
Yahoo Dataselsyn;
0.08
0.036
Yahoo Dataselsyng
0.025
0.062
Yahoo Datasetsyn;
0.047
0.018
Yahoo Datasetsyng
0.034
Yahoo Datasetsyny
0.017
0.0069
0.034
0.088
Sensor Datusetsyn:
0
0
0.108
0.39
Sensor Datasetsyng
0
0
0.146
0.36
GE DatasetSyn
0
0.017
0.0104
0
GE DatasetSyn2
0
0.11
0.096
Table Various probability values on different quantile threshold parameters (See lemma of the main paper
B LEMMA
Lemma 1:
For an univariate dataset D, the probability of an anomaly P(A) = P(E > anigh) + P(F < alow)) where Ohighs ORow are the higher and lower level quantile thresholds respectively.
13

---

Under review as a conference paper at ICLR 2023
Forget Gate
oiL
Ourput
Impul Gate
PEF
h
Figure 3: The structure of LSTM cell.
Proof. A datapoint declared an anomaly its quantile value is higher than ('high lower than Olow. Here Ohighh. Olow are the higher and lower quantile threshold value. PEE (hiigh is the prob- ability of an anomaly whose quantile value higher than Orhigh- On the other side, P(F < alow is the probability of quantile value of anomalous datapoint lower than Olow. Hence the presence of an anomaly dataset possible if one of the events true. Therefore
P(A) = P(E > Ohhigh UF < (How.
P(A) = P(E anigh) + P(F (low)) - P(E > Othigh ni < (low)
Both the events &, Fare mutually exclusive. Hence the above Equation can be written as P(A) = P(E > ahigg) + P(F < (low)
C BACKPROPAGATION OF LSTM WITH PEF
For backward propagation, it is required to compute the derivatives for all major components of the LSTM. is the cost function and the relationship between i, and hidden state hy is 1, wy +h, + by. The predicted value y' so softmar (v.). The derivative of the hidden state can be shown as follow
aJ JJ Ov.
Th. Dig Thy
JJ JJ H(m, * h4s + bv)
Oh Ov
Th+
JJ
ØJ
wv
Ōh, Dit
The variable involved the output gate is Oz-
0J OJ Oh.
Jo: Th, Dor
0J OJ a(o. PEF(CC))
0o: Thu
So
JJ AJ PEF(C.)
Jo: The
C is the cell state and the chain rule for cell state can be written
DJ OJ ahe ac, Th, aC,
0J value already we have calculated part of hidden state equation.
Th,
Oh, a(o, * PEF((.))
ac
ac,
a0t
(IC, + +12
14

---

Under review as a conference paper at ICLR 2023
After setting the value of th ac
in equation 4
0J
ØJ Q0:
ac,
0h, (1Cgl + 1)2
The chain rule for is
0J 0J ac,
06,
ac, dG,
We need derive only
0Cs 06,
since
0J is available from equation 5. 00,
ac, aife Cc-1 + G * x())
de
dc,
it
After replacing the value of &Cs deh
aJ 0J
*it
dé, ac,
Similar way Da. De,
The following derivatives for input gate
[1a.+111
ŌJ aJ 4
0i. ac,
0J AJ
(ir
(1-))) i()
Da. oc,
For forget gate, below the derivatives
0J 0J
Ct-1
0f:
0C,
0J
DJ
Cr-iff(1- - ft))
daf
ac,
Ze the concatenation of the 1, Tt. The derivatives of the weights are as follow
0J JJ Z,
ow Daz
OJ AJ Z₁
Ow, Ja:
AJ JJ
h
Ow Dt
JJ AJ Zt
Owo dao
C.I PARAMETERIZED ELLIOT FUNCTION
One of the major benefit of the parameterized Elliot function is that it further decreases the rate of saturation comparison to the non-parameteriie Elliot function. We have applied one parameter a, which controls the shape of the Elliot function. There will be different derivatives if we apply parameterize Elliot function LSTM.
PEF
1 1 a
After the introduction of the PEF, the hidden state equation is follow
h4 PEF.CC.
15

---

Under review as a conference paper at ICLR 2023
As per the chain rule, the derivative for de will be
JJ 0J Filiol
do. ah,
da.
DJ
0J
O Elliot(C.)
Da.
ah
After each iteration, the ag is updated as per equation
5
OJ
(a 0 +
(5)
ao.
Similarly, we can derive og and update the parameter.
D INTUITION WITH AN EXAMPLE:
Firstly, we attempt to find variance for two sigmoid activations in a network. The derivative each activation output is approximately 0.25( (0) = 0.25), as the weights uniformly initialized, and the input features are assumed to have the same variance. Hence,
f((s)) = 0.25
Var[= Varl [ar[=]] (a).25) or [W (0.25) V Var(W2")
We see that this diminishing factor of 0.25 steeply drops the variance during the forward pass. Similarly, we observe that the gradient,
0Cost = ('s) k. 0gl+1
aCost
0s
has f'(s(.) one of the factors, and thus the diminishing factor is tied to the variance. Even when N 2 the variance reduces by a factor of 44 256.
Let's compute variance for neural network with two hidden layers using sigmoid and tanh activa- tions. For tanh, if the initial values are uniformly distributed around O, the derivative is f'(xk) ) = 1. Therefore, the variance for the second layer output is, Var[=²] V Var (ar==a)(((00)))))) ((0.25) *Var ( n2 V [ar( We see that the diminishing factor is just 42 = 16, and this results in much better variance when compared to the previous case. Therefore, using different AFs instead the same implies reduction in vanishing gradients and results in much better flow of information because the variance value is preserved for longer.
16

---

Under review as a conference paper at ICLR 2023
E DATASET PROPERTIES
Dutaset Name
Anomaly
Size
Missing Data
Modul
Distribution
#Variables
TW/ Period
Publicly available actual industrial data
AWS Dataset,
No
Unimodal
Weiball
Univariate
84/168
AWS Dutaseta
2486
No
Unimodal
Weibull
Univariate
38/152
AWS Datasety
0.066
1499
No
Multimodal
Weiball
Univariate
38/152
Yahoo Dataset,
0.14
1421
No
Unimodal
Weibull
Univariate
20/60
Yahoo Datasety
0.54
1462
No
Unimodal
Gamma
Univariate
30/90
Yahoo Datuseta
0.55
1440
No
Unimodal
Weibull
Univariate
10/120
Yahoo Dataset,
0.28
1422
No
Bimodal
Weiball
Univariate
105/210
Yahoo Dutasets
0.63
1421
No
Bimodal
Log-nornal
Univariate
24/22
Yahoo Dutaseta
0.28
1421
No
Multimodal
Weibull
Univariate
74/148
Yahoo Dataset,
0.53
1680
No
Unimodal
Weibull
Univariate
125/250
Datasets
Unimodal
Log-normal
Univariate
116/232
Yahoo Dataseta
0.47
1680
No
Unimodal
Weibull
Univariate
Machine Temperature Dataset
0.19
501
No
Multimodal
Weibull
Univariate
38/
Private actual industrial data
GF Datasel;
0.18
1609
No
Unimodal
Exponential
Univariate
117/234
GE Datascto
1.47
544
No
Multimodal
Weibull
Univariate
50/150
Publicly available synthetic industrial data
AWS DatasetSyn,
1.03
1059
No
Unimodal
Exponential
Univariate
84/168
AWS DutasetSyn,
0.877
2506
No
Unimodal
Weibull
Univariate
38/152
AWS DatasetSyn,
1509
No
Unimodal
Gamma
Univariate
38/152
Yahoo DatasetSyn,
1431
No
Univariate
Yahoo DatasetSyny
1.22
1472
No
Unimodal
Gamma
Univariate
30/0
Yahoo DatasetSyng
1.24
1450
No
Unimodal
Weibull
Univariate
10/120
Yahoo DatasetSyn;
0.977
1432
No
Bimodal
Weiball
Univariate
105/210
Yahoo DatasetSym,
1.32
1431
No
Bimodul
Weibull
Univariate
24/72
Yahoo DatasetSya,
0.97
1431
No
Multinodal
Weiball
Univariate
74/148
Yahoo DutasetSym;
1.12
1690
No
Unimodal
Weibull
Univariate
125/250
Yahoo DutasetSyn,
1.18
1690
No
Multimodal
Log-normal
Univariate
116/232
Yahoo DatasetSyng
1.065
1690
No
Bimodal
Weibull
Univariate
3090
synthetic industrial
DatasetSyn;
0.80
1619
No
Unimodal
Log-normal
Univariate
117/232
GE DatasetSyng
3.30
554
No
Multimodal
Exponential
Univariate
5IV150
Table 6: Anomaly Dataset Properties.
F COMPARISON ON NON-INDUSTRIAL DATASETS
Since, we used industrial datasets in the initial comparison with multiple deep learning based anomaly detection techniques, thus to provide further survey, here non-industrial datasets
17

---

Under review as a conference paper at ICLR 2023
(See Table 7. Here, Deviation Networks gives NA because it does not work for single anomaly containing datasets.
Dataset
Anomaly
quantile-LSTM
Autoencoder
GAN
DevNet
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
Trave/Timegs7
0.011
0.67
0.33
0.024
0.33
0.01
0.33
Trave[Timees
1
0.006
0
0.016
NA
NA
Occupancysoos
0.03
0
0.007
NA
Occupancye<00s
2
0.06
0.438
0.5
0.014
0.5
0.02
Speedgoor
1
0.014
1
0.103
0.009
NA
NA
Speed777s
0.086
0.792
0.2
0.9
0.16
0.75
Speedss013
2
0.053
0.75
0.5
0.043
0.1
Table 7: Performance comparison various quantile LSTM techniques on non-industrial datasets.
G COMPARISON WITH TREE BASED ALGORITHMS
In this section, we have compared the quantile-LSTM with other tree based algorithms, such as UVMGTree, MGTree (Sarkar et al. 2022). In table 8 two performance metrics, Precision and Recall have been considered. has been noticed from the same table that quantile-LSTM has produced better Recall, compared with other tree based anomaly identifier algorithms. However, both the tree based anomaly identifier algorithms have better Precision in comparison to quantile- L.STM. Since, Recall has been given higher priority, quantile-LSTTM is competitive enough with other tree based anomaly detection algorithms.
Dataset
UVMGTTree(--eenm
MG7Tree(K-means)
quantile-I.ST
Precision
Recall
Precision
Recall
Precision
Recall
AWS Dataset,
100%
100%
100%
100%
5.2%
AWS Datasela
18%
100%
100%
50%
2.2%
Yahoo Dataset,
15%
100%
50%
50%
3.6%
100%
Yahoo Dataset,
66%
22%
100%
61%
100%
Yahoo Datasets
87%
87%
100%
11%
Yahoo Datasets
33%
33%
100%
33%
10%
Yahoo Datasets
100%
100%
100%
100%
22%
100%
Yahoo Dataset,
77%
63%
50%
27%
16%
63%
Yahoo Datasets
60%
60%
100%
14%
80%
Yahoo Dataseta
100%
100%
100%
62%
33%
100%
Table 8: The performance comparison of UVMGTree with other standard anomaly identifier algo- rithms
18

---

Under review as a conference paper at ICLR 2023
H PERFORMANCE COMPARISON BETWEEN ELLIOT FUNCTION AND PEF AS ACTIVATION FUNCTION
In order to compare performance of Elliot function and parameterized Elliot function activation functions, we experimented them by using them as activation functions in the LSTM layer of the models and comparing the results after they run on multiple datasets. The results are shown in Table
Dataset
Elliot Function
Purameterized Ellior Function
Precision
Recall
Precision
Recall
AWS Dutaset,
0.041
AWS Dataseta
0.002
0.0042
Datasety
0.0181
AWS DatasetSyn;
0.02
0.73
0.909
AWS DatasetSyny
0.39
0.77
0.6875
AWS DatasetSyn;
0.06
0.73
1
Yahoo Dataset;
0.006
0.25
0.0465
Yahoo Datasety
0.02
0.375
Yahoo Datasets
0.05
0.088
Yahoo Dataset,
0.022
0.66
Yahoo Datasets
0.002
0.17
0.0275
Yahoo Dataset,
0.03
0.09
0.066
0.54
Yahoo Datasets
0.017
0.4
0.028
0.3
Yahoo Dataset,
0.43
0.75
0.0208
0.75
Yahoo DatasetSyn,
0.14
0.86
0.375
Yahoo DatasetSyng
0.04
0.72
0.611
Yahoo DatasetSyng
0.78
0.6
DatasetSyn;
0.0625
Yahoo DatasetSyng
0.015
0.69
0.764
0.928
Yahoo DatasetSyn:
0.35
0.43
0.411
0.66
Yahoo DatasetSyns
0.024
0.5
0.197
0.7
Yahoo DatasetSyng
0.27
0.67
0.94
Table Comparison of Precision and Recall score for LSTM with Elliot Function and parameter- ized Elliot Function as Activation Function.
According to the data gathered after running the models, we found that parameterized Elliot function has better Precision and Recall for except four of the datasets. Thus, we could conclude that using parameterized Elliot function as an activation function gave better performance for quantile- LSTM.
I BASELINE METHODS
We have identified multiple baseline methods, such as iForest, Elliptic envelope, Autoencoder and deep learning based approaches for comparison purposes. iForest is isolation-based approach, whereas elliptic envelope is an unsupervised method. Isolation Forest: iForest is unsupervised anomaly detection algorithm Liu et al. (0088 which
19

---

Under review as a conference paper at ICLR 2023
creates a forest of binary trees based random partition. It is ensemble algorithm and the au- thors have shown that is possible to isolate an anomaly with a minimum number of partitions. We have implemented the iForest using sklearn with the default configuration provided in the library. Ellipticc Envelope: : It is another unsupervised approach which relies on a geometric configuration- ellipse to detect the anomalies. It creates an imaginary elliptic area the datasets. The datapoints falling outside this elliptical area are considered as an anomaly. We have applied sklearn package of the python with default parameters to implement this algorithm.
Autoencoder: We have considered Autoencoder for comparison purpose. Like quantile-LSTM, Autoencoder is trained on the normal datapoints. It relies on the reconstruction error to identify any anomaly present in the dataset. We have computed the mean square error (MSE) to measure the reconstruction error. detects the anomalies based on an upper and lower threshold on the MSE. Deep Autoencoding Gaussian Mixture Model (DAGMM): It is an unsupervised DL based anomaly detection algorithm Zong et al.] (2018), where it utilizes a deep autoencoder to generate a low-dimensional representation and reconstruction error for each input datapoint and it is further fed into a Gaussian Mixture Model (GMM). Normal datapoints have been used for training purpose of the model. We have referred the implementation by Nakae Nakae (2018) and the configuration applied for KDDCUP datasets. However, we have implemented upper and lower threshold on the scores instead of percentile based threshold detect the anomalies.
Generative Adversarial Networks (GAN) GANs are DL based generative models that generate samples similar to the training dataset by learning the true data distribution. It is an unsupervised approach and it is designed to find regularities or patterns in the input dataset in such way that it generates output very similar to the original dataset. The model consists of two sub-models, one is generator and another discriminator. The discriminator tries to identify the generated output ei- ther real or fake. We have considered the GAN implementation as a reference Sankar & Harper 2021). Like previous approach, we have applied upper and lower threshold instead of percentile based threshold to detect the anomalies.
Deviation Network (DevNet) This is novel method that hamesses anomaly scoring networks, Z- score based deviation loss and Gaussian prior together to increase efficiency for anomaly detection. In this model first the anomaly scoring network is used get scalar anomaly score. Then the anomaly score is learned by a reference score generator to create a reference score. Lastly the refer- ence score and its standard deviation are put into the deviation loss function optimize the losses in the anomaly scores with reference to the mean of the reference scores.
LSTM AutoEncoder The LSTM autoencoder network architecture uses the first couple of neural network layers to create the compressed representation of the input data, the encoder. A repeat vec- tor layer to distribute the compressed representational vector across the time steps of the decoder then deployed. The final output layer of the decoder provides the reconstructed input data. Increase in reconstruction loss in the autoencoder used to flag anomalies.
Autoencoder, GAN, DAGMM and DevNet all Deep Learning based baseline algorithms considered two thresholds ( upper threshold and lower thresholds) on reconstruction errors or predicted value. The thresholds are the same for all baseline algorithms. We have considered these two thresholds to align the baseline methods with the quantile based approaches. For example, in Autoencoder, upper and lower threshold considered follow:
Upper threshold mean_erron + std srr.error
Lower_thresiie threshold = meean_rroor - 2 std-error
.where meean_error and std_error are the mean and standard deviation of the reconstruction error. Equations 6 and have been applied by all DL based baseline methods.
J F1 COMPARISON ON BENCHMARK DATASETS
Another metric which is very well-known for performance comparison is Fl scores. There are situ- ations, where it is difficult to compare algorithms based on precision and recall. In those cases, Fl scores can be effective to evaluate the algorithms. Table I0 demonstrates the performance compar- ison of the quantile based LSTM techniques with other baseline algorithms in terms of F1 scores. is evident from Table that quantile based LSTM approaches have outperformed others in most of the datasets.
20

---

Under review as a conference paper at ICLR 2023
Dulnset
kF-LSTM
median L.1M
quanile- LSTM
GAN
DAGMM
Autoencoder
AWS Dacaser
0.66
0.8
0.09
0.22
0.86
0.057
AWS Dataset
0.33
0.044
0.088
0.03
0.21
0.16
0.166
AWS Dasaset,
0,071
0.055
0.1
0.066
0,035
Yahoo Damaset
0.14
0.077
0.08
0.125
0.013
0.66
0.16
Yalooo Danseet
0.5
0.88
0.54
0.29
0.14
0.4
0.4
Yahoo Datasets
0.76
0.20
0.16
0.2
0.24
0.42
0,875
0.84
0.42
0.086
Datasety
Danaet,
Yahooo Dataweto
0.42
Dataset,
0.038
0.09
0.06
0.074
0.17
0.20
GE Dataseto
0.8
0.088
Table 10: The F1 measure comparison of quantile based LSTM techniques with other standard anomaly identifier algorithms.
K CHARACTERISTICS OF THE LSTM TECHNIQUES
Here, we will discuss the strength and limitations of the approaches, proposed in this paper.
K.1 QUANTILE-LSTM
The advantages of the quantile-LSTTM are
1. It does not need the entire dataset in memory for anomaly identification.
2. Two thresholds (Upper, Lower) are used in quantile-LSTM and the thresholds are flexible.
3. It has exploited quantile, which is distribution agnostic.
However, it has several disadvantages.
1. This technique needs two LSTM models and two thresholds.
2. A special case of quantile-LSTM, iqr-LSTM requires three LSTM. Building multiple mod- els can be expensive.
K.2 MEDIAN-LSTM
The median-LSTM, has a couple of advantages over other methods. They are listed as follow:
1. It has applied single LSTM model, which computes the median unlike quantile-LSTM, which has applied two LSTM models.
2. median-LSTM does not identify the range of the normal data points; rather, it computes the distance of the observed value from the median.
The weakness of the approach are:
1. median-LSTM expects the entire time series dataset to be present from where it tries to identify the anomalies. In contrast, quantile-LSTM does not need the whole dataset to present. Therefore, median-LSTM may require higher memory for huge number of datapoints.
2. In the case of quantile-LSTM, the upper and lower quantile values may differ for different datasets. However, it is always 0.5 quantile for median-LSTT LLSM inrespective the datasets. Hence it is not that flexible like other approaches.
21

---

Under review as a conference paper at ICLR 2023
L VARYING THRESHOLDS
Autoencoder, GAN, DAGMM and DevNet all Deep Learning based baseline algorithms considered two thresholds upper threshold and lower thresholds) on reconstruction errors predicted value. To understand the impact of different thresholds on the performance, we have considered three baseline algorithms GAN, Autoencoder and Devnet. The baseline methods have considered five different sets of threshold values for upper and lower thresholds. The sets are shown in column head of tables [ 12 and 13 where the first threshold the upper percentile/value and the second threshold is the lower percentic/valuue
GAN
99.25 and 0.75
99.75
0.25
99.9 and 0.1
1:5 std
std
Dataset
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
Yahoo Datasct;
0.09
0.25
0.5
1
0.0086
0.007
1
Yahoo Dataset,
0.348
0,333
0.375
0.4
0.25
0.037
0.875
0.013
Yahoo Datasety
0.28
0.5
0.444
0.286
0.28
0.5
0.0543
0.5
0.0322
0.875
Yahoo Datasct,
0.007
0.007
0.007
Yahoo Datasct,
0.18
0.4
0
Yahoo Datasct,
0.154
0.364
0.3
0.273
0.5
0.182
0.041
0.63
0.04
0.63
Yahoo Datasets
0.038
0.1
0.1
0.1
0,25
0.1
0
0
Yahoo Dataseto
0.192
0.625
0.5
0.625
0.5
0.25
0.0052
0.375
Table 11: Comparison of Precision and Recall score for GAN with varying thresholds for anomaly Upper Bound and Lower Bound(as mentioned in each column head).
Autoencoders
99.25 and 0.75
99.75
0.25
and 0.1
std
std
Dataset
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
Yahoo Datasct;
0.5
0.07
0.5
0.036
0.5
0.019
0.0069
0.5
0.0099
1
Yahoo Dataset,
0.5
0.4
0,333
0.5
0.2
0.5
0.25
0.25
Yahoo Datasety
0.44
0.5
0.4
0.5
0.25
0.333
0.7
0.875
0.122
0.875
Yahoo Dataset,
0.13
0.375
0.33
0.6
0.042
0.33
0.035
0.33
Yahoo Datasct,
0.18
0.33
0.5
0.5
0.023
0.022
Yahoo Dataset,
0.5
0.5
0.5
0.5
0.5
0.5
0.043
0.36
0.032
0.45
Yahoo Datasets
0
0
0.25
0.1
0.008
0.1
0.004
0.1
Yahoo Dataseto
0.18
0.625
0.5
0.625
0.33
0.125
0.5
0.12
0.625
Table 12: Comparison of Precision and Recall score for Autoencoders with varying thresholds for anomaly Upper Bound and Lower Bound(as mentioned in each column head).
evident from the above tables that performance vary significantly based on the thresholds de- cided by the algorithm. Therefore it is very important to decide correct threshold which can identify all the probable anomalies from the dataset. Most of the cases, it is not feasible to know the appropriate threshold before applying the baseline algorithms.
M VLDB
Using the VLDB Benchmark, generated timeseries dataset of 5500 datapoints containing 40 anomalies. We used various Deep Learining based algorithms on this generated non-industrial dataset. From table [4] can clearly LSTM) works almost as well as GAN and far better than the other two. While in terms precision Median-L -SMM gives the best values, thus performing better than the other algorithms overall.
that in terms of recall our proposed
(eerithm(Meeiann
22

---

Under review as a conference paper at ICLR 2023
Devnet
99.25 and 0.75
99.75 and 0,25
99.9 and 0.1
mean =:1.3 std
mean +:1
std
Dataset
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
Yahoo Dataset,
0.002
0.002
0.001
0.004
Yahoo Dataset,
0.005
0.005
0.004
0.4
Yahoo Dutaset,
0.0078
0.0078
0.0078
0.727
0.0354
0.909
Yahoo Datasets
0.111
0.5
0.333
0.5
0.333
0.5
0.0279
0.0153
Yahoo Dutaseta
0.167
0,5
0.5
0,667
0,333
0.0347
1
Yahoo Dutaset,
0.054
0.125
0.2
0.25
0.2
0.033
0.571
0.0106
0.714
Dutasels
Yahoo Datasct,
0
Table 13: Comparison of Precision and Recall score for Devnet with varying thresholds for anomaly Upper Bound and Lower Bound(as mentioned in each column head).
VLDB Dataset
Precision
Recall
Median-LSTM
0.513
0.976
GAN
0.0072
Autoencoders
0.013
0.025
Devnet
0.0357
0.158
LSTM-Autoencoddrrs
0.064
0.143
Table 14: Comparison of Precision and Recall score for VLDB generated dataset for various deep learning anomaly detection techniques.
VLDB Dataset
F1-score
Median-LSTM
0.673
GAN
0.0143
Autoencoders
0.0171
Devnet
0.0582
LS SSTM-Autoencoders
0.0884
Table 15: Comparison of Fl-score for VLDB generated dataset for various deep learning anomaly detection techniques.
23

---

Under review as a conference paper at ICLR 2023
N TABLES
We have experimented each of the algorithms on AWS datasets times and mean(() and std deviation(ar) of the results have been computed and shown Table T6 The superior results are marked bold. Iqr-LSTM (a variant of quantile-SSTM) has outperformed others in AWSI and AWS3.
Dosavet
iqp-LSTM
GAN
Astomeoder
DAGNM
Procisionly-
Raaally
Procisionty- +h))
Recallije +
Precisionlp
Recallyy + e)
Prodisionty o)
Recallyp
AWS Dulaset:
875.0.216
11.
0.06110.000 0o8
110
0.02110.0111
1:0
0.00100004
0.2510.43
Dateset
0.126-0.007
0.16:0.0275
0.14:0.076
0
0.0425-0.008
Datesel,
0.099=0.000
Table 16: Statistical analysis of the algorithms on AWS datasets.
Dataset
after training
a initial value
AWS Datasetj
1.612
0.1
AWS Datasetz
0.895
0.1
AWS Dataset;
1.554
0.1
AWS DatasetSyn;
1.537
0.1
AWS DatasetSyng
0.680
0.1
AWS DatasetSyng
1,516
0.1
Yahoo Dataset
1,432
0.1
Yahoo Dataset2
1,470
0.1
Yahoo Datasety
1.658
0.1
Yahoo Datasets
1,686
0.1
Yahoo Datasets
1.698
0.1
Yahoo Dataset,
1.725
0.1
Yahoo Datasets
1,850
0.1
Yahoo Datasetg
1.640
0.1
Table 17: Different values for each Dataset after the training.
24


Reviews:
[{"review_id": "pILJPKBxbb", "reviewer": "Reviewer_ZaFe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on LSTM architectures for time-series anomaly detection. In particular, the paper proposes different solutions for taking into consideration conditional quantities, which help in the identification of anomalies. In addition, instead of using existing activation functions, the work proposes to use a learnable/parametrized function. A comparison against multiple baselines and datasets demonstrates the efficacy of the proposed solutions.", "summary_of_the_review": "Check above for the reasoning. The paper can have a significant impact if these issues are improved but for now seems somewhat as a rushed/preliminary work", "strength_and_weaknesses": "Strengths\n\n1. Anomaly detection over time-series data is a timely, important, and well-studied problem\n2. Well-written and easy-to-follow - well motivated\n3. Experimental results support the claims in the paper\n\nWeaknesses\n\n1. The focus on LSTMs is not justified\n2. Missing relevant baselines\n3. Missing relevant datasets\n4. Missing relevant evaluation measures to assess accuracy\n\nComments:\n\nThe paper focuses on LSTM without sufficient justification. Why? Is there proof that LSTMs is the best for this task? A strong argument is needed in that direction and likely experimental results to support this claim\n\nThe parametrization of the activation function is interesting. It would be better though if evaluated across different architectures/settings to understand the true impact and if indeed this is the reason for the improved performance\n\nThe work unfortunately misses many recent advances in the area, which as a result make the comparisons/evaluations somewhat obsolete.\n\nFor example, TSB-UAD is a new benchmark in that space. VUS is a new family of measures for evaluating time-series anomaly detection methods. The work should perform an evaluation using the benchmarking datasets, the baselines to demonstrate that indeed LSTMs is the way to go, and use the new evaluation measures which solve many flaws in the current evaluations.\n\n\"Tsb-uad: an end-to-end benchmark suite for univariate time-series anomaly detection.\" Proceedings of the VLDB Endowment 15.8 (2022): 1697-1711.\n\n\"Volume under the surface: a new accuracy evaluation measure for time-series anomaly detection.\" Proceedings of the VLDB Endowment 15.11 (2022): 2774-2787.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ksPv53Bx94", "reviewer": "Reviewer_Qabh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper combines ideas in quantile regression and anomaly detection with LSTM networks. The authors further propose a new activation function as an alternative to more established activation functions for LSTMs and run a range of experiments comparing their newly proposed method to other approaches for anomaly detection.", "summary_of_the_review": "Most of the individual contributions of the paper seem rather marginal and disconnected, wherefore they don’t add up to an altogether interesting contribution. I am willing to change my mind if the authors are able to make a concrete proposal on what method they think is best for specific anomaly detection problems.", "strength_and_weaknesses": "The main ideas of the paper are intuitive and clear, but the authors could do a better job tying the paper together. There is a) no clear explanation of how the two main ideas of the paper are connected and b) a proper discussion of the strengths and weaknesses of the various proposed alternatives.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iib850y4RB", "reviewer": "Reviewer_bCty", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an algorithm to estimate conditional quantiles for time-series predictions, and proposes a new parameterized Elliot activation function in LSTM gates. The conditional quantiles in predictions are used to identify anomalies.", "summary_of_the_review": "The paper lacks literature review in the specific domain (quantile based anomaly detection) and the experiments have missed out comparison among different activation functions.", "strength_and_weaknesses": "1. There should be a set of experiments that compare the anomaly detection performance of different activation functions since the paper proposes an activation function (PEF) as a novel contribution. This seems to be missing.\n\n\n2. Section 4.2: \"For comparison purposes, we have first compared the Recall.\" -- This is not the correct or standard practice. An algorithm that simply reports every data point as an anomaly will have recall 1 and beat all other algorithms. The paper should instead compare algorithms on the basis of F1 scores which is a combination of recall and precision.\n\n\n3. Table 3, 4: The text in these tables is too small and hard to interpret. Should highlight best algorithms as per statistical significance analysis. Also, need to show std. errors.\n\n\n4. Section 5 Related Work: We need more discussion on existing quantile-regression based anomaly detection. A simple search on Google Scholar for \"anomaly detection quantile\" comes up with existing published papers. e.g. [r1] which also works with multivariate data, unlike the proposed algorithm in this paper. Relevant works (more than one if applicable) should be treated as additional benchmarks to be compared with in experiments. The paper should clearly explain what is the novel contribution in the current work in light of the existing literature. Note that contribution in point 1 of the abstract \"... estimate conditional quantiles ...\" is not novel. And the evidence for the effectiveness of contribution point 2 of the abstract is not strongly convincing.\n\n\nReferences:\n[r1] Tambuwal, Ahmad Idris, and Daniel Neagu. \"Deep Quantile Regression for Unsupervised Anomaly Detection in Time-Series.\" SN Computer Science 2.6 (2021): 1-16. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PXV8x3_WOG", "reviewer": "Reviewer_9RXE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes three different approaches to estimating quantiles from time series data using long short-term memory networks (LSTMs). In addition, a parameterized Elliott function (PEF) is proposed and used as an activation function in each of the quantile-based LSTM variants. When compared to several industrial time series datasets, quantile-based LSTMs outperform several conventional anomaly detection methods as well as deep learning models for anomaly detection.", "summary_of_the_review": "Overall, this paper proposes a simple yet effective approach to anomaly detection that seems to outperform several state-of-the-art anomaly detection baselines. The work lacks methodological novelty and is rather incremental in that respect. Note that this does not diminish the importance of the performance improvements introduced by the proposed quantile-based approaches across several anomaly detection datasets. That being said, considering: \n\n(1) the degree to which this work is application-oriented rather than a work that makes novel methodological contributions in representation learning, and \n\n(2) the absence of a comparison of the proposed approaches to variants of the baselines run with different thresholds; \n\nI am not convinced that this work is a good fit for a venue such as ICLR. Nevertheless, I am looking forward to the authors’ response and I would be willing to adjust my score in case I have misunderstood or misinterpreted certain aspects of the work.", "strength_and_weaknesses": "Strengths:\n\n* This work is simple yet effective. Making use of simple statistics such as quantiles, the authors devise three LSTM-based architectures that exhibit state-of-the-art anomaly detection performance when compared to both well-established anomaly detection methods as well as anomaly detection approaches based on deep learning.\n\n* The paper is well motivated by the distribution-agnostic properties of quantiles and their suitability for capturing tail behavior.\n\n* One of the main advantages of the proposed approaches is their advantage over threshold-based anomaly detection methods. Namely, many anomaly detection methods rely on global thresholds for deciding whether a data point is anomalous or not, which are specific to the domain and the model being used (e.g., autoencoder-based approaches require determining a specific threshold for their reconstruction errors); and thus determining reasonable thresholds may be rather difficult. In contrast, relying on quantiles, the proposed approaches make use of adaptive, domain-invariant and dataset-invariant thresholds that do not need to be set beforehand.\n\n* The proposed quantile-based approaches make no assumption about the data distribution. This is supported empirically by showing that the distributional variance does not impact the predictive performance of the approaches.\n\n* The proposed PEF activation function satures slower/later than other activation functions such as sigmoid and tanh. Its saturation ratio is less pronounced and the authors demonstrate that such an activation function leads to improved anomaly detection performance.\n\n* The scaling parameter $\\alpha$ in PEF is learned automatically from data which allows for flexibility in determining the shape of PEF that is better suited for isolating anomalies. This is supported by the observation that different $\\alpha$ values have been learned and have shown to be beneficial for the different datasets used in the experiments.\n\n* The authors have conducted relatively thorough experiments to assess the effectiveness of the proposed approaches in comparison with the baseline methods.\n\n* The paper is clear and fairly well-written. The used notation is easy-to-follow and consistent throughout the entire paper.\n\n-------------------------------------------------------------------------------------------------\n\nWeaknesses:\n\n* Due to the large number of parameters that can be introduced, LSTM networks are arguably not always well-suited for (very) long sequences. In that regard, I am wondering if the proposed approaches are limited to short time series. I would encourage the authors to elaborate on this point in their response. Also, including the length of the times series for each of the datasets in the paper (or in the Appendix) would be beneficial.\n\n* It is not clear whether the autoencoder baseline leverages LSTMs as encoder and decoder networks or regular feed-forward neural networks. If the latter is the case, I am wondering why the authors did not consider comparing to such a relevant baselines as an LSTM autoencoder for anomaly detection. I would like to ask if the authors can clarify this in their response as well as in the paper.\n\n* Towards the end of Section 2.1.3, the authors discuss their choice of $\\omega$. Namely, the authors claim that, when $\\omega=2$, 95.45% of the data points are within two standard deviations distance from the mean value. Nevertheless, I am wondering how this parameter can be determined without making any assumptions about the data distribution? For example, for a certain dataset, the claimed 95.45% of the data points might fall within two standard deviations of the mean, but this might not be the case for another dataset. Thus, I would encourage the authors to include more specific details on the choice of $\\omega$ and how this choice depends on the underlying distribution of the dataset at hand.\n\n* Lemma 1 seems to be self-evident. Moreover, its proof appears to be rather trivial. Therefore, I am wondering if this lemma is necessary to be a part of the main paper. Moreover, I believe that this lemma would hold for “outliers”, not necessarily for “anomalies”, as data points that are not within a certain range of quantiles are typically referred to as “outliers” which can be data points that are distant from the mean or location of a distribution but not necessarily represent abnormal behavior. On the other hand, the notion that anomalies can also be declared by means of lower and higher quantiles is essentially the hypothesis made in this paper. It would be appreciated if the authors can provide some clarity on this point in their response.\n\n* To assess the anomaly detection performance of the proposed approaches and the baselines, recall and precision have been calculated. Although the proposed quantile-based approaches are threshold-invariant, recall and precision must have been calculated at a particular threshold for the threshold-dependent baselines. Were different thresholds determined for each of these baselines or the same threshold was applied across all of them? I would suggest that the authors include in the paper the threshold(s) that was/were used for each of the datasets. Lastly, I believe that this work would have benefited from an analysis of the effect that different thresholds have on the performance of the baselines. Such an analysis would have verified whether the performance improvements hold in case different thresholds are applied to the baselines.\n\n-------------------------------------------------------------------------------------------------\n\nMinor weaknesses:\nThere are also certain grammatical and typographical errors and remarks that require attention. Some of them are summarized as follows:\n- Page 2: “Our contributions are three folds” should be replaced with “Our contributions are three-fold”.\n- Usage of informal language: “Thus, in time series data, where LSTM architecture has shown to come in handy”. In this sentence, consider replacing “come in handy” with “useful”, “practical” or “beneficial”.\n- The second sentence on page 4 starts with “quantile-LSTM” which is not capitalized. Same applies to the first sentence in Section 2.1.2.\n- A full stop is missing at the end of Section 2.1.2.\n- Page 4, Section 2.1.3: “After computing the differences on the entire dataset, for every window $D_p$, mean ($\\mu_p$) and standard deviation ($\\sigma_p$) for the individual time period $D_p$.” —> In this sentence, the phrase “are calculated” should be inserted between “($\\sigma_p$)” and “for”.\n- Table 2 is included as a figure whereas it should be included as a table using the {table} environment.\n- In the caption of Figure 3, “learn” should be replaced with “learnt” or “learned”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject