Time Series Anomaly Detection via Hypothesis Testing for Dynamical Systems
Authors: Haowei He
Jingzhao Zhang
Yanan Wang
Benben Jiang
Shaobo Huang
Chen Wang
Yang Zhang
Xuebing Han
Dongxu Guo
Guannan He
Minggao Ouyang
Venue: Submitted to ICLR 2023 (2023)
Keywords: anomaly detection
dynamical system
hypothesis testing

Abstract: Real world systems---such as robots,  weather, energy systems and stock markets---are complicated and high-dimensional. Hence, without prior knowledge of the system dynamics, detecting or forecasting abnormal events from the sequential observations of the system is challenging. In this work, we address the problem caused by high-dimensionality via viewing time series anomaly detection as hypothesis testing on dynamical systems. This perspective can avoid the dimension of the problem from increasing linearly with time horizon, and naturally leads to a novel anomaly detection model, termed as DyAD (Dynamical system Anomaly Detection). Furthermore, as existing time-series anomaly detection algorithms are usually evaluated on relatively small datasets, we released a large-scale one on detecting battery failures in electric vehicles. We benchmarked several popular algorithms on both public datasets and our released new dataset. Our experiments demonstrated that our proposed model achieves  state-of-the-art results.

Under review as a conference paper at ICLR 2023
TIME SERIES ANOMALY DETECTION VIA HYPOTHESIS TESTING FOR DYNAMICAL SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
Real world systems- as robots, weather, energy systems and stock markets- are complicated and high-dimensional. Hence, without prior knowledge of the system dynamics, detecting forecasting abnormal events from the sequential observations of the system is challenging. In this work, we address the problem caused by high-dimensionallty via viewing time series anomaly detection as hy- pothesis testing on dynamical systems. This perspective can avoid the dimension of the problem from increasing linearly with time horizon, and naturally leads to a novel anomaly detection model, termed DyAD (Dynamical system Anomaly Detection). Furthermore, as existing time-series anomaly detection algorithms are usually evaluated relatively small datasets, released a large-scale one detecting battery failures in electric vehicles. We benchmarked several popular algorithms on both public datasets and our released new dataset. Our experiments demonstrated that our proposed model achieves state-of-the-an results.
INTRODUCTION
Hypothesis testing aims to decide whether the observed data supports or rejects a default belief known as the null hypothesis. Applications are abundant. In this work, we view anomaly detection as an application of hypothesis testing. This perspective is nothing profound proplund-samples from the null hy- pothesis can be viewed as in-distribution, and rejection can be viewed as detecting anomalies. Despite being rather straightforward, this view was not carefully investigated in large-scale anomaly detection tasks, because most classical hypothesis testing methods suffer from the curse of dimensionality.
In this work, we address the problem incurred by high-dimensionality via focusing on time series data collected from unknown dynamical systems. We exploit the structure dynamical systems and show that although the time series data can be high dimensional due to the long time horizon, the problem still remains tractable. More specifically, the concentration that leads to statistical confidence does not come from independent variables but from martingales. We turn the high dimensionality caused by the long time horizon into our favor. Furthermore, our analysis leads to a detection procedure in which the anomaly in systems (e.g., errors and attacks) be isolated from the rarity of system input (c.g., control commands), and hence reduces misclassification rates.
By combining the above analysis with autoencoder-be based probabilistic models, we develop new model termed DyAD (DYnamical system Anomaly Detection). We show that the -motvvedo DyAD model can achieve state-of-the-art performances on public datasets including MSL (Mars Science Laboratory rover) (Hundman et al. 2018) and SMAP (Soil Moisture Active Passive satel- lite) (O'Neill et al. 2010). To further validate our finding, we then release a much larger (roughly 50 times in terms of data points) dataset to benchmark several popular baselines.
Our released dataset focuses on the battery safety problem in electric vehicles. recent years, electric vehicle (EV) adoption rates increased exponentially due to their environmental friendliness, improved cruise range and reduced costs brought by onboard lithium batteries (Schmuch et al.) 2018 Mauler et al. 2021). Yet, large-scale battery deployment can lead unexpected fire incidents and product recalls (Deng et al. 2018). Hence, accurately evaluating the health status of EV batteries is crucial to the safety of drivers and passengers. To promote research in this field, release dataset collected from 301 electric vehicles recorded over months to years. Only battery-relater data at charging stations released for anonymity purposes. 50 of the 301 vehicles eventually suffered from battery

---

Under review as a conference paper at ICLR 2023
failure. Experiments the EV battery dataset confirm that proposed model achieves better performance for system anomaly detection.
In summary, contributions are:
We formulate hypothesis testing based on data observed from dynamical systems and derive generalized likelihood ratio test that exploits the Markovian structure of observations from dynamical systems.
We show that the above formulation leads novel model, termed DyAD, for anomaly detection on dynamical systems.
We release a large dataset collected from 301 electric vehicles, out of which 50 suffered from battery failure. In addition to benchmarking anomaly detection algorithms, the dataset may be of independent interest for machine learning tasks in nonlinear systems.
RELATED WORKS
2.1 ANOMALY DETECTION AND OUT-OF-DISTRIBUTION DETECTION
The difference between -iittiiiiiiittoonnn detection (OOD) and anomaly detection (AD) subtle. Up to the authors' knowledge, anomaly detection, as compared to OOD detection, refers to identifying samples that differ more drastically but rarely from the in-distribution samples. However, the mathematical formulations for the two problems are the same, and hence we use the two terms interchangeably within the context of this manuscript.
The core idea of AD is to develop metric that differs drastically between normal and abnor- mal samples. Some previous works find that the model output probability for normal samples is higher (Hendrycks & Gimpel| 2016 Golan & El-Yaniv 2018) in image tasks. Some previous works focus on detecting anomalies in the feature space by Norcing/assuming the feature concentration of normal samples (Schölkopf et al. [999; Lee et al. 2018). Some enhance the representation power of networks by introducing contrastive learning [Winkens et al., 2020 Taak et al.) 2020) and data transformation (Golan & El- Yaniv! 2018). |Ren al. (2019) partition the input into semantic and background parts and define the log-likelihood difference between a normal model and a background model as the likelihood ratio to distinguish anomalies. More recently, Ristea al. (2022) propose a self-supervised neural network composed of masked convolutional layers and channel attention modules for vision tasks, which predicts a masked region in the convolutional receptive field. Roth et al.] (2022) utilize memory bank learned from nominal samples and nearest neighborhood search to detect anomalies on industrial images.
2.2 TIME SERIES ANOMALY DETECTION
Since the battery system is a complex system that consists of multi-dimensional time series data, the most relevant deep learning research topic is multivariate time series anomaly detection. We will briefly introduce recent progress and the common datasets used in this area.
Several recent works focus on multivariate time series anomaly detection, Malhotra al. (2016) propose to model reconstruction probabilities of the time series with an LSTM-based encoder-
decoder network and use the reconstruction errors detect anomalies. Hundman et al. (2018)
leverage the prediction етгогs of the LSTM model to detect telemetry anomaly data. Su et al. (2019) propose OmniAnomaly to find the normal patterns through stochastic recurrent neural network and use the reconstruction probabilities to determine anomalies. /Zhao et al. (2020) capture multivariate correlations by considering each univariate series as an individual feature and including two graph attention layers to learn the dependencies of multivariate series in both temporal and feature dimensions. Deng & Hooi (2021) adopt graph neural networks to learn the vntablvariable interactions.
There are several public time series datasets for anomaly detection. The SMAP (Soil Moisture Active Passive satellite) dataset is collected by NASA's Earth Environment Satellite Observation Satellite reile et al. 2010). The MSL (Mars Science Laboratory rover) collects data sequences to determine if Mars ever able support microbial life (Hundman et al. 2018). The water
2

---

Under review as a conference paper at ICLR 2023
treatment physical test-bed datasets, SWaT (Secure Water Treatment) (Mathur & Tippenhauer 2016) and WADI (Water Distribution) (Ahmed et al. 2017). are sensor data recording simulated attack scenarios of real-works water treatment plants. The TSA (Time Series Anomaly detection system)
contains time series data collected from Flink (Zhao et al. 2020).
The anomaly labels in these time series datasets marked when anomaly event happens. In contrast, battery system failures can only be marked on the vehicle level rather than the event level. Therefore, though the mentioned multivariate time series anomaly detection algorithms achieve good performance on their respective datasets, the performance of these algorithms on our battery system failure detection dataset needs be reassessed.
ANOMALY DETECTION FOR DYNAMICAL SYSTEMS
In this section, start by formulating time series anomaly detection hypothesis testing dynamical systems. We then study likelihood-based tests in this context and exploit the Markovian structure of the time-series data collected from dynamical systems. By adapting classical analysis, we provide a guarantee for the false discovery rate of our model.
3.1 HYPOTHESIS TESTING AND ANOMALY DETECTION
Hypothesis testing on time series tries to decide whether collection of random samples {21, .T, provides enough evidence (in terms of statistical significance) reject the null hypothesis that the samples come from a null distribution Po as opposed to a family alternative distributions. In practice, the null distribution Po is often not known. To carry standard hypothesis tests, Po needs to be estimated empirically from independent copies random samples
(11, (2, DT Ji,
1,2,...,
the estimated is accurate enough, one can still apply standard hypothesis testing algorithms. To detect anomaly given collection of time series }xi where x; {-1, 22,,.., 2T }T, the model can learn the normal distribution po- Then, given a new sample x, the model predicts whether it is from the normal distribution (i.e., not rejects the null hypothesis) from an abnormal distribution (i.e., rejects the null hypothesis).
The procedure is conceptually simple, yet the challenge lies in efficiently approximating the distri- bution Po- In the following parts, we show how viewing 2y as outputs from dynamical systems can relieve the curse of dimensionality.
3.2 HYPOTHESIS TESTING IN DYNAMICAL SYSTEMS
In this subsection, formulate hypothesis testing in dynamical systems and highlight its difference against the classical setup. For simplicity, we consider a discrete dynamical system with a fixed time length in Fuclidean space. A more generalized and formal study is left future directions. In particular, consider a random mapping
X x
where f describes the transition probability that maps an inner state E X, a system input ue E U. and time-invariant system parameter e θ random next state 2++1 E X. More formally, for t = 2,,....,
Ftt1
f(r,, 0 ue),
Furthermore, we consider the case when system inputs are sampled from a distribution that is independent from the system itself, u1.7 U.
Our goal is to detect whether observed sample comes from normal system, where system parameters are sampled from the null hypothesis Ho, or from an abnormal system, where the parameters are sampled from the alternative hypothesis Hf;
Hg ~ Oo.
H1 :0 θ1, for some θ

---

Under review as a conference paper at ICLR 2023
The above formulation subsumes many real-world problems. For example, if we aim to detect abnormal electric vehicle batteries, can describe battery health, whereas the signals 2y are recorded by the battery management system under the charging current U,
The benefit of viewing time series as observations from dynamical systems is that it turns the high dimensionality caused by the long time horizon into our favor. Intuitively, the problem dimension is determined by the dimension of the system dynamics whereas additional observations 2,, 4g tell us more about the unobserved parameter 0. We pursue this idea in the next subsection.
3.3 LIKELHOOOD TESTS FOR THE DYNAMICAL SYSTEMS
Following the notations above, the anomaly detection task can be stated via the optimization problem below:
max -. (c 1}}
s.t EU [[((II u1:) 11 S a..
where, I denotes the indicator function, the hypothesis class F is subset of prediction functions {c: y&T XUST - {0, 1]}. and the parameter a controls the false discover rate. Here, the objective aims to maximize the power of the test on the alternative hypothesis, i.e., the percentage of anomaly detected. For now, we assumed that the alternative hypothesis is simple and single-valued.
With the above goal in mind, we then have the proposition below.
Proposition 1. For any a, the optimal solution to the optimization problem above (also known as the uniformly most powerful test) can be written thresholding the conditional likelihood ratio below
e(TI.T, uI:T) = I{ II pi (oe 1,, It-1) > C
{
(1
where Po denotes the likelihood under the null hypothesis, P1 denotes the likelihood under the alternative hypothesis, and is chosen such that E (Iff [[f())r)l))))) = a.
The proof is direct application of the famous Neyman- Pearssn theorem along with the fact that
(11.,, (hr))
Pi (Ze) M4 #
II
Pu (ZI.T u1:r)
Pu D U1-1, xt-1)
where we use the Bayes rule and the Markovian property of the dynamical system.
The interesting observations from the above theorem twofold. First, the optimal detector is independent of the input distribution una ~ U, but only depends on the conditional distribution p(x1r)lu::) (. Second, variables, which suggests that we may get stronger statistical significance from martingale-style concentration bounds.
get product form that resembles likelihood ratios for independent
In practice very often the alternative hypothesis is composite instead of simple. In this setup, the uniformly most power test may not exist. Alternatively, have the following guarantee on false discover rate:
Proposition 2. for any y, x, u, there exists a such that (2 u) 1, then we have that
f MITT) =I (II Po (1 < <cj:
is the generalized likelihood ratio test. Under the null hypothesis. the likelihood ratio within the indicator function converges in distribution to x-squared distribution with freedom d, where d the dimension of the parameter space.
The proof applics Wilk's theorem to the following equation:
Pi (I (I.)) sup (o(T u1:))
II
Po( Aereuuu-i.. 21-1)
4

---

Under review as a conference paper at ICLR 2023
The above proposition suggests that, asymptotically, could simply reject the hypothesis by thresholding the likelihood according to the x2 test and control the false discovery rate. Since log- logliikeihhod is monotonic in likelihood, it is equivalent to finding classifier of the following form:
flarr, UITT) =I ((0) + Σ
11 21 <
where we dropped the subscript and use 1(0, x, u) = Po u) to denote the log- rigelikelihood under the distribution of the null hypothesis. The form in equation motivates us to propose the anomaly detection algorithm in the next section.
4 DYAD: AUTO-ENCODER-BEL -BASED ANOMALY DETECTION MODEL
have seen above that the key to anomaly detection via hypothesis testing is to learn the distribution of f ~ Gu. We adopt the variational inference (as in variational autoencoder (Kingma & Welling 2013), diffusion models (Ho et al.) 2020), etc.) formulation for this task. In particular, we parameterize the family of likelihoods L via weights in neural networks. Then want identify the likelihood E L that minimizes the KI. divergence between the empirical distribution Po and the probability function pr- induced by the learned likelihood /* For more detailed discussion on the variational inference, we refer the readers to Section 2.2 in Kingma & Welling (001)). Hence, we get,
Po (x;, U, 0,)
min D (Po. pt-) = Ex,u,,00pp log LEC
Pt- (x,, Us 0.)
where u, x; are shorthands for the deh sampled input and output sequences. We note that as the numerator independent of we can equivalently solve
1(0') + Muaaliin
max Ex, B...B.B. IEC
IST
where we used the fact that likelihood can be rewritten as products due the Markovian structure.
However, the above problem cannot be solved, because in practice we can only observe the system inputs and outputs u,, X,, whereas the system parameter 0 remains unknown. Hence, a natural fix to infer 0 from observed data.
Σ Laaa=tuu 11 0.)
(L.Ja)EC
where we marginalize over the unobserved @ by learning a posterior distribution. In practice, we cannot optimize over the entire probability space. Hence we simplify the problem with the following approximations:
1. We assume the distribution of the system parameters 0 can be reparameterized (e.g.. through a neural network) as multivariate Gaussian 0 ~ ~N(00, I).
2. We assume that x, Gaussian conditioned on Wr- 21- 0, i.e.,
2, (( o);
where Pa is function to be learned and ar is hyper-paramett
3. We assume the posterior is also Gaussian, i.c..
N ((ueuu-11 11-1), 1), (2 v))))
where Me and a are functions to be learned.
Putting everything together and use the ELBO (evidence lower bound) trick, we get something that is similar to the LSTMMautoencodee model, but with one key difference, that the system input is sent into the decoder. The optimization problem now becomes:
min Ex, Do DRi(N(ve, (())))))) (vi ma) -
5

---

Under review as a conference paper at ICLR 2023
Input
Variational Regularization loss
Encoder
Latent Variable
Reconstructed Obervvatio x'
Decoder
System input 2
System response
Reconstruction loss
Figure I:
The network architecture of DyAD. The input data are split into system input and system response x. The model parameters are updated by minimizing an autoencoder style loss Eqn.
where D denotes the KL divergence, take the negative sign and use that log- log-likelihood of Gaussian is quadratic. The notations fdd, Me are parameterized by recurrent neural networks and are shorthands for
Me Me Me u:), -²2 (x, u:).
Md 2/
We further simplify the KL divergence using properties of Gaussian distributions and get
min Ex, U. 8 NNNiee,0 | + tr(oor - log(lo2) +
Σ SS
(v)
pa) 10
(5)
variationall reg. luss
recon, lnes
view He, de as output the encoder network and nld as output of the encoder network, then we can adopt the variational autoencoder training procedure with the modification specified in the Figure []] The key difference is that, instead of requiring the network learn to retrieve all data dimensions, our model aims learn the internal states 0 of the system. Therefore, the decoder in DyAD, which now serves as a dynamical system, is responsible for simulating the system, and retrieving the response from the latent representation and the internal states. We note that some recent works (c.g., Girin et al. (2020); Mehta et al.) (202T)) also applied autoencoders to learning dynamical systems, yet our result provides the first derivation and application via anomaly detection.
We briefly comment on why our proposed algorithm could outperform existing methods before moving testing the empirical performance of DyAD. Existing deep-learning based methods mostly build on the intuition that model could learn generalizable normal patterns in the time series. Our analysis formalizes this intuition with Proposition | and from which we make the following improvements. First, by proposing a dynamical system probabilistic model, we isolate rare input signals (e.g.. battery charging protocol) from abnormal systems (e.g.. battery status). Second, by focusing on hypothesis testing in the system parameter space 0, the dimension of the distribution significantly reduced. These two changes allow our model utilize data more efficiently.
BATTERY SYSTEM ANOMALY DETECTION
Before we present the anomaly detection results, we briefly describe our released EV dataset. As one of the core components of EVs, the battery system not only needs to support long cruising range, but also guarantee the safety of drivers and passengers. Effective and efficient detection of battery system failure reduces the product recall rate and improves the user experience. Due to its wide range of applications and great significance, we release a large- battery dataset, which contains over 88M battery charging time steps collected from 301 vehicles (see detailed stats in Table MT. The multi- dimensiialii time series features include current, voltage, temperature and SOC (state of charge) information. An example is shown in Figure 2(b) and 2(c) More examples

---

Under review as a conference paper at ICLR 2023
MSL
SMAP
SWaT
EV (ours)
Number of dimensions
55
25
50
Number of time frames
132, 046
562, 800
92. 255
88, 135, 040
Anomaly ratio
10.27%
13.13%
5.95%
N/A
Table I:
The statistics of our EV Dataset and public datasets.
can be found in Appendix B Our dataset and code available at https: /ttp///drvv..a// wttt..ms///aa..
310 -gP ?e e- nBBM The usage of our data is under CC BY-NC-
SA license.
Normal/Abmormal label
hormal
Alnoral maheche
Comer Le
Jan
Jan 79
24
Unlabeled charging records the vehidle
Trrolapp
Tesmilappia
(a)
(b)
(c)
An illustration of our EV dataset. (a) The data structure of our EV dataset. Only vehicle- level anomaly labels are available. Charging snippets are collected within months and years. (b)-(c) The voltage and current of charging snippets collected from a normal and an abnormal vehicle.
Figure 2:
We highlight that unlike previous datasets where anomalies marked when unexpected event happens in time series, the anomaly labels the battery dataset have a natural hierarchical structure shown in Figure 2(a). Specifically, abnormalities in the battery system chronic (such aging), so the abnormality labels are at the vehicle level (one per vehicle) rather than the event level (one per timestamp), which is more challenging time series anomaly detection task. To address the problem, we propose the following procedure to summarize piece-wise prediction into vehicle-level prediction.
5.1 DETECT VEHICLE LEVEL ANOMALIES
To handle the hierarchical dataset structure and sparsity of anomaly labels, we develop robust scoring function to map the snippet-level scores obtained from DyAD to vehicle-level predictions. The scoring function is shown in Alg. [] In particular, we predict the abnormal degree of a charging snippet by thresholding the reconstruction error at value and then predict whether a vehicle is abnormal by averaging the top h percentile errors. Both 7 and h are fine-tune mnnettundd on the validation dataset. Notice that the straightforward idea is to average all charging snippets belonging to vehicle, but we will show that our robust scoring procedure achieves better performance in experiments. We apply both the robust scoring function and the averaging scoring function all deep learning baselines.
Algorithm Pseudo code of the robust scoring function Hyperparameterss percentile h and threshold т.
Input: Charging snippet scores F ((::),: from a vehicle.
Output: Vehicle-level prediction.
1: Sort F by from large small.
2: Take the mean of the largest 4% as the vehicle's score.
3: Predict vehicle abnormal if the average of the largest 4% is greater than т.
5.2 EXPERIMENT RESULTS
The interpolated averaged ROC curves and the mean and variance of AUROC values are shown in respectively. Our proposed algorithm DyAD achieves the best performance
Figure 3 and Table 2
7

---

Under review as a conference paper at ICLR 2023
DyAD (Ours)
SVDD LSTMAD
MTAD-GAT
AE
GDN
Without robust score
With robust score
1.0
1.0
True positive rate
True positive rate
0.6
0.6
0.4
0.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False positive rate
False positive rate
Figure 3: Interpolated averaged ROC curves of several algorithms on our EV battery dataset. Shaded area represents the five-fold variance.
on our large-scale dataset against well-established autoencoder-basse and graph-based algorithms. Meanwhile, this is the first time to deploy deep learning algorithms to detect electric vehicle battery system failure in such large dataset. Furthermore, DyAD benefits from the proposed robust scoring function with a 2.0% improvement. suggests that sometimes averaging on all snippets belongs to vehicle the best choice for such hierarchical dataset, which is of the difference between our dataset and traditional time series anomaly detection datasets.
Table 2:
Mean and standard variance of test AUROC (%) values all vehicles from three makes. Among all the considered algorithms,, DyAD achieves best detection results by 3.1% ~ 20.0% AUROC boost and relatively small variance compared to the second best algorithm both averaging (Left) and robust scoring (Right) criteria. Further, our robust scoring function consistently improves baseline algorithms. Bold denotes the best results.
AUROC (%) by Averaging Score
AUROC (%) by Robust Score
Algorithm
AE Deep SVDD
66.1 3.1
69.4 2.6
51.3 + 4.2
2..2
LSTMAD
60.0
63.1
MTAD-GAT
0.9
2.3
GDN
68.9
3.2
DyAD (Ours)
73.2 + 2.5
75.2 + 2.7
Table 3: Mean and standard variance of test AUROC (%) values each make. Bold denotes the best results.
Algorithm
Make
Make
AE Deep SVDD
65.3 7.0
59.4 i: 8.4
5.7
58.5 13.7
LSTMAD
56.0
58.6 9.5
MTAD-GAT GDN DyAD+Robust Score (Ours)
37.1
50.9 12.5
64.8 11.5
65.3 1 4.3
78.0 - 5.2
84.4 + 3.8
We further examine the difficulty of detecting anomalies in each make by training models on data from the same make only. Make is omitted since it has only four anomaly vehicles thus can not do the five-fold cross-validation. The AUROC values of different algorithms are shown in Table 3 It seems that learning one make of vehicles individually would lead to a larger variance of the results. One possible reason is due to the smaller amount of data. When the variance of the results becomes significantly larger, the mean value of the results becomes less reliable in comparing different algorithms affected by outlier values. Therefore, we recommend using a larger number of vehicles reflect the performance of the algorithms in the population. In all five algorithms, the
8

---

Under review as a conference paper at ICLR 2023
variance of DyAD, although also increased, is still at lower level than the other four. Meanwhile, the change of the other four algorithms is within variance. Based on the results in Table 3 detecting anomalous vehicles from Make appears to be simpler than that from Make 1.
By comparing Table 3 against Table 2 we notice that some algorithms(c.g. GDN) benefit from training on all vehicles altogether, whereas some (e.g., DyAD) benefit from learning different models for each Make. Hence, there is still room for researchers to improve the detection of electric vehicle battery system failure. Limited time and computing resources, we only implement these algorithms. We also welcome researchers to further develop algorithms on the dataset to improve EV safery.
6 TIME SERIES ANOMALY DETECTION IN OTHER REAL-WORLD TASKS
To supplement our results, we implement DyAD on two spacecraft system time-series anomaly detection datasets, MSL (Mars Science Laboratory rover) and SMAP (Soil Moisture Active Passive satellite), and water treatment test-bed system dataset, SWaT (Secure Water Treatment). The two spacecraft datasets are released by NASA (O'Neill et al. 2010 Hundman et al. 2018), recording the spacecraft's 's telemetry channel sequences and command sequences encoded as 0 or The water treatment dataset integrates digital and physical elements to control and monitor system behaviors. They are widely used as benchmarks for anomaly detection (Hundman et al. 2018 Zhao et al.] 2020) Chen et al. 2022 Deng & Hooi 2021). For these three datasets, the anomaly events are defined on time frames by another log. And the commonly used metrics are Fl score, Precision, and Recall.
We implement the same baseline algorithms with necessary modification for dataset dimension. The detection results are shown in Table 4 For the spacecraft system, we treat the command dimensions as system input and the telemetry data system response. More training details can be found in Appendix C As shown in the table, DyAD achieves the best F1 scores consistently. Specifically, we improve the SOTA algorithm by 3.9% and 3.0% on the MSL and SMAP datasets respectively. For the water treatment system, the sensor dimensions are treated as system input and the actuator dimensions as system response. More details can be found in Appendix C We can see from the table that DyAD outperforms other baseline algorithms by 0.4%.
Table 4: F1, Precision and Recall on the two spacecraft datasets. Bold denotes the best results.
MSL
SMAP
SWaT
Method
F1
Precison
Recall
F1
Precision
Recall
F1
Precision
Recall
AE
0.5774
0,5507
0.6070
0.6215
0.8606
0.4864
0.7961
0.9452
0.6876
Deep SVDD
0.6804
0.8270
0.5779
0.2965
0.1752
0.9630
0.7870
0.8633
0.7231
LSTMAD
0.4502
0.3831
0.5458
0.6944
0.8914
0.5687
0.8007
0.9570
0.6883
MTAD-GAT
0.9084
0.8754
0.9440
0.9013
0.8906
0.9123
0.8359
0.9271
0.7612
GDN
0.7660
0,7598
0.7723
0.7069
0.7556
0,6641
0.8082
0.9935
0.6812
DyAD (Ours)
0.9438
0.9215
0.9673
0.9313
0.9501
0.9132
0.8399
0.9824
0.7335
7 CONCLUSIONS AND DISCUSSIONS
In this work, we have seen that properly formulating the probabilistic model underlying the data-in our case, viewing time series as outputs from dynamical systems- can greatly improve sample effi- ciency and lead to better model design. Our analysis naturally motivates a variant of the autoencoder model that is tailored for encoding dynamical systems. The performance of our proposed model is validated on both our released large-scale EV dataset and on existing public datasets.
There is much to be done. It is known that models based on likelihood alone do not achieve the best performances, as observed in Nalisnick et al.] (2018). The idea is that although likelihood provides a guarantee of the false discovery rate, it does not necessarily give the best detection power, which would depend on the distribution of the alternative hypothesis. For this reason, multiple improvements (Ren et al. 2019; Xiao et al. 2020 Choi et al. (018)) have been made to further boost detection performances by improving the scores generated by likelihoods. As these investigations orthogonal to our focus within this work, it remains to see whether similar ideas can be applied to improve our models.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Chuadhry Mujeeb Ahmed, Venkata Reddy Palleti, and Aditya P. Mathur. WADI: water distribution testbed for research in the design of secure cyber physical systems. In Panagiotis Tsakalides and Baltasar Beferull-Lozane (eds.), Proceedings of the 3rd International Workshop on Cyber-Physice Systems for Smart Water Networks, CySWATER@ CPSWeek 2017, Pittsburgh, Pennsylvania, USA, April 21, 2017, pp. 28- ACM, 2017. doi: 10. 10.1145/305536633055777 URLhttps:: https://doi 3055375
22g/1001145/22
Wenchao Chen, Long Tian, Bo Chen, Liang Dai, Zhibin Duan, and Mingyuan Zhou. Deep variational graph convolutional recurrent network for multivariate time series anomaly detection. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 3621 3621-3633, PMLR, 2022. mlr wtt.llp.peeseeeeeee//eem chen22x. html
URL http:/tsss/poccceiiiniee
Hyunsun Choi, Eric Jang, and Alexander A Alemi Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXtv:/880.011392. 2018. 2018.
Ailin Deng and Bryan Hooi. Graph neural network-based anomaly detection in multivariate time series. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Confer- ence on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021. pp. 4027-4033 AAAI Press, 2021. URL https:// aaai org/index: index. php/ AAAI//
article ew 6523
Jie Deng, Chulheung Bae, James Marcicki, Alvaro Masias, and Theodore Miller. Safety modelling and testing of lithium-ion batteries in electrified vehicles. Nature Energy, 3(4):261-266, 2018.
Laurent Girin, Simon Leglaive, Xiaoyu Bie, Julien Diard, Thomas Hueber, and Xavier Alameda- Pineda. Dynamical variational autoencoders: A comprehensive review. CoRR, abs/2008.
2020. URL hoLntts::/// http:://axxvvvvvo wttpa//www..//b//////aae//// 12595
Izhak Golan and Ran El-Yaniv. Deep anomaly detection using geometric transformations. In NeurIPS 2018, 2018.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02133. 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jichao Hong, Zhenpo Wang, and Yongtao Yao. Fault prognosis of battery system based accurate voltage abnormity prognosis using long short-term memory neural networks. Applied Energy, 251: 113381, 2019.
Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Söderström. Detecting spacecraft anomalies using Istms and nonparametric dynamic thresholding. In Yike Guo and Faisal Farooq (eds.), Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, pp. 387-395 ACM, 2018. doi: 10.1145/3219819:22198228 httpttts::/ /doi corii.oogg1000 1145/ 3219819. 3219845
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 1rXiv:1312.611 14,66/11,200 2013.
arXiv preprint
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-t--disttibuution samples and adversarial attacks. In NeurlPS 2018, 2018
Da Li, Zhaosheng Zhang, Peng Liu, Zhenpo Wang. and Lei Zhang. Battery fault diagnosis for electric vehicles based voltage abnormality by combining the long short-term memory neural network and the equivalent circuit model. IEEÉ Transactions on Power Electronics, 36(2):1303-1315
2020.
10

---

Under review as a conference paper at ICLR 2023
Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. Lstm-based encoder-deeoder for multi-sensor anomaly detection. CoRR, abs//607,00118, 2016. /RLpt::://axxxv.oo
abs
Aditya P. Mathur and Nils Ole Tippenhauer. Swat: a water treatment testbed for research and training on ICS security. In 2016 International Workshop on Cyber-physical Systems for Smart Water Networks, CySWater@ CPSWeek 2016, Vienna, Austria, April 11, 2016, pp. 31-36. IEEE Computer URL /esttp::/ddo..ogg/////
Society, 2016. doi: 10. W000CCS5aaer2216..777444 Cyswater. 2016 7499060
Lukas Mauler, Fabian Duffner, Wolfgang G Zeier, and Jens Leker. Battery cost forecasting: review of methods and results with an outlook to 2050. Energy Environmental Science, 2021.
Viraj Mehta, Ian Char, Willie Neiswanger, Youngseog Chung, Andrew Nelson, Mark Boyer, Egemen Kolemen, and Jeff Schneider. Neural dynamical systems: Balancing structure and flexibility in physical prediction. In 2021 60th IEEE Conference on Decision and Control (CDC), pp.
3735-3742. IEEE, 2021.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayaman, Do deep generative models know what they don't 't know? arXiv preprint arXiv:1810.09136 2018.
Peggy O'Neill, Dara Entekhabi, Eni G. Njoku, and Kent H. Kellogg. The NASA soil moisture active passive (SMAP) mission: Overview. In IEEE International Geoscience & Remote Sensing Symposium, IGARSS 2010, July 25-30, 2010, Honolulu, Hawaii, USA, Proceedings, pp. 3236-3239. IEBE, 2010. doi: 10.
URL https:://doli http:///ooooor///..../
IGARSS 2010 5652211
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for nut-o--distribution detection. Advances in neural information processing systems, 32, 2019.
Nicolae-Cätte Cätälin Ristea, Neelu Madan, Radu Tudor Ionescu, Kamal Nasrollahi, Fahad Shahbaz Khan, Thomas B. Moeslund, and Mubarak Shah. Self-supervised predictive convolutional attentive block for anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13576-13586, June 2022.
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14318-14328, June 2022.
Richard Schmuch, Ralf Wagner, Gerhard Hörpel, Tobias Placke, and Martin Winter. Performance and cost of materials for lithium-based rechargeable automotive batteries. Nature Energy, 3(4): 267-278, 2018.
Bernhard Schölkopf. Robert C. Williamson, Alexander J. Smola, John -aylor,,ylor, and John C. Platt. Support vector method for novelty detection. In NeurIPS 1999, 1999.
Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (eds.), Proceedings the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pp. 2828 2837. ACM, 2019. doi: 10.1145/3292500. 3330672. URL wttttt:///eooo /tt:://ooi..co org. 10 1145/
111.1..111122222 3205000222222222 0tt.:/22250033003 3330672
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: novelty detection via contrastive learning on distributionally shifted instances. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hssua-Tiie Lin (eds.), Advances in Neural Information Pro- cessing Systems 33: Annual Conference Newral Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020. Ssser/2020/hastee http:///aaaa//6666636333333 88888896662222 7777777777670 77wp:///7777770220////00----aaa0
/c/
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved dutto--distributtin detection. arXiv preprint arXiv:2007.05566 2020.
11

---

Under review as a conference paper at ICLR 2023
Hui Wu, Denys Zhuo, Desheng Kong, and Yi Cui. Improving battery safety by early detection of internal shorting with a bifunctional separator. Nature communications, 5(1):1-6, 2014.
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. Advances neural information processing systems, 33:20685-20696. 2020.
Qiao Xue, Guang Li, Yuanjian Zhang, Shiquan Shen, Zheng Chen, and Yonggang Liu. Fault diagnosis and abnormality detection of lithium-ion battery packs based on statistical distribution. Journal of Power Sources, 482:228964, 2021.
Ruixin Yang, Rui Xiong, Suxiao Ma, and Xinfan Lin. Characterization of external short circuit faults in electric vehicle li-ion battery packs and prediction using artificial neural networks. Applied Energy, 260: 14253, 2020.
Xiao-Guang Yang. Teng Liu, and Chao- Yang Wang. Thermally modulated lithium iron phosphate batteries for mass-marked electric vehicles. Nature Energy, 662):176-185, 2021.
Hang Zhao, Yujing Wang, Juanyong Duan, Congrui Huang, Defu Cao, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. Multivariate time-series anomaly detection via graph attention network. In Claudia Plant, Haixun Wang. Alfredo Cuzzocrea, Carlo Zaniolo, and Xindong Wu (eds.), 20th IEEE International Conference on Data Mining, ICDM 2020, Sorrento, Italy, November 17-20, 2020, pp. 841-850. IEEE, 2020. doi: 10.1109/CDD50000. 10.109900DMM0000020000000000 202.0000000000...... URL https:://oi..om wtwp:/..rr1.... 000.://...................0..... 200.::01...22.....0.0....... 00093
Yuejiu Zheng, Yifan Lu, Wenkai Gao, Xuebing Han, Xuning Feng, and Minggao Ouyang. Micro- short-circuit cell fault identification method for lithium-ion battery packs based on mutual informa- tion. IEEE Transactions on Industrial Electronics, 68(5):4373-4 -3841 2020.
12

---

Under review as a conference paper at ICLR 2023
A RELATED WORK ON BATTERY SAFETY
To the best of our knowledge, there is no well-established deep learning study of battery system failure with large-scale datasets. On one hand, deep learning technology has not been widely used in detecting battery failure, and many studies still use statistical methods (Xue et al. (2021) or canonical machine learning (Zheng et al. 2020). On the other hand, the absence of large public datasets hinders the progress of deep learning techniques in this area.
Existing researches mainly study lithium battery safety through battery physical/chemical structure and charging/disscargingg data. Physic-based approaches usually aim to improve battery safety by changing/adding the structure/component inside battery. [Wu et al. (2014) add a bifunctional separator inside the battery to achieve early detection of lithium dendrites. Yang ct al. (2021) design a thermally modulated I.FP (lithium iron phosphate) battery to improve cruise range. Advanced data-based methods analyze battery faults with deep learning models. |Hong et al. (2019) adopt
an LSTM network to predict multi-forward-step voltage value and judges the battery safety with a threshold voltage value. [i et al. (2020) propose achieve higher battery fault diagnosis reliability and accuracy by combining the LSTM model and the equivalent circuit model. Yangg et al. (2020) employ multi-layen neural network to estimate the current of the short-circuited cell and then predict maximum temperature increase with a 3D dlectro-therma coupling model.
Meanwhile, although deep learning is a data- huta-h technique, existing related studies do not incorporate enough vehicles in their datasets. We notice that Li et al. (2020) collects data from 9 vehicles and divides them into three categories, Hong et al. (2019) record time series information of one electric taxi, Zheng et al. (2020) use four cells with inconsistent capacities, and Yang et al. (2020) use eight cells. However, they cannot used as a benchmark for measuring battery system anomalies because we believe that a conclusive result requires validation on much larger dataset beyond several vehicles and their inconsistent data form is another problem.
B DATASET EXAMPLES
Some charging snippets from the battery dataset are depicted in Figure 4 The feature dimensions include voltage, current, max/min single cell voltage and max/min cell temperature. From the figure, we can see that the time series are mixed, and there is no simple rule to distinguish the charging snippets of abnormal vehicles from normal charging snippets. This is also of the main reasons why anomaly labels are on the vehicle level rather than the snippet level.
Normal vehicle
Normal vehicle vehicce
4.1
Abnormal vehicle
vehicle
Voltage (V)
Current (A)
-1-
-10*
3.9
-2.
-107
-3x
Normal vehicle Abnoraal vehicle
3.7
-4
-10²
100
100
100
Timestamp (s)
Timestamp (s)
Timestamp (s)
4.1
Normal vehicle Abnormal vehicle
40
Min single voit (V)
40
4.0
Max temp C)
Min temp C)
3.9
30
Normal vehicle Abnormal vehicle
20
20
3.7
Normal vehicle Abnormal vehicle
10-
3.6
50
100
50
100
50
100
Timestamp
Timestamp (5)
Timestamp (s)
Figure 4: Charging snippet examples from snippets of normal and abnormal vehicles.
13

---

Under review as a conference paper at ICLR 2023
Here also give conceptual figure to help to understand what anomaly in battery system would be like. Figure 5 there are three charging snippets collected from a vehicle, where the first two have normal max temperature value and the third shows sharp increase in temperature. Notice that the different snippets are not contiguous, and there are discharging phase between snippets. Our dataset does not contain the third type of snippets, early enough to prevent potential hazard in advance.
goal is detect battery system failures
This example also explains that the labels in the dataset can only be labeled the vehicle level, but not at the charging snippet level, since even a battery system that is about to fail may exhibit a normal charging pattern early on.
Charging snippet
TL
Charging sniopet Charging snippet 3
47-
Nax
30 Charging snippet
Charging
Charging snippet
30
6200
2063
(a)
Figure 5: An example of the anomaly in the EV dataset. (a) The max temperature of vehicle with three charging snippets. (b) The SOC of the battery system with three charging snippets. The third snippet shows a runaway charging temperature, which means that the battery system is already broken.
C ALGORITHMS IMPLEMENTATION DETAILS
The ultra implementation details can be found in our released code. We partially use the official and public code resources from PyOd LSTMAD GDN and MTAD-GAT All the experiments run on machine with four 2080 Tī GPUs.
C.1 AUTOENCODER
The network adopts encoder-decoder structure with batch normalization layers, drop out layers and sigmoid activation functions. It is built with several fully connected layers. The latent dimensions [64, 32, 32, 64] in the encoder and decoder. We train the network 20 epochs with batch size of 128 for the battery dataset, and epoch for the two spacecraft datasets. We use the Adam optimizer with learning rate of 0.001.
C.2 DEEP SVDD
The feature extraction network of the deep SVDD is an autoencoder with hidden dimensions [64, 32, 32, 64], which is similar to the autoencoder above. The SVDD loss is computed on the middle 32-dimensional latent feature. We also adopt the reconstruction loss to help the network learn better data representation. We train 10 epochs with batch size of 64 using the Adam optimizer with learning rate of 0.001 for both the battery dataset and the spacecraft datasets.
httas://iithbbb.comm///a/mmm///
http://mmmmmaattttttttteta pyt-chh
14

---

Under review as a conference paper at ICLR 2023
C.3 LSTMAD
The LSTMAD uses the LSTM layers and LSTM cells to encode and decode the input data, respec- tively. The latent feature dimension 32. We use Adam optimizer with a learning of 0.001 and train 20 epochs with a batch size of 128 on both the battery dataset and the spacecraft datasets. For the two spacecraft datasets, the window length to 128. The reconstruction loss function is the mean absolute eπror
C.4 MTAD-GAT
For the spacecraft dataset, the detection results are directly from the official paper Zhao et al. (2020). For the battery dataset, set the window length to 100 and train the graph network 30 epochs with a batch size 256. Only the feature dimension number is modified. The other parameters are set to the default value.
C.5 GDN
We adapt the graph layers to fit our input data. Stacked fully-connected layers are attached at the end of the graph layer. The latent dimension is 128. The window length of the time series data is set to 32 for the battery dataset and 128 for the spacecraft datasets. We use an Adam optimizer with learning rate 0.001 to train 20 epochs with batch size of 128.
C.6 DYAD
For the battery dataset, annealing Adam optimizer. The hidden size and latent size are set to 64 and 32, respectively. The spacecraft dataset is pre-processed following Zhao et al. (2020). Different snippets are concatenated together. To deal with this special data structure, we use LSTM as the recurrent cell and add convolutional layer to the decoder. delayed signal of the encoder passed to the decoder to compensate for sparse command channels. We train the MSL dataset for 10 epochs and the SMAP dataset for epochs. Both with a cosine annealing Adam optimizer.
GRU as the recurrent unit and train the network 3 epochs with cosine
The SWaT dataset provides an official document interpreting each dimension of the data, including the type (sensor or actuator) and a brief description for each dimension. We treat the sensor dimensions as system input and the actuator dimension system response. We directly apply DyAD on SWaT with window size 128.
We notice that another water treatment dataset, WADI (Ahmed et al., 2017), is also used in recent works (Deng & Hooi 202T). However, as shown on the official dataset website, the current version of WADI has 127 dimensions, while the reported number of dimensions is 1 112 in recent research works (Deng & Hooi 2021). Because of the version mismatch, we do not consider using WADI in experiments.
15


Reviews:
[{"review_id": "zzPoK1Z88ei", "reviewer": "Reviewer_4Vb4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors propose a novel anomaly detection scheme based on hypothesis testing, that is directed towards multivariate problems where higher dimensionality can prove to be challenging for statistical testing. The authors leverage an autoencoder architecture for modelling the underlying dynamical system, and come up with a robust approach for determining whether a time series is anomalous or not. The contributions of the paper are framed around a new dataset on the battery power of electric vehicles that was curated for this paper (and made publicly available), but the authors also feature additional experiments on more widely-used datasets such as SMAP. Notable performance improvements are observed across all settings. ", "summary_of_the_review": "This paper frames its contributions around a novel dataset that poses different challenges to other anomaly detection datasets that are usually considered in the literature. The authors also develop a new model (DyAD) and robust scoring procedure that outperforms several other competing techniques. Nevertheless, I believe that some of the concepts put forward in the paper are not developed clearly enough, and the novelty of the modelling contributions were particularly unclear while reading the paper. Overall, I would encourage the authors to revise how the paper is currently structured in order to reduce confusion in regards to these aspects, and highlight the contributions more clearly.\n", "strength_and_weaknesses": "- I appreciate that the authors have introduced a new complex dataset for anomaly detection, whose properties may overlap with time series encountered in other real-world domains. I believe that moving beyond the standard datasets used for evaluation is a particularly effective means of identifying new research problems and directions. \n- However, the paper treads a fine line between tackling an application-specific problem and proposing a more general solution to anomaly detection in multivariate time series datasets, which can be confusing at times. I believe the Battery System problem itself could be better explained early on in the paper via more illustrative examples of what a fault looks like. One other thing I was unsure of while reading the paper is whether the problem set-up of detecting anomalous vehicles (here described by several collected time series), would be better posed as a classification problem overall rather than time series anomaly detection. Given that the detection appears to be carried out offline, it is unclear whether there may be other more obvious means of detecting issues or degradations in this case.\n- I found that there was a bit of a disconnect between sections 3 and 4, where it wasn’t always easy to understand how certain aspects introduced in Section 3 are then manifested in the architecture of DyAD.\n- The use of auto-encoders for time series anomaly detection is fairly common, and I would have liked the novel aspects of this formulation to be more clearly expressed. The takeaways on why DyAD is more appealing than competing techniques was not clear to me by the end of reading the paper beyond the performance improvements obtained in the experiments.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WoGTIpkmU5O", "reviewer": "Reviewer_V5eP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors developed an auto-encoder-based anomaly detection model called DyAD (Dynamic system Anomaly Detection) for time series anomaly detection. They primarily dealt with the high-dimensionality issue caused via viewing time series anomaly detection as hypothesis testing on dynamical systems. They demonstrated the performance of their model on both public dataset and on a newly prepared EV dataset.", "summary_of_the_review": "This is an interesting study for time series anomaly detection. I believe the research community may find value from this study. Even though the authors released the source code, it would be helpful if they could briefly discuss their implementation details in the manuscript (maybe in the appendix). Please add a comma (,) after e.g. (page 6).\n\nThe study demonstrated state-of-the-art results in several datasets, including an EV dataset and two spacecraft datasets. Although the authors claim that they validated the efficacy of their model using these datasets, I have a little reservation about the diversity of the datasets and the generalizability of the proposed model. I would like the authors to extend their experiments on one or more general purpose tasks, such as energy systems, stock analysis and meteorological data to evaluate the robustness of the model. We cannot draw conclusions about the robustness of a model based on limited experiments.\n \nOverall, I am bit concerned about the generalizability of the model since the experiments are not conducted on a variety of datasets. I therefore do not recommend accepting the paper in the current form considering a high bar to ICLR.\n", "strength_and_weaknesses": "Strengths:\n- The study demonstrates that a well-formulated probabilistic model can significantly increase sample efficiency and result in better model design.\n- The dataset released by the authors could be useful for the community.\n- The proposed model performed significantly better than the other candidate models.\n\nWeaknesses:\n- Properly formulating the probabilistic model underlying the data is not a brand-new concept. This is more prevalent in literature.\n- Experiments are somewhat limited\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AaajNllRxqm", "reviewer": "Reviewer_R2Bu", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper suggests a specific method for anomaly detection in time series data. The approach is based on hypothesis testing. Given the distribution of the normal data and the empirical one from test data, the decision of normal vs. abnormal is made. The authors suggest as an application area the evaluation of the battery health of e-cars and provide a new data set for future benchmarks in anomaly detection. In the experiments, they evaluate their method on such data. In addition, two public benchmarks are used to compare with existing methods. The experimental results seem to support the claims of the authors", "summary_of_the_review": "If the contribution by the authors is the solution for the battery system anomaly detection (Sec. 5), then it boils down to Algorithm 1 (basically an averaging operation) and a variance test. This is not sufficient for me as the contribution, even if there might be more complex problems around that might be tackled by their method. Furthermore, the theoretical part of their paper assumes Gaussianity for the time series process variables (page 6, enumeration 1-3). I cannot see any novelty in this since the authors do not relate such dynamic systems to the Kalman filter.", "strength_and_weaknesses": "Time series anomaly detection is an important area nowadays to monitor dynamic systems (for example, weather) and to gain insights into unknown processes by analysing detected anomalies. This usually comes with a continuous time series in which intervals that show anomalous behaviour/relationship of the dynamic process variables involved have to be detected. However, this paper seems to treat anomaly detection as an outlier detection method in static data built from a time series model, i.e. the whole time series is input to the system to determine whether it shows anomalous behaviour. This observation is a major criticism since the paper title is - at least for me - misleading. However, such preconditions/assumptions might be of importance in some applications, like the introduced battery health data set. But even in this case, the paper lacks originality and contribution to the area. Another criticism is the strong assumption of the Gaussianity of the involved variables. I wonder whether Kalman-Filter with control would do a similar job with a proper evaluation function of the model match. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject