Anomaly detection in dynamical systems from measured time series
Authors: Andrei Ivanov
Anna Golovkina
Venue:  (2021)
Keywords: dynamical systems
polynomial neural networks
anomaly detection

Abstract: The paper addresses a problem of abnormalities detection in nonlinear processes represented by measured time series. Anomaly detection problem is usually formulated as finding outlier data points relative to some usual signals such as unexpected spikes, drops, or trend changes. In nonlinear dynamical systems, there are cases where a time series does not contain statistical outliers while the process corresponds to an abnormal configuration of the dynamical system. Since the polynomial neural architecture has a strong connection with the theory of differential equations, we use it for the feature extraction that describes the dynamical system itself. The paper discusses in both simulations and a practical example with real measurements the applicability of the proposed approach and it's benchmarking with existing methods.

Under review as a conference paper at ICLR 2021
ANOMALY DETECTION IN DYNAMICAL SYSTEMS FROM MEASURED TIME SERIES
Anonymous authors Paper under double-blind review
ABSTRACT
The paper addresses a problem of abnormalities detection in nonlinear processes represented by measured time series. Anomaly detection problem is usually for- mulated as finding outlier data points relative to some usual signals such as expected spikes, drops, or trend changes. nonlinear dynamical systems, there are cases where time series does not contain statistical outliers while the pro- cess corresponds to an abnormal configuration the dynamical system. Since the polynomial neural architecture has strong connection with the theory of differ- ential equations, we use it for the feature extraction that describes the dynamical system itself. The paper considers both simulations and practical example with real measurements. The applicability of the proposed approach and it's bench- marking with the existing methods discussed.
INTRODUCTION
Most the works related to anomaly detection in time series data are referred to the detection of the "observation which deviates much from other observations to arouse suspicions that it was generated by a different mechanism" (see Hawkins (1980). Anomaly detection problem is usually usual signals such unexpected spikes,
formulated as finding outlier data points relative drops, trend changes, and level shifts. Baazque et al. (2020) provides a literature review that deals exclusively with time series data and provides taxonomy for the classification of outlier detection techniques according to their main characteristics.
In contrast these methods, we address the problem of anomaly detection dynamical systems from measured time series. In this case, we are interested in detecting the anomalies which do not deviate so much from other observations while they were still generated by a different configuration of the dynamical system. To better explain this issue, let us consider a dynamical system in the form of the ordinary differential equation (ODE):
d x F(t,
where X = (æ1, [21,,22.... ] is state vector, F is nonlinear function depending on two scalar pa- rameters 01 and 02. The dynamical system (1) generates a trajectory in the form of the multivariate time series as a particular solution for given initial condition Xp. Let us also assume for sim- plicity that the initial conditions X(0)= = X₂ are always the same for different trajectories but only parameters are varied and belong to the normal distribution.
Since the system TT) is nonlinear, the distribution of the trajectories is unknown in advance and may differ from the normal one. This may cause that trajectories corresponding abnormal system parameters are located somewhere among the other trajectories. Fig. demonstraass that parameters of the system (I) taken from the tail of the normal distribution correspond to centered trajectories in the time-space.
This example formulates the problem of anomalies detection in the dynamical systems represented only by the measured trajectories. We are interested in the unsupervised methods for recovering the representative features of the dynamical system from the time series. Such method should calculate features that are correlated with the dynamical system itself but not just with a time series that is generated by the dynamical system. Also, collecting massive training sets in industrial applications

---

Under review as a conference paper at ICLR 2021
1.00
0.03
0.00
0.50
6-
-0.03
0.25
-0.05
0.00
0.55
time, sec
Figure I: The abnormal parameters of the dynamical system from the tail of the normal distribution may correspond to centered trajectories in the time-space. The normal distribution is not preserved due to the nonlinear differential transformation that describes the dynamical system behavior.
is often not feasible. This makes it difficult to apply existing approaches that are based on either the extraction of the statistical features or state-of-the-ar neural networks for temporary dependencies modeling. To address these lack-of-the-data challenges, we rely on the polynomial neural networks that were recently gain the attention of researchers in the field of learning of dynamical systems from measured data.
The most relevant to our research work the paper of authors López et al. (993). where the con- nection between the system of ODEs and polynomial neural networks (PNN) is introduced. Further, the PNN architectures were also widely highlighted in the literature. For example, Zjavka (201T) proposes polynomial neural architecture that approximates differential equations. The Legendre polynomial is chosen as a basis by Yang et al. (2018). In all these papers, the authors suggest either integrating the parametrized polynomial ODEs or training PNN as a black-box model for identifica- tion of the dynamical systems from measured time series and prediction of new trajectories.
In opposite to these works, Ivanov et al. 2020 suggests a deterministic algorithm to translate an arbitrary nonlinear ODE to PNN without training. If the dynamics of the system follows approxi- mately given ODE, the Taylor mapping approach allows calculating weights of the neural network (TM-PNN) directly from the equations. This TM-PNN equals initial ODEs with the necessary level of accuracy and can fine-tuned to recover true dynamics from real measurements without nu- merical integration of the ODEs. Further, Ivanov & Agapov| (2020) applied TM-PNN architecture in practical application to control one of the larges X-ray source. A deep TM-PNN is initialized from the ODEs that describe the charged partiele motion and consists of more than 1500 hidden polynomial layers with unique weights that are fine-tuned with only one measured trajectory of the real system.
For the introduced problem of anomaly detection in the dynamical systems represented by time series, utilize both PNN and TM-PNN architectures. Similar to López al.) (1993), train PNN from scratch with measured trajectories. But instead of using the resulting model for dynamics prediction, we focus on the feature extraction problem and consider the weights of the PNN as features describing the dynamical system itself. Since this approach does not require any a priory knowledge about the dynamical system the form of the ODEs and relies only on data, we compare it with statistical feature extraction and LSTM --sedd anomaly detection for time series.
For the benchmarking with simple ODE- based parameters estimation, we utilize the TM-PNN architecture and follow the idea proposed in Ivanov & Agapov| (2020). Given the system of ODEs that defines the dynamical system with an initial assumption of zero values for all free parameters, translate the equation to PNN using the Taylor mapping technique. Then compare the direct tuning of parameters of ODE with the fine-tuning of weights of the pre-initialized TM-PNN.
Though the Anonymous Company from the automotive industry provides the real measured data related to the introduced problem, the dataset is small and contains only 50 time series with only one true anomaly. Moreover, the anomaly scores of normal data unknowns. For benchmarking purposes, we use simulated datasets of the same size based on the Van der Pol oscillator. Then we apply PNN for the analyses of real-world datasets that were generated in production settings.
2

---

Under review as a conference paper at ICLR 2021
2 SIMULATED DATA
In this section, we implement a simplified example with the Van der Pol oscillator. This equation widely used in the physical sciences and engineering and can be used for the description of the pneumatic hammer, steam engine, periodic occurrence of epidemics. economic crises, depressions, and heartbeat. The equation has well-studied dynamics and widely used for testing of numerical methods (see e.g. Pan & Duraisamy (2018)).
For simplicity, we parametrize the Van der Pol oscillator with two scalar parameters di and 02:
x' = y. y/ - y (1 + a1jk - az)x²y
were means derivative on time, (01, E EN(0, with = 0.001 for clarity and as identity two dimensional matrix.
For experiments with each method, we generate 100 datasets of 50 trajectories each for different sampled parameters (41, (2), Each trajectory is the numerical solution of the system (2) with con- stant initial condition (Io, yo) = (3.0, 0) during 5 see time interval with fixed time step 0.01 sec. To solve the system of ODEs we use the traditional Runge Kutta method of forth order and then add white noise to all variables and y.
After this procedure, each trajectory represented by two-variable time series {x,, yi with 500 stamps and corresponds to certain values of parameters (01, 02). The true score of abnormality for each trajectory associated with the probability values of these parameters. Fig. represents an example of the generated dataset. On the left plot, the result of the sampling the (a1, (22 is demonstrated. The right plot shows values each trajectory with a corresponding anomaly score. Since the system (2) is nonlinear, the normal distribution in parameter space is not preserved in time-space for trajectories. Namely, the true abnormal trajectories can be positioned the arbitrary locations inside the bunch of all generated trajectories.
In the next section, we explore different methods for feature extraction from these time series data. Since extracted features have different physical meanings and units, algorithm to score trajectories in terms of abnormality. After scores are estimated and converted ranks,
the Isolation Forest
both the Kendall rank correlation coefficient (r) and Spearman's rank correlation coefficient (p) to evaluate the method performance in comparison with true ranks associated with the probability values of parameters (a1, 02). Both 7 and p metries between variables will be high when observations have similar rank, and low when observations have a dissimilar rank.
We also introduce a heuristic accuracy metric. In each dataset of 50 samples, we consider suc- cessful anomaly detection if the trajectory with the worst true score is detected within three less scored trajectories by the given method. As an accuracy, we calculate the fraction such successful detections among all generated datasets.
The intuition of this heuristic metric is the following. Though the rank correlation coefficients are mathematically rigorous metrics, it is expected in the practical application that anomalies arise not very often. It may be more important to detect true anomaly among trajectories with the lowest scores rather than rank normal trajectories correctly. Three trajectories with the lowest scores among 50 samples roughly correspond to the 5% threshold that selected arbitrarily.In our opinion, the rank correlation coefficients along with such a heuristic accuracy demonstrate both the performance of the extracted features and the ability to detect anomalies in practical applications.
2.1 PNN ARCHITECTURE AND PNN-BASED BASED FEATURES
Since each time series has 500 stamps and generated by randomly defined dynamical system, we construct deep PNN architecture with 500 polynomial layers with shared weights. Each layer is described by the transformation
x 37
[1++1
x; yi
M:
Wi
+
W
W
(3)
Mi+1
2
V
3

---

Under review as a conference paper at ICLR 2021
40
Hi+1
5500
M
M
M
yb
y
y/++
Y500
Figure 2: Deep PNN architecture with shared weights.
with initialized from uniform transform weights W1 = 1, W22 0, W3 0.
Fig. represents a multioutput deep PNN architecture with shared weights that propagate initial values (Zo, yo] the beginning of the trajectory along the time. Since it is indicated by López et al. (1993) that such architecture corresponds to a system of ODEs, we train a separate PNN for each trajectory and consider fitted weights Wi, 22, W3 features for further anomaly scoring.
For the training, we use Adam optimizer with default parameters implemented in Keras/TensorFlow library. Though each layer M represents local dynamics with small time step, the multioutput architecture allows processing noisy trajectories considering loss function as mean squared error (MSE) for all time steps (24, yi) altogether.
2.2 STATISTICAL FEATURES (TSFRESH)
The benchmarking method for abnormal process detection in the industry is a threshold-based deci- sion based on some statistical characteristics of the measured processes. The exact rules for anomaly detection can be adaptive and depend on the testing equipment specification but still depends on the statistical features.
To represent such threshold-basee anomaly detection, we use isfresh library that was developed by Christ et al. (2016) and allows us to calculate a comprehensive number of features. After automatic feature extraction from the generated time series, we apply the classical principal component analy- sis (PCA) to preserve only components. The range from 2 to 10 components was investigated but only weak influence on the anomaly scores is explored.
Tabel [ contains a comparison of scoring of trajectories based on statistical features extracted from time series and PNN-based weights that are fitted during 1000 epochs. Regardless of the accuracy and (T,p) p) performance, statistical features sometimes disagree with true anomaly scores calculated from the distribution of parameters of the dynamical systems.
On the other hand, the minimum values of rank correlation coefficients are always positive for the PNN weights along with the higher accuracy value. This makes the PNN- approach at least an alternative for industrial applications where threshold-based rules do not detect all anomalies in the production processes.
Table 1: Comparison tsfresh and PNN
METHOD
ACCURACY
METRIC
MEDIAN
AVG
STD
MIN
MAX
7 0.19
0.21
0.25
-0.32
0.72
tsfresh
0.59
P 0.28
0.26
0.30
-0.40
0.85
0.34
0.39
0.14
0.20
0.66
PNN
0.80
0.44
0.47
0.16
0.23
0.79
2.3 TRADITIONAL NEURAL NETWORKS (LSTM AUTOENCODER)
Another possible solution for anomaly detection in time series is using recurrent convolution neu- ral networks. Unfortunately, most existing deep learning efforts in the time series anomaly detection related to statistical outliers and do not tackle problem formulation. For the addressed prob- lem, only relatively small datasets of unlabeled data are available. Nonetheless, for benchmarking purposes, apply the STM-based autoencoder extract latent features of time series.
4

---

Under review as a conference paper at ICLR 2021
Since the LSTM neural network is general-purposs architecture, it is not possible to train separate neural network for each time series. Instead, one has to train LSTM on the whole training batch that causes issue with increasing number of training epochs in the considered task. For our simulated data, the LSTM-hasee autoencoder does not generalize data but memorize all trajectories. The increase in number of the latent variables from to 15 did not improve the situation. The scoring performance is reduced while training epochs are increased (see Tabel 2
Table 2: Memorization of the LSTM-AE with the increase of training epochs
METHOD
ACCURACY
METRIC
MEDIAN
AVG
STD
MIN
MAX
LSTM-AE
0.14
0.17
0.37
-0.46
0.70
0.34
(1000 epochs)
P
0.15
0.19
0.44
-0.54
0.78
LSTM-AE 0.28
0.08
0.16
0.20
-0.15
0.52
(5000 epochs)
0
0.12
0.19
0.24
-0.23
0.57
In opposite this, PNN training demonstrates convergence with increased training epochs. One of the possible reasons for this is that the PNN architecture corresponds to some priory unknown system of ODEs and during training with a single trajectory PNN more accurately recovers physical properties of the dynamical system. A brief explanation of this property is presented in the discussion section. The theoretical examination along with the exploration of other -urpoos neural architectures not covered by this research work.
Table 3: Convergence of the PNN with the increase of the training epochs
METHOD
ACCURACY METRIC
MEDIAN
AVG
STD
MIN
MAX
PNN
0.80
T
0.34
0.39
0.14
0.20
0.66
(1000 epochs)
P
0.44
0.47
0.16
0.23
0.79
PNN
0.92
T
0.51
0.50
0.09
0.36
0.68
(5000 epochs)
P
0.60
0.60
0.11
0.40
0.82
2.4
EQUATION-BASED PARAMETERS FITTING
While in the sections 2.2 and 2.3 we train PNN from scratch, in this section, we take into account a priori knowledge about the dynamical system in the form of the ODEs. Also, compare the fine- tuning of the pre-initialized PNN (TM-PNN) with the Taylor map and the direct fitting of ODEs' parameters.
The traditional fitting of the parameters (a1 02) in the equation 2 is implemented as a search procedure through numerical solving of the ODE with Runge- Kutta method of forth order. Given the generated trajectory, the parameters in the equation are identified as the solution of the inverse problem starting with the initial approach (01, 02) = (0, 0).
Since in the ODE-based fitting we parametrize equation and preserve its structure, let us also con- priori available knowledge about the dynamical
sider the equation (2) with aj 0 and d2 system that were used to generate given trajectory:
x dt
xz = -
x 2
(-1
0
+
0
0
0
0
2
dt
V
0
ry
y
PxX + P [ |9]
where X = (a, y) and X 2020) one calculate Taylor map for this equation in the form (3)
(x3 x² y. xy . ). Following algorithm presented by Ivanov et al.
X W,X, WaX!" WaX!1
(5)
5

---

Under review as a conference paper at ICLR 2021
that in the combination with the equation rrssulss in the ODE
X P,WiX, P1 x + (PIWs + PaW1 X O(X,
(6)
where | means the k-th Kronecker power. Comparing 6 with yields a new system of equa-
tions concerning the weight matrices:
W P.W1,
W P.W2,
W'J P.W3 + P.W1.
with initial conditions Wi(0) = 1, W2(0) = 0, Wa(0) = 0 for the time interval [0, 5] results Taylor map M with matrices Wi (T'),, (2( and W3(T):
The solution of the equation
7
0.99995933855004993
0.00904044997261332
W
W2 0,
0.0090400499226132 1.00899987822663
(8)
W3 1.2204426e 07
-4.0741619e
05
-2.4518692e 07
-5.5367007e 10
4.0741619e - 05 -0.0090808848
-8.1971951e - 05
-2.4684793e - 07
that describes the dynamics of the Van der Pol oscillator for 01 = 0 and a2 = O. Since the calcu- lated map M valid for arbitrary initial conditions, the TM-PNN can be constructed by sequential concatenation of maps with shared weights.
The only difference between the TM-PNN and the PNN used the previous section is the initial weights. As long as we compare the method with ODE-based fitting, we can incorporate initial knowledge form of the system ODEs inside the PNN by calculating the initial approach for weights (8). Further, these weights are fine-tuned with generated time series without solving ODE. For both approaches, we stop fitting when MSE for both variables in the trajectory becomes less than 10-5.
Table 4: Comparison of the ODE-based parameters fitting with the TM-PNN fine-tuning
METHOD
ACCURACY
METRIC
MEDIAN
AVG
STD
MIN
MAX
0.50
0.52
0.05
0.48
0.59
ODE
0.98
P
0.61
0.60
0.03
0.56
0.62
0.48
0.49
0.06
0.41
0.56
TM-PNN
0.96
P
0.55
0.57
0.04
0.53
0.63
EXPERIMENTAL DATA
The dataset consists of 50 two-dimensional time series with true abnormal trajectory that is generated in the real-world production settings. Fig. (left) demonstrates the time-space with the filtered trajectories for the first variable. The simple moving average is applied remove noise from measurements. Note that the true scores of all normal trajectories are unknown and only information about the true abnormal trajectory is available.
To detect this anomaly within 50 trajectories, we apply the considered above methods and score the time-series based on the extracted features with the help of the Isolation Forest algorithm. Both Isfresh-based features and LSTM latent variables do not detect abnormal trajectory while the PNN- based features provide meaningful result. The right plot in Fig. presents scoring based on the PNN weights.
As polynomial layer the PNN, we use the second-order transformation (3) with the additional matrix Wo for free terms in the Taylor map. Given the two-dimensional state vector of the mea- surements, the total amount weights equals to 12. Fig. shows the weights of each trajectory in parallel coordinates. The weights for the true abnormal curve differ from the rest of the trajectories and anomaly can be easily detected with the Isolation Forest approach within the space of the PNN weights.

---

Under review as a conference paper at ICLR 2021
0.15
025
0.5
0.00
14
-0.50
-0.50
-075
02
-1.00
DmC
Figure 3: Input measured trajectories with a labeled true anomaly (left) and the same trajectories scored with PNN weights (right).
DISCUSSION
In the experiments, PNN-hased features outperform both classical statistical features and latent vari- ables from the LSTT autoencoder. Moreover, the performance of the TM-PNN compatible with the simple benchmark of the fitting of the parameters of the ODE.
For the dynamical systems, the PNN is suitable architecture because of its connection differential equations. After fitting PNN with a single trajectory, PNN not only learns the given trajectory but approximates the dynamical systems itself. To explain this briefly, let describe simple way to go back from the fitted PNN to ODE.
Let us assume that after training PNN with single trajectory, the weights equal W1, W2, W3 and define a polynomial layer in the form of the Taylor map (3). Let us also consider that this Taylor map corresponds the discretization of the unknown ODE
dt = f(X)
with the Euler approximation
X,+1 = X4 + f(X)tt.
Comparing 9 with 3 one recover right-hand side of the differential equation using the fitted weights the PNN:
8 =x=((x) = (Wi-1)X W2 + W2X!1s) /At.
dt
Using different approximation schemes, one can obtain different exact formulas for the ODE. The main point is that the weights of the PNN correspond to the parameters of some system of the differential equations that is unknown a priori. This approach allows us to approximately reconstruct the system of ODEs that is supposed generate the training trajectory without processing of ODEs with numerical solvers at all. Only weights of the PNN are tuned during training from time series.
1.00
1.0
0.5
0.50
0.25
0.0
Figure 4: The PNN weights for each trajectory in parallel coordinates.

---

Under review as a conference paper at ICLR 2021
5 CONCLUSION
The PNN architecture for dynamical systems becomes natural one in terms of the connection to systems of ODEs. It is demonstrated in the paper that weights of the PNN fitted from the time series can be considered as parameters of the initially unknown system of ODEs. Since ODE-based models are one of the most general approaches for describing the dynamical systems, the PNN-based representation learning outperform traditional methods for the introduced problem.
For the first set of experiments, train PNN with zero initial weights from data without any initial knowledge about the dynamical system. PNN-based features outperform both classical statistical features and traditional neural networks. Also, it is still open question how select general- purpose neural architecture to solve the problem. On the other hand, the deep PNN provides a clear understanding of how to tune the number of hidden layers and orders of nonlinearities for weight matrices depending on the input time series.
For the second experiment, we incorporate initial knowledge about the system inside PNN using the Taylor mapping technique and compare the approach with the ODE-based parameters fitting. Both TM-PNN and ODE-based parametrization have compatible performance. Though the ODE-based fitting slightly outperforms TM-PNN in the simulated experiment. it demonstrates that PNN can replace classical ODE-based parametrization when the system of ODEs completely unknown or complex enough to be solved numerically in practice.
In the last example, we consider real-world application of anomaly detection in production data. The use case is provided by Anonymous Company and represents complex assembly testing in automotive industry. Since nothing is known about the process in terms of the possible ODEs and the labeled anomaly is currently passed the existing testing equipment, the PNN can provide additional insights for anomaly detection and quality control the production environment.
REFERENCES
A. Blázquez- Garzquee-GGrcia, A. Conde, U. Mori, and J. Lozano. A review outlier/anomaly detection in time series data. arXiv.org e-Prints: 2002.04236, 2020. URL
2002. 04236. pdf
M. Christ, A.W. Kempa-Liehr, and M. Feindt. Distributed and parallel time series feature extraction for industrial big data applications. arXiv.org e-Prints: 1610.07717, 2016. URL http://
adsabs
D. Hawkins. Identification of Outliers. Springer, Dordrecht, 1980. doi: https://doi.or//10.1000 978-94-015-3994-4
A. Ivanov and I. Agapov. Physics- deep neural networks for beam dynamics in charged particle accelerators. Phys. Rev. Accel. Beams, 23:074601, Jul 2020. doi: 10. 10.1UB//hyyRReeAceeBeasss 23.074601. URL https:// aps http:///o.//....1..////eeeeee//e
23. 23.074601
A. Ivanov, A. Golovkina, and U. Iben. Polynomial neural networks and taylor maps for dynamical systems simulation and learning. In 24th European Conference on Artificial Intelligence 2020, Spain, Frontiers Artificial Intelligence and Applications, pp. 1230-1237, Netherlands, August 2020. IOS Press. ISBN 978-1-64388 1-44668-100-9
doi: 10.323//FAIA200223,
V. López, R. Huerta, and J. R. Dorronsoro. Recurrent and feedforward polynomial modeling of coupled time series. Neural Comput.. 5(5):795-811, September 1993. ISSN 0899-7667. doi: 10.1162/neco. 1993.555.999995 URL /tL
nwwo. 19w..o...99..... 1993.999395
S. Pan and K. Duraisamy. Long-time predictive modeling of nonlinear dynamical systems using neural networks. Hindawi Complexity, 4801012, 2018
Y. Yang, M. Hou, and J. Luo. A novel improved extreme learning machine algorithm in solv- ing ordinary differential equations by legendre neural network methods. Advances in Difference Equations, 4(1 (164 2018.
L. Zjavka. Differential polynomial neural network. Journal of Artificial Intelligence, 4 (1):89-99, 2011.
8


Reviews:
[{"review_id": "Qmw6yXybBrW", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes the use of polynomial neural architectures (PNN) for anomaly detection in dynamical systems, and more precisely, for feature extraction. The choice of the architecture is motivated by the links between PNNs and ODEs that describe the evolution of dynamical systems. \n\nQuality, clarity, originality and significance:\nPro: The paper is well written and approaches an important inverse problem, how to find anomalies in a dynamical system from some measured time series. The problem of anomaly detection is clearly significant, and the paper could benefit even more from a more extensive experimental setting on real world data. Cons: While the paper is mostly well argumented, I find that it leaves relevant information out that hinder its clarity as detailed below. \n\nThe main motivation of the paper is exemplified in Fig. 1 which shows that abnormal parameters of a dynamical system from the tail of the distribution might lead to normal trajectories, therefore misleading the anomaly detection. While this example is explaining already well the problem, I have a few questions. What are the ODEs generating this data, is this an example of the Van der Pol oscillator? If yes, it would be good to mention that. If it is a different system, is this system one-dimensional (x) or higher-dimensional? When PNN will be applied to this data, will it be applied on the fully observed system (let’s imagine two-dimensional with variables x and y) or will it be applied on the partially observed system (here x)? If the input to PNN will be only x then I believe the Figure is ok, because we try to find anomalies only from x and the plot on the right shows that normal trajectories still correspond to abnormal parameters. However, if the PNN will be applied on other variables in addition to x (let’s say x and y), then we would need to see the trajectories in this two-dimensional (or 3D if adding time) space. Because a trajectory can seem normal in one direction (here x) but appear as abnormal if we plot it in a different space. I believe the trajectories should be shown in the space of the data that will be used as input to PNN. In Sect. 2, is the full data (x and y) given to PNN always (this seems to be the case from eq 3 and Fig. 2)? \n\nI believe the abstract would benefit from a sentence about the results.\n\nEq (1): Why only two parameters a1 and a2? If this hints to the Van der Pol oscillator, then X should be two-dimensional not higher-dimensional. Otherwise, if it is a general dynamical system, $\\theta$ could be used for parameters. And then for illustration purposes as in Fig. 1 assume $\\theta$/$a$ is two-dimensional.\n\nWhat happens if the PNN architecture chosen in eq (3) changes? Are the results robust to the order of the polynomials?\n\nCould the authors provide a reference for Isolation Forest and describe it a little bit and justify its choice?\n\nSect. 2.1: Do we need the same number of layers as the number of stamps? If yes, how to address scalability to real world applications?\n\nSect. 2.2: “calculate a comprehensive number of features” – how many features were computed and could the authors give some examples? Why is PCA needed on these features, and what is the impact of the number of PCs on the final results?\n\nWould it be possible to combine the different tables of results into one table for an easier comparison? I am very much surprised by the difference in performance between the LSTM in Table 2 and PNN in Table 3. If LSTMs behave so poorly, are there other methods closer to PNN in performance?\n\nSect.2.4: Some aspects, eg eq (8) could be added to the appendix. Not sure I fully understand how the TM-PNN is constructed by “sequential concatenation”. Could the authors detail on this? What are the running times for ODE and PNN is Table 4?\n\nSect. 3: Would it be possible to have more details on the data? Some details are mentioned in the last paragraph from Conclusions, but could be moved here. What does this data represent, unless there are confidential issues? Eq (3) is a third order, not second order – which one is used here? Fig. 3, similar to Fig.1, should show the trajectories in the same space as the input to RNN, which might be more than one dimension (here x). \n\nFig. 4: I might be missing smth, but not sure I understand what are the parallel coordinates?\n\nRelated literature: there is a strong trend in ML in using Neural ODEs for modeling dynamical systems. Could the authors discuss the similarities and differences between Neural ODEs and PNNs?", "extra_scores": {}, "extra_text": {}}, {"review_id": "M9ogzBRE6LQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the problem of anomaly detection in nonlinear processes. The main idea is to employ polynomial neural architecture for feature extraction that describes the dynamical system. The experiment results in both simulations and a practical example showed the effectiveness of the proposed technique.\n\nStrengths:\n+ Anomaly detection in dynamic systems is an interesting problem to study.\n+ The proposed PNN architecture is technically sound.\n\nWeaknesses:\n- Several related works are not mentioned or compared.\n- Only one practical dataset is used for evaluation.\n\nMy major concern is that many related works are not mentioned or compared in this work, e.g., One-class SVM[1], DSEBM[2], Deep SVDD[3], DAGMM[4], and Neural ODE [5].\n\n[1] One-Class SVMs for Document Classification, JMLR, pp. 139-154, 2001\n\n[2] Deep structured energy based models for anomaly detection. In International Conference on Machine Learning, pp. 1100–1109, 2016.\n \n[3] Deep One-Class Classification, ICML 2018\n\n[4] Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection, ICLR 2018\n\n[5] Neural Ordinary Differential Equation, NIPS 2019\n\nI would suggest the authors provide a comprehensive review of existing anomaly detection methods and discuss why they cannot address the anomaly detection problem in measured time series.\n\nAnother concern is that only one small dataset is used for evaluation. More real-world datasets should be used to justify that the proposed technique can be generalized to different use scenarios.\n\nWhat's the computational complexity of PNN? What's the empirical running time?\n\nPNN is only evaluated over two-dimensional time series, I am wondering whether it can be generalized to multivariate time series. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gl0TTKlnMJE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is exploring anomaly detection problems where a time series does not contain statistical outliers. It shows the proposed polynomial neural network-based approach, which uses the fitted weights as extracted features, outperforms both classical statistical features and an LSTM autoencoder-based neural network.\n\nIt is indeed interesting and of practical value to distinguish any anomaly condition resulting from an abnormal configuration of the dynamical system, especially when it is difficult to accomplish using classical methods. \n\nHowever, there are some places where this paper needs to further clarify:\n- It is mentioned the proposed method will “train a separate PNN for each trajectory and consider fitted weights as features for further anomaly scoring”. If this is the case, I don’t see how two similar trajectories ($X,t$), with one deemed anomalous, would result in significantly different PNN weights based on equations (2), (3), and (9).\n- The anomaly score needs to be clearly defined. From Figure 3, it seems a lower anomaly score indicates a higher chance of a trajectory being abnormal. However, I was thinking of the exact opposite while reading through the text.\n- It should be further clarified that the color code on the left subplot of Figure 1 should be the probability of getting a specific ($a_1, a_2$) pair, rather than the anomaly score on the right subplot.\n- Is it possible to use standard ODE-based fitting without any prior knowledge of the dynamic system (the $a_1=a_2=0$ assumption in Section 2.4)? If so, can this method be used in the last real-world application example of anomaly detection?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on anomaly detection in dynamical systems from time series measurement. The originality of the contribution is to detect anomalies not based on the detection of OOD observations but from identified parameters or statistics of the dynamical system. They are using “polynomial neural networks. All the reviewers agree that the paper is not yet mature both in the form and in the technical content. The authors did not provide a rebuttal.
Decision: Reject