Recurrent Neural Network Architecture based on Dynamic Systems Theory for Data Driven Modelling of Complex Physical Systems
Authors: Deniz Neufeld
Venue:  (2021)
Keywords: dynamic system identification
recurrent networks
explainable AI
time series modelling

Abstract: While dynamic systems can be modelled  as sequence-to-sequence tasks  by deep learning using different network architectures like DNN, CNN, RNNs or neural ODEs, the resulting models often provide poor understanding of the underlying system properties. We propose a new recurrent network architecture, the Dynamic Recurrent Network, where the computation function is based on the discrete difference equations of basic linear system transfer functions known from dynamic system identification. This results in a more explainable model, since the learnt weights can provide insight on a system's time dependent  behaviour.  It also introduces the sequences' sampling rate as an additional model parameter, which can be leveraged, for example, for time series data augmentation and model robustness checks. The network is trained using traditional gradient descent optimization and can be used in combination with other state of the art neural network layers. We show that our new layer type yields results comparable to or better than other recurrent layer types on several system identification tasks.

Under review as a conference paper at ICLR 2021
RECURRENT NEURAL NETWORK ARCHITECTURE BASED ON DYNAMIC SYSTEMS THEORY FOR DATA DRIVEN MODELLING OF COMPLEX PHYSICAL SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
While dynamic systems can be modeled as sequence-to-scquenee tasks by deep learning using different network architectures like DNN, CNN, RNNs neural ODEs, the resulting models often provide poor understanding the underlying system properties. We propose new recurrent network architecture, the Dynamic Recurrent Network (DYRNN), where the computation function is based on the discrete difference equations of basic linear system transfer functions known from dynamic system identification. This results in more explainable model, since the learnt weights can provide insight on a system's 's time dependent behaviour. It also introduces the sequences' sampling rate an additional model parameter, which can be leveraged, for example, for time series data augmentation and model robustness checks. The network is trained using traditional gradient descent opti- mization and can be used in combination with other state of the art neural network layers. We show that our new layer type yields results comparable to or better than other recurrent layer types on several system identification tasks.
INTRODUCTION
Dynamic systems occur in many different areas of life (Isermann & Münchhof (201 From biol- ogy, engineering, medicine to economics and more: Often, if system changes its state based on a external input, this system can be viewed as dynamic system. Dynamic system identification is the process of modelling the system's 's properties. Such models can be used, for example, for anomaly detection, controller design outcome prediction. For linear systems, this identification task is already well understood and state of the art methods exist.
However, if system exhibits non- non-linear behaviour, for example slip-stick-effect due to mechanical friction, the applicability of these methods is limited. In this case different approaches implemented in the state of the range from white-box black-box models. Generally, increasing system complexity raises the need for more powerful and often less understandable model architectures in order to produce satisfactory results: White box (based on differential equations or numerical simulations of the physical system components), black box systems (like Gaussian processes, deep neural networks, Support Vector Machines) and grey box models, which often employ mix of linear and non-linear building blocks.
One example of a tool used in engineering are Hammerstein Wiener which are a combination of linear and (prior known) non-linear equations (shown in Figure []. The linear model parameters are determined based on the training data. The non-linear behaviour of models is modeled using lookup tables or user defined non-linear functions.
In this work we present a new type of recurrent neural network layer called the Dynamic Recur- rent Neural Network (DYRNN). It is designed for data based modelling of dynamic systems in sectuence-t--sqqueee
manner based on input (x(t)) and output (y(t) data. With it, we intend to bridge the gap between dynamic systems theory and recurrent neural networks. The layer's internal computation is based on elemental transfer blocks from linear system identification. By combining it with non-linear neural networks, Hammerstein-Wiiet style model is emulated. This way, the

---

Under review as a conference paper at ICLR 2021
model can offer additional knowledge about the examined system's 's internal properties. Further- more, while the model is trained on sampled data of one sampling rate can be applied to data of different sampling rate. This can be used to check the robustness of the model
the same system or to save time during training. We show that our network produces results which are better than or comparable to other recurrent networks (RNN, LSTM, GRU) three different problem datasets. Since the layer can be implemented to be compatible to current deep learning frameworks, it can be combined with state of the art neural network layers (like convolutional or fully connected layers) and training techniques.
utt)
g(t)
y(t)
Figure 1: Hammerstein-Wicee Wiener model. Static non-linearities before and after a linear differential equation model g(t) can be used to model non-linear dynamic systems.
2 RELATED WORK
Dynamic system identification can be viewed as saaeence-to-seqenee task of the modelling of systems' output based on certain inputs. [sermann Münchhof (201T), for example, list several different tools like ARIMA processes for linear systems and multiple neural network architectures for non-linear systems. Examples for the latter are locally recurrent locally feedforward networks (LRGF), Multi Layer Perceptrons (MLP) and Radial Basis Function (RBF) networks of different types of dynamics. These model structures are generalized, however, and as we will show, further theoretical background linear systems theory could be leveraged.
Generally, deep learning offers multiple neural network layer types that can be employed when dealing with sequence-to-sequenee problems, like fully connected (FC) networks, convolutional networks (CNN) recurrent networks. Recurrent networks also known sequential models (like RNN, LSTM by Hochreiter & Schmidhuber (1997) and GRU by [ho et al. (2014)) and have been used successfully for text based sequence-to-see so-quuence problems like machine translation or text processing. Wang (217)) demonstrates a concept of LSTM for dynamic system identification by using several parallel LSTM layers which predict the systems behaviour based on its input and prior predictions and their derivatives. A different approach of modelling dynamic systems are neu- ral ordinary differential equations (ODEs) by Chen et al. (2018). These networks learn dy dt of a function f with v() = (=(()) and the resulting ODE model is used an numerical integrator/sol (like Runge-Kutte to compute yt). This has the advantage of a varying sampling step size which is determined by the solver. but these methods are agnostic of dynamic systems theory knowledge. Similarly, RRissi et al. (2019) use deep learning to learn Partial Differential Equations (PDE) of physical systems in FC model combined with numerical integrator. Furthermore, since the evaluation of ODE/PDE models is done using a numerical integrator, the model is difficult to apply in combination with other neural network layers like for example convolutional or recurrent layers. In terms of sampling frequency of the measurement data, recurrent network architectures can only be trained on one specific data frequency, and do not provide the functionality generalize other sampling rates of the same system. In such case one would have to resample the new data to the frequency of the training set.
Explainability approaches for sequential models text processing deduce which parts of the sen- tences are relevant for the model's prediction based on the activation of the internal gates (as shown by e.g Krakovna & Doshi Velez (2016)). Interpretability of RNN/LSTM or GRU models for con- tinuous measurement data has not been explored yet to our knowledge.
DYNAMIC RECURRENT NETWORK
The complexity of the modelling of dynamic systems does not only result from the potential non- linearity, but also from the fact that the model has to keep track of the system's 's current and past states in order to predict the output based on new input. We intend model dynamic system in
2

---

Under review as a conference paper at ICLR 2021
Input
RNN
Output
time
Hidden State
Figure 2: Example of eequence-lo-sequeeee translation using recurrent network layer. The layers hidden state is used for the computation of the next timestep's result.
a sequence-to-sequenee task, which is why we chose a recurrent network architecture. Recurrent neural networks can be seen units which iterate over given sequence and predict output. During this computation, a hidden state is computed which is leveraged for the prediction of the next time step. Our network differs from other state of the recurrent networks in its computation function, which is derived from linear dynamic systems theory.
In the following we explain the theoretical background used in this work. Then we describe the structure of our DYRNN. Finally, we show different advantages that result from this structure like training and prediction at different signal sampling rates and interpretability of the models.
3.1 LAYER STRUCTURE
The background knowledge this section covered by Föllinger al. (2016) and Yarlagadda 2010). In dynamic systems theory linear system with the external input u and resulting system output yt) is expressed as the differential equation
v(t + a1 (t)
an (m) (t) bo u(()) b2 úi(t) + ... bn u (t).
Therefore, linear system acts transformation of the input unt) based to convolution (+) with the system's 's lincar transfer function g(t) to produce the system output y(t with
v() v(1) g((() = ( (()))(( -
The transfer function g()) used in engineering for, amongst others, controller design, system stability analysis or frequency response estimation. Larger, more complicated transfer functions can be seen as several basic functions which are interconnected in circuit type fashion in parallel or in series (see Figure 3)
u(t)
g(t)
y(t)
u(t)
(1)
(1
y(t)
Figure 3: Examples for transfer function circuits. Linear systems modeled by chaining together several transfer functions.
Dynamic systems theory identifies five basic linear transfer functions from which all linear dynamic systems can be modeled, which we use in the construction of the DYRNN: P, I, D, PT1 and PD. For further information and and visualization see Appendix Section A They have the following functionalities:
P: Proportional gain of the input, implemented multiplication with constant
I: Integrating component, meaning the step-wise summation of input over time.
D: Differential component acting high-pass filter

---

Under review as a conference paper at ICLR 2021
p(k)
i(k-1)
i(k)
x(k-1)
x(k)
D
d(k)
(k)
(k:)
(1
(tl(k-1)
(kk)
PT1
ptl(k)
(R)
Ml.a(k (k)
1)
Myx (k)
Mn.pell
PD
pd(k)
Myst (k)
(a) Design of DYRNN unit with five different inner equations. This architecture results
(b) Resulting model when stacking DYRNN layers. Each of output channels would produce another five output chan- nels for each connected cobb-oommnnnt.
multiples five output channels.
Figure 4: Structure of DYRNN and how appending layers performed.
PT1: Proportional multiplication of input with time delay, used to model e.g. voltage in capacitors in RC circuits. This function also acts as a low-pass filter
PD: Proportional increase with differential component
In the following equations K stands for constant with influence on the output amplitude, while T a time constant which influences the speed of the system's reaction. K and T act as the trainable weights in DYRNN layer. For usage in recurrent layer, these differential equations discretized with in our case first degree forward Euler. This is common in many engineering applications, and means replacing i() with
((t) y(k) y(k 1)
At
This results in discrete recurrence equations, with the sample number and the time distance be- tween samples At. The respective equations of the basic transfer functions follows:
p(:) Kp .<(k)
Al x(k)
i(A:) = i(k: 1) +
(5)
K,
KD.
d(b) At
(ar(k) x(k 1))
At
ptl (k) pt1(k - -1) (KrTı x(k) pt1 1)) At + TeTı
pd(k) = KPD (x(k) + TPD (a(k) x(k: 1))),
with the input æ(k) and all K > O.
The equations above are implemented as the computation function of recurrent network layer as described in Appendix Section A. K and T in the equations become trainable weights of the layer, while the hidden state consists of <kkk i)k i(k - 1) and pl(k 1). In the following we refer to these internal computations as subcomponents. We explore two different network variants in our work: The DYRNNS with all five known subcomponents and the DYRNN3 with just P. PD and PTI. The reason for this is, that D subcomponent can be approximated by PD and by PT. Since integrators can also cause model instabilites, we explore both variants in our experiments.
We formulate unit's computations based on the number of input channels nie and the number of outputs per subcomponent type Noc. The number output channels per layer Mayer amounts to (subcomponent c Thw 3 * noe for DYRNN3 and 5tn, for DYRNNS (see Figure 4a).
The fact that DYRNN can implemented with an interface compatible with other recurrent net- works allows the modelling of systems properties by training in sequence-to-sequence fashion
4

---

Under review as a conference paper at ICLR 2021
using out of the box backpropagation through time, gradient descent and optimizer functions im- plemented in state of the art frameworks like Pytorch (by Paszke et al. (2019) and Tensorflow (by Martin Abadi et al. (2015)).
Since each imput of the layer is connected to five basic components, stacking DYRNN layers results in increasing number of output channels, shown in Figure 4b In experiments, achieved good results with two cascading DYRNN layers followed by a per-time step linear FC layer with one neuron. This final layer can be enhanced using non-linear activation functions and more layer- s/neurons, which would result in similar structure to Hammerstein-Wer Wienerr models for modelling of static non-line non-linearities in the data. case of noisy input data, for example due to sensor noise, additional CNN layers could be used as well.
3.2 GENERALIZATION FOR DIFFERENT SIGNAL SAMPLING RATES
The new parameter At means that a model can be used on datasets with a different sampling rates by adjusting the At when interacting with the model. This can be leveraged as means to perform time series data augmentation. Training on different rates and testing on another sampling rate also provide insight about model's robustness. Additionally, training time be reduced by subsampling the dataset during training, and using the original sampling rate while inferencing for greater prediction accuracy (see Figure 5p
Subsampled Dataset
Subsampled Dataset
0.50.5
1.0
0.2
0.72
0.30
Amplitude
Amplitude
-075
10
16
Time in Seconds
Sample Number
Figure 5: Example dataset y(k) sin(x(k)) at different sampling rates. Keeping the amount of samples the same, a larger sampling rate results in larger range of covered time.
3.3 INTERPRETABILITY OF RESULTING MODELS
Based on our layer, it is possible to interpret the significance of a subcomponent towards the final dataset well as the properties of its influence. While the DYRNN layers learn the time depen- dant system behaviour, a per-timestep fully connected network selects the relevant dynamics of the model. The higher the FC's learnt weight connected for the modelling of the dynamic system (see Figure (). As an example, assume a network one DYRNN5 layer and a fully connected (FC) layer as shown in Figure 6 Since the subcomponents P. and PTI are weighted with 0.5, 1.5 and 0.8 as opposed to the D and PD with 0.1 and 0.2, the modeled system mostly displays P. and PTI properties.
specific subunit, the more significant it is
Additionally, we can analyse the model by evaluating its equation in the Laplace and the Fourier domain. This means replacing the discrete subcomponents with their respective Laplace formulas as described in the Appendix Section A while keeping the structure of the network as well as the trained values for K and T. This yields new equation depending on the parameter s, which for the example network above results in:
((s) (0.) 1.5Kr
0.8K
+ 0.18K
0.2K + sTpD))
(9)
(1+ (Tp)
(s) = U(s) 0.1s*KDTPT + (0.1KK + + (L.5K) + 1.5Kg
(10)
1.0s"Tpr 1.0s
5

---

Under review as a conference paper at ICLR 2021
DYRNN
FC
3pp (:(A))
0.5
1.5
Uo (k)
Ma(k)
0.1
((k)
pp (k)
0.8
Mnd (k)
0.2
Figure 6: Interaction of one DYRNN layer and a FC network layer. Via the FC layer's learnt weights. it is possible to gain insight of the underlying system dynamics.
The Laplace form has the advantage that convolution with the transfer function y(t) u(t) g can be replaced with a multiplication with (s) = G(s) Us).. It allows an engineer to analyse the model for stability, and visualize different kinds of information on the model, e.g. in a pole- zero plot. The transfer function G(s) can be examined for system stability by replacing s with iw (w = 2T f. with f as the signal frequency) yields the Fourier from of the transfer function. This can be used for frequency response and root locus analysis (Föllinger et al. (2016)), as we show in our experiments.
The transformation of G(s) for our example network into the time domain yields the following differential equation:
+ +.)) + ((( + +.. + u(t)1.5,, (t(t)Trr + v - = =()))==" (0((000
(11)
Transforming larger networks - for example the one used in our experiments works along the same lines. Such extracted differential equations can further be leveraged to analyze the model.
EXPERIMENTS
We evaluated our network on three different datasets. The first is an electrical RC-circuit simulated using Matlab Simulink, shown in the Appendix in Figure [3 The other datasets are from the Database for Identification of Systems "DaISy", namely the Heating System De Moor B.L.R. (b) and the Thermic Resistance De Moor B.L.R. (a)) datasets. In the following we will refer the datasets as Circuit, Heating and Thermic. In these experiments we focus on learning the correct dynamics independent from the initial value of y(t). Therefore, we subtract the mean time offset of the prediction sequence towards the label before the final evaluation with Mean Squared Error.
u(k)
concatenate
Recurrent Layer
Cascaded Recurrent Layers
FC with 1 Unit
Figure 7: Architecture used in experiments.
The first goal of the experiments is to evaluate our models (DYRNN3 and DYRNN5) compared to RNN, LSTM and GRU as implemented in Tensorflow. The network architecture is shown in Figure 7 The first part of the network is built in a cascaded fashion of two recurrent layers which are concatenated with the original input. Its results are passed to linear fully-connected layer with one unit. The number units in the other recurrent layers were chosen to result in a similar parameter distribution DYRNN5, as shown in Section A.2 For experiment similar count of parameters in the recurrent part the network and the same layer constellations were used (see Table 4 and 6). Due to the design of the DYRNN5 network, its FC part contains parameters than the other networks.

---

Under review as a conference paper at ICLR 2021
Additionally, a DYRNN5 model was trained for different sampling rates for the Circuit dataset to examine its ability to generalize to other sampling rates. For this, the dataset was subsampled at different frequencies using linear interpolation. The constellation of sampling rates in these three runs are described in Table All networks were trained with the Adam Optimizer by Kingma & Ba (2014) and the Huber Loss function with standard parameters defined in Tensorflow. The amount of training iterations was 3000 epochs. Evaluation of results is done on a separate testing dataset.
Table 1: Amount of parameters per network layer and recurrent layer type.
Run
Training set resampled at
Prediction set resampled at
A
0.7 Atorig
1.0 Atorig
B
1.0 Atorig
0.7 At Atorig
C
0.5 Morig. 1.0 Atorig
0.7 Morig
After training the DYRNN models are transformed into Laplace transfer functions G(s) using Sympy (Meurer et al. (2017) ) and three different views of this function are generated: G(s) over s, the frequency response ((w)) over w and the root locus curve of G(iw) over w to compare the different model dynamics per dataset.
5 RESULTS
After 10 runs each, the different recurrent networks' performance on the testing set is shown in Figure Since an activation function in the FC layer was omitted, it amounts a linear combination of all channels from the cascaded part. Therefore, it can be assumed that better performance of the DYRNN stems from its computation functions. In this case different network configuration and more parameters may enable the other layer types to achieve better results.
Section A.3 in the Appendix shows the best and worst runs on the testing set for all datasets and layer types well as the transfer functions learnt by DYRNN3 rounded to two significant digits in Section A.4 The transfer function analysis plots in Figures runs which performed best the testing datasets.
and were generated for the
On the Circuit dataset, both DYRNNS and DYRNN3 perform consistently better than each of the other recurrent networks. This is to be expected since an idealized capacitor acts PTI element, so the DYRNN is predestined for this modelling task. Nonetheless, all layer types perform well on the dataset if training a meaningful way on some runs (Appendix Figure T4p. Figure [0] shows that G(s) for DYRNN5 has more poles. This is to be expected because of the structure of the network, but can be an indicator for instability in some frequency areas. G(s) of DYRNN3 follows the given plots better. The Frequency responses of both DYRNN networks are similar to the actual system, but the root locus curve of DYRNN5 differs significantly from the actual one: its starting point is far outside of the plotted region.
successful. The RNN network, for example, was unable to fit the data in
On the Heating dataset, the GRU network performed slightly better than DYRNN3 and DYRNNS.
noticeable that, while the datasets is visually similar to a PT1 element, the DYRNN3 model displays similar frequency response plot and root locus curve in Figure Ilb) and c) seen in the actual model in the PT1 system in the Circuit dataset (Figure I0).
On the Thermic dataset, DYRNN3 performed similarly to GRU. The dataset consists of two inputs and one output, yielding two transfer functions G1(s) and G2(s). These can be analysed separately shown in Figure 12 for DYRNN3. While G(s) over is similar to the one in Circuit, the frequency response and root locus of both (s and (2(s) are very different from Circuit and Heating.
In total, DYRNN5 performed worse than DYRNN3 the non-simulated datasets. We assume that the reason for this is that Heating and Thermic do not have a strict zero-level at the start and the end of the datasets' input and output. That might enforce more stable model.
Our results concerning resampling to different sampling rates (in show that training for 0.7 At and predicting 1.0 AL performs worse than the other two variants. From this, it can be concluded that
7

---

Under review as a conference paper at ICLR 2021
the total amount of time covered by the training data is more important than the training different sampling rates for this dataset. also shows that the model able to extrapolate for each of the different sampling rates another. Our implementation, which allows the per batch sampling rate during training is computationally more expensive than the one which relies upon fixed sampling rate, the inexpensive version possible alternative for larger datasets.
Test set MSE
10
10
10-2
10-5
10
10-3
10-8
10
RNN
LSTM
GRU
DYRNN3
DYRNNS
RNN
LSTM
GRU
DYRNN3
DYRNNS
RNN
LSTM
GRU
DYRNN3
DYRNN5
(a) Capacitor Circuit
(b) Heating System
(c) Thermic Resistance
Figure 8: Results on different evaluation datasets after 10 runs each using the same architecture with
varying layer types.
Table 2: MSE and Standard deviation of the different layer types the experiment datasets.
RNN
LSTM
GRU
DYRNN3
DYRNNS
Circuit
4.4e-3+t7.6e-3
3.8e-4 +1.3e-4
1.3e-4 + 9.2e-5
9.3e-6 7
9.0e-6 +: 1.2e-5
Heating 3.6e-2 + 2.8e-2
1.3e-2 + 1.4e-2
3.7e-3+ 1.9e-3
4.5e-3 + 6.0e-4
4.4e-3+ 3.1e-4
Thermic 5.0e-3 1.7e-3
2.7e-3 +2.2e-4
2.3e-3+ 4.4e-4
2.3e-31 6.3e-4
1.9e-2+ 1.8e-2
Test set MSE
10 2
10-4
A
B
C
10-6
Mean Loss
1.43e-3
4.76e-06 2.21e-06
5.63e-06
A
B
C
Loss Std.-dev.
1.4e-3
7.11e-06
Experiment Version
Figure 9: Results the A: 0.7Alorig 0.5Atorig 1.0Atorig - 0.7Atorig
1.0Aoorigi
B: 1.0Aborig - 0.7Alorigi C:
FUTURE WORK
We have shown new type of recurrent layer designed for the modelling of dynamic systems. Based on this, we see several potential areas for future work. One point that was not part of this work was the in depth analysis of the learnt transfer functions, and how the transfer function is to be interpreted in case of non-linear activation functions in the network. The plots shown of the transfer functions are accessible and interpretable mainly for engineers. Another area of research would be on how to make these results interpretable for scientists without engineering background.
Our experiment showed competitive results on three datasets of system identification. A different area of application be model based reinforcement learning tasks, since the layers' computation blocks are also commonly used in control engineering.
CONCLUDING REMARKS
In this work present recurrent network architecture which is based dynamic systems theory. We show that the leannt system dynamics can produce models which can extrapolate to dif- ferent sampling rates in the data. Due the specific meaning of the cells' computation function, the
8

---

Under review as a conference paper at ICLR 2021
400
ModeG Madeb
Mode5 Aztual
200
001
Nodels
Actual
Modoll Start
Nodel' Start
Aztual Start
20
10
60
(a) Transfer Function G(s) over
(b) Frequency Response
(c) Root Locus Curve
Figure 10: Circuit model transfer function. Actual data could be computed since the resistance and capacitance of the circuit known.
600
Madell
Mode Mode
1.00
Models
Midelt
X
Model Start
-10
0.5
600
-15
0.00
-25
-0.50
-200
-30
-0.75
-1.00
-0.25
0.50
(a) Transfer Function G(s) over
(b) Frequency Response
(c) Root Locus Curve
Figure 11: Heating system transfer function for DYRNN3 and DYRNN5. Even though the models are similar performance, differences can be seen using this advanced visualization.
400
co
25
7.5
200
0.0
-2.5
(1/kal
G2/wl
G2tkal Start
200
-10
10.00
20
30
He
(a) Transfer Function G(s) over s
(b) Frequency Response
(c) Root Locus Curve
Figure 12: DYRNN3 results on the Thermic dataset. Since the system has two different inputs, the model provides one transfer function each.
leamt model can be leveraged to gain insight on the underlying system dynamics and differential equation model of the system can be extracted.
REFERENCES
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Dif- ferential Equations, 2018. URL Un.htttp//aarwivv.rr///////// 000.:/0000//10000076665

---

Under review as a conference paper at ICLR 2021
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Ceroodder--ecdeere for Statistical Machine Translation: GRU, 2014. URL
1078v3
De Moor B.L.R. DalSy: Database for the Identification of Systems: Heat flow density through two layer wall, Thermic Systems, [99-011], URLphttp:/hhomss.co esat www.eeeeessstt.....ee.e...... be/
smm// daisy/
De Moor B.L.R. DalSy: Database for the Identification of Systems: Heating system, Thermic Systems, [99-001], b. hotthhtt/////ttt esat
wtwpi//en.eeesmmmddaaaaaaaa daisy/
Otto Föllinger, Ulrich Konigorski, Boris Lohmann, Günter Roppenecker, and Ansgar Trächtler. Regelungstechmik: Einführung in die Methoden und ihre Anwendung. Lehrbuch Studium. VDE Verlag GmbH, Berlin and Offenbach, 12., überarbeitete auflage edition, 2016. ISBN 978-3-8007 4201-1.
Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 175, 1997. ISSN 0899-7667. doi: 10.1 166////co.1997.99 1...9.9...177. 1735.
Rolf Isermann and Marco Münchhof. Identification of Dynamic Systems. Springer Berlin Heidel- berg, Berlin, Heidelberg, 2011. ISBN 978-3-540-78878-22 doi: 1111007978-3355--788777999
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, 2014. URL
httt:://axxv..ogg http://df/1411// 6980v9
Viktoriya Krakovna and Finale Deshi-Velez. Increasing the Interpretability of Recurrett Neural Networks Using Hidden Markov Models, 2016. URL http:://axxiv.vrg/ppd/////
05934v1
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Mur- ray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, 1lya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-Scale Ma- chine Learning on Heterogeneous Systems, 2015. URLhtttss http://wwttts.teeoo www.eeensofoo..oo org/
MATLAB. Version 9.5.0.9444144R221999 2018
Aaron Meurer, Christopher P. Smith, Mateusz Paprocki. Ondiej Čertik, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rath- nayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Štěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e January 2017. ISSN 2376-5992. ht1/ http://ooooorgg///// /10 771 http/pperrecce -ss...jjjsss.... 103
doi: 10.77177peerj-cs.. 103.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Des- maison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chin- tala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'xeetquutes Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024- 8035. Curran Associates, Inc, 2019. URL http: http:://pap.ss..e....ppppppppppp CC ctpppprrrccppp 9015
ibrary.
pdf
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informer neural networks: A deep leamn ing framework for solving forward and inverse problems involving nonlinear partial differen- tial equations. Journal of Computational Physics, 378:686-707, 2019. ISSN 00219991. doi:
10. 10.1006//.ccp.201.. 10.045.
10

---

Under review as a conference paper at ICLR 2021
Yu Wang. A new concept using LSTM Neural Networks for dynamic system identification. In 2017 American Control Conference ACC, pp. 5324-5323 5329. 2017. doi: 10229919/ACCC222177963222
Radha K. Rao Yarlagadda. Analog and digital signals and systems. Springer, New York, 2010. ISBN
978-1-4419-0033-3. URL http:
loc, http://.....p////ttttaaaaaaaaaaa
2009929744-d.html
A APPENDIX
BASE FUNCTIONS IN DYNAMICAL. SYSTEMS THEORY
In dynamic systems theory, the basic elements are visualized using their response towards an input unit step function (t) starting t=0 with the amplitude of 1. In Table we show short summary over the blocks used in our network layer. Blocks display a specific behaviour, which is visualized by looking the response towards the Unit step function.
A.1 IMPLEMENTATION DETAILS
Sequential models keep hidden state while iterating over an input sequence, to encode information necessary for the computation of the next step's s result.
There are two ways to implement the model: one where the sampling rate is kept constant between batches and one where the sampling rate can set per batch. While the first version can be imple- mented using trivial matrix multiplication, the per-batch version is implemented as follows:
pkk) = u(k) o |Kl
(22)
i(k) = i(k 1) +
i(k)
(23)
A/
К
d(k) (u(A) - u(k 1))
|Kpl
(24)
At
ptl (k) = pl(k - 1) + (u(k) |Krrl - ppf1 -1))
(25)
(TPTI +1
pd(k) ((())) + ((u(k) u(k 1)) At [TpD])
|KpDl.
(26)
(27)
With u(t)'s dimensionality of [batch x sample x input channels] and Af(t) as [batch x sample x 1] or At [batch x 1]. the multiplications expressed Einstein's 's sum formation (which can be implemented as described in e.g. as "bi, ij - bij', as "bi, - bi' and as bij, bij->bij', i.e. the Hadamard product operator. The dimensions of the trainable parameters result to
docs/pythow/tf/enssm/) are:
Kr E Rexo
K₁ E Rixx
KD E piexo
Kyrı E RRiexoe Trm Rixxo
KPD E Riexoe TrD E Ric
given this setup. The weights P.D.PTI PD were initialized random uniform distribution, with range 0.2). Its important to initialize I such that the first iteration does not become unstable. For datasets with around 500 elements, the initialization used in the experiments was in the range of [110.0At, 110.0A1].
A.2 FURTHER EXPERIMENT DOCUMENTATION
We evaluated our network on three different datasets. One is our own dataset from a Matlab (MAT- LAB (018)) simulation shown in Figure 13 The input of our model is the input voltage of the

---

Under review as a conference paper at ICLR 2021
Table 3: Basic linear components, input (t) and system response y(t) Step response and differential equations of the used linear components.
System
Unit Step
Function
Re-
General Block Output Func- tion
Laplace Form
sponse
P
Amplitude
v(1) K u(l)) (12)
Y(s) K U(s)
(13)
Amplitude
5(t) = K - u(t)
(14)
Y(8) = U(s)/s
1
(15)
Amplitude
5(6) K -(t)) (16)
Y(s) U(s)
(17)
PT1
Amplitude
T T-)(1+ v()) K u(l)
K
Y(s) = U(s)
1
(18)
(1+T T-T
(19)
PD
Amplitude
y(t) = K (ū(t) T + u(t))
-(s)= = U(s) K (1+T.5)
1
(20) (21)
circuit, while the output value is the voltage measured over the capacitor. The simulation's sampling rate is 0.005s. We simulated three time series for training, validation and testing with length of seconds. The input consists of square signal with randomly changing amplitudes from 0.5s 1. 1.5s. The remainder of the time, the input voltage is OV.
The other Thermic) and the "Heating system" (De Moor B.L.R. (6) (in short Heating) datasets from the DalSy system identification benchmarks:
the "Heat flow density through a two layer wall" (De Moor B.L.R. (a) (in short
Heating System Benchmark: Prediction of halogen lamp temperature based on input volt- age
12

---

Under review as a conference paper at ICLR 2021
1
Signal 1
0.1F
Discrate
0. 10000
powergui
Figure 13: Simulation used for the generation of experiment data. It contains a resistor of 2 Ω and capacitor of O.IF. Input sequence is the source voltage, while output to predict is the voltage measured over the capacitor.
Two Walls Benchmark: Prediction of heat flow through two walls based on measurements before and between the walls
After training, the models were evaluated testing set split from the complete dataset. The DalSy Datasets do not have a zero-level starting interval like in our circuit dataset. For simplicity's 's sake, chosen network architecture does not take the initial value of the output time series into account. Therefore an offset could be seen in predictions on the testing set for all layer types. In a post processing step this offset computed using the median difference between prediction and label data, and subtracted from the prediction prior to the final MSE evaluation.
The amount of units and parameters for all experiments shown in the Tables
Table 4: Amount of units per network layer and recurrent layer type for the the Circuit dataset.
Values in parentheses denote the parameter count of that layer.
Units (Param. Count)
RNN
LSTM
GRU
DYRNNS
DYRNN3
Units in first recurrent layer
3 (15)
2 (32)
2 (30)
1 output per
1 output per S.c. (5)
s.c. (7)
Units in second recurrent layer
5 (45)
(40)
(36)
1 output per
1 output per
s.c. (35)
S.c. (15)
Number of parameters in FC layer
10
32
14
Number of parameters total
70
78
72
74
34
Table Amount units per network layer and recurrent layer type for the Heating System dataset.
Units (Param. Count)
RNN
LSTM
GRU
DYRNNS
DYRNN3
Units in first recurrent layer
3 (15)
2 (32)
2 (30)
2 output s.u. (14)
output per S.u. (5)
Units in second recurrent layer
7 (77)
3 (72)
3 (63)
1 output per
I output per
s.u. (70)
s.u. (15)
Number of parameters in FC layer
12
63
14
Number of parameters total
104
111
100
147
34
13

---

Under review as a conference paper at ICLR 2021
Table 6: Amount of units per network layer and recurrent layer type for the Thermic Wall Res. dataset.
Units (Param. Count)
RNN
LSTM
GRU
DYRNN5
DYRNN3
Units in first recurrent layer
(18)
(40)
(36)
1 output per s.u. (14)
output per S.u. (10)
Units in second recurrent layer
(77)
(72)
(63)
1 output per
output per
s.u. (70)
s.u. (30)
Number of parameters FC layer
63
27
Number of parameters total
108
120
107
147
67
A.3
BEST AND WORST RESULTS ON DATASETS
The Figures 14
15
and
16
show the best and worst testing results of all 10 runs per network type.
0.5
0.2
Input RNN
Capacitor Voltage
LSTM
-0.2
GRU
DYRNN3
-0.4
DYRNN5
00.
Label
-0.6
2
0
2
time S
time
(a) Best results after 10 runs.
(b) Worst results after 10 runs.
Figure 14: Comparison of best and worst models the Circuit dataset.
0.8
Input RNN
0.5
LSTM
Output Heat
0.6
GRU
DYRNN3
DYRNN5
0.4
Label
2
1
2
time
time
(a) Best results after 10 runs.
(b) Worst results after 10 runs.
Figure 15: Comparison of best and worst models on the Heating dataset.
14

---

Under review as a conference paper at ICLR 2021
Input1 RNN
Heat behindehinn second Wall
0.5
0.5
LSTM
GRU DY N3
-0.5
DY N5
Label
-0.5
0
2
3
time s
time
(a) Best results after 10 runs.
(b) Worst results after 10 runs.
Figure 16: Comparison of best and worst models on the Thermic dataset.
A.4 RESULTING TRANSFER FUNCTIONS
Below the transfer functions of the best runs are documented. The transformation to differential equations is trivial. Notice that due to the rounding significant digits, the differential equations listed here are unlikely to yield accurate results. Additionally, simplification of these transfer functions is possible if the poles and zeros the function match, but this is outside of the scope of this work. The transfer functions of the best runs of DYRNN3 are as follows:
Circuit:
9.2E
10s
3.2E - 75 + 3.6E - 5s 0.0021s 0.75s + 3.9s 5.0
G(s)
(28)
1.6e 554 + 0.03s 0.3s² 0.98s 1.0
Heating:
G(s) 2296 -97 0.04s + 0.85s + 3.8 + 4.5s + 2.5s + 0.68s + 0.073
(29)
3.6s5 + 14.0s4 + 14.0s + 6.2s² + 1.3s 0.099
Thermic Resistance:
G(s) Gi(s) + G2(s)
(30)
Yi(s) (s) G)(s)
(31)
Y²(x) X2(x) G2(a)
(32)
(s(s) -0.0025 0. 0.12 2.1 431.0s 40.0s²
7.9s 1.8
(33)
2.3 + 13.0s + 4.33s 7.3s
G2(s) 2.2F 85 0.0013.
(0.015+
0.4s² 2.0s 1.5
(34)
7.6E 7s + 0.045s 0.06s 0.69s 1.0
This sum transfer functions can used to simulate y(t) by first simulating (t( and y2(t)
separatly and then building the of the results.
15


Reviews:
[{"review_id": "tGHf_ON5HBh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The theory of dynamical systems has such a rich and extensive history dating to more than a century ago, that Poincaré himself would marvel to see what we can do now with modern technology.  Perhaps he’d not be pleased to see that extensive research in dynamical systems from the previous century is ignored in this paper, however the authors are aiming at reaching more interpretability on NN models as stated in the abstract.\n\nI think the paper has definitely technical merits, but I remain puzzled with the lack interpretability section (3.4). I do not see how I can use the methodology to reconstruct the underlying physical processes that drive the physical phenomena. Since the authors are claiming in the abstract that interpretability is a distinctive feature of the methodology, I was expecting to see a full fleshed content, but it’s essentially just a couple of paragraphs.\n\nThese are the questions that I’d love the authors to address:\n1/ Since the authors are inserting an explicit integration method by virtue of the Euler step through Eqs 3 to 8, the method inherently carries A-unstable. Can you estimate the conditions for any system that would prevent the A-stability risk?\n2/ What are the differences of the methodology for Hamiltonian and dissipative dynamical systems? \n3/ If the methodology that the authors are proposing works, it should be the case that any transient trajectory can be explained by DYRNN. It means that a trained DYRNN on a small set of trajectories (or one sufficiently large one if we have an ergodic system), can fully reproduce any other transient trajectory of the dynamical system. In other words, produce a train and test trajectories and prove that the error on the test set is small enough.\n4/ At the core of classical dynamical system theory is understanding the set of local and global bifurcations that explain the physical/chemical/biological phenomena. If DYRNN can predict the bifurcation points respect to a control parameter, then the method is rock&roll, otherwise it’s just another method.\n5/ Interpretability: If the authors can map back the DYRNN to a set of ODEs/PDEs then I’d be fully convinced on the value of the method. Otherwise, I remain unconvinced on the value of the interpretability.\n\nI am not ready to accept this paper in its current form, but if the authors could prove (1,2) or 3 or 4 or 5 then I’ll happily change my rating.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "I3rNKMigZLP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nThis paper aims at defining a new architecture, Dynamic Recurrent Neural Network, that would be based on discrete differential equations of basic linear system transfer functions known from dynamic system identifiation. They show also an application example.\n\nThe paper is correctly written, and the subject is of interest. But I don't really see the point as it looks like the method removes all of what is 'recurrent', as no weights from the new unit are learned during the training. The learning is only on some fully connected layers afterwards: I feel that the authors just made an (interesting) 'feature extraction' instead of a recurrent unit, by using standard transfer functions without any weights to be learned (or haven't I understood correctly?).\n\nThe claim that the paper is the first to present a method where the sequence sampling rate, or 'delta t', is a parameter that can be modified without needed to re-train the network seems odd. In fact, I am not very familiar with this field but it looks to me that a lot of works that are now combining neural nets and dynamical systems are by definition able to do that. For example, I know this work 'Learning Dynamical Systems from Partial Observations', Ayed et al. 2019, which has a section called 'Benefits of Continuous-Time.', where I can read 'this allows us to accommodate irregularly acquired observations, and as demonstrated by the experiments, allows interpolation between observations.' So it is not something 'new' to the community and I would guess that papers closer to yours would also have the same feature?\n\nQuestions/ remarks:\n- please explain better how your network is still recurrent : where is the 'hidden memory' that is passed? And where are the weights inside the DYRNN that are learned during the training? For me it looks like only the FC model placed after, that is moreover without any non-linearity (ReLU, etc), contains weights that are updated during training.\n- 'and are state of the art layer types for text based sequence-to-sequence problems like machine translation or text processing --> no, the recurrent NN are not state of the art in translation anymore... transformers are. But RNN could be state of the art in other problems.\n- 'The number of output channels per layer n layer amounts to (base component count * n oc = 5 * n oc ), as shown in Figure 3b.' --> not clear: there is 5 output channels, so noc = 5, but what is this 'base component count'? Why do we have to multiply it, and what is a layer? Or do you mean that there are 5 DYRNN units? or that every 'component' (P, I, ..) outputs the 5 outputs? This is not what is represented here.\n- Fig. 6 might not be necessary, we can understand the concept without it.\n- Modell --> Model\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fk8xgggCKGE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims at proposing Dynamic Recurrent Network to understand the underlying system properties of RNNs. By first showing five basic linear transfer functions in dynamic systems theory, the paper formulates DYRNN units. To solve the increasing number of layers issue, they concatenate inputs and intermediate results before passing into an FC layer. It is interesting to see how adjusting\n\\deta t is related to the model’s robustness. Though not fully explained, this paper provides a method to partially explain the RNN insights through FC layers learnt weights.  \n\nThe paper is well written to convey the central ideas. The overall idea is interesting, and experiments are clear to demonstrate the proposed method. It will be better to test the method on some benchmark datasets so that it will be easy to compare with the state-of-the-art.\n\nA small advice: do you mean varying instead of variing in section 6?", "extra_scores": {}, "extra_text": {}}, {"review_id": "3oSIzHJI2HE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a method for incorporating basic concepts from linear systems theory into the standard structure for training artificial neural networks. They compare results of their approach against standard approaches for 3 simple datasets.\n\nBroadly speaking the work is quite unclear, and takes several passes over to have a basic sense of the approach. There are too many shortcomings to enumerate them all, so I will just present one. Figure 5 is presented in the section \"example architecture\" which might lead one to believe the authors implement this network (which it appears they do not). I believe this is included only to indicate a hypothetical architecture, but the presentation is too poor to glean this with any quickness. This is of course, in itself, not sufficient grounds for rejection, but speaks broadly to the poor presentation of the work. It does not seem ready for publication.\n\nAs for the significance, the work clearly falls short. Although the motivation of constructing a \"more explainable model\" is a good one, this should not come at an extreme cost of model expressivity. It seems obvious that richer models, such as LSTMs etc., correctly trained, should be able to account for the linear transformations the authors include in their \"novel layer.\" That their work is competitive with these richer models is simply an indication of the simplicity of the tasks they chose, which (as far as I can tell) can all be accounted for using linear systems analysis (although it's hard to say, since they work so poorly explains the second two tasks). It's completely unclear how effective the authors' approach would be over standard, richer models, on tasks that cannot be accounted for by linear systems analysis, and I am doubtful that the suggested approach could offer much over these richer models.\n\nLikewise, an alternative view of the authors' work is as a learnable filter bank applied to data to create a representation of the data better suited for post-hoc learning with a richer model, which is certainly an useful idea, but it is not clear to me (and the authors haven't shown) that their choice for this filter-bank is superior to many other choices (e.g. convolutional layers applied prior to FC layers, which is standard for deep networks).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new RNN architecture called Dynamic RNN which is based on dynamic system identification.

Reviewers questioned the expressivity of the proposed model, practical application/impact of the proposed model, and interpretability of the proposed model. Even though the authors attempted to convince the reviewers, 3 out of 4 reviewers think that this work is not ready for publication. 

Specifically, R4 recommends 5 ways to strengthen the paper. I recommend the authors to incorporate this feedback and make a stronger resubmission in the future.
Decision: Reject