Recurrent Neural Network Architecture based on Dynamic Systems Theory for Data Driven Modelling of Complex Physical Systems

1 INTRODUCTION . Dynamic systems occur in many different areas of life ( Isermann & Münchhof ( 2011 ) ) . From biology , engineering , medicine to economics and more : Often , if a system changes its state based on a external input , this system can be viewed as a dynamic system . Dynamic system identification is the process of modelling the system ’ s properties . Such models can be used , for example , for anomaly detection , controller design or outcome prediction . For linear systems , this identification task is already well understood and state of the art methods exist . However , if a system exhibits non-linear behaviour , for example slip-stick-effects due to mechanical friction , the applicability of these methods is limited . In this case different approaches implemented in the state of the art range from white-box to black-box models . Generally , increasing system complexity raises the need for more powerful and often less understandable model architectures in order to produce satisfactory results : White box ( based on differential equations or numerical simulations of the physical system components ) , black box systems ( like Gaussian processes , deep neural networks , Support Vector Machines ) and grey box models , which often employ a mix of linear and non-linear building blocks . One example of a tool used in engineering are Hammerstein-Wiener models which are a combination of linear and ( prior known ) non-linear equations ( shown in Figure 1 ) . The linear model parameters are determined based on the training data . The non-linear behaviour of models is modeled using lookup tables or user defined non-linear functions . In this work we present a new type of recurrent neural network layer called the Dynamic Recurrent Neural Network ( DYRNN ) . It is designed for data based modelling of dynamic systems in a sequence-to-sequence manner based on input ( x ( t ) ) and output ( y ( t ) ) data . With it , we intend to bridge the gap between dynamic systems theory and recurrent neural networks . The layer ’ s internal computation is based on elemental transfer blocks from linear system identification . By combining it with non-linear neural networks , a Hammerstein-Wiener style model is emulated . This way , the model can offer additional knowledge about the examined system ’ s internal properties . Furthermore , while the model is trained on sampled data of one sampling rate it can be applied to data of the same system at a different sampling rate . This can be used to check the robustness of the model or to save time during training . We show that our network produces results which are better than or comparable to other recurrent networks ( RNN , LSTM , GRU ) on three different problem datasets . Since the layer can be implemented to be compatible to current deep learning frameworks , it can be combined with state of the art neural network layers ( like convolutional or fully connected layers ) and training techniques . 2 RELATED WORK . Dynamic system identification can be viewed as a sequence-to-sequence task of the modelling of a systems ’ output based on certain inputs . Isermann & Münchhof ( 2011 ) , for example , list several different tools like ARIMA processes for linear systems and multiple neural network architectures for non-linear systems . Examples for the latter are locally recurrent locally feedforward networks ( LRGF ) , Multi Layer Perceptrons ( MLP ) and Radial Basis Function ( RBF ) networks of different types of dynamics . These model structures are generalized , however , and as we will show , further theoretical background on linear systems theory could be leveraged . Generally , deep learning offers multiple neural network layer types that can be employed when dealing with sequence-to-sequence problems , like fully connected ( FC ) networks , convolutional networks ( CNN ) or recurrent networks . Recurrent networks are also known as sequential models ( like RNN , LSTM by Hochreiter & Schmidhuber ( 1997 ) and GRU by Cho et al . ( 2014 ) ) and have been used successfully for text based sequence-to-sequence problems like machine translation or text processing . Wang ( 2017 ) demonstrates a concept of LSTM for dynamic system identification by using several parallel LSTM layers which predict the systems behaviour based on its input and prior predictions and their derivatives . A different approach of modelling dynamic systems are neural ordinary differential equations ( ODEs ) by Chen et al . ( 2018 ) . These networks learn dy/dt of a function f with y ( t ) = f ( x ( t ) ) and the resulting ODE model is used an numerical integrator/solver ( like Runge-Kutta ) to compute y ( t ) . This has the advantage of a varying sampling step size which is determined by the solver , but these methods are agnostic of dynamic systems theory knowledge . Similarly , Raissi et al . ( 2019 ) use deep learning to learn Partial Differential Equations ( PDE ) of physical systems in a FC model combined with a numerical integrator . Furthermore , since the evaluation of ODE/PDE models is done using a numerical integrator , the model is difficult to apply in combination with other neural network layers like for example convolutional or recurrent layers . In terms of sampling frequency of the measurement data , recurrent network architectures can only be trained on one specific data frequency , and do not provide the functionality to generalize to other sampling rates of the same system . In such a case one would have to resample the new data to the frequency of the training set . Explainability approaches for sequential models in text processing deduce which parts of the sentences are relevant for the model ’ s prediction based on the activation of the internal gates ( as shown by e.g.Krakovna & Doshi-Velez ( 2016 ) ) . Interpretability of RNN/LSTM or GRU models for continuous measurement data has not been explored yet to our knowledge . 3 DYNAMIC RECURRENT NETWORK . The complexity of the modelling of dynamic systems does not only result from the potential nonlinearity , but also from the fact that the model has to keep track of the system ’ s current and past states in order to predict the output based on new input . We intend to model a dynamic system in a sequence-to-sequence task , which is why we chose a recurrent network architecture . Recurrent neural networks can be seen as units which iterate over a given sequence and predict an output . During this computation , a hidden state is computed which is leveraged for the prediction of the next time step . Our network differs from other state of the art recurrent networks in its computation function , which is derived from linear dynamic systems theory . In the following we explain the theoretical background used in this work . Then we describe the structure of our DYRNN . Finally , we show different advantages that result from this structure like training and prediction at different signal sampling rates and interpretability of the models . 3.1 LAYER STRUCTURE . The background knowledge in this section is covered by Föllinger et al . ( 2016 ) and Yarlagadda ( 2010 ) . In dynamic systems theory a linear system with the external input u ( t ) and resulting system output y ( t ) is expressed as the differential equation y ( t ) + a1 · ẏ ( t ) + . . .+ an · y ( n ) ( t ) = b0 · u ( t ) + b1 · u̇ ( t ) + . . .+ bn · u ( n ) ( t ) . ( 1 ) Therefore , a linear system acts as transformation of the input u ( t ) based to a convolution ( ∗ ) with the system ’ s linear transfer function g ( t ) to produce the system output y ( t ) with y ( t ) = u ( t ) ∗ g ( t ) = ∫ u ( τ ) g ( t− τ ) dτ . ( 2 ) The transfer function g ( t ) is used in engineering for , amongst others , controller design , system stability analysis or frequency response estimation . Larger , more complicated transfer functions can be seen as several basic functions which are interconnected in a circuit type fashion in parallel or in series ( see Figure 3 ) . Dynamic systems theory identifies five basic linear transfer functions from which all linear dynamic systems can be modeled , which we use in the construction of the DYRNN : P , I , D , PT1 and PD . For further information and and visualization see Appendix Section A ) . They have the following functionalities : • P : Proportional gain of the input , implemented as a multiplication with a constant • I : Integrating component , meaning the step-wise summation of an input over time . • D : Differential component acting as a high-pass filter • PT1 : Proportional multiplication of input with time delay , used to model e.g . voltage in capacitors in RC circuits . This function also acts as a low-pass filter • PD : Proportional increase with a differential component In the following equations K stands for a constant with influence on the output amplitude , while T is a time constant which influences the speed of the system ’ s reaction . K and T act as the trainable weights in a DYRNN layer . For usage in a recurrent layer , these differential equations are discretized with – in our case – first degree forward Euler . This is common in many engineering applications , and means replacing ẏ ( t ) with ẏ ( t ) = y ( k ) − y ( k − 1 ) ∆t . ( 3 ) This results in discrete recurrence equations , with the sample number k and the time distance between samples ∆t . The respective equations of the basic transfer functions are as follows : p ( k ) = KP · x ( k ) ( 4 ) i ( k ) = i ( k − 1 ) + ∆t KI · x ( k ) ( 5 ) d ( k ) = KD ∆t · ( x ( k ) − x ( k − 1 ) ) ( 6 ) pt1 ( k ) = pt1 ( k − 1 ) + ( KPT1 · x ( k ) − pt1 ( k − 1 ) ) · ∆t ∆t+ TPT1 ( 7 ) pd ( k ) = KPD · ( x ( k ) + TPD ∆t · ( x ( k ) − x ( k − 1 ) ) ) , ( 8 ) with the input x ( k ) and all K , T > 0 . The equations above are implemented as the computation function of a recurrent network layer as described in Appendix Section A.1 . K and T in the equations become trainable weights of the layer , while the hidden state consists of x ( k − 1 ) , i ( k − 1 ) and pt ( k − 1 ) . In the following we refer to these internal computations as subcomponents . We explore two different network variants in our work : The DYRNN5 with all five known subcomponents and the DYRNN3 with just P , PD and PT1 . The reason for this is , that a D subcomponent can be approximated by PD and I by a PT . Since integrators can also cause model instabilites , we explore both variants in our experiments . We formulate one unit ’ s computations based on the number of input channels nic and the number of outputs per subcomponent type noc . The number of output channels per layer nlayer amounts to ( subcomponent count ) * noc = 3 * noc for DYRNN3 and 5 * noc for DYRNN5 ( see Figure 4a ) . The fact that DYRNN can be implemented with an interface compatible with other recurrent networks allows the modelling of a systems properties by training in a sequence-to-sequence fashion using out of the box backpropagation through time , gradient descent and optimizer functions as implemented in state of the art frameworks like Pytorch ( by Paszke et al . ( 2019 ) ) and Tensorflow ( by Martı́n Abadi et al . ( 2015 ) ) . Since each input of the layer is connected to five basic components , stacking DYRNN layers results in an increasing number of output channels , as shown in Figure 4b . In our experiments , we achieved good results with two cascading DYRNN layers followed by a per-time step linear FC layer with one neuron . This final layer can be enhanced using non-linear activation functions and more layers/neurons , which would result in a similar structure to Hammerstein-Wiener models for modelling of static non-linearities in the data . In case of noisy input data , for example due to sensor noise , additional CNN layers could be used as well .

Summary: This paper aims at proposing Dynamic Recurrent Network to understand the underlying system properties of RNNs. By first showing five basic linear transfer functions in dynamic systems theory, the paper formulates DYRNN units. To solve the increasing number of layers issue, they concatenate inputs and intermediate results before passing into an FC layer. It is interesting to see how adjusting