## 4 Brief Literature Survey on ANNs Applied to Time Series Modeling
### Time Series Forecasting with Recurrent Neural Networks  
[PERSON], [PERSON], and [PERSON] [96] propose a series of architectural modifications aimed at improving the performance of recurrent neural networks (RNNs) applied to time series forecasting tasks. Given the universal approximation properties of RNNs, they can be used to forecast time series in the form of nonlinear state space models [155]. [PERSON], [PERSON], and [PERSON] [96] rely on this general framework in order to incrementally build their models. They start out with a given RNN architecture, and then propose a refined version, seeking to correct empirically observed deficiencies found in the initial model. The RNN architectures discussed in their work are listed next:  
1. Basic time-delay RNNs in state space formulation, which model _open dynamical systems_ (i.e., partly autonomous and partly externally driven dynamical systems);
2. Error-correction neural networks (ECNNs), which refine the basic RNN model by adding an error-correction term in order to handle missing information from unknown external drivers of open dynamical systems;
3. Historically consistent neural networks (HCNNs), which refine ECNNs by internally modeling external drivers, thus transforming ECNNs (and basic RNNs) into _closed dynamical systems_;
4. Causal-retro-causal neural networks (CRCNNs), which refine HCNNs by incorporating into their usual information flow from past into future (causal flow) the effects of rational decision-making and planning via an information flow from future into past (retro-causal flow). Each model in this listing is useful in its own right for particular real-world applications. [PERSON], [PERSON], and [PERSON] [96] provide useful hints that facilitate the construction and training of these models. They also point out real-world scenarios where these models are employed; for instance, they mention that ECNNs have been employed successfully to forecast the demand for finished products and raw materials within the context of supply chain management. These architectures and their related algorithms have been implemented in a software system developed by Siemens Corporate Technology called simulation environment for neural networks (SENN). In the remainder of this subsection, we summarize the main points presented in [PERSON], [PERSON], and [PERSON] [96]. **1. Basic RNN. ** [PERSON], [PERSON], and [PERSON] [96] show that an open dynamical system can be used to create a vector time series \(\vec{y}_{\tau}\), which can be described in discrete time \(\tau\) using an output equation and a state transition [34]:  
\[\vec{s}_{\tau+1} =f(\vec{s}_{\tau},\vec{u}_{\tau}) \text{State transition} \tag{29}\] \[\vec{y}_{\tau} =g(\vec{s}_{\tau}) \text{Output equation}\]  
\(\vec{s}_{\tau}\) is the current hidden system state, \(\vec{s}_{\tau+1}\) is the upcoming system state, and \(\vec{u}_{\tau}\) represents external factors. This is called an open dynamical system. The data-driven system identification is based on the selected parameterized functions \(f()\) and \(g()\). Parameters in \(f()\) and \(g()\) are chosen such that an appropriate error function, such as \(\frac{1}{T}\sum_{\tau=1}^{T}\|\vec{y}_{\tau}-\vec{y}_{\tau}^{T}\|^{2}\), is minimized (\(\vec{y}_{\tau}^{T}\) are the target observations). Typically, without loss of generality, \(f(\vec{s}_{\tau},\vec{u}_{\tau})=\tanh(\vec{A}\vec{s}_{\tau}+\vec{B}\vec{u}_{ \tau})\) and \(g(\vec{s}_{\tau})=\vec{C}\vec{s}_{\tau}\); the hyperbolic tangent is the activation function in the network's hidden layer, while the output function is specified as a linear function; \(\vec{A}\) (autonomous dynamics or memory), \(\vec{B}\) (external factors), and \(\vec{C}\) are weight matrices that model the open dynamical system. The technique of finite unfolding in time [156] is employed in order to solve the selection of appropriate matrices \(\vec{A}\), \(\vec{B}\), and \(\vec{C}\) that minimize the error function.