## 3 Historically consistent neural networks (HCNNs)
### Applying Echo State Networks to Time Series Forecasting  
Training recurrent neural networks (RNNs) is a difficult task; however, they integrate a large dynamic memory and highly flexible computational capabilities, making them a very powerful tool. Error backpropagation (BP) is the standard method to train networks, especially feedforward neural networks (FFNNs), and it has also been extended to RNNs. This extension, however, has not been straightforward: RNNs are dynamical systems, and training them with BP sometimes leads to bifurcations, so chaos (non-convergence) occurs. Echo state networks (ESNs) are an alternative approach for training RNNs, as proposed by [PERSON] (2016). In the classical ESN approach, an RNN structure is called a _reservoir_, so ESN methodology is often known in the literature as reservoir computing (RC), which is, at the moment, a prolific research area in RNNs ([PERSON] and [PERSON], 2017). In [PERSON] (2017), practical techniques and recommendations for successfully applying ESNs are presented, with emphasis on the time series forecasting problem. [PERSON] (2017) points out that ESNs are conceptually simple and easy to implement, but experience and insight are a must for training them successfully. In the remainder of this subsection, we summarize the main points addressed in this important contribution made by [PERSON] (2017). **The basic ESN model**. ESNs are used to supervise temporal machine learning tasks where, for a given training input signal \(\mathbf{u}(n)\in\mathbb{R}^{N_{\mathbf{u}}}\), a desired target output signal \(\mathbf{y}^{\text{target}}(n)\in\mathbb{R}^{N_{\mathbf{y}}}\) is known. The discrete time is \(n=1,\ldots,T\) and the number of data points in the training dataset is \(T\). The task is to learn a model with output \(\mathbf{y}(n)\in\mathbb{R}^{N_{\mathbf{y}}}\), where \(\mathbf{y}(n)\) matches \(\mathbf{y}^{\text{target}}(n)\) as best as possible, minimizing an error measure \(E(\mathbf{y},\mathbf{y}^{\text{target}})\) and, importantly, generalizing well to unseen data. The error measure \(E\) is typically a mean-squared error (MSE). The update equations are  
\[\mathbf{\tilde{x}}(n)=\tanh\Bigl{(}\mathbf{W}^{\text{in}}[1;\mathbf{u}(n)]+ \mathbf{W}\mathbf{x}(n-1)\Bigr{)} \tag{33}\]  
and  
\[\mathbf{x}(n)=(1-\alpha)\mathbf{x}(n-1)+\alpha\mathbf{\tilde{x}}(n), \tag{34}\]  
where \(\mathbf{x}(n)\in\mathbb{R}^{N_{\mathbf{x}}}\) is a vector of reservoir neuron activations and \(\mathbf{\tilde{x}}(n)\in\mathbb{R}^{N_{\mathbf{x}}}\) is its update, all at time step \(n\), \(\tanh(\cdot)\) is applied element-wise, \([\cdot;\cdot]\) stands for a vertical vector (or matrix) concatenation, \(\mathbf{W}^{\text{in}}\in\mathbb{R}^{N_{\mathbf{x}}\times(1+N_{\mathbf{u}})}\) and \(\mathbf{W}\in\mathbb{R}^{N_{\mathbf{x}}\times N_{\mathbf{x}}}\) are the input and recurrent weight matrices, respectively, and \(\alpha\in(0,1]\) is the leaking rate. It is important to mention here one fundamental difference between regular RNNs and ESNs: while ESN's input and recurrent weight matrices \(\mathbf{W}^{\text{in}}\) and \(\mathbf{W}\) remain fixed once initialized, the weights in RNN's state transition matrices are updated after each iteration during the learning process. A graphical representation of an ESN is depicted in Figure 8.  
The linear readout layer is defined as  
\[\mathbf{y}(n)=\mathbf{W}^{\text{out}}[1;\mathbf{u}(n);\mathbf{x}(n)] \tag{35}\]  
where \(\mathbf{y}(n)\in\mathbb{R}^{N_{y}}\) is the network output, and \(\mathbf{W}^{\text{out}}\in\mathbb{R}^{N_{y}\times(1+N_{u}+N_{x})}\) is the output weight matrix. The RC algorithm introduced with ESNs by [PERSON] [162] consists of the following steps:  
1. Generate a random reservoir RNN \((\mathbf{W}^{\text{in}},\mathbf{W},\alpha)\);
2. Run the reservoir using the training input \(\mathbf{u}(n)\) and collect the corresponding reservoir activation states \(\mathbf{x}(n)\);
3.