## 3 Historically consistent neural networks (HCNNs)
### Applying Echo State Networks to Time Series Forecasting
Compute the linear readout weights \(\mathbf{W}^{\text{out}}\) from the reservoir, minimizing the MSE between \(\mathbf{y}(n)\) and \(\mathbf{y}^{\text{target}}(n)\);
4. Use the trained network on new input data \(\mathbf{u}(n)\) to compute \(\mathbf{y}(n)\) by using the trained output weights \(\mathbf{W}^{\text{out}}\). **Producing a reservoir. ** At the same time, the reservoir acts as a nonlinear expansion and as a memory of input \(\mathbf{u}(n)\). The reservoir can be described as a nonlinear, high-dimensional expansion \(\mathbf{x}(n)\) of the input signal \(\mathbf{u}(n)\). For classification tasks, input data \(\mathbf{u}(n)\) that are not linearly separable in the original space \(\mathbb{R}^{N_{u}}\) often become so in the expanded space \(\mathbb{R}^{N_{x}}\) of \(\mathbf{x}(n)\), where they are separated by \(\mathbf{W}^{\text{out}}\). _Reservoir's global parameters._ Given the RNN models (33) and (34), the reservoir is defined by the tuple \((\mathbf{W}^{\text{in}},\mathbf{W},\alpha)\). The input and recurrent connection matrices \(\mathbf{W}^{\text{in}}\) and \(\mathbf{W}\) are generated randomly. The leaking rate \(\alpha\) of the reservoir nodes in (34) can be regarded as the speed of the reservoir update dynamics discretized in time. **Setup for parameter selection**. Learning the results is fast in ESNs. This should be leveraged to evaluate how well a reservoir is generated by a particular set of parameters. To evaluate a reservoir, we train the output (35) and measure its error by applying cross-validation or training error. Randomly generated buckets, even with the same parameters, vary slightly in their performance. Keep the random seed fixed and averaged over multiple reservoir samples to eliminate random fluctuations in performance. **ESN ensemble**. Training many small ESNs in parallel and averaging their outputs, in some cases, has drastically improved the performance of the basic ESN approach [164; 165]. **Removing initial transient**. Usually \(\mathbf{x}(n)\) data from the beginning of a long training sequence are discarded (i.e., not used for learning \(\mathbf{W}^{\text{out}}\)) since they are contaminated by initial transients. The initial transient is a result of an arbitrary setting of \(\mathbf{x}(0)\), which is typically \(\mathbf{x}(0)=\mathbf{0}\). An unnatural initial state is introduced that is not normally visited once  
Figure 8: An echo state network (ESN) in schematized form. \(\mathbf{u}(n)\) is the training input signal, 1 is a constant input signal (intercept), \(\mathbf{W}^{\text{in}}\) contains input weights while \(\mathbf{W}\) contains recurrent weights (both \(\mathbf{W}^{\text{in}}\) and \(\mathbf{W}\) in the reservoir remain fixed after initialization), \(\mathbf{x}(n)\) contains neuronal activations and is the reservoir’s output. \(\mathbf{W}^{\text{out}}\) are trainable output weights, and \(\mathbf{y}(n)\) is the ESN’s output signal. \(\mathbf{W}^{\text{out}}\) weights minimize error \(E(\mathbf{y},\mathbf{y}^{\text{target}})\) after linear training. the network has been \"prepared\" for the task. The number of time steps to discard depends on the network memory, which in turn depends on the reservoir parameters, and are normally at the order of tens or hundreds. From this, we see that regular RNNs (discussed in Section 4.5) and ESNs have differing approaches when dealing with the uncertainty of the initial state in the dynamical system they are trying to identify: RNNs inject noise into the initial state in order to identify a stable dynamical system, while ESNs discard a few initial transient states, relying on their echo state property to achieve stability.