2020 International Symposium on Nonlinear Theory and Its Applications,NOLTA2020, Virtual, November 16-19, 2020

Echo State Network with Chaos Noise for Time Series Prediction
Yoko Uwate†, Martin Schule ‡, Thomas Ott ‡ and Yoshifumi Nishio††Dept. of Electrical and Electronic Engineering, Tokushima University
2-1 Minami Josanjima, Tokushima, Japan
Tel: +81-88-656-7662, Fax: +81-88-656-7471
Email: {uwate, nishio}@ee.tokushima-u.ac.jp‡Institute of Applied Simulation, Zurich University of Applid Sciences
Einsiedlerstrasse 31a, 8820 Waedenswil, Switzerland
Email: {scli, thomas.ott}@zhaw.ch
Abstract—In this study, performance of chaos noise injected to Echo State Network for time series prediction isinvestigated. For the evaluation of the chaos noise, twoparameters of the logistic map are selected to produce different features as intermittency chaos and fully developed
chaos. By computer simulations, it is confirmed that thethree-periodic intermittency chaos noise is better performance than the fully developed chaos noise for time series
prediction.
1. Introduction
Today, every time we open a newspaper, the page contains AI characters, and new technologies that apply deeplearning are being developed everywhere in the industry.
However, deep neural networks have problems such as requirement of enormous amount of training data and difficult to implement in hardware. In recent years, research on
reservoir computing has been actively investigated in orderto facilitate hardware implementation and reduce the calculation cost. The network structure of reservoir computing
is composed of an input layer, a hidden layer called a reservoir, and an output layer. Representative examples of reservoir layers that play a particularly important role in learning are the Echo State Network (ESN) [1], [2], in which theneuron network is random and the weight is fixed, and theLiquid State Computing (LSC) [3], which uses the spiking
neuron model.
Jaeger et al. have proposed adding the state noise to theupdate equation of ESNs [4]. They confirm that adding
state noise is computationally more expensive, but appears
to have the additional benefit of stabilizing solutions inmodels with output feedback. Hayakawa et al. pointed
out the chaos near the three-periodic window of the logistic
map gains best performance of Hopfield Neural Networks(HNNs) for solving traveling salesman problems (TSPs)[5]. In our previous studies, we have proposed a methodof modeling chaos noise to prove the effectiveness of chaosnoise, and confirmed that the noise obtained by modelingshows the same performance as chaos noise [6], [7]. Therefore, we consider that chaotic noise is effective for ESNs.
We investigate the performance of chaos noise when it

is injected to the ESN for time series prediction. Computer
simulated results show that the intermittency chaos noise
gains good performance for long-term prediction of timeseries.
2. Echo State Network
Figure 1 shows an echo state network. ESNs are applied
to supervised machine learning tasks where for a given
training input signal u(n) and a desired target output signal ytarget (n) is known. The task of ESN is to minimize
the difference between the output value y(n) and the target
value ytarget (n). The error E is calculated by the following
equation which called a Mean-Square Error (MSE).
1X(yi (n) − ytarget(n))2iT n=1
T

E(y, ytarget ) =(1)where T is the number of data points in the training dataset.
The update equation of ESNs is described as follows.
x̃(n) =x(n) =tanh(Win [1; u(n)] + Wx(n − 1)),(1 − ε)x(n − 1) + εx̃(n)(2)

where x(n) is a vector of reservoir neuron activations andx̃(n) is its update with all at time step n and ε denotes theleaking rate.

Figure 1: An echo state network.
For the computer simulations, the parameter of the number of neurons in reservoir is set to 1,000 and the leakingrate (ε) is fixed with 0.3.

- 274 -

3. Chaos Noise

4. Time Series Prediction

In this section, we explain chaos noise injected to theESN. The logistic map is used to generate the chaos noise:

We use the Mackey-Glass equation for the time series
prediction. The Mackey-Glass equation is the representative example of delay induced chaotic behavior. The equation is the nonlinear time delay differential equation.

v(n + 1) = αv(n)(1 − v(n)).

(3)

Varying the parameter α, Eq. (3) behaves chaotically via
a period-doubling cascade. Further, it is well known thatthe map produces intermittent chaos just before periodic
windows appear.
Figure 2 (a) shows an example of the intermittency chaos
near the three-periodic window obtained from Eq. (3) for
α = 3.8274. As we can see from the figure, the chaotic
time series could be divided into two phases; laminar partsof periodic behavior with period three and burst parts ofspread points over the invariant interval. As increasing α,
the ratio of the laminar parts becomes larger and finally thethree-periodic window appears. For the comparison, wealso carry out computer simulations for the case of fully
developed chaos in Fig. 2 (b) for α = 4.0000. This noise ismuch more similar to the random noise.
The state equation of ESNs with chaos noise is describedby the following equation.
x(n) =(1 − ε)x(n − 1) + εx̃(n) + δv(n).

(4)

where δ is the control parameter of the amplitude of thechaos noise. For the simulation, δ is set to 0.0001.
1"logistic3_1.txt"

chaos noise

0.8

0.6dzzτ=β− γz,dt
1 + zmτγ, β, m > 0(5)An example of time series of Mackey-Glass is shown inFig. 3.

Figure 3: An example of time series of Mackey-Glassequation (τ = 17).
For the training of ESNs, the number of points of timeseries is set to 2,000. After the training, 2,000 time series
points will be predicted with free running of ESNs. For
implementation of ESNs, we referred the manuscript [8].
5. Simulation Results

0.4

0.2

00500

1000
t

1500

2000

(a) 3-periodic intermittency chaos (α=3.8274).
1"logistic4_1.txt"

chaos noise

0.8

0.6

0.4

0.200500

1000t

1500

2000

(b) Fully developed chaos (α=4.0000).

Figure 2: Two types of chaos noise generated from logisticmap.

The computer simulation results of 50 time average areshown in Fig. 4. We calculate two types of MSE after training. The first one is that MSE is calculated for the first 500
time steps. The second one is that MSE is calculated forthe whole 2,000 time steps. Figure 4 (a) shows the simulation results of MSE with first 500 time steps. From thisfigure, we can see that the ESN without noise has the lowest MSE. The performance of intermittency chaotic noiseand fully developed chaos noise of MSEs are almost thesame.
Figure 4 (b) shows the simulation results of MSE with2,000 time steps. From this figure, it can be seen that in
the case of ESN without noise, the range of values taken
by MSE is the largest and fully developed chaos is thesmallest. ESNs using intermittency chaos noise has a lower
range of values that MSE can take than other ESNs.
Next, we summarize the 50 times average results of MSE
using three types of ESNs, ESN with intermittency chaosnoise, ESN with fully developed chaos noise and ESN
without noise (Table 1). This result is very interesting. Thisis because ESN with no noise shows the best performancein MSE for first 500 steps, whereas ESN with intermittency- 275 -chaotic noise shows the best accuracy in MSE for whole
2,000 steps. In other words, the accuracy immediately after learning is good for ESN without noise, however it canbe said that ESN with intermittency chaos noise is effective
for long prediction.
+,$-.&""/*#&"$'")*#""$'")!#&"$'")!#""$'")&#""$'"("#""$%""!
0123450223167

89::-6;<=>

1='1=0>3(a) ESN with 3-periodic intermittency chaos (α=3.8274).

(a) MSE (500).
,-./$0"""1"#"+"#"*"#")"#"("#"'"#"&"#"%"#"$"#"!
"!
2345672445389:;<<=8>?@A

3@B3@2A5(b) MSE (2,000).
(b) ESN with Fully developed chaos (α=4.0000).

Figure 4: Simulation results of MSE.

Table 1: Simulation results (50 times average)
MSE (500) MSE (2,000)
Intermittency chaos
1.02e-04
1.96e-02
Fully chaos
9.04e-05
2.55e-02
No noise
6.43e-06
2.25e-02

Figure 5 shows the example of simulation results of target data and predicted data after the training of ESNs. In
the case of ESN without noise, the target and predictedvalue are in good agreement up to the first 500 steps, but
after that, the error becomes large. While, in the case ofESN with intermittency chaos noise, it can be seen that theerror is small over 2000 steps.
Finally, the obtained output weights (Wout ) of ESNs areshown in Fig. 6 It seems that the ESN with chaos noise hasa higher (Wout ) value than the ESN without noise. A moredetailed study of the learning process of chaos noise andthe performance of ESNs is one of our future topics.

(c) ESN without state noise.

Figure 5: Simulation results of time series prediction using
three types of ESNs.

- 276 -References[1] H. Jaeger, “”Echo State” Approach to Analyzing andTraining Recurrent Neural Networks,” Technical Report GMD Report 148, German National ResearchCenter for Information Technology, 2001.
[2] H. Jaeger, “Echo State Network,” Scholarpedia,
2(9):2330, 2007.
[3] M. Lukosevicius, “A Practical Guide to Applying EchoState Networks,” Neural Networks: Tricks of the Trade,7700, pp. 659-686, 2012.

(a) ESN with 3-periodic intermittency chaos (α=3.8274).

[4] H. Jaeger, M. Lukosevicius, D. Popovici and U. Siewert, “Optimization and Applications of Echo State Networks with Leaky-integrator Neurons,” Neural Networks, 20, pp. 335352-686, 2007.
[5] Y. Hayakawa and Y. Sawada, “Effects of the Chaotic
Noise on the Performance of a Neural Network Modelfor Optimization Problem,” Phys. Rev. E, State. Phys.
Plasmas Fluids Relat. Interdiscip Top., vol. 51, no. 4,pp.2693-2696, 1995.
(b) ESN with Fully developed chaos (α=4.0000).

[6] Y. Uwate, Y. Nishio and A. Ushida, “Markov Chain
Modeling of Intermittency Chaos and its Application
to Hopfield NN,” IEICE Transactions on Fundamentals, vol. E87-A, no. 4, pp. 774-779, Apr. 2004.
[7] Y. Uwate, Y. Nishio, T. Ueta, T. Kawabe and T.
Ikeguchi, “Performance of Chaos and Burst Noises Injected to the Hopfield NN for Quadratic Assignment
Problems,” IEICE Transactions on Fundamentals, vol.
E87-A, no. 4, pp. 937-943, Apr. 2004.
[8] M. Lukosevicius, “A Practical Guide to Applying Echo
State Networks,” Neural Networks: Tricks of the Trade,7700, pp. 659-686, 2012.

(c) ESN without state noise.

[9] T. Yamane, Y. Katayama, R. Nakane, G. Tanaka and D.
Nakano, “Wave-based Reservoir Computing by Synchronization of Coupled Oscillators,” Proceedings ofInternational Conference on Neural Information Processing (ICONIP’15), pp. 198-205, 2015.

Figure 6: Output weights after training.
6. Conclusions
When using ESN in real-world nonlinear modelingtasks, the ultimate goal is to minimize test errors. Theadding state nose to ESN has been proposed to have thebenefit of stabilizing solutions in models with output feedback. In this study, we investigated the performance ofchaos noise when it is injected to the ESN for time series prediction. Computer simulated results showed thatthe intermittency chaos noise gains good performance forlong-term prediction of time series.
In future works, we would like to make clear the reason
why intermittency chaos noise better performance than theothers for time series prediction. Furthermore, we would
like to extend this study and investigate reservoir computing using chaotic circuits in the reservoir layer [9], [10].

[10] J. Choi and P. Kim, “Reservoir Computing based onQuenched Chaos,” Chas, Solitons and Fractals, 140,
110131, https://doi.org/10.1016/j.chaos.2020.110131,
2020.

- 277 -