Master’s Thesis 要旨

Analysis of relationship between structure and performance in echo
state networks
Dept. of Mathematical Informatics 48-176230
Supervisor

1 Introduction
Echo state networks (ESNs) are powerful computing
systems in predictions of time series [1]. However,
the mechanism of the computational ability and thememory capacity of the ESN has not been fully clarified. In this thesis, we investigate how the structuresof the ESN, including the activation function and theconnection weights, influence its performance. In order to reveal the mechanism of the ESN in predicting time series, we compute the Jacobian matrix ofthe ESN system with respect to the past input signals, and analyze the relationship between the Jacobian matrix and the structures of the ESN. We findthat it is the eigenvalue distribution of the recurrent
connection matrix that determines the dynamical behavior of the ESN. Based on this result we propose a
new method to design the recurrent connection matrix which shows better performance than that in thestandard ESN.

2 Echo State Network
The dynamics of the ESN is described by:x[t] = Cr[t−1] + Winp u[t] + Wf eed z[t−1] ,r[t] = f (x[t] ),z[t]=W(1)out [t]r ,where x[t] is the internal state at time t, and u[t] andz[t] are the input and output signals at time t, respectively. C represents the recurrent connection weights(inside the reservoir). The matrices Winp , Wout
and Wf eed represent the input, readout and feedback
connections weights of the ESN, respectively, and fdenotes the activation function which is element-wisein this formula. We only train the readout connection weights Wout of the ESN. The other connectionweights are initialized randomly (Gaussian distribution is commonly used) and remained unchanged.
Therefore the initialization method for other connections is of great importance. In this study, we investigate how the structure, such as the activation
function f and the recurrent connection weights C,
influences the computational performance of ESNs.

3 Relationship between Structure andPerformance in ESNs
3.1 Relationship between Feedback Connections andLong Term Memory
In the analysis of the role of feedback connections, weused the three-bit-flip-flop (TBFF) task as an exam-Hei Ze
Assoc. Prof. Gouhei Tanaka

ple. TBFF task has three input channels and output
channels. The input signals are given by sequencesof pulses with irregular intervals. The peak value ofall pulses is 1 or −1. This task requires the output
to be 1 or −1, and to have the same sign as that of
the most recent input pulse of the same channel.

Fig. 1. PCs of ESNs with feedback connections[2]and those without feedback connections

To investigate the role of the feedback connections
in TBFF tasks, we trained two ESNs with and without the feedback connections on the TBFF task. And
compared the first three principle components (PCs)of the reservoir states of these two ESNs. The resultis shown in Fig. 1. With the feedback connections,
the ESN can perform the TBFF task. There are eight
attractors in the dynamics of the ESN. When the output state was changed by the input pulse, the state of
the reservoir jumped from one attractor to another,which is the origin of the long term memory. Without feedback connections, there is only one attractor. When there is a pulse of the input, the state ofthe reservoir will be pushed away, and soon fall into
the attractor as the input pulse vanishes as shown inFig. 1. The color denotes the channel of the input,
and the values of three channels compose the RGB
space of the color. From this result, we can draw theconclusion that feedback connections are importantin the formation of the long term memory of the ESN.
3.2 Relationship between Recurrent Connections andShort Term Memory
The echo state property means that the influence ofthe past input signals gradually vanishes. Mathematically this means that the derivative of the state x[t]with respect to the past input signal u[t−m] converges
to 0 as m increases. This can be regarded as the origin of the short term memory of the ESN. The valueof the derivative is naturally included in the Jacobian matrix of the ESN. With the chain rule, we cancompute the Jacobian matrix as follows:( out )T [t]∂z [t]F CF[t−1] C · · · F[t−m] CWinp ,=W∂u[t−m](2)

Analysis of relationship between structure and performance in echo state networks (Hei Ze)

where F[t] is an n × n diagonal matrix with the i′[t]th diagonal term f (xi ). We find that F[t] can be
regarded as constant signals with random perturbations if the scale of the input is small. Therefore wecan make the approximation F[t] ≈ Fa , and for a hyperbolic tangent or sigmoid activation function, Fa
approximately equals to the identity matrix multiplied by a constant. With the eigen-decompositionof Fa C, we can obtain the relationship between theeigenvalues of Fa C and the short term memory. Especially for the k time delay task, which requires theESN to output the time series value k time steps before the current time, we can compute the optimal
readout weights as follows:Woptimal =1kM (λλT )[1λk1

λk2···

λkn]T,(3)
where λ = [λ1 , λ2 , · · · , λn ] is a vector of eigenvaluesof Fa C, and n denotes the reservoir size and M is avariable which depends on the max value of λ, whichMM +1satisfies (max(λ)) ̸= 0 and (max(λ))= 0. Thematrix Woptimal is the optimal value of W which is alinear transform of the readout weights Wout . Whenk is large, which means the required memory for thistask is large, we can expect that the term λλT isneither rapidly expanding nor contracting.
3.3 The Eﬀect of Activation Function on Computational Ability of ESN
We used the NARMA task in the analysis of the roleof feedback connections. For a random time series u
as the input signal, the target z is generated from uwith the next recursive formula:zt = αzt−1 + βzt−1k∑zt−i + γut−k ut−1 + δ,proved by the next experiments, where we take thediﬀerent order Taylor expansions of the hyperbolic
tangent function as activation functions, and compare the performance of ESNs with these activation
functions. The result is shown in Fig. 2 which showsthat the computational ability of the ESN with thelinear activation function is much worse than thoseof nonlinear activation functions.

4 Design Strategy for Echo State Networks
From the result obtained in section 3.2, we see thatthe mean value of the non-zero eigenvalues of Fa C isimportant in the formation of the short term memory. Since Fa approximately equals to the identity
matrix, we just consider the recurrent matrix C. In
order to adjust the eigenvalue distribution C, we first
generate a diagonal block matrix which is composedof multiple 2 × 2 matrices with controllable pairs ofeigenvalues. By denoting the radius of each pair by
ri , we can define: r[m] = [r1m , r2m , · · · , rnm ], where nis the reservoir size. Then we can adjust the radius
distribution of the eigenvalues of the recurrent matrix by choosing diﬀerent m which is shown in Fig. 3(left). We can also compare the performance of ESNs
with this matrix of diﬀerent m on the time delay task
(Fig. 3, right).

(4)

i=1

where α, β, γ and δ are constant values, and k indicates the required memory of this task. Eq. (4)is extremely complicated and therefore the NARMAtasks can be used to evaluate the computational ability of the ESN. From Eq. (4) we computed the condi-Fig. 3. Eigenvalue distributions of matrices Wm
generated with r[m] (left). And the performance of ESNs with the recurrent matrix
as Wm on the time delay task (right).

From the result we found that when the eigenvalues
are distributed near the edge of the circle (green lines,
with m = 0.0001, 0.001, 0.01, 0.1), the performance isexcellent. When m = 1, the eigenvalue distribution(bottom right corner in the left figure) is similar tothat of the random matrix (upper left corner in theleft figure), and their performance (red line and blueline) are both worse than the others.

Bibliography
Fig. 2. Performance of ESNs with diﬀerent order
Taylor expansions of the activation function

tion for the Jacobian matrix, and found that for tasks
whose target time series are generated from a polynomial of degree n, the n-th derivative of the activation function is required to be non-zero. This can be[1] H. Jaeger, “The echo state approach to analysing
and training recurrent neural networks-with an erratum note.,” German National Research Center
for Information Technology GMD Technical Report,
vol. 148, no. 34, p. 13, 2001.
[2] D. Sussillo and O. Barak, “Opening the black box:
low-dimensional dynamics in high-dimensional recurrent neural networks.,” Neural computation, vol. 25,no. 3, pp. 626–649, 2013.