数理解析研究所講究録
第 760 巻 1991 年 71-87

71

リカレントニューラルネッ トによる非線形力学系の学習
Learning Nonlinear

Dynamics by Recurrent Neural
Networks

ATR 視聴覚機構研究所
ATR Auditory and Visual Perception Research Laboratories

佐藤雅昭

村上由彦

Masa-aki Sato and Yoshihiko Murakami

任意のフィードバック結合を持つニューラルネッ ト(リ カ

レントネッ ト)は、 複雑な非線形ダイナミスを持つシステムであり、 リミ

トサイクルやカオスなどの様々な時間的振るッ

舞いを示す。 我々はこれらの現象を情報処理に利用する目的
で、 リカレントネヅ

トを研究している。 本稿では、 非線形ダ

イナミクスの学習に関する研究を紹介する。

72

Learning Nonlinear Dynamics by Recurrent Neural
Networks

Masa-aki Sato and Yoshihiko Murakami
ATR Auditory and Visual Perception Research Laboratories

Sanpeidan Inuidani Seika-cho Soraku-gun Kyoto, 619-02, Japan

73
ABSTRACTA recurrent network, which can approximate auniversal class of nonlinear dynamic systems, and its
learning algorithm are presented. The possibility oflearning chaotic dynamics by the recurrent network
was investigated. The Lorentz attractor was used as
an example of chaotic dynamics. When the trajectoryof the Lorentz attractor was used as the teacher
signal, the network was able to acquire the time
evolution rule of the Lorentz dynamics and generated
a chaotic attractor similar to the Lorentz attractor.
The possibility of learning the hidden chaotic
dynamics was also investigated.

1. INTRODUCTION
There are three types of neural networks. The first type is amultilayerd feed-forward network. It has been shown that a threelayer network can approximate any nonlinear function. The second
type is a relaxation network, such as the Hopfield network.
Although its output changes in time, only the stable output is used
for information processing. Therefore, these two types of networks
can be considered static information systems. The third type is arecurrent neural network with arbitrary feedback connections.
Since the recurrent networks are complex nonlinear dynamic
systems, they exhibit a variety of complex temporal behavior, suchas limit cycle and chaos. Our main aim is to use the nonlinear
behavior of a recurrent network for information processing $[1,2]$ .
This may open new areas for active and dynamic information
processing.
In fact, chaos and other nonlinear phenomema have been.found in many biological systems including squid giant axons, rat
hippocampus, rabbit olfactory bulb and brain $EEG[3,4,5]$ . These2

74
nonlinear dynamic phenomena seem to play an important role for
information processing in biological systems [3]. We would like tocontrol the chaotic dynamics by using recurrent networks. As a firststep, we trained the recurrent network to learn the chaotic
dynamics [1]. Although it is impossible to learn the long term
behavior of chaotic dynamics because of the initial value sensitivity[6], it is possible to leam the time evolution rule of the chaotic
dynamics.
Recently, Lapedes and Farber [7] trained feedforward
backpropagation networks [8] to learn discrete chaotic maps, andstudied the accuracy of the network’s short time prediction. In our
approach, on the other hand , the recurrent network can acquire
the time evolution rule of the chaotic dynamics described by
nonlinear differential equations.
In our previous paper [1], we proposed a new recurrent
neural network architecture for general purposes. It is composed of
two types of units. One is a dynamic unit whose output isdetermined by a differential equation. The other is a sigmoid unit
which transforms an input to an output through a sigmoid function.
These are connected each other by feedback connections. It isshown that this recurrent network can approximate a universal
class of nonlinear dynamic systems if a sufficient number of hidden
units is introduced. A supervised learning rule for this recurrent
network was also derived. In this article, we summarized theprevious results of the recurrent network architecture and thelearning algorithm, and presented simulation results in detail.
In the computer simulation, the Lorentz attractor was used
as an example of chaotic dynamics. The trained recurrent networks
were composed of three dynamic units, which correspond to thethree dynamic variables.in the Lorentz dynamics, and thirty hidden
sigmoid units. In one simulation, the trajectory of the Lorentz
attractor was used as the teacher signal. After 30,000 weight
updates, the recurrent network generated a chaotic attractor whose
structure was very similar to that of the Lorentz attractor. The
value of the largest Liapunov exponent calculated by the trained$\cdot$75

network was 0.85 (desired value: 0.90), which means that therecurrent network was able to learn the instability of the chaotic
dynamics.
Next, we investigated the possibility of learning the hidden
dynamic variables of the chaotic dynamics. When one variable washidden, the trained network generated a chaotic attractor after
50,000 weight updates. The trayectories for visible variables werevery chose to those of the Lorentz attractor, while the hidden
variable trajectory was deviated from that of the Lorentz attractor.
The implication of this result is also discussed.

2. UNIVERSAL APPROXIMATION FOR NONLINEAR DYNAMIC
SYSEMS
Most of nonlinear dynamic systems can be described by thefollowing equations of motions if sufficient number of auxiliary
variables are introduced:$dX(t)/dt=F(X(t), U(t))$

(2.1)

and
represent a N-dimensional vector dynamic
variable, a K-dimensional vector external force and a N-dimensional
vector nonlinear function which is called a vector field,
respectively. For example, any Hamilton system can be written inthis form.
Recently, it was shown that any nonlinear function can beapproximated by a finite sum of sigmoid functions $[9,10]$ . Let $G(x)$be a sigmoid function. Let
be a compact region of a space spannedbyand . The vector field $F(X,U)$ is assumed to be continuous in, there exists an integer $M$ and real
. Then, for an arbitrary
constant’s $WA_{im},WB_{mi},WC_{mk},$ $WD.(i=1,..,N;m=1,.., M;k=1,.., K)such$that the following relation is hold:

where

$X,$

$U$

$F$

$\Omega$$X$$\Omega$$U$$\epsilon>0$$ma\kappa|F_{i}(X,U)-H_{i}(X,U)|<\epsilon$( $X.Uje\Omega$

(2.2)

76
,where $H(X,U)$ is defined by$H_{i}(X,U)= \sum_{m=1}^{M}WA_{m}\cdot G(\sum_{j=1}^{N}WB_{mj}\cdot X_{j}+\sum_{k=1}^{K}WC_{mk}\cdot U_{k}+WD_{t\hslash})$

.

(23)

If the nonlinear dynamic system defined by (2.1) is structurally
stable [61, the vector field $F(X,U)$ can be approximated by the vector
function $H(X,U)$ in the compact region . Therefore, universal classof nonlinear dynamic systems described by the equation (2.1) canbe approximated by recurrent neural networks defined by thefollowing equations of motions:$\Omega$$dX(t)/dt=WA\cdot Z(t)$(2.4a)$Z(t)=G(WBX(t)+WC\cdot U(t)+WD)$(2.4b)

where the N-dimensional vector $X(t)$ and the M-dimensional vector$Z(t)$ represent outputs of dynamic units andsigmoid units,
respectively. The dynamic units receive signals from the sigmoid
units through the N X $M$ connection weight matrix $WA$ . The sigmoid
units receive the M-dimensional vector bias $WD$ , signals from theconnection weight matrix $WB$ anddynamic units through the M X
connection weight matrix $WC$ .
external inputs through the M X
They transform these inputs to outputs through a sigmoid function,. In the learning process, some dynamic units receive desired
temporal behavior as teacher signals. They are called visible units
and denoted by $VD$ . The other dynamic units have no teacher signal
and are called hidden dynamic units. They are denoted by $HD$ . The
sigmoid units are all hidden since there is no teacher signal forthem. The structure of the network is shown in fig.1.
$N$

$K$

$G$

3. LEARNING ALGORITHM
In this section, a supervised learning algorithm for the5

77

recurrent network defined by (2.4) is derived $[1,13]$ . Although we
can derive a learning rule for any error function, here we will use

the teacher forcing error function $[11,12]$ . In the teacher forcing
method, the visible units are clamped to the teacher signal, $Q(t)$ , byreceiving additional external forces,$J\iota(t)=dQ_{i}(t)/dt-(WA\cdot Z(t))\iota$

for$i\in VD$.

The magnitude of the external forces can be considered as thedeviation from the desired network. Therefore, an error function isdefine by$E= \int_{t1}^{J2}dt\sum_{i\in\gamma D}J_{i}^{2}(t)$

.
(3.1)

By introducing the Lagrange multipliers,function can be written as:$E= \int_{t1}^{t2}dt[\sum_{i\in\nu D}J_{i}^{2}-\sum_{ieHD}PX_{i}(\alpha/dt-WA\cdot Z)$$PX$

and $PZ[13]$ , the error;$- \sum_{m}PZ_{n\prime}(Z_{\hslash}-G((WB\cdot X+WC\cdot U+WD)_{n}))]$

.

(3.2)

Let us calculate the variation of the error function in order to get
the expression for the gradient of the error function. The calculationis straightforward. The equations of motions for Lagrange
multiplier can be.derived from the requirement that the coefficientof the variations
andshould be vanish:$\delta X$$\delta Z$$d(PX_{i})/dt=- \sum_{n}PZ_{m}\cdot G’((WB\cdot X+WC\cdot U+WD)_{m})\cdot(WB)_{n\dot{u}}$(3.3a)

and$PX\iota(t2)=0$

for(3.3b)$i\in PD$

where $G’(x)$ represents the gradient of the sigmoid function, and

6

78

for

$PZ_{m}= \sum_{i}P_{i}\cdot(WA)_{\dot{\nu}m}$$m=1,\ldots,M$ ,(3.3c)

where$P\iota=- J_{i}$for$i\in HD$

and.
Then the variation of the error function can be written as$P\iota=PX_{i}$for$i\in HD$$\delta E=\int_{t1}^{t2}dr[P^{T}\cdot\delta WA\cdot Z+(PZ\cdot G’(WB\cdot X+WC\cdot U+WD))^{T}$$( \mathscr{N}B\cdot X+\delta WC\cdot U+\delta WD)]+\sum_{i\in HD}PX_{i}(t1)\cdot M_{i}(t1)$

.

(3.4)

where matrix notations are used and the superscript
denotes thetranspose of a vector. The derivatives of error function with respect
to adjustable parameters $WA,$ $WB,$ $WC,$ $WD$ and $X(tl)$ are given by$\delta WC,$the coefficients ofin (3.4),
andrespectively. The adjustable parameters can be modified by using
the steepest descent method or other method like conjugategradient algorithm so that the error value will decrease.
The leaming schedule is as follows [2]. First, the network is
run forward in time fromto $(T+TB)$ . The outputs of the hidden
units are calculated by clamping the visible units to the teacher
signals. Second, the error response variables, $PX$ and $PZ$ , arecalculated backward in time from $(T+TB)$ to , following
Then, the weight values are modified to decrease
equation(3.3).
the error function. The initial value for hidden dynamic units arealso updated. Finally, the recurrent network with new parametervalues is run forward in time fromto $(T+TF)$ , and the current
time, , is updated to $(T+TF)$ . The above steps are repeated until
the error value becomes sufficiently small. There are somecomments on the initial condition in the above learning scheme.
Although initial condition for the visible units are known, the initialcondition for the hidden units are not known. An improper choiceof the initial condition for the hidden units causes errors of the$T$$\delta WA,$$\delta WD$$\delta WB,$$\delta X(tl)$

$T$

$T$

$T$

$T$

7

79

visible units even for the desired weight values. Therefore, theinitial values for the hidden units are considered as learningparameters in our learning scheme. When the desired trajectory is
chaotic motion, it is impossible to impose a initial condition at a
fixed time because of sensitive dependence on the initial condition[61. Therefore, the initial condition should be reset for each learning
trial and the learning interval $TB$ should not be large comparedwith the time scale corresponding to the largest Lyapunov exponent[6]. This means that the recurrent network learns different
trajectories in the chaotic attractor for each learning trial. Since
these trajectories are derived from the same time evolution rule,
one can expect that the recurrent network is able to aquire the time
evolution rule of the chaotic dynamics.

4. LEARNING CHAOTIC DYNAMICS
4.1

Lorentz Attractor
In this section, the possibility of learning chaotic dynamics
by the recurrent network is investigated. The Lorentz attractor(fig.2) is used as an example of the chaotic dynamics. It is definedby the following differential equations [6].
$d\kappa/dt=Fl(x,y,z)=10\cdot(x-y)$(4.1a)$dy/dt=F_{2}(x,y,z)=-y+(28-z)\cdot x$(4.1b)$dz/dt=F_{3}(x,y,z)=-(8/3)\cdot z+x\cdot y$(4.1c)

This is an autonomous system and there is no external input.
The trained network was composed of three dynamic units
and thirty sigmoid units. In the numerical simulation, thesedifferential equations were approximated by the second order
Runge-Kutta method. The time step was set to 0.01. The initial
weights of the network were chosen randomly. In the learning
phase, the learning internal $TB$ and the free running time $TF$ are set
to 1.0 and 0.5, respectively.

80
4.2 All Visible CaseIn one simulation, all the dynamic units received the teacher
signals $x(t),$ $y(t)$ and $z(t)$ calculated by the equation (4.1). As
learning proceeded, the network exhibited numerous bifurcations
and the error increased at these points because of instability near
the bifurcation points. Accordingly, we observed considerable
qualitatively different behavior such as fixed points, limit cycles,etc (fig.3). After 30,000 weight updates, the recurrent network
generated the chaotic attractor shown in fig.4. The structure of theattractor is very close to the Lorentz attractor. The accuracy of theapproximation for the dynamic evolution rule (4.1) can beevaluated by the difference between the vector field
for thefor theLorentz dynamics (4.1) and the effective vector fieldrecurrent network (2.4). The error for the vector field $F\iota(x, y, z)$ in a
2-D section of the phase space is shown in fig.7, where the averagewith respect to the remaining axis is taken. One can see that theerror on the attractor is very small. The error inside the attractor isalso small, while the error outside the attractor becomes large. One
should note that the network has never been supplied the teacher
signal in these regions. The error average over the attractor was
0.0002%. The above results show that the Lorentz dynamics (3.1)
are well approximated by the recurrent network in theneighborhood of the attractor. We also calculated the largest
Liapunov exponent [6] which characterize the degree of theinstability of the chaotic dynamics. The value for the trained
network was 0.85 (the value for the Lorentz attractor was 0.90).
This indicated that the recurrent network was able to learn the
instability of the chaotic trajectories in the Lorentz attractor.
$F_{i}$$(WA\cdot Z)\iota$

4.3

Learning Hidden Dynamics
Next, we investigated the possibility of learning the hidden
dynamic variables of the chaotic dynamics. Chaotic behavior does
not appear for continuous dynamic systems with less than threedegrees of freedom [6]. When only two dynamic variables,
and ,
$y$

9

$z$

81

were used as the teacher signals, the recurrent network should
estimate hidden dynamics in order to produce the chaotic attractor.
However, there is an ambiguity corresponding to the coordinatetransformation of the dynamic variables, since there is no teacher
signal for$x$. Under the coordinate transformation,$X=h(x’,y’,z’)$(4.2a)$y=y’z=z$$(42c)(4.\cdot 2b)$’

do not change. Then, the hidden unit ofcould correspond to the transformed
The equations of motion for the transformed variables,the trajectory ofandthe recurrent network$y$variable $x’$ .
are given by$z$$d\kappa’/dt=[F_{1}(h(x’,y’,z’),y’,z’)-p_{2}(h(x’,y’,z’)_{J}y’,z’)\cdot\partial h/\phi’-$$F_{J}(h(x’,y’,z’),y’,z’)\cdot\partial h/\partial z’]/(\partial h/\partial x’)$

(4.3a)$dy’/dt=F_{2}(h(x’,y’,z’),y’,z’)$(4.3b)$dz’/dt=F_{J}(h(x’,y’,z’),y’,z’)$(4.3c),and the trained recurrent network may aquire this time evolution
rule. In this case, the vector field of the recurrent network isdifferent from that of the Lorentz equation (4.1), although the
dynamics of both systems are equivalent.
In the simulation, the recurrent network generated thechaotic attractor shown in fig.5 after the 50,000 weight updates.
are very close to that
The trajectories for visible variables
andof the Lorentz attractor, while the hidden variable trajectories are
deviated from those of the Lorentz attractor. The vector field errors
corresponding to the visible variables are very small while that
corresponding to the hidden variable is large (fig.8). The largest
Liapunov exponent calculated by the trained network was 0.75.
The above results seem to indicate that the hidden. unit of
the trained network corresponds to the transformed variable $x’in$
$y$

10

$z$

82

(4.2). An attractor transformed from the Lorentz attractor by thecoordinate transformation$x=x’- 2y’$

(4.4a)$yz=z=y$ ,$(44c)(4.\cdot 4b)$is shown in fig.6. The attractor generated by the trained network(fig.5) is more similar to the transformed attractor (fig.6) than theLorentz attractor (fig.2). However, we have not yet find the precise
form of the transformation by which the trained recurrent network
is mapped into the Lorentz attractor. There is another possibility
that there exists different dynamics which generates the same
trajectories for. some of the variables of the Lorentz attractor. We
are still investigating this problem.
,

5. Conclusion
A recurrent network, which can approximate a universal
class of nonlinear dynamic systems, and its learning algorithm were
presented. The possibility of learning chaotic dynamics wasinvestigated. The Lorentz attractor was used as an example of thechaotic dynamics. When the trajectories of all the dynamic
variables were used as the teacher signal, the recurrent network
was able to acquire the time evolution rule of the Lorentz dynamics
and generated a chaotic attractor which was very similar to theLorentz attractor. The possibility of learning the hidden chaotic
dynamics is still an open problem and we will study it further in
our future publication. We hope recurrent networks and chaos may
open a new area of active and dynamic information processing.

Reference[1]

M.Sato, Y.Murakami and K.Joe,“Learning chaotic dynamics byrecurrent neural networks“ Proc.Inter.Conf.on Fuzzy Logic &

11

83

[21

[3]

Neural Networks, 601 (1990)
M.Sato, K.Joe and T.Hirahara,’“APOLONN brings us to the realworld“ IJCNN Vol.1, 581-587 (1990)
C.A.Skarda and W.J.Freeman, “How brains make chaos in orderto make sense of the world”, Behavior and Brain Science, 10,

161-195
[4]
[51
[6]

[7]

(1987)

Proceeding of Intemational Conference on Fuzzy Logic and
Neural Networks, at IIZUKA, JAPAN(1990)
H.G.Schuster, “Deterministic chaos“, VCH (1988)
J.Guckenheimer and P.Holmes, “Nonlinear oscillations,dynamical systems, and bifucations of vector field”, SpringerVerlag, New York (1983)
Lapedes A and Farber , “Nonlinear signal processing usingneural network” LA-UR-87-2662, Los Alamos National Lab.
$R$

(1987)[81[91

Hinton GE, Rumelhart DE and Williams RJ “Learninginternal representation by error propagation“, in Parallel
Distributed Processing I, Rumelhart DE and McClelland JL,
M.I.T. Press, Cambridge, MA, 318-362, (1986)
Funahashi $K,$
the approximate realization of continuous
mapping by neural networks” Neural Networks, 2, 183-192$\prime\prime on$(1989)

B. Irie and S. Miyake, ”Capabilities of three-layered
perceptrons“ Proc.of IJCNN88, 1, 641-648 (1988)[11] Pearlmutter BA, “Learning state space trajectories inrecurrent network“, Neural Computation, 1, 263-269 (1989)[12] Williams RJ and Zipser , “A leaming algorithm forcontinually running fully recurrent neural network”, NeuralComputation, 1, 270-280 (1989)[13] M. Sato, “A Learning Algorithm to Teach Spatiotemporal
Patterns to Recurrent Neural Networks“ Biol. Cybernetics, 62,

[10]

$D$

259-263(1990)

12

84

Fig.1 The structure of the recurrent network

x-y plane
Fig.2 Lorentz attractor

y-z planez-x plane

Fig.3 A limit cycle generated by the recurrent net under training

85

z-x planey-z plane

Fig.4 The attractor generated by the all visible recurrent net

z-x plane
Fig.5 The attractor generated by the one hidden recurrent net

x’-y’ planey’-z’ plane

Fig.6 Transformed Lorentz attractor

86$\dot{o}0\sigma v$$\dot{s}\circ$億$>\wedge$.

X$\perp\infty$$\overline{\underline{o}}$叫く\supset$\overline{u\underline{\underline{o_{[}}}}$$\triangleleft\overline{\triangleleft}$$t^{N}h$$\sim\dot{o}o$

!り

$-*$
$\mu_{r}$$.—$$\wedge N$; 8$\underline{\triangleleft}$$\wedge^{\wedge}$$\vee X^{-}$

$0$

—- 日何
$\overline{\vee O}$$\overline{\triangleright}\iota$$\triangleleft$$\Phi O$$\overline{\underline{o}}$$\triangleright$$\underline{\Phi}$:
1$\Phi$$\underline{\overline{o}}$$\infty$$rightarrow$$.\overline{\triangleright}$

占$\overline{O}$$rightarrow$$q_{}$$\overline{O}$$\overline{\alpha}$$g$$\overline{\overline{\geq}}$

国$\dot{\ddagger}^{\frac{b}{h}}\triangleright_{\dot{0}}$$\wedge>^{-}N\backslash$X火$\overline{\sim Q}$–$\frac{o}{\frac{}{u!}}$87

科$\sim\dot{o}o$$\wedge N$$\wedge^{-}$

X匹$\overline{\underline{o}}$$\llcorner$$th^{)}\propto$$\iota u:o$$\triangleleft@$$Ac\triangleleft$$tl\dot{O}\circ$$-\wedge-\overline{g}$$\approx$$\mathfrak{t}h$$\wedge N$$\underline{\triangleleft}$$\wedge^{\wedge}$$\underline{4)}$

$b$

X–

8

へ

.

$\underline{O}$日何$\overline{\triangleright}_{\backslash }$$\triangleleft$$\Phi O$$\approx$$\succ$$\overline{\vee 0}$$\triangleleft V$4)$\circ$$\overline{-}$$\dot{\overline{B}}$$\overline{O}$$\approx\Phi$$\overline{\underline{o}}$

血田

$b$

$\overline{r^{O}b_{i}}$$\sim\dot{o}0$$\infty 6\dot{O}$$\wedge N$$\dot{\overline{A}}$$>^{-}\backslash$

X
$p-$

火
$\underline{\overline{o}1}$$L^{\overline{O}}:\coprod$$O$$\frac{\not\in}{\overline{\geq}}$