# Building Recurrent Neural Networks to Implement Multiple Attractor Dynamics Using the Gradient Descent Method

Authors: Jun Namikawa; Jun Tani
Journal: Advances in Artificial Neural Systems
 (2009)

---

## Abstract

The present paper proposes a recurrent neural network model and learning algorithm that can acquire the ability to generate desired multiple sequences. The network model is a dynamical system in which the transition function is a contraction mapping, and the learning algorithm is based on the gradient descent method. We show a numerical simulation in which a recurrent neural network obtains a multiple periodic attractor consisting of five Lissajous curves, or a Van der Pol oscillator with twelve different parameters. The present analysis clarifies that the model contains many stable regions as attractors, and multiple time series can be embedded into these regions by using the present learning method.


---

## Body

## 1. Introduction

Recurrent  neural networks (RNNs) have been successfully applied to the modeling  of various types of dynamical systems. Since the universal approximation ability
of multilayer neural networks has been proved, RNNs can model arbitrary
dynamical systems and turing machines [1–3]. However, applying RNNs to a
desired model may be very difficult even if such RNNs exist [4]. For example, building RNNs
to implement required multiple attractor dynamics is a difficult problem for
standard training, such as the gradient descent method. Doya and Yoshizawa
[5] demonstrated that
RNNs can acquire two limit cycles in the gradient descent method using
initialization with small connection weights, whereas learning for more than
three limit cycles is difficult [6]. This is due to the fact that the learning of several
time series causes a conflict with respect to the changing of the connection weights.
How to form RNN models that can learn several temporal sequence patterns has
proved to be a challenging problem.There have been some approaches to this problem. In
order to avoid conflicts in the change of parameters, the
mixture-of-experts-type architecture has been
investigated [7, 8].
The mixture-of-experts model consists of RNNs as experts and a hierarchical
gating mechanism. At the end of successful learning, each expert implements
attractor dynamics as locally represented knowledge, and a gating mechanism
chooses only one expert at any time. The system can acquire many attractor
patterns although there is a disadvantage in that the system does not have the
generalization ability on the attractor patterns. As the other approach to
implement multiple patterns, the parametric bias (PB) method has been developed
to improve the learning capability of RNNs [9, 10]. In an RNN that employs the PB method (RNNPB), PB
values provide the information needed in order to individualize each sequence.
It has been reported that the number of time series that RNNPBs can learn is
greater than that which RNNs without PB can learn. However, the PB method
cannot avoid the conflict caused by each attractor learning. Therefore,
learning multiple time series by an RNNPB tends to fail when the number of time
series increases.In the present study, we will focus on the training
method for RNNs to learn multiple attractor dynamics. Furthermore, we will show
that the present research is related to research into RNNs with contraction
transition functions. In recent years, RNNs with contraction transition mapping
have been investigated with respect to the performance of time series learning
[11–13], generalization ability [14], and memory capacity [15]. Jaeger [11, 12] demonstrated that an “echo
state network,” which is an RNN with contraction mapping, successfully learns
the Mackey-Glass chaotic time series, a well-known benchmark system for time
series prediction. In order to formally express the generalization ability,
Hammer and Tiňo proved that RNNs with contraction are
distribution-independent learnable in the probably approximately correct (PAC)
sense [14]. From the
above results, RNNs with contraction might be regarded as powerful tools for
modeling dynamical systems. However, RNNs with contraction have difficulty in
representing multiple attractor dynamics because dynamic states governed by the
contraction transition function are globally attracted to one point. In this
paper, the representation capability of RNNs with contraction mapping will be
improved such that the RNNs can obtain multiple attractor dynamics.We start by defining the concepts of the RNN and the
training method for multiple attractor dynamics. The RNN has the Elman net-type
architecture, and the training method for RNNs is basically based on the
backpropagation through-time (BPTT) algorithm [16]. We then show in numerical
simulation that the RNNs can acquire multiple periodic attractors constituted
by five Lissajous curves, or a Van der Pol oscillator with twelve different
parameters. Moreover, we consider why the RNNs successfully learn multiple
attractors and how the performance of learnability depends on parameters of the
RNNs. Finally, we link the results obtained herein to other learning strategies,
and consider other advanced research topics.


## 2. Model

### 2.1. Recurrent Neural Network

We first consider a neural network model with
recurrent connection, such as the Elman net [17] (see Figure 1). The RNN contains I/O units, orthogonal
units, and internal units. We denote the dynamic states of I/O units,
orthogonal units, and internal units at time step n by xn∈ℝN1,rn∈ℝN2,
and un∈ℝN3,
respectively. The RNN is defined by functions fθ and gθ with a parameter θ≡(W1,W2,W3,V,b,d),
where fθ:ℝN1×ℝN2×ℝN3→ℝN3 and gθ:ℝN3→ℝN1×ℝN2 are of the forms(1)fθ(x,r,u)=(1−ϵ)u+ϵ(W1⋅x+W2⋅r+W3⋅F(u)+b),(2)gθ(u)=F(V⋅F(u)+d),
where W1∈ℝN1×N3,W2∈ℝN2×N3,W3∈ℝN3×N3,
and V∈ℝN3×N1N2 are matrices, b∈ℝN3 and d∈ℝN1N2 are vectors, ϵ∈ℝ is a time constant that satisfies 0≤ϵ≤1,
and F denotes a componentwise application such as Fi=tanh.Figure 1
Architecture of the recurrent neural
network. Solid arrows, dotted arrows, and boxes represent fixed connections,
adjustable connections, and network states, respectively.Dynamic states of the RNN at time stepn are updated according to(3)(xn,rn)=gθ(un),un+1=fθ(xn,rn,un)=fθ(gθ(un),un).
From these equations, the RNN
can be represented by an N3-dimensional dynamical system.We now define bistability for the RNN.Definition 1.
Assumefθ:ℝN1×ℝN2×ℝN3→ℝN3 is as above. The function fθ is bistable with respect to the third
variable u if a real value ω>1 and an integer Ns exist such that(4)wij=ωifi≤Ns,i=j,wij=0ifi≤Ns,i≠j,|wij|<1N3otherwise,for every element wij of the matrix W3.The bistability of a functionfθ is a key concept of our learning method. We
will show in Section 4.1 that the bistable function fθ plays an important role in the learning of
multiple attractor dynamics.


### 2.2. Learning Method

We present a formulation of the training procedure for
the RNN with a multiple teacher I/O time series. For every1≤k≤m and Lk∈ℕ,
we assume that (x1(k),…,xLk(k)) is a sequence of teacher I/O of length Lk.Initialization of Parameters
We initialize every element of matricesW1(0), W2(0),
and V(0) and vectors b(0) and d(0) randomly from the uniform distribution in the
interval (−1/N3,1/N3).
A matrix W3(0) is randomly assigned such that fθ is bistable. For all 1≤k≤m,u1(k)(0) is randomly initialized in the interval [−1,1].
Assume thatrn(k)(0) is an m-tuple of vectors (sn(k,1)(0),…,sn(k,m)(0)) for 1≤k≤m and 1≤n≤Lk,
and that the dimension of sn(k,l)(0) is equivalent to that of sn′(k′,l′)(0) if l=l′.
We initialize sn(k,l)(0) such that(5)sn(k,l)(0)={0ifk=l,−1otherwise.Run Network with Teacher I/O and Compute Error Function
For every1≤k≤m,
the sequence (x^1(k)(t),…,x^Lk(k)(t)) of I/O units of the RNN at learning step t is defined by(6)(x^n(k)(t),r^n(k)(t))=gθ(t)(un(k)(t)),un+1(k)(t)=fθ(t)(xn(k),rn(k)(t),un(k)(t)).
The error function E(k)(t) of the RNN at learning step t with the kth teacher I/O time series is defined
by(7)E(k)(t)=∑n=1Lke(x^n(k)(t),xn(k))+e(r^n(k)(t),rn(k)(t)),where e denotes the mean square error function e(x,x′)=(1/2)(x−x′)T⋅(x−x′).
Finally, the error function E(t) at learning step t is defined by(8)E(t)=∑k=1mE(k)(t).Update Parameters
Letρ(t)∈{W1(t),W2(t),V(t),b(t),d(t)} be a parameter of the RNN at learning step t.
We determine the parameter ρ(t+1) by(9)ρ(t+1)=ρ(t)+αΔρ(t),Δρ(t)=βΔρ(t−1)−∂E(t)∂ρ(t),
where Δρ(0)=0;α and β are the constants of the learning rate and
momentum, respectively. On the other hand, a connection matrix W3(t) is not changed as W3(t+1)=W3(t) in order to hold the bistability condition. We
compute the initial state u1(k)(t+1) of the internal units at learning step t+1 such that(10)u1(k)(t+1)=u1(k)(t)+α′Δu1(k)(t),Δu1(k)(t)=βΔu1(k)(t−1)−∂E(t)∂u1(k)(t),
where Δu1(k)(0)=0,
and α′ is the constant of the learning rate of the
initial state. Assume that sn(k,l)(t) is a vector as a component of the orthogonal
units rn(k)(t),
such as rn(k)(t)=(sn(k,1)(t),…,sn(k,m)(t)).
The vector sn(k,l)(t+1) is defined by(11)sn(k,l)(t+1)={sn(k,l)(t)+α′′Δsn(k,l)(t)ifk=l,−1otherwise,(12)Δsn(k,l)(t)=βΔsn(k,l)(t−1)−∂E(t)∂sn(k,l)(t),
where Δsn(k,l)(0)=0,
and α′′ is the constant of the learning rate of the
orthogonal units.
Note that the maximum value of the error functionE(t) depends on the number of units and the length
of the teacher I/O time series. Thus, we should scale the learning rates α,α′,
and α′′ with the number of units and length of
sequences. In the present paper, we consider parameters γ,γ′,
and γ′′ such that α=γ/(N1+N2)∑kLk,α′=γ′/(N1+N2),
and α′′=γ′′/(N1+N2).


## 2.1. Recurrent Neural Network

We first consider a neural network model with
recurrent connection, such as the Elman net [17] (see Figure 1). The RNN contains I/O units, orthogonal
units, and internal units. We denote the dynamic states of I/O units,
orthogonal units, and internal units at time step n by xn∈ℝN1,rn∈ℝN2,
and un∈ℝN3,
respectively. The RNN is defined by functions fθ and gθ with a parameter θ≡(W1,W2,W3,V,b,d),
where fθ:ℝN1×ℝN2×ℝN3→ℝN3 and gθ:ℝN3→ℝN1×ℝN2 are of the forms(1)fθ(x,r,u)=(1−ϵ)u+ϵ(W1⋅x+W2⋅r+W3⋅F(u)+b),(2)gθ(u)=F(V⋅F(u)+d),
where W1∈ℝN1×N3,W2∈ℝN2×N3,W3∈ℝN3×N3,
and V∈ℝN3×N1N2 are matrices, b∈ℝN3 and d∈ℝN1N2 are vectors, ϵ∈ℝ is a time constant that satisfies 0≤ϵ≤1,
and F denotes a componentwise application such as Fi=tanh.Figure 1
Architecture of the recurrent neural
network. Solid arrows, dotted arrows, and boxes represent fixed connections,
adjustable connections, and network states, respectively.Dynamic states of the RNN at time stepn are updated according to(3)(xn,rn)=gθ(un),un+1=fθ(xn,rn,un)=fθ(gθ(un),un).
From these equations, the RNN
can be represented by an N3-dimensional dynamical system.We now define bistability for the RNN.Definition 1.
Assumefθ:ℝN1×ℝN2×ℝN3→ℝN3 is as above. The function fθ is bistable with respect to the third
variable u if a real value ω>1 and an integer Ns exist such that(4)wij=ωifi≤Ns,i=j,wij=0ifi≤Ns,i≠j,|wij|<1N3otherwise,for every element wij of the matrix W3.The bistability of a functionfθ is a key concept of our learning method. We
will show in Section 4.1 that the bistable function fθ plays an important role in the learning of
multiple attractor dynamics.


## 2.2. Learning Method

We present a formulation of the training procedure for
the RNN with a multiple teacher I/O time series. For every1≤k≤m and Lk∈ℕ,
we assume that (x1(k),…,xLk(k)) is a sequence of teacher I/O of length Lk.Initialization of Parameters
We initialize every element of matricesW1(0), W2(0),
and V(0) and vectors b(0) and d(0) randomly from the uniform distribution in the
interval (−1/N3,1/N3).
A matrix W3(0) is randomly assigned such that fθ is bistable. For all 1≤k≤m,u1(k)(0) is randomly initialized in the interval [−1,1].
Assume thatrn(k)(0) is an m-tuple of vectors (sn(k,1)(0),…,sn(k,m)(0)) for 1≤k≤m and 1≤n≤Lk,
and that the dimension of sn(k,l)(0) is equivalent to that of sn′(k′,l′)(0) if l=l′.
We initialize sn(k,l)(0) such that(5)sn(k,l)(0)={0ifk=l,−1otherwise.Run Network with Teacher I/O and Compute Error Function
For every1≤k≤m,
the sequence (x^1(k)(t),…,x^Lk(k)(t)) of I/O units of the RNN at learning step t is defined by(6)(x^n(k)(t),r^n(k)(t))=gθ(t)(un(k)(t)),un+1(k)(t)=fθ(t)(xn(k),rn(k)(t),un(k)(t)).
The error function E(k)(t) of the RNN at learning step t with the kth teacher I/O time series is defined
by(7)E(k)(t)=∑n=1Lke(x^n(k)(t),xn(k))+e(r^n(k)(t),rn(k)(t)),where e denotes the mean square error function e(x,x′)=(1/2)(x−x′)T⋅(x−x′).
Finally, the error function E(t) at learning step t is defined by(8)E(t)=∑k=1mE(k)(t).Update Parameters
Letρ(t)∈{W1(t),W2(t),V(t),b(t),d(t)} be a parameter of the RNN at learning step t.
We determine the parameter ρ(t+1) by(9)ρ(t+1)=ρ(t)+αΔρ(t),Δρ(t)=βΔρ(t−1)−∂E(t)∂ρ(t),
where Δρ(0)=0;α and β are the constants of the learning rate and
momentum, respectively. On the other hand, a connection matrix W3(t) is not changed as W3(t+1)=W3(t) in order to hold the bistability condition. We
compute the initial state u1(k)(t+1) of the internal units at learning step t+1 such that(10)u1(k)(t+1)=u1(k)(t)+α′Δu1(k)(t),Δu1(k)(t)=βΔu1(k)(t−1)−∂E(t)∂u1(k)(t),
where Δu1(k)(0)=0,
and α′ is the constant of the learning rate of the
initial state. Assume that sn(k,l)(t) is a vector as a component of the orthogonal
units rn(k)(t),
such as rn(k)(t)=(sn(k,1)(t),…,sn(k,m)(t)).
The vector sn(k,l)(t+1) is defined by(11)sn(k,l)(t+1)={sn(k,l)(t)+α′′Δsn(k,l)(t)ifk=l,−1otherwise,(12)Δsn(k,l)(t)=βΔsn(k,l)(t−1)−∂E(t)∂sn(k,l)(t),
where Δsn(k,l)(0)=0,
and α′′ is the constant of the learning rate of the
orthogonal units.
Note that the maximum value of the error functionE(t) depends on the number of units and the length
of the teacher I/O time series. Thus, we should scale the learning rates α,α′,
and α′′ with the number of units and length of
sequences. In the present paper, we consider parameters γ,γ′,
and γ′′ such that α=γ/(N1+N2)∑kLk,α′=γ′/(N1+N2),
and α′′=γ′′/(N1+N2).


## 3. Numerical Experiments

In this section, we conduct two types of experiments
as examples of using the training method for RNNs proposed in Section2. The
first experiment shows the learning of five Lissajous curves. The second
experiment shows the training of multiple attractors of a Van der Pol
oscillator with 12 different parameters.


### 3.1. Experiment 1: Lissajous Curves

#### 3.1.1. Teacher I/O Time Series

Our first task is to learn the five Lissajous curves
defined by(13)xn,1(1)=45sin(2πnM),xn,2(1)=45cos(2πnM),xn,1(2)=45sin(4πnM),xn,2(2)=45cos(2πnM),xn,1(3)=xn,2(2),xn,2(3)=xn,1(2),xn,1(4)=45sin(2πnM),xn,2(4)=45cos(4πnM),xn,1(5)=xn,2(4),xn,2(5)=xn,1(4),
and we consider constants M=32 and Lk=200 for all 1≤k≤5 (see Figure 2).Trajectories of the teacher I/O time
series in experiment 1.
(a)(b)(c)(d)(e)


#### 3.1.2. Learning and Testing

We now describe the specific conditions applied to RNN
training. The time constantϵ is set to 0.1.
The number N2 of orthogonal units is 10,
and the dimension of a vector sn(k,l) is 2 for all 1≤l≤5.
Suppose that fθ is bistable with N3=30,Ns=15,
and ω=2.5.
The learning rates and momentum are given by γ=0.1,γ′=γ′′=0.01,
and β=0.9,
respectively.Figure3 shows the error function E(k)(t) for 20 000 learning steps. We also show the
Kullback-Leibler divergence between the teacher I/O time series and a sequence
of I/O units in the RNN computed by (3) which do not use external
perturbation by the teaching sequences. We use the Kullback-Leibler divergence
as a measure of the discrepancy between two sequences. Formally, the
Kullback-Leibler divergence between two probability distributions p and q is defined as(14)dKL(p,q)=∫(logp(x)−logq(x))p(x)dx.By definition, in order to
compute the Kullback-Leibler divergence, it is necessary to obtain probability
distributions of the teacher I/O time series and a sequence of I/O units.
However, obtaining the probability distribution of a sequence of I/O units is
very difficult. Therefore, we quantize a time series of real-valued vectors
into a symbolic sequence such that if the real value is less than 0,
then the symbol 0 is appropriated, and otherwise the symbol 1 is appropriated. In addition, we use the
probability distribution whereby sub-blocks with a block length of l appear in the symbolic sequence given by the
above quantization.Error and Kullback-Leibler
divergence between the teaching sequences and output generated by the RNN for 20 000 learning steps in experiment 1.
(a)(b)Figure4 describes attractors of the trained RNN
computed by (3) of which the initial state of internal units is u1(k)(t) for each 1≤k≤5.
By comparing the attractors with the teacher I/O time series displayed in
Figure 2, we can see that the RNN can generate sequences similar to training
data.Time seriesxn generated by the trained RNN in experiment 1.
For each time series, only the initial state u0 is different.
(a)(b)(c)(d)(e)In Figure5, examples of attractors for the RNN with
random initial states are displayed. This shows that, in addition to the
attractors corresponding to teacher I/O time series, there exist many
attractors of the RNN.Figure 5
Time seriesxn generated by the trained RNN with random
initial state u0 in experiment 1.


### 3.2. Experiment 2: Van der Pol Attractors

#### 3.2.1. Teacher I/O Time Series

Our second task is to learn multiple attractors given
by the Van der Pol oscillator with different parameters. The Van der Pol
oscillator defined by(15)d2ydt2−μ(1−y2)dydt+y=0is a model of an electronic
circuit that appeared in very early radios. It is well known that there exists
a limit cycle for the Van der Pol oscillator. In this experiment, we consider
twelve teacher I/O time series, where the kth teacher I/O time series xn(k) is given by(16)xn,1(k)=ay(t)+bk,xn,2(k)=ady(t)dt+ck,t=nτk,for μ=0.25 and a=0.15,
where bk and ck are constant parameters representing the
center position of the limit cycle, and τk is a time constant of the oscillator. We assume
that the parameters bk,ck,
and τk are given by combining the values of bk=±0.4, ck=±0.4,
and τk=2,4,6.
Figure 6 shows the teacher I/O time series given by (16). The length of
training data is Lk=200 for 1≤k≤12.Teaching sequences of experiment 2. (a) Trajectories onℝ2.
(b) Temporal trajectories of teaching sequences.
(a)(b)


#### 3.2.2. Learning and Testing

The parameters for learning are set as follows. Letfθ be bistable with N3=40 and Ns=20.
The dimension of the vector sn(k,l) is 1 for every 1≤l≤12 so that N2=12.
Other parameters are the same as in experiment 1.The error function and the Kullback-Leibler divergence
for 200 000 learning steps are displayed in Figure7.
Figure 8 shows attractors of the trained RNN, and the initial state of the
internal units of which is set to u1(k)(t) for every 1≤k≤12.Error and Kullback-Leibler divergence between
the teaching sequences and output generated by the RNN for 200 000 learning steps in experiment 1.
(a)(b)Time
seriesxn generated by the trained RNN in experiment 2.
For each time series, the initial state u0 is the same as the training phase. (a)
Trajectories on ℝ2.
(b) Temporal trajectories.
(a)(b)This result allows us to consider that the RNN acquires multiple periodic
attractors constituted by the teacher I/O time series.


## 3.1. Experiment 1: Lissajous Curves

### 3.1.1. Teacher I/O Time Series

Our first task is to learn the five Lissajous curves
defined by(13)xn,1(1)=45sin(2πnM),xn,2(1)=45cos(2πnM),xn,1(2)=45sin(4πnM),xn,2(2)=45cos(2πnM),xn,1(3)=xn,2(2),xn,2(3)=xn,1(2),xn,1(4)=45sin(2πnM),xn,2(4)=45cos(4πnM),xn,1(5)=xn,2(4),xn,2(5)=xn,1(4),
and we consider constants M=32 and Lk=200 for all 1≤k≤5 (see Figure 2).Trajectories of the teacher I/O time
series in experiment 1.
(a)(b)(c)(d)(e)


### 3.1.2. Learning and Testing

We now describe the specific conditions applied to RNN
training. The time constantϵ is set to 0.1.
The number N2 of orthogonal units is 10,
and the dimension of a vector sn(k,l) is 2 for all 1≤l≤5.
Suppose that fθ is bistable with N3=30,Ns=15,
and ω=2.5.
The learning rates and momentum are given by γ=0.1,γ′=γ′′=0.01,
and β=0.9,
respectively.Figure3 shows the error function E(k)(t) for 20 000 learning steps. We also show the
Kullback-Leibler divergence between the teacher I/O time series and a sequence
of I/O units in the RNN computed by (3) which do not use external
perturbation by the teaching sequences. We use the Kullback-Leibler divergence
as a measure of the discrepancy between two sequences. Formally, the
Kullback-Leibler divergence between two probability distributions p and q is defined as(14)dKL(p,q)=∫(logp(x)−logq(x))p(x)dx.By definition, in order to
compute the Kullback-Leibler divergence, it is necessary to obtain probability
distributions of the teacher I/O time series and a sequence of I/O units.
However, obtaining the probability distribution of a sequence of I/O units is
very difficult. Therefore, we quantize a time series of real-valued vectors
into a symbolic sequence such that if the real value is less than 0,
then the symbol 0 is appropriated, and otherwise the symbol 1 is appropriated. In addition, we use the
probability distribution whereby sub-blocks with a block length of l appear in the symbolic sequence given by the
above quantization.Error and Kullback-Leibler
divergence between the teaching sequences and output generated by the RNN for 20 000 learning steps in experiment 1.
(a)(b)Figure4 describes attractors of the trained RNN
computed by (3) of which the initial state of internal units is u1(k)(t) for each 1≤k≤5.
By comparing the attractors with the teacher I/O time series displayed in
Figure 2, we can see that the RNN can generate sequences similar to training
data.Time seriesxn generated by the trained RNN in experiment 1.
For each time series, only the initial state u0 is different.
(a)(b)(c)(d)(e)In Figure5, examples of attractors for the RNN with
random initial states are displayed. This shows that, in addition to the
attractors corresponding to teacher I/O time series, there exist many
attractors of the RNN.Figure 5
Time seriesxn generated by the trained RNN with random
initial state u0 in experiment 1.


## 3.1.1. Teacher I/O Time Series

Our first task is to learn the five Lissajous curves
defined by(13)xn,1(1)=45sin(2πnM),xn,2(1)=45cos(2πnM),xn,1(2)=45sin(4πnM),xn,2(2)=45cos(2πnM),xn,1(3)=xn,2(2),xn,2(3)=xn,1(2),xn,1(4)=45sin(2πnM),xn,2(4)=45cos(4πnM),xn,1(5)=xn,2(4),xn,2(5)=xn,1(4),
and we consider constants M=32 and Lk=200 for all 1≤k≤5 (see Figure 2).Trajectories of the teacher I/O time
series in experiment 1.
(a)(b)(c)(d)(e)


## 3.1.2. Learning and Testing

We now describe the specific conditions applied to RNN
training. The time constantϵ is set to 0.1.
The number N2 of orthogonal units is 10,
and the dimension of a vector sn(k,l) is 2 for all 1≤l≤5.
Suppose that fθ is bistable with N3=30,Ns=15,
and ω=2.5.
The learning rates and momentum are given by γ=0.1,γ′=γ′′=0.01,
and β=0.9,
respectively.Figure3 shows the error function E(k)(t) for 20 000 learning steps. We also show the
Kullback-Leibler divergence between the teacher I/O time series and a sequence
of I/O units in the RNN computed by (3) which do not use external
perturbation by the teaching sequences. We use the Kullback-Leibler divergence
as a measure of the discrepancy between two sequences. Formally, the
Kullback-Leibler divergence between two probability distributions p and q is defined as(14)dKL(p,q)=∫(logp(x)−logq(x))p(x)dx.By definition, in order to
compute the Kullback-Leibler divergence, it is necessary to obtain probability
distributions of the teacher I/O time series and a sequence of I/O units.
However, obtaining the probability distribution of a sequence of I/O units is
very difficult. Therefore, we quantize a time series of real-valued vectors
into a symbolic sequence such that if the real value is less than 0,
then the symbol 0 is appropriated, and otherwise the symbol 1 is appropriated. In addition, we use the
probability distribution whereby sub-blocks with a block length of l appear in the symbolic sequence given by the
above quantization.Error and Kullback-Leibler
divergence between the teaching sequences and output generated by the RNN for 20 000 learning steps in experiment 1.
(a)(b)Figure4 describes attractors of the trained RNN
computed by (3) of which the initial state of internal units is u1(k)(t) for each 1≤k≤5.
By comparing the attractors with the teacher I/O time series displayed in
Figure 2, we can see that the RNN can generate sequences similar to training
data.Time seriesxn generated by the trained RNN in experiment 1.
For each time series, only the initial state u0 is different.
(a)(b)(c)(d)(e)In Figure5, examples of attractors for the RNN with
random initial states are displayed. This shows that, in addition to the
attractors corresponding to teacher I/O time series, there exist many
attractors of the RNN.Figure 5
Time seriesxn generated by the trained RNN with random
initial state u0 in experiment 1.


## 3.2. Experiment 2: Van der Pol Attractors

### 3.2.1. Teacher I/O Time Series

Our second task is to learn multiple attractors given
by the Van der Pol oscillator with different parameters. The Van der Pol
oscillator defined by(15)d2ydt2−μ(1−y2)dydt+y=0is a model of an electronic
circuit that appeared in very early radios. It is well known that there exists
a limit cycle for the Van der Pol oscillator. In this experiment, we consider
twelve teacher I/O time series, where the kth teacher I/O time series xn(k) is given by(16)xn,1(k)=ay(t)+bk,xn,2(k)=ady(t)dt+ck,t=nτk,for μ=0.25 and a=0.15,
where bk and ck are constant parameters representing the
center position of the limit cycle, and τk is a time constant of the oscillator. We assume
that the parameters bk,ck,
and τk are given by combining the values of bk=±0.4, ck=±0.4,
and τk=2,4,6.
Figure 6 shows the teacher I/O time series given by (16). The length of
training data is Lk=200 for 1≤k≤12.Teaching sequences of experiment 2. (a) Trajectories onℝ2.
(b) Temporal trajectories of teaching sequences.
(a)(b)


### 3.2.2. Learning and Testing

The parameters for learning are set as follows. Letfθ be bistable with N3=40 and Ns=20.
The dimension of the vector sn(k,l) is 1 for every 1≤l≤12 so that N2=12.
Other parameters are the same as in experiment 1.The error function and the Kullback-Leibler divergence
for 200 000 learning steps are displayed in Figure7.
Figure 8 shows attractors of the trained RNN, and the initial state of the
internal units of which is set to u1(k)(t) for every 1≤k≤12.Error and Kullback-Leibler divergence between
the teaching sequences and output generated by the RNN for 200 000 learning steps in experiment 1.
(a)(b)Time
seriesxn generated by the trained RNN in experiment 2.
For each time series, the initial state u0 is the same as the training phase. (a)
Trajectories on ℝ2.
(b) Temporal trajectories.
(a)(b)This result allows us to consider that the RNN acquires multiple periodic
attractors constituted by the teacher I/O time series.


## 3.2.1. Teacher I/O Time Series

Our second task is to learn multiple attractors given
by the Van der Pol oscillator with different parameters. The Van der Pol
oscillator defined by(15)d2ydt2−μ(1−y2)dydt+y=0is a model of an electronic
circuit that appeared in very early radios. It is well known that there exists
a limit cycle for the Van der Pol oscillator. In this experiment, we consider
twelve teacher I/O time series, where the kth teacher I/O time series xn(k) is given by(16)xn,1(k)=ay(t)+bk,xn,2(k)=ady(t)dt+ck,t=nτk,for μ=0.25 and a=0.15,
where bk and ck are constant parameters representing the
center position of the limit cycle, and τk is a time constant of the oscillator. We assume
that the parameters bk,ck,
and τk are given by combining the values of bk=±0.4, ck=±0.4,
and τk=2,4,6.
Figure 6 shows the teacher I/O time series given by (16). The length of
training data is Lk=200 for 1≤k≤12.Teaching sequences of experiment 2. (a) Trajectories onℝ2.
(b) Temporal trajectories of teaching sequences.
(a)(b)


## 3.2.2. Learning and Testing

The parameters for learning are set as follows. Letfθ be bistable with N3=40 and Ns=20.
The dimension of the vector sn(k,l) is 1 for every 1≤l≤12 so that N2=12.
Other parameters are the same as in experiment 1.The error function and the Kullback-Leibler divergence
for 200 000 learning steps are displayed in Figure7.
Figure 8 shows attractors of the trained RNN, and the initial state of the
internal units of which is set to u1(k)(t) for every 1≤k≤12.Error and Kullback-Leibler divergence between
the teaching sequences and output generated by the RNN for 200 000 learning steps in experiment 1.
(a)(b)Time
seriesxn generated by the trained RNN in experiment 2.
For each time series, the initial state u0 is the same as the training phase. (a)
Trajectories on ℝ2.
(b) Temporal trajectories.
(a)(b)This result allows us to consider that the RNN acquires multiple periodic
attractors constituted by the teacher I/O time series.


## 4. Numerical Analysis

### 4.1. Contraction and Bistability

Assume thatX and U are sets and that U is equipped with a metric structure. A
function  f:X×U→U is a contraction with respect to U if a real value C∈[0,1) exists such that the
inequality(17)∀u1,u2∈U∀x∈X|f(x,u1)−f(x,u2)|≤C|u1−u2|holds for all x∈X and u1,u2∈U.
Lemma 2.
Let
one consider a dynamical system onℝN3 defined by the transition  function un+1=fθ(gθ(un),un),
where fθ and gθ are defined in (1) and (2), respectively.
Assume that each element wij of the matrix W3 satisfies (4), and ν∈ℝ is the maximum absolute value of elements in W1, W2,
and b.
If there exist three solutions of(18)x−ωtanh(x)+v(N1+N2+1)=0,x∈ℝ,then
(1)
there are2Ns invariant sets of a dynamical system un+1=fθ(gθ(un),un);(2)
suppose thatU⊂ℝN3 is an invariant set of the dynamical system;
then, the restriction of fθ to ℝN1×ℝN2×U is a contraction with respect to U and the maximum norm |⋅|∞,
where |u−u′|∞≡max1≤i≤N3|ui−ui′|.Proof.
We suppose that (18) has three
solutions, such as x1>x2>x3 (see Figure 9). In general, x1,x2>0 and x3<0.
(1) Assume1≤i≤Ns and x1≥un(i)≥x2.
Then, the expression(19)un(i)−ωtanh(un(i))+v(N1+N2+1)≤0⇒un(i)−ωtanh(un(i))−On(i)≤0⟺−ϵ(un(i)−ωtanh(un(i))−On(i))≥0⟺(1−ϵ)un(i)+ϵ(ωtanh(un(i))+On(i))≥un(i)⟺un+1(i)≥un(i)is satisfied for all xn∈ℝN1,rn∈ℝN2,
and un,un′∈ℝN3,
where On(i) is the ith element of the vector On=W1⋅xn+W2⋅rn+b.
Hence, un+1(i)≥un(i) if x1≥un(i)≥x2.
Furthermore, if un(i)≥x1,
then un+1(i)≥x1 because(20)un(i)≥x1⇒tanh(un(i))≥tanh(x1)(becausex1>0)⇒(1−ϵ)un(i)+ϵ(ωtanh(un(i))+On(i))≥(1−ϵ)x1+ϵ(ωtanh(x1)+On(i))⇒un+1(i)≥x1(becausex1≥un(i)≥x2⇒un+1(i)≥un(i)).Therefore, the region [x1,∞) is a stable set of the ith element of vector un satisfying the fact
that if un(i)∈[x1,∞) then un+k(i)∈[x1,∞) for any k∈ℕ.
Similarly, we can easily show that ifun(i)≤−x1,
then un+1(i)≤−x1.
Thus, there are two stable regions of the ith element of vector un for each 1≤i≤Ns.
Then, there are 2Ns invariant sets.
(2) LetU⊂ℝN3 be the invariant set presented above. Assume
that un,un′∈U.
For any1≤i≤Ns,
the inequality(21)|un+1(i)−un+1′(i)|=|(1−ϵ)(un(i)−un′(i))+ϵω(tanh(un(i))−tanh(un′(i)))|<|(1−ϵ)(un(i)−un′(i))+ϵ(un(i)−un′(i))|=|un(i)−un′(i)|holds because if |x|≥x1,
then ω|dtanh(x)/dx|<1.
On the other hand, for everyNs≤i≤N3,(22)|un+1(i)−un+1′(i)|=|(1−ϵ)(un(i)−un′(i))+ϵ∑jwij(tanh(un(j))−tanh(un′(i)))|≤(1−ϵ)|un(i)−un′(i)|+ϵ|∑jwij(tanh(un(j))−tanh(un′(j)))|≤(1−ϵ)|un(i)−un′(i)|+ϵ|∑jwij(un(j)−un′(j))|≤(1−ϵ)|un(i)−un′(i)|+ϵN3maxj|wij(un(j)−un′(j))|≤(1−ϵ)|un(i)−un′(i)|+ϵN3maxj|wij|maxj|un(j)−un′(j)|≤((1−ϵ)+ϵN3maxj|wij|)maxj|un(j)−un′(j)|<maxj|un(j)−un′(j)|.
Then,|un+1−un+1′|∞<|un−un′|∞ is obtained for any un,un′∈U.
Accordingly, the restriction of fθ to ℝN1×ℝN2×U is a contraction with respect to U and the maximum norm |⋅|∞.Figure 9
Schematic diagram of (18).For anyN1,N2∈ℕ and ν≥0,
there is a real number q such that if ω≥q,
then (18) has three solutions. Thus, if ω is large enough and matrices W1 and W2 represent small connection weights, then fθ contains 2Ns invariant sets, and each restriction of fθ to an invariant set is a contraction with
respect to a third input. Moreover, the integer N3−Ns is the effective degree of freedom for each
contraction mapping restricted to an invariant set. If N3−Ns is a large value, then RNN can acquire a more
complex time sequence. In Figures 10 and 11, we plot the Kullback-Leibler
divergence of the trained RNN for parameters ω and Ns,
in which the training data are the same as those for experiment 1. These
results imply that it is necessary that ω,2Ns,
and N3−Ns be large values in order to learn multiple
attractor dynamics.Figure 10
Kullback-Leibler divergence between
the teaching sequences and output generated by the trained RNN withϵ=0.1,N2=10 (dimsn(k,l)=2), N3=30,
and Ns=15.Kullback-Leibler divergence between the
teaching sequences and output generated by the trained RNN withϵ=0.1,ω=2.5,
and N2=10 (dimsn(k,l)=2). (a) N3=20,
(b) N3=30,
and (c) N3=40.
(a)(b)(c)


### 4.2. Orthogonality

In the last paragraph of the previous section, we have
shown that RNNs have many stable regions, and the existence of the stable
regions plays an important role in the learning of multiple sequences. However,
the existence of multiple stable regions is not sufficient for success in the
multiple attractor learning because if the change of parameters corresponding
to each time series influences other changes, each time series cannot
necessarily be embedded into each region. Similarly, this problem appears in
the method of RNNPB.In the training algorithm defined in Section2, each
state of orthogonal units r=(s(1),…,s(m)) is trained by (5) and (11). Thus, firing of s(k) only occurs in the generation
of the kth teaching sequence. This implies that
orthogonal units allow the conflict of parameter changes caused by multiple
time series learning to be avoided because orbits corresponding to each
teaching I/O time series run around the orthogonal state space of the trained
RNN.In order to show the effect of the orthogonal units on
the conflict among teaching sequences, we consider thekth learning ratio dvij(k)(t) defined by(23)dvij(k)(t)=∂E(k)(t)∂vij(∂E(t)∂vij)−1,where vij is an element of the matrix V.
If dvij(k)(t) is nearly equal to 1,
then the change in vij is approximately independent of teaching
sequences rather than the kth sequence. In Figure 12, we plot the value dV(k)(t) determined by(24)dV(k)(t)=1(N3−Ns)|R(k)|∑i∈R(k)∑j=NsN3dvij(k)(t),where R(k) is a set of indices corresponding to the
elements of the vector s(k).
The value dV(k)(t) represents the average of the kth learning ratio for connections between
internal units and orthogonal units s(k).
In this numerical experiment, for each learning step, dV(k)(t) is clearly larger than 1/m=0.2,
where m is the number of teaching sequences. Then, the
sum of the kth learning ratios of connection weights
between internal units and orthogonal units s(k) is dominant. Therefore, in changing matrix V,
there is no conflict generated by multiple teaching sequences. However, we
could not find a strong bias of the learning ratio for the matrices W1 and W2 and every element vij of V with i<Ns.
Thus, we consider that connection weights between internal units and orthogonal
units encode information on an individual time series, and other connection
weights encode whole information.Figure 12
AveragedV(k)(t) of the kth learning ratio for the connections between
internal units and orthogonal units s(k) for 200 000 learning steps in experiment 1.


## 4.1. Contraction and Bistability

Assume thatX and U are sets and that U is equipped with a metric structure. A
function  f:X×U→U is a contraction with respect to U if a real value C∈[0,1) exists such that the
inequality(17)∀u1,u2∈U∀x∈X|f(x,u1)−f(x,u2)|≤C|u1−u2|holds for all x∈X and u1,u2∈U.
Lemma 2.
Let
one consider a dynamical system onℝN3 defined by the transition  function un+1=fθ(gθ(un),un),
where fθ and gθ are defined in (1) and (2), respectively.
Assume that each element wij of the matrix W3 satisfies (4), and ν∈ℝ is the maximum absolute value of elements in W1, W2,
and b.
If there exist three solutions of(18)x−ωtanh(x)+v(N1+N2+1)=0,x∈ℝ,then
(1)
there are2Ns invariant sets of a dynamical system un+1=fθ(gθ(un),un);(2)
suppose thatU⊂ℝN3 is an invariant set of the dynamical system;
then, the restriction of fθ to ℝN1×ℝN2×U is a contraction with respect to U and the maximum norm |⋅|∞,
where |u−u′|∞≡max1≤i≤N3|ui−ui′|.Proof.
We suppose that (18) has three
solutions, such as x1>x2>x3 (see Figure 9). In general, x1,x2>0 and x3<0.
(1) Assume1≤i≤Ns and x1≥un(i)≥x2.
Then, the expression(19)un(i)−ωtanh(un(i))+v(N1+N2+1)≤0⇒un(i)−ωtanh(un(i))−On(i)≤0⟺−ϵ(un(i)−ωtanh(un(i))−On(i))≥0⟺(1−ϵ)un(i)+ϵ(ωtanh(un(i))+On(i))≥un(i)⟺un+1(i)≥un(i)is satisfied for all xn∈ℝN1,rn∈ℝN2,
and un,un′∈ℝN3,
where On(i) is the ith element of the vector On=W1⋅xn+W2⋅rn+b.
Hence, un+1(i)≥un(i) if x1≥un(i)≥x2.
Furthermore, if un(i)≥x1,
then un+1(i)≥x1 because(20)un(i)≥x1⇒tanh(un(i))≥tanh(x1)(becausex1>0)⇒(1−ϵ)un(i)+ϵ(ωtanh(un(i))+On(i))≥(1−ϵ)x1+ϵ(ωtanh(x1)+On(i))⇒un+1(i)≥x1(becausex1≥un(i)≥x2⇒un+1(i)≥un(i)).Therefore, the region [x1,∞) is a stable set of the ith element of vector un satisfying the fact
that if un(i)∈[x1,∞) then un+k(i)∈[x1,∞) for any k∈ℕ.
Similarly, we can easily show that ifun(i)≤−x1,
then un+1(i)≤−x1.
Thus, there are two stable regions of the ith element of vector un for each 1≤i≤Ns.
Then, there are 2Ns invariant sets.
(2) LetU⊂ℝN3 be the invariant set presented above. Assume
that un,un′∈U.
For any1≤i≤Ns,
the inequality(21)|un+1(i)−un+1′(i)|=|(1−ϵ)(un(i)−un′(i))+ϵω(tanh(un(i))−tanh(un′(i)))|<|(1−ϵ)(un(i)−un′(i))+ϵ(un(i)−un′(i))|=|un(i)−un′(i)|holds because if |x|≥x1,
then ω|dtanh(x)/dx|<1.
On the other hand, for everyNs≤i≤N3,(22)|un+1(i)−un+1′(i)|=|(1−ϵ)(un(i)−un′(i))+ϵ∑jwij(tanh(un(j))−tanh(un′(i)))|≤(1−ϵ)|un(i)−un′(i)|+ϵ|∑jwij(tanh(un(j))−tanh(un′(j)))|≤(1−ϵ)|un(i)−un′(i)|+ϵ|∑jwij(un(j)−un′(j))|≤(1−ϵ)|un(i)−un′(i)|+ϵN3maxj|wij(un(j)−un′(j))|≤(1−ϵ)|un(i)−un′(i)|+ϵN3maxj|wij|maxj|un(j)−un′(j)|≤((1−ϵ)+ϵN3maxj|wij|)maxj|un(j)−un′(j)|<maxj|un(j)−un′(j)|.
Then,|un+1−un+1′|∞<|un−un′|∞ is obtained for any un,un′∈U.
Accordingly, the restriction of fθ to ℝN1×ℝN2×U is a contraction with respect to U and the maximum norm |⋅|∞.Figure 9
Schematic diagram of (18).For anyN1,N2∈ℕ and ν≥0,
there is a real number q such that if ω≥q,
then (18) has three solutions. Thus, if ω is large enough and matrices W1 and W2 represent small connection weights, then fθ contains 2Ns invariant sets, and each restriction of fθ to an invariant set is a contraction with
respect to a third input. Moreover, the integer N3−Ns is the effective degree of freedom for each
contraction mapping restricted to an invariant set. If N3−Ns is a large value, then RNN can acquire a more
complex time sequence. In Figures 10 and 11, we plot the Kullback-Leibler
divergence of the trained RNN for parameters ω and Ns,
in which the training data are the same as those for experiment 1. These
results imply that it is necessary that ω,2Ns,
and N3−Ns be large values in order to learn multiple
attractor dynamics.Figure 10
Kullback-Leibler divergence between
the teaching sequences and output generated by the trained RNN withϵ=0.1,N2=10 (dimsn(k,l)=2), N3=30,
and Ns=15.Kullback-Leibler divergence between the
teaching sequences and output generated by the trained RNN withϵ=0.1,ω=2.5,
and N2=10 (dimsn(k,l)=2). (a) N3=20,
(b) N3=30,
and (c) N3=40.
(a)(b)(c)


## 4.2. Orthogonality

In the last paragraph of the previous section, we have
shown that RNNs have many stable regions, and the existence of the stable
regions plays an important role in the learning of multiple sequences. However,
the existence of multiple stable regions is not sufficient for success in the
multiple attractor learning because if the change of parameters corresponding
to each time series influences other changes, each time series cannot
necessarily be embedded into each region. Similarly, this problem appears in
the method of RNNPB.In the training algorithm defined in Section2, each
state of orthogonal units r=(s(1),…,s(m)) is trained by (5) and (11). Thus, firing of s(k) only occurs in the generation
of the kth teaching sequence. This implies that
orthogonal units allow the conflict of parameter changes caused by multiple
time series learning to be avoided because orbits corresponding to each
teaching I/O time series run around the orthogonal state space of the trained
RNN.In order to show the effect of the orthogonal units on
the conflict among teaching sequences, we consider thekth learning ratio dvij(k)(t) defined by(23)dvij(k)(t)=∂E(k)(t)∂vij(∂E(t)∂vij)−1,where vij is an element of the matrix V.
If dvij(k)(t) is nearly equal to 1,
then the change in vij is approximately independent of teaching
sequences rather than the kth sequence. In Figure 12, we plot the value dV(k)(t) determined by(24)dV(k)(t)=1(N3−Ns)|R(k)|∑i∈R(k)∑j=NsN3dvij(k)(t),where R(k) is a set of indices corresponding to the
elements of the vector s(k).
The value dV(k)(t) represents the average of the kth learning ratio for connections between
internal units and orthogonal units s(k).
In this numerical experiment, for each learning step, dV(k)(t) is clearly larger than 1/m=0.2,
where m is the number of teaching sequences. Then, the
sum of the kth learning ratios of connection weights
between internal units and orthogonal units s(k) is dominant. Therefore, in changing matrix V,
there is no conflict generated by multiple teaching sequences. However, we
could not find a strong bias of the learning ratio for the matrices W1 and W2 and every element vij of V with i<Ns.
Thus, we consider that connection weights between internal units and orthogonal
units encode information on an individual time series, and other connection
weights encode whole information.Figure 12
AveragedV(k)(t) of the kth learning ratio for the connections between
internal units and orthogonal units s(k) for 200 000 learning steps in experiment 1.


## 5. Discussion

In this report, we have investigated a method of
embedding multiple time series into a single RNN. In order to clarify the
characteristics of the proposed approach, we compare the proposed approach with
other approaches with respect to information representation of multiple
sequences in the models. The mixture-of-RNN-experts-type model composes local
representation in an RNN for each sequence. The local representation provides
robustness against changing the parameters in learning, but it lacks the
ability to extract common patterns included in the sequences because of the
independency of the local representation. In the proposed model, the local
representation is constructed into orthogonal units, while the global
representation is also constructed into internal units using the connection
weights between I/O units and internal units. Since each sequence generated by
the proposed model shares the state space and connection weights, the model can
extract common patterns of the sequences as well as conventional neural
networks.Another characteristic, which clarifies the difference
between our model and other models, is whether the classification of each time
series is self-organized into the state space. For example, in the
mixture-of-RNN-experts-type model, the allocation of time series to each RNN is
determined automatically. As another example, in the RNNPB model, PB values are
self-organized such that the PB can individualize each time series. On the
other hand, the proposed model needs the information of orthogonalization for
each time series. Since the sparse firing patterns which appear in orthogonal
units, corresponding to time series, are given as teaching information
externally, the classification of sequences is not self-organized. The
characteristic whereby the time series cannot be automatically classified is a
disadvantage of the proposed model. However, the time series can be classified
using other clustering techniques before applying the proposed method. Thus, by
combining the proposed method and other clustering techniques, an algorithm
that automatically classifies and generates multiple time series can be
constructed.


## 6. Conclusion

In this paper, we have presented an RNN model and a
learning algorithm that can acquire the ability to generate multiple sequences.
The RNN model consists of two distinct properties called bistability and
orthogonality. Bistability guarantees the existence of multiple attractor
structures in RNNs, and provides the RNNs with contraction transition mapping.
Orthogonality, which is given as a function of the orthogonal vectors of RNNs,
helps prevent conflicts with respect to parameter changes caused by multiple
training sequences. In the numerical experiments, RNNs which have bistability
and orthogonality can learn multiple periodic attractors constituted by five
Lissajous curves or 12 Van der Pol oscillators. Based on these results, the
proposed model can be applied to the modeling of various types of dynamical
systems that include multiple attractors.



---

*Source: 846040-2008-10-27.xml*