Context-Dependent Processing of
Spatiotemporal Patterns Based on Interaction
Between Neurodynamical Systems
Takashi Hasuo1 , Ken Yamane1 , and Masahiko Morita1
Graduate School of Systems and Information Engineering, University of Tsukuba,
Tsukuba-city, 305-8573, Japanhasuo@bcl.esys.tsukuba.ac.jp
Abstract. Dynamics of traditional neural network models are generally time-invariant. For that reason, they have limitations in contextdependent processing. We present a new method, dynamic desensitization, of varying neurodynamics continuously and construct a basic modelof interaction between neurodynamical systems. This model comprises
two nonmonotone neural networks storing sequential patterns as trajectory attractors. The dynamics of respective networks are modified according to the states of other networks. Using numerical experiments,
we also show that the model can recognize and recall complex sequenceswith identical patterns in diﬀerent positions.

1

Introduction

The brain can be regarded as an assemblage of dynamical systems comprising
many neurons that perform spatio-temporal pattern processing. Consequently,
neurodynamical systems comprising artificial neural networks with recurrent
connections and continuous-time dynamics are thought to oﬀer great potential
for brain-like information processing. However, existing neurodynamical systems
have limited capability, particularly in terms of context-dependent processing.
Two explanations for this limited ability are the following. First, the dynamics are usually time-invariant (at least in the short run). For that reason, thesystem always makes the same transition at the same state unless it receives
external input. Second, no appropriate method is known for making diﬀerent
neurodynamical systems interact. If two networks are interconnected in a usual
manner, they constitute a single neurodynamical system with time-invariant dynamics, rather than two diﬀerent systems. In such a case, the influence of system
A on system B depends on the state of A, but not on that of B.
In this paper, we propose a method of varying the dynamics of a recurrent neural network according to another network. We construct a model forcontext-dependent processing such as recognition and recall of complex sequential patterns.

2

Dynamic Desensitization

As a method of varying the dynamics of a neural network discretely, Moritaet al.[1] proposed selective desensitization. This method desensitizes about halfFig. 1. Methods of varing neurodynamics:
Thick arrows represent trajectories of thenetwork state. Planes represent subspaces.
The dashed arrows represent trajectory attractors in the subspaces.

Fig. 2. Model structure.

of the neurons or renders their output as neutral, depending on a given modification pattern. Through this operation, the modified state of the network is
projected onto a subspace of the state space, and produces transitions according
to the dynamics in the subspace. If trajectory attractors[2] are formed in diﬀerent subspaces, the network state reaches diﬀerent target patterns according tomodification patterns, as shown schematically in Fig. 1(a).
In this model, however, the modification pattern is restricted to a static
pattern so that trajectory attractors can be formed within individual subspaces.
For that reason, we call this “static desensitization.”
Here we propose “dynamic desensitization” by extending the modification
pattern of static desensitization to a dynamic or spatiotemporal pattern. Thenthe set of desensitized neurons, or the subspace onto which the network state isprojected, varies continuously with time, as depicted in Fig. 1(b). In this case, asingle trajectory attractor to the target state cannot be formed. If the change ofthe modification pattern is suﬃciently slow, however, the network state makes
short transitions in each of a series of subspaces and can thereby reach the target.

3

Model

Using the method described above, we constructed a basic model of interaction
between neurodynamical systems (Fig. 2). This model consists of two nonmonotone neural networks that have the same number of neurons and obey the samedynamic equation.
Specifically, the i-th neuron (i=1,. . . ,n) of either network acts according toτnXdui
wij yj + zi , yi = g(vi ) · f (ui ),= −ui +dt
j=1(1)

where ui denotes the instantaneous potential, yi is the output, zi is the external
input. In addition, wij represents the synaptic weight from the j-th neuron ofTable 1. Input sequences: Letters represent
n-dimensional binary patterns selected randomly; {OABCD}T denotes a spatiotemporal pattern changing from a pattern O
via A, B and C into D over a period of T .
Each trajctory overlaps with others in every unit section, e.g., {AB} is included ins1 , s2 and s3 , and {BC} is included in s1 ,s2 and s5 .

Fig. 3. State transition diagram.

s1 = {OABCD}T s2 = {OABCB}T
s3 = {OABDC}T s4 = {OACBD}T
s5 = {OBCBD}T s6 = {OBDAC}T
s7 = {OCADB}T s8 = {OCADA}T
s9 = {ODADC}T s10= {ODCDA}Tthe same network, vi is the instantaneous potential of the corresponding neuronof the other network, and τ is a time constant. Also, g(vi ) is a sigmoid function
which takes a value from 0 to 1, and f (ui ) is a nonmonotonic output function[2].
If vi takes a positive large value, the output is yi ' f (ui ), but if vi takes anegative large value, then g(vi ) and yi are nearly zero, which means that theneuron is desensitized. Here we consider xi ≡ sgn(ui ) (sgn(u) = 1 for u > 0 and−1 for u ≤ 0) and refer to the binary vector x = (x1 , . . . , xn ) as the current
state of the network.
The two networks have a similar structure, but they diﬀer in function (Fig. 3).
One network receives an input of complex spatiotemporal patterns from theoutside and stores them. This network is termed the input part, but it also
shows the recalled pattern when no input pattern is fed. The other network,
termed the context part, does not receive the external input directly, but it
autonomously changes its state from a fixed initial state O0 , via some branch
points, to a terminal state C µ in accordance with the state transitions of theinput part.

4

Computer SimulationWe carried out simulation experiments using the 10 spatiotemporal patterns
s1 , . . . , s10 presented in Table 1. Their dimensions (and size of each network)
were n = 800 and their temporal length was T = 20τ . We also generated cµ ={O0 • • • C µ }T (µ = 1, . . . , 10) corresponding to sµ , where • represents a branch
point described above. Subsequently, we trained the model by giving sµ and cµ
as learning signals to the input and context parts, respectively, to store them as
trajectory attractors (see refs. [2] and [1] for the learning algorithm).
After completion of 30 training cycles, we gave various input patterns andexamined the behavior of the model. Figure 4 presents two examples in which(a) s2 was fed with 10% noise and with temporal expansion and contraction,and (b) only short sections of s9 were fed. The state of the context part reaches
C 2 in (a) and C 9 in (b), meaning that the model recognized the input pattern as

Fig. 4. Behavior of the model: Similarities of the network state (upper and middlegraphs) or the current input (lower graph) to reference patterns are plotted against
time scaled by the time constant τ .

s2 or s9 . It is also apparent that a spatiotemporal pattern similar to the originalis recalled in the input part.
This result, together with those of many other experiments, indicates thatthe two networks act autonomously and cooperatively through mutual dynamicdesensitization.

5Concluding Remarks

We have proposed a dynamic desensitization method to modify a neurodynamical system using another system. We have also shown that two systems which
are interconnected by this method can recognize and recall complex sequenceswith many overlapping sections, even if they have substantial noise, have someblank sections, or are temporally expanded or contracted.
Interaction through dynamic desensitization diﬀers greatly from interaction
which takes place through ordinary synaptic connections between neurons: theinfluence on dynamics of one system depends on the states of both systems.
This method also presents an advantage in that the number of interconnections
required is merely n. By developing this method, more than two neurodynamicalsystems can interact to provide more powerful capability, which remains as asubject for future study.

References
1. Morita, M., Murata, S., Morokami, S.: Context-dependent sequential recall by a
trajectory attractor network with selective desensitization. Proceedings of the Third
International Conference on Neural Networks and Artificial Intelligence. (2003) 235—238
2. Morita, M.: Memory and learning of sequential patterns by nonmonotone neuralnetwork. Neural Networks 9 (1996) 1477—1489