Context-dependent Sequential Recall by a Trajectory Attractor
Network with Selective Desensitization
Masahiko MORITA 1), Kazuhiko MURATA 2), Shigemitsu MOROKAMI 2)
1) Institute of Engineering Mechanics and Systems, University of Tsukuba
2) Doctoral Program in System Information Engineering, University of Tsukuba
1-1-1 Tennodai, Tsukuba, Ibaraki 305-8573, Japan
E-mail: mor@bcl.esys.tsukuba.ac.jp
URL: http://volga.esys.tsukuba.ac.jp/~mor/
Abstract: We present a model composed of nonmonotonic
neurons that recalls various target patterns from the samecue pattern depending on a given context pattern, which
was essentially difficult for conventional neural networkmodels. In this model, the state of the network is projected
onto a subspace by desensitizing a part of neurons
depending on the context, and shifts along a trajectory
attractor in the subspace to reach the target state. The
model can simulate an arbitrary finite state automaton
without limitations of size, keeping the merits of distributedrepresentation.
Keywords: state transition, trajectory attractor,
context-dependent association, nonmonotonic neuron,
finite automaton.

1. INTRODUCTION
Let us consider a simplified problem ofcontext-dependent association in a neural network withfully distributed representation. Specifically, target patternT µν to be recalled depends not only on cue pattern S µν(µ = 1,L, p) but also on pattern C (ν = 1,L, q) representing
context, where the elements of patterns S µ , Cν and T µ ,νtake 1 or -1 randomly with equal probabilities. Although
this problem appears very simple, conventional neuralnetworks have serious difficulty in solving it.
For example, a two-layer network shown in Fig.l (a) inwhich the input units representing S µ and Cν are directly
connected to the output units cannot realize theinput-output relation expressed by Table 1. This is because
the set of target patterns are the same for every S µ and forevery Cν . Even if all target patterns are mutually different,
no connection weights solve the problem when p and q arelarge enough, since the average of many target patterns
associated with the same S µ (or Cν ) does not vary muchwith µ (ν ) because of the averaging effect. Moreover,this difficulty is not essentially resolved by introducing
hidden units, between the input and output units unless we
use local representation, pq hidden units each of which
encodes a particular combination of S µ and Cν .
This kind of contextual modification such that S (C ) ,an object S in a context C , is represented by theconcatenated pattern ( S ,C ) has been used in most modelsof contextual processing. However, they do not work wellunless the size (p and q) of the problem is small enough,due to the above difficulty caused by one-to-manycorrespondence. This means that contextual processing byneural networks based on fully distributed representation
has strong restriction.
This paper presents a model of context-dependentassociation that solves the above problem without using
local representation.

2. THE MODEL2.1. Dynamics and contextual modification
The present model has been developed by modifying
the dynamics of the analog Hopfield model, namely
continuous-time fully recurrent neural networks.
Specifically, the i-th neuron (i =1,L, n) acts according toτndui= −ui + ∑ wij y j + zi ,dtj =1

yi = gi ⋅ f (ui ) ,

(1)
(2)

where ui denotes the instantaneous potential, wij thesynaptic weight from the j-th neuron, y i the output, zi theexternal input, τ a time constant, gi output gain, and f (u )the activation function. These equations are the same asconventionally used except that gi is controllable andf (u ) is a nonmonotonic functionf (u ) =1 − e − cu 1 − e c′(|u |− h ),⋅1 + e − cu 1 + e c′(|u|− h )(3)where c, c' and h are positive constants (we substitute c=50,c'=10, h=0.5 in the experiments described later).
Since the polarity of ui is important in this model, weconsider xi = sgn(ui ) and treat the vector x = ( x1 ,L, xn ) as
the network state, where sgn(u ) = 1 for u > 0 and -1 foru ≤ 0 . The network state x at an instant is represented by apoint in the state space consisting of 2n possible states.
When x changes, it almost always moves to an adjacentpoint in the state space because xi changes
asynchronously. Consequently, x leaves a track with time,
which we call the trajectory of x.
If gi ≡ 1 for all neurons, the network is identical to anexisting model known as the nonmonotone neural network(NNN[1]). In the present model, however, the gain gi isTable 1. Example of the relation of target patterns to cue
and context patterns that is difficult to realize.

Cν

Sμ

Sμ
Tμν
・・Cν

(a)

・・・・・・・・・・

Tμν

(b)

Fig.1 – Methods of contextual modification: (a)conventional concatenation, and (b) selective desensitization
used in the present model.

set to zero depending on the context. Since the output y iof the neuron with gi = 0 does not vary with the input, werefer to this operation as desensitization, and about half ofthe neurons are `desensitized' when the network receives acontextual modification. The gain vector G = (g1 ,L, g n ) is
specified by the context and is different in different
contexts, in other words, G represents the context.
In the following, we regard G in the same light as thecontext pattern C , and deal with the simplest case thatg i = (1 + ci ) / 2 , where ci (i = 1,L, n ) is the i-th componentof C and takes ± 1 . We also define the network state
modified by context C as x (C ) ≡ (g1 x1 ,L, g n xn ) , which is athree-valued (1, 0, -1) vector. In the same way, we define
the patterns S and T modified by C as S (C ) ≡ (g1s1,L, gnsn )and , T (C ) ≡ (g1t1 ,L, g ntn ) respectively.

2.2. Forming trajectory attractors
A NNN can have many string-shaped attractors,
termed trajectory attractors, in its state space, along which
the network stably makes continuous state transitions [2].
This enables the network to associate a cue with anarbitrary target, but not to recall different targets
depending on contexts. Thus in the present model, we form
trajectory attractors in the subspace composed of the valid(not desensitized) neurons in each context. The concretealgorithm for it is as follows.
First, we give an initial state x (Cν ) = S µ (Cν ) and input
the learning signal r = S µ (Cν ) in the form zi = λ ri to thenetwork modified by Cν , where λ denotes the input
intensity and ri is the i-th element of r.Then we change r
bit by bit to T µ ,ν (Cν ) over some period (5 τ in thesimulations below), modifying synaptic weights wij
between valid neurons according toτ′dwij
dt= − wij + α ri y j ,C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 C 10
S T 1 T 2 T 3 T 4 T 5 T 6 T 7 T 8 T 9 T 10
S 2 T 10 T 1 T 2 T 3 T 4 T 5 T 6 T 7 T 8 T 9
S 3 T 9 T 10 T 1 T 2 T 3 T 4 T 5 T 6 T 7 T 8
S 4 T 8 T 9 T 10 T 1 T 2 T 3 T 4 T 5 T 6 T 7
S 5 T 7 T 8 T 9 T 10 T 1 T 2 T 3 T 4 T 5 T 6
S 6 T 6 T 7 T 8 T 9 T 10 T 1 T 2 T 3 T 4 T 5
S 7 T 5 T 6 T 7 T 8 T 9 T 10 T 1 T 2 T 3 T 4
S 8 T 4 T 5 T 6 T 7 T 8 T 9 T 10 T 1 T 2 T 3
S 9 T 3 T 4 T 5 T 6 T 7 T 8 T 9 T 10 T 1 T 2
S 10 T 2 T 3 T 4 T 5 T 6 T 7 T 8 T 9 T 10 T 1
1(4)where τ ′ denotes a time constant of learning (τ ′ >> τ ) and
α is a learning coefficient (since performance of learning
is better when α is a decreasing function of | ui | [2], we
substitute α = 2.0 xi yi in the simulations).

While r is moving in the state space, x (Cν ) follows
slightly behind. When r reaches the end T µ ,ν (Cν ) , we keepr = T µ ,ν (Cν ) for a while (1 τ ) so that x (Cν ) may
sufficiently approach r.
In addition, we input r = T µ ,ν for a short time to thenetwork without contextual modification (G= (1,L,1) ) sothat the state x= T µ ,ν may become a point attractor (thisoperation is unnecessary when T µ ,ν naturally becomes anattractor, which is the case with the simulations below).
We apply the above procedure to all µ and ν , andrepeat it over some (about 10 to 30) cycles, gradually
decreasing λ . If x (Cν ) follows r even when λ = 0, thentrajectory attractors have been formed.

2.3. Context-dependent recall
The process of recall is schematically shown in Fig. 2
where the n-dimensional state space of the network isexpressed three dimensionally.
For example, when the network is modified by acontext pattern C 1 , the network state x is projected onto asubspace and it becomes x (C 1 ) = S (C1 ) .Then x (Cν ) moves
along the trajectory attractor to T 1 (C1 ) , when it is expectedthat x = T 1 because T 1 is an attractor (note that generally
the potential ui of desensitized neurons is not zero).Even ifx ≠ T 1 , the exact pattern T 1 is recalled when we release all
neurons from desensitization. In the same way, if the statex = S receives modification by C 2 and C 3 , the network
recalls T 2 and T 3 , respectively, by the state transitions
along the trajectory attractors in respective subspaces.
It should be noted that Fig. 2 is somewhat misleadingin that the projection directions of x by C 1 , C 2 and C 3 areorthogonal, indicating no overlap of desensitized neurons;
but in fact, on average, half of the desensitized neurons arecommon to two different contexts. It should also be notedthat the output y i of desensitized neurons should be at themean level y (estimated to be 0 in this model) tominimize interference between different subspaces.

xc

C1

T 3(C 3)

S1

S(C 3)T3

S(C 2)

4T1
S

C4xb

S

C2 C
14 C

C2

C1

S2

C3

3

C

C3

C4C2

4C2 C

1

T 2(C 2)
T2

S(C 1)

C
T1

1(C )

xa
Fig.2 – Recall process in the state space.

Fig.3 – Association capacity of the model.

3. COMPUTER SIMULATIONS
3.1. Capacity
We carried out computer simulations to examine theperformance of the model. Fig. 3 shows one of the results,
in which the maximum number pq of associations that themodel can correctly form is plotted against the number n ofneurons. In this experiment, cue patterns S 1 ,L, S p andcontext patterns C 1 ,L, C q are of the same number (p=q)
and generated at random; the target pattern T µ ,ν is one ofthe patterns T 1 ,L, T p generated at random, and isdetermined by the remainder for | µ − ν | divided by p asshown in Table 1.
As we see from the graph, the capacity of this modelincreases nearly proportionally with n. In contrast, when
we use the conventional method of contextual
modification (concatenating C to S ), the capacity isalmost zero even with a NNN, as shown by broken lines inFig. 3.
We tested the model on various other conditions
including the case p ≠ q and the capacity was larger than
0.16n in any case, indicating that this model is free fromthe above difficulty in context-dependent association. Thisis thought to be because neither S µ or Cν is directly
associated with T µ ,ν .

S3
C3Fig.4 – Example of the state transition diagram.

3.2. Simulating a finite automaton
In our model, the target pattern may be identical to oneof the cue patterns. Then another target pattern can berecalled from the recalled pattern, and by repeating this,target patterns can be sequentially recalled. In this process,we can control the sequence of patterns is also possible byswitching the current context. In other words, the statetransition of the network is controllable.
For example, any state transition among S 1 ,L, S p ispossible using p context patterns, as shown in Fig. 4 for thecase of p=4. Also Table 2 shows an example of S µ - Cνrelation that enables arbitrary state transitions in the casep=q=10.
In practice, we trained the network to act according toTable 2. Fig. 5 shows the behavior of the model when S µis not externally given except for the initial state, wheresimilarities ∑ xi siµ / n of the network state x to patterns S 1
to S 10 are plotted against time scaled by the time constantτ.
We see in (a) that x shifts from the initial state S 6 to S 5 ,moving via S 7 , S 4 , S 8 , etc. to S 1 while C = C 1 is kept
throughout. On the other hand, x makes different
transitions by switching C in (b). It should be noted thatswitch of C does not have to be made when x = S µ exactly
holds; that is, we may switch C in the middle of statetransitions except when x is midway between S µ , asshown in (b) at t=38τ .
As we see from this example, the model can recall S µ in
any order and preserve S µ for any time. This means thatthe model simulates a finite state automaton (FSA) with pstates and q inputs, as is apparent from Fig. 4. Since thenumber of trajectory attractors or transition paths that themodel can form increases proportionally with n, it follows
that this model possesses the ability to simulate anarbitrary FSA of any size without explosion of neurons.

Table 2. State transition rules used for the experiment.

C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 C 10
S 1 S 1 S 2 S 3 S 4 S 5 S 6 S 7 S 8 S 9 S 10
S 2 S 10 S 2 S 1 S 3 S 4 S 5 S 6 S 7 S 8 S 9S 3 S 9 S 10 S 3 S 1 S 2 S 4 S 5 S 6 S 7 S 8S 4 S 8 S 9 S 10 S 4 S 1 S 2 S 3 S 5 S 6 S 7
S 5 S 7 S 8 S 9 S 10 S 5 S 1 S 2 S 3 S 4 S 6
S 6 S 5 S 7 S 8 S 9 S 10 S 6 S 1 S 2 S 3 S 4S 7 S 4 S 5 S 6 S 8 S 9 S 10 S 7 S 1 S 2 S 3
S 8 S 3 S 4 S 5 S 6 S 7 S 9 S 10 S 8 S 1 S 2S 9 S 2 S 3 S 4 S 5 S 6 S 7 S 8 S 10 S 9 S 1
S 10 S 1 S 6 S 7 S 2 S 8 S 3 S 9 S 4 S 5 S 10S61.0S5S

7S4S8S

3S9S2SAlthough the present model can simulate any FSA, it is
not equivalent to a FSA, since the model is based on fully
distributed representation and has high ability ofgeneralization. For example, the model can receive anunknown combination of cue and context patterns andrecall a proper target pattern by analogy. This implies thatthe model has, in principle, a potential for surpassing
conventional intelligent systems based on symbolic
manipulation.
To examine this possibility, however, further
investigation is required, using not random but structured
patterns.
We also point out that the principle of our model isbiologically plausible. First, trajectory attractors can beformed in a network composed of pairs of excitatory andinhibitory neurons with a sigmoid activation function [3],and it is suggested that recall in the inferior temporal
cortex is performed by a pattern shift in the neuronal
activity [4,5] like the state transition along the trajectory
attractor. Second, there exists good physiological evidence
supporting that selective desensitization of neurons is used
for contextual processing in the brain, but furtherdiscussion on this subject will be given at some other time.

S1

100.8

Similarity

0.6
0.4
0.2
0.0

C1-0.2

5. REFERENCES

-0.4
0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

9S8S7Time(a)
1.0

S6S

5S

7S8

S1

S

10S

0.8

Similarity

0.6
0.4
0.2
0.0

C1-0.2

C8C7C

5-0.4
0.0

5.0

10.0

15.0

20.0

25.0

4. CONCLUDING REMARKS

30.0Time(b)Fig.5 – Behavior of the model.

35.0

40.0

45.0

[1] M. Morita, “Memory and learning of sequential
patterns by nonmonotone neural networks,” NeuralNetworks, vol.9, no.8, pp.1477-1489, 1996.
[2] M. Morita, “Associative memory with nonmonotonedynamics,” Neural Networks, vol.6, no.1, pp.115-126,
1993.
[3] A. Suemitsu, S. Morokami, K. Murata, and M.
Morita, “Computational examination on the dynamics ofrecall activity in the inferior temporal cortex,” Proc.IJCNN
2002, pp.136-141, 2002.
[4] M. Morita, and A. Suemitsu, “Computationalmodeling of pair-association memory in inferior temporalcortex,” Cognitive Brain Research, vol.13, pp.169-178,
2002.
[5] M. Morita, “Computational study on the neuralmechanism of sequential pattern memory,” CognitiveBrain Research, vol.5, pp.137-146, 1996.