Neural Network Model of Context-Dependent
Neuronal Activity in Inferotemporal Cortex
Atsuo Suemitsu

Masahiko Morita

Department of Mathematics and Computer Science
Interdisciplinary Faculty of Science and Engineering
Shimane University
Shimane 690-8504, JAPANE-mail: sue@cis.shimane-u.ac.jp

Graduate School of Systems and information Engineering
University of Tsukuba
Ibaraki 305-8573, JAPANE-mail: mor@bcl.esys.tsukuba.ac.jp

Abstract— Neuronal activities related to context-dependent
recall have been found in the monkey inferotemporal cortex.
If we set the same task for an artificial neural network, however,a serious computational difficulty arises. In the present paper,
we overcome this difficulty by implementing a novel methodof contextual modulation, termed selective desensitization, andconstruct a neural network model that performs the samecontext-dependent memory task as that assigned to the monkey.
The model, being consistent with the anatomical structure of
the inferotemporal lobe, as well as with physiological findings,
not only reproduces the empirical data well but also gives aclear account for a phenomenon that had not been explicable todate. This strongly suggests that the brain implements contextdependent recall based on the same principle as adopted in themodel.

I. INTRODUCTION
Naya et al. [1] reported neuronal activities related tocontext-dependent recall in the inferotemporal cortex (IT)of monkeys. They used two kinds of visual memory tasks
and found neurons that changed activity when the task was
switched. This finding is interesting because it reflects therecall process of the target that depends on the context.
If we try to model such context-dependent recall using artificial neural networks, however, we face a serious computationaldifficulty caused by many-to-many correspondence between
the input patterns and the target patterns. Accordingly, no existing model explains the above neuronal activities sufficiently.
It has recently been shown, however, that the above difficulty can be resolved by implementing a novel method ofcontextual modulation, termed selective desensitization [2],[3]. In the present study, we construct a neural network modelof context-dependent recall in IT on the hypothesis that the
selective desensitization method is used in the brain, andcompare simulation results with the empirical data.
II. CONTEXT-DEPENDENT ACTIVITY IN
INFEROTEMPORAL CORTEX
To investigate memory mechanisms in the primate brain,
the delayed matching-to-sample (DMS) and the delayed pairassociation (DPA) tasks have often been used. In these tasks, acue picture is presented for a short period, and the monkey isrequired to judge whether a test picture presented after a delay

period is the target or not. The two tasks are different in thatthe target is the same picture as the cue in the former whereas itis the paired associate of the cue in the latter. Combining these,
Naya et al. [1] developed a task termed the pair-association
with color switch (PACS) task.
In their experiment (Fig. 1), 12 pairs (G1 , C1 ),..., (G12 ,C12 ) of computer-generated pictures were prepared, each paircontaining a green picture and a cyan picture. One of them
was used as a cue, and the target was the same picture in trials
for the DMS task or its paired associate in trials for the PACS
task. In a DMS trial, delay period 1 (d1) after cue presentation
was 5 s, during which a stimulus (color signal) of the samecolor as the cue was shown on a display. After d1, the color
signal changed to gray and delay period 3 (d3) of 1 s began. A
test picture was presented at the end of d3. In a PACS trial, d1
lasted only 2 s and delay period 2 (d2) started at the switchingof the color signal to a different (the same as the target) color;d2 lasted 3 s and the color signal became gray in d3.
Naya et al. recorded neuronal activities in the anterior IT
while two well-trained monkeys were performing these tasks.
Fig. 2 shows the results of analyzing 15 neurons which exhibit
selective delay responses, where firing rates of each neuron
during d1 and d3 of the DMS task and the PACS task areplotted; (a) and (b) are cases where the best picture, in which
the neuron exhibits the highest cue response, is presented asa cue and where the paired associate of the best picture ispresented, respectively. There is little difference in d1 firing
rates between the two tasks, but there is a significant differencein d3. This means that subsequent neuronal activity changes
significantly after the color switch.
III. SELECTIVE DESENSITIZATION METHOD
In the previous study [4], we constructed a computationalmodel for the DPA task (referred to as the DPA model). In thismodel, trajectory attractors connecting from the cue-coding totarget-coding states are formed by learning, and when a cue
pattern is given, the target pattern is recalled by continuous
state transitions along the corresponding trajectory attractor.
Although the DPA model agrees well with some empiricalresults, this model cannot perform the above DMS/PACS task
because it always recalls the same target pattern from the same

warning (w)
(1 s)

cue
(0.5 s)delay1 (d1)
(2 s)delay2 (d2)
(3 s)delay3 (d3)
(1 s)choice(1.2 s)match

DMS
non-match
gray

green

cyan

gray
match

PACS
non-match

Fig. 1. Illustration of DMS and PACS tasks. Warning, gray square (1 s in both tasks); cue, 1 of 24 pictures presented as a cue stimulus (0.5 s); delay period
1, the same color as the cue stimulus (5 s in the DMS task; 2 s in the PACS task); delay period 2, the same color as the paired associate of the cue stimulus(3 s in the PACS task); delay period 3, gray square (1 s in both tasks); choice, a choice of two stimuli (1.2 s in both tasks), the same picture as the cue(matching) and the others (nonmatching) in the DMS task, or the paired associate of the cue (matching) and the others (nonmatching) in the PACS task

10

Spikes, no./s

Spikes, no./s

10

5

5

0

w

d1
PACS

d3

d1
DMS

d3

(a)

0
w

d1
DMS

d3

d1
PACS

d3

(b)

Fig. 2. Average firing rates of each neuron during delay periods of the DMS and PACS tasks. (a) Trials in which the best pictures are presented as a cue.
(b) Trials in which the best picture’s associates are presented as a cue. (Reproduced from [1])cue pattern. Accordingly, we must modify this model so thatit can recall different targets from the same cue depending on
the context, namely, whether or not the color signal changes
during the delay period.
The simplest method for associating plural targets with asingle cue is to concatenate the cue pattern and the context pattern to be the input pattern. Although this method
has generally been used, it causes a computational difficultywhen the number of the targets corresponding to one cue isincreased. Let us briefly explain the issue taking the case of

the DMS/PACS task as an example (refer to [3] for detailed
discussions).
Assume the cue picture to be G1 and thus the initial color
signal is “green”; then the target is G1 if the color signal
remains “green” and C1 if the color signal switches to “cyan”.
It seems easy to recall the correct target by associating G1 with“green” and C1 with “cyan”. However, “green” and “cyan”
should be associated also with other pictures G2 ,...,G12 andC2 ,...,C12 , respectively. Consequently, the direct connections
between the units representing the color signal and thosexc

desensitization

T 3(C 3)S(C 3)T3
S(C 2)

T1rrelease ofdesensitization

N1
x

trajectory
attractor

s

S

c

xb

N2

T 2(C 2)
T2

S(C 1)T 1(C 1)xa
Fig. 3. Schematic process of context-dependent recall. The n-dimensional
state space of the network is expressed three dimensionally.

representing the target are diluted because of the averaging
effect. This means that the color signal does not aid the
network to recall a particular target pattern.
This difficulty, called the problem of averaging caused by
one-to-many correspondence, is deep-rooted and is unavoidable for conventional models unless they involve local representations, or hidden units each of which codes a particular
combination of cue and context patterns. Such local representations, however, do not accord with physiological findings
about IT. Thus we use the selective desensitization method,which was devised to resolve the problem of averaging without
using local representations [2], [3].
In this method, some units are desensitized depending on
the context, and the desensitization pattern or the configurationof desensitized units corresponds to the context pattern. Here,“desensitize” means to fix the output of units at a neutral value
regardless of the input, which is implemented by setting theoutput gain to zero. In mathematical terms, assuming that theoutput xi of the i-th cell (i = 1, . . . , n) is given by
xi = gi · (f (ui ) − x̄i ) + x̄i ,(1)where ui denotes the potential of the unit, f (u) is theactivation function and x̄i is the mean level of xi , the output
gain gi = 0 for desensitized units and gi = 1 otherwise;
accordingly, desensitized units emit a constant value of x̄i ,whereas valid (not desensitized) units emit f (ui ). For simplicity, we regard the gain vector (g1 , . . . , gn ) in the samelight as the context pattern.
Although desensitization (setting xi = x̄i ) is a different
concept from inactivation (decreasing xi to near 0), the twoare approximately equivalent if x̄i  0, namely, the mean
output level is low. Since the above IT neurons responded
to only a few pictures, and also for simplicity of the model,
we use the mechanism of sending strong inhibitory signal tosome units to implement desensitization. It should be notedthat there exist other possible biological mechanisms such aspresynaptic inhibition and neuromodulators.

xFig. 4. Block diagram of the model. Network N1 receives the output r ofN2 and stores memories, and its units are desensitized depending on the color
signal c. N2 receives the output x of N1 and the external input s to generate
the learning signal for N1 .

Fig. 3 schematically shows how different targets can berecalled depending on the context, where the n-dimensional
state space of a fully recurrent neural network is expressed
three dimensionally. When the network at state S is modified
by a context pattern C 1 = (c11 , . . . , c1n ), that is, unit i isdesensitized when c1i = 1, the network state x is projected
onto a subspace composed of the valid units and changes
into S(C 1 ). Then x moves from S(C 1 ) to T 1 (C 1 ) along the
trajectory attractor formed in the subspace. By removing thecontextual modification or releasing desensitization thereafter,x changes into T 1 when T 1 is an attractor. In the same way,
when the network is modified by C 2 and C 3 , x reaches T 2
and T 3 , respectively, via state transitions along the trajectory
attractors in respective subspaces.
IV. THE MODEL
We constructed the model for the DMS/PACS task by applying the selective desensitization method to the DPA model,
and thus it has much in common with the DPA model. We
here describe the model briefly, concentrating on the different
parts (for a detailed explanation for the common parts, refer
to [4]).
The model is composed of two interconnected neural networks: association network N1 in which memories are stored
and trainer network N2 which generates the learning signal
required for memory formation (Fig. 4). The output patternx = (x1 , . . . , xn ) of N1 is sent to N2 and also fed recurrently
into N1 , and the output pattern r = (r1 , . . . , rn ) of N2 is fed
back to N1 as the learning signal. The external input patterns = (s1 , . . . , sm ) is fed into N2 ; although N2 should receives via N1 , the direct input path to N1 is omitted for simplicity.
In addition, the color signal c = (c1 , . . . , cn ) is given to N1
to desensitize some units.
The internal structures of networks N1 and N2 are shownin Fig. 5 and Fig. 6, respectively, and their dynamics are

x1

xi

xn

xc

desensitization
release ofdesensitizationr 2 (c 2)
C1+

Ci+

−

C1

Cn+

Ci−

ri
ci

xa

cn

Fig. 5. Structure of network N1 . A pair of excitatory cell Ci+ and inhibitory
cell Ci− composes a unit, and all units are mutually connected.

r

1

x

ri x i

1

C1

s

Cn

si

(2)

j=1

τ

dui
dt

= −ui +xi

= f (ui ) ,mn= −vi +
pij sj +qij xj

dvi
dt

j=1

+
wij
xj − wi∗ yi + λri ,

j=1

−ρ



(3)
(4)

j=1

rj + σri + h,

(5)

j=i

ri
f (u)

= f (vi ) ,1=,
1 + e−10u

r 1 (c 1 )Fig. 7. Attractors formed in the state space of N1 . Patterns r1 and r2 arepoint attractors, and paths from r1 (c2 ) to r2 (c2 ) and from r2 (c1 ) to r1 (c1 )are trajectory attractors. N1 can then undergo state transitions from r1 to r2
and vice versa by receiving color signal c2 and c1 , respectively.

+
dwij
+
= −wij+ αri xj ,
(8)
dt
−
dwij
−
= −wijτ− β1 ri xj + β2 xi xj + γ,
(9)dt
while each cell of the model is running according to Eqs. (2–7) [5]. Here, τ  is a time constant (τ   τ ), α, β1 , andβ2 are learning coefficients (β1 < β2 ), and γ is a positive
constant representing the lateral inhibition between units.
Coefficient α may be a positive constant, but because thelearning performance is better when α is a decreasing functionof xi , we adopt α (κ − xi ) (xi < κ)
(10)
α=
0
(xi ≥ κ),τsm

described mathematically by


n

−
yi = f wij
xj + ζci − θ  ,nxb
r 2 (c 1)

unit, respectively, wi∗ represents the efficiency of the inhibitory+
synapse from C−
i to Ci , and pij and qij are random weights
of input synapses to the i-th cell Ci of N2 . ui and vi are
the potentials of C+
i and Ci , respectively, λ denotes the input
intensity of r, h is an offset, and θ, ζ, τ , ρ and σ are positive
constants.
Learning of this model is performed by modifying thesynaptic weights of valid N1 units according to

Fig. 6. Structure of network N2 . For this network, the present model is thesame as the DPA model [4].

τr1

rn x n

Ci

1

2

(c )

rn

c1

trajectory
attractor

Cn−

r1

r1

r2

(6)
(7)

+
−
where wij
and wij
are the synaptic weights from the j-th−
unit to excitatory and inhibitory cells C+
i and Ci of the i-thwhere κ ≡ β1 /β2 and α is a positive constant.
By repeating this synaptic modification several times, apattern ‘close’ (in the sense of the angle between pattern
vectors) to learning signal r becomes a point attractor in
the state space when r stands still. If r varies continuously,
however, r leads the move of the network state x, producing agentle flow in the direction of motion of r; that is, a trajectory
attractor is formed along the track of r [6].
The purpose of learning is to form attractors in the state
space of N1 so that N1 can make bidirectional state transitions
between two patterns, as schematically shown in Fig. 7. In thisfigure, r1 and r2 are point attractors coding paired stimulus

s

1

c

ss

1

1 1

c

1

ss

3 2

c

ss

2

c

2 2

ss c

2

4

1

1

c

2

ss c
2

1

1

c

2

ss c
4

2

2

c

1

ss c
1

2

2

c

1

s

3

x

1

x

12

0

5

10

15

20

25

30

35

40

45

50

55

60

Time [sec]Fig. 8. Behavior of the model after learning. Response of individual units in N1 to various cue and test patterns are vertically arranged. Thin and thickvertical lines represent the borders between periods and between trials, respectively. The first four trials are for the DMS task and the last four trials are for
the PACS task.

patterns s1 and s2 , respectively, and line arrows from r1 (c2 ) tor2 (c2 ) and from r2 (c1 ) to r1 (c1 ) represent trajectory attractors,
where rµ (cν ) denotes rµ modified by color signal cν .
After learning, the model performs the DMS/PACS task
as follows. First, when a cue pattern, for example, s1 , isgiven with color signal c1 in the cue period, the network
state x changes into r1 (c1 ). This state is maintained after
the cue period, since it is a point attractor in the subspacecorresponding to the modification by c1 and c1 is continuously
fed during d1.
In the DMS trial in which no color switch occurs, d3 starts
after d1, when the color signal is removed or c is set toO ≡ (0, ..., 0); it should be noted that in the actual task, themonkey no longer needs the color signal after the onset of
d3. Consequently, x moves to the point attractor r1 and stays
there. If we then input s1 again as a test pattern, it is thoughtthat many units show a stronger response than when the test
pattern is sµ (µ = 1).
In contrast, in the PACS task, the color signal is switchedfrom c1 to c2 at the onset of d2, and x changes immediately
into a state near r1 (c2 ); then x moves successively towardr2 (c2 ) along the trajectory attractor. When the color signal isremoved in d3, x is attracted by r2 , and thus strong responses
to the test input of s2 are expected.
V. COMPUTER SIMULATION
We carried out computer simulation using the model witha size of n = 1000. First, we randomly generated 12 pairsof patterns that were 1000-dimensional (m = 1000) binary
vectors. We then input these pairs of patterns sequentially,
applying the above learning procedure. Training was repeated
20 times for each pair. The parameters wereτ = 100ms, τ  = 50000τ, θ = 2.1, wi∗ = 10,
ζ = 0.12, λ = 0.3, ρ = 0.0105, σ = 0.6α = 50, β1 = 25, β2 = 50, γ = 0.05.

Synaptic weights pij and qij were set to random numberswith mean of 0.005 and 0.001 and variance of 0.05 and 0.01,
respectively.
After training, we tested the model in performing each task,
varying the combination of cue and test patterns. The responseof the model in 8 trials is shown in Fig. 8, where the timecourse of the outputs of 12 units is plotted. These units wereselected from the units encoding s1 or s2 or both such thatvarious kinds of activity are displayed. The first half andsecond half of the trials are for the DMS and PACS tasks,
respectively; the odd-numbered trials are matching trials inwhich the test and target patterns are identical and the others
are nonmatching trials.
We can see that units showing high activity in d3 further
increase their output in matching trials, whereas they are
depressed in nonmatching trials. We confirmed that we candistinguish between the target and nontarget patterns according
to the output distribution in the test period. This indicates thatthe model can perform the DMS/PACS task well.
To compare the results with physiological data, we plotted
the graphs shown in Fig. 9, where outputs of 20 units areplotted in the same manner as in Fig. 1. These units wererandomly selected from among the units encoding s1 or s2 or
both. We see that the graphs in Fig. 9 are very similar to thosein Fig. 1 if we compensate the difference in the activity level
during the warning period.
VI. DISCUSSION AND CONCLUSION
The most notable point in the above results is that in d3 of
the DMS task, some units rapidly increase their output. Thisis because these units are desensitized during d1 and released
from desensitization in d3.
Such activity enhancement was actually observed in the
IT neurons, as shown in Fig. 1a, in which activity increaseof three to four times is seen for 4 of 15 neurons. Thisphenomenon is unprecedented and has not been explainedso far, since in the DMS task, the monkey is thought to1.0

OutputOutput

1.0

0.5

0.0

0.0
w

d1
PACS

d3

d1

d3
DMS

(a)
Fig. 9.

0.5

w

d1
DMS

d3

d1
PACS

d3

(b)

Responses of each unit in both tasks when the cue is (a) the pattern encoded by the unit and (b) its paired associate.

remember the same picture throughout the delay period. The
agreement between the simulation results and the empirical
data, therefore, not only provides a reasonable explanation
for this phenomenon, but also suggests strongly that selective
desensitization actually occurs in IT.
In conclusion, we have constructed a neural network model
for context-dependent association and demonstrated that themodel can reproduce the distinctive activities, including an
inexplicable one, of the monkey IT neurons. This supports
our hypothesis that the selective desensitization method isused for context-dependent information processing in IT, andpresumably in other areas of the brain as well.
ACKNOWLEDGMENT
This work was supported by Grant-in-Aid for Scientific
Research (B) from JSPS (#15300068) and Grant-in-Aid for
Scientific Research on Priority Areas “Informatics” (Area#006) from MEXT (#16016207).
R EFERENCES[1] Y. Naya, K. Sakai, and Y. Miyashita, “Activity of primate inferotemporal
neurons related to a sought target in pair-association task,” Proc. Natl.
Acad. Sci. USA, vol. 93, pp. 2664–2669, 1996.
[2] M. Morita, K. Murata, and S. Morokami, “Context-dependent sequential
recall by a trajectory attractor network with selective desensitization,”
Proc. ICNNAI2003, pp. 235–238, 2003.
[3] M. Morita, K. Matsuzawa, and S. Morokami, “A model of contextdependent association using selective desensitization of nonmonotonic
neural elements,” Systems and Computers in Japan, vol. 36, no. 7, pp. 73–
83, 2005.
[4] M. Morita and A. Suemitsu, “Computational modeling of pair-association
memory in inferior temporal cortex,” Cognitive Brain Research, vol. 13,
pp. 169–178, 2002.
[5] M. Morita, “Computational study on the neural mechanism of sequential
pattern memory,” Cognitive Brain Research, vol. 5, pp. 137–146, 1996.
[6] M. Morita, “Memory and learning of sequential patterns by nonmonotone
neural networks,” Neural Networks, vol. 9, pp. 1477–1489, 1996.