1660 • The Journal of Neuroscience, February 18, 2004 • 24(7):1660 –1665

Brief Communication

A Neural Correlate of Reward-Based Behavioral Learning in
Caudate Nucleus: A Functional Magnetic Resonance Imaging
Study of a Stochastic Decision Task
Masahiko Haruno,1 Tomoe Kuroda,1 Kenji Doya,1,2 Keisuke Toyama,3 Minoru Kimura,4 Kazuyuki Samejima,1,2
Hiroshi Imamizu,1 and Mitsuo Kawato1
1Computational Neuroscience Laboratories, Advanced Telecommunications Research Institute, Kyoto 619-0288, Japan, 2Core Research for Evolutional
Science and Technology, Japan Science and Technology Agency, Kyoto 619-0288, Japan, 3Shimadzu Technical Research Laboratory, Kyoto 619-0237, Japan,
and 4Kyoto Prefectural University of Medicine, Kyoto 605-8566, Japan

Humans can acquire appropriate behaviors that maximize rewards on a trial-and-error basis. Recent electrophysiological and imaging studies
have demonstrated that neural activity in the midbrain and ventral striatum encodes the error of reward prediction. However, it is yet to be
examined whether the striatum is the main locus of reward-based behavioral learning. To address this, we conducted functional magnetic
resonance imaging (fMRI) of a stochastic decision task involving monetary rewards, in which subjects had to learn behaviors involving different
task difficulties that were controlled by probability. We performed a correlation analysis of fMRI data by using the explanatory variables derived
from subject behaviors. We found that activity in the caudate nucleus was correlated with short-term reward and, furthermore, paralleled themagnitude of a subject’s behavioral change during learning. In addition, we confirmed that this parallelism between learning and activity in thecaudate nucleus is robustly maintained even when we vary task difficulty by controlling the probability. These findings suggest that the caudate
nucleus is one of the main loci for reward-based behavioral learning.
Key words: basal ganglia; imaging; learning; motivation; reinforcement; reward

Introduction
Guided only by reward and penalty information, animals canadapt their behaviors so that maximal rewards are obtained in thelong run, even in unfamiliar and stochastic environments. This
reward-based behavioral learning problem has been modeled inseveral ways (Sutton and Barto, 1998; Breiter et al., 2001). The
central learning algorithm in the reinforcement learning models
changes behaviors in proportion to reward prediction errors.
Some computational models have proposed that the signal transmission in the striatum is modified by synaptic plasticity for behavioral learning, while being guided by reward prediction error
conveyed by midbrain dopamine neurons (Houk et al., 1995). In
their pioneering work, Hollerman and Schultz (1998) have accumulated compelling evidence that dopamine neurons in themonkey midbrain encode reward prediction errors. Human imaging studies have revealed that the activity in the ventral striatum and putamen (Berns et al., 2001; Breiter et al., 2001; McClureet al., 2003; O’Doherty et al., 2003) is correlated with the reward
prediction errors in classical conditioning tasks.

The dorsal striatum, which receives inputs from the dopamine
neurons and constitutes loop circuits with many cerebral corticalareas, can potentially be the main locus of reinforcement learning inwhich behavioral changes are induced by synaptic plasticity while
controlled by the reward prediction error. However, it has not yet
been demonstrated that the neural activity of the dorsal striatum
parallels the behavioral change or reward prediction errors during
reward-based learning of new behaviors. To investigate the neuralmechanism of reward-based behavioral learning (instrumental conditioning), experimental sessions should contain at least several trials
of behavioral learning for a reliable correlation between behavioral
changes and neural activity. In addition, the correlation would be
more reliable if the rate or difficulty of learning could be quantitatively controlled by task parameters. Here, we developed a new stochastic decision task that satisfies all of these prerequisites and demonstrate that the activity of the caudate nucleus parallels the
behavioral change during learning as well as the amount of shortterm reward by using functional magnetic resonance imaging(fMRI).

Materials and Methods
Received June 16, 2003; revised Dec. 22, 2003; accepted Dec. 23, 2003.
This study was supported by the Telecommunications Advancement Organization of Japan, and by grants to M.K.
from the Human Frontier Science Program. We thank Drs. Shigeru Kitazawa, Manabu Honda, Katsuyuki Sakai, andChris Miall for helpful comments on this manuscript.
Correspondence should be addressed to Dr. Masahiko Haruno, Department of Cognitive Neuroscience, Computational Neuroscience Laboratories, Advanced Telecommunications Research Institute, 2-2-2 Hilaridai Seikacho,
Sorakugun, Kyoto 619-0288, Japan. E-mail: mharuno@atr.co.jp.
DOI:10.1523/JNEUROSCI.3417-03.2004
Copyright © 2004 Society for Neuroscience 0270-6474/04/241660-06$15.00/0

Experimental paradigm. In a Test block of the task (Fig. 1 A), subjects
were required to move a start disk (green; displayed at 0 sec) located in
one of two boxes to the target box where a target disk (red; displayed 0.5
through 1.0 sec) is located by pushing the left or right button after a soundcue. Note that the start and target disk positions can overlap. All of thesubjects pushed the buttons with their right-hand index or middle finger. If
the green disk moved to (or stayed at) the target disk box successfully, thetarget box lighted up, and the subject earned a positive reward (⫹5 yen).

Haruno et al. • fMRI of Reward-Based Behavioral Learning

J. Neurosci., February 18, 2004 • 24(7):1660 –1665 • 1661

stochastic with dominant probabilities of 0.8,
0.675, and 0.5, respectively. Therefore, rules 1,
2, and 3 became more difficult to learn in thisorder. Rule 4, with equal 0.5 probabilities, iscompletely random, so no effective learning
was possible. An essential and attractive property of the current task is that the task difficulty
is controlled by only one parameter (i.e., thedominant probability in a principled way). Because preparatory experiments found that previous exposition to rule 4 sometimes deteriorates subsequent learning in other rules andincreased differences among subjects, rules 1– 4
were used in this fixed order in scanning sessions for all of the subjects without explicit instructions concerning task difficulty. As expected, actual learning became slower in thisorder.
One Test block included 12 trials. In a Control block, the subjects were required to push
the same buttons as in the preceding Test block
after a visual instruction given as the green disk
position. There was no reward or penalty givenin the Control block. The Test and Control
blocks were interleaved. One session for each
Figure 1. Experimental design. A, Visual displays, button pushing, and their timing in the stochastic decision task. B, C, transition rule included 15 Test/Control blocks
Graphical explanation of transition rule (rule 2). The two circles denote the two (left or right) positions of the disk. The arrows and containing 180 Test trials and lasted for 24 min(4 sec ⫻ 12 trials ⫻ 2(Test ⫹ Control) ⫻ 15
attached numbers represent disk movements and their corresponding probabilities when the displayed button is selected.
blocks). The subjects were told that the disk
would move in a stochastic but systematic manOtherwise, the subject suffered the same amount of penalty (⫺5 yen). The
ner according to the pushed button and were encouraged to earn as muchaccumulated reward was displayed above the boxes and was updated after
monetary reward as possible, which was actually given to them in addieach button push. Successive trials were initiated using the final disk position
tion to the basic compensation.
of the previous trial as a start disk position, with a randomly selected target
Explanatory variables. Four explanatory variables were derived fromdisk position.
subject behaviors. The short-term reward (SR) denotes the amount of
The disk movement was stochastically dependent on the selected butmoney (yen) obtained in one term, defined as one-half of a block (sixton according to the transition rules described below (rules 1– 4). For
trials). The accumulated reward (AR) denotes the amount of money
example, in rule 2, the left button push (Fig. 1 B) moved the disk to theaccumulated up to the current term in each transition rule.
right with a probability of 0.8 and to the left with a probability of 0.2,
The learning rate index (LRI) quantifies the change in button pushregardless of whether the disk was initially located at the left or right.
behavior from one term to the next. Because the subject’s behavior in theConversely, the right button push (Fig. 1C) moved the disk to the left
ith term can be described by how often button b was pushed when start s
with a probability of 0.8 and to the right with a probability of 0.2. Thereand target t are provided (s, t, and b take a value of either left or right), we
fore, in rule 2, the optimal behavior for the right target, for example, as inrepresent it as probabilities P i 共b兩s,t兲. Therefore, the differences in button
Figure 1 A, was to push the left button.
push behaviors can be captured by a distance measure between the two
One transition rule consists of two 2 ⫻ 2 matrices corresponding to acorresponding probabilities. The KL distance (Cover and Thomas, 1991)
left or right button push, respectively. Each element of the matrices
defined below is the most standard measure of distance between two
shows the disk movement probability for a given start and target position
probabilities p and q, where the summation is taken for all of the possible
in the following format as displayed in the first matrix in rule 1: (1) firstevents to calculate expectation. The KL distance formally represents how
row, starting from left, (2) second row, starting from right, (3) firstmuch information (bits) is lost when a probabilistic distribution p iscolumn, moving to left, and (4) second column, moving to right.
compressed by another distribution q instead of p. It takes a non-negative
move left move rightvalue and equals 0 only if p and q are identical:
start left
Rule 1: Pleft ⫽
1.000
0.000
start right
p
0.000
1.000
KL 共 p 兩 q 兲 ⫽p log
q0.000 1.000
P right ⫽ 1.000 0.000
The LRI of the ith term representing behavioral change in the adjacent ith
and (i ⫹ 1)th terms is a straightforward application of the KL distance
0.200 0.800
Rule 2: Pleft ⫽between the two sets of probabilities P i⫹1 共b兩s,t兲 and P i 共b兩s,t兲. If a given
0.200 0.800
transition rule is learnable for a subject (rules 1–3), the subject is expected
0.800 0.200
to change behavior a lot at the beginning of learning, but not to change it
P right ⫽ 0.800 0.200
much at the later stage of learning. In this case, LRI is expected to look likean exponentially decaying learning curve, which approximately reflects
0.675 0.325
0.325 0.675
how much synaptic plasticity takes place for behavioral changes.
Rule 3: Pleft ⫽
P
⫽
right
0.325 0.675
0.675 0.325
The learning convergence index (LCI) represents a memory consolidation process for optimal decision-making. Because the progress in
0.500 0.500
0.500 0.500
Rule 4: Pleft ⫽Pright ⫽ 0.500 0.500
learning can be measured by how close the current button push behavior
0.500 0.500
is to the final one, LCI was defined as a similarity index between thecurrent and final button push behaviors. Because a negated value of theRule 1 is deterministic, consisting of probabilities 0 and 1, and always
moves the disk in the same way. Rules 2, 3, and 4 are increasingly more
KL distance between P i 共b兩s,t兲 and P final共b兩s,t兲 represents a similarity of

冋

冋
冋
冋
冋
冋

册
册
册
册
册

册

冋
冋

冘

册
册

Haruno et al. • fMRI of Reward-Based Behavioral Learning

1662 • J. Neurosci., February 18, 2004 • 24(7):1660 –1665

Figure 2. Behavioral results of learning. Shown are data from the least successful subject ( A), the most successful subject ( B), and the average and SDs of all eight subjects ( C). The time coursesof SR, AR, LRI, and LCI are shown from top to bottom.
behaviors in the ith and final term, we defined the LCI of the ith term bynormalizing the negated KL distance between 0 and 1. Therefore, LCI
becomes 0 when the current button push behavior is maximally distantfrom the final one and approaches 1 as the current behavior becomes
more similar to the final one, in other words, as the optimal (except rule4) behavior is acquired.
The correlation coefficient between two explanatory variables for one
subject was highest at 0.68 between LCI and SR and ⬍0.41 for any othercombination. The mean and SD over subjects between LCI and SR were0.61 and 0.07, respectively. The multicolinearity among the explanatory
variables was evaluated by the variance inflation factor (VIF) (Chatterjee
and Price, 1977). The VIF of one variable is 1/(1 ⫺ R 2), where R is amultiple correlation coefficient of the given variable fitted by the remaining variables. Typically, the statistical result is assumed unreliable if
VIF ⬎ 10. In our experiment, the maximum value of VIF for all of thesubjects was 2.21.
MRI acquisition and analysis. Eight healthy adults (24 –33 years of age;
two females, six males; all right-handed) participated in the experiment.
The informed consent of the participants was obtained beforehand, andthe protocol was approved by the ethics committee of Advanced Telecommunications Research Institute. MRI scanning was done with a 1.5
tesla Marconi scanner. For each subject, 480 scans ([Test (8 scans) ⫹Control (8 scans)] ⫻ 15 ⫻ 2) of Bold images (repetition time, 6 sec; echotime, 55 msec; flip angle, 90°; field of view, 192 mm; resolution, 3 ⫻ 3 ⫻3 mm) were acquired for the first two rules. Each fMRI session contained
four preliminary dummy scans corresponding to six Control trials toallow for T1 equilibration effects. After a break, the same procedure wasrepeated for the other two rules. High-resolution structure images werealso acquired for each subject. The data were analyzed by statistical parametric mapping (SPM99) (Friston et al., 1995). Before the statisticalanalysis, we conducted motion correction and nonlinear transformation
into the standard space of the Montreal Neurological Institute coordinates as
implemented in SPM99. These images were smoothed with a 6 mm fullwidth half-maximum isotropic Gaussian kernel. The transformation into
the Talairach coordinates (Talairach and Tournoux, 1998) was done byaffine transformation after the entire analysis was completed.

Regression analysis was conducted on all of the fMRI data of the four
rules. In addition to LRI, LCI, SR, and AR, we added four binary variables(each representing one rule). The regression results were masked with theTest–Control contrasts obtained for all of the rule sessions ( p ⬍ 0.05,
corrected), under the assumption that all of the learning-related brain
activities are included in Test–Control. During Control blocks, LRI, LCI,
and SR were set to 0, and AR was set to the AR of the preceding Test
condition. AR-correlated voxels in Figure 3A–C were for only the rule 4
condition (Elliot et al., 2000), because the monotonic increases in AR for
the other rules may absorb physiological and mechanical noises.

Results
Behavioral Data
Figure 2 shows how the reward acquisition and button push behaviors changed during the Test blocks for the least ( A) and most( B) successful subjects in terms of total monetary reward, as wellas the average of the eight subjects ( C). The SR continued toincrease during the entire session for rules 1–3. The horizontal
lines in the top row of Fig. 2C show theoretical maximum values
for SR that can be expected for optimal button pushing (30, 18,
10.5, and 0 yen for rules 1– 4, respectively). ARs increased almost
monotonically for rules 1–3 and exhibited increasingly smaller
positive slopes for rules 1–3 but did not increase for rule 4. SRs in
the final terms were not significantly different from the above
theoretical maximum values ( p ⬎ 0.4). Correspondingly, ARs in
the final terms (excluding rule 4) were significantly larger thanzero ( p ⬍ 0.0001). These observations demonstrate that learning
certainly took place for rules 1–3.
In the deterministic task (rule 1), the LRI decreased to 0 and
the LCI approached 1 within 10 terms for all of the subjects. In themoderately stochastic task (rules 2 and 3), the decrease in LRI and
the increase in LCI became gradually slower than for rule 1. In the
random task (rule 4), LRI and LCI tended to continuously fluctuate until the very final stage. Figure 2C indicates that the aver-Haruno et al. • fMRI of Reward-Based Behavioral Learning

J. Neurosci., February 18, 2004 • 24(7):1660 –1665 • 1663

Table 1. The stereotactic coordinate and the peak t value within brain regions correlated with LRI, LCL, SR, and AR
Learning rate
Learning convergence
Short-term reward
ROI
1 L SP
2 R SP
3 L IP
4 R IP
5 L PM
6 R PM
7 SMA
8 L PF
9 R PF
10 L OF
11 R OF
12 L CN
13 R CN
14 L GP
15 R GP
16 L CB
17 R CB
18 R ST
19 L OC
20 R OC

Stereotactic
coordinates

t
value

23, ⫺70, 36

6.86

19, 20, 27

6.49⫺30, 43, 7
31, 49, 13⫺40, 43, ⫺6
30, 43, ⫺6⫺0, ⫺3, 0
12, 3, 10⫺14, ⫺9, 2
12, ⫺3, 5

7.21
8.64
6.94
7.26
6.70
5.87
6.08
5.40

34, ⫺73, ⫺2541, ⫺32, ⫺2⫺25, ⫺94, 8
26, ⫺85, ⫺5

7.13
7.47
6.09
6.19

Stereotactic
coordinates⫺11, ⫺73, 38
10, ⫺73, 38⫺32, 3, 53
28, 0, 53⫺6, ⫺9, 47⫺30, ⫺59, ⫺35

t
value

Accumulated reward

Stereotactic
coordinates

t
value

Stereotactic
coordinates

t
value

⫺38, ⫺79, 30
34, ⫺76, 35

5.48
4.34

⫺22, ⫺64, 52
26, ⫺64, 47

13.14
7.41

⫺35, 5, 48
23, 0, 55⫺3, ⫺3, 47⫺37, 32, 12
23, 46, 13

11.69
9.80
9.17
10.58
11.21

23, 43, ⫺1

19.79

52, ⫺53, 8⫺19, ⫺76, 22
15, ⫺70, 23

8.56
12.65
8.82

7.54
7.54
7.54
7.72
5.74

⫺16, ⫺3, ⫺18

3.79⫺27, ⫺64, 2
26, ⫺59, 2

4.60
4.20

5.74

The regions of interest (ROIs) were superior parietal cortex (SP), intraparietal sulcus (IP), dorsolateral premotor cortex (PM), supplementary motor area (SMA), prefrontal cortex (PF), orbitofrontal cortex (OF), caudate nucleus (CN), globus
pallidus (GP), cingulate cortex (CC), cerebeller cortex (CB), amygdala (AM), superior temporal lobule (ST), and occipital cortex (OC). L, Left; R, right.

age LRI across subjects was large at early terms and decreased aslearning progressed, while individual subjects sometimes did notchange their behaviors for the first few trials and their LRI started
from 0 (Fig. 2 A, rule 3). All of the subjects reported in retrospective inquiries that they tried in vain to discover the rules between
the button push and the disk movement even for rule 4, and fourof them reported that they eventually fixed their behavior. Theseobservations indicate that learning difficulty was effectively controlled by the stochastic parameter.
fMRI study
Subtraction analysis
We first determined which brain areas were more strongly activated in the Test condition than in the Control condition for eachtransition rule ( p ⬍ 0.05, corrected for multiple comparisons inrules 2– 4; p ⬍ 0.001, uncorrected in rule 1). Rule 1 induced
activation only in the bilateral intraparietal sulcus, bilateral superior parietal cortices, and left cerebellum. In addition to theseareas, rules 2 and 3 strongly activated the basal ganglia, rightcerebellum, bilateral premotor, bilateral orbitofrontal, bilateralsuperior parietal, bilateral occipital, and right prefrontal cortices,
as well as the supplementary motor area (SMA). In rule 4, in
addition to the above areas, the brain activity extended to the left
prefrontal cortex, right amygdala, and right superior temporal
lobule. Although these neural activities were generally strongest
in rule 4, signal intensity in the caudate nucleus, the globus pallidus, and the orbitofrontal cortex were rather constant, that is, tvalues for rules 2– 4 were 6.05, 5.47, and 5.80 in the left caudate
nucleus, 7.09, 6.93, and 6.10 in the left globus pallidus, and 7.52,
7.24, and 7.34 in the left orbitofrontal cortex, respectively.
Regression analysis
To further investigate the brain structures found in the subtraction, we performed a multivariate regression analysis of fMRI
data with LRI, LCI, SR, and AR. The threshold for LRI, LCI, andAR was p ⬍ 0.05, corrected for Test–Control volume, and that for
SR was p ⬍ 0.001, uncorrected. Table 1 and Figure 3 summarize
the brain areas revealed by the analysis. LRI had significant cor-

relations with activity in the bilateral caudate nucleus, globus
pallidus, orbitofrontal, prefrontal, and occipital cortices, rightparietal, premotor, and temporal cortices, and cerebellum. LCI
exhibited significant correlations with activity in the bilateral
dorsal premotor, parietal, supplementary motor area, and left
cerebellum. SR was correlated with activity in the left caudate
nucleus, bilateral occipital, and parietal cortices. AR had correlations with activity in the bilateral prefrontal, premotor, parietal,
and occipital cortices, supplementary motor area, and right orbitofrontal cortex.
Figure 3A shows that the activity of the caudate nucleus significantly correlated with LRI and SR. The caudate activity was stronger
on the left side probably because subjects used their right hand. In
Figure 3A, it is also observed that activity of the globus pallidus wascorrelated with LRI, and activity of the dorsal premotor cortex and
SMA was correlated with LCI and AR. Importantly, the contiguous
voxels correlated with both LRI and SR in the entire brain werelocated only in the dorsolateral bank of the lateral ventricle. Simultaneous correlation with LRI and SR is computationally essential for
reinforcement learning loci, because synaptic plasticity (LRI related)should be induced by reward prediction errors (SR related). Furthermore, three-dimensional reconstructions of these LRI- and SRcorrelated voxels (Fig. 3D) were in good agreement with the threedimensional shapes of the caudate nucleus head and body, as well as
the globus pallidus. In the caudate nucleus, the correlation with LRI
was stronger in the ventral region and the correlation with SR was
confined to the dorsal part. In addition, the activity of the left lateral
cerebellum was correlated with LCI (Fig. 3B), and that of the orbitofrontal and prefrontal cortices was correlated with LRI and AR (C).
Bold signal trends in the ventral caudate nucleus
The essential role of the ventral caudate nucleus in reward-based
behavioral change was also confirmed by a direct assessment ofneural activity that was measured as a Bold signal increase in each
Test block compared with the subsequent Control (baseline)
block. This analysis was conducted separately for each transition
rule (rules 1– 4). The activity in the ventral part of the left caudate

Haruno et al. • fMRI of Reward-Based Behavioral Learning

1664 • J. Neurosci., February 18, 2004 • 24(7):1660 –1665

caudate nucleus and that in behavioral change (LRI) for the four
different levels of learning difficulty (rules 1– 4). More specifically, the decrease in the Bold signal in the caudate nucleus as wellas the decrease in the magnitude of changes in button-push behaviors between the two neighboring terms were statistically significant for only rules 1–3, in which learning was possible, andtheir negative slopes became smaller as learning became moredifficult (rule 1 ⬎ rule 2 ⬎ rule 3 ⬎ rule 4).

Discussion

Figure 3. Results from the correlation analysis. A, B, C, The numbers attached to the brain
loci are defined in Table 1. The red, green, blue, and yellow regions denote the voxels correlatedwith LRI, LCI, SR, and AR, respectively. The color scaling of the t value is shown in the bar graphs.
A voxel correlated with two variables is represented as a mosaic of the two colors with black
outline. The frontal sections highlight the Talairach coordinates of the left caudate nucleus ( A),left lateral cerebellum ( B), and right orbitofrontal cortex ( C), with thin horizontal and verticallines. D, Three-dimensional distribution of voxels correlated with LRI and SR within a rectangular parallelepiped (⫺18 ⬍ x ⬍ ⫺9; ⫺12 ⬍ y ⬍ 6; ⫺3 ⬍ z ⬍ 24), where x, y, and z
represent the Talairach coordinates. CNH, CNB, and GP denote the head and body of the caudate
nucleus and the globus pallidus, respectively. P denotes the peak voxel correlated with LRI
whose location is (⫺9, ⫺3, 0). The most ventral voxel correlated with LRI was located in theplane of z ⫽ ⫺3. E, Bold signal increase averages and SDs over 11 voxels and eight subjects.

nucleus (11 voxels marked by asterisks in Fig. 3D) around thepeak (marked by P in Fig. 3D) of LRI correlation exhibited atendency to decrease during the tasks with all of the rules except
rule 4. The rate of decrease (the negative slope of the regression
using all of the data from the eight subjects) became smaller withgreater randomness of probability (Fig. 3E) (⫺0.022, ⫺0.014,⫺0.007, and ⫺0.006 for rules 1– 4, respectively). These slopes
were significantly negative (⬍0) for rules 1–3 ( p ⬍ 0.05). Furthermore, the average regressions for individual subjects, considering intersubject variance, confirmed that the slopes of rule 1
and rule 2 were significantly ⬍0 ( p ⬍ 0.005) and also significantly less than the slope of either rule 3 or rule 4 ( p ⬍ 0.005).
Correspondingly, there was also a decrease in LRI (Fig. 2) withsimilar dependence on the randomness of probability (rules
1– 4). The curve fitting by exponential functions was statisticallysignificant (rules 1–3; p ⬍ 0.05) and also logical, because LRI is
positive by definition and approaches 0. The exponential decay
rates for rules 1– 4 were ⫺0.134, ⫺0.111, ⫺0.031, and 0.009,
respectively. Thus, the analysis of Bold signal trends confirmedthat there was parallelism between the decrease in activity of the

The most important finding in our study was that activity in theventral part of the caudate nucleus exhibited a strong correlation
with the magnitude of behavioral change during learning (instrumental conditioning). Bold signal analysis revealed parallelism
between the decrease in caudate nucleus activity and that in LRI
in wide variations in task difficulty (rules 1– 4). Among possible
cognitive elements that can be captured by LRI, our first concern
is the behavioral change in the context of the reinforcement
learning theory (Sutton and Barto, 1998), assuming that behavioral change is guided by reward prediction errors. Experimental
evidence is now accumulating on the roles played by midbrain
dopamine neurons and the ventral striatum in representing reward prediction error. Monkey neurophysiological studies demonstrated that dopamine neurons in the monkey midbrain encode reward prediction errors (Hollerman and Schultz, 1998).
Human imaging studies using reward (classical conditioning)tasks revealed that activity in the ventral striatum (Berns et al.,
2001; Breiter et al., 2001) and putamen (McClure et al., 2003;
O’Doherty et al., 2003) is correlated with the reward prediction
error. In the context of reward-based behavioral learning, computing a subject’s reward prediction error is difficult, and noattempt has ever been made to estimate it. Therefore, we took abehavior-based approach and computed LRI only from subjects’
behaviors without making any additional assumptions. In theframework of the reinforcement learning theory, LRI is expected
to reflect synaptic plasticity responsible for behavioral change,
which is a product of the reward prediction error, inputs forbehavior generation, and an adaptively changing learning coefficient. Therefore, LRI and reward prediction error may be correlated but could be significantly different from each other. Our
results suggest that the caudate nucleus plays an important role inbehavioral learning guided by reward prediction error, which issent from the midbrain, as proposed in several computationalmodels (Houk et al., 1995; Montague et al., 1996).
It is probable that LRI-correlated brain activity also involves
higher cognitive functions, such as inference and hypothesis testing about the task structure, although the stochastic decision task
was originally designed with the simple reinforcement learningtheory as the guiding principle. Because we expect that more
random tasks (rules 3 and 4) are more likely to invoke these
cognitive functions than simple tasks (rules 1 and 2), it is noteworthy that, in the subtraction analysis, the activity in the cerebral cortical areas, including the dorsolateral prefrontal cortex,
tended to increase in accordance with the task difficulty (rules
2– 4). This may suggest that these cortical areas were partly involved in such higher cognitive learning more than the caudate
nucleus. Related to the caudate activity correlation with LRI, Parkinson patients were reported to have difficulty in learning aprobabilistic decision-making task (Knowlton et al., 1996).
It was also remarkable that no overlapping correlation was
found between LCI and LRI, both of which are learning-related
variables. As learning proceeds, LRI decreases, while LCI increases and saturates. Therefore, LRI and LCI may well corre-

Haruno et al. • fMRI of Reward-Based Behavioral Learning

spond to initial nonroutine learning with attention and later routine behavior with less attention, respectively (Fig. 2C, LRI and
LCI). In the reinforcement learning interpretation, we note thatLRI corresponds to synaptic plasticity responsible for behavioral
changes, and LCI corresponds to memory consolidation for optimal behaviors. The LRI-correlated caudate activity is consistent
with reports that the anterior striatum was active and essential
when a monkey learned a new motor sequence (Miyachi et al.,1997, 2002). LCI-correlated activity was found in the bilateral
dorsal premotor and intraparietal cortices, SMA, and left lateral
cerebellum. The dorsal premotor cortex and SMA exhibited additional correlation with AR. This may indicate that these areas
are involved in the intermediate phase of learning by selecting anappropriate action on the basis of the previous experience of
rewards. This view is consistent with the following human imaging studies. In a positron emission tomography study, the precision of the subjects’ recall of a stimulus sequence was correlatedwith the dorsal premotor cortex and supplementary motor area
activity (Honda et al., 1998). An fMRI study also reported activation of the SMA and precuneus in the intermediate and late stagesof motor sequence learning, respectively (Sakai et al., 1998). It isalso interesting that the locus for the LCI-correlated activity inthe left lateral cerebellum was close to that involved in learning
novel tool use (Imamizu et al., 2000) and possibly related to thevisuomotor transformation (internal model) that routinely maps
a visual input to an appropriate selection of behavior. This interpretation is also supported by the subtraction analysis showingthat only the parietal cortex and cerebellum were activated in rule
1, in which the learning converged rapidly.
The results concerning reward variables (SR and AR) were ingood agreement with previous studies. Primarily, the dorsal partof the caudate nucleus (Kawagoe et al., 1998) and orbitofrontal
cortex (Elliot et al., 2000) were correlated with SR and AR, respectively. The small overlapped activation in the caudate nucleus by
SR and LRI can be explained by the difference in temporal characteristics of LRI and SR; LRI represents a low-frequency decaying component, whereas SR represents a high-frequency fluctuating component attributable to stochasticity in the reward
schedule.

References
Berns GS, McClure MS, Pagnori G, Montague PR (2001) Predictability
modulates human brain response to reward. J Neurosci 21:2793–2798.

J. Neurosci., February 18, 2004 • 24(7):1660 –1665 • 1665
Breiter HC, Aharon I, Kahneman D, Dale A, Shizgal P (2001) Functional
imaging of neural responses to expectancy and experience of monetary
gains and losses. Neuron 30:619 – 639.
Chatterjee S, Price B (1977) Regression analysis by example. New York:
Wiley.
Cover TM, Thomas JA (1991) Elements of information theory. New York:
Wiley.
Elliot R, Friston KJ, Dolan RJ (2000) Dissociable neural responses in humanreward systems. J Neurosci 20:6159 – 6165.
Friston KJ, Holmes AP, Worsley K, Poline JB, Frith C, Frackowiak RSJ (1995)
Statistical parametric maps in functional brain imaging: a general linear
approach. Hum Brain Mapp 2:189 –210.
Hollerman JR, Schultz W (1998) Dopamine neurons report an error in thetemporal prediction of reward during learning. Nat Neurosci 4:304 –309.
Honda M, Deiber M, Ibanez V, Pascual-Leone A, Zhuang P, Halet M (1998)
Dynamic cortical involvement in implicit and explicit motor sequencelearning a PET study. Brain 121:2159 –2173.
Houk JC, Adams JL, Barto AG (1995) A model of how the basal ganglia
generate and use neural signals that predict reinforcement. In: Models of
information processing in the basal ganglia (Houk JC, Davis JL Beiser DG,eds), pp 249 –270. Cambridge, MA: MIT Press.
Imamizu H, Miyauchi S, Tamada T, Sasaki Y, Takino R, Putz B, Yoshioka T,
Kawato M (2000) Human cerebellar activity reflecting an acquired internal model of a new tool. Nature 403:192–195.
Kawagoe R, Takikawa Y, Hikosaka O (1998) Expectation of reward modulates cognitive signals in the basal ganglia. Nat Neurosci 1:1411–1416.
Knowlton BJ, Mangels JA, Squire LR (1996) A neostriatal habit learning
system in humans. Science 273:1399 –1402.
McClure SM, Berns GS, Montague PR (2003) Temporal prediction errors ina passive learning task activate human striatum. Neuron 38:339 –346.
Miyachi S, Hikosaka O, Miyashita K, Karadi Z, Rand MK (1997) Differential roles of monkey striatum in learning of sequential hand movement.
Exp Brain Res 115:1–5.
Miyachi S, Hikosaka O, Lu X (2002) Differential activation of monkey striatal neurons in the early and late stages of procedural learning. Exp Brain
Res 146:122–126.
Montague PR, Dayan P, Sejnowski T (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16:1936 –1947.
O’Doherty J, Dayan P, Friston K, Critchley H, Dolan RJ (2003) Temporal
difference models and reward-related learning in the human brain. Neuron 38:329 –337.
Sakai K, Hikosaka O, Miyauchi S, Takino R, Sasaki Y, Putz B (1998) Transition of brain activation from frontal to parietal areas in visuomotor
sequence learning. J Neurosci 18:1827–1840.
Sutton RS, Barto AG (1998) Reinforcement learning. Cambridge, MA: MIT
Press.
Talairach J, Tournoux P (1998) Co-planar stereotaxic atlas of the humanbrain. New York: Thieme.