Reprinted from American Laboratory April 2002

Functional imaging of the human temporal
cortex during auditory sentence processing
Kuniyoshi L. Sakai, Fumitaka Homae, Ruyichiro Hashimoto, and Kei Suzuki
ensory information processing in the cerebral cortex has been characterized as involving multiple areas that are hierarchically organized and functionally specialized.1 It is known thatspeech recognition critically depends on specialized
cortical regions, including auditory areas as well as
language areas.2 While cytoarchitectonic studies haveidentified multiple auditory areas in humans,3–6 therole of these multiple areas in auditory analysis per se
has yet to be elucidated. Using recently developed
imaging techniques, differential cortical responses tospeech stimuli could reveal functional differentiationin the auditory cortex. One notable drawback is thatmost functional imaging studies have tested languagetasks at the word level, such as lexical decision andword-generation tasks, thereby neglecting the syntactic aspects of the language faculty that may be primarily manifested at the sentence level.
This paper presents the results of recent functional
imaging studies intended to identify cortical areas involved in auditory sentence processing. Functional
magnetic resonance imaging (fMRI) and optical topography (OT) were used. OT is a newly developed technique that measures temporal changes in hemoglobin
oxygenation simultaneously at multiple regions.7–9
While near-infrared spectroscopy (NIRS) measuresspectroscopic reflection and scattering at a single
region with a light emitter and a detector,10,11 OT canacquire a topographical image of cortical activity. The
current direction of research in cognitive neurosciencehas established the usefulness of these neuroimaging
techniques for understanding essential aspects of human cognition, including speech recognition.

S

Experiment
Subjects
Seven native Japanese-speaking males (aged 20–32)
participated in the study. They demonstrated righthandedness (laterality quotients 81–100) according to
the Edinburgh inventory.12 The subjects’ consent was
obtained according to the Declaration of Helsinki. Approval for the human experiments was obtained fromthe institutional review board of the University ofTokyo, Graduate School of Arts and Sciences (Tokyo,
Japan).

Auditory stimuli
Speech sounds were sentences taken from fairy

tales. These were digitized (16 bit, 11,025 Hz) using
Oshaberi-mate speech-synthesis software (Fujitsu,
Tokyo, Japan), which converts Japanese written text
into sound waveforms. Sine wave tone and white
noise used in the control task were synthesized by
Sound Forge sound-editing software (Sonic Foundr y,Inc., Madison, WI). In fMRI experiments, auditory
stimuli were presented to the subject’s ears every 1
sec through separate tubes, and the scanning sounds
were confined within the interstimulus interval of
300–600 msec. In OT experiments, stimuli were presented with a stereophonic headphone.

Task paradigm
The behavioral tasks used in the fMRI study were acontrol (CON) condition in which nonspeech sounds
were presented to both ears, and two listening conditions for speech sounds: a diotic (binaural) listening(DIO) condition and a dichotic listening (DIC) condition. All stimuli consisted of targets to be responded toand nontargets to be neglected by the subjects. Under
the DIO condition, identical stimuli were presented toboth ears simultaneously. Target phrases were presented in the order of an original story with interventions of nontargets, which were made by randomizing
the order of syllables of the corresponding target.
These jumbled stimuli conformed to the rules ofJapanese phonotactics but had no meaning. The subject was asked to respond to a target stimulus by pressing a bulb attached to a pneumatic switch. Under the
DIC condition, either a pair of target and nontarget orthat of two different nontargets was simultaneously
presented to different ears. The subject had to choose
one side with a target and pay attention to that target
selectively. Therefore, in addition to the recognition of
speech sounds required by the DIO condition, the DIC
condition also demanded selective attention to target
stimuli in either ear.
In the OT study, a dichotic listening paradigm wasused for all tasks. Target stimuli and nontarget stimuli
were simultaneously presented to different ears every 2sec, and a target was alternatively presented to either
the left or right ear at random intervals. Subjects wereasked to track targets and to press a button when a target was shifted to the other side. In a control task, atone and white noise were presented as targets andnontargets, respectively. In order to confirm that subjects performed tasks by recognizing targets, a tone ofdifferent pitch was presented as a nontarget at a lowerinfrared laser diodes with two wavelengths (782–793 nm and 823–832nm) were used as the light sourc e s(maximum intensity 2 mW/mm 2, intensity modulation 1–10 kHz). The
reflected lights were detected withavalanche photodiodes located 30
mm from an incident position. Using
lock-in amplifiers, the detected signalb
was separated into individual lights o u rces with each wavelength. 7
Twenty-two points in e ach hemisphere we re simultaneously measured at minimum spatial intervals of
21 mm, and each point was samplede v e ry 500 msec. The measured reFigure 1 Representative activity of multiple auditory areas in one subject. a) Series gion in each hemisphere centered onof t-maps of (DIO + DIC) vs CON on horizontal brain slices. b) Series of
the Sylvian fissure and covered an
t-maps of DIC vs DIO. Color bars denote t-values for each comparison. The left
area of 6 × 12 cm2. A correlation coefside of the brain is shown on the left. The centers of the slices are z = 4, 12, 20,
and 28 from the left panel to the right. Multiple activated regions were found in the ficient (r) of hemoglobin time points
auditory and language-related areas, mostly in the left hemisphere of this subject. with a boxcar waveform was calculated for each measurement point,
rate. These probe stimuli prevented subjects from perand an r-map was created from the r-values. As aforming the tasks by tracking nontargets only. Two
threshold for the r-values, 0.73 or –0.73 was chosen
speech-recognition tasks were used: 1) a repeat task, infor statistical significance (P = 0.05, corrected).
which the targets consisted of one repeated sentence;
Results and discussion
and 2) a story task, in which the targets were successive
different sentences of a continuous story. A sentence
Functional differentiation within the auditory
different from the target for the repeat task and contexcortex and language areas
tually anomalous phrases for the story task were used as
Multiple regions were identified that were actiprobe stimuli. Therefore, these tasks could not be comvated under the DIO and DIC conditions (Figure 1).14
pleted appropriately by identifying speech sounds
In the t-map of (DIO + DIC) vs CON, activation was
without paying attention to their meanings.
observed in multiple auditory and language areas: theprimary auditory area (A1), secondary auditory area
fMRI data acquisition and analysis(A2, both lateral and medial portions), planum temporale (PT), superior temporal gyrus (ST), suprafMRI data were acquired using an MRH-1500 1.5
marginal gyrus (SMG), and inferior frontal gyrus (IF).
Tesla MRI system (Hitachi Medical Corp., To k y o ,In the t-map of DIC vs DIO, on the other hand, reJapan). Horizontal slices were scanned over with a gragions in A2 (medial portion), PT, ST, and IF were idendient echo echo-planar imaging (EPI) sequence (repetitified that responded more prominently to the DIC
tion time 2 sec, field of view 384 × 384 mm2, resolution
condition(Figure 1b). In A2, the medial portion,
3 × 3 × 8 mm3). Analyses of fMRI time-series data werecalled A2m (medial A2), clearly showed DIC selectivfirst done on a single-subject basis using in-house softity, though this selectivity was not apparent in theware.13 Time-series data of each voxel were converted
lateral portion, A2l (lateral A2). Anterior ST (STa) wasto percent signal changes from the initial CON block,also identified as an activated region just posterior to
and were then averaged for multiple sessions after corthe Heschl’s gyrus and on the lateral surface of the anrection for head movements between scans. The actiterior superior temporal gyrus.
vation under the DIO and DIC conditions was estiFurther, it was confirmed that the spatial relationshipmated using the following two types of t-tests for eachbetween A2m, A21, and STa was consistent among all
voxel: (DIO + DIC) vs CON (t > 2.6), combining DIO
subjects. These results clearly demonstrate that the DIO
and DIC before comparison with CON, and DIC vsand DIC conditions modulate cortical responses differDIO (t > 2.3) as direct comparison between two experientially among multiple auditory areas. Some areas
mental conditions. The significance level for each actishowed greater responses to the DIC condition than tovated region after Bonferroni correction was P < 0.02
the DIO condition, while other areas did not show suchand P < 0.04, respectively.
a difference. These contrasting response patterns suggest
functional differentiation among multiple auditory arOT data acquisition and analysis
eas. It is notable that both cortical language areas, WerOT data were acquired using two OT systems withn i c k e ’s area (PT and ST) and Broca’s area (IF), arethe same calibration (ETG-100 and ETG-A1, Hitachiparceled into two types of regions with different reMedical Corp. ), one for each hemisphere. Nearsponse patterns. All the regions identified in the left
a

a

b

for speech recognition experiments.15
There are several advantages to using
OT over other functional mapping
techniques. First, it is possible to independently measure the temporalchanges in oxyhemoglobin concentration (Coxy) and deoxyhemoglobin concentration (Cdeoxy). Second, there is no
scanning noise to interfere with the experimental auditory stimuli. Third, its
signal-to-noise ratio is as high as fMRI,
allowing
for the observation of corticalFigure 2 Differential responses under the DIO and DIC conditions in multiple au activity with a small number of trials.
ditory areas. Histogram comparing the percent signal changes (mean ± SEM(standard error of the mean) of subjects) for each auditory condition vs CON is
One major disadvantage of OT is thatshown in each region. Filled and open bars denote signal changes under DIO
its measurement is restricted to the corand DIC, respectively. a) Regions in the left hemisphere. b) Regions in the righttical surface. Nevertheless, OT has thehemisphere. Signal changes are shown for A1, A2m, A21, and STa, which have potential to introduce a new dimension
close anatomical relationships. They show progressive signal increases under the to the mapping of human cognitive
DIO and DIC conditions. Single (*) and double (**) asterisks denote statistical
functions.
significance at P < 0.05 and P < 0.005, respectively.
Left-dominant activation in the suhemisphere were also found in the right hemisphere atperior temporal cortex (the superior and middle temsimilar coordinates of the brain.
poral gyri) was observed preferentially during the story
It was found that A1, A2m, A2l, and STa showed atask over the repeat task when compared with the conprogressive increase in responses to both the DIO andtrol task (Figure 3). Wider regions were clearly more acDIC conditions (Figure 2), suggesting that these regions
tivated during the story task than during the repeat
constitute a functional pathway on the order of A1,
task in both hemispheres. This finding is consistentA2m, A2l, and STa. In other words, these regions havewith the results of a previous positron emission tomogprogressively greater selectivity for the speech stimuli
raphy (PET) study that showed activation of the left supresented. If auditory areas at the early stage process
perior temporal cortex when subjects listened to conboth the story phrases and the nonsense words in thetinuous speech in their native language.16 The direct
course of figure-background segregation, the DIC concomparison between the hemodynamics in the story
dition would elicit larger responses than the DIO contask and that in the repeat task showed focal activation
dition. In contrast, at the later stage, after filtering out
in the left superior temporal cortex (Figure 4). Both an
the nonsense words as unattended stimuli, responses
increase in Coxy and a decrease in Cdeoxy were signifito the DIO and DIC conditions would become compacantly larger in the middle temporal gyrus than in therable.
superior temporal gyrus. The temporal changes in theThis “filter” process agrees with the decrease in DIC
left superior temporal cortex are shown in Figure 5.
selectivity along the pathway of A2m, A2l, and STa. The
With a delay of 6 sec, an increase in Coxy was synchroauditory pathway proposed here corresponds to previnized with each onset of the story task and, after reachous anatomical findings. Based on the gradient ining a plateau, Coxy returned to the baseline level at theacetylcholinesterase staining and cytochrome oxidase
end of the task. Although a decrease in Cdeoxy was also
activity, a hierarchical order of A1, PA (posterior area),synchronized with each period of the story task, Cdeoxy
LA (lateral area), and STA (superior temporal area) has
did not exactly mirror the temporal dynamics of Coxy.
been proposed.6 Specifically, the spatial configuration
Similar, but smaller, changes in both Coxy and Cdeoxyof A1, PA, LA, and STA corresponds to that of A1, A2m,
were observed in the repeat run. These results suggest
A2l, and STa. It is notable that the anatomical study
that the hemodynamics in the midlateral part of theand the functional study match with respect to theleft temporal cortex reflect cognitive factors involved
anatomical location of these areas and the order of thein the processing of sentences. A critical difference beproposed hierarchical levels. These findings further
tween the story and repeat tasks would be the load ofsuggest that A2 and cortical language areas, such as
speech stimuli to be processed. Recognition of succesWernicke’s and Broca’s areas, play a pivotal role insive different sentences of a story demands more audicombining differential mechanisms of speech recognitory, memory, and language information processing
tion and in processing language beyond the primarythan the recognition of repeated sentences. The temauditory area.
poral cortex activation reported here is also consistent
with the role of the primate temporal association area
in memory storage and memory retrieval.17,18
Temporal cortex activation during speech
Based on converging evidence from aphasic andrecognition
imaging studies, a modular specialization of the cortiTo address the issue of sentence processing in thecal language areas that are critically involved in sencerebral cortex further, the OT technique was adopted
tence processing has been proposed. 1 9 This rather

a

a

b

Figure 3 Superior temporal cortex activation during speech
recognition. a) Story task vs control. The left brain is shownon the left. b) Repeat task vs control. OT images in r-maps
averaged among all subjects were superimposed on lateralimages of a brain. The color bar at right denotes correlation
coefficients (r) of Coxy time points with a boxcar waveform(each period 36 sec, delay 6 sec). Note the prominent acti vation in the midlateral part of the bilateral temporal cortex.

Figure 4 Focal activation in the left superior temporal cor tex during sentence processing. OT images in r-maps for a
direct comparison between story and repeat tasks. Left:r-maps of Coxy temporal changes. Right: r-maps of Cdeoxy
temporal changes. Activation is localized in the middle
temporal gyrus in both comparisons.

broad delineation at the current stage of imaging
studies reflects the difficulty of dissociating linguistic
components from confounding cognitive factors. The
language system does not stand alone but interacts
with other systems of perception, memory, and consciousness, as well as with the speech output system.
Future work combining electrophysiological eventrelated brain potential and magnetoencephalography, hemodynamic (PET, fMRI, OT), and transcranial magnetic stimulation (TMS) methods will allow
the further parceling of language processing in the
cerebral cortex and will enable us to address questions
about module-specific brain areas and how they actually perform linguistic computations of sentences.

References
1. Felleman DJ, Van Essen DC. Distributed hierarc h i c a lprocessing in the primate cerebral cortex. Cereb Cortex
1991; 1(1):1–47.
2. Geschwind N. Specializations of the human brain. Sci
Amer 1979; 241(3):158–68.

b

Figure 5 Hemodynamics in speech-recognition tasks rela tive to the control task. a) Story and control tasks. b) Repeat
and control tasks. There were three periods of either thestory task (S) or the repeat task (R) in each run. Red lines
show the mostly positive temporal changes of Coxy, whereas
the blue lines reveal the mostly negative temporal changesof Cdeoxy. These temporal changes were calculated from av eraged data among subjects.

3. Brodmann K. Comparative localization of the cerebral
cortex in its principles represented on the basis of thecell structure. Leipzeig, Germany: Verlag von Johann
ambrosius Barth, 1909.
4. Von Economo C, Horn L. On the convolution, mass,and coritical architecure of the superior temporal area:
its individual and side difference. Z Neurol Psychiat
1930; 130:678–757.
5. Galaburda AM, Sanides F. Cytoarchitectonic organization of the human auditory cortex. J Comp Neurol
1980; 190:597–610.
6. Rivier F, Clarke S. Cytochrome oxidase, acetylcholinesterase, and NADPH-diaphorase staining in human supratemporal and insular cortex: evidence for multiple auditory areas. Neuroimage 1997; 6(4):288–304.
7. Maki A, Yamashita Y, Ito Y, Wantanabe E, Mayanayi Y,
Koizumi H. Spatial and temporal analysis of human motor activity using noninvasive NIR topography. Med Phys
1995; 22(12):1997–2005.
8. Yamashita Y, Maki A, Koizumi H. Near-infrared topographic measurement system: imaging of absorbers localized in a scattering medium. Rev Sci Instrum 1996;
67(3):730–2.
9. Koizumi H, Yamashita Y, Maki A, et al. Higher-orderbrain function analysis by trans-cranial dynamic NIRS
imaging. J Biomed Opt 1999; 4(4):403–13.
10. Kato T, Kamei A, Takashima S, Ozaki T. Human visualcortical function during photic stimulation monitoringby means of near-infrared spectroscopy. J Cereb Blood
Flow Metab 1993; 13:516–20.
11. Chanc e B, Zhuang Z, Unah C, Alter C, Lipton L.
Cognition-activated low-frequency modulation oflight absorption in human brain. Proc Natl Acad Sci
USA 1993; 90:3770–4.
12. Oldfield RC. The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 1971;
9:97–113.
13. Sakai K, Watanabe E, Onodera Y, et al. Functional mapping of the human somatosensory cortex with echoplanar magnetic resonance imaging. Magn Reson Med
1995; 33:736–43.
14. Hashimoto R, Homae F, Nakajima K, Miyashita Y, Sakai
KL. Functional differentiation in the human auditory
and language areas revealed by a dichotic listening task.
Neuroimage 2000; 12(2):147–58.
15. Sato H, Takeuchi T, Sakai KL. Temporal cortex activation
during speech recognition: an optical topography
study. Cognition 1999; 73(3):B55–66.
16. Mazoyer BM, Tzourio N, Frak V, et al. The cortical representation of speech. J Cognit Neurosci 1993;
5(4):467–79.
17. Sakai K, Miyashita Y. Memory and imagery in the temporal lobe. Curr Opin Neurobiol 1993; 3(2):166–70.

18. Sakai K, Naya Y, Miyashita Y. Neuronal tuning and associative mechanisms in form representation. Learn Mem
1994; 1(2):83–105.
19. Sakai KL, Hashimoto R, Homae F. Sentence processingin the cerebral cortex. Neurosci Res 2001; 39(1):1–10.

The authors are with the Department of Cognitive and Be havioral Science, Graduate School of Arts and Sciences,
University of Tokyo, Komaba, 3-8-1 Komaba, Meguro-ku,
Tokyo 153-8902, Japan; tel./fax: (81) 3-5454-6261; email: sakai@mind.c.u-tokyo.ac.jp. Dr. Sakai is also with
CREST, Japan Science and Technology Corp., Kawaguchishi, Japan. The authors would like to thank Dr. Ya s u s h iMiyashita, and Dr. Kyoichi Nakajima, of the Department of
Physiology, University of tokyo School of Medicine; Mr. Hi roki Sato and Mr. Tatsuya Takeuchi, in the authors’ team,for collaborating on the fMRI and OT experiments; and Ms.
Hiromi Matsuda for her administrative assistance. This work
was supported by a CREST grant from Japan Science andTechnology Corp. and a grant from Toyota Physical andChemical Research Institute Aichi, Japan, to K.L. Sakai.