ORIGINAL RESEARCH ARTICLE
published: 22 October 2013
doi: 10.3389/fnhum.2013.00681

HUMAN NEUROSCIENCE

Functional and anatomical correlates of word-, sentence-,and discourse-level integration in sign language
Tomoo Inubushi 1,2 and Kuniyoshi L. Sakai 1,3*
1Department of Basic Science, Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan
Japan Society for the Promotion of Science, Tokyo, Japan
3
CREST, Japan Science and Technology Agency, Tokyo, Japan
2

Edited by:John J. Foxe, Albert Einstein Collegeof Medicine, USA
Reviewed by:
Einat Liebenthal, Medical College of
Wisconsin, USA
Timothy M. Ellmore, The City
College of New York, USA*Correspondence:
Kuniyoshi L. Sakai, Department of
Basic Science, The University ofTokyo, 3-8-1 Komaba, Meguro-ku,
Tokyo 153-8902, Japane-mail: sakai@mind.c.u-tokyo.ac.jp

In both vocal and sign languages, we can distinguish word-, sentence-, and discourse-level
integration in terms of hierarchical processes, which integrate various elements intoanother higher level of constructs. In the present study, we used magnetic resonanceimaging and voxel-based morphometry (VBM) to test three language tasks in Japanese
Sign Language (JSL): word-level (Word), sentence-level (Sent), and discourse-level (Disc)decision tasks. We analyzed cortical activity and gray matter (GM) volumes of Deaf
signers, and clarified three major points. First, we found that the activated regions in
the frontal language areas gradually expanded in the dorso-ventral axis, corresponding
to a difference in linguistic units for the three tasks. Moreover, the activations in eachregion of the frontal language areas were incrementally modulated with the level of
linguistic integration. These dual mechanisms of the frontal language areas may reflecta basic organization principle of hierarchically integrating linguistic information. Secondly,
activations in the lateral premotor cortex and inferior frontal gyrus were left-lateralized.
Direct comparisons among the language tasks exhibited more focal activation in theseregions, suggesting their functional localization. Thirdly, we found significantly positive
correlations between individual task performances and GM volumes in localized regions,even when the ages of acquisition (AOAs) of JSL and Japanese were factored out. Morespecifically, correlations with the performances of the Word and Sent tasks were found in
the left precentral/postcentral gyrus and insula, respectively, while correlations with thoseof the Disc task were found in the left ventral inferior frontal gyrus and precuneus. The
unification of functional and anatomical studies would thus be fruitful for understanding
human language systems from the aspects of both universality and individuality.
Keywords: cerebral cortex, hemispheric lateralization, fMRI, frontal lobe, neuroimaging

INTRODUCTION
All human languages involve various elements at different levelsof hierarchical linguistic processing (Chomsky, 1995; Jackendoff,
2002). Indeed, multiple phonemes or morphemes are combined into single words through word-level integration; multiple
content words and function words are merged into single sentences through sentence-level integration, and multiple sentences
are further incorporated into discourses through discourse-level
integration. While higher-level elements integrate linguistic information from lower-level elements, associated meanings and contextual information emerge simultaneously. It has been proposedthat unification processes of phonological, syntactic, and semantic elements are gradually represented from the ventral part ofthe left lateral premotor cortex [L. LPMC, the lateral side ofBrodmann’s areas (BAs) 6/8] to the pars orbitalis of the left inferior frontal gyrus (L. F3O, BA 47) with a caudo-rostral gradient(Hagoort, 2005; Uddén and Bahlmann, 2012). In contrast, wehave proposed that syntax and sentence comprehension are organized in the dorso-ventral axis of the left lateral side of BAs 6/8,
44/45, and 47 (Sakai, 2005). Consistent with this latter possibility,

Frontiers in Human Neuroscience

previous neuroimaging studies have reported selective activationin the L. LPMC and/or pars opercularis and triangularis of theleft inferior frontal gyrus (L. F3op/F3t, BAs 44/45) for syntactic
processing (Stromswold et al., 1996; Dapretto and Bookheimer,
1999; Embick et al., 2000; Indefrey et al., 2001; Hashimoto andSakai, 2002; Sakai et al., 2002; Friederici et al., 2003; Musso et al.,
2003; Ben-Shachar et al., 2004; Sahin et al., 2006; Iijima et al.,2009; Inubushi et al., 2012), and in the L. F3O for sentence comprehension (Dapretto and Bookheimer, 1999; Homae et al., 2002,
2003; Sakai et al., 2005). Here we define the frontal language areas
as the regions consisting of the L. LPMC, L. F3op/F3t, and L. F3O.
The goal of the present study was to determine the functionalorganization of these frontal language areas. Although linguistic processes are localized in different regions according to these
postulates, there has been no direct evidence regarding how these
multiple regions are organized. Indeed, it is unknown whether
each specific region within the frontal language areas is overactivated, unchanged, or underactivated at higher levels of linguistic
integration, when compared with lower levels. Furthermore, if
the frontal language areas play fundamental roles in various

www.frontiersin.org

October 2013 | Volume 7 | Article 681 | 1

Inubushi and Sakai

Hierarchical integration in sign language

linguistic processes, then the functional organization of theseregions should be independent of input modalities, including
speech sounds, letters, and signs. We predicted that different levels of integration would be associated with increased activationsin the dorso-ventral axis of the frontal language areas.
In the present study, we used the Japanese Sign Language (JSL).
It might be thought that a sign language is a unique or atypicallanguage, because it is used only in a visual mode without auditory representations for most words. It is true that Deaf (we follow
the recent trend in capitalizing this term to refer to a culturalgroup) participants almost exclusively rely on visual information for language processing throughout their lives, which is not
the case for individuals with normal hearing. However, sign languages share the same basic properties of word-, sentence-, anddiscourse-level processes with vocal languages (Sandler and LilloMartin, 2006), which can be easily illustrated by the JSL sentences
we used (Figure 1A, Table 1). On the other hand, we included
some children (seven participants younger than 19 years old), andrecruited disproportionate numbers of females (20 females outof 28 participants). Moreover, the Deaf participants showed theindividual variability in JSL proficiency, which has been primarily
due to limited opportunities for communication and education in JSL and the written Japanese (JPN) for deaf children(Table 2). While the specific developmental changes associatedwith maturation of the language system, as well as the role of leftlateralization in various brain regions, are still under intensive
investigation, it is nevertheless well accepted that the representation of language varies with experience and the acquisitionof language skills. The relationship between language processing
and neural activity may be also different as a function of genderand age. We acknowledge a source of variance within the study
population from the inclusion of children and disproportionate
numbers of females in the sample, as well as from the wide range
in experience and duration of exposure (DOE) to JSL and JPN.
All of these factors may limit the generalization of the results toan adult population with typical language development, but it isstill challenging to examine whether the functional organization
predicted from the previous neuroimaging studies can be demonstrated by such a unique sample of participants. We believe thatour participants provide us a rare opportunity to examine theuniversality of language processing in the brain beyond modality
differences and human diversities, and to elucidate the universal relationship between the underlying neural organization andhierarchical linguistic processes.
A recent functional magnetic resonance imaging (fMRI) studyof Deaf adults reported that activations in the left frontal andsome other regions during a grammatical judgment task were
negatively correlated with the ages of acquisition (AOAs) of
American Sign Language (ASL) (Mayberry et al., 2011). Because
AOAs were also negatively correlated with the task performancesin this previous study, it is critical to separate the effects of proficiency and AOAs, and to confirm common activations among
all participants. For our functional analyses, we tried to exclude
the effects of the following factors by regarding them as nuisancefactors: the effect of age, handedness, gender, age of hearing loss,
AOAs of JSL, and AOAs of JPN. On the other hand, some previous voxel-based morphometry (VBM) studies have shown that

Frontiers in Human Neuroscience

FIGURE 1 | An experimental paradigm with language tasks inJapanese Sign Language (JSL). (A) There were three language tasks: aword-level decision (Word), a sentence-level decision (Sent), and a
discourse-level decision (Disc) task. The stimuli used in the Disc task werea long dialog articulated by two signers who were taking turns(see Table 1). In both the Word and Sent tasks, the sentences from this
dialog were presented in a randomized order. In these three languagetasks, some words or phrases were replaced with anomalous probes(indicated by asterisks before the translated words in the figure), to which
the participants were asked to respond by pressing a button. For example,
in the Word task, the word “do?” (denoted by †1) articulated by the firstsigner was replaced by a pseudo-sign (the picture with a red border). In the
Sent task, the word “do?” was replaced by a syntactically anomalous JSL
sign “done?” (the picture with a light green border) as a perfective aspectmarker expressing the past tense. In the Disc task, the word “do” (denotedby †2) articulated by the second signer was replaced by a contextuallyanomalous sign “sleep” (the picture with a blue border). The linguistic units
for these three tasks are shown in color bars separated by dots: Word (red),
Sent (light green), and Disc (blue). (B) The general linear model design
matrix for one typical participant. Run-specific predictors are shown in theright-hand columns (denoted as 1st–8th) that model differences amongindividually averaged activations in eight runs. For each of the left-handcolumns representing the first to seventh runs, the regressors of the R,
Disc (D), Sent (S), and Word (W) tasks (convolved with a hemodynamic
function), as well as the realignment parameters of three translations andthree rotations obtained from preprocessing (i.e., due to head movements),
were included in the design matrix in this order. For the eighth run, theregressors of the R and Disc tasks, as well as the six realignment
parameters, were included in the design matrix.

Deaf individuals have an increased gray matter (GM) density inthe left motor cortex (Penhune et al., 2003) and an increased
GM volume in the insula (Allen et al., 2008), compared withhearing non-signers. It has also been shown that AOAs of ASL

www.frontiersin.org

October 2013 | Volume 7 | Article 681 | 2

Inubushi and Sakai

Hierarchical integration in sign language

Table 1 | English translation of JSL discourse sentences.

Table 2 | Participants’ profiles and behavioral data for the task.

A:(a Deaf woman): Next time, would you like to do a “Momo-taro”
play?
(“Momo-taro” is a Japanese folk tale translating as “Peach Boy.”)B:(a Deaf man): Let me see . . . Yes, I would.

A:

Where would you like to do the play?

B:

In the backyard of my house.

A:

Isn’t it small for a play?

B:

No, it isn’t. It’s large enough for 20 people to gather.

Range

Mean

SD

Age (years old)

12–54

30

14

LQ

50–10093

14

Age of hearing loss (years old)

0–3

0.5

1.0

AOA (years old)

JSL

0–22

8.7

7.3

JPN

0–18

4.2

4.6

A:

Cool! I’m sure to come.

B:

We can watch the play while having BBQ.

A:

That sounds great! Should it be free?

B:

Nonsense! I’ll ask a thousand yen including BBQ and the play.

A:

What roles are included in the play?

B:

There are an old man, an old woman, Momo-taro, a dog, a bird, amonkey . . .

A:Yeah, the dog, bird, and monkey.

B:

Right. They went to Oni-ga-shima (the “Demons’ Island”).

A:

Which won?

Word

0.84–1.00

0.97

0.05

B:

Momo-taro won.

Sent

0.79–1.00

0.94

0.07

A:

Watching the play seems boring to me. BBQ and talking would be
more fun.

Disc

0.79–1.00

0.91

0.05

R

0.97–1.00

1.00

0.01

DOE (years)

Hit rate

Correct rejection rate

JSL

1–54

21

17

JPN

6–50

26

13

Word

0.20–1.00

0.78

0.22

Sent

0.10–1.00

0.57

0.23

Disc

0.50–1.00

0.76

0.12

R

0.78–1.00

0.97

0.05

B:

No, you must take part in the play, too.

A:

No way! I’m too nervous.

Word

0.16–3.58

2.63

0.84

B:

Really? For the BBQ, we will have thick, juicy, and delicious steak . . .

Sent

0.16–3.26

1.78

0.75

A:

Oh, you’re drooling . . .

Disc

1.34–3.81

2.18

0.54

B:

Now, will you take part in the play?

R

3.24–4.76

4.44

0.44

A:

OK, I want to take the role of Momo-taro.

B:No, you should take a Demon, and I’ll take Momo-taro.

A:

It’s not fair! Let’s toss a coin to decide.

B:

Which do you prefer, Momo-taro or a steak?

dRTs (ms)Word5857–11753

7747

1097

Sent

4223–9472

6696

902

Disc

4714–7890

5910

703

R

543–4802

2549

1180

A:

Wait a minute . . . a steak.

B:

You can have a steak, if you play a Demon.

LQ, laterality quotients; AOA, age of acquisition; DOE, duration of exposure;A:

All right. Who cares, I’ll be a Demon.

RTs, reaction times; JSL, Japanese Sign Language; JPN, Japanese; Word, word-B:

Heh-heh-heh.

level decision; Sent, sentence-level decision; Disc, discourse-level decision;R, repetition.

were negatively correlated with the GM density in the left precuneus (Pénicaud et al., 2013). We thus hypothesized that there
would be an anatomical signature for the linguistic proficiency of
individuals in the frontal language areas and these regions, evenwhen the functional organization of the frontal language areas
was common to individuals.

MATERIALS AND METHODSPARTICIPANTS

Thirty-seven Deaf signers participated in the present study. Nine
participants were excluded from the data analyses because of
neurological abnormalities, excessive head movements even ina single run, weak left-handedness, later onset of hearing loss
(more than 3 years old), and/or poor task performances (nullhit rates). Table 2 shows detailed profiles of the remaining 28
participants (20 females and 8 males), including two students
from a bilingual-bicultural school for the Deaf (Meisei Gakuen,
Shinagawa-ku, Japan) and three students from a facility for Deaf
children (Kanamachi Gakuen, Katsushika-ku, Japan). All participants showed right-handedness [laterality quotients (LQ) = 50],

Frontiers in Human Neuroscience

as determined by the Edinburgh inventory (Oldfield, 1971).
According to audiological reports, all participants had binauralhearing losses of >75 dB. Most of the 28 participants had experienced the cued speech method and/or the oral method to learn
JPN, but there has been no established testing in JSL or JPN for
the Deaf. Neither the grouping of early/late bilinguals nor a binary
distinction between native/non-native speakers (i.e., first/second
languages) was taken into account in the present study. Written
informed consent was obtained from all participants, as well asfrom their parents/guardians for the juvenile participants, according to the Declaration of Helsinki. The study was approved byboth the school and facility, and by the review board of The
University of Tokyo, Komaba.
STIMULI

A dialog between two Deaf signers (a woman and a man) was
prepared in JSL (see Table 1 for the entire dialog). Each stimulus consisted of familiar signs, which were sufficiently easy forthe juvenile participants to comprehend, and did not containwww.frontiersin.org

October 2013 | Volume 7 | Article 681 | 3

Inubushi and Sakai

Hierarchical integration in sign language

fingerspellings. Each dialog sentence was articulated by one oftwo signers who were taking turns, i.e., one person questioning/proposing, then the other responding, thereby completing
one or two sentences (Figure 1A). The rationale for using two
signers was to present samples of natural discourse with richprosodic cues, thereby providing contextual information throughan actual conversation (Sakai et al., 2005). Video-taped signers were always in a full-face shot, because gaze directions are
crucial in sign languages. For example, the video images of an
inquiring facial expression looking forward represented an interrogative to the other signer in the dialog. Video images of the twosigners were presented with an eyeglass-like MRI-compatible display (VisuaStim XGA; Resonance Technology, Northridge, CA)(resolution = 720 × 480, frame rate = 30 fps).
TASKS

There were three language tasks: word-level or lexical (Word),
sentence-level (Sent), and discourse-level (Disc) decision tasks.
In the Word and Sent tasks, each dialog sentence was presented
in a random order; in the Disc task, the dialog sentences were
presented in the original order, and completed in seven separate blocks. Therefore, the overall stimuli were physically equatedamong the language tasks. On the other hand, individual tasks
imposed different “task sets” (i.e., an effective intention for atask to attend the specific operations demanded by the task) as
explained below, while these tasks basically included probe detection in common in that anomalous probes infrequently appeared(in 10 out of 29 or 43 dialog sentences). It should be noted that
there was only one type of probes (lexical, syntactic, or contextual errors) included in each task block. The purpose of includingprobes was to ensure the participants’ full attention to lexicalinformation, sentence expressions, or discourse flows, thereby
allowing assessment of their linguistic proficiency for each task.
A number of linguistic studies with lexical, syntactic, or contextual decision tasks inherently involved anomaly detection in
the tasks, and we have already established that activations of thefrontal language areas depend on the type of decisions, but not on
the anomaly of stimuli themselves (Suzuki and Sakai, 2003). We
performed fMRI experiments in a block design, which measured
overall responses during each block, and thus was unaffected by
the presence of probes (mostly less than 5 s). Trials with the sentences containing anomalous probes were included in the block
design analysis. To maintain the natural flow of signs, both normal and anomalous versions of an entire sentence were filmedwith the same signers and settings, as if each sentence constituted
a normal continuous discourse even with a probe. Each block of
the language tasks consisted of four or five dialog sentences, andlasted for 24–48 s.
Using both JSL and written JPN, we instructed the participants
to respond to a probe by pressing a button while a sentence containing the probe was presented. At the initiation of each block,the task type was visually presented for 1.3 s in Japanese: “kotoba”(“word”) for the Word, “hyougen” (“expression”) for the Sent,
and “kaiwa” (“conversation”) for the Disc task. In the Word task,probes were pseudo-signs freely devised by the native signers, andthe participants were asked to detect the probes by focusing on
word-level information among the disconnected dialog sentences.

Frontiers in Human Neuroscience

Lexical decision critically involves word-level integration; for sign
languages, elements of handshape, location, and movement are
combined into real words. In the Sent task, probes were syntactically anomalous JSL expressions, and the participants were asked
to detect the probes by focusing on sentence expressions (e.g.,
word-to-word relationships) among the disconnected dialog sentences. These syntactic errors included violations of tense (seeFigure 1A), person (e.g., an agreement error between first andthird persons), word order (an ungrammatical order of lexical
items), etc. In the Disc task, probes were contextually anomalous
signs in the flow of the dialog, and the participants were asked todetect the probes by focusing on the flow of discourse, i.e., using
discourse-level integration. Every sentence containing a probe in
the Disc task was not only syntactically normal, but also semantically plausible, if the sentence was free from a given context. In ourparadigm, the levels of linguistic integration necessary for eachlanguage task can be characterized by distinct linguistic units. In
the Word, Sent, and Disc tasks, linguistic units were individual
words, disconnected dialog sentences, and consecutive sentences,respectively.
As a baseline for the language tasks, a repetition (R) task wastested with the same probe detection. At the initiation of each R
block, the task type, “kurikaeshi” (“repetition”), was visually presented for 1.3 s. In the R task, normal sentences used in the threelanguage tasks were played backward and presented in a randomized order, and a probe was a successively repeated “backward
sentence.” During each R block, video images of only one signer(e.g., a woman for an R block, a man for the next R block, etc.)
were presented, since successive presentation of different signers could not become a probe. The participants reported that it
was impossible to comprehend sentences composed of backward
signs, although some of the signs were recognizable as meaningful. Here, it should be noted that the stimuli used in the language
and R tasks were physically equivalent, i.e., in terms of the visual
stimuli themselves, which included hand shapes, facial expressions, and body movements. Activation by the contrasts between
the language and R tasks (e.g., Word − R) thus reflected the processes of individual words (in common for the language tasks)
and associated linguistic integration (different among the language tasks), while general cognitive factors such as lower-levelvisual perception, probe detection, short-term memory, response
selection, and motor responses were fully controlled.
Every participant underwent a total of eight scanning sessions,each of which had seven blocks and lasted for 220.7–221.1 s. In
the first seven sessions, four blocks of the baseline R task werealternately presented with a block of the Word, Sent, or Disc task
(appearing once in a pseudo-random order). In the last session,
four blocks of the R and three blocks of the Disc task were alternately presented, with a part of the original dialog being repeatedonce more with new probes in the Disc task. In each block of
the Disc and R tasks, there was always a single probe, and ineach block of the Word and Sent tasks, there were either one or
two probes. There were a total of ten different probes for eachof the Word, Sent, and Disc tasks, whereas there were 32 probes
for the R task. In each language task, no sentence appeared morethan twice. The use of more blocks for the Disc task might haveincreased the brain activations, but the number of blocks waswww.frontiersin.org

October 2013 | Volume 7 | Article 681 | 4

Inubushi and Sakai

Hierarchical integration in sign language

largest for the R task (Figure 1B). Since the number of blocks wasequal for the Word and Sent tasks, the change in activation by thelevels of linguistic integration, if any, cannot be explained by thedifferences in the number of blocks.
The stimulus presentation and button-press signal acquisition were controlled using the Lab-VIEW software package andinterface (National Instruments, Austin, TX). The accuracy ofeach task was evaluated with d , which was computed from theZ-scores of hit rates and correct rejection rates. If these rates
were 1.00, the formula of 1 – 1/(2N) was used (Macmillan andCreelman, 2005), where N was the total number of probes in eachtask (Word, 10; Sent, 10; Disc, 10; R, 32) for hit rates, and that ofother stimuli in each task (Word, 19; Sent, 19; Disc, 33; R, 110) for
correct rejection rates. Reaction times (RTs) were calculated fromthe onset of each dialog sentence containing a probe.
MRI DATA ACQUISITION

The participants were in a supine position, and their heads were
immobilized inside the radio-frequency coil with straps. The MRI
scans were conducted on a 3.0 T MRI system (GE Signa HDxt
3.0T; GE Healthcare, Milwaukee, WI). We scanned 32 axial slicesthat were 3-mm thick with a 0.3-mm gap, covering from −42.9
to 62.7 mm from the anterior to posterior commissure (AC-PC)
line in the vertical direction, using a gradient-echo echo-planar
imaging (EPI) sequence [repetition time (TR) = 3 s, echo time(TE) = 60 ms, flip angle (FA) = 90◦ , field of view (FOV) =192 × 192 mm2 , resolution = 3 × 3 mm2 ]. In a single scanningrun, we obtained 77 volumes following three dummy images,
which allowed for the rise of the MR signals. High-resolution T1weighted images of the whole brain (192 axial slices, 1.0 × 1.0 ×
1.0 mm2 ) were acquired with a three-dimensional fast spoiled
gradient-echo (3D FSPGR) sequence (TR = 9 ms, TE = 3 ms,
FA = 25◦ , FOV = 256 × 256 mm). These structural images wereused for normalizing fMRI and VBM data.
fMRI ANALYSES

We performed group analyses with statistical parametric mapping software (SPM8; Wellcome Trust Center for Neuroimaging,
London, UK) run on MATLAB software (MathWorks, Natick,MA). The acquisition timing of each slice was corrected using
the middle (sixteenth in time) slice as a reference. We realigned
the functional volumes to the first volume and removed runsthat included data with a translation of >4 mm in any of thethree directions and with a rotation of >2.5◦ . Each individual’s
structural image was coregistered to the mean functional image
generated during realignment. The coregistered structural image
was spatially normalized to the standard brain space as definedby the Montreal Neurological Institute (MNI) using the unified
segmentation algorithm with very light regularization, which is a
generative model that combines tissue segmentation, bias correction, and spatial normalization in the inversion of a single unified
model (Ashburner and Friston, 2005). After spatial normalization, the resultant deformation field was applied to the realigned
functional imaging data, which was resampled every 3 mm using
seventh-degree B-spline interpolation. All normalized functional
images were then smoothed by using an isotropic Gaussian kernel of 9 mm full-width at half maximum (FWHM). Task-specific

Frontiers in Human Neuroscience

effects were estimated with a general linear model (random effects
model).
In the first-level analysis, each participant’s hemodynamic
responses induced by the tasks were modeled with a box-car
function, and this function was convolved with a hemodynamic function. To minimize the effect of head movements,
the six realignment parameters obtained from preprocessing
were included as a nuisance factor in a general linear model(Figure 1B). The images of the Disc, Sent, Word, and R tasks werethen generated for each participant, and used for a second-level
analysis. To regress out the effect of age, LQ, gender, age of hearing
loss, AOAs of JSL, and AOAs of JPN, we included these nuisancefactors as covariates in the design matrix of the second-level analysis. The statistical parametric maps were thresholded at a voxel
level of uncorrected p < 0.0001, and at a cluster level of correctedp < 0.05 using the false discovery rate (FDR). For the anatomical identification of activated regions, we used the Anatomical
Automatic Labeling method (Tzourio-Mazoyer et al., 2002) andAnatomy toolbox (Eickhoff et al., 2005).
VBM ANALYSESVBM analyses on MR images were performed using SPM8
software. After alignment to the AC-PC line, T1-weighted
images were bias-corrected and segmented to the GM, whitematter, and cerebrospinal fluid by using default tissue probability maps and a New Segment tool, which uses an affine
regularization to warp images to the International Consortiumfor Brain Mapping (ICBM) East Asian brain template. Intersubject registration was achieved with Diffeomorphic Anatomical
Registration Through Exponentiated Lie Algebra (DARTEL)(Ashburner, 2007). Jacobian-scaled (“modulated”) and warped
tissue class images were then created with DARTEL’s Normalize
to MNI Space tool, which spatially normalized images to the
MNI space, converted voxel sizes to 1.5 × 1.5 × 1.5 mm3 (the sizeof the DARTEL template), and smoothed images with a standard Gaussian filter of 8-mm FWHM. To avoid possible edge
effects (partial volume effects) around the border of the GM,
voxels with a value greater than 0.2 were used for analyzing themodulated GM images. A multiple regression analysis was performed on the smoothed GM images to determine regions inwhich GM volumes showed a correlation with the d value ofeach language task. The total GM volumes of individual brains
were entered into the model as a proportional scaling factor toregress out the general size difference across the participants. Thed of the R task, as well as nuisance factors used in the fMRI analyses, was included as a covariate in the design matrix of VBM
analyses. The statistical parametric maps of GM volumes were
thresholded at a voxel level of uncorrected p < 0.001, and at acluster level of FDR-corrected p < 0.05. To account for the nonisotropic smoothness of the VBM data (Ashburner and Friston,2000), non-stationary cluster correction implemented in SPM8
was applied.

RESULTSBEHAVIORAL DATA

The task performances were well above the chance level, as indicated by the fact that the value of d was significantly larger thanwww.frontiersin.org

October 2013 | Volume 7 | Article 681 | 5

Inubushi and Sakai

Hierarchical integration in sign language

zero (all, p < 0.0001) (Table 2). These high task performancessuggest that the participants successfully detected different typesof probes. According to a One-Way repeated measures analysis
of variance (rANOVA) on the d data, there was a significant
main effect of the task [F(3, 81) = 129, p < 0.0001]. Paired t-tests
showed that the R task was significantly easier than the other threetasks (Bonferroni corrected p < 0.05). Based on a comparison of
the d values for the three language tasks, the Word task was theeasiest, whereas the Sent task was the most difficult (correctedp < 0.05).
Correlation analyses on d among the participants showedthat the performance of the Word task was significantly correlated with that of the Sent task (r = 0.61, p = 0.0004) (Table 3).
If some participants had inadvertently and partially switchedbetween the Sent and Word tasks irrespective of task instructions, then the performances of one of the two tasks would
have become worse, because the use of one task set, e.g.,
the detection of pseudo-signs, did not help in the proper useof the other task set, as there were no pseudo-signs at allin the Sent task. However, the performances on the Wordtask were found to be positively correlated with those on the
Sent task, indicating that the participants, who discriminatedone of the two tasks well, could also discriminate the othertask, even when the same set of disconnected sentences wasused.
On the other hand, the ages of the participants were strongly
correlated with the DOE of JSL (r = 0.91, p < 0.0001); the ages
and DOE of JPN were also correlated (r = 0.94, p < 0.0001). The
AOAs and DOE of JSL were negatively correlated with each other(r = −0.62, p = 0.0003), while the AOAs and DOE of JPN were
not correlated (r = 0.08, p = 0.68), probably due to the smaller
variances for the AOAs of JPN. The correlation between the AOAs
of JSL and JPN was not significant (r = −0.22, p = 0.28), indicating that these AOAs were independent of each other for theparticipants.
According to a One-Way rANOVA on the RTs, there was asignificant main effect of the task [F(3, 81) = 165, p < 0.0001].
Paired t-tests showed that the RTs for the R task were significantly shorter than those for the other three tasks (correctedp < 0.05). Among the RTs for the three language tasks, the RTs

Table 3 | Correlation matrix of participants’ profiles and behavioral
data.
Age

Age
AOA (JSL)

1.00

AOA

DOE

d

d

d

d

(JSL)

(JSL)

(Word)(Sent)(Disc)

(R)
−0.03

−0.25

0.91*

−0.18

−0.06

0.02

1.00

−0.62*

0.04

0.22

−0.05

0.15

1.00

−0.16

−0.14

0.00

−0.08

DOE (JSL)d  (Word)d  (Sent)d  (Disc)d  (R)

1.00

0.61*

0.40

0.29

1.00

0.29

0.17

1.00

−0.03
1.00

Correlation coefficients (r) are shown here. Asterisks denote significant correlations (corrected p < 0.05).

Frontiers in Human Neuroscience

for the Word task were the longest, whereas those for the Disctask were the shortest (corrected p < 0.05). In the VBM analyses,
we used the d of each task to represent the linguistic proficiency
of individuals, because d controls for any response bias the participants may have, and thus is more reliable than RTs in thiscontext.
CORTICAL ACTIVATIONS MODULATED BY THE LEVELS OF LINGUISTICINTEGRATION

To elucidate the functional organization of the frontal language
areas in an unbiased manner with respect to other corticalregions, we adopted whole brain analyses for fMRI and VBM.
Figures 2A–C show cortical responses in the Word, Sent, andDisc tasks compared with those in the R task (FDR-correctedp < 0.05). Corresponding to a difference in the linguistic units
for the Word, Sent, and Disc tasks (see Materials and Methods,
Tasks), activated regions in the frontal language areas gradually
expanded in the dorso-ventral axis, i.e., from the L. LPMC tothe L. F3O, via the L. F3op/F3t. In Word − R, significant activation was observed bilaterally in the LPMC and dorsal F3op/F3t(dF3op/F3t), as well as in the pre-supplementary motor area
(pre-SMA, the medial side of BAs 6/8) (Table 4). In Sent − R
and Disc − R, consistent activation was observed in the bilateral
LPMC/F3op/F3t, R. F3O, pre-SMA, left angular gyrus (L. AG, BA39), bilateral middle/superior temporal gyrus (MTG/STG, BAs
21/22), and bilateral caudate. In Disc − R, activation in the F3O
and AG became bilateral, indicating that the Disc task recruitedexactly mirrored regions. We also observed the left cerebellum
activation in Sent − R, and the medial precuneus (BA 7) activation in Disc − R. Taking these results together in an overlaid map(Figure 2D), the hierarchical integration in the frontal language
areas was striking, such that the more dorsal regions activated at
the lower levels of linguistic integration were almost completely
included in the wider regions activated at the higher levels. In theright lateral frontal cortex, pre-SMA, and right temporal regions,in contrast, such an integration was unclear, and some regions
activated at the lower levels were outside those activated at thehigher levels.
We focused on the frontal language areas with clear hierarchical integration, and chose three regions of interest (ROIs). The
local maximum of each region of the L. LPMC, L. F3op/F3t, andL. F3O was taken serially from significant activation in Word −R, Sent − R, and Disc − R, respectively (Table 4), to ensure an
unbiased selection of local maxima (i.e., not necessarily selective
to Disc − R alone). We examined whether activations of these
ROIs were incrementally modulated with the level of linguistic
integration (Figure 2E). A Two-Way rANOVA with the ROI (L.
LPMC, L. F3op/F3t, L. F3O) × task (Word, Sent, Disc) showedsignificant main effects of ROI [F(2, 54) = 14, p < 0.0001] andtask [F(2, 54) = 8.2, p = 0.0008] with no significant interaction[F(4, 108) = 1.0, p = 0.39]. Paired t-tests showed that the signal
changes for Disc − R were significantly higher than those forWord − R in all three regions (Bonferroni corrected p < 0.05).
Task difficulty cannot explain these modulation patterns, as theSent task was the hardest among the three language tasks; thepatterns were also independent of general short-term memory,
because we subtracted responses in the R task. Therefore, notwww.frontiersin.org

October 2013 | Volume 7 | Article 681 | 6

Inubushi and Sakai

Hierarchical integration in sign language

Table 4 | Cortical activations modulated by the levels of linguistic
integration.
BrainregionBA

Side xyz

Z value

VoxelsLPMC/
dF3op/F3t

6/8/44/45

L

−48

17

37

5.1

224

LPMC/
dF3op/F3t

6/8/44/45

R

48

20

31

4.9

136

pre-SMA

6/8

M

−9

44

46

5.0

143
538

WORD − R

SENT − R
LPMC

6/8

L

−39

11

43

5.5

F3op/F3t

44/45

L

−51

20

19

5.2

*

LPMC

6/8

R

39

14

37

6.2

437

F3op/F3t

44/45

R

45

29

19

5.7

*

F3O

47

R

48

26

1

4.2

47

pre-SMA

6/8

M

−9

38

46

6.2

323

AG

39

L

−42

−58

25

4.8

139

MTG/STG

21/22

caudatecerebellum,
crus I

L

−60

−34

−2

4.6

36

R

48

−28

−5

5.5

103

L

−15

5

16

4.6

80

R

12

5

13

4.5

77

L

−18

−76

−32

4.9

34

966

DISC − R

FIGURE 2 | Cortical activations modulated by the levels of linguistic
integration. (A–C) Cortical activations in each of the Word (A), Sent (B),and Disc (C) tasks, compared with the R task, are projected onto the lateralsurfaces of a standard brain in the MNI space. Significantly activatedregions are shown in red (FDR-corrected p < 0.05). Note the mostprominent activation in the frontal language areas. (D) An overlaid map ofcortical activations in Word − R (red), Sent − R (light green), and Disc − R(blue), using transparent overlays in this order (Word − R topmost). For
example, when a region was activated in both Sent − R and Disc − R, itscolor became blue-green (see the region in the right temporal cortex). Notethe gradual expansion of activation from the dorsal to ventral regions within
the frontal language areas. We focused on three regions in the frontal
language areas: the L. LPMC, L. F3op/F3t, and L. F3O [shown as yellow
dots in (A–C), and as black dots in (D)]. (E) Signal changes of each local
maximum for Word − R (red), Sent − R (light green), and Disc − R (blue).
Error bars indicate the SEM of participants, and asterisks denote thesignificance level of corrected p < 0.05.

only the dorso-ventral expansion of activated regions in thefrontal language areas, but also overactivation in each of theseregions was primarily influenced by the level of linguistic integration.

Frontiers in Human Neuroscience

LPMC

6/8

L

−39

8

43

6.6

F3op/F3t

44/45

L

−51

20

22

6.5

*

F3O

47

L

−30

26

−5

6.5

*

MTG/STG

21/22

L

−48

−31

−5

6.3

*

LPMC

6/8

R

42

17

43

6.2

577

F3op/F3t

44/45

R

54

23

16

5.6

*

F3O

47

R

36

26

−8

6.1

*

MTG/STG

21/22

R

54

−25

−5

5.6

79

pre-SMA

6/8

M

0

35

46

6.9

562

AG

39

L

−33

−70

40

5.6

333

R

57

−64

22

5.5

195

precuneus

7

caudate

M

−3

−67

31

5.1

93

L

−15

11

10

5.2

99

R

15

17

7

4.8

103

Stereotactic coordinates (x, y, z) in the MNI space (mm) are shown for eachactivation peak of Z values (corrected p < 0.05). BA, Brodmann’s area; L, left
hemisphere; R (in the Side column), right hemisphere; M, medial; LPMC, lateral premotor cortex; dF3op/F3t, pars opercularis and triangularis of the dorsal
inferior frontal gyrus; F3O, pars orbitalis of the inferior frontal gyrus; pre-SMA,
pre-supplementary motor area; AG, angular gyrus; MTG/STG, middle/superior
temporal gyrus. The region with an asterisk is included within the same clustershown one row above.

LATERALIZATION AND FUNCTIONAL LOCALIZATION OF CORTICAL
ACTIVATIONS

To determine which regions showed significant lateralization,
we further performed a flip method with voxel-wise analyses,
which has been shown to be superior to the ROI-based lateralization indices method for such determination (Baciu et al., 2005).
For this purpose, activations (i.e., in Word − R, Sent − R, orwww.frontiersin.org

October 2013 | Volume 7 | Article 681 | 7

Inubushi and Sakai

Hierarchical integration in sign language

Disc − R) of the brain images flipped from side to side (i.e.,
mirror-reversed images derived from fMRI first-level analyses)
were subtracted from the cortical activations of the normal
images shown in Figures 2A–C. These comparisons correspondto the interaction of task and hemisphere (left vs. right); theresultant activations in the left brain represent (Disc − R) ×
(left − right), etc., whereas those in the right brain represent(Disc − R) × (right − left), etc. (FDR-corrected p < 0.05). In
Sent − R, we observed clear left-lateralized activations in the L.
F3op/F3t and L. LPMC, as well as right-lateralized activations in
the MTG/STG (Figure 3A, Table 5). In Disc − R, we observed
clear left-lateralized activations in the L. LPMC alone (Figure 3B),
while there was no significant lateralization of activations in
Word − R.
Next, we compared activations among the language tasks.
By adding the Word task to the R task in Sent − R,i.e., Sent − (Word + R), we examined overactivation during sentence-level processes when compared with lower levels. This contrast exhibited more focal activation in the L.
LPMC and L. F3op/F3t (Figure 3C, Table 5), consistent withthe left-lateralized activation in these regions (Figure 3A). On
the other hand, the Disc − (Sent + R) contrast resulted inan activation pattern (Figure 3D) similar to that in Disc −R (Figure 2C), but the L. F3O activation was clearly separated. A direct comparison of activations between the Disc andWord tasks showed significant activation in the bilateral LPMC
and R. AG (Figure 3E), while a direct comparison between the
Disc and Sent tasks showed focal activation in the L. LPMC
(Figure 3F). This L. LPMC activation is consistent with the leftlateralized activation in this region (Figure 3B). These results
clarified the functional localization of the L. LPMC, L. F3op/F3t,
and L. F3O.

Table 5 | Lateralization and functional localization of corticalactivations.
Brain region

Side

x

y

z

Z value

VoxelsSENT − R, NORMAL − FLIP
LPMC

6/8

L

−42

−4

52

4.2

25

F3op/F3t

44/45

L

−51

14

13

4.5

21

MTG/STG

21/22

R

39

−28

−5

4.6

31

L

−45

2

46

4.9

56
208

DISC − R, NORMAL − FLIP
LPMC

6/8

SENT − (WORD + R)
LPMC

6/8

L

−42

11

43

5.0

F3op/F3t

44/45

L

−51

20

19

4.5

*

LPMC

6/8

R

39

11

37

6.1

342

F3op/F3t

44/45

R

45

26

19

5.4

*

AG

39

L

−42

−49

16

4.9

103

R

57

−58

19

5.7

151

MTG/STG

21/22

R

48

−31

−2

5.1

106
491

DISC − (SENT + R)
LPMC

6/8

L

−42

8

46

6.5

F3op/F3t

44/45

L

−57

20

19

5.8

*

LPMC

6/8

R

39

17

46

5.6

290

F3op/F3t

44/45

R

54

23

19

4.6

*

F3O

47

L

−30

26

−2

4.7

38

R

36

29

−5

5.9

62

pre-SMA

6/8

M

9

29

49

6.0

352

AG

39

L

−45

−64

22

4.6

130

R

57

−61

22

4.6

114

MTG/STG

21/22

L

−48

−28

−8

5.3

58

DISC − WORD
LPMC
AG

L

−39

8

46

5.6

159

R

39

17

46

4.7

151

39

R

57

−64

19

4.9

197

6/8

L

−36

8

49

4.6

50

6/8

Disc − Sent

POSITIVE CORRELATIONS BETWEEN INDIVIDUAL TASKPERFORMANCES AND GM VOLUMES

LPMC

We further examined correlations between the individual task
performances and regional GM volumes. Multiple regression
analyses revealed a significantly positive correlation between thed of each language task and GM volumes in localized regions(FDR-corrected p < 0.05). Between the d of the Word task andGM volumes, we found a prominent correlation in the dorsolateral surface of the left precentral gyrus and postcentral gyrus(L. PreCG/PostCG, BAs 4/3/1/2) [MNI coordinates of its peak:(x, y, z) = (−49, −22, 59), Z(19) = 4.6, p < 0.001, 1227 voxels] (Figure 4A). This region corresponded to the “hand area” ofthe primary motor and somatosensory cortices. Between the d
of the Sent task and GM volumes, a significant correlation wasobserved in the right insula [(32, −6, 17), Z(19) = 4.9, p < 0.001,
1440 voxels] (Figure 4B). The second largest cluster was located in
the left insula [(−29, −9, 8), Z(19) = 3.9, p < 0.001, 449 voxels],
which was just below the threshold of FDR-corrected p < 0.05.
Finally, we also found a strong correlation between the d ofthe Disc task and GM volumes in the L. F3O [(−46, 36, −24),
Z(19) = 4.0, p < 0.001, 593 voxels] and in the left precuneus[(−19, −54, 41), Z(19) = 4.2, p < 0.001, 590 voxels] (Figure 4C).
The L. F3O cluster was anteroinferior to the region identified
in the Disc − R (Figure 2C), and the left precuneus cluster was

Frontiers in Human NeuroscienceBA

The region with an asterisk is included within the same cluster shown one row
above.

lateral to the region identified in the Disc − R. These results
suggest an anatomical signature for the linguistic proficiency of
individuals in a task-dependent manner.

DISCUSSION
Here we analyzed cortical activity and GM volumes of Deaf participants, and clarified three major points. First, we found that
activated regions in the frontal language areas gradually expandedin the dorso-ventral axis, corresponding to a difference in linguistic units for the three language tasks. Moreover, activations
in each region of the frontal language areas were incrementally modulated with the level of linguistic integration. Thesedual mechanisms of the frontal language areas may reflect abasic organization principle of hierarchically integrating linguistic information. A previous fMRI study with passive (i.e., without
on-line tasks) reading of English texts has reported that activations in the bilateral frontal and temporal regions increasedin magnitude and spatial extent for each of these regions, withwww.frontiersin.org

October 2013 | Volume 7 | Article 681 | 8

Inubushi and Sakai

Hierarchical integration in sign language

the greatest increase being induced by narratives, followed inorder by unconnected sentences and word lists (Xu et al., 2005).
However, neither hierarchical integration nor activation modulation by the linguistic levels free from stimulus differences has
been previously clarified. We experimentally manipulated the task
sets and linguistic units, separately from the stimuli themselves(see Materials and Methods, Tasks), and clarified that active integration processes actually modulated activations in the frontal
language areas. Secondly, activations in the LPMC and F3op/F3t
were left-lateralized, and direct comparisons among the languagetasks exhibited more focal activation in these regions and the L.
F3O, suggesting their functional localization. Thirdly, we foundsignificantly positive correlations between individual task performances and GM volumes in localized regions, even when the
AOAs of JSL and JPN were factored out. More specifically, correlations with the performances of the Word and Sent tasks were
found in the L. PreCG/PostCG and insula, respectively, while correlations with those of the Disc task were found in the L. F3O
and left precuneus. These correlations suggest anatomical specialization of these regions related to individual abilities in sign
languages, irrespective of a wide range of AOAs in both sign andvocal/written languages. These results demonstrate functional
and anatomical correlates of hierarchical linguistic integration in
the frontal language areas.
The present results suggest that the L. LPMC, L. F3op/F3t, andL. F3O may primarily subserve word-, sentence-, and discourselevel integration, respectively. Hagoort and others have proposed a caudo-rostral gradient of the left inferior frontal cortex,
in which unification processes of phonological, syntactic, andsemantic elements are gradually represented in three corresponding regions from the left ventral BA 6 to BA 47 (Hagoort, 2005;

FIGURE 3 | Lateralization and functional localization of corticalactivations. (A,B) Cortical activations in the Sent − R or Disc − R of
the brain images flipped from side to side (Flip) were subtracted fromthose of the normal images (Normal). In Sent − R, Normal − Flip (A),note the significant effect of hemispheres in the L. LPMC, F3op/F3t,

Frontiers in Human Neuroscience

Uddén and Bahlmann, 2012). In our previous fMRI and lesion
studies, however, the L. LPMC was more crucially recruited forprocessing scrambled sentences (grammatical in JPN) than active
sentences, even when phonological factors were thoroughly controlled (Kinno et al., 2008, 2009). The overactivation in the L.
LPMC in the Disc task compared with the Word and Sent tasks(Figures 3E,F) can be explained by the hierarchical nature of linguistic integration, such that the Disc task with discourse-level
integration requires more intensive checking processes at wordand sentence-levels than do the other tasks without discourselevel integration. We propose that the input-driven information
integrated at the word-level in the L. LPMC is transmitted to thecenter of sentence-level integration in the L. F3op/F3t, and that
the L. F3O integrates the information from these regions into
coherent and meaningful discourses.
The hierarchical integration in the frontal language areas in thepresent study may be reminiscent of the recently proposed ideaof a “temporal receptive window” (TRW) (Hasson et al., 2008;
Lerner et al., 2011), which is the time length of effective stimuli
for a cortical region, in analogy with the spatial receptive fields ofneurons in the visual system (Hubel, 1988). By using an intersubject correlation analysis for fMRI data, Hasson et al. have showna hierarchical organization of TRWs from lower- to high-orderareas within the bilateral occipital and temporo-parietal cortices
for silent movies and speech sounds, respectively. In the bilateral frontal cortex, however, responses were scattered and limited
to longer TRWs in these previous studies. In the present study,
in contrast, hierarchical integration was most prominent in thefrontal language areas, and this organization was closely linked
with integration of linguistic information, rather than that of sensory information. Moreover, each specific region of the frontal

and R. MTG/STG. In Disc − R, Normal − Flip (B), there was asignificant effect of hemispheres in the L. LPMC alone. (C) Regions
identified by Sent − (Word + R). (D) Regions identified by Disc −(Sent + R). (E) Regions identified by Disc − Word. (F) Regions
identified by Disc − Sent.

www.frontiersin.org

October 2013 | Volume 7 | Article 681 | 9

Inubushi and Sakai

Hierarchical integration in sign language

FIGURE 4 | Anatomical signature for the linguistic proficiency of
individuals in JSL. (A–C) Correlations between regional gray matter (GM)volumes and individual performances (d  ) of the Word (A), Sent (B), andDisc (C) tasks. In the left column, correlation maps (FDR-correctedp < 0.05) are projected onto the left lateral surfaces of a standard brain for
the L. PreCG/PostCG (A) and for the L. F3O (C). An axial slice (z = 14) isshown for the largest cluster in the right (R.) insula, as well as for thesecond largest cluster in the left insula (B). A sagittal slice (x = −19) isshown for the largest cluster in the L. precuneus (C). In the right column,
scattered plots and regression lines are shown for partial correlations
between the standardized d  of each task and the standardized GM
volumes at the peak voxel, after removing the effects of age, LQ, gender,
age of hearing loss, AOAs of JSL, AOAs of JPN, and d  of the R task.

language areas was overactivated at higher levels of linguistic integration (Figure 2E), whereas the regions with shorter TRWs (e.g.,
the primary visual and early auditory cortices) responded similarly to stimuli regardless of a larger temporal context in their

Frontiers in Human Neurosciencestudies. The sources of these distinctions could be the different
stimuli, tasks, or analyses employed, as well as the hierarchicalnature of linguistic integration. Nevertheless, it would be interesting to imagine that specific linguistic functions take over a generalorganizing principle based on TRW in the frontal language areas.
Lesion studies have shown that, as in vocal languages, damage in the left hemisphere caused aphasia in sign languages, while
body movements or production of non-linguistic gestures wererelatively spared (Corina et al., 1992; Hickok et al., 1996; Marshallet al., 2004). Many previous neuroimaging studies have also confirmed the left-lateralized activations in the fronto-parietal or
fronto-temporal regions during phonological (MacSweeney et al.,2008), lexical (Leonard et al., 2012), and grammatical (Mayberry
et al., 2011) judgment tasks in sign languages, while the spatial aspects of sign languages may activate the right hemisphere(Emmorey et al., 2002; Newman et al., 2002). In our previous
fMRI study (Sakai et al., 2005), we examined cortical activity bycontrasting similar Disc and Word tasks in JSL, and reported leftlateralized activation in the frontal and temporo-parietal regionsin the JSL task for Deaf signers and hearing bilinguals (children of Deaf adults, CODA). By introducing the Sent task inthe present study, we further demonstrated that syntax-selective
activations in the L. LPMC and L. F3op/F3t were clearly leftlateralized (Figure 3A). Our results establish that some aspectsof the functional organization of frontal language areas in word-,
sentence-, and discourse-level processing are common to all individuals, even if there is a considerable individual variability inlinguistic proficiency.
In addition to the lateral frontal cortex, we observed consistent activation in the pre-SMA in the language tasks (Table 4).
The pre-SMA has been associated with motor control for sequential articulations of syllables (Ziegler et al., 1997; Bohland andGuenther, 2006), and with visuo-spatial transformation in tasks
such as mental rotation (Ecker et al., 2006), either of which may
explain the consistent pre-SMA activation in the present study. In
both the Sent − R and Disc − R, we also observed activation in
the temporo-parietal regions, including the bilateral MTG/STG
and the left AG, which are involved in phonological and lexicosemantic processes (Dronkers et al., 2004; Hickok and Poeppel,
2007). It is possible that these regions are recruited more intensively during the Sent and Disc tasks, i.e., at higher levels of
integration of the phonological and lexico-semantic information.
On the other hand, the right AG activation was selective to theDisc task (Figures 2D, 3E), indicating its recruitment only at thehighest level of linguistic integration. In addition to these corticalactivations, we observed significant activation in the cerebellumin Sent − R, and that in the caudate in both Sent − R and Disc− R. Some previous studies have suggested that on-line linguistic
processing is also controlled by the internal model of the cerebellum (Ito, 2008; Lesage et al., 2012), and that the caudate plays animportant role in bilingual switching (Crinion et al., 2006; Tanet al., 2011). Our task actually required on-line detection of linguistic errors with a time constraint, and most of the participants
were bilinguals in JSL and JPN.
Previous VBM studies have shown that increased GM volumes
in distinct regions were associated with proficiencies in different aspects of vocal languages, including vocabulary, literacy, or

www.frontiersin.org

October 2013 | Volume 7 | Article 681 | 10

Inubushi and Sakai

Hierarchical integration in sign language

syntactic abilities (Mechelli et al., 2004; Lee et al., 2007; Carreiras
et al., 2009; Nauchi and Sakai, 2009). The increased GM volume
in the dorsolateral surface of the L. PreCG/PostCG (i.e., the “handarea”) may be related to the better acquisition of subtle and complex hand movements by signers, who would therefore have better
lexical knowledge in JSL as well. A previous VBM study has shownthat GM volumes in the bilateral primary motor and somatosensory cortices were larger in professional keyboard players thannon-musicians (Gaser and Schlaug, 2003). It is striking to notethat the signers with right-hand dominance for general motor
controls in our study showed a prominent correlation in the left
PreCG/PostCG. Indeed, even two-hand signs have certain phonological constraints, such that the handshapes and movements
must be either more complex in the dominant hand or symmetric in both hands (Sandler and Lillo-Martin, 2006). As regards
the insula with a significant correlation between its GM volumes
and the d of the Sent task, previous voxel-based lesion symptom
mapping and fMRI studies have shown that the anterior insula
is critical for coordinating speech articulation (Dronkers, 1996;
Bohland and Guenther, 2006; Kemeny et al., 2006; Baldo et al.,2011). The precuneus, in which we observed a significant correlation between the GM volumes and the d of the Disc task, hasbeen shown to be involved in the shifting attention between different locations in space necessary for the coordination of motor
control (Wenderoth et al., 2005), and also to be necessary forthe production of signs with both hands. Based on the suggested
motor-related functions of the L. PreCG/PostCG, insula, and precuneus, we suspect that these regions have supportive roles inaccomplishing correct linguistic decisions in sign languages. In
contrast, the L. F3O, the GM volumes of which were significantly
correlated with the d of the Disc task, would also be functionallyspecialized in discourse-level integration, further supporting its
pivotal role in sentence comprehension.
In spite of the fact that we utilized JSL and examined Deaf signers, which included some children and disproportionate numbersof females, we found consistently left-lateralized activation in
the frontal language areas among the participants we tested(Figures 2, 3). Moreover, each of these activated regions precisely

REFERENCES
Allen, J. S., Emmorey, K., Bruss, J., andDamasio, H. (2008). Morphology of
the insula in relation to hearing status and sign language experience.
J. Neurosci. 28, 11900–11905. doi:
10.1523/JNEUROSCI.3141-08.2008
Ashburner, J. (2007). A fast diffeomorphic image registration algorithm. Neuroimage 38, 95–113. doi:
10.1016/j.neuroimage.2007.07.007
Ashburner, J., and Friston, K. J. (2000).
Voxel-based morphometry - themethods. Neuroimage 11, 805–821.
doi: 10.1006/nimg.2000.0582
Ashburner, J., and Friston, K. J.
(2005). Unified segmentation.
Neuroimage 26, 839–851. doi:
10.1016/j.neuroimage.2005.02.018
Baciu, M., Juphard, A., Cousin, E.,
and Le Bas, J. F. (2005). Evaluating

Frontiers in Human Neuroscience

fMRI methods for assessing hemispheric language dominance inhealthy subjects. Eur. J. Radiol. 55,
209–218. doi: 10.1016/j.ejrad.2004.
11.004
Baldo, J. V., Wilkins, D. P., Ogar, J.,
Willock, S., and Dronkers, N. F.
(2011). Role of the precentral gyrusof the insula in complex articulation. Cortex 47, 800–807. doi:
10.1016/j.cortex.2010.07.001
Ben-Shachar, M., Palti, D., and
Grodzinsky, Y. (2004). Neuralcorrelates of syntactic movement:
converging evidence from two
fMRI experiments. Neuroimage
21, 1320–1336. doi: 10.1016/j.
neuroimage.2003.11.027
Bohland, J. W., and Guenther, F. H.
(2006). An fMRI investigationof syllable sequence production.

matched one of those reported by the previous neuroimaging
studies with vocal languages, which showed the involvement of
the L. LPMC and/or the L. F3op/F3t in syntactic processing, as
well as that of the left F3O in sentence comprehension (see theIntroduction). The new finding of the present study is that thefrontal language areas in the dorso-ventral axis are hierarchicallyorganized in correspondence with the different levels of linguistic
integration. The striking consistency of the organization within
our Deaf population indicates the universality of linguistic processing beyond modality differences and human diversities, which
would also be generalizable to other individuals in any natural languages that are based on universal grammar (Chomsky,1995). In conclusion, our study demonstrated the functional
and anatomical correlates of hierarchical linguistic integrationin the frontal language areas and other regions. The unificationof functional and anatomical studies would thus be fruitful forunderstanding human language systems from the aspects of both
universality and individuality.

AUTHOR CONTRIBUTIONS
Conceived and designed the experiments: Tomoo Inubushi andKuniyoshi L. Sakai. Performed the experiments: Tomoo Inubushi
and Kuniyoshi L. Sakai. Analyzed the data: Tomoo Inubushi
and Kuniyoshi L. Sakai. Wrote the paper: Tomoo Inubushi andKuniyoshi L. Sakai.

ACKNOWLEDGMENTS
We thank H. Miyashita, K. Iijima, S. Ohta, and T. Nakai for theirhelpful discussions, Y. Morimoto, M. Yahagi, Y. Kazumi, and A.
Sunada for their assistance with the experiments, N. Komoro
for her technical assistance, and H. Matsuda for her administrative assistance. This research was supported by a Core Research
for Evolutional Science and Technology (CREST) grant from theJapan Science and Technology Agency (JST), by a Grant-in-Aid
for Scientific Research (S) (No. 20220005) from the Ministryof Education, Culture, Sports, Science and Technology, and by
a Grant-in-Aid for Japan Society for the Promotion of Science(JSPS) Fellows (No. 24·9028).
Neuroimage 32, 821–841. doi:
10.1016/j.neuroimage.2006.04.173
Carreiras, M., Seghier, M. L., Baquero,S., Estévez, A., Lozano, A., Devlin,
J. T., et al. (2009). An anatomical signature for literacy. Nature
461, 983–986. doi: 10.1038/
nature08461
Chomsky, N. (1995). The Minimalist
Program. Cambridge, MA: The MIT
Press.
Corina, D. P., Poizner, H., Bellugi,
U., Feinberg, T., Dowd, D.,
and O’Grady-Batch, L. (1992).
Dissociation between linguistic andnonlinguistic gestural systems: acase for compositionality. Brain
Lang. 43, 414–447. doi: 10.1016/
0093-934X(92)90110-Z
Crinion, J., Turner, R., Grogan, A.,
Hanakawa, T., Noppeney, U.,

www.frontiersin.org

Devlin, J. T., et al. (2006). Language
control in the bilingual brain.
Science 312, 1537–1540. doi:
10.1126/science.1127761
Dapretto, M., and Bookheimer, S. Y.
(1999). Form and content: dissociating syntax and semantics insentence comprehension. Neuron
24, 427–432. doi: 10.1016/S08966273(00)80855-7
Dronkers, N. F. (1996). A new brain
region for coordinating speech
articulation. Nature 384, 159–161.
doi: 10.1038/384159a0
Dronkers, N. F., Wilkins, D. P., Van
Valin, R. D. Jr., Redfern, B. B., andJaeger, J. J. (2004). Lesion analysisof the brain areas involved in language comprehension. Cognition 92,
145–177. doi: 10.1016/j.cognition.
2003.11.002

October 2013 | Volume 7 | Article 681 | 11

Inubushi and Sakai

Ecker, C., Brammer, M. J., David,
A. S., and Williams, S. C. (2006).
Time-resolved fMRI of mental
rotation
revisited-dissociating
visual perception from mental rotation in female subjects.
Neuroimage 32, 432–444. doi:
10.1016/j.neuroimage.2006.03.031
Eickhoff, S. B., Stephan, K. E.,
Mohlberg, H., Grefkes, C., Fink,
G. R., Amunts, K., et al. (2005). A
new SPM toolbox for combining
probabilistic
cytoarchitectonic
maps and functional imaging data.
Neuroimage 25, 1325–1335. doi:
10.1016/j.neuroimage.2004.12.034
Embick, D., Marantz, A., Miyashita,Y., O’Neil, W., and Sakai, K. L.
(2000). A syntactic specializationfor Broca’s area. Proc. Natl. Acad.
Sci. U.S.A. 97, 6150–6154. doi:
10.1073/pnas.100098897
Emmorey,K.,
Damasio,
H.,
McCullough, S., Grabowski, T.,
Ponto, L. L. B., Hichwa, R. D., et al.
(2002). Neural systems underlying
spatial language in American sign
language. Neuroimage 17, 812–824.
doi: 10.1006/nimg.2002.1187
Friederici, A. D., Rüschemeyer, S.-A.,
Hahne, A., and Fiebach, C. J. (2003).
The role of left inferior frontal
and superior temporal cortex insentence comprehension: localizing
syntactic and semantic processes.
Cereb. Cortex 13, 170–177. doi:
10.1093/cercor/13.2.170
Gaser, C., and Schlaug, G. (2003). Brainstructures differ between musicians
and non-musicians. J. Neurosci. 23,
9240–9245.
Hagoort, P. (2005). On Broca, brain,
and binding: a new framework.
Trends Cogn. Sci. 9, 416–423. doi:
10.1016/j.tics.2005.07.004
Hashimoto, R., and Sakai, K. L. (2002).
Specialization in the left prefrontal
cortex for sentence comprehension. Neuron 35, 589–597. doi:
10.1016/S0896-6273(02)00788-2
Hasson, U., Yang, E., Vallines, I.,
Heeger, D. J., and Rubin, N. (2008).
A hierarchy of temporal receptive windows in human cortex.
J. Neurosci. 28, 2539–2550. doi:
10.1523/JNEUROSCI.5487-07.2008
Hickok, G., Bellugi, U., and Klima,
E. S. (1996). The neurobiology
of sign language and its implications for the neural basis of language. Nature 381, 699–702. doi:
10.1038/381699a0
Hickok, G., and Poeppel, D. (2007).
Opinion - the cortical organizationofspeechprocessing.
Nat.
Rev. Neurosci. 8, 393–402. doi:
10.1038/nrn2113
Homae, F., Hashimoto, R., Nakajima,K., Miyashita, Y., and Sakai, K. L.

Frontiers in Human Neuroscience

Hierarchical integration in sign language(2002). From perception to sentencecomprehension: the convergenceof auditory and visual informationof language in the left inferior
frontal cortex. Neuroimage 16,
883–900. doi: 10.1006/nimg.2002.
1138
Homae, F., Yahata, N., and Sakai,
K. L. (2003). Selective enhancement of functional connectivity
in the left prefrontal cortex
duringsentence
processing.
Neuroimage 20, 578–586. doi:
10.1016/S1053-8119(03)00272-6
Hubel, D. H. (1988). Eye, Brain, and
Vision. New York, NY: Scientific
American Library.
Iijima, K., Fukui, N., and Sakai, K.
L. (2009). The cortical dynamics
in building syntactic structuresof sentences: an MEG study
in a minimal-pair paradigm.
Neuroimage 44, 1387–1396. doi:
10.1016/j.neuroimage_2008.10.041
Indefrey, P., Brown, C. M., Hellwig,
F., Amunts, K., Herzog, H., Seitz,
R. J., et al. (2001). A neural correlate of syntactic encoding during speech production. Proc. Natl.
Acad. Sci. U.S.A. 98, 5933–5936. doi:
10.1073/pnas.101118098
Inubushi, T., Iijima, K., Koizumi, M.,
and Sakai, K. L. (2012). Left inferior frontal activations dependingon the canonicity determined by
the argument structures of ditransitive sentences: an MEG study. PLoS
ONE 7:e37192. doi: 10.1731/journal.pone.0037192
Ito, M. (2008). Control of mental
activities by internal models in the
cerebellum. Nat. Rev. Neurosci. 9,
304–313. doi: 10.1038/nrn2332
Jackendoff, R. (2002). Foundationsof Language: Brain, Meaning,
Grammar,
Evolution.
Oxford:Oxford University Press.
Kemeny, S., Xu, J., Park, G. H.,
Hosey, L. A., Wettig, C. M., andBraun, A. R. (2006). Temporal dissociation of early lexical accessand articulation using a delayed
naming task - an fMRI study.
Cereb. Cortex 16, 587–595. doi:
10.1093/cercor/bhj006
Kinno, R., Kawamura, M., Shioda,
S., and Sakai, K. L. (2008). Neuralcorrelates of noncanonical syntactic
processing revealed by a picturesentence matching task. Hum.
Brain Mapp. 29, 1015–1027. doi:
10.1002/hbm.20441
Kinno, R., Muragaki, Y., Hori, T.,
Maruyama, T., Kawamura, M., andSakai, K. L. (2009). Agrammatic
comprehension caused by aglioma in the left frontal cortex. Brain Lang. 110, 71–80. doi:
10.1016/j.bandl.2009.05.001

Lee, H., Devlin, J. T., Shakeshaft,C., Stewart, L. H., Brennan,A., Glensman, J., et al. (2007).
Anatomical traces of vocabularyacquisition in the adolescent brain.
J. Neurosci. 27, 1184–1189. doi:
10.1523/JNEUROSCI.4442-06.2007
Leonard,M.K.,Ramirez,N.F.,Torres,C.,
Travis, K. E., Hatrak, M., Mayberry,
R. I., et al. (2012). Signed words
in the congenitally deaf evoke typical late lexicosemantic responses
with no early visual responses
in left superior temporal cortex.
J. Neurosci. 32, 9700–9705. doi:
10.1523/JNEUROSCI.1002-12.2012
Lerner, Y., Honey, C. J., Silbert,
L. J., and Hasson, U. (2011).
Topographic mapping of ahierarchy of temporal receptive
windows using a narrated story.
J. Neurosci. 31, 2906–2915. doi:
10.1523/JNEUROSCI.3684-10.2011
Lesage, E., Morgan, B. E., Olson, A.
C., Meyer, A. S., and Miall, R.
C. (2012). Cerebellar rTMS disrupts predictive language processing. Curr. Biol. 22, R794–R795. doi:
10.1016/j.cub.2012.07.006
Macmillan, N. A., and Creelman, C.
D. (2005). Detection Theory: A
User’s Guide. Mahwah, NJ: Lawrence
Erlbaum Associates.
MacSweeney, M., Waters, D., Brammer,
M. J., Woll, B., and Goswami, U.
(2008). Phonological processing indeaf signers and the impact ofage of first language acquisition.
Neuroimage 40, 1369–1379. doi:
10.1016/j.neuroimage.2007.12.047
Marshall, J., Atkinson, J., Smulovitch,E., Thacker, A., and Woll, B.
(2004). Aphasia in a user ofBritish sign language: dissociation
between sign and gesture. Cogn.
Neuropsychol. 21, 537–554. doi:
10.1080/02643290342000249
Mayberry, R. I., Chen, J.-K., Witcher, P.,
and Klein, D. (2011). Age of acquisition effects on the functional organization of language in the adultbrain. Brain Lang. 119, 16–29. doi:
10.1016/j.bandl.2011.05.007
Mechelli, A., Crinion, J. T., Noppeney,
U., O’Doherty, J., Ashburner, J.,
Frackowiak, R. S., et al. (2004).
Structural plasticity in the bilingual brain - proficiency in a second language and age at acquisition
affect grey-matter density. Nature
431, 757. doi: 10.1038/431757aMusso, M., Moro, A., Glauche, V.,
Rijntjes, M., Reichenbach, J.,
Büchel, C., et al. (2003). Broca’s
area and the language instinct.
Nat. Neurosci. 6, 774–781. doi:
10.1038/nn1077
Nauchi, A., and Sakai, K. L. (2009).
Greater leftward lateralization of

www.frontiersin.org

the inferior frontal gyrus in second
language learners with higher syntactic abilities. Hum. Brain Mapp.
30, 3625–3635. doi: 10.1002/hbm.
20790
Newman, A. J., Bevelier, D., Corina, D.,
Jezzard, P., and Neville, H. J. (2002).
A critical period for right hemisphere recruitment in American
sign language processing. Nat.
Neurosci. 5, 76–80. doi: 10.1038/
nn775
Oldfield, R. C. (1971). The assessment
and analysis of handedness:the
Edinburgh
inventory.
Neuropsychologia 9, 97–113. doi:
10.1016/0028-3932(71)90067-4
Penhune, V. B., Cismaru, R.,
Dorsaint-Pierre, R., Petitto, L.A., and Zatorre, R. J. (2003). The
morphometry of auditory cortex
in the congenitally deaf measured using MRI. Neuroimage 20,
1215–1225. doi: 10.1016/S10538119(03)00373-2
Pénicaud, S., Klein, D., Zatorre, R.
J., Chen, J.-K., Witcher, P., Hyde,K., et al. (2013). Structural brainchanges linked to delayed first language acquisition in congenitally
deaf individuals. Neuroimage 66,
42–49. doi: 10.1016/j.neuroimage.
2012.09.076
Sahin, N. T., Pinker, S., and Halgren,
E. (2006). Abstract grammaticalprocessing of nouns and verbs in
Broca’s area: evidence from fMRI.
Cortex 42, 540–562. doi: 10.1016/
S0010-9452(08)70394-0
Sakai, K. L. (2005). Language acquisition and brain development. Science
310, 815–819. doi: 10.1126/science.
1113530
Sakai, K. L., Noguchi, Y., Takeuchi, T.,
and Watanabe, E. (2002). Selective
priming of syntactic processing by
event-related transcranial magnetic
stimulation of Broca’s area. Neuron
35, 1177–1182. doi: 10.1016/S08966273(02)00873-5
Sakai, K. L., Tatsuno, Y., Suzuki, K.,
Kimura, H., and Ichida, Y. (2005).
Sign and speech: amodal commonality in left hemisphere dominancefor comprehension of sentences.
Brain 128, 1407–1417. doi: 10.1093/
brain/awh465
Sandler, W., and Lillo-Martin,
D. (2006). Sign Language andLinguistic Universals. Cambridge,
UK:
Cambridge
University
Press. doi: 10.1017/CBO978113
9163910
Stromswold, K., Caplan, D., Alpert, N.,
and Rauch, S. (1996). Localizationof syntactic comprehension bypositron emission tomography.
Brain Lang. 52, 452–473. doi:
10.1006/brln.1996.0024

October 2013 | Volume 7 | Article 681 | 12

Inubushi and Sakai

Suzuki, K., and Sakai, K. L. (2003).
An event-related fMRI study of
explicit syntactic processing of normal/anomalous sentences in contrast to implicit syntactic processing. Cereb. Cortex 13, 517–526. doi:
10.1093/cercor/13.5.517
Tan, L. H., Chen, L., Yip, V., Chan,
A. H. D., Yang, J., Gao, J.-H., et al.
(2011). Activity levels in the left
hemisphere caudate-fusiform circuit predict how well a second language will be learned. Proc. Natl.
Acad. Sci. U.S.A. 108, 2540–2544.
doi: 10.1073/pnas.0909623108
Tzourio-Mazoyer, N., Landeau, B.,
Papathanassiou, D., Crivello, F.,
Etard, O., Delcroix, N., et al. (2002).
Automated anatomical labeling ofactivations in SPM using a macroscopic anatomical parcellation

Frontiers in Human Neuroscience

Hierarchical integration in sign languageof the MNI MRI single-subject
brain. Neuroimage 15, 273–289. doi:
10.1006/nimg.2001.0978
Uddén, J., and Bahlmann, J. (2012). A
rostro-caudal gradient of structured
sequence processing in the left
inferior frontal gyrus. Philos. Trans.
R. Soc. B Biol. Sci. 367, 2023–2032.
doi: 10.1098/rstb.2012.0009
Wenderoth, N., Debaere, F., Sunaert,
S., and Swinnen, S. P. (2005). The
role of anterior cingulate cortex and
precuneus in the coordination of
motor behaviour. Euro. J. Neurosci.
22, 235–246. doi: 10.1111/j.14609568.2005.04176.x
Xu, J., Kemeny, S., Park, G., Frattali,C., and Braun, A. (2005). Language
in context: emergent features ofword, sentence, and narrative
comprehension.
Neuroimage

25, 1002–1015. doi: 10.1016/j.
neuroimage.2004.12.013
Ziegler, W., Kilian, B., and Deger,
K. (1997). The role of the left
mesial frontal cortex in fluent
speech: evidence from a case ofleft supplementary motor area
hemorrhage. Neuropsychologia 35,
1197–1208. doi: 10.1016/S00283932(97)00040-7
Conflict of Interest Statement: The
authors declare that the research
was conducted in the absence of any
commercial or financial relationshipsthat could be construed as a potential
conflict of interest.
Received: 30 July 2013; accepted: 27
September 2013; published online: 22
October 2013.

www.frontiersin.org

Citation: Inubushi T and Sakai KL(2013) Functional and anatomicalcorrelates of word-, sentence-, anddiscourse-level integration in sign language. Front. Hum. Neurosci. 7:681. doi:
10.3389/fnhum.2013.00681
This article was submitted to the journal
Frontiers in Human Neuroscience.
Copyright © 2013 Inubushi and Sakai.
This is an open-access article distributed under the terms of the Creative
Commons Attribution License (CC BY).
The use, distribution or reproduction
in other forums is permitted, provided
the original author(s) or licensor are
credited and that the original publication in this journal is cited, in accordance with accepted academic practice.
No use, distribution or reproduction ispermitted which does not comply withthese terms.

October 2013 | Volume 7 | Article 681 | 13