Brain Activity During Turn-Taking Using Virtual Characters: An fMRI Study on Humans Interactingwith Virtual Characters1
Masahie Yuasa, Natsumi Watanabe and Akira Toyomura
In this study, we examined turn-taking behavior by measuring brain activity using functional magnetic resonance imaging(fMRI). We developed virtual characters that exhibited turn-taking behavior. These characters displayed facial expressions
that indicated a willingness to continue talking, and regulated its timing in conversational turn-taking. We then confirmed thevalidity of these behaviors through behavioral experiments that used a pseudo-conversation between these virtual charactersand human participants. In addition, we conducted an fMRI experiment where the participants were required to read part of a
scenario in a conversation with the character. The results showed that the insular cortex and the right superior temporal gyrus
were significantly activated when the character behaved properly in turn-taking. These results suggest that the activation of
the right superior temporal gyrus was related to mutual understanding. The insular cortex has been shown to be related toempathy in previous studies. Thus, turn-taking may have a significant role related to mutual understanding and empathy inconversation. Our experimental design and findings may provide an objective framework for virtual character design, andcontribute to the study of human conversation.
1. Creation of the turn-taking scenarios and virtual
character behavior<1・1> The planned conversational turn-taking scenarios
Previous research has measured brain activity when hearing
words which are “closely related” and “completely unrelated”
to a previous word and when hearing sentences “for which thecontext is correct” and “for which the context is not correct.”
Similarly, we created an “easy-to-continue” and a “difficultto-continue” conversational turn-taking scenario andinvestigated brain activity by recording differences in thefMRI measurement results.
<1・2>Planning the conversational turn-taking scenarios
We assumed a conversation with three parties anddeveloped the face-to-face system shown in Fig. 1.
Although turn-taking occurs with two people as well,conversation participants may begin speaking at the sametime during turn-taking between three parties or more and so
it is necessary for the conversation participants to attempt tojudge and resolve who will continue the conversation (1)(43).
As this study focuses on the continuation of conversation,
we will use three party turn taking, which makes judgment
and resolution for the continuation and maintenance of theconversation more essential.
As shown in Fig. 1, character A and character B wereplaced on the left and right of the screen and assumed
conversation with the experiment participant who was infront of the display screen. The television production
software TVML was used in creating the scenario (44). TVML
displays the virtual characters on the screen and their actions
and speech can be controlled by a script (language). “Lipsyncing,” in which the size of the mouth changes according
to the volume of the synthesized speech (or speech file), wasalso implemented (45)(46) using TVML to allow the face-toface conversation to continue without noticeable
discrepancies between the image of the mouth and thecharacter’s speech (7)～(9).

1

IEEJ Transactions on Electronics, Information and Systems
Vol.137 No.5 pp.724–734

Further, the following was designed for this experimentwith consideration to the later brain activity measurement
experiment.
(1) The fixed speaking order of the characters and theexperiment participant (speaking order: character A,
character B, experiment participant, character B again)(2) The advance creation of the script (spoken words).
The characters spoke a dialogue created in advance. Whenit was the experiment participant’s turn to speak, theirdialogue was presented onscreen as a caption and they readit aloud.
As the development and flow of conversation between
people is usually unpredictable and difficult to control in anexperiment, a virtual turn-taking situation was produced bydetermining the speaking order and dialogue of thecharacters and the experiment participant in order to control
their actions.
An example of the flow of conversation is shown in Fig. 1.
First, character A on the left says, “Have you finished yourwork for today?” (Fig. 1 (1)). In response, character B says,“Yes, I’ve finished my work for today” (Fig. 1 (2)). Next, acaption appears at the bottom of the screen. At this time, as
shown in Fig. 1 (3), character A and character B look forward(they look at the experiment participant, or look at thecamera, so to speak). The experiment participant reads thecaption of “That’s good. You’ll go home early, won’t you?”as their own line, developing a pseudo-conversation. After
this, character B’s response is “Yes, I want to go home earlyso that I can watch my favorite video” (Fig. 1 (4)). The final
action of character B is an expression which matches any of
the four previously mentioned categories: “Change,” “Nochange,” “Natural,” or “Delayed” (Table 1). The characters’
voices are human voices recorded in advance. Note that thevoices were from the same person and it is thought that there
is no difference in the impression from the voice (tone) in
each experimental condition.

Fig.1.The Flow of Conversation by Virtual Characters and Participant.

Fig.2. Changes in Facial Expression.
Table 1. Stimulus 1–4
Stimulus 1

Stimulus 2

Stimulus 3

Stimulus 4

MotionChangeChangeNo Change

No Change

TimingNaturalDelay

NaturalDelay

After the experiment participant spoke their line, theexperimenter (author) would cause character B to say theline from Fig. 1 (4) by manually pressing a button on the
keyboard while listening to the experiment participant speak.
Note that the operation from the keyboard button pressed by
the experimenter was remote and was not seen by theexperiment participant. During the instruction beforehand,
it was emphasized in the explanation that “the characters are
moving in response to your (the experiment participant’s)
voice.” Accordingly, it was believable that the experimentparticipant and characters were having a conversation.

<1・2・1> Investigation of behavior during
conversational turn-taking
In this study, we focused on “change” and “speech timing”
which are connected to the continuation of conversational
turn-taking.
“Change” is the attitude displayed by the listener during
turn-taking. Conversation participants anticipate who wants
to speak and who wants to listen (2). For the purpose ofcontinuing a conversation, it is particularly important todisplay an attitude of wanting to speak (8). For example,
changes such as smiling or leaning forward indicate a desire

to speak and lead to positively and actively progressing theconversation. Meanwhile, if one converses with noexpression, this demonstrates that they do not really want totalk or continue the conversation which does not lead to acontinued conversation.
“Speech timing” is the timing in which the next speaker
begins speaking (the interval before they begin speaking).
Nagaoka et al. noted that the interval while the speaker isswitching has a major effect on the expression of specificemotions (47). Further, Trimboli states that the intervalchanges depending on the type of conversation(cooperative conversation, confrontational conversation)(48). Timing is one essential element composing
conversation. Speaking with timing that smoothly
advances the conversation with no discomfort leads tomaintaining a conversation (47). Meanwhile, leaving aninterval after the previous speaking has finished before one
begins speaking, or speaking while another is still speaking,do not lead to maintaining a conversation (47). The
following four behaviors were created based on the above
investigations. Combinations of these behaviors compose
experimental stimuli 1 through 4. Table 1 shows the
combinations in experimental stimuli 1 through 4.
・“Change”: Expression of a desire to speak. Can beunderstood as a desire to pleasantly continue
conversation.
・“No Change”: No facial expression. Can beunderstood as not really wanting to speak and not wanting
to continue conversation.
・“Natural Speech Timing”: Beginning to speak withappropriate timing (interval) to progress the conversation
with no discomfort.
・“Delayed Speech Timing”: Beginning to speak too
long after the previous speaker has finished, making it
difficult to progress the conversation.
<1・2・2> Implementation of conversational turn-taking
behavior
The “Change” used in this experiment is based on
previous research (8)(49) and consists of: (1) raising theeyebrows, (2) slight upward extension of the upper half ofthe body and the neck, and (3) raising the corners of themouth (both ends). Fig. 2 demonstrates an example
implemented with a character. Past research has verified
that these changes are an expression of a desire to speak(an expression claiming the right to speak) (8).
In stimuli in which the “speech timing” was “Natural”(experimental stimuli 1 and 3), character B began speaking
immediately after the participant finished speaking. Whenthe speech timing was “delayed” (experimental stimuli 2
and 4), there was an interval (0.6 sec) between when theparticipant finished speaking and when character B began
speaking. A value of 0.6 seconds was used as this is thetechnical limit for the facial CG synthesis to appear natural;any shorter than this and it will appear unnatural. Enomoto
et al. report that turn-taking ceases to be smooth when aninterval of one second or longer is left (50) and so an intervalof 0.6 seconds is not too long. The experimenter controlled
the speech timing of the characters (“Natural” or“Delayed”) by pressing a button on the keyboard while
listening to the participant speak.
2. Verification of the interpretation of behavior andconversation continuity by questionnaire<2・1> Character behavior verification experiment<2・1・1> Items assessed by the questionnaire
A questionnaire experiment was implemented to explore theinterpretation of the experimental stimuli created. To selectthe items assessed by the questionnaire, an interview was
carried out regarding the experimental stimuli in advance andsix assessment items representing the characteristics of theexperimental stimuli were selected ((a) Comfortable Uncomfortable, (b) Natural tempo - Unnatural tempo, (c)Smooth - Rough, (d), Comfortable - Uncomfortable, (d)Standard - Strangeness, (f) Mutual Understanding Misunderstanding). Each assessment item was assessed on
five levels. Further, to explore differences in conversation
continuity for each experimental stimulus, the items of “(g)Character’s willingness to continue talking (whether theparticipant thinks the character on the right “wants to keep
talking”)” and “(h) Participant’s willingness to continue
talking (whether you feel, based on the words and actions of
the character on the right, you “want to continue talking”)”
were created and explored. Responses to (g) and (h) were
established as seven level assessments.
Experiment participants were asked to carry on a pseudoconversation with the characters as shown in Fig. 1. It wasconveyed to participants that: “this is a three-party
conversation between you and the characters,” “thecharacters will listen and respond to your voice,” and “yourdialogue will be shown as a caption, so please read it.”Further, it was confirmed that the participants understood
the flow of conversation from the practice beforehand. The
characters’ spoken lines were the same; only the non-verbal
behaviors for experimental stimuli 1 through 4 werechanged. Each execution of the questionnaire items andexperimental stimuli used a random order.
<2・1・2> Experiment results
Respondents were university students. Assessment items(a) through (f) were answered by 17 individuals and (g)through (h) were answered by 19 individuals.
<2・1・3> Discussion of the questionnaire experiment
The results demonstrated that, compared to other stimuli,
experimental stimulus 1 had significantly higher
assessments for the survey items of “highly comfortable,”“natural tempo,” “smooth,” “comfortable,” “character’s
willingness to continue talking” and “participant’s
willingness to continue talking.” Therefore, experimental
stimulus 1 consisted of behavioral expression consistent
with the realization of a continuous conversation.
Meanwhile, compared to other stimuli, experimental
stimulus 4 was lower on all assessed values: unnatural
tempo, rough, uncomfortable, etc. and it is unlikely that thisstimulus would encourage a continuous conversation. It isthought that the created behaviors themselves were natural,not strange, and there were no problems using these as
experimental stimuli for turn-taking.
<2・2> Summary of the questionnaire experimentapproved experiment explanation form and documents
after which consent was obtained in writing.

From the above, it was found that experimental stimulus 1
included change and timing for the purpose of continuing a
conversation, while experimental stimulus 4 included
change and timing making it difficult to continue a
conversation. As previously mentioned (section 2.1), in thisstudy, we took fMRI measurements using experimental
stimuli 1 and 4 to compare an “easy-to-continue” and a“difficult-to-continue” conversational scenario. Reference(5) notes that, in conversational turn-taking, conversation
participants interpret intentions and behaviors (whether one
wants to talk) from a combination of facial expressions andtheir timing to determine who should speak next. It can besaid that conversational turn-taking behaviors areintegrative and combine both change and expression timing.
In order to identify the brain areas active during turn-taking,we attempted to find differences in brain activation between
the two stimuli by selecting experimental stimulus 1,
consisting of integrative turn-taking behavior, andexperimental stimulus 4, consisting of the opposite, andtaking fMRI measurements.
From the experiment results, the created behaviors arenatural and there is no problem using them as experimental
stimuli. From these findings, it was thought that, even if the
behaviors are made by a digital character, conversational
turn-taking plays its full role (effect) and we next proceeded
to fMRI measurements.

<3・1・3> Experiment conditions

3. fMRI measurements of conversational turn-taking
with characters<3・1> Experiment method<3・1・1> Changes from the questionnaire experiment
Unlike the above-mentioned questionnaire, it wasnecessary here to get a sufficient number of scans forinterpretation. Accordingly, the conversation topic was
supplemented. The supplemented lines are shown inAppendix 1. Words and sentences were selected as much as
possible so that each line would be the same length. Further,
they were thoroughly screened in advance and authors andparticipants (not including fMRI experiment participants)
removed or reworded sections that were difficult to hear,difficult to read aloud, or easily misread by the experimentparticipant. Additionally, in the experiment, the effect of anindividual character’s appearance was reduced by replacing
the character with a different one after one topic ofconversation was finished.
<3・1・2> Experiment participants
Experiment participants were eight university students(ages 20 to 23, average age 21.4; 4 male, 4 female). The
experiment contents were approved by the ethical review
board of the relevant facility (The Human Bioethical
Review Board of Tokyo Denki University). The
experiment objective, details, and warnings (risks,
protection of personal information, etc.) were fully
explained to the experiment participants using a similarly

A 1.5T superconducting magnet MRI scanner (Stratis-II)(installed at the Tokyo Denki University Chiba New Town
Campus) produced by Hitachi Medical Corporation wasused. The experiment participant, lying on their side on the
bed within the MRI machine, wore prism glasses to watch
the image of the characters depicted on a screen located bytheir feet. Experiment participants could hear the character’s
voices through a pair of non-magnetic headphones and theparticipant’s speech was conveyed to the experimenter using
a non-magnetic microphone. As with the questionnaire
experiment, the experimenter manually pressed buttons on a
keyboard while listening to the experiment participant speak.
<3・1・4> Measurement method
Measurement used a block design which divided thevarious tasks and rests of the stimuli into fixed times. Fig. 4
shows the composition of one scan while Fig. 5 shows thetasks and rests in one session. During rests, a cross-shaped
mark was presented in the center of the screen. Task stimuli
were one topic from Fig. 1 (1) to (4). A sparse imaging
method was used in which, after conversation on one topic,
there was a two-second interval followed by one threesecond scan (51). The dialogue lines used in the experimentare shown in Appendix 1. To eliminate an effect due topresentation order, the lines were presented in random orderduring the task. During one task either eight scans using
experimental stimulus 1, or eight scans using experimental
stimulus 4 were completed. Including these tasks and rests,
a total of 52 brain image scans were taken in one session. To
reduce effects from presentation order, sessions whichswitched the tasks of experimental stimulus 1 andexperimental stimulus 4 were also prepared, and three
sessions were conducted with each participant. The total
measurement time for all 3 sessions was approximately 45
minutes.

Fig.4. The composition of a topic (Utterances, aninterval and a scan)Table 2. Brain regions activated during turn-taking (uncorrected p< .001, k > 10, L=left, R=right, Stimulus 1 > Stimulus 4)

Fig.5. Structure of a session

Imaging conditions were as follows: FOV: 240mm ×240mm，TR/TE: 3000 / 50.5ms，Flip Angle: 90deg.,slice thickness 4.0mm, space between slices 1.0mm. The
EPI image Voxel size was 3.75 × 3.75 × 5.0 mm and thefull width at half maximum (FWHM) was 10mm. Imaging
was implemented in an Echo-Planar Imaging (EPI)-GE
sequence.
<3・1・5> Analysis method
Significant signals were selected based on the intensity of
the BOLD signals of the various tasks and rests. These were
mapped onto the image of a standard brain to obtain a brain
image of the experiment results. The analysis process was
as follows: (1) Realignment: Correction of disparities
caused by participant movement during the experiment. (2)Normalization: Converting the participant’s brain to a
standard brain (Talairach). (3) Smoothing: Smoothing noise
included in the imaging and improving the S/N ratio. (4)Statistics: t-test for each Voxel. Analyses used the medical
imaging analysis software SPM8(52). Activation areas weresought through group analysis using the contrast image
between the two conditions obtained for each participant. A
3D brain image indicating the hypothesized activation
pattern with an SPM{Z}Map (53)(54) was created.
<3・2> Experiment results
Fig. 6 shows the images resulting from the experiment.
Table 2 shows the areas activated when the activation datafrom experimental stimulus 4 tasks was subtracted from theactivation data from experimental stimulus 1 tasks as wellas the Brodmann area (BS), coordinates, and Z-Score(uncorrected p < .001).
The analysis results primarily demonstrated activation in
the right cerebral hemisphere. Activation in the insula andsuperior temporal gyrus (STG) of the right brain weremarked. Activation was also seen in the putamen, precentral
gyrus, around Heschl’s gyrus, and the cerebellum.

4. Discussion and Conclusions
The measurement results demonstrated activation in theright STG. This activation is believed to correspond to theabovementioned results by Stolk et al. (31)(32). The game from
Stolk et al.’s experiment involved progression through turntaking while understanding one’s own and one’s opponent’s
hand (game pieces), similar to the conversational turn-taking
in which one interacted while understanding one’s own
words and the characters’ behavior. When we look at theexperiment by Stolk et al. and the present experiment
together, it is implied that activation of the right STG isrelated to understanding the actions of the self and others.
Different from the interactions between words or between
sentences handled in past research, conversational turntaking is an interaction accompanied by mutual
understanding and it is possible that we were able to graspthat brain activity in this study.
Furthermore, significant activation was observed in theright insula in this experiment. In addition to reports that theinsula is active during measurements related to taste(55)～(57),emotion, and facial expression, it is also reported to be
involved when one is linking visual and auditory stimuli(62)(63),empathizing with another person’s pain(64)～(67), seeking an
indulgence(68)(69), or making a decision(70)(71). There are alsostudies reporting that the insula is involved in speech(72).
However, the experimental stimuli for both conditions(experimental stimulus 1 and 4) in the present experiment
included speech, and it is thought that the difference in insula
activation due to speech was cancelled out when the
difference between the two conditions was analyzed.
Accordingly, it is unlikely that the insula activation detectedin this study is related to speech.

Fig.6.Brain areas activated during turn-taking (Stimulus 1 > Stimulus 4)
As mentioned above, conversational turn-taking is thought
to be accompanied by mutual understanding, and it ispossible that the insula activation in this study was related toempathy. Previous experiments related to empathy for pain
included brain measurements when watching another person
receive an electric shock(64)(65) and empathy can be thoughtof as feeling what another is feeling as though one wasfeeling it themselves. When continuing a conversation as
well, it can be hypothesized that the participant felt
something close to “empathy,” in which they felt that “thecharacter wants to continue talking” based on theirconversational turn-taking behavior and therefore felt
similarly themselves.

References（1）

H. Sacks, E. Schegloff, and G.A. Jefferson: “A simplest systematics for theorganization of turn-taking for conversation”, Language, Vol.50, No.4,
pp.696–735 (1974)（2）

S. Duncan: “Some signals and rules for taking speaking turns inconversations”, Journal of Personality and Social Psychology, Vol.23,
pp.283–292 (1972)（3）

小磯花絵・伝 康晴：「円滑な話者交替はいかにして成立するか—
会話コーパスの分析にもとづく考察—」，認知科学，Vol.7, No.1,
pp.93–106, (2000)

（4）

伝 康晴：
「受け手になること，次話者になること - 話者交替再考」，
人工知能学会研究会資料，SIG-SLUD-A203, pp.107–112 (2003)

（5）

（6）

（7）

徳永弘子・武川直樹・寺井 仁・湯浅将英・大和淳司：「会話における参与役割志向態度に基づく順番交替分析：—人の「話したい」「聞きたい」態度が次話者を決定する仕組み」，知能と情報，Vol.25,
No.5, pp.889–900 (2013)
K. Otsuka, J. Yamato, Y. Takemae, and H. Murase: “Quantifying interpersonal
influence in face-to-face conversations based on visual attention patterns”, in
Proc. ACM CHI Extended Abstract, pp.1175–1180 (2006)
木村幸士・湯浅将英・武川直樹：「多人数エージェントによる会話の雰囲気生成—文字ばっかり読んでないで空気読め—」，HAI シンポジウム 2007, 1E-3 (2007)（8）

湯浅将英・武川直樹・徳永弘子：「発話交替エージェントによる発話
義務と発話権利の表現—「話すべき」「話したい」による社会的コミュニケーションモデルの提案—」，HAI シンポジウム 2011 (2011)（9）

湯浅将英，木村幸士，武川直樹，寺井 仁：「多人数エージェントの発話交替により現れる雰囲気の分析～気まずい会話・仲良し会話～」，信学会 HCS2010-17，pp.91–96 (2010)（10） S.A. Kotz, S.F. Cappa, D.Y. vonCramon, and A.D. Friederici: “Modulation of
the lexical–semantic network by auditory semantic priming: An eventrelated
functional mri study”, Neuroimage, Vol.17, No.4, pp.1761–1772 (2002)（11） J. Rissman, J.C. Eliassen, and S.E. Blumstein: “An event-related fmri
investigation of implicit semantic priming”, Journal of Cognitive
Neuroscience, Vol.15, No.8, pp.1160–1175 (2003)（12） D.R. Gitelman, A.C. Nobre, S. Sonty, T.B. Parrish, and M.-M. Mesulam:“Language network specializations: an analysis with parallel task designs andfunctional magnetic resonance imaging”, Neuroimage, Vol.26, No.4, pp.975–985 (2005)（13） J. Obleser and S.A. Kotz: “Expectancy constraints in degraded speech
modulate the language comprehension network”, Cereb Cortex, Vol.20, No.3,
pp.633–40 (2010)（14） W. Ni, R.T. Constable, W.E. Mencl, K.R. Pugh, R.K. Fulbright, S.E. Shaywitz,
B.A. Shaywitz, J.C. Gore, and D. Shankweiler: “An event-related
neuroimaging study distinguishing form and content in sentence processing”,
Journal of Cognitive Neuroscience, Vol.12, No.1, pp.120–133 (2000)（15） R. Caplan and M. Dapretto: “Making sense during conversation: an fmristudy”, Neuroreport, Vol.12, No.16, pp.3625–3632 (2001)（16） E.C. Ferstl and D.Y. vonCramon: “What does the frontomedian cortex
contribute to language processing: Coherence or theory of mind?”,
NeuroImage, Vol.17, pp.1599–1612 (2002)（17） F.T. Sieborger, E.C. Ferstl, and D.Y. vonCramon: “Making sense of nonsense:an fmri study of task induced inference processes during discourse
comprehension”, Brain Research, Vol.1166, pp.77–91 (2007)（18） J.R. Binder, R.H. Desai, W.W. Graves, and L.L. Conant: “Where is thesemantic system? a critical review and meta-analysis of 120 functional
neuroimaging studies”, Cerebral Cortex, Vol.19, No.12, pp.2767–2796 (2009)（19） A.D. Friederici: “The cortical language circuit: from auditory perception tosentence comprehension”, Trends in cognitive sciences, Vol.16, No.5,
pp.262–268 (2012)（20） K. Rothermich and S.A. Kotz: “Predictions in speech comprehension: fmri
evidence on the meter-semantic interface”, NeuroImage, Vol.70, No.15,
pp.89–100 (2013)（21） E.F. Lau, C. Phillips, and D. Poeppel: “A cortical network for semantics:(de)constructing the n400”, Nature Reviews Neuroscience, Vol.9, pp.920–
933 (2008)（22） C. Whitney, E. Jefferies, and T. Kircher: “Heterogeneity of the left temporal
lobe in semantic representation and control: priming multiple versus single
meanings of ambiguous words,” Cerebral Cortex, Vol.21, pp.831–844 (2011)（23） G. Jobard, F. Crivello, and N. Tzourio-Mazoyer: “Evaluation of the dual route
theory of reading: a metanalysis of 35 neuroimaging studies”, NeuroImage,Vol.20, pp.693–712 (2003)（24） E.C. Ferstl, M. Rinck, and D.Y. vonCramon: “Emotional and temporal aspectsof situation model processing during text comprehension: an event-related
fmri study”, Journal of Cognitive Neuroscience, Vol.17, No.5, pp.724–739(2005)（25） E. Maguire, C. Frith, and R. Morris: “The functional neuroanatomy ofcomprehension and memory: the importance of prior knowledge”, Brain,
Vol.122, pp.1839–1850 (1999)（26） G.J. Stephens, L.J. Silbert, and U. Hassonc: “Speaker-listener neural couplingunderlies successful communication”, Academy of Sciences of the UnitedStates of America, Vol.107, No.32, p.14425–14430 (2010)（27） S. Bookheimer: “Functional mri of language: New approaches tounderstanding the cortical organization of semantic processing”, Annual
Review of Neuroscience, Vol.25, pp.151–188 (2002)（28） K.D. Federmeier: “Thinking ahead: The role and roots of prediction inlanguage comprehension”, Psychophysiology, Vol.44, pp.491–505 (2007)（29） M. Jung-Beeman: “Bilateral brain processes for comprehending natural
language”, Trends in Cognitive Sciences, Vol.9, pp.512–518 (2005)（30） K.S. Laura Menenti, S.M.E. Gierhan, and P. Hagoort: “Shared language:
Overlap and segregation of the neuronal infrastructure for speaking andlistening revealed by functional mri,” Psychological Science, Vol.22, No.9,
pp.1173–1182 (2011)（31） A. Stolk, L. Verhagen, J.-M. Schoffelen, R. Oostenveld, M. Blokpoel, P.
Hagoort, I. vanRooij, and I. Toni: “Neural mechanisms of communicative
innovation”, Proceedings of the National Academy of Sciences of the UnitedStates of America, Vol.110, No.36, pp.14574–14579 (2013)（32） A. Stolk, M.L. Noordzij, L. Verhagen, I. Volman, J.-M. Schoffelen, R.
Oostenveld, P. Hagoort, and I. Toni: “Cerebral coherence between
communicators marks the emergence of meaning”, Proceedings of the
National Academy of Sciences of the United States of America, Vol.111,
No.51, pp.18183–18188 (2014)（33） 藤巻則夫・早川友恵：「言語処理の脳活動及び解析方法」，情報通信研究機構季報，Vol.50, No.3, pp.45–62 (2004)（34） S. Garrod and M.J. Pickering: “Why is conversation so easy?”, Trends in
Cognitive Sciences, Vol.8, No.1, pp.8–11 (2004)（35） L. Menenti, M.J. Pickering, and S.C. Garrod: “Toward a neural basis ofinteractive alignment in conversation”, Frontiers in Human Neuroscience,
Vol.6, No.185 (2012)（36） J.P. Morris, K.A. Pelphrey, and G. McCarthy: “Regional brain activation
evoked when approaching a virtual human on a virtual walk”, Journal of
Cognitive Neuroscience, Vol.17, No.11, pp.1744–1752 (2005)（37） K.A. Pelphrey, R.J. Viola, and G. McCarthy: “When strangers pass processing of mutual and averted social gaze in the superior temporal sulcus-”,
Psychological Science, Vol.15, No.9, pp.598–603 (2004)（38） L. Schilbach, A.M. Wohlschlaeger, N.C. Kraemer, A. Newen, N.J. Shah, G.R.
Fink, and K. Vogeley: “Being with virtual others: Neural correlates of social
interaction”, Neuropsychologia, Vol.44, No.5, pp.718–730 (2006)（39） I. Benbasat, A. Dimoka, P.A. Pavlou, and L. Qiu: “Incorporating social
presence in the design of the anthropomorphic interface of recommendation
agents: Insights from an fmri study”, Proc. of International Conference on
Information Systems (ICIS), p.228 (2010)（40） M.J. Tarr and W.H. Warren: “Virtual reality in behavioral neuroscience andbeyond”, Nature Neuroscience, Vol.5, pp.1089–1092 (2002)（41） R. Adolphs: “Cognitive neuroscience of human social behaviour”, Nature
Reviews Neuroscience, Vol.4, pp.165–178 (2003)（42） M.V. Sanchez-Vives and M. Slater: “From presence to consciousness throughvirtual reality”, Nature Reviews Neuroscience, Vol.6, pp.332–339 (2005)（43） 榎本美香：「多人数インタラクションの多様性とダイナミズム－多人
数インタラクションでは何が多くなるのか？」，社会言語科学，Vol.9,
No.2, pp.154–158 (2007)（44） NHK 放送技術研究所：TVML プレイヤー（45） 平谷淳一・林 正樹：「ユーザーカスタマイズ可能なテレビの検討」，
情報処理学会全国大会講演予稿集，8U-04 (2001)（46） 林 正樹・折原 豊・下田 茂・上田博唯・横山敏明・八重樫一仁・栗原恒弥・安村通晃：「テレビ番組記述言語 TVML の言語仕様と CG
記述方法」，第 3 回知能情報メディアシンポジウム，pp.75–80 (1997)（47） 長岡千賀・小森政嗣・M.R. Draguna・河瀬 諭・結城牧子・片岡智嗣・中村敏枝：「協調的対話における音声行動の 2 者間の一致–意見固持型対話と聞き入れ型対話の比較–」，ヒューマンインタフェース
シンポジウム 2013 論文集，pp.167–170 (2004)（48） C. Trimboli and M.B. Walker: “Switching pauses in cooperative andcompetitive conversations”, Journal of Experimental Social Psychology,
Vol.20, No.4, pp.297–311 (1984)

（49） M. Yuasa, N. Mukawa, K. Kimura, H. Tokunaga, and H. Terai: “An utterance
attitude model in human-agent communication: From good turntaking tobetter human-agent understanding”, Proc. of CHI Extended Abstracts 2010,
pp.3919–3924 (2010)（50） 榎本美香・伝 康晴：「3 人会話における参与役割の交替に関わる非言語行動の分析」，人工知能学会研究会資料，Vol.SIG-SLUD-A301，
pp.25–30 (2003)（51） D.A. Hall, M.P. Haggard, M.A. Akeroyd, A.R. Palmer, A.Q. Summerfield,
M.R. Elliott, E.M. Gurney, and R.W. Bowtell, ““sparse” temporal samplingin auditory fmri”, Human Brain Mapping, Vol.7, pp.213–223 (1999) （52）
SPM, “http://www.fil.ion.ucl.ac.uk/spm/”.
（53） 石井一成：「脳機能画像統計解析—spm と 3d-ssp—」，最新医学，
Vol.60,No.5, pp.980–987 (2005)
河村誠治・上野雄文：「脳核医学および fmri における統計学的画像解析法」，日本放 射線技術 学会雑 誌，Vol.59, No.5, pp.594–603(2003)（55）
B. Cerf-Ducastel, P.-F.V. deMoortele, P. MacLeod, D.L. Bihan, and A.
Faurion: “Interaction of gustatory and lingual somatosensory perceptions
at the cortical level in the human: a functional magnetic resonance imaging
study”, Chemical senses, Vol.26, No.4, pp.371–383 (2001)（56）
K. Rudenga, B. Green, D. Nachtigal, and D. Small: “Evidence for an
integrated oral sensory module in the human anterior ventral insula”,
Chemical senses, Vol.25, No.8, pp.693–703 (2010)（57）
M. Saito, H. Toyoda, S. Kawakami, H. Sato, Y. Bae, and Y. Kang:“Capsaicin induces theta-band synchronization between gustatory andautonomic insular cortices”, The Journal of Neuroscience, Vol.32, No.39,
pp.13470– 13487 (2012)（58）
A.R. Damasio, T.J. Grabowski, A. Bechara, H. Damasio, L.L. Ponto, J.
Parvizi, and R.D. Hichwa: “Subcortical and cortical brain activity during
the feeling of self-generated emotions”, Nature neuroscience, Vol.3, No.10,
pp.1049–1056 (2000)（59） A. Craig: “Interoception: the sense of the physiological condition of thebody”, Current opinion in neurobiology, Vol.13, No.4, pp.500–505 (2003)（60） N. Tsuchiya and R. Adolphs: “Emotion and consciousness”, Trends in
Cognitive Sciences, Vol.11, No.4, pp.158–167 (2007)（54）（61） 寺澤悠理・梅田 聡・斎藤文恵・加藤元一郎：「右島皮質損傷によってネガティブ表情の識別に混乱を示した一例」，高次脳機能研究，
Vol.30, No.2, pp.349–358 (2010)（62） K.O. Bushara, J. Grafman, and M. Hallett: “Neural correlates ofauditoryvisual stimulus onset asynchrony detection”, The Journal of
Neuroscience, Vol.21, No.1, pp.300–304 (2001)（63） K.O. Bushara, T. Hanakawa, I. Immisch, K. Toma, K. Kansaku, and M.
Hallett: “Neural correlates of cross-modal binding”, Nature Neuroscience,
Vol.6, No.2, pp.190–195 (2003)（64） 有田秀穂：「他者の痛みに共感する」，pp.14–17，中外医学社，人間性のニューロサイエンス–前頭前野，帯状回，島皮質の生理学–(2011)（65） CA. Brown, B. Seymour, W. El-Deredy, and A.K. Jones: “Confidence inbeliefs about pain predicts expectancy effects on pain perception andanticipatory processing in right anterior insula”, Pain, Vol.139, No.2,
pp.324–332 (2008)（66） O. FeldmanHall, T. Dalgleish, R. Thompson, D. Evans, S. Schweizer, and
D. Mobbs: “Differential neural circuitry and self-interest in real vshypothetical moral decisions”, Social Cognitive and Affective
Neuroscience, Vol.7, No.7, pp.743–751 (2012)（67） T. Singer, B. Seymour, O’Doherty J., H. Kaube, R.J. Dolan, and C.D. Frith:“Empathy for pain involves the affective but not sensory components ofpain”, Science, Vol.303, pp.1157–1162 (2004)（68） A. Brody, M. Mandelkern, E. London, A. Childress, G. Lee, R. Bota, M.
Ho, S. Saxena, L.J. Baxter, D. Madsen, and M. Jarvik: “Brain metabolicchanges during cigarette craving”, Archives of general psychiatry, Vol.59,
No.12, pp.1162–1172 (2002)（69） H. Myrick, R.F. Anton, X. Li, S. Henderson, D. Drobes, K. Voronin, and
M.S. George: “Differential brain activity in alcoholics and social drinkers
to alcohol cues: Relationship to craving”, Neuropsychopharmacology,
Vol.29, No.2, pp.393–402 (2004)（70） M. Paulus, C. Rogalsky, A. Simmons, J. Feinstein, and M. Stein:“Increased activation in the right insula during risk-taking decision making
is related to harm avoidance and neuroticism”, Neuroimage, Vol.19, No.4,
pp.1439– 1448 (2003)（71） L. Clark, A. Lawrence, F. Astley-Jones, and N. Gray: “Gambling
nearmisses enhance motivation to gamble and recruit win-related brain
circuitry”, Neuron, Vol.61, No.3, pp.481–490 (2009)（72） H. Ackermann and A. Riecker: “The contribution(s) of the insula to speech
production: a review of the clinical and functional imaging literature”,
Brain Structure and Function, Vol.214, No.5, pp.419–433 (2010)（73） 小谷賢太郎：「人間特性評価論文をどう書くべきか」，ヒューマン
インタフェース学会誌，Vol.13, No.3, pp.147–150 (2011)