Automatic Detection of Very Early Stage of Dementia
through Multimodal Interaction with Computer Avatars
Hiroki Tanaka

Nara Institute of Science andTechnology, Japan

Hiroyoshi Adachi

Osaka University Health Care
Center, Japan

Norimichi Ukita

Nara Institute of Science andTechnology, Japan

hadachi@psy.med.osakaukita@is.naist.jp
u.ac.jpTakashi Kudo
Satoshi Nakamura

hiroki-tan@is.naist.jp

Osaka University Health Care
Center, Japan

kudo@psy.med.osakau.ac.jp

Nara Institute of Science andTechnology, Japans-nakamura@is.naist.jp

ABSTRACT

lenging, especially in its very early stages [21]. Currently,
there is no powerful tool that gives a reliable detection ofdementia: rather, the patient has to go through a series ofcognitive tests conducted by a professional neurologist forassessments. This process involves a certain amount of anxiety and stress. Especially in the case of the very early stage
detection, complementary tests include the analysis of samples of cerebrospinal fluid taken from the brain, a magnetic
resonance brain imaging test, and a blood test [11]. Suchmethods are invasive, bring discomfort to the participants,
are relatively costly and require a significant amount of eﬀort
and time. Thus, there is an increasing need for additional
noninvasive and/or cost-eﬀective tools, allowing identification of participants in the preclinical or early clinical stagesof dementia.
As noninvasive and cost-eﬀective approaches, previous works
have attempted to detect dementia from their speech andlanguage attributes [8, 12, 5, 18, 1]. For example, Aramaki et al., [1] reported that mild cognitive impairments
tend to speak less and use easier words than healthy control. However, there is no existing powerful tool to detect
dementia from audiovisual features, and most of them used
non-interactive data such as picture description, narrative,
and cognitive tasks. In contrast, there was a study that applies interactive computer avatars with spoken dialogue todetect user’s social behaviors [23].
This paper proposes a new approach to detect very earlystage of dementia automatically. We develop a computeravatar with spoken dialog functionalities that produces simple natural spoken queries referring to cognitive tests suchas Mini Mental State Examination (MMSE) and Wechsler
Memory Scale-Revised (WMS-R) as well as other related
medical questions [7].

This paper proposes a new approach to detecting very earlystage of dementia automatically. We develop a computeravatar with spoken dialog functionalities that produces natural spoken queries referring to Mini Mental State Examination, Wechsler Memory Scale-Revised and other related
questions. Multimodal interactive data of spoken dialogues
from 18 participants (9 dementias and 9 healthy controls)
are recorded, and audiovisual features are extracted. We
confirm that the support vector machines can classify into
two groups with 0.94 detection performance as measured byareas under ROC curve. It is found that our system has
possibilities to detect very early stage of dementia throughspoken dialog with our computer avatars.

CCS Concepts•Applied computing → Health care information systems;

Keywords
Dementia, spoken dialogue, avatars

1. INTRODUCTION
Dementia is broadly defined as deterioration in memory,
thinking, and behavior that decreases a person’s ability tofunction independently [14]. Early diagnosis of dementia is
important for several reasons. The most important reason isthat early diagnosis allows the patient and family to plan for
the future and identify outside sources of assistance. Moreover, as potentially useful and proven treatments become
available, early diagnosis of dementia will become increasingly important [19]. The early detection of dementia is chal-

2. RELATED WORK
Several works have attempted to detect or diagnose dementia from speech and language features, and have demonstrated the potential of the approaches to identifying dementia.
Orimaye et al [16] proposed a diagnostic method to identify people with Alzheimer’s disease using a large numberof language features extracted from transcribed audio files

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from Permissions@acm.org.

ICMI’16, November 12–16, 2016, Tokyo, Japanc 2016 ACM. 978-1-4503-4556-9/16/11...$15.00
http://dx.doi.org/10.1145/2993148.2993193

261

from the DementiaBank dataset1 . They used 242 sample
files for both healthy controls and people with Alzheimer’s
disease. They compared five diﬀerent machine learning algorithms, achieving a 74% classification accuracy using a
support vector machine (SVM) classifier with 10% crossvalidation. Konig et al [10] performed an experiment of using four cognitive vocal tasks (a counting backward task,a sentence repeating task, an image description task, anda verbal fluency task) with participants divided into threegroups: healthy controls, people with mild cognitive impairment, and people with Alzheimer’s disease. They extracted
features from the audio recordings. They achieved to classify
into healthy controls and mild cognitive impairment with anaccuracy of 79%, healthy controls and Alzheimer’s disease
with an accuracy of 87%. Recently and achieving higher
accuracy, Fraser et al [5] studied the potential of using language features to identify Alzheimer’s disease. They used
speech recordings along with their manually transcribed files
derived from the DementiaBank dataset. They selected 240
speech recordings with a set of 370 speech and language features. Then, they applied two machine learning algorithms
and obtained a highest accuracy of 92% in distinguishing
between healthy controls and Alzheimer’s disease.
There have been a small number of papers examining facial expression of dementia as we surveyed so far. For example, the ability to exhibit facial expressions was studiedin four patients with severe dementia of the Alzheimer’s disease, by means of the Facial Action Coding System (FACS)
under pleasant and unpleasant stimulus conditions. The results showed that some types of dementia tend to mute facial
expression [2].
This paper proposes a new approach to detect very earlystage of dementia automatically. We develop an interactive
computer avatar with spoken dialog functionalities that produces natural spoken queries.

(b) Gaze: The system displays a small dot on a computer
screen, and users are directed to gaze the moving dot.
(c) Reading: The system displays a document of Wechsler
Logical Memory I (immediate) task in the WMS-R [24,18], and users read aloud the sentence.
(d) Fixed Q&A: The system asks a total of three fixed
queries as referring to the MMSE [4]. It consists of1) What is the date today?, 2) What is your memorable story?, 3) How did you come here today?
(e) Random Q&A: The five queries are randomly produced.
A total of 13 questions were prepared for random questions such as “Do you have any appetite?,”“Please tellme about Shigeo Nagashima” (who is a famous baseball player in Japan), as referring to e.g. [7].
(f ) Retelling: The system read aloud a document of a different part of the Wechsler Logical Memory, and users
retell the sentence.
During interaction with a computer avatar, the system
records user’s video and audio using built-in camera and microphone. The system waits 10 seconds after a user’s finalutterance for closed / easy to answer questions, and waits
15 seconds for diﬃcult questions to go on to a next question. Two graduate students evaluated load and diﬃcultyof each task. Then, we ordered tasks according to its loadand diﬃculty (ascending order). A total amount of time tocomplete all dialogues is around 10-15 minutes.

4. EXPERIMENTAL EVALUATIONIn this section, we represent an experimental evaluation
using the system.

4.1 User Study

3. INTERACTIVE SYSTEM
We used MMDAgent2 as a computer avatar. The proposed system is a Japanese spoken dialogue integrating speech
recognition, dialogue management, text-to-speech, and behavior generation. MMDAgent works as a Windows application. The system was adapted to elderly people by displaying subtitles and slower speaking rate. This development
process was conducted in discussion with a professional psychiatrist. We selected an animated female character who
is similar to an actual human, as opposed to an animallike character, but not more realistic human-like one, as wehope that this will make the conversation interesting. Whenthe system recognizes an utterance, after a few seconds thesystem nods its head. The nodding behavior motions werecreated by MikuMikuDance3 .
We prepared a total of 6 continuous dialogue procedures
to detect very early stage of dementia, and summarize themas follows (in Figure 1):(a) Self-introduction: The system introduces herself andasks user’s name and age. This process is to makeusers comfort to interact.
1

https://talkbank.org/DementiaBank/
http://www.mmdagent.jp/
3
http://www.geocities.jp/higuchuu4/2

262

We recruited a total of 20 participants. Each participant
gave informed consent before the data recording. 10 participants (9 males and 1 female) were recorded at Osaka University Hospital as dementia group, and other 10 participants(7 males and 3 females) were recorded at Nara Institute of
Science and Technology as healthy controls. Participantsof the dementia group were diagnosed as very early stageof dementia by expert clinicians at the Osaka University
Hospital according to Diagnostic and Statistical Manual of
Mental Disorders, 4th. Edition (DSM-IV). We collected information on age and the MMSE score from all participants.
We used a laptop (Surface Pro 3) to record interaction, andconfirmed 20 participants could complete all procedures ofthe dialogue as mentioned in the previous section. An amplitude of microphone and distance between users and thelaptop is consistent as long as possible in the two separate
locations. One person manually transcribed the recordeddata.
As shown in Table 1, we finally used audiovisual dataof 18 participants. Two participants were removed fromthe experiment because one in the dementia group did notreceive a diagnosis of dementia yet and one in the healthy
controls obtained 22 of the MMSE score (below the cut-oﬀ
score). The MMSE score was significantly diﬀerent between
two groups by using two-tailed Student’s t-test (p < .05),
and age was matched (p > .05).

Anna Thompson of South Boston,
employed as a cook in a school cafeteria,
reported at the police station that she had
been held up on State Street the night
before and robbed of fifty-six dollars. She
had four small children, the rent was due,and they had not eaten for two days. The
police, touched by the woman’s story, took
up a collection for her.

(a) Self-introduction
Self-introduction
Sub:
What
is yourname?
“Whatis yourname?”(b) Gaze(c) Reading(f) Retelling
Retelling(f)“Pleasefollowinga story.
sentences”
Sub:remember
Please remember(e) Random
Random questions
questions
(e)
“DoDo
you
have
any
appetite?”
Sub:
you
have
any
appetite?

(d) Fixed questions“What
is monthSub:
Whatis the anddatedate?”
today?

Figure 1: Dialogue procedures. Japanese sentences and subtitles are translated into English (Sub).
utterances. We extracted the number of tokens (Tokens) as
well as the ratio of fillers (Fillers) (e.g. “umm,” or “eh” inJapanese) from an output of the Mecab. Type-token ratio(TTR) represents the ratio of the total vocabulary to theoverall words, and is a simple measure of vocabulary size.
We extracted TTR as referring to [3] as follows.

Table 1: Participant demographics with mean andSD values of age and the MMSE score.
Group
Dementia
Controls

N
9
9

Age
76.4 (7.4)
73.9 (4.5)

MMSE
21.4 (1.6)
26.4 (5.7)

TTR =4.2 Feature Set

4.2.3

For speech feature extraction, we used the Snack soundtoolkit4 . Here, we considered fundamental frequency, power,speech rate, and voice quality. We did not extract meanvalues of fundamental frequency because those features arestrongly related to individuality. Thus, we extracted statistics of coeﬃcient of variation (F0cov) for fundamental frequency. We extracted mean value of power (Power), andspeech rate (SR), which is a feature dividing the numberof words by the number of voiced seconds. Voice quality
is also computed using the diﬀerence between the first harmonic (h1) and the third formant (a3) (h1a3) which definedby [6] as follows.

P ause = ta − tq .

4.2.4

(3)

Visual Features

We extracted the ratio of smiling as a facial expression
feature. To analyze the recorded video, fist we extracted anumber of facial features by using a constrained local model[20] based face tracker7 . From a total of 66 feature points,
we calculated the features of outer eye-brow height, inner
eyebrow height, outer lip height, inner lip height, eye opening, and lip corner distance following Naim et al. [15]. Using
these features, we modeled smiling faces using the Japanese
female facial expression database [13]. For video, we predicted whether the label belongs to the smiling or neutralclass in each frame, and the proportion of smiling frames(1)

4.2.2 Language Features

We used Mecab5 for part-of-speech tagging in Japanese
5Interactive Features

We extracted pauses before new turns (Pause). Here, wedenote values of pauses before new turns as a time between
the end of the avatar’s question (tq ) and the start of theuser’s answer (ta ). A maximum waiting time is 10 or 15
seconds in the system, and thus the feature ranges between
0 and 15 as follows.

4.2.1 Acoustic Features4(2)

We also computed word diﬃculty as the ratio of nouns
with above intermediate level to the overall nouns (Diﬃculty). We used word diﬃculty dictionary6 to obtain nouns
with above intermediate level.

We extracted acoustic, linguistic, interactive, and visualfeatures from videos of the answers’ speech of the three fixed
queries. We averaged each feature of the three answers.
We selected user-independent and insightful audiovisual features based on previous works which found diﬀerences between psychiatric disorders and early stage of dementia, andhealthy controls [23, 9, 1]. Features were compared between
groups by using Student’s t-test (two-tailed).

h1a3 = h1 − a3.

number of types.
number of tokens

6

http://www.speech.kth.se/snack/http://taku910.github.io/mecab/

7263

http://jreadability.net/
https://github.com/kylemcdonald/FaceTracker

4.3 Classifier

0.6
0.0

0.6

0.8

1.0

rectly detect one person in the dementia group. This isbecause the person spoke the largest number of tokens andwas relatively early to answer the questions in the dementia
group.

5. CONCLUSION

Table 2: Feature ranking sorted by p-values. The
fifth column (Trend) shows the direction of the trend(increasing or decreasing) of the dementia group.
Bold fonts indicate significant diﬀerences (p < 0.05).
Cohen’s d
1.69
1.31
1.15
1.17
0.99
0.92
0.88
0.64
0.42
0.08

0.4

Figure 2: ROC curve. Red and blue lines show true
positive rate and false positive rate of SVM and Logistic regression respectively.

First, we sorted each feature by their p-values of t-test as
well as Cohen’s d values as shown in Table 2.

p-value
0.003
0.017
0.028
0.031
0.067
0.076
0.079
0.195
0.383
0.867

0.2

False positive rate

4.4 Results

Feature
Pause
h1a3
TTR
Tokens
Smile
F0cov
Power
Diﬃculty
Fillers
SR

SVM
Logistic

0.0

We used two machine learning algorithms for detecting
very early stage of dementia from healthy controls. SVM
with a sigmoid kernel and logistic regression were compared
as classifiers. In this experiment, we used all audiovisual features as an input of the classifiers, and the classifiers trained
to predict the label belonging to the dementia group andthe healthy controls. We evaluated a classification performance with leave-one-subject-out cross-validation, and plotted ROC curve with areas under ROC curve (AUC) [18].

Rank
1
2
3
4
5
6
7
8
9
10

0.4
0.2

True positive rate

0.8

1.0

among all frames was named the ratio of smiling (Smile).
To verify that the model generalizes to other speakers andvideo, we used the NOCOA+ database [22], which contains
derisive and friendly videos of four Japanese men. We extracted the ratio of smiling from these videos and confirmedthat friendly videos have a significantly larger ratio of smiling compared to derisive videos (p=0.002). Because the system did not record images below the chest, we did not take
into account non-facial gestures.

We developed a computer avatar with spoken dialog functionalities that produces natural spoken queries. As the initial analysis, 18 participants were recorded and audiovisualfeatures were extracted. The results of the analyses showedthat several features were eﬀective to distinguish between
very early stage of dementia and healthy controls. It wasalso confirmed that SVM can classify two groups with 0.94
detection performance as measured by AUC. We found thatthe system has possibilities to detect very early stage of dementia through spoken dialog with a computer avatar.
This work is a first attempt to detect dementia throughmultimodal interaction with computer avatars. However,the several modules are not adapted to elderly people. For
example, to achieve more precise prediction of facial expressions, we have to train facial points model using data ofelderly people. Also, this work is limited in terms of 1)
the number of participants, 2) feature set, and 3) question
set. First, we plan to collect a large number of patients atthe Hospital as well as age-matched controls. Second, multimodal integration should be considered such as eye gaze
tracking, actual user responses, and richer linguistic features (e.g. syntax complexity measures [17]). Third, weshould take other question sets or actual user responses into
consideration incorporating with automatic speech recognition (ASR). For other future directions, we plan to examine
diﬀerent types of dementia [8].

Trend
↑
↑
↑
↓
↑
↑
↓
↓
↓
↓

The pause was significantly diﬀerent between two groups(p=0.003), indicating patients with very early stage of dementia tend to delay responses to the system than the healthy
controls. The other features such as the h1a3, the TTR,
the Tokens were also significantly diﬀerent (p=0.028). Here,the TTR was not consistent with the previous work, whichshowed that most of the healthy controls had lexically richer
speech than dementia participants [3]. This was caused because our data was limited in terms of the number of tokens,especially in the dementia group. Rest of features were notsignificantly diﬀerent (p > .05), but have possibilities to contribute to further classification. These results showed that
our small number of features were eﬀective to distinguishbetween the dementia group and the healthy controls.
Figure 2 shows ROC curve, and we confirmed that machine learning algorithms classified two groups with 0.94 for
SVM and 0.91 for logistic regression as measured by AUC.
This result indicates a high detection performance compared
to previous works [12, 5, 18]. The classifiers could not cor-

6. ACKNOWLEDGMENTS
This work is supported by Yanmar Lab 2112.

264

7. REFERENCESINTERSPEECH, pages 1039–1042, 2012.
[13] M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba.
Coding facial expressions with gabor wavelets. In
Proceedings of Third IEEE International Conference
on Automatic Face and Gesture Recognition, pages
200–205. IEEE, 1998.
[14] G. M. McKhann, D. S. Knopman, H. Chertkow, B. T.
Hyman, C. R. Jack, C. H. Kawas, W. E. Klunk, W. J.
Koroshetz, J. J. Manly, R. Mayeux, et al. The
diagnosis of dementia due to alzheimer’s disease:
Recommendations from the national institute on
aging-alzheimer s association workgroups on
diagnostic guidelines for alzheimer’s disease.
Alzheimer’s & dementia, 7(3):263–269, 2011.
[15] I. Naim, M. I. Tanveer, D. Gildea, et al. Automatedanalysis and prediction of job interview performance.
arXiv preprint arXiv:1504.03425, 2015.
[16] S. O. Orimaye, J. S.-M. Wong, and K. J. Golden.
Learning predictive linguistic features for alzheimer’s
disease and related dementias using verbal utterances.
In Proceedings of the 1st Workshop on Computational
Linguistics and Clinical Psychology (CLPsych), pages
78–87, 2014.
[17] B. Roark, M. Mitchell, and K. Hollingshead. Syntactic
complexity measures for detecting mild cognitive
impairment. In Proceedings of the Workshop on
BioNLP 2007: Biological, Translational, and ClinicalLanguage Processing, pages 1–8. Association for
Computational Linguistics, 2007.
[18] B. Roark, M. Mitchell, J.-P. Hosom, K. Hollingshead,
and J. Kaye. Spoken language derived measures for
detecting mild cognitive impairment. Audio, Speech,
and Language Processing, IEEE Transactions on,
19(7):2081–2090, 2011.
[19] K. S. Santacruz and D. Swagerty. Early diagnosis ofdementia. Am Fam Physician, 63(4):703–713, 2001.
[20] J. M. Saragih, S. Lucey, and J. F. Cohn. Facealignment through subspace constrained mean-shifts.
In IEEE 12th International Conference on Computer
Vision, pages 1034–1041, 2009.
[21] V. Taler and N. A. Phillips. Language performance inalzheimer’s disease and mild cognitive impairment: acomparative review. Journal of Clinical andExperimental Neuropsychology, 30(5):501–556, 2008.
[22] H. Tanaka, S. Sakti, G. Neubig, T. Toda, and
S. Nakamura. Nocoa+: Multimodal computer-based
training for social and communication skills. IEICE
Transaction on Information and Systems,
E98-D(8):1536–1544, 2015.
[23] H. Tanaka, S. Sakti, G. Neubig, T. Toda, H. Negoro,
H. Iwasaka, and S. Nakamura. Automated social skills
trainer. In Proceedings of the 20th International
Conference on Intelligent User Interfaces, pages
17–27. ACM, 2015.
[24] D. Wechsler. WAIS-III: Administration and scoring
manual: Wechsler adult intelligence scale.
Psychological Corporation, 1997.

[1] E. Aramaki, S. Shikata, M. Miyabe, and A. Kinoshita.
Vocabulary size in speech may be an early indicator of
cognitive impairment. PloS one, 11(5):e0155195, 2016.
[2] K. Asplund, A. Norberg, R. Adolfsson, and H. M.
Waxman. Facial expressions in severely demented
patients―a stimulus–response study of four patientswith dementia of the alzheimer type. International
Journal of Geriatric Psychiatry, 6(8):599–606, 1991.
[3] R. Bucks, S. Singh, J. M. Cuerden, and G. K.
Wilcock. Analysis of spontaneous, conversational
speech in dementia of alzheimer type: Evaluation ofan objective technique for analysing lexical
performance. Aphasiology, 14(1):71–91, 2000.
[4] M. F. Folstein, S. E. Folstein, and P. R. McHugh.
“mini-mental state”: a practical method for grading
the cognitive state of patients for the clinician.
Journal of psychiatric research, 12(3):189–198, 1975.
[5] K. C. Fraser, J. A. Meltzer, and F. Rudzicz. Linguisticfeatures identify alzheimer’s disease in narrative
speech. Journal of Alzheimer’s Disease, 49(2):407–422,
2015.
[6] M. Hanson. Glottal characteristics of female speakers.
Harvard University. PhD thesis, Ph. D. dissertation,
1995.
[7] R. M. Harden and F. Gleeson. Assessment of clinicalcompetence using an objective structured clinicalexamination (osce). Medical education, 13(1):39–54,
1979.
[8] W. Jarrold, B. Peintner, D. Wilkins, D. Vergryi,
C. Richey, M. L. Gorno-Tempini, and J. Ogar. Aided
diagnosis of dementia type through computer-based
analysis of spontaneous speech. In Proceedings of the
ACL Workshop on Computational Linguistics andClinical Psychology, pages 27–36, 2014.
[9] S. Kato, H. Endo, A. Homma, T. Sakuma, andK. Watanabe. Early detection of cognitive impairment
in the elderly based on bayesian mining using speechprosody and cerebral blood flow activation. In 2013
35th Annual International Conference of the IEEE
Engineering in Medicine and Biology Society (EMBC),
pages 5813–5816. IEEE, 2013.
[10] A. König, A. Satt, A. Sorin, R. Hoory,
O. Toledo-Ronen, A. Derreumaux, V. Manera,
F. Verhey, P. Aalten, P. H. Robert, et al. Automatic
speech analysis for the assessment of patients withpredementia and alzheimer’s disease. Alzheimer’s &Dementia: Diagnosis, Assessment & Disease
Monitoring, 1(1):112–124, 2015.
[11] C. Laske, H. R. Sohrabi, S. M. Frost, K. López-de
Ipiña, P. Garrard, M. Buscema, J. Dauwels, S. R.
Soekadar, S. Mueller, C. Linnemann, et al. Innovative
diagnostic tools for early detection of alzheimer’s
disease. Alzheimer’s & Dementia, 11(5):561–578, 2015.
[12] M. Lehr, E. T. Prud’hommeaux, I. Shafran, andB. Roark. Fully automated neuropsychologicalassessment for detecting mild cognitive impairment. In

265