iiWAS2019 Papers

Proceedings of iiWAS2019

Prototyping and Preliminary Evaluation of Mind Monitoring
Service for Elderly People at Home
Chisaki Miura1 ,Haruhisa Maeda1 , Sachio Saiki1 , Masahide Nakamura1,3 , Kiyoshi Yasuda2
1 Graduate School of System Informatics, Kobe University, Japan
2 Riken AIP, Tokyo

3 Faculty of Informatics, Osaka Institute of Technology, Japan

cmiura@ws.cs.kobe-u.ac.jp,haruhisa@ws.cs.kobe-u.ac.jp,sachio@carp.kobe-u.ac.jp
masa-n@cs.kobe-u.ac.jp,fwkk5911@mb.infoweb.ne.jp

ABSTRACT

1

In order to support sustainable in-home long-term care, it is essential to monitor mental states of elderly people at home, as well as
to encourage their ability of self-care. However, the technical challenges include the limitations on human interventions and sensorbased monitoring, as well as daily recording and externalization ofmental states. In this research, we propose Mind Monitoring Service, which aims to monitor mental states and promote self-careof elderly people at home. In the proposed service, an agent asks
a user specific questions to acquire his/her mental state. Based on
the answers, the service then assesses the mental state and sends
feedback. We implement a prototype service, and evaluate the feasibility of the service through a preliminary experiment. The results show that data characterizing mental states of individual subjects was obtained successfully, and that some subjects externalized their minds by feedback from the service.

Japan is facing a super-aging society. The proportion of people
over 65 years old in the total Japanese population was less than 5%
in 1950, but it increased to 28.1% in 2018[3]. Under these circumstances, there are chronic shortage of nursing facilities and care
workers. To cope with the problem, The Japanese government isshifting the policy from the conventional facility-based care into
the in-home long-term care. The Ministry of Health, Labor and Welfare in Japan declares the Community-based Integrated Care System[12], which ensures the provision of health care, nursing care,prevention, housing, and livelihood support by the year 2025. The
system consists of four principles: self-care, mutual care, public
support, governmental aid. Due to the limitation of the security
cost, the government especially expects elderly people to conduct
the self-care under the system.
However, the self-care and independent living are not easy formost elderly people, as their physical abilities and cognitive functions are being declined. Thus, some external supports must be
needed. Moreover, elderly people have higher risk of mental depression, compared to those of younger ages due to their lost experiences [11]. Taking these facts into consideration, in the in-homelong-term care, it is essential to monitor their “mind”, and to provide appropriate supports according to the state of the “mind”.
In monitoring the psychological aspects of elderly people at home,
we consider the following three challenges:

CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI); Ubiquitous and mobile computing systems andtools; • Applied computing → Health care information systems.

KEYWORDS
in-home long-term care, elderly monitoring system, mental state,sensing, agent, chatbot
ACM Reference Format:
Chisaki Miura1 ,Haruhisa Maeda1 , Sachio Saiki1 , Masahide Nakamura1,3 ,Kiyoshi Yasuda2 . 2019. Prototyping and Preliminary Evaluation of Mind
Monitoring Service for Elderly People at Home. In The 21st International
Conference on Information Integration and Web-based Applications & Services (iiWAS2019), December 2–4, 2019, Munich, Germany. ACM, New York,NY, USA, 7 pages. https://doi.org/10.1145/3366030.3366127

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
iiWAS2019, December 2–4, 2019, Munich, Germany© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-7179-7/19/12.
https://doi.org/10.1145/3366030.3366127

INTRODUCTION（P1）Limitations on human interventions and sensor-based
monitoring: Traditionally, the mental states of elderly people have been assessed via human intervention such as inquiries and counseling by professionals. However, it is notrealistic to conduct such interventions every day at home.
Recently, the elderly monitoring systems using sensors and
IoT come onto markets. However, they can only monitor
externally observable events. Thus, the conventional monitoring systems do not cover internal mental states of elderly
people.
（P2）Challenge in recording and externalizing mental state:
Every mental illness of an elderly person (e.g., depression) is
caused by various factors. The symptoms also vary from one
person to another [10]. Thereby, the mental illness is often
mistaken as just the aging phenomenon or poor physicalcondition. Furthermore, without any opportunity, elderly
people would not spontaneously externalize their mental
states into words or written sentences.

439

Proceedings of iiWAS2019
iiWAS2019, December 2–4, 2019, Munich, Germany

iiWAS2019 Papers
Chisaki Miura1 ,Haruhisa Maeda1 , Sachio Saiki1 , Masahide Nakamura1,3 , Kiyoshi Yasuda2（P3）Challenge in realizing mind monitoring and appropriate support based on the mental state: As a resultof (P1) and (P2), it is difficult for any third person to objectively grasp, record and monitor the mental states of elderly
people at home. The elderly also have no opportunity to reflect their mental states. Therefore, it is yet challenging tosupport elderly people based on their internal mental state.

To overcome these challenges, we propose Mind Monitoring Service in this paper. For elderly people at home, the proposed service
aims to monitor their mental states, as well as to promote self-carebased on the states. More superficially, we tackle the above challenges (P1) to (P3) by the following (A1) to (A3), respectively.
(A1)Interaction using Mind Sensing Service: Our research grouphas been studying Mind Sensing. In the Mind Sensing, a virtual agent (VA) autonomously talks to a target person, triggered by an environment/behavior sensing as well as thetime schedule. The target then answers the question and thesystem records the answer. This process externalizes his/herinternal mind at that time as words [7][4]. The Mind Sensing Service is a core service of the Mind Sensing, by which
a user can define the questions from the VA systematicallyand flexibly. In this research, we extensively utilize the Mind
Sensing Service to acquire the mental state of the elderly via
the interaction between VA and elderly people.
(A2) Inquiry method specialized for acquisition of mental
state: We propose an inquiry method specific for acquiring mental states of elderly people. Based on mental tests
and counseling techniques in the clinical psychology, wedevelop questions that can be easily answered by the elderly. As the VA in (A1) asks these questions, the target person responds to them and externalizes the current mental
state. Repeating this process, the proposed service records
the mental states of the elderly people at home, continuously.
(A3) Self-care assistance by monitoring and feedback of mental state: The answers to the questions are aggregated on a
weekly basis to observe the mental states. Depending on theobservation, the proposed service creates feedback to thetarget person, including further questions and advice. The
feedback makes the target person recognize his/her own
mental state, and encourages the self-care. Moreover, if anabnormal state is detected, the service connects to external
supporters and asks for appropriate instructions.
Thus, the proposed service is able to acquire mental states of
elderly people at home continuously, which has been difficult sofar. It can also provide monitoring and self-care of “mind”, withappropriate interventions based on the acquired mental states.
In this paper, we implement a prototype of the proposed service,
and perform a preliminary experiment. Through the experiment,
we investigate the practical feasibility and implementation issuesof the proposed service.

2 PRELIMINARIES
2.1 Concepts of Health and Disease
The World Health Organization (WHO) defines the concept of healthas follows [13]:Health is a state of complete physical, mental and social well-being and not merely the absence of diseaseor infirmity.
According to this definition, health is a state that can be characterized by three aspects: physical, mental and social aspects.
On the other hand, there are various ways of defining disease. A
medical anthropologist, Young [14], classified the concept of disease into the following three types: Disease is an impairment thatcan be judged objectively. Illness is a disorder that is perceived subjectively. Sickness is used as a broad meaning including disease andillness, and is also considered as a state that the person cannot accomplish the social role completely.

2.2

Mental Illness of Elderly people

Typical mental illnesses elderly people tend to develop include depression and anxiety disorder. Depression is a psychiatric disorderthat causes a variety of mental and physical symptoms, such asa strong depressive mood, loss of interest or pleasure and disorders of appetite and sleep [11]. Anxiety disorder is a mental illness
that causes excessive anxiety, restlessness and the loss of concentration [10]. A major factor that causes the elderly to suffer fromsuch mental illnesses lies in their experiences of loss. The experiences include the deterioration of physical ability due to aging,
the loss of social role by the retirement, and the bereavement offamiliar people [10].
Moreover, it is often difficult to detect the mental illness of elderly people. For example, since elderly people with depression
tend to be less expressive, their sorrow, bad moods, and thoughtof depression are not well observed. Thus, the depression of elderly may be overlooked by a regular diagnostic criteria [11]. Similarly, the symptoms of general anxiety disorder, such as being
tired, lack of concentration, and sleeping disorder are commonly
observed among elderly people. Therefore, the mental illness ofelderly should be treated comprehensively by various factors including changes of daily thinking and physical declines.

2.3

Psychological Assessment Tools

Unlike physical illness which causes symptoms on specific partsof the body, mental illness causes global disorders in thought, consciousness, and physics. Hence, it is necessary to grasp the stateof the person from various viewpoints. The psychological assessment tools, including tests, scales, and questionnaires, are tools toquickly assess the mental state of the person.
A questionnaire sheet consists of multiple questions, each of which
the subject writes the answer for. To avoid burdening the subject,
every question is generally written in short and simple sentences.
The subject answers the question with Yes and No, or chooses
one from multiple scale. Representative questionnaires for psychological assessment include, GDS-15 (Geriatric depression scale15) [9]: the depression scale for the elderly, PHQ-9 (Patient Health440

iiWAS2019 Papers

Proceedings of iiWAS2019

Mind Monitoring Service for Elderly People at Home

iiWAS2019, December 2–4, 2019, Munich, Germany

Questionnaire-9) [6]: assessment of general depression, GAD-7 (Generalized Anxiety Disorder-7) [6]: measuring the degree of anxiety
disorder, and GHQ (General Health Questionnaire) [2]: assessmentof nephropathy. Although these questionnaires are not directly
used for diagnose, they are widely used for the screening purpose,
assessing the degree of symptoms and signs.

$QHOGHUO\SHUVRQ

Mind Sensing Service [5]Figure 1 shows the overall system architecture of the Mind Monitoring Service. As seen in the figure, the proposed service consistsof three parts. They achieves the three approaches (A1) to (A3) described in Section 1, respectively.

3K\VLFDOLW\$VNVTXHVWLRQV&UHDWHTXHVWLRQV$QVZHUVTXHVWLRQV

The Mind Sensing (“kokoro” sensing in Japanese) [7][4] is a new
type of sensing technique being developed by our research group.
It aims to record the internal mind of a target person that cannot be
observed externally by general sensors or IoT. In the Mind Sensing,a virtual agent (VA) or chatbot asks the person various questions
to externalize his/her mind as words. The Mind Sensing Service [5]is a core service of the Mind Sensing, which defines and manages
the questions, and automates the delivery of the questions and thecollection of the answers.
The Mind Sensing Service mainly consists of two parts: actions
and rules. An action defines a delivery of a concrete message tousers, which is specified by three items: target person(s) to which
the message is sent, the body of the message, and a service to deliver the message. The services currently implemented include VA(a virtual agent [8] on user’s PC), LINE (a dedicated LINE chatbot[1] on user’s smartphone), and MAIL (user’s email). For example,
suppose that we want to implement an action where the LINE chatbot asks a user A “How are you feeling now?”. Then, we define anaction: act1 = {targets: ["A"], messagebody: "How are younow?", service: "LINE"}.
A rule defines a trigger to execute actions, which is typed byeither time-based or event-based. A time-based rule repeatedly executes actions at the designated time and time interval. It is specified by actions to be executed, the start time, the end time, and thetime interval by minutes. For example, suppose that we want to execute the above action act1 every two hours from 8 o’clock to 20
o’clock. Then, we define a time-based rule: rule1 = {actions:["act1"], since: "8:00", until: "20:00" , interval:
120}.
An event-based rule triggers actions by an event input from anexternal system. It is specified by actions to be executed, an enabling condition over the event, and the cooling time to the next
execution. For example, we can define an event-based rule like:the VA says “Good morning! Did you sleep well?” when a behavior recognition system detects that user A wakes up in a bedroom
between 7 o’clock and 8 o’clock in the morning. Then, we create
an event-based rule: rule2 = {actions: ["act2"], since:"7:00", until:"8:00", location: "bedroom", event: "behavior
recognition .WAKEUP", cooling: 60}, where act2 = {targets:["A"], messagebody: "Good morning! Did you sleep well?",
service: "VA"}.

3 PROPOSED METHOD
3.1 System Architecture$ʁ,QTXLU\PHWKRGVSHFLDOL]HG
IRUDFTXLVLWLRQRIPHQWDOVWDWH$JHQW

6XSSRUW
RULQWHUYHQWLRQ

2.4

n0LQG|0RQLWRULQJ6HUYLFH$ʁ,QWHUDFWLRQXVLQJ
0LQG6HQVLQJ6HUYLFH([WHUQDO6XSSRUWHUV,QTXLU\'%

0HQWDOLW\
6RFLDOLW\$ʁ6HOIFDUHDVVLVWDQFHE\PRQLWRULQJDQGIHHGEDFNRIPHQWDOVWDWH$QDO\]HDQVZHUV)HHGEDFNV*HQHUDWH
IHHGEDFNV0HQWDO6WDWH'%

8QGHUVWDQG
DQGPRQLWRU
PHQWDOVWDWH

'HWHFWLRQRIDQ$EQRUPDO6WDWH

Figure 1: System Architecture(A1) Interaction using Mind Sensing Service: Using the Mind
Sensing Service, the proposed service periodically collectsinternal states of an elderly person at home, through interaction between the agent and the person.
(A2) Inquiry method specialized for acquisition of mental state: We develop inquiries specific for acquiring mental states of the elderly person. The inquiries are stored ina database. The inquiries are then encoded by actions andrules of the Mind Sensing Service.
(A3) Self-care assistance by monitoring and feedback of mental state: Every time the elderly person answers a question,
the answer is stored in a database with timestamp. With anappropriate period, the service then analyzes the collected
mental states. According to the result, the service produces
feedback including further questions and advise. If an abnormal state is detected, the service connects to external
supporters, and asks for appropriate instructions.

3.2

A1: Interaction using Mind Sensing Service

By using the Mind Sensing Service, the agents (i.e., a VA or a chatbot) can listen to and record the internal minds of elderly people,
instead of human caregivers. In the proposed service, we let an
agent send questions to elderly people at a fixed time every day.
For this purpose, we apply the time-based rule of Mind Sensing
Service. For the human-agent interaction, we use the LINE chatbot over a well-known messaging application on smartphone.
The reason why we chose the LINE chatbot as the interaction
tool is that the chatbot is easier and more reliable to exchange thequestions and the answers, compared to the VA on the PC or Email. The questions (described in detail in A2) for acquiring themental states are somewhat technical. Thus, the text messages overthe message chat is clearer than the voice dialogue with the VA.
In addition, LINE has an intuitive UI and text input support withspeech recognition. It is already accepted by many elderly people.
Therefore, we consider that LINE is more suitable than e-mail.

3.3

A2: Inquiry Method Specialized for
Acquisition of Mental State

In order to understand mental states of elderly people systematically, we first define a framework characterizing mental health.
For this, we integrate the concepts of health and disease described441

Proceedings of iiWAS2019
iiWAS2019, December 2–4, 2019, Munich, Germany

in Section 2.1, and characterize the mental health in the following
three perspectives: Physicality, Mentality, and Sociality.
Physicality corresponds to the physical aspect of health and disease. It targets physical symptoms that can be explained by objective factors. In mental illness, such physical symptoms are quite
important and should be assessed. In the physicality perspective,
we try to grasp the mental health, according to the presence orabsence of the physical symptoms.
Mentality corresponds to mental aspect of health and illness. It
covers subjective feelings such as emotions and moods. Even withthe same physical symptoms, the mental states of elderly people
generally vary from one person to another. It is therefore essential to elicit a subjective sense through inquiries. In the mentality
perspective, we characterize the mental health by subjective assessment.
Sociality corresponds to the social aspect of health and sickness.
It covers the evaluation from others and self-evaluation. The evaluation from others means the state of the target person that canbe pointed out by a third person. The self-evaluation means how
the target person thinks himself (or herself). In general, the social
health focuses on the evaluation from others only. However, in thepsychological approach, the subject’s self-awareness should also
be considered. For example, a patient with depression often has astrong sense of guilty, and regards himself as a worthless person
for no reason. Thus, it is crucial to ask how the subjects assesses
themselves. Through the two types of evaluations, we try to grasp
the mental health.
From each of the three perspectives, we develop inquiries thatreveal the current mental state of elderly people. We create thequestions within the inquiries by referring to the practical psychological assessment tools introduced in Section 2.3. For each question, we assume that the target elderly person answers by choosing
one from four choices: “No, I do not think so at all”, “No, I do notthink so much”, “Yes, somewhat I think so” and “Yes, I really thinkso”. We intentionally omit a marginal choice “Yes or No”, in orderto encourage the target person to carefully reflect his/her current
state.
To achieve sustainable and reasonable Mind Monitoring, we consider that the inquiries should be at most twice a day, and the number of questions in each inquiry should be at most three. Each inquiry is sent every day at a fixed time, considering the life rhythmof the target person. It is also necessary to change questions every
day, and to send encouraging messages, so that the person does
not get tired and quit answering.
We then encode the developed inquiries by actions and rules ofthe Mind Sensing Service. Finally, the Mind Sensing Service automatically sends the questions according to the specified rules, andcollects the answers.

3.4

A3: Self-care Assistance by Monitoring andFeedback of Mental State

By analyzing the answers to the questions, the service grasps themental state of the target person. As explained in A2, there are
four choices for answering each question. We assign the score of2, 1, -1, -2 to these choices, so that 2 points are for most healthy
choice and -2 points are for most unhealthy choice. Each question

iiWAS2019 Papers
Chisaki Miura1 ,Haruhisa Maeda1 , Sachio Saiki1 , Masahide Nakamura1,3 , Kiyoshi Yasuda2

belongs to one of the three categories defined in A2. Therefore,
the score of the response represents the degree of psychologicalhealth in the corresponding category. A positive score representsa healthy mental state, whereas a negative score represents a unhealthy mental state.
For example, let us consider a question “Do you often wake up
while sleeping?” The question is classified in Physicality, as its intention is to examine the sleeping disorder. As for the four choices“No, I do not think so at all”, “No, I do not think so much”, “Yes,somewhat I think so” and “Yes, I really think so”, score points 2, 1, 1, -2 are assigned to them, respectively. If the target person answers"Yes, I really think so", the current mental state of the Physicality
perspective is evaluated to be -2 at this time.
The proposed service puts these scores in a time-series analysis,or statistically aggregates the scores on a weekly or monthly basis. Based on the analysis, the proposed service implements Mind
Monitoring. The time-series analysis can detect abnormal mental
states if the following observations are found: the negative scores
last for a long time, the positive scores lasting for a long time suddenly changed to negative, or the target person suddenly quit answering.
The proposed service also generates feedback that encourages
the target person to conduct self-care. The most basic feedback isto present problems or concerns that can be inferred from the answers. It also presents countermeasures to alleviate the situation.
When the service detects an abnormal state that cannot be solved
by self-care, it connects to external human caregivers to request
human intervention or support.
For example, let us consider again the previous question, “Do
you often wake up during the night?” The answer “Yes, I reallythink so” represents that the person may have sleep disorder. Then,the service makes a further question: “Recently, you may not sleep
well. There may be suspicion of sleep disorder. Please let me know
if you have any trouble or worry.” On receiving such feedback,the person can recognize the possibility of sleep disorder, and reflect the cause of it. Besides, if the service tells countermeasures
to alleviate sleep disorders, or introduces medical institutions, it ispromising to reduce the subject’s anxiety, significantly.
Operating the proposed service continuously enables the Mind
Monitoring of elderly people at home. Based on the accumulated
answers, statistics, and feedback, individual elderly people can reflect their minds by themselves. Also, the accumulated data can beused as strong evidence for personalized medical care and nursing,
optimized for individual subjects.

4 PROTOTYPE IMPLEMENTATION
4.1 Configuring Mind Sensing Service
We implemented a prototype system to verify the practical feasibility of the proposed service. In this implementation, we configured
the Mind Sensing Service (see Section 2.4), so that the LINE agent
sends three questions twice a day (in the morning and the evening).
Figure 2 shows a screen where we applied this configuration to the
Mind Sensing Service.
In Figure 2, Miura_Morning0,1,2,3 and Miura_Evening0-3 (theprefix Miura_ is omitted hereafter) represent 8 actions. For eachaction, we can see that target people, a message body, LINE as442

iiWAS2019 Papers

Proceedings of iiWAS2019

Mind Monitoring Service for Elderly People at Home

iiWAS2019, December 2–4, 2019, Munich, Germany

Figure 2: Mind Sensing Service Console
a message delivery service. Morning0 (or Evening0) are greetingmessages to start an inquiry in the morning (or evening, respectively). In Morning1,2,3 (or Evening1,2,3), we define concrete
three questions asked in the morning (or evening, respectively) inquiry.
MorningRule and EveningRule represent time-based rules that
execute the inquiries in the morning and evening, respectively.
MorningRule executes four actions Morning0,1,2,3 at 6:30, whereas
EveningRule executes Evening0,1,2,3 at 21:30. In both rules, theinterval is set to 1440 minutes, so that MorningRule and EveningRule
are executed exactly once a day. Thus, each target user receives
three questions following a greeting message every morning at
6:30, as well as every evening at 21:30.
As mentioned in Section 3.3, the contents of the questions should
be changed every day, preferably. However, we have not yet implemented a feature that automatically updates actions of the Mind
Sensing Service. Therefore, in this prototype, we manually updated
the message body of Morning1,2,3 and Evening1,2,3 every day.

4.2

Creating Questions

In order to create concrete questions in the inquiry, we extensively
used the existing questionnaire sheets for mental illness. Specifically, based on GDS-15, PHQ-9, GAD-7, and GHQ60 described in
Section 2.3, we created 42 questions in total. The primary goal of
the proposed service is sustainable mind monitoring, but not rigorous screening tests of specific mental illnesses. Therefore, we
intentionally excluded too strong questions such as those asking

death or suicide. We also unified the end of sentences of all questions, so that the elderly people could answer them easily.
We classified the 42 questions into three categories defined in
Section 3.3. As a result, 13 questions were classified into Physicality, 21 were into Mentality, and 8 were into Sociality, as shown in
Figure 3 (a) (b) (c), respectively. In the figure, sentences in bold face
represent questions where the answer “Yes, I really think so” is assessed as most healthy. Other sentences represent questions where“Yes, I really think so” means the worst healthy.
Each subject was asked to answer three questions in the morning, and other three in the evening. Since six questions were consumed every day, the total 42 were covered in one week.

4.3

Analysis and Feedback

In the prototype system, the proposed service aggregates obtained
scores within every week, and analyzes statistics. The service thenidentifies questions to which a user marked negative scores. Finally, as the feedback to the user, the service sends a further question that makes the user reflect himself with respect to the negative
questions.
For this purpose, we extracted a symptom inferred by each question. The results are shown in the column “inferred symptom” inFigure 3 (a) (b) (c). In the feedback, the service shows the user thesymptoms inferred from questions that have been negatively assessed in the past week. Showing concrete symptoms would promote the user to conduct self-reflection and externalize thought
and anxiety as words.

443

Proceedings of iiWAS2019

iiWAS2019 Papers
Chisaki Miura1 ,Haruhisa Maeda1 , Sachio Saiki1 , Masahide Nakamura1,3 , Kiyoshi Yasuda2

iiWAS2019, December 2–4, 2019, Munich, Germany
4XHVWLRQVʤ3K\VLFDOLW\ʥLQIHUUHG V\PSWRP'R\RXWKLQN\RXKDYHWURXEOHIDOOLQJDVOHHSRUDZDNHLQWKHPLGGOHRIQLJKW"

VOHHS GLVRUGHU'R\RXWKLQN\RXVOHHSWRRPXFKUHFHQWO\"VOHHS GLVRUGHU'R\RXWKLQN\RX IHHOWLUHGDQGGRQRWZDQWWRGRDQ\WKLQJ"IDWLJXH DQGLQHUWLD'R\RXWKLQN\RXKDYHQRWKDGPXFKDSSHWLWHUHFHQWO\"DQRUH[LD'R\RXWKLQN\RXKDYHHDWHQWRRPXFKUHFHQWO\"RYHUHDWLQJ'R\RXWKLQNLWLVGLIILFXOWWRIRFXVRQUHDGLQJQHZVSDSHUVRUZDWFKLQJ79"

GLIILFXOW\ RI
FRQFHQWUDWLRQ'R\RXWKLQN\RXIHHOJRRGDQGKHDOWK\"XQKHDOWKLQHVV'R\RXWKLQN\RXZDQWWRWDNHDIDWLJXHUHFRYHU\DJHQW"IDWLJXH'R\RXWKLQN\RXIHHOWLUHG"

IDWLJXH'R\RXWKLQN\RXKDYHDORWRIKHDGDFKHVUHFHQWO\"KHDGDFKH'R\RXWKLQN\RXRIWHQIHHOOLNH\RXUKHDGLVKHDY\"KHDGDFKH'R\RXWKLQN\RXIHHOIOXVKHGRUDFKLOO"KHDW RUFKLOO'R\RXWKLQN\RXZDNHXSLQWKHPLGGOHRIWKHQLJKW"

VOHHS GLVRUGHU(a) Physicality
4XHVWLRQVʤ6RFLDOLW\ʥLQIHUUHG V\PSWRP'R\RXWKLQN\RXIHHOVDWLVILHGZLWK\RXUGDLO\OLIH"

GLVVDWLVIDFWLRQZLWKOLIH'R\RXWKLQNOLIHLVHPSW\"HPSWLQHVVRIOLIH'R\RXWKLQN\RXRIWHQIHHOERUHGHYHU\GD\"

ERULQJOLIH'R\RXWKLQN\RXDUHKDSS\"XQKDSSLQHVV'R\RXWKLQN\RXRIWHQIHHOWKDW\RXDUHSRZHUOHVV"SRZHUOHVVQHVV
LQDFWLYLW\'R\RXWKLQN\RXDUHIXOORIHQHUJ\"'R\RXWKLQNWKDWWKHVSHHGRIPRYHPHQWDQGVSHDNLQJKDVEHFRPHVRVORZWKDWRWKHUVQRWLFHLW"

PHQWDOGLVRUGHU'R\RXWKLQN\RXFDQQRWFDOPGRZQDVPXFKDVRWKHUVQRWLFH"

PHQWDOGLVRUGHU

4XHVWLRQVʤ0HQWDOLW\ʥLQIHUUHG V\PSWRP'R\RXWKLQN\RXUGDLO\DFWLYLWLHVDQGLQWHUHVWVKDYHGHFOLQHG"

GHFOLQH RIDFWLYLW\DQGLQWHUHVWV'R\RXWKLQN\RXDUHRIWHQGULYHQE\YDJXHDQ[LHW\LQWKHIXWXUH"IXWXUHDQ[LHW\'R\RXWKLQN\RXZDQWWRVWD\KRPHUDWKHUWKDQJRLQJRXWRUGRLQJQHZWKLQJV"GHFOLQH RIDFWLYLW\'R\RXWKLQN\RXDUHPRUHZRUULHGDERXWIRUJHWWLQJWKLQJVWKDQDQ\WKLQJHOVH"

FRQFHUQ RYHU
IRUJHWIXOQHVV'R\RXWKLQN\RXIHHOWKDWWKHUHLVQRKRSH"

GHVSDLU'R\RXWKLQN\RXKDYHOLWWOHLQWHUHVWRUHQMR\PHQWRIWKLQJV"GHFOLQH RILQWHUHVWV'R\RXWKLQN\RXIHHOGHSUHVVHG DQGKRSHOHVV"GHVSDLU'R\RXWKLQN\RX IHHOQHUYRXVRUDQ[LRXVUHFHQWO\"WHQVLRQ DQ[LHW\RYHUVHQVLWLYH'R\RXWKLQN\RXKDYHEHHQWRRZRUULHGUHFHQWO\"ZRUU\'R\RXWKLQNLWLVGLIILFXOWWRUHOD["

GLIILFXOW\ RIUHOD[LQJ'R\RXWKLQN\RXIHHOUHVWOHVV"UHVWOHVVQHVV'R\RXWKLQN\RXIHHODQQR\HGDQGDQJU\UHFHQWO\"

DQJHU'R\RXWKLQN\RXPLJKWEHDIUDLGWKDWVRPHWKLQJWHUULEOHZLOOKDSSHQ"IHDU'R\RXWKLQN\RXFDQQRWVOHHSEHFDXVHRIZRUULHV"ZRUU\'R\RXWKLQN\RXDOZD\VIHHOVWUHVV"FKURQLF VWUHVV'R\RXWKLQN\RXPLJKWJHWIUXVWUDWHGDQGDQJU\"

DQJHU'R\RXWKLQN\RXDUHVFDUHGRIVRPHWKLQJIRUQRSDUWLFXODUUHDVRQ"IHDU'R\RXWKLQNHYHU\WKLQJLVPRUHEXUGHQVRPHIRU \RXWKDQXVXDO"VWUHVV'R\RXWKLQN\RXIHHODQ[LHW\RUWHQVLRQ"

DQ[LHW\WHQVLRQ'R\RXWKLQN\RXRIWHQVHHP WREHLQDJRRGPRRG"

EDGPRRG'R\RXWKLQN\RXFDQQRWVWRSZRUU\LQJ"

ZRUU\(b) Mentality(c) Sociality
Figure 3: Created Questions

5.2

Figure 4: Box Plots Visualizing the Scores

5 EXPERIMENTAL EVALUATION
5.1 Overview
We have performed a preliminary experiment using the prototype
implementation. The experiment was conducted with six subjects
for seven days from July 2nd to 20th, 2019. The subjects were twomen in the 70’s, a man in the 40’s, and three men in 20’s. The proposed service was originally intended for the elderly generation.
However, we also recruited young subjects in the experiment inorder to evaluate practical feasibility of the service.
The experiment has been approved by the Research Ethics Committee of Kobe University Graduate School of System Informatics(No. R01-02).

Experimental Results

First, we describe the results related to the practical feasibility.
During the seven-day experiment, five of the six subjects (let thefive be A, B, C, D, and E) were able to respond to the inquiries,
successfully. However, the one in the 70’s (let the one be F ) gave
almost no response. After the experiment, we asked subject F thereason why he did not answer the questions. He told us that hedid not notice the question messages due to the narrow screen setting of the LINE application. We should have checked the screen
setting before the experiment. Furthermore, we found that the responses from the subjects were not always on time. We assumed
that subjects should answer three questions in the morning, andother three in the evening. However, there were some cases where
six questions were answered at once. We also found that some answers were missing. Some subjects complained that it was balky
to input the answers of the three questions by text typing.
Next, we see the result of the mind monitoring achieved by the
proposed service. Figure 4 shows box plots visualizing the scoresof the five subjects. Each box plot represents the distribution ofscores for three categories for a subject. By focusing on averagevalues of each category, we can observe characteristics on individual’s mental states. For example, the average values of subject
D are -0.08, -0.21, 1.00 for Physicality, Mentality, and Sociality, respectively. We can see that his mental states were a bit inadequate
in physical and mental aspects, and that states in social aspect wasmaintained. The height of each box indicates the degree of variance in the answers. Considering four subjects except subject C,

444

iiWAS2019 Papers

Proceedings of iiWAS2019

Mind Monitoring Service for Elderly People at Home

iiWAS2019, December 2–4, 2019, Munich, Germany

we can see that Mentality was the most variable, and Sociality wasless variable.
Based on the results, feedback was created using the method
described in Section 4.3, and sent to each subject. The followingshows an example of feedback sent to subject B.

Good morning, Mr. B! According to your answers during 7/27/8, it seems that you were recently feeling- decline of activity and interests- fatigue and inertia- difficulty of relaxing- anxiety- decline of concentration- concern over forgetfulness- chronic stress
Do you have any thoughts in mind? If you have any concern
or anxiety, please feel free to talk to me.

Then, subject B responded to this feedback as follows.
Thank you. Recently, I have been very busy with my work,and I had no time to relax. I tend to forget various things
because there are too many tasks to do. I believe that is why
I am feeling chronic stress. However, the work is going well,and everyone is doing his or her best, so I think that this kindof anxiety will disappear soon.

6

CONCLUSION

In this paper, we have proposed Mind Monitoring Service that monitors internal minds of elderly people to support sustainable inhome long-term care. Using our technique of Mind Sensing, an agent asks the user questions to acquire the mental state. The service assesses the mental state from viewpoints of Physicality, Mentality, and Sociality, by scoring the answers. The service finallysends the feedback showing symptoms inferred from questions
with negative scores. We also implemented a prototype of the proposed service and conducted a preliminary experiment. The experimental results show that data characterizing mental states ofindividual subjects was obtained successfully, and that some subjects externalized their minds by feedback from the service.
In our future work, we conduct more extensive experimentswith many elderly people. For this purpose, we plan to improve thehuman-agent interaction and the feedback method. It is also a challenging but important issue to present appropriate self-support
 methods for individual mental states.

 ACKNOWLEDGMENTS

This research was partially supported by JSPS KAKENHI Grant
Numbers JP19H01138, JP17H00731, JP18H03242, JP18H03342, JP19K02973.

REFERENCES[1] LINE Corporation. [n.d.]. LINE. https://line.me/en/.
[2] David P. Goldberg, Yasuaki Nakagawa, and Ikuo Daibo. 1985. The General Health
Questionnaire. Nihon Bunka Kagakusha.
 [3] Government of Japan. 2019. Annual Report on the Aging Society (2019).
http://www.cao.go.jp/.
It is clearly seen that subject B looked back on himself, and ex[4] Haruhisa Maeda, Sachio Saiki, Masahide Nakamura, and Kiyoshi Yasuda. 2019.
ternalized his mind owing to the feedback. In other words, theMemory Aid Service Using Mind Sensing and Daily Retrospective by Virtual
feedback promoted his self-reflection. From this fact, we can conAgent. In 10th International Conference, DHM 2019, Held as Part of the 21st HCIInternational Conference, HCII 2019, Vol. LNCS 11582. 353–364. Orland, USA.
firm the effectiveness of the feedback to a certain extent. After the[5] Haruhisa Maeda, Sachio Saiki, Masahide Nakamura, and Kiyoshi Yasuda. 2019(toexperiment, we interviewed the subjects what they felt about theappear). Rule-Based Talking Service to Elderly at Home for Efficient Mind Sensing. In IEICE Technical Report.
service. We obtained various comments from the subjects. They
[6] Kumiko Muramatsu. 2014. An up-to-date letter in the Japanese version of PHQ,
are, for example, “In feedback, there were some items that did not
PHQ-9, PHQ-15. https://ci.nii.ac.jp/naid/120005482683/. , 35-39 pages.
apply to my situation”, “I found that my psychological condition[7] Masahide Nakamura, Kenji Hatano, Jun Miyazaki, Kiyoshi Yasuda, Noriaki
Kuwahara, Hiroaki Kazui, Sachio Saiki, Seiki Tokunaga, Mihoko Otake, Naoki
was dangerous, as too many things were pointed out.”.
Kodama, and Naoko Kosugi. 2019-2023. Developing a System for Self-care andMutual-aids of Elderly People at Home Based on Externalization of InternalStates. JSPS KAKENHI, Grant-in-Aid for Scientific Research (A) 19H01138 (20195.3 Discussion2023).
[8] Shota Nakatani, Sachio Saiki, Masahide Nakamura, and Kiyoshi Yasuda. 2018.
The preliminary experiment with the prototype went mostly very
Generating Personalized Virtual Agent in Speech Dialogue System for People
well. On the other hand, from the results and the subsequent inwith Dementia. In Digital Human Modeling 2018 (DHM 2018), Held as Part of HCI
terview, we found that the prototype still has two limitations.
International 2018, Vol. LNCS 10917. Springer, 326–337. Las Vegas, USA.
[9] Morihiro Sugishita and Takashi Asada. 2009. Geriatric Depression Scale andFirstly, there is an issue of immediate response, which means that
minus; Short Version-Japanese, GDS-S-J. Japanese Journal of Cognitive Neurothe answer to the inquiry can be delayed or forgotten. LINE is anscience 11, 1 (2009), 87–90. https://doi.org/10.11253/ninchishinkeikagaku.11.87[10] Masatoshi Takeda and Toshihisa Tanaka. 2007. E de Miru Kokoro no Hokenexcellent chat communication tool, but there is no mechanism toshitu(Mental clinic with pictures). Alta Publishing Co., Ltd.
guarantee the response. The fact that the subjects had to use the[11] The Ministry of Health, Labor and Welfare in Japan. 2009.
text input for answering questions was also affecting the responDepression prevention and support manual (revised version).
https://www.mhlw.go.jp/topics/2009/05/dl/tp0501-1i.pdf.
siveness. We need to devise methods for easy response and moti[12] The
MinistryofHealth,Labor
andWelfareinJapan.
vation supports.
2013.
Community-based
Integrated
Care
System.
Secondly, the effectiveness and satisfaction of feedback werehttps://www.mhlw.go.jp/stf/seisakunitsuite/bunya/hukushi_kaigo/kaigo_koureisha/.

also inadequate. The proposed feedback method pointed out all
symptoms inferred from questions that have been evaluated negatively even once. However, considering the comments such as“there were items that did not apply to me”’ or “there were too
many items pointed out”, this method may be generating hypersensitive and pessimistic feedback. Further improvement in generating the feedback will be necessary.

[13] World Health Organization. 1946. Constitution of the World Health Organization. https://www.who.int/.
[14] A Young. 1982. The Anthropologies of Illness and Sickness. Annual Review of
Anthropology 11 (1982), 257–285.

445