Memory Aid Service Using Mind Sensing
and Daily Retrospective by Virtual Agent
Haruhisa Maeda1(B) , Sachio Saiki1 , Masahide Nakamura1,2 ,and Kiyoshi Yasuda3
1

Graduate School of System Informatics, Kobe University,
1-1 Rokkodai, Nada, Kobe, Japan
haruhisa@ws.cs.kobe-u.ac.jp, sachio@carp.kobe-u.ac.jp,
masa-n@cs.kobe-u.ac.jp
2
Riken AIP, 1-4-1 Nihonbashi, Chuo-ku, Tokyo 103-0027, Japan
3
Osaka Institute of Technology, 5-16-1, Omiya, Asahi, Osaka, Japan
fwkk5911@mb.infoweb.ne.jp

Abstract. Our research group has been studying smart home services
for elderly people, which detect their daily activities based on the environmental sensors in the house. However, such sensors can only obtain
limited information. To execute more optimized care, we must retrieve
not only external events but internal states. Furthermore, to support
memory aid, it is important to be able to retrieve the recorded information at any time. In this paper, we propose a new memory aid service,
which records the self-talk of elderly people and utilizes the recorded
information. Speciﬁcally, we develop mind sensing, which is to externalize the inside of elderly’s heart as a sentence using a virtual agent. Then,the recorded information by mind sensing is cleansed through calibration and classiﬁcation based on dialogue between an elderly person anda virtual agent. These information can be retrieved by classiﬁcation or
arbitrary keywords. In this way, the proposed service enable elderly torecord and retrieve what they thought anytime anywhere.
Keywords: Smart health-care · Dialogue agent · Mind sensing

1

Introduction

Japan is currently facing hyper-aging society, which causes many social issues.
According to a report by the Government of Japan, the number of elderly people
over the age of 65 is 35.15 million in 2017, which is 27.7% of the total population[2]. Due to the long average life and the low birthrate, the number of elderly
over 65 is increasing in Japan and will reach 40% of the total in 2050. There
are many challenges, for example, the shortage of human resources and facilities
in the nursing care ﬁeld. Therefore, the government is focusing on supporting
home care. However, there is a possibility that the burden on family caregivers
increases in home care. Furthermore, the elderly at home has various diﬃcultiesc Springer Nature Switzerland AG 2019

V. G. Duﬀy (Ed.): HCII 2019, LNCS 11582, pp. 353–364, 2019.
https://doi.org/10.1007/978-3-030-22219-2_27

354

H. Maeda et al.

in daily life due to the decline of cognitive function. In particular, as elderly
people are getting forgetful, they can not do what they had done until then,so that they feel strong anxiety, confusion, and deterioration of self-esteem. In
order to support these elderly people, many research on assistive technology has
been carried out in various organizations in recent years.
Our research group has been developing the activity recognition service basedon the environmental change [6]. This service read the variations of environmental sensors in elderly’s home and detects their activities by machine learningtechnology. The detected activities are notiﬁed to the mobile terminals of elderly,
and they respond to the notiﬁcations based on the their current situation. This
service recognizes elderly’s activity and at the same time realizes the creation ofcommunication opportunities for them.
On the other hand, the previous research is limited to the activity recognition
that can be observed with traditional sensors. To execute more personalized carefor elderly, it is important to obtain not only such external events but the internal states such as human emotions and physical condition. The internal state is
directly linked to human health, and it is important information for monitoring [1]. However, the internal states are usually acquired through inquiries andcounseling by experts that can not be done permanently at home. Moreover, it
is diﬃcult to measure the changes of internal states by sensors.
The purpose of this research is to support elderly people who feel uneasy
about forgetfulness. In this paper, especially, we propose a new memory-aid
service using virtual agents, which are 3D humanoid robots. The proposed service
mainly has the following two functions.
– Agent-Assisted Mind Sensing: A system that an agent asks users
about their minds, to require and record the internal states of them
– Memory-Aid Retrospective: A system that an user and an agent
look back the day using the data recorded by mind sensing
In order to solve the limitation of the previous research, we propose mind
sensing that a virtual agent directly asks the elderly about their internal states.
With various events as a trigger, the agent sends a message to the elderly mobile
terminal asking for the current physical condition and emotion. After the user
answers the question, the detailed information of the answer is automaticallystored in the database. As a result, we can grasp the internal state of the elderly
and use it for more sophisticated care.
Retrospective is that the elderly and the agent review the day at an arbitrary
sense of time utilizing the accumulated internal states of them. While viewing
the list of messages entered by elderly by mind sensing, they calibrates and classiﬁes these messages one by one. Thus, elderly people can prevent forgetfulness
by looking back at what happened recently. Furthermore, they can search theinternal states later on the basis of classiﬁcations, words, period, and can retrieve
them whenever necessary.
In this paper, we also implement a prototype system that is equippedwith the function of mind sensing and retrospective. Through the preliminary

Memory Aid Service Using Mind Sensing

355

experiments, we conﬁrm that the proposed system can be expected to be
extremely eﬀective forgetfulness.

2

Preliminary

2.1

Elderly Care at Home

Along with the aging society, single households whose head is over 65 years old isincreasing in Japan. However, it is diﬃcult to accept and carry out the care forall elderly at medical facilities. On the other hand, there are a lot of elderly that
prefer to receive care at their own home or their family’s home rather than toreceive at nursing home or care house. Against this background, in recent years,
many attempts to care for elderly people at home have been made in various
ﬁelds. Japanese Government is focusing on training human resources for homehealth care and promoting cooperation in each local government, and aims torealize Community-based integrated care systems [3] in which residences, medicalcare, and nursing care are integrally provided so that elderly can continue theirown living in a familiar area even if they become severely in need of nursing care.
The general companies are starting to work hard on developing new services that
relieve elderly of loneliness using a communication robot. These eﬀort aims to
improve the quality of life (QOL) of the elderly.
2.2

Virtual Agent

In elderly care, it is expected to use care robots that utilize various assistive
technologies to assist daily living. Among these technologies, virtual agent is a
3D friendly humanoid robot displayed on the PC screen, and can use speech
recognition and speech synthesis technology to interact with users. By introducing such an agent to the medical ﬁeld, it is expected that instead of an actualcare provider, it becomes a communication partner of the elderly and reducethe burden of caregiver. In our previous research, we have developed a system,
called Virtual Care Giver (VCG), using the VA [7]. VCG was designed to beable to cooperate with Web services, to integrate IoT, smart home and cloud.
However, there is also a limitation caused by the looking of VA. The care andadvices from unfamiliar avatar do not always motivate the elderly very well. In
a recent study, to solve such a limitaion, we has been studying MPAgent systemusing virtual agent technology [5]. Figure 1 shows a screen shot of MPAgent.
This service creates a 3D model of a face and dynamically generates an agent
based on the feature points of the face acquired from the facial photograph. It
also realizes a realistic dialogue by lips movement and changing the expression
by the utterance. Delegating the communication care with MPAgent, a humancaregiver can concentrate on human-centric tasks that cannot be done with ICT.

356

H. Maeda et al.

Fig. 1. Screen shot of MPAgent

2.3

Elderly Monitoring Service

As one of the technologies to support home care for the elderly, a monitoring
system at home using ICT is considered promising. In the general process ofmonitoring, ﬁrst, the system senses real-world data, and grasp the situation of
the elderly and their environment. The data to be sensed includes biometric datasuch as pulse and acceleration of arm or leg, and environmental data such astemperature, humidity, and illuminance. Second, using the obtained data, thesystem recognizes the elderly’s activities of daily living. Finally, based on therecognized activity, the system leads to actual care. Such a monitoring system
reduces the eﬀorts of nursing care and aims at improving the quality of care that
only human beings can. However, in the conventional monitoring system, there
were issues such as introduction cost, invasiveness problem, failure to achieve
personal adaptation using the collected data, and lack of communication to theelderly. In order to solve these problems, in previous research, our research group
has proposed and developed an elderly state notiﬁcation service [6] using activity
recognition based on environmental change. This service consists of an activity
recognition service that predicts elderly’s activity by using environment sensor
data, and a care execution service that makes notiﬁcations based on predictedactivity using SMS application. With this system, it is possible to record activity by non-invasive sensing and to create communication opportunities for theelderly. On the other hand, even by utilizing various sensors and devices, it is
only detecting external events such as environmental data and user’s expressions
and behaviors, and it is impossible to observe the internal states within the user’smind. The internal state is directly related to human health, and it is important
information for monitoring. Obtaining internal states helps to grasp the moredetailed status of the elderly and helps more sophisticated care. In addition toacquiring conventional external events, it is important for the monitoring system
to continue to monitor the internal state.

3

Proposed Service

3.1

Requirement

In this research, we propose a service to acquire and accumulate the internal
state of the elderly who could not be obtained by the conventional monitoring

Memory Aid Service Using Mind Sensing

357

system. Also, it contributes to the forgetfulness of elderly based on the accumulated internal condition. The proposed service realizes the following two system
requirements.
– R1: Externalize the mind of the elderly into words– R2: Accumulate the words and retrieve them at any time later
We implement a mind sensing service with the function of R1, that acquire
the internal states of elderly by the virtual agent. We also implement a retrospective service depending on the function of R2, that have elderly to lookback on the acquired internal state using a virtual agent and lead to prevent
forgetfulness.
3.2

Use Case Scenario

Here, we explain how the user actually uses the proposed system. When the user
starts the mind sensing service, ﬁrst of all, the system registers information of
sensors installed in the house in order to link with sensor data. By doing this,various notiﬁcation is sent to the user triggered by user’s activity recognition, so
the user returns the current feeling, what happened on that day, future schedule,
and so on.
When doing retrospective service, while interacting with the agent, the usercorrects and classiﬁes the data input so far. By classifying the contents of thedata into classes deﬁned by the users themselves, they are used for later retrieval.
If the user wishes to remember what happened in the past, he can use various
queries to search and conﬁrm the data he entered by mind sensing, at any time.
In addition, if the user wants to change personal settings, the user can makesettings such as the agent’s questioning timing and user information change.
3.3

System Architecture

In this research, to support memory aid of elderly people, we build an architecture that can manage important information for the elderly’s life. Figure 2 shows
the system architecture of the proposed service. The proposed system is made up
of two services. One is Agent-Assisted Mind Sensing Service that acquires
and records the feelings of the user. The agent triggers various events such as
the result of the user’s activity detection presumed by the existing system andasks the user about his mind using a smartphone. Then, the service obtains theanswer from the user and stores it in the database. By managing such internal
state for users, we will use it for other services. For instance, the doctor and the
caregiver refer to the accumulated information, they can grasp the state of theelderly and make appropriate care plan. The second is Memory-Aid Retrospective Service based on the data recorded by mind sensing. In this service,while interacting with the agent, the user looks back inside the mind that theuser himself inputs on one day. As a result, the user can remember what happened on the day and what he thought. Furthermore, the user can search the

358

H. Maeda et al.

data of his mind based on various queries and check them at any time. Since
elderly people can retrieve important information anytime in their daily lives, it
is possible to ease the anxiety of forgetfulness. It is expected these retrospective
views will lead to memory aid of elderly. We discuss these two services in detailin Sects. 3.4 and 3.5 below.

SensorBox

Activity recognition system
Activity
Recognition

SensorDB

Monitorenvironment

Trigger sensing

Exposeinternal states
SMS app on
smartphone

Ask states
API
Record states

Mind
DB

Send dialogue
Review pastmessages

API
Virtual Agent
on PC

Query
messages

Mind
SensingMemory-Aid
Retrospective
Core services on cloudFig. 2. The proposed system architecture

3.4

Agent-Assisted Mind Sensing

This system promotes the expression of the inner mind of each elderly person,and record it for high-quality care and communication to be done later. Especially, we use an external agent service to understand the internal conditions suchas the elderly’s psychology and physical conditions and at the same time create richer communication. The agent inquires of the user about various internal
conditions, then, stores the obtained answer in the database.
3.4.1 User Interaction
The agent gives questions according to the situation of the user. For example,
at a ﬁxed time every day, the agent asks the user about the physical condition
at morning wake-up and about what happened today before sleeping at night.
Furthermore, in cooperation with the existing system, the proposed system consider the observable user situation. Speciﬁcally, based on the user’s activity ofdaily living recognized by the existing system, the agent inquires accordingly atthat time. The proposed system sends messages to the user using the agent service existing as an external service, triggered by the detection from the previous
research.

Memory Aid Service Using Mind Sensing

359

3.4.2 Record of Internal States
This service asks the user to input his self-talk using a smartphone in a waythat he answer questions of the agent, or spontaneously speak what he thinkat the moment the user likes. The user input messages by text or voice using asmartphone, and the message is automatically collected through the Web-API
at the time when transmitted. The data structure of the message we manage is
as follows.
message ID : ID
datetime : The time when the user sended a message
contents : A message sent by a senderfrom: An user who sent a message
to: An user who received a message
dataType : The type of the message (text, image, etc.)
category : What type of topic the message is

Category is a schema for users to classify their messages in the retrospective system, and it is empty when each message is ﬁrst stored. Accordingly, weorganize information from the viewpoint of when, who, and what, so that, it ispossible to grasp them based on various queries.
3.5

Memory-Aid Retrospective

This system helps the user to remember the memory by looking back on themessages of the mind entered by the user. While interacting with the agent, theuser can perform the following operations.
3.5.1 Data Cleansing
The user calibrates and classiﬁes the most recently entered messages. Initially, alist of messages for one day is displayed on the screen, and the user selects one
by one. This is to correct unexpected conversion errors and sentence delimiters
caused by voice input. If there is nothing to change, the user continues theretrospective without doing anything here. When calibration is completed, theuser then sorts the messages into the categories deﬁned by the user himself. It is
not classiﬁed at the time when the user ﬁrst inputs the message. The informationof the messages on the database is updated when the user classiﬁes them. By
arranging and adding information to the original data in this way, they becomericher contents, and it is useful for later searching under various conditions.
3.5.2 Search
When the user wants to remember necessary information, the user can search fordata by three conditions: category, word and period. It is possible to conﬁrm thedetails of messages from each search result, and the user can acquire necessary
information at any time.

360

H. Maeda et al.

Search by classification
The user can search tweets according to the classiﬁcation of them. Whenthe user selects a classiﬁcation, the user can retrieve a list of the relevant
tweets. The user can conﬁrm.
Search by word
The user can get a list of tweets based on the words of which the user isinterested.
Search by date
When the user designates a period, he can acquire the tweet inputted within
the period. Therefore, the user can remember what happened at that time.
3.5.3 Personal Settings
The proposed system manages the setting information of each user in the
database. The user can change various system settings on the screen. In thissection, we will describe the following three functions.
Classification settings
The user can deﬁne the types of categories as described in Sect. 3.5.1. Thischange is reﬂected when the user classiﬁes the data.
Agent settings
In the proposed system, we use the previous research MPAgent as a virtual
agent. Since the MPAgent system generates an agent from an arbitrary
facial picture, users can customize to their preferred agent. It is possible torealize a conversation with a person familiar with the user, such as relatives,children, close friends, etc.
Sensing settings
The user can set a trigger for the agent to interact with the user. By setting
the time and the place that the user wishes to notify, and what kind ofcontent he wishes to be notiﬁed. Since notiﬁcations such as simple greetings,
forgotten records, and important schedules arrive when the user needs it,
the user will not have to worry about forgetting. By intervention in life using
an agent, we can expect that elderly people can easily reconsider their lives.

4

Prototype Development

In this paper, we implement a prototype system that is equipped with the function of mind sensing and retrospective consisting of server-client system. In particular, we build a mind sensing service with the LINE Messaging API [4] for
acquiring the self-talk of the elderly. To accumulate the internal state of elderly
people obtained by mind sensing, we prepare a database and implement the
Web-API for inserting and retrieving data. Furthermore, in collaboration withan agent which is an external service, we design a memory-aid service for a daily
retrospective to help elderly memory. The following sections describe the detailsof the prototype.

Memory Aid Service Using Mind Sensing

4.1

361

Agent-Assisted Mind Sensing

In previous research, we have been developed the communication agent service
using LINE application. This service is specialized in the function of interactive
interaction and implemented as a web service. When an event such as message
transmission/reception occurs, a POST request can be sent to the speciﬁed URL.
In this paper, we develop a database which manages the internal states of users,
and APIs to store and retrieve data. In mind sensing, when various events suchas the result of activity recognition occur, or at a speciﬁc time, a message asking
the user’s mind is sent to the user on LINE. When a message is replied from theuser, a POST request is sent to the created API and the content of the message,
the time when it was sent and so on, is accumulated in the database.
4.2

Memory-Aid Retrospective

When the user starts using the retrospective service, a menu screen is displayed,
and the user can select any one of buttons “Daily Retrospective”, “Search”, and“Settings”.
When the user begins “Daily Retrospective”, the messages accumulated
in the database are acquired by the API and displayed as a list. When theuser selects any one of the messages, the list screen changes to the modiﬁcation/classiﬁcation screen. If the message is changed here, the data on the
database is updated by the API. After this work is ﬁnished, the screen returns
to the list screen again and the list of the messages to be displayed is also dynamically changed. The user can visually see the message and prevent forgetfulness
by remembering what happened recently. Figure 3 shows the screenshot when
the user experiences the retrospective function. We deploy MPAgent as an agent
that interacts with the user on the left of the screen. On the right of the screen,
we prepare a user interface for the user to perform various operations. In thisﬁgure, the contents the agent is talking about and a list of messages entered by
the user are displayed.
When the user starts “search”, the list screen changes to the search screen.
The user here can search past messages by category, word, and period, andeasily switch search type by switching the tabs. In the search by category, it ispossible to acquire the messages based on the category decided by the user at
the time of daily retrospective. In the search by word, it is possible to acquire themessage including the speciﬁed word, and in the case of the search by the period,
the message within the speciﬁed period can be acquired. The user can retrieve
necessary information at any time, so there is no worry about forgetfulness.
4.3

Development Environment

The development environment is as follows.
– Development language: Java, JavaScript, HTML, CSS– Database: MongoDB-3.4.17– Web server: Apache Tomcat 7.0.77– Web service framework: Jersey framework

362

H. Maeda et al.

Fig. 3. Screen shot of the proposed system

5

Preliminary Experiment

We conducted preliminary experiments targeting actual elderly people using
prototypes. The purposes of the experiment are to evaluate the quality at thetime of use for each function by having the user use the system. In this section,
we explain the outline of the experiment using the proposed system, and consider
the results based on the two requirement shown in ??.
5.1

Outline

The target of the experiment is a male in his 60s living with a couple. Whenthe mind sensing system detects waking up of the tester, it asks the physicalcondition at that time. Similarly, when the system detects the activity of thetester, it makes a question depending on the situation. At the end of the evening,
the tester use a retrospective system to look back on the day’s actions. The
experiment period is 38 days from December 25, 2018 to January 31, 2019.
5.2

Result and Discussion

We conﬁrm that there were messages about the pain of a body and physical condition of the tester. Such information cannot be obtained by traditional sensors,
and we fulﬁll the requirement of mind sensing. In other words, it is possible toobtain internal states by mind sensing. On the other hand, the tester recorded
measured values such as blood pressure and weight. It is expected to be able todevelop further services by analyzing the information.

Memory Aid Service Using Mind Sensing

363

From the viewpoint of usability, the tester said to input by speech recognition
is more convenient than that by ﬂick operation. However, due to many operations
until entering his internal states, it may be somewhat diﬃcult for elderly people
with advanced dementia to deal with the system.
In mind sensing service, the number of messages sent from the tester to theservice was 337, and 331 of them were text messages, and 6 were images. In
addition, 229 out of 337 messages were classiﬁed by the tester in retrospective
service. In this preliminary experiment, the tester deﬁned the six classiﬁcations,and the number of messages classiﬁed into each is as shown in the table below(Table 1).
Table 1. The number of messages in each category
Event Schedule Health Study Memorandum Other
81

36

83

19

32

21

Here, messages classiﬁed into multiple categories are counted for each category. This result indicates that we can grasp what the tester is interested in. The
reason why all the messages are not classiﬁed is that the user forgot to look backon the day or that the classiﬁcation was not performed correctly due to systematic error. We should consider the need to prompt users to do retrospective and,of course, we must increase the reliability of the system.
The tester said that the function of short-term retrospective was eﬀective inthat when he was relaxing, it was able to correct typos of messages and realized
what they did. On the other hand, there were opinions on the diﬃculty in using
the user interface, such as many operations.
Regarding the search function, the search most frequently used by the tester
was to search messages classiﬁed as “Memorandum”. This was why he accumulated what he might forget in advance, and used it as a way to prevent
forgetfulness by checking later. In this way, by the search function, the tester
analyzed the number and contents of the classiﬁed messages.
Through the preliminary experiments, we conﬁrm that the proposed system
can be expected to be extremely eﬀective forgetfulness.

6

Conclusion

In this paper, we proposed a new memory-aid service supporting elderly whofeel uneasy about the decline in cognitive function. Especially, the proposedservice obtains and records the internal states of the user by mind sensing. Then,
users and agents will look back on the day based on the accumulated data. We
also implement prototyping and conﬁrm the eﬀect of the proposed service by
preliminary experiments on an actual elderly person. For future research, wewill extend the proposed system to support multiple users and evaluate in detail364

H. Maeda et al.

through experiments on many more people. In addition, we will develop a service
that acquires new insights by analyzing accumulated internal states of elderly.
Acknowledgements. This research was partially supported by the Japan Ministry of Education, Science, Sports, and Culture [Grant-in-Aid for Scientiﬁc Research(B) (16H02908, 18H03242, 18H03342), Grant-in-Aid for Scientiﬁc Research (A)(17H00731)], and Tateishi Science and Technology Foundation (C) (No.2177004).

References
1. Di Cesare, G., Di Dio, C., Marchi, M., Rizzolatti, G.: Expressing our internal states
and understanding those of others. Proc. Nat. Acad. Sci. 112(33), 10331–10335(2015)
2. Government of Japan: Annual report on the aging society, June 2018. http://www.
cao.go.jp/
3. Government of Japan: Community-based integrated care systems, June 2018.
https://www.mhlw.go.jp/index.html. (in japanese)
4. LINE Developers: Messaging api, July 2018. https://developers.line.biz/ja/services/messaging-api/
5. Nakatani, S., Saiki, S., Nakamura, M., Yasuda, K.: Generating personalized virtual
agent in speech dialogue system for people with dementia. In: Duﬀy, V.G. (ed.)
DHM 2018. LNCS, vol. 10917, pp. 326–337. Springer, Cham (2018). https://doi.
org/10.1007/978-3-319-91397-1 27
6. Tamamizu, K., Sakakibara, S., Saiki, S., Nakamura, M., Yasuda, K.: Machine learning approach to recognizing indoor activities based on detection of environmental
change. In: 11th World Conference of Gerontechnology (ISG2018), vol. 17, p. 118s,st. Petersburg, USA, May 2018
7. Tokunaga, S., Tamamizu, K., Saiki, S., Nakamura, M., Yasuda, K.: VirtualCareGiver: personalized smart elderly care. Int. J. Softw. Innov. 5(1), 30–43 (2016).
https://doi.org/10.4018/IJSI.2017010103. http://www.igi-global.com/journals/abs
tract-announcement/158780