一般社団法人 電子情報通信学会
THE INSTITUTE
OF ELECTRONICS,
社団法人 電子情報通信学会
INFORMATIONAND COMMUNICATIONENGINEERS
THE INSTITUTE
OF ELECTRONICS,

信学技報
IEICE 信学技報
Technical Report
SC2018-40(2019-03)
TECHNICALREPORT OF IEICE.

INFORMATION AND COMMUNICATION ENGINEERS

エージェントによる「こころ」センシングを活用した物忘れ支援サービスの提案
前田

晴久†

佐伯

幸郎†

中村 匡秀†

安田清††† 神戸大学 〒 657–8501 神戸市灘区六甲台町 1–1†† 大阪工業大学 〒 535–8585 大阪市旭区大宮 5–16–1
E-mail: †haruhisa@ws.cs.kobe-u.ac.jp, †††masa-n@cs.kobe-u.ac.jp
あらまし

我々の研究グループでは認知機能に不安や問題を抱える在宅高齢者を支援するために，センサを用いて宅内の環境情報を取得し，ユーザの日常動作などのコンテキストを推定するスマートホームサービスを開発している．しかしながら，センサによって観測できる高齢者の情報は限定的である．より個人に即したケアを行うためには，ユーザの心理や体調などの内面的な心の内も併せて取得，記録する必要がある．さらに，物忘れ支援のためには，記録した情報をいつでも取り出せるようにすることが重要である．本研究では，高齢者の心の内を記録し，記録した情報を活用できる新しい物忘れ支援サービスを提案する．具体的には，従来の環境センシングによって推定されたコンテキ
ストをトリガとし，仮想エージェントが高齢者に問いかけて，その心の内を言葉に外化させ記録する「こころ」セン
シングを開発する．また「こころ」センシングによって記録された情報は，高齢者と仮想エージェントの対話に基づ
く校正・分類を通してクレンジングされる．クレンジングされた情報は，分類や任意のキーワードで取り出すことができる．提案サービスによって，高齢者はいつでもどこでも情報を記録し，取り出すことができるようになる．
キーワード スマートヘルスケア，対話エージェント，「こころ」センシング

Proposal of Memory Aid Service Using Mind Sensing by Virtual Agent
Haruhisa MAEDA† , Sachio SAIKI† , Masahide NAKAMURA† , and Kiyoshi YASUDA††† Kobe University Rokkodai-cho 1–1，Nada-ku，Kobe，Hyogo，657–8501 Japan†† Osaka Institute of Technology 5–16–1, Omiya, asahi-ku, Osaka, 535–8585 Japan
E-mail: †haruhisa@ws.cs.kobe-u.ac.jp, †††masa-n@cs.kobe-u.ac.jp
Abstract Our research group has been studying smart home services for elderly people, which detect their daily
activities based on the environmental sensors in the house. However, such sensors can only obtain limited information. To execute more optimized care, we must retrieve not only external events but internal states. Furthermore,to support memory aid, it is important to be able to retrieve the recorded information at any time. In this paper, we
propose a new memory aid service, which records the self-talk of elderly people and utilizes the recorded information. Specifically, we develop mind sensing, which is to externalize the inside of elderly’s heart as a sentence using
a virtual agent. Then, the recorded information by mind sensing is cleansed through calibration and classification
based on dialogue between an elderly person and a virtual agent. These information can be retrieved by classificationor arbitrary keywords. In this way, the proposed service enable elderly to record and retrieve what they thought
anytime anywhere.
Key words Smart health-care，Dialogue agent，Mind Sensingof Japan, the number of elderly people over the age of 65 is

1. Introduction

35.15 million in 2017, which is 27.7% of the total popula-

Japan is currently facing hyper-aging society, which causes

tion [1]. Due to the long average life and the low birthrate,

many social issues. According to a report by the Government

the number of elderly over 65 is increasing in Japan and will

- 19 -—1—

This article is a technical report without peer review, and its polished and/or extended version may be published elsewhere.

Copyright ©2019 by

IEICE

reach 40% of the total in 2050. There are many challenges,

result, we can grasp the internal state of the elderly and use

for example, the shortage of human resources and facilities

it for more sophisticated care.

in the nursing care field. Therefore, the government is focus-

Retrospective is that the elderly and the agent review theing on supporting home care. However, there is a possibility

day at an arbitrary sense of time utilizing the accumulated

that the burden on family caregivers increases in home care.

internal states of them. While viewing the list of messages

Furthermore, the elderly at home has various diﬃculties inentered by elderly by mind sensing, they calibrates and clas-

daily life due to the decline of cognitive function. In par-sifies these messages one by one. Thus, elderly people canticular, as elderly people are getting forgetful, they can notprevent forgetfulness by looking back at what happened re-

do what they had done until then, so that they feel strong

cently. Furthermore, they can search the internal states later

anxiety, confusion, and deterioration of self-esteem. In orderon the basis of classifications, words, period, and can retrieve

to support these elderly people, many research on assistive

them whenever necessary.
In this paper, we also implement a prototype system thattechnology has been carried out in various organizations in

is equipped with the function of mind sensing and retrospec-

recent years.
Our research group has been developing the activity recog-

tive. Through the preliminary experiments, we confirm thatnition service based on the environmental change [2].This ser-the proposed system can be expected to be extremely eﬀec-

vice read the variations of environmental sensors in elderly’s

tive forgetfulness.

home and detects their activities by machine learning tech-

2. Preliminary

nology. The detected activities are notified to the mobile
terminals of elderly, and they respond to the notifications

2. 1 Elderly Care at Home

based on the their current situation. This service recognizes

Along with the aging society, single households whose headelderly’s activity and at the same time realizes the creation ofis over 65 years old is increasing in Japan. However, it is diﬃ-

communication opportunities for them. On the other hand,cult to accept and carry out the care for all elderly at medical

the previous research is limited to the activity recognition

facilities. On the other hand, there are a lot of elderly thatthat can be observed with traditional sensors. To execute

prefer to receive care at their own home or their family smore personalized care for elderly, it is important to obtainhome rather than to receive at nursing home or care house.

not only such external events but the internal states such as

Against this background, in recent years, many attempts tohuman emotions and physical condition. The internal statecare for elderly people at home have been made in various

is directly linked to human health, and it is important infor-

fields. Japanese Government is focusing on training humanmation for monitoring [3]. However, the internal states areresources for home health care and promoting cooperation inusually acquired through inquiries and counseling by experts

each local government. The general companies are startingthat can not be done permanently at home. Moreover, it is

to work hard on developing new services that relieve elderly

diﬃcult to measure the changes of internal states by sensors.

of loneliness using a communication robot. These eﬀort aims

The purpose of this research is to support elderly people

to improve the quality of life (QOL) of the elderly.
2. 2 Virtual Agent

who feel uneasy about forgetfulness. In this paper, especially,
we propose a new memory-aid service using virtual agents,

In elderly care, it is expected to use care robots that utilize

which are 3D humanoid robots. The proposed service mainly

various assistive technologies to assist daily living. Among

has the following two functions.

these technologies, virtual agent is a 3D friendly humaniod•

Agent-Assisted Mind Sensing: A fucntion thatrobot displayed on the PC screen, and can use speech recog-

an agent asks users about their minds, to require andnition and speech synthesis technology to interact with users.

record the internal states of them

By introducing such an agent to the medical field, it is ex-•

Memory-Aid Retrospective: A fucntion thatpected that instead of an actual care provider, it becomes aan user and an agent look back the day using thecommunication partner of the elderly and reduce the burden

data recorded by mind sensingof caregiver. Our research group has been studying MPAgent

In order to solve the limitaion of the previous research, weusing virtual agent technology [4]. Fig.1 shows a screen shot

propose mind sensing that a virtual agent directly asks theof MPAgent. This service creates a 3D model of a face andelderly about their internal states. With various events as adynamically generates an agent based on the feature points

trigger, the agent sends a message to the elderly mobile ter-of the face acquired from the facial photograph. It also real-

minal asking for the current physical condition and emotion.

izes a realistic dialogue by lips movement and changing theAfter the user answers the question, the detailed information

expression by the utterance. Delegating the communicationof the answer is automatically stored in the database. As acare with MPAgent, a human caregiver can concentrate on- 20 -—2—

obtained by the conventional monitoring system. Also, ithuman-centric tasks that cannot be done with ICT.

contributes to the forgetfulness of elderly based on the accumulated internal condition. The proposed method realizes
the following two system requirements.
•

R1: Externalize the mind of the elderly intowords
図 1 Screen shot of MPAgent•

R2: Accumulate the words and retrieve themat any time later
2. 3 Elderly Monitoring Service

We implement a mind sensing service with the function of

As one of the technologies to support home care for the

R1, that acquire the internal states of elderly by the virtual

elderly, a monitoring system at home using ICT is consid-

agent. We also implement a retrospective service depend-ered promising. In the general process of monitoring, first,

ing on the function of R2, that have elderly to look back on

the system senses real-world data, and grasp the situation

the acquired internal state using a virtual agent and lead toof the elderly and their environment. The data to be sensed

prevent forgetfulness.

includes biometric data such as pulse and acceleration of arm

3. 2 Use Case Scenario

or leg, and environmental data such as temperature, humid-

Here, we explain how the user actually uses the proposed

ity, and illuminance. Second, using the obtained data, thesystem. When the user starts the mind sensing service, firstsystem recognizes the elderly’s activities of daily living. Fi-of all, the system registers information of sensors installed innally, based on the recognized activity, the system leads tothe house in order to link with sensor data. By doing this,actual care. Such a monitoring system reduces the eﬀorts

various notification is sent to the user triggered by user’sof nursing care and aims at improving the quality of care

activity recognition, so the user returns the current feeling,that only human beings can. However, in the conventional

what happened on that day, future schedule, and so on.

monitoring system, there were issues such as introduction

When doing retrospective service, while interacting withcost, invasiveness problem, failure to achieve personal adap-

the agent, the user corrects and classifies the data input sotation using the collected data, and lack of communication

far. By classifying the contents of the data into classes de-to the elderly. In order to solve these problems, in previous

fined by the users themselves, they are used for later retrieval.

research, our research group has proposed and developed an

If the user wishes to remember what happened in the past,elderly state notification service [2] using activity recognition

he can use various queries to search and confirm the data hebased on environmental change. This service consists of anentered by mind sensing, at any time. In addition, if the user

activity recognition service that predicts elderly’s activity by

wants to change personal settings, the user can make settings

using environment sensor data, and a care execution service

such as the agent’s questioning timing and user informationthat makes notifications based on predicted activity usingchange.

SMS application. With this system, it is possible to record

3. 3 System Architecture

activity by non-invasive sensing and to create communica-

In this research, to support memory aid of elderly people,

tion opportunities for the elderly. On the other hand, even

we build an architecture that can manage important informa-

by utilizing various sensors and devices, it is only detecting

tion for the elderly’s life. Fig.2 shows the system architecture

external events such as environmental data and user’s ex-of the proposed method. The proposed system is made up ofpressions and behaviors, and it is impossible to observe thetwo services. One is Agent-Assisted Mind Sensing Ser-internal states within the user’s mind. The internal state isvice that acquires and records the feelings of the user. The

directly related to human health, and it is important infor-

agent triggers various events such as the result of the user’s

mation for monitoring. Obtaining internal states helps toactivity detection presumed by the existing system and asks

grasp the more detailed status of the elderly and helps more

the user about his mind using a smartphone. Then, the ser-sophisticated care. In addition to acquiring conventional ex-vice obtains the answer from the user and stores it in the

ternal events, it is important for the monitoring system todatabase. By managing such internal state for users, we willcontinue to monitor the internal state.

use it for other services. The second is Memory-Aid Ret-

3. Proposed Methods

rospective Service based on the data recorded by mind
sensing. In this service, while interacting with the agent, the3. 1 Requirement

user looks back inside the mind that the user himself inputs

In this research, we propose a service to acquire and ac-on one day. As a result, the user can remember what hap-

cumulate the internal state of the elderly who could not be

pened on the day and what he thought. Furthermore, the- 21 -—3—

user can search the data of his mind based on various queries

message_ID : IDand check them at any time. Since elderly people can retrieve

datetime

important information anytime in their daily lives, it is pos-fromsible to ease the anxiety of forgetfulness. It is expected thesetoretrospective views will lead to memory aid of elderly. We
discuss these two services in detail in chapters 3. 4 and 3. 5

: The time when the user sended a message: An user who sent a message: An user who received a message

dataType: The type of the message (text, image, etc.)

contents: A message sent by a sendercategory: What type of topic the message isbelow.

SensorBox

Activity recognition systemin the retrospective system, and it is empty when each mes-

Activity
Recognition

sage is first stored. Accordingly, we organize information

SensorDB

Monitorenvironment
Exposeinternal states
SMS app on
smartphone

Ask statesAPI

3. 5 Memory-Aid Retrospective

Record states

This system helps the user to remember the memory by

Mind
DB
API

Virtual Agenton PC

possible to grasp them based on various queries.

Mind
Sensing

Send dialogue
Review pastmessagesfrom the viewpoint of when, who, and what, so that, it is

Trigger sensing

looking back on the messages of the mind entered by the

Memory-Aid
Retrospective

Querymessagesuser. While interacting with the agent, the user can performthe following operations.

Core services on cloud

3. 5. 1 Data Cleansing

図 2 The proposed system architecture

The user calibrates and classifies the most recently enteredmessages. Initially, a list of messages for one day is dis-

3. 4 Agent-assisted Mind Sensing

played on the screen, and the user selects one by one. This

This system promotes the expression of the inner mind

is to correct unexpected conversion errors and sentence de-of each elderly person, and record it for high-quality carelimiters caused by voice input. If there is nothing to change,

and communication to be done later. Especially, we use an

the user continues the retrospective without doing anything

external agent service to understand the internal conditions

here. When calibration is completed, the user then sorts the

such as the elderly’s psychology and physical conditions andmessages into the categories defined by the user himself. Itat the same time create richer communication. The agent

is not classified at the time when the user first inputs theinquires of the user about various internal conditions, then,message. The information of the messages on the databasestores the obtained answer in the database.

is updated when the user classifies them. By arranging and3. 4. 1 User Interaction

adding information to the original data in this way, they be-

The agent gives questions according to the situation ofcome richer contents, and it is useful for later searching underFor example, at a fixed time every day, thevarious conditions.

agent asks the user about the physical condition at morning

3. 5. 2 Search

the user.

wake-up and about what happened today before sleeping at

When the user wants to remember necessary information,night. Furthermore, in cooperation with the existing system,

the user can search for data by three conditions: category,the proposed system consider the observable user situation.

word and period. It is possible to confirm the details ofSpecifically, based on the user’s activity of daily living recog-

messages from each search result, and the user can acquire

nized by the existing system, the agent inquires accordingly

necessary information at any time.

at that time. The proposed system sends messages to the

In the search by category, the user can search messages

user using the agent service existing as an external service,according to the category of them. When the user selects

triggered by the detection from the previous research.

one of the categories, the user can retrieve a list of the rel-

3. 4. 2 Record of Internal States

evant messages. In the search by word, the user can get aThis service asks the user to input his self-talk using alist of messages based on the words of which the user is in-smartphone in a way that he answer questions of the agent,terested. In the search by period, when the user designatesor spontaneously speak what he think at the moment thea period, he can acquire the messages inputted within theuser likes. The user input messages by text or voice using

period. Therefore, the user can remember what happened ata smartphone, and the message is automatically collectedthat time.

through the Web-API at the time when transmitted. The

3. 5. 3 Personal Settings

data structure of the message we manage is as follows.

The proposed system manages the setting information of

Category is a schema for users to classify their messages

each user in the database. The user can change various sys-- 22 -—4—

4. 2 Memory-Aid Retrospective

tem settings on the screen. In this section, we will describe

When the user starts using the retrospective service, athe following three functions.
•menu screen is displayed, and the user can select any one

Classification settings

The user can define the types of categories as described inof buttons "Daily Retrospective", "Search", and "Settings".
When the user begins "Daily Retrospective", the mes-

3. 5. 1. This change is reflected when the user classifies thesages accumulated in the database are acquired by the API

data.
•

and displayed as a list.

Agent settings

When the user selects any one

In the proposed system, we use the previous research MPA-of the messages, the list screen changes to the modifica-

gent as a virtual agent. Since the MPAgent system generates

tion/classification screen. If the message is changed here, thean agent from an arbitrary facial picture, users can customize

data on the database is updated by the API. After this work

to their preferred agent. It is possible to realize a conversa-

is finished, the screen returns to the list screen again andtion with a person familiar with the user, such as relatives,

the list of the messages to be displayed is also dynamicallychildren, close friends, etc.

changed. The user can visually see the message and pre-•vent forgetfulness by remembering what happened recently.

Sensing settings

The user can set a trigger for the agent to interact with theFig.3 shows the screenshot when the user experiences the ret-user. By setting the time and the place that the user wishes

rospective function. We deploy MPAgent as an agent thatto notify, and what kind of content he wishes to be noti-

interacts with the user on the left of the screen. On thefied. Since notifications such as simple greetings, forgottenright of the screen, we prepare a user interface for the user

records, and important schedules arrive when the user needs

to perform various operations. In this figure, the contents

it, the user will not have to worry about forgetting. By in-

the agent is talking about and a list of messages entered bytervention in life using an agent, we can expect that elderly

the user are displayed.

people can easily reconsider their lives.

4. Prototype Development
In this paper, we implement a prototype system that is
equipped with the function of mind sensing and retrospective
consisting of server-client system. In particular, we build amind sensing service with the LINE Messaging API [5] for
acquiring the self-talk of the elderly. To accumulate the internal state of elderly people obtained by mind sensing, we
prepare a database and implement the Web-API for inserting
and retrieving data. Furthermore, in collaboration with an

図 3 Screen shot of the proposed systemagent which is an external service, we design a memory-aid
service for a daily retrospective to help elderly memory. The
following sections describe the details of the prototype.

When the user starts "search", the list screen changes tothe search screen. The user here can search past messages

4. 1 Agent-assisted Mind Sensing
In previous research, we have been developed the communication agent service using LINE application. This service is
specialized in the function of interactive interaction and implemented as a web service. When an event such as message
transmission/reception occurs, a POST request can be sent
to the specified URL. In this paper, we develop a databasewhich manages the internal states of users, and APIs to store
and retrieve data. In mind sensing, when various events suchas the result of activity recognition occur, or at a specificby category, word, and period, and easily switch search typeby switching the tabs. In the search by category, it is possible to acquire the messages based on the category decided
by the user at the time of daily retrospective. In the search
by word, it is possible to acquire the message including thespecified word, and in the case of the search by the period,
the message within the specified period can be acquired. The
user can retrieve necessary information at any time, so there
is no worry about forgetfulness.

time, a message asking the user’s mind is sent to the user
on LINE. When a message is replied from the user, a POST
request is sent to the created API and the content of the message, the time when it was sent and so on, is accumulated in
the database.

- 23 -

4. 3 Development Environment
The development environment is as follows.
•

Development language: Java, JavaScript, HTML, CSS•

Database: MongoDB-3.4.17•

Web server: Apache Tomcat 7.0.77

—5—

•

correctly due to systematic error. We should consider the

Web service framework: Jersey frameworkneed to prompt users to do retrospective and, of course, we

5. Preliminary Experimentmust increase the reliability of the system.

We conducted preliminary experiments targeting actual el-

The tester said that the function of short-term retrospec-

derly people using prototypes. The purposes of the experi-

tive was eﬀective in that when he was relaxing, it was able

ment are to evaluate the quality at the time of use for eachto correct typos of messages and realized what they did. On

function by having the user use the system. In this section,

the other hand, there were opinions on the diﬃculty in using

we explain the outline of the experiment using the proposed

the user interface, such as many operations.
Regarding the search function, the search most frequently

system, and consider the results based on the two requirement shown in 3. 1.

used by the tester was to search messages classified as "Mem-5. 1 Outline

orandum". This was why he accumulated what he might

The target of the experiment is a male in his 60s living

forget in advance, and used it as a way to prevent forgetful-

with a couple. When the mind sensing system detects wak-ness by checking later. In this way, by the search function,

ing up of the tester, it asks the physical condition at thatthe tester analyzed the number and contents of the classified

time. Similarly, when the system detects the activity of themessages.

tester, it makes a question depending on the situation. At

Through the preliminary experiments, we confirm that thethe end of the evening, the tester use a retrospective system

proposed system can be expected to be extremely eﬀective

to look back on the day ’s actions. The experiment period isforgetfulness.

38 days from December 25, 2018 to January 31, 2019.

6. Conclusion

5. 2 Result and Discussion
We confirm that there were messages about the pain of aIn this paper, we proposed a new memory-aid service sup-body and physical condition of the tester. Such information

porting elderly who feel uneasy about the decline in cog-cannot be obtained by traditional sensors, and we fulfill thenitive function. Especially, the proposed method obtains

requirement of mind sensing. In other words, it is possible toand records the internal states of the user by mind sens-

obtain internal states by mind sensing. On the other hand,ing. Then, users and agents will look back on the day based

the tester recorded measured values such as blood pressure

on the accumulated data. We also implement prototyping

and weight. It is expected to be able to develop further ser-and confirmed the eﬀect of the proposed method by prelimi-

vices by analyzing the information.

nary experiments on an actual user. For future research, we

From the viewpoint of usability, the tester said to input
by speech recognition is more convenient than that by flick

will extend the proposed system and evaluate in detail byexperiments on many more people.
謝辞

operation. However, due to many operations until entering

この研究の一部は，科学技術研究費（基盤研究 B

his internal states, it may be somewhat diﬃcult for elderly

16H02908，18H03242, 18H03342，基盤研究 A 17H00731），おpeople with advanced dementia to deal with the system.

よび，立石科学技術振興財団の研究助成を受けて行われている．

In mind sensing service, the number of messages sent fromthe tester to the service was 337, and 331 of them were text
messages, and 6 were images. In addition, 229 out of 337
messages were classified by the tester in retrospective service. In this preliminary experiment, the tester defined thesix classifications, and the number of messages classified into
each is as shown in the table below.
表 1 The number of massages in each category

Event Schedule Health Study Memorandum Other
81

36

83

19

32

21

Here, messages classified into multiple categories arecounted for each category. This result indicates that we cangrasp what the tester is interested in. The reason why all
the messages are not classified is that the user forgot to look

文献[1] 内閣府，“平成 30 年度版高齢社会白書,” http://www.cao.go.
jp/，June 2018．
[2] K. Tamamizu, S. Sakakibara, S. Saiki, M. Nakamura,and K. Yasuda, “Machine learning approach to recognizing indoor activities based on detection of environmental change,” 11th World conference of Gerontechnology (ISG2018), vol.17, p.118s, May 2018. St. Petersburg,
USA.
[3] G. Di Cesare, C. Di Dio, M. Marchi, and G. Rizzolatti,“Expressing our internal states and understanding those ofothers,” Proceedings of the National Academy of Sciences,vol.112, no.33, pp.10331–10335, 2015.
[4] S. Nakatani, S. Saiki, M. Nakamura, and K. Yasuda, “Generating personalized virtual agent in speech dialogue system for people with dementia,” Digital Human Modeling2018 (DHM 2018), Held as Part of HCI International 2018,
vol.LNCS 10917, pp.326–337, Springer, July 2018. Las Vegas, USA.
[5] LINE Developers, “Messaging api,” https://developers.
line.biz/ja/services/messaging-api/, July 2018.

back on the day or that the classification was not performed- 24 -—6—