Proceedings of iiWAS2019

iiWAS2019 Papers

Rule-Based Inquiry Service to Elderly at Home
for Efficient Mind Sensing
Haruhisa Maeda1 , Sachio Saiki1 , Masahide Nakamura1,2 , Kiyoshi Yasuda3
1 Graduate School of System Informatics, Kobe University, Japan
2 Riken AIP, Tokyo

3 Faculty of Informatics, Osaka Institute of Technology, Japan

haruhisa@ws.cs.kobe-u.ac.jp,sachio@carp.kobe-u.ac.jp,masa-n@cs.kobe-u.ac.jp,fwkk5911@mb.infoweb.ne.jp

ABSTRACT
To support in-home long-term care, we are studying techniques of
Mind Sensing, which externalizes internal states of elderly people
as words through conversations with agents or robots. We havepreviously developed a prototype system of Mind Sensing, integrated with an activity recognition system and an LINE chatbot.
However, the system was tightly coupled with the fixed systems,
it was difficult to add or change the setting of questions from thechatbot to individual elderly people.
In this paper, we propose the Mind Sensing Service, which allows
a service operator to define and manage the questions flexibly, andto automate the delivery of the questions and the collection of theanswers. The proposed service consist of two elements: actions andrules. An action defines the contents of specific questions such as
what message is sent to which elderly people. A rule defines theconditions on when, where, and by what event, in order to executethe action. The proposed service makes it possible to implement
more systematic and flexible Mind Sensing.

CCS CONCEPTS• Information systems → RESTful web services; • Humancentered computing → Ubiquitous and mobile computing
systems and tools; • Applied computing → Health care information systems;

KEYWORDS
smart healthcare, smart home, Web service, mind sensing
ACM Reference Format:
Haruhisa Maeda1 , Sachio Saiki1 , Masahide Nakamura1, 2 , Kiyoshi Yasuda3 .
2019. Rule-Based Inquiry Service to Elderly at Home for Efficient Mind
Sensing. In The 21st International Conference on Information Integration andWeb-based Applications & Services, December 2–4, 2019, Munich, Germany.
ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3366030.3366114

1

INTRODUCTION

According to Japanese Government, the number of Japanese people over the age of 65 is 35.15 million in 2017, which is 27.7% of thePermission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
iiWAS2019, December 2–4, 2019, Munich, Germany© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-7179-7/19/12. . . $15.00
https://doi.org/10.1145/3366030.3366114

total population [3]. The number continues to increase and willreach 40% of the total in 2050. Similar situations are observed inmost developed countries in the world. Such a super-aging society causes chronic shortage of human resources, nursing facilities,
and the cost of social security. The Japanese government is therefore shifting the policy from the conventional facility care to inhome long-term care. It is ideal for all elderly people to live in their
beloved homes. However, they generally face with various difficulties in daily life, due to the decline of physical and cognitive functions. Also, the home care would pose a heavy burden on family
caregivers. In order to support these elderly people at home, theassistive technologies have been studied and developed in variousorganizations in recent years.
A promising technology to support the in-home long-term careis the elderly monitoring system using ICT. Exploiting sensors, wearable devices, and smart home technologies, the system tries to recognize daily activities of elderly at home. Various methodologies
have been studied, for example, the method with ambient sensing[9] [8], the method with accelerometers of mobile phones [4], themethods with wearable sensors [11], and the method using a smart
home with indoor positioning and power meters [10].
On the other hand, monitoring with sensors has a limitation thatthe system can detect only externally observable events. To performperson-centered home care, it is also important to recognize internal states of elderly people. Here, the internal state refers to a statusof a person that cannot be observed externally, including moods,
pains, conditions, desires, and intentions. Since the internal state is
directly linked to human health, it is important to monitor within
the home care [2]. However, the internal state is usually assessed
through inquiries and counseling by experts. Hence, it is challenging to monitor the internal state regularly at home.
In order to overcome the limitation, our research group is studying techniques of Mind Sensing (“kokoro” sensing in Japanese [6]).
It externalizes the internal states of elderly people as “words” via
conversations with agents or robots.
In our previous study [5], we have developed the memory-aid
service, which exploits a Mind Sensing method with an activity
recognition system [9] and the LINE chatbot. Triggered by the timeor an activity recognized in a smart home, the chatbot asks a user
(i.e., elderly person) a question that externalizes the internal state
as words. As the user answers the question by text (as well as voice
and images), the conversation is automatically recorded in a database with a timestamp. The service then provides the retrospective
process, where the user can review, correct, classify, and search the666

iiWAS2019 Papers
iiWAS2019, December 2–4, 2019, Munich, Germany

Proceedings of iiWAS2019
Haruhisa Maeda1 , Sachio Saiki1 , Masahide Nakamura1,2 , Kiyoshi Yasuda3

recorded information of own at any time. Thus, the service is designed for the memory-aid purpose of healthy elders as well as
people with cognitive impairment.
To make context-aware questions, our previous system [5] regularly monitors the log generated by the activity recognition system [9]. When a specific activity of a user is detected, the system
instructs the LINE chatbot to ask the user a question related to theactivity. However, the previous system was tightly coupled withthe activity recognition system and the LINE chatbot. That is, theMind Sensing can be only triggered by the specific activity recognition, and the inquiry is performed only by the LINE chatbot. Also,
all the questions were hard-coded within the program. Thus, ourprevious system lacks the flexibility, where it is quite difficult to
add or change the configuration of Mind Sensing, adapting to individual elderly people.
To cope with the limitation, this paper presents a new service
that allows more flexible and systematic Mind Sensing. Specifically, we introduce a rule-based inquiry system, to allow individual
users to define custom Mind Sensing. In the proposed service, a
Mind Sensing is defined by a rule and associated actions. An action
consists of the address of the target user, the concrete question toask, and the message service to deliver the question. A rule is either
time-based or event-based. A time-based rule executes a given setof actions based on designated time and frequency. A event-based
rule executes a given set of actions based on the event notified fromthe external smart home service. Based on the rules, the proposedservice automatically sends questions and collects answers to andfrom the target users.
The created rules and actions are managed within a database.
The user can easily add, update, and delete them through GUI of
Web application. The proposed service makes it possible to ask
questions with a high degree of freedom in Mind Sensing. Thus,
it can be expected to acquire various aspects of internal states of
elderly people. In this paper, we implement a prototype of the proposed service, and conduct a case study. Through the case study,
we check the feasibility of the proposed service, and collects user’s
feedback for further improvement.

2

PREVIOUS STUDY: MEMORY-AID SERVICE
FOR HOME ELDERLY CARE [5]2.1 Overview and System Architecture
In [5], we have developed the memory-aid service to support elderly people at home who are anxious about age-related forgetfulness, as well as cognitive impairment. The key idea is to provide
an external memory for a user (i.e., an elderly person), so that theuser can easily record whatever important for the user, and can retrieve the recorded information at any time. Major challenges were
(1) how to encourage the user to record the essential information(including the internal states) in the external memory, and (2) how
to use and maintain the recorded data for better quality life.
Figure 1 shows the system architecture of the memory-aid service, consisting of the following two functionalities:
Chatbot-based Mind Sensing is a mechanism that externalizes and records the information in user’s mind, through conversations with the LINE chatbot on a mobile phone. Triggered by timeor an event, the chatbot autonomously asks a question to the user

SensorBox

Activity recognition system
Activity
Recognition

SensorDB

Monitorenvironment

Trigger sensing

Exposeinternal states
SMS app on
smartphone

Ask states
API

Mind
Sensing

Record states

Mind
DB

Send dialogue
Review pastmessages

API
Virtual Agent
on PC

Query
messages

Memory-Aid
Retrospective
Core services on cloud

Figure 1: System architecture of memory-aid service [5]over the LINE app. We integrated the activity recognition system[9] in order to generate context-aware questions. Typical questions
include “What are you doing now?”, “How are you feeling today?”,“Did you sleep well?”, “What did you do today?” and “Where do
you want to go?”As the user responds to the chatbot by manual text input or the
speech-to-text input of the mobile phone, the text message is sent
to Web-API, and recorded in a database with a timestamp. Moreover, regardless of the questions, the user can spontaneously talkto the chatbot to record any information, such as memorandums,
schedules, appointments, and thoughts.
Memory-Aid Retrospective is an application that provides anopportunity where the user can look back the recorded information. Exploiting voice interaction with a virtual agent [7] and intuitive graphical user interface, the user can perform two types ofretrospective processes on a PC.
In the short-term retrospective process, the system displays thelist of fresh messages recorded within one day before now. While
viewing the list at the end of the day, the user can remember what
happened today. To maintain the quality of the data, the application tells the the user to correct typos and errors in the existing
records, or to add supplementary messages if necessary. Moreover,for every record, the user can associate user-defined categories that
explain the type of the information.
In the long-term retrospective process, the user can view andsearch all the existing records. By simple user interface assistedwith the virtual agent, the user can retrieve messages by category,
keywords, and date. Thus, the user can find important information whenever necessary, which would relieve the anxiety for theforgetfulness.

2.2

Prototype

We have implemented a prototype of the memory-aid service as a
Web application operated on both a smart phone and PC. In particular, we implemented the chatbot with the LINE Messaging API[1]. To manage the data obtained by the Mind Sensing, we deployed
MongoDB. For inserting and retrieving data via the Internet, we
implemented REST Web-API using the Jersey RESTful Web Services framework.
To evaluate the practical feasibility, the prototype system has
been deployed and operated within actual households. Especially,
Subject A (a 77-year-old man) and Subject B (70-year-old man) are
actively cooperating on our experiment. The current rules of the667Proceedings of iiWAS2019

iiWAS2019 Papers

Rule-Based Inquiry Service to Elderly at Home
for Efficient Mind Sensing

iiWAS2019, December 2–4, 2019, Munich, Germany

mind sensing are as follows. When the activity recognition system
detects the wake up of the subject, the chatbot on mobile phoneasks the physical condition of the subject. When the system detects any activity, the chatbot asks a question depending on thecontext. In the end of the day, the chatbot encourage to performthe retrospective process.

2.3

Limitation of the Previous System

The previous prototype system was tightly coupled with the activity recognition system and the LINE chatbot. Therefore, the capability of Mind Sensing was limited by the capability of thesesystems. In the previous system, based on the result of the activity recognition, the following message is automatically sent to theuser via the LINE application and stored in the database.

Figure 2: System architecture of proposed service

Figure 2 shows the system architecture of the proposed service.
In the proposed service, each Mind Sensing is defined by a rule,
specifying which question is sent, to whom, at when, by what
event, and with which message service. Once a rule is defined,
the service automatically sends the questions to the target users,and collects the answers. In Figure 2, there are various smart home
services to support elderly users at home, including activity recognition service, indoor positioning service, and anomaly detection
service. Each of these services generates events and manages theevent log. The proposed service can accept these events, and use the events in triggering conditions of Mind Sensing.
A rule specifies an enabling condition when the Mind Sensing
The item to represents an unique ID associated with each user.
is executed. The condition is based on either time or event. A timeThe item contents describes a question that aims to externalize
based rule is triggered when the designated time is arrived, while
the internal state of the user. In this example, the question contains
an event-based rule is executed when an event matching the conthe time, the location, and the recognized activity (i.e., “eating”),
dition is notified. Each rule is associated by a set of actions. An
the accuracy, and the type of sensor used for recognition.
action corresponds to an inquiry to a user, consisting of the adTo perform the Mind Sensing for a user, the system regularly
dress of the user, the concrete question to ask, and the message
monitors the database for the latest recognition result. When theservice to deliver the question. We adopt various messaging sersystem finds a new activity of the user, the system checks the type,
vices, including SMS (short messaging service), Email, and Slack, tothe location, and the time of the activity, and generates a question
inquiry the questions to the target user. By supporting interaction
by filling concrete values within a preset message template.
with various devices such as smartphone and PC, we can perform
On the other hand, the above procedure of generating the quesMind Sensing, according to the lifestyle of individual user.
tions is hard-coded within the system. If a user wants to change
As the user responds to the question in the natural language
the question, the user has to directly edit the message template
text, the answer is then recorded in the database with a timestamp.
in the source code, which is cumbersome. Furthermore, the Mind
The stored conversations between a user and a chatbot are later
Sensing can be triggered only by the activity recognition system.
used in services such as Memory-Aid Service. This allows users to
Although there exist various events captured by other smart homereview, correct, classify, and search the information they recorded
services (e.g., appliance operations, energy usage), they cannot bethemselves. In addition, through appropriate access control, they
cooperated with the Mind Sensing. Also, the user may prefer tocan be accessed by third parties such as doctors and caregivers foruse another message delivery service than LINE, for instance, anperson-centered care treatments.
email. However, the previous system does not allow it. Thus, theprevious system lack the extensibility and flexibility, which limits
3.2 Action
the capability of the Mind Sensing.
An action defines a configuration of concrete inquery of Mind Sensing. The configuration includes three items: targets specifies tar3 PROPOSED METHOD
get user(s) to inquiry, messageBody specifies the content of the
3.1 System Architecture
question message, and serviceType specifies a service to deliver
To achieve more flexible and systematic Mind Sensing, we prothe message.
pose the Mind Sensing Service in this paper. The proposed service
It is possible to specify multiple users in targets, and a desigexploits a rule-based system which allows individual users to denated question can be sent to the multiple users simultaneously.
fine custom Mind Sensing methods. The key idea is to de-couple
When an action is executed, the system refers to a name aggregathe definitions of the Mind Sensing from the surrounding systems.
tion table, which maps a user ID within the proposed service to a
_id: "5d2abade3efbc011fb13f849"
timestamp: "2019-07-14T14:17:18+0900"
from: "LINE bot"to: "U921012a7a9b4c86ec69782677ed2a3f3"
dataType: "text"
contents: "Hi, Haruhisa. How are you? It is 14:17
on July 14th. In the living room, are you eating?
[Question number:14005](Accuracy:71.428%) (Detected by humidity sensor)"

668

iiWAS2019 Papers
iiWAS2019, December 2–4, 2019, Munich, Germany

Proceedings of iiWAS2019
Haruhisa Maeda1 , Sachio Saiki1 , Masahide Nakamura1,2 , Kiyoshi Yasuda3

user ID of the concrete message service specified in serviceType.
After resolving the user ID, the service invokes Web-API of themessage service, passing the text described in the messageBody,
the destination address of the user ID. For example, suppose that
we define the following action: act1 = {targets: ["Maeda"],messageBody: "How is your current condition？", service:"LINE"}. The act1 defines an action that the LINE chatbot send amessage "How is your current condition?" to a LINE user ID corresponding to “Maeda”.

3.3

Time-Based Rule

A time-based rule (TBrule, for short) is a rule that repeatedly executes actions at time interval within a set period. It defines an inquiry without depending on any event from external services. The
TBrule can be used when asking questions regularly or when sending messages at a fixed time. A TBRule is defined by four items:
actions specifies a list of actions to execute, since specifies thestart time, until specifies the end time, and interval specifies
minutes of the repetition interval. For example, suppose that we
define the following TBrule: tbrule1 = {actions: ["act1"],
since:"10:00", until:"16:00", interval: 60} . The tbrule1
defines a TBrule that action act1 is executed every hour from 10
o’clock to 16 o’clock every day.
When the service starts, all TBrules in the database are read.
Each TBRule has the tasks that are executed every interval fromthe start time on the server. The task checks if the current time iswithin the execution period (between since and until) and executes
the action if it is.

3.4

Event-Based Rule

An event-based rule (EBrule, for short) is a rule that is triggered byan event notified from an external service, based on when, where,
and what event is notified.
An EBrule is defined by three items: actions specifies a list ofactions to execute, conditions specifies one or more conditions
to be satisfied by the event, and breakTime specifies minutes ofcooling time to the next execution.
When an event is notified, an EBrule is triggered only if all theconditions are satisfied. Each condition is defined by the 5W perspective (i.e., WHO, WHOM, WHAT, WHEN, WHERE). This perspective can cover most events issued by external systems. Eachcondition is defined by six items: from, to, event, since, until,
location.
cid: condition ID
from: The subject of eventto: The object of event
since: Whether the event took place after this time
until
: Whether the event took place before this time
event
: The contents of event
location: The location of event
description : The description of event
For example, suppose we define the following condition: con1 ={from: "Activity recognition", to:"Maeda", since:"06:00",
until:"10:00", event:"Waking up", location:"Bedroom",} The con1 defines a condition that activity recognition service
detects user Maeda’s waking up in the bedroom.

Next, let us define the following EBrule: ebrule1 = {actions:["act1"], conditions: ["con1"], breakTime: 30} . The
ebrule1 defines a rule that action act1 is executed only when thecondition con1 is fulfilled. That is, when that activity recognition
service detects that Maeda wakes up, then send him a question“How is your current condition?” by LINE. Once ebrule1 is executed, it will not run for the next 30 minutes.
To receive the event notification from external systems, the proposed service exposes REST API, with a method postEvent(from, to,event, time, location). When the external system executes the API,
the service evaluates conditions of every EBrule against the given
values of the parameters. For example, when postEvent("Activity
recognition service", "maeda", "10:42:24", "Waking up", "bedroom") is
executed for the above con1, it returns false because the perspective of the WHEN is not met.

4 CASE STUDY
4.1 Outline
As a case study, we conduct an experiment to obtain the user’s
mental state by sending questions using the proposed service. The
purpose of the case study is to verify the operation of the systemand to gather user’s opinion for better improvement. In the experiment, a questioner created 42 questions in total by referring to themental illness questionnaire sheets, and created rules to ask thesubjects three questions twice a day in the morning at 6:30 and in
the evening at 21:30. Each subject answered every question witha four-level choice. The answer was stored in a database for later
analysis. After the user answered 42 questions in 7 days, we finallysent a review of the past week and feedback that intervened in themental state of the subject.

4.2

Creating Rules

For the experiment, we implemented a Web application that allows the questioner to easily create and update actions and rules
on a Web browser. Figure 3(a) shows the screen of the prototype,where the user creates actions and rules. It shows a list of actions
MorningAction0-3, each of which is defined by targets, messageBody, and serviceType. The rule at the bottom of the figure showsa TBrule, executing MorningAction0-3 at 6:30 every morning.
In this experiment, we needed to send four messages at eachtime, twice a day, at 6:30 and 21:30. Accordingly we created eightactions: MorningAction0,1,2,3 and EveningAction0,1,2,3. Actions MorningAction0 and EveningAction0 are greeting messages
to start an inquiry. MorningAction1,2,3 and EveningAction1,2,3
define concrete three questions asked in the morning (or evening,respectively). For example, MorningAction1 is described as follows: MorningAction1 = {targets: ["maeda", "yasuda",...,"nakamura"], messagebody: "Q1. Do you think you feelsatisfied with your daily life?", service: "LINE"}.
We then created two TBrules MorningRule and EveningRule.
In both rules, the interval was set to 1440 minutes, so that MorningRule and EveningRule were executed exactly once a day. Thus,
each subject received three questions following a greeting message every morning at 6:30, as well as every evening at 21:30. For
example, MorningRule is described as follows: MorningRule ={actions: ["MorningAction0", "MorningAction1","Morning

669

Proceedings of iiWAS2019

iiWAS2019 Papers

Rule-Based Inquiry Service to Elderly at Home
for Efficient Mind Sensing

iiWAS2019, December 2–4, 2019, Munich, Germany(a) Question setting screen(b) Interaction screen

Figure 3: Screenshots of applications in case study
Action2", "MorningAction3"], since: "06:30", until: null,interval:1440} .

4.3

Results and Feedback

Figure 3(b) shows the screen of LINE application, where a LINE
chatbot were interacting with the subject. The chatbot sent a message asking the mental state by MorningAction0,1,2,3 at 6:30,
as defined in MorningRule. In the bottom of the figure, the subject
responded to these questions.
In this case study, the questioner, who made the questions fromquestionnaire sheets of mental illness, defined the actions and rules
via the Web application by herself. She said that the proposed service was very useful, and much more efficient than the manual survey she had been doing before. On the other hand, she requested
some improvement on the usability.

5

CONCLUSION

In this paper, we have proposed MindSensingService to capture internal mental states of elderly people at home. Introducing a rulebased system, we have de-coupled the definition of Mind Sensing
from surrounding systems, which allows more systematic and flexible Mind Sensing. We also conducted a case study asking questions for understanding mental states of the subjects. The proposedservice allows to define the method of Mind Sensing independentlyof specific systems or messaging service. Thus it is expected to acquire various aspects of the internal states. As for future research,
we plan to conduct a clinical experiment with more elderly people
to evaluate practical feasibility and limitations.

ACKNOWLEDGMENTS

REFERENCES[1] 2018. LINE Messaging API. https://developers.line.biz/ja/services/messaging-api/.

[2] Giuseppe Di Cesare, Cinzia Di Dio, Massimo Marchi, and Giacomo Rizzolatti. 2015. Expressing our internal states and understanding those of others.
Proceedings of the National Academy of Sciences 112, 33 (2015), 10331–10335.
https://doi.org/10.1073/pnas.1512133112[3] Government of Japan. 2018. Annual Report on the Aging Society (2018).
http://www.cao.go.jp/.
[4] Jennifer R Kwapisz, Gary M Weiss, and Samuel A Moore. 2011. Activity recognition using cell phone accelerometers. ACM SigKDD Explorations Newsletter 12,
2 (2011), 74–82.
[5] Haruhisa Maeda, Sachio Saiki, Masahide Nakamura, and Kiyoshi Yasuda. 2019.
Memory Aid Service Using Mind Sensing and Daily Retrospective by Virtual
Agent. In 10th International Conference, DHM 2019, Held as Part of the 21st HCIInternational Conference, HCII 2019, Vol. LNCS 11582. Springer, 353–364.
[6] Masahide Nakamura, Kenji Hatano, Jun Miyazaki, Kiyoshi Yasuda, Noriaki
Kuwahara, Hiroaki Kazui, Sachio Saiki, Seiki Tokunaga, Mihoko Otake, Naoki
Kodama, and Naoko Kosugi. 2019. Developing a System for Self-care andMutual-aids of Elderly People at Home Based on Externalization of InternalStates. JSPS KAKENHI, Grant-in-Aid for Scientific Research (A) 19H01138 (2019).
[7] Shota Nakatani, Sachio Saiki, Masahide Nakamura, and Kiyoshi Yasuda. 2018.
Generating Personalized Virtual Agent in Speech Dialogue System for People
with Dementia. In Digital Human Modeling 2018 (DHM 2018), Held as Part of HCIInternational 2018, Vol. LNCS 10917. Springer, 326–337. Las Vegas, USA.
[8] Long Niu, Sachio Saiki, and Masahide Nakamura. 2018. Using Non-Intrusive
Environmental Sensing for ADLS Recognition in One-Person Household. International Journal of Software Innovation (IJSI) 6, 4 (August 2018), 16–29.
[9] Kazunari Tamamizu, Seiji Sakakibara, Sachio Saiki, Masahide Nakamura, andKiyoshi Yasuda. 2018. Machine Learning Approach to Recognizing Indoor Activities based on Detection of Envi-ronmental Change. In 11th World conferenceof Gerontechnology (ISG2018), Vol. 17. 118s. St. Petersburg, USA.
[10] K. Ueda, M. Tamai, and K. Yasumoto. 2015. A method for recognizing living activities in homes using positioning sensor and power meters. In 2015 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops). 354–359. https://doi.org/10.1109/PERCOMW.2015.7134062[11] Zhihua Wang, Zhaochu Yang, and Tao Dong. 2017. A Review of Wearable Technologies for Elderly Care that Can Accurately Track Indoor Position, Recognize
Physical Activities and Monitor Vital Signs in Real Time. Sensors 17, 2 (2017).
https://doi.org/10.3390/s17020341

This research was partially supported by JSPS KAKENHI Grant
Numbers JP19H01138, JP17H00731, JP18H03242, JP18H03342,
JP19H04154, JP19K02973.

670