Recording Daily Health Status with Chatbot
on Mobile Phone - A Preliminary Study Haruhisa Maeda, Sachio Saiki, Masahide Nakamura

Kiyoshi Yasuda

Graduate School of System Informatics, Kobe University
1-1 Rokkodai, Nada, Kobe, Hyogo 657-8501, Japan
Email: haruhisa@ws.cs.kobe-u.ac.jp, masa-n@cs.kobe-u.ac.jp

Faculty of Informatics, Osaka Institute of Technology
5-16-1, Omiya, Asahi, Osaka, 535-8585 Japan
Email: fwkk5911@mb.infoweb.ne.jp

Abstract—To support in-home long-term care, we are studying
techniques of mind sensing, which externalizes internal states of
elderly people as words through conversations with agents or
robots. We previously developed the memory-aid service, where achatbot on a mobile phone autonomously talks to elderly people,
to record their conditions, events, and memorandums. Duringexperiments with healthy elders, we found that they regularly
talked to the chatbot about health status, such as weight andblood pressure. This motivated us to use the mind sensing as anaffordable and practical means to record daily health status.
In this paper, we present a method where individual users
can declare health metrics of their interests, and record themthrough the mind sensing. Specifically, for each user-defined
metric, the chatbot asks the user the current value of the metricat the designated time. The text conversations are then put
in a data mining process to extract time-series values of themetric. The time-series data is finally visualized as a graph, withwhich the user can review the health status. Our preliminary
experiment shows that individual health metrics can be recorded
and visualized successfully even without “connected” measuring
instruments.
Index Terms—smart healthcare, mind sensing, data mining,
chatbot, conversational agent, IoT

I. I NTRODUCTION
According to the Japanese Government, the number ofJapanese people over the age of 65 is 35.15 million in 2017,
which is 27.7% of the total population [1]. The number
continues to increase and will reach 40% of the total in 2050.
Similar situations are observed in most developed countries
in the world. Such a super-aging society causes a chronic
shortage of human resources, nursing facilities, and the cost ofsocial security. The Japanese government is therefore shifting
the policy from the conventional facility care to in-homelong-term care. It is ideal for all elderly people to live intheir beloved homes. However, they generally face various
difficulties in daily life due to the decline of physical andcognitive functions. Also, home care would pose a heavy
burden on family caregivers. In order to support these elderly
people at home, the assistive technologies have been studied
and developed in various organizations in recent years.
A promising technology to support the in-home long-termcare is the elderly monitoring system using ICT. Exploiting sensors, wearable devices, and smart home technologies,
the system tries to recognize daily activities of elderly athome. Various methodologies have been studied, for example,

the method with ambient environmental sensing [2] [3], themethod with accelerometers of mobile phones [4], the methods
with wearable sensors [5], and the method using a smart homewith indoor positioning and power meters [6].
On the other hand, monitoring with sensors has a limitationthat it detects only externally observable events. To performperson-centered home care, it is also important to recognize
internal states of elderly people. Here, the internal state refers
to a status of a person that cannot be observed externally,
including moods, pains, conditions, desires, and intentions.
Since the internal state is directly linked to human health,
it is important to monitor within the home care [7]. However,
the internal state is usually assessed through inquiries andcounseling by experts. Hence, it is challenging to monitor theinternal state regularly at home.
In order to cope with the limitation, our research groupis studying techniques of mind sensing (“kokoro” sensing inJapanese [8]). It externalizes the internal states of elderly
people as “words” through conversations with agents or robots.
In our previous study [9], we developed the memory-aid
service, exploiting a mind sensing method with a chatbot ona mobile phone. Triggered by time or an event within a smart
home, the chatbot asks a user (i.e., elderly person) a questionthat externalizes the internal state as words. As the user
answers the question by text (as well as voice and images),the conversation is automatically recorded in a database with atimestamp. The service then provides the retrospective process,where the user can review, correct, classify and search therecorded information of own at any time. Thus, the service
is designed for the memory-aid purpose of healthy elders aswell as people with cognitive impairment.
To evaluate the practical feasibility, we are experimentally
operating this memory-aid service within actual householdsof healthy elderly people. Then, we found that they often
talked about their health status, such as pains, weight, bloodpressure measurements and the number of steps walked. Of
course, the metrics of interests and the way of recording variedamong individuals. However, recording such health status ona daily basis is quite important and should be promoted more
explicitly. Although there exist “connected” devices measuring
such health data online, they are not yet adopted widely byelderly people, and would be intrusive for their daily life. This
motivated us to use the mind sensing as an affordable andpractical means to record daily health status.
In this paper, we investigate a method where individual users
can declare health metrics of their interests, and record themetrics through the chatbot-based mind sensing. Specifically,for each user-defined metric, the chatbot asks the user thecurrent value of the metric at the designated time. As theusers respond with the value, the conversation is recorded
in a database. Applying data mining process in the recorded
messages, we extract time-series values of the metrics. The
time-series values are finally visualized as a graph, with which
the user can review the health status.
We have conducted a preliminary experiment that visualizes
health status from the mind sensing records of two elderly
people. The visualized metrics include blood pressure measurements, consumed calories, the number of steps of walking,the weight, the health condition, the time of wake up and go to
bed and the record of defecation. In the time-series graphs, weobserved that changes in health status are well observed. Thus,
the individual health metrics can be recorded and visualized
successfully by the proposed method with the chatbot-based
mind sensing.
This study has been approved by the research ethics committee of Graduate School of System Informatics, Kobe University. Written informed consent was obtained from subjects
for publication of this paper and accompanying images.
II. P REVIOUS S TUDY: M EMORY-A ID S ERVICE [9]A. Overview and System Architecture
In [9], we have developed the memory-aid service to support
elderly people at home who are anxious about age-related
forgetfulness, as well as cognitive impairment. The key idea
is to provide an external memory for a user (i.e., an elderly
person), so that the user can easily record whatever important
for the user, and can retrieve the recorded information at any
time. Major challenges were (1) how to encourage the user torecord the essential information (including the internal states)in the external memory, and (2) how to use and maintain therecorded data for better quality life of the user.
Figure 1 shows the system architecture of the memory-aid
service, consisting of the following two functionalities:
Chatbot-based Mind Sensing is a mechanism that externalizes and records the information in user’s mind, throughconversations with a chatbot on a mobile phone. Triggeredby time or an event issued by an external system, the chatbot
autonomously asks a question to the user. In [9], we integrated
the activity recognition system [2] with SensorBox [10], inorder to generate context-aware questions. Typical questions
include “What are you doing now?”, “How are you feelingtoday?”, “Did you sleep well?”, “What did you do today?”and “Where do you want to go?”As the user responds to the chatbot by manual text input
or the speech-to-text input of the mobile phone, the text
message is sent to Web-API, and recorded in a database with atimestamp. Moreover, regardless of the questions, the user canspontaneously talk to the chatbot to record any information,such as memorandums, schedules, appointments, and thoughts.

SensorBox

Activity recognition system
Activity
Recognition

SensorDB

Monitorenvironment

Trigger sensing

Exposeinternal states
SMS app on
smartphone

Ask states
API

Mind
Sensing

Record states

Mind
DB

Send dialogue
Review pastmessages

API
Virtual Agent
on PC

Query
messages

Memory-Aid
Retrospective
Core services on cloudFig. 1. System architecture of memory-aid service [9]Memory-Aid Retrospective is an application that provides
an opportunity where the user can look back the recorded
information. Exploiting voice interaction with a virtual agent[11] and intuitive graphical user interface, the user can perform
two types of retrospective processes on a PC.
In the short-term retrospective process, the system displays
the list of fresh messages recorded within one day before
now. While viewing the list at the end of the day, the user
can remember what happened today. To maintain the qualityof the data, the application tells the user to correct typosand errors in the existing records, or to add supplementary
messages if necessary. Moreover, for every record, the user
can associate user-defined categories that explain the type of
the information.
In the long-term retrospective process, the user can viewand search all the existing records. By simple user interface
assisted with the virtual agent, the user can retrieve messages
by category, keywords, and date. Thus, the user can find important information whenever necessary, which would relieve
the anxiety for the forgetfulness.
B. Deploying Prototype in Actual Households
We have implemented a prototype of the memory-aid service as a Web application operated on both a smartphone and
PC. In particular, we implemented the chatbot with the LINE
Messaging API [12]. To manage the data obtained by the mind
sensing, we deployed MongoDB. For inserting and retrieving
data via the Internet, we implemented REST Web-API using
the Jersey RESTful Web Services framework.
Figure 2(a) shows a screenshot of the Chatbot-based Mind
Sensing, where the LINE chatbot asked the user what happened in the Japanese room, and the user answers his activities.
Figure 2(b) shows the screen of the short-term retrospective
process, where the user corrects and classifies the latest
messages. Figure 2(c) shows the screen of the long-term
retrospective process, where the user searches messages withthe “health” category.
To evaluate the practical feasibility, the prototype system
has been deployed and operated within actual households.
Especially, Subject A (a 77-year-old man) and Subject B (70year-old man) are actively cooperating on our experiment. The
current rules of the mind sensing are as follows. When the(a) Chatbot-based Mind Sensing(b) Short-term Retrospective(c) Long-term Retrospective

Fig. 2. Screenshots of the memory-aid service

activity recognition system detects the wake up of the subject,
the chatbot on the mobile phone asks the physical condition of
the subject. When the system detects any activity, the chatbot
asks a question depending on the context. At the end of
the day, the chatbot encourages to perform the retrospective
process.
To respond to the chatbot, Subject A manually inputs the
text on the LINE application, whereas Subject B is using
the speech-to-text feature of the mobile phone. The subjects
sometimes talk to the chatbot just for the recording purpose,even when there is no question.
III. R ECORDING H EALTH S TATUS WITH M IND S ENSING
A. Motivating Example

B. Declaring Metrics to Record
There are various kinds of data characterizing one’s healthstatus, and the data of interest varies among individuals.
One may be interested in weight, while another may beinterested in blood pressure. To make the framework flexible
and sustainable, the proposed method first requires every user
to declare metrics that he/she wants to record within thesystem. When a user declares a metric, the proposed methodasks the user to define the following attributes:
Name: defines the identifier of the metric (e.g., weight,
condition, blood pressure).
• Type: defines the type of the value of the metric bynumeric, enumerate, structured, or time.
• Unit: defines the unit of the metric (e.g., kg, None,mmHg).
• Reporting Time: defines the ideal time for the user toreport the measurement.
•

In the experiment in Section II-B, we found that the subjects
talked to the chatbot about various topics. In particular, these
healthy elders spontaneously recorded health-related data. For
example, let us translate the third message in Figure 2(b).
 Let us see the previous example. If the user wishes to record
2019-06-25 (Tue) 8:39
the body temperature, condition, and blood pressure, the user
During 7:45 and 8:34, I measured blood pressure. The
submits the following three declarations to the system:values were 145 60 48, ... (top, bottom, heart rate, respectively). At 8:00, body temperature was 36.0 Celsius.
d1 = {name:"body temperature",
I took the medicine... The condition is OK. Today, I will
type:numeric, unit:"Celcius",
go to the training gym after one month absence. ...
rep_time:08:00:00}
 d2 = {name:"condition", type:enumerate
Applying a simple text mining to this message can retrieve{"OK":1, "NG":0}, unit:None,
important health metrics helpful to in-home self-care. Of
rep_time:08:00:00}
course, these subjects were considerably motivated for theird3 = {name:"blood pressure", type:
health. However, we believe that recording the health data
structured {"top":{type:numeric,within the mind sensing is possible for general elderly people,
unit:"mmHg"}, "bottom":{type:with a solid framework and appropriate intervention. In thisnumeric, unit:"mmHg"},
paper, we investigate a method that exploits the chatbot-based
rep_time:08:00:00}
mind sensing for the purpose of recording health status.
Data-Mining Algorithm:
Declaration d1 defines the body temperature, where it takes
a numeric value, the unit is Celsius, and the user wishes torecord around 8 o’clock. Next, d2 defines the condition by
enumerate type of “OK” or “NG” (No Good) whose values are
encoded by 1 or 0, respectively. Finally, d3 defines the bloodpressure by structured data, consisting of “top” and “bottom”
both of which take numeric values with the unit of “mmHg”.
C. Recording Health Status with Chatbot

Input: declaration of metric M , user u
Output: time-series values M (u, t) of M of u at time t
Procedure:
1) From all messages sent by u, find messages containing M.name as words. Let SM be the set ofthe found messages.
2) For each message m ∈ SM , search each word
after M.name, and find a value v that conforms
to M.type. Let t be the timestamp of m. Output v
as M (u, t).
Suppose that Mr. Maeda sent the message in Section III-C
on 2019-06-27 (Thr) 08:15:33. Then, the algorithm derives

Once health metrics are declared by a user, the system
encourages the user to record the values. For each declared
metric, when the reporting time comes, the system commands
the chatbot to ask the user the value of the metric. For example,
for the declarations of d1, d2, d3, the chatbot “Mei-chan”
talks to the user at eight o’clock every day, like:body_temperature(maeda, t)=35.7
blood_pressure(maeda, t)={135, 88}
Mei-chan: Good morning, Mr. Maeda!
condition(maeda, t)= OK
How are you? Please report yourwhere t = 2019-06-27T08:15:33.
- body temperature
The step 1) of the procedure can be performed by a search- contition (OK or NG)
queryof the database. The step 2) can be implemented by text
- blood pressure (top and bottom)
patternmatching with regular expressions.
to me. Have a wonderful day!


E. Visualizing Time-Series Data
On receiving the message from the chatbot, the user meaFinally, the time-series values of each metric are plottedsures the values using measuring instruments of own, andina graph, with which the user can review the history ofsends the answer to the chatbot. For this, the user can talkthe
health status. According to the basis of the once-daily
by his/her own words, but the name of each metric should
recording,we downsample the time-series values to daily
precede its value, without interleaving other metrics.

frequency, discarding the time of the day. If the resolutionof daily frequency is too fine, we can downsample the data to
Maeda: Good morning, Mei-chan.
weekly frequency, with appropriate aggregation.
My body tempreature is 35.7. Blood
The downsampled time-series data of M (u, t) is then plotpressure measurements were 135 and 88.
ted
on a graph, where the horizontal axis represents t and theCondition is OK. Thank you!
verticalaxis plots the value of M (u, t).

The answer message to the chatbot is then recorded in the
database with a timestamp. Of course, the user could maketypos and errors in the message, especially when using the
speech-to-text input on the mobile phone. Once the message
is sent, these errors cannot be corrected on the mobile phone.
Instead, the user can use the Memory-Aid Retrospective (seeSection II-A) to correct the errors and proof-read the message
again. This is also good for the user to reflect the health status.
For sustainable recording, we consider that the systemshould not force the user to be punctual. Even if the values
are not measured at the reporting time, the message from thechatbot would stimulate the user to record them later. Ourcurrent aim is once-daily recording of health status.
D. Extracting Metrics Values from Messages
Applying data mining to the recorded messages in the
database, we extract values of metrics. Here, we suppose thateach message follows the convention that “the name of each
metric should precede its value without interleaving othermetrics”. Then, the following algorithm derives time-series
values M (u, t) of a metric M of a user u at time t.

IV. P RELIMINARY E XPERIMENT
A. Assumption and Health Metrics
To see the practical feasibility, we conduct an experimentthat visualizes daily health status from messages recorded by
the mind sensing. In fact, the integration of the proposed
method into the memory-aid service is yet under implementation. Therefore, in this preliminary study, we analyze theexisting data of Subjects A and B (see Section II-B), assumingthat they followed the instruction of the proposed method.
Investigating the existing data, we found that each subject
regularly recorded the following metrics:
Subject A (based on data since 2019-03-28 till 2019-06-27):• blood pressure (top, bottom, and heart rate)• the number of steps in daily walking• condition (OK or NG)• calory consumption in walking and the training gym
Subject B (based on data since 2018-12-18 till 2019-06-27):• blood pressure (top, bottom, and heart rate)• weight• hours of sleep and get up

150(a) Subject A: Blood pressure

100000top (mmHg)
bottom (mmHg)
heart rate (bpm)

100
75

60000
40000
20000

50
04
2019

05
2019

06
2019

07
2019

(c) Subject A: Condition

0

04
2019

05
2019

06
2019

calories(kcal)

8000
6000

0.6
0.4

4000

0.2

2000

Condition(OK=1, NG=0)
0.0
04
05
2019
2019

06
2019

07
2019

0

(e) Subject B: Blood pressure

04
2019

05
2019

06
2019

100

25

weight(kg)

70
top (mmHg)
bottom (mmHg)
heart rate (bpm)
01 02 03 04 05 06 07
2019 2019 2019 2019 2019 2019 2019(g) Subject B: Hours of sleep / getup

20
15
10

07
2019

(f) Subject B: Weight
72

50

07
2019

(d) Subject A: Calorie consumption (PW)

0.8

150

steps

80000

125

1.0

(b) Subject A: # of steps (PW)

68
01 02 03 04 05 06 07
2019 2019 2019 2019 2019 2019 20196(h) Subject B: Defecation scale
scale

4

sleep
getup

2

5
0
02
03
2019 2019

04
2019

05
2019

06
2019

07
2019

0

01 02 03 04 05 06 07
2019 2019 2019 2019 2019 2019 2019

Fig. 3. Visualization of health metrics of Subjects A and B recorded in the memory-aid service

defecation scale (Bristol stool form scale [13])For each metric, we specified the declararion. Then, we applied the proposed data-mining algorithm and the visualization
to the recorded data of the memory-aid service.
•

B. Results
Figure 3 shows graphs that visualize the individual metricsof the subjects. As for graphs (b) and (d), the time-series values
have been aggregated per-week (PW) by total, since the dailyfrequency was too fine to visualize. From the graphs, we canclearly observe changes in their health status.
First, let us see the health status of Subject A. In graph(a), we can observe sharp fluctuations in blood pressure andheart rate around the beginning of June 2019. Accordingly, thecondition dropped to no-good shown in graph (c). Recently,
we interviewed Subject A what happened, and he told usthat he suffered from mild bacterial pneumonia during that
period. Fortunately, he has completely recovered from thedisease now. After the recovery, we slowly restarted the daily
exercise. However, he is intentionally reducing the amount of
the exercise, as we can see the number of steps and the calorie
consumption in graphs (b) and (d).
Next, let us see the result of Subject B. From graph (e), we
can see the blood pressure is stable within acceptable range.
The sudden drop of the top value at the end of May 2019was due to the error of recording, as he mistakenly inputs thevalue 022 instead of 122. From graph (f), Subject B is losing
weight significantly. As we asked, he has been on a diet to
reduce the weight to alleviate knee pain. It is recommended
by his physical therapist. From graph (g), we can see that
Subject B regularly gets up around five o’clock, however, hesometimes goes to bed in unusual hours. From graph (h), thevalue of the Bristol scale is usually 4 or 3, which is quite
normal. However, the scale is not recorded every day as we
can see a lot of zero values.
V. D ISCUSSION AND C ONCLUDING R EMARKSIn this paper, we have presented a method that allows auser to record and review the daily health status throughconversation with a chatbot on a mobile phone. For each
metric declared by the user, the chatbot autonomously asks
the current value, and the user answers to record. The timestamped conversations stored in a database are then put in data
mining process to visualize time-series values of the metrics.
We have also conducted a preliminary experiment visualizing health status of two healthy elders from the existing data of
our previous study. As shown in the experiment, the proposed
method successfully extracted and visualized the health statusfrom the records of mind sensing.
In fact, many elderly people are measuring own healthdata at home. However, most data is volatile without being
recorded, and is not utilized effectively. Recently, “connected”
health-care devices appear on the market. However, even
without such devices, the chatbot-based mind sensing can bea practical method to record and utilize the health data.

Of course, there are many open issues. Firstly, it is important
to consider how to design attractive dialogues of the chatbot,
so as to keep the motivation of general elderly people. Also, in
addition to the visualization, the system should provide appropriate intervention depending on the health status. Previously,
a medical examination by interview was the only way for adoctor to know daily circumstances of elderly people. Sharing
the recorded data between the user and his/her family doctor
would be a great help for efficient medical treatments. Theseissues will be addressed in our future work.
ACKNOWLEDGMENT
This research was partially supported by JSPS KAKENHI
Grant Numbers JP19H01138, JP17H00731, JP18H03242,
JP18H03342, JP19K02973.
R EFERENCES[1] Government of Japan, “Annual report on the aging society (2018),”
http://www.cao.go.jp/, June 2018.
[2] K. Tamamizu, S. Sakakibara, S. Saiki, M. Nakamura, and K. Yasuda,“Machine learning approach to recognizing indoor activities basedon detection of envi-ronmental change,” in 11th World conference ofGerontechnology (ISG2018), vol. 17, May 2018, p. 118s, st. Petersburg,
USA.
[3] L. Niu, S. Saiki, and M. Nakamura, “Using non-intrusive environmental
sensing for adls recognition in one-person household,” International
Journal of Software Innovation (IJSI), vol. 6, no. 4, pp. 16–29, August
2018.
[4] J. R. Kwapisz, G. M. Weiss, and S. A. Moore, “Activity recognition using cell phone accelerometers,” ACM SigKDD Explorations Newsletter,vol. 12, no. 2, pp. 74–82, 2011.
[5] Z. Wang, Z. Yang, and T. Dong, “A review of wearable technologies for
elderly care that can accurately track indoor position, recognize physicalactivities and monitor vital signs in real time,” Sensors, vol. 17, no. 2,
2017. [Online]. Available: https://www.mdpi.com/1424-8220/17/2/341[6] K. Ueda, M. Tamai, and K. Yasumoto, “A method for recognizing
living activities in homes using positioning sensor and power meters,”
in 2015 IEEE International Conference on Pervasive Computing andCommunication Workshops (PerCom Workshops), March 2015, pp. 354–
359.
[7] G. Di Cesare, C. Di Dio, M. Marchi, and G. Rizzolatti, “Expressing
our internal states and understanding those of others,” Proceedings of
the National Academy of Sciences, vol. 112, no. 33, pp. 10 331–10 335,
2015.
[8] M. Nakamura, K. Hatano, J. Miyazaki, K. Yasuda, N. Kuwahara,
H. Kazui, S. Saiki, S. Tokunaga, M. Otake, N. Kodama, and N. Kosugi,“Developing a system for self-care and mutual-aids of elderly people
at home based on externalization of internal states,” JSPS KAKENHI,
Grant-in-Aid for Scientific Research (A), no. 19H01138, 2019-2023.
[9] H. Maeda, S. Saiki, M. Nakamura, and K. Yasuda, “Proposal of memory
aid service using mind sensing by virtual agent,” in IEICE Technical
Report, vol. 118, no. 511, SC2018-40, March 2019, pp. 19–24.
[10] S. Sakakibara, S. Saiki, M. Nakamura, and S. Matsumoto, “Indoor
environment sensing service in smart city using autonomous sensor
box,” in 15th IEEE/ACIS International Conference on Computer andInformation Science (ICIS 2016), June 2016, pp. 885–890, okayama,
Japan.
[11] S. Nakatani, S. Saiki, M. Nakamura, and K. Yasuda, “Generating
personalized virtual agent in speech dialogue system for people withdementia,” in Digital Human Modeling 2018 (DHM 2018), Held as Partof HCI International 2018, vol. LNCS 10917. Springer, July 2018, pp.
326–337, las Vegas, USA.
[12] “LINE Messaging API,” https://developers.line.biz/ja/services/messagingapi/, July 2018.
[13] S. J. Lewis and K. W. Heaton, “Stool form scale as a useful guideto intestinal transit time,” Scandinavian Journal of Gastroenterology,
vol. 32, no. 9, pp. 920–924, 1997, pMID: 9299672. [Online]. Available:
https://doi.org/10.3109/00365529709011203