2022 The 1st International Conference on Software Engineering and Information Technology (ICoSEIT)

Study of Multi-Modal Diary Service Using Spoken
Dialogue Agent for Self-Care in Elderly People
1st Sinan Chen
1-1 Rokkodai-cho, Nada
Kobe University
Kobe, 657-8501, Japan
chensinan@ws.cs.kobe-u.ac.jp

2nd Masahide Nakamura
RIKEN Center for Advanced Intelligence Project
Kobe University
Tokyo, 103-0027, Japanmasa-n@cs.kobe-u.ac.jp

Abstract—With Japan’s aging population and low birth rate,
the self-care of older adults at home has become increasingly indemand. Although the rapid progress of AI-enabled healthcare
is developing, diversity and reusability of in-home daily living
data to be recorded and analyzed are still challenging points.
In our previous study, we have developed a system utilizing
spoken dialogue agents for listening to the minds of older adults.
The purpose of this study is to propose a multi-modal diaryservice that records and analyzes dialogue data with a diversiﬁedviewpoint. Our key idea is to retrieve the text logs, audio,
and images and execute multi-modal visualization. The proposed
method describes the main steps, including heterogeneous data
extraction, Web API development, multi-modal diary generation,and heterogeneous data analysis. Also, we have discussed theapproach to extending heterogeneous data. Using the proposed
method, older adults can recollect the past for their own better.
In this way, the correct improvement of in-home care and healthmanagement for all individuals is promising.
Index Terms—aging society, elderly self-care, speech dialogue
agent, multi-modal visualization, smart healthcare

I. Introduction
As the aging population around the world, the demand
for nursing care is increasing every year [1]. Especially forthe elderly at home, improving self-care using engineeringbased assistive technology is widely studied and developed. In
the smart healthcare ﬁelds, monitoring the Activities of Daily
Living (ADL) to understand the events and status of the elderly
is a common approach [2] [3]. Moreover, to record daily living
and assist the memory of elderly people at home, performing
an interaction using a robot or smartphone is proposed [4].
Recording life videos to analyze has also been developed [5].
The challenging points of this study include the lack ofdata diversity and reusability. Such collecting data using onlya single form and recording complete data often contains
many redundant data. Our interest is in providing a mind
sensing [6] system for elderly people at home. In our previous
study, a Listening Service [7] integrating Virtual Agent(VA) technology and IoT sensors was developed. This service
aims to collect minds through “spoken dialogue” between
elderly people and the VA. Using WebSocket and Pub/Sub
architectures chen2020improving, a general computer thatlinks MMDAgent [8] to Web Speech Application Programming
Interface (API) and IoT sensors has been deployed in actual
households.
The goal of this paper is to propose a multi-modal diaryservice for elderly self-care using the spoken dialogue agent.

978-1-6654-7303-3/22/$31.00 ©2022 IEEE

The daily living data includes a wide range of modalities, suchas not only texts [9], but also audio and images [10]. Our key
idea is to record and analyze thick and rich dialogue data basedon the previous study. In our approach, we divided the fourmain steps into the proposed method: (1) Heterogeneous data
extraction. (2) Web API development. (3) Multi-modal diary
generation. (4) Heterogeneous data analysis. In this way, asystem with spoken dialogue agents will be more meaningful
to perform spoken dialogue and think back better for theelderly with diversiﬁed viewpoints.
II. Preliminaries
A. Aging Society and Self-Care in Elderly People
In recent years, advanced countries around the world arefacing an increasing demand for nursing care in an aging
society [1]. Many older adults with chronic diseases, dementia,
and living alone expand every year in Japan. The shortageof medical welfare personnel and facilities is becoming more
severe. In response to this, the Japanese government is actively
strengthening a Community-based Integrated Care System 1 .
It comprises self-help, mutual aid, cooperation, and public
assistance. Nonetheless, due to the decreasing birthrate and
COVID-19 inﬂuence, it is diﬃcult for mutual aid, cooperation,and public assistance to advance. Hence, self-help will play
an essential role in the future.
Related work for self-help in the elderly generally includes
maintenance of Activities of Daily Living (ADL) and mental
self-care, encouraging them to focus entirely on their eﬀorts
and needs. Especially in mental self-care for the elderly living alone, stress-dealing, health management, and enhancing
memory are often not easy. In order to improve the above
cases, assistive technology using engineering methods in thehealthcare ﬁeld has been rapidly progressing worldwide. Oneof the most representative systems is to monitor older adults
at home. It is common to understand the events and status of
the elderly by the machine [2] [3].
B. Technical Challenges
To achieve the self-care of older adults conveniently athome, we consider two signiﬁcant challenges.
The ﬁrst challenge lies in data diversity. It is challenging toassist older adults eﬀectively by integrating data in multiple
1 https://www.mhlw.go.jp/english/policy/care-welfare/care-welfareelderly/dl/establish e.pdf

120Microservice

TABLE I
Example of the question contents asked by the agent every three hours.

Cloud Server
Scenarios

PubSub | Web Sockect

Personal Dialogue Logs

Home

Motion Pressure

Elderly Person

IoT Sensors

VA

Browser

General Computer

Period of Time

Question Contents

6:00 ∼ 8:59

Can you tell me how you are feeling today?

9:00 ∼ 11:59

Is there anything you want to talk about now?

12:00 ∼ 14:59

Can you tell me what you did this morning?

15:00 ∼ 17:59

Can you talk to me about whatever your mind?

18:00 ∼ 20:59

Can you tell me what you did this afternoon?

21:00 ∼ 23:59

Can you tell me what you felt and did today?

Fig. 1. A conceptual diagram of the Listening Service for older adults.

forms (e.g., image, audio, and text data), to reﬂect on theirmemories of speciﬁed things. For elderly people at home, it is
not uncommon to experience forgetfulness that occurs in daily
life at home. Such as “where is the wallet away?”, “what is thename of a celebrity on TV?”. For recording and analyzing thecontent that the elderly were not able to notice, the existingstudies include a life watching service using an interactive
home robot, prospective memory functioning improvement
using a smartphone, and remote acoustic voice recording andanalysis in clinical practice. However, most studies collect data
in a single form (i.e., only text or audio data).
The second challenge is data reusability. Indeed, there arealso some related studies recording life videos to aid thememory of older adults at home. Retrieving the daily living
context of older adults is also an important research topic,
using video analysis technology. However, accumulating fullvideos every day that contain a lot of repetitive video clips(e.g., moving around the room) requires ample storage space.
They are often not eﬀectively re-used for retrieving valuable
information. As for video analysis, it is unrealistic to introduce
them into general households widely, due to requires a massive
amount of costs and computing resources. Hence, to improvedata reusability, data themselves should be lightweight at ﬁrst.
C. Previous Study: Spoken Dialogue Agent for Elderly People
Our research group has been studying and developing themind sensing [6] for the elderly at home. Its core part isutilizing the Virtual Agent (VA) technology and IoT devices.
Unlike the conventional approach using pure IoT sensors, we
focus on Human-Computer Interaction (HCI) in the form of“dialogue”. One of the newest achievements is to deploy andoperate a Listening Service [7]. As a virtual lister, it canperform spoken dialogues with elderly people and reply togreetings (i.e., Aizuchi in Japanese) at random. Meanwhile,
many existing Web microservices are also used conveniently invoice interaction with the agent, such as watching the YouTube
videos, searching keyword information, and managing thepersonalized Google calendar. Figure 1 shows a conceptual
diagram of the Listening Service for older adults.
More speciﬁcally, on the one hand, it consists of two
hardware components: a general computer and IoT sensors(Phidgets motion and pressure sensors). On the other hand, itconsists of two software components: a general computer tolink MMDAgent [8] (i.e., agent material and speech synthesis)
and a Web browser (i.e., speech recognition using Web Speech
API, dialogue content UI). Note that the unique User-ID islinked with the virtual agent of each elderly person. It is
extended to a Web service that records and manages the basic
information (i.e., nickname and birthday date) of individual
elderly people. Integrating the User-ID into WebSocket and
Pub/Sub architectures [11], the speciﬁc general computer andIoT sensors are connected. In this way, the agent asks aquestion once automatically and regularly. It performs whenthe elderly person at home is “present” as detected by IoT
sensors. An example of the question contents asked by theagent every three hours during each day shows in Table I.
III. Proposed Method: Multi-Modal Diary Service
A. Goal and Key Idea
To improve the two challenges described in Section II-B,
we aim to propose a multi-modal diary service based on
spoken dialogue agents. The long-term goal of this study is
to make older adults think back on their memories better. For
example, some things that cannot be noticed and remembered
can be improved by checking the multi-modal diary on the
Web page. Our key idea is to record and analyze dialogue data
in multi-modal form, containing text, images, and audio. It will
make a system with spoken dialogue agents more meaningful,which improves communication skills and enhances memoryaid using multi-modal visualization for older adults at home.
B. Overall Architecture
Figure 2 shows the overall architecture of the proposed
multi-modal diary service for older adults at home. Based on
the previous study, the timing console plays a crucial role inasking questions or replying to greetings timely by spoken
dialogue agents. The proposed method divides into four main
steps.
More speciﬁcally, Step 1 describes collecting the dialogue
content of an older adult in the form of text, audio, andimage. In this step, the composed dialogue content with thedate time from the client is sent to the cloud database. As
a transfer station between the cloud database and the client
device, Step 2 presents developing a Web API for building a
way of retrieving heterogeneous data. In Step 3, generating

121

Step 2

Step 3

Multi-Modal
Diary Service

Step 4Multi-Modal Diary
Web API

Home

Database

Timing Console
User Interface
Speech-to-Text
Speaker
AgentDeveloper

Questions andGreetings

FamilyDoctor

Dialogue Text
Logs

Sentiment Analysis

Dialogue Audio
Logs

Prosodic Extraction

Dialogue Image
Logs

Emotion Estimation

Text
Mic

Recording Audio
Audio

Sensors
Elderly Person

CameraGeneral Computer

Capturing Image
Image
Step 1

Fig. 2. The overall architecture of the proposed multi-modal diary service for older adults at home.

a multi-modal diary in the Web browser is proposed, making
the multi-modal visualization of heterogeneous data. Step 4
explains an extension approach for the collected data, including text sentiment analysis, voice emotion detection, and facial
emotion estimation. Additional value-added services are also
discussed.
C. Step 1: Heterogeneous Data Extraction
Step 1-1: Speech-to-Text
In Step 1-1, to construct a smooth interaction between older
adults and spoken dialogue agents, recording the said dialogue
context in the form of text is required. Speech-to-text is atechnology for recognizing speech audio and extracting thetext in a speciﬁc language. It is usually used in the form of
Web API. The typical techniques include Web Speech API
with the Web browser, Speech to Text APIs with many cloudservices (e.g., Google, Microsoft, and Watson).
Like the previous study described in Section II-C, a systemusing Web Speech API (Speech Recognition) and MMDAgent(Speech Synthesis) is proposed. The converted text logs weresent to the MongoDB database on the cloud. Text logs ofquestions or greetings from spoken dialogue agents were also
sent to MongoDB. A data sample with dialogue context tothe cloud database is shown in Listing 1. Note that the timeinterval is required between speech recognition and synthesis.
It should apply to the spoken speed of individuals.
Listing 1. Data sample with dialogue context sending to the cloud database.

{"_id ": "data id","from ": "user -id","to": "keicho -bot"," contents ": "I had a backache at night"," dataType ": "text"," timestamp ": "2021 -09 -08 T18 :02:25+09:00" ,"tag ": "va. keicho .user -id","time ": "2021 -09 -08 T09 :02:19.475 Z"}

End S1-2

Start S1-2
Start S1-1

NextCloud

End S1-1S1-1: Speech to Text

S1-2: Recording Audio

MongoDB

Fig. 3. The schematic diagram represents recording audio and text.

Step 1-2: Recording Audio
In Step 2-1, to record the raw speech data from older adults,
the most naive approach is to build an audio recording function
embedded in the Web browser by JavaScript programming.
One of the typical techniques is the Web Audio API. It
generally creates a media stream source to input audio buﬀer
array using the function navigator.getUserMedia. In theprevious study, the recorded audio-buﬀer array was sent to
the NextCloud database that was saved as an audio ﬁle in
the format wav. Note that the process of recording the audio
contains speech-to-text. Its schematic diagram is shown in
Figure 3. There is a challenging point: smooth dialogue andlistening among diﬀerent functions.
Step 1-3: Capturing Image
In Step 1-3, to collect images while spoken dialogue fromolder adults, the most naive approach is to construct an image
capturing function embedded in the Web browser. In the periodof Step 1-2, to capture image data on time series, the relatedparameters are shown in Table II. Firstly, a video stream
is created into the Web browser. Next, the HTML5 canvas
is drawn from the video snapshot every second using thefunction drawImage(). Then, the raw data in the format
base64 encode is extracted from the canvas by the function

122

TABLE II
Related parameters of capturing image data on time series.

Shooting device

Camera (Lenovo ldeaPad C340)

Shooting method

Drawing canvas from video stream

Shooting interval

One second

HTML5 canvas size (pixel)

320 × 240

Computer browser

Chrome (Version 84.0.4147.125)

canvas.toDataURL(). Finally, the base64 encoding is sent
to the NextCloud database that is saved as an image ﬁle in theformat jpg. Note that all ﬁle names use the date and time.
D. Step 2: Web API Development
Step 2-1: Retrieving Spoken Dialogue Text
In Step 2-1, to retrieve the past dialogue text logs from thecloud database, a Web API named Keicho-Log is developed.
It can generally be developed using various programming
languages, such as Java and Node.js. An example of Web
API speciﬁcation for retrieving dialogue text data is shown in
Listing 2. Based on the previous study, a You-ID Service
is proposed that indicates diﬀerent users by the ID (i.e., uid).
By integrating the uid and Web API, all the past dialogue
text logs can not only be extracted but also be searched by
the keyword, date, and limited count. Moreover, by approving
the Cross-Origin Resource Sharing (CORS) on the cloud, thedialogue text logs can be extracted from the client using
function jQuery.ajax() or Fetch API.
Listing 2. Example of Web API speciﬁcation for retrieving dialogue text data.

GET
GET
GET
GET
GET
GET
GET

˜/ uid ={ uid }/˜/ uid ={ uid }/ count ={ count}
˜/ uid ={ uid }/ keyword ={ keyword }˜/ uid ={ uid }/ date ={ date}
˜/ uid ={ uid }/ keyword ={ keyword }/ count ={ count }˜/ uid ={ uid }/ date ={ date }/ count ={ count}
˜/ uid ={ uid }/ date ={ date }/ keyword ={ keyword }Step 2-2: Extracting Spoken Dialogue Audio
In Step 2-2, to retrieve audio data from the cloud database,the critical step is to search for the speciﬁed date and timefrom the saved ﬁle names. Note that the ﬁle names are diﬀerent
between audio and images due to the timing diﬀerence (seeFigure 3). Using the NextCloud database, the Web URL shows
the ﬁle ID or names (e.g., nextcloud/apps/ﬁles/?dir=ﬁleid)with the endpoint.
Step 2-3: Loading Spoken Dialogue Image
In Step 2-3, to retrieve image data from the cloud database,the critical step is to search for the speciﬁed date and timefrom the saved ﬁle names. The common point between Step
2-2 and Step 2-3 is that the raw data for audio is in the format
buﬀer array but for images is in the format base64 encode.
E. Step 3: Multi-Modal Diary Generation
Step 3-1: Displaying Text by Questions
In Step 3-1, to retrieve the spoken dialogue text logs, checkthe context (e.g., The reason why said this topic from theelderly) in the records is required. For this, answers of older
adults after the speciﬁc questions (see Table I) are retrieved
and displayed on the Web page that ended with the particular
keyword. Note that the keyword for stopping the spoken
dialogue must be set in advance.
Step 3-2: Integrating Audio and Images with Texts
In Step 3-2, to connect audio and images with texts, a uniquegroup including audio, image, and text with min time intervals
can be calculated and retrieved by their ﬁle names. In this way,
the response text and its group can be shown in the list foreach question. Note that the displayed time is extracted fromthe image ﬁle name, which is easier to send the image than
the text and audio.
Step 3-3: Designing Web User Interface
In Step 3-3, to design a Web user interface to be easier
understood by elderly people, we create a speciﬁc HTML
template put into the cloud server that includes the user’s
nickname, date, and each dialogue log in the time series withmultiple forms (i.e., text, audio, image).
F. Step 4: Heterogeneous Data Analysis
Step 4-1: Sentiment Analysis with Text
In Step 4-1, to analyze the text sentiment from dialogue
logs, many techniques developed in the ﬁeld of Natural Language Processing (NLP) can be used. The typical techniques
include Morphological Analysis and Syntax Analysis. They
are often applied to sentiment-labeled dictionaries to check thepositive and negative degree of the meanings of target texts(e.g., negaposi-analyzer-ja 2 in Japanese).
Step 4-2: Prosodic Extraction to Audio
In Step 4-2, to extract the prosodic from the audio of
dialogue logs, many existing cloud services were developed in
the ﬁeld of Audio Signal Processing (ASP). They are generally
used as a Web Application Programming Interface (API)linking the edge devices. The extracted data often includes
the audio’s rate, pitch, and volume. Moreover, the function ofemotion estimation based on the raw data and training modelis also achieved (e.g., User-Local API 3 in Japanese).
Step 4-3: Emotion Estimation Using Image
In Step 4-3, many existing techniques with cloud computing
and edge computing can be used to recognize the user’s
emotion from the image of dialogue logs. The basic processincludes facial detection, facial feature extraction, and facial
feature analysis. Unlike cloud computing, edge computing
is performed by the pre-training model based on supervisedmachine learning that is not required to connect the network.
The representative edge computing service is the face API
driving tensorﬂow.js [12].
IV. Implementation
A. Technology and Workﬂow
The speciﬁc component of the multi-modal diary service
based on the spoken dialogue agent must be implemented by

123

2 https://github.com/azu/negaposi-analyzer-ja
3 https://emotion-voice-ai.userlocal.jp/

from a dialogue image is drawn in the original image using
the Face API and tensorﬂow.js. (c) The emotional tendency isdetected from the dialogue audio with the User-Local API.

Cloud
Cloud Server- Web Socket- Web Page Source- Database API
Driving

Local
Sensor Driver- Motion- Pressure

Post
Get
Get

Database- Text (MongoDB)- Audio (Nextcloud)- Image (Nextcloud)V. Case Study

Post

A. Designing Usage

Web Browser- Speech Recognition- Multi-Modal Diary UI- Other Source Code

Driving
MMDAgent- Speech Synthesis- Character Material

Edge Server- Tomcat
Driving - ApacheFig. 4. The speciﬁc architecture for implementing multi-modal diary service.

To assist self-care in elderly people at home, the interactionof the elderly with the text, audio, and image recording
functions requires further design clearer. For this, we consider
the following two modes to ﬂexibly use the multi-modal diaryservice based on the spoken dialogue agent:1) Active Mode: The elderly person could ask the virtualassistant the diary contents when he/she could actively
browse the diary on a computer. Our key concept isregarding the multi-modal diary service as a dialogue
scenario that allows calling by the speciﬁc keyword.
2) Passive Mode: The virtual assistant could oﬀer advice
proactively based on the analyzed results of the pastdiary contents. Furthermore, it could also summarize theanalyzed results in a move period of the past time rather
than one single time point. In this way, we believe thatthe dialogue generated will be more personalized.
B. Discussion

Fig. 5. Actual image of calling “diary” with spoken dialogue agent. Copyright2009-2018 Nagoya Institute of Technology (MMDAgent Model “Mei”)

developers to evaluate the usability of the proposed method forelderly people at home. Using the proposed method describedin Section III, we implement a multi-modal diary service by
the following technologies:• General Computer: Lenovo IdeaPad C340.
• Sensor Module: Pressure and Motion (Phidgets).
• Character Material: MMDAgent.
• Speech Synthesis: MMDAgent.
• Speech Recognition: Web Speech API.
• Dialogue Interface: Chrome (version: 100.0.4896.127).
• Data: text, audio, image.
• Database: MongoDB (text), NextCloud (audio, image).
• Cloud Server: Node.js Server.
• Edge Server: Apache Tomcat.
• Communication: Pub/Sub Messaging, WebSocket.
Figure 4 shows the speciﬁc architecture for implementing
multi-modal diary service. An actual image of calling “diary”
with spoken dialogue agent is shown in Figure 5.
B. Results
The example of outputted results is shown in Figure 6: (a) A
dialogue text log is analyzed by the negative-positive judgmentthat colors used to express the degree. (b) The emotional score

An advantage of this study, it extends the spoken dialogue
data from pure texts to audio and images and makes the dialogue logs richer and thicker. Meanwhile, the 1-second photos
extracted from the video stream are the functional approach todata reduction. From the collected data, we found that the size
range of one image and audio is 200 to 500 KByte and 500
to 1200 KByte, respectively. However, as for the limitationof this study, privacy and security issues are important andneed to be studied. The performance of speech recognitionand emotion analysis also has not been experimented with yet.
Moreover, the diary cannot be meant to deal with everyday
actions at home if their related descriptions have not been
received from the elderly. Hence, we consider that it may
require richer information (e.g., non-verbal communication)in further widely period besides dialogue data.
C. Related WorkOne of the critical steps to assist the elderly in self-care athome is collecting daily living data. They include a wide rangeof modalities, from language data in the form of texts andspeech [9], visual data in the form of images and video [10],as well as environmental data such as temperature, humidity
or light [13]. In next-generation healthcare, to provide a moredetailed description of daily living for individuals, collecting
heterogeneous data on time series and providing a multimodal visualization is an excellent signiﬁcant research topic.
Moreover, further analyzing heterogeneous data on time series
is also promising. Such as sentiment analysis to text data [14],emotion estimation to facial image data [15], and prosodic
features extraction to audio data. Their results on time series
can extend to a variety of value-added services in the future.

124

Example of multi-modal diary in JapaneseMulti-modal visualization and analysis

Good morning! How are you feeling this morning?

I feel normal today
There is a meeting in the morning
I have to be conscious of your health from today(a)I have to prepare various things and do my best

CalendarConfirm
Today's schedule

6DGQHVV

Create3HDFH'LVJXVW

10 o’clock today

6XUSULVH$QJHU

Joint research meeting

(b)

+DSS\

)HDU

(c)

Fig. 6. Example of multi-modal diary service in Japanese: (a) Text sentiment analysis. (b) Audio emotion detection. (c) Facial emotion estimation.

VI. Conclusion
In this paper, a multi-modal diary service based on the
spoken dialogue agent is proposed. As a main result, toassist self-care by interaction with the spoken dialogue agent,
it can automatically record the text, audio, and image of
dialogue with the virtual agent when the elderly person sits infront of a computer screen. By the implementation and casestudy, the advantage and limitation of the proposed method
are discussed, including usage and data reduction. In future
work, we plan to deploy the multi-modal dialogue service
in actual households and evaluate the eﬀectiveness of the
proposed method. Extending the multi-language version of
spoken dialogue agents is also an exciting task.
Acknowledgment
This research was partially supported by JSPS KAKENHI
Grant Numbers JP19H01138, JP20H05706, JP20H04014,
JP20K11059, JP22H03699, JP19K02973, Grant-in-Aid for
JSPS Research Fellow (No.22J13217), and Tateishi Scienceand Technology Foundation (C) (No.2207004).
References[1] S. Takatsuki, “Are we ready for the upcoming super-aging society?”
Circulation Journal, pp. CJ–21, 2021.
[2] S. Chen, S. Saiki, and M. Nakamura, “Proposal of home context
recognition method using feature values of cognitive api,” in 2019 20th
IEEE/ACIS International Conference on Software Engineering, Artiﬁcial
Intelligence, Networking and Parallel/Distributed Computing (SNPD).
IEEE, 2019, pp. 533–538.
[3] ——, “Nonintrusive ﬁne-grained home care monitoring: Characterizing
quality of in-home postural changes using bone-based human sensing,”
Sensors, vol. 20, no. 20, p. 5894, 2020.
[4] K. Abe, H. Fuse, M. Okumura, and M. Isshiki, “Development of a life
watching service for elderly people by using interactive home robot,”
International Journal of Informatics Society (IJIS), vol. 12, no. 1, pp.
59–67, 2020.

[5] S. L. Schneider, L. Habich, Z. M. Weston, and C. A. Rosen, “Observations and considerations for implementing remote acoustic voicerecording and analysis in clinical practice,” Journal of Voice, 2021.
[6] H. Maeda, S. Saiki, M. Nakamura, and K. Yasuda, “Rule-based inquiry
service to elderly at home for eﬃcient mind sensing,” in Proceedings ofthe 21st International Conference on Information Integration and Webbased Applications & Services, 2019, pp. 664–668.
[7] H. Ozono, S. Chen, and M. Nakamura, “Study of microservice execution
framework using spoken dialogue agents,” in 2021 IEEE/ACIS 22nd International Conference on Software Engineering, Artiﬁcial Intelligence,
Networking and Parallel/Distributed Computing (SNPD). IEEE, 2021,
pp. 273–278.
[8] A. Lee, K. Oura, and K. Tokuda, “Mmdagent-a fully open-source toolkit
for voice interaction systems,” in 2013 IEEE International Conferenceon Acoustics, Speech and Signal Processing. IEEE, 2013, pp. 8382–
8385.
[9] H. Maeda, S. Saiki, M. Nakamura, and K. Yasuda, “Recording daily
health status with chatbot on mobile phone-a preliminary study,” in 2019
Twelfth International Conference on Mobile Computing and Ubiquitous
Network (ICMU). IEEE, 2019, pp. 1–6.
[10] S. Chen, S. Saiki, and M. Nakamura, “Toward ﬂexible and eﬃcient homecontext sensing: Capability evaluation and veriﬁcation of image-based
cognitive apis,” Sensors, vol. 20, no. 5, p. 1442, 2020.
[11] F. Chen, P. Liu, J. Zhu, S. Gao, Y. Zhang, M. Duan, Y. Wang, andK. Hwang, “Improving topic-based data exchanges among iot devices,”
Security and Communication Networks, vol. 2020, 2020.
[12] H. Agarwal, G. Verma, and L. Gupta, “Student attendance system basedon the face recognition,” Asian Journal For Convergence In Technology(AJCT) ISSN-2350-1146, vol. 7, no. 2, pp. 70–73, 2021.
[13] K. Tamamizu, S. Sakakibara, S. Saiki, M. Nakamura, and K. Yasuda,“Capturing activities of daily living for elderly at home based on
environment change and speech dialog,” in International Conference on
Digital Human Modeling and Applications in Health, Safety, Ergonomics
and Risk Management. Springer, 2017, pp. 183–194.
[14] S. Chen and M. Nakamura, “Generating personalized dialogues basedon conversation log summarization and sentiment analysis,” in The
23rd International Conference on Information Integration and WebIntelligence, 2021, pp. 217–222.
[15] K. Hirayama, S. Chen, S. Saiki, and M. Nakamura, “Toward capturing
scientiﬁc evidence in elderly care: Eﬃcient extraction of changing facial
feature points,” Sensors, vol. 21, no. 20, p. 6726, 2021.

125