Communication Robotics— Life technologies to catch, communicate and connect —Yoichi Yamazaki

Dept. of Home Electronics
Faculty of Creative Engineering
Kanagawa Institute of Technology

1. Introduction

With the spread of AI and IoT, homes and other living
environments are advancing daily, and the government has
designated its vision for Japanese society in the future as “Society
5.0.” In the government’s 2016 5th Science and Technology Basic
Plan, Society 5.0 is advocated as, “A human-centered societythat balances economic advancement with the resolution of social
problems by a system that highly integrates cyberspace (virtual
space) and physical (real) space”[1]. In the 2021 6th Science,
Technology and Innovation Basic Plan, the image of society forSociety 5.0 was revised to reflect rapid changes in Japan andinternationally, and to emphasize the diversity in how people
pursue happiness, as: “Build sustainability and resilience for the
threats faced by society and uncertain conditions yet to come, toensure safety and security for citizens and a society where people
can achieve well-being in their own diverse ways” [2].
To realize this, people must be able to receive meticulous
services anywhere in society, accommodating their needs,
whatever the situation. This will require technologies that enrichdaily life in society, connecting people with systems, with places
and with each other. ICT, AI, IoT, virtual reality (VR), andalso human interface technologies will play a central role inthis. The author is from the Department of Home Electronics,
Faculty of Creative Engineering at Kanagawa Institute ofTechnology, where he conducts R&D in the Life Tech field.
This field connects various issues faced by people in life and workwith engineering, with three core elements of livelihood, social
environment and education. This article describes technologies■ Figure 1: Catch, Communicate and Connect technologies
and applications

that the Communication Robotics Laboratory is working on,
as examples of research to realize richer lifestyles for people, inthree themes: “Catch” technologies that help systems understandpeople, “Communicate” technologies that help robots show
emotion and intentions to people, and “Connect” technologies
that help people connect with each other through use of robots.
We also include examples of collaboration with enterprise on thesetechnologies. Examples of applications in these three themes,“Catch,” “Communicate,” and “Connect” are shown in Figure 1.

2. Life Tech Communication Robotics

As ICT, AI and IoT systems advance and spread, robots
are anticipated to serve as interfaces to these systems. In orderfor such robots will coexist with people in our living spaces,
direct communication such as speaking will be important, but
so will familiarity and non-verbal behaviors, and as we havewith ordinary communication, common rules for these will beneeded[3]. Communication Robotics is a field of research oncommon forms of behavior and expression for robots in contact
with humans, so they can convey information to humans. The
Communication Robotics Laboratory is developing technologies
for robotic movement, representations and systems to convey
information to humans in a casual manner, through developmentof robotic systems focused on expressing emotion and other forms
of non-verbal expression.
In this research, we have selected three technical issues
discussed below, as elemental technologies for realizing richlifestyles for people. The first issue is information technologythat will enable systems to handle emotion and other humanambiguities. General purpose technology for recognizing an
individual’s emotions already exists and in the future, frameworks
and technologies to determine conditions using data from multiple
people will be needed. The second is how to present information
to people. Considering the smart-house example, when a system
works with a person to control an appliance automatically, thesystem must present the actions it is planning to the person
casually, in a way that does not seem troublesome. The third issue
is technology to connect people to others and to spaces throughthe network. The entire living environment will be connected to
the network, providing communication experiences beyond video
calling and email, so that new lifestyles and experiences can beprovided through the internet.
The following sections of this article present technologies

New Breeze Autumn 2023

5

being developed to solve the problems described above on threethemes, which are “Catch” technologies that help systems that canunderstand people, “Communicate” technologies that help robots
show emotion and intention to people and “Connect” technologies
that use robots to help people connect with other people. In recent
years AI technology has also spread rapidly, and basic technologies
that have been accumulating are starting to be implemented inreal living spaces, so that cases of trial research with enterprise are
increasing. Some of these will also be introduced.

3. “Catch” technologies that help systems
understand people

With the spread of IoT, residences are also being converted
to smart houses rapidly, with living-space sensors and home
appliances being connected to networks, enabling them to use
each other. In such an environment, each appliance can beconsidered as an actuator able to control the living environment,
inf luencing not only superf icial physical aspects such astemperature and humidity, but also psychological features suchas the moods of people in the space. To do so, information
technology to enable systems to handle the ambiguity of people, as
well as know-how to apply it in real environments will be essential.
This section introduces an “atmosphere” recognition technologythat is able to understand people and conditions within a residence.
Technology that recognizes an individual’s emotion is alreadycommon; utilizing language and also various other approaches,such as speech rhythm, images and bodily information. However,to combine such emotional information from multiple people andquantify the atmosphere in a space requires a different theoreticalframework. The method for quantifying atmosphere describedin this article uses existing technologies for recognizing emotion
and adopts a bottom-up approach to compute the atmosphereof a space from the emotional information of multiple people.
To model an individual’s emotion, an aff inity-pleasurearousal space[4] is used, and we use a method from Onishi etal.[5] to transform emotional information from several people
to an atmosphere space with three axes indicating friendliness,
liveliness, and casualness. An overview of the living-room
atmosphere-recognition system is shown in Figure 2. Whenfitting ordinary emotion information into an emotion model, it
must be formulated according to the state of the environment,
which will be original know-how.
This technology is being applied in development of a smart
living room that uses a game device as a sensor and links withhome appliances to perform child care[6], in smart appliancesthat calm the atmosphere when watching a TV program, andin development of smart robots[7][8]. It can also be applied toconnect people in distant locations, to develop atmosphere-sharing
systems that connect people for telework, online lessons or otherapplications (Figure 3).

6

New Breeze Autumn 2023

■ Figure 2: Living-room atmosphere-recognition systemusing emotion information from individuals■ Figure 3: Co-creative atmosfield and remote atmospheresharing for telework or online learning

4. Casual, non-verbal expression technologies
that “Communicate” to people

To implement services suited to people and conditions as
required for Society 5.0 will require presentation of detailed
information. On the other hand, conveying an excess of
information could seem annoying and troublesome to people. To
solve this problem requires careful examination and presentationof the minimal necessary information, and the information must
be presented in a casual and familiar way so that it does not
feel troublesome. This section introduces examples of casual
information presentation, focusing on non-verbal expression
technology for familiar information presentation.
4.1 Familiar information presentation
Familiarity is an important element in systems that
present information to people. This is more than showing theinformation concisely, and giving it familiarity can have effects
including increasing a person’s understanding and recall of theinformation[9]. For familiarity, emotional elements are particularly
important. This section introduces a home robot called AHOBO,
which provides on-going frailty care to the elderly in their homes
for everyday lives (Figure 4)[10].
Frailty refers to various changes in faculties and loss of ability
with advancing age, and prevention of frailty in the elderly, or■ Figure 4: Recognition training (left) and health support(right) given by AHOBO

frailty care, contributes to preventing illness and accidents in thehome. Frailty care requires continuous support in everyday life,
and the AHOBO robot was developed to provide such support.
AHOBO is a table-top communication robot that shows emotion
with its antennas and eyes, and implements two types of system tosupport daily life of the elderly, for physical health and to prevent
dementia. For physical health frailty care, AHOBO supports
measurement of blood pressure, and for psychological frailty care,it has a recreational function that provides cognitive training.
4.2 Casual information presentation
When too much information is presented to people at once,
they tend to feel it is troublesome, so it is necessary to present theminimal amount of information in a casual way. In particular,
while walking, driving or otherwise moving, it can be dangerous
to present too much information, so a safe means to convey
information is needed. This section introduces a method ofpresenting information when traveling that is not troublesome,using a device called “FUN’IKI Ambient Glasses.”
FUN’IKI Ambient Glasses are an information device
developed by Namae Megane Inc., that are glasses incorporating
full-color LEDs and a compact speaker. They are able to present
information linked to a smartphone in an intuitive way, using
sounds and LED lights that switch on and off [11]. An example of
FUN’IKI Ambient Glasses emitting light is shown in Figure 5.
By turning on LEDs in the frame of the glasses, the entire lens
glows, and this can be used to convey information.
The benefits of FUN’IKI Ambient Glasses include: (1)information can be checked while working, without looking awayfrom the work, (2) the information presented does not obstruct
the field of view, (3) information can be presented intuitively, with■ Figure 5: FUN’IKI Ambient Glasses■ Figure 6: Presenting bicycle riding conditions using
FUN’IKI Ambient Glasses■ Figure 7: Navigation system using FUN’IKI Ambient
Glasses

low cognitive load, using color and sound, and (4) it is possible
to present information to the surroundings at the same time aspresenting information to the wearer. This could be suitable fortypes of mobility that require a certain amount of attention, suchas walking or driving, because it requires less cognitive load thansmart glasses that display text. The FUN’IKI Ambient Glasses
have been used in applications such as visualizing conditions whenriding a bicycle[12], or providing map guidance when walking or
running[13].

5. T echnologies that “Connect” people withpeopleEntire living environments will become connected to
networks, creating environments with communication experiences
beyond video calling and email, making new lifestyles andexperiences possible through the internet. We are aiming tocreate disruptive innovation initiated in Japan, and to implementa cybernetic avatar platform that will enable anyone to participate
in various social activities, as part of goal 1 of the “Moon-shot
research and development program” being promoted: “To realize
a society in which people are freed of constraints of body, mind,
space and time, by 2050” [14]. More specifically, we aim to expand
the opportunities and potential for social activity by remotely
operating cybernetic avatars, which includes robots and virtual

New Breeze Autumn 2023

7

8avatars, as “additional bodies for the user.” This will require
technologies that connect people with other people, and people
with spaces, as discussed in Section 2. The technology introducedin Section 3 for sharing atmosphere for telework or online learning
is an example. This section describes trials of another example of a“Connect” technology, doing remote work and avatar work at theAvatar Robot Café, using avatar robots that OryLab is promoting
in collaboration with the authors.
Robots that operate from a remote location through theinternet in place of a person are called telepresence robots. Various
types of telepresence robot are being developed, such as thenewme (avatarin Inc.)[15], which emphasizes communication andcombines a mobile robot with a display, and humanoid robots thatemphasize a sense of presence.
Telepresence robots enable lifestyles in which people canbe transported instantly, overcoming constraints of space, butthey also have potential to extend a person’s bodily capabilities.
This is very significant, particularly for persons with disability.
OryLab Inc. has set its mission to eliminate loneliness throughtechnology, and is developing avatar robots that enable people
who have given up on communication due to disability, to go out
and communicate with people through the internet[16]. There
are two types of avatar robot: a table-top model called OriHime(Figure 8) and a mobile model called OriHime-D (Figure 9).
Both types of OriHime are equipped with a camera, microphoneand speaker and can be operated through the internet using a PC,
tablet or smartphone application (Figure 10). In addition to input
using a mouse or tablet, line-of-sight input using an intentioncommunication device (OriHime eye+Switch) is also available.
Providing OriHime at schools or companies can enable those withdisabilities that make it difficult to go out, to participate in society,attending school or work while they are at home or in the hospital.
OryLab has proposed Avatar work as a system that can enable
those with difficulty going out to perform remote work using
OriHime from home or hospital, and is operating a café space for■ Figure 9: OriHime-D■ Figure 8: OriHime■ Figure 11: Avatar Robot Cafe concept image

New Breeze Autumn 2023

■ Figure 10: OriHime-D operating systemavatar work called “Avatar Robot Café DAWN Ver. β,” as a trialsocial implementation[17]. A concept image of Avatar Robot Café
DAWN Ver. β is shown in Figure 11.
Avatar Robot Café DAWN Ver. β was opened as a timelimited-trial shop in November 2018, and after trials in various
regions, a permanent trial shop was opened in Nihonbashi, Tokyoin June 2021.
To work at Avatar Robot Café, operators (pilots) that havedifficulty leaving home provide service by operating OriHime
and OriHime-D robots remotely. OriHime are used to provide
customer services such as taking orders, and OriHime-D are used
to perform service tasks that require physical movement, such as
bringing drinks to the table.
One of the important benefits of avatar robots is the ability toshare the experience of a place with others using a physical robot.
By sharing an experience with the person in front of the avatar
robot through the robot, a person can get a real sense of actually
being in the place, even from a remote location. In research so far,working at the Avatar Robot Café has been shown to be mentally
fulfilling for pilots and has been suitable for many disabled people
desiring participation in society[18]. As with Avatar Robot Café,
by using avatar robots that expand bodily capabilities of people
with disabilities and by creating a society where avatar robots are
ubiquitous, we believe we could realize a society in which people
can participate in various activities, even if they are bedridden, forexample.

Smart Meter and Appliances in Home Energy Management System,” Proc. of the 2011IFSA World Congress and the 2011 AFSS(IFSA-AFSS 2011), RW-101, Jun.2011.
[10]	Yoichi Yamazaki, Masayuki Ishii, Takahiro Ito, and Takuya Hashimoto, “Frailty Care Robot
for Elderly and its Application for Physical and Psychological Support,” J. Adv. Comput.
Intell. Intell. Inform., Vol. 25, No.6 , pp. 944 -952 , 2021. DOI：10. 20965/jaciii.2021.p0944[11]	FUN’IKI Ambient Glasses (Namae Megane Inc.) http: /fun-iki.com/[12]	Wataru Kutsuzawa, Yoichi Yamazaki, Kei Shiratori, Shusaku Tane, Kazunori Kawamura,“Visualization of Driving Situation in Bicycle by LED Color of FUN’IKI Ambient Grasses,”
HCG Symposium 2017 Proceedings, HCG2017-B-4-3, 2017.12[13]	Takumi Ishikawa and Yoichi Yamazaki, “Distracted Walking Accident Prevention by LED
Color and Sound of Glasses-type Wearable Devices,” 2020 IEEE International Conference
on Consumer Electronics (ICCE), Las Vegas, NV, USA, 2020, pp. 1-2, doi: 10.1109/
ICCE46568.2020.9043080.
[14]	Cabinet Office, “Moonshot Research and Development Program”, https://www8.cao.
go.jp/cstp/moonshot/[15]	newme (avatarin Inc.), https://about.avatarin.com/[16]	OriHime (OryLab Inc.), https://orihime.orylab.com/[17]	Avatar Robot Café “DAWN” Ver. β (OryLab Inc.), https://dawn2021.orylab.com/en/[18]	Kazuaki Takeuchi, Yoichi Yamazaki and Kentaro Yoshifuji, “Avatar Work: Telework for
Disabled People Unable to Go Outside by Using Avatar Robots “OriHime-D” and Its
Verification”. In Proceedings of the 2020 ACM/IEEE International Conference on Human
Robot Interaction(HRI’20), March 23 26, 2020, Cambridge, UK. ACM, New York, NY, USA
8 pages. https://doi.org/10.1145/3371382.3380737

6. Conclusion

This article has introduced examples of communication
robotics research to enrich the lives of people in the three themesof “Catch” technologies, which help systems understand people;“Communicate” technologies, which help robots express intention
to people in familiar and casual ways; and “Connect” technologies,
which help people connect with others through use of robots,
including examples of collaboration with enterprise. These
technologies will contribute to creating the society that Japan isaiming for, in which each and every person can realize their own
version of well-being.
References[1] Cabinet Office, “5th Science and Technology Basic Plan,” 2016.
[2] Cabinet Office, “6th Science, Technology and Innovation Basic Plan,” 2021.
[3]	Yoichi Yamazaki, Yutaka Hatakeyama, Hajime Nobuhara, Kaoru Hirota, “Eyeball Robot
Proposal for Mascot Robot Systems,” IEICE Technical Report SIS 2005-70 (2006-03), pp.
61-64, 2006.
[4]	Yoichi Yamazaki, Yutaka Hatakeyama, et al., “Fuzzy Inference based Mentality Expression
for Eye Robot in Affinity Pleasure-Arousal Space,” Journal of Advanced Computational
Intelligence and Intelligent Informatics(JACIII), Vol.12 No.3, pp.304-313, 2008[5]	Kazuhiro Ohnishi, Yojiro Adachi, Yoichi Yamazaki, Fangyan Dong,Kaoru Hirota,“Transform Functions from Affinity Arousal-Pleasure Space into Atmosfield”, Proceedingsof the 28th Fuzzy systems Symposium, pp. 904-909 (TL2-1), 2012.9[6]	Yoichi Yamazaki, Takuya Hayama, Masanobu Nakamura, Wataru Kutsuzawa, Kazuaki
Takeuchi and Masao Isshiki, “Home Atmosphere Visualization Based on Motion Sensing
in Smart Living Room”, Proc. of 2020 IEEE 2nd Global Conference on Life Sciences andTechnologies (LifeTech), pp.280-281, Mar. 2020[7]	Naoki Sato, Kazuaki Takeuchi, Masao Isshiki, Yoichi Yamazaki, “Eye contact expression
promoting spontaneous behavior by robot using atmosphere estimation,” Proceedings ofthe 79th National Convention of IPSJ, pp. 3-509-510 (7V- 06), 2017.3[8]	Yoichi Yamazaki, Takuya Shinkai, and Masao Isshiki, “Smart Robot based on Web
Resource with Friendly Communication and Its Application to Library Guide,” Proc. ofInternational Workshop on Informatics 2017
(IWIN2017), pp.249-253, Sept. 2017.
[9]	Yoichi Yamazaki, Yasutaka Yoshida, Makoto Motoki, Takuya Hashimoto, Yutaka
Hatakeyama, Fangyan Dong, and Kaoru Hirota, “Qualitative Expression using AHOGE for

Cover Art
Yōrō Waterfall (Yōrō notaki), Mino Province,from Famous Views of the
60 Provinces

Utagawa Hiroshige (1797-1858)
Source: National Diet Library,NDL Image Bank(https://rnavi.ndl.go.jp/imagebank/)

New Breeze Autumn 2023

9