Photo based improvement of virtual living room
system using video call to connect family moderately
Genki Uryu

Kenji Funahashi

Shinji Mizuno

Nagoya Institute of Technology, Japang.uryu.880@stn.nitech.ac.jp

Nagoya Institute of Technology, Japankenji@nitech.ac.jp

Aichi Inst. of Tech., Japans mizuno@aitech.ac.jp

Index Terms—Virtual living room, Video call, BLE beacon

II. V IRTUAL LIVING ROOM SYSTEM
A. Real living room

I. I NTRODUCTION
Many senior citizens stay in a hospital, or they are moved
into sheltered accommodation. When they live apart from theirfamily, there is a risk of them suffering from serious disturbance in mental ability including delirium due to stress [1][2].
Their family can visit hospital or sheltered accomodation andmeet senior citizens so as to reduce their stress and prevent
them from suffering serious disturbance. However they might
not be able to do so due to epidemics. They can communicate
with their family as in the case of life when they live insame house by making constant video calls instead of visit.
However, they may feel uncomfortable with constant shooting.
We feel a need of the system that they can communicate
without feeling uncomfortable even if senior citizens live apartfrom their family. When we came to think of the place where
the families lived in comfortably, the living room came to ourmind. Family live freely in the living room, spending much of
their time without a strong connection each other. But a person
can start a conversation at any time by talking to anotherperson, and the conversation will end naturally without clear
termination signal. For this reason, we have proposed “Virtualliving room system” [3] that had these feature and reproduceda real living room. We made an experimental system in orderto verify the validity of the system and conducted experimentsthat we had subjects live while using it. In the experiment,
we conﬁrm the usefulness of the system. However, we also
conﬁrm two problems through the opinions of participants.
One is the lack of presence. The system displays an icon on thescreen that represents the the information of whether personthat lives away from the user is in a certain room or not. Theicon is not related to the user, so it may not have made theusers feel that they live in the same place. The other is thedifﬁculty of setting up for using the system. Setting up thissystem is not easy for people who are not very familiar withcomputer operation. The system will actually be used in a
hospital or nursing home. Therefore the staff work in would
have to set up the system and help the user use it. They may
be unfamiliar with computer operation and probably cannot
set up the system and use it by theirselves. In this paper, we
propose an improved system based on photos that is createdin view of the problem and the actual users.

In this section, the main points considered in developing
the prototype system [3] is described. The living room iswhere the family lives comfortably. We think that the system
reproduces the real living room connects senior citizens wholive apart from their family and their family moderately.
Therefore we think that using the system enable the seniorcitizens to communicate their family and reduce their stressdue to environmental changes. We think about the features
the living room and some of the features we should have thesystem reproduce. We decide these feature as below as thefeature we should have the system reproduce.
• A person in the living room can see at a glance whether
another person is there.
• A person can start a conversation at any time by talking
to another person.
• The conversation a person starts ends naturally without
clear termination signal.
• When someone leave the living room, people feel thething.
B. Implementation method
In this section, we discuss the technical elements to build theactual construction of the prototype system [3]. In the livingroom, you can see at a glance whether family is there. A person
needs to be able to know whether the others are the sameroom when using virtual living room system. On the otherhand, people feel a person’s leaving the living room when
he leaves the living room and a person’s entering the livingroom. A person needs to be able to feel the partner’s leaving
the living room and entering the living room. The system wehave proposed has three functions. One is to show the statusof the partner, and another is to activate a video call by voicerecognition of a belief word (Fig. 1) and closed it if they staysilent each other for a certain period of time. The other isto display animation when the partner leave the living roomand enter the room on the screen (Fig. 2). These functions
reproduce the features of the real living room. Therefore wethink that these functions can make the users to feel that they
are in the same place. The system shows an icon that represents
the information whether partner that lives away from the user
is in a certain room or not on screen (Fig. 3). The partner

Fig. 1. Video call activated by belief word

Fig. 3. Previous system screen and appearance of experimentIII. I MPROVEMENT
A. Photo based notice
We thought that an icon displayed on the screen would be
sufﬁcient for the users to know at a glance the status of theother. However, there was no immersive experience, and they
did not feel that they were in the same space. Thus, we propose
the system that displays users’ actual photos superimposed ona photo of the room in which the users actually live insteadof the icon and a plain background (Fig. 4). However thesefunctions are not enough because it may be difﬁcult for theuser to imagine what the partner does by seeing the photo,and he may not feel that they are in the same room. For thisreason, we feel the need of the function that indicates the area
where partner is on to the user. The photo of each person
moves on one photo of the living room, like moving from one
area to the other area, according to the actual their movement(Fig. 5). In addition, the farther away people are, the smaller
people looks. Therefore our system displays people who are
farther away.
B. Fixed installation beacons and mobile phone receivers

Fig. 2. Animation displayed on the previous system screen when the partner
leaves the room

can see the status of the other at a glance. Skype is used asa subsystem instead of developing original video call system.
The main system sends the necessary information to the othersystem through Skype text chat function (Fig. 3). BLE beaconsthat are been worn by them are used to judge whether theyare.

However, it is difﬁcult by using only beacons and one PC asa receiver like as our previous system to realize the functions
that estimates the area where the user is on for indicating
the area where partner is on to the user. Our previous systemcannot estimate the area of a person wearing a beacon when
they are in a large room like living, dining and kitchen areas.
One solution is to increase the number of PC as a receiver andput on the PCs around the room, but it has a problem withcost. By the way, many people have mobile phones recently.
We think that we use the mobile phones as receivers of signalof beacons. We decide that the user sets beacons on the roomand carries their mobile phone. We develop an application thatsignal strength of the beacons and sends the information to thePC. User puts the beacons on the room separately and carries
the mobile phone instead of carrying the beacon (Fig. 6). Thesystem can judge the area the partner is on and the user can
conﬁrm it.
C. How to setup the system
However it takes time to register photos such as section A
and it is difﬁcult for the users to understand what they should
do such as section C. In short, it is difﬁcult for the people who

Fig. 4. The screen user sees of the system

are unfamiliar with computer operation to setup the system.
This system will be used in hospital or nursing home, andthe staff may be unfamiliar to computer operations. Thus, we
provide a GUI-based conﬁguration function so that even suchpeople can use the system easily. For setting the system interms of hardware, user should follow instruction displayedon the system as below.
1–1) For the family side, user puts three beacons on threeplaces of the room separately.
1–2) User clicks the button displayed on the screen of thesystem works on each person’s mobile phone and leaves
them as they are (Fig. 7).
1–3) User clicks on the point with the beacon in the room
picture from near to far from the camera (Fig. 8).
Fig. 5. The change of the system screen user sees

2–1) For the elderly side, user puts the beacon on the centerof the room where elderly lives in and click the button
displayed on the system screen and collects it.
2–2) User puts the beacon on near the exit of the room where
elderly lives in and click the button displayed on thesystem screen and collects it.
For setting the both systems in terms of software, user should
follow instruction displayed on the system as below.
1) User takes a picture of the room where the family live inand a picture of the room where the elderly lives in.
2) User takes pictures of the each family member and theelderly so that their bodies are the center of picture.
3) User saves the pictures of the people on one side and theroom where they live in on PC used on the other side.
4) User clicks the button displayed on the screen to makethe system read the pictures. Photos will be cropped to aﬁxed size.
If the user is not familiar with computer operations, the users
can setup the system we propose by following the instruction
displayed on the system screen step by step in this way ifthey ﬁnish making Skype’s account creation and preparing
execution environment of the system.

IV. E XPERIMENT
A. Outline
We need to see if our systems reproduce the real living roomand make the users feel that they live at same place. On theother hand, we need to see if our new system overcome theproblem our previous system have. Therefore we conducted
experiment that we let the subjects use both our previoussystem and our new system. First, the participants, an elderly
woman and her family, used our previous system in theirFig. 6. Usage example of the system

respective homes, which were physically separeted from eachother. After a while they used our new system (Fig. 9), andcompared these two systems. We obtained opinions from theparticipants about two systems.
B. Opinion from subjects
In regards to the previous system, we got positive opinionsfrom the subjects as below.
• I felt more freely and comfortably when I live while using
this system than when I have lived with constant shooting
because camera is on only when necessary.
• I felt free to communicate with the partner, because I
did not feel that the connection termination of video chatmeant loss of communication with the partner.
• I talked with the partner and communicated with him
naturally as when we were in the same space because
the video chat terminated when we stayed silent eachother.
However, we also got an negative opinion from a subject. An
elderly person of them also said that she did not feel that she
lived together with their family at the same place because thestatus of her family that was displayed on screen was an icon.
In regards to our new system, they also said that the system
reduced the uncomfortable feeling about the constant video
call. In addition to the positive opinion, we also obtained apositive opinion from the elderly who was one of the subjects
about the presence and operability of the system.
• I felt that we lived together with my family at same placebecause the system displayed photos of my family andthe room photo that we lived in instead of the icon.
• I could have understood the instruction displayed on
easily and I could have ﬁnished the setting up of thesystem.
• I was able to ﬁnish setting up the system and using thesystem thanks to easy-to-follow instructions displayed on
PC.
C. Consideration
We conﬁrmed three things about our new system. We
conﬁrmed that the systems reduced the uncomfortable feelingFig. 7. The example of placing the beacons and taking the picture of theroom (family side)that they felt when they lived with constant video call. Also,
we conﬁrmed that our new system overcame the problem of
the previous system about the presence. We conﬁrmed thatpeople who are not familiar with computer operation can use
our new system easily.

V. C ONCLUSION
In this paper, we proposed an improved system using photos
based on our previously proposed system. It was conﬁrmedthat the new system was improved about the presence, andmade users feel that they live together with their family at
the same place. It was also conﬁrmed that the system is easy
to use, even for those unfamiliar with computer operations.
However, the participants in this experiment did not live ina hospital or nursing home, and the experiment was a simple
one, using the system for a few hours. In the future, we would
like to conﬁrm the usefulness of this system by having hospitaland nursing home residents and their families participate in theexperiment and asking for their opinions.

Fig. 8. Setting up the system

Fig. 9. Appearance of experiment (elderly person side)ACKNOWLEDGMENT
This work was supported in part by JSPS KAKENHI Grant
Number JP20K11918 and JP22K12337.
R EFERENCES[1] A. Matui, M. Yatsuzuka, S. Takabatake, Y. Mukaiyama, K. Hasegawa,and K. Tazawa, “Study on the delirium-inducing factors after operationin aged surgical patient,” The journal of the Nursing Society of theToyama Medical and Pharmaceutical University, vol. 6. no. 1. Japan,July 2005.
[2] American Psychiatric Association, “American Psychiatric Association
Practice Guideline for the Treatment of Patients With Delirium,” 1999.
[3] G. Uryu, K. Funahashi, “Virtual living room system based on video callto connect nursing home resident and family moderately,” NICOGRAPH
International 2021 , Japan, July 2021.