TVRSJ Vol.25 No.3 pp.277-283, 2020

基礎論文 Paper

Development of a Telediagnosis System using
Telexistence
Junkai Fu*1 Fumihiro Kato*2 Yasuyuki Inoue*2 Susumu Tachi *2
Abstract --- This paper proposes a telediagnosis system that allows medical staff to examine
remote patients through a telexistence robot with tactile sensor/display devices. The system
consists of three components, including a Six Degrees of Freedom (6 DoF) movable audio-visualtelexistence system for telecommunication, body temperature and heartbeat measurement
equipment, and a skin-imitated tactile display equipped with thermal/pressure presentation
devices. Compared to conventional videophones, this multimodal mixed reality (MR)telediagnosis system is expected to allow medical staff to examine patients more carefully as if
they are observing patients in person．Twelve experts (ten medical doctors and two nurses)
evaluated the system. The results indicate that the system shows a positive feasibility for
improving the realism of telediagnosis. Subsequently, the effects of telexistence with tactile
information on telediagnosis are discussed.
Keywords: Telediagnosis, Telexistence, Haptics, Mixed Reality

1 Introduction
In recent years, several countries have turned into an
aging society. The cost for the elderly to receive adiagnosis is always high, in terms of time spent andexpenses. Hence, healthcare targeted at the aged must be
improved urgently. A suitable means to achieve thisobjective is telemedicine. Traditional telemedicine
concentrates on using information technology such as
audio and video to provide medical diagnosis and othertreatment from professional doctors to patients. It
reduces the cost of medical treatment and cross-infection
risk for infectious diseases.
However,conventional
telemedicine
through
information technology (IT) systems such asvideophones has several defects, such as its use only
after initial diagnosis. The patient’s condition has alreadystabilized, and the diagnosis process is more inclined tomake a consultation rather than a medical examination.
This is because only a small amount of medical
information can be transmitted to the doctor. The initialdiagnosis always contains several examinations to obtainessential bio-signals, which conventional telediagnosis
cannot provide. Without this information, medical staff
cannot determine the disease.
2 Telemedicine System
Combining virtual reality (VR) with healthcare has

been proposed for decades. Early surveys reported that
research using virtual reality to perform surgery andmedical education dates as far back as the 1990s[1].
After that, many studies were conducted about haptic
telemedicine[2, 4], online[5], and massive patienttransaction[3].
With the development of VR/AR/MR, an increasing
number of VR/AR/MR applications and studies intelemedicine has emerged. Telepresence surgery has
already been commercialized for several years. Moreover,due to its high accuracy, surgery systems such as the daVinci telerobotic surgical system have been increasingly
used[6]. Additionally, palpation based on VR simulation
has been proposed. Coles et al. use artificial force tosimulate the feeling of human skin and apply the system
into palpation and insertion training[7]. To achieve
telemedicine, a robot with its partial function as a humanis also a viable way. Garingo et al. apply a mobile robot
into neonatal intensive care to help people from a
distance[8].
3 System Requirements
To enable palpation between people in different places,
this paper proposes a telediagnosis system with a tactile
display. We hope this tactile display can be used indiagnosing difficult illnesses that cannot be determined
without touch. Moreover, certain illnesses, whentransferred into numerical data, seem obscure; however,symptoms can be easily understood upon being*1 Graduate School of Frontier Sciences, The University of Tokyo*2 Institute of Gerontology, The University of Tokyo

―277―

Transactions of the Virtual Reality Society of Japan Vol.25, No.3, 2020

diagnosed by touch. The proposed system reduces the
distance between the patient’s home and the medical
institution, eliminates the need for the aged to check in toa hospital, and enables them to receive a diagnosis at
home.
The basic skills of medical examination include
percussion, auscultation, and palpation. Pulse, bloodpressure, and body temperature are usually measured asvital signs. We decided to reproduce palpation to acquire
these vital signs as a minimum requirement for
telediagnosis. Through palpation, medical staff can know
the patient’s pulse rate, sweating, swelling, tenderness,and temperature, or other conditions. Several diseases
can be judged through palpation. In our system, we focus
on reproducing temperature and pulse as these are vital
information but more comfortable to measure andreplicate. A human’s pulse regularly ranges between
60–100 bpm. If the proposed system works effectively,
additional bio-signals can be added to it to enhance its
usability. Certain diseases may cause an increased heart
rate beyond the normal range. For example, paroxysmal
supraventricular tachycardia (PSVT) is the mostcommon symptom of tachycardia. In this condition, thepulse rate can reach 140–180 bpm. In addition, thetemperature and pulse are positively correlated. Usually,
pulse increases by 10 bpm when the temperature rises by
0.5℃. Therefore, in response to the possibilities, weneed to set the reproducing pulse range between 40–200
bpm on a surrogate arm to simulate arrhythmia. To thisend, we choose a surrogate robot and use a surrogate arm
to achieve the system. The system must follow theseprinciples[9]. It must be real-time to ensurecommunication quality. A doctor must be able to see,hear, and touch the patient to judge the condition, andalso see their body in the virtual space to ensure
immersion and interaction.
4 System Design & Implementation

Fig.1 System schematic diagram

The system consists of two parts: the patient’s side, andthe doctor’s side (Fig. 1, Fig. 2).
Patient’s side: The patient is made to wear a glove
with sensors and sit in front of a 6 DoF robot. Withsensors embedded in the robot and the glove,
audio-video information and bio-signals of the patient
are recorded and transmitted to the doctor’s side. Thispart comprises two components: (1) A doctor-surrogate
robot perceives the surrounding information, including
the patient’s appearance and voice and replicates thevoice and gestures recorded from the doctor’s side. (2)Sensors on the glove measure the patient’s pulse andtemperature. The sensor data are stored and processed ina micro control unit (MCU), and subsequently, bio-signal
information is transmitted to the doctor’s side.
Doctor’s side: The doctor wears a head-mounted display(HMD) connected to the surrogate robot on the patient’sside, which reconstructs the surrounding of the patient’s

Fig.2 System data diagram―278―

Fu, Kato, Inoue, Tachi : Development of a Telediagnosis System using Telexistence

room. In addition, the HMD is equipped with a stereo
camera to achieve video-seethrough capability; that is,the doctor can recognize their hands overlapping in theimage of the patient’s side. Except for the hands, theimage on the doctor’s side is eliminated using chroma
key screens in a Unity project. This method realizes
mixed reality. Furthermore, a patient’s surrogate arm
equipped with several devices is placed in front of thedoctor to reproduce the patient’s bio-signal information.
The system is thus expected to allow the doctor to feel as
if they are diagnosing the patient face-to-face
4.1 Audio-video Reproduction
4.1.1
Surrogate robot at patient’side
The robot (RT-Telbee) used on the patient’s side isequipped with a stereo camera (ZED mini), binauralmicrophone, and speaker, enabling it to communicate
like a human. Further, it has a 6 DoF mechanical system(Fig. 3), which allows the doctor’s side to connect it toan HMD (Oculus Rift CV2, Facebook Technologies,
LLC)[10] to control its movement. A stereo camera isaffixed on the robot’s head. When the robot moves, theview of the camera is also simultaneously transferred tothe doctor’s side.

Fig.4 MR environment in the HMD

4.2 Bio-signal reproduction
4.2.1
Detecting glove
The patient’s side uses a glove equipped with a pulse
sensor (PulseSensor, World Famous Electronics llc.)[11],a negative temperature coefficient (NTC) thermistor(56A1002-C3, Alpha Sensors, Inc.), and an MCU(ESP32-WROOM-32D, Espressif Systems Pte. Ltd) todetect the pulse and temperature (Fig. 5).

Fig.5 Bio-signal detecting glove

4.2.2
Surrogate for reproduction
The doctor’s side receives bio-signal and reproduces
the pulse and temperature on a surrogate arm, as depicted
in Fig. 6. Under the artificial skin (a mixture of silicone
and Al2O3 powder), a heater generates heat. A
mechanical crank system applies pressure to the liquid in
the tube under the wrist to imitate the pulse.
Fig.3 Construction of 6 DoF robot RT-Telbee

4.1.2
Mixed reality at doctor-side
To increase the realistic feeling and maintain thepresence of the doctor’s side, we attached a stereo
camera on the HMD and extracted the doctor’s handfrom the local surroundings by using chroma key
composing. The doctor can see his hand while wearing
the HMD, and the hand is overlapped with thebackground transmitted from the patient’s side, as seen in
Fig. 4. As the HMD is tracking with the help of sensors
and connected with the surrogate robot on the patient’sside, while they move, the vision in the HMD alsochanges.

Fig.6 Mechanism of the reproduction system

However, we found that the shape of the surrogate
does not matter and, therefore, developed a smaller one
with a cooling module to replace the surrogate arm, as
seen in Fig. 7. A heater cooperates with a Peltier element
to match the temperature to that of the patient.

―279―

Transactions of the Virtual Reality Society of Japan Vol.25, No.3, 2020

The results are reported in Fig. 9:

Fig.7 New surrogate with a cooling module

Fig.9 Evaluation of User ExperienceFig.8 MR system in two sides (upper) and the view from thedoctor/patient (below)

5 Experiments
The purpose was to test if this telediagnosis system isuseful, and more specifically, to determine if it could
provide a better reality as compared to the traditional
telediagnosis system. Can tactile display offer more clues
and benefits to the telediagnosis? We conducted anexperiment with 12 medical experts (ten medical doctors
and two nurses). The MR system used in the experiments
is shown in Fig. 8. The participants were required tomake a complete diagnosis akin to a real medical
examination by using this MR system and a traditional
videophone. The procedure was almost identical with the
exception of the videophone method, which did not offer
palpation on a tactile display.
After the experience, participants were administered
two questionnaires to evaluate the system, followed byan interview.
6 Results
After using each system (VR and videophone),questionnaires included the following four statements:
Q1. I felt as if I was actually in the room with the patient.
Q2. I felt as if the patient was actually in front of me.
Q3. I felt as if the medical examination was actually
performed.
Q4. I felt tense while performing the medical examination.

Evaluations by 12 experts indicated that scores for the
VR system are significantly better than the videophonesystem, with the exception of question2, i.e., “patient’s
presence.”
For the first question, the score of self-presence in thisVR system is much better than the videophone. This
indicated that telexistence was accomplished; the user’s
presence was successfully transferred to a remote place.
This difference is a benefit from the high immersion of
the telexistence system; while the participant moves his
body, what they see will also change according to his
movements. When the participant reaches out for
palpation, they can also feel the tactile feedback.
However, these features cannot be realized using thevideophone alone. These factors ensure that the system
has a better self-presence. During the experiment of the
VR system, we noticed that doctors sometimes tried toread medical records. However, due to the poor qualityof the image in the HMD, it is difficult to recognize thetext. If the video quality can be upgraded, or if a virtual
medical record can be added as an MR object to allow
the doctor to read the text naturally and comfortably,
self-presence can be further improved.
Second, regarding the patient’s presence, thedifference between the two average scores is not very
obvious. Moreover, through the t-test, the p-value is alsomuch higher than 0.13. From the interview after theexamination, we ascertained that the main cause of thissituation is the image quality in HMD. The vision in the
HMD, which is cached by the stereo camera, isstereopsis like a human. It is supposed to be better thanthe videophone in presence. The reason several experts
rated the VR system lower is probably due to the quality
degradation of head tracking while performing theexamination. According to experimenters’ observations,when experts looked down for palpation, the angles of―280―

Fu, Kato, Inoue, Tachi : Development of a Telediagnosis System using Telexistence

the experts’ head occasionally tilted beyond the robot’s
range of motion, causing visual dislocation. While thepulse and temperature could be felt, the touched area was
not the wrist in the participant’s perspective. This
inconsistent feeling could significantly decrease thepatient’s presence. Moreover, compared to question 1,
although the images from both sides are cached by thestereo camera, the local side’s image quality is usually
better than the remote side benefited from thetransmission method and has less noise. Therefore, thisvalidates the result that the performance of self-presence
is much better than the presence of the patients.
Furthermore, the tactile display lends an indispensable
devotion to the score of question 1.
Third, regarding the realism of examination, theexperts show a positive overall judgment indicating thatby using the VR system, participants could have a more
realistic feel for the examination. This questionnaire item
refers to their usual consultation experience. As all thesubjects who participated were expert medical staff withsignificant diagnosis experience at hospitals, they could
compare the proposed systems with their daily diagnosis,
and the scores could be graded easily.
Fourth, scores of tensions show an importantdifference between the two methods. The tension heremeans the level they devoted to this telediagnosis. It
reflects how seriously the experts treat the process of thediagnosis to a certain degree. Before the experiment, we
intended to find something like the index of theconcentration during the examination. After discussions
with the medical staff, we assumed that actual medical
practice creates a certain degree of mental stress for them,and the tension is directly proportional to theconcentration[12]. The results revealed that the VR
system brings more stress. As anticipated, this means thatthe experts treat it more seriously and like an actual
diagnosis. However, according to the interview after theexperiment, some participants said that the unfamiliar
experience in VR environment will also induce somestress. Therefore, this influence should also be accounted
for in this result.
After all the experiments, participants were required toshare their impressions of the VR diagnosis system.
As reported in Table 1, most participants had neverusedsimilartelemedicine
system(including
videophones) before. However, some reported that they
had on occasion performed examinations using atelephone. This suggests that even videophones were yet
to become popular in clinical examinations. Furthermore,

Table 1 Answer of questionnaire2
Q1: Have you ever experienced another

YES

NO

1

11

1

0

10

1

12

0

10

1

11

1

telemedicine system?
Q2.1: If Q1 is “YES,” do you think the VR
system is more real?
Q2.2: If Q1 is “NO,” do you think the VR
system is real?
Q3: Do you think VR experience is usefulfor the examination?
Q4: Do you think the tactile display is usefulfor the examination?
Q5: Do you want to use a VR telediagnosis
system?

most participants agreed that the VR system presented arealistic experience akin to an actual examination. They
generally indicated a positive attitude towards the VR
system with the tactile display and agreed on thesystem’s usefulness in clinical examinations. Therefore,a majority of participants answered that they wanted totry the VR telediagnosis system in real practice.
According to questionnaire results, experts rated the
proposed system higher than traditional systems.
However, interviews revealed that the positive evaluation
is likely to include expectations of the VR telediagnosis
system developed in the future. This is because problemsof the current system—particularly in terms of thequality of video image—were significantly evident. Even
though some respondents answered with a positive
evaluation of the system in questionnaires, they also said
later that the view of the VR system was poorer thanvideophones. Therefore, several factors emerged for theirinconsistent evaluation. First, the resolution of the HMD
used in the system is still insufficient to provide a clear
view in comparison to the normal PC display used forvideophones. Moreover, the quality of the video image
captured by the stereo camera is still poor. Second,
because the robot’s head movement had delays andvibration, video images of the VR system were degraded.
These could cause distraction and reduced immersion to
the experts during the examination. However, limitations
caused by hardware issues can be resolved by theirimprovement[13]. In addition, if they kept their heads
steady, the degradation of the video image disappeared.
With many sensory clues such as immersive stereo
view and touch, it is probable that they can convince theusability of the VR telediagnosis, even though thecurrent system is still developing. In fact, we received―281―

Transactions of the Virtual Reality Society of Japan Vol.25, No.3, 2020

many comments on the current system limitations. For
example, skin color is often used to judge the presence ofsymptoms such as inflammation. Therefore, chromaticaberration of the camera image obstructs the correct
judgment of the expert. One respondent claimed thatchromatic clues must be presented more correctly.
Actually, the determining process of disease iscomplicated. Therefore, to derive a correct conclusion, it
is necessary to obtain many kinds of bio-signals withhigh accuracy. Only with enough information about thepatient can the doctor confidently determine the disease.
In this context, palpating the heart rate and bodytemperature is insufficient. According to anotherparticipant’s comments, the skin’s thickness, hardness
and surface moisture always help with the diagnosis.
Nowadays, medical staff perform palpation not only on
the patient’s wrist but also on the neck, foot, and otherbody parts, which are more effective in judging local
lesions. Therefore, this system should have wilder
applicability. Since the surrogate’s shape is irrelevant, it
can be improved to adapt more reproduction of
bio-signals.
Finally, the participants were dissatisfied with comfort
levels when wearing the HMD. This aspect must be
considered in real practice. Many participants haveshown VR sickness after using this VR telediagnosis
system. Moreover, the helmet is too heavy to be worn fora long time. Medical staff will tire easily, and therefore,
it is impractical in daily telediagnosis. Thus, it isnecessary to improve their VR experience when
diagnosing. The weight of the helmet can only be solved
when the hardware undergoes improvements. However,the software can be better developed. Due to limitationsof a surrogate robot, such as visual dislocation, thedoctor can only move in a small range. Using a robot
with more DoF may be more effective. In diagnosis,
reading and writing medical records is necessary. Adding
a function of the medical record will greatly improve thesystem. In order to have efficient communication withthe patients, it is proposed that the patient side’s
surrogate robot arm is better equipped to simulate thedoctor pressing the patient’s skin, helping the patient
more effectively realize the doctor’s instructions.

7 Conclusion
Aiming at the current situation of telemedicine, thispaper proposes a telediagnosis system to allow medical
staff to palpate a remote patient in a MR environment,
allowing the doctor to obtain a better observation andmore necessary essential information to determine
diseases. The system is based on telexistence and tactile
display with a 6 DoF surrogate robot and a new surrogate
unit. On the patient’s side, a 6 DoF robot and sensors areused to measure body temperature and pulse rate, which
are then reproduced on the doctor’s side using MR andtactile display devices. Through experiments andcomparisons with the traditional telediagnosis method, it
was concluded that this MR system had better
performance, and the effects of telexistence and tactile
display on telediagnosis reflect a positive attitude. The
feasibility of the system for improving the realism oftelediagnosis is therefore verified
Acknowledgements
This work was supported by JST ACCEL Grant
Number JPMJAC1404, Japan. We would like to thankProf. Hiroyuki Shinoda for his support of the project. We
would also like to thank Prof. Katsuya Iijima, Prof.
Hiroshi Oyama, Prof. Masahiro Akishita, and Assoc. Prof.
Sumito Ogawa and MD. Ayaka Fujii for their advices
from doctor's point of view.
Reference[1]

J. Moline : Virtual reality for health care: A survey; Stud.

[2]

J. Kim, B. Ahn, Y. Kim and J. Kim : Inclusion detection

Health Technol. Inform., 44, 3–34, (1997).
with haptic-palpation system for medical telediagnosis;
2009 Annual International Conference of the IEEE
EngineeringinMedicineandBiology

Society,

Minneapolis, MN, 4595-4598, (2009).
[3]

Murphy RL Jr, Bird KT : Telediagnosis: a newcommunity

healthresource.

Observations

on

the

feasibility of telediagnosis based on 1000 patienttransactions; Am J Public Health., 64(2), 113–119, (1974)[4]

Methil-Sudhakaran,

N.,

Shen,

Yantao,Mukherjee,

Ranjan , and Xi, Ning.: Development of a Medical
Telediagnostic System with Tactile Haptic Interfaces;
Proceedings, IEEE/ASME International Conference on
Advanced Intelligent Mechatronics, (2005).
[5]

Yantao Shen, Nandagopal S. Methil, Craig A. Pomeroy,
Ning Xi, Ranjan Mukherjee, Danyu Zhu, Zhiwei Cen,
James Yang, Matt W. Mutka, Carol A. Slomski, Keith N.

―282―

Fu, Kato, Inoue, Tachi : Development of a Telediagnosis System using Telexistence

Apelgren : Supermedia Interface for Internet-based
Telediagnostics of Breast Pathology; The International
Journal of Robotics Research, 26(11–12), 1235–1250,(2007).
[6]

G. H. Ballantyne , F. Moll,: The da Vinci telerobotic
surgical

system:The virtual

operative

field

and

telepresence surgery; Surg. Clin. North Am., 83(6)1293–1304, (2003).
[7]

T. R. Coles, N. W. John, D. Gould, and D. G. Caldwell :Tokyo. He received Ph.D. degree in Department ofComputational Intelligent and System Science from
Tokyo Institute of Technology 2016, and a JSPS research
fellow for young scientists since 2010. He had beenworking at TOSHIBA R&D since 2014 to 2016 aspermanent researcher. His research interests include
virtual reality, haptics, telexistence, physical simulation.
He received Miura Prize from the Japan Society of
Mechanical Engineering in 2010.

Integrating haptics with augmented reality in a femoral
palpation and needle insertion training simulation; IEEE
Trans. Haptics, 4 (3), 199–209, (2011).
[8] A. Garingo, P. Friedlich, L. Tesoriero, S. Patil, P. Jackson,
and I. Seri: The use of mobile robotic telemedicine
technology in the neonatal intensive care unit; J.
Perinatol., 32(1), 55–63, (2012).
[9]

SusumuTachi:

Telexistence

2nd

Edition;World

Scientific, (2015).
[10] Oculus : https://www.oculus.com[11] PulseSensor. https://pulsesensor.com/[12] R. R. Santriono Refki: Comparison of Face-to-Face andVideophone Communication Modalities for Delivering

Yasuyuki Inoue (Regular member)
2010, completed a doctoral courseof Graduate School of ElectronicandInformation
Engineering,
Toyohashi
UniversityofTechnology, Japan. 2015, Project
Researcher of Prof. Tachi’slaboratory
atInstituteofGerontology, University of Tokyo,
Japan, at the present day. The field of expertise are
cognitive psychology, human informatics and VR. Ph.
D. degree in Engineering.

Anxiety and Stress Psychoeducation to Deaf Individuals
in a University Counseling Center; 4(1), 75–84, (2019),[13] M. Chen, S. Member, Y. Jin, T. Goodall, X. Yu, and A. C.
Bovik: Study of 3D Virtual Reality Picture Quality; IEEE
Journal of Selected Topics in Signal Processing, 14(1),89-102, (2020).

(Received: March 31, 2020)[Author Biographies]
Junkai Fu (Student member)
2015, received the B.E in
ElectronicandInformation
EngineeringfromBeihang
University, China. 2020, he
received the MSc in Departmentof Complexity Science andEngineering from the Universityof Tokyo, Japan.
Fumihiro Kato (Regular member)
He is currently Project Researcherof The University of Tokyo, Japan,
and is belonging to JST ACCEL
Embodied Media Project at Tachi
Laboratory,
InstituteofGerontology, The University ofSusumu Tachi (Fellow)
He is Professor Emeritus of The
University of Tokyo, and theFounding President of the Virtual
Reality Society of Japan. He
received his B.E. and Ph.D. degrees
from The University of Tokyo, in1968 and 1973, respectively. He
joined the Faculty of Engineering of
The University of Tokyo in 1973, and in 1975, moved to
the Mechanical Engineering Laboratory, Ministry ofInternational Trade and Industry, Japan. From 1979 to
1980, Dr. Tachi was a Japanese Government Award
Senior Visiting Scientist at the Massachusetts Institute ofTechnology, USA. In 1989, he rejoined The Universityof Tokyo, and served as Professor at the Department ofInformation Physics and Computing till March 2009. Heserved also as Professor and Director of the International
Virtual Reality Center at Keio University, Japan from
April 2009 till March 2015.He initiated and foundedInternational Conference on Artificial Reality andTelexistence (ICAT) in 1991 and International-collegiate
Virtual Reality Contest (IVRC) in 1993. He received the2007 IEEE VR Career Award and served as General
Chair of IEEE Virtual Reality Conferences.

―283―