第 15 回テレイグジスタンス研究会(2020 年 2 月)

Study on Telexistence C:
A Telediagnosis Platform based on Telexistence:
Investigation of the Roles of Presence and Tactile
Information in Telemedicine
Junkai Fu1), Fumihiro Kato2) , Yasuyuki Inoue2), Susumu Tachi 2)1) 東京大学 新領域創成科学研究科 （〒277-8561 千葉県柏市柏の葉 5-1- 5, junkai@tachilab.org ）2) 東京大学 高齢社会総合研究機構 （〒113-0032 東京都文京区本郷 7-3-1, { fumihiro.kato, y-inoue,tachi}@tachilab.org）

Abstract： This paper proposes a telediagnosis system that allows medical staff to examine remotepatients through telexistence robot system with tactile sensor/display. The system consists of threecomponents, an audio-visual telexistence system for telecommunication, a skin-like tactile display
equipped with thermal and heartbeat display devices, and body temperature and heartbeat
measurement equipment. In comparison with conventional telephone and videophone, this VR
system is expected to allow the medical staff to examine the patient more carefully as if he or she isobserving the patient face to face．To test this, 10 medical doctors and 2 nurses evaluated this system.
According to the results, the influence and effect of telexistence and tactile display on telediagnosis
are discussed. Also, the feasibility of the system for improving the reality of telediagnosis is verified
and discussed.
Keywords：Teledignosis, Telexistence, Haptics, Mixed reality

1. Introduction

With the development of VR/AR/MR, more and more

In recent years many countries have come into an aging society.

applications and research in telemedicine emerged. Telepresence

The cost for the elder to have a diagnosis always spends a lot nosurgery has already commercialized for many years. Also, due tomatter the time or money on the trip. Hence the health care

the high accuracy, this kind of surgery system such as the datargeted at the aged is an urgent matter. Telemedicine is suitable

Vinci telerobotic surgical system, has been increasingly used[2].

to do this. Traditional telemedicine concentrates on using

Moreover, palpation based on VR simulation has also been

information technology such as audio, video to provide medical

proposed. Timothy et al. use artificial force to simulate thediagnosis and other treatment from professional doctor to thefeeling of human skin and apply the system into the palpation

patient. It can reduce the cost of medical treatment and the risk

and insertion training[3]. To achieve telemedicine, a robot ownof cross-infection for infectious diseases. However, thepartial function as a human is also a viable way. Garingo et al.

conventional telemedicine system through IT systems such as

apply a mobile robot into neonatal intensive care to help people

videophone is not an efficient way, which limits the transmitted

from a distance[4].

information for a doctor to diagnosis. By using telexistence, thisThis research targets the daily diagnosis, which is before theresearch is devoted to enhancing the communication between

confirmation of the disease and surgery. The frequency may be

doctor and patient and narrow the gap like in the same room. By

very high and last for a long time. To makes the palpation

using a surrogate that can simulate the patient’s pulse andbetween people in different places possible, we propose atemperature, the doctor's side will feel like palpating directly.

telediagnosis system with a tactile display. We hope this tactile

2. Telemedicine System

display can be used in diagnosing a difficult illness, which cannot

Combining virtual reality (VR) with health care has been

be determined without touch and some illness transferred into

proposed for decades of years. Early surveys reported thatdata in number is obscure to understand, but once being touched,research using virtual reality to do surgery, medical educationthese symptoms can be easily judged. In this way, the system

can go back to the 1990s[1].

shortens the distance between the patient’s home and medicalinstitution. There is no need for the aged to stay at the hospital

Sensors on the glove measure the pulse and temperature of theevery day. With this, he/she can accomplish a diagnosis just atpatient. The sensor data are stored and processed in a micro

home.

control unit (MCU), and then the bio-signal information is

For that purpose, we choose a surrogate robot and use atransmitted to the doctor-side.

surrogate arm to achieve the system. The system must follow
these principles[5]. It must be real-time to ensure communication

Doctor-side: The doctor wears a Head-mounted display (HMD)

quality. A doctor can see, hear, and touch the patient to judge theconnected to the surrogate robot in the patient-side, which

condition, and able to see his body in the virtual space to ensurereconstruct the surrounding of the patient’s room. Also, the HMD

immersion and make interaction. We choose pulse andis equipped with a stereo camera for achieving video-seethrough;temperature to measure from the patient and reproduce to thethat is, the doctor can recognize his/her own hands overlapping

doctor precisely and realistically because it is comfortable forin the image of the patient-side. The doctor-side’s image except

reproduce but always essential in all the diagnosis. If this idea

for the hands is eliminated by using chroma key screens in aworks, the more bio-signal can be added to the system to enlarge

Unity project. This method realizes the mixed reality.

its usability.

Furthermore, a patient surrogate arm equipped with many

The human’s regular pulse is between 60~120 bpm. The

devices is placed in front of the doctor to reproduce the bio-signal

disease may cause the heart rate beyond the range. For example,

information of the patient. Owing to the system, it is expectedparoxysmal supraventricular tachycardia (PSVT) is the mostthat the doctor can feel as if he/she were diagnosing the patients

common symptom of tachycardia. The pulse can reach 140~180

face to face.

bpm. Besides, the temperature and pulse are in positive
correlation. Usually, pulse increases by 10 bpm when thetemperature rises by 0.5℃. Therefore, in response to possible
occasions, we need to set the reproducing pulse range between
40~200 bpm on the surrogate arm to simulate arrhythmia.

3. System Design & Implementation
The system consists of two parts. One is the patient-side, theother is the doctor-side.
Patient-side: The patient is supposed to wear a glove withsensors and sit in front of a 6 DOF robot. With the sensors
embedded in the robot and glove, the audio-video information

Figure 1: System schematic diagram

and bio-signal of the patient are recorded and transmitted to thedoctor-side. This part is made up of two components: (1) A

3.1 Audio-video Reproduction

doctor-surrogate robot is supposed to perceive the surrounding’s

3.1.1 Surrogate robot at patient’side

information including the patient’s appearance, voice, andThe robot (RT-Telbee) used in patient-side equipped with areplicate the voice and gestures recorded from the doctor-side. (2)stereo camera, a binaural microphone, and a speaker, making

Figure 2 : System data diagram

itself able to communicate like a human. It also has a 6-DOF

a heater and flexible tube to generate heat and pulse.

mechanical system, which allows the doctor-side connects it withan HMD (Oculus Rift CV2, Facebook Technologies, LLC)[6] tocontrol its movement．

Figure 6: Mechanism of the reproduction system
But we found the surrogate in what kind of shape does not
Figure 3: Construction of 6 DOF robot RT-Telbee

matter, so we developed a smaller one with a cooling module toreplace the surrogate arm like figure 7. A heater cooperates with

3.1.2

Mixed reality at doctor-side

To increase the realistic feeling and maintain the presence ofthe doctor-side, we attached a stereo camera on the HMD anda Peltier element to keep the temperature the same with thepatient. A mechanical crank system is responsible for applying
pressure to the liquid in the tube to imitate the pulse.

extract the doctor’s hand from the surrounds in the local byusing chroma key composing. The doctor is able to see his hand
while wearing the HMD, and the hand is overlapped with thebackground transmitted from the patient-side. Because theHMD is tracking by sensors and connected with the surrogate
robot at the patient-side, while he moves, the vision in theHMD will also change like figure 4.
Figure 7: New surrogate with a cooling module

Figure 4: MR environment in the HMD
3.2 Bio-signal reproduction
3.2.1 Detecting glove
Patient-side uses a glove equipped with a pulse sensor(PulseSensor, World Famous Electronics llc.)[7], A negative
temperature coefficient (NTC) thermistor (56A1002-C3, Alpha

Figure 8: System in two sides (upper) and the view from theSensors, Inc.) and an MCU (ESP32-WROOM-32D, Espressif

doctor/patient (below)

Systems Pte. Ltd) to detect the pulse and temperature.

4. Experiment
To test this telediagnosis system is useful or not. Morespecifically, compared to the traditional telediagnosis system,
this system can provide a better reality or not. Can the tactile
display bring more clues and do benefits to the telediagnosis. We
did an experiment with 12 medical experts. The participants were
Figure 5: Bio-signal detecting glove

required to make a complete diagnosis like a real medical
examination by using this VR system and the traditional

3.2.2 Surrogate for reproduction
Doctor-side receives the bio-signal and reproduces the pulseand temperature on a surrogate arm like figure 6. Under theartificial skin(a mixture of silicone and Al2O3 powder), there arevideophone separately. The procedure is almost the same except
for that in the videophone method; there is no palpation on atactile display.
After the experience, they are asked to answer twoquestionnaires to evaluate the system. At last, we had an

the system defects, such as image quality, and the limitation tointerview to listen to their comments.

determine disease just by pulse and temperature.

5. Results

6. Conclusion

After using each system (VR and videophone), there four

In our research, we proposed a telediagnosis system based on
Telexistence. This system contains a surrogate robot and a tactile

questions as follows:
Q1. I felt as if I were actually in the room where the patient is in.

display. By using this system, the medical staff can palpate theQ2. I felt as if the patient were actually in front of me.

patient like face to face. Then, we did an experiment to compare

Q3. I felt as if the medical examination was actually performed.

this system with the traditional telediagnosis system. ThisQ4. I felt tense while performing a medical examination.

telediagnosis system performed a better ability in diagnosis,

The score is shown as follows:

according to the medical experts. But there are also many defects
that need to be improved. In the future, the most urgent need is
to improve image quality. Secondly, the tactile display should
provide more information such as blood pressure or other biosignal to enhance its usability. And versatility should also beconsidered, other parts like neck or foot also can be developed in
the tactile display. Finally, due to the high reality, this system alsocan be used in medical training to improve the practical skills ofmedical staff.

Acknowledge

Figure 9 : Evaluation of User Experience
From figure 9, we can see that the scores for the VR systemwere significantly better than the videophone system except forThis project is supported by JST ACCEL Embodied Media
Project (JPMJAC1404).

Reference

the Q2 “patient’s presence.” We had an interview after theexperiment and know that they are particularly dissatisfied with

[1]J. Moline, “Virtual reality for health care: A survey,”

the image quality. Compared to the video in skype, the image

Stud. Health Technol. Inform., vol. 44, pp. 3–34, 1997,

quality in HMD is very poor. The patient’s image isn’t clear

doi: 10.3233/978-1-60750-888-5-3.

enough, so the patient’s presence will also be damaged. The Q3[2]

G. H. Ballantyne and F. Moll, “The da Vinci

shows a positive overall judgment. It means that, by using VR

telerobotic surgical system: The virtual operative fieldsystem, the participants could have a more realistic feel for theand telepresence surgery,” Surg. Clin. North Am., vol.

examination. Finally. for Q4, the VR telediagnosis brings more

83, no. 6, pp. 1293–1304, 2003, doi: 10.1016/S0039-tension. As our thought, it means the experts treat it more

6109(03)00164-6.

seriously and more like the actual diagnosis.

[3]

the impression of the VR diagnosis system.

in a femoral palpation and needle insertion training
simulation,” IEEE Trans. Haptics, vol. 4, no. 3, pp.

Table 1: Answer of questionaire2
Q1: Have you ever experienced anothertelemedicine system?
Q2.1: If Q1 is “YES,” do you think the VR system
is more real?
Q2.2: If Q1 is “NO,” do you think the VR system
is real?
Q3: Do you think VR experience is useful for theexamination?
Q4: Do you think the tactile display is useful forthe examination?
Q5: Do you want to use a VR telediagnosis
system?

T. R. Coles, N. W. John, D. Gould, and D. G.
Caldwell, “Integrating haptics with augmented reality

After all the experiments, they were required to answer about

YES
1NO
1110

Jackson, and I. Seri, “The use of mobile robotic

101

telemedicine technology in the neonatal intensive care

120101

111

199–209, 2011, doi: 10.1109/TOH.2011.32.
[4]

A. Garingo, P. Friedlich, L. Tesoriero, S. Patil, P.

unit,” J. Perinatol., vol. 32, no. 1, pp. 55–63, 2012,
doi: 10.1038/jp.2011.72.
[5]

Scientific, 2015.
[6]

The results are shown in table1. Most of them agreed that thisVR system is presented with a realistic experience like an actual
examination. Furthermore, all of them agreed that the VR systemand the tactile display is useful while performing a clinicalexamination. But after the interview, they also said a lot about

Susumu Tachi, Telexistence 2nd Edition. World“Oculus.” [Online]. Available:
https://www.oculus.com.

[7]“PulseSensor.” [Online]. Available:
https://pulsesensor.com/.