3B-05
This article is a technical report without peer review, and its polished and/or extended version may be published elsewhere

第 24 回日本バーチャルリアリティ学会大会論文集（2019 年 9 月）

Study on Telexistence XCV:A Telediagnosis Platform Based on Mixed Reality andBio-signal Display Device
Junkai Fu1)，Yasuyuki Inoue2)，Fumihiro Kato2) ，Susumu Tachi 2)1) 東京大学 新領域創成科学研究科 （〒277-8561 千葉県柏市柏の葉 5-1- 5, junkai@tachilab.org ）2) 東京大学 高齢社会総合研究機構 （〒113-0032 東京都文京区本郷 7-3-1, {y-inoue,fumihiro.kato,tachi}@tachilab.org）

Abstract： We propose a telediagnosis system that allows medical staff to interact with a remotepatient through telexistence robot system. The system consists of an audio-visual telexistencesystem (TX-toolkit), a surrogate arm equipped with thermal and pressure displays, and bodytemperature and heartbeat measurement equipment. By using this system, the medical staff canexamine the patient as if he or she is observing the patient face to face．
Keywords：Teledignosis, Telexistence, Haptics, Mixed reality

1. Introduction

Moreover, palpation based on VR simulation has also been

Recent years many countries have come into an aging society.

proposed. Timothy et al. use artificial force to simulate theThe aged have to live by themselves because their children havefeeling of human skin and apply the system into the palpationno time to take care of them due to their hard work. Hence theand insertion training[3]. To achieve telemedicine, a robot own

health care targeted at the aged is an urgent matter.

partial function as a human is also a viable way. Garingo et al.

Telemedicine is suitable to do this. Traditional telemedicineapply a mobile robot into neonatal intensive care to help people

concentrates on using information technology such as audio,from a distance[4].

video to provide medical diagnosis and other treatment from

This research targets at the daily diagnosis, which is before

professional doctor to the patient. However, medical treatment

the confirmation of the disease and surgery. The frequency may

through a screen is not an efficient way, which limits thebe very high and last for a long time. We propose atransmitted information. The doctor cannot teach the patient

telemedicine system with palpation simulation, which seems

like face to face. The patient may misunderstand what thelike the study[3]. Also implement a function of telediagnosis,

doctors mean and feel more nervous. By using mixed reality,

which makes the palpation between people in different places

this research is devoted to enhancing the communication

possible. This system shortens the distance between thebetween doctor and patient and narrow the gap like in the same

patient’s home and medical institution. There is no need for theroom. By using a surrogate arm which can simulate theaged to stay at the hospital every day. With this, he canpatient’s pulse and temperature, the doctor side will feel likeaccomplish a daily diagnosis just at home.

palpating truly.

For that purpose, we choose a surrogate robot and use a

2. Telemedicine Systemsurrogate arm to achieve the system. The system must follow

Combining virtual reality (VR) with health care has beenthese principles. It must be real-time to ensure communication

proposed for decades of years. Early surveys reported thatquality. A doctor can see, hear, and feel the patient to judge theresearch using virtual reality to do surgery, medical education

condition, and able to see his body in the virtual space to ensure

can go back to 1990s[1].

immersion and make interaction. The vital medical information

With the development of VR/AR/MR, more and morein daily diagnosis such as temperature, pulse need to beapplication and research in telemedicine emerged. Telepresence

gathered from the patient and reproduced to the doctor precisely

surgery has already commercialized for many years. Also, dueand realistically.

to the high accuracy, this kind of surgery system such as da

Human’s regular pulse is between 60~120 bpm. The disease

Vinci telerobotic surgical system has been increasingly used[2].

may cause the heart rate beyond the range. For example,Ⓒ 2019 日本バーチャルリアリティ学会-3B-05-

3B-05paroxysmal supraventricular tachycardia (PSVT) is the mostgot from the patient-side, which makes the artificial arm

common symptom of tachycardia. The pulse can reach 140~180

more like the patient’s real arm. The doctor would feelbpm. Besides, the temperature and pulse are in positive
correlation. Usually, pulse increases by 10 bpm when theas if he is diagnosing the patients face to face.
3.1 Transmission of Behavior Information

temperature rises by 0.5℃. Therefore, in response to possible
occasions, we need to set the reproducing pulse range between
40~200 bpm on the surrogate arm to simulate arrhythmia.

Figure 1: System overview: Patient-side and doctor-side
Figure 3: Mechanical construction of TX-toolkit

3. System Design & Implementation
The system consists of two parts like Figure 1. One is

3.1.1

patient-side, the other is the doctor-side.
⚫

⚫

Telexistence Toolkit

Telexistence is a concept which represents the technology

Patient-side: This part is made up of two components. A

reappearing one’s existence from a distance[6]. To achieve

surrogate robot is supposed to collect the surrounding’s

telexistence, a robot which has the same enginery as a human isinformation and copy the posture and voice got fromnecessary. However, popularizing a traditional robot with highdoctor-side. Sensors read the bio-signal measured fromcostthe patient. An MCU saves the signal and then send the(TX-toolkit) is designed to achieve the telecommunication withdata to the doctor-side.

low cost but high telepresence[7]. As Figure 3, TX-toolkit

Doctor-side: The Head-mounted display (HMD) will

equipped with a stereo camera, a binaural microphone, and areconstruct the scene of the patient’s room. The HMD

speaker, making itself able to communicate like a human. It has

mounts an Ovrvision[5], which allows the doctor toa 3-DOF mechanical system and network function, which allow

recognize his arm by using chroma key. A surrogate arm

the user connects it with an HMD (Oculus Rift)[8] to control its

equipped with many devices reproduces the bio-signal

movement．

isEconomicallyFigure 2 : System data diagramⒸ 2019 日本バーチャルリアリティ学会-3B-05-

unbearable.

Telexistence-toolkit

3B-05thermistor calculates the temperature at the same time. An
ESP32 (ESP32-WROOM-32D, Espressif Systems Pte. Ltd.)saves both types of data. ESP32 is an MCU with integrated
Wi-Fi function. It gathers the bio-signal then sends it to thedoctor-side.

Figure 4: Synchronization between TX-toolkit and user
3.1.2Telecommunication

The telecommunication system works as Figure 2 upper halfshows. With TX-toolkit, the image, sound at patient-side can betransferred to the doctor-side through the video, audio stream
Figure 5: Bio-signal importand playback on the HMD. By this way, the doctor will feel likesitting in front of the patient and move his sight as he wishes.
As Figure 4 the robot has a 3-DOF neck, which allows it to3.2.2

Bio-signal Reproduction

copy the movement same with doctor’s head. The movement
information is got from the HMD. Moreover, the patient canrealize the doctor’s eyesight and hear his voice through therobot’s rotation and its speaker. Hence Patient-side will know
about what the doctor is looking at and follow his guidance. By
that, it provides an efficient way for each other to communicate
freely and realistically.
Besides, in order to see his own arm to create a better
realistic at doctor-side, an Ovrvision is equipped in front of theHMD. The doctor can recognize his own side’s surrounding
through the stereo camera, yet the view of his own side willFigure 6: Pulse mechanism with Esp32

cover the view got from the TX-toolkit. Therefore, to eliminate
the unwanted pixel, we use chroma key with doctor-side’s

Doctor-side’s PC receives the message and shows thebackground all in blue, the doctor can see nothing except for hisown arm presenting in the scene of the patient’s room.

heartbeat and temperature on the HMD’s screen. Next, the3.2 Transmission of Bio-signal

message will be sent to another ESP32. This ESP32 connects toTransmission of bio-information can be divided into twoan actuator unit and a temperature control unit.

parts. One is gathering information from the patient-side. The

Actuator unit is responsible for rebuilding the pulse. The

other is reproducing the bio-signal on the surrogate arm throughmechanism is shown in Figure 6. A crank mechanism issome mechanism. The surrogate arm is a silicone arm with

attached to a DC motor (380K75, Tamiya Inc.). The rotationsome rubber tube under the skin, which is usually used inrate keeps the same as the patient’s heartbeat. When the motor

medical practice to train injection skill. The tube distributes likeruns, the crank will drive a syringe back and forth. The syringe

an artery. Fill the tube with water and add pressure, then theconnects to surrogate arm’s rubber tube, within it, water will

system will be like blood flowing in the arteries. We simulate

take tunes being squeezed and relaxed, which makes thethe flow of blood to produce a pulse.

process same as blood being pumped by the heart. The

3.2.1

temperature control unit contains three components. A

Bio-signal Gathering

We choose the pulse and temperature as the primary function tothermistor is used to monitor the temperature. A pair of film

reproduce. A sensor unit is attached to the patient’s finger asheater (P-13611) and Peltier element (TES1-12705) keep theshown in Figure 5, which is composed of Pulsesensor[9] and atemperature the same as the data received from patient-side.

thermistor

4. Results(56A1002-C3,AlphaTechnics.).

Pulsesensor

measures the heartbeat by photoplethysmogram (PPG) whileⒸ 2019 日本バーチャルリアリティ学会-3B-05-The system prototype was implemented with the following3B-05
specification.

only contains his own arm and what the robot sees at the⚫

Robot: TX-toolkit with a built-in stereo camera, binauralpatient-side. Therefore, the doctor just like sitting in the same

microphone, speaker, and 3-DOF mechanical system.

room and naturally put his own arm on the patient’s wrist to⚫

MCU: ESP32-WROOM-32Ddiagnose. The surrogate arm reproduces the bio-signal got from⚫

VR headset: Oculus Rift CV2 with Ovrvision

the patient-side. So, the doctor can feel the warmth and the⚫

Sensor: Pulsesensor and NTC thermistorpulse as he is touching the real arm.

⚫

Display: 380K75 motor, 4cm×4cm Peltier element, and5. Future Workfilm heater on a surrogate arm.

Usability testing is in the top priority. A series of verification
test among medical staff is being planned and conduct in thefuture to test whether the system is realistic or not.
Furthermore, more function of bio-signal test can be added tothe system such as ECG. With more function, the practicality
will increase considerably.

6. Conclusion
This research proposes a telediagnosis system that allows
medical staff to interact with a remote patient throughtelexistence robot. Also, a surrogate arm equipped with athermal device and actuator is used to present the patient’s
Figure 7: Linear relationship between pulse and voltage

temperature and heartbeat. This system provides a more
realistic way for remote diagnosis.

The relationship between the voltage applied to the DC

Referencemotor and the pulse is shown in Figure 7. Voltage (V) and pulse
rate (R) are close to a proportional relationship as follow.

[1]

J. Moline, “Virtual reality for health care: A survey,”
Stud. Health Technol. Inform., vol. 44, pp. 3–34, 1997.

[2]

G. H. Ballantyne and F. Moll, “The da Vincitelerobotic surgical system: The virtual operative field

Required pulse range corresponds to a voltage range ofand telepresence surgery,” Surg. Clin. North Am., vol.

1.5~6V. By controlling the applied voltage, the pulse rate can be83, no. 6, pp. 1293–1304, 2003.

displayed correctly.
[3]

T. R. Coles, N. W. John, D. Gould, and D. G. Caldwell,“Integrating haptics with augmented reality in afemoral palpation and needle insertion trainingsimulation,” IEEE Trans. Haptics, vol. 4, no. 3, pp.
199–209, 2011.

[4]

A. Garingo, P. Friedlich, L. Tesoriero, S. Patil, P.
Jackson, and I. Seri, “The use of mobile robotic
telemedicine technology in the neonatal intensive careunit,” J. Perinatol., vol. 32, no. 1, pp. 55–63, 2012.

[5]“Ovrvision.” [Online]. Available:
http://ovrvision.com/.

[6]
Figure 8: System in different angles and doctor-side’s view:
Patient-side’s image mixed with the doctor’s hand

Susumu Tachi, Telexistence 2nd Edition. World
Scientific, 2015.

[7]

Y. Inoue, F. Kato, and S. Tachi, “テレイグジスタン
スロボットにおける身体表現機能の VR 拡張,”This system achieves a simple application scenario fortelediagnosis. The patient sits in front of the TX-toolkit and put

vol. 24, no. 1, pp. 137–140, 2019.
[8]

his arm on the desk, following the doctor’s guidance. At thesame time, the doctor faces a blue background box. In the box,

https://www.oculus.com.
[9]

the surrogate arm is also in blue. With the Ovrvision, all thebackground and the surrogate arm will become transparent byusing chroma key. Like Figure 8, the image shown on the HMDⒸ 2019 日本バーチャルリアリティ学会“Oculus.” [Online]. Available:-3B-05-“PulseSensor.” [Online]. Available:
https://pulsesensor.com/.