Development of a Telepresence System for Sharing 3D Physical Objects
Sharing real-time point clouds between multi users by using Mixed Reality○ Daichi Ishikawa *1, Tomohiro Fukuda *2 and Nobuyoshi Yabuki *3*1 Graduate Student, Div. of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka Universit y*2 Assoc. Prof., Div. of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka University, Ph.D.
*3 Prof., Div. of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka University, Ph.D .
Keywords: Telepresence; sharing 3D physical objects; Mixed Reality (MR); real-time point clouds; interactive.

Figure 1. The proposed system configuration
The research presented in this paper aims to develop a1. Introduction
Consensus building between stakeholders is necessary totelepresence system adapted into multi users communication: aenhance the quality of the decision-making. Telepresence allows

sender and some receivers. This is accomplished by sharing theone to feel as though other users in remote areas were presently

coordinates of 3D virtual objects between MR-HMDs theat the same location1).

receivers wearing.

Virtual Reality (VR) and Mixed Reality (MR) can provide
immersive experiences with people wearing VR/MR-Head

2. System configuration

Mounted Display (HMD). Kantonen et al. proposed an MR

Figure 1 shows the proposed system configuration. At first, an

telepresence system, Augmented Collaboration in Mixed

RGB-D camera simultaneously captures the RGB images andEnvironment (ACME) that can share 3D virtual objects made by

depth data in real-time to generate real-time point clouds. After

Building Information Modeling (BIM) between stakeholders2).

point cloud processing for point cloud segmentation, real-time

However, this system requires 3D virtual models to be made inpoint clouds are classified into individual clusters by theadvance, making it difficult to share 3D physical objects whose

Euclidean Cluster Extraction method4).

shapes continuously change in real-time.

Point cloud data with XYZ coordinates, RGB color values,

Real-time point clouds enable modeling of 3D physicaland cluster information of real-time point clouds are transmitted

objects in real-time by using an RGB-D camera. In our previousto MR-HMD devices in a remote area connected via the sameresearch, Ishikawa et al. proposed an immersive telepresence

Local Area Network (LAN). Receivers wearing MR-HMD issystem to improve operability of real-time point clouds in orderable to observe, in the real world, 3D virtual objects generated by

to share and manipulate the 3D physical objects more easily inpoint cloud data through MR in real-time. Connecting each MR-

remoteareas3).

Thissystemonly

allowsone-to-one

communication: a sender and a receiver.

日本建築学会情報システム技術委員会
第 42 回情報･システム･利用･技術シンポジウム論文集， 254-255 ， 2019 年 12 月，東京
Proceedings of the 42nd Symposium on Computer Technology of Information ，Systems and Applications ， AIJ ， 254-255 ， Dec. ， 2019 ， Tokyo

HMD with a server PC, the coordinates of 3D virtual objects
among MR-HMDs are shared.

－ 254 －(a) The receiver A’s view
through HoloLens(b) The state of receiver Aand the receiver B(c) The receiver B’s view
through HoloLens

Figure 2. Sharing the coordinates of 3D virtual objects
3. Experiment and Result

4. Limitations

In this experiment, two receivers in the same room realized for

In the proposed system, some limitations still exist.

sharing the coordinate of 3D virtual objects generated using realtime point cloud data acquired from a remote area.
In Figure 6a and 6c, while the receiver A touched 3D virtual

The first is the usage environment. In order to use the system,
an RGB camera and MR-HMDs are necessary under the same
LAN environment.

objects, the receiver B could observe this situation. From thisThe second is the light environment. An RGB-D camera isresult, we confirmed that the coordinates of 3D virtual objects

weak to light reflection due to using laser to capture depth data.

were synchronized between the receiver A and the receiver B

To run this system better, we prepare a room not too bright.

Figure 3. Floor plan showing two sites

References
1） Buxton, W. (1992). Telepresence: Integrating shared task andperson spaces, In Proceedings of graphics interface, Vol. 92, No.
1992, pp. 123-129
2） Kantonen, T., Woodward, C., and Katz, N. (2010) Mixed reality invirtual world teleconferencing, In Virtual Reality Conference (VR),2010 IEEE, pp. 179-182..
3） Ishikawa, D., Fukuda, T., and Yabuki, N. (2019). Development ofa Telepresence System for Sharing 3D Physical Objects Using
Real-time Point Clouds. AIJ HOKURIKU 2019, Summaries ofTechnical Papers of Annual Meeting, Information Systems
Technology, 13-144） Rusu, R. B. (2010). Semantic 3d object maps for everyday
manipulation in human living environments. KI-KünstlicheIntelligenz, 24(4), 345-348.

(a) Around sender(b) The view of RealSense
D435i
Figure 3. The state of the sender

インタラクティブ発表 Z103－ 255 －