Development of a Telepresence System for Sharing 3D Physical Objects
Improving visibility of virtual objects by increasing the density of real-time point clouds○ Daichi Ishikawa *1, Tomohiro Fukuda *2 and Nobuyoshi Yabuki *3*1 Graduate Student, Div. of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka Universit y*2 Assoc. Prof., Div. of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka University, Ph.D.
*3 Prof., Div. of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka University, Ph.D .
Keywords: Telepresence; sharing 3D physical objects; Mixed Reality (MR); real-time point clouds; visualization.

and manipulate the 3D physical objects more easily in remote

1. IntroductionIn

the architectural field,

consensus building between

stakeholders such as architects and clients is necessary toareas5). However, the visual quality of the 3D virtual objects
using real-time point clouds were insuffficient.

enhance the quality of the decision-making. Communication

The research presented in this paper aims to improve thesystems play an important role in providing a platform on whichaesthetic of 3D virtual objects generated by using real-time point

ideas can be shared between several people. Although conducting

clouds. Although increasing the number of point clouds leads toremote meetings such as video conferencing has manya better quality in visuals, it also increases the processing speed

advantages to time, cost, and energy used for traveling when

due to the increasing data size. Thus, we propose a new filtering

compared with face-to-face meetings, there is a lack of presence

method for point clouds; point cloud clipping which has abilityfelt when communicating with other people. Telepresence allows

to improve the point cloud processing speed. The proposed

one to feel as though other users in remote areas were presently

system will then be compared to that of the previous systemat the same location, otherwise known as pseudo-face-to-facebased on whether aesthetics and point cloud processing speedcommunication1). Telepresence has the ability to combine face-were enhanced.

to-face meeting with remote meeting.
Virtual Reality (VR) and Mixed Reality (MR) can provide

2. Proposed Method

immersive experiences with people wearing VR/MR-Head

Our proposed system is an immersive telepresence system that

Mounted Display (HMD). Kantonen et al. proposed an MR

allows a receiver through an MR-HMD to observe andtelepresence system, Augmented Collaboration in Mixed

manipulate 3D virtual objects generated using real-time point

Environment (ACME) that can share 3D virtual objects made incloud data acquired from remote areas (Figure 1).

Building Information Modeling (BIM) software between
stakeholders2). However, this system requires 3D virtual models
to be made in advance, making it difficult to share 3D physical2.1. GENERATING REAL-TIME POINT CLOUDS
At first, an RGB-D camera simultaneously captures the RGB

objects whose shapes continuously change in real-time.

images and depth data (X, Y and Z direction) in real-time. Using

Point clouds are a cluster of points with 3D coordinates andpoint cloud clipping at this point ensures that the RGB-D camera

RGB color values, etc. captured by a 3D laser scanner. Point

does not acquire depth data (excluding point in the Z direction)

clouds enable the surveying of terrain and modeling of 3Dby excluding points which exist further than the distance set byphysical objects to be done easily and precisely. An RGB-D

the user. Using the point cloud clipping method before othercamera can acquire point clouds in real-time (hereinafter referredfiltering processes will improve the processing speed.

to as “real-time point clouds”). Kowalski et al. proposed

To generate real-time point clouds, the coordinates of RGB

LiveScan3D which automatically reconstructs 3D objects using

images correspond to that of the depth data coordinates because

real-time point clouds3). Fukuda et al. developed a telepresence

these coordinates differ when acquired by the RGB-D camera.

system using MR and real-time point clouds which allowed users
to share 3D physical objects4). This system focused only on

2.2. POINT CLOUD PROCESSING

displaying real-time point clouds. In our previous research,

To enable a user wearing MR-HMD the ability to manipulate 3D

Ishikawa et al. proposed an immersive telepresence system toobjects individually, we need to classify real-time point clouds

improve operability of real-time point clouds in order to shareinto clusters using point cloud segmentation. In the proposed

日本建築学会情報システム技術委員会
第 42 回情報･システム･利用･技術シンポジウム論文集， 216-219 ， 2019 年 12 月，東京
Proceedings of the 42nd Symposium on Computer Technology of Information ，Systems and Applications ， AIJ ， 216-219 ， Dec. ， 2019 ， Tokyo－ 216 －

Figure 1. The proposed system architecture
system, point cloud segmentation is done using the Euclidean

clouds in 3D and can then manipulate them using a pinch hand

Cluster Extraction6) method which classifies point clouds based

gesture.

on the distances between each point. Before point cloudsegmentation, some point cloud processing is required as

3. Experiment and Results

follows:

3.1. EXPERIMENTAL CONDITION

 Filtering: removing unnecessary points from the point

The proposed system requires some devices as shown in Table 1.

cloud data depending on the threshold set by the user in X,

Although RealSense D435i enables capturing depth data in highY, and Z direction to accelerate point cloud processing

resolution (1280×720), we set the resolution as 640×360 in thespeed.

It is unnecessary to consider in the Z direction

proposed system to accelerate point cloud processing speed. We

because unnecessary points have already removed byalso set the threshold of point cloud clipping (Z direction) as 0—point cloud clipping.

1.5 m and that of filtering (X and Y direction) as -1.0—1.0 m.

 Extracting and removal planes: extraction and removal

Point cloud processing and transferring point cloud data areof some planes such as desk or floor surfaces using

conducted by the laptop PC. HoloLens receives point cloud data

Random Sample Consensus (RANSAC). This process isand displays virtual objects generated by real-time point clouds.

necessary for the Euclidean Cluster Extraction method toprevent point clouds from being recognized as the samecluster through some planes.
 Reducing noise: eliminating point clouds which exceed
the threshold set by the user to achieve better visuals of 3D
virtual objects represented as real-time point clouds.
2.3. TRANSFERRING POINT CLOUD DATA
Point cloud data with XYZ coordinates, RGB color values, andcluster information of real-time point clouds are transmitted to an
MR-HMD device in a remote area connected via the same Local
Area Network (LAN) by using a TCP socket.
The proposed system saves the processed data as a PLY file,which is a file format used for point cloud transmittance and thenplays the recorded video output continuously on a PC display.

Table 1. Required devices of the proposed system
Name

Use

FeatureRealSense
D435i

RGB-D
cameraDepth Technology: Active IR Stereo
Minimum Depth Distance: 0.105 m

Let’s note
MX4

Laptop PC

RAM: 8.0 GB
CPU: Intel® Core™ i5-5300U
GPU: Intel® HD Graphics 5500

HoloLens

MR-HMD

Wi-Fi 802.11ac

3.2. POINT CLOUD CLIPPING METHOD
In the proposed system, we modified the filtering process byadding a point cloud clipping method in order to accelerate thepoint cloud processing speed. To evaluate the feasibility of theproposed system, the processing speed of the initially proposedsystem is compared with the improved system. We measured
relations between the number of output points and the processing

2.4. TELEPRESENCE ON MR-HMD
A receiver wearing MR-HMD is able to observe, in the real world,
3D virtual objects generated by point cloud data through mixed
reality in real-time. The user is able to observe real-time point

報告 H26speed three times when generating real-time point clouds basedon two static 3D physical models on a desk while changing theposition of them (Figure 2).

－ 217 －

(a)
(b)
(c)
Figure 2. Input RGB image (top), clipped input points (bottom left), unclipped input points (bottom right), respectively
Figure 6 shows screenshots of the view of the receiver
captured by the HoloLens. The receiver in site B was able toobserve the sender and the 3D building model in remote area site
B in 3D using real-time point clouds.
Next, comparison of the visibility of our proposed system ismade with that of the previous research5) (Figure 7). Visuals in
the improved system in Figure 6 is compared to that of visuals
created in the previous research in Figure 7. It is evident that theFigure 3. The relation between the number of output points andpoint cloud processing speed
In Figure 3, the green dots indicate the result of the relation
between the number of output points and the processing point
cloud speed when not using the point cloud clipping method,while the blue dots show the improved system using the clipping
method. Regarding the shape of dots, circles, triangles, andsquares indicate in case (a), (b), and (c), respectively. In all cases,
necessary objects are represented. Consequently, using the point
cloud clipping method was successful in reducting point cloud
processing speed.
3.3. THE VISIBILITY OF 3D VIRTUAL OBJECTSThis section evaluates whether the aesthetics of the 3D virtualobjects is enhanced by using the proposed system. As shown in
Figure 4, an experiment is conducted via Local Area Network(LAN) on two sites separated by a wall. In site A, real-time point
clouds that represent a sender and a static physical 3D building
model are generated by acquiring RGB images and depth datausing RealSense D435i (Figure 5a). In site B, a receiver wearing
HoloLens receives the processed point cloud data transferredfrom site A and observes the state of the sender and 3D building
model from site A through real-time point clouds in mixed reality

visibility of objects represented by real-time point clouds wasimproved by increasing the number of output point clouds.
This experiment was done using a laptop PC for point cloud
processing. If a PC with better specifications is used instead ofthe laptop PC, the point cloud processing speed in our proposedsystem is almost same as that of previous research. Priority ofdesired outcome should be made regarding aesthetics of virtualobjects, faster the processing speed or mobility depending on thesituation when using this system.
4. Discussion
As described in sections 3.2, we could improve the visibility of
3D virtual object by increasing the density of real-time point
clouds in the proposed system

Improving the visibility leads tobetter understandings and smooth consensus building between
stakeholders. In addition, we confirmed the point cloud clipping
method could reduce the point cloud processing speed.
However, our proposed system still has some challenges.
The first is the lack of bidirectional connection. Currently
only the sender is able to transmit point cloud data and thereceiver can only view and manipulate the objects. A desired
outcome for future works is the sender’s ability to view andmanipulate point cloud data as well; making the system(Figure 5b). The receiver is also able to manipulate 3D virtualbidirectional.
The second is the limitation of environment. In order to use

objects generated by real-time point clouds using a hand pinch

the system on a greater scale, adapting this system into thegesture.

報告 H26

internet environment is essential.

－ 218 －Figure 4. Floor plan showing two sites(a) Sender (site A)(b) Receiver (site B)
Figure 5. System users

Figure 6. The receiver’s view through HoloLens

Figure 7. Insufficient views of real-time point clouds inprevious research
5. Conclusion and Future works
We proposed an immersive telepresence system that has
improved aesthetics for sharing objects between users generatedusing real-time point clouds. Although latency is slower due tothe increased number of point clouds, we confirmed that
including the clipping method during the filtering process has theability to accelerate the point cloud processing speed.
Future works include implementation of bidirectional
connection between a sender and a receiver and adapting the
proposed system into the internet environment to realize more
effective telepresence.

報告 H26

References1） Buxton, W. (1992). Telepresence: Integrating shared task andperson spaces, In Proceedings of graphics interface, Vol. 92, No.
1992, pp. 123-129.
2） Kantonen, T., Woodward, C., and Katz, N. (2010) Mixed reality invirtual world teleconferencing, In Virtual Reality Conference (VR),2010 IEEE, pp. 179-182.
3） Kowalski, M., Naruniec, J., and Daniluk, M. (2015). Livescan3d:
A fast and inexpensive 3d data acquisition system for multiple
kinect v2 sensors. In 2015 International Conference on 3D Vision
pp. 318-325.
4） Fukuda, T., Zhu, Y., and Yabuki, N. (2018) Point Cloud Stream on
Spatial Mixed Reality Toward Telepresence in Architectural Field,
Proceedings of the 36th eCAADe Conference (Education andResearch in Computer Aided Architectural Design in Europe), Vol.
2, pp.737-744.
5） Ishikawa, D., Fukuda, T., and Yabuki, N. (2019). Development ofa Telepresence System for Sharing 3D Physical Objects Using
Real-time Point Clouds. AIJ HOKURIKU 2019, Summaries ofTechnical Papers of Annual Meeting, Information Systems
Technology, 13-14.
6） Rusu, R. B. (2010). Semantic 3d object maps for everyday
manipulation in human living environments. KI-KünstlicheIntelligenz, 24(4), 345-348.

－ 219 －