Integration of Real-Time Video Communication and Semantic Segmentation
over the Internet Toward Dynamic Occlusion Handlingin Mobile Mixed Reality for Landscape Simulation○ Daiki KIDO*1, Tomohiro FUKUDA*2 and Nobuyoshi YABUKI*3*1 Grad. Student, Div. of Sustainable Energy and Environmental Engineering, Grad. School of Engineering, Osaka University.
*2 Assoc. Prof., Div. of Sustainable Energy and Environmental Engineering, Grad. School of Engineering, Osaka University, Ph.D.
*3 Prof., Div. of Sustainable Energy and Environmental Engineering, Grad. School of Engineering, Osaka University, Ph.D.
Keywords: Real-Time video communication; semantic segmentation; Internet; mobile Mixed Reality;
dynamic occlusion handling; landscape simulation.

1. Introduction

Depth-based methods acquire the depth information of theTo form a good landscape, it is essential to discuss the project

occlusion target from a 3D sensing camera in real-time such asactively among stakeholders. Stakeholders consist of experts

RGB-D camera or stereo camera, and compare the depth of thesuch as architects and government, and non-experts such as

virtual objects with the acquired depth information to handle

residents. Then it is difficult for stakeholders, especially theocclusion

non-experts, to imagine the planned landscape, especially adynamically in real-time, however have a limitation of thethree-dimensional (3D) object that has not existed yet. To

distance to obtain depth information and are unsuitable for a

facilitate the lively discussion, it is effective to visualize thelarge-scale landscape simulation outdoor.

planned landscape in the planning and design phase.

3),

4).

These methods can handle occlusion

Contour-based methods detect and track the silhouette of

Recently, the use of mixed reality (MR) has attracted

the occlusion target and handle occlusion. In our previous study,

attention in the architecture, engineering and construction

we developed a MR system for landscape design simulation that(AEC) field. MR is a technology that merges the real and virtual

performs dynamic occlusion handling using real-time semantic

worlds 1), and MR can produce a realistic landscape simulation

segmentation based on deep learning 5) (Figure 1). Semanticby overlaying a 3D computer graphic (CG) model of a newsegmentation is a technique to link each pixel in an image to abuilding on a camera view of the planning site.

class label. Since real-time semantic segmentation processing

One of the technical challenges in MR is the occlusion

involves heavy processing and needs a high-end computer, weproblem, which occurs when virtual objects hide physicaldeveloped a system in which client device transfers image

objects that should be rendered in front of virtual objects. As

frames to a server, which performs semantic segmentation

MR is realized by overlaying 3DCG models on a camera view,processing on those frames and sends processed frames back to

it is difficult to render the correct order of the real and virtual

the client. To realize the client-server communication in thisobjects from the line-of-sight. Incorrect occlusion may cause

system, a web application was developed and utilized. However,user confusion related to the depth perception of the user.

the client-server communication based on a web application is

Properly occlusion handling is an essential issue for the spatial

not a versatile method in terms of computer security. It is

understanding between the real and virtual objects.

necessary to access the server computer for the use of the web

Previous studies of occlusion handling methods have been

application. If the web application was deployed to access from

divided into three categories: model-based method, depth-based

everywhere, we have to manage and secure the server computer

method and contour-based method.

and it takes a lot of care and cost.

Model-based methods create a 3DCG model of theocclusion target (occlusion model) in pre-processing andcompare the depth of the virtual objects with the occlusion
model to handle occlusion 2). In these methods, if the occlusion
targets move or change its shape over time such as vegetation, it
is difficult to handle occlusion because these methods need tocreate an occlusion model in pre-processing.
日本建築学会情報システム技術委員会
第 42 回情報･システム･利用･技術シンポジウム論文集， 212-215 ， 2019 年 12 月，東京
Proceedings of the 42nd Symposium on Computer Technology of Information ，Systems and Applications ， AIJ ， 212-215 ， Dec. ， 2019 ， Tokyo

Figure 1. MR w/ occlusion using semantic segmentation 5)－ 212 －Then, we focused on real-time video communication
technology

overInternetconnection, the stream data that is to be transferred should be

without deploying the web

also specified. Then, a client device can specify the image

application. When we use this video communication technology,

frames acquired from a webcam connected to a client device as

it is not necessary to manage the server computer in terms of

the stream data, however a server computer doesn’t have any

computer security and possible to use this MR systemdata to transfer. Therefore a canvas element that has no data waseverywhere. Therefore, in our study, we aim to integrate thespecified as the stream data in a server computer at first. After

real-time video communication and semantic segmentation

the P2P connection is established, processed frames wereprocessing over the Internet based on deep learning toward

copied to the canvas element that was specified as the stream

dynamic occlusion handling in mobile MR for landscapedata and transferred to the client device (Figure 3).

everywhere

toconductneeds to be established. At the timing of establishing a P2Pclient-server

communication

thesimulation.
2.3. BROWSER-PYTHON COMMUNICATION (SERVER)
2. Proposed system

A server computer performs semantic segmentation processing

2.1. SYSTEM OVERVIEW

on the acquired webcam frames. However, the webcam frames

The overview of our proposed system is shown in Figure 2.

were acquired on the web browser written in JavaScript andConsidering that the MR system for landscape simulation was

semantic segmentation processing was performed with python

developed in a game engine in our previous study5),

weintegrated a game engine and video communication technology.

programming

language.

Thusinoursystem,aJavaScript-Python communication was implemented
In JavaScript, Ajax (asynchronous JavaScript and XML)2.2. REAL-TIME VIDEO COMMUNICATION

was used for JavaScript-Python communication. With Ajax,

As explained in chapter 1, we focused on real-time video

web applications can send and receive data from a server

communication

asynchronously without reloading the whole web page. In ourtechnology

torealize

client-server

communication for real-time semantic segmentation processing

system, webcam frames were transferred to python with Ajax.

in mobile MR. To realize real-time video communication, weIn

python,flask

wasusedforJavaScript-Python

adopted WebRTC (Web Real-Time Communications) 6), which

communication, which is a micro web application framework

enables web applications and sites to capture and optionally

written in python. In our system, webcam frames were received

stream audio and/or video media between browsers in real-time.

with flask and performed semantic segmentation processing,

WebRTCapplicationand sended processed frames back to JavaScript. The

programming interfaces (APIs) and protocols which workcommunication flow between JavaScript and python is shown

together to achieve real-time communication.

in Figure 4.

consistsofseveral

interrelated

In our system, the packages of WebRTC were installed toboth a client device and a server computer. When the2.4.

communication with WebRTC is started, peer to peer (P2P)(CLIENT)BROWSER-GAMEENGINECOMMUNICATION

connection between the client device and the server computer

Webcam frames and semantic segmentation images in the webbrowser need to be transferred to a game engine in the client
device to develop a MR system. Since it is difficult to send
those images from JavaScript to game engine directly, wedeveloped an intermediate server at the localhost in client
device using flask.

Figure 3. WebRTC communication in our system (Left: At theFigure 2. System overview.

報告 H19timing of P2P connection, Right: After the P2P connection).

－ 213 －and fence.
In the accuracy test, 3 frames of each communication wereselected and IoU (Intersection over Union) was calculated,which is the index to measure the accuracy of image
segmentation. IoU was calculated with the ground-truth
annotation image and predicted segmentation image (Figure 6).
In the client-server communication, the server computer
was connected to LAN/Internet with a cable and the client
device was connected to LAN/Internet with wi-fi. The
communication speed between client-server over the LAN wasmeasured with the ping command, and the communication
speed on the Internet was measured using a web site 10).
Measured communication speed was shown in Table 1. We useda surface pro 6 tablet computer with Intel Core i5-8250U ofCPU, 8GB of RAM, and Intel UHD Graphics 620 of GPU as aFigure 4. Communication flow between JavaScript and python.

client device and video camera, and a desktop computer withIntel Core i7-8700K of CPU, 32GB of RAM, and NVIDIA

In JavaScript, we adopted Ajax as explained in 2.3 andGeForce GTX 1080Ti 11GB of GPU as a server computer. The

transferred the webcam frames and semantic segmentationclient device was panned to acquire the video (Figure 7). The

images to the intermediate server.

resolution of the acquired video was 1024 × 576 pixels.

In a game engine, GET method in HTTP communication

Verification results are shown in Figure 8, 9 and Table 2.

was used to request the data and the game engine received theIn this verification, we confirmed that the real-time video

webcam frames and semantic segmentation images from thecommunication over the Internet and semantic segmentationintermediate server (Figure 5).

processing were integrated and video frames and semantic
segmentation images were displayed in Unity.
As Figure 8, 9 and Table 2 show, the accuracy of semantic

3. Verification experiment
A verification experiment was conducted to evaluate thedeveloped system. We adopted ICNet7)

segmentation over the Internet was lower than over the LAN. It

as a semantic

is considered that this lower accuracy over the Internet wassegmentation method, cityscapes 8) as a dataset for ICNet based

caused by the bitrate of the video communication with WebRTC.

on our previous study 5), and Unity as a game engine. We used aBitrate is the number of bits that are conveyed or processed per

WebRTC communication platform provided by Skyway 9).

unit of time. The transferred video frames were compressed andWe compared the accuracy of the semantic segmentation

the data size became smaller because of the low bitrate of video

processing and processing speed in Unity between the case that

communication. Thus, it is necessary to improve the bitrate forthe communication was over the local area network (LAN) andover the Internet. In the LAN communication, Unity and python
to perform semantic segmentation were directly communicatedwithout video communication based on our previous study 5).
The target site of semantic segmentation was selected the samesite whose view from a camera consists of vegetation, building

Figure 6. IoU (Intersection over Union) calculation.

Figure 5. Communication between JavaScript and game engine.

Figure 7. Scene of verification experiment.

報告 H19－ 214 －Table 1. Measured communication speed.
Communication

Speed [Mbps]

In this study, we integrated the real-time video communication

Client-Server

19.2

and semantic segmentation processing based on deep learning

Brower(cli.)-Server10)

Up: 113.8, Down: 118.7

over the Internet toward dynamic occlusion handling in mobile

Browser(ser.)-Server10)

Up: 626.5, Down: 625.5

MR for landscape simulation. We conducted a verification

LANInternet4. Conclusions and future work

experiment,

andweconfirmedthat

real-time

video

communication over the Internet and a game engine wereintegrated.
In the future works, the bitrate of video communication andthe communication between the intermediate server and a gameengine in client device should be improved. And this systemshould be integrated with a MR system for dynamic occlusion
handling.
Figure 8. Verification results.
AcknowledgementsThis research has been partly supported by JSPS KAKENHI Grant
Number JP19K12681.

more accurate segmentation.
As Table 2 shows, processing speed in Unity over theInternet was much faster than over the LAN. However, thoughclient-server communication over the Internet for uploading
video frames and downloading segmentation images in client
device was about 25 fps (frames per second), the processing
speed of intermediate server-Unity communication in client
device was about 5 fps. It was revealed that the communication
speed between the intermediate server and Unity in the client
device should be improved.
With this system, the camera view can be segmented andhandled as one likes in a game engine. We can use this system
not only for dynamic occlusion handling in MR for landscapesimulation as explained in chapter 1, but also for visualenvironmental assessment in real-time; the ratio of each objectfor example.
Table 2. Processing speed in Unity.
Communication

Processing speed [fps]LAN

6~7Internet

15~16
Building

IoU (Intersection over Union)

1

Fence

Sky

Vegetation

Internet2

Internet3

0.9
0.8
0.7
0.6
0.5
0.4
0.3

0.2
0.1
0LAN1

LAN2

LAN3

Internet1

ImagesReferences1） Milgram, P. and Kishino, F.: 1994, A Taxonomy of Mixed Reality
Visual Displays, IEICE Transactions on Information Systems,E77-D, 12, 1321-1329.
2） Inoue, K., Fukuda, T., Cao, R., and Yabuki, N.: 2018, Tracking
Robustness and Green View Index Estimation of Augmented andDiminished
RealityforEnvironmental
Design:PhotoAR+DR2017 project, Proceedings of the 23rd International
Conference on Computer-Aided Architectural Design Research in
Asia (CAADRIA 2018), 339-348.
3） Du, C., Chen, Y., Ye, M., and Ren, L.: 2016, Edge snapping-based
depth enhancement for dynamic occlusion handling in augmentedreality, In the 15th IEEE International Symposium on Mixed andAugmented Reality (ISMAR 2016), 54-62.
4） Holynski, A. and Kopt, J.: 2018, Fast depth densification forocclusion-aware augmented reality, ACM Transactions onGraphics (TOG), 37 (6), 194, 1-11.
5） Kido, D., Fukuda, T., and Yabuki, N.: 2019, Development of a
Semantic Segmentation System for Dynamic Occlusion Handlingin Mixed Reality for Landscape Simulation, Proceedings of the37th eCAADe and 23rd SIGraDi Conference, 1, 641-648
6） Jennings, C., Hardie, T., and Westerlund, M.: 2013, Real-timecommunications for the web, IEEE Communications Magazine,51, 4, 20-26.
7） Zhao, H., Qi, X., Shen, A., Shi, J., and Jia, J.: 2018, ICNet forReal-Time Semantic Segmentation on High-Resolution Images,
In Proceedings of European Conference on Computer Vision(ECCV 2018), 418-434.
8） Cordts, M., Omran, S., Ramos, S., Rehfeld, T., Enzweiler, M.,
Benenson, R., Franke, U., Roth, S., and Schiele, B.: 2016, The
cityscapes dataset for semantic urban scene understanding, In
Proceedings of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR 2016), 3213-3223.
9） Skyway – Enterprise Cloud WebRTC Platform – :<https://webrtc.ecl.ntt.com/> (accessed 3 October 2019).
10） Speedtest by Ookla – The Global Broadband Speed Test: <https://www.speedtest.net/> (accessed 3 October 2019).

Figure 9. IoU of semantic segmentation images.

報告 H19－ 215 －