Automatic Generation Method of Building Mask Images
by Using the 3D Model with Aerial Photograph for Deep Learning
Toward authentic optimal placement system of outdoor loudspeakerfor communication of disaster prevention information○ Kazunosuke Ikeno*1, Tomohiro Fukuda *2 and Nobuyoshi Yabuki *3*1 Div. of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka University*2 Assoc. Prof., Div. of Sustainable Energy and Environmental Engineering, Graduate School of Engineerin g, Osaka University, Ph.D.
*3 Prof., Div. of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka University, Ph.D.
Keywords: Disaster prevention; Disaster administrative radio; Machine learning;
Automatic design; Semantic segmentation; Mask image.

1. Introduction
Theinformation about the building placement in GIS. It is difficult

importanceof

theadministrativeauthorities

to confirm the present state. There is a method to confirm landcommunicates information about disaster prevention isuse from aerial photographs. This method enables to get theincreased. When a natural disaster occurs in Japan,

current information easily and inexpensively.

administrative authorities communicate this information using

Recently, the method of object detection by using

Common Antenna TeleVision (CATV), emergency alert email

machine learning is proposed. We can detect the target objects

and disaster administrative radio1).

on images quickly using this method. It is also possible to

CATV and emergency alert email can convey the opticaldetect buildings on aerial photographs using this method4).

information clearly, however only many people can get

The accuracy of building detection is greatly influenced byinformation. On the other hand, disaster administrative radio

the quantity and features of the dataset. It is necessary to trainsystem has been developed in roughly 82% of thethe model adequately for each target area. However, it takes

municipalities in Japan2). There are two ways, individualmuch time to generate mask images from aerial photographs

house receivers and outdoor loudspeakers for transmission

for training models because there are many buildings in one

methods. Ministry of Internal Affairs and Communications

aerial photograph and many sets of aerial photographs andrecommends distribution individual house receivers to allmask images are needed to train the model.

family because the transmission method using individual

The method to generate mask images automatically byhouse receivers hardly receive influence from the weather,using Virtual Reality (VR) models for machine learning ishowever, many municipalities use not individual houseproposed5). We can generate mask images easily by using 3D

receivers but outdoor loudspeakers due to financial mattersvirtual models that are divided into each object such as

and limited cost-effectiveness. However, there is a problem

buildings, roads, and terrain. However, the representation ofthat some regions don’t receive the information from outdoor

the VR models is sometimes deficient to be used as datasets

3)

loudspeakers .

Municipalities

needtoreconsider

theplacement of outdoor loudspeakers.

for training the model. In this research, by pasting aerial
photographs on 3D models as materials , representation of theMunicipalities request experts to consider the optimal

VR models expects to be improved. We can generate mask

placement of outdoor loudspeakers when they prepare theimages automatically by using these VR models and prepare

update plan. The experts make an investigation into the land

proper datasets.

use from Geographic Information System (GIS) and map

This study aims to develop the automatic optimal

owned by each municipality. However there are some cases

placement system of outdoor loudspeakers by analyzing landthat the information provided by the local government don’t

use from aerial photographs. This article describes the method

correspond to present state in provincial cities with a large

to detect buildings on aerial photographs by the model trained

population size and regions that land use changes frequently.

by using the existing dataset and generate mask images

The information on local cities in GIS is updated only

automatically from VR models for training models.

occasionally and it is expensive. In some cases, there is no
日本建築学会情報システム技術委員会
第 42 回情報･システム･利用･技術シンポジウム論文集， 172-175 ， 2019 年 12 月，東京
Proceedings of the 42nd Symposium on Computer Technology of Information ，Systems and Applications ， AIJ ， 172-175 ， Dec. ， 2019 ， Tokyo－ 172 －2. System development
2.1. PROPOSED SYSTEM
Our proposed system detects buildings on aerial photographs
using semantic segmentation, understand land usage patternand provide optimal placement by using geometric analysis.
Semantic segmentation is one of the object detection methods
using machine learning. Conceptual diagram of our proposedsystem is presented in Figure 1.

Figure 3. Flowchart of generating mask images
2.2. SYSTEM IMPLEMENTATION
2.2.1. DEVELOPMENT ENVIRONMENT
The libraries and software for detecting buildings on aerial
photographs using machine learning are shown in A) and B).
The OS of the PC is Ubuntu 16.04 LTS.
A) TensorboardX 1.6
TensorboardX can show loss output during training inFigure 1. Conceptual diagram of our proposed system

real-time using Application Programming Interface (API). It
was used to confirm the training state of the model.

The system for generating mask images to train the modelB) Jupyter Notebook

is developed. This system loads 3D models include terrain

Jupyter Notebook is one of the analytical tools. It canand building objects, classify the building class and others

manage programs, explanations and execution results at once.

class, switch between the model include all objects and theIt was used to output segmentation results.

model include only building and generate two upper view

The libraries and software for generating sets of mask

images of the models. This system can generate plural sets ofimages and aerial photographs automatically are shown in C)

mask images and aerial photographs from one 3D model.

and D). The OS of the PC is Windows 10 Education 64bit.

Conceptual diagram and flowchart of generating the dataset isC) Autodesk InfraWorks 2019presented in Figure 2 and Figure 3.

Autodesk InfraWorks is one of the infrastructure design
software. It was used to create 3D models of the target area.
D) Unity 2017.3.1f1 (64-bit)
Unity is one of the game engines that can load 3D models.
It was used to build systems.
2.2.2. PRELIMINARY EXPERIMENTAs preliminary experiment, we train the model by using
existing dataset. Our proposed system used U-Net6) because it
is a type of methods to detect objects in pixel units forsegmentation of buildings on aerial photographs. The dataset
for training was SpaceNet7) include satellite images andground-truth building footprints in Rio de Janeiro, Vegas,Paris, Shanghai, Khartoum, and Atlanta. Our proposed systemused the dataset of Rio de Janeiro because they have mostbuilding labels (polygons) in the six datasets. The satellite

Figure 2. Conceptual diagram of generating mask images

images of SpaceNet are too large (19584 px×19584 px) to be
used for training the model. Then we divided the original

報告 H25－ 173 －images into 6940 small images (438 px×406 px) andto take images on the 3D model. The x-coordinates andclassified them into three classes. The model was trained onz-coordinates of the camera were changed by 100 inU-Net by using the images of ground-truth building and5600×5200 (world coordinates of Unity). The camera

satellite images. After that, we evaluated the accuracy of theoutputted images include only aerial photographs and images

model. For verification, Intersection over Union (IoU)8) which

only building masks as png files (400 px×400 px). Conceptualis one of the metrics that evaluate how similar predicted area

diagram of the system to generate sets of aerial photographs

is to the ground truth area was used. IoU is given as Eq. (1).

and mask images automatically is presented in Figure 4.

IoU = TP / ( TP + FN + FP )(1)

TP (True Positive), FN (False Negative), FP (False Positive)

3. Results
The results of detecting buildings on aerial photographs in thetest class by the trained model are presented in Figure 5. The

2.2.3. GENERATING MASK IMAGES AUTOMATICALLY

red area is the predicted and true area of buildings. IoU that

The 3D model of target area was created. by using Autodesk

calculated using sets of aerial photographs and mask images

InfraWorks. The building placement were determinedin verification class is shown in Table.1. The results ofaccording to Fundamental Geospatial Data9) (FGD) provided

detecting buildings on aerial photographs in Sakaiminato by

the Geospatial Information Authority of Japan (GSI). The

the trained model are presented in Figure 6.

aerial photographs are pasted on objects of terrain.
The objects of the created model were classified into

Table 1. IoU of the trained modelbuilding objects and others on Unity. We defined the camera

IoU

0.602

Figure 4. System to generate sets of aerial photographs and mask images

Input

Prediction

TrueFigure 5. Results of detecting buildings on test photographs

報告 H25Input

Prediction

True

Figure 6. Results of detecting buildings on Sakaiminato photographs－ 174 －

The sets of aerial photographs and mask images thatgenerated automatically by our proposed system are presented

5. Conclusion
The conclusions of the present study are shown below.
 Proposed the system to generate sets of aerial

in Figure 7. Our proposed system generated 2912 sets in 190
seconds.

photographs and mask images from 3D models.
 Compared mask images generated automaticallyand manually.
The future works is to adjust the color tone of target aerial
photographs to the photographs in the dataset. In the next step,
we train the model using the mask images generated by ourproposed system and evaluate the accuracy of the trainedmodel.

Photographs

Mask images

Automatically generated

Mask images
Manually generated

Figure 7. Generated aerial photographs and mask images
4. Discussion
Our proposed system detected buildings on aerial photographs
in the existing dataset with the accuracy that IoU is 0.602.
IoU of this system exceeds the threshold value (IoU = 0.5)10).
This system can detect buildings sufficiently. However, there
are some misdetected and undetected buildings in the resultsof detecting buildings on target aerial photographs. There are
two reasons for these problems. First, the color tone of aerial
photographs in dataset used to train the model significantly
different to target aerial photographs. Second, there are few
vacant lots and farmlands in the aerial photographs in thedataset of Rio de Janeiro. It is necessary to consider thesethings for the building segmentation on local cities that have
many vacant lots and farmland.
The time to generate mask images is reduced bygenerating them from 3D models automatically. The mask
images generated by our proposed system are almost the sameas the mask images generated manually. This system cangenerate mask images in detail shapes. However, it cannot
generate mask images of small warehouses. Some aerial
photographs that include thin lines on boundaries between

References1） Fire and Disaster Management Agency: 2017, Chiho bosaigyoseino genkyo [Present situation of the local disaster prevention administration] <http://www.fdma.go.jp/disaster/chihoubousai/pdf/27/genkyo.pdf> (accessed 24 January 2019).
2） Ministry of Internal Affairs and Communications: 2019, Sh
ichoson bosai gyosei musen tou seibi jokyo [States of development of disaster administrative radio system] <https://www.tele.soumu.go.jp/j/adm/system/trunk/disaster/change/inde
x.htm?print> (accessed 3 February 2019).
3） Ministry of Internal Affairs and Communications: 2017, Sa
igai joho dentatsu syudan no saikin no doko [Recent trendof the disaster communication means] <http://www.soumu.go.jp/main_content/000525713.pdf> (accessed 9 October 2018).
4） Adachi, K. and Iwata, T.: 2019, A Study on Recognition ofBuildings in Aerial Photograph Based on Machine Learning,
Summaries of technical papers of Annual Meeting Architectural
Institute of Japan, Information systems technology, 57-58.
5） Fukuda, T., Novak, M. and Fujii, H.: 2019, Development ofSegmentation-Rendering on Virtual Reality for Training
Deep-learning, Simulating Landscapes and Advanced User
Experience, Proceeding of the 37th eCAADe and 23th SIGraDi
Conference, Volume 2, 433–440.
6） Ronneberger, O., Fischer, P. and Brox, T.: 2015, “Unet:Convolutional networks for biomedical image segmentation”.
In International Conference on Medical Image Computing andComputer-Assisted Intervention, 234–241. Springer.
7） DigitaalGlobe: 2019, SpaceNet <https://explore.digitalglobe.com/SpaceNet-Thank-You.html> (accessed 9 September 2019).
8） Everingham, M., Ali Eslami, S. M., Van Gool, L., Williams, C.
K. I., Winn, J. and Zisserman, A.: 2015, The Pascal VisualObject Classes Challenge: A Retrospective, International
Journal of Computer Vision, 111(1), 98–136.
9） Geospatial Information Authority of Japan: 2019, Kibanchi
zu johosaito [Site of basic map information] <https://www.
gsi.go.jp/kiban/> (accessed 10 September 2019).
10） Jabbar, A., Farrawell, L., Fountain, J. and Chalup, S. K.: 2017,
Training Deep Neural Networks for Detecting Drinking Glasses
Using Synthetic Images, Neural Information Processing, Part 2,354–363.

each object of the 3D model. It is necessary to prescreen
generated mask images. This system can be used for otherdetected targets that location information is provided.

報告 H25－ 175 －