ISBN978-4-600-00276-3

ICCBEI 2019
Proceeding of the 4th International Conferenceon Civil and Building Engineering Informatics
November 7-8, 2019, Sendai, Japan

Edited by Koji Makanae and Nobuyoshi Yabuki
ICCBEI 2019 Organizing Committee

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

ICCBEI 2019

4th International Conference on Civil and Building Engineering Informatics

Organizers

CEI Asian Group for Civil Engineering Informatics (AGCEI)
Committee on Civil Engineering Informatics,
Japan Society of Civil Engineers (JSCE)

Supporters
Japan Construction Information Center

Miyagi University

Sponsors
Gold Sponsors:

Forum 8 Co., Ltd.
Kajima Corporation
Applied Technology Co., Ltd. and Autodesk Inc.
Sendai Tourism, Convention
and International Association (SenTIA)

Silver Sponsors:

Chodai Co.,Ltd
Chuo Fukken Consultants Co.,Ltd.
JIP Techno Science Corporation
Shimizu Corporation
Trion Corporation
Obayashi Corporation
Yachiyo Engineering Co.Ltd

ISBN978-4-600-00276-3
Edited by Koji Makanae and Nobuyoshi Yabuki
Published by ICCBEI 2019 Organizing Committee
COPYRIGHT NOTICE: Copyright © 2019 by the authors of the individual papers.
All materials are distributed under the Creative Commons Attribution-NonCommercialNoDerivatives 4.0 International License (CC-BY-NC-ND 4.0).

The full papers of this proceeding can be download from:https://www.iccbei2019.com

ⅠPREFACE
We are very pleased to hold the International Conference on Civil and Building
Engineering Informatics (ICCBEI 2019) in Sendai City, Miyagi Prefecture, Japan. The
organizer of ICCBEI, Asian Group for Civil and Building Informatics (AGCEI), wasdevelopmentally organized from the series of Asian Construction Information Technology
Roundtable Meeting sponsored by Japan Society of Civil Engineers (JSCE) and Japan
Construction Information Center (JACIC). AGCEI is the sister organization of the International
Society for Computing on Civil and Building Engineering (ISCCBE), which holds theInternational Conference on Computing in Civil and Building Engineering (ICCCBE) in theeven number years. AGCEI decided to hold the international conference in the Asian andPacific region in the odd number year and held the conference in Tokyo in 2013, then Tokyoin 2015, and Taipei in 2017. ICCBEI 2019 is the fourth conference, which is the first conference
outside of Tokyo in Japan.
In a decade, information and communication technologies (ICT) have advanced
continuously and more rapidly. Various new technologies such as 3D scanning using drones,
computer vision, IoT sensors and actuators, robot and machinery control, and so on have beendeveloped and applied to civil and building engineering. Also, artificial intelligence (AI) has
been able to analyze big data generated by accumulating a large amount of information andreplace human knowledge and cognition. Furthermore, the information or data-centric concept
has evolved, and the information and data are becoming a core in linking for Cyber-Physical
Systems. In every phase in the lifecycles of building and infrastructure, building information
modeling(BIM) is becoming increasingly important as the core for storing information. In thistrend, “informatics” in the title of the conference is becoming a key academic area in civil andbuilding engineering, and we should make further progress.
In response to our Call for Papers, 78 abstracts were submitted, and our international
scientific committee has finally accepted 62 full papers after rigorous reviews. These accepted
papers have included in the proceedings with three abstracts of keynote lectures. The papers
can be categorized into nine academic categories without industrial/ technical category:
Building and Construction Information Modeling, AI and Data Analysis, Image Processing andComputer Vision, Visualization and XR(VR/AR/MR), IoT/Sensors and Monitoring, Laser and
Image
Scanning,
Facility
andInfrastructureManagement,
Computational
Mechanics/Engineering, and Information and Process Management. As these categories show,
the papers target innovative topics in civil and building informatics. We hope that ICCBEI 2019
and the publication of the proceeding will contribute to the development and dissemination of
Civil and Building Engineering long into the future.
Furthermore, the Sendai area where ICCBEI 2019 will be held was affected by thedisaster of the large earthquake in 2011. Especially, the coastal area was heavily hit by Tsunami,
and some cities were flown out. Many reconstruction projects in the coastal area are underway,
and many applications of ICT are tried. We hope that our discussions at this conference will be
useful for the reconstruction of disaster areas.
Koji Makanae
Chair, ICCBEI 2019 Organizing Committee
Chair, Committee on Civil Engineering Committee,
Japan Society of Civil Engineers (JSCE)
Professor, Miyagi University, Japan

ⅠI

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Organization of ICCBEI 2019

Asian Group for Civil Engineering Informatics (AGCEI)Board of Director
President:
Nobuyoshi Yabuki, Osaka University, Japan

Members:
Weng Tat Chan, National University of Singapore, Singapore
Atsushi Fukasawa, Japan Construction Information Center, Japan
Shang-Hsien (Patrick) Hsieh, National Taiwan University, Taiwan
Jun Shang Kuang, The Hong Kong University of Science and Technology, Hong Kong
Sang-Ho Lee, Yonsei University, Korea
Veerasak Likhitruangsilp, Chulalongkorn University, Thailand
Zhiliang Ma, Tsinghua University, China
Koji Makanae, Miyagi University, Japan
Xiangyu Wang, Curtin University, Australia

Committee on Civil Engineering Informatics,

Japan Society of Civil Engineers (JSCE)
Chair:
Koji Makanae, Miyagi University, Japan
Vice-Chair:
Katsutoshi Yasui, Obayashi Corporation, Japan
Osamu Okamoto, Ibaraki National College of Technology
Secretary-General:
Hiroaki Mori, Chuo Fukken Consultants Co., Ltd., Japan
Members:
27 Standing Committee Members and Other Sub-committee members.

Supporters:
Japan Construction
Information CenterⅢProceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Committees of ICCBEI 2019
Organizing Committee:
Chair:
Koji Makanae, Miyagi University, Japan
Vice-Chair:
Nobuyoshi Yabuki, Osaka University, Japan
Members:
Yoichiro Chiba, Pacific Consultants International Group, Japan
Hiroshi Fukumori, Shimizu Corporation, Japan
Teppei Ishiuchi, Miyagi University, Japan
Hirotoshi Ito, Eight-Japan Engineering Consultants Inc., Japan
Hirofumi Matsuda, Trion Corporation, Japan
Hiroaki Mori, Chuo Fukken Consultants Co., Ltd., Japan
Masaki Sawa, Hazama Ando Corporation, Japan
Yoshihide Sekimoto, University of Tokyo, Japan
Masato Shiozaki, Mitsui Sumitomo Construction Co., Ltd., Japan
Ko Ueyama, CTI Engineering Co., Ltd., Japan
Katsutoshi Yasui, Obayashi Corporation, Japan

Scientific Committee:
Chair:
Nobuyoshi Yabuki, Osaka University, Japan
Vice-Chair:
Koji Makanae, Miyagi University, Japan
Members:
Robert Amor, The University of Auckland, New ZealandWeng Tat Chan, National University of Singapore, Singapore
Albert Y. Chen, National Taiwan University, Taiwan
Jack C. P. Cheng, The Hong Kong University of Science and Technology, Hong Kong
Hiroshige Dan, Kansai University, Japan
Tomohiro Fukuda, Osaka University, JapanLei Hou, Royal Melbourne Institute of Technology, Australia
Patrick Shang-Hsien Hsieh, National Taiwan University, Taiwan
Satoshi Kanai, Hokkaido University, Japan
Teppei Ishiuchi, Miyagi University, JapanKei Kawamura, Yamaguchi University, JapanJun Shang Kuang, The Hong Kong University of Science and Technology, Hong Kong
Sang-Ho Lee, Yonsei University, Korea
Veerasak Likhitruangsilp, Chulalongkorn University, Thailand
Vincent Yu-Cheng Lin, National Taipei University of Technology, Taiwan
Wilson W.S. Lu, The University of Hong Kong, Hong Kong
Zhiliang Ma, Tsinghua University, China
Tomohiro Mizoguchi, Nihon University, Japan
Kantaro Monobe, Tohoku Gakuin University, JapanAli Motamedi, École de Technologie Supérieure, Canada
Tatsunori Sada, Nihon University, Japan
Masayuki Saeki, Tokyo University of Science, Japan
Raj Kapur Shah, Liverpool John Moores University, UK
Yoshihiro Yasumuro, Kansai University, Japan
Xiangyu Wang, Curtin University, Australia
Yang Zou, The University of Auckland, New ZealandⅣProceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table of Contents
KEYNOTE LECTURE-1
Information and Communication Technologies (ICT) in Civil and Building Engineering

Prof. Kincho H. Law (Stanford University).............................................................................................................................................................. 1

KEYNOTE LECTURE-2
Digitalization in AEC– German Perspectives from Strategy to Implementation

Prof. Katharina Klemt-Albert (Leibniz Universität Hannover).............................................................................................................................................................. 3

KEYNOTE LECTURE-3
Creation, Integration and Management of BIM Information

Dr. Jack C. P. Cheng (The Hong Kong University of Science and Technology).............................................................................................................................................................. 5

AI and Data Analysis
NLP-based Method for Auto-Correcting Data of Public Constructions

Meng-Lin Yu (National Taiwan University of Science and Technology),
Hao-Yung Chan (National Taiwan University of Science and Technology),
Meng-Han Tsai (National Taiwan University of Science and Technology).............................................................................................................................................................. 8

Analysis of Power Usage and Building Residents Relationships for Energy Savings Using
Social Network Analysis

Ru-Guan Wang (National Central University), Chien-Cheng Chou (National Central University)............................................................................................................................................................ 17

As-built Detection of Steel Frame Structure Using Deep Learning

Ryu Izutsu (Osaka University), Nobuyoshi Yabuki (Osaka University),
Tomohiro Fukuda (Osaka University)............................................................................................................................................................ 25

Video-based Construction Vehicles Type Recognition

Chen-Hsuan Wang (National Taiwan University of Science and Technology),
I-Tung Yang (National Taiwan University of Science and Technology),
Meng-Han Tsai (National Taiwan University of Science and Technology)............................................................................................................................................................ 33

GA-based Optimization for Construction Sequence of Precast and Cast-in-place Concrete
Components

Songyang Li (Tsinghua University), Zhiliang Ma (Tsinghua University)............................................................................................................................................................ 41

Basic Study on Detecting Destruction Sounds of Structural Members by Using Machine
Learning

Takeo Izumita (Tokyo University of Science), Masayuki Saeki (Tokyo University of Science)............................................................................................................................................................ 49

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Computer Vision-based In-building Human Demand Estimation for Installation of AutomatedExternal Defibrillators

Wen-Xin Qiu (National Taiwan University), Albert Y. Chen (National Taiwan University)............................................................................................................................................................ 55

Multi-Dimensional Sequence Alignment for Context-Aware Human Action Analysis of BodySensor Data

Nipun Nath (Texas A&M University), Amir Behzadan (Texas A&M University),
Prabhat Shrestha (Texas A&M University)............................................................................................................................................................ 61

Application of Intelligent Systems in the Underground Excavation Industry: A Short Review

Mohsen Ramezanshirazi (Sapienza University of Rome),
Mohammad Norizadeh Cherloo (University of science and technology), Orod Zarrin............................................................................................................................................................ 69

Determination of Automated Construction Operations from Sensor Data Using Machine
Learning

Aparna Harichandran (Curtin University), Benny Raphael (Indian Institute of Technology Madras),
Abhijit Mukherjee (Curtin University)............................................................................................................................................................ 77

Image Processing and Computer Vision
Study on Automatic Chalk Marks Recognition for Concrete Tunnel Inspection Using Deep
Learning

Yao Zhang (Yamaguchi University), Kei Kawamura (Yamaguchi University),
Cuong Nguyen Kim (Mientrung University of Civil Engineering),
Koji Oshikiri (Ricoh Company, Ltd.), Taro Kikuchi (Ricoh Company, Ltd.)............................................................................................................................................................ 86

GA-CNN Based Automatic Crack Detection and Classification Method for Concrete
Infrastructures

Cuong Nguyen Kim (Mientrung University of Civil Engineering),
Kei Kawamura (Yamaguchi University), Yao Zhang (Yamaguchi University)............................................................................................................................................................ 92

As-built Modeling of Steel Structures Using Symmetry

Takuya Suzuki (Nihon University), Tomohiro Mizoguchi (Nihon University).......................................................................................................................................................... 100

A Review of Research on Advanced Autonomous Technologies for Bridge Inspection

Rina Hasuike (Gifu University), Koji Kinoshita (Gifu University), Lei Hou (RMIT University).......................................................................................................................................................... 108

A Performance Evaluation of Feature Detectors and Descriptors for Unmodified Infrastructure
Site Digital Images

Natthapol Saovana (Osaka University), Nobuyoshi Yabuki (Osaka University),
Tomohiro Fukuda (Osaka University).......................................................................................................................................................... 113

Visualization and XR (VR/AR/MR)
Efficient Access to Inspection Data Based on Augmented Reality Using a “BRIDGE - CARD”

Hisao Emoto (KOSEN), Hiroki Komuro (KOSEN), Takehiko Midorikawa (KOSEN),
Hideaki Nakamura (Yamaguchi University), Kei Kawamura(Yamaguchi University).......................................................................................................................................................... 122

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Bidirectional Linking Of 4D-BIM Planning with Virtual and Augmented Reality

Felix Dreischerf (University of Applied Science Erfurt),
Habeb Astour (University of Applied Science Erfurt).......................................................................................................................................................... 130

Verification About Work Efficiency Improvement by Using an Augmented Reality and Wearable
Computer

Satoshi Yamanaka (Obayashi Corporation), Shinya Sugiura (Obayashi Corporation),
Ryo Tajima (Obayashi Corporation), Takafumi Yamanaka (Obayashi Corporation).......................................................................................................................................................... 136

Visualization of Construction Process by Construction Site Sensing and Digital Twins

Atsushi Takao (Okumura Corporation), Nobuyoshi Yabuki (Osaka University),
Kohei Seto (Okumura Corporation), Iwao Miyata (Okumura Corporation).......................................................................................................................................................... 141

AR Visualization of Physical Barrier for Wheelchair Users Using Depth Imaging

Rio Takahashi (Kansai University), Hiroshige Dan (Kansai University),
Yoshihiro Yasumuro (Kansai University).......................................................................................................................................................... 147

Facility and Infrastructure Management
Steel Bridge Information Delivery Model for Earned Value Management (EVM)

Teruaki Kageyama (Japan Construction Information Center), Nobuyoshi Yabuki (Osaka University).......................................................................................................................................................... 154

A Proposal of Deterioration Prediction Model for Tunnel Lighting Facilities Using Markov
Stochastic Process

Noriaki Maeda (Yamaguchi University), Kei Kawamura (Yamaguchi University).......................................................................................................................................................... 162

VRP-based Model for Lane Marking Assessment with MRU Vehicle

Yu-Chun Lin (National Taiwan University), Si-Ting Liao (National Taiwan University), Chieh Wang
(National Taiwan University), Albert Y. Chen (National Taiwan University).......................................................................................................................................................... 170

The Gateway to Integrating User Behavior Data in “Cognitive Facility Management”

Jinying Xu (The University of Hong Kong), Weisheng Lu (The University of Hong Kong),
Jing Wang (The University of Hong Kong).......................................................................................................................................................... 177

Building and Construction Information Modeling (BIM/CIM)
Research on BIM-based Infrastructure Platform with International Standards

Katsunori Miyamoto (Japan Construction Information Center).......................................................................................................................................................... 185

A BIM-based AR Application for Construction Quality Inspection

Nai-Wen Chi (National Taiwan University of Science and Technology),
Yi-Wen Chen (Moldex3D Co., Ltd.), Shang-Hsien Hsieh (National Taiwan University),
Jen-Yu Han (National Taiwan University), Lung-Mao Huang (Reiju Construction Co., Ltd.).......................................................................................................................................................... 191

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Application of BIM Technology on Virtual Mock-up and Implementation for General
Contractor in Construction

Rodrigo Samuel Ortiz Chaparro (National Taipei University of Technology),
Yu-Cheng Lin (National Taipei University of Technology).......................................................................................................................................................... 197

Deep Learning-based Scan-to-BIM Framework for Complex MEP Scenes Using Laser Scan
Data

Chao Yin (The Hong Kong University of Science and Technology),
Boyu Wang (The Hong Kong University of Science and Technology),
Jack C.P. Cheng (The Hong Kong University of Science and Technology).......................................................................................................................................................... 204

Conversation-based Building Information Delivery System for Facility Management

Kuan-Lin Chen (National Taiwan University of Science and Technology),
Meng-Han Tsai (National Taiwan University of Science and Technology).......................................................................................................................................................... 212

Development of a New District in the State of Hessen “FlexQuartier” ProjectJose Alberto Lagunes Ronzon (Technische Hochschule Mittelhessen),Moritz Hofmann (Technische Hochschule Mittelhessen),
Milena Potpara (Technische Hochschule Mittelhessen),
Dirk Metzger (Technische Hochschule Mittelhessen),
Joaquin Diaz (Technische Hochschule Mittelhessen).......................................................................................................................................................... 219

Process Re-engineering in Owner Organizations to Improve BIM-based Project Delivery Using
Requirements Management Platform

Ali Motamedi (École de technologie supérieure), Sylvain Vaudou (École de technologie supérieure),
Romain Leygonie (École de technologie supérieure), Daniel Forgues (École de technologie supérieure).......................................................................................................................................................... 227

A Framework for Visual BIM-based Maintenance Management in MRT Stations

Yi-Shian Huang (National Taipei University of Technology),
Yu-Cheng Lin (National Taipei University of Technology).......................................................................................................................................................... 235

Modeling the Last-mile Problem of BIM Adoption

Jing Wang (the University of Hong Kong), Weisheng Lu (the University of Hong Kong),
Jinying Xu (the University of Hong Kong).......................................................................................................................................................... 243

BIM-based Wall Framing Calculation Algorithms for Detailed Quantity Takeoff

Chavanont Khosakitchalert (Osaka University), Nobuyoshi Yabuki (Osaka University),
Tomohiro Fukuda (Osaka University).......................................................................................................................................................... 251

BIM-Supported Compliance Verification of Performance-based Car-park Ventilation Design

Johannes Dimyadi (Compliance Audit Systems Limited), Robert Amor (University of Auckland).......................................................................................................................................................... 259

The Study of Automatic CIM Models Development for Infrastructure Projects

Tzu-Tin Huang (National Taipei University of Technology),
Yu-Cheng Lin (National Taipei University of Technology).......................................................................................................................................................... 266

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

An Automated BIM-integrated System for Change Order Cost Impact Evaluation

Veerasak Likhitruangsilp (Chulalongkorn University), Tantri Handayani (Universitas Gadjah Mada),
Nobuyoshi Yabuki (Osaka University), Photios Ioannou (University of Michigan).......................................................................................................................................................... 273

Simulation and Optimization of Utility Tunnels Construction as Linear Project

Mohamed Sherif (The American University in Cairo), Abdelhamid Abdallah (Helwan University),
Khaled Nasser (The American University in Cairo).......................................................................................................................................................... 279

Integrating BIM into Green Residential Building Assessment: A Case Study

Fatma Abdelaal (University of Canterbury), Brian Guo (University of Canterbury),
Yang Zou (University of Auckland), Mazharuddin Syed Ahmed (Ara Institute of Canterbury).......................................................................................................................................................... 287

Enhanced Underground Utilities Management Integrated CIM Technologies

Sheng-Lun Zhuo (National Taipei University of Technology),
Yu-Cheng Lin (National Taipei University of Technology).......................................................................................................................................................... 293

Developing Efficient Mechanisms for BIM Model Simplification

Jack C.P. Cheng (The Hong Kong University of Science and Technology), Keyu Chen (The Hong Kong
University of Science and Technology),
Weiwei Chen (The Hong Kong University of Science and Technology).......................................................................................................................................................... 300

Computational Mechanics/Engineering
Evaluation of Pile Performance in Different Layers of Soil Investigating Pile Behavior by
OpenSeesPL

Orod Zarrin (The University of Newcastle), Mohsen Ramezanshirazi (Sapienza University of Rome).......................................................................................................................................................... 309

Development of Static and Dynamic Modeling Approaches Using Frame Models for City
Seismic Response Analysis

Pher Errol B. Quinay (University of the Philippines Diliman),
Aileen Rachelle Fader (Wallcrete Company, Inc.),
Franz Marius Carangan (Al Abbar Aluminum Philippines, Inc.).......................................................................................................................................................... 318

Structural Shape Grammars used in Intelligent Generation Design of Discrete Structures

Xianzhong Zhao (Tongji University), Ruifeng Luo (Tongji University).......................................................................................................................................................... 325

Numerical Evaluation of Seismic Response of Anchorage Foundation installed in Switchboard
CabinetSang-Moon Lee (Gangneung-Wonju University), Woo-Young Jung (Gangneung-Wonju University),
Ga-Ram Kim (Gangneung-Wonju University).......................................................................................................................................................... 330

IoT, Sensors, and Monitoring
Framework for a BIM-based Real-time Evacuation Guidance System in Smart Buildings

Kayla Manuel (Osaka University), Nobuyoshi Yabuki (Osaka University),
Tomohiro Fukuda (Osaka University).......................................................................................................................................................... 335

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Development of a Method to Detect Earthquake-Related Changes in Images Taken by CCTV
Cameras Surveying Civil Infrastructure

Arata Konno (Ministry of Land, Infrastructure, Transport and Tourism),
Hirotaka Sekiya (Ministry of Land, Infrastructure, Transport and Tourism),
Hideyuki Ashiya (Ministry of Land, Infrastructure, Transport and Tourism).......................................................................................................................................................... 343

A Human Following Robot for Assisting Tunnel Inspectors

Chia-Hsing Ho (Turing Drive Inc.),
Yo-Ming Hsieh (National Taiwan University of Science and Technology).......................................................................................................................................................... 349

A Building Structural Health Management System by BIM and IoT Collaboration

Narito Kurata (Tsukuba University of Technology), Kenro Aihara (National Institute of Informatics),
Takahiro Konishi (Applied Technology Co., Ltd.), Hirofumi Yamaoka (Applied Technology Co., Ltd.),
Shinichi Kondo (Applied Technology Co., Ltd.).......................................................................................................................................................... 335

Development of an Anomaly Detection System of Road Signs Using Mems Accelerometers

Naomasa Haibara (Tokyo University of Science), Masayuki Saeki (Tokyo University of Science).......................................................................................................................................................... 361

Laser and Image Scanning
Automatic Indoor Environment Modeling from Laser-scanned Point Clouds Using Graph-Based
Regular Arrangement Recognition

Hayato Takahashi (Hokkaido University), Hiroaki Date (Hokkaido University),
Satoshi Kanai(Hokkaido University).......................................................................................................................................................... 368

Detecting Building Façade Deteriorations: Evaluation of 3D Laser Scanning and Image-based
Reconstruction Approaches to Determine Feasible Settings in Data Collection

Zhuoya Shi (NYU Tandon School of Engineering), Semiha Ergan (NYU Tandon School of Engineering).......................................................................................................................................................... 376

Automated UAV Route Planning for Bridge Inspection Using BIM-GIS Data

Yang Zou (University of Auckland), Molood Barati (University of Auckland),
Enrique Del Rey Castillo (University of Auckland), Robert Amor (University of Auckland),
Brian H.W. Guo (University of Canterbury), Jiamou Liu (University of Auckland).......................................................................................................................................................... 384

Measuring Railway Facilities by Using Two Mobile Laser Scanners Directly Above the Rails

Kohei Yamamoto (PASCO Corp.), Nobuyoshi Yabuki (Osaka University).......................................................................................................................................................... 392

Information and Process Management
A Feasibility Study for LDAP Certification in Collaboration with Exiting Accounts in RDBMS

Yoshiyuki Yokoyama (Japan Construction Information Center).......................................................................................................................................................... 399

Semantic Modeling of Building Construction Emission Knowledge

Wenkai Luo (RMIT University), Guomin Zhang (RMIT University), Lei Hou (RMIT University),
Malindu Sandanayake (Victoria University).......................................................................................................................................................... 405

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Application and Analysis of System Architecture Model for Construction Project

Tatsuru Tomii (Kokusai Kogyo Co., Ltd.), Koji Makanae (Miyagi University),
Raj Kapur Shah (Liverpool John Moores University).......................................................................................................................................................... 411

Smart Construction Objects (SCOs): A New Theory of Smart Construction Is Born
Weisheng Lu (The University of Hong Kong), Yuhan Niu (Construction Industry Council),
Chimay Anumba (University of Florida).......................................................................................................................................................... 418

Industrial/Technical Papers
Collection Data Using New Tool to Satisfy Specifications

Yasushi Kawanai (Japan Construction Information Center).......................................................................................................................................................... 427

Educational Activity Aimed at Improving Productivity in Japanese Construction Industries
Through 3D-CAD

Yasuyuki Kikyo (Japan Construction Information Center).......................................................................................................................................................... 430

Improvement of Electronic Bidding Core System

Hiroyuki Ishiwata (Japan Construction Information Center).......................................................................................................................................................... 434

Integration of 3D Models of Structures and Geological Composition as an Underground
Infrastructure Model

Toshiaki Hakoda (JGC Corporation), Syoichi Nishiyama (OYO Corporation),
Takaki Omori (Nikken Sekkei Civil Engineering Ltd.), Isao Shiozaki (Engineering
Advancement Association of Japan), Mamoru Narusawa (ESCA-SC),
Nobuyuki Yabuki (Osaka University).......................................................................................................................................................... 438

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Keynote LectureProceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

KEYNOTE LECTURE-1

Information and Communication Technologies (ICT)in Civil And Building Engineering
Prof. Kincho H. Law
Professor, Civil and Environmental Engineering, Stanford University, USA. Email: law@stanford.edu
Abstract: Civil and Building Engineering has had a long and successful history in adopting computing
technologies, from computer graphics, CAD, engineering analyses, virtual simulations, to project management.
As technologies continue to advance, there are many new opportunities that can take advantage of information
science and computing technologies in engineering. Technologies such as building information modeling, virtual
reality, computer vision, sensors, Internet and cloud computing, etc., are now being deployed in civil andconstruction engineering. This presentation will provide an overview of current trends of computing technologies
in the AEC domain. Specifically, the discussions will focus on technologies related to building information
modeling and enterprise integration, and the applications of Internet of Things (IoT) and machine learning in thecivil and building industry.

1

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Profile of Prof. Kincho H. Law
Dr. Kincho H. Law is Professor of Civil and Environmental Engineering at Stanford University. He received his
B.Sc. in Civil Engineering and B.A. in Mathematics from the University of Hawaii in 1976, and M.S. and Ph.D.
in Civil Engineering from Carnegie Mellon University in 1979 and 1981, respectively. After serving as AssistantProfessor at Rensselaer Polytechnic Institute from 1982 to 1988, he joined Stanford University in 1988. Prof.
Law’s research interests focus on computational and information science in engineering. His research has dealt
with various aspects of high performance computing; sensing, monitoring and control of engineering systems;
legal and engineering informatics; smart manufacturing; web services, cloud and Internet computing.
Prof. Law was the recipient of the ASCE Computing in Civil Engineering Award in 2011. He has received anumber of best paper awards from ASCE, ASME, IEEE and Digital Government Society; these include Best Paper(on Data Analytics for Advanced Manufacturing) at IEEE Big Data Conference in 2016, Best Paper at the ASME
Manufacturing Science and Engineering Conference in 2015, Best Paper in the ASCE Journal of Computing in
Civil Engineering in 2014, Best Research Paper (on Resilience and Smart Structures) at the International
Workshop on Computing in Civil Engineering in 2013, Best Research and Practice Paper at 6th International
Conference on Electronic Governance in 2012, Meritorious Paper at the 4th International Conference on Electronic
Governance in 2010, Best Research Paper at the 9th International Conference on Digital Government Research in
2008, and others. Prof. Law was elected Distinguished Member of the American Society of Civil Engineers in
2017, Fellow of the American Society of Mechanical Engineers in 2017, Life Member of the American Society of
Civil Engineers in 2018, and Senior Member of the Institute of Electrical and Electronics Engineers in 2019.

2

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

KEYNOTE LECTURE-2

Digitalization in AEC– German perspectives from strategy to implementation
Prof. Katharina Klemt-Albert
Professor for Construction Management und Digital Engineering, Leibniz Universität Hannover, Germany. Email:klemt-albert@icom.uni-hannover.de
Abstract: Germany has recognized the potential of the BIM method and is proceeding to develop necessary
guidelines and framework conditions at national level as well as implementation acts. This process is driven by
the Federal Ministry of Transport and Digital Infrastructure (BMVI) and the Federal Ministry of the Interior,
Building and Home Affairs (BMI). In 2015, BMVI published the phased plan Digital Planning and Building as a
strategy for the introduction of BIM for federal traffic infrastructure. The plan describes the step-wise path to thegradual introduction of BIM in the area of responsibility of the BMVI. Selected transport infrastructure projectsof the road, rail and waterways are currently carried out as pilot projects. Federal institutions are pursuing an open
exchange of data (OpenBIM). An examination of the application of BIM in building construction projects of theFederal Government with an estimated investment volume of EUR 5 million or higher is demanded by BMI.
Starting in 2020, both ministries will run the national BIM competence center in order to coordinate the BIM
implementation as well as to act as a service hub of BIM-related knowledge and technology transfer.
In addition, guidelines and instructions for applying the BIM methodology are developed and issued by major
German private and public companies. They include BIM execution plans (BEP) and employer’s information
requirements (EIR) as well as legal designs (contract) and tender procedures. Noteworthy here is, for example,
Deutsche Bahn as a major public contractor in the infrastructure sector, which has developed specifications for theapplication of the BIM method for railway projects. The federal project company in the field of road infrastructure,
DEGES, has published guidelines and templates for road construction. Other private companies have proceeded
in a similar manner. These guidelines include specifications for framework conditions, such as processes, roles
and project standards, as well as BIM use cases, such as specifications for collaboration and information
requirements (including platform concepts and integrated communication tools). Specific and crucial roles bothon employer’s and contractor’s side (e.g. BIM manager and BIM coordinator) are defined to guarantee satisfying
project execution due to BIM use.
All sectors require a solid, consistent and state-of-the-art training and education of engineers and specialists. Thisis executed both in the public and the private sector. Legal entities both on national and state level can and mustbe partner of (regional) economies in order to support especially SMEs in the change process. The Institute ofConstruction Management and Digital Engineering (ICoM) is a leading German educator not only on theuniversity level, but also transferring knowledge to working professionals from beginner to expert levels via useof new methodical approaches as well as their digital laboratories (X-Lab). Innovative methods of education (e.g.
blended learning and virtual reality) find their way into the educational programs. Post-educational, certified
courses for professionals and specialists round off academic qualification.

3

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Profile of Prof. Katharina Klemt-Albert
Prof. Dr.-Ing. Katharina Klemt-Albert leads the Institute for Construction Management and Digital Engineering
at Leibniz University Hanover since 2016. Her focus in research and teaching is on digital transformation anddigitalization of the building and construction industry. She started her academic career studying civilengineering at Ruhr University Bochum. In 2001 she received her doctorate with distinction from Technical
University of Darmstadt in cooperation with Northwestern University/USA. Professor Klemt-Albert is anexperienced leader and manager of mega projects in Germany and worldwide. For 14 years, she held positions ofhigh responsibility at Deutsche Bahn AG, most recently as Member of the Board of an international engineering
company with 1,500 employees. Her personal interest has always been the development and implementation oftechnical innovations. Her work focuses on the integration of science and practice. Major research topics at her
institute are GreenBIM, Digital Methodologies in AEC, Digital Construction and Major Projects. Professor
Klemt-Albert is honored as one of Germany‘s Top 25 most influential women in engineering. She won the prize
for excellence in teaching of Leibniz Universität Hannover. She initiated several BIM networks such as the BIM
Cluster Lower Saxony, a statewide association of government, industry and academia. Furthermore, Professor
Klemt-Albert is founder and CEO of albert.ing GmbH, which offers engineering services and consulting indigital construction.

4

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

KEYNOTE LECTURE-3

Creation, Integration and Management of BIM Information
Dr. Jack C. P. Cheng
Associate Professor, Department of Civil and Environmental Engineering, The Hong Kong University of Scienceand Technology, Hong Kong. E-mail: cejcheng@ust.hk
Abstract: Building information modeling (BIM) technology is increasingly used in various architecture,engineering, construction and operations applications. BIM is even compulsory for public projects in somecountries and regions like UK and Hong Kong. This leads to the opportunities yet challenges to the availability,
representation, use and management of building information models. In this presentation, the generation andrepresentation of building information models will be discussed firstly. In specific, model requirement definition
and advanced approaches based on machine learning and AI to automatically generate building information models
from point cloud data will be presented. Secondly, examples and lesson learnt of integrating BIM information withother technologies such as GIS, Internet of Things and robotics will be presented. Thirdly, this presentation will
discuss the storage, sharing and management of building information models and associated information in a multiuser, collaborative environment. Potential opportunities and challenges of relevant technologies and infrastructures
such as blockchain and common data environment will be discussed.

5Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Profile of Dr. Jack C. P. Cheng
Dr. Jack Cheng is an Associate Professor of Civil and Environmental Engineering, Director of the RFID
Center, and Associate Director of the GREAT Smart Cities Institute at the Hong Kong University of Scienceand Technology (HKUST). He obtained his PhD degree from Stanford University. His research interests
include BIM, Internet of Things (IoT), computer vision and deep learning, construction informatics,
construction and facility management, green and low carbon buildings, and sustainable construction. He iscurrently the Chair of the Hong Kong Construction Industry Council (CIC) BIM Standards (Phase 2) Task
Force, Chair of ASCE Global Center for Excellence in Computing, Chairman of Autodesk Industry Advisory
Board (AIAB), President-Elect of ASCE Hong Kong Section, and Honorary Treasurer of Hong Kong
Institution of Building Information Modeling (HKIBIM). He is a Professional Member of HKIBIM, CIC
Certified BIM Manager, and Certified Carbon Auditor Professional (CAP). He has received the Construction
Industry Outstanding Young Person Award in 2019 and the Young BIMer of the Year Award in 2014 fromthe CIC. He has co-authored over 200 referred journal and conference publications.

6Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

AI and Data Analysis

7Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

NLP-BASED METHOD FOR AUTO-CORRECTING PUBLIC CONSTRUCTIONS
DATA
Meng-Lin Yu​1​, Hao-Yung Chan​2​, Meng-Han Tsai​3
1)
2)
3)

Graduate Student, Department of Civil and Construction Engineering, National Taiwan University of Science andTechnology, Taipei, Taiwan. Email: m10705510@mail.ntust.edu.tw
Ph.D. student, Department of Civil and Construction Engineering, National Taiwan University of Science andTechnology, Taiwan. Email: d10705005@mail.ntust.edu.tw
Assistant Professor, Department of Civil and Construction Engineering, National Taiwan University of Science andTechnology, Taiwan. Email: menghan@mail.ntust.edu.tw

Abstract:
This research developed a term dictionary in the construction area​ ​for increasing segmentation quality of
auto-correcting the data in the public construction cost estimate system (PCCES) data. According to theadministrative rules in Taiwan, all the information, such as bidding price, construction time, construction
material, etc., of infrastructure projects need to be recorded into the PCCES. The database with enormous
amounts of historical data allows the government to have statistical analysis and learn from the historicalexperience. However, in the database, most of the data is stored as non-structural formats, mainly texts, whichmakes analyzing those data a tedious and time-consuming work. Therefore, this research aims to structuralize
the non-structural data in the PCCES database through natural language process (NLP) methods. We construct a
dictionary to eliminate inconsistency of terms and common names automatically. This dictionary contains theterms and common names used in the field of civil engineering in Taiwan to facilitate the use of NLP parsing
and classification to correct erroneous data.
Keywords: natural language processing (NLP), public construction cost estimate system (PCCES), Chinesesegmentation
1. INTRODUCTION
According to the regulations of government contracts in Taiwan since 2002, all government publicconstructions require to use PCCES to record names, materials, human resource, quantity, machine type,model number, specifications and price (Archnowledge Inc., 2016; Public Construction Commission,
Executive Yuan, 2016; Public Construction Commission, Executive Yuan, 2017). PCCES assists government
agencies and manufacturers in several works, such as preparing project budget, estimating pricing, changing
design, and comparing funds. PCCES may improve those things mentioned above, standardize the budget,increase the credibility of funds, avoid re-establishment of systems by government agencies, import theperformance of government agencies, etc. At the same time, it stores a lot of public constructions
information.
Despite its good intention, the PCCES database stores a lot of wrong or non-structure data due toseveral causes, including manual typing error, lack of knowledge of the system, existed error, simultaneousmaintenance of the same project by multiple users, having different definitions of the same work item, having
two or more different names referring to the same material, etc. For example, the data should be"(0331026005)​混凝土，預拌，​140kgf/cm2"(Concrete, Premix, 140kgf/cm2) but in real data it may with thewrong unit like"(0331026005)​混凝土，預拌，​140kgf/c​㎡​" or it is a non-structure data like"(0331026005)140​第一型預拌混凝土​"(140premix Portland cement type 1).
In the past, tasks, such as correcting data, statistics, comparing and auditing data, were done
manually. For example, in correcting data, humans could read and understand what the meaning of the nameof the work item and compare with the work item code to make sure if it was correct. However, the manual
work cost an enormous amount of time, and the quality was unstable.
To solve such problems, we need an automated correction toolkit. The development of information
technology (IT) and the improvement of computation in the present have changed how people handle thedata. Nowadays, we develop a more automatic process on computers to replace manual works. In the field ofIT, natural language processing (NLP) enables computers to process human languages. NLP is alreadyapplied in the engineering field (Ryan, 1993; Bai et al., 2012), making NLP a feasible solution to the
problems of PCCES. In this research, we adopted NLP and developed a dictionary which included all theterms and common names to avoid wrong segmentation. This dictionary automatically corrects the wrong
data in the PCCES system database.

8

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2. LITERATURE REVIEWNatural Language Processing (NLP)
Natural language processing (NLP) is a subfield of computer science, information engineering, andartificial intelligence concerned with the interactions between computers and human (natural) languages, inparticular how to program computers to process and analyze large amounts of natural language data. Challenges
in natural language processing frequently involve speech recognition, natural language understanding, andnatural language generation. It is widely used in our daily life, such as Siri and Google assistant. Siri and Google
convert human speech to text and understand the meaning. Data mining from literature text, Siri and Google will
provide us useful data within a few seconds. It helps us get rid of heavy dirty works on text. Those scenarios are
pretty close to our life. But until now, we don't have universal NLP tools to solve all the problems that require
NLP technology. Every problem requires a customized solution to solve it.
If we want to use NLP to solve problems the first thing is to cut sentences to words and then store thecategories and information of the words in a dictionary. The computer can then use an algorithm to compute
those data to understand the meaning (Goldberg and Levy, 2014; Rong, 2016).
Segmentation
Cutting sentences to words is relatively simple in English because the word boundaries are explicit;
English uses spaces to separate words. With this, we can cut words very easily. We can map words between thearticle to know the meaning in the dictionary. Also, English words themselves contain their origins such as
roots, prefixes, and suffixes. Such characteristics enhance the possibility to guess the meaning when
encountering terms which do not exist in the dictionary.
By contrast, in Chinese, this task remains unsolved. First, Chinese characters are derived fromhieroglyphics, and the characteristics are all in the strokes. Second and most importantly, there is no separation
between two Chinese words. For that reason, when we apply NLP in Chinese, we have to cut the sentences first.
If we want to cut the sentence, we need to know which combination of characters is a word. Even though wehave a dictionary, wrong segmentation always happens. For example, a statement such as "if students get good
grades, they will be happy" (如果拿到好成績，學生會開心) after segmentation could become, "if, students,
get good grades, they will be, happy" (如果, 學生, 拿到, 好成績，學生會, 開心). ​"Student" and "student
union" are different. In the civil construction area, wrong segmentation also happens because some materials
have different names. Those different names all refer to the same material. It makes it more challenging to teachthe computer to understand even more difficult (Le and Jeong, 2017; Noy, 2004).
Segmenter
When dealing with natural languages, a segmenter can help obtain a good quality of segmentation.
Segmenters cut input sentences using dictionaries and algorithms and return the result of word segmentation as
the output. In this research, we survey three common Chinese segmenters: ​CKIP​, ​CoreNLP​, and ​Jieba​. The
comparison is shown in Table 1.
CKIP is a segmenter developed by the Academia Sinica, Taiwan (Ma and Chen, 2003). ​CoreNLP isdeveloped by Stanford University, supporting six languages including (Modern Standard) Arabic, (Mainland)
Chinese, French, German, and Spanish (Manning et al., 2014). ​Jieba is a popular open-source project in Python
published on GitHub, allowing users to use it to resolve NLP problems; it only supports Chinese (Sun, 2012).
We believe that ​CKIP may be closer to Taiwanese terms among other segmenters. On the other hand, ​Jieba
supports customized dictionary to expand its vocabulary.
Table 1. Comparison chart of the segmenters

CKIP

CoreNLP

Jieba

Inventor

Chinese knowledge andinformation processing,
Academia Sinica, theNation Academy of
TaiwanThe Stanford Natural
Language ProcessingGroup

Sun Junyi

LicenseCustomized License

GPL v2+

MIT License

Supported Languages

Traditional Chinese

Arabic, Simplified
Chinese, French,
German, and Spanish

Simplified Chinese,
Traditional Chinese

9Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3. OBJECTIVE
This research aims to develop a dictionary for increasing segmentation quality of auto-correcting
PCCES system data. The performance of recent segmenters is not good enough for developing auto-correcting
toolkits. In this research, we not only developed a dictionary which includes all the terms and common names toavoid wrong segmentation but also improved the performance of segmenter. We expect this new segmenter toincrease the quality of auto-correcting PCCES system data. The developed dictionary should eliminate theinconsistency of terms and common names in PCCES data.
4. METHODOLOGY
In this research, we proposed a process of making the dictionary. The dictionary covers terms andcommon names in the civil engineering area. The process is composed of six steps: dictionary initialization, datacollection, preprocessing, segmentation, quality checking, and dictionary updating (Figure 1).
First of all, we made an initial version of the dictionary with a suggested term list in the instruction
manuals of PCCES. Secondly, we obtained real project data from actual practitioners who used the PCCES
system at work and preprocessed the real data. Then, we used the dictionary to segment the real data. After that,
we checked the segmentation quality with specialists. The specialists manually fixed wrong segmentation andadd the new extracted terms into the dictionary. This process repeats until specialists accept the segmentation
quality, then we have a final version of the dictionary.

Figure 1. System process
Dictionary Initialization
The PCCES system has several instruction manuals on its website. We went to the PCCES system
website to download all the documents about materials, these documents include hundreds of suggested terms.
We extracted terms without symbols, units and numerical from these documents and sort by the length of thesewords in descending order.
Data Collection
We collected real data from actual practitioners who use the PCCES system at work. These real data arecloser to daily use than the example provided by the Public Construction Commission (PCC), Executive Yuan,
Taiwan. That is why we do are not only use the example provided by PCC because the example does not havecommon names. Also, terms and common names are the key points of segmentation quality.
Preprocessing
In data preprocessing, we removed and replaced words and symbols because they may lead to wrong
segmentation. Many symbols in Chinese are entirely different from English, such as “​～​”, “​。​”, “​，​”, “​（​”, etc.
We also replace unit symbols to the real letter or number, such as “​㎡​” to “m2” and “​㎏​”(square kg) to “kg”.
Besides, we removed unnecessary prepositions such as “​及​” (and), “​含​” (included). In the real data, there aremany symbols which we have to remove or replace because it would cause wrong segmentation when we
segment real data (Figure 2).

10Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 2. Data reprocessing
Segmentation
We surveyed some segmenters and chose one to segment the real data with the dictionary. The rule ofthe dictionary is to retain the words as close to its original meaning as possible. It sometimes needs to besegmented from the longest form of words. The first word that appears in the dictionary would be used first,
which is why we sort words by the length in descending order at 4.1.
Quality Checking
After we segmented real data, we check the segmentation quality manually with specialists. The
specialists will confirm each term and common name if the term and common name are complete or not in thereal data segmentation results. We will finish this process if the quality is acceptable to the specialists.
Dictionary Updating
After quality checking, if the quality is not acceptable to the specialists, they will manually fix wrong
segmentation and add the new extracted terms into the dictionary. For example, the wrong segment, “​竹節鋼筋​”(deformed bar) becomes “​竹節​, ​鋼筋​”(bamboo joint, rebar). The specialists will pick up these two terms “​竹節​,鋼筋​” and combine them back into “​竹節鋼筋​” and add it into the dictionary. Then, we use the updated
dictionary to segment the real data again.
5. IMPLEMENTATION
Dictionary Initialization
The PCCES system provided several instruction manuals on its website. We visited the PCCES system
website to downloaded all the documents about materials and pick up all the suggested terms from these files.
The example is shown in Table 2. Because the first word appears would be used first in ​Jieba dictionary, wewant to use words as close to their original meanings as possible and sort terms by the length in descendingorder. The example is shown in Table 3.
Table 2. Margins of the manuscript
Document.1

Document.2

緩疑劑

Set Retarder

第​1​型水泥Type 1 Portland
cement減水緩凝劑Water-reducing
retarder

第​4​型水泥Type 4 Portland
cement

高性能減水緩凝劑High-performancewater-reducing
retarder

輸氣第​1A​型水泥

Type 1A Portland
cement

高性能減水劑

High-performancewater reducer

白水泥White cement

Table 3. Margins of the manuscript
Original dictionary

Terms in English

11

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

高性能減水緩凝劑High-performance water-reducing
retarder

輸氣第​1A​型水泥

Type 1A Portland cement

高性能減水劑

High-performance water reducer

減水緩凝劑

Water-reducing retarder

第​1​型水泥Type 1 Portland cement

第​4​型水泥Type 4 Portland cement

緩疑劑

Set Retarder

白水泥White cement

Data Collection
To collect more terms and common names, we collected real data provided by actual practitioners who
work on PCCES. We collected 5847 real data include 417 work items, 5369 materials, 16 machines, 3 humanresources, and 40 others. We found a lot of terms not in suggested terms list, for example, “​宜蘭石​”(Stone from
Yilan). These terms and common names are the key points of segmentation quality.
Preprocessing
Although the Public Construction Commission (PCC) provided the instruction manuals, it is not easy tofollow in practice. There are a lot of Chinese symbols, unit symbols and Roman numbers, such as "。", "㎡", "㎏", and "Ⅱ" in the real data. In data preprocessing, we first replaced all "，" to ", ", unit symbols and Roman
numbers symbols with letters like "㎡", "㎏" and "Ⅱ", then removed Chinese symbols like "。" We also
replaced unnecessary prepositions such as "及" (and), "含" (included) to ", " because we only want terms andcommon names. The example is shown in Table 4.
Table 4. Margins of the manuscript
BeforeReplaceAfter

滲透性底塗層​0.3kg/​㎡㎡​→m2

滲透性底塗層​0.3kg/​m2

產品，結構用混凝土，預拌，​280​㎏​f/​㎝​2㎏​→kg, ​㎝​→cm產品​,​結構用混凝土​,​預拌​,
280​kg​f/​cm​2

粒料，細粒料​。

。​→remove

粒料​,​細粒料

產品，預拌混凝土材料費，​280kgf/cm2​，第​Ⅱ​型水泥

Ⅱ→2產品​,​預拌混凝土材料費​,
280kgf/cm2,​第​2​型水泥產品，預拌混凝土材料費，​210kgf/cm2​，工地交貨(​含​圓柱試驗​)含​→,產品​,​預拌混凝土材料費​,
210kgf/cm2,​工地交貨​(​,​圓柱試驗​)產品​，​粒料​，​極白石，​→,產品​,​粒料​,​極白石

Segmentation
There are four reasons why we chose Jieba. First, the reason for developing Jieba is to solve the Chinese
word segmentation problem in the beginning. Second, Jieba supports Traditional Chinese. Third, Jieba supports
customized dictionaries. Last and the most important is that Jieba is an open-source project that we cancustomize the features we want. The details are shown in Table 5. We used Jieba with the dictionary to segment
the real data. We used this segmenter to segment Table 4, and the expected result to look like Table 6. After thesegmentation, we invited two specialists who both work in the civil engineering area; one of them has beenworking for four years, and the other has been working for 14 years. We provide the segmentation result to thespecialists to check the quality.

12

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 5. Margins of the manuscript

Term

MethodEnglishMandarin

CKIP

CoreNLP

Jieba

Jieba
+Dictionary

Deformedbar

竹節鋼筋

竹節​, ​鋼筋

竹節​, ​鋼筋

竹節​, ​鋼筋

竹節鋼筋

Stone from
Yilan

宜蘭石

宜蘭石

宜蘭石

宜蘭​, ​石

宜蘭石

Spiralcolumn螺箍筋柱螺箍筋柱螺箍筋​, ​柱

螺箍筋柱

螺箍筋柱

Effective
depth

有效深度

有效​, ​深度

有效​, ​深度

有效​, ​深度

有效深度

Table 6. Margins of the manuscript
BeforeAfter

滲透性底塗層​0.3kg/m2

滲透性底塗層​, 0.3kg/m2

產品​,​結構用混凝土​,​預拌​,
280kgf/cm2

產品​, ​結構用混凝土​, ​預拌​,
280kgf/cm2

粒料​,​細粒料

粒料​, ​細粒料

產品​,​預拌混凝土材料費​,
280kgf/cm2,​第​2​型水泥產品​, ​預拌混凝土材料費​,
280kgf/cm2, ​第​2​型水泥產品​,​預拌混凝土材料費​,
210kgf/cm2,​工地交貨​(,​圓柱試驗​)產品​, ​預拌混凝土材料費​,
210kgf/cm2, ​工地交貨​, ​圓柱試驗產品​,​粒料​,​極白石產品​, ​粒料​, ​極白​, ​石

Quality Checking
After we used ​Jieba with the dictionary to segmented real data, we showed the segmentation results tothe specialists to check the quality. This process stops when the quality is acceptable to the specialists. If there is
any wrong segmentation such as the example shown in Table 6 where “​極白​, ​石​” should be “​極白石​“, thespecialists then add “​極​白石​“ to the dictionary. We segmented the real data again using ​Jieba with the updated
dictionary. The result of the updated dictionary is shown in Table 7.
Table 7. Margins of the manuscript
BeforeAfter產品​, ​粒料​, ​極白​, ​石(product, pellet, white, rock)產品​, ​粒料​, ​極白石
(product, pellet, white rock)

6. VALIDATION
To evaluate the quality of the developed dictionary, we compared the results before and after using thedictionary to cut sentences by applying ​word2vec (Goldberg and Levy, 2014; Rong, 2016) to calculate and findthe top 10 items that are the closest to cement and concrete. Also, we used a data visualization tool to illustrate13

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019the result of ​word2vec to show the distribution of words. The validation results are shown in Table 8 and Figure
3. After applying the developed dictionary, the distance between cement and concrete in the vector space has
increased from about 0.9989 to 0.995; the density of distribution has significantly improved.
7. RESULT AND DISCUSSION
The distance between cement and concrete becomes farther after applying the developed dictionary.
This result is counter-intuitive because ​they are often used interchangeably in common situations, and cement is
an ingredient of concrete. However, it may be proof that the correctness of the word segmenter that has been
improved. More words that only have a relationship with one of them appear, which makes the distance farther.
Also, the distance only increased by 0.389%, which means that they are still very close relatives.
On the other hand, after using the dictionary, the quality of the word segmentation is significantly
improved, shown in Table 8. and Figure 3., The distribution of words is also denser in the ​word2vec vector
space, shown in Table 9. and Figure 3.
Table 8. distance between cement and concrete

Without a dictionary

With a dictionary

0.9989132343

0.9950177034

Figure 3. Words distribution in the word2vec vector space before and after. The left figure shows the
distribution before applying the developed dictionary; the right figure shows the distribution after applying thedictionary.
Table 9. Margins of the manuscript
Without a dictionary
水泥​ (cement)

混凝土​ (concrete)

CM2 (square centimeter)

0.999641

劑​ (medicament)

0.9996675

交貨​ (delivery)

0.999611澆置​ (pouring)

0.9996522

工地(construction site)

0.999581

化學​ (chemistry)

0.9996463

第​ (prefix before a number)

0.999526、​ (fullwidth comma)

0.999641

型​ (type)0.999493

回填​ (backfill)0.9996373

14

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

140KGF (140 kilogram-force)

0.99949）​ (fullwidth left parenthesis)0.9996329

費​ (cost)

0.999488機拌​ (machine mixing)

0.9996316

280KGF (280 kilogram-force)

0.999462

10CM (10 centimeter)

0.9996298

350KGF (350 kilogram-force)

0.999397

及​ (and)0.9996291

80KGF (80 kilogram-force)

0.99939

控制性​ (controllability)

0.9996286

With a dictionary
水泥​ (cement)

混凝土​(concrete)

材料費​ (material fee)

0.9971849產品​ (product)

0.9972743

工地​ (construction site)

0.9971505

CM3 (cubic centimeters)

0.9972516

KG (kilogram)

0.9970934

水泥​ (cement)

0.9971778產品​ (product)

0.9970929

水中​ (in the water)

0.997173280

0.9970812

交貨​ (delivery)

0.9971293

CM2 (square centimeter)

0.9970585

預拌混凝土(ready mixed concrete)

0.997112

預拌混凝土​ (ready mixed concrete)

0.9970556

第一型水泥(portland cement type I)

0.9970986

水中​ (in the water)

0.9970452

KGF (kilogram-force)

0.9970934

140

0.9970348

MM (millimeter)

0.9970911

設計強度(design strength of concrete)

0.9970232

工地​ (construction site)

0.9970759

8. CONCLUSIONSIn this research, we developed a dictionary for improving the segmentation quality of auto-correcting
PCCES system data and proposed a process of making a dictionary that covered terms and common names in
the civil engineering area. We used original suggested terms to construct the first version dictionary andprocessed the real data to obtain daily used common names before we segmented the real data. We checked thequality of segmentation and repeated the whole process until the quality was acceptable to specialists; then usedthe ​word2vec to validate the results. After using the dictionary, we found that the segmentation quality wassignificantly improved, and words in ​word2vec vector space were much closer than before. However, ourcurrent implementation cannot automatically recognize new terms or common names. We plan in the future to
implement an automatic discrimination system for new terms and common names.
ACKNOWLEDGMENTS
We thank Dr. Ming-Da Tsai, Mr. Liang-Yuan Liu and Mr. Yi-Hao Lin for their professionalsuggestions. This work was financially supported by the Taiwan Building Technology Center from The
Featured Areas Research Center Program within the framework of the Higher Education Sprout Project by the
Ministry of Education in Taiwan.
REFERENCES
Archnowledge Inc. (2016). Budget preparation manual of Public Construction Cost Estimate System (version
Win4.3) (in Chinese). Retrieved from Public Construction Commission, Executive Yuan website:
https://pcces.pcc.gov.tw/csinew/Default.aspx?FunID=Fun_12_11&SearchType=E
Bai, L., Wang, J., Jiang, Y., and Huang, D. (2012). Improved Hybrid Differential Evolution-Estimation ofDistribution Algorithm with Feasibility Rules for NLP/MINLP Engineering Optimization Problems,
Chinese Journal of Chemical Engineering, 20 (6), 1074-1080.
Day, M.-Y. and Lee, C.-C. (2016). Deep Learning for Financial Sentiment Analysis on Finance News Providers,2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining(ASONAM), San Francisco, CA, USA, pp.1127-1134.

15

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019Goldberg, Y. and Levy, O. (2014). word2vec Explained: deriving Mikolov et al.'s negative-sampling
word-embedding method. Retrieved from arXiv website:
https://arxiv.org/abs/1402.3722
Ma, W.-Y. and Chen, K.-J. (2003). Introduction to CKIP Chinese Word Segmentation System for the FirstInternational Chinese Word Segmentation Bakeoff, Proceedings of ACL, Second SIGHAN Workshop
on Chinese Language Processing, Sapporo, Japan, pp.168-171.
Le. T., and Jeong, H. D. (2017). NLP-Based Approach to Semantic Classification of HeterogeneousTransportation Asset Data Terminology, Journal of Computing in Civil Engineering, 31(6).
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., and McClosky, D. (2014). The Stanford
CoreNLP Natural Language Processing Toolkit, Proceedings of the 52nd Annual Meeting of the
Association for Computational Linguistics: System Demonstrations, Baltimore, Maryland, USA,
pp.55-60.
Noy, N. F. (2004). Semantic integration: A survey of ontology-based approaches, ACM SIGMOD Record,
33(4), 65-70.
Public Construction Commission, Executive Yuan. (2016). Report of information service procurement ofplanning and construction of the second generation public works price database system (in Chinese).
Public Construction Commission, Executive Yuan.ISBN: 9789860492149.
Public Construction Commission, Executive Yuan. (2017). Report of expansion information service
procurement of planning and construction of the second generation public works price database system(in Chinese), Public Construction Commission, Executive Yuan. ISBN: 9789860536751.
Rong, X. (2016). word2vec Parameter Learning Explained (v4). Retrieved from arXiv website:
https://arxiv.org/abs/1411.2738v4
Ryan, K. (1993). The role of natural language in requirements engineering, Proceedings of the IEEE
International Symposium on Requirements Engineering, San Diego, CA, USA, pp.240-242.
Sun, Junyi. (2012). Jieba. Retrieved from GitHub website:
https://github.com/fxsjy/jieba

16Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

ANALYSIS OF POWER USAGE AND BUILDING RESIDENTS RELATIONSHIPSFOR ENERGY SAVINGS USING SOCIAL NETWORK ANALYSIS
Ru-Guan Wang1 and Chien-Cheng Chou2
1) Graduate Research Assistant, Information Technology for Disaster Prevention Program, Department of Civil Engineering,
National Central University, Taoyuan, Taiwan. Email: rubyw666@gmail.com
2) Professor, Information Technology for Disaster Prevention Program, Department of Civil Engineering, National Central
University, Taoyuan, Taiwan. Email: ccchou@ncu.edu.tw

Abstract: Reducing carbon footprints in the building sector can be achieved by altering power consumption
behavior of building residents. Due to the influence of today’s declining birth rate and population aging, thestructure of human society is changed, requiring the identification of key persons active in a community topersuade the others into saving electricity. This research aims at applying the technique of social network analysis(SNA) to a publicly available smart meter data set for building residents in Germany. Traditionally the head of acommunity can serve as the role of broadcasting energy-saving information, although its effectiveness varies withdifferent circumstances. In the proposed SNA-based approach, the German data set is firstly examined and preprocessed, such as augmenting building occupancy data and relationships among residents. Then, different SNA
indexes are explored in order to derive a generalized procedure for such identification of key persons. Moresustainable societies can be established if key persons of a community can be identified and get involved by using
the proposed approach. Energy-saving information specific to each type of home appliance can be broadcast
effectively and efficiently so that building residents can persuade easily.
Keywords: Social network analysis; smart meter data analytics; energy conservation.
1. INTRODUCTION
As scholars and policy makers are becoming increasingly interested in the establishment of a sustainable
society, reducing its carbon footprints such as implementing power saving tips has been regarded as one of themain tasks (Costanzo et al. 1986; Chen et al. 2012). The earth gets gradually warmer, the sea level is constantly
rising, the desert area is expanding annually, and even the emergence and spread of new diseases not only bringthreats to the environment but have shown the need of a systematic method to investigate the interactions amongpeople causing the aforementioned problems. Human beings almost change every aspect of the earth. It seems thatthe earth is signaling warnings to people. Therefore, in recent years, such sustainability issues are valued bygovernment around the world. In addition to pursuing the quality of people's living environment, it also needs togive friendly treatment for all animals and plants, reflecting the performance of loving the earth. Everyone changes
his or her life style a little, which jointly can contribute to the overall protection of the earth.
It has been recognized in the literature that reducing carbon footprints in the building sector can beachieved by altering power consumption behavior of building residents (Azar & Menassa, 2014). Although there
are numerous energy-saving devices or approaches designed for the building sector, they may all become fruitless
if building residents who actually consume energy disregard such energy-saving tips or instructions (Jain et al.
2013). Due to the influence of today’s declining birth rate and population aging, the structure of human society ischanged, requiring the identification of key persons active in such campaigns advocating energy saving orconservation. Therefore, such key persons of a community may collectively persuade the others into changing
their energy usage behavior, creating a positive feedback loop to promote further energy savings.
Before the technique of social network analysis (SNA) has yet to appear, it is difficult to analyze theinteraction among individuals of a community. SNA originates from the adaptive network used in physics. Basedon mathematical graph theories and other quantitative analysis methods, SNA can be used to help understandlarge-scale social systems. Currently, SNA has been employed in the fields of sociology, psychology, anthropology,
and several science-related domains since the 1970s (Mankoff et al. 2010). By definitions, a social network is used
to represent the exchange of resources between human beings, and a resource can represent a message, tangible orintangible object (Du et al., 2016). In this research, A building resident is modeled as a node in the network, andhis or her influence on power-saving related issues for the other resident is seen as a link or flow between the two
nodes. The following sections describe the use of SNA for a publicly available smart meter data set for building
residents in Germany. Section 2 presents the data set and SNA. Section 3 shows the SNA results of the data set,
while Section 4 contains further discussions. Section 5 lists relevant conclusions and future research directions.
2. METHOD
2.1 Smart meter data set and overall structure of the social network
The data set utilized contains power consumption or solar generation data from 11 households in southern
Germany, and there are approximately 16,000 records, each with cumulative, kilowatt-hour values for the17Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

household and appliance levels. The collection time period is between 2015 and 2017, with some buildings locatedin urban areas while some in suburbs (Minde, 2017).
In SNA, a weight is defined as the degree of a resource needed for the link between two nodes. In theliterature, weight definitions are often customized to better fit research needs, e.g., the number of friends a person
can have as a weight for analyzing communication issues. Hence, this research assumes that node Ni to node Nj is
weighted, and its weight Wij(t) is defined as the intensity of one node affecting the other in a certain time interval
about power usage issues. The value of Wij(t) can be 0 to 3, as shown in Equation (1), where Wij(t) not necessarilyequal to Wji(t), i.e., weights are directional. If a resident initiating a weight link does not have any influence on theother resident’s power usage, probably because of no friendship, its weight is zero, and such a link can be discarded
as well. Meanwhile, three methods, introduced in Section 2.2, are employed to assign an appropriate value (1-3)
to each weight. Briefly, both the first and the third methods use survey questions to obtain relationships amongresidents, and because of the nature of questionnaires, all such weights are qualitative and sometimes containmisleading indication. Thus, it is suggested to seek for other sources, such as cell phones usage data, to support
the survey results (Peschiera & Taylor, 2012).
0: there is no influence between the two residents ′ electricity use behavior;
1.0 − 1.9: the degress of influence from the first resident to the second via survey;
𝑊𝑖𝑗 (𝑡) {2.0 − 2.9: the correlation coefficient of actual power usage between the two residents;
3: the first resident is the elected leader of a group;(1)

The second method calculates correlation coefficients of actual electricity consumption data for any pairof two households. Figure 1 shows the original data format (CSV) of the data set, and a simple transformation
function (depicted in the D9 cell) is used to calculate each incremental power consumption value between two
consecutive time intervals. Since the second method is based on actual data, such weights are more reliable,compared with the first and third methods, and are used extensively to form SNA relationships for the data set.
Totally, 61.5% of the weight links are determined by the second method.

Figure 1. Actual power consumption data: from cumulative to incremental
2.2

Design of weights allocation
The first method, basically a survey, is designed to allocate 1.0 to 1.9 to weights for a group of residents.
The survey asks for the list of a given resident’s all friends that can affect his or her power usage. For example, if
a resident thinks that some of the neighbors or friends can affect how he or she uses electricity, several weightlinks will be created from the set of the residents to the given resident. After all weight links have been identified,a typical normalization procedure can be utilized to identify the top ten weights, where 1.9 assigned to the resident
with the most number of outgoing weight links, 1.8 assigned to the second most, and so on.
The second method is designed to allocate 2.0 to 2.9 to weights and is based on the calculation ofcorrelation coefficients for power consumption actual data for any pair of two households. Since a correlation
coefficient ranges between -1.0 and 1.0 and is in fact bi-directional, the value needs to be transformed and mapped
into the value range of SNA weights. Additionally, because each household of the data set can be classified aseither urban or suburban areas, there will be 𝐶2𝑛 correlation coefficients in each group, where n is the number ofhouseholds in urban or suburban areas. After all correlation coefficients have been calculated in each group, thetop ten pairs are selected, each assigned from 2.9 to 2.0 accordingly. Note that one pair consists of two householdswith two links, each with the same weight to show the bi-directional relationship. The higher the correlation
coefficient, the more likely that the two households have similar power usage patterns. For example, if thecorrelation coefficient is close to 1.0, at 7AM, the two families all consumed 70W and at 8AM, they both consumed
80W, etc. Although such relationships are not indicated by human beings, this research assumes that when a powersaving tip may work for the two households having such relationships, it is easier to collectively change their18

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

power usage behavior due to the same patterns they share (Kamilaris et al. 2013).
In addition, it is possible to have a relationship between one resident in urban areas and another resident
in suburbs. To consider this phenomenon, the aforementioned method is utilized again to identify top ten
relationships between the urban and suburban groups. In other words, if one resident of the urban group shares themost similar power usage patterns with another resident of the suburban group, a link is created with the weightassigned to 2.5. Again, if such the relationship reflects the second most similar power usage patterns, a link iscreated with the weight assigned to 2.4, and so on. It should be noted that such weight values are generally lessthan the values created by the second method, which implies that it would be easier to change power usage
behaviors of the residents living in the same area.
The third method uses another survey to simply identify the leader of a community. Usually, such a leader
is responsible for all administrative affairs for the community and is elected every 2-4 years. This research assumes
that if a resident can correctly identify the leader, a link will be created from the leader to the resident with theweight equal to 3, which implies that the leader can affect power usage behavior of the resident.
Finally, in addition to weights determination, each household can be associated with several attributes,
each indicating how a specific appliance consumes electricity. In each such attribute, three levels (1-3) are used todenote the degree of electricity used for the given appliance. This research assumes that 1 is assigned to thehouseholds who consume more electricity of the given appliance, considering total average energy usage per day.
2.3

SNA indexes
Traditionally in SNA, the characteristics of nodes and links can be used to interpret many social
phenomena. In addition to establishing weight links between households, some attributes data may increase
discrimination and then grasp node and link characteristics of the overall network. In this section, the following
three SNA indexes are presented and applied to the data set: degree centrality, closeness centrality, betweenness
centrality (Ekpenyong et al. 2014; Ekpenyong et al. 2015).
Degree centrality refers to the number of nodes connected to a node, i.e., the number of links as shown inEquation (2) and Figure 2. However, in a directional network, the centrality of in-degree and out-degree must be
calculated separately. As shown in Figure 2, in-degree is used to indicate the number of incoming links pointing
to a given node, and out-degree is used to indicate the number of outgoing links that a given node extends.
𝐶𝐷 (𝑣) = 𝑑𝑒𝑔(𝑣)(2)Figure 2. In-degree and out-degree diagram
Closeness centrality is defined as a measure of centrality in a network, calculated as the reciprocal of thesum of the length of each shortest path (denoted as d) between a given node (denoted as x) and all other nodes(denoted as y), as shown in Equation (3). If a node has the highest closeness centrality value, it is geometricallylocated at the center of the network. Additionally, this index does not necessarily mean the so-called core node ofa network, which possesses the maximal number of links. Basically, the smaller the sum is, the shorter the pathfrom this node to all other nodes, and the bigger the closeness centrality. Note that this index can be used to show
the willingness to convey information to different groups of nodes.
1
C(𝑥) = ∑(3)𝑦 𝑑(𝑦,𝑥)

Betweenness centrality refers to the number of paths that pass through a given node, provided that there
are many paths in the network. As shown in Equation (4) and Figure 3, 𝜎𝑠𝑡 represents the number of shortest
paths between nodes s and t, and 𝜎𝑠𝑡 (𝑣) is the number of such paths passing through node v. The higher this index,
the more important role it may play in the communication aspect of the network, which can be regarded as a bridgefrom one node to another. Note that the betweenness centrality of a node scales with the number of pairs of nodes
as implied by the summation.
𝜎 (𝑣)C𝐵 (𝑣) = ∑𝑠≠𝑣≠𝑡 𝑠𝑡
(4)
𝜎𝑠𝑡

19

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 3. Betweenness centrality
3. SNA RESULTS
3.1 Overall structure
Two common SNA tools, UCINET and NetDraw (Borgatti et al. 2002), are utilized to examine the data
set. Firstly, the region type of each household is selected so as to split the data set into two groups: urban andsuburban. As shown in Figure 4, each household displayed as a node will be marked as blue if it is located in urban
areas. A household marked as green means it is located in suburban areas. Each line with an arrow and a valuemeans a weight link for SNA.

Figure 4. SNA-based overall structure of the data set: urban households marked as blue and suburban households
marked as green
3.2

Degree centrality
Using the data set, degree centrality generally can express the number of contacts between a household
and others. Because a directional network is used in this study, it is necessary to calculate both in-degree centrality
and out-degree centrality. As listed in Figure 5, one record consists of node ID, out-degree and in-degree values.
These records are sorted according to out-degree centrality. In principle, the order of out-degree values is not
related to the order of in-degree values. Figures 6 and 7 show the visualization results of degree centrality. In
Figure 6, the larger the node, the higher the in-degree centrality value, which means that the outside world is more
likely to affect these larger nodes. Similarly, in Figure 7, the larger the node, the higher the out-degree centrality
value, which means that these larger nodes are more likely to affect the outside world.

Figure 5. Degree centrality analysis results

20Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 6. In-degree centrality structure

Figure 7. Out-degree centrality structure
3.3

Closeness centrality
Close centrality can be used to analyze the degree of interaction between a household and all householdswith different attributes. Because this study uses a directional network, it is necessary to calculate the two indexes,
in-closeness and out-closeness. Figure 8 lists the two index values based on the in-closeness order. As shown in
Figure 9, the outside world with different attributes most easily affects Nodes 1, 7, 8, and et al. Similarly, the incloseness order is not necessarily the same as the out-closeness order. Figure 10 shows that Nodes 1, 2 and 7 are
the top three and are more likely to affect the outside world with different attributes. In other words, if a household
does not frequently contact other households, using Nodes 1, 2, 7, and et al. to communicate with the family is
still easier to persuade them to save energy.

Figure 8. Closeness centrality analysis results

21

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 9. In-closeness centrality structure

Figure 10. Out-closeness centrality structure
3.4

Betweenness centrality
Betweenness centrality implies that how good a given household is at socializing. Using UCINET, theoutput results consist of two columns, as shown in Figure 11. The first column, Betweenness, contains the valueof each betweenness centrality without performing normalization, while the second column, nBetweenness,
contains the normalized values. In fact, the rankings of the two columns are the same, and Figure 12 shows thevisualization structure based on the first column. Basically, the larger the node, the better the betweenness
centrality. In other words, all the messages in the community may rely on the node with the largest betweenness
centrality, i.e., Node 1, in order to broadcast information. Typically, this node is the elected leader or opinion
leader of a community.

Figure 11. Betweenness centrality analysis results

22Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 12. Betweenness centrality structure
3.5

Cluster analysis based on SNA
When there are many households in a community, in addition to finding one or two leaders to promoteenergy-saving practices, the community may also need several facilitators to help convince and broadcast energysaving information. Since each node in a SNA network can have attributes, the so-called clustering operation
against selected attributes can be performed first to generate several groups. Then, the traditional SNA operations
can be performed for each group. For example, the previous data set is split into urban and suburban areas. The
same data set can be split into three groups (low, medium and high) based the amount of energy usage of heaters.
Indeed, previous researchers have combined two or more attributes to split their data sets into a hierarchy so thattraditional SNA operations can be applied.
Figure 13 shows the results of the proposed SNA clustering analysis. The blue nodes are in urban areas
and the green nodes are in suburbs. The larger the nodes, the better the betweenness centrality. The thicker thenode border, the better the out-closeness centrality. Using the NetDraw tool, one can specify the Scrunch factor
equal to 2, and the tool shows two groups: the left part of Figure 13 contains Nodes 8, 11, 10, 7, 9, 3, 12, and theright part of Figure 13 contains Nodes 2, 4, 5, 1, 6. As shown in Figure 13, it seems that Node 7 is the leader ofthe left group. Because a leader cannot play two roles, Nodes 3, 6, 9, 10 can be selected one to play the facilitatorrole, which means Nodes 3, 6, 9, 10 have a good closeness centrality value, implying that all messages should passthrough this node. Again, for the right group of Figure 13, Node 1 can serve as leaders, and Nodes 1, 2 have agood closeness centrality value, while Node 2 seems to be the best facilitator. Because a leader cannot play tworoles.

Figure 13. Cluster analysis of complete communities

23

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

4. DISCUSSION
The closeness centrality and betweenness centrality in SNA has their own characteristics. One can use
cluster analysis to dynamically form several groups, each with special characteristics, and then apply SNA to eachsuch group. For example, the cluster analysis is performed first, and all households with high electrical heater
consumption can be obtained. Then, using SNA can identify both closeness and betweenness centralities, whichcorrespond to leader(s) and facilitator(s) of a group. Using closeness centrality means finding a leader to have theshortest distance from the leader to all other nodes in the network. Such leaders possess better communication
channels to all households of a community. Meanwhile, using betweenness centrality can identify some nodes
where almost all messages pass through. It means that this household is suitable for disseminating knowledge, butthere may be no communication channel with all kinds of people.
5. CONCLUSIONS
Through the analysis of social network analysis, the degree of contact between the households and thevisualization of the electricity habits of the households can effectively find key people in the community andspread the energy-saving information, in addition to the influence and imitation willingness of the appliance. It
can also achieve the effect of saving time and cost. It is not necessary to carry out training or monetary consumption
and induction for each family but to use the spread of energy conservation and carbon reduction issues to deepen
the people's concept and importance on this issue. One of the goals of saving money is that the entire communityor the entire community can effectively improve their electricity habits and the continuous progress among theresidents will lead to a better and more eco-friendly world.
ACKNOWLEDGMENTS
The research was supported in part by the Ministry of Science and Technology of Taiwan under Grant
MOST108-2623-E-008-005-D.
REFERENCES
Azar, E. and Menassa, C.C. (2014). Framework to Evaluate Energy-Saving Potential from Occupancy
Interventions in Typical Commercial Buildings in the United States, Journal of Computing in Civil
Engineering, 28(1), 63-78.
Borgatti, S.P., Everett, M.G., and Freeman, L.C. (2002). Ucinet for Windows: Software for Social Network
Analysis. Analytic Technologies, Harvard, Massachusetts, USA.
Chen, J., Taylor, J.E., and Wei, H.H. (2012). Modeling building occupant network energy consumption decisionmaking: The interplay between network structure and conservation, Energy and Buildings, 47, 515-524.
Costanzo, M., Archer, D., Aronson, E., and Pettigrew, T. (1986). Energy Conservation Behavior: The Difficult
Path from Information to Action, American Psychologist, 41(5), 521-528.
Du, F., Zhang, J., Li, H., Yan, J., Galloway, S., and Lo, K.L. (2016). Modelling the impact of social network onenergy savings, Applied Energy, 178, 56-65.
Ekpenyong, U.E., Zhang, J., and Xia, X. (2014). Mathematical modelling for the social impact to energy
efficiency savings, Energy and Buildings, 84, 344-351.
Ekpenyong, U.E., Zhang, J., and Xia, X. (2015). How information propagation in social networks can improveenergy savings based on time of use tariff, Sustainable Cities and Society, 19, 26-33.
Jain, R.K., Gulbinas, R., Taylor, J.E., and Culligan, P.J. (2013). Can social influence drive energy savings?
Detecting the impact of social influence on the energy consumption behavior of networked users
exposed to normative eco-feedback, Energy and Buildings, 66, 119-127.
Kamilaris, A., Taliadoros, G., and Pitsillides, A. (2013). Social Electricity: When Awareness about Electricity
Becomes Social. Retrieved from ERCIM News website: https://ercim-news.ercim.eu/en93/ri/socialelectricity-when-awareness-about-electricity-becomes-social
Mankoff, J., Fussell, S.R., 3, Dillahunt, T., Glaves, R., Grevet, C., Johnson, M., Matthews, D., Matthews, H.S.,
McGuire, R., Thompson, R., Shick, A., and Setlock, L. (2010). StepGreen.org: Increasing Energy
Saving Behaviors via Social Networks, Proceedings of the Fourth International Association for the
Advancement of Artificial Intelligence (AAAI) Conference on Weblogs and Social Media, Washington,
D.C., USA.
Minde, A. (2017). Household Data. Retrieved from Open Power System Data Platform website:
https://doi.org/10.25832/household_data/2017-11-10
Peschiera, G. and Taylor, J.E. (2012). The impact of peer network position on electricity consumption inbuilding occupant networks utilizing energy feedback systems, Energy and Buildings, 49, 584-590.

24

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

AS-BUILT DETECTION OF STEEL FRAME STRUCTURE
USING DEEP LEARNING
Ryu Izutsu1, Nobuyoshi Yabuki2, and Tomohiro Fukuda3
1) Master Course Student, Division of Sustainable Energy and Environmental Engineering, Osaka University, Suita, Japan.
Email: izutsu@it.see.eng.osaka-u.ac.jp
2) Ph.D., Prof., Division of Sustainable Energy and Environmental Engineering, Osaka University, Suita, Japan. Email:yabuki@see.eng.osaka-u.ac.jp
3) Ph.D., Assoc. Prof., Division of Sustainable Energy and Environmental Engineering, Osaka University, Suita, Japan.
Email: fukuda@see.eng.osaka-u.ac.jp

Abstract: At the construction site, as-built management is generally performed by taking pictures andcomparing them with drawings or Building Information Modeling (BIM) models. Since this work is
time-consuming and prone to human error, a more accurate and efficient method of capturing the progress isdesired. The purpose of this research is to construct a system that can efficiently capture the progress of theconstruction by detecting each structural steel frame component such as a beam and a column under construction
from images taken by a camera. First, we developed a Convolutional Neural Network (CNN) that could detect
structural steel frame components under construction from images by fine-tuning the existing Object Detectionand Segmentation CNNs. Next, we constructed a system that can capture each structural steel frame member
from an image by integrating two constructed CNN models. Finally, we conducted accuracy verification andevaluated the developed system.
Keywords:

Convolutional Neural Network, Deep Learning, As-built Detection, Steel Frame, Segmentation.

1. INTRODUCTION
In recent years, visual capturing of the construction progress has become possible by using thephotographs taken at the construction site, detailed drawings, and BIM model of the construction site. However,this work is not only time-consuming but also prone to human error. Therefore, an automatic system is desired tocapture construction progress more accurately and efficiently.
Recently, the technology of object detection using deep learning has been developed. The recognition
accuracy of objects using deep learning has been developed with the advent of Caltech 101 (Fei-Fei et al., 2007)
which is a data set of digital images and ImageNet (Deng et al., 2009) which is an image database. In ImageNet
Large Scale Visual Recognition Challenge (ILSVRC) which is a large-scale object recognition competition thathas been held since 2010, the object detection accuracy using deep learning has been higher than humandetection accuracy since 2015. Researchers have been developing advanced detection systems using thedetection results.
In this research, we constructed a CNN that can detect each structural member such as a beam, column,
and joint under construction by fine-tuning on the existing object detection and segmentation CNNs. Then, wedeveloped a system that can perform object detection and segmentation for steel structures by integrating two
CNN models. First, since the actual construction site is often covered with scaffolds and soundproof sheets, asteel frame scaled model was made and photographed to prepare an image dataset for deep learning. Next, inorder to detect structural members using deep learning, we changed the training weights of existing deeplearning models such as YOLO (You Only Look Once) (Redmon et al., 2016) and U-Net (Ronneberger et al.,2015) by fine-tuning using model photos. Then, we develop a system that can detect specified target structuresfrom images by integrating two CNN models. Finally, we conducted accuracy verification and evaluated thedeveloped system.
2. LITERATURE REVIEW
2.1 Object Detection Using Deep Learning
Currently, many object detection algorithms using networks similar to Region-CNN (R-CNN) (Gidaris& Komodakis, 2015) have been proposed. First, R-CNN is one of the state-of-the-art CNN-based deep learning
object detection approaches. This network resized the input image to CNN and calculated the number of featuresby extracting about 2000 candidates of the region in which the object appears in the input image. Next, Faster
R-CNN (Ren et al., 2015) with faster inference speed and learning speed than R-CNN was announced. The
speedup was achieved by incorporating the Region Proposal, which had been time-consuming in previous neuralnetworks, into CNN. In addition, Faster R-CNN uses a learning technology called Multi-task loss and has an
end-to-end structure that allows the entire model to learn. As a representative deep learning neural network
constructed using the same idea as the R-CNN algorithm for object detection, YOLO and Single Shot Detector(SSD) (Liu et al., 2016) was developed. YOLO can be trained with the same end-to-end structure as Faster
R-CNN. Furthermore, the conventional method is very time-consuming since it focuses on generated object

25

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

proposals (Region Proposals). YOLO can detect in real time since it can be detected by looking at the entire
image only once. However, the detection accuracy of the model published in 2016 is low, and the accuracy rateof the object detection position is also low. This problem is caused by various limitations such as the limit of thenumber of detectable objects and only one class could be identified in the grid. In order to solve the above
problems, YOLOv3 (Redmon & Farhadi, 2018) was announced. This deep learning model not only solved theabove problems but also was able to cope with the scales of different objects present in the image. It helps toimprove the accuracy of object detection.
2.2

Related Research in Detecting Building Components
While automation is attempted in various fields, there are technological researches that automate inprogress and production management in construction. At the construction site, a system has been proposed torecognize the completion part of the construction by using object recognition from the captured image data andautomatically recognizes three-dimensional objects (Fathi et al., 2015). However, this system can only confirm
the installation situation on the image, and the recognition accuracy is not high performance.
Also, management systems using three-dimensional models have been proposed. A method has been
proposed for detecting structural members such as slabs and girders in existing bridges from point cloud data bysegmenting and constructing a three-dimensional model from the detection results (Lu et al., 2019). This method
makes it possible to construct a three-dimensional model efficiently from the point cloud. However, it is notpossible to detect concrete bridges or truss bridges with complicated geometrical shapes. Furthermore, it isshown that the detection performance of structural members is affected when the point cloud data is distributed
at narrow intervals and unevenly. In addition, in order to create a detailed BIM of an existing facility, a method
has been proposed in which three-dimensional measurement values are obtained as point cloud using a laser
scanner, and BIM model of a building is made from the obtained point cloud (Tang et al., 2010). This method
can be considered as an efficient BIM creation algorithm since it can construct BIM models of existing structures.
However, in this method, verification is performed on only simple planes. Therefore, there are problems to besolved, such as occlusion and modeling of complex structures. In recent years, with the development ofcomputer vision technology, it has become possible to automate human work. A system was developed toautomatically detect structural members in a room by utilizing two-dimensional image data (Hamledari et al.,2017). However, the tasks that can be performed by computer vision technology are limited, and there is a needfor a system that can perform various detections with high accuracy.
3. PROPOSED METHOD
Figure 1 shows an overview of the proposed system processes. The proposed system is able to detect asteel structure by creating a training data-set, using a scaled steel frame model and performing fine-tuning on
CNN. In this paper, we use two CNNs to detect the steel structure. The reason is that it takes much time tocreate training data when using a recent segmentation system such as Mask R-CNN. Therefore, we reduced
the time to create training data by combining YOLOv3, which makes it easy to create learning data such as
object detection, and U-Net, which can detect only the shape of the object by segmentation. In addition, it
is difficult to detect with Mask R-CNN because steel structures such as joints are small in the image. Figure
2 describes the procedure for detecting actual steel structures. Figure 2 (a) shows the method of the proposedsystem which confirms each structural steel frame member, such as a beam, column, and joint, from the image.
The image of the actual steel frame structure is input, and detection is performed for each structural steel framecomponent. Thereafter, regional segmentation is performed by superimposing the segmented results. This
superimposing is done by combining the deep learning model results and the detection results. Figure 2 (b)shows the method of the as-built detection system. The dimensions of the photos that were taken at theconstruction site are then changed to 512 x 512 pixels. Thereafter, completed construction aspects in the image
are detected using a combination of the learned CNN and a mask of the image. The result of the mask image isused for area division by returning the created mask image to the original image size.
3.1

Object Detection Using a Scaled Steel Frame Model
The objective of the proposed system is to detect each member in a structure in order to track
construction. However, building structures cannot be detected using existing image databases, such as ImageNet.
Since ImageNet does not include images of structural steel frame components in its database. To resolve thisissue, we attempted to make our own image data to be used as the data set for learning and these images would
be from the actual construction site. However, scaffolds and soundproofing sheets at most construction sites
conceal the building, this makes it is difficult to get direct photographs of the columns and beams needed for thisstudy. To solve this, digital images are taken of a 1:30 scaled steel frame model. The obtained images are thenused as a training data set. We found that it was possible to detect structural steel frame members such as a beam,column, and joint of steel frame structures in the actual construction site by fine-tuning with it this data set. 2000
images of the steel frame model were prepared as training image data sets and 400 of them were selected at

26

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

random as test images. First, annotation work was performed using LabelImg (Tzutalin, 2015) in order to use thecreated photograph data of the structural steel frame model as a training data set (Figure 3). Next, the number ofclasses to be detected is set as three, and annotation data is made from the captured images to detect three typesof structural members; a beam, column, and joint. In Figure 3 the XML file shows the position of the rectangle
surrounding each member in the image.

Figure 1. Development procedure of proposed system

Figure 2. Detection procedure for actual steel structure
SelectingimagesCreatedrectangles

column beam
ObjectclassesCreating
rectanglejointTraining data

Input data

Sample of a column object class for training
Figure 3. An example of created annotation data27

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3.2

Segmentation by CNN Using a Scaled Steel Frame Model
The data set for training was made to perform segmentation detection of structural steel frame members.
Unlike the process mentioned in 3.1 when performing rectangle detection, the mask image is created bychanging the beam, column, and joint of the detection target color to white and the background color to black.
The created mask image is then used as a training data set (Figure 4). The image size is adjusted to 512 × 512
pixels since each training data set does not work well if the image size is too large during input phase. Bylearning at this size, capturing the desired features inside the target image is easier.
4. SYSTEM DEVELOPMENT
The training data set made in 3.1 was changed to an appropriate format. Thereafter, YOLOv3 wasfine-tuned, and the object detection was performed on the steel frame model using the proposed system.
YOLOv3 is the excellent object recognition CNN, as it is can handle various object classes and positions bygiving tasks to handle position information in an image. Then, using the training data set made in 3.2, fine-tuning
was performed to change the learning weight of U-Net. Based on U-Net, we constructed a CNN capable ofperforming segmentation detection on targeted members (Figure 5). It was originally proposed for segmentationof medical images. In order to accurately represent the boundary of the detected objects, measurement has been
made to detect, adjustments to the image can then be made, such as increasing the loss at the boundary between
the object to be detected and the background. On the other hand, SegNet (Badrinarayanan et al., 2017), which isone of a popular model for Semantic Segmentation, is a deep learning model that performs segmentation oflandscape images faster and using less memory. Since the Encoder-Decoder structure is adopted, processing canbe performed at higher speeds. However, in the proposed system, it is necessary to accurately detect only theconstruction parts/ elements that are completed by using the photograph of the detection object. Therefore, wechose to use U-Net in the proposed system as it does not emphasize real-time processing performance at highspeeds.
Instance segmentation is implemented by combining the result of segmentation detection and the resultof object detection using the scaled structural steel frame model. The detected structural steel frame members are
displayed in corresponding colors to match each area from the segmentation results by overlapping these withthe detected rectangular areas. Based on this detection result, it is possible to capture the parts whereconstruction is completed only by confirming the image. Furthermore, the attribute information of the member
detected from the photograph can also be captured. Similar to the results obtained for object detection using
YOLOv3, we highlighted the beams in green, columns in pink, and joints in orange (Figure 6).

Figure 4. Constructed scaled steel frame model and data set for learning

Figure 5. Structure and contents of learning of CNN based on U-Net

28

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Combined results

Figure 6. Result of instance segmentation using mask result（a）Construction site image（b）Detection result for the image of (a)Figure 7. Detection result of proposed object detection system
5. RESULTS
5.1 Object Detection Result Using Actual Construction Site Images
We verified whether it is possible to detect a structural steel frame member from a picture of the actual
construction site using YOLOv3 which can detect beams, columns, and joints of a scale-model by fine-tuning.
As a verification experiment to detection accuracy, we prepared an image with different shapes of the detection
target and compared it to the CNN training data set and performed detection (Figure 7). As shown in Figure 7 (a),the target structure is concealed by the crane. This image is not included in the training data set.
Although the result from the detection of the image in the real construction site has high accuracy, it was
not able to detect all the objects (Figure 7 (b)). The detection accuracy is high since the left side of thephotograph contains structural steel frame members that do not overlap, which makes detection easier. However,in the right side of the photograph the boundary line is obscured due to the overlapping of multiple objects;
beams, columns, and joints. This problem occurs since the structure of the detected object in the image iscomplicated. A possible solution to improve the detection method is to add the image data of the actual
construction site to the data set for learning in the future and giving diversity to the constructed system
5.2

Segmentation Result Using Actual Construction Site Images
Similarly, we examined whether it is possible to detect a structural steel frame member by inputting thephotograph of an actual construction site into U-Net, which fine-tuned by the image data of the scaled structuralsteel frame model. The accuracy of the system was verified using Intersection over Union (IoU) (Figure 8). IoU
is the segmentation result that shows the ratio of the correctly recognized areas among the areas predicted to be
the detection target. The accuracy of CNN for steel frame structures was about 80%. However, the accuracy of
the joints is worse than that of the beams and the columns. A possible cause for this is that the structure of thejoint is complicated compared to the other two detection targets, and structures that are complex in nature suchas scaffolding are erroneously designated as joints. The result of instance segmentation is shown by combining
the segmentation result and the object detection result (Figure 9). The detection results of two CNNs are
obtained by JPG format. Therefore, two result images are combined by utilizing the illustrator. First, the shape ofthe steel structure is extracted from the segmentation result. Next, the color of the label is added to thesegmentation result part overlapping with the label of the object detection result. By adding color only to theobject detection part, it is possible to segment the detection result (Figure 10).

29

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 8. Calculation method of IoU

Input actual construction site image

Object detection result

Segmentation result

Detected and segmented steel frame image
Figure 9. Result of detection for actual steel frame structure

Combined result of
the extract part
Each result

Result of instancesegmentation

Extraction of
detection results
Figure 10. Extraction of detection results and combination procedure

Table 1. Accuracy obtained by instance segmentation
Structural steel frame component Intersection over Union（%）
Category
Scale model of steel frame
Beam
95.5
structure
Column
95.7
Joint
84.1
Actual construction site
Beam
78.4
Column
80.3
Joint
50.4

30

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 1 shows the accuracy of the constructed deep learning model for the steel frame structure modeland the actual construction site image (Table 1).
Since the data set for learning in this study was constructed using a scale model of the steel frame, we
expected that the detection accuracy for the scale model of the steel frame would be high. However, even at theactual construction site, it is possible to secure a certain level of detection accuracy. This detection accuracy canbe further improved by adding various actual construction site image data to the training data set.
6. CONCLUSIONS
Currently, the technology of object detection using deep learning has been developed by researchers allover the world. However, only using existing training data sets, the objects that can be detected are limited.
Therefore, when trying to detect a new object using deep learning, it takes much time for the tasks to becompleted, such as collecting image data of various objects and adding the images.
In proposed study, since the actual construction site is often covered with scaffolds and soundproofing
sheets, a 1:30 scale steel frame structure model was made to verify whether it was possible to detect structuralsteel frame members at construction sites. Initially, 2000 images of a training data set were made of a steel frame
model fine-tuned to CNN. By using the constructed CNN, it was possible to confirm about 80% accuracy isachievable, even at the actual construction site. In addition, we constructed a system that can detect the as-built
structure from the image by utilizing deep learning CNN. In this paper, the detection was performed on three
types of steel frame components; beams, columns, and joints. For future development, we will add othercomponents/ building elements to the training data. Then, make training data sets for structural steel frame
members to improve the accuracy and increase the types of detection targets. Moreover, in order to improve theaccuracy of CNN that performs segmentation, we will develop a system that can capture feature quantities moreaccurately.
REFERENCES
Badrinarayanan, V., Kendall, A., and Cipolla, R. (2017). SegNet: A deep convolutional encoder-decoderarchitecture for image segmentation, IEEE transactions on pattern analysis and machine intelligence,
39(12), 2481-2495.
Belsky, M., Sacks, R., and Brilakis, I. (2016). Semantic enrichment for building information modeling,
Computer‐Aided Civil and Infrastructure Engineering, 31(4), 261-274.
Bosche, F., Haas, C. T., and Akinci, B. (2009). Automated recognition of 3D CAD objects in site laser scans for
project 3D status visualization and performance control, J. Comput. Civ. Eng. 23 (6) 311–318.
Brilakis, I., Lourakis, M., Sacks, R., Savarese, S., Christodoulou, S., Teizer, J., and Makhmalbaf, A. (2010).
Toward automated generation of parametric BIMs based on hybrid video and laser scanning data,
Advanced Engineering Informatics, 24(4), 456-465.
Davila Delgado, J. M., Butler, L. J., Gibbons, N., Brilakis, I., Elshafie, M. Z., and Middleton, C. R. (2016).
Management of structural monitoring data of bridges using BIM.
Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., and Fei-Fei, L. (2009). ImageNet: A Large-Scale Hierarchical
Image Database, In CVPR09.
Ding, L., Zhou, Y., and Akinci, B. (2014). Building Information Modeling (BIM) application framework: Theprocess of expanding from 3D to computable nD, Automation in construction, 46, 82-93.
Fathi, H., Dai, F., and Lourakis, M. (2015). Automated as-built 3D reconstruction of civil infrastructure using
computer vision: Achievements, opportunities, and challenges[J]. Advansed Engennering
Informatics,29:149-161.

Fei-Fei, L., Fergus, R., and Perona, P. (2007). Learning generative visual models from few training
examples: An incremental bayesian approach tested on 101 object categories, Computer vision andImage understanding, 106(1), 59-70.
Gidaris, S. and Komodakis, N. (2015). Object detection via a multi-region and semantic segmentation-aware cnnmodel, In Proceedings of the IEEE International Conference on Computer Vision, 1134-1142.
Hamledari, H., McCabe, B., and Davari, S. (2017). Automated computer vision-based detection of componentsof under-construction indoor partitions[J], Automation in Construction,74:78-94.
Khaloo, A. and Lattanzi, D. (2015). Extracting Structural Models through Computer Vision[C], Portland, OR,
USA: American Society of Civil Engineers (ASCE), 538-548.
Kropp, C., Koch, C., and König, M. (2018). Interior construction state recognition with 4D BIM registeredimage sequences, Automation in Construction, 86, 11-32.

LabelImg. (2015). Github. Retrieved from github website: https://github.com/tzutalin/labelImg
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., and Berg, A. C. (2016). Ssd: Single shot
multibox detector, In European conference on computer vision, 21-37.
Long, J., Shelhamer, E., and Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation, The

31

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3431-3440.
Lu, R., Brilakis, I., and Middleton, C. R. (2019). Detection of structural components in point clouds of existing
RC bridges, Computer‐Aided Civil and Infrastructure Engineering, 34(3), 191-212.
Omar, T. and Nehdi, M. L. (2016). Data acquisition technologies for construction progress tracking[J],Automation in Construction, 86:11-32.
Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016). You only look once: unified, real-time object
detection[C], IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV,
USA: IEEE, 779-788.
Redmon, J. and Farhadi, A. (2018). YOLOv3: An Incremental Improvement. arXiv:1804.02767[cs.CV].
Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with
Region Proposal Networks, Advances in Neural Information Processing Systems 28.
Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image
Segmentation, Medical Image Computing and Computer-Assisted Intervention (MICCAI) Vol. 9351,
234-241.
Tang, P., Huber, D., Akinci, B., Lipman, R., and Lytle, A. (2010). Automatic reconstruction of as-built building
information models from laser-scanned point clouds: A review of related techniques, Automation inconstruction, 19(7), 829-843.
Zhu, Z. and Brilakis, I. (2010). Concrete Column Recognition in Images and Videos[j], Journal of Computing in
Civil Engineering, ASCE, 24(6):478-287.
Zhu, Z., Ren, X., and Chen, Z. (2017). Integrated detection and tracking of workforce and equipment fromconstruction jobsite videos[J], Automation in Construction, 81: 161-171.

32

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Video-based construction vehicles type recognition
Chen-Hsuan Wang1, I-Tung Yang2, Meng-Han Tsai3
1) Graduate Student, Department of Civil and Construction Engineering, National Taiwan University of Science andTechnology, Taiwan. Email: m10705506@mail.ntust.edu.tw
2) Chairman, Professor, Department of Civil and Construction Engineering, National Taiwan University of Science andTechnology, Taiwan. Email: ityang@mail.ntust.edu.tw
3) Assistant Professor, Department of Civil and Construction Engineering, National Taiwan University of Science andTechnology, Taiwan. Email: menghan@mail.ntust.edu.tw

Abstract: This research is an ongoing project that aims to recognize construction vehicles automatically for saving
the labor cost. Monitoring the construction vehicle that enters and leaves the construction site can help withtracking the construction progress and recognizing the abnormal vehicles. In modern construction sites, IP cameras
have been widely utilized for security purposes. The enormous amounts of gathered videos bring opportunities for
us to monitor the construction vehicle without assigning a gate guard. However, although IP cameras can store all
the historical images, it is very complicated and time-consuming to calculate the traffic flow of each vehicle typefrom the surveillance images of the day. Therefore, this research proposed a real-time construction vehicle
recognition system. The proposed system contained three major parts: an image recognition algorithm forrecognizing the vehicle type, a database for storing the recognition results, and a dialogue system for supplying
the structuralized results to the site manager. For the image recognition algorithm, the YOLOv3 algorithm wasutilized for processing the video frame by frame and recognizing the vehicle type. The results of the recognition
would be analyzed and stored in a structuralized database for further use. Lastly, a chatbot-based dialogue system
was developed to provide the site information and push instant warning to the related personnel. After a feasibility
test, the unstructured construction video data could be transferred into structured information autonomously. Also,
additional labor resources could be saved and thus increase the efficiency of the overall process of construction
site management.
Keywords:

object detection, object location, image segmentation, YOLOv3 algorithm, dialogue system.

1. INTRODUCTION (ALL CAPITAL, 10 PT., BALD, TIMES NEW ROMAN)
1.1 BACKGROUND
When all industries in Taiwan are trying to use artificial intelligence to reduce manpower and costs, thecivil engineering industry still wastes a lot of human resources on miscellaneous works, whether it is recording
traffic flow on paper every day or hire the security guard to permit the vehicle to enter the site at the entrance. For
calculate the traffic flow, employees must calculate the traffic flow by looking at security camera frame by frame.
It causes a lot of human resources, paper waste, and high error rate. It is difficult to find specific information rightaway because the paper information is easy to lose, and the amount of information is very large. The humid climate
in Taiwan causes the paper to be easily corroded and damaged, and it is difficult to save. When the situation occurs,
paper and human resources cannot be notified by the site manager in time.
1.2 Objective
In order to solve the unnecessary expenditure of labor costs and the shortcomings of paper records, thisresearch developed a system of recognizing the construction vehicle automatically that can be connected to thewhole site vehicle entering and leaving control, combined with two technologies: dialogue system and artificial
intelligence, to save many costs. The accuracy rate notifies the site manager in real-time when there is any vehicle
entering the site.
2. LITERATURE REVIEWIn this research, we apply the YOLOv3 Neural Network algorithm and dialogue system to develop asystem to recognize the construction vehicles automatically. The following section briefly discusses YOLOv3 anddialogue system. The first part introduces the features (Redmon & Farhadi, 2018), applications (Qu et al., 2018;
Yanan et al., 2018; Benjdira et al., 2019; Arai & Rahul, 2019), network architecture (Redmon & Farhadi, 2018)
and detection process (Redmon, & Farhadi 2018) of YOLOv3, and the second part introduces chatbot of dialoguesystem (Litman & Silliman, 2004) and application (Tsai et al., 2019).
2.1 Introduce YOLOv3
2.1.1 Feature
YOLOv3 is a state-of-art image recognition algorithm. Compared with the previous version of YOLOv2,
the network layer of extraction features is improved. The main feature of YOLOv3 is that it uses an end-to-end
detection method. Input the original image into the convolutional network, which can output the position and33

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019category of all targets in the image. It makes YOLOv3 is faster than other convolutional networks. YOLOv3 iscomposed of 53 layers of convolution layers, implemented on the darknet framework, and superimposed byresiduals.
2.1.2

Application
YOLOv3’s detection accuracy at the small object is higher than SSD, and it has higher performance(Redmon & Farhadi, 2018). This advantage makes it more suitable for engineering, and the accuracy is muchbetter than SSD. YOLOv3 can be used for pedestrian detection, railway and road detection, using UAV for
detection, can be used in the self-driving car. These tasks are required to be able to detect immediately.
2.1.3 Network architecture
The network architecture is shown in Figure 1. The first part is the convolution layer. The size of the input
image is presented in 416x416, the number of channels is 3, and the operation is performed using a convolution
kernel with a size of 3x3 and a step of 1 to obtain a 32 channels 416x416 feature map. The second part is the res
layer, and the res layer is derived from resnet. Resnet is the Deep Residual Network. The process of calculating
the residual network is shown in figure 2 (He et al., 2015) In order to solve the phenomenon of gradient diffusion
or gradient explosion of the network. We propose to change the deep network from step-by-step training to stageby-stage training. Using the shortcut connection method, each segment is trained on the residual to achieve asmaller loss finally as well as to control the gradient to avoid disappearing or exploding. There are 53 convolution
layers from 0-layer to 74th-layer of darknet-53, and thus it is summarized as darknet-53. The others are res layers,
and the structure uses a series of 1x1 and 3x3 convolution kernels, the effect is much better than resnet-19. After
maintaining accuracy as resnet-152, it is 1.5 times faster than Resnet-101, twice faster than resnet-152. After 75thlayer is the YOLO part. YOLOv3 can be predicted on three different scales by down-sampling, the first time is at
82nd-layers, and the previous 81 layers are for down-sampling. To make the 81st- layer, the step range is 32, using
a 1x1 convolution kernels to output a feature map of 13x13x255. The second detection is at 94th-layer. The
feature depth from the 79th-layer and 61st-layer use up-sampling, using 1x1 convolution kernels to output a feature
map of 26x26x255. A similar step is used with the up-sampling to deep connect the 91th-layer and 36th-layer
feature. 1x1 convolution kernels are then used to create a third scale on the 106th-layer to output a feature map of
52x52x255. It gives YOLOv3 a certain degree of accuracy when detecting small objects. YOLOv3 detects large
objects at 13x13, middle objects in 26x26 and small objects at 52x52. The neural network predicts three boundingboxes for each size with the formula as in (1), which contains 4 bounding box offsets, 1 target prediction, and 80
Classification prediction.
Tensor = N ∗ N ∗ [(bounding box) ∗ (offset + object + class)](1)

Figure1 YOLOv3 network architecture

Figure 2. Residual learning: a building bloc

2.1.4 Process
The detection process of YOLOv3 is divided into four parts:(1) Calculating bounding box: YOLOv3 divides the image into S×S cells, predicting four coordinates (tx, ty,
tw, th) for each bounding box. The predicted mesh is based on the offset (cx, cy) in the upper left corner.
Previously obtained pw and ph can predict bounding boxes as formula (2). When calculating, use the sum ofsquared error loss to calculate the error for it to fall between 0 and 1.

34

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019bx = σ(t x ) + Cx
by = σ(t y ) + Cy
bw = Pw etw
tw{ bh = ph e

(2)(2) Dimension clustering: The other convolution network usually set the anchors box manually, resulting ininaccurate precision. After YOLOv2, k-means is used to predict the size required by the anchor box. YOLOv3
inherits the method of YOLOv2, using the IOU to determine the final score to evaluate which anchor box is
the most appropriate, the distance function used by k-means as formula (3):d (box , centroid) = 1 − IoU(box , centroid)(3)(3) IoU score: IoU is the ratio of overlap rate for detection result and ground truth. The user can set thethreshold. The object will be recognized if the IoU exceeds the threshold. The IoU as formula (4). If the IoU
exceeds the IoU setting, Identified will be predicted as an object.
detection result ∩ ground truth
𝐼𝑜𝑈( detection result, ground truth ) = detection result ∪ ground truth(4)(4) Classification: The paper of YOLOv3 (Redmon, 2018) use binary cross-entropy instead of softmax because
binary cross-entropy can classify an object to multiple labels, but softmax only can classify an object to asingle label. Formulas such as formula (5):1𝑐 = − 𝑛 ∑𝑥 ( 𝑦 𝑙𝑜𝑔𝑦̅ + (1 − 𝑦)log(1 − 𝑦̅))(5)
2.2 Introduce to the chatbot
Chatbots provide suitable interfaces (Tsai et al., 2019) for users to simulate human conversations via text
or speech. In the past, chatbots were mostly used for telephone customer service (Wu et al., 2015). They cananswer customer questions through pre-set context and dialogue process. In recent years, a chatbot can be divided
into two types, retrieval-based and generative-based (Yang, 2016). Retrieval-based, according to the pre-set
command, make the user get information which user required, the correct rate is 100%, mostly used in specificareas to solve problems. Generative-based uses natural language processing of artificial intelligence to understandthe keywords in the sentence, generate appropriate answers automatically, not suitable for use in specific areas
because the accuracy rate cannot reach 100%, Chatbot is used in disaster management for engineering. Since
disaster-related data is complicated, Ask Diana (Tsai et al., 2019) is developed to solve this problem. It provides
users with an intuitive interface for mobile devices. Users can ask questions via Ask Diana. Whether how is theweather or disaster data and allows users to develop treatment policies effectively.911 bot is a Facebook
Messenger-based application which can report the emergencies to relevant authorities. It provided an intuitive
interface with buttons, which can allow the user easily to use by clicking the button (Crook, 2018). In Taiwan, theNational Science and Technology Center for Disaster Reduction (NCDR) cooperated with Line to develop a
dialogue system which provides alarms for earthquakes, floods and typhoons in real-time (Typhoon Database,2019).
3. METHODOLOGY
In order to autonomously detect the construction vehicles and provide instant messages to the site manager,
this research proposed a video-based construction vehicle recognition system. The system utilized the deeplearning technologies for vehicle type recognition and used a dialogue-based chatbot system to play the role of theinformation provider. The following subsections will provide the introduction about the system structure, the datapreprocess and image recognition algorithm, the database, and the dialogue system respectively.
3.1 System architecture
Figure 3 illustrates the overall system architecture. The system contained three major parts: the datapreprocess and image-recognition, the database, and the chatbot system. As an input to the image recognition, we
set up an IP camera at the gateway of the site for collecting the video. The collected videos are then transmitted tothe device that recognizes the frames. The YOLOv3 algorithm was utilized to process the video frame by frame.
By doing so, the construction gate video could be converted into lists of information, including vehicle type,entering time, count of vehicles, and so on. The information will then be stored in the designed database. Finally,a dialogue-based chatbot system will transmit the relevant information to the user or as an instrumentality for theuser to query the image recognition result.

35

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 3. system architecture
3.2 Data preprocessing and image recognition
The second part is data pre-processing and image recognition. Figure 4 shows the workflow of the data
preprocess and image recognition. In order to enable the recognition system to have high accuracy at different
times and in different environments, it is necessary to collect data of different environments in the data collectionpart, such as morning and night, because the visibility at night is low. Weather is also a factor to be considered as
the accuracy rate is decreased by strong light (Morten, 2017) that the visibility is low on sunny days. After
collecting the data, the first we should do is to remove the data which cannot be used to avoid the error rate. The
second is to process the data into the format that can be trained through the data format specified by the YOLOv3
algorithm, and then divide the data into a training set and a test set to avoid overfitting. Overfitting will make therecognition system work only in a specific place. It cannot be used in the various construction sites. IoU will be
set during training. Only when the IoU of the vehicle type exceeds pre-set IoU score will the vehicle be identified.
The traffic flow calculation line will also be set. Recognition will recognize the type of vehicle and calculate thetraffic flow when the vehicle goes through the traffic flow calculation line.

Figure 4. Data pre-processing and image recognition
3.3 Database
As a site response system, the system requires a database to store data and provide the information requiredby the user. This database is designed to conform to the dialogue system and can provide any corresponding
information in the dialogue system. The database will include raw video files and detection results. The detection
results will be stored as a structuralized table format, which contains information about types of vehicles,
timestamp, traffic flow, and so on.
3.4 Dialogue system
The fourth part is the dialogue system. In this research, we used the dialogue system that in a specificdomain, we need it with high precision. Therefore, the dialogue system we chose is based on retrieval and has an
intuitive interface to set buttons for the user. The dialogue system can increase the accuracy to 100% by clicking
the button. The system architecture is shown in Figure 5. Users can receive the vehicle information automatically36

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

when vehicles are entering the site. Users can also get information by specifying commands or buttons.

Figure 5. Dialogue system architecture
4. PROTOTYPINGIn this implementation, we performed data preprocessing, filter the data and mark the features, and use
YOLOv3 as our algorithm for image recognition, identify the vehicle and then make subsequent judgments onwhether this type of vehicle is wrong, in import and export. The traffic calculation line is used to calculate thetraffic flow in the entrance and exit, and the traffic of the entire day or currently can be counted for management
purposes. The processed data is stored in the database, and the information is transmitted to the user using the
dialogue system.
4.1

Description of the Hardware and Software tools
In this experiment, it is divided into four parts, IP camera, computer, database and dialogue system. The
IP camera recorded the scene of the site entrance and exit. We chose two IP camera that can record 1080P. The
image is transmitted to the computer. The computer hardware device selects the six-core I7-8700 of our CPU. In
the graphic card part, because it needs to be applied to the GPU operation, we chose the Nvidia, and after the film
is processed, we will transmit the result to the database. We selected MySQL, MySQL has high portability,
excellent performance, and easy to develop instructions. Finally, we used Line, a commercial messaging platform,
to be our dialogue system to transmit the data required by the user to the user. The complete system specifications
are given in Tables 1.
Table 1. configurations of the computerdevicemodel
CPUIntel Core I7-8700 (six cores)

Graphic card
Operating system
Dialogue system
RAM
Database
dialogue system

Nvidia GTX 1050, 2GB GDDR4Windows10 (x64-based processor)
Line
32 GB RAM
MySQL
Line

4.2

Data pre-processing
This research used the excavator and agitating lorry as the research object. In order to implement thisresearch, we divide into a training set and a test set. The training set contains every 500 frames of excavators andagitating lorries. The test set contains every 50 frames of excavator and agitating lorries. We collected the datafrom the search engine and photographed by ourselves. In order to ensure the effectiveness and generality of theexperiment, We try to shoot photos of a different time, different environments, different scales, first we use"labelImg" to label the photos manually as figure 6 and remove the image that the target is too small, and createdinto a VOC dataset. The VOC dataset format file is first converted to the YOLO format, and then set obj.data andobj.name and pre-training file. These processes are executed by darknet framework.

37

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 8. Traffic calculation line

Figure 6. Label
4.3

YOLOv3 image recognition algorithm
In this research, the object recognition algorithm uses YOLOv3 algorithm to recognize and locate theconstruction vehicle. We used the YOLOv3 provided code and the default configuration. The value for both height
and width are set to 416×416 and use stochastic gradient descent to optimize our training. Concerning the YOLO
v3 parameters the momentum set to 0.9, decay= 0.0005, learning rate= 0.001, batch= 24 and subdivisions= 8.
Anchors that overlaps the ground truth object by less than a threshold value (0.7) are ignored. The recognition
results are shown in Figure 7.

Figure 7. vehicle detection using YOLOv3
4.4 Traffic flow calculation
In this research, we use OpenCV to calculate the traffic flow (CH.Tseng, 2019). First, we will use the
YOLOv3 algorithm to locate and classify and find the bounding box of the car and find the centroid of the car. We
add a traffic flow calculation line on the detection frame. Recognition system will calculate the traffic flow whenthe vehicle goes through the traffic flow calculation line.
We determine the driving direction of the car by the color of the traffic flow calculation line. If the vehicle
is pass to the blue calculation line first, after that pass to the green line. The vehicle can be confirmed that is toleave from the construction site. If the vehicle is press to the green calculation line first, after that press to the blueline, the vehicle can be confirmed that is to enter the construction site. We added the calculation line to the hidden
layer, it is shown in figure 8, there are two lines to determine the direction of the car quickly, we will broaden thecalculate line as the detection area, it can ensure system not calculate time and again when the speed of a vehicle
is too slow.
4.5 DatabaseIn this research, we use the MySQL to be our database, because MySQL is open source, support manysystems, programming languages, high portability, excellent performance, and easy to develop, the fields in the
database have the type, time, count and frame. The database stores the vehicle type, the timestamp after the frame
is recognized, these allow users to view specific images by vehicle type or specific times, this identification frame
includes several vehicles of the same type and the file name of the stored image (figure 9).

38

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 9. The screenshot of the designed database

Figure 10. Chatbot button setting

4.6

Dialogue system
In this research, our dialogue system is base on Line platform. On the operation interface, when the vehicle
enters the site, the chatbot will notify the user. We have developed four buttons: Total Traffic Flow, Specific Frame,
Each Vehicle Traffic Flow, video. Total Traffic Flow can get the total traffic flow up until now. The user can get
specific time's image through click Specific Frame button. The user can get each type of vehicles' traffic flow
through Each Vehicle Traffic Flow. The user can also get full video link through click Video. For instance, theuser wants to get total traffic flow up until now to check whether the total flow correct, can get through the "Total
Traffic Flow" button, Then the user can get each type of vehicles traffic flow through click "Each Vehicle Traffic
Flow" button to find error , and if the user wants to watch full video, the user can click the "Video" button to get
the video link (Figure 10).
5. CONCLUSIONSIn this research, we developed a system combine chatbot for recognizing types of construction vehicle inreal-time, which can recognize the type of vehicle on-site entrance and exit, and get the traffic flow anytime via
chatbot, it can respond immediately. Our system can run in real-time with simple hardware. All we rely on an IP
camera and a PC with an image capture board. OpenCV will calculate traffic flow when the vehicle goes throughthe calculation line. The image recognition algorithm, YOLOv3, is used to classify whether it is a construction
vehicle according to the characteristics of the vehicle. According to the accuracy and performance of the detection,
the chatbot effectively informs the user that this is an excellent system to reduce human resources and timeeffectively. We also use images from different environments and different times as training data so that we can beforecast in different status. We believe that many systems can be applied to their own needs in accordance withthe prototype of this system.
For future work, we plan to divide into two parts. The first part is improving the safety of pedestrians andworkers. In the aspect of pedestrians, we will serve a warning on vehicles and pedestrians when simultaneous
detection of people and construction vehicles on the calculation line. In the aspect of the worker, we will predict
the trajectory of the crane so that when danger is predicted, we will serve a warning to notify the workers to escape.
We think this system can reduce the risk of disasters in the site. The second part is saving time and human resources
about filling the construct diary. We will make video summarization and image caption to make the whole video
become abstract about what happened today.
ACKNOWLEDGMENTS
This work was financially supported by the Taiwan Building Technology Center from The Featured Areas
Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of
Education in Taiwan.
REFERENCES
Arai, K., and Rahul, B. (2019). Advances in Information and Communication: Proceedings of the 2019 Future ofInformation and Communication Conference (FICC), 1, pp. 261-270
Benjdira, B., Khursheed, T., Koubaa, A., Ammar, A and Ouni, K. (2019) Car Detection using Unmanned Aerial.
Vehicles: Comparison between Faster R-CNN and YOLOv3, (2019), Retrieved from arXiv website:
https://arxiv.org/abs/1812.10968.
Crook, J. (2018) 911bot is a Chat Bot That Could Save Your Life. Retrieved from 911bot website:
https://www.chatbotinsider.ai/industry-news/911-chatbot/

39

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

He, K., Zhang, X., Ren, S, and Sun, J. (2016) Deep residual learning for image recognition. Retrieved from arXiv
website: https://arxiv.org/abs/1512.03385.
Jensen, M.B., Nasrollahi, K., and Moeslund, T.B. (2017), Evaluating State-of-the-Art Object Detector on.
Challenging Traffic Light Data, 2017 IEEE Conference on Computer Vision and Pattern Recognition
Workshops (CVPRW), Honolulu, HI, pp. 882-888.
Kanjaruek, S., and Li, D. (2014) Data-Information Retrieval based Automated Ontology framework for Service
Robots, 2014 International Conference on Manipulation, Manufacturing and Measurement on the
Nanoscale (3M-NANO), pp. 90-94.
Litman, D. J. and Silliman S. (2004). ITSPOKE: an intelligent tutoring spoken dialogue system, HLT-NAACL-Demonstrations '04 Demonstration Papers at HLT-NAACL 2004, pp. 5-8
Qu, H., Yuan, T., Sheng Z., and Zhang Y. (2018). A pedestrian detection method Based on YOLOv3 model and.
Image enhanced by Retinex, 2018 11th International Congress on Image and Signal Processing,
BioMedical Engineering and Informatics (CISP-BMEI), Beijing, China, pp. 1-5.
Redmon, J. and Farhadi, A. (2018). YOLOv3: An incremental Improvement. Retrieved from arXiv website:
https://arxiv.org/abs/1804.02767.
Meng-Han Tsai, James Yichu Chen, Shih-Chung Kang. (2019). Ask Diana: A Keyword-Based Chatbot System.
for Water-Related Disaster Management, Water, 11(2), 234
Tseng, CH (2019), Retrieved from Tseng, CH website: https://github.com/ch-tseng
Typhoon
DataBase(2019).
Retrievedfrom
TyphoonDatabase
website:http://rdc28.cwb.gov.tw/TDB/ntdb/pageControl/ty_warning
Wu, J., Li, M., and Lee, CH. (2015) A probabilistic framework for representing dialog systems and entropy-based
dialog management through dynamic stochastic state evolution. IEEE/ACM Transactions on Audio,
Speech and Language Processing (TASLP), 23(11):2026–2035.
Yanan, S., Hui, Z., Li, L., and Hang, Z. (2018). Rail Surface Defect Detection Method Based on YOLOv3 Deep
Learning Networks, 2018 Chinese Automation Congress (CAC), Xi’an, China, pp. 1563-1568.
Yang
Justin.
(2016),
Chatbot
developmentideas.
Retrievedfromgithubwebsite:http://zake7749.github.io/2016/12/17/how-to-develop-chatbot/

40

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

GA-BASED OPTIMIZATION FOR CONSTRUCTION SEQUENCE OF PRECAST
AND CAST-IN-PLACE CONCRETE COMPONENTS
Songyang Li1, Zhiliang Ma2
1) Ph.D. Candidate, Department of Civil Engineering, Tsinghua University, Beijing, P.R. China. Email: skuyinlee@gmail.com
2) Ph.D., Prof., Department of Civil Engineering, Tsinghua University, Beijing, P.R. China. Email: mazl@tsinghua.edu.cn

Abstract: To reduce the time and cost, as well as to improve the efficiency and safety of precast concrete
construction, the construction sequence of precast concrete (PC) components and cast-in-place (CiP) components
needs to be optimized. To solve this problem, a Genetic Algorithm-based method is proposed in this paper. Firstly,
based on the real project experience and some reasonable assumptions, the construction sequence of PC and CiP
components problem is analyzed and several optimization objectives and constraints are summarized. Then a GAbased method to solve the construction sequence problem is implemented, which includes the coding method,
fitness function as well as crossover, mutation and selection operations. Finally, a case study is conducted to verify
the proposed method. The results indicate that the method can effectively find the optimal solution for theconstruction sequence of PC components and CiP components.
Keywords: Precast concrete buildings, Construction sequence, Genetic Algorithm, Optimization
1. INTRODUCTION
With the encouragement of Chinese government and implementation by construction enterprises,
prefabricated buildings have emerged as a trend in China. Compared with traditional buildings, prefabricatedbuildings are more attractive due to their higher quality, higher construction efficiency, less waste andenvironment-friendly characteristic (Cao et al., 2015; Jaillon and Poon, 2008; Jaillon et al., 2009; Tam et al., 2007).
As a result of market environment and traditions, precast concrete (PC) buildings dominate other types ofprefabricated buildings, such as wood structure and steel structure, especially for residential buildings in China.
For a PC building, part of the concrete components of the building, including walls, slabs, columns, beams, stairs
and so on, are produced in a factory, and then transported to the construction site for on-site assembly. Since theprocessing, structural technology and management of prefabricated buildings have been improved (Yang et al.
2019), the prefabrication rate of PC buildings is getting higher, which means more and more components are PC
components. This makes the construction process of PC buildings more like assembling a LEGO set, in which thesequence is crucial. Construction sequence of building components have direct impacts on the time, cost, difficulty
and safety of PC building construction, thus needs to be optimized.
Up to now, such optimization has been carried out mainly for in-factory production phase of PC buildings.
For example, a multi-objective Genetic Algorithm-based searching technique is proposed to solve extended
Flexible Job Shop Scheduling Problems for precast production (Anvari et al., 2016); a flowshop scheduling model
for multiple production lines for precast production is proposed and a GA-based optimization approach to facilitate
optimized scheduling is developed, considering workstation idle time, contract penalty and storage cost, makespanand type change of precast components as optimization objectives (Yang et al., 2016); and an approach for
optimizing shop floor rescheduling of multiple production lines for flowshop production of PC components is
proposed in case the original schedule cannot be continued (Ma et al., 2018). However, only a little attention has
been paid to the sequencing problem for PC construction phase. In a previous study, the prefabricated wall panel
sequencing problem is solved with the objective to minimize the work required to position and brace panels, whilepanel interference is ignored (Shewchuk & Guo, 2011). In a recent study, the assembly sequence of PC walls
considering the weights and occupied spaces of the walls and the interference is optimized using GA and BIMbased approach (Wang et al., 2018). In real PC projects, some building components are PC components, whileothers remain to be cast-in-place (CiP) components. Both kinds of components are usually constructed in parallel.
However, existing studies about PC construction sequence only consider the PC components and the CiP
components are omitted.
This paper proposes a GA-based method to obtain the optimal Construction Sequence of Precast Concrete
components and Cast-in-Place components (CS-PC&CiP). Firstly, the CS-PC&CiP is analyzed to summarize theobjectives and constraints of the optimization in Section 2. Next, a GA-based method to solve the CS-PC&CiP isintroduced in detail in Section 3. Then, a case study is conducted to verify the proposed method in Section 4.
Finally, the study is concluded in Section 5.
2. ANALYSIS ON CS-PC&CIP
After several field studies of PC projects and consulting experts from the industry, the authors found thatthe most common type of PC buildings in China is shear wall structure. In the following analysis and discussion,
PC shear wall structure is the main focus and used as an example.

41

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2.1 Basic Analysis
For PC shear wall structure, the general construction sequence is in the following order: bottom slabs,
exterior walls, interior walls, beams, stairs and top slabs. Usually, all slabs are PC components and they areassembled one by one following the sequence that they are arranged on the plan without other components being
constructed simultaneously. Similarly, all exterior walls are PC components and they are usually constructed one
by one in the clockwise direction or counterclockwise direction. Therefore, the construction of slabs and exterior
walls has a natural sequence which is easy to get and need no optimization. What concerns us the most is theconstruction sequence of interior walls in a PC shear wall structure.
Interior walls consist of both PC and CiP components. Most interior walls are PC walls, while in somepositions, such as stairwells, elevator shafts and bathrooms, CiP walls are still being used to improve the integrityof the whole structure. In practice, PC walls and CiP walls are constructed in parallel. However, due to the quantity
differences between two kinds of walls, CiP walls are always completed much earlier than PC walls, which may
cause a cost due to the idling of workers. What is more, the simultaneous construction of PC walls and CiP walls
may have some conflicts in space, which hinder the efficiency and safety. Therefore, it is a vital work to optimize
CS-PC&CiP for interior walls under some objectives and constraints. It deserves to note that the junctions between
PC walls are also cast-in-place, but they are not considered in this study for simplicity.
Before the analysis of the objectives and constraints for the optimization for CS-PC&CiP, several
assumptions are made: 1) only one PC wall and only one CiP wall can be constructed in parallel by two different
work groups at one time; 2) the time needed for constructing a wall is proportional to the volume of the wall, forboth kinds of walls; 3) PC walls are constructed one by one continuously without interruption, while theconstruction of CiP walls can be interrupted to avoid the conflicts with the PC walls.
Under these assumptions, the known conditions include the volume of each PC wall and CiP wall, andparameters to indicate if a PC wall and a CiP wall interfere with each other when they are constructed in parallel.
The variables to be optimized for CS-PC&CiP are the construction sequence of PC walls and that for CiP walls.
For the construction sequence of CiP walls, the interruptions between the construction of different walls should beconsidered.
The objectives and constraints for the optimization for CS-PC&CiP are summarized in Table 1 and thedetails of them are discussed in the following subsections.
No.
1
2
3
4
5

Table 1. Optimization objectives and constraints for CS-PC&CiP
Items
Minimization of the makespan of CiP walls construction
Optimization objectives Minimization of the interval between the completion time of two kinds ofwallsClassification

Optimization
constraints

Constraint of volumes of PC walls
Constraint of volumes of CiP walls
Constraint of interferences between two kinds of walls

2.2 Optimization Objectives(1) Minimization of makespan of CiP walls construction.
As mentioned in the assumptions, PC walls are constructed continuously. That means the makespan of
PC walls construction is fixed, which is the sum of the construction time of each individual wall. However, theconstruction of CiP walls can be interrupted to avoid the interferences with the PC walls being constructed, whichmakes the makespan of CiP walls construction longer or equal to the sum of the construction time of eachindividual wall. To reduce the idling of the work group for CiP walls during the interruptions, the makespan ofCiP walls construction needs to be minimized when optimizing the CS-PC&CiP.
(2) Minimization of the interval between the completion of two kinds of walls
Due to the quantity difference between PC walls and CiP walls, the completion of CiP walls is always
ahead of the completion of PC walls. However, the optimal condition is the constructions of both kinds of walls
are completed at the same time and subsequent construction of beams and top slabs can start after the completionof walls to reduce the idling after the completion of CiP walls construction. Therefore, the interval between thecompletion of two kinds of walls needs to be minimized.
2.3 Optimization Constraints(1) Constraints of volumes of PC walls
One step of PC walls construction is to lift the wall from PC components stacking site to the designed
position for assembly. But occupation of stacking site cause cost. To save the storage cost of PC walls as much as
possible, the PC walls occupied more space should be lifted to designed position as earlier as possible. Thus, thePC walls with the largest volumes should always be constructed first.
(2) Constraints of volumes of CiP walls

42

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

One step of CiP walls construction is to bind the reinforcement of the wall. The CiP walls with larger
volumes usually have more complicated reinforcement skeleton inside. To reduce the difficulty of construction,
the CiP walls with larger volumes should be constructed as earlier as possible when the space of the floor is lesscrowded.
(3) Constraints of interference between two kinds of walls
If the simultaneously constructed PC walls and CiP walls intersect with each other or are very closely
located, their constructions will interfere with each other and may increase the difficulty of construction, as wellas cause some safety issues. Thus, this kind of condition should be avoided when optimizing the CS-PC&CiP.
3. GA-BASED METHOD TO SOLVE THE CS-PC&CIP
Genetic algorithm is one kind of commonly used algorithm to solve the optimization problems. In thisstudy, GA is implemented to solve the construction sequence of precast concrete and cast-in-place components.
The process of GA is shown in Figure 1 and detailed implementations of each step to solve the CS-PC&CiP isthoroughly discussed in the following subsections.
Ge e a
e

a

a

C

e

M a
Se ec
N
Sa

I d d ac d

Ge e a

a

F e
e a a

a

Te

a e?

Ye

E d

Figure 1. Flowchart of Genetic Algorithm
3.1 Individual Coding and Population Initialization
To get the optimal CS-PC&CiP, each possible construction sequence is coded into a chromosome as one
individual as Figure 2 shows.

x1 x2 … xm y1 y2 … yn
Figure 2. Individual coding of one possible sequence
Each chromosome consists of two parts of genes. The first part is the sequence of 𝑥" to 𝑥# , which is
integer coded. Each element of the sequence, 𝑥$ , is a unique integer representing the ID of PC wall that will be
constructed in 𝑖th order and 𝑚 is the quantity of PC walls. Because PC walls are constructed continuously
without interruption, only the construction sequence of each wall need to be coded, the exact starting time andcompletion time of each wall can be easily computed using this sequence. The second part is the sequence of 𝑦"to 𝑦( , which is real number coded. Each element of the sequence, 𝑦) , is a positive real number representing thestarting time of the construction of CiP wall with the ID of 𝑗 and 𝑛 is the quantity of CiP walls. Because theconstruction of CiP walls can be interrupted, the starting time of each wall must be specified in the code, and theconstruction sequence of CiP walls can be inferred accordingly.
When initializing the first population, each individual is randomly generated under the constraints that 𝑥$
is a unique integer in the range of 1 to 𝑚 and 𝑦) is a positive real number. Besides, 𝑦) should also satisfy
Equation (1) and Equation (2).
(1)
0 ⩽ yj ⩽ TPC j = 1,…, nyk + TCiPk ⩽ yl

k = 1,…, n(2)where 𝑇-. is the makespan of PC walls construction, 𝑇.$-/ is the construction time of CiP wall with the ID of𝑘. Equation (1) means the starting time of each CiP walls should be before the completion of all PC walls. Equation(2), in which 𝑙 is the subsequent wall to be constructed after the completion of wall 𝑘, means the starting timeof one CiP wall should be after the completion of its previous wall to avoid the overlap of CiP walls construction.
3.2 Fitness Function

43Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Fitness function is used to evaluate the fitness of each individual. In the proposed method, individual withthe minimum fitness value has the highest fitness, and thus, is the optimal solution for the CS-PC&CiP. Fitness
function should reflect the aforementioned optimization objectives and constraints in the previous section, andeach of them are implemented as follows.
(1) Minimization of the makespan of CiP walls construction
The makespan of CiP walls construction, 𝑇-$. , is calculated by using Equation (3), and needs to beminimized.
(3)
min TCiP = max{yj + TCiP } − min{yk} j = 1,…, n k = 1,…, nj(2) Minimization of the interval between the completion of two kinds of walls
The interval between the completion of all PC walls and all CiP walls, 𝑇$(3 , is calculated by using
Equation (4), and needs to be minimized.
(4)
min Tint = | max{yj + TCiP } − TPC | j = 1,…, nj(3) Constraints
When solving the CS-PC&CiP, three aforementioned constraints usually cannot be satisfied
simultaneously. Thus, the violation of the constraints is allowed in the proposed method. To introduce theconstraints into the fitness value, penalty function method (Yeniay, 2005) is used. Once a constraint is violated, apenalty value will be added to the fitness value, and the total penalty value should also be minimized whensearching for the optimal solution.
For the constraints of volumes of PC walls, referring the previous study (Wang et al., 2018), a penalty
matrix, 𝑀, is defined in Equation (5), where 𝑉-./ is the volume of PC walls with the ID k. Once a smaller PC
wall is constructed before a larger one, a penalty value of the volume ratio of two walls, which is larger than 1,
will be added to the fitness value. The more the difference between the pair of walls, the larger the penalty will be.
The total penalty value for the constraints of volumes of PC walls, 𝑃-. , is calculated by using Equation (6), andneeds to be minimized.
VPCj, VPCj > VPCi
M = Mij = VPCi
(5)0,
VPCj ⩽ VPCi

min

PPC =

m−1

m

∑ ∑

i=1 j=i+1

m×m

Mxi xj(6)

The similar penalty matrix, 𝑁, and total penalty value, 𝑃.$- , for the constraints of volumes of CiP walls
are defined or calculated by using Equation (7) and Equation (8) respectively. In Equation (8), 𝐽 is the ID set ofCiP walls which are constructed after the CiP wall 𝑖.
VCiPj, VCiPj > VCiPi
N = Nij = VCiPi(7)0,
VCiPj ⩽ VCiPi

min

PCiP =

n−1

∑∑

n×n

Nij

(8)

i=1 j∈J

For the constraints of interference between PC walls and CiP walls, a penalty matrix, 𝐿, is defined in
Equation (9). Once a CiP wall is being constructed simultaneously with a PC wall that interfere with it, a penalty
valued of 1 will be added to the fitness value. The total penalty value for this constraint, 𝑃$(3 , is calculated byusing Equation (10), where 𝐽 is the ID set of PC walls which are constructed simultaneously with CiP wall 𝑖.
1, CiPi interferes with PCj
L = Lij ={[]0, otherwisemin

Pint =

n

(9)
n×m

L
∑ ∑ ij(10)i=1 j∈J(4) Integrated Fitness Function
Finally, all optimization objectives and constraints are integrated into one fitness function as shown inEquation (11).
TCiP − TCiPmin
PPC
PCiP
T
P
min F = w1
+ w2 int + w3
+ w4
+ w5 int(11)
TCiPmax − TCiPmin

T*
int

P*
PC

P*
CiP

P*
int

where 𝑤$ is the weight allocated to the 𝑖th item in the equation, which reflects the importance level of that44

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

objective or constraint when searching for the optimal solution. It deserves to note that the final optimal solution
might be sensitive to the weights allocated to the three constraint items. The larger the weight is, the less likely the
∗
∗
∗
∗
, 𝑃-.
, 𝑃.$, 𝑃$(3in the denominators are the maximum values ofcorresponding constraint will be violated. 𝑇$(3∗∗are the maximum and minimum values of
the corresponding variables in the nominators. 𝑇.$-<=> and 𝑇.$<?@∗
the makespan of CiP walls construction respectively. 𝑃-. is calculated using Equation (6) with the input solution∗
is calculated using Equationin which the PC walls are constructed from the smallest one to the largest one. 𝑃.$∗(8) with the input solution in which the CiP walls are constructed from the smallest one to the largest one. 𝑃$(3∗∗∗cannot be calculated directly, and thus is estimated by simulations. 𝑇$(3 , 𝑇.$-<=> and 𝑇.$-<?@ are calculated byusing Equation (12). These parameters are used to normalize the corresponding variables because these variables
have different scales and units. After normalization, each item in Equation (11) will have a value in the range of 0
to 1.

T*
= max{max{TCiPi}, | TPC −
int

n

∑
j=1

TCiPj | } T*
CiP

max

= TPC + max{TCiPi} T*
CiP

min

=

n

T
∑ CiPj(12)

j=1

3.3 Generation of a New Population
In the GA, the child generation is generated by conducting crossover operation, mutation operation andselection operation on the parent generation. Since the chromosome in this method is coded with both integer andreal number, different rules need to be mixed up for the crossover and mutation operations.
(1) Crossover Operation
For the first part of the chromosome, which is integer coded, a similar single point matching crossover
method (SPMCM) is used as introduced in previous study (Wang et al., 2018). For the second part of thechromosome, which is real number coded, the crossover operator defined in Equation (13) is used.
(13)
yc = αyp + (1 − α)yp 0 < α < 112where 𝑦A is the child chromosome, 𝑦BC and 𝑦BD are parent chromosomes.
(2) Mutation Operation
For the first part of the chromosome, which is integer coded, the mutation operation is realized byrandomly exchanging two genes of the chromosome. For the second part of the chromosome, which is real number
coded, the non-uniform mutation (Zhao et al., 2007) is used. The mutated gene is randomly generated from intervalΩ defined in Equation (14).
t c(14)Ω = [yk − s(t)(yk − Lk), yk + s(t)(Uk − yk)] s(t) = 1 − r (1− T )where 𝑦E is the gene value before mutation, [𝐿E , 𝑈E ] is the range of 𝑦E , 𝑇 is the maximum generation number,𝑡 is the current generation number, 𝑟 and 𝑐 are parameters.
If the child generated by crossover operation and mutation operation fails to satisfy Equation (2), it will
be abandoned.
(3) Selection Operation
After crossover operation and mutation operation, selection operation is finally conducted to generate thenew population. Roulette wheel selection is used in this method, by which the individuals with higher fitness(lower fitness value) are more likely to be selected into next generation.
4. CASE STUDY
In this section, the proposed GA-based method to solve the CS-PC&CiP is applied on a hypotheticalpenalized building to solve optimal construction sequence of PC interior walls and CiP interior walls.
4.1 Case Description
The plan view of the case floor that used to test the proposed GA-based method is shown in Figure 3. It
contains of 11 PC interior walls and 5 CiP interior walls as indicated by the labels near the walls.
PC4
PC2

PC5

CiP1

CiP3

CiP2
PC1

PC3
CiP4
PC11

PC9

Bathroom

PC7
CiP5

PC10

PC8

PC6

Figure 3. Plan view of the case floor
The detailed information of the case floor is summarized in Table 2. All the walls have the same height45

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

and thickness. The first row is about PC walls, and the first column is about CiP walls. The number next to thewall ID is the length (unit: meter) of the corresponding wall. In the case a PC wall and a CiP wall interfere witheach other if constructed simultaneously, the intersecting cell of two components is filled with 1. In this case study,
the interferences are recognized when the two walls intersect with each other or the distance between the walls is
less than 2.0 meter. And more comprehensive rules could be formulated according to the situations when applying
in real projects.
In this case study, the length of wall is used to represent its volume. Since only ratios are used in the whole
optimization process, using length as a substitute will have no influence. Besides, assumptions about theconstruction time are used: the construction time of PC walls per unit volume is 1 unit time, while the construction
time of CiP walls per unit volume is 1.5 unit time.
PC1
1.5
CiP1
CiP2
CiP3
CiP4
CiP5

PC2
2.0

2.0
2.3
1.7
1.9
1.7

Table 2. Detailed information of the case floor
PC3
PC4
PC5
PC6
PC7
PC8
3.2
1.6
2.0
1.8
3.2
1.8
1
1
1
1
1
1
1
1
1
1
1

PC9
2.8

PC10
1.8

PC11
3.0

4.2 Configuration of Parameters
The parameters configuration of the GA-based method for this case study are summarized in Table3.
Table 3. Configuration of parameters
Value
Parameter
0.4
Elite rate
0.2
Population size
0.1
Maximum Generation 𝑇
0.1
α
0.2
𝑟
0.8
𝑐
0.05

Parameter
𝑤"
𝑤L
𝑤M
𝑤N
𝑤O
Crossover rate
Mutation rate

Value
0.05
200
1000
0.5
0.5
3

4.2 Results and Discussion
The algorithm is terminated at the fixed maximum generation which is 1000 in the case study. The optimal
construction sequence for the case floor obtained by the proposed GA-based method after 1000 generations isshown in Figure 4. The upper row is the sequence of PC walls and the lower row is the sequence of CiP walls. The
width of each rectangle represents the construction time of each wall, which is proportional to the volume of thewall.
The final fitness value of this optimal solution is 0.338. The constraints of volumes of CiP walls and theconstraints of interference between two kinds of walls are all satisfied in this solution. However, the constraints ofvolumes of PC walls are violated by PC Wall 5, as the red part of Figure 4 shows. This is because PC Wall 5 willinterfere with CiP Wall 2 or CiP wall 4 if it is sequenced by the volume order. However, constraints of interference
have a larger weight in the fitness function. For the objective of minimization of the interval between thecompletion of two kinds of walls, this optimal solution fails to optimize the interval to zero, as the blue line in
Figure 4 shows. This is because PC Wall 4 will interfere with CiP Wall 3 if the whole process of CiP walls
construction is dragged forward to keep up with the completion of PC walls. For the objective of minimization of
the makespan of CiP walls construction, current solution fails to make it optimal. The intervals marked by green
ellipses in Figure 4 can be eliminated without violating any constraint of interference if more generations of
population are created.
3

7

5

11

9

2

8

10

6

4

1

PC

CiP
2

1

4

3

5

Figure 4. Optimal solution for the case study (1000 generations)

46

time

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The convergence curve for the case study is shown in Figure 5. The horizontal axis is the generation
numbers and the vertical axis is the fitness values. The lower black points are the best fitness value for eachgeneration and the upper blue points are the mean fitness value for each generation.
Best: 0.0338182 Mean: 0.0535477

0.45Best fitness
Mean fitness

0.4
0.35

Fitness value

0.3
0.25
0.2
0.15
0.1
0.05
0
0

200

400

600

800

1000

Generation

Figure 5. Convergence curve for the case study

Current best individual

5. CONCLUSIONS
Current Best Individual25
In this study, a GA-based
method to solve the optimal construction sequence of precast concrete
components and cast-in-place components is proposed. The CS-PC&CiP problem is analyzed firstly, and the
optimization objectives and constraints are put forward, based on practical experience and some reasonableassumptions. Then a GA-based20method to solve CS-PC&CiP problem is implemented. The mix of integer coding
and real number coding is used to assemble the chromosomes of the individuals. An integrated fitness function isconstructed to reflect the objectives
and constraints summarized before. The crossover operation and mutation
15
operation for the mixed coding are explained in detail. Finally, a PC shear wall structure is used as a case study toverify the proposed method. The optimal solution is obtained and the superiorities and drawbacks of this solution
are discussed.
10
However, this study still has some limitations. Firstly, the structural relationships among different
components are omitted. Secondly, in real projects, both PC components construction and CiP components
5
construction contains many different
operations which need to be considered separately because different
operations may have different resource requirements. Thirdly, the input data about the PC building for GA-based
method are recognized and collected manually, which will need a lot of human effort if being used in real projects.
0
Therefore, in future studies, theshouldto the model; the construction
0 structural2
4 considerations
6
8
10
12be added14
16
sequencing of scheduling problem should be modeled
andsolvedonoperation-level;
and Building Information
Number of variables (16)Modeling (BIM) should be involved to these kinds of problem to realize the automation of the whole process.
ACKNOWLEDGMENTS
This study is supported by the Joint Research Center of Tsinghua University and Glodon Co., Ltd.
REFERENCES
Anvari, B., Angeloudis, P., and Ochieng, W. Y. (2016). A multi-objective GA-based optimisation for holistic
Manufacturing, transportation and Assembly of precast construction, Automation in Construction, 71,
226-241.
Cao, X., Li, X., Zhu, Y., and Zhang, Z. (2015). A comparative study of environmental performance between
prefabricated and traditional residential buildings in China, Journal of cleaner production, 109, 131-143.
Jaillon, L., and Poon, C. S. (2008). Sustainable construction aspects of using prefabrication in dense urban
environment: a Hong Kong case study, Construction Management and Economics, 26(9), 953-966.
Jaillon, L., Poon, C. S., and Chiang, Y. H. (2009). Quantifying the waste reduction potential of using prefabricationin building construction in Hong Kong, Waste Management, 29(1), 309-320.
Ma, Z., Yang, Z., Liu, S., and Wu, S. (2018). Optimized rescheduling of multiple production lines for flowshop
production of reinforced precast concrete components, Automation in Construction, 95, 86-97.
Shewchuk, J. P., and Guo, C. (2011). Panel stacking, panel sequencing, and stack locating in residential
construction: lean approach, Journal of Construction Engineering and Management, 138(9), 1006-1016.
Tam, V. W., Tam, C. M., Zeng, S. X., and Ng, W.C. (2007). Towards adoption of prefabrication in construction,47

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Building and Environment, 42(10), 3642-3654.
Wang, Y., Yuan, Z., and Sun, C. (2018). Research on assembly sequence planning and optimization of precast
concrete buildings, Journal of Civil Engineering and Management, 24(2), 106-115.
Yang, H., Lv, Y., He, y., and Zhu, J. (2019). Research and prospect on the present situation of assembled buildings
in China, In IOP Conference Series: Earth and Environmental Science (Vol. 242, No. 6, p. 062083). IOP
Publishing.
Yang, Z., Ma, Z., and Wu, S. (2016). Optimized flowshop scheduling of multiple production lines for precast
production, Automation in Construction, 72, 321-329.
Yeniay, Ö. (2005). Penalty function methods for constrained optimization with genetic algorithms, Mathematicaland computational Applications, 10(1), 45-56.
Zhao, X., Gao, X.S., and Hu, Z.C. (2007). Evolutionary programming based on non-uniform mutation, Applied
Mathematics and Computation, 192(1), 1-11.

48

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019BASIC STUDY ON DETECTING BREAKING SOUNDS OF STRUCTURAL
MEMBERS BY USING MACHINE LEARNING
Takeo Izumita1, Masayuki Saeki2
1) Master’s Student, Tokyo University of Science, Chiba, Japan. Email: 7618503@ed.tus.ac.jp
2) Dr. Eng., Prof., Tokyo University of Science, Chiba, Japan. Email: saeki@rs.noda.tus.ac.jp

Abstract: In case of a large earthquake, it is necessary to grasp the damage state of structures in disaster areas assoon as possible. Therefore, we have been trying to develop a sensor which is able to detect the destruction of
structural members.
In this research, we focus on the breaking sound of wood that is easily recorded using a smartphone andis expected to be discerned from any other sounds with the help of machine learnings. As a basic research, firstexperiments were carried out to record the breaking sounds of woods as well as other sounds such as talking voice,crashing dishes, closing door, etc. Second, Mel Frequency Cepstral Coefficients (MFCC), that is widely used as
acoustic features in the field of speech recognition, were calculated by analyzing the recorded samples. Third,
Multi-Layer Perceptron (MLP) and Support Vector Machine (SVM) were applied to the MFCCs to categorize
them into two groups which were destruction sound and the others. In the analysis, 156 MFCCs were prepared astraining data. In case of SVM, about 70% of these data were properly recognized. On the other hand, MLP with
one hidden layer gave a success rate of about 90%. In addition, the effect of the selection of training data sets andthe MLP models on the success rate was investigated.
Keywords:

Mel Frequency Cepstral Coefficients, Machine Learning, breaking sound, damage estimation

1.

INTRODUCTION
In case of a large earthquake, it is necessary to grasp the damage state of structures in disaster areas assoon as possible for using the limited resources effectively. In Japan, several seismic networks have been
constructed and seismic intensities were estimated. The information is helpful for narrowing the damaged area,
however the spatial resolution might be still not enough to specify the damaged structures. Therefore, we havebeen trying to develop a sensor which is able to detect the destruction of structures.
In this research, we focus on the breaking sound of wood to detect the damaged structures because humanbeings can identify the breaking sounds of wood even under the situation that various sounds simultaneously occur.
Besides, the sound can be easily recorded by a smartphone and be analyzed in it with the help of machine learnings.
In recent years, speech recognition technology has been improved due to the rapid development of machinelearnings and used in various systems such as a smartphone, smart speaker, voice search, etc (Kawahara, 2015).
In addition, other technologies related to the sound recognition have been studied. For example, Tani et al. (2017)
develop a system for crime prevention, which continuously records various sounds in a city and estimates thesurrounding conditions. The other example is the research in which a falling object’s shape, material, and thefalling height are estimated from the sound emitted when the falling object touches onto the ground (Zhang, et al,2017). These technologies are expected to be applied to the situation recognition problem in case of earthquake.
The objective of this research is to investigate the feasibility of detecting breaking sounds of woods bymeans of machine learnings. As a basic research, first some experiments were carried out to record the breaking
sounds of woods as well as other sounds such as talking voice, crashing dishes, closing door, etc. Second, Mel
Frequency Cepstral Coefficients (MFCC), that is widely used as the acoustic features in the field of speech
recognition, were calculated by analyzing the recorded samples. Third, Multi-Layer Perceptron (MLP) and Support
Vector Machine (SVM) were applied to the sound recognition problem of categorizing the MFCCs into two groups
which were breaking sounds of woods and the others. In the analysis, 156 MFCCs were prepared as training data
sets. The results showed that only about 70% of these data were properly recognized in case of SVM. On the other
hand, MLP with one hidden layer achieved a success rate of about 90%. In addition, the effect of the selection oftraining data sets and the MLP models on the success rate were investigated. We also investigate the features of
echo of the breaking sounds because it is not sure whether the echo can be used as the training data of the breaking
sounds of woods or not. If the echo can be treated as the breaking sounds, the number of training data is easily
increased. So, in the analysis, 120 MFCCs corresponding to the echo were prepared as the evaluation data. The
results show that about 67% of echo were recognized as breaking sounds of woods.
2. PREPARATION OF TRAINING DATA SETS
2.1 Experiments of recording sounds
In the experiments, the breaking sounds of woods were emitted by bending a timber, destroying a part ofhalf-lap joint of woods, or splitting a timber in a silent room. The waveforms of sound pressure were recorded
using the sound-level meter (LA-4440, ONOSOKKI) and the sound recorder (DR-7100, ONOSOKKI) in this49Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019research. The sampling rate was set to be 51.2 kHz which covered the audible range. Figure 1 shows a photo of
the experiment of destroying a part of half-lap joint of woods.
The above observation was carried out thirty times from September 26th, 2016 to January 23rd, 2019 by
three students separately. The other sounds such as crashing dishes, talking voice, closing door etc., which might
be emitted at a large earthquake, were also recorded in the same period. Table 1 shows the number of experiments
for each pattern.

Figure 1. A photo of experiment of recording a breaking sound of woods
Table 1. Number of experiments for each pattern
Breaking sound of woods
Other sounds
Breaking half-lap joint
22 times
Closing door
Bending a timber
5 times
Crashing dishes
Splitting a timber
3 times
Talking voice
Other sounds
2.2

8 times
7 times
5 times
13 times

Calculation of MFCC as an acoustic feature
For updating the parameters of sound detection model, the sound pressure waveforms themselves or the
acoustic features calculated from the waveforms can be used as learning data. In this research, Mel Frequency
Cepstral Coefficients (MFCC) are used as an acoustic feature. The calculation procedure of MFCCs is shown in
Figure 2.
Pick up a 20 ms frame data from recorded soundApply a Hamming window and Fourier Transform to the frame

Calculate the power spectrum of the frame

Apply Mel filter bank to the power spectrum

Calculate the logarithm of the energy passing through each filter

Apply Discrete Cosine Transform (DCT) to the calculated energy

Figure 2. Calculation procedure of MFCC

50

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

MFCCs are well known to be the dominant features used for speech recognition (Logan, 2000). Mel is thescale which represents well human auditory characteristics. The frequency can be converted to the Mel using theequation (1).
𝑓𝑓
+ 1�
700

𝑚𝑚𝑚𝑚𝑚𝑚 = 1125 × ln �

(1)

In the calculation of MFCCs, first, a data frame of 20 ms is extracted from a time series data of the recorded
sound. 20 ms is the data length generally used in the speech recognition. It is not sure whether the data length isoptimal or not for detecting the breaking sounds of woods. Second, the hamming window and the Fourier transform
are applied to the data frame to obtain the Fourier spectrum. Then, the power spectrum is calculated from theFourier spectrum. At the next step, Mel filter bank is applied to the power spectrum to calculate the energy passing
through each filter. The Mel filter bank is the group of 20 filters in which each filter is arranged at an equal intervalof the Mel scale. Finally, Discrete Cosine Transform (DCT) is applied to the logarithm of the energy passing
through each filter to calculate MFCCs. Therefore, in this research, 20 ms sound data is converted to the MFCC
which is a vector with 20 components.
Training data sets prepared in this research
In this research, data frames of 20 ms are extracted from the recorded sound every 10 ms in the same way
to the process used for speech recognition, and MFCCs are calculated for every frame when the sound pressure
exceeds the noise level. The calculated MFCCs are classified into three groups named “BROKEN”, “ECHO” and“OTHERS”. If the data frame extracting from the breaking sounds of woods includes the maximum amplitude,
the corresponding MFCC is defined as “BROKEN”. The MFCCs calculated from the following data frames aredefined as “ECHO”. Considering an earthquake situation in which the breaking sounds are recorded by using adevice such as smartphone in a house, it is necessary to take the feature of echo into account. The MFCCs
calculated from the other sounds such as crashing dishes and talking voices are defined as “OTHERS”.
In the experiments, 78 MFCCs of BROKEN are obtained from the 30 experiments. Therefore, the same
number of MFCCs are selected from OTHERS as training data. The MFCCs of ECHO are not used as training
data in this research because it is not sure whether the ECHO should be categorized to BROKEN or not. So, 156
MFCCs are prepared for the training data in this research. In machine learnings, the MFCCs of BROKEN arelabeled as 1 and those of OTHERS as 0. Besides, all MFCCs are normalized as pre-processing since the amplitudes
are very different from each other.
2.3

3. TRIALS OF MACHINE LEARNINGS
To investigate the performance of machine learnings, two methods are compared in this research. One is
Support Vector Machine (SVM) and the other is Neural Network (NN). Both the methods are used as a 2 levels
classification problem.
3.1

Simulation results of Support Vector Machine
SVM is one of the kernel-based learnings and is the method of classification to create the patterndiscriminator with using of linear threshold elements and kernel functions. The model parameters are optimized
by maximizing the margin which is defined as the minimal distance of a sample to the decision surface. The marginof linear classifier is the minimal distance of any training point to the hyperplane (Muller et al, 2001).
In the analysis, the module supported by scikit-learn (Sklearn.svm.SVC, 2019) on application of
Anaconda (Anaconda, 2019) is used for machine learning. A linear kernel function is selected as a kernel functionof SVM. The 156 MFCCs mentioned above are input as training data. The estimated success rates are shown in
Table 2. In this result, all MFCCs of OTHERS are correctly classified as OTHERS. On the other hand, only 47.4%
of MFCCs of BROKEN are classified as BROKEN. The success rate is totally 73.7%.
Table 2. The success rates of SVM
BROKENOTHERS
TOTAL
47.4%
100%
73.7%
3.2

Simulation results of Neural Network
In this research, the code of NN is written by using Keras module (Keras Documentation, 2019) on
application of Anaconda. In the analysis, two types of NN architectures are tested. Both the architectures have 3
layers, that is the input, hidden and output layer, but one has 20 nodes in the hidden layer and the other has 40
nodes. The reason why two types of NN architectures are tested is to examine the dependence on the number ofnodes in the hidden layer. As for the input layer, 20 nodes are set since the MFCC is a vector with 20 components.
The number of nodes in the output layer is defined as 1 to set the problem as two levels classification. The

51

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019activation function in the hidden layer is set the Rectified Linear Unit (ReLU) and the one in the output layer is
set sigmoid function which is commonly used for two levels classification.
In the analysis, machine leanings are conducted with 4 batch seizes (20, 25, 30, 35) and 2 epoch settings(2000, 4000). Therefore, 8 different settings are tested. The same MFCCs data set is used as the training data.
Table 3 shows the success rates obtained in this analysis.
As shown in Table 3, the number of nodes in the hidden layer, the batch sizes and the epoch numbers do
not affect the success rates in this problem. The MFCCs of OTHERS are perfectly classified to the OTHERS inany cases. About 80% of the MFCCs of BROKEN are also correctly detected as the BROKEN. Totally about
90% of the MFCCs are correctly classified. This success rate is obviously better than that of SVM.

Batch size

Epoch
2000
4000
2000
4000
2000
4000
2000
4000

20
25
30
35

Table 3. The success rates of NN
20 nodes in the hidden layer
40 nodes in the hidden layer
BROKENOTHERSTOTALBROKENOTHERS
TOTAL
82.1%
100%
91.0%
82.1%
100%
91.0%
82.1%
100%
91.0%
82.1%
100%
91.0%
82.1%
100%
91.0%
82.1%
100%
91.0%
82.1%
100%
91.0%
82.1%
100%
91.0%
80.8%
100%
90.4%
82.1%
100%
91.0%
82.1%
100%
91.0%
82.1%
100%
91.0%
80.8%
100%
90.4%
80.8%
100%
90.4%
82.1%
100%
91.0%
82.1%
100%
91.0%

Comparison between the results of SVM and NN shows that the NN gives better success rates than SVM.
All MFCCs classified as OTHERS in the simulation of NN are classified as OTHERS in the simulation of SVM,
too. Conversely, some of MFCCs belonging to BROKEN are incorrectly classified as OTHERS in SVM but
successfully classified to BROKEN in NN. The miss-classified MFCCs are mainly found out in the experimental
pattern of bending / splitting a timber.
4. ECHO DATA ANALYSIS
To investigate the feature of echo of breaking sounds, the MFCCs of ECHO are input to the NN model
described in the previous section. In the analysis, first, the NN models are trained using only the data of BROKEN
and OTHERS which are the same to the data used in the previous analysis. Then, 120 MFCCs of ECHO are input
to the trained NN model and the output results are investigated. Table 4 shows the rates of ECHO classified to
BROKEN. The settings of number of nodes in the hidden layer, batch sizes and epoch numbers are the same tothe previous section.
As shown in Table 4, about 67% of ECHO is recognized as the breaking sound of woods in this simulation.
The same 40 MFCCs are almost always recognized as OTHERS in any settings. To investigate the difference
between the samples recognized as BROKEN and OTHERS, the components of MFCCs are visualized in Figure
3. The x- and y-axis represent the component number of MFCC and the values of components, respectively. The
green lines represent the MFCCs of BROKEN. The red lines are the MFCCs of EHCO recognized as BROKEN
and the blue lines are them recognized as OTHERS. The behaviors of blue lines are apparently different from theothers especially in the lower component numbers.
There is no clear relationship between the MFCCs of BROKEN and ECHO. Even if the MFCCs of
BROKEN are defined as OTHERS, some of the MFCCs of ECHO calculated from the same experiment are
defined as BROKEN. There is the reverse, too.

Batch size
20
25
30
35

Table 4. The results of classification of ECHO as BROKEN
20 nodes in the hidden layer 40 nodes in the hidden layer
Epoch
BROKENBROKEN
2000
67.5%
67.5%
4000
67.5%
67.5%
2000
66.7%
66.7%
4000
67.5%
67.5%
2000
66.7%
66.7%
4000
67.5%
67.5%
2000
66.7%
65.8%
4000
67.5%
67.5%

52

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

1.0
ECHO recognized as OTHERS

0.8

ECHO recognized as BROKEN

0.6

BROKEN using for learning

Value of MFCCs

0.4
0.2
0
-0.2
-0.4
-0.6
-0.8
-1.0

1

3

5

7
9
11
13
Element number of MFCC

15

17

19

Figure 3. Behavior of components of MFCCs

1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

Value of MFCCs

Value of MFCCs

Figure 4 shows two examples of MFCCs mentioned above. The x- and y-axis are the same as Figure 3.
The green lines represent the MFCCs of BROKEN which NN models learned correctly. The red lines are the
MFCCs of BROKEN which are not recognized as breaking sounds. The blue lines are MFCCs of ECHO which
are not recognized as breaking sounds. The orange lines are the MFCCs of ECHO recognized as breaking sounds.

0
-0.2
-0.4

BROKEN recognized as breaking sounds
BROKEN not recognized as breaking sounds
ECHO not recognized as breaking sounds

-0.6
-0.8
-1.0

1

3

5

7
9
11
13
Element number of MFCC

15

17

0
-0.2
-0.4

BROKEN not recognized as breaking sounds-0.6

ECHO not recognized as breaking sounds

-0.8
-1.0

19

ECHO recognized as breaking sounds
1

3

5

7
9
11
13
Element number of MFCC

15

17

19

Figure 4. Two examples of BROKEN and ECHO MFCCs calculated from the same experiment
5. CONCLUSIONSIn this research, we investigate the feasibility of detecting breaking sounds of woods by means of machinelearnings. First, some experiments are conducted to record the breaking sound of woods and the others such as
crashing dishes, talking voices and door closings. MFCCs are calculated from the recorded sound as an acoustic
features and input to the sound recognition models. In the analysis, SVM and NN models are tested. As the results,
NN models give better success rates than SVM. We also study the acoustic features of echo of breaking sounds by
inputting them into the trained NN model. As the result, some of echo are recognized as the breaking sounds ofwoods but the others are not. The reason is not revealed yet.
In future work, we will try to deconvolute mixed sounds into several sounds using NN models because
various sounds simultaneously occur in case of large earthquake. Besides, further experiments of recording sounds
will be conducted since the number of sounds recorded in this research is not enough to make sure the feasibilityof the present method.

53

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

REFERENCES
Anaconda. (2019). Retrieved from Anaconda website: https://www.anaconda.com/distribution/
Kawahara, T. (2015). State of the Speech Recognition Technology. The Journal of the Institute of Electronics,
Information and Communication Engineers, Vol.98, No.8, 2015, 710-717. (in Japanese)
Keras Documentation. (2019). Retrieved from Keras Documentation website: https://keras.io/
Logan, B. (2000). Mel Frequency Cepstral Coefficients for Music Modeling, Proc. of the International
Symposium on Music Information Retrieval (ISMIR) 2000, Plymouth, USA.
Müller, K. R., Mika, S., Rästch, G., Tsuda, K., and Schölkopf, B. (2001). Introduction to Kernel-Based Learning
Algorithms, IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL.12, NO.2, 181-201.
Sklearn.svm.SVC. (2019). Rettived from scikit-learn website: https://scikitlearn.org/stable/modules/generated/sklearn.svm.SVC.html
Tani, M., Komatsu, T., Narisetty, C., Kondo, R. (2017). NEC Acoustic Situation Awareness Technology and Its
Feasibility Test in Singapore, The Institute of Electronics, Information and Communication Engineers,
Technical Report PRMU2017-30, SP2017-6, 29-32. (in Japanese)
Zhang, Z., Li, Q., Huang, A., Wu, J., Tenenbaum, J. B., Freeman, W. T. (2017). Shape and Material from Sound,
31st Conference on Neural Information Processing Systems, Long Beach, CA, USA

54

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019COMPUTER VISION-BASED IN-BUILDING HUMAN DEMAND ESTIMATIONFOR INSTALLATION OF AUTOMATED EXTERNAL DEFIBRILLATORS
Wen-Xin Qiu1, Albert Y. Chen2
1) Graduate student, Civil Engineering, National Taiwan University, Taipei, Taiwan. Email: r07521505@ntu.edu.tw
2) Assoc. Prof., Civil Engineering, National Taiwan University, Taipei, Taiwan. Email: AlbertChen@ntu.edu.tw

Abstract: To support a better indoor location distribution of automated external defibrillators (AEDs), the realdemand, the actual human distribution, should be known. In this research, a computer vision-based process is
proposed. The data are collected in the form of image sequences, such as surveillance cameras, in the building.
Image based human detection and tracking are applied, and the depth estimation from a deep neural network is
utilized to estimate the location of each person. The appearing and disappearing locations are clustered to estimate
the node in the indoor traveling network to other spaces in the image. The amount of people are accumulated, forlocations in the building, to serve as the demand distribution, and a network based model for AED location
optimization can be further integrated.
Keywords: Automated external defibrillators, Human detection and tracking, Depth estimation
1. INTRODUCTION
Automated external defibrillator (AED) is a medical device for treating people with sudden cardiac arrests.
Recommended to be placed in crowded working areas such as office buildings (U.S. Dept. of Labor, 2003), AEDs
improve survival odds and they also have the characteristics of easy-to-use. The concept of public access
defibrillator (PAD) is proposed. The optimal response time of accessing AED is stated to be less than 3 minutes.
However, there is a lack of standard of the appropriate number and location for AEDs (U.S. Dept. of Health andHuman Services and General Services Administration, 2009). In Taiwan, the regulations only state the type of
public places in which PADs need to be installed; the precise indoor location as to where to install PADs is not
specified (Public Emergency Medical Service Equipment Control Regulations, 2013).
There are some research efforts on location of AEDs by 3D indoor spatio-temporal location modeling(Dao et al., 2010) and location in high-rise buildings (Chan, 2017). Nevertheless, the demand is usually an assumed
number or ratio through experience. We hope to estimate the real demand by obtaining actual human distribution
in the building to have a better judgement of where and how many PADs should be installed in a building.
For observation of the human distribution, extraction of information from image sequences is considered
to be a reasonable way. The existing surveillance cameras could be used as sensors. For example, the occupancy
distribution is estimated using the surveillance cameras in buildings (Zhang & Jia, 2017). The computer-vision
based method is also proposed for the AED location problem (Chen & Chen, 2017). However, the previous
research works still need some manual process to make the model perceive the correspondence between the images
and the geometric or network model, which is worthy of investigation.
2. METHOD
The objective of the research is to collect image sequences from surveillance cameras and accumulate thehuman counts in each area to estimate the demand of the indoor location model of AED. The process is preferred
to be highly automatic. The input is the image sequences and the corresponding network model. The output shouldbe the demand on the nodes in the network.
2.1 Network
The network model is generated from the geometric model of the building (Chen & Huang, 2015). Nodesof the network represent rooms and spaces, and the demand is accumulated on nodes. The links connects nodes
across doors and stairs. Locations of surveillance cameras are specified, including the field of view (FoV) as shownin Figure 1.

$BNFSB 1 'P7$BN2 'P7

Figure 1. Network and camera locations
Camera Location
The locations of the cameras are given. Most of the cameras are located at the hallways and corridors. The
rooms do not need to be seen by installed cameras, but can be observed by the entrances or doors of the rooms, i.e.
the links connected to the node.

2.2

55

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2.3

Image Sequence Processing Approach
The approach of accumulating humans from image sequences is show un Figure 2. The process starts
from obtaining image sequences from surveillance cameras. Then the following steps are carried out.

Video (images)
from cameras

Depth estimationon the video(relative depth)

Human detection
and tracking

Clustering
the points that humans
appear/ disappear

Model:
1. Distance of links
2. Location anddirection ofcamerasis known

Assigning clusters to
nodes(= doors or exits)

Recording the count ofeach node

Figure 2. Approach(1) Human Detection and Tracking
The first step is to detect human in the images. Tracking is to ensure a single person is not counted twice
to the same node. The model we adopt in the process is YOLOv3 (Redmon & Farhadi, 2018) for human detectionin each frame. Then a tracking model extending from Simple Online and Realtime Tracking (SORT) (Wojike etal., 2017) is implemented.
(2) Depth Estimation
The image depth of the detected human is to determine the location of the human in the network. Depth
can be estimated from the image by several approaches, such as using RGB-D cameras or calculating disparity todepth from stereo images. In this research, the monocular depth estimation is adopted. The basic concept is training
and estimating depth from every single image taken by a monocular camera by a deep convolution neural networkmodel (Eigen et al., 2014). In recent research works, stereo images or video sequences are used for training theunsupervised model (Godard et al., 2017; Zhou et al., 2018).
(3) Clustering
After getting the tracks and the corresponding depth of human, the appearing/disappearing points ofhuman in the image is recorded. The points are determined based on: (a) a person coming in from or going out ofa connected space through a door; (b) a person stays at the same space but goes in/out of the FoV of the camera;and (c) the failure of the tracking model. Here (c) is what we do not wish to get.
Although each point has its estimated depth already, the clustering step is to recognize the locations ofentrances and doors in the image. The entrances and doors are assumed to have multiple people coming in and out,
so each cluster can represent an entrance or door.
(4) Assignment
Each track is assigned to the node in the FoV as a temporal count. We pay attention to the counts of thenodes coming in/out the rooms, because people usually stay in the rooms longer than on the hallway. Each point
contains the in/out data, location in the image, timestamp, depth from the camera, and cluster ID. The count of thenode in the room is added a count if the point belongs to the cluster correspond to the node, and vice versa.
3. RESULTS
The method is performed on a test case of a 3 minutes image sequence. It is taken by a single camera at a
hallway in the building of the Department of Civil Engineering at National Taiwan University.
3.1 Network
A part of the network is used in the test case. The details of the network are shown in Figure 3.

CameraԮԯԱ԰Figure 3. Network used for test case

56

Node
(Room)
Camera1

Distance (cm)

2
3
4

736
1260.42226.40250

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3.2

Image Sequence Processing
The package for detect and track is called deep_sort_yolov3 (Qidian213, 2019). In the image sequence,
25 persons are tracked by the detect and track, and the ground truth are 20 persons in total. Most of the persons
are tracked once, but one of the tracks is split to several parts.
The depth estimation model adopted in our approach was implemented by Godard et al. (2017), and thedepth is calculated from the disparity between the image pair by Equation 1.

d" =

b×f
d

(1)

where d" is the depth, b the baseline, f the focal length, and d the disparity.
Since we do not tune the model by ourselves, the baseline and focal length remains unknown. We use(1/disparity) to estimate the relative depth. The estimation is performed on the 5405 frames from the 3 minutes
image sequence. Figure 4 shows the original input image and the estimated depth map, and Figure 5 plots thedepth of each track.

Estimated depth

Figure 4. Original image and estimated depth map

frame #

Figure 5. Estimated depth of tracks
From the 25 tracks, 50 points are generated. Three cases for extracting depth value are considered: (a)depth of the location of the foot; (b) depth of the center of the human; and (c) median over the bounding box.
("1, %1)

center ∶". + "' %. + %'
,
2
2

1223:
". + "', %'2("' , %' )Figure 6. The center and foot of a detected humanSome cluster methods are tested. The K-means clustering is believed to be reasonable for our case.
Numbers of clusters from 1 to 9 are tested based on the three cases of depths. From inertia and elbow analysis,shown in Figure 7, cluster numbers 3, 4, or 5 may be the best choice. However, from the visualization in Figure
8, K = 5 and using depth of foot separates the doors best.

57

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 7. Inertia (SSE, L2-norm) of clusters

Figure 8. Some results after clustering
Additionally, the points containing and without depth information is trialed to proof the depth informationis required to have a more reasonable output, as Figure 9 shows.

Figure 9. Left: Clustering with (x,y) but without depth. Doors of node 3 and 4 are mixed.
Right: Clustering with depth. Green ones are the points coming in/out the FoV.
Finally, the case of K = 5 and using depth of foot is chosen for further analysis, Figure 10 illustrates theclusters and the corresponding doors.

58

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Ա

԰

ԯ

Ԯ

Figure 10. Clusters and doors
The depth of each cluster is extracted and compared to the first door. The error between the estimateddepths of clusters and the measured distances in the network are within 30% and have a high correlation inregression, as shown in Figure 11. The detailed numbers are depicted in Table 1.
Depth of clusters
200= 0.9748

180

Estimated depth

160
140
120
100
80
60
40
20
0
0

500

1000

1500

2000

2500

Network distance (cm)Figure 11. Relationship between distance and estimated depth
Table 1. The distance (depth) of the clusters and the distance of the network (can be seen as ground truth)
Cluster
Network
NodeError of scaledistance
scale
distance
scale(Room)
X
348.777
15.2304
185.0281.000
2226.41.000
0%
3121.9410.659
1260.40.566
16%
243.437
0.235
736
0.331
29%
1
22.901
0.124
250
0.112
10%
camera0000After the assignment, the counts are accumulated to each node. Table 2 shows the count of the nodes in
the room.
Table 2. The count of testing case and the ground truth (counted manually)
NodeCluster
Ground truth(Room)
in
out
in
out
X
4
4
2
1
4
6
7
5
7
3
7
7
5
5
2
5
5
4
4
1
3
2
2
2
4. DISCUSSION
From the tracking step, errors start to occur. The most is due to the occlusion between humans in theimages. The tracking model can be changed for a better performance, and would not change the whole procedure.

59Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Similarly, the depth estimation model can also be changed for obtaining the absolute depth. This may provide theapproach with more reliable data since the distances may not need to match based on their ratio.
In the clustering tests, the best cluster number is 3, 4, or 5 from analysis, and determined to be 5 in thevisualization with the network. There is the consideration of the (x, y) of points of different doors are close in theimage due to the perspective. Figure 9 also supports that besides (x, y), if the depth is used for clustering, theperformance of splitting door 3 and 4 is better.
5. CONCLUSIONSIn this research, we proposed a computer-vision based method to deal with the collection of human
distribution for a building. The procedure combines multiple information from the images, including humantracking and depth estimation, to have an improvement on automation of human distribution data collection.
More testing image sequences are being collected to find the outliers and the weak part of the method.
Additionally, the location of cameras is considered to be limited by the FoV.
REFERENCES
Chan, T. C. Y. (2017). Rise and shock: Optimal defibrillator placement in a high-rise building, Prehospital
Emergency Care, 21 (3), 309-314.
Chen, C.-C., and Chen, A. Y. (2017). Video-Based Indoor Human Detection for Decision-Making of the
Installation Locations for Automated External Defibrillators, International Workshop on Computing in
Civil Engineering (IWCCE 2017), Seattle, WA, USA, June 25-27.
Chen, A. Y. and Huang, T. (2015) Toward a BIM Enabled Decision Making for in-Building Response Missions.
IEEE, Transactions on Intelligent Transportation Systems, Vol. 16, No. 5, pp. 2765-2773.
Dao, T. H. D., Zhou, Y., Thill, J.-C., and Delmelle, E. (2012). Spatio-temporal location modeling in a 3d indoor
environment: the case of aeds as emergency medical devices. International Journal of Geographical
Information Science, 26 (3), 469-494.
Eigen, D., Puhrsch, C., and Fergus, R. (2014). Depth map prediction from a single image using a multi-scale
deep network, Advances in Neural Information Processing Systems, 3, 2366-2374. Neural information
processing systems foundation.
Godard, C., Aodha, O. M., and Brostow, G. J. (2017). Unsupervised monocular depth estimation with left-rightconsistency, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6602-6611.
Public Emergency Medical Service Equipment Control Regulations [公共場所必要緊急救護設備管理辦法].
(2013). R.O.C. (Taiwan).
Qidian213. (2019). deep_sort_yolov3. Retrieved from github: https://github.com/Qidian213/deep_sort_yolov3
Redmon, J. and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767.
U.S. Department of Health and Human Services and General Services Administration. (2009). Guidelines for
Public Access Defibrillation Programs in Federal Facilities, 41133-41139.
U.S. Department of Labor. (2003). Saving Sudden Cardiac Arrest Victims in the Workplace. Retrieved from
Occupational Safety and Health Administration website: https://www.osha.gov/Publications/3185.html
Wojke, N., Bewley, A., and Paulus, D. (2017). Simple online and realtime tracking with a deep association
metric, 2017 IEEE International Conference on Image Processing (ICIP), 3645-3649.
Zhang, C. and Jia, Q.-S. (2017). An occupancy distribution estimation method using the surveillance cameras inbuildings, 2017 13th IEEE Conference on Automation Science and Engineering (CASE), 894-899.
Zhou, T., Brown, M., Snavely, N., and Lowe, D. G. (2017). Unsupervised learning of depth and ego-motion
from video, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
1851-1858.

60

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

MULTI-DIMENSIONAL SEQUENCE ALIGNMENT FOR CONTEXT-AWARE
HUMAN ACTION ANALYSIS OF BODY-SENSOR DATA
Nipun D. Nath1, Amir H. Behzadan2, and Prabhat Shrestha3
1) Ph.D. Student, Zachry Department of Civil Engineering, Texas A&M University, College Station, Texas, USA. Email:nipundebnath@tamu.edu
2) Associate Professor, Department of Construction Science, Texas A&M University, College Station, Texas, USA. Email:abehzadan@tamu.edu
3) M.S. Student, Department of Construction Science, Texas A&M University, College Station, Texas, USA. Email:prabhat1993@tamu.edu

Abstract: Time-motion human data are critical to analyzing activities of construction field crew and their
spatiotemporal interactions. A major limitation of the current practice, however, is that the context in which datais collected is rarely incorporated in data analysis. This paper investigates the problem of incorporating context
into human activity recognition (HAR). This is achieved by using a multi-dimensional sequence alignment (MSA)
method that transforms raw body-mounted sensor data to basic human actions in a one-step process. In this paper,an action is defined as a single, isolated effort (e.g., kneeling, pushing), while an activity refers to an ongoingprocess over a period of time (e.g., pipefitting, welding). When grouped together, actions performed in a specificsequence form an activity. The designed MSA method enables the recognition of human actions throughcomparing the similarities between several time series sequences of body-sensor data, thus fusing contextual
information on temporal dependency with classification. The method is tested on a publicly available dataset insubject-dependent and subject-independent classifications. It is found that in both cases, the 5-fold cross-validation
classification accuracy is >97% which is on par with or higher than the performances achieved in previous studies.
Keywords: Multi-dimensional sequence alignment, machine learning, human activity recognition, smartphone
sensors.
1. INTRODUCTION
Recent advancements in sensing and instrumentation technologies have resulted in a wealth of data inarchitecture, engineering, construction, and facility management (AEC/FM) domains. This has provided great
opportunities for improving the quality and timeliness of decision-making through the use of data analytics anddata-driven decision support tools. The widespread adoption of data-driven applications at the operations level forcrew analysis, however, is still hindered by key challenges such as low data quality and inherent noise (Zamalloa& Krishnamachari, 2007), rigidity of data processing frameworks (Islam et al., 2012), and the absence of context
integration in data analysis (Akhavian, 2015; Leite et al., 2016, Shrestha & Behzadan, 2017), leading to low
reliability of the generated process-level knowledge. In particular, incorporating context in machine learning (ML)classification is not trivial. Previous research in other domains has shown that adding contextual information cansignificantly increase the performance of data analysis. For instance, in a land cover classification project,integrating contextual information (e.g., edge detection, iterative centroid linkage) allowed users to add or remove
classification components (e.g., labels assigned at pixel level) (Stuckens et al., 2000). In another application,
relevant items were recommended in an online marketplace when contextual knowledge was added to theinformation on buyer’s item of interest (Adomavicius et al., 2005). Similarly, researchers were able to improveclassification accuracy of visual data using deep learning, particularly, convolutional neural networks, by merging
information of multiple adjacent pixels rather than information contained in a single pixel (LeCun et al., 2015).
In this paper, the prospect of using a bioinformatics technique called multi-dimensional sequencealignment (MSA) for improving the reliability of human activity recognition (HAR) is examined. In the past, HAR
using vision-based or sensor-based techniques has shown to provide useful information in a variety of applications
such as ergonomic assessment (Nath et al., 2018; Chen et al., 2017; Cheng et al., 2012), safety monitoring (Lee etal., 2011; Golparvar-Fard & Niebles, 2013; Ray & Teizer, 2013; Zhang et al., 2015; Ding et al., 2018), andproductivity measurement (Gong & Caldas, 2009; Gong & Caldas, 2011). The use of bioinformatics methods inthis research is inspired by the fact that living organisms at both macro (nature) and micro (species) levels are
extremely data-intensive yet superiorly efficient in processing vast amounts of data generated over time and acrossmultiple generations, making them capable of thriving despite significant anomalies. For example, during the DNA(deoxyribose nucleic acid) replication, nucleotides (i.e. individual instances) are copied from parental DNA to thechild DNA while preserving the genetic information coded in the sequence of nucleotides (i.e. context) in the child
DNA (Rosenberg, 2009).
MSA is an extended form of sequence alignment (SA), a bioinformatics technique for evaluating thedegree of similarity between two strings of data by aligning one sequence with another using a series of heuristic
or probabilistic methods (Rosenberg, 2009). The authors have previously designed and tested a one-dimensional
SA technique (Shrestha et al., 2018) to post-process the output of HAR from body-mounted smartphone sensors

61

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019(accelerometer, linear accelerometer, and gyroscope) by incorporating knowledge (i.e. context) about dominantaction sequences into the analysis. The goal of the authors’ previous study was to test the performance of several
ML algorithms in detecting basic human actions (i.e. climb up/down the stairs, walk, squat, lift, jump). To note,these actions are the building blocks of more complex activities performed by construction workers (Nath, 2017).
For example, in a construction site, activities such as carrying or loading/unloading materials (e.g., drywall, bricks,
pipes) can be broken down into a sequence of basic actions (e.g., squat, lift, pull/push, walk). Thus, designingalgorithms that can achieve significance performance in classifying basic human actions can ultimately help
improve the classification accuracy of more complex construction activities. In light of this, the MSA method
introduced in this paper aims at streamlining (while maintaining accuracy and computational efficiency) the
previously developed two-step process (HAR followed by SA post-processing) of recognizing basic human actions
from raw sensor data. The developed approach is validated on a publicly available dataset containing sensor
recordings for general applications in HAR.
2. LITERATURE REVIEW
2.1 Fundamentals of Sequence Alignment (SA)SA was originally developed as an alternative method for investigating the trends and comparing
information in DNA, RNA (ribonucleic acid), and protein sequences, by answering questions such as the function
or source organism of an unknown gene sequence, the relatedness of organisms, or grouping of sequences fromclosely related organisms (Copasaro, 2018). These alignment principles were later expanded to other domains suchas social sciences (Gauthier et al., 2010) to advance the analysis of socio-economic data by producing normalized
data trends and comparing each data point to the trend. SA has also been used in the development of linguistics
algorithms to generate sentence-level paraphrases from unannotated corpus data (Barzilay & Lee, 2003) andanalysis of the sequential aspects within the temporal and spatial dimensions of human actions (Shoval & Isaacson,2007). Furthermore, Huang et al. (Huang et al., 2010) illustrated the application of SA in studying dynamic humaninteractions using passive radio-frequency-identification (RFID) from objects (describing parameters such as
location, motion, and orientation) to train a model for recognizing daily human actions in a home environment.
The variations between different instances of the same person and different people performing the exact sameactions were dealt with by using SA to recognize common patterns of change for each action.
As shown in Figure 1, SA evaluates two data sequences and expresses the degree of similarity between
these sequences by measuring the number of operations required to make them identical. These operations include
deletion, insertion, and substitution, where an element in the target sequence is removed, added, or replaced,
respectively (Shoval & Isaacson, 2007). The score of similarity produced as a result is inversely proportional to
the number of operations required to reach at identical sequences.

Figure 1. The three primary alignment operations in SA algorithm

Multi-dimensional Sequence Alignment (MSA)
MSA is an expanded form of SA in which more than one attributes (i.e. dimensions) of source and target
sequences are compared, and degrees of similarity obtained from the comparison in each of the attributes is used
to calculate an aggregate score. With the advent of new computing capacities, and the expansion of SA beyond
bioinformatics, the principles of SA were also expanded to include evaluation of sequential datasets containing
more than one attribute (Joh et al., 2002). For example, in experiments previously conducted by the authors(Shrestha et al., 2018), SA post-processing was applied to the output of HAR, which was essentially a vector (1dimensional sequence) of action labels. These action labels constituted the attributes of the target (predicted)
sequence and were compared with a source (actual or ground truth) sequence using a scheme similar to Figure 1,
in order to identify and correct anomalies in action recognition. In contrast, MSA looks at the multi-dimensionalraw sensor data (preceding HAR) and compares target and source sequences by aligning them across multiple
sensor data dimensions. In this paper, the authors demonstrate that using this approach, the two-step process of
HAR (for initial action recognition) and SA (for post-processing) can be replaced with a single-step MSA
implementation.
In principle, both MSA and HAR use some of the same ideas from supervised (inductive) ML. However,
although both algorithms use labeled (training) data to classify unlabeled (test) data, they vary in how they use
prior information. HAR is based on defining a variety of distinctive statistical features for a particular window,
2.2

62

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

while MSA relies on the fact that various classes can be represented by the differences in the sequence of
information within a particular time window. Thus, the general trend in the sequence is used as the basis ofcomparison and classification in MSA. A basic comparison between HAR and MSA is illustrated in Figure 2.

Figure 2. Basic comparison of HAR and MSA methods
In Figure 2, a target sequence is compared against a source sequence using HAR and MSA. In MSA, thediscretized form of the continuous sensor data is compared, whereas HAR uses extracted features. The former
utilizes the variation within the sequences whereas the latter produces several features that are constant within thewindow. This new approach comes with several advantages, namely the flexibility in choosing window sizes andthe starting position for each window, and the ability to choose a variable overlap length between windows thatcan be changed during the implementation. In contrast, changing the starting position or the overlaps in HAR
requires re-extraction of features, which can significantly complicate the implementation and reduce
computational efficiency. In particular to extracting information on construction worker actions, the advantagesof MSA are unmatched since the pace of human motions in performing different actions varies significantly (e.g.,quickly lifting a brick versus slowly plastering a wall), and thus being able to select a variety of window sizes
might be helpful for more accurately detecting these actions.
3. METHODOLOGYAs previously stated, the main goal of MSA implementation in this research is to classify an unknown
sequence of raw sensor data belonging to a particular action by comparing it with several data sequences withknown action classification. These comparisons are performed across several dimensions, where a dimension isdefined as data along a specific axis for a particular sensor (e.g., accelerometer data along z-axis). Therefore, thetotal dimensions of collected data is the product of the number of sensor units, types of sensors in each unit (e.g.,
accelerometer, gyroscope, magnetometer), and the number of axes (e.g., x, y, z axes) along which data is collectedfrom each sensor. The alignment process is conducted individually for each dimension of corresponding data ofsource and target sequences and a score of sequence alignment is obtained. The scores collected from allcomparisons are then used to collectively assess the classification of the target sequence. The process by whichscores are calculated and evaluated is illustrated in the phase diagram of Figure 3.

Figure 3. Basic comparison of HAR and MSA methods

63Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

As shown in this Figure, the process consists of first collecting data required to implement the algorithm,
followed by a training phase where the optimal combination of parameters for MSA is identified, and a testing andclassification phase where action labels for unknown sequences are generated. These steps are explained in thefollowing Subsections.

Training Phase
In this phase, correctly labeled data is used as source sequence (i.e. ground truth) against which unknown
data points are compared. The time series data first go through a normalization process to remove subjectdependency, as different people perform the same actions differently. Once normalized, time derivative (i.e., slope)of each sensor reading is calculated to factor in the general trend between successive normalized amplitude values.
If readings of a particular sensor are 𝑎𝑖 and 𝑎𝑖−1 at time stamp 𝑡𝑖 and 𝑡𝑖−1 , respectively, the slope of the𝑎− 𝑎reading at 𝑡𝑖 is calculated as 𝑖 𝑖−1 . This is repeated for data across all dimensions, actions, and subjects.
𝑡 − 𝑡
3.1

𝑖

𝑖−1

The comparison principles used in MSA are primarily based on ordinal categories and therefore,
continuous sensor data needs to be discretized (Garcia et al., 2013). Discretization is a commonly used techniquefor data reduction while improving performance, especially in supervised ML applications. In this research, global
discretization in a direct equal-frequency splitting framework is implemented, as described by Liu et al. (2002).
Global discretization methods incorporate information available in the entire space, therefore the obtained discrete
data can be examined in the context of all available data points (Garcia et al., 2013). Furthermore, in order tonegate the effect of outliers, frequency splitting techniques (Garcia et al., 2013) are used which rely on positional
information such as percentiles to determine the boundaries of categories.
In particular, percentile ranks are used to discretize the continuous data points and assign each point to
one of 20 categories by comparing against all the data points available in a particular dimension. Each categorycontains 5 percentile ranks, i.e., the first category includes data in the 0th to the 5th percentiles, the second category
includes data in the 6th to 10th percentiles, and so on. It is worth mentioning that the number of categories in this
implementation is limited to 20 since existing SA algorithms were originally designed for bioinformatics
applications to compare sequences of amino acids. Given that most organic matter is made up of 20 basic aminoacids (Simoni et al., 2018), current SA algorithms are limited to an alphabet representing the 20 amino acids.

Identification of Optimal SA Parameters
The accuracy of action classification is dependent on the scores derived from SA which in turn are affectedby a number of parameters. The optimal set of parameters is selected from a pool, and comprise those that perform
most consistently across different datasets. This is ensured by implementing cross validation in the source data
across different folds. The combination of parameters that produces the highest average accuracy is selected as the
optimal combination. The parameters that can be varied are as follows, Number of data points compared: Changing this parameter can affect the time interval used in thecomparison. For example, a sequence with 50 data points will cover 2 seconds in a 25 Hz dataset, whereas
one with 100 data points covers 4 seconds.
 Scoring matrix criteria: A score generally increases for a positive match and decreases in case of anegative match. Here, different combinations of ratios of positive and negative matches can be tested.
Moreover, since the comparison is based on categories derived from continuous data, the distance
between the various categories also has significant implications in terms of the general trend. For instance,a mismatch between categories representing the 4th percentile and 6th percentile would have lesssignificance compared to a mismatch between categories representing the 5th percentile and 97th
percentile. Thus, the negative score assigned to the former mismatch would presumably be different thanthe one assigned to the latter.
 Number of comparisons: In each iteration, the target sequence is aligned with a number of sources
sequences, and this is varied as well. In theory, best results occur with the maximum number ofcomparisons. However, this is tested by varying the number of comparisons.
 Comparison window overlaps: In a dataset collected from an uncontrolled environment, it is difficult tocomputationally identify when a new action cycle begins or how long an action cycle lasts. Moreover,this can vary across instances and among different people. In order to minimize the effect of out-of-phase
SA comparisons, overlaps between different comparison windows can be used to reduce the number ofsuch comparisons. This process is also essential to identify the most useful metric of overlap.
3.2

Testing and Classification Phase
Similar to the source sequence, for the target sequence, normalized slope values are calculated and thendiscretized into 20 ordinal values. However, the percentile ranks of the target sequence data points is identified bycomparing against available source sequence values. The reason is that in online classification (i.e., classification
performed as new data points are added), only a limited sample of the target sequence is available at the time of
3.364Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019classification. As illustrated in Figure 4, each target sequence window is compared against several source sequence
windows, across all dimensions. Each comparison produces a score, resulting in a collection of scores dependingon the parameters of comparison. Each SA comparison is conducted with sequences of 𝑠 × 𝑓 data points, where𝑠 is the size of the window (in seconds) and 𝑓 is the frequency of data collection (in Hz).

Figure 4. Score matrix generation for a particular target and source window across actions and dimensions
For a given known action 𝑥, dimension 𝑛, and window 𝑘 in the source sequence, a corresponding score
𝑆𝑥,𝑛,𝑘 is calculated. The dimension sum score (𝑆𝑆𝑥𝑘 ) is then calculated by adding individual scores across all
dimensions, i.e., 𝑆𝑆𝑥𝑘 = ∑𝑛 𝑆𝑥,𝑛,𝑘 . For instance, in the process illustrated in Figure 4, a target data sequence
window with 𝑁 dimensions is compared with source sequence windows. This is repeated for each of the actions
and a score matrix containing the scores of alignments for all dimensions and actions for this particular target
sequence window is obtained. The sum of the scores across different dimensions yields 𝑆𝑆𝑥𝑘 , as illustrated in
Figure 4. The calculation of 𝑆𝑆𝑥𝑘 is performed for every labeled window of each action in each source sequence.
Using this holistic approach, the comparison yielding the highest 𝑆𝑆𝑥𝑘 determines the action label of the unknown
window in the target sequence. Since there are 𝑘 windows in the source sequence, this step identifies 𝑘 actions.
Finally, majority voting is used to select the action label of for the particular window in the target sequence.
4. EXPERIMENTThe experimental data used in this research is obtained from the University of California Irvine (UCI)
Machine Learning Repository (Lichman, 2013) which houses a publicly available dataset (Barshan & Yüksek,
2014), containing sensor recordings collected for general applications in HAR. The dataset contains data from 8
subjects (4 males and 4 females, between the ages of 20 to 30) performing five common human actions (stand,walk upstairs, walk, run on a treadmill, and jump) in indoor (a sports hall building) and outdoor areas. Subjects
performed each action for 5 minutes while data was collected from 5 sensor units mounted on left and right arms,
left and right legs, and torso. Data was collected at 25Hz along x, y, and z axes, and each sensor unit comprised ofan accelerometer, a gyroscope, and a magnetometer. Table 1 contains a description of the collected data.

Category
Actions
Subjects
Sensor locations
Sensor types
Orientation
Dimensions
Frequency
Data duration
Total data points

Table 1. Summary of different parameters of the input dataset
Count
Description
5
stand, walk upstairs, walk, run (on treadmill), jump
8
4 males, 4 females, 20-30 years old
5left and right arms, left and right legs, torso
3
accelerometer, gyroscope, magnetometer3
along x, y, z axes
45
5 data units × 3 sensor types × 3 orientations = 45
25 Hz
25 data points per second
5 min.
per action per person
300,000
8 people × 300 seconds × 25 Hz × 5 actions65Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

5. DATA ANALYSIS AND DISCUSSION
5.1 Evaluating the Effectiveness of MSA for Action Identification
The UCI dataset is used for subject-dependent and subject-independent classification. The MSA algorithm
was implemented in High-Performance Research Computing (HPRC) clusters at Texas A&M University (HPRC,
2018).
For subject-dependent classification, for each subject, 25 minutes of available data is divided into 15
minutes of training and 10 minutes of testing data constituting a breakdown of 60%-40% for training and testing.
The training sample is used to identify the optimal combination of parameters of the most accurate classification.
To achieve this, a 5-fold cross validation is implemented with different combinations of score matrix, window
length, and the particular sensors used. Figure 5(a) illustrates the classification confusion matrix obtained for all
subjects using a 45-dimensional MSA applied to data from all 5 sensors. In this implementation, 4 of the 5 actions,
namely stand, walk upstairs, walk, and run are classified with very high accuracy (100%, 99%, 100%, and 100%,
respectively), whereas the accuracy of classifying action jump is slightly lower, and this action is confused withaction walk upstairs in ~6% of instances.

Figure 5. Confusion matrix for subject-dependent (a) and subject-independent (b) MSA classification using datafrom all 5 sensors (45 dimensions)
The designed MSA algorithm is also tested for subject-independent classification. In this case, data from
4 of the 8 subjects is designated as training sample, whereas data from the remaining 4 subjects is used for testing.
This selection of training and testing subjects is done randomly and repeated 8 times to produce 8 different
combinations of training and testing samples. Figure 5(b) illustrates the confusion matrix of subject-independentclassification implemented on these 8 combinations using 45-dimensional data collected from all 5 sensors.
Comparing Figures 5(a) and 5(b), it is seen that the classification accuracies are almost comparable with a slightdecrease in some cases for subject-independent classification. For example, while actions stand, walk upstairs, andrun are classified with extremely high accuracy in both scenarios, classification accuracy of actions walk and jump
decreases by 7% (from 99% to 92%) and 5% (from 94% to 89%), respectively in subject-independent classification.
This reaffirms that variations in how subjects perform can impact the overall accuracy of action classification.
5.2

Benchmarking the Performance of the Designed MSA Implementation
The effectiveness of the designed MSA implementation is benchmarked against previous work that used
the same dataset in subject-independent implementation. As listed in Table 2, the algorithms implemented inreference applications include Bayesian decision-making (BDM), rule-based algorithm (RBA), least-squares
method (LSM), k-Nearest neighbor (k-NN), dynamic time warping (DTW), support vector machines (SVMs),
artificial neural networks (ANN) (Altun et al., 2010), and time series shapelets (Liu et al., 2015). Table 2 indicatesthat the classification accuracy achieved by the designed MSA approach in this research (97.5% for subjectindependent classification) is on par with or higher than those achieved in previous studies. The highest accuracy
was achieved by Altun et al. (2010) using the BDM algorithm.
Table 2. Comparison of classification accuracies obtained from different algorithms
Classification algorithm
Reference
Accuracy
Bayesian decision making (BDM) Altun et al. (2010)
99.1%
Rule-based algorithm (RBA)
Altun et al. (2010)
81.0%
Least-squares method (LSM)
Altun et al. (2010)89.4%
k-Nearest neighbor (k-NN)
Altun et al. (2010)
98.2%
Dynamic time warping (DTW)
Altun et al. (2010)
82.6%
Support vector machines (SVMs) Altun et al. (2010)
98.6%
Artificial neural networks (ANN) Altun et al. (2010)
86.9%
Time series shapelets
Liu et al. (2015)
87.3%
HAR-SA (two-step)
Shrestha et al. (2018) 98.0%
This work: MSA (single-step)
97.5%

66

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

6. CONCLUSIONSIn this paper, a new MSA method for contextualization and classification of basic human actions waspresented. Using this MSA implementation, actions were classified with high fidelity by comparing various
attributes (i.e. data dimensions) of raw time-motion data sequences, and aggregating the obtained alignment scores.
Further, a distinction was drawn between actions and activities by defining action as a single, isolated effort, andactivity as an ongoing process over a period of time. Actions can thus be viewed as building blocks of complex
activities performed by construction workers (e.g., activity welding can be broken down into a sequence of actions
such as kneeling, holding, extending arm). It was therefore established that designing algorithms that can achievesignificance performance in classifying basic human actions can ultimately help improve the classification
accuracy of more complex construction activities. Compared to traditional HAR algorithms, the designed MSA
approach considers trends between data points and action dependency information as part of the classification
process, and provides more flexibility in dealing with variable window sizes. Overall, both HAR and MSA processdata through reduction by identifying representative subsets of data. While HAR uses statistical features, MSA
relies on categorical representation, which is a common technique in supervised ML and has been validated innumerous applications. To examine the performance of the designed MSA algorithm, a publicly available dataset
consisting of data collected using 5 sensor units mounted on 8 subjects performing 5 actions for 5 minutes each
was used in both subject-dependent and subject-independent scenarios. On average, after several iterations, theaccuracy of action recognition was 98% for subject-dependent classification, and 97% for subject-independentclassification when data from all 5 sensor units (i.e. 45 dimensions) was used.
In the designed MSA implementation, the number of ordinal categories used was limited to 20. Further
work in this research will investigate ways to remove this constraint with the expectation that with greater
granularity in categorization, the accuracy of classification is improved. Moreover, while this study investigatedthe performance of MSA algorithm on secondary data (collected by other researchers), future work will also aim
at applying MSA to primary data which will include sensor data from construction worker actions.
ACKNOWLEDGMENTS
The presented work has been supported by the U.S. National Science Foundation (NSF) through grant
CMMI 1800957. The authors gratefully acknowledge the support from the NSF. Model training was performedon Texas A&M University’s High Power Research Computing (HPRC) clusters. The authors thank HPRC for
providing the necessary computational resources. Any opinions, findings, conclusions, and recommendations
expressed in this paper are those of the authors and do not necessarily represent the views of the NSF or HPRC.
REFERENCES

Adomavicius, G., Sankaranarayanan, R., Sen, S., and Tuzhilin, A. (2005). Incorporating contextual
information in recommender systems using a multidimensional approach. ACM Transactions on
Information Systems (TOIS), 23 (1), 103-145
Akhavian, R. (2015). Data-driven simulation modeling of construction and infrastructure operations using
process knowledge discovery, PhD. Thesis, University of Central Florida, FL.
Altun, K., Barshan, B., and TunÃ§el, O. (2010). Comparative study on classifying human actions withminiature inertial and magnetic sensors. Pattern Recognition, 43 (10), 3605-3620.
Barshan, B., and Yüksek, M.C. (2014). Recognizing daily and sports actions in two open source machinelearning environments using body-worn sensor units. The Computer Journal, 57 (11), 1649-1667
Barzilay, R., and Lee, L. (2003). Learning to paraphrase: an unsupervised approach using multiple-sequencealignment. Proceedings of the 2003 Conference of the North American Chapter of the Association for
Computational Linguistics on Human Language Technology-Volume 1, pp.16-23.
Chen, J., Qiu, J., and Ahn, C. (2017). Construction worker's awkward posture recognition through supervised
motion tensor decomposition. Automation in Construction, 77, 67-81.
Cheng, T., Migliaccio, G.C., Teizer, J., and Gatti, U.C. (2012). Data fusion of real-time location sensing andphysiological status monitoring for ergonomics analysis of construction workers. Journal of Computing
in Civil engineering, 27 (3), 320-335.
Copasaro, G. (2018). An introduction to applied bioinformatics. Retrieved from readiab website:
http://readiab.org/book/0.1.3/.
Ding, L., Fang, W., Luo, H., Love, P.E., Zhong, B., and Ouyang, X. (2018). A deep hybrid learning model
to detect unsafe behavior: integrating convolution neural networks and long short-term memory.
Automation in Construction, 86, 118-124.
Gauthier, J.A., Widmer, E.D., Bucher, P., and Notredame, C. (2010). Multichannel Sequence Analysis
Applied to Social Science Data. Sociological methodology, 40 (1), 1-38
Garcia, S., Luengo, J., Sáez, J. A., Lopez, V., and Herrera, F. (2013). A survey of discretization techniques:67Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Taxonomy and empirical analysis in supervised learning. IEEE Transactions on Knowledge and Data
Engineering, 25 (4), 734-750.
Golparvar-Fard, M., Heydarian, A., and Niebles, J.C. (2013). Vision-based action recognition of earthmoving
equipment using spatio-temporal features and support vector machine classifiers. Advanced Engineering
Informatics, 27 (4), 652-663.
Gong, J., and Caldas, C.H. (2009). Computer vision-based video interpretation model for automatedproductivity analysis of construction operations. Journal of Computing in Civil Engineering, 24 (3), 252263.
Gong, J., and Caldas, C.H. (2011). An object recognition, tracking, and contextual reasoning-based videointerpretation method for rapid productivity analysis of construction operations. Automation inConstruction, 20 (8), 1211-1226.
HPRC(2018).
Introduction
to
ADA.
RetrievedfromHPRC
website:https://hprc.tamu.edu/wiki/Ada:Intro#Hardware_Summary.
Huang, P.C., Lee, S.S., Kuo, Y.H., and Lee, K.R. (2010). A flexible sequence alignment approach on patternmining and matching for human action recognition. Expert Systems with Applications, 37 (1), 298-306.
Islam, M.M., Hassan, M.M., Lee, G.W., and Huh, E.N. (2012). A survey on virtualization of wireless sensor
networks. Sensors, 12 (2), 2175-2207.
Joh, C.H., Arentze, T., Hofman, F., and Timmermans, H. (2002). Action pattern similarity: amultidimensional sequence alignment method. Transportation Research Part B: Methodological, 36 (5),385-403.
LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature, 521 (7553), 436.
Lee, H.S., Lee, K.P., Park, M., Baek, Y., and Lee, S. (2011). RFID-based real-time locating system forconstruction safety management. Journal of Computing in Civil Engineering, 26 (3), 366-377.
Leite, F., Cho, Y., Behzadan, A.H., Lee, S., Choe, S., Fang, Y., Akhavian, R. and Hwang, S. (2016).
Visualization, information modeling, and simulation: Grand challenges in the construction industry. J.
of Computing in Civil Engineering, 30 (6), 04016035.
Lichman, M. (2013). UCI Machine Learning Repository, (2013). Retrieved from UCI website:
http://archive.ics.uci.edu/ml.
Liu, H., Hussain, F., Tan, C.L., and Dash, M. (2002). Discretization: An enabling technique. Data mining
and knowledge discovery, 6 (4), 393-423.
Liu, L., Peng, Y., Liu, M., and Huang, Z. (2015). Sensor-based human action recognition system with amultilayered model using time series shapelets. Knowledge-Based Systems, 90, 138-152.
Nath, N.D. (2017). Construction ergonomic risk and productivity assessment using mobile technology andmachine learning, Thesis, Missouri State University, MO.
Nath, N.D., Chaspari, T., and Behzadan, A.H. (2018). Automated ergonomic risk monitoring using bodymounted sensors and machine learning. Advanced Engineering Informatics, 38, 514-526.
Ray, S.J., and Teizer, J. (2013). Computing 3D blind spots of construction equipment: Implementation andevaluation of an automated measurement and visualization method utilizing range point cloud data.
Automation in Construction, 36, 95-107.
Rosenberg, M.S. (Ed.). (2009). Sequence alignment: methods, models, concepts, and strategies. Univ. of
California Press.
Shoval, N., and Isaacson, M. (2007). Sequence alignment as a method for human action analysis in space
and time. Annals of the Association of American geographers, 97 (2), 282-297.
Shrestha, P., and Behzadan, A.H. (2017). An evolutionary method to refine imperfect sensor data forconstruction simulation. Proceedings of the Winter Simulation Conference (WSC), Las Vegas, NV,
pp.2460-2471.
Shrestha, P., Nath, N.D., and Behzadan, A.H. (2018). An Exploratory Study of Sequence Alignment for
Improved Sensor-Based Human Action Recognition. Proceedings of the Construction Research
Congress, New Orleans, LA, pp.347-357.
Simoni, R.D., Hill, R.L., and Vaughan, M. (2002). The Discovery of the Amino Acid Threonine: the Workof William C. Rose. Journal of Biological Chemistry, 277 (37), E25.
Stuckens, J., Coppin, P.R., and Bauer, M.E. (2000). Integrating contextual information with per-pixel
classification for improved land cover classification. Remote sensing of environment, 71 (3), 282-296.
Zamalloa, M.Z., and Krishnamachari, B. (2007). An analysis of unreliability and asymmetry in low-power
wireless links. ACM Transactions on Sensor Networks (TOSN), 3 (2), 7.
Zhang, S., Sulankivi, K., Kiviniemi, M., Romo, I., Eastman, C.M., and Teizer, J. (2015). BIM-based fallhazard identification and prevention in construction safety planning. Safety science, 72, 31-45.

68

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

APPLICATION OF INTELLIGENT SYSTEMS IN THE UNDERGROUNDEXCAVATION INDUSTRY: A SHORT REVIEW
Mohsen Ramezanshirazi1, Mohammad Norizadeh cherloo2, Orod Zarrin3
1) Ph.D., Department of structural and geotechnical engineering, Sapienza University of Rome, Italy.Email:Mohsen.ramezanshirazi@uniroma1.it
2) Eng., University of science and technology (IUST). Email: M.norizadeh1369@gmail.com

Abstract: According to the growing trend of the world population, the need for underground transportation isincreasing. To satisfy the urban development the use of the underground space acquired increasing importance andconsequently tunnels are playing an essential role in the development of urban infrastructures.
A variety of construction methods have been developed for tunneling; in an urban area, the mechanized excavationby using a tunnel boring machine (TBM) is the more effective option. Tunnels are constructed under various kinds ofgeological conditions, different from hard rock to very soft soils. By respect to the number of empirical and semi-empiricalmethods obtainable for predicting phase in mechanized tunneling, there is a beneficial advantage to using intelligence
instrument. This paper aimed to the presented the advantages of inteligence prediction tool with used artificial neuralnetwork(ANN). The ANN crate based on different advance algorithms and data collected from the real Tunnel project. Carryout networks optimization and advantages of the method on time and human resource in projects proposed. The relevant
results proved that the capability of this method for prediction and shown good agreements between prediction results .The
main aim of this study is to introduce the capable Artificial Neural Networks (ANNs) as well as the effectiveness of ANNs inorder to predict the TBM performances employed for tunnel excavation.
Keywords: Intelligent system, ANNs, Tunnel Boring Machine, TBMs.
1. INTRODUCTION
Among the primary tasks of during tunneling, there needs to be a continuous heedful inspection toprevent unintentional damage to the existing on/underground infrastructure. To this regard, the on-site engineer
should examine the ground distortion and its settlement at every stage of the work and ensure that theobservations are in line with those previously calculated during the design process. In fact, there exist a number ofmethods, varying depending on the complexity of the problem, reported in the literature of tunnelling engineering
which tries to predict the ground deformation induced by tunnelling. In circumstances that the tunnelling being
done in soft ground, numerous studies have been conducted to account for various environmental features.
The artificial intelligence methodology developed in the past few decades can be quite promising,
especially as an artificial neural network (ANN) has been proved to be quite strong when the problem underinvestigation is associated with understandable non-linearity in its essence. As a matter of fact, there is always aconsiderable concern when it comes to scooping out soil in highly occupied urban regions, where not only thereis the likelihood of inadvertently damaging the adjacent structures, but also there is a high risk of fatality or
injuries for both construction laborers and people living in the vicinity. To this respect, realizable estimation model
which can be able to approximate the true ground settlement and its distortion due to tunnelling is a vital researchline that can save lives and money and bring along a safer workplace with less risk of failure. To meet this end,
the scope of the present study is to propose a prediction model based on ANNs which are particularly popular
due to their ability to foreseeing the non-linearity of geotechnical problems. As time passes, an enriched record ofdata related to tunnelling is available which can be useful when training and building up the ANN model. In
addition, ANN models can acquire higher accuracy in their prediction when they are fed with on-going recorded
data from the previous projects or even the concurrent ones. There are numerous features which boost thecapability of ANN models. Generalization is one of the primary factors in ANN models make a developed model
to be used for even non-trained data. Once, sufficient data is provided for an ANN model, the developed model
can later be used to estimate the outputs from any other input data as long as they are in the same scope. In otherwords, any ANN model is trained to learn the features underneath the problem under investigation, and once themodel is developed, there is no need for re-training and it can be used instantly to predict new dataset. The
literature of neural network models’ application in tunnelling involves both excavations using TBM (e.g.
Suwansawat and Einstein, 2006) and sequential excavation method.
2. ARTIFICIAL NEURAL NETWORKs (ANNs)
Artificial neural networks include a user-defined number of layered-neurons connected pairwise to eachother where the neurons are tuned to during the training phase when fed by the input data to estimate theoutput. To this69

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

regard, the general processes in ANN can be categorized into (1) learning; and (2) recall. During the training process,the network is fed by the input data from the first layer of neurons where the number of neurons in the first layer isequivalent to the number of features. Later, these input data are mapped into other dimensional feature spaces
according to the number of neurons in the hidden layer, and then they are converted into the output layer where thenumber of neurons is equal to the number of output. The recall process is the diverse procedure taken from the output
layer reaching the first layer of input. ANN is the most popular technique of artificial intelligence techniques able tolearn from the input data to predict any other unseen problems Wasserman, 1989; Limpon, 1987; Kosko,
1958; Nielsen, 1998. ANN attempts to resemble what a human brain actually does during its information handlingprocess. The great popularity of the application of ANN models in engineering and especially geotechnical engineering
originates from its ability in accounting for non-linearity among the features of the problem without the need for the
analyser to understand all these non-linear correlations Agrawal et al. (1994)stated that ANN is a suitable means ofapproximating the functions for mapping from the sample input data to the output. The greatest advantage of ANN incomparison with existing statistical methods is its self-sufficiency without the need for a precise and exhaustive
mathematical model.
During the past few decades, ANN has found its grounds for applicability in geoengineering with respect tonumerous subjects Ni et al. (1995) could successfully propose a fuzzy-based neural network to assess the failure slope.
Juang and Chen (1999)Tailored ANN models which were able to estimate the liquefaction resistance and potentialfrom cone penetration test (CPT) based on a huge historical dataset. Juang et al. (2000) Took advantage of thecapabilities of ANN to estimate the cyclic resistance ratio (CRR) based on several soil parameters particularly being
used when investigating the potential for liquefaction. The architecture of the ANN i.e., the number of layers as wellas the number of neurons in each layer, can be designed to solve considerably complex problems (Hornik et al. 1989).
In multi-layer perceptron neural network with three layers of neurons, as shown in Figure. 1, the input layer isconnected to the output layer through interconnected neurons lying on the middle layer, called the hidden layer. The
mapping function between the input layer and the hidden layer takes the input variables, x j, multiplied by their weights,
w j, and summed and added to the threshold value denoted by w0 which is formulated as follows:
W0+ ∑𝒎𝒎 𝒘𝒘𝒘𝒘𝒘𝒘𝒘𝒘
𝒋𝒋=𝟏𝟏

(1)

Here, m denotes the number of input variables. In almost all ANN application cases in geo-techniqueengineering, the mapping function is preferred among the possible options as Berry and Linoff 1997: the sigmoid,linear or hyperbolic tangent. To identify which number transfer function is more efficient in mapping between thelayers’ input, simply several functions can be tested and their results can be compared against each other to choose
the best mapping function. In theoretical aspects, the mapping functions can vary from one layer to another, but in thecontext of engineering practices, this is not a recommended approach since it cannot change the ANN capability
considerably. The architecture of the ANN is set by the number of layers and the number of neurons in each layer.
However, the input layer and the output layer have a fixed size in terms of a number of neurons which are specified
by the problem.
The input parameters show the ANN input data; whereas the output data are the desired physical quantities.
The optimal layout of the neural network architecture does not have a general rule, but the best practice to set the
number of hidden layers and the neurons in each layer is using trial and error. More details on the architectural designof the ANN can be found in Fausett (1994) and Philippe (1997). For an instance, the ANN trained and proposed byKim et al. (2001) was set to consider 19 features in the input layer while mapping the input variables to a single output,
ground surface settlement, testing the network performance with 24, 47, and 94 neurons in the hidden layer. Neaupane
and Adhikari (2006) Presumed five and nine neurons lying in the two hidden layers with six input neurons in the input
layer to be used for estimating one single output, horizontal displacement. Suwansawat and Einstein (2006) trained an
ANN model with three layers of 13 input neurons in the first layer, 20 neurons on the hidden layer and only one single
output. The sample input data during the training phase is divided into train and test where the weights in the ANN are
adjusted using the training dataset and then the error of the prediction is compared against the test data.

70

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 1: General Concept of ANN and MLP
The ANN training procedure follows an algorithm to minimize the error index on the test data set when the
network is trained using the training data set. The learning procedure of the ANN is called back-propagation algorithm
in which the network is once fed forward using the input data and then using the deviation from the estimated output
data versus the real output data, the network path is reversed to adjust the weights in order to minimize the deviation.
This entire process is repeated several times to reduce the error in prediction. This whole procedure is illustrated
visually in Figure. 1 (Haykin 1994; Patterson 1996), once the error falls below a pre-defined threshold, it can beclaimed that the ANN model is trained successfully. When the weights in the network are adjusted and fixed, then the
network is called a trained network which can later be used for prediction of any other data which was previously
given or not given to the mode as long as the input data values lie within the range of training limits. Apart from thenecessity of dividing the data into train and test data set, another set of data is useful to be considered, which is called
the validation set. The validation set is used to ensure that the network is not “overtrained” or in more sampler words
the network has not memorized the correlation between the input data and out data sets.
This problem is often known as overfitting problem which states that although the error of the trained anddeveloped ANN model lies within an acceptable range when a new data is given to the model, the accuracy falls down
significantly. This is because that the fitted non-linear function between the input and output datasets has memorized
the correlations rather than learning it. Overfitting can be caused when the number of neurons or hidden layers isincreased, meaning that a higher degree of non-linearity is allowed to be considered in the mapping function
Suwansawat and Einstein (2006). Data set division for training and validation phases in ANN models is an importantfactor that affects the performance of the ANN. The importance of the selection of the data to be included in the training
data set is vital since it needs to represent the whole data set. In addition, the maximum and minimum limits for eachfeature (input data) should be included in the training set since the extrapolation cannot be done in the ANN model.
There are several studies available in the literature investigating the importance of data division in developing ANN
models. It is once proposed by Javadi (2006) to arrange the data in 11 combinations wherein each one could be used
to train and develop the ANN model and then choose the one with the best result Yoo, C., and Kim (2007). Used a ruleof thumb data division rule of 80%-20% wherein 80% percent of the data were used for the training phase and 20%
was used for the test. However, Suwansawat and Einstein (2006) examined its developed ANN by dividing the data
into two equal sample sizes for training and test. Although the increase in the number of input data increases theaccuracy, it should be noted that the computational burden is also heightened. In brief, if the effect of the total numberof data is neglected, there is not a general prescription for the division of the data into training/test data set. If the
training sample is too small then the training algorithm of ANN will not be able to achieve high accuracy of prediction;
meanwhile, if the training data size is large not only the computational effort must be increased but also the overfitting
problem might occur.
As a general rule, it is often recommended to use an optimization procedure to determine the number of datain the training sample data. In the present paper, the optimization analysis resulted in 70% of the data being used forthe training set and the remaining 30% for the test phase.
3. ANNs IN TUNNELLING
Often, the existing structures at the vicinity of the working place making the process of tunnelling extremely
more complicated and the on-going plans for enlarging the cities is aggravating the situation. Hence, it is quite a morerational approach to be prepared for possible scenarios being thought of during the planning phase of the construction.
One of the most common techniques to overcome such difficulties in tunnelling projects is to use surface
instrumentation to ensure the quality of the project.
The influential factors when investigating the ground movement can be divided into three main aspects:• Geometrical characteristics: Cross-section area (equivalent diameter), excavation face height, type of71

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019support, overburden, depth, single or twin tunnel, pillar width for twin tunnels, etc.
• Ground characteristics: Deformability modulus, Poisson’s ratio, lateral earth pressure coefficient,permeability, friction angle, cohesion, unit weight, etc.
• Excavation and support process: Excavation method (Sequential Excavation or TBM), excavation type (fullface or sequential mining), distance from the face to support, support time, support methods (anchoring, shotcrete,steel sets), advance rate, etc.
In real practice, their numerous factors actually affecting the tunnelling construction; however, the common
practice is to collect only those parameters with the most significant influence on the outcome. Otherwise, it might beeven impossible to undertake the analysis due to the computational burden. In tunnelling project, since the data ofground settlement and movements are known thus the ANN models used in tunnelling engineering is also known assupervised learning analysis, and in so many studies the MLP have been already used such as Shi et al., 1998; An etal., 2004; Suwansawat, 2004 ; Suwansawat and Einstein, 2006. Suwansawat (2004) states that the ANN models arehighly efficient in handling the non-linear correlation between the influential input parameters. The ANN models areable to approximate the output with quite a high accuracy and thus it has become a widely accepted technique to be
used in tunnelling projects. Shi et al. (1998) implemented the MLP neural network which was trained and validated toestimate the settlement during the project time span of sequentially excavated tunnels for the subway in Brasília. The
purpose of their study was to estimate the settlements in the remaining phases of the projects according to the recorded
data from the initial phases. To this end, the first 6 km of the tunnel was used during the training process and then 500
m of it was used for the test step. The ability of ANN was proved to be almost 50% better than other conventional
methods of estimation since the former had 33.4 mm of error whereas the latter had 70 mm of error. The correlation
coefficients for their developed ANN models were 0.832 and 0.575 for the training and test data set, respectively. An
et al. (2004) chose a subway tunnel project in Shanghai as his case problem to test their developed MLP neural network.
In their study, they used the optimization algorithm of genetic algorithm to identify the optimal values for the neuralnetwork architecture parameters, such as the number of hidden layers and the number of neurons in each layer.
Although a significant difference was observed when predicting the settlement rough, the ANN model was quite
promising in terms of its accuracy with respect to tither methods. Suwansawat and Einstein (2006) selected the tunnels
in Bangkok as to test their MLP neural network. Among their input parameters were TBM operational parameters,
groundmass characteristics, and surface movements. The most difficulty they had when gathering the input data wasto collect the parameters associated with instruments. Their study was limited to shallow tunnels; however, their study
also proved the efficiency of the ANN models in tunnelling projects.
4. THE EXAMPLE OF CAPABLE ANNS
4.1 RADIAL BASIS FUNCTION (RBF)
Radial Basis Function is categorized as one of the several existing types of multi-layer neural network
typically includes three layers of neurons (Schalkoff, 1997), which is the case used here. The radial basis function
neural networks are generally trained using a diverse set of learning algorithms, e.g., two-step hybrid method (Haykin,1994). The hidden layer in radial basis function neural networks can be interpreted as a classifier on the input data fed
to the network, thus are often called cluster centers. As the name of the neural network implies, unlike the ANNs where
the sigmoid function is used to convert the input data into a range of 0 and 1, the radial basis function neural networks
employ a Gaussian Kernel (Haykin, 1994). The radial basis function (RBF) neural networks can be concentrated at apoint prescribed by the given set of unit weights. Widths and positions in the Gaussian functions are identified during
the training procedure of the network. The output neurons employ a linear relationship between the RBFs. There is
not a general rule or any demonstration to judge whether to use the RBF networks and backpropagation multi-layer
perceptron (MLP) (Pal and Srimani, 1996). One of the advantages of RBF networks is that they can be trained fasterthan the general MLP networks; though that they are adaptable i.e., not easily implemented once a modification isneeded (Attoh-Okine et al., 1999). The best statement to state which method should be used is to check the output
performance. Assume that a classification problem needs a Kohonen network, where several types of ANN models
such as BP and RBF might be practical, there is the need that the parameters affecting the ANN should be optimized.
To this end, the authors can refer to a study conducted done by Hudson and Postma (1995). In the process of RBF, N
number of basis functions, one for each input data, are used which are defined as φ (x − x) p where φ(x) is a non-linear
function. The subtraction of x − x p actually governs the function φ, next the following formulation is used:𝑁𝑁

𝑓𝑓(𝑥𝑥) = ∑ 𝑊𝑊𝑊𝑊φ(ǁ𝑋𝑋 − 𝑋𝑋𝑝𝑝)
𝑃𝑃=1

72

(2)

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The algorithm aims to find the weights, Wp such that it can cover all the inputs. Three widely-used layers in
RBF are input layer, hidden layer, and output layer. The RBF only has one layer of neurons in its hidden layer unlike
what ANN models which might have several numbers of mapping layers. To this regard, the RBF is quite efficient insaving time and computational effort often needed for training the MLP ANN models. The RBF is able to transformthe non-linearity existing in the problem due to some unseen correlation between the influential factors, to a linear
output. The radial activation function usually defined by “Gaussian function” is used to map the input data into theneurons in the hidden layer. Last, the final output is the weighted summation of the non-linear inputs mapped to the
output layer, which changes the non-linearity in the problem into a linear relationship which is quite easier to be solved.
RBF technique of neural network is used for making accurate predictions. Figure 4, shown the architecture of the RBF
network.

Figure 2: The Radial Basis Function (RBF) network structure
4.2 Extreme Learning Machine (ELM)
The extreme learning machine (ELM) as an emerging learning technique provides efficient unified solutions
to generalized feed-forward networks including but not limited to (both single- and multi-hidden-layer) neuralnetworks, radial basis function (RBF) networks, and kernel learning. ELM theories1–4 show that hidden neurons areimportant but can be randomly generated and independent from applications and that ELMs have both universal
approximation and classification capabilities; they also build a direct link between multiple theories (specifically,ridge regression, optimization, neural network generalization performance, linear system stability, and matrix theory).
Consequently, ELMs, which can be biologically inspired, offer significant advantages such as fast learning speed, easeof implementation, and minimal human intervention. They thus have strong potential as a viable alternative techniquefor large-scale computing and machine learning.
4.3 Using Back- Propagation Algorithm and Delta-Rule
One of the main advantages of using a neural network revolves around its ability to learn from samples andto gradually improve its performance through a learning stage. In general, a neural network formulates a relation
between input and output values through an interactive process of weights and bias adjustment. Accordingly, a more
knowledgeable network about the mentioned relation will be developed after each iteration of the learning process.
Basically, the back-propagation learning algorithm consists of two parts pass through different layers of a network as
follows: a forward pass and a backward pass. Considering the forward pass, as explained, an input pattern is applied
to the nodes in a forward direction from the input to the output layers of the network with propagated effects layerby-layer. Finally, an output is produced as a response to the network. In case the response is different from the desired
actual value, error corrections will be required by distributing error values back through layers to adjust weights andbias of the network.
This is what the back-propagation or error back propagation algorithm is responsible. As shown in Figure
4.10, a multi-layer perceptron network is a fully connected network, in a way that a neuron in any layer is completely
linked to all nodes/neurons in the previous and/or next layers. As mentioned before, the processing signal flows
through layers in a forward direction, i.e. from left to the right of the network. Figure 5.8 represents a selected sectionof a multi-layer perceptron network. Considering a back propagation network, two kinds of signal can be defined asfollows: (1) Function Signal: This signal performs as an input signal (stimulus) which starts at the input layer,
propagates forward (neuron by neuron and layer by layer), until it finally reaches the output layer known as the output
signal. (2) Error Signal: An error signal originates from an output neuron of the network to be distributed backward
throughout the network at layer by layer level.

73

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 3: Directions of two basic signal flows in a multi-layer perceptron: forward propagation of function signals
and back-propagation of error signals
5. AN EXAMPLE OF PREDICTION RESULTS OF TBM PERFORMANCES BY USING ANN EXAMPLE
AND DISCUSSION
In the face of training numbers presence often organized it is known that “predictability” of an ANN different
depending on the designed network and training relations. Moreover, ANN learning should accomplish by highaccuracy to assurance general approximation for other applications. Though, as mentioned in literature there is no
method applicable of existence used for the selection of optimal parameters and relations of neural network training,
excepting of “pilot Training Method” which have been introduced by M. Ramezanshirazi, et al (2019). Based on apilot study, tried to select optimal network and adapted training factors to decide about the final pattern.
According to compression between results of trained and tested data, the optimal network selected as areasonable tool for prediction of maximum surface settlement. Pursuant to the value of Coefficient of determination(R2) and Mean squared error (MSE). Moreover, as illustrated fitting results figure 4, there is a good agreement inlearning and testing part and, there are good agreements with measured and predicted data and with this results, it ispossible to check the ability of the network to predict the new data with the optimal weight of learned network. In thiscase, when obtained reasonable results from each network, should save the optimal weights and keep it for applying
on a new case. So, the advantages of the new models will be considered to optimize time on a real site. Is worth tonote that it is possible to present to the engineers just matrix of optimal weights, not the complete network.

Figure 4: Prediction of TBM performances through the ANNs
5.1 Effect of Hidden Neurons
Arguably, different numbers of hidden layers and/or nodes will result in performance variation. Basically,
they both result in higher mapping accuracy between the input and output data as well as developing a more complex
function to fit the input data closer. However, a higher number of hidden layers and/or nodes cannot necessarily imply
as a more accurate and optimal network in comparison to the one with a lower number of hidden layer and/or nodes.
Like before, such network will be able to provide almost perfect answers to the training set of problems (i.e. MSE = 0
or R2 = 1) but fails to generalize a valid solution to different new problems. This issue is known as "overfitting". In
this regards, figure 5 illustrated the improvement trends of prediction results due to the calibration of a number of74

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019hidden neurons.

Figure 5. Prediction results of settlement by elm networks and different hidden neuron (10, 20, 30, and 40)
5.2 Effect of Training Epoch
Since each neural network can learn and correct their errors in each iteration they go through backpropagation
algorithm. Therefore, a higher number of iterations or epochs results in greater error reduction during the learning
period that leads the network to closely follow training data patterns to be fitted to them. By increasing the epoch
number, it can be seen that the performance of all networks improves with a lower error or MSE value.

Figure 6. The improvement trends of the fitting results according to the number of epochs
CONCLUSION
The Artificial Neural Networks (ANNs) procedure introduced as a predictor tool for using prediction of TBM
performances due to mechanized excavation. The proposed method includes introduced three capable models basedon neural networks for using to tunnelling excavation in different ground condition. The relevant results proved thatthe capability of the aforementioned methods for prediction of excavation performance. According to the procedureof ANNs, a number of advantages such as simplifies of input data, general approximation, simple designing andaccuracy in trained and tested results are non-negligible. Moreover, using networks in homogenise ground condition
is more significant to operate this instrument.
Based on several studies on prediction of excavation performance by a number of different networks, thisresearch could be beneficial and recommended for using as one accurate predictable tool to the optimization of time,
instruments and human resource in projects. The important limitation of ANNs needs to the preparation of enough75

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

database for the training part.

REFERENCES
Agrawal, G., Frost, J.D., Chameau, C.L.A., (1994). Data analysis and modeling using an artificial neural network.
In: Gulhati, S.K. (Ed.), Proceedings of XIII International Conference of Soil Mechanics and Foundation
Engineering, New Delhi, vol. 4, pp. 1441– 1444.
An, H., Sun, J., Hu, X., (2004). Study on intelligent method of prediction by small samples for ground settlement inshield tunnelling. In: Proceedings of the 30th ITA-AITES World Tunnel Congress, C38, Singapore.
Berry, M.J.A., and Linoff, G. (1997). Data mining techniques: for marketing, sales, and customer support. Wiley &Sons Inc., New York.
Fausett, F. (1994). Fundamentals of neural networks: Architectures, algorithms, and applications. Prentice-Hall,Englewood Cliffs, N.J.
Haykin, S. (1994). Neural networks: A comprehensive foundation. Macmillan Publishing, New York.
Hornik, K., Stinchcombe, M., and White, H. (1989). Multilayer feedforward networks are universal approximators.
Neural Networks, 2(5): 359–366. doi:10.1016/0893-6080(89)90020-8.
Javadi, A.A. (2006). Estimation of air losses in compressed air tunnelling using a neural network. Tunnelling andUnderground Space Technology, 21(1): 9–20. doi: 10.1016/j.tust.2005.04.007.
Juang, C.H., Chen, C.J., (1999). Cpt-based liquefaction evaluation using artificial neural networks. Journal ofComputer-aided Civil and Infrastructure Engineering 14, 221–229.
Juang, C.H., Chen, C.J., Tang, W.H., Rosowsky, D.V., (2000). Cptbased liquefaction analysis. Part i: Determinationof limit state function. Geotechnique 50 (5), 583–592.
Kim, C.Y., Bae, G.J., Hong, S.W., Park, C.H., Moon, H.K., and Shin, H.S. (2001). Neural network based predictionof ground surface settlements due to tunnelling. Computers and Geotechnics, 28(6–7): 517–547. doi:
10.1016/S0266-352X (01)00011-8.
Limpon, R.P., (1987). An introduction to computing with neural nets. IEEE ASSP Magazine, 4–22.
M.Ramezanshirazi. ,(2019) Application of Artificial Neural Networks to Predict Excavation Performances of
Mechanized Tunnelling Machines, Ph.D. Thesis, Sapienza University of Rome, Itay.
Neaupane, K.M., and Adhikari, N.R. (2006). Prediction of tunnelling- induced ground movement with the multilayer perceptron. Tunnelling and Underground Space Technology, 21(2): 151– 159. doi:
10.1016/j.tust.2005.07.001.
Ni, S.H., Lu, P.C., Juang, C.H., (1995). A fuzzy neural network approach to evaluation of slope failure potential.
Journal of Microcomputers in Civil Engineering 11, 59–66.
Nielsen, R.H., (1998). Neurocomputing, picking the human brain. IEEE Spectrum 25 (3), 36–41.
P. T. W. Hudson, E. O. Postma, Choosing and using a neural net, Artificial Neural Networks an Introduction to
ANN Theory and Practice, (1995), 273-87.
Pal and Srimani, (1996) S.K. Pal, P.K. Srimani Neurocomputing: motivation, models and hybridization Computer,29 (3) 1996, pp. 24–27.
Patterson, D. (1996). Artificial neural networks. Prentice Hall, Singapore.
Ramezanshirazi, M., Sebastiani, D., & Miliziano, S. (2019, July). Artificial Intelligence to Predict Maximum
Surface Settlements Induced by Mechanized Tunnelling. In National Conference of the Researchers ofGeotechnical Engineering (pp. 490-499). Springer, Cham, doi : 10.1007/978-3-030-21359-6_52
Robert J. Schalk of, Artificial Neural Networks,1997, book.
Shi, J., Ortigao, J.A.R., Bai, J., (1998). Modular neural networks forpredicting settlements during tunneling. ASCE
Journal of Geotechnical and Geoenvironmental Engineering 124 (5), 389– 394.
Suwansawat, S., Einstein, H.H., (2006). Artificial neural networks for predicting the maximum surface settlement
caused by EPB shield tunneling. Tunnell. Undergr. Space Technol. 21 (2), 133–150.
W_S (1994) _ Sarle_ Neural Networks and Statistical Models_ In Proceedings of the Nineteenth Annual SAS Users
Group International Conference_ pages 1994.
Wasserman, P.D., (1989). Neural Computing Theory and Practices. Van Nostrand Reinhold, New York.
Yoo, C., and Kim, J.-M. (2007). Tunnelling performance prediction using an integrated GIS and neural network.
Computers and Geotechnics, 34(1): 19–30. doi: 10.1016/j.compgeo.2006.08.007.

76Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

DETERMINATION OF AUTOMATED CONSTRUCTION OPERATIONS FROMSENSOR DATA USING MACHINE LEARNING
Aparna Harichandran1, Benny Raphael2 and Abhijit Mukherjee3
1) Joint Doctoral Candidate, Department of Civil Engineering, Indian Institute of Technology Madras, Chennai, India, andSchool of Civil and Mechanical Engineering, Curtin University, Bentley, WA 6102, Australia. Email:aparnaharichandran@gmail.com
2) Professor, Department of Civil Engineering, Indian Institute of Technology Madras, Chennai, India. Email:benny@iitm.ac.in
3) Professor, School of Civil and Mechanical Engineering, Curtin University, Bentley, WA 6102, Australia. Email:abhijit.mukherjee@curtin.edu.au

Abstract: Automated construction creates an intricate working environment involving workers and machines.
The added complexity of automated construction demands a rigorous monitoring system compared toconventional construction. The first stage of developing such a monitoring system is the identification of
construction operations. This paper discusses a methodology for the identification of construction operations
from sensor data. The methodology is illustrated using the case study of a coordinated lifting equipment
implemented in a laboratory. The data is collected from a small scale structural frame consisting of steel
modules in a controlled laboratory condition. The automated system follows a top-down construction method
where the major construction operations are performed at the ground level and the structure is lifted upwards instages. Strain and acceleration measurements were collected from the structure during construction. Each
operation is associated with a unique pattern of measurements at each sensor location. The measurement data isused for analysis by support vector classification. Parameters like error penalty (C) and width of Gaussian kernel(σ) were varied to obtain the best prediction results. The results of the analysis show that the linear classification
gives better results compared to the nonlinear classification for all operations except coordinated lifting.
However, coordinated lifting is the best-predicted operation with an accuracy of 96%. Selection of optimalvalues of C and σ enhances the accuracy of classification. The features extracted from data seems to highly
influence the learning of the algorithm and the performance of prediction. The results show the potential forusing machine learning techniques for monitoring automated construction operations.
Keywords:

Automated Construction, Construction Monitoring, Machine Learning, Support Vector Machines

1. INTRODUCTION AND BACKGROUND
The construction industry is reported to be one of the most unsafe working environments in the world(Bureau of Labor Statistics, 2018). Labors as human beings have an inherent tendency to avoid accidents which
can be anticipated. However, accidents are inexorable due to unpredicted circumstances and numerous parties
involved. The sophisticated collaboration of machines and manpower in an automated construction creates amore intricate scenario than conventional construction. The instability and subsequent failure of the structure
under construction can be one of the major reasons which cause catastrophic accidents at construction sites.
Therefore, continuous monitoring of the structure and operations is essential for automated construction.
1.1 Automated Construction Methods and Monitoring Systems(Bock & Linner, 2016) have analyzed automated or robotic construction systems adopted around theworld and classified them based on the working direction and location of the main operations into 10 categories.
Four of these automation systems are called sky factories in which the central operation unit is situated on the
topmost floor and construction works are progressing in the upward direction. Three of the systems are called
ground factories where building units are constructed at ground level and pushed upwards. Two of the systems
involve a combination of conventional and automated construction. The last category is a combination of bothoff-site and on-site factory.
Most of the automated construction systems have a control room from which they monitor the entire
construction through video cameras, or barcodes/RFID data of the components, in some cases sensors also(Bock& Linner, 2016). However, some of the systems are not having a real-time monitoring system. None of these
automation systems seems to have a monitoring system which provides integrated information about the wholeautomated construction. This is essential to take appropriate control actions during construction(Harichandran,Raphael, & Mukherjee, 2019). Conventional method of construction can be improved with automatedmonitoring systems by monitoring quality (Zhong, Li, Cui, Wu, & Liu, 2018), safety (J. Park, Kim, & Cho,2017), productivity (Joshua & Varghese, 2014) and progress of the work (Soman, Raphael, & Varghese, 2017).
For ensuring safe and reliable operations, an automated monitoring system is a necessity for automatedconstruction.
1.2 Applications of Machine Learning in Construction
Recent advancements in sensing technologies and artificial intelligence have revolutionized various

77Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

fields of construction. Accuracy and speed of onsite data collection improved the development of decision
support systems(M. Y. Cheng & Chen, 2002; H. S. Park, Lee, Adeli, & Lee, 2007; Zhang & Bai, 2015). Machinelearning techniques have demonstrated its ability to resolve numerous challenging construction problems whichwere beyond the capability of a limited number of supervisors at the site. This includes work-related issues likelabor activity recognition (Akhavian & Behzadan, 2016; Joshua & Varghese, 2011), ergonomic risk detection(Akhavian & Behzadan, 2016), injury forecast (Tixier, Hallowell, Rajagopalan, & Bowman, 2016), work
progress (Harichandran, Raphael, & Varghese, 2018), work safety assessment (Poh, Ubeynarayana, & Goh,2018); quality assurance and quality control tasks like structural damage detection (Chaiyasarn et al., 2018),
durability estimation (Taffese & Sistonen, 2017); construction management problems like project success (M.-Y.
Cheng, Wu, & Wu, 2010), project control (Wauters & Vanhoucke, 2014), cost estimation (M. Y. Cheng, Peng,
Wu, & Chen, 2010), decision making (M. Y. Cheng & Wu, 2009) etc. Even though machine learning techniques
are widely used for innumerable applications in construction including monitoring, the feasibility of using it for
ensuring the stability of the structure under construction is never explored.
2. OBJECTIVE
This research focuses on monitoring the structural stability in an automated construction from sensor
measurements taken from the structure. In order to develop a monitoring framework, construction operations andconditions of the structure have to be identified from the sensing data. This paper discusses how automatedconstruction operations are determined from the sensing data using machine learning techniques. The sensor
measurements are collected from controlled automated construction experiments conducted in a laboratory
condition.
3. METHODOLOGY
The research methodology consists of experimental studies and analysis of measurement data. It consistsof four major phases; 1) Establishment of measurement system 2) Controlled experiments 3) Analysis of data 4)
Validation of monitoring framework.
3.1 Establishment of Measurement System
Automation of construction requires sensors for taking control actions and monitoring. The type,
number, and position of sensors and configuration of the experiments are decided according to the methodology
for the design of the measurement system. This paper does not cover the design of the measurement system.
However, the basic criteria for the selection of sensors and data acquisition systems are described as follows.
Sensors are selected based on range, sampling frequency, and measurement requirements. Constraints
related to measurement locations and the number of sensors are determined based on cost, the structural system
as well as lifting machines adopted. Strain gauges and accelerometers are selected for the current study. Stain
gauges are intended to capture the static and slow operational states, whereas accelerometers are for covering the
dynamic operational states during the automated construction. Nine linear strain gauges (resistance: 120 ohm,
gauge length: 5 mm) and eight monoaxial piezoelectric accelerometers (range: -5 g to +5 g, sensitivity: 1000mv/g) are placed on the structural system as shown in Figure 1.

Figure 1. Schematic diagram of the structural system which shows positions of sensors(All dimensions are in mm)

78Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The data acquisition system is selected based on types of sensors, communication strategy, and data
storage systems. All the sensors used in the experiments discussed in this paper are wired and fixed on the
topmost beams and column modules. This assembly has to be kept undisturbed. Hence, construction of this partof the structure is not covered in the current experiments. The testing of beam member connections is planned
for the future with wireless sensors. QuantumX MX840B and QuantumX MX1615B are the data acquisition
systems used for accelerometers and strain gauges respectively at a sampling frequency of 200 Hz.
3.2 Controlled Experiments
A small scale structural frame is built by an automated construction system in a controlled laboratory
condition (Figure 1 and Figure 2). The structural frame is made of mild steel pipe sections of 60 mm outer
diameter, threaded externally on both ends. All the six columns of the structure are composed of column
modules each of 400 mm length. Column modules are connected by 35 mm long couplers with internal threading.
All other connections are made by joints as shown in Figure 2.

Figure 2. Automation machine and the structure under construction
The developed automation system follows a top-down method of construction (Raphael, Rao, &Varghese, 2016). In top-down construction, the topmost structural components are constructed first and thecompleted structure will be lifted upwards. These operations are followed by connection of the components
below the already constructed one. The newly assembled structure will be lifted up again and the operations will
be repeated.
The automation system has six lifting machines each having 2-ton lifting capacity. These are placed at
the location of columns. All lifting and lowering operations are automated, while connection operations are
performed manually in the current study. Advanced levels of automation for these operations are underconsideration. During the connection of column modules, individual machines are operated to lower and lift the
supporting platforms. The configuration of the structure designed in such a way that, every stage of thisautomated modular construction, the overall stability of the structure is ensured. All of the six machines areoperated simultaneously to lift the completed structure at the end of each cycle of operations. This process iscalled coordinated lifting. The five different categories of operations involved in one cycle of this automatedmodular construction method are listed in Table 1.
Due to constraints of time and resources, it is impossible to experimentally evaluate all these 99
operations. However, planning the right set of experiments which covers all major categories of operations will
provide enough data for developing a monitoring system for the entire automated modular construction
operations. Since the geometrical configuration of the structure is symmetrical, 41 operations are experimentally
evaluated capturing data for all the operations listed as in Table 2.

79Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 1. Automated construction operations
Sl. No.

Automated construction operations

1
2
3
4
5

Connection of beam members
Coordinated lifting of the completed structure
Lowering of supporting platforms
Connection of column modules
Lifting of supporting platforms until load transfer
Total number of operations

Number ofoperations
per category
2
7
30
30
30
99

Table 2. Experimentally evaluated automated construction operations/ states
Number of operations
Sl. No. Operation category
Automated construction operations
per category1
OP1
Idle state (Ambient Vibration)2
2
OP2
Coordinated lifting of completed structure3
3
OP3
Lowering of supporting platforms
12
4
OP4Connection of column modules
12
5
OP5
Lifting of supporting platforms until load transfer
12
Total number of operations
41
During each set of experiments, the instrumented structure is constructed by the top-down construction
method by the automated system developed. It follows the five categories of operations in the order as listed inTable 2. It starts with OP1, followed by the other four operations at each location of columns at a time. After
OP5 at column number 6, one cycle of operations complete. Then the next cycle begins with OP1. The
experiments are conducted for two cycles of operations and repeated for six times. Measurements are recorded
from 17 different sensor positions of the structure during the entire course of the experiments.
3.3 Analysis of Data and Validation of Monitoring Framework
The major objective of the research is to identify the automated construction operations from the sensor
measurements to develop a robust monitoring system. The machine learning-based framework developed formonitoring automated modular construction is shown in Figure 3. The first two phases are covered in the earlier
sections. The data analytics and validation phase are discussed here.

Figure 3. Framework for identifying automated construction operations
The measurement data collected from controlled experiments are used for classification analysis. Each
operation is associated with a unique pattern of measurements at each sensor location. The major aim is todetermine the operation from its sensing pattern. Five sets of measurement data for each operation are used for80Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019training the classifier. One set of measurement data (which were not used training) for each operation is used fortesting. The performance of classification analysis is determined as the percentage of correct predictions. Digitalrecords of all the operations along with timestamp are maintained for all experiments. The prediction results arecompared with the recorded data for validation.
The operations are identified from the sensing patterns using Support Vector Machines (SVM)
classification (Harichandran et al., 2019, 2018). This machine learning technique comes under the category ofsupervised learning. SVM uses binary classification to separate the supplied data points into two classes by a
hyperplane (classifier). When a particular operation has to be identified, the data corresponds to that operation
are labeled as positive. All other data are labeled as negative. During the learning process, the classifier isdetermined by maximizing its distance from the closest data points on both sides. The classification methods andcorresponding parameters used for the current study are shown in Table 3, where RBF is Radial Basis Function
classifier, C is error penalty value and σ is the width of Gaussian kernel.
Table 3 SVM classification methods and parameters used for data analysis
Sl.No.

Method of classification

Parameters and range of values

Number of analysis per
operation1

Linear Classification

C: 10 - 100

102Nonlinear Classification with RBF

C: 10 – 100

Total number of analysis for 5 operations

σ : 0.5 – 1.5

110
600

In order to make the pattern recognition problem more simple and effective, features are extracted fromthe raw experimental data (Bishop, 2006). The feature extraction is one of the most important tasks in
pre-processing. The features are selected in such a way that computation of features should be fast and it should
contain enough information to discriminate the patterns. The average, standard deviation, minimum value,
maximum value and histogram (ranging from -3 σ to +3 σ, σ: standard deviation) calculated for moving time
windows are selected as features in this study.
4. RESULTS AND DISCUSSION
The classification analysis results are plotted in Figures 4 – 6. Figure 4 shows the effect of C on
prediction results. Except for ambient vibration (idle condition) when C= 30 and C= 40, all other operations haveprediction accuracy above 60% using linear classification. Addition of column modules is the best-predicted
operation with linear classification, followed by the lowering of supporting platform. All of the operations have a
dip in performance for C= 30 and C= 40. Each operation has different optimum value for C which gives the best
prediction results.
Figure 5 and 6 show the effect of σ on prediction results. Coordinated lifting is the best-predicted
operation among all. It has a clearly discriminating pattern in all sensor data. This is the only operation which
has good prediction results with nonlinear classification. The optimum value of σ for coordinated lifting is foundto be equal to 0.9. All other operations have an accuracy of prediction less than 50%. Higher values of C in
nonlinear classification did not seem to improve the prediction. Even though all operations except coordinatedlifting gives marginally low accuracy of prediction, σ value from 0.7 to 0.9 gives the best predictions for these
operations by nonlinear classification.
Table 4 shows a summary of the best prediction results for all operations. The coordinated lifting is thebest-predicted operation among all. This operation is best recognized by nonlinear classification with the optimalvalue of σ equal to 0.9 irrespective of the values of C. All other operations are best identified by linear
classification with different optimal values of C. Ambient vibration (idle state) has the least prediction
percentage. This operational state has the lowest number of data points among all operations. The accuracy of
prediction for this state is improved to 72% by augmentation of existing training data.

81

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Percentage of Correct Prediction (P)Linear Classification
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
0

10

20

30

Addition of Column Modules
Lifting of Supports

40
50
60
70
Error Penalty Value (C)Ambient Vibration
Lowering of Supports

80

90

100

110

Coordinated Lifting

Figure 4. Effect of error penalty on prediction results

Percentage of Correct Prediction (P)Nonlinear Classification (RBF): Coordinated Lifting
100%
90%
80%
70%
60%
50%
0

C= 10
C= 20
C= 30
C= 40
C= 50
C= 60
C= 70
C= 80
C= 90
C= 1000.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.6
Width of Gaussian Kernel (σ)
Figure 5. Effect of σ on prediction results

Percentage of Correct Prediction (P)Nonlinear Classification (RBF): Addition of Column Modules
50%
40%
30%
20%
10%
0%
0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.6
Width of Gaussian Kernel (σ)
Figure 6. Effect of σ on prediction results

82

C= 10
C= 20
C= 30
C= 40
C= 50
C= 60
C= 70
C= 80
C= 90
C= 100Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 4. Comparison of prediction results
Sl.No.

Operation/StateClassification type

C

σ

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

Coordinated Lifting
Coordinated Lifting
Coordinated Lifting
Coordinated Lifting
Coordinated Lifting
Coordinated Lifting
Coordinated Lifting
Coordinated Lifting
Coordinated Lifting
Coordinated Lifting
Addition of Column modules
Lowering of Supports
Lowering of Supports
Lifting of supports
Lifting of supports
Lifting of supports
Ambient Vibration

Nonlinear Classification
Nonlinear Classification
Nonlinear Classification
Nonlinear Classification
Nonlinear Classification
Nonlinear Classification
Nonlinear Classification
Nonlinear Classification
Nonlinear Classification
Nonlinear Classification
Linear Classification
Linear Classification
Linear Classification
Linear Classification
Linear Classification
Linear Classification
Linear Classification

10
20
30
40
50
60
70
80
90
100
10
10
20
80
90
100
60

0.9
0.9
0.9
0.9
0.9
0.9
0.9
0.9
0.9
0.9

Best percentage ofcorrect predictions
96%
96%
96%
96%
96%
96%
96%
96%
96%
96%
90%
86%
86%
80%
80%
80%
72%

5. CONCLUSIONSThe prospect of machine learning techniques for identifying automated construction operations fromsensing data is explored in this paper. SVM classification is used for analyzing the strain and acceleration datacollected from the structure during construction. The analysis of data is conducted by varying the input
parameters (C and σ) of the classification to obtain the best prediction results.
Linear classification gives better results compared to nonlinear classification for all operations except
coordinated lifting. Selection of optimal values of C and σ enhances the accuracy of classification. The features
extracted from data seems to highly influence the learning of the algorithm and the performance of prediction.
The operations having more number of data points are better identified. However, data augmentation improves
the prediction of operations with small data sets. The operations like coordinated lifting, having clearly
discriminating pattern of measurements are best identified compared to other operations.
The current framework for identifying automated construction can be improved by incorporating
domain knowledge. Development of a systematic feature selection method will enhance the learning process.
This will result in better predictions. Simulated data can be used for operations which are difficult to recreate in alaboratory setup. Studies for merging the current framework with model-based system identification methods formonitoring automated construction are in progress.
ACKNOWLEDGMENTS
Authors deeply acknowledge the support provided by the faculties and staffs of Structural Engineering
Laboratory, Building Automation Laboratory, and Workshop at the Department of Civil Engineering, IIT
Madras. The scholarship from the Ministry of Human Resource Development (MHRD), Government of India
and Curtin International Postgraduate Research Scholarship (CIPRS) and Research Stipend Scholarship from
Curtin University, Australia support the research works of the first author. The funding for this research projectis provided by the Department of Science and Technology (DST), Government of India(DST/TSG/AMT/2015/234).
REFERENCES
Akhavian, R., & Behzadan, A. H. (2016). Smartphone-based construction workers’ activity recognition andclassification.
AutomationinConstruction,
71(Part2),
198–209.
https://doi.org/10.1016/j.autcon.2016.08.015
Bock, T., & Linner, T. (2016). Site Automation Automated/Robotic On-site Factories. New York: Cambridge
University Press.
Bureau of Labor Statistics. (2018). National census of fatal occupational injuries in 2017. Retrieved fromhttps://www.bls.gov/news.release/pdf/cfoi.pdf
Chaiyasarn, K., Khan, W., Ali, L., Sharma, M., Brackenbury, D., & Dejong, M. (2018). Crack Detection in83Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Masonry Structures using Convolutional Neural Networks and Support Vector Machines. 35th
International Symposium on Automation and Robotics in Construction (ISARC 2018). Retrieved fromhttps://www.iaarc.org/publications/fulltext/ISARC2018-Paper030.pdf
Cheng, M.-Y., Wu, Y.-W., & Wu, C.-F. (2010). Project success prediction using an evolutionary support vector
machineinferencemodel.
AutomationinConstruction,
19(3),
302–307.
https://doi.org/10.1016/j.autcon.2009.12.003
Cheng, M. Y., & Chen, J. C. (2002). Integrating barcode and GIS for monitoring construction progress.
Automation in Construction, 11(1), 23–33. https://doi.org/10.1016/S0926-5805(01)00043-7
Cheng, M. Y., Peng, H. S., Wu, Y. W., & Chen, T. L. (2010). Estimate at completion for construction projects
using evolutionary support vector machine inference model. Automation in Construction, 19(5), 619–629.
https://doi.org/10.1016/j.autcon.2010.02.008
Cheng, M. Y., & Wu, Y. W. (2009). Evolutionary support vector machine inference system for construction
management. Automation in Construction, 18(5), 597–604. https://doi.org/10.1016/j.autcon.2008.12.002
Harichandran, A., Raphael, B., & Mukherjee, A. (2019). Identification of the Structural State in AutomatedModular Construction. 36th International Symposium on Automation and Robotics in Construction (ISARC
2019), 187–193. Banff, Canada.
Harichandran, A., Raphael, B., & Varghese, K. (2018). Inferring Construction Activities from Structural
Responses Using Support Vector Machines. 35th International Symposium on Automation and Robotics inConstruction(ISARC2018),
332–339.
Retrievedfrom
http://www.iaarc.org/publications/fulltext/ISARC2018-Paper080.pdf
Joshua, L., & Varghese, K. (2011). Accelerometer-Based Activity Recognition in Construction. Journal ofComputing in Civil Engineering, 25(5), 370–379. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000097
Joshua, L., & Varghese, K. (2014). Automated recognition of construction labour activity using accelerometersin field situations. International Journal of Productivity and Performance Management, 63(7), 841–862.
https://doi.org/10.1108/IJPPM-05-2013-0099
Park, H. S., Lee, H. M., Adeli, H., & Lee, I. (2007). A new approach for health monitoring of structures:
Terrestrial laser scanning. Computer-Aided Civil and Infrastructure Engineering, 22(1), 19–30.
https://doi.org/10.1111/j.1467-8667.2006.00466.x
Park, J., Kim, K., & Cho, Y. K. (2017). Framework of Automated Construction-Safety Monitoring Using
Cloud-Enabled BIM and BLE Mobile Tracking Sensors. Journal of Construction Engineering andManagement, 143(2), 05016019. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001223
Poh, C. Q. X., Ubeynarayana, C. U., & Goh, Y. M. (2018). Safety leading indicators for construction sites: A
machinelearning
approach.
AutomationinConstruction,
93,
375–386.
https://doi.org/10.1016/J.AUTCON.2018.03.022
Raphael, B., Rao, K. S. C., & Varghese, K. (2016). Automation of modular assembly of structural frames forbuildings. Proceedings of the 33rd International Symposium on Automation and Robotics in Construction(ISARC 2016), Auburn, USA, (33), 412–420.
Soman, R. K., Raphael, B., & Varghese, K. (2017). A System Identification Methodology to monitor
construction activities using structural responses. Automation in Construction, 75, 79–90.
https://doi.org/10.1016/j.autcon.2016.12.006
Taffese, W. Z., & Sistonen, E. (2017). Machine learning for durability and service-life assessment of reinforced
concrete structures: Recent advances and future directions. Automation in Construction, 77, 1–14.
https://doi.org/10.1016/j.autcon.2017.01.016
Tixier, A. J. P., Hallowell, M. R., Rajagopalan, B., & Bowman, D. (2016). Application of machine learning toconstruction injury prediction. Automation in Construction, 69(January 2018), 102–114.
https://doi.org/10.1016/j.autcon.2016.05.016
Wauters, M., & Vanhoucke, M. (2014). Support Vector Machine Regression for project control forecasting.
Automation in Construction, 47, 92–106. https://doi.org/10.1016/j.autcon.2014.07.014
Zhang, Y., & Bai, L. (2015). Rapid structural condition assessment using radio frequency identification (RFID)
based
wirelessstrainsensor.
AutomationinConstruction,
54,
1–11.
https://doi.org/10.1016/j.autcon.2015.02.013
Zhong, D., Li, X., Cui, B., Wu, B., & Liu, Y. (2018). Technology and application of real-time compaction
quality monitoring for earth-rockfill dam construction in deep narrow valley. Automation in Construction,
90, 23–38. https://doi.org/10.1016/J.AUTCON.2018.02.024

84

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Image Processing and Computer Vision85

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

STUDY ON AUTOMATIC CHALK MARKS RECOGNITION FOR CONCRETE
TUNNEL INSPECTION USING DEEP LEARNING
Yao Zhang1, Kei Kawamura 2, Cuong Nguyen Kim 3, Koji Oshikiri 4, and Taro Kikuchi 51) Graduate Student. Graduate School of Science & Technology for Innovation, Yamaguchi University, Japan. E-mail:i503vg@yamaguchi-u.ac.jp
2) Ph.D., Assoc. Prof., Graduate School of Science & Technology for Innovation, Yamaguchi University, Japan. Email:kay@yamaguchi-u.ac.jp
3) Dr. Eng., Faculty of Highway & Bridge, Mientrung of Civil Engineering, Vietnam. E-mail: nguyenkimcuong @ muce.edu.vn
4) Ricoh Company, Ltd. Japan.E-mail:kohji.oshikiri@jp.ricoh.com
5) Ricoh Company, Ltd. Japan. E-mail:taroh.kikuchi@jp.ricoh.com

Abstract: Current methods of practice for inspection of concrete tunnel typically involve visual assessments anddrawing notes conducted manually by trained inspectors. The labor intensive and time consuming natures ofmanual inspection have engendered research into development of method for automated damage identification
using computer vision techniques. Due to the limitation of the information reflected by the current photography
technology, in this study, compared with other studies that directly detect cracks from images (Eftychios
Protopapadakis et al., 2019; F. Panella et al., 2018; Suguru Yokoyama & Takashi Matsumoto, 2017), the author
proposed to identify chalk marks that can be determined to be deformation. In this study, in order to improve theefficiency of tunnel inspection, it verified the automatic recognition of chalk marks in a concrete tunnel with areasonable accuracy. As a result, a chalk mark classifier using a deep convolutional neural network on deeplearning has been generated to detect the existence of chalk marks as a stage study of recognition. The proposed
method is evaluated in terms of accuracy, precision and recall is shown.
Keywords: concrete inspection, deep learning, convolutional neural network, image recognition
1. INTRODUCTION
In Japan, many existing concrete tunnels have already been in operation for more than half a century
since they were built in the economy high-growth period, the aging of concrete structures such as tunnels is
becoming a serious problem. In order to cope with the problems caused by the aging of tunnels, the Ministry ofLand, Infrastructure, Transport and Tourism has stipulated to conduct periodic inspections once every five years
for tunnels since July, 2014 (Ministry of Land, Infrastructure, 2014).
In the current tunnel inspection method, the tunnel is inspected by close visual observation, and thedefects such as cracks are illustrated by the chalk marks on the concrete wall surface, then the defects will besketched in the filed books, referring to the recorded chalk marks. After the on-site work defection map will bemade on CAD (computer-aided design) in office according to the filed books.
In recent years, digital technologies have been applied to inspection activities, the tunnel surface marked
with chalk can be scanned into a layout panorama by inspection car which mounts cameras and lights, instead ofbeing drawn into filed books. Then the office work can be proceeded in accordance with the layout panorama.
However, it also takes a lot of time to create defects map by handwork, even through a layout panorama.
On the other front, deep learning methods using an artificial neural network have been applied to image
recognition. In the benchmark tests of image recognition, the technology using deep learning set a new record overthe past one, showing its usefulness in this field.
Based on these conditions, for the purpose of improving the efficiency of tunnel inspection and reducing
the maintenance work cost, this paper proposed to recognize the chalk marks automatically from layout panoramaof concrete tunnel utilizing CNN (convolutional neural network) which is a method of deep learning.
In order to realize this automatic recognition, as shown in Figure 1, the first step is to detect the position
or existence of the chalk mark from the image. In the second step, the symbol and trace marked with chalk will besegmented, then the final step is to recognize the meaning of the symbol and trace after the segmentation.

Figure 1. The flow of automatic recognition of chalk marks

86Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

In this study, authors studied the method of detecting the presence of chalk marks in an image, whichcorresponds to "Step1" in the flow of automatic recognition of chalk marks shown in Figure 1. As a result,possibility of this method was be presented with a reasonable accuracy.
2. METHOD
2.1 Procedure of this researchThis study was carried out by the following four procedures.

Figure 2. Examples of target image
Step 1. Image collection on the walls of the tunnel
In this study, 225 tunnel wall surface images obtained from photography vehicle in regular tunnel
inspections were selected. All of those images are in gray scale and marked with chalk marks. Figure 2 shows anexample of two images that will be studied. The image size shown in Figure 2 is 1600×1200 pixels, and the sizeof the photographic area is 4800×3600(mm).
In this study, considering the effect of deep learning and the performance of the computer, the image
used for deep learning was segmented into 48 sub-images each image, and the final number of sub-images after
cut was 10,800. It should be noted that the cutting size is 224×224 pixels.
Step 2. Create data sets

Chalk markNo chalk markFigure 3. Class definition
The images of the tunnel wall surface were divided into two classes (Chalk mark, No chalk mark) shownas Figure 3. Then the number of image data was augmented to three times (32400) by mirror inversion and rotationin 180 degrees. After that the data were divided into training, validation and test dataset for deep learning andevaluation. The number of each dataset is shown as the Table 1 below. Here, the training dataset is the actual
dataset that will be used to train the model (weights and biases in the case of neural network). The model sees andlearns from this data. The validation dataset is the sample of data that is used to provide an unbiased evaluation
for a model fit that is performed on training dataset with model hyperparameters tuning. And the test dataset is the
sample of data used to provide an unbiased evaluation for performance of a final model that generated from thetraining dataset.

Training Dataset
Chalk markNo chalk mark
5000
5000

Table 1.The number of dataset
Validation Dataset
Chalk markNo chalk mark
5000
5000

87

Test Dataset
Chalk markNo chalk mark413
485

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Step 3. Generation of the chalk mark classifier
A chalk mark classifier was generated through deep learning using training dataset and validation
dataset.
Step 4. Evaluation of identification performance
Test data set is used to evaluate the recognition performance of the chalk mark classifier.
2.2 Deep Learning Network
In this study, authors chose VGG16 as the deep learning network for the generation of Chalk MarkClassifier.
The structure of the VGG16 (Karen Simonyan & Andrew Zisserman, 2014) is shown in Figure 4 (here,
FC in the figure is Full Connection). VGG16 is a highly acclaimed convolutional network for classification anddetection of the 2014 Image Recognition Competition (ILSVRC)(Olga Russakovsky et al., 2014). In the training
process of the model, the gray scale single channel image will be taken as the input, and the neural network will
output classification scores as the result. According to the result, the images for prediction will be set to be Chalkmark or No chalk mark while the corresponding classification scores above the threshold 0.5.

Figure 4. Network structure of VGG16
2.3 Deep Learning Methods
At the time of deep learning, 50 epochs were performed. Here, all of the training and validation data
were loaded and learned during each epoch. In order to improve the versatility of the model, hold-out validation
was performed, which is a method that randomly using 50% of data from the total dataset that consist of the
training and validation datasets to training, using remaining 50% of the data for validation. The hold-out validation
was performed twice that were named Model 1 and Model 2 respectively.
3. RESULTS
3.1 Evaluation method of results
In this study, the classification of the results from chalk marks prediction is expressed by threat score
shown in Table 2.
Table 2. Classification of chalk mark identification results

True
Chalk markNo chalk mark

Chalk markTP(True Positive)

FP(False Positive)

No chalk mark

FN(False Negative)

TN(True Negative)

Predicted

Then, the performance of the classifier was evaluated by the Accuracy shown in equation (1).
Accuracy ＝

𝑇𝑃+𝑇𝑁
𝑇𝑃+𝐹𝑃+𝑇𝑁+𝐹𝑁

Here, the accuracy is the percentage of the data which correctly predicted to be Chalk mark or No chalk mark.

88

(1)

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The results that obtained from training of deep learning were evaluated using accuracy that calculatedby the prediction results that generated from training dataset and validation dataset.
3.2 Results of deep learning
The datasets were put into the deep learning network for learning, and the chalk mark classifier was
generated after each epoch. The learning results are shown in the Figure 5 and Figure 6 below. Here the training
accuracy is calculated by the equations (1) using the prediction value from training dataset that performed after
each epoch. The validation accuracy is calculated by the equation (1) using the prediction value of validation
dataset with the trained model obtained from each epoch. The best-performing results will be used as a classifier
for subsequent test.

Figure 5. Training results of VGG16 (model.1）

Figure 6. Training results of VGG16 (model.2）

From the results data, both of the models reached almost 97% in validation accuracy by VGG16.
Therefore, the effect of deep learning has reached the requirement in this aspect.
3.3 Verification of identification performance
In order to verify the performance of the classifier, the test dataset, which was not used for learning, ispredicted by the VGG16 trained classifier. The two best-performing classifiers mentioned in the last section wereused for prediction of test dataset, and the two predicted values were averaged as the final prediction results. The
final prediction results are shown in Table 3.
The accuracy was 96.7% calculated by the equation (1) using the data shown in Table 3.
Table 3. Prediction results of VGG16

True
Chalk markNo chalk mark

Chalk markTP(395)

FP(12)

No chalk markFN(18)TN(473)

Predicted

In order to further evaluate the identification performance in detail, especially for the target class "Chalkmark", Precision and Recall which commonly used in pattern recognition and binary classification problems wereadopted. The calculations of Precision and Recall are shown in equation (2) and equation (3) below. In aclassification task, a precision score of 1.0 that obtained from a class classification means that every item predicted
to this class does indeed belong to corresponding class, whereas a recall of 1.0 means that all the items belonging
to some class were detected.
Precision ＝Recall ＝

𝑇𝑃
𝑇𝑃+𝐹𝑃
𝑇𝑃

𝑇𝑃+𝐹𝑁

(2)
(3)

After using the above equation (2), the value of precision reached 0.971, which means that 97.1%
images that belong to class "Chalk mark" were correctly predicted. And the value of recall reached 0.956 whichmeans that the 95.6% of the images that actually belonging to class "Chalk mark" were detected.

89

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

4. DISCUSSION
Figure 7 shows the prediction results with a combined image. The correct answer of the areas thatmarked with "0" is "No chalk mark", and the correct answer of the image with "1" is "Chalk mark". The areas
surrounding with green line are the data predicted incorrectly.

Figure 7. Recognition of VGG16
Then, in the images with wrong prediction by VGG16, three examples with wrong answers "Chalkmark" and four examples with wrong answers "No chalk mark" are shown in Figure 8 and Figure 9, respectively.
For Figure 8, there are many deformations that are look similar to chalk marks which are difficult to berecognized even by the naked eyes. Specifically, image A in Figure 8 is free lime on the tunnel wall, image B isbelt‐shaped free lime caused by water leakage, image C is a unknown white point of the concrete, and image D isbelt‐shaped free lime taken under better lighting conditions than image B. These confusable image data are
believed to be the main cause of mistaken predictions as "Chalk mark".

Image A

Image B

Image CImage D

Figure 8. The wrong prediction as "Chalk mark"

Image E

Image F

Image G

Figure 9. The wrong prediction as "No chalk mark"Figure 9 shows wrong predictions as "No chalk mark". In image E, there is free lime in the surrounding90

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019area above the chalk mark, which interferes with the prediction of the image. Image F is the chalk mark that became
lighter after a long time since it was recorded, while the chalk mark in image G is located in the left bottom of theimage, which is difficult to be recognized due to its small area. These image data are believed to be the main cause
of incorrect predictions as "No chalk mark".
As the results obtained from figures above showed, the complexity of the images that draw with various
chalk mark pattern that are even difficult for the naked eye to recognize and chalk-like free lime led to the incorrect
predictions. Through the analysis of the above data, in order to improve the accuracy of the classifier, it is necessary
to increase the proportion of similar confusable images in the learning data set.
5. CONCLUSIONSIn this study, in order to detect chalk marks from tunnel wall images, the classifier for prediction was
successfully generated using VGG16 which is based on deep learning. Secondly, the performance of the automatic
prediction classifier was verified and the verification results achieved 96% accuracy. From these points of view,the research of automatic detection using chalk mark classifier basically achieved a satisfactory result. However,there are also some erroneous predictions from the test dataset. The authors consider that it is the insufficiency ofcomplexity that the data have caused false predictions. From the above, the authors will enrich the diversity ofdata sets in future studies.
ACKNOWLEDGMENTS
This work was supported by The Japan Society for the Promotion of Science (JSPS) KAKENHI Grant
Number 19K04580.
REFERENCES
Eftychios Protopapadakis, Athanasios Voulodimos, Anastasios Doulamis, Nikolaos Doulamis and Tania
Stathaki. (2019). Automatic crack detection for tunnel inspection using deep learning and heuristic
image post-processing. Springer Science plus Business Media, LLC, part of Springer Nature．
F. Panella, J. Boehm, Y. Loo, A. Kaushik and D. Gonzalez. (2018). Deep learning and image processing forautomated crack detection and defect measurement in underground structures. The International
Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2．
Karen Simonyan and Andrew Zisserman. (2014). Very Deep Convolutional Networks for Large-Scale Image
Recognition．
Ministry of Land, Infrastructure, Transport and Tourism. (2014). Road Tunnel Periodic inspection guidelines.

website: www.mlit.go.jp/common/001044575.pdf
Olga Russakovsky, Jia Deng and Hao Su et al. (2014). ImageNet Large Scale Visual Recognition Challenge
2014. International Journal of Computer Vision (IJCV).Vol 115,Issue 3,pp.211–252．
Suguru Yokoyama and Takashi Matsumoto (2017). Development of an automatic detector of cracks in concrete
using machine learning, Procedia Engineering, 171, 1250-1255．

91Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

GA-CNN based automatic crack detection and classification method for concrete
infrastructures
Cuong Nguyen Kim1), Kei Kawamura2), Yao Zhang3), and Amir Tarighat4)

1) Dr. Eng, Faculty of Highway & Bridge, Mientrung of Civil Engineering, Vietnam. E-mail: nguyenkimcuong @muce.edu.vn
2) Assoc. Prof., Graduate School of Science & Technology for Innovation, Yamaguchi University, Japan. Email:kay@yamaguchi-u.ac.jp
3) Graduate Student. Graduate School of Science & Technology for Innovation, Yamaguchi University, Japan.E-mail:i503vg@yamaguchi-u.ac.jp
4) Assoc. prof, Department of Civil Engineering, Shahid Rajaee Teacher Training University, Iran, tarighat@srttu.edu

Abstract: Automatic crack detection is a main task in a crack map generation of the existing concrete
infrastructure inspection. This paper presents an automatic crack detection and classification method based on
genetic algorithm (GA) to optimize the parameters of image processing techniques (IPTs). The crack
detection results of concrete infrastructure surface images under various complex photometric conditions still
remain noise pixels. Next, a deep convolution neural network (CNN) method is applied to classify crack
candidates and non-crack candidates automatically. Moreover, the proposed method is compared with thestate-of-the-art methods for crack detection. The experimental results validate the reasonable accuracy inpractical application.
Keywords: Crack detection, genetic algorithm, convolution neural network.
1. INTRODUCTION
Many concrete components of existing infrastructure systems such as bridges, and tunnels havesuffered from various geologic, loading and environmental conditions cause to cracks which make influent toquality of operations. Therefore, the condition assessment of the existing infrastructures is an important task
not only for warming against deterioration but also for guaranteeing soon maintenance. Concrete cracks areimportant indicators reflecting the safety of infrastructure. The automatic crack detection based on image data
has been considered significantly due to accuracy, objectivity and timing inspection. This technique can be
implemented using some of different image data captured from ultrasonic device, infrared and thermal device,laser scanning, and commonly digital cameras.
Image processing techniques consist of three approaches: edge detection, threshold technique (Fujita etal.2006) and mathematical morphology (Nguyen et al 2016). Machine learning algorithms (MLAs) arecommonly used to decide the parameter value of IPTs. Therefore, they are applied to detect and classify
concrete infrastructure surface cracks.
In recently years, many automatic crack detection and classification methods based on a combinationof IPTs and MLAs are implemented as decision tree (DT) (Kei et al.2013), support vector machine (SVM), kclustering nearest neighbour (K-NN), and artificial neural network (ANN) (Wang et al.2014; Bang Yeon Lee
et al. 2011; Li Li et al.2014).
Moreover, the parameter-optimization algorithms of IPTs such as genetic algorithm (GA), particle
swarm optimization (PSO), artificial bee colony (ABC), and differential evolution (DE) are typically utilized.
In this article, the image processing parameters (IPPs) are adjusted to the optimized value in order toincrease the accuracy of crack detection before use of convolution neural network (CNN) to eliminate noncrack candidates. As a result, the crack map will be improved significantly.
2. PROPOSED METHOD
Figure 1 shows a pipeline of automatic crack detection and classification based on IPTs combinedwith GA-CNN.
2.1 Image Processing Techniques (IPTs)
The image processing techniques compose of three main parts. Namely, there are filtering image part,
binary image part, and feature extraction part. Therein, filter image part comprises of consecutive dilation-

92

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

erosion transform and contrast enhancement of the gray-scale image. The binary image part consists ofbinarization, and dilation-erosion transform of the binary image.

Input imagee

Filter image

Dilation/
ErosionEnd

Optimized parameters

Start

Yes
Ye

Initialial / N
Next
population

Evaluation of eachindividual
individ

Stopping
criterion

GA
FitnessfunctionContrast
Enhancement

N
No

EvolutionnSelection
electio

CrossoverBinary image

Binarization
Dilation/Erosion

Mutation

Input
Layer

OutputLayer

Deep convolution neural network

Linear degree

crack
OutputImage

Noncrack

Crackcandidates
Stage-1

Convolution layer
Batch-normalization
Activation function
Max-PoolingStage-2

Stage-3

FCL
128
neurons

Figure 1. Pipeline of the automatic crack detection and classification using GA-CNN
To the end, geometric transform composes of labelling and linear degree. The detailed steps refer in(Cuong et.al 2018). The purpose of the first part is to make blurred images as well as eliminate noises andshading. The main aim of the binarization is to segment grayscale image into binary image depending on a
threshold value. The purpose of dilation/erosion that is processed in the binary image part is to connect
fragment images of crack meanwhile noises are separated from the cracks. In the end, the threshold defined inlinear degree is to decide whether the single objects in the binary image are removed.
The productivity of the filtered image depends on the parameter value of structuring element size in
dilation and erosion transform. In binarization, if the threshold value is too high, many crack pixels are lost. If
threshold value is too low, more noise will occur. Such it is necessary to find out an optimum threshold value.
Similarly, the parameter values of dilation/erosion and linear degree in the binary image part also affect to thequality of output image. Therefore, these parameters are adjusted to the optimized values based on GA.
2.3 Application of GA to The Image Processing Parameters Optimization
GA is one of the optimization algorithms based on solution population inspired biologically behavior.
A solution candidate is encoded to a chromosome (individual) which contains information of tuned parametervalues. Some weak individuals will be removed as well as some elite members would keep in the next
generation. Other individuals will be selected to crossover with respect to its fitness value. The probability of
the crossover and crossover point are predefined. Further, mutation following specific probability ratio is so as
to improve the quality of each generation. Advantage of GA is to avoid a local optimization as otherconventionally evolutional algorithms.

93Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

1 1 1 0 1 0 1 0 1 1 1 0 1 1 0 0 1 1 1 0 1 1 1 0
b: 8 bits

f: 6 bits

d: 4 bits

l: 6 bits

Figure 2. A represented chromosome for solution candidates
Table 1 Properties of parameter
Variable

Range

s
t
d
l

Steps

[1 127][0 255][1 31][0 32.5]

2
1
0.5
0.5

Bits
6
8
4
6

Initial/Next Population

Input

Output1 1 1 0 1 0 1 0 1 1 1 0 1 1 0 0 1 1 1 0 1 1 1 0
bi: 8 bits

fs: 6 bits

di: 4 bits

li: 6 bits

0 1 1 0 1 1 0 0 1 1 0 0 1 1 0 0 1 0 1 0 1 1 0 0

Original
image(k)bi: 8 bits

fs: 6 bits

di: 4 bits

li: 6 bits..
.

Crack Image(st, bi,di li)k
(f1,f2,f) k

1 1 1 0 1 0 1 1 1 1 1 0 1 1 0 1 1 1 1 0 0 1 1 0
bi: 8 bits

fs: 6 bits

Target image
f1

Loss
; f2
Crack

di: 4 bits

Coding individuals of
candidates (20 individuals)
Noise
Back

f

1

the

li: 6 bits

solution

1
1( f1 )2  ( f 2 ) 2 f  >0,1@2
2

GA performance

Figure 3. Procedure of genetic algorithm(1) Represented chromosome Design for solution candidates
The IPPs are combined together for creating an individual in a population. Next each individual isrepresented by a chromosome encoded to a binary string, as shown in figure 2.
Namely, the size of structuring element (s) is assigned by 6 bits, the threshold value of binarization (t) isexpressed by 8 bits, dilation transform parameter (d) is expressed by 4 bits, and the linear degree is expressedby 6 bits (l). Table 1 shows the parameter value range which design based on the preliminary experiments.
(2) Genetic algorithm
Figure 3 indicates a sequence of GA including into the crucial three stages. Namely, they consist of
the initial population generation, fitness evaluation of each individual in the current population, and evolution
operation to create the next generation. Namely, the detailed steps are presented as the following three steps:
Step1. Generate initial population randomly

94Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

An initial population including 20 individuals was generated randomly with respect to 20 phenotypes
to start fitness evaluation. To assess the fitness of the individual in the current population, an objectivefunction to assess crack detection accuracy is defined as the Eq.(1). Loss and noise are computed based oncomparison between the processed image and the target image shown in Fig.4. As a result, the objectivefunction (f) has to ensure the accuracy of extracted crack information with the minimum noises and losses asmuch as possible. The accuracy and the processing time can be considered as evaluation costs. As the firststep, the evaluation cost based on the noise and loss ratios were applied, as shown in the Eq.(1).
Step2. Evolution operation
The evolution operation comprised of selection, crossover, and mutation is repeated until finding bestsolution. Each binary string encoded from the searching range of the parameter values has a corresponded
fitness value. The probability of each string to be selected is proportional to its fitness value based on the
Roultte wheel rotation randomly. The process is repeated for the second parent. Two elite members are kept
forward to the next generation.
To improve quality of individual fitness, the crossover operation is used to create two new children
from two selected parents with predefined probability. Crossover point is point laid on between 0 to the end ofchromosome length. In this study, the single crossover point is selected. The part of the first parent
chromosome that runs until the crossover point is spliced with the part of the second parent chromosome that
includes, and runs after, the crossover point shown in Fig.2. The whole new generation is selected in thismanner. The mutation of bit strings ensue through bit flips at random positions. The purpose of the mutation
operation is to create genotype diversification in the population in order to avoid local optimization leading tofinding the best solution. Mutation point is chosen randomly. However, mutation rate is very small under 1%
to avoid collapsing the genetic structure of the current population.
Step3. Stopping criterion
Evaluation of each individual meets the predefined maximum generation

(a)

(b)

(c)

Figure 4. Result of supervised learning algorithm with one-pixel accuracy. (a): Ground-truth image;(b): Processed image; (c): Compared image. A number of red pixels in the compared image are noise pixels
annotated Noise. A number of green pixels in the compared image are loss pixels annotated Loss.

f=1  w1f12  w 2 f 2 2 , f  [0 1]
f1 =

Loss
Noise
;f 2 =
Crack
Back

(1)

Where Crack and Back are the number of black pixels and white pixels in the ground-truth image,respectively. In this paper, the weight parameters of the objective function w1=w2=0.5.
f1, f2 are loss rate and noise rate, respectively. f measures the accuracy of crack detection. f is larger value, theaccuracy is higher.
3. DEEP NEURAL NETWORKThe output results of the image processing technique are crack images including crack pixels andcrack-like non-crack pixels. The brightness of crack-like non-crack pixels is similar to the brightness of thecrack pixels. The brightness is intensity value of pixel of binary image in 0 (black) or 255 (white). Figure 5
shows a crack image result of fully automated crack detection method using IPTs combined GA. As a result,95

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019the crack image contains many noise and crack like non-crack. The major challenge is how to classify them
automatically in order to only keep true crack pixels.

Figure 5. Application of deep learning method for true-crack pixel detection

Figure 6. An example of convolution operation on a matrix of size 5x5 with a kernel of size 3x3.

Table 2 Dimension of layers and parameters
Layer
Name

Numberof Filter

Size

StridePadding

Layer
Name

Numberof Filter

Size

StridePadding

C1

32

16x16

2

2

C3

128

6x6

2

2

M-P1

1

3x3

2

-

M-P3

1

2x2

2

-

Dr1

1

Dr2

1

C2

64

7x7

2

2

FCL1

1

128

-

-

M-P2

1

3x3

2

-

FCL2

1

2

-

-

3.1 Convolution Layer
The DCNN composed of three stages. Each stage comprises of convolution layer, batchnormalization layer, max-pooling layer with drop out, and activate function (Cha et al.2017). The last layer isfully-connected layer so as to map noise candidates or crack candidates. Figure 6 shows the convolution

96Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019layers consisted of the number of filters and size of a filter. The purpose of performing convolution layer is to
extract local feature map. To the best of our knowledge, given a training data set {(ui,vi); i=1,2,..,n}, ui  R3 isthe input data, the target output vi  R. The convolution layers compute feature maps in the following
equation:(2)Fj= ) (6ui×Wj+bj)(3)
Wj=[wij]kwhere Wj, k, and bj are the weight vector, size of the weight matrix and bias vector, respectively, of the
convolution of kernel j. Wj is calculated from equation 3. wij are connecting the weight parameters. wij will be
updated continuously from one convolution layer to another layer. Therefore the different convolution layer
kernel will be extract different features from the input data. )( ) illustrates as an activation function. The
rectified linear unit (RELU) function was utilized in our experiment. Fj represents the feature maps achievedby convolutional kernel j.
As shown in figure 6, the output size is calculated in the following form:ூିிାଶൈ௉
ହିଷାଶൈ଴൅ͳൌ൅ ͳ ൌ ͵
(4)
ܱൌ
ௌ

ଵ

I: input matrix size; F: Filter size; P: Padding size; S: Stride size. As a result, the information of the output
matrix is compressed depending on the predefined mask.
3.2 Max Pooling Layer

5

1

3

7

4

4

3

5

1

1

7

7

3

8

2

9

Figure 7. An example of max pooling layert theth computationt ti complexity,
l it bbutt it also
l capturestll of relative
Pooling layer is adapted to mitigate
allglobal features of the training data invariantly. In this experiment, max pooling layer is used to keep thestrongest features and reduce the size of previous layer. The stride size of max pooling is 2 pixels shown infigure 7.
Table 2 indicates dimension of layers of DCNN shown in Figure 1 as well as parameters of stride andpadding. Where C1, C2, and C3 are the convolution layers; M-Pl, M-P2, and M-P3 are the max pooling.
FCL1and FCL2 are fully-connected layers. Dr1and Dr2 are drop out with a predefined probability ratio (Tonget al.2018). In this paper, these probability ratios are 0.4.
4. IMAGE DATASETThe image dataset are acquired by digital camera Canon 50EOS and iphone 6 plus with (4752x3168)
pixels and (3264x2248) pixels, respectively. The total of data image is 350 images. They were divided into
300x300 pixels to process images and extract crack/noise candidates. These have the different size. To
increase training images and to avoid over-fitting phenomenon, the original sub images rotate clock wises,counter clockwise, upside down, and leftside right. Moreover, several noise pixels along the shape of crack
candidates extracted from IPTs were cleaned to improve training progress. Therefore, the total of training data
include 5000 images of crack candidates and 5000 images of noise candidates resized to 100x100 pixels
divided to 5 sub sets (1000, 2000, 3000, 4000, and 5000 images) to test the accuracy of the proposed model.
Therein, the training and validation data are divided into 80% and 20% percentage, respectively. The purposeof the validation data is to adjust the parameters of the training progress until acquired an optimum-learningmodel. The testing data will be tested by this model. Figure 8 shows some samples in the training data. The
accuracy of training model is depending on the diverse of image data.

97

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019Non-crack candidates

Crack candidates

Figure 8. Some samples of training data with 100x100 pixels
5. EXPERIMENT RESULTS
The accuracy of the classification is expressed as the following equation:

ACC

TP  TNTP  TN  FP  FN

(5)

Where TP is the total of crack images detected correctly, TN is the total of non-crack images detected
correctly. FP is the total of the crack images detected incorrectly. FN is the total of the non-crack images
detected incorrectly.
Figure 9 shows the experiment results of the proposed training model with various image data sizes. Training
maximum accuracy (ACC) is 96.1%, the test data accuracy is 91% with 3000 images for each class. Moreover,the minimum testing accuracy belonged to 4000 image data set is 87.5%. Furthermore, Figure 10 shows
comparison of the training accuracy between the proposed method and Cha_2017 method. The result expresses
that the proposed method outperforms the other, and it gains maximum accuracy at epoch 30.
6. CONCLUSION
This paper found the optimization of parameters of IPTs using GA. Moreover, the IPTs results
combined with DCNN method resulted in high automatic crack detection. The accuracy of training model isdepending on the diverse of input data. The difference of accuracy between training and testing is large. It isnecessary to improve more deep CNN layers to get high score.
The final purpose was to create crack map therefore requiring the pixel-level accuracy automatically.
Disadvantage of DCNN method needed a large number of input data so as to gain the high fixed accuracy.
More, the computation of DCNN model is heavy relied on GPU and computer configuration.
ACKNOWLEDGMENT
This research is supported by JSPS KAKENHI grant number 15K06180.
REFERENCES
Barharnsakun (2017). A Hybrid ABC-ANN for Pavement Surface Distress Ditection and Classification, Int.J.
Mach. Learn & Cyber, Vol 8, pp.699-710.

98

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The accuracy of different training data sizes
Training-Accuracy
94.53
91

1000

93.75
89.55

2000

Testing-Accuracy

96.09

94.53
91

95
90.5

87.5

3000

4000

5000

Figure 9. The experiment results of the various training image data.

Figure 10. Comparison result between the proposed method and Cha_2017

Cha,Y., Choi,W. (2017). Deep Learning-Based Crack Damage Detection Using Convolutional Neural
Networks, Computer-aided Civil and Insfrastrure Engineering , vol. 32, pp. 361–378.
Fujita,Y., Mitani Y., and Hamamoto Y. (2006). A method for crack detection on a concrete structure,
ICPR2006: IEEE 18th Int.Conf, on Pattern Recognition, Vol.3, New York, pp.901-904.
Kawamura, K., Yoshino, K., Nakamura, H., Sato, Tarighat, A. (2013). A Valid Parameter Range
Identification Method of a Digital Image Processing Algorithm for Concrete Surface Cracks Detection Using
Genetic Algorithm and Decision Tree, Proc. Japan Soc. Civ. Eng. Vol.69.
Kawamura, K., Miyamoto, Y., Nakamura, A., H., Sato, R. (2003). Proposal of a crack pattern extraction
method from digital images using an interactive genetic algorithm, Proc. Japan Soc. Civ. Eng. Vol.72,
pp.115-131.
Kim, H., Ahn, E., shin, M., Sim., S. (2018) Crack and Non-crack classification from Concrete Surface Image
Using Machine Learning, Structural Health Monitoring, pp.1-14.
Nguyen, C., Kawamura, K., Tarighat, A. (2016) A study on semi-automatic concrete cracks detection using
interactive genetic algorithm, JCI, vol 38(1), pp.2061-2066.
Tong, Z., Gao, J., Sha, A., and Hu, L., (2018). Convolution neural network for asphalt pavement surface
texture analysis, Computer-aided civil and infrastructure engineering, Vol 0, pp.1-17.
Zhang, W., Zhang, Z., Qi, D., Liu,Y. (2014). Automatic crack detection and classification method for subway
tunnel safety monitoring, Sensors .Vol.14, pp.19307-19328.

99

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

AS-BUILT MODELING OF STEEL STRUCURES USING SYMMETRY
Takuya Suzuki 1, Tomohiro Mizoguchi 2
1) Master Student, Department of Computer Science, College of Engineering, Nihon University, Fukushima, Japan.
2) Associate Prof., Department of Computer Science, College of Engineering, Nihon University, Fukushima, Japan. Email:mizoguchi.tomohiro@nihon-u.ac.jp

Abstract:
As numerous number of civil infrastructures are aging in Japan, new information technologies are required tosupport efficient and effective maintenance of infrastructures. As-built modeling is one of the techniques thatreconstructs 3D model reflecting the current status of the structure from point cloud acquired by laser scanner.
Created modes are effectively used in various stages of inspection, such as prior scheduling before inspection andrecording of inspection results on the 3D model after inspection. We focus on steel structures in bridges in thiswork. Many steel members are used in bridge and they are arranged in complicated manner. However, from theviewpoint of function and aesthetics of bridges, steel members are arranged according to the specific rules definedby the designer. For example, steel members are oriented in the vertical direction and are arranged at regularintervals in the horizontal direction. In addition, they are arranged symmetrically in the front-rear and right-leftsides of the bridge. Therefore, detecting such symmetry from point cloud and utilizing them in modeling process
can improve efficiency of as-built modeling, the quality of the 3D reconstructed model, and robustness to theabsence of point cloud, compared with the conventional methods in which each steel member is modeled
individually. In this paper, we propose a new method for reconstructing high-quality 3D model of steel structuresof bridges from laser scanned point cloud enhancing symmetry. The method first detects multiple planar reflection
symmetry which dominate the large portion of bridge by shape matching. The next step detects each steel members
efficiently by simplifying point cloud based on voxelization, extracting skeleton structure, and evaluating point
distribution using principal component analysis. Then the members arranged in symmetrical positions and posture
are grouped together, and finally 3D template models are fitted to the group members simultaneously undersymmetry constraints. Experimental results demonstrate good performance of our proposed method.

Keywords:

Terrestrial Laser Scanner, As-Built Modeling, Symmetry.

1. INTRODUCTION
1.1 Background
Recently aging of civil infrastructure structures in Japan built during the period of high economic growthhas become a serious problem. Consequently, the importance of maintenance is increasing to prevent big accidents.
However, it is difficult to inspect all the bridges finely since it needs the significantly high cost and the number ofskilled inspectors is decreasing. Therefore, new technologies are required to support the maintenance. One of suchtechnologies is 3D scanning of the structures using laser scanner. This enables the detailed documentation of thestructure as a dense point cloud reflecting the current states. Many works have been studied to construct lightweight as-built 3D models from point cloud (Kanai, 2018, Nan, 2010). This modeling convert large point cloudwith hundreds million points to light-weighted polygonal models with only a few thousands triangles, which makes
post-processing easier. The constructed 3D models can be widely used in various stages of inspection ranging
from prior scheduling to post recording of the inspection results such as captured images of damages or degradation(Tanaka, 2018). We focus on steel structures of bridges as shown in Figure 1. Many steel members are used in a
structure and are arranged in a complex manner. In addition, access is limited in the laser scanning process and theentire structures are not scanned in detail, thus occlusions are unavoidable and the several areas are missing in thescanned point cloud. Therefore, many manual operations are required to select individual members in point cloudusing commercial software. Thus the automatic methods are expected to create 3D models of steel structures.
As-built modeling process of steel structures consists of two main steps. The first step extracts subsets ofpoints each of which corresponds to a single steel member, and the second step creates 3D polygonal model by
fitting a template model (Lu, 2018). In the modeling process of steel structures, many members must be extracted
individually from incomplete point cloud with many occlusions. Moreover, the posture and positions of models
must be estimated when the corresponding point cloud is missing. On the other hand, from the viewpoints offunctionality and aesthetics, steel members are arranged under the symmetry constraints that the designer specifies.
For examples, members with vertical directions are arranged at the constant intervals of distances and angles along
the horizontal directions as shown in Figure 2(a). In addition, left-right and front-back sides are both symmetricalso that they follow planar reflection configurations as in Figure 2(b). Thus detecting such symmetry or repetitions
and utilizing them in the modeling process has several advantages in the sense of modeling efficiency, quality of
the resulting models, and robustness to the occlusions, over the previous methods where each parts are modeled100Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019individually (Nan, 2010).
1.2

Related Works
Many woks have been reported for 3D modeling from laser scans using symmetry in computer graphics
field (Nan, 2010). These methods try to find subsets of points corresponding to repetitively appearing objects, suchas windows of multistory buildings, and their repetitive patterns, such as intervals and directions. This enables toestimate the positions and postures of the models for the missing area, and to create entire models by fitting
template models simultaneously under the repetition constrains. The method (Lin, 2013) focused on low-rise
residences and proposed to detect planar reflection symmetry from point cloud and to use it in the 3D modelingprocess so that models can be created when the large portions of the residences are missing. However, thesemethods cannot be directly used for steel structures in which many members are arranged in a complex manner.
As for the as-built modeling of steel structures, the method in (Laefar, 2017) recognizes cross sectional
2D shape of the member from registered laser scan, and create 3D models by extruding the shape. Another method
is proposed by (Kanai et al., 2018) where multiple planes are robustly extracted from point cloud of a steel member,and its cross-sectional shape and size are estimated by analyzing geometric relations of extracted planes. The bothmethods focused on a single member and creating a 3D models by evaluating the cross sectional shape. In contrast
to them, our focus is not only the detection of multiple steel members, but also the automatic detection of symmetry
from the point cloud that should be defined in the designing stages of the bridge and the efficient creation of theentire model of a steel structure by analyzing mutual relations of multiple members under the symmetry constraints.

(b) Point cloud(a) Steel bridge(c) Point cloud of steel members

Figure 1. Laser scanning of steel bridge(a) Equal distance and angles(b) Symmetry of left and right,
and front and back sides

Figure 2. Examples of symmetry in steel structures
1.3

Our Purpose and Overview of Our Proposed Method
In this paper, we propose a method for efficiently creating a high-quality 3D as-built model from point
cloud of a steel structure of bridge using symmetry. Figure 3 shows the overview of our proposed method. Givena laser scan of a steel structure, multiple global reflection symmetries are extracted which dominate large portionsof the structure (Step1). Next, multiple steel members are detected efficiently (Step2). Then, members arranged
symmetrically are grouped together (Step3), and finally template models are simultaneously fitted to multiple
member points in a group (Step4).
The advantages of our method are summarized as follows: (1) it can efficiently detect multiple steel
members from large point cloud, (2) it can estimate the positions and postures of members in the data missing area101Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019using symmetry constraints, and (3) it can create a high-quality as-built model enhancing symmetry.
Step1
Point
cloud

Multiple planar r eflection symmetr y

Detec tion ofsy mmetry• Nor mal computation• ICP alignment

Step2
Detec tion ofsteel members• Vox elization• Thinning
• PCA

Step3
Grouping ofsteel members

Step4

3D modelc reation• Evaluation of pointto point distance

3D model• Simultaneous fitting oftemplate models

Figure 3. Overview of our proposed method
1.4

Test Site and Scanner
We used the point cloud of steel bridge shown in Figure 1(a) captured by TOPCON GLS-2000L scanner.
Figure 1(b) shows the entire point cloud, and Figure 1(c) shows the part of it which are used in this paper. The
numbers of point cloud are 7,847,897 and 3,434,436 respectively.

2. OUR PROPOSED METHOD
In this chapter, we describe the details of our proposed method.
2.1

Detection of Symmetry (Step1)
The proposed method first detects multiple global reflection symmetries which cover the large portion ofpoint cloud through alignment of pair of point cloud (Mitra, 2006; Mizoguchi, 2013). Figure 4 shows the overview
of this method. The input point cloud M is reflected to an arbitrary plane, and its reflected point cloud O is
generated. Then O is transformed to O’ such that O’ is tightly aligned to M by Iterative Closest Point (ICP) method.
The point-to-point correspondences between M and O’ can be obtained as a result of alignment. In figure 4, forexample, p1 in M is matched to q3 in O’. The point q3 is generated by reflecting p3 in M. Thus p1 and p3 can bematched under a certain planar reflection. Similarly, seven pairs can be found in this illustration. As shown in theillustration, the set of midpoints of point pairs are distributed on the reflection plane, which can be simply
computed by fitting a least-squares plane to them. In the ICP matching process, using normal vectors of each point
leads to higher accuracy of alignment and symmetry detection. Figure 5 shows the result of symmetry detection
where two dominant reflection planes are visualized.
Arbitrary plane
Point cloud M

p1

p4

p2

p5

p3

p6

q3
p7 q
7

q6

Reflected
point cloud O

q2

q5(1) Reflection to arbitrary plane

Cor r espondence

p1

q3

p2

q2

Reflection plane

p3

p1

q1

p2

p3

q1
p4q7

q4

p5q6

p6q5

p7q4

p4

(2) Alignment

Figure 4. Overview of our symmetry detection method

Figure 5. An example of symmetry detection results

102

p5

p6

(3) Plane fittingp7Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2.2

Detection of Steel Members (Step2)
Next, many steel members are efficiently extracted individually from a point cloud. The shape of members
is long and narrow. The straightforward way to extract such member points is first to evaluate local point
distribution by Principal Component Analysis (PCA) to the local neighborhood. However, considering the size ofcross sectional shape of a member, e.g., a few tens of centimeters, the neighbor size must be set relatively larger
in the kd-tree based searching, e.g., 1 meter, to appropriately compute the linearity of the members, which makes
the computational cost very high. To solve this problem, we introduce voxelization and point thinning process toefficiently compute local point distribution. Figure 6 shows the process of our method. The method first converts
point cloud to its voxel representation. And then the thinning process is applied to the voxel model using voxel tovoxel connectivity. Then for each voxel, neighboring voxels are efficiently searched using the connectivity, andlinearity is computed at each voxel by evaluating point distribution. The voxels with high linearity is extracted bythresholding, the extracted voxels are clustered and labeled, and the clustering results are mapped to the original
point for extracting steel member points individually.
近傍サイズ:大Lar geneighbor hood size

Low linear ity

Computation of linear ity
on point cloudHigh linear ity

Computational cost is ver y lar ge

Point cloud

Point cloud

Feedback topoint cloudVox elization

Small
近傍サイズ:小neighbor hood size

Computationof linear ity

Thinning

Smaller number ofr epr esentative points

Linear ity at eachr epr esentative point

Figure 6. The overview of our steel member detection method

Step2-1: Voxelization
Point cloud is converted to the voxel representation. The size is set to 0.1 meter through various
experiments. The number of voxels is approximately 400×150×50 for the example in Figure 1(c). The cell
including at least one point is called active cell. The number of active cell is 55,552. This number is muchsmaller than that of point cloud, which makes the following computation much faster.
Step2-2: Thinning by Laplacian Smoothing
Next, the representative points are computed as the barycenter of included points at each active cell, andthe graph is constructed by connecting the representative point using voxel connectivity. Laplacian smoothing isapplied to the graph iteratively, and the coordinates of the representative points are updated, resulting in thethinning of points. The number of iteration is set to 20 for the example in this paper. After thinning process, thelinearity can be appropriately computed at each voxel with smaller neighboring size, leading to faster computation.
Figure 7(a) and 7(b) shows the examples of thinning results.

Step2-3: Linearity Computation
Then, for each cell, the neighboring cells are searched. The neighboring size was set to 0.5 meter in thispaper. The PCA is applied to the set of the neighboring cells and the linearity ki is computed using three eigenvaluesby eq.(1).
ki 

max
1  2  3

(1)

Here 1 , 2 , 3 are the three eigenvalues, and max is the maximum value among them. The linearity ki represents
how well point sets are distributed along the line. The linearity ki = 1.0 shows that the points are completely
distributed on a line. Similarly, ki = 0.5 and ki = 0.33shows that the points are completely distributed on a plane or103Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019sphere respectively. The computed result is shown in Figure 7(c). As shown in this figure, linearity is relatively
high at the point of steel members, and small at other portions. By using the threshold thline, points with highlinearity are extracted as shown in Figure 7(d). The number of extracted cells is 19,129. We set thline = 0.85 in thisexample. This results show that our method can detect individual steel members appropriately.

(a) Point cloud(b) Representative points
1(c) Linearity(d) Extracted points

Figure 7. Thinning process and extraction of steel member points

Step2-4: Parts Detection by Clustering
Finally, extracted cells are recursively searched and clustered, and the part label is allocated to eachconnected cell sets. We removed the cluster for the post processing with less than 15 cells. Allocated labels are
mapped to the original points, and the point subsets each of which corresponds to a steel member can be extracted.
2.3Grouping of Steel Members using Symmetry (Step3)
The next step finds the congruent sets of steel members under the planar reflection detected in step 1 andgroup them. To do this, each clustered points are reflected and found the congruent clusters by ICP matching.
This process is applied to all the pair of clusters.

2.4

3D Model Creation by Simultaneous Template Fitting (Step4)
In the last step, 3D simplified model is constructed by simultaneously fitting template models to eachgroup of point subsets using symmetry. Figure 8 shows the overview of this method. To simplify the fitting process,our method first transforms the point subsets in a group under the planar reflections and combines the point subset.
Then for the combined points, a template model if fitted by ICP matching. In this work, template model is also
defined as a point cloud. After the ICP matching, a template model is transformed under the inverse planarreflections. This enables the construction of multiple steel member model simultaneously. In this method, thematching error is distributed evenly to each member.
The template models are created using commercial software Rapdiform XOR3. For this, the crosssectional shapes are evaluated by slicing point cloud of steel members, and the size is approximately defined. We
prepared two models in this example.

(1) Combination of gr ouped
point by r eflection oper ation(2) Template model fitting to
combined point cloud(3) Inver se r eflection oper ation
to fitted template model

Figure 8. Overview of our simultaneous fitting process of a template model using symmetry104

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3. RESULTS AND DISCUSSION
This chapter describes the experimental result of symmetry detection (Step 1), extraction of steel member(Step 2), and 3D model reconstruction by simultaneous template fitting (Step 4) respectively.
3.1

Results of Symmetry Detection (Step1)
Accuracy of symmetry detection is strongly related to ICP alignment accuracy. Figure 9(a) and 9(c) show
the results of point cloud alignment for front-back planar reflection, and Figure 9(b) and 9(d) for left-rightsymmetry. In both figures, red represents original point cloud, and blue show the points reflected. Figure 9(c) and9(d) show the alignment error. In these figures, points within 30mm error are represented by color, and others withblack. As for the front-back symmetry, the ratio of points within 30mm error was 71.8%, and the averaged error
was 10.0mm. As for the left-right symmetry, the ratio within 30mm error was 57.8%, and the averaged error was
6.6mm. Reasonable results were obtained in both cases. The reason for the difference of averaged error should be
the point density. In the alignment for front-back symmetry, the portion with high point density was aligned tothose with low density, resulting in relatively large alignment error. On the other hand, in the alignment for leftright symmetry, the portion with high point density was aligned to those with high density, and vice versa. This
type of pair can be tightly aligned with low alignment error.

(a) Alignment for front and back sides symmetry(b) Alignment for right and left sides symmetry
30mm

30mm

0mm

0mm

(c) Error distribution for front and back sides symmetry(d) Error distribution for right and left sides symmetry

Figure 9. Accuracy evaluation of symmetry detection
3.2

Results of Steel Member Extraction (Step2)
Figure 10(a) shows the results of steel member extraction. In this example, 32 members are used, and our
method could extract all of them individually as shown in figure10(a). Figure 10(b) shows the classification of
extracted members. As shown in red, single members were divided into multiple parts by occlusions, and extracted
as a member. As shown in blue, only some parts were scanned and the corresponding points were detected as amember. As for the computational time, our voxelization and thinning process enabled the fast computation ofsteel member extraction in 23 seconds.

(a)

(b)

(c)

Figure 10. Extraction of steel member and grouping
3.3

Results of 3D Model Reconstruction (Step4)
Figure 11(a) shows the fitting results of the template model and the error evaluation. In figure 11(a) left,
blue and red represent the input point cloud and fitted template model as a point cloud format respectively. The
averaged error was 8.9mm. Normal vectors were not used in this matching process. Therefore, the front and backsurfaces on the member were not distinguished in the matching process, and the relatively large errors were105

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

generated compared to the laser scanning accuracy, i.e., 2 or 3 millimeters. Figure 11(b) shows the simultaneous
fitting results of a template model to four members in a group. The averaged error was 9.2mm and the large
variations could not be found between four members. The generated error was approximately the same with thecase of a single member, and thus our method enabled the accurate and efficient fitting of template models using
symmetry.
Figure 12(a) shows the result of simultaneous fitting for all the group. The average error was about 9.5mm.
Some variations were generated ranging from 11.7mm to 8.6mm as shown in figure 12(b). Figure 12(c) shows thegrouped point cloud and figure 12(d) shows the fitted template models in point cloud format. The advantage ofthis method can be found in this example, where the template models could be appropriately fitted to the portions
with occlusion as in figure 12(c) and 12(d).
20mm

20mm

0mm

0mm

(a)

(b)

Figure 11. Accuracy evaluation of simultaneous template model fitting to steel members
20mm

0mm

(a)

(b)

(c)

(d)

Figure 12. Result of simultaneous template model fitting
4. CONCLUSIONSIn this paper, we proposed a new method for efficient and high quality as-built modeling of steel structuresusing symmetry. We demonstrated that multiple planar reflection symmetries were accurately extracted, and thatmultiple steel members could be efficiently extracted through voxelization and thinning process. In addition, we
verified that template models can be simultaneously and accurately fitted to multiple steel members.
Future works includes the extension of this method to multiple scans. More accurate fitting of templatemodels using normal vectors must be solved. Many applications can be developed using a constructed 3D model,such as 3D recording of inspection results. The constructed models are light-weight polygonal models, and thus
images can be attached as texture for realistic visualization. And such light-weight models can be visualized fast
on tablet or AR-glasses, and thus developing new applications is an interesting direction.
ACKNOWLEDGMENTS
We thank Mutsu Tech Consultant Co. Ltd for providing the laser scanner.
REFERENCES
Lin，H., Gao, J., Zhou, Y., Lu, G., Ye, M., Zhang, C., Liu, L., and Yang, R. (2013). Semantic Decomposition andReconstruction of Residential Scenes from LiDAR Data, ACM Transaction on Graphics, 32(4), 66
Laefer, D. F. and Truong-Hong, L. (2017). Toward Automatic generation of 3D Steel Structures for Building
Information Modeling, Automation in Construction, 74, 66-77.
Kanai, S., Hashikawa, M., and Date, H. (2018). Automated recognition and 3D CAD modeling of standardized
steel bridge members in a laser scan, ICCCBE2018 17th International Conference on Computing in Civil
and Building Engineering.
Lu, R., Brilakis, I., and Middleton, C. R. (2018). Detection of Key Components of Existing Bridge in Point Cloud106

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Datasets, ICCCBE2018 17th International Conference on Computing in Civil and Building Engineering.
Tanaka, F., Hori, M., Onosato, M, Date, H., and Kanai, S. (2018). Bridge Information Model Based on IFC
Standards and Web Content Providing System for Supporting an Inspection Process, ICCCBE2018 17th
International Conference on Computing in Civil and Building Engineering.
Mizoguchi, T., Kanai, S., Date, H., and Tanaka, H. (2013). Robust and Exhaustive Method for Symmetry Detection
from Scanned Meshes, Journals of Advanced Mechanical Design, Systems, and Manufacturing, 7 (5),pp.862-875.
Mitra, N., Guibas, L. J., and Pauly, M. (2006). Partial and Approximate Symmetry Detection for 3D Geometry
Symmetry in 3D Geometry, SIGGRAPH .
Nan, L. L., Sharf, A., Zhang, H., Cohen-Or, D., and Chen, B. (2010). SmartBoxes for Interactive Urban
Reconstruction, ACM Transaction on Graphics, 25 (3), 560-568.

107

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

A REVIEW OF RESEARCH ON ADVANCED AUTONOMOUS TECHNOLOGIESFOR BRIDGE INSPECTION
Rina Hasuike1, Koji Kinoshita2, Lei Hou3
1) Ph.D. Candidate, Department of Civil Engineering, Gifu University, Gifu, Japan. Email: w3912010@edu.gifu-u.ac.jp
2) Dr. Eng., Associate Professor, Department of Civil Engineering, Gifu University, Gifu, Japan. Email: kinosita@gifu-u.ac.jp
3) Dr. Eng., Senior Lecturer, School of Civil Engineering, RMIT University, Melbourne, Australia. Email: lei.hou@rmit.edu.au

Abstract: Nowadays, many of these infrastructure projects are having serious aging issues, and therefore, timely
inspection and repair works are crucial to ensure these structures are safe to use. So far, bridge inspection has been
conducted primarily based on visual inspection by human inspectors. However, the downside of visual inspection
is apparent. Firstly, conducting visual inspections can be quite time-consuming, especially in large and complex
bridge structures; and secondly, this method cannot well address occupational health and safety issues such as
working at height. Under this consideration, applying advanced autonomous technologies such as Unmanned Aerial Vehicles (UAVs) and image processing techniques to support human inspectors has attracted growing interestin the industry. In this study, a literature review was conducted with a focus placed around identifying the bestpractice of current technological approaches and proposing the futuristic research directions for technological refinement and improvement. Methodically, this study examined the publications over the past 20 years, i.e., from1999 to 2018, and leveraged the tool of CiteSpace to derive significant review findings on UAV and defect detection research.
Keywords: unmanned aerial vehicles, image processing, bridge inspection
1. INTRODUCTION
Infrastructure projects range across critical civil structures such as bridges, roads and tunnels. Nowadays,
many of these infrastructure projects are having serious aging issues, and therefore, timely inspection and repair
works are crucial to ensure these structures are safe to use. That is because those aging issues are caused by lackof structure issue identification. If these problems cannot be promptly identified, use of these aging bridges would
eventually incur fatal structure failures such as collapse. It is therefore understood that constant structure inspections with precise understanding on structure problems are critical to assure the safety and functional performanceof these assets.
A bridge has to be designed to meet the structural requirements, which involve load, vibration, weather
and many other design codes. However, the lack of appropriate inspections and shortage of fixing for damages
caused by the aging or some unexpected load such as earthquake, it could lead to a downgraded condition level ofthe structure. In order to prevent such downgrades, periodic inspection of bridges is a common practice around theworld. For instance, in United States (U.S), bridge inspection is mandated by the U.S. department of Transportation(2004). In Australia, Part 7 of the Austroads Guide to Bridge Technology (AGBT) was published in 2009. In the
AGBT, it is mandated that Level 1 inspections, which is routine maintenance/surveillance inspection, are conducted on a six month to one year cycle depending on a risk assessment (Austroads, 2018). In Japan, it is mandated
that in every 5 years time, there has to be at least one inspection by close distance has been conducted since 2014
based on Ordinance for Enforcement of Road Act (Road Bureau, 2019a). In practice, bridge inspections in thosecountries are typically conducted primarily by human inspectors. However, the downside of visual inspection isapparent. Firstly, conducting visual inspections can be quite time-consuming especially in large and complex
bridge structures; and secondly, this method cannot well address occupational health and safety issues such as
working at height. Under this background, how to apply advanced autonomous technologies such as Unmanned
Aerial Vehicles (UAV) and image processing techniques to assist human inspectors has become a heated research
topic in recent years. For example, in Japan, many kinds of advanced autonomous technology for bridge inspection
have been developed and some trials have been conducted at existing bridges (Salaan at el., 2018), and image
processing is especially used for crack identification (Chun & Igo, 2015). In addition, the project titled "Infrastructure Maintenance, Renovation and Management", which is one of the projects under the Cross-ministerial Strategic
Innovation Promotion Program (SIP), was conducted by Cabinet Office, Government of Japan in order to accelerate the research and development on maintenance technologies from 2014 to 2018. In this project, many kindsof new advanced autonomous technology were developed and tested. Therefore, the development of technology
which uses for infrastructure inspection has become a hot research field.
In this study, a literature review had been conducted with a focus placed around identifying the best practice of current technological approaches and proposing the futuristic research directions for technological refinement and improvement. Methodically, this study had examined the publications over the past 20 years, i.e., from1999 to 2018, and leveraged the tool of CiteSpace to derive significant review findings including to use thoseadvanced autonomous technologies for bridge inspection and to improve the safety of inspectors.

108

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2. REVIEW METHOD
In order to find out the current research on the applying advanced autonomous technologies to bridge
inspection in recent years, the search of articles was implemented based upon a web system named Web of Science(WoS). In this study, the search code of TS, a.k.a. topics, was set as “bridge inspection”. Furthermore, the scopeof the search was set to be between 1999 and 2018 to identify changes in heated research areas. As a result, 419
articles were identified (a total number of 90 between 1999 and 2008, a total number of 21 between 2009 and 2013
and a total number of 208 between 2014 and 2018). The increased amount of publications over the last two decades
indicates that bridge inspection has become a heated research topic. The research topic includes some themes, forexample, utilizing UAV/UAS, non-destructive testing (NDT) and damage detection by using image processing,etc.
Subsequently, CiteSpace 5.3.R9 was used to find research hotspots and analysed the relations of researchby conducting co-citation network analysis. Based on the result of the analysis, the best practice of current technological approaches and the details of researches were clarified.
3.

ANALYSIS OF THE KNOWLEDGE DOMAINS AND KEY CLUSTERS
Figure 1 shows the result of co-citation analysis and labeling clusters with title terms. In this study, adocument co-citation network that contains 252 nodes and 819 links was generated. The modularity Q and themean silhouette scores are two important metrics that can tell about the overall structural properties of the network(Chen, 2014). In this analysis, the modularity Q was 0.78 which means the network is reasonably divided intoloosely coupled clusters. The mean silhouette score of 0.46 suggests that the homogeneity of these clusters on
average is not very high, but not very low either. According to the network, the articles were divided to 49 clusters
by title terms, and some large clusters were shown in Figure 1. Clusters are shown with “#” and the numbers andthe size of words mean the cluster’s scale. These significant clusters identified that the highlighted research areain “bridge inspection” researches. As shown in Figure 1, the clusters show some advanced autonomous technologies such as #0 unmanned aerial system, #1 defect detection, #2 unmanned aerial vehicle application, #3 infrared
thermography and so on.
Both of cluster #0 and #2 indicated about the unmanned aerial vehicle (UAV), which include developing
UAV, enhancing UAV’s performance, proposing the efficient flight path and suggesting utilizing ways. One of
the biggest problems in developing UAV is to keep stable while the flights in various environments. To overcome
this problem, many kinds of UAV have been developed and compared in some test places (Khaloo et al., 2018;
Dorafshan et al., 2018b; Ikeda et al., 2018; Salaan et al., 2018). In addition, a crack detection method, which uses
the images that were obtained by UAV, has been established by using image processing, deep learning and so on(Kim et al., 2018; Zhong et al., 2018; Lei et al., 2018; Leea et al., 2018) . Not only crack detection but also tocreate 3D models by images or using laser scanning have been a hot research area (Khaloo et al., 2017; Kim et al.,2018). Therefore, researches about utilizing UAV have been conducted to obtain images and to use those data
effectively. The issues are also clarified in those studies, i.e., to get a stable flight in a GPS-denied environment

Figure 1. An overview of the co-citation network

109

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019because it relies heavily on GPS signals (Dorafshan et al., 2018b; Reagan et al., 2018), and in unfavorable weather
conditions (Seo et al., 2018; Zhong et al., 2018). Regarding to apply UAVs for bridge inspection, a survey was
conducted in the U.S. by Duque et al. (2018). According to the survey for Departments of Transportation (DOTs),
one state had used a drone for bridge inspection and six DOTs were planning to use drones in the near future(Duque et al., 2018). As shown in this survey, interest in utilizing drones becomes increasing.
Cluster #1 focusses on defect detection, which includes detecting cracks, rebar corrosion, and other many
types of defects. In this research area, concrete crack detection is one of the most highlighted areas (Dorafshan etal., 2018a; Kim et al., 2018; Zhong et al.,2018). There are two main detection methods, one based on image(Dorafshan et al., 2018a ; Dorafshan et al., 2018b ; Kim et al., 2018) and the other based on point cloud data
(Turkan et al., 2018; Huthwohl & Brilakis, 2018), and deep learning is used in each. In terms of using deep learningfor image-based crack detection, Dorafshan et al. (2018a) compared the performance of common edge detectors
and deep convolutional neural networks (DCNN) in concrete structures, and it showed that the method used DCNN
was better than the common one. Fan et al. (2018) proposed a crack detection algorithm based on the mesoscale
geometric features to effectively distinguish cracks and false cracks. Huthwohl & Brilakis (2018) presented amethod to automatically identify regions of interest in order to reduce the inspection space to areas which can thensubsequently be inspected by a human engineer or by an automated defect classifier. Those crack detection methods target is not only concrete but also fatigue crack in steel (Dorafshan et al., 2018b). Moreover, a new technique,which possesses the ability to extract bridge dynamic properties from the responses of a vehicle that passes overthe bridge at high speed was introduced (Elhattab et al., 2018), and to detect rebar corrosion in concrete structureby using ground penetrating radar (GPR) has been used (Eisenmann et al., 2018). Thus, the researches about defectdetection have been conducted from many kinds of approach.
Cluster #3 mentions infrared thermography which has been used to bridge inspections. Hiasa et al. (2017a,
2017b, 2017c) evaluated the accuracy and reliability of infrared thermography (IRT) for high-speed applications
to detect concrete delaminations. This technology does not require the closing lane, and the test result showed highaccurate damage detection. On the other hand, it is considered that to combine IRT and other technologies such as
UAV or GPR (Abu Dabous et al., 2017; Mader et al., 2016). Abu Dabous et al. (2017) introduced an integrated
method utilizing IRT and GPR technologies to enhance the detection of concrete bridge defects.
4. THE BEST PRACTICE AND PROBLEM
As summarized in Section 3, the autonomous technologies which are used for bridge inspection have been
researched from various areas ranging from data acquisition to analysis. These technologies are developed not
only separation but also combination, e.g. to obtain data by using UAV and to analyse those data by using image
processing (Kim et al., 2018; Zhong et al., 2018). Thus, it is clear that combining techniques will allow more
efficient inspection. Each technology is in the stage of demonstration or experiments at existing bridges and comparison with conventional methods, and the results show possibilities of those technologies utilization for bridge
inspection.
Focusing on the subject of those technologies, many technologies and researches target concrete bridges,
and there is a tendency that many researches focused on cracks among damage. It is assumed that cracks in concrete
are selected as the damage that is easy to detect and check by using image processing or deep learning technology,
as the first step of using those technologies for bridge inspections. Subsequently, in addition to cracks, other typesof damage have been attempted to be detected as well, e.g. delamination and spalling of the concrete deck. In theconventional inspection method of those damages requires traffic control because the inspectors need to be in close
distance with those damages. To conduct an effective inspection of them, research on using NDE which does not
require traffic control is actively implemented (Elhattab et al., 2018). Thus, it can be seen that the development ofdefect detection technologies for concrete bridges is actively promoted. However, autonomous technologies forsteels, such as detection of corrosion of rebar in reinforced concrete (Eisenmann et al., 2018) and fatigue cracks
on steel bridges (Dorafshan et al., 2018b), are fewer than those technologies for concrete. It will be desired thatfurther researches about steel in the future.
In many researches, the main purposes are to objectively evaluate the damage, increase the efficiency offieldwork, and ensure the safety of inspection work. These purposes have been achieved by the currently developed
technology. However, when applying these technologies to an existing bridge’s inspection, it is difficult becauseof the limitation by the law about inspection, and that utilization is only as a support for inspectors at this time. As
technology development and its verification progress, it is desired to ensure safety when using those technologies
and to draft the law about using autonomous technologies for bridge inspection.
5. DISCUSSIONAs the development of various technologies progresses, it can be said that the following certification or
evaluation are desired to apply for bridge inspection; (1) objective evaluation of technology performance (e.g.
specification of crack detection limit size and application condition); (2) certification of the safety on the structureevaluated by the technology (e.g. verification by comparison with the evaluation by conventional methods); (3)110Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019certification of the safety when using the technologies. If they are not proved, it will be difficult to draft the law
about using autonomous technologies or to mandate using those technologies in the law.
With regard to (1) and (2), evaluations are in progress by each technology developer, e.g. by comparison
with existing methods. One of the criteria for evaluating technology is to satisfy specific numerical values basedon the bridge inspection guidelines established by the local municipalities that manage the target bridge. Therefore,
further technical development is expected by defining and publishing the required performance when each local
municipalities use autonomous technology.
With regard to (3), it is a particularly legally sensitive part, for example in U.S. and Japan, there are strictly
flight safety restrictions for UAV (Federal Aviation Administration, 2019; Civil Aviation Bureau, 2019). To clear
those requirements, the research which improves the stability of the UAV and simplifies the operation have been
conducted. Some NDEs, including IR and GPR, have also been developed that can implement the inspection without traffic control, and it is suggested that those technologies can overcome those limitations.
To accelerate the application of autonomous technology, it is desired for each country to draft a law on
the application of those techniques to bridge inspection. In Japan, the Guideline for Periodic Road Bridge Inspection was revised in March 2019, and the order about visual inspection has been relaxed (Road Bureau, 2019b). It
is expected that further technological development and introduction will be promoted.
6. CONCLUSIONSIn this study, a literature review was conducted with a focus being placed around identifying the bestpractice of current technological approaches and proposing the futuristic research directions for technological refinement and improvement. The results are summarized as follows.
(1) The researches related to UAV are the most frequent. Those researches are being carried out on hardware such as the development of the UAV and enhancement of commercial UAV, and software suchas the proposal of flight routes for efficient information acquisition. Test flights on existing bridges
are also being conducted. It is pointed out for problems which include flight during bad weather, lossof image quality, and stable flight under non-GPS environment.
(2) The researches related to defect detection technology, high precision detection has become possible,
such as using image processing and deep learning. Many researches have been conducted on thesubject of concrete crack detection. In addition, researches using NDE for other damage such as delamination of concrete are also increasing. There are still few research concerning rebar in concrete,fatigue cracks and corrosion in steel bridges, etc., and further studies are desired.
(3) Research on NDE technology such as IR and GPR are also in progress. These technologies greatly
increase the possibility of conducting inspections without traffic controls.
(4) With the development of various technologies, when considering the application to existing bridges,
the application may be difficult due to legal restrictions. It is desired to draft the law after ensuring
the accuracy and safety of each technology.
REFERENCES
Abu Dabous, S., Yaghi, S., Alkass, S. and Moselhi, O. (2017). Concrete bridge deck condition assessment using
IR Thermography and Ground Penetrating Radar technologies, Automation in Construction, 81, 340-354.
Austroads. (2018). Part 7 of the Austroads Guide to Bridge Technology. Retrieved from Austroads website:
https://austroads.com.au/publications/bridges/agbt07
Chen, C. (2014). The CiteSpace Manual. Retrieved from CiteSpace website: http://cluster.ischool.drexel.edu/~cchen/citespace/CiteSpaceManual.pdf
Chun, P. and Igo, A. (2015). Crack detection from image using random forest, Journal of Japan Society of Civil
Engineers, Ser.F3 (Civil Engineering Informatics), Vol.71, No.2, Ⅰ_1-Ⅰ_8.
Civil Aviation Bureau. (2019). Japan’s safety rules on Unmanned Aircraft (UA)/Drone. Retrieved from Ministryof Land, Infrastructure and Transport, Japan website: http://www.mlit.go.jp/en/koku/uas.html
Dorafshan, S., Thomas, R. J. and Maguire, M. (2018a). Comparison of deep convolutional neural networks andedge detectors for image-based crack detection in concrete, Construction and Building Materials, 186,
1031-1045.
Dorafshan, S., Thomas, R. J. and Maguire, M. (2018b). Fatigue Crack Detection Using Unmanned Aerial Systems
in Fracture Critical Inspection of Steel Bridges, Journal of Bridge Engineering, 23(10).
Duque, L., Seo, J. and Wacker, J. (2018). Synthesis of Unmanned Aerial Vehicle Applications for Infrastructures,
Journal of Performance of Constructed Facilities, 32(4).
Eisenmann, D., Margetan, F. J. and Ellis, S. (2018). On The Use of Ground Penetrating Radar to Detect Rebar
Corrosion in Concrete Structures, 44th Annual Review of Progress in Quantitative Nondestructive Evaluation, Vol. 37 (Vol. 1949).

111

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Elhattab, A., Uddin, N. and Obrien, E. (2018). Drive-By Bridge Frequency Identification under Operational Roadway Speeds Employing Frequency Independent Underdamped Pinning Stochastic Resonance (FI-UPSR),
Sensors, 18(12).
Fan, Y. X., Zhao, Q. L., Ni, S. D., Rui, T., Ma, S. and Pang, N. (2018). Crack detection based on the mesoscale
geometric features for visual concrete bridge inspection, Journal of Electronic Imaging, 27(5).
Federal Aviation Administration. (2019). Unmanned Aircraft Systems Retrieved from Federal Aviation Administration website: https://www.faa.gov/uas/
Hiasa, S., Birgul, R. and Catbas, F. N. (2017a). Effect of Defect Size on Subsurface Defect Detectability and
Defect Depth Estimation for Concrete Structures by Infrared Thermography, Journal of Nondestructive
Evaluation, 36(3).
Hiasa, S., Birgul, R. and Catbas, F. N. (2017b). Investigation of effective utilization of infrared thermography(IRT) through advanced finite element modeling, Construction and Building Materials, 150, 295-309.
Hiasa, S., Catbas, F. N., Matsumoto, M. and Mitani, K. (2017). Considerations and Issues in the Utilization of
Infrared Thermography for Concrete Bridge Inspection at Normal Driving Speeds, Journal of Bridge Engineering, 22(11).
Huthwohl, P. and Brilakis, I. (2018). Detecting healthy concrete surfaces, Advanced Engineering Informatics, 37,
150-162.
Ikeda, T., Yasui, S., Minamiyama, S., Ohara, K., Ashizawa, S., Ichikawa, A., Okino, A., Oomichi, T. and Fukuda,
T. (2018). Stable impact and contact force control by UAV for inspection of floor slab of bridge, Advanced
Robotics, 32(19), 1061-1076.
Khaloo, A., Lattanzi, D., Cunningham, K., Dell'Andrea, R. and Riley, M. (2018). Unmanned aerial vehicle inspection of the Placer River Trail Bridge through image-based 3D modelling. Structure and Infrastructure
Engineering, 14(1), 124-136.
Kim, I. H., Jeon, H., Baek, S. C., Hong, W. H. and Jung, H. J. (2018). Application of Crack Identification Techniques for an Aging Concrete Bridge Inspection Using an Unmanned Aerial Vehicle, Sensors, 18(6).
Leea, J. H., Yoon, S. S., Kim, I. H. and Jung, H. J. (2018). Diagnosis of Crack Damage on Structures based on
Image Processing Techniques and R-CNN using Unmanned Aerial Vehicle (UAV), Sensors and Smart
Structures Technologies for Civil, Mechanical, and Aerospace Systems 2018, Vol. 10598.
Lei, B., Wang, N., Xu, P. C. and Song, G. B. (2018). New Crack Detection Method for Bridge Inspection Using
UAV Incorporating Image Processing, Journal of Aerospace Engineering, 31(5).
Mader, D., Blaskow, R., Westfeld, P. and Weller, C. (2016). POTENTIAL OF UAV-BASED LASER SCANNER
AND MULTISPECTRAL CAMERA DATA IN BUILDING INSPECTION, XXIII ISPRS Congress,
Commission I, Vol. 41, 1135-1142.
Reagan, D., Sabato, A. and Niezrecki, C. (2018). Feasibility of using digital image correlation for unmanned aerial
vehicle structural health monitoring of bridges, Structural Health Monitoring-an International Journal,
17(5), 1056-1072.
Road Bureau. (2019a). Ordinance for Enforcement of Road Act. Retrieved from Ministry of Land, Infrastructure
and Transport, Japan website: https://elaws.e-gov.go.jp/search/elawsSearch/elaws_search/lsg0500/detail?lawId=327M50004000025_20180401_430M60000800037&openerCode=1 (in Japanese)
Road Bureau. (2019b). Guideline for Periodic Road Bridge Inspection. Retrieved from Ministry of Land, Infrastructure and Transport, Japan website: http://www.mlit.go.jp/road/sisaku/yobohozen/yobohozen.html (inJapanese)
Salaan, C. J. O., Okada, Y., Mizutani, S., Ishii, T., Koura, K., Ohno, K. and Tadokoro, S. (2018). Close visualbridge inspection using a UAV with a passive rotating spherical shell, Journal of Field Robotics, 35(6),850-867.
Seo, J., Duque, L. and Wacker, J. (2018). Drone-enabled bridge inspection methodology and application, Automation in Construction, 94, 112-126.
Turkan, Y., Hong, J., Laflamme, S. and Puri, N. (2018). Adaptive wavelet neural network for terrestrial laser
scanner-based crack detection, Automation in Construction, 94, 191-202.
U.S. department of transportation. (2004). National Bridge Inspection Standards Regulations. Retrieved from U.S.
department of transportation website: https://www.fhwa.dot.gov/bridge/nbis.cfm
Zhong, X. G., Peng, X., Yan, S. K., Shen, M. Y. and Zhai, Y. Y. (2018). Assessment of the feasibility of detecting
concrete cracks in images acquired by unmanned aerial vehicles, Automation in Construction, 89, 49-57.

112

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

A PERFORMANCE EVALUATION OF FEATURE DETECTORS AND
DESCRIPTORS FOR UNMODIFIED INFRASTRUCTURE SITE DIGITAL IMAGES
Natthapol Saovana1, Nobuyoshi Yabuki2, and Tomohiro Fukuda3
1) Ph.D. Candidate, Division of Sustainable Energy and Environmental Engineering, Graduate of School of Engineering, Osaka
University, Osaka, Japan. Email: u089147k@ecs.osaka-u.ac.jp
2) Ph.D., Prof., Division of Sustainable Energy and Environmental Engineering, Graduate of School of Engineering, Osaka
University, Osaka, Japan. Email: yabuki@see.eng.osaka-u.ac.jp
3) Ph.D., Assoc. Prof., Division of Sustainable Energy and Environmental Engineering, Graduate of School of Engineering,Osaka University, Osaka, Japan. Email: fukuda@see.eng.osaka-u.ac.jp

Abstract: Image registration is a process to match and align a group of images. It is the first step ofphotogrammetric techniques such as Structure from Motion (SfM) to construct a model. It uses feature detector
and descriptor algorithms to detect features inside each image and link these similar feature pairs to solve theunknown variables that is required for further processes. Before using feature detector and descriptor algorithmwith a specific knowledge, a performance evaluation of these algorithm is necessary to be conducted because eachalgorithm has different advantages and disadvantages over specific expertises. Infrastructure project digital images
are unique because the structures have similar shapes and smooth surfaces, which can complicate the image
registration quality. Moreover, the uncontrolled lighting can further decrease the clearness and sharpness of images.
Therefore, it is very challenging to get satisfied output by processing unmodified infrastructure images throughphotogrammetric techniques. Although these algorithms have various types and directly affect the quality of themodel, there are no performance evaluations in the domain of infrastructure images to be seen. This study proposes
the performance evaluation of robust feature detector and descriptor algorithms. The evaluation is separated into
two categories, which are the feature coverage inside the region of interest (ROI) and the performance of feature
matching. The result shows that Oriented FAST and Rotated BRIEF (ORB) can detect the highest amount offeatures with the shortest amount of time. However, Speeded Up Robust Features (SURF) can better detect featuresinside the ROI, which may lead to the better output quality. Finally, SURF128, which is SURF that was extended
to utilize 128 floats, can finish the entire process at the fastest speed. This study can serve as a suggestion whenfeature detector and descriptor algorithms have to be chosen to solve questions inside the infrastructure domain.
Keywords: Image registration; feature detector and descriptor algorithms; feature detection; feature matching;infrastructure digital images
1. INTRODUCTION
In the current practice, construction companies are using digital images due to their cost-effectiveness,accuracy, and easy utilization (Hamledari et al., 2017; Zhang et al., 2009). By using digital images, they can beprocessed through Structure from Motion (SfM) technique to create a model, for example, a building (Musialski
et al., 2013), a field (Carbonneau & Dietrich, 2017), a river bank (Micheletti et al., 2015), and a pile of embankment(Wrózyński et al., 2017). SfM is a simple and inexpensive type of photogrammetry-based techniques that utilizes
an algorithm called the feature detector and descriptor to automatically detect and match numerous suiting
attributes from intersections between images to construct a model (Westoby et al., 2012). Therefore, the quality of
the SfM model greatly relies on the performance of this detection and matching.
There are numerous feature detectors and descriptors, which have their own advantages and disadvantages,
to be implemented. Among these feature detectors and descriptors, Scale Invariant Feature Transform (SIFT)(Lowe, 2004), Speeded Up Robust Features (SURF) (Bay et al., 2008), KAZE (Alcantarilla et al., 2012),
Accelerated-KAZE (AKAZE) (Alcantarilla et al., 2013), Oriented FAST and Rotated BRIEF (ORB) (Rublee etal., 2011), and Binary Robust Invariant Scalable Keypoints (BRISK) (Leutenegger et al., 2012) have their own
feature detectors and descriptors inside their algorithms. They are robust to image distortion such as rotation, affine
transformation, and the zooming in or out between images (Tareen & Saleem, 2018). Moreover, they can be utilize
easily with the built-in libraries inside OpenCV (Bradski, 2000). There were studies that proposed the comparative
performance analysis of these algorithms with public image datasets (Hietanen et al., 2016; Işık & Özkan, 2014;
Tareen & Saleem, 2018). However, the comparative performance analysis is still needed before using thesealgorithms because a specific algorithm that can perform well with some expertises does not guarantee that it can
execute well with other study topics (Rusinol et al., 2015). Some conclusions from previous comparative
performance analyses on unique expertise datasets are, SIFT could perform well with night vision goggles images(Mouats et al., 2018), SIFT, ORB, and BRISK accomplished with document images (Rusinol et al., 2015), andSURF had high performance with images from a drone for emergency landing (Cowan et al., 2017).
Digital images with feature detector and descriptor algorithms in civil engineering have been studied fora while now, for instance, rectangle concrete columns detection (Zhu & Brilakis, 2010), sparse point cloud
generation using features from two calibrated cameras (Fathi & Brilakis, 2011), bricks calculation (Hui et al.,113

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 20192014), absolute-scale point cloud generation of civil structures from a cube (Rashidi et al., 2015), inspection offrame manufacturing from CAD images (Martinez et al., 2019), and crack detection in civil infrastructure using
digital images (Kong & Li, 2019). Anyway, the feature detector and descriptor algorithms have not been focusedas the main topic in the civil engineering scope. Studies often verified the performance of their system, not theperformance of feature detectors and descriptors themselves.
Furthermore, infrastructure construction site digital images are unique because they consist of rocks,
vegetation, and same textureless planar structures from location to location. This situation can complicate thedetection and matching. The uncontrollable lighting of the outdoor environment can also decrease the quality ofimage matching by declining the color clearness and texture sharpness (Henderson & Izquierdo, 2015). To
enlighten on this problem, a sample image dataset was processed through Agisoft Photoscan which is considered
to have high performance for SfM technique (Remondino et al., 2012). Figure 1 shows the result from Agisoft
Photoscan. It can be seen that the columns in the original image lose their caps although there are some detectedfeatures. This is the result of the failed image registration, which the feature detector and descriptor algorithm
cannot find the reliable matchings of the column caps between images. Although the feature detector and descriptor
algorithm inside Agisoft Photoscan is not openly shown to the public (Verhoeven et al., 2015), one of the technicalsupport staff said that they utilized an algorithm that similar to SIFT (Semyonov, 2011). In order to further clarify
this problem, an exploratory testing was conducted by using SIFT as a feature detector and descriptor algorithm
to do the feature matching of this dataset. Figure 2 shows the result of the matching and it can be seen that there
are no matchings between column caps at all. This situation could happen when the algorithm had lower confidence
to match the column caps therefore, it decided to instead match the boulders and vegetation that it had higher
confidence in matching. Consequently, there is a need for a performance comparison between feature detectors
and descriptors to find the suitable algorithms for infrastructure project digital images, which has not been seen.
The objective of this study is to evaluate the performance of feature detector and descriptor algorithms
when they are implemented with the unmodified infrastructure site digital images. Each algorithm is comparedfrom their abilities to detect features inside images and match their similarity. The computational time in eachprocess is also compared to gain the insight of the speed of each algorithms with infrastructure domain.

a) The original imageb) Detected featuresc) The final mesh

Figure 1. Final results of the SfM technique from Agisoft Photoscan

Figure 2. The result of the feature matching by SIFT
2. PROPOSED METHOD
The proposed method is separated into five parts, which are shown in Figure 3. Infrastructure digital
images are utilized as the input of this study. These images are classified based on their original capturing tools,
put into classified dataset, and processed through an image processing software to be manually cropped. These
cropped images are also formed other datasets that have no background to visualize the impact of the background.
Moreover, the rim of these cropped images will be served as the ROI of the testing. Next, the system implementing
feature detector and descriptor algorithms is developed to extract and match features between each image pair. For
the ROI coverage testing, every images from datasets that have the background are processed into the system tomark the feature inside each image. These images are further processed into an image processing software to
calculate the number of pixels that each feature detector and descriptor can detect. After the number of pixel inside
the entire picture is achieved, the ROI is utilized to find the number of pixels inside the ROI and calculate the114Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

coverage that each algorithm can detect. Then, every dataset is processed into the system again to do the matching
between each image pair. In this part, each algorithm detects and highlights the features inside each image.
Afterwards, each feature inside the first image is utilized as the reference to find other similar features inside thesecond image. If the similar feature is found, the system will match these two features by drawing a yellow line
between these two features. The evaluation is separated into two criteria, which are the performance of thematching and the computational time in each step. These criteria are two important factors for the image
registration. The higher number of the good matching shows the better ability to find the similarity between images.
Meanwhile, the computational time shows the rapidness of the system, which will be necessary when the speed of
the image registration is concerned. Finally, the results are compared, discussed, and concluded.

Figure 3. Research procedure
3. Experimentation
Eight digital image pairs from a real infrastructure project in Thailand were captured by a drone and acamera. This case study was chosen because it contained numerous type of smooth surface reinforced concrete
structures and had disordered environment. The drone used in this study was DJI Phantom 4 and the camera was
Olympus EM-10 Mk III. Figure 4 shows the sample images from each image pair. Then, these images weremanually cropped to extract the interested structures and utilized as the ROI for further evaluations. Figure 5 showsthe example of the original image and the cropped image that its rim will be utilized as the boundary of the ROI
coverage evaluation. Consequently, there were four datasets which were images from a camera, images from acamera that have no background, images from a drone, and images from a drone that have no background. Eachdataset consisted of eight images (four image pairs).
Next, both ROI and normal images were processed through a system programed on Spyder with OpenCV
3.4.2 to detect the feature points inside these pictures. The specifications of the testing computer were Intel(R)Core(TM) i5-8400 CPU @ 2.80GHz and 8.00 GB RAM. Feature detector and descriptor algorithms that were
evaluated in this research were SIFT, SURF (64-Floats), SURF128 (Extended SURF that used 128-Floats), KAZE,AKAZE, ORB, and BRISK. The output of the system was black images that the positions of feature points weremarked in red color. These images were processed through Photoshop CS2 to find the number of red pixels andcalculate for the percentage of ROI coverage (red pixels inside the yellow line). Figure 6a) shows the output image
from the system. The yellow line in Figure 6b) was manually marked to show the boundary of the ROI. It used therim of Figure 5b) as the reference.

a) Pictures from a camerab) Pictures from a drone

Figure 4. Sample pictures inside each dataseta) The original imageb) The cropped image

Figure 5. The example of the original image and the cropped structure inside no background dataset

115

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019a) The output image from the systemb) The boundary of the ROI

Figure 6. The output image after processed Figure 5a) through the system and the boundary of the ROI
Next part was the evaluation of the feature matching by each feature detector and descriptor algorithm.
The matching scheme between features was Nearest Neighbor Distance Ratio, which utilized first two nearest
neighbors from a feature set of the first image as the references and then, searched through similar features fromanother picture. The threshold ratio of this matching scheme was set at 0.7 to eliminate the low confidence matches.
For SIFT, SURF, SURF128, and KAZE descriptors matching, Least Absolute Deviations or L1-norm was utilized
for the image registration. Meanwhile, Hamming distance (Robinson, 2008) was used for AKAZE, ORB, and
BRISK descriptors matching.
When the matching was on going, it had high probability to form bad matchings that will affect the qualityof the image registration. Therefore, these bad matchings had to be removed by using RANdom Sample Consensus(RANSAC) algorithm (Fischler & Bolles, 1981). The iteration of RANSAC was set at 2,000 iterations and 99.5%
confidence like Tareen and Saleem (2018). In order to raise the confidence of the time measurement of each featuredetector and descriptor algorithms, the entire matching was executed 100 times for each algorithm and each image
pair to minimize the risk of computational errors.
4. RESULTS
4.1 The comparison of the detection coverageFirstly, sixteen images from camera and drone datasets were processed into the system. Each sample was
evaluated separately between each feature detector and descriptor algorithms. Figure 7a) shows the sample image
from the camera dataset Figure 7b) to h) shows the output from each feature detector and descriptor algorithms.
Red dots were features that each algorithm detected and the yellow lines were manually draw to show the ROI
boundary of the image. Table 1 shows the result of the example in Figure 7. The calculation started by finding thenumber of pixel in the original image. In this case, the original image was 1080 x 1440 pixels, which equaled to1,555,200 pixels. Then, this amount was utilized to divide the total detected features of each algorithm to find thepercentage of the coverage for the entire image. Next, the numbers of red dots inside the ROI were calculated tobe divided by the number of pixels inside ROI, which was 484,526 pixels in this case.

a) Original imageb) SIFTc) SURFd) SURF128e) KAZEf) AKAZEg) ORBh) BRISK

Figure 7. The sample coverage results of the study

From Table 1, it could be seen that ORB can detect the highest amount of feature inside an image.
However, when focusing inside the ROI of the image, SURF could do better with the coverage about 25%of the entire ROI. Furthermore, all of the results from every image pair were used to calculate the average
percentage of the coverage. The final average result of this part is shown in Table 2. The final result went
along with the example in Table 1 and demonstrated that ORB had the highest performance in detecting

116Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

features inside an image. Anyway, SURF had the highest percentage of the features detection inside ROI.

Feature detector
and descriptor
algorithms
SIFT
SURF
SURF128
KAZE
AKAZE
ORB
BRISK

Table 1. The result from the evaluation of Figure 7
Percentage of detected
Total detected
Total detectedfeatures compared to thefeatures inside ROI
features (Pixels)
entire image (%)(Pixels)
136,933
8.805
29,225258,399
16.615125,206
204,238
13.133
91,27473,5934.732
15,102
63,0474.054
15,80778,718340,587
21.900210,614
13.54325,140ROI Coverage
(%)
6.032
25.84118.838
3.117
3.262
16.246
5.189

Table 2. The comparison of the average percentage of detected features and ROI coverage
Feature detector
Average percentage of detected features
Average ROI coverage by
and descriptor algorithms
compared to the entire picture (%)
detected features (%)
7.877
SIFT
11.552
SURF
18.86423.80217.218SURF128
15.130
6.946
KAZE
6.955
AKAZE
6.672
7.034
20.976
ORB
30.882
12.020
BRISK
19.9644.2

The performance evaluation of each feature detector and descriptor algorithm
In this part, all of the image pair, sixteen in total, were processed into the system to find the matching
performance of each feature detector and descriptor algorithm. Figure 8 shows the example results of an image
pair from the camera dataset and Table 3 shows the summary of the performance evaluation between featuredetector and descriptor algorithms. Please be noted that µ stands for Micro (10-6).

a) Features and matchings by SIFTb) Features and matchings by SURFc) Features and matchings by SURF128d) Features and matchings by KAZEe) Features and matchings by AKAZEf) Features and matchings by ORBg) Features and matchings by BRISK
Figure 8. The sample matching results of the study117Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 3. Summary of the performance evaluation between feature detector and descriptor algorithms
Time for features
Time
Detected features
Time
Featuredetectedfor(Points)for
Total
detector
Matched Matched(µSecond)
outliersfeaturestime
andfeaturesinliers
to bematched(µSec.)
descriptor(Pairs)(Pairs)
1st
2nd
1st
2nd
rejected(µSec.)
algorithms
image
image
image
image
(µSec.)SIFT
SURF
SURF128
KAZE
AKAZE
ORB
BRISK

1,522
4,130
2,716
1,167
1,205
7,007
2,142

SIFT
SURF
SURF128
KAZE
AKAZE
ORB
BRISK

10,974
14,626
10,7495,474
4,821
63,30820,619SIFT
SURF
SURF128
KAZE
AKAZE
ORB
BRISK

1,200
3,079
2,108
1,190
1,101
9,192
3,171

SIFT
SURF
SURF128
KAZE
AKAZE
ORB
BRISK

11,196
17,22012,7787,455
6,397
69,771
28,510SIFT
SURF
SURF128
KAZE
AKAZE
ORB
BRISK

6,223
9,764
7,088
3,821
3,381
37,31913,610

Camera dataset with no background190
121 12.431 10.265
7.941
6.652
396
236
312
202
3.457
2.830
103
50 61.432 50.729
77
36 10.936
8.763
308
162
1.525
1.288
104
61
3.390
3.153
Camera dataset
7,876
248
119 12.178 11.691
13,141
480
250 10.787 10.095
9,510
387
214
4.640
4.963
5,110
102
42 54.821 54.658
4,117
76
24
9.813
9.651
114
5.029
47,596499
4.010
15,610
141
64
9.404
7.684
Drone dataset with no background
1,200
108
66
8.057
8.011
2,988
290
5.245
5.246
178
2,054
215
144
2.488
2.341
1,144
136
63 36.585 36.155
1,063
77
42
6.854
6.587
172
8,880
318
1.191
1.160
3,036
130
83
3.116
3.042
Drone dataset
11,879
418
146
9.790
9.912
16,884529
202 10.122 10.043
12,487
413
159
4.701
4.615
7,575
457
190 39.119 38.208
6,626
186
67
7.632
7.450
5.344
5.254
70,982805
222
27,932459
202 11.249 10.870
Average of all image pairs from all dataset
1,470
3,839
2,551
1,306
1,244
7,115
2,221

5,606
9,213
6,651
3,784
3,263
33,643
12,200

241
424
332
199
104
483
208

113
216
180
86
42
167
103

10.6148.524
3.902
47.989
8.809
3.272
6.790

9.970
8.009
3.607
44.937
8.113
2.928
6.187

0.971
2.262
1.034
0.428
0.389
4.341
0.928

0.112
0.106
0.102
0.109
0.088
0.100
0.095

23.77916.961
7.423
112.697
20.1767.254
7.566

23.40623.04612.4503.975
2.273
144.843
31.587

0.084
0.098
0.093
0.088
0.076
0.089
0.078

47.35944.02522.147113.542
21.813153.971
48.753

0.766
1.680
0.861
0.425
0.285
7.811
2.098

0.070
0.075
0.073
0.075
0.061
0.076
0.068

16.90412.2455.763
73.24013.786
10.2398.325

26.51529.24717.482
5.873
3.289
187.449
61.542

0.076
0.083
0.076
0.083
0.061
0.081
0.080

46.29349.49526.87583.28418.432
198.12883.74012.91414.059
7.957
2.675
1.559
86.11124.039

0.086
0.090
0.086
0.089
0.071
0.087
0.080

33.58430.68215.55295.69118.55292.39837.096

The result showed that ORB could detect the highest number of features in every case. The number ofmatched features in the datasets that contained fewer details such as no background datasets were very competitive
between ORB and SURF algorithms, which in average, ORB could perform slightly better than SURF. However,the number of matching from ORB algorithm decreased drastically after RANSAC removed the outlier matching.
SURF had the highest amount of inliers instead of ORB. For the time spent in each step, although ORB could
detect the highest amount of features in each image, ORB still spent the least period of time in cases that had fewer

118

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019detail such as the datasets that had no background. For the datasets that had background, SURF128 could perform
faster. Furthermore, AKAZE spent the least amount of time in average for features matching and outliers rejecting.
Finally, SURF128 utilized the least amount of time in average for the entire process.
5. DISCUSSION
In general, ORB was a very promising algorithm for detecting features inside infrastructure images. It
could detect the biggest amount of features and still utilized the shortest period of time in order to detect them incases that had fewer detail like no background datasets. BRISK and SURF were other two algorithms that could
detect desirable amount of features inside images. ORB had great performance in feature matching but SURF also
had high performance, especially in the dataset that had fewer detail like the camera dataset that had no background.
Moreover, although ORB had considerably high amount of matching, more than half of these matchings were
rejected by RANSAC as poor matchings. The huge amount of time that ORB used to do the matching reflected on
this aspect also because the higher amount of detected features meant the higher possible matchings. However, in
this case, these matchings could be wrong and waste the time instead. These matchings also did not cover the ROIof the images as expected because it could detect features up to about 30% of the image but could detect only
about 20% of the ROI. SURF had the highest amount of ROI coverage which might result in the better quality
model of ROI. Although it could detect around 18% of the image, it could detect the ROI up to about 24%.
AKAZE could match features rapidly. It spent the smallest amount of time finishing the matching and theoutliers rejecting. Besides, although the speed was very fast comparing to other algorithms, its matching number
was very poor and had the smallest amount of final matchings. Finally, SURF128 was the overall fastest algorithm
when it had to detect and match features inside unmodified images of infrastructures. The ROI coverage wasconsiderably satisfactory with the third rank behind SURF and ORB.
6. CONCLUSION
This study presents a performance evaluation between seven feature detector and descriptor algorithms,
which are SIFT, SURF, SURF128, KAZE, AKAZE, ORB, and BRISK. They have both a detector and a descriptor
available in their algorithms. They are also robust to the rotation, the affine transformation, and the zooming ofimages. The evaluation used unmodified infrastructure digital images as testing datasets. This study can serve asa suggestion for choosing the feature detector and descriptor algorithms with the infrastructure domain. The results
showed that ORB had the highest performance in detecting features. It also spent the least amount of time infinding these features. Anyway, if the image had gigantic amount of features, it would slow down this process andlose to SURF128. SURF had a competitive amount of detected features but the dispersion of detected featuresinside the ROI was better than ORB and had the most promising result in ROI coverage testing. AKAZE was thefastest algorithm to finish the matching and outliers rejecting. However, AKAZE performed poorly when it cameto the quantity of feature matching. Interestingly, SURF128 spent the least amount of time for the overall process.
Moreover, it could also detect the features inside the ROI very well and got the third rank in the ROI coverage
testing.
The limitation of this study was the number of the sample, which was small (sixteen image pairs, thirtytwo images in total). Therefore, more images should be implemented to solidify the result.
REFERENCES
Alcantarilla, P. F., Bartoli, A., & Davison, A. J. (2012). KAZE Features. ECCV 2012, Part VI, 214–227.
Alcantarilla, P. f., Nuevo, J., & Bartoli, A. (2013). Fast Explicit Diffusion for Accelerated Features in Nonlinear
Scale Spaces. Procedings of the British Machine Vision Conference 2013, 13.1-13.11.
Bay, H., Ess, A., Tuytelaars, T., & Van Gool, L. (2008). Speeded-Up Robust Features (SURF). Computer Vision
and Image Understanding, 110(3), 346–359.
Bradski, G. (2000). The OpenCV Library. Dr. Dobb’s Journal of Software Tools.
Carbonneau, P. E., & Dietrich, J. T. (2017). Cost-effective non-metric photogrammetry from consumer-gradesUAS: implications for direct georeferencing of structure from motion photogrammetry. Earth Surface
Processes and Landforms, 42(3), 473–486.
Cowan, B., Imanberdiyev, N., Fu, C., Dong, Y., & Kayacan, E. (2017). A performance evaluation of detectors anddescriptors for UAV visual tracking. 2016 14th International Conference on Control, Automation,
Robotics and Vision, ICARCV 2016.
Fathi, H., & Brilakis, I. (2011). Automated sparse 3D point cloud generation of infrastructure using its distinctive
visual features. Advanced Engineering Informatics, 25(4), 760–770.
Fischler, M. a, & Bolles, R. C. (1981). Random Sample Consensus: A Paradigm for Model Fitting with
Apphcatlons to Image Analysis and Automated Cartography. Communications of the ACM, 24(6), 381–
395.
Hamledari, H., McCabe, B., Davari, S., & Shahi, A. (2017). Automated Schedule and Progress Updating of IFC-

119

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Based 4D BIMs. Journal of Computing in Civil Engineering, 31(4).
Henderson, C., & Izquierdo, E. (2015). Robust feature matching in the wild. Proceedings of the 2015 Science andInformation Conference, SAI 2015, 628–637.
Hietanen, A., Lankinen, J., Kämäräinen, J. K., Buch, A. G., & Krüger, N. (2016). A comparison of feature detectors
and descriptors for object class matching. Neurocomputing, 184, 3–12.
Hui, L., Park, M., & Brilakis, I. (2014). Automated In-Place Brick Counting for Facade Construction Progress
Estimation. Computing in Civil and Building Engineering, 29(6), 958–965.
Işık, Ş., & Özkan, K. (2014). A Comparative Evaluation of Well-known Feature Detectors and Descriptors.
International Journal of Applied Mathematics, Electronics and Computers, 3(1), 1.
Kong, X., & Li, J. (2019). Non-contact fatigue crack detection in civil infrastructure through image overlappingand crack breathing sensing. Automation in Construction, 99(December 2018), 125–139.
Leutenegger, S., Chli, M., & Siegwart, R. Y. (2012). BRISK: Binary Robust Invariant Scalable Keypoints Stefan.
In 2011 International Conference on Computer Vision. Barcelona, Spain.
Lowe, D. G. (2004). Distinctive Image Features from Scale-Invariant Keypoints. International Journal ofComputer Vision, 60(2), 91–110.
Martinez, P., Ahmad, R., & Al-Hussein, M. (2019). A vision-based system for pre-inspection of steel frame
manufacturing. Automation in Construction, 97(November 2018), 151–163.
Micheletti, N., Chandler, J. H., & Lane, S. N. (2015). Investigating the geomorphological potential of freely
available and accessible structure-from-motion photogrammetry using a smartphone. Earth Surface
Processes and Landforms, 40(4), 473–486.
Mouats, T., Aouf, N., Nam, D., & Vidas, S. (2018). Performance Evaluation of Feature Detectors and Descriptors
Beyond the Visible. Journal of Intelligent and Robotic Systems: Theory and Applications (Vol. 92). Journalof Intelligent & Robotic Systems.
Musialski, P., Wonka, P., Aliaga, D. G., Wimmer, M., Van Gool, L., & Purgathofer, W. (2013). A survey of urban
reconstruction. Computer Graphics Forum, 32(6), 146–177.
Rashidi, A., Brilakis, I., & Vela, P. (2015). Generating Absolute-Scale Point Cloud Data of Built Infrastructure
Scenes Using a Monocular Camera Setting. Journal of Computing in Civil Engineering, 29(9).
Remondino, F., Del Pizzo, S., Kersten, T. P., & Troisi, S. (2012). Low-Cost and Open-Source Solutions for
Automated Image Orientation – A Critical Overview. 4th International Conference on Cultural Heritage,
EuroMed 2012, Limassol, Cyprus, 7616, 40–54.
Robinson, D. J. S. (2008). An Introduction to Abstract Algebra. Berlin, Boston: De Gruyter.
Rublee, E., Rabaud, V., & Konolige, K. (2011). ORB : an efficient alternative to SIFT or SURF. Intl. Conf.
Computer Vision, 1–5.
Rusinol, M., Chazalon, J., Ogier, J. M., & Llados, J. (2015). A comparative study of local detectors and descriptors
for mobile document classification. Proceedings of the International Conference on Document Analysis
and Recognition, ICDAR, 2015–Novem, 596–600.
Semyonov, D. (2011). Algorithms used in Photoscan. Retrieved March 21, 2019, fromhttps://www.agisoft.com/forum/index.php?topic=89.0
Tareen, S. A. K., & Saleem, Z. (2018). A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK.
2018 International Conference on Computing, Mathematics and Engineering Technologies: Invent,Innovate and Integrate for Socioeconomic Development, ICoMET 2018 - Proceedings, 2018–Janua, 1–
10.
Verhoeven, G., Karel, W., Štuhec, S., Doneus, M., Trinks, I., & Pfeifer, N. (2015). Mind your grey tonesExamining the influence of decolourization methods on interest point extraction and matching forarchitectural image-based modelling. International Archives of the Photogrammetry, Remote Sensing andSpatial Information Sciences - ISPRS Archives, 40(5W4), 307–314.
Westoby, M. J., Brasington, J., Glasser, N. F., Hambrey, M. J., & Reynolds, J. M. (2012). “Structure-from-Motion”
photogrammetry: A low-cost, effective tool for geoscience applications. Geomorphology, 179(June 2018),
300–314.
Wrózyński, R., Pyszny, K., Sojka, M., Przybyła, C., & Murat-Błazejewska, S. (2017). Ground volume assessment
using “Structure from Motion” photogrammetry with a smartphone and a compact camera. Open
Geosciences, 9(1), 281–294.
Zhang, X., Bakis, N., Lukins, T. C., Ibrahim, Y. M., Wu, S., Kagioglou, M., … Trucco, E. (2009). Automating
progress measurement of construction projects. Automation in Construction, 18(3), 294–301.
Zhu, Z., & Brilakis, I. (2010). Concrete Column Recognition in Images and Videos. Journal of Computing in Civil
Engineering, 24(6), 478–487.

120

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Visualization and XR (VR/AR/MR)

121

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

EFFICIENT ACCESS TO INSPECTION DATA BASED ON AUGMENTEDREALITY USING A “BRIDGE - CARD”
Hisao Emoto1, Hiroki Komuro2, Takehiko Midorikawa3, Hideaki Nakamura4, Kei Kawamura51) Dr. Eng., Assoc. Prof., Department of Civil and Environmental Engineering, KOSEN (National Institute of Technology),
Fukushima, Japan. Email: emoto@fukushima-nct.ac.jp
2) Student at Department of Civil and Environmental Engineering, KOSEN (National Institute of Technology), Fukushima,
Japan. Email: komuro-h@city.iwaki.lg.jp (Present affiliation is Iwaki City Office in Fukushima)3) Dr. Eng., Prof., Department of Civil and Environmental Engineering, KOSEN (National Institute of Technology), Fukushima,
Japan. Email: midorikawa@fukushima-nct.ac.jp
4) Dr. Eng., Prof., Division of Electrical, Electronic and Information Engineering, Graduate School of Science and Technology
for Innovation, Yamaguchi University, Yamaguchi, Japan. Email: nakahide@yamaguchi-u.ac.jp
5) Dr. Eng., Assoc. Prof., Division of Electrical, Electronic and Information Engineering, Graduate School of Science andTechnology for Innovation, Yamaguchi University, Yamaguchi, Japan. Email: kay@yamaguchi-u.ac.jp

Abstract: In Japan, the time has come to rapidly rebuild bridges and other civil infrastructure during this periodof economic growth to improve the service life of bridges and to develop long-standing government policy
measures to maintain bridges. Normally, bridge inspections are performed by a close visual inspection on site.
However, the number of professionals have decreased. For this reason, it is important to efficiently accessnecessary data on site to save labor. To aid in this, the AR (Augmented Reality) technique has recently beendeveloped.
In order to identify the bridge, a method of accessing data using “marker type” AR technology is
employed. As in previous studies, a marker directly pasted on a bridge near a noted condition, such as a crack etc.,
has been proposed. This is useful for accessing the bridge’s condition data. But the data is not suitable for a bird'seye viewing of the whole bridge. Instead, it is easy to access whole bridge data using a “Bridge-Card”, functioning
like a business card, on which bridge specifications are documented. In this way, the Bridge Card is a marker.
This study aims to efficiently save labor for visual inspections using “bridge-cards”, which are the sizeof a business card and developed with a bridge inspection support system for smart phones. This system delivers
specification data, inspection data, repair and reinforcement data, among other information used by AR when the“bridge-cards” is read by the smart phone. This paper discusses the suitability of the use of a marker, whileconsidering the reality that it is useful to access data off-site, in addition to location-based AR, which requires abridge site visit.
Keywords: bridge maintenance support system, visual inspection, bridge-card, augmented reality
1. INTRODUCTION
Recently in Japan, civil infrastructure renovation and construction during this high economic growth
period is needed to maintain and extend service life (Abe et al., 2006; Miyagawa et al., 2008). This paper targets
the civil infrastructure of bridges. The bridge inspection method is a visual inspection. The Ministry of Land,
Infrastructure, Transport and Tourism revised the ministerial ordinance on March 31, 2014 to establish specificstandards for road maintenance based on the provisions of Article 35-2, paragraph 2 of the Road Law Enforcement
Order. It requires a visual inspection check once every five years. Without engineering knowledge of the structure,material and environment of a bridge, however, a visual inspection is not sufficient. Unfortunately, the number ofengineering inspectors is insufficient to perform the necessary visual inspections, with the retiring of the babyboom generation, decreasing birthrate, and aging population. The Ministry of Land, Infrastructure, Transport andTourism uses ICT as “i-Construction”, in order to efficiently solve this problem (Tateyama, 2016). Furthermore,
the cabinet office tried to use SIP (Cross-ministerial Strategic Innovation Promotion Program) in order tocollaborate with other ministries for civil infrastructure management. We focus on Virtual Reality in the ICT field,
developing an AR (Augmented Reality) technique based on image detection or identification of an area using
advanced AI (Artificial Intelligence) techniques (Tachi et al., 2011). This AR technique is used by the famous
smart phone “Pockemon GO”, an application shown in “Pocketmonster” on smart phone cameras.
Basic specification, historical inspection, historical repair and strength data are necessary for the visualinspection. Basic specification data consists of the bridge name, location, year place in service, type of bridge andso on. The historical inspection data confirms existing cracks or spalling and any progress made in these conditions.
In addition, if the historical repair or strength data exists, the inspector needs to check it. Before 20 to 30 years
ago, these data existed only on paper. In the past ten years, these data have become electronic, and more electronic
data is expected in the future. These electronic data are printed out for the target bridge when inspectors make asite visit. The printing of data in preparation of a visual inspection is inefficient. Although smart phones or tablet
PCs are popular, they are almost never used in civil infrastructure maintenance. Hardware problems in the field
include exposure to water and dust, while the software issues include not having a standard electronic data format.

122Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

First, hardware problems can be resolved with more costly equipment, while dustproofing and rainproofing will
improve advanced technology. We investigate the question of the standard electronic data format. In order to solve
this question, we use the AR technique (Tachi et al., 2011; Kobayashi, 2010; Nikkei Communication edited, 2009;
I/O edited, 2016 ).
Upon accessing bridge inspection data using the AR technique, the system needs to identify where bridge
it is. There are several methods for accessing inspection data using AR. The primary way is through location-based
AR. Bridge location is identified by GPS-measured longitudes and latitude, depending on the positioning accuracy
and GPS capability. Another method is vision-based AR, with which it has recently become possible to identify abridge using image recognition technology and space recognition technology. Vision-based AR has both marker
and markerless types. The marker type is often realized by QR code, among other methods. However, any sort ofmarker is sufficient for specifying the bridge, and this can be realized with a business-card sized card, a symbol or
the like. On the other hand, the markerless type used by space recognition techniques needs high-spec CPU
performance because of the amount of calculations it requires. In the future, with advancing hardware, this will
become popular.
In this study, we discuss data access using markers for bridge identification. Previous research proposed
directly accessing data about bridge damage conditions with QR code, because QR code paste on near point ofeach damaged condition. Thus, each damaged condition datum would be accessed, but data about the whole bridge
damage condition could not be accessed.
In order to easily access whole bridge data, such as bridge damaged condition or specifications, businesscard-sized bridge-cards were created for use as a marker. This study is concerned with the appropriateness ofbridge-cards. These are more useful than location-based AR, which require a bridge site visit.
The purpose of this study is to develop an efficient bridge inspection support system by using the bridgecard, which is AR technique user-friendly, and to propose how to efficiently inspect bridges on site.
2. OUTLINE OF A BRIDGE INSPECTION SUPPORT SYSTEM USING A “BRIDGE-CARD”
2.1 How to apply the AR system to the bridge inspection support system
When a bridge inspector checks a bridge by visual inspection, they require basic specifications andhistorical inspection information. Figure 1 shows each user role in the bridge management DB. How to accessthese data from the management DB depending on user, place or purpose differs. Here, we assumed the following
users: a bridge manager (Administrator), a system developer (Developer), and an inspection engineer (Bridge
Inspectors). As shown in Figure 1, the bridge manager (Administrator) checks overall data via an office PC todevelop a maintenance plan. For this purpose, a GIS system is efficient. The system developer (Developer) updates
data from the DB via a PC in an office or datacenter and directly accesses data from the DB through the server
engine client tools. The inspector engineer (Bridge Inspectors) accesses bridge specifications, like a bridge name,Figure 1. Role of users in bridge inspection support system by using AR

123

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019location, and bridge type, on the bridge site and further refers to historical inspection results during the visualinspection to check whether the number of damage conditions have increased or whether any damage conditions
have progressed. In this situation, it is important to easily access these data from the DB by using a smart phoneor tablet PC. In this study, we suggest data access via a “Bridge-Card” on an AR system. In general, there aremethods to directly access data from a URL or QR-code. But a “Bridge-Card” is easily printed out like a businesscard and allows the user to quickly recognize bridge name, type of bridge and a bridge form from a picture
overview.
2.2

About AR technique
AR (Augmented Reality ) is one representation of “Reality”. Other methods include VR (Virtual Reality)
and others. AR exists between the real environment and the virtual environment by creating a computervisualization, as shown in Figure 2 (Paul, Kishino ,1994; Ronald, 1997 ). MR (Mixed Reality) consist of the realand virtual environment. AR techniques make virtual images and sounds in the real environment. The conditionsof AR are “1.The fusion of the real environment and VR environment by computer ”, “2. Interaction with the realenvironment” and “3. Three Dimensions” (Kobayashi, 2010; Ronald, 1997).
A type of fusion VR environment is shown in Table 1. This kind is roughly divided into location-based
AR and vision-based AR. Location-based AR identifies a place by GPS. In this, it is important to measure location
with a GPS. On the other hand, vision-based AR identifies a place by image recognition from a picture on a smart
phone. This is achieved by marker and markerless types of AR. With a marker type, a place is identified by amarker; whereas, with a markerless type, place is identified by image recognition. Table 1. show the features ofeach AR type.
The procedure for using marker-type AR follows the steps in Figure 3: Step 1 “Capture image fromcamera”, Step 2 “Recognize marker and pattern matching”, Step 3 “Measure 3D-location and direction”, and Step
4 “Mix CG(VR) and image from camera”.
2.3

Bridge management and utility AR
Utility AR used for bridge management deals with how to access the DB. Table 2 shows the original
method. A “bridge location” target uses latitude and longitude with GPS measurement. A “Whole bridge” targetand “Bridge name board” is needed for high-performance smart phone or tablet PC location identification. In
general, camera movies consist of 30 frames per 1 second. This means that the processing time is 33 mill second
per 1 frame. Thus, a short processing time is clearly needed. A “Bridge-card” target needs a card, but as a “Bridgecard” is a marker, it is possible to access data in practice. In addition, as a “Bridge-card” is a card, it is easy toknow bridge data on site as printed in basic specifications (bridge name, length of bridge, location and type ofbridge).

Mixed Reality
Real
Environment

Augmented
Reality

Augmented
Virtuality

VirtualEnvironment

Figure 2. Outline of AR
Table 1. Type of AR and features
Type of AR

Advantage

Disadvantage

Location data as latitude and longitude;
Depending on the GPS system, position errors
direction and tilt data can be obtained on
may occur.
any platform.

Location-based AR

Marker type

It is necessary to put a marker on the bridge
It is possible to display additional data due site in the real world. If it is difficult to put a
to marker accuracy.
marker on a physical place in the environment,
then it can't be used.

Vision-based AR
Calculations are necessary to identify the
It is possible to directly show thespace and shape of things like a bridge. This
Markerless type additional data without any physical placerequires a high-performance client with large
marker by considering the landscape.
computational complexity.

124

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Step1.
Captureimage oncameraStep3.
Measure 3-D
location anddirection

Step2.
Recognize
marker

Step4.
Mix CG andimage oncameraFigure 3. Procedure of marker-type AR
Table 2 . Target of identification and AR type for bridge management
TargetAR TypeFeatureBridge location(Log. Lat.)

Location-based AR

Requires measuring location with GPS
and accessing DB on site.

Whole bridge

Vision-based AR, Markerless type

Needs a high-performance client torecognize an area, and it can access DB
on site.

Bridge name board

Vision-based AR, Markerless type

Needs a high-performance client torecognize an area, and it can access DB
on site.

Bridge-Card

Vision-based AR, Marker type

It can access DB anywhere (bridge site,
office). It creates "Bridge-Cards", and it
needs "Bridge-Cards"

From the above, in this study we apply a marker type AR (“Bridge-card”) to confirm basic specifications
on the bridge site with a smartphone or tablet PC.
2.4

System goals and environmental development
The objective of this system study is to make visual inspection work more efficient for bridge management.
The main user target is the bridge inspector. We also look at how a bridge inspector can easily use this system onsite once training tools are provided. This system uses a method of visualizing the data with a bridge-card as an
AR marker and with the camera function of a smartphone, as shown in Figure 4. For this method, it is necessary
to make a “Bridge-card” and record it as an AR marker in advance.
Development environments are Unity, Vuforia, and Android Studio. Unity is IDE (Integrated
Development Environment) for Games. As Unity is multiplatform, it is possible to apply any platform that has aclient PC and a smart phone (android or iOS) under the same programing code and content. Vuforia is a libraryfor making AR suggestions from Qualcomm Inc. It is known to have a high potential for recognition accuracy,
and in addition, it is easy to utilize for planar marker recognition, 3-D marker recognition, cloud recognition, andtracking recognition when the marker shows the camera. Android Studio is the official IDE for Android application
development based on IntelliJ IDEA. IntelliJ IDEA is a powerful code editor and development tool used to
improve efficiency and productivity of android apps.

Figure 4. Outline of this system

125Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The smart phone for application verification is the Nexus 5 (Google) android version 6.01 and Mate 10
Pro(Huawei) android version 8.00.
3. DEVELOPING THE SYSTEM AND RESULTS
3.1 System Procedure
Figure 5. shows procedures for this system. This system needs basic specification data and historical datafrom the target bridge. Next, a bridge-card is created based on pictures from the whole bridge. Here, as show in 1
to 3 of Figure 5, the “Bridge-Card”, as an AR marker, is registered in the Vuforia system. The Vuforia system has
an accuracy value to identify an image as an AR marker, which needs to exceed a certain threshold. After
registering the image in Vuforia, it is possible to carry out an AR system using Unity. In Unity, a bridge modelshowing historical data for bridge management is created. Finally, an application for a smart phone is compiled
with “Android Studio” and “Unity”.
3.2

Making “Bridge-Cards”
A “Bridge-Card” image is made using a general image editor application. The information for the “BridgeCard” is bridge name, a panoramic view of the whole bridge, the bottom of the girder, and the bridge surface.
Figure 6. shows a “Bridge-Card”, and Figure 7 shows a “HYBRIDGE-CARD” made by the Japan Bridge
Association Inc. for public promotion about the attraction of steel bridges to everyone from elementary school
students to bridge enthusiasts. The “HYBRIDGE-CARD” is effective for use as a “Bridge-Card” in this study.

Start
Summary of DB

Collecting target bridge data
Making bridge-card
1.

Inputting data in Vuforia

2.

Implementing AR on Unity

Improve

3． Linking Android Studio and Unity
Making an application
Confirming an application andextracting problems
End

Figure 5. System Procedures

Figure 6. Example of a Bridge-Card

Figure 7. Example of a HYBRIDGE CARD

126

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Therefore, the “HYBRIDGE-CARD” was registered as the “Bridge-Card” in Vuforia. In the following section,
we discuss the “Bridge-Card” recognition rate.
3.3

Results of the bridge-card identification rate
The Vuforia system has a “Rating” value, which means it uses recognition rate (“Rating” value) as a
threshold level. This “Rating” is expressed in five levels for an identified image. When the “Rating” value is “0”,recognition rate is low. When the “Rating” value is 5, the recognition rate is high. Normally, when the “Rating”
value is “0”, a smart phone cannot identify the Bridge-Card via the camera, and it is impossible to display the data.
Therefore, “Bridge-Card” investigate how to make image recognition of “Bridge-Cards” more effective. Several
kinds of “Bridge-Card” images trend towards a larger, desired “Rating” value. Figure 8 (a) shows “Rating”(recognition rate) of a “Bridge-Card”. In Figure 8 (a), No. 1 and No. 2 are different bridges. No.1 and No.2 featuredetailed text information, surrounded by a black rectangle. No. 3 does not have these features; although it pictures
the same bridge as in No. 1. The “Rating” of No. 3 is 2. Figures 8 (b) and (c) show the average and standard
deviation of brightness value for No. 1 and No. 2 on the vertical axis and “Rating” (recognition rate) on thehorizontal axis. Here, the brightness value is applied as red, green and blue.
Next, Figure 9 shows the result of the “HYBRIDGE-CARD” made by the Japan Bridge Association Inc.
As an example, the Rating of “HYBRIDGE-CARD” from 2 to 5 is shown in Figure 9 (a). Figures 9 (b) and (c)plot each of five points at Ratings of 3 to 5 and one point at a Rating of 2. This figure shows the average andstandard deviation of brightness (R) value on the vertical axis and “Rating” (recognition rate) on the horizontal
axis. Here, brightness (R) is summarized by just the red value.
The brightness value is expressed as 0 to 255 for each pixel of the three primary colors, red, green, andblue. For example, red is expressed by (255, 0, 0), black is expressed by (0, 0, 0), white is expressed by (255, 255,
255). This means that as the numerical value increases, brightness increases.
4. DISCUSSIONAs “Rating” increases, the average brightness value also increases, as shown in Figure 8 (b) and Figure 9(b). The results show that this trend towards larger Ratings follows a brighter value for the “Bridge-Card”.
We see from Figure 8 (c) and Figure 9 (c) that as “Rating” increases, it trends towards a higher standard

No
No

Bridge-Card
橋梁カード

1

RatingRating

4

Averageof brightness value
輝度値の平均

160
140
120
100
80
60
40
20
0
0

1

2

Low

3

Rating

4

5

6

High

(b) Average of brightness value
2

4

100
90

2

輝度値の標準偏差3Standard deviation ofbrightness value

80
70
60
50
40
30
20
10
0
0

1

Low

(a) Bridge-Card and Rating

2

3

Rating45High(c) Standard deviation of brightness value

Figure 8. Bridge-Card Rating (recognition rate) and statistics of brightness value

127

6

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

HYBRIDGE CARD

RatingHYBRIDGE CARD

Rating5

HYBRIDGE CARD

4

RatingHYBRIDGE CARD

Rating

3

2

180
160
140
120
100
80
60
40
20
0

Standard deviation ofbrightness
value
輝度値の標準偏差Average 輝度値の平均
of brightness value(a) HYBRIDGE CARD and these Rating ( Recognition rate)

0

1

Low

2

3

Rating

4

5

6

100
90
80
70
60
50
40
30
20
10
0
0

1

Low

High

2

3

Rating

4

5

6

High

(c) Standard deviation of brightness value(b) Average of brightness value

Figure 9 . HYBRIDGE CARD Rating (recognition rate) and statistics of brightness value (R)Figure 10. Application in this systemdeviation of the brightness value. This means that a brightness value of 0 to 255 appears uniformly. In other word,
there is a trend towards a larger recognition rate when various colors are used.
In summary, it is able to improve the recognition rate by increasing brightness value. Here, “Bridge-Card”
recognition rate (“Rating”) do not differ once a high recognition rate was designed procedure in the creation of the“Bridge-Card”. In addition, to improve the recognition rate (“Rating”), the “Bridge-Card” was surrounded by black
rectangular line, and string information was added, which resulted in a recognition rate improvement of 2 to 4.
To summarize the above, low recognition rates (“Rating”) can be improved by redesign.

128Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

5. Applied system example
As shown in Figure 10, when a “Bridge-Card” registered in Vuforia system is used via smart phone in an
android application, basic bridge specifications are displayed on the device monitor. Figure 10 illustrates a bridge
inspection model where a bridge inspector on site references data required for visual inspection by AR.
When a bridge inspector makes a visual inspection on site, they draw damage conditions, such as cracks
and spalling, on paper. In these circumstances, it is important to easily access the DB. To improve the accuracy ofthe damage condition drawing, a bridge inspector must confirm historical repairs. At this time, it is useful to accessdetailed data on a smart phone or tablet PC using a “Bridge-Card”, which can fit easily into the inspector’s pocket,
like a business card. In addition, since it can be easily used even in severe field environments, it can use for an
administrator in the office. A bridge inspector can check the data in the office before going to the bridge site,
without needing to print out a drawing.
The advantage of using physical paper cards (Bridge-Bard) is that they efficiently access data, without
taking a photo of the whole bridge when moving downstream or upstream on the bridge site, allowing easy
confirmation of the data in the office. Furthermore, the readability of the information on physical paper cards(Bridge-Cards) makes the data easily understood.
6. CONCLUSIONSIn this study, we described how to easily access the damage condition drawing data, historical repair
method data, and basic specifications for bridge inspection. This method uses the AR technique, based on a
“Bridge-Card”, which is like a business card. In this “Bridge - Card”, there is a whole photo of the bridge, alongwith bridge name, construction year and so on. The developed application reads a “Bridge-Card” on a smart phone
camera. If the application identifies a bridge in this system, it displayed the bridge data for inspection use.
The following results were obtained through this study:1) With the aim of efficient bridge inspections, we developed and used a visual inspection support systemthat utilized a “Bridge-Card” and AR.
2) We created the “Bridge-Card”, and we also used the “HYBRIDGE-CARD”. If the recognition rate(“Rating”) was high, Vuforia image recognition rate increased.
3) We suggest that on site bridge inspectors can easily access the data required for visual inspection inthis application using AR.
ACKNOWLEDGMENTS
The author of this study would like to express their heartfelt thanks to the Japan Bridge Association Inc.
for providing the “HYBRIDGE-CARD”.
REFERENCES
Abe, Masato, Abe, Makoto, and Fujino,Y. (2006). Development and characteristics of infrastructure
maintenance of Japan with emphasis on bridges, Journal of JSCE, Ser. F, 63(2), 190-199.
Miyagawa, T., Yasuda, K., Iwaki, I., Yokota, H., Hattori, A. (2008). Asset management for civil engineers –from a concrete structures viewpoint - , Journal of JSCE, Ser. F, 64(1), 24-43.
Tateyama, K. (2016). i-Construction and CIM, Japan Construction Information Center,114,5-8.
Tachi, S., Sato, M., Hirose, M., (2011), Virtual Reality, The Virtual Reality Society of Japan.
Kobayashi, A.,(2010), AR-Kakucho Genjitsu, Mainichi Communications Inc.
Nikkei Communication edited, (2009), AR no subete Keitai to Net wo kaeru Kakucho Genjitsu, Nikkei BP Inc.
I/O edited, (2016), “VR””AR”Gizyutu Gaidobukku, Kougakusya Inc.
Paul, M.,Kishino, F., (1994), A taxonomy mixed reality visual displays, IEICE Transactions on Information
System, E77-D(12).
Ronald, T., Azuma, (1997), A survey of augmented reality, Teleoperators and virtual environments , 6(4), 355385.

129

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Bidirectional linking of 4D-BIM planning with
Virtual and Augmented Reality
Felix Dreischerf1 and Habeb Astour2
1) Ph.D. Candidate, Department of Civil Engineering, University of Applied Science Erfurt, Erfurt, Germany.
Email: felix.dreischerf@fh-erfurt.de2) Dr.-Ing., Prof., Department of Civil Engineering, University of Applied Science Erfurt, Erfurt, Germany.
Email: habeb.astour@fh-erfurt.de

Abstract: Building information modeling (BIM) currently provides a significant benefit to digitization in theconstruction industry in and outside of Germany. New techniques such as Augmented Reality (AR) and Virtual
Reality (VR) can contribute to the spread of the BIM methodology in the construction industry and it willcomplement traditional planning and execution methods. Certainly however, these innovative technologies havethe potential to simplify the entire planning and execution process.
The BIM methodology, VR, and AR are not well-known nor well applied in Germany. BIM is used inbuilding design, but mostly in pilot projects. It needs to be implemented in the daily workflow. By combining itwith new visualization methods, the added value of the BIM methodology will be increased and the process of its
implementation will possibly be accelerated.
Today, the linking of BIM with AR or VR is done only in basic features. BIM combined with computer
aided planning methods and AR or VR enable new approaches and procedures. This leads to a simplification ofplanning and execution. That means that the combination of BIM and VR/AR can be one of the most important
methodologies in the future of building design. However, this combination and its use cases need to be developed
to increase its practicability.
The following paper explains a concept for linking BIM with AR and VR. Therefore, a short overview ofthese elements and the linking methods is given. The focus of the linking is set on the connection between
3D design and construction schedule. The combination is known as 4D BIM. The result is a concept of connecting
4D BIM with AR and VR.
Keywords:

Building Information Modeling, BIM, Virtual Reality, VR, Augmented Reality, AR, 4D, Planning

1. INTRODUCTION
Augmented Reality (AR) and Virtual Reality (VR) devices have become affordable in recent years. For
example, an Oculus Rift S costs about 450 euro in Germany. This allows application in various areas. The usage
and spread in the gaming industry have increased significantly. In May 2019, the number of connected VR headsets
on Steam every month surpassed 1 million devices (Lang, 2019). Large companies like Google, Microsoft, andFacebook push developments in the right direction, in contrast to companies in the construction industry.
A relatively new planning method, Building Information Modeling (BIM), has been established forseveral years in the construction industry. Autodesk, one of the biggest vendors of software for architecture,engineering, construction, etc., has applied BIM since 2002 (Autodesk, 2002). With BIM, the information of abuilding during the planning, execution, and operation is available in all project phases and for all of the project
participants. These participants constantly work with the same data and evaluate it continuously. By combining
this method with a project platform or a common data environment (CDE), continuous data exchange can beomitted. Everyone involved in the planning, execution, and operation has access to a shared work platform.
By combining the new visualization forms with BIM, the construction industry can develop newworkflows, concepts, and use cases. The BIM methodology thus provides very good data and a good informationbase for this new kind of technology. Examples of currently available use cases include a visualization in VR using
existing 3D/BIM models (Enscape, 2019), or a virtual view of a structure from all sides in AR(JBKnowledge Inc., n.d.). In these use cases, the combination is called a unidirectional link and it is made only in
one data transfer direction. Neither a change in AR nor in VR has a direct impact on the 3D/BIM model.
It begs the question: why do the technologies not use a bidirectional link? One approach to solve thisquestion will be pointed out in the following paper. The detailed description of the approach follows after a brief
explanation of BIM, AR, and VR.
2. BUILDING INFORMATION MODELING 4D IN VIRTUAL AND AUGMENTED REALITY
2.1 Building Information Modeling
BIM is a planning method in the construction industry that covers the whole life cycle of a building fromplanning to execution and operation. Consistent and simplified data management and data provision enables
customers, contractors, and operators to benefit from this methodology. One reason is to reduce the overhead,
which is otherwise required for communication. The knowledge that is generated about the building during its
lifecycle is maintained, and not lost due to missing or inadequate communication or interpretation.

130

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

In addition to the X and Y dimensions, a 3D/BIM model has also a Z dimension. This explanation is notmeant to give the impression that BIM is equal to 3D. Furthermore, BIM is not only a 3D model, it is an objectoriented method. An object knows, for example, that it is a reinforced concrete wall and has many of attributes to
describe it better.
There are two basic types of BIM. The first one, Closed BIM, is the use of only one software solution or
platform component. The transfer of data between the software of the stakeholders is mostly uncomplicated. The
problem is only a particular software solutions must be used by all project partners (Hudson, 2017).Open BIM is
the second type of BIM implementation. It provides open and easy data exchange between different software
solutions. For using this BIM type in projects, buildingSMART an international not-for-profit organization, hasdeveloped an exchange format called Industry Foundation Classes (IFC) (buildingSMART international, n.d.).
The problems of IFC is that any building has to be transferred into IFC, and that any software used must be able
to handle this format. A software publisher must implement functionality to correctly export and import the data
in IFC format.
Another very important aspect by using BIM is the exchange of Information. The typical way to deal withthe structure of information exchange in the context of BIM is a CDE. This is a cloud or server solution, which all
stakeholders can access. This obtains many advantages such as an up-to-date and redundancy-free information for
all project partner. Therefore, an information exchange for example Emails in a BIM project should be avoided.
To implement BIM in a construction project are two documents the Employer's Information Requirement(EIR) and the BIM Execution Plan (BEP) essential. The content of EIR includes how the BIM planning must lookand what the use cases and the objectives are (Scottish Futures Trust - EIR, n.d.). The BEP describes systematically
how BIM should be applied for a project (Scottish Futures Trust - BEP, n.d.). For example, which Software,
hardware and processes are needed. In both documents, the country-specific regulations have to be considered.
Another feature is that high planning efforts are partially shifted from later stages to earlier ones. This isbecause of an easier extension of detail depth during planning up until Construction Administration (CA). Figure
1 illustrates the shifting of planning effort to earlier stages. This also has the virtue that a change in planning does
not influence the building cost significantly, and that is why changes can easily be made. It alludes to the fact that
there may be an overhead for the first BIM implementations in a company. The aforementioned positive effects
occur in an optimal process structure.

Figure 1. Efforts phase shift because of BIM (The Construction Users Roundtable, 2004)
In the BIM methodology a construction schedule or building cost can be linked to a 3D model. Thisconnection with the construction schedule is known as 4D. The addition of another dimension, the deposit of thecosts, will make it 5D. The two main advantages of 4D and 5D (nD) are the error avoidance in manual transmission
between the different software solutions, and a consistent process of data storage. Both planning methods are
already in use. There are already possibilities and software solutions, for example, to connect a 3D model fromREVIT with iTWO from RIB.
An additional advantage of nD is that changes of quantity in a 3D model are updated directly to theconstruction schedule and cost plan. One challenge for the implementation of nD is the interoperability. Thatmeans that one has to provide appropriate plugins of the used software to import and export the suitable file format.
The exchange formats occasionally do not include all the information that are present in the original model. In

131Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019other words, if the data set is not properly imported into the software solutions, important information may be lost.
The troubleshooting is hard because the missing data is probably not easy to detect.
2.2

Virtual and Augmented Reality
In addition to the BIM methodology, the approach of using AR and VR for construction planning (as wellas the technology itself) is still quite young. These forms of digital realities belong to the concept of a virtual
continuum, which contains every kind of visualization from the real world to the virtual world.
If the real world is expanded or supplemented by information, it is called AR. Virtual objects in the realworld is the next step on the way to an exclusively virtual world. This step is called the Augmented Virtuality(AV). The mix of AR and AV is Mixed Reality (MR). It also includes all the things in-between these two kinds ofreality. Another step on this virtual continuum is VR, which is an entirely computer-generated reality(Milgram & Kishino, 1994). The collective term for AR, VR, and MR is known as Extended Reality (XR)
(Shaptunova, 2018). Figure 2 graphically explains the aforementioned context.

Figure 2. Virtual Continuum (adapted from Milgram & Kishino)(1) Mixed Reality
MR is a combination of AR und AV. In AR, the real environment is most important, and it is only
supplemented by additional, useful information. This could be, for example, the display of countries’ flags between
the swimming lanes at the Olympic Games. In the case of AV, the focus is set on virtual content. Reality steps into
the background and the virtual objects are positioned in the real world. An example of this is a house that appears
and can be viewed from all sides.
The visualization of MR is currently seen most often with smartphones or Head Mounted Displays(HMDs), such as HoloLens or Daqri. The first AR system was developed in the 1960s, and it allowed the landingof helicopters at night. This operated with infrared cameras in front of the helicopter (Kiyokawa, 2007).
Applications for AR and AV have been increasing in recent years. For example, the filters on Instagram
are AR applications that are no longer perceived as such. They already have the acceptance of society, and nearly
everyone is familiar with them. On the other hand, the HMDs are price intensive and, therefore, still rare.
Distribution into the mass market must take place. HMDs are already more accepted and common for VR thanthey were before.
(2) Virtual Reality
VR is immersion in a computer-generated world or reality. This exists through devices such as computer
screens or HMDs.
Virtual worlds as we interpret today have become popular since the invention of video games. It must be
mentioned that there were other kinds of virtual realities before. The first ones were simple cave paintings. These
were, of course, not virtual. Real objects like mammoths or people in hunting scenes were on cave walls for a longtime. Through looking at these pictures, it was possible to enter another reality in one’s own thoughts.
Due to the various kinds of art, the immersion in another reality has continuously evolved over time. With
increasing skills in diverse arts such as painting, writing, music, theater, and film, it has become more pleasant andeasier to enter a non-existent world. People occasionally use this opportunity to break out of their daily grind andimagine a different, sometimes beautiful, life. Later in 1958, the first interactive computer game Tennis for Two
was developed by the atomic physicist William Higginbotham (Overmars, 2012). This can be seen as a kickoff for
VR. It was the first virtual world in which one could immerse oneself.
VR has taken the next step towards complete immersion and presence because of the invention of HMDs.
Immersion is the feeling that one is in a virtual world. Presence is the state of being completely in a place. As
HMDs become better and more user-friendly, they will be used by everyone. This makes technology easier toaccept by society and integrate into everyday life.

132

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2.3

Linking Planning With Virtual and Augmented Reality
A link of 3D models with construction schedules separately or in simple combinations is already possible
by using the right software. As mentioned above, this can already be done for example with REVIT connected to
iTWO. This procedure is already used in different construction projects in Germany at big companies like forexample Züblin. Further examples from bibliographic and software research perspectives can be found in the
publication of Alcínia Zita Sampaio with the title Enhancing BIM Methodology with VR Technology (Sampaio,2018). In these examples a bidirectional link is not provided.
An additional opportunity in planning and using of 3D, 4D and 5D models is a new and innovative
connection with AR or VR. The basic idea behind the link is to edit the 3D model and the construction schedule
directly in AR or VR. To make this possible, an AR or VR software, hereafter referred to as XR-Software must be
developed. This link may create a great benefit in the early stages of planning.
With VR it will be much easier to make changes, because one has controllers and the whole world isgenerated. A controller can be a keyboard or 3D-Mouse, and the users do not have to change their way of planningin the first step. This includes the 3D model and the schedules. On the other hand, in AR one manipulates thebuilding structures just by using one’s fingers. For example, one can drag and drop a wall or create a new one byusing the interface. Changing the schedules using the fingers or gestures will be much more difficult. The exactly
new methodology must be developed and learned first.
The following explanations only look at a unidirectional link. Data will be sent from the planning softwareto the XR-Software, but not the same way back. The more detailed explanation of the XR software, as well as thebidirectional link, is described in section 2.4.
The output from the planning software is carried out at the beginning via the native file formats. From acertain perspective, the IFC interface is used for the handover of the 3D model data. The data is entered or read
like the output via the native file formats and later over IFC.
The XR-Software can get the data, which is needed in two ways: the manual or automated transfer of datafrom the planning software to the XR-Software. It only takes one transfer to create the link between the planningand the visualization. In the long run of the XR-Software, this time investment is very profitable in comparison to
doing the whole process manually. Moreover, the link can be used for all other construction projects using thesame planning software and can be applied without further adjustments. The two transmission possibilities are
explained in more detail below.
(1) Manual Data Transfer
The simplest version of programming implementation is the manual handover of a 3D model andconstruction schedule to XR-Software. This involves a manual output of the required data from the planningsoftware solutions and manual input into the XR-Software. While performing this step, attention must be paid tothe file formats in order to lose as little information as possible. There is no direct or automated connection between
software products. The only connection is the human who performs these processes.
The advantage of manual data transfer with a well-known process chain is a simpler and faster workaround.
However, if changes occur, this approach leads to problems. In the case of making a change in AR or VR, it mustbe accurately remodeled in the planning software too. This will be followed again by the manual output of the datafrom the planning software and manual input into the XR-Software. This is tedious, time-consuming, and generates
a high level of personnel cost. Automated data transfer is a solution to these problems.

(2) Automatic Data Transfer
To use automatic data transfer, the XR-Software must be customized to use the available Application
Programming Interfaces (APIs) of the used planning software. The data is therefore directly disposable to the XRSoftware. A plugin or app may also need to be programmed for the respective planning software. It includes anautomatic mechanism for handing over the data. For example, during the storage process a dataset can betransferred instantly to the XR-Software. The users only need to save after every change they want seen in AR or
VR. This transfer is done automatically, which is the reason it is called an automated data transfer.
The advantage of this type of transfer is the elimination of manual output and input. This makes the XR
software more user-friendly. It should be mentioned that a disadvantage is that the XR-Software must be active,at least in the background, during the planning process. Otherwise, the data of the planning software has to becached elsewhere. When starting the XR-Software, the cached data has to be imported.
Changes to planning data need to be reimported during visualization within the XR-Software. In this case,only a unidirectional link exists. Such programs are already available on the market. However, it is not possible
for them to make changes directly in AR or VR. An example for this software is Enscape, which offers aunidirectional link. It can have plugins for REVIT, SKETCHUP, RHINO, or ARCHICAD, which enablevisualizations in VR.

133

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2.4

Bidirectionality
Unidirectional linking already exists. But data can now be found for a bidirectional link as well. To havea bidirectional link, the data has to be transferred in both directions. This causes an adjustment in the planningsoftware in the event of changes made in AR or VR. Either the changes will be performed directly or they will
only be displayed. In the case of changes only being displayed, users must confirm the correctness manually one
by one.
The XR-Software in its final form is intended to support the bidirectional link. Therefore, rules forautomated transmission must be defined. Changes will, and should, only be transferred if the conditions are
fulfilled. A simple rule or restriction is the rule to block various elements. With this rule a change is excluded, orin the event of a change, the transfer is forbidden. Without these restrictions, consistent data retention is not inplace and bidirectionality cannot correctly be implemented. Figure 3 shows the condition query in a simplified
graphic representation. A more detailed explanation can be given later, when the programming has begun. Actual
this is just a first thought and not finalized yet.

Figure 3. Bidirectional Dink
One difficulty is the realization of the condition query in the ongoing planning process. This applies bothto the processing in the planning software and within the XR-Software. Probably the easiest way to solve this is
to allow automated data transfer to take place after the project is stored. The rules must determine the conflicts
between the setpoint and actual value. A variant of conflict resolution can be done automatically by comparing
timestamps. The other opportunity can be handled by issuing a protocol or performing step-by-step change
management, after which users can accept or reject the changes. In this variant, after handling all conflicts, themodified data must be automatically transferred to the planning software and the XR-Software. This prevents
having two data inventories. Without this, the positive effect of a bidirectional link would be lost.
It has to be mentioned that there will be two separate XR-Software solutions. One for AR and another one
for VR. Both the layout and the visual presentation of the XR-Software solutions are not exactly defined at thistime. However, controllers or hand tracking will be an important component. Hand tracking is the simplest andmost intuitive way of manipulating data within AR or VR, because people perform many interactions of everyday
life with their hands.
3. RESULTS AND DISCUSSION
This paper explains how bidirectional linking can be done. It describes a first and unfinalized concept tocreate the XR-Software. Programming and implementation are now in progress in department of Building
Information Modeling at Erfurt University of Applied Science in Germany. In this early stage, there might be also
a risk that linking is not possible with this explained method.
The explained concept is a first possible solution to solve the problem of the bidirectional link between
the BIM methodology respectively 3D and 4D models and AR or VR. After the successful implementation of the
4D XR-Software, an extension for 5D planning could also be possible.
4. CONCLUSION
Modern techniques such as AR and VR, in conjunction with BIM, are leading the construction industry
into a new age of digitization. The potential is as great as the evolution from hand drawing to computer-supported
digital drawings.
The widespread use of this new processing method depends on a number of factors. Users must accept the
new technology and the changing processes. The application must be intuitive and easy to learn. Working with the
XR-Software should be user friendly and take less time, and should generally shorten the whole planning process.
Due to the bidirectionality, the planned XR-Software offers the potential to take the next step towards better
planning, execution, and operation in the construction industry.
ACKNOWLEDGMENTS
The author of this paper would like to thank the author of the template for the 4th International Conferenceon Civil and Building Engineering Informatics (ICCBEI 2019).

134

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

REFERNCES
Autodesk, Inc. (2002). Building Information Modeling. Retrieved from website:
http://www.laiserin.com/features/bim/autodesk_bim.pdf.
buildingSMART international (n.d.). Technical Vision. Retrieved from website:
https://www.buildingsmart.org/standards/technical-vision/.
Enscape GmbH. (2019). Real-Time Rendering & Virtual Reality. Retrieved from website:
https://enscape3d.com/.
Hudson, N. (2017). Issue One - Closed BIM Versus openBIM. Retrieved from website:
https://thebimhub.com/magazines/bim-journal/issue/1/pdf/, 4-9.
JBKnowledge, Inc. (n.d.). SmartReality – the ultimate BIM interface. Retrieved from website:
https://smartreality.co/.
Kiyokawa, K. (2007). An Introduction to Head Mounted Displays for Augmented Reality, 1.
Lang, B. (2019). Analysis: Monthly-connected VR Headsets on Steam Pass 1 Million Milestone. Retrieved fromwebsite: https://www.roadtovr.com/monthly-connected-vr-headsets-steam-1-million-milestone/.
Milgram, P., Kishino, F. (1994). A Taxonomy of Mixed Reality Visual Displays, 2-4.
Overmars, M. (2012). A Brief History of Computer Games, 1-2.
Sampaio, A. (2018). Enhancing BIM Methodology with VR Technology. Retrieved from website:
https://www.intechopen.com/books/state-of-the-art-virtual-reality-and-augmented-realityknowhow/enhancing-bim-methodology-with-vr-technology.
Scottish Futures Trust - EIR. (n.d.). Create the Employers Information Requirements - BIM Level 2 Guidance.
Retrieved from website: https://bimportal.scottishfuturestrust.org.uk/level2/stage/2/task/8/create-theemployers-information-requirements.
Scottish Futures Trust - BEP. (n.d.). Supply Chain Response:- The BIM Execution Plan. Retrieved from website:
https://bimportal.scottishfuturestrust.org.uk/level2/stage/3/task/12/supply-chain-response-bimexecution-plan.
Shaptunova, Y. (2018). What Is Extended Reality and What Can We Do with It. Retrieved from website:
https://www.sam-solutions.com/blog/what-is-extended-reality-and-what-can-we-do-with-it/.
The Construction Users Roundtable (CURT) (2004). Collaboration, Integrated Information, and the Project
Lifecycle in Building Design, Construction and Operation. Retrieved from website: https://kcuc.org/wpcontent/uploads/2013/11/Collaboration-Integrated-Information-and-the-Project-Lifecycle.pdf, 4.

135

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Verification about work efficiency improvement by using an Augmented Reality andWearable computer
Satoshi Yamanaka1, Shinya Sugiura2, Ryo Tajima3, and Takafumi Yamanaka4
1) Chief Engineer, Obayashi CO., Ltd., Tokyo, Japan. Email: yamanaka.satoshi@obayashi.co.jp
2) Manager, Obayashi, CO., Ltd., Tokyo, Japan, Email: sugiura.shinya@obayashi.co.jp
3) Deputy Manager, Obayashi CO., Ltd., Tokyo, Japan, Email: tajima.ryo@obayashi.co.jp
4) Dr. Eng., Chief Engineer, Obayashi CO., Ltd., Tokyo, Japan, Email: yamanaka.takafumi@obayashi.co.jp

Abstract: Conventionally, in construction site, workers used to imagine a structure to be constructed based on adesign drawing, and executed construction using an explicit thing such as stake or marking on the site. However,whether they can imagine the finished product accurately depends on the ability of the individual, so work
mistakes and rework may occur. Also, there was a problem that it took a lot of time for preparatory work such as
installation work of the stake.
In the construction industry in Japan, the labor shortage is getting worse because the number of
employees has declined and are aging, but construction investment tends to recover. Even under suchcircumstances, improvement of work efficiency is required for the entire industry to ensure quality. The use ofInformation and Communication Technology (ICT) is mentioned as a means to improve work efficiency, andamong them we focused on Augmented Reality (AR) technology and Wearable computer and worked to solve
the problem.
In this study, we installed the U-shaped drain of berm drainage of 330 meters by using the AR and thewearable computer omitting the installation work of the stake which we have done so far, and the effect by use
was verified. In AR, we superimposed a three-dimensional model created from Computer-Aided Design (CAD)
software using image recognition method, and adopted “Hololens” it's kind of main display type, among
wearable computers which can wear on the head. In addition, automatic tracking type Total Station was used forfine adjustment and position confirmation of the U-shaped drain to be installed. As a result of carrying out theactual construction after the test construction and verifying it, it was possible to shorten the working time byabout 48%, and it was confirmed that using this technology contributes to the improvement of working
efficiency.
Keywords:

ICT, Augmented Reality, Productivity Improvement

1. INTRODUCTION
Conventionally, in the construction site construction workers imagined the completed shape in theirhead while checking the drawing at the worksite and carrying out the work. And by utilizing explicit things suchas stake and marking, everything involved in construction made it possible to construct the same finishedproduct. However, since it depends on individual experience and competence as to whether they can precisely
imagine the finished product in this method, there was also a risk of incorrect work or rework. In addition, it
took a lot of time to install the stake and sometimes resulted in a delay in construction.
In Japan's construction industry, the number of employed people has dropped and the majority of elderly
generations have left the job, and the number of employees have continued to decline, and the aging of
employees are proceeding. On the other hand, construction investment is on a recovery trend compared with2010 which was about half of the peak time, there are situations in which infrastructure improvement has to be
carried out with a small number of people (Ministry of Land, Infrastructure and Transport, 2017). Amid suchcircumstances, it is required to further improve the efficiency of construction while securing the quality of theinfrastructure.
In recent years, utilization of ICT has been regarded as means to contribute to improvement of working
efficiency. Among them, a lot of efforts using Augmented Reality (AR) have been reported, which is atechnology that can be expected to improve work efficiency (Nobuyoshi Yabuki, 2016). For example, Takashi
Tahara (2015) built a system that makes it easy to grasp the plan by superimposing the structure shape
constructed using the AR and the Industrial Television (ITV) camera by the three-dimensional (3D) model.
Tomoya Kaneko et al. (2017) has confirmed that using AR with a tablet device is effective in forming consensus
and helps to prevent redo. And he confirmed that the system can easily share information. Hatori et al. (2013)
developed a system to display safety measures and construction process at the time of construction by AR andconfirmed its effectiveness. In this way, efforts to improve efficiency by using 3D models and AR are reported
for the propose of information sharing and process confirmation. In addition, with the miniaturization ofcomputers, cases of using wearable computers are increasing even in construction work, and AR can also be
used with wearable computers.
We omitted the installation of stake by using 3D model indication and wearable computer of AR for U-shaped136Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

drainage channel of road surface drainage installed on excavated surface of road construction constructed in
Yamada-cho, Iwate Pref. In this paper, we report on the effectiveness of work efficiency improvement confirmedin the construction at that time.
2.

SELECTION OF EQUIPMENT AND SOFTWARE TO USE
We selected of three equipment and software to user as follows: (1) a wearable computer for using AR,(2) an AR software including conversion to data for creating 3D models and browsing with devices, and (3) asurveying instrument to check if the structure is installed in the correct position for the design. Details of each
item are as follows.
2.1

Selection of wearable computer to use AR
A wearable computer is a general term for terminals attached to the body and means terminals attached
to various positions, but the mainstream is a type to be worn on the wrist or the head.
In this paper, we will refer to the device attached to the head. A wearable computer is equipped with adisplay, and it is roughly divided into a main display type which displays large on the front of the user's own
field of view and a sub display type which arranges the sub display around the view field (Horikoshi, 2014). In
this case, AR is used instead of the stake, but the wearer may stumble or fall if the line of sight deviates from thefront as in the case of the sub display. We adopted a main display type in which the displayed objects match fordirection of the line of sight from the viewpoint of use and safety. And, among devices, we decided to compare
the viewing angle size, ease of viewing, function and adopt Microsoft's "Hololens".
2.2

AR software for creating and viewing 3D models
In this construction site, we adopted Construction Information Modeling (CIM) and we have alreadyused 3D model in construction of backhoe Machine Control (MC) system and so we decided to add data to be
used in AR to existing 3D model. The MC system is a system that assists the operation of the operator bypartially auto controlling the operation from bulldozer's blade and backhoe bucket. Sensors such as Global
Navigation Satellite System (GNSS) and inclinometer are attached to the car body, and automatic control is
performed using the output. Since the 3D model is created with LandXml of data format, we add a model of
U-shaped drain to the data, we decided to use "Trend-Core"（Fukui Computer, Fukui, Japan） as software.
There are two major categories of software that uses AR, the methods of superposition model in virtual
space and real space. One is a method (Vision Based AR) that superimposes coordinate points on athree-dimensional model and actual coordinate points by image recognition using a mark called "marker". In
order to use this method at the construction site, it is necessary to derive the position of the marker by surveying.
The second method is to use a sensor (direction, GNSS, distance, etc.) in cooperation with the devices to acquire
information to calculate the positional relationship in the real space, and to match it with the same position(Sensor Based AR) (Yuko Uematsu, 2010). For example, Hiroshi Miki et al. (2018) used GNSS locationacquisition and AR to figure out the location of underground objects using AR. When using this method in thefield, surveying work is not required, but high accuracy is required for the sensor. Hololens can be used in either
way. Considering the trouble of work, it is preferable to use Location Based AR which allows positioning in thefield environment without surveying work at the time of construction.
In this construction, the initial superimposed of the 3D model is important for ensuring construction
accuracy. Since the target precision with the cut is within the range of ± 50 mm, the Location Based AR whichcalculates the position in space by the sensor is influenced by its accuracy. However, it can be said that it is notsuitable as a method because accuracy required for small stage drainage is within ± 30 mm. Therefore, it isnecessary to locate the marker by the surveying instrument, but Vision Based AR which derives the accurate
marker position is adopted. We decided to use “GyroEye Holo” (Infomatics, Kawasaki, Japan) as software.
2.3 Surveying instrument for checking whether the structure to be installed is in the correct position
with match to the design
We confirmed in advance the position error which occurs when installing the structure using AR at
Hololens. It was found that a deviation of 40 mm occurred at a point of at least 20 m. This is because theterminal computes the distance traveled from the origin while updating the spatial information, it is conceivablethat the measurement error of each sensor accumulates. Since an error of 40 mm is not an allowable error in
installing the U-shaped drain, we decided to use a surveying instrument for the final position adjustment. The
surveying instrument adopted an automatic tracking type Total Station (TS) that can measure with an error of 1
mm, confirmed the adjustment amount with a tablet terminal attached to the machine, fine-tuned and set the
U-shaped drain at the design position.

137

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3. ON-SITE VERIFICATION
3.1 Construction procedure
The construction procedure was set as follows: (1) creating a 3D model of U-shaped drain used for bermusing Trend-Core, (2) in order to read the 3D model into Hololens, convert the model with the converter of
GyroEye Holo and import it into the terminal via the cloud server, (3) place the markers used to determine theposition and orientation of the model displayed in the field using the TS, (4) the worker installs the real
U-shaped drain while looking at the model on Hololens, and (5) check the installation position error using TS
and make final adjustment. The image is shown in Fig.1.
Step2 Convert the data file “GyroEye Holo” andimport it into Hololens via cloud server.

Step1 Create 3D model for “Trend-Core”.

Cloud server
Convert and upload

Download and import

Hololens

Step3 Set the marker by TS.

Step4 Install U-shaped drain looking at Step5 Check the position ofdrain by TS.
3D model with AR.

X=-5863.xxx m
Y=95613.xxx m
Z=32.xxx m
Displayed modelZ

markerY
X

markerFigure 1. Construction procedure
3.2

Test construction
In consideration of the safety of the worker who wears the terminal before starting the actual
installation, the test construction was carried out to confirm the display method of the model suitable for thefitting feeling, workability, work environment. First, after installing the stakes, we installed a U-shaped drain.
Next, Hololens was worn by one worker, making a situation without stake and carrying out the installation workwith only the 3D model visible in AR. The construction situation is shown in Fig.2. After finishing the work, weinterviewed the workers and extracted the improvement items for this construction the results obtained from thehearing. The contents are as follows: (1) even though Hololens is wearing it, I can secure visibility and it does
not mind, (2) they can understand the installation line of the drain, however, it isn't clear the height, (3) fineadjustment of a few millimeters is difficult because the position seems to be different depending on the viewing
angle. Combined use of surveying instruments including final confirmation is necessary and (4) displaying theentire 3D model makes it difficult to distinguish between the actual and overlapping parts. It is better to limit thepart to be displayed. For example, corners at the top of the structure are good.
Install stake

Install drain

Check position by using stake

Conventional construction method138Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Check situation

Check model of AR

Install drain

virtualreal

Screen inside the Holones

Installation method using AR and wearable computer

Screen inside the Hololens

Figure 2. Test construction situation
3.3

Main construction

We carried out main construction reflecting the knowledge gained from the test construction. As shownin Fig.3 wears Hololens and constructed it so as to match the line while watching a simplified 3D model as a line
representing the upper end of the U-shaped drain displayed on the device. Then we measured the working timeand compared it with the time required for installation by the conventional construction method. The target work
time is as follows: (1) preparation time to plan the place and interval where stake is installed, (2) meeting time to
the work procedure to set up the planned drain on site and the time to consult with workers concerning loadingof materials, (3) the time of creating 3D model for using AR at the site, (4) installation time for stake for thedrain installation work, including surveying time, and (5) installation time of installing U-shaped drain excluding
during excavation.
Displayed line like drain’s edge
TS

prism

Position check and fine adjustment

Install the drain roughly

Figure 3. Main construction situation
4.

RESULTAs shown in Table 1, it was possible to reduce the working time by about 48% compared with theconventional construction method, as shown in Table 1, when the construction was carried out using AR,
wearable computer and automatic tracking type TS. Although there was no major change in the installation work
itself, we are able to omit the stake and leveling that was done before the work conventional, because we are able
to work while looking at the 3D model. In addition, we reduced the time of the meeting. Since the time to create
a 3D model is an item which does not exist in the conventional construction method, the time has increased, but
overall it is possible to shorten the operation time, and it was possible to confirm the improvement of working
efficiency.
Table 1. Result of Verification139

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

However, we confirmed the items that we need to improve in the future as shown below: (1) the end
point of the structure that they want to confirm at the time of installation can not be confirmed directly by
Hololens. The Hololens specification is because the viewing angle is narrow and you can not see the model
within about 800 mm from themselves. However, it may be possible to improve by emphasizing alignment on
the displayed line instead of the end point, or by lowering the line of sight of the worker and confirming theposition, (2) the position accuracy decreases as the worker moves away from the origin where the model issuperimposed. Therefore, it is necessary to perform the fine adjustment using the TS and the confirmation workof the accuracy, which requires more time and labor, (3) since it is not possible to measure and display thedifference between the position of the AR model and the real structure in real time during the installation work,
so it is necessary to consider methods. For example, arrange the scale model on the 3D model or cooperate withthe surveying instrument then using output results.
5. CONCLUSION
It was confirmed that AR and wearable computer are used for actual construction in this construction,
confirming the effect and improving work efficiency. We used surveying instruments together to compensate forthe deterioration of the narrow viewing angle and positional accuracy of the wearable computer, but in the future
we can further shorten the working time if we can use it only by the wearable computer. For that purpose, wewould like to aim for improvement of work efficiency throughout the construction process by working withcross-cutting research, development and execution by collaborating with not only our company but also terminaland software vendors to solve problems.
REFERENCES
Ministry of Land, Infrastructure and Transport. (2017). White paper on Land, Infrastructure, Transport andtourism in Japan 2017: http://www.mlit.go.jp/common/001216008.pdf, pp.2-4.
Nobuyoshi Yabuki, (2016): Research and Applications of VR/AR in Civil and Construction Engineering,
Journal of the Society of Instrument and Control Engineers, Vol.55, No.6, pp.483-488.
Tomoya Kaneko, Nakabayashi Takumi, Suzuki Masashi, Masakazu Kobayashi, Hideyuki Horiuchi, (2017),
FutureShot® - an Augmented Reality Tool Using a Tablet Device, REPORT OF OBAYASHICORPORATION TECNICAL RESEARCH INSTITUTE, No.81
Takashi Tahara, Nobuyoshi Yabuki. (2015): Investigation of utilization of AR system using 3D model for
construction work plan, Journal of Japan Society of Civil Engineers F3, Vol71, No.2, pp.I_123-I_133.
Fumio Hatori, Nobuyoshi Yabuki, Emi Komori, and Tomohiro Fukuda(2013)：Application of the Augmented
Reality Technique using multiple markers to construction site, Journal of Japan Society of Civil
Engineers, Ser.F3(Civil Engineering Informatics), Vol69, No.2, pp.I_24-I_33.
Tsutomu Horikoshi. (2014): Wearable devices: current features and future perspective, Memoirs of Shonan
Institute of Technology, Vol.49, No.1, pp.65-73.
Yuko Uematsu(2010): Augmented Reality : Foundation 2 : Geometrical Registration Technique, Magazione ofInformation Proccecing, Vol21, No4, pp.373-378.
Hiroshi Miki, Osamu Okamoto, Kuniharu Nishihara, (2018): AR technology using underground GNSS
“Underground buried object visualization system”, Foundation Engineering & Equipment, Monthly,
Vol.46, No.9, pp.78-80.

140Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

VISUALIZATION OF CONSTRUCTION PROCESS BY CONSTRUCTION SITESENSING AND DIGITAL TWINS
Atsushi Takao1, Nobuyoshi Yabuki2, Kohei Seto3, and Iwao Miyata 41) Okumura Corporation, Tokyo, Japan. Email: atsushi.takao@okumuragumi.jp
2) Division of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka University, Osaka,
Japan. Email: yabuki@see.eng.osaka-u.ac.jp
3) Okumura Corporation, Tokyo, Japan. Email: kohei.seto@okumuragumi.jp
4) Okumura Corporation, Tokyo, Japan. Email: iwao.miyata@okumuragumi.jp

Abstract: This paper presents efforts to improve the productivity of construction sites by visualizing construction
process. As a visualization method, we adopted Digital Twins that combine 3D models and sensing data(construction machinery on-board sensor, AI analysis of cloud camera image). Reproducing the situation at theconstruction site with Digital Twins made it easier for engineers to understand and analyze the works. Result of
the construction site analysis showed the improved efficiency.
Keywords:

Digital Twins, Cloud camera, Sensor, 3D model, Visualization

1. INTRODUCTION
In the construction industry in Japan, there is a concern about labor shortage caused by the recent
retirement of a large number of the elder and young people’s unwillingness to work at construction sites. Therefore,
the productivity of construction must be improved. The Ministry of Land, Infrastructure, Transport and Tourism(MLIT) is carrying out "Project on introducing and utilizing innovative technologies to dramatically improve theproductivity of construction sites" in response to i-Construction aimed at improving the productivity of
construction sites, and an integrated innovation strategy. In order to improve productivity in the constructionindustry, waste of each work must be eliminated and work plans should be improved. In this research, we focusedon visualization that using a 3D model to identify the waste of work.
As visualization using 3D model, efforts are made to solve the problem of planning and decision making
by reflecting various information in 3D model in real time like Virtual Singapore (Singapore Government, 2018).
In addition, Autodesk and University of Illinois used UAV photogrammetry to create a point cloud at aconstruction site, and conduct an effort to efficiently grasp the current situation and calculate the amount of soil
(Autodesk, 2015).
In this paper, as a visualization method using 3D model, we constructed a digital twin model combining
handy laser scanner measurement data and sensing data. The construction machinery was equipped with an onboard sensor to sense its position and movement. A cloud camera system was deployed to capture the situations at
the site. We were able to visualize the situation in the field by analyzing the digital twin model and sensing data,and streamline the field work.
2. DIGITAL TWINS CONSTRUCTION METHOD
2.1 Construction of 3D models
We created 3D models of structures and terrain for the purpose of reflecting the progress of construction
site to digital space (Figure 1). The structure model was developed based on the design drawing. As for the terrain
model, the plan view, the Geographical Survey Institute geography mesh, and the survey data were combined tocreate the terrain around the site. In addition, according to the construction progress, point cloud data at that point
in time as shown in Figure 2 was acquired using a handy laser scanner. When measuring the terrain by the handy
laser scanner, markers were placed at the reference points and coordinates were adjusted. Based on the point cloudmeasured by the handy laser scanner, a 3D model was created at each construction stage.

Figure 1. 3D model

Figure 2. Handy laser scanner and measurement data141

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2.2

Construction machinery on-board sensor
In order to measure the position and work status of construction machinery (Backhoe, Bulldozer, Road
Roller) at the site, we mounted onboard sensor on it. As shown in Figure 3, the sensor consists of a sensor body, a
receiver, and a wireless base for reception. The data measured by the sensor was uploaded to the cloud server
every 10 seconds through the wireless LAN. Figure 4 shows the installation status of the sensor. The effective
range of the receiver was a radius of up to 300 m. If the target site exceeds the effective range of the receiver, thereceptor shown in Figure 3 was installed to expand the effective range of the sensor. The information that thesensor acquires shown in Table 1. The sensor data were subjected to state determination by the following method.
The work discrimination flow is shown in Figure 5.
ReceiverSensor body

Reception

Figure 3. Construction machinery on-board sensor

Sensor type

No.
１－１

Acceleration

１－２
１－３

Geomagnetism

GPS

２－１
３－１
３－２
３－３

Figure 4. Installation situation

Table 1. Information acquired by the sensor
Data transmission
ParameterTypefrequency
X axisNumberdirection
10 times per
Y axisNumberseconddirection
Z axisNumberdirection
Angle
NumberOnce per second（degree）
Latitude
NumberLongitude
Once every
Number30 seconds
Above sea
Numberlevel（m）

Figure 5. Work discrimination flow

142

Remarks
Including gravitational
acceleration

Clockwise 0 ~ 360 °assuming north 0 °Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

When determining sensor data, we classified in the order of traveling, operating, idling, and stopping.
Traveling is defined as the state where construction machinery is moving to the work place without work.
Operating is the state engaged in the construction work which analyzes. Idling is a state in which the engine ofconstruction machinery is operating but construction work and movement are not performed. Stopping is a statein which the engine is not operating.
GPS position information was used to determine traveling. If the distance calculated from the GPS
position information of two seasons is equal to or greater than the threshold, it is traveling.
The accelerometer value was used to determine operating and idling. Threshold 1 was set for operating
and threshold 2 for idling. The acceleration threshold was an acceleration at the time when the state of theconstruction machinery in the video of the construction machinery changed. Operating and idling were determinedby Equation (1).
(1)

𝐺 = (𝑥 − 16389) + 𝑦 + 𝑧

where x, y and z in equation (1) are values of the accelerometers. If the average value of Gi for 10 seconds is equal
to or greater than threshold 1, it is designated as operating, and if it is less than threshold 1 and equal to or greaterthan threshold 2, it is designated as idling.
When it was neither of traveling, operating, idling, it was stopping.
In order to determine the direction of the construction machinery, the determination of the swing state was
performed. Geomagnetic sensor values were used to determine the swing state. The swing state was calculated byadding the angle parameter Di. The angle parameter Di was calculated by Equation (2).
(2)

𝐷 = min(𝑑, 360 − 𝑑)where d is the absolute value of the difference with the previous angle data. The swing state was classified into
three stages of less than 60 °, 60° to 120 °, and 120 ° or more.
2.3

Cloud camera image analysis
The scene image with a cloud camera was cut out as a still image, and the object recognition by AI was
performed for each image. The AI model recognized objects learned in advance, and stored the object type andthe position on the image in the database. Also, objects of Worker, Construction machinery (Dump Truck, Backhoe,
Bulldozer, Road roller), and Materials (Skin plate, Strip bar) are regarded as AI recognition targets. Since
construction machinery may overlap in the image, the image of that condition is learned in advance to improve thedetermination accuracy of AI.
After determining the objects, the state of each object and the states of a plurality of objects are combined,
and logic processing is performed to determine the work of the objects according to the rule base. An outline ofwork determination based on object recognition results is shown in Figure 6. Combine data with the same ID anddifferent time, and determine the flow line of the object. The work content is determined from the state of theposition and posture in the flow line of the object.
Processing

Data

Recognition result

Site image

Still image
AI imagerecognition

Logic processing
Flow line
connection

Recognition floe line

Recognition result

Flow line

Object statusdetermination

State determination

State
Work determination

Work
Process
determination

Process

Discriminationof work frommultiple object states

Process determination

Process identificationfroma combination of work

Figure 6. Outline of process determination from object recognition result143Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2.4

Digital Twins
The construction site was reproduced on a web browser by superimposing construction machinery onboard sensor data and AI analysis data of cloud camera images on a 3D model on a cloud server. The construction
machinery on-board sensor data reproduces the position information of the machine, and the AI recognition databy the cloud camera image reproduces the position information of man, material and dump. Data was updated
once every 10 minutes.
In addition, location information of construction machinery, worker, and materials on the model can be
manipulated, added, and deleted on the Web browser to simulate the situation in the site. By manipulating the timebar, I could also display the model of any time. The web page of Digital Twins is shown in Figure 7.

Simulation operation button

Model information

Time bar
Figure 7. Web page of Digital Twins
The sensor data displayed the latest position information among data within 60 minutes from the current
time for each individual data. As for image analysis data, among the data of worker, materials, and construction
machinery, only the latest 20 data within 60 minutes from the current time and the latest data were displayed.
3. RESULTS AND DISCUSSION
The digital twin model was tried at the construction site shown in Table 2. Among the target construction
work, we worked on visualization of the progress of construction and improvement of the work plan for the 3900
m2 reinforced soil wall construction method. The reason for choosing the reinforced soil wall construction method
is that it is possible to operate analysis and improvement of the construction procedure in a short cycle because
there are many repetitive operations. Reinforcement work will repeat the three steps of wall work assembly,
backfilling and reinforcement installation after the foundation is installed. After repeated work, construction will
be carried out according to the following procedures: concrete placement, protection fence foundation installation,and tour arrangement. In this paper, we apply the digital twin model in three iterations. The situation of each workof reinforced soil wall work is shown in the Figure 8.

Name
Orderer
Construction
periodConstruction
site
Construction
outlineTable 2. Construction site information2017 Tokai ring road Takatomi IC north district construction
Chubu Regional Development Bureau, Ministry of Land, Infrastructure, Transport and Tourism(MLIT).
February 8, 2018 - April 26, 2019
Nishi-Fukase, Yamagata City, Gifu Prefecture, Japan
Construction extension L = 500 m
Retaining wall work Reverse T Retaining wall
Reinforcing earth wall work 3, 900 m2
Other

144

L = 65 m

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Wall work assembly

Backfilling
Reinforcement installation
Figure 8. Work of reinforced soil wall work

A platform that utilizes Digital Twins is shown in Figure 9. When logged in from the platform, adashboard screen was displayed, and it was possible to confirm the current operation status of heavy equipmentfrom the data of construction machinery on-board sensor. When construction machinery was in operation, theoperation status marker turned on and the operation time was displayed. Also, by moving to the sensor graph page,
the working time per unit time can be confirmed. On the work discrimination page, it was possible to see theclassification of the work content obtained by the AI analysis of the cloud camera image. The cloud camera pageand Digital Twins page were able to visually confirm the situation at the site.

Digital Twins page

Login page

Operation status marker
Dashboard page
Operating timeTime
Sensor graph page
Work discrimination page
Cloud camera page
Figure 9. Web platform
The 3D model of the terrain data was modeled by executing handy scanner measurement once a month.
The situation of the site was grasped by six cloud camera. In determination of work contents by AI analysis ofcloud camera images, dumping of soil by dump truck, spreading of soil by bulldozer, rolling pressure of filling byroad roller, and transportation of skin plate by worker and backhoe Members were classified into installation of145

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

strip bars, material transportation by backhoe, filling, and leveling work.
The acceleration data of the road roller is shown in Figure 10 as an example of the construction machineryon-board sensor. A value less than or equal to the threshold 1 is determined to be stopping, idling is determined
between the thresholds 1 and 2, and operating is determined to be 2 or more. When a value exceeding the threshold
2 is at least, the construction machinery is visualized in the graph as operating.
Acceleration data (road roller)

Acceleration

Acceleration

Acceleration data (road roller)

Threshold 1

Threshold 2Threshold 1

Time

Time
Sensor data

Threshold 1

Sensor data

Threshold 1

Threshold 2

Figure 10. Acceleration data of the road roller
The 3D model used for Digital Twins had to reduce the amount of data in order to ensure operability on
the web browser. Therefore, as for the topographic map display of the terrain model configured by TIN, adjacentfaces with a slope of 20 ° or less were integrated, and the number of faces was reduced by about 55%. With regard
to cloud cameras, it was necessary to change the installation location as construction progressed.
The analysis data made it possible to streamline the transport installation procedure of the skin plate in
the reinforced soil wall method. The effective use of construction machinery waiting time during installation work
has reduced the waiting time that accounted for 32% of construction machinery work time to 12%. In addition, thetransport time was reduced to 50% before the change due to the change of the wall material transport machine.
4. CONCLUSIONS
The field information could be efficiently visualized by utilizing the digital twin model that could reflect
the field sensing data using sensors and cloud cameras. By utilizing these data, the work could be streamlined andcontributed to the improvement of the productivity at the construction site.
Although the location information of construction machinery and workers is real time, updating of
topographic data is once a month, so there was a problem that deviation from progress becomes large. Since
updating of terrain data involves measurement and modeling tasks, it is difficult to improve real-time performanceby increasing the measurement frequency. In the future, we will prepare a model in which the completed shape ofterrain data is divided into blocks in advance, and consider a method to increase the display of soil on digital twin
according to the construction progress that can be acquired by sensing at the site. In addition, the heavy equipment
sensor measured at 10 times per second, and the amount of data was large, which also caused a problem that it
took time for the process determination processing by AI. In the future, improvements will be made such as
temporary processing within the sensor to speed up processing.
Because it can confirm the situation in the field with Digital Twin through the Web browser, versatility ishigh. By displaying using virtual reality, various usage methods such as detailed on-site simulation, on-siteexperience at remote places, and residents' explanations can be considered. In addition, I think that it will lead tothe improvement of the transparency of the site and the rapid formation of agreement because the information on
the site can be shared easily.
ACKNOWLEDGMENTS
This research was funded by the Ministry of Land, Infrastructure, Transport and Tourism for “Project on
introducing and utilizing innovative technologies to dramatically improve the productivity of construction sites”
and was commissioned to order by the Chubu Regional Development Bureau.
REFERENCES
Singapore Government. (2018). Virtual Singapore. Retrieved from Virtual Singapore website:
https://www.nrf.gov.sg/programmes/virtual-singapore
Autodesk. (2015). Development of a new technology that streamlines construction progress management with
UAV photos by Autodesk and University of Illinois. Retrieved from Autodesk website:
https://www.autodesk.co.jp/press-releases/archive/2015-04-23

146

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

AR VISUALIZATION OF PHYSICAL BARRIER FOR WHEELCHAIR USERSBASED ON REALTIME DEPTH IMAGING
Rio Takahashi1, Hiroshige Dan2, and Yoshihiro Yasumuro3
1) Graduate Student, Graduate School of Science and Engineering, Kansai University, Suita City, Japan. Email:media.englab.tostemkuniv15@gmeil.com
3) Ph.D., Assoc. Prof., Department of Environmental and Urban Engineering, Kansai University, Suita City, Japan. Email:dan@kansai-u.ac.jp
4) Ph.D., Prof., Department of Environmental and Urban Engineering, Kansai University, Suita City, Japan. Email:yasumuro@kansai-u.ac.jp

Abstract: Since Japan is facing the issue of rapid aging society, the number of potential wheelchair users isovergrowing, as more and more seniors need long-term cares. The government has worked on prevailing thebarrier-free environment by establishing a law regarding promotion of smooth transfer for elderly and physicallydisabled people. Holding the 2020 Olympics is also a vital situation for infrastructure development in Japan.
Barrier-free maps, for example, are prepared in many communities, but they cover only existences of facilities,e.g., lifts, slopes, and handrails, in major public institutions. Therefore, they miss the details of physical barriers,
such as bumps and width clearances on the path, needed for the independent mobility of the wheelchair users. This
paper addresses a method to find out physical barriers by using a depth camera to visualize them efficiently throughaugmented reality, which may be useful for the facility managers. A depth camera acquires 3D point cloud in thetarget space and checks the existence of interference between the environment and the volume of an actual
wheelchair. The proposed system also performs back-projection of the detected barriers onto RGB color video
frames for 2D AR representation. The verification of the achieved accuracy of the barrier check, as well as theimplementation scheme, are reported in the paper.
Keywords: Augmented Reality, Wheelchair user, Barrier-free, Depth imaging
1. INTRODUCTION
In recent years, with the declining birthrate and aging population in Japan, the percentage of seniors in thenational population is increasing. As of 2015, the number of seniors is about 34 million, and the aging rate is
26.7%, and it is estimated that the number of seniors will continue to increase in the future after the Tokyo
Olympics and Paralympics. Besides, 40% of seniors go out almost every day, and more than 90% go out morethan once a week (according to the survey of the Ministry in 2017). As a result, further increase in physically weakpeople is predicted, and it is required to expand a barrier-free society in which physically weak people can coexist
with healthy people. Examples of barrier-free measures include the new barrier-free laws and the barrier-free maps
implemented by national and local governments. However, not only the target facilities are limited, but delicate
barrier information such as wheelchair accessibility in the daily life of disabled people cannot be obtained, andthus the barrier-free environment from the viewpoint of the users is not prevailed. In this study, we propose asystematic solution to verify the barriers for wheelchair users in their familiar environment where physical barriers
are routinely found. The aim is to make it possible for both wheelchair users and healthy people, who will be
increasing in the future, to coexist in an easy-to-move, easy-to-use society (as shown in Fig. 1).

Figure 1. Schematic image easy-to-use barrier inspection tool
2. METHOD
Yasumuro et al. created a 3D model of the target space by acquiring 3D coordinates of the real space witha depth camera (Yasumuro 2015). The barrier verification is performed by combining the depth map and thewheelchair's minimum traffic trajectory and checking for interference. A 3D model can be constructed from data147

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019obtained by capturing the target area from multiple viewpoints with a depth camera, but it takes time and labor forprocedures such as alignment of data for model creation and interference judgment with a wheelchair model.
Therefore, there is a need for a means that can carry out on-site barrier verification immediately. And a convenient
tool is desirable so that a facility manager or a care manager can perform barrier confirmation in daily work.
In this research, in order to provide accurate and easy-to-understand barrier information in real time in aneasy-to-use manner, we aim to visualize the barrier in AR, taking into consideration the relationship between thecamera position and the real space and the shape of the wheelchair. Also, the barrier display on 2D color images
is performed in parallel to support the association with real space. Furthermore, we propose a barrier verification
tool that can be used even on mobile terminals equipped with 3D sensing camera technology in order to make iteasier and more accessible for the users.
A floor surface is detected from depth image data via a depth camera, and interference judgment between
a 3D volume model with actual dimensions of a wheelchair and the surrounding physical objects is realized. The
process chain of the system is shown in Fig.2. First, a wheelchair model is prepared based on the specification of
the standards for wheelchairs and the rotation range as shown in Fig.3. 3D depth image is acquired from a depth
camera’s data stream and dealt with as 3D point cloud data. A plane approximation is performed for a given radius
range from the center coordinates of the acquired depth image (Takahashi 2018). In planar approximation,RANSAC (Fischler 1981) is used to perform the least-squares method, excluding outliers of the points data fromthe sample range. Then, the wheelchair’s volume is set based on the estimated normal of the floor plane, and theintersecting 3D points from the real space through the depth camera is checked out. As a result, the 3D points
detected as barrier part is highlighted in depth images. Also, the color image acquired in advance with a USB
camera is associated with the pixels of the depth camera, since the color imaging capability is not equipped withthe depth camera we used in this paper. By calculating a homography matrix so that the color-pixel frame of theUSB camera and the depth frame are fit by pixel-to-pixel correspondences. After the barrier verification in thedepth frame, the pixels of the color image corresponding to the 3D point cloud determined to be the barrier portion
are highlighted to make the barrier in 2D display with AR manner in realtime as well.

Figure 2. General structure of reports, papers, and essays

Figure 3. 3D wheelchair volume model, considering an ergonomic dimensions and industrial standards
3. IMPLEMENTATION AND EXPERIMENTS
We used a TOF (Time of Flight) type of near-infrared camera, SwissRanger SR4000 (Mesa Imaging Inc.)(Lange, 2001) to acquire depth images of 176 x 144 pixels in 60 fps rate (Fig. 4 (right)), and a color camera
UCAM-C750FBBK (Elecom Inc.) for capturing color image frames with 1024x768 pixels (Fig. 4 (left)). We used
Visual Studio 2010 (Microsoft Corporation) as the development environment with OpenCV library for image
processing, and OpenGL API for both 2D and 3D AR display. With a laptop PC and the camera set connected as148

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019shown in Fig. 4, the barrier location is verified by walking in the target space with the camera facing on the floor.
Example pair of a depth and color image is shown in Fig. 5. Since the angle of view differs in these cameras, weapply a conversion to the color camera that matches the depth camera with the narrower angle of view. Using the
3D information in the depth image frame, the RANSAC algorithm effectively estimates the floor plane by sampling
the yellow points as shown in the Fig. 6. The sampled points lie only on the floor, excluding the different heightof s furnisher leg, which is eventually highlighted in red as a barrier. The green cylindrical object depicts thevolume model of a wheelchair. On the other hand, since the power cord is a small step less than 2 cm, it is notjudged as a barrier.

Figure 4. Devices used for the implementation; A TOF camera and a color USB camera

Figure 5. Color image captured with a USB color camera (left) and depth image from a TOF camera (right)

Figure 6. Floor estimation based on sampling with RANSACEXPERIMENTAL RESULTS
In this research, we assumed barrier verification in flow lines in wheelchair user's daily life andexperimented with indoor space as target space. This time, as shown in Fig. 9, in a general classroom of Kansai
University in Japan where wheelchair users also use, verification was conducted in the passage between desks and

4.

149

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019around the desk for wheelchair users as shown Fig. 7. The verification results and the color image of theverification area are shown in Fig.8. The viewpoint was placed right above the wheelchair model, and the barrier
judgment was carried out based on the presence or absence of the highlighted points in red. It was also found that
there was not enough space for wheelchair users to drive the wheels on their own, as the entire classroom wasinclined from back to the front. The detected barrier location is superimposed on a color image to display AR in
2D as well (Figure 8 (right row)).
In the classroom environment, the desk in the front row closest to the entrance door is not fixed to thefloor and can be moved for dedicating to wheelchair users. For the experiment, the desk was turned at an angle sothat wheelchair users could turn around, and the model diameter was set to 160 cm for verification in considerationof the rotation movement of the wheelchair (Fig. 7 (bottom)). It turned out that the barrier is complicated in thespace and it cannot quickly turn around, for example, when the space between the wall and the desk legs is thenarrowest when getting by the desk, or when the top plate of the desk interferes. Since the desk's legs were anangled shaft, if it was unclear at first glance where it would get in the way. Even so, it was easy to judge the barrier
through our system. As a result, in this case, it was confirmed that wheelchair users had a difficult design to sit
freely (Fig.9).

Figure 7. Case studies at a class room; normal passage (upper) and movable desk for wheelchair users (lower)

Figure 8. Visualization result for the normal passage between the desks

150

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 9. Visualization result for the movable desk space
5.

CONCLUSION
We proposed a handy system to display physical barriers for wheelchair users in AR. The barrier
verification can be carried out so quickly, even for the complicated 3D spatial configuration without bringing aphysical wheelchair. The barrier verification is conducted in 3D and the results are represented in AR on 2D
perspective.
Our next step is focusing on easy-to-record capability for accumulating the verification results for facility
management. Also, we aim to implement our algorithm on a more compact type of mobile device with a camera
with a wide angle of view, such as a smartphone, as a handy device close to wheelchair users. One of thepossibilities is to simplify barrier verification by linking with technologies such as SLAM technique in integrated
devices of a depth camera and a color camera such as Google Tango platforms. (D. Keralia, 2014)
ACKNOWLEDGMENTS
Part of this research is supported by the Takenaka Corporation.
REFERENCES
Cabinet Office, Government of Japan (2017), Aged Society White Paper, pp. 2-6. (in Japanese) website:
http://www8.cao.go.jp/kourei/whitepaper/w-2017/zenbun/pdf/1s1s_01.pdf
Keralia D., Vyas K. K., and Deulkar K., (2014), Google Project Tango – A Convenient 3D Modeling Device,
International Journal of Current Engineering and Technology, Vol.4, No.5. pp. 3139-3142.
Han C. S., et al., (2002) A Performance-based Approach to Wheelchair Accessible Route Analysis OriginalResearch Article, Advanced Engineering Informatics, Vol.16, Issue 1, pp. 53-71.
Sato M. (2011), Practice Knowledge of Barrier-free Renovation, Ohmsha, Ltd. (in Japanese)
Fischler M. A., Bolles R. C., (1981) Random Sample Consensus: A Paradigm for Model Fitting with Applications151Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

to Image Analysis and Automated Cartography, Communications of the ACM, vol. 24, no. 6, pp. 381-395.
Lange R., Seitz P., (2001) Solid State Time-Of-Flight Range Camera, IEEE Journal of Quantum Electronics, vol.
37, Issue 3, pp. 390-397.
Takahashi R., Matsushita S., Dan H., Yasumuro Y., (2018) Visualization of Physical Barrier for Wheelchair Users
Using Depth Imaging, Proceedings of the 18th International Conference on Construction Applications ofVirtual Reality (CONVR2018), pp. 87-94.
Yasumuro Y. and Dan H., (2015) Web-based 3D Barrier-Free Verification for Wheelchair Access, Proc ofInternational Conference on Civil and Building Engineering Informatics (ICCBEI2015), p.46 (ID:70).

152

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Facility and Infrastructure Management

153Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

STEEL BRIDGE INFORMATION DELIVERY MODELFOR EARNED VALUE MANAGEMENT (EVM)
Teruaki Kageyama1, Nobuyoshi Yabuki2
1) Senior Researcher, Research Department, Japan Construction Information Center (JACIC), Tokyo, Japan.
Email: kageyamt@jacic.or.jp
2) Ph.D., Prof., the Division of Sustainable Energy and Environmental Engineering, Graduate School of Engineering,Osaka University, Japan. Email: yabuki@see.eng.osaka-u.ac.jp

Abstract: Building Information Modeling (BIM) has been widely adopted in the building industry, and its
established methods and technologies show enormous potential in benefiting the civil engineering industry.
Through the rapid growth of BIM in the civil engineering industry, mandatory use in government procurements,
and utilization for improving productivity, the importance of 3D product models in the civil engineering industry
is becoming increasingly prominent. The purpose of this paper is to propose for government organizations to
implement Earned Value Management (EVM) using 3D product models. Firstly, the government contract process
in line with the acts, guidelines and standards are mapped using an Information Delivery Manual (IDM). Next, inorder to implement EVM using 3D product models, it is necessary to organize the relationship between the Work
Package (WP), the minimum units of the Work Breakdown Structure (WBS), and the government contract units.
For this reason, 3D product models with government contract units are defined as Information Delivery Elements(IDE). Finally, by using the Steel Bridge Information Delivery Model (SB-IDM), which has been integrated with
the IDE of the steel bridge structure, the usability of EVM is evaluated.
Keywords:

BIM for infrastructure, product model, information delivery manual, earned value management

1. INTRODUCTION
Since 2004, the term BIM has begun to spread throughout the building industry, and over the past
decade, BIM technology has rapidly become commonplace. BIM, as defined by the U.S. National Institute of
Building Sciences, is “a digital representation of physical and functional characteristics of a facility. A BIM is ashared knowledge resource for information about a facility forming a reliable basis for decisions during its life
cycle; defined as existing from earliest conception to demolition” (NIBIS, 2016). BIM is now used widely forgovernment procurements and productivity improvement. Mandatory use of BIM for UK government
procurements started in 2016, and the EU BIM Handbook (EU BIM Task Group, 2018), which highlights
government-led initiatives of EU nations, are prime examples of the widespread adoption of BIM. For government
procurements in Japan, the Ministry of Land, Infrastructure, Transport and Tourism (MLIT) launched a trial projectcalled Construction Information Modeling/Management (CIM) in 2014. CIM is defined as a method to “streamline
and improve a series of construction manufacturing processes via measures including the use of 3D product models
from the initial stages of the civil engineering industry, such as planning, surveying, and engineering, and thecoordination and development of 3D product models at each stage of construction and operational management”(Yabuki, 2016). Putting together the knowledge accumulated through the CIM trial projects (survey, design, andconstruction), the MLIT, in 2016 fiscal year, formulated the “CIM Implementation Guidelines.” The document
covers the development of 3D product models and case studies of implementation over project phases ranging
from survey and design to operational management. As proven by the fact that the streamlining of consensus
building is recognized as the most notable benefit, CIM trial projects have proven the effectiveness of 3D productmodels as a tool to aid communication between stakeholders. However, there is an insufficient number of casestudies on tasks where CIM is fundamentally considered effective, such as supervision and inspection, calculationof work volumes, and project schedules. According to the MLIT, this is partly due to the “lack of guidelines andrules” related to the development process of 3D product models (MLIT, 2017).
The process control in public works, the inspector has confirm the reports and progress rate of theprocess that the contractor has manage. In addition, the adequacy of the plan and the evaluation of the progress are
judged by the personal experience of the inspector. For this reason, there is a need for a management method that
objectively performs construction progress management based on the government procurement and makes aquantitative judgment. The purpose of this paper is to propose for government organizations to implement EVM
using 3D product models. Firstly, the government contract process, in line with the acts, guidelines and standards,
is mapped using an IDM. The IDM is used to provide an integrated reference for the process and data required by
BIM (buildingSMART International, 2010). Next, EVM is used as a project management system to help project
managers measure and analyze performance-related factors of projects such as costs and schedules. In order toimplement EVM using 3D product models, it is necessary to organize the relationship between the WP, theminimum unit of the WBS, and the government contract units. For this reason, 3D product models, together withthe government contract units, are defined as IDE. Finally, by using the SB-IDM, which has been integrated with154

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

the IDE of the steel bridge structure and is generated in accordance with the IDM of the government contract units,
the usability of EVM is evaluated.
2. EXISTING STUDIES
2.1 BIM in the Civil Engineering Industry
BIM has been widely adopted in the building industry, and its established methods and technologies
show enormous potential in benefiting the civil engineering industry. The application of BIM in the civilengineering industry is referred to as CIM in Japan, but it is also called Bridge Information Modeling (BrIM),
Civil Integrated Management, Civil Information Modeling, or Virtual Design and Construction (VDC) by
researchers and organizations (Costin et al., 2018). The use of BIM in the United States has a long history withextensive research in this area, but Europeans are also rapidly achieving maturity in BIM, and the UK, Germany,and France are now using BIM as a favorable technology for design and management of their infrastructures.
There is a significant growth in adoption of BIM for infrastructure between 2012 and 2017 in Europe, and the rateof BIM implementation for infrastructure projects has risen from 20% to 52% in this period (SmartMarket report,
2019). Although initially intended for buildings and vertical construction, there have been various international
efforts by buildingSMART International (bSI) groups to use and expand the Industry Foundation Classes (IFC),
which have provided a comprehensive specification of the information about the building industry. In order toincrease the scope to include infrastructure development for other infrastructure models (bridges, roads, tunnels,etc.), bSI created the “Infrastructure Room” in 2010 to serve as the center of various international groups
implementing IFC for infrastructure. Since then, various projects are in progress.
EVM
The American National Standards Institute/Electronic Industries Alliance (ANSI/EIA) approved theEarned Value Management Systems (EVMS) standard in June 1998. EVMS is a project management system thathelps project managers measure, analyze, and manage performance-related factors for projects such as costs andschedules through a comparison and analysis of the Planned Value (PV), Earned Value (EV), and actual costs(Fleming and Koppelman, 2010). This method uses a Cost Performance Index (CPI) and Schedule Performance
Index (SPI), which are two management elements for project managers to control costs and schedules to help themunderstand the construction progress. In order to implement EVMS, it is necessary to establish a hierarchicaldecomposition of the total scope of work to be carried out by the project team to accomplish the project objectives
and create the required deliverables. The operation is called WBS, and the smallest components of WBS are called
Work WP.
2.2

2.3

EVM with BIM
According to a study of management methods for construction projects conducted by Russel et al.
(2015), 50 international construction projects used one or more systemic project control mechanisms such as BIM,
EVM, and Location Based Management (LBM). Of that subset, the document also reports that 38.1% used two
and 23.8% used all three for project control. Other studies on EVM in the civil engineering industry involving 3D
product models include a study by Yabuki et al. (2004) on the development of construction management systems
for cut and fill earthworks based on 4D-CAD and EVM. The study aims to streamline and improve construction
management and implement progress payment methods. In addition, a study by Mohamed et al. (2014)
implementing an EVM concept on BrIM, which is a method for the 3D product modeling of bridges developed by
the Federal Highway Administration (FHWA, 2016), and which investigates a system for controlling costs andscheduling construction works. However, there are no existing studies focusing on the relationship between 3D
product models and government contract units/project management. A report on a literature review and criticalanalysis of BIM for transportation infrastructure by Costin et al. (2018) pointed out research gaps regarding BIM
to apply for infrastructure management: “Project management methodologies including EVM and Integrated
Project Delivery (IPD) have been applied and studied with BIM capabilities. Integration of other project
management methodologies with BIM for infrastructure is a considerable gap that could bring significant value
for infrastructure projects.”

3. IDM AND IDE
3.1 Acts, Guidelines and Standards
In public construction works ordered by the MLIT and local public organizations in Japan, the MLIT
performs a quantity survey prior to procuring a project using the “ceiling price,” which is set in accordance withthe “Laws and regulations on the procedure for determining the contract amount.” With an aim of establishing
fairness in the ceiling price calculation for contract works, the MLIT specifies the “Operation of Civil Engineering
Cost Estimate Guidelines and Standards” and “Civil Engineering Construction Quantity Survey Outlines andGuidelines (MLIT, FY2018 Edition).” The document specifies items required for calculating costs of the relevant155Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

construction projects which are needed to be registered for the design specifications when a contractor undertakes
a civil engineering project under direct control of the MLIT. Figure 1 shows the Acts, Guidelines and Standards
regarding government contracts in Japan.

Figure 1. Acts, Guidelines and Standards regarding government contracts in Japan
In order to reevaluate the traditional quantity survey system and create a unified and consistent scheme,
the MLIT is currently working on the “New Cost Estimating System of Public Works” to create a new quantity
survey framework. As part of this process, the government contract units are standardized to calculate direct
construction costs. This standardization process includes work type-based categorization to cover the number of
hierarchical layers for quantity surveys, definitions of the hierarchical elements, and segmentation, as well as astandardization of terminologies and quantity survey units. The government contract unit standardization is asystem tree diagram which subdivides diverse construction items in a standardized manner and consists of sevenhierarchical layers (levels). Using these levels, standardized work elements required for derivatives of the relevant
construction work can be identified. Table 1 shows hierarchical layers (levels) for standardized government
contract units.
Table 1. Hierarchical layers (levels) for government contract units
Level

Identifier

Level 0Business Classification

Level 1

Construction Classification

Level 2Work Item

Level 3

Class

Level 4

Subdivision

Level 5Specification

Level 6

Quantity Survey Element

Content
Classification based on budgeting system and business
operations.
Level 0 divided to address work order lot and client
General term for a series of operations required to constructcertain structural elements (part of Level 1 elements).
Level classification to connect Levels 2 and 4 to clarify theoverall standardization system.
Level to indicate units and contract units that are basic derivative
units or temporary structure units of the relevant construction
work.
Objective information on materials, standards, and contractspecified conditions for Level 4 configuration components.
Level 4 cost calculation components generally not disclosed incontracts.

156

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3.2

IDM for Steel Bridge Project
BIM is a new approach for describing and displaying the information required for the design,
construction, and operation of constructed facilities. It can bring together the numerous threads of different
information used in construction into a single operating environment, reducing, and often eliminating, the need forthe many paper documents currently in use. The IDM aims to provide an integrated reference for process and datarequired by BIM by identifying the discrete processes undertaken within building construction, the information
required for its execution, and the results of that activity. The IDM comprises “Process Maps” and “Exchange
Requirements,” which can be described as layers within the architecture. The purpose of a process map is to help
with understanding how the work is undertaken for achieving a well-defined objective. The preferred approach todeveloping a process map within IDM is to use the Business Process Modeling Notation (BPMN), where an
exchange requirement represents the connection between process and data. BPMN applies relevant information
defied within an information model to fulfil the requirements of an information exchange between two businessprocesses at particular stages of the project. Figure 2 shows IDM for the design, quantity survey, and costestimation stages of a steel bridge project.

A1: Client

Civil Engineering Construction Common Specifications, Chapter VI Road
Article 6804: Detailed Bridge Design, Quantity Survey and Cost Estimation[T01-6804]
Detailed Bridge Design[T02]
Cost Estimation
Construction Quantity
Generalization List
Operation of Civil Engineering Cost Estimate
Guidelines and Standards

Civil Engineering Construction
Common Specifications

Work Item Standard of Productivity

A2: Designer

Civil Engineering Construction Common Specifications, Chapter VI Road
Article 6804: Detailed Bridge Design, and Quantity Survey[T6804-1]
DesignPlanning[T6804-2]
Field
Survey[T6804-3]
DesignCondition[T6804-4][T6804-5]
DesignCalculation[T6804-6]
DesignDrawing

Selection of
Bridge Type

Steel Bridge Information Delivery Model (SB-IDM)

ER: Model Elements
ER: Information Delivery Elements (IDEs)[T6804-7]
Quantity
Survey[T6804-8]

LandscapeConsideration[T6804-9]
TemporaryWork Plan.

[T6804-10]
DesignVerification

Civil Engineering Construction Quantity Survey Outlines andGuidelines【Legend】
Event (Start)

Sequence Flow

Message

Event (End)Message Flow

Exchange Requirement

Task

Relationship

Guidelines and Standards

Figure 2. IDM for the design, quantity survey, and cost estimation stages of a steel bridge project
3.3

IDE

In order to implement EVM using 3D product models, it is necessary to organize the relationshipbetween the WP, the minimum units of the WBS, and the government contract units. The government contract
units standardized in Japan define Level 4 as “the level to indicate units and contract units that are a basic derivative
unit or temporary structure unit of the relevant construction work.” Meanwhile, Level 5 defines materials andspecifications (objective descriptions) and conditions disclosed in contracts, and Level 6 defines component
elements for cost calculations. To determine the government contract units when conducting a quantity survey,
Level 4 (Subdivision) and Level 5 (Specification) work items specified in the “Quantity Survey Items andClassification Conditions,” set forth by the Civil Engineering Quantity Survey Outlines, must be calculated. IDEof the steel bridge as an exchange requirement are shown in Figure 3. IDE equipped with Level 4 and Level 5
information can be used to quantify work items in accordance with the Quantity Survey Items and Classification
Conditions (the procurement unit of government organizations). Figure 4 shows a class diagram for the IDE linked
with the EVM.

157

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Level 1
Construction Class

Steel Bridge Upper
Deck

Level 3
Level 2
Class
Work Item
ER: Model Element

Fabrication

Level 4
Subdivision

Main Girder
Fabrication

Upper Flange

Level 5
Level 6
Specification
Quantity Survey Element
Quantity Information
Steel Manufacturring
Manufacturing
Steel Material

Mass

Bolt / Nut Type

Unit

Stud Dowel Type

Unit

Concrete

VolumeReinfoce

Type

Mass

Fundation

Type

VolumeWeb Plate
Lower Plate
Sway Baracing

Bridge Substructure

Abutment

Concrete Abutment

Wing Wall
Ballst Wall
Abutment Wall

【Legend】
EntityRelationship

Figure 3. IDE for a steel bridge

Figure 4. Class diagram of the IDE linked with the EVM
4. SB-IDM AND EVM
The potential of EVM coordination was evaluated by outputting the government quantity survey dataobtained from the SB-IDM, which has integrated the IDE of the steel bridge structure generated in accordancewith the newly defined data structure. The newly defined SB-IDM was developed using Autodesk Revit 2018
(hereinafter referred to as Revit). With Revit, geometric shapes can be saved with additional attribute information.
To count work items, Revit’s “Calculate Totals” feature was used. Parameters such as units, numerical digits, andsteel material calculation methods were adjusted to ensure that the appropriate IDM components could be produced.
Table 2 shows design conditions of the steel bridge upper deck (girder fabrication: simple plate girders) used forthis evaluation, and Figure 5 shows components of the main girder and connection materials. Using thisinformation, work items can be quantified at a per-material basis. In quantity surveys by government organizations,the manufacturing and material costs, as well as the assembly time, welding time, and temporary assembly time,
are calculated based on the weight of the steel plates per material type after quantifying the work items. Whenperforming this process, the assembly time shall be calculated using a block length of the component materials.
The total length of welding required in quantity surveys by government organizations can be calculated bypredefining the welding edges of each component material. Figures 6 show the SB-IDM with IDE components inplace.

158

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

FormatBridge Length
Girder Length
Skew Angle
Live Load
Large Vehicle Traffic Volume
Slab Thickness
Design Lateral Seismic Coefficient

Table 2. Design conditions
Unitmm
DegreeNumber of Vehicles / Day / DirectionmmSimple Plate Girder122.2040.60@30°
B Live Load2000 or greater250
Kh = 0.25

Figure 5. Main girder components and connection member components

Figure 6. SB-IDM
4.1

Quantity Survey and Cost Estimation
A quantity and cost estimation summary (Table 3) was generated using the quantification obtained fromthe SB-IDM. For the quantity and cost estimation summary, quantity and cost information from the IDE produced
in Revit were exported to a CSV file. Verification was made to ensure work resources, excluding the work hours
required for the WP of WBS, and could be exported to each document format.
4.2

EVM
The construction procedure was to simultaneously construct the A1 and A2 abutments during the steel
bridge manufacturing period and to construct the P1 and P2 piers sequentially after completion. The steel bridge
will be constructed from the A1 side, and the construction period is set to 9 months, avoiding the flood period. In
addition, the workable days of each month are unified on the 20th. Table 4 shows the results of the calculation ofvarious EVM parameters by inputting data for 5 months for verification. Figure 7 shows a graph of the verticalaxis with the SPI and the horizontal axis with the CPI to evaluate the work progress by EVM. In the case of thedata for verification, the start month indicates the area of low cost/slow process in the lower left where the EV
falls below the PV through the confirmation of site construction conditions and preparation of materials andequipment. The predicted completion period (Tec 2) will be 390 days, which can be expected to be extended by
30 days from the initial construction period. With the progress of construction, the state of the process and cost159

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019stabilizes, and it is possible to read the state of transitioning to the high cost/early process area in the upper right.
The predicted completion period (Tec 1) will be 358 days and the delay will be recovered.
Table 3. Quantity and Cost Estimation Summary

Table 4. Results of calculation of various EVM parameters

Figure 7. SPI and CPI

160

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

5. CONCLUSIONS
There have been a limited number of studies that have been able to apply BIM for the civil engineering
industry. However, basing the previous work that has been undertaken, the research presented in this paper has
purpose for EVM using 3D product models that objectively performs construction progress management based on
the government procurement and makes a quantitative judgment. The key benefits of the SB-IDM that have beendeveloped include: Organized in IDM include the acts, guidelines and standards that must be followed in order to use 3D productmodels in the civil engineering industry; The SB-IDM which are integrated the IDE are defined to use 3D product models with government contractunit.
 EVM based on the government procurement quantity can confirm the progress of construction objectively
and quantitatively;
However, the government procurement system in japan, in principle, it is a design-bid-build. When
applying the method proposed in this report at the time of construction, it is need to be checked the SB-IDM
according to the temporary equipment of the construction method and the construction by the contractor. In
addition, the steel bridge verified in this paper is a basic structure, and it is necessary to increase the number ofverifications and improve the accuracy to develop of the Model View Definition (MVD). I will continue to researchvarious industry organizations and trends including buildingSMART International.
REFERENCES
buildingSMART International. (2010). Information Delivery Manual Guide to Components and Development
Methods. http://iug.buildingsmart.org/idms/development/IDMC_004_1_2.pdf. Retrieved on 20 April
2019.
Costin, A., Adibfar, A., Hu, H., and Chen, S. S. (2018). Building Information Modeling (BIM) for transportation
infrastructure - Literature review, application, challenges, and recommendations, Automation inConstruction, Vol. 94, pp. 257-281.
EU BIM Task Group. (2018). Handbook for the Introduction of Building Information Modeling by the European
Public Sector. http://www.eubim.eu/downloads/EU_BIM_Task_Group_Handbook_FINAL.PDF.
Retrieved on 20 April 2019.
Fleming, Q. and Koppelman, J. (2010). Earned value project management (4th ed.) Newtowns Square: ProjectManagement Institute.
Jones, S. A. and Laquidara-Carr, D. (2017). The business value of BIM infrastructure, SmartMarket report, Rep.
to Dodge Data & Analytics, Balford, MA.
https://images.autodesk.com/adsk/files/business_value_of_bim_for_infrastructure_smartmarket_report__2012.pdf. Retrieved on 2 March 2019.
Ministry of Land, Infrastructure, Transport and Tourism (MLIT). (2017). “Technical Research” Homepage, Part
4 CIM Implementation Promotion Committee, http://www.mlit.go.jp/common/001197208.pdf.
Retrieved on 22 August 2017.
Mohamed, M. and Mohamed, H. (2014). Implementing Earned Value Management using Bridge Information
Modeling, KSCE Journal of Civil Engineering, pp. 123-130.
National Institute of Building Sciences (NIBIS). (2016). About the National BIM Standard United States,
https://www.nationalbimstandard.org/about. Retrieved on 12 January 2019.
Nobuyoshi, Y. (2016). Introduction to CIM - Innovation of Construction Production System, Rikoh Tosho, p.
208.
Nobuyoshi, Y., Tomoaki, S., and Yoshikazu, S. (2004). Development of a Cut and Fill Earthworks Management
System Based on 4D CAD and EVMS, Journal of Construction Management (JSCE), Vol.11, pp. 9198.
Russell, K. and Roby, H. (2015). Construction Project Control Methodologies and Productivity Improvement:
EVM, BIM, LBM, Proceedings of the 6th International Conference on Engineering, Project, andProduction Management, pp. 57-66.
U.S. Department of Transportation, Federal Highway Administration. (2016). Bridge Information ModelingStandardization. Retrieved from the Federal Highway Administration website:
https://www.fhwa.dot.gov/bridge/pubs/hif16011/ on 20 April 2019.

161

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

A PROPOSAL OF DETERIORATION PREDICTION MODEL FOR TUNNEL
LIGHTING FACILITIES USING MARKOV STOCHASTIC PROCESS
Noriaki Maeda1, and Kei Kawamura2
1) Graduate School, Doctor Course, Sciences and Technology for Innovation, University of Yamaguchi, Yamaguchi, Japan.
Email: g005wc@yamaguchi-u.ac.jp
2) Dr Eng., Assoc. Prof., Sciences and Technology for Innovation, University of Yamaguchi, Yamaguchi, Japan. Email:kay@yamaguchi-u.ac.jp

Abstract: Asset management systems (AMS) are powerful tool for maintenance of civil infrastructures. The
management system model consists of the four steps cycle model that is PDCA(Plan-Do-Check-Action) in the
total quality management system. This paper proposes a deterioration prediction model on the management system
for tunnel lighting facilities. The model is a time-based Markov stochastic process for predicting the deteriorationof health degree of a tunnel unit. The tunnel unit is equivalent to all of tunnel lighting facilities in a tunnel.
Firstly, this paper explains a characteristics of the Markov stochastic process model applied to the deterioration
prediction of a tunnel unit. Secondly, the deterioration of health degree of a tunnel unit is formulated by solving
the master-equation of Markov stochastic process. Then, the applicability and validity of the deterioration
prediction model is shown by illustrating numerical examples using visual inspection data of existing tunnels.
Finally, the paper indicates how the model functions in AMS.
Keywords: AMS, tunnel lighting facilities, Markov stochastic process, deterioration, prediction, health degree,
master-equation, inspection
1. INTRODUCTION
In Japan, civil infrastructure constructed in a high economic growth period have been deteriorating
remarkably. It is necessary for civil infrastructure to maintain with safety and minimum cost. Asset management
systems (AMS) is effective for maintaining civil infrastructure with safety and economy (Miyagawa et al., 2008).
AMS is total quality engineering management system consists of PDCA(Plan-Do-Check-Action) four steps. Four
steps of PDCA correspond P to planning of renew and repair, D to performing of inspection and repair, C topredicting of future condition, and A to modifying of current plan from future condition in maintenance of civil
infrastructure. It is most important for rotating PDCA cycle to predict and evaluate the condition from inspectionof civil infrastructure. Therefore, it is necessary to develop a model of predicting easily future condition of
infrastructure for performing each step effectively in PDCA cycle.
In general, the prediction of future infrastructure condition has practical application issues. In many cases,
the civil infrastructure documents such as inspection result have only a small amount analyzable data. This reason
is caused to not accumulate all data from function start time to the present. In addition, the accumulated data style
is not electronic style but paper style. An inspection result of civil infrastructure consists of condition state fordiscrete condition rating evaluated by the sensitivity of inspectors. It is necessary for analyzing inspection result
to convert discrete condition state to continuous condition value.
Therefore, this study proposes a model to predict future condition of tunnel lighting facilities considering
these issues for tunnel in civil infrastructure. The model having the characteristic of comprehensible to all persons
concerned with AMS solves to these issues using simple methods. The first method is the utilization for acontinuous scale of health degree to discrete inspection result. The health degree indicates deterioration of the
facility on a continuous condition scale of 0 to 100 such as the Pavement Condition Index (Madanat et al., 1995).
It is converted for initial value of health degree corresponding to each discrete condition state of inspection result
by sensitive of experienced inspectors. The second method is the formulation by state probability and transition
probability of Markov stochastic process. The state probability is defined the existence probability for eachcondition state of the facilities in a tunnel. The transition probability is defined the moving probability of condition
state from one before time to present time inspection of facilities in a tunnel. The state probability is solved bymaster-equation based on Markov stochastic process. The master-equation is consisted in difference probability
flux of macro-statistical ensemble from time-based transition probability and state probability. The health degree
is given to multiply a weight as initial value of health degree to state probability solved by master-equation.
Therefore, the model enables to predict deterioration of the tunnel lighting fixture in the unit of not individual
lighting fixtures but a tunnel. The model is called simplified dynamic macro model from comprehensible method
for persons concerned with AMS (Maeda & Kawamura, 2015).
In this study, the applicability and validity of the model is shown for numerical examples by adapting thehealth degree for the existing tunnel lighting facilities. Then, it is shown how the model functions as a role of
PDCA four steps in AMS.

162

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2. TUNNEL LIGHTING FACILITY
2.1 Function
A tunnel lighting facility has two functions. The first is to solve of light and dark adaption for reducing
visual difference between inside and outside of a tunnel. The second is to provide stable visual brightness to drive
safety in a tunnel. The structure consists of lighting-fixture, lamp, ballast, mounting brackets, anchor-bolts etc. asshown in Figure 1. These materials are currently made of stainless steel, but the past materials were made of steel.
In a tunnel, the material of metal is to easily rust for reasons of adhering road surface anti-freezing agent and oftunnel spring water. Thus, the metal deterioration tends to remarkably progress corrosion and rusting with the age
on the facilities in tunnel. Therefore, it is important for preventing deterioration to plan and perform preventive
repair and replace based on period inspection.

Figure 1. A tunnel lighting facility
2.2

Inspection
Generally, there are three types of inspection on the facility. The first is a daily inspection to evaluate acondition of facility by inspector's human senses. The second is a detail inspection to evaluate function of facilityby the measuring instruments. The third is a periodical inspection that periodically evaluates structural
deterioration by inspector's human senses. In this study, periodical inspection is adopted to manage to prediction
and repair using structure condition on the facility. The method of periodical inspection is to observe thedeterioration state with visual and tentacle of inspectors. The frequency of inspection is carried out every several
years in each of a division tunnel. The example of performing the inspection in same tunnel for three years isshown in Figure 2. The inspection parts are a lump-body, fixtures, fasteners, anchor bolts, hinges and latches, andlump-cover. The state classification of inspection result is evaluated in four classes for each inspection part asshown in Table 1. The evaluation for a facility unit is adopted the lowest evaluation of each part in view of safety
as shown Figure 3. Thus, the condition state of inspection result is a discrete evaluation. Therefore, it is necessary
for performing effectively AMS to convert discrete evaluation to continuous value such as health degree.
State classification
D

C
B
A

Table 1. Criteria for state classification of periodic inspection
Criteria
There are serious structural deficiencies due to deterioration and/or damage.
The element should be immediate repair.
There are some structural deficiencies due to deterioration and/or damage.
The element should be repaired.
There are no serious structural deficiencies due to deterioration and/or damage.
The element has no problem.

Figure 2. Inspection cycle

163Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

total evaluation of facility

facility

evaluation of component parts

fixtureb

attachmentcanchor boltbconcreteaC

Figure 3. Relation between component evaluation and total evaluation of facility
3. MODEL FORMULATION
In general, Markov chain model is traditional model of deterioration prediction model for the reason foreasy to aggregate data on the model (Shin & Madanat, 2003; Yina et al., 2016; Mark et al., 1992). After, it has
been proposed to Markov chain hazard models (Aoki et al., 2005; Tsuda et al., 2006) and Weibull hazard model
as considering the characteristic member for parts of the facility (Aoki et al., 2005; Qing et al., 2014). However,these models have two problems. The first is complication for calculation process of these prediction models. The
second is rather difficult to calculate in case of the small amount of data. Therefore, in case of tunnel lighting
facility, it is proposed a macro model calculating the deterioration prediction easily as a tunnel unit not individual
lighting fixture. This model is called simplified dynamic macro model as a time-based dynamic Markov chain
model having master-equation. In detail, each condition state is determined by solving master-equation which havethe difference probability flux of time-based probability consisted in state probability Pi of state i and transition
probability from state i to state j. In this study, it is predicted health degree as continuous value of deterioration fora tunnel unit using the simplified dynamic macro model from discrete inspection result.
This model procedure consists of five steps as shown Figure 4.

Figure 4. Flow-chart of procedure for simplified dynamic macro model164

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The first step is to decided health degree of a continuous score from 0 to 100 corresponding to eachcondition state of discrete inspection result. In this study, it is adopted a method of hearing to ten experienced
inspectors about health degree for each condition state of four classes shown Table 1. For the result from themethod, the health degree corresponding to each condition state is shown in Table 2.
Table 2. Health degree corresponding to condition state

Condition state

A

B

C

D

health degree (%)

100

50

30

10

The second step is to decide patterns of transition probability. The patterns of transition probability arebased on the following two conditions.
・holding state (transition from condition state i to condition state j)
Pi → i (i = j : condition state j is equal to condition state i)・adjoining transition (transition from condition state i to adjoining condition state j)
Pi → j (i≠j : condition state j is not equal to condition state i)
All the patterns of transition probability are shown in Table 3.
Table 3. Patterns of transition probability
initial state

final statetransition probability

i

j

Pi→j

1

A

A

PA→A

2

A

B

PA→B

3

B

B

PB→B

4

B

C

PB→C

5

C

C

PC →C

6

C

D

PC →D

pattern

The third step is calculation of transition probability from inspection result of one before time and present
time. Let Ai and Bi denote a quantity of lighting fixture of initial state (i.e. before transition) and final state (i.e.
after transition) for state i in a same tunnel. Transition probability Pi → j and Pi → i are shown as follows:
𝑃 → = |(𝐴 − 𝐵 )|⁄𝐵
(1)
(2)

𝑃→ = 1 − 𝑃→

Therefore, Transition probability matrix P which consists of the elements of both Pi → i and Pi → j, iswritten as follows:

𝑃
0
𝑷=
0
0

𝑃
𝑃
0
0

0
𝑃
𝑃
0

0
0
𝑃
𝑃

(3)

Where Pi i = Pi →i, Pi j = Pi → j, i = A, B, C, D.
The forth step is calculation of state probability by the master-equation. The master-equation is consisted
in difference probability flux of time-based transition probability and state probability. When let Pi denote state
probability on condition state i at time t, the master-equation for Pi is expressed as follows:𝑑𝑃(𝑃 ∙ 𝑃 → ) −(𝑃 ∙ 𝑃 → )
=
𝑑𝑡
,

,

(𝑘 ≠ 𝑚)(4)

Where Pi and Pj are state probability of condition state i and j, respectively. Pi → j is transition probability
from condition state i to condition state j.
Therefore, each master-equations for state probability Pi, i = A, B, C, D are written as follows:165

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019𝑑𝑃 ⁄𝑑𝑡 = −𝑃 ∙ 𝑃 →(5)

𝑑𝑃 ⁄𝑑𝑡 = 𝑃 ∙ 𝑃 → − 𝑃 ∙ 𝑃 →(6)

𝑑𝑃 ⁄𝑑𝑡 = 𝑃 ∙ 𝑃 → − 𝑃 ∙ 𝑃 →(7)

𝑑𝑃 ⁄𝑑𝑡 = 𝑃 ∙ 𝑃 →(8)

The state probability Pi, i = A, B, C, D is obtained by the solving from Equation (5) to Equation (8).
The fifth step is calculation of health degree for a tunnel unit using state probability Pi and an initial healthdegree of corresponding to condition state i decided the first step. Let Wi denote an initial health degree on
condition state i shown Table 2 as a weight. When let Hi denote the health degree of a tunnel unit on condition
state i, the health degree Hi, i = A, B, C, D is expressed using Wi and Pi as follow:
𝐻 = 𝑊 ∙𝑃
(9)
Thus, when let H denote the health degree of tunnel unit, H is written from Equation (9) as follows:𝐻 =𝐻(i = A, B, C, D)(10)

4. TRANSITION PROBABILITY CALCULATION AND ESTIMATION RESULTS
4.1 Transition Probability Calculation
In this study, the method described from step 1 to step 5 in Chapter 3 applies to tunnel lighting facilitiesin 59 existing tunnels. In detail, the facilities are 27 tunnels made of steel and 32 tunnels made of stainless steel,for each material.
The transition probability Pi → j made of steel and stainless steel respectively calculated from procedureof step 2 in Chapter 3 are shown in Table 4. In steel, the characteristic is that PB → C and PC → C are very higher
than another transition probabilities. Thus, in the condition state B, it tends to make a transition to the adjoining
condition state C for a reason of progressing very speedy the deterioration for steel. In the condition state C, it
tends to hold the same condition state C for a reason of repairing for the deterioration parts as corrosion and rust.
On the other hand, in stainless steel, the characteristic is that PA → A, PB → B, and PC → C are very higher thananother transition probabilities. Thus, in the condition state A, B, and C, it tends to hold the same condition. The
reason of condition state A and B is considered of progressing slowly the deterioration for stainless steel. The
reason of condition state C is indicated of repairing for the deterioration as same case of steel.
Therefore, in the transition probability, it is indicated the condition state B have a great difference for steel
and stainless steel.
Table 4. Transition probability for patterns

4.2

Estimation Results
The state probability Pi is calculated by using from Equation (5) to Equation (8). The distribution diagramof Pi, i = A, B, C, D is shown in Figure 5 and Figure 6.
In steel, it is predicted that PC reaches 70% of the whole five years later. Further, ten years later, it ispredicted that PD reaches 20% of the whole. In this time, PC and PD are above 95% of whole.
On the other hand, in stainless steel, it is predicted that PC reaches 60% of the whole seventeen years later.
Further, twenty-five years later, it is predicted that PC and PD are about 90% of the whole.
Thus, the deterioration of steel is indicated to progress very speedy for stainless steel.

166

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 5. Distribution diagram of state probability for steel

Figure 6. Distribution diagram of state probability for stainless steel
Finally, this study calculates the health degree with Equation (9) and Equation (10). The time-based
function of health degree called health degree curve is shown in Figure 7. There is a difference for progress speedof the deterioration the steel and stainless steel.
It is predicted to arrive at the condition state C in eight years later of steel and twenty-four years later ofstainless steel. The difference of deterioration progress depends on the transition probability of steel and stainless.
The transition speed from condition state B to C is 4.5 times speedy of steel than of stainless. The difference causes
the difference of sixteen years between steel and stainless steel. The predicted age which reach health degree of
20% and 30% is shown in Table 5.

167

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 7. Health degree curve for steel and stainless steel
Table 5. Prediction age for health degree for steel and stainless steel

In steel, it is predicted that the repair occurs for sixteen years from eight years of the health degree 30%
to twenty-four years of the health degree 20%. On the other hand, in stainless steel, it is predicted that the repair
occurs for twenty-one years from twenty-four years to forty-five years. The results are indicated to progress thedeterioration slowly after reaching health degree of 30% corresponding to the condition state C. The total cost of
repair after reached health degree of 30% will increase with time passes. Then, the health degree is forecasted todecent more slowly and to reach health degree of 10% corresponding to the condition state D.
Therefore, it is desirable to decide the optimum time of replace for the tunnel unit by considering the effectand cost of the repair for the time from health degree 30% to 20%.
5. CONCLUSIONSIn this study, it is predicted the health degree of tunnel lighting facilities by using the simplified dynamic
macro model. It is indicated the difference of progressing the deterioration from transition probability of steel andstainless steel. These results are verified for the model to predict the deterioration of the health degree converted
from visual inspection. These are contributed to decide of a time to replace the facilities for a tunnel unit.
This model products the future health degree of output data from the inspection result of input data. The
information of health degree and the inspection result is shared by all members concerning with AMS. This
information sharing is expected to perform each step of PDCA more effectively for all members on AMS(Wittenbaum et al., 2004). The decision support is created by the discussion with the information sharing basedon each member’s experience of having performed PDCA of each step effectively. And then, the manager decides
an optimum time of replace for the facilities based on the substance of the decision support from the discussion.
The new management style of the decision making from the discussion to the decision support with sharing thesame information is expected to contribute to development for AMS.
Therefore, the model is contributed to perform each step effectively in PDCA cycle on AMS. On the new
management style, the model is indirectly contributed to make decision such as a time of replace for facilities asabove mentioned. The model is possible to apply similarly to other infrastructure treating as macro-statisticalensemble for a purpose of widely utilization on AMS.

168

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

REFERENCES
Miyagawa, T., Yasuda, K., Iwaki, I., Yokota, H., and Hattori, A. (2008). Asset management for civil engineersFrom a concrete structures viewpoint-, JSCE Journal F, .64(1), 24-43.
Madanat, S., Mishalani, R., and Wan Ibrahim, W. H. (1995). Estimation of infrastructure transition probabilities
from condition rating data, J. Infrastructure sys., ASEC, 1(2), 120-125.
Maeda, N. and Kawamura, K. (2015). Prediction of health degree for tunnel lighting facilities using simplified
dynamic macro model, JSCE Journal F4, 71(1), 19-32.
Shin, H. and Madanat, S. (2003). Development of a stochastic model of pavement distress initiation, J.
Infrastructure Plan. and Man., JSEC, 744/Ⅳ(61), 61-67.
Yina, F. M., Alexander, P., Hanns, D. L. F., Joaquin, V. F., and Guilherme, M. S. (2016). Estimating Bridge
Deterioration for Small Data Sets Using Regression and Markov Models, World Academy of Science,
Engineering and Technology International Journal of Urban and Civil Engineering, 10(5), 663-670.
Mark, A. C., Carlos, S., Carl, T., and Erik, H.V. (1992). Modeling Bridge Deterioration with Markov Chains,
Journal of Transportation Engineering, 118(6), 820-833.
Aoki, K., Yamamoto, K., and Kobayashi, K. (2005). Estimating hazard models for deterioration forecasting, JSCE
Journal, 791/ VI(67), 111-124.
Tsuda, Y., Kaito, K., Aoki, K., and Kobayashi, K. (2006). Estimating Markovian transition probabilities for bridge
deterioration forecasting, Structural Eng./Earthquake Eng., JSCE, 23(2), 241s-256s.
Aoki, K., Yamamoto, K., Tsuda, Y., and Kobayashi, K. (2005). A deterioration forecasting model with multistaged Weibull hazard functions, JSCE Journal, 798/ VI(68), 125-136.
Qing, Z., Cheng, H., and Guanghua, X. (2014). A mixture Weibull proportional hazard model for mechanicalsystem failure prediction utilising lifetime and monitoring data, Mechanical Systems and Signal
Processing, 43,103-112.
Wittenbaum, G. M., Hollingshead, A. B., and Botero, I. C. (2004). From cooperative to motivated information
sharing in groups; Moving beyond the hidden profile paradigm, Communication Monographs, 71(3), 286310.

169

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

VRP-BASED MODEL FOR LANE MARKING ASSESSMENT WITH
MRU VEHICLE
Yu-Chun Lin1, Si-Ting Liao2, Chieh (Ross) Wang3, and Albert Y. Chen4
1) Master Student, Department of Civil Engineering, National Taiwan University, Taipei, Taiwan. Email:r06521515@ntu.edu.tw
2) Master Student, Department of Civil Engineering, National Taiwan University, Taipei, Taiwan. Email:r07521501@ntu.edu.tw3) R&D Staff, Energy and Transportation Science Division, Oak Ridge National Laboratory, Knoxville, TN, USA. Email:cwang@ornl.gov
4) Associate Professor, Department of Civil Engineering, National Taiwan University, Taipei, Taiwan. Email:albertchen@ntu.edu.tw

Abstract: Evaluation of lane marking conditions has advanced from visual and/or manual inspections, which
can be unsafe and time-consuming, to mobile assessments using vehicle-mounted devices that allow transportation
agencies to collect retroreflectivity data at a large scale in a safer and efficient manner. However, cost-effectively
routing and operating these mobile retroreflectivity units (MRUs) at a large scale can be a unique challenge. This
study proposes a vehicle routing problem (VRP) based model to optimize the routing of MRUs. The model takes
into account the additional costs of daily and weekly operations of MRUs, such as remounting the device andtraveling between tasks.
Keywords: vehicle routing problem (VRP); pavement markings; mobile retroreflectivity unit (MRU)
1. INTRODUCTION
Lane markings are an important type of traffic control devices that is crucial to roadway safety. Periodical
assessments of lane marking conditions, e.g., visibility and retroreflectivity, ensure the proper function andperformance of lane markings. Traditionally, methods for lane marking assessment involve visual surveys andmanual data collection using handheld devices, which can be limited in scale, time-consuming, and unsafe. Recent
developments of the Mobile Retroreflectivity Unit (MRU) – a laser-based equipment that is mounted on a vehicle
capable of measuring pavement marking retroreflectivity at highway speeds – have provided opportunities forstate and local transportation agencies to collect pavement marking retroreflectivity data in a large scale. State
departments of transportation (DOTs), such as the Florida DOT, have started to conduct state-wide pavement
marking condition evaluation using MRUs (Choubane, Sevearance, Holzschuher, Fletcher, & Wang, 2018).
Usually, state DOTs rely on field engineers and technicians to artificially plan the routes and carry out thetesting. However, collection of state-wide pavement marking data on all state-maintained highways on an annual
basis under factors (e.g., fog, traffic, wet weather, hurricanes, etc.) that can significantly affect testing schedules
can be challenging. Therefore, there is a need to minimize the operational cost, in terms of time spent and distancetraveled, of the data collection schedule. One way of minimizing the effort is to optimize the daily and weekly
routing plans.
Each MRU testing task can be considered as a node of a network and the distance between nodes denotes alink of the network. Vehicle routing problem (VRP) models can be used for solving the sequence of the nodes
being visited and the links being traveled on, which then ensures the optimal solution of the objective under various
constraints. If there is a complete network, we can form the VRP model and the testing schedule can be optimized.
The purpose of this study is to formulate an integer program (IP) for optimization of pavement marking assessment
scheduling and provide a structured way to generate an efficient assessment plan.
2. LITRATURE REVIEW
2.1 Lane Marking Assessment and Mobile Retroreflectivity Unit (MRU)
Pavement markings are important for road users and are key contributors to roadway guidance and traffic
safety. Especially at night, the visibility of pavement markings is most critical. To ensure pavement markings areadequately maintained, quantified measurements are to be provided for monitoring (Choubane, Sevearance,
1

This manuscript has been authored in part by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the
US Department of Energy (DOE). The US government retains and the publisher, by accepting the article forpublication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license
to publish or reproduce the published form of this manuscript, or allow others to do so, for US government
purposes. DOE will provide public access to these results of federally sponsored research in accordance with the
DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

170

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Holzschuher, Fletcher, & Wang, 2018). Early tools for performance assessment of pavement markings are throughhandheld devices and visual inspections. However, use of such tools is dangerous due to cumbersome traffic(Holzschuher et al., 2010). An instrument mounted on a vehicle driving in usual traffic can conquer those shortages.
The Mobile Retroreflectivity Unit (MRU) is a vehicle mounted with a mobile retroreflectometer capable ofmeasuring pavement marking retroreflectivity at highway speed. The safety and efficiency of MRUs have been
considered by more and more highway agencies as an attractive alternative to the old tools.
Mounting equipment on the vehicle implies there is a configuration and specific procedure. Figure 1.
demonstrates three types of lane markings: type 1 is a center line, type 2 is a skip line and type 3 is an edge line.
In order to measure these different pavement markings (such as yellow center lines, white skip lines, and whiteedge lines), the MRU must be set on the specific side of the vehicle. For example, the retroreflectometermust be mounted on the right side of the vehicle if the testing lane is an edge line. Calibration may impact the
precision of measurements. Choubane et al. (2013) pointed out that MRUs are fairly sensitive instrument and ifthey are not properly calibrated, accuracy and precision of the tests will be affected. Therefore, in addition to theneed of proper and routine calibrations, properly mounting and remounting the MRU is crucial.

Figure 1. Types of lane marking.
2.2 Vehicle Routing Problem
The Vehicle Routing Problem (VRP) is an extension of the Traveling Salesman Problem (TSP) (Dantzig &Ramser, 2008). The VRP seeks best route that services a number of customers with a fleet of vehicles from depots.
The objective of the VRP can be minimizing transportation costs (traveling time or distance). Kulkarni &Bhave(1985) purposed requirements of the VRP model, including (1) model formulation needs to meet all customers’
demands; (2) there are certain restrictions on the number of consumers/nodes assigned to each vehicle; and (3) the
total travel cost per vehicle and the service capacity cannot exceed the upper limit of each vehicle. Moreover, there
are two assumptions in the VRP. First, the maximum demand at a location 𝑖 is less than the minimum capacity ofa vehicle, and the second is whenever a customer i is serviced, the demand at i is fulfilled.
A basic VRP restricts the relation between vehicles, nodes and arcs. Constraints in VRP must: (1) ensure avehicle only visits and leaves one customer once; (2) if a vehicle visits one customer, it must leave the same
customer; (3) constraints must guarantee all routes start and end at the depot; (4) capacity limit and maximum costlimit of each vehicle are imposed by inequalities; and (5) subtours should be eliminated.
The complexity of reality brings about extensions of VRP in academic research and practical applications.
For example, there are VRPs with time windows (VRPTW), fleet size and mix VRPs (FSVRP), VRPs withmultiple use of vehicle (VRPM), and many more variations. When solving VRPs in a larger scale a solution
algorithm is usually needed. Gracia et al. apply VRP to address the biomass collection problem (Gracia et al.,2014). Since the various machines interact with each other during the harvesting process, the path and work orderof the different machines becomes important. Meta heuristics such as the Genetic algorithms (GA) and local searchmethods are used to obtain solutions (Zhang et al., 2008).
2.3 Summary
Using vehicles equipped with detection tools to improve safety and efficiency of assessment tasks is thetrend of future maintenance procedures. The VRP has been extended to handle many similar applications. The
relationship of tasks in lane marking assessment can be described through a network of nodes and links, and webelieve planning of the maintenance schedule is suitable to be solved by a VRP model.
3. METHODOLOGYThis section describes the formulation of a VRP-based model for routing of MRU mounted vehicles
conducting lane marking assessments. The objective contains the operational cost, reconfiguration cost and travelcost. The goal of the model is to provide a system-wise assessment schedule.

171

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3.1 Model Formulation

𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒: 𝛼 * 𝑌, + 𝛽 * * 𝑍,2 + 𝛾 * * * 𝑆𝑇7,2 + 𝛿𝑇𝑑,∈.

2∈3 ,∈.

2∈3 ,∈. 7∈8(1)
Subject to* * * * * 𝑋<=7,2> = 1∀𝑑 ∈ 𝑁>∈D 2∈3 ,∈. 7∈8 <∈?∪A(2)* * * * * 𝑋<=7,2> = 1∀𝑜 ∈ 𝑁>∈D 2∈3 ,∈. 7∈8 =∈?∪A(3)* * * 𝑋<H7,2> = * * * 𝑋H=7,2> ∀n ∈ N ∪ H, v ∈ V, t ∈ T>∈D 2∈3 <∈?∪A>∈D 2∈3 =∈I∪A(4)* * * 𝑋R=7,2> ≤ 1 ∀ℎ ∈ 𝐻, 𝑣 ∈ 𝑉, 𝑡 ∈ 𝑇>∈D 2∈3 =∈I(5)* * * 𝑋<R7,2> ≤ 1 ∀ℎ ∈ 𝐻, 𝑣 ∈ 𝑉, t ∈ T>∈D 2∈3 <∈?

(6)𝑢< − 𝑢= + (N + H) ∗ * * * * 𝑋<=7,2> ≤ 𝑁 + 𝐻 − 1 ∀o, d ∈ N, 𝑜 ≠ 𝑑>∈D 2∈3 ,∈. 7∈8(7)𝑌, ≥ 𝑋<=7,2> ∀o ∈ 𝑂, d ∈ 𝐷, v ∈ 𝑉, t ∈ 𝑇, w ∈ 𝑊, s ∈ 𝑆(8)𝑍,2 ≥ 𝑋<=7,2> ∀o ∈ 𝑂, d ∈ 𝐷, v ∈ 𝑉, t ∈ 𝑇, w ∈ 𝑊, s ∈ 𝑆
(9)
** *

*

>∈D 2∈3 =∈I∪A <∈?∪A𝐶<=𝑋≤ 𝐹 ∀𝑣 ∈ 𝑉, 𝑡 ∈ 𝑇
𝐵<= <=7,2>
(10)

* *

*

>∈D =∈I∪A <∈?∪A𝐶<=𝑋≤ 𝑓 ∀𝑣 ∈ 𝑉, 𝑡 ∈ 𝑇, 𝑤 ∈ 𝑊
𝐵<= <=7,2>(11)

𝑆𝑁7,2H> ≥* 𝑋<H7,2>l + * 𝑋H=7,2> − 1 ∀𝑛 ∈ 𝑁, 𝑣 ∈ 𝑉, t ∈ T, w ∈ 𝑊, 𝑠, 𝑠 n ∈ 𝑆, 𝑠′ ≠ 𝑠
<∈?∪A

=∈I∪A

(12)

172

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

𝑆𝑇7p2 ≥ * * 𝑆𝑁7p2H> ∀𝑣 ∈ 𝑉, t ∈ T, w ∈ W>∈D H∈q(13)
𝑇𝑑 ≥ * * * * ** 𝐶<= 𝑋<=7,2>>∈D 2∈3 ,∈. 7∈8 =∈I∪A <∈?∪A

(14)
*

*

* 𝑋H=7,2> ≥ * * 𝑋<H7,s> ∀n ∈ 𝑁, 𝑣 ∈ 𝑉, t ∈ 𝑇, 𝑤 ∈ 𝑊>∈D 2∈stuvw =∈I∪A>∈D <∈?∪A(15)
𝑌,xy ≤ 𝑌, ∀t ∈ 𝑇 − 1(16)
𝑍,(2ty) ≤ 𝑍,2 ∀t ∈ 𝑇, w ∈ 𝑊 − 1(17)Operating cost, configuration cost and travel distance are considered in the model, the model use weight toevaluate the performance of assessment schedule. Operating cost in this model means the total time cost forfinishing all tasks. Configuration cost can be defined as time unit, or penalty of change equipment side. Travel
distance is the sum of all travel distance of used link.
The formulation extends the basic VRP model. Eq. (2)-(7) are the basic VRP constraints. Test vehicles ineach period are ensured arrive and leave each node (expect depot node) once in Eq. (2) and (3). Eq. (4) indicatesflow conservation. Each vehicle only can leave home node less than once in Eq. (5). Based on Eq. (4), Eq. (6)indicates the vehicle must back to depot node if it has set out from depot node. Eq. (7) is subtour elimination
constraints.
This formulation considers practical conditions, including the test vehicles may need maintenanceperiodically, the labors won’t drive vehicle on their day off, and the test vehicles are requested to return home nodeperiodically. The formulation uses operating hours in a period to satisfy this request in Eq. (10). And Eq. (11) is
another inequality for operating hours per workday. Eq. (8) and (9) calculate the total operating period and workday,
which are used to measure performance.
Tools installation or reconfiguration increase cost operating time. To avoid the assessment schedule with aninefficient plan, Eq. (12) indicates whether labors change configuration types in the node (i.e., the vehicle entersnode with configuration type 1 but leaves node with configuration type 2, then value of RHS in Eq. (12) is 1,
therefore, variable value of LHS will more than 1). Eq. (13) calculates the total times of reconfiguration. In practice,labors prefer not to change configuration type in a workday, and this formulation can use penalty to ensure same
type of equipment.
Eq. (14) sum the total travel distance and the LHS variable is for readability. The workdays in a period havetime sequence, Eq. (15) determines vehicles won’t violate workday sequence, for example, Eq. (15) means if thevehicle enters the node in workday w, it can’t visit remining nodes in the day before workday w of same period
anymore.
Eq. (16) and Eq. (17) express time. Eq. (16) indicates if the previous period doesn’t operate, then thefollowing period can’t operate. Eq. (17) has the same concept but replace period into workday. These twoconstraints won’t change objective value of solution. However, it can reduce execution time when the capacity
exceeds demand (ex. All task need one periods for operating but there is a large period set.)
3.2 Notation(1) Variables
𝑋<=7,2> is binary variable that indicating whether link of node o to node d uses vehicle v in period t,•
workday w with tool installed in side s is functional or not. 1 means link is used and 0 otherwise.
𝑌, is binary variable representing period t is used or not.
•
•
•
•
•

𝑍,2 is binary variable representing period t, workday w is used or not.
𝑈H is variable for subtour elimination.
𝑆𝑁7,2H> is binary variable indicating whether vehicle v operating in period t, workday w changes
installation type of test tool from other type to type s or not. 1 means change side and 0 otherwise.
𝑆𝑇7,2 is variable indicating reconfiguration times of vehicle v in period t, workday w.

173

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019•
𝑇𝑑 is variable describing total travel distance.
•
All variables above are positive.
(2) Sets
𝑁 is set of nodes expect depot, and O, D are same meaning but for readability.
•
𝐻 is set of depot nodes•
𝑉 is set of vehicle fleet•
𝑇 is set of periods•
𝑊 is set of workdays•
𝑆 is set of configuration types.
•(3) Parameters
𝐶<= is travel distance from node o to node d•
𝐵<= is speed limit of node o to node d•
F is allowed working hours per period•
f is allowed working hours per working day•𝛼, 𝛽, 𝛾, 𝛿 are constant of variables in objective function•
4. CASE STUDY
In this section, the proposed methodology is applied to the Florida Department of Transportation (FDOT)’s
MRU program. The program has a yearly schedule and there are inventory lane markings needed to be measured
every year. The tasks of lane marking section are located throughout the entire Florida state.
For the implementation of the formulation, Python is chosen as the programming language and Gurobi isutilized as the solver. The analysis is executed on a computer with 8GB 2,133 MHz LPDDR3 memory and a 2.3
GHz Intel Core i5 CPU.
4.1 Nodes and Arcs
To enable a VRP-based model, a network has to be available. In this work, the nodes of network aretransformed from tasks of the MRU program. Each task represents a trip needed to be measured, and each task
contains an origin mile point and a destination mile point. The origin point and destination point are different nodes
in the network. More clearly, there are three types of arcs:(1) Arcs between each task (blue dotted arrow in figure 2-c): each task are divided into two nodes. One isthe origin node and the other is the destination node. An origin node only links to a destination node of thesame task.
(2) Arcs between different tasks (blue solid arrow in figure 2-c): destination node of a task only can direct
to origin nodes of other tasks or the depot node.
(3) Arcs between tasks and depot (black solid arrow in figure 2-c): vehicle can move to origin nodes of alltasks from the depot and only comes back to the depot from a destination node.
Nodes will be visited according to arcs, which implies through the manipulation of arcs value, nodes of sametask are guaranteed to be visited in turn. Figure 2-a, 2-b, 2-c demonstrate design of the network.

Figure 2-a.

Figure 2-b.

Figure 2-c.

Travel distance is used for arcs. There are two types of travel distance in this case. One is distance between
different tasks, which is extracted from the Open Street Routing Machine (OSRM) API. The OSRM runs on
OpenStreetMap data and provides the environment and API to request the distance between locations. The otheris distance between nodes of same task, which is extracted from the starting and ending mile points of each task.
4.2 Sets of MRU program
In the MRU program, there are two MRU vehicles for testing retroreflectivity of lane markings. Drivers
depart from the depot on Monday and come back to the depot on Thursday, and all measure tasks need to befinished in 52 weeks. In correspondence with the model formulation, there are 52 periods and 4 workdays in each174

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

period in the MRU program. There are two types of configuration of MRUs. The retroreflectivity is measured by
MRU vehicle in this case and tasks of lane marking are to be conducted with the MRU mounted on different sides.
There will be a reconfiguration cost if the tasks need different types of equipment configuration.
4.3 Results
Tasks in Gilchrist county is conducted to verify the correctness of the model. Different types of tasks are
selected in both cases. The result displays fleets are dispatched to reduce the operation period and workday.
Table1: Results
Case
Tasks
Constants(𝛼 | 𝛽 | 𝛾 | 𝛿)Operating cost
Reconfiguration
cost
Travel distance (m)Execution time

1
6 tasks in Gilchrist county
600,000|100,000|50,000|1
1 vehicle operate in 1 week with 1 workday
1 time of reconfiguration
306,5386.16
85.46sRoute

5. CONCLUSION
In this study, a VRP-based model is proposed for lane marking assessment with vehicles with MRUs. The
network is composed of lane marking tasks and links considering travel distance. The model considers operating
cost, including operational period and workdays, reconfiguration cost (i.e. mounting of equipment from one sideof the vehicle to another in a workday), and total travel distance. The model proposed can be a generalization toconsider many different cost combinations, because the aforementioned costs weights that are adjustable in theobjective function of our proposed model.
ACKNOWLEDGMENT
The authors would like to thank Bouzid Choubane, Charles Holzschuher, and Eddie Offei of the State Materials
Office of the Florida Department of Transportation for providing data and input. In addition, the authors thank the
Ministry of Science and Technology for the support of grant MOST 107-2119-M-002-018 that has made this work
possible.
REFERENCE

175

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Choubane, B., Sevearance, J., Holzschuher, C., Fletcher, J., & Wang, C. (Ross). (2018). Development and
Implementation of a Pavement Marking Management System in Florida. Transportation Research Record.
https://doi.org/10.1177/0361198118787081
Choubane, B., Sevearance, J., Lee, H. S., Upshaw, P., & Fletcher, J. (2013). Repeatability and Reproducibility ofMobile Retroreflectivity Units for Measurement of Pavement Markings. Transportation Research Record:
Journal of the Transportation Research Board. https://doi.org/10.3141/2337-10
Dantzig, G. B., & Ramser, J. H. (2008). The Truck Dispatching Problem. Management Science.
https://doi.org/10.1287/mnsc.6.1.80
Gracia, C., Velázquez-Martí, B., & Estornell, J. (2014). An application of the vehicle routing problem tobiomass transportation. Biosystems Engineering. https://doi.org/10.1016/j.biosystemseng.2014.06.009
Holzschuher, C., Choubane, B., Fletcher, J., Sevearance, J., & Lee, H. S. (2010). Repeatability of Mobile
Retroreflectometer Unit for Measurement of Pavement Markings. Transportation Research Record:
Journal of the Transportation Research Board. https://doi.org/10.3141/2169-11
Kulkarni, R. V., & Bhave, P. R. (1985). Integer programming formulations of vehicle routing problems.
European Journal of Operational Research. https://doi.org/10.1016/0377-2217(85)90284-X
Yueqin Zhang, Shiying Chen, Jinfeng Liu & Fu Duan. (2008). The application of genetic algorithm in vehicle
routing problem. Proceedings of the International Symposium on Electronic Commerce and Security,
ISECS 2008, 0(3), 3-6

176

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

THE GATEWAY TO INTEGRATING USER BEHAVIOR DATA IN “COGNITIVEFACILITY MANAGEMENT”
Jinying Xu1, Weisheng Lu2, and Jing Wang3
1) Ph.D. Candidate, Department of Real Estate and Construction, The University of Hong Kong, Hong Kong. Email:jinyingxu@connect.hku.hk
2) Ph.D., Assoc. Prof., Department of Real Estate and Construction, The University of Hong Kong, Hong Kong. Email:wilsonlu@hku.hk3) Ph.D. Candidate, Department of Real Estate and Construction, The University of Hong Kong, Hong Kong. Email:jingww@connect.hku.hk

Abstract: In the face of current predicaments of facility management (FM), the concept of cognitive FM is
proposed with a view to providing active intelligent management of a facility. In order to achieve such cognitiveFM, how to integrate user behavior data into a cognitive FM system has to be solved. This paper serves as method
guidance for it by putting forward the idea that location can serve as a gateway for the integration. Ultra-wideband(UWB) is recommended as the device layer to construct the 3D local positioning system for the cognitive FM
system after comparison between different local positioning technologies from the accuracy, scalability, and costdimensions. The way to bridge the user behavior data with facilities through coordinate transformation andlocation/distance computation is briefly introduced. Such of a uniform 3D coordinate system with high accuracy
and scalability for FM situation can provide a common language for communication and computational
applications. Finally, application scenarios for various facilities such as commercial building, office building,hospitals, warehouses, airports, and transportation stations are discussed.
Keywords:
UWB.

Facility management, cognitive system, data integration, user behavior, local positioning system,

1. INTRODUCTION
Cognitive facility management (FM) is defined as the active intelligent management of a facility, which
can perceive through cognitive systems, learn in the manner of human cognition with the power of cognitive
computing, and act actively, adaptively, and efficiently via automated actuators, to improve the quality of people’s
life and productivity of core business (Xu et al., 2019). The proposition of cognitive FM is to shift the current
predicament that a facility fails to provide satisfactory services to people, organizations, and businesses (Wang etal., 2018). The tight spot is caused by the passiveness of current FM systems which cannot meet the differentiatedand changing requirements of users in a facility. Various passive FM systems are pre-programmed, which cannot
respond to complicated, flexible, changing situations in real life. FM needs to be updated with intelligence to ahigher level akin to human beings’ cognitive capability (e.g., to perceive, to learn, and to act). Cognitive FM is acyber-physical-social system (CPSS) where cyber (e.g., facility model, computer-aided FM system), physical (e.g.,furniture, air conditioning system), and social (e.g., user behavior) information are integrated. The first step forcognitive FM to proactively perceive the requirements of users is to collect user behavior data, with which user
preference and requirements can be learned.
However, user behavior data is hard to collect due to privacy issues and the lack of appropriate maturetechnologies. Cameras are ideal equipment to capture user behavior but limited by the notoriety of privacy
encroachment (Chen et al., 2018), especially for private buildings. Many research and practices have made plenty
efforts to collect user behavior data using portable devices (e.g., smartphones) and wearable devices (e.g., smart
watches) (Lee et al., 2016; Liu & Huang, 2016; Rosenberger et al., 2016; Anjomshoa et al., 2017; Doryab et al.,2018). The lack of GPS information can also constrain it for location-based behavior data since GPS usually cannot
work properly for indoor areas. Therefore, the collection of location-based behavior data at indoor environment
remains a gap to be filled. A further gap is the integration of user data into an FM system. Majority of the FM
systems neglect the consideration of user data (Kang & Choi, 2015). Thus the integration of user behavior datafrom the social aspect and the cyber and physical data in FM is, by and large, an uncharted territory.
This paper aims to develop an appropriate way to collect location-based behavior data at indoor
environment using Ultra Wideband (UWB) technology and furthermore find out a gateway to integrate user
behavior data (social) with the cyber and physical FM information system for further cognitive FM application.
The rest of the paper is organized as follows: Section 2 is a detailed literature review of location-based user
behavior data collection technologies and the user behavior data integration approaches; Section 3 is the
deployment of location-based user behavior data collection system; Section 4 is the gateway development for user
behavior data integration in FM information system. Section 5 is the discussion and conclusion.

177Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2. LITERATURE REVIEW
2.1 Location-based User Behavior Data Collection Technologies
Commercial analysts and consultants have long studied user behavior data collection. Log (Cheng et al.,2017), search (Kim et al., 2015), payment (Liébana-Cabanillas et al., 2018), and click (Wang et al., 2017) behaviors
of web or mobile users are the most common behavior studied among others. Recently, with the rise of emerging
pervasive wearable and smart devices such as smartwatches and smartphones, the behavior data of movement(Vuković et al., 2018), eating (Kalantarian & Sarrafzadeh, 2015), and sleeping (Alfeo et al., 2018) becomes
possible to be collected. The detection of movement is facilitated by the motion sensor, mostly triaxial gyroscopes,
embedded in smartphones or smartwatches (Kalantarian et al., 2015).
Meanwhile, location information is becoming an indispensable part of smart services (Niu et al., 2016).
In current practices, the location information of users is detected by the GPS module, satellite-based positioning
system, in the smart devices. However, a weak point of GPS is that it can’t work properly inside buildings andother places because line-of-sight transmission between receivers and satellites is not possible in an indoor
environment (Gu et al., 2009). The failure of GPS is featured by either the loss of signal or low accuracy, which
weakens the usability of smart devices that are supported by GPS for location-based services.
Facilities, especially buildings, are the common cases where GPS fails, which makes indoor navigation abottleneck problem troubling both business and academia. Many efforts have been made to tackle the issue by
developing indoor positioning systems (Liu et al., 2017). Techniques used for indoor positioning systems include
infrared (Aitenbichler & Muhlhauser, 2003), vision analysis (Kawaji et al., 2010), magnetic signals (Li et al.,2012), audible sound (Mandal et al., 2005), ultrasound (Hazas & Hopper, 2006), Bluetooth (Feldmann et al., 2003),
RFID (radio-frequency identification) (Saab & Nakad, 2010), WLAN (wireless local area network) (Yang & Shao,2015), and UWB (ultra wideband) (Mahfouz et al., 2008). Some research also tried hybrid methods of different
techniques. The comparison of the main features of the technologies is shown in Figure 1 (Maute, 2012; Sakpere
et al., 2017). Among the three main features, accuracy denotes the typical resolution of a positioning technology,
scalability means the capacity a local positioning system (LPS) can be scaled up, and cost is the investment ofdevices. The choice of technology is dependent on the targeted applications. For cognitive FM, the accuracy
required in, pedestrian navigation, product tracking, ambient assisted living applications should be cm-m level,
the scalability of the system should be relatively high, but the cost can be limited. To synthesize these three featuresand requirements of cognitive FM applications, UWB can be an acceptable choice to collect location data.

Figure 1. Comparison of different indoor positioning technologies
2.2 User Behavior Data Integration Approaches
To collect user behavior data is one important thing, to integrate such data for further analysis andapplication is another more significant problem. Current user behavior analysis and modeling are mostly based on
online behavior data, be they social network data, mobile data, or web data. The integration of such data with178

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

customized services can be found at the field of intelligence business, such as search engine optimization (Ghose
et al., 2012), customized recommendations of products and services for online shopping, reading, andentertainment (Besbes et al., 2015).
Some energy use studies also pay much attention to user behavior (Hoes et al., 2009). Such researchmainly investigates the impact of user behaviors (e.g., presence, movement, and use of control system) on building
energy performance by simulation or correlation analyses (Zhang et al., 2018). However, the integration of user
behavior data with location-based services is still unattended. Therefore, the customized comfort and convenience
services for facility users is still on its way. This paper aims to develop the gateway to integrating the location offacility user with user behavior and preference for better facility management such as energy management, security
services, and indoor navigation.
3. UWB SYSTEM FOR LOCATION-BASED USER BEHAVIOR DATA COLLECTION
As discussed at Section 2, UWB is a relatively optimal choice for indoor positioning services regarding
its high accuracy (several to tens centimeters level), high scalability (its signal can research 30 to 50 meters), andmedium cost ($50 per tag which requires low energy). UWB is a mature technology for real-time indoor
positioning system based on trilateration (Sakpere et al., 2017). The advantages of UWB, despite the high accuracy
and scalability, include low power transmission, multi-path fading robustness, ultra-fine time resolution, andmultiple simultaneous transmissions (Chu & Ganz, 2005). However, metallic and liquid materials may cause UWB
signal interference, the use of more UWB anchors, and strategic placement of UWB anchors could overcome thisdisadvantage (Liu et al., 2007). Moreover, UWB was proved to have the ability to support real-time 3D dynamic
indoor positioning (Li & Yang, 2015).
The working mechanism of UWB 3D indoor positioning system is as illustrated in Figure 2. At least fouranchors will be installed, among which three of them are at the same height for the calculation of x and ycoordinates, while the other is at a different height for the calculating of z coordinate. Tags can transmit signals
with the anchors; the anchors can also transmit the received signals sent back by the tags to the datum anchor,
based on the TDOA (time difference of arrival) and trilateration principle, the 3D coordinates of the tags can becalculated.

Figure 2. Mechanism of UWB 3D positioning system
Users equipped with the UWB tags can be tracked at real time with high accuracy. With the accurate realtime location data of users, their spatial-temporal database and behavior patterns (e.g., going to room 724 around
9:30 am every; buying fruits at the supermarket every Saturday morning) can be collected and analyzed. Therefore,
services relating to the location (e.g., controlling of lights, air conditioning system based on occupancy, schedulingelevators, auto parking license, intelligent access control, and indoor navigation) can be provided proactively andefficiently. Besides, when an emergency happens, the accurate location will make precise and in-time notification,
alert, and rescue possible. Additionally, the positions of facilities and assets can also be recorded and traced for179

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

further interaction between users. In a word, the UWB 3D LPS gives visibility and traceability of facilities andusers for cognitive FM.
4. THE GATEWAY OF USER BEHAVIOR DATA INTEGRATION IN COGNITIVE FM SYSTEM
After collecting user behavior data, a gateway is required to integrate such data into the cognitive FM
system for further FM application processing and actuating (Chen et al., 2015). Location is argued in this paper as
the gateway. With every facility and sub-facility having their location, the location of users can be attached to thefacility as a gateway to link the users and the facilities. In the cognitive FM system, the location of sub-facilities
serves as the reference system of applications for users. That is to say, given every facility has its coordinate
information in the cognitive FM system, once users’ locations are tracked, the relative position of users to thefacilities is determined. The coordinate information of facilities could be marked and stored in a facility
information model. The coordinate information of users collected by the UWB LPS can be registered andcomputed in the coordinate system of the facilities by coordinate transformation (Tiemann et al., 2015). The
transformation of a triplet coordinate in the UWB LPS to the facility coordinate system is shown in Equations (1)to (6). Thus, the indoor positioning information of users will be integrated into the cognitive FM system with thefacility coordinaKANGte system as a reference system. Similarly, the facility coordinate system can also betransformed to the universal coordinate system to link up with GPS (Global Positioning System) to ensure theseamless connection between the LPS with GPS.

Figure 3. The gateway between user behavior to cognitive FM applications
𝐷𝑋𝑖,𝑗 = |𝑙𝑥𝑖 − 𝑙𝑥𝑗 |,𝐷𝑌𝑖,𝑗 = |𝑙𝑦𝑖 − 𝑙𝑦𝑗 |,𝐷𝑍𝑖,𝑗 = |𝑙𝑧𝑖 − 𝑙𝑧𝑗 |,𝑜𝑥𝑗 + 𝐷𝑋𝑖,𝑗 , 𝑙𝑥𝑖 > 𝑙𝑥𝑗
𝑓𝑥𝑖 = {,
𝑜𝑥𝑗 − 𝐷𝑋𝑖,𝑗 , 𝑙𝑥𝑖 < 𝑙𝑥𝑗
𝑜𝑦𝑗 + 𝐷𝑌𝑖,𝑗 , 𝑙𝑦𝑖 > 𝑙𝑦𝑗
𝑓𝑦𝑖 = {,
𝑜𝑦𝑗 − 𝐷𝑌𝑖,𝑗 , 𝑙𝑦𝑖 < 𝑙𝑦𝑗
𝑜𝑧𝑗 + 𝐷𝑍𝑖,𝑗 , 𝑙𝑧𝑖 > 𝑙𝑧𝑗
𝑓𝑧𝑖 = {,
𝑜𝑧𝑗 − 𝐷𝑍𝑖,𝑗 , 𝑙𝑧𝑖 < 𝑙𝑧𝑗
Where
i, j
I
DXi,j
DYi,j
DVi,j
lxi
lyi
lzi
fxi
fyi
fzi
oxj

∀𝑖, 𝑗 ∈ {1, 𝐼}, 𝑖 ≠ 𝑗∀𝑖, 𝑗 ∈ {1, 𝐼}, 𝑖 ≠ 𝑗∀𝑖, 𝑗 ∈ {1, 𝐼}, 𝑖 ≠ 𝑗

Indoor point numbers;
The total number of indoor points;
Distance between point i and j in x axis;
Distance between point i and j in y axis;
Distance between point i and j in z axis;
Local coordinate of point i in x axis;
Local coordinate of point i in y axis;
Local coordinate of point i in z axis;
Facility coordinate of point i in x axis;
Facility coordinate of point i in y axis;
Facility coordinate of point i in z axis;
Facility coordinate of original point j in x axis;

180

(1)
(2)
(3)

∀𝑖, 𝑗 ∈ {1, 𝐼}, 𝑖 ≠ 𝑗(4)∀𝑖, 𝑗 ∈ {1, 𝐼}, 𝑖 ≠ 𝑗(5)∀𝑖, 𝑗 ∈ {1, 𝐼}, 𝑖 ≠ 𝑗(6)

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

oyj
ozj

Facility coordinate of original point j in y axis;
Facility coordinate of original point j in z axis.

By attaching UWB tags to users, the indoor location of users will be easily traced. Assigning every tag aunique ID, the user ID and his/her tag ID can be bonded together, which will make the identification andtraceability of the users at fingertips. The UWB LPS itself can record sort of user behaviors such as moving fromplace A (lxa, lya, lza) to place B (lxb, lyb, lzb), staying at place C (lxc, lyc, lzc) for a duration Tc, or waiting at place D(lxd, lyd, lzd) for elevator E at (lxe, lye, lze). Therefore, the user behavior data and the facility data will be bridged
by the location data after coordinate transformation in the cognitive FM system. When computing for cognitive
FM applications, the path of the users, their accurate location in the facility, their distance from facilities and assets
can be used as essential parameters in algorithms. For example, the smart access control application will need thelocation data and ID of the user to match with the access authentication database for auto-control of entrances ofrestricted areas when authorized user is at the entrance. To sum up, the uniformed system with every point has its
unique 3D coordinate has the following advantages:(1) The triplets of points are more precise than GPS information and therefore more suitable for FM uses.
Moreover, it is contextualized in a specific facility than the GPS to provide a more situated understanding.
(2) The 3D coordinate system provides a uniform positioning methodology that is scalable across thefacility rather than focusing on a smaller area such as a corridor or a room. Very often, cognitive FM applications(e.g., targeted safety alerts when an emergency happens) should coordinate different facilities rather than a smaller
part only.
(3) When communicating something (e.g., positions of the facility entrance or an elevator, or users) in thefacility, the cognitive FM system can have a uniform coordinate system to which they can refer. The chance of
miscommunication can be significantly reduced.
(4) When developing AI or robotics to perform cognitive FM applications and services (e.g., automaticallycalling elevators, path recommendation, smart access control, etc.) in the facility, developers can have a coordinate
system with spatial-temporal information as the underlying elements to develop their algorithms.
Such a uniform coordinate system for cognitive FM has the potential to be adopted in various facilities,
such as commercial buildings, hospitals, office buildings, warehouses, railway stations, and airports. Used incommercial buildings, by being aware of the location and path of the customers, it can guide customers find their
way to a specific store or even a specific product, help merchants study the preference of customers and send out
customized recommendations and advertisements, and further analyze the efficiency of the streamline design and
FM. For hospitals and other healthcare facilities, cognitive FM system with uniform coordinates offers an
undisruptive solution to track the patients with dementia, detect falling-down, guide the way to find the right room,
locate the nearest nurse for smart nurse calling, and find an important asset faster to save lives. For office buildings,
automatic elevator calling, smart access control, parking lot guidance for guests, and auto-control of HVAC
(Heating, Ventilation, and Air Conditioning) systems for energy saving become easier to implement. The accurate
locating of vehicles and items will lay a solid foundation for the unmanned warehouses. Additionally, at airports
and railway/bus/underground transportation stations, an accurate indoor positioning system will support smart
guidance for passengers, premier services for important guests, and the locating of baggage.
5. DISCUSSION AND CONCLUSIONSThis paper serves as method guidance for the user behavior data integration in cognitive FM system todisrupt the stagnant development of FM domain. It puts forward the idea that temporal-spatial information canserve as a gateway for the integration between the physical aspects of the facility and the social aspects of the users
and their behaviors in the cyber aspects of the FM. Besides, by comparing the accuracy, scalability, and cost ofdifferent local positioning technologies including infrared, vision analysis, magnetic signals, audible sound,ultrasound, Bluetooth, RFID, WLAN, and UWB, UWB is recommended as the device layer to construct the 3D
LPS, a uniform 3D coordinate system for the cognitive FM system. User behavior data marked associated withlocation data can be bridged with facilities, which also has their coordinate by coordinate transformation andcomputation. Thus, the relative position and distance of users and facilities can be recorded and computed forcognitive FM applications. The advantages of such a uniform 3D coordinate system include high accuracy andscalability for FM situation, providing a common language for communication, constructing a uniform coordinate
system for algorithms development of AI and robotics. Application scenarios for different facilities such ascommercial buildings, hospitals, office buildings, warehouses, railway stations, and airports are also furtherly
discussed. Future research should try to develop a prototype or do experiments to demonstrate the implementationof such a system and validate its feasibility, and develop system architecture and implementation methods toexecute its aims in different application scenarios. The validation and evaluation of the system is still underdevelopment but based on similar research and practical applications, its realization can be expected.
ACKNOWLEDGMENTS
The author of this paper would like to thank The University of Hong Kong to provide the Seed Fund for181Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Translational and Applied Research.
REFERENCES
Aitenbichler, E., and Muhlhauser, M. (2003). An IR local positioning system for smart items and devices. 23rd
International Conference on Distributed Computing Systems Workshops, Providence, USA, pp. 334339.
Alfeo, A. L., Barsocchi, P., Cimino, M. G., La Rosa, D., Palumbo, F., and Vaglini, G. (2018). Sleep behavior
assessment via smartwatch and stigmergic receptive fields. Personal and ubiquitous computing, 22(2),227-243.
Anjomshoa, F., Aloqaily, M., Kantarci, B., Erol-Kantarci, M., and Schuckers, S. (2017). Social behaviometrics
for personalized devices in the internet of things era. IEEE Access, 5, 12199-12213.
Besbes, O., Gur, Y., and Zeevi, A. (2015). Optimization in online content recommendation services: Beyond
click-through rates. Manufacturing & Service Operations Management, 18(1), 15-33.
Chen, A. T. Y., Biglari-Abhari, M., Kevin, I., and Wang, K. (2018). Context is King: Privacy Perceptions of
Camera-based Surveillance. 2018 15th IEEE International Conference on Advanced Video and Signal
Based Surveillance (AVSS), Auckland, New Zealand, pp. 1-6.
Chen, K., Lu, W., Peng, Y., Rowlinson, S., and Huang, G. Q. (2015). Bridging BIM and building: From aliterature review to an integrated conceptual framework. International Journal of ProjectManagement, 33(6), 1405-1416.
Cheng, J., Lo, C., and Leskovec, J. (2017). Predicting intent using activity logs: How goal specificity andtemporal range affect user behavior. Proceedings of the 26th International Conference on World Wide
Web Companion, Perth, Australia, pp. 593-601.
Chu, Y., and Ganz, A. (2005). A UWB-based 3D location system for indoor environments. 2nd International
Conference on Broadband Networks, 2005, Boston, USA, pp. 1147-1155.
Doryab, A., Chikarsel, P., Liu, X., and Day, A. K. (2018). Extraction of Behavioral Features from Smartphone
and Wearable Data. arXiv preprint arXiv:1812.10394.
Feldmann, S., Kyamakya, K., Zapater, A., and Lue, Z. (2003). An indoor bluetooth-based positioning system:
Concept, implementation and experimental evaluation. International Conference on Wireless Networks,
Las Vegas, USA, vol. 272.
Ghose, A., Ipeirotis, P. G., and Li, B. (2012). Designing ranking systems for hotels on travel search engines bymining user-generated and crowdsourced content. Marketing Science, 31(3), 493-520.
Gu, Y., Lo, A., and Niemegeers, I. (2009). A survey of indoor positioning systems for wireless personal
networks. IEEE Communications Surveys & Tutorials, 11 (1), 2009.
Hazas, M., and Hopper, A. (2006). Broadband ultrasonic location systems for improved indoor
positioning. IEEE Transactions on Mobile Computing, 5(5), 536-547.
Hoes, P., Hensen, J. L. M., Loomans, M. G. L. C., de Vries, B., and Bourgeois, D. (2009). User behavior inwhole building simulation. Energy and buildings, 41(3), 295-302.
Kalantarian, H., Alshurafa, N., and Sarrafzadeh, M. (2015). Detection of gestures associated with medication
adherence using smartwatch-based inertial sensors. IEEE Sensors Journal, 16(4), 1054-1061.
Kalantarian, H., and Sarrafzadeh, M. (2015). Audio-based detection and evaluation of eating behavior using thesmartwatch platform. Computers in Biology and Medicine, 65, 1-9.
Kang, T. W., and Choi, H. S. (2015). BIM perspective definition metadata for interworking facility management
data. Advanced Engineering Informatics, 29(4), 958-970.
Kawaji, H., Hatada, K., Yamasaki, T., and Aizawa, K. (2010). Image-based indoor positioning system: fast
image matching using omnidirectional panoramic images. Proceedings of the 1st ACM International
Workshop on Multimodal Pervasive Video Analysis, Firenze, Italy, pp. 1-4.
Kim, J., Thomas, P., Sankaranarayana, R., Gedeon, T., and Yoon, H. J. (2015). Eye‐tracking analysis of user
behavior and performance in web search on large and small screens. Journal of the Association for
Information Science and Technology, 66(3), 526-544.
Lee, J. G., Kim, M. S., Hwang, T. M., and Kang, S. J. (2016). A mobile robot which can follow and lead humanby detecting user location and behavior with wearable devices. 2016 IEEE International Conference on
Consumer Electronics (ICCE), Berlin, Germany, pp. 209-210.
Li, B., Gallagher, T., Dempster, A. G., and Rizos, C. (2012). How feasible is the use of magnetic field alone forindoor positioning?. 2012 International Conference on Indoor Positioning and Indoor Navigation(IPIN), Sydney, Australia, pp. 1-9.
Li, X., and Yang, S. (2015). The indoor real-time 3D localization algorithm using UWB. 2015 International
Conference on Advanced Mechatronic Systems (ICAMechS), Beijing, China, pp. 337-342.
Liébana-Cabanillas, F., Muñoz-Leiva, F., and Sánchez-Fernández, J. (2018). A global approach to the analysis ofuser behavior in mobile payment systems in the new electronic environment. Service Business, 12(1),182

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 201925-64.
Liu, H., Darabi, H., Banerjee, P., and Liu, J. (2007). Survey of wireless indoor positioning techniques andsystems. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications andReviews), 37(6), 1067-1080.
Liu, Z., and Huang, X. (2016). Reading on the move: A study of reading behavior of undergraduate smartphoneusers in China. Library & Information Science Research, 38(3), 235-242.
Mahfouz, M. R., Zhang, C., Merkl, B. C., Kuhn, M. J., and Fathy, A. E. (2008). Investigation of high-accuracy
indoor 3-D positioning using UWB technology. IEEE Transactions on Microwave Theory andTechniques, 56(6), 1316-1330.
Mandal, A., Lopes, C. V., Givargis, T., Haghighat, A., Jurdak, R., and Baldi, P. (2005). Beep: 3D indoor
positioning using audible sound. Second IEEE Consumer Communications and Networking Conference,
2005, Las Vegas, USA, pp. 348-353.
Mautz, R. (2012). Indoor positioning technologies. ETH Zurich. https://doi.org/10.3929/ethz-a-007313554.
Niu, Y., Lu, W., Chen, K., Huang, G. G., and Anumba, C. (2016). Smart construction objects. Journal ofComputing in Civil Engineering, 30(4), 04015070.
Rosenberger, M. E., Buman, M. P., Haskell, W. L., McConnell, M. V., and Carstensen, L. L. (2016). 24 hours of
sleep, sedentary behavior, and physical activity with nine wearable devices. Medicine and Science inSports and Exercise, 48(3), 457.
Saab, S. S., and Nakad, Z. S. (2010). A standalone RFID indoor positioning system using passive tags. IEEE
Transactions on Industrial Electronics, 58(5), 1961-1970.
Sakpere, W., Adeyeye-Oshin, M., and Mlitwa, N. B. (2017). A state-of-the-art survey of indoor positioning andnavigation systems and technologies. South African Computer Journal, 29(3), 145-197.
Tiemann, J., Schweikowski, F., and Wietfeld, C. (2015). Design of an UWB indoor-positioning system for UAV
navigation in GNSS-denied environments. 2015 International Conference on Indoor Positioning andIndoor Navigation (IPIN), Banff, Canada, pp. 1-7.
Vuković, M., Car, Ž., Pavlisa, J. I., and Mandić, L. (2018). Smartwatch as an Assistive Technology: Tracking
System for Detecting Irregular User Movement. International Journal of E-Health and Medical
Communications (IJEHMC), 9(1), 23-34.
Wang, G., Zhang, X., Tang, S., Wilson, C., Zheng, H., and Zhao, B. Y. (2017). Clickstream user behavior
models. ACM Transactions on the Web (TWEB), 11(4), 21.
Xu, J., Lu, W., and Li, L. (2019). Cognitive facilities management: definition and architecture. International
Conference on Smart Infrastructure and Construction (ICSIC), Cambridge, U.K, pp. 115-122.
Yang, C., and Shao, H. R. (2015). WiFi-based indoor positioning. IEEE Communications Magazine, 53(3), 150157.
Zhang, Y., Bai, X., Mills, F. P., and Pezzey, J. C. (2018). Rethinking the role of occupant behavior in building
energy performance: A review. Energy and Buildings, 172, 279-294.

183

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Building and Construction
Information Modeling (BIM/CIM)

184

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

RESEARCH ON BIM-BASED INFRASTRUCTURE PLATFORMS WITH
INTERNATIONAL STANDARDS
Katsunori Miyamoto1
1) Senior Researcher, Construction Information Research Institute, Japan Construction Information Center General
Incorporated Foundation (JACIC), Tokyo, Japan. Email: miyamotk@jacic.or.jp

Abstract: The Ministry of Land, Infrastructure, Transport, and Tourism (MLIT) released the “action plan for theoverseas development of infrastructure systems” in 2016. In consideration of this plan, we support the acquisitionof the international standards in addition to the Japanese domestic design standards originating in global activity
to promote original Japanese construction technologies and knowhow to minimize risks for differences in designconcepts between the standards.
Therefore it is becoming increasingly important to participate in international BIM standards meetings toshare information and study the major benefits of initiating the international standards related to BIM for
infrastructure. These benefits enables 3D collaborative design to improve the efficiency of the construction
production system through managed infrastructure lifecycle innovation and facilitates communication by using astandardized format for 3D data modeling, adding the attribute shape and materials information to enable datainteroperability between each phase of the production processes of the infrastructure systems.
In this paper, we considered a new method for solutions for civil infrastructure platforms based on theinternational standards that will maximize efficiency and productivity of public works. While watching the trendsof ISO and other international standards, we tried this new method to save time, costs, and human resources, andto improve safety and the production process while working on its standardization. The purpose of this paper is toprovide information to introduce a BIM-based framework using the Industry Foundation Classes (IFC) model inconsideration of a use case and tools from various Information and Communication Technology (ICT) atconstruction sites. In order to confirm the amendment results, we applied BIM to a complex, model-related
geography and geological 3D model at the time of disaster restoration by utilizing JACIC’s complimentary PhotogCAD® and performing case studies of investigations on small-scale areas and short-term lifecycles in a conceptualmodel.
Keywords:

BIM, InfraBIM, International Standards, Product Model, Model-Based Design, Cooperative Design

1. INTRODUCTION
This Document is a plan that should build discussion, create a system that describes the correspondence
policies of our country for international standardization promotion activities, and spread the BIM infrastructure(BIM for Infrastructure) in Japan and i-Construction. We are considering research that focuses on the methods ofthe development plan on the basis of a standardization trend of infrastructure BIM while more effective
infrastructure management is demanded. In this study, we arranged a conceptual model effect necessary for an
infrastructure platform with international standards using the Balanced Scorecard (BSC) technique fordevelopment and management of the project strategy.
This study attempts to plans affairs efficiently by letting users share various data from the construction
production system at each phase as an approach for improving productivity, which the MLIT promotes andprovides cooperation. The study reviewed the technical applicability of three-dimensional (3D) from each territory,
such as roads and rivers widely used in 2010, formulated a changed standard, and demonstrated ICT constructions
to promote the measure. 3D model IFC in the field of building was authenticated by the International Standard
ISO16739 in 2013, and was then expanded to the field of engineering works. However, for "OpenINFRA" it wasnecessary that the uniformity with foreign countries included a review of the data exchange standard of the
produced engineering works territories and it required stated goals, recommendations in the case of foreigncountries expanding the construction work of our country, and an exported infrastructure system.
An "international engineering works committee" was installed to review the construction production
system architectural policy and measures based on the infrastructure BIM in 2017. This committee holds a council
for domestic parties and reviews communication, and adjustment. We intend to create more profit utilization of
InfraBIM in Japan by examining domestic data exchange standards on the basis of the domestic correspondence
policy and international standards for international trends. In addition, for activities conducted in consideration of
the integration of the field of building and engineering works, the workbook about the IFC standards in our country
is necessary for confronting the standardization trends of association and the association between BIM andinfraBIM as urgent problems.
The original purpose of this paper is to provide information to achieve greater efficiency in theconstruction industry and public works (such as dams or bridges) in the near future, and to contribute to improving
productivity to build a safe and trusted infrastructure in our country.

185

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2. METHOD
2.1 Information topic management according to KJ method
First, we extracted topics using KJ method about problems within the scope of BIM for Infrastructure
standardization activities. KJ method is fundamentally similar to mind-mapping. It was developed as the Affinity
Diagram by Jiro Kawakita in the 1960s and became one of the Seven Management and Planning Tools used in
Total Quality Control (TQC) and became one of the ‘Seven management (New) tools’ of modern Japanese quality
management and uses values of Buddhism intended as structured meditation.
The basic cycle can be used to build up a problem-solving method through repetition, except it uses nested
clusters rather than a tree structure
1. Card making: all relevant facts and information are written on individual cards and collated. This step
could be adapted to use Brainstorming or Constrained Brainwriting, to generate a supply of ideas on cards.
2. Grouping and naming: The cards are shuffled, spread out and read carefully. For each group write an
apt title and place it on top of its group of cards. Repeat the group making, using new titles.
3. Chart making: We have less than 10 groups. In this study, BIM standards performed grouping at four
viewpoints of BSC and BIM technical elements performed grouping at four management elements.
4. Explanation: We express what the chart means, writing notes. Ideas for the solution are often developed
whilst explaining the structure of the problem. In this study, in eight groups become in four viewpoints,
management element from four management technologies and elemental technologies. (Figure 1.)

Figure 1. Basic figure of KJ method
2.2 Information rearranging, mapping according to BSC
We considered planning the duties efficiently with a construction production system which utilizes 3D
object data based on the international standards, and proofs of concept projects were performed for the purpose of
shortening work periods and, ensuring quality for productivity improvement of the construction work. However,there are areas still remaining such as the implementation.
Each country has its own strategy, and it is all the countries of the world, and it is standardization torepresent ISO that becomes the key issue. The process rule, communalization of 3D objects, and ground utilization
are necessary to push infrastructure businesses forward strategically; therefore we investigated international
standardization trends and a relationships, and arranged a concept model.
BIM for Infrastructure standards and technologies support it are introduced with many divergences.
Therefore, we used the Balanced Scorecard (BSC) of the management technique with scientific administration andmanagement for the present problem by rearranging and searching for a solution.
The BSC was originally developed by Dr. Robert Kaplan of Harvard University and Dr. David Norton asa framework for measuring organizational performance. Traditionally project managers used the BSC added
additional strategic measures only for short-term financial performance as a measure of success. Today it is usedwidely for systems such as the unification strategy of management including the KJ method and for brainstorming.

186

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The BSC is a strategic planning and management system that project managers use to:Convey what they are going to achieve
Arrange strategic works, group them, and attach titles

Prioritize projects, products, and services
Measure and monitor progress towards strategic targets
The system connects the dots between big picture strategy elements such as mission (our purpose), vision(what we aspire for), core values (what we believe in), strategic focus areas (themes, results and/or goals), andmore operational elements such as objectives (continuous improvement activities), measures (or key performanceindicators, called “KPIs,” which track strategic performance), targets (our desired level of performance), andinitiatives (projects that help you reach your targets). (Figure 2.)

Figure 2. Basic figure of BSC
3. RESULTS
The BSC shows that we acquire knowledge on objects from information to develop objectives, measures(KPIs; key performance indicators), targets, and initiatives (actions) relative to each of one of the following four
points of view:Ⅰ .Viewpoint of learning and growth and organizational capacity: Views organizational
performance through the lenses of human capital and infrastructure
Ⅱ. Viewpoint of internal processes: Views organizational performance through the lenses ofquality and efficiency related to key processes
Ⅲ. Viewpoint of customer and stakeholder: Views organizational performance from the point ofview of key stakeholders

Ⅳ. Viewpoint of financial or stewardship: Views organizational financial performance and the useof financial resources
The hierarchy of the BSC is written from the "Viewpoint of Financial or Stewardship in Services" to the"Viewpoint of Learning and Growth and Organizational Capacity in Standards Competence," but the enforcement
procedure is phase II, phase III, and phase IV from phase Ⅰ. It is essential to invest in this frame using ROI
provided in phase IV for phase I. This spiral process continuously improves and adds sophistication to the project.
First, we extracted a standard that became the standards such as with the ISO and European Committee
for Standardization (CEN), German Industrial Standard (DIN), and British Standards Institution (BSI) through the
KJ method, and mapped them onto four viewpoints according to techniques using the balance scorecard.
Then, we mapped concept models such as the "digital twin" that was a measure of German "Industry 4.0"
in consideration the standard item. We mapped them onto Social Implementation (SI), Project Management (PM),Change Management (CM), and Elemental Technology (ET) techniques that we accepted for four phases of thebalance scorecard. (Figure 3.)187

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 3. Standards information rearranging and mapping according to BSC
4. DISCUSSION
A precise plan, enforcement, and effect observation are enabled by it being said that the whole is mostsuitable, and mapping the major elements. At first, 3D CAD is set up to demand specifications depending on needs,
a profit utilization scene appears in the market early, and an engineer and the person concerned build available
environments early and are expected to apply them continuously. Specifically, 3D object models and process rules
are put together according to the order of the BSC with a spiral cycle. It is desirable for a high framework of
precision, and it is necessary to arrange the work with this balance scorecard as a package. Therefore, it is essential
to put events, such as the acquisition of software authorization by the IFC and the qualification authorization of
the BIM management, updates to the baseline, and monitoring. (Figure 4.)

Figure 4. Effects from BSC using the principle of leverage
This paper discusses the research on conceptual model for IFC Software Certification and its validationfrom all aspects. Proof of concept (PoC) is a realization of a certain method or idea in order to demonstrate its
feasibility. We extracted the standards about the BIM official approval from this BSC and applied a detailed flow
about IFC official approval based on this BSC and inspected the validity. The sophistication by the cycle whenthe IFC official approval continued because MVD exists every use case is demanded. (Figure 5.)188Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 5. PoC of IFC Software Certification
We extracted standards and technical elements from the BSC table as cafeteria system about CDE of thecollaboration environment using common data and verified a proof of Community Cloud conceptual model.
(Figure 6.)

Figure 6. PoC of Community Cloud Environment
Furthermore, we inspected the on desk about the BIM individual ability certification equally. Therefore
from these, it can be understood that it is the pivot when process rules, communities, and competence person
promote BIM. (Figure 7.)

Figure 7. PoC of Competence Certification

189

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

5. CONCLUSIONS
MLIT places BIM/CIM (Building and Construction Information Modeling/Management) with an engineof i-Construction to improve the productivity of construction sites 20% by 2025, carries out the utilization projects
as case studies, and is introducing BIM/CIM in earnest from 2019. In addition, problems that i-Construction and

BIM/CIM should support include labor inadequacy, environmental problems, disaster prevention, innovationand technical traditions, severe financial conditions, work-related accidents, globalization of productivity,and deterioration of facilities of the infrastructure due to population decline, low birthrates, and aging.
Furthermore, as IoT tools for Disaster simulation, Photog-CAD® is free software that supports
disaster recovery work and other survey work. As process rule of the, Photog-CAD® has been published in"the disaster notebook" said to be the effective tool of disaster recovery programs in Japan. Upon using
Photog-CAD® based on photogrammetry, 3D terrain models of the disaster sites can be built conveniently
using household digital cameras without surveying instruments. The function of this software is composedof the photogrammetry module and the design and cost estimation module controlled by CAD.
In this paper, we knew that this was strong tool for the three factors of "Common Information
Infrastructure (Platform)," which distributed the "process rule," "3D object model data", and data according to theanalysis by the BSC of this report about the reason why the life cycle management of infrastructure and, theinspection of common ground did not advance a digital change in the construction sector, and that supporting achange was important. As a result, we became able to expect greater efficiency in the construction sector andimprovement of productivity. (Figures 8.)

Figure 8.

Extension from three factors to four factors

ACKNOWLEDGMENTS
We would like to extend thanks to those organizations who kindly allowed us to evaluate the areas under
their charge. The photogrammetry module of this software is based on “3DiVision” a Tokyo Denki University.
REFERENCES
Miyamoto, K. (2012). A research study on the information sharing/exchange system during construction,
International Conference on Computing in Civil and Building Engineering (ICCCBE), Moscow, Russia,
236-237.
Miyamoto, K. (2013), Disaster recovery project support system, and assistance on site with Photog-CAD®, The
15th construction information research institute presentation lecture document(ICCBEI), Tokyo, Japan,
477-484.
Miyamoto, K. (2014). A research study on lifecycle infrastructure management with shared product models on
collaborative information systems, International Conference on Computing in Civil and Building
Engineering (ICCCBE), Orlando, Florida, United States of America.
Miyamoto, K. (2015). CIM for sustainability appraisal of conceptual ILCM of disaster restoration work,
Proceedings of the 2nd International Conference on Civil and Building Engineering Informatics(ICCBEI), Tokyo, Japan, pp.83.
Miyamoto, K. (2015). A framework for early disaster recovery support using a CIM model based on
photogrammetric techniques, Proceedings of the World Engineering Conference and Convention,
Kyoto, Japan.
Miyamoto, K. (2016). A Framework for the Data Coordination Method of Maintenance Data and a 3D
Conceptual Model on a CIM Based Database, Proceedings of the 16th International Conference on
Computing in Civil and Building Engineering (ICCCBE), Osaka, Japan, pp.185.
Miyamoto, K. (2017). Research on Infrastructure Lifecycle Management Platform based on CIM with
International Standards, Proceedings of the 3nd International Conference on Civil and Building
Engineering Informatics (ICCBEI), Taipei, Taiwan, pp.335-338.

190

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

A BIM-based AR Application for Construction Quality Inspection

Nai-Wen Chi1, Yi-Wen Chen2, Shang-Hsien Hsieh3, Jen-Yu Han4, and Lung-Mao Huang5

1) Project Assistant Professor, Department of Civil and Construction Engineering, National Taiwan University of Scienceand Technology, R.O.C. Email: nwchi@mail.ntust.edu.tw2) Research and Development Engineer, Moldex3D Co., Ltd., R.O.C. Email: owen0407@gmail.com
3) Professor, Department of Civil Engineering, National Taiwan University, R.O.C. Email: shhsieh@ntu.edu.tw
4) Professor, Department of Civil Engineering, National Taiwan University, R.O.C. Email: jyhan@ntu.edu.tw
5) Assistant Vice President, Reiju Construction Co., Ltd., R.O.C. Email: ko0957@mstc.reiju.com.tw

Abstract: Quality inspection is always an important stage during construction projects. Effective quality control
can help construction projects avoid schedule delay and budget overrun. However, effective and efficientconstruction quality control is always a challenging task. It not only requires experienced inspectors but also
various tools such as design drawings, tape measures and cameras to support the on-site quality inspections. In
recent years, with the popularity of portable devices, and also the maturity of Building Information Modeling (BIM)
and Augmented Reality (AR), it has become possible to integrate the two techniques on portable devices fordeveloping an innovative construction quality control platform. In this paper, a prototype BIM-based AR quality
inspection system is proposed. It can assist inspectors to discover the construction errors in the early stage of aconstruction project by comparing the on-site construction objects (especially for the falseworks) with theircorresponding elements in the BIM model. The framework and the user interface of the prototype system are
discussed and the application scenarios for improving the current quality inspection procedure are demonstrated.

Keywords: Building Information Modeling (BIM), Augmented Reality (AR), Quality Inspection, Site Inspection

1. INTRODUCTION
Quality control is an important process performed by general contractors. To execute quality control plan,inspectors must perform quality inspections with various equipment and supporting tools. For example, they must
check out the element size on design drawings, and then measure the actual size of the corresponding real objects
to check if they are correctly constructed. In addition to size, inspectors also need to check out the form, as wellas the specifications of the building objects, which requires their experiences on interpreting the information on
design drawings because it may not fully reflect actual appearance of real objects. These are the reasons why
traditional quality inspection process is inefficient and may delay construction schedule.
Integration of Building Information Modeling (BIM) and Augmented Reality (AR) provides a possible
solution to address the aforementioned issue. BIM can provide not only the geometric information but also 3D
visualization for a building. Comparing with design drawings, BIM model can provide a comprehensive reference
for identifying the differences between the 3D model and real facility and can therefore deliver more accurate
information for inspection. There has been also existing research on BIM-based quality inspection. For example,

191

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Li (2016) combined BIM and database, and developed a web-based quality inspection system. However, many ofsimilar researches were implemented on PC, which still face the issue of limited portability. To make use of BIM
on construction site is not an easy task because many of BIM software packages (e.g., Autodesk Revit or Trimble
TEKLA Structures) often have high hardware requirements and cannot be easily and directly executed on portabledevices. Inspectors can hardly bring their laptops to the construction site and operate BIM software for assisting
quality inspections. To address the problem, Tsai et al. (2014) developed a BIM-based mobile approach for quality
inspections. It enables the mobile devices to interact with BIM models. In addition, AR also extends thepossibilities on BIM applications. For example, Li (2017) proposed a BIM-based AR application for quality
inspection that completes spatial information acquisition using AR. It transfers spatial information to the BIM
model, and applies geomatics technique for image positioning. Inspectors can review the correctness of theconstructed facility with 3D models that mapped with photorealistic textures. Based on similar concepts, theauthors want to apply BIM and AR on quality inspection, especially for comparing the differences between BIM
model components and real objects. Therefore, a prototype BIM-based AR system, Smart Quality Inspector was
implemented to achieve this goal.
In our previous research (Chang et al., 2018), we focused on the user interface design and developmentof a mobile application. In this paper, we extend the system implementation to the web server that uses ASP.NET
Web API and Autodesk Revit API, respectively, to develop applications for assisting communication between
portable devices and web server, and for extracting BIM models into portable devices. We also test the prototype
system developed in real construction sites. The results show that the proposed Smart Quality Inspector, thoughjust a prototype system, is able to improve the current quality inspection process.

2. THE SMART QUALITY INSPECTOR SYSTEMThe Smart Quality Inspector system can be mainly divided into three modules, model transformer, mobile
application, and web server, as Figure 1 shows:192

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 1. The framework of the Smart Quality Inspector system

The frameworks and functions of the three modules are further elaborated as follows:1.

Model transformer: In order to make good use of a BIM model on portable devices, some preparatory works
must be applied to the BIM model. For reducing the system burden on portable devices, we only extract
necessary information from the BIM model. We developed Autodesk Revit API applications that can export
both the mesh model (which can be displayed in Unity frameworks) and the floor plans from a Revit model.
The user interface of model transformer is shown in Figure 2. Users can select specific floors (i.e., they can
only export part of the BIM model, as the total building may be too huge for mobile applications) from the
BIM model and export them to mesh models for mobile applications.

Figure 2. The user interface of model transformer

193

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 20192.

Mobile application: The mesh models and the floor plans generated in the previous step are then loaded into
the mobile application before quality inspections are performed. Although portable devices are light-weighted
and are convenient for quality inspectors, their computing performance is not as good as desktop or laptop
computers. As a result, it is not an easy task to browse a BIM model directly on portable devices. In addition,
portable devices cannot support BIM software such as Autodesk Revit as well. On the contrary, most of
portable devices with iOS/Android operating system can support the Unity-based VR/AR applications.
Therefore, we adopt the mesh format supported by Unity to represent the geometry information of the BIM
model. Quality inspectors must first pinpoint their current locations on the map, and switch to the camera viewto locate their approximate coordinates in the 3D model. After that, they can start their inspections byoverlaying the transparent 3D model to real objects manually in their portable devices, as Figure 3

shows.

During the inspection process, the gyroscope sensor in the mobile device can help inspectors to overlay themodel and the real object smoothly. The inspector can then lock the AR view of the screen when theperspective of the model has matched well the real objects. A quality inspector can not only check if theconstruction objects meet the requirements but also make text/voice/image records for found defects on theconstruction objects.

Figure 3. The user interface of mobile application3.

Web server: All the inspections performed by quality inspectors will be uploaded to a web server, which isdesigned for managing different construction projects under a general contractor. A manager can check all theinspection records and perform several administrative applications (e.g., quantity takeoffs) through the webbased interface. The system is implemented by the ASP.Net framework and supported by Microsoft SQL
Server database, and uses the ASP.NET Web API to interact with mobile applications. Inspectors withportable devices can upload their photos/records/voices to the web server. In practice, we can divide thequality inspection works into many parts and assign the tasks to different inspectors. The data on the web194Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019server will be synchronized with the mobile devices held by both quality inspectors and quality managers. An
example screenshot of the web-based administrative system is shown in Figure 4.

Figure 4. A sample screenshot of the web-based administrative system

3. TESTING SCENARIO
For validating the usability of the proposed Smart Quality Inspector system, the authors have tested thesystem in a real construction site. The construction project is a public housing building located in the Taipei city.
The Reiju Construction Co., Ltd., which is an A-class construction company in Taiwan, is the general contractor
for this construction project, and also the investor of the proposed Quality Inspector system. The BIM model ofthe public housing building is available for the proposed system. We applied only part of the BIM model (fromB1F to 2F) because most of the building is still under construction.
Figure 5 shows an example scenario about how we perform quality inspection on construction objects.
Traditionally, quality inspectors have to check the size of opening on the design drawings, and then measure theactual sizes of the opening on the wall. By operating the Smart Quality Inspector system, we can simply check if
the edges of the translucent AR model (transparency of the model is adjustable) can match the real object. From
the figure, we can see they are perfectly matched. As a result, we select the wall on the AR model, take a photo as
an evidence that the wall has passed quality inspection.

Figure 5. Checking the actual size of opening by the mobile application

195

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

4. CONCLUSIONSIn this paper, we proposed a smart Quality Inspector system, which takes advantage of the BIM and AR
technologies. The prototype system helps demonstrate improvement over the traditional quality inspection
scenarios on two aspects: (1) various support tools are integrated in a simple portable device to facilitate
comprehensive inspection in a reasonable timeframe, and (2) compared with design drawings, a 3D AR model ismuch easier to comprehend and be compared with the corresponding real objects. In the future, the authors willcontinue to enhance the functionality and usability of the mobile application, and also develop more applications
for administrative tasks.

REFERENCES
Li, Y. C. (2017). Integrating BIM and Geomatics Techniques for Construction Quality Control, Master Thesis,
Department of Civil Engineering, National Taiwan University (in Chinese).
Li, P. C. (2016). A BIM enabled Construction Quality Inspection Approach, Master Thesis, Department of Civil
Engineering, National Taiwan University (in Chinese).
Chang, Y. Y., Yang, Y., Hsieh, S. H., Han, J. Y., and Huang, L. M. (2018). Integrated Application of BIM andAR for Construction Quality Inspection: User Interface Design, Proceedings of the 22nd Symposium on
Construction Engineering and Management, Taipei, Taiwan, R.O.C (in Chinese)Tsai, Y. H., Hsieh, S. H. and Kang, S. C. (2014). A BIM-Enabled Approach for Construction Inspection,
Proceedings of the 2014 International Conference on Computing in Civil and Building Engineering,
Orlando, Florida, USA, USA, pp. 721-728.

196

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

APPLICATION OF BIM TECHNOLOGY ON VIRTUAL MOCK-UP
IMPLEMENTATION FOR GENERAL CONTRACTOR IN CONSTRUCTION

AND

Rodrigo Samuel Ortiz Chaparro *1, Yu-Cheng Lin 2
1) Master graduate student, Department of Civil Engineering, National Taipei University of Technology, No.1. Chung-Hsiao
E. Rd., Sec.3, Taipei, Taiwan. Email: rodrigo.ortizchap@gmail.com
2) Professor, Department of Civil Engineering, National Taipei University of Technology, No.1. Chung-Hsiao E. Rd., Sec.3,
Taipei Taiwan. Email: yclin@ntut.edu.tw* To whom correspondence should be addressed. E-mail: rodrigo.ortizchap@gmail.com

Abstract: Building information modeling (BIM), the visual technology that has been constantly used inarchitectural, engineering and construction (AEC) industry. Some of the major benefits of applying BIM includes
presenting and simulating construction operations through 3D illustration, as well as providing an advanced review
for construction management. Two dimensional CAD drawings are still widely used for construction projects.
However, to avoid misunderstanding of the design concept in projects of larger complexity, construction project
teams will require more reliable and detailed information. This study proposes a BIM-based virtual mockup
approach for general contractors, to enhance the performance of analysis review for construction management inconstruction phase of projects. The proposed approach is then applied in the construction phase of a selected
project case study to discuss the effectiveness of BIM-based virtual mockup implementation approach in practice.
Finally, this study identifies and summaries the benefits, limitations, conclusion, and suggestions for further
applications.
Keywords: Building information modeling; BIM; virtual mock-up; general contractor; construction
management.
1. INTRODUCTION
BIM is a powerful tool that gives the project staff a better way of working, providing many accessible
tools that maximize and increase the efficiency and productivity in civil engineering during a project’s life cycle.
It’s an exciting time in the AEC industry because just as application are improving, so are many technologies that
support its use (Barnes & Davies, 2015). Different from 2D models, 3D virtual mock-up representations help toperceive spaces in an identical way to it physical counterpart this allowing owners, engineers and designers tobuild/refurbish, visualizing 3D models of buildings is also essential in assisting clients’ decision making to accept
or change design parameters/criteria. A detailed virtual mock-up will positively affect constructability, studies,work sequences, scheduling, potential savings in terms of time and cost, improve coordination and design
flexibility. In areas where there is no proper skilled labor and there are no proper conversation skills (language
problem among project participants) it is the 3D model that acts as a perfect universal language. Apart from helping
workers to understand the unfinished product (construction process, detailing, and workflow simulations) it will
also help to satisfy the client demands.
For general contractors, BIM technology is considered recently as an essential mock-up simulation tool
to simulate full coordination of various designs before construction for advanced construction management.
Therefore, the main objective of this study is to propose approach for virtual BIM-based mock-up, permitting
everyone involved in the construction process the ability to visualize (by way of virtual 3D model walkthrough)what is actually to build and the future results of the construction project. Finally, methods for addressing thechallenges, conclusions and suggestions are drawn from the analysis of the case study.
2. LITERATURE REVIEWFor the past years, BIM software industry has been continuously developing new modeling tools. These
tools are providing general contractors with accessible management integration methods (Hardin, 2015). A virtual
reality integrated construction mock-up system is a tool that can enable the project team to solve issues or do
improvements in a very early stage of the construction phase of a project. The concept of virtual mockups is,similar to physical mockups, an accurate replica of a portion of the building envelope systems in the virtualenvironment (Maing, 2012). BIM-based virtual mock-up will assist with the ability to do more “what if” scenarios,
such as looking at various sequencing options, site logistics, temporary structures, decision-making framework for
scaffolding structure (Barnes & Davies, 2015). In addition, virtual mock-up with a combined consolidate model
can be used for communicating constructability information between design and construction teams (Krzyzosiak,2016), which will allow construction team members to have a clearer visualization of construction projects andhelp to perform site inspection more efficiently by identifying high-frequency and high-severity events within aconstruction site for mitigation or hazard removal (Barnes & Davies, 2015).
At the same time, the use of virtual mock-up technology for general contractors will enhance the197Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

capacity of integration of construction processes by bringing data, building information, and models from reliable
sources together to generate unique data platform, it will also enable project team members to work together,creating easier ways to address common problems or analyzing construction information (Krzyzosiak, 2016).
It is necessary to consider use of LOD (level of Development) for BIM-based virtual mock-u
p during the process. The LOD is defined by the AIA E203 as “the minimum dimensional, spatial, qua
ntitative and qualitative, and other data included in a model element to support the authorized uses asso
ciated with such LOD.” (Hardin et al., 2015; AIA., 2013). The LOD and model information are gener
ally defined for the key stages of the project (Barnes & Davies, 2015). However, for more detailed andcomplex operation, the level of information may not be complete for the construction purposes. During
the construction phase of a project, LOD in BIM model increases as the project proceeds, often basedin the first instance, on existing information (Sawhney, 2017).
This model based coordinated work will lead to a better workflow (B. Hardin, 2015), enhanceon site communication efforts and act as a time safer which are some of the major problems of const
ruction site in the AEC field (Eastman et al., 2011). Changes in design of the construction project will
be easier due to the constant communication, accessible and reliable building information.
Given all this good aspects, there are also many challenges associated with implementation ofBIM-based virtual mock-up. The usage of the BIM-based virtual mock-up will reduce fewer human errors and reduce the cost for physical mock-ups.
The employment of BIM and virtual models will present potential challenges:
1. Further application of BIM in practice requires a collaborative approach, coordination among otherparties to exploit the potential of BIM (Gholizadeh et la., 2017), constant availability of information
technology from different contractors and labor personnel (Beliz & Ugur, 2017).
2. Merge information from different sources and software, this task will require a certain expertise in
the most common software used in BIM (AutoCAD, Revit, Tekla, Navisworks, Civil 3D, Excel, Sk
etch up, etc.).
3. For developing countries equipping the personal with necessary information and skills will be a cha
llenge because of the accessibility of new technologies and information in the field, this kind of op
eration methods are not accepted by any public agencies yet (Beliz & Ugur, 2017).
4. Resistance to implementation of Virtual mock-up, which could partly result from cultural resistance(Ruoyu, et la., 2017).
5. Understanding the value, customer needs or the cost and return on investment, BIM education andlack of training (Ruoyu, et la., 2017; Hardin, 2015).
6. Creation of new roles in the construction site for the implementation of a virtual mock up, assign
a top-level management responsible for the development of an adoption plan, creating an internal te
am of key managers and engineers responsible for the development of the virtual mock-up and wor
k advices (Eastman et al., 2011).
3. VIRTUAL MOCK-UP IMPLEMENTATION APPROACH
In order to handle a virtual mock-up using BIM specifically for general contractors, the study proposes a
conceptual framework for the adoption of virtual mock-up implementation. For each phase of the construction or
installation project, project manager and project engineers may utilize an existing BIM model to simulate theexpected result thought virtual mock-up before the job execution. However, it is important to first verify that all
the requirements are met. The BIM model need to be modified or revised in advance according to the different
purposes or phases of it usage.
This study develops the flowchart for the utilization of virtual mock-up implementation for generalcontractor during the construction phase (Fig. 1).
Due to the complexity or short budget available for some projects the BIM models are usually less detailed,
it is necessary to identify which content of BIM model are more critical, susceptible of mistakes or will be produced
in massive numbers, then construction project participants that are directly involved on the project (including
project manager, project engineer, or onsite engineer) can select the content that will require a more enhanced or
focused attention for virtual mock-up development and implementation. In this stage of the development theelements that also require a higher level of detail have to be selected. All the virtual mock-ups will be planned anddeveloped based on the project team members’ meeting and according the project needs.

198Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Start

`
The development of
BIM model isnecessary for Markup
in practiceNo
To confirm
Yes

To determine the elements of
BIM model for virtual mockup

To determine thecontent for virtual
mock-upTo determine LOD of
BIM model for virtual
mock-upNo
To implement BIMbased virtual mock-up

Yes

To develop BIM
model for virtual
mock-up

To confirm
No

To meet therequirement

No

Yes

End

Figure 1. Flow chart for the BIM-based virtual mock-up design and implementation.
During any stage of a construction phase of a project, project manager (or construction manager) andproject engineers have to work together for the design and development of BIM-based virtual mock-up. Table 1
illustrates the major responsibilities for project contributors involved in the creation and implementation of a BIMbased virtual mock-up
From the table we can note that cooperation and communication between the project team members is
significant for a construction project, the participation of a BIM engineer is also indispensable for building thevirtual mock-up. However, most of the projects will not have enough budget to settle a complete BIM specialist
engineer’s team. One proposed solution that will be necessary to carry out the implementation of virtual mock-up
will be the creation of a new responsibility for the role of project manager. This role will allow the engineer toperform a better management and coordination of information between different supplier companies, MEP(Mechanical, Electrical, and Plumping) subcontractors, different services companies and construction project teammembers facilitating the management and creation of virtual mock-up. The similarity of the BIM manager andproject manager duties can make this extra task possible, bringing a non-cost solution that can be applied in smaller
projects of general contractors that lack of the tools adjacent to virtual mock-up.
General Contractors will have to encourage construction managers to go through trainings andcapacitation on BIM technology in order to learn how others are using each piece of software and do the applicationof this kind of process of work. As mention before, a certain level of development for the proposed Virtual Mockup will be requested, a project management can rely on the already existing potential to generate coordination andmanagement of information. Its expertise in construction will add more reliable source of information to findcritical factors that usually delay construction projects, or even cause conflicts between different contractors.

199

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 1. Responsibilities for participants involved in the implementation of the BIM-based virtual mock-up
Role
Project engineer

BIM manager

BIM engineer
Project manager (orconstruction
manager)

Responsibilities
To supervise on-site technical tasks and propose requirements for the creation of thevirtual mock-up to then apply in the project.
To certify the subcontractors or client demands.
To set up the level of development needed for the virtual mock-up.
To manage and facilitate all the process necessary to create and manage BIM. This
involves coordinating all the information from architecs, consulting engineers, andsubcontractors. The BIM manager also coordinates critical reference points andpropose solutions for the use of the virtual mock-up.
To develop and revise the BIM models based on the proposed requirements during
the process of creation of the specific virtual mock-up.
To confirm BIM-based virtual mock-up result.
Initiate or plan BIM-based virtual mock-up implementation.

Although the design and development of a BIM-based virtual mock-up it is not complex work for generalcontractor, the successfully implementation on a construction site where it will be acting as the main reference fordifferent contractors is what it makes it more challenging. Fig. 2 shows the management framework of the BIMbased virtual mock-up implementation. There are three parts in the framework of the BIM-based virtual mock-up
implementation. They are BIM model for virtual mock-up, people, and BIM model management. The part thatmake implementation of virtual mock-up easier is the people involved in the application, as it was mentioned
before there are a few engineers that will play a key role in the implementation of the BIM-based virtual mock-up
in constructions, the main job of these engineers is to fully understand the BIM model management by executionof detailed information, revision of considerations for design, coordinated cooperation works and agreements toaccess information and references from subcontractors and suppliers. The build-up of a BIM-based virtual mockup will be executed only after going through several analyses like: BIM LOD of the existing BIM models andproposed virtual mock-ups, clients and architecture requirements, construction code requirements, clearance ofdetailed information of the elements and materials needed for the specific sections, constructability, installation
works, and feasibility analysis of construction site scenario. Therefore, in order to achieve a full application of thevirtual mock-up the construction participants should also work for the execution plan.
BIM Model for
Virtual Mock-up
BIM LOD

Requirements

BIM Element

Scenario

BIM Content

Detail Information

People

BIM Engineer

Project Engineer
BIM-based
virtual mock-up

BIM Manager

Project Manager

Main BIM Model

Revised BIM Model

Model Management

BIM Model Management

Figure 2. The management framework of the BIM-based virtual mock-up implementation200

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

4. CASE STUDY
4.1
Case study introduction
The following case study involves a general contractor with twenty years of specific experience in the
Taiwanese construction office building project. In this case working in cooperation with three different
subcontractors, and five suppliers during the construction phase of a project. Furthermore, after collecting essential
data of materials from different suppliers, the general contractor employed BIM technology for the design of avirtual mockup of specific critical points, encouraging all onsite engineers and project manager to use this BIMbased virtual mockup to identify important interfaces and requirements for better construction management.
4.2
Detail virtual mock-up implementation in construction
The mock-up is an essential tool when it comes to visualize complex building information that seemvery complicated in a 2D plan. Fig. 3 and Fig. 4 demonstrate a complex cross section of a virtual model, detailingthe materials, fabrication specifications, and different elements used, which helps project team member for
assembly and installation procedures.

Waterproof
Membrane

Marble Door
Threshold
Stainless Steel
Plate

Figure 3. Detailed cross section of a door frame and the design of waterproof membrane.

Insulation
Foam

Z-shaped Ceiling
Joint
Ceiling Board

Aluminum
Alloy Frame
Plate Glass
Silicone SealantFigure 4. Detailed cross section of an office section
This 3D cross sections virtual mock-ups are part of the mentioned construction project that required a fastmodeling and had complicated construction operations. The assistance and expertise of the BIM engineer
helped with the problem solution discussions, as a result the proposed modeling goals were quickly
achieved and completed. However, engineers’ expertise and long business consolidation in this sector were201

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019not sufficient to immediately convince full coordination agreement between the parties involved.
Once cooperation was achieved, the implementation approach was applied to facilitate the coordination andworking efficiency, each design team made its design data available for the project, models were
integrated, checked, approved, and validated as fit for coordination.
4.3
Discussion
4.3.1 Major advantages for use of Virtual Mock-up in the case study
1. Building virtually allows project stakeholders to flesh out a project’s true design without the use of
conventional real mock-up.
2. Give a clear understanding of the concept design and prevent conflicts between different constructions
works allowing full coordination of various consultants’ design and contractors.
3. Highlight critical issues, assist in the creation of a working plan and execution.
4. Determinate the exact material used for every exact location by having a clear visualization of the virtual
representation of the detailed model, materials can be labeled and described.
5. Effectively control construction project costs to maximize profits.
6. Virtual mock-up permits a higher reliability of expected site conditions and measurements.
7. Allow manufacturers of prefabricated materials to build contract requirements based on BIM-based virtual
mock-up, which will minimize the problem of nonconformance between onsite unsuitable measures andproject requirements.
4.3.2 Major limitations for use of Virtual Mock-up in the case study
1. Construction site implementing BIM-based virtual mock up must have site engineers that are trained or atleast familiar with BIM technology process and tools in order to immediately build detail models if they
are required.
2. A BIM-based mock-up cannot provide the construction project team with a 1:1 scale mock-up.
3. Different form physical mock-up, virtual mock-up cannot have real material samples.
4. The material suppliers couldn’t present material specifications and requirements before construction starts
making it more difficult for the general contractor to build the virtual mock-up with more reliable
information.
5. Labor personnel still prefer using 2D plans as reference because they are more used to it.
6. Cooperation agreement between subcontractors and general contractor.
7. Weekly check reports of updates of onsite or offsite changes to then apply in the current BIM-based virtual
mock-up.
5. CONCLUSION
The research indicates that technology has presented many advances in tools and methods that can enhance theefficiency and effectiveness of a project construction phase. BIM models and Virtual mock-ups can be an
integrated part of architectural and engineering education since the future will dependent on new technologies,
having a complete catalog of critical points and its respective virtual mock-up will improve construction operations.
However, construction managers are still working on the adoption of this new tools, implementation of BIM-based
mock-ups still faces challenges in some key areas that are to be improved:
1.
Team oriented challenges: General contractors will not always have enough budget to get the necessary andrecommended number of engineers that are required for the implementation of virtual mock-ups. As statedearlier, a whole set of engineers are needed and they will likely need a great BIM knowledge and field
experience to successfully delivery the application of virtual mock-ups on a construction site.
2.
Employment of new tools and technology in the construction site: The concept of a BIM-based virtual mockup is still new and it is not commonly adopted throughout construction site operations. Even though BIM isa well-known operation process, and it has been successfully implemented in many construction projects.
Achievement of a good operation management with this new tool is still under-development, which will
require contractors to spend a lot of effort to master the necessary technical set of software and data.
3.
Strategic planning: In order to increase the operational efficiency in construction, weekly meetings will beneeded, people involved in the construction project have to be present: field personnel, subcontractors,
engineers and architects. The main purpose is to communicate modifications, construction instructions, andcoordination of work sequences. The requirements and analysis of the situations of critical factors for thedesign and implementation of new virtual mock-ups will have to be discussed.
4.
Maintaining a construction project schedule: Weekly reports and virtual mock-up designs will have to becompleted in a reduced time. However, due to the many working activities happening at the construction
site, these activities usually require a quick action or full attention of the engineers in charge, reducing the
expected attention on virtual mock-ups. Therefore, regardless the effort it will not always meet the
expectations.

202

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 20195.

Information availability: Having a well accumulated BIM asset can add to the accuracy and design
considerations taken in the designing phase of the virtual mock-up. Currently not all the subcontractors are
employing BIM operation process.
In order to address this challenges, existing tools are being adapted and new ones created. This increased demandof new management processes tools, due to complex projects, are bringing the necessity of implementation of new
virtual mock-ups that can replace its traditional physical counterpart. This, brings many positive results andincreasing BIM usage experience, facilitating the communication with client and project team members to deliver
better construction outcomes. Furthermore, field experience is a positive factor for quick and reasonable decision
making to problems that engineers face in every project.
The research summarizes the following suggestions that advocate for the implementation of a virtual mock-up:
1. Enhances construction projects by the adaptation of this methodology to suit the project requirement. Present
improvements to the construction industry by boosting the visualization strategies, which allows for thecreation and exchange of knowledge during the design process or implementation of the virtual mock-up.
2. Facilitates site inspections and constructability analysis by establishing critical points and highlighting
repetitive elements on construction projects.
3. Improves installation tolerance decision and timeframe optimization.
4. Increases engineers’ awareness of the construction implications of the design processes before or during theconstruction phase of a project.
5. A key factor for the successful adoption of virtual mock-up is the full support and collaboration of construction
project team members.
6. The following procedures will directly benefit the execution of virtual mock-ups:
 Focus on small gains, Set weekly meetings in order to collect field information,
 Share expertise, and highlight critical points that will require a virtual mock-up with higher LOD level.
6. REFERENCES
B. Hardin D. McCool. (2015). BIM and Construction Management (2nd ed.). Wiley.
Peter Barnes and Nigel Davies. (2015). BIM in Principle and in Practice (2nd ed.). ICE Institution of Civil
Engineers.
Beliz Ozorhon and Ugur Karahan. (2017). Critical Success Factors of Building Information Modeling
Implementation. J. Manage. Eng., 10.1061/(ASCE)ME.1943-5479.0000505
R.Y Sharahily, B. Medjdoub, M. Kashyap & M. L. Chahal. (2015). Communication framework to support
more effective onsite construction monitoring WIT Press - BIM in Design, Construction and Operations.
Xiaozhi Ma, S.M. ASCE, Feng Xiong, A.M. ASCE, Timothy O. Olawumi, Na Dong and Albert P. C. Chan.
(2018). Conceptual Framework and Roadmap Approach for Integrating BIM into Lifecycle ProjectManagement. J. Manage. Eng., 10.1061/(ASCE)ME.1943-5479.0000647.
Patrick Krzyzosiak. (2018). Let's Build. Compelling Reasons to Use Virtual Mock-ups for Fabrication. Retrieved
from Building Tehc Insider website: http://blog.rsconstruction.com
Ruoyu Jin, Craig Hancock, Llewellyn Tang, Chao Chen, Dariusz Wanatowski, M.ASCE, and Lin Yang. (2017).
Empirical Study of BIM Implementation-Based Perceptions among Chinese Practitioners. J. Manage.
Eng., 10.1061/(ASCE)ME.1943-5479.0000538.
Gholizadeh, P., B. Esmaeili, and P. Goodrum. (2017). Diffusion of building information modeling function in theconstruction industry. J. Manage. Eng. 34 (2): 04017060. https://doi.org/10.1061/(ASCE)ME.194354790000589
Peter Fewings. (2013). Construction Project Management, An Integrated Approach (2nd ed.). Routledge.
Anil Sawhney., Atul R Khanzode., Saurabh Tiwari. (2017). Building Information Modelling for Project Manager.
RICS Insight Paper
American Institute of Architects (AIA). (2013). Building Information Modelling and Digital Data Exhibit.
Document E203
Eastman, C., Teicholz, P., Sacks, R., and Liston, K. (2011). BIM Handbook: A Guide to Building Information
Modeling for Owners, Managers, Designers, Engineers, and Contractors (2nd ed.). Wiley.
Minjung Maing (2012). Effectiveness of virtual mockups of building envelope systems. Building Enclosure Science& Technology (BEST3) Conference Paper.

203

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

DEEP LEARNING-BASED SCAN-TO-BIM FRAMEWORK FOR
COMPLEX MEP SCENES USING LASER SCAN DATA
Chao YIN1 & Boyu Wang1, and Jack C.P. Cheng1
1) Ph.D. Student, Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology,
Hong Kong, China. Email: cyinac@connect.ust.hk
2) Ph.D. Student, Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology,
Hong Kong, China. Email: bwangbb@connect.ust.hk
3) Ph.D., Assoc. Prof., Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology,
Hong Kong, China. Email: cejcheng@ust.hk

Abstract: Mechanical, electrical and plumbing (MEP) systems play an important role in buildings and civil
infrastructure. As laser scanning technology can capture geometric information with a high accuracy, point clouds
have been widely used to reconstruct the as-built models of MEP systems. However, due to the complexity of point
clouds collected from industrial scenes, processing point cloud data is mostly conducted manually over a long period.
In order to speed up the modeling work, this paper proposes an automated BIM model generation framework basedon deep learning technique for complex MEP scenes, with noise and heavy occlusion. There are four main proceduresin our framework: (1) semantic segmentation: a state-of-the-art deep learning model named PointNet is adapted andleveraged to segment point clouds into four categories; (2) instance segmentation: retrieved point clouds of different
classes are further segmented into instances with clustering methods; (3) geometrical information extraction andalignment: for pipes, an algorithm is developed to extract geometric information for pipes; and (4) BIM generation: a
BIM model is generated parametrically using Dynamo based on geometric information and alignment parameters.
The proposed approach was validated using a real MEP scene. The results show that our proposed framework cansatisfactorily provide an automated and robust solution to BIM model generation for MEP scenes using laser scanningdata.
Keywords: As-built BIM, SCAN-to-BIM, Point Cloud Processing, Deep Learning, MEP, Laser Scanning
1. INTRODUCTION
Building information modeling (BIM), which generally refers to creating semantically rich digital models, iswidely used to manage the project data through a facility's life cycle (Tang et al. 2010). BIM models can serve as acentral data hub that can store and exchange information about a facility. The creation of as-built models for existing
facilities is of great importance based on two reasons: 1) The as-design CAD-based models cannot reflect how thefacility is actually built; 2) Some old facilities built many years ago do not have corresponding 3D model. 3D Laser
scanning technology has made it tractable to create as-built BIM models (Tang et al. 2010). With laser scanning,
engineers can reconstruct the 3D scenes into point cloud data with extremely high accuracy within several minutes.
Mechanical, electrical, and plumbing (MEP) works constitute a large portion of construction costs thus need
periodical inspection (Bosché et al., 2013; Ahmed et al., 2014). A typical MEP scene consists of pipes, pumps, filters,
tanks, valves and other components. Currently, creating as-built BIM for MEP scenes from point clouds is mostly
done manually by professional reverse engineers over a long working period. Compared with other laser scanningdata, the point clouds of MEP scenes have the following features: 1) Huge data volume; 2) Complex environments: alot of noises and clutters; 3) Large-scale occlusion: point cloud of the top part of tanks and pipes near walls are always
missing due to the limitation for laser scanners’ location; 4) The size of the components is inconsistent: tanks are
generally huge while the size of valves are small; The pipes are characterized by its length and slimness. All thesecharacteristics increase the difficulty to automate the reconstruction of MEP models.
This study proposes an automated BIM generation framework for complex MEP scenes, which can directly
consume point cloud data with noise and heavy occlusion. This paper is organized as follows. Section 2 provides
details of our proposed methods to create as-built BIM for complex MEP scenes. An illustrative example to validate
the proposed framework is given in Section 3. The discussion is elaborated in Section 4. Finally, conclusion is drawn
in Section 5.

204

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2. METHOD
This section illustrates the proposed framework of BIM generation based on deep learning technique for complex
MEP scenes based on 3D laser scanning data. Our proposed framework (shown in Figure 1) involves four steps: 1)
automated data segmentation, 2) geometry information extraction for pipes, 3) point cloud alignment for objects withirregular shapes, and 4) parametric BIM model generation. In automated data segmentation process, laser points ofdifferent categories (pipe, filter, tank, and pump) are firstly segmented using our trained deep learning framework
PointNet and then instance point clouds of each category can be achieved using our developed region growing methods.
For geometry information extraction from pipes, a PCA and cross section fitting algorithm is developed. With regard
to objects with irregular shapes like filter, tank and pump, a combined registration algorithm which integrate SAmple
Consensus Initial Alignment (SAC-IA) and Iterative Closest Point (ICP) algorithm is developed to obtain the accurate
information of their positions and poses. Finally, parametric 3D BIM models of the complex MEP scene are generated
automatically with Autodesk Revit and Dynamo.

Figure 1. Proposed framework of as-built BIM generation based on deep learning technique for complex MEP
scenes
2.1 Automatic Segmentation
Segmentation of point cloud is the process of grouping points into multiple classes with similar properties.
Classification and segmentation of the point cloud data are of great importance since they form the starting point for
further processing and better scene understanding. Generally, it can be divided into two levels: semantic segmentationand instance segmentation.
2.1.1

Semantic Segmentation
PointNet (Qi et al., 2016), is a light-weighted and robust end-to-end neural network for various tasks including
point cloud classification, part segmentation and scene semantic segmentation. As this network can directly consume
point clouds, the number of the parameters used in this network is significantly smaller than other deep learning
networks which render point cloud into images or voxels. Compared with images, the most striking difference is thatpoint cloud data in 3D space is unordered and each point is permutation invariant. In order to deal with this issue, maxpooling layer which can be regarded as a single symmetric function was used before the fully connected layer for
selecting informative points and encoding the reasons behind the selection. After mapping the initial coordinate
information to high-dimension space (1024-dimension space) with 1 by 1 kernels, the max pooling layer is applied to
select the critical points which can be interpreted as the skeleton of the object. Following the max pooling layer, fully
connected layers are used for digesting all learnt information for further classification and segmentation. For semantic
segmentation, global feature will be further concatenated with the local embedding for the final output.
Based on PointNet, point clouds of the MEP scene are segmented into four categories including pipe, filter, tankand pump. We train the neural network on model-generated point cloud and use the well trained model to segment the
point clouds collected by laser scanner. The whole pipeline can be divided into 3 steps: 1) Prepare MEP point clouddata for training: considering the great number of points in point cloud, it is time-consuming and tedious to manually205

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019label all the points in the point cloud. Instead, we leverage the as-designed BIM model of the MEP scenes to generate
point cloud with labels. In order to achieving better training result without overfitting, we jitter the point cloud andadd some Gaussian noises to model-generated point cloud data. When we prepare the training and testing data, wecrop the whole industrial space into 1m by 1m blocks and set the center of the block as local origin. Normalized
coordinates can be calculated in the block. Considering the area of the industrial scene is much larger than that ofindoor scenes, it is meaningless to use the global coordinates. Due to that the point cloud data used for training is
generated from the 3D model, the uniform color may cause overfitting for the training process. Based on these twopoints, the global coordinate channels and color channels are not used as input information, only the local coordinates
are preserved as input. 2) Train the neural network using the prepared MEP point cloud data from scratch 3) with thewell trained model, we segment the scanned MEP point cloud to 4 pre-defined classes. The overall classification
accuracy is 84.72%, and mean IoU is 53.09%. The qualitative results are shown in Figure 2.

(a)

(b)

Tank
Pipe
FilterPump(c)Fig. 2. (a) Original point cloud data to be segmented, (b) Ground truth for semantic segmentation, (c) Semantic
segmentation result with well-trained PointNet

2.1.2
Instance Segmentation
From the sematic segmentation result, we firstly retrieve the point clouds labeled as filter, tank and pump. For
the prediction result with some noises, we use region grouping, also known as Euclidean clustering, to remove thewrongly recognized points. This algorithm starts randomly with a seed point in the dataset and search all its neighbors
within certain distance. All the neighbors will be considered as the member of the current cluster and become newseed points. If the number of points in one cluster is smaller than the predetermined threshold, it will be recognized asnoise. Taking filters as an example, the result after removing the noisy points are shown in Figure 3(a). It should benoticed that the point clouds segmented by neural network is sparser than the original one because the neural network
will subsample the point cloud data to a unified number (4096 in our case), which is not conducive to registration
work implemented in later stage. As a solution, after getting clusters without noise, we calculate the mean (x, y, z)
coordinates for each clusters and extract the point clouds in original point sets within predetermined distance to that
position. Through this action, we can extract the dense point clouds for certain category (Figure 3(b)). These instances
will be dealt with robust registration algorithm explained in chapter 2.2.2.
After retrieving all the point clouds labeled as pipe, we firstly estimate the normals and curvatures of the point
clouds with the algorithm given in point cloud library (PCL), which formulate the problem as least-square plane fitting
estimation and solve it by singular value decomposition (SVD). The parallel pipes are segmented into instance levelwith region growing method proposed by Rabbani et al. (Rabbani et al., 2005). The key idea is to find the smoothsurface based on normal and curvature similarity constraints. Considering that different pipes do not contact with each206

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019other after carefully removing the noise points, all winding pipes can be separated successfully. The region growing
result are shown in Figure 3(c). One problem which may be encountered is that one continuous pipe may be regarded
as two parts due to the occlusion issue. In the later stage, after extracting all parameters for the pipes, we can merge
two parts together according the collinearity.

(a)
(b)
(c)
Figure 3. (a) Retrieving point cloud labeled as filter and removing noises by region grouping, (b)According to theposition information given by deep learning result, retrieving dense point cloud for instances in original point set,(c) Region growing result for pipes

2.2 Geometrical Information Extraction and Alignment
2.2.1
Pipes
Each continuous pipe consists of several straight parts and elbows connecting them (3 segments for the pipeshown in Figure 4(a)). Two common methods detecting and fitting cylinders are Hough transform and Random Sample
Consensus (RANSAC). Methods based on RANSAC (Bolles et al., 1981) will randomly select the least points tocalculate the model and check the percentage of inliers according to a designated threshold. After exhausting search,
the algorithm will choose the model with most points within the threshold. The fatal drawback of the algorithms withRANSAC is that they cannot fit multiple models simultaneously, which means each straight pipes need to besegmented manually before implementing these methods. Hough transform (Rabbani et al., 2005; Ahmed et al., 2014;
Bosché et al., 2013; Bosché et al., 2015; Patil et al., 2016), as a voting algorithm, is robust to noises. However, forcylinders, Hough transform methods need to work in 5-dimensional space which introduce high computational
expenses and cannot be implemented efficiently.
In this research, an algorithm is developed to automatically determine the pipeline route according to point cloud
distribution. Compared to those traditional methods which highly rely on the detailed piping layout, our method isdata-based and can be implemented without enough pre-knowledge. Starting from one end of the pipe, all points
within predetermined distance to the starting point will be used for initial analysis (Figure 4(a)). The distance should
be large enough to reflect the direction of the straight parts, but not too big to reach the next corner points (0.3 m inour case). For this portion of the straight pipe, principal component analysis (PCA) is applied to extract the principal
direction of the straight part. Due to the non-uniformity of the point clouds, the direction got from PCA may have alittle bias from the actual centerline. We try to solve this problem by iteratively using circle fitting method. With thetransformation matrix got from PCA, all points within the virtual sphere will be transformed to a local coordinate
system. Afterwards, these points are resampled to several thin slices at certain interval along the axis (Figure 4(e)).
For each slice, using the direct least-square fitting algorithm (Pratt, 1987), the local center and the radius of the circle
can be extracted. The matching result is shown in Figure 4(f). It should be noted that the algorithm will drop thosecross sections where the number of points is too small to reliably determine the circle. The multiple consecutive centerswill be connected and the centerline of the straight pipe can be extracted by fitting a line to the circle centers.
So far, the parameters we got above is only for a small portion of the pipe. With the initial centerline direction,
we can extract all the points within the certain distance to the centerline (1.5R for our case, R is the initial radius
extracted in last step). After that, we can repeat the workflow (PCA and circle fitting) again for the whole straight pipeand get accurate parameters for it including direction, one point on the centerline and pipe radius. After that, point
clouds on this straight part will be removed and one point at the end of this segment will be selected as the new starting
point (Figure 4(c), (d)). In this way, the algorithm can proceed to the next straight segment. Till now, we have theparameters for all straight segments of the pipe. In order to reconstruct model of the piping system, it is necessary to207

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019reconstruct the piping network by calculating the intersection of the pipes (Patil et al., 2016). In the 3D space, twolines do not necessarily intersect with each other. We find the points on one line, which has the smallest distance tothe other line and approximate the intersection point as the average of the two points. After getting all intermediate
points and two end points for a winding pipe, we draw all parameters needed. They can be used for parametric model
generation in the later chapter (Chapter 2.3).

(a)

(b)

(c)

(d)

(e)
(f)
Figure 4. (a) One pipe segmented by region growing, (b) Selecting the starting point and retrieving all points
within certain distance (0.3 m in our case) to the starting point, (c)(d) The endpoint of the last segment become
the new starting point of the next segment, retrieving points in the sphere for initial analysis, (e) Cut the straightpipe into slices at certain interval along the axis, (f) Circle fitting result for the thin slice

2.2.2
Components with Irregular Shape
Different from pipes which are in regular shapes, the other MEP components like filter have extremely irregularshapes. Undoubtedly, these filter components are difficult to extract the geometrical information to create a parametricmodel. Considering this observation, our method is to register or align the point cloud of one component from laser
scanner with multiple same types of point cloud generated by a 3D model library to gain the optimal BIM model;
Take filter type as an example, this registration process can be summarized as two steps. 1) Build the 3D model librarycomprised of meshes and their corresponding point clouds; 3D filter models are gathered and stored in our 3D model
library with the common 3D model format like PCD or OBJ formats. Each type of filter model can be utilized tosample a uniform point cloud. As a result, a number of filter models and their corresponding point clouds are collectedin the 3D library. 2) Register the filter point cloud from laser scanner with multiple filter point clouds from 3D model
library; an optimal point cloud of the filter type with the highest score and the rigid transformation matrix can begained. Therefore, registration or data alignment is the core step of modeling the component with irregular shapes.
Registration in point cloud processing is a process of transforming multiple point cloud datasets into the same
coordinate system aiming at aligning the overlapping components of the datasets. Considering the datasets we process
are mainly comprised of rigid objects, rigid registration is the main registration type we consider in this paper. It canbe modeled by a 4 by 4 rigid transformation matrix made up of 6 Degrees of Freedom (DOF). We adopt the coarseto-fine registration strategy which is commonly leveraged for point clouds registration.

208

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

(a)
(b)
(c)
Figure 5. (a) point cloud of filter collected from laser scanner, (b) point cloud of filter generated from 3D model
library, (c) Registration results using SAC-IA and ICPSpecifically, we integrate SAmple Consensus Initial Alignment (SAC-IA) and Iterative Closest Point (ICP)
algorithm to complete the registration process. SAC-IA is an efficient coarse registration algorithm (Rusu, et al., 2009)
which utilizes a discriminative local descriptor named fast point feature histogram (FPFH) and RANSAC ideas toiteratively find the best rigid transformation of 2 point clouds which yields the best error metric. Details of thisalgorithm is illustrated in his paper (Rusu, et al., 2009). As a consequence, an initial rough alignment can be gained.
Regarding the ICP algorithm, it is an iterative algorithm proposed by Besl and McKay (Besl and McKay, 1981) whichtries to minimize the distances between two clouds of points. However, ICP has a drawback which requires that initial
location of point clouds should be near without large distance differences. If not, ICP might get trapped in a local
minimum and converge slower. Based on these observations, combining SAC-IA and ICP will help avoid the local
minimum faced with ICP.
Based on Point Cloud Library (PCL), we implement this integrated algorithm which take the point cloud fromlaser scanner and multiple point clouds from 3D model library as inputs. And it can output the optimal model andcorresponding matrix for the point cloud from laser scanner. The final registration result for a filter is shown in Figure
5. So far, we can get two rigid transformations respectively from the SAC-IA and ICP registration. The overall
transformation can be computed by multiplying these two transformation matrices. In later stage, this matrix will be
stored in an excel spreadsheet for automatic modeling in dynamo.
2.3 Parametric BIM Generation
Parametric model is a digital model which is determined and pre-programmed by a finite number of parameters.
Specifically, the digital model or its corresponding elements can be generated in automation by a finite of logic
parameters instead of by being manually manipulated. Knowing these parameters, all other information of the model
will be defined. For parametric BIM creation of a typical MEP scene, objects can be categorized into objects withregular shapes (e.g.: pipe type) and those with irregular shapes (e.g.: filter type) considering the regularity of objects’
shapes in the scenes. A pipe whose geometry can be thought of as a cylinder is determined by a radius and two center
points of the ends. As for filter type, the optimal digital BIM model can be gained by alignment method illustrated in
section 2.2.2. Therefore, the key to its modeling is to find the correct position and pose which is determined by therigid object transformation matrix containing 6 parameters (3 rotation and 3 translation parameters).
After extracting geometrical information of each pipe component from point cloud data and obtaining rigid object
transformation matrix of each filter component, these parameters will be stored in an Excel spreadsheet. Furthermore,
self-defined Revit family types are constructed in Revit based on these parameters. For pipe type, a custom pipe Revit
family is created which is determined by a radius and two center points. For filter type, a custom filter Revit family isalso invented so that each filter can be placed on any arbitrary locations with different poses. Finally, a Dynamo
workflow reading all the above parameters in an excel spreadsheet is developed to generate BIM models of a typical
MEP scene. In this workflow, we mainly leverage the Dynamo MEPover and built-in packages to generate pipes, itscorresponding connectors—elbows and place the filters with correct poses in the right positions.

209

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3. RESULTS
3.1 Point Cloud Data
To validate the proposed framework, a complex MEP plant is scanned using Leica BLK360 laser scanner which
is a portable laser reality capture solution capable of generating a full 3D representation in just minutes. The whole
scene of the MEP plant are composed of pipe, filter, tank, pump, other small piping components and clutters. Due tothe complexity of the scanned structure, multiple scans were conducted with over 8 scan locations to reduce occlusions.
Then Leica’s Cyclone Register 360 was leveraged for data pre-processing including noise filtering and scanregistration. The resolution approximates 5 mm point spacing at 10 m distance and data scanning took about 15
minutes in total. The final output is a single BLK format file containing XYZ coordinates, RGB value of each point
and panoramic images. For better illustration of our proposed framework, a typical MEP scene is selected consistingof 31 pipes, 4 filters and some small piping components.

(a)(b)
Figure 6. (a) Point cloud of a typical MEP scene in CloudCompare, (b) BIM model of the scene in Revit
3.2 Conduction of Proposed Framework
First, segment the whole point cloud of the MEP scene into point clouds of 4 categories (pipe, filter, tank, pump)using our trained PointNet deep learning model. Then further segment the point cloud of a category into instances bymeans of our developed region growing algorithms. Henceforth, point cloud instances of pipe, filter, tank, and pump
can be acquired. Second, extract geometric information of pipes; a PCA and sphere-based algorithm is developed to
extract the geometric dimensions from a zigzag long pipe, including at each end. After this step, the geometric
information involving pipe’s radius and two center point’s coordinates will be stored in an Excel file. Third, utilize
the integrated SAC-IA-ICP algorithm to find the optimal fitting model in 3D library for the input point cloud. Similarly,
the transformation parameters will be stored in the Excel file. Finally, a workflow for generating pipes, elbows andplacing filters was developed in dynamo, which is a graphic programming tool working with Autodesk Revit. The
final BIM modeling result is shown in Figure6. (b).
4. DISCUSSION
The modeling of BIM from point cloud data includes 3 aspects: modeling the geometry of components, assigninga category label and other properties to a component and creating relationships between components (Tang, et. Al,2010). In our article, we primarily focus on the parts of geometric modeling and object recognition. Although weachieved a good result with 84.72% accuracy on the automated segmentation using PointNet, more labeled training
datasets in MEP scenes are still needed for avoiding possible overfitting issues. With more labeling datasets, ourlearned model would generalize better on the MEP point cloud data. Therefore, a more robust and higher segmentation
result can be gained. It still requires some manual works in the instance segmentation of pipes since applying region
growing algorithm to the pipe point clouds can over-segment the data. As a result, separated pipes needs beconcatenated to a complete pipe. Different from pipes’ modeling, modeling filter components needs a 3D model
library. If 3D filter models are not rich in the library, the optimal filter model might not be able to gain. However, ourmethod provides a good alternative way to generate BIM models for irregular shapes using registration algorithms. In

210Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

BIM generation using dynamo, we did not extract the geometry information of elbow components but utilized the twopipes which were connected by the elbow component to infer their geometry information.
5. CONCLUSIONSThis paper proposes an automated comprehensive framework to create as-built BIM models for complex MEP
scenes from 3D laser scan data. First, a pioneering deep learning framework named PointNet is adapted, trained fromscratch and acquired a good semantic segmentation result when inputting a point cloud of a complex MEP scene. As
a result, point clouds of different categories including pipe, filter, tank, and pump can be obtained. Second, we
conducted instance segmentation on the point cloud of each category to separate objects using a region growing
algorithm. Then, regarding pipes, we developed a PCA and sphere-based algorithm which is capable of extracting thegeometric dimensions from a zigzag long pipe. For filters, pumps and tanks which are made up of irregular shapes, aregistration algorithm SAC-IA-ICP integrating SAC-IA and ICP algorithm is leveraged to find the optimal model ina 3D model library with the highest fitting score. Finally, Dynamo, a visual programming tool working with Revit, isutilized to automatically accomplish the generation of BIM models based on extracted pipes’ geometric and modeltransformation parameters from the registration process. The proposed framework was validated on a complex MEP
scene composed of multiple pipes and other piping components. During geometry information extraction, 31 out of
31 pipes, 4 filters and other small piping components were extracted successfully.
REFERENCES
Ahmed, M. F., Haas, C. T., & Haas, R. (2014). Automatic detection of cylindrical objects in built facilities, Journalof Computing in Civil Engineering, 28(3), 04014009.
Besl, P. J., & McKay, N. D. (1992). Method for registration of 3-D shapes. In Sensor Fusion IV: Control Paradigms
and Data Structures (Vol. 1611, pp. 586-607).
Bolles, R. C., & Fischler, M. A. (1981, August). A RANSAC-Based Approach to Model Fitting and Its Application
to Finding Cylinders in Range Data. In IJCAI (Vol. 1981, pp. 637-643).
Bosché, F., Guillemet, A., Turkan, Y., Haas, C. T., & Haas, R. (2013). Tracking the built status of MEP works:
Assessing the value of a Scan-vs-BIM system, Journal of computing in civil engineering, 28(4), 05014004.
Bosché, F., Ahmed, M., Turkan, Y., Haas, C. T., & Haas, R. (2015). The value of integrating Scan-to-BIM and
Scan-vs-BIM techniques for construction monitoring using laser scanning and BIM: The case of cylindrical
MEP components, Automation in Construction, 49, 201-213.
Pătrăucean, V., Armeni, I., Nahangi, M., Yeung, J., Brilakis, I., & Haas, C. (2015). State of research in automatic asbuilt modelling, Advanced Engineering Informatics, 29(2), 162-171.
Patil, A. K., Holi, P., Lee, S. K., & Chai, Y. H. (2017). An adaptive approach for the reconstruction and modeling of
as-built 3D pipelines from point clouds, Automation in Construction, 75, 65-78.
Pratt, V. (1987), Direct least-squares fitting of algebraic surfaces. In ACM SIGGRAPH computer graphics (Vol. 21,
No. 4, pp. 145-152).
Rabbani, T., & Van Den Heuvel, F. (2005). Efficient hough transform for automatic detection of cylinders in point
clouds, Isprs Wg Iii/3, Iii/4, 3, 60-65.
Rabbani, T., Van Den Heuvel, F., & Vosselmann, G. (2006). Segmentation of point clouds using smoothness
constraint, International archives of photogrammetry, remote sensing and spatial information sciences,36(5), 248-253.
Rusu, R. B., Blodow, N., & Beetz, M. (2009, May). Fast point feature histograms (FPFH) for 3D registration. In
2009 IEEE International Conference on Robotics and Automation (pp. 3212-3217).
Tang, P., Huber, D., Akinci, B., Lipman, R., & Lytle, A. (2010). Automatic reconstruction of as-built building
information models from laser-scanned point clouds: A review of related techniques, Automation inconstruction, 19(7), 829-843.
Qi, C. R., Su, H., Mo, K., & Guibas, L. J. (2017). Pointnet: Deep learning on point sets for 3d classification andsegmentation, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp.
652-660).

211

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Conversation-based Building Information Delivery System for Facility Management
Kuan-Lin Chen1, Meng-Han Tsai2
1) Graduate Student, Department of Civil and Construction Engineering, National Taiwan University of Science andTechnology, Taiwan. Email: m10705504@mail.ntust.edu.tw
2) Assistant Professor, Department of Civil and Construction Engineering, National Taiwan University of Science andTechnology, Taiwan. Email: menghan@mail.ntust.edu.tw

Abstract: This research aims to develop a conversation-based building information delivery system as a bridge
between equipment maintenance engineers and facility information management platform. At present, most of the
new buildings using BIM technology are handed over the managing system to the property management companies.
On-site equipment maintenance engineers obtain building model information through these management platforms
or emails. However, for an engineer who is on site to repair a particular piece of equipment, there is too much
unnecessary model information in the system and it may cause their work to be inefficient. With the developmentof IT technology, the conversation-based system is more flexible and has been gradually applied in various
engineering fields. We developed a conversation-based system through natural language processing (NLP). By
collecting and training the query keywords that equipment maintenance engineers often use, this conversationbased system can quickly understand the user's query intent. At the same time, this method integrates various tools
for transmitting building information, and equipment maintenance engineers can complete the receiving andquerying equipment models in the same interface. In order to verify the effectiveness of this method, we tested
several devices in a residential building to analyze the benefits of importing conversation-based system during the
FM phase. The results demonstrated that the conversation-based system can correctly identify statements fordifferent sentences but with the same intent and this method can improve the efficiency of equipment maintenanceengineers to retrieve building model information.
Keywords:
Building Information Modeling, Facility management, Conversation-based system, Natural
language processing
1. INTRODUCTION
The system development technology of Building Information Modeling management platform is
gradually mature. On the same platform, people can view 3D models, detailed model information, equipment
maintenance records, reports, and so on. These interfaces for querying model information are mostly limited to thefacility information management platform developed by each company, such as V3DM or WeBIM Sync in Taiwan.
When there is a special condition, the system can notify the relevant equipment maintenance engineers through Email or SMS. After receiving the notification, the equipment maintenance engineers can obtain detailed equipment
by scanning the QR Code on the faulty device. However, the vast amount of information that exists in the databaseis not necessarily a good thing for an equipment maintenance engineer. For example, the equipment maintenanceengineers of the pumping motor only need a few specific information, but the information sent by the system
contains a lot of unnecessary information. These factors may reduce the efficiency of equipment maintenanceengineers in retrieving building model information if lots of equipment needs to be inspected and repaired.
Therefore, the interaction between the user and the operating interface is important. In the case of equipment
maintainers, effective communication improves the efficiency of the job when the equipment maintainer operates
the device that manages the BIM model information. A well-designed conversation-based system needs to consider
the user's operational behavior and motivation. In the past, many conversation-based systems have been used indifferent fields as a tool for effective communication.
The objective of this research is to develop a conversation-based building information delivery system for
retrieving information on building models in the field of facility management. Through the developed
conversation-based system, the conversation-based system can analyze the words input by the user and thentransmit the model information in the database to the user according to the keywords in the semantics. In addition,it also plays the role of interface integration. Equipment maintenance engineers do not have to rely on a specificmanagement platform or other developed apps to query related information that can be done through common chatsoftware. Compared to the way of retrieving model information in the past, the flow of operations is much
simplified, and through natural language processing techniques, retrieval is more efficient in a large model
database. Furthermore, the conversation-based system can also store the user's past operation records, which canbe used as an auxiliary aid for equipment maintenance. Taking a residential building as an example, we use thedeveloped conversation-based system to retrieve the equipment inside the building. The results show that thisconversation-based system can understand the user's intention and quickly find the model information that the user
wants.

212

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2. LITERATURE REVIEW
Applications and problems of BIM in FM stage
The most significant benefit of BIM in the life cycle of a building is in the FM phase. Facility managers
who use the BIM database effectively can get a lot of benefits to help manage large amounts of information. (Azizet al., 2016) However, the application of BIM in the FM phase still has gaps and challenges, and the data is noteffectively utilized. Therefore, BIM and facility management systems must be further integrated. (Naghshbandi,2016) Although most of the software and tools in FM focus on operations and maintenance development, agile
software also plays an important role in the FM industry. The capabilities of equipment management are not only
in maintenance management but also in non-core activities that are also relevant to people, locations and processes.
Agile software provides convenience. However, there are many types of facility management software on themarket that confuse the facility manager when choosing the most suitable software.
Efficiency improvements of retrieving BIM data
Related research is to solve the problem that BIM technology is too inefficient to retrieve information in
the FM stage. For example, develop an integrated system to get information and knowledge about building
maintenance. The system helps maintenance personnel tracking related operational records of building
components and retrieving relevant information about maintenance. (Motawa & Almarshad, 2012) In order toimprove the efficiency of the retrieval model, a semantic search engine BIMSeek was developed. Using contextual
analysis techniques and constructing ontology, an automatic query extension method is proposed to retrieve online
model information. (Gao et al., 2015) Based on natural language processing and ontology to understand thesemantics of the user, and extracting the target keyword and the restriction sequence from the sentences outputtedby the user to improve the retrieval efficiency. (Wu et al., 2019)
Chatbot applications based on conversation-based systems
Although the function of the chatbot is limited, when the scope of the conversation is reduced, the accuracy
and speed of the message transmitted by the chatbot can surpass human. Thus, having a chatbot that is able toretrieve answers for all questions quickly and efficiently is very helpful. (Atiyah et al., 2018) According to thesurvey, users have a high degree of adhesion to communication software. A chatbot is considered a suitable tool
for retrieving data, using keyword search or natural language techniques to understand the intent of users and assist
users in querying data. In recent years, there have been many successful mission-oriented chatbot applications indifferent fields. There are already some successful application cases, such as Machine Learning based chatbots
that make the results more practical, especially using NLP technology to allow chatbots to understand the context
of the conversation, allowing companies or schools to serve as advisors. (Singh et al,.2018) Ask Diana is a chatbot
system consisting of a water-related disaster database, a user intent mechanism, and an intuitive mobile-devicebased user interface. The user can query the chatbot directly or operate the image-based menu to retrieve the flood
information and formulate corresponding strategies effectively. (Tsai et al., 2019) There is also a chatbot for mental
healthcare, which provides conversational services for psychiatric counseling through natural language
understanding (NLU) technology and provides appropriate responses based on different case studies. (Oh et al.,2017)
In short, a chatbot can be used as a tool to effectively transfer information and understand the user's
semantics to assist in retrieving information. It has been used in many different fields, but in the past research,
there were very few chatbots used in civil engineering; and so far, no chatbot has been applied in the FM stage of
BIM technology. Equipment maintenance has a huge amount of data. If the chatbot can be used as an interface forsearching model information, the efficiency of the searching model will be greatly improved. The users also candirectly manipulate and inquire the operation records to maintain the equipment and receive immediate feedback.
3. METHOD
The introduction of the method mainly covers two parts: system architecture and named entity recognition(NER). The system architecture includes the mechanism by which the conversation-based system understands theuser's intent; the flow of natural language processing; and how to obtain BIM model information. NER is part of
the natural-language understanding technology. This paragraph describes how to perform word segmentation,
token tagging, and weight calculation of keywords for user-entered words.
3.1

System architecture
The design of the system architecture can be divided into three major modules, as shown in Figure 1. The
first module is the decision mechanism used by the chatbot. The second module is the module of natural language
processing (NLP). We put the statement marked by the entity into the pre-processed corpus. This corpus is used
to calculate the subsequent similarity. The last module is the BIM source, which is from Forge's cloud platform
and BIM software.

213

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 1. Conversation-based system architecture
3.2

Decision mechanism
The general chatbot can be divided into a rule-based model, a retrieval-based model, and a split model.
The conversation-based system developed in this paper adopts the rule-based model and the retrieval-based model.
With regard to the rule-based model, we provide the user intuitive operation interface and manually write thestatements that the user often enters as a rule for design. Unlike the rule-based model, the retrieval-based model is
responsible for processing the user's input irregularity statement, and determining the type of reply to the user by
calculating the weight of the keyword.
3.3

Natural language processing (NLP)
This module contains Natural Language U1nderstanding (NLU), pre-processed corpus, TF-IDF(similarity calculation) and Natural Language Generation (NLG). Natural language understanding is through word
segmentation, Tokenizer and Named Entity Recognition (NER); and the corpus contains pre-processed statementswith transcripts in it. Based on these processed statements and the language entered by the user, the similarities
are calculated. Finally, Natural Language Generation (NLG) here is based on the similarity value to determine thetype of response to the user's answer. If it is lower than the preset value, the conversation-based system will directly
send the corresponding answer to the user. If it is lower, the conversation-based system will send several possible
answers and then pass them back to the user. The module contains three parts: Named-Entity Recognition (NER),
Word Segmentation and Feature Extraction and Similarity Calculation.
3.3.1

Named-entity recognition (NER)
NER is used to retrieve some of the entities in the text, allowing the machine to automatically find theentities we are interested in. This method helps the computer understand the user's intentions and subsequent
similarity calculations. The following is divided into three paragraphs: defining the scope, word segmentation andfeature extraction, and similarity calculation.
3.3.2

Word segmentation and feature extraction
We disassemble each of the collected corpora, and after obtaining these words, we use the rule-basedtagging to label each word. For example, the equipment manufacturer and equipment number will be in the
categories of equipment information, and the pump and air conditioner will be in the category of equipment name.
Take a model information as an example. Table 1 is about the pumping machine. The model information of thepump contains length, width, height, etc., that are used to describe the shape of the model. We classify these pieces
of model information as the same class. Information such as device photos, device names, floors, etc., is used torepresent the different events of the model; we also attribute them to different classes.

214

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 1. Entity classification
Pump related information
Entity name
Length, width, height…
Equipment information
1F,2F,3F…
Floor location
Pump, hydro-exhauster…
DevicePicture, Image…
Picture3.3.3

Similarity calculation
With these named-entity tags, we can use these modules as a benchmark for similarity comparisons. We
use inverse document frequency (TF-IDF) to find the important keywords in the sentence by word frequency. The
text similarity is calculated by Term Frequency - Inverse Document Frequency (TF-IDF). The TF-IDF algorithm
consists of two parts, word frequency calculation (TF) and inverse document frequency (IDF). The formulas areas (1), (2), (3).
𝑐𝑜𝑢𝑛𝑡(𝑤)tf𝑤,Di = |𝐷 |(1)𝑖ⅈ ⅆf𝑤 = log

𝑁𝑁

1+∑𝑖=1 𝐼(𝑤,𝐷𝑖 )tf − ⅈⅆf𝑤,Di = tf𝑤,Di × ⅈ ⅆf𝑤
⚫
⚫
⚫
⚫
⚫

(2)
(3)

tf𝑤,Di : The frequency at which a particular entity name (𝑤) appears in sentence (𝐷𝑖 ).
𝑐𝑜𝑢𝑛𝑡(𝑤): The number of occurrences of the entity name (𝑤).
𝐷𝑖 : The number of all words in a sentence.
𝑁: Total number of statements in the corpus.
𝐼(𝑤, 𝐷𝑖 ): Whether the statement 𝐷𝑖 contains the entity name in the corpus(If included, the value is 1 and the value is 0).

Using the same method applied to the statement as entered by the user, we find the important keywords
in the user statement. Next, the cosine similarity (formula 4) is used to calculate the similarity to the statements inthe corpus and determine the corresponding answer that best matches the user's intention. Cosine similarity is avector generated based on the word frequency in the TF-IDF algorithm. Next, the cosine similarity of the two
vectors is calculated. A larger value means that the two statements are similar.
ncos θ =⚫
⚫
⚫

∑i=1(Ai ×Bi)
n

n

(4)

√∑i=1(Ai )2 x√∑i=1(Bi )2

Ai : In the user statement Ai , the frequency of each word after being disassembled.
Bi : In the statement of the corpus, the frequency of each word ⅈ after being disassembled.
cosθ: The closer the angle is to 0 degrees, the more similar the two vectors are.

3.4

BIM Source
The BIM source that mainly provides the user's answer consists of two parts. The first one is the Forgecloud platform. The cloud platform integrates the BIM software information and related reports, and the crawler
sends the information to the conversation-based system through the open API. The second part is to directly export
the model data of the BIM-related software into a specific IFC format and then store it in the database, and provide
answers according to the user's intention. It confirms the user's intentions and information through a decisionmaking mechanism. Finally, the logical form of the natural-language generation (NLG) is processed to generate areply to the user's statement. The message content of these statements is based on the source of the BIM data.
It is important to define the range of languages that we want to deal with; and within a certain range, theaccuracy of entity identification can be improved. In this study, we designed the natural language response
mechanism to be the domain knowledge, focusing on the knowledge base of equipment maintenance. Entities aretagged based on terminology, query terms, or building model information commonly used by equipment
maintainers.
4. IMPLEMENTATION
We implemented a question and answer system for retrieving the BIM model to test the feasibility of the
proposed method. First, we collect the sentences and terminology commonly used by equipment maintenanceengineers to construct a dictionary in the knowledge base. Based on this constructed dictionary, we candisassemble and calculate the similarity of the user's statements. Among them are the following suite technologies:

215

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019⚫
⚫

Jieba: The Python-based Chinese word segmentation library. In addition to the included word segmentation
algorithm, it also provides us with a customized vocabulary.
Genism: A free python module dedicated to working with raw, unstructured text that automatically extracts
semantic topics from documents. The library contains Word2vector for vectorizing words.

The sentences in Table 2 are several marking processes for identifying entities and user intent. We divide
into two different intents for training: query device information and query model pictures. Within each intent,
different statements are manually set with entity tags. The first intent is to query device model picture, whichcontains the entities for the search, device, photo, and model. The second intent is to query device information
including retrieval, device, and device information.
Table 2. Marking entity
Intent 1: Query device model picture
Intent 2: Query device information
I want to <find> the (picture) of the [pump].
I want to <find> the #length# of the [pump].
<Search> for [pump]’s (picture).
I want to <find> the #weight# of the [pump].
(Picture) of the [pump].
Please <give> me the #date# of #manufacture# of the[pump].
The 3D {model} of [pump].
#length# of the [pump].
[Pump]’s {3Dmodel}.
#weight# of the [pump].
Help me <find> the (picture) of the [pump].
#manufacturer# of the [pump].
The (picture) of the [pump].
#length# [pump].
I want to <find> the (picture) of the [pump].
#weight# [pump].
[Pump] 3D {model}#manufacturer# [pump].
[PUMP] {model}
Help me <find> the #length# of the [pump].
<Search> for [pump]’s (image).
Who is the #manufacturer# of this [pump]?
(image) of the [pump].
<Search> for #length# of this [pump]?
[Pump] 3D {model}#MANUFACTURER# [PUMP].
The (image) of the [pump].
#WEIGHT# [PUMP].
(image) [PUMP]#LENGTH# [PUMP].
Each symbol in the sentence: <> represents the entity of Query, () represents the picture, ## represents device
information, [] represents the device name, {} represents the model in BIM.
5. RESULT AND DISCUSSION
We tried a few different sentences to test the models we trained. The result is shown in Figure 2, that themodule can identify the entities in the statement and classify them into the correct intent. We divide the intent into
Query device information and Query device model picture; and entities are divided into query, device, equipment
information, and picture. Case1 shows that when the user expresses the " Help me find the length of the pump."
statement, the conversation-based system recognizes that " Query device information " is the user's real intention
through a higher intention score (0.989). Case2 and Case3 show that users use different expressions but have thesame query intent. The user in Case2 expresses " I want to search the pump 3D model.", and the user in case3
expresses " I need to find the model picture of the pump.", but their user intent is classified as " Query device
model picture ".

Figure 2. Identify entities and intentions216

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Finally, we connect the LINE platform chatbot in series and take a query pumping device information as
an example to test the sentences input by different equipment maintenance engineers. The test results of theconversation-based system are shown in Figure 3. In the first picture, the user expresses "Give me the length, widthand height of pump1"; the second picture, the user expresses "Pump1 length, width, height"; the third picture, theuser only expresses "I need the length, width and height of PUMP1". The results demonstrated that theconversation-based system successfully understands the user's intentions. Through different sentences but the same
intent, the system can accurately provide the BIM model information that the user wants. The information sent by
the chatbot is a menu. In addition to containing important model information, the menu provides users with moredetailed content, links to model data, and pictures of the building model. Users can click on the menu directly or
choose to enter other keywords to query.

Figure 3. Test results of the conversation-based system
5. CONCLUSIONSThis research proposes a conversation-based building information delivery system to assist on-siteequipment maintenance engineers to obtain model information efficiently from the facility information
management platform. This system consists of three modules, including decision mechanisms, natural language
processing and BIM source modules. Decision mechanisms help classify user-entered statements into key-basedor natural-language processing logic. The natural language processing module determines the real intent of theuser by training the collected corpus. The BIM source module collects model information from different BIM
software through a database to manage large model data. We used three different cases to test the feasibility of thismethod. The results show that for different statements entered by the user, the conversation-based system can findthe words marked in the statement and calculate the similarity to find the corresponding user intent. With theconversation-based system we developed, the task of obtaining BIM model information for the facility
management stage can be completed on a common chat platform, and through Natural Language Processing (NLP),
the efficiency of retrieving the building model is improved. In future work, we will design a complete user interface
and add more conversation-based system features, such as instant push information, emergency notifications, etc.
Moreover, this conversation-based building information delivery system will be applied to the equipment
management of the building to assist the maintenance of on-site equipment maintenance engineers.
ACKNOWLEDGMENTS
This work was financially supported by the Taiwan Building Technology Center from The Featured Areas
Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of
Education in Taiwan.

217

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

REFERENCES
Atiyah, A., Jusoh, S., and Almajali, S. (2018). An Efficient Search for Context-Based Chatbots, The 8thInternational Conference on Computer Science & Information Technology, DOI:
10.1109/CSIT.2018.8486187
Aziz, N.D., Nawawi, A.H., and Ariff, N.R.M. (2016). Building Information Modelling (BIM) in Facilities
Management: Opportunities to be Considered by Facility Managers, AMER International Conference on
Quality of Life, 234,353 – 362.
Gao, G., Liu, Y.S., Lin, P., Wang, M., Gu, M., and Yong, J. H. (2015). A query expansion method for retrieving
online BIM resources based on Industry Foundation Classes, Automation in Construction, 56, 14-25.
Motawa, I. and Almarshad, A. (2013). A knowledge-based BIM system for building maintenance, Automation inConstruction, 29, 173-182.
Naghshbandi, S.N. (2016). BIM for Facility Management: Challenges and Research Gaps, Civil Engineering
Journal, 2 (12), 679-684.
Oh, K.J., Lee, D.K., KO, B.S., and Choi H.J. (2017). A Chatbot for Psychiatric Counseling in Mental Healthcare
Service Based on Emotional Dialogue Analysis and Sentence Generation, 2017 IEEE 18th International
Conference on Mobile Data Management, 371-375.
Singh, R., Paste, M., Shinde, N., Patel, H., and Mishra, N. (2018). Chatbot using TensorFlow for small
Businesses, 2018 Second International Conference on Inventive Communication and Computational
Technologies (ICICCT), DOI: 10.1109/ICICCT.2018.8472998.
Tsai, M.-H., Chen, J. Y. and Kang, S.-C. (2019). Ask Diana: A Keyword-Based Chatbot System for WaterRelated Disaster Management, Water, 11(2), 234.
Wu, S., Shen, Q., Deng, Y., Cheng, J. (2019). Natural-language-based intelligent retrieval engine for BIM object
database, Computers in Industry, 108, 73-88.

218

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

DEVELOPMENT OF A NEW DISTRICT IN THE STATE OF HESSEN“FLEXQUARTIER” PROJECTJose Alberto Lagunes Ronzon1, Moritz Hofmann2, Milena Potpara3, Dirk Metzger4 and Joaquin Diaz5
1) Master student, Department of Architecture and Civil Engineering, Technische Hochschule Mittelhessen, Giessen, Germany.
Email: jose.lagunes.ronzon@bau.thm.de2) Master student, Department of Architecture and Civil Engineering, Technische Hochschule Mittelhessen, Giessen, Germany.
Email: moritz.fabio.hofmann@bau.thm.de3) Ph. D. student, Department of Architecture and Civil Engineering, Technische Hochschule Mittelhessen, Giessen, Germany.
Email: milena.potpara@bau.thm.de4) Professor, Department of Architecture and Civil Engineering, Technische Hochschule Mittelhessen, Giessen, Germany.
Email: dirk.jens.dieter.metzger@bau.thm.de5) Ph. D. Professor, Department of Architecture and Civil Engineering, Technische Hochschule Mittelhessen, Giessen,
Germany. Email: joaquin.diaz@bau.thm.de

Abstract: A new district is planned to be built in Hessen, a state in Germany. This new district is called“FlexQuartier” and it is intended to include residential units and other kinds of floor uses. It is a goal of theGovernment of Germany to develop new districts with characteristics based on smart city and sustainability
principles. The main technology in this project is an innovative High-Temperature Storage System in combination
with a battery storage and a water tank storing hot water, this combined with the use of a Quartier Information
Model (QIM). This papers explains the steps that are being followed in the beginning of the project (conception
phase) and also the steps that will be followed in order to create a model that builds upon itself and can store not
only information about the district, but also can help in the design, construction, commissioning and management
phases of the project.
Keywords: FlexQuartier, Smart Cities, BIM, QIM, High-Temperature Storage, Interoperability, Sensors, Energy,
Facility management.
1. INTRODUCTION
As a part of the Energy Research Program of the Federal Government of Germany, in the State of Hessen,a new district is planned to be built. This new urban district with a total of 7.9 ha. will be built on an inner area ofthe city of Giessen with over 400 residential units and more than 38,000 m² of floor space. The project is being
designed forming an energy efficiency district with contemporary features as well as innovative solutions in thefield of energy and system services. It is called “FlexQuartier” (Figure 1).
For the first time in Germany, there is a combination of the concepts of BIM, 5D methodology, energy
efficiency and Smart Cities planned altogether following a new methodology gathering the needed information
continuously in a QIM. The key technology of the “FlexQuartier” is an innovative hybrid storage system consistingof a novel high-temperature storage device (HTS) in combination with a state-of-the-art battery storage forelectricity and a large-volume hot water stratified storage tank for the waste heat of the HTS. The innovative hightemperature storage system (HTS) will be used to take care of the fact that the volatility of renewable energies
requires innovative and cost-effective energy storage and the possibility to include this HTS system in the market
of energy storage devices has to be analyzed and proved.

Figure 1. FlexQuartier preconcept.

219

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

In the planning phase, a system for the representation of the entire neighborhood in a virtual neighborhood
information model, a “Quartier Information Model” (QIM) based on the Building Information Modeling (BIM)
methodology, will be developed and applied, also including aspects of energy monitoring in order to obtain an
integral methodology for the planning process and management of the “FlexQuartier” project. The requirementsof the monitoring of the urban district must, therefore, be taken into account in the concept and in the planningphase as well as in the realization phase. The QIM will be created and used at the beginning of the project and in
this phase the model will be called Conception Model; during the planning and construction, three models will becreated sequentially, these will be the Planning Model, Realization Model and Commissioning Model; and at theend, for the management of the new district, a Management Model will be created. The operation of the energy
system will be carried out with the Management Model.
All the models will be built upon themselves, this means, the Conception Model created at the conception
phase will then become the Management Model at the end of the project. These models will be compatible withthe open source format IFC so that they can be transferred and used in other software so that a series of simulations
and analyses can be performed, e. g. to quantify the size of the energy storage system and the whole usage ofenergy in general. An important objective of the project is developing a methodology that is capable of being
replicable outside of Germany, and not only in the European context but also in any other city around the world.
There are several stakeholders from the state involved in the project, these are: the Technische Hochschule
Mittelhessen (THM): for the High-temperature storage, reconversion, heating networks; the network
serviceability, energy and system services; and for the developing of the “Quartier Information Model” (QIM) and
Building Information Modeling. Also, the City of Giessen as main beneficiary of the implementation of thesedifferent technologies and Smart Power GmbH & Co. KG as one of the main contractors for energy supply in theState of Hessen.
In this paper, it will be explained: the methodology used for the design of the “FlexQuartier”, the different
parameters to be used for the simulation of the energy in the district as well as a brief analysis of the benefits.
2. STATE OF THE ART
In recent years, the concept of smart sustainable cities has become increasingly important and getting
worldwide attention as a response to the challenge of urban sustainability. However, this is applicable largely tothe technologically advanced nations (Bibri and Krogstie, 2017). According to the United Nations, 66% of theworld population will live in cities by 2050 (United Nations, 2015). Cities consume about 70% of the world’s
resources and since the citizens are major consumers of energy they become important contributors to greenhousegas (GHG) emissions due to the amount of urban population and the economic and social activities, in addition to
the inefficiency of the built environment (Bibri and Krogstie, 2017). Recent studies have determined that by
optimizing the operation and management of the energy in buildings and houses, it can be saved up to between 20
and 30% of the energy consumption. All of this can be made without changing the buildings structure and/or the
hardware configuration (Guan, Xu and Jia, 2010). This means that the way cities are build and managed has to befaster and more efficient providing the necessary spaces and services that the large number of people will demandin the future. Smart sustainable cities seek creating healthy, livable, and prosperous human environments with aminimum consumption of resources (energy, material, etc.) and a minimum impact on the environment (Bibri and
Krogstie, 2017). It is necessary to implement technological tools in order to achieve this goal.
One of the tools that is revolutionizing the construction and facility management industries is BIM
software. It has many advantages, some of them are the geometric and semantic information that the different
objects inside the BIM models possesses about the building through its life cycle (Song et al., 2017). By using thismethodology, the different stakeholders involved in the project have a common goal of achieving complete androbust systems providing an improved collaboration and, with this, enhance the efficiency and create more positive
impacts in future projects (Tah, Oti and Abanda, 2018). The urban 3D models nowadays are a great tool for thecreation of Smart City platforms for the proper management of neighborhoods and cities and the planning anddesign of urban developments (Álvarez et al., 2018).
The limitation of BIM software is its capability to process complex information inside one single software.
It is very easy to store information such as cost of equipment, name of manufacturers, dates for maintenance, etc.
inside software such as Revit or ArchiCAD. The design intent is often defined quantitatively, and this allows abuilding model to be used to check for these requirements. For qualitative requirements (e.g. this space should benear another or a bathroom that must comply with government regulations, etc.), the 3D model can also support
automatic evaluations (Eastman, Charles M., et al., 2018). However, to make complex analyses such as calculating
the amount of energy that a whole building will generate and use, is when these programs find themselves
exceeded. To tackle this problem, the industry has created external programs or plug in’s that can be used toperform the more complex tasks. To do this, it is necessary to load the information of the buildings, in most of thecases, in an universal standard format or Industry Foundation Classes (IFC) (Ignatova, Zotkin and Zotkina, 2018).
The combination of BIM and external software allows us to extend the information that the models have.
The early integration of the geometric and semantic data become valuable to the facility management teams during

220Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

the district’s operations, especially when it comes to monitoring district performance (Pärn, Edwards and Sing,2017). To monitor and eventually improve this performance, the models are capable of having information about
the different energy storage systems. Several technologies for energy storage exist now in the market, (e. g.
batteries, superconducting magnetic energy storage, super capacitors, flywheels, molten salt power towers, etc.).
All of these technologies have advantages and disadvantages, while some are extremely expensive, others canstore a little energy and others have a quick discharge rate. The combination of a High Temperature system witha battery storage for electricity and a large-volume hot water stratified storage tank has to be further reviewed.
3. DEVELOPMENT
3.1 General overview
The city of Giessen, which is located in the State of Hessen, and SWG AG, as a municipal utility company,
are seeking an answer to the question of what the energy-efficient, as renewable as possible-supplied andenergetically integrated urban district of the future can look like. Due to the long useful life of neighborhoods(more than 200 years) (Figure 2), planned neighborhoods should already be measured against the energy policy
goals for 2050. The participating institutes contribute with their interdisciplinary scientific expertise for thedevelopment of sustainable solutions.

Figure 2. Lifespan of urban neighborhoods: Development of a new, sustainable and intelligent urban district withat least 200 years of imputed life expectancy.
The interoperability of the whole district has to be considered from a holistic point of view. There arenecessary intermediate steps achieving a sustainable operation of the neighborhood. These are the virtual step-bystep development of a conception model, a planning model, a realization model, a commissioning model and amanagement model. These models must all build on each other and be integrated together. The models(Conception, Planning, Realization, Commissioning and Management) have different phases: District phase,
Building Phase and Management Phase (Figure 3).

Figure 3. Different models and phases
Each of these different models have a series of steps to be followed in order to achieve complete
integration between one another:(1) Conception Model: assessment of boundary conditions (such as local climate data, urban planning,etc.), agreement/definition of minimum energy standards, building typological differentiation in groups/clusters(e. g. terraced house, multi-family residential building, public institution, etc.) on the basis of basic urban planning221

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

data, definition of cluster typical energy profiles, conception/creation of a district model (QIM), transmission ofcluster typical energy profiles - specific energy needs, definition of cluster-typical energy gains (PV, solar
radiation, etc.), integration information storage technology, definition/integration of control systems, evaluationof the overall model with forecast of the energetic behavior.
(2) Planning Model: definition requirements for individual buildings according to minimum energy
standards, definition of requirements components according to the minimum energy standard, definition of
integration sensor technology in components and anchoring in the overall system, integration in model of sensors,
definition requirements modeling/planning of individual buildings, consulting planning/modeling, integration of
the individual building plans as replacement of the clusters (building instances) with changes/optimizations in theoverall model, gradual evaluation of the overall model with forecast of the energetic behavior(3) Realization Model: preparation based on the detail of the building modeling and definition of theexecution-specific requirements, tender of the project (based on model).
(4) Commissioning Model: realization of the “As-Builts” in order to be transferred to the QIM, integrate
all the communication connection scenarios into a Quartier Information System (QIS).
(5) Management Model: transfer of the Quartier Information System (QIS) into the operating phase,
optimization during operation.
The use of the QIM makes it possible for the first time to taxonomically relate the formerly separate
disciplines of power generation, distribution, storage (energy system technology) and energy requirements at
district, infrastructure and building level (architecture and building infrastructure) together (Figure 4). The
different models that will be developed must be interoperable between different software. The THM team, involvedin the planning part of the project, is working on the compatibility with the IFC format. The project can be used
serving as a basis for other cities around the world who may want to replicate this kind of project.

Figure 4. Taxonomy of the QIMWith the use of the IFC format, there will be different kinds of exports during the lifecycle of the district.
During the Conception Phase, the IFC exports will be used for, for example, the purposes of simulating the districts
energy behavior and also the different attributes of the whole elements inside. During the Building Phase, the IFC
exports will be made for the planners to have the precise scope of the project and they can create the 3D models
accordingly; contractors and subcontractors to have the necessary information to develop schedules and costestimations as precise as possible; for building owners to have the required information for a smoothcommissioning of the properties and also for them to be able to quantify their new spaces for leasing, selling or
operation purposes. During the Management Phase, the amount of information contained in the model will be sohigh that not every stakeholder will be interested or have the permission to access to this, it is for this reason, that
the IFC exports can be customizable for the different purposes. For example: in the Management Phase, the project
team can easily program the BIM software to create three different types of IFC exports: IFC_Ow for owners,
IFC_FM for facility managers and IFC_EU for end users, each one of this models containing the right information
for the right stakeholder without risking any breach of private information.
3.2 District Phase
The first QIM is the Conception Model, in this, different clusters of buildings will be created according
to the requirements of the project. In figure 5, for example, the red cluster are residential buildings with two wallsthat are adjacent to another building and two walls in contact with the exterior. The green cluster are residentialbuildings with one wall adjacent to another building and three walls in contact with the exterior. This information
is important for the calculation of the energy consumption of the buildings.

222

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 5. Different cubes in clusters.
In order to create a QIM that will be used and improved, a series of information inputs have to be taken
into consideration in each one of the different phases. For the first model, the Conception Model, characteristics
that are mandatory according to German law, characteristics of the site, characteristics of the usage of the buildings,
etc., were taken into consideration.
The “Grundflächenzahl” (GRZ) (in English: Site occupancy index) and the “Geschossflächenzahl” (GFZ)(in English: Floor space index) are two inputs that are given by the German law “Verordnung über die bauliche
Nutzung der Grundstücke § 19 Grundflächenzahl zulässige Grundfläche” (in English: Ordinance on the Structural
Use of Land § 19 Base area number, permissible ground area). The GRZ is the base area number which indicatesthe area proportional of a building site that may be overbuilt, it is a dimensionless number. Example: for a GRZ
of 0.6 it means that maximum 60% of the land area may be overbuilt. In this same scenario, if a building inside aproperty has a floor area of 300 m² and the area of the property where the building is equals to 500 m², then theratio between the two numbers is 0.6.
The GFZ is the number that indicates the ratio of the total floor area of all solid floors of the buildings on
the area of the building plot, it is a dimensionless number. Example: a property has an area of 500 m² and a GFZ
of 1.0, the sum of the floor area in all buildings located on the property must therefore also be 500 m², this means,
one could build a four story building with 125 m² of floor space per floor (4 × 125 m² = 500 m²). In the same
example, if the GFZ is 0.5, the maximum allowed total floor area would be 250 m² (500 m² × 0.5 = 250 m²).
Other important values considered are: the number of people that will use the building(“Bewohneranzahl”), the width of the building (“Breite”), total energy demand (“Gesamtenergiebedarf”), numberof story’s (“Geschosse gesamt”), ground area (“Grundfläche”), construction area (“Konstruktionsfläche”), among
others. Figure 6 shows the information for one of the buildings or “cubes” in the Conception Model and how the
information is stored.
Due to the current nature of the land purchase agreements, photovoltaic systems will be installed on atleast 50% of the residential roof areas. Depending on the design of the commercial buildings in the district, between
500 and 1,500 kW of photovoltaic capacity will be installed, which is going to be available as a secured installedcapacity within the project.

Figure 6. Attributes of one cube.

223

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The hybrid storage system that will be build has to be capable of storing the necessary energy for theproper operation of the new district in the times when the energy will not be produced from the PV panels (e. g. atnight and during cloudy days). The district's energy supply, which has to be as renewable as possible, combinedwith efficient building services, results in temporary energy surpluses, which can be stored in the hybrid storage
system both in the short and long term (Figure 7). The city of GI is also planning electric cars charging points inthe western part of the district, the THM will set up four charging stations directly at the power station. There will
be further charging stations in the planned parking garage in the eastern part of the quarter. These charging stations
force the district to have an amount of available energy at all times.
An evaluation will be made from an overall urban planning perspective in order to work out connections
between urban development, municipal energy supply infrastructure, sustainable housing and the mobility sector
and inner-city integration. The use of electric chargers located in different points of the district for electric vehicles
must also be taken into consideration. Conceptually, the use of new information and communication technologies
must already be taken into account in the planning of the district in order to control the entire energy systemtechnology including the high-temperature storage during later operation. In the conception model, the electric
mobility and the associated charging infrastructure of the modern urban quarter, which must also be included in
the concept, can then be considered. Future requirements must be implemented, and new technologies integrated.
Most of the buildings will have different sorts of sensors to help in the future with the monitoring and control ofthe district. In this conception phase, three sensors are already taken into account in the model: “Sensor 1 –Temperatur”, for temperature; “Sensor 2 – Wasserzähler”, for water consumption; and a non-compulsory sensor,“Sensor 3 – Gaszähler”, for gas consumption. The plan from the project team is for this data to be sent in real time
directly to the QIM model for the district managers to have the input as accurate and timely as possible.

Figure 7. Hybrid Energy Storage System interaction
With all the data included in the conception model, and with the help of external software (e. g. iTWO),
it is possible to simulate the energy production, transportation, consumption and storage of the entire district.
Figure 8 shows how it is possible to get the sum of the residential building’s rooftops from the entire district. Thenthe model can be programed to have 50% (or any other input that the planners want to give it) of residential
rooftops as PV panels. Having the amount of PV panels, planners can then simulate the amount of energy
production of the district by changing several inputs such as the percentage of PV panels in roofs, and the GRZ
and GFZ ratios. Then a simulation of the whole district can be performed. For the FlexQuartier project, it is ofgreat importance to determine the size of the HTS for the energy storage, with a simulation, the project team will
determine the size of the device as well as the performance that it will have for the next 200 years of operation.
3.3 Building Phase
Once the Conception Phase concludes, during the Building Phase, the Conception Model will be first
converted into the Planning Model, with all the information gathered in the Conception Model, the city planners
will have a broad understanding of how the district will perform with the given conditions. Also, if there is an
unexpected change, thanks to the nature of the model, city planners will have the ability to change the configurationof the district if they need to. This process can be as iterative as the district requires.

224

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 8. Programming of residential rooftops in iTWO.
During the construction, as it has been studied, the use of BIM with methodologies, like Lean, is becomingof great importance in the construction industry. Lean provides the principles to be followed and when usedcorrectly, it can help by getting quality right the first time, improve upstream flow variability, reduce production
variability, reduce inventory, increase flexibility, reduce change, use an appropriate production control approach,use pull systems, standardize, promote continuous improvement, use visual management, simulate production
processes, design the production system for flow and value, among others. With this approach, contractors will
have a model that will provide with all the necessary inputs that may be needed in order to carry out theconstruction process with the least amount of changes or unexpected issues. Subcontractors can benefit from this
too, since they can start with the production of the necessary materials from a very early stage of the project.
District managers and the city of Giessen will make sure during the commissioning of the project that all
systems and components were designed, built, installed, tested, and maintained according to the operational
requirements. Thanks to the information in the model, the city of Giessen will be sure that the handover isperformed in a safe and orderly manner from the contractors to the end users, guaranteeing its operability in termsof performance, information traceability, reliability and safety.
The Commissioning Model can help in the logistics of the delivery from the contractor to the owner since
they can easily create punch lists to detect any possible missing document, object or information. The attributes
mentioned before can help the facility managers to control which equipment was recently maintained, when doesit needs to be maintained, how much can it cost, who will be the technician to make the job and how long will theequipment be useful, all with one single model.
3.4 Management Phase
Once the building is delivered and the operation and management begins, end users and owners can get
help from the Management Model, they can easily plan, perform and keep track of the maintenance of the facilities,
assess the impact of system failures and taking quick decisions to solve them, visually assess the areas that can beaffected with future renovations or adaptations, etc. District managers will have in real time inputs from the sensors
installed along the district, helping them with the monitoring and control of water, gas and temperature (for theenergy consumption). Facility managers will have a visual tool to assess all the elements of the buildings before
city inspectors or auditors can detect any problem regarding any component of the building.
This same model can later be used by facility managers not only for the energy management, but also for
simulations such as crowd behavior tools and emergency evacuations or response to certain events (e. g. a terrorist
attack, an earthquake, a flooding). They can, for example, use the GRZ and GFZ values as an aid to calculate thepossible renting spaces (e. g. in the case of an office building). With these same values, facility managers candetermine how much area needs to be cleaned and calculate the amount of people to do the job or determine theprice that it will cost to pay an external company to do the cleaning.
With the amount of information contained in the Management Model, facility managers can calculate thecost of maintenance per floor of used space in the same way that district managers can calculate the cost of
maintenance of the whole district per building and house.

225Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

4. CONCLUSION
The FlexQuartier project has the potential of becoming a role model for future projects, not only inGermany, but also in the rest of the world. The stakeholders must pay close attention to the boundaries and theinformation needed in order to make this a smart sustainable city. There is not a list of specific characteristics orrequirements that need to be fulfilled to call a city “smart”, however, as the time passes by, more and more literature
about the subject is developed. The city of Giessen is making a great effort to create its first smart district which
can lead to a full smart city concept.
All cities change. New buildings are built, new recreational areas are created, and people come and go.
This constantly changing environment makes it difficult for governments to keep up with the changes. With one
QIM model (which as we saw will change from conception to management) during the Conception Phase, theplanners can calculate with a high level of certainty the amount of energy, materials, and money that will be
required to develop such a project. During the Building Phase, they can estimate with high precision the time itwill take to build the entire district as well as the cost of each of the construction phases. During construction, thecontractors and subcontractors will be able to tackle the changes that during construction will happen (because if
there is a general consensus in project management, it is that changes are inevitable), also they can have access to
the latest versions of the models in real time due to the interoperability of them. In the commissioning of thebuildings, contractors and district managers will be able to provide with all the communications and services thatthe district will require, as well as the as-built model. During the Management phase, the city of Giessen and the
district managers will have an innovative tool that allows them to tackle changes in a real time manner as well as
have the total control of the energy that in the district gets generated, transported, spent and stored. They will have
inputs from the sensors installed to monitor and control the district and take well thought decisions with thisinformation and this will enable them to take quick decisions.
We can conclude that the use of BIM in combination with other technologies can improve significantly
all phases of a project. With the implementation of 5G networks and Internet of Things (IoT) it will be possible
for end users of the buildings to interact with the district. With the implementation of machine learning andartificial intelligence, end users can easily get information from the district about particular conditions in the
district, like traffic, possible maintenance to roads, problems with energy, special events happening in the district,new businesses opening, etc. The possibilities for the usage of BIM are endless and right now we are just looking
at the surface of it all and realizing the extents of its capabilities.
5. REFERENCES
Álvarez, M. et al. (2018) “3D Urban Virtual Models generation methodology for smart cities,” Informes de laConstrucción, 70(549), pp. e237–e237. doi: 10.3989/id.56528.
Bibri, S. E. and Krogstie, J. (2017) “Smart sustainable cities of the future: An extensive interdisciplinary
literature review,” Sustainable Cities and Society. Elsevier B.V., pp. 183–212. doi:
10.1016/j.scs.2017.02.016.
Eastman, Charles M., et al. BIM handbook: a guide to building information modeling for owners, managers,
designers, engineers and contractors. Hoboken, New Jersey: Wiley, 2018. Print.
Guan, X., Xu, Z. and Jia, Q. S. (2010) “Energy-efficient buildings facilitated by microgrid,” IEEE Transactions
on Smart Grid. IEEE, 1(3), pp. 243–252. doi: 10.1109/TSG.2010.2083705.
Ignatova, E., Zotkin, S. and Zotkina, I. (2018) “The extraction and processing of BIM data,” IOP Conference
Series: Materials Science and Engineering, 365(6). doi: 10.1088/1757-899X/365/6/062033.
Pärn, E. A., Edwards, D. J. and Sing, M. C. P. (2017) “The building information modelling trajectory in facilities
management: A review,” Automation in Construction. Elsevier B.V., 75, pp. 45–55. doi:
10.1016/j.autcon.2016.12.003.
Song, Y. et al. (2017) “Trends and Opportunities of BIM-GIS Integration in the Architecture, Engineering andConstruction Industry: A Review from a Spatio-Temporal Statistical Perspective,” ISPRS International
Journal of Geo-Information, 6(12), p. 397. doi: 10.3390/ijgi6120397.
Tah, J. H. M., Oti, A. H. and Abanda, F. H. (2018) “A state-of-the-art review of built environment information
modelling (BeIM),” Organization, Technology and Management in Construction: an International
Journal, 9(1), pp. 1638–1654. doi: 10.1515/otmcj-2016-0030.
United Nations. (2015). World urbanization prospects. the 2014 revision. New York: Department of Economic
and Social Affairs. http://esa.un.org/unpd/wup/ Publications/Files/WUP2014-Report.pdf (Accessed
08.05.2019).
World Economic Forum, 2015. WEF report 2015.

226

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

PROCESS RE-ENGINEERING IN OWNER ORGANIZATIONS TO IMPROVE BIMBASED PROJECT DELIVERY USING REQUIREMENTS MANAGEMENTPLATFORM
Ali Motamedi1, Sylvain Vaudou2, Romain Leygonie3, Daniel Forgues4
1) Ph.D., Assoc. Prof., Department of Construction Engineering, École de technologie supérieure, Montreal, Canada. Email:ali.motamedi@etsmtl.ca2) BIM Director, CAP INGELEC., Bordeaux, France. Email: sylvainvaudou@hotmail.com
3) M.A.Sc. Candidate, Department of Construction Engineering, École de technologie supérieure, Montreal, Canada. Email:romain.leygonie.1@ens.etsmtl.ca4) Ph.D., Prof., Department of Construction Engineering, École de technologie supérieure, Montreal, Canada. Email:daniel.forgues@etsmtl.ca

Abstract: BIM-assisted project execution methods are being increasingly adopted by designers and contractors
due to client demand and the efficiency gains they provide. However, many owner organizations do not take fulladvantage of BIM in assessing the quality of deliverables, and do not use digital deliverables during the operation
phase. Additionally, although many owner organizations have BIM implementation plans, their design review anddata handover processes still follow conventional methods due to a lack of BIM knowledge related to facilities
management. Requirements related to Facility Management (FM) operations are usually transferred using
traditional methods, and requirement elicitation and management processes often do not consider the BIM
potential (such as information provision, automatic data transfer, data quality assurance and control). In thisresearch, the operation of a major provincial government owner organization in Canada, which is one of thepioneers in adopting BIM-based project execution, is analyzed in an observational study to identify process gaps
when it comes to benefiting from BIM during the lifecycle. The study showed that although many BIM-based
processes are adopted for projects, many core processes on the owner side (such as gathering and communicationof FM requirement, data quality control and non-geometric data handover) still follow traditional methods. Using
extensive data gathering and use case analysis, process gaps were identified and process re-engineering
recommendations prepared. Additionally, the use of a requirements management system is proposed and verified
to tackle issues related to requirement documentation and tracing, knowledge preservation, data handover, anddata quality assurance and control.
Keywords: BIM, Requirements management, Quality assurance, Facilities management, Process re-engineering
1. INTRODUCTION
Many governmental and public organizations have started mandating the use of Building Information
Modeling (BIM) in a bid to improve productivity and information management. According to a NIST report(Gallaher et al., 2004), the major benefit of BIM lies in the cross-platform interoperability it offers for data transfer
and its asset management information centralization ability. This report identified a lack of integration between
project management (PM) and asset management (AM) for information capture and transfer as a major issue that
reduces owners’ ability to carry out proper maintenance activities due to a lack of information.
To manage asset maintenance and development, AM brings together activities that are split between
different departments with no common vision or governance. In PM, the production and exchange of information
focuses on monitoring construction documentation and verifying performance according to predefined objectives(Dawood et al., 2002). One of the challenges of conventional PM is that available information is on static
documents (PDF, paper documents), which makes the data retrieval process costly and error-prone and slows thebuilding commissioning process (IFMA, 2013). BIM platforms offer electronic support, including faster dataprocessing and the possibility of information reuse.
This study proposes a framework that formalizes the concept of integrated asset lifecycle management.
Further, it adapts systems engineering product lifecycle management practices from other industries (e.g.,
automotive, manufacturing) to the Architecture, Engineering, Construction, and Operations (AECO) industry.
Systems engineering offers a systematic and rigorous approach for requirements management. Thus, theframework proposes an integrated approach for the management of asset information, which ensures the alignment
between needs and deliverables. It includes a requirements management system for capturing information
requirements, integration with BIM authoring tools, and re-engineering of related processes.
The proposed method harmonizes information management practices between PM and AM, as well asplanning and design. It was implemented in a major owner organization in Canada, for verification and validation.
When such organizations change their practices after thorough studies and validations in pilot projects, they canthus serve as locomotives for other members of the industry. The objectives of this research are: (1) to identify
how a requirements management system can improve current practices in an owner organization, including quality227

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019assurance (QA) and quality control (QC); (2) to propose a new framework to integrate requirements management
practices for BIM-based project delivery; (3) to implement, verify, and validate the proposed framework in a realworld case study, using a customized commercial requirements management platform.
2. LITERATURE REVIEW
2.1 Requirements engineering
A requirement is: “A statement that identifies a product or process operational, functional, or design
characteristic or constraint, which is unambiguous, testable or measurable, and for product or process acceptance(by consumers or internal quality assurance guidelines).” (IEEE, 2005). A well-defined requirement is complete,unequivocal, consistent, achievable, concise and traceable (Kamara et al., 2000).
Requirements engineering consists of an integrated and systematic management of requirements that are
retrieved, analyzed, and tracked throughout the product lifecycle (Hull et al., 2011). To allow an understanding of
the changes and the evolution of the construction of the product, requirements management utilizes two
fundamental principles: traceability and stage-gate system (Sommerville & Sawyer, 1999). Traceability helps tounderstand the evolution of information, the relationship between requirements, as well as the impact of changes
made (Hull et al., 2011). Stage-gates for their part are transitions that verify whether the necessary information has
been provided. Depending on the results obtained, the development of the system can be stopped, corrected orcontinued at the same pace (Hull et al., 2011).
The requirements for a construction project are defined and described in the Functional Plan (FP) – a
paper or PDF document – serving as the only transcription of the customer’s wishes, in the form of text, diagrams
and plans. This document is prepared and shared at the beginning of the project, but as the amount of information
grows during the project, the FP document is usually not updated, and as a result, is unfaithful to the as-built
building (Whyte et al., 2010).
2.2

Information exchanges
There are three main problems faced in transferring building data. These relate to: the quality of the data
(accuracy, completeness, timeliness); the correspondence of the data to requirements, and the lack of a standard
digital format and structure for data exchange (Gallaher et al., 2004; Whyte et al., 2010). AM teams are often
reluctant to retrieve information from post-construction data. Robust processes for capturing and managing
information requirements, information provision and data transfer after commissioning can address this issue.
Kiviniemi & Fischer (2004) suggest that the solution for establishing continuity between PM and AM liesin developing an interface between the client requirements and design tools. This interface can increase the usageof requirements documentation throughout the design and construction process. It allows for tight management of
requirements and provides more automation, which results in reduced efforts and costs in lifecycle management.
The COBie format is an example of such an interface, which enables bridging the gap between the client
requirements and the design tool. This open data format describes the information exchanges between theconstruction and operations phases of a project (GSA, 2011), and provides consistent and structured asset
information to the owner.
2.3

BIM for facility management
ISO 55000 defines AM as the coordinated activity of an organization to realize value from assets (physicalor financial). For an organization to reach its objectives, it must use valuable data and information for decisionmaking. What is thus clear here is that how an organization processes its data is important. In an AM culture, theinformation itself is considered as an asset that must be managed and maintained.
Key owners are aware of the opportunities offered by BIM for the capture of information needed to
manage building system performance, to establish appropriate maintenance practices and to evaluate the feasibilityof proposed expansions or renovations. BIM models can be used as databases to store, organize, analyze, control,and exchange structured information, and therefore, can enable a well-designed FM system by managing building
and assets lifecycle data. Among other things, BIM can be used for managing spaces, populating the FM database,and anticipating maintenance needs (Becerik-Gerber et al., 2012).
Facility managers do not usually use these BIM models, because either they do not contain the necessary
information, or they include superfluous information (Jawadekar, 2012). Defining a BIM-enabled FM thus requires
specifying the relevant information to be added to the model and carried on to the O&M phase. Some studies havediscussed the process flow for creating an FM-BIM and its integration with O&M platforms, but this requires
further exploration (Motamedi et al., 2018).
2.4

Quality management
Kulusjärvi (2012) states that Quality Control (QC) focuses on assessing the quality of building designs.
This activity should be defined within the BIM Plan as a detailed approach to assess the degree of compliance ofthe final deliverables with the quality standards (Motamedi et al., 2018). To properly check the model, a228

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019verification of the systems in the model must be performed to determine if they include the necessary major assets(Zadeh et al., 2015). Motamedi et al. (2018) proposed QC checklists of assets to be included, alongside a purge of
unnecessary information. Having a lightweight integrated model that is interoperable and is enriched with FM datais the final goal of the model preparation (Motamedi et al., 2018). A detailed determination of what should be
included in the model and automated quality assessment approaches are directions that require further work.
2.5

Identified research gaps
The literature review highlights the divide between AM and PM, which leads to an interruption in theflow of information at the end of construction. The implementation of requirements management systems can
respond to many challenges related to the use of data in the operation phase, including data availability and data
quality. However, implementing such systems requires process re-engineering and an assessment of impacts.
Additionally, the mapping of contractual requirements into BIM requirements, as well as the control processes of
FM-BIM models, need to be further explored. The literature does not provide a detailed analysis and assessmentof a case study of the implementation of such platforms and re-engineering processes in large owner organizations.
4. PROPOSED METHODThis research proposes the establishment of a requirements management framework for communicating
the requirements and verifying the compliance of information throughout the project. It complements current
information management systems and it will serve as a backbone to the project information management process,starting from the planning phase, for producing and communicating the requirements throughout the project
lifecycle. The proposed framework includes a requirements management system integrated in a BIM authoring
software. The implementation of this system requires the re-engineering of various existing processes.
The requirements are captured during the planning, design and construction phases, and are clearlyrecorded and organized in the requirements management system (as opposed to text-based definitions in traditional
FP document). The system holds the relationships of the requirements with the objects and data, which are centrally
stored in the BIM models. The system communicates the requirements to project stakeholders by producing reports
or providing access to the database. Additionally, it verifies the information by analyzing its compliance with therequirements and tracking its history of changes. Additionally, the system automatically transfers captured data toplatforms used for the building operation. Finally, it supports open standards for data transfer, and thus allows theintegration of various tools in the project lifecycle.
Furthermore, the integrated system supports lifecycle QA processes by providing a guide for designers
regarding the information needed in the models (i.e., the assets and their data) and tips to ensure that the qualityof information is maintained. Figure 1 presents a systems architecture. It shows that during each of the major
phases of the project lifecycle, stakeholders (an architect, for example) query the requirements management system
to retrieve information, which they use to carry out modeling activities (Fig. 1 data entry arrow). The resulting
information goes through a quality control layer and is entered into an interoperable BIM database. The databasecan be queried, and its data is retrieved for various BIM uses (e.g., 5D calculation).

Figure 1: Systems architecture and interactions
Using the requirements templates, the owner creates the initial requirements during planning and generates
the FP accordingly (Figure 2). In the design and construction phases, the owner, engineers, and contractors
successively query the system to retrieve the requirements, which they use to enrich the models in the BIM

229

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

database. Similar to the FP, the construction contract requirements are automatically generated by the system.
Eventually, the data gathered in the BIM is exported to the CMMS during the handover phase.

Figure 2: Information flow through project lifecycle with the use of the requirements management system
In addition to the QC activities performed at the stage-gates between phases of a construction project,
additional QC activities are proposed in the quality management process. These activities consist in assessing
whether the data in the model comply with the requirements expressed in the system. Furthermore, the proposed
method allows gradual QC and a continuous monitoring of the requirements solicitation status throughout the
project lifecycle (Figure 3). As the requirements management system is embedded in the BIM authoring tool, manyof the QC activities, such as the generation of compliance reports, can be automated, instead of using 2D plans
and static documents.

Figure 3: Proposed stage-gates and QC activities
5. IMPLEMENTATION IN AN OWNER ORGANIZATION
The following steps were taken in building an IT solution to support the information management
processes of a major owner organization: (1) context analysis by reviewing documents, collecting project data,and performing interviews, (2) development of a framework by establishing a concept of operation for capturing,verifying, and validating information for AM based on requirements management, (3) implementation andexperimentation of the solution based on the framework in a case study through the realization of a proof of concept
involving the use of a commercial platform, and (4) evaluation and conclusion by ensuring that the proof of concept
criteria for success are met.
5.1

Context analysis
Prior to this study, the organization had started adopting BIM processes through several pilot projects to230

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019evaluate the potential for a larger scale adoption. After successful evaluation of the pilot projects, the organization
adopted a roadmap for mandating the use of BIM in all large projects.
The Building Support and Coordination Office is responsible for activities such as the determination of
information requirements and the identification of key control points in requirements tracking. The requirements
describe the asset types, physical and operating properties, costs, and equipment manufacturing and maintenancedata.
In order to manage the information exchanges, the organization uses an IT architecture implemented for
BIM-based projects that consists of four main components: a File Transfer Protocol (FTP) datastore for the storage
and sharing of files and models; an external portal service (i.e., SharePoint) acting as an information hub; an
Electronic Document Management System (EDMS) (i.e., Codebook) used for recording some of client’s
requirements, and an E-transit application for the transfer of the models. However, the current architecture faces
certain issues, such as a lack of integration between the above-mentioned components with the BIM authoring tool.
The process is not fully digitalized, and the transfer of data is file-based. Finally, the EDMS is not optimally
exploited, and supports neither proper centralization of the requirements, nor automated QC. Consequently, therequirements are still in the FP.
A process map was developed that includes projects steps and actors, inputs and outputs for each step. In
the process map, the activities affected by our proposed method were identified. The main identified activities arereal estate analysis, design review, and data handover to AM. For each of these activities, use cases were developed
highlighting the actors and their actions related to requirements management.
5.2

Use cases and identified issues
Real estate analysis is the stage where the initial requirements are expressed by the client. The
organization creates tender documents based on the information gathered in this step. The FP is created by
combining the client’s needs with the functional and technical content defined by the organization. Figure 4 shows
the actions and stakeholders involved during this step of the project. The shaded activities show the actions thatcan be notably improved using a requirements management platform. Similar use cases have been developed forother identified project steps.

Figure 4: real-estate analysis use case
Design review is the stage where the compliance of the proposed design with the requirements and thedecisions made in the real estate analysis stage is assessed. The designers submit technical choices and 2D
drawings to the organization, responding to the original FP. A report is sent to the project manager, who carries
out manual verifications from 2D drawings and sends modification requests to the designers.
It is during the data handover process that the as-built information is retrieved, controlled and transmitted
to the asset managers, mainly in static documents. The organization verifies whether the required documents havebeen delivered by the contractors, but it does not verify the content of the documents in detail and can thus not
ensure their compliance. Moreover, the quality of the delivered models produced by the engineers and the generalcontractor is not adequately verified. As a result, many issues in the models failed to be detected. The models are
only partially updated by the engineers with as-built data, and the information required for the operations is
manually recorded in the CMMS from Excel spreadsheets, which is a costly and error-prone endeavor. Moreover,
the recorded data is often incomplete. This handover method illustrates the divide in the flow of information
between PM and AM.
An analysis of the use cases allows highlighting the areas of improvement and the functionalities related
to information and requirements management to which the proposed system responds. Table 1 summarizes the231Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

findings. The areas where the improvement opportunities are most visible are: the automation of informationtransfers between phases; the communication between the stakeholders; the control of the information and themodel, and finally, the inclusion of the requirements of the property managers throughout the lifecycle.
Table 1. Problems identified and corresponding solutions
Problem•••••

Generation of duplicatesReal estate personnel not involved
Technical Specifications are a compilation of existing documents
Lack of data needed for the operations phase
Many information transfer steps•

Requirements, design concepts and end-of-project documentation
shared in paper document•

Manual checks based on 2D plans and on-site verification•

Transfer of information to CMMS based on paper documents

•
•
•
•
•
•
•
•
•

Solutions and opportunities
Structure the information in a requirements management platform
Involve real estate personnel in requirements definition
Use a requirements management platform to centralize knowledge intemplates
Simplify access to the information through the platform
Propose a dynamic way to communicate requirements to engineers
Link the requirements management platform to authoring software tofacilitate the export of information
General contractor enters the data in the platform with the as-built
information in response to the requirements
Carry out automated QC on the platform for requirements complianceTransfer the information to the CMMS from the platform adapted toclassification systems

5.3

Proposed solution
The proposed solution is based on the method explained in section 4. It includes a re-engineering ofexisting processes and the use of a requirements management platform integrated in a BIM-authoring software.
This platform allows monitoring the content of the model to assess compliance with the requirements. The abovementioned problems are addressed through process re-engineering to accommodate the use of the requirements
management platform. The divide in the flow of information between PM and AM is tackled by taking advantageof the platform’s ability to centralize project requirements, and to share, organize and retrieve the requirements
produced at each stage, as well as communicate them to the next phase. It also allows the retrieval of data fromthe BIM model to be automatically transferred to the CMMS and CAFM platforms.
During the planning phase, the information requirements for the operations are entered in the platform by
the actors involved in the real estate analysis. The use of templates that formalize accumulated knowledge in theorganization allows for greater efficiency.
During the design phase, the integration between the platform and the authoring software allows accessing
data requirements within the authoring tool. The quality of the design is assessed following a stage-gate process.
At each stage of the design development, the models are checked to verify their accuracy and their compliancewith the requirements. The integrated requirement management platform facilitates this process by automaticallychecks the compliance. The design proceeds to next stage only after the confirmation of compliance of all required
information. The involvement of the AM team in this process is essential. For the design review process, thereports automatically generated by the platform are used to verify compliance with the requirements expressed in
the FP.
During the construction phase, the contractors enrich the model through the platform. At the end of theconstruction, all the information added by the contractors and by the engineers throughout the project is retrievablein the platform. Consequently, the stage-gate for receiving post-construction information is processed more
efficiently at the commissioning step. Figure 5 shows various types of information added in the requirements
management platform during each phase of the project. For instance, during the planning phase, the rooms and theassets requirements are added to the platform, which helps automatically generate the FP document.

Figure 5: Evolution of the content of the platform232

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

6. CASE STUDY: A COURTHOUSE EXTENSION PROJECT
Our proposed solution was verified in a BIM-based courthouse renovation and extension project in Canada.
The organization used an EDMS platform (i.e., Codebook) that was successfully launched in the planning phase.
However, it was not effectively used in the proceeding project phases, due to the lack of a contractual obligation
to use it, and the absence of a clear definition of required information. The objectives of the pilot project were toverify and validate the applicability of our proposed method, which includes the use of a customized tool and theimplementation of re-engineered processes. The requirements related to the Heating, Ventilation, and Air
Conditioning (HVAC) were especially considered to verify the applicability of the proposed method.
dRofus was chosen due to its unique requirements management capabilities derived from a Project
Lifecycle Management (PLM) system. It consists of a collaborative data planning and management tool and is
integrated in a BIM authoring software (i.e., Revit). The platform features various functionalities and is especially
useful for defining room specifications. It allows carrying out QC in the BIM authoring software and generating
reports. In addition, it can be used to create templates and to track changes made during the project lifecycle.
The lengthy FP was thoroughly analyzed by the research team in order to map its items (i.e., therequirements categories) to the available modules of the platform. The requirements related to the organization of
spaces, the descriptions of the rooms, and the properties of the items can have their compliance checked
automatically by the platform.
In the Real estate analysis, the main goal of the proposed solution is to centrally compile the stakeholders’requirements and to structure information. The client records the requirements in the platform and the project
manager reviews the information added in the platform. The templates are used to define general requirements andequipment types that are assigned to spaces to quickly generate Room Data Sheets. The client also uses templates
that are pre-made by the owner, which contain generic requirements and standard and regulatory specifications.
The use of templates enables a quicker definition of the room list and improved consistency as it allows similar
configuration for spaces with the same functionalities. Afterwards, property managers define the information
requirements related to the equipment. After the requirements are recorded in the platform, the FP is created byexporting the information already contained in the platform. The platform can further define, communicate andverify information requirements about rooms and spaces, as well as the type and amount of equipment they contain.
For the Design Review, the integration of dRofus within Revit allows centralizing information andautomatically comparing the model against the requirements. The objects and their attributes in the model arelinked to the requirement items recorded in the platform. This link enables quicker modeling since the platforms
generates and places all the required items in the center of the rooms once the room datasheets are linked to theplatform. Additionally, the platform reports the difference between the planned and the modeled items, whichallows the designers to rapidly adjust the number of items in each room.
The new process includes the delivery of BIM models by designers and the submission of equipment
choices directly in the platform. The project manager and the technical experts control the compliance of the
proposed design choices with automatically published reports. The platform illustrates the difference between theplanned and the modeled items, which allows the designers to quickly adjust the number of items in each room.
Moreover, it is possible to automatically monitor the progress of the design project.
During the Data handover, the general contractor updates the models with the as-built information andthe project manager and the commissioning agent assess the completeness and quality of information directly inthe platform. At this stage, information that is already available in the requirements management database isexported to the FM platform. In this case study, the data in the platform is exported in Excel format, and thenmapped to a template provided by the CMMS platform (i.e., GuideTI). The above-mentioned activities are
summarized in Figure 8, which shows the interactions between the stakeholders and dRofus throughout the project
lifecycle, as well as the evolution of contractual documents and BIM models.
7. CONCLUSION
The present research builds on system engineering practices in advanced industries and adapts them to theconstruction domain. The proposed framework consists of the implementation of a requirements management
system integrated in a BIM authoring software. Additionally, it proposed process re-engineering activities tocollect, communicate and track requirements during the project lifecycle. It allows an integrated QA and an
automated QC of models and their information, as well as the collection and retention of knowledge. The flow of
information between stakeholders and the requirements management system throughout the project lifecycle wasbeen elaborated. The framework was assessed in a project at a major owner organization. Use cases were developed
to clarify process re-engineering and implementation steps based on the proposed method. Data transfer between
PM and AM was also evaluated.

233

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 6: Interactions with the requirements management platform
The development of the use cases enabled an assessment of the applicability of a commercial platform by
evaluating its response to the requirements expressed in the FP document. The re-engineered processes thatfacilitate the use of the platform enabled stakeholders to better communicate the requirements and to control thecompliance of design choices more efficiently. The main benefits of the implemented solution are its ability tocentrally structure and store the requirements, to generate compliance reports, to enable automatized QC of theBIM models, and to facilitate information transfer to the CMMS.
Although the data in the platform can be transferred to the CMMS system, the data exchange isunidirectional. An integrated system that allows data synchronization between the CMMS, the BIM model and therequirements management database is desired. Additionally, the requirements can be formally presented by IFCobjects and stored in the BIM model. Finally, changes in the contract templates, such as mandating the use of arequirements management platform, should be considered.
REFERENCES
Becerik-Gerber, B., Jazizadeh, F., Li, N., & Calis, G. (2012). Application Areas and Data Requirements for
BIM-Enabled Facilities Management. Journal of Construction Engineering and Management, 138(3),431–442. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000433
Dawood, N., Akinsola, A., & Hobbs, B. (2002). Development of automated communication of system for
managing site information using internet technology. Automation in Construction, 11(5), 557–572.
https://doi.org/10.1016/S0926-5805(01)00066-8
Gallaher, M. P., O’Connor, A. C., Dettbarn, Jr., J. L., & Gilday, L. T. (2004). Cost Analysis of Inadequate
Interoperability in the U.S. Capital Facilities Industry (No. NIST GCR 04-867; p. NIST GCR 04-867).
https://doi.org/10.6028/NIST.GCR.04-867
GSA. (2011). GSA BIM Guide For Facility Management. General Services Administration.
Hull, E., Jackson, K., & Dick, J. (2011). Requirements engineering (3rd ed). London ; New York: Springer.
IEEE. (2005). IEEE Std 1220-2005 IEEE Standard for Application and Management of the Systems Engineering
Process.
Jawadekar, S. (2012). A Case Study of the Use of BIM and Construction Operations Building Information
Exchange (COBie) for Facility Management (Master Thesis). Texas A&M University.
Kamara, J. M., Anumba, C. J., & Evbuomwan, N. F. O. (2000). Establishing and processing client
requirements—a key aspect of concurrent engineering in construction. Engineering, Construction andArchitectural Management, 7(1), 15–28. https://doi.org/10.1108/eb021129
Kiviniemi, A., & Fischer, M. (2004). Requirements Management Interface to Building Product Models. 12.
Kulusjärvi, H. (2012). COBIM Series 6: Quality Assurance (p. 27).
Motamedi, A., Iordanova, I., & Forgues, D. (2018). FM-BIM Preparation Method and Quality Assessment
Measures. Presented at the 17th International Conference on Computing in Civil and Building
Engineering (ICCCBE), Tampere, Finland.
Sommerville, I., & Sawyer, P. (1999). Requirements engineering: a good practice guide. Retrieved fromhttp://app.knovel.com/web/toc.v/cid:kpREAGPG05
Teicholz, P. M., & IFMA Foundation (Eds.). (2013). BIM for facility managers. Hoboken, New Jersey: Wiley.
Whyte, J., Lindkvist, C., & Nurain, I. (2010). Value to Clients through Data Hand-Over: A Pilot Study.
Summary Report to Institution of Civil Engineers (ICE) Information Systems (IS) Panel.
Zadeh, P., Staub-French, S., & Pottinger, R. (2015). Review of BIM Quality Assessment approaches for Facility
Management. Presented at the International Construction Specialty Conference, Vancouver.

234

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

A Framework for Visual BIM-based Maintenance Management in MRT Stations
Yi-Shian Huang 1, Yu-Cheng Lin2
1) Master graduate student, Department of Civil Engineering, National Taipei University of Technology, No.1. Chung-Hsiao
E. Rd., Sec.3, Taipei, Taiwan. Email: mh70804@gmail.com
2) Professor, Department of Civil Engineering, National Taipei University of Technology, No.1. Chung-Hsiao E. Rd., Sec.3,
Taipei Taiwan. Email: yclinntut@gmail.com

Abstract:
The long-term operation of MRT station facilities will cause wear and tear. How to improve themaintenance and management is an important issue in MRT station facilities management. In order to improvethe efficiency of MRT station facility maintenance and management, this study use the building information
model (BIM) visualization technology to discuss the facility management. Although there are many previous
studies focused on applications of BIM-based MM works, few studies are focus on the implementation models
for BIM-based visual BIM-based maintenance management in MRT Stations. Therefore, the study proposes theframework for visual BIM-based maintenance management special for MRT station in Taiwan. The proposed
framework was applied in a selected MRT station in Taiwan for case study to verify our proposed frameworkand to demonstrate the effectiveness. Finally, the study identifies the benefits, limitations, and suggestions forfurther applications.
Keywords:

Building Information Modeling (BIM), Maintenance Management, MRT Station, Operation Phase

1. INTRODUCTION
Building Information Modeling (BIM) technology has been applied in construction projects, for now it
has also been extended to MRT engineering and civil engineering. In present time, when there is an abnormal
condition for facilities, the facilities maintainer can only get the feedback from a black and white pictures insingle-oriented facilities. As a result, there is no way to receive all the status for facilities immediately, which
reduces the efficiency of facility maintenance and management. So this research is to study and planning of BIM
in the MRT station facilities maintenance and management, including maintenance procedures and personnel,
and defined maintenance management system in the model after combining visual rendering. By import theactual case, to study for the benefits and difficulties in effective planning management and extend the value ofbuilding facilities.
2. LITERATURE REVIEW
BIM technology applied to building through three-dimensional visualization to information sharing,
analytical data, provide decision, and information management. Because the operation and maintenance stage
takes most of the time in the life cycle of the building, in order to maintain the long-term use and smooth
operation of the facility, Facility Management (FM) comes into being. International Facilities Management
Association (IFMA) presents FM is a profession that encompasses multiple disciplines to ensure functionality,
comfort, safety and efficiency of the built environment by integrating people, place, process and technology.
Sorting out the previous definitions, the concept of facility management is an important phase in the overall life
cycle of the maintenance building.
Marzouk (2014) used BIM and wireless sensing systems to monitor the quality of the indoor
environment of the subway, for a maintenance indicator that in paper provided. Marzouk, ATY (2012) proposed
the application of BIM in subways by modeling different components including structural, mechanical, electrical,and HVAC, assist facility managers in making decisions. Chen and Chu (2016) integrated analysis andapplication through the MRT station model and the Time dependent Vehicle Routing Problem (TDVRP) model
to assist in the search for rescue and evacuation operations in the MRT building. Timothy Justin Brooks andJason D. Lucas (2014) identified the keys to successful post-construction BIM utilization and how it can besupported by the contractor, to bridge the gap between the contractor and the owner. Sreelatha Chunduri et al.
(2013) developed a theoretical framework of technical requirements for using BIM-server as a multi-disciplinary
collaboration platform. Lather et al. (2017) present the development of a framework for leveraging the 3D natureof spaces, their attributes, and sensor location data in conjunction with building management system (BMS), theframework offers an initial system to integrate two typically disparate data, to more easily access building data inthe operations and maintenance of facilities. Ruikar et al. (2007) explored the scope of using the semantic web tomanage information management processes in the construction industry. Develops the hypothesis that
information can be managed using appropriate tools and techniques and develops a roadmap that shows the way
in which a solution can be achieved. Motawa, and Almarshad (2013) developed an integrated system to capture
information and knowledge of building, use BIM module to capture relevant information and Case-Based235

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Reasoning (CBR) module to capture knowledge, to help maintenance teams learn from previous experience.
Mayo, G. et al. (2012) assist facility owners in the adoption and mature implementation of BIM processes byassessing their current state of BIM execution. Becerik-Gerber et al. (2011) proposed an online survey andface-to-face interviews were conducted to assess the current status of BIM implementations in FM, Highlighting
the synergy between the two, help professionals recognize potential areas in which BIM can be useful in FM
practices.
3. A FRAMEWORK FOR VISUAL BIM-BASED MAINTENANCE MANAGEMENT
In present time, there are two methods used for BIM to management in MRT Stations. They areschedule maintenance model and call for maintenance model.
1. Scheduled maintenance:
According to the condition of the facility, establish the time interval for the maintenance of the facility.
In addition, to make the scheduled date and operation content for the maintenance.
2. Call for maintenance:
By the passengers or station staff to carry out the maintenance notice, and through the system to carryout the maintenance notice. Or by the regular inspection found abnormal after notification.
There are two ways used for BIM-based visual BIM-based maintenance management in MRT Stations. They areschedule maintenance model and call for maintenance model.
BIM is applied to the operation and maintenance mode of MRT station in the following two categories:
1. Scheduled maintenance mode:
The BIM model is used by the maintenance sectors, which establishes and sets the time limit forperiodic maintenance of the facility. It also input the data into the information field for the model. When the
periodic maintenance of the facility reaches, the inspector will go to the site to maintain the facilities after
receiving the notification which highlighted by BIM component facilities. If there are problems been found andcannot be solved, the problems will be reported to the maintenance engineer in the system, and the BIM
component will continue to be displayed until the issue is solved (Fig. 1).

The model ishighlighted
NoticeMaintenanceengineer

Inspector

MaintenanceengineerUpdate to the BIM model

NoticeReturn the maintenanceinformation

The model is unlit

Figure 1. Scheduled maintenance mode
2. Call for maintenance:
Once MRT station’s passengers or the station staff found facilities failure, by inputting the data throughthe mobile device, BIM components will be highlighted to remind the maintenance engineers. The maintenanceengineers need to confirm the location of facilities and the reasons for the failure, also need to analyzed whether
there is a repeat accident of the documents, the date for maintenance will scheduled after the issue is confirmed.
The MRT station facility vendors will be informed, who will go to found and repair any abnormal facilities.
After the completion of repair, the content of reparation will be integrated and published to the system to informthe maintenance engineer. After receiving the information, the maintenance engineer will confirmed that there is
no loss for any information, and upload the content to the system to update the information (Fig. 2).

236

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Station staff
or passenger

Confirm whether the report is repeated

Maintenanceengineer

Maintenanceengineer

updated to the system

Return information through the system

The system model is
unlit

Call for repair

SystemNoticeMRT station
facility vendor

Figure 2. Call for maintenance mode
The BIM facility management implementation team of MRT station is different from the generalbuildings. Table 1 mainly definitions the different responsibilities for the main personnel who perform facility
maintenance and management.
Table 1. Responsibilities description of participants for BIM-based maintenance management in MRT station
Personnel
Maintenanceengineer

Duties and responsibilities
Must have maintenance management experience, basic graphic knowledge and software
operation ability. Provide the model information for site personnel and the experience forprevious maintenance. To assist maintenance applications in regular management.

Inspector

For the personnel stationed at the MRT station, the main work is regular inspection, initial
repair and emergency maintenance for facilities.

MRT station
facility vendor

Mainly to provide facilities information and received repair and replacement notices frommaintenance engineers.

Auditor
Station staff
or passenger

Check and evaluate the situation of current operating procedures.
It is the user of MRT station's facilities. If the facilities are abnormal who can inform themaintenance operation of the facilities.

In order to let the maintenance personnel be more intuitive to find the abnormal facilities, the following
two methods are proposed for the visual presentation of the planning model in the system:
1. Visual transparency mode
In order to prevent maintenance team need to spend more time into the model to search for the abnormal
facility which blocked by the external structure, the set of opacity to will adjusted to 80% in this study (seeFig.3) .

Figure 3. Transparent appearance

237

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2. Abnormal facility caution status
To promote efficient for the management personnel to maintenance abnormal facilities. This study alsoplans the definition of the colors for the facilities states. The following two colors definition modes are divided
into “scheduled maintenance of facilities” and “call for maintenance of facilities”:
Scheduled maintenance of facilities: the date of periodic maintenance shall be set according to the statusof each facility when the completed model is delivered. Ahead of 7 days and 14 days the scheduled maintenancecan be calculated (see Table 2).
Table 2. Scheduled maintenance of facility status color definition table
Facility
situation

NormalThe 14 days before
schedule maintenance

The 7 days before
schedule maintenance

Color

Original

Blue■

Orange ■Scheduledmaintenanceon the day
Pink ■

Waiting
approval
Green ■

Call for maintenance of facilities: it is not easy to predict the abnormal state of the facilities, so thedefinition of the state and color of the facilities is different from that of scheduled maintenance (see Table 3).
Table 3. Call for maintenance of facility status color definition table
Facility situation

NormalAbnormal

Color

Original

Red ■

Under
maintenanceYellow ■Waiting
approval
Green ■

4. CASE STUDY
The case of this study is the ground floor of a MRT station in Taipei city. The shape of the MRT stationis slender. There are many MRT station service facilities, and the appearance of facilities with the same functionis the same, which may cause the maintenance personnel to repair the normal facilities after receiving the error
message.
After receiving the completed model, the maintenance team shall first set the requirements for the
maintenance and management of the facility, modify the model according to the requirements, then check andrevise the model by personnel. After the revision, the model can be combined with the BIM maintenancemanagement mode and visual presentation proposed by the research institute, so that the maintenance personnel
can effectively identify the status of facilities, provide engineers and supervisors who inspect the model toquickly and effectively know the site conditions, and monitor the equipment tasks that have not been completed(Fig. 4).

Figure 4. The illustration of facility maintenance of MRT stations integrated BIM model(1) Scheduled maintenance operation mode of MRT station facilities:
Based on the BIM model and system of MRT station, taking emergency lifesaving instant exit as an
example, the scheduled maintenance of station facilities in MRT station is divided into the following procedures:238

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

1.

2.

3.

The maintenance unit sets the maintenance date for the emergency lifesaving instant exit. The model isbright blue for 14 days before the expiration date, orange for 7 days before the expiration date, and pink
for the day on regular maintenance.
On the day of scheduled maintenance, the maintenance personnel shall know the location of theemergency lifesaving instant exit which is need to be repaired on the day first, and go to the location of
the facility to perform the operation of periodic maintenance, and record the contents of scheduled
maintenance into the system form. In the process, if the emergency lifesaving instant exit is found ofabnormal conditions, the maintenance notice should be relevant.
After the overall maintenance, the model will change from pink to green (see Table 5) when the personnel
records the maintenance form to the system center for review by the maintenance unit supervisor.
Table 4. Comparison diagram of emergency lifesaving instant exit in maintenance mode
Facility
situation

NormalThe 14 days before The 7 days before
scheduleschedulemaintenancemaintenance

Scheduledmaintenanceon the day

Waiting
approval

BIM
component
stateThis study based on the method of scheduled maintenance after the introduction of the case, and theflow chart shown in Figure 5.

MaintenanceengineerStart

The 14 days
beforeschedulemaintenance

ScheduledmaintenancenoticeBlue

The 7 days
beforeschedulemaintenance

OrangeInspector

Scheduledmaintenanceon the day

View facility
location

Pink

Inspection andmaintenance

Change BIM
component color

Green

Check pending
review
facility

Determine
Yes
whether the facility has
passedChange BIM
component color

End

AuditorRed

No

Call for repair

MRT station
facility
vendor

NoticeRepairYellow

Determine
if the facility isrepaired

No

Yes

Green

Figure 5. Flow char for scheduled maintenance operation mode of facilities in case study(2) Call for maintenance operation mode of MRT station facilities:
Taking a machine that top up the Easycard as an example, the call for maintenance of station facilities in
MRT station is divided into the following procedures (see Fig. 6):
a. After station staff found out that the machine that top up the Easycard had faults, station staff notified
the system and the maintenance engineer confirmed that the faults were not repeated notifications. Then theyclicked on the model of the gate and turned to red, and informed the MRT facility manufacturer to come for239

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

maintenance through the system.
b. When the MRT station facility vendor comes for maintenance, the model change to yellow by
selecting the BIM component, so that other maintenance personnel know that the facility is under maintenance.
c. After the facility maintenance is completed, the MRT station facility vendor enters the system andselects the model. At this time, the model should be changed to the green color to waiting approval.

Figure 6. The state color of the fault repair operation model is presented
This study proposes the flow chart of call for maintenance after the introduction of the case (see Fig. 7).

MaintenanceengineerCall formaintenancenoticeStart

On the day

View facility
location

MRT station
facility
vendorChange BIM
component color

Yellow

Red

Distribution

RepairYES
Determinewhether the facility has
passedChange BIM
component color

NO
Green

Auditor

Check pending
review
facility

Determine
YES
whether the facility isrepaired

Change BIM
component color

End

NO

Figure 7. The flow chart of Call for maintenance operation mode of MRT station facilities
4.2 Discussion for Benefits and difficulties
The main benefits are summaries as following:1.
To let the computer to work that do not need human resources, which can shorten the time for humanresources to find facilities, and solve the problem of data transmission that may cause loss.
2.
To quickly receive the location of the facility in the first time, understand whether there are facilities
around the facility that cannot be reached, and quickly remove the failure of the facility according to thesurrounding conditions.
3.
The facilities in the building will affect each other. Through consulting the model, the facility
manufacturer can know whether the maintenance of the required facilities will affect other facilities andinterfere with their operation.
4.
To transform this management mode into active management, and automatically issue notices tomaintenance engineers through system investigation, so as to improve the efficiency of maintenance.
5.
To quickly identify the current status of the facilities through the color discrimination of the facilities, and240

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

immediately carry out emergency repair of the facilities or order the follow-up repair and maintenance ofthe facilities.

1.

2.
3.
4.

The main limitations are summaries as following:
At present, MRT stations have been accustomed to using two kinds of maintenance management
operations: operation management system (OMIS) and maintenance management system (MMS), which
may be difficult to accept new maintenance methods.
If the contents are filled into the mobile device in the filling mode, the newly used personnel may sufferfrom inefficiency due to unaccustomed reasons.
As MRT needs to combine a large amount of professional information, which may come from different
companies, how to integrate and combine the information needs to be discussed.
At present, space annotation is been hidden. Although it can make personnel find the location of the
facility more efficiency, it may not be able to immediately connect the location, which reduces theefficiency of communication.

5. CONCLUSIONS AND SUGGESTIONSThis study first understands the current inspection mode of MRT stations, plans to integrate maintenancespace and facility status with BIM, and then apply it into MRT stations for facility maintenance management.
The conclusions are as follows:1.
The maintenance and management work must have the MRT station BIM maintenance team, and inaccordance with the standards and specifications of the set, and, in view of the existing equipment
maintenance management mode according to the process of the operation personnel to define the related
responsibilities and authorities, in view of the overall state of maintenance and management to monitor
application integration management and so on.
2.
The transparency of the peripheral structure of the original completed model can make the maintenance
unit more intuitively understand whether there is any abnormal condition of the facility and improve theefficiency of the facility maintenance.
3.
In view of the facility maintenance integration, make full use of the characteristics of information carrier,so that the maintenance, overhaul and inspection personnel can actively and effectively complete thefacility maintenance, increase the service period of facilities and improve the quality of MRT stations.
However, the platform and maintenance process of this study still have shortcomings. The following
Suggestions are proposed for this study:
1.
Personnel who originally used the traditional facility maintenance and management mode may be excluded
due to the change of the old mode. It is suggested that each personnel should have a certain understandingof the facility management mode combined with BIM.
2.
BIM is only applied to the maintenance and management of station facilities in MRT stations as the objectof discussion. In the future, other platforms can be integrated and managed to make the management levelof operation and maintenance more complete.
3.
Maintenance personnel can observe the abnormal facilities visually in the study. In the future, the system
function setting on the interface can be customized to meet the requirements of each type of facility
maintenance.
6. REFERENCE
IFMA, What Is FM? Retrieved from IFMA website: https://www.ifma.org/about/what-is-facility-management
Marzouk, M. and Aty A. A., (2012). Maintaining Subway Infrastructure Using BIM, Proceedings ofConstruction Research Congress 2012: Construction Challenges in a Flat World, pp. 2320-2328.
M Marzouk, A Abdelaty, (2014). BIM based framework for managing performance of subway stations,
Automation in Construction, vol.41, pp. 70-77.
A Y. Chen, J C. Chu, (2016). TDVRP and BIM integrated approach for in building emergency rescue routing,
Journal of Computing in Civil Engineering, vol.30, no.5 C4015003
Timothy Justin Brooks., Jason D. Lucas., (2014). A Study to Support BIM Turnover to Facility Managers for
Use after Construction, Computing in Civil and Building Engineering, pp. 243-250.
Sreelatha Chunduri., Ralph Kreider., John I. Messner., (2013). A Case Study Implementation of the BIM
Planning Procedures for Facility Owners, Architectural Engineering Conference, pp. 691-701.
J.I. Lather, R.Amor, and J.I. Messner, (2017). A case study in data visualization for linked building information
model and building management system data, ASCE International Workshop on Computing in Civil
Engineering 2017, pp. 228-235.
D. Ruikar, C.J. Anumba, A. Duke, P.M. Carrillo, N.M. Bouchlaghem, (2007). Using the semantic web for241Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

project information managemen, pp.507-524.
Motawa, I., Almarshad, A., (2013). A knowledge-based BIM system for building maintenance, Automation inConstruction, pp. 173-182.
Mayo, G., Giel, B., Issa, R.R.A. (2012). BIM Use and Requirements Among Building Owners, ASCEInternational Conference on Computing in Civil Engineering, pp. 349-356.
B. Becerik-Gerber, F. Jazizadeh, N. Li , and G. Calis, (2011). Application areas and data requirements for
BIM-enabled facility management, Journal of Construction Engineering and Management, pp.431-422.

242

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

MODELING THE LAST-MILE PROBLEM OF BIM ADOPTION
Jing Wang1, Weisheng Lu2, and Jinying Xu3
1) Ph.D. Candidate, Department of Real Estate and Construction, the University of Hong Kong, Hong Kong. Email:jingww@connect.hku.hk
2) Associate Professor, Department of Real Estate and Construction, the University of Hong Kong, Hong Kong. Email:wilsonlu@connect.hku.hk3) Ph.D. Candidate, Department of Real Estate and Construction, the University of Hong Kong, Hong Kong. Email:jinyingxu@connect.hku.hk

Abstract: In recent years, the high expectation of Building Information Modeling (BIM) has increasingly attracted
the attention of organizations in developing countries. To catch up with the leading BIM practice, those ‘late mover’
organizations tend to benchmark and adopt BIM practices that have been proven effective by global leaders.
However, the uptake of BIM use is largely stuck by the “last-mile problem”. While developers diffusing theirstandardized or generalized solutions to global users, organizations often find it difficult to adopt such solutions
due to the contextual difference between such standardized and generalized BIM solutions and their use
environments. This paper aims to firstly define the “last-mile” problem in BIM adoption and then, propose aconceptual model of such problem. In this paper, the last-mile BIM adoption is defined as “a decentralized process
involving the linear diffusion of BIM solutions from its source developers to destination users”. Synchronizing
literature on BIM and last-mile problems in various domains, a last-mile BIM adoption model is proposed by
identifying the model components and developing a design framework. This study has both academic and practical
implications. It offers a set of formal language to systematically describe the last-mile problem of BIM adoption,
leading to an improved understanding of the last-mile process and problems therein. For practitioners, the study
facilitates them to analyze last-mile problems and develop strategies accordingly.
Keywords:

Building Information Modeling (BIM), last mile, model, adoption, diffusion

1. INTRODUCTION
In recent years, Building Information Modeling (BIM) has been increasingly subscribed by the global
architecture, engineering, and construction (AEC) industry to enhance construction productivity. Particularly,
leading organizations in developing countries have started to show interests in BIM adoption. To minimize
learning cost, one common strategy is to adopt the technology and related BIM practices that have been tested by
the global leaders (Cao et al., 2014). For example, the majority BIM explorers in developing countries adopt
mainstream technologies developed from the U.S. as their uppermost technical solutions (Herr and Fisher, 2019).
Some authorities have developed localized BIM guidelines based on the early versions of U.S. and European
countries.
During the course of BIM adoption, the late-mover organizations seem suffering from the “last-mile”
problem. A major difficulty lies in adopting BIM from developed countries, considering the contextual differences
between the environments where BIM is developed and used. This incurs huge efforts to adapt the given BIM
practice to suit the specific organizational environment. While the BIM user organizations desire a set of tailormade BIM solutions, software vendors hardly provide such a solution. From their point of view, such toolkits areagainst the massive production spirit, incurring higher development costs bringing inconvenience for maintenance.
Similar concerns are also identified amongst international standardization bodies. In this regard, a feasible solution
to bridge the last-mile BIM adoption should balance the concerns from different stakeholders.
The solutions to the last-mile problems have been widely discussed in telecommunications, e-commerce,
logistics, and supply chain management studies. Although the last-mile problems vary in these domains, theyshared a similar two-step solution: to formally describe the last-mile problems by identifying the its structuralcomponents and needs of different stakeholders therein, and find an optimal distribution structure thatsynchronizes the concerns of multiple stakeholders (Aized et al., 2015; Harrington et al., 2016; Wang and Odoni,2014). However, in the context of BIM, the last-mile problems are still poorly theorized in literature. There lacks
a rigorous language to clearly define the problems for further analysis and solution development.
The primary aim of this paper is to articulate the “last-mile” problem of BIM adoption by an organization.
It does so by proposing a conceptual model comprising of a working definition, components, and a design
framework. This paper takes the first step to provide a set of systematic language to describe the last-mile problemof BIM adoption. It enhances the understanding of last-mile process, and can be used to analyze the last-mile
barriers for designing solutions. The remainder of the paper is organized as follows. Subsequent to this introductionis the research design of the study. Literature review is presented subsequently, focusing on last-mile studies inthe telecommunication networks, supply chain management, and transportation planning, as well as BIM adoption
literature. Then, the last-mile model is presented, including a working definition, the model components, and thedesign framework of last-mile BIM adoption. Discussions and conclusion are drawn in the last section.

243

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2. RESEARCH METHODS
The research design of this study combines literature review and industry engagement. The former method
facilitates to develop the definition, model, and design framework, while the latter enables to verify the outcomesusing practical evidence from real-life cases. According to Fink (1998), literature review is a “systematic, explicit,
and reproducible design for identifying, evaluating, and interpreting the existing body of recorded documents”. It
facilitates identify conceptual contents in the domain by summarizing the conceptual patterns, themes, and issuesof existing literature (Meredith, 1993). Even though there lack studies directly shed light on specific last-mile
problems of BIM adoption, the last-mile problems in other domains and BIM adoption have been popular topics
in the past twenty years.
Given this situation, relevant publications were searched and reviewed in two rounds. The first-roundreview concerns the literature on last-mile problems in the domains of telecommunication, supply chain
management, and public transit. By doing so it is expected to identify the important conceptual elements fordefining and articulating the last-mile problem in a general setting. Keywords for searching include “last mile”,“first mile”, “definition”, “model”, “framework”, and “dimensions”. One author quickly scanned the contents todetermine the inclusion of the papers, while another author double checked to reduce potential bias. Following asimilar process, the second-round selection focused on BIM adoption studies to contextualize the term “last mile”
and its conceptual components in organization’s BIM adoption practice. Keywords used in this round include“Building Information Modeling OR BIM”, “adoption OR implementation OR use”, “process”, “practice”, “case”,
and “barriers OR obstacles OR difficulties”. After selection, a detailed study was then conducted to criticallyanalyze the selected papers, especially their arguments and scopes of applications concerning (1) BIM adoption
stages, (2) BIM adoption barriers at each stage, (3) the stakeholders involving in ONE organization’s BIM adoption
practice, and (4) the factors influencing BIM adoption.
The literature review findings allowed to propose the definitions and last-mile model of BIM adoption.
Yet, the outcomes should be verified by empirical evidence to ensure its credibility. The last-mile model, especially
its typology, should be also further explored in real-life cases. In this study, the two objectives were achieved
through the author’s industry engagement. During the past two years, the authors constantly engaged with a costconsultant, a building client, a main contractor, and two software vendors. Working closely with the stakeholders,
the researchers were able to offer advice and design strategies to help with BIM practice of each stakeholder.
Meanwhile, the engagement enabled researchers to immerse, observe, and reflection from the real-life practice toverify the proposed last-mile model of BIM adoption. Data was collected through in-depth group discussions,informal interviews, reports, emails, and reflection of practical situations. Data was then analyzed by summarizing
BIM adoption stages, the last stretch of the adoption process, the ways that BIM user organizations integrate BIM
solutions with their own practice, and the methods that software vendors promote their BIM solutions. If the
proposed model fails to consider all practical concerns, it will be modified accordingly; if the model contains someelements not identified in practice, it will be discussed with the AEC professionals to determine if the elements
should be deleted or kept.
3. LITERATURE REVIEW
3.1 Last Mile and Last-Mile Problems in General
The term “last mile” originates from the telecommunication to denote the last leg of a wide area networkthat runs from the nearest aggregation point to users (Speta, 2000). Despite its seemingly short leg to the main
backbone facilities, the last-mile connection is usually the most problematic one. It is usually the technologicalbottleneck that limits the bandwidth and consequently speed of data transfer to customers (Cotter and Taylor,2001). It also incurs a huge amount of cost to install and maintain the last-mile connections, as these connections
directly link to numerous users and their wider variety of equipment that a standardized service fails to support(Nandi et al., 2016). “Last mile” is now widely used as a metaphor to describe the problems in the last stretch of a
network of service delivery in many domains, such as parcel delivery from centralized transits to geographicallydispersed users in supply chain management (Gevaers et al., 2011), and transportation planning that ships
passengers from public transportation nodes to their final destination (Wang and Odoni, 2016). In a nutshell, “lastmile”, in general, can be regarded as a decentralization process that delivers people, goods, products or service
from a centralized point to a dispersed destination.
Underneath the last-mile problems is a need to coordinate the requirements of multiple stakeholders in thelast-mile network. A typical example is the last-mile logistics, which usually involves manufacturers, retailers,deliverers, and customers (Niu et al., 2016) in the upstream and downstream flows of products, services, finances,
and/or information (Mentzer et al., 2001). Hitherto, existing studies have identified many variables influencing thelast-mile process. The variables can be classified based on the stakeholders involved, such as transportation
operators (or industry level), public administrators (or institutional level), and end-customers (Russo and Comi,
2011; Taniguchi and Tamagawa, 2005). In some studies, merchandise-oriented variables are also included as one
important level for analysis (Frederick et al., 2018; Harrington et al., 2016). The design variables can also be
categorized into technical, social, economic and environmental dimensions, each contains smaller items (Gevaers

244Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

et al., 2011; Harrington et al., 2016). The categorization into different levels and dimensions depends on thespecific perspectives and requirements in analyzing the last-mile problem.
To find an optimal solution for the complex last-mile problem, a prerequisite is to describe the problem
clearly, concisely and comprehensively. This makes it necessary to systematically model the last-mile network.
For instance, Nandi et al. (2016) characterized the last-mile telecommunication network development in rural areas
as the selection of suitable technologies and deployment methods considering the geographic locations, economic
conditions, motivation and adoptability, and sustainable business framework (e.g., skilled workers and funding
support). In the business to customer (B2C) delivery, the last-mile delivery is modeled by identifying its structuralelements, typology of structures, hierarchical levels, and design framework (Aized et al., 2014; Frederick et al.,2018). Even though contextualizing in different scenarios, these models highlighted some conceptual elements indefining the last-mile model, e.g., start and end point, the “trunk” business to which last mile is attached, major
stakeholders, last-mile structures and typology, and design variables. These should be addressed in defining thelast-mile BIM adoption.
3.2

BIM Adoption
The definition of BIM adoption should be first clarified to develop a last-mile BIM adoption model.
Despite numerous studies on BIM adoption, there still lacks a canonical definition. Indeed, BIM adoption tends to
be used interchangeably with BIM implementation (e.g., Arayici et al., 2011, Ding et al., 2015), which blurs thedistinctions between these two concepts. According to Rogers (2003), adoption is a “decision to make full use of
an innovation as the best course of action available”, while implementation is “a phase which occurs once aninnovation has been put into use”. In the BIM context, Succar and Kassem (2015) differentiate these concepts
considering BIM implementation occurs at sub-organizational scales (e.g., individuals and groups), while adoption
denotes a more generic term to overlay the connotations of implementation and diffusion (i.e., BIM use across theglobal construction industry). In another stream of literature (e.g., Papadonikolaki, 2018), BIM adoption andimplementation are used in different levels – adoption of BIM by firms, and implementation in projects. Despite
their nuances, these studies converge on a common understanding that adoption could be considered as a more
holistic term than implementation (Ahmed and Kassem, 2018). In proposing the last-mile BIM adoption model,
this study recognizes the need for a more holistic definition of adoption to cover more than just a specific phase ora milestone.
BIM adoption at organization level can be divided into several stages. According to Succar and Kassem(2015), BIM adoption can be achieved via a three-phased approach, namely readiness to BIM-based tools,
workflow and protocols, capability built on willful experiments and implementation, and maturity as organizations
gradually and continuously improving quality, repeatability, and predictability of BIM adoption. A critical leg hereis the last stretch occurring after the point of adoption, at which organizational readiness transforms into
organizational capability/maturity. In another stream of studies that follows Rogers’ Innovation-Decision Model,
BIM adoption is stretched to include an organization’s exploration and decision-making process, e.g., awareness,
intention and interest, point of adoption, implementation and confirmation (Ahmed et al., 2017; Hochscheid andHalin, 2019). In a broader sense, BIM adoption can be also portrayed as an iterative rather than a linear process,which can be modeled by multiple spiral cycles to diagnose the problems of adoption, plan executive strategies,
execute plans, and evaluate the application results (Arayici et al., 2011). The model also highlighted theinteractions amongst organizations and solution providers, who provide technical assistance for mutual adaptationof technology and organization practice for BIM adoption.
The explanations of the slow BIM uptake in organizations lie in various barriers of BIM adoption. A
consensual typology of these barriers is yet to be agreed, but efforts have been paid to exploring it. Some widely
mentioned hurdles include the attitudinal, technical, procedural, and economical (Azhar, 2009; Bryde et al., 2013;
Chang, 2014; Niu et al., 2016). The attitudinal hurdles can influence the early stage BIM adoption as organization
gaining knowledge and making decisions, while the effects of the rest can be prolonged to the later adoption stages.
Lacking skillful staff is a significant factor hindering a scale-up BIM adoption, as identified in organizations ofboth BIM leading/following countries regardless of their sizes (Bui et al., 2016; Hosseini et al., 2016). Several
researchers also attribute the slow uptake to the environment where BIM is adopted, e.g., internal management
support, organization structures, external coercive/mimetic/normative pressures and the contextual differences
between BIM developing and use environments (Ahmed and Kassem, 2018; Cao et al., 2014; Peansupap andWalker, 2005; Xu et al., 2014). These factors were further analyzed and modeled by Ahmed and Kassem (2018)
into a holistic three-level hierarchy, namely BIM innovation characteristics, the external environment
characteristics, and the internal environment characteristics. These factors should be taken into consideration indeveloping the last-mile BIM adoption framework.
4. THE LAST-MILE PROBLEM OF BIM ADOPTION
4.1 Definition
Organizations generally undergo a series of learning and decision-making process to adopt BIM (Figure245

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 20191). The journey starts with awareness of BIM, following a serious of information collection and knowledge
acquisition to evaluate the cost and benefits of BIM adoption. If the evaluation is positive, organizations will
initiate the BIM practice and explore feasible implementation strategies, e.g., suitable BIM toolkit, the BIM-related
workflow, information standard, organization structure, staff training. These strategies may be tested in trial
projects to identify their applicability in real-life cases. Once determined, organizations will implement thestrategies for their existing practice. Notably, during the last stage, organizations will continuously interact withmultiple stakeholders to acquire BIM solutions, including software vendors, technical developers, BIM consultants,
policymakers and other BIM users. These solutions, however, may not be suitable for an organization’s specificcontext, as the solution providers tend to develop, standardize, and promote its solutions to a larger portion of users
worldwide. This leads to the last-mile problem.

Figure 1. The stretch of last-mile BIM adoption
Synthesizing the previous studies on both last mile and adoption, this paper proposed a working definitionof last mile in BIM adoption as follows:
The last mile is the last stretch in the BIM adoption process. It is a decentralized process involving thelinear diffusion of BIM solutions from its source developer to destination users. It starts at the point whenorganizations determine/purchase BIM solutions to the point BIM is truly integrated into the organization practice.
4.2

Modeling the last-mile problems of BIM adoption
There are four conceptual components of the last-mile problem of BIM adoption, namely the product,transit point, drop point and destination (Figure 2).
(1) Product: BIM Innovations
Giving the real-life organization needs on BIM adoption, BIM solutions here should be pursued in a
broader sense that includes not only technology, but its accommodating process and guidelines. These dimensions
has been well categorized by Succar (2009), Liang et al., (2016), and various BIM ontological and implementation
frameworks (e.g., Jung and Joos, 2011). Specifically, BIM technology refers to a collection of tools and techniquesthat support BIM’s functionalities. It not only includes BIM data, software, and hardware, but also integral
platforms to support effective communication and collaboration amongst different stakeholders. BIM process
denotes a series of ordering work activities of BIM creation, management, and utilization to support the projecttasks, while BIM policy can be regarded as a course of action adopted or proposed by a government, business, orindividual as a reference to guide the BIM-based. An observation is that these solutions, mainly developed in U.S.
and Europe, will be gradually diffused to other countries/regions and the organizations therein.
(2) Transit Point: The Point BIM Solutions Introduced to Users
The transit point denotes the state that a BIM solution is first removed from its original developing settings246

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019and introduced to the users. This is also the starting point of the last mile. At this point, the BIM solution may not
be suitable to be used in the local organization settings, which may be conflicted to the embedded
original/laboratory settings in the BIM solutions. Some common transit points include the in-house promotion
department of the software vendors, the local subsidiary of a headquarter developer, and the third-party promotion
by professional institutes, governments, and so on.

Figure 2. Modelling the last-mile problem of BIM adoption(3) Drop Point: The Transfer Point of BIM Solutions
The drop point is where the BIM solutions are transferred. Technically, the drop point can be any point
between the source developer and the final BIM adoption organization. Some typical examples include thedeveloper’s own context, a continent/region-wide market, a country/district, or specific local organizations.
Currently, the selection of drop point is largely dominated by the mainstreaming BIM solution developers(4) Destination: User Organizations
Organizations in AEC industries generally vary in attitudes and behaviors on BIM adoption. A small
portion are actively embracing and explore the innovations with an aim to improve its existing business practice,
while the majority are more hesitate and passive in adoption BIM. The organizational behaviors in the last mile
can be attributed to its inherent characteristics, such as the scale, business nature, technical capabilities, humanresources, financial resources, and culture. These will directly or indirectly influence an organization’s attitude on
BIM, and thus BIM expectations and implementation strategies. Notably, organizations are confined to specificlocal conditions, to which BIM solutions diffused from its origins may not be suitable to be applied. This makes
it necessary for local agencies or BIM users to tailor the BIM solutions for the specific local regulation, economic,
social, and cultural environment.
From the preceding section, a typology of the last-mile BIM adoption process can have three basic forms,
namely the passive, active and mixed mode (see Figure 3). In the passive mode, BIM solutions are directly sent to
fit the organization’s specific context without much organization’s involvement. In other words, BIM solutions
are removed from its origin settings, customized and localized to meet the specific local requirements by agencies
other than users. Examples include the country-specific versions of BIM software, local BIM software plug-ins,
BIM specifications initiated by the local governments. In the active mode, BIM solutions are “fetched” by the userorganization. It denotes a more active state that an organization constantly involves in the adaptation of BIM tosuit its own requirements, e.g., in-house development of technology, regulation developed based on examples fromother countries. The mixed mode is an intermediate state, in which developers provide BIM solutions customized
to a certain degree, then users adapt the solutions to tailor their requirements, e.g., the customization sections in
Industry Foundation Class (IFC) schema.

247Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 3. The typology of the mode of last-mile process
4.3

The design framework and variables for last-mile problems of BIM adoption
The design framework is a structural arrangement of variables that enables analyzing, optimizing, anddesigning a solution for the last-mile problem. Inspired by Lim et al (2018), the variables can be classified into
structural variables and contingency variables (Figure 4). The structural variables are descriptive indicators
describing the last-mile configurations, such as the geographical distance between the start point and end point,
the geographical coverage concerning the geographic area of the group of local users, and the mode of last-mileprocess. Comparatively, contingency variables are factors influencing the permutations of structural variables.
Contingency variables can be grouped into several dimensions based on different criteria. For example, they canbe grouped according to components in the last-mile model, i.e., BIM dimension, organization dimension,developer dimension and industry dimension (e.g., the external environment of BIM adoption). The variable canbe also categorized into technical, procedural, economic, and social dimensions. The selection of dimensions
depends on the perspective of analysis, BIM expectations and requirements on the last mile.

Figure 4. The framework for analysing the last-mile problems of BIM adoption
Properly harnessed, the framework can help different stakeholders to achieve their BIM objectives. For
examples, based on the analytic framework, software vendors can set up product development andcommercialization strategies to enhance market penetration. BIM users can analyze the critical barriers hindering
the last-mile adoption in their existing organization practice, and thus adjust BIM adoption strategies. They may
also identify the specific parts of BIM solutions to modify. The framework also assists policymakers or
administrators to identify the weakest part in the local last-mile BIM adoption, and devise regulations,specifications, mandates, and incentive mechanism to further promote BIM adoption.
5. DISCUSSION AND CONCLUSION
Amidst the rapid diffusion of BIM in the AEC industry, an increasing number of organizations in the
globe are willing to explore and adopt BIM. However, the existing BIM solutions, be they software, informationhub, execution plan, and specifications, are dominated and led by a few countries such as the U.S and Europe. For
the majority organizations, to catch up with the world-class practice, they tend to benchmark, select and follow
the practices that have already been proven effective by global leaders.
The problems lie in the last mile, which is referred to as the period from the point of the determinant of
BIM solutions until those solutions are integrated into organizational practice. For organization users, they find it
difficult to dramatically change its mindset, practice, and culture to adopt BIM. The difficulties become larger

248

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

when adopting BIM solutions from overseas countries due to the differences between the environments where thesolutions are originated and adopted. This incurs huge efforts to adapt BIM solutions to a specific local setting.
For software vendors, they hardly provide highly customized solutions targeting every market, in considering thehigh cost of development and maintenance. International standardization bodies and best-practice organizations
also hold similar concerns. Thus, a critical problem here is how to synchronize and balance the different
requirements of such a last-mile network.
This paper aims to take the first step by articulating the last-mile problem of BIM adoption. It does so by
proposing a working definition, identifying the components and typology of the last-mile model, and devising aframework to analyze the problems in the last mile. Specifically, a last-mile model consists of three components– the BIM solutions to be diffused, a transit point where BIM solutions are first removed from its origins andintroduced to users, and a destination organization in which BIM will be adopted. The last-mile process can havethree forms, namely the active, passive and mixed mode based on the involvement of organizations in “fetching”
and adapting BIM solutions based on their specific needs.
The proposed last-mile model can be used to analyze the last-mile BIM adoption barriers and design BIM
strategies accordingly. For example, from a holistic point of view, the existing last-mile process is largely active;
the majority of organizations tend to spend considerable efforts to adapt BIM technology, its accommodatingorganization practice and supporting specifications for their specific application scenarios. This costs lessdevelopment efforts for software vendors. However, it incurs huge costs of BIM adoption especially for the early
adopters of a locality and keeps a great number of organizations, especially the small- and medium-sizeorganizations from even trying and exploring BIM. To solve the problem, incentives can be designed to encourage
mainstreaming software vendors, local agencies, or BIM users to develop localized BIM solutions.
This study has both academic and practical implications. It offers a set of languages to systematicallydescribe the last-mile problem of BIM adoption, which leads to an improved understanding of the last-mile process
and problems therein. This also enables a deep dialogue among different stakeholders to work collaboratively tobridge the last-mile problem. For practitioners, this paper facilitates to analyze the last-mile problems of BIM
adoption, and develop strategies accordingly – product development and commercialization strategies for software
vendors, BIM adoption and adaptation guidelines for organizations, and incentive mechanisms or mandates forpolicymakers. Future research can work on evaluating the last-mile model in real-life practices, and exploring how
the different contingencies influence the last-mile problem of BIM adoption at organizational level.
REFERENCES
Ahmed, A. L., & Kassem, M. (2018). A unified BIM adoption taxonomy: Conceptual development,
empirical validation and application. Automation in Construction, 96, 103-127.
Ahmed, A., Kawalek, J., & Kassem, M. (2017, July). A conceptual model for investigating BIM adoption
by organisations. In LC3 2017: Volume I–Proceedings of the Joint Conference on Computing in Construction(JC3). Heraklion, Greece.
Aized, T., & Srai, J. S. (2014). Hierarchical modelling of Last Mile logistic distribution system. The
International Journal of Advanced Manufacturing Technology, 70(5-8), 1053-1061.
Arayici, Y., Coates, P., Koskela, L., Kagioglou, M., Usher, C., & O'Reilly, K. J. S. S. (2011). BIM
adoption and implementation for architectural practices. Structural survey, 29(1), 7-25.
Azhar, S., Brown, J., & Farooqui, R. (2009). BIM-based sustainability analysis: An evaluation of building
performance analysis software. In Proceedings of the 45th ASC annual conference (Vol. 1, No. 4, pp. 90-93).
Brady, T., & Davies, A. (2004). Building project capabilities: from exploratory to exploitative learning.
Organization studies, 25(9), 1601-1621.
Bui, N., Merschbrock, C., & Munkvold, B. E. (2016). A review of Building Information Modelling forconstruction in developing countries. Procedia Engineering, 164, 487-494.
Cao, D., Li, H., & Wang, G. (2014). Impacts of isomorphic pressures on BIM adoption in construction
projects. Journal of Construction Engineering and Management, 140(12), 04014056.
Chang, C. Y. (2014). An economic framework for analyzing the incentive problems in building
information modeling systems. In Proceedings of Academy of Management annual meeting, Philadelphia, USA.
Ding, Z., Zuo, J., Wu, J., & Wang, J. Y. (2015). Key factors for the BIM adoption by architects: A China
study. Engineering, Construction and Architectural Management, 22(6), 732-748.
Elmualim, A., & Gilder, J. (2014). BIM: innovation in design management, influence and challenges of
implementation. Architectural Engineering and design management, 10(3-4), 183-199.
Fink, A. (1998), Conducting literature research reviews: from paper to the Internet, Sage Publications,Inc., Thousand Oaks, CA.
Gevaers, R., Van de Voorde, E., & Vanelslander, T. (2011). Characteristics and typology of last-milelogistics from an innovation perspective in an urban context. City Distribution and Urban Freight Transport:
Multiple Perspectives, Edward Elgar Publishing, 56-71.

249

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Harrington, T. S., Singh Srai, J., Kumar, M., & Wohlrab, J. (2016). Identifying design criteria for urban
system ‘last-mile’solutions–a multi-stakeholder perspective. Production Planning & Control, 27(6), 456-476.
Herr, C. M., & Fischer, T. (2019). BIM adoption across the Chinese AEC industries: An extended BIM
adoption model. Journal of Computational Design and Engineering, 6(2), 173-178.
Hochscheid, E., & Halin, G. (2018). A model to approach BIM adoption process and possible BIM
implementation failures.
Jung, Y. and Joo, M. (2011), “Building information modelling (BIM) framework for practicalimplementation”, Automation in construction, 20(2), pp.126-133.
Liang, C., Lu, W., Rowlinson, S. and Zhang, X. (2016), “Development of a multifunctional BIM maturity
model” Journal of Construction Engineering and Management, 142(11), p.06016003-1-06016003-9.
Lim, S. F. W., Jin, X., & Srai, J. S. (2018). Consumer-driven e-commerce: A literature review, design
framework, and research agenda on last-mile logistics models. International Journal of Physical Distribution &Logistics Management, 48(3), 308-332.
Mentzer, J. T., DeWitt, W., Keebler, J. S., Min, S., Nix, N. W., Smith, C. D., & Zacharia, Z. G. (2001).
Defining supply chain management. Journal of Business logistics, 22(2), 1-25.
Meredith, J. (1993), “Theory building through conceptual methods”, International Journal of Operations& Production Management, Vol. 13 No. 5, pp.3-11.
Nandi, S., Thota, S., Nag, A., Divyasukhananda, S., Goswami, P., Aravindakshan, A., ... & Mukherjee, B.
(2016). Computing for rural empowerment: enabled by last-mile telecommunications. IEEE Communications
Magazine, 54(6), 102-109.
Niu, Y., Lu, W., Liu, D., Chen, K., Anumba, C., & Huang, G. G. (2016). An SCO-enabled logistics andsupply chain–management system in construction. Journal of Construction Engineering and Management, 143(3),04016103.
Papadonikolaki, E. (2018). Loosely Coupled Systems of Innovation: Aligning BIM Adoption with
Implementation in Dutch Construction. Journal of Management in Engineering, 34(6), 05018009.
Peansupap, V., & Walker, D. (2005). Exploratory factors influencing information and communication
technology diffusion and adoption within Australian construction organizations: a micro analysis. Construction
Innovation, 5(3), 135-157.
Russo, F., & Comi, A. (2011). A model system for the ex-ante assessment of city logistics measures.
Research in transportation economics, 31(1), 81-87.
Speta, J. B. (2000). Handicapping the race for the last mile: A critque of open access rules for broadband
platforms. Yale J. on Reg., 17, 39.
Succar, B. (2009). “Building information modelling framework: A research and delivery foundation forindustry stakeholders”. Automation in construction, Vol. 18 No. 3, pp.357-375.
Succar, B., & Kassem, M. (2015). Macro-BIM adoption: Conceptual structures. Automation inconstruction, 57, 64-79.
Taniguchi, E., & Tamagawa, D. (2005). Evaluating city logistics measures considering the behavior ofseveral stakeholders. Journal of the Eastern Asia Society for Transportation Studies, 6, 3062-3076.
Wang, H., & Odoni, A. (2014). Approximating the performance of a “last mile” transportation system.
Transportation Science, 50(2), 659-675.
Xu, H., Feng, J. and Li, S. (2014), “Users-orientated evaluation of building information model in the
Chinese construction industry”, Automation in Construction, Vol. 39, pp.32-46.

250

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

BIM-BASED WALL FRAMING CALCULATION ALGORITHMS FOR DETAILEDQUANTITY TAKEOFF
Chavanont Khosakitchalert1, Nobuyoshi Yabuki2, and Tomohiro Fukuda3
1) Ph.D. Candidate, Division of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka
University, Japan. Email: khosakitchalert@it.see.eng.osaka-u.ac.jp
2) Ph.D., Prof., Division of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka
University, Japan. Email: yabuki@see.eng.osaka-u.ac.jp
3) Ph.D., Assoc. Prof., Division of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka
University, Japan. Email: fukuda@see.eng.osaka-u.ac.jp

Abstract: Although an automated quantity takeoff using building information modeling (BIM) is proved to befaster and more reliable than the traditional quantity takeoff method, the information and geometries in a BIM
model must be input correctly. Drywall is a wall type that consists of wall framings as a core structure layer anddrywall sheets as finish layers. During a tendering phase, the area of each layer of drywall is used for costestimation. However, during a construction phase, the material quantity of wall framings must be calculated inlength in order to purchase the materials effectively. If the wall framings do not exist in the BIM model,
construction practitioners have to create them or calculate their length manually. Creating wall framing elements
in a BIM model is a time-consuming and error-prone task, especially in a large scale project. The increased
geometries in a BIM model also affect the working performance of the software. This research proposes a methodthat automatically calculates the lengths of vertical and horizontal members of wall framings from the extracted
wall surfaces and the input spacing values. The method also eliminates the region of walls that overlap withstructural elements such as columns and beams. The validation is done by using an interior construction project asa case study. It showed that the proposed method provides an accurate wall framing quantity when compared with
the quantity results from the BIM model that has wall framings and the quantity results from the manual calculation
methods. With this method, the wall framing elements do not need to be created in a BIM model for quantity
takeoff. The modeling time can be saved while construction practitioners can get an accurate wall framing quantity
for purchasing material during a construction phase.
Keywords:

Building Information Modeling (BIM), Quantity takeoff, BIM-based quantity takeoff, Wall framing

1. INTRODUCTION
Quantity takeoff, a measurement of building elements, is a time-consuming and error-prone task in thedesign and construction processes (Holm et al., 2005; Monteiro & Martins, 2013). Although the development of
information technology such as computer-aided design (CAD) and spreadsheet software increases conveniencefor practitioners and improve their work efficiencies, the process of measurement still based on 2D drawings,
which require manual measurement and human interpretation.
Building Information Modeling (BIM) technology introduces a new method for quantity takeoff byextracting quantities directly from object-oriented digital models of a building (Sacks et al., 2018). This method iscalled BIM-based quantity takeoff. It delivers less working time and more reliable outcome when compared to the
traditional method (Bečvarovská & Matějka, 2014; Sattineni & Bradford, 2011). However, because the quantities
are extracted from geometries and information of building elements, they can be wrong if the geometries orinformation do not exist or are not correct. Therefore, the accuracy and reliability of the extracted quantities dependon the completeness and correctness of the BIM model.
Quantity takeoff of building elements that consist of sub-components is complicated in both traditionaland BIM-based method. In the traditional method, quantity surveyors have to measure each sub-component
manually. In the BIM-based method, BIM modelers have to model every sub-component in order to extract
accurate quantities. Walls are the example of this complication because they consist of a core structure layer andfinish layers, which require separated measurements. During a tendering phase, only the area of each wall layer ismeasured and used for cost estimation (Royal Institution of Chartered Surveyors (RISC), 2012). However, in actual
construction, walls are made up of sub-components such as bricks in masonry walls and wall framings in drywalls.
Although the quantities of these sub-components are not necessary for cost estimation, it is necessary for materialpurchase and material optimization in a construction phase (Peansupap & Thuanthongdee, 2016; Sacks et al.,2018).
The focal building element in this research is a wall framing of drywall. The drywall is made up of wall
framings as a core structure layer and drywall sheets as finish layers. During the construction phase, a generalcontractor or a sub-contractor has to estimate the length of wall framings for material purchase. According to theLevel of Development (LOD) Specification 2019 (BIMForum, 2019), the drywall at LOD 350 represents thedetails of wall framings and the drywall at LOD 400 represent the connection details of wall framings. Therefore,
the drywall at LOD 350 and higher can be used for wall framing takeoff. Nevertheless, a BIM model that251

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

contractors receive from design teams usually do not contain all necessary information for quantity takeoff andhence they have to develop their own BIM models (Sattineni & Bradford, 2011). Developing model elements in a
BIM model from one LOD level to another is time-consuming and error-prone especially in a large scale project.
The tight time constraints in a construction site are also challenging for construction practitioners to develop BIM
models (Liu et al., 2018). Furthermore, the details of sub-components in a large scale project increase thegeometries and the file size of a BIM model which can affect the working performance of the software.
The objective of this research is to develop an automated calculation approach that can estimate thequantities of the wall framings without creating model elements in a BIM model. The main contribution of thispaper is to propose a new method for calculating wall framings quantities from a BIM model which is faster andeasier than making detailed framing models. The proposed method uses the surfaces of drywalls in a BIM modeland the input spacing values to calculate the lengths of the vertical and horizontal members of wall framings. The
calculation algorithms are developed according to the construction method of light gauge steel framing system for
non-load bearing walls. The proposed method is validated using a case study by comparing the quantity resultsfrom the proposed method with the quantity results from the BIM model that has wall framings and the quantity
results from the manual calculation methods. After that, the results are discussed and the conclusion is summarized.
2. BACKGROUND AND RELATED RESEARCH
2.1 Construction method for light gauge steel framing system
A light gauge steel framing is a structural system of drywall that is widely used as an alternative to a woodframing in buildings nowadays (Packer, 2016). The principle of construction is similar to wood framing. The
framing system consists of horizontal and vertical members. The horizontal members are called track or U-track
which is named by the U-shaped section of the steel member. The vertical members are called stud or C-stud whichis named by the C-shaped section of the steel member. Figure 1 illustrates components in a light gauge steel
framing system.

Figure 1. Components in a light gauge steel framing system (front view)
The construction method for a light gauge steel framing system is inquired from installation guides bymaterial suppliers (The Siam Cement Group Public Company Limited, 2016; The Siam Gypsum Industry(Saraburi) Company Limited, 2015; USG Australasia, 2011). It can be summarized that the ordinary spacing for
C-stud is 600 mm maximum on center. The spacing for U-track depends on the need of nogging to prevent studrotation and bucking. The intersection and corner details are divided into two methods (see Figure 2) which dependon the condition of the construction site. The method one needs one extra stud at a corner and a T-intersection andtwo extra studs at a cross intersection (see Figure 2a). The method two only needs one extra stud at a crossintersection (see Figure 2b). The studs at both sides of openings (doors and windows) are to use two C-studs
combined into boxed studs (see Figure 1a). The header or lintel and the sill trimmer are to use U-track (see Figure
1b and Figure 1c). The jack studs between header or sill trimmer and top or bottom track are to use the same sizeand spacing as other C-studs (see Figure 1d).
The specific construction details used in the case study such as the spacing of members and the intersection
and corner details are obtained from an interview with the construction manager of the case study building. This
information is then used as the setups for the prototype system in the validation section.

252

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 2. The intersection and corner details of light gauge steel framing system
2.2

Quantity Takeoff
According to Holm et al. (2005), the traditional quantity takeoff methods for wall framings are divided
into two calculation methods: by wall framing spacing and by wall framing per area. Additional interviews withconstruction practitioners were conducted to gain more details about both methods which can be summarized asfollows. The first method calculates the length of C-studs and U-tracks by wall dimensions and spacing between
members. As for the C-stud, the number of studs is calculated by dividing the total wall length by the stud spacing,then round up the result to the whole number and plus one for the stud at the end. The extra studs at the intersections,corners, and openings should be counted separately and add to the number of studs. After that multiplying the total
number of studs by the wall height to find the total length of C-studs. As for the U-tracks, the number of tracks is
calculated by dividing the total wall height by the track spacing, then round up the result to the whole number andplus one for the track at the end. After that multiplying the total number of tracks by the wall length to find thetotal length of U-track. The second method calculates the length of C-studs and U-tracks by multiplying the net
wall area by the length of studs or tracks per one square meter. Then the extra length of studs at the intersections,corners, and openings should be added to the results. It should be noted that both methods are an estimation. The
quantity results may be more or less than the actual use. The most accurate calculation would be the method thatdraws the actual number of studs and tracks and measures them. However, this method is very time-consuming
and not practical in a large scale project.
BIM has a capability to extract quantities information from building model elements (Sacks et al., 2018);
therefore, the actual number of studs and tracks can be extracted instantly if they exist in a model. However, a
BIM model from a design stage usually does not contain all necessary information (Olsen & Taylor, 2017;
Sattineni & Bradford, 2011). Developing a BIM model to higher LOD within a limited time constraint become achallenge for construction practitioners (Liu et al., 2018). The modeling time increases in the range of two toeleven times when developing a BIM model from one LOD to another (Leite et al., 2011). Furthermore, theincreased LOD also increases the geometries and the file size of a BIM model, which causes scalability problems.
Large scale project that contains millions of objects consumes memory resource and increases the processing timeof a computer (Sacks et al., 2018). Therefore, it may not necessary to create every detailed in building model
elements (LOD 350 or higher).
Several pieces of research have been done to improve the process of construction and quantity takeoff
using computer-aided design (CAD) or BIM approach. Our previous research proposed an automatic method thatenhances the accuracy of the quantities of each wall and floor layer in a BIM model by using clash detection
capability (Khosakitchalert et al., 2018, 2019). Choi et al. (2015) proposed a quantity takeoff process and a
prototype system for estimation of building frame in an early design stage. Rajabi et al. (2015) developed a systemthat can estimate quantities of mechanical, electrical, and plumbing (MEP) from a BIM that has no detailedbuilding model available. Cho & Chun (2015) proposed a cost estimation system for reinforced concrete structuresusing BIM-based quantity takeoff and data mining. Manrique et al. (2015) proposed a method that automaticallygenerates shop drawings of wood-framing in a 3D CAD environment. Lim et al. (2016) proposed a rebar estimation
algorithm to quickly evaluate the financial performance of the design. Zaki et al. (2017) developed algorithms that
can automatically generate a masonry wall assembly at LOD 400 in a given BIM. Liu et al. (2018) developed arule-based BIM approach to minimize material waste for drywall sheets.
However, the estimation algorithms for light gauge steel framing of drywall that use the capability of BIM
has not been proposed. This proposed method will not change the original BIM model. Therefore, the speed andthe accuracy of the proposed method would benefit for construction practitioners who need a quantity of wall
framings but does not want to develop every detail of walls in a BIM model.
3. PROPOSED METHOD
The proposed method uses the surfaces of drywalls in a BIM model and the spacing values of wall
framings to calculate the total length of vertical and horizontal members of wall framings. To implement the253Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

proposed method, a prototype system is developed using Dynamo 1.3.3.4111, a visual programming extension in
Autodesk Revit (“Dynamo BIM,” 2016). This is because this software platform can connect with BIM models
directly and easily for both non-programmers and programmers to develop and visualize the results.
The process flowchart of the proposed method is shown in Figure 3. The first step is the preparation of
surfaces of drywall. The wall elements are imported from a BIM model and the wall types that are not drywall will
be filtered out. The location lines and the wall heights are extracted from the drywall. They are then used to form
wall surfaces. The structural columns, structural framings (beams), floors, doors, and windows are imported froma BIM model and combined into an integrated geometry. This geometry is used to check the intersection with thewall surfaces. The regions of the wall surfaces that overlap with the geometry will be subtracted. The final products
are the net surfaces of walls that have been subtracted with columns, beams, floors, doors, and windows. The net
wall surfaces, the wall heights, and the wall location lines will be used to calculate the length of wall framings in
the following steps.
Start
Input structural columns,
structural framings, and floors

No

Input
doors

Combine into an
integrated geometry

Input
windows

Extract wall
location lines

Input
walls

Extractwallheights

Select only
drywalls

Check forintersection

Yes

Createsurface fromextrusion(a)Manualinput
track
spacing

The netwall
surfaces

Subtract theoverlapping
regions

Extract
perimeter lines

Select the top horizontal lines(represent header tracks)

Extract
perimeter lines

Select the top and bottom
horizontal lines (represent
header and sill trimmer tracks)

Divide wallheights byspacing

Createsequence ofpoints on Z axis

Extract
perimeterlines

Duplicate location lines
to the points (exclude thefirst points)

Select the bottom
horizontal lines (represent
bottom U-tracks)

Calculatethe lengthof lines

The length ofbottom U-tracks

Select the top horizontal
lines (represent top Utracks)

Calculatethe lengthof lines

The length of
Top U-tracks

Intersect lines
with wall
surfaces

The length ofmiddle U-tracks

Calculate the length of
lines

(b)
Manualinput studspacing

Find points on thecorners of doors
and windows
Intersect lines
with wall
surfaces

Project them on
the wall location
lines

Remove lines thatintersect with doors
and windows

Input
doors

Combine into
an integrated
geometry

Input
windows

Extract
perimeter lines

Select the
top andbottomhorizontal
lines

Split thelines by the
pointsCalculate thelength of eachline

Divide thelengths byspacing

Create sequenceof points on thelines

Create vertical lines
on sequence of points

Calculate the lengthof lines (represent
common C-studs)

Calculate thelength of each
line

The
length of
C-studs

Divide thelengths byspacing

Create verticallines on sequenceof points

Createsequence ofpoints the lines

Find intersection
points of the wall
location lines

Points of thecornerPoints of the
T-intersection
Point of the crossintersection(d)Intersectlines withwall surfaces

Calculate the lengthof lines (represent
jack studs)

Manual input
jack stud spacing(c)

Plus theend pointof the lines

Select the side
vertical lines ofdoors and windows

Find C-stud linesthat overlap theside vertical lines

Find Cstud linesthat areon thosepointsSelect lines(represent
extra studs atintersections
and corners)

Select lines (represent
extra studs at both sides
of openings)

Calculatethe lengthof lines

Calculatethe lengthof lines

The length of
extra C-studs
at theintersections
and corners

The length of
extra C-studs
at both sides
of openings

End

Figure 3. The process flowchart of the proposed method
3.1

Calculation algorithms for U-tracks
U-tracks are the horizontal members of light gauge steel framing system. They consist of top tracks,
bottom tracks, and middle tracks. Figure 3a shows the flowchart of the calculation algorithms for U-tracks. The
length of the top and bottom tracks can be obtained from the perimeter lines that are extracted from the net wall
surfaces. The top horizontal lines will be selected as the representative of the top U-track. The bottom horizontal

254

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019lines will be selected as the representative of the bottom U-track. These lines will be then calculated their lengths.
The middle U-tracks consist of the tracks that are used as noggings, the tracks that are used as headers at doors
and windows, and the tracks that are used as sill trimmers at windows. The length of header tracks is calculated
from the top horizontal line extracted from door and window geometries. The length of sill trimmer tracks is
calculated from the bottom horizontal line extracted from window geometries. The length of nogging tracks is
calculated from the following process. First, the wall heights are divided by the input track spacing. The results
are rounded up to the whole number and used to create the sequence of points on the Z axis of the walls. Second,
the wall location lines are duplicated to those points except the first points which are the location of the bottom Utracks. Third, the lines are intersected with the net wall surfaces to eliminate the part of lines that are not inside thesurfaces. Finally, the length of noggin tracks is calculated from the lines and they will be combined with the lengthof header tracks and still trimmer tracks to form the total length of middle U-tracks. Figure 4a shows thevisualization of U-tracks represented by lines in Dynamo.
3.2

Calculation algorithms for C-studs
C-studs are the vertical members of light gauge steel framing system. They consist of common studs andjack studs. Figure 3b shows the flowchart of the calculation algorithms for C-studs. The length of common studs
is calculated from the following process. First, the points on the corners of doors and windows are detected fromthe net wall surfaces. They are then projected on the wall location lines. These points are used to split the wall
location lines into segments. Second, the lengths of line segments are divided by the input stud spacing. The results
are rounded up to the whole number and used to create the sequence of points on the lines. In addition, the end
points of the lines are added to the sequence of points. The vertical lines are then created on these points. Third,
the lines are intersected with the net wall surfaces to eliminate the part of lines that are not inside the surfaces.
Fourth, the lines are checked the intersection with door and window geometries in order to remove the lines thatintersect with doors and windows. Finally, the length of the common stud is calculated from the lines. Similar tothe common studs, the length of jack studs is calculated from the following process. First, the top and bottom
horizontal lines are selected from the perimeter lines of door and window geometries. Secondly, the lengths of line
are divided by the input stud spacing. The results are rounded up to the whole number and used to create thesequence of points on the lines. The vertical lines are then created on these points. Third, the lines are intersectedwith the net wall surfaces to eliminate the part of lines that are not inside the surfaces. Finally, the length of thejack stud is calculated from the lines and they will be combined with the length of common studs to form the total
length of C-studs. Figure 4b shows the visualization of C-studs represented by lines in Dynamo.
3.3

Calculation algorithms for extra C-studs at intersections and corners
Different numbers of extra studs at intersections and corners are used according to different construction
details (see Figure 2). When two walls meet each other, the calculation algorithms for C-studs only calculate one
stud per wall at the intersection point. Therefore, the calculation for extra studs is needed. Figure 3c show the
flowchart of the calculation algorithms for extra C-studs at the intersections and corners. The wall location lines
are used to find the intersection points between walls. The points are divided into points of the corner, points of
the T-intersection, and points of the cross-intersection. Then the C-stud lines that are on the points will be selected.
The lengths of C-stud lines are calculated again for the extra C-studs at the intersections and corners. Figure 4cshows the visualization of extra C-studs at intersections and corners represented by lines in Dynamo.
3.4

Calculation algorithms for extra C-studs at both side of openings
Figure 3d show the flowchart of the calculation algorithms for extra C-studs at both side of openings.
These extra studs are calculated when the studs at both sides of openings are two C-studs combined into boxed
studs. The side vertical lines are selected from the perimeter lines of door and window geometries. Then the Cstud lines that overlap with the side vertical lines will be selected. The lengths of C-stud lines are calculated again
for the extra C-stud at both sides of openings. Figure 4d shows the visualization of extra C-studs at both sides of
openings represented by lines in Dynamo.

Figure 4. (a) Lines represent U-tracks. (b) Lines represent C-studs. (c) Lines represent extra C-studs atintersections and corners. (d) Lines represent extra C-stud at both sides of openings.

255

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

4. VALIDATION
A case study was used to verify the prototype system and validate the proposed method. It is an interior
construction project on the fifth floor of Chulapat 14 building located at Chulalongkorn University, Bangkok,
Thailand (see Figure 5a). This case study covers all scenarios of walls interact with other walls, structural columns,
beams, and floors. The gross internal area of this floor is 1,280 m2. Drywalls are used to divide the interior space
into 28 rooms. This construction project uses construction details as follows. The typical wall height is 3.65 m andthe wall height under beams is 3.45 m. The spacing of vertical members (C-studs) is 600 mm maximum on center.
The spacing of horizontal members (U-tracks) is also 600 mm maximum on center, which is divided into sevenrows. The intersection and corner detail is the detail (a) in Figure 2. The studs at both sides of doors and windows
are to use two C-studs combined into boxed studs. The header and still trimmer are to use U-track and the jackstuds are to use C-stud with the same 600 mm spacing.
Two BIM models were created in Autodesk Revit 2018.2. The first BIM model is the accurate detailed
BIM model used as a baseline for the comparison. This model has drywalls at LOD 350. The wall framings wereprecisely modeled according to actual construction details (see Figure 5b). The quantities of wall framings wereextracted directly from the BIM model. The second BIM model has drywalls at LOD 300. The drywalls are single
model elements with defined material layers (see Figure 5c). Some walls were created by overlapping withstructural columns and beams. The prototype system was applied to this BIM model. Figure 5d shows thevisualization of wall framings represented by lines in Dynamo. The quantities of wall framings were automaticallycalculated by the prototype system and the results were compared with the results from the accurate detailed BIM
model.

Figure 5. (a) The photo of the case study. (b) The wall framings in the accurate detailed BIM model (drywall
sheets are hidden from this view). (c) The BIM model at LOD 300. (d) The visualization of wall framings
represented by lines in Dynamo.
Furthermore, manual quantity takeoff methods were used to compare the performance of the proposed
method. According to section 2.2, the manual quantity takeoff methods for wall framings are divided into two
methods. In this validation, the first method is named manual method 1. It calculates the length of studs and tracks
from wall dimensions and spacing between members. The total wall length measured from the construction
drawings is 314.20 m. The wall height and the spacing of stud and track is described in the first paragraph ofsection 4. The second method is named manual method 2. It calculates the length of studs and tracks from net wallarea and length of studs and tracks per one square meter. The net wall area measured from the construction
drawings is 947.91 m2. The length of studs per one square meter is 2 m. The length of tracks per one square meteris 2 m. The net wall area excludes the area of openings and the area that overlap with structural columns and beams.
The length of extra studs at the intersections, corners, and openings were measured separately and added to theresults of both manual method 1 and manual method 2. The numbers of corners, T-intersections and crossintersections are 20, 24, and 6 respectively. The number of doors is 47 and the number of windows is 5.
The quantity results from the proposed method, the manual method 1, and the manual method 2 werecompared with the quantity results from accurate detailed BIM model. Table 1 shows the comparison and thedeviation of each method when compared to the baseline.

256

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 1. Comparison of the wall framing quantities among the accurate detailed BIM model, the proposed
method, the manual method 1, and the manual method 2.
Accurate
Detailed
Proposed Method
Manual Method 1
Manual Method 2
BIM Model
Wall Framings
LengthLength Deviation Length Deviation Length Deviation
(%)
(%)
(%)
(m)
(m)
(m)
(m)
C-studs
2,726.18 2,729.35
0.12 2,857.95
4.83 2,837.52
4.08
U-tracks (Total)

1,971.49

1,974.66

0.16

2,199.40

11.56

1,895.82-3.84- Top tracks

320.30321.800.47

315.30

-1.56

N/A

N/A

- Bottom tracks

263.98261.90-0.79

315.30

19.44

N/A

N/A

- Middle tracks

1,387.21

1,390.96

0.27

1,576.50

13.65

N/A

N/A

The quantity of wall framings from the accurate detailed BIM model is the most accurate, but the timespent on creating wall framings in the BIM model is massively high when compared to the execution time of the
proposed method. Additionally, the increased geometries in a BIM model slow down the viewport rendering speed,
especially in a computer with less memory and a low-end graphics card.
The quantity of C-studs from the proposed method is 0.12% higher than the accurate detailed BIM modeland the quantity of U-tracks from the proposed method is 0.16% higher than the accurate detailed BIM model.
The deviation of C-studs and U-tracks are less than 0.20% which can be said that the wall framing quantities fromthe proposed method are accurate when compared to the accurate detailed BIM model. The slightly excess
quantities are not significant because generally, contractors will add a percentage of waste on top of the calculated
quantities when purchasing materials.
Furthermore, the result from the proposed method is more accurate than the result from the manual method
1 and 2. The quantity of C-studs from the manual method 1 is 4.83% higher than the accurate detailed BIM modeland the quantity of C-studs from the manual method 2 is 4.08% higher than the accurate detailed BIM model. The
quantity of U-tracks from the manual method 1 is 11.56% higher than the accurate detailed BIM model. The
deviation of U-tracks from the manual method 1 is the highest because this method does not subtract the U-trackat doors and windows. All of these deviations are usable in practice because the quantities results are higher than
the actual use. However, leftover materials could occur, especially when a percentage of waste is added to thedeviated quantities which causes much more excess materials. On the other hand, the quantity of U-tracks fromthe manual method 2 is 3.84% lower than the accurate detailed BIM model. The absent quantity would be a
problem when purchasing materials because the quantity of U-tracks may not be enough during the construction.
The percentage of waste that is added may or may not cover the absent quantity.
In summary, the proposed method produces the quantity results that is as accurate as the results from theaccurate detailed BIM model, but it spends much less time to execute. Both manual methods are usable in practice,
but not as efficient as the proposed method. The results from different manual methods are not constant and theexcess quantities plus a percentage of waste would cause tremendous excess materials in a construction site.
5. CONCLUSION
This paper presented a method to estimate the quantity of wall framing from a BIM model that does notcontain wall framing elements. The proposed method extracts surfaces of drywall in a BIM model and eliminates
the parts of the surfaces that overlap with structural elements by using clash detection capabilities. Afterward, thesurfaces and the spacing values of wall framings are used to calculate the positions and the lengths of the verticaland horizontal members of wall framings. The prototype system is developed using Autodesk Revit and Dynamo
extension.
The validation was done by using an interior construction project on the fifth floor of Chulapat 14 building
located at Chulalongkorn University, Bangkok, Thailand. A BIM model with accurate wall framings was createdas a baseline for the comparison. A BIM model with walls that are single model elements was used to apply the
prototype system. Two manual quantity takeoff methods were also done to compare the performance of the
proposed method. The comparison revealed that the quantity of wall framing from the proposed method is accurate
when compared to the results from the baseline. Moreover, the results from the proposed method have lessdeviation than the results from the manual methods. With this method, construction practitioners do not need tocreate wall framing elements in a BIM model to measure quantities, which is time-consuming and error-prone,especially in a large scale project. The accurate estimation of the quantity of wall framings is beneficial for materialpurchase during a construction phase because excess materials and waste can be reduced. The principle of the
proposed method can be used to develop a wall framing estimation system in any BIM software products.

257

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

REFERENCES
Bečvarovská, R. and Matějka, P. (2014). Comparative Analysis of Creating Traditional Quantity Takeoff
Method and Using a BIM Tool. Construction Maeconomics Conference 2014.
BIMForum. (2019). Level of Development (LOD) Specification Part 1 & Commentary For Building Information
Models and Data. Retrieved from https://bimforum.org/wp-content/uploads/2019/04/LOD-Spec-2019Part-I-and-Guide-2019-04-29.pdf
Cho, J. and Chun, J. (2015). Cost Estimating Methods for RC Structures by Quantity Takeoff and Quantity
Prediction in the Design Development Stage. Journal of Asian Architecture and Building Engineering,14(1), 65–72.
Choi, J., Kim, H., and Kim, I. (2015). Open BIM-based quantity take-off system for schematic estimation ofbuilding frame in early design stage. Journal of Computational Design and Engineering, 2, 16–25.
Dynamo BIM. (2016). Retrieved from website: http://dynamobim.org/
Holm, L., Schaufelberger, J.E., Griffin, D., and Cole, T. (2005). Construction Cost Estimating: Process andPractices (1st ed.). Pearson.
Khosakitchalert, C., Yabuki, N., and Fukuda, T. (2018). The Accuracy Enhancement of Architectural Walls
Quantity Takeoff for Schematic BIM Models. In Proceedings of the 35th International Symposium on
Automation and Robotics in Construction (ISARC) (pp. 768–775). Berlin, Germany.
Khosakitchalert, C., Yabuki, N., and Fukuda, T. (2019). Improving the accuracy of BIM-based quantity takeoff
for compound elements. Automation in Construction, 106, 102891.
Leite, F., Akcamete, A., Akinci, B., Atasoy, G., and Kiziltas, S. (2011). Analysis of modeling effort and impact
of different levels of detail in building information models. Automation in Construction, 20(5), 601–609.
Lim, C., Hong, W.K., Lee, D., and Kim, S. (2016). Automatic Rebar Estimation Algorithms for Integrated
Project Delivery. Journal of Asian Architecture and Building Engineering, 15(3), 411–418.
Liu, H., Singh, G., Lu, M., Bouferguene, A., and Al-Hussein, M. (2018). BIM-based automated design andplanning for boarding of light-frame residential buildings. Automation in Construction, 89, 235–249.
Manrique, J. D., Al-Hussein, M., Bouferguene, A., and Nasseri, R. (2015). Automated generation of shop
drawings in residential construction. Automation in Construction, 55, 15–24.
Monteiro, A. and Martins, J.P. (2013). A survey on modeling guidelines for quantity takeoff-oriented BIM-based
design. Automation in Construction, 35, 238–253.
Olsen, D. and Taylor, J.M. (2017). Quantity Take-Off Using Building Information Modeling (BIM), and Its
Limiting Factors. Procedia Engineering, 196(2017), 1098–1105.
Packer, A.D. (2016). Building Measurement: New Rules of Measurement (2nd ed.). Routledge.
Peansupap, V. and Thuanthongdee, S. (2016). Levels of Development in BIM for Supporting Cost Estimation of
Building Construction Projects. In Proceedings of the 16th International Conference on Computing in
Civil and Building Engineering (ICCCBE) (pp. 671–678). Osaka, Japan.
Rajabi, M., Bigga, T., and Bartl, M.A. (2015). Optimization of the quantity take-off (QTO) process for
Mechanical, Electrical and plumbing (MEP) trades in tender estimation phase of the construction projects.
In Proceedings of the 32nd International Symposium on Automation and Robotics in Construction(ISARC) (pp. 1–8). Oulu, Finland.
Royal Institution of Chartered Surveyors (RISC). (2012). NRM 2: Detailed measurement for building works.
Retrieved from https://www.rics.org/globalassets/rics-website/media/upholding-professionalstandards/sector-standards/construction/nrm-2-detailed-measurement-for-building-works-1st-editionrics.pdf
Sacks, R., Eastman, C.M., Lee, G., and Teicholz, P.M. (2018). BIM Handbook: A Guide to Building Information
Modeling for Owners, Designers, Engineers, Contractors, and Facility Managers (3rd ed.). Wiley.
Sattineni, A. and Bradford, R. (2011). Estimating with BIM: A Survey of US Construction Companies. In
Proceedings of the 28th International Symposium on Automation and Robotics in Construction (ISARC)(pp. 564–569). Seoul, Korea.
The Siam Cement Group Public Company Limited. (2016). SCG Smartboard Installation Manual. Retrieved
from https://www.scgbuildingmaterials.com/th/Download/Manual.aspx
The Siam Gypsum Industry (Saraburi) Company Limited. (2015). ProWall Partition Metalprofile. Retrieved
from https://www.usgboral.com/content/dam/USGBoral/Thailand/Website/Documents/English/brochurecatalogues/ProWall.pdf
USG Australasia. (2011). Steel Stud & Track Installation Details. Retrieved fromhttps://www.usgboral.com/content/dam/USGBoral/Australia/Website/Documents/English/installationguide/usg-steel-stud-installation-7-11-aus.pdf
Zaki, T., Nassar, K., and Hosny, O. (2017). Parametric Blockwall-Assembly Algorithms for the AutomatedGeneration of Virtual Wall Mockups Using BIM. In Proceedings of the Architectural Engineering
Conference (pp. 844–854). Reston, VA: American Society of Civil Engineers.

258

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

BIM-SUPPORTED COMPLIANCE VERIFICATION OF PERFORMANCE-BASEDCAR-PARK VENTILATION DESIGN
Johannes Dimyadi1, Robert Amor2
1) PhD, CEO, Compliance Audit Systems Limited, Auckland, New Zealand. Email: jdimyadi@complianceauditsystems.com
2) PhD, Prof., School of Computer Science, University of Auckland, Auckland, New Zealand. Email: trebor@cs.auckland.ac.nz

Abstract: Modern regulatory frameworks, such as those enforceable in the UK, Australia, and New Zealand, allow
car-park ventilation systems to be designed to comply with performance-based objectives. This has provided
designers with the opportunity to propose an alternative solution when prescriptive requirements are either too
restrictive or costly to implement. The input data required for computations or numerical simulations to support
the performance-based design is often complex and may include the geometry, building usage or activities andbuilding system characteristics, as well as normative compliant-design parameters. Conventionally, this input data
is gathered manually from paper-based drawings and other written documentation including normative standards,
which is an inefficient and error-prone process. In this paper, we investigate to what extent can simulation input
data be generated from the available information shared through the Building Information Modelling (BIM)
collaborative process. Additionally, we explore an open standard computable representation of normativerequirements that can be used to automate some of the preparatory computations to further complement thesimulation input data. An exemplary use case is also described to illustrate the approach.
Keywords: BIM, CO, CFD, car-park ventilation, performance-based design.
1. INTRODUCTION
1.1 Conventional Car-park Ventilation Design
The requirements and performance of the ventilation design of enclosed car-parks has been studied
extensively for decades (Chan, Burnett, & Chow, 1998; Demir, 2015; Ho, Xue, & Tay, 2004; Li & Xiang, 2013).
There are generally two main objectives, namely the removal of fire-generated smoke pollutant and toxins in theair (Lu, Wang, Zhang, & Zhang, 2011), and the removal or dispersal of accumulated contaminants such as carbon
monoxide (CO) due to vehicle exhaust (Chow, 1995). This paper describes the latter and focuses on the designthat has the objective of maintaining the CO to a safe concentration level. Ventilation can be achieved naturally
by means of passive systems involving strategically placed wall openings having adequate sizes to induce a crossflow through the space (Chan & Chow, 2004), which is the most energy-efficient option. However, this option isoften constrained by the physical limitation of the car-park enclosure. Alternatively, a mechanical ventilation
system can be used to provide the required volumetric air change per hour (ACH) in the space using a system of
extract and supply air through a network of ducting. Mechanical ventilation systems may also be constrained by
the physical configuration of the enclosure and associated structural elements, particularly with respect to theavailable head room to accommodate ductwork and fans. A combination of passive natural ventilation system
supplemented by some mechanical means may be the best option in some cases.
The ventilation of semi-enclosed car-parking spaces has conventionally been designed to satisfy a set ofprescriptive requirements and to comply with the acceptable threshold for the minimum volumetric air change ratein the space or stipulated minimum air extract and supply rates. These prescriptive requirements are often overly
conservative and inflexible, which result in costly installations and a poor operating efficiency. Consequently, theperformance-based car-park ventilation design approach has become a popular alternative solution recognised byregulatory frameworks in some countries including UK, Australia, and New Zealand, due to its significant costsaving potential and the flexibility to adopt more innovative solutions (Chan, 2001). Recent technology inmechanical ventilation design includes space and energy saving air-moving devices such as impulse fans or jet
fans that can replace the need for ductwork (Špiljar, Drakulić, & Schneider, 2018; Viegas, 2009). Traditionally,mechanical ventilation systems have been designed to operate either continuously or on a preset daily on/off
schedule regardless of any variation in the space activity (Chan et al., 1998). In recent years, CO sensors have been
incorporated into the design to control the operation of the fans on demand depending on the level of COconcentration in the space, which is an energy saving feature that also reflects the actual activity in the car-park(Burnett & Chan, 1997).
However, the performance-based design approach demands more robust design procedures that often
include a verification of the proposed design by means of computational fluid dynamics (CFD) simulations(Gaekwad, 2017). Such a computational or simulation-oriented design process would require considerably more
complex input data. The collection of this data has conventionally been undertaken manually from paper-basedarchitectural drawings, written specifications, and normative standards conveyed in natural language for humaninterpretation, which is an inefficient and error-prone process.
Every car-park is unique, but there are also common features shared by all car-parks that can be used259

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

towards automating some aspects of the data collection process. This paper describes what information can beshared through the Building Information Modelling (BIM) collaborative process, which is gaining popularity
among building designers, to facilitate the transfer of the building geometry and related information to the CFD
simulation input data. Furthermore, the paper describes a method of representing normative requirements in an
open standard that can then be used towards the initial computations necessary to further complement thegeneration of the CFD simulation input data.
1.2 Normative Requirements for Car-Park Ventilation Design
The ventilation design of enclosed or semi-enclosed car-parks is generally governed by normativerequirements. In the UK, Australia, and New Zealand, satisfying the prescriptive requirements stipulated in the
Building Regulations is only one way of complying to the intent of the governing legislation. The regulatory
framework in these countries provides the opportunity for designers to propose an alternative solution that isverified by a robust industry-standard design methodology. CFD simulations are commonly used for this purpose
and are generally performed to verify that the accumulated CO concentration levels at different locations and forspecified durations are within the legal thresholds.
2. SHARING OF BUILDING INFORMATION
2.1 Industry Foundation Classes (IFC)
Since 2013, the ISO16739-1 Industry Foundation Classes (IFC) has progressively been accepted as thepreferred standard for exchanging BIM data among project stakeholders in the Architecture, Engineering,
Construction and Facilities Management (AEC/FM) domain. Given the progressive uptake of BIM in the domain,
more common building information is becoming readily available for sharing among project stakeholders.
The latest IFC4 schema has 767 entities such as IfcSite, IfcBuilding, IfcBuildingStorey, IfcSpace, IfcZone,
IfcOpeningElement, IfcDoor, etc. Each entity has a range of property sets such as Pset_SpaceCommon,
Pset_SpaceThermalRequirements, Pset_ZoneCommon, etc., as well as a range of quantity sets such as
Qto_SpaceBaseQuantities. The building geometry, common building elements and their properties including basic
quantities such as the floor area, perimeter length, floor to ceiling height, and volume of each space are inheritedin the building model being shared and reasonably straightforward to transfer as input to downstream applications.
Specifically, a car-parking space is exchanged in the latest IFC4 schema as a property set of its coreschema, namely Pset_SpaceParking, which can be identified as all occurrences of IfcSpace entities with an
attribute value for ObjectType set to “Parking”. The Pset_SpaceParking property set has several single value
properties (Table 1) that are useful to convey the design intent. For example, the IsAisle property can be used torepresent a scenario where a row of vehicles is releasing CO in the atmosphere while queuing to exit a car-park
enclosure or to access individual parking units.
Table 1. Single Value Properties of Pset_SpaceParking
Key
Value
ParkingUse
Not enumerated, but can include car, motorcycle, truck, etc.
ParkingUnits
Parking space for the transportation type, i.e. usually one per space
IsAisleTRUE if it is an aisle for accessing the parking unit
IsOneWay
TRUE if IsAisle is TRUE and designed for one-way traffic
The requirement for ventilation in a space is exchanged by the property set of an IfcSpace entity, namely
Pset_SpaceThermalRequriements. If a space is intended to be naturally ventilated, then the NaturalVentilation
property of Pset_SpaceThermalRequriements must be set to TRUE. If a mechanical ventilation is desired, then the
NaturalVentilation property must be set to FALSE. There are other ventilation design parameters that can be
specified in the IFC model such as NaturalVentilationRate or MechanicalVentilationRate, given in terms of ACH.
For the car-park ventilation design, the IfcFan and related entities and property or quantity sets can also
be used to convey the design intent and equipment specifications. IfcFanTypeEnum is an enumeration of different
types of fan such as CentrifugalForwardCurved, TubeAxial, etc. Furthermore, NominalAirFlowRate andNominalStaticPressure are two properties of Pset_FanTypeCommon that are important in specifying the fancharacteristics.
CO sensors are exchanged by the IfcSensor entity with IfcSensorTypeEnum set to “CO2SENSOR”. The
IfcSensorTypeEnum includes a long list of other enumerated sensor types such as HEATSENSOR,HUMIDITYSENSOR, TEMPERATURESENSOR, as well as USERDEFINED and NOTDEFINED. The
IfcSensor is also associated with a range of property sets such as Pset_SensorTypeCO2Sensor, which has one
single value property, namely SetPointConcentration. The value of CO concentration set point can be specified
by setting the IfcPropertyBoundedValue.SetPointValue in the model, although different BIM authoring tools may
have slightly different methods of achieving this.

260

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3. SHARING NORMATIVE INFORMATION
Normative requirements such as provisions in the building legislations, building regulations, building
codes, standards, by-laws, and judicial rulings, as well as contractual documents are conventionally written innatural language for human interpretation, which is not readily processable by machines. In recent years, there has
been an increasing interest in different domains to represent normative knowledge in a computable form for humanreadability and machine processability. Notably, LegalDocML and LegalRuleML are two emerging open
standards that can be used to represent the entire literal and logical content of normative documents, which canthen be used to automate some of the compliance audit processes (Dimyadi, Governatori, & Amor, 2017).
In regard to the performance-based car-park ventilation design, there is a set of parameters, which areoften specified by normative standards, that must be used as the basis for design. For example, the New Zealand
Building Code (NZBC) Acceptable Solutions G4/AS1 document sets the performance-based compliance criteria
as complying with relevant requirements of the Australian Standards AS1668.2 (Standards Australia, 2012). Someof these requirements include prescribed input parameters that must be taken into account for design purposes,
such as certain factors to use for determining the total effective number of vehicles and the maximum driving
distance in a worst-case scenario, as well as calculations on the required volume of air intake or exhaust, etc. For
example, the factor that shall be used to determine the effective number of vehicles in a worst-case scenario in aretail occupancy is 0.7 of the total capacity, which can be represented by a LegalRuleML rule (Figure 1).

Figure 1. A computable representation of an exemplar normative requirement
4. USE CASE
4.1 Exemplar Car-park ModelAn underground semi-enclosed car-park in a retail building in Auckland, New Zealand has been used as
the basis for the exemplar building model developed to illustrate the use case of the approach (see Figure 2). A jet
fan can be seen installed on the ceiling in Figure 2.

Figure 2. Photo showing the exemplar car-park building used as the basis for the use case261

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

4.2 BIM Data
A building model of the car-park has been developed using the Revit 2019 BIM authoring software
application based on the original design documentation and site observations. All essential ventilation systemcomponents have been incorporated into the model as closely as possible to reflect the actual installation. For the
purposes of the data exchange exercise, some assumptions have been made on the system operating characteristics,
such as the fan capacity, static pressure, sensor set points, etc. For example, jet fans and CO sensors and theiroperating characteristics have been incorporated into the model accordingly. Jet fans and CO sensors are part ofthe ventilation system to assist with the dispersal and ultimately removal of CO by pushing the contaminated air
along in the direction of the exhaust air opening. The car-park has two wall openings, which represent the entrance
and exit ramps on opposite walls. These have been incorporated into the building model to represent the outside
air intake and exhaust air openings to induce passive natural cross-flow across the enclosure.
To enable querying and information extraction, an IFC model has been obtained by exporting from Revit.
Five spaces within the enclosure designed to be parking aisles have been modelled as five separate unboundedrooms, each is exchanged by IFC as an IfcSpace object with its ObjectType set to “Parking” and exchanged with
the IsAisles property set to TRUE in the Pset_SpaceParking property set. The remaining space within the enclosure
has also been modelled as a room (IfcSpace entity) with the IsAisles property set to FALSE. Figure 3 is the 3Dview of the IFC model showing the five rectangular shaped unbounded rooms representing the parking aisles thatprovide access to individual parking units. Jet fans are exchanged as the IfcFan entity with its corresponding
properties, namely NominalAirFlowRate and NominalStaticPressure values set based on the assumed operating
characteristics. The CO sensor activation set point value has been extracted as described in Section 2.1 above as
being 20 ppm, which is an assumed value incorporated into the model.

Figure 3. The IFC model view showing five rectangular spaces specified as parking aisles
The Qto_SpaceBaseQuantities gives the GrossFloorArea of this car-park model as 4,957 m2. Thisquantity set also gives the NetFloorArea and GrossPerimeter values as 3,750 m2 (excluding spaces dedicated asparking aisles) and 290,432 mm, respectively. A tally of all IfcSpace entities with ObjectType = ‘Parking’ and the
IsAisles property set to “FALSE” gives a total parking unit count of 140, which represents the car-parking capacity
in the building. This information is required when modelling the worst-case scenario and for determining the likely
CO production rate (see Section 4.3).
4.3 Transferring of IFC model to Simulation Data
There are several methods to transfer the geometry of the building to a CFD simulation input for tools
such as the NIST’s Fire Dynamics Simulator (FDS) (McGrattan et al., 2015), as reported in the literature of asimilar work (Dimyadi, Solihin, & Amor, 2018). Figure 4 shows a view (using the SmokeView companion tool toFDS) of the FDS computational model that has been mapped from the IFC geometry via bounding boxes.

Figure 4. FDS Simulation model showing parking aisles represented by CO generator strips262

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

FDS has been selected as the CFD simulation tool for this exercise as it has been validated against several
published experimental studies on similar cases involving low-speed fluid flow containing species such as CO thatis not thermally-driven (Viegas, 2009). For the purposes of simulating the CO generation, the parking aisles havebeen transformed in the FDS model as rectangular strips representing surfaces that release CO in the air. Each
rectangular strip is intended to represent the exhaust of a row of vehicles queuing to exit the car-park with idlingengines as in a worst-case scenario described in Section 4.5.
4.4 Normative Design Parameters
The NZBC G4/AS1 compliance document calls upon AS1668.2 Standard as the compliance criteria to besatisfied when proposing a performance-based car-park ventilation design. This Standard specifies that any
ventilation system shall ensure that concentrations of atmospheric contaminants within the car-park enclosure do
not exceed the listed occupational or community exposure thresholds. As part of the compliant-design procedure,
the standard specifies a factor of 0.7 for determining the effective total number of vehicles in a retail occupancy.
Another design parameter is a factor of 0.5 for determining the maximum driving distance to exit the car-park
enclosure in the same retail occupancy. Furthermore, the applicable compliance criterion for this exemplar carpark is that the CO concentration in the enclosure must be maintained to a maximum average value of 60 ppm
everywhere between 900 mm and 2500 mm above the floor for one hour, or a peak value of 100 ppm. These
prescribed design parameters can all be represented as LegalRuleML rules such as that shown in Figure 4.

Figure 4. A rule representing the provision in the AS1668.2 specifying a prescribed design parameter
4.5 Modelling Worst Case Scenario and CO Production Rate
The CO emission from a vehicle varies depending on the type of fuel, the age of the vehicle, engine
temperatures and the environmental conditions. The ASHRAE Handbook “HVAC Applications” (ASHRAE,2015) provides a range of estimated vehicle CO emission based on EPA Mobile3 prediction for 8 km/h driving
speed in car-parking spaces (Table 2).
Table 2. Vehicle CO Emission (ASHRAE, 2015)
SeasonHot engine (g/min)
Cold engine (g/min)
Summer 32 degC
2.54
1.89
4.27
3.66
Winter, 0 degC
3.61
3.38
20.74
18.96
For simulation purposes, the average CO emission rates for hot and cold engines in the winter condition263

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019has been selected, i.e. 11.7 g/min. Given the car-parking capacity of 140 parking units, as obtained from the IFC
data (Section 4.2) and given the normative usage factor for a retail occupancy of 0.7 (rule in Figure 4), the effective
number of vehicles leaving the car-park per hour can be calculated as 0.7 x 140 = 98 vehicles per hour. Additionally,
the effective travel distance to exit the car-park is specified by AS1668.2 as being half of the car-park grossperimeter (see Section 4.4 above), which can now be calculated as 145 m, knowing the GrossPerimeter from IFC(Section 4.2). If the driving speed in the car-park is 6 km/h, which is a reasonable assumption, it would take eachvehicle 1.45 min on average to drive out of the car-park. Therefore, the total CO production rate per vehicle canbe calculated as 11.7 g/min x 1.45 min = 17 g per vehicle. Knowing that there are effectively 98 vehicles per hourin the enclosure, the cumulative CO production in the enclosure can be calculated as 1,666 g/h. Please note thatthe presence of other gas species in the atmosphere within the car-park enclosure has been assumed to be negligible.
In the FDS input data, each CO generator strip contributes 1/5 to the total CO released in the enclosure,i.e. 1,666/5 = 333 g/h. Each strip has been modelled in FDS as having a top surface of 10 m2 in area and positioned
at 300 mm above the floor, which is the average height of a vehicle exhaust pipe. The mass flux of CO released
from that surface can then be calculated as 333/10/3600 = 0.0093 g/s/m2, which is entered as an input in the FDS
simulation data.
4.6 Simulation Output and Result
The FDS simulation output shows a steady state condition being reached starting at about 1,500 seconds
with the average CO concentration anywhere in the region of the car-park between 0.9 m to 2.5 m above thefinished floor level remains below 30 ppm for the hour (Figure 5). This is well within the 60 ppm compliancethreshold. Additionally, the maximum recorded CO concentration level for one-hour simulation is 47.3 ppm, whichis also within the prescribed 100 ppm peak threshold. At one hour into the simulation, the CO seems to have beendispersed and maintained within a safe concentration level (Figure 6).

Figure 5. Average CO concentration for one hour within the 0.9 to 2.5 m region above the floor

Figure 6. CO dispersion and concentration levels at 3600 s and 1.8 m plane above the floor

264Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

5. CONCLUSIONS
An approach to sharing BIM and normative data to support the performance-based design of ventilationin enclosed car-parking spaces has been described with an exemplary use case based on a real car-park building.
For this exercise, the data exchange process is semi-automatic using data query and manual look-up. However, theentire approach can be automated in an executable workflow where each task would query different data sources
such as BIM and a rule set and then performed the corresponding calculations, which can then be used to generate
the input data for the CFD simulation. The same approach can be applied to other performance-based design that
relies on the readily available information from BIM and normative standards. The challenge would be to ensurethat the required information is available in the building model in the first place, which may not be guaranteed
depending on when, at which stage of the design process, and which model is being shared. Another challenge isto ensure that the computable representation of normative requirements is endorsed and maintained officially andserved from a central repository to guarantee that it is the latest official version.
REFERENCESASHRAE (2015). ASHRAE Handbook: HVAC Applications. SI Edition. American Society of Heating,
Refrigerating and Air-Conditioning Engineers. USA. ISBN-10: 1936504936.
Burnett, J., & Chan, M. Y. (1997). Criteria for Air Quality in Enclosed Car Parks. In Proceedings of the Institutionof Civil Engineers - Transport (pp. 102–110).
Chan, C. Y. (2001). A performance-based approach to car park ventilation design. In Proceedings of AIRAH 2001
National Conference (pp. 1–8).
Chan, M. Y., Burnett, J., & Chow, W. K. (1998). Energy use for ventilation systems in underground car parks.
Building and Environment, 33(5), 303–314. https://doi.org/10.1016/S0360-1323(97)00075-9
Chan, M. Y., & Chow, W. K. (2004). Car park ventilation system: performance evaluation. Building andEnvironment, 39(6), 635–643. https://doi.org/10.1016/j.buildenv.2003.10.009
Chow, W. K. (1995). On ventilation design for underground car parks. Tunnelling and Underground SpaceTechnology, 10(2), 225–245. https://doi.org/10.1016/0886-7798(95)00010-V
Demir, A. (2015). Investigation of Air Quality in the Underground and Aboveground Multi-Storey Car Parks in
Terms of Exhaust Emissions. Procedia - Social and Behavioral Sciences, 195(216), 2601–2611.
https://doi.org/10.1016/j.sbspro.2015.06.461
Dimyadi, J., Governatori, G., & Amor, R. (2017). Evaluating LegalDocML and LegalRuleML as a Standard for
Sharing Normative Information in the AEC/FM Domain. In Proceedings of the Joint Conference on
Computing in Construction (JC3) (pp. 637–644).
Dimyadi, J., Solihin, W., & Amor, R. (2018). Using IFC to Support Enclosure Fire Dynamics Simulation. In I.
Smith & B. Domer (Eds.), Advance Computing Strategies for Engineering - Lecture Notes in Computer
Science (Vol. 10864, pp. 339–360). Lausanne, Switzerland. https://doi.org/10.1007/978-3-319-91638-5_19
Gaekwad, J. (2017). Carbon Monoxide Dispersion in Enclosed Car Parks: Pollutant Source Modelling Methods.
In Proceedings of the 15th IBPSA Conference (pp. 654–661). San Francisco, USA.
https://doi.org/https://doi.org/10.26868/25222708.2017.168
Ho, J. C., Xue, H., & Tay, K. L. (2004). A field study on determination of carbon monoxide level and thermal
environment in an underground car park. Building and Environment, 39(1), 67–75.
https://doi.org/10.1016/j.buildenv.2003.07.006
Li, Y., & Xiang, R. (2013). Particulate pollution in an underground car park in Wuhan, China. Particuology, 11(1),
94–98. https://doi.org/10.1016/j.partic.2012.06.010
Lu, S., Wang, Y. H., Zhang, R. F., & Zhang, H. P. (2011). Numerical Study on Impulse Ventilation for Smoke
ControlinanUnderground
Car
Park.
ProcediaEngineering,11,
369–378.
https://doi.org/10.1016/j.proeng.2011.04.671
McGrattan, K., McDermott, R., Weinschenk, C., Overholt, K., Hostikka, S., & Floyd, J. (2015). Fire Dynamics
Simulator User’s Guide (FDS V6). National Institute of Standards and Technology. Retrieved fromhttp://dx.doi.org/10.6028/NIST.SP.1019
Špiljar, Ž., Drakulić, M., & Schneider, D. R. (2018). Analysis of Jet Fan Ventilation System installed in an
Underground Car Park with Partition Walls. Journal of Sustainable Development of Energy, Water andEnvironment Systems, 6(2), 228–239. https://doi.org/10.13044/j.sdewes.d5.0180
Standards Australia. (2012). AS1668.2-2012 The Use of Ventilation and airconditioning in buildings. Part 2:
Mechanical ventilation in buildings.
Viegas, J. C. (2009). The Use of Impulse Ventilation to Control Pollution in Underground Car Parks. International
Journal of Ventilation, 8(1), 57–74. https://doi.org/10.1080/14733315.2006.11683832

265Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019The Study of Automatic CIM Models Development for Infrastructure Projects
Tzu-Tin Huang1 and Yu-Cheng Lin2
1) Graduate Student, Department of Civil Engineering, National Taipei University of Technology, Taipei, Taiwan. Email:wchwch8899@gmail.com
2) Professor, Department of Civil Engineering, National Taipei University of Technology, Taipei, Taiwan. Email:yclinntut@gmail.com

Abstract: Building Information Modeling (BIM) has been widely used in civil engineering and constructionindustry. Its technology has effectively reduced many mistakes of this huge industry and reduced the cost of
construction projects. It has established industrial trends for countries all over the world. However, Civil
Information Modeling (CIM), is the latest concept in the construction engineering, its application in the is slowly
becoming more present in the construction industry. The field of civil engineering construction is relatively large,
and the work content in the design and planning stage is highly variable, especially when the model is being
constantly updated on the constrcution phase of the project. In this stage CIM engineer encounters many
complicated and cumbersome operations and integration of information problem are detected. If complicatedprojects are built with the limitations of the software itself, improving work efficiency will become a more difficult
task. Recently, visualization program design for the application of engineering has gradually become a trend.
Visual programming tools can assist the CIM model designs by providing improvement in the solutions ofproblems encountered in the original construction method. Therefore, this study uses the Autodesk Revit platform
combined with Dynamo as a research tool to write programs, import into case application, and study how these
two softwares simplify the repetitive work when designing projects. Finally, the research demonstrates theapplication of the proposed tool into a case of the North Road model guardrail for the design and development ofa model with support of automatic generation of elements. With the successful development and implementationof the automation program, this project was able effectively shorten designing and construction time even thought
the complexity of the project, reduce human errors and omission. Lastly, this paper will present the discussion of
the main benefits, difficulties and limitations by case study as a reference for follow-up research and application
will be introduced.
Keywords:

Civil Information Modeling, CIM, Automation, Visual Programming, Dynamo.

1. INTRODUCTION
Civil engineering is a discipline that deals with complex designing works, construction and maintenanceof the physically and naturally built environment, especially public sector works such as roads, bridges, dams,
highways, airports, pipelines, sewage and drainage systems, railways, etc. In the process of building a CIM model,
there are often many manual operations that are repetitive. Correctly, most of the CIM model are still being builtwith standard software functions. However, visual programming tools can be used to make CIM models designs
an easier task. Related users can use software like Dyanamor to create visual computer programs instead of text
programs by operating program components. This clear and intuitive programming approach make understanding
procedures easier for non-programmers or new programmers. Most of the programs that can be used to assist theCIM model are usually developed by the Application Program Interface (API). Nevertheless, for CIM-related
users who don’t have any background or lacks knowledge about writing programs, their API-assisted CIM
automation program will not meet all the requirements in a short time. Thus, users will have to identify whether
there is an auxiliary API plug-in on the network, or cooperate with the programming engineer to find the proposedsolution, and then manually repeat the operation to complete the construction.
To introduce CIM technology in any project, the first step will be the build and design of a CIM model.
Then during the process of building a CIM model, engineers may encounter many repetitive operations andinformation that needs to be integrated to understand the problem. In addition, the main purpose of this study isuses the Visual Programming Language (VPL) to carry out the auxiliary development tools for engineering CIM
model construction automation, solve the complicated manual operation of the current CIM model, improve thepossible omission of errors in the construction, accelerate the speed of model construction, improve originalfunctions in the CIM model, and make the CIM model construction engineer work more reliable to achieveeffectiveness and reduce errors. The purpose of this study is as follows:
1. To identify automatic CIM models development for infrastructure projects, assist CIM model build automatedoperations, improve the construction model encountered by the repeated work, and reduce the build of errors.
2. With the development of case introduction, explore the benefits, difficulties, and limitations of using visualprogramming to apply to CIM models.
2. LITERATURE REVIEWBuilding information modeling (BIM) is a set of interacting policies, processes and technologies

266

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

generating a “methodology to manage the essential building design and project data in digital format throughout
a building life-cycle” (Penttilä, 2006). BIM,which enables a facility to be digitally represented by object-basedmodeling, not only changes how a facility is created from traditional CAD solutions, but also remarkably alters
the key delivery processes involved in constructing a facility. Therefore, BIM is not only a technology change, butalso a process change. Civil information modeling (CIM) is a term commonly used in the AEC industry to referto the application of BIM for civil infrastructure facilities, such as highways, bridges, tunnels, water supply
channels, etc. (Cheng et al, 2016). CIM technology can use the previous data to backup the new project data, in
the project management, plan, design, build and facility management, so engineers can easily hit their target of the
project (Japan Construction Information Center, 2016). CIM technology is reference BIM technology, but there
are three different purposes between CIM technology and BIM technology. The first difference is that in BIM
most of the projects are building projects and these projects are always located in a fixed surface level. In the other
hand, CIM is subjected to gradual changes due to the terrain situation and due to its wide range of coverage. The
second difference is the professional vocabulary for both technologies are different. For example, in buildings
project the vertical structure to support the building are call column, but in civil projects vertical structure to
support the project are call pier. The third difference is the construction method, in building project the project isbuilt floor by floor, but in civil project is built pier by pier (Cheng et al, 2016).
Graphical programming language or VPL, main concept is based on graphical data flow (Vectorworks Inc,
2016). Engineers can compile and link all processes by creating flowcharts, describing their ideas and algorithms(Charntaweekhun & Wangsiripitak, 2006). The process of creating visual programs through step-by-step
operation, following by the logic of input, processing and output; rather than a series of imperative instructions,unlike the API (see Figure 1) (Mode Lab, 2015). The application tool of this research should include the graphicalinterface of visual program design language and auxiliary geometry modeling, which can control the visualization
program tool of the parameters within the CIM model through the programming language design in the nodes.
Related auxiliary CIM model of visual programming tools such as the following sets of software tools: (1)Marionette, (2) Grasshopper, (3) Generative Components, (4) Dynamo. Futthermore, Table 1 illustrates thedifference between the software and what else.

(a) Text program(b) Visual program

Figure 1. Difference between text and visual program languages

267

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 1. Comparison of visual programming software for auxiliary CIM models
Marionette

Generative Components

Grasshopper

DynamoSoftwarecompany

Vectorworks

Bentley

McNeelAutodesk

Designplatform

Vectorworks
Architect

MicroStation

Rhino

Revit

Independent
software

No

No

Yes

Yes

Free
software

No

Yes

Yes

Yes

Advantage

Visualization programwith advanced 2Dinterface and powerful
3D modelingcapabilities.

It can analyze and design
the structure of free-form
objects.

The calculation
The data structure isefficiency is good, and flexible, and the ability tothere are many readycall Revit data is an
made algorithms.
advantage.

Nodepresentation
mode

The current visual programming tools for the relevant auxiliary CIM software are widely used in the
Grasshopper plug-in developed by Sia McNeel, USA, primarily as a secondary parameter modeling tool for the
Rhino software platform. This study employ Autodesk's Dynamo as a research tool. The reason is Dynamo is an
open original code graphical programming language environment for CIM that enlarge the parametric informationof Autodesk Revit software elements. And the tool can be run independently of the environment. With Dynamo's
graphical programming language, it is possible to build complex geometric CIM models that were previously
unattainable in Revit software, automatimg the development by applying a large amount of information andspecifications to the CIM model and running.
The following consolidation-related studies, such as, Kensek (2014) through the Arduino board andDynamo tools, detecting lighting and verify the correct position of the shading device on the wall. In another case,it was proved that the Arduino and Dynamo or Revit API could be used to connect environmental sensors tobuilding information. Cheng et al. (2016) presents a framework for evaluating practices of various civil
infrastructure using the CIM. Feng and Lu (2017) using Dynamo development program, an automatic construction
system of construction frame model is established. Aaron et al. (2018) developed an application to collect andanalyze BIM transport infrastructure and critical literature review, and implement a variety of algorithms to help
automate the analysis and review. Marek Salamak et al. (2018) develop an automated program system via Dynamo
which corrects faulty analytical model obtained from BIM geometry, providing better automation for preparing
FEM model. Mojtaba Valinejadshoubi et al. (2019) through the Dynamo and SHM systems, developing aninspection program about effectively manages building construction. And check whether the component isdamaged or abnormal in the structure. Brandon Bortoluzzi et al. (2019) develop management and integration
program system through Dynamo, minimizing the two-dimensional plans and elevations to build 3D models andlarge amounts of data required for maintenance.
3. RESEARCH CONTENT
All construction components in the CIM model contain geometric, architectural and project information. In
the process of construction engineering design and construction, the digital information in the CIM model can be
managed automatically, and the entity parameter components can be corresponded to through object-oriented
information structure. However, when construct the CIM model, with the project size increases, if the construction
can not be automated, a single editing set and place the manual build mode, resulting in poor job efficiency quality.
Therefore, it is necessary to reduce the repeated construction work, reduce the complexity of operation, and assist
in modifying the design, can be automatically entered into the required parameter information, one click to
generate or quickly simulate the desired location.
In this study, a visual programming tool for the Dynamo, Dynamo can improve the company and related268

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

projects to build CIM workflow, this tool enables CIM information can be managed through the opening of theonline search node package within Dynamo, can download various function node-related demand, and then carryon programming, so that the original work of the CIM can be more flexible operation. This study presents visualprogramming tool for application development Dynamo aid of Revit software, can be divided into the following
seven applications are as follows: (1) Information import export; (2) analysis simulation to optimize the design;(3) foundation turning operation mode; (4) irregular geometry build; (5) encodes build various types of automation;(6) automated management information parameter; (7) the subject of nuclear design specifications. The purposeof this study is to improve the CIM model build process, complex repetitive operations jobs, it will apply for the
direction of automation, development. From automated program development, CIM parameter information will
be integrated with the design process, and thus be able to CIM information parameter automation and information
management remit, to quickly get the relevant parameters and automatically generate the number of components.
Therefore, this study proposes the application of visual programming in CIM model building process (seeFig. 2). When building engineering CIM model, if we encounter complex and repetitive operation problems, weneed to analyze the application requirements, evaluate whether we need to develop automation programs to assist
CIM model, and consider whether we can develop the required automation functions through visual programming,
if we can further enter the programming. Through the node repository, function nodes with different requirements
are established. Code Block node can realize the design script programming language, provide powerful visualprogramming function, and directly write Python script through Python Script node. The function of different
function nodes is used to establish the program flow, confirming the connection relationship between each node.
If there is any problem in the design flow, the node will be highlighted. The program design test should be repeatedto check whether the correct connection between nodes is confirmed due to the selection of non-conforming nodes
or code errors. In the process of execution, we can know whether the development program meets the requirementsof CIM model building automation operation. If the program does not meet the requirements and needs to bemodified, we can feedback the opinions to program planning and update it.

Start

Understand therequirement for
CIM models

Need Repeat
work?

YES

Need the content for
automation plan?

NO

YES

Confirmed theprogramming
content of models

PlanprogrammingNO

Develop theCIM models
manually

Setup
function
nodes

Develop theCIM models
manuallyNO

Setup
PythonscriptingNO
NO

Confirm theresultConfirm theresults

Need to setup
node connection

YES

YES

YES

Run visualprogramming
programEnd

Need the final
result?

YES

Integrate manuallymodels andprogramming
modelsModels combined andconflict analysis

YES

Confirm theresults

NO

NO

NO

Figure 2. Visual programming applied to CIM model construction workflow
4. CASE STUDYThis study applied the research model to practical engineering in Taiwan. This case is channel north road.
The problem is limited to the construction of the guardrail support. The problems for the construction of CIM
model includes built bridge panel, guardrails and other originals. After the building, one by one manually placedin the planned position, need to spend more time. For the manpower, this job is less able to clearly carry out thedivision of labor. With the help of automated program, the construction and planning time can be shortened quickly,
and the trouble of division of labor can be reduced. Therefore, this study uses the Dynamo program to write and269

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

automate the construction of the guardrail support.
Figure 4 shows the results of performing automated program construction. In this case, the program design
process is divided into four memory blocks. The first three memory blocks are the main design processes forautomatic generation of guardrail support. The fourth block is to adjust the rotation angle of the guardrail support
after the automatic construction of the program execution, so that it can be closely combined with the guardrail.
Figure 5 shows the program design process.

Figure 4. Program design process

Figure 5. The implementation of automated programs build the guardrail support
This case is applied to CIM model construction automation work through the proposed visualization
program design, and the automatic generation program is developed for the needs of the case. From this visualprogram design, you can understand the entire program operation process, and the general construction method
differences (see Table 2).

270

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 2. Method comparison of case execution
Construction process
Degree of impact
Spend time
Create bridge decks one by Depending on the size of the At least three hours.
Copy
one, and adjust the upperproject, as the project increases,
construction floors, then build thethe number of repeatedmethod guardrails, and then copy the operations increases, and thespacing and adjust the curve. construction process is prone toerrors.
Set the distance between the Although the time will not
About two and a half
increase as the size of thehours.
Array bridge deck and thebuilding guardrail, enter the number project, the number of arrays
to be arrayed, and adjust the is unknown, so you need tomodeselect curve to complete the calculate it yourself before
model.
you can enter it.
The bridge can be built
Because the program canEntering values andthrough the program and the directly obtain the planning executing programsreference line andtakes less than 3
Auto-build upper level can beseconds.
program automatically adjusted. Then automatically generate
only the distance between the components, it will not be
guardrails can be input, and affected by the scale.
the curve curvature can be
input to complete the model
automatically.
Methods

Other functions
No.

No.

1. After the programis executed, you can
get the number ofholders and the ID ofeach component.
2. For the automation
function, the position
can be modified
directly during thesubsequentdiscussion. It only
takes about 5 seconds
to enter theconditions. If you
need to include theprogram open time, it
only takes 16 seconds
to complete the setup.

Through automated development of this research program, making the relevant work can improve
efficiency and shorten time to build. The benefits for the development program are described below:
1.
Through the procedure, carrying out the automatic generation of guardrail support, can shorten the previous
time spent on construction of the model. Once the visual program is completed, it can solve the operation of
repeated construction operations and improve the efficiency of work.
2.
During the construction process, CIM engineer has any need to comply with the regulations, but it has
problems with the location of the site. After discussion and adjustment, it can directly move the guardrail
bearing without considering the base. Then, after confirming the modified position, through the automatedprocedure, select the reference line planning below the guardrail mount, it can get the component withparameter information directly.
3.
After the implementation of the program, can quickly produce the required number of build, it can also be
used as a reference site related projects.
However, after the actual operation, according to the results of the case study,
the difficulties andlimitations are as follows:1.
The position of the guardrail support is placed by input spacing. However, if the curvature of the ramp is toolarge or because of high-level relationship and not placed in the correct position, it will eventually cause afence bearing floating in the air or voids in the ramps. Therefore, the bending angle and high rise of theguardrail support should be calculated. And write an additional part of the node program, so that the guardrail
support are built in the correct position.
2.
During the development process, it may encounter unfamiliarity with the use of the node database, resulting
in spending a bigger amount of time on writing programs and ongoing testing programs, enabling thedevelopment program to meet the requirements of the case and to generate the required automation.
3.
Dynamo built-in function node library has a large amount of node information, it is difficult to understandit in a short period of time, the use and function of each node is not very clear through the process ofprogramming. So it still necessary reading through the relevant functions and books to fully understand all
the information, and pay attention to any relevant information on the official website to increase development
inspiration and application.

271

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 20194.

The development of this research visual program can apply to all versions of Revit, and the design
development environment is in Dynamo version 1.3.4. Therefore, the program design process needs to be in
the version 1.3.4 or higher, and the lower-level software version cannot execute the program developed inthis study.

5. CONCLUSIONS AND SUGGESTIONS
5.1 CONCLUSIONS
1.
By applying the visual program design proposed in this study on CIM model, the main purpose of buildinga CIM model automation is to enhance the repetitive model operation and evaluate requirements ofapplication. Therefore, it is necessary to fully understand the requierments of any project to then start the
automation program implantation and establish the automated workflow
2.
After the development and implementation of automation programs, efficiency can be improved in building
operations. When the user need to modify the CIM model, it can apply the existing program and re-enter therequired parameters. Then, it can automatically generate the required CIM model components, reducing thetime and cost of redesign.
3.
Using visual programming tools to achieve automated CIM model in construction through visual oriented
programs have improved CIM-related users acceptance of learning programs. The development of users withentry level on the designing field is way easier than text programming language, but to get a deeper
understanding the application experience, users and researches must constantly accumulate new knowledge
about functions and programming nodes by improvimg their logic and skills of graphics programming.
5.2 SUGGESTIONS
1.
The automatic generating program of guardrail support developed by this research institute is designed for
only one point element. However, the application of CIM for the relevant planning of the ramp, there arestill many elements can be built on automatically. Therefore, it is suggested that further automateddevelopment can be developed for CIM-relaed components above two point elements which can highlights
the benefits of automation simulation for CIM-based use.
2.
Application visual programming tools (such as Dynamo) not only develops automation functions on thebuild, but also develops for model checking in the future. Through the visual program design to develop the
automatic inspection model, improve the CIM model construct correctness, assist CIM related engineers tocarry out model inspection.
REFERENCES
Penttilä, H., (2006). The effects of information and communication technology on architectural profession,
ECPPM - eWork and eBusiness, Valencia, Spain, 13-15 September 2006. in: M Martinez and R Scherer(eds), Architecture, Engineering and Construction, pp. 615-622.
Japan Construction Information Center, (2016). The Key-Base Information Station for the Construction Industry
to pave way for the Future.
Cheng C. P., Qiqi Lu and Deng Y., (2016). Analytical review and evaluation of civil information modeling,
Automation in Construction, Vol 67, p.31-47.
Vectorworks Inc. VPL Introduction. Retrieved from http://www.vectorworks.net.2016-01.
K. Charntaweekhun and S. Wangsiripitak, "Visual Programming using Flowchart", International Conference on
Communications and Information Technologies, King Mongkut's Institute of Technology Ladkrabang,
Bangkok, 2006. pp. 1062-1065.
Mode Lab. (2015). The Dynamo Primer. Retrieved from http://dynamoprimer.com, 2015-12.
Dynamo Primer website, https://primer.dynamobim.org, 2019.

272

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

AN AUTOMATED BIM-INTEGRATED SYSTEM FOR CHANGE ORDER COST
IMPACT EVALUATION
Veerasak Likhitruangsilp1, Tantri N. Handayani2, Nobuyoshi Yabuki3, Photios G. Ioannou4
1) Ph.D., Assoc. Prof., Department of Civil Engineering, Faculty of Engineering, Chulalongkorn University, Bangkok,
Thailand. Email: Veerasak.L@chula.ac.th2) Ph.D., Department of Civil and Environmental Engineering, Faculty of Engineering, Universitas Gadjah Mada,
Yogyakarta, Indonesia. Email: tantri.n.h@ugm.ac.id3) Ph.D., Prof., Division of Sustainable Energy and Environmental Engineering, Osaka University, Osaka, Japan. Email:yabuki@see.eng.osaka-u.ac.jp
4) Ph.D., Prof., Department of Civil and Environmental Engineering, University of Michgian, Ann Arbor, USA. Email:photios@umich.edu

Abstract: Change orders are inevitable in construction projects and usually have a major impact on project costs.
Evaluating change order impacts by using conventional 2D drawings often leads to the inaccurate estimatedquantity of modified works, which subsequently yields unreliable impact costs. Building information modeling(BIM) can address such challenges because it facilitates automated quantity takeoff and cost estimating,integrating relevant information, and visualization. Yet, implementing BIM for this purpose requires awell-defined methodology and a well-organized information structure. This paper develops an automated systemfor evaluating the cost impact of a construction change order. The system analyzes the modified building
elements using the model checker and estimates direct and indirect costs of such change order (e.g., demolition
and relocation costs). Since the proposed system performs a thorough cost impact analysis with minimal
subjectivity, the results are accurate and reliable for all project participants. In addition, the system is equippedwith a recording system for tracking the dynamic changing of project costs throughout the construction phase.
The system is applied to an actual high-rise building project to illustrate its capability and practicality.
Keywords: Building information modeling (BIM), change order, variation order, automated cost evaluation,
cost impact, claim management
1. INTRODUCTION
A construction change order is a modification towards the building conditions as a result of the project
owner’s instruction to change. It is an important tool for correcting any errors in project design or specifications.
Despite its benefits, a change order can be a major cause of additional project costs (Love & Li, 2000). Sincecosts are one of the determinant factors of successful construction projects, evaluating altered costs due tochange orders is extremely crucial. During construction, both owner and contractor need to evaluate the cost
impact resulting from the construction activities performed per a change order. The common practice for the cost
impact evaluation is primarily based on 2D drawings (e.g., CAD), which provides limited visualization andinformation. As a result, the evaluation is usually time-consuming and inaccurate. In addition, without a clear
methodology, the results may trigger conflicts among the project stakeholders. Thus, it is necessary to introduce
a transparent and systematic methodology with less subjectivity for evaulating change order cost impacts.
Building information modeling (BIM) is a potential tool for establishing such platform. BIM provides
geometrical visualization and facilitates construction information management. The quantity takeoff by BIM
begins with identifying work conditions, which are related to building elements. An appropriate work item andits associated cost are assigned to the quantity results (Lee et al., 2015). This process can also be extended to costestimating and monitoring (Elbeltagi et al., 2014; Olatunji et al., 2010). It allows us to extend the automatedquantity takeoff to the cost recording system with a high level of accuracy because the visualization function ofBIM can strenghten the estimating process in terms of accuracy and speed (Shen & Issa, 2010).
This paper proposes a BIM-integrated system that can automatically assess altered costs due toconstruction change orders. We introduce a new approach of construction cost impact evaluation based on thebuilding condition assessment. The system encompasses a thorough cost impact analysis with minimal
subjectivity. It can quatify the variation of construction materials as well as assess both direct costs (e.g.,material and labor costs) and indirect costs (e.g., relocation and demolition costs). The dynamic changing of
construction costs can also be recorded.
2. SYSTEM ARCHITECTURE
The system presented in this paper is part of the Building Information Modeling (BIM)-Integrated
System for Evaluating the Impact of Change Orders (BIM-ISICO) (Handayani, 2018; Likhitruangsilp et al.,2018). The BIM-ISICO is a comprehensive system that can evaluate the three main impacts of a constructionchange order: (1) altered physical conditions of the building, (2) time impact, and (3) cost impact. This paper
focuses only on the cost impact evaluation.

273

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The three unique features of the proposed system are:1) The model checker module, which can identify the altered building elements due to a change order
2) The cost evaluation module, which can assess not only direct costs but also indirect costs (e.g.,
relocation and demolition cost)3) Thorough analysis with minimal subjectivity
These main features are integrated into a system architecture, which elaborates the detailed methodologyof the system, as depicted in Figure 1. As can be seen, the system consists of three modules: (1) the datapreparation module, (2) the model checker module, and (3) the cost evaluation module.
This system is created using several BIM-based software such as Autodesk Revit, Microsoft Excel,
Visual Basic for Applications (VBA) and Dynamo. Each of the software plays an important role in data andinformation transfer or cost impact analysis.
2.1

Data Preparation Module
The data preparation module administers data collection and preparation. The necessary data are the 5D
BIM model of the project, construction progress, and project costs. The 5D BIM model refers to the 3D BIM
model, in which every building element is integrated with project cost information, namely, cost ID and unit cost(e.g., labor cost and material cost). Another input data is construction progress prior to the issuance of a changeorder for every element. The virtual building model must be updated to reflect the latest conditions of thebuilding in accordance with all prior change order issuances. Consequently, the most updated 5D BIM model isobtained and called the basic BIM model.
The owner’s instruction of a construction change is gathered in this step. Based on the instruction, thebasic BIM model is modified to reflect the owner’s desire. The result of the modification is a new BIM model
called the modified BIM model.
2.2 Model Checker Module
The objective of the module checker module is to assess the building conditions and identify themodified elements (Likhitruangsilp et al., 2018). It also classifies the modified elements based on the change
categories defined by the system.
The module checker module first creates a new BIM model called the merged BIM model, which is
obtained by inserting the basic BIM model into the modified BIM model. This linking process does notcompromise the information contained in both models because each of the BIM models contains independent
information.
The analysis of the model checker module is performed by using two system applications: Dynamo andVisual Basic for Applications (VBA). Dynamo is designed to transfer relevant information from the basic BIM
model and the modified BIM model to the spreadsheet. In the spreadsheet, VBA compares the geometricalparameters (e.g., quantity, location, and material) associated with the elements of the basic BIM models to thoseof the modified BIM models. There are five possible results of the comparison:
1) Addition refers to the scenario where a new element ID exists in the modified BIM models, butdoes not exist in the basic BIM model.
2) Deletion refers to the scenario where an element ID in the basic BIM model does not longer
exist in the modified BIM model.
3) Relocation refers to the scenario where an element ID found in both types of BIM modelentails different location information.
4) Quantity modification refers to the scenario where the same element ID in both types of BIM
model entails different values of quantity parameters.
5) Material alteration refers to the scenario where the same element ID entails different material
parameters in both models.
Every modified element detected in this module is presented with its cost information. For example, as
an element is categorized as “quantity modification,” this module provides the identity (i.e., element ID, element
name, and level), quantity before and after the change, material before and after the change, cost ID, and unit
cost.
2.3

Cost Evaluation ModuleThis module is the core of the analysis. The cost impact is evaluated by interpreting the results of themodel checker module. The cost and progress information of the modified elements are used to calculate thealtered direct cost and the indirect costs such as demolition and relocation costs. An important assumption is thatany works that have been done according to the basic BIM model must be reimbursed. Other rules for cost
impact evaulation are as follows:
1) The element classified as “addition” increases the quantity of the associated cost item regardless ofthe value of the progress. If the element is not included in the database, the analyst must input a new274

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

cost item.
2) The element classified as “deletion” reduces the quantity of the associated cost item if the progressparameter is zero. In contrast, if the progress is greater than zero (i.e., the works have been partiallyor completely finished), the identified quantity shows the amount of demolition works.
3) The element classfied as “relocation” is considered a relocation work if the progress is greater thanzero.
4) The element classified as “quantity modification” may increase and decrease the associated cost
item if the progress is zero. If the progress value is otherwise, the demolition cost is considered.
5) The element classified as “material alteration” can add or subtract the associated cost item. Thisdepends on the availability of the new material in the database. The assignment of the new element
may change the element into a different cost item. If the new cost item is not listed in the database,the analyst must input it. If the progress is greater than zero, demolition works are required.
As the altered work is quantified, the unit cost is derived from the analyst's assumptions. The project
cost database before the change is named as the basic project cost. Thus, the modification of project cost leads to
a new project cost database, the modified project cost.

Figure 1. System architecture of the proposed system275

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3. APPLICATION EXAMPLE
The proposed system is applied to an actual building project to illustrate its capacity and practicality.
The project is an 18-storey academic building, which hourses laboratories, classrooms, seminar halls, and sports
facilities. The project owner, Chulalongkorn University, instructed the contractor to adjust a pipeline shaft in
restrooms. Since the shaft connects a pipe of every floor, this modification was performed identically for eachfloor.
Figure 2(a) shows the building conditions before the change was issued. This is the basic BIM model, in
which all information, including progress and building conditions, has been updated. The change orderinstruction was applied to the basic BIM model, which results in the modified BIM model, as shown in Figure2(b). Lastly, Figure 2(c) shows the merged BIM model, which contains the information from the basic andmodified BIM models.
The implementation of the model checker module yields a list of the modified elements, as shown in
Figure 3. For this change order, the module detects almost 200 elements, which include walls, floors, ceilings,
and sanitary wears. Each of the modified element is characterized by its identity, change type, compared quantity,
compared material, and cost information. Herein, we focus on the material cost and the labor cost, which
establish the direct cost impact.
The first modified element displayed in Figure 3 is the wall with the element ID of 348487, which iscategorized as “quantity modification” (from 8 to 14.9 m2). This element belongs to the cost ID of C.01.01,
which refers to the half piece of brickwork cost item. The increasing quantity of this element triggers theincreasing of the associated cost item. This element also contributes to the increasing quantity of 103 m2 for thecost item of the half piece of brickwork, as shown in Figure 3. This result implies additional construction works(i.e., demolition and relocation), which need to be performed on the building element that has already been
completed. Any equipment or plant demobilization must be postponed to support such additional works.
It should be noted that the quantity modification is applied to not only wall elements but also ceilings
and floors. The F10 element located on the 1M floor is categorized as “quantity modification.” The quantity
decreases from 14.5 to 13.5 m2. This element contributes to the reduction of the cost item of coarse-surface
ceramic for a total of 16 m2. The similar calculation is applied to the ceiling cost item.

(a) basic BIM model(b) modified BIM model(c) merged BIM model
Figure 2. Development of the three types of BIM model

276

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

As shown in Figure 4(a), the direct cost is reduced by 13,380 Thai Baht (THB). This results from thedecreased quantity of wall tiles, the floor tiles, and the ceiling. The brick wall item is increased though.
The element ID of 348533, P1-100mm wall, is detected as “deletion.” Since the progress of the work is
100% (i.e., complete), the demolition work is required. The quantity of the element (1.5 m2) shows the numberof demolition work. As shown in
Figure 3, this element belongs to C.01.01, which is the half piece of the brickwork. Thus, the demolition
work is recorded to this particular cost item in Figure 4(b).

Figure 3. Results of the model checker module

Direct cost
Cost ID
C.01
C.01.01
C.01.04
C.02
C.02.09
C.04
C.04.02

Home

Collapse

ExpandBasicquantity

Item
Walls
Half piece brickwork
Ceramic tile wall size 600x600
Tiles
Coarse‐surface ceramic tile floor size 600x600
Ceiling and roof
Gypsum board type damp proof frame ‐ bar 0
Grand total

Δ Quantity

Modified
quantity

Unit

Unit cost (THB)
Material cost
Labor cost

Basic total cost
(THB)

Modified total
cost (THB)

11883
3062

103
(51)

11,986 Sq.m.
3,011 Sq.m.

400
700

140
150

6,416,897
2,602,547

6,472,247.09
2,559,196.66

1985

(18)

1,967 Sq.m.

700

150

1,686,826

1,671,526.11

879

(16)

863 Sq.m.

500

130

553,820
137,267,137

543,740.40
137,253,757(13,380)

Collapse

Expand(a) direct cost calculation

Indirect cost
Home
Cost ID
C.01.01
C.01.04
C.04.02

Quantity (sq.m or series)
Price (THB)
Demolition Relocation
Demolition Relocation
Half piece brickwork
4.5
0
1,000
Ceramic tile wall size 600x600
3
12.4
1,000
2,000
Gypsum board type damp proof frame
2
0
1,000
Cost item

Grand total

Total price
(THB)
4,500
27,8002,000

34,300(b) indirect cost calculation
Figure 4. Results of the cost evaluation module277

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The similar evaluation is applied to any modified element classified as “relocation.” For example,
P5-10mm wall with the element ID of 354902 is identified as “relocation” as the progress shows that the work
has already been finished. Thus, the relocation work is required. The quantity is derived from
Figure 3 whereas the unit cost is provided by the analyst. The relocation work means that the contractor
has to demolish the current walls and rebuild a new one with the same specifications at another location. The
relocation cost is recorded in Figure 4(b) showing that the contractor needs the indirect cost of 34,300 THB,
which consists of the demolition of walls, ceramic walls, and ceiling.
4. DISCUSSION
The 5D BIM concept where the virtual building model is embedded with cost information greatly
benefits change order management. In the conventional practice, 2D drawings and cost databases are usuallysegregated and are not linked to each other. The 5D BIM concept interrelates both sources of information, whichaccommodates the tracking of cost impacts, regardless of the number of change orders issued during
construction. In addition, it helps the analyst calculate the quantity and expense of altered works and extra
activities such as demolition and relocation. These attribute to a comprehesive and reliable project cost database.
By integrating the 5D BIM concept and the progress information, this paper proposes a new method for
estimating the demolition and relocation costs. In practice, the quantities of these works are usually derived fromthe estimator’s assumptions or forecast. An automated and systematic calculation of these extra costs yields
accuate and reliable results.
Some previous research works used the model checker concept for assessing the modified building
conditions due to change orders [e.g., Moayeri et al. (2015)]. Yet, our system further extends the model checker
to evaluate the cost impact of change orders. Since our analysis is associated with minimal subjectivity, the resultof the model checker module can be referenced by the project stakeholders while negotiating project costadjustments. This can avoid conflict and disagreement among all parties.
5. CONCLUSION
This paper presents a new BIM-integrated system, which is equipped with a comprehensive
methodology and supporting tools for evaluating the cost impacts of construction change orders. The system
consists of three interrelated modules, namely, the data preparation module, the model checker module, and thecost evaluation module. The system is applied to an 18-story building to illustrate and verify its efficacy andpracticality. It yields direct costs and the cost of additional works such as demolition and relocation for
implementing each change order. The system can also monitor the dynamic changing of project costs due tochange orders throughout project life cycle.
REFERENCES
Elbeltagi, E., Hosny, O., Dawood, M., and Elhakeem, A. (2014). BIM-based cost estimation/monitoring forbuilding construction, International Journal of Engineering Research and Applications, 4(7), 56-66.
Handayani, T. N. (2018). A BIM-based system for evaluating impacts of change orders in construction projects,
Ph.D. thesis, Chulalongkorn University, Bangkok, Thailand.
Lee, H. W., Oh, H., Kim, Y., and Choi, K. (2015). Quantitative analysis of warnings in building information
modeling (BIM), Automation in Construction, 51, 23-31.
Likhitruangsilp, V., Handayani, T. N., Ioannou, P. G., and Yabuki, N. (2018). A BIM-enabled system forevaluating impacts of construction change orders, Proceedings of Construction Research Congress
2018, Louisiana, USA.
Likhitruangsilp, V., Handayani, T. N., and Yabuki, N. (2018). A BIM-enabled change detection system forassessing impacts of construction change orders, Proceedings of the 17th International Conference on
Computing in Civil and Building Engineering, Tempere, Finland.
Love, P. E., and Li, H. (2000). Quantifying the causes and costs of rework in construction, Construction
Management & Economics, 18(4), 479-490.
Moayeri, V., Moselhi, O., and Zhu, Z. (2015). Design change management using a BIM-based visualization
model, Proceedings of the 5th International/11th Construction Specialty Conference, Vancouver,
British Columbia.
Olatunji, O., Sher, W., and Ogunsemi, D. (2010). The impact of building information modelling on construction
cost estimation, Proceeding of the W055-Special Track 18th CIB World Building Congress 2010,
Salford, United Kingdom.
Shen, Z., and Issa, R. R. (2010). Quantitative evaluation of the BIM-assisted construction detailed cost estimates,
Journal of Information Technology in Construction (ITcon), 15, 234-257.

278

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

SIMULATION AND OPTIMIZATION OF UTILITY TUNNELS CONSTRUCTION
AS LINEAR PROJECTS
Mohamed A. Sherif 1, Abdelhamid Abdallah 2, and Khaled Nasser 3
1) MSc Candidate, Department of Construction Engineering, The American University in Cairo, Fifth Settlement, Egypt.
Email: mohamed_ayman_sherif@aucegypt.edu
2) Teaching Assistant, Department of Architectural Engineering, Helwan University, Cairo, Egypt.
Email: abdelhamid.abdullah@aucegypt.edu
3) Ph.D., Assoc. Prof., Department of Construction Engineering, The American University in Cairo, Fifth Settlement, Egypt.
Email: knassar@aucegypt.edu

Abstract: The utility tunnels are major features in several projects on different scales ranges from different sizesof campuses like university and hospital campuses to large scale utility tunnels used in mega residential projects
and airports. They are usually the housing space for different utilities required for the projects that are on multibuilding scale. The simulation of the construction of utility tunneling hasn’t received enough attention in research
while tunneling in general has been studied significantly. The scheduling and planning of utility tunnels as an
example of linear projects are facing big challenges of optimizing the project duration, work balance and utilizationof the available resources. Linear Projects consist of several similar elements with repeated activities from oneelement to the other. This paper will propose a discrete event simulation model to plan the construction sequenceof utility tunnels with optimum duration and resources. It also represents a case study as an example for a utility
tunnel construction in Egypt.
Keywords: Utility, tunnels, Scheduling, Planning, Linear Projects, Simulation, Modeling, Optimization,
Construction, Management.
1. INTRODUCTION
The definition of the term “Tunneling” is used to define activities related to the underground excavation
operations (AbouRizk, Ruwanpura, Er, & Fernando, 1999) aiming to achieve certain functions based on different
types of such operations. Service tunnel or a “Utility tunnel” is an underground tunnel smaller in width andelevation from the Normal Vehicles Crossing Tunnels with the functions of water distribution, sustainability ofthe environment, limiting the urban expansion, distribution of sewerage systems, increasing green spaces, reducing
energy use and reducing emissions and noise levels productions. All these functions are associated with many
challenges with the most significant challenge of Space Requirement which is why the use of underground Utility
tunnel is an alternative way to solve the problems of urban utility placement since it saves a lot of upper surface
space for better use. Utility tunnels have lots of features in comparison to the normal systems used, some examplesof these features are the Capability of Locating, Maintaining and Repairing utility systems rapidly, the Capabilityof avoiding the disruption of the environment of the upper surface and protecting the community from emergencies
regarding the infrastructure systems as much as possible (Ex. Outrages), the reduction of the future Maintenanceand repairing Costs since the permeant Easy accessibility that the service tunnels can provide, the Easily
Cooperation between the systems provided since they are all exposed and accessible in one place (No Need for
digging and Excavation on the long term) and finally Service tunnels can help terminate the need for Manholes inroads (Broere, 2016). Service Tunnels can have many possible sizes based on the required function, it can be large
enough to fit the Utility Systems required or it can be very large to accommodate Humans or even utility transfer
Vehicles (Balasubramanian, 2014).
The methods of construction of the utility tunnels can be divided into two methods which are Precast
Concrete (Prefabricated Concrete) where large ready blocks are prepared, poured off-site and transferred to thesite location to be connected together, the advantages of this method is reduction of the amount of reinforcement
used in the concrete and the reduction of the cracks and repair of the concrete. The Building scale samples of thismethod are railway tunnels, water transport tunnel, gas pipeline tunnels and service tunnels (De la Fuente, 2013).
but this method disadvantages are the difficult transportation method of the prefabricated items, the uneconomic
suitability of the use of this method in small projects, the need for the availability of a complete well-defined scopeof work and large capital cost (Priya & Neamitha, 2018). for the previous reasons the Cast on site concrete (Onsitu Concrete) is used in common projects which also have a very wide scale ranges from small linear projects due
to its acceptable cost to very large linear projects due to its high load capacity relative to its cost.
The Utility tunnels are an example of Linear Projects which are projects that consists of several similarelements where the activities included in each element are repeated from one element to the other. This repetition
can be either due to the uniform repetition of activities through the project (for example multiple similar houses
and high-rise buildings consists of typical floors) or due to geometrical layout of the project (for example highway
and Utility tunnels) (F. A. E.-M. Agrama, 2011).

279

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Scheduling and resourcing of such linear projects represent a major challenge during the planning phasebecause the schedules of such project needs to guarantee the optimum smooth flow of resources while obeying thestructure elements’ dependency, achieving the minimum planned duration to complete the project, accounting for
all the possible interruptions and risks that might face the project and determine the optimum number of resources
required which gives the maximum utilization of such resources (F. A. Agrama, 2012). In order to achieve suchrequirements, an optimization process needs to be prepared that reflects all the feasible alternatives and reach a
final optimum decision.
2. METHOD
2.1 Planning of linear projects “Utility tunnels”
The traditional way for the optimization process of resources and duration is using the Microsoft Excel
spreadsheets with defining the logical difference between continuous repeated activities then representing theprocess and results on the Line of Balance (LOB) graph. This method could reserve the resources allocation andthe logical dependencies between the construction elements, determining the start and finish time for each activity
but without the discussion of the optimization of the work balance (F. A. E.-M. Agrama, 2011). The Line of
Balance is a Resource driven technique which means that the resources are the governing factor forming the Lineof Balance to reach a full utilization of these resources in linear projects. However, several characteristics of theLine of Balance don’t match those of linear projects schedules models (Su & Lucko, 2015). The Genetic
Algorithms optimization models are widely used in Linear repetitive projects with the objective of minimizing the
project duration, Cost or both in a Multi Objective optimization model which has shown promising results(Mathew, Paul, Dileeplal, & Mathew, 2016).
2.2

Discrete Event Simulation
The Discrete Event Simulation technique is defined as a way of modeling the operations of a system as adiscrete (Independent) sequence of events happening over a certain period. Each event happens at a certain instantin time and marks a variation of state in the system. Between each consecutive event, no change in the system isassumed in order to reach leaping criteria of the simulation from one event to the next one (Sharma, 2015).
Different from the commonly used scheduling tools, the discrete event simulation can be used for the planning of
linear projects while taking into consideration the effect of variability in the construction process; well consider
the effect of changes in the durations of activities along the project construction chain which can lead to
optimization of the project total duration and better allocation of the available resources (Forcael, González,Muñoz, Ramis, & Rodriguez, 2018). Also, the discrete event simulation can count for the dynamic nature of theconstruction project’s activities durations and resources; which correct the assumption of fixed durations andavailable resources for a project along the total project duration in order to reflect the real nature of the construction
projects (Pinha & Ahluwalia, 2016)
2.3

The proposed model
The aim of this model is to find a new approach to manage and plan the liner projects. this new approachis prepared using discrete event simulation as a tool to plan the construction time schedule, number of resources,
compare different numbers of resources or any other results that could affect the construction management of thelinear projects. Accordingly, this model is developed as a conceptual model depending on the Quantity Surveying
data from the Building Information Model (BIM) of the project or these data can be prepared manually by
surveying on a Microsoft Excel spreadsheet. These data are prepared, integrated in the Discrete Event Simulation
as the inputs to the model, then the model examine the dependency relations between the different elements of the
projects which means in order to start working in a certain construction element (for example the Reinforced
Concrete foundation) what needs to be prepared or finished before the start of this successor task (for example thePlain Concrete foundation and the insulation). The model also examines the resources available for eachconstruction process and assign each required resource to the activities then consider when to release such resourcefrom the assigned activity to start work on the next task. Finally, the results are computed from the model in theform of how many resources are needed for each construction process and for the overall project, the percentagesof utilization of each resources and how many parts the project should be divided into in order for the best use of
the available resources and space (Work Balance) as Illustrated in figure 1.

Figure 1: The Main Logic Behind the Discrete Event Simulation Conducted280

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The Components of Cast in Site Utility Tunnels can by classified either with respect to the general layout where
the utility tunnels first consists of an Entrance and an Exit; they can be Ramps so that the Utility transfer Vehicles
can walk on, second comes the Channels of the Tunnel which are typically a rectangular section; Most of theCables and lines run through these Channels, third is the Stations of the Tunnels which are bigger than the Channels
in width and Elevations and may have an extra floor; these Stations are used to accommodate Equipment andmachines of the Tunnel if needed, forth is Control Rooms where the Electrical control panels of the tunnel areconnected in certain places, finally, there may be some Command Towers to monitor the tunnel. The developmentof this model is based on the Sequential process of the Construction Elements where the utility tunnels consistof: Plain Concrete Foundation then the Reinforcement Concrete Foundation which carry the Retaining Concrete
walls that are capable of tolerating the lateral load from the backfilling around the tunnel, the Concrete Columns
that exist in the stations part of the tunnel to accommodate for the extra elevations required and finally the roofslabs as illustrated in figure 2.

Figure 2: The Sequential Construction Elements of the Utility tunnels

The final model developed for the discrete event Simulation logic can be broken down into Five stages which aredescribed as the following. First, the Model Inputs which are the Quantity Surveying data table that can be preparedon a Microsoft Excel Spreadsheet or by using a Building Information Model Software (BIM) for more accurate
and detailed data. Second, the Construction Elements that describes the components, dependencies between thembased on the data from the Quantity Surveying and the Construction rates defined by the user. Third, the Holdfactor that controls the flow of elements in order to maintain the dependency relations, in other words, each element
wouldn’t start unless the required number of elements holding it has finished. Fourth, the construction process or
the delay duration for each element for example the duration of fixing the steel reinforcement or for pouring theconcrete, these delays depends directly on the quantities from the spreadsheet and on the resources available. Fifth,
the deliverables of the model in term of the optimum number of division parts (Work Balance) and the optimum
number of resources required for best utilization of such resources. All the Five Stages are illustrated in Figure 3.

281

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 3: Graphical representation of the Five Stages of the model logic

2.4

The proposed model
For the proposed model in this paper Anylogic software was used as a discrete event simulation software.
The quantity surveying data was extracted from the Autodesk Revit software to the excel sheet then to the model
itself on Anylogic. Any logic software was able to provide a very similar simulation to the real-life construction
project since it depends on nodes that can be used as a delay node (each construction process) and between thesedelays are lines describing the conditions and the relations between these delays. The concept of describing eachdelay as a separate node is very suitable for the discrete event simulation logic. Figure 4 represents the sequenceof construction of the model’s elements; The first construction element of the detailed model is the plain concreteelement which is required for the start of the second construction element which is the reinforced concrete. The
start of a certain length of the reinforced concrete foundation depends on the finish of a certain length of the plain
concrete foundation, this number of lengths is the required optimization result (work balance) which can lead tothe best utilization criteria for resources. The same concept is applied to the third element which is the concrete
walls then the columns and finally the roofs. This condition that describes the start point of each element isFigure 4: Resources and Model development282

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

controlled by the hold nodes which needs the queueing or the waiting nodes for the elements, these holding nodes
were coded effectively to simulate these start points of different parts. There are seize and release nodes responsible
for the assignment of the required resources for each construction process or delay. Finally comes the sink nodes
that defines the finishing points of the elements and that the successor construction element process can start likethe case of plain and reinforced concrete foundation.
2.5

The model parameters
The Main Parameters of the model can be divided into User Inputs which can be the delay rates for eachprocess, then the Resources which can be the amount of formwork and the number of pumps needed and finally
the Crews required that can be the number of Carpenters and Steel Fixers as illustrated in the following Figure 5.

Figure 5: Example of the model Parameters

3. CASE STUDY
3.1 Case description
The Proposed Case study is the Construction of a Utility tunnel in Egypt. This project was done in the New
Capital of Egypt by Orascom Construction Industry as the main Contractor in the International Shooting Club
Project. The Tunnel is divided into 23 Zones of 10 Stations, 10 Channels and 2 ramps as shown in Figure 6. The
Overall length of the tunnel is 720 m with average of 30 m for each zone.

Figure 6: Utility Tunnel Construction "New capital of Egypt"

The Scheduling and resourcing of such project was done based only on duration that the tunnel should be finishedin without consideration of the available resources and without doing any work balance for such project whichmeans that the work progress will depend only on the accessibility of zone and resources that can be provided.
The Schedule of the Project is done on Primavera P6 Software where the tunnel is divided into 5 Phases (4 Phasesof 4 Zones and 1 Phase of 3 Zones). This approach led to a huge loss in resources and time. The proposed modelis used as an efficient alternative way to that approach to come up with the optimum work balance and required
resources. The quantity surveying data were extracted from the BIM model which was prepared on the Autodesk
Revit then represented on a Microsoft Excel spreadsheet in a way to serve the model requirement from the elements
order. The User can enter the Number of parts for the division of the tunnel manually as well as the number ofresources and crews as initial values for the model as shown in Figure 7 to get the results of the model run which
are the utilization percentage and Overall Duration.

283

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 7: The User Initial Inputs

The main part of the model is its Functions and Parameters which controls the retrieving process of data from the
Excel Spreadsheets, the dependency relations between the model elements which is described by the Hold Nodes,
calculating the overall duration of the project, Assigning the required resources at the right time to the construction
process and computing the utilization rates of these resources for the purpose of optimization. All this process is
illustrated in Figure 8.

Figure 8: Illustration of the model functions and parameters

3.2

Running the model results

Figure 9: Utilization results of Crews and Overall Duration284

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The First Level of the Model is Normally Running the model after Entering the Data to the model, thesimulation runs for all the project’s elements retrieving each element one after the other till the end of the elements
then the simulation stops. The Results of running the model are the utilization percentage and number of eachFormwork type and each Crew type and finally the Overall duration as presented in Figure 9. These results are
based on the input parameters previously entered.
3.3

Comparing results
The Second Level of the model is Comparing the results system where the user fixes all the variable andincrement one required variable by a certain amount for several runs in order to compute its effect on the overall
duration. By this concept, Increasing the number of parts (Span Division) while fixing the variables of the resources
available will decrease the overall duration of the project to a certain limit until the number of parts will not be thegoverning factor of the flow of activities but the amount of available resources will be the governing factor as
shown in Figure 10. The same concept can oppositely be applied by Fixing the variable of the Number of Parts
and Incrementing the Available Number of Crews Available to see its effect on the overall duration until it stops
being the governing factor of the activities flow. The Same can be Done for the Number of Formworks Variable.
This level of the model can provide the user with an accurate estimate of the required resources and work balance
as a base for the construction process with the study of each parameter effect on the overall duration of the project.

Figure 10: Comparing the Number of Parts Results

3.4

Model Optimization
The third and highest level of the model is the Optimization process; the optimization of each of theprevious resources can be done by Optimizing the Number of Span Parts first then using the results in the
Optimization of the number of Crews available and using the results of both to Optimize the number of Formworks
available then Considering all this as a first iteration then repeating the Iterations again by using the results to get
the Optimum number of Span parts again for a certain number for iterations while considering that increasing thenumber of iteration will provide a more accurate results of the model. Figure 11 illustrates Optimization of thenumber of Span Parts by running the Optimization Model while fixing all other factors (Work Balance).

Figure 11: Optimization Work balance by fixing other factors

285Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Then optimization of the number of Crews available is done by fixing the number of Span Parts factor resulting
from the Previous Optimization process and optimizing the Number of Crews available. The next step is the
optimization of the number of Formworks Available by running the Optimization Model while fixing the numberof Span Parts and the number of Crews available factors resulting from the Previous Optimization processes andoptimizing the Number of Formworks available. The previous steps are repeated for a certain number of iterations
while considering the results of the previous iterations as a base for the next iteration until the results are flattened
and these results are the required optimum solutions.
4. CONCLUSIONThe Scheduling and Planning of the linear projects are considered of big construction challenges due tothe need for the optimum resourcing and transacting from one element to the next while obeying the structureelements’ dependency on each other and the need for minimizing the overall duration for such projects. Suchchallenges can be solved by using the discrete event simulation as an approach for providing the optimum workbalance and best utilization of resources which will all lead to minimizing the duration of the project. The proposedmodel can be used for determining the duration of a certain amount of resources, comparing results of different
resources situation to compute the effect of each resource on the overall duration and for Optimization of thenumber of resources, work balance and overall Duration. In Addition, this model can be used to get a detailed timeschedule for the activities of a construction processes. Also, this model represents a good representation of theelement’s dependency relations.
5. RECOMMENDATIONThe use of the simulation models has a lot of valuable incomes in the decision support systems, utilizationof resources and leveling and optimization of the project duration. The simulation models are easy to use systems
as they are computational based systems that can be developed without difficult mathematical approach and canbe modified to suite different situation that can be faced. Also, the automation of such systems can guarantee afaster result when required. That is why this paper recommend more work in the development process of such
simulation models especially those related to the construction process.
REFERENCES
AbouRizk, S. M., Ruwanpura, J. Y., Er, K. C., & Fernando, I. (1999, 5-8 Dec. 1999). Special purpose simulation
template for utility tunnel construction. Paper presented at the WSC'99. 1999 Winter Simulation
Conference Proceedings. 'Simulation - A Bridge to the Future' (Cat. No.99CH37038).
Agrama, F. A. (2012). Multi-objective genetic optimization of linear construction projects. HBRC Journal, 8(2),144-151.
Agrama, F. A. E.-M. (2011). Linear projects scheduling using spreadsheets features. Alexandria Engineering
Journal, 50(2), 179-185.
Balasubramanian, A. (2014). Tunnels-types and importance.
Broere, W. (2016). Urban underground space: Solving the problems of today’s cities. Tunnelling and UndergroundSpace Technology, 55, 245-248. doi:https://doi.org/10.1016/j.tust.2015.11.012
De la Fuente, A. (2013). Advances on the use of fibres in precast concrete segmental linings.
Forcael, E., González, M., Muñoz, J., Ramis, F., & Rodriguez, C. (2018). Simplified Scheduling of a Building
Construction Process using Discrete Event Simulation.
Mathew, J., Paul, B., Dileeplal, J., & Mathew, T. (2016). Multi objective optimization for scheduling repetitive
projects using GA. Procedia Technology, 25, 1072-1079.
Pinha, D., & Ahluwalia, R. (2016). Dynamic Project Scheduling using Discrete Event Simulation.
Priya, P. K., & Neamitha, M. (2018). A REVIEW ON PRECAST CONCRETE. International Research Journalof Engineering and Technology.
Sharma, P. (2015). Discrete-event simulation. International journal of scientific & technology research, 4(4), 136140.
Su, Y., & Lucko, G. (2015). Comparison and renaissance of classic line-of-balance and linear schedule concepts
for construction industry. Organization, technology & management in construction: an international
journal, 7(2), 1315-1329.

286

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

INTEGRATING BIM INTO GREEN RESIDENTIAL BUILDING ASSESSMENT: A
CASE STUDY
Fatma Abdelaal1, Brian Guo2, Yang Zou3, and Mazharuddin Syed Ahmed4
1)

Ph.D. Candidate, Department of Civil and Natural Resources Engineering, University of Canterbury, Christchurch, New Zealand. fatma.abdelaal@pg.canterbury.ac.nz

2)

Ph.D., Lecturer, Department of Civil and Natural Resources Engineering, University of Canterbury, Christchurch, New Zealand. brian.guo@canterbury.ac.nz

3)

Ph.D., Lecturer, Department of Civil and Environmental Engineering, University of Auckland, Auckland, New Zealand. yang.zou@auckland.ac.nz

4)

Ph.D., Ara Institute of Canterbury, Christchurch, Canterbury, New Zealand. mazharuddin.syedahmed@ara.ac.nz

Abstract: Building Information Modeling (BIM) technology provides numerous benefits for green building
design and assessment. Recent years have seen attempts to integrate BIM into the commercial and industrial
building assessment process. However, research on BIM applications to green residential buildings is limited.
Thus, this research aims at investigating the potential of integrating BIM into green residential building rating
systems such as Homestar, the green residential assessment system in New Zealand. First, the required data andinformation to calculate Homestar points has been identified by reviewing the Homestar v4 technical manual, inorder to determine the proportion of the design rating points that can be calculated based on BIM functions. Second,
the required data were matched against the data available in a BIM model of a case study of a residential building,the building is the first 10 Homestar rating house in Christchurch, New Zealand. The results reveal that 76 of 120
points of Homestar rating credits can be achieved with BIM software, with 100% computable for Density andResources Efficiency, 67% for Energy, 21% for Water, %17 for Waste, 33% for Management, 100% for Materials,
and 67% for Site. Future efforts can be made to develop a fully automated BIM-based green residential buildings
assessment framework that can help designers optimize the design and assist the assessors with the rating process.
Such a BIM-based assessment framework has a significant potential to simplify the rating process and reduce
assessment costs.
Keywords: Building Information Modeling (BIM), Green Building Assessment, Residential Buildings,
Homestar, New Zealand, Sustainability.
1. INTRODUCTION
The Architecture, Engineering, and Construction (AEC) industry has been responsible for 40-50% of
global greenhouse gas emissions and 30-40% of all primary energy use (Asif et al., 2007). As a result, the valueof sustainable development has been increased, and it becomes a necessity for the industry to adopt sustainability
in order to reduce cost, improve buildings efficiency, and reduce the environmental impacts (Ramesh et al., 2010).
Whilst New Zealand is moving towards its 2050 challenge of achieving Net-Zero buildings, it is vital to highlighttools and methods that assist in achieving that goal. Green building rating systems (GBRS) are tools for evaluating
the environmental performance of sustainable buildings in the face of the increased demand for sustainability.
They provide “a way of structuring environmental information, an objective assessment of building performance,
and a measure of progress towards sustainability” (Ding, 2008). Over the past two decades, several green
assessment tools have been issued and applied internationally, such as LEED (Alwan et al., 2015). Green Star,
Homestar, and the National Australian Built Environment Rating System for New Zealand (NABERSNZ) are
rating systems introduced by the New Zealand Green Building Council (NZGBC) to measure and rate thesustainability performance of existing and new buildings in New Zealand according to the local specific conditions.
In addition to these rating systems in New Zealand, there are several initiatives that promote green residentialbuildings in New Zealand, such as “HomeFit”, “SuperHome Movement”, and “Warm and Dry Program”. The
main purpose of these tools is to raise the standard for healthier, energy-efficient houses.
Building Information Modeling (BIM) technology is considered as a revolutionary movement for the AEC
industry since it provides an integrated and data-rich model which has the ability to carry out complex analysis
such as cost and environmental analysis (Azhar et al., 2009). Previous studies demonstrated that BIM affords
potentials for optimizing green building rating process (Yahya et al., 2016). It can aid in different aspects ofsustainable design, for instance, building orientation, building massing, daylight analysis, water harvesting, energy
modeling, and materials selection (Krygiel & Nies, 2008). Barnes & Castro-lacouture (2009) demonstrated that
13 credits of LEED can be directly evaluated through BIM software. In addition, BIM has been used to assist inachieving more than 35% of the credits in the LEED rating system (Azhar et al., 2011). 26 credits of the Hong
Kong ‘BEAM Plus’ sustainable building rating system can be achieved by BIM tools (Wong & Kuan, 2014).
Gandhi & Jupp (2014) stated that nearly 90% of Green Star Australia credits can be assisted through BIM.
However, Wu & Issa (2013) pointed out that the development of green BIM as ‘immature and unsystematic’. Its
adoption rate in green buildings is still very low and its full potential is yet to be explored (Bueno et al., 2018).
Note that recent research on BIM-based green building assessment has mainly been focused oncommercial and industrial buildings. Efforts that integrate BIM into green residential building design and287

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019assessment process have been very limited. In the context of New Zealand, New Zealand Green Building Council(NZGBC) stated that although a significant interest in registration for Homestar was seen in recent years, the actual
number of the certified houses remains low due to the significant time and cost required (Kirpensteijn, 2017). BIM
and green building assessment have not yet matured in New Zealand, and it is expected to face barriers andchallenges along with their integration regarding the availability of data, interoperability, and lack of knowledge
and experience. Thus, this research aims to investigate the potential of integrating BIM into green residentialbuilding assessment systems (i.e., Homestar).
2. METHOD
A two-step method was adopted in this study. The first step aimed to identify all data that are required toassess green residential building points. For this purpose, Homestar was considered as the case study. Based on
the Homestar, houses are assessed based on seven categories: (1) Energy, Health and Comfort, (2) Density andResource Efficiency, (3) Water, (4) Waste; (5) Management; (6) Materials; (7) Site; (8) Innovation (optional)(NZGBC, 2017). A brief introduction of Homestar assessment categories and points is presented in Table 1. A six(6) or higher Homestar rated buildings provide assurance that the house is warmer, drier, and cost less to run thana typical new house built to the building code standards. A ten (10) Homestar rated house means the house isworld-leading in terms of its environmental performance. In the first step, each Homestar criterion was studied to
identify the necessary data and parameters for the assessment.

Category
DensityandResource
Efficiency
Energy,Healthand Comfort
Water
Waste

Management
Materials

Site

Innovation

Total

Table 1. Homestar assessment categories and points
Description
Points available
Rewards smaller dwellings and residential
8developments with smaller footprints andwhich require fewer resources to build, operate
and occupy.
Rewards attributes that contribute to a60
reduction in energy use
Rewards attributes that contribute to reduced
14
water consumption
Rewards attributes that provide the ability to
6readily recycle waste, as well as construction
practices that reduce waste going to landfill
Rewards attributes that contribute to making a6safe, secure and adaptable dwelling
Rewards the use of responsibly-sourced
14
products and material that have lower
environmental impacts over their lifetime.
Rewards the attributes of the site such as
12
effective stormwater management, thecontribution to local ecology, and the locationof the dwelling in relation to key amenities
Rewards the uptake of building initiatives
Optional
which significantly reduce the environmental
impact of the dwelling
120

Percentage of total
6%

50%
12%
5%

5%
12%

10%

0%

100%

Note: All information is based on the Homestar v4 Technical Manual (NZGBC, 2017).

The second step was to identify available data that can be extracted from BIM modelling platforms tocalculate points of each Homestar category. The study considered Graphisoft ArchiCAD 22 as a BIM modelling
platform as it is one of the common BIM platforms that is being used in New Zealand. Element properties andclassification systems of Graphisoft ArchiCAD 22 were analyzed. All available data from Graphisoft ArchiCAD
22 were matched against data required to assess Homestar points. This was aimed to decide which Homestar
categories and how many points can directly or indirectly be calculated based on BIM data.
3. RESULTS
The findings of the research are presented according to the eight categories of the Homestar assessment
system. The Innovation category is optional and thus it was not included in this research.
3.1 Density and Resource Efficiency
This category promotes and rewards smaller dwellings that have higher resource efficiency and higher
density ratio. Points of resource efficiency can be calculated based on a matrix of conditioned space and number

288

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019of bedrooms. For example, if a 6-bedroom house has conditioned space of fewer than 148 m2, it can be rewarded
a maximum of 5 points. Density ratio can be calculated by dividing the gross floor area by the building footprint.
Houses with a density ratio equal to or greater than 5 can be rewarded a maximum of 3 points. Table 2 shows thedata required to calculate the points underlying the category of Density and Resources Efficiency. Note that all therequired categories can be calculated directly from the BIM model, and so obtain 100% of the category points.
Table 2. Density and Resource category required data in the BIM model
Key data required for Homestar rating
Available data from BIM-based
calculation
BIM
pointsResource Efficiency Conditional floor area
Yes
Bedroom number
Yes
8 points
Density Ratio
Gross floor area
Yes
Building footprintYes
3.2 Energy, Health, and Comfort
The second category of the Homestar rating system is Energy, Health, and Comfort with 50% of Homestar
total score, and it is considered one of the key categories. As shown in Table 3, 40 out of 60 points can be calculated
based on the data available in the BIM model. Thermal comfort and natural lighting can be achieved using theIntegrated Environmental Solution (IES) software.
Table 3. Energy, Health, and Comfort category required data in the BIM model
Sub-category

Key data required for Homestar rating

Thermal Comfort(20 points)
Efficient Space Heating(6 points)

Climate zone location
Windows, Walls, Flooring, Roof data
Heating devices specifications
Areas/zones covered by each heater
IES thermal modeling report
Intermitted extract ventilation
Range hood for the cooking hop
Extraction system for bathrooms
Net openable area to outside
Background ventilators for bedrooms
Continuous extract ventilation
Bathroom extract configures for the home
Interior doors are undercut or have grills
Exterior doors fully sealed
Insulated mechanical ventilation system
Ventilation systems specifications
Energy modeling report

Available datafrom BIM
Yes
Yes
No
Yes
Yes
No
Yes
No
Yes
No
No
No
Yes
Yes
No
No
Yes

Water heaters specifications
Water pressureLighting efficacy (lumens per watt)
Lighting control systems
Lighting plans
IES daylight modelingSound transmission class
Acoustic ceiling total area
Sound-insulating trickle vents in rooms
Design plans
Washing lines space and length

No
No
No
No
Yes
Yes
No
Yes
Yes
Yes
Yes

Ventilation(4 points)

Surface and Interstitial Moisture(5 Points)
Hot Water Heating(6 points)
Lighting (2 points)

Natural Lighting (3 points)
Sound Insulation (3 points)

Inclusive Design (3 points)
Energy-Efficient Drying(1 point)

BIM-basedcalculation points
20 points
4 Points

1 point

5 points
0 points
1 point

3 points
2 points

3 points
1 point

3.3 Water
Water is the third category of the Homestar rating system with 14 points total and two (2) sub-categories;
water use and sustainable water supply. Only 3 of them can be calculated via the BIM model (Table 4), otherrequired data related to the home fixtures specification, and the collection and harvesting systems used for the
rainwater.

289

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 4. Water category required data in the BIM model
Sub-category

Key data required for Homestar rating

Water Use in the Home
(10 points)
Sustainable
Water
Supply (4 points)

Fixtures specifications

Available data fromBIM
No

Rainwater collection system
Roof catchment area
Roof type and slope
Rainwater tank size
Rainwater harvesting system

No
Yes
Yes
Yes
No

BIM-basedcalculation
points0 points
3 points

3.4 Waste
Table 5 shows that only 1 point can be obtained through the BIM model for the Waste category since theinternal and external recycling bins space and capacity are available in the design plans for the rated home. Site
waste minimization plan (SWMP) and onsite sorting requirements are an external database that is not available in
the BIM model.
Table 5. Waste category required data in the BIM model
Sub-category

Key data required for Homestar rating

Construction Waste
Minimization (5 points)
Household Waste
Minimization (1 point)

Site waste minimization plan
Onsite sorting requirements
Internal recyclables bins space
External recycling bin space

Available datafrom BIM
No
No
Yes
Yes

BIM-based calculation
points0 points
1 point

3.5 Management
Two (2) points out of six (6) can be achieved for the security sub-category in the Management category,
while the required data for the home user guide and responsible contracting are not included in the BIM model, as
shown in Table 6.
Table 6. Management category required data in the BIM model
Sub-category

Key data required for Homestar rating

Security (2 points)

Defined house boundary
One window or door is seen from the road
The main entrance is well definedHouse design strategy
Energy operating and maintenance ins.
Water operating and maintenance ins.
Waste bins times of collection
Maintenance schedule
Landscaping and ecology information
House plans and construction details
Appliance manuals
Warranties and guarantees
Local transport information
Environmental Management Plan (EMP)
Contractor’s certificate or ISO

Home User Guide(2 points)

Responsible Contracting(2 points)

Availablefrom BIM
Yes
Yes
Yes
No
No
No
No
No
No
Yes
No
No
No
No
No

data

BIM-basedcalculation points
2 points

0 points

0 points

3.6 Materials
BIM demonstrates its ability to provide all the required data for achieving the Materials category points
as seen below in Table 7. BIM is an information management tool contains the building materials and its type,specification, and quantities.
Table 7. Materials category required data in the BIM model
Sub-category

Key data required for Homestar rating

Sustainable Materials (10 points)

Materials type and specification
Materials quantities
Materials type and specification
Materials quantities

Healthy Materials; interior finisheswith low volatile organic compounds
VOC (4 points)

290

Available datafrom BIM
Yes
Yes
Yes
Yes

BIM-basedcalculation points
10 points
4 points

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3.7 Site
While the all required data for the sub-category native planting can be calculated using the BIM model,
almost half of the points required for the other three (3) sub-categories are available in the model, therefore and as
shown in Table 8, 8 points out of 12 can be achieved.
Table 8. Site category required data in the BIM model
Sub-category

Key data required for Homestar rating

Stormwater Management(4 points)

Site area
Roof area
Permeable area
Onsite stormwater management system
Site/landscape plan
Total land area
Location/site plan

Available data from
BIM
Yes
Yes
Yes
No
Yes
Yes
Yes

Public transport schedule
Site plan
Parking facility specification

No
Yes
No

Native Planting (2 points)
Neighborhood Amenities(4 points)
Cycling (2 points)

BIM-basedcalculation points
3 points

2 points
2 points

1 point

4. DISCUSSION AND CONCLUSIONSThis paper represents a preliminary effort to develop a fully automated BIM-based Homestar assessment
tool. It investigated the potential of integrating BIM into green residential building assessment. Homestar andGraphisoft ArchiCAD 22 were adopted as a case study to demonstrate the link between BIM and green residentialbuilding assessment. The findings indicated that 67 of 120 points can be achieved by integrating BIM into thedesign rating process. As shown in Table 9, 100% of the points underlying the category of Density and Resources
Efficiency, and Materials can be obtained using BIM. Energy, Health, and Comfort category account for 50% ofHomestar total score, and over 66% of the total points can be computed based on BIM. It should be noted that thedata required to calculate Homestar points matched against ArchiCAD models only. Future effort should be madeto consider other BIM platform such as Revit. In addition, future research can be focused to develop an automatedBIM-based Homestar assessment system. This can promote the digitalization and automation in green building
design and assessment. In specific, such a BIM-based assessment system can help designers obtain initial scoresof Homestar and thus facilitate sustainable design optimization. In general, there is a lack of motivation to apply
Homestar rating, due to cost and time involved. The BIM-based system has significant potential to simplify therating process and reduce assessment time and costs. This study recommends developing a data structure of BIMbased green residential building rating systems. This can improve interoperability between different BIM platforms,
and facilitates a fully automated assessment process.
Table 9. BIM-based green residential building assessment results summaryHomestar Category
Points Category Weighting
BIM-based Credits Points
Density and Resources Efficiency
86%8Energy, Health, and Comfort
60
50%
40
Water
14
12%
3
Waste
6
5%
1
Management
65%2Materials
14
12%
14
Site
12
10%
8
Total
120
100%
76

Proportion
100%
66.7%
21.4%
16.7%
33.3%
100%
66.7%
63.3%

Considering the fact that current Homestar assessment process is mainly based on experts (e.g., Homestar
assessors), BIM-based Homestar assessment tools have significant potential to reduce time and cost involved in
the process. More importantly, BIM-based Homestar assessment tools enable designers to analyze energy
efficiency and environmental impacts of different design schemes and scenarios in the early stage of design.
Currently, this can only be done by Homestar assessors until the design is completed. This would promote
communication between designers and clients and support decision making.
It should be noted that there are major challenges to fully integrating BIM into Homestar assessment. First,
ArchiCAD 22 cannot provide all data that are required to assess all Homestar categories. It also lacks the analytic
and reasoning ability to automate the assessment process. This is understandable, as it was not specificallydesigned for building sustainability assessment. Future research is needed to identify other BIM platforms and291

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

tools (e.g., Green Building Studio) can be used to complement ArchiCAD 22. This, however, will raise issues ofinteroperability between different BIM tools and platforms. In addition, Homestar assessment involves rule-based
reasoning. Therefore, developing an ontology to represent the domain knowledge of residential green building
assessment would be helpful. Semantic Web Rule Language (SWRL) rules can be developed based on the ontology
and Homestar manual to automate partial assessment processes.
REFERENCES
Alwan, Z., Greenwood, D., & Gledson, B. (2015). Rapid LEED evaluation performed with BIM-basedsustainability analysis on a virtual construction project. Construction Innovation, 15(2), 134–150.
https://doi.org/10.1108/CI-01-2014-0002
Asif, M., Muneer, T., & Kelley, R. (2007). Life cycle assessment: A case study of a dwelling home in Scotland.
Building and Environment, 42(3), 1391–1394. https://doi.org/10.1016/j.buildenv.2005.11.023
Azhar, S., Brown, J., & Farooqui, R. (2009). BIM-based Sustainability Analysis: An Evaluation of Building
Performance Analysis Software. 45th ASC Annual Conference, (2016), 1–4.
Azhar, S., Carlton, W. A., Olsen, D., & Ahmad, I. (2011). Building information modeling for sustainable design
and
LEED
®
ratinganalysis.
AutomationinConstruction,20(2),
217–224.
https://doi.org/10.1016/j.autcon.2010.09.019
Barnes, S., & Castro-lacouture, D. (2009). BIM-enabled Integrated Optimization Tool for LEED Decisions. In
International Workshop on Computing in Civil Engineering (pp. 258–268). Texas.
Bueno, C., Pereira, L. M., & Fabricio, M. M. (2018). Life cycle assessment and environmental-based choices atthe early design stages: an application using building information modeling. Architectural Engineering and
Design Management, 14(5), 332–346. https://doi.org/10.1080/17452007.2018.1458593
Ding, G. K. C. (2008). Sustainable construction-The role of environmental assessment tools. Journal ofEnvironmental Management, 86(3), 451–464. https://doi.org/10.1016/j.jenvman.2006.12.025
Eboss, & Productivity Partnership. (2014). BIM in New Zealand — an industry-wide view. Eboss, 4–13. Retrieved
from http://www.eboss.co.nz/bim-in-nz-an-industry-wide-view
Gandhi, S., & Jupp, J. (2014). BIM and Australian Green Star Building Certification, 275–282.
https://doi.org/10.1061/9780784413616.035
Kirpensteijn, H. (2017). Planning for Green Building Design and Technology in New Zealand. Retrieved fromhttps://researcharchive.lincoln.ac.nz/bitstream/handle/10182/8022/Kirpensteijn_MPlan_open.pdf?sequence=5
Krygiel, E., & Nies, B. (2008). Green BIM: Successful Sustainable Design [M]. John Wiley (Vol. 4).
NZGBC. (2017). Homestar v4 Technical Manual, (February), 29–30. https://doi.org/10.1007/978-3-319-049540_9
Ramesh, T., Prakash, R., & Shukla, K. K. (2010). Life cycle energy analysis of buildings: An overview. Energy
and Buildings, 42(10), 1592–1600. https://doi.org/10.1016/j.enbuild.2010.05.007
Wong, J. K. W., & Kuan, K. L. (2014). Implementing “BEAM Plus” for BIM-based sustainability analysis.
Automation in Construction, 44, 163–175. https://doi.org/10.1016/j.autcon.2014.04.003
Wu, W., & Issa, R. (2013). Integrated Process Mapping for BIM Implementation in Green Building Project
Delivery. In Proceedings of the 13th International Conference on Construction Applications of Virtual
Reality, (October), 30–31.
Yahya, K., Boussabaine, H., & Alzaed, A. N. (2016). Using life cycle assessment for estimating environmental
impacts and eco-costs from the metal waste in the construction industry. Management of Environmental
Quality: An International Journal, 27(2), 227–244. https://doi.org/10.1108/MEQ-09-2014-0137

292

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

ENHANCED UNDERGROUND UTILITIES MANAGEMENT INTEGRATED CIM
TECHNOLOGIES
Sheng-Lun Zhuo*1, Yu-Cheng Lin2
1) Master graduate student, Department of Civil Engineering, National Taipei University of Technology, No.1. Chung-Hsiao
E. Rd., Sec.3, Taipei, Taiwan. Email: z0933957902@gmail.com
2) Professor, Department of Civil Engineering, National Taipei University of Technology, No.1. Chung-Hsiao E. Rd., Sec.3,
Taipei Taiwan. Email: yclinntut@gmail.com*To whom correspondence should be addressed. E-mail: z0933957902@gmail.com

Abstract: Civil Information Modeling (CIM) is commonly used in civil engineering projects, such as tunnels,
bridges, roads, underground pipelines. CIM can provide a better understanding of engineering problems andonsite status through the visual advantages. The CIM model can present a parameter-based visual effect thatonce established can effectively achieve communication and coordination. Over the years many adjustments
have been made in the existing infrastructure of the pipelines in addition to the new pipelines systems that havebeen switched. Sometimes, due to an error in the underground pipeline drawings or specifications, unnecessary
disasters may occur this would occur because there is not a periodic update in the 2D drawings. In some cases,
after the pipeline location is excavated, underground utilities in service could be affected since they weretraditionally designed with 2D drawings. If the original pipeline is found during construction or maintenance, 2D
drawings will probably show a change or addition, therefore, it will be a difficult task for the construction
company to update or verify the precision of the available maps according to the actual situation on site. There
are also cases where the current or new pipeline location is not shown or specified in the as-built drawings,
making more difficult dinging or other related operations, miscalculation in the construction site for excavation
can cause thousands of dollars of over cost and would affect thousands if the drinking water or electricity
distribution is affected. This study will use CIM in underground pipeline management applications by building,planning and researching about the processes and standards taken for the underground pipeline models. It is
expected that CIM will enable field personnel and construction personnel to keep clear visualization of theunderground soil conditions. And explore the benefits, difficulties and restrictions of building a CIM model forunderground utilities. Finally, through the 3D model for communication coordination and inspection, and using
3D models and virtual space, through 3D visualization to present pipeline conflicts, review the spatial and spatial
needs of facilities and buildings, and analyze the models, and achieve communication between members.
Coordinate and integrate the effectiveness of collaboration.
Keywords: Building Information Modeling (BIM), Civil Information Modeling (CIM), Underground Utilities,
Underground Pipeline Maintenance.
1. INTRODUCTION
The underground pipeline maintenance and installation in civil engineering are one of the mostimportant operations. Due to numerous underground pipeline maintenance units, manufacturers who modify theexisting models cannot be explicitly in the specification and creation of new 3D models, the original drawings
containing information provided by each pipeline unit has some gaps comparing with the actual undergroundinformation, frequently causing accidental digging of underground and undesirable pipelines, therefore causing
disasters. This study will confirm the location of the pipelines in the excavation for the construction
manufacturer through the 3D model. When miss-digging occurs during excavation, changes will be applied in
the model directly during execution, at the same time this will clarify the responsibility of everyone involve, in
simple words the purpose of this study is to better understand the corresponding position of each pipeline in theunderground during the maintenance phase by CIM technology, and improve the limitations encountered in
traditional methods of inspection.
Understanding 2D drawings it is not easy, in order to present pipeline elevation, pipeline
misalignment site surveys must be performed. Furthermore, enhancing the efficiency of traditional maintenanceinspection would be a task for the new CIM model by reducing the cognitive blind spots formed in the processof communication and coordination in different fields for different subcontractors, this CIM model can also
help to eliminate delays in making decisions and reducing the overall cost of the project.This study proposesthat CIM technology is introduced into the maintenance phase of underground pipelines by designing CIM
models that can facilitate pipeline review and improving pre-operation preparations, providing the owners,
manufactures and government agencies with the management references.
2. LITERATURE REVIEWBuilding Information Modeling (BIM) is a new technology, method, concept of operation process. It

293

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019refers to the construction facilities (Including buildings, bridges, roads, and tunnels) during the whole project life
cycle, techniques for creating and maintaining digital information for construction facilities and their engineering
applications. In a way that is easier to imagine and understand, BIM technology is a simulation of realengineering in a computer virtual space that is mainly used to assist engineers in building life cycle planning,
design, construction and operation. (Guo & Xie, 2011) since BIM started in the building industry, it has now
extended to the infrastructure/civil industry (Hong, 2015). Bharathwaj et al. (2016) defined CIM technology as
Civil Integrated Management. Nobuyoshi Yabuki defines CIM technology as Construction Information
Modeling (Yabuki, 2012). CIM is defined that civil engineers start adopting the application of a new concept that
focuses more in civil engineering projects (Cheng, and Lu, 2016). New technologies of operation process likeBIM and CIM provides construction projects with new management and engineering operations methods andconcepts, one of them is the 3D visual presentation and collaborative work across the project area, which provide
high production efficiency by reducing costs and shortening construction time. These new technologies extend
the follow-up operations and maintenance. However, CIM technical terms have not yet been popularized, whichmake the current industry dependent on terms used for implementation of projects with BIM, including technicalterms that represent and standardize operations. In construction industry a lot of the projects are currently
working with BIM operation processes by implementing it in civil engineering projects. CIM is similar to BIM,
many projects use BIM technical nouns to refer to them during execution of CIM projects.
This study will explore laying pipelines in streets, locating the critical points where it is difficult toclearly understand the current status of underground utilities through 2D floor plans, the main reason is that in2D drawings there could be overlaps of more than two pipelines hence turning point is less obvious (see Fig. 1).
Therefore, this study will highlight the necessity in understanding the pipeline system by designing a 3D modelfrom the 2D drawings (see Fig. 2). The difference in clearness of information is displayed in this two models

where 3D models shows a better organization of the given information giving more reliable reference of thesite situation. (Land & Surveyors, 2018)

2D drawings visual conflic

Figure 1. Underground utilities 2D drawing

294Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Use of CIM model to present
pipelines in the first place.

Figure 2. Underground utilities in a 3D Schematic.
Establishing a 3D underground pipeline model is a key advantage. It can bring together different data
and analyze the environment through overall insight (Chen, 2011). Through the combination of GIS and BIM,
the integrated model can achieve different levels of detail. Life cycle data of objects related to urban planningand management processes are shared (Ma & Ren, 2017). BIM provides detailed information about individual
objects, while GIS helps to present spatial information on a large scale (Harris & Batty, 1993). For underground
pipeline management, in particular, BIM technology is mainly used to create detailed models with detailed
attributes, providing all the information for multi-view and organizational communication and decision-making(Chen, 2017) due to its horizontal and vertical orientation. Complex distribution, underground pipeline
construction not only needs to consider the attributes of the sewer itself (such as material, depth, size,
connection), but also the interaction with other objects (roads, manhole covers, etc.) (Davies et al., 2001).
Therefore, it is necessary to store and present geometric related information in underground pipelinemanagement (Zhu et al., 2004). Although there are many software for 3D modeling at this stage, many software
can only be used in a single application, and it is necessary to integrate the applications through multiple
software to achieve the actual desired results. (Morris & Janée, 2009).
The underground infrastructures commonly include the physical piping, sewers, electric cables,
telecom lines, underground water, underground traffics etc. (Acaddrafting, 2017). For underground pipeline
network, the underground BIM requires a model that shows not only the distribution of network but also theconnection with surface objects (Drogemuller, 2009). As a part of the underground pipeline network, applying
BIM into the sewer construction project will promote the plan optimisation and increase the work efficiency(Zhou, 2016).
3 APPLICATION OF CIM FOR UNDERGROUND UTILITIES MANAGEMENT
In the densely populated areas and commercial areas, busy roads, and due to the intricate andcomplicated underground pipelines network in the metropolitan area, the difficulty in maintaining theunderground pipelines is quite high. For example, the old gas pipelines cause gas explosions and cause
casualties, the break of energy pipeline could cause a black out around a whole district. The unknown situationof underground utilities has also become a time bomb with potential for damage to habitants on an unimaginable
scale. In order to carry out pipeline maintenance with safety and effectiveness new construction methods have tobe implemented to improve the operation and service life of pipelines and ensure the normal supply of gas,
electricity and water. Through the CIM model, general contractor personnel can understand the information andmaintenance essentials of underground pipelines. At present, underground pipelines are provided by thecompetent authority or various pipeline units to provide 2D maps and construction at relative positions. In thisstudy, after the CIM model has been drawn for the pipelines that have not yet been laid, the construction units
are provided for pipeline burial according to the model output 2D map. As the pipeline burial regulations295

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

become more and more cumbersome, the pipelines are allowed to be buried as usual, and the excavation processoften encounters underground problems. The construction cost is continuously improved, and many
manufacturers have not yet understood the underground conditions for construction, which affects thesubsequent new pipeline layout.
1. To check visual conflict
Using CIM to solve visual conflict and executes review can solve engineering problems in theconstruction site, so when CIM models are completed and used as main reference, complete management andcoordination integration is achieved, by applying CIM to the information sharing mode of planning and back-end
operation management and maintenance in the design and construction stage. Analysis through civil engineering
information model, construction of a 3D stereoscopic component database, construction information system
integration and maintenance management stages. However, with 3D graphic models drawings completion,
project interface, collision problem, exploring the conditions under the stratum, thus finding an error wouldn’t be
a problem if immediate correction onsite its executed. CIM model integration before construction is also a wayof avoiding this situation because it not only promotes the smoothness of the construction process, it is also themain reason for the development of CIM in the field of civil engineering at the construction stage.
2. To review use space
Via CIM model direct judgment, limitations on ways to improve the review of traditional floor plans,
simplified illustration of nesting and artificial imagination, and CIM model export to virtual reality software,
investigate into dynamic model, and visually check the position between the pipelines, make the use of spacecheck work easier and more realistic.
3. To review dimensional elevation planning
Engineering planners need better engineering experience, a good interpretation ability and spatial
imagination, and when drawing a 2D plan, you must consider the conflict of the three-dimensional space and thearrangement of the pipelines in the underground, when there is a temporary change in design during pipeline
construction or when faced with problems with existing pipelines, must re-correct the relevant chart.
4. To query underground pipeline attribute
In the underground pipeline maintenance management, there are eight main categories,
telecommunications, electric power, fresh water, stained rainwater, gas, oil delivery, irrigation works, these are
the most common pipeline systems. In recent years, the underground pipelines of various counties and cities
have not been fully completed, looking for a systemic problem, pipeline leaking, pipeline interlaced presentation,
detailed position on the map of each pipeline haven’t been well determinate making it very hard to find specificconnections. After building a 3D model, construction personnel can use the model to render outside, it is also
make major pipeline space information and construction operation standards more clear. By rendering through3D models, it is possible to express the actual status of each pipeline, 3D model can be also be used to query
pipeline attributes, pipeline interlacing, and user takeover and current status of each pipeline.
4. CASE STUDYThis study introduces CIM applications and uses "visual communication" and "conflict collision
analysis" as case tests and in the establishment of underground pipeline models, to find ways to use manual
methods to build pipeline difficulties and benefits, so as to increase the way of automation having CIM as a basis
for reference. Finally, it can be imported into the construction site to enable on-site construction personnel toinquire, view pipeline details, model information, and follow-up management and maintenance data.
This case is a case of underground pipelines in a redrawing area of New Taipei City, and uses
Autodesk Revit to draw the telecommunications pipelines and Gas pipeline in the area. The reason for selecting
the telecom pipeline and Gas pipeline is that the pipeline is relatively complete in the underground burial.
Therefore, this study finds the pipeline by drawing the pipeline process. Difficulties in the pipeline are examinedin the following case, providing the basis and solution for the engineers to use it instead of manual drawings ofpipeline.
This study draws the case of underground pipeline embedding in a redrawing area of New Taipei City
into a CIM model, and then discusses and analyzes the preliminary results of the model with the on-site
construction personnel, and enables the on-site construction personnel to communicate effectively andunderstand the underground conditions through the model (Fig. 3 & Fig 4). As shown in the above, and throughthe feedback of the on-site construction unit to modify the comments, the subsequent underground pipelinemaintenance information is more accurate. At present, only the telecom pipeline is focused on the fact that the
telecommunications pipeline is relatively complete, because this case is still in progress, if other pipelines aresubsequently buried, such as: tap water, gas, oil pipelines, etc.

296Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 3. Case expectation of telecom pipeline and Gas pipeline results

Figure 4. Telecommunications and Gas pipeline line in a 3D Schematic

297

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

In this study, the CIM model is drawn. In order to avoid the collision of pipelines and thepossible errors in design and construction, when the design route and the buried position and extension pathof various underground pipelines conflict, the pipelines may overlap each other (see Fig. 5).It cancoordinate with the pipeline ownership units as early as possible, and plan design plans or constructionplans such as line type, pipeline temporary migration and permanent migration.

Figure 5 Pipeline overlaps each other
5. CONCLUSIONSThis study built the underground pipeline into a CIM model. In the design stage of this model, engineers
can be provided with information to better understand the correspondence between underground pipelines
existing drawings and the construction site situation thus avoiding design errors, reduce future changes in design,
reduce risk and improve communication efficiency between units to achieve good construction quality. Thismodel can also be applied to the maintenance and operation phase. This study concluded the following points:
1.
When the underground pipeline is built, it must meet all the requirements of the pipeline design plan andproceed with the integration with the current situation of the construction site. Finally, it must be
submitted to the construction company for inspection, if the pipeline construction details and generalsituation of construction still have differences, the pipeline model should be taken into consideration toundergo a feasibility evaluation.
2.
CIM engineers need to have experience in construction or supervision and need to be familiar with CIM
software, subcontractor’s personnel with relevant experience are also required to participate in providing
information, underground situation of pipelines and operation progress reports, to ensure that the model isin line with the actual needs of the scenario, full communication and coordination are necessary between
the CIM engineers and the construction team members.
3.
Through this study after CIM model is designed, the major pipelines are defined in the model interface
integration and advance simulation analysis of construction problems are executed, this work can be in thedesign process even in an early discussion with the construction team, future construction interface
conflict will be reduced, and this improved information can be used to deliver future maintenance work.
4.
Traditional underground pipeline is more time consuming to bury, although the CIM imported
underground pipeline model is not convenient, but the follow-up application can understand the current
situation of the underground through the CIM model, and the current needs of the relevant units orpractitioners to use CIM more and more, therefore, through CIM model to avoid pipeline mis-excavation ,but the key is to spend time and training so that the relevant units or practitioners can be familiar with theoperation of CIM software.
6. RECOMMENDATION

The major recommendations are summarized as following:
1.

The capability of CIM model not only stops at the simulation and performance of 3D geometry, theinformation contained in its architecture, it will be updated and expanded with the progress of the plan life
cycle, so the exact grasp of underground information will expand the value of the CIM model, and expand
from the general design and construction integration to the long-term benefits of operation and298

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2.

3.

4.

5.

maintenance.
The information provided by the CIM model (including pipeline categories, pipes, pipe diameters,
construction and depth of burial data) can greatly improve the efficiency of the operating process, reduce
human error and cost waste, so that the responsible unit can invest more resources in pipeline topic
analysis, program research and research development, to provide a reference for construction units to havemore underground information.
When build underground pipeline, underground pipeline due to the different units responsible, therefore,
during the construction phase in all relevant units via personnel embedded in the CIM model, extraction,
update and modify information, collaboration can be achieved in different units of support and response tofollow-up on the model more convenient updated.
For this study, the results of CIM model imported case of underground pipeline, underground pipeline
parametric model CIM information can be accumulated as after a new pipeline to have a reference orreuse of the basis, enhance operational efficiency and operating experiences in the model of accumulation.
Since the CIM modeling software version is updated once a year, the automation module Dynamo isadded after the 2018 version, but the functional status of the module opening is not discussed in depth. It
is suggested that the underground pipeline can be further explored by using automation.

7. REFERENCES

Acaddrafting. (2017). THE ENGINEERING DESIGN. THE ENGINEERING DESIGN. Retrieved fromhttps://www.theengineeringdesign.com/bim-utility-infrastructure-engineering-design/
Bharathwaj Sankaran, William J. O’Brien, Paul M. Goodrum,Nabeel Khwaja, Fernanda L. Leite, andJoshua Johnson “Civil Integrated Management for Highway Infrastructure” ,2016. pp10-17
Chen, R. (2011). The development of 3D city model and its applications in urban planning. 2011 19thInternational,Conference,on,Geoinformatics,pp1–5.
https://doi.org/10.1109/GeoInformatics.2011.5981007
Chen, M. (2017). Comprehensive application of BIM and GIS technology in underground integrated pipings
network. Construction in Shanxi, pp4–5.
Https://doi.org/10.13719/j.cnki.cn14-1279/tu.2017.19.141
Davies, J. P., Clarke, B. A., Whiter, J. T., & Cunningham, R. J. (2001, March 1). Factors influencing thestructural deterioration and collapse of rigid sewer pipes. Urban Water. Elsevier.
https://doi.org/10.1016/S1462-0758(01)00017-6
Drogemuller, R. (2009). Can BIM be civil? Queensland Roads, (7),.Retrieved from
Http://eprints.qut.edu.au/27991/,pp47–55
Guo Rongqin, Xie Shangxian, (2011), BIM Technology and Public Works, Public Works Electronic News, Issue
pp38-42.

Hong Jiajun, (2015), “Research on BIM Import Operation and Maintenance Management Pre-Operation
Planning”, Master's thesis, National Taipei University of Science and Technology Disaster
Prevention Master's Program, Taipei.
Harris, B., & Batty, M. (1993). Locational Models, Geographic Information and Planning Support
Systems. Journal of Planning Education and Research, 12(3),pp184–198.
https://doi.org/10.1177/0739456X9301200302

Jack C.P. Cheng, Qiqi Lu , Yichuan Deng“Analytical review and evaluation of civil information
modeling”2016.
Mcgarva, G., Morris, S. P., & Janée, G. (2009). Technology watch report: preserving geospatial data,(May).Retrieved,Form
www.dpconline.org/component/docman/doc_download/363-preserving-geospatial-data-by-guy-mcgar
va-steve-morris-and-gred-greg-janee
Ma, Z., & Ren, Y. (2017). Integrated Application of BIM and GIS: An Overview. Procedia Engineering,
196(June),pp1072–1079.
Https://doi.org/10.1016/j.proeng.2017.08.064

Topographic Land & Property Surveyors，Locate Underground Utilities – 6 levels of Utility Mapping,
2018.
Yabuki,N. “BIM and Construction Information Modeling (CIM) in Japan.”Proceedings of the International
Conference on Computational Design in Engineering，(CODE 2012).
Zhu, L., Wu, X., & Liu, X. (2004). Application of 3D GIS in urban underground space planning CONG.
Rock and Soil Mechanics, (06), 882–886.

299

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

DEVELOPING EFFICIENT MECHANISMS FOR BIM MODEL SIMPLIFICATIONJack C. P. CHENG1, Keyu CHEN2, Weiwei CHEN3,
1) Associate Professor, Department of Civil and Environmental Engineering, The Hong Kong University of Science andTechnology, Hong Kong. E-mail: cejcheng@ust.hk
2) Ph.D. Student, Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology,
Hong Kong. E-mail: kchenal@connect.ust.hk
3) Ph.D. Student, Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology,
Hong Kong. E-mail: wchenau@connect.ust.hk

Abstract: Building information modeling (BIM), which can provide both geometric and semantic information forusers to visualize and interact with each building component, is attracting increasing attention in the architecture,engineering, construction and operations (AECO) industry nowadays. However, complicated and huge BIM
models can increase the time for model transfer, increase the computation work load while rendering, and reduce
the fluency during visualization and interaction. Therefore, this paper aims to develop efficient mechanisms for
BIM model simplification to better utilize large BIM models. This paper mainly focuses on two types ofcomponents: walls and cylindrical components. Walls that are connected to each other have a large number of
redundant polygons, thus an algorithm is proposed to minimize the number of polygons without altering the walls.
Cylindrical components, which have a large amount of redundant polygons on the side surfaces, can be simplified
by removing all redundant polygons. Illustrative examples that evaluates the performance of these developed
mechanisms are also provided.
Keywords:

AR/VR; BIM; Model simplification; Polygon reduction

1. INTRODUCTION
Defined as an advanced approach to the exchange and interoperability of digital information for building
design, construction and facility management, building information modeling (BIM) has been adopted in the whole
lifecycle of building facilities (Chen et al. 2018). Nowadays, BIM is attracting increasing attention in thearchitecture, engineering, construction and operations (AECO) industry for the reason that it can comprise a set oftechnologies, policies and processes with an aim to manage data in digital format throughout the lifecycle of a
project (Penttilä2006). The digital representation of building facilities in BIM is object-based and informationrich (Tan et al. 2017), indicating the advantage of using BIM as data source for information management ofbuilding facilities. With the geometric and semantic information provided by BIM, users can visualize and interact
with each building facility in a 3D environment. With the development of both hardware and software,visualization of BIM models and interaction between users and BIM can be greatly enhanced by emerging
technologies, such as augmented reality (AR) and virtual reality (VR). AR can overlay digital informationsuch as images, videos and virtual models onto the real world to allow users to interact naturally with thesurroundings (Cheng et al. 2017). By linking building facilities with corresponding data source, AR makes
it possible to query and update the information on building facilities. VR provides a complete virtualenvironment which enables AECO professionals to visualize project design even before the project is built.
To integrate BIM with AR/VR, BIM models have to be converted into a particular file format andimported into an AR/VR engine. However, BIM models take a lot of time to transfer and much computation torender because of their sheer size and complexity. When these BIM models are used for visualization andinteraction, the process becomes less fluent. Some building components generated by BIM software have large
numbers of redundant polygons which can be merged while keeping the original shape identical. Some curved
components can be greatly simplified while maximize the consistency of the overall shape. Several algorithms
have been proposed for model simplification (Garland and Heckbert 1997; Rossignac and Borrel 1993; Schroeder
et al. 1992), based on which some software or some functions of software are also developed (PolygonCruncher
2018; Zbrush 2018). With the existing algorithms or software, users can simplify models by setting a particularreduction ratio or the number of polygons to be reduced. However, the current existing algorithms and software
for model simplification cannot provide optimized solutions for polygon reduction dealing with different types ofbuilding component according to their different features. The efficiency of model simplification can be greatly
increased if different types of component can be simplified with corresponding appropriate algorithms, rather thansimplifying all models with a particular ratio.
Therefore, this paper aims to develop mechanisms for BIM model simplification for VR/AR. Different
from the solid modeling methods of BIM software such as Boundary Representation (BRep) and Constructive
Solid Geometry (CSG), 3D models in AR/VR engines (e.g. Unity (Unity 2018), and UNREAL ENGINE(UNREAL 2018)) are represented with a surface modeling method—polygonal meshes, which are composed ofpolygons and vertices. This paper focuses on 3D models composed of triangles, which are commonly used in300

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

AR/VR engines. This paper mainly focuses on two types of components: walls and cylindrical components. Walls
that are connected to each other have a large number of redundant polygons, thus an algorithm is proposed tominimize the number of polygons without altering the walls. Cylindrical components, which have a large amountof redundant polygons on the side surfaces, can be simplified by removing all redundant polygons. Illustrative
examples that evaluates the performance of these developed mechanisms are also provided. The developed
mechanisms for BIM model simplification is shown in Figure 1.

Data transfer for geometric information

Simplecomponents

BIM models

GeometricmodelsProblem
identificationand components
classification

Walls

Cylindercomponents

Polygonreduction
algorithm 1

Light weight versionof geometric models

Polygonreduction
algorithm 2

Figure 1. Flow chart of the mechanisms for BIM model simplification
2. MODEL SIMPLIFICATION FOR WALLS
2.1 Problem statement
Walls are a special case of BIM models that contain no curved surfaces. Although most walls are formedby flat surfaces, the meshes of the walls may not be formed by the minimum number of triangles. While
representing building facilities, the surface modeling method of polygonal meshes will generate redundant
triangles at the junction part if walls are connected with each other. As shown in Figure 2(a), if two walls areconnected with each other, the junction part is regarded as an independent surface and triangles of these surfaces
will be generated accordingly. In a large building, it is very common that a wall is connected with a number ofother walls, indicating that many redundant triangles will be generated, as shown in Figure 2(b). For BIM models,
walls are one of the most common components and are connected with each other in most cases. Therefore, theabove mentioned problem leads to a large number of redundant triangles.

Figure 2. Examples of walls: (a) a wall is connected with a single wall; (b) a wall is connected with multiple
walls
2.2 Polygon reduction
As mentioned in the above section, walls represented by polygonal meshes have redundant triangles at thejunction part. An algorithm is proposed to make sure all surfaces of a wall only contain the minimum number oftriangles. The algorithm is derived from a polygon reduction method named progressive meshes (Hoppe 1997),
which can collapse two or more vertices into a single vertex. In this research, the progressive meshes is achieved

301

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019by iteratively collapsing one vertex onto a nearby vertex. When a vertex is collapsed onto a nearby vertex, thetriangles that contain both vertices will be removed while the triangles that only contain the vertex to be collapsed
but not contain the nearby vertex will be reshaped by replacing the vertex to be collapsed with the nearby vertex.
For example, in Figure 3, vertex U is collapsed onto its nearby vertex V. Triangles ∆1 and ∆2 are removed as they
contain both U and V. Triangles ∆3, ∆4, ∆5, and ∆6 are reshaped by replacing U with V. Triangles ∆7 and ∆8
remain the same.

Figure 3. An example of vertex collapse
The key to this method is to decide which vertices can be collapsed without changing the original shape.
In the case of walls, all surfaces are flat. If two vertices are connected by an edge of a triangle, they can be regarded
as neighbors of each other. For a vertex and one of its neighbors, if all triangles that contain this vertex but do notcontain that neighbor are coplanar with the line segment between the two vertices, this vertex can be collapsed
onto this neighbor. As shown in Figure 4, vertex U can be collapsed onto vertex V while keeping the original shapeidentical, as line UV is coplanar with triangles ∆1, ∆2 and ∆3. However, vertex V cannot be collapsed onto U, asline UV is not coplanar with triangles ∆4 or ∆5.

Figure 4. An example of deciding which vertices to collapse
The vertices to be collapsed can be found using Equation (1). If the sum of the dot products of vector UV
with the normal of each triangle that contains U but does not contain V equals zero, U can be collapsed onto V.
𝑛̅̅̅̅̅̅̅̅̅̅̅̅̅̅
∑|𝑉
𝑈𝑉 ∙ 𝑁𝑜𝑟𝑚𝑖 | = 0𝑖(1)

where ̅̅̅̅̅
𝑉𝑈𝑉 is the vector from U to V, ̅̅̅̅̅̅̅̅̅
𝑁𝑜𝑟𝑚𝑖 is the normal of the ith triangle that contains U but does not containV, and n is the number of triangles that contain U but not V. After U is collapsed onto V, the triangles that containboth U and V are removed from the list of triangles, and the triangles that contain only U but not V are reshapedby replacing U with V. Then the vertex V is also removed from the list of vertices.
By merely using the above equation, it is probable that the reshaped triangles overlap with openings on
the wall. As illustrated in Figure 5, vertex U can be collapsed onto both V and W according to Equation (1).
However, if vertex U is collapsed onto V, the reshaped triangle will overlap with the openings. To avoid theoverlap between the reshaped triangles and the openings, it should be guaranteed that all edges of the reshaped302

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

triangles do not intersect with the edges of triangles that are not changed (intersections at end points are notconsidered as an intersection in this method).

Figure 5. An example of overlap between a reshaped triangle and openings
Equations (2) and (3) are used to detect whether two line segments intersect with each other in 3D space.
Firstly, Equation (2) is used to determine whether two line segments are coplanar. Then Equation (3) is used tofind the intersection point of the two lines where the two line segments are located. Finally, the intersection point
is compared with the end points of the two line segments to determine if it is also the intersection point of the twoline segments.
|𝑉̅3 ∙ (𝑉̅1 × 𝑉̅2 )| = 0(2)(𝑉̅1 × 𝑉̅2 ) ∙ (𝑉̅3 × 𝑉̅2 )(3)
𝑃 = 𝑃1 + 𝑉̅1 []|𝑉̅1 × 𝑉̅2 |2
where 𝑉̅1 and 𝑉̅2 are the vectors of the two line segments, 𝑉̅3 is the vector from the starting point of 𝑉̅1 to thestarting point of 𝑉̅2 , 𝑃1 is the starting point of 𝑉̅1 , and P is the intersection point of the two lines where the twoline segments are located.
The work flow of polygon reduction for walls is as follows. Firstly, all vertices and triangles are extracted
from meshes. Next, each triangle and each neighbor are assigned to the corresponding vertices. Then the lists ofvertices and triangles are renewed iteratively with the collapse of vertices based on Equation (1) while avoiding
overlap between the reshaped triangles and openings based on Equations (2) and (3). At the end, the meshes are
generated with the new lists of vertices and triangles.
2.3 Results and discussion
The proposed polygon reduction algorithm for walls was tested with different walls and the effect turned
out to be significant. The performance of the algorithm was evaluated by analyzing the change of the number ofvertices and triangles. Besides, plugins are developed to export the reshaped meshes into OBJ file format tocompare the file size of models.
Two examples are provided to demonstrate the performance of the algorithm. The building component
shown in Figure 6 is a wall shared by several rooms in a residential building. As shown in Table 1, the number ofvertices and triangles of the wall were reduced by 71% and 77%, respectively. The file size of the model was also
reduced by 73% accordingly. After going through polygon reduction, the shape of the wall remained exactly thesame, while each surface of the wall only contained two triangles, which is the minimum number of triangles toform a rectangle.

303

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 6(a) The original mesh of a wall with no openings; (b) the simplified mesh of the wall

Before reduction
After reduction

Table 1. Results of polygon reduction for the wall in the first example
Number of vertices
Number of triangles
File size
28
522584 bytes
8
↓ 71%
12
↓77%
690 bytes↓73%

The building component shown in Figure 7 is a wall with a rectangular opening. As shown in Table 2, thenumbers of vertices and triangles of the wall were reduced by 43% and 43%, respectively. The file size of themodel was also reduced by 43% accordingly. As illustrated in the above two examples, the polygon reduction ratioof walls is determined by the number of redundant triangles, which depends on how many walls are connected tothe simplified wall.

Figure 7(a) The original mesh of a wall with a rectangular opening; (b) the simplified mesh of the wall

Original mesh
Simplified mesh

Table 2 Results of polygon reduction for the wall in the first example
Number of vertices
Number of triangles
File size
28
562522 bytes
16
↓ 43%
32
↓43%
1444 bytes↓43%

3. MODEL SIMPLIFICATION FOR CYLINDRICAL COMPONENTS
3.1 Problem statement
Cylindrical components are also very common in BIM models. The round pipes of a plumbing system
contain a huge number of cylindrical components. Besides, round ventilation ducts and round columns are also
cylindrical components. For polygonal meshes, all surfaces are formed by polygons, indicating that the top andbottom surfaces of a cylinder are represented by regular polygons, rather than circles. Meanwhile, the side surface
of a cylinder can be represented by triangles that connect the top surface and the bottom surface. The regularpolygons representing the top and bottom surfaces of a cylinder look like circles in rendering mode as there aresufficient number of edges of the polygons, thus the corresponding prism looks like a cylinder. In AR/VR engines,

304

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019a cylinder can be represented by a top surface, a bottom surface and a side surface, while the number of triangles
on a side surface equals the total number of edges of the top and bottom regular polygons regardless of the lengthof the cylinder (shown in Figure 8(a)). However, for the polygonal meshes of BIM models, an increasing numberof triangles will be generated as the length of the cylinder increases (shown in Figure 8(b)). The cylindricalcomponents can be greatly simplified if the redundant triangles can be removed from the side surface.

Figure 8(a) A cylinder generated by an AR/VR engine; (b) a cylinder imported into an AR/VR engine from a
BIM software
3.2 Polygon reduction
As described in Section 3.1, the top and bottom surfaces of a cylinder are represented accurately by regularpolygons, while the side surface of the cylinder may have many redundant triangles, especially when the cylinder
is very long. Therefore, a polygon reduction method is proposed to simplify cylindrical components byregenerating the side surfaces with the top and bottom surfaces. Detailed descriptions of this method are given
below.
Firstly, all vertices and triangles are extracted from the mesh of a cylinder to be simplified and stored in
two lists (shown in Figure 9(a)). Secondly, all triangles with the same unit normal are grouped together to get thetriangles and vertices on the top surface and the bottom surface. It is because the top and bottom surfaces areformed by triangles with the same unit normal while no more than two triangles on the side surface have the same
unit normal. Thirdly, all vertices and triangles on the top and bottom surfaces are kept while all other vertices andtriangles are deleted (shown in Figure 9(b)). Fourthly, the vertices on the top and bottom surfaces are sorted incounterclockwise order with the following three steps: (1) a vertex is selected as the starting point; (2) vectors arecomputed between the starting point and each of the other vertices; (3) the cross product of each pair of vectors is
computed to get the order of the vertices. For example (shown in Figure 9(c)), V, V1, and V2 are vertices on the
top surface of a cylinder and V is selected as the starting point. The cross products of vector ̅̅̅̅̅
𝑉𝑉1 and vector̅̅̅̅̅
𝑉𝑉2 have the same direction as the normal of the top surface, indicating that vertex V1 comes before vertex V2
in the counterclockwise sequence. Similarly, V2 comes before V3, and V3 before V4. At the end, the triangles ofthe side surface are generated with the reordered vertices of the top and bottom surfaces (shown in Figure 9(d)).
Half of the triangles are formed by two adjacent vertices from the top surface and one corresponding vertex (thevertex that is closest to the other two vertices) from the bottom surface, while the other half of the triangles areformed by two adjacent vertices from the bottom surface and one corresponding vertex from the top surface.

305

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 9(a) Extract vertices and triangles; (b) Remove vertices and triangles of the side surface; (c) Reorder
vertices on the top and bottom surfaces; (d) Regenerate triangles on the side surface
3.3 Results and discussion
The proposed polygon reduction algorithm for cylindrical components was tested with several round pipes
and the effect turned out to be significant. An example is provided to demonstrate the performance of the algorithm.
As shown in Figure 10, two plumbing pipes with different lengths are simplified using the proposed method. Aftersimplification, the shapes of the pipes remain almost the same.

Figure 10. An example of the proposed polygon reduction method for cylindrical components
Table 3 illustrates the results of the proposed polygon reduction method for cylindrical components. For
the pipe with a length of 500mm, the numbers of vertices and triangles were reduced by 63% and 61, respectively,
and the file size was also reduced by 63% accordingly. For the pipe with a length of 1500mm, the numbers ofvertices and triangles were reduced by 87% and 86%, respectively, and the file size was also reduced by 87%
accordingly. It can be seen that the longer the cylindrical components, the more significant the polygon reduction
method will be.

306

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

LengthPipe 1
Pipe 2500mm
1500mmTable 3 Results of polygon reduction for the pipes
Number of vertices
Number of triangles
File size
Original
Simplified
Original
Simplified
Original
128
48
236
92
12.0KB
364
48
656
92
34.8KB

Simplified
4.4KB
4.4KB

4. CONCLUSIONSThe complicated BIM models can increase the time of model transfer, increase the computation work load
while rendering, and reduce the fluency during visualization and interaction. Therefore, mechanisms for efficient
BIM model simplification are developed in this paper. This paper mainly focuses on walls and cylindricalcomponents and different model simplification algorithms are proposed accordingly. The algorithms were tested
with single components. As illustrated in the test, the mechanisms can greatly simplify BIM geometric models,
reduce the time for model transfer and improve the fluency during visualization. However, only walls andcylindrical components are included in this paper while other irregular shapes are not considered. In the future,mechanisms for model simplification of other irregular shapes will be developed.
REFERENCES
Chen, W., Chen, K., Cheng, J. C., Wang, Q., and Gan, V. J. (2018). "BIM-based framework for automatic
scheduling of facility maintenance work orders." Automation in Construction, 91, 15-30,
https://doi.org/10.1016/j.autcon.2018.03.007.
Cheng, J. C. P., Chen, K., and Chen, W. "Comparison of Marker-based AR and Marker-less AR: A Case Study on
Indoor Decoration System." Lean and Computing in Construction Congress (LC3): Proceedings of the
Joint Conference on Computing in Construction (JC3).
Garland, M., and Heckbert, P. S. "Surface simplification using quadric error metrics." Proceedings of the 24th
annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley
Publishing Co., 209-216.
Hoppe, H. "View-dependent refinement of progressive meshes." Proceedings of the 24th annual conference on
Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., 189-198.
Penttilä, H. (2006). "Describing the changes in architectural information technology to understand design
complexity and free-form architectural expression." Journal of Information Technology in Construction(ITcon), 11(29), 395-408.
PolygonCruncher (2018). Available at: http://www.mootools.com/plugins/us/polygoncruncher/index.aspx(Accessed on date 18-09-2018).
Rossignac, J., and Borrel, P. (1993). "Multi-resolution 3D approximations for rendering complex scenes."
Modeling in computer graphics, Springer, 455-465.
Schroeder, W. J., Zarge, J. A., and Lorensen, W. E. "Decimation of triangle meshes." ACM siggraph computer
graphics, ACM, 65-70.
Tan, Y., Song, Y., Liu, X., Wang, X., and Cheng, J. C. (2017). "A BIM-based framework for lift planning intopsides disassembly of offshore oil and gas platforms." Automation in construction, 79, 19-30.
Unity (2018). Available at: https://unity3d.com/ on date 18-09-2018).
UNREAL (2018). Available at: https://www.unrealengine.com/ (Accessed on date 27-09-2018).
Zbrush (2018). Available at: https://store.pixologic.com/zbrush-2018/ (Accessed on date 18-09-2018).

307

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Computational Mechanics/Engineering308Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019EVALUATION OF PILE PERFORMANCE IN DIFFERENT LAYERS OF SOIL
INVESTIGATING PILE BEHAVIOR BY OPENSEESPL
Orod Zarrin1, Mohsen Ramezanshirazi2
1) Centre for Infrastructure Performance and Reliability, The University of Newcastle, Callaghan, NSW 2308 (Australia)
2) PhD Candidate, Sapienza University of Rome, Italy, Mohsen Ramezanshirazi, mohsen.ramezanshirazi@uniroma1.it

Abstract: Pile foundations technique is developed to support structures and buildings on soft soil. The mostimportant dynamic load that can affect the pile structure is earthquake excitation. From the 1960s, thecomprehensive investigation of pile foundations during earthquake excitation indicate that piles are subject todamage by affecting the superstructure integrity and serviceability.
The main part of researches has been focused on the behavior of liquefiable soil and lateral spreading
load on piles. During an earthquake, two types of stresses can damage the pile head. Inertial load that is due tosuperstructure and deformation that is caused by surrounding soil. Inertial load and soil deformation are associatedwith the acceleration developed in an earthquake. The acceleration amplitude at the ground surface depends onthe magnitude of earthquakes, soil properties and seismic source distance. According to the investigation, thedamage is between the liquefiable and non-liquefiable layers and also soft and stiff layers. This damage crushes
the pile head by increasing the inertial load, which is applied by the superstructure. On the other hand, the cracks
on the piles due to the surrounding soil are directly related to the soil profile.
In this study, the performance and behavior of pile foundations during different earthquakes areinvestigated. This investigation has done by OpeenSeesPL in three different layers of soil that contain liquefiable
and non-liquefiable layers.
Keywords: Pile, Earthquake, Liquefaction, Non-Liquefiable, Damage.

1. INTRODUCTION
According to the recent researches, pile foundations are very vulnerable during earthquake excitation indifferent soil layers. The surrounding soil has direct influence on piles behavior. Soils with different stiffness
have different behavior during earthquake and casus drastic deformation and stress through the pile length (Bobet,
et al., 2001). The previous observations on the piles performance in liquefied layers showed that the negative
effect of liquefaction has rang from structural damage to excessive deformations.
Predicting the piles behavior in liquefiable soil that subjected to earthquake excitation is very complex,
which need to considered the inertial and kinematic loads (Wilson, et al., 2000). Kinematic and inertial interactions
are the two major effects of earthquake on the pile foundation. Kinematic interaction is based on the pile effects
on free field ground motion and inertial interaction, is based on the inertial loads of superstructure that transfer tothe piles (Chu andTruman, 2004).
From the last earthquakes throughout the world, so many valuable experience in terms of soil liquefaction
behavior during earthquakes have been found such as Loma (1989), Luzon (1990), Manjil (1990), Kobe (1995),
Manzanillo (1995), Chi-Chi (1999), Kocaeli (1999), and Bhuj (2001) (Bird andBommer, 2004). Saturated soils
above the water table are always susceptible to liquefaction, which can be a major hazard for piles and deep
foundations (Arab, et al., 2011).
The foundation of buildings, roads, bridges and other infrastructural facilities are always affected byliquefaction phenomena during an earthquake. And the common problems for piles are cracking, rupturing andrupturing of pile connections, or in extreme cases permanent lateral and vertical deflection and rotations of pileheads with impact on the superstructure (Abe and Kuwabara, 2004).
Tokida et al. 1992; Girard and Taylor 1994; Towhata and Mizutani 1999 have proofed that the resistance
between pile and liquefiable soil will change by different velocity of pile movements (Mizutani, et al., 1999;
Tokida, et al., 1992). The experimental research showed that the resistance of pile would increase linearly in thesaturated sand. Kutter and Voss (1995) have found that the resistance of pile in the saturated soil affected bydrainage of pore pressure around the piles. Towhata (1999) revealed, “The rate of change of drag resistance withpipe velocity in liquefied sand increased with increasing sand density while torsional shear tests on the same sandshowed that its undrained modulus was relatively insensitive to strain rate”.
Abdoun, et al. (1997) and Dobry and Abdoun (1998) experimental program results revealed that thelateral pressure will increase in saturated liquefiable layer that place between to non-liquefiable layers and lead toincrease the pile stress and displacement. In this study, the performance and behavior of pile foundations during
three different layers are investigated. A liquefaction sand layer placed between two non-liquefiable clay layerswith different stiffness. The soil and pile behavior in the liquefiable and non-liquefiable layers have been modeled
and evaluated by OpenSeesPL.

309

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2. LIQUEFACTION PROCESSThe process of liquefaction related to the layers that cannot tolerate more pressure and behave like a liquid
by losing strength. Clay-free deposits of sand is the layer that susceptible to the liquefaction. The liquefaction is
being occurred when the soil is subjected to seismic waves and affected the saturated layers by distorting its
structure and make particles of soil to lose their strength (Figure 1).
In the layers, that drainage is impossible the collapse can increase the pore-water pressure among thegrains. By increasing the pore-water pressure to the level of the soil weight, the viscous liquid action start andliquefaction occur.
In the liquefied layers, the deformation of soil may happen easily by little shear resistance and it candamage buildings and other structures. The liquefaction phenomena depend on the soil looseness’s, the amountof clay or cementing between particles and the amount of drainage restriction. Furthermore, the following reason
can affect the soil deformation during liquefaction such as ground slop, thickness, depth and distribution of loadby structures. Totally the looser and younger sediment along the high water tables, the more susceptible toliquefaction.

Figure 1: process of liquefaction (water-saturated sand grains) (Tokida, et al., 1992)

3. LATERAL SPREADS
Lateral spreading is another phenomena which impose additional loading to piles and in most of cases
produce diagonal cracks at the pile head in the non-liquefied and liquefied layer (Bobet, et al., 2001). Lateral
spreading is involved with lateral displacement of a large block of soil, which is the results of liquefaction on thesubsurface layers (Figure 2). Displacement in the lateral happens in response of two forces that generated byearthquakes: gravitational and inertial forces. Lateral spreading; usually develop on slopes with less than 3 degrees
and displacement toward a free face like river channel. And also has a horizontal displacements up to several
meters (Youd and Hoose, 1978).

Figure 2: Diagram of a lateral spread (Tokida, et al., 1992)
Many piles, which damage or collapse by the earthquake is located in the lateral spreading zone and it
can cause large kinematic loads due to the lateral ground movement. About the piles head damage two items have
involve; the free filed and lateral spreading zone. The superstructure and lateral ground movement cause inertial
and kinematic loads respectively. The key factors for pile design that can tolerant the earthquake excitation are
shear forces and bending moments. The corrected prediction that goes follow can help piles to survive:

310

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019(a) Estimating the lateral pressure acting on the pile by the lateral spread and then evaluating the consequent
pile response.
(b) First estimating the ground displacement within the free-field soil profile, and then applying thisdisplacement profile to the pile through a Winkler soil model.
(c) Treating the displacing soil as a viscous fluid, and computing the resultant pile actions (Amiri, 2008).
The timing of lateral spreading in the strong ground shaking can be affected by various factors; one ofthese important factors that should be mentioned is excess pore pressures during earthquake excitation.

4. DAMAGE LOCATION
Pile failures first happen at the pile head, where shear and bending moment are maximized. However,some samples show that cracks can occur:a) In the layers with great differences in stiffness.
b) In the layers of liquefied and non-liquefied.
c) In the second largest moment.
d) In the location of bars reduction (Chu andTruman, 2004).
The damages usually happen in the layers of stiff and soft soil, where the concentrate of strain at different
stiffness of the soil is high. In addition, there is a same action in the liquefied and non-liquefied layers, because
during an earthquake the stresses in the liquefied layers reduce the soil stiffness. The liquefied, loose and soft
layers are susceptible to the second biggest moment. The construction of a plastic hinges at the upper part of thepile after the head failure and the redistribution of bending moments makes new horizontal cracks (Amiri, 2008).

Figure 3: Damage location of pile in different layer soil (Bobet, et al., 2001).

5. SOIL PROFILE (LIQUEFIABLE AND NON-LIQUEFIABLE LAYER)
The liquefaction soil in this research has three section. The two non-liquefiable layers at the top (surface
layer) and bottom (base layer) supporting the liquefiable layer. The most dramatic point is in the liquefiable layer,
which the free field displacement appeared due to the lateral soil pressure (Karlsrud, 2012). Liquefaction during
earthquake cause a stiffness reduction in the liquefiable layer and make a large deformation on the piles. At thistime, the non-liquefiable layer moved with liquefiable soil and it applies large lateral load to the piles(Cubrinovski, et al., 2006).

6. SOIL AND EARTHQUAKE PROPERTIES
Table 1 and 2 list the three different soil and earthquake that used in this study.
Table 1: Soil properties
Name

Date

StationPGA (g)M

Tabas - Iran
Hyogo-ken Nanbu –JapanNorthridge -USA

1978

Taft

0.400

7.5

1995

Kobe

0. 837

7.4

1994

USGS OFR

0.843

7.2

311

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 2: Earthquake properties
Cohesion
(kPa)

Mass
Density(Mg/m3)Layer of Soil

U-Clay 2 (First
layer 5 m)

Saturated Sand
(Second Layer 3 m)

U-Clay 2 (Third
layer 2 m)

Numberof Yield
SurfacePeak Shear Strain

Multiply by((Sqrt(3)/2)

Multiply by ((Sqrt(2)/3)

1.8

75

3

20

1.7

0

10

20

1.8

75

10

20

7. THE TIME HISTORY ANALYSISAs mention before, the acceleration of earthquake has strong influence on the piles that can be interpreted
by below graphs in different depth and layers. The behaviour of saturated sand is substantial, when the water porepressure increase up to the zero effective stress condition. Figure 4 show the recorded of time histories ofaccelerations along the pile length at specified depth. It is clearly observed that the amplitudes of acceleration
increase as the depth decreases, especially in the liquefiable layer (saturated sand). The below Figures show thatby passing the liquefiable layer and reaching the stiffer soil the acceleration amplitude decreased significantly(Figure 4, depth 9).
Depth 5 m

0.15

Tabas

0.06

Hyogo-Ken-Nanbu

0.04

Northridge

Tabas
0.1

Hyogo-Ken-Nanbu
Northridge

Acceleration (g)Acceleration (g)Depth 0 m
0.08

0.02
0
-0.02

0.05

0

-0.04
-0.05

-0.06
-0.08

-0.1

0

5

10

15

20

25

30

35

0

Time (s)Depth 6.5 m

2

15

20

Time (s)

25

30

35

Depth 8 m
Tabas

Hyogo-ken-Nanbu

0.1

Northridge

Acceleration (g)Acceleration (g)

10

0.15

Tabas

1

5

0

-1

Hyogo-ken-Nanbu
Northridge

0.05
0
-0.05

-2

-0.1
0

5

10

15

20

25

Time (s)

30

35

0

5

Acceleration (g)Depth 9 m

1015Time (s)20

0.06

Tabas

0.04

Hyogo-ken-Nanbu

25

30

35

Northridge

0.02
0

-0.02
-0.04
-0.06
0

5

10

15

20

25

30

35

Time (s)Figure 4: Accelerations of pile along the length at different depth

8. EXCEED PORE-WATER PRESSURE
Figures 5 shows the pore-pressure time histories for three earthquake excitations. During the first layerthat contain clay, in all three earthquake, profile behaviour show a very little excess pore pressure because of the
non-liquefiable soil and also in the depth 6.5 m no pore pressure recorded for Tabas earthquake.
In the second part, the sand layer is saturated and profile behaviour show a stronger excess pore pressurethat can be considered the liquefiable layer causes this behaviour. In this layer, the Tabas earthquake shows thehigher effect due to the great peak acceleration in comparison to the other earthquakes. The last part that belong

312Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

to the stiffer clay, the significant degradation happen, even though it placed under the water table level. The sandlayer obviously showed a better behaviour than upper layer on the frequency and amplitude. The transition
behaviour in the liquefiable layer in all earthquakes showed a sharp degradation in pore pressure that coincidencewith sharp increase in accelerations. It can be explained that the coincidence of sharp pore pressure degradation
and acceleration rises was due to the deliquefaction shock waves.
Depth 6.5 m

1.5

Hyogo-Ken Nanbu
Northridge

1
0.5
0
-0.5
-1
-1.5

Tabas
Hyogo-ken-Nanbu
Northridge

120
100
80
60
40
20
0

0

10

20

30

Time (s)0510Depth 9 m

20

Exceed Pore Pressure (kPa)

500
400
300
200
Tabas
Northridge
0

5

10

15

20

25

Time (s)

30

30

35

200
150
100

Tabas
Hyogo-ken-Nanbu

50

Northridge

Hyogo-ken-Nanbu

025

Depth 10 m

250

100

15

Time (s)600
Exceed Pore Pressure (kPa)

Depth 8 m

140
Exceed Pore Pressure (kPa)

Exceed Pore Pressure (kPa)

2

0
0

35

10

20

Time (s)30

40

Figure 5: Exceed pore pressure for three earthquakes
The Figure 6 shows the variations of exceed pore pressure along the pile length and by reaching theliquefaction layer, the soil behaviour changed. Liquefaction of the entire sand layer achieved around 5s for Hyogoken Nanbu and Northridge earthquake and 10s for Tabas earthquake at the depth of 8 m. The time of liquefaction
behaviour changed in the depth of 9 m by 17s for the Hyogo-ken Nanbu and Northridge earthquake and about 32s
for Tabas earthquake. The rapid increase of pore pressures during the aforementioned seconds are due to the highacceleration. And it is a common behaviour that at lower frequencies the amplitude decreases (Abe, et al., 2004).
0
1

Tabas

2

Hyogo-ken-Nanbu

Depth (m)3Northridge

4
5
6
7
8
9
10

-40

60

160

260
360
Exceed Pore Pressure (kPa)

460

560

Figure 6: Exceed pore pressure along the pile length for three earthquakes
The significant part of saturated layer interaction with pile is during the increasing of water pore pressure.
In this period, the bending moments of pile considerably increase. However, by saturating the sand layer, theliquefied layer interaction along the pile decrease which present small bending moments [(ru) ore pressure ratio= excess pore pressure / initial effective stress].

313

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

9. P-Y ANALYSIS
Winkler model is a suitable method to evaluate the soil-pile interaction by representing p-y curves. A py curve shows the resistance or reaction of soil to the pile lateral displacement or in some cases the relative
displacement between soil and pile. The reaction force p can be obtained either from double differentiation ofbending moments or directly from the data recorded by the soil pressure transducers (experimental test). The yparameter that shows the pile displacements were calculated by integrating the pile rotations along the pile.
Figure 7 to 9 display the typical interactive p-y curves for three earthquakes at different depths.
Accordingly, by increasing the depth of pile the curves show a soften behaviour. It is clearly observed that at 8m
depth, the pile shows a large displacement in all earthquakes. This depth represents the liquefaction layer andindicates that the saturated soil has undergone a higher displacement in comparison to the other depths of pile. Bylooking at the first and last layer of soil, it is obvious that they had greater resistance in comparison to theliquefiable layer due to the poor cohesiveness and non-liquefaction property. Figure 10 (depth 5 and 6.5 m) show
a complex behaviour of different mechanism of pile. In these Figures, hardening happened after reaching theultimate strength. It is clear that by increasing the pore pressure, the behaviour of soil changed and after reaching
the maximum pore pressure the force (P) decreased.
In the Hyogo-Ken Nanbu and Northridge, the pore pressure increased early around 5s and for Tabas it
started around 10s. From Figures 4, it is clear that the displacement showed several peaks with the same magnitude
that related a substantial reduction of pile stiffness. The softer behaviour of pile in all earthquake is mainly due tothe high pore pressure with larger displacement. The softening trend of Figures 7 to 9 show that the pore pressure
increased continuously up to 10s and after around 12s it started to decrease almost in all earthquake (Figures 4).
In the liquefiable layer, the Figures 4 show the softer behaviour of p-y, which is related to the water pore pressure
degradation.

Tabas

1000

0 m (depth)

03.33 m (depth)
5 m (depth)-1000

6.5 m (depth)
8 m (depth)-2000

9 m (depth)
-3000
-0.01

-0.005

0

0.005

0.01

0.015

Displacement (m)Figure 7: p-y behaviour at different depth for Tabas Earthquake

Hyogo-Ken Nanbu
5000
4000

(Load) Shear Force (kn)(Load) Shear Force (kn)2000

0 m (depth)

3000

3.33 m (depth)
2000

5 m (depth)
1000

6.5 m (depth)08 m (depth)-1000

9 m (depth)

-2000
-3000

-0.03

-0.025-0.02-0.015-0.01-0.005

Displacement (m)

314

0

0.005

0.01

0.015

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 8: p-y behaviour at different depth for Hyogo-Ken Naubu Earthquake

Northridge
3000(Load) Shear Force (kn)

2000
1000

0 m (depth)

03.33 m (depth)-1000

5 m (depth)-2000

6.5 m (depth)-3000

8 m (depth)-4000

9 m (depth)

-5000

-0.02

-0.01

0

0.01

0.02

0.03

Displacement (m)Figure 9: p-y behaviour at different depth for Northridge Earthquake
In the Figure 10, by starting the liquefiable layer, the behavior of pile has been changed and graph showsa permanent displacement. However, in the stiffer layers the pile behavior shows an elastic trend (Figure 10, depth
0, 3.33 and 9 m).
Depth 0 m

Depth 3.33 m
2000

500

0

Tabas
Hyogo-Ken-Nanbu

-500

-1000
-0.03

Northridge

-0.02

-0.01

0

0.01

0.02

(Load) Shear Force (kn)(Load) Shear Force (kn)

1000

1000

0
Tabas
-1000

Hyogo-Ken-Nanbu
Northridge

-2000
-0.04

0.03

-0.03

-0.02

Depth 5 m

0.01

0.02

0.03

Depth 6.5 m

1000
500
0
Tabas

-500

Hyogo-Ken-Nanbu-1000

Northridge(Load) Shear Force (kn)(Load) Shear Force (kn)

0

3000

1500

-1500
-0.03

-0.01

Displacement (m)Displacement (m)

Tabas

2000
Hyogo-Ken-Nanbu

1000

Northridge

0
-1000
-2000
-3000

-0.02

-0.01

0

0.01

0.02

-0.02

0.03

-0.015-0.01-0.005

0

0.005

0.01

Displacement (m)Displacement (m)

315

0.015

0.02

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Depth 8 m

Depth 9 m
3000

3000

Tabas

2000

Hyogo-Ken-Nanbu

1000

Northridge

0
-1000
-2000
-3000
-4000
-0.008(Load) Shear Force (kn)(Load) Shear Force (kn)

4000

2000

Tabas
Hyogo-Ken-Nanbu

1000

Northridge
0
-1000
-2000
-3000

-0.006-0.004-0.002

0

0.002

0.004

0.006

0.008

-0.002

Displacement (m)-0.001

0

0.001

0.002

Displacement (m)Figure 10: p-y behavior at different depth for three earthquake

10. CONCLUSION
To evaluate the piles response in the lateral spreading, the liquefiable layer should be considered carefully.
Piles always susceptible to damage in the layers with different stiffness, which can undergo a large displacement.
The other important fact that worth to mention is the properties of surrounding soils that have strong influence on
the seismic interaction and the displacement of the piles. One of the fundamental principles of piles behavior is in
the liquefaction soil, which can cause degradation in the stiffness and lateral support. The pile damages usually
happen in the layers of stiff and soft soil, where the concentrate of strain at different stiffness of the soil is high.
In addition, there is a same action in the liquefiable and non-liquefiable layers, because during an earthquake thestresses in the liquefiable layers reduce the soil stiffness.
Liquefaction is one of the most important reason that leads to heavy damage in piles. When the load ofinertial axial applies to the foundation, piles adherence is subjected to the critical situation. Experimental test andp-y analyses revealed that bending moments induced in the piles when the sand layer liquefies. Pile geometry
helped reducing undesirable torsional effects during earthquake.
This paper analysis the pile behavior in different layers of soil during three earthquakes excitation. The
results of pile behavior represent the interaction of soil and pile in terms of p-y behavior, the effect of water porepressure, the pile hysteresis behavior, and the time history behavior of pile in different depth. The p-y behavior
displayed some characteristics of stress-strain through the layers of soil. According to the p-y analyses, theliquefiable sand experienced greater bending moments. The maximum bending moment (pore pressure anddisplacement) in all earthquake happened in the layer where the liquefaction sand placed between two stiffer
layers.
The lateral resistance p in liquefiable layer was less than the other layers because of the saturated situationin sand layer. In addition to less resistance in the sand layer, the softening trend can be seen in this layer during
the earthquakes as the excess pore pressures increased. The lateral p-y resistance of liquefiable sand is a complex
phenomenon that has been shown to be significantly affected by relative density, cyclic degradation, excess porepressures, phase transformation behavior, prior displacement history, and loading rate. Estimating force anddisplacement demands on pile-supported structures in liquefiable sand requires the challenging task of estimating
the effects of liquefaction on the dynamic response of the soil profile (including any permanent deformations) andthe soil-pile–structure system.

REFERENCES
Abdoun, T, Dobry, R, and O'Rourke, TD. (1997). Centrifuge and numerical modeling of soil-pileinteraction during earthquake induced soil liquefaction and lateral spreading. Paper
presented at the Observation and Modeling in Numerical Analysis and Model Tests in
Dynamic Soil-Structure Interaction Problems.
Abe, Akio, and Kuwabara, Fumio. (2004). ANALYSIS OF SATURATED SAND-PILE
INTERACTION UNDER EARTHQUAKE CONDITION BASED ON EQUIVALENT
LINEAR METOD.
Abe, Akio, Meneses, Jorge, Sato, Masayoshi, and Mohajeri, Masoud. (2004). Near-full scale testing
and analysis of saturated sand-pile interaction under earthquake condition. Paper presented
at the Proc. 13th World Conference on Earthquake Engineering, Paper.

316

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Amiri, Sharid Khan. (2008). The earthquake response of bridge pile foundations to liquefaction
induced lateral spread displacement demands: University of Southern California.
Arab, A, Shahrour, I, and Lancelot, L. (2011). A laboratory study of liquefaction of partially saturated
sand/Estudio en laboratorio sobre licuefacción de arena parcialmente saturada. Journal of
Iberian Geology, 37(1), 29.
Bird, Juliet F, and Bommer, Julian J. (2004). Earthquake losses due to ground failure. Engineering
geology, 75(2), 147-179.
Bobet, Antonio, Salgado, Rodrigo, and Loukidis, Dimitrios. (2001). Seismic design of deep
foundations. Joint Transportation Research Program, 188.
Chu, D, and Truman, KZ. (2004). Effects of pile foundation configurations in seismic soil-pilestructure interaction. Paper presented at the 13th World Conference on Earthquake
Engineering.
Cubrinovski, Misko, Ishihara, Kenji, and Poulos, Harry. (2006). Pseudo-static analysis of piles
subjected to lateral spreading.
Dobry, Ricardo, and Abdoun, Tarek. (1998). Post-triggering response of liquefied sand in the freefield and near foundations. Paper presented at the Geotechnical Earthquake Engineering andSoil Dynamics III.
Karlsrud, Kjell. (2012). Prediction of load-displacement behaviour and capacity ofaxially loaded
piles in clay based on analyses and interpretation of pile load test results.
Kutter, Bruce L, and Voss, Tom. (1995). Analysis of data on plow resistance in dense, saturated,cohesionless soil: Naval Facilities Engineering Service Center.
Mizutani, T, Towhata, I, and Anai, K. (1999). Shaking table tests on seismic behavior of sheet pile
quay walls subjected to backfill liquefaction. A. A. Balkema, P. O. Box 1675 NL-3000 BR
Rotterdam The Netherlands, 551-554.
Tokida, Ken-ichi, Matsumoto, Hideo, and Iwasaki, Hideaki. (1992). Experimental study on dragacting on piles in ground flowing by soil liquefaction. In Technical Report NCEER (Vol. 92,
pp. 511-523): US National Center for Earthquake Engineering Research (NCEER).
Towhata, I., and Mizutani, T. . (1999). Effect of subsurface liquefaction
on stability of embankment resting on surface. Paper presented at the 2nd International Conference on
Earthquake Geotechical Engineering Balkema, Rotterdam, Netherlands.
Wilson, Daniel W, Boulanger, Ross W, and Kutter, Bruce L. (2000). Observed seismic lateralresistance of liquefying sand. Journal of Geotechnical and Geoenvironmental Engineering,
126(10), 898-906.
Youd, T Leslie, and Hoose, Seena N. (1978). Historic ground failures in northern California
triggered by earthquakes (Vol. 993): US Govt. Print. Off.

317

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

DEVELOPMENT OF STATIC AND DYNAMIC MODELING APPROACHES USINGFRAME MODELS FOR CITY SEISMIC RESPONSE ANALYSIS
Pher Errol B. Quinay1, Aileen Rachelle Fader2, Franz Marius Carangan3
1) Dr. Eng., Assoc. Prof., Institute of Civil Engineering, University of the Philippines Diliman, Quezon City, Philippines.
Email: pbquinay2@up.edu.ph2) Quantity Surveyor - Technical Team, Wallcrete Company, Inc., Quezon City, Philippines, Email:aileenrachellef.wci@gmail.com
3) Junior Structural Engineer, Al Abbar Aluminum Philippines, Inc., Pasig City, Philippines, Email:FranzC@alabbargroup.com

Abstract: Methods that can estimate the city response to seismic events are valuable tools that can assist in disasterrisk reduction efforts. With the increasing availability of large computing resources, physics-based approaches thatcan be used for this purpose are becoming more practicable. This study aims to develop static and dynamic
modeling approaches for city seismic response analysis. For both modeling approaches, tools were developed togenerate and analyze models that are suitable to the available GIS and BIM data. To check the accuracy of thedeveloped tools, validation tests were conducted by comparing with commercial software that is commonly usedin structural analysis and design in the Philippines. Validations for static and dynamic analysis show that the resultsof the developed tools are within 6% and 8% of the results that were computed using a commercial software,
respectively. These results are considered acceptable given the low computation cost per building in the developed
approaches. As a demonstrative example, two cities in Metro Manila were considered for scenario earthquake
analysis. Low to midrise reinforced concrete structures were analyzed and floor displacements were computed.
From these results, maximum responses were obtained and visualized in city-level. Another example wasconducted with the aim to compare the computed period of vibration with that of a previous study that performedexperiment on a three-story building in Metro Manila. Results show that for the considered standard model of thebuilding, the periods of vibration can be closely estimated by the developed tool.
Keywords:

City seismic response analysis, finite element method, static and dynamic analysis

1. INTRODUCTION
In community disaster mitigation planning and preparation, determining the number of affected structures
and the critical areas in the city for different earthquake scenarios can provide relevant inputs for resource
allocation and in developing effective evacuation and recovery plan. At present, the area-based approach iscommonly-used to estimate the response of different cities to earthquake scenarios. This approach combines dataon hazard, exposure, and vulnerability, through maps and pre-computed vulnerability and fragility curves (Bautista
et al., 2011). Because most of the data are processed from validated exposure information and recorded damage,the approach is considered empirical and reliable for damage estimation. However, extending the applicability ofthis approach to model the dynamic behavior of individual structures is difficult, because area-based approach is
not fully-developed to account for the influence of geometry, material properties, and site conditions. An
alternative approach is to analyze the response of individual buildings in the city, as in the simulation-based
approach (Hori et al., 2018).
Simulation-based approach aims to estimate the overall response of a city to an earthquake scenario by
combining the peak responses of individual buildings. Its reliability in estimation relies on taking into account theeffect of each building’s geometry and material properties to its dynamic response. One advantage of this approachis that multiple analysis can be performed to study the variability in response given the variability in material
properties or input loading. This results to a large database of responses which can be used in assessing thestructural integrity of buildings following an earthquake event.
A major challenge in using simulation-based modeling for seismic response of cities is the significant
computer resource that may be required to process and store information. There are many related studies using
available data, such as GIS data to generate analysis models (Homma et al., 2014; Fujita, et al. 2015, Quinay &Ichimura, 2016, Quinay et al., 2018). These studies show that even with the use of GIS data, with lower resolution
compared to building drawings, there is still a need to develop in-house tools to manage the computation, especiallyfor the analysis of large cities. Thus, it still remains a challenge to use building drawings as models for use in city
seismic analysis.
This study aims to develop a computationally-efficient approach for city seismic response analysis. Twomodeling approaches are proposed: static and dynamic modeling. The static modeling approach aims to provide ameans for rapid estimation of building’s response from calculated base shear due to the seismic demand. The
dynamic modeling approach aims to determine the dynamic properties, such as natural period of vibration of the318Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019building. For both modeling approaches, tools were developed to generate and analyze models suitable to theavailable GIS and BIM data. The tools were also implemented with computing techniques to anticipate the large
computation cost of analyzing up to thousand structures in the target city.
2. THE STATIC AND DYNAMIC MODELING APPROACHES
In numerical modeling, the resolution of available data can limit the accuracy of the analysis model thatcan be generated. For city seismic response analysis, it is important to tailor the modeling approaches according
to the available data. In the case of Metro Manila, GIS data are widely available, but digitized engineering drawings
are relatively few.
2.1 Static Modeling Approach
The static modeling approach aims to provide a means for rapid estimation of building’s response due tothe seismic demand. For this purpose, the required level of detail of the analysis models is generally not high, andthe widely-available GIS data may be used. The GIS data considered here have resolutions as fine as in the orderof a meter. This resolution is fine enough to describe the shape of the building footprint. Together with available
height information, the three-dimensional geometry of the structure can be approximated, including details, suchas the number of floor levels, or building type. In the approach developed in this study (see Figure 1), templatemodels – each representing particular building types in the target area, are generated. The template models are
generated as BIM models, where details in geometry and material properties may be easily inputted. BIM is usedhere to anticipate the wide availability of BIM for building structures in Metro Manila in the future. A tool wasdeveloped to automatically generate frame models from the BIM template models (Fader & Quinay, 2018). The
frame models are composed of line elements with six-nodal degrees of freedom. Frame models are fixed at thebase (ground level), and the horizontal force loading, representing earthquake loading, are computed from baseshear that are redistributed to floor levels. The results of the static analysis are displacements and story drifts,
which can be compared to threshold values to classify the structure as requiring further analysis.
2.2 Dynamic Modeling Approach
The dynamic modeling approach aims to determine the dynamic properties, such as natural period ofvibration of the building. Because of the demand for higher resolution models in this approach, it is suited forbuilding models with available structural details, such as engineering drawings (see Figure 2). For this approach,a three-dimensional solid finite element model is generated and dynamic analysis is performed using a timevarying input. In this analysis, it is expected that there is a significant jump in the computation cost, and it isimportant to introduce computational techniques in order to realize the computation in practical time.
The model generation follows the finite element mesh generation approach of Ichimura et al. (2007)
wherein the mesh of tetrahedron and hexahedron elements are placed strategically in the model to generate highaccuracy in modeling complicated geometries while reducing the computation cost. This is achieved by using theoctree technique and multiresolution grid. In their approach, the model’s three-dimensional geometry is alreadydefined before the meshing procedure. In this study, the mesh and geometry are fixed at the same time, and instead,uses a level set approach to combine and trim elements to generate the desired cross sections (Carangan, 2018;
Carangan & Quinay, 2018).
For the dynamic analysis, the displacement, 𝒖𝒏+𝟏 at time, {n+1} is solved using the Equation (1):
𝟐

𝟒

𝟐

𝟒

𝟒

(𝑲 + ∆𝒕 𝑪 + ∆𝒕𝟐 𝑴) 𝒖𝒏+𝟏 = (∆𝒕 𝑪 + ∆𝒕𝟐 𝑴) 𝒖𝒏 + (𝑪 + ∆𝒕 𝑴) 𝒖𝒏̇ + 𝑴𝒖𝒏̈ + 𝒇𝒏+𝟏

(1)

where, 𝑲, 𝑴, 𝑪 are the stiffness, mass, and damping matrices. The vectors { 𝒖}, {𝒖̇ }, {𝒖̈ } and {𝒇} refer to thedisplacement, velocity, acceleration, and external force vectors, respectively. ∆𝑡 is the time increment and 𝑛 is
the time step. The boundary conditions are time-varying displacements inputted at the bottom of the model.
Viscous absorbing boundaries were placed at the bottom and side surface of the soil part of the model to truncate
the infinite domain.
2.3 Implemented computational techniques
All developed tools were coded in low-level programming languages (C and Fortran). The computations
were designed wherein the building data can be partitioned according to the number of available processors. To
achieve this, distributed-memory approach through Message Passing Interface (MPI) was implemented. Thus, allmodel generation (line and solid elements), analysis, and postprocessing, are done in parallel. A simple partitioningof building model data based on the number of nodes was used to reduce the imbalance in computation load foreach processor, and also to achieve almost the same runtime.

319

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 1. Procedure for the static modeling approach

Figure 2. Procedure for the dynamic modeling approach

3. VALIDATION OF GENERATED ANALYSIS MODELS
Validation of the models generated by static analysis code was conducted by comparing the resulting
displacement with that of a commercial software (CS) used for structural analysis and design. For the load settings,
the static earthquake loads used in CS and FEM are the same. Figure 3 shows the displacements and deformed
shape of the models. For a 5-story structure, the difference in the displacement values, with respect to CS results,
ranged from 0.20 – 5.73% for the top-floor nodes.

320Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 3. Comparison of results of static analysis code developed in this study and commercial structural
analysis software

Validation for the developed dynamic analysis code was also conducted by comparing the results forthe displacement (in frequency domain) between the code and CS. The same input dimensions and material(concrete) was set. Multiple cases with input loadings of Ricker wavelets of increasing center frequency wereconducted. The time-varying displacement results are then postprocessed by Fast Fourier Transform to determine
the amplitude-frequency distribution corresponding to each loading case.
Figure 4 shows the analysis models and a comparison of a nodal response at the top floor. The graphshows the results after inputting a Ricker wavelet with 1.1 Hz center frequency. Red line is the response of the 3Dmodel generated by this study, and the black line is the response of the CS model. In all the tests, the obtained
maximum relative difference of results is 7.91%.

Figure 4. Analysis models and comparison of results of dynamic analysis code developed in this study andcommercial software (CS) (red line is computed by this study while black line is computed by CS).

4. DEMONSTRATIVE EXAMPLES
Two cities in Metropolitan Manila were considered for a scenario earthquake analysis. A Mw 7.2
hypothetical earthquake was set to compute for the input seismic load for each building model. For this study, only
low- to mid-rise concrete structures in the city were considered. Table 1 shows distribution of this building material
type with varying floor levels in the two cities. City 1 has a total of 3,036 structures, while City 2 has 924. For

321

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

generating the analysis models, we used the information from GIS dataset output of GMMA READY Project 2013.
Templates of building models corresponding to those present in the GIS data were generated. To compute for the
input floor horizontal loading, we used the NSCP 2015 static force procedure. An important parameter indetermining the lateral load is the natural period of the building. The magnitude of the floor loading includes the
selfweight of the structure as well as superimposed loadings.
Figure 5 shows the results of the analysis. For City 1, the obtained maximum displacement is about 6 cm.
While for City 2, the maximum is about 3 cm. Post computations show that the frame models exhibited higher
displacements and story drifts with increasing floor height. Moreover, as shown for City 1, there are clustering ofbuildings that obtained displacements ranging from 1.5 cm to 3 cm. Whereas, in City 2, buildings with similar
displacements are more scattered throughout the whole city.
Table 1. Distribution of building floor levels of the two cities
No. of floors
No. of buildings
City1
City 2
2
2,209
838
3
559
72
4
192
13
5
60
0
6
16
1

Figure 5. Result of scenario earthquake modeling: (a) visualized raw GIS data ; (b) generated building models
and response to the scenario earthquake
To check the applicability of the developed dynamic analysis code, a three-story school building in Metro
Manila, which was previously instrumented for a study in determining the period of vibration (Kubo et al., 2004),
was considered for analysis. Given the standard floor plan and structural design, the analysis model (structure andsoil) was constructed (see Figure 6). The model was subjected to multiple inputs of Ricker wavelets of increasing
center frequency. Table 2 shows the comparison of the period derived from experiment and computed in this study.
As shown, the obtained period for the longitudinal direction closely matched with that of the experiment. However,in the case of the transverse direction, there is a large difference (33.8%), which is attributed to possible
inconsistencies in material properties and member geometries of the analysis model and the constructed building.
Since the model generated in this study is based on the standard design for that particular building type, the actual322

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019settings of the building as constructed in the field remains to be further verified.

Figure 6. The finite element model of the thee-story building used for validation test of dynamic analysis code

Table 2. Periods derived from experiment and computed in this study
Direction
Experiment (s)Computed (s)Difference (%)(Kubo et al., 2004)(This study)
Longitudinal
0.24
0.22
6.2
Transverse
0.15
0.20
33.8

5. CONCLUSIONSThis study aimed to develop static and dynamic modeling approaches for city seismic response analysis.
For both modeling approaches, tools were developed to generate and analyze models applicable to the available
GIS and BIM data.
To check the accuracy of the developed tools, validation tests were conducted wherein the results
obtained from these tools were compared with those of commercial software that are commonly used in structural
analysis and design. Validations for static and dynamic analysis show that the results of the developed tools arewithin 6% and 8% of the results that were computed using a commercial structural analysis and design software,
respectively.
As a demonstrative example, two cities in Metro Manila were considered for scenario earthquake
analysis. Low- to mid-rise reinforced concrete frame structures were analyzed and floor displacements werecomputed. From the results, maximum story drifts were computed, and visualized in city-level to compare theresponses of the two cities. Another example conducted was on the computation of the period of vibrations of athree-story building in Metro Manila. Results show that for the considered standard model of a three-story building,the periods of vibration can be closely estimated.
The developed modeling approaches are continuously being improved to be capable in handling thecomputation of other structure types, such as tall buildings with dual systems, hybrid, and makeshift structuresthat are local to Metro Manila cities. Moreover, with many buildings in the cities that are now being installed withsensors for health monitoring, the recorded data can be used to validate the analysis models.
ACKNOWLEDGMENTS
This research was carried out in part using the CoARE Facility of the Advanced Science and Technology
Institute and the Computing and Archiving Research Environment, Diliman, Quezon City. The demonstrative
application part of this study has received technical assistance from DOST-PHIVOLCS Project, FEATURE:
Feature-based Earthquake Analysis Toolset for Urban Response Estimation.
REFERENCES
Bautista, M L P, Bautista B C, Narag I C, Melosantos M L P, Lanuza A G, Papiona K, Enriquez M C, Salcedo J
C, Perez J S, Deocampo J B, Punongbayan J T, Banganan E L, Grutas R N, Olavere E A B, Hernandez
V H, Tiglao R B, Figueroa M, Solidum R U, Jr., Punongbayan R S. (2011). The Rapid Earthquake
Damage Assessment System (REDAS) Software. Proceedings of the Ninth Pacific Conference on
Earthquake Engineering: Building an Earthquake-Resilient Society, 12-16 April, Auckland, New
Zealand.
Carangan F M, Quinay P E (2018) Development of an automated soil-structure model generation tool for large
scale 3D dynamic finite element analysis. In Proceedings of 8th Regional Symposium for Infrastructure323

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Development in Civil Engineering (RSID 2018), UP Diliman, Quezon City, October 25-26.
Carangan, F M (2018) Development of an octree-based tool for generation of three-dimensional structure models
using PSM Approach, Undergraduate Research Project, University of the Philippines Institute of Civil
Engineering
Fader A R, Quinay P E (2018) Developing a structural analysis procedure utilizing BIM and HPC for application
to large scale urban seismic response estimation. Proceedings of the 1st International Conference on
Concrete and Steel Technology Engineering and Design (CASTED 2018), May 24-26, Manila,
Philippines.
Fujita K, Ichimura T, Hori M, Maddegedara L, Tanaka S. (2015) Scalable multicase urban earthquake simulation
method for stochastic earthquake disaster estimation, Procedia Computer Science, 14th International
Conference on Computational Science 2015; 51: 1483-1493.
GMMA READY Project 2013. Hazards Mapping and Assessment for Effective Community Based Disaster Risk
Management, Phase II. Project implemented by National Disaster Risk Reduction and Management
Council-Office of Civil Defense – Collective Strengthening of Community Awareness on Natural
Disasters Agencies (NDRRMC-CSCAND), administered by United Nations Development Programme(UNDP) and funded by Australian Government through the Australian Agency for International
Development (AusAID).
Homma S, Fujita K, Ichimura T, Hori M, Citak S, Hori T. (2014) A physics-based Monte Carlo earthquake
disaster simulation accounting for uncertainty in building structure parameters. Procedia Computer
Science, 14th International Conference on Computational Science 2014; 29: 855-865.
Hori M, Ichimura T, Wijerathne L, Ohtani H, Chen J, Fujita K, Motoyama, H (2018) Application of highperformance computing to earthquake hazard and disaster estimation in urban area. Frontiers in Built
Environment Ed. By Katsuichiro Goda, Vol. 4, Article 1, pp. 1-13.
Ichimura T, Hori M, Bielak J. (2009) A hybrid multiresolution meshing technique for finite element threedimensional earthquake ground motion modeling in basins including topography, Geophysical JournalInternational, 177: 1221-1232.
Kubo, T., Arai, H., Aono, M., Achiwa, T., Bautista, B., Lanuza A. (2004) Seismic vulnerability evaluation ofurban structures in the Metro Manila Part 4: Evaluation of seismic performance of school building”. In
Proceedings of Asia Conference on Earthquake Engineering – Manila, Philippines.
NSCP (2015) National Structural Code of the Philippines 2015 Volume I Buildings, Towers, and Other Vertical
Structures, Association of Structural Engineers of the Philippines, Inc.
Quinay P E, Ichimura T (2016) An improved fault-to-site analysis tool towards fully HPC-enhanced physicsbased urban area response estimation. Journal of Earthquake and Tsunami, 10, (5).
Quinay P E, Grutas, R, Ichimura T, Bautista B, Hori, M (2018) A two-step procedure for estimation of seismic
response of urban areas in the Philippines. 11th U.S. National Conference on Earthquake Engineering,Los Angeles, California.

324

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Structural Shape Grammars used in Intelligent Generation Design of Discrete
Structures
Xianzhong Zhao1, Ruifeng Luo 2
1) Ph.D., Prof., College of Civil Engineering, Tongji University, Shanghai 200092, China. Email: x.zhao@tongji.edu.cn.
2) Ph.D. Candidate, College of Civil Engineering, Tongji University, Shanghai 200092, China. Email: lrf@tongji.edu.cn

Abstract: The intelligent construction industry involves three aspects: intelligent generation design, intelligent
fabrication, and intelligent operation and service. A conceptual and intelligent generation design method of discrete
structures was introduced. The method, based on the computational synthesis, is capable of generating an optimal
structure from all alternative ones at the concept design stage to meet the optimization objectives and constraints.
This generation method contains three parts: structural shape grammars, structural evaluation, and structural
optimization algorithm. Structural shape grammars in this method can generate, optimize and transform discrete
structural shapes using geometric topology within the design domain. The characters of structural shape grammars
were obtained through existing shape grammars. An innovational tetrahedral shape grammar was presented aiming
to improve the adaptability of structural shape grammar in spatial issues. Through classifying the topologicalrelations of five points in space, the tetrahedral shape grammar determines not only the topological transformation
rules but also the nodes adding and deletion rules of discrete structure. Besides the topological transformation
method, this tetrahedral shape grammar also takes the shape transformation and size transformation into
comprehensive consideration. Besides, prospective applications of intelligent generation design method were put
forward.
Keywords: intelligent generation design method, spatial shape grammar, tetrahedron topology.
1. INTRODUCTION
In the rapid development era of information and digitization, construction labor productivity has not kept
pace with the overall economic productivity. The construction industry is among the least digitized according to
McKinsey global institute industry digitalization index(Agarwal et al. 2016), only higher than agriculture andhunting. Thus, the concept of intelligence construction was made for realizing digitization in design, fabrication,and operation during the whole construction life-cycle. The intelligent construction industry involves three aspects:
intelligent generation design, intelligent fabrication, and intelligent operation and service. In this paper, theintelligent generation design method is mainly focused. Based on computational synthesis (Shea & Cagan, 1999),
this intelligent generation method contains three parts: structural shape grammars, structural evaluation, andstructural optimization algorithms, which are capable of generating the optimal structure from all alternative onesat the concept design stage to meet the optimization objectives and constraints.
Structural shape grammar is a semantic algorithm for topological transformation, which can generate,
optimize and transform discrete structural shapes using geometric topology within the design domain for
optimization. The grammar, consisting of a series of shapes and shape combination rules, defines a language of
discrete structures by specifying design transformations that implicitly represent form-function relations(Campbell & Shea, 2014). For discrete structures, the structural shape grammar links the structure shape with theoverall structural performance through a series of precisely set shape transformation rules to realize the calculationand synthesis of structures. Previous studies have mostly focused on generation two-dimensional structures, forthe complexity and multi-solution of the three-dimensional problems. The general approach for generating spatial
structures is based on the ground structure method (Dorn, 1964), while the spatial grammar rules are rarely
mentioned for discrete structures. Compared with the conventional structural generation optimization method,
methods based on shape grammars can more comprehensively consider the comprehensive factors affecting thestructural form, which is gradually introduced into the industrial design field for industrial products design(Suppapitnarm et al., 2004; Zimmermann et al., 2018;).
Therefore, through classifying the topological relations of five points in space, an innovational tetrahedral
shape grammar was presented aiming to improve the adaptability of structural shape grammar in spatial issues.
The tetrahedral shape grammar determines not only the topological transformation rules but also the nodes adding
and deletion rules of discrete structure.
2. THE METHOD OF INTELLIGENT GENERATION DESIGN
The intelligent generation design method is developed from structural topology and shape annealing
algorithm (Shea & Cagan, 1999), which can find the most suitable structure under certain criterions. The generation
method is composed of structural shape grammar, structural evaluation, and structural optimization algorithm andits basic flow chart is shown in Figure 1.
The generation method takes the simple initial shape as the basic element, generates new shapes through325

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019the rules defined by preset shape grammars. All possible design schemes will be included through the combinationof the shape grammar rules. As for the structural evaluation, design objectives and constraints are given according
to the specific design information and requirements, combining effects of safety, economy, ease of construction
and architectural elegance, through structural analysis and other relevant structural indicators. As for the structural
optimization algorithm, it is a kind of heuristic algorithm, and the simulated annealing was chosen in these
problems. The heuristic algorithm has some differences in the optimization mechanism but has great similarity inthe optimization process, which is a kind of "neighborhood search" structure. It starts from an initial solution,
generating several neighborhood solutions through neighborhood functions under the control of the criticalparameters of the algorithm, updating the current state according to the acceptance criteria, then modifying thecriteria according to the critical parameters. The above search steps are repeated until the convergence criterion of
the algorithm is satisfied, and finally, the best design and the process data is obtained.

Figure 1. Flow chart of the intelligent generation design method
3. STRUCTURAL SHAPE GRAMMARS OF SPATIAL STRUCTURES
Plane structural shape grammar rules of section size, geometric position, and topological transformation
have been constructed to generate planar truss structures and single-layer reticulated shell structures (Wang &Zhao, 2011). It has also been extended to the space truss beam structure (pseudo-3D) subjected to two-dimensional
forces (Zhao and Shea, 2010). However, for the more widely used and general spatial structures, complete
grammar rules have not been formed. Therefore, the focus of this paper is to establish the shape grammar rules ofthe spatial structure with tetrahedron as the essential element, preparing for the later structural intelligent design.
2.1 Features of structural shape grammars
From the existing planar grammar and pseudo-3D structural shape grammars, the following two key
features should be considered in the establishment of structural shape grammar rules.
(1) Geometric completeness: The rules of shape grammar satisfy the reversibility to ensure the completeness ofgeometric topological deformation of the generated structure, to optimize to the optimal solution that meetsthe objectives and constraints under the guidance of heuristic algorithm. That is, there are both "add" rules
and "subtract" rules, and the rules should be reversible and in pairs to avoid the complex and complicatedlocal optimal solution.
(2) Mechanical completeness: Spatial structures such as space trusses are generally hinged structures. The
generated structures must satisfy the mechanical principle of geometric invariability. For two-dimensional
plane problems, Maxwell criterion can be used to facilitate the implementation. For three-dimensional
problems, it is difficult to embed the criterion directly. At the beginning of grammar establishment, geometricinvariance of grammar rules should be fully considered.
2.2 Tetrahedral shape grammar rules
In three-dimensional space, four points that are not coplanar can form a tetrahedron. Among many
stereoscopic figures, tetrahedrons are the simplest three-dimensional figure (3-simplex), and each face consists oftriangles, which is similar to the two-dimensional problem, so the tetrahedron is chosen as the basic element of the𝑎, 𝑏, 𝑐, 𝑑, 𝑒 , and consider the possible spatial
spatial shape grammar. Suppose there is a spatial point set 𝑃
topology of five point structure, and the result is shown in Figure 2.

326

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 2. Classification of five-point structure
When any four points are not coplanar, the topology of spatial point set P5 can be divided into two
categories. One is two kind convex hull hexahedrons, and the other is convex hull tetrahedron. In hexahedron A,
two sub-tetrahedrons T-abcd and T-ebcd are formed (see Figure 3a); In hexahedron B, three sub-tetrahedral Tabce, T-acde and T-abde are formed (see Figure 3b). In convex tetrahedron t-abcd, point e is embedded into thetetrahedron, and four sub-tetrahedrons, T-abcd, T-dced, T-aebd, and T-bced are formed. (see Figure 3c).

Figure 3. Any of four points are not coplanar topology
When there are four points coplanar, the complex three-dimensional problem can be transformed into aplane problem. As shown in Figure 4, if a, b, c, and d are coplanar, two-dimensional triangulation can be carriedout, and then the points are connected with e to form two (see Figure 4a, Figure 4b) or three (see Figure 4c)
tetrahedrons.

Figure 4. Four points are coplanar topology
Based on the above classification of spatial five-point structure topology, a tetrahedral space shapegrammar is proposed, which includes three kinds of rules: section, shape, and topology. The shape modification
rule can change the location of a single point, and the size modification rule can change the cross-sectional area of
each member. The topology exploration can be realized by the topology modification rules, which contains threepairs of rules and are all reversible. The details are shown in Figure 5, and the illustrative applications of topology
modification rules are shown in Figure 6.
Rule 1 is to add or subtract one point inside the tetrahedron, while rule 1.1 and rule 1.2 are special cases
when four points are coplanar. These two rules can avoid the appearance of thin tetrahedrons as adapting rule 1,for the thin tetrahedrons always have poor mechanical properties. Rule 2 is to add or subtract one point outside thetetrahedron while Rules 2.1 and 2.2 are also the situation of four points coplanar. Rule 3 is a unique rule in threedimensional problems. The same distribution of points has different topological configurations due to different327Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019connections. These three types of rules are combined stochastically with each other to generate the tetrahedral
topology that satisfies the objectives and constraints.

Figure 5. Tetrahedral shape grammar
11

rule 1.2

2

2

4

4

3

5

3

1
6

rule 2.2

2
4
5

3

1
2

1

rule 1.2

rule 1.2
6

2

4
3

4

5

3

5

Figure 6. The illustrative applications of topology modification rules

328Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

When solving three-dimensional practical problems, the feasible solution space is enormous. On the one
hand, the heuristic optimization algorithm can reduce the search time; on the other hand, for specific problems tobe solved, some additional requirements are added to the tetrahedral shape grammar, which can significantly
reduce the random search range and improve the optimization efficiency. For example, when generating spacetruss structures, additional requirements can be applied such as: (1) the Specified fixed points and fixed edges (2)the minimum length of truss members; (3) the minimum angle between members; (4) the minimum volume
proportion of the new generating tetrahedron and the original tetrahedron and other reasonable requirements.
4. CONCLUSIONSFor discrete structures, an innovational tetrahedral shape grammar was presented aiming to improve theadaptability of spatial shape grammars in this intelligent generation method based on computational synthesis.
This tetrahedral shape grammar makes structural modification in shape, size and topology level, which improve
the generation and optimization efficiency of discrete spatial structures. This grammar can be widely used, notonly in the generation of spatial structures but also in the design of the metamaterial micro-unit design or someindustrial products. Next, the tetrahedral approach needs to be practiced in combination with specific cases,
considering the specific design constraints, and objectives in later research.
REFERENCES
Agarwal, R., Chandrasekaran, S., and Sridhar, M. (2016). Imagining construction’s digital future. Mckinsey
insights Report, Mckinsey & Company and Mckinsey Global Institute, Jun.
Dorn, W. (1964). Automatic design of optimal structures. J. de Mecanique, 3, 25-52.
Königseder, C., & Shea, K. (2014). Systematic rule analysis of generative design grammars. Artificial
Intelligence for Engineering Design, Analysis and Manufacturing, 28(3), 227-238.
Shea, K., and Cagan, J. (1999). Languages and semantics of grammatical discrete structures. Artificial
Intelligence for Engineering Design, Analysis and Manufacturing, 13(4), 241-251.
Suppapitnarm, A., Parks, G. T., Shea, K., and Clarkson, P. J. (2004). Conceptual design of bicycle frames bymultiobjective shape annealing. Engineering Optimization, 36(2), 165-188.
Wang R., and Zhao X. (2011). Shape grammar driven configuration synthesis of spatial truss structures. Journalof Computer-Aided Design & Computer Graphics, 23(11), 1924-1930.
Zhao X., and Shea, K. (2010). Intelligent generation and design of spatial truss structure. Journal of Building
Structures, (9), 63-69.
Zimmermann, L., Chen, T., and Shea, K. (2018). A 3D, performance-driven generative design framework:
automating the link from a 3D spatial grammar interpreter to structural finite element analysis andstochastic optimization. Artificial Intelligence for Engineering Design, Analysis and Manufacturing,32(2), 189-199.

329

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Numerical Evaluation of Seismic Response of Anchorage Foundation installed in
Switchboard Cabinet
Sang-Moon Lee1, Ga-Ram Kim2, Woo-Young Jung3
1) Ph.D. Candidate, Department of Civil Engineering, Gangneung-Wonju University, Gangneung-si, Gangwon-do, South
Korea. Email: idealmoon@naver.com
2) Master Student, Department of Civil Engineering, Gangneung-Wonju University, Gangneung-si, Gangwon-do, South Korea.
Email: kgr3288@naver.com
3) Professor, Department of Civil Engineering, Gangneung-Wonju University, Gangneung-si, Gangwon-do, South Korea.
Email: woojung@gwnu.ac.kr

Abstract: In this study, the seismic response of anchor bolt for freestanding equipment, such as switchboard
cabinets, at the hydroelectric power plant was presented based on the results of numerical simulations. From theexperimental study, shaking table tests were performed to investigate the overall structural behavior of switchboard
cabinets and seismic-induced damages including a rocking problem that leads to the deformation occurred at thecabinet bottom were observed during the test. The FE modeling was conducted by using the ABAQUS similar toan actual cabinet panel and 3D dynamic nonlinear analysis was performed using seismic loadings like long-period
earthquakes. For validating the proposed FE model, the maximum displacement and the mode shapes were usedin the comparison of numerical analysis results and a reasonable agreement was made when seeing in the maximum
displacement of the switchboard cabinets. From the analysis, a slight difference was found in the analysis due tothe rocking problem which causes cup-like deformation at the bottom of the cabinet during the tests. Finally, using
the validated numerical model, the effects of long period earthquake wave on cabinet behavior was analyzed. The
Von-Mises stress at the anchor bolt embedded in concrete foundation occurred 1.068 MPa and the horizontal
maximum displacement of the cabinet was 24.8 cm. From the basis of this research, there is needed to compareinterrelation between nonstructural components such as a cabinet and the seismic waves which have various period
in the future research.
Keywords:

Switchboard Cabinet, Shaking Table Test, Seismic Waves, Nonstructural Components

1. INTRODUCTION
1.1 Earthquake Damage of the Nonstructural Components
Since an earthquake occurs in a short time, there is notenough time to actively take action when the earthquake occurs,
resulting in a greater loss of life or damage to property than othernatural disasters. The frequency of earthquakes with such a
danger has been increasing in recent years, and damage is also
increasing. Not only the damage to the entire facility was a
problem, but also the damage to nonstructural components
installed in the affected facility were also found to significantly
disrupt the performance of the entire facility. However, in thedomestic construction structural standard, which includes theseismic design criteria are not clearly defined for the mechanicaland electrical non-structural elements of these power generation
facilities.

Fig. 1. Earthquake damage of thenonstructural components

2. NUMERICAL MODEL VERIFICATIONIn this study, the test and the analytical results were compared with each other to verify the reliability of
the numerical model. The earthquake loads were generated by using the shaking table and its mechanism can be
confirmed the model by using test data and then, there is analyze the behavior characteristics of an anchorage
connecting electric cabinet subjected to earthquake loading by using the numerical analysis. Ultimately, thereliability of the numerical analysis simulating the switchboard cabinet based on the Abaqus element was verified
by comparing it with the test results.
2.1 Experimental Simulation Method
2.1.1 Description of Geometry
The specifications of the cabinet used in the study are shown in Table 1. The Fig. 2 shows the overall
setup for the experiment, and the cabinet modeling by using Abaqus program based on the Table 1.

330

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Specimen
CabinetLengthSingle door

800

Table 1. Test specimens specifications
Dimension (mm)Weight(kg)
Width
Height
800

2,350

480

Boundary(Fixed)M16 Anchor(8ea)

Figure 2. The whole system regarding test and, component of 3D-Model by using Abaqus program
2.1.2

Checking Resonance Frequency
A resonance search experiment was performed with a sine sweep test to check the possible structuraldeformations before and after the experiment. It was performed once for each axis direction (X), (Y), and (Z) andthe magnitude of the acceleration signal was 0.07g in order to minimize the damage of the cabinet. The searchrange was 1 ~ 50Hz and the frequency increase was 2 octave / min. The procedure of the shaking table experimentis shown in below Table 2.
2.1.3

Resonance Search and Time History Test
As shown in Fig. 3, the TRS (Test Response Spectrum) obtained a response measured after the time history
experiment by installing an accelerometer for three directions (X, Y, Z) on the bottom of the shaking table.
Table 2. Overall test sequence
Classification

Step
1
2
3
4
5
6
7
8
9

X
Y
Z
Reg.160
UHS
AC 156X
Y
Z

Pre-resonance search test

Time history test

Post-resonance search test

Figure 3. Sensor location
2.2 Analysis Simulation Method
2.2.1 Material Properties
As shown in below Table 3 presents the material properties applied in the numerical analysis. The
Damping value is obtained proceeding Eigenvalue-analysis by using Abaqus program and then the values required
for the numerical analysis were calculated by using Matlab program.

Material
Steel (SS400)

Table 3. Test specimens specifications
Elastic
Density(ton/mm)
Young’s Modulus
Height7.85e-09
2.1e+050.3

2.2.2

Damping (5%)
Alpha
Beta
3.57756.99e-04

Interface Mechanism
The interface between cabinet bottom and top of the jig was designed to have the sliding effect by using
contact surfaces. In addition, the contact mechanism was applied to the anchor used to connect between the cabinet
and the jig by using Abaqus function.

331

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2.3 Results Comparison and Model Verification
2.3.1 Natural frequency
The resonance frequency of the cabinet was checked and the resonance search experiment was performed
to check the structural changes and the dynamic characteristics during the experiment. It was performed at a levelof 0.07 g from 1 Hz to 50 Hz, and as a result, a natural frequency value of 15.30 Hz was obtained. The natural
frequency value obtained through the numerical model is 16.56 Hz, and it is found that there is some difference
from the experimental data. It is that the main reason of the difference value is not considered completely for amass of the actual cabinet about the numerical modeling.
2.3.2

Displacement
The below Fig. 4 (a) shows the data obtained by using the displacement response of the LVDT(Linear
Variable Differential Transformer) installed at the top (D1) and bottom (D2) position of the cabinet as a transfer
function and that is a showing the maximum displacement value of the cabinet under the seismic wave. The Fig.
4 (b) shows the maximum displacement which obtained by using numerical analysis for the same conditions.

(a)

(b)

Fig. 4. The relative displacement of cabinet: (a) test results: 1.23mm (b) analysis results: 0.402mm
For validating the proposed FE model, the maximum displacement and the mode shapes were used in thecomparison of numerical analysis results and a reasonable agreement was made when seeing in the maximum
displacement of the switchboard cabinets. From the analysis, a slight difference was found in the analysis due tothe rocking problem which causes cup-like deformation at the bottom of the cabinet during the tests.
3. ANALYSIS OF VULNERABILITY
3.1 Vulnerable Characteristics under the Seismic Waves
Using the validated numerical model, the effects of long period type of earthquake wave on cabinet
behavior was analyzed. The below Table 4 shows the amplitudes of the earthquakes applied to analyze thevulnerability of the cabinet to kind of each earthquake.

PeriodTable 4. Applied amplitude and analysis results
Applied amplitude
Applied to analysis amplitudeLong(El centro)

3.2

Results and Analysis
The below Table 5 shows that is the results about effect of long period of earthquake on the anchors
supporting and the overall behavior of cabinet. The Von-Mises stress at the anchor bolt embedded in concrete
foundation occurred 1.068 MPa and the horizontal maximum displacement of the cabinet was 24.8 cm.

PeriodTable 5. Influence regarding long period of earthquake condition
Maximum StressMaximum DisplacementLong(El centro)
1.068 MPa332

2.481e+02 mm

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

4. CONCLUSIONS
From the results, it can be seen that the load on the anchor supporting the cabinet does not significantly
increase, but on the contrary, it has a great influence on the overall cabinet movement. It is expected that theflexible stiffness of the cabinet resulted in more structural weakness in a long - period earthquake wave. From thebasis of this research, there is needed to compare interrelation between nonstructural components such as a cabinet
and the seismic waves which have various period in the future research.
ACKNOWLEDGMENTS
This research was supported by a grant (19IFIP-B128598-03) from Industrial Facilities & Infrastructure
Research Program (IFIP) funded by Ministry of Land, Infrastructure and Transport of Korean government.
REFERENCES
FEMA E-74. (2012). Reducing the Risks of Nonstructural Earthquake Damage – A Practical Guide.
ICC-ES. (2015). AC156-2010 Acceptance Criteria for Seismic Certification by Shake-table Testing of
Nonstructural Components.
REGULATORY GUIDE 1.60 DESIGN RESPONSE SPECTRA FOR SEISMIC DESIGN OF NUCLEAR
POWER PLANTS. (2014).
Jieun Hur. (2012). Seismic Performance Evaluation of Switchboard Cabinets Using Nonlinear Numerical
Models.
Rhee, H. M., Kim, M. K., Sheen, D. H. and Chol, I. K. (2013). Analysis of Uniform Hazard Spectra for
Metropolises in the Korean Peninsula, Journal of the Earthquake Engineering Society of Korea, Vol. 17,
No. 2, pp. 71~77.

333

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

IoT, Sensors, and Monitoring334

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

FRAMEWORK FOR A BIM-BASED REAL-TIME EVACUATIONGUIDANCE SYSTEM IN SMART BUILDINGS
Kayla Manuel1, Nobuyoshi Yabuki2, and Tomohiro Fukuda3
1) Master Course Student, Division of Sustainable Energy and Environmental Engineering, Osaka University,
Japan. E-mail: manuel@it.see.eng.osaka-u.ac.jp
2) Ph.D., Prof., Division of Sustainable Energy and Environmental Engineering, Osaka University, Japan. E-mail:yabuki@see.eng.osaka-u.ac.jp
3) Ph.D., Assoc. Prof., Division of Sustainable Energy and Environmental Engineering, Osaka University, Japan.
E-mail: fukuda@see.eng.osaka-u.ac.jp
Abstract: In emergency situations, such as a fire, mass panic can result in many unnecessary deaths. Some deaths
are caused not as a result of the fire, but due to the herds of people seeking egress. If alternative exits were madeknown to those seeking egress, congestion at prominent exits could be alleviated and unnecessary deaths
prevented. Current two-dimensional evacuation signs in buildings show a single means of escape and are
insufficient because they are static and relying on them for information is inconvenient in a time-sensitive situation.
A Building Information Modeling (BIM) model is a semantically rich three-dimensional representation of abuilding with geometric and semantic data. Integration of this built environment information and Internet ofThings (IoT), termed SMART buildings, are commonly used for operation and maintenance purposes. The
proposed system integrates BIM and IoT to create an evacuation guidance system to assist users in building egress.
The proposed system continuously monitors the current fire situation, the building and its’ occupant in order toprovide real-time customized instruction from each user’s respective location/s to safety. Moreover, should theirinitial path of escape be obstructed by dynamic events, such as obstructions caused by an escalated fire orinoperative doors, the pre-simulated paths would suggest an alternative route that an egress-seeker should take.
Because visibility is low, due to smoke density, egress instruction is given as voice commands to customized
wireless earphones so that users can be hands-free to crawl, should the need arise. The system goes through adatabase of evacuation paths and based on proximity and other factors the evacuation instruction is relayed. The
proposed system relies heavily upon knowing where occupants are in the building at all times in order toimmediately instruct them from their respective initial positions to a designated exit.
Keywords: Building Information Modeling (BIM), Egress, Dynamic Real-time navigation, Indoor Localization,
Smart Buildings, Internet of Things (IoT).
1. INTRODUCTION
In an emergency situation, visitors and those who frequent the building, such as employees andmaintenance staff, may find it difficult to escape. Aside from users being disoriented amidst the chaos, they may
not be aware of certain exits in the building, such as service exits (Chu et al., 2015). Under normal circumstancesthese areas are restricted for obvious security reasons. However, during an emergency situation, where human
lives are priority, knowing where these exits are can alleviate congestion at prominent ‘front-of-house’ exits.
Lujak et al., (2017) proposed a route guidance system that divides occupants from each area of each floor equally
amongst the available exits to alleviate bottlenecks.
Traditionally, singular instruction is given to all evacuees under the assumption that users develop a
crowd-like mentality during an emergency situation (Lujak et al., 2017). Signage displaying evacuation plans are
installed in static locations and can easily be occluded by crowds or smoke. These signs provide a single
evacuation path because under extreme conditions, information overload can otherwise paralyze users. In theevent that this singular path suggestion becomes inaccessible users are left without alternatives. Other fire signage
in a building, although installed at ceiling heights and their luminescence is useful in the dark, their existence isfutile once concealed by smoke. Previous studies suggest retrofitting signs at more appropriate heights, whichcomes at an addition cost to company (Motamedi et al., 2016). Physical signage can easily be destroyed orconcealed in the fire whereas virtual signage in the BIM model is unaffected. Luminescent skirting found at an
appropriate height can also be utilized for wayfinding, however, the direction in which one should travel is notindicative. An intercom system broadcasting evacuation instruction is easily muffled by fire-related noise
pollution. One could argue that the proposed system faces the same vulnerability, however, the proximity of theearphones to the user is much closer than that of the intercom speaker system. Moreover, the proposed systemprovides instructions in the user’s preferred language thereby resolving any language barriers.
Each occupant’s location and characteristics vary, so too should their evacuation instruction. The
proposed system will rely heavily upon knowing occupants are in the building in order to immediately instructthem from their respective locations to the exits. Extensive research has been conducted to improve Real-time
Location Systems (RTLS) which track the indoor location of users in a building (Siddiqui et al., 2014; Atila et al.,335

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 20192018; Zadeh & Rüppel, 2013). The proposed system utilizes Bluetooth Low Energy (BLE) beacons to track users’
locations in the building through their respective mobile devices. Although further advancement in indoor location
accuracy is needed, by installing multiple beacons and triangulation, tracking accuracy can be further enhanced(Zafari et al., 2019; Zuo et al., 2018). Distinguishing characteristics of users include; age, gender, dependants,
body size, respiratory capacity and mobility, amongst others. Physical disabilities which can hinder a user’s
evacuation ability include occupants who are visually-impaired or wheelchair-bound. Health-related issues
affecting evacuation ability include respiratory capacity which can lower an occupants chance of survival if they
suffer from asthma or emphysema (Atila et al., 2018).
2. LITERATURE REVIEWExtensive research has been conducted to improve current emergency evacuation navigation systems
integrating IoT and BIM systems (Cheng et al., 2017; Wang et al., 2014; Ahn & Han, 2012; Wu & Chen, 2012).
Although most studies have similar objectives, in the majority of research reviewed BIM semantic properties areusually overlooked. Aside from providing 3D visuals and a platform on which to perform simulations, semantic
information in BIM models can be advantageous in designing an evacuation guidance system.
Chen et al., (2018) developed a system to report on the legitimacy of a fire emergency by combining BIM
and IoT processes. After verifying that there is a fire emergency, customized smoke detectors illuminate the path
to be navigated. In the aforementioned study, however, the role of BIM was limited to geometry and semantic
properties were overlooked.
Several studies exist wherein BIM-based simulations are used to improve fire-related practices. Zadeh(2010) developed a system using geometry and material information of building elements in BIM as boundary
conditions to generate potential escape routes. Fire safety engineers would use the created system to modify andtest the feasibility of proposed escape routes in a virtual reality environment to evaluate them visually. Althoughsemantic BIM information in conjunction with Fire Dynamics Simulator (FDS) models were used in this study ittargeted fire safety engineers, whereas the proposed system is intended for use by egress-seekers.
Wang et al., (2014) created a system comprising of four modules; evacuation assessment, escape route
planning, safety education, and equipment maintenance. The equipment maintenance module made use of
semantic BIM information for monitoring the maintenance of fire equipment. However, in the module pertaining
to escape route planning geometric BIM information was solely used as boundary conditions for path creation.
Atila et al., (2018) proposed a ‘user-specific’ evacuation system comprising of three fundamental
functionalities; an intelligent dynamic routing engine, an indoor positioning system and a user-interactive mobile
application. The application proposed by the authors hereinafter SmartEscape, uses an Artificial Neural Network(ANN) in conjunction with sensor data and a spatial database to provide individual evacuation instruction
according to egress-seekers individual characteristics. Evacuation paths in SmartEscape are influenced by threemain factors; the users, the building and the current fire conditions in real-time. User characteristics considered inthis study were far more comprehensive than that of previous works. Aside from users having varying respiratory
and mobility traits, environmental factors were also considered such as temperature, carbon monoxide levels,
smoke density, human density, etc. After taking these factors into consideration, the shortest and safest route is
calculated for each egress member. In order to track user locations, Radio Frequency Identification Tags (RFIDtag) were used. Although RFID tag readers are not currently in mobile devices, the authors believe in future thiswill become a standard built-in function.
Although comprehensive, the SmartEscape system only used spatial constraints plotted by the RFID tag
coordinates. In the proposed system, however, by incorporating BIM with the inclusion of both geometric andsemantic data as boundary conditions more informed evacuation instruction is improved through additional
constraints such as materiality, building elements in relation to exits, etc. Moreover, the definition of ‘exits’ in the
proposed system not only refers to doors but windows as well, provided that they are at a suitable height for escape.
Under the assumption of a worst-case scenario, wherein all doors are blocked, it is imperative that an evacuation
plan include windows as an alternative exit.
3. PROPOSED SYSTEM OVERVIEW
The proposed system provides real-time navigation instruction to egress-seekers through modified
wireless earphones able to detect users’ head orientation. Initially, sensory data about the fire, building and theusers are broadcast from their respective IoT devices (Figure 1.1 and Figure 1.2). Thereafter, pre-simulated
evacuation paths are calculated using BIM model information, the A* pathfinding algorithm and user traits which
are then filtered according to various environmental factors such as temperature, smoke and crowd density (Figure
1.3 and Figure 1.4). The subsequent egress engine is then built out from the game engine to a tablet device whichacts as a server (Situation Monitor), from which all information related to the current fire situation can be viewed
in real-time (Figure 1.5). User devices act as a client receiving the path instruction based on their individual
characteristics as well as their immediate surroundings (Figure 1.6).

336

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 1. Overview of the Proposed System.
Table 1. Specifications of the hardware and software used in the proposed system in alphabetical order.
# COMPONENTS
SPECIFICATION
Hardware
Beacons
Estimote Proximity Beacons with Bluetooth® SoC, ARM® Cortex®-M4 32-bit
1
processor with FPU, 64 MHz Core speed, 512 kB Flash memory, 64 kB RAM
memory5 Computer
Windows 10 Education (64 bit)
6 Earphones
Akiki TWS- P10 True Wireless earphones
FOD (face
LP Research Motion Sensor Bluetooth version 2 (LPMS-B2) with 9-Axis Inertial
6
orientation device)
Measurement Unit (IMU) / AHRS with Bluetooth Classic and BLE Connectivity
6 Phone (user)
iPhone (IOS 12.0) and Android devices (minimum API level 24)
Tablet (situation
Samsung Galaxy Tablet S3. (Android Oreo: API level 28)
6monitor)
Software
3 Path Database
Multiple cached graphs within game engine. A* pathfinding algorithm.
BIM modeling
Autodesk Revit 20194software
BIM to IFC
Tridify and BIMImporter – Plugins to convert BIM to IFC for use in game engine
4
converter
5 Game Engine
Unity (version 2018.3.5f1) build Situation Monitor (SM) out to Tablet device
Online server
User locations and sensor data are sent via PHP to a MySQL database. The game2
engine queries this database for the most recent data.
Sensor Data
-Environment readings from beacons and other IoT devices
1-Simulated fire alarms and smoke detectors created within the game engine-Doors, windows and staircase enclosure doors are retrofit with status sensors.

337

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

4. DETAILING SYSTEM COMPONENTS
The sections to follow detail the proposed system components and are to be read in conjunction with thedetailed system architecture diagram as shown in Figure 2 below.

Figure 2. Detailed system architecture.
4.1 Sensor Data
For environmental data, beacons are used for two purposes; to provide each users location data as wellas temperature information of each area. In the prototype of the proposed system, initial temperature is measured
using actual beacons, however once the simulated fire event is created simulated data is used. Simulated data
includes temperature, fire alarms and smoke detectors created within the game engine. Door, window and staircase
enclosure doors will be retrofit with sensors to detect whether they are opened or closed.
4.2 Positioning System
The proposed system relies heavily upon knowing the location of each occupant in order to immediately
instruct them from their respective positions to their final exits. Sparsely placed beacons are used as a median totriangulate the position of users’ mobile devices. In order to track user location in real-time, as they navigate
through the building, data packets are periodically sent via PHP to a MySQL server. The game engine then queries
this MySQL database for the most recent position data and moves the users’ avatar within Unity to the equivalentvantage point (Davies, 2019). Object persistence is required so that when a user in the real world collides with thevirtual waypoint the relevant instruction is given (Figure 3).
User in the real world
User avatar in the virtual world
Waypoints with Evacuation instruction

Figure 3. Diagram showing persistence of real and virtual world objects.

338

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

4.3 Path Database
After importing the BIM model into Unity a path database is created. The path database is made up of
several path maps which are then cached to an external server to be localized within by the egress-seeker. Presimulated paths are calculated beforehand from every possible point on the floor plan and every possible route isaccounted for, awaiting instantiation as a voice instruction to users. Each line in a path is made by connecting an
initial starting point and a target destination. The target destinations are game objects and can be changed in theinspector panel of the game engine. In our case, windows and doors will be set as target objects. The A*
pathfinding algorithm is used to compute paths, as this method covers less area and computation time is fasterthan that of the Dijkstra’s algorithm (Wang et al., 2014). Each evacuation path is made up of nodes (see Figure 4)
with each node representing a sound file.

BIM model
objects withcolliders (green)

Graph Nodes
detect object
colliders

Figure 4. A screenshot within Unity of the graph nodes and graph cache system for path database creation.
4.4 Building Information Model (BIM)
A Building Information Model (BIM) is a digital three-dimensional semantic representation of a physicalentity (Kensek & Noble, 2014). Integrating built environment information and Internet of Things (IoT), termed
SMART buildings, is a common practice in operation and maintenance sectors. The proposed system integrates
BIM and IoT to create an evacuation guidance system able to assist users in building egress. Using a conversion
tool, the BIM model is exported as an Industry Foundation Classes (IFC) file in order to preserve the semantic
BIM data. Thereafter, in conjunction with the path database (Section 4.3), the system is built-out to a tablet device,
which acts as a server in charge of monitoring the entire fire situation (Section 4.5). Aside from providing building
geometry, BIM models include semantic information, for instance the flammability related to different building
components such as; plastered, wallpapered and tiled walls for example. The BIM model elements that inform
evacuation paths include walls, doors, windows, staircases and staircase enclosures, furniture and fire equipment.
Apart from the position of these elements, several crucial factors are taken into consideration. In terms of door
and window types; the system is aware of their respective methods of operation such as; standard door and windowlevers, sliding doors and sliding windows and finally fire doors with push-bar handles. In terms of windows two
aspects are taken into account; the sill height of windows and whether a window has access to a landing. This isimportant, because windows which are at an appropriate height and have access to a refuge area, are also included
as alternative exits when evacuation paths are configured.
4.5 Game Engine
Within the game engine, Unity, graphs are overlaid on the imported BIM model (see Figure 4). An avatarof each user and his/her varying properties are instantiated once they connect to the Situation Monitor (SM)
network. Thereafter their location is tracked as they move throughout the building. Unity then queries a MySQL
server for the most recent user position data and their avatar within Unity will move to the equivalent vantage
point (Davies, 2019). Once a fire emergency commences, based on each users’ characteristics, location, faceorientation and environmental factors the relevant evacuation path instruction is sent to the users’ earphones.
4.6 Mobile Application(1) Server: Situation Monitor (SM)In order to provide users with dynamic real-time egress instruction, three main aspects are continuously
monitored by the tablet device; hereinafter the Situation Monitor (Figure 5). The three main aspects that are339

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

simultaneously monitored are; building, fire/s and user circumstances. Using the proposed system, decision
makers and emergency personnel can then have portable real-time information about current fire situation. Beacon
identifiers used in the system prototype are shown in Figure 6.

Figure 5. A screenshot of the proposed Situation Monitor (SM) in the game engine.

Figure 6. A screenshot of Beacon identifiers used in the prototype system from the Estimote cloud portal.
(2) Client: User-Interface
In previous systems to view evacuation wayfinding arrows, users are required to hold their devices. The
proposed system, however, considers the low-visibility in smoke-filled rooms and proposes modified wirelessearphones providing additional hands-free operation. The proposed system provides unconventional evacuation
instruction, which under normal circumstances seem drastic, however, during an emergency situation could mean
the difference between life and death. Users receive evacuation instruction through modified wireless earphones;
this modification allows the system to track each user’s head orientation in order to ensure accurate path
instruction is given. In the proposed system, a user’s head and body movements are tracked separately due to theheads independent movement of the body. The wireless earphones are customized by attaching a 9-axis inertial
measurement unit (IMU) and an altitude and heading reference system (ARHS) with Bluetooth capabilities to oneof the earphones (Figure 7). The IMU unit uses three different Micro-Electro-Mechanical System (MEMS)
sensors. A 3-axis gyroscope, a 3-axis accelerometer and a 3-axis magnetometer which can achieve drift-free, highspeed orientation data around all three axes. The modified earphones hereinafter, the Face Orientation Detector(FOD), tracks the movement of users heads while user phones, irrespective of orientation, are tracked by the
beacons. The FOD device transmits the monitored data via Bluetooth packets to the Situation Monitor (SM) server
to know users’ face orientation and where they are in the building. In order to understand the need for thismodification, consider the following example. The blue spheres in the figures below represent the user and theirrespective x, y and z coordinates in the three-dimensional space (Figure 8). If the system sends an evacuation
instruction saying ‘go straight’, based solely on the users x, y and z positions, should the user not be facing thedirection of the path, the evacuation instruction will send them in the wrong direction. When the face orientation
is known the system can then accurately guide users as shown in image on the right in Figure 8. Moreover, shoulda user turn their head to look back, they are essentially still facing forward, and the system needs to know thedifference thus, the Face Orientation Detector (FOD) device is only attached to one earphone. Users faceorientation may seem trivial, however, in the proposed system the worst case scenario is assumed wherein smoke

340

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019density is at its’ maximum and visibility is equivocal to that of being blind, or in a dark room and because theBIM model contains the necessary information, the system can adequately guide users (Figure 9).

Figure 7. Face Orientation Detector (FOD)

Figure 8. Diagram of incorrect and correct face orientation, each node represents an instruction sound file.
User sensor data INPUT
earphone*FOD
Phone
beacon

Voice Instruction OUTPUT to the user

Situation Monitor

Situation Monitor

earphoneUser Information

User Information*FOD

Face Direction

Face Direction

User Location(x,y,z coordinates)

User Location(x,y,z coordinates)
Building

Building

Fire/s Information

Fire/s Information

Phone
beaconFigure 9. Input and output between the Situation Monitor (SM) interface and the user’s device.
*FOD: Face Orientation Device.
5. CONCLUSION AND FUTURE WORKS
Current evacuation guidance systems are insufficient as they are outdated methods still being used toescape buildings’ which are rapidly growing in complexity, height, and technological advancement such as thedevelopment of Smart buildings. The framework outlined in this paper, defines a real-time BIM-based evacuation
guidance system. The proposed system combines BIM and IoT technologies to provide egress-seekers with theirown personal fire evacuation guide, readily at hand should the need arise. The system is constantly monitoring
the location of occupants by using beacons for location tracking and should an emergency occur, the proposedsystem can immediately guide users accordingly, without users wasting valuable time attempting escape on theirown. The benefits of BIM, aside from geometric information, is that it contains semantic information of all theelements found in the building.
Two main aspects set the proposed system apart from previous studies; firstly, the navigation instruction
provided is unlike existing evacuation guidance systems. For instance, if all of the exits on the ground floor areblocked, the system would instruct egress-seekers to escape by going up the stairs to utilize the second floor bridge
which connects to the adjacent building or alternatively to access the roof and await further assistance. Secondly
the modified wireless earphones which track the direction a user is facing. These modified earphones are an
essential feature in order to achieve hands-free operation.
The wireless earphones are modified by attaching a 9-axis inertial measurement unit (IMU) and altitudeand heading reference system (ARHS) device with Bluetooth capabilities to one of the earphones. This allows the
tracking of head movement while the user’s phone, irrespective of orientation, is tracked by beacons. The Situation341

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Monitor (SM) on a portable device displays all egress-seeker information and path instruction while
simultaneously monitoring the conditions of the building and the fire/s conditions in real-time.
A possible pitfall of the system is its heavy reliance on vocal evacuation instruction and noise pollution
such as; alarms, explosions, falling building elements and people screaming could impede a user’s ability to hear
the egress instructions. A possible solution to this would be adding a white noise sound underlay in the backgroundof the voice instructions to help filter out background noise, or perhaps in future noise-canceling headphone
technology will be industry standard in wireless earphones.
Future works involve testing the prototype system to validate whether it is a reliable evacuation guidancesystem. A case study will be created based on a worst-case scenario; wherein smoke density is at its highest andvisibility is at its lowest. The testing environment will be done in a dark room, equivalent to that of being blind or
attempting to navigate a smoke-filled room to emphasize users’ reliability on the evacuation instructions. For theexperiment, two users on opposite ends will attempt to navigate in the dark from their respective locations to theirtarget destinations. In the first scenario, users will rely solely on the proposed system presented in this work,whereas in the second scenario users attempt to escape on their own. The duration of each scenario is documented
and timestamps are compared to one another in order to determine system feasibility and whether the proposed
work is able to efficiently save lives.
REFERENCES
Ahn, J. and Han, R. (2012). An indoor augmented-reality evacuation system for the Smartphone using
personalized Pedometry, Human-centric Computing and Information Sciences, 2(1), pp.18.
Atila, U., Ortakci, Y., Ozacar, K., Demiral, E. and Karas, I. (2018). SmartEscape: A Mobile Smart Individual Fire
Evacuation System Based on 3D Spatial Model, ISPRS International Journal of Geo-Information, 7(6),pp.223.
Davies, C. (2019). Parallel Reality: Tandem Exploration of Real and Virtual Environments, PhD, University of
St Andrews, pp. 116-124.
Chen, X., Liu, C. and Wu, I. (2018). A BIM-based visualization and warning system for fire rescue, Advanced
Engineering Informatics, 37, pp. 42-53.
Cheng, M., Chiu, K., Hsieh, Y., Yang, I., Chou, J., and Wu, Y. (2017). BIM Integrated Smart Monitoring
Technique for Building Fire Prevention and Disaster Relief, Automation in Construction, 84, pp.14-30.
Chu, M., Parigi, P., Law, K. and Latombe, J. (2015). Simulating individual, group, and crowd behaviors inbuilding egress, Simulation: Transaction of the society for modeling and simulation International
Journal, 91(9), pp. 825-845.
Kensek, K., Noble, D. and Eastman, C. (2014). Building Information Modeling: BIM in Current and Future
Practice, (1st ed.) Hoboken: John Wiley & Sons, Inc., pp. 23-28.
Lujak, M., Billhardt, H., Dunkel, J., Fernández, A., Hermoso, R. and Ossowski, S. (2017). A distributed
architecture for real-time evacuation guidance in large smart buildings, Computer Science andInformation Systems, 14(1), pp. 257-282.
Motamedi, A., Yabuki, N., Wang, Z., Fukuda, T., and Michikawa, T. (2016). Automatic Signage Visibility
Checking System Using BIM-enabled VR Environments, Proceedings of the 16th International
Conference on Computing in Civil and Building Engineering (ICCCBE2016), pp.1-8
Siddiqui, H., Vahdatikhaki, F. and Hammad, A. (2014). Performance analysis and data enhancement of wirelessUWB real-time location system for tracking construction equipment, 21st International Workshop of theEuropean Group for Intelligent Computing in Engineering, EG-ICE, Cardiff, United Kingdom.
Wang, B., Haijiang, Li., Rezgui, Y., Bradley, A., and Hoang, O. (2014). BIM Based Virtual Environment for Fire
Emergency Evacuation, The Scientific World Journal, pp. 1-22.
Wu, C. and Chen, L. (2012). 3D spatial information for fire-fighting search and rescue route analysis withinbuildings, Fire Safety Journal, 48, pp.21-29.
Zadeh, P. (2010). BIM-based Immersive Indoor Graph Networks for Emergency Situations in Buildings, In:International Conference on Computing in Civil Engineering, pp.1-7.
Zadeh, P. and Rüppel, U. (2013) A Method for the Application of Numerical Simulations during Firefighting
Operations Using Pre-Simulated, Model-Based Fire Scenarios, Open Journal of Civil Engineering (2A),
pp. 9-17.
Zafari, F., Gkelias, A. and Leung, K. (2019). A Survey of Indoor Localization Systems and Technologies, Instituteof Electrical and Electronics Engineers (IEEE) Communications Surveys & Tutorials, pp.1-32.
Zuo, Z., Liu, L., Zhang, L. and Fang, Y. (2018). Indoor Positioning Based on Bluetooth Low-Energy Beacons
Adopting Graph Optimization. Sensors, 18(11), pp.1-20.

342

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

DEVELOPMENT OF A METHOD TO DETECT EARTHQUAKE-RELATEDCHANGES IN IMAGES TAKEN BY CCTV CAMERAS SURVEYING CIVIL
INFRASTRUCTURE
Arata Konno1, Hirotaka Sekiya2, and Hideyuki Ashiya3
1) Head, Disaster Prevention and Relief Division, Okayama River Management Office, Chugoku Regional Development
Bureau, Ministry of Land, Infrastructure, Transport and Tourism, Japan. Email: konno-a85aa@mlit.go.jp
2) Ph.D., Head, Information Platform Division, Research Center for Infrastructure Management, National Institute for Landand Infrastructure Management, Ministry of Land, Infrastructure, Transport and Tourism, Japan. Email:sekiya-h92tb@mlit.go.jp
3) Director for Information Engineering Affairs, Electricity and Telecommunication Office, Engineering Affairs Division,
Minister’s Secretariat, Ministry of Land, Infrastructure, Transport and Tourism, Japan. Email: ashiya-h82ac@mlit.go.jp

Abstract: Following an earthquake, the government’s disaster management bureau immediately seeks to clarify
the degree of damage to infrastructure. Closed Circuit Television (CCTV) cameras have been deployed andutilized to survey infrastructure on roads and rivers. Use of an automated system to select CCTV cameras inmunicipal areas where seismic activity has exceeded a preset level (set by the Japan Meteorological Agency)
was studied. This system is equipped to obtain images taken by some testing CCTV cameras at regular times tocompare images pre- and post-earthquake.
In this paper, we validated a method of detecting regions of change caused by earthquakes using threesteps. First, we rendered non-photorealistic images as anomalous by drawing artificial damage after theearthquake. For example, falling objects can be captured by CCTV cameras surveying roads. Second, we
validated an optimal number of images so as to remove moving objects such as cars and waving trees. We used
the median value of each pixel in the images. Third, we validated algorithms containing parameters to adjust
image sensitivity. Accounting for daylight changes, we used a joint intensity histogram of an anomalous image
and a normal image. The joint intensity histogram was a two-dimensional combined intensity for the same pixel
between anomalous and normal images. We regarded changes outside the constant multiple of the standard
deviation of the joint intensity histogram as legitimate.
Using several filters, we removed moving objects with five images for four camera images. Under thiscondition, it was optimal to regard 10 times the standard deviation as artificial damage for 18 CCTV cameras.
We are trying to detect a variety of changes more robustly for more CCTV cameras.
Keywords:

support of the early stage after earthquake, Closed Circuit Television, image change detection

1. INTRODUCTION
Following an earthquake, the government’s disaster management bureau immediately seeks to clarify
the degree of damage to infrastructure. The larger the earthquake, the longer it takes to clarify damages. Closed
Circuit Television (CCTV) cameras have been utilized to survey infrastructure on roads and rivers. However, ittakes time to complete a review of all the images obtained by these cameras in disaster areas. Therefore, we
studied an automated system to select CCTV cameras in municipal areas, where seismic intensity detected by theJapan Meteorological Agency exceeds a preset level. This automated system is equipped to obtain images taken
by some testing CCTV cameras at regular times to compare images before and after an earthquake. We support
this effort.
In general, there are two approaches to detecting changes in the images. First, a change can be detectedfrom the previous image in the frames. An absolute difference greater than a threshold is marked as a change foreach pixel. This threshold needs to be established for each individual camera. This approach an also detect noisesuch as moving objects (such as a person and a car). Second, many methods of background subtraction havebeen proposed. Noise can be eliminated by rendering background images. This approach can be applied to avariety of CCTV cameras.
No perfect method exists despite the many methods that have been proposed in recent years. An ideal
background system should eliminate moving objects, gradual illumination change, waving trees, etc (Toyama etal., 1999). Piccardi (2004) classified background methods as seven types. Some types needs over a hundred
images to model images statistically. More precise models use more images.
However, there is little time to detect changes soon after an earthquake. This means only a few frames
from CCTV cameras are usable after an earthquake. While statistical approaches with many camera frames are
performed, no study as yet has sought to detect changes using fewer images. One of the objectives of this paper
was to eliminate noise with just a few frames.
Changes resulting from an earthquake are one anomaly. We cannot consider that all damage has been
done by an earthquake. Therefore, we do not have sufficient training images. We rendered non-photorealistic343

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

images as anomalous images by drawing artificial damage after the earthquake. The second objective of thispaper was to determine how to detect imaginable changes from an earthquake using multiple CCTV cameras.
We have developed a method toward that goal.
2. METHOD
We composed an algorithm with two parts as shown in figure 1. For the first part, we eliminated noises
with some frames. Note that we had few frames after the assumed earthquake. We discuss the optimal number offrames in Section 3. For the second part, we detected changes caused by an earthquake. Note that we had few
actual images showing earthquake damage. We rendered non-photorealistic images as anomalous by drawing
artificial damage after the earthquake in Section 3.

Figure 1. Composition to detect changes
2.1 Eliminating noises
A background image from a CCTV camera at time t was obtained by n frames: I (t-n+1), I(t-n+2), ･･･I(t).
The value I represents the intensity level (0–255) within the gray scale. Four types of background modelingclassified by Piccardi seem to be widely applied. The first type fits a Gaussian probability density function on
the last n pixel’s values (Wren & Azarbayejani, 1997), (Koller et al.,1994). The second type uses the medianvalue of the last n frames (Lo & Velastion, 2001), (Cucciara et al., 1997). The third fits a mixture of Gaussians tocope with multiple background objects (Stauffer & Grimson). The fourth is based on eigenvalue decomposition(Oliver et al., 2000). As mentioned in the previous section, we had few images soon after the earthquake. We
used a median filter in the second type because the median value was not affected by outliers compared to othertypes. A background image Bij (t) of a pixel (i, j) was then defined by equation (1):
(
) (
)
( )
( )
𝐵 = median(𝐼,𝐼, ･･･, 𝐼 )(1)where a pixel (i, j) ranges from (1,1) to (H, W). Smoothing filters are effective to denoise unnecessary
fluctuations in images. Before calculating the median value, we used a Gaussian filter defined by equations (2)and (3) for each image:𝐼 = ∑( , )∈,𝐺 (𝑥, 𝑦) =𝑒

(2)

𝐼 , 𝐺 (𝑖 , 𝑗 )
(

)

(3)

where the kernel Kij denotes a square neighborhood of fixed size and centered on a pixel (i, j), and the parameterα represents dispersion of the Gaussian. We adjusted the strength of filtering by changing the value of α. Aftercalculating the median value, we used a non-local mean filter GNL (Buades et al., 2005) using equations (4) and
(5):

𝐼 =

∑( ,

)∈

∑( ,

(, , , )

,

,
)∈

(4)

(, , , )

,
∑
(

,

)∈

,

,

,

(5)

𝐺 (𝑖, 𝑗, 𝑖 , 𝑗 ) = 𝑒

where the window size K’ij denotes a square region centered on both a single pixel (i, j) and a neighboring pixel(i’, j’), and β represents the degree of filtering. The non-local mean filter did not degrade or remove the finedetails. We show the values of these parameters in Table 1.
Gaussian Filter
non-local mean filter

Table 1. Values of parameters used in this paper
kernel size Kij: 5×5, α = 1kernel size Kij: 5×5, window size K’ij: 21×21, β =15344

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2.2 Detected changes
Radke et al. (2005) described two common types of operations for detecting image changes. The first
type involves geometric adjustments by analyzing the sensitivity of a difference in pixel values. The second type
involves intensity adjustments by analyzing pixel intensity values. In this paper, we used the joint intensity
histogram (JIH) corresponding to the latter type. In addition, we use the automated threshold method of Otsu
(1980) for comparison.
We defined the JIH as a two-dimensional histogram of combined intensity pairs, (B(1)ij, B(2)ij), where(1)
B ij represents the normal background image, and B(2)ij represents the background image obtained after theearthquake. To consider the mean intensity of a background image, we calculated average image B(k)ij (k = 1,2)
using equation (6):𝐵( ) =

∑, 𝐵

( )

(6)

Suppose we could disregard gradual illumination change and no change occurs, B(1)ij = B(2)ij = bij for all (i, j). The
value bij ranges from 0 to 255. Therefore, the value bij on two axes of B(1)ij and B(2)ij are plotted on a line l with aslope of 45°as shown in figure 2. Suppose some change occurs in images after the earthquake B(2)ij, and aregion will be observed outside of this line l. The difference in intensity level Dij between B(1)ij and B(2)ij iscalculated by equation (7):

𝐷 =𝐵

( )

−

( )
( )

𝐵

( )

(7)

Considering the effect of gradual illumination change, we regarded this line l as a Gaussian distribution with adispersion γ. Therefore, we regarded the region of JIH outside of the Gaussian as the change after the earthquake.
We introduced the integer parameter s with dispersion γ by equation (8):𝐷 > 𝑠𝛾(8)

The larger value s extended the noise region (illumination change) in JIH, whereas a small change was notdetected. We evaluate the parameter s by counting false positive and false negatives in 3.2.
We eliminated the tiny change with three steps. First, we used dilation as morphology once with a size 3
×3. Second, we used the median filter with a size 3×3. Third, we eliminated the region whose area was no
more than 120.

Figure 2. Example of the JIH (Kita, 2008)
2.3 Implementation
Our implementation was derived from the publicly available Python3.5 and OpenCV3.1.0. Note that theparameters in Table 1 are normalized to OpenCV.
3. EXPERIMENT AND DISCUSSION
3.1 Validation of the optimal number of frames
We used four camera images on “ChangeDetection Net” (Goyette et al., 2012). We regarded moving
cars, waving trees, wet snow, and car headlights as representative noises. We rendered non-photorealistic images
with these images: we drew cracks, falling objects, stacked vehicles, and falling rocks for each camera images.
As a result, we determined five frames to be sufficient to eliminate these noises. For more details, see ourprevious work (Konno, 2018).
3.2 Specification of the optimal value s in JIH
We tested 18 CCTV cameras deployed on roads, rivers, and towers of offices in the regional bureau. For
each camera, we extracted 10 frames at five-second intervals out of the movies. Then, we prepared two datasets:
the “normal datasets (NDs)” and “anomalous datasets (ADs).” In NDs, we directly input 10 frames. Changes
were not detected in NDs. If changes were not detected by JIH, we counted this camera image as true detection.

345

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

In ADs, we directly input only the first five frames. For the last five frames, we input the
non-photorealistic images, i.e., we drew artificial damage in the last five frames. We showed one normal frameand one artificial frame for each CCTV camera in figure 3. For example, we drew falling objects on roads,
cracks on banks of rivers, fire at a distance, sloping houses, etc. Only those regions with artificial damage should

Figure 3. A normal frame (left) and non-photorealistic frame (right) for each CCTV camera

346

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

have been detected by JIH. Other regions should not be detected. If regions with artificial damages were detected
and other regions were not detected, then we counted this camera image as true detection.
In this way, we counted the truly detected camera images with the parameter s ranging from 3 to 12. The
maximum number of truly detected camera images was 36 (18 in NDs and 18 in ADs). We show the results infigure 4. For s =3, almost all artificial damage was detected in ADs, whereas no true detection was observed in
NDs. This was because noise regions in JIH were too small for many other regions to be regarded as changed
falsely. However, with the larger value of s, the number of truly detected camera images in NDs increased. At
the same time, the number of truly detected camera images in ADs decreased about s > 7. This tells us that we
detected artificial changes and eliminated noises (illumination change) simultaneously, when the noise region in
JIH was optimal. In both datasets (NDs and ADs), the best number of truly detected cameras were observed withs = 10. We show the detected images of camera No. 17 deployed on towers of offices in figure 5, as an exampleof eliminating false positives. We also show, again, one of the last five frames for reference at the right of thisfigure. For s = 4, a false positive was detected in NDs, whereas the fire in the distance was truly detected in ADs.
The large vehicle affected by illumination was falsely detected in NDs. This indicated that intensity levels in theregion containing this vehicle were outside of the noise region of JIH. This large vehicle was not detected with s= 10. The distant fire was also truly detected with s = 10.

Figure 4. Relationship between number of truly detected cameras and the value of s

Figure 5. Detected images with various values of s and one of the last five frames (camera No.17)347

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3.3 Limitation of this method
We specified the optimal values s in JIH using images at noon in the previous section. However, anearthquake can occur under bad lighting conditions such as night and rain. Related works have proposed toeliminate raindrops in images. We will work to improve this method to detect changes even at night. In this case,we drew an artificial blackout as representative damage.
In addition, using few actual images during and after an earthquake is another problem. One approach tothis problem may be to consider the method of unsupervised learning. We are working on this approach.
4. CONCLUSIONSIn this paper, we demonstrated a method to detect regions of change by earthquakes in images taken by
CCTV cameras deployed on roads, rivers, and office towers in the regional bureau. We developed a systemequipped with the ability to obtain a sequence of images both before and after an earthquake. We used fiveframes for calculating the median value for all pixels to eliminate noises (for example, moving cars). Changes
were calculated with a JIH. As a result of testing for 18 CCTV cameras, the optimal parameter was s =10.
However, we used artificial images in this test. We need to go beyond artificial post-earthquake damage.
For example, unsupervised learning for each CCTV camera can be effective. We can continue to accumulate
images taken by CCTV cameras. Our work to improve the accuracy of this system is ongoing.
ACKNOWLEDGMENTS
This work was supported by the Cross-Ministerial Strategic Innovation Promotion Program on Council
for Science, Technology and Innovation: “Enhancement of Social Resiliency against Natural Disasters” managed
by the Japan Science and Technology Agency.
REFERENCES
Konno, A., Sekiya. H., and Ashiya. H. (2018). Validation of the optimal number of images for eliminating noises
to detect changes by the earthquake, Proceedings of infrastructure planning, Vol.58, p.212 (in Japanese).
Toyama, K., Krumm, J., Brumitt, B., and Meyers, B. (1999). Wallflower: Principles and Practice of Background
Maintenance, International Conference on Computer Vision, pp.255-261.
Piccardi, M. (2004). Background subtraction techniques: a review, IEEE International Conference on Systems,
Man and Cybernetics, pp.3099-3104.
Wren, C., Azarbayejani, A., Darrell, T., and Pentland, A. (1997). Pfilter: Real-time Tracking of the Human Body,
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, pp.780-785.
Koller, D., Weber, J., Huang, T., Malik, J., Ogasawara, G., Rao, B., and Russell, S. (1994). Toward Robust
Automatic Traffic Scene Analysis in Real-time, Proceedings of the International Conference on Pattern
Recognition, pp.126-131.
Lo, B., and Velastion, S. (2001).Automatic congestion detection system for underground platforms, Proceedingsof the International Symposium on Intelligent Multimedia, Video and Speech Processing, pp.158-161.
Cucchiara, R., Grana, C., Piccardi, M., and Prati, A. (1997). Detecting Moving Objects, Ghosts, and Shadows inVideo Streams, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25, No. 10,
pp.1337-1442.
Stauffer, C., and Grimson, W.E.L. (1999). Adaptive background mixture models for real-time tracking,Proceedings of the International Conference on Pattern Recognition, pp.246-252.
Oliver, N., Rasorio, B., and Pentland, A. (2000). A Bayesian Vision System for Modeling Human Interactions,
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 8, pp.831-843.
Buades, A., Coll, B., Morel, J. (2005). A non-local algorithm for image denoising, IEEE Computer Society
Conference on Computer Vision and Pattern Recognition, Vol. 2, pp.60-65.
Radke, R. J., Andra, S., Al-Kofahi, O., and Roysam, B. (2005). Image Change Detection Algorithms: A
systematic Survey, IEEE transactions on image processing, Vol. 14, No. 3, pp.294-307.
Otsu, N. (1980). An automatic threshold selection method based on discriminant and least squares criteria, Thetransactions of the Institute of Electronics, Information and Communication Engineers, Vol. J63-D, No.
4, pp.349-356 (in Japanese).
Kita, Y. (2008). A study of change detection from satellite images using joint intensity histogram, proceedings of
19th International Conference on Pattern Recognition, pp.351-356.
Goyette, N., Jodoin, P., Porikli, F., Konrad, J., and Ishwar, P. (2012). changedetection.net: A new change
detection benchmark dataset, IEEE Workshop on Change Detection. Website:
http://www.changedetection.net/

348

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

A HUMAN-FOLLOWING ROBOT FOR ASSISTING TUNNEL INSPECTORS
Chia-Hsing Ho1 and Yo-Ming Hsieh2
1) Engineer, Turing Drive Inc., Taipei City, Taiwan, Email: chiahsing@turing-drive.com
2) Associate Professor, Department of Civil and Construction Engineering, National Taiwan University of Science andTechnology, Taipei City, Taiwan, Email: ymhsieh@mail.ntust.edu.tw

Abstract: Tunnels require regular inspections to maintain their safety. Tunnel inspectors have modern tools suchas smartphones and tablets to help them record observations via picture-taking, voice-recording, hand-writing, andtext-typing. However, there is a lack of automated ways to obtain localization information inside tunnels. In thiswork, a person-following robot is developed for assisting tunnel inspectors by providing localization information
inside tunnels. The robot contains four essential ingredients: person-detection, self-driving, localization, andcommunication with smart devices. This paper presents these ingredients and their integration. The developed
robot was field tested inside buildings and pedestrian tunnels with success and issues. These results and issues arealso presented in this paper.
Keywords: localization, person following, robot, tunnel inspection
1.

INTRODUCTION
Tunnels are used for highways, railroads, subways, etc., that are very important for our traffics. Since
tunnel is an enclosed space, it would cause traffic jam or casualties once it had any unexpected damage on structure.
According to domestic and foreign statistics, the most common accident in tunnel is fire (Chou et al., 2004). Whenfire accidents occurred in tunnel, it would cause damage to part of the tunnel structure and equipment, andhampering the initial relief implementation, which will more likely develop into major disasters (Tsai and Chang,2012). Therefore, tunnels require regular inspections to maintain their safety. The inspection needs to ensure thefire protection equipment and the structure of the tunnel is in good condition.
To maintain tunnels, visual inspections are regularly performed. Tunnel inspectors have modern tools
such as smartphones and tablets to help them record observations via picture-taking, voice-recording, hand-writing,and text-typing. However, locations are roughly estimated and recorded by inspectors. Vision based localization(Hsieh and Liao, 2014) can help inspectors to get localization information via two cameras, but it requires
inspectors to fix cameras’ position and keep horizontal between two cameras, which is pretty annoying to checkthe cameras’ status constantly during the inspection. There is a need to provide localization information
automatically to reduce inspectors’ works.
In this article, a human-following robot is developed for assisting tunnel inspectors by providing
localization information inside tunnels, as shown in Figure 1. The robot can follow a person through a stereo
camera and locate itself while following the person. The hardware and software design, main methods of the robot
are introduced in next section. Then, in Section 3, the test result of person detection and a field test in a pedestriantunnel is described. The issues of this work are described in Section 4. Finally, the conclusion is described in
Section 5.

Figure 1. Human-following robot (left). Robot following a person in a pedestrian tunnel (right).
2.

METHOD
The developed robot contains four essential ingredients: person-detection, self-driving, localization, andcommunication with smart devices. The overall flowchart of the robot is shown in Figure 2. There are three mainparts that are parallelly processing in the robot, while self-driving and localization are considered in the same part.

349

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Each part can work independently, which means if any other part crushed, it won’t influence the other parts’working. The reason we designed the program this way is because the camera we used crushes sometimes. We
used inter-process-communication to let different processes access the same data. The hardware of the robot will
be introduced in Section 2.1. The software of the robot will be introduced in Section 2.2, and the explanation ofeach part will be described in Section 2.3-2.5.
startPersondetection

Self-driving
andlocalization

Communication
with smart
devicesFigure 2. Overall flowchart of the robot
2.1

Hardware
The hardware components of the robot are listed in Table 1. The main arithmetic unit is Nvidia Jetson
TX1, an embedded system-on-module for building autonomous machines. Its computing performance is usefulfor deploying deep learning and computer vision. The camera we used for obtaining images and detecting person
is Intel RealSense ZR300, a stereoscopic depth-sensing camera that can be used for object detection, recognition,and tracking. The base of the robot is mini T101-D car platform, which contains two DC motors for controlling
the tracked wheel. On each DC motor there is an encoder for counting the turns of the motor, in order to calculate
the odometry of the robot. To control the DC motors, a L298N module is used. A HC-SR04 module is used formeasuring the distance between the robot and the wall on the robot’s right side to draw the robot’s moving route.
Finally, there are two power banks used for supplying power for TX1 and DC motors.
Table 1. Hardware Components
Component / Module
PurposeNvidia Jetson TX1Main arithmetic unit
Obtaining Depth Image, Person
Intel RealSense ZR300
detectionmini T101-D Smart Robot Tank
Base of the robot
Chassis Tracked Car Platform
DC motor
Controlling the wheel of the robot
Motor Encoder
For counting the turns of DC motor
L298N Dual H-Bridge Motor
For controlling DC motor
Controller Module
HC-SR04 Ultrasonic Ranging
For creating odometry map of theModulerobot
Battery Pack
Supplying power for the robot
2.2

Software
The software components of the robot are listed in Table 2. Jetpack 3.1 is the OS image for Jetson TX1,
along with libraries and APIs such as CUDA, cuDNN, and OpenCV. PCL stands for point cloud library, an open
source library for 2D/3D image and point cloud processing. PCL is used for filtering noise of the image fromcamera and person detecting. BlueZ is the official Linux Bluetooth stack. It provides, in its modular way, support
for the core Bluetooth layers and protocols. BlueZ is used for creating and broadcasting Eddystone Beacon
Messages through Bluetooth Low Energy (BLE). The last one is Linux IPC, stands for Inter-Process350Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Communication, allows processes to manage shared data. The robot system uses IPC/shared memory to transfer
data between processes.
Table 2. Software Components
Component / Module
PurposeJetpack 3.1
OS for Nvidia Jetson TX1
PCL 1.8.1
Image filter, Person detection
For obtaining depth image fromlibrealsense 1.12.1
ZR300
For creating Eddystone BeaconBluezmessages
For data transferring between
Linux IPC/shared memory
processes
2.3

Process of person-detection part
The method we used for detecting person is based on PCL people detector, which can detect a person andits pose in a point cloud (Buys et al., 2014). It splits RGB-D image to depth and RGB image first. Then use a
pretrained kinematic model to label the depth image with human body parts, in order to filter the noise of the image.
After that, combine the filtered depth image with RGB image and do the labeling again. Thus, the RGB-D image
would be able to label out the human body.
The flowchart of person detection is shown in Figure 3. At the beginning, capture RGB-D image fromcamera, transform it into a point cloud. Then, check each point in the point cloud, filter out points with depth value
out of range. Do the person detection, once it finds person, calculate the person’s center point coordinates and thedepth of center point, which means how far between the person and the robot. If not, grab another image and do
the detection again. Finally, write the person information, center and depth of the person, into shared memory to
let other processes get it.

start

Image
capturePersondetection

Noise filter

Write person
information intoshared memory

Get person
center pointand depth

yes

Find
perso

no

Figure 3. Flowchart of person detection
2.4

Process of self-driving and localization part
The aim of this process is to follow the person detected. The flowchart of process is shown in Figure 4.
Once it gets person data from shared memory, it will check if the data is sent long time ago. If it is, then it means
the person may had disappeared from camera, so it will stop moving and wait until the person shows again. If theperson data is right on time, then it will compare the data with former one, to make sure the person is not getting
weird moves. After that, use PID controller to calculate the movement of the robot. Then use the output result tomove the robot. After robot moving, get the feedback movement from motor encoder, calculating the displacementof the robot, which is the robot’s odometry. Finally, write the moving information into shared memory to let thenext process get it.
For the robot moving, we used two PID controllers to control it, one is for controlling the direction andanother one is for the depth. Concept of each controller is explained below.
(1) PID in direction
To keep the person in the center, so he/she wouldn’t be out of the image, PID controller is trying to makesure the person’s center point be the center of the image. Which is, if the x-axis range of the image is between 0
to 1, the controller will keep the x-axis value of the center point 0.5. When the value is bigger than 0.5, meaning
the person is at the right side of the image, the controller should then make the robot moving to the right side. Thus,
the person may approach to the center.

351

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The output of this PID controller are two different speed of the left and right motor on the robot. Whenthe person is inclined to one side, the motor on the opposite side should raise its speed. The more offset the person
is apart from center, the faster the speed should be.

Stop moving

yes
Get person
information fromshared memory

start

Write moving
information intoshared memory

Reading

Calculatedisplacement

noMove therobot

Comparison
with former
person data

PID
controller

Figure 4. Flowchart of self-driving and localization(2) PID in depth
To keep the person in front of the robot, this controller is controlling the distance between the robot andthe person it detected. Because the range of depth value the camera can get is between 0.55 to 2.8 meters, thedesigned value of depth is 1.5 m, the average of the range. When the person’s depth value is bigger than 1.5, therobot needs to go forward, otherwise it needs to go backward.
The output of this controller is the speed of two motors on the robot. The sum of this output and the output
from controller in direction will be sent to the motor control unit, which is the speed value of the two motor on
both side of the robot.

(3) motor control and localization
The method used for motor control is PWM control. By changing the PWM working period, the speed ofthe motor can be controlled according to the sum of the outputs of two PID controllers. The encoder on the motor
can count the turns of the motor. By multiplying the turns of the motor and the radius of the wheel of the robot,
we can then obtain the displacement of the robot. In this way we can get the distance from the beginning that the
robot has moved.
2.5 Process of communication with smart devices partIn this part we are trying to get some robot information while it is moving. So, we packed up the person
and robot moving data into Eddystone beacon type and broadcast it as beacon URL type messages. The flowchart
is shown in Figure 5. At first, get the person data and the robot moving data from shared memory. Then, transfer
them into beacon message. Send it over. Then we can easily get the message through any apps that can get beacon
message on smart phone/ tablet.

start

Get person andmoving informationfrom shared memory

Transfer data into
Eddystone
beacon URL type

Send beacon
message

Figure 5. Flowchart of communication with smart devices
3. RESULTS
The developed robot was field tested inside buildings and pedestrian tunnels. The result of person
detection is shown in Figure 6. On the left side of Figure 6 is the image from camera, which has been filteredwithout the points whose depth value is bigger than 2.8 m. On the right side is the result image of person detection.
Different body parts of the person are labeled on the image, since the robot is at the back side of the person, it352

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

can’t label the person’s head.
As the image shows, the person detector can only detect the upper part of the body. Maybe it is because
the camera on the robot is upward, the tilt angle is too big to capture the whole-body parts of person. There is aneed to optimize the tilt angle of the camera in order to get the best performance of the person detection.

Figure 6. Filtered image from camera (left). Person detection result (right).
On the other hand, the tracking result is shown in Figure 7. We used an ultrasonic sensor on the right sideof the robot to get the distance between the robot and its right-side wall. The distance from wall and thedisplacement from encoders of motors can then create the charts. Let the value of distance from wall be the x-axis
and the displacement of the robot has moved be the y-axis. Both of the chart shows the route of robot and target
person in a pedestrian tunnel. As it shows, although the robot keeps moving left and right, it can successfully
follow the target person.
The reason of the robot keeps moving left and right might because of the ultrasonic sensor’s position is
not the right place to add on. Since we placed the sensor on the right side of the robot body, it may get different
distance when the robot turns left or right.

Figure 7. Field test result in pedestrian tunnels

353

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

4. DISCUSSION
The developed robot is a prototype assistant for helping tunnel inspectors to get the localization
information in the tunnels. However, there are some issues it has to be described. First of all, the orientation of thecamera needs to be considered as an impact factor. It is hard to check the camera’s tilt angle during the test, since
the robot doesn’t have a way to display image immediately. If the tilt angle is too big, it may only catch the target
person’s upper-body. However, if the tilt angle is too small, it may only catch the lower part of the body. Thus,
finding a way to optimize the camera’s orientation or display the image from camera on teleoperated devices isnecessarily.
During the robot movement, we may use encoders of the motor to gain wheel odometry. However, it onlyshows the displacement of the robot, it needs to combine with data from ultrasonic sensor to draw the route of therobot. The ultrasonic sensor is unstable and prevents smooth movement of the robot. Even the robot is tracking
the target person well, the route of it may still shifting on the chart due to the ultrasonic sensor. Thus, finding
another way to draw the route of the robot may solve this problem.
5. CONCLUSIONSIn this work, we use an embedded system to build a human-following robot, which can follow a person
through a stereo camera and locate itself while following the person. The developed robot can then provide tunnel
inspectors their localization information for their recording of inspecting. The person following behavior isachieved by people detection and PID controller algorithm. People detection is done by PCL, an open source
library, that processes RGB-D images and detects pixels belong to human. PID controller algorithm controls the
motor in the robot to achieve person-following. The localization information is gained from wheel odometry of
the robot. The status of the robot is transmitted using Bluetooth Low Energy (BLE) that broadcasts Eddystone
Beacon messages. The message can be shown to the user by Apps in mobile devices to monitor the robot, and also
provide localization information to tunnel inspectors.
Through field tests inside buildings and pedestrian tunnels, the developed robot in this article cansuccessfully follow people. However, the orientation of the camera and the way to show the route of the robot
needs improvement.
ACKNOWLEDGMENTS
The author of this article would like to thank PCL library, author of librealsense, and all the other opensourced library. Thanks to those people’s dedication, we don’t have to build the wheels ourselves and have moretime to developing things to further level. Together, we can make the world a better place.
REFERENCES
Chou, Y.D., Hsin, Y.F., and Chang, S. C. (2004), Discussion on the strategy of safe operation of tunnel operation
management from major long tunnel accidents at domestic and foreign in recent years., Taiwan
Highway Engineering, 30(7).
Tsai, T.M. and Chang, P.Y. (2012), A Research on Security Problems and Management Mechanisms for Long
Tunnels - A Case Study on Hsueh Shan Tunnel. Retrieved from NDLTD in Taiwan website:
https://hdl.handle.net/11296/ae7h6hHsieh, Y.M. and Liao, Y.C. (2014), Localization inside Tunnels Using Machine Vision. Proceeding of the 31st
International Symposium on Automation and Robotics in Construction and Mining (ISARC 2014), pp.
860 - 867, Sydney, NSW, Australia. (Paper #123).
Buys, K., Cagniart, C., Bashkeev, A., Laet, T., Schutter, J., and Pantofaru, C. (2014), An adaptable system for
RGB-D based human body detection and pose estimation., Journal of Visual Communication and Image
Representation, 25 (1), 39-52.

354

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

A BUILDING STRUCTURAL HEALTH MANAGEMENT SYSTEM BY BIM AND
IoT COLLABORATION
Narito Kurata1, Kenro Aihara2, Takahiro Konishi3, Hirofumi Yamaoka4, and Shinichi Kondo5
1) Ph.D., Prof., Faculty of Industrial Technology, Tsukuba University of Technology, Tsukuba-City, Ibaraki, Japan. Email:kurata@home.email.ne.jp
2) Ph.D., Assoc. Prof., Digital Content and Media Sciences Research Division, National Institute of Informatics, Chiyoda-ku,
Tokyo, Japan. Email: kenro.aihara@nii.ac.jp
3) Corporate Officer, General Manager, Business Strategy Division, Applied Technology Co., Ltd., Kita-ku, Osaka, Japan.
Email: ta-konishi@apptec.co.jp
4) IT Consultant, Business Strategy Division, Applied Technology Co., Ltd., Kita-ku, Osaka, Japan. Email:yamaoka@apptec.co.jp
5) BIM Consultant, Business Strategy Division, Applied Technology Co., Ltd., Kita-ku, Osaka, Japan. Email: shkondo@apptec.co.jp

Abstract: The authors have developed an autonomous time-synchronization sensing system that holds highprecision absolute time information by using a Chip-Scale Atomic Clock (CSAC) (Kurata, 2016; Kurata, 2018).
Even if the CSAC-mounted sensor module/data logger is installed over a wide area and at high density, it ispossible to acquire measurement data with time synchronization without relying on a network or a global
positioning system (GPS) signal. The CSAC-mounted sensor module as an Internet of Things (IoT) device wasinstalled in an actual building, and a structural health evaluation was performed for each subsequent earthquake(Kurata et al., 2008). A building structural health management system was constructed wherein acceleration dataon each floor during an earthquake as measured by the CSAC-mounted sensor module is transmitted to a cloudserver by 3G/LTE, and the acceleration, the inter-story drift angle and a structural health evaluation of the building
are displayed in 3D model by a Building Information Modeling (BIM). Forge (Applied Technology, 2019;
Autodesk, 2019) is employed as a development platform to realize this. This paper gives an outline of the CSACmounted sensor module as an IoT device and the building where it was installed, along with case results of seismic
observations and inter-story deformation, a proposal for a building structural health management system, and its
envisaged future. As a demonstration, an autonomous time-synchronization sensing system as an IoT device waslinked with Forge, the BIM cloud expansion support development platform, and the results of a structural healthevaluation during an earthquake were shown on a BIM model.
Keywords: Building Management System, Building Information Modeling (BIM), Internet of Things (IoT),
Structural Health Monitoring, Earthquake Observation, Time Synchronization, Chip-Scale Atomic Clock.
1. INTRODUCTION
In the process of building design, construction and maintenance, it is necessary to use BIM models
consistently for the purposes of reducing costs, improving work efficiency, shortening the construction period andcreating added value. In addition, for building maintenance and management, it is expected that the IoT will be
used, which enables the use of many sensors, with a view to introducing fifth-generation mobile communication
systems (5G). In recent years, the IoT, which collects real space information (Kawakatsu et al., 2019), and clouddevelopment platforms that link to a BIM model, have become available. The authors developed an autonomous
time-synchronization sensing system on which a CSAC is mounted as an ideal IoT system to maintain highprecision absolute time information (Kurata, 2016; Kurata, 2018). Even if the CSAC-mounted sensor module/datalogger is installed over a wide area and at high density, it is possible to acquire measurement data with timesynchronization without relying on a network or a GPS signal. Meanwhile, the authors are also developing a BIM
service with the name of “toBIM” (Applied Technology, 2019), which allows the use of Forge, the BIM cloud
expansion support development platform (Autodesk, 2019). Using these devices and the service, the authors haveproposed a building structural health management system linking the IoT and BIM, and are promoting its
commercialization. Specifically, a CSAC-mounted sensor module was installed in an actual building, and theacceleration data measured for each subsequent earthquake was sent to a cloud server by 3G/LTE. By means ofForge, a building structural health management system was constructed that displays accelerations, inter-story
deformation angles and a structural health evaluation of a building in 3D on a BIM model of the building.
2. AUTONOMOUS TIME SYNCHRONIZATION SENSING SYSTEM
In order to install multiple sensors indoors, where GPS signals cannot be received, and to performmeasurements while ensuring time synchronization, it is necessary to lay a dedicated line, a wired or wirelessnetwork, and the like. A sensor module/data logger was developed that eliminates these requirements andmaintains high-precision absolute time information autonomously, regardless of the environment (Kurata, 2016;
Kurata, 2018). The main control unit, the sensor unit and the wireless communication unit are housed in a case, as355

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019shown in Photo. 1. The main control unit is equipped with a central processing unit (CPU), the CSAC, a field
programmable gate array (FPGA), a memory, storage, and a network interface that uses Ethernet. The sensor unit
is equipped with a three-axis micro-electro-mechanical system (MEMS) acceleration sensor, a three-channel
external analog sensor input interface, a temperature sensor, an anti-aliasing filter, and an A/D converter. The
module functions as a single sensor by using the built-in MEMS acceleration sensor, and functions as a data logger
by connecting an external sensor with the external analog sensor input interface. The wireless communication unit
is implemented by a commercially available Raspberry Pi, has a communication function that uses Wi-Fi and
3G/LTE, and can rapidly adapt to new communication methods (Kurata, 2016; Kurata, 2018). With this module,
an autonomous time-synchronization sensing system using various sensors can be constructed. By linking this IoT
device with a BIM model, a building structural health management system has been constructed that reduces costand creates added value, using BIM as its standard (Fig. 1).
CSAC

Sensor Unit

Wireless Communication Unit

Main Control Unit

Photograph 1. CSAC-mounted autonomous time synchronization sensor module

Display on BIM ModelIoT Sensor

Forge Cloud Platform

Figure 1. Outline of building structural health management system by BIM and IoT collaboration
3. STRUCTURAL HEALTH EVALUATION OF THE BUILDING DURING AN EARTHQUAKEThe building to which the autonomous time-synchronization sensing system was applied is a reinforced
concrete three-story building (Photo. 2) in the Tsukuba area. Four CSAC-mounted sensor modules were installed
in a dedicated space beside the stairs on each floor, as shown in Fig. 2. For this application, all modules used abuilt-in MEMS acceleration sensor. Table 1 is a list of the earthquakes measured during October and November
2017. The trigger setting for seismic measurement was set to a level that generally exceeded 1 cm/sec2, and data
for earthquakes with a seismic intensity of 1 or more that occurred in the Tsukuba area were measured.

Photograph 2. External appearance of applied building356

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

X

Location of sensors on each floor

Z
Y

Figure 2. Plan view of 2nd floor of the building and locations of CSAC-mounted sensor module installed
Table 1. List of earthquake record measured by CSAC-mounted sensor modules
Date

Time

Name ofEpicenter

Magnitude/
Depth(km)Local/Max. Intensity

1

06/10/2017

16:59

Fukushimaken-oki

6.3/57

1/2

2

06/10/2017

23:56

Fukushimaken-oki

5.9/53

2/5 lower

3

07/10/2017

16:20

Ibarakiken-nanbu

3.4/43

1/1

4

12/10/2017

15:12

Fukushimaken-oki

5.2/261/2

5

15/10/2017

19:05

Ibarakiken-hokubu

3.0/7

1/1

6

18/10/2017

07:40

Ibarakiken-nanbu

3.7/451/2

7

02/11/2017

22:31

Ibarakiken-oki

4.3/74

1/3

8

03/11/2017

21:38

Ibarakiken-hokubu

4.8/8

2/3

9

05/11/2017

17:40

Ibarakiken-nanbu

2.9/43

1/1

10

15/11/2017

01:21

Ibarakiken-nanbu

3.8/20

1/2

11

26/11/2017

15:55

Ibarakiken-hokubu

3.9/4

1/2

12

30/11/2017

22:02

Ibarakiken-naubu

3.9/42

1/3

Y dir. deformation (cm)Y dir. deformation (cm)Y dir. deformation (cm)No

X direction
acceleration

X dir. deformation (cm)Y direction
acceleration
X dir. deformation (cm)X dir. deformation (cm)Figure 3. Maximum acceleration and inter-story deformation calculated by sensor data357

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Among the measurement data shown in Table 1, for the Ibaraki North earthquake (Table 1, No. 8) which
recorded a maximum seismic intensity of 3, the inter-story deformation was calculated by integrating theacceleration waveform of each floor. Fig. 3 shows the inter-story deformation and the distribution of maximum
acceleration for each floor. As this was a small earthquake with a seismic intensity of 3, the maximum inter-story
deformation angle was small and had no effect on structural integrity. However, it is possible to quantitatively
confirm structural integrity by acquiring measurement data and assessing the inter-story deformation.
4. STRUCTURAL HEALTH EVALUATION DURING AN EARTHQUAKE BY LINKING FORGE AND
IoT EARTHQUAKE SENSING SYSTEM
Forge supports the expansion of a desktop application-based BIM to a cloud-based BIM. Its biggest
feature is that it has a 3D viewer that displays BIM data on a browser, and it can customize representations suchas the addition, movement, rotation, and coloring of 3D graphic elements by WebGL. It can also receive data froma device connected via the Internet, and allow it to be visualized. Fig. 4 shows the toBIM service platform deployedby the authors. Forge is a cloud foundation of a building management system that can be linked to this platform.
When a specific URL is accessed by a browser, a BIM model on the cloud is acquired to provide 3D display
functions tailored to the purpose. This makes it possible to provide various services utilizing BIM to users who do
not have BIM software (non-BIM users). For example, services such as “an Facility Management (FM) system
which allows selection of the equipment installed on a BIM and confirms the model type, performance, andmaintenance period”, and “a monitoring system that uses an indoor positioning sensor and a biosensor attached toa person to display information such as the person’s position and body temperature on the BIM in real time” canbe provided.
As an attempt to implement a building structural health management system, the autonomous timesynchronization sensing system as an IoT device, as shown in Section 3, was linked to Forge for a demonstration.
The cloud environment of the building structural health management system is built on AWS Cloud, as shown in
Fig. 5, and Fluentd (Fluentd Project, 2019) was used to exchange data between the cloud and the IoT. Fluentd is
open source software that facilitates secure transfer, collection, aggregation, and storage of data distributed tomultiple locations, and which is continually increasing. The authors used open source software because thesetechnologies are still evolving, and improvements in quality and security can be expected due to their scrutiny bymany users. The received data is analyzed by Amazon Elasticsearch Service, and presented in graph form in realtime in a link-up with KIBANA (Elasticsearch, 2019). Logs are collected by Amazon S3, and results collected
previously can be browsed.

Working with BIM software
Working without BIM software
Maximize efficiency in creating BIM data Maximize efficiency in using BIM data

Figure 4. Outline of toBIM service platform

358

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 5. Cloud configuration of building structural health management system
The customization of Forge’s 3D viewer to display a BIM model is shown in Figs. 6 and 7. A display
node is shown on the end of the pillar closest to the IoT seismic sensor installed on each floor of the building, andits color represents the magnitude of the acceleration. By moving the nodes while they were connected with line
elements, the inter-story displacement and inter-story deformation angle of each floor were successfully visualized.
The measurement data was time history data sampled every 0.01 seconds. Acceleration data was extracted for each
earthquake event, and the magnitude of the displacement of each floor was calculated by double integration. By
visualizing this continuously, an animation can be displayed. Further, by extracting the maximum inter-story
deformation angle from the measurement results, the deformation status at that moment could be found. From thisdemonstration, it was confirmed that visibility can be dramatically improved by using a 3D representation thatutilizes BIM for the structural safety assessment of buildings during an earthquake. The improvement of visibility
is considered to be an important item in a service for general users without specialized knowledge.

Figure 6. Visualization of seismic measurement results on BIM model by Forge

①
②
③
④
⑤

Selection of Past Earthquake
Animation
Adjustment of Playback Speed
Maximum Story Displacement
Mode Magnification

Figure 7. Display command

359

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

5. CONCLUSIONSThis paper gave an outline of a CSAC-mounted sensor module as an IoT device, the building where it was
installed, a series of seismic observations and an assessment of inter-story deformation, and proposed a newbuilding structural health management system and its envisaged future. As a demonstration, a case was reported
where an autonomous time-synchronization sensing system as an IoT device was linked to Forge, the BIM cloud
expansion support development platform, to represent on a BIM model structural health evaluation results during
an earthquake. By linking the IoT with BIM, the authors plan to work on implementing a building structural healthmanagement system that leads to cost reductions and creates added value.
ACKNOWLEDGMENTS
This research was partially supported by the New Energy and Industrial Technology Development
Organization (NEDO) through the Project of Technology for Maintenance, Replacement and Management of Civil
Infrastructure, Cross-ministerial Strategic Innovation Promotion Program (SIP). This research was also partially
supported by JSPS KAKENHI Grant Number JP16K01283.
REFERENCES
Applied Technology. (2019). toBIM. Retrieved from Applied Technology website: https://tobim.net/
Autodesk. (2019). FORGE:Cloud based developer tools from Autodesk. Retrieved from Autodesk website:
https://forge.autodesk.com/
Elasticsearch. (2019). kibana. Retrieved from Elasticsearch website: https://www.elastic.co/jp/products/kibana
Fluentd Project. (2019). fluentd. Retrieved from Fluentd Project website: https://www.fluentd.org/
Kawakatsu, T., Aihara, K., Takasu, A., and Adachi, J. (2019). Deep Sensing Approach to Single-Sensor Vehicle
Weighing System on Bridges, IEEE Sensors Journal, 19 (1), 243-256.
Kurata, N., Suzuki, M., Saruwatari, S., and Morikawa, H. (2008). Actual Application of Ubiquitous Structural
Monitoring System using Wireless Sensor Networks, Proceedings of the 14th World Conference on
Earthquake Engineering (14WCEE), Beijing, China, pp.1-8, Paper ID:11-0037.
Kurata, N. (2016). Basic Study of Autonomous Time Synchronization Sensing Technology Using Chip Scale
Atomic Clock, Proceedings of the 16th International Conference on Computing in Civil and Building
Engineering (ICCCBE2016), Osaka, Japan, pp.67-74.
Kurata, N. (2018). Improvement and Application of Sensor Device Capable of Autonomously Keeping AccurateTime Information for Buildings and Civil Infrastructures, Proceedings of the Ninth International
Conference on Sensor Device Technologies and Applications (SENSORDEVICES 2018), Venice, Italy,
pp.114-120.

360

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

DEVELOPMENT OF AN ANOMALY DETECTION SYSTEM OF ROAD SIGNSUSING MEMS ACCELEROMETERS
Naomasa Haibara1, Masayuki Saeki2
1) Master’s Student, Tokyo University of Science, Chiba, Japan. Email: 7618527@ed.tus.ac.jp
2) Dr. Eng., Prof., Tokyo University of Science, Chiba, Japan. Email: saeki@rs.noda.tus.ac.jp

Abstract: It is recognized as an important issue how to efficiently maintain an enormous number of social
infrastructures. For example, it is said that there are approximately 10 million road signs in Japan, and they are
visually inspected once every five years. The visual inspections take a lot of time and cost. Therefore, we havebeen trying to develop an anomaly detection system for infrastructures. In this research, as a basic experiment, aroad sign was constructed in our university. Then, MEMS (Micro Electronic Mechanical Systems) accelerometers
was fixed on the back of the signboard, and the acceleration responses were automatically observed every 4 hours
for about 18 months to investigate the daily variation. A single measurement was performed at 100 Hz sampling
rate and 3 minutes data length. As a result, it was found that the eigenfrequency of the first mode varied by about
0.01 Hz. We also simulated the damage of the boundary condition which was often seen in damage cases of roadsigns and examined the change in responses. As reducing the thickness of the root part by about 0.5 mm whichcorresponded to 12 % decrement of bending rigidity, the eigenfrequency decreased by about 0.016 Hz. Since thechange in eigenfrequency was larger than the temporal variation, the anomaly was successfully detected. On theother hand, in the case of 0.1 mm which corresponded to 5 % decrement, the change in eigenfrequency could not
be detected because of the small variation. In addition, we also examined the direct density ratio estimation method
based on KLIEP (Kullback-Leibler Importance Estimation Procedure) to detect the change point and abnormal
value of the anomaly detection index.
Keywords: eigenfrequency, MEMS accelerometer, anomaly detection, road signs, density-ratio estimation,
KLIEP
1. INTRODUCTION
There is a need to develop a method to efficiently inspect the social infrastructures because a large numberof infrastructures are there, and their deteriorations may cause serious accidents. In recent years, large amounts ofacceleration data can be easily acquired because of the development of IoT (Internet of Things) technology andthe widespread use of MEMS (Micro Electronic Mechanical Systems) accelerometers. The big data obtained bylong-term monitoring may let us detect very small changes in vibration characteristics that are not visible before.
In addition, the widespread use of reasonable MEMS accelerometers might make it possible to locate numerous
numbers of sensors on structures. Based on such backgrounds, Ozaki et al. (2017) experimentally constructed aroad sign in our university and fixed a MEMS accelerometer to the back of the signboard. In their experiments,
the vibration responses were manually measured to investigate the temporal variations as a basic research.
In this study, a prototype of wireless sensor equipped with a MEMS accelerometer was developed andfixed to the road sign constructed in our university. The vibration responses were automatically measured every 4hours for about 18 months. The observed acceleration data were analyzed to develop anomaly detection indexes
and to investigate their daily variations. The direct density ratio estimation method was also applied to the timeseries data of anomaly detection indexes to find out their change points.
2. VIBRATION MONITORING USING THE PROTOTYPE OF WIRELESS ACCELEROMETER
2.1 Overview of the monitoring experiment
To investigate the daily variations of vibration characteristics, a road sign was constructed in our
university. The measurement schedule is shown in Figure 1.
First, the vibration responses were measured manually from December 8, 2016 to March 7, 2017. In thisperiod, a damage experiment was also conducted on January 27, 2017 to make clear whether the change in response
due to the damage could be detected or not. Major case examples of damage to road signs includes root corrosion,
bolt loosening, crack and deformation. These damages result in the decrease of eigenfrequency. In the experiment,
the thickness of the root part was decreased by 0.5 mm to simulate a root corrosion. The decrement of thickness
corresponds to about 12 % reduction in bending rigidity. The eigenfrequency of the first mode was analyzed as ananomaly detection index. As a result, the eigenfrequency decreased by 0.016 Hz after the damage.
During the period from June 6, 2017 to January 25, 2018, vibration response was automatically measured
using a prototype of wireless accelerometer which has been developed in our laboratory. This sensor consists of a
MEMS (Micro Electronic Mechanical Systems) accelerometer, wireless communication module, MCU (MicroController Unit), lithium-ion battery and small solar panel. In that period, acceleration responses were measured361

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 20192017/1/27
Damage experiments 1
12 % reduction in bending rigidity

2016/12/8

2017/3/7

2017/12/22
Damage experiments 2
5 % reduction in bending rigidity

2018/1/25

6/6

Now continue
Manualmeasurement

Automatic measurement
Figure 3
Figure 8

Figure 7
Figure 1.

Monitoring experiment schedule

450[mm]920[Hz] wireless module
Solar panel
3136[mm]

MEMS

Li-ion battery

Figure 2. Road sign and wireless accelerometer for monitoring experiment

Eigenfrequency [Hz]

4.8
4.7
4.6
4.5
4.4
4.3
4.2
2016/12/8~2017/3/7

2017/6/6~2018/1/25

1/27~2/4

2/10~3/11

3/31~7/20

7/20~9/28

2018/9/28~2019/4/18

Figure 3. Monitoring results of eigenfrequency of the first mode

every 4 hours at 100 Hz sampling rate for 3 minutes. The voltages of the solar panel and lithium-ion battery werealso observed every 5 minutes to check the condition of sensor. Figure 2 shows the prototype of the wirelessaccelerometer fixed to the road sign constructed in our university.
On December 22, 2017, another damage experiment was conducted with the aim of detecting the change
in vibration response due to the damage smaller than the previous one. In this experiment, the surface of the rootpart was additionally scraped off by 0.1 mm which corresponded to 5 % reduction in bending rigidity. This
monitoring experiment was continued up to January 25, 2018. After that the road sign was removed for safety.
The other road sign of the same type and size was newly constructed on the same day at the different location.
January 27, 2018, a long-term monitoring was started using the same wireless accelerometer as before and iscontinuing even now.
Figure 3 shows the temporal variation of the estimated eigenfrequency of the first mode. There are several
discontinuities which are mainly caused by the replacement of the sensor. The replacement of sensor changes its
mass and position, and results in the change in responses. Data lacks are also happened many times because of (a)362

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 201960

Temperature of battery [℃]

Temperature of battery [℃]battery exhaustion due to improper charging, (b) server shutdown, (c) radio receiver power-off, and (d) radio
interference. The details are given below.
(a) Battery exhaustion due to improper charging was occurred many times in summer season. In
consideration of safety, the sensor is set to stop charging when the temperature of the lithium-ion battery exceeds
40 ℃. Figure 4 and 5 show the temporal variations of the battery’s temperature and voltage, respectively. The left
figures represent the result of summer seasons and right figures winter season. From Figure 4, in summer season,
it is seen that there are many days exceeding the upper limit of battery temperature. Even if the air temperature is
below 40 ℃, the battery temperature easily exceeds 40 ℃ due to the heat generation during charging. Therefore,
enough charging time could not be assured in the daytime and monitoring at night could not be performed without
solar power generation. This is the reason why the monitoring was interrupted in summer season. In the experiment,
the monitoring was resumed by replacing a battery with a fully charged one when the battery voltage drops under
3.0 V. After the replacement, the voltage rapidly recovers as shown in Figure 5. On the other hand, in other reasons,monitoring was continued without the battery charging problem.
(b) Server shutdown and (c) radio receiver power-off sometimes happened. The acceleration datameasured by the wireless accelerometer was sent to a server settled in our laboratory about 60 m away from theroad sign. Sometimes, the mobile PC used as a server froze and the data was not be saved. In addition, a power
failure temporarily occurred at the time of equipment inspection. Since the wireless receiver obtained the powerfrom the electrical outlet, it didn’t work at that time.
(d) Radio interference sometimes occurred when multiple wireless accelerometers were used at the sametime. The data in that case are excluded from the analysis.
Table 1 shows the estimated results of the mean value and standard deviation of eigenfrequency of thefirst mode. From a series of experiments, it was found that the standard deviation of eigenfrequency of the firstmode was about 0.01 Hz.

50
40
30
20
10
0.0

Summer season-10
2017/6/29

8/8

9/23

60

50
40
30

20
10

0.0

Winter season-10
2017/10/1 10/28 11/15 12/4 12/21 1/11 1/25

9/30

4.5
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5 Summer season
0.0
2017/6/29

Voltage of battery [V]Voltage of battery [V]Figure 4. Monitoring results of voltage and temperature of battery

8/8

9/23

9/30

4.5
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5 Winter season
0.0
2017/10/1 10/28 11/15 12/4 12/21 1/11 1/25

Figure 5. Monitoring results of voltage of battery
Table 1.

Estimated results of eigenfrequency of the first mode

2016/12/8~2017/3/7

2017/6/6~2018/1/25

1/27~2/4

2/10~3/11 3/31~7/20 7/20~9/28

2018/9/28~2019/4/18

Ave.

4.57

4.65

4.27

4.43

4.55

4.65

4.65

std

0.02140.00982

0.01640.00851

0.02500.01210.0105

3. RESULTS OF LONG-TERM MONITORING OF ROAD SIGNS USING MEMS ACCELEROMETERS
3.1 Trial of other anomaly detection index
Optimal anomaly detection index might be stable in a general state and sensitive to the damage of structure.
To find out better anomaly detection index, we examined the power spectrum area ratio which obtained from the363Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

power spectrum of the acceleration.
Abe et al. (2016) note that when the structure is damaged, the peak of the power spectrum shifts to thelower frequency band. The algorithm is to calculate the area of low frequency band (𝑆𝐴𝐿 ) and take the ratio to thearea of whole frequency band (𝑆𝐴𝐴 = 𝑆𝐴𝐿 + 𝑆𝐴𝐻 ) including the low band. As shown in the Figure 6, if somedamage occurs in the structure, the peak of the power spectrum shifts lower frequency. And then, the power
spectrum area ratio, given in equation (1), approaches the minimum value of 1.
α=

𝑆𝐴𝐴 𝑆𝐴𝐿 + 𝑆𝐴𝐻
𝑆𝐴𝐻
=
= 1+
𝑆𝐴𝐿
𝑆𝐴𝐿
𝑆𝐴𝐿

(1)

𝑆𝐴𝐴
𝑆𝐴𝐿

Power spectrum [

/ ]

𝑆𝐴𝐻

ー：undamaged
ー：damaged

Frequency [Hz]Figure 6. Conceptual diagram of power spectrum area ratio
3.2

Monitoring results of an anomaly detection index
Figure 7 shows the monitoring results of each anomaly detection index from December 2016 to March2017. In this period, on January 23, 2017, the surface of the root part was scraped off by 0.5 mm whichcorresponded to about 12 % reduction in bending rigidity. There is a clear difference after the damage.
Figure 8 shows the other results during June 2017 to January 2018. On December 22, 2017, the thicknessof the root part was decreased by 0.1 mm which corresponded to about 5 % reduction in bending rigidity. In thiscase, it is difficult to detect the change due to the damage simply from the change in the mean value because the

2016/12/8

2017/1/27

2016/12/8

3/7

2017/1/27

3/7

Figure 7. Monitoring results of anomaly detection index (12 % reduction in bending rigidity)

2017/6/6

12/22 2018/1/25

2017/6/6

12/22 2018/1/25

Figure 8. Monitoring results of anomaly detection index (5 % reduction in bending rigidity)364

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Moving average [Hz]

4.7

4.6

4.5
10 days

4.6

4.5
20 days

4.4

4.4

4.7

4.7

Moving average [Hz]Moving average [Hz]Moving average [Hz]

4.7

4.6

4.5
10 days
4.4

4.6

4.5

20 days
4.4

Figure 9. Moving average of eigenfrequency(Upper: About 12 % reduction in bending rigidity, Lower: About 5 % reduction)
variation of the index is larger than the amount of change.
Moving average was also calculated to make clear the trend of change in indexes. Figure 9 shows theresults of moving averages for 10 and 20 days. The x- and y-axis are the same to that of Figure 7 and 8. In the
Figure 8, it is difficult to detect the change in the index in case of stiffness reduction of about 5 % (small damage).
However, calculating the moving average make it easier to see the trend.
4. CHANGE POINT DETECTION METHOD OF ANOMALY DETECTION INDEXIn section 3.2, moving averages are shown as a method to suppress the influence of temporal variation ofindexes. However, if the noise level is larger than the change in the index due to damage, it is still difficult to
detect the change in the index.
Therefore, we applied the density ratio direct estimation method (hereinafter simply referred to as “density
ratio estimation”) to the time series of anomaly detection indexes. Density ratio estimation is proposed by
Kawahara et al. (2012), which directly estimates the ratio of probability density functions estimated from thenormal data and the following abnormal data. This method can be applied to the problem of detecting outliers orchange points in time series data.
The ratio of probability density function is calculated by equation (2). In the algorithm, first, the normal
data (𝑌𝑟𝑓 : reference samples) and test data (𝑌𝑡𝑒 : test samples) are defined from time series. And then, the kernel
model defined as equation (3) are estimated by using all data. In actual monitoring, the data obtained later will betest samples. In the equations, 𝐾𝜎 represents a Kernel function found based on 𝑌𝑟𝑓 , 𝛼𝑙 is a weight parameter,and σ is determined by LCV (Likelihood Cross Validation).
Figure 10 shows the estimated ratio of probability density function. This result is obtained using the samedata to the result shown in section 3. In this study, Kernel functions is obtained with the 𝑌𝑟𝑓 which consists of 10
samples selected from the undamaged data. In figure 10, there is a significant difference in the index before andafter the damage corresponding to about 12 % reduction in bending rigidity. On the other hand, it is not possible
to confirm a difference enough to detect the damage before and after the damage corresponding to about 5 %reduction in bending rigidity.
𝑛𝑡𝑒𝑤
̂ (𝑌) = ∑ 𝛼𝑙 𝐾𝜎 (𝑌𝑟𝑓 , 𝑌𝑡𝑒 (𝑙))

(2)

𝑙=1

2

‖𝑌𝑟𝑓 − 𝑌𝑡𝑒 ‖𝐾𝜎 (𝑌𝑟𝑓 , 𝑌𝑡𝑒 ) = exp (−
)
2𝜎 2

365

(3)

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Eigenfrequency

2016/12/8

2017/1/27

Power spectrum area ratio

3/7

2016/12/8

Eigenfrequency

2017/6/6

2017/1/27

3/7

Power spectrum area ratio

12/22 2018/1/25

2017/6/6

12/22 2018/1/25

Figure 10. Estimated results of density ratio(Upper: About 12 % reduction rate in bending rigidity, Lower: About 5 % reduction rate)
5. CONCLUSIONSIn this paper, we described a results of monitoring experiments in which a road sign was monitored by theanomaly detection system using MEMS wireless accelerometer.
First, a prototype of wireless sensor equipped with a MEMS accelerometer was developed and fixed to
the road sign constructed in our university. The acceleration responses were measured automatically for about 18
months. As a result of long-term monitoring, the standard deviation of the eigenfrequency of the first mode wasabout 0.01 Hz. We also examined the power spectrum area ratio as an anomaly detection index. In addition,
damage experiments were conducted to make clear whether the change in response due to the damage could be
detected or not. In the case of 12 % reduction in bending rigidity, changes in the index could be detected. On theother hand, in the case of 5 % reduction in bending rigidity, the noise level is larger than the change in the index
due to the damage. Therefore, it is difficult to detect the change in the index. Furthermore, moving average anddensity ratio estimation were examined as a method to detect the change of the index. In the future, we try to findout better anomaly detection index. We will also consider a method to detect the change of the index.
The long-term monitoring also revealed various problems of the wireless accelerometer developed in thisresearch. At present, monitoring has been interrupted many times. In the future, we will try to suppress the rise inbattery temperature in summer season. In addition, we also solve the other problems to acquire continuous data.
REFERENCES
Ozaki, D., Tsuji, T., and Saeki, M. (2017): BASIC STUDY ON ANOMALY DETECTION OF ROAD SIGNSUSING MEMS ACCELEROMETERS, The 3rd International Conference on Civil and Building
Engineering Informatics in conjunction with 2017 Conference on Computer Applications in Civil andHydraulic Engineering, Taipei, Taiwan, April 19-21．
Abe, K., Natori, T., Kominato, Y., Sekiguchi, T., Yamano, A., and Lin Wang. (2016). Monitoring Method ofSoundness of Railway Bridge by Using Soundness Diagnosis Indices Correlated with NaturalFrequency, Journal of Japan Society of Civil Engineers, Ser. A1 (Structural Engineering & Earthquake
Engineering (SE/EE)), Vol.72, No.1, 21-40. (in Japanese)
Kawahara, Y., and Sugiyama. M. (2012). Sequential Change-Point Detection Based on Direct Density-Ratio
Estimation, Statistical Analysis and Data Mining, vol.5, no.2, pp.114-127.

366

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Laser and Image Scanning367

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

AUTOMATIC INDOOR ENVIRONMENT MODELING FROM LASER-SCANNED POINT
CLOUDS USING GRAPH-BASED REGULAR ARRANGEMENT RECOGNITION
Hayato Takahashi1, Hiroaki Date2, and Satoshi Kanai 31) Master Course Student, Graduate School of Information Science and Technology, Hokkaido University, Sapporo, Japan.
Email: h_takahashi@sdm.ssi.ist.hokudai.ac.jp
2) Ph.D., Assoc. Prof., Graduate School of Information Science and Technology, Hokkaido University, Sapporo, Japan. Email:hdate@ssi.ist.hokudai.ac.jp
3) Ph.D., Prof., Graduate School of Information Science and Technology, Hokkaido University, Sapporo, Japan. Email:kanai@ssi.ist.hokudai.ac.jp

Abstract: In this paper, an automatic regularized 3D modeling method of indoor environments from laserscanned point clouds is proposed. The method can efficiently create 3D indoor environment models which consistof a floor, a ceiling, and walls with regular arrangements of orthogonal, parallel, coplanar, and regular intervals.
In the method, first, a floor and ceiling are extracted by plane fitting and parallelized by the least-square fitting.
Then, wall line segments on the ceiling plane are estimated from boundary points on the ceiling. Next, graphs forthe wall line segments are successively constructed to recognize the regular arrangements of the walls. In the graph,a node corresponds to one or more wall line segments, and an edge represents regular arrangement relationships
between the walls. The resulting graph expresses the simplest regular arrangement relationships of the walls, andmodified wall line segments are obtained by the least square fitting of lines to corresponding points under theconstraints of the regular arrangements. Finally, a 3D model is created by sweeping the wall line segments along
a vertical direction. Experimental results using point clouds from the laser scan simulation of a 3D model and laser
scanning of a real indoor environment show the effectiveness of the proposed method.
Keywords:

Laser scan, point clouds, 3D indoor environment modeling, regular arrangement.

1. INTRODUCTION
Currently, middle and long-range laser scanning technology is used in a wide range of areas, such as plant,
civil engineering, architecture, surveying, and mapping. The technology allows us to acquire dense and precise 3D
point clouds of large-scale objects, structures, and environments. In the plant and architecture areas, 3D modelingof artificial structures using laser-scanned point clouds is now becoming common. The resulting precise 3D models
are used in drawing, renovation planning, and maintenance. Manual 3D modeling from the point clouds is verytime consuming, therefore automatic or semi-automatic modeling methods are strongly required. We focus on the
3D modeling of indoor environments from laser-scanned point clouds.
Many 3D modeling methods of indoor environments based on 3D laser scanning technology have been
proposed (Díaz-Vilariño et al., 2015; Hong et al., 2015; Jung et al., 2014; Monszpart et al., 2015; Nan et al., 2010;
Previtali et al., 2018; Shi et al., 2019; Valero et al., 2012). These methods can produce 3D models automaticallyor semi-automatically from point clouds. The researches solve several problems for 3D model generation frompoint clouds, such as handling noisy data and large data sets, interpolating occluded regions, robust extraction ofprimitives, recognizing and imposing several regular arrangements, recognizing additional structures on the wallsuch as windows and doors, recognizing semantics, and useful CAD data (B-rep) generation.
In our research, we mainly tackle a problem to recognize, represent, and impose the useful regulararrangements of walls in indoor environment modeling. Artificial structures and man-made objects such as indoor
environments often have regular arrangements, such as orthogonal and regular intervals. Therefore, considering
regular arrangements in indoor modeling has many advantages. For example, firstly, recognition of regulararrangements from point clouds helps to understand the scanned environments (Monszpart et al., 2015). Secondly,
the regular arrangements allow us to make the model representation compact. In addition, using regulararrangements often simplify and stabilize the modeling and recognition processes (Nan et al., 2010; Previtali et al.,2018). Although the models with regular arrangements have relatively lower accuracy compared with the models
generated by individual precise surface fitting, they are useful for several applications, such as efficient andbeautiful visualization of the environments, facility arrangement planning, drawing creation for renovation
planning, air conditioning simulations, and inspection of regularities of real environments.
Many methods for indoor modeling impose or use orthogonal relationships of walls, ceiling and floors
under Manhattan-world assumptions. However, co-planar and regular intervals are not often handled (e.g. Honget al., 2015), although many walls in the buildings follow those relationships. Monszpart et al. (Monszpart et al.,2015) proposed a method for finding and imposing several regular arrangements of planes. A mixed-integer
program is used to find the regular arrangements, and robust extraction of regular arrangements for orthogonal andparallel planes is achieved. However, it is difficult to handle several types of regular arrangements because of its
time complexity. Previtali et al. (Previtali et al., 2018) proposed a modeling method of windows and doors on thewalls considering the same shape, same alignment, and same interval. The regularity found by simple line-distance368

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Plane extraction parameters etc.
Ceiling and floor
detection andparallelization
A1- RANSAC- PCA

Tolerance of contour
detection etc.

Tolerance forsimplification
etc.

Wall line
segment
extraction

A2- Contour detection- RANSAC

Point clouds

Ceiling andfloor planes

Ceiling points

Regularizationof wall lines
A3
Graph-based
regularization

Wall line segments

Regularizedwall line
segments
3D modelgeneration

3D model(polygon)
A4

Sweep

Figure 1. Flow of the proposed indoor environment modeling method
evaluations is used as constraints for object’s parameter determination. The method can be extended to walls, but
relationships of the objects are not explicitly represented, and over-constraints and compact representation of thearrangements are not discussed.
In this paper, an automatic and efficient indoor environment modeling method considering four useful
regular arrangements of walls, i.e. parallel, orthogonal, coplanar and regular interval, is proposed. In the method,
initial walls are detected from the points on the ceiling in order to realize robust modeling for occlusions. The
regular arrangements of the walls are detected by the successive construction of graphs. The detection method isbased on the one proposed by Li et al. (Li et al., 2011). We extended the method for handling coplanar and regularinterval of walls. In addition, simple primitive fitting methods considering the detected regular arrangements ofwalls are described for efficient modeling. Our method is applied to synthesized data from the laser scan simulation
and real laser-scanned point clouds, and the effectiveness of the method is quantitatively shown with experimental
results.
2. INDOOR MODELING METHOD CONSIDERING REGULAR ARRANGEMENTS
2.1 Overview of the method
The input of our modeling method is registered laser-scanned point clouds of indoor environments. In our
research, we assume that the ceiling and floor are parallel, the ceiling can be approximated by flat regions, andwalls are orthogonal to the ceiling under the Manhattan-world assumptions. Four regular arrangements for walls,
i.e. orthogonal, parallel, co-planar and regular intervals, are achieved in the modeling. In our method, initial walls
are estimated from contours of points of the ceiling in order to make our wall modeling method robust to obstacles
near the walls such as shelfs and furniture. The walls are represented by straight line segments on the ceiling
planes, and their regular arrangements are detected using the graph. In the graph, each node corresponds to asingle wall or multiple walls with regular arrangements. Successive graph construction based on simple rules
detects the regular arrangements without redundancy and over-constraints, and minimum parameters for the
regular arrangements are also extracted. The wall line segments are modified by the least square fitting to thecorresponding points under constraints of the detected regular arrangements.
The flow of our modeling method is shown in Figure 1. In the method, first, the ceilings and floor planes
and corresponding points are recognized by using a well-known plane fitting method. Then, wall lines which areprojections of the walls onto the ceiling, are extracted by a contour detection method and line fitting to theboundary of ceiling points. Next, regular arrangements of wall lines are recognized using a graph-based method,
and wall lines are modified according to the recognized regular arrangement. Finally, a 3D indoor model isobtained by sweeping the wall lines by the height of the room. In the regularization step, first, orthogonal andparallel relationships are extracted, and wall lines are modified so as to be parallel and orthogonal by constrained
fitting of the lines to the points. Then, for parallel lines, coplanar and regular intervals are detected.
In the following sections, details of each step (A1~A3) shown in Figure 1 except for the final step (A4)are described, because the final step is a simple sweeping operation of the resulting wall line segments by theroom height.
2.2

Ceiling and floor detection and parallelization
In the ceiling and floor detection (Figure 1, A1), first, using RANSAC (Random Sample Consensus)
which is a robust model fitting method based on the iteration of hypothesis and verification (Fischler & Bolles,
1981), planes are fitted to the point clouds. Then, via thresholding for number of points on the planes and normal
direction evaluation, horizontal large planes are extracted. The planes with heights larger than a threshold arerecognized as ceiling planes, and a plane with the lowest height is recognized as a floor. Then, a normal vector for369

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Input wall
primitives
w1
w6

Resulting Graph
w2

w1,5

w2

w4

w3,6

w4

w3
w5

null
Insert(w5, w6)(a) An example of a graph for orthogonal and parallel

w5

w5

w5

w5

w2

w6

w6

w3,6

w3,6

w4

InsertCollapse(w3, w6) w
3

Insert(w2, w4)

Parallel
Orthogonal(b) Graph construction process

Figure 2. Graph representation of the orthogonal and parallel relationship of the wall lines
the ceiling and floor planes are determined by fitting planes with the normal based on the least square method. As
a result, parallel planes for ceilings and floor are obtained.
Wall line segment extraction
In the wall line segment extraction (Figure 1, A2), first, a binary image similar to the existing methods(Hong et al., 2015; Shi et al.,2019; Valero et al., 2012) is created by projecting points of the detected ceiling planes
onto the highest ceiling plane, grid definition, and occupied cell identification. Then, contours of the image are
extracted by the Suzuki algorithm (Suzuki & Abe, 1985) and corners of the contours are extracted by the Douglas
Peucker method (Douglas & Peucker, 1973) which is the polygon approximation method. Finally, wall line
segments {wi} are extracted via line fitting by RANSAC to projected points between the corner points detected by
the Douglas Peucker method. The inliers of RANSAC are assigned to each wall line segment wi as corresponding
points Pi.
2.3

Regularization of orthogonal and parallel line segments
In the regularization of wall lines, first, orthogonalization and parallelization are applied to the initial wallline segments. In our method, based on the method proposed by Li et al. (Li et al., 2011), the graph which represents
the orthogonality and parallelism of lines are constructed. Figure 2(a) shows an example of the graph representing
orthogonal and parallel relationships. Each node corresponds to one or more than one line segments. Multiple
segments in a node are recognized as parallel relationship, and the edge represents orthogonal relationship between
segments in end nodes. Mutiple orthogonal frames can also be handled. In the graph, the connected component
corresponds to each orthogonal frame.
The graph is constructed by successively adding line segment pairs. Before graph construction, in orderto determine line pairs to be added to the graph and the order of addition, the relationship of each line segment
pair is roughly recognized, and the score of each line segment pair for regularity is calculated. First, the relationshipof each line segment pair (wi, wj) is roughly recognized according to the smaller intersection angle θij of the linesof the pair. If θij is less than π/4, the pair (wi, wj) is classified as parallel. The others are classified as orthogonal.
Then, a score of the regular arrangement, sij, for each line pair (wi, wj) is calculated as follows.
2.4

(1)

The value of sij becomes larger for the line pairs which are closer to the ideal state for the classified typeof regular arrangement.
An example of this graph construction process is illustrated in Figure 2(b). After the score calculation, thepair with the maximum sij is successively added to the graph. In this addition, if a line segment of the pair to be
added is not in the current graph, a node of the line segment is created. The edge between two nodes corresponding
to the line segment pair are added while assigning the label of the parallel of orthogonal. In this process, redundantnodes and edges are not added. After the addition, if an edge with the parallel label is created, the edge is collapsed.
The addition of the pair is iterated until all pairs with sij larger than a given threshold are added to the graph.
Modified wall line segments are determined independently for the segments in each connected component
which corresponds to an individual orthogonal coordinate frame. In the modification, as shown in Figure 3,
direction vectors of two orthogonal lines are first determined, and positions of lines are then decided. First, thepoint sets Pi of each line segment wi in a connected component of the graph are translated so that its barycenter
becomes an origin as shown in Figure 3(b). Here, a combined set of points corresponding to line segments in anode after this translation is denoted by P, and the translated points of the other node are denoted by O. Then,
using the following minimization, direction vectors orthogonal to each other (d and Rd) can be determined.

370

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

wi

y

wi,k

wj

wj
d

wj,l

wl

wi
x
Rd

wk

(a) Lines to be orthogonal/parallel
and corresponding pointswl(b) T ranslated points andline fittingwk(c) Modified lines

Figure 3. Wall line modification to satisfy parallel and orthogonal relationships2
min  ∑ (dpi ) 2 + ∑ ( (Rd)pi ) 
d
i∈O
 i∈P


(2)

Here, || d || =1, pi is the (translated) position of point i, and R is a rotation matrix for rotating 90 degrees.
After deriving the d, based on the least square fitting manner, lines with a directional vector d or Rd passing thebarycenter of each point set Pi are created as modified wall line segments as shown in Figure 3(c).
2.5Regularization of coplanar and regular interval line segments
Coplanar constraints are imposed to a set of parallel line segments whose distances are 0, and regularinterval constrains are applied to a set of line segment pairs which have the same interval. In the regularization for
coplanar and regular intervals, directions of wall line segments are not changed, and only the positions of each line
are modified. For simplicity, in the following part of the explanations, we represent the position of each line
segment wi by a scalar di along its normal direction. In our method, regularization for the coplanar and regularintervals is done in three steps of (1) grouping line segments, (2) relationship extraction using the graph, and (3)line segment fitting under regular arrangement constraints.
(1) Grouping line segments
Coplanar and regular intervals are detected based on the distance between line pairs. Therefore, first, foreach parallel line pair (wi, wj), the unsigned distance between the lines sij is calculated by Equation (3).

s=| di − d j |ij(3)Then, the wall line pairs are sorted by an increasing order of sij. A set of pairs with sij less than τ (in our(i =experiments, τ = 30 mm is used) is recognized as a coplanar group GC. Then, regular interval groups
1,…,n) are created from remaining pairs. In this process, if difference of sij of neighboring pairs is less than a given
threshold τ, they are grouped. As a result, a set of initial groups for regularization are obtained as shown in Figure4(b).
(2) Relationship extraction using the graph
Satisfying the co-planar relationship is completed by finding common positions {ck} for each set of
coplanar line segments in GC as shown in Figure 4(c) left. On the other hand, satisfying the regular intervalasrelationship is completed by finding base positions { } for the pairs and common intervals {gi} in eachshown in Figure 4(c) right (in the figure, for simplicity, some relationships such as regular intervals related tocoplanar lines are not represented). However, as shown in the figure, for example, the interval ge can be represented
by other intervals, i.e. ge = gb + gc, and some base positions can be represented by combinations of others. Deriving
intervals independently may result in over constraints. To remove these redundancies of the parameters and to findthe minimum parameters which represents all regular arrangements, in our research, successive graph construction
is used. The node in the graph represents a wall line segment or a set of wall line segments, and each edge has aninterval gi for regular intervals as an attribute.
Graph construction is done by adding nodes and edges of the groups with a smaller average of {sij}successively, according to the following rules. Figure 4(e) shows an example of the graph construction.
Coplanar group[process for nodes] Add nodes corresponding to each wall line segment pair. If the node corresponding to a wallline to be added already exists in the graph, a new node is not created.

371

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019Line segment pair (sorted by sij)(a) An example ofwall line segments(b) Line segment pair groups(c) All parameters to be determined forcoplanar and regular interval

null

wX,Y a node corresponding both lines wX and wY(d) Resulting parameters to be determinedfor coplanar and regular interval(e) Graph construction process

Figure 4. Graph construction for coplanar and regular interval wall lines[process for edges] Add edges corresponding to the nodes of the pair and collapse the edge. Here, a self-loop edge
is not created. As a result, each node includes coplanar wall line segments.
Regular interval group[process for nodes] Check whether two nodes of the pair are already included in the same connected component
in the current graph. If the nodes are included, represent its interval gi by the sum of the parameters of theother groups in the current graph. If not, create the nodes and add them into the graph.
[process for edges] Create an edge with an attribute gi between the node pair which is not included in the sameconnected component. Doubly lined edges and self-loops are removed. If only an edge whose endpoint is
a node of coplanar line segments is created after adding pairs of a group, the edge is removed because theregular interval relationship can be satisfied by a coplanar relationship.
Figure 4(e) shows an example of the resulting graph for wall segments in Figure 4(a). Figure 4(d)represents the results of detected basic components of regularities and parameters to be determined. One baseis selected in each connected component of the graph. It can see that compact and minimum
position
representation of the regular arrangements is obtained from the graph.
(3) Line segment fitting under regular arrangement constraints
Using the resulting graph, a position dj of a wall line segment wj in a connected component along thecorresponding normal direction ni can be represented by using a position bi of a base wall line wi and interval gl
by using Equation (4).
(4)

is an integer coefficient determined by the path from the node of the base wall line segment to
Here,
wj. A set of line segments which satisfies the coplanarity and regular interval and fits to corresponding point clouds
can be derived by minimizing an energy function E of residuals between points and lines defined by Equation (5)with respect to positions {bi} of base wall line segments and the intervals {gl} of each regular interval.
(5)Here, C is a set of the connected components of the graph, Ci is a set of wall line segments in the i-th
connected component, Oj is a set of points corresponding to the wall line segment j, pk is the position of a point kalong the corresponding normal direction ni. The results are independent from the selection of the base wall line372

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Table 1. Point clouds used in the experiments and processing times
Dataset A
Dataset B

Source#scans / #points
Room size
Processing time[s]Blensor (simulation)
3 / 1.6M
8m x 5m
1.8

FARO FOCUS 3D S120
5 / 133.4M
21m x 8.5m
195.1
R2 R3 R5(a) Original model and point cloud(c) Parallel and orthogonal
R1
R2
R3
R4
R5

(b) Resulting 3D model(d) Coplanar(e) Regular interval

Figure 5. Point clouds and experimental results of Dataset A
R3(a) Point Cloud

R1
R2
R3
R4
R5
R6

(c) Parallel and orthogonal

(II)
(I)

(b) Resulting 3D model(d) Coplanar(e) Regular interval

Figure 6. Point clouds and experimental results of Dataset B
segment, therefore, we select the base wall line randomly in each connected component. The minimization of Ewith respect to {bi} and {gl} can be solved by the least square method.
3. EXPERIMENTS AND EVALUATIONS
The proposed method is applied to two datasets. Dataset A is created by a laser-scanning simulation of a
CAD data of a room including explicit regular arrangements by a scan simulator (Blensor). The point cloud ofDataset A includes scanning errors of 3 mm at 25 m (three sigma in Gaussian noise). Dataset B is acquired bylaser scanning of real environments using a terrestrial laser scanner, FARO FOCUS 3D S120. Information of eachdataset is summarized in Table 1, and point clouds are shown in Figure 5(a) and 6(a). In Figures 5(b) and 6(b), theresulting 3D models with regular arrangements are shown. The top views of detected regular arrangements of thewalls are shown in (c)-(e) of each figure.
From the modeling results shown in Figures 5(b) and 6(b), it was confirmed that the walls are correctly
estimated, and 3D models of indoor environments are automatically generated from the point clouds by our method.
Using the contour of points on the ceiling could estimate the appropriate wall lines from large occlusions of thewalls by tall shelfs in Dataset B. In the resulting 3D model of Dataset B, few incorrect narrow walls (short walllines) were observed. The cause of them can be considered as the lack of points of the ceiling by occlusions andlarge incident angles of the laser. Beautification for removing small wall line segments will solve these problems
and is included in future works.
Figures 5(c) and 6(c) show detected parallel and orthogonal wall lines. Lines drawn by the same color aredetected as parallel lines, and orthogonal relationships of lines with different colors are also recognized. Figures5(d) and 6(d) represent detected coplanar walls. Walls with the same color are recognized as coplanar. Figures5(e) and 6(e) show detected regular interval relationships. In the figures, a dashed line with a label represents agroup of regular intervals. Red thick lines of each group are detected regular intervals. From the results, it was373

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Photo and point cloudResulting 3D model

Point cloud andresulting 3D model(a) A window side

Photo and point cloudResulting 3D model

Point cloud andresulting 3D model(b) A wall with tall shelves

Figure 7. Comparison of photos, point clouds and resulting 3D models

Dataset A
Dataset B

Table 2. Parameters used in the experiments
RANSAC for extracting
RANSAC for estimating
ceilings and floors
wall line segments
The number Inlier distance The number Inlier distanceof iterations threshold[mm] of iterations threshold[mm]
100
25
100
5
100
25
100
5

Pixel sizeof binaryimage[mm]Grouping
thresholdτ [mm]

45
15

30
30

observed that all regular arrangements defined on the 3D model used in the scan simulation for creating Dataset
A could be correctly recognized. For Dataset B, although the ground truth of the all regular arrangement are notclear, almost major regular arrangements which can be recognized by visual confirmations, such as regularintervals for the side faces of the two columns shown in the lower part of the room in Figure 6(c), were detected.
In our method, regular intervals were detected from two parallel planes whose intervals were larger than thegrouping threshold τ = 30 mm. Some unexpected regular intervals also were found by the method. These results
show that the proposed method is useful to find four kinds of regular arrangements of the walls from the point
clouds of indoor environments.
Figure 7 shows photos, point clouds, and resulting 3D models of two different views (in Figure 6(b), (I) awindow side and (II) a wall with tall shelves) in Dataset B. The walls were robustly estimated from point clouds
where the walls are not sufficiently scanned by window glasses and occlusions. The modeling error was checkedby measuring distances between some parallel walls. To check the nominal distances, a laser range finder is used
for measurement of the room of Dataset B. Three distances were evaluated for Dataset A, and the errors were 8,
16, and 24 mm. Four distances were measured for Dataset B, and the errors were 4, 13, 9, and 3 mm. Theoretically,
the errors depend on the scanning errors, registration errors, and parameters in some approximation and fitting
operations of the algorithm, such as grid sizes of binary image generation and inlier distance thresholds for
RANSAC in wall line segment extraction. The parameters used in the experiment are summarized in Table 2.
Although the parameters in our experiments are determined experimentally, optimal parameter decisionconsidering point density and accuracy will result in more accurate modeling results.
The processing times for modeling from point clouds are shown in Table 1. In our method, a 3D modelfrom point clouds with 133 million points could be created in about three minutes using a standard PC (CPU:Core-i7 4790, RAM: 32GB). The results show that our method can efficiently create a 3D model from point cloudsof practical size. Further evaluations for several indoor environments are included in future works.

374

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

4. CONCLUSIONSIn this paper, a 3D indoor modeling method from laser-scanned point clouds considering four kinds ofregular arrangements of the walls is proposed. Parallel, orthogonal, coplanar, and regular intervals are detectedusing a graph-based representation of the relationships, and a minimum representation (parameters) is also
extracted as a result of successive graph construction. The least square fitting of wall line segments underconstraints from regular arrangements and point cloud approximation derives the final wall line segments ofregularized models. The experimental results for point clouds from a scan simulation showed that our method
could extract all regular arrangements in the original room model. Major regular arrangements which can berecognized by visual confirmations in real environments are detected from the laser scanned point clouds. The 3Dmodels with regular arrangements could be efficiently generated by our method from point clouds of practical data
size. Future work includes investigating the best parameter setting according to point cloud quality, application toseveral indoor environments, and detail structure modeling such as windows and doors considering regulararrangements.
REFERENCES
Blensor (2018). https://www.blensor.org
Díaz-Vilariño, L., Khoshelham, K., Martínez-Sánchez, J., and Arias, P. (2015). 3D modeling of building indoor
spaces and closed doors from imagery and point clouds, Sensors, 15, 3491-3512.
Douglas, D. H. and Peucker, T. K. (1973). Algorithms for the reduction of the number of points required torepresent a digitized line or its caricature. Cartographica: The International Journal for Geographic
Information and Geovisualization, 10 (2), 112-122.
Fischler, M. A. and Bolles, R. C. (1981). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM, 24 (6), 381-395.
Hong. S., Jung, J., Kim, S., Cho, H., Lee, J., and Heo, J. (2015). Semi-automated approach to indoor mappingfor 3D as-built building information modeling, Computers, Environment and Urban Systems, 51, 34-46
Jung, J., Hong, S., Jeong, S., Kim, S., Cho, H., Hong, S., and Heo. J. (2014). Productive modeling fordevelopment of as-built BIM of existing indoor structures, Automation in Construction, 42,68-77.
Li, Y., Wu, X., Chrysanthou, Y., Sharif, A., Cohen-Or, D., and Mitra, N. J. (2011). GlobFit: consistently fitting
primitives by discovering global relations, ACM Transactions on Graphics, 30 (4), Article No.52.
Monszpart, A., Mellado, N., Brostow, G. J., and Mitra, N. J. (2015). RAPTER: Rebuilding man-made scenes
with regular arrangements of planes, ACM Transactions on Graphics, 34 (4), Article No. 103.
Nan, L., Sharf, A., Zhang, H., Cohen-Or, D., and Chen, B. (2010). SmartBoxes for interactive urban
reconstruction, ACM Transactions on Graphics, 29(4), Article No. 93.
Previtali, M., Díaz-Vilariño, L., and Scaioni, M. (2018). Towards automatic reconstruction of indoor scenes fromincomplete point clouds: door and window detection and regularization, International Archives of the
Photogrammetry, Remote Sensing and Spatial Information Sciences, 42(4), 507-514.
Shi, W., Ahmed, W., Li, N., Fan, W., Xiang, H., and Wang, M. (2019). Semantic geometric modelling of
unstructured indoor point cloud, International Journal of Geo-Information, 8 (1), 9.
Suzuki, S. and Abe, K. (1985). Topological Structural Analysis of Digitized Binary Images by Border
Following, Computer Vision, Graphics and Image Processing, 30 (1), 32-46.
Valero, E. Adán, A., and Cerrada, C. (2012). Automatic method for building indoor boundary models fromdense point clouds collected by laser scanners, Sensors, 12, 16099-16115.

375

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Detecting Building Façade Deteriorations: Evaluation of 3D Laser Scanning and Imagebased Reconstruction Approaches to Determine Feasible Settings in Data Collection
Zhuoya Shi1, Semiha Ergan2
1) Ph.D. Candidate, Department of Civil and Environmental Engineering, NYU Tandon School of Engineering, Brooklyn, NY,
USA. Email: zhuoyash@nyu.edu
2) Ph.D., Assist. Prof., Department of Civil and Environmental Engineering, NYU Tandon School of Engineering, Brooklyn,
NY, USA. Email: semiha@nyu.edu

Abstract: To identify façade conditions that may cause public injury, many cities mandate façade inspection
programs. With around one million aging buildings, New York City (NYC) requires buildings more than six stories
to be inspected every five years. Thus, a safe, comprehensive, and objective inspection method is important. Laser
scanners can capture as-is conditions rapidly and safely, while terrestrial laser scanners are influenced by the highincidence-angle in capturing detailed crack information at required heights. Images captured through drones or
stationary equipment at elevated heights can aid with the height problem by generating point clouds via 3D
reconstruction technology; however, is influenced by the accuracy of reconstruction algorithms. The objective of
this study is to compare the point clouds obtained from these two modalities of technologies for the visibility andmeasurement accuracy of cracks and define the settings of equipment in capturing the right quality of point clouds
at all floors including six levels and up. The experiments use a series of simulated cracks with different widths andorientations. These cracks were generated at sizes and thresholds indicated in building inspection programs (e.g.,
Local Law 11/98 in NYC) to understand the capability of two technologies in detecting such thresholds. The rootmean-square error in the measurement of crack lengths as compared to the ground-truth data was utilized toevaluate the accuracy of crack detection using point clouds obtained with these two modalities. Findings are
influential for choosing and utilizing technologies at the right settings for effective periodical evaluations ofbuilding façades.
Keywords: building façade inspection, 3D laser scanning, 3D reconstruction.
1. INTRODUCTION
In the densely populated cities like New York City (NYC), the façade inspection program is mandated toensure public safety. The Department of Buildings (DOB) of NYC requires buildings more than six floors to beinspected every five years (DOB, 2011). There are more than 14,000 buildings that need to be inspected in thecurrent inspection cycle at NYC (DOB, 2019). Current inspection regulations require qualified inspectors to
conduct visual inspections, take photos, and draw sketches of façade conditions during an inspection (DOB, 2011).
After each inspection, DOB requires a report submission with a description of the building condition, a summaryof findings, and the assessment of the inspector on the safety status of the building. Images and sketches attachedto the report can keep a record of the defects identified in the current inspection circle and provide a baseline forcomparison for the next inspection cycle. The existing work process is labor-intensive and time-consuming in datacollection and subjective in assessing the safety/condition of façade components. For example, the inspector needs
at least 1.5 hours to inspect only one side of an apartment building for the first inspection cycle; more time will beneeded for the comparison work if there is a previous inspection from the last cycle. Façade inspection reports
submitted to related city agencies need to go through both administrative and plan examiners’ reviews for
acceptation or rejection of the façade condition assessments. Several conditions can lead to the rejection of reports
such as when the assessment results are inconsistent with the images of a building’s façade, and when clear color
images for the reported assessment results are missing (DOB, 2019). Thus, an objective dataset that can record
façade component details and provide evidence for the condition rating of façade components is necessary for boththe inspection and the review process. A process that reduces the labor-intensive and time-consuming nature of
the inspection process is needed.
Point cloud data that can capture the as-is conditions of façades can help with the façade inspection process
by quickly capturing the required data, accurately documenting the building and component conditions, objectively
providing evidence for comparison to the data and results from previous inspection cycles. 3D point cloud data ofa surface can be obtained through 3D laser scanning and 3D reconstruction from images. Earlier studies thatleveraged point cloud data from laser scanners (e.g., Tang et al., 2009; Laefer et al., 2014) and image reconstruction
technology (e.g., Torok et al., 2014) resulted in algorithms to detect cracks and spalls mainly on concrete surfaces(e.g., beams, columns) and pavements, or façades with less than four floors. These studies also evaluated several
influence factors that change the quality of the data for defect detection (e.g., Kavulya et al., 2011). However, fordefect detection specifically on building façades, the influence of the vertical distance between defects andscan/camera locations on the visibility of defects and the measurement accuracy has not been studied.
In this study, we designed an experiment to compare the point cloud data obtained with terrestrial laser376

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019scanners and 3D reconstruction technology to support the façade inspection process. Foam boards with simulated
cracks of different widths and orientations were attached to the façade of a building at different elevations. Several
influence factors, i.e., crack size and orientation, the horizontal incidence angle, the distance between scan/camera
locations and the façade, the vertical distance on the façade where cracks were simulated, and scan/camera
resolution were considered to collect datasets to evaluate the visibility of the cracks for measurements. The results
can help with guiding point cloud-based façade inspection for determining the feasible boundaries of datacollection for comprehensive and accurate façade inspections.
2. BACKGROUND RESEARCH
Terrestrial laser scanners are capable of capturing the as-is condition of surfaces with millimeter level ofaccuracy, and have long been studied for damage estimation, defect detection, and health monitoring of a variety
of infrastructure elements such as pavement, tunnels, bridges, and buildings (Tang et al., 2009; Tsai and Li, 2012;
Kim et al., 2014; Yu et al., 2014; Laefer et al., 2014; Sedek and Serwa, 2016). Different algorithms were compared
for their ability to automatically detect and quantify the spalling and cracks on concrete surfaces (Tang et al., 2009;
Kim et al., 2014). The feasibility of applying 3D laser scanning to crack detection was also studied on brick andconcrete buildings (Laefer et al., 2010). However, only buildings with three or four floors were tested for thedetection reliability, while the minimum height of buildings that need façade inspection is six floors and up (DOB,2019). A systematic evaluation of façade conditions at various elevations using the two technologies is missing.
Studies in this area also provided a point of departure on the factors influencing the quality of data andmeasurements. The influence of surface color (Kavulya et al., 2011), the resolution of laser scanners (Anil, 2015),
the distance from a surface to laser scanners(Kavulya et al., 2011; Anil, 2015), and the incidence angle(Soudarissanane et al., 2007; Zamecnikova et al., 2015; Kavulya et al., 2011) were considered on the quality ofobtained point cloud data. However, the influence of location of cracks with respect to the elevation on the façade
was not considered for façade taller than four floors. Thus, there is a need to evaluate the impact of these influence
factors as a whole on the quality of the point cloud data needed to support the façade inspection process withcomprehensive and accurate façade condition information.
3D reconstruction technology, as another option to create a point cloud, was also adopted by many
researchers for crack detection on structure elements and building condition assessment (Jahanshahi and Masri,
2012; Torok et al., 2014; Zheng, 2014; Zhou et al., 2015; Liu et al., 2016). 3D reconstruction technology was used
to obtain the camera setting parameters and camera pose, which is necessary for crack detection with images. The
images with detected cracks can be projected back to the 3D model surface for defects documentation (Liu et al.,2016). Both methods require the images to be taken at close proximity with the target object. Regarding the usageof 3D reconstruction on buildings, it was mainly on the damage assessment on residential buildings with a focus
on building elements’ (i.e., windows, doors, and walls) displacement and deflections rather than the defects likecracks and spalls. Detection for cracks or other surface discontinuities based on the difference between the default
plane normal and local plane normal were only studied with single structure elements (e.g., columns and beams)(Torok et al., 2014; Zheng, 2014). Previous research studies have mainly focused on algorithms development for
automatic crack detection using images and did not focus on streamlining the data collection process. For the
façade inspection program, a lot of images are taken by inspectors during the current process, which can work as
a data source for the reconstruction of a 3D model. There is a need to study whether the images taken during theinspection process can help with crack detection and further comparison.
3. CASE STUDY
3.1 Experiment Setup for Data Collection
In this study, we designed a series of experiments to investigate the influence of factors on the quality ofthe obtained point cloud data and guide the façade inspection of buildings using point clouds. The 3D laser scannerused for data collection was FARO Focus S 150 (Figure 1a), and the camera used for image capture was Nikon
D3200 (Figure 1b). Two important parameters, i.e., quality and resolution, impact the density (or sparsity) of point
clouds captured with laser scanners. The combination of the quality and resolution determines the point distancein the obtained point cloud data, which is the distance between two captured points with a scanning distance of
10m. The quality setting of the 3D laser scanner was set to be constant as 4X for the experiment, meaning for eachpoint’s incoming laser light, the observation time was 8 microseconds. This longer observation time (as comparedto the observation time of 1 microsecond when the quality is set as 1X) for the incoming laser light can let thesensor get a stronger signal, leading to a reduction in ranging noise in the obtained point cloud data. By setting ahigher quality, we ensure the range accuracy. The resolution of the scanner was changed between ¼ to 1/1 toevaluate the effect of point density on the visibility and measurements on façade defects. Resolution value of ¼,½, and 1/1 indicate that the point distance for the obtained point cloud data would be around 6.1 mm, 3.1 mm, and
1.5 mm for 10 m, respectively. For image capturing, the focal length of the camera was set constant as 55mm,
which was the largest focal length available on the camera to capture objects further away, and the images captured
with the camera was 6,016 × 4,000 in resolution.

377

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 1. Experiment Setup. (a) FARO Focus S 150; (b) Nikon D3200; (c) Simulated cracks;
Based on their width, cracks can be divided into three groups: hairline cracks (opening < 1mm),
slight/small cracks (1mm < opening < 5mm), and large cracks (opening > 5mm) (DOB, 2016). The experimenttestbed consisted of four foam-boards with simulated cracks on a computer with three different widths (1mm,5mm, and 10mm) and orientations (horizontal, vertical, and inclined 45°) to represent different groups of cracks(Figure 1c). The length of all the cracks was set as 10 cm, which represented the minimum detected crack length
by manual facade inspection (Laefer et al., 2010). Cracks were simulated by printing instead of manually cutting
them to avoid introducing human error.

Figure 2. Experiment Setup. (a) Building façade used in the experiments; (b) Plan view of scan and camera
locations with respect to the façade; (c) Laser distance meter.
Four foam boards of the exact configuration above were printed and were attached to the 2nd, 4th, 6th andthe top floor (i.e., 8th floor) of a building located in NYC to investigate the quality of obtained point cloud data to
guide comprehensive and accurate façade inspection (Figure 2a). To analyze the influence of range distance fromthe target building to the laser scanner locations and the horizontal incidence angles, nine locations were selected
separately at distances of 10m (min range of the scanner), 15m, and 20m from the target building with horizontal
incidence angles of 0°, 15°, and -15° with respect to the vertical line at the center of the foam boards (Figure 2b)
by controlling the distance from the center line in Figure 2b. One scan was performed at each scan location tocover the front façade of the building, and 100 images were captured for the 3D reconstruction technology at eachlocation. A digital laser distance meter with a measurement accuracy of ± 1.5mm was used to control the distancefrom the laser scanner/camera to the target façade (Figure 2c). Table 1 summarizes the investigated influence
factors on the resulting data quality both for laser scanners and 3D reconstruction technology.
Table 1. Influence factors investigated on the data quality
Technology
Laser scanner/ 3D
reconstruction

Influence factor
The elevation of the simulated
cracks on the façade
Crack width (mm)Crack orientation (°)

378

Set values
2nd, 4th, 6th, 8th
1, 5, 100°, 45°, 90°

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Horizontal distance from façade(m)
Horizontal incidence angle (°)
Scanning resolution*

10, 15, 20
0°, 15°, -15°
1/1, ½, ¼*: Only applicable to the laser scanner. With the 4X quality, point distance for the selected resolutions are: 1.5mm/10m(resolution = 1/1), 3.1mm/10m (resolution = 1/2), and 6.1mm/10m (resolution = 1/4).

3.2

Processing and Analysis of the Captured Datasets
Point cloud data obtained with the 3D laser scanner was processed, and measurements of the simulated
cracks were taken with a robust point cloud processing tool. Figure 3a shows an example point cloud data of the
façade. Six measurements were taken so far for every visible crack and will be automated with random samplingin the future work. If the crack was not visible in the captured dataset, the length was marked as 0 for themeasurement result, and corresponding cell in the results table was greyed out. If the cracks were occluded due totrees and other objects, they were marked as occluded for that scan location and setting. The length of eachsimulated crack captured through both technologies was measured and compared to the ground truth value (i.e.,
10 cm of length for each crack width) for accuracy evaluation.
Three widely-used commercial and open source photogrammetry tools were evaluated for 3D
reconstruction to generate accurate and geo-referenced the point cloud data based on the images captured with thecamera. Figure 3b-3d show examples of reconstructed point cloud data of the target building façade generated by
the three photogrammetry tools. Among the three tools, application 1 generated the sparsest point cloud that onlyshows the edges of columns and window frames (Figure 3b). This is due to the lack of textures on the foam boards
and its color being white, where the application benefits from the richness of texture and color. Application 2
generated denser point clouds as compared to application 1, with more noise in the data (Figure 3c). Application
3 generated photo-textured mesh model and point cloud data (Figure 3d). The point cloud data generated with the
3D reconstruction was too sparse to conduct measurement evaluation regardless of the tools used. The textured
mesh model generated with application 3, however, provided a chance to measure the simulated cracks with good
accuracy. Application 3 worked well with images taken at the distances of 20 m and 15 m for 3D reconstruction.
With the images taken from 10 m, the 3D reconstruction tool did not work well due to the high vertical incident
angle at higher elevations. The measurement of cracks in the reconstructed model was completed so far only forthe 6th floor’s cracks.

Figure 3(a) Point cloud data obtained with the terrestrial laser scanner; (b) point cloud data generated
with application 1; (c) point cloud data generated with application 2; (d) 3D model generated with application 3
4. MEASUREMENT RESULTS
To evaluate the accuracy of measurements on the captured point cloud data, the root-mean-square error(RMSE) was adopted for the accuracy assessment. The Equation (1) shows RMSE calculation (Chai and Draxler,
2014), in which 𝑑𝐷𝑖 is the crack length measured in the point cloud data, and 𝑑𝑟 is the ground truth value of themeasured crack length, and n represents the total number of measurements taken.
∑(𝑑𝐷𝑖 − 𝑑𝑟 )2

𝑅𝑀𝑆𝐸 = √𝑛(1)

It is obvious that the point cloud data obtained with a 3D laser scanner (Figure 3a) is much denser thanthe 3D reconstructed point cloud (Figure 3b). From the reconstructed point cloud with Applications 1 and 2,
building elements such as columns and windows were easy to find, while the simulated cracks were not visible.
Hence, the measurements were only conducted with the data reconstructed via application 3.

379

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Measurement accuracy is provided based on the RMSE values to help understand the influence factors on
quality of façade inspection data and suggest required settings to capture the right quality data. However, it should
be noted that during a façade inspection project, the actual length and width of cracks are less important ascompared to the visibility and orientation of the cracks in the obtained dataset. This conclusion was drawn fromthe shadowing of the façade inspection process for three different buildings in NYC.

4.1 Results for visibility of cracks and RSME in the point cloud data obtained with the terrestrial laser
scanner
The results of the assessment on the laser scanner-based dataset are provided in Figure 4. The results areprovided as RSME values where cracks were visible, as “O” when occluded, and as greyed out cells if cracks wereinvisible. So, wherever RSME reported in the figure, it should be perceived that the corresponding cracks werevisible for inspectors, which is the essential information inspectors collect during inspections. Given the results,
none of the 1mm hairline cracks simulated could be seen in the point clouds and were not presented in Figure 4.
Regardless of the resolution and horizontal incidence angle set, 10mm cracks in all orientations (i.e., horizontal,
vertical, inclined) were visible for 2nd floor cracks at 15m scan range and at 10m scan range for all crack widths(if occlusions were not present), as shown with dashed green highlights in Figure 4. Regardless of horizontal
incidence angles, resolutions, and crack type/orientation, nothing was visible at 8 th floor at 10m and 15m (except
the largest vertical cracks at highest resolution at -15° horizontal incidence angle). However, at shorter scan ranges
and for cracks located at higher elevations, introducing a horizontal incidence angle seemed to help for visibility.
Also, resolution ¼, in general, generated very poor-quality data for detecting any of the simulated cracks as shownin Figure 4 with vertical red highlights in Figure 4.
When we analyze Figure 4 vertically, it was observed that with the highest resolution (i.e., 1/1), all thevertical and inclined cracks were captured at a distance of 20m with all horizontal incidence angles at all elevations
with RMSE around 1cm (mainly around 0.5cm). This resolution could also capture cracks up to 6th floor at10m
and 15m with worsened RMSE. With the same settings at 20 m range, the horizontal cracks on the 2nd, 4th, and 6th
floor could be captured with RMSE values smaller than 0.8cm, while the horizontal cracks on the 8th floor could
not be captured. When the horizontal incidence angle changed to 15° for 1/1 resolution at 20 m range, visibility
at certain levels got effected along with the measurement accuracies (5mm cracks on the 6th floor for 15° and 5mmhorizontal cracks on the 2nd floor for -15°). It should be noted that as the location of the scanner/camera comescloser to the building, the visibility of cracks on the higher levels (the 6th and 8th floor) decreases due to the highvertical incidence angle in general. The visibility of the vertical and inclined cracks on the 6th floor is not affectedby the scan location on 1/1 and ½ resolution except for ¼ resolution. However, it is negatively impacted forhorizontal ones at 10m and 15m. When the scanning range was 10m with a horizontal incidence angle as 15°, all
the cracks on the 2nd floor were occluded by the trees near to the building, however the cracks would be visible formeasurements with RMSE values around 1cm (e.g., horizontal incidence angle of 0° or -15° from 10m). In
general, it was observed that 10 m horizontal distance was too close to the façade and should not be used for
capturing the façade inspection data.

380

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Bold: The point cloud data that has good measurement accuracy (RMSE smaller than 1cm); “O”: Cracks occluded by tree
branches. Gray cells indicate invisibility of cracks; V: vertical crack; H: horizontal crack; I: inclined crack.

Figure 4. Average RMSE for measurements of cracks on the data captured with the laser scanner (Unit: cm).
When the resolution was changed to ½, and we keep analyzing the table vertically, all types of cracks
were visible for 2nd and 4th floors with the RSME values around 0.8cm from 15m and 20m regardless of thehorizontal incidence angle (except for the 5mm horizontal cracks on the 4th floor at 0°, and at -15° at 20m). Whenthe distance decreases to 10m, the visibility of the horizontal cracks on the 4th floor and the measurement accuracyof the visible cracks were negatively affected. The visibility of the 10mm vertical cracks on the 6 th floor was notaffected by the horizontal incidence angle when the scan range was 15m, while the RSME value increases when
it changes from 0° and -15° to 15°. The 10mm vertical crack on the 6th floor was also captured from 20m with ahorizontal incidence angle of 0° and -15° and 10m with a horizontal incidence angle of ±15°. The RSME values
for the captured 10mm vertical cracks on the 6th floor were all around 1cm. The inclined 10mm crack on the 6th
floor was only possible to capture from 15m when the horizontal incidence angle was -15° with an RMSE valueof 0.5cm. None of the cracks on the 8th floor was captured with ½ resolution regardless of the horizontal incidence
angles and the scan ranges.
With a resolution of ¼, all types of cracks on the 2nd floor were captured from 10m, as the horizontal
incidence angle was small at that height of cracks. 10mm cracks on the 2nd floor were captured from 15m regardlessof the horizontal incidence angle except for the occluded ones, which would be measured if not occluded. The
RMSE values for the captured cracks from 15m were all below 1.5cm. At 20m as compared to 15m, the visibility
and the measurement accuracy were negatively affected by the increase of the distance (e.g., 10mm cracks on the
2nd floor). At ¼ resolution, the 10mm vertical and inclined cracks on the 4th floor were captured from 15m and
20m with a horizontal incidence angle of 0° and 15° with an average RMSE value of 1.3cm. The 10mm inclined
cracks on the 4th floor were only captured from 15m with ±15° horizontal incidence angle. No cracks were captured
with ¼ resolution from 20m with a horizontal incidence angle of -15. In general, the ¼ resolution should not be
used for capturing the façade inspection data since it cannot capture the cracks on the higher floors (i.e., 6th and 8th
floors) of buildings.
To summarize the discussions so far, the minimum settings to capture the cracks on each floor and achievea good measurement accuracy (i.e., 1cm or less) in the captured point cloud set would be as the following. Taking
both the measurement accuracy and visibility point of view, for buildings that have more than 6 floors, the best
resolution (i.e., 1/1) should be used and the scanning location should at around 20m from the building with 0°
horizontal incidence angle with respect to the façade plane. For buildings that have 6 floors or below, lower
resolution (1/2) can cover the façade for crack identification from 15m with a horizontal incidence angle of 15°
and 20m with a horizontal incidence angle of 0° and can be considered for a shorter time per scan. Since theresolution of ¼ and 10m range were not able to capture the higher floors’ cracks, it is not recommended for façade
inspection projects.
4.2

Results for visibility of cracks and RSME in the point cloud data obtained with 3D reconstruction
The measurements of cracks using the 3D reconstruction data could be completed only on the 6th-floorso far. The measurement accuracy is presented for images obtained from 20m and 15m because the 3D
reconstruction model for images obtained from 10m was hardly constructed. For each captured crack, six
measurements were captured, and the RMSE was used for the measurement accuracy evaluation. To compare themeasurement accuracy with point cloud obtained with 3D laser scanning, average RMSE values for the 6th-floor
cracks were calculated and presented in Table 2. Again, no hairline cracks were visible in the reconstructed data
and are not reported in Table 2.
Table 2. Average RMSE for measurements of cracks on the point cloud data reconstructed from images(Unit: cm)
Horizontal incidence angle
0 degree15 degree-15 degree
Scan
range
20m
15m

Elevationof crack
6th
floor
6th
floor

Crack
width
5mm

V

H

I

V

H

I

V

H

I

0.48

0.78

0.92

0.33

0.33

0.65

0.36

0.72

1.04

10mm

0.78

1.20

0.34

0.66

1.09

0.32

0.37

0.73

0.28

5mm

0.86

1.09

0.59

0.49

1.04

0.59

0.38

0.73

0.30

10mm

0.80

1.03

0.71

0.31

1.00

0.54

0.29

0.80

0.26

Bold: RMSE value smaller than 1cm. V: vertical crack; H: horizontal crack; I: inclined crack
From Table 2, it is apparent that it is possible to measure the cracks on the 6 th floor with a 3D model381

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

generated with reconstruction technology (no visibility issues for the 6th floor)., indicating that the 3D
reconstruction performs better than laser scanning regarding on the coverage of cracks in different direction. The
distance change did not affect much of the visibility and measurement accuracy of reconstruction model since theaverage RMSE value of images obtained from 20m (0.63cm), and 15m (0.65cm) was similar. When the horizontal
incidence angle equals 0, the measurement accuracy of all types of cracks was negatively affected by the decreaseof distance, and the measurement accuracy of vertical (around 0.75cm at 20m) and inclined cracks (around 0.40cm
at 20m) was better than the horizontal cracks (around 1.1cm at 20m), which concurs the measurement result of 3D
laser scanning data. When the horizontal incidence angle is 15°, the measurement accuracy of inclined andhorizontal cracks was not affected by the distance change, while the average measurement accuracy of verticalcracks decreased when the distance changed from 20m to 15m. With the horizontal incidence angle as -15°, themeasurement accuracy, in general, is better than the reconstruction model generated with images obtained fromthe horizontal incidence degree of 0° and 15°. The measurement accuracy of all types of cracks was not affectedby the increase of the distance, and the average RMSE value for all types of cracks is around 0.5cm when thehorizontal incidence angle is -15°, which is close to the measurement result in point cloud obtained from -15 degreewith the best resolution (1/1) at a distance of 10m. One thing that needs to notice is that the 3D reconstruction
technology can capture the 5mm cracks very well, while the laser scanning technology can only capture cracks ofthis width with the best resolution.
Based on Table 2, the vertical and inclined cracks were captured with higher accuracy than thehorizontal cracks regardless of the horizontal incidence angle and distance in general, and the measurement
accuracy with data obtained with -15° horizontal incidence angle was better than the ones with 0° and 15°, which
is different from the result from the 3D laser scanning data. This might because the photos imposed to the 3D
reconstruction model get distorted, leading to the inaccurate start and end point of each crack. The 3D
reconstruction from images was able to cover the cracks on the 6th floor with a pretty good measurement accuracy
that is near to the data captured with the terrestrial laser scanner.

5. CONCLUSION & FUTURE WORKThe objective of this study is to evaluate the settings (e.g., resolution, scan range, vertical incidence angles,etc.) at which laser scanners and cameras should be operated to generate a point cloud dataset for enabling acomprehensive façade inspection. This study specifically evaluated the visibility of cracks of different orientationand sizes, however, will be extended for other information required for a comprehensive façade inspection (e.g.,visibility of spalls, openings, loose components). The results show that the 3D laser scanning data perform morestable than the 3D reconstruction from images in terms of measurement accuracy. To obtain crack information at,
below, and beyond the 6th floor of buildings, a resolution of 1/1 and a distance of 20m is recommended for laser
scanners. However, only from the visibility point of view, it is sufficient to run the equipment at 15 m range at a
½ resolution for buildings up to the 6th floor. For the 3D reconstruction technology, the data collection distance
did not affect much on the visibility and measurement accuracy for upper floor (i.e. 6th floor). The images obtained
from -15° performed best in terms of the average measurement accuracy. One thing that needs to be noticed is thatthe minimum crack width (i.e., 1mm) was not able to be captured with both terrestrial laser scanner and the image
reconstruction. The terrestrial laser scanner is recommended for stable performance and better measurement
accuracy across elevations with improved RSME. The result of this case study can provide guidance for façade
inspection projects’ survey method selection. Although the simulated cracks in this presented case study have greatcolor contrast with the background, it did not affect the measurement accuracy of the dataset, which is our target
information. This study will be extended to find the heights/floors where visibility of cracks is not possible. Future
work will focus on the implementation of algorithms that can automatically detect and classify cracks based on
the width in the point cloud data.
ACKNOWLEDGMENTS
The authors of this paper would like to thank graduate students Poornachandra Venkatesh and Rashmi J.
Nadauvinmath for assistance with the crack measurement process.
REFERENCES
Anil, E. (2015). Utilization of As-is Building Information Models Obtained from laser Scan Data for Evaluationof Earthquake Damaged Reinforced Concrete Buildings (Doctoral dissertation).
Chai, T., and R. R. Draxler. (2014). Root mean square error (RMSE) or mean absolute error (MAE)?—Arguments
against avoiding RMSE in the literature. Geoscience. Model Development, 7 (3), 1247-1250.
Department of Buildings (2011).1 RCNY 103-04 “Periodic Inspection of Exterior Walls and Appurtenances of
Buildings." Retrieved from: https://www1.nyc.gov/assets/buildings/rules/1_RCNY_103-04.pdf.

Last accessed on April 03, 2019.

382

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Department of Buildings (2019). Facade Inspection & Safety Program (FISP) Filing Instructions. Retrieved from:https://www1.nyc.gov/site/buildings/safety/facade-inspection-safety-program-fisp-filinginstructions.page. Last accessed on April 23, 2019.
Department

of

Buildings(2016).

Presentations:

FaçadeConditions.

Retrievedfrom:https://www1.nyc.gov/assets/buildings/images/content/misc/FacadePresentation.pdf. Last accessedon April 21, 2019.
Jahanshahi, M.R. and Masri, S.F. (2012). Adaptive vision-based crack detection using 3D scene reconstruction
for condition assessment of structures. Automation in Construction, 22, 567-576.
Kavulya, G., Jazizadeh, F., and Becerik-Gerber, B. (2011). Effects of color, distance, and incident angle on qualityof 3D point clouds. Proceedings of the International Workshop on Computing in Civil Engineering 2011,
Miami, Florida, United States, pp.169-177.
Kim, M., Sohn, H., and Chang, C. (2014). Localization and quantification of concrete spalling defects using
terrestrial laser scanning. Journal of Computing in Civil Engineering, 29 (6): 04014086.
Laefer, D.F., Gannon, J, and Deely, E. (2010). Reliability of crack detection methods for baseline conditionassessments. Journal of Infrastructure Systems, 16 (2), 129-137.
Laefer, D.F., Trurong-Hong, L., Carr, H., and Singh, M. (2014). Crack detection limits in unit based masonrywith terrestrial laser scanning. NDT&E International, 62(2014), 66-76.
Liu, Y., Cho, S., Spencer, B.F., and Fan, J. (2016). Concrete crack assessment using digital image processing and
3D scene reconstruction. Journal of Computing in Civil Engineering, 30 (1): 04014124.
Sedek, M. and Serwa, A. (2016). Development of new system for detection of bridges construction defects using
terrestrial laser remote sensing technology. The Egyptian Journal of Remote Sensing and Space Science,
19 (2), 273-283.
Soudarissanane, S., Ree, J.V., Bucksch, A., and Lindenbergh, R. (2007). Error budget of terrestrial laser scanning:
influenceoftheincidence
angle
on
the
scan
quality.
Retrievedfrom:https://www.semanticscholar.org/paper/Error-budget-of-terrestrial-laser-scanning-%3A-of-theSoudarissanane-Ree/600cb11f8f8848e76d2914832a740f900d946777#similar-papers. Last accessedon March 4, 2019.
Tang, P., Huber, D, and Akinci, B. (2009). Characterization of three algorithms for detecting surface flatness
defects from dense point clouds. Proceedings of IS&T/SPIE Conference on Electronic Imaging, Scienceand Technology, San Jose, California, United States, vol. 7239.
Torok, M.M., Golparvar-Fard, M., and Kochersberger, K.B. (2014). Image-based automated 3d crack detection
for post-disaster building assessment. Journal of Computing in Civil Engineering, 28 (5): A4014004.
Tsai, Y.J., and Li, F. (2012). Critical assessment of detecting asphalt pavement cracks under different lighting andlow intensity contrast conditions using emerging 3d laser technology. Journal of Transportation
Engineering, 138 (5). 649-656.
Yu, Y., Li, J, Guan, H, and Wang, C. (2014). 3D crack skeleton extraction from mobile LiDAR point clouds.
Proceedings of 2014 IEEE Geoscience and Remote Sensing Symposium, Quebec City, QC, Canada, pp:914-917.
Zheng, P. (2014). Crack detection and measurement utilizing image-based reconstruction. Retrieved from:https://vtechworks.lib.vt.edu/bitstream/handle/10919/48963/crack_detection_and_measurement_uti
lizing_image_based_reconstruction.pdf?sequence=1&isAllowed=y .Last accessed on March 20,
2019.
Zhou, Z., Gong, J, and Guo, M. (2015). Image-based 3D reconstruction for post hurricane residential building
damage assessment. Journal of Computing in Civil Engineering, 30 (2), 04015015.

383

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

AUTOMATED UAV ROUTE PLANNING FOR BRIDGE INSPECTIONUSING BIM-GIS DATA
Yang Zou1, Molood Barati2, Enrique del Rey Castillo3, Robert Amor4, Brian H.W. Guo5, Jiamou Liu6
1) Lecturer, Department of Civil and Environmental Engineering, University of Auckland, Auckland, New Zealand.
yang.zou@auckland.ac.nz
2) Research Assistant, Department of Civil and Environmental Engineering, University of Auckland, Auckland, New Zealand.
mbar468@aucklanduni.ac.nz
3) Lecturer, Department of Civil and Environmental Engineering, University of Auckland, Auckland, New Zealand.
e.delrey@auckland.ac.nz
4) Professor, Department of Computer Science, University of Auckland, Auckland, New Zealand. trebor@cs.auckland.ac.nz
5) Lecturer, Department of Civil and Natural Resources Engineering, University of Canterbury, Christchurch, New Zealand.
brian.guo@canterbury.ac.nz
6) Senior Lecturer, Department of Computer Science, University of Auckland, Auckland, New Zealand.
jiamou.liu@auckland.ac.nz

Abstract: Unmanned Aerial Vehicle (UAV) has been increasingly used for bridge inspection in the past few years
and the recent development of 3D bridge models generated by UAV photogrammetry is highly promising for
inspection purposes. However, the current practices of UAV-enabled bridge inspection require human control,
during which process the UAV is flying close to the bridge location to take high-resolution images according tohuman commands. To overcome this gap, a novel method to automate the UAV route planning for bridge
inspections is proposed, which uses Building Information Modelling (BIM) and Geographic Information System(GIS) data as input to drive the automated UAV flying and operations. The underlying assumptions include, 1) the
integrated BIM-GIS system can represent the physical world, and 2) the 3D bridge model generated by UAV
photogrammetry can provide highly reliable and accurate data for bridge inspection. Under such route planning,
the BIM-GIS system provides the bridge’s geospatial and surrounding environment information (e.g. bridge’s
geometry, elevation, orientation) to guide the UAV flying the shortest path to gather all required information forthe bridge inspection according to a novel algorithm. The next stage of this project will validate and test the
proposed method for real bridges.
Keywords: Unmanned Aerial Vehicle (UAV), Bridge inspection, BIM-GIS integration, Route planning,
Photogrammetry.
1. INTRODUCTION
The objective of bridge inspection is to identify changes from previous inspections, determine the physicaland functional condition of the bridge and understand if the bridge is safe and meets the service requirements.
Bridge inspections are time consuming and labour intensive, and need to be performed on a regular basis and inan efficient way (Yeum et al., 2015). But the current bridge inspection practice heavily relies on manual datacollection and subjective estimates of bridge condition and often require road closure and traffic management(Adhikari et al., 2014).
Unmanned Aerial Vehicle (UAV) is a new alternative to make the bridge inspection faster, safer, and lessexpensive in comparison with traditional bridge inspection methods (Chan et al., 2015). Using UAV for bridge
inspection should follow a number of steps including data collection, data transmission, data analysis, and decision
making (Zou et al., 2019). Data collection is the first step but one of the most important steps involving the UAV
flying close to the bridge to collect the data for inspection purposes. Compared with the traditional method thatcollects individual images for further analysis, the recent development of UAV photogrammetry that converts aseries of high-resolution images into 3D point cloud has the potential to improve the understanding of the bridgeand collect better data management and visualisation. Understanding how to apply UAV photogrammetry forbridge inspection is necessary to collect more accurate UAV imagery because poorly captured images may lead to
redundant efforts. To determine an effective UAV flight route bridge engineers not only require the geometry of
the bridge but also detailed information about the surrounding environment. Building Information Modelling(BIM) provides a semantically-enriched 3D geometrical model at the bridge level that assists experts in modellingand constructing a structure (Lu et al., 2017) while Geographic Information System (GIS) supports spatial or
geographic data for the surrounding environment (Chen et al., 2019). The integration of BIM and GIS data
improves the process of data sharing among tools and provides a representation of the physical world by virtual
3D models (Zhu et al., 2018).
A novel UAV route planning algorithm for the bridge inspection by exploiting BIM-GIS data is proposedin this paper, with the objective being to automatically fly an UAV on the shortest possible path to collect imagery
data. The captured data using UAV photogrammetry can be visualised in a 3D point cloud environment to assist
experts for the bridge inspection, but the quality of the 3D point cloud must be evaluated for further algorithm384

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019refinements. The next stage of this project is to validate and check the efficiency of the proposed algorithm forreal bridges.
The reminder of this paper is structured as follows. The related work is reviewed in Section 2. Section 3
describes the details of proposed method for the bridge inspection. The significance and new insights in this paper
as well as future work are discussed in Section 4.
2. LITERATURE REVIEWThe literature review reported in this section has been classified into three main categories including(2.1) BIM-GIS integration, (2.2) Bridge inspection using UAVs, and (2.3) UAV route planning.
2.1 BIM-GIS integration
BIM-GIS integration is a widely discussed topic in the construction industry, with one of the main
challenges being the lack of a schema that defines the mapping between the BIM and GIS data models. Semantic
Web (SW) technologies and ontology matching approaches are used to develop BIM-GIS platforms for integration.
On this line of study, Karan et al. (2015) proposed a method for BIM-GIS integration using SW technologies. In
the proposed method, BIM and GIS data firstly translated into the SW common data format. Then, a number ofstandardised ontologies are generated to integrate and query the heterogeneous spatial and temporal data. Volk etal. (2014) developed a data transformation platform using Mike2Shp toolbox and Extensible Markup Language(XML). Through employing this platform, the geological coordinates were recorded. Mignard et al. (2014)
proposed another integration method using ACTIVe 3D and ontology. A software designed by Kang et al. (2015)
developed for integrating BIM into a GIS-based Facilities Management (FM) system by utilising domain extension
and open source Web. Similar methods proposed (Amirebrahimi et al., 2016; Rafiee et al., 2014).
Besides SW-based methods, several applications are also developed for BIM-GIS integration. For
example, Jusuf et al. (2017) proposed a workflow using FME as a data transform platform which can manipulate
BIM data from AutoCAD Revit 2016 and import the modified data into Graphisoft Archicad 20 as a format of
CityGML. In another similar method, Wu et al. (2007) developed a tool for transforming IFC data to GML format
through data acquisition, original coordinate determination, coordinate transformation, and model regeneration
processes. El Meouche et al. (2013) also provided a comprehensive summary of software available for integrating
BIM data in a GIS software. Google Earth, ArcGIS, and AutoCAD Map 3D are some examples in this field. Some
popular data formats imported into such software are DWG for 2D drawing, DGN for extracting a set of data
points, and IFC that is the ISO standard for representing BIM data developed by buildingSMART.
2.2 Bridge inspection using UAVs
To inspect a bridge, the UAV should be able to complete two main tasks. The first task is to navigate
around the bridge and will be discussed under Section 2.3. The second task is to collect data for damage detection
and safety condition determination. Identification of bridge damage using UAVs can be achieved in many ways.
One method that Lei et al. (2018) proposed used specific algorithms to process UAV images to determine cracks
on the bridge. Promising results of identifying cracks were also reported which can be improved by utilizing
machine learning methods. Work by Chen et al. (2019) specified many different types of data such as lasers data
and thermal scanners data for inspection. Chan et al. (2015) also offered UAVs as a cost-effective tool for
inspecting constructions such as bridges, and explained how to utilize UAVs for conducting visual bridge
inspection by considering the obstacles that should to be integrated into current practice.
Besides UAVs’ advantages for bridge inspection, it is important to use advanced drone operation
applications such as DJI GS Pro to fly drones by setting specific features. Different features should to be checked
before flying UAV such as overlap and altitude. In photogrammetry, overlapping affects the possibility of building
a complete 3D model (Dong et al., 2016). Due to the duplicity of data from each point being collected from at least
three positions. It has been recommended to set side overlap ratio and front overlap ratio 75% and 80%
respectively (Dandois et al., 2015). In case of failing to trigger a photo by the UAV on the exact time, thisconfiguration captures every feature at least three times. Altitude is another important feature in photogrammetrythat affects the accuracy of 3D models (Zhang et al., 2017). The UAV camera and altitude define Ground Sampling
Distance (GSD) which determines how big each pixel is on the object. The UAV should not fly too close to asurface or thousands of images will be obtained, which takes too long to be processed by computers. Thus, themaximum and minimum GSD should be defined for capturing more accurate photos.
2.3 UAV route planning
Flying UAVs without preparing route planning can have a negative impact on the inspection processsince UAV’s performance depends on several parameters such as battery life and flight range (Teo et al., 2016),
which highlights the important of pre-flying planning. Technically, the goal of route planning is to find the most
optimum route (i.e., path) by minimizing the distance travelled and time taken. Various machine learningalgorithms have been proposed to discover the shortest route such as Travelling Salesman problem, Dijkstra’s385

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

algorithm, Random Walk, etc.
On this subject, He et al. (2017) studied four different geometry search algorithms by comparing their runtime, complexity, and path length. This study compared the performance of Dijkstra, Floyd, A*, and Ant colonyalgorithms. The study also defined the environment as a grid map to find the path from the initial grid to the target
grid. The simulation results suggested that Dijkstra’s algorithm had the best performance with the shortest run
time and path length in an environment with obstacles. It is important to mention that the simulation was tested ina static environment and with fixed obstacles. The study also provided an overview of existing popular solutions
to solve path planning problems for UAVs.
Similarly to He and Zhao (2017), Savuran et al. (2015) studied the path optimisation problem for acarrier launched UAV using Genetic Algorithm (GA). Generally, GA focuses on the phenomenon of adaptation
as it occurs in nature and tries to bring the concept of adaptation into computer systems (Maulik et al., 2000). The
experimental results revealed a better performance to compare with Nearest Neighbor Heuristic in terms ofminimizing total route length covering all given targets. In this study, the starting point and target point are
dynamic as the carrier moves in the real environment.
Goel et al. (2018) also proposed a solution based on Glowworm Swarm algorithm to find a collision
free path for a 3D environment. The difference of this study with others is that the starting point and target point
are fixed, but the algorithm is still designed for a real-world environment with unknown obstacles. In the scenario
of structural inspection using UAVs, Guerrero et al. (2013) developed a method to find optimised path using
Traveling Salesman Problem in the windy environments. This study considered the battery limitation of the UAV
and validated the proposed method by computing an optimised route for a simulated structure.
3. METHOD
A new UAV route planning algorithm for bridge inspection using BIM-GIS data is proposed in thispaper. The imagery data captured by UAV will be processed by photogrammetry and visualised in 3D point cloudfor bridge inspection. The proposed framework is illustrated in Figure 1 and consists of three main modules: (3.1)
BIM-GIS system, (3.2) Route planning and data collection module, and (3.3) Data analysis module. The details ofeach module have been described in the following subsections. Note that this method has been proposed based on
two main assumptions with the first assumption being that the integrated BIM-GIS system can represent thephysical world, and the second one being that 3D bridge model generated by UAV photogrammetry can provide
highly reliable and accurate data for bridge inspection.

Figure 1: The framework of automated UAV route planning using BIM-GIS data
3.1 BIM-GIS system
The process of aggregating and visualising multiple sources of information is crucial for selecting
appropriate inputs for developing automated UAVs route planning. Autodesk InfraWorks was used to integrate
GIS into 3D design BIM model of bridges. The new features in Autodesk InfraWorks allow to integrate GIS data
in a more seamless manner to design BIM models. To improve the process of decision making, Autodesk
InfraWorks provides a working platform that enables users to quickly and easily visualise preliminary designs in
a 3D virtual environment. Autodesk InfraWorks also allows integrated BIM-GIS data to be easily managed andviewed. In this way, the BIM-GIS data can be used at the time of bridge inspection to provide up-to-date
information of the site condition.
In order to select the desired dataset for route planning, filtering data by inspecting the surrounding
environment of bridge is important. The geometry of each structural component of bridge are first chosen to be386

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019the main parameters for the route planning algorithm. Additionally, potential hazards such as trees and lamp posts
should be marked for risk management.
3.2 Route planning and data collection module
The effective UAV route planning for bridge inspection not only relies on the bridge’s geometry,orientation and location information but also on surrounding environment information. The process of collecting
imagery data is explained in this section. As represented in Figure 1, route planning and data collection module
contains two main subsystems, being (I) Route Planning Algorithm, and (II) Drone mapping deployment. The
overall workflow of Route planning and data collection module is illustrated in Figure 2. In this module, the firststep is to extract the details of the bridge’s geometry, barriers information, and surrounding environment
information. The second step is to define multiple circular-shaped layers on top of the bridge. The next step is todetermine interest points on the layer where the UAV is going to capture imagery, and to extract coordinates andmeasure the distance between each pair for developing route planning using Travelling Salesman Algorithm. In
Step 4, the UAV should be programmed using related parameters in Step 2. For example, the flight height can be
defined by considering the height of the bridge and obstacles. Then, the automated UAV should automatically fly
the possible shortest path using Travelling Salesman Algorithm to collect imagery data (Steps 5 and 6). The reason
for using Travelling Salesman Algorithm is because of drone battery limitation. In fact, the drone flight is restricted
in maximum travel distance supplied by the battery. To reduce the flight cost, the proposed method takes advantage
of Travelling Salesman Algorithm. Generally, Travelling Salesman problem aims to find a solution for thefollowing question: given a list of points (i.e., nodes) and the distances between each pair of points, what is theshortest possible route that visits each point and returns to the origin point (Rosenkrantz et al., 1977)?

Figure 2: The workflow of route planning and data collection module
The three types of aerial photographs are vertical, low oblique, and high oblique. A photograph isvertical if the camera lens axis is vertical or perpendicular at the moment of exposure to the object. The scale ofvertical photographs is constant and the shape of ground area is square or rectangle. Low oblique photographs
taken with the drone camera tilted about 30ºfrom vertical line were used in this study. The low oblique does notshow the horizon and the ground area covered is a trapezoid. Oblique aerial images can be characterized by Ground
Sampling Distance, image quality, and overlap. The following section briefly explains the features that should beconsidered for developing route planning algorithm and collecting imagery data.
Ground Sampling Distance. Figure 3 shows an example of oblique images. Photographing with a tilt
results in imagery with variable scale. In order to describe an oblique image, the size of pixel orthogonal to the387Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

camera is usually determined. In the oblique aerial images, Ground Sampling Distance (GSD) value differs in thefront and the back of foot print of the oblique images as the ground area shape is a trapezoid.

Figure 3: Parameters of an oblique image
The general equations for measuring orthogonal GSD (𝐺𝑆𝐷𝑜𝑟𝑡ℎ ), façade GSD (𝐺𝑆𝐷𝑓𝑎𝑐𝑎𝑑𝑒 ), and ground
GSD (𝐺𝑆𝐷𝑔𝑟𝑜𝑢𝑛𝑑 ) are given by (Höhle, 2013):ℎ cos⁡(𝛽 − 𝑡)
𝐺𝑆𝐷𝑜𝑟𝑡ℎ = 𝑝𝑒𝑙 ′ ( )(
)
𝑐
cos 𝛽
cos⁡(𝛽 − 𝑡)
𝐺𝑆𝐷𝑓𝑎𝑐𝑎𝑑𝑒 ⁡ ≈ ⁡ 𝐺𝑆𝐷𝑜𝑟𝑡ℎ ⁡()
sin 𝛽cos(𝛽 − 𝑡)
𝐺𝑆𝐷𝑔𝑟𝑜𝑢𝑛𝑑 ⁡ ≈ 𝐺𝑆𝐷𝑜𝑟𝑡ℎ (
)
cos 𝛽

(1)
(2)
(3)

where 𝑝𝑒𝑙 ′ =pixel size in image, ℎ=flying height above ground, 𝑐 =camera constant, 𝑡=tilt of camera axis,
𝛽=angle between a direct line from the lens to a target and the vertical line.
Image quality. The forward motion of the camera may cause blurred images if the exposure time is toolong. The image motion (∆𝑠 ′ ) can be measured by means of Equation 4 (Höhle, 2013):∆𝒔′ = (

𝒗. ∆𝒕′) . 𝒑𝒆𝒍′𝑮𝑺𝑫(4)where 𝑣=velocity, ∆𝑡 ′=exposure time, 𝑝𝑒𝑙′= pixel size in image, and 𝐺𝑆𝐷=ground sampling distance.
Overlap. The overlap ratio directly effects on the accuracy of 3D models. The shape of the overlap area
is a trapeze in oblique aerial images and the scale is usually variable in different scenes. The recommended standard
forward and lateral overlap is about 60% in this type of photography (Höhle, 2013).
3.2.1 Route Planning Algorithm
In the following section, the preliminary idea of Route Planning Algorithm is proposed for the bridge
inspection using the reinforced-concrete (RC) beam bridge. The algorithm receives BIM-GIS data as input andgenerates Shortest Path data as output. In this study, multiple circular-shaped layers are defined on the top of thebridge. The idea has been inspired by running several manual experiments for generating 3D models of the bridge.
Flying the drone on circular-shaped layers results in better 3D models to compare with horizontal or verticalmotions. The results revealed that by using circular-shaped layers and setting the required coverage and overlap
more details can be captured for the triangulation process. The final path shape of each layer is a polygon since
the drone stops in discrete points to collect imagery data.

388

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Route Planning Algorithm
Input: BIM-GIS data
Output: Shortest Path
1.
2.
3.
4.
5.
6.
7.

8.

Extract the details of bridge’s geometry, barriers information, and surrounding environment
information.
Define multiple circular- shaped layers on top of the bridge.
Determine interest points on the layers that the UAV is going to capture imagery;
Extract coordinates of interest points on the layers.
Calculate distance between each pair;
Check if any of point is overlapping with the barriers;
Set following parameters:
 Set camera angle of view; Set the distance between the camera and the bridge;
 Define UAV speed; Define the flight height;
 Set an adequate overlap of adjacent images;
Find the shortest path using Travelling Salesman Algorithm to capture imagery data;
return Shortest Path

3.2.2 Drone mapping deployment
In this project, the drone mapping deployment subsystem has been designed to show the effectiveness of
Route Planning Algorithm. DJI Co., Ltd is extending its drone control tools for third party development. By using
suitable Application Programming Interface (API), developers can create customized tools to automatically control
UAVs. DJI Software Development Kit (SDK) tools enable developers to design their custom missions. The mobile,window, and onboard SDKs are some available APIs that provide the ability of setting high and low level flight
as well as aircraft state by telemetry and sensor data, obstacle avoidance, camera and gimbal control, battery andremote controller, and predefined waypoint and hot-point. Note that mobile SDK supports iOS and Android
platforms, while the onboard SDK is supported by Windows, Linux, and others. It is also interesting to know that
onboard planning provides a new application called Ground Station that allow developers to create 2D and 3D
maps from photos captured by DJI Phantom.
3.3 Data analysis module
The goal of the proposed Data analysis module shown in Figure 1 is to visualise 3D point cloud using
imagery data captured by UAV photogrammetry in Route planning and data collection module. This system
contains two main subsystems including (I) 3D visualisation, and (II) Evaluation.
3D visualisation is a subsystem designed for representing 3D models of bridges using imagery datacollected by UAV photogrammetry. The aim of photogrammetry is to make measurements from photographs torecover the exact positions of surface points. Photogrammetric analysis can use high-speed photography andremote sensing to evaluate and collect 2D and 3D motion using imagery analysis. The number of photos used inphotogrammetry will determine the level of detail and accuracy of the measured area. In this project, Agisoft
Metashape, a popular 3D reconstruction software, is used to produce 3D point cloud by processing imagery dataof the bridges. The compilation process starts with digitising 3D surface data at a regular interval from imagery.
Then, these mass points will be completed by break lines at significant topography changes such as bridge edges
and hydrographic features. Such 3D models are convenient for visual inspection and creating database models.
The objective of the Evaluation subsystem is to check the quality and accuracy of 3D models. Due tothe lack of a benchmark for checking the accuracy of 3D model, highly reliable and accurate data has been assumed
for the 3D bridge models generated by UAV photogrammetry. Therefore, the accuracy of 3D models will beassessed by bridge engineers as there may be a variety of issues with the 3D model such as incomplete data, outlier
noise, geometric inaccuracies, etc. In each of these issues an appropriate modification is required to improve theaccuracy of 3D models such as gathering more data, changing camera orientation, applying relevant filter to images,
using GPS data alignment, changing flight height and overlapping ratio, etc. To this end, an Evaluation andmodification step is embedded in the framework to modify features of the route planning algorithm as well as
drone mapping deployment.

4. DISCUSSION AND CONCLUSION
The current practices of UAV-enabled bridge inspection require human controlling, which is time-389

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019consuming, labor intensive, and error-prone. In response to this problem, a novel route planning algorithm toautomatically fly UAVs over the shortest possible path using BIM-GIS data is proposed in this paper. The main
benefit of the route planning algorithm is to provide the capability of autonomous UAV flight to avoid redundantmanual commands. The autonomous flight does not occur until the UAV automatically decides where to fly
without a pilot interaction. To this end, future work may be needed not only to further improving the UAV’s
capabilities (e.g. sensor system, real-time data transmission), but also to advance data processing capabilities (e.g.
embedding computer vision algorithms to detect hazards in real-time to meet the requirements.
Interim results of an on-going project are reported in this paper. The next-step will be to test and validate
the effectiveness of the proposed route planning algorithm for real bridges. Additionally, the authors would liketo compare the accuracy of 3D bridge models generated by UAV photogrammetry with laser scanning. The
proposed method is suitable for a bridge consisting of small to medium spans. However, based on the orientationand size of the bridge, the proposed method may need to define different forms of layers to cover the whole surface.
The second part of this project is to extend the study to generate 3D models of larger bridges and combine circularshaped layers with horizontal and vertical layers. Another important direction for future work is to measure theoptimum angle of camera for the oblique photogrammetry in different scenarios. Given the recommended overlap
ratio, the optimum distance between two cameras will need to be investigated to obtain the best quality of data.
ACKNOWLEDGMENTS
Foremost, the authors would like to acknowledge the financial support by the University of Auckland
Faculty Research Development Fund (FRDF).
REFERENCES
Adhikari, R., Moselhi, O., & Bagchi, A. (2014). Image-based retrieval of concrete crack properties for bridge
inspection. Automation in construction, 39, 180-194.
Amirebrahimi, S., Rajabifard, A., Mendis, P., & Ngo, T. (2016). A framework for a microscale flood damage
assessment and visualization for a building using BIM–GIS integration. International Journal of Digital
Earth, 9(4), 363-386.
Chan, B., Guan, H., Jo, J., & Blumenstein, M. (2015). Towards UAV-based bridge inspection systems: A review
and an application perspective. Structural Monitoring and Maintenance, 2(3), 283-300.
Chen, S., Laefer, D. F., Mangina, E., Zolanvari, S. I., & Byrne, J. (2019). UAV Bridge Inspection throughEvaluated 3D Reconstructions. Journal of Bridge Engineering, 24(4), 05019001.
Dandois, J., Olano, M., & Ellis, E. (2015). Optimal altitude, overlap, and weather conditions for computer vision
UAV estimates of forest structure. Remote Sensing, 7(10), 13895-13920.
Dong, S., Shao, X., Kang, X., Yang, F., & He, X. (2016). Extrinsic calibration of a non-overlapping camera
network based on close-range photogrammetry. Applied optics, 55(23), 6363-6370.
El Meouche, R., Rezoug, M., & Hijazi, I. (2013). Integrating and managing BIM in GIS, software review.
International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2, W2.
Goel, U., Varshney, S., Jain, A., Maheshwari, S., & Shukla, A. (2018). Three Dimensional Path Planning for
UAVs in Dynamic Environment using Glow-worm Swarm Optimization. Procedia computer science,
133, 230-239.
Guerrero, J. A., & Bestaoui, Y. (2013). UAV path planning for structure inspection in windy environments.
Journal of Intelligent & Robotic Systems, 69(1-4), 297-311.
He, Z., & Zhao, L. (2017). The comparison of four UAV path planning algorithms based on geometry searchalgorithm. Paper presented at the 2017 9th International Conference on Intelligent Human-Machine
Systems and Cybernetics (IHMSC).
Höhle, J. (2013). Oblique aerial images and their use in cultural heritage documentation. Proc. Int. Archives of the
Photogrammetry, Remote Sensing and Spatial Information Sciences, 5, W2.
Jusuf, S. K., Mousseau, B., Godfroid, G., & Hui, V. S. J. (2017). Integrated modeling of CityGML and IFC forcity/neighborhood development for urban microclimates analysis. Energy Procedia, 122, 145-150.
Kang, T. W., & Hong, C. H. (2015). A study on software architecture for effective BIM/GIS-based facility
management data integration. Automation in construction, 54, 25-38.
Karan, E. P., & Irizarry, J. (2015). Extending BIM interoperability to preconstruction operations using geospatial
analyses and semantic web services. Automation in construction, 53, 1-12.
Lei, B., Wang, N., Xu, P., & Song, G. (2018). New crack detection method for bridge inspection using UAV
incorporating image processing. Journal of Aerospace Engineering, 31(5), 04018058.
Lu, Y., Wu, Z., Chang, R., & Li, Y. (2017). Building Information Modeling (BIM) for green buildings: A criticalreview and future directions. Automation in construction, 83, 134-148.
Maulik, U., & Bandyopadhyay, S. (2000). Genetic algorithm-based clustering technique. Pattern recognition,
33(9), 1455-1465.

390

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Mignard, C., & Nicolle, C. (2014). Merging BIM and GIS using ontologies application to urban facility
management in ACTIVe3D. Computers in Industry, 65(9), 1276-1290.
Rafiee, A., Dias, E., Fruijtier, S., & Scholten, H. (2014). From BIM to geo-analysis: view coverage and shadow
analysis by BIM/GIS integration. Procedia Environmental Sciences, 22, 397-402.
Rosenkrantz, D.J., Stearns, R.E., & Lewis II, P.M. (1977). An analysis of several heuristics for the travelingsalesman problem, SIAM J. Comput. 6, 563-581.
Savuran, H., & Karakaya, M. (2015). Route optimization method for unmanned air vehicle launched from a carrier.
Lecture Notes on Software Engineering, 3(4), 279.
Teo, T.-A., & Cho, K.-H. (2016). BIM-oriented indoor network model for indoor and outdoor combined route
planning. Advanced Engineering Informatics, 30(3), 268-282.
Volk, R., Stengel, J., & Schultmann, F. (2014). Building Information Modeling (BIM) for existing buildings—
Literature review and future needs. Automation in construction, 38, 109-127.
Wu, I. C., & Hsieh, S. H. (2007). Transformation from IFC data model to GML data model: methodology and tool
development. Journal of the Chinese Institute of Engineers, 30(6), 1085-1090.
Yeum, C. M., & Dyke, S. J. (2015). Vision‐ based automated crack detection for bridge inspection. Computer‐
Aided Civil and Infrastructure Engineering, 30(10), 759-770.
Zhang, Y., Yuan, X., Fang, Y., & Chen, S. (2017). UAV low altitude photogrammetry for power line inspection.
ISPRS International Journal of Geo-Information, 6(1), 14.
Zhu, J., Wright, G., Wang, J., & Wang, X. (2018). A critical review of the integration of geographic information
system and building information modelling at the data level. ISPRS International Journal of GeoInformation, 7(2), 66.
Zou, Y., Gonzalez, V., Lim, J., Amor, R., Guo, B. H., & Jelodar, M. B. (2019). Systematic Framework for Postearthquake Bridge Inspection through UAV and 3D BIM Reconstruction. CIB World Building Congress(WBC) 2019, 17-22 June 2019, Hong Kong.

391

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

MEASURING RAILWAY FACILITIES BY USING TWO MOBILELASER SCANNERS DIRECTLY ABOVE THE RAILS
Kohei Yamamoto1 and Nobuyoshi Yabuki2
1) PASCO Corp., Japan. Email: kootho1810@pasco.co.jp
2) Ph.D., Prof., Division of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka
University, Suita, Japan. Email: yabuki@see.eng.osaka-u.ac.jp

Abstract: To measure three-dimensional shape of the rail is important for the safe operation of trains.
Conventional rail track measuring methods tend to be time-consuming, labor-intensive, and error prone. To
grasp positions and shapes of railway facilities which are close to the clearance gauge, rapidly, efficiently andprecisely, this research proposes a new measurement method by using two mobile laser scanners directly above
the rails. In this method, two Mobile Laser Scanners (MLSs) are installed directly above rails so thathigh-precision point cloud data can be obtained more rapidly than the single MLS method. The point cloud dataobtained by two MLSs directly above the rails can be accurately coincided using the position of rail track center
lines. The point cloud data of MLSs are compared with the point cloud collected by Terrestrial Laser Scanner(TLS). Finally, this method was applied to the test rail line where two electric poles are installed and it was
identified within 0.01m of relative accuracy compared with TLS. The applicability of this method has beenverified to both curve and straight tracks.
Keywords:

Mobile Laser Scanning, 3D Point Clouds, Registration, Facility Management, Monitoring.

1. INTRODUCTION
Since the safe operation of trains is of utmost importance, the railroad is regularly monitored with therequired accuracy and frequency according to the standards set by railway operators. The conventional methods
that the railway maintenance service operation currently uses tend to be time-consuming and costly.
For the local railway operators it is often challenging to meet the highest safety level with limited
resources and budgets. Major railway companies, such as JR group, have train collision accidents caused by
obstruction to the clearance gauge for the construction limitation in recent 5 years, even they regularly monitored
the condition of the spatial environment with the clearance gauge measuring train which costs hundreds million
yen. Mainly clearance gauge are grasped with a measuring ruler or instruments by manpower to quantitatively
grasp the difference between obstacles and clearance gauges (Wei, et al., 2013). It is now common to measure
the rail track accurately with various laser measurement methods (Soni, et al., 2014). The monitoring usage oflaser scanner has been exemplified with TLS data at a train station. Using extremely highly accurate data set theresult of fitting to 3D models of the rail was better than 3mm in relative accuracy.
Mobile Mapping Systems (MMS) are able to realize fast, redundant and automatic data collection with
Mobile Laser Scanning (MLS) on the vehicle. The challenge is to process with massive amounts of data sets forrailway monitoring and management. The algorithms of the automatic rail extraction with ICP algorithm (Besl,& Mackay, 1992) combined with principal component analysis (PCA) had been proposed (Niina et al., 2018).
The accuracy of the experiment was less than 3mm. It had mentioned that the result of extracted position was
inappropriate at the curve section of rails. Furthermore, the advanced algorism for the rail detection andextraction from MLS data with machine leaning had been proposed (Sánchez-Rodríguez et al., 2019). It could beclassified through Support Vector machine although it is difficult to separate point clouds regarding to rails
manually from the ground.
There are many researches to improve efficiency for fitting rail models to point clouds (Asscheman,
2017; Benito, 2012; Elberink & Khoshelham, 2015; Pastucha, 2017; Yang & Fang, 2015). The voxel-based
segmentation algorithm and reconstruction with ground mesh had applied to improve the discernibility of point
cloud at the railway environment (Asscheman, 2017). As the algorithm of more focused to the properties of therail gauge the algorithm to use pieces of rail models had been developed (Benito, 2012). The iterative calculation
had adapted to automatically fit to laser point clouds and to reconstruct models with a curvature adjustment tolonger direction. Moreover, the essential properties that a pair of rail tracks is parallel and locally linear had
applied to the model fitting algorithms (Elberink & Khoshelham, 2015). The accuracy of the extracted
centerlines was between 0.01 and 0.03m. Usually the partial analysis about the gap of height around the track
had performed to find the rail. To improve the detection ratio the data-driven approach which utilized the
combination of geometric and radiometric information had been applied (Yang & Fang, 2015). On the other
hand the data-driven approach tends to be influenced to point density and quality of point clouds. Furthermore,
the stepwise narrowing window to determinate the rail head position had been created with properties of rail
track parameters for analysis (Pastucha, 2017). This algorithm contains the avoidance of problematic sections for
extraction and simply line fitting with cross sectioned point clouds of the rail heads.

392

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

In addition, focusing on the single rotation of the laser scanner data the rail model matching method for
localization and detection had been proposed (Heckel, et al., 2015; Stain, 2017). The method had analyzed theline of scanner data around the rail position including the noise echo data which appear the corner of the rail
head with the accuracy of 0.01m so that the position of railway switch can be detected. However, the position of
the head of laser scanner was at relatively high position up above the center of the track. Therefore, it could be
difficult to treat the specular reflectance noise at the rail head surface.
In particular, many methods fitting the rail model to the point cloud with noise data for specularreflectance often omit the data with the outlier filter because point clouds which acquired with the fixed scanningor the mobile scanning with low speed are enough dense even after the filtering. On the other hand, withpractical speed point cloud data of mobile laser scanning is so sparse that it can’t be omitted to shape the rail
head object for the extraction of track centerline.
To reduce the influence of the reflective noise from the rail head we developed an MLS system directly
above the rails (Yamamoto, et al., 2018). Accurate track centerlines which are calculated according to railparameter sets can be extracted. Moreover, simultaneous data acquisition from forward and backward direction
by two mobile laser scanners can obtain majority of the surface of the railway facilities.
In this paper, we verify the applicability of this system installed on the road-rail vehicle running on
actual railway environment. The experiment showed the same level of accuracy as the TLS method. According
to the experimental result the accuracy can be as same as TLS and better about the efficiency.
2. METHOD
2.1 Set up Two Mobile Laser Scanners directly above the rails
MLS can collect point cloud data which is processed with several sensor data such as Global Navigation
Satellite System (GNSS), Inertial Measurement Unit (IMU) and an odometer. The laser scanner accumulates
radiometric, distance and angle information which can be reconstructed as 3D coordinates with extrinsic
information calculated with data from other sensors.
The deployment of the laser scanner is usually a center of the feature object to be measured. Therefore,
in several researches in regard to rail position, MLS are equipped on the center of a rail track at the middle pointof the gauge. With this deployment one laser scanner can irradiate only the gauge side surface of rails. On thesurface of the rail, when the angle of incidence around the rail head surface exceeds 60 degree angle the point
cloud can be reconstructed incorrect position and called echo noise. Furthermore, there occurs speckle noisearound the gauge side corner of the rail head surface. Thus, point clouds of scanline collected from the rail would
probabilistically contain false positions of the rail.
In order to make the best use of the laser scanner point cloud data this paper propose to make simple
about the position relation between one laser scanner and rail head. The position of the light emitted part of thelaser scanner is deployed on the center of the rail position so that the effect of measurement noise can be
confined to the distance from the light emitted part to the rail head. Figure 1 shows the condition of the
deployment of two mobile Laser scanners directly above the rails.
If it were measured with one mobile laser scanner directly above one rail and measured for another railin turning way the gap of each point clouds would become larger than the case of two mobile laser scanners
because of time difference of the accuracy of GNSS. Moreover, for time-saving the efficiency of data acquisition

Figure 1. Deployment of Two Mobile Laser Scanners directly above the rails393

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019by the simultaneous use of two laser scanners increases more than the single scanner.
Figure 2 shows the overall processing of the algorithm for two mobile laser scanners directly above therail. First, once the data acquired, 3D reconstruction for point data will be processed and every scanline will be
extracted with the time of GNSS/IMU. Second, the rail position will be calculated from each scanline which isanalyzed with parameter set of the rail. Third, the center position of track which means in the middle point
between both rail heads will be extracted. Fourth, two sets of scanlines can be matched using track center points.
Finally, two sets of the data will be matched and be able to manipulate with 3D viewer.
2.2

Parametric extraction of the rails
Figure 3 shows a scanline of Laser Scanner A of Figure 2. Using the time attribute from GNSS/IMU
each scanlines can be extracted from the point clouds. Rail A locates directly below the Laser Scanner A andRail B exists just beside Rail A as a narrow gauge. Therefore, the scanline around the rail A can be geometricallyextracted from the condition of the gauge size. Figure 4 shows parameters that can be applied to point clouds ofthe rail head surface of both rails. Point clouds of the head top surface of Rail A has lineally symmetricalignment with the center axis of Rail A. On the other hand the scanline of Rail B is asymmetric.
For the Rail A the coordinates of the rail center position can be calculated with the point clouds.
However, it must be corrected from the influence from speckle noise around the rail head surface. First, withcollection of the point clouds around the rail head surface, the rail center position can be calculated by using
mean and weighted mean to the coordinates. Horizontal coordinates can be determined by average value.
Furthermore, vertical coordinates can be determined by weighted means because the influence of specularreflection is restricted to the Z value with proposed method. It can be defined as follows.
nX

X w  i 1
n

i

n

Y

Yw  i 1
n

(1)

i

n

W Z

Z w  i 1n

i

i

W
i 1

i

Here, Xi, Yi, Zi (i=1,2,…,n) means coordinates of one point around the rail head surface, Wi (i = 1, 2, …,n) means the value of reflection for weighting. Xw, Yw, Zw means the coordinates of the rail center position.

Second, for the Rail B the cant point can be extracted as the maximum height of the point clouds
around the rail head surface. Therefore, the edge of the rail gauge can be calculated with position of thepoint of the rail head side around 14mm bellow from the gauge line. The gauge line is drawn between thecenter position of Rail A and cant point of Rail B. Subsequently, perpendicular line can be drawn down to
the gauge line and intersection points can be obtained as the edge of the gauge line.
Finally, the midpoint of the gauge line segment can be determined as the track center position of
the scan line. These calculations can repeat for the scanlines continuously so that points of the rail trackcenter positions will form the rail track centerline.

394

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 20192.3

Registration with track center positions
To make maximum use of the two sets of the point cloud acquired by MLSs it is popular to apply two
types of registration method. One is to use control points and the other is the calibration method. Both methods
require performing an accurate direct survey for control points and features which takes cost and time. Therefore,
this paper proposes a quick registration method using with the rail track center positions. The rail track center
position is common for each data set of MLSs. Moreover, there exists corresponding positions between the rail
track center positions because every point cloud includes the time attribute from GNSS/IMU.
The difference between rail track center lines can be considered as the gaps of each scanline of two setsof point clouds of each MLS. From these gaps the translation vectors for scanline can be generated. The
threshold of gaps will be given to the prediction error value of GNSS/IMU so that the irregular error value canbe excluded. Between each translation vectors liner interpolation can be applied because the longitudinalintervals of scanlines are relatively short compared to the change of vertical position of the rail. The scanlines of
MLS of which the prediction error value is not superior to the others will translate to fit to the corresponding
scanlines according to the translation vectors.
Thus, the position of point cloud will be matched to the others according to the accuracy of determinedtrack center positions. Both of the laser scanning data will be able to display on the 3D Viewer at the same time.
The environment of 3D Viewer software enables to examine with cross and longitude sections at any position.
3. RESULTS
3.1 Data Registration of point clouds simultaneously acquired by Two Mobile Laser Scanner
In order to evaluate proposed method we carried out the experiment to verify the position and shape ofthe rail and facilities of the railway. We used the test rail for construction vehicles at Kobe machinery
maintenance center of Nikken Corporation in Kobe City of Hyogo Prefecture. The test rail composed of thestraight rail and the curve rail which merged with the left-hand railroad switch. Figure 6 shows the external viewof MLS on road-rail vehicle. Table 1 shows the specification of the MLS data acquisition. The target of speed of
the vehicle are 14.4km per hour so that the separation of scanlines for the longer direction 0.02m. The
deployment of MLS is Front and Rear in Figure 6 and the other pattern is to deploy side by side on the Rear
Deck. MLS run one round trip of which course includes curve and straight rails. Table 2 shows the result ofcomparison between MLS and TS coordinates. The accuracy of the position seemed to be satisfied with thespecification of the MLS because the horizontal accuracies were within 0.02m. On the other hand, regarding to
the vertical accuracy of Laser Scanner A had a difference from the target. Therefore, target of the registration is
Laser Scanner B because of the smaller difference than Laser Scanner A regarding the vertical accuracy.
Figure 7 shows the example of rail position analysis. The tool which we programed read each scanlines

Table 1. Specification of MLS data acquisition
Specification
Speed of vehicles
Product Name
Absolute accuracy
Height of
Laser Scanner
Scan rate
Laser
Scanner
Pulse rate
Relativeaccuracy
Scanline Moving direction
interval
Lateral direction

Contents
14.4km/h
Leica Pegasus Two
0.02m

Pegasus Two

Pegasus Two

2.7 or 2.2m
1.016 Million pixel /s
12000rpm (200Hz)±0.005m
0.02m
0.005mFigure 6. External view of Two MLSs on road-rail vehicle

Table 2. Comparison MLS to TS target coordinates
Laser
ScannerA
Laser
Scanner
B

Standard
deviation
AverageStandard
deviation
Average

x(m)
0.012

y(m)
0.010

z(m)
0.012

-0.0050.007-0.0030.007-0.0780.017-0.007-0.004-0.005Gauge line
Rail position
Scanlines

Figure 7.

395

Rail position analysis for straight railProceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

200 (mm)

Track Center position
after registration

Registered Point Cloud
Laser Scanner A

Track Center position
before registration100

Laser Scanner B

0

-600

-500

-400

-300

-200

-100

0

200

100

300

500

400

600

-100

Figure 8. Cross-section of the point cloud data after the registration with track center positon
Pole #1

Pole #2
Laser
Scanner A
Laser
Scanner B
MeasurepointsMeasurepoints

Laser
Scanner A
Laser
Scanner B
MeasurepointsMeasurepoints

TLS

TLS

Figure 9. Horizontal cross section of point clouds around electric poles of railway

Table 3. Comparison MLS to TS about radius of electric poles
TLS
Pole#1(A)

147.97

Pole#2(B)

Measurements (mm)MLS Curve
MLS Straight
Pole#2
Pole#1
Pole#2
Pole#1
(BMC)
(AMS)
(BMS)
(AMC)

ΔAMC=
A-AMC

ΔBMC=
B-BMC

ΔAMS=
A-AMS

ΔBMS=
B-BMS

150.09145.15

2.81

2.56

2.94

0.42

147.52145.03149.67

Differences (mm)of data set of MLS and analyzed center position of the rail head surface and the geometrical position of the rail
gauge according to the parameter set. The same process had applied to other MLS data.From the track centre
points of gauge lines the translation vector was calculated and applied the proposed algorithms. Figure 8 shows
the cross-section of the point cloud data after the registration with track center position. The difference of trackcentre position is within 0.005m.
3.2

Data evaluation
To confirm the availability of proposed method the registration of point clouds had evaluated with TLS
point clouds data with the electric pole diameter. In the railway environment the laser scanning method has
difficulty to acquire whole part of the cylindrical object such as an electric pole. Even TLS requires manyscanner positions to cover cylindrical surface from every directions. Figure 9 shows the horizontal cross section
at the 1.5m from the ground for two poles. At the pole #1 TLS acquired from one direction beside Mobile Laser
Scanning on the rail had irradiated from forward or backward direction so it could collect a half to two third of
the whole direction at the cylindrical surface of the pole. On the other hand at the pole #2 TLS point clouds had
formed perfect circle.
In order to evaluate the object integrity the radius of pole objects were compared with the least squares
method. The radius was calculated with three points extracted from inside of a circle or arc shapes on thehorizontal section display so that the relative accuracy of point cloud after registration with two MLS can becompared with TLS. MLS measured poles from the straight track and the curve track so that the effect to theregistration between the straight track and the curve track can be compared.
Table 3 shows the results of the calculation of radius. The measurements AMC, BMC and AMS show within
3.0mm difference from TLS results. On the other hand at BMS shows almost the same length of B.
4. DISCUSSION
According to the result the relative accuracy of proposed methods can be satisfied within10mm of396

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

accuracy which is required for facility management of the railway. Simultaneous data acquisition of two mobile
laser scanners can contribute to the high accuracy of data registration because the positions and attitude of MLSs
are not so different from each other. The result of registration is enough accurate to apply to collect the positionof facilities of the rails. Moreover the result indicated that the possibility to automate future extraction such aspoles can satisfy the accuracy of 10mm. It is possible to select the point cloud and to apply with the least squares
method to estimate the center position of the circle or arc shape. To apply iteratively to the point cloud with thisoperation the center line of pole object can be extracted and can create pole models with radius length of thevalue. Challenges are to control the quality of the registration and to identify the point clouds around the poles.
For the control of the registration quality the roll angle control can be considered. To adjust the height value ofthe extracted rail gauge line such as a smoothing method to the longer direction it can improve the roll angle ofregistration. And for the identification method of the poles, with the sliced plane in the certain elevation point
clouds can be reduced and can be evaluated with the shapes of the circle. The plane can be transformed theimage and be analyzed with the shape of circle shape to identify the center position of the circle approximately.
5. CONCLUSIONSThis research proposes a new measurement method by using two mobile laser scanners directly above
the rails. In this method, two Mobile Laser Scanners (MLSs) are installed directly above rails so thathigh-precision point cloud data can be obtained more rapidly than the single MLS method. In addition to the dataacquisition the data processing method which extracts the position of rails referring the parameters of the gaugeof the rail are developed. Moreover for the prevention of the influence of specular reflection it was effective toapply weighted means for the calculation of rail head position. The point cloud data acquired by two MLSs
directly above the rails can be accurately coincided using the position of rail track center lines. The point clouddata of MLSs are compared with TLS so that the proposed method was applied to the test rail line where twoelectric poles are installed and it was identified within 0.01m of relative accuracy.
The experiment showed the same level of accuracy as the TLS method. According to the experimental
result the accuracy can be as same as TLS and better about the efficiency.
REFERENCES
Asscheman, M.J. (2017). Automatic object segmentation and reconstruction in LIDAR Point Clouds of railway
environments, MS thesis, Utrecht University Virtual worlds Computer Science.
Benito, D. (2012). Automatic 3D modeling of Train Rails in a LiDAR Point Cloud, MSc thesis, University ofTwente Faculty of Geo-Information and Earth Observation (ITC).
Besl, P.J. and Mckay, N.D. (1992). A Method for Registration of 3-D Shapes, IEEE Transactions on Patten
Analysis and Machine Intelligence, Vol.14, No.2, pp.239-256.
Elberink, S. O. and Khoshelham, K. (2015). Automatic Extraction of Railroad Centerlines from Mobile Laser
Scanning Data, Remote Sens, Vol.7, no.5, pp.5565-5583.
Hackel, T., Stein, D., Maindorferz, I., Lauery, M., and Reiterer, A. (2015). Track detection in 3D laser scanning
data of railway infrastructure, In: Proc. IEEE Int. Instrum. and Meas. Technol. Conf. pp. 693–698.
Niina, Y., Honma, R., Honma, Y., Kondo, K., Tsuji, K., Hiramatsu, T., and Oketani, E. (2018). Automatic Rail
Extraction and Clearance Check with a Point Cloud Captured by MLS in a Railway, The International
Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2,
pp.767-771.
Pastucha, E. (2017). Point Cloud Classification and track center Determination in point cloud collected by MMS
on Rail, INGEO 2017 – 7th International Conference on Engineering Surveying.
Sánchez-Rodríguez, A., Riveiro, B., Soilán, M., and González-deSantos, L.M. (2018). Automated detection anddecomposition of railway tunnels from mobile laser scanning datasets, Automation in Construction, 96,
pp.171–179.
Soni, A., Robson, S., and Gleeson, B. (2014). Extracting rail track geometry from static terrestrial laser scans
for monitoring purposes, The International Archives of the Photogrammetry, Remote Sensing andSpatial Information Sciences, Volume XL-5, pp.553-557.
Yamamoto, K., Yaoita, A., and Yabuki, N. (2018). Modeling Method of Rail Clearance Gauge by Using Two
Mobile Laser Scanners Directly Above The Rails. Journal of Japan Society of Civil Engineers, Ser. F3(Civil Engineering Informatics), 74 (2), pp I_70-I_81.
Yang, B. and Fang, L. (2014). Automated Extraction of 3-D Railway Tracks from Mobile Laser Scanning Point
Clouds, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 7, pp.
4750–4761.
Wei, H., Zue, H., Wang, Z., and Wu, W.(2013). Center Line Coordinates Survey for Existing Railway by 3-D
Constraints Method, TELKOMNIKA, Vol. 11, No.5. pp.2816-2821.

397

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Information and Process Management398Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

A FEASIBILITY STUDY FOR LDAP CERTIFICATION IN COLLABORATIONWITH EXISTING ACCOUNTS IN RDBMS
Yoshiyuki Yokoyama1
1) Project Manager, Department of Systems Engineering, Japan Construction Information Center, Minato, Tokyo, Japan.
Email: yokoyamy@jacic.or.jp

Abstract: For the purpose of improving productivity, we, JACIC, have been trying to apply a cloud storage
product on our already existing application. This idea will have our users use a new login account instead of their
existing account to be certified by a cloud storage product. However, since this would put a burden on users byrequiring them to keep track of multiple accounts, we have focused on finding a single sign-on solution with LDAP.
OpenLDAP, a product that implements LDAP, as well as BerkeleyDB and HierarchicalDB, are used as a backend system for storing login accounts. However, RDBMS also could have been used at least from its documents.
If the RDBMS is attached as the back-end for OpenLDAP, the cloud will be able to access the existing login
accounts through an LDAP interface. Thus, users will be certified by the cloud without having to register multiple
accounts. However, there are a few OpenLDAP + RDBMS test cases online which have made us hesitate to choose
LDAP as our single sign-on solution. This feasibility study shows some points of attention for creating andoperating an OpenLDAP + RDBMS environment.
Keywords: single sign-on, OpenLDAP ,back-sql, slapd-sql, RDBMS
1. INTRODUCTION
1.1 Issue
JACIC has been operating a web-based information service for supporting the 3Rs (Reduce, Reuse,
Recycle) in public construction projects (Our Service). Our Service enables information sharing among
administration offices, construction project owners and construction companies via digitalized recycling
plan/report forms written by the construction companies. Our Service has contributed to improving part of theconstruction process for more than 15 years by digitalizing recycling information. However, one issue was pointed
out in practice. In actual use, many processes which are paper-based, such as manual writing, are still being used.
For instance, in private construction projects, the recycling plan/report form is written by hand, and photos anddrawings must be submitted with the recycling plan at the beginning of the project. The administration office willstore them in a physical holder on a shelf as the basis of their workflow. This forces construction companies toreprint the recycling plan form on paper even if they have already digitalized it into Our Service.
The Author chooses the proper application for each task and for word processing, calculates withspreadsheets and prepares for presentations as part of the daily tasks, then, consequently, prints the data for
confirmation and elaboration as well as saving paper documents which were handed out at meetings into PDF files
before disposing of them as Figure 1 shows. Finally, the Author stores them as a set of documents related to
identical projects into spaces placed around the desktop PC or in shared folders on the network. This shows ourtasks will be completed by not only well-structured data formats generated by sophisticated business applications,but also non-structured data files generated by office suites such as general applications. Therefore, the same
situation might happen during a construction project.

Figure 1. paper based submitting399

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Digitalizing only a part of the process out of the whole construction process forces users to have a mixof handling digital files and papers, which is a burden. On the other hand, digitalizing all of the remaining paperbased processes will require a lot of coordination. The processes are different at each organization and a wide
diversity of functionality of applications is needed. There will be many difficulties to solve and it will take a longtime to figure it all out, but people will never be free of this type of burden in their practice, so we have been
considering file sharing via a cloud storage service as a short term solution as Figure 2 shows.

Figure 2. Shared in Cloud storage
In this solution, we considered that enabling existing user accounts in Our Service without using new
accounts for the cloud will make it so that our users will not have to have multiple accounts.
2.1 Types of Single Sign-On
Single Sign-On (SSO) allow users to access many kinds of IT services by using one account. There areseveral types of SSOs with different ways of implementation and histories, as Table 1 shows.
LDAP can be called the first SSO solution via the internet (IETF, 1993). After LDAP, SOAP/XML
technology was released and made it easier to become connected between different internet domains. It seems that
SOAP/XML led to the appearance of SAML (OASIS, 2002), and later, the advancement of the federation method.
In 2006, OpenID, which was based on the RESTful interface for its communication, made its debut (OpenID
Foundation, 2006). Since that time, people have been able to access a large number of websites using a single SNS
account. Currently, OpenID is integrated with OAuth 2.0 into OpenID Connect (OpenID Foundation, 2014).
Table 1. Types of SSO

2.2 Products for the trial
In this study, there were some restrictions to choose the products for the trial for both hardware andsoftware. Regarding hardware, such as the server machine, was two guest OSs on a hypervisor. We allowed useof 4MB of memory and two sockets of virtual CPU. Then, for software, we used free, open source software,
including the OS, database and cloud storage.
As the result of our survey, we found that ownCloud 9, which was a candidate for our open-sourced cloudstorage product (owncloud, 2017), has an LDAP-based user certification add-on module as described in the bold

400

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

frame in Table 2 below. Thus, we decided to use LDAP for an SSO in this study.

Table 2. ownCloud’s add-on module for SSO

CentOS, which was used in the trial, has OpenLDAP 2.4.40(OpenLDAP, 2014) as its distributed package
by default. OpenLDAP has a functionality to use RDBMS for its data store even it isexperimental(OpenLDAP,2018a). The functionality is called back-sql or slapd-sql (back-sql).
However, there were not enough test cases or examples of back-sql use online, especially with Oracle, toevaluate for feasibility. Since this is a feasibility study for how to build an SSO environment with back-sql, thatmeans a combination of OpenLDAP and Oracle.
2. METHOD
2.1 Planned structure for trial scenario
Figure 3 shows a planned structure which was combined with OpenLDAP and Oracle attaching to backsql. Our Service’s users were already using Oracle. The user certification add-on module was installed into cloudstorage, which, in this case, was ownCloud 9.

Figure 3. Planned structure
When a user tries to access the cloud storage, a user certification add-on module will ask OpenLDAP tosearch. Next, OpenLDAP will send SQL to Oracle to `SELECT` user.
After Oracle returns the user account, meaning the user ID and password, openLDAP gets the result via
back-sql and returns user certification to the add-on module. Finally, the user ID and password are evaluated tomatch against the text the user typed in the sign-in screen for the cloud storage.
LDAP Schema Mapping
For the RDBMS tables (Schema mapping), back-sql needs to map the ldap. In particular, a treestructured LDAP schema into two dimensions for the RDBMS tables. For the trial, there are sample sql files in
openLDAP’s directory for creating tables.
We analyzed sample sql files bundled in “servers/slapd/back-sql/rdbms_depend/” to understand therelationship between the LDAP schema and the RDBMS table. Figure 4 is a minimum set of tables to create andtheir values for our study.
2.2

401

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 4. Schema mapping
2.3 View for “persons” table
A view definition is needed in Oracle for selecting user account information so that PostgreSQL canpretend to have the persons table itself, because openLDAP will search for the “persons” table from PostgreSQL
via back-sql. Also, for avoiding mismatched password text between that typed by users and the original Oracle
database, adding “{CRYPT}” is needed before the original password text (e.g., “{CRYPT}passwordtext”) in theview definition. Figure 5 is the view definition we used in Oracle and foreign table definitions in PostgreSQL.

Figure 5. View for “persons” table
2.3 View for “ldap_entries” table
In order to synchronize the user accounts and “ldap_entries” table dynamically, a view definition in
PostgreSQL is needed. Figure 6 is the view definition we made.

Figure 6. View for “ldap_entries” table402

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

3. RESULT
There were some obstacles we encountered when building an SSO environment. After repeating sometrial and error, we found that our planned structure in Figure 1 could not enable LDAP SSO. The following are
major obstacles related to product level issues we encountered during the trial which forced us to change theplanned structure.
(1) PHP
ownCloud’s add-on module for LDAP could not call ldap-related functions.
PHP 5.6.31 in CentOS 6.9 was not compiled with an LDAP module.
Changed PHP 5.6.25 from the Red Hat Software Collections edition (RHSCL, 2015), which was compiled
with an ldap module.
(2) OpenLDAP“ldapsearch” ended in an SQL error.
OpenLDAP 2.4.xx can only properly access PostgreSQL.
Generated SQL “SELECT” statement, including the “text()” function, which is dependent on PostgreSQL.
(3) PostgreSQL
PostgreSQL under Ver. 8 cannot access the external database.
Changed OS from CentOS 6 to CentOS 7 for upgrading from PostgreSQL Version 8 to 9.
Installed an FDW (Foreign Data Wrapper) plug-in, which enables PostgreSQL to connect to the Oracle
database.
Our final result is in Figure7, which we arrived at after avoiding many obstacles, especially (2) and (3).
We needed to place PostgreSQL between OpenLDAP and Oracle for the intimidated functional element to get
access from the LDAP request and to take user accounts from Oracle as the external foreign database.
Finally, we were able to sign in from the cloud storage by using existing user accounts in the Oracle
database.

Figure 7. Final structure
4. DISCUSSION
We succeeded in building an SSO environment by trying back-sql, which was used in combination withLDAP and Oracle. However, we had to use PostgreSQL for gluing openLDAP and Oracle under the conditions of
the selected version and the configuration parameters. As a result, we think we still need further improvements
such as in the following two areas.
(1) Password synchronization
Users cannot change their passwords while they are using cloud storage.
back-sql seems to not have an updated SQL statement.
We need to find reasonable solutions to synchronize passwords.
(2) Performance tuning
The response time is around five seconds on average under single usage, which is too long to wait formost users.
Performance tuning by indexing and caching-like solutions is needed.
5. CONCLUSIONSIn this study we proofed the feasibility of LDAP certification in collaboration with existing accounts in
RDBMS, which is important for Oracle-based applications. Collaborating with LDAP gives users the chance to
use their existing accounts as SSO platforms without any additional programing. However, for this solution,
further improvements are needed for practical use.

403

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

During our trial, we changed ownCloud 9 to Nextcloud 13 as our cloud storage product. We, unexpectedly,
were also given a chance to try the federation method of SSO. As the bold frame in Table 3 shows, Nextcloud 13
has other add-on modules for user certification by using SAML and OpenID Connect. We will try to build an
SSO environment for trying the federation method in the future.
Table 3. Nextcloud’s add-on modules for SSO

REFERENCES
IETF. (1993). RFC. Retrieved from IETF website: https://www.rfc-editor.org/rfc/rfc1487.txt
OASIS. (2002). Standard. Retrieved from OASIS website: https://www.oasis-open.org/standards#samlv1.0
OpenID Foundation. (2006). Obsolete Specifications, Yadis. Retrieved from OpenID Foundation website:
http://openid.net/specs/yadis-v1.0.pdf
OpenID Foundation. (2014). Specifications retrieved from OpenID Foundation website:
https://openid.net/specs/openid-connect-core-1_0.htm
ownCloud (2017). Older ownCloud versions. Retrieved form owncloud website:
https://owncloud.org/changelog/server/v9/
OpenLDAP(2014). older versions of OpenLDAP Software. Retrieved from OpenLDAP FTPsite:
ftp://ftp.openldap.org/pub/OpenLDAP/openldap-release
OpenLDAP(2018a).OpenLDAP Software 2.4 Administrator's Guide. Retrieved form OpenLDAP website:
http://www.openldap.org/doc/admin24/backends.html
Nextcloud(2018).Latest Nextcloud Server version. Retrieved from Nextcloudwebsite:https://nextcloud.com/changelog/#latest13
RHSCL(2015). Red Hat Software Collections Product Life Cycle. Red Hat website:
https://access.redhat.com/support/policy/updates/rhscl
OpenLDAP(2018b). OpenLDAP Technical Mailing List Interface. Retrieved form OpenLDAP website:
http://www.openldap.org/lists/openldap-technical/

404

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

SEMANTIC MODELING OF BUILDING CONSTRUCTION EMISSIONKNOWLEDGE
Wenkai Luo1, Guomin Zhang2, Lei Hou3, Malindu Sandanayake4
1) Ph.D. Candidate, School of Engineering, RMIT University, Melbourne, VIC, Australia. Email: s3691927@student.rmit.
edu.au2) Ph.D., Prof., School of Engineering, RMIT University, Melbourne, VIC, Australia. Email: kevin.zhang@rmit.edu.au
3) Ph.D., School of Engineering, RMIT University, Melbourne, VIC, Australia. Email: lei.hou@rmit.edu.au
4) Ph.D., School of Engineering and Science, Victoria University, Melbourne, VIC, Australia. Email: Malindu.Sandanayake@vu. edu.au

Abstract: Previous works on estimating construction carbon emissions from energy consumption were primarily
capitalizing on the quantity of construction materials, equipment and an emission inventory. It is envisaged thatwith the aid of Building Information Modeling (BIM) technologies that are featured by semantic-rich data andinformation, the present-day practice of energy and emission estimation can be well improved. Despite an ideal
BIM model typically encompasses information that ranges across building design, construction and operation
details, a rationale around how to leverage semantic-rich BIM to address building energy consumption and carbon
emission topics is still unclear. Under this backdrop, this study is centered on formulating a semantic-rich building
energy consumption ontological model that is capable of accurately calibrating the energy consumption andemissions of a building. The formulated model will consider various factors that can affect the calibration andestimation such as materials, fabrication, logistics, processing, and the like.
Keywords:

Construction Phase, Ontology, Emission Estimation

1. INTRODUCTION
Buildings provide great contributions to environmental emission throughout their life cycles (Hong et al.,2015; Chau et al., 2012). In previous studies, efforts were primarily put on the use phase, because it produces 8090% of the total emissions in the life cycle of buildings (Jonsson et al., 1998; Wu et al., 2012). In recent years,
research interest has gradually shifted to other phases such as the construction phase. The reasons of this change
are mainly from two aspects: 1) the advancement of applying energy-efficient materials and designs lead to thedecrease of energy consumption at the use phase (Li et al., 2010; Sandanayake et al., 2016a); 2) the realization of
the short-term intensive emission may cause more damage to the environment and society rather than the longterm mild one (Guggemos, 2005; Tam et al., 2002; Yan et al., 2010).
To reduce emissions in the construction phase, accurate and efficient estimation must be implemented
primarily. Although there are plenty of different emission estimation methods available from different studies,Life Cycle Assessment (LCA) is the most widely adopted technique by researchers to measure and compare
environmental impacts of a certain product or process (Finkbeiner et al., 2006). LCA usually has three analysis
approaches, Input/Output (I/O) based, Process-based, and Hybrid based. To analyze construction phase emissions,the process-based approach is easier to compute and define the assumptions, limitations and objectives(Sandanayake, 2016). However, this approach requires the input data with high quality and accuracy which isdifficult to be assured (Hendrickson et al., 1997), and a huge amount of data input is extremely time-consuming.
Along with the development of information technology in the Architecture, Engineering and Construction(AEC) industry, Building Information Modeling (BIM) has been widely used by different participants. BIM has
successfully helped to achieved efforts saving in many different scenarios in the AEC industry. Information storedin BIM could facilitate the construction emission estimation, however, information from BIM is not enough forthe estimation. Combination of information from other sources such as construction plan, emission inventory ishighly required. In the previous study, different methods were raised to satisfy the needs of connecting data fromdifferent resources to fulfill automation in many tasks (Pauwels et al., 2017). Based on various studies, semantic
web technology is deemed as a promising technique to link information across domains (Berners et al., 2001). As
ontologies are the core of a semantic web, combining information from different sources relies on the quality of
constructing the domain ontology (Giri, 2011).
This study aims to develop a construction emission domain ontology which contains the knowledge ofboth construction process and air emission. Connecting it with the building product ontology, known as the BIM
model, will enable more automatic and accurate estimation of the construction phase emission. This paper is
structured as follows: In Section 2, the research methodology for developing the proposed ontology is elaborated.
After that, the construction emission estimation methods and parameters involved are discovered in Section 3.
Following the discovery, the general architecture of the semantic model is designed in Section 4. The taxonomical
structure and attributes of a construction plan are explored in Section 5. At last, Section 6 summarises thecontributions and suggestions for the future work.

405

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2. RESEARCH METHODOLOGY
The key purpose of applying semantic web technology in this study is to link data across domains. In thiscase, reusing the existing ontology from other domain if available becomes the most natural and efficient choice.
According to the popular ontology developing method (Noy & McGuinness, 2000), there are seven steps asfollows: 1) determining the domain and scope of the ontology; 2) considering reuse of existing ontologies; 3)
enumerating important terms; 4) defining the classes and the class hierarchy; 5) defining the properties of classes;
6) defining the values for the properties; 7) creating class instances. Because this study is about an applicationlevel ontology development, it is more reasonable to firstly identify instances that have the relation with emission
calculation from construction activities rather than determine concepts/classes at first. Hence, the developing
process of this study follows the bottom-up procedures including the following four steps.
2.1 Defining the purpose and the scope of the building construction emission ontology
The purpose of developing the building construction emission ontology is not only formalizing theknowledge of construction emissions, but also supporting the integration of the knowledge with building
information models to simulate the generation of emission from construction activities. The data required in thesimulation only covers parts of the whole knowledge base of construction and emission domain. Therefore, on the
one hand, the ontology to be developed in this study is an application-oriented ontology within a sub-domain. On
the other hand, it’s essential to investigate the existing ontologies from different domains to identify what classes
are already defined, and how they can be utilized to formulate the emission estimation method. Due to the length
limit of the paper, only air emission is chosen as the object of this research.
2.2 Analyzing building construction emission mechanisms
Previous studies reveal that the LCA process-based method is the most accurate one to estimate theconstruction emission provided the data quality can be guaranteed. Because the purpose of developing the ontology
is to satisfy the needs of simulating the construction emission automatically through integrating emission
knowledge and building information models, it is essential to clarify the principle behind the LCA equations andthe origin of each individual factor. Carefully analyzing mechanisms can provide a profound understanding how
and to what aspects the construction activities may affect the emissions. Moreover, intermediate processes todetermine the equation factors from the existing ontological classes are required, and it is also required that certain
new classes be formalized through the analysis.
2.3 Matching in existing ontologies
As discussed previously, reusing existing ontologies or parts of them is one of the principles behind thesemantic web technology, that is sharing knowledge across different domains. Thus, acquiring data from existing
ontology and linking them with factors from the LCA equations is rational. Setting-up connections among thoseclasses and factors may help form the structure of the ontology and discover omissions of the classes and properties.
2.4 Developing building construction emission ontology
A building information model can certainly be information-rich, it is, however, limited by furnishingequivalently-adequate data related to construction activities (EI-Diraby, 2012). For instance, although BIM format
IndustryFoundation
Classes(IFC)(buildingSMART,
2016)
contains
a
class
named
Pset_EnvironmentalImpactIndicators, this class does not provide information about the emission volume and its
relation to the construction activities. the architecture of the proposed building construction emission ontology
needs to consider the interactions among building components, construction activities and different emission
substances. In this study, the ontology is modeled and edited using one of the most popular open-source tools
named Protégé.
3. ANALYSING THE CONSTRUCTION EMISSION MECHANISMS FOR BUILDING
3.1 Existing process-based emission estimation methods
Within process-based LCA methods, principles behind all equations are very similar in general. Airemission is generated from different sources, and the amount of the air emission depends on the quantity ofdifferent source and emission factors, while those factors reflect the features of different construction activities
and the features of different machine. Thus, the emission could be calculated through Equation (1) theoretically.
(1)‫ܧ‬௜ ൌ ܳ௝ ൈ ‫ܨܧ‬௜௝Where, Ei is the amount for emission type i, Qj is the quantity of related source j, and EFij is the emission
factor for emission type i from source j.
For different types of emission, this equation can be further developed based on the features of emission,
types of machine and the accessibility of related data. In previous work (Sandanayake, M., 2016b), a criterion forthe selection of emission estimation methods and standards was set up in the Australian context. Furthermore, thein-depth direct and in-direct emission mathematical models were developed. These models can depict the activity
level generation mechanism of different types of air emission.
(1) Estimation of Green House Gas (GHG) emissions from material transportationாிೕ ൈ௘ೕ ൈௗൈ௪(2)‫ܧ‬ሺீுீሻ் ൌଵ଴଴଴Where, ej is the energy consumption of the vehicle in GJ/ton-km, d is the one-way distance denoted by406

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

km and w is the total weight of the vehicle in tons. The emission factor EFj can be retrieved from the National
Greenhouse Accounts Factors (NGAF) (2018) depending on the fuel type.
(2) Estimation of GHG emissions from equipmentா஼ೕ ൈாிೕ ൈ௙೐ ൈ௅ிൈ்

(3)‫ܧ‬ሺீுீሻ௘௤ ൌଵ଴଴଴Where E(GHG)eq is the GHG emissions from construction equipment in kg, ECj is the energy content factor
of fuel type j (gigajoules per kilolitre or per cubic metre) used for equipment, fe is the fuel consumption of theequipment at full-load capacity, T is the hours of use of the equipment for the activity considered, and LF is theload factor, which is the fraction of available power during the operation of equipment.
(3) Estimation of non-GHG emissions from material transportation஺ೖ ൈாிሺಿಸሻೖ
(4)‫ܧ‬ሺேீሻ்ǡ௞ ൌଵ଴଴଴Where k is the non-GHG considered, Ak is the vehicle activity in km and EF(NG)k is the exhaust emission
factor for non-GHG k in kg/km, which can be obtained from the Australian National Inventory Report (NIR)(2011).
‫ܣ‬௞ ൌ

௙ൈ௘ೕ ൈௗൈ௪
ா஼ೕ(5)

Where f corresponds to the fuel capacity of the vehicle in km/L, ej is the energy consumption of the vehicle
in GJ/ton-km, d is the distance denoted by km, w is the weight of the loaded vehicle in tons and ECj is the energy
content factor of fuel type j in GJ/kl.
(4) Estimation of non-GHG emissions from equipment(6)‫ܧ‬ሺேீሻ௘௤ǡ௞ ൌ ‫ܨܧ‬௞ ൈ ܲ ൈ ܶ ൈ ‫ܨܮ‬Where EFk is the non-GHG emission factor for the emission substance k for equipment eq considered inkg/(kW-hr) and P is the rated power output of the equipment considered in kW, T is the usage hours and LF is theload factor, which is the fraction of available power during the operation of equipment.
(5) Embodied emissions from materials(7)‫ܧ‬௠ ൌ σ ܳ஻ைொ ൈ ሺͳ ൅ ߤሻ ൈ ݁௠
Where QBOQ is the quantity of a type of materials indicated in the bill of quantity (BOQ) of a construction
activity, μ is the waste factor for the material m and em is the emission factor for material m in kgCO2-eq/kg.
(6) Estimation of emissions from electric equipment௉ൈఎൈ௛ൈ௘೐೗೐೎
‫ܧ‬௘௟௘௖ ൌ(8)ଵ଴଴଴
Where p is the power of equipment in kW, ηis the efficiency of the equipment determined by the featureof equipment, h is the usage hours for activities considered, eelec is the emission factors of purchased electricity in
kgCO2-e/kWh.
(7) Estimation of emissions from construction waste(9)‫ܧ‬ௐ ൌ ܹ௜ ൈ ܳ௜
Where Wi is the waste factor for the type of waste material, and Qi is the amount of material delivered tothe site for the specific activity.
3.2 Mechanisms analysis
Although some variables from the above-listed in-depth models may imply inter-correlations with theexisting ontological classes, VWLOO are there some classes need to be formalized to underpin variables value
assignment. In Table 1, each variable is assigned with depending objects, which describe the mechanisms about
how the value of that variable is determined. Furthermore, data sources are categorized into two clusters, namely,
direct and indirect ones. A direct source means variables could find data directly from possible sources, while
those sources could be a building information model, an air emission report or a machine inventory. An indirectsource can only provide basic information for variables calculation. Take the variable d in the GHG estimationfrom transportation as an example, it requires the location information of both the project and vendors forcomputing purposes. To accurately determine the information for a specific variable, it is typically required toapply metadata to constitute the filter rule parameters.
There are some variables, for example, LF and T in Equation (3), that their values are mainly determinedfrom company norms or other previous studies’ empirical data. These data barely reflect the status or features ofthe project to be evaluated, in the meanwhile, only a little part of the construction plan information is involved in
the value calculation. Therefore, special attention on classes defining and relations setting up will need to be paid
to incorporating these types of information into the newly designed ontology.
4. THE STRUCTURE DESIGN OF CONSTRUCTION EMISSION ONTOLOGY
The intention of this study is to develop a model to formalize the construction emission knowledge.
Because an existing ontology named ifcOWL has already included concepts and relationships of construction
products and processes, they can be linked to the new construction emission ontology. Besides the building productmodel, there are another two sub-models that need to be developed, which form the construction emission ontology.

407

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

In my method, the construction sub-model and emission sub-model inherited from part of the construction processontology (EI-Gohary & EI-Diraby, 2010) and AIR_POLLUTION_Onto (Oprea, 2009). Figure 1 demonstrates thedeveloped ontology.
Table 1. Summary of data source of emission estimation variables
Meta Data Source
Direct Source
Emissions Variables Depending Objects
E(GHG)T
EFj
Fuel type
BIM_Resource
NGAF (2018)
ej
Vehicle features
BIM_VehicleModel
Vehicle InventorydProject location
BIM_Project
Information
vendors’ location

E(GHG)eq

E(NG)T

E(NG)eq

Em

w
ECj
EFj
fe
LF

Vehicle features
Fuel type
Fuel typeequipment features
equipment type
operation condition

T

Planning working
timeidle timeEF(NG)k fuel type, vehicle
type and age classfvehicle featuresej
vehicle featuresdProject location andfabricators’ locationw
vehicle weightECjfuel typeEFk
fuel typePequipment featuresTtask amountidle time
LF

machine type andoperation condition

QBOQ
μmaterial amountcompany's
capabilities
material type

em
Eelec

EW

P
η
h

equipment features
equipment features
task amount,schedule plan

eelec
Wi
Qi

Indirect Source

Vendors
Information

BIM_VehicleModel
Vehicle Inventory
BIM_Resource
NGAF (2018)
BIM_Resource
NGAF (2018)
BIM_EquipmentModel Equipment Inventory
BIM_EquipmentModel
Vehicle Inventory
BIM_Task
Company Norms orPrevious Studies
BIM_Schedule
Company Norms orPrevious Studies

BIM_Task
BIM_Resource
BIM_VehicleModel
BIM_VehicleModel
BIM_VehicleModel

Vehicle Inventory
Vehicle Inventory
Vehicle Inventory
BIM_Project
Information

BIM_VehicleModel
Vehicle Inventory
BIM_Resource
NGAF (2018)
BIM_Resource
NGAF (2018)
BIM_EquipmentModel Equipment Inventory
BIM_Quantity
BIM_Task
Company Norms orPrevious Studies
BIM_EquipmentModel
Vehicle Inventory
BIM_Task
Company Norms orPrevious Studies
BIM_Quantity
Company Norms orPrevious Studies
BIM_Material
BIM, National
inventories
BIM_EquipmentModel Equipment Inventory
BIM_EquipmentModel Equipment Inventory
BIM

BIM_Materialmaterial amount408

National inventories
Company Norms orPrevious Studies
BIMProceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

ifcOWL Ontology
Air Pollution Onto

Building Construction
Emission Ontology

Construction Process
Onto

Figure 1. Ontology Structure
5. BUILDING CONSTRUCTION EMISSION SEMANTIC MODEL DEVELOPMENT(1) Concepts of air emission
In this study, parent classes named AIR_EMISSION, EMISSION_SOURCE, EMISSION_FACTOR are
adapted from the AIR_POLLUTION_Onto; meanwhile sub-classes named TRANSPORTATION, SO2, PM and
CONSTRUCTION_ACTIVITY are inherited from AIR_POLLUTION_Onto. Other enriched sub-classes include
EMISSION_FACTOR, RESOURCE CONSUMPTION, etc..
(2) Concepts of the construction process
Parent concepts named ACTION, RESOURCE, ACTORS are based on IC-PRO-Onto (EI-Gohary & EIDiraby, 2010) whereas PRODUCT is excluded because it is already modeled in ifcOWL. Only those specificconcepts related to air emission are selected for sub-classes. Furthermore, some essential air emissions-related
concepts such as CONSTRUCTION EQUIPMENT, VEHICLE, VENDOR, and so on are not found when creating
the IC-PRO-Onto.

Figure 2. Taxonomy of Building Construction Emission Ontology
In the ontology, the hierarchy (a.k.a. taxonomy) of the classes represents the structure of the domain
knowledge shown in Figure 2. Each axiom binds two classes or individuals with specific relations. In Protégé,
Domain and Range are used to define the axiom of the subject domain in the Object properties dialogue, forinstance, the property CONSUME_MATERIAL links Subject TASK and Object MATERIAL.
6. CONCLUSION AND FUTURE WORKIn this study, the methodology of semantic modeling of building construction emission knowledge has

409Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

been illustrated. To formalize the knowledge of construction emission, a construction emission ontology isdeveloped. This ontology covers part of both construction and air emission domains according to the mechanismof emission generation from construction activities. Connecting this ontology with a building information model
can implement the simulation of air emissions from different construction activities and provide an understandingof building emission at the construction phase. It can also be utilized to help builders or contractors identify thewhereabouts of emissions and viable optimisation solutions.
Future work will be placed around: firstly, leveraging software, interview and pilot studies to validate theontology pertinent to syntax and semantic quality; secondly, extending the ontology to different emission types;
and finally, integrating the construction emission ontology with environmental impact, cost and other related
knowledge to facilitate the optimization of emission throughout the whole life cycle.
REFERENCES
Berners, T.L., Hendler, J., Lassila, O., (2001). The semantic web, Scientific American. 284 (5), 35–43.
buildingSMART. (2016). IFC4 ADD2 Release. Retrieved from buildingSMART website:
http://www.buildingsmart-tech.org/specifications/ifc-releases/ifc4-add2
Chau, C.K., Hui, W.K., NG, W.Y., Powell, G. (2012). Assessment of CO2 emissions reduction in high-rise
concrete office buildings using different material use options. Resources, Conservation and Recycling,61, 22-34.
El-Diraby, T. E. (2012). Domain ontology for construction knowledge. Journal of Construction Engineering andManagement, 139(7), 768-784.
El-Gohary, N. M. and T. E. El-Diraby (2010). "Domain ontology for processes in infrastructure andconstruction. Journal of Construction Engineering and Management, 136(7), 730-744.
Finkbeiner, M., Inaba, A., Tan, R., Christiansen, K. & Klüppel, H.-J. (2006). The new international standards forlife cycle assessment: ISO 14040 and ISO 14044. The international journal of life cycle assessment, 11,80-85.
Guggemos, A.A. & Horvath, A. (2005). Comparison of environmental effects of steel-and concrete-framedbuildings. Journal of Infrastructure Systems, 11, 93-101.
Giri, K. (2011). Role of ontology in semantic web. DESIDOC Journal of Library & Information Technology,
31(2),116-120.
Hendrckson, C. T., Horvath, A., Joshi, S., Klausner, M., Lave, L. B. & Mcmichael, F. C. (1997). Comparing twolife cycle assessment approaches: a process model vs. economic input-output-based assessment.
Electronics and the Environment, ISEE-1997., Proceedings of the 1997 IEEE International Symposium
on, 1997. IEEE, 176-181.
Hong, J., Shen, G.Q., Feng, Y., Lau, W.S.-T., Mao, C. (2015). Greenhouse gas emissions during the construction
phase of a building: a case study in China. Journal of Cleaner Production, 103, 249-259.
Jones, D.M., Bench-Capon, T.J.M., Visser, P.R.S. (1998). Methodologies for Ontology Development,
Proceedings of IT and Knowledge Conference of the 15th IFIP World Computer Congress, Budapest,
Hungary.
Jonsson, A., Bjorklund, T., Tillman, A.-M. (1998). LCA of concrete and steel building frames. The International
Journal of Life Cycle Assessment, 3, 216-224.
Li, X., Zhu, Y., Zhang, Z. (2010). An LCA-based environmental impact assessment model for construction
processes. Building and Environment. 45, 766-775.
Noy, N., McGuinness, D.L. (2000). Ontology Development 101: A guide to Creating your First Ontology,
Stanford Medical Informatics Technical Report No. SMI-2001-0880, 2000. URL:
www.smi.stanford.edu/projects/protege/publications/ontology_development/ontology101.pdf.
Oprea, M. M. (2009). AIR_POLLUTION_Onto: an ontology for air pollution analysis and control. IFIP
International Conference on Artificial Intelligence Applications and Innovations, Springer,135-143.
Pauwels, P., Zhang, S., & Lee, Y. C. (2017). Semantic web technologies in AEC industry: A literature overview.
Automation in Construction, 73, 145-165.
Sandanayake, M., Zhang, G., Setunge, S. (2016a). Environmental emissions at foundation construction stage ofbuildings e two case studies. Building and Environment, 95, 189-198.
Sandanayake, M. (2016b). Models and Toolkit to Estimate and Analyze the Emissions and Environmental
Impacts of Building Construction.
Tam, C.M., Z.M. Deng, and S.X. Zeng. (2002). Evaluation of construction methods and performance for highrise public housing construction in Hong Kong. Building and Environment, 37(10), 983-991
Wu, H., Yuan, Z., Zhang, L., Bi, J. (2012). Life cycle energy consumption and CO2 emission of an officebuilding in China. The International Journal of Life Cycle Assessment, 17, 105-118.
Yan, H., Shen Q.P., Fan C.H., Wang Y.W., Zhang L. (2010). Greenhouse gas emissions in building construction:
A case study of One Peking in Hong Kong. Building and Environment, 45(4), 949-955.

410

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

APPLICATION AND ANALYSIS OF SYSTEM ARCHITECTURE MODELFOR CONSTRUCTION PROJECT
Tatsuru Tomii1, Koji Makanae2, Raj Kapur Shah3
1) Software developer, Kokusai Kogyo Co., Ltd., Tokyo, Japan. Email: tatsuru_tomii@kk-grp.jp
2) Ph.D., Prof., School of Project Design, Miyagi University, Miyagi, Japan. Email: makanae@myu.ac.jp
3) Ph.D., Senior Lecturer, Faculty of Engineering and Technology, Liverpool John Moores University, Liverpool, United
Kingdom. Email: r.shah@ljmu.ac.uk

Abstract: In a smart society where physical space and cyber space are integrated. Similarly, the construction
project life cycle is being considered to be mutually related on both physical and cyber spaces. The project life
cycle becomes complicate when integrating the relationship between physical and cyber space. This paper presents
the design of a system architecture model in which the project life cycle is hierarchized, which is based on thescope of each process in the project. Additionally, an analysis of the information technology that supports eachprocess from the scope of this model is also presented. The paper concludes that the proposed model is a useful
tool to the construction project life cycle, and it helps to organize the process and system of the project components.
Keywords:

System architecture model, V-model, Construction project life cycle, Cyber-Physical systems

1. INTRODUCTION
In recent years, there are international movement to innovate the manufacturing industry throughadvanced digitization, such as Advanced Manufacturing Partnerships (AMP) in the United States, Industry 4.0 in
Germany, and Made in China 2025 in China. For example, in Industry 4.0, they are working to improve
productivity by integrating digital and the actual world (Bitkom et al., 2015). The concept of integrating the digitaland the actual world is based on Cyber-Physical Systems (CPS), which is an integration of cyber space and physicalspace. The CPS is a mechanism that communicate information on physical processes to a computer via a networkand feedback the computed result to objects (Lee, 2006).
In Japan, there are promoting the activities of Society 5.0 to realize "Super Smart Society" by CPS(Cabinet Office, Government of Japan, 2016). In the construction of social infrastructure, Building Information
Modeling (BIM) and Construction Information Modeling/Management (CIM) also promote the integration andmanagement of the digital and actual world. The construction project process is a complex life cycle in which
various industries are involved, and CIM's efforts are to reduce the waste such as communication loss, mistakes
and rework, and to improve productivity (MLIT, 2017a).
In the advanced digitized construction project life cycle by CPS, it can be considered as a complex system
in which elements of both physical space and cyber space are related. In this paper, a system architecture model ispresented for the life cycle of social infrastructure that is complicated by CPS. The model clarifies the scope ofthe entity's responsibility by defining the scope of each process in the project life cycle. In addition, an analysis ispresented for the information technology, which is used in each process based on the scopes of the project. The
next section describes about the proposed system architecture model.
2. SYSTEM ARCHITECTURE MODEL
2.1 Standards and Proposal Model
There are various standards in the project life cycle process of the system development. Standards of theInternational Organization for Standardization (ISO) include ISO/IEC/IEEE 15288 and ISO/IEC/IEEE 12207, andISO/IEC/IEEE 24748 provides guidelines for system life cycle management applying these processes (ISO, 2018).
In these standards, several models are suggested as how to execute each process in the project development.
The waterfall model, which is one of the project life cycle models, is a model that sequentially advances
a series of processes in one life cycle, as shown in Figure 1. If requirements are stable, they can be developed
efficiently. However, the upper the process is adopted with the higher and additional cost if there is need to change
a decision.
The incremental model is a method of dividing the requirements and developing them gradually. Initially,
the system is released with partial functions, but repeating the version, upgrade will be completed. If limited
functions can be used in business, the initial release can be developed in a short period because the initialdevelopment volume is small. However, requirements may change during the development, and it may benecessary to consider the changes in the initial plan.
The evolutionary model is a method of developing while repeating release like the incremental model, but
this is different in that all requirements are unknown at beginning. In this model, development is performed frompartial requirements definition but the following requirements are defined from the result, and the version is411

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019upgraded. Therefore, it is necessary to advance development while obtaining feedback from the users frequently.
Planning
Requirements Definition
Realization and Decomposition

Basic Design
Detailed Design
Implementation
Unit Testing
Integrated Testing

Verification and Integration

System Testing
Operation

Figure 1. Example of waterfall model for software development
Planning

Operation
Verification

Requirements Definition
Realization andDecomposition

System Testing

Basic Design

Integrated Testing

Detailed Design

Verification andIntegration

Unit Testing
Implementation

Figure 2. Example of V-model for software development
In the case of a sequential life cycle such as waterfall model, the processes comprise two phases, which
are realizing and dividing phases from the planning to implementation including the evaluating and integrating
after the implementation. The V model is a V-shaped representation of these phases (Boehm, 1979), as shown in
Figure 2. The V model is used for the system development of projects such as V-Modell XT in Germany (Biffl etal., 2006), and this is used for ITS (Intelligent Transportation Systems), development guidelines for spacecraft
software (USDOT & Caltrans, 2009; JAXA, 2013). As shown in Figure 2, the V model correspondence can bemade between left and right processes, such as the requirements definitions and system test, basic design andintegration test, and so on. This helps the verification of the corresponding to the contents performed in eachprocess of realization, and the evaluation process can be planned at the stage of realization. By deciding theverification content at an early stage, it is possible to detect deficiencies in each process at an early stage and to
improve the quality and productivity.
In addition, since the left side of V is the phase of system dividing and the right side is the phase of system
integrating, it can be considered that the scope of upper process is wider and scope of lower process is contracted.
In the example shown in Figure 3, the top to bottom target range is layered into systems, subsystems, andcomponents. The processes in each layer have a Plan-Do-Check relationship that planning the future on left process,whereas the execution to the lower layer and the evaluation of the results on the right process. However, each layer
can be separated and made independent.
The system elements in accordance with the scope of hierarchy can be considered the pyramid structuresuch as Figure 4(a). Figure 4(b) shows the combination of the system structure and the V-model. The processes
and system elements of each layer in the V-model are associated. For example, if a layer consists of five elements,
it means that 5 life cycles of each element will be executed in parallel. This model is similar to the Dual Vee model(Forsberg et al., 2005), but by making the relationship between the scope and elements corresponding to the412Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

process more obvious, the person of each process can make it easier to understand the scope of the responsibility.
Planning

SystemOperation

Requirements Definition

System Testing
Subsystem

Basic Design

Integrated Testing

Component

Detailed Design

Unit Testing

Implementation
Plan

Check

Do

Figure 3. Processes hierarchy and Plan-Do-Check relationship

ScopesSystemLife cycle
Elements

ScopesSubsystem

Component
Elements
4(b) Proposal Model4(a) System Element Structure

Figure 4. System architecture model
2.2 Comparison with Other Models
The system architecture can be represented by other models, as shown in Figure 5. In this paper, the systemarchitecture is called as the linear model, the circular model and the cubic model because of its shape.
Element

Elements

Elements
Verification

Life cycle

5(a) Linear Model5(b) Circular Model

Element Types

ScopesLife cycle

ScopesLife cycle
5(c) Cubic Model

Figure 5. Other models
The linear model is represented the life cycle with straight line, as shown in Figure 5(a). In this model, thenested diagram instead of the tree can represent the system structure. This linear life cycle can be regarded as atimeline, and it is easy to understand which element to develop in which process. However, it may be difficult to413

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

notice the Plan-Do-Check relationship of processes because it is separated between planning and verification
processes. This model is easy to combine with production management tools such as Gantt charts, but it would bebetter to limit the target scope.
The circular model is a circle or spiral diagram of the life cycle, as shown in Figure 5(b). The system
structure diagram can be represented in the same tree structure as the V-model. This model is similar to the Vmodel except that it returns from the last process to the first process. Therefore, this model matches iterative
development such as the incremental model or the evolutionary model well. However, managing parallel tasks ina time series may be more difficult than the linear model.
The cubic model is the diagram that represented three viewpoints by three axes, as shown in Figure 5(c).
One cell separated by each viewpoint is a system element, and the arrangement of each cell can represent thesystem structure. This model is used in the Smart Grid Reference Architecture Model (SG-CG, 2012), and theReference Architecture Model Industry 4.0 (Bitkom et al., 2015). This model makes it easy to understand thecorrespondence between elements and each viewpoint, and it can represent the detailed structure. However, inorder to confirm the internal structure, it is necessary to represent it in the sectional views or use the 3D userinterface.
In this paper, straight-line time axes can represent the proposed architecture model. It is easy to classify
the processes by scope, which is planar shape and simple. The model is presented and applied to the social
infrastructure and analyzed its system.
3. SYSTEM ARCHITECTURE MODEL OF CONSTRUCTION PROJECT
3.1 V-model of the Construction Project Life Cycle
The production processes of the social infrastructure are transferred to the maintenance phase afterplanning, design, construction, completion and inspection (Sacks et al., 2018). Since these are usually used in a
project, a project plan is formulated before the production processes. In addition, the project is based on the policies
of organizations. In the case of the government, the policies are such as master plans and laws. Projects and policies
implemented based on the appropriate PDCA cycle will evaluate those plans and evaluate the results. Figure 6
shows these processes in a V-model. In this figure, the scope of each process is expressed in the social layer ofpolicy and policy evaluation, the project layer of project planning and project evaluation, and the structure layerof design, construction, completion and inspection.
Operation, MaintenanceSociety

Project

Policy Evaluation

PolicyPlanning

Project Evaluation

DesignIn-service

Completion Inspection

Structure
Construction

Figure 6. V-model of construction project life cycle
3.2 System Structure of the Road Traffic
Figure 7 shows the elements of a road traffic system in Japan based on the three-layered structure. It isstipulated by law that road administrators such as countries and prefectures manage roads and their database (RoadAct Article 18 and Article 28). Therefore, hardware elements of road traffic in the society layer is the road networkand the database, and they are dealt with in the policy and policy evaluation processes. The road network is madeup of many courses, and the course construction is implemented as a project. At the project layer, one course isadopted from several proposals with abstract plans, and detailed plan is carried out. The course plan is evaluatedafter the project is completed. The course is made up of various structures such as superstructure, substructure,
and accessories. These elements production are dealt with in the structure layer processes.
Traffic which is a service provided by the road is managed by collecting, analyzing, and controlling. The
Public Safety Commission (police) separately from the road administrators manages traffic, and overallmanagement such of the road network is dealt in the traffic control center. For individual courses, methods oftraffic information collection and traffic control, such as installation of sensors and traffic signals, are considered.
Road signs and road information facilities that constitute road accessories are managed by the roadadministrator, but elements related to traffic regulations such as traffic lights and the traffic administrator manages

414

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019traffic signs even if they are attached to the road. Similarly, elements such as sensors for traffic informationcollection, which are set for each purpose, are managed by each organization. However, as road data is needed for
traffic analysis and control, elements should be shared and usable by each organization. Such elements can befacilitate to share in the system by the elements integration and centrally management in the upper layer. For
example, Japan Digital Road Map Association (DRM) maintains national road network data (DRM, 2013), andJapan Road Traffic Information Center (JARTIC) collects and delivers traffic information (Ozaki, 2003).
ScopesSystem Element StructureLife Cycle

Road Traffic
Society

Road Network

Traffic

Regional Road DB
CourseProject

Collector

National
Road DB

Analyzer

Traffic
Information

PolicyEvaluation

Planning

Project
Evaluation

DesignCompletion
Inspection

Controller
Controller

Collector

PolicyStructure
Superstructure

Substructure
Sign

Accessory

SensorTraffic Light

Sign

Road information facility

Construction

Figure 7. System architecture model of the road traffic
3.3 Information Technology to Support Processes
This hierarchical structure is different scope of the concern for each layer. It is necessary to select asuitable information technology used in each process. Figure 8 shows an example of the information technology
used in each process.
Range of
Information

Life cycle

Information Technology

Society

Construction

CG / VR

IndividuallyConstruction Robot

Structure

CAD

Completion
Inspection

Survey / Sensing

DesignSpecified

GIS

Project
Evaluation

Database

Planning

WidelyNumerical Simulation

PolicyEvaluation

Statistics

Project

PolicyMathematics for Planning

Operation

Figure 8. Information technologies to support processes
In the example shown in Figure 7, it is necessary to plan and evaluate from a wide point of view so thatthe target element of the social layer is a wide area road network. It is necessary to use comprehensive methods
such as statistics, mathematics for planning, GIS (Geographic Information System), database, and numerical415

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

simulation in the process. In the project layer, more detailed planning and evaluation are performed to a specificscope. In this layer, technologies in the social layer is used to a specific area, but the methods such as a sensing
for terrain, weather, and other environmental information is used with more detailed information. In the structure
layer, individual structures are targeted. It is used technologies such as CAD (Computer-Aided Design),
construction robots, and confirming the completion by Computer Graphics (CG) and Virtual Reality (VR).
These information technologies may use common information. For example, geospatial information about
the actual space is used from spatial analysis for wider area to the location of individual structure. It is necessary
to use the same data by any process in this case, but exactly the same data is not always optimal considering thedensity of the information. Three-dimensional models may be used in the structure layer, but the surface data of
the space may be used in the society layer. Therefore, the data should be managed so as not to lose necessary
information throughout the life cycle while adding information or converting the format in order to share
information between processes.
3.4 Extension and Division of System Architecture Model
The construction process is organized in a hierarchy of society, projects and structures, but it is possible
to add more or more layers. For example, just as the United Nations' sustainable development goals are classified
in views of the global environment, society and economy, this model can be expressed as a configuration basedon environmental policy. Conversely, it is possible to append the more detailed layer of the structure member
below the structure layer, as shown in Figure 8.
In addition, the layers can be divided into specified layers because the components have independence.
For example, it is possible to model focusing on each element by dividing three-layer model into individual
projects, as shown in Figure 9. However, it is necessary to be strongly aware of the policies and evaluation there
because these projects are based on the upper processes.
Environment

Environmental policy and evaluation

Society
Infrastructure
based onenvironmental
policyProject
StructureMemberMember production

Figure 9. Example of an extension of the system architecture model
Road Traffic
Society
Project
Structure

Project
Structure
Infrastructure

Traffic

Figure 10. Division of the system architecture model
5. CONCLUSIONSThe paper presents a system architecture model with the integration and management of each element ofphysical space and cyber space. The model aims to realize a highly digitized construction project life cycle in thesmart society. A model was designed by combining the V-model and the system structure for the architecture416

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019model. In addition, a definition is presented for a hierarchical structure based on the target scope of each process
and applied it to the project life cycle and system structure.
Authors represent a system by a V-model that consists of the processes for the policy, project planning,
design, construction, completion inspection, project evaluation, and policy evaluation in order to apply this modelto the project life cycle. In addition, the process is classified by the hierarchical structure of society, business andstructure from the scope of each process.
Finally, an analysis is presented in the paper for the components of a road project and the information
technology used in each process based on these layers. The road components are largely divided into infrastructure
and traffic, but there are elements that should be shared with each element. These can be made them common
assets by separating, integrating. There are common data that is used at different levels in each layer such as
geospatial information. Such data needs management that can be shared without losing information while changing
its format throughout the project life cycle. The paper concludes that a system architecture model is useful tool for
integrating the relationship of a construction project between physical and cyber space.
ACKNOWLEDGMENTS
JSPS KAKENHI Grant Number 19K04641 and the Daiwa Foundation Small Grants from The Daiwa
Anglo-Japanese Foundation supported this work.
REFERENCES
Biffl, S., Winkler, D., Höhn, R., and Wetzel. H. (2006). Software process improvement in Europe: potential of the
new V-modell XT and research issues, Software Process: Improvement and Practice, 11(3), 229-238.
Bitkom, VDMA, and ZVEI. (2015). Implementation Strategy Industrie 4.0: Report on the results of the Industrie
4.0
Platform.
Retrieved
from
ZVEI
website:https://www.zvei.org/fileadmin/user_upload/Presse_und_Medien/Publikationen/2016/januar/Implement
ation_Strategy_Industrie_4.0_-_Report_on_the_results_of_Industrie_4.0_Platform/ImplementationStrategy-Industrie-40-ENG.pdf
Boehm, B. W. (1979). Guidelines for Verifying and Validating Software Requirements and Design Specifications,
Proceedings of Euro IFIP 79, London, England, pp.711-719.
Cabinet Office, Government of Japan. (2016). The 5th Science and Technology Basic Plan. Retrieved from Cabinet
Office, Government of Japan website: https://www8.cao.go.jp/cstp/english/basic/5thbasicplan.pdf
DRM.
(2013).
Database
CreationandUpdate,RetrievedfromDRM
website:http://www.drm.jp/english/drm/database/structure.html
Griggs, D. (2013). Sustainable development goals for people and planet, Nature, 495(7411), 305-307.
Forsberg, K., Mooz, H., and Cotterman, H. (2005). Visualizing Project Management. John Wiley & Sons.
ISO. (2018). ISO/IEC/IEEE 24748-1 Systems and software engineering - Life cycle management - Part 1: Guidelines for life cycle management.
JAXA. (2013). Independent Verification and Validation (IV & V) Guidebook (The bible of IV & V). Retrieved
from JAXA website: https://repository.exst.jaxa.jp/dspace/handle/a-is/19895
Lee, E. A. (2006). Cyber-Physical Systems - Are Computing Foundations Adequate? Position Paper for NSF
Workshop on Cyber-Physical Systems: Research Motivation, Techniques and Roadmap, 2, 1-9.
MLIT. (2017a). Draft guidelines for CIM introduction: Part 1. Retrieved from MLIT website:
http://www.mlit.go.jp/tec/it/pdf/guide01.pdf
MLIT. (2017b), White Paper on Land, Infrastructure, Transport and Tourism in Japan, 2017. Retrieved from
MLIT website: http://www.mlit.go.jp/en/statistics/white-paper-mlit-2017.html
Ozaki, H. (2003). Evolution of ITS in Japan in the Last and Next Decades, Telektronikk, 99(1), 36-40. Retrieved
from Telenor website: https://www.telenor.com/wp-content/uploads/2012/05/T03_1.pdf#page=38
Sacks, R., Eastman, C., Lee, G., and Teicholz, P. (2018). BIM Handbook: A Guide to Building Information Modeling for Owners, Designers, Engineers, Contractors, and Facility Managers (3rd ed.). John Wiley & Sons.
SG-CG. (2012). Smart Grid Reference Architecture, Retrieved from CEN & CENELEC, website:
https://www.cencenelec.eu/standards/Sectorsold/SustainableEnergy/SmartGrids/Pages/default.aspx
Stockholm Resilience Centre. (2016). How food connects all the SDGs, Retrieved from Stockholm ResilienceCentre website: https://www.stockholmresilience.org/research/research-news/2016-06-14-how-foodconnects-all-the-sdgs.html
USDOT and Caltrans. (2009). Systems Engineering Guidebook for Intelligent Transportation Systems Version 3.0.
Retrieved from USDOT website: https://www.fhwa.dot.gov/cadiv/segb/files/segbversion3.pdf

417

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

SMART CONSTRUCTION OBJECTS (SCOS): A NEW THEORY OF SMARTCONSTRUCTION IS BORN
Weisheng Lu1, Yuhan Niu2, and Chimay Anumba3
1) Associate Professor, Department of Real Estate and Construction, Faculty of Architecture, The University of Hong Kong,
Pokfulam, Hong Kong
2) Construction Industry Council (CIC), Hong Kong
3) Dean and Professor College of Design, Construction and Planning, University of Florida, USA

Abstract: Despite of the prolific development of smart systems in the construction industry, the theorizing workof ‘smart construction’ is rather stagnant. The fundamental concepts, definitions, and models of smart construction
are yet to be fully explored. It is against this backdrop that this study seeks to advocate smart construction objects(SCOs) as an innovative concept and the basic elements to define, understand, and achieve a new theory of smart
construction. It does so by adopting a mixed-method strategy at the kernel. It establishes the conceptual anddeployment elements of SCOs and tests them in two case studies. This study reveals that the concept of SCOs cansteer the field of smart construction towards a new theory by (a) enhancing the theoretical lucidity on smart
construction, and (b) providing a generalizable framework for realizing it. One elegance of SCOs lies in that they
can be adopted and implemented without radically changing the prevailing construction practice and process.
Advancing smart construction through this direction can be expected to go more promisingly than existing
directions.
Keywords: Smart construction, smart construction objects, theory.
1. INTRODUCTION
Smart construction has increasingly been advocated in recent years, with “smart” becoming a global
buzzword. The enthusiasm for smart has infiltrated almost every aspect of life from the device level (e.g., smart
phone and smart watch), the industry level (e.g., smart health and smart transportation), to the city or country level(e.g., the smart city initiatives in New York, Tokyo, Seoul, Glasgow, Ontario, and Singapore). The constructionindustry is no exception by strenuously exploring the concept of smart to solve its many chronic problems such as
delayed delivery, escalating cost, unsatisfactory quality, and stagnant productivity.
Regardless of the growing interest, studies on smart construction, however, are stagnating at a primitive
stage. “Smart construction” is conveniently used to refer to anything that is different from “traditional”
construction. For example, there is a “smart construction site” where workers, materials, and machinery can betracked and monitored (Hammad et al., 2012); ‘smart building construction’ as an indispensable element of thesmart city (Angelidou, 2015); or “smart construction lift car toolkit” that allows automated recognition of thelogistic items in construction (Cho et al., 2011). Likewise, with the resurgence of interest in artificial intelligence(AI) and robotics for construction, several AI - or robotics-based systems have been developed under thenomenclature of “smart construction”. These include the sensing system to monitor workers’ exposure to
vibrations (Kortuem et al., 2007), the contour crafting system for automatic fabrication of building structures onsite (Khoshnevis, 2004), or the mechanical arms to help worker handle heavy materials (Lee et al., 2006).
Despite the research efforts on smart construction by employing ideas from AI, robotics, and analogous
concepts, there is still widespread frustrations in the industry in respect of smart construction. In contrast to theadvanced development of smart systems in manufacturing, automobile, civil aviation, and logistics and supply
chain management, the fundamental concepts, definitions, and models of smart construction are yet to besystematically explored. Successful cases of smart construction have emerged in a piecemeal fashion, hence having
little generalizability. In addition, smart systems introduced from other industries have been disruptive to existing
construction practice, resulting in practitioner reluctance to harness their potential. There is a need to develop a
new theory to guide smart construction development, and it is against this backdrop that this study was initiated.
Unlike most studies of this kind tending to develop a smart construction technology or system, this study takes thechallenge to look at smart construction as a more general issue from a theoretical perspective.
Building on previous studies of smart construction objects (SCOs), this research argues that thedevelopment of SCOs is leading towards a new theory of smart construction. It demonstrates that SCO
development offers a perspective from which to (a) systematically define, understand, and achieve smart
construction; (b) provides a new perspective to solve problems beyond the scope of existing theories; and (c)address limitations in existing studies on smart construction, including lack of theoretical lucidity, and disruptive
and piecemeal application with limited generalizability.
2. THE NEED FOR A NEW THOERY
According to Oxford dictionaries, a theory is a “supposition or a system of ideas intended to explain
something, especially one based on general principles independent of the thing to be explained”. There are two418Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019broad types of theory: explanatory theory and change theory. An explanatory theory is used as a plausible
general principle or body of principles offered to explain a phenomenon. A theory is a model capable ofpredicting future occurrences or observations, being tested through experiment or otherwise verified throughempirical observation (De Benetti, 2009). However, even the best explanations may not be enough bythemselves to fully guide change, e.g., in smart construction development. Change models are desired in thiscase. Theories are by their nature abstract and not content- or topic-specific but they have empirical relevance.
There are non-negligible limitations in current studies on smart construction. First, it is lack of theoretical
lucidity, meaning that the fundamental concepts, definitions, and models of smart construction have yet to be fully
explored. Amid proliferating studies on smart construction, its definitions are still nebulous. Worse yet, researchon smart construction is partitioned into isolated sub-disciplines, mostly too focused on technological tools suchas sensors, networking, or automatic control. Smart construction is stuck in theoretical muddle and murk. Lucidtheories on smart construction are necessary for its future research and development to proceed on a more solid
footing.
The second limitation in existing smart construction studies is their piecemeal application with limited
generalizability. Rooted in different theoretical bases from disciplines including computer science, information
communication technology, and the manufacturing industry, most existing smart construction studies emphasizea cross-sectoral learning approach for achieving smart construction. Even though there are successful cases ofsmart construction application such as the application of expert systems in construction (Andersen and Gaarslev,
1996; McGartland and Hendrickson, 1985), the complex nature of projects, unique site conditions, non-repeatable
construction processes, and heterogeneity and fragmentation of the construction industry all make it difficult to
directly transplant well-developed smart systems from other industries.
The third limitation is rooted in the disruptive nature of current approaches for achieving smart
construction. Notably, most smart systems introduced from other industries are intrusive to existing construction
working practices. With a view to increasing the adaptability of smart systems to the construction industry, somestudies advocate alteration of the traditional working environment or working procedures to accommodate newtechnologies, rather than integrating new technologies into the current working practices. Examples include thedevelopment of mobile platforms to cater for the operation of robots (Zied, 2007) and the strategy of robot-oriented
design (ROD) (Bock and Linner, 2015). Consequently, the intrusive smart systems create reluctance on the partof workers and managers to harness their full potential while the industry is criticised for being ‘notoriously slow’
to embrace change (Liu et al., 2018; Woudhuysen and Abley, 2004). To date, not all smart systems have beentreated not as a natural and endogenous ally, but instead some of them as a potentially disruptive adversary toconstruction.
These three limitations create a desire for a new theory of smart construction. Within the theory, ideally,
conceptual elements of smart construction can be clearly defined to provide theoretical lucidity. Also required issystematic development of a generic system framework to enable the practical deployment of smart construction
under the dynamic, sophisticated and diverse conditions of construction. More importantly, instead of continuously
mandating construction personnel to change and adapt to new technologies, a non-disruptive approach that caninnovatively accommodate the complex nature of construction is desired. Unlike similar studies which havedeveloped a particular smart construction technology or a system, this study takes on the challenge of looking at
the theoretical development of smart construction, seeking the fundamentally theoretical elements to define smart
construction and the generalizable solutions in advancing the field.
3. RESEARCH DESIGN AND METHODSThis study adopts abductive reasoning logic for its research design. Based on a logic chain of exploration,modelling and production, experimentation, rectification and evaluation, abduction allows researchers to move
between the creation of explanations and acquisition of knowledge from empirical phenomena (Downward andMearman, 2006). This makes it particularly appropriate for a study in construction, which is primarily a practicebased research domain encompassing aspects of both natural and social science, and collaboration between
academia and industry can lead to knowledge that is both academically insightful and practically actionable. The
research design for this study and respective methods at each stage are elaborated as follows:(1)Exploration stage: Literature review and industry engagement are conducted to understand thebackground, related concepts, and various models in construction. Based on the literature review and the reflectionof the authors’ industry engagement, the need for a smart construction theory and current limitations are identified,with initial findings summarized in Sections 2 and 3 of this paper.
(2)Modelling stage: Based on the understanding gained from the literature-based discovery, theconceptual and deployment elements of SCOs are systematically developed. This step involves desktop studies
and discussions with practitioners and particularly with visionary scholars. In order to understand and define smart
construction, the conceptual elements including the definition and core properties of SCOs are theoretically
proposed, while taxonomic relationship with other concepts is also articulated. Besides, in order to apply SCOs in

419

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

the, considerable efforts are paid to develop the deployment framework of the SCOs-enabled management systemand technical solutions.
(3)Experimentation and rectification stage: In the experimentation and rectification stage, case study
method is used to empirically test SCOs and the SCO-enabled smart management framework in context-driven
and problem-focused project practice. This method is valuable in situations where existing knowledge is limited
and shallow for more and deeper insights (Harris and Ogbonna, 2002). Prototypes of SCOs are developed andrectified to cater for practical needs and variance of contexts. By conducting two in-depth case studies, theunderstanding on the concept and framework of SCOs are substantiated in different application scenarios.
(4)Evaluation stage: Findings and reflections are provided with the theorizing and discussions in theevaluations stage. During the development of the theoretical elements of SCOs and the implementation of two
progressive case studies, a continuous dialogue takes place between the author’s pre-understandings and theempirical data. New insights are generated as an evolution of the authors’ understanding, as summarised in thediscussion section.
During the entire research study process, multiple data collection and analysis methods are applied,
including interviews, direct observations and participatory-observations, documentation analysis, and theoreticaldebates. Certainly, it is not a linear process. Rather, the mixed methods approach unfolds in a reiterative fashion.
Triangulations of literature-based discovery, theoretical debates, and co-production are repeated throughout the
study, and are blended in narratives ensure a coherent argument for ease of reading.
4. SMART CONSTRUCTION OBJECTS (SCOS) AS A NEW THEORY
4.1 Conceptual elements
The concept of SCOs is developed as a basic element to define, understand, and achieve smart
construction. Inspired by the concept of the smart object (SO) (Kortuem et al., 2010; López et al., 2012), SCOs is
proposed as a step towards ubiquitous computing and smartness in the construction context. SCOs are defined as
construction resources made ‘smart’ by augmenting them with smart properties (Niu et al., 2015). These resources
could be machinery, tools, devices, materials, components, and even temporary or permanent structures. To
explain the smartness SCOs could confer, three core properties of SCOs are proposed in Figure 2: awareness,
communicativeness, and autonomy, denoting the sensing ability, data sharing ability, and autonomous actiontaking ability of SCOs (Niu et al., 2015). Each of the three core properties is subdivided into several types, while
they may function in cooperation depending on needs and requirements in different application scenarios.
As the basic elements of smart construction, SCOs offer a way to define and understand smart
construction. Understanding of SCOs and smart construction are deepened when their taxonomic relationship withcyber-physical systems (CPSs) and the Internet of things (IoT) are elucidated (see Figure 1). Differences andsimilarities between the three concepts are articulated by Niu et al. (2018). For example, although the three
concepts share similar underlying technology tools, each operates at a different level (SCOs at the component level,
a CPS the system level, and the IoT the infrastructure level) (Niu et al., 2018). A synergetic deployment framework
to integrate the three concepts has been proposed with the objective of harvesting the synergy between them when
adopting smart construction.

Figure 1. The development of conceptual elements of SCOs
4.2 Deployment elements
While flexible combinations of their three core properties (awareness, communicativeness and autonomy)
enable SCOs to provide individual smart functions, the true power of SCOs lies in an integrated, responsive smart

420

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

construction system in which they are linked. A generic framework for this SCO-enabled smart management
system is developed for practical deployment (see Figure 2). By providing a multi-layered structure with theconnecting relationships in between, the system framework for the SCO-enabled smart management system clearly
illustrates the process of turning traditional construction objects into smart and customizable SCOs, the functions
units to be included in the smart management platform (SMP), and the typical demand-oriented applications of
SCOs. It demonstrates how SCOs could interact with people or each other to support construction management byenabling a more connected world of construction.
Whilst this study proposes SCOs as a paradigmatic development of smart construction rather than atechnical solution per se, it is important to introduce the technical foundation of SCOs to demonstrate theirfeasibility for smart construction. Awareness, communicativeness, and autonomy of SCOs can be achieved byaugmenting construction objects with various modules into construction objects, including computing,communication, sensing, and locationing modules (Liu et al., 2015). To encapsulate these modules in an integrated
manner, a standalone, programmable, extendable integrated electronic chip, named i-Core, is developed in thisstudy as one of the technical solution (see Figure 3) (Lu et al., 2016). Able to be implanted into machinery, devices,and materials, and similar to a computer central processing unit (CPU), the i-Core turns dumb construction
components and plants into SCOs and makes smart construction possible. Implementation of the three core SCO
properties relies on integration of various computing, sensing, and communicating modules into the i-Core. To
meet changing needs of construction sites and achieve different functions, these modules are extensible and canbe selected and customized case by case.

Figure 2. The development of deployment elements of SCOs
5. THE TESTING AND VALIDATION OF SCOS
5.1 Case A: SCOs for logistics and supply chain management (LSCM) in construction
In Case A, the first attempt to put the SCO concept and conceptual framework proposed in this study into
practice, prefabricated beams were made into SCOs to support LSCM in a public housing prefabrication projectin Tuen Mun, Hong Kong. The beams were enabled to sense real-time location, push the information to a cloudbased smart management platform (SMP), and update their LSCM status. With these functions, a real-time bidirectional information flow between SCOs and the SMP was achieved, along with concurrent information andmaterial flow during the LSCM process (Niu et al., 2016).
The implementation of Case A validates the practicality and customizability of SCOs in achieving smart
construction, demonstrating flexibility in property combinations, a customizable framework, well-performing
prototypes of the i-Core, SCOs, and the SMP (see Figure 4), and the non-disruptive nature of SCOs. The
collaborating practitioners, having used both radio frequency identification (RFID) tags and SCOs for LSCM,
expressed a preference for SCOs because they require less manual work and fewer changes in working practice.
These industry partners’ interest in collaborating further lends support to the practicability of SCOs.

421

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 3. Testing and validation of SCOs in Case A
5.2 Case B: SCOs for construction occupational health and safety (OHS) management
With the unfailing support from the industry partners, Case B was conducted to enhance the validity andtransferability of SCOs. While Case A involved the relatively new construction practice of prefabrication, Case B
targeted conventional onsite tower crane operations and associated safety problems. By making a tower crane intoa SCO and field testing it, Case B demonstrates that an SCO-enabled occupational health and safety (OHS)
management system can support worksite monitoring, hazard detection, alerting, and data visualization to identify
and respond autonomously to dangerous situations. Having validated the autonomy of SCOs in a controlled lab
test, Case B demonstrates that SCOs could help control dangerous situations more quickly and with greater
accuracy than human reactions by taking in-time, autonomous actions such as halting a machine (Niu et al., 2019).
The customizability of SCOs and the SCO-enabled smart management system was further substantiatedin Case B when applied to suit a completely different application scenario. The different versions of i-Coredeveloped for the tower crane and customized SMP also show the operability of SCOs for achieving smart
construction (Figure 4). In addition, when looking the non-disruptive nature of SCOs in Case B, the AI-based
solution provided by SCOs can be understood as an extra layer of protection. It does not require the crane operator
or the safety manager to relinquish their existing OHS strategies such as wearing personal protective equipment(PPE) or conducting safety training, while reinforcing the OHS management from an additional perspective.
To sum up, the Cases A and B together deepen the understandings of SCOs towards a new theory of smart
construction from an empirical perspective. By addressing each of the major limitations in existing studies on
smart construction, including “lacking of theoretical lucidity”, “piecemeal applications and limited
generalizability”, and “disruptive in nature”, the in-depth explorations of SCOs in the two cases vividly
demonstrate their practicality, customizability and non-disruptive nature in filling theses gaps.
6. FINDINGS AND DISCUSSIONS
6.1 Enhancing the theoretical lucidity of smart construction
The development of SCOs is definitely not the first study on smart construction. However, it makes asignificant endeavour to enhance the theoretical lucidity. By systematically proposing the concepts, core properties,422

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019and system framework in smart construction, the development of SCOs has established the fundamental elements
to define and understand smart construction. In this study, considering each SCO as a basic element of smart
construction, the SCO-enabled smart management system and i-Core are developed to achieve smart constructionby harnessing the synergetic power of SCOs. These conceptual and deployment elements serve as essential
components in deepening understanding of smart construction.

Figure 4. Testing and validating of SCOs in Case B
The theoretical lucidity of smart construction is further enhanced when understanding of the proposed
concepts is substantiated in the context of industry practice. Applying the technical solution i-Core, an electricalcircuit integrated with sensing, communication, and control modules, existing construction resources such as smart
prefabricated beams and a smart tower crane (Cases A and B), are made into SCOs. These SCOs serve traditional
functions while also behaving in a smarter way. The smart prefabricated beams, for example, still serve as the
structural elements to take load in buildings, during the LSCM process, while they can sense and report their realtime locations to the cloud-based platform. When the conceptual elements of smart construction are understood
not only literally but also from a practical setting, the move towards smart construction is improved.
The enhancement on the theoretical lucidity made by SCOs can also be perceived from the perspectivethat the deployment of SCOs are presenting management implications for the smart construction. By comparing
the SCO-enabled LSCM with the traditional LSCM, it was found that SCOs can facilitate informed decisionmaking by providing real-time bi-directional information flow between SCOs and the SMP. Also, when comparedwith RFID-enabled LSCM, SCO-enabled LSCM is acknowledged by industry practitioners as a less disruptive
approach since it requires less manual effort. Likewise, deployment of SCOs for OHS management reveals theability of SCOs to support AI-based solutions and autonomous action-taking. Comparing the SCO-enabled smart
construction model with the human-centric decision science model, SCOs could prevent dangerous situations fromdeveloping into fatal accidents by taking active and prompt actions in conditions that would overload humanthinking and reacting abilities.

423

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

6.2 The generalizability and customizability of SCOs
The three core properties of SCOs, namely, awareness, communicativeness, and autonomy, are presented
in a tri-axial diagram. Each axis carries one dimension of smart property that is divided into different types andlevels. Different combinations of the core smart properties enable customization of smart solutions to almost
unlimited construction scenarios as widely seen in construction. The tri-axial diagram is resilient and extendable,
which is open to enrichment such as by adding a supplementary dimension of a particular type of smartness oradditional smart properties. The resilience of the theoretical constructs can also be perceived in the framework of
the generic SCO-enabled smart management system, which is not only generalizable but also customizable. The
three layers comprising the SCOs, the SMP, and the application layer, were applied in both Cases A and B
scenarios. The successful implementation of SCOs in both scenarios, together with practitioners’ openness to
SCOs as a means of achieving smart construction, reveal the practicability, generalizability, and customizabilityof SCOs towards the theory of smart construction. More importantly, the generic system framework also provides
a clear direction and sufficient detail for the research community to replicate and enrich the work, serving as aclassic element in supporting paradigmatic development.
From another perspective, the technical solution adopted in existing studies to bring SCOs to life, i-Core,can also be considered as a generalizable and customizable solution to achieve smart construction. Various sensors,
location tracking modules, communicating modules, and control units can be integrated into the i-Core flexibly.
Showing diverse traits and functions in different SCO application scenarios, the i-Core has the versatility to turndifferent construction resources into SCOs.
6.3 The non-disruptive nature of SCOs
Central to the new theory of smart construction through SCOs is the ideological shift from radical change
to a non-disruptive approach, in achieving smart construction. The non-disruptive nature of SCOs is rooted in therationale of ubiquitous computing from which SCOs are developed. Ubiquitous computing initiates the rethinking
on a radical ideological departure from the tradition of putting intelligent machines out for use and having people
adapt to them (Weiser and Brown, 1997). Notably, it advocates embedding of smart computing abilities in theenvironment, anywhere and everywhere (Weiser, 1999). For initiating this new perspective towards smart
computing, ubiquitous computing is gaining acceptance as a new theory in the smart computing researchcommunity (Abowd and Mynatt, 2000; Greenfield, 2010). Likewise, developing SCOs is a step towards ubiquitous
computing and smartness in the construction context. Instead of continuously inventing and bringing in new
intelligent machines for construction personnel to learn, SCOs offer a new way of achieving smart construction
by making existing construction resources and objects smart.
The development of SCOs should not be perceived as a technical solution per se, in contrast to mostexisting studies on smart construction applications. Instead, the aim is to promote an ideological shift towards the
new theory of smart construction. In construction, the prevailing stagnant progress in innovation intake andtechnology acceptance suggest that the industry does not need or cannot afford radical change. The value residingin the non-disruptive nature of SCOs is clear from the positive response of construction practitioners in the casestudies, in contrast with their resistance to other, disruptive smart systems.
Nevertheless, by arguing against radical change in established construction practice, this study does notassert that traditional construction practices should remain unchanged. It is a question of priority. The construction
practice should be changed with the regard to improve performance and productivity, but not to make room for
introducing new technologies. The motivations for the changes should always be enhancing productivity, quality,
safety performance, and reducing cost. Likewise, the introduction of new technologies, innovations, should serve
the same purpose of facilitating such improvement.
7. CONCLUSIONS
The global construction industry has long been plagued with problems such as delayed delivery,
escalating cost, and unsatisfactory quality. Smart construction has attracted considerable attentions as a new
direction to solve these problems while the understanding of smart construction is literally nebulous. By
developing, prototyping, and testing the concept of smart construction objects (SCOs) in a systematic manner, thisstudy argued that SCOs can be regarded as a new theory of smart construction. It demonstrated that SCOs, byproviding the enriching conceptual elements for defining and understanding smart construction, as well as byoffering deployment elements to achieve smart construction in industry practices, could foster a new theory ofsmart construction.
Linking the development of SCOs to the major limitations in smart construction studies, the elegance of
SCOs were identified as threefold:(a) SCOs enhanced the theoretical lucidity of smart construction by offering the theoretical concepts andframework to define and understand smart construction;(b) SCOs allowed the generalizability and customizability of smart construction by providing resilient
property diagram, system framework, and technical solutions to support implementation in diverse scenarios;424

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019(c) SCOs provided a non-disruptive approach to smart construction that were more likely to be welcomedby the conventional construction practitioners.
With SCOs serving as the basic elements of smart construction, it is envisaged that their development
will be further enriched in a variety of application scenarios and synergetic studies with other emerging
technologies in construction.
REFERENCES
Abowd, G. D., and Mynatt, E. D. (2000). Charting past, present, and future research in ubiquitous computing. ACM
Transactions on Computer-Human Interaction (TOCHI), 7(1), 29-58.
Andersen, T., and Gaarslev, A. (1996). Perspectives on artificial intelligence in the construction industry.
Engineering, Construction and Architectural Management, 3(1/2), 3-14.
Angelidou, M. (2015). Smart cities: A conjuncture of four forces. Cities, 47, 95-106.
Bock, T., and Linner, T. (2015). Robot oriented design. Cambridge University Press.
Cho, C. Y., Kwon, S., Shin, T. H., Chin, S., and Kim, Y. S. (2011). A development of next generation intelligent
construction liftcar toolkit for vertical material movement management. Automation in Construction, 20(1),14-27.
Downward, P., and Mearman, A. (2006). Retroduction as mixed methods triangulation in economic research:
Reorienting economics into social science. Cambridge Journal of Economics, 31(1), 77-99.
Greenfield, A. (2010). Everyware: The dawning age of ubiquitous computing. New Riders.
Hammad, A., Vahdatikhaki, F., Zhang, C., Mawlana, M., and Doriani, A. (2012). Towards the smart construction
site: Improving productivity and safety of construction projects using multi-agent systems, real-time
simulation and automated machine control. In Proceedings of the Winter Simulation Conference.
Harris, L. C., and Ogbonna, E. (2002). The unintended consequences of culture interventions: A study of
unexpected outcomes. British Journal of Management, 13(1), 31-49.
Khoshnevis, B. (2004). Automated construction by contour crafting-related robotics and information technologies.
Automation in Construction, 13(1), 5-19.
Kortuem, G., Alford, D., Ball, L., Busby, J., Davies, N., Efstratiou, C., . . . Kinder, K. (2007). Sensor networks or
smart artifacts? An exploration of organizational issues of an industrial health and safety monitoring system.
Springer Berlin Heidelberg.
Kortuem, G., Kawsar, F., Fitton, D., and Sundramoorthy, V. (2010). Smart objects as building blocks for theinternet of things. Internet Computing, IEEE, 14(1), 44-51.
Lee, K.Y., Lee, S.Y., Choi, J.H. Lee, S.H., and Han, C.S. (2006). The application of the human-robot cooperative
system for construction robot manipulating and installing heavy materials. SICE-ICASE International Joint
Conference, Busan, Korea.
Liu, D., Lu, W., and Niu, Y. (2018). Extended technology-acceptance model to make smart construction systems
successful. Journal of Construction Engineering and Management, 144(6), 04018035.
López, T. S., Ranasinghe, D. C., Harrison, M., and McFarlane, D. (2012). Using smart objects to build the internetof things. IEEE Internet.
Lu, W. Niu, Y., Liu, D., Chen K. and Ye, M. (2016). i-Core: Towards a customizable smart construction system
for Hong Kong. Innovation in Construction, 1, 71-79, ISSN 2312-8291.
Liu, D., Lu, W., Niu, Y., and Wong, H. (2015). A SCO-based tower crane system for prefabrication construction.
Proc. of CRIOCM2015 International Symposium on Advancement of Construction Management and Real
Estate, Hangzhou, China.
McGartland, M. R., and Hendrickson, C. T. (1985). Expert systems for construction project monitoring. Journalof Construction Engineering and Management, 111(3), 293-307.
Niu, Y., Anumba, C., and Lu, W. (2018). Taxonomy and deployment framework for emerging pervasive
technologies in construction projects. Journal of Construction Engineering and Management, in press.
Niu, Y., Lu, W., Xue, F., Liu, D., Chen, K., Fang, D., Anumba, C. (2019). Towards the “third wave”: An SCOenabled occupational health and safety management system for construction. Safety Science, 111, 213-223.
Niu, Y., Lu, W., Liu, D., Chen, K., Anumba, C., and Huang, G. G. (2016). An SCO-enabled logistics and supply
chain-management system in construction. Journal of Construction Engineering and Management, 143(3),04016103.
Niu, Y., Lu, W., Chen, K., Huang, G. G., and Anumba, C. (2015). Smart construction objects. Journal ofComputing in Civil Engineering, 30(4), 04015070.
Weiser, M. (1999). The computer for the 21st century. Mobile Computing and Communications Review, 3(3), 311.
Weiser, M., and Brown, J. S. (1997). The coming age of calm technology. Beyond calculation. Springer, New
York, NY.
Woudhuysen, J., and Abley, I. (2004). Why is construction so backward? Wiley Academy.

425

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Industrial/Technical Papers

426Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

COLLECTION DATA USING NEW TOOL
TO SATISFY SPECIFICATIONS
Yasushi Kawanai1
1) Registered Perfect Engineer (Civil), Director of E-bidding Support Center, Japan Construction Information Center, Akasaka,
Tokyo, Japan. Email: kawanaiy@jacic.or.jp

Abstract: As part of E-Bidding Support Center’s work with the Japanese Ministry of Land, Infrastructure,
Transportation and Tourism (MLIT), our help desk business for e-bidding is partially controlled by the‘‘performance regulations’’ on contract specifications, not amount regulations. The specifications say that thenecessary facilities and staff should be prepared for dealing with daily telephone inquiries. This does not involveamount regulations. And the specifications also say that we should avoid making people wait when they call.
Instead, immediate responses are requested.
In order to prove that people are not waiting when they call, we need a new data. Therefore, I found a tool
to obtain this data and prove that we met this requirement. By using a type of Interactive Voice Response (IVR)
system, we can get the data to prove people is not waiting when they call. As a result, we were able to prove that
we satisfied the MLIT specifications by the introduction of this system.
Keywords:

Interactive Voice Response (IVR), data analysis, MLIT, specifications, performance regulations

1. INTRODUCTION
The Japan Construction Information Center’s (JACIC) E-bidding Support Center has been entrusted every
financial year with work by the MLIT called ‘‘e-bidding system support work’’. This work mainly involves help
desk for questions. The contract specifications included ‘’amount regulations’’ in past years in order to prepare
the stuff and facilities for receiving 12,000 phone calls every year. However there were no amount regulations for
2018, and from this year, there are ‘‘performance regulations’’ to prepare the stuff and facilities not to make people
wait when they call.
How can we prove that people are not waiting when they call our center? What is needed for us to indicatethat we satisfied the specifications? In order to answer these questions, we will probably need new data, which wehave to get through daily work, and which presents a new and difficult task.
2. METHOD
Our e-bidding Support Center planned to get the new data as follows.
Another help desk office at the JACIC is using a tool called ‘‘Interactive Voice Response’’ (IVR2430-2,
TAKACOM Co. Ltd.). Our center has figured out how to use data recording with this tool and we think that if thisIVR system in able to keep daily log records and it can be connected with a telephone exchange box, we can get
the data we need.
Figure 1 shows a joint model using these tools.
The original function of IVR is to issue a message several times, even if the operators are busy on otherlines. The messages will be, for example, ‘‘our help desk is busy at the moment. Please hold a little ’’Figure 1. Joint system for IVR

427

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

However IVR also has a function to record how many telephones are being used at the same time andhow many messages are recorded every day, and can calculate a total of those records.
When we examine the data carefully, we hope we can determine whether any callers had to wait for someamount of time before speaking with an operator and whether the specifications were satisfied.
Our Support Center has 9 operators and 7 telephone lines. We also planned one additional line and used
it for IVR application.
3. RESULTS
We started IVR in April 2018 and got data for one year. The result is as follows; we received 12,090
telephone calls by the March 2019. There were 20 instances of 7 lines that were completely busy where the 8th
additional line worked, but there were only 2 messages that were interacted. Busy time was usually in the morninghour.
The first interactive message was planned to appear within the first 15 seconds after answering and thenevery 20 seconds after the first message at intervals. A second message didn’t appear in this data.
Table 1 shows reports of IVR on the June 2018.It shows how many times the additional line was used andthe number of IVR messages. During the mornings of June 12th,13th, and 18th, there were some instances when 7lines were completely busy. Furthermore, on the 12th and 13th, the additional IVR line was used. However bothtimes it did not last for 15 seconds and the first message was not voiced.
Table 1. Daily IVR Report

From this data we can imagine that on the 12th and 13th, the operator finished the previous call and took
the next with the additional line just before the message transferred within 15 seconds.
I myself had an observation that the telephone lines were full as shown in Figure 2. There are many such
instances between 10 and 11 in themorning.

Figure 2. Image of 7 busy lines428

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Before the start of this experiment, we imagined that there would be some instances where the message
would be played and the results turned out to be what we expected. However a second message (35 seconds later)
did not appear. There were just two instances of the first messages. Our service center receives more than 12,000
calls every year, so only two times is very few. We believe we were able to satisfy the specifications from thesedata.
In regard to the number of operators and telephone facilities, the data show that we did not make any
callers wait a long time for someone to answer, which means that our facilities are sufficient. Therefore weconclude that we have fulfilled the performance regulations.
4. DISCUSSION
We still believe that these data are not perfect. Areas of insufficiency include the probability of more calls
coming in at the same time and cases of calls suddenly being cut off before 15 seconds. We were able to check thestatus of the 8th call, but in the case of the 9th person to call at the same time, we were not able to recognize theircall. In addition if someone were to end the call within 15 seconds, we would not be able to recognize their behavior.
12,090 times’ call means 40 times a day on average when there are 300 working days in a year. And every
help desk have to answer only 5 calls or so in a day on average. That number is not so many. However there willbe almost 3 times calls of them at busy time. We have to prepare the facilities to meet the busy time. It is not sogood productivity. We have to consider how to make it to be average, too.
5. CONCLUSIONSThe MLIT entrusted work for our help desk had performance regulations that specified that we had toprepare enough stuff and facilities to make sure that no one would have to wait for response from an operator when
they called. For this request to be satisfied, JACIC introduced a tool called IVR and obtained data to use its
recording function. From the results of the given data, we believe that we fulfilled the specifications required forthis work.
However the current data seem to be insufficient. We cannot know what is behavior of each caller or
operators distinctively by this way and so we will try to get more data in the future.
REFERENCES
TAKACOM Co.Ltd. sales catalog (2016), IVR-2430Ⅱ.

429

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

EDUCATIONAL ACTIVITIES AIMED AT IMPROVING PRODUCTIVITY IN
JAPANESE CONSTRUCTION INDUSTRIES THROUGH 3D-CAD
Yasuyuki Kikyo1
1) Senior Researcher, Construction Information Research Institute, Japan Construction Information Center, Tokyo Japan.
Email: kikyoy@jacic.or.jp

Abstract: In 2014, the Japan Construction Information Center Foundation (JACIC), began a technical andinstitutional plan about applying CIM. At the same time, the Ministry of Land, Infrastructure, Transport andTourism (MLIT) announced its policy for CIM, but there is no educational organization that provides instruction
for 3D-CAD software in the civil engineering field.
We believe it is urgent to develop human resources who can use 3D-CAD software in the construction field.
Therefore, we started holding educational activities in Japan called "CIM Challenging Education and Training" asa non-profit service. We developed a special curriculum for 3D-CAD in the construction field with the cooperationof organizations such as buildingSMART Japan. To date, we have held this program eight times from 2015 to2018.
Keywords: BIM/CIM, develop human resources, 3D-CAD.
1. INTRODUCTION
1.1 JACICThe Japan Construction Information Center Foundation
CIM
Standard
Construction(JACIC) was founded in 1985 under the authority of the Ministry
Information
izationModeling/
Managementof Construction, Japan. The aim of the organization is to promote
the application of information and communication technology
for the sake of efficient and reliable execution of construction
Improvement of
productivity for
projects as shown Figure 1. Since that time, the JACIC has
Construction sector
CORINS
Cost
conducted various activities as a public organization with a
Estimation/TECRISneutral standpoint.
Construction Information System
We are managing four major information systems related to
public construction work in Japan as follows.
Construction
E‐Bidding(1) CORINS and TECRIS, Construction and Technicalby‐products
Procurement
Soil
Information
Consulting Records Information
generated(2) Construction Byproducts Resource/Surplus Construction
Figure 1. JACIC activities
Soil Information Exchange System(3) Public Procurement Information Service (PPI)(4) Research and development on the function and information service of a public works cost estimation system
In 2012, the JACIC created project teams consisting of members from various departments to tackle the subjects
that the JACIC considers social problems. The activities of the teams had to be completed within a year. Since
then, the project teams replaced their members to tackle subjects reflecting trends in the construction andinformation technology fields.
1.2 Purpose of starting educational activities
In 2014, the JACIC began to assess its technical and institutional plan about applying CIM. At the same time,
the Ministry of Land, Infrastructure, Transport and Tourism (MLIT) announced its policy for CIM, but there were
no educational organizations that provided instruction about 3D-CAD software in the civil engineering field. We
thought that it was urgent to develop human resources who can use 3D-CAD software in the construction field so
we started educational activities in Japan called "CIM Challenging Education and Training" as a non-profit service.
We developed the special curriculum for 3D-CAD in the civil engineering field with the cooperation oforganizations such as buildingSMART Japan.
2. METHOD
2.1 Plan for the curriculums
In November 2015, we collected the opinions of cooperative firms (e.g., training organizations and 3D-CAD
vendors) that could realize training curriculums related to five civil engineering fields (bridges, rivers, roads, dams,
and tunnels) on the 14 examples we presented. Based on the feedback from the cooperative firms, we made training
curriculums for civil engineering design that contained a road route design-related cutting and filling section aspart of civil engineering. We were unable to create any other training curriculums because they could not be
addressed with the use of 3D-CAD.

430

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

In selecting road routes with 3D-CAD, bridges and tunnels are generated parametrically as the assumed outline
design. In order to help trainees understand the advantages of 3D-CAD, we considered that the aim of thiscurriculum would be a rough estimate of the cost of construction and explained to residents about the overview of
the construction. Through explanations to residents with 3D-CAD, the consensus is easily reach and reduce any
reworking of the construction design. In addition, trainees can estimate the outline costs of the construction that
calculates the quantity of soil with 3D-CAD. There were no 2D-CAD processes that calculate the soil quantity andrough cost. After the November 2016 program, we organized a committee to consider more practical curriculums
because the trainees demanded ones that would be more practical. For example, other types of 3D-CAD and othercategories of civil engineering except for roads, soil embankments, and soil cutting. The committee included four
engineers who utilize 3D-CAD-related public works and civil engineering. For more practical 3D-CAD
curriculums, the committee proposed the following.
(1) It is hard to express the default object.
(2) Expression of underground supporting ground layers(3) Simulation in 3D-CAD
In August 2018, we can use IFC-tag in 3D-CAD so we included it in the curriculum. Furthermore, we added thecurriculums as follows.
1. Revise the borders of roads and sidewalks as well as tunnels used by pedestrians under embankments, and theshapes of crossings.
2. Simulations of sunlight along the embankments of houses.
3. Expression in the lower areas of bridges and ground layers and cost comparisons when changing the types ofupper parts of bridges
2.2 Offering curriculums to cooperative firms
At first we offered cooperative firms our curriculums without considering how to realize them. From thefirst class to the sixth, our curriculum was mainly about road route design. Before the seventh class, a committeethat considers how to revise the curriculums was organized. On May 2018, we tried to understand what wasdifficult with the curriculums with the cooperative firms and concluded our results that October.
2.3

Facilities and logistics
Early on we decided to have gatherings take place at a meeting room that we reserved for four daysin central Tokyo. We took into account transportation, power outlets, Wi-Fi, and management issues when
we selected the meeting room. Furthermore, there were demands for additional locations, so we also held thelectures in Niigata.
In addition, in 2017, we held meetings at an educational center where the trainees could stay for acheaper price because the trainings continued for three days.
PC specifications are as follows:
CPU Core i7 (8650U) -1.9 GHz
GPU Intel HD 620
16 GB RAM
500 GB HDD
At this training, we needed 36 PCs for 3 classes. Each class needed the following:
10 PCs for trainees
2 spare PCs
1 PC for the lecturer

At times we encountered software issues, but it was necessary to continue the training so we switched todifferent computers. We set the return due date for the rental PCs and the dealer collected them on the final day.
At the sixth training event, to reduce the operational burden, we occupied part of the distribution centerand installed the applications.
2.4

Recruits trainee
we recruited participants from web site and newspaper, and be full soon at 2016. However, at 2017
participants did not gather in two class.
After revised the curriculums, at 2018 be full soon again. We think that there are the demand of morepractical BIM/CIM curriculums.

431

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

2.5 Correct the opinions from trainee
After the classes, we collected the opinions of the trainees.
Around 95% of trainees had a positive opinion and it seems that they were able to apply 3D-CAD to theircivil engineering designs.
Table 1. Opinions
Strongly
Questionagree
1. Were you able to make out the
63.2%
instruction of this course?
2. How much did you understand the21.1%
contents and the practice in this course?
3. Are you available to adopt to your47.6%
business of the contents of this course?

Agree

DisagreeStrongly
disagree

36.8%

0.0%

0.0%

78.9%

0.0%

0.0%

47.4%

5.0%

0.0%

3. RESULTS
We held the training classes 8 times.

Class No.
1
2
3
4
5
6
7
8

Table 2. Dates of the manuscript
Date
Number of classes
February. 2016
2
March 20163
September 2016
3
October 2016
3
November 2016
3
August 20172
December 20172
August 20182

Fig.2 August 2018 Class 8-1

Number of trainees
21
18
23
22
15
22
13
21

Fig.3 August 2018 Class8-2

4. DISCUSSION
At this course, we gave the trainee that main concepts and operations related 3D-CAD in prior 2 days, andactually they operate the 3D-CAD related the concepts and the operations in latter 1 days.
And we guided trainees how to solve the question about the concepts and the operations related 3D-CAD.
There are a few fields where 3D-CAD can be applied to civil engineering. Therefore, it is necessary toincrease the fields where it can be applied.
Before the event, it was a burden to install hardware and software, but now it is expected that 3D-CAD
can be used with PCs without hardware or software restrictions.
5. CONCLUSIONSTo improve productivity in the construction field, as well as developing hardware and software, it isnecessary to continue human ability development. Both orders and receivers continually improve methods and soon, and we need to continue the activity of education for civil engineering technicians.

432

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019ACKNOWLEDGMENTS
To carry out this project, we had the cooperation of four engineers of the committee, educational center,
software developers and software venders.
REFERENCES
Nobuyoshi, Y. (2013). The needs and problems of the three-dimensional product model for CIM realization,
Public Works Management Journal, pp. 54-57.
MLIT Committee of promoting CIM. (2017). Guideline of promoting CIM, pp.9.

433

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

IMPROVEMENT OF ELECTRONIC BIDDING CORE SYSTEM
Hiroyuki Ishiwata1
1) Manager, Systems Engineering Department, Japan Construction Information Center Foundation (JACIC), Tokyo, Japan.
Email: ishiwath@jacic.or.jp

Abstract: Most Japanese public procurements were made through paper-based procedures before the year 2000.
However, there were some problems with these paper-based processes such as inefficiency, non-competitiveness,
and non-transparency. Furthermore, there was a serious issue involving unfairness in transportation costs and time
between the participating companies.
The Japanese government officially announced the “e-Japan Strategy” in 2001 and active application of“information, communication, and technology.” For public procurement, an electronic procurement method using
the internet was planned. According to the policy, JACIC had developed package software for an electronic
bidding system (Electronic Bidding Core System) in 2002. Afterwards, we launched some updates and are now
on our latest version, Version 6, released in March 2016. Currently, 800 public organizations have adopted oursoftware.
For 15 years, we worked with several types of IT technology and launched update modules for thissoftware. Then, a serious problem occurred in December 2016 that Java plugin technology will be abolishedstarting from the future version, Next Java SE Version 9. However, in September 2017, when Version 9 wasreleased, it was announced that the abolition schedule of the Java plugin had been pushed ahead. The majority ofJapanese public systems depend on Java plugin technology, and our Electronic Bidding Core System was no
exception, so we immediately got to work on an emergency project for our system.
This paper describes that project.
Keywords:

procurement, electronic bidding, JACIC

1. INTRODUCTION
In this article, I describe the background of this project. Electronic bidding systems (e-bidding systems)
have been adopted by most Japanese public organizations such as central ministries, prefectures, local governments,
and incorporated organizations. In particular, the Electronic Bidding Core System (EBCS), provided by the JACIC,
has been adopted by 5 central ministries, 47 prefectures, 756 local governments, 19 incorporated organizations inJapan. EBCS is package software which was developed so that each public organizations could build its own
electronic bidding system and it contains all of the necessary functions for that purpose, allowing publicorganizations to customize their systems as they see fit.
In September 2017, the IT vendor who provide Java announced that they would not implement Java plug-in function with Java 9 and would no longer offer free support for the Java -plug-in function for Java 8 in
September 2018 (Later it was postponed to January 2019). We had heard from the vendor that the Java -plug-infunction would be available with Java 9, so we were confused by the sudden change with their policy. Followingthat announcement, many public organizations, including those that used EBCS, were forced to consider othermeasures.
According to that announcement, JACIC had started to come up with a new method to replace the Javaplug-in in EBCS. Issues include not being dependent on vendor specific products, not using the unique functions
offered by each product, using common functions for multiple products, using standardized functions bystandardization organizations as long as possible, using the most appropriate systems, minimizing the impact on
the system used by public organizations, not asking contractors to pay fees, selecting the best language for the
program, and ensuring security.
2. METHOD
2.1 Conditions for alternative method
In order to decide the alternative method of Java plug-in, we asked the IT vendors, who are regularmembers of EBCS Development Consortium. We set several conditions for the proposal.
The first was to minimize the impact on public organizations. EBCS is package software for e-bidding
system, and each public organization uses EBCS to build their own specific e-bidding system. If each publicorganization had to implement large-scale customization when applying alternative functions, it would affect theirbudget and their application schedule.
The next condition was that we could receive technical support from the vendor. E-bidding system had
already become an infrastructure for public procurement, and it is most important to secure security, so if thevendor ended their support, public organizations had to return back to paper bidding. And that affect not only
public organizations but also bidders. Therefore, if problems arise in the technology used by the e-bidding system,434

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019it will be necessary for the vendor to promptly come up with a solution.
The schedule was another key issue. Since the Java plug-in vendor support period had been decided,development and application of public organizations must completed by that date, and since there are more than
200 systems, it is necessary to secure a one-year application period for each public organization.
The next condition was in regard to security. Security for the Java plug-in function had been ensured
through the vendor’s support, but when building a new application, it is necessary for security to be verified by athird-party security consultant company.
The final condition was how to select a development language. The program size of current Java applets
was large and the scale of development is different depending on the selected language. In addition, if we use a
programming language that has a completely different philosophy from Java, a large design change is required,
which leads to a delay in the development period.
2.2

Selection of Development Language
Until then, Java language had been available for about five years, but the vendor’s policy had changed,
and the usage period had been reduced to six months. Due to the version upgrade, it was assumed that the interface
and functions would change significantly, and we had to prepare for system updates every six months. Since thespan of system renewals for public organizations was about every five years, it is difficult to conduct a system
renewal every six months. Therefore, we decided to select a language other than Java.
First, we examined the languages available from the following point of view using general technicalbooks and information found online. Support could be expected and it had received more than five years ofdevelopment with regular security updates and bug fixes as well as being made compatible with drivers that usean IC card. It also featured high Java portability and had the ability to reduce the amount of system development,
and it could handle security and used a framework that could be expected to absorb the effects of OS version
upgrades. As a result, we decided to compare and examine C++ language, C# language, and C language.
Furthermore, to ensure security, we asked the company providing security inspections to verify thesystem configuration, programs, and prototypes.
3. RESULTS
3.1 Selection of alternative method(1) Windows application
First, we examined a method to create a new Windows application. Instead of the current Java applet, it
was created by developing a new, unique application that relays the browser and IC card reader. Communication
between the browser and the application was made using standard technologies such as JavaScript, WebSocket,
and Web Messaging API.
The advantages with this method were that was not affected by Applet's retirement or browser updates
and it could use the current external interface, so it could reduce the amount of external program changes and had
less impact on existing programs. In addition, because web standard technology was used, it could be expectedthat it could be used for a relatively long time.
Disadvantages include an increase in the burden on the help desk in response to inquiries regarding
Windows applications and a possibility that it would require higher performance specifications for the user’s client
PC and for the user’s PC than what was used currently. There was also a possibility that the capacity required forthe program to be installed may be increased.
The main issue was that it was a proprietary method, so security indicators and verification methods had
not been established and verification by a third party was required, so it was necessary to consider how to distributethe applications, update methods, and help desk operations.
(2) Browser Extension Usage
For the next method, we examined browser extension usage by setting the target browsers as Chrome orEdge, and made the Java Applet's browser extension (JavaScript). In addition, we defined a new interface that calls
DLL from JavaScript and IC card access function.
The advantage was that it was not affected by the applet being discontinued and it could be used toreduce the burden of installation work newly generated in the user PC depending on the distribution method, suchas using Google Store.
However, there were several disadvantages. As HTML 4 cannot be used, web standard design using
HTML 5 must be introduced, IE dependent screens must be changed, and the use of the Chrome Web Store should
be considered. In addition, it was also affected by Chrome updates and required redefinition of external interfaces
and application signing.
Furthermore, there were issues with how to use application signatures, the redefinition of an externalinterface, and a distribution method for browser extensions.

435

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019(3) JWS
As a third method, we examined the JWS application method. Instead of a Java plugin, we used JWS(Java Web Start, a technology provided by a Java vendor) to create a new Java application that replaces the Java
applet and only converts screens that require signatures and signature verification into applications such as Java.
The screen was only displayed using a browser.
The advantages were that it was a system recommended by the Java provider, it could be expected to be
used for a relatively long time, and the modified application was downloaded as JWS so installation was notnecessary.
Disadvantages included the need to change the design of some screens, the need to change the design ofthe customization screen of the user group, and the need to control multiple screens for communication between
the browser and the Java application. The security level was lowered because the random number signature of thescreen transition was changed from all screens to just some of them.
We compared and examined the above three methods and found that the second method was affected bybrowser specifications that change frequently, and the third method was a unique, applet-like function provided
by the Java provider, who may decide to suddenly end support of the application. It was also determined that anunderstanding of the user group could not be obtained. Although the development period would be longer thanwith the second and third methods, we decided to adopt the first method because it can minimize the impact ofvendors.
3.2

Selection of development language
There were hundreds of development languages and each had its own features. Therefore, it wasnecessary to select the most suitable language in consideration of the frequency of updates and security responses,
as well as the availability of support, available OS, and type and size of application to be created.
Since e-bidding systems were used by public organizations, they might be stable and available for a
relatively long period of time. We also narrowed down the language with this in mind since it was most importantthat the operation period be available for five years, taking into account the system update span of each ordering
agency.
(1) JavaScript
First we examined JavaScript as a script language. JavaScript was characterized by being the mostwidely used programming language in the world and was capable of describing from the server side to the clientside. In addition, it was easy to modify since it was a script language.
(2) C++
Next, we examined the C++ language, which was a Java-based language. C++ could be described indetail, but the specifications were huge, complicated, and difficult. Furthermore, memory management, which wasa hotbed for bugs, must be performed by the users themselves, and since C++ did not consider portability, it wasnecessary to create programs specific to each OS.
(3) C#
Next, we examined C#, which was generally considered a language close to Java. Both C# and Java
were based on C++, and C# had a history of initially referring to Java. Both languages had the common purposeof removing complexity from C++ to ease the burden on programmers and to have the same code work in any
environment. The processing system also handled memory management, which was prone to bugs.

Overview

Table 1. Selection of development language
Java (current
CC++
system)Introduced in 1994.
Introduced in 1973. Introduced in 1980.
Java runs on the Java C is one of the most C++ is one of the
Virtual Machine
widely-used
most widely-used(JVM). Its
programming
programming
specifications have a languages. In
languages and is anstrong focus onparticular, it
extension of the C
being used in aprovides many
language.
network
opportunities forenvironment.
use in thedevelopment ofprograms that
directly control

436

C#
C# is a softwaredevelopment
language for theMicrosoft .NET
environment
announced byMicrosoft in 2000.
It incorporates thebest of the major
programming
languages.

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Porting fromJava language－

RuntimeEnvironment
Developers

Java Runtime
Environment (JRE)Manyhardware.
Relatively easy

Relatively easy
- C# is called“Microsoft's Java”

Windows OS

Relatively difficult- Language
specification iscomplicated- Memory
management
Windows OS

Relatively few

Relatively few

Many.NET Framework

We decided to choose C # because of the following advantages.
-It can be expected to be used for a long time because it is the main language of Microsoft-By using .Net Framework, it is possible to reduce development volume and ensure quality and security.
-No need for memory management coding-Relatively easy porting from Java
3.3

Conduct security verification
We asked an external security consultant company to verify our specifications and our programs, andafter conducting security diagnostics, we confirmed that there were no security issues.
4. DISCUSSION
By examining alternatives to the Java plugin, we were able to break away from old technologies that arehampered by security problems. Then, we were able to switch to a new language by changing the programming
language. I think that it is worthwhile to take measures against the crucial effects of the changes in vendor policy.
However, we decided to use the .Net framework as an execution environment to reduce the amount of development
and increase the quality efficiency of applications. As a result, new challenges have arisen that will require keeping
an eye on the vendor's policy of providing the .Net framework.
4.1 Problem with using communication port
As an implementation issue, there was the use of communication port.
The PC side application developed this time always occupies the communication port for communication with thebrowser. Since the communication port is freely available on each PC, it may conflict with existing PC-side
applications.
Although it is possible to change the port according to the setting on the PC side, there are many users who
participate in the electronic bidding who are builders and are not familiar with the PC. We need to be careful about
the design so that they won't be able to bid because they don't know the settings of the PC. Therefore, we chose aport that is not commonly used, and asked the user that the selected port is not used by the existing application.
We chose to fix the port as a method to minimize the possibility of users experiencing trouble.
5. CONCLUSIONS
From this examination, we were able to avoid the worst situation that the electronic bidding system
would not be available. If we cannot present this measure to public organizations, it will likely result in reducing
the reliability of our electronic bidding system and some ordering agencies will probably return to paper bidding.
The electronic bidding system is a system that uses a web server, AP server, DB server, Windows server,
Linux, Solaris, Windows 10, and Windows 8.1, as well as an IC card, an electronic certificate, and an encryption
algorithm. We will work to collect vendor support policies for these products, and if we see the possibility of an
end of support, we will try to systematically take early action.
ACKNOWLEDGMENTS
The author of this paper would like to thank all electronic bidding system users.
REFERENCES
JACIC (Japan Construction Information Center Foundation) website:
http://www.jacic.or.jp/english/index.html, accessed on April 15, 2019.
Electronic Bidding Core System website:
http://www.cals.jacic.or.jp/english/coreconso/, accessed on April 15, 2019.

437

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

INTEGRATION OF 3D MODELS OF STRUCTURES AND GEOLOGICALCOMPOSITION AS AN UNDERGROUND INFRASTRUCTURE MODEL
Toshiaki Hakoda1, Syoichi Nishiyama 2, Takaki Omori3, Isao Shiozaki4, Mamoru Narusawa5,and Nobuyoshi Yabuki61) Assistant General Manager, Overseas Infrastructure Project Division, JGC Corporation, Yokohama, Japan. Email:hakoda.toshiaki@jgc.com
2) P.E., JP (Applied Science), General Manager, Social System Business Division, OYO Corporation, Tokyo, Japan. Email:nisiyama-syoichi@oyonet.oyo.co.jp
3) P.E., JP (Applied Science), General Manager, Engineering Division, Nikken Sekkei Civil Engineering Ltd., Email:oomorita@nikken.jp
4) Dr. Eng., General Manager, Geo-space Engineering Center (GEC), Engineering Advancement Association of Japan (ENAA)
Email: shiozaki@enaa.or.jp
5) Deputy Manager, Facility Management Department, ESCA-SC, Email: narusawa@esca-sc.com
6) Ph.D., Prof., Division of Sustainable Energy and Environmental Engineering, Graduate School of Engineering, Osaka
University. Email: yabuki@see.eng.osaka-u.ac.jp

Abstract: To develop three-dimensional (3D) models as examples of the infrastructure model concept which
would be useful for the operation and maintenance or design and construction of underground facilities, we
integrated 3D models of the geological composition, the structure and the lifeline facilities (water supply andsewerage system, city gas line, electrical power and communication lines) for the underground mall named ESCA
in Nagoya City, Japan. After the integration of the above 3D models, we were able to show the owner, ESCA-SC,
that the soil layers were suitable to ensure the stability of the structure. They also checked the 3D models by
utilizing the Mixed Reality (MR) system and found the integrated 3D models would be very useful for making aplan for the modification and renovation of their facilities.
Keywords: 3D geological model, underground mall and lifeline, Mixed Reality (MR)
1. INTRODUCTION
Creating a digital platform of three-dimensional models of buildings has become quite common inadvanced cities in the world, although most underground facilities and lifelines including geological composition
are not modeled yet because of insufficient information for these.
The Geo-space Engineering Center (GEC) of the Engineering Advancement Association (ENAA) of
Japan has organized a Committee for the Integration of Underground 3D models (CIU) and its Working Group(WG) with engineering consultants, OYO Corporation (OYO) and Nikken Sekkei Civil Engineering, Ltd. (NSC),
a construction company, Shimizu Corporation (SMZ) and an engineering company, JGC Corporation (JGC). The
WG has surveyed several major local municipalities and authorities in Japan since 2017 and has found that most
local municipalities and authorities have managed their facilities mainly relying on the two-dimensional (2D)digital data and printed documents, except for some authorities which utilize the 3D model for consensus formation
between the parties or personnel concerned. CIU decided to proceed by directing the WG to collect sample data,
including information on the geological composition to be used for the preparation of a 3D model of undergroundfacilities. CIU selected ESCA underground mall connected to the west entrance of Nagoya Station in the Tokai
area of Japan, for NSC had already created a 3D model of the ESCA underground mall and parking lot for thesimulation of seismic strengthening works and an evacuation plan of the mall for shoppers and guests in emergency
cases.
Before commencing the modeling work by the WG, ESCA-SC (the owner of ESCA) joined this study andissued Employer’s Information Requirement (EIR) and the WG made a BIM Execution Plan (BEP) for thismodeling work. In addition to the EIR, the WG proposed to utilize a Mixed Reality (MR) system for the planningof the modification of utilities and confirmation of the lifelines into tie-in points of ESCA. ESCA-SC acceptedthat proposal and the WG demonstrated the MR system to ESCA-SC at the site of ESCA.
The WG also studied the latest sample of the utilization of Augmented Reality (AR) for (a) underground
piping works by a construction company and (b) 3D modeling for oil and gas plants by an engineering company.
In the former case, an ingenious technique for using the AR system is described in 2.1 Civil works. In the latter
case, the task of Information Management for an engineering company is presented in 2.2 Plant works.
This paper describes the results of the above studies and integrated 3D models.
2. EXAMPLES OF 3D PIPING MODELS FOR CIVIL AND PLANT WORKS
2.1 Civil works
One effective use of 3D models for civil works is for the visualization of buried underground facilities
before excavation by using AR. When all lifelines were visualized in 3D models through the AR system, a user of438Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

SMZ noticed that levels of existing lifelines are visually perceived as objects above ground. To avoid this false
perception, a limited area to be excavated is highlighted as a 3D model and other lifelines are projected on a roadas a 2D image. Figure 1 shows a 3D model of lifelines which are recognized below ground level in the highlighted
area of excavation.

Figure 1. AR Image of Lifelines in Highlighted 3D model and 2D model Projected on a Road
2.2 Plant works
For oil and gas plant works, 3D modeling of complicated piping and equipment by using Smart Plant
3D of Intergraph and PDMS of AVEVA is extremely useful. These 3D models are mainly utilized for clash
detection in the design stage and simulation of the construction sequence as a 4D system. Figure 2 shows a sample
of a 3D model of piping and valves designed by JGC. Recently, major clients have mandated the submission of
total 3D models with detailed properties of the whole plant for operation and maintenance as the contract
conditions of Information Management. To fulfill that requirement, JGC is developing a total data-centric
engineering system for new gas plants.

Figure 2. 3D Model of Piping and Equipment of Oil and Gas Plant
3. PROCEDURE OF 3D MODELING FOR ESCA UNDERGROUND MALL
A list of software packages used for the modeling of each object is shown in Table 1. 3D models aredeveloped by the WG for each object and integrated into one model.
Table 1. Software for modeling
Model
SoftwareArchiCAD, Revit, AutoCAD
Structure of ESCA undergroundmall, parking and staircases
Geological Composition
GEO-CRE, OCTAS Modeler
Ground Anchors
Revit
Lifelines
AutoCAD, Revit
Integration of models
Navisworks
Mixed Reality
GyroEye Holo
3.1 Development of geological composition ESCA underground mall
Before making a detailed image of the geological composition of the land immediately below the ESCA
underground mall, the WG checked the existing public geological report for a wider area around the Nagoya
Station. After that, the WG utilized OYO’s software named “GEO-CRE” for geological comparison work, and the“OCTAS Modeler” for solid modeling work based on the data of twelve borehole logs given by ESCA-SC dated
in 1969 before the commencement of construction of the ESCA underground mall.

439

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

The procedure for analysis of the 3D geological model was made in accordance with “3D geologicalanalysis manual Ver.1.5 2019” published by the 3D Geological Analysis Technology Consortium, Japan. The
algorithm of interpolation of spacing for the calculation of the surface model is based on BS-Horizon (Fortranprogram for generating the geological surface using cubic B-Spline) (Masumoto et al., 2004).
The 3D model of ground anchors was made with Revit from the discarded materials for construction of
the retaining wall and combined with the 3D geological composition. The WG was able to confirm that the groundanchors reached into a firm layer.
Figure 3 shows a sectional view of the structure of ESCA resting on a firm condition of bearing soil layers.
Figure 4 shows the soil conditions of the bottom level for ESCA and the surround of retaining walls. Figure 5
shows the soil conditions around ground anchors which are assumed to be stable in a firm bearing layer of soil atthe construction period of retaining walls.

Figure 3. Section of geological composition and structure of ESCA

Figure 4. Soil Conditions of Bottom Level and Surround of Retaining Walls for ESCA

Figure 5. Soil Condition Surrounding the Ground Anchors
3.2 Development and integration of lifelines and structural model of ESCA underground mall
3D structural model of ESCA underground mall including the parking floor is developed by NSC basedon as-built drawings. Lifelines into ESCA are buried beside the retaining wall of ESCA. In order to visualize
lifelines and clarify the positions of tie-in points into ESCA, the WG developed 3D models of lifelines andcombined these with structural models of ESCA.

440

Proceedings of the 4th International Conference on Civil and Building Engineering Informatics, 2019

Figure 6 shows the 3D lifelines model of the southwest area of ESCA underground mall. City gas lines
and electrical power lines are omitted from this image due to a reason of security requested by those operationcompanies.

Figure 6. 3D Lifelines Model at Southwest Area of ESCA Underground Mall
The WG and ESCA-SC share the above 3D models in “box” cloud storage (application for Salesforce.com
Inc.) and can be accessed by using Autodesk Navisworks Freedom.
3.3 3D lifelines model and 3D structural model of ESCA checked by the MR system
The management department of ESCA-SC checked the 3D model of lifelines near ESCA by the MR
software named GyroEye Holo of Informatix, Inc., through a device of Microsoft HoloLens from inside andoutside of ESCA. They confirmed the positions of lifelines with the structure of ESCA and found that the MR
system could be very useful to make a renovation plan for utility lines. Figure 7 shows a captured image of the 3D
lifelines model and 3D structural model of ESCA by the GyroEye Holo as seen through the head-mounted display
worn by the observers in the picture on the right.

Figure 7. Captured Image of Lifeline and Structure of ESCA by GyroEye Holo
4. CONCLUSIONSThis report showed that the integration of 3D geological model, underground structures, remained groundanchors and lifelines were effective for making a plan for the maintenance and renovation of ESCA undergroundmall. On the other hand, we recognized the difficulties posed by 3D modeling from complicated descriptions of2D drawings for lifelines given by operators and local authorities, which would be a critical issue for 3D modelingin the future. This study is the first step for developing a 3D underground infrastructure model including geologicalmodel, and we hope that these models shown as samples here would be useful for operators of facilities and forsurveying, design consultants and construction companies to improve quality, cost-effectiveness and time
efficiency.
An especially significant aspect of this study is believed to be the integration of the 3D modeling of thegeological composition together with that of underground facilities.
ACKNOWLEDGMENTSENAA GEC would like to thank subsidy business, JKA, for their support of this study of the undergroundinfrastructure model. Whole reports written in Japanese of 3D modeling of underground information as anunderground infrastructure model for 2017 and 2018 fiscal year can be referred in the website of ENAA GEC.
(URL: https://www.enaa.or.jp/jka-subsidy-business/h29, https://www.enaa.or.jp/jka-subsidy-business/h30)
REFERENCES
Masumoto, S., Raghavan, V., Yonezawa, G., Nemoto, T. and Shiono, K. 2004. Construction and Visualization ofa Three Dimensional Geologic Model Using GRASS GIS, Transactions in GIS, 8, 211-223.

441

Proceeding of the 4th Interenational Conference on Civil and Building Engineering Informatics
ISBN978-4-600-00276-3
https://www.iccbei2019.com

ICCBEI 2019 Organizing Committee