Automated Damage Detection and Structural Modelling with Laser Scanning

Inspection of aging and deteriorating infrastructure such as bridges, dams, power supply infrastructures are crucial to assure their reliability as well as to estimate their remaining life. This study proposes to use small, low flying autonomous unmanned aerial vehicles, coupled with three dimensional laser scanning, high resolution imaging and state of art modeling and analysis abilities in order to provide high precision damage detection, condition assessment and modelling of structures and infrastructure systems. In recent years, laser scanning has been shown to be effective in capturing three dimensional geometrical information with a high degree of accuracy, which enables automatic visual damage detection and creation of high precision information and computational models. This paper proposes a set of strategies for processing captured texture mapped laser point cloud automatically in order to detect surface damage and large element deformation of the infrastructure system as well as to create high fidelity computational models. Both predicted surface and determined geometric properties, which are extracted from captured texture mapped point clouds, are used for locating, differentiating, quantifying and documenting surface damage on infrastructures. Also, high quality volumetric finite element meshes are generated through processing the raw 3D point cloud using proposed mesh generation strategies and validated against experimental specimen. In order to show the robustness and effectiveness of these capabilities, the proposed strategies for damage detection and computational model generation are validated and tested against both synthetic and real point cloud data.  The recent ASCE Report Card for infrastructure in the U.S. (released in July 2014) revealed that the average grade for all infrastructure types was D+ which emphasizes the importance of assessing the current conditions and performing health monitoring on infrastructure systems. Current routine inspection requires sending inspectors to observe, 1 Graduate Research Assistant, Northeastern University, yan.yuj@husky.neu.edu 2 Assistant Professor, Hacettepe University, burcuguldur@hacettepe.edu.tr 3 Robotics Engineer, Robotic Institute, Carnegie Mellon University, lyoder@cmu.edu 4 Graduate Research Assistant, Carnegie Mellon University, vkasired@andrew.cmu.edu 5 Senior Systems Scientist, Robotic Institute, Carnegie Mellon University, huber@andrew.cmu.edu 6 Systems Scientist, Robotic Institute, Carnegie Mellon University, basti@andrew.cmu.edu 7 Paul Christiano Professor of Civil and Environmental Engineering, bakinci@cmu.edu 8 CDM Smith Professor and Department Chair, Northeastern University, JF.Hajjar@neu.edu measure and record defects manually, which is labor intensive and dangerous. This inspection process usually requires shutting down a portion of or all of the structure's daily operation, which may result in significant inconvenience. Results recorded in this way are error prone since they are subject to transcription errors, measurement errors, and dependence on the personal judgment of the inspector. There is also typically no automated way to compare inspection results over a period of time, so it is difficult to track the propagation of damage. In order to assess, document, and monitor the health condition of infrastructure systems in a more rapid and practical way, in this work we are developing the use of a small, lowflying robotic assistant (i.e. an unmanned aerial vehicle) mounted with a threedimensional laser scanner and a high resolution camera, to capture texture mapped laser point clouds representing the geometrical information of infrastructures. The captured texture mapped point clouds are processed later through state of the art damage detection, modelling, analysis strategies. This robot provides assistance not only for routine inspection but also for the rapid assessments after disasters like earthquakes, severe storms and explosions, especially for reaching dangerous and inaccessible locations. To achieve this purpose, six main steps are proposed in this work, as shown in Fig. 1.  Research themes Manually flying the robot in close proximity to a structure is difficult for a human pilot, and it may be difficult for a human pilot to choose a flight path from which the robot can collect complete data from the structure. Flying safely close to structures and guaranteeing complete data collection are two low level tasks well suited for robotic algorithms (Yoder and Scherer 2015) When a human pilot is relieved of low level piloting tasks, they are free to guide the mission from a high level. In this work, an approach has been developed in which a pilot specifies the structure to be scanned by the robot using a 3D bounding box and a graphical user interface. As the robot completes the mission, the user is free to change the bounding box, or to specify new bounding boxes for the robot to perform scans. Once all surfaces are modeled inside the bounding boxes, the robot detects that the mission has been completed and returns home with high quality data (Yoder and Scherer 2015) Conventional terrestrial laser scanners are able to produce accurate registered point clouds because the scanner's base is stationary. In the case of an aerial laser scanner where the vehicle is moving, each measurement comes from a different aerial vehicle position. Creating a registered map in the aerial laser scanner case requires using additional sensors like inertial measurement units (IMUs) and global positioning systems (GPS) Since infrastructure inspection robots often operate in environments without sufficient GPS coverage, they must use their onboard sensors to estimate the position of the aerial vehicle. This problem is known as simultaneous localization and mapping (SLAM) since the vehicle must simultaneously estimate the aerial vehicle's position using the laser point cloud and register the point cloud using the vehicle's position. Our solution uses an IMU and a Velodyne VLP 16 laser scanner mounted on a rotating gimbal, as shown in Fig. 2(a) The IMU and laser data is processed using a novel SLAM approach to produce low latency aerial vehicle position estimates and a registered point cloud (Zhang et al. 2014)  (a) Low flying robots; (b) Automatic segmentation and semantic modelling Registered point clouds provide several capabilities such as distance measurement and coarse visualization. However, in order to gain full benefit of the 3D data, we propose using semantic modeling algorithms to transform the point cloud into an information rich, objectoriented representation the infrastructure equivalent of a Building Information Model (BIM) Such a representation enables an inspector to interact with the robot at the symbolic level, for example by commanding the robot to inspect a bridge's columns at a 2 cm resolution and provide a report rather than having to specify each column's location and extent manually. Furthermore, the representation can provide a pathway to create computational models needed for structural assessment. Our approach uses computer vision techniques to segment components of the structure, identify the coarse topology, and recognize and model individual elements, such as columns, beams, truss structures, and abutments on a bridge as shown in Fig. 2(b) (Song, 2015 and Li, 2015) Once the components of a structure are segmented, it is possible to determine the surface damage by using the captured texture mapped point cloud. Both 3D point data and color information associated with each 3D point in the registered cloud are used for locating, differentiating, quantifying and documenting several damage types that include small deformations (cracks, spalling, etc. large deformations with no change in topology (crosssections of the components remain nearly intact) and large deformations with localized change in topology (localized cross sectional changes where the alignments of the components are not effected entirely) The extracted surface properties along with the geometric properties of the investigated components are used for capturing the deviations from the initial conditions. These detected deviations are recorded and certain visual inspection criteria is used for assigning a condition rating to the components of the investigated structures. This process is then followed by the finite element model generation by using the components extracted from the captured point cloud to perform a detailed assessment of the current condition of the structure (Guldur, B. Hajjar, J. 2014c) In order to assess and monitor the health condition of infrastructure through finite element analysis, computational models including up to date geometry, material and damage information should be generated. The structural modelling procedure developed for this study aims to enable the automatic generation of computational modelling from collected 3D raw point cloud data, derived semantic/information models, or a combination of both. Macro level computational models using beam type element will be generated to enable efficient linear and nonlinear analysis for characterizing the shortcomings and identifying key components of infrastructures. In order to assess the health conditions more accurately, high fidelity continuum models with nonlinear material and geometric properties are also created. The resulting computational models are correlated with observed damage information. These computational models coupled with high precision nonlinear finite element analysis will provide information on up to date conditions and behavior of infrastructure systems with possibilities for documenting damage information, damage propagation, remaining life, and related assessments.  Automated generation of finite element model of damaged steel girder bridge system The final theme in this project is immersive and information r