Industrial application of 3D mapping
By: Lyubomyr Demkiv
Tags: Robotics, Slam, Mapping, Vision, 3d Reconstruction

Photo by Phil Hearing on Unsplash

This article is written in co-authorship with Denys Kotelovych

Where is 3D mapping most expected in the first place?

3D mapping (also called projection mapping or spatial augmented reality) of the environments is a maturing technique with a high demand on the market. The problem may be split into two tasks: outdoor mapping and indoor mapping. Though being seemingly similar and using akin algorithms, these tasks are performed by various companies, using different sensors. For example, twelve National Mapping Agencies in Europe joined in a 3D Special Interest Group in the framework of “European Spatial Data Research” [1]. Their main focus is increasing the mapping scale, using less expensive and, at the same time, more appropriate sensors. The obtained 3D data can potentially be used in simulating flooding, analyzing solar potential, and maintaining dikes in European countries. Moreover, 3D city maps [2] would enable visualization and increase understanding of areas of interest in the urban design-context in a spatial, timely, and virtual manner. Additionally, 3D mapping may be used for campus, offices, events, and retail infrastructure mapping.

The approaches that are used to solve the mapping problems may be divided into three following groups (see fig. 1):

• Geographic information systems (GIS) and computer-aided design (CAD) that use the corresponding software to design 3D models of the environment (the results of such an approach for an ancient Ukrainian city may be found by following this link: https://bit.ly/2JyF4h4).

• Mobile mapping systems (MMSs) like drones and UARs equipped with cameras (e.g. RGB-D cameras) and other sensors that navigate the terrain performing either autonomous or human-controller 3D mapping.

• Photogrammetry that requires a significant amount of 2D images that will be further augmented into a 3D map of the environment. One of the examples of this approach may be found here http://tiny.cc/outfmz

Figure 1: Common 3D mapping techniques

Among the industries (see fig. 2), the majority of outdoor mapping applications are focused on examining the buildings, analyzing the geo- infrastructure, inspecting real estate, analyzing mining and managing emergency situations, and performing precise agriculture and forestry.

Figure 2: Outdoor 3D mapping by industry

Generally, MMSs that are used for mapping can extract detailed 3D data with high-quality texture information of the 3DFigure 2: Outdoor 3D mapping by industry building model. However, mapping urban areas by MMSs is expensive and requires advanced mathematical approaches with complicated steps. In particular, a commercial MMS, which generally includes two GPS receivers, is an expensive device that costs about $1 million.

Currently, most technological tools are used for outdoor mapping. Although, indoor mapping applications are gaining a significant popularity.

There are multiple indoor mapping methods ( see [3]) available to the community. Among them are traditional terrestrial laser scanners that are rather slow; the research prototypes Aalto VILMA and the Würzburg backpack that are promising only through experimental proof of concepts; the FGI Slammer, which may be used only on a flat surface, and a commercial indoor mapping system Matterport, which is inaccurate and immobile; NavVis, which works only with close-to-flat surfaces, Zebedee, which has a low data capture rate; Stencil, which eventually generates double surfaces, and Leica Pegasus: Backpack, which has lower performance than its competitors.

As you can see, though being developed for many years by multiple teams, indoor localization is still an open problem. One of the main challenges of mobile mapping is integrating and synchronizing an image, lidar, and georeferencing systems from the mechanical, electrical, and software point of view to provide images and point clouds precisely georeferenced in a geographical reference system. Moreover, there are environments and assets geometry that are challenging for 3D mapping: a concrete wall with high uniformity, a homogenous or a shiny painting, a shiny metallic surface, stainless steel, glass, windows. On the contrary, you can achieve the best performance of the algorithms in environments with dirty surfaces, algae, natural rocks, brick walls, old painting, cracks, and corrosion. From an environment geometry point of view, large assets that allow navigating separate lines to cover the same areas from different angles are preferable to linear assets, such as ducts, sewers, or narrow chimneys. The assets comprising of a single volume (such as cylindrical or spherical tanks, large chimneys, boilers, etc.) where all parts of the asset can be more or less seen from everywhere in the asset, are preferable to the assets split into several volumes, connected by manholes, corridors, or narrow passages. The assets containing long linear features (such as welds, pipes, etc.) are preferable to assets that lack such visual features.

For indoor 3D mapping, there is a growing demand in many domains, among which are retail, finance (insurance), construction, and others. In fig.3, you can find some of the most promising domains and their demand. There are also some tasks that are common to all the domains (disconnected, marked in the black color), like interactive mapping, emergency planning (evacuation path), and integration with local data and systems.

Figure 3: Indoor 3D mapping (the tasks in black rectangles are common to all domains)

Same as for the outdoors, MMSs are the most promising in creating 3D maps of the indoor environment. In robotics, Robot Operating System (ROS) is de-facto a standard. Therefore, we are using it for all our robotics solutions. Among all the variety of approaches, we will focus on three methods: COLMAP, [4] Open3D [5] (a complete library for processing RGB-D images and their enhancement) and RTAB-Map [6] (flexible in terms of hardware, which is often used on robots as an online routine). All approaches described below are compatible with ROS and may be used for different MMSs.

Below, we will compare pros and cons of each approach using two datasets:

living and office rooms that are captured using plain and RGB-D cameras (Microsoft Kinect v2) and are placed onto TurtleBot for RTAB-Map case. The differences and comparison of the outcome of the methods are also presented.

2. 3D Mapping methods used by industry

2.1. Photogrammetry approach: COLMAP

By having multiple pictures from different viewpoints, it is possible to extract a 3D structure from them. COLMAP consists of a command-line interface (CLI) as well as a graphic user interface (GUI) for a better visualization and parameter tuning. The pipeline has the following steps (see fig. 4):

Figure 4: COLMAP reconstruction pipeline

• Images. Prepare images for the future object or place to be reconstructed (in general, it is better to have at least 40–60% of overlapping between the consecutive images).

• Feature extraction. For each image from the dataset visual features extracted (SIFT, [7] SURF [8], BRISK [9], etc.).

• Matching. The extracted features are matched between the images to find a correspondence for the future processing.

• Geometric verification. Some of the matches might not be correctly paired, so in this step, COLMAP estimates a fundamental matrix and tries to improve the future matching by filtering some of the errors.

• Initialization for incremental reconstruction. An initial pair of images is chosen in this step to start a reconstruction process (it is crucial to pick a pair with a lot of correspondences because it will influence the next steps a lot).

• Image registration. Camera pose and intrinsic parameters (if needed) are estimated for the new image.

• Triangulation. New points will be added to the scene only if the existing camera poses can see them from the different viewpoints (see [10]).

• Bundle adjustment. Since image registration and triangulation influence each other a lot, a small error in one step increases the error in another one, so it must be refined. A reprojection error could be minimized by attempting to refine the camera and point parameters (see [11])

• Outlier filtering. Observations with large reprojection errors are filtered.

The results obtained with both datasets are given on the fig. 5 and fig. 6.

Figure 5a: COLMAP results of the office room

Figure 5b: COLMAP results of the office room

Figure 6a: COLMAP results of the living room

Figure 6b: COLMAP results of the living room

2.2. Alternate approach: Open3D

Open3D is an open-source library written in C++ with Python API. It is relatively easy to integrate it in any project with great high-level interfaces for dealing with algorithms and data structures (Listing 1). This method may work with the data obtained from multiple sources either by a robot or someone else, which makes this method an alternant one. It is one of the cons of this approach. The data structures are separated into the point clouds, meshes, and RGB-D images. For each of them, I/O, sampling, visualization, and data conversion is implemented as well.

Listing 1: Open3D reconstruction system

The key components of the reconstruction system are the following:

• Make fragments. Build local geometric surfaces (fragments) from a sub-sequence of RGB-D images (a batch of images is processed and “merged” into the one-point cloud).

• Register fragments. The fragments are aligned into a global coordinate system for the loop closure detection.

• Refine registration. The alignment of fragments is refined.

• Integrate scene. Mesh of the model generation.

You can see the obtained results with both data sets on the fig. 7 and fig. 8.

Figure 7a: Open3D results of the office room

Figure 7b: Open3D results of the office room

Figure 8a: Open3D results of the living room

Figure 8b: Open3D results of the living room

2.3. Pure robotics approach: RTAB-Map



Real-Time Appearance-Based Mapping (RTAB-Map) is a popular graph-based SLAM library. It can be used as a standalone cross-platform application and as a ROS package. Since it is flexible, almost any component can be changed and adapted to any kind of robot with multiple cameras and sensors. In general, many configurations can be built using their nodes but, in this paper, we will use the followings ones:

• Visual odometry. The features are extracted from the image sequences and visual odometry is calculated in real-time and then refined with a local bundle adjustment.

• RGB-D sync. Color and depth images are taken in different timestamps, so we will have to synchronize them before we use them.

• Kalman filter [12]. It is used to combine and smoothen visual, wheel, and laser odometries into a single odometry output, which can then be used in the RTAB-Map node (not a part of the package).

• RTAB-Map. Odometry and point clouds that are generated by 2D

LIDAR as inputs are used here to create, update, and refine graphs.

The structure of the map is depicted as nodes and links. After the RGB-D sync, the Short-Term Memory (STM) module creates nodes with an odometry pose and other information at a fixed rate, which is then used in the next modules. A link contains a rigid transformation between two nodes. In general, memory is divided into Working Memory (WM) and Long-Term Memory (LTM), and after the node is transferred to LTM, it is no longer available in WM. By using this approach, it is possible to stash already processed data into the database and use it to further refine and post-process it. For a loop closure, the RTAB-Map uses a bag-of-words approach, so when a new node is created in STM, then the features are extracted from RGB image and added to a visual word vocabulary, afterward, a node is compared to other nodes in WM. Also, proximity detection is used to localize the nodes close to the current with laser scans (can be seen while mapping when robot is moving backward). Afterward, graph optimization is performed to minimize map errors (g2o [13]).

You can see the results got with both data sets on the fig. 9 and fig. 10.

Figure 9a: RTAB-Map results of the office room

Figure 9b: RTAB-Map results of the office room

Figure 10a: RTAB-Map results of the living room

Figure 10b: RTAB-Map results of the living room

2.4. Pros and cons of the 3D mapping methods

Two different rooms are picked for reconstruction and both intentionally with a large number of visual features. We use Kinect v2 to capture RGB-D images with QHD resolution (960x540), also the camera is properly calibrated. 200 images are used for a living room and 300 — for the office room in RTab-Map and Open3D, and 500 and 1000 images for COLMAP correspondingly.

You can visually see that the texture and color of an office room is better transferred and mapped using COLMAP (Figure 5) but the structure is slightly better with RTAB-Map (fig. 9), although, a living room has different results and COLMAP gives better results on color (fig. 6b). But there may be lots of errors in structure (fig. 6a), while Open3D and RTAB-Map give almost the same results respectively (fig. 8, 10).

All three approaches have their own advantages. RTAB-Map is flexible in terms of hardware and use while giving fast and rough results on large rooms and even buildings, but the downside is sensitivity to computation powers and sensors quality (especially odometry and IMU). In general, if the hardware is tuned correctly, then, we can see nice results. COLMAP does not need anything except for a set of images to start processing, but a large number of visual features must be seen from multiple points of view to give better results. The same goes for Open3D — it is easy to set up and use, but it is sensitive to parameter tuning. It is worth mentioning that the GPU was not used in this case. Its use together with a more sophisticated camera will significantly increase the quality of mapping.

The pros and cons of the methods above are shortly summarized in table 1.

Table 1: 3D Mapping methods Pros and Cons

3. What is the most efficient way to create a 3D map?

World industrial tendencies are focused on saving people from routine operations. Therefore, all newly implemented processes are introducing robots to increase performance and both users’ and employees’ satisfaction to ease their work. Getting up-to-date environmental maps allows to significantly increase inspection, and, in such a way, the decision capabilities of the operational team.

Taking into account that any type of camera can be mounted directly on UGV, UAV, UUV, etc., both MMS approach and photogrammetry can be performed using robots. Moreover, you can mount the camera on the robotic arm, and it would even further increase the number of MMS’s degrees of freedom and, therefore, would improve the quality of the final map.

The choice between a real-time mapping and an offline mapping influences the hardware that is used. It also changes the operational costs. The photos and video may be processed directly on the robot. It, in turn, requires more computational power and larger batteries that are streamed to a distinct server for processing or stored on the robot for further offline mapping. The final project’s architecture depends on the project’s requirements and should consider the pain points that are being faced.

At the end of the day, we may say that industrial 3D mapping allows a

company to stand out from its competitors now. Although in the future, it

will become just another common tool. 3D environmental maps will be a

regular set of data that everyone would want to have in the greatest numbers possible.

References:

[1] Jantien Stoter, Bruno Vallet, Thomas Lithen, Maria Pla, Piotr Wozniak, Tobias Kellen-berger, Andre Streilein, Thobias Risto Ilves, Hugo Ledoux. “State-of-the-art of 3D national mapping”. In: Special session — SpS 12 — EuroSDR: Innovative technologies and methodologies for NMCAs. (2016).

[2] Byungyun Yang. “ Developing a Mobile Mapping System for 3D GIS and Smart City Planning”. In: Sustainability 2019, 11, 3713. (2019).

[3] Ville V. Lehtola, Harri Kaartinen and Andreas Nuchter, Risto Kaijaluoto, Antero Kukko, Paula Litkey, Eija Honkavaara, Tomi Rosnell, Matti T. Vaaja, Juho-Pekka Virtanen, Matti Kurkela, Aimad El Issaoui, Lingli Zhu, Anttoni Jaakkola, Juha Hyyppa. “ Comparison of the Selected State-Of-The-Art 3D Indoor Scanning and Point Cloud Generation Methods”. In: Remote Sensing, 2017, 9, 796. (2017).

[4] Johannes Lutz Schönberger and Jan-Michael Frahm. “Structure-from-Motion Revisited”. In: Conference on Computer Vision and Pattern Recognition (CVPR). (2016).

[5] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. “Open3D: A Modern Library for 3D Data Processing”. In: arXiv:1801.09847. (2018).

[6] M. Labbé and F. Michaud. “RTAB-Map as an Open-Source Lidar and Visual SLAM Library for Large-Scale and Long-Term Online Operation”. In: Journal of Field Robotics, vol. 36, no. 2, pp. 416–446. (2019).

[7] David G. Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. In: International Journal of Computer Vision 60, 91–110. (2004).

[8] Tinne Tuytelaars, Luc Van Gool Herbert Bay, Andreas Ess, “Speeded-Up Robust Features(SURF)”. In: Computer Vision and Image Understanding, Volume 110, Issue 3.(2008).

[9] Roland Y. Siegwart, Stefan Leutenegger, Margarita Chli, “BRISK: Binary Robust invariant scalable keypoints”. In: ICCV ’11 Proceedings of the 2011 International Conference on Computer Vision, pp.2548–2555. (2011).

[10] R. I. Hartley and P. Sturm. “Triangulation”. In: Computer Vision and Image Understanding, Volume 68, Issue 2.(1997).

[11] I. Hartley A. Fitzgibbon. B. Triggs P. F. McLauchlan. “Bundle Adjustment — A Modern Synthesis”. In: International Workshop on Vision Algorithms, Corfu, Greece. pp.298–372. (2000).

[12] Rudolph Emil Kalman. “A New Approach to Linear Filtering and Prediction Problems”. In: Transactions of the ASME–Journal of Basic Engineering 82.Series D, pp. 35–45. (1960).

[13] g2o — General Graph Optimization link: https://github.com/RainerKuemmerle/g2o