# A Local Enhancement Method for Large-Scale Building Facade Depth Images using Densely Matched Point Clouds

Authors: Jiao Guo; Ke Li; Hao Xu
Journal: Advances in Multimedia
 (2022)

---

## Abstract

In recent years, laser scanning systems have been widely used to acquire multi-level three-dimensional spatial objects in real time. The laser scanning system is used to acquire the three-dimensional point cloud data of urban scenes. Due to the large-scale characteristics of urban scenes, and the problems of scanning occlusion, scanning path, and limited scanning laser range, the laser scanning system cannot scan every object in the scene comprehensively, multidirectionally and finely, so the corresponding three-dimensional point cloud data collected by many objects are incomplete, and the data images are relatively sparse and unevenly distributed. The existing point cloud denoising and enhancement algorithms, such as AMLS, RMLS, LOP, and WLOP, all use local information to enhance the missing or sparse parts of the point cloud. This point cloud enhancement method is only limited to a small range and cannot do anything for the larger missing area of the point cloud. Even if it is done reluctantly, the effect is not satisfactory. There are a lot of repetitive and similar features in urban buildings, such as the repetitive areas of floors and balconies in buildings. These repetitive areas are distributed in different positions of point clouds, so the repetitive information has non local characteristics. Based on the nonlocal characteristics of building point cloud data and the repetitive structure of buildings, this article proposes a nonlocal point cloud data enhancement algorithm, which organizes the point cloud data in the repeated area into a set of basic geometric elements (planes). The structures are registered in a unified coordinate system, and the point cloud is enhanced and denoised through two denoising processes, “out-of-plane” and “in-plane.”


---

## Body

## 1. Introduction

With the rapid development of technologies such as laser scanning, remote sensing, virtual reality, meta universe, network image library, and the concept of “Digital Earth,” cities around the world are competing to build their own virtual city scenes [1]. Digital Earth is the use of digital technology and methods to organize the spatial and temporal changes of the Earth and its activities and environment, according to the earth’s coordinates, it is stored in computers distributed around the world to form a digital model of the earth. In urban planning, urban monitoring, intelligent transportation, 3D maps, urban cultural heritage protection, real games, virtual battlefields, accident emergency response, pollutant diffusion and other fields, 3D models of large-scale virtual urban scenes can be built [2]. Simulation and simulation play an important role. With the improvement of urban informatization, 3D models of urban scenes will be more widely used [3].The complexity and diversity of urban scenes require effective collection of urban scene shape information and accurate and rapid 3D reconstruction, which is very challenging. At present, the information collection of 3D urban model is still a work with great investment and complicated process. Its technology is the main influencing factor on the formation process and design quality of 3D urban model [4]. How to improve the intelligence of information collection of 3D urban model is the bottleneck of building 3D urban virtual environment. Scene 3D data recovery through pictures and videos is the most efficient means to collect 3D point cloud information [5]. In the same coordinate system, it requires a lot of computation to quickly register images and obtain high-density 3D point clouds through stereo matching. In addition, because of the height self similarity of buildings in urban environment, it is easy to have registration problems. Therefore, how to quickly and efficiently register images and obtain high-density 3D point clouds in the unified coordinate system is the focus of the current discussion. It can be divided according to the characteristics of the 3D laser scanning system and its related performance, as shown in Table 1. This paper distinguishes three methods according to different equipment, operating principle, and effective distance. The specific classification is as follows:Table 1
Classification of 3D laser scanning systems.
3D laser scanning systemDifferent platformsAirborne 3D laser scanning systemGround 3D laser scanning systemPortable 3D laser scanning systemWorking principlePulsed 3D laser scanning systemPhase 3D laser scanning systemImpulse-phase 3D laser scanning systemEffective scanning distanceShort-range 3D laser scannerMid-range 3D laser scannerLong-range 3D laser scannerAviation 3D laser scannerHowever, due to the occlusion between objects and the performance limitations of the data acquisition device, the acquired data often present phenomena such as missing and uneven distribution. The main premise of the next generation model is to process the collected information and obtain comprehensive and balanced information. The final goal of 3D reconstruction is to automatically generate the model. However, due to the particularity of the data itself and the reconstruction process itself, the grid pattern obtained by gridding cannot meet the needs of high accuracy, visualization, and data. How to automatically create a grid model that meets the requirements is a key step to complete the three-D reconstruction of the urban environment [6].The method of extracting 3D images of a scene from a plurality of pictures taken from various perspectives is called multidimensional stereovision [7]. The current multidimensional stereo vision algorithms do not consider some characteristics of man-made objects (e.g., buildings) themselves, and the generated three-dimensional point cloud can be applicated. The current multidimensional stereoscopic vision method cannot consider some characteristics of artificial objects (such as buildings) and the generated 3D point cloud is only determined according to the image matching with multiangle pictures and the smoothness with adjacent images, resulting in sparse. Point clouds generated in places with unclear texture characteristics are missing or inconsistent [8]. Due to the rapid development of laser scanning technology, on-board and on-board laser scanning devices are widely used as 3D data collection in urban living sites [9]. Cloud data are missing and unevenly distributed, which seriously affects the next model generation. In view of the above problems, this article studies the use of building self-similarity to complete, enhance, and denoise LiDAR point clouds. These problems are currently difficult and hot issues in the field of computer vision and computer graphics, and there are important theoretical issues to study these problems.


## 2. State of the Art

### 2.1. Overview of 3D Reconstruction of Buildings

The final grid module must go through many stages such as 3D information collection, 3D data processing and 3D reconstruction. The information obtained can be mainly divided into long-distance information (satellite images, aerial images, and airborne laser scanning data) and short-range data (short-range photogrammetry data, short-range laser scanning data, and manual measurement data). The 3D data processing refers to the work data processing by processing the obtained 3D information [10]. For example, the shielding of the scanning point cloud data information by the on-board laser, the limitation of the laser range and the scanning area and other reasons cause the missing and uneven distribution of the 3D point cloud data. Before taking the newly acquired point cloud data as the next generation model, it can be enhanced and denoising [11]. Buildings, roads, trees, and other objects must be gridded to form the surface mesh of the object when carrying out the 3D point cloud. Since the 3D discrete point cloud technology cannot be accurate according to the surface structure of the object, the mesh modeling technology can not only clearly see the surface contour of the building but also carry out the next step of image generation, background rendering, and other operations. In the virtual reality and simulation design of building space, Zhou et al. build the mathematical model of building location and parameters and expand the mathematical model of the target building space based on the mathematical model of the building space. The results show that the method has good point and line rendering effect and can obtain more realistic gym simulation design results based on 3D virtual building [12]. Grid representation refers to using the position and normal vector information of 3D points to generate polygons (usually represented by triangular patches) to form the surface of an object [13]. Grid-based description has become a classic technology in 3D engineering. Figure 1 shows the flowchart of 3D reconstruction of urban virtual scene, which mainly includes three parts: 3D point cloud data acquisition, data processing, and 3D scene reconstruction [14].Figure 1
Flowchart of 3D reconstruction of urban building scene.


### 2.2. Overview of 3D Point Cloud Data Acquisition

The complexity and diversity of urban environment makes it challenging to effectively collect scene shape information [15]. The collection of environmental characteristic information is an important step in the reconstruction of the 3D urban scene, and its technology is also the main influencing factor that directly affects the formation and design quality of the 3D urban model. The main information sources of 3D urban model include (1) 3D point cloud data obtained by static laser scanning equipment; (2) 3D point cloud data generated based on 2D data (such as aerial photos, satellite, and ground images); (3) The on-board point cloud information directly obtained by the airborne laser scanning device and the 3D point cloud information. Static laser scanning equipment refers to fixing the equipment in one place, adjusting the posture of the equipment, scanning part of the shape of the scanned object, then moving the scanning equipment to scan other parts, and finally passing the position information of the equipment and the scanned 3D point cloud from the 3D point clouds scanned at different positions that are registered in a unified coordinate system. The 3D point cloud obtained by the static laser scanner has extremely high accuracy, but due to its complex use and low degree of automation, it is mostly used for high-precision 3D data acquisition and accurate modeling. Table 2 presents the comparison of the CPU running time of the PSUR algorithm and the incremental SfM algorithm [16].Table 2
Performance comparison between PSUR algorithm and incremental SfM algorithm.
Test data#imgs.view (m)f.graph (s)SfM (m)Total (m)RIAD (m)4-sided (a)44652400119178673Hotel (b)2352725481113751Kinder. (c)2753226889126713Dorn (d)64575645125209128212-sided (e)594695451582361476MedSchool (f)44852294138195135592-buildings (g)49458633The abbreviation of motion structure is SFM, which mainly refers to the method of finding out the corresponding points on the image sequences of several different dimensions, and calculating the position of 3D nodes and camera parameters in the image through the corresponding points [17]. The theoretical cornerstone of SFM is the basic principle of perspective projection geometry, that is, to establish the world relationship of 3D and 2D by introducing perspective projection mode [18]. Assuming that there is a point X in the three-dimensional space, and its two-dimensional corresponding pixel on the image I is x, then there is a perspective projection matrix.(1)P=KR|t.where K represents the internal parameters of the camera, R represents the orientation of the camera, and t represents the position of the camera so that the projection formula is established.(2)x=PX.The basic principle of SfM is to extract matching two-dimensional feature points from images taken at different angles, namelyx1, x2,…(matching points on different images), and calculate the two-dimensional feature points using the two-dimensional corresponding matching points and the projection formula [19]. The 3D signals of the matching points including the camera essence, position, and distance of each image are shown in Figure 2.Figure 2
Schematic diagram of SfM.


### 2.3. Lidar Scanning Point Cloud Acquisition

Vehicle and airborne LiDAR (Ranging) is a new measurement technology obtained by interdisciplinary research in recent years. It is based on laser detection method, electronic computer, precision moving carrier attitude detection method, and precision moving [20]. The rapid development of GPS differential positioning technology provides a more powerful scientific and technical guarantee for the acquisition of large-scale urban scene 3D modeling data [21]. It can quickly, in real time, and automatically acquire dense 3D point cloud information in the measurement area and is a new and fast 3D point cloud data acquisition method. Compared with other traditional measurement methods, the vehicle-mounted and airborne lidar scanning system has the characteristics of a high degree of automation, short data production cycle, low influence by weather, and high measurement accuracy. It is the most advanced real-time 3D data information acquisition system.Now, there are also new laser scanning control systems in the market, such as the 3D laser test vehicle lynx of the UK Optech Co., Ltd. and the streetmapper of the UK 3D laser test Co., Ltd. As shown in Figure3, this laser scanning control system is installed on the roof of the car, which can scan the scene on both sides of the street at high speed under normal speed, and can directly obtain 360-degree 3D point cloud data. Meanwhile, the attached HD camera can also obtain point cloud data synchronously. The corresponding texture map is very suitable for obtaining 3D graphics and surface texture data of larger urban scenes. The 3D point cloud data of buildings used in this paper is obtained from the lynx test.Figure 3
Schematic diagram of 3D laser scanning.


## 2.1. Overview of 3D Reconstruction of Buildings

The final grid module must go through many stages such as 3D information collection, 3D data processing and 3D reconstruction. The information obtained can be mainly divided into long-distance information (satellite images, aerial images, and airborne laser scanning data) and short-range data (short-range photogrammetry data, short-range laser scanning data, and manual measurement data). The 3D data processing refers to the work data processing by processing the obtained 3D information [10]. For example, the shielding of the scanning point cloud data information by the on-board laser, the limitation of the laser range and the scanning area and other reasons cause the missing and uneven distribution of the 3D point cloud data. Before taking the newly acquired point cloud data as the next generation model, it can be enhanced and denoising [11]. Buildings, roads, trees, and other objects must be gridded to form the surface mesh of the object when carrying out the 3D point cloud. Since the 3D discrete point cloud technology cannot be accurate according to the surface structure of the object, the mesh modeling technology can not only clearly see the surface contour of the building but also carry out the next step of image generation, background rendering, and other operations. In the virtual reality and simulation design of building space, Zhou et al. build the mathematical model of building location and parameters and expand the mathematical model of the target building space based on the mathematical model of the building space. The results show that the method has good point and line rendering effect and can obtain more realistic gym simulation design results based on 3D virtual building [12]. Grid representation refers to using the position and normal vector information of 3D points to generate polygons (usually represented by triangular patches) to form the surface of an object [13]. Grid-based description has become a classic technology in 3D engineering. Figure 1 shows the flowchart of 3D reconstruction of urban virtual scene, which mainly includes three parts: 3D point cloud data acquisition, data processing, and 3D scene reconstruction [14].Figure 1
Flowchart of 3D reconstruction of urban building scene.


## 2.2. Overview of 3D Point Cloud Data Acquisition

The complexity and diversity of urban environment makes it challenging to effectively collect scene shape information [15]. The collection of environmental characteristic information is an important step in the reconstruction of the 3D urban scene, and its technology is also the main influencing factor that directly affects the formation and design quality of the 3D urban model. The main information sources of 3D urban model include (1) 3D point cloud data obtained by static laser scanning equipment; (2) 3D point cloud data generated based on 2D data (such as aerial photos, satellite, and ground images); (3) The on-board point cloud information directly obtained by the airborne laser scanning device and the 3D point cloud information. Static laser scanning equipment refers to fixing the equipment in one place, adjusting the posture of the equipment, scanning part of the shape of the scanned object, then moving the scanning equipment to scan other parts, and finally passing the position information of the equipment and the scanned 3D point cloud from the 3D point clouds scanned at different positions that are registered in a unified coordinate system. The 3D point cloud obtained by the static laser scanner has extremely high accuracy, but due to its complex use and low degree of automation, it is mostly used for high-precision 3D data acquisition and accurate modeling. Table 2 presents the comparison of the CPU running time of the PSUR algorithm and the incremental SfM algorithm [16].Table 2
Performance comparison between PSUR algorithm and incremental SfM algorithm.
Test data#imgs.view (m)f.graph (s)SfM (m)Total (m)RIAD (m)4-sided (a)44652400119178673Hotel (b)2352725481113751Kinder. (c)2753226889126713Dorn (d)64575645125209128212-sided (e)594695451582361476MedSchool (f)44852294138195135592-buildings (g)49458633The abbreviation of motion structure is SFM, which mainly refers to the method of finding out the corresponding points on the image sequences of several different dimensions, and calculating the position of 3D nodes and camera parameters in the image through the corresponding points [17]. The theoretical cornerstone of SFM is the basic principle of perspective projection geometry, that is, to establish the world relationship of 3D and 2D by introducing perspective projection mode [18]. Assuming that there is a point X in the three-dimensional space, and its two-dimensional corresponding pixel on the image I is x, then there is a perspective projection matrix.(1)P=KR|t.where K represents the internal parameters of the camera, R represents the orientation of the camera, and t represents the position of the camera so that the projection formula is established.(2)x=PX.The basic principle of SfM is to extract matching two-dimensional feature points from images taken at different angles, namelyx1, x2,…(matching points on different images), and calculate the two-dimensional feature points using the two-dimensional corresponding matching points and the projection formula [19]. The 3D signals of the matching points including the camera essence, position, and distance of each image are shown in Figure 2.Figure 2
Schematic diagram of SfM.


## 2.3. Lidar Scanning Point Cloud Acquisition

Vehicle and airborne LiDAR (Ranging) is a new measurement technology obtained by interdisciplinary research in recent years. It is based on laser detection method, electronic computer, precision moving carrier attitude detection method, and precision moving [20]. The rapid development of GPS differential positioning technology provides a more powerful scientific and technical guarantee for the acquisition of large-scale urban scene 3D modeling data [21]. It can quickly, in real time, and automatically acquire dense 3D point cloud information in the measurement area and is a new and fast 3D point cloud data acquisition method. Compared with other traditional measurement methods, the vehicle-mounted and airborne lidar scanning system has the characteristics of a high degree of automation, short data production cycle, low influence by weather, and high measurement accuracy. It is the most advanced real-time 3D data information acquisition system.Now, there are also new laser scanning control systems in the market, such as the 3D laser test vehicle lynx of the UK Optech Co., Ltd. and the streetmapper of the UK 3D laser test Co., Ltd. As shown in Figure3, this laser scanning control system is installed on the roof of the car, which can scan the scene on both sides of the street at high speed under normal speed, and can directly obtain 360-degree 3D point cloud data. Meanwhile, the attached HD camera can also obtain point cloud data synchronously. The corresponding texture map is very suitable for obtaining 3D graphics and surface texture data of larger urban scenes. The 3D point cloud data of buildings used in this paper is obtained from the lynx test.Figure 3
Schematic diagram of 3D laser scanning.


## 3. Local Point Cloud Enhancement Technology

### 3.1. Plane Extraction

In the first step of the algorithm, using the RANSAC algorithm, each repeating region is divided into a set of planes, and at the same time, the normal vector of the plane is assigned to each 3D point belonging to the plane. After obtaining this set of planes, a filter is performed to remove the planes containing a small number of point clouds from the plane set, using 50 as the threshold in the experiment. The most robust way to describe line segments is the calculation formula in weighting, where the weighted confidence is the average width of each line segment. Here, line segments can be replaced by planes, and how to determine the confidence level of each plane will be described. Each plane Pi will belong to planePi.The set of three-dimensional points of is denoted as {p1, p2, …}. Here, we use their scales to describe the confidence of these points as planes, denoted ϕ(Pi). So, the confidence of a plane is a combination of these three factors:(3)ωPi=σPi⋅ηPi⋅ϕPi.Because the point cloud is unordered, the loss function designed must be able to avoid the uncertainty of the result caused by the disorder of the point cloud. Next, two loss functions are introduced: Chamfer distance (CD) and Earth mover’s distance (EMD).CD is defined as follows:(4)dChamferX^p,Xp=∑x∈X^Pminy∈Xpx−y22+∑y∈X^Pminx∈Xpx−y22,where Xp is the real distributed point cloud. For each predicted point, the point in Xp that has the smallest distance from this point is found, then the smallest distance is calculated, and this is done for all predicted points. In turn, according to each point in Xp, the minimum distance is found from this point in the predicted point cloud, this operation is performed on all real points, and the distances are added up, and the final distance obtained is the Chamfer. (Note: Although this algorithm is simple and fast to calculate, it is obvious that the Chamfer distance algorithm cannot guarantee the consistency between the two point cloud distributions.)EMD is defined as follows:(5)dEMDX^p,Xp=minϕ:Xp⟶Xp∑x∈X^px−ϕx2.EMD can alleviate the problem brought by CD, where Ø is a bijection, since it can make one point set map to another point set, which can ensure the consistency between the two point sets, so EMD has a significant impact. In view of this, the article uses CD and EMD in stages. First, after the sparse point cloud is generated for the first time, the use of EMD can keep the generated point cloud and the real point cloud with the same distribution and can reduce the amount of calculation. Then all subsequent dense point clouds use CD to calculate the loss.


### 3.2. Triangulation Model

The triangulation principle of the heterogeneous stereo vision system is the same as that of the isomorphic stereo vision system, both of which use the parallax of two images to measure the distance of a certain point according to the triangular model. The imaging principle of the stereo vision system is similar to that of the human eye. Humans are able to perceive the distance of an object due to the difference in imaging a certain point between the two eyes. The further apart the points are, the smaller the difference. The greater the distance of the points, the greater this difference, which is often referred to as parallax.As shown in Figure4, P is a certain point in the environment to be tested, and OR and OT are the optical centers of the two cameras, respectively. The imaging points of point P on the two camera sensors are P1 and P2, respectively, f is the focal length, B is the baseline distance, XR and XT are the abscissa of the ordinary image and the abscissa of the panoramic image, respectively, and Z is the depth distance to be solved. Let the distance between P1 and P2 be D, then we have(6)D=B−XR−XT.Figure 4
Schematic diagram of triangulation.According to the principle of similar triangles, we have(7)B−XR−XTB=Z−fZ.where(8)Z=fBXR−XT.where the focal length f can be obtained by single-target positioning of the camera, and the baseline distance B can be obtained by manual measurement, so as long as the depth information Z can be obtained by obtaining the parallax XR-XT.


### 3.3. Plane Clustering

In this step, similar planes are grouped together. The similarity measure uses the plane parameter equation, and the L2-distance of the plane parameter is used as the similarity measure of the two planes. The planeP can be expressed as a parametric equation:(9)n⟶⋅p+d=0,of which (10)n⟶=nx,ny,nz,represents the normal vector of the plane and(11)n⟶=1,p represents a point on the plane P and d represents the distance from the origin to the plane. For a plane, its parameters can be used to represent, namely (nx, ny, d), which represents a plane as a three-dimensional point. The distance between two planes is calculated using the parameter l two distances. Planar clustering can be transformed into a clustering problem on three-dimensional points. However, if there is no point cloud in some overlapping areas, or even the point cloud distribution on some planes is inconsistent, if the same calculation is made for each plane, some planes below the confidence level will also change the clustering results. The plane will affect the clustering results.For the stability of clustering, a sorted clustering process is introduced here, and planes are added to each class one by one. The planes are ordered in all repeat regions according to confidence. According to the sorting, each time a plane with the highest confidence is taken from the sequence and placed into the existing class. If the L2-distance of the plane to all planes in the existing class is greater than a threshold, a new class is generated. Finally, some classes that contain fewer planes are discarded.


### 3.4. Node Merge

In this step, the wall-directed graph is simplified by selecting two redundant nodesvi,vj to merge (Figure 5). Redundant nodes contain wall images of the same wall segmented in different images. To merge two redundant nodes vi,vj, the following two conditions must be met: (1) The wall images in vi,vj have representational similarity, which is measured using the L2-distance of the GIST global descriptor, called GIST similarity; (2) Two nodes vi,vj are subsequent nodes of the same node, or vj have the same subsequent nodes, and the similarity measure is called sequential similarity.Figure 5
Error edges introduced by occlusion.The sequential similarity is defined by the weights and angles of the edges of the two nodes to the target nodevi,vj. The metric function that fuses GIST similarity and order similarity is defined as follows:(12)Si,j=NiNjNi+Nj,ifNi≥v,Nj≥v,Ai,j<ηdi,j<ε,0,otherwise.where d (i, j) represents the L2-distance of the GIST global descriptor; Ni and Nj represent the edge weights between the node vi,vj and the target node, respectively; A(i, j) represents the two nodes with the angle difference between the target nodes. First, only nodes that satisfy d(i, j) less than ε and A(i, j) less than η can be merged; second, vi,vj where both Ni and Nj are relatively large are candidate node pairs for merging. The parameters ν, ε, and η were used in the experiments as 2, 0.8, and 5°, respectively. Figure 6 shows the two steps of node merging. The two values on the edge in the figure represent the angle and weight, respectively. Nodes A and E are selected to merge (middle image), and nodes B and D are selected to merge (right image).Figure 6
Two processes of node merging.


## 3.1. Plane Extraction

In the first step of the algorithm, using the RANSAC algorithm, each repeating region is divided into a set of planes, and at the same time, the normal vector of the plane is assigned to each 3D point belonging to the plane. After obtaining this set of planes, a filter is performed to remove the planes containing a small number of point clouds from the plane set, using 50 as the threshold in the experiment. The most robust way to describe line segments is the calculation formula in weighting, where the weighted confidence is the average width of each line segment. Here, line segments can be replaced by planes, and how to determine the confidence level of each plane will be described. Each plane Pi will belong to planePi.The set of three-dimensional points of is denoted as {p1, p2, …}. Here, we use their scales to describe the confidence of these points as planes, denoted ϕ(Pi). So, the confidence of a plane is a combination of these three factors:(3)ωPi=σPi⋅ηPi⋅ϕPi.Because the point cloud is unordered, the loss function designed must be able to avoid the uncertainty of the result caused by the disorder of the point cloud. Next, two loss functions are introduced: Chamfer distance (CD) and Earth mover’s distance (EMD).CD is defined as follows:(4)dChamferX^p,Xp=∑x∈X^Pminy∈Xpx−y22+∑y∈X^Pminx∈Xpx−y22,where Xp is the real distributed point cloud. For each predicted point, the point in Xp that has the smallest distance from this point is found, then the smallest distance is calculated, and this is done for all predicted points. In turn, according to each point in Xp, the minimum distance is found from this point in the predicted point cloud, this operation is performed on all real points, and the distances are added up, and the final distance obtained is the Chamfer. (Note: Although this algorithm is simple and fast to calculate, it is obvious that the Chamfer distance algorithm cannot guarantee the consistency between the two point cloud distributions.)EMD is defined as follows:(5)dEMDX^p,Xp=minϕ:Xp⟶Xp∑x∈X^px−ϕx2.EMD can alleviate the problem brought by CD, where Ø is a bijection, since it can make one point set map to another point set, which can ensure the consistency between the two point sets, so EMD has a significant impact. In view of this, the article uses CD and EMD in stages. First, after the sparse point cloud is generated for the first time, the use of EMD can keep the generated point cloud and the real point cloud with the same distribution and can reduce the amount of calculation. Then all subsequent dense point clouds use CD to calculate the loss.


## 3.2. Triangulation Model

The triangulation principle of the heterogeneous stereo vision system is the same as that of the isomorphic stereo vision system, both of which use the parallax of two images to measure the distance of a certain point according to the triangular model. The imaging principle of the stereo vision system is similar to that of the human eye. Humans are able to perceive the distance of an object due to the difference in imaging a certain point between the two eyes. The further apart the points are, the smaller the difference. The greater the distance of the points, the greater this difference, which is often referred to as parallax.As shown in Figure4, P is a certain point in the environment to be tested, and OR and OT are the optical centers of the two cameras, respectively. The imaging points of point P on the two camera sensors are P1 and P2, respectively, f is the focal length, B is the baseline distance, XR and XT are the abscissa of the ordinary image and the abscissa of the panoramic image, respectively, and Z is the depth distance to be solved. Let the distance between P1 and P2 be D, then we have(6)D=B−XR−XT.Figure 4
Schematic diagram of triangulation.According to the principle of similar triangles, we have(7)B−XR−XTB=Z−fZ.where(8)Z=fBXR−XT.where the focal length f can be obtained by single-target positioning of the camera, and the baseline distance B can be obtained by manual measurement, so as long as the depth information Z can be obtained by obtaining the parallax XR-XT.


## 3.3. Plane Clustering

In this step, similar planes are grouped together. The similarity measure uses the plane parameter equation, and the L2-distance of the plane parameter is used as the similarity measure of the two planes. The planeP can be expressed as a parametric equation:(9)n⟶⋅p+d=0,of which (10)n⟶=nx,ny,nz,represents the normal vector of the plane and(11)n⟶=1,p represents a point on the plane P and d represents the distance from the origin to the plane. For a plane, its parameters can be used to represent, namely (nx, ny, d), which represents a plane as a three-dimensional point. The distance between two planes is calculated using the parameter l two distances. Planar clustering can be transformed into a clustering problem on three-dimensional points. However, if there is no point cloud in some overlapping areas, or even the point cloud distribution on some planes is inconsistent, if the same calculation is made for each plane, some planes below the confidence level will also change the clustering results. The plane will affect the clustering results.For the stability of clustering, a sorted clustering process is introduced here, and planes are added to each class one by one. The planes are ordered in all repeat regions according to confidence. According to the sorting, each time a plane with the highest confidence is taken from the sequence and placed into the existing class. If the L2-distance of the plane to all planes in the existing class is greater than a threshold, a new class is generated. Finally, some classes that contain fewer planes are discarded.


## 3.4. Node Merge

In this step, the wall-directed graph is simplified by selecting two redundant nodesvi,vj to merge (Figure 5). Redundant nodes contain wall images of the same wall segmented in different images. To merge two redundant nodes vi,vj, the following two conditions must be met: (1) The wall images in vi,vj have representational similarity, which is measured using the L2-distance of the GIST global descriptor, called GIST similarity; (2) Two nodes vi,vj are subsequent nodes of the same node, or vj have the same subsequent nodes, and the similarity measure is called sequential similarity.Figure 5
Error edges introduced by occlusion.The sequential similarity is defined by the weights and angles of the edges of the two nodes to the target nodevi,vj. The metric function that fuses GIST similarity and order similarity is defined as follows:(12)Si,j=NiNjNi+Nj,ifNi≥v,Nj≥v,Ai,j<ηdi,j<ε,0,otherwise.where d (i, j) represents the L2-distance of the GIST global descriptor; Ni and Nj represent the edge weights between the node vi,vj and the target node, respectively; A(i, j) represents the two nodes with the angle difference between the target nodes. First, only nodes that satisfy d(i, j) less than ε and A(i, j) less than η can be merged; second, vi,vj where both Ni and Nj are relatively large are candidate node pairs for merging. The parameters ν, ε, and η were used in the experiments as 2, 0.8, and 5°, respectively. Figure 6 shows the two steps of node merging. The two values on the edge in the figure represent the angle and weight, respectively. Nodes A and E are selected to merge (middle image), and nodes B and D are selected to merge (right image).Figure 6
Two processes of node merging.


## 4. Experimental Results and Analysis

In this section, we test five real data, namely Simple wall, Detail, Tall, Cylinder, and Fat. These five sets of data have different characteristics. The simple wall is mainly a simple building wall composed of planes. Detailed drawing of building wall with balcony was performed, with detailed structural features. The point cloud is sparse or missing due to occlusion. A cylinder is a building containing a cylinder. Fat is a typical modern residential building. Before testing the real data set, we used the synthesized 2D data to test the influence of the number of repeated regions and the noise level of 3D points on the Nola algorithm. Cylinder is a building containing cylinders. Fat is a typical modern residential building. Before testing on the real dataset, we use a synthetic 2D data to test the effect of the number of repeat regions and the noise level of 3D points on the NOLA algorithm.The experimental group of “processing result parameters of the algorithm in the article” conducted the initial registration and fine registration stages on the mean square error and time-consuming performance indicators of the classic SAC-IA algorithm, the algorithm in the literature, and the improved ICP algorithm. The relevant performance comparison parameter information is shown in Table3. Among them, the literature in the initial registration stage uses the classic SAC-IA algorithm, so the accuracy and time results of their initial registration are exactly the same.Table 3
Summary of the parameters of the algorithm processing results in this article.
Point cloud group nameInitial registrationFine registrationTotal timeClassic SAC-IA algorithmLiterature algorithmsImproved SAC-IA algorithm of this articleClassic ICP algorithmLiterature algorithmsImproved ICP algorithm of this articleMean squared errorTime consumptionMean squared errorTime consumptionMean squared errorTime consumptionMean squared errorTime consumptionMean squared errorTime consumptionMean squared errorTime consumptionClassic SAC-IA algorithm + ICP algorithmLiterature algorithmsCombining improved SAC-IAalgorithm and ICP algorithmArmadillo4.6 × 10−4684.6 × 10−4682.1 × 10−4314.9 × 10−5463.1 × 10−5221.6 × 10−6151149049Goddess6.1 × 10−4853.2 × 10−4853.5 × 10−4553.2 × 10−5672.1 × 10−5371.1 × 10−62215212277Gate9.1 × 10−41969.1 × 10−41965.5 × 10−4979.7 × 10−51516.6 × 10−5743.1 × 10−652347270149Through a comprehensive analysis of the relevant parameters of the algorithm processing results in the text from the experimental results in Table3, it can be known that:(1)
In the initial registration stage, the improved SAC-IA algorithm constructed in this article is better than the classical SAC-IA algorithm in terms of mean square error and calculation time. Compared with the classical SAC-IA algorithm, the IA algorithm has improved the initial registration accuracy of Armadillo data, Goddess data, and Gate data by 54.35%, 42.62%, and 39.56%, respectively; in terms of the time consumption of initial registration calculation, the improved SAC-IA algorithm is more efficient than the classical SAC algorithm. IA algorithm reduces the time consumption of Armadillo, Goddess, and Gate by 54.41%, 35.29%, and 50.51% respectively.(2)
In the fine registration stage, the improved ICP algorithm constructed in this article is significantly improved in mean square error and operation speed compared with the classical ICP algorithm and the literature algorithm: In terms of fine registration accuracy, the improved ICP algorithm of this article is better than the classical ICP algorithm. The algorithm’s improvements in Armadillo data, Goddess data, and Gate data are 96.73%, 96.56%, and 96.8%, respectively, which are 94.84%, 94.76%, and 95.30% higher than those in the literature in Armadillo data, Goddess data, and Gate data. In terms of aspects, the improved ICP algorithm of this article reduces the time consumption of Armadillo, Goddess, and Gate data by 67.39%, 67.16%, and 65.56%, respectively, compared with the classical ICP algorithm, which is decreased by 31.82%, 40.54%, and 29.73%, respectively.The experimental group of “ICP algorithm comparison and evaluation” compared the registration error and registration time-consuming performance of the classic ICP algorithm, the literature algorithm, and the improved ICP algorithm in the fine registration stage. The relevant performance comparison parameters are detailed in Table4.Table 4
Comparison and evaluation of ICP algorithms.
Point cloud group nameClassic ICP algorithmLiterature algorithmsImproved ICP algorithm of this articleRegistration error/mRegistration time/sRegistration error/mRegistration time/sRegistration error/mRegistration time/sArmadillo5.4 × 10−5462.8 × 10−5222.1 × 10−515Goddess4.7 × 10−5671.9 × 10−5371.1 × 10−522Gate4.8 × 10−51513.6 × 10−5743.1 × 10−552Through a comprehensive analysis of the ICP algorithm from the experimental results in Table4, it can be seen that: through the three sets of data of Armadillo, Goddess, and Gate, the classical ICP algorithm, the algorithm in the literature, and the improved ICP algorithm of this article are used for algorithm experiments. The improved ICP algorithm of this article is better than the classic ICP algorithm. The registration errors are increased by 61.11%, 76.6%, and 35.42%, respectively, and the required registration time is reduced by 67.39%, 67.16%, and 65.56%, respectively, and the required registration time is reduced by 31.82%, 40.54% and 29.73%, respectively.In this article, the above algorithm is tested on a PC configured with an intel (R) Core(TM) i7 2.80 GHZ processor and 4G memory. Table5 shows the performance evaluation of the algorithm in this chapter. #pts. represents the number of point clouds of the test data; #queries represents the number of repeated region types; #repet. represents the number of repeated regions of each repeated region type; prep. time represents the preprocessing time, including RANSAC plane extraction, repeated regions detection, etc.; cons. time represents the processing time of point cloud enhancement.Table 5
Performance statistics.
Model# pts# queries# repetPrep. time (s)Cons. timeSimple wall181,486212 | 12121115 sDetail128,55825 | 660160Tall433,32537 | 17 | 2555135 sCylinder1,354,30515[3–32]320215 sFat737,72312[6–8]30099 sThe WLOP is a method that uses local information to strengthen point clouds, establish local operators, and iteratively adjust the size and displacement of normal vectors of each point cloud. In this section, we compare the enhancement results of local point cloud enhancement method (WLOP) and non local point cloud enhancement calculation (NOLA) in constructing 3D point cloud. It can be seen from the conclusion that WLOP cannot preserve some flat and line segment features of the point cloud itself. Therefore, after WLOP is performed, the point cloud range in which line segment features are displayed in the input point cloud is curved. Even if we replace the input data with data synthesized from repeated regions, the results given by WLOP are still suboptimal. The NOLA algorithm uses basic geometric elements as processing objects (planes, cylinders, and line segments) rather than a single 3D point. During the processing, the point cloud maintains the characteristics of the original plane, cylinder, and line segment, so the ideal point cloud enhancement results are obtained.The GFSR algorithm is implemented in the development platform of VS2008.net under Windows system, and the test is completed on a PC using Intel (R) core (TM) i72.80ghz processor and four gigabytes of memory. Table6 shows the performance evaluation of the calculation in this chapter, including the scale data of the grid model. Where PTS. shows the scale of the enhanced node cloud; gram. seg. represents the time required for segmentation according to the syntax principle; model rec. describes the working time spent in the model making stage. The number of vertices in the mode represents the number of vertices of the final mesh model; and model#faces indicates the number of faces in the final mesh model.Table 6
Performance evaluation and grid model scale statistics.
Test data#pts. (k)gram. seg. (s)model rec. (s)model #verticesmodel#facesDetail81574647153887690Tall19196752552762670Synthetic6733191753405042Neat45037627100843922Fat16885172278441150Simple1092431923004042Bar3393470218084


## 5. Conclusion

In recent years, laser scanning systems have been widely used in real-time acquisition of multilevel 3D space targets, especially vehicle-mounted laser scanning systems are used to acquire 3D data in urban scenes. The vehicle-mounted laser scanning system can quickly and accurately obtain 3D point cloud information such as urban buildings, trees, bridges, and roads. The corresponding 3D point cloud data are incomplete or sparse. Especially in urban buildings, due to the high density of buildings, it is easy to block each other, and the trees on the roadside will also block the buildings. Also, due to the limited range of laser scanning equipment, the point clouds of the high-rise parts of the building are sparse or missing. The current local point cloud enhancement algorithm mainly considers the use of neighborhood information to enhance the point cloud. This calculation cannot repair the missing areas of large-scale point clouds. In this method, a nonlocal point cloud method is established by considering the non local (i.e., repeated) region and the high-level semantic information such as the plane cylinder. The algorithm enhances the input point cloud by repeating the three steps of region detection, “out-of-plane” denoising, and “in-plane” denoising. Finally, the full practice proves the accuracy, rationality, and correctness of this method. The practice proves that the method we provide can effectively improve the efficiency of the input point cloud, fill the missing area of the data, and eliminate the impact.



---

*Source: 3175998-2022-08-12.xml*