# A Semantically Aware Multi-View 3D Reconstruction Method for Urban Applications
## 3 Results and Discussion
### Semantic-Based SGM  
The dense reconstruction of point clouds can be approached using various methods, such as PatchMatch and SGM. We opted for the SGM method for semantic optimization. The specific computation process of SGM includes the following steps:  
1. Initialization: select the best neighborhood frames for each image based on three criteria--the angle of co-visibility points between two images, the area covered, and the scale similarity. 2. Depth map initialization: initialize a coarse depth map for each image using the Delaunay triangulation method based on sparse point clouds. 3. Perform epipolar rectification on the image pairs and calculate the matching cost per pixel by row using the WZNCC consistency measure. 4. Aggregate the one-dimensional path costs from various directions to approximate the calculation of the optimal two-dimensional disparity. 5. After cost aggregation, find the disparity value with the minimum cost for each pixel. 6. Fuse the three depth maps generated from the three sets of image pairs involved in the calculation for each image. 7. Perform a semantically based fusion on the corresponding dense point clouds calculated for all images. We performed dense reconstruction of Tsinghua University's Old Gate and Life Sciences Building following the steps outlined above, and we compared the results of the original method with those reconstructed using the semantic SGM method. After obtaining the disparity maps of pairwise images reconstructed by means of semantic SGM, in step f, we selected three sets of depth maps reconstructed from adjacent images for each image to be fused. During this process, we fused the images based on the semantic consistency across different images. The result is shown in the Figure 10.  
In step g, it is not sufficient to simply combine the dense point clouds reconstructed from all images, as points from different images may correspond to the same point in real three-dimensional space. Thus, it is necessary to evaluate points that are redundantly present. During the fusion process, there are instances where the same point possesses different semantic attributes in different images. We performed a weighted statistical analysis of the semantics of such a point across various depth  
Figure 10: Dense point cloud reconstruction results: (**a**) the dense point cloud of the Tsinghua Life Sciences Building reconstructed using the original SGM; and (**b**) the dense point cloud of the Life Sciences Building reconstructed using sSGM based on semantics. Yellow points represent buildings, green points represent vegetation, gray points represent static-other, dark blue points represent bicycles, and light blue points represent sky. maps, and we assigned the attribute with the highest weight to the final, fused dense point cloud of the scene. The resulting target dense point cloud is shown in Figure 7.  
The dense point cloud after semantic reconstruction not only possesses coordinates in the XYZ space but also carries semantic properties. We assigned colors based on these semantic attributes: vegetation is colored green, people are magenta, vehicles are blue, buildings are yellow, the sky is blue, and static clutter in the scene is gray. It is evident that the results obtained from our semantic SGM reconstruction exhibit less noise around the edges of objects, such as the edges of building rooftops. Moreover, dense reconstruction based on semantics yields dense point clouds containing semantic information, which is extremely important for scene perception. After fusing a dense point cloud from all views, one can choose whether to optimize the dense point cloud using one of three options: REMOVE_SPECKLES, FILL_GAP, and ADJUST_FILTER. To assess the effectiveness of our semantic optimization, we conducted a comparison of the results before and after optimization. The comparison results for the reconstruction of the Tsinghua Life Sciences Building are shown in Figure 11.  
The above figure shows the reconstruction results for the Tsinghua Life Sciences Building.