# A Semantically Aware Multi-View 3D Reconstruction Method for Urban Applications
## 4 Conclusions  
In this study, we addressed the limitations present in the traditional multi-view 3D reconstruction process by proposing a semantic-based optimization method. Our approach was experimentally validated using a dataset comprising complex urban scenarios, which demonstrated the superiority of our semantic optimization. Initially, we selected a dataset comprising urban architectural objects. To acquire a semantic segmentation model for the city, we trained the model using the open-source Cityscapes dataset. As the original dataset's focus differs from our areas of interest, we adjusted its categories by reducing the original 19 categories to 10. Our aim was twofold: firstly, we identified and separated dynamic targets in an urban environment that could significantly affect reconstruction accuracy, and secondly, we distinguished architecturally significant buildings needing detailed reconstruction from other elements in the setting. To ensure the precision of semantic segmentation, we compared three state-of-the-art (sota) methods, including DeepLab V3+, OCRNet, and MaskZformer, for training and conducting validations using both the Cityscapes dataset and the reconstructed dataset. Ultimately, we chose the MaskZformer network, which exhibits the highest accuracy. It achieved a pixel accuracy of 88% in the validation test using the Cityscapes dataset and 85% in the validation test using the reconstructed dataset, thus meeting the requirements for subsequent semantic reconstruction tasks. Following the acquisition of semantic maps corresponding to the images, we optimized the SfM process by segregating static and dynamic elements, while excluding dynamic targets (such as people and vehicles) and semantically inconsistent feature matches. This semantic optimization process generally enhanced the SfM's accuracy. In the reconstruction results of the two scenarios, the overall accuracy improved by 49%. After obtaining more stable camera external parameters, we applied semantic optimization to the dense point cloud reconstruction. We  
Figure 13: Comparison of results of mesh reconstruction of Tsinghua Life Sciences Building: (**a**) reconstruction results using the sSGM method; (**b**) reconstruction results using the original SGM method; and (**c**) reconstruction results using the PatchMatch method. semantically enhanced the classic SGM algorithm for dense reconstruction by semantically weighting a penalty term during cost computation. This improvement allowed for a better identification of depth discontinuities within the scenes. The sSGM method significantly improves the reconstruction accuracy of object boundaries. Through comparison with the ground-truth data from LiDAR scanning, it is evident that our semantically optimized sSGM reconstruction achieves an overall increase of 32.79% in accuracy compared to the original SGM reconstruction. Compared to the other mainstream algorithm PatchMatch, our method's accuracy is 63.06% higher. The performance of the method proposed in this paper depends on the number of categories included in semantic segmentation. A finer semantic segmentation enables a higher accuracy in reconstruction. Another factor that restricts the performance of our method is the influence of different individuals of the same category after semantic segmentation. In future work, we will attempt to classify more detailed semantic segmentation or instance segmentation. Conceptualization, R.W. and H.D. ; methodology, R.W. ; software, D.W.; validation, C.Z. and X.A. ; formal analysis, X.A. ; investigation, X.A. ; resources, H.P. ; data curation, R.W. ; writing--original draft preparation, R.W. ; writing--review and editing, H.D. ; visualization, C.Z. ; supervision, H.P. ; project administration, R.W. ; funding acquisition, H.P. All authors have read and agreed to the published version of the manuscript. This research was funded by the Innovation Program CX-387 of the Shanghai Institute of Technical Physics. Data Availability Statement: Data are contained within the article.