# A Semantically Aware Multi-View 3D Reconstruction Method for Urban Applications
## 2 Materials and Methods
### Static and Dynamic Separation-Based SfM  
The method of SfM typically involves extracting feature points from images, solving the external parameters between cameras through feature matching, and computing sparse point clouds. This method poses no problem for fixed scenes, but in urban 3D reconstruction, the presence of numerous continuously moving objects, such as cars, people, animals, and vegetation swaying in the wind, presents challenges. Feature points extracted from these dynamic objects possess unstable characteristics which can introduce errors when solving for the camera's external parameters.  
In earlier works, scholars utilized the frame difference and optical flow methods to obtain information on moving objects in order to optimize the effects of 3D reconstruction. However, the frame difference method is only applicable to cameras with fixed positions, and the optical flow method is limited to sequential images and thus presents certain limitations. With the development of deep learning, some scholars proposed the image masking method [31]. Although image masking can eliminate dynamic targets in a scene, it impacts the extraction of feature points. Therefore, we proposed an SfM based on the separation of dynamic and static elements. Initially, we used high-level semantic information to intelligently determine the attributes of image feature points to enhance the overall accuracy of SfM by filtering out unstable feature points. Then, before proceeding with the SfM calculations, we assessed the semantic consistency of the feature point matching results, thus further improving the precision of feature point matching.  
We performed semantic segmentation on the reconstructed multi-view images using deep learning to obtain the corresponding semantic images as follows:  
\[SemImg_{i}=segmentation(RGB_{i}) \tag{1}\]  
In this formula, \(i=1,2,\ldots,N\), with \(N\) being the total number of images used for reconstruction. We calculated the SIFT feature points for each image and input the semantic attributes of the features during the calculation process:  
\[sFeat_{i},Desc_{i}=SIFT(gray_{i}) \tag{2}\]  
The obtained feature information is as follows:  
\[sFeat_{i},=\left[f_{i}^{1},f_{i}^{2},\ldots,f_{i}^{k},\ldots,f_{i}^{n}\right],\ Desc_{i}=\left[d_{i}^{1},d_{i}^{2},\ldots,d_{i}^{k},\ldots,d_{i}^{n}\right] \tag{3}\]  
In the formula, \(f_{i}^{k}\) is the information of the k-th feature point of image \(i\), and \(d_{i}^{k}\) is the 128-th dimensional descriptor of the k-th feature point of image \(i\), where:  
\[f_{i}^{k}=\left[x,\ y,scale,orientation,semantic\right] \tag{4}\]  
The last item of \(f_{i}^{k}\) is the semantic attribute of the feature point.  
Figure 2 shows the principles of semantic SfM. The first is feature matching based on semantic consistency. During the feature matching process, feature point pairs are evaluated to determine whether they belong to the same semantic category, thereby reducing mismatches. The second approach is to use semantic information to separate dynamic target feature points and reduce the impact of unstable feature points in the scene on calculations.