# Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images
## 2 Related Work  
Right before the renaissance of deep learning, it was a hard task to get a good pixel-wise segmentation map on images. [PERSON], [PERSON] et al. [8] take object-level semantic information to constrain camera extrinsic. Some other methods perform the segmentation directly on the point cloud or meshes, according to their geometric characteristics. [PERSON] et al. [9] and [PERSON] et al. [10] take the random forest classifier to do point segmentation, while [PERSON], [PERSON] et al. [11; 12] and [PERSON] et al. [13] treat it as an energy minimization problem in a Conditional Random Field (CRF). Ray potential (likelihood) is frequently adopted in semantic point cloud generation.  
The flourishing CNN-based semantic segmentation methods are quickly outperforming traditional methods in image semantic segmentation tasks; take, for example, the Fully Convolutional Network (FCN) [1] and Deeplab [3]. High-level computer tasks such as scene understanding and semantic 3D reconstruction are now steady and rudimentary processes. The goal of 3D semantic modeling is to assign a semantic label to each 3D point rather than each 2D pixel. Several learning-based approaches follow the end-to-end manner, analyzing the point cloud and giving segmentation results directly in 3D. Voxel-based methods such as ShapeNets [14] and VoxNet [15] were proposed naturally. Some methods learn a spatial encoding of each point and then aggregate all individual point features to a global point cloud signature [16; 17]. However, current deep learning-based segmentation pipelines cannot handle noisy, large-scale 3D point clouds. Thus, a feasible method is required to firstly perform pixel-wise semantic segmentation on 2D images and then back-project these labels into 3D space using the calibrated cameras to be fused. The methods above handle the point cloud directly, which means they carry a costly computational burden. In other words, they cannot manage large-scale 3D scenes without first partitioning the scene. More than that, because the morphological gap between point clouds in different scenarios is too large. These algorithms may be poorly generalized.  
There are several methods doing semantic segmentation on 2D image and making use of multi-view geometric relationships to project semantic labels into 3D space. For RGBD-based approaches, once good semantic maps of each image are acquired, the semantic point clouds can easily be fused. [PERSON], [PERSON] et al. [18] took advantage of a random forest to classify 2D features to get semantic information, while [PERSON], [PERSON] et al. [19] used FCN with CRF-RNN to perform segmentation on images. [PERSON], [PERSON] et al. [20] and [PERSON], [PERSON] et al. [21] proposed incremental semantic label fusion algorithms to fuse 3D semantic maps. For RGB-based approaches, also addressed as Structure from Motion (SfM) and MVS, each point in the generated 3D structure corresponds to pixels on several images. Following the prediction of 2D labels, the final step is to assign each 3D pixel a semantic label [20; 22]. The refinement process is as essential as the generation process of the semantic point cloud itself. [PERSON], [PERSON] et al. [7] and [PERSON], [PERSON] et al. [23] filter the mismatching by semantic labels of feature points. With the motivation of denoising, [PERSON], [PERSON] et al. [24] proposed a Hough-transform-based algorithm called FC-GHT to detect plane on point cloud for further semantic label optimization. [PERSON], [PERSON] et al. [23] used semantic information as a mask to wipe out the meshes belonging to the semantic class _sky_. These methods have two primary drawbacks. Firstly, they only use the final semantic maps, which means the probabilities of other categories are discarded. Secondly, they contain no global constraints integrated into their algorithms. In response, we propose some ideas for improvement.