# A Building Segmentation Network Based on Improved Spatial Pyramid in Remote Sensing Images
###### Abstract  
Building segmentation is widely used in urban planning, disaster prevention, human flow monitoring and environmental monitoring. However, due to the complex landscapes and highdensity settlements, automatically characterizing building in the urban village or cities using remote sensing images is very challenging. Inspired by the recent deep learning methods, this paper proposed a novel end-to-end building segmentation network for segmenting buildings from remote sensing images. The network includes two branches: one branch uses Widely Adaptive Spatial Pyramid (WASP) structure to extract multi-scale features, and the other branch uses a deep residual network combined with a sub-pixel up-sampling structure to enhance the detail of building boundaries. We compared our proposed method with three state-of-the-art networks: DeepLaby3+, ENet, ESPNet. Experiments were performed using the publicly available Inria Aerial Image Labelling dataset (Inria aerial dataset) and the Satellite dataset II(East Asia). The results showed that our method outperformed the other networks in the experiments, with Pixel Accuracy reaching 0.8421 and 0.8738, respectively and with mIoU reaching 0.9034 and 0.8936 respectively. Compared with the basic network, it has increased by about 25% or more. It can not only extract building footprints, but also especially small building objects. CNN; semantic segmentation; super resolution; remote sensing; spatial pyramid; ResNet 2021115069 11 April 202126 May 2021  
(such as color and shape). In recent years, there have been more and more researches in this field, and some remarkable achievements have been made. Conventional remote sensing image segmentation methods usually rely on hand-crafted features to extract spatial and texture information of the image, and consider a building as a combination of low-level features, such as spectral information [6], tone [7], texture [8], and geometric shape [9; 10]. Many classic powerful feature extractors can also achieve high efficiency, such as support vector machines (SVMs) [11], multinomial logistic regression [12; 13], boosted decision trees (DTs) [14], neural networks [15; 16; 17], Scale-Invariant Feature Transform (SIFT) [18], Histogram of Oriented Gradient (HOG) [19], and Speeded Up Robust Features (SURF) [20]. These methods do, however, have high requirements for the researchers themselves, and cannot represent the high-level semantic features of the image well. A deep Convolutional Neural Network (CNN) [21] predicts using only features learned from training data, instead of hand-engineered features. Essential to their success is the built-in invariance of deep CNNs to local image transformations, which allows them to learn increasingly abstract feature representations. The challenge of CNN-based semantic segmentation comes from the inner content, shape, and scale variations of the same objects, as well as the easily confused and fine boundaries among different objects. First, the convolution layer has both shift and spatial-invariant characteristics. While invariance is clearly desirable for high-level vision tasks, it may hamper low-level tasks, such as pose estimation and semantic segmentation, in which precise localization is required, rather than the abstraction of spatial details. Second, CNNs are able to capture informative representations with global receptive fields, by stacking convolutional layers and down-sampling; however, the use of down-sampling to compress data is irreversible, resulting in information loss and further causing translation invariance and smooth results. This also leads to inaccurate object boundary acquisition. Third, the multiple scales of objects are also a challenge for CNNs. The features extracted by CNNs usually have a limited receptive field, where the feature mainly describes the core region and largely ignores the context around the boundary, no matter whether the receptive field is large or small.