###### Abstract
[19] designed a deep network for extracting building footprints, in which the polygon regularization is conducted on the initial result to obtain a rectangle building map. [PERSON] et al. [20] presented a CNN model that combined full-scale skip connections and an edge guidance module to improve the location accuracy of buildings. [PERSON] et al. [21] introduced the active contour model to the CNN, combining remote sensing images and LiDAR data for precise building extraction. However, while most of the above methods need accurate edge information, auxiliary data or sophisticated structures, it is difficult to achieve accurate edges due to poor spatial resolution, spectral similarity, and mixed pixels. Furthermore, CRF-based methods do not sufficiently extract features from the images, lack adequate information propagation, and use post-processing that also lowers model performance. Improving multiscale feature extraction ability is also an important way to enhance building extraction performance. Based on FCN and VGG16, the multiscale decoding network (MSDNet) was designed for the semantic segmentation of images [31]; an inception module, which combines the un-pooling, transposed convolution, and dilated convolution paths, is treated as its decoding part. [PERSON] et al. [32] proposed the GMEDN framework, in which a distilling decoder part is used to extract the multiscale features for precise prediction results. [PERSON] et al. [33] provided the UNet-AP model for accurate building footprint extraction from very-high resolution remote sensing satellite imagery. In which atrous convolution is applied to feature extraction for enhancing the representation of objects at different scales in the image. The USPP framework was provided for building segmentation in high-resolution remote sensing [34], which enables extraction of features at multiple spatial scales and at the same time up-samples the feature maps to learn global contextual information by incorporating a spatial pyramid pooling module. These models try to extract and fuse the multiscale features in the network; however, to reduce the complexity of the model or improve its efficiency, the multiscale feature extraction modules are mainly implemented in the decoding part of the network. Thus, extraction of the multiscale features is not enough, for example, lacking refinement of the feature maps extracted by the network to reduce negative feature information, the fusion of deep and shallow features in the encoding stage, the direct multiscale features from the input images, etc. In order to fully utilize different-level feature maps, residual networks or skip-layer connections are used to fuse the shallow layers and deep layers in some proposed models. For example, the dilated convolutions are used to increase the receptive field instead of using pooling layers [22; 23; 24]. However, dilated convolution-based models must perform convolutional operations on high-resolution feature maps through the entire network, which makes such models difficult to train and computationally expensive. Another important approach is to use skip connections, such as U-Net [25], UNet++ [26], SegNet [27], etc. They directly connect a convolutional layer in the encoder and a corresponding layer in the decoder. Thus, the fusion of the feature maps of earlier layers and the discriminative feature maps up-sampled at the end of the encoder step can obviously refine the segmentation map. Therefore, the skip-connections approach is widely used in encoder-decoder models for improving the performance of remote sensing classification [35; 36]. [PERSON] et al. [37] proposed the ResUNet-a model, in which U-Net was taken as the encoder/decoder backbone, and the pyramid scene parsing pooling module, residual connections module and multi-tasking module were combined. The above methods can perform well in most cases, but while these networks are able to improve the overall segmentation results, the boundaries between two different semantic classes often can not be well defined.