# Damaged Building Detection with Improved Swin-Unet Model

Authors: Su Xu; Xiping He; Xiaoli Cao; Jian Hu
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

Automatic detection of damaged buildings from satellite remote sensing data has become an urgent problem to rescue planners and military personnel. Unfortunately damaged buildings are in different dimensions and shapes with different roofs depending on the type of the material to be painted. In this study, we present an improved Swin-Unet approach that comprises three main operations. First, improved Swin-Unet as a Unet-like pure Transformer is used for multitemporal image segmentation. Second, different multitemporal features are extracted using hyperspectral image classification algorithm. Finally, a binary change map is generated, and evaluation results are obtained. This article takes AIST building change detection scene as the example, and compared with the conventional approaches tested, overall accuracy, mean intersection over union, and separated Kappa in the proposed method were improved by at least 23.36, 0.1725, and 0.202, respectively. Furthermore, different scenes, such as Gaofen-2/Jilin-1 multitemporal optical images and satellite imagery dataset (xBD), have also come to the same conclusion. Thus, it provides advantageous capabilities for monitoring damaged buildings along coastal areas.


---

## Body

## 1. Introduction

Natural disasters, such as earthquakes, tsunamis, or hurricanes, are inevitable, which may cause catastrophic damage to coastal areas. These buildings tend to bear the brunt of the disaster region. Therefore, it is of utmost importance for rescue planners to automatically detect damaged buildings from satellite remote sensing data. Unfortunately damaged buildings are in different dimensions and shapes with different roofs depending on the type of the material to be painted have. Furthermore, detecting roofs in damaged buildings [1, 2] have related to many factors, such as illumination, cloud cover, satellite observation and imaging conditions, background environment, and side wall occlusion. In the past few decades, researchers have attempted to detect changes in buildings using different tools, such as synthetic aperture radar (SAR) [3], light detection and ranging (LiDAR) [4], spectral information, and high-spatial resolution imagery [5]. However, SAR [3] cannot provide more detailed features than texture, whereas human eyes can. These are often critical to determining whether a building is damaged. Nonetheless, the cost of LiDAR [4] data acquisition is very high that it cannot offer a broad range of monitoring capabilities. At the same time, it is difficult to obtain high-precision results and guarantee the reliability and stability of spectral information method due to many factors such as shadows, occlusions, relief displacement, and spectral variation of buildings. Hence, it is a huge challenge to design an automated method for extracting buildings from remote sensing images, which could achieve high accuracy, especially with increasing spatial resolution. Many building extraction approaches have been studied in the past decade, which can be generally classified into two categories: (1) supervised machine learning approaches, which combine spectral information with morphological indices, and (2) deep learning methods. Geometrical features, such as histogram of oriented gradients (HOGs) [6], spectral/spatial priors [7, 8], shadow/building symbiosis [9], morphological building index [10], scale detection and dome reshaping [11], and segmentation, corner detection, and adaptive windowed Hough transform [12], have a great influence on automatically building extraction. However, irregular (nonrectangular) roof coverings or irregular structural buildings make these methods unavailable. At the same time, the extraction of color, reflectance, and other features often requires manual adjustment, and different buildings have different materials. Therefore, how to better extract the characteristics of the building has become the difficulty of this project. Recently, deep learning [13–19], as the other category, has been widely used for the automatic detection of damaged buildings in color aerial images. One major difference between the two categories concerns threshold selection or training data. A deep convolution neural network based complete framework [14] can be improved using conditional random field. The framework is very good at detecting individual houses, but poor at identifying large buildings because of the locality of convolution operation. An automated building detection framework [15] based on Markov random field ImageNet was proposed by Vakalopoulou et al. However, multiclass learning toward the classification of various classes in single and multitemporal very high-resolution multispectral datasets remains unsettled. To extract buildings holistically, an improved DeconvNet method [16] can achieve an effective performance boost by multisource information fusion. Nevertheless, it usually feels quite helpless in the face of various scales, rich topological appearances, and complex backgrounds. Alshehhi et al. [17] used spatial features of adjacent superpixels to enhance multiclass prediction. This method performs well in the localization of urban objects; however, it requires additional processing to outline boundaries more accurately. An optimized U-Net method (SE-Unet) [18] can effectively strengthen network capability by squeeze-and-excitation modules and be useful in other sample imbalanced situations by dice and cross-entropy loss function. Meanwhile, DTRF-Unet [19] can enhance edge features of buildings and reduce information loss after downsampling through U-Net. However, computing power, model segmentation, and time performance require much improvement. Recently, inspired by Transformer’s great success in the natural language processing domain [20], researchers have tried to bring Transformer into the vision domain [21]. In [22], a vision transformer (ViT) is proposed to perform the image recognition task. Using two-dimensional (2D) image patches with positional embedding as inputs and pertaining on a large dataset, ViT achieved comparable performance with the CNN-based methods. Furthermore, a data-efficient image transformer (DeiT) is presented in [23], indicating that Transformer can be trained on midsize datasets, and a more robust Transformer can be obtained by combining it with the distillation method. In [24], a hierarchical Swin Transformer is developed. The authors of [24] achieved state-of-the-art performance on image classification, object detection, and semantic segmentation by considering Swin Transformer as a vision backbone. The success of ViT, DeiT, and Swin Transformer in image recognition tasks demonstrates the potential for Transformer to be applied in the vision domain. To eliminate the weaknesses, we study Swin Transformer to obtain better results from Swin-Unet [24]. Specifically, hierarchical Swin Transformer [25] with shifted windows is used as the encoder to extract irregular structural building features for hyperspectral image segmentation. However, multitemporal hyperspectral images can be usefully applied to avoid the influence of existing trees, varying sizes, and orientations. Thus, it can fuse hierarchical Swin Transformer [26] and HOG, normalized difference vegetation index, and features from accelerated segment test features to obtain a new optimized method to further improve the precision of building edge detection. This is the intuitive idea of this paper. However, hyperspectral images have a 3D structure, with each point consisting of two spatial coordinates (the position of the pixel) and a spectral coordinate (the corresponding wavelength). It is very tedious for a standard Swin-Unet network structure with segmentation labels in a slice-by-slice manner, resulting in spectral information loss. Thus, an improved Swin-Unet network structure for damaged building change detection using hyperspectral imagery is presented. The main contributions of this study are summarized as follows.
(1)
For the properties of multitemporal images, we introduce a Swin-Unet model by deforming its sampling and pooling patterns to increase the ability to effectively capture different multitemporal features on adaptive regions. Thus, an improved Swin-Unet network (myNet) has been proposed by aggregating the (upsampled) representations from all parallel convolutions(2)
To fit the structure of the hyperspectral image, a new model, called myNet, is developed. It adopts a 3D body as input and carries out the corresponding 3D operation, especially 3D convolution, 3D max-pooling, and 3D upconvolution layer(3)
According to the properties of hyperspectral image segmentation problem, we adopt different geometrical feature to a new optimized method to obtain better evaluation results(4)
The rest of this paper is organized as follows. The details of our proposed method are developed in Section2. Based on the previous analysis, all postulates are confirmed by experimental results in Section 3. Comparative discussions in Section 4 and conclusions in Section 5 are presented, respectively.


## 2. Problem Formulation and Analysis

Hyperspectral imagesχ∈RH×W×D, where D represents the number of bands and H and W denote the width and height in each band, respectively. Figure 1 shows the overall architecture of our proposed model (myNet), comprising two encoders, a bottleneck, one decoder, and eight skip connections. The details of each part are given as follows: In the first stage, multitemporal hyperspectral images are split into nonoverlapping patches with about 4∗4∗D in size by HOG features and is loaded into different encoders separately. Then, the patch feature is projected into sequence embedding in the linear embedding layer from an encoder. The third step of this process is the most important part: feature representation learning. The basic unit of the proposed myNet is the Swin Transformer block used for feature representation and long-range semantic information interactive learning. To construct the bottleneck to learn the deep feature representation, two successive Swin Transformer blocks [26] adopt (shifted) window-based multihead self-attention ((S)W-MSA) modules, respectively, as shown in Figure 1. From the mathematical viewpoint, self-attention is presented as follows:
(1)AttentionQ,K,V=softmaxQKTd+BV,where Q,K,V∈RM2×d denote the query, key, and value matrices and B comes from the bias matrix B^∈R2M−1×2M+1. Here M2 and d represent the number of patches in a window and the dimension of the query or key, respectively. If shifted windows, LayerNorm, multihead self-attention (MSA), and multilayer perceptron (MLP) keep work, two successive Swin Transformer blocks [27] can be expressed as follows:
(2)z∧l=W−MSALNzl−1+zl−1,(3)zl=MLPLNz∧l+z∧l,(4)z∧l+1=SW−MSALNzl+zl,(5)zl+1=MLPLNz∧l+1+z∧l+1,wherez∧l and zl represent the outputs of the (S)W-MSA and MLP modules of the lth block, respectively. To reduce the loss of spatial information caused by downsampling, skip connection is used to fuse these learning features from the Swin Transformer block with these upsampled features. The final step is the patch merging layer in the decoder. Extracted deep features were upsampled. Difference information between multitemporal images was retained.Figure 1
Improved Swin-Unet network structure.


## 3. Results

To evaluate the effectiveness of the proposed approach, the most widely used remote sensing image (RS) datasets for building damage assessment, such as Gaofen-2/Jilin-1 multitemporal optical images, satellite imagery dataset (xBD), and AIST building change detection (ABCD) scene, were collected for model evaluation, covering different areas from multitemporal imaging modalities. For comparison, various kinds of architectures (e.g., SegNet, U-Net, SE-Unet, and DTRF-Unet) and assessment indices (e.g., mean intersection over union (mIoU) and separated Kappa (SeK)) are applied with minibatches of size 4. Mathematically, assessment indices (the higher, the better) can be defined as follows:(6)mIoU=0.5∗TNTN+FP+FN+0.5∗TPTP+FN+FP,(7)SeK=Kappa∗e0.5∗TP/TP+FN+FP−1,where Kappa coefficient (Kappa), true negative (TN), true positive (TP), false negative (FN), false positive (FP), and overall accuracy (OA) are the most commonly used scientific measurements of scene change detection algorithm. All system specifications and simulation tools are shown in Table 1. And these experiment parameters are shown in Table 2.Table 1
System specifications and simulation tools.
HardwareConfigurationCPUIntel (R) Core (TM) i7-4790 K 4.00 HzGPUNVIDIA GEFORCE GTX 1080RAM16 GBHard diskToshiba SSD 5126SystemUbuntu 16.04Software packagePython 3.7.3+ tensorflow1.13.2 + Numpy1.17.4+ keras2.1.5 + h5py2.10.0Support libraryCUDA10.2 + cudnn7.6Table 2
Experimental parameters.
Tuning parametersValuesEpochs160Iterations2000Minibatches2Input image size256×256Patch size24Initial learning rate0.001Dropout rate0.5Eval size0.1Momentum0.9Weight decay1.00E-04


### 3.1. Multitemporal Optical Images

The multitemporal optical images [27] were derived from the fifth group of the 2021 Gaofen Challenge. Gaofen-2 and Jilin-1 provide 4 m multispectral image files with a spatial resolution that can reach 0.8 m. Figure 2 shows all multitemporal images mark changed buildings. Each pair of image files is six channels in tiff format, whereas annotation files are single-channel in png format.Figure 2
Multitemporal optical images: (a) before changed, (b) after changed, and (c) change annotation file.
(a)(b)(c)The detection results from multitemporal optical images [27] are shown in the first line (top) of Figure 3. The second line (bottom) displays four colors: red depicts missing classified part, green depicts correctly classified part, blue depicts misclassified part, and black depicts the background part.Figure 3
Performance of different models (from top/left to bottom/right): (a) SegNet, (b) U-Net, (c) SE-Unet, (d) DTRF-Unet, and (e) myNet.
(a)(b)(c)(d)(e)As shown in Figure3, SegNet has worst-case performance in that it has a simple network structure, as shown in Figure 3(a). Although U-Net has a symmetrical analysis–synthesis structure, the lower center of Figure 3(b) has not been detected when it faces 3D structures of hyperspectral images. Aiming at further improvement, SE-Unet (Figure 3(c)) and DTRF-Unet (Figure 3(d)) maintain high-resolution representations, but the upper right part is still missing. Only the proposed myNet model outperforms others and closely resembles ground truth (GT). One of the critical components to turning this around was introducing an attention mechanism. However, these evaluation indicators from image segmentation tasks in Table 3, such as F1, Ks, and mIoU, support this conclusion. Accuracy increases from 67.73% in SegNet to 98.44% in the proposed model. The F1 values have dramatically increased from 12914 in SegNet to 26328 in the proposed myNet. The growth in Ks has increased approximately 30%, whereas the increase in mIoU tripled, reaching 98.44%. After overall consideration, we conclude that the proposed model is effective for image segmentation tasks.Table 3
Results from multitemporal optical images.
ModelF1KsmIoUAccuracySegNet12914129140.25010.6773U-Net16621163530.33990.7053SE-Unet17016165650.35070.7315DTRF-Unet19944165040.45370.8622Proposed26328182410.89870.9844


### 3.2. xBD Dataset

The second building damage assessment dataset, xBD [28, 29], was proposed by Gupta et al. The xBD dataset is used by the xView 2 prize challenge, covering eight disaster types, 15 countries, and over 800,000 building annotations across over 45,000 km2 of the imager. For simplicity, 256×256 pixels are cropped from palu-tsunami_00000096 in xBD, as shown in Figure 4. White depicts damaged building roof polygons.Figure 4
Imagery from palu-tsunami (from left to right): (a) original image, (b) predisaster imagery, (c) postdisaster imagery, and (d) its label.
(a)(b)(c)(d)Figure5 and Table 4 present the details of different methods on xBD. With respect to the accuracy, the performance of SegNet is more easily affected by noise and inevitably degraded. As can be clearly seen, the best satisfactory performance is provided using the proposed myNet. This is because it utilizes hyperspectral data with abundance information and high antinoise property. On the other hand, other coefficients of myNet are also very high, indicating that the consistency between myNet and GT map is almost perfect.Figure 5
Performance of different models (from top/left to bottom/right): (a) SegNet, (b) U-Net, (c) SE-Unet, (d) DTRF-Unet, and (e) myNet.
(a)(b)(c)(d)(e)Table 4
Results from xBD.
ModelF1KsmIoUAccuracySegNet4824.06116510.18540.3914U-Net4326.95148100.24230.4412SE-Unet8149.83185790.27730.4799DTRF-Unet9511.88196590.28820.5174Proposed10275.57272060.38740.8992


### 3.3. ABCD Scene

ABCD scene [30] was collected from an exhaustive investigation after the great tsunami in the Tohoku region of Japan on March 11, 2011. It comprises 8,506 pairs of pretsunami and posttsunami spectral images, whose scale and resolution are fixed as 160×160 pixels and 40 cm/pixel, respectively. Over 220,000 buildings were assigned to their corresponding labels, representing the degree of damage. Among them, 4,253 target buildings had taken a central place in pretsunami images and were flattened in posttsunami ones. Figure 6 shows the 4161st and 15738th scenes in ABCD [30].Figure 6
Imagery from the great tsunami (from left to right): (a) pretsunami image, (b) its label, (c) posttsunami image, and (d) its label.
(a)(b)(c)(d)Experiments on the ABCD scene demonstrate that the proposed approach had significant performance and closely resembles GT. The third line (bottom) displays four colors: yellow depicts missing damaged part, green depicts correctly damaged part, blue depicts misclassified part, and black depicts background part, as shown in Figure7.Figure 7
Performance of different models (from top/left to bottom/right): (a) SegNet, (b) U-Net, (c) SE-Unet, (d) DTRF-Unet, and (e) myNet.
(a)(b)(c)(d)(e)The values of F1 have dramatically increased from 3230.7 in SegNet to 18764.3 in myNet, whereas Ks significantly increased from 2536 in SegNet to 16037 in myNet. Moreover, the evaluation indicators, mIoU, SeK, and OA, also draw a compelling conclusion as shown in Table5.Table 5
Results from ABCD scene.
ModelF1KsmIoUSeKOASegNet4824.06116510.18540.152447.3450U-Net4326.95148100.24230.154448.0698SE-Unet8149.83185790.27730.176951.1181DTRF-Unet9511.88196590.28820.178452.9514Proposed10275.57272060.38740.324970.7184


## 3.1. Multitemporal Optical Images

The multitemporal optical images [27] were derived from the fifth group of the 2021 Gaofen Challenge. Gaofen-2 and Jilin-1 provide 4 m multispectral image files with a spatial resolution that can reach 0.8 m. Figure 2 shows all multitemporal images mark changed buildings. Each pair of image files is six channels in tiff format, whereas annotation files are single-channel in png format.Figure 2
Multitemporal optical images: (a) before changed, (b) after changed, and (c) change annotation file.
(a)(b)(c)The detection results from multitemporal optical images [27] are shown in the first line (top) of Figure 3. The second line (bottom) displays four colors: red depicts missing classified part, green depicts correctly classified part, blue depicts misclassified part, and black depicts the background part.Figure 3
Performance of different models (from top/left to bottom/right): (a) SegNet, (b) U-Net, (c) SE-Unet, (d) DTRF-Unet, and (e) myNet.
(a)(b)(c)(d)(e)As shown in Figure3, SegNet has worst-case performance in that it has a simple network structure, as shown in Figure 3(a). Although U-Net has a symmetrical analysis–synthesis structure, the lower center of Figure 3(b) has not been detected when it faces 3D structures of hyperspectral images. Aiming at further improvement, SE-Unet (Figure 3(c)) and DTRF-Unet (Figure 3(d)) maintain high-resolution representations, but the upper right part is still missing. Only the proposed myNet model outperforms others and closely resembles ground truth (GT). One of the critical components to turning this around was introducing an attention mechanism. However, these evaluation indicators from image segmentation tasks in Table 3, such as F1, Ks, and mIoU, support this conclusion. Accuracy increases from 67.73% in SegNet to 98.44% in the proposed model. The F1 values have dramatically increased from 12914 in SegNet to 26328 in the proposed myNet. The growth in Ks has increased approximately 30%, whereas the increase in mIoU tripled, reaching 98.44%. After overall consideration, we conclude that the proposed model is effective for image segmentation tasks.Table 3
Results from multitemporal optical images.
ModelF1KsmIoUAccuracySegNet12914129140.25010.6773U-Net16621163530.33990.7053SE-Unet17016165650.35070.7315DTRF-Unet19944165040.45370.8622Proposed26328182410.89870.9844


## 3.2. xBD Dataset

The second building damage assessment dataset, xBD [28, 29], was proposed by Gupta et al. The xBD dataset is used by the xView 2 prize challenge, covering eight disaster types, 15 countries, and over 800,000 building annotations across over 45,000 km2 of the imager. For simplicity, 256×256 pixels are cropped from palu-tsunami_00000096 in xBD, as shown in Figure 4. White depicts damaged building roof polygons.Figure 4
Imagery from palu-tsunami (from left to right): (a) original image, (b) predisaster imagery, (c) postdisaster imagery, and (d) its label.
(a)(b)(c)(d)Figure5 and Table 4 present the details of different methods on xBD. With respect to the accuracy, the performance of SegNet is more easily affected by noise and inevitably degraded. As can be clearly seen, the best satisfactory performance is provided using the proposed myNet. This is because it utilizes hyperspectral data with abundance information and high antinoise property. On the other hand, other coefficients of myNet are also very high, indicating that the consistency between myNet and GT map is almost perfect.Figure 5
Performance of different models (from top/left to bottom/right): (a) SegNet, (b) U-Net, (c) SE-Unet, (d) DTRF-Unet, and (e) myNet.
(a)(b)(c)(d)(e)Table 4
Results from xBD.
ModelF1KsmIoUAccuracySegNet4824.06116510.18540.3914U-Net4326.95148100.24230.4412SE-Unet8149.83185790.27730.4799DTRF-Unet9511.88196590.28820.5174Proposed10275.57272060.38740.8992


## 3.3. ABCD Scene

ABCD scene [30] was collected from an exhaustive investigation after the great tsunami in the Tohoku region of Japan on March 11, 2011. It comprises 8,506 pairs of pretsunami and posttsunami spectral images, whose scale and resolution are fixed as 160×160 pixels and 40 cm/pixel, respectively. Over 220,000 buildings were assigned to their corresponding labels, representing the degree of damage. Among them, 4,253 target buildings had taken a central place in pretsunami images and were flattened in posttsunami ones. Figure 6 shows the 4161st and 15738th scenes in ABCD [30].Figure 6
Imagery from the great tsunami (from left to right): (a) pretsunami image, (b) its label, (c) posttsunami image, and (d) its label.
(a)(b)(c)(d)Experiments on the ABCD scene demonstrate that the proposed approach had significant performance and closely resembles GT. The third line (bottom) displays four colors: yellow depicts missing damaged part, green depicts correctly damaged part, blue depicts misclassified part, and black depicts background part, as shown in Figure7.Figure 7
Performance of different models (from top/left to bottom/right): (a) SegNet, (b) U-Net, (c) SE-Unet, (d) DTRF-Unet, and (e) myNet.
(a)(b)(c)(d)(e)The values of F1 have dramatically increased from 3230.7 in SegNet to 18764.3 in myNet, whereas Ks significantly increased from 2536 in SegNet to 16037 in myNet. Moreover, the evaluation indicators, mIoU, SeK, and OA, also draw a compelling conclusion as shown in Table5.Table 5
Results from ABCD scene.
ModelF1KsmIoUSeKOASegNet4824.06116510.18540.152447.3450U-Net4326.95148100.24230.154448.0698SE-Unet8149.83185790.27730.176951.1181DTRF-Unet9511.88196590.28820.178452.9514Proposed10275.57272060.38740.324970.7184


## 4. Discussion

General discussion took place in this section. Compared with other results, SegNet has worst-case performance in that its simple network structure. Although U-Net, as a typical model with encoder-decoder structure, has shown the potential in various fields, yet only some test results were correctly detected in our experiments, largely because the encoder-decoder structure themselves still have room for improvement. SE-Unet can not only effectively strengthen network capability by squeeze-and-excitation modules but also can be useful in other sample imbalanced situations by dice and cross-entropy loss function, while DTRF-Unet can not only enhance edge features of buildings but also reduce information loss after down sampling by U-Net. However, computing power, model segmentation, and time performance have plenty of room for improvement. Only myNet model outperforms the others and closely resembles GT. And one of the reasons that is most responsible for that hierarchical Swin Transformer in myNet model is used as the encoder to extract irregular structural buildings features for hyperspectral image segmentation. On the other hand, multitemporal hyperspectral images be usefully employed to avoid the influence of existing trees, varying sizes, and orientations. Also, we adopt different geometrical feature to a new optimized method to obtain better evaluation results.


## 5. Conclusion

Automatic extraction of damaged buildings from remote sensing images is essential to emergency disaster assessment and military target reconnaissance. In this paper, a building extraction method is proposed on hyperspectral images based on an improved Swin-Unet model. The experimental result shows that the proposed method can effectively extract damaged buildings. Under the constraint of current hardware, these parameters are set on the basis of experience or the performance of proposed method. The research can definitely try to effectively extract damaged buildings. Further studies can undoubtedly create a multisource, multiscale, and high-resolution building dataset. With the help of the fieldwork, more data spectral curves are gained for the research region and built up a single-surface building material spectral library. So adaptively determining parameters, spectral libraries, building extraction, and accuracy evaluation would be addressed in the future.



---

*Source: 2124949-2022-07-15.xml*