<figure><img src="image_1.png" /></figure>

HAL
open science

# Deep Siamese Network with Contextual Transformer for Remote Sensing Images Change Detection

Mengxuan Zhang, Zhao Liu, Jie Feng, Licheng Jiao, Long Liu

► To cite this version:

Mengxuan Zhang, Zhao Liu, Jie Feng, Licheng Jiao, Long Liu. Deep Siamese Network with Contextual Transformer for Remote Sensing Images Change Detection. 5th International Conference on Intelligence Science (ICIS), Oct 2022, Xi'an, China. pp.193-200, 10.1007/978-3-031-14903-0_21. hal-04666436

HAL Id: hal-04666436

https://hal.science/hal-04666436v1

Submitted on 1 Aug 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Deep Siamese Network with Contextual
Transformer for Remote Sensing Images Change
Detection*

Mengxuan Zhang¹, Zhao Liu¹, Jie Feng¹, Licheng Jiao¹, and Long Liu²,*

¹ Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, School of Artificial Intelligence, Xidian University, Xi’an, Shaanxi Province 710071, China

² School of Electronic Engineering, Xidian University, Xi’an, Shaanxi Province 710071, China

**Abstract.** Change detection is one of the most important and challenging tasks in remote sensing images processing. Deep learning has gradually become one of the most popular technologies in remote sensing image change detection. Recently, the success of self-attention mechanism in computer vision provides new ideas for change detection task. In this paper, a new method based on deep siamese network with self-attention mechanism for bi-temporal remote sensing image change detection is proposed. In order to obtain more powerful image features, the contextual transformer module is added into the feature extractor. In order to make full use of the low-level and the high-level features from the feature extractor, the multi-scale fusion strategies are applied to integrate features. Furthermore, the obtained image features are input into the transformer to get more refined pixel-level features. The proposed model is testified on CCD dataset, and the results demonstrate its effectiveness.

**Keywords:** Change detection · Contextual Transformer · Siamese network · Transformer.

# 1 INTRODUCTION

With the development of remote sensing technology, remote sensing images have become a common data source for monitoring the Earth's surface [1]. Change detection is an effective task to observe changes in the Earth's surface, which aims to compare the surface differences in the same area at different times. It is widely used in the fields of land cover survey, natural disaster monitoring, resource exploration, environmental detection, and urban expansion [2].

* This work was supported in part by the Natural Science Basic Research Program of Shaanxi (No. 2022JM-336), the Fundamental Research Funds for the Central Universities (No. XJS211906 and No. XJS210205), the Postdoctoral Science Foundation of China (No. 2018M633467). (Corresponding author: Long Liu. Email: lunar_xdun@163.com).
Over the past few decades, the researchers have proposed various approaches to deal with the change detection task. The traditional change detection methods can be divided into three categories: the image arithmetic-based method, the image transformation-based method and the post-classification method. The image arithmetic-based methods generate the image change maps by comparing the pixel values directly and apply the thresholds to classify pixels into changing and invariant classes. Since the image arithmetic-based method is affected by radiation and noise easily, it is difficult to select an appropriate threshold. In the image transformation-based methods, the image spectral combination is converted into a specific feature space to distinguish the changed pixels. The post-classification is a supervised method which uses a trained classifier to classify two-phase images and compares the resulting feature maps pixel-by-pixel to obtain the change maps.

In recent years, the deep learning techniques have achieved a great success. With its powerful generalization ability, deep learning breaks the limitations of traditional algorithms. Therefore, many deep learning-based methods have been proposed in the field of remote sensing images change detection. Daudt et al. [3] proposed three U-Net-based Siamese networks, FC-EF, FC-Siam-Conc and FC-Siam-Diff. The FC-EF connects the bi-temporal images as a single input to the fully convolutional network. The FC-Siam-Conc contains two skip connections, which are concatenated in the decoding part for later fusion. The FC-Siam-Diff concatenates the absolute value of the difference of two skip connections. Liu et al. [4] proposed LGPNet with a local and global pyramid-structured feature extractor. The local and global feature pyramid module capture buildings of different scales from multiple perspectives and avoid to miss the buildings of different scales and shapes. Peng et al. [5] proposed an improved U-Net++ model. A deep supervised module is applied to capture subtle changes in challenging scenes. Chen et al. [6] proposed a spatial-temporal attention neural network with two kinds of self-attention modules, a basic spatial-temporal attention module and a pyramid spatial-temporal attention module. Zhang et al. [7] proposed IFN. The VGG-16 is utilized to extract bi-temporal image features, then the channel and spatial attention modules are used in the decoder to concatenate image features. Furthermore, the intermediate layers of the IFN are trained with deep supervision. Fang et al. [8] proposed SNUNet, which is composed of the Siamese network and the NestedUNet. The channel attention module is utilized to refine the features in each level of the decoder.

Although the deep learning methods have made outstanding contributions to the development of change detection, there still exist some problem. To improve the performance of change detection, many existing change detection methods introduce an attention mechanism into the final prediction stage and the decoder. Unlike above methods, the self-attention module is added into the feature extractor to get refined feature representations in this paper. A deep Siamese network with contextual transformer (DSNCoT) for bi-temporal image change detection is proposed. In order to get more obvious and distinguishable features, the contextual transformer (CoT) [9] is integrated into the network to improve
the ability of feature extraction. A transformer module [10] is adopted to add
more useful spatial and semantic information so as to enhance the pixel-level
features of the image.

<figure><img src="image_4.png" /><figcaption>Fig. 1. The Architecture of DSNCoT.</figcaption></figure>

## 2 METHODOLOGY

As shown in Fig. 1, the proposed DSNCoT is a Siamese-based network, which contains three parts: Firstly, a weights-shared feature extractor to extract multi-scale features from bi-temporal images. Secondly, a transformer module excapates image pixel-level features, which is helpful to identify the change of interest and exclude irrelevant changes. Thirdly, a prediction module generates the final change map.

### 2.1 Feature Extractor

Given a set of bi-temporal images, a weight-sharing feature extractor is utilized to
obtain the different levels of the features for each image. The feature extractor is
designed on the basis of ResNet-50. The feature extractor consists of five stages.
The first stage utilizes a convolutional layer with the kernel size 7×7 to extract
the low-level features from the input image. Then, a 2×2 max-pooling layer
<figure><img src="image_5.png" /><figcaption>Fig. 2. The details of the CoT module.</figcaption></figure>

with stride 2 is applied to resize the feature map to half the size of the input image. Four bottleneck blocks constitute the remaining four stages. As shown in Fig. 2, all the 3×3 convolutional layers are replaced by the CoT [9] module in ResNet-50 which aims to improve the capability of feature extractor. The CoT module consists of two parts. The static contextual features are obtained by 3×3 convolution. Then, a new designed self-attention mechanism is used to integrate the static and the dynamic contextual features. Finally, the static and the dynamic contextual information are combined together. In order to make full use of the sufficient spatial information of the low-level features and semantic information of the high-level features, the nonlinear interpolation is utilized to integrate the features obtained at each stage.

## 2.2 Transformer Module

In order to get more obvious feature pairs, as shown in Fig. 3, the transformer [10] is introduced to make the feature pairs distinguish easily. The characteristics of the original pixel space are enhanced by using the relationship between each pixel and the semantic tokenizer. Firstly, the Siamese semantic tokenizer is utilized to extract compact semantic tokens from the feature map. Similar to the tokenizer of NLP, it splits the input sentence into several elements and represents each element with a token vector. The Siamese semantic tokenizer splits the feature maps from the feature extractor into a few visual words, and each
<figure><img src="image_6.png" /><figcaption>Fig. 3. The transformer module.</figcaption></figure>

word corresponds to one token vector. Then, the token vectors are transformed into sets of features in the spatial space, which are named as the token sets. Secondly, these token sets are input into a transformer encoder to model the context between each token so as to generate more context-rich token representations. Finally, the token representation output by the transformer encoder is input into the transformer decoder to obtain pixel space-level features.

## 2.3 Prediction module

In this paper, a very shallow FCN is used for change discrimination. The FCN consists of two 3×3 convolutional layers with BatchNorm, and the output channels of each convolution are "32,2". We put the bi-temporal upsampling feature map $F_{new}^1, F_{new}^2 \in R^{H \times W \times C}$ obtained by DSNCoT into FCN to get the change map $P \in R^{H \times W \times 2}$, where H and W represents the height and width of the original image respectively. The formula can be written as follow:

$$P = \delta(f(D)) = \delta(f(F_{new}^1, F_{new}^2)) \quad (1)$$

Where $D \in R^{H \times W \times C}$ is the absolute value of the element subtraction of the bi-temporal feature map. $f(\cdot)$ is the change classifier, where the number of output channels changes from C to 2. $\delta(\cdot)$ represents a softmax operation on the classifier output.

## 2.4 Details of Loss Function

Given a set of training image pairs and corresponding ground truth maps, the objective function is optimized to obtain accurate predictions. In change detection, the number of unchanged pixels is always much more than the number of changed pixels. In order to alleviate the influence of sample imbalance, the objective function is defined on two different loss functions in this paper.
In the training stage, we combine two loss functions to optimize the parameters of the entire network. The overall loss function is defined as follow:

$$L(X, Y) = L_{CE}(X, Y) + \lambda L_{DICE}(X, Y) \quad (2)$$

Where X is the prediction map and Y is the ground truth map. L is the overall loss, LCE is the cross-entropy loss and LDICE is the dice loss. λ is the weight coefficient of dice loss which is set to 0.1.

The cross-entropy loss function is usually used for image classification. The change detection task can be viewed as a binary classification for partitioning the observed image into the changed and unchanged regions. The dice loss can alleviate the problem of sample imbalance. The above two loss function formulas are written as:

$$L_{CE}(X, Y) = \frac{1}{H \times W} \sum_{h=1, w=1}^{H,W} l(X_{hw}, Y_{hw}) \quad (3)$$

$$L_{DICE}(X, Y) = 1 - \frac{2|X \cap Y|}{|X| + |Y|} = 1 - \frac{1}{W} \sum_{w=1}^{W} \frac{2 \sum_{h=1}^{H} X_{h,w} Y_{h,w}}{\sum_{h=1}^{H} X_{h,w} + \sum_{h=1}^{H} Y_{h,w}} \quad (4)$$

Where Xhw and Yhw are the prediction map and the label map for the pixel at location (h, w) respectively. H and W are the height and width of the original image, respectively.

# 3 EXPERIMENT

## 3.1 Dataset

The CCD [11] dataset contains two types of image pairs, the artificially synthesized images and the remote sensing images. The remote sensing images are observed in our experiments. These real season-varying remote sensing images are obtained through the Google Earth API. The CCD dataset contains 16,000 remote sensing image pairs. The training set contains 10,000 pairs. The validation set and test set contain 3,000 pairs. The size of each image is 256×256, and the spatial resolution is 0.03-1m/pixel.

## 3.2 Implementations

All of the experiments are completed by the PyTorch framework and a work station with GeForce RTX 3090 and 24G memory. The stochastic gradient descent with momentum is used to optimize the model. The values of the momentum, the weight decay and the learning rate are set to 0.99, 0.0005 and 0.01 respectively. The entire model is trained for 200 epochs until it converges. To verify the effectiveness of the proposed model, five typical metrics are applied: precision (Pre), recall (Rec), F1-score (F1), intersection over union (IoU) and overall accuracy (OA).
<table><caption>**Table 1.** The Numerical Results on CCD Dataset.</caption><thead><tr><th>Method</th><th>Pre (%)</th><th>Rec (%)</th><th>F1 (%)</th><th>IoU (%)</th><th>OA (%)</th></tr></thead><tbody><tr><td>FC-EF [3]</td><td>60.91</td><td>58.31</td><td>59.20</td><td>64.81</td><td>91.47</td></tr><tr><td>FC-EF-Conc [3]</td><td>70.92</td><td>60.32</td><td>63.71</td><td>66.46</td><td>92.52</td></tr><tr><td>FC-EF-Diff [3]</td><td>76.26</td><td>57.37</td><td>65.33</td><td>68.72</td><td>93.15</td></tr><tr><td>SNUNet [8]</td><td>92.07</td><td>84.64</td><td>88.20</td><td>78.69</td><td>97.33</td></tr><tr><td>IFN [7]</td><td><strong>97.46</strong></td><td>86.96</td><td>91.91</td><td>85.03</td><td>98.19</td></tr><tr><td>DSNCoT</td><td>96.88</td><td><strong>95.81</strong></td><td><strong>96.34</strong></td><td><strong>92.94</strong></td><td><strong>99.14</strong></td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>**Fig. 4.** Visualization results on CDD dataset: (a) the images at time1, (b) the images at time2, (c) the ground truth, (d) the results of FC-EF, (e) the results of FC-Siam-Conc, (f) the results of FC-Siam-Diff, (g) the result of IFN, (h) the results of SNUNet, (i) the results of the proposed model.</figcaption></figure>

## 3.3 Results and analyses

To show the effectiveness of our proposed DSNCoT, we compare it with five change detection methods, including the FC-EF [3], the FC-Siam-conc [3] and the FC-Siam-diff [3], the IFN [7] and the SNUNet [8]. All the numerical results are given in Table 1. It can be observed that the proposed DSNCoT model outperforms other five methods with higher values of Rec, F1, IoU and OA. The visualization results on the CCD dataset are shown in Fig 4. FC-EF, FC-EF-diff and FC-EF-conc can capture the large-sized changing regions, but they neglect some small changes in the observed images. The IFN can get better change maps on the most change areas of large targets, but its ability to catch small targets is
not good enough. Although the SNUNet can get the complete change regions in
most cases, it is not sensitive to some edges in the change regions. The proposed
DSNCoT can obtain both the complete large-scaled changing areas and the small
changing areas, which include cars, roads, villages and new urban buildings.

4 Conclusion

In this paper, we propose a new deep learning method based on deep siamese
network with self-attention mechanism for change detection of bi-temporal re-
mote sensing images. The CoT module is introduced into the feature extractor to
obtain bi-temporal image features with refined representations. The transformer
module is added to model the bi-temporal image features at the pixel level, in
order to obtain more refined bi-temporal image features. Compared with other
five change detection methods, DSNCoT can detect more detailed changes and
show better performance on the CCD dataset.

References

1. D. Hong et al.: More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification. IEEE Transactions on Geoscience and Remote Sensing 59(5), 4340-4354(2021)

2. J. Xiao et al.: Evaluating urban expansion and land use change in Shijiazhuang, China, by using GIS and remote sensing. Landscape and Urban Planning 75(1), 69-80(2006)

3. C. Daudt et al.: Fully Convolutional Siamese Networks for Change Detection. 2018 25th IEEE International Conference on Image Processing (ICIP), 4063-4067(2018)

4. T. Liu et al.: Building Change Detection for VHR Remote Sensing Images via Local-Global Pyramid Network and Cross-Task Transfer Learning Strategy. IEEE Transactions on Geoscience and Remote Sensing 60, 1-17(2022)

5. D. Peng et al.: End-to-End Change Detection for High Resolution Satellite Images Using Improved UNet++. Remote Sensing 11(11), (2019)

6. H. Chen et al.: A Spatial-Temporal Attention-Based Method and a New Dataset for Remote Sensing Image Change Detection. Remote Sensing 12(10), (2020)

7. C. Zhang et al.: A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sensing images. ISPRS Journal of Photogrammetry and Remote Sensing 166, 183-200(2020)

8. S. Fang et al.: SNUNet-CD: A Densely Connected Siamese Network for Change Detection of VHR Images. IEEE Geoscience and Remote Sensing Letters 19, 1-5(2022)

9. Y. Li et al.: Contextual Transformer Networks for Visual Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, (2022). https://doi.org/10.1109/TPAMI.2022.3164083

10. H. Chen et al.: Remote Sensing Image Change Detection With Transformers. IEEE Transactions on Geoscience and Remote Sensing 60, 1-14(2022)

11. M. Lebedev et al.: Change detection in remote sensing image using conditional adversarial networks 42(2), 565-571(2018)