# Effective Risk Detection for Natural Gas Pipelines Using Low-Resolution Satellite Images
## 3 Change Detection with Transformers  
In CD, a model takes as input two co-registered images and produces a binary map indicating the change. During training, the model has to learn to differentiate between relevant changes, e.g., constructions, and non-relevant changes, e.g., seasonal changes or reflections. In this work, we use a state-of-the-art deep-learning-based CD model, ChangeFormer [9], shown in Figure 2. This section aims to give insights into the intermediate representations of ChangeFormer and how its attention mechanism is used for CD. For a more in-depth explanation of the computations carried out by each module, we refer to the original work of [9]. ChangeFormer consists of three stages:  
1. Feature extraction via four downsampling and transformer blocks (blue);
2. Feature comparison using four difference modules (red);
3. A decoder stage with a multilayer perceptron (MLP) and upsampling (yellow). Originally, transformer models were introduced for natural language processing tasks [14] but have been adapted to work with images (cf. vision transformer [15]) and perform equally as well on visual tasks as architectures based on convolutional neural networks (CNNs). Transformers take as input a sequence of tokens. In the computer vision context, images are split into patches, which serve as the transformer input tokens. Figure 2: **The ChangeFormer model adapted from [9]**: Four transformer modules produce features on different scales for the input images. The features from both images serve as input to the difference modules. These modules produce comparison features, which are forwarded through a multilayer perceptron (MLP) and upsampling module to output the binary change map. Unlike in the vision transformer, ChangeFormer, features are extracted by convolutions that downsample and extract features from overlapping patches in the input image. Afterwards, a positional encoding is added to the feature maps using another CNN and MLP step. Figure 3 shows how these feature maps are used as tokens for the transformer blocks. Each pixel in the feature map corresponds to one token with length C, which corresponds to the feature dimension. Together, all tokens (i.e., pixels) make up the input sequence for one transformer block. At the heart of every transformer is an attention mechanism applied in the transformer block. Attention mechanisms produce a vector that weighs up which parts of the input sequence are essential for solving the specific task. The ChangeFormer model uses the scaled self-attention mechanism from [8]:  
\[\textbf{Attention}(\textbf{Q},\textbf{K},\textbf{V}))=Softmax(\frac{\textbf{Q} \textbf{K}^{T}}{\sqrt{d_{head}}}\textbf{V}) \tag{1}\]  
In self-attention, every input token can look at features from the same sequence of input tokens. \(K\), \(Q\), and \(V\) are all linear transformations of the input sequence. Every token emits features to find changes via its key \(K\). The query \(Q\) binds to these features, and together, \(QK\) can be seen as the affinity of every token with each other. By multiplying by the value \(V\), we obtain the aggregation of all important tokens for every token. The tokens are then enriched with the attention features, which is repeated for three rounds of attention. There are four transformer blocks that apply downsampling and the attention mechanism, and the attention-enriched feature maps from the previous block serve as the input for the next block. Intuitively, these attention-enriched feature maps contain features from each pixel and features from other pixels, which contain valuable information for the specific pixel. Furthermore, we can look for changes on different scales as all transformer blocks together extract hierarchical features on four different scales. Figure 4 shows attention maps on four scales after the first round of attention for the _before_ image. In the next step, each difference module compares concatenated features from both images with the same scaling.