# Effective Risk Detection for Natural Gas Pipelines Using Low-Resolution Satellite Images
## 4 Experimental Evaluation  
In this section, we demonstrate that ChangeFormer is a viable choice for detecting pipeline risks and compare it to other models. We first examine examples qualitatively and  
Figure 4: **Attention maps**: ChangeFormer produces attention maps on different scales. Here, the mean attention map of all feature dimensions is shown after the first round of attention. The four scales correspond to the four transformer blocks shown in Figure 2. Areas containing houses are considered more on the 64 \(\times\) 64 scale. On the lower scales, more high-level features are considered as the resolution decreases. then perform a quantitative evaluation of our collected data. The aim of our experimental evaluation is to answer the following questions:  
* Can ChangeFormer capture visual properties of relevant changes? * How good is ChangeFormer at detecting relevant changes? * Can we adjust for higher recall without significantly sacrificing precision? We present the precision, recall, and F1 scores on a _pixel-_ and _object-based_ level to compare the models concerning the change class. Additionally, we report the Intersection over Union (IoU) at the pixel level to measure the prediction's overlap with the GT. The mean and standard deviation are shown over five runs for every model. In the _pixel-based_ metrics, the class of each pixel is checked. For the _object-based_ metrics, we group pixel blobs by selecting all connected pixels of the change class, as they usually belong to a single change. A pixel blob is a true positive if it overlaps with any GT blob by more than four pixels; otherwise, it is a false positive. To avoid favoring models with many minor predictions, two prediction blobs overlapping the same GT blob are counted as one true positive. The _object-based_ metric is helpful for pipeline monitoring since our primary focus is detecting wherever there was a change rather than precisely determining its exact boundaries. **Models**: We test the vanilla ChangeFormer with randomly initialized weights. Additionally, we provide the results of ChangeFormer with pretrained weights from training on the LEVIR-CD dataset [17], which we call **ChangeFormer-pt**. LEVIR is a high-resolution CD dataset that focuses on building changes and is close to our setting. We use the weighted cross-entropy loss (Section 3) to allow for higher recall in exchange for precision. Different values for the weighting parameter \(\alpha\) (2, 4, 8) are considered to punish missed changes more than false alarms and we call these cost-adjusted variants **ChangeFormer-ca**. A value of 2 imposes twice the penalty for missed changes compared to a false alarm, whereas a value of 8 imposes an even more severe penalty for missed changes. As our baseline, we compare with other state-of-the-art transformer-based models and a CNN model, which was the best-performing CNN on the LEVIR dataset in [9]. * **FC-Siam-conc**[18] is a CNN-based Siamese network. It consists of an encoder producing features for both input images and a decoder that takes the concatenation of features on different scales as input to produce the output binary map. In [18], the authors also proposed **FC-Siam-diff**, which uses the difference of features in the decoder. We could not produce any predictions using this model, so we did not include it in our comparison. * **Bitemporal Image Transformer (BIT)**[10] relies on a transformer encoder and decoder to enrich tokens obtained from a CNN. The tokens are then projected back to pixel space, where their difference is computed via subtraction, and another CNN is used to produce the binary change map. * **DTCDSCN**[6] combines a change detection module with two semantic segmentation modules, which are trained together. Image segmentations and the binary change mask are provided as labels during training. This addition helps the model to focus more on object-level features and improves the performance on the evaluated task of building change detection.