# Effective Risk Detection for Natural Gas Pipelines Using Low-Resolution Satellite Images
## 4 Experimental Evaluation
Note that in this work, we only use the CD part of the network, which involves, similar to the other BIT and ChangeFormer, obtaining features via convolutional layers and enriching them via attention. * **TinyCD**[19] is again a Siamese structure that uses the attention mechanism over features extracted from a CNN. The main difference is in its size. While the current state-of-the-art models have millions of parameters, such as ChangeFormer with over 41 million, TinyCD only has a fraction with 280k parameters and still can achieve good performance on benchmark CD datasets. For a thorough comparison, we refer to the TinyCD paper [19]. **Training:** All models are trained for 200 epochs, and we use the AdamW optimizer and, as mentioned before, the cross-entropy loss with different weightings \(\alpha\) for ChangeFormer. We use the default hyperparameters (batch size of 16, learning rate is 0.0001 with a linear decay to zero) from [9] as they produce the best results and apply the same data augmentation during training. The same applies to the baselines, where we use the default parameters, loss functions, and data augmentation. We select the best-performing model on the validation split and report its performance on the unseen test set. **Q1. Qualitative evaluation**: Figure 5 shows examples of six satellite images from the collected dataset. We observe that the ChangeFormer predictions overlap well with the GT. When the GT comprises more complex shapes, the model loses some details but obtains the right overall form. In two cases, the model predicts small blobs not in the GT (red circles). In Figure 6, ChangeFormer is compared with the baselines on our running example from Figure 1. All models detect the changes here but make another false prediction, except DTCDSCN, which only predicts one object. Qualitatively, ChangeFormer, TinyCD, and DTCDSCN match the GT the best, while the FC-Siam-Conc model predicts a significant change that is not in the GT. These results indicate that ChangeFormer learns the correct features to predict relevant changes. In some cases, it lacks some details but obtains the right overall shape of the GT. Table 1 shows the _pixel-based_ performance. Here, we can observe that ChangeFormer and TinyCD have the highest IoU, which aligns with our qualitative observations--almost every second predicted pixel overlaps with the GT. FC-Siam-Conc and the BIT perform worse with \(\sim\)13% less IoU. Figure 5: **Qualitative results:** Shown from the top to bottom row are the _before_ and _after_ images, the prediction, and the GT from the ChangeFormer model [9]. The model can detect almost all changes represented in the GT, even in the case of three changes in one image, as in the fourth column. In some images, it predicts false positives (red circles). The last column again shows a false positive as it predicts the field being harvested as a change, but it also misses a small construction being started (yellow circle). Figure 6: **Qualitative comparisons**: In our running example (Figure 1), all models detect the construction site in the GT but miss its exact shape. Additionally, some models predict another change in the upper left corner that is not in the GT. **Q2. Model performance**: On the _pixel-based_ comparison in Table 1, we observe that TinyCD and ChangeFormer perform the best by a considerable margin compared with the BIT and FC-Siam-Conc baseline. The recall and precision of ChangeFormer are balanced. TinyCD has the best precision values and slightly worse recall values than ChangeFormer. FC-Siam-Conc achieves competitive precision values but has a lower recall, and DTCDSCN has slightly worse performance than the best-performing models. Table 2 depicts the _object-based_ performance. Here, DTCDSCN has the best F1 score followed by ChangeFormer. ChangeFormer detects approximately 80% of the changes, with a reasonably high precision exceeding 60%. Overall, most values are higher as this metric only considers wherever there was a change rather than how good predictions overlap with the GT.