# Satellite On-Board Change Detection via Auto-Associative Neural Networks
## 3 Results
### Quantitative Performance Analysis  
Typical metrics for image classification quantitative performance analysis are:  
\[\text{Accuracy}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{FP}+\text{TN}+\text{ FN}} \tag{2}\]  
\[\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}} \tag{3}\]  
\[\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}} \tag{4}\]  
\[\text{False Positive Rate}\left(\text{FPR}\right)=\frac{\text{FP}}{\text{FP}+ \text{TN}} \tag{5}\]  
\[\text{F1}=2\times\frac{\text{Precision}\times\text{Recall}}{\text{Precision}+ \text{Recall}} \tag{6}\]  
where:  
* TP = true positives, i.e., samples labelled as changed (red markers) and correctly predicted by the model;
* TN = true negatives, i.e., samples labelled as not changed (blue markers) and correctly predicted by the model;
* FP = false positives, i.e., samples labelled as not changed (blue markers) and wrongly predicted as changed by the model;
* FN = false negatives, i.e., samples labelled as changed (red markers) and wrongly predicted as not changed by the model. In the metrics computations, samples related to patches with less than 10% of pixels changed (i.e., green markers) were discarded. As highlighted in the ground truth images, the number of changed pixels is significantly lower than the number of unchanged ones, and this class imbalance was considered during the results evaluation. In the case of an unbalanced dataset, accuracy is not useful, while the F1 score represents a proper measure as a combination between precision and recall. The receiver operating characteristic (ROC) curve is also a valid method for evaluating the performance of an algorithm. It plots the recall against the FPR, and the area under the curve (AUC) measures the performance of the model for all possible change thresholds. The AUC ranges from 0.5 for a random classifier to 1 for a perfect classifier. Figures 16 and 17 show the ROC curve obtained by validating with OSCD and CVA ground truth images, respectively, for test images 1 and 2 in Figure 11. Figure 16 highlights that the lowest AUC value occurs for test image number 2, in the DWT case. With the AANN algorithm, the value improves by approximately 0.2. Test image number 1 exhibits better AUC values; the best is 0.909 for the AANN algorithm. By validating with the CVA (Figure 17), the values increase, reaching almost 1 for test image 2 with the AANN algorithm, but the other cases also reach high AUC values. Table 2 summarizes the results obtained for test image number 1, shown in Figure 11(a\({}_{1}\),a\({}_{2}\)), validated with the OSCD and CVA reference images. The results obtained with the AANN and DWT techniques were compared; the table also reports the AUC values. The change detection with AANN reduction performed better than DWT, as already demonstrated in Figures 12 and 14. F1 achieves 0.86 with CVA validation and AANN reduction, which is more than 0.2 higher than DWT. The result also increases by the same value when validating with OSCD reference image, although, in this latter case, the best absolute value of F1 is 0.6. Recall represents an important metric to analyze, since it provides a measure of the number of missed changes, and these are the main focus of a change detection algorithm. Recall doubles by performing reduction with the AANN, with both OSCD and CVA validations. However, in an operational scenario, the threshold value should be previously defined, in order for the algorithm to run automatically. The choice of an optimum general value can be a very difficult task because it definitely depends on the land cover contained in the image; a scene with a prevalence of urban areas would certainly provide an optimum threshold value different from that of a rural one. To address this issue, we selected from the dataset the two most similar images in terms of land cover: test image number 1 (shown in Figure 11a1,a2) was compared to test image number 3 (shown in Figure 3). Based on the  
Figure 16: ROC curve for test image 1 in Figure 11a1,a2 (**left**) and test image 2 in Figure 11d1,d2 (**right**).