## 3 Experiments and Results
### Results  
The dataset consists of 17,159 samples of 256 \(\times\) 256-pixel RGB images, labeled to indicate whether they contain roads (53.5%) or not (46.5%). The images are from the National Plan for Aerial Orthophotometry (PNOA) with original pixel resolutions of 0.25 or 0.50 m. They were taken from a Web Map Tiled Service (OGC:WMTS) of the IGN at zoom level 18 which corresponds to a scale of 1:2.132 so the equivalent pixel size on the ground is 0.597 m. The samples correspond to eight areas of the Spanish geography that can be considered representative: Albacete, Balearic Islands, Ciudad Real, Galicia, Huelva, Murcia, Navarre and Segovia. We refer to this dataset as IGN-PNOA hereafter. Sample results of a designed network's output for three random sample groups (not used for training or validation) outside of the dataset (orthophotos were from random geographical areas of Spain with no associated cartography available) to check their predictions over unknown data are shown in Figure 4. Under each image of the group is the network predictive output for the class \"contains road\". In all these examples, the somewhat extreme output values given by the network are due to the use of the softmax activation on the output layer. We selected the nine highest scoring networks (measured on the validation partition) designed by VGNet; all models were trained from scratch using the train and validation examples. To evaluate  
Figure 4: Sample predictions for three randomly selected zones outside the dataset that were not used for training. “p_road” shows the network prediction confidence for the class “contains road”. their test accuracy, k-fold cross-validation (CV) was used with typical values of k = 5, 10 [81,82]. Regarding k = 10, the authors of [81,83] showed that 10-fold CV obtain similar results to leave-one-out cross-validation (LOOCV) and has lower computational cost. Test metrics are included in Figures 5 and 6 and detailed in Table 2. Their respective accuracy (95% confidence intervals, using the calculation method detailed by [PERSON] [84]) are in Table 3 to show the expected accuracy for each model. In all cases, the same parameters used for designing the networks (\(epochs=200\), \(alpha=1e^{-4}\), \(optimizer=Adam\) and batch size calculated for each model based on GPU memory available) were used to train them on this final stage. Finally, for the two best scoring models in Table 4, additional metrics were calculated on the test partition: precision, recall, F1 score and Area Under the ROC-Curve (AUC). To give an example of one of the designed networks, a high-level architectural overview of model \"v2m2\" is shown in Figure 7 with sequences of operations grouped into blocks for clarity (as detailed in the figure legend). This particular model contains two Inception-ResNet-like sections at the areas with parallel operator branches. Regarding execution times, on average k-fold (k = 5) evaluation takes 52.8 h on IGN1 server (4\(\times\) Tesla V100 GPUs). Test accuracies for some well-known architectures on the IGN-PNOA dataset obtained by members of the Cartobot project team [41] by applying transfer learning techniques (in this case using partitioning train/validation/test with ratio 50:25:25) are for Inception-v3 (87.88-91.19%) or Inception-ResNet (91.28-92.53%). In addition, in [43], the mean accuracies for the same dataset are reported, among others, for ResNet-50 (91.7%), VGG (93.9%) and for two different ensembles (94.6% and 95.6%, respectively). Here, ensemble refers to the combination of multiple networks into one meta-classifier, namely their results are combined to improve predictions. \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Model** & **Precision** & **Recall** & **F1 Score** & **A.U.C.** \\ \hline v2m2 & 0.936 & 0.904 & 0.920 & 0.950 \\ v2m3 & 0.931 & 0.884 & 0.907 & 0.952 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Additional metrics for the two best models on the IGN-PNOA dataset on the test partition.