# Accounting for Training Data Error in Machine Learning Applied to Earth Observations
## 3 Case Studies
### Infrastructure Mapping
#### 3.1.2 Detecting Roads from Satellite Imagery  
Road networks constitute a critical geographical data layer used to assist national decision makers in resource allocation, infrastructure planning, vaccination campaigns, and disaster response, among others. However, accurate and up-to-date road networks are not available in many developing countries. High-resolution satellite imagery, paired with deep-learning methods, provides the capacity to detect and map roads at large spatial scales. This important goal, however, is dependent on availability of local high-quality TD.  
To evaluate the impact of local TD availability on predicted road network accuracy, a study was carried out in Kumasi, Ghana [194]. Two datasets were used to train ML models: (1) the SpaceNet ([[https://spacenetchallenge.github.io/](https://spacenetchallenge.github.io/)]([https://spacenetchallenge.github.io/](https://spacenetchallenge.github.io/))) dataset [195] in Khartoum, Sudan, and Las Vegas, USA, and (2) OSM data in Kumasi, Ghana. The SpaceNet Dataset includes high quality road labels with human expert validation, but unfortunately was not available in Kumasi, Ghana. Therefore, the latter study site relied on OSM data, consisting of crowdsourced labels with no accuracy assessment or expert validation. A series of experiments were carried out to assess the feasibility of using transfer learning, using the Raster Vision Python library for training and evaluation. For all MobileNet V2 models introduced in the following list, the image chip size was set to 300 \(\times\) 300 pixels, and the training/validation split was 80/20.  
The Las Vegas Model was trained and validated on SpaceNet data in Las Vegas and produced very high accuracy predictions. However, when this model was used in Kumasi, it predicted very few roads, with only scattered road segments. The Khartoum model was also trained using SpaceNet data in Khartoum. The Kumasi model used Maxar WorldView-3 imagery and labels from OSM as input. OSM was used to test the quality of crowdsourced labels in training a road detection model. The Khartoum Model was then fine-tuned on OSM labels in Kumasi for three different steps of 100 K, 50 K and 10 K. All models used the same hyperparameters, to isolate the role of TD on model performances.  
To validate the models' performance using an independent dataset, a set of expert labels was generated over a small part of Kumasi. Figure 9 shows the region with human expert data vetting, along with the three model predictions. The Las Vegas model is excluded from this figure as it does not have any meaningful prediction in Kumasi. Quantitative performance metrics were calculated using the expert-created labels, to which the models had been blind during training. The results indicate that, as shown by Figure 9, the F1 score for roads was substantially higher for the Kumasi model (0.6458) than when using the Khartoum model (0.3780). However, by retraining and fine-tuning the Khartoum model, the F1 score for roads increased to 0.6135. The full accuracy results for this experiment are presented in Table S3, and prediction maps are shown in Figure S1.  
Based on these results, it is concluded that: (1) lack of diverse TD significantly limits the geographic applicability of models, as the types, surfaces, and arrangements of roads varies substantially between regions; (2) regional training datasets are essential for the model to learn the feature of roads in that region; and (3) transfer learning from a reasonably similar geography can help train models.