# Accounting for Training Data Error in Machine Learning Applied to Earth Observations
## 3 Case Studies
### Infrastructure Mapping  
#### 3.1.1 Incorporating Noisy Training Label Data  
Automated building footprint detection is an important but difficult mapping task, with potential benefits for a wide range of applications. The following case study illustrates the use of Raster Vision ([[https://rastervision.io/](https://rastervision.io/)]([https://rastervision.io/](https://rastervision.io/))), an open source deep learning framework, to train several models for automated building detection from high resolution imagery (Additional detail available at: [[https://www.azavea.com/blog/2019/08/05/noisy-labels-deep-learning/](https://www.azavea.com/blog/2019/08/05/noisy-labels-deep-learning/)]([https://www.azavea.com/blog/2019/08/05/noisy-labels-deep-learning/](https://www.azavea.com/blog/2019/08/05/noisy-labels-deep-learning/))). These models perform best when trained on a large number of correctly labeled examples, usually generated by a paid team of expert labelers. An alternative, less costly approach was conducted in which a building segmentation model was trained using labels extracted from OSM. However, the labeled training polygons generated from OSM contain errors: some buildings are missing, and others are poorly aligned with the imagery or have missing details. This provides a good test case for experimentation on how noise in the labels affects the accuracy of the resulting model. To measure the relationship between label noise and model accuracy, the amount of label noise was varied while holding all other variables constant. To do this, an off-the-shelf dataset (the SpaceNet Vegas buildings data set) was used in place of OSM, into which label errors were systematically introduced. Missing and imprecisely drawn building errors were systematically introduced to this relatively large training data set (~30,000 labeled buildings) ([[https://spacenetchallenge.github.io/datasets/spacenetBuildings-V2](https://spacenetchallenge.github.io/datasets/spacenetBuildings-V2) summary.html]([https://spacenetchallenge.github.io/datasets/spacenetBuildings-V2](https://spacenetchallenge.github.io/datasets/spacenetBuildings-V2) summary.html)), and then the resulting model accuracy was measured. The experimental design consisted of two series of six datasets each, with random deletion or shift of buildings at increasing probabilities and magnitudes, respectively. For each dataset, a UNet semantic segmentation model with a ResNet18 backbone was trained using the fastai/PyTorch plugin for Raster Vision ([[https://github.com/azavea/raster-vision-fastai-plugin](https://github.com/azavea/raster-vision-fastai-plugin)]([https://github.com/azavea/raster-vision-fastai-plugin](https://github.com/azavea/raster-vision-fastai-plugin))). These experiments, including data preparation and visualization, can be replicated using code at [[https://github.com/azavea/raster-vision-experiments/tree/master/noisy_buildings_semseg](https://github.com/azavea/raster-vision-experiments/tree/master/noisy_buildings_semseg)]([https://github.com/azavea/raster-vision-experiments/tree/master/noisy_buildings_semseg](https://github.com/azavea/raster-vision-experiments/tree/master/noisy_buildings_semseg)). Figure 7 shows the ground truth and predictions for a variety of scenes and noise levels, showing that the quality of the predictions decreases with the noise level. The background and central portions of buildings tend to be predicted correctly, whereas the outer periphery of buildings presented a greater challenge. These results are quantified in Figure 8, which shows F1, precision, and recall values for each of the noise levels below (see Table S2 for terminology description). The precision falls more slowly than recall (and even increases for noisy drops), which is consistent with the pattern of errors observed in the prediction plots. Pixels that are predicted as building tend to be in the central portion of buildings, leading to high precision. Figure 7: _Cont_.