# Accounting for Training Data Error in Machine Learning Applied to Earth Observations
## 3 Case Studies
### Agricultural Monitoring  
Two agricultural cases illustrate how TD error can impact both categorical and quantitative remotely sensed measures. The first relates to cropland mapping and is drawn from an ongoing study focused on mapping smallholder agricultural fields at high spatial resolution (3-4 m) in Ghana. The mapping method is based on \"active learning\", in which a random forest-based [124; 198; 199] ML algorithm is iteratively trained and validated by a crowdsourcing platform. This platform enlists human trainers to visually interpret and digitize field boundaries in the imagery (PlanetScope visual and near-infrared surface reflectance [128]) being classified [149; 150; 198]. During this process, a protocol is used to assess the accuracy of training labels, in which each worker is periodically directed to a training reference site where the field boundaries are already known but are invisible to the worker. Using these training reference sites, the interpreters' maps are then scored using a multi-dimensional accuracy  
Figure 10: Coefficient of determination (R\({}^{2}\)) and root mean squared error (RMSE) of the water, energy, and carbon with artificial neural networks (WECANN) model output against ground measurements from FLUXNET towers in comparison to the three datasets used to generate the target training data for latent heat flux (LE) (**a,b**), sensible heat flux (H) (**c,d**) and gross primary production (GPP) (**e,f**). assessment algorithm [150], resulting in an average TD accuracy score for each worker ranging from 0 (complete disagreement with reference) to 1 (perfect agreement). Each label site is mapped by at least five workers, and the resulting worker-specific accuracy scores are used within a Bayesian merging algorithm to combine the five sets of labels into a single consensus label, which is then used to train the random forest classifier. Here we use the worker-specific training accuracy scores to assess the impact of label quality on map accuracy by assessing three variants of two random forest-generated maps, one over Central Ghana (\(\sim\)3400 km\({}^{2}\)) and one over Northern Ghana (\(\sim\)3100 km\({}^{2}\)). The first two maps were trained using labels generated by the worker with the least accurate TD, the second two by the most accurate worker, and the third using the consensus labels. The accuracy of each pair of maps was then assessed against the validation set (reserved consensus labels) using the true skill statistic [90] (sensitivity + specificity - 1, with scores ranging from \(-\)1 to 1). The results show a substantial difference in accuracy between the maps trained with the least and most accurate workers' labels (Figure 11A), with the former having 7-9% more skill than the latter, while maps based on consensus labels have \(\sim\)3% more skill than those of the most accurate workers' labels. The second case relates to remotely sensed crop estimates of wheat yields collected in 48 smallholder fields in Bihar, India in 2016-17 [200]. Yield data were collected via eight \(2\times 1\) m\({}^{2}\) crop cuts within each field, and PlanetScope-derived green chlorophyll vegetation indices (GCVI) were calculated over each field from imagery collected over four dates during the growing season (13 January, 25 February, 12 March, and 14 April 2017). A random forest regression was trained on the yield measured for each field, using the four dates of GCVI values as predictors. To test the effect of TD error on the resulting yield predictions, three types of noise were artificially introduced into the yield data used for training: (1) a systematic 0.5 ton/ha overestimate with randomly distributed errors sampled from a normal distribution with a mean of 0 ton/ha, (2) random noise with standard deviations of 0.5 ton/ha, and (3) random noise with standard deviations of 1 ton/ha. A baseline model fit to unperturbed data was also developed.