# Accounting for Training Data Error in Machine Learning Applied to Earth Observations
## 2 Sources and Impacts of Training Data Error
### Impacts of Training Data Error
Mislabeling 25% of TD samples reduced RF accuracy by 3-7% for a binary classifier and 7-10% for a multiclass model, with the ranges in accuracy loss also varying according to TD sample size [48]. Classification certainty was more heavily impacted by label error, dropping by 45-55%, as measured by the mean margin [48]. Other classification models showed larger impacts due to label noise, including 11-41% kappa declines for a kNN (28% label noise [52]), and 24% [126; 181] and 40-43% accuracy loss for a kernel perceptron and NN, respectively, when each is trained with 30% of TD labeled incorrectly [59; 126; 181]. Single decision-tree models were most sensitive to label error, registering 39% to nearly 70% accuracy declines for 30% label noise [59; 126; 181]. The research described above provides substantial information on how TD error can impact the accuracy and certainty of older-generation ML classifiers. Further understanding of the consequences of these errors can be inferred from literature examining the impact of errors in map reference data. Map reference errors can substantially bias areal estimates of land-cover classes, as well as the estimation of variance in those classes, particularly when examining land-cover change [46; 182; 183]. While methods exist to incorporate map reference data error into map accuracy assessments and area estimates [38; 46; 47], and also to account for TD uncertainty in assessing classifier accuracy [48], there has been little work that shows how to address both TD and map reference error. Less information is available regarding the ways in which TD error may propagate beyond the map it initially creates. Initial research by [PERSON] et al. [33] examined how error propagates from a primary land-cover map into subsequent derived products. This work used a high-quality reference cropland map to quantify the errors in 1 km cropland fractions derived from existing land cover datasets and measured how these errors propagated in several map-based analyses drawing on cropland fractions for inputs. The results suggest that downstream errors were in some instances several fold larger than those in the input cropland maps (e.g., carbon stock estimates, Figure 4), whereas in other cases (e.g., evapotranspiration estimates) errors were muted. In either case, the degree to which the error magnifies or reduces in subsequent maps is difficult to anticipate, and the high likelihood that error could increase means that any conclusions based on such land cover-derived maps must be treated with caution when error propagation is not quantified. This analysis suggests how TD errors might impact the maps they generate and provides a potential method for quantifying their impacts on map accuracy. Figure 4: An examination of how error in pixel-wise cropland fractional estimates (expressed as a percentage, top row) can propagate error (expressed as a percentage) in maps that use land-cover data as inputs, such as estimates of carbon density (bottom row). Figure adapted from [PERSON] et al. [33]. The impact of map input errors can also be seen in the practice of using well-known standard datasets, such as the National Land Cover Map (NLCD, [184]), to map quantities of interest, such as urban tree canopy biomass. Urban trees play a crucial role but in regional carbon cycles [185; 186; 187] but are often omitted from EO studies of carbon dynamics, e.g., MODIS Net Primary Productivity [188]. As urban lands are expected to triple between 2000 and 2030 [189; 190], the need to factor them into carbon accounting is pressing, but remotely mapping urban tree cover is limited by (a) spatial resolutions that are too coarse for highly variable urban landscapes and (b) TD that are often biased to forested, agricultural, and other rural landscapes. For these reasons, the Landsat-derived NLCD Percent Tree Cover (PTC) product [191], which estimates canopy cover at 30-m resolution across the US, provides a practical input for empirical models to map tree biomass.