# Accounting for Training Data Error in Machine Learning Applied to Earth Observations
## 2 Sources and Impacts of Training Data Error
### Impacts of Training Data Error  
TD errors carry through to impact the map production process and outcomes. From a design perspective, the size and class composition of TD is particularly impactful on ML algorithms, which are susceptible to overfitting and class imbalance problems [31; 73]. Additionally, the assumption of representativeness of training pixels is often overstated, and many TD may in fact not be generalizable to broader scales (discussed by [PERSON] et al. [154]). TD errors arising from the collection process also impact map quality. Both design- and collection-related errors may be particularly hard to discern, or quantify in absolute terms, if the error in the map reference data errors are unknown. Several studies reviewed in Section 1.2.2 provide insight into how much TD error can impact ML-generated land-cover maps, focusing on aspects of sample size and balance (design-related errors) and labeling error (collection-related error). This work shows that the impact of each error source varies according to the algorithm used. For example, SVMs were relatively insensitive to changes in sample size, with accuracy dropping by only 3%-6% under TD size reductions of 85-94% [28; 180]. Random forests (RF) also proved robust to TD sample size, showing slightly higher accuracy drops of \(\
eg\)4-10+% when TD was reduced by 70-99% [48; 51; 180]. Sample size also impacts the certainty of RF classification by lowering the mean margin (a measure of certainty related to the number of class votes) by \(\sim\)50% for sample size reductions of 95% [48]. In contrast to SVM and RF, maps classified with single decision trees are highly affected by TD size, with 13% accuracy loss for TD reductions of 85% [28], and up to 50-85% loss with TD size reductions of 50-70% [51; 59]. NNs show varying responses to sample size, depending on their algorithmic design: one NN based on adaptive resonance theory showed accuracy reductions of \(\sim\)30% to \(\sim\)65% when TD samples were halved [59], while a feed-forward NN lost just 2% accuracy when TD was reduced by 85% [28]. Classifiers are also sensitive to class balance within the training data. For example, the accuracy of RF-generated maps declined by \(\sim\)12% to \(\sim\)23% and classification confidence fell \(\sim\)25% to \(\sim\)50% when TD class balances were highly skewed [48]. Notably, the ranges in these accuracy and confidence declines were attributable to differing TD sample sizes, showing the synergistic effect of sample size and class balance sensitivities. [PERSON] et al. [31] provide a more comprehensive review of class imbalance for RF, SVM, NN, and k-nearest neighbors (kNN) classifiers, finding that all models were sensitive to class imbalance, but the accuracy impact was largest for rare classes, as opposed to overall map accuracy. The impact of TD labeling errors, also referred to as noise, varies substantially between mapping algorithms. SVMs and closely related derivatives appear least sensitive to mislabeling. SVMs lost just 0-5% in land-cover classification accuracy when 20-30% of TD samples were mislabeled either randomly or uniformly across classes [30; 52; 126]. Relative vector machines (RVMs) were even less sensitive under these conditions (2.5% accuracy loss for 20% mislabeling [30]), and an SVM designed specifically for handling noisy TD (context-sensitive semi-supervised SVM) was even more robust (2.4% reduction in kappa for 28% mislabeling [52]). However, the impact of TD noise was greater for all three models when mislabeling was confined to specific classes. SVMs lost 9% accuracy and 31% kappa when 20-28% of samples in spectrally similar classes were mislabeled [30; 52]. The RVM showed a 6% accuracy loss [30], and the specialized SVM showed a 12% kappa reduction [52] under the same conditions. As with sample size, RF is the next least sensitive to TD noise [48; 51].