# Accounting for Training Data Error in Machine Learning Applied to Earth Observations
## 2 Sources and Impacts of Training Data Error
### Sources of Training Data Error
#### 2.1.2 Collection-Related Errors  
There are several common forms of error that occur when collecting both TD and map reference data. The first of these are errors of interpretation [39], which are mistakes created in the process of manual image interpretation. Image interpretation is widely used to generate TD, but often this technique leads to inconsistent labels between interpreters for the same areas of interest [34; 37; 99; 155]. Interpreters may lack experience in the task or be unfamiliar with the context of the study area, e.g., [156]. In an unusually thorough analysis of error in image interpretation, [PERSON] et al. [99] showed that inter-interpreter agreement was on average 86% but ranged from 46 to 92%, depending on land cover. This research, which relied on trained image interpreters, concluded that transitional land cover classes produce substantial interpretation uncertainty, which is particularly problematic since much land cover mapping effort is directed towards change detection. Another image interpretation study that used a crowdsourcing platform found that interpreters' average accuracy in digitizing crop field boundaries in high-resolution imagery was \(\sim\)80%, based on comparisons against training reference data [150]. This result held true whether the interpreters mapped several hundred sites or \(<\)50 (Figure 2), indicating that increased interpreter experience does not necessarily eliminate labeling error, even when analysts are highly seasoned [99]. These findings underscore the need to assess uncertainty in TD, as well as map reference data, using predefined training reference data or inter-interpreter comparisons [46; 60; 99; 157; 158]. Labeling error may also result from inadequate or poorly communicated semantic class definitions [159; 160], particularly when identifying human land use as opposed to biophysical land cover [161]. This is especially evident in urban environments, which exhibit high spatial and spectral heterogeneity (even within HR/VIR imagery [162]), and are also semantically vague (i.e., hard to define) even at the ground level. For example, Figure 3 shows a typical example of TD collection for mapping informal settlements (i.e., slums), in Nairobi, Kenya, in which several trained interpreters separately delineate the same area [163]. Because slums may be defined by sociodemographic factors in addition to spatial and spectral properties, TD creation for such areas is prone to error stemming from semantic issues [160]. Complex classes such as slums may exhibit high variability between study areas, as local idiosyncrasies link the definition of slums to different physical, remotely observable characteristics. These characteristics make it hard to develop a generalizable mapping capability for land uses such as informal settlements. These results further illustrate the importance of consensus mapping for image interpretation, particularly for spatially, spectrally, or temporally heterogeneous LCLU classes, which may have vague or regionally idiosyncratic semantic definitions. Categorical mapping projects typically define a crisp set of non-overlapping categories, rather than a fuzzy set [164; 165]. However, many human and natural land covers exhibit continuous gradation between classes, implying that crisp map legends will necessarily cause semantic ambiguity when image pixels in transitional areas are labeled [166; 167]. This problem is particularly acute with moderate- and coarse-resolution imagery [26]. Local variance is highest when scene objects approximate the spatial dimension of the image resolution, leading to poor classification accuracy [168]. While substantial research has been devoted to the issue of mixed pixels [137; 169; 170; 138; 167; 139; 138], crisp categories are still often relied on during the training and testing phases of image classification [172]. Alternative approaches based on fuzzy set theory are available, but have seen limited adoption [165; 173].