# Accounting for Training Data Error in Machine Learning Applied to Earth Observations
## 2 Sources and Impacts of Training Data Error
### Sources of Training Data Error  
#### 2.1.1 Design-Related Errors  
With respect to TD sampling design, errors primarily stem from failures to adequately represent the spatial-temporal-spectral domains of the features of interest in the manner most suited to the specific ML algorithm being used [53]. This problem may be exacerbated in cases where TD are collected exclusively using the same rigorous probability-based specifications used to collect map reference data, which may be overly restrictive for the purposes of TD collection. While the use of such standards to collect TD may be possible provided that there is a large enough data set (e.g., a large benchmark data set), smaller training data sets and/or cases of geographically sparse target classes/objects will benefit strongly from the increased flexibility afforded to TD collection standards, which are less restrictive than those for map reference data (e.g., allowing for purposive rather than purely probabilistic sampling). A lack of geographic representation of the phenomena of interest results in a disparity between the distribution of TD compared to the true distribution of the mapped phenomenon in geographic and/or feature space [28; 29; 30; 31]. This problem is highly relevant in ML approaches, which are sensitive to TD quality, including class balance, labeling accuracy, and class comprehensiveness relative to the study area's true composition [30]. Temporal unrepresentativeness is also a common source of error in the response design of TD, due to the prevalence of image interpretation as a source for TD. In this case, error arises when obsolete imagery is interpreted to collect training points or polygons and their associated labels [39; 61]. The problem is illustrated in Figure 1, which contrasts smallholder fields that are clearly visible in a satellite base map (Bing Maps) with ground data collected in 2018. Center pivot fields were installed after the base map imagery was collected, but before ground data collection, causing a temporal mismatch between the base map and the in situ data. Labels generated from the base map would therefore introduce substantial error into an ML algorithm classifying more recent imagery. New HR/VHR satellites that have more frequent acquisitions (e.g., PlanetScope [128]) can help minimize such temporal gaps for projects that are designed to map present-day conditions (e.g., 2018 land cover), but cannot solve this problem for mapping projects covering earlier time periods (i.e., before 2016). The same can be said for aerial and unmanned aerial vehicle acquisitions, which are typically limited in geographic and temporal extent [129]. While hardcopy historical maps can help supplement temporal data gaps, these data sources come with their own problems, such as errors introduced during scanning and co-registration, and unknown production standards and undocumented mapping uncertainties. Spatial co-registration can be a substantial source of response design error when training with HR and VHR commercial satellite imagery. Due to their narrow swath widths, HR/VHR sensors are often  
Figure 1: An example of potential training data error that can arise when image interpretation is conducted on older imagery. The underlying imagery is from Bing Maps, which shows smallholder agricultural fields near Kulpawn, Ghana. The white polygons were collected by a team of mappers (hired by Meridia) on the ground using a hand-held Global Positioning System (GPS) in 2018. The smallholder fields were replaced by larger center-pivot irrigation fields sometime after the imagery in the base map was collected. tasked, resulting in substantially off-nadir image acquisitions [61]. Due to large view zenith angles and the lack of adequate digital elevation models, side overlapping imagery for stereo photogrammetry, or other relevant control points, HR/VHR imagery often does not meet the same orthorectification standards as coarser resolution, government-operated satellites [130; 131; 132].