# Accounting for Training Data Error in Machine Learning Applied to Earth Observations
## Guidelines and Recommendations
### Step 3: Minimize Collection-Related Errors  
There are numerous ways to collect TD for categorical and continuous mapping projects, each with their own sources of error. There are thus many potential approaches for minimizing the associated collection errors, which may be quite specific to a particular variable (e.g., for agricultural area estimates [214]). However, there are several general approaches that can be followed to minimize TD collection errors. Our focus here is primarily on error in image-interpreted TD, which is one of the most common approaches used to training ML mapping algorithms. We also touch on the specific case of model-derived training data. Whenever possible, we recommend using protocols that incorporate training reference data to independently assess TD accuracy, particularly for image-interpreted TD, e.g., [150]. Training reference datasets can be limited in size compared to the ultimate sample size, provided that training reference locations are randomly presented to interpreters during the data creation campaign [150]. Active feedback during training label creation can also help reduce errors on a rolling basis by providing interpreters with information regarding their performance [174]. If comparison against training reference data is not possible, then consensus methods for generating TD may be the next best alternative. Consensus between at least 3 interpreters is recommended to allow for majority voting [34; 46], but more complex land covers may require up to 7 interpreters [46]. Consensus among several domain experts may also be the best and most practical measure for collecting both training reference data and map reference data [34; 57]. In the case of image-interpreted samples, consensus approaches should employ multiple interpreters to label the same site. For continuous variables, several independent or repeated in situ measurements should be made and aggregated. For modeled variables where the error is unknown, as in the surface flux case study, training based on the outputs of multiple independent models is recommended. The agricultural case study shows how multiple mappings can be used to quantify label uncertainty (Figure 12A) and minimize the amount of labeling error, yielding improved map accuracy (Figure 11A). The surface flux case study demonstrates these same benefits across several continuous variables (Figure 10). The number of separate measures or interpreters needed will vary depending on the application. For example, in the cropland mapping case study, 5 interpreters labeled each consensus training sample, and in the continuous surface flux example, 3 separate modeled inputs were used. Further steps can be taken to minimize TD collection errors arising from image interpretation. Interpreters should be given thorough training regarding the task [34], which may include instruction on remote-sensing principles as well as local or regional contextual information. Local domain expertise is particularly helpful for consistent identification of idiosyncratic land covers [163]. Interpreter education is particularly important for crowdsourcing or citizen science data collection campaigns, as participants typically lack formal experience in image interpretation [151; 215]. As described in Step 2 above, image interpretation is inadvisable when the available imagery does not support the legend categories in terms of spatial, spectral, temporal, or radiometric resolution [216; 217; 218]. Researchers must be especially cautious in the similar but potentially more hazardous case that HR/VHR imagery is used to create training samples that are then used with coarser resolution imagery when ingested into the ML model.