# Accounting for Training Data Error in Machine Learning Applied to Earth Observations
## 5 Conclusions  
Current practices in EO research are generally inattentive to the need to evaluate and communicate the impact of TD error on ML-generated maps. This oversight undermines the goals of scientific reproducibility and may compromise the insights drawn from the resulting maps. Improving these practices is important due to the increasing use of TD-intensive ML algorithms, which have motivated our review and recommendations. To resolve terminological differences arising from the influence of non-EO disciplines, and to help contextualize TD considerations relative to established map accuracy assessment practice, we distinguish between four types of \"truth\" data used in ML-based mapping projects (training, validation, training reference, and map reference data), and define the appropriate role for each (Section 1.2). We identify causes of error in TD as well as map reference data, distinguishing where these vary (Section 2.1). We then explore the impacts of TD error (Section 2.2) and provide a set of case studies to illustrate the consequences of such error across a range of ML-based mapping applications (Section 3). We then provide a set of guidelines for minimizing error arising from the design and collection of TD samples, and present recommendations for measuring and accounting for the impact of these errors (Section 4). Many of these guidelines and procedures also relate to map reference data generation, and we ground our recommendations in the existing best practices for map accuracy assignment (Sections 1.2.1 and 4.1). We conclude by defining three tiers of TD error accounting and reporting standards, which are designed to accommodate a wide range of ML-based mapping projects. The highest tiers should be adopted when creating open training libraries and public map products, both of which are increasingly being developed to meet the growing demand for EO-derived maps. In this context, there is a pressing need to rigorously evaluate the training requirements and relative performance of deep-learning models as they become more widely used for EO [36]. While TD is more visible in the context of LCLU and other categorical mapping projects, the need for rigorous, well-documented TD is also critically important for continuous variable applications in Earth System Sciences (e.g., hydrological research [228]). If adopted within the peer-reviewed literature, the standards we propose for TD treatment may improve confidence in scientific findings drawn from map-based research, which can otherwise be confounded by poorly quantified map errors [33; 57]. The following are available online at [[http://www.mdpi.com/2072-4292/12/6/1034/s1](http://www.mdpi.com/2072-4292/12/6/1034/s1)]([http://www.mdpi.com/2072-4292/12/6/1034/s1](http://www.mdpi.com/2072-4292/12/6/1034/s1)), Figure S1: Sample prediction results in Kumasi, Ghana. (a) Input imagery. (b) Predictions from the Las Vegas model. (c) Predictions from the Khartoum model. (d) Prediction from the Kumasi model. (e) Predictions from the Khartoum Model retrained in Kumasi. Figure S2: Schematic of product selection using the Triple Collocation approach. Table S1: List of peer-reviewed publications retrieved using Google Scholar search algorithm results. Table S2: Summary of commonly used error metrics. Table S3: Quantitative results of comparing each of the three models trained for the road detection case in Kumasi, Ghana to the validation labels. Table S4: Template and procedure for documenting training data. **Author Contributions:** This article synthesizes the ideas of the 20 authors resulting from a workshop focused on issues of error in training data for Machine Learning approaches in Earth Observation research. Conceptualization, L.E. and A.E. ; formal analysis and investigation, L.E., H.A., R.A., J.R.E., L.F., D.K., D.L., A.B.R., L.S., Y., Z.-F.Y. ; writing--original draft preparation, A.E., L.E., H.A., L.F., D.K., D.L., R.G.P.J., A.B.R., Z.-F.Y.