# Accounting for Training Data Error in Machine Learning Applied to Earth Observations
## 1 Introduction  
Recent technological advancements have led to a new era in Earth observation (EO, also known as remote sensing), marked by rapid gains in our ability to map and measure features on the Earth's surface such as land cover and land use (LCLU), e.g., [1; 2], vegetation cover and abundance [3], soil moisture [4], infrastructure [5; 6], vegetation phenology [7; 8; 9], land surface albedo [10; 11; 12], and land surface temperature [13; 14]. The resulting data are used by an expanding set of disciplines to gain new insights into socioeconomic and environmental dynamics, such as community-level poverty rates [15], changes in surface water [16] and forest cover [17], and carbon accounting [18]. As such, EO is increasingly shaping our understanding of how the world works, and how it is changing. These breakthroughs are facilitated by several technological advances, particularly the increasing availability of moderate (5-30 m), high-resolution (1-5m, HR), and very high resolution (<1 m, VHR) imagery, as well as new machine-learning (ML) algorithms that frequently require large, high quality training datasets [19; 20; 21; 22; 23; 24]. Large training datasets have been necessary for decades in the production of continental and global maps [1; 2; 25; 26]. In the current data-rich era, the impact of training data (TD) quality and quantity on map accuracy is even more relevant, especially for maps generated by data-hungry ML algorithms [27; 28; 29; 30; 31; 32]. Errors in these products also impact the veracity of any downstream products into which they are ingested [33]. While progress in algorithmic performance continues apace, standards regarding the collection and use of TD remain uncoordinated across researchers [34]. Additionally, much of the research and development of big data and ML is occurring in industry and the fields of computer science and (non-spatial) data science, leaving a potential knowledge gap for EO scientists [35; 36]. The measurement and communication of map accuracy is a mature topic in EO and related fields, with a variety of metrics and approaches tailored to different data types, analyses, and user groups [37; 38; 39; 40; 41; 42; 43; 44; 45]. This includes substantial work to measure error in map reference data (i.e., the independent sample used to assess map accuracy) and account for its impact on map assessment [38; 46; 47; 34]. However, focus on the quality and impacts of TD error has been less systematic. While several efforts have been made to use and evaluate the impact of different aspects of TD quality (noise, sample design, and size) on classifiers [30; 32; 48; 49; 50; 51; 52; 53], much of this work focuses on exploring these issues for specific algorithms [31; 48; 53; 54]. Previous research shows that the impact of TD error can be substantial but varied, suggesting that a more comprehensive approach to this issue is warranted. Furthermore, while TD and map reference data are often collected using the same approaches [55; 56; 57] and often subject to the same errors, the existing procedures to minimize and account for map reference errors [38; 46; 47; 34] are not necessarily relevant for quantifying the impacts of TD error. The problems associated with TD error can be summarized as follows:  
1. The \"big data\" era vastly increases the demand for TD. 2. ML-generated map products rely heavily on human-generated TD, which in most cases contain error, particularly when developed through image interpretation. 3. Uncertainty in TD is rarely assessed or reported, and TD are often assumed to have perfect accuracy [30] (which is also common with map reference data [57]). 4. TD errors may propagate to downstream products in surprising and potentially harmful ways (e.g., leading to bad decisions) and can occur without the map producer and/or map user's knowledge. This problem is particularly relevant in the common case where TD and reference data are collected using the same methods, and/or in cases where map reference data error is not known or accounted for, which is still common [57].