# Review of Remote Sensing Methods to Map Coffee Production Systems
## 3 Results
### Accuracy Assessment  
Map validation is a crucial component of any remote sensing classification methodology, as it assesses the success and reliability of the product and subsequent analysis. In this section, we describe the validation methods used in the literature to determine the accuracy of classification results. Due to the different approaches used for validation (e.g., considerable heterogeneity in the number of classes mapped, the validation sampling strategies, and sample size), the accuracy results cannot be directly compared across studies. Instead, we summarize studies by key characteristics of an accuracy assessment, specifically: sampling design, reference data, and analysis as outlined by [PERSON] et al. [95]. Supplementary Table S3 provides a complete list of the validation methods used in the 43 papers examined in this review.  
Sampling design or selecting the types of validation data and the number of validation points for the assessment, is the first step of an accuracy assessment. The number of validation data points used for each study in our review varied widely. [PERSON] et al. [55] used only 43 data points in total (10 data points for the coffee class), whereas [PERSON] et al. [93] used 1754 data points. It is important to note, however, that the size of the study areas in these two classifications were drastically different. [PERSON] et al. [55] classified a small area in southwestern El Salvador, while [PERSON] et al. [93] classified the entire African continent. This review identified three main sampling strategies in the literature. First, 13 studies performed sampling using their classified land cover maps. Of these studies, 5 used a random sampling method, 5 used a stratified sampling method, 2 papers targeted points based on areas deemed to be inaccurate, and 1 paper did not use sample points, but instead performed an accuracy assessment of the entire referenced classification. [PERSON] et al. [95] recommend a stratified sampling to limit biases caused by less prevalent land cover classes. An additional 16 studies used a class occurrences dataset collected before the mapping exercise and split the dataset into a training dataset and a validation dataset. Finally, 7 studies used separate independent data sources for both the calibration and validation datasets.  
Response design is the process of determining the reference land cover class in each respective sampling unit [95]. All studies compared classification results to an independent reference dataset to obtain context for the validity of the land cover product. In 4 studies, the reference classes were collected solely through the interpretation of high-resolution imagery. Another 12 studies used data collected in the field, while the final 18 studies used a combination of high-resolution imagery interpretation, field visits, and/or previously collected data. For example, [PERSON] et al. [89] compared remote sensing results to both data collected in the field and survey data at the district level.  
The validation analysis specifies the metrics used to present the accuracy of the product [95]. Our review revealed that researchers employed multiple measures of accuracy. The Kappa index of agreement [96], presented by 28 papers, aims to assess map accuracy by accounting for the random distribution of mapped areas of each class as well as the random distribution of class geolocation. In addition, 30 studies reported a confusion matrix [97] and 31 studies reported overall accuracy. The minimum overall accuracy reported was 18.6% [67], and the maximum was 99.8% [79]. For a complete set of overall, producer's, and user's accuracies, see Supplementary Table S2. Four of the reviewed studies presented an estimation of uncertainty of validation metrics (Supplementary Table S3).