# Best Practices for Applying and Interpreting the Total Operating Characteristic
## 1 Introduction
\begin{table}
\begin{tabular}{c p{227.6 pt}} \hline \hline
**ID** & **Description** \\ \hline
1 & Specify the purpose, particularly whether the TOC measures calibration or validation. \\
2 & Report what is known concerning data quality. \\
3 & Show an overlay of the maps at the time points that bound the time intervals. \\
4 & Show maps of the independent and rank variables. \\
5 & Compare to a non-random baseline ranking. \\
6 & Mask pixels that are not candidates for the particular type of change. \\
7 & Describe the sampling scheme and how the method accounts for the sampling. \\
8 & Plot TOC curves including the baseline in the same parallelogram. \\
9 & Consider extent reduction so the curve first touches the upper bound at the right corner. \\
10 & Include threshold markers on the curves. \\
11 & Label relevant thresholds, especially the threshold at the correct quantity. \\
12 & Interpret slopes of the segments of the TOC curves relative to the uniform line. \\
13 & Discusses the reasons for any changes in the concavity of the curves. \\
14 & Investigate the points where TOC curves touch or cross. \\
15 & Zoom into the origin of the TOC parallelogram to interpret early thresholds. \\
16 & Show maps of misses, hits, false alarms, and correct rejections for relevant thresholds. \\
17 & Report exclusively the metric(s) that relate to the research question. \\
18 & Test the sensitivity of results to the threshold selections. \\
19 & Avoid stating model performance in simple universal words such as “good”. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Best practices when using the Total Operating Characteristic. characteristics that TOC curves do not necessarily show. Practice 5 is important because a temptation can be to use randomness as a baseline when randomness is uninteresting. An example of a non-random baseline for a land change simulation model is to predict that the future gain of a class will occur near the edge of the existing class. Practice 6 is important when the purpose is to predict exclusively the gain of a class, in which case pixels that show the class at the start of the prediction are not candidates for gain, and thus must be masked from the analysis. Practice 7 is important especially for machine learning algorithms that use sampling. Practice 8 is essential to see where TOC curves cross. Practice 9 is important because the points are on the upper bound to the right of the point where the curve touches the upper bound. Practices 10-11 will inspire authors to avoid smoothing the curves; smoothing makes readers blind to the thresholds. Practice 12 inspires the interpretation of the slope of each segment as the intensity of the presence within each segment of the TOC curve. Practice 13 relates to 12 because concavity is related to slope. Practice 14 is important when comparing various methods of classification. Practice 15 is relevant when the thresholds near the origin are most relevant to the research question. Practice 16 reveals spatial relations that the TOC curves do not show. Practice 17 is crucial because many irrelevant metrics exist. Authors must not report metrics merely because the metric is popular. Practice 18 is essential when the algorithm does not choose each unique rank as a threshold. Practice 19 will force authors to think deeply concerning the precision of the research question. Our literature search found publications by early adopters of TOC curves. Several articles demonstrate numerous best practices; in particular, [PERSON] et al. (2018) interpreted the shape of TOC curves in their comparison of several algorithms to a non-random baseline model that predicted deforestation in Borneo (2018). Other applications would have benefitted from applying the best practices. Some publications show the ROC or TOC curves but do not show the thresholds, in which case interpretation is difficult ([PERSON], 2018; [PERSON], 2018).