# Best Practices for Applying and Interpreting the Total Operating Characteristic
## 4 Discussion
### Examples of TOC Curves in the Literature  
This subsection examines three particularly revealing journal articles where the authors used some best practices but not others. We did not receive responses when we contacted the authors to help us interpret their publications. One of the best decisions by the authors was to use and show the TOC rather than the ROC, because the insights below would have been impossible using the ROC. Insights from the best practices allow readers to draw conclusions that differ from the authors' conclusions. Figure 7 shows the TOC curves of [PERSON] and [PERSON] (2016) [36]. Their purpose is to compare three algorithms that simulate urban growth. CA-logistic is a cellular automata model that uses logistic regression, which serves as a non-uniform baseline. PSO-CA is a particle swarm optimization algorithm that uses cellular automata. ABC-CA is an artificial bee colony algorithm that uses cellular automata. Their goal is to examine urban growth, not urban presence; thus, it is necessary to eliminate the urban areas at the start of the simulation when generating the TOC curves. The article does not report whether the authors masked the existing urban areas from the spatial extent. The shapes of the TOC curves make us suspect that existing urban areas are included in the spatial extent, which can have a profound effect on the shapes of the TOC curves and the AUCs. All three algorithms generate TOC curves on the uniform line near the origin, which indicates that all three models are as accurate as random for the earliest ranking pixels. Then, the TOC curves tend to run parallel to the left bound of the TOC parallelogram to near the upper left corner of the parallelogram. If the authors did not eliminate the existing urban areas, then those straight segments likely indicate urban persistence rather than urban gain. The tops of those straight segments show more hits than errors, which would imply a Figure of Merit greater than a half, but this contradicts the results in the authors' Table 4. Then, the curves are flat immediately to the right of where the segments are parallel to the left bound. This could indicate that the models are worse than random at predicting the urban gain. Then, the curves show a change in concavity as the slopes of the TOC segments increase, which indicates that the later-ranked pixels are more intensively concentrated in the places of urban gain. If the urban at the start of the validation interval were masked, as best practices dictate, then the models would likely show AUC values of less than a half. Table 4 in [36] compares the three models at distinct thresholds, where the threshold for logistic regression shows the most change while the threshold for ABC-CA shows the least change. These thresholds make it challenging to compare the models because each threshold predicts a distinct quantity. When models simulate less change, they tend to generate fewer false alarms. The ABC-CA threshold simulated the least change, and thus appears to perform best according to the Figure of Merit. The TOC curves are much more helpful for evaluating the models because the reader can compare each model at any particular quantity. Figure 8 presents the TOC curves from [PERSON] et al. (2016), which compare two models of urban growth [37]. ABC-CA is an artificial bee colony algorithm that uses cellular automata. ACO-CA is an ant colony optimization algorithm that uses cellular automata. Figure 8 is similar to Figure 7 concerning the shapes of the curves; thus, many of the comments in Figure 7 apply also to Figure 8. The TOC curves are parallel to the left bound until they reach the thresholds at slightly less than the correct quantity at the upper right corner of the parallelogram. The loss of urban during the calibration interval combined with the lack of masking existing urban would produce this pattern in the TOC curve. Similar patterns exist in other journal articles [34; 35; 38].