# Modernising Receiver Operating Characteristic (ROC) Curves +
## Appendix A ROC, Item Response Theory, Control Charts, Rankits and Other Measures  
### ROC in the Presence of Counted-Fractions Ordinality  
A generic version of a ROC plot is given by [PERSON] [11] for the case of a binary decision \(A\) or \(B\) about an entity subject to conformity assessment based on a signal (\(SN\) with noise) in the presence of noise, \(N\), where the vertical axes of the plot can be either the probability \(P(A|SN);0\to 1\) or \(P(B|SN);1\gets 0\) while the horizontal axes can be either \(P(A|N);0\to 1\) or \(P(B|N);1\gets 0\). This description has wide applicability, including for both pregnancy testing (as in the present case study) as well as in assessing the performance of machine learning classification algorithms [9]. These probabilities can be calculated from the raw testing data (exemplified in Table 1, with expressions such as those given in the following \"decision matrix\" (1)) (Figure 11). A traditional ROC curve is produced by calculating the probabilities \(P(A|SN)\) and \(P(B|SN)\) at each assigned value (hCG (IU/L) in the present case study) to yield estimates of TPR and FPR, respectively. In making the summations, each local \(SL\) is allowed to \"sweep successively over\" the complete range of concentration levels for each device type from the actual concentration level and upwards [11]. ROC curves for each device are then plotted with TPR against FPR, shown for the present case study in Figure 1.  
A poorly performing instrument would have a ROC curve as a 45\({}^{\circ}\) straight line between the coordinates (0,0) and (1,1). \(AUC\) (area under curve) values are traditionally used to estimate relative classification performance, calculated as in [21]:  
\[AUC=\big{(}FPR_{j}-FPR_{j+1}\big{)}\cdot TPR_{j}\]  
or in [22]:  
\[AUC=\frac{\big{(}FPR_{j}+FPR_{j+1}\big{)}}{2}\cdot\big{(}TPR_{j+1}-TPR_{j} \big{)}\]  
Other diagnostic accuracy indices regularly used in ROC analysis include:  
* The Youden index, which is a measure of the vertical distance between the 45\({}^{\circ}\) line (Section 3.1) and the corresponding point on the ROC curve, as given by the expression \(Youdenindex=sensitivity+specificity-1\). A better diagnosis has a higher Youden index value [23]
* _Sensitivity + Specificity_, which is a simple addition where a better diagnosis has a higher index value. * _Distance to corner_, a better diagnosis which has a smaller distance, \(d\), to the top-left corner of the ROC curve for each cut-off value, \(d=\sqrt{\left(1-sensitivity\right)^{2}+\left(1-specificity\right)^{2}}\) All of these and similar traditional ROC performance indices unfortunately do not take account of the effects of uncompensated ordinality, particularly the well-known non-linearity of data scaling due to the effects of so-called 'counted fractions'; put simply, any score \(X_{j}\%_{\_}{\_}{\_}{\_}{\frac{X_{j}}{k-1},X_{\_}{k}}\%\) on a fixed scale bounded by 0 and 100% becomes increasingly non-linear at either extreme [5]. A difference of, for example, 1% percentage points in score \(X\) is a different amount at mid-scale (50%) compared with scores at 5% or 95%. Classic test theory (CTT) does not compensate for 'counted fractions' ordinality of percentage raw scores nor makes a separation of classifier ability from task difficulty which are confounded in the raw data, as described in Appendix A.2. Merely using a non-parametric test, such as Kruskal-Wallis or other ranking tests, to deal with any ordinality instead of adopting modern measurement theory can make it challenging to detect significant correlations, for instance those between cognitive scores and biomarkers, or in assessing classification performance as studied here with ROCs. The under-estimation due to counted fractions at the high end of a cognitive scale [24], for instance, can easily be misinterpreted in terms of other effects, such as 'cognitive reserve'. Rankits are considered further in Appendix A.3.