# Efficacious End User Measures—Part 1: Relative Class Size and End User Problem Domains

Authors: E. Earl Eiland; Lorie M. Liebrock
Journal: Advances in Artificial Intelligence
 (2013)

---

## Abstract

Biological and medical endeavors are beginning to realize the benefits of artificial intelligence and machine learning. However, classification, prediction, and diagnostic (CPD)
errors can cause significant losses, even loss of life. Hence, end users are best served when they have performance information relevant to their needs, this paper’s focus. Relative class size (rCS) is commonly recognized as a confounding factor in CPD evaluation. Unfortunately, rCS-invariant measures are not easily mapped to end user conditions. We determine a cause of rCS invariance, joint probability table (JPT) normalization. JPT normalization means that more end user efficacious measures can be used without sacrificing invariance. An important revelation is that without data normalization, the Matthews correlation coefficient (MCC) and information coefficient (IC) are not relative class size invariants; this is a potential source of confusion, as we found not all reports using MCC or IC normalize their data. We derive MCC rCS-invariant expression. JPT normalization can be extended to allow JPT rCS to be set to any desired value (JPT tuning). This makes sensitivity analysis feasible, a benefit to both applied researchers and practitioners (end users). We apply our findings to two published CPD studies to illustrate how end users benefit.


---

## Body

## 1. Introduction

Biological compounds and systems can be complex, making them difficult to analyze and challenging to understand. This has slowed applying biological and medical advances in the field. Recently, artificial intelligence and machine learning, being particularly effective classification, prediction and diagnostic (CPD) tools, have sped applied research and product development. CPD can be described as the act of comparing observations to models, then deciding whether or not the observations fit the model. Based on some predetermined criterion or criteria, a decision is made regarding class membership (x∈A or x∉A). In many domains, class affiliation is not the end result, rather it is used to determine subsequent activities. Examples include medical diagnoses, bioinformatics, intrusion detection, information retrieval, and patent classification. The list is virtually endless. Incorrect CPD output can lead to frustration, financial loss, and even death; correct CPD output is important. Hence, a number of CPD algorithms have been developed and the field continues to be active.Characterizing CPD effectiveness, then, is necessary. For example, CPD tool developers need to know how their particular modification affects CPD performance, and practitioners want to make informed choices between CPD options before deploying a tool in the field.1 Jamain and Hand, summarizing their results in a classifier meta-analysis, comment: The real question a user generally wants to answer is “which classification methods [are] best for me to use on my problem with my data …”  [1].This question has not been addressed in studies we have read. Indeed, Jamain and Hand generalize the sentiment of R.P.W Duin’s comment regarding comparing automated, heavily parametrized classifiers.It is difficult to compare these types of classifiers in a fair and objective way [2].Seemingly, the research community has viewed the end user’s need as too complex to address. Thus, for the most part, researchers have focused on addressing their own needs. End user issues, when discussed, have been constrained to specific problem domains. It might be fair to state that each end user’s need is, in some way, unique. However, that does not mean that the apparent complexities faced by end users cannot be identified and managed. Ideally, a means of satisfying end user needs without sacrificing researcher needs will emerge. At a minimum, it should be possible for end users to be enlightened regarding measure suitability (which measures best quantify how a CPD will impact their situation). This paper is a first step in identifying a general structure of CPD problems2 faced by end users and using that structure to identify CPD evaluation measures and tools relevant to end users. To the extent that research studies present CPD performance information by which end users can estimate impact in their situation, the studies provide improved service to the end user.Our primary focus is on summary statistics. In the current context, summary statistics are formulae that take measurement suite elements as input3 and output a single value which represents the target CPD’s overall quality. However, because multiple values are condensed into a single value, information is lost. To the extent essential information is retained, the summary statistic can prove useful for CPD evaluation. A key characteristic of summary statistics is that they are not monotonic; they have optima. Useful summary statistic optima indicate overall classifier quality. Ideally, these summary statistics also quantify some aspect of classifier output efficacious to end users. End users can directly use such values to estimate how the CPD will impact their situation.As presented by Hand [3], measurement theory distinguishes between two entity attribute types: intrinsic, those that are part of an entity’s definition (e.g., density or mass) and extrinsic, those that are expressions of the entity’s interaction with the environment (e.g., weight). Attributes such as density and weight can be quantified, so we can also talk about intrinsic and extrinsic measures. When reported in joint probability tables (JPTs), CPD output is partitioned into four distinct categories: T+, F+, F-, and T-. After any dataset has been tested, the final object count in each category is influenced by the environmental factors rCS and boundary (B). (rCS is the relative sizes of the classes in the test set (rCS=Y¯/Y). B is an n element vector that defines a “surface” that encloses one class, for example, “class A.” In every case, there will also be an optimum boundary4 (B*). All elements outside that surface are in class “A¯”, rather than class “A.” Because T+, F+, F-, and T- are sensitive to rCS and B, they are extrinsic measures.


### 1.1. Nomenclature

Although this paper applies well-established stochastic concepts, not all discussions use the same terminology. To avoid confusion, we define our lexicon for quantities measured (each being the size of the defined set):T+: correctly identified events in class A, the “class of interest” (if such a class exists);T-: correctly identified events of class A¯, the other class;F+: class A¯ events incorrectly flagged as class A;F-: class A events incorrectly flagged as class A¯;Z: events flagged as class A;Z¯: events flagged as class A¯;Y: actual class A events in the data set;Y¯: actual class A¯ events in the data set;N: the data set;These values are often presented in an JPT as in Table 1. When appropriate, these symbols will also be used to represent populations. Context will determine whether a quantity or a population is being referenced.Table 1
Values in the lexicon are often organized into a joint probability table (JPT), such as this.
Actual target classification
A
A
-
Totals↓
Testresult
Positive
T
+
F
+
Z
Negative
F
-
T
-
Z
-
Totals
Y
Y
-
NEnd users are interested in how a process will function in their environment, so they need measures sensitive to extrinsic factors. From a purely academic perspective, the goal for many researchers is to characterize the CPD process independent of extrinsic factors; thus, they want intrinsic measures. Presumably because of the immediacy of the need, significant progress has been made in identifying and characterizing intrinsic measures.5 We are interested in extrinsic measures useful for end users; little attention has been paid to their needs.Because of the disparity between researcher and end user needs, providing for end user needs requires careful consideration. A researcher is interested solely in CPD performance; effects caused by external factors must be accounted for, if not eliminated. In contrast, end users need to incorporate external factors, not compensate for or eliminate them. Thus, in order to have research reports that are readily applicable by end users may require providing values that hold little relevance for researchers. We propose an “end user efficacious” measure suite and a means by which end users can tailor research results to their specific environment.This study builds on Sokolova et al. and other CPD summary statistic characterization studies [4–14]. A challenge categorical problem evaluators face, when comparing to CPD results reported by others, is adjusting for data set effects. One of the major data set issues is that test sets used may well have different rCSs, with different applicability and/or utility. This can cloud results. As an example, we ran a CPD on two test sets drawn from the same class populations. Since both the class source populations and CPD were the same for each test, one would expect statistically indistinguishable output. However, since JPT categories are extrinsic, the anticipated similarity may be masked. The only difference between the test outputs shown in Table 2 is one test set has a relative class size of 9 : 1 (rCS=9) and the other a relative class size of 1 : 1 (rCS=1). The CPD performs equally well in each test; however, rCS introduces a bias in the JPTs that makes the CPD performance equality difficult to recognize. When rCS=1, there are twice as many F- observations as F+. However, when rCS=9, the ratio between F- and F+ goes to 18 : 1! Without knowledge of the test sets used, an observer could well conclude that these were two significantly different CPDs, with significantly different applicability and/or efficacy. This is an obvious problem for researchers, thus significant effort has been applied to mitigate it; a selection of rCS invariant measures are available: (i)
the Youden index [15];(ii)
two related measures, diagnostic odds ratio (DOR) [16] and diagnostic power (DP) [17] (DP=(3/π)log(DOR);(iii)
the Matthews correlation coefficient (MCC) [18];(iv)
the receiver (or relative) operating characteristic6 area under the curve (AUC) [19, 20];(v)
information theoretic measures such as the information coefficient (IC) [21, 22].Table 2
This table shows the totalF+ and F- for two tests with the same CPD on equally sized data sets (2250 observations), drawn from the same populations. The only difference is the samples have different rCSs. Because the tests were run on data sets with different sized classes, the equivalence of the CPD’s effectiveness is not obvious. It would be easy for an observer to erroneously conclude the CPDs were significantly different.
Relative class size
F
+
F
-
1 : 1
125
250
1 : 9
25
450Youden was addressing the rCS’s biasing effect in 1950; thus, the problem has been known for well over half a century, yet reports regarding mitigation are still entering the literature [9, 23]. In the works reviewed, consideration of end user problem environment was tightly constrained and the view of the data virtually unrestricted. We invert these criteria; first identifying problem interactions with rCS (a broad view of end user needs), then viewing the data such that it addresses the question posed by Jamain and Hand (a constrained view of the data).rCS is generally confounding in the research environment; this is presumably also true for some end users. However, for other end users, rCS may be important for their problem. For these end users, basing decisions on rCS invariant measures may be misleading. Hence, we start by asking two questions:(i)
“Is rCS important for all end user CPD problem domains?”(ii)
“If not, what characteristics define when to incorporate rCS?” Consider relating these two questions to a pair of real-world problems. A less effective CPD used with a rheumatoid arthritis test could lead to either more people than necessary being treated, or fewer. Likewise, a poorly selected intrusion detection boundary could cause an IT system to have excessive errors (false alarms or missed attacks).In order to consider the two questions posed above, we will use a statistical nomenclature to describe a supervised CPD test bed. Viewed from a statistical perspective, observations on the dataset processed provide estimates of the underlying (class) population probabilities (rCS is the odds expression of that probability). Observations in the test system input are drawn from the specific populations (because the source class populations are known for each observation, a “ground truth” exists). The source population relative class sizes can be represented as a probability, for example, the probability that a randomly selected input will be a member of classA. This is the leading probability (Pleading), the probability before the inputs interact with the defined process.7 In the examples stated, class A members would consist of RA-positive individuals and malicious information system activity. In the field, “ground truth” for any particular individual cannot be known prior to being processed (otherwise there would be no need for evaluation). However, in the test scenario being described, ground truth is known for each test set member. Since the source class is known for each CPD input element, input uncertainty does not affect any individual CPD output.The balance of this paper is organized as follows. Section2 considers relative class size. Section 3 describes the research protocol used in this study. Section 4 discusses efficacious measures for end users and considers existing summary statistics. Section 5 considers a cause of rCS invariance in measures and implications thereof. This is followed in Section 6 where we present two examples using the proposed format and tool. The main body of this paper closes with a summary of our findings and presents future work in Section 7. Four appendices with equation derivations and additional JPT normalization details wrap up the paper.


## 1.1. Nomenclature

Although this paper applies well-established stochastic concepts, not all discussions use the same terminology. To avoid confusion, we define our lexicon for quantities measured (each being the size of the defined set):T+: correctly identified events in class A, the “class of interest” (if such a class exists);T-: correctly identified events of class A¯, the other class;F+: class A¯ events incorrectly flagged as class A;F-: class A events incorrectly flagged as class A¯;Z: events flagged as class A;Z¯: events flagged as class A¯;Y: actual class A events in the data set;Y¯: actual class A¯ events in the data set;N: the data set;These values are often presented in an JPT as in Table 1. When appropriate, these symbols will also be used to represent populations. Context will determine whether a quantity or a population is being referenced.Table 1
Values in the lexicon are often organized into a joint probability table (JPT), such as this.
Actual target classification
A
A
-
Totals↓
Testresult
Positive
T
+
F
+
Z
Negative
F
-
T
-
Z
-
Totals
Y
Y
-
NEnd users are interested in how a process will function in their environment, so they need measures sensitive to extrinsic factors. From a purely academic perspective, the goal for many researchers is to characterize the CPD process independent of extrinsic factors; thus, they want intrinsic measures. Presumably because of the immediacy of the need, significant progress has been made in identifying and characterizing intrinsic measures.5 We are interested in extrinsic measures useful for end users; little attention has been paid to their needs.Because of the disparity between researcher and end user needs, providing for end user needs requires careful consideration. A researcher is interested solely in CPD performance; effects caused by external factors must be accounted for, if not eliminated. In contrast, end users need to incorporate external factors, not compensate for or eliminate them. Thus, in order to have research reports that are readily applicable by end users may require providing values that hold little relevance for researchers. We propose an “end user efficacious” measure suite and a means by which end users can tailor research results to their specific environment.This study builds on Sokolova et al. and other CPD summary statistic characterization studies [4–14]. A challenge categorical problem evaluators face, when comparing to CPD results reported by others, is adjusting for data set effects. One of the major data set issues is that test sets used may well have different rCSs, with different applicability and/or utility. This can cloud results. As an example, we ran a CPD on two test sets drawn from the same class populations. Since both the class source populations and CPD were the same for each test, one would expect statistically indistinguishable output. However, since JPT categories are extrinsic, the anticipated similarity may be masked. The only difference between the test outputs shown in Table 2 is one test set has a relative class size of 9 : 1 (rCS=9) and the other a relative class size of 1 : 1 (rCS=1). The CPD performs equally well in each test; however, rCS introduces a bias in the JPTs that makes the CPD performance equality difficult to recognize. When rCS=1, there are twice as many F- observations as F+. However, when rCS=9, the ratio between F- and F+ goes to 18 : 1! Without knowledge of the test sets used, an observer could well conclude that these were two significantly different CPDs, with significantly different applicability and/or efficacy. This is an obvious problem for researchers, thus significant effort has been applied to mitigate it; a selection of rCS invariant measures are available: (i)
the Youden index [15];(ii)
two related measures, diagnostic odds ratio (DOR) [16] and diagnostic power (DP) [17] (DP=(3/π)log(DOR);(iii)
the Matthews correlation coefficient (MCC) [18];(iv)
the receiver (or relative) operating characteristic6 area under the curve (AUC) [19, 20];(v)
information theoretic measures such as the information coefficient (IC) [21, 22].Table 2
This table shows the totalF+ and F- for two tests with the same CPD on equally sized data sets (2250 observations), drawn from the same populations. The only difference is the samples have different rCSs. Because the tests were run on data sets with different sized classes, the equivalence of the CPD’s effectiveness is not obvious. It would be easy for an observer to erroneously conclude the CPDs were significantly different.
Relative class size
F
+
F
-
1 : 1
125
250
1 : 9
25
450Youden was addressing the rCS’s biasing effect in 1950; thus, the problem has been known for well over half a century, yet reports regarding mitigation are still entering the literature [9, 23]. In the works reviewed, consideration of end user problem environment was tightly constrained and the view of the data virtually unrestricted. We invert these criteria; first identifying problem interactions with rCS (a broad view of end user needs), then viewing the data such that it addresses the question posed by Jamain and Hand (a constrained view of the data).rCS is generally confounding in the research environment; this is presumably also true for some end users. However, for other end users, rCS may be important for their problem. For these end users, basing decisions on rCS invariant measures may be misleading. Hence, we start by asking two questions:(i)
“Is rCS important for all end user CPD problem domains?”(ii)
“If not, what characteristics define when to incorporate rCS?” Consider relating these two questions to a pair of real-world problems. A less effective CPD used with a rheumatoid arthritis test could lead to either more people than necessary being treated, or fewer. Likewise, a poorly selected intrusion detection boundary could cause an IT system to have excessive errors (false alarms or missed attacks).In order to consider the two questions posed above, we will use a statistical nomenclature to describe a supervised CPD test bed. Viewed from a statistical perspective, observations on the dataset processed provide estimates of the underlying (class) population probabilities (rCS is the odds expression of that probability). Observations in the test system input are drawn from the specific populations (because the source class populations are known for each observation, a “ground truth” exists). The source population relative class sizes can be represented as a probability, for example, the probability that a randomly selected input will be a member of classA. This is the leading probability (Pleading), the probability before the inputs interact with the defined process.7 In the examples stated, class A members would consist of RA-positive individuals and malicious information system activity. In the field, “ground truth” for any particular individual cannot be known prior to being processed (otherwise there would be no need for evaluation). However, in the test scenario being described, ground truth is known for each test set member. Since the source class is known for each CPD input element, input uncertainty does not affect any individual CPD output.The balance of this paper is organized as follows. Section2 considers relative class size. Section 3 describes the research protocol used in this study. Section 4 discusses efficacious measures for end users and considers existing summary statistics. Section 5 considers a cause of rCS invariance in measures and implications thereof. This is followed in Section 6 where we present two examples using the proposed format and tool. The main body of this paper closes with a summary of our findings and presents future work in Section 7. Four appendices with equation derivations and additional JPT normalization details wrap up the paper.


## 2. Relative Class Size

Abstractly described, test set elements interact with the defined process. This interaction “modifies” the elements (perhaps only by adding a tag indicating strength of the match with a model), leading to a test for classA¯ membership. The probability that a randomly selected output will be detected as a member of class A¯ is the subsequent probability (Psubsequent). Psubsequent describes the state of the data stream after interacting with the defined process and is the combined result of the input mix (quantified as a probability (Pleading=Y/N) or an odds ratio (rCS=Y¯/Y)) and the defined process. The defined process contributes its own uncertainty (Pevent) to the observed output. Thus, the test system can be described by the equation Psubsequent=f(Pleading,Pevent). The CDP test set model is illustrated in Figure 1.Figure 1
The test system “ground truth” inputs have a specific mix, representing the underlying probability for the system (Pleading). The test system outputs have a specific mix (Psubsequent), representing the interaction of the defined process and the inputs. The defined process contribution to the uncertainty observed in the output is represented by Pevent. Often, the results are presented in JPTs.Tying the test set model to the examples,Psubsequent consists of the patient’s RA diagnosis and the stream of intrusion detector classifications. Pevent for the RA diagnosis consists of the strength of the match between the compound assayed and RA, test quality and the boundary used to determine class membership (diseased, not diseased). Similarly, Pevent for the intrusion detection example consists of the appropriateness of the model that represents the malicious activity, the reliability of tags defining the activity, and the algorithm (or perhaps rule set) used to make malicious/non malicious determination.In the CPD test system described,P^leading is a characteristic of the input test dataset; hence, it is always fixed.8,9 It is, in fact, related to rCS:
(1)P^leading=1-rCS1+rCS.
We can now restate the original question within our framework: “are there problem domains where Pleading is important rather than confounding?”One such situation could arise where individual results are significant only to the extent to which they contribute to a cumulative result. Consider setting intrusion detection boundaries. The end user is interested in limiting the impact of intrusions and intrusion prevention. The impact is cumulative, with each evaluation activity contributing. In this case, relative class size (expressed asPleading or rCS) is important. If the end user were to base its boundary on Pevent by using a rCS invariant measure (a measure that could not reflect the end user’s estimate of their attack rate), there would likely be either excessive false alarm processing costs or excessive expenses due to missed attacks. Cases of this type, where each individual outcome contributes to a cumulative result, require knowledge of both Pleading and Pevent.Are there conditions in whichPleading, rather than being essential, might instead cause errors? We suggest that one such situation is when individual results are important and cumulative results are not. Consider a person tested for rheumatoid arthritis (RA). Depending upon the physician’s office ordering the test, the frequency of RA+s tested could vary considerably.8 If each office set test boundaries to minimize their respective error rates, there would be a range of test scores that would be classified differently by different offices. Clearly, both diagnoses cannot be correct; a person cannot be simultaneously RA+ and RA-. In this case, considering the physician’s rCS-based Pleading does not minimize the error for the patient.With regard to rCS, we see that while basic research benefits from rCS invariant measures, these measures are not suitable for all end users; rCS invariance will be confounding for some end users. For these end users, any specific environment can have any of (literally) an infinite number of rCS values. Indeed, an end user’s expected rCS can vary over time. Thus, a “one size fits all” solution will not be particularly efficacious. Our goal to provide for end user rCS needs, thus, resolves into two tasks:(i)
identify both rCS-sensitive and rCS-invariant measures that are efficacious for end users;(ii)
identify a means by which end users can tailor reported CPD results to reflect performance for their expected rCS.


## 3. Research Protocol

Although in many problem domains, populations tend to be normally distributed, this is not universal. In order to avoid limiting the applicability of our results, we use analytic procedures that are insensitive to distribution. To preserve generality, our analysis is strictly nonparametric; medians are used instead of means and quantiles are used instead of standard deviations. We also execute our tests with the Monte Carlo method, a nonparametric analytical tool often used when problem complexity (in our case, potential end user problem complexity) is not amenable to mathematical analysis.CPD evaluation studies can be partitioned into two groups: those that use “real-world” data and those that use simulated data. Characterizing CPD evaluation measures requires observing how the measures respond as CPD output varies. Real-world data, such as those available in repositories, for example, the UCI Machine Learning Repository, provide the opportunity to test against a wide variety of complex data types [24]. However, observing the effect of incremental changes on real-world data is difficult at best. For our purpose, we use simulated CPD output. Although any distribution could be used, we assert normality when generating datasets. All data sets used in this study were generated such that the classes were normally distributed (N(m¯,σ2); m¯ is the distribution mean and σ is the standard deviation). The figures displayed were based on four hundred datasets consisting of two hundred thousand randomly drawn observations from two source populations:  positive = N(1.0,0.0225) and negative = N(2.0,0.0625). Separate tests were run with datasets having rCSs of
(2)20:1,21:1,22:1,…,213:1.
A total of 5,600 independent data sets were used in this study.For each summary statistic evaluated, we observed how the reported metric was affected by rCS versus boundary versus metric output. The 3D results are presented as contour plots. Because the measure values are asymptotic to one (thus nonlinear), we use the median of the four hundred runs for each test case; means are not valid for non-linear scales. It is impractical to present confidence intervals on 3D data, but on the 2D graphs in AppendixD, the ninety percent confidence interval (90% CI) is displayed for select test series. To illustrate, the 90% CI is indicated by the vertical lines at each rCS tested in Figure 2; the horizontal line indicates the median.Figure 2
The vertical bars in this graph indicate the 90% confidence interval for measurements at each point observed. The horizontal line is the median.This protocol provides the flexibility and repeatability necessary for analysis, yet abides by the constraints necessary for analysis of less tractable problem domains with difficult problem environments (e.g., complex CPD input and output distributions).


## 4. End User Efficacy

End users have two activities: CPD selection and CPD application. Regardless of any end user problem distinctions, these two activities address common interests:(i)
process application is concerned with the accuracy of the CPD for both possible outcomes:
(1)
“given that the test is positive, to what extent can the result be relied upon?”(2)
“given that the test is negative, to what extent can the result be relied upon?”Mathematically, this can be expressed as a conditional probability, or a conditional odds. These values are monotonic, so difficult to use for optimum boundary identification;(ii)
process selection needs to choose the CPD with the best expected accuracy (“given the set of choices for CPD, which CPD will provide the best results and to what extent can its results be relied upon?”). Summary statistic output (based on the two monotonic measures) can inform end users for process selection.The end user efficacious measures differ for the two rCS problem types; so they are further discussed in the following sections.


### 4.1. When rCS Is Important

Measure efficacy depends upon whether or not the impact on the end user is cumulative. The two CPD application questions can be expressed mathematically as(i)
“given that the test is positive, to what extent can the result be relied upon?”Ω(T+∣Z)=T+/F+;(ii)
“given that the test is negative, to what extent can the result be relied upon?”Ω(T-∣Z¯)=T-/F-.Proportions, being asymptotic to one, are not ratio measures [25] and, thus, have limited utility. We use odds ratios instead.The effect is cumulative (or additive); so the two conditional expressions, instead of being a measure suite, provide ancillary information. For CPD selection, end users will be interested in the proportion of the input stream that can be expected to be correct. Thus, a measure such as the total accuracy rate (TAR)(3)TAR=T++T-T++T-+F++F-
represents the per element expected accuracy. TAR, being asymptotic to one, is not a ratio measure; therefore, averaging is not a valid operation. The total accuracy odds ratio (TOR) would be a better choice:
(4)TOR=T++T-F++F-.Proportions, such as TAR and odds ratios such as TOR, are alternate expressions of the same CPD output. In fact, odds ratios can be transformed into proportions using (1).Figure3 shows TAR, as boundary and rCS vary. The TAR contour appears to vary little and be relatively constant over a wide boundary range. The optimum boundary (shown on the graph as the black dashed line) intersects the x-axis at around 1.45 and slopes toward 1.0. Additionally, the contour around the optimum boundary flattens as rCS increases. We can also see the optimum boundary and the reported accuracy rate both change as rCS varies. TAR is sensitive to rCS; thus, it is useful for problem domains where cumulative effects are important.Figure 3
Being the sum of the observed correct classifications, TAR is a good measure for evaluating rCS sensitive CPDs. It is significant that the dashed line, indicating the optimum boundary, is not vertical; this shows that TAR is rCS sensitive.There are two other commonly seen rCS-sensitive summary statistics:F-score and Matthews correlation coefficient9 (MCC). Another measure, information coefficient (IC); is becoming more prevalent, so we consider it as well.10


#### 4.1.1.F-Score

F-score is the complement of a summary statistic proposed by van Rijsbergen [26]. The measure suite for F-score is recall and precision; van Rijsbergen’s measure is based on information retrieval performance criteria put forth by Cleverdon [27]. Cleverdon’s criteria address practitioner needs in information retrieval. Recall quantifies a CPD’s completeness (the probability that the desired observations in the database are correctly identified). Precision quantifies the probability that undesired observations are mistakenly labeled as desired. For the information retrieval domain, these data11 seem to be what end users need to know (F-score is now being seen in other problem domains.)Recall and precision correspond to the conditional probabilitiesP(T+∣Y) and P(T+∣Z) (also known as “True positive rate” (TPR) and “positive predictive value” (PPV)). In the problem domain within which they were introduced (information retrieval), these measures quantify how well an CPD relates an object to a concept, such as selecting a document based on keywords. F-score is defined as
(5)Fβ=(1+β2)(precision)(recall)(β2)(precision+recall),
where β is the relative weight of precision and recall:
(6)β=importance  of  precisionimportance  of  recall.
If precision and recall have equal weights, then Fβ=F1, which is the harmonic mean of precision and recall:
(7)F1=2(precision)(recall)precision+recall.
Since precision and recall are conditional probabilities, we can convert the F-score equation into JPT values. After substitution and rearranging terms,
(8)F1=T+T++F+/2+F-/2.
The derivation is provided in Appendix A. Notably, using the harmonic mean results in T+, F+ and F- not being equally weighted in the denominator. While this may be suitable for the information retrieval domain and some others, it is hardly universal.In contrast to TAR,F-score’s rCS sensitivity varies, depending upon the class monitored. The effect can be seen in Figures 4(a) and 4(b). Interestingly, both TAR and F-score are well-accepted measures. This may indicate the existence of another CPD problem structure element. Analysis of this possibility is postponed for later consideration.These graphs show thatF-score, a summary statistic commonly used to compare CPD effectiveness, is rCS-sensitive. This is a desirable characteristic for problem domains such as information retrieval. In addition to rCS sensitivity, F-score is also sensitive to the target class.
(a)
The relatively flat area around the optimum boundary (black dashed line in the graph) with low rCS suggests a low boundary sensitivity. The ridge follows the same optimum boundary as that of the total accuracy rate
(b)
If, instead of selectingT+, we select T-, then F-score looks remarkably similar to TAR. Thus, F-score quantifies the categorical process’s effect on a specific category


#### 4.1.2. Matthews Correlation Coefficient

The Matthews correlation coefficient (MCC) is a more recent measure, introduced by Matthews [18]. MCC is the application of Pearsons correlation coefficient to CPD evaluation. In a subsequent classifier measure survey, Baldi et al. restated the measure in the form commonly seen today [21]:
(9)MCC=(T+*T-)-(F+*F-)Y*Y¯*Z*Z¯.
Although not mentioned explicitly in Matthews’ work, Baldi et al. note that the equation requires normalized distributions:
(10)a-a¯σA,wherea∈{A},A={a1,a2,a3,…,as} is the class of the input dataset and s=|A|. a¯ is the mean of A. A¯ is treated in the same manner. One effect of distribution normalization is class size equalization: rCS=1 (this is discussed in Section 5). The expressions used to generate normalized JPTs are shown in Table 3 the class size equalization is indicated by the ones in the “normalized totals” row. To demonstrate Matthews initial intent to use normalized distributions, we recalculated Matthews et al.’s original results using both actual and normalized JPTs. The values using normalized JPTs matched Matthews results; the values using actual JPT values varied from Matthews reported values by approximately a factor of twenty. Thus, MCC, when introduced, was intended to be calculated on normalized JPTs.Table 3
The expressions in this JPT normalize the category values.
Actual target classification
Y
Y
-
Test result
Positive
T
+
/
Y
F
+
/
Y
-
Negative
F
-
/
Y
T
-
/
Y
-
Normalized totals
1
1
2The JPT normalization prerequisite applied by Matthews et al. and noted by Baldi et al. seems to have been lost, although the belief that MCC is rCS-invariant persists [28–34].12 As a consequence, Baldi et al.’s equation is sometimes applied without first normalizing the JPTs. In two of these reports, Cannon et al. [28] and Mirceva et al. [34] include JPTs. Upon recalculating their results, we determined that the values presented were based on nonnormalized JPTs. In both cases, there were substantial differences between the results on normalized and nonnormalized JPTs. In the Cannon et al. results, the difference affected not only the values, but also the process rankings. Using normalized JPTs, MOLPRINT, the process ranked last, moved into the upper fifty percent of processes tested. Having rankings of processes substantially change due to such changes could result in selection of a suboptimal process for use in real-world settings.Figure5 shows how normalization impacts the rCS sensitivity for MCC. Figure 5(a) shows MCC’s response when the raw JPT values are used. The peak boundary (indicated on the graph by the dashed line) shifts as rCS varies and the value decreases as rCS increases (indicated by the sloping dashed line that intersects the contours). In contrast, Figure 5(b) shows that with JPT normalization, the peak boundary and calculated optimum MCC value are fixed (indicated by a vertical dashed line and contours that do not intersect the peak boundary). Exactly the same data sets were used for both graphs; the only difference is the presence or absence of JPT normalization.If MCC inputs are not normalized, it is rCS-sensitive.
(a)
MCC exhibits rCS sensitivity
(b)
On normalized JPTs, MCC exhibits rCS invariance. The increasing boundary curvature whenrCS>26 is a JPT normalization artifact explained in Appendix DComparing MCC results is complicated by the fact that the published reports we surveyed did not identify whether or not the MCC values reported were on normalized JPTs or not. As seen in Figures5(a) and 5(b), comparing results across tests where rCS is not normalized could lead to errors. Since rCS affects the optimum boundary when raw JPTs are used, a simple correction of reported values by recalculating MCC on normalized JPTs will most probably be for a suboptimal boundary; thus, the corrected MCC value will also be suboptimal.13  In this section, we characterized MCC using nonnormalized data. Appendix C derives an rCS-invariant MCC expression.Relative to end user interests, it is unclear what MCC quantifies and under what context the value will be relevant; MCC’s end user efficacy is limited to optimum boundary identification.MCC’s complexity makes determining an underlying measure suite difficult. This detail will be addressed in the future.


#### 4.1.3. Mutual Information Coefficient

Rost and Sander introduced an information-theory-based measure into the literature in 1993 [22]. It was subsequently included in a measure comparison by Baldi et al. [21]. Since then, it has gained some traction in biological literature [35–43] and has been seen in network management literature [44]. The measure is sometimes called the information coefficient or mutual information coefficient; we use the acronym IC.As explained by Baldi et al., IC is the mutual information (I) normalized by the entropy in ground truth (H); I is the mutual information contained in ground truth regarding the test set S(Y∪Y¯) and the CPD prediction of ground truth, as contained in Z∪Z¯:
(11)IC=I(Y∪Y¯,Z∪Z¯)H(Y∪Y¯).
Expressing I and H in terms of JPT categories,
(12)I(Y∪Y¯,Z∪Z¯)=-H(T+N,F+N,F-N,T-N)-T+Nlog(|Y|*|Z|)-F+Nlog(|Y¯|*|Z|)-F-Nlog(|Y|*|Z¯|)-T-Nlog(|Y¯|*|Z¯|),
where
(13)H(T+N,F+N,F-N,T-N)=-T+NlogT+N-F+NlogF+N-F-NlogF-N-T-NlogT-N.Information-theory-based measures are gaining traction in the literature [35–43]. Some of these reports indicate the belief that the measures are rCS-invariant [38, 40, 43]. Solis and Rackovsky [41] note that their particular information theoretic measure may not be rCS-invariant. The belief that information theoretic measures are rCS-invariant comes from the fact that information theory applies to probability density functions, which are always normalized (rCS=1) [45, 46]. Unless JPTs are normalized prior to use, IC and related measures cannot be guaranteed to be rCS-invariant.Like other measures, IC compares target CPD output to an CPD using random classification. However, it differs in that IC is based on the entropy existing in the test set and CDP output. If the input and output are the same, thenIC=1; if the output of the process is equivalent to that of random selection, then IC=0. A side effect of IC’s use of logs is increased computational complexity. All of the other measures evaluated have a complexity of O(N), IC is O(N2). This may limit IC’s utility for large data sets. IC’s computational complexity did affect our analysis. Had we calculated IC on the two hundred thousand element test sets used for the other measures, it would have taken approximately six months. Consequently, we tested IC on twenty thousand element test sets. In Figure 6, we can see that the peak boundary shifts as rCS increases; thus, IC is not rCS-invariant. As with the other rCS-sensitive measures, JPT normalization can confer rCS invariance.Figure 6
The sloped dotted line on the contour graph shows that IC is notrCS-invariant.


### 4.2. Findings regarding rCS Sensitive Problems

(i)
Of the four rCS-sensitive summary statistics reviewed, TAR andF-score appear to be efficacious.
(ii)
Because the reaction to rCS of TAR andF-score are opposite, this may indicate the existence of other elements in the CPD problem structure (we will address that in future work.)
(iii)
MCC and IC, regardless of their apparent utility for researchers, do not seem to quantify information directly usable by end users. This section has not covered how end users can take a single reported value and convert it into one applicable to their specific rCS environment. This will be discussed in Section 5.


### 4.3. When rCS Is Confounding

When rCS is confounding, in addition to quantifying end user issues, efficacious measures must be rCS-invariant. The following discussion will apply normalized JPT input when necessary.The CPD application expressions for this problem type are normalized versions of those for rCS-sensitive problems.“Given that the test is positive, to what extent can the result be relied upon?” Mathematically, this can be expressed as a conditional probability, or a conditional odds on normalized JPTs. For the reason mentioned in Section4, we use the odds, normalized Ω(T+∣Z)=T+Y-/F+Y.“Given that the test is negative, to what extent can the result be relied upon?” The odds expression for this is normalizedΩ(T-∣Z-)=T-Y/F_Y-.Test selection needs to choose the CPD with the best expected accuracy (“given that a result will be rendered, to what extent can the result be relied upon?”). The two CPD application questions provide operational information, but are also the basis for this noncumulative CPD problem selection decision. As such, they can be considered the measurement suite for the CPD selection decision. The CPD selection decision requires an end user efficacious summary statistic. The expected prediction accuracy (EPA) is the average of the two odds ratios identified in the previous paragraph. Each CPD event is independent and the conditional values are normalized. The special conditions that dictate applying either the geometrical mean (a compounding effect) or the harmonic mean (unequal set sizes) do not exist; so the arithmetic mean of the conditional odds on normalized JPTs is appropriate:(14)EPA=(T+Y¯)/(F+Y)+(T-Y)/(F-Y¯)2.
To our knowledge, this end user summary statistic is not found in the literature. We apply this summary statistic in the meta-analysis in Section 6.As noted in the introduction, rCS-invariant summary statistics are already in use. We review three commonly seen rCS-invariant summary statistics:(i)
the Youden index [15];(ii)
two related measures: diagnostic odds ratio (DOR) [16] and diagnostic power (DP) [17] (DP=(3/π)log(DOR));(iii)
the receiver (or relative) operating characteristic area under the curve (AUC) [19, 20]. Two other summary statistics, the Matthews correlation coefficient (MCC) [18] and mutual information coefficient (IC) [21] are commonly held to be rCS-invariant, but in fact are not. They were discussed in Section 4.1.


#### 4.3.1. Youden Index

The Youden index (traditionally represented byJ) was proposed in 1950 and is seen in medical diagnostic studies [15]. There are a number of expressions of J. The original is
(15)J=12[T+-F+T++F++T--F-T-+F-].
Perhaps a more common representation is
(16)J=sensitivity+specificity-1,
where
(17)sensitivity=T+Y,specificity=T-Y¯.
Further, sensitivity is also known as the true positive rate (TPR) and specificity is the complement of the false positive rate; specificity = 1-FPR=1-(F+/Y-). Hence, an even simpler (thus better, according to the minimum description length principal) definition would be
(18)J=TPR-FPR.
In this form, the Youden index can be taken to be a summary statistic of the measure suite {TPR,FPR}.J is special in that J=0 indicates an CPD with an output equal to that of tossing a fair coin. J=1 with a perfect CPD and J=-1 for an CPD that misclassifies everything. As noted in their respective literature bases, J shares a characteristic with AUC, in that it is insensitive to rCS. On our source populations, the optimum boundary is approximately 1.4. This can be seen in Figure 7. There is an issue with end user efficacy, however. J quantifies the spread between the TPR and FPR. This information has little bearing on the “pretest” question posed at the beginning of this section.Figure 7
The Youden index has a very uniform shape and the optimum boundary lies along the peak of the Youden index ridge. This exhibits the expected rCS invariance.


#### 4.3.2. Diagnostic Odds Ratio (DOR) and Discriminant Power (DP)

Two related measures are the diagnostic odds ratio (DOR) [16] and discriminant power (DP) [17]. DOR is defined as
(19)DOR=T+/F-F+/T-,
where T+/F-is true positive odds (TPO) and F+/T- is false positive odds (FPO). After simplification,
(20)DOR=T+T-F+F-.
Discriminant power is defined as
(21)DP=3π(logX+logW),
where
(22)X=sensitivity1-sensitivity,Y=specificity1-specificity.
Recasting the equation, we get
(23)DP=3πlog(T+T-F+F-).
The derivation is provided in Appendix B. Comparing the two measures, we see that
(24)DP=3πlog(DOR).DOR and DP are found in medical research. Interestingly, DP=-∞ and DOR=0 when either T+=0 or T-=0, both need not equal zero. Similarly, DP=∞ and DOR=∞ when either F+=0 or F-=0, both need not equal zero. Hence, an CPD can classify some observations correctly (total Accuracy > 0), yet have DP=-∞ and DOR=0. This is counterintuitive, since one would expect DP=-∞ and DOR=0 to indicate a totally ineffective CPD and DP=∞ and DOR=∞ to indicate a perfect CPD, rather than something in between. Since T+ and T- are (statistically) independent,14 (as are F+ and F-), the DP and DOR could, in a probabilistic sense, be interpreted as the odds that, given two random observations, one will be classified T+ and the other T- (one will be classified F+ and the other F-). While the question seems similar, the fact that the DOR and DP optimum boundaries are different from the other inherently rCS invariant measures tested suggests that the two questions are significantly different. Perhaps this is because the DP and DOR treat the problem as a multiplicative function; we identify the problem as an additive function. This value would seem to be directly relevant in niche CPD scenarios, but not to general CPD problem types.In medical studies, when the event tested for (T+) has a low probability, DOR approximates relative risk: the rate at which the event was observed in group A versus the rate observed in group B. This is valuable information. However, when applied in the more general CPD domain, there is a problem. In any specific CPD task, the category of interest may not have a sufficiently low probability T+; thus, the approximation may not always be acceptably close.Unfortunately, DOR and DP have a challenging sensitivity to boundary; the optimum boundary is indicated bymin(DOR) (or min(DP)). Thus, for any test run, the boundary with the smallest T+T- relative to F+F- gives the best accuracy. Not only is this counterintuitive, but also a potential error source. The problem originates from the fact that the greater the min(DOR) (or min(DP)), the better the results. Thus, if the boundary used to partition the test output is not at min(DOR) (or min(DP)), the results may appear better than they really are.15 Most observations regarding DOR apply to DP as well. For example, DP=0 when T+T-=F+F-.One important characteristic of DOR and DP is that they are rCS invariant. An important difference between DOR/DP and the other rCS-invariant measures is that their optimum boundaries, although constant in our tests, are offset from the “minimum error boundary.” These effects can be seen in Figure8.16 Since DOR and DP are minima, they follow a valley in the contour graph, instead of a ridge, Also contrary to the other measures, DOR decreases when the absolute class size effect becomes noticeable. This means that the contours are closed, instead of open as seen for the other measures. DOR’s vertical optimum boundary line and constant value (seen in  Figure 8(b)) indicate that DOR (and hence, DP) is rCS-invariant. Because of this boundary bias, they may not be useful for selecting boundaries. For example, in our test environment, TAR at the common optimum boundary is 0.994, TAR at DOR optimum boundary is 0.958; the difference is significant at the 95% confidence level.Instead of the optimum value being maxima, like the other measures evaluated, the optimum DOR value is a minimum. Hence, the contours show a valley instead of a ridge. Also contrary to the other measures, DOR decreases when the absolute class size effect becomes noticeable. This means that the contours are closed, instead of open like the others. DOR’s vertical optimum boundary line and constant value (seen in Figure8(b)) indicate that DOR (and hence, DP) is rCS-invariant. DOR/DP optimum boundaries (approx. 1.2) are offset from the optimum boundaries seen in the other rCS-invariant measures (approx. 1.4). DP, the log form of DOR, has the same characteristics as DOR.
(a)
Contour graph of rCS versus boundary versus DOR value. Scaling makes the measure seem somewhat rCS sensitive. However, Figure8(b) shows DOR is actually rCS invariant
(b)
Graph of rCS versus DOR, with error bars


#### 4.3.3. Receiver Operating Characteristic Area under the Curve (AUC)

ROC has a solid history. Swets campaigned diligently to establish it as the evaluation criterion of choice [20, 47, 48]. The {TPR,FPR} measurement suite is the basis for the AUC summary statistic. The title originates from the fact that it is the area under a “ROC curve,” a curve defined by false positive (FPR=F+/Y-) and true positive (TPR=T+/Y) rates. These values are calculated from JPTs of CPD output for a number of boundaries across the observed range, then graphed as the ROC curve [19, 49].17 The literature describes the ROC curve (AUC) as being rCS-invariant as well as boundary-invariant. Because it is boundary-invariant, AUC is a popular tool in our present research environment. However, AUC has been criticized on theoretical terms recently [50, 51].In contrast to the other summary statistics reviewed, AUC is generally accompanied by the ROC curve (indeed, the ROC curve may be presented without providing AUC). To a person skilled in the art, the ROC curve provides a great deal more information regarding CPD performance than does the single value AUC summary statistic18 (this is, of course, true for any measure suite, since consolidation of multiple values into a single summary statistic value means that information is lost).Compared to our end user focused criteria, ROC-AUC, being boundary-invariant, is not useful for boundary identification. Nor is it efficacious for end users.There are numerous ROC-AUC variants [52, 53]. Vanderlooy and Hüllermeier determined in their comparison, that despite intuitive appeal, none of the variants confer any CPD selection improvement. From the end user perspective, since the underlying measure units remain the same, they all have the same limited efficacy.Figure9 shows the optimum boundary versus rCS for the proposed normalized PPV, NPV average and existing summary statistics, normalized MCC, Youden index, and DOR. As can be seen in the figure, DOR peaks at a different boundary than the other rCS-invariant measures tested, and (excepting DOR) the optimum boundary is relatively stable until rCS>26, after which the detected optimum boundary starts dropping rapidly.19 For our context, a key finding from Figure 9 is that not only does DOR have weak end user efficacy, it also should not be used to identify the optimum boundary.Figure 9
Other than DOR/DP, all of the rCS-invariant measures tested identified the same optimum boundary.


### 4.4. Findings regarding rCS-Invariant End User Problems

For problems requiring rCS invariance, we find that(i)
end users need three values. For CPD selection, the expected total predictive accuracy (a summary statistic)EPA=((T+Y-)/(F+Y)+(T-Y)/(F-Y-))/2 is important. When the CPD is used in the field, the summary statistic value has no meaning. Instead, end users need the information provided by the two measure suite elements; the positive predictive value odds ratio PPV=(T+Y-)/(F+Y) and the negative predictive value odds ratio NPV=(T-Y)/(F-Y-).(ii)
Although many commonly seen summary statistics can be used to identify the optimum boundary, as seen in this study, not all can (e.g., ROC-AUC and DOR). Figure9 shows how the optimum boundary identified by DOR differs from that identified by the other summary statistics tested.(iii)
Of the rCS sensitive summary statistics evaluated, only the EPA output answers the end user’s CPD selection question. The others may be useful for niche problems, but provide little useful information for the “common” end user.


## 4.1. When rCS Is Important

Measure efficacy depends upon whether or not the impact on the end user is cumulative. The two CPD application questions can be expressed mathematically as(i)
“given that the test is positive, to what extent can the result be relied upon?”Ω(T+∣Z)=T+/F+;(ii)
“given that the test is negative, to what extent can the result be relied upon?”Ω(T-∣Z¯)=T-/F-.Proportions, being asymptotic to one, are not ratio measures [25] and, thus, have limited utility. We use odds ratios instead.The effect is cumulative (or additive); so the two conditional expressions, instead of being a measure suite, provide ancillary information. For CPD selection, end users will be interested in the proportion of the input stream that can be expected to be correct. Thus, a measure such as the total accuracy rate (TAR)(3)TAR=T++T-T++T-+F++F-
represents the per element expected accuracy. TAR, being asymptotic to one, is not a ratio measure; therefore, averaging is not a valid operation. The total accuracy odds ratio (TOR) would be a better choice:
(4)TOR=T++T-F++F-.Proportions, such as TAR and odds ratios such as TOR, are alternate expressions of the same CPD output. In fact, odds ratios can be transformed into proportions using (1).Figure3 shows TAR, as boundary and rCS vary. The TAR contour appears to vary little and be relatively constant over a wide boundary range. The optimum boundary (shown on the graph as the black dashed line) intersects the x-axis at around 1.45 and slopes toward 1.0. Additionally, the contour around the optimum boundary flattens as rCS increases. We can also see the optimum boundary and the reported accuracy rate both change as rCS varies. TAR is sensitive to rCS; thus, it is useful for problem domains where cumulative effects are important.Figure 3
Being the sum of the observed correct classifications, TAR is a good measure for evaluating rCS sensitive CPDs. It is significant that the dashed line, indicating the optimum boundary, is not vertical; this shows that TAR is rCS sensitive.There are two other commonly seen rCS-sensitive summary statistics:F-score and Matthews correlation coefficient9 (MCC). Another measure, information coefficient (IC); is becoming more prevalent, so we consider it as well.10


### 4.1.1.F-Score

F-score is the complement of a summary statistic proposed by van Rijsbergen [26]. The measure suite for F-score is recall and precision; van Rijsbergen’s measure is based on information retrieval performance criteria put forth by Cleverdon [27]. Cleverdon’s criteria address practitioner needs in information retrieval. Recall quantifies a CPD’s completeness (the probability that the desired observations in the database are correctly identified). Precision quantifies the probability that undesired observations are mistakenly labeled as desired. For the information retrieval domain, these data11 seem to be what end users need to know (F-score is now being seen in other problem domains.)Recall and precision correspond to the conditional probabilitiesP(T+∣Y) and P(T+∣Z) (also known as “True positive rate” (TPR) and “positive predictive value” (PPV)). In the problem domain within which they were introduced (information retrieval), these measures quantify how well an CPD relates an object to a concept, such as selecting a document based on keywords. F-score is defined as
(5)Fβ=(1+β2)(precision)(recall)(β2)(precision+recall),
where β is the relative weight of precision and recall:
(6)β=importance  of  precisionimportance  of  recall.
If precision and recall have equal weights, then Fβ=F1, which is the harmonic mean of precision and recall:
(7)F1=2(precision)(recall)precision+recall.
Since precision and recall are conditional probabilities, we can convert the F-score equation into JPT values. After substitution and rearranging terms,
(8)F1=T+T++F+/2+F-/2.
The derivation is provided in Appendix A. Notably, using the harmonic mean results in T+, F+ and F- not being equally weighted in the denominator. While this may be suitable for the information retrieval domain and some others, it is hardly universal.In contrast to TAR,F-score’s rCS sensitivity varies, depending upon the class monitored. The effect can be seen in Figures 4(a) and 4(b). Interestingly, both TAR and F-score are well-accepted measures. This may indicate the existence of another CPD problem structure element. Analysis of this possibility is postponed for later consideration.These graphs show thatF-score, a summary statistic commonly used to compare CPD effectiveness, is rCS-sensitive. This is a desirable characteristic for problem domains such as information retrieval. In addition to rCS sensitivity, F-score is also sensitive to the target class.
(a)
The relatively flat area around the optimum boundary (black dashed line in the graph) with low rCS suggests a low boundary sensitivity. The ridge follows the same optimum boundary as that of the total accuracy rate
(b)
If, instead of selectingT+, we select T-, then F-score looks remarkably similar to TAR. Thus, F-score quantifies the categorical process’s effect on a specific category


### 4.1.2. Matthews Correlation Coefficient

The Matthews correlation coefficient (MCC) is a more recent measure, introduced by Matthews [18]. MCC is the application of Pearsons correlation coefficient to CPD evaluation. In a subsequent classifier measure survey, Baldi et al. restated the measure in the form commonly seen today [21]:
(9)MCC=(T+*T-)-(F+*F-)Y*Y¯*Z*Z¯.
Although not mentioned explicitly in Matthews’ work, Baldi et al. note that the equation requires normalized distributions:
(10)a-a¯σA,wherea∈{A},A={a1,a2,a3,…,as} is the class of the input dataset and s=|A|. a¯ is the mean of A. A¯ is treated in the same manner. One effect of distribution normalization is class size equalization: rCS=1 (this is discussed in Section 5). The expressions used to generate normalized JPTs are shown in Table 3 the class size equalization is indicated by the ones in the “normalized totals” row. To demonstrate Matthews initial intent to use normalized distributions, we recalculated Matthews et al.’s original results using both actual and normalized JPTs. The values using normalized JPTs matched Matthews results; the values using actual JPT values varied from Matthews reported values by approximately a factor of twenty. Thus, MCC, when introduced, was intended to be calculated on normalized JPTs.Table 3
The expressions in this JPT normalize the category values.
Actual target classification
Y
Y
-
Test result
Positive
T
+
/
Y
F
+
/
Y
-
Negative
F
-
/
Y
T
-
/
Y
-
Normalized totals
1
1
2The JPT normalization prerequisite applied by Matthews et al. and noted by Baldi et al. seems to have been lost, although the belief that MCC is rCS-invariant persists [28–34].12 As a consequence, Baldi et al.’s equation is sometimes applied without first normalizing the JPTs. In two of these reports, Cannon et al. [28] and Mirceva et al. [34] include JPTs. Upon recalculating their results, we determined that the values presented were based on nonnormalized JPTs. In both cases, there were substantial differences between the results on normalized and nonnormalized JPTs. In the Cannon et al. results, the difference affected not only the values, but also the process rankings. Using normalized JPTs, MOLPRINT, the process ranked last, moved into the upper fifty percent of processes tested. Having rankings of processes substantially change due to such changes could result in selection of a suboptimal process for use in real-world settings.Figure5 shows how normalization impacts the rCS sensitivity for MCC. Figure 5(a) shows MCC’s response when the raw JPT values are used. The peak boundary (indicated on the graph by the dashed line) shifts as rCS varies and the value decreases as rCS increases (indicated by the sloping dashed line that intersects the contours). In contrast, Figure 5(b) shows that with JPT normalization, the peak boundary and calculated optimum MCC value are fixed (indicated by a vertical dashed line and contours that do not intersect the peak boundary). Exactly the same data sets were used for both graphs; the only difference is the presence or absence of JPT normalization.If MCC inputs are not normalized, it is rCS-sensitive.
(a)
MCC exhibits rCS sensitivity
(b)
On normalized JPTs, MCC exhibits rCS invariance. The increasing boundary curvature whenrCS>26 is a JPT normalization artifact explained in Appendix DComparing MCC results is complicated by the fact that the published reports we surveyed did not identify whether or not the MCC values reported were on normalized JPTs or not. As seen in Figures5(a) and 5(b), comparing results across tests where rCS is not normalized could lead to errors. Since rCS affects the optimum boundary when raw JPTs are used, a simple correction of reported values by recalculating MCC on normalized JPTs will most probably be for a suboptimal boundary; thus, the corrected MCC value will also be suboptimal.13  In this section, we characterized MCC using nonnormalized data. Appendix C derives an rCS-invariant MCC expression.Relative to end user interests, it is unclear what MCC quantifies and under what context the value will be relevant; MCC’s end user efficacy is limited to optimum boundary identification.MCC’s complexity makes determining an underlying measure suite difficult. This detail will be addressed in the future.


### 4.1.3. Mutual Information Coefficient

Rost and Sander introduced an information-theory-based measure into the literature in 1993 [22]. It was subsequently included in a measure comparison by Baldi et al. [21]. Since then, it has gained some traction in biological literature [35–43] and has been seen in network management literature [44]. The measure is sometimes called the information coefficient or mutual information coefficient; we use the acronym IC.As explained by Baldi et al., IC is the mutual information (I) normalized by the entropy in ground truth (H); I is the mutual information contained in ground truth regarding the test set S(Y∪Y¯) and the CPD prediction of ground truth, as contained in Z∪Z¯:
(11)IC=I(Y∪Y¯,Z∪Z¯)H(Y∪Y¯).
Expressing I and H in terms of JPT categories,
(12)I(Y∪Y¯,Z∪Z¯)=-H(T+N,F+N,F-N,T-N)-T+Nlog(|Y|*|Z|)-F+Nlog(|Y¯|*|Z|)-F-Nlog(|Y|*|Z¯|)-T-Nlog(|Y¯|*|Z¯|),
where
(13)H(T+N,F+N,F-N,T-N)=-T+NlogT+N-F+NlogF+N-F-NlogF-N-T-NlogT-N.Information-theory-based measures are gaining traction in the literature [35–43]. Some of these reports indicate the belief that the measures are rCS-invariant [38, 40, 43]. Solis and Rackovsky [41] note that their particular information theoretic measure may not be rCS-invariant. The belief that information theoretic measures are rCS-invariant comes from the fact that information theory applies to probability density functions, which are always normalized (rCS=1) [45, 46]. Unless JPTs are normalized prior to use, IC and related measures cannot be guaranteed to be rCS-invariant.Like other measures, IC compares target CPD output to an CPD using random classification. However, it differs in that IC is based on the entropy existing in the test set and CDP output. If the input and output are the same, thenIC=1; if the output of the process is equivalent to that of random selection, then IC=0. A side effect of IC’s use of logs is increased computational complexity. All of the other measures evaluated have a complexity of O(N), IC is O(N2). This may limit IC’s utility for large data sets. IC’s computational complexity did affect our analysis. Had we calculated IC on the two hundred thousand element test sets used for the other measures, it would have taken approximately six months. Consequently, we tested IC on twenty thousand element test sets. In Figure 6, we can see that the peak boundary shifts as rCS increases; thus, IC is not rCS-invariant. As with the other rCS-sensitive measures, JPT normalization can confer rCS invariance.Figure 6
The sloped dotted line on the contour graph shows that IC is notrCS-invariant.


## 4.1.1.F-Score

F-score is the complement of a summary statistic proposed by van Rijsbergen [26]. The measure suite for F-score is recall and precision; van Rijsbergen’s measure is based on information retrieval performance criteria put forth by Cleverdon [27]. Cleverdon’s criteria address practitioner needs in information retrieval. Recall quantifies a CPD’s completeness (the probability that the desired observations in the database are correctly identified). Precision quantifies the probability that undesired observations are mistakenly labeled as desired. For the information retrieval domain, these data11 seem to be what end users need to know (F-score is now being seen in other problem domains.)Recall and precision correspond to the conditional probabilitiesP(T+∣Y) and P(T+∣Z) (also known as “True positive rate” (TPR) and “positive predictive value” (PPV)). In the problem domain within which they were introduced (information retrieval), these measures quantify how well an CPD relates an object to a concept, such as selecting a document based on keywords. F-score is defined as
(5)Fβ=(1+β2)(precision)(recall)(β2)(precision+recall),
where β is the relative weight of precision and recall:
(6)β=importance  of  precisionimportance  of  recall.
If precision and recall have equal weights, then Fβ=F1, which is the harmonic mean of precision and recall:
(7)F1=2(precision)(recall)precision+recall.
Since precision and recall are conditional probabilities, we can convert the F-score equation into JPT values. After substitution and rearranging terms,
(8)F1=T+T++F+/2+F-/2.
The derivation is provided in Appendix A. Notably, using the harmonic mean results in T+, F+ and F- not being equally weighted in the denominator. While this may be suitable for the information retrieval domain and some others, it is hardly universal.In contrast to TAR,F-score’s rCS sensitivity varies, depending upon the class monitored. The effect can be seen in Figures 4(a) and 4(b). Interestingly, both TAR and F-score are well-accepted measures. This may indicate the existence of another CPD problem structure element. Analysis of this possibility is postponed for later consideration.These graphs show thatF-score, a summary statistic commonly used to compare CPD effectiveness, is rCS-sensitive. This is a desirable characteristic for problem domains such as information retrieval. In addition to rCS sensitivity, F-score is also sensitive to the target class.
(a)
The relatively flat area around the optimum boundary (black dashed line in the graph) with low rCS suggests a low boundary sensitivity. The ridge follows the same optimum boundary as that of the total accuracy rate
(b)
If, instead of selectingT+, we select T-, then F-score looks remarkably similar to TAR. Thus, F-score quantifies the categorical process’s effect on a specific category


## 4.1.2. Matthews Correlation Coefficient

The Matthews correlation coefficient (MCC) is a more recent measure, introduced by Matthews [18]. MCC is the application of Pearsons correlation coefficient to CPD evaluation. In a subsequent classifier measure survey, Baldi et al. restated the measure in the form commonly seen today [21]:
(9)MCC=(T+*T-)-(F+*F-)Y*Y¯*Z*Z¯.
Although not mentioned explicitly in Matthews’ work, Baldi et al. note that the equation requires normalized distributions:
(10)a-a¯σA,wherea∈{A},A={a1,a2,a3,…,as} is the class of the input dataset and s=|A|. a¯ is the mean of A. A¯ is treated in the same manner. One effect of distribution normalization is class size equalization: rCS=1 (this is discussed in Section 5). The expressions used to generate normalized JPTs are shown in Table 3 the class size equalization is indicated by the ones in the “normalized totals” row. To demonstrate Matthews initial intent to use normalized distributions, we recalculated Matthews et al.’s original results using both actual and normalized JPTs. The values using normalized JPTs matched Matthews results; the values using actual JPT values varied from Matthews reported values by approximately a factor of twenty. Thus, MCC, when introduced, was intended to be calculated on normalized JPTs.Table 3
The expressions in this JPT normalize the category values.
Actual target classification
Y
Y
-
Test result
Positive
T
+
/
Y
F
+
/
Y
-
Negative
F
-
/
Y
T
-
/
Y
-
Normalized totals
1
1
2The JPT normalization prerequisite applied by Matthews et al. and noted by Baldi et al. seems to have been lost, although the belief that MCC is rCS-invariant persists [28–34].12 As a consequence, Baldi et al.’s equation is sometimes applied without first normalizing the JPTs. In two of these reports, Cannon et al. [28] and Mirceva et al. [34] include JPTs. Upon recalculating their results, we determined that the values presented were based on nonnormalized JPTs. In both cases, there were substantial differences between the results on normalized and nonnormalized JPTs. In the Cannon et al. results, the difference affected not only the values, but also the process rankings. Using normalized JPTs, MOLPRINT, the process ranked last, moved into the upper fifty percent of processes tested. Having rankings of processes substantially change due to such changes could result in selection of a suboptimal process for use in real-world settings.Figure5 shows how normalization impacts the rCS sensitivity for MCC. Figure 5(a) shows MCC’s response when the raw JPT values are used. The peak boundary (indicated on the graph by the dashed line) shifts as rCS varies and the value decreases as rCS increases (indicated by the sloping dashed line that intersects the contours). In contrast, Figure 5(b) shows that with JPT normalization, the peak boundary and calculated optimum MCC value are fixed (indicated by a vertical dashed line and contours that do not intersect the peak boundary). Exactly the same data sets were used for both graphs; the only difference is the presence or absence of JPT normalization.If MCC inputs are not normalized, it is rCS-sensitive.
(a)
MCC exhibits rCS sensitivity
(b)
On normalized JPTs, MCC exhibits rCS invariance. The increasing boundary curvature whenrCS>26 is a JPT normalization artifact explained in Appendix DComparing MCC results is complicated by the fact that the published reports we surveyed did not identify whether or not the MCC values reported were on normalized JPTs or not. As seen in Figures5(a) and 5(b), comparing results across tests where rCS is not normalized could lead to errors. Since rCS affects the optimum boundary when raw JPTs are used, a simple correction of reported values by recalculating MCC on normalized JPTs will most probably be for a suboptimal boundary; thus, the corrected MCC value will also be suboptimal.13  In this section, we characterized MCC using nonnormalized data. Appendix C derives an rCS-invariant MCC expression.Relative to end user interests, it is unclear what MCC quantifies and under what context the value will be relevant; MCC’s end user efficacy is limited to optimum boundary identification.MCC’s complexity makes determining an underlying measure suite difficult. This detail will be addressed in the future.


## 4.1.3. Mutual Information Coefficient

Rost and Sander introduced an information-theory-based measure into the literature in 1993 [22]. It was subsequently included in a measure comparison by Baldi et al. [21]. Since then, it has gained some traction in biological literature [35–43] and has been seen in network management literature [44]. The measure is sometimes called the information coefficient or mutual information coefficient; we use the acronym IC.As explained by Baldi et al., IC is the mutual information (I) normalized by the entropy in ground truth (H); I is the mutual information contained in ground truth regarding the test set S(Y∪Y¯) and the CPD prediction of ground truth, as contained in Z∪Z¯:
(11)IC=I(Y∪Y¯,Z∪Z¯)H(Y∪Y¯).
Expressing I and H in terms of JPT categories,
(12)I(Y∪Y¯,Z∪Z¯)=-H(T+N,F+N,F-N,T-N)-T+Nlog(|Y|*|Z|)-F+Nlog(|Y¯|*|Z|)-F-Nlog(|Y|*|Z¯|)-T-Nlog(|Y¯|*|Z¯|),
where
(13)H(T+N,F+N,F-N,T-N)=-T+NlogT+N-F+NlogF+N-F-NlogF-N-T-NlogT-N.Information-theory-based measures are gaining traction in the literature [35–43]. Some of these reports indicate the belief that the measures are rCS-invariant [38, 40, 43]. Solis and Rackovsky [41] note that their particular information theoretic measure may not be rCS-invariant. The belief that information theoretic measures are rCS-invariant comes from the fact that information theory applies to probability density functions, which are always normalized (rCS=1) [45, 46]. Unless JPTs are normalized prior to use, IC and related measures cannot be guaranteed to be rCS-invariant.Like other measures, IC compares target CPD output to an CPD using random classification. However, it differs in that IC is based on the entropy existing in the test set and CDP output. If the input and output are the same, thenIC=1; if the output of the process is equivalent to that of random selection, then IC=0. A side effect of IC’s use of logs is increased computational complexity. All of the other measures evaluated have a complexity of O(N), IC is O(N2). This may limit IC’s utility for large data sets. IC’s computational complexity did affect our analysis. Had we calculated IC on the two hundred thousand element test sets used for the other measures, it would have taken approximately six months. Consequently, we tested IC on twenty thousand element test sets. In Figure 6, we can see that the peak boundary shifts as rCS increases; thus, IC is not rCS-invariant. As with the other rCS-sensitive measures, JPT normalization can confer rCS invariance.Figure 6
The sloped dotted line on the contour graph shows that IC is notrCS-invariant.


## 4.2. Findings regarding rCS Sensitive Problems

(i)
Of the four rCS-sensitive summary statistics reviewed, TAR andF-score appear to be efficacious.
(ii)
Because the reaction to rCS of TAR andF-score are opposite, this may indicate the existence of other elements in the CPD problem structure (we will address that in future work.)
(iii)
MCC and IC, regardless of their apparent utility for researchers, do not seem to quantify information directly usable by end users. This section has not covered how end users can take a single reported value and convert it into one applicable to their specific rCS environment. This will be discussed in Section 5.


## 4.3. When rCS Is Confounding

When rCS is confounding, in addition to quantifying end user issues, efficacious measures must be rCS-invariant. The following discussion will apply normalized JPT input when necessary.The CPD application expressions for this problem type are normalized versions of those for rCS-sensitive problems.“Given that the test is positive, to what extent can the result be relied upon?” Mathematically, this can be expressed as a conditional probability, or a conditional odds on normalized JPTs. For the reason mentioned in Section4, we use the odds, normalized Ω(T+∣Z)=T+Y-/F+Y.“Given that the test is negative, to what extent can the result be relied upon?” The odds expression for this is normalizedΩ(T-∣Z-)=T-Y/F_Y-.Test selection needs to choose the CPD with the best expected accuracy (“given that a result will be rendered, to what extent can the result be relied upon?”). The two CPD application questions provide operational information, but are also the basis for this noncumulative CPD problem selection decision. As such, they can be considered the measurement suite for the CPD selection decision. The CPD selection decision requires an end user efficacious summary statistic. The expected prediction accuracy (EPA) is the average of the two odds ratios identified in the previous paragraph. Each CPD event is independent and the conditional values are normalized. The special conditions that dictate applying either the geometrical mean (a compounding effect) or the harmonic mean (unequal set sizes) do not exist; so the arithmetic mean of the conditional odds on normalized JPTs is appropriate:(14)EPA=(T+Y¯)/(F+Y)+(T-Y)/(F-Y¯)2.
To our knowledge, this end user summary statistic is not found in the literature. We apply this summary statistic in the meta-analysis in Section 6.As noted in the introduction, rCS-invariant summary statistics are already in use. We review three commonly seen rCS-invariant summary statistics:(i)
the Youden index [15];(ii)
two related measures: diagnostic odds ratio (DOR) [16] and diagnostic power (DP) [17] (DP=(3/π)log(DOR));(iii)
the receiver (or relative) operating characteristic area under the curve (AUC) [19, 20]. Two other summary statistics, the Matthews correlation coefficient (MCC) [18] and mutual information coefficient (IC) [21] are commonly held to be rCS-invariant, but in fact are not. They were discussed in Section 4.1.


### 4.3.1. Youden Index

The Youden index (traditionally represented byJ) was proposed in 1950 and is seen in medical diagnostic studies [15]. There are a number of expressions of J. The original is
(15)J=12[T+-F+T++F++T--F-T-+F-].
Perhaps a more common representation is
(16)J=sensitivity+specificity-1,
where
(17)sensitivity=T+Y,specificity=T-Y¯.
Further, sensitivity is also known as the true positive rate (TPR) and specificity is the complement of the false positive rate; specificity = 1-FPR=1-(F+/Y-). Hence, an even simpler (thus better, according to the minimum description length principal) definition would be
(18)J=TPR-FPR.
In this form, the Youden index can be taken to be a summary statistic of the measure suite {TPR,FPR}.J is special in that J=0 indicates an CPD with an output equal to that of tossing a fair coin. J=1 with a perfect CPD and J=-1 for an CPD that misclassifies everything. As noted in their respective literature bases, J shares a characteristic with AUC, in that it is insensitive to rCS. On our source populations, the optimum boundary is approximately 1.4. This can be seen in Figure 7. There is an issue with end user efficacy, however. J quantifies the spread between the TPR and FPR. This information has little bearing on the “pretest” question posed at the beginning of this section.Figure 7
The Youden index has a very uniform shape and the optimum boundary lies along the peak of the Youden index ridge. This exhibits the expected rCS invariance.


### 4.3.2. Diagnostic Odds Ratio (DOR) and Discriminant Power (DP)

Two related measures are the diagnostic odds ratio (DOR) [16] and discriminant power (DP) [17]. DOR is defined as
(19)DOR=T+/F-F+/T-,
where T+/F-is true positive odds (TPO) and F+/T- is false positive odds (FPO). After simplification,
(20)DOR=T+T-F+F-.
Discriminant power is defined as
(21)DP=3π(logX+logW),
where
(22)X=sensitivity1-sensitivity,Y=specificity1-specificity.
Recasting the equation, we get
(23)DP=3πlog(T+T-F+F-).
The derivation is provided in Appendix B. Comparing the two measures, we see that
(24)DP=3πlog(DOR).DOR and DP are found in medical research. Interestingly, DP=-∞ and DOR=0 when either T+=0 or T-=0, both need not equal zero. Similarly, DP=∞ and DOR=∞ when either F+=0 or F-=0, both need not equal zero. Hence, an CPD can classify some observations correctly (total Accuracy > 0), yet have DP=-∞ and DOR=0. This is counterintuitive, since one would expect DP=-∞ and DOR=0 to indicate a totally ineffective CPD and DP=∞ and DOR=∞ to indicate a perfect CPD, rather than something in between. Since T+ and T- are (statistically) independent,14 (as are F+ and F-), the DP and DOR could, in a probabilistic sense, be interpreted as the odds that, given two random observations, one will be classified T+ and the other T- (one will be classified F+ and the other F-). While the question seems similar, the fact that the DOR and DP optimum boundaries are different from the other inherently rCS invariant measures tested suggests that the two questions are significantly different. Perhaps this is because the DP and DOR treat the problem as a multiplicative function; we identify the problem as an additive function. This value would seem to be directly relevant in niche CPD scenarios, but not to general CPD problem types.In medical studies, when the event tested for (T+) has a low probability, DOR approximates relative risk: the rate at which the event was observed in group A versus the rate observed in group B. This is valuable information. However, when applied in the more general CPD domain, there is a problem. In any specific CPD task, the category of interest may not have a sufficiently low probability T+; thus, the approximation may not always be acceptably close.Unfortunately, DOR and DP have a challenging sensitivity to boundary; the optimum boundary is indicated bymin(DOR) (or min(DP)). Thus, for any test run, the boundary with the smallest T+T- relative to F+F- gives the best accuracy. Not only is this counterintuitive, but also a potential error source. The problem originates from the fact that the greater the min(DOR) (or min(DP)), the better the results. Thus, if the boundary used to partition the test output is not at min(DOR) (or min(DP)), the results may appear better than they really are.15 Most observations regarding DOR apply to DP as well. For example, DP=0 when T+T-=F+F-.One important characteristic of DOR and DP is that they are rCS invariant. An important difference between DOR/DP and the other rCS-invariant measures is that their optimum boundaries, although constant in our tests, are offset from the “minimum error boundary.” These effects can be seen in Figure8.16 Since DOR and DP are minima, they follow a valley in the contour graph, instead of a ridge, Also contrary to the other measures, DOR decreases when the absolute class size effect becomes noticeable. This means that the contours are closed, instead of open as seen for the other measures. DOR’s vertical optimum boundary line and constant value (seen in  Figure 8(b)) indicate that DOR (and hence, DP) is rCS-invariant. Because of this boundary bias, they may not be useful for selecting boundaries. For example, in our test environment, TAR at the common optimum boundary is 0.994, TAR at DOR optimum boundary is 0.958; the difference is significant at the 95% confidence level.Instead of the optimum value being maxima, like the other measures evaluated, the optimum DOR value is a minimum. Hence, the contours show a valley instead of a ridge. Also contrary to the other measures, DOR decreases when the absolute class size effect becomes noticeable. This means that the contours are closed, instead of open like the others. DOR’s vertical optimum boundary line and constant value (seen in Figure8(b)) indicate that DOR (and hence, DP) is rCS-invariant. DOR/DP optimum boundaries (approx. 1.2) are offset from the optimum boundaries seen in the other rCS-invariant measures (approx. 1.4). DP, the log form of DOR, has the same characteristics as DOR.
(a)
Contour graph of rCS versus boundary versus DOR value. Scaling makes the measure seem somewhat rCS sensitive. However, Figure8(b) shows DOR is actually rCS invariant
(b)
Graph of rCS versus DOR, with error bars


### 4.3.3. Receiver Operating Characteristic Area under the Curve (AUC)

ROC has a solid history. Swets campaigned diligently to establish it as the evaluation criterion of choice [20, 47, 48]. The {TPR,FPR} measurement suite is the basis for the AUC summary statistic. The title originates from the fact that it is the area under a “ROC curve,” a curve defined by false positive (FPR=F+/Y-) and true positive (TPR=T+/Y) rates. These values are calculated from JPTs of CPD output for a number of boundaries across the observed range, then graphed as the ROC curve [19, 49].17 The literature describes the ROC curve (AUC) as being rCS-invariant as well as boundary-invariant. Because it is boundary-invariant, AUC is a popular tool in our present research environment. However, AUC has been criticized on theoretical terms recently [50, 51].In contrast to the other summary statistics reviewed, AUC is generally accompanied by the ROC curve (indeed, the ROC curve may be presented without providing AUC). To a person skilled in the art, the ROC curve provides a great deal more information regarding CPD performance than does the single value AUC summary statistic18 (this is, of course, true for any measure suite, since consolidation of multiple values into a single summary statistic value means that information is lost).Compared to our end user focused criteria, ROC-AUC, being boundary-invariant, is not useful for boundary identification. Nor is it efficacious for end users.There are numerous ROC-AUC variants [52, 53]. Vanderlooy and Hüllermeier determined in their comparison, that despite intuitive appeal, none of the variants confer any CPD selection improvement. From the end user perspective, since the underlying measure units remain the same, they all have the same limited efficacy.Figure9 shows the optimum boundary versus rCS for the proposed normalized PPV, NPV average and existing summary statistics, normalized MCC, Youden index, and DOR. As can be seen in the figure, DOR peaks at a different boundary than the other rCS-invariant measures tested, and (excepting DOR) the optimum boundary is relatively stable until rCS>26, after which the detected optimum boundary starts dropping rapidly.19 For our context, a key finding from Figure 9 is that not only does DOR have weak end user efficacy, it also should not be used to identify the optimum boundary.Figure 9
Other than DOR/DP, all of the rCS-invariant measures tested identified the same optimum boundary.


## 4.3.1. Youden Index

The Youden index (traditionally represented byJ) was proposed in 1950 and is seen in medical diagnostic studies [15]. There are a number of expressions of J. The original is
(15)J=12[T+-F+T++F++T--F-T-+F-].
Perhaps a more common representation is
(16)J=sensitivity+specificity-1,
where
(17)sensitivity=T+Y,specificity=T-Y¯.
Further, sensitivity is also known as the true positive rate (TPR) and specificity is the complement of the false positive rate; specificity = 1-FPR=1-(F+/Y-). Hence, an even simpler (thus better, according to the minimum description length principal) definition would be
(18)J=TPR-FPR.
In this form, the Youden index can be taken to be a summary statistic of the measure suite {TPR,FPR}.J is special in that J=0 indicates an CPD with an output equal to that of tossing a fair coin. J=1 with a perfect CPD and J=-1 for an CPD that misclassifies everything. As noted in their respective literature bases, J shares a characteristic with AUC, in that it is insensitive to rCS. On our source populations, the optimum boundary is approximately 1.4. This can be seen in Figure 7. There is an issue with end user efficacy, however. J quantifies the spread between the TPR and FPR. This information has little bearing on the “pretest” question posed at the beginning of this section.Figure 7
The Youden index has a very uniform shape and the optimum boundary lies along the peak of the Youden index ridge. This exhibits the expected rCS invariance.


## 4.3.2. Diagnostic Odds Ratio (DOR) and Discriminant Power (DP)

Two related measures are the diagnostic odds ratio (DOR) [16] and discriminant power (DP) [17]. DOR is defined as
(19)DOR=T+/F-F+/T-,
where T+/F-is true positive odds (TPO) and F+/T- is false positive odds (FPO). After simplification,
(20)DOR=T+T-F+F-.
Discriminant power is defined as
(21)DP=3π(logX+logW),
where
(22)X=sensitivity1-sensitivity,Y=specificity1-specificity.
Recasting the equation, we get
(23)DP=3πlog(T+T-F+F-).
The derivation is provided in Appendix B. Comparing the two measures, we see that
(24)DP=3πlog(DOR).DOR and DP are found in medical research. Interestingly, DP=-∞ and DOR=0 when either T+=0 or T-=0, both need not equal zero. Similarly, DP=∞ and DOR=∞ when either F+=0 or F-=0, both need not equal zero. Hence, an CPD can classify some observations correctly (total Accuracy > 0), yet have DP=-∞ and DOR=0. This is counterintuitive, since one would expect DP=-∞ and DOR=0 to indicate a totally ineffective CPD and DP=∞ and DOR=∞ to indicate a perfect CPD, rather than something in between. Since T+ and T- are (statistically) independent,14 (as are F+ and F-), the DP and DOR could, in a probabilistic sense, be interpreted as the odds that, given two random observations, one will be classified T+ and the other T- (one will be classified F+ and the other F-). While the question seems similar, the fact that the DOR and DP optimum boundaries are different from the other inherently rCS invariant measures tested suggests that the two questions are significantly different. Perhaps this is because the DP and DOR treat the problem as a multiplicative function; we identify the problem as an additive function. This value would seem to be directly relevant in niche CPD scenarios, but not to general CPD problem types.In medical studies, when the event tested for (T+) has a low probability, DOR approximates relative risk: the rate at which the event was observed in group A versus the rate observed in group B. This is valuable information. However, when applied in the more general CPD domain, there is a problem. In any specific CPD task, the category of interest may not have a sufficiently low probability T+; thus, the approximation may not always be acceptably close.Unfortunately, DOR and DP have a challenging sensitivity to boundary; the optimum boundary is indicated bymin(DOR) (or min(DP)). Thus, for any test run, the boundary with the smallest T+T- relative to F+F- gives the best accuracy. Not only is this counterintuitive, but also a potential error source. The problem originates from the fact that the greater the min(DOR) (or min(DP)), the better the results. Thus, if the boundary used to partition the test output is not at min(DOR) (or min(DP)), the results may appear better than they really are.15 Most observations regarding DOR apply to DP as well. For example, DP=0 when T+T-=F+F-.One important characteristic of DOR and DP is that they are rCS invariant. An important difference between DOR/DP and the other rCS-invariant measures is that their optimum boundaries, although constant in our tests, are offset from the “minimum error boundary.” These effects can be seen in Figure8.16 Since DOR and DP are minima, they follow a valley in the contour graph, instead of a ridge, Also contrary to the other measures, DOR decreases when the absolute class size effect becomes noticeable. This means that the contours are closed, instead of open as seen for the other measures. DOR’s vertical optimum boundary line and constant value (seen in  Figure 8(b)) indicate that DOR (and hence, DP) is rCS-invariant. Because of this boundary bias, they may not be useful for selecting boundaries. For example, in our test environment, TAR at the common optimum boundary is 0.994, TAR at DOR optimum boundary is 0.958; the difference is significant at the 95% confidence level.Instead of the optimum value being maxima, like the other measures evaluated, the optimum DOR value is a minimum. Hence, the contours show a valley instead of a ridge. Also contrary to the other measures, DOR decreases when the absolute class size effect becomes noticeable. This means that the contours are closed, instead of open like the others. DOR’s vertical optimum boundary line and constant value (seen in Figure8(b)) indicate that DOR (and hence, DP) is rCS-invariant. DOR/DP optimum boundaries (approx. 1.2) are offset from the optimum boundaries seen in the other rCS-invariant measures (approx. 1.4). DP, the log form of DOR, has the same characteristics as DOR.
(a)
Contour graph of rCS versus boundary versus DOR value. Scaling makes the measure seem somewhat rCS sensitive. However, Figure8(b) shows DOR is actually rCS invariant
(b)
Graph of rCS versus DOR, with error bars


## 4.3.3. Receiver Operating Characteristic Area under the Curve (AUC)

ROC has a solid history. Swets campaigned diligently to establish it as the evaluation criterion of choice [20, 47, 48]. The {TPR,FPR} measurement suite is the basis for the AUC summary statistic. The title originates from the fact that it is the area under a “ROC curve,” a curve defined by false positive (FPR=F+/Y-) and true positive (TPR=T+/Y) rates. These values are calculated from JPTs of CPD output for a number of boundaries across the observed range, then graphed as the ROC curve [19, 49].17 The literature describes the ROC curve (AUC) as being rCS-invariant as well as boundary-invariant. Because it is boundary-invariant, AUC is a popular tool in our present research environment. However, AUC has been criticized on theoretical terms recently [50, 51].In contrast to the other summary statistics reviewed, AUC is generally accompanied by the ROC curve (indeed, the ROC curve may be presented without providing AUC). To a person skilled in the art, the ROC curve provides a great deal more information regarding CPD performance than does the single value AUC summary statistic18 (this is, of course, true for any measure suite, since consolidation of multiple values into a single summary statistic value means that information is lost).Compared to our end user focused criteria, ROC-AUC, being boundary-invariant, is not useful for boundary identification. Nor is it efficacious for end users.There are numerous ROC-AUC variants [52, 53]. Vanderlooy and Hüllermeier determined in their comparison, that despite intuitive appeal, none of the variants confer any CPD selection improvement. From the end user perspective, since the underlying measure units remain the same, they all have the same limited efficacy.Figure9 shows the optimum boundary versus rCS for the proposed normalized PPV, NPV average and existing summary statistics, normalized MCC, Youden index, and DOR. As can be seen in the figure, DOR peaks at a different boundary than the other rCS-invariant measures tested, and (excepting DOR) the optimum boundary is relatively stable until rCS>26, after which the detected optimum boundary starts dropping rapidly.19 For our context, a key finding from Figure 9 is that not only does DOR have weak end user efficacy, it also should not be used to identify the optimum boundary.Figure 9
Other than DOR/DP, all of the rCS-invariant measures tested identified the same optimum boundary.


## 4.4. Findings regarding rCS-Invariant End User Problems

For problems requiring rCS invariance, we find that(i)
end users need three values. For CPD selection, the expected total predictive accuracy (a summary statistic)EPA=((T+Y-)/(F+Y)+(T-Y)/(F-Y-))/2 is important. When the CPD is used in the field, the summary statistic value has no meaning. Instead, end users need the information provided by the two measure suite elements; the positive predictive value odds ratio PPV=(T+Y-)/(F+Y) and the negative predictive value odds ratio NPV=(T-Y)/(F-Y-).(ii)
Although many commonly seen summary statistics can be used to identify the optimum boundary, as seen in this study, not all can (e.g., ROC-AUC and DOR). Figure9 shows how the optimum boundary identified by DOR differs from that identified by the other summary statistics tested.(iii)
Of the rCS sensitive summary statistics evaluated, only the EPA output answers the end user’s CPD selection question. The others may be useful for niche problems, but provide little useful information for the “common” end user.


## 5. JPT Normalization

In statistical circles, standardizing distributions is a well-established technique. One effect of standardization is that the area under the probability density function (pdf) equals 1. This simplifies pdf analysis, since the area of any segment of the area under the curve can be interpreted as the probability of an event occurring within that segment. Similarly, distribution standardization facilitates pdf comparisons. Since the CPD analysis domain considers processes with overlapping pdfs, it intersects with the pdf comparison domain, but is neither a superset nor a subset.20 Where appropriate, distribution standardization is a useful tool.In CPD analysis, distribution standardization takes the form of JPT normalization. Table4 shows a JPT displaying “raw” data—actual category cardinality. After normalization, the class totals (bottom row in Table 5) are one. Thus, JPT normalization seems to be a cause for rCS-invariance in measures. As such, it provides a benefit to end users with rCS invariant problems: any JPT-based CPD evaluation measure will have rCS invariant output, if the input JPTs are normalized. (Illustrated in Figure 10).Table 4
This JPT holds actual category counts.
Actual target classification
A
A
-
Testresult
Positive
T
+
F
+
Z
Negative
F
-
T
-
Z
-
Total
Y
Y
-
NTable 5
The values in this JPT have been normalized. Normalization results in equal class sizes (both total both equal one).
Actual target classification
A
A
-
Test result
Positive
T
+
/
Y
F
+
/
Y
-
Negative
F
-
/
Y
T
-
/
Y
-
Normalized total
1
1
2The normalized accuracy rate andF-score seem to be relatively invariant to  rCS. Not only is the value relatively constant, but the boundary stays constant as well.
(a)
Contour graph of normalized Accuracy rate. The reader may note that, other than the contour values, the graph is almost exactly the same as the Youden index graph
(b)
Contour graph of normalizedF-score. As with Youden index and normalized accuracy rate, the optimum boundary follows the “minimum error boundary”Although any measure can be rCS-invariant when the JPTs are normalized, some measures have emerged which have intrinsic rCS invariance. These inherently rCS invariant measures all have{TPV,FPV} (ratios that normalize the JPTs) as measure suites, thus rather than being counter examples, they provide empirical evidence that JPT normalization is the root cause for rCS invariance in measures; proof is beyond the scope of this paper. An overview of commonly seen rCS invariant measures is provided in Appendix D.There is also a benefit for end users with rCS-sensitive problems. Statisticians use distribution standardization to mitigate rCS effects; however, the process is reversible. JPTs withrCS=1 can be “tuned” to any desired rCS simply by multiplying one class by a constant c so that cY-/Y equals the desired value.21 Thus, an end user with an rCS-sensitive problem can adjust reported results to fit their need. JPT tuning also allows end users to execute sensitivity analyses and estimate how the CPD will perform in their environment, over the expected rCS range. These insights are applied to a real-world problem in Section 6; a comparison of two RA diagnostic tests and an intrusion detection problem.However, the optimum boundary is rCS-dependant; thus, the tool is not complete. To apply to all end users, results for all possible optimum boundaries would need to be provided.22 This is impractical, if not impossible, for CPD test reports to include. As illustrated in Endnote 15, the tuned JPTs will indicate trends, but cannot be considered definitive. Nonetheless, JPT tuning extends JPT normalization in a way we have not previously seen in the literature and provides end users with a useful capability.


## 6. Examples

In this section, we use our proposed end user efficacious data analysis on two real-world problems, a meta-analysis23 comparing two medical diagnostic tests for rheumatoid arthritis (RA) by [54] and data from a cyber security masquerading study by [55].


### 6.1. A Meta-Analysis of Rheumatoid Arthritis Diagnostic Tests

The meta-analysis is quite thorough and accounts for many potential variations between studies. Three hundred and two relevant studies were found; eighty-six satisfied the rigorous inclusion criteria. The team concludes that one test is better than the other, however, does so without using a summary statistic. Our reanalysis adds the three recommended measures identified in Section4.3.The study uses two measures: positive likelihood ratio (LR+) and negative likelihood ratio (LR-). Given a typical test using supervised inputs (where ground truth is known), these two values are efficacious for researchers. They are less efficacious where end users have only the CPD output and ground truth is unknown. As we note in Section 4.3, PPV and NPV are more relevant for end users.On pooled data, theLR- differences between the tests were statistically insignificant. However, the LR+ results were statistically significant. On the pooled data, the “anti-CCP”24 positive test results were more frequently correct than the “RF”25 diagnoses. The authors make one important point regarding rheumatoid arthritis treatment; it is harmful and costly to treat persons with false positive results. Hence, it is important to correctly diagnose negatives as well as positives: total expected predictive accuracy (EPA) is important for rheumatoid arthritis treatment. JPT normalization allows measurement of EPA that is rCS-invariant. Using normalized JPT data,
(25)EPA=T+/F++T-/F-2.
In our extension to Nishimura et al.’s report, we calculate the normalized EPA, PPV, and NPV on the pooled test data. Table 6 shows the original likelihood ratios reported by Nishimura et al. and EPA (the parenthesized range is the 95% confidence interval.26) LR-s and PPV show similar values, but LR+ is about one-sixth of the NPV; end users should be cautious when interpreting likelihood ratios. Comparing EPAs for each test and keeping in mind the end user context requires rCS invariance, the anti-CCP test correct diagnosis rate is a little more than twice the correct diagnoses rate of the RA test. This is true, even though, as can be seen in Table 7, the RF test actually more accurately detects RA’s presence.Table 6
Rheumatoid arthritis is a disease where both nontreatment and unnecessary treatment have negative consequences. Thus, knowing the overall predictive accuracy rate is useful information for a practitioner. These tables show the summary likelihood ratios originally reported and the corresponding normalized predictive accuracy odds ratios. Although the anti-CCP test is significantly better, its overall accuracy is not as great, nor is the negative predictive value as poor as one might believe, based simply on the likelihood ratio. (the parenthesized range in this and subsequent tables is the 95% confidence interval).
(a)
Test
Normalized odds ratio measures
LR
+
LR
−
Anti-CCP
12.46 (9.72–15.98)
0.36 (0.031–0.042)
RF
4.86 (3.95–5.97)
0.38 (0.33–0.44)
(b)
Test
Normalized odds ratio measures
PPV
NPV
EPA
Anti-CCP
13.4 (13.0–17.0)
2.88 (2.71–3.0)
8.14 (7.86–10.0)
RF
4.6 (4.25–5.0)
2.74 (2.63–2.87)
3.67 (3.44–3.93)Table 7
These normalized JPTs of  Nishimura et al.’s [54] pooled anti-CCP and RF test data were generated using their reported sensitivities and specificities. A person without RA is far less likely to be misdiagnosed than one with the disease, when the anti-CCP test is used.
(a)
Actual RA condition
Diseased
Not diseased
Anti-CC test result
Positive
0.67 (0.65–0.68)
0.05 (0.04–0.06)
Negative
0.33 (0.32–0.35)
0.95 (0.94–0.95)
Total
1
1
(b)
Actual RA condition
Diseased
Not diseased
RF test result
Positive
0.69 (0.68–0.7)
0.15 (0.14–0.16)
Negative
0.31 (0.3–0.32)
0.85 (0.84–0.86)
Total
1
1The authors note that “the better accuracy of anti-CCP antibody was mainly due to its higher specificity.” In comparing the JPTs in Table7, the anti-CCP pooled data summary in Table 7 shows that the anti-CCP test is actually less accurate in detecting diseased individuals—and at a statistically significant level (0.67 for anti-CCP is (statistically) significantly worse than the 0.69 reported for RF). We see that the anti-CCP actually detects RA less reliably than the RF test; the improvement is, in fact, entirely due to better specificity (correctly identifying nondiseased). In a case such as this, where each test is more accurate on one class, rather than one test being more accurate on both classes, it may not always be clear if there is any net diagnostic improvement. Normalized total predictive accuracy quantifies net diagnostic improvement in a way that may help clarify these issues.This RA example is an rCS confounding type problem; in order to mitigate rCS bias, JPT normalization should be applied. We can now apply JPT tuning to illustrate how rCS can skew results; it is possible to estimate the cumulative test results that GPs and RA specialists will actually observe in their respective practices. The method actually “tunes” the JPTs; any desired rCS can be set.27 A general practitioner may occasionally test for RA. Actual testing rates do not appear to be publicly available; so for computational simplicity, we assume the odds are one to one hundred that someone tested actually has the disease. Because of his/her specialty, a rheumatologist may have a new patient base that is highly skewed toward RA-positive. We assume a one hundred to one ratio. What total accuracy ratios will the two offices observe for the two tests? An JPT tuned to the rheumatologists’ patient base is shown in Table 8, an JPT tuned to the GP’s patient base is shown in Table 9. The EPA odds ratio observed by the rheumatologist would be anti-CCP: 670, RF: 250; the GP would observe total accuracy ratios of anti-CCP: 144, RF: 131, a statistically insignificant difference. Summary Table 10 shows that practices will have radically different experiences with the two tests, although the anti-CCP test is still best for the patient, regardless of the office.Table 8
The above two JPTs have been “tuned” to a population where the diseased population is one hundred times the undiseased population. In this environment, the cumulative results will cause the anti-CCP test to appear to outperform the RF test.
(a)
Actual RA condition
Diseased
Not diseased
Anti-CC test result
Positive
67
0.05
Negative
33
0.95
Totals
100
1
EPA
670 (850–650)
(b)
Actual RA condition
Diseased
Not diseased
RF test result
Positive
69
0.15
Negative
31
0.85
Totals
100
1
EPA
230 (250–212)Table 9
The top two JPTs have been “tuned” to a population where the diseased population is one hundredth of the undiseased population. In this environment, the two tests appear statistically indistinguishable.
(a)
Actual RA condition
Diseased
Not diseased
Anti-CCP
Positive
0.67
5
test result
Negative
0.33
95
Totals
1
100
EPA
144 (136–150)
(b)
Actual RA condition
Diseased
Not diseased
RF test
Positive
0.69
15
Result
Negative
0.31
85
Totals
1
100
EPA
137 (131–143)Table 10
This table shows the total accuracy ratios for RA tested populations of 100 : 1 (the rheumatologist) and 1 : 100 (the general practitioner).
Test
Patient bases (diseased : undiseased)
Rheumatologist (100 : 1)
General practitioner (1 : 100)
Anti-CCP
670 (850–650)
144 (136–150)
RF
230 (250–212)
137 (131–143)


### 6.2. A Cyber Security Masquerade Study

The cyber security problem domain is one where cumulative effects (e.g., processing false alarms (F+)) are important. Consider an end user desiring to detect masquerading attacks, in which an attacker pretends to be an authorized user in order to gain access to a system. Determining the appropriate boundary for the detector is necessary in order to balance the effects of false alarms and missed attacks. This balance is subject to the relative volume of normal and masquerade system activity; thus, rCS is important; end users will want to incorporate rCS.Schonlau et al. simulate a masquerade attack by capturing UNIX commands resulting from specific users, then inserting UNIX commands generated by another user into the original command stream. They compare a number of detection algorithms. The best performing was based on data compression. The Bayesian classifier [55] they used did not perform as well. We compare the two algorithms using the seven summary statistics discussed earlier.Schonlau et al. use ROC curves to compare their various detection algorithms. From a research perspective, this is appropriate, since ROC is invariant to rCS and does not require boundary selection. However, as noted in Section4.3.3, ROC provides limited information to end users. To illustrate the effect of using an inappropriate measure type, we reanalyze one of the user command streams with both rCS-sensitive and rCS-invariant measures. Table 11 shows the results.Table 11
These tables show the results for the compression-based classifier and the Bayesian classifier. The measures output on different scales and measure different characteristics; they cannot be directly compared. Because these are two different classifiers, the output ranges differ.
(a)
Compression classifier
rCS-sensitive measures
rCS-invariant measures
Measure
Value
Boundary
Measure
Value
Boundary
IC
0.528
0.800
Youden
0.791
0.200
TAR
0.930
0.800
DOR
6.78
0.200
MCC
0.786
0.600
ROC-AUC
0.851
NA
T
+
F-score
0.829
0.800
(b)
Bayesian classifier
rCS-sensitive measures
rCS-invariant measures
Measure
Value
Boundary
Measure
Value
Boundary
IC
0.068
188
Youden
0.057
-
228
TAR
0.620
638
DOR
0.206
638
MCC
0.053
-
228
ROC-AUC
0.505
NA
T
+
F-score
0.543
-
387For all seven summary statistics considered, the classifier with the higher value is better. Clearly, regardless of the measure, the compression algorithm outperforms the Bayesian classifier. The summary statistic values and associated optimum thresholds, however, vary widely.What do we learn about the two classifiers from the measure values? The IC measures information content; MCC measures covariance. Youden index and DOR/DP quantify more esoteric characteristics. All four measure classifier performance relative to random selection using a fair coin, an issue particularly relevant to researchers, who generally consider a fair coin to be the most ineffective classifier. ROC-AUC, being rCS- and boundary-invariant, also has attractive characteristics for research. End users, however, are concerned about the net result, not distance from random selection. While each of these five measures quantify a characteristic related to the classifier performance characteristic of interest, none can be transformed into a value useful in the intrusion detection domain.JPT tuning can help end users make more informed decisions. Schonlau et al.’s test sets consisted of one hundred blocks of concatenated UNIX commands. For “user 24,”rCS=3.7. “In the wild,” one would expect rCS to be considerably smaller. For this example, we will assume that the end users expect rCS∈[10K,100K]. For the intrusion detection problem domain, TAR or F-score may provide the most information regarding end result. TAR includes both T+ and T-; F-score only includes T+.28 An IT system administrator may be most concerned about intrusion risk and detection overhead, thus not so concerned about T-. If so, then F-score may be most relevant when comparing cyber security tools. Consider the ΩF-score of Schonlau et al.’s raw data in Table 12. The system administrator can tell that when rCS=3.7, the correctly detected masquerade activity should be almost five times as frequent as errors; this is the system administrators greatest area of concern. A corporate executive might be concerned about the effect all four categories could have on the enterprise’s performance; thus, TOR would be most relevant. Consider the TOR score of Schonlau et al.’s raw data in Table 12. The executive can tell that when rCS=3.7, there will be over 13 correctly classified events for every misclassification. Based on these values, both persons might decide that performance is acceptable. JPT tuning, however, changes the picture considerably. The executive will see accuracy triple, but the system administrator will see a decrease in accuracy of more than three orders of magnitude. The executive and system administrator may now have different opinions.Table 12
This table shows how JPT tuning can assist end users in estimating how an CPD will work in their environment. An executive looking at TOR will see that there are 25 correct classifications for every incorrect in the expected operating range. The IT system administrator looking atΩF-score will see that there will be thousands of errors for every correct T+. Their decisions regarding the usefulness of this CPD may differ.
rCS =
TOR
Ω
F-score
1.0
7.8
7.1
3.7
13.3
4.9
1,000
19
0.042
10 K
25
0.0021
100 K
25
0.0004Another problem with selecting an inappropriate summary statistic can be seen in Table11. Not all measures have the same optimum threshold. An end user relying on an inappropriate summary statistic to determine a useful boundary for masquerade detection may be disappointed with their results.After a classifier is selected, the two perspectives can lead to different system optimizations. When made available to end users, TAR andF-score values can help stakeholders such as executives and IT managers make more informed decisions. Since TAR/TOR and F-score/ΩF-score may have different optimum boundaries, practitioners and decision makers may benefit from having both values reported for each optimum boundary over an rCS range. That way, end users will have an appreciation of the tradeoff associated with selecting a particular solution.


## 6.1. A Meta-Analysis of Rheumatoid Arthritis Diagnostic Tests

The meta-analysis is quite thorough and accounts for many potential variations between studies. Three hundred and two relevant studies were found; eighty-six satisfied the rigorous inclusion criteria. The team concludes that one test is better than the other, however, does so without using a summary statistic. Our reanalysis adds the three recommended measures identified in Section4.3.The study uses two measures: positive likelihood ratio (LR+) and negative likelihood ratio (LR-). Given a typical test using supervised inputs (where ground truth is known), these two values are efficacious for researchers. They are less efficacious where end users have only the CPD output and ground truth is unknown. As we note in Section 4.3, PPV and NPV are more relevant for end users.On pooled data, theLR- differences between the tests were statistically insignificant. However, the LR+ results were statistically significant. On the pooled data, the “anti-CCP”24 positive test results were more frequently correct than the “RF”25 diagnoses. The authors make one important point regarding rheumatoid arthritis treatment; it is harmful and costly to treat persons with false positive results. Hence, it is important to correctly diagnose negatives as well as positives: total expected predictive accuracy (EPA) is important for rheumatoid arthritis treatment. JPT normalization allows measurement of EPA that is rCS-invariant. Using normalized JPT data,
(25)EPA=T+/F++T-/F-2.
In our extension to Nishimura et al.’s report, we calculate the normalized EPA, PPV, and NPV on the pooled test data. Table 6 shows the original likelihood ratios reported by Nishimura et al. and EPA (the parenthesized range is the 95% confidence interval.26) LR-s and PPV show similar values, but LR+ is about one-sixth of the NPV; end users should be cautious when interpreting likelihood ratios. Comparing EPAs for each test and keeping in mind the end user context requires rCS invariance, the anti-CCP test correct diagnosis rate is a little more than twice the correct diagnoses rate of the RA test. This is true, even though, as can be seen in Table 7, the RF test actually more accurately detects RA’s presence.Table 6
Rheumatoid arthritis is a disease where both nontreatment and unnecessary treatment have negative consequences. Thus, knowing the overall predictive accuracy rate is useful information for a practitioner. These tables show the summary likelihood ratios originally reported and the corresponding normalized predictive accuracy odds ratios. Although the anti-CCP test is significantly better, its overall accuracy is not as great, nor is the negative predictive value as poor as one might believe, based simply on the likelihood ratio. (the parenthesized range in this and subsequent tables is the 95% confidence interval).
(a)
Test
Normalized odds ratio measures
LR
+
LR
−
Anti-CCP
12.46 (9.72–15.98)
0.36 (0.031–0.042)
RF
4.86 (3.95–5.97)
0.38 (0.33–0.44)
(b)
Test
Normalized odds ratio measures
PPV
NPV
EPA
Anti-CCP
13.4 (13.0–17.0)
2.88 (2.71–3.0)
8.14 (7.86–10.0)
RF
4.6 (4.25–5.0)
2.74 (2.63–2.87)
3.67 (3.44–3.93)Table 7
These normalized JPTs of  Nishimura et al.’s [54] pooled anti-CCP and RF test data were generated using their reported sensitivities and specificities. A person without RA is far less likely to be misdiagnosed than one with the disease, when the anti-CCP test is used.
(a)
Actual RA condition
Diseased
Not diseased
Anti-CC test result
Positive
0.67 (0.65–0.68)
0.05 (0.04–0.06)
Negative
0.33 (0.32–0.35)
0.95 (0.94–0.95)
Total
1
1
(b)
Actual RA condition
Diseased
Not diseased
RF test result
Positive
0.69 (0.68–0.7)
0.15 (0.14–0.16)
Negative
0.31 (0.3–0.32)
0.85 (0.84–0.86)
Total
1
1The authors note that “the better accuracy of anti-CCP antibody was mainly due to its higher specificity.” In comparing the JPTs in Table7, the anti-CCP pooled data summary in Table 7 shows that the anti-CCP test is actually less accurate in detecting diseased individuals—and at a statistically significant level (0.67 for anti-CCP is (statistically) significantly worse than the 0.69 reported for RF). We see that the anti-CCP actually detects RA less reliably than the RF test; the improvement is, in fact, entirely due to better specificity (correctly identifying nondiseased). In a case such as this, where each test is more accurate on one class, rather than one test being more accurate on both classes, it may not always be clear if there is any net diagnostic improvement. Normalized total predictive accuracy quantifies net diagnostic improvement in a way that may help clarify these issues.This RA example is an rCS confounding type problem; in order to mitigate rCS bias, JPT normalization should be applied. We can now apply JPT tuning to illustrate how rCS can skew results; it is possible to estimate the cumulative test results that GPs and RA specialists will actually observe in their respective practices. The method actually “tunes” the JPTs; any desired rCS can be set.27 A general practitioner may occasionally test for RA. Actual testing rates do not appear to be publicly available; so for computational simplicity, we assume the odds are one to one hundred that someone tested actually has the disease. Because of his/her specialty, a rheumatologist may have a new patient base that is highly skewed toward RA-positive. We assume a one hundred to one ratio. What total accuracy ratios will the two offices observe for the two tests? An JPT tuned to the rheumatologists’ patient base is shown in Table 8, an JPT tuned to the GP’s patient base is shown in Table 9. The EPA odds ratio observed by the rheumatologist would be anti-CCP: 670, RF: 250; the GP would observe total accuracy ratios of anti-CCP: 144, RF: 131, a statistically insignificant difference. Summary Table 10 shows that practices will have radically different experiences with the two tests, although the anti-CCP test is still best for the patient, regardless of the office.Table 8
The above two JPTs have been “tuned” to a population where the diseased population is one hundred times the undiseased population. In this environment, the cumulative results will cause the anti-CCP test to appear to outperform the RF test.
(a)
Actual RA condition
Diseased
Not diseased
Anti-CC test result
Positive
67
0.05
Negative
33
0.95
Totals
100
1
EPA
670 (850–650)
(b)
Actual RA condition
Diseased
Not diseased
RF test result
Positive
69
0.15
Negative
31
0.85
Totals
100
1
EPA
230 (250–212)Table 9
The top two JPTs have been “tuned” to a population where the diseased population is one hundredth of the undiseased population. In this environment, the two tests appear statistically indistinguishable.
(a)
Actual RA condition
Diseased
Not diseased
Anti-CCP
Positive
0.67
5
test result
Negative
0.33
95
Totals
1
100
EPA
144 (136–150)
(b)
Actual RA condition
Diseased
Not diseased
RF test
Positive
0.69
15
Result
Negative
0.31
85
Totals
1
100
EPA
137 (131–143)Table 10
This table shows the total accuracy ratios for RA tested populations of 100 : 1 (the rheumatologist) and 1 : 100 (the general practitioner).
Test
Patient bases (diseased : undiseased)
Rheumatologist (100 : 1)
General practitioner (1 : 100)
Anti-CCP
670 (850–650)
144 (136–150)
RF
230 (250–212)
137 (131–143)


## 6.2. A Cyber Security Masquerade Study

The cyber security problem domain is one where cumulative effects (e.g., processing false alarms (F+)) are important. Consider an end user desiring to detect masquerading attacks, in which an attacker pretends to be an authorized user in order to gain access to a system. Determining the appropriate boundary for the detector is necessary in order to balance the effects of false alarms and missed attacks. This balance is subject to the relative volume of normal and masquerade system activity; thus, rCS is important; end users will want to incorporate rCS.Schonlau et al. simulate a masquerade attack by capturing UNIX commands resulting from specific users, then inserting UNIX commands generated by another user into the original command stream. They compare a number of detection algorithms. The best performing was based on data compression. The Bayesian classifier [55] they used did not perform as well. We compare the two algorithms using the seven summary statistics discussed earlier.Schonlau et al. use ROC curves to compare their various detection algorithms. From a research perspective, this is appropriate, since ROC is invariant to rCS and does not require boundary selection. However, as noted in Section4.3.3, ROC provides limited information to end users. To illustrate the effect of using an inappropriate measure type, we reanalyze one of the user command streams with both rCS-sensitive and rCS-invariant measures. Table 11 shows the results.Table 11
These tables show the results for the compression-based classifier and the Bayesian classifier. The measures output on different scales and measure different characteristics; they cannot be directly compared. Because these are two different classifiers, the output ranges differ.
(a)
Compression classifier
rCS-sensitive measures
rCS-invariant measures
Measure
Value
Boundary
Measure
Value
Boundary
IC
0.528
0.800
Youden
0.791
0.200
TAR
0.930
0.800
DOR
6.78
0.200
MCC
0.786
0.600
ROC-AUC
0.851
NA
T
+
F-score
0.829
0.800
(b)
Bayesian classifier
rCS-sensitive measures
rCS-invariant measures
Measure
Value
Boundary
Measure
Value
Boundary
IC
0.068
188
Youden
0.057
-
228
TAR
0.620
638
DOR
0.206
638
MCC
0.053
-
228
ROC-AUC
0.505
NA
T
+
F-score
0.543
-
387For all seven summary statistics considered, the classifier with the higher value is better. Clearly, regardless of the measure, the compression algorithm outperforms the Bayesian classifier. The summary statistic values and associated optimum thresholds, however, vary widely.What do we learn about the two classifiers from the measure values? The IC measures information content; MCC measures covariance. Youden index and DOR/DP quantify more esoteric characteristics. All four measure classifier performance relative to random selection using a fair coin, an issue particularly relevant to researchers, who generally consider a fair coin to be the most ineffective classifier. ROC-AUC, being rCS- and boundary-invariant, also has attractive characteristics for research. End users, however, are concerned about the net result, not distance from random selection. While each of these five measures quantify a characteristic related to the classifier performance characteristic of interest, none can be transformed into a value useful in the intrusion detection domain.JPT tuning can help end users make more informed decisions. Schonlau et al.’s test sets consisted of one hundred blocks of concatenated UNIX commands. For “user 24,”rCS=3.7. “In the wild,” one would expect rCS to be considerably smaller. For this example, we will assume that the end users expect rCS∈[10K,100K]. For the intrusion detection problem domain, TAR or F-score may provide the most information regarding end result. TAR includes both T+ and T-; F-score only includes T+.28 An IT system administrator may be most concerned about intrusion risk and detection overhead, thus not so concerned about T-. If so, then F-score may be most relevant when comparing cyber security tools. Consider the ΩF-score of Schonlau et al.’s raw data in Table 12. The system administrator can tell that when rCS=3.7, the correctly detected masquerade activity should be almost five times as frequent as errors; this is the system administrators greatest area of concern. A corporate executive might be concerned about the effect all four categories could have on the enterprise’s performance; thus, TOR would be most relevant. Consider the TOR score of Schonlau et al.’s raw data in Table 12. The executive can tell that when rCS=3.7, there will be over 13 correctly classified events for every misclassification. Based on these values, both persons might decide that performance is acceptable. JPT tuning, however, changes the picture considerably. The executive will see accuracy triple, but the system administrator will see a decrease in accuracy of more than three orders of magnitude. The executive and system administrator may now have different opinions.Table 12
This table shows how JPT tuning can assist end users in estimating how an CPD will work in their environment. An executive looking at TOR will see that there are 25 correct classifications for every incorrect in the expected operating range. The IT system administrator looking atΩF-score will see that there will be thousands of errors for every correct T+. Their decisions regarding the usefulness of this CPD may differ.
rCS =
TOR
Ω
F-score
1.0
7.8
7.1
3.7
13.3
4.9
1,000
19
0.042
10 K
25
0.0021
100 K
25
0.0004Another problem with selecting an inappropriate summary statistic can be seen in Table11. Not all measures have the same optimum threshold. An end user relying on an inappropriate summary statistic to determine a useful boundary for masquerade detection may be disappointed with their results.After a classifier is selected, the two perspectives can lead to different system optimizations. When made available to end users, TAR andF-score values can help stakeholders such as executives and IT managers make more informed decisions. Since TAR/TOR and F-score/ΩF-score may have different optimum boundaries, practitioners and decision makers may benefit from having both values reported for each optimum boundary over an rCS range. That way, end users will have an appreciation of the tradeoff associated with selecting a particular solution.


## 7. Conclusion

This paper is a first step in identifying the structure of CPD problems faced by end users. Using that structure, we characterize how CPD evaluation measures are relevant to end users and identify end user relevant evaluation tools. To that end, we have defined rCS’s importance to end user problems, identified measures that are efficacious for end users, and shown how JPT normalization and JPT tuning are useful for end user CPD evaluation.Depending upon whether the end user is interested in the cumulative output or each individual CPD output, rCS is either an important factor or confounding. For maximum end user utility, research reports should include information efficacious for both problem types:(i)
for “rCS is confounding” problems, end users need a summary statistic,EPA=((T+Y¯)/(F+Y)+(T-Y)/(F-Y¯))/2 and the underlying measurement suite, PPV=(T+Y¯)/(F+Y), NPV=(T-Y)/(F-Y¯). All three values are based on normalized JPTs. If the values used are from normalized JPTs, then Y and  Y¯ both equal one, thus are unnecessary.(ii)
For “rCS is important” problems, end users must be able to tailor results to suit their individual rCS environments. We identify one appropriate summary statistic; the total accuracy odds ratioTOR=(T++T-)/(F++F-). Another, F-score, is already in use: Fβ=((1+β2)(precision)(recall))/((β2)(precision+recall)), where β is the relative weight of precision and recall: β is the importance of precision relative to the importance of recall. End users can apply JPT tuning to tailor results for their environment. To do so, they will require the base JPT values ({T+,F+,F-,T-}). Consolidating these findings, we propose that end users will be better served if research reports include PPV, NPV, EPA (or Fβ, if it is prevalent in the domain), and the four normalized base JPT values.Future work will continue to develop a CPD problem structure. The disparity between TAR andF-score suggests that at least one more characteristic exists. Also, without compensating for the effect of the shift in optimum boundary, JPT tuning does not fully address the end user’s need to tailor research results. We will be considering means of addressing that deficiency.



---

*Source: 427958-2013-02-26.xml*