The Confusing Matrix
By: Ron Itzikovitch
Tags: Statistics, Mathematics, Data Science, Probability, Machine Learning

Question Types

There are many types of questions we can ask about the various performance metrics. Here we will only cover a few types of questions, and you can build upon these and come up with many more variations.

Type A

Only a single metric is presented in the question, and we condition on the true condition of the subject (sick/healthy).

The answer could either be the metric, 1 minus the metric, or the question can’t be answered using the available information.

QUESTION FORMAT:

The <metric> of a test is X%.

We perform the test on a <healthy / sick> subject. What is the probability that the test will return a <positive / negative> result?

EXAMPLE:

The specificity of a test is 95%.

We perform the test on a healthy subject. What is the probability that the test will return a positive result?

The question asks for the false positive rate, P(+|not D) .

Given specificity and applying the rule P(A|B) = 1 — P(not A|B) we know that P(+|not D) = 1 — P(-|not D) = 1 — specificity , and so the answer is 1 — specificity = 1 — 0.95 = 0.05 .

Type B

Same as the previous question type, but only with added irrelevant information.

QUESTION FORMAT:

The <metric> of a test is X% and its <other metric> is Y%.

We perform the test on a <healthy / sick> subject. What is the probability that the test will return a <positive / negative> result?

EXAMPLE:

The specificity of a test is 95% and its precision is 90%.

We perform the test on a healthy subject. What is the probability that the test will return a positive result?

We don’t need precision to answer the question; specificity suffices. The answer to this question is the same as the answer to the previous question type: 1 - specificity .

Type C

Only a single metric is presented in the question alongside the prevalence or the proportion of positive/negative test results. Note that in this case the question will not ask us to find a conditional probability.

QUESTION FORMAT:

The <metric> of a test is X% and we are told that the <prevalence / proportion of positive test results> is Y%.

We perform the test on a random subject. What is the probability that the test will return a <true / false>-<positive / negative>?

EXAMPLE:

The precision of a test is 95% and we are told the test always finds that 10% of the subjects are positive.

We perform the test on a random subject. What is the probability that the test will return a true-positive?

The questions asks us to find P(+ and D) .

We know that P(A and B) = P(B|A)P(A) and so

P(+ and D) = P(D|+)P(+) = (precision) * (proportion positive results)

And so we get 0.95*0.1 = 0.095 .

Type D

Similar to the previous type, but with added redundant information, and this time we may ask about a conditional probability.

QUESTION FORMAT:

The <metric> of a test is X%. We also know that the test’s <other metric> is Y% and that these metrics were obtained when the test was evaluated on a dataset where the prevalence was Z%.

We perform the test on a single subject, and it returns a <negative / positive> result. What is the probability that it is a <false / true>-<negative / positive>?

EXAMPLE:

The false discovery rate of a test is 5%. We also know that the test’s negative predictive value is 95% and that these metrics were obtained when the test was evaluated on a dataset where the prevalence was 10%.

We perform the test on a single subject, and it returns a negative result. What is the probability that it is a false negative?

The question asks for P(D|-) and we know that

P(D|-) = 1 — P(not D|-) = 1 — negative predictive value = 1 — 0.95 = 0.05.

All information other than the test’s negative predictive value is not required in order to answer the question and is just noise.

Type E

The question includes two metrics and the prevalence or the proportion of positive test results given by the test.

QUESTION FORMAT:

The <metric> of the test is X% and its <other metric> is Y%. Assume we apply the test to a dataset where the <prevalence / proportion of positive results> is Z%. What is the probability that a subject is <healthy / sick> given that the test returned a <positive / negative> result?

EXAMPLE:

The sensitivity of a test is 90% and its specificity is 90%. Assume we apply the test to a dataset where the prevalence is 1%. What is the probability that the subject is sick, given that the test returned a positive result?

The question asks us to find the test’s precision, P(D|+) .

To find P(D|+) we can apply Bayes’ rule and after writing out the terms explicitly, we see that we simply need to plug-in the values of sensitivity, specificity and the prevalence to get the value of precision.

Image by Author

Type F

Same as the previous question type, only nastier — the question includes information which seems relevant, but the question cannot be answered.

EXAMPLE:

Replace specificity with negative predictive value in the previous question type example. Work out the form of precision when we rewrite it using Bayes’ rule, and you will see we end up with terms whose value is not given in the question, and so it cannot be solved.

A Few More Examples

Assume that a test has a specificity of 95%. We run this test on a dataset where none of the subjects has the disease. Will the test return any positive results?

ANSWER: Yes, the test will label 5% of the dataset as positives, even though they are all actually negative for the disease. This is because the false positive rate can also be expressed as 1 — specificity , and so we have 1 — 0.95 = 0.05 . Assume that a test has a sensitivity of 60%. The prevalence in the dataset is 70% and we also know that 80% of the results returned by the test are positive. What is the precision of the test?

ANSWER:

The question asks us to find P(D|+) .

We can solve it using Bayes’ rule:

P(D|+) = P(+|D)P(D) / P(+)

= sensitivity * prevalence / proportion positive results

= 0.6 * 0.7 / 0.8 = 0.525 Assume that a test was evaluated on some dataset and was found to have a precision of 95%. We now run this test on a different dataset where 50% of the subjects have the disease. Will the test return any positive results?

ANSWER: We cannot answer based on the information given in the question.

Considering the information we have we know that P(D|+) = 0.95 and that P(D) = 0.5 .

The question asks for P(+) , or more specifically it asks if P(+) > 0 .

P(+) can be also written as

P(+) = P(D)P(+|D) + P(not D)P(+|not D) .

As we do not know the value of P(+|D) (sensitivity) and we also don’t know the value of P(+|not D) (false positive rate) we cannot answer the question.

Just to note, we do know the value of P(not D) , since 1−P(D) = P(not D) . Assume that a test has a sensitivity of 95% and a specificity of 95%. We also know that the prevalence is 10%. We run the test on 1000 subjects. How many positives will we the test find?

ANSWER: We previously showed how to solve a very similar question using Bayes’ rule. We will now show an alternative way.

We are given the prevalence of the disease, so we know that 10% of 1000 subject have the disease, and so there are 100 actual positives.

The test sensitivity is 95% and so 95 of the 100 actual positives will be found by the test, and they are true-positives.

The test specificity is 95% and so 855 of the 900 actual negatives will be found by the test and they are true-negatives. The 45 remaining actual negatives will be erroneously detected as positives by the test, and will be false-positives. This is the same as noting that our false positive rate is 5% since the false positive rate equals 1 – specificity.

We end up with 95 true-positives + 45 false-positives for a total of 140 positives, which is the quantity we were asked to find.

By the way, we now also know that the test precision is 95/(95+45)=67.9% Assume that a test has a sensitivity of 95% and a specificity of 95%. We also know that the prevalence is 10%. Is it possible that the test’s negative predictive value is 50%?

ANSWER: No.

Following very similar calculations to the previous question, and assuming we ran the test on 1,000 subjects for simplicity:

The prevalence is 10%, so we have 100 actual positives and 900 actual negatives.

95% sensitivity provides us with 95 true-positives and 5 false-negatives.

95% specificity produces 855 true-negatives and 45 false-positives.

The negative predictive value is

TN/(TN + FN) = 855/(855+5) = 99.4% ≠ 50%. A company tries to sell you their novel test which is (supposedly) highly accurate in detecting some disease. They claim that it has a sensitivity of 95%, a specificity of 95% and 95% precision. Is it possible this isn’t some marketing trick and that such a test does exist?

ANSWER: As we saw, we can write precision as a function of prevalence, sensitivity and specificity. Let’s look at it again:

Image by Author

We note that we can rearrange the above so that prevalence will be expressed as a function of sensitivity, specificity and precision. We then plug-in the metrics reported by the company like so:

Image by Author

As it turns out, given the metrics provided by the company, they evaluated their test on a dataset where the prevalence was 50%, which is very high, and might be completely unrelated to the real prevalence we expect to see in the population we intend to use the test on. In other words, applying the same test to a a group of subjects where the prevalence is 2% will result in different performance metrics, and not all of them can be 95%.

Using Bayes Factor

Another way to answer question 4 is by using the Bayes factor, and the method is detailed in this video: “The medical test paradox, and redesigning Bayes’ rule”. I won’t cover the method here since I think the video explains it exceptionally well and I highly recommend watching it.