STATWI0002.04 Cover Page – page 0 2018-03-23

Source document: 

CVB Quality Manual Chapter 5

USDA Center for Veterinary Biologics

Statistics Section

Work Instructions

This document approved for the indicated purposes

Internal use Yes

External distribution Yes

CVB public web site Yes

Document: STATWI0002.04

Title: Technical Notes for Estimating Diagnostic Sensitivity and Specificity

Author: Monica Reising

Approved by: David Siev on 2018-03-23
Technical Notes Regarding Estimating Diagnostic

Sensitivity and Specificity

Statistics Section

Center for Veterinary Biologics

United States Department of Agriculture

1920 Dayton Avenue

Ames, Iowa, USA

March 23, 2018

1 Introduction

Veterinary Services Memorandum 800.73 provides guidance on studies required for the

licensure of diagnostic test kits. This document provides technical details related to estimating diagnostic sensitivity (Sn) and diagnostic specificity (Sp). A simulation method

is described that makes use of the performance characteristics of one or more reference

tests, as well as the prevalence of each population sampled. A software package is available

to implement the procedure. This document assumes that the reader has a substantial

background in mathematical statistics.

2 Notation and Properties

This section provides notation that will be used throughout. Further, properties of probabilities are provided to allow derivations of expected probabilities and hence, expected

counts.

Let

 P(•) represent a marginal probability.

 P(•|•) represent a conditional probability

 D+ indicates disease positive

1

STATWI0002.04 page 1 of 9
2 NOTATION AND PROPERTIES 2

 D− indicates disease negative

 T+ indicates test positive

 T− indicates test negative

 T? indicates test suspect

 i = 1, 2, . . . , I where I is the number of tests (i = 1 refers to the experimental test

and i = 2, . . . , I refers to the (i − 1) reference tests)

 l = 1, 2, . . . , L where L is the number of populations sampled

 k = 1, 2, . . . , K represents the k

th unique combination of test outcomes where K is

the total number of unique combinations of test results

 Si =

(

2, for a 2-state test

3, for a 3-state test

 K =

QI

i=1 Si

, which represents the number of unique test combinations

 πi = Sni = P(Ti+ | D+) for the i

th test

 θi = Spi = P(Ti− | D−) for the i

th test

 ψi = P(Ti? | D+) for the i

th test

 φi = P(Ti? | D−) for the i

th test

 δi =

(

0, if the i

th test has 2 states

ψi

1−πi

, if the i

th test has 3 states

 γi =

(

0, if the i

th test has 2 states

φi

1−θi

, if the i

th test has 3 states

 pl = Pl(D+) for the l

th population (lth population prevalence)

 xk,i =







Ti − if the i

th test is negative in the kth unique combination of test outcomes

Ti + if the i

th test is positive in the kth unique combination of test outcomes

Ti ? if the i

th test is suspect in the kth unique combination of test outcomes

 Mk is a vector of length I with elements (xk,1, xk,2, . . ., xk,I ) representing the k

th

unique combination of test outcomes; Mk 6= Mk

0 for all k 6= k

0

 Pl(Mk) denotes the probability of observing the k

th unique combination of test outcomes within the l

th population.

 nk,l denotes the number of samples (observed count) exhibiting the k

th unique combination of test outcomes within the l

th population.

 Nl =

PK

k=1 nk,l is the total number of samples tested from the l

th population.

STATWI0002.04 page 2 of 9
3 ESTIMATING SN AND SP 3

 nˆk,l denotes the expected count for the k

th unique combination of test outcomes within

the l

th population.

 nˆk,l =

h

[

QI

i=1 P(ωk,i | D+)] Pl

(D+) + [QI

i=1 P(ωk,i | D−)] Pl

(D−)

i

Nl

Law of Total Probability: Let B1, B2, . . . , Bk be a collection of mutually exclusive and

exhaustive events, then for any event A,

P(A) = X

k

i=1

P(A ∩ Bi)

which implies

P(T−|D+) = 1 − π

P(T+|D−) = 1 − θ

in an instance where the test method has 2-states (positive and negative) and

P(T−|D+) = 1 − π −

P(T+|D−) = 1 − θ − φ

in an instance where the test method has 3-states (positive, negative and suspect).

Conditional Independence:

Two test methods are conditionally independent given disease status if and only if

P(T1, T2|D) = P(T1|D)P(T2|D)

3 Estimating Sn and Sp

3.1 Simulation Method

Identify one or more reference tests. Provide information on the performance characteristics

(sensitivity and specificity) of each reference test. Provide information on the prevalence

of each of the sampled populations. Estimate the Sn and Sp of the experimental kit by the

simulation method. If an infallible reference test is available, follow the method in Section

3.2.

STATWI0002.04 page 3 of 9
3 ESTIMATING SN AND SP 4

3.1.1 Sn and Sp

 The firm provides reasonable assumed values for Sn (π) and Sp (θ) for each

reference test used based on the available information.

 The firm provides reasonable assumed values for δ and γ for each 3-state reference

test used based on available information. These express the probability of a

suspect result as a fraction of the non-correct test results. See Section 2 Notation.

 When information exists for the distribution of Sn (π) and Sp (θ) it is typically

crude or limited. Unless specific data are available about the form and nature

of the distribution, rely on the software package default approach of simulating

from a beta distribution with fixed variance.

3.1.2 Prevalence

 Provide a known or reasonable assumed value of the prevalence for each population sampled. Please refer to VSM 800.73 for further details regarding prevalence

estimates.

 When information exists for the distribution of prevalence it is typically crude

or limited. Unless specific data are available about the form and nature of the

distribution, rely on the software package default approach of simulating from a

beta distribution with fixed variance.

3.1.3 For each simulation cycle, the Sn (π) and Sp (θ) for the experimental test are estimated as the values that minimize the sum of the squared deviations between the

observed and expected counts. For a 2-state experimental test,

min

π1, θ1

X

K

k=1

X

L

l=1

(nk,l − nˆk,l)

2

For a 3-state experimental test,

min

π1, θ1,δ1,γ1

X

K

k=1

X

L

l=1

(nk,l − nˆk,l)

2

3.1.4 Using the above equations, interval estimates are obtained from the set of optimized

values for each set of simulations by the highest density method. A software package1

in the R language may be found online.

3.2 Estimating Sn and Sp with an Infallible Reference test

An infallible reference test is one that can detect either the presence or absence of a disease

without error. There are very few instances in which an infallible reference test exists for

1https://github.com/ABS-dev/DiagTestKit/blob/master/README.md

STATWI0002.04 page 4 of 9
A APPENDIX 5

determining the positive or negative status of a sample. If an infallible reference test is

available for one or the other disease status, then the corresonding parameter estimate (Sn

or Sp) is a simple fraction.

For example, for a 2-state experimental kit the results of the experimental kit and the true

disease status can be cross-classified as follows:

Disease

Test Positive Negative

Positive a b

Negative c d

Then Sn and Sp are determined as follows:

Sn = P(T+|D+) = a

a + c

Sp = P(T−|D−) = d

b + d

Use the Clopper-Pearson (Clopper & Pearson (1934)) method for obtaining interval estimates.

A Appendix

A.1 A Single 2-State Reference Test, One population, 2-State

Experimental Test

When a single 2-state reference test (T2) is used to estimate the sensitivity and specificity

of a 2-state experimental test (T1), there are 4 unique test combinations (i.e. K = 4). If

only one population is sampled, there are a total of 4 expected counts.

Let

 M1 = (T1−, T2−)

 M2 = (T1−, T2+)

 M3 = (T1+, T2−)

 M4 = (T1+, T2+)

STATWI0002.04 page 5 of 9
A APPENDIX 6

Therefore,

P1(M1) = P(T1 − |D+)P(T2 − |D+)P1(D+) + P(T1 − |D − )P(T2 − |D − )P1(D − )

= (1 − π1)(1 − π2)p1 + θ1θ2(1 − p1)

P1(M2) = P(T1 − |D+)P(T2 + |D+)P1(D+) + P(T1 − |D − )P(T2 + |D − )P1(D − )

= (1 − π1)π2p1 + θ1(1 − θ2)(1 − p1)

P1(M3) = P(T1 + |D+)P(T2 − |D+)P1(D+) + P(T1 + |D − )P(T2 − |D − )P1(D − )

= π1(1 − π2)p1 + (1 − θ1)θ2(1 − p1)

P1(M4) = P(T1 + |D+)P(T2 + |D+)P1(D+) + P(T1 + |D − )P(T2 + |D − )P1(D − )

= π1π2p1 + (1 − θ1)(1 − θ2)(1 − p1)

The expected counts are

 nˆ1,1 = P1(M1)N1

 nˆ2,1 = P1(M2)N1

 nˆ3,1 = P1(M3)N1

 nˆ4,1 = P1(M4)N1

Sensitivity and specificity of the experimental kit are estimated as the values that minimize

the sum of the squared residuals. Specificially,

min

π1, θ1

h

(n1,1 − nˆ1,1)

2 + (n2,1 − nˆ2,1)2 + (n3,1 − nˆ3,1)2 + (n4,1 − nˆ4,1)2

i

This can easily be extended to multiple populations.

A.2 Two, 2-State Reference Tests, One population, 3-State Experimental Test

When two, 2-state reference tests are used to estimate the performance characteristics of a

3-state experimental test, there are 12 unique test combinations (i.e. K=12). If only one

population is sampled, there are a total of 12 expected counts.

Let

 M1 = (T1−, T2−, T3−)

 M2 = (T1−, T2−, T3+)

STATWI0002.04 page 6 of 9
A APPENDIX 7

 M3 = (T1−, T2+, T3−)

 M4 = (T1−, T2+, T3+)

 M5 = (T1+, T2−, T3−)

 M6 = (T1+, T2−, T3+)

 M7 = (T1+, T2+, T3−)

 M8 = (T1+, T2+, T3+)

 M9 = (T1?, T2−, T3−)

 M10 = (T1?, T2−, T3+)

 M11 = (T1?, T2+, T3−)

 M12 = (T1?, T2+, T3+)

Therefore,

P1(M1) = P(T1−|D+)P(T2−|D+)P(T3−|D+)P(D+) + P(T1−|D−)P(T2−|D−)P(T3−|D−)P(D−)

= (1 − π1)(1 − δ1)(1 − π2)(1 − π3)p1 + θ1θ2θ3(1 − p1)

P1(M2) = P(T1−|D+)P(T2−|D+)P(T3+|D+)P(D+) + P(T1−|D−)P(T2−|D−)P(T3+|D−)P(D−)

= (1 − π1)(1 − δ1)(1 − π2)π3p1 + θ1θ2(1 − θ3)(1 − p1)

P1(M3) = P(T1−|D+)P(T2+|D+)P(T3−|D+)P(D+) + P(T1−|D−)P(T2+|D−)P(T3−|D−)P(D−)

= (1 − π1)(1 − δ1)π2(1 − π3)p1 + θ1(1 − θ2)θ3(1 − p1)

P1(M4) = P(T1−|D+)P(T2+|D+)P(T3+|D+)P(D+) + P(T1−|D−)P(T2+|D−)P(T3+|D−)P(D−)

= (1 − π1)(1 − δ1)π2π3p1 + θ1(1 − θ2)(1 − θ3)(1 − p1)

P1(M5) = P(T1+|D+)P(T2−|D+)P(T3−|D+)P(D+) + P(T1+|D−)P(T2−|D−)P(T3−|D−)P(D−)

= π1(1 − π2)(1 − π3)p1 + (1 − θ1)(1 − γ1)θ2θ3(1 − p1)

P1(M6) = P(T1+|D+)P(T2−|D+)P(T3+|D+)P(D+) + P(T1+|D−)P(T2−|D−)P(T3+|D−)P(D−)

= π1(1 − π2)π3p1 + (1 − θ1)(1 − γ1)θ2(1 − θ3)(1 − p1)

P1(M7) = P(T1+|D+)P(T2+|D+)P(T3−|D+)P(D+) + P(T1+|D−)P(T2+|D−)P(T3−|D−)P(D−)

= π1π2(1 − π3)p1 + (1 − θ1)(1 − γ1)(1 − θ2)θ3(1 − p1)

STATWI0002.04 page 7 of 9
A APPENDIX 8

P1(M8) = P(T1+|D+)P(T2+|D+)P(T3+|D+)P(D+) + P(T1+|D−)P(T2+|D−)P(T3+|D−)P(D−)

= π1π2π3p1 + (1 − θ1)(1 − γ1)(1 − θ2)(1 − θ3)(1 − p1)

P1(M9) = P(T1?|D+)P(T2−|D+)P(T3−|D+)P(D+) + P(T1?|D−)P(T2−|D−)P(T3−|D−)P(D−)

= δ1(1 − π1)(1 − π2)(1 − π3)p1 + γ1(1 − θ1)θ2θ3(1 − p1)

P1(M10) = P(T1?|D+)P(T2−|D+)P(T3+|D+)P(D+) + P(T1?|D−)P(T2−|D−)P(T3+|D−)P(D−)

= δ1(1 − π1)(1 − π2)π3p1 + γ1(1 − θ1)θ2(1 − θ3)(1 − p1)

P1(M11) = P(T1?|D+)P(T2+|D+)P(T3−|D+)P(D+) + P(T1?|D−)P(T2+|D−)P(T3−|D−)P(D−)

= δ1(1 − π1)π2(1 − π3)p1 + γ1(1 − θ1)(1 − θ2)θ3(1 − p1)

P1(M12) = P(T1?|D+)P(T2+|D+)P(T3+|D+)P(D+) + P(T1?|D−)P(T2+|D−)P(T3+|D−)P(D−)

= δ1(1 − π1)π2π3p1 + γ1(1 − θ1)(1 − θ2)(1 − θ3)(1 − p1)

The expected counts are

 nˆ1,1 = P1(M1)N1

 nˆ2,1 = P1(M2)N1

 nˆ3,1 = P1(M3)N1

 nˆ4,1 = P1(M4)N1

 nˆ5,1 = P1(M5)N1

 nˆ6,1 = P1(M6)N1

 nˆ7,1 = P1(M7)N1

 nˆ8,1 = P1(M8)N1

 nˆ9,1 = P1(M9)N1

 nˆ10,1 = P1(M10)N1

 nˆ11,1 = P1(M11)N1

 nˆ12,1 = P1(M12)N1

Sensitivity and specificity of the experimental kit are estimated as the values that minimize

the sum of the squared residuals. Specificially,

min

π1,θ1,δ1,γ1

X

K

k=1

(nk,1 − nˆk,1)

2

.

STATWI0002.04 page 8 of 9
REFERENCES 9

References

Clopper, D., & Pearson, E. (1934). The use of confidence or fiducial limits illustrated in

the case of the binomial. Biometrika, 26 , 404–413.

STATWI0002.04 page 9 of 9