Background
==========

The manner in which accuracy of clinical tests is mathematically summarised in the biomedical literature has important implications for clinicians. Appropriate accuracy measures would be expected to sensibly convey the meaning of the study results with scientifically robust statistics without exaggerating or underestimating the clinical significance of the findings. Lack of use of appropriate measures may lead authors of primary accuracy studies to draw biased conclusions.\[[@B1]\] In systematic reviews of test accuracy literature, there are many ways of synthesising results from several studies, not all of which are considered to be scientifically robust. For example, measures such as sensitivity and specificity commonly used in primary studies are not considered suitable for pooling separately in meta-analysis.\[[@B2]\] Variations in reporting of summary accuracy and use of inappropriate summary statistics may increase the risk of misinterpretation of clinical value of tests.

A recent study evaluated a small sample of meta-analytical reviews of screening tests to demonstrate the variety of approaches used to quantitatively summarise accuracy results.\[[@B3]\] This study confined itself to a limited Medline search. It exclusively examined meta-analytical studies so reviews not using quantitative synthesis were excluded. It did not look at accuracy measures used to report results of primary studies separately from those used for meta-analyses. In order to address these issues, we undertook a comprehensive search to survey systematic reviews (with and without meta-analysis) of test accuracy literature to assess the measures used for reporting results of included primary studies as well as their quantitative synthesis.

Methods
=======

We manually searched for relevant reviews in the *Database of Abstracts of Reviews of Effectiveness* (DARE).\[[@B4]\] In order to limit the impact of human error inherent in manual searching, we complemented it with electronic searching. DARE was searched electronically with word variants of relevant terms (diagnostic, screening, test, likelihood ratio, sensitivity, specificity, positive and negative predictive value) combined using OR. From 1994 to 2000 DARE\[[@B4]\] has identified 1897 reviews of different types by regular electronic searching of several bibliographic databases, hand searching of key major medical journals, and by scanning grey literature (search strategy and selection criteria can be found at <>). The structured abstracts of these reviews were screened independently by the authors to identity systematic reviews of test accuracy. The full texts were obtained of those abstracts judged to be potentially relevant. Reviews addressing test development and diagnostic effectiveness or cost effectiveness were excluded. Any disagreements about review selection were resolved by consensus.

Information from each of the selected reviews was extracted for the measures of test accuracy used to report the results of the primary studies included in the review. If a meta-analysis was conducted, information was also extracted for the summary accuracy measures. The various accuracy measures are shown in Table [1](#T1){ref-type="table"}. We sought the following in the primary studies: sensitivity or specificity, predictive values, likelihood ratios and diagnostic odds ratio. For meta-analysis, we sought the summary measures pooling the above results and summary receiver operating characteristics (ROC) plot or values. All extracted data were double-checked. We divided the reviews into two groups arbitrarily according to time of publication; one group covering the period 1994--97 (50 reviews) and another covering 1998--2000 (40 reviews). This allowed us to assess whether there were any significant differences in measures being used to report test accuracy results among reviews published earlier and those published later. As the approaches to summarising results are not mutually exclusive, we evaluated and reported the most commonly used measures and their most common combinations. We used chi-squared statistical test for comparison of differences between proportions.

::: {#T1 .table-wrap}
::: {.caption}
######

Measures of accuracy of dichotomous test results
:::

  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  **Measures for primary studies**
   **Sensitivity (true positive rate)**
   The proportion of people with disease who are correctly identified as such.
   **Specificity (true negative rate)**
   The proportion of people with disease who are correctly identified as such.
   **Positive predictive value**
   The proportions of test positive people who truly have disease.
   **Negative predictive value**
   The proportions of test negative people who truly do not have disease.
   **Likelihood ratios (LR)**
   The ratio of the probability of a positive (or negative) test result in the patients with disease to the probability of the same test result in the patients without the disease.
   **Diagnostic odds ratio**
   The ratio of the odds of a positive test result in patients with disease compared to the odds of the same test result in patients without disease.
  **Measures for meta-analysis**
   **Summary sensitivity, Specificity, predictive values, likelihood ratios, and diagnostic odds ratio**
   Pooling of the above accuracy measures obtained from multiple primary studies (usually averaged and weighted according to size of individual studies).
   **Summary receiver operating characteristics curve (ROC)**
   A method of summarising the performance of a test as found in multiple primary studies, which takes into account the relationship between sensitivity and specificity.
  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
:::

Results
=======

Of the abstracts available in DARE, 150 were considered to be potentially relevant. Excluding reviews that addressed test development and diagnostic effectiveness or cost, 90 reviews of test accuracy were left for inclusion in our survey. There were 45 reviews of dichotomous test results, 42 reviews of continuous results dichotomised by the original authors, and 3 reviews that contained both result types. Meta-analysis was used in 60/90 (67 %) reviews, 50 in 1994--97 and 40 in 1998--2000. (See [Additional File](#S1){ref-type="supplementary-material"}: BMC_IncludedRefList_04032002 for a complete listing of the 90 reviews included in our study).

As shown in Table [2](#T2){ref-type="table"}, sensitivity or specificity was used for reporting the results of primary studies in 65/90 (72%) reviews, predictive values in 26/90 (28%), and likelihood ratios in 20/90 (22%). For meta-analysis, independently pooled sensitivity or specificity was used in 35/60 (58%) reviews, pooled predictive values in 11/60 (18%), pooled likelihood ratios in 13/60 (22%), and pooled diagnostic odds ratio in 5/60 (8%). Summary ROC was used in 44/60 (73%) of the meta-analyses. There were no significant differences between reviews published earlier and those published later as shown in Table [2](#T2){ref-type="table"}.

::: {#T2 .table-wrap}
::: {.caption}
######

Measures of test accuracy reported in review of diagnostic literature (1994--2000)
:::

  ------------------------------------------------- ------------------ -------------------------------------- ------------ ------------------------------------- -------------------
                                                    **Time periods**

  **Measures of test accuracy**                     **1994--1997**     **1998--2000**

                                                    ***n***            ***% (95% confidence interval)*^+^**   ***n***      ***% (95% confidence interval*^+^**   **p-value^\*\*^**

  Included primary studies                          **50^\*^**                                                **40^\*^**
  Sensitivity or specificity                        35                 70 (55--82)                            30^a^        75 (59--87)                           0.77
  Predictive values                                 13^b^              26 (15--40)                            13^c^        33 (19--49)                           0.66
  Sensitivity, specificity and predictive values    12                 24 (13--38)                            12           30 (17--47)                           0.69
  Likelihood ratios                                 9                  18(9--31)                              11           28 (15--44)                           0.41
  Diagnostic odds ratios                            0                  0 (0--7)                               0            (0--9)                                \-
  **Meta-analysis**                                 **38^\*^**         76(62--87)                             **22^\*^**   55 (38--71)                           0.16
  Independently pooled sensitivity or specificity   22                 58 (41--74)                            13^d^        62 (36--79)                           0.86
  Pooled predictive values                          9^e^               24(11--40)                             2^f^         10(1--29)                             0.29
  Pooled likelihood ratios                          5                  13 (4--28)                             8            38 (17--59)                           0.08
  Pooled diagnostic odds ratios                     5                  13 (4--28)                             0            0(0--15)                              0.20
  Summary ROC plot or values                        23                 61 (43--76)                            11           52 (28--72)                           0.60
  ------------------------------------------------- ------------------ -------------------------------------- ------------ ------------------------------------- -------------------

^\*^ numbers do not add up to totals because some reviews used more than one measures of accuracy; ^\*\*^ chi sq. test with Yates\' correction; ^+^ Exact (Clopper-Pearson) 95% confidence interval a includes study that only reported either sensitivity or specificity; b,c. includes study that only reported either positive or negative predictive value; d includes meta-analyses that only reported either pooled sensitivity or specificity; e,f. includes meta-analyses that only reported either pooled positive or negative predictive values
:::

Discussion
==========

Our study showed that sensitivity and specificity remain in frequent use, both for primary studies and for meta-analyses over the time period surveyed. Sensitivity and specificity are considered inappropriate for meta-analyses, as they do not behave independently when they are pooled from various primary studies to generate separate averages.\[[@B2]\] In our survey, separate pooling of sensitivities or specificity was used frequently in meta-analyses where summary ROC would have been more appropriate. \[[@B5]-[@B7]\].

Our findings about reporting of summary accuracy measures in meta-analyses are different to those reported previously.\[[@B3]\] We found a higher rate of use of summary ROC, though use of independent summaries of sensitivity, specificity and predictive values were similar. These differences may be due to differences in searching strategies (databases and time frames) and selection criteria. Our search was more recent and comprehensive, using DARE\[[@B4]\], which has covered seven different databases (Medline, CINAHL, BIOSIS, Allied and Alternative Medicine, ERIC, Current Contents clinical medicine and PsycLIT), and hand-searched 68 peer-reviewed journals and publications from 33 health technology assessment centres around the world since February 1994. Moreover, as we did not restrict our selection to meta-analytical reviews only, we were able to examine reviews summarising accuracy results of primary studies without quantitative synthesis, which constituted 33% (30/90) of our sample. Therefore, compared to the previous publication on this topic,\[[@B3]\] our survey provided a broader and more up-to-date overview of the state of reporting of accuracy measure in test accuracy reviews.

Conclusions
===========

The use of inappropriate accuracy measures has the potential to bias judgement about the value of tests. Of the various approaches to reporting accuracy of dichotomous test results, likelihood ratios are considered to be more clinically powerful than sensitivities or specificities.\[[@B8]\] Crucially, it has been empirically shown that authors of primary studies may overstate the value of tests in the absence of likelihood ratios.\[[@B1]\] There is also evidence that readers themselves may misinterpret test accuracy measures following publication.\[[@B9]\] It is conceivable that the problem of inconsistent usage of test accuracy measures in published reviews, as found in our survey, may contribute to misinterpretation by clinical readership. The reason for variation in reported accuracy measures may, in part, be attributed to a lack of consensus regarding the best ways to summarise test results. It is worth noting that despite authoritative publications about appropriate summary accuracy measures in the past,\[[@B5],[@B7],[@B10]\] (we have only quoted a few references) inconsistent and inappropriate use of summary measures has remained prevalent in the period 1994--2000. Our paper highlights the need for consensus to support change in this field of research.

Competing interest
==================

None declared

Pre-publication history
=======================

The pre-publication history for this paper can be accessed here:

<>

Supplementary Material
======================

::: {.caption}
###### Additional file

Reference list of the 90 reviews included in the survey
:::

::: {.caption}
######

BMC_IncludedRefList_04032002
:::

Acknowledgement
===============

We wished to acknowledge J. Dinnes, J. Glanville and F. Song for their contribution in the searches and selection of the reviews for the survey.

Funding
=======

RCOG Wellbeing grant number K2/00


Background
==========

Most current methods of cancer early detection, such as mammography or cervical cytology, are based on anatomic changes in tissues or morphologic changes in cells. Recently, various molecular markers, such as protein or genetic changes have been proposed for cancer early detection \[[@B1]-[@B4]\]. This has spurred many investigators with long-term cohort studies to serially collect and store blood or tissue specimens. The aim is to later perform a nested-case control study, where specimens from subjects with a particular type of cancer (cases) and specimens from a random sample of subjects without the cancer (controls) are tested for various molecular markers. Sometime this sort of study is called a retrospective longitudinal study \[[@B6]\] although retrospective longitudinal data could arise in other ways, as well. Unlike cross-sectional study designs, the markers are measured on specimens collected well before the onset of clinical disease in cases. This avoids the potential confounding effect of the target disease on the marker.

For example, in the ATBC (alpha-tocopherol, beta-carotene) \[[@B7]\] and CARET \[[@B8]\] studies, subjects were randomized to placebo or drug to in a long-term study to determine the effect of the drug on lung cancer mortality. During the course of the trial serum was serially collected and stored in a biorepository. In a subsequent nested case-control study, stored serum samples from all cases of prostate cancer and a random sample of controls were tested for prostate-specific antigen (PSA).

Importantly the nested case-control study of early detection biomarkers may be distinct from the original long-term study from which serum were collected. It is designed to answer a different question, it typically studies subjects with a different disease, and it often ignores the intervention in the original long-term study.

Methods
=======

We had three considerations in formulating appropriate guidelines. First we wanted to link the analysis to the goal of study, namely, to help decide on further study of the biomarker as a trigger for early intervention. Second we wanted to minimize possible biases in the selection of cases and the controls and in the investigation of many markers. Third, we wanted to extract as much information as possible relevant to the evaluation.

Results
=======

We offer the following guidelines for the design and analysis of nested case-control studies of early detection cancer biomarkers.

1\. For the clearest interpretation, statistics for binary markers should be based on true and false positive rates or predictive values based on the true prevalence -- not odds ratios, relative risks, or predictive values based on the prevalence in the study

A promising marker should have a high degree of accuracy in discriminating between subjects who are likely to get cancer from those who are not. For a binary marker, which is either positive or negative, the basic measures of this type of accuracy are the true positive and false positive rates. Consider the hypothetical data in the Table [1](#T1){ref-type="table"} from a nested case-control study of early detection biomarkers. The true positive rate (TPR), or the test sensitivity, is the probability the marker is positive given cancer. The false positive rate (FPR), or 1 -- specificity, is the probability the marker is positive given no cancer. In Table [1](#T1){ref-type="table"}, TPR is estimated by 80/100 = .80, and FPR is estimated by 10/1000 = 01. For a perfect test, TPR = 1 and the FPR = 0. There is always a tradeoff between the TPR and FPR, so it is meaningless to assess one without assessing the other. For example, one could make the TPR equal to 1 simply by classifying every subject as positive for cancer. But this would be a poor classification rule because the FPR would also be 1.

::: {#T1 .table-wrap}
::: {.caption}
######

Hypothetical data for a binary marker from a nested case-control study
:::

              Marker -   Marker +   Total
  ----------- ---------- ---------- -------
  No cancer   990        10         1000
  Cancer      20         80         100

True positive rate (TPR) = 80/100 = .80 = sensitivity False positive rate (FPR) = 10/1000 = .01 = 1 -- specificity
:::

Guidelines for FPR and TPR

Because the underlying prevalence of cancer is so low in average risk populations, for acceptable cancer screening of asymptomatic people, the FPR should be very small \[[@B9]\]. As a starting point we recommend basing guidelines for FPR and TPR on the FPR and TPR for mammography, which, as discussed below, is FPR=.01 and TPR=.80. A mammogram is analogous to a biomarker test for cancer but there is extra information from various studies that makes it useful for setting guidelines.

One reason for using FPR and TPR from mammography as a standard is that, unlike biomarker measurements in nested case-control studies, in mammography studies there is a biopsy at the time of a positive test. This biopsy is the gold standard for determining cancer status in subjects with positive mammograms and is used for computing TPR in a way not possible with biomarkers in nested case-control studies. The TPR for mammography is the probability of a positive biopsy as a direct result of mammography in women with cancer and is estimated via mathematical models or data collected after following subjects not biopsied. As discussed in Baker and Pinsky \[[@B10]\], estimates ranged from .74 to 1.00 with .80 a conservative value.

Importantly the estimated TPR for mammography is not likely to be affected by overdiagnosis, which means that some screening-detected cancers would never have caused medical problems during the patient\'s life \[[@B11]\]. This would make the biomarker appear more promising than actually the case. Results from the HIP screening trial of mammography and clinical self-examination \[[@B12]\] suggest that if there were overdiagnosis with mammography, it would be relatively small. At the time of the last breast screening in the HIP trial, there were more cancers in the group randomized to screening than in the controls. But with further follow-up the number of cancers in the control group eventually equaled the number in the screened group, which would not have occurred if there were substantial overdiagnosis.

A second reason for using FPR and TPR from mammography is that, based on various randomized trials with cancer mortality endpoint, mammography is generally considered an acceptable screening modality. The implication is that a similar FPR and TPR for a biomarker would lead to an acceptable screening modality. For a particular biomarker, these target values of FPR and TPR from mammography may need modification depending on various factors. One factor is the invasiveness of a follow-up procedure to investigate a positive test (e.g. needle biopsy of the prostate to investigate an abnormal PSA versus laparotomy to investigate an abnormal CA125). The more invasive the follow-up procedure, the lower the FPR must be to gain acceptance in practice. A second factor is additional work-up prior to a biopsy. If a positive biomarker is unlikely to trigger additional diagnostic work-ups prior to biopsy, a higher FPR might be acceptable.

One caveat when using FPR from mammography is to be careful as to its definition. The restricted definition is the probability of a positive biopsy as a direct result of mammography in women without cancer. The less restricted definition is the probability of a suspicious mammogram warranting additional diagnostic follow-up of any type in women without cancer. Typically nested-case control studies of early detection biomarkers do not provide information on additional diagnostic follow-up. Therefore they cannot be used to estimate a less restricted FPR involving diagnostic follow-up. However, because nested case-control biomarker studies provide data on cancer diagnosis, they can be used to estimate a more restricted FPR based on unnecessary biopsies. Therefore the target FPR is based on the more restricted definition of FPR in mammography. For mammography the more restricted FPR is estimated by the fraction of women screened by mammography who received a biopsy in which no cancer was detected. As discussed in Baker and Pinsky \[[@B10]\], estimates of FPR from three studies ranged from .005 to .013 with a middle value of around .010.

Inappropriateness of odds ratio and relative risk

When evaluating binary early detection markers, it is inappropriate to report an odds ratio or relative risk, as is common in epidemiology or clinical trials. Because an odds ratio or relative risk is a single number, it does not capture the tradeoff between correctly classifying cancer and incorrectly classifying non-cancers. Also the odds ratio or relative risk can lead to an overoptimistic impression of the performance of an early detection test if the interpretation is based on experience in epidemiology or clinical trials. In the latter settings an odds ratio of 3 is often considered large. Much larger odds ratio are needed from early detection tests for useful application in the screening setting \[[@B13]\]. For example, for the target values of FPR = .01 and TPR = .80, the odds ratio equals (TPR × (1-FPR)/ ((1-TPR) × FPR) = 396, as in Table [1](#T1){ref-type="table"}.

Appropriate computation of predictive values

It is sometimes useful to use the FPR and TPR to compute the predictive value negative (PVN), the probability of no disease if the marker is negative, and the predictive value positive (PVP), the probability of disease if the marker is positive. For cancer screening, it is the PVP that is most important to the physician in clinical decision-making. Because the likelihood of any individual cancer type in an asymptomatic person is nearly always very low, a negative early detection test usually adds little information to the clinical impression. The computation of the PVN and PVP depend on the prevalence of cancer as well as on the FPR and the TPR, as shown below,

PVP = (TPR × prevalence) / (TPR × prevalence + FPR (1-prevalence))

PVN = (1-FPR) × (1-prevalence) / ((1-TPR) × prevalence + (1-FPR) (1-prevalence))

When the prevalence is small, as with cancer screening, a small absolute change in FPR, but not TPR, can have a large impact on the PVP. It is erroneous to directly compute the prevalence from data in the nested case-control study, as in Table [1](#T1){ref-type="table"}, because the ratio of cases to controls in the study is not the same as in the population. For example suppose FPR = .01, TPR = .80, and prevalence = .003. We obtain,

PVP = = (.8 × .003) /(.8 × .003 + .01 .997)= .19

PVN = (.99 × .997) / (.2 × .003 + .99 × .997) = .999

If we had incorrectly substituted the apparent prevalence in Table [1](#T1){ref-type="table"} of 100/1100 = .091, we would have incorrectly computed PVP = .89 and PVN = .980. For this reason, calculation and reporting of PVP and PVN using only data from a nested case-control study is not useful or appropriate.

Extension to ordered categories via ROC curves

Many markers for the early detection of cancer can be reported as ordered categories. Some markers, such as sputum cytology, inherently involve ordered categories, such as no evidence of cancer, slight atypia, moderate atypia, severe atypia, and frank cancer. Other markers, such as PSA, involve a continuous measure for which higher values indicate a greater probability of cancer. Dividing these continuous measures into ranges (either based on predetermined values or percentiles) gives ordered categories.

With ordered categories, the statistics should still be based on FPR and TPR. Each ordered category is a possible cutpoint, where values equal to or higher than the cutpoint are called positive and values lower than the cutpoint are called negative. For each cutpoint, one can compute FPR and TPR (Table [2](#T2){ref-type="table"}) and generate a receiver-operating characteristic (ROC) curve, which is a plot of TPR versus FPR for the various pairs \[[@B14]\]. (See Figure [1](#F1){ref-type="fig"}). The higher and farther left the points on the ROC curve the better the test performance. As mentioned previously, acceptable cancer screening requires very small false positive rates. Therefore, for evaluating cancer biomarkers, we are only interested in the leftmost sliver of the ROC curve in Figure [1](#F1){ref-type="fig"}.

![ROC curve derived from hypothetical data in Table [2](#T2){ref-type="table"}. Because acceptable cancer screening requires a very small false positive rate (FPR) and a reasonable true positive rate (TPR), only the four points farthest to the left are relevant for biomarker evaluation.](1471-2288-2-4-1){#F1}

::: {#T2 .table-wrap}
::: {.caption}
######

Hypothetical Data for an Ordered Marker From a Nested Case-Control Design
:::

              1     2    3    4    5
  ----------- ----- ---- ---- ---- ---- ------
  No cancer   960   20   10   8    2    1000
  Cancer      0     10   10   20   60   100

For cutpoint 4, the true positive rate is (20+60)/100 = .80 For cutpoint 4, the false positive rate is (8 + 2)/1000 = .01
:::

2\. To avoid overdiagnosis bias, cases should be diagnosed as a result of symptoms rather than on screening

For the TPR in the biomarker study to reflect the true TPR, cases should be diagnosed as a result of symptoms rather than on screening. For example in the study of PSA in the ATBC trial \[[@B7]\], cases were subjects diagnosed with prostate cancer as a result of symptoms. If the prostate cancer cases were detected as the result of screening, say with ultrasound, the TPR could be artificially elevated if there were overdiagnosis, as previously discussed.

3\. To minimized selection bias, the spectrum of control conditions should be the same in study and target screening populations

For the FPR in the nested case-control study to reflect the true FPR in the target population, the spectrum of control conditions should be the same as in the target population. By control conditions, we mean characteristics of the population, such as the presence of other diseases or certain known risk factors that could elevate the false positive rate.

The spectrum of conditions could differ considerably if the retrospective biomarker study were embedded in a randomized trial with strict eligibility requirements. For example, consider a biomarker for the early detection of lung cancer where the data comes from a biorepository arising from a randomized trial of healthy subjects. It would be inappropriate to apply the results to a population with a high prevalence of chronic obstructive lung disease, bronchitis, or viral pnuemonitis because these conditions could increase the number of positive readings in subjects without lung cancer. Because FPR is very small for screening to be acceptable, this spectrum bias could have important consequences in a clinical application.

It would not always be possible to identify all relevant control conditions, but to the extent possible, the control conditions should be similar in both populations.

4\. To extract additional information, criteria for a positive test should be based on combinations of individual markers and changes in marker levels over time

Data from multiple markers present an opportunity to extract potentially valuable information not available by analyzing markers separately. Consider the hypothetical data in Table [3](#T3){ref-type="table"}. The left side corresponds to subjects without cancer, and the right corresponds to subjects with cancer. In our idealized circumstance, markers A and B are independent for subjects without cancer and are perfectly correlated for subjects with cancer. The indicated region for A = 2 or A = 3 and B = 2 or B = 3 designates a positive test that has FPR = .04 and the TPR = .80. Suppose that marker A and marker B were evaluated separately. The indicated region corresponding to A only, namely A = 2 or A = 3 and the indicated region corresponding to B only, namely B = 2 and B = 3, each designates a positive test that has FPR=. 20 and TPR=. 80. Thus, in this particular example, the combination of markers leads to a much better test than the separate markers, with a smaller FPR (.04 versus .20) for the same TPR of .80. One could create a similar example with any number of marker levels. For an ideal test in which FPR = 0 and TPR = 1, the region would encompass all subjects with cancer but no subjects without cancer.

::: {#T3 .table-wrap}
::: {.caption}
######

Hypothetical data for two ordered markers, A and B
:::

               Subjects without cancer (N = 1000)   Subjects with cancer (N = 100)
  ------------ ------------------------------------ -------------------------------- ------------ ------------ ---- ----------- ----------- -----------
  B = 1        640                                  80                               80           800          20   0           0           20
  B = 2        80                                   **10^a^**                        **10^a^**    **100^e^**   0    **20^b^**   **0^b^**    **20^f^**
  B = 3        80                                   **10^a^**                        **10^a^**    **100^e^**   0    **0^b^**    **60^b^**   **60^f^**
  **A only**   800                                  **100^c^**                       **100^c^**                20   **20^d^**   **60^d^**

**For A and B combined;**^a^False positive rate (FPR) of indicated region (10+10+10+10)/1000=. 04; ^b^True positive rate (TPR) of indicated region = (20 +0+0+60)/100=. 80 **For A only**; ^c^False positive rate (FPR) of indicated region =(100+100)/1000=. 20; ^d^True positive rate (TPR) of indicated region = (20+60)/100=. 80 **For B only**; ^e^False positive rate (FPR) of indicated region =(100+100)/1000= .20; ^f^True positive rate (TPR) of indicated region = (20+60)/100=.80
:::

A real application of how marker combinations provide extra information comes from a recent study by Mok et al \[[@B15]\] on CA125 and Prostasin as markers for ovarian cancer. Although the data are not from a nested case-control study and have not been validated in subsequent studies, they are useful for illustration. Based on Figure 5 of Mok et al \[[@B15]\] with approximate values for the outpoints, we investigated regions with FPR=.02. The combination for a positive test of CA 125 greater than or equal to 40 U/ml and Prostasin greater than or equal to 5 μg/ml had FPR = .02 and TPR = .92. Evaluating CA 125 alone, the criterion for a positive test of CA 125 greater than 80 U/ml had FPR=.02 and TPR =.78. Evaluating Prostasin alone, the criterion for a positive test of Prostasin greater than 15 μg/ml had FPR=. 02 and TPR=. 32. Thus, in this real example, the combination of markers led to a better test than the separate markers, with a larger TPR (.92 versus .78 or .32) for the same FPR of .02.

To more generally compare performance of a combination of markers with a single marker, we need to compare ROC curves. Creating an ROC curve from a combination of markers is different from creating an ROC curve from a single marker. With a combination of markers, the number of possible regions, as in Table [3](#T3){ref-type="table"} a, for calling a marker positive is extremely large. Some choices of regions correspond to AND rules, for example A\>1 and B\>1, as in Table [3](#T3){ref-type="table"} a. Some choices of regions correspond to OR rules, for example A\>2 or B\>2. Other choices are also possible, but for biological reasons, one would usually require all regions to be contiguous. A plot of the FPR and TPR for each region would lead to a cloud of points, rather than the smooth curve in Figure [1](#F1){ref-type="fig"}. To create the best ROC curve, one should select those points that are highest and farthest to the left, which is generally all that would need to be presented and only for small false positive rates. For complicated situations, Baker \[[@B16]\] proposed an algorithm to select the regions creating the best ROC curve without the need to enumerate all the regions. Mathematically, in any sample of data, the best ROC curve for a combination of markers must be as good or better than the ROC curve for any of the markers evaluated separately. The reason is that the set of possible regions for calling a combination of markers positive includes as a special case the regions for calling any single marker positive.

Alternative approaches that do not directly optimize the ROC curve include linear logistic regression or linear discriminant analysis \[[@B17]\], which choose regions based on linear combinations of the markers, and neural networks \[[@B18]\], which choose regions in a very complicated nonlinear manner. Due to the potential for overfitting (to follow), it is not possible to make a blanket statement as to which approach for choosing regions is best. If one takes the set of regions for calling a combination positive that gives a good ROC curve in a random sample of data, it may give a poor ROC curve in another random sample of the data, simply due to selecting chance patterns in the first sample. As discussed in the section on overfitting, this motivates splitting the data into two random samples, training and test, and using the regions from the training sample to compute the definitive ROC curve based on data in the test sample.

Changes in marker values over time also provide potentially valuable information not available when examining markers at a single time point. With marker measurements at two different times per subject (and approximately the same interval between times), a common summary measure is the slope. If investigators believe that both slope and baseline level predict cancer, the combination can be evaluated using the previously discussed methods for evaluating multiple markers, namely, treating baseline level and slope as two \"separate\" markers. With measurements at more than two times per subject, investigators may identify a more complicated feature, such as whether or not there is a sudden increase in marker levels \[[@B19]\].

When biomarker measurements occur at regular time intervals (and allowing different numbers of measurements for each subject), one can estimate TPR and FPR by using a first order Markov chain in reverse time, as described by Baker and Tockman \[[@B20]\] for the analysis of precancerous lesions for lung cancer.

5\. To avoid overfitting, the criteria for a positive marker combination developed in a training sample should be evaluated in a random test sample from the same study and, if possible, a validation sample from another study.

With a single marker and a large number of subjects, there is usually no concern with overfitting. However with many combinations of markers, overfitting could invalidate results. Overfitting is often associated with step-wise regression models \[[@B21]\] but it can occur in other situations as well. Overfitting of a larger number of markers to a relatively small number of subjects produces a model that is overly sensitive to chance fluctuations in the data. As a simple example, overfitting occurs when a sports announcer reports that a baseball player had a very high batting average against left-handed pitchers in ballpark X over the past month. This average is not very reliable for future predictions because the particular set of factors, left-handed pitchers and ballpark X, were selected to give a high average. In reality the high average is more likely the result of chance factors that coincided with left handed pitchers at ballpark X during that particular month. One way to adjust for overfitting is to apply the factors in another sample not used for initial reporting. The first sample is known as the training sample, and the second sample is known as the test sample. For example, the prediction of the average against left handed pitchers in the ballpark X could be tested on data from a different month. Similarly, a standard statistical approach to adjust for overfitting is to randomly split the data into a training and test samples. This is called the split-sample approach. Promising marker combinations are identified in the training sample, but more reliable FPR and TPR measurements are made in the test sample because it involves different data. Baker \[[@B16]\] used the split-sample approach to evaluate the performance of four markers for prostate cancer.

More sophisticated adjustments for overfitting include cross-validation and bootstrapping. Suppose the interest lies in a single statistic, such as the area under the part of the ROC curve corresponding to a small FPR. In a typical cross-validation analysis (i) the data are randomly split into ten samples each with 10% of the data, (ii) one of the 10% samples is deleted, (iii) the statistic is computed by applying the classification rule recalculated from the remaining 90% of the sample to the deleted 10% sample and (iv) an average is taken of the statistic over all ten 10% samples \[[@B22]\]. Although one could use a different fraction than 10%, ROC computation it is not possible with the most extreme version of cross-validation that involves leaving out one subject at a time. With bootstrapping, (i) the entire data set is resampled with replacement many times, (ii) on each resampling the statistics is computed, and (iii) the statistics from each resampling are combined in a special way \[[@B22]\]. In a recent study comparing these adjustments with other types of statistics in a different setting, Steyerberg et al \[[@B22]\] found that the split-sample approach tended to underestimate performance, cross-validation performed poorly on some statistics that were not normally distributed, and bootstrapping performed best overall. For our purposes of estimating an entire ROC curve rather than a summary statistic, more research is needed for cross-validation and bootstrapping, as it is not clear how best to combine ROC curves over different samples.

Regardless of the method used to adjust for overfitting in forming a classification rule, to obtain the most reliable FPR and TPR measurements, the classification rule should ideally be evaluated in a *validation* sample from a *different* study, as in Baker \[[@B16]\].

6\. To identify biomarkers with true and false positive rates similar to mammography, the training, test, and validation samples should each include at least 110 randomly selected subjects without cancer and 70 subjects with cancer (as based on FPR and TPR for mammography)

The sample size is based on the need to determine if the biomarker is sufficiently promising for investigation as a trigger for early intervention in a future trial. As discussed previously, based on considerations from mammography, our target values are FPR = .01 and TPR= .80. In most situations, we think it would be of interest to specify a 95% confidence interval for TPR of (.70, .90). Using a normal approximation the target standard error is approximately .05. Setting the standard error of TPR, TPR × (1-TPR) / (square root of n), equal to .05 and solving for the sample size n, we obtain n= 64, which we round up to 70. In addition we think that in most situations the largest reasonable value of FPR would be .03 which is 3 times the number of false positives as with mammography screening. Because FPR is so small, we do not use a normal approximation. We specify a sample size of n = 110, so that under the binomial distribution with FPR=.01, the upper 2.5% bound equals .03 × 110. Strictly, these sample sizes apply only after a single criterion for a positive test has been identified. For a training sample, one might consider larger sample sizes.

Conclusion
==========

A major advantage of nested case-control studies for early detection biomarkers is that they can be done quickly if serum from a long-term study has been stored in a biorepository. Importantly the retrospective aspect does not compromise the validity. There are none of the usual problems with retrospective studies such as recall bias. Thus we anticipate that in the coming years, there will be many reports in literature from studies of this design.

These guidelines should greatly help investigators design and analyze nested case-control studies for early detection biomarkers and help readers of the literature to interpret them. It bears emphasis that these studies do not prove clinical efficacy of the markers. Rather, they suggest which markers or marker combinations are the most promising candidates for further study as a trigger for early intervention in definitive trials with cancer-mortality endpoints.

Competing interests
===================

None declared

Pre-publication history
=======================

The pre-publication history for this paper can be accessed here:

<>

Acknowledgement
===============

The authors thank David Ransohoff for his helpful comments.