When is measuring sensitivity and specificity sufficient to evaluate a diagnostic test, and when do we need randomized trials?

Diagnostic tests are used to confirm, exclude, classify, or monitor disease to guide treatment. Their clinical value depends on whether the information they provide leads to improved patient outcomes; this can be assessed by randomized trials that compare patient outcomes from the new diagnostic test versus the old test strategy. However, randomized trials of test-and-treatment strategies are not routinely performed. They are not required for marketing approval, and they are not always feasible because they require large sample sizes. As a result, new diagnostic tests frequently enter clinical practice without evidence of improved patient outcomes. Studies of diagnostic test accuracy can show how well diagnostic strategies that include a new test identify the presence or absence of disease compared with an old test strategy by comparing each with the results of a reference standard test. However, if clinicians use only information about test accuracy to decide whether to adopt a new diagnostic test, they sometimes may harm patients (for example, if subsequent treatments are unsafe or ineffective). It is therefore worthwhile to investigate how best to decide if clinicians can rely on evidence about test accuracy or if they need to wait for patient outcome results from randomized trials. Fryback and Thornbury (1) described a hierarchy of 6 levels of evidence for the assessment of a diagnostic test: 1) the technical quality of test information; 2) diagnostic accuracy; 3) change in the referring physician's diagnostic thinking; 4) change in the patient management plan; 5) change in patient outcomes; and 6) societal costs and benefits. This framework does not provide guidelines about if and when lower levels of evidence are adequate to assess a test and always requires randomized trials for conclusions about improved patient outcomes. Some researchers have suggested that accuracy studies alone may sometimes suffice (2-6). We provide a practical framework to help clinicians decide whether a new diagnostic test can be adopted on the basis of evidence of test accuracy alone or if they need to wait for results from randomized trials. A Framework for Deciding if Evidence of Test Accuracy Will Suffice Studies of diagnostic test accuracy can suffice if clinicians already have evidence from randomized trials showing that treatment of the cases detected by the diagnostic test improved patient outcomes (Figure 1). This approach may seem straightforward; however, it requires a clear understanding of the proposed use and benefits of the new test, as well as careful consideration of whether the cases detected are representative of the patients included in treatment trials. Figure 1. Trial evidence versus linked evidence of test accuracy and treatment efficacy. The benefits of a new diagnostic test will vary according to how it is used. Investigators of a new diagnostic test need to explicitly state who will be tested, where the new test will fit in the existing diagnostic pathway, and what tests it will supplement or replace. This information will allow them to identify the expected benefits of adopting the new test and, therefore, the most relevant questions to ask to assess its value. Test attributes generally fall into 3 categories: 1) The test is safer or is less costly; 2) the test is more specific (excludes more cases of nondisease) and thus avoids unnecessary treatment; and 3) the test is more sensitive (detects more cases of disease) and thus promotes more appropriate treatment. We propose a simple sequence of questions to guide decisions about whether evidence of test accuracy will suffice for each of these 3 categories (Figure 2). The first step in assessing a new diagnostic test is to classify it according to whether it is more sensitive than the old test. We describe the key concepts of this framework using simple examples to describe situations where the new test offers better safety or specificity with similar sensitivity, followed by consideration of situations where the test is more sensitive. We also consider other, more complex scenarios. In each of these examples, we have some existing evidence of treatment efficacy for cases detected by an old test, and therefore, the rationale for testing has already been established. Figure 2. Assessing new tests using evidence of test accuracy, given that treatment is effective for cases detected by the old test. When a New Test Has Similar Sensitivity to an Old Test If a new diagnostic and old diagnostic test have similar sensitivity, it is generally reasonable to assume that they will detect the same true cases of disease. However, a new test may offer other positive attributes, such as better safety or more specificity than an old test. If studies of test accuracy show that the new test offers other positive attributes without a loss of sensitivity, it is logical to assume that cases detected by either test will show the same response to treatment. Therefore, new trials assessing treatment efficacy in the cases detected by the new test are not needed. When a new test and an old test show similar sensitivity and specificity, the value of the new test corresponds to the benefits of avoiding the adverse events or costs associated with the old test. For example, Doppler ultrasonography has been adopted as a less invasive replacement for venography for diagnosis of deep venous thrombosis (7). Given that trials have shown that treatment of this condition is effective (8), studies of test accuracy showing that Doppler ultrasonography and venography have similar sensitivity and specificity suffice to determine the value of the new test (Table, example 1). Table. New Diagnostic Test Assessment Framework and Examples If a new test is more specific than an old test, it avoids the harms of unnecessary further investigation or treatment. In this scenario, evidence showing that the new test has similar sensitivity but better specificity than the old test will suffice to determine the new test's value. For example, new immunochemical fecal occult blood testing kits have been proposed to offer increased specificity, without a loss of sensitivity, for detecting human blood, reducing the high false-positive rate of the older guaiac-based test kits (9). Given that trials have already shown the benefit of detecting colorectal cancer with guaiac test kits, studies of test accuracy will suffice to determine whether the immunochemical tests have increased specificity without any substantial loss in sensitivity (Table, example 2). Our assessment is more complex when we consider a new test that involves a tradeoff between positive and negative attributes. We deal with this situation later. When the New Test Is More Sensitive than the Old Test If a new test is more sensitive than an old test but has similar specificity, its value is directly related to the treatment response in the extra cases detected. If treatment response has already been assessed by treatment trials enrolling patients detected by the new test, decisions to use the test will be based on whether these trials showed that treatment improves patient outcomes. In this instance, evidence of test accuracy linked with evidence of treatment efficacy replaces the need for new randomized trials (Figure 1). There will also be a good match between tested and treated populations if the results for patients identified by a new test are analyzed in a treatment trial as potential predictors of treatment response. For example, trials of adjuvant tamoxifen among women with early breast cancer have shown that the estrogen receptor status of the tumor determines its response to treatment (10). These trials demonstrate the clinical value of testing for estrogen receptor status, as well as the effectiveness of tamoxifen therapy. However, more commonly, treatment trials have only enrolled cases detected by an old test. In these situations, clinicians need to consider whether the results apply to cases detected by the new test. Assessing Whether the Extra Cases Detected by a New, More Sensitive Test Respond to Treatment Clinicians first need to ask whether the extra cases detected by a new diagnostic test represent the same spectrum or subtypes of disease as those included in treatment trials. If they do, then randomized trials may not be required (Figure 2). If they do not, or if clinicians cannot be certain that they do, then clinicians also need to ask whether treatment response is known to be similar across the range of disease spectrum or subtype. These considerations are important regardless of the magnitude of the difference in sensitivity between tests. At one extreme, a good match between tested and treated populations is possible if the reference standard used to determine test sensitivity and specificity is also the starting point for trials conducted in similar populations. For example, computed tomography colonography is more sensitive for the detection of large colorectal polyps when scanning is performed with the patient in prone and supine positions versus supine positioning alone, without reduced specificity (11). Treatment trials showing improved survival following the early detection and treatment of colorectal polyps have been based on cases detected by colonoscopy, the reference standard for the computed tomography colonography accuracy studies. All of these tests are used to identify the same disease characteristic (adenomatous colorectal polyps) and spectrum of disease (classified by polyp size), so it is reasonable to conclude that computed tomography colonography with dual positioning will improve patient outcomes compared with supine positioning alone (Table, example 3). At the other extreme, a good match between tested and treated populations will not be possible if a new test measures a different biological characteristic to define disease and leads to a different selection of cases for treatment. For example, a new nucl