When should a new test become the current reference standard?

Key Summary Points Whether a new test (such as polymerase chain reaction) should replace a current reference test (such as culture) can be determined by a fair umpire test. A fair umpire test may have errors even larger than the tests under evaluation. What makes it a fair umpire is that its errors are independent of those of both tests. Possible umpires include causal exposures, concurrent testing, prognosis, or response to treatment. To decide whether the new test or current reference standard test is more accurate requires the fair umpire to be applied only to cases in which these tests' results differ. Time is often a better diagnostician than the best anatomical pathologist. Clifton Meador New diagnostic tests do not simply alter our diagnostic processes; they may also change whom we classify as having a disease. Gold standard tests are often imperfect, which leads to modification or change of the standardsometimes by deliberate decision and sometimes by stealth. A definitional shift occurs when a new test detects additional cases of apparent disease but creates uncertainty about whether these additional cases should be classified and treated in the same way. For example, magnetic resonance imaging for suspected multiple sclerosis has widened the diagnosis in practice, but the correlation of lesions found on imaging with eventual clinical course is poor (1); hence, some patients will be falsely labeled and may receive unnecessary treatment. This problem is not unique to multiple sclerosis; new enzyme tests, such as troponins (2); DNA methods in microbiology, such as polymerase chain reaction tests (3); new cardiac hormone tests, such as B-type natriuretic peptide for heart failure (4); and new imaging methods, such as magnetic resonance imaging and positron emission tomography in neurologic and musculoskeletal diseases (5), are changing the spectrum of patients considered to have specific diseases. These changes have led to discussion and dispute about how the additional cases detected should be managed. As our diagnostic armamentarium continues to expand and improve, this dilemma will increasingly challenge us. One trigger for changing the reference test is known flaws in the current reference test. For example, the insensitivity of culture for viruses and difficult-to-culture bacteria created interest in polymerase chain reaction methods, and problems with diagnosing diastolic heart failure created interest in alternatives to echocardiography. A second trigger is when a new technology, such as magnetic resonance imaging or positron emission tomography, suggests new diagnostic possibilities or stages of disease not previously perceived. This need or perception does not necessarily translate into greater diagnostic accuracy that leads to greater clinical benefits. Of course, not all new diagnostic technologies will lead to reconsideration of the reference test. Some may be only improvements on a triage test, or may be less invasive or more convenient replacements of other tests (6). But tests that do change our definition of disease require more than a consideration of accuracy; they also require assessment of the clinical consequences of the change. What principles should guide replacement of a current clinical reference test? Investigators may claim that the new test is better than existing reference standard tests on theoretical grounds, or is more reliable, or has better sensitivity. If the current reference standard is flawed, then we have no obvious means of deciding whether the proposed new reference standard is really better. Most previous work on this topic has considered the problem of estimating the accuracy of the new test (7) by using such methods as combined tests (using several tests as the reference standard) or discrepant analysis (810) (use of a third test to resolve disagreements between the 2 tests). However, estimation of accuracy is not essential in the decision to adopt a new reference standard (11). Any principles or criteria for replacing the current reference standard need to adequately assess the consequences of the switch, both nosologically and clinically. We aim to set out the criteria and principles for accepting a new test as a better reference standard or component of a reference standard. The 3 key principles that assist with deciding on the value of the new test are as follows. The consequences of the new reference test can be understood through the disagreements between the old and new reference tests. Resolving the disagreements between old and new test requires a fair, but not necessarily perfect, umpire test. Possible umpire tests include causal exposures, concurrent testing, prognosis, or the response to treatment. We examine each principle in turn and then discuss how they may work together. Disagreements between Old and New Tests A new reference standard test may lead to a diagnostic spectrum shift by either broadening or narrowing a diagnosis. Most commonly, an apparently more sensitive test broadens the diagnosis by detecting earlier or less consequential cases. Troponins in chest pain and magnetic resonance imaging in breast disease or suspected multiple sclerosis are typical examples of this. Less commonly, the test will narrow the diagnosis by excluding cases that were previously diagnosed, either because the disease was inconsequential or the reference test yielded an incorrect diagnosis. For example, nerve conduction studies narrowed the range of patients considered to have carpal tunnel syndrome. Narrowing also occurs when specific diagnoses are removed from a broad nonspecific category, such as when antibody testing led to reclassification of some patients previously labeled as having the irritable bowel syndrome (12). To understand the consequences of switching reference tests, we may use the following principle to focus investigation: Principle 1: The consequences of the new reference test can be understood through the disagreements between the old and new reference tests. The shaded cells in Table 1 show these consequences through the hypothetical comparison of a new and old reference standard test. The consequence of switching from the old to the new reference test is that the spectrum of disease in the patients being treated shifts from cell C (old test positive; new test negative) to cell B (old test negative; new test positive). Table 1. Possible Consequences of Switching from Old to New Reference Tests The possible disagreements in Table 1 may be divided into 2 simpler cases (Table 2). First, the new test may detect extra possible cases. This apparent additional sensitivity of the new test may also involve a shift to the detection of earlier or less severe cases, such as additional cases of myocardial ischemia detected by troponin (2) or cases of celiac disease detected by endomysial antibody (13). However, these extra cases may also be false positives or cases of less severe illness, so careful assessment is required. Second, the new test may detect fewer possible cases. This apparently better specificity of the new test may also involve some shift from earlier or less severe and less consequential cases. For example, concerns about white-coat hypertension have led to increasing use of ambulatory blood pressure monitoring to reduce false-positive diagnoses of hypertension. Similar concerns have been expressed about overdiagnosis of many conditions, particularly from screening techniques (such as lung and breast cancer screening) (14). However, these cases may also be false negatives, and careful assessment is again required. For example, because we would wish to see that persons classified as nonhypertensive by ambulatory blood pressure monitoring have a similar prognosis to those with no hypertension, the clinical consequences for the discordant group need careful assessment. Table 2. Performance of New Test if (I) More Sensitive or (II) More Specific It is possible to have cases in both cells B and C of Table 1. The crucial question is not simply the numbers in the discordant cells (B and C), but the nature of those cases. Are the additional cases detected by the new test actual (consequential) cases of the disease or merely false-positive results? And are the cases not detected by the new test actual (consequential) cases of the disease (that is, is the test yielding false-negative results?) If the numbers were equal but the additional cases detected by the new test (cell B) were more serious cases that had a greater gain from treatment than the missed cases (cell C), then the new test would provide a net benefit. Conversely, if the additional cases were less consequential and benefited less from treatment, then the shift would be undesirable. Deciding on the balance of consequences of the new test is not simple. A randomized trial may be required to demonstrate the value (or nonvalue) of treatment in patients in cells B and C. For example, the introduction of troponins has altered our ability to detect ischemic myocardial damage but led to uncertainty about appropriate treatments for this new spectrum of patients. This uncertainty led to several randomized trials in nonST elevation myocardial infarction, which, when pooled, suggest that early revascularization is beneficial (15), at least for some subgroups (16). Other examples or scenarios may be simpler, such as the identification of additional surgically correctable epileptic foci by positron emission tomography (5), and well-documented beforeafter case series may be sufficient (17). Randomized trials will only need to be considered for patients in the discordant cells (cells B and C in Table 1). However, such trials are sometimes unnecessary (18) or not feasible. We should begin with an examination of any known flaws in the current reference test (19) to see whether these might explain the differences. This is a useful start but alone is insufficient, and the next 2 principles provide gu