Guidelines for Meta-analyses Evaluating Diagnostic Tests

Clinicians must decide whether to use a diagnostic test in a patient and how to interpret the result [1]. Policy-makers must assess the overall value of a test, compare it to alternatives, and decide whether the test should be made available. Both clinical and policy decisions should be based on a thorough evaluation of the test [2]. A crucial step in evaluation is the assessment of diagnostic accuracythe ability of the test to determine correctly the presence or absence of the disease of interest. This is done by comparing test results with those obtained using a reference (gold, criterion, or comparison) standard. Estimates of the diagnostic accuracy of a test may differ among studies. Each study may have included too few patients to give precise estimates or too selected a population to allow general applicability. Therefore, meta-analysis, the critical review and statistical combination of results of previous research [3-6], is potentially useful for assessing diagnostic accuracy. Using meta-analysis, we can 1) provide an overall summary of diagnostic accuracy; 2) determine whether estimates of diagnostic accuracy depend on the study design characteristics [study validity] of the primary studies; 3) determine whether diagnostic accuracy differs in subgroups defined by the characteristics of the patients and test; and 4) identify areas for further research. New hypotheses may be generated or the attempt to meta-analyze data may highlight deficits that need to be addressed in future primary studies before a useful meta-analysis can be done. To help researchers do and readers assess meta-analyses of diagnostic accuracy, we suggest guidelines on how they should be conducted, reported, and critically appraised. Our guidelines are based on current concepts of how to assess diagnostic tests and conduct meta-analyses (Table 1). Because other guidelines exist for meta-analysis in general [3-8], we emphasize those issues that are particular to assessing diagnostic accuracy. For each step, the guidelines are used to review 11 journal articles published from January 1990 through December 1991 whose primary purpose was to assess the accuracy of a diagnostic test against a concurrent reference standard using meta-analysis [9-19]. The 11 articles are all the meta-analyses published in this 2-year period that we could identify through various search procedures. The guidelines were applied to each article independently by three of the authors, and the majority view was accepted. (See Appendix 1 for details of search and review procedures.) Table 1. Steps in Conducting a Meta-analysis of a Diagnostic Test and Summary of Guidelines Determine the Objective and Scope of the Meta-analysis Because the type of meta-analysis we are assessing compares a diagnostic test against a reference standard for the disease of interest, a clear statement about what diagnostic test is being evaluated and how the reference standard is defined should be provided. Diagnostic tests are frequently used against the background of clinical information that is available before testing. A test may appear useful if only the association between the test and the reference standard is examined. However, it may not be useful if its information is carried by data that are already available to the clinician or are easy to obtain without ordering the test [20]. Test evaluation should examine the incremental or marginal value of the test. For example, consider an evaluation of exercise thallium scans against the reference standard of coronary angiography in patients with symptoms suggestive of coronary artery disease. An electrocardiogram would have been obtained routinely at the time of exercise. Therefore, the relevant question is whether the thallium scan has any incremental value over the information already available from the electrocardiogram. Similarly, a test may have poorer diagnostic accuracy in a tertiary care facility than in a primary care setting because the patient population has already been selected on the results of other tests [21]. Thus, it is imperative that meta-analysts clearly state the clinical circumstances for which they wish to evaluate diagnostic accuracy. Not all primary studies identified will have used the meta-analyst's ideal reference standard or cover the exact tests or clinical context desired by the meta-analyst. This can be dealt with by using only the subset of papers that do so or by examining the extent to which deviation from the ideal changes the findings. In general, we suggest the latter approach and describe it further when we discuss how to assess the effect of variation in study validity and in the characteristics of patients and test. Often, the comparative value of several tests is being assessed. As when only a single test is being evaluated, this comparison should be viewed against the background of previous information, which should be equivalent for the tests being compared. Review. The reference standard was stated in 10 and the test of interest in all of the 11 meta-analyses. Two meta-analyses gave a clear statement about the clinical background against which the tests' incremental value was being evaluated. Some authors, for example, Pinson and colleagues [18], stated that this was a problem in the primary studies, pointing to the need for improving their quality. Seven meta-analyses compared tests, whereas 4 evaluated an individual test. Retrieve the Relevant Literature Literature searching should be done using various methods, including searching computerized bibliographic systems, consulting content experts, searching appropriate journals, and identifying further studies and search strategies using the reference lists of studies already obtained [3, 8, 20]. The final MEDLINE search strategy should be reported in sufficient detail to allow others to repeat it if they wish to update the meta-analysis after publication. Except for those clearly outside the scope of the meta-analysis, the relevance of all other papers should be judged by applying inclusion and exclusion criteria. This result could be accomplished by using the same methods outlined in the next section for the extraction of data. Reporting the reason for exclusion of potentially eligible papers helps readers understand how the criteria were applied. Publication bias may arise because of the selection for publication of papers with more extreme results because they are more interesting or statistically significant [22, 23]. Methods for dealing with publication bias have been developed [24, 25], but their applicability to diagnostic test assessment has not been explored. Review. Literature retrieval methods were described in 7 of the 11 meta-analyses, all of which used MEDLINE searching. Three articles gave their search terms, and 2 of these explained how these were linked [10, 14]. Criteria for including or excluding papers were given in 8 articles. Four articles gave information about excluded studies. Publication bias was discussed in 3 of the 11 articles, although no estimates could be made of its likely effect. Extract and Display the Data Assessing study relevance and extracting data require the use of judgment. Various procedures have been suggested to reduce the possibility of obtaining biased estimates of diagnostic accuracy, including independent review by two readers and resolution of differences by a third reader or by discussion between the original two readers [3, 8]. Sometimes, readers are blinded to details of authorship and study results [8]. Although theoretically sound, no evidence exists that blinding results in a decrease in bias. Until such evidence is obtained, the decision on whether to use blinding depends on the resources available. Whatever the decision, authors should state how the judgments were made. Publishing a full list of diagnostic accuracy and study characteristics (for example, design features, patient characteristics) for each primary study allows other researchers to decide if they agree with the judgments made and enables re-analysis applying different analytic techniques, using a subset of studies, or adding studies published after the meta-analysis was done [26]. Review. Two meta-analyses stated that each of the primary studies was assessed by two or more readers [13, 19]. One of the articles [13] also mentioned that disagreements were settled by consensus or a third party. Gianrossi and colleagues [11] offer an example of extensive display of data about diagnostic accuracy and study characteristics. Estimate Diagnostic Accuracy Before discussing the estimation of diagnostic accuracy in meta-analyses, some key concepts of test assessment need to be addressed [20, 27, 28]. Most measures of diagnostic accuracy are based on the comparison of the test with a reference standard that determines the presence or absence of the disease of interest. An ideal diagnostic test discriminates between diseased and nondiseased individuals without error. Test error can be characterized by various measures. The two most commonly reported measures are sensitivity, the probability that a test result is positive in patients with the disease of interest, and specificity, the probability that a test result is negative in patients without the disease of interest. These measures rely on a single threshold (cut-point or positivity criterion) for classifying a test result as positive. Changing the threshold to increase sensitivity decreases specificity and vice versa. This trade-off between sensitivity and specificity makes it imperative that they be considered jointly. When studies use different criteria to define positive and negative test results, as in our example, they differ in their explicit threshold. Even when studies use the same explicit threshold, their implicit thresholds may differ, especially if interpretation of the test requires judgment. For example, radiologists may agree to use the same words to describe imagi