Assessing sensitivity and specificity in new diagnostic tests: the importance and challenges of study populations.

To the editor- Hospital epidemiology frequently involves evaluating new diagnostic tests. This is particularly relevant in the example of active surveillance for multidrug-resistant organisms (MDROs) in high-risk populations. Determining the sensitivity and specificity, and therefore effectiveness, of potentially new diagnostic tests requires these surveillance methods to be epidemiologically and statistically assessed. Advice on study design and biostatistical methods for evaluating diagnostic tests has recently been published, but there are additional practical concerns for hospital epidemiology and other disciplines where the incidence of disease is low that are only addressed in this editorial. Biostatistical advice on study design and reporting results for new diagnostic tests is limited. The STARD (STAndards for the Reporting of Diagnostic accuracy studies) statements discuss methods and provide recommendations on reporting and designing studies, but do not delve into numerous and important biostatistical issues.1,2 In order to help fill this gap we recently published an article discussing statistical methods for evaluating the sensitivity and specificity of novel dichotomous, diagnostic tests.3 Our publication provides recommendations for sample size calculations during the design stage when assessing diagnostic tests, as well as detailing the importance of reporting confidence intervals around sensitivity and specificity results. As briefly mentioned at the conclusion of the article, there is a tradeoff between statistical rigor and the “practical realities of sample collection.” In order to achieve narrow confidence intervals around estimates of sensitivity (and specificity), a minimum number of positive (and negative) patients must be drawn from the population to whom the test is going to be applied. Simply put, the sample size must include a large enough positive population so the confidence interval around the sensitivity can be precise. This is especially difficult when the prevalence is inherently low; as is often the case in hospital epidemiology. When assessing the sensitivity of a new test, confidence intervals reflect both the sample size and the inherent sensitivity. Small sample sizes yield wide confidence intervals, and therefore a greater statistical uncertainty of the results, even when the new diagnostic test has a high sensitivity. For example, the most recently reported prevalence of methicillin-resistant Staphylococcus aureus (MRSA) among hospital inpatients in US healthcare facilities is 66.4 per 1000 patients, 6.6%.5 As shown in the table below, the low prevalence significantly affects the confidence intervals around the sensitivity when assessing a new MRSA surveillance test for hospitals. Sensitivity of New MRSA Diagnostic Test 80% 85% 90% 95% Total Sample Size (Estimated # of Positive Patients) 100 (7) 50.4%-100.9% 58.5%-111.4% 67.7%-112.2% 78.9%-111.1% 500 (33) 66.6%-93.4% 72.9%-97.0% 79.9%-100.1% 87.7%-102.3% 1000 (65) 70.4%-89.6% 76.4%-93.6% 82.8%-97.2% 89.8%-100.2% View it in a separate window 95% Confidence Intervals for 4 different predicted sensitivities, using 3 different samples sizes and taking into account 6.6% prevalence of MRSA. These large sample sizes needed to achieve moderate to narrow confidence intervals can be daunting from a time and cost perspective. However, attempts to find alternatives to sampling large populations can prove difficult and come with their own set of epidemiological issues. One tempting solution is to find known positive patients identified by clinical cultures, and then test the new surveillance method to determine sensitivity. This practice of “cherry-picking” positive patients to get the population sizes needed for narrow confidence intervals potentially introduces even larger biases. Screening and diagnostic tests should be evaluated in the same population in which the test is going to be used.6,7 Knowing the patient is positive can skew the interpretation of results and lead to an overestimation of sensitivity for the new test (known as diagnostic review bias).2,4,8 As well, positive patients, as opposed to the true population in question, are likely to have different demographics, risk factors for the outcome, and different confounding variable distributions. These sometimes extreme differences between the population tested and the population in which the test is to be used lead to issues with generalizing results and, more importantly, yield incorrect sensitivity and specificity estimates for the test.7,8 Evaluating new diagnostic tests becomes even more complicated when there is no gold standard against which to compare the new test. Without a gold standard, additional methods are needed in order to maintain the validity of the analysis or risk incorrect sensitivity and specificity results. In these cases, options such as composite reference standard (recommended), latent class analysis or discrepant analysis, are needed in order to accurately assess the test.3,4 All three of these methods require an additional sample to be collected or test to be performed and further add to the laboratory workup and analysis.4 Finding a balance between the need for narrow confidence intervals and the practical problems of finding a large and appropriate sample population is critical. Poorly designed or reported studies can lead to premature adoption of tests and the incorrect care of patients.2,4,8 We believe that: a) studies should use larger sample sizes and b) the test should be evaluated in the population among which it is to be implemented, in order to provide the most accurate and meaningful results. Reporting confidence intervals and study population demographics will assist in this endeavor. Finding an appropriate balance will become increasingly more important as diagnostic tests, especially surveillance for MDROs, are used more routinely due to legislative requirements and the critical need to quickly and accurately test patients.