Spectrum Bias or Spectrum Effect? Subgroup Variation in Diagnostic Test Evaluation

When considering a new diagnostic test, clinicians must decide whether an evaluation of a new test is valid and whether the results are generalizable to their own patient population. An optimal study of a new diagnostic test includes a broad spectrum of persons who would normally undergo the test in a clinical setting. This broad spectrum is necessary to produce valid, precise, and generalizable estimates of test performance (1-3). However, test performance often varies with patient characteristics. For example, the sensitivity (the probability of positive test results given disease) (P[T+ |D+]) and specificity (the probability of negative test results given no disease) (P[T |D]) of a diagnostic test may be higher or lower in certain patient subgroups (4, 5). Evaluations of diagnostic tests must attempt to achieve valid, precise, and generalizable measurements of test accuracy, despite this heterogeneity. In this paper, we reexamine the issue of patient spectrum first described by Ransohoff and Feinstein (6). We provide guidance for clinicians interpreting reports of diagnostic test evaluations as well as suggestions for data analysis and presentation. Origins of the Concept of Spectrum Bias Ransohoff and Feinstein introduced the concept of patient spectrum in a landmark article in 1978 (6). They observed that the performance of a test in practice may be misrepresented by clinical studies that include too narrow a range of diseased case-patients or too narrow a range of nondiseased controls. They highlighted characteristics to consider when designing a study that examines the efficacy of a test, including pathologic, clinical, and comorbid features of the case-patient and control groups. Their concern that the relationship between disease state and test performance may change according to characteristics of the patient sample has profoundly influenced diagnostic test evaluation. Although Ransohoff and Feinstein did not use the term spectrum bias explicitly, their discussion of patient spectrum in a paper examining biases affecting diagnostic test research fostered the use of the term to describe subgroup variation (5, 7-11). The term spectrum bias implies that patient spectrum is a problem requiring correction. Consequently, a wide variety of remedies has been proposed, including reporting the average sensitivity and specificity of the sample tested (3, 7, 8, 12), using covariate adjustment (13), reporting results of subgroup analyses (2-4, 7, 8, 10, 12, 14, 15), and reporting study population characteristics as a minimum (2, 3, 7, 8, 14). Others have argued that many estimates of sensitivity and specificity are useless because study samples are too dissimilar to actual patient populations (11, 16, 17). Spectrum Effect: An Alternative Term for Subgroup Variation in Test Performance The term spectrum bias is commonly applied when the diseasetest relationship is heterogeneous across patient subgroups and the study draws preferentially from a limited portion of the patient spectrum. Including a broad spectrum of patients has been the most common approach to addressing subgroup heterogeneity (7, 8, 14, 16, 18). However, spectrum bias is a misnomer because it falsely suggests a systematic error in study design, data collection, or analysis that compromises the validity of the results. We suggest that the term spectrum bias be replaced with the term spectrum effect, which reflects the inherent variation in test performance among population subgroups. Subgroup variation is not a bias but is clinically relevant information to be identified and reported with appropriate analyses. If a spectrum effect is possible, heterogeneity should be assessed by subgroup analyses of test performance. The stability and validity of the subgroup estimates will be limited by the sample size of each category. The most common approach for addressing patient spectrumincluding a variety of participants and reporting a single estimate of test performancemay actually produce an estimate with limited clinical utility. When heterogeneity is present, the overall population estimate is not generalizable to any specific patient population. Ignoring the spectrum effect will lead to a population estimate that is a weighted average of test performances across subgroups (7, 12, 13, 19, 20). This estimate will vary with the proportion of patients in each subgroup included in the weighted average. The primary issue for the spectrum effect is generalizability. If test performance varies substantially by sex, there is little clinical value in using the population estimate of test performance that combines men and women to determine post-test probability for any individual patient. Instead, the subgroup estimates for men should be applied in men and the subgroup estimates for women should be used in women. Furthermore, estimating test performance in a clearly defined, narrow spectrum of patients is not biased but is instead a valid estimate for a specific patient subgroup. However, such estimates should not be generalized to other patient subgroups. A simple example illustrates the spectrum effect (Table 1). Consider a new diagnostic test that performs heterogeneously across age. Among persons younger than 50 years of age, the test performs with a sensitivity and specificity of 95% and 75%, respectively. Among persons at least 50 years of age, the test performs with a sensitivity of 75% and a specificity of 95%. In an investigation of this test, the study sample is drawn preferentially from the source population so that 75% of persons younger than 50 years of age are included in the study and only 25% of persons 50 years of age or older are included. The sensitivity and specificity of the test in this study sample are found to be 90% and 80%, respectively. In a second study, only 25% of persons younger than 50 years of age are included and 75% of persons 50 years of age or older are included. The sensitivity and specificity are determined to be 80% and 90%, respectively. Comparing these two studies, we find that both have included a reasonable spectrum of participants but provide differing population-average estimates of sensitivity and specificity. However, estimates of sensitivity and specificity within strata of age are unbiased, that is, identical to those in the source population. Table 1. Performance of a Hypothetical Diagnostic Test under Different Spectrums of Age Assessing the Spectrum Effect Stratified Analysis of Sensitivity, Specificity, Likelihood Ratios, and Receiver-Operating Characteristic Curves Sensitivity, specificity, likelihood ratios, and receiver-operating characteristic (ROC) curves are commonly used indicators of test accuracy. Each of these measures can be evaluated for spectrum effect with stratified analyses of patient subgroups. Subgroup variation in sensitivity and specificity estimates can be examined analytically by stratifying on the characteristic defining the subgroup and by employing a simple chi-square test of association. An association between a patient characteristic and the result of a diagnostic test may indicate the need to report estimates for subgroups of the sample. The clinical significance of the difference in sensitivity or specificity must also be considered when the P value of the chi-square test is being examined. Small cell sizes may cause the test to miss a clinically meaningful difference. Conversely, very large cell sizes may result in a statistically significant association that is not clinically meaningful, as often occurs with specificity estimates arising from cross-sectional studies of populations with a low prevalence of disease. The positive (P[T+ |D+]/P[T+ |D]) and negative (P[T |D+]/P[T |D]) likelihood ratios provide an estimate of the change in the odds for disease given a particular test result. Because the algebraic construction of likelihood ratios is analogous to risk ratios (21), the spectrum effect can be evaluated in likelihood ratios by using a test of homogeneity, available in many software programs. Statistical tests of homogeneity have low power to detect an effect; therefore, values are often less conservative for these tests (for example, = 0.10) (22). Receiver-operating characteristic curves plot the false-positive rate (1 specificity) against the true-positive rate (sensitivity). The area under this curve indicates how accurately a test discriminates between true-positive and true-negative results, independent of the positive test threshold (23). Subgroup-specific ROC curves can be constructed and compared. The curve that lies farther toward the upper left quadrant has better discriminatory accuracy. Calculating the critical ratio and comparing it with a normal distribution can provide a quantitative comparison of these uncorrelated ROC curves (24). Stratified Analysis of Enzyme Immunoassay for Chlamydia trachomatis As an example of addressing spectrum effect, we assessed the performance of an enzyme immunoassay for Chlamydia trachomatis (25). We examined age and clinic type as two representative factors with the potential for subgroup variation. Compared with a reference standard of ligase chain-reaction assay, the overall sensitivity and specificity for enzyme immunoassay were 73.4% and 99.4%, respectively (Table 2). When stratified by age, sensitivity and specificity varied significantly between older and younger patients; the test performed better in the younger subgroup (P 0.001) (Table 2). Positive and negative likelihood ratios also differed significantly after stratification by age. The enzyme immunoassay ROC curves and their areas revealed considerable heterogeneity after stratification by age (Figure). The enzyme immunoassay seemed to have the highest discriminatory accuracy among participants who were 24 years of age or younger. This finding was confirmed by formal comparison of the areas under the curve (critical ratio, 3.18; P 0.001), verifying our visual assessm