High and variable prevalence does matter in reviews of diagnostic accuracy in cerebral palsy

In reviewing the literature on the prediction of cerebral palsy (CP) diagnosis, Bosanquet et al. found 19 studies with an estimated prevalence of CP ranging from 5 to 59%; the studies were conducted in hospital neonatal or intensive care units, with all but one sampling primarily from preterm infants. The prevalence of CP should affect our appraisal of these studies and their implications for prediction. High prevalence is not necessarily problematic and is typical for studies of diagnostic accuracy because the precise estimation of sensitivity depends on having many true cases. The unbiased estimation of sensitivity and specificity requires no assumptions about the way in which patients were sampled in relation to prevalence. However, difficulties arise if we are encouraged, as I think we are here, to think of sensitivity and specificity as measuring the accuracy of ‘prediction’, because predictive accuracy (e.g. positive predictive value) depends acutely on prevalence. At any given diagnostic accuracy it is harder to predict from a positive test and easier from a negative test when there are fewer cases in the setting where the test is used. It matters not at all in this regard whether the ‘prognostic’ assessments were made before or after the criterion standard diagnosis was obtained in a research study. The sensitivity and specificity do not answer questions about predicting from test results. For example, Romeo et al.’s study has the highest Standards for Reporting Diagnostic Accuracy (STARD) quality score cited in support of the ‘good evidence for the predictive accuracy of developing CP using the General Movements Assessment. ’ The estimated sensitivity of the assessment of fidgety movements at 3 months of age is 98% for CP at 2 years of age. Given the way Romeo et al. found their cases in an Italian intensive care unit, the estimated prevalence is about 6%. An infant with a positive test in this setting has about a 53% chance of a CP diagnosis at 2 years old. The same positive general movements test used in a general practice setting where the prevalence is closer to 0.01–0.02% of live births will be correct about 3% of the time, but more than 92% of the time for settings such as Ferrari et al. where the prevalence is over 50%. But which of these figures is good evidence for the predictive accuracy of general movement assessments? I suggested initially that high prevalence is not necessarily a problem for the estimation of diagnostic accuracy, but it can be prognostic of methodological problems associated with bias. In particular, high and variable prevalence is a clue to where to look for spectrum bias. The 6.3% prevalence in the Romeo et al. study results from a process in which all preterm infants in an intensive care unit were assessed over a long enrollment period with few additional criteria. This is high prevalence, but probably not high for preterm infants whose CP is plausibly not qualitatively different in features or severity than CP among term births. In diagnostic accuracy studies, high risk is good if it produces more children with CP, but causes spectrum bias if it produces different children with CP, that is, different in terms of the ease of classification. If 6% prevalence is what you get when you test every preterm infant that comes through the intensive care unit, how different is the diagnostic problem with settings and procedures that achieve prevalence 2–9 times higher (14– 52%) for the five other studies of general movements? What explains the 38% prevalence of ‘severe CP’ in a test of the same general movements in five similar European neonatal units studied by Prechtl et al., achieved in spite of having 26% term births and 56% classified as ‘low risk’ by cranial ultrasound? Possibly there are differences in the spectrum, but Prechtl et al. also tell us that neurological examination ‘was done repeatedly until at least 24 months of age’ with no mention of blinding for the criterion standard classification of CP. When the tested assessment can influence subsequent diagnosis, this causes work-up or incorporation biases. The general point is that high and variable prevalence is a sign of methodological trouble, particularly when all of the studies in this review come from apparently similar settings. When we cannot count on variations in study characteristics to average out methodological problems in a review, the best conclusions may come from a close reading of the few best studies we can find.