Reply : A critical reflection on the grading of the certainty of evidence in umbrella reviews

I would like to thank Dr. Schlesinger and colleagues for their critical reflection on the grading of the certainty of evidence in umbrella reviews. Even though this field is relatively new and we will learn a lot as it develops, there are certain aspects of this approach that are not as arbitrary as Schlesinger and colleagues are presenting. The goal of my commentary [1] is not to provide a short “manual” on how to perform an umbrella review. The commentary specifically highlights the complexity and the expertise needed to perform such a review by elaborating on the different aspects that need to be very carefully taken into consideration in the design, analysis and interpretation of the evidence. The classification for meta-analyses is described as Convincing (Class I), Highly suggestive (Class II), Suggestive (Class III) and Weak (Class VI). The wording aims to work against simple black-and-white dichotomization of significant and non-significant results. There may be no formal consensus yet on the use of this classification from recognized professional organizations but they have been repeatedly used from large teams with long lasting expertise in evidence synthesis. Moreover, I should clarify that the criteria for class I–IV categories are meant to be applied to meta-analyses of observational studies of risk factors. They are not aimed to be applied to meta-analyses of interventions (typically studied with randomized trials). I am surprised to see the authors’ claim that the overall approach and grading criteria are relying solely on statistical significance expressed as p values. Actually, the opposite is true. The proposed criteria should not be judged on a standalone basis but collectively, because they complement each other and do not substitute the careful appraisal of the evidence before any classification is performed. Passing a p value threshold of 0.05 provides very weak evidence, if any, to support an observational association, it is very easy to achieve but it means close to nothing nowadays [2]. Several methodologists proposed to shift the threshold from 0.05 to 0.005 for new discoveries [3] and many fields, e.g. in omics already use far more stringent p value thresholds. Therefore, the proposed threshold for umbrella reviews is 10−6 for classes I and II and 10−3 for class III. The strength of the proposed association is reinforced by applying the number of events or cases criterion which is more than 1000 and requires a large total sample size and decent power to detect any other than small effects. The proposed criteria use two different ways to evaluate inconsistency and its impact in meta-analysis. One is the 50% cut-off for I2 and the other is the 95% predictive intervals, which convey much more information than statistical significance of the meta-analysis they derive from. The I2 is a relative measure of heterogeneity which can be used to compare the amount of inconsistency across different metaanalyses and even with different number of studies. There is a long lasting debate about the use and the interpretation of these cut-offs, but the Cochrane handbook still refers to them in finer categorization than dichotomy. In addition, the 95% prediction intervals, which unfortunately are not so commonly reported but much more insightful, are very well suited to evaluate the variability of the effect size in different settings [4], not to judge the statistical significance. Several of the Bradford Hill criteria for causation (specificity, temporality, biological gradient, plausibility, coherence, and analogy) do not necessarily involve quantification and they can be addressed in the steps described before applying the criteria described in the commentary. Bradford Hill was very careful not to consider any of them as definitive and the performance of these criteria in the current research environment is often limited or problematic [5]. Schlesinger et al. also did not notice that there is a clear referral to the quality assessment of the included studies in the general appraisal of the evidence. * Stefania Papatheodorou spapathe@hsph.harvard. edu