Cochrane Risk of Bias : ‘Your common man has no conception of the zeal that animates a scientific investigator, the fury of contradiction you can arouse in him’

Dear Editor-in-Chief, ‘Your common man has no conception of the zeal that animates a scientific investigator, the fury of contradiction you can arouse in him’ As Editors of the Cochrane Eyes and Vision group we felt that Professor Wilkins’ somewhat sweeping assertion that the Cochrane Risk of Bias tool was not ‘up to the job’ should not go unchallenged. Those whose interest in systematic reviews has been aroused by conflicting views amongst senior academics may wish to read a recent report which has systematically synthesised published comments upon the strengths and weaknesses of the Cochrane Tool. Jørgensen et al. found that the Tool has become the standard approach to assess risk of bias in randomized clinical trials but that it has frequently been implemented in a non-recommended manner. Inconsistent implementation may in part explain the poor agreement rates which have been identified in the use of the tool. Poor agreement may, however, not matter if the reviewer’s conclusions would be the same and the fact that the tool’s configuration facilitates checking should not be overlooked. Meta-epidemiological studies have proved that studies with inadequate or unclear sequence generation, allocation concealment or masking procedures tend to overestimate treatment effects compared with trials of adequate quality, especially for subjective outcomes. What matters in the Cochrane Risk of Bias tools is that, despite the fact that agreement in assessing trial characteristics may be moderate, its domains are based on constructs that are definitely the fundamentals of modern biomedical science. In their systematic review of the effect of coloured overlays and lenses on reading performance, Griffiths et al. do not report an explicit statement as to why they chose not to conduct a meta-analyses of the included studies. Typical reasons for avoiding quantitative synthesis within a systematic review revolve around heterogeneity – the included studies being thought to be too different, either statistically or clinically (patients, interventions or outcomes). In the discussion, Griffiths et al. state that ‘Based upon our view that a large majority of the literature we reviewed is at high risk of bias it is not clear that this field of research is ready for meta-analysis’. However, if the authors have set inclusion criteria that allow the inclusion of such studies, high risk of bias in most studies is a reason for assigning low certainty to the available evidence rather than avoiding a meta-analysis. In his comments regarding the Griffiths’ et al. review, Wilkins found it inappropriate that the Wilkins Rate of Reading Test was rated as ‘High risk of bias’ in an additional domain called ‘External bias’. It is important to note that this domain is not part of the Cochrane Risk of Bias tool but was used by the review authors to assess a range of issues that include patient sampling and surrogacy of outcome measures. In a standard Cochrane review, these issues would be dealt with in a following step that leads to the preparation of a Summary of Findings Table using GRADE to integrate risk of bias assessment with other dimensions: indirectness, whether the evidence from the included studies is directly relevant to the review question in terms of patients, interventions and outcomes; inconsistency, whether the effects are consistent across studies; imprecision, whether the effects are measured enough precisely to allow an unequivocal clinical interpretation of their direction and magnitude; and publication bias. In Griffiths’ et al. review, the authors consider the use of Wilkins test as providing indirect evidence, but this is a clinical, not a methodological decision. Because Griffiths et al. did not conduct a meta-analysis, but gave no valid reason for not doing so, and did not formally assess the certainty of the evidence for dimensions other than risk of bias using GRADE, their review is systematic but its analysis is narrative. Narrative summaries allow more room for subjectivity and may sometimes be misleading although it is fully acknowledged that quantitative analysis which may capture and model uncertainties is not always viable. Constructive criticism has great value in identifying where improvement in any instrument is required but a systematic review of current comments indicates that the Tool is very much more than a ‘a set of untraceable opinions of uncertain validity, dressed up to look like science’.