Finding the Evidence is All in the Methodology

Statistics has never been my favourite subject, but all practising clinicians need to understand evidence synthesis and the principles of statistical analysis. All of the scientific articles at the Journal of Orthodontics are sent for independent statistical review and I am extremely grateful to our statistical advisors for the essential job that they do in this regard. However, like much anonymous peer review, it is not uncommon for this process to highlight differences of opinion between statisticians responsible for data within a particular manuscript and those on our editorial board. These types of argument can also be found in the letters pages of orthodontic journals and recently I became aware of one such example of this, which indirectly discussed some work that I had been involved in. To me, it illustrates the importance of appropriate methodology and statistical analysis but also highlights the difficulties of interpretation when disseminating the results of clinical studies. Systematic review with meta-analysis is rightly regarded as being at the pinnacle of the evidence hierarchy in clinical research. However, much like the research it attempts to synthesise, informative metaanalysis is reliant upon appropriate methodology. Systematic reviews often bring together studies that are diverse, so clinical and methodological heterogeneity is to be expected (Higgins et al., 2002; 2003; Higgins, 2008). However, authors have a responsibility to pool data appropriately. Studies included in a review need to be consistent, and whilst statistical tests are available to evaluate this, there can still be disagreement and debate. In 2010, two systematic reviews independently reported on the clinical effectiveness of self-ligating bracket systems (SL) in comparison to those with conventional ligation (CL). One concluded that there was insufficient high quality evidence to support the use of SL brackets (Fleming and Johal, 2010); the other suggested that whilst evidence for the advantages of SL brackets was generally lacking, slightly less lower incisor proclination did appear to be one significant advantage of SL systems over CL (Chen et al., 2010). A conclusion that was later reiterated in an ‘Ask the Experts’ forum in the American Journal of Orthodontics and Dentofacial Orthopedics (Marshall et al., 2010) and which has been further discussed in the context of Damon SL brackets (Wright et al., 2011). How can two systematic reviews carried out at the same time and analysing a relatively small number of the same trials come to such different conclusions? To answer this question requires a closer look at the methodology. Fleming and Johal concluded that there were insufficient trials of low or medium-risk bias to allow meta-analysis with regard to the outcome of lower incisor proclination. In contrast, Chen and co-workers included three studies in their analysis of lower incisor change. One was a randomized controlled trial (Scott et al., 2008; here I must declare, that this was a study that I was involved in), whilst the other two were prospective cohort studies (Jiang and Fu, 2008; Pandis et al., 2009). These prospective cohort studies were considered by the review authors to be at high and low risk of bias, respectively. Indeed, the study by Jiang and Fu actually found in favour of CL brackets, but there were only 13 participants in each group, the lower arch crowding only amounted to a mean of 3.5 mm and there were wide standard deviations in incisor change. However, all these investigations had analysed Damon SL brackets and their inclusion was justified on the basis that the literature on SL brackets was sparse, these were the only studies that provided data that could be analysed and the relatively low weight assigned to the Jiang study meant that it did not actually influence the overall outcome with regard to lower incisor proclination (Chen et al., 2011). This potential bias has been highlighted and the data re-analysed using only the two low-bias studies (Kaklamanos and Athanasiou, 2011). As might be expected, exclusion of the Jiang and Fu study still results in a meta-analysis that suggests an advantage for SL brackets in producing less lower incisor proclination; however, in the absence of this study, a value called I goes up. I is an index of heterogeneity and represents the percentage total variation across the studies being considered. The higher the value of I, the more likelihood that variability across the studies is due to heterogeneity rather than chance, with values of 25%, 50% and 75% categorized as low, moderate and high, respectively (Higgins et al., 2003). In our example, excluding Jiang and Fu increases I from EDITORIAL Journal of Orthodontics, Vol. 41, 2014, 165–166