How can we make the results of trials and their meta‐analyses using continuous outcomes clinically interpretable?

Our scientific community seems to have by now come to a broad agreement that repetitive transcranial magnetic stimulation (rTMS) produces statistically significantly greater improvement than sham rTMS in the treatment of acute depression. However, the clinical relevance of this statistical superiority remains unclear, and Lepping and colleagues in this issue of Acta Psychiatrica Scandinavica addressed this problem with a novel and clinically intuitive approach (1). They first systematically searched for clinical trials that compared rTMS against sham rTMS in the treatment of depression and that used the Hamilton Depression Rating Scale (HAMD). They then calculated the percentage change in HAMD scores for each arm, took their weighted average across all arms of rTMS or sham rTMS, and then converted this average percentage change to the corresponding Clinical Global ImpressionImprovement Score (CGI-I), using the established conversion method (2). Because the CGI-I has clearly interpretable anchor scores, ranging from 1 = very much improved, 2 = much improved, 3 = minimally improved, 4 = no change through 7 = very much worse, it is now easy to assess the effects of rTMS or sham rTMS qualitatively and clinically. The improvement on rTMS was 36% on HAMD, which corresponded with 2.9 on CGI-I, while that on sham rTMS was 23% on HAMD and 3.4 on CGI-I. The difference is therefore 0.5 between rTMS and sham rTMS on a 7-point scale of CGI-I, which does appear small and the review authors concluded ‘Whilst rTMS appears to be efficacious (. . .), the clinical relevance of its efficacy is doubtful’. Two of the most recent and comprehensive reviews on the same topic reached different conclusions, using different summary methods. Slotema et al. (3) conducted a standard systematic review and meta-analysis of rTMS versus sham rTMS and found an effect size of 0.55 (P < 0.001) and concluded ‘It is time to provide rTMS as a clinical treatment method for depression’. Berlim et al. (4) focused on high-frequency rTMS and the dichotomized outcomes of response rates and remission rates and found an odds ratio of 3.3 (P < 0.0001) of rTMS over sham rTMS for both outcomes and concluded that rTMS ‘seems to be associated with clinically relevant antidepressant effects’. Acknowledging that these three reviews included overlapping yet different sets of trials and focused on related yet different outcomes, how can the conclusions still be so different? Basically there are three ways to make the continuous outcomes, such as ratings of depression severity, interpretable: (i) conversion to SD (standard deviation) units, (ii) conversion to dichotomous outcomes, and (iii) conversion to natural units. Slotema et al. took the first approach: the effect size, also known as standardized mean difference, is the difference in the mean scores of the two groups, divided by the standard deviation. ‘0.55’ is an enigmatic number, unless we use the rough rule of thumb for interpreting the effect size proposed by Cronbach that 0.2 represents a small effect, 0.5 a moderate effect, and 0.8 a large effect (5). Berlim et al. used the second approach by meta-analyzing response and remission rates. Because this is a meta-analysis of dichotomous outcomes, the results can be legitimately summarized as an odds ratio. Unfortunately, here again, clinicians are at a loss to evaluate if an odds ratio of 3.3 represents a large, moderate, or small effect.