Authors' response to V Johnson and Y Yuan

© Society for Clinical Trials 2007 10.1177/1740774507079433 We thank Johnson and Yuan for their very interesting comments. We also noted upfront that significance-chasing behavior and between-study heterogeneity may be confounded. This is also a typical problem of all previously proposed tests for publication bias [1]. Nevertheless, an empirical evaluation of 6873 meta-analyses from the Cochrane database shows that only 25% of them have high or statistically significant heterogeneity [2]. The most common problem in applying ‘publication bias diagnostics’ is the inadequate number of studies, with fewer than ten studies in 87% of these meta-analyses [2]. The proposed exploratory test would not be very useful in investigating small meta-analyses of few studies, with or without heterogeneity. It is intended for use primarily in assessing large meta-analyses and even more so in assessing whole domains of many meta-analyses for an excess of statistically significant findings, something that traditional ‘publication bias diagnostics’ are currently not set to do. A word of caution is due about observed betweenstudy heterogeneity. Observed heterogeneity does not necessarily reflect genuine diversity across true treatment effects. Significance-chasing behavior may often generate between-study heterogeneity on its own. Diverse effect sizes across studies may simply reflect the play of bias – or even simple unintentional errors and lack of standardization. The relative contribution of genuine heterogeneity versus bias and errors requires careful evaluation with large-scale empirical evidence. However, in the conduct of individual participant data metaanalyses, we have frequently come across grouplevel between-study heterogeneity that evaporated once definitions, methods, outcomes and analyses were standardized across data sets and errors were corrected that had affected the original grouplevel data. An excess of statistically significant findings can be caused by biases that go beyond the traditional definition of publication bias, and these may sometimes affect the observed heterogeneity. Our test aims to address the composite effects of all significance-chasing behaviors: results not published because of being negative (publication bias), results distorted (in analyses and in selection of outcomes and what to report) so that they show large positive effects and positive results fabricated out of nonexisting data. We feel that the focus on publication bias to date has been misleading, since the other two significance-chasing behaviors may be more important than publication bias in many fields [3–5]. All three significance-chasing behaviors cause a relative excess of significant findings, in principle detectable with our proposed test. Focusing only on publication bias has caused a ghost hunt for invisible studies (by default intractable after their disappearance), whereas the problem may lie at the visible studies in front of our eyes. The exact impact of significance-chasing on observed between-study heterogeneity can follow different patterns. One pattern is extreme conformity: if there is very strong belief and/or conflicts of interest to support a specific effect, the published literature is forced to spurious homogeneity. Extreme homogeneity of effects may be tested [6]. Our exploratory test would also pick consistent efforts to pass the threshold of significance provided enough studies are available. In a second pattern, some genuine null effects may manage to get published as such, if the barrier against negative results is not absolute. In this case, the studies that reflect the true effects would be the ones seemingly causing the between-study heterogeneity. In a third pattern, significance-chasing infiltrates only a portion of the literature. Then the studies suffering from significance-chasing bias would be the ones seemingly causing the heterogeneity. In a fourth pattern, extremely contradictory results (e.g., significant in the opposite direction) are not only unwelcome, but they are even encouraged by the publication environment. This so-called Proteus phenomenon is common in some observational research, for example genetic epidemiology [7]. Our exploratory test may be applied also in all these situations, but with due caution in application and interpretation. We fully agree with Johnson and Yuan that reported P-values eventually do not protect their consumers in the current research environment. This requires an urgent paradigm shift that is overdue in Authors’ response to V Johnson and Y Yuan