Estimating Publication Bias in Meta-Analyses: A Meta-Meta-Analysis Across Disciplines and Journal Tiers

Selective publication and reporting in individual papers compromise the scientiﬁc record, but are meta-analyses as compromised as their constituent studies? We systematically sampled 63 moderately large meta-analyses (at least 40 studies per meta-analysis) in PLOS One , top medical journals, top psychology journals, and Metalab, an online, open-data database of developmental psychology meta-analyses. We empirically estimated publication bias in each. Across all meta-analyses, “statistically signiﬁcant” results in the expected direction were only 1 . 20 times more likely to be published than “nonsigniﬁcant” results or those in the unexpected direction (95% CI: [ 0 . 94 , 1 . 53 ]), with a conﬁdence interval substantially overlapping the null. Comparable estimates were 0.82 for meta-analyses in PLOS One , 1.23 for top medical journals, 1.54 for top psychology journals, and 4.68 for Metalab. We estimated that for 87% of meta-analyses, the amount of publication bias that would be required to attenuate the point estimate to the null exceeded the amount of publication estimated to be actually present in the vast majority of meta-analyses from the relevant scientiﬁc discipline (exceeding the 95 th percentile of publication bias). Study-level measures (“statistical signiﬁcance” with a point estimate in the expected direction and point estimate size) did not indicate more publication bias in higher-tier versus lower-tier journals, nor in the earliest studies published on a topic versus later studies. Overall, the mere act of performing a meta-analysis with a large number of studies (at least 40) and that includes non-headline results may largely mitigate publication bias in meta-analyses, suggesting optimism about the validity of meta-analytic results. scientiﬁc method How much does publication bias compromise the integrity of meta-analysis results? In a systematic, interdisciplinary sample of meta-analyses, we found that “statistically signiﬁcant” results in the expected direction were only 1.20 times more likely to be published than “nonsigniﬁcant” results, and statistical uncertainty was also consistent with no publication bias on average. These estimates ranged from 0.82 to 4.68 for meta-analyses from diﬀerent sources. For the large majority of meta-analyses, conclusions are robust to plausible amounts of publication bias. There seemed to be no more publication bias in individual studies published in higher-tier journals versus lower-tier journals, nor in the earliest studies published on a topic. Our results suggest optimism about the validity of most meta-analytic estimates. 0 . 76 , 1 . 05 ]; p = 0 . 19 ) times as likely to be “signiﬁcant” as results in lower-tier 229 journals and estimated that early results were 0 . 94 (95% CI: [ 0 . 81 , 1 . 08 ]; p = 0 . 38 ) times as likely 230 to be “signiﬁcant” as later results. We also conducted a sensitivity analysis in which we excluded 231 from the “higher-tier” designation a single journal ( Journal of Educational Psychology ) that had 232 contributed 47 % of the higher-tier point estimates. After excluding this journal, higher-tier point 233 estimates appeared less likely than lower-tier point estimates to be aﬃrmative (Supplement). 234