How should journals handle scientific criticism?
Channel: Center for Open Science
Date: 2023-05-26

Session: How should journals handle scientific criticism?

Speaker: Tom Hardwicke, Research Fellow, University of Melbourne

Abstract: Journals exert considerable control over criticism of prior research submitted in the form of letters, commentaries, or online comments (post-publication critique). Our recent study of 330 top-ranked journals across 22 scientific disciplines found that many (37%) do not accept post-publication critique at all and those that do sometimes impose strict limits on length (e.g., 175 words) and time-to-submit (e.g., 2 weeks). In practice, many journals that accepted post-publication critique in principle rarely published them in practice (~1.9% of 2066 randomly selected empirical articles published in 2018 were associated with at least one critique). While most critiques received an author reply (44/58), authors typically stated that their original conclusions remained unchanged (41/44). There was considerable intra-disciplinary variation, with medical journals publishing the most critiques, but also imposing the strictest length and time-to-submit limits. We suggest several journal policies that may facilitate the cultivation, documentation, and dissemination of post-publication critique.