Open Review for Open Science? Assessment of Methodological Problems and Quality in Journal Peer Review
Creators: name: Karhulahti, Veli-Matti
name: Backe, Hans-Joachim
Publisher: name: Open Science Framework
Year: 2022
Subjects: subject: Arts and Humanities
subject: Social and Behavioral Sciences
subject: Journalology
subject: Peer review
subject: Qualitative

description: For decades, a key topic in journalology – study of the academic publishing process – has been the peer review process: how to verify the scientific quality of articles in a way that is ethical, reliable, and possible to carry out in practice? (E.g. Burnham 1990; Smith 1997; Bornmann 2008.) Arguably, the foremost current conversations of this issue align with the open science dialogue and, in particular, the question whether journals should employ closed or open peer review practices (e.g. Ford 2013; Moylan et al. 2014; Manchikanti et al. 2015). In general, the consensus seems to be that fully closed peer practices are considered good for providing reviewers a safe anonymous space to criticize honestly and facilitating the speed of the review process, whereas fully open review processes are considered good for making the process transparent and motivating the reviewers to take responsibility of their critique, thus increasing the quality of the feedback overall (e.g. Leek et al. 2011; Pöschl 2012; Tomkins et al. 2017; Walsh et al. 2000). In writing on open research, one finds suggestions for universal regulations and practices, with social sciences and humanities (SSH) often singled out as potentially requiring different solutions (Nosek et al. 2015; Morey et al. 2016). A facile view of the issue might be to consider the acceptance of open review practices as determined by discipline. Yet even within strongly data-driven disciplines, e.g. medicine, the debate about the advantages and disadvantages of open review processes continues (Groves 2010; Khan 2010; van den Eynden et al. 2016). At the same time, some social science research vocally champions open research (Vuong 2017) and review practices (Risam 2014). The challenges in developing universally applicable procedures for open research and review appear to be at least partially rooted in conceptual differences (such as the meaning of originality) (Guetzkow et al. 2004) and resulting methodologies (Peels &amp; Bouter 2018). In order to better understand how open science, here as the proliferating call for open peer review processes (see Shanahan &amp; Olsen 2014; Morey et al. 2016), impacts humanities and social sciences, we aim at twelve qualitative semi-structured interviews with chief editors of highly ranked academic humanities and social science journals. [Note: here we use the term "highly ranked" to indicate that the journal in question is respected in its own field and is being recognized as a quality publication platform among the scientists of its research area.] Three research questions guide the study. RQ1: In what way do highly ranked humanities and social science journals perceive of open peer review processes and how do these perceptions materialize in the actual processes they employ? We hypothesize (H1) a correlation between dominant approaches and review practices, with qualitatively oriented journals preferring closed and quantitatively oriented journals preferring open review practices. RQ2: How do the chief editors position their journal’s review and publication practice between policy, ethics, and pragmatism? We hypothesize (H2) that smaller and younger publications adhere more strongly to changing standards in publication practices and enforce open practices more vehemently than bigger and more established journals. RQ3: How do the chief editors situate their own powerful role in the peer review process and in what way is that role negotiating open science principles of the peer review process? We hypothesize (H3) the degree of adaptation of open review practices directly correlating with the role of the chief editor in a given journal, where empowered editors in hierarchical structures will prefer closed practices. References Bornmann, L. (2008). Scientiﬁc peer review: An analysis of the peer review process from the perspective of sociology of science theories. Human Architecture: Journal of the Sociology of Self-Knowledge, 6(2), 3. Burnham, John C. (1990). "The Evolution of Editorial Peer Review". JAMA: The Journal of the American Medical Association. 263 (10): 1323 Ford, E. (2013). Defining and characterizing open peer review: A review of the literature. Journal of Scholarly Publishing, 44(4), 311-326. Groves, T. (2010). “Is Open Peer Review the Fairest System? Yes.” BMJ (Clinical research ed.) 341:c6424. doi:10.1136/bmj.c6424. Guetzkow, J., Lamont, M. &amp; Mallard, G. (2004). “What Is Originality in the Humanities and the Social Sciences?” Am Sociol Rev 69 (2): 190–212. doi:10.1177/000312240406900203. Khan, K. (2010). “Is Open Peer Review the Fairest System? No.” BMJ (Clinical research ed.) 341:c6425. doi:10.1136/bmj.c6425. Leek, J. T., Taub, M. A., &amp; Pineda, F. J. (2011). Cooperation between referees and authors increases peer review accuracy. PloS one, 6(11). Manchikanti, L., Kaye, A. D., Boswell, M. V., &amp; Hirsch, J. A. (2015). Medical journal peer review: Process and bias. Pain Physician, 18(1), E1 Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D., Lewandowsky, S. et al. (2016). “The Peer Reviewers' Openness Initiative: Incentivizing Open Research Practices Through Peer Review.” Royal Society open science 3 (1): 150547. doi:10.1098/rsos.150547. Smith Richard. Peer review: reform or revolution? Time to open up the black box of peer review.” BMJ 1997; 315 :759. Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D., ... &amp; Vanpaemel, W. (2016). The Peer Reviewers' Openness Initiative: incentivizing open research practices through peer review. Royal Society Open Science, 3(1), 150547. Moylan, E. C., Harold, S., O’Neill, C., &amp; Kowalczuk, M. K. (2014). Open, single-blind, double-blind: which peer review process do you prefer?. Nosek, B. A., Alter, G., Banks, G. C. , Borsboom, D. , Bowman, S. D., Breckler, S. J., Buck, S. et al. (2015). “Promoting an Open Research Culture.” Science (New York, N.Y.) 348 (6242): 1422–25. doi:10.1126/science.aab2374. Peels, R., Bouter, L. (2018). “The Possibility and Desirability of Replication in the Humanities.” Palgrave Commun 4 (1). doi:10.1057/s41599-018-0149-x. Pöschl, U. (2012). Multi-stage open peer review: scientific evaluation integrating the strengths of traditional peer review with the virtues of transparency and self-regulation. Frontiers in computational neuroscience, 6, 33. Risam, R. (2014). “Rethinking Peer Review in the Age of Digital Humanities.” Ada: A Journal of Gender, New Media, and Technology, Number 4 (4). doi:10.7264/N3WQ0220. Shanahan, D. R., &amp; Olsen, B. R. (2014). Opening peer-review: the democracy of science. Journal of Negative Results in Biomedicine. Tomkins, A., Zhang, M., &amp; Heavlin, W. D. (2017). Reviewer bias in single-versus double-blind peer review. Proceedings of the National Academy of Sciences, 114(48), 12708-12713. van den Eynden, V., Knight, G., Vlad, A., Radler, B., Tenopir, C., Leon, D., Manista, F., Whitworth, J., Corti, L. (2016). “Survey of Wellcome Researchers and Their Attitudes to Open Research.” https://doi.org/10.6084/m9.figshare.4055448.v1 Vuong, Q. H. (2017). “Open Data, Open Review and Open Dialogue in Making Social Sciences Plausible.” Nature: Scientific Data Updates, 1–5. Walsh, E., Rooney, M., Appleby, L., &amp; Wilkinson, G. (2000). Open peer review: a randomised controlled trial. The British Journal of Psychiatry, 176(1), 47-51.