Reviewers’ Self-Awareness and Its Impact on the Peer Review Process

In medical publishing, the peer review process, which is a cornerstone of academic research, is also fundamental to the operation of a scientific journal. Editors rely on reviewers as a source of unbiased and independent expertise that allows them confidentially to appraise submissions on a wide range of topics. Reviewers willingly give their time to the process, generally without remuneration. In return, they may gain early access to information that keeps them abreast of the latest scientific developments in their field of interest, and, more recently, even some academic kudos.1Curry S. Peer review is essential to good science – it's time to credit expert reviewers.https://www.theguardian.com/science/2017/jun/01/peer-review-is-essential-to-good-science-its-time-to-credit-expert-reviewersDate accessed: September 30, 2019Google ScholarMany journals use an anonymous peer review process, whereby the identity of the reviewer is not disclosed to the authors. The influence of anonymity on communication and behaviour has been extensively studied in relation to virtual, computer mediated interactions. In this arena, it has been shown to encourage freedom of expression and disinhibition, but also foster negative, aggressive, or unconstructive criticism of others.2Haines R. Hough J. Cao L. Haines D. Anonymity in computer-mediated communication: more contrarian ideas with less influence.Group Decis Negot. 2014; 23: 765-786Crossref Scopus (31) Google Scholar,3Suler J. The online disinhibition effect.Cyber Psychol Behav. 2004; 7: 321-326Crossref PubMed Scopus (2128) Google Scholar It is thus possible that anonymity may have similar effects on a reviewer acting for a medical journal, although there is no evidence that peer review, in itself, fosters negative, aggressive, or unconstructive criticism.Responsibilities of the reviewerWhen accepting an assignment, a reviewer agrees to perform a task for the editor. Although most reviewers would have received no training in such tasks, there is abundant literature to help them fulfil their obligation to the journal commissioning their expertise.4Publons How to write a peer review: 12 things you need to know.https://publons.com/blog/how-to-write-a-peer-review-12-things-you-need-to-know/Date accessed: September 30, 2019Google Scholar, 5Stiller-Reeve M. How to write a thorough peer review.https://www.nature.com/articles/d41586-018-06991-0Date accessed: September 30, 2019Google Scholar, 6Elsevier The role of a reviewer.https://www.elsevier.com/en-gb/reviewersDate accessed: July 25, 2020Google Scholar, 7The BMJ Resources for reviewers.https://www.bmj.com/about-bmj/resources-reviewersDate accessed: July 25, 2020Google Scholar The primary responsibility of reviewers is to validate the scientific process. Less obviously, perhaps, reviewers also have a professional obligation to the authors of a paper, as their role is not limited to a judgement on suitability for publication, but includes the recognition of modifiable flaws in the research and/or its presentation, and the constructive proposal of modifications that could improve the relevance, resonance, and overall impact of an eventual publication.In our experience as seasoned and not always successful authors, we have generally encountered reviewers who were willing to work with us to improve our papers. Consequently, when our work is rejected, we understand the reasons behind the editors’ decision. On occasion, however, we have been on the receiving end of unconstructive, negative criticism, which we have often found difficult to address, as it was not directed to a specific aspect of our work (“your manuscript is very muddled”, stated the anonymous reviewer of a recent submission of ours). Reassuringly, such reviews are rare and tend to be at odds with the opinion and criticism of the editor and other referees. Nevertheless, in such instances, it is difficult to avoid a temporary feeling of dejection, despite the rational awareness that the opinion stated in the review in question may be unfounded.Inexperienced authors inevitably may be more sensitive to critical comments than their more seasoned peers. For some of these often young researchers, it is possible to envisage that a negative, unconstructive comment on their work may blunt their enthusiasm for further academic practice. For this reason, we believe, a good reviewer should have sufficient self awareness to understand his/her potential impact not only on the submission at hand, but also on the professional progress of its authors.Some journals have adopted an open process, whereby the identity of the reviewers is disclosed to the authors. Some journals even publish the whole peer review process alongside manuscripts. This approach may, however, be counterproductive by reducing the freedom of expression of a reviewer. For example, a junior reviewer may feel obliged to give a senior author a favourable appraisal of his/her submission. Some may even decline an assignment altogether, not willing to be seen to criticise the work of colleagues. This could be a particular issue for papers on very specialised subjects, on which there may only be few experts who know each other well. This potential problem, in particular, may also dissuade editors from abandoning anonymity. Furthermore, it has been shown that single blind reviewers (i.e., referees that are aware of the names and affiliations of the authors, but the whose identity is not disclosed) are more likely than their double blind counterparts to accept review requests for work from well known authors, top universitie, and top companies, and recommend their papers for acceptance.8Tomkins A. Zhang M. Heavlin W.D. Reviewer bias in single- versus double-blind peer review.Proc Natl Acad Sci U S A. 2017; 114: 12708-12713Crossref PubMed Scopus (211) Google ScholarIt should be noted that medical journals, particularly prestigious ones, may reject good quality manuscripts because space for publication is limited and priority is given to better quality or more relevant research. However, occasionally, the vagaries of the peer review process will lead to poor papers being accepted.Feedback mechanisms to inform peer reviewers on the perceived quality of their review could have a positive impact on the process. Such a theoretical framework is supported by research evidence both in academia and in the business world.9Baliram N.S. Youde J.J. A meta-analytic synthesis: examining the academic impacts of feedback on student achievement.IDE J. 2018; 5: 76-90Google Scholar,10Center for Effective OrganizationsPerformance feedback culture drives business impact.https://ceo.usc.edu/wp-content/uploads/2019/07/Performance-Feedback-Culture-Drives-Business-Performance-i4cp-CEO-002-1.pdfDate accessed: October 1, 2019Google Scholar Peer review quality and reviewer performance may improve in a performance feedback culture involving authors and members of the editorial board. A reviewer recognition platform, which also serves as an evaluation forum, has been recently launched and is being embraced by an increasing number of medical journals.11Publons.https://publons.com/about/home/Date accessed: October 1, 2019Google Scholar Feedback mechanisms in medical publishing may result in wider use of relevant courses on peer review provided by relevant publishing bodies and organisations (Table 1). Notably, the European Journal of Vascular and Endovascular Surgery has regularly run reviewer training at meetings of the European Society for Vascular Surgery. Such practices may potentially enhance the peer review process and ultimately the quality of academic research.Table 1Good practice in peer reviewTips for peer review1. Be professional: Take the role of peer reviewer seriously, work timely and deliver a professional scientific work.2. Be kind: Be considerate and respectful of colleagues'/peers' scientific work and treat it as if it were your own team's work.3. Be thorough: Provide general and specific comments in the context of the entire existing scientific evidence, after you perform your own literature researches. The report should be stratified into the following distinct sections: general comments, major specific comments, and minor comments4. Be constructive: Provide helpful, specific suggestions on how to improve the scientific work. Challenge the authors to answer questions that will be asked by the readership.5. Be realistic: Bearing in mind that you are invited to peer review a completed scientific work, refrain from making unrealistic suggestions pertaining to peer review of a study protocol and focus on pragmatic, realistic changes that would enhance the value of the work.6. Provide positive feedback: Do highlight strengths of the study with relevant arguments to support such statements.7. Be organised: Approach the reviewed work methodically commenting on the scientific context, research methodology/methods of data analysis, presentation of results, and interpretation of study findings.8. Be aware of your own biases: Unconscious bias may range from your own approaches to and preferences for clinical management to being aware of the authors' or the author institution's previous scientific work.9. Acknowledge your own limitations: If you are unfamiliar with one or more aspects of the study, for instance the statistics or risk of bias assessment, say so explicitly.10. Be a collaborator: See your role in the author's scientific work as collaborative rather than authoritative.Pitfalls1. Sarcasm2. Being argumentative3. Personal criticism4. Profanity5. Bias6. Conflicts of interests Open table in a new tab It remains unclear whether peer review improves the scientific quality of a journal's content.12Righi S. Takács K. The miracle of peer review and development in science: an agent-based model.Scientometrics. 2017; 113: 587-607Crossref PubMed Scopus (20) Google Scholar Nevertheless, the process is here to stay, at least while scientific journals exist in their present form, as an alternative mechanism does not appear to exist. To reviewers, we therefore remind that, whenever possible, any comment disclosed to the authors should be constructive, even when rejection is recommended (Table 2). The responsibility of the editors may need to expand to “review the reviewers”, i.e., remove unhelpful commentary, or perhaps even provide feedback to reviewers if they have included noticeably unconstructive comments. Authors on the receiving end of bad, unconstructive reviews should not be discouraged, as these are par for the course: they should take a deep breath and either move on to the next journal, or politely explain to the editor that they would gladly respond to constructive criticism, should any be forthcoming.Table 2Peer review coursesPublons Academy: https://publons.com/community/academy/Elsevier Researcher Academy: https://researcheracademy.elsevier.com/navigating-peer-review/certified-peer-reviewer-courseSpringer's “How to peer review” tutorial: https://www.springer.com/gp/authors-editors/authorandreviewertutorials/howtopeerreview Open table in a new tab In medical publishing, the peer review process, which is a cornerstone of academic research, is also fundamental to the operation of a scientific journal. Editors rely on reviewers as a source of unbiased and independent expertise that allows them confidentially to appraise submissions on a wide range of topics. Reviewers willingly give their time to the process, generally without remuneration. In return, they may gain early access to information that keeps them abreast of the latest scientific developments in their field of interest, and, more recently, even some academic kudos.1Curry S. Peer review is essential to good science – it's time to credit expert reviewers.https://www.theguardian.com/science/2017/jun/01/peer-review-is-essential-to-good-science-its-time-to-credit-expert-reviewersDate accessed: September 30, 2019Google Scholar Many journals use an anonymous peer review process, whereby the identity of the reviewer is not disclosed to the authors. The influence of anonymity on communication and behaviour has been extensively studied in relation to virtual, computer mediated interactions. In this arena, it has been shown to encourage freedom of expression and disinhibition, but also foster negative, aggressive, or unconstructive criticism of others.2Haines R. Hough J. Cao L. Haines D. Anonymity in computer-mediated communication: more contrarian ideas with less influence.Group Decis Negot. 2014; 23: 765-786Crossref Scopus (31) Google Scholar,3Suler J. The online disinhibition effect.Cyber Psychol Behav. 2004; 7: 321-326Crossref PubMed Scopus (2128) Google Scholar It is thus possible that anonymity may have similar effects on a reviewer acting for a medical journal, although there is no evidence that peer review, in itself, fosters negative, aggressive, or unconstructive criticism. Responsibilities of the reviewerWhen accepting an assignment, a reviewer agrees to perform a task for the editor. Although most reviewers would have received no training in such tasks, there is abundant literature to help them fulfil their obligation to the journal commissioning their expertise.4Publons How to write a peer review: 12 things you need to know.https://publons.com/blog/how-to-write-a-peer-review-12-things-you-need-to-know/Date accessed: September 30, 2019Google Scholar, 5Stiller-Reeve M. How to write a thorough peer review.https://www.nature.com/articles/d41586-018-06991-0Date accessed: September 30, 2019Google Scholar, 6Elsevier The role of a reviewer.https://www.elsevier.com/en-gb/reviewersDate accessed: July 25, 2020Google Scholar, 7The BMJ Resources for reviewers.https://www.bmj.com/about-bmj/resources-reviewersDate accessed: July 25, 2020Google Scholar The primary responsibility of reviewers is to validate the scientific process. Less obviously, perhaps, reviewers also have a professional obligation to the authors of a paper, as their role is not limited to a judgement on suitability for publication, but includes the recognition of modifiable flaws in the research and/or its presentation, and the constructive proposal of modifications that could improve the relevance, resonance, and overall impact of an eventual publication.In our experience as seasoned and not always successful authors, we have generally encountered reviewers who were willing to work with us to improve our papers. Consequently, when our work is rejected, we understand the reasons behind the editors’ decision. On occasion, however, we have been on the receiving end of unconstructive, negative criticism, which we have often found difficult to address, as it was not directed to a specific aspect of our work (“your manuscript is very muddled”, stated the anonymous reviewer of a recent submission of ours). Reassuringly, such reviews are rare and tend to be at odds with the opinion and criticism of the editor and other referees. Nevertheless, in such instances, it is difficult to avoid a temporary feeling of dejection, despite the rational awareness that the opinion stated in the review in question may be unfounded.Inexperienced authors inevitably may be more sensitive to critical comments than their more seasoned peers. For some of these often young researchers, it is possible to envisage that a negative, unconstructive comment on their work may blunt their enthusiasm for further academic practice. For this reason, we believe, a good reviewer should have sufficient self awareness to understand his/her potential impact not only on the submission at hand, but also on the professional progress of its authors.Some journals have adopted an open process, whereby the identity of the reviewers is disclosed to the authors. Some journals even publish the whole peer review process alongside manuscripts. This approach may, however, be counterproductive by reducing the freedom of expression of a reviewer. For example, a junior reviewer may feel obliged to give a senior author a favourable appraisal of his/her submission. Some may even decline an assignment altogether, not willing to be seen to criticise the work of colleagues. This could be a particular issue for papers on very specialised subjects, on which there may only be few experts who know each other well. This potential problem, in particular, may also dissuade editors from abandoning anonymity. Furthermore, it has been shown that single blind reviewers (i.e., referees that are aware of the names and affiliations of the authors, but the whose identity is not disclosed) are more likely than their double blind counterparts to accept review requests for work from well known authors, top universitie, and top companies, and recommend their papers for acceptance.8Tomkins A. Zhang M. Heavlin W.D. Reviewer bias in single- versus double-blind peer review.Proc Natl Acad Sci U S A. 2017; 114: 12708-12713Crossref PubMed Scopus (211) Google ScholarIt should be noted that medical journals, particularly prestigious ones, may reject good quality manuscripts because space for publication is limited and priority is given to better quality or more relevant research. However, occasionally, the vagaries of the peer review process will lead to poor papers being accepted.Feedback mechanisms to inform peer reviewers on the perceived quality of their review could have a positive impact on the process. Such a theoretical framework is supported by research evidence both in academia and in the business world.9Baliram N.S. Youde J.J. A meta-analytic synthesis: examining the academic impacts of feedback on student achievement.IDE J. 2018; 5: 76-90Google Scholar,10Center for Effective OrganizationsPerformance feedback culture drives business impact.https://ceo.usc.edu/wp-content/uploads/2019/07/Performance-Feedback-Culture-Drives-Business-Performance-i4cp-CEO-002-1.pdfDate accessed: October 1, 2019Google Scholar Peer review quality and reviewer performance may improve in a performance feedback culture involving authors and members of the editorial board. A reviewer recognition platform, which also serves as an evaluation forum, has been recently launched and is being embraced by an increasing number of medical journals.11Publons.https://publons.com/about/home/Date accessed: October 1, 2019Google Scholar Feedback mechanisms in medical publishing may result in wider use of relevant courses on peer review provided by relevant publishing bodies and organisations (Table 1). Notably, the European Journal of Vascular and Endovascular Surgery has regularly run reviewer training at meetings of the European Society for Vascular Surgery. Such practices may potentially enhance the peer review process and ultimately the quality of academic research.Table 1Good practice in peer reviewTips for peer review1. Be professional: Take the role of peer reviewer seriously, work timely and deliver a professional scientific work.2. Be kind: Be considerate and respectful of colleagues'/peers' scientific work and treat it as if it were your own team's work.3. Be thorough: Provide general and specific comments in the context of the entire existing scientific evidence, after you perform your own literature researches. The report should be stratified into the following distinct sections: general comments, major specific comments, and minor comments4. Be constructive: Provide helpful, specific suggestions on how to improve the scientific work. Challenge the authors to answer questions that will be asked by the readership.5. Be realistic: Bearing in mind that you are invited to peer review a completed scientific work, refrain from making unrealistic suggestions pertaining to peer review of a study protocol and focus on pragmatic, realistic changes that would enhance the value of the work.6. Provide positive feedback: Do highlight strengths of the study with relevant arguments to support such statements.7. Be organised: Approach the reviewed work methodically commenting on the scientific context, research methodology/methods of data analysis, presentation of results, and interpretation of study findings.8. Be aware of your own biases: Unconscious bias may range from your own approaches to and preferences for clinical management to being aware of the authors' or the author institution's previous scientific work.9. Acknowledge your own limitations: If you are unfamiliar with one or more aspects of the study, for instance the statistics or risk of bias assessment, say so explicitly.10. Be a collaborator: See your role in the author's scientific work as collaborative rather than authoritative.Pitfalls1. Sarcasm2. Being argumentative3. Personal criticism4. Profanity5. Bias6. Conflicts of interests Open table in a new tab It remains unclear whether peer review improves the scientific quality of a journal's content.12Righi S. Takács K. The miracle of peer review and development in science: an agent-based model.Scientometrics. 2017; 113: 587-607Crossref PubMed Scopus (20) Google Scholar Nevertheless, the process is here to stay, at least while scientific journals exist in their present form, as an alternative mechanism does not appear to exist. To reviewers, we therefore remind that, whenever possible, any comment disclosed to the authors should be constructive, even when rejection is recommended (Table 2). The responsibility of the editors may need to expand to “review the reviewers”, i.e., remove unhelpful commentary, or perhaps even provide feedback to reviewers if they have included noticeably unconstructive comments. Authors on the receiving end of bad, unconstructive reviews should not be discouraged, as these are par for the course: they should take a deep breath and either move on to the next journal, or politely explain to the editor that they would gladly respond to constructive criticism, should any be forthcoming.Table 2Peer review coursesPublons Academy: https://publons.com/community/academy/Elsevier Researcher Academy: https://researcheracademy.elsevier.com/navigating-peer-review/certified-peer-reviewer-courseSpringer's “How to peer review” tutorial: https://www.springer.com/gp/authors-editors/authorandreviewertutorials/howtopeerreview Open table in a new tab When accepting an assignment, a reviewer agrees to perform a task for the editor. Although most reviewers would have received no training in such tasks, there is abundant literature to help them fulfil their obligation to the journal commissioning their expertise.4Publons How to write a peer review: 12 things you need to know.https://publons.com/blog/how-to-write-a-peer-review-12-things-you-need-to-know/Date accessed: September 30, 2019Google Scholar, 5Stiller-Reeve M. How to write a thorough peer review.https://www.nature.com/articles/d41586-018-06991-0Date accessed: September 30, 2019Google Scholar, 6Elsevier The role of a reviewer.https://www.elsevier.com/en-gb/reviewersDate accessed: July 25, 2020Google Scholar, 7The BMJ Resources for reviewers.https://www.bmj.com/about-bmj/resources-reviewersDate accessed: July 25, 2020Google Scholar The primary responsibility of reviewers is to validate the scientific process. Less obviously, perhaps, reviewers also have a professional obligation to the authors of a paper, as their role is not limited to a judgement on suitability for publication, but includes the recognition of modifiable flaws in the research and/or its presentation, and the constructive proposal of modifications that could improve the relevance, resonance, and overall impact of an eventual publication. In our experience as seasoned and not always successful authors, we have generally encountered reviewers who were willing to work with us to improve our papers. Consequently, when our work is rejected, we understand the reasons behind the editors’ decision. On occasion, however, we have been on the receiving end of unconstructive, negative criticism, which we have often found difficult to address, as it was not directed to a specific aspect of our work (“your manuscript is very muddled”, stated the anonymous reviewer of a recent submission of ours). Reassuringly, such reviews are rare and tend to be at odds with the opinion and criticism of the editor and other referees. Nevertheless, in such instances, it is difficult to avoid a temporary feeling of dejection, despite the rational awareness that the opinion stated in the review in question may be unfounded. Inexperienced authors inevitably may be more sensitive to critical comments than their more seasoned peers. For some of these often young researchers, it is possible to envisage that a negative, unconstructive comment on their work may blunt their enthusiasm for further academic practice. For this reason, we believe, a good reviewer should have sufficient self awareness to understand his/her potential impact not only on the submission at hand, but also on the professional progress of its authors. Some journals have adopted an open process, whereby the identity of the reviewers is disclosed to the authors. Some journals even publish the whole peer review process alongside manuscripts. This approach may, however, be counterproductive by reducing the freedom of expression of a reviewer. For example, a junior reviewer may feel obliged to give a senior author a favourable appraisal of his/her submission. Some may even decline an assignment altogether, not willing to be seen to criticise the work of colleagues. This could be a particular issue for papers on very specialised subjects, on which there may only be few experts who know each other well. This potential problem, in particular, may also dissuade editors from abandoning anonymity. Furthermore, it has been shown that single blind reviewers (i.e., referees that are aware of the names and affiliations of the authors, but the whose identity is not disclosed) are more likely than their double blind counterparts to accept review requests for work from well known authors, top universitie, and top companies, and recommend their papers for acceptance.8Tomkins A. Zhang M. Heavlin W.D. Reviewer bias in single- versus double-blind peer review.Proc Natl Acad Sci U S A. 2017; 114: 12708-12713Crossref PubMed Scopus (211) Google Scholar It should be noted that medical journals, particularly prestigious ones, may reject good quality manuscripts because space for publication is limited and priority is given to better quality or more relevant research. However, occasionally, the vagaries of the peer review process will lead to poor papers being accepted. Feedback mechanisms to inform peer reviewers on the perceived quality of their review could have a positive impact on the process. Such a theoretical framework is supported by research evidence both in academia and in the business world.9Baliram N.S. Youde J.J. A meta-analytic synthesis: examining the academic impacts of feedback on student achievement.IDE J. 2018; 5: 76-90Google Scholar,10Center for Effective OrganizationsPerformance feedback culture drives business impact.https://ceo.usc.edu/wp-content/uploads/2019/07/Performance-Feedback-Culture-Drives-Business-Performance-i4cp-CEO-002-1.pdfDate accessed: October 1, 2019Google Scholar Peer review quality and reviewer performance may improve in a performance feedback culture involving authors and members of the editorial board. A reviewer recognition platform, which also serves as an evaluation forum, has been recently launched and is being embraced by an increasing number of medical journals.11Publons.https://publons.com/about/home/Date accessed: October 1, 2019Google Scholar Feedback mechanisms in medical publishing may result in wider use of relevant courses on peer review provided by relevant publishing bodies and organisations (Table 1). Notably, the European Journal of Vascular and Endovascular Surgery has regularly run reviewer training at meetings of the European Society for Vascular Surgery. Such practices may potentially enhance the peer review process and ultimately the quality of academic research. It remains unclear whether peer review improves the scientific quality of a journal's content.12Righi S. Takács K. The miracle of peer review and development in science: an agent-based model.Scientometrics. 2017; 113: 587-607Crossref PubMed Scopus (20) Google Scholar Nevertheless, the process is here to stay, at least while scientific journals exist in their present form, as an alternative mechanism does not appear to exist. To reviewers, we therefore remind that, whenever possible, any comment disclosed to the authors should be constructive, even when rejection is recommended (Table 2). The responsibility of the editors may need to expand to “review the reviewers”, i.e., remove unhelpful commentary, or perhaps even provide feedback to reviewers if they have included noticeably unconstructive comments. Authors on the receiving end of bad, unconstructive reviews should not be discouraged, as these are par for the course: they should take a deep breath and either move on to the next journal, or politely explain to the editor that they would gladly respond to constructive criticism, should any be forthcoming.