Reviewing Peer Review
Current Status, Future Challenges
Asia Pacific Association of Medical Journal Editors
APAME Convention 2013
Japan Medical Association Building, Tokyo, Japan, August 4, 2013

JF Lapeña Jr. M.A., M.D., F.P.C.S.
lapenajf@upm.edu.phReviewing Peer Review
I acknowledge the generous support of the JAMJE Organizing
Committee to attend this conference.
I believe I have no financial or other (including personal)
relationships, intellectual passion, political or religious beliefs,
and institutional affiliations that might lead to a conflict ofinterest in making this presentation.

DISCLOSURESReviewing Peer Review
This presentation is based on:
Dominy, Peggy and Jay Bhatt. “Peer Review in the Google Age: Is technology
changing the way science is done and evaluated?” Cited July 13, 2013. Available
from: eprints.rclis.org/7411/1/Peer_Review_in_the_Google_Age%5B1%5D.ppt‎
Eklof, Tony. “Is Peer Review Peerless?” James Joyce Library. University of Dublin.
LIR Seminar 2009. LIR HEAnet User Group for Libraries. Cited July 13, 2013.
Available from:lirgroup.heanet.ie/sites/default/files/Seminars/2009/Tony%20Eklof.ppt
Siegel , Vivian. “Reviewing Peer Review” Departments of Medicine and Cell andDevelopmental Biology, Vanderbilt University. Cited July 13, 2013. Available from:medicine.mc.vanderbilt.edu/sites/.../reviewing%20peer%20review%204.ppt

REFERENCES

Reviewing Peer Review
This presentation is based on:
Rosenfeld, Richard M. How to review journal manuscripts. Otolaryngology –Head and Neck Surgery (2010) 142, 472-486
Pöschl, Ulrich. Multi-stage open peer review: scientific evaluation integrating
the strengths of traditional peer review with the virtues of transparency andself-regulation. Frontiers in computational neuroscience. (2012) 6, 33: 1-16.

REFERENCES

Reviewing Peer Review
WHY do you Publish?
• To document or claim priority for a discovery• To share what you have learned and how you havelearned it, so that others can build on that knowledge
HOW do you Publish?
• Through peer reviewed journals of the highest
possible stature
Siegel, 2008.

Reviewing Peer Review
WHY do journals use Peer Review?
• To determine whether conclusions drawn by the research
are justified and‎“ new”
• To get some gauge as to the significance or potential
significance of the work
• Adds‎a‎“ human judgment “‎ element to the academic
process and assists/lessens workload of the Editor• Ideally, this is then turned into a decision that isconstructive, transparent, timely, and fair and will
enhance the final writing product
Rosenfeld 2010; Eklof, 2009; Siegel, 2008.

Reviewing Peer Review
WHY do journals use Peer Review?
• Quality control for scholarly information; Filter
More‎ papers‎ submitted‎than‎coul d‎ be‎“pr inted”
Weeds‎out‎ fraud‎a nd‎ eliminates‎ “bad”‎science,‎ pseudoscience, harmful science; upholds ethical standards• Promotes originality, academic rigour and improves
critical thinking/writing skills of editors, reviewers,authors (mechanism for improvement of manuscripts)• Reduce bias and improve quality of published articles• Collegial stamp of approval• Professional obligation to‎the‎ principles‎ of‎o ne’s‎ discipline
Rosenfeld 2010; Eklof, 2009; Dominy & Bhatt, 2006

Reviewing Peer Review“Medieval” Peer Review
Some‎ would‎ say‎t hat‎ “Peer‎Review”‎ goes‎ back‎as‎f ar‎ as‎
the 17th century, when it wasknown‎ as‎“The ‎I nquisition‎ of‎the Holy Roman and CatholicChurch”.‎ ‎Scholars’‎ works‎were‎
examined for any hints of“heresy”.

Dominy & Bhatt, 2006. From Wikipedia
http://en.wikipedia.org/wiki/Peer_review

Reviewing Peer Review“Modern” Peer Review

Peer review (“refereeing” in someacademic fields) is used in:
1. Ethics / Institutional Approval &Publication Process
2. Awarding of Research Funding
3. Patents and IP Deliberations
4. Standards and Guidelines

Each involves slightly different
practices, but ultimately
colleagues are evaluating eachother.

Dominy & Bhatt, 2006

Reviewing Peer Review
From Start to Finish:
Typical Linear Flow of the Scholarly Publishing Process
Author submits manuscript* to academic journal editor
Editor determines whether manuscript has sufficient merit to bereviewed by editorial board or selected external reviewers
Manuscript sent back to the author with a rejection letter or on toreviewers (typically, a double-blind process)*after complying with institutional /ethical review boards, etc
Publish, Not Perish: The Art and Craft of Publishing in Scholarly Journals.
University of Colorado 2006. Available from http://www.publishnotperish.org

Reviewing Peer Review
From Start to Finish:
Typical Linear Flow of the Scholarly Publishing Process
Reviewers return the manuscript to the editor with comments andrecommendations (depending on peer review model)
Editor sends manuscript back to the author with either a rejection
letter or a request for revisions in compliance with reviews
Author revises manuscript and resubmits to editor
Editor (sometimes) sends revised manuscript back to external
reviewers
Publish, Not Perish: The Art and Craft of Publishing in Scholarly Journals.
University of Colorado 2006. Available from http://www.publishnotperish.org

Reviewing Peer ReviewFrom Start to Finish:
Typical Linear Flow of the Scholarly Publishing Process
Editor accepts or rejects manuscript
Copy Editor edits the manuscript and galley proofs are laid out (injournals with long queues, this may involve updating revisions)
Author provides editing or proofing of final copy before publication
Paper is eventually published in journal
Journal is disseminated in print and electronic forms
Publish, Not Perish: The Art and Craft of Publishing in Scholarly Journals. Universityof Colorado 2006. Available from http://www.publishnotperish.org

Reviewing Peer Review
SO, WHAT’S THE PROBLEM?

Famous papers that were published and did NOT get peer
reviewed:

Siegel, 2008; Dominy & Bhatt, 2006.

Reviewing Peer Review
SO, WHAT’S THE PROBLEM?

Famous papers that were published and did NOT get peer
reviewed:
Watson & Crick’s 1951 paper on the structure of DNA in
NatureAbdus Salam’s paper‎ “Weak‎an d‎e lectromagnetic‎
interactions”‎(19 68).‎ Led‎ to‎No bel‎Priz e
Alan Sokal’s “Transgressing‎ the‎Bou ndaries...”‎ in‎19 96‎
turned out to be a hoax. Now known as the Sokal Affair.

Eklof, 2009; Siegel, 2008; Dominy & Bhatt, 2006.

Reviewing Peer Review
SO, WHAT’S THE PROBLEM?

INTRODUCTION

Dominy & Bhatt, 2006

Reviewing Peer Review
SO, WHAT’S THE PROBLEM?

Famous papers that were published and passed peer review
that later proved to be fraudulent:
Jan Hendrik Schon of Bell Labs submitted and passedpeer review 15 papers published in Science and Nature(1998-2001) found to be fraudulent.
Igor and Grichka Bogdanov 1999 & 2002 publishedpapers in theoretical physics believed by many to be jargonrich nonsense.

Siegel, 2008. Dominy & Bhatt, 2006

Reviewing Peer ReviewSO, WHAT’S THE PROBLEM?

Famous papers that were published and passed peer review
that later proved to be fraudulent:
Yoshitaka Fujii fabricated data in 172/249 papers.
Beginning his career in falsification in 1993 while at theTokyo Medical and Dental University, he continued it at theUniversity of Tsukuba, and at Toho University in Tokyo,
where he was finally dismissed in February 2012.
Diederik Stapel was found to have published fabricateddata in 30 peer-reviewed social psychology papers from hisposts in the University of Groningen and Tilburg University.

Reviewing Peer Review
SO, WHAT’S THE PROBLEM?

Famous papers that were published and passed peer review
that later proved to be fraudulent:
Woo Suk Hwang of Seoul National University was the centerof one of the biggest investigations of scientific fraud, with theretractions of two landmark papers in stem cell research:• Patient-Specific Embryonic Stem Cells Derived from Human SCNT
Blastocysts. W. S. Hwang et al. Science 308,1777-1783; 2005
• Evidence of a Pluripotent Human Embryonic Stem Cell Line
Derived from a Cloned Blastocyst. W. S. Hwang et al. Science 303,1669-1674; 2004

Reviewing Peer Review
SO, WHAT’S THE PROBLEM?

Famous papers that got rejected that later turned out to be
seminal works:
Krebs & Johnson’s 1937 paper on the role of citric acid on
metabolism,‎ rejected‎ by‎ Nature‎as‎b eing‎ of‎“insuf ficient‎
importance”,‎w as‎e ventually‎pub lished‎ in‎t he‎ Dutch‎j ournal‎
Enzymologia. This discovery, now known as the Krebs Cycle,
was recognized with a Nobel prize in 1953.
Black & Scholes 1973‎pa per‎on ‎ “the‎pricing ‎ of‎op tions‎ and‎corporate‎ liabilities”,‎ rejected‎m any‎tim es,‎ was‎ev entually‎
published at the intersession of Merton Miller to get it accepted bythe Journal of Political Economy. This work led to the Nobel Prize
Siegel, 2008. Dominy & Bhatt, 2006

Reviewing Peer Review
DOES PEER REVIEW WORK?
• Reviews often fail to address technical quality of the work
• Reviews are often dismissive• Reviewers often seem to ask for additional work justbecause they can or they think they should• Editors often insist on additional experiments that‎wo n’t‎change the paper• If a paper is rejected, the whole process has to be started
over again,‎ because‎ (in‎ge neral)‎ the‎ reviews‎ can’t‎ be‎used at another journal
Siegel, 2008

Reviewing Peer Review
It is ironic that, in an era known for the great
speed and availability of information -- wherewe could choose to blog our results rather thansubmit them to journals -- publishing papers
seems slower and more painful than ever before.

Siegel, “The Promise of Peer Review”, DMM, 2008

Reviewing Peer Review
The traditional ways of scientific publishing andpeer review do not live up to the needs ofefficient communication and quality assurance intoday’s highly diverse and rapidly developingworld of science.

Pöschl, 2004

Reviewing Peer Review
Besides high profile cases of scientific fraud,science, and society are facing a flood ofcarelessly prepared scientific papers that arelocked away behind subscription barriers, dilute
rather than enhance scientific knowledge, lead toa waste of resources and impede scientific andsocietal progress.
Pöschl, 2004

Reviewing Peer Review
On the other hand, the spread of innovative ideas
and concepts is often delayed by inertia andobstruction in the hidden review process of
traditional mainstream scientific journals.

Pöschl, 2004

Reviewing Peer Review
DOES PEER REVIEW WORK?
• Slows down the research process• Time consuming for reviewers• Difficult to find qualified experts for some manuscripts orproposals• Bias of reviewers• Arbitrary, secret and subjective!
• Process can frustrate researchers; valuable time spent on‘lateral’‎ research
Eklof, 2009; Dominy & Bhatt, 2006

Reviewing Peer Review
DOES PEER REVIEW WORK?
Is Peer Review Broken?"
McCook, Alison. The Scientist - Magazine of the Life
Sciences 2006 Feb; 20(2):26. Cited July 18, 2013.
Available from: http://www.the-scientist.com/2006/2/1/26/1/
Submissions are up, reviewers are overtaxed, and authors
are lodging complaint after complaint about the process at
top-tier journals. What's wrong with peer review?
Eklof, 2009; Dominy & Bhatt, 2006

Reviewing Peer Review
NATURE’S PEER REVIEW TRIAL"Journal lays bare remarks from peer reviewers"
Marris, Emma. Nature 2006 Feb; 439(9):642 Cited July18, 2013; Available from:http://www.nature.com/nature/journal/v439/n7077/full/439
642b.html
Cloak of anonymity shed by new publication. Editors of ajournal launched this week are out to revolutionize peer
review. By publishing signed reviews alongside papers,
they hope to make the process more transparent andimprove the quality of the articles.
Dominy & Bhatt, 2006

Reviewing Peer Review

Reviewing Peer Review
SO, WHAT’S CHANGED?
Papers‎ can‎be‎“publ ished”‎ on‎the‎ Web‎ without theconstraints (peer reviewers and editors) of traditional“journals”.
Papers‎ can‎be‎“publ ished”‎ on‎the‎ Web‎ with the constraints(peer‎r eviewers‎ and‎ editors)‎of ‎tradi tional‎ “journals”.
Is this a good thing or a bad thing?
Dominy & Bhatt, 2006

Reviewing Peer Review
WITHOUT CONSTRAINTS• Wild and Open—no gatekeepers—no censors?
• Will the good stuff filter to the top?
• Discernment—how‎ do‎you‎kno w‎ if‎ it’s‎goo d?
• Who’s‎t he‎authority?

Dominy & Bhatt, 2006

INTRODUCTION
Reviewing PeerReview
WITH CONSTRAINTS• Is the good stuff really vetted?
• Is scholarship really monitored and maintained?
• Are availability and access assured?
• Publishers‎ “sharing”‎ content‎w ith‎ the‎ internet‎( a‎ short‎ list):•American Physical Society•American Mathematical Society•BioMed Central•Blackwell•BMJ•Cambridge University Press•IEEE•MIT Press

•ACM
•AIP
•IOP
•Public Library of Science•Royal Society of Chemistry•Kluwer•PNAS

Dominy & Bhatt, 2006

INTRODUCTION
Reviewing Peer
Review
OPEN ACCESS AND PEER REVIEW(1) fully compatible with traditional peer review;
enables interactive & transparent forms of review/discussion
open to all interested members of the scientific community andthe public (open peer review).

(2) gives reviewers more information to work with;

provides unlimited access to relevant publications acrossdifferent scientific disciplines and communities(interdisciplinary scientific discussion & qualityassurance).
Pöschl, 2012

INTRODUCTION
Reviewing Peer
Review
OPEN ACCESS AND PEER REVIEW(3) facilitates development and implementation of new metrics forthe impact and quality of scientific publications (combination ofcitation, download/usage, commenting, ranking by various groupsof readers and users, respectively; Bollen et al., 2009).

(4) helps to overcome the obsolete monopoly structures ofscientific publishing & statistical analysis of publication contents
and citations/ references, which limit opportunities for innovation inscientific publishing and evaluation.
Pöschl, 2012Multi – Stage Open Peer Review as practiced in ACP & ACPDINTRODUCTION
Reviewing Peer
Review
OPEN ACCESS AND PEER REVIEW• The effects and advantages of open access, public review,
and interactive discussion can be efficiently and flexibly
combined with the strengths of traditional scientific publishingand peer review (Pöschl, 2009a, 2010a,b).
• conserves strengths of traditional peer review while
overcoming its major weaknesses.

Pöschl, 2012

INTRODUCTION
Reviewing Peer
Review
OPEN ACCESS AND PEER REVIEW• compatible with the structures of traditional scientificpublishing and quality assurance• enables efficient transition from the operational but suboptimal past of subscription-based journals & hidden peer
review to the future of free exchange and transparent
evaluation of scientific information on the internet.

Pöschl, 2012

INTRODUCTION
Reviewing Peer
Review
MULTI-STAGE PEER REVIEW & PUBLICATION PROCESS• Effectively resolves the dilemma between rapid scientificexchange and thorough quality assurance• Offers a win-win situation for all involved parties (authors,referees, editors, publishers, readers/ scientific community).

Pöschl, 2012

INTRODUCTION
Reviewing Peer
Review
MULTI-STAGE PEER REVIEW & PUBLICATION PROCESS
Positive effects & advantages compared to traditional forms ofpublication with closed peer review:
1. Discussion papers offer free speech & rapid disseminationof novel results & original opinions w/o revisions that might
delay or dilute innovation (authors’‎ and‎rea ders’‎ advantage).
2. Interactive peer review & public discussion offer direct
feedback & public recognition for high quality papers(authors’‎ advantage);‎

Pöschl, 2012

INTRODUCTION
Reviewing Peer
Review
MULTI-STAGE PEER REVIEW & PUBLICATION PROCESS• Prevent or minimize opportunity for hidden obstruction andplagiarism (authors’‎ advantage);• Provide complete & citable documentation of criticalcomments, controversial arguments, scientific flaws, &complementary information (referees’‎ &‎rea ders’‎ advantage);‎• Reveal deficiencies & deter submissions of carelessly
prepared manuscripts, thus helping to avoid/minimize thewaste of time and effort for deficient submissions (referees’,‎editors’,‎ publishers’,‎and‎r eaders’‎ advantage).‎

Pöschl, 2012

INTRODUCTION
Reviewing Peer
Review
MULTI-STAGE PEER REVIEW & PUBLICATION PROCESS
3. Final revised papers offer maximum scientific information
density & quality assurance achieved by full peer review (withoptional anonymity of referees) & revisions based on referees’‎comments plus additional comments from other interestedscientists (readers’‎ advantage).‎

Pöschl, 2012

INTRODUCTION
Reviewing Peer
Review
RECOMMENDATIONS
1. Promote open access• Traditional publishers- reluctant to undermine profits aslong as they can rely on rigid subscription schemes;• Those ready to serve science will swiftly adapt to new
financing schemes illustrated by the open choice modeland acquisition of BioMed Central by Springer.
• Others can be substituted by new service providers as
indicated by the swiftly growing number, size, and visibilityof open access publishers and journals.
Pöschl, 2012

INTRODUCTION
Reviewing Peer
Review
RECOMMENDATIONS
1. Promote open access2. Promote multi-stage open peer review in new and existing
journals, repositories, and other publication platforms.
• Public review & interactive discussion are technicallystraightforward & can be flexibly adjusted to different
scientific communities,• Care should be taken when dealing with key features ofpeer review and scientific discourse (optional anonymity fordesignated referees, permanent archiving, and citability of
published manuscripts and comments, etc.). Pöschl, 2012

INTRODUCTION
Reviewing Peer
Review
RECOMMENDATIONS
1. Promote open access2. Promote multi-stage open peer review in new and existing
journals, repositories, and other publication platforms.
3. Promote development and implementation of new andimproved metrics for the impact and quality of scientificpublications (combination of citation, download/usage,commenting, and ranking by various groups of readers andusers, respectively).
Pöschl, 2012

INTRODUCTION
Reviewing Peer
Review
RECOMMENDATIONSOpen access: urgently needed to stimulate innovation bycompetition in this field which has long been hampered bymonopoly structures.
The working capacities of librarians & related information
professionals that may be liberated by the end of thesubscription business are urgently needed for structuring,processing, quality assurance, digital preservation of scientificcontents, bibliometric data, & statistical analyses both atscientific institutions and at commercial service providers.
Pöschl, 2012

Reviewing Peer Review

MARAMING SALAMAT PO