Editorial Peer Review: Its Strengths and Weaknesses.

Health sciences librarians study and teach the principles of evidence-based medicine (EBM), search evidence-based health care (EBHC) resources, and strive to practice evidence-based librarianship (EBL) [1]. In this work, Weller extends “EB” awareness to evidence-based scientific publishing. By providing a systematic review of empirical studies on the editorial peer review process from 1945 to 1997, the author assembles the available evidence on the value and validity of that process and its effect on the quality of the published literature. The author presents the strengths and weaknesses of peer review, analyzes the benefits and shortcomings, makes recommendations for further research, and provides information for improving future studies. With perhaps 6,000 to 7,000 scientific articles written every day [2] and with the review process for just one journal estimated to cost about $1 million a year [3], questioning the worth of this process is appropriate. Though most understand what editorial peer review is, or have experienced it firsthand when submitting manuscripts for review, few have considered its diverse aspects in detail. This book provides insight and brings to light nagging questions such as the following: “What is the evidence that the ‘best’ science or scholarly material is published and the ‘worst’ is rejected?” (p. 51); “What is the value of the review process to authors?” (p. 120); “What is known about the overall quality of reviewers' reports?” (p. 158); “To what degree do reviewers agree with each other when evaluating the same manuscript?” (p. 182); “Is the submission versus acceptance rate for manuscripts different depending on the gender or ethnicity of the author?” (p. 227); and “What kind of statistical errors have been identified through studies of published articles?” (p. 255). Editorial Peer Review offers analyses of research studies that may answer these and other questions. For each question, the author methodically states the criteria for study inclusion, appraises the validity of the studies that address that issue, and, finally, proposes recommendations and draws conclusions based upon the evidence. The preface of this book succinctly states its purpose and its methodology and describes the structure of the book. It is important to understand this before plunging into the substance of the work. Following the introductory chapter, which includes a brief history of the topic and the process used to compile the relevant studies, chapters appear on rejection rates, editors and editorial boards, authors and authorship, role of reviewers and quality of their reviews (including review agreement and bias), and statistical review of manuscripts. Each of these sections is extensive and substantive. One chapter addresses peer review in an electronic environment, and the final one presents key conclusions and recommendations. Because the literature covered in the work does not extend beyond 1998, Weller's chapter entitled “Peer Review in an Electronic Environment” is necessarily limited. The rapidity of change in this arena requires conjecture rather than analysis, as few relevant studies are either completed or in progress. The author was obliged to address the electronic aspects of scholarly publishing, because she previously had given the transition from print to electronic as one of the primary reasons for undertaking this project. “As publication moves from print to electronics and the editorial peer review process may undergo change as a result, now is an excellent time to examine the cumulated information on editorial peer review and critically evaluate the entire process” (p. 3). This reviewer looks forward to the author's future investigation and analysis of peer review in a digital world, as the process of electronic publishing matures and a new area of research emerges. For all other topics covered in this work, the number of studies cited and the depth of analysis are astounding, so much so that the more casual reader may be overwhelmed by the sheer number of references, author names, and study descriptions. Although tables are used effectively throughout the work to present salient characteristics and findings of a large number of research studies, the reader without knowledge of research design or not already familiar with the literature of peer review could find it challenging to digest. Such a reader would do well to attack each chapter in the following manner: (1) read the introductory and background sections, (2) note the research questions and study inclusion criteria, and, (3) finally, move to the recommendations and conclusions. The detail that is included, though necessary for a systematic review of this nature, will be most useful to those who are designing research studies to address the issues of peer review or to journal editors attempting to improve the process. The core elements of the work are of interest to a wide range of readers concerned with scholarly communication, from reviewers, to publishers, writers, and librarians. There is a simple reason for the complexity of fields of study that encompasses peer review and the volume of the literature included in this systematic review—editorial peer review is not limited to a single discipline. Although the field of medicine has apparently produced a large number of the studies, Weller points out that “Since editorial peer review is not a discipline-specific field, literature on the subject could and does exist in almost every scholarly field with a journal publication outlet” (p. 8). The need to cover disciplines that range from medicine, nursing, education, agriculture, and management science has resulted in an in-depth and extensive treatment. The author of this work is well known for her investigations of the scientific peer review and publishing process. She is also a highly regarded and active member of the Medical Library Associations' Research Section. Using a clear and precise writing style and declaring her own fascination with the topic, she compares the process of tracking down relevant studies to the design of a mystery novel. In the preface, she reveals that “a strategy somewhat akin to Sherlock Holmes' methodology was needed to identify and locate all studies in this field. With an eagerness similar to Holmes' enthusiastic, ‘The game is afoot,’ I relentlessly tracked down all leads” (p. xiv). In the end, the author's zeal, knowledge of the field, and skills in writing and analysis lead this reviewer to speculate that, given the time, the author herself just might be able to resolve every one of the hundreds of research questions remaining in the field of editorial peer review! Surely few are more qualified to make the attempt.