Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine

Publication of medical research has high stakes: the communication and legitimization of medical research, the advancement of authors' careers, priorities in funding decisions, the direction of future research, and the visibility and prestige of journals themselves. Peer review and editing play central roles in the publication process, affecting the acceptance of a manuscript and the form in which it appears. The most commonly heard justification of peer review is that it is an indispensable aid to an editor in assessing the importance of a scientific question and in assessing how well that question has been answered [1, 2]. However, it has also been criticized as being inherently conservative, censorial, and, perhaps worst of all, arbitrary [3]. A frequently heard charge is that peer review delays the dissemination of crucial medical information without commensurate benefit [4-7]. During the last several years, some medical journal editors decided that the value of peer-review and editing practices should be examined with the same rigor demanded for testing medical hypotheses [8, 9]. The First International Congress on Peer Review was organized in 1989 [10], bringing together medical journal editors and other interested scholars to present and discuss research on peer review; a second Congress was held in 1993. The peer-review process has two components: the assessments by external reviewers and the decisions and actions taken by editors, which are partially affected by comments from the reviewers. To our knowledge, no study has evaluated the effects of peer review and editing on manuscript quality once the decision to accept has been made, and a computerized search of Index Medicus back to 1966 failed to locate any such studies. In this paper, we present the results of such a study, assessing the change in a manuscript between the times of provisional acceptance and final publication. We studied whether the quality of accepted manuscripts was improved by peer-review and editorial processes and, if it was, which aspects were most improved. Methods Setting The study was conducted at the editorial offices of Annals of Internal Medicine. Annals, a specialty journal in internal medicine, is published twice monthly and has a circulation of approximately 100 000. Annals receives approximately 2400 manuscripts annually, of which half are reports of original research. During the period of this study, the investigators included the editors of Annals (RHF and SWF) and a statistical associate editor (SNG). The Review Process No change was noted in the usual review and editing procedures at Annals during the time of this study. All manuscripts received at Annals were initially reviewed by one of two full-time editors or one of two half-time deputy editors, as well as by one of seven associate editors, all of whom are faculty members of medical schools in Philadelphia and have subspecialty interests (for example, infectious disease, gastroenterology). Half of the submissions were returned to authors without further review and half were sent to at least 2 outside reviewers, selected by the associate editor from a database of about 7000 reviewers. After comments from the reviewers were received, the original editor and associate editor reassessed each manuscript and chose which ones would be discussed at a weekly editorial conference of editors, deputy editors, medical associate editors, and two statistical associate editors. Factors that affected acceptance decisions included the quality of the research, the importance of the question, the contribution of the finding to its field, the utility and interest for Annals readers, the quality of the presentation, the priority relative to other articles, and available space. Authors were notified either that the editors would not accept the paper, that the editors were willing to reconsider the paper after major revisions, or that the paper was provisionally accepted, pending satisfactory revision. Approximately one third of the articles evaluated by outside reviewers were accepted, 15% of submitted original research articles. Papers to be considered further were sent to authors, along with the comments of the two outside reviewers, comments of one of the statistical editors, and a letter from one of the editors or deputy editors (which summarized the discussion at the weekly conference, the ideas of the associate editor, and suggestions from the editor). In addition, each manuscript was reviewed by a production editor, and directions for changes in manuscript wording or layout of figures and tables were included. All revised manuscripts were reviewed by the editor or deputy editor in charge of the manuscript, the appropriate associate editor, the statistical editor, and the production editor. Some revised manuscripts were also reassessed by the original outside reviewers. Approximately half of the revised manuscripts were returned to authors for further revision. Most revised manuscripts (>95%) were ultimately published. The time taken by this process was approximately 2 weeks for the initial decision to review or reject, 8 additional weeks to review and make an acceptance decision, 8 weeks until final acceptance, and about 4 months until publication. More than 95% of manuscripts submitted to Annals had a provisional acceptance or rejection decision sent to the authors within 3 months. The average time from submission to publication was about 7 months, with initial peer review accounting for approximately 6 weeks. Manuscript Selection and Study Design All original research manuscripts (articles) accepted for publication by Annals from March 1992 to March 1993 were entered into the study after obtaining the author's consent. Commentaries, reviews, expository pieces, editorials, and brief reports were not included. This study had a beforeafter design, in which two versions of each manuscript were evaluated: the version originally submitted and the version sent to the printer for publication after all modifications based on peer review, editors' comments, and copyediting. All before and after manuscripts were in electronic form and were reformatted to make the appearance of the two versions identical. Authors' names and affiliations were removed. The design of the study was approved by the Institutional Review Board of the University of Pennsylvania School of Medicine. Definition of Quality Manuscript quality can be separated conceptually into two components: the quality of the research itself, and the quality of the research report. The quality of the research report was evaluated in this study. It was defined as follows on the cover sheet of the quality assessment instrument: Whether the authors have described their research in enough detail and with sufficient clarity so a reader could make an independent judgment about the strengths and weaknesses of their data and conclusions. Manuscript Quality Assessment Instrument A 34-question instrument was developed to structure the assessment of the quality of a manuscript (Appendix). Items were derived from published checklists [11-14], articles about the contents of journal articles [15-18], the authors' editorial experience, and the comments of journal editors and methodologists who reviewed drafts of the instrument. Each question could be answered on a 5-point ordinal scale, where 1 was worst and 5 was best. The instrument was organized along the same dimensions as a standard journal article: Title and Abstract (2 items), Introduction (2 items), Methods (7 items), Results (15 items), Discussion and Conclusions (4 items), and General Evaluation (4 items). An additional question asked for a subjective assessment of the manuscript's overall quality on a 10-point scale. The instrument differed from previously published quality scoring schemes in several ways. It was not a checklist but rather was a set of structured judgments, graded ordinally, allowing users the discretion not to penalize a manuscript if a detail was omitted that was not critical to the study's interpretation. Also, in keeping with the definition of quality given above, each question was about the adequacy of the reporting rather than the quality of the research itself. Assessment A panel of 44 physicians and epidemiologists with training in research methods and in critically assessing the medical literature was recruited to serve as an independent panel of expert assessors (experts). They did not receive any formal training in the use of the assessment instrument, although general guidelines were given on the cover sheet (Appendix). The panel was masked to the design and aims of the study; they were told only that they were participating in a study of manuscript quality for Annals. Before and after versions of each manuscript were randomly assigned to different experts to prevent the bias that might have been introduced if they could infer the design of the study and thereby which manuscript had been through the editorial process. Thirty-two manuscript versions were given to two or three experts to assess the reliability of the instrument; all others were assessed by only one expert. Statistical Analyses The study was designed to have 90% power to detect a 0.5 unit change in average score, assuming a within-manuscript standard deviation of 1 scale unit, using = 0.05. The main outcome measure was the percentage of items that were scored 3 or higher on the 5-point scales (percentage score). The average of all score components (average score) was also analyzed. Linear regression was used to assess the effect of revision on each of these outcome measures, with terms controlling for manuscript and reviewer. Item-specific analyses were done on dichotomized item scores (0 for ratings 2 and 1 for ratings 3). The change from before revision to after revision in individual items was statistically assessed with conditional logistic regression, which allowed for variable group s