The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie.

In modern science, peer review has advanced to become the most important instrument for assessing scientific work. Supporters of the peer review process deem it indispensable, because only active researchers within the same research field—peers—are in a position to assess the scientific quality of their colleagues research. Critics of peer review see as weaknesses of the process that different referees assessments of one and the same piece of scholarly work rarely agree (problem of interreferee agreement), and that recommendations and decisions in peer review show systematic biased judgments, resulting in little correlation between the decision to publish and the impact of the refereed manuscript following publication (problem of predictive validity). Research on peer review, which in recent years has examined the criticisms of the peer review process, deals for the most part with journal peer review and somewhat less frequently with peer review for research grant proposals and fellowship applications. For Godlee and Jefferson, “the biggest surprise is how little we know about its workings. For a system that demands ever increasing rigor and levels of proof from scientists, it remains itself remarkably untouched by the rigors of science. ” Many peer review studies have methodological weaknesses, and “most of the publications on journal peer review are more opinion than research, often the ruminations of a former editor. Likewise, most of the many letters to editors on the topic, the comments of one kind or another are predominantly opinion. ” The present study examines whether the criticisms of the peer review process are justified at a journal that publishes original research papers. In the study Guardians of Science, Daniel evaluated the peer review process at Angewandte Chemie in the mid-1980s. We once again examined the quality of peer review at Angewandte Chemie according to the criteria of reliability (agreement among referees) and predictive validity, using an optimized study design and modern methods of statistical data analysis. To investigate reliability we determined the degree of agreement among referees. Judgments are called reliable if there is agreement of the publication recommendations of several independent referees for a manuscript. To check the predictive validity of the peer review process, studies on manuscript review use, for lack of other operationalizable indicators, the citation counts of manuscripts accepted for publication and manuscripts rejected by a journal but then published elsewhere. According to van Raan, citations provide “a good to even very good quantitative impression of at least one important aspect of quality, namely international impact. ” For Lindsey, citations are “our most reliable convenient measure of quality in science—a measure that will continue to be widely used. ” Scientific judgments of whether journal manuscripts are worthy of publication are said to show predictive validity if there is a statistically significant difference between the citation rates of manuscripts accepted for publication and those of manuscripts rejected by a journal but then published elsewhere.