Publication of research on controversial topics: The early acceptance procedure

Research with the potential to produce controversial findings is important to progress in the sciences. But scientific innovators often meet with resistance from the scientific community. Much anecdotal evidence has been provided about the reception accorded to researchers who have obtained controversial findings. While many of these cases occurred long ago (e.g., Copernicus and Galileo), the problem continues to the present. This problem has been addressed to some extent in that nearly all universities grant their faculty tenure to protect their right to publish their findings. Still, the right to publish one's findings does not remove the barriers to publication of controversial findings. Perhaps the major barrier to publication is peer review. Peer review serves many useful functions such as correcting errors and providing a fair way to allocate journal space and research funds. But it also suppresses innovation. Below, I discuss how peer review affects the publication of controversial findings, discuss what is currently being done, and then recommend another solution to this problem. Comments Postprint version. Published in International Journal of Forecasting, Volume 12, Issue 2, June 1996, pages 299-302. Publisher URL: http://dx.doi.org/10.1016/0169-2070(95)00626-5 This journal article is available at ScholarlyCommons: http://repository.upenn.edu/marketing_papers/106 Published in International Journal of Forecasting, 11, 1995, Notes: 1-4. Publication of Research on Controversial Topics: The Early Acceptance Procedure J. Scott Armstrong The Wharton School, University of Pennsylvania Research with the potential to produce controversial findings is important to progress in the sciences. But scientific innovators often meet with resistance from the scientific community. Much anecdotal evidence has been provided about the reception accorded to researchers who have obtained controversial findings. While many of these cases occurred long ago (e.g., Copernicus and Galileo), the problem continues to the present. This problem has been addressed to some extent in that nearly all universities grant their faculty tenure to protect their right to publish their findings. Still, the right to publish one's findings does not remove the barriers to publication of controversial findings. Perhaps the major barrier to publication is peer review. Peer review serves many useful functions s uch as correcting errors and providing a fair way to allocate journal space and research funds. But it also suppresses innovation. Below, I discuss how peer review affects the publication of controversial findings, discuss what is currently being done, and then recommend another solution to this problem. Effects of Peer Review on the Publication of Controversial Findings Papers with findings that challenge existing beliefs are often treated harshly by journal reviewers. For example, Gans and Shepherd (1994) examined the reception given, to new ideas in economics. In this paper, leading economists discuss their frustrations because their best work has often received unfavorable reviews. This perception of a bias against innovative findings was found in a systematic survey of applications for National Cancer Institute funding in 1980. Respondents were asked about their attitudes toward reviewers. For example, in response to the question whether reviewers were “biased against women and/or minorities,” only 4.9% of the 336 respondents agreed. When asked whether reviewers were “reluctant to support unorthodox or high-risk research,” 60.8% agreed (Chubin and Hackett, 1990, p. 66). In a survey of the editors of 16 leading American Psychological Association journals, Armstrong and Hubbard (1991) found that the editors were discouraged because few papers with controversial findings were submitted. Furthermore, of those submitted, few were favorably reviewed by the referees. For the two year period covered by the survey, only one paper with controversial findings had received all favorable reviews. (In this case the editor told us that he was determined to publish the paper so he sought out referees who would provide favorable reviews.) This represents one favorably reviewed paper for 32 “journal years.” Reviewers arc unlikely to say they reject a paper because they disagree with the findings. They are probably not even aware of such a prejudice. However, experimental studies (Goodstein and Brazis, 1970; Abramovitz et al., 1975; Mahoney, 1977) show that such a bias does exist. This prejudice poses a risk for those who would study controversial topics. For example, the findings about the poor predictive performance of Box-Jenkins procedures were greeted with a negative response (e.g., see the commentary on Makridakis and Hibon, 1979). Some researchers even thought that these findings should not have been published.