Editorial Policies for the Publication of Controversial Findings

Armstrong and Hubbard (1991), in a survey of editors of 20 psychology journals, found a bias against the publication of papers with controversial findings. The 16 editors who responded said that they received few papers with controversial findings "during the last two years." When they did receive such papers, the reviewers rejected them. Some of these editors expressed dismay over this situation and said that their referees usually rejected such papers. The study encountered only one instance where the reviewers agreed that a paper with controversial findings should be published. The editor who handled this case was blunt: he picked referees who would agree to its publication. Comments Postprint version. Published in International Journal of Forecasting, Volume 8, Issue 4, December 1992, pages 543-544. Publisher URL: http://dx.doi.org/10.1016/0169-2070(92)90064-G This journal article is available at ScholarlyCommons: http://repository.upenn.edu/marketing_papers/109 Published in International Journal of Forecasting, 8, 1992, 543-544. Editorial Policies for the Publication of Controversial Findings J. Scott Armstrong The Wharton School, University of Pennsylvania Armstrong and Hubbard (1991), in a survey of editors of 20 psychology journals, found a bias against the publication of papers with controversial findings. The 16 editors who responded said that they received few papers with controversial findings “during the last two years.” When they did receive such papers, the reviewers rejected them. Some of these editors expressed dismay over this situation and said that their referees usually rejected such papers. The study encountered only one instance where the reviewers agreed that a paper with controversial findings should be published. The editor who handled this case was blunt: he picked referees who would agree to its publication. I believe that part of the problem lies with the roles of the editors and referees. Referees often believe that their job is to decide whether they agree with the results of a study, while editors often believe that their job is to summarize the recommendations of the referees. Such roles are unlikely to lead to the publication of papers with controversial findings. Mahoney's (1977) experiment found that referees rejected papers that had controversial findings. They did so, they reported, because of “poor methodology.” Meanwhile, an equivalent sample of referees accepted papers having identical methodology but noncontroversial findings; these referees approved of the technology. Goodstein and Brazis (1970) obtained similar findings. The Editors of the International Journal of Forecasting are committed to the notion that important advances in the field depend upon the publication of papers with controversial findings. Such papers are sought by the Editors. Unfortunately the supply is limited. Editorial procedures have been instituted to aid in the acceptance of such papers. One procedure is that authors can propose a set of reviewers and the Editors will ask at least one of these to review the paper. Another procedure is the “Note to Referees,” where the author is invited to submit the design of the study without results. Referees would review this before they received the results. A number of authors have provided a list of possible referees. Few have used the “Note to Referees.” Perhaps the problem is similar to that noted for psychology journals (Armstrong and Hubbard, 1991); the IJF does not receive many papers that have controversial findings. In an effort to increase the number of papers with controversial findings, and at the same time provide safeguards against the publication of incorrect findings, I believe that the IJF should add new procedures. New Editorial Procedures – Some Suggestions Decisions on Papers with Controversial Findings For papers with controversial findings, the editors, not the referees, should make the decision about publication. I believe that the editorial board should seek important papers and ensure that they are published. The role of the referees in this case is to ensure that the paper is as good as it can be, rather than asking them to make a recommendation about publication. Papers with controversial findings can be so identified by the authors, by the Editor, or by the Associate Editor handling the paper.