Experimenter Fraud: What Are Appropriate Methodological Standards?

LA FRAUDE DE L'EXPERIMENTATEUR: QUELS SONT LES STANDARDS METHODOLOGIQUES APPROPRIES EXPERIMENTATORBETRUG: WAS SIND DIE GEEIGNETEN METHODOLOGISCHEN MASSSTABE? FRAUDE DEL EXPERIMENTADOR: ? CUALES SON LAS NORMAS METODOLOGICAS APROPIADAS? The recent extensive discussions of methodological issues for psychological research have included the topic of experimenter fraud (John, Loewenstein, & Prelec, 2012; Simonsohn, 2013; Strobe, Postmes, & Spears, 2012). In extending these discussions to parapsychology. Stokes (2015) and Kennedy (2014) raised the possibility that experimenter fraud may be more extensive than is generally realized and may significantly compromise the research findings, particularly when combined with other forms of methodological bias. Palmer (2016) and Roe (2016) challenged these points. Virtually all discussions of fraud in parapsychology describe the Levy case as the most definitive evidence of experimenter fraud. As one who was involved in the Levy expose, my observation is that the various discussions miss the key factors that were involved and the important implications of the Levy case. In addition, writers often make surmises about what happened without direct knowledge. For example, Rogo's (1985) description contains numerous errors and is not a reliable source of information (Kennedy, 2017). In this paper I discuss key points and implications of the Levy case that have not been discussed before and describe my experiences working in research environments where measures to prevent fraud were standard procedure. I also make recommendations for dealing with fraud based on my experiences combined with the findings from published studies of fraud. As a frame of reference, it is useful to describe three alternative positions or expectations with regard to experimenter fraud. 1. The expectation of fraud-proof experiments is based on the assumption that research can and should be conducted in a way that completely precludes the possibility of experimenter fraud. As Palmer (2016) pointed out, this is the standard for extreme skeptics such as Hansel (1966, 1980). Palmer also pointed out that this expectation is currently widely rejected in science. For me this expectation is eliminated by the fact that no measures could prevent collusion among experimenters to produce fraudulent results. 2. The expectation that fraud by one experimenter should be very difficult or impossible is based on the recognition that the vast majority of cases of fraud start when an experimenter is alone with the data and finds that data manipulation or fabrication is easy and tempting with very little possibility of detection. This standard has long been recommended in parapsychology (Akers, 1984; Dalton et al., 1996; Rhine, 1974, 1975) and was a methodological standard in my experience working in regulated medical research (described below). 3. The expectation that procedures to prevent experimenter fraud are unnecessary is usually based on the argument that independent replication will reveal and overcome fraud. Palmer (2016) raised a related argument that procedures to prevent experimenter fraud could create a paranoid work environment that should be avoided. He also argued that discussion or modeling of possible fraud for specific studies are implicit accusations of fraud and therefore are not ethical in the absence of strong evidence that fraud has occurred. Background on Experimenter Fraud Experimenter fraud is an established factor in scientific research (Broad & Wade, 1982; John et al., 2012; Stroebe et al., 2012; Retraction Watch, n.d.). The extent of occurrence of fraud is unknown because undetected instances are likely and institutions have often been reluctant to make cases of fraud known publicly. Independent replication and peer review for publication have generally not been effective at detecting even extensive fraud (Broad & Wade, 1982; Strobe et al. …