a : Approved For Release 2000/08/08 : CIA-RDP96-00789R002200450001-8 1 CRITERIA FOR ACCEPTABLE EXPERIMENTS One challenge that parapsychologists face as they strive for greater uniformity and more cleanly defined experimental methodologies is the variety of (putative) phenomena and corresponding experiment types that are being pursued. In the psi ganzfeld experimental regime, 18 for example :-- which focuses on ESP effects -- the experimental setup, equipment, and methodology are quite different from those employed in REG experiments, which attempt to measure psychokinesis. Other types of remote viewing and PK experiments have still other setups and protocols. Within each category, moreover, different researchers use different setups, equipment, and methodologies from each other and over time. This is true, of course, in any field of research; but the critical scrutiny applied to parapsychology militates against the customary flexibility. The current movement in the direction of greater uniformity and cleaner methodologies is thus, in large part, an attempt to strengthen the case for parapsychology in defense against its critics. In their joint communique, Hyman and Honorton made several recommendations for the design and conduct of future psi ganzfeld experiments. In general, they said, such experiments should strive for: elimination of possible means of sensory leakage; better procedures and documentation for the randomization of targets; more thorough documentation of judging and feedback procedures: avoidance of arbitrary multiple analyses; avoidance of selective reporting of statistically significant results; correct use of valid statistical tests and methods; and painstakingly thorough documentation of experimental procedures and other information on experimenters and subjects as well as certain raw data. oo 0 0000 Detailed suggestions were made in each of these areas. 19 The authors noted that many of the recommendations would apply equally well to other types of parapsychology research besides ganzfeld. During the workshop, Robert Jahn described several criteria or "tactics" that he and his colleagues had found to be "useful in helping us to understand the phenomena that we are examining." While some were general suggestions for improving the standing of the field as a whole, others were specific recommendations for strengthening parapsychology experiments, particularly in the PK area. The latter are: o employ impeccable experimental equipment and technique, using the highest possible precision; o work with the largest possible data base (to separate the marginal effects from background noise); o concentrate on operator-specific, statistical replicability; 18 Ganzfeld, or "whole field," ESP experiments Place the receiver in an environment of reduced external stimuli for attempted receipt of a visual image "transmitted" telepathically by a sender located remotely, 19 Hyman, R. & C. Honorton, op. cit., pp. 355-361. Approved For Release 2000/08/08 : CIA-RDP96-00789R002200450001 -8
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200450001-8 o use a "tripolar protocol," i.e., collect data under three interspersed conditions (e.g., intention to raise the number of events, intention to lower the number of events, and baseline); and o within the experimental program, maintain equal respect for the aesthetic/impressionistic and rigorous/analytical aspects of the research on this unusual topic. One additional point that was made repeatedly in the workshop regarded the necessity of distinguishing between exploratory and confirmatory (proof- oriented) experiments. The former are generally shorter, more broadly defined, and not reported. There was general agreement that the explorational objectives should be clear and the temptation to report positive results as "proof" of a hypothesis should be resisted. Instead, exploratory studies should be used to suggest and develop hypotheses for later proof-oriented experiments which should be highly defined and constrained. Exploratory studies can also be used to develop and refine experimental procedures. III. ANALYSIS OF DATA As was noted earlier, the scientific evidence for psi consists chiefly of statistically significant deviations from a theoretical chance expectation or experiential baseline, Consequently, the validity of statistical analyses employed is crucial to the strength and acceptability of parapsychological claims. According to Jessica Utts, a statistician participating in the OTA workshop, parapsychologists in general tend to be more careful with statistical methodology than researchers in other fields. They are generally more responsive to criticism on this score than other researchers are, she said. There is general understanding among them, Utts said, that human responses cannot be assumed to be statistically random. It is also well understood that selection of targets with replacement is preferable because it allows independent trials. She also referred to the formal policy of parapsychology for reporting negative results. However, it was noted, not all statistical analysis by parapsychologists (or by their critics) is of high caliber. Much of the discussion centered around a single broad topic: how to inject the maximum robustness and power into the empirical data base. It was recommended that parapsychologists should confine themselves to methodologies that yield such robust data that the outcome does not depend largely on subtle statistical points. There was general agreement on the various suggestions that were made to this end. For example, there was considerable discussion of the traditional notion that a "successful" experiment has a significance level (p value) of .05 or less (meaning that there is no more than a 5 percent chance that the results are due to chance). Partly as a result of this concept, Utts noted, nonsignificant results (that is, where p >.05) are too often interpreted erroneously by critics as insignificant results. An example was given to demonstrate that sample size (n) has a substantial impact on the statistical outcome; if n is small there is very little chance for a "successful" outcome, even if relatively high actual probabilities of success based on ESP are assumed. Meta-analysis of all the forced-choice precognition experiments conducted from the 1930s on showed that Approved For Release 2000/08/08 : CIA-RDP96-00789R002200450001-8