a Approved For Release 2000/08/08 : CIA-RDP96-00789R002200450001-8 falsifiable -- that is, that there is no definitive defense against the charge of "general sloppiness". Without detailed criteria for what constitutes acceptable research, they say, criticism of an experiment cannot be anticipated or refuted. In their view, it is an example of the unfairly high standards that are set for parapsychology research, in which no amount of care is ever "enough." The critics present at the workshop tended to agree in principle with the unfairness of “non-falsifiability," but they believe that the “dirty test tube" charge is nevertheless valid because the success of the experiments rests on whether a small but significant departure from some statistical norm or observed norm has in fact been seen, making it essential to eliminate extraneous factors. Parapsychologists object that the charge itself is irrelevant -- that if the alleged flaws or "dirt" have no demonstrable connection with the experimental outcome, it is capricious to argue that the experiment is invalid. The litany of charge and countercharge could be detailed much further. Suffice it to say that the controversy surrounding parapsychology is nowhere more heated and complex than in the area of experimental methodologies. REPLICABILITY For critics of parapsychology, a key argument against the existence of psi phenomena is the difficulty of replicating (repeating) successful experiments -- a difficulty that also plagues many accepted areas of the social and behavioral sciences. If the same effect could be produced many times by different experimenters using different equipment but the same procedures, then the likelihood that an extraneous variable is producing the effect would be much lower and the validity level of the anomalous results would be higher. However, positive results are in most cases maddeningly (to parapsychologists) difficult to replicate, even with the same operator. (Because the effects seem to be highly operator-specific, results are even more difficult to replicate with different operators or groups of operators.) Parapsychologists attribute this replicability problem to the elusiveness and weakness of the psi "signal," and to its vulnerability to subtle subjective factors, most of them unknown. Critics contend that the near non-replicability merely demonstrates the effects of randomness combined with multiple flaws in experiment design and conduct. The parapsychologists are well aware that experimental flaws, both actual and alleged, become less important as a phenomenon becomes more replicable. Therefore, in recent years they have placed great emphasis on replicability. As one noted in his presentation, "The final criterion of the success or failure of my research program is the extent to which I'm able to develop procedures that I can articulate with sufficient precision that other people are able to obtain similar results." In the case of ESP-oriented experiments (such as remote perception or “ganzfeld" psi experiments), the number of these more standardized experiments is still not large. There, parapsychologists have relied upon what is termed "meta-analysis" of many studies, taken as a group, to give statistical confirmation for a pattern of positive results. The rationale is that "an Approved For Release 2000/08/08 : CIA-RDP96-00789R002200450001-8
a Approved For Release 2000/08/08 : CIA-RDP96-00789R002200450001-8 experiment or an effect may be considered replicated if a series of replication attempts provides statistically significant evidence for the original effect when analyzed as a series."l2 It ig also contended that this approach allows the analyst to assess methodological flaws to determine their effect empirically across the data base. Using the meta-analytic approach, parapsychologists have calculated an overall "success" rate, in achieving statistically significant effects (p <.05), of 45% for ganzfeld ESP studies and 21% for random event generator (REG) studies. However, critics have objected strongly to these findings, saying that many of the studies included are not directly comparable due to differences in methodology. They also claim that many unsuccessful experiments have not been published and are therefore not in the data base. This "file drawer problem", they assert, artificially increases the success ratio and the overall effect size.1 They also cite a corresponding "retrospective study" problem, in which small exploratory studies are included in the data base after the fact if their results turn out positive. 14 Furthermore, they believe, the proliferation of flaws in methodology simply perpetuates the same erroneous results. The debate over replicability, particularly in the context of meta-analysis, has been heated and continues to be so. However, the nature of the controversy is currently being outlined in detail and areas of agreement are being identified. For example, the recent joint communique by two of the workshop participants refers to "the growing role we believe meta-analysis will play, both in the evaluation of research quality and in the assessment of moderating variables. We urge parapsychological investigators to plan and report their experiments with the idea that their single experiment will contribute to a future meta-analysis." This idea relies in part on guidelines for greater uniformity in experiment design and conduct which the two authors specified in their article. Such uniformity or consistency is viewed as crucial by many moderate critics, who point to the fact that there has never been a successful parapsychology experiment which any and all competent researchers, following the same procedure, can replicate exactly. As one of the workshop participants noted in a recent article, "Although one cannot set precise standards that evidence of psi must meet, judgment should be suspended until there is at least some consistency among research findings from a body of methodologically irreproachable experiments, at least some of which are repeatable [by any other researcher]." 12 Rao, K.R. & J. Palmer. The anomaly called psi: Recent research and eriticism. Behavioral and Brain Sciences, 1987, 10(4): 539-551 (p. 544). 13 It should be noted that at least one critic (Hyman) has acknowledged the file drawer problem to be less evident in parapsychology than in other areas of science. (Hyman, R. & C. Honorton, op. cit., p.359) 14 The selective reporting of "positive" results by either nonreporting or retrospective reporting is prohibited by Parapsychological Association policy. 15 Hyman, R., op. cit., pp. 8-15. 16 Hyman, R. & C. Honorton, op. cit., p. 361. 17 Alcock, J.E. Parapsychology: Science of the anomalous or search for the soul? vioral and Brai ences, 1987, 10(4): 553-565 (p. 559). Approved For Release 2000/08/08 : CIA-RDP96-00789R002200450001-8