Meta-Analysis Part 1 Approved For Release LhDw Adio SSA RURRS-00789R002200390001-5 we developed (May, Radin et al, 1985) requires binary statistics. In addition, to avoid the possibility that reported p values or z scores were rounded up, whenever possible we recorded the reported number of trials (bits generated in an experiment) and hits (number of times the designated bit was obtained) in these experiments. Of these 332 binary experiments, 188 were reported injournals and conference proceedings, of which 58 were reported significant at p < .05, 2-tailed, (against 9.4 expected by chance). The probability of observing 58 significant studies out of 188 is less than 10757, We refer to this body of data as the “survey.” The remaining 144 experiments were obtained from the Princeton Engineering Anomalies Research Laboratory (Dunne, Jahn and Nelson, 1982). Of these experiments, 13 were significant 2-tailed, resulting in p < .04 (corrected for continuity). We refer to these experiments as the “Princeton” data. Experiments with incomplete descriptions Of the 188 survey studies, 30 were simulated by Monte Carlo techniques because the experiment was reported as nonsignificant but neither the z score nor the number of trials and hits were provided. To perform the simulation, we had a pseudorandom generator (cf. May, Humphrey and Hubbard, 1980) choose a z score at random from a normal distribution [N(0,1)], but bounded between 10725 to 1.64 and -10 5 to -1.64.3 In five additional studies, the results were reported as significant and p or z values were provided, but the number of trials or hits were not given. For these five studies, since the z score was known or could be calculated from a p value, the trials or hits (whichever was missing) were calculated. Table 1 shows a breakdown of the number of experiments reported in each of the seven reference sources we used. It is clear that the reports provided in the Research in Parapsychology series are not as detailed as one might have wished, but it is not surprising since the contents of this reference are only abstracts of the full papers presented at the annual Parapsychological Association conventions. Table 1. Experiment breakdown by source of reference. Experiments Experiments 5 Journa) of the American Society for Psychical Research European Journal of Parapsychology 6 Journal of the Society for Psychical Research 9 Proceedings of the Parapsychological Association® 32 Journal of Parapsychology 49 Research in Parapsychology Princeton Engineering Anomalies Research Lab © for the years 1971 and 1984 3. We did not generate z scores of zero because this data was ultimately used in an evaluation of our model (May, Radin et al, 1985), in which log(z) is taken. UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200390001-5
Meta-Analysis Part 1 Approved For Release UN GloASSIBLER .00789R002200390001 -5 In summary, of the 332 experiments we considered (188 from the survey and 144 from Princeton), 71 were reported significant at p < .05, 2-tailed, for an overall binomial probability of p< 5.4 x 107s, ADDRESSING CRITICISMS OF THE DATA Taken as prima facie evidence, one might think that this body of published data provides indisputable evidence that an anomaly exists. But there are numerous reasons why the data may be suspect. The main criticisms (Akers, 1984; Hansel, 1980; Hyman, 1985; Kurtz, 1981) include . Results are due to chance . Basic statistical assumptions are violated . Only significant studies are published . Experiments are not replicable . RNGs are nonrandom . Poorer studies are included with better studies NMmAdbWD Be Let us consider each of these six steps as successive filters for the reliability of the data. If each criticism can be satisfactorially refuted or countered, then a persuasive case for an anomalous effect can be made. i. Results are due to chance In any one experiment we cannot establish the reality of a phenomenon, regardless of the significance level, unless strong theoretical predictions have preceeded the experiments. For example, the recent experiments suggesting that Bell’s inequality is violated (e.g. Aspect, Dalibard, and Roger, 1982; Aspect, Grangier and Roger, 1982) have been widely accepted within the physics community on the basis of only a few empirical studies despite its profound implications on our view of the nature of reality (cf. d’Espagnat, 1979, Mermin, 1985; Rohrlich, 1983). Parapsychology, however, has had the disadvantage of not having a firm theoretical base on which to stand. Thus the nature of the claim (any claimed psi effect) understandably requires extremely persuasive evidence. One wonders how statistically strong an effect must be to bring about a consensual agreement within the scientific community that a psi effect on RNGs is real. Would p < 10743 be sufficient? If this figure were revised to take into account all of the criticisms noted above, and the end result were say, 1075, would that be sufficient? Clearly an overall p = .1 would not satisfy anyone, so there is a decision curve related to this question. This curve is probably different according to individual prejudices and predilictions, but the resolution of this question is beyond the scope of the present paper. Note that if an anomaly did exist, it would not necessarily imply that psi was the mediating factor. Such an anomaly may, for example, reveal some heretofore unknown statistical peculiarities about random numbers. B-7 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200390001-5