Meta-Analysis Part 1 Approved For Release Jobe ba SiS HESS .00789R002200460001-3 experiments (3 groups), 30 experiments (subject by subject analysis), and so on. How do we decide what to use? We resolved this issue for this first-pass analysis in the following way: For cases where there were multiple hypotheses under test and multiple analyses of the data, we chose as the experimental] unit the largest possible accumulation of data compatible with a single “direction of effort” assigned to the subjects. A clearly defined direction of effort meant that the experimental protocol required either more 1’s or more 0’s from the RNG to successfully complete the assigned task, regardless of whether or not the subjects actually knew their task in detail. Say, for example, a hypothesis predicted that group A would score higher than group B, and it was stated that “higher” meant more 1 bits. "Then we would take this Study as two experiments: Group A’s and group B’s scores. In this particular case, since group A was predicted to score higher than group B, if in fact the difference between z(A) and z(B) were Significant, then both z scores would be taken as positive, regardless of the reported z’s. Thus if z(A) = 1.5, z(B) = -1.0, then the z-score difference between them would be significant one-tailed with zaitt= 1.77. If the number of trials run in each case were 10000, then the number of hits assigned per experiment would be hits(A)= 5075 and hits(B) = 5050, which are both positive deviations; similarly the z scores would be recorded as z(A) = 1.5, z(B) = 1.0. If z(A) = 1.2 and z(B) = -1.0, the z scores would be recorded as originally reported since zai is not significant. The same would be true if z(A) = -2.0 and z(B) = 2.0. (Fortunately, such problems of interpretation were not often encountered in the survey.) As another example, if groups A, B, and C all tried to influence an RNG in a particular way, and no predictions were made as to interactions, then their overall result would be combined as one experiment. In this way, we attempted to emphasize in the meta-analysis the underlying question of whether or not observers could influence or otherwise affect the statistical output of an RNG according to the stated intention of the experimenter. Results of literature review We found 73 pertinent references in the journals and reports.2 These references included 381 experiments contributed by 38 different principal investigators, representing about 10 different laboratories around the world. We say “about 10 laboratories” because over the years labs have come and gone, researchers have moved among different labs, and in many cases, one or two individuals at an academic or private research institution are considered a “laboratory.” Breakdown of experiments _ Of the 381 experiments found, 332 (in 56 remaining references) were described as using binary generators based on either radioactive decay or electronic noise. For this meta-analysis, we considered only studies using binary RNGs (or any study in which the hit rate was defined or could be interpreted as 50%) for three reasons: First, since 87% of the experiments (332/381) employed binary generators, we felt that this sample was representative of the entire RNG database; second, for the sake of simplicity; and, third because the test of a model ; 2. These references are listed under the heading “Meta-Analysis” in the references at the end of this paper. * B-5 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
Meta-Analysis Part 1 Approved a Release UN Geb AS SH ELEQ 00789R002200400001 -3 we developed (May, Radin ef al, 1985) requires binary statistics. In addition, to avoid the possibility that reported p values or z scores were rounded up, whenever possible we recorded the reported number of trials (bits generated in an experiment) and hits (number of times the designated bit was obtained) in these experiments. Of these 332 binary experiments, 188 were reported in journals and conference proceedings, of which 58 were reported significant at p < .05, 2-tailed, (against 9.4 expected by chance). The probability of observing 58 significant studies out of 188 is less than 10757. We refer to this body of data as the “survey.” The remaining 144 experiments were obtained from the Princeton Engineering Anomalies Research Laboratory (Dunne, Jahn and Nelson, 1982). Of these experiments, 13 were significant 2-tailed, resulting in p < .04 (corrected for continuity). We refer to these experiments as the “Princeton” data.* - ‘ Experiments with incomplete descriptions Of the 188 survey studies, 30 were simulated by Monte Carlo techniques because the experiment was reported as nonsignificant but neither the z score nor the number of trials and hits were provided. To perform the simulation, we had a pseudorandom generator (cf. May, Humphrey and Hubbard, 1980) choose a z score at random from a normal distribution [N(0,1)], but bounded between 10-25 to 1.64 and -10@5 to -1.64.9 In five additional studies, the results were reported as significant and p or z values were provided, but the number of trials or hits were not given. For these five studies, since'the z score was known or could be calculated from a p value, the trials or hits (whichever was missing) were calculated. Table 1 shows a breakdown of the number of experiments reported in each of the seven reference sources we used. It is clear that the reports provided in the Research in Parapsychology series are not as detailed as one might have wished, but it is not surprising since the contents of this reference are only abstracts of the full papers presented at the annual Parapsychological Association conventions. * Table 1. Experiment breakdown by source of reference. Experiments Experiments Journa) of the American Society for Psychical Research European Journal of Parapsychology Journal of the Society for Psychical Research Proceedings of the Parapsychological Association*® Journal of Parapsychology Research in Parapsychology Princeton Engineering Anomalies Research Lab * for the years 1971 and 1984 3. We did not generate z scores of zero because this data was ultimately used in an evaluation of our mode] (May, Radin et al, 1985), in which log(z) is taken. UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3