Approved For Release LIGA SSIFIEDS co7sorddsso0svO5dTs' Another reason why it may be difficult to produce significant experiments at will is the well-known “experimenter effect” (Rosenthal, 1976). This effect is ubiquitous to all the sciences, but parapsychology seems to be especially vulnerable (see, e.g. White, 1977). The experimenter effect may help explain why some critics of parapsychology claim that they have never obtained significant results in their attempts to replicate psi experiments (e.g. Kurtz, 1981, p.16; Neher, 1980, p.147). Of course, the odds of never obtaining a significant study can be astronomical, depending on the number of studies conducted. Unfortunately, critics rarely report the number and details of their claimed replications, so a good estimate of the probability of their never seeing a significant result cannot be made. It should be noted that experimenter effectyis only one of many confounding problems. involved in the quest for the significant replication. For example, selection of subjects, experimenters, task conditions, experimental protocols, statistical procedures, environmental conditions, feedback techniques and generation of random numbers are all reflected in the ultimate outcome of an experiment. Regardless of how well controlled an experiment may be, a change in any one of these factors will affect the entire experiment in a complex, poorly understood way. In any case, experimenter bias is unavoidable, and we must rely on well-controlled experiments with features like automated data recording to help eliminate this bias. In spite of tight controls, however, it is known that even parapsychologists who would like to replicate RNG studies cannot guarantee significant results. Thus, critics would perhaps claim that any reported significant studies are due more to unconscious or intentional experimenter bias (i.e. fraud or carelessness) rather than there being a real effect. To address the issue of what effect different experimenters may have had in the reported RNG experiments, we ran two analyses on the survey data. The first involved calculating the overall z score obtained by each principal investigator; the second was a test of the homogeneity of mean z scores repored by different investigators. Combined z score results | Table 2 shows a combined z and mean z calculated for each of 28 different principal investigators. This list is comprised of only those studies where sufficient detail was published for us to calculate z scores from the number of trials and hits in an experiment (332 total - 35 partially detailed experiments = 297 experiments). The z(overall) scores per investigator were calculated by summing the z scores for all experiments contributed by that investigator and dividing by the square root of the number of experiments. In effect, this weights each experiment equally, regardless of the number of trials (bits) actually used in the experiment. (The number of trials run in these experiments ranged between 144 and 2 million.) B-15 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3 |
nea DML oe pees I sy Mg ea For ‘peteasebN Glide SSAFABIRs- 00789R002200400001-3 Table 2: Overall z score per investigator Principal investigator * References | Experiments z(overall) Andre 2.413 Bierman Braud Broughton Debes Dunne Edge Giesler Heseltine Hill Honorton Houtkooper Jungerman Kelly Matas May 2 Millar Morris Morrison Palmer Pantas Radin Randall Schechter Schmeidler Schmidt Shafer Winnett MNreNNe wW FRE On nN el ell el ee ae UMNORNADARW 1 This is the name of the first author as listed in the references. Pd The study by May, Humphrey, and Hubbard (1980) is not included in this survey because their sequential analysis data collection technique is not amenable to z score analysis. As seen in Table 2, the overall z scores for these investigators ranged between ~2.384 to 13.224. The grand total z score, obtained by summing the 28 z scores and dividing by V 28 is z = 8.548, for an overall p < 1.27 x 10“ (2-tailed). If we remove Schmidt’s 30 studies, since he obtained the largest overall z score and is responsible for the largest number of references in our survey, we find the grand total z = 6.160, p < 7.31 x 107° (2-tailed). If we also remove the Princeton data, which comprise nearly half of the reported experiments, we get a grand total z = 5.480, p < 4.25 x 1078 (2-tailed). Thus, after removing the two largest contributors to the database, we are left with a fairly impressive overall result: Odds against chance of about 1 in 23,000,000. In addition, we find that 39% (11/28) of the experimenters obtained overall 2-tailed significance and 68% (19/28) obtained positive z scores. Test for homogeneity of effect size Do different experimenters tend to observe about the same effects in their experiments? Or are there some individuals who consistently obtain significant results and others do not? In the present context, to test for homogeneity of effect size among UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
a a ; Meta-Anaiuysis P 1 | Approved For ReleaselOGUAR SGHFABI6-00789R002200400001-3 different experimenters, we believe it makes more sense to test the individual z scores obtained in each experiment rather than use effect sizes such as d, d’, r, or so on, as discussed by Rosenthal (1984) and others. The reason is the following: Effect size may be defined as significance test = [effect size] x [size of study] where “significance test” can be a z, t, r, chi-square, or any other statistical test. In the studies we found in the literature, it is clear that if the effect size were constant regardless of the size of the study (say, N trials), we should be observing enormous z scores when N is even moderately large. For example, if an investigator ran a study with N = 100 and obtained a z score = 20, this would imply that the effect size (defined as r = 2Ap = ziVN for a binary RNG) would be r= 2.0/¥100 = 2.0/10.0 = .2. If this effect size were constant, then if we ran the same experiment again but with N = 10000, the z score for this experiment would be z = (2Ap) ¥10000 = .2( 100) = 20.0. Z scores of this magnitude are simply not reported in individual experiments, thus our effect size is almost certainly n-dependent. Indeed, this phenomenon has been observed repeatedly in a variety of experiments and has been called a goal-directed effect (e.g. Kennedy, 1978; May, Radin et al, 1985; Schmidt, 1974). , To take the effect size n-dependence into account, we must multiply the effect size by a function of the size of the study, which brings us back to a significance test, as noted above. For the sake of convenience, we can use the z score calculated for each experiment. To see whether different experimenters reported about the same magnitude z scores, we performed an analysis of variance; the results are shown in Table 3 (on the next page). It is clear from the results of the ANOVA that different experimenters do indeed obtain different mean z scores, although with 25% (7/28) of the principal investigators reporting mean z scores greater than 2 or less than —2, it is not the case that only one or two experimenters have obtained large mean z scores. B-17 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
Meta-Analysis Part 1 Approved For Release 200 Ode GS HED 00789R002200400001-3 Table 3: Results of one-way analysis of variance Grand mean N MEAN Z SD SE 297 0.5979 1.5823 0.0918 Person N MEAN Z SD SE Andre 4 1.2065 1.9091 0.9546 Bierman 2 2.7570 1.3863 0.9803 Braud 4 1.8797 0.9378 0.4687 : Broughten 4 ~0.2347 0.3048 0.1524 Debes 8 0.1260 1.8205 0.6437 Dunne 144 0.3386 1.1842 0.0987 Edge 10 0.1166 2.0067 0.6346 Giesler 12 0.7778 0.8011 0.2313 Heseltine 19 -0.0885 1.7124 0.3928 Hill 1 2.9498 , Honorton 14 0.4071 1.1328 0.3028 Houtkooper 4 0.4906 1.4944 0.7472 Jungerman 1 2.3322 Kelly 2 2.3799 0.3015 0.2132 Matas 2 0.3625 2.9522 2.0875 May 1 -2.3841 Millar 2 —-0.6187 1.6406 1.1601 Morris 5 0.8206 1.0562 0.4723 * Morrison 3 0.7746 0.4926 0.2844 Palmer 1 1.7500 Pantas 4 0.7625 2.4453 1.2226 Radin 4 2.1712 0.8822 0.4411 Randall 6 ~0.0120 1.1753 0.4798 Schechter 2 -0.7496 4.2411 2.9989 Schmeidler 1 -1.2728 Schmidt 30. 2.4144 2.0341 0.3714 Shafer 2 -1.0178 1.1158 0.7890 Winnett 5 -0.0396 0.5795 0.2592 SOURCE SS df MS F p person 197.9629 27 7.3320 3.631 2.78x 10 8 error 543.1467 269 2.0191 B-18 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3