: CIA-RDP96-00789R002200610001-0 Approved For Release 2000/08/08 284 The Jou rnal of Pa rapsych ology ‘TABLE | OVERALL SIGNIFICANCE LEVEL AND EFFECT SIZE z ES anne Mean 0.65 0.020 SD 2.68 0.100 Lower 95% confidence estimate 0.40 0.011 Combined z = 11.41, p = 6.3 x 10 °*° “Fail-safe N” = 14,268 WES) = 3.51, 308 df, p = .00025 ee population is students (in approximately 40% of the studies); the least frequently used populations are the experimenters themselves and animals (each used in about 5% of the studies). Though a few studies tested subjects through the mail, more typ- ically subjects were tested in person, either individually or in groups. ‘Target selection methods included no randomization at all (studies using “quasi-random” naturalistic events), informal methods includ- ing manual card-shuffling or dice-throwing, and formal methods, primarily random number tables or random number generators. The time interval between the subjects’ responses and target gen- eration varied from less than one second to one year. OVERALL CUMULATION Evidence for an overall effect is strong. As shown in the top part of Table 1, the overall results are highly significant.” Lower bound (one-tailed) 95% confidence estimates of the mean z score and ES are displayed in the bottom portion of Table I. ‘Ninety-two studies (30%) show significant hitting at the 5% level, and significant outcomes are contributed by 40 different investiga- tors. The z scores correlate significantly with sample size: (307) = .156, p = .003. The mean number of trials for significant studies is 34% larger than the mean number of trials for nonsignificant stud- ies. ? The statistical analyses presented here were performed using SYSTAT (Wilk- inson, 1988). When ¢ tests are reported on samples with unequal variances, they are culeulated using the separate virtinces within groups for the error and degrees of freedom following Brownlee (1965). Unless otherwise specified, p levels are one- tailed. Combined 2z’s are based on Stouffer’s method (Rosenthal, 1984). A Meta-Analysis of Forced-Choice Precognilion Experiments 285 , 0 (e) o oO (e) Investigator yrtted titi CIA-RDP96-00789R002200610001-0 -0,2 -0.1 -0.0 OF 0.2 03 Ot 0.5 Mean effect size Figure 1. Mean effect size by investigator. N = G62 investigators. Replication Across Investigators Virtually the same picture emerges when the cumulation 1s investigator rather than study as the unit of analysis; the combin z is 12.13, and 23 of the 62 investigators (37%) have overall oyy comes significant at the 5% level: The mean (investigator) effect siae is 0.033 (SD = .093). LL There is a significant difference in the mean ES across invest gators, but it is surprisingly small: Kruskal-Wallis one-way ANOV@ by ranks, x°(61) = 82.71, p = .034. The effect is clearly not due ® a few major contributors. If investigators contributing more tha three studies are eliminated, leaving 33 investigators, the combing z is still 6.00 (p = 1.25 x 107°) and the mean ES is .028 (SD = 091). Figure | shows the mean effect sizes by investigator. ‘These results indicate substantial cross-investigator replicability and directly contradict the claim of critics such as Akers (1987) that efedse 2000/08/08 fe)
286 The Journal of Parapsychology successful parapsychological outcomes are achieved by only a few investigators. oO t Poe Filedrawer Problem oO Oo = A well-known reporting bias exists throughout the behavioral apences favoring publication of “significant” studies (e.g., Sterling, £859). The extreme view of this “filedrawer problem” is that “the Qunals are filled with the 5% of the studies that show Type I er- vrs, while the filedrawers back at the lab are filled with the 95% of & studies that show nonsignificance...” (Rosenthal, 1984, p. 108). Recognizing the importance of this problem, the Parapsychological Ssociation in 1975 adopted an official policy against selective re- rting of positive results.” Examination of the parapsychological lit- ture shows that nonsignificant results are frequently published, Mad, in the precognition database, 70% of the studies have reported RRasignificant results. Nevertheless, 75% of the precognition studies were published before 1975, and we must ask to what extent selec- <. publication bias could account for the cumulative effects we ob- @ Phe central section of Table 1 uses Rosenthal’s (LO84) “failsafe @’ statistic to estimate the number of unreported studies with z Bores averaging zero that would be necessary to reduce the known @plabase to nonsignificance. ‘The filedrawer estimate indicates that Ber 46 unreported studies must exist for each reported study to Qiduce the cumulative outcome to a nonsignificant level. ® A different approach to the filedrawer problem is described by iwes, Landman, and Williams (1984; personal communication Bom Dawes to Honorton, July 14, 1988). Their truncated normal @erve analysis, like Rosenthal’s “fail-safe N.” is based on normal qurve assumptions. ‘Uheir null hypothesis is that z scores above some Gitical level (e.g., z = 1.65, 1.96, etc.) are randomly sampled from 940,1) above that critical level. The alternative to the null hypothesis @ that, because there is some real effect, the distribution of z’s is Gifted to the right of 0 and the z’s will be larger than predicted by @e null. For a critical level of z = 1.65, the expected mean z is 2.06 ad the variance is .14. In the precognition database, there are 92 Shadies with z’s > 1.65. Their average is 3.61, not 2.06 as predicted * Analyses indicate no significant difference in the magnitude of reported study outconies | ind after The mean ES for studies prior to [975 is 0.021 (sD ), and for studies reported thereafter the mean is 0.017 (S20 = .106): 1307) 28, p = .782, two-tailed. A Meta-Analysis of Forced-Choice Precognition Experiments 287 by the null hypothesis. Since the variance of the normal truncated above 1.65 is .14, the ¢est z (using the Central Limit Theorem) com- paring 3.61 to 2.06 is 39.84 [1.55 divided by (.14/92)'"]. Here, p is & virtually zero. Similar results are found with cut points of 1.96, 2.33, 4 and 2.58. On the basis of these analyses, we conclude that the cumulative © significance of the precognition studies cannot satisfactorily be ex- %& plained by selective reporting. 00 OUTLIER REDUCTION 0789R0022006 Although the overall z scores and effect sizes cannot reasonably be attributed to chance, inspection of the standard deviations ino Table 1 indicates that the study outcomes are extremely heteroge- neous. Given the diversity of methods, subject populations, anda, other study features that characterize this research domain, this isQ not surprising. a The study outcomes are in fact extremely heterogeneous. Al-<f though a major objective of this meta-analysis is to account for theO variability across studies by blocking on differences in study quality,¢6 procedural features, and sampling characteristics, the database© clearly contains extreme outliers. The z scores range from —5.1 too 19.6, a 25-sigma spread! The standardized index of kurtosis (g2) is 9.47, suggesting that the tails of the distribution are much too longS for a normal distribution. ar. We eliminated the extreme outliers by performing a “10 percent® trim” on the study z scores (Barnett & Lewis, 1978). This involves eliminating studies with z scores in the upper and lower 10% of. the distribution, and results in an adjusted sample of 248 studies. ‘They trimmed z scores range from — 2.24 to 3.21 (@g, = — 1.1). The re-w vised z scores and effect sizes are presented in Table 2. 2 Elimination of extreme outliers reduces the combined z scores bygy approximately one half, but the outcomes remain highly significant. Twenty-five percent of the studies (62/248) show overall significant hitting at the 5% level. Lower bound confidence estimates show thaig, the mean z’s and effect sizes are above 0 at the 95% confidence level Elimination of outliers reduces the total number of investigators from 62 to 57, but the results remain basically the same when the analyses are based on invesuigators rather than studies. “The com- bined z is 6.84; 18 of the 57 investigators (31.6%) have overall. sig- 9