Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2
Meta-analysis of Forced-Choice Precognition Experiments
OVERALL CUMULATION
Evidence for overall directional hitting and for successful prediction of
hitting and missing tendencies is strong.
1 As shown in the top part ofTable
1, the overall results are highly significant. The meanpredicted z
is twice as
large as the mean directional z, indicating the advantage of making focused
predictions (and the lawfulness implied by being able to do so).
Thirty
percent of the studies show overall significant hitting at the 5% level. Nearly
40% are significant on the basis of the investigators’ predictions.
TABLE 1:
Overall Precognition Significance Levels
Zdir
Zpred
Mean
0.69
138
Standard Deviation
2.65
236
Combined (Stouffer) z
12.14
24.23
pz
6 x 10'27
4 x 10’52
Filedrawer Estimate
16,529
66,687
Lower 95% Confidence Estimate
0.44
1.16
Lower 99% Confidence Estimate
034
1.07
Lower 99.9% Confidence Estimate
0.22
0.97
Lower-bound confidence estimates of the meanz-scores displayed in the
bottom portion of Table 1 indicate that the mean directional and predicted
z-scores are well above zero at the 95%, 99%, and 99.9% confidence levels.
Significance levels, not surprisingly, are related to sample
size.
The
correlation (r) is 0.151 for the directional z’s (307 df,p = .0044), and for the
1The statistical analyses presented here were performed using SYSTAT (Wilkinson,
1988). When f-tests are reported on samples with unequal variances, they are calculated using
the separate variances within groups for the error and degrees offreedom following Brownlee
(1965). Unless otherwise specified, p-Ievels are one-tailed.
Psychophysical Research Laboratories
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2
Meta-analysis of Forced-Choice Precognition Experiments
predicted z’s, r is 0.242 (307 df, p =
8.4 x 10"^).
Directionally significant
studies have a mean sample size that
is 37% larger than the mean for
directionally nonsignificant studies.
Using the predicted z-score criterion,
significant studies have a mean sample size that is more than double that of
the nonsignificant studies.
The effect size analysis is presented in Table 2.
Both directional and
predicted outcomes are
significantly above zero, and
again, the mean
predicted effect size is twice as large as the directional mean ES.
TABLE 2:
Overall Effect Sizes
ES&ir
ESprtd
Mean
0.022
0.041
Standard Deviation
0.098
0.092
/(308)
4.01
7.88
P»
4 x 10‘5
3 x 10'14
Lower 95% Confidence Limit
0.012
0.032
Lower 99% Confidence Limit
0.008
0.029
Lower 99.9% Confidence Limit
0.005
0.025
Replication across investigators
Virtually the same picture emerges when the cumulation
is by
investigator rather than study as the unit of analysis. The combined z’s are
12.71
for directional outcomes and 22.12 for predicted outcomes.
Twenty-four of the 62 investigators (39%) have directional outcomes
significant at the 5% level, and 39 investigators (63%) have significant
predicted outcomes. The mean (investigator) directional effect size is 0.036
(sd = .091), and the mean predictedES is 0.050 (sd = .087).
These results indicate a substantial level of cross-investigator replicability
and
directly contradict the claim of critics such as Akers (1988)
that
Psychophysical Research Laboratories
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2
Meta-analysis of Forced-Choice Precognition Experiments
successful parapsychological outcomes are achieved by only a few
investigators.
The Filedrawer Problem
A well-known reporting bias exists throughout the behavioral sciences
favoring publication of “significant” studies
(e.g.,
Sterling,
1959).
The
extreme view of this “filedrawer problem,” as Robert Rosenthal describes
it, “is that the journals are filled with the 5% of the studies that show type I
errors, while the filedrawers back at the lab are filled with the 95% of the
studies that show nonsignificance.
.
.” (Rosenthal,
1984,
p.
108).
Recognizing the importance of this problem, the Parapsychological
Association in 1975 adopted an official policy against selective reporting of
positive results. Examination of the parapsychological literature shows that
nonsignificant results are frequently published and
in the precognition
database, 60% to 70% of the studies have reported nonsigiiificant results.
Nevertheless, 75% of the precognition studies were published before 1975,
and we must ask to what extent selective publication bias could account for
the cumulative effects we observe.2
The central section ofTable 1 uses Rosenthal’s (1984) filedrawer statistic
to estimate the number of unreported studies with z-scores averaging zero
that would be necessary to reduce the known database to nonsignificance.
The filedrawer estimate suggests that over 50 unreported studies must exist
for each reported study to reduce the cumulative
hitting (directional)
outcomes to a nonsignificant
level.
For the predicted outcomes, the
filedrawer ratio is more than 200:1.
Another approach to the filedrawer problem
is described by Robyn
Dawes (Dawes, Landman and Williams, 1984; personal communication to
Honorton, July
14, 1988).
Dawes calculates the expected mean z
and
variance for various significance levels on the assumption that reported
significant outcomes reflect nothing more than type I error. He then tests
2Analyses indicate no significant differences in the magnitude ofreported study outcomes
before and after 1975.
The mean directional z-score for studies prior to 1975
is 0.719
(sd = 2.6) and for studies reported thereafter the mean is 0.605 (sd = 2.81) (t = 0325, 307
df,p =
.746, 2-tailed). For predicted 2-scores, the comparable values are 1.43 (sd = 2.29)
and 1.22 (sd = 2.60); t = 0.675, 307 df,p = .675, 2-tailed).
Psychophysical Research Laboratories
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2