Approved For Release
SSiFTED'00789*0^20^90001^
Figure 4. Method of estimating filedrawer size (see text)
(4a)
Now that we had a
full description of curves
1 and
2, we assumed that the area labeled “b” in
Figure 4a was the number of observed studies with
|z| < 1,65 (188 - 76 = 112), that area “c + d”
was composed of 76 observed studies with
|z| ^ 1.65, and that the total area “a + b + c + d” was
calculated at 283 studies 5
.
Doing the subtraction 283 - 112 - 76 = 95, we estimate 95 unreported
or unretrieved nonsignficant studies in the actual filedrawer. We believe that this number is a more
realistic estimate than the 4700
studies determined by equation
(2).
In
fact,
studies would
require each of 10 parapsychology laboratories to have
filed only about 0.6 studies per year over
the
15 year survey period
(as opposed to 2.6 per month ,
as 4700 studies would require).
Now
if we combine
the
188
observed
survey
studies
with
new,
nonsignificant
z
scores
(generated by Monte Carlo technique with z chosen
at random from a normal
distribution, and
bounded between 10 ~25 and ±1.64), we find of the 283 resulting studies, mean z = .462, overall
z = 7,768, and p <
8.03 x 10 ~ 15
.
Again applying formula
(2) to the new values
(for the sake of
curiosity), we find X = 4078 additional nonsignificant studies needed to bring this overall p value
down to p =
.05,
2-tailed.
Finally, combining
all survey, newly estimated, and Princeton studies (188+144), we find that
for the 425
total studies the mean
z =
.420,
overall
z = 8.684, and p < 3.9 x 10~18
.
Applying
formula
(2), we find we would need 7778 additional nonsignificant studies in the filedrawer. Thus,
from several different perspectives,
it seems that the filedrawer issue is not as serious a problem as
many have thought.
5.
This calculation was based on the curve-fitted standard deviations for the two Gaussian curves and the
observed number of studies in areas b and c + d.
B-13
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5
Wp>pr6vdi*iFBEirtRelease ^^(^^g^pj)^^-00789R002200390001 -5
Incidentally, testing the standard deviation of the z scores observed in these 425 studies
(sd =
1.414) against the expected variance of 1.0 for a normal, unperturbed z distribution, results in a
chi-square value of 853.7 (424 df), for a p < 5.9 x 10“«.
Table
1 (below) and 7
(at end of the
paper) summarize these findings.
Table
1.
Summary of z score analyses
Source of
reference
studies
(N)
z
Zz's
Vn-
p
(2-tail)
z score
sd
variance test
against
a =
X
p
(2-tail)
Survey
188
0.738
10.114
4.9 x 10
1.739
568.5
- 47
4.9 x 10
Princeton
144
0.339
4.063
4.9 x 10' 5
1.184
201.9
0.001
Estimated
(simulated)
-0.084
-0.820
0.412
0.661
41.5
0.51
filedrawer
Combined
425
0.420
8.684
3.9 x 10‘ 18
1.414
853.7
5.9 x 10' 34
4 . Experiments are not replicable
Occasional
significant
effects may be
impressive, but the
existence
of the claimed anomaly
cannot be established on the basis of results reported by only a few individuals. 6
The same effect
must be replicated by many others.
Is
it true, as Kurtz (1980)
claims, that
The basic problem
...
is the lack of replicability by other experimenters.
Apparently,
some experimenters
a relative few — are able to get similar results, but most are
unable to do so.
(Italics in the original, p.12)
In fact, of the 332 experiments we considered, 78.6% failed to reach significant levels.
It
is
hardly surprising, then, that on the basis of examining individual experiments
it is easy to reach the
conclusion that the effect
is elusive and non-replicable.
At
this
failure
rate, nearly
4 out of 5
experiments will
fail to reject the null hypothesis.
(Of course,
if just chance were operating, 19 out
of 20 experiments would
fail to reject the null hypothesis.)
.
Actually, compared to experimental psychology, experimental parapsychology
is in much better
shape as far as replication rates go.
Honorton (1975),
for instance, describes a study by Bozarth
and Roberts (1972), who, in a survey of 1334 articles from psychology journals, found only eight
articles involving replications of previously published work.
In this present meta-analysis alone
parapsychology is a factor of 40 ahead of psychology.
B-14
Approved For Release 2uuu/08/08
: Cl
00789R002200390001-5
Approved For Release
Another
reason why
it may be
difficult
to produce
significant
experiments
at
will
is
the
well-known “experimenter effect”
(Rosenthal, 1976).
This effect
is ubiquitous to all the sciences,
but parapsychology seems to be especially vulnerable
(see,
e.g. White,
1977).
The experimenter
effect may help explain why some
critics of parapsychology claim that they have never obtained
significant results
in their attempts to replicate
psi experiments
(e.g.
Kurtz,
1981,
p.16; Neher,
1980, p.147).
Of course, the odds
of never obtaining a
significant study can be astronomical,
depending on the number of studies conducted.
Unfortunately,
critics rarely report the number
and details of their claimed replications, so a good estimate of the probability of their never seeing a
significant result cannot be made.
It should be noted that experimenter effect is only one of many confounding problems involved
in the quest for the significant replication.
For example, selection of subjects, experimenters, task
conditions,
experimental
protocols,
statistical
procedures,
environmental
conditions,
feedback
techniques and generation
of random numbers are
all reflected
in the ultimate outcome
of an
experiment.
Regardless of how well controlled an experiment may be, a change in any one of these
factors
will affect the entire experiment in a complex, poorly understood way.
In any case, experimenter bias is unavoidable, and we must rely on well-controlled experiments
with features like automated data recording to help eliminate this bias.
In spite of tight controls,
however,
it
is known that even parapsychologists who would like to replicate RNG studies cannot
guarantee significant results.
Thus, critics would perhaps claim that any reported significant studies
are due more to unconscious or intentional experimenter bias
(i.e. fraud or carelessness) rather
than there being a real effect.
To address the issue of what effect different experimenters may have had in the reported RNG
experiments, we ran two analyses on the survey data.
The
first involved calculating the overall z
score obtained by each principal investigator; the second was a test of the homogeneity of mean z
scores reported by different investigators.
Combined z score results
Table 2 shows a combined z and mean z calculated for each of 28
different principal investigators.
This
list
is comprised of only those studies where sufficient detail
was published for us to calculate z scores from the number of trials and hits in an experiment (332
total - 35 partially detailed experiments = 297 experiments).
The z(overall) scores per investigator
were calculated by summing the
z scores for
all experiments contributed by that investigator and
dividing by the square root of the number of experiments.
In effect, this weights each experiment
equally, regardless of the number of trials
(bits) actually used in the experiment.
(The number of
trials run in these experiments ranged between 144 and 2 million.)
B-15
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5