MApptotoa FW Release aTOipf
CIA|PfE»-00789R002200390001-5
2 . Basic statistical assumptions are violated
This criticism incorporates such problems as the improper application of statistics to a particular
experimental design, violation of assumptions of independence, performing multiple analyses on the
same
data, and
so
on.
In
this meta-analysis, one
of the reasons we only
considered binary
generators was to simplify the statistical assumptions to the point where we could avoid many such
problems.
Another
reason
was
to
avoid
the
“apples
vs.
oranges”
comparison problem we
mentioned earlier.
Because we were interested only in RNG experiments that reported (or where
we could calculate) the number of hits and trials, we were in fact comparing apples only with apples
(actually
bits
with
bits).
While
it
is
true
that
there were many
different
psychological and
physiological conditions involved in these experiments, as well as human and non-human subjects,
the underlying question we asked was the same for each experiment: What was the behavior of the
RNG as compared to the pre-specified direction of effort defined in the experimental task?
The
statistics
in
these RNG
experiments
are
described by
the
well
understood
binomial
distribution, and the central limit theorem allows us to use the normal approximation to further
simplify the statistical treatment for the range of trials observed in the data (200 to 2 million trials in
a single experiment).
Violation
of the assumption
of independence
can be
the
downfall
of an
otherwise
tightly
controlled experiment.
In the present case, however, the random events are based on sources that
are quantum-mechanical (QM) in nature — radioactive decay of alpha, beta, or gamma particles,
or electronic noise from various semiconductor devices such as tunnel diodes. QM theory states
that
random
numbers
based
on QM
events
are
in
principle
indeterminant
and
therefore
independent of each other, provided that the RNG device is properly designed and constructed*
In this meta-analysis, under the null hypothesis of no psi effect we can assume independence of
random
bits.
Note
that the assumption
of independence among bits does not override proper
concern about whether the RNGs used in the experiments produced bits with equal probabilities.
It
is entirely possible, for example, to produce bits that are completely independent, but with p(l) =
.6 and p(0) =
.4.
This
is addressed in point 5 below.
3 . Only significant studies are published — the Filedrawer problem
The filedrawer problem,
in which only significant studies are reported and the nonsignificant
studies languish in filedrawers,
will inflate the results of a meta-analysis because there will be too
many small p values
(or
equivalently,
to many
large
z
scores).
To
address
this problem, we
followed a procedure proposed by Rosenthal (1984, p.
108),
in which the average
z score
for
all
combined studies
is applied to the formula:
4. Note that some of the diodes used in noise-based RNGs are not QM in nature,
RNGs that use avalanche
diodes,
for example, derive their noise from fluctuations in charge carrier multiplication, which can be
described by classical electromagnetic theory.
B-8
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5
Approved For Release ^^|!ftS8tFI€B_00789R0^®^^ta&
K[KZ
2 - 2.72]
(1)
2.72
where K
is the number of studies combined, Z
is the mean Z obtained for the K studies, and X
is
the number of new, filed , or unretrieved studies averaging null results required to bring the new
overall p level to a designated level.
The value 2.72 in equation
(1)
is the square of 1.65, the z
value
for p =
.05
(the p
level
that Rosenthal
uses).
To make our
filedrawer
estimate more
conservative, we chose a 2-tailed p =
.05,
z =
1.96.
Thus the formula we used was
K[KZ
2 - 3.92]
We shall consider the Princeton studies separately from the rest of the survey because we have
good reason to believe that all of the Princeton data was, in fact, published, thus their data has no
filedrawer problem.
[(Publishing all data
is a part of the Princeton Laboratory’s philosophy (Jahn,
1982)].
In the
188 survey experiments, the mean z score = 0.738.
A mean
z of this value over 188
experiments produces an overall z = 10.114, for a 2-tailed p < 4.9 x 10“24 (see Table
1).
Note that
this method of estimating the overall probability
is more accurate than determining the binomial
probability
of 71
successes out
of
188
samples
at p <
.05,
as described
earlier
in
this paper.
Applying Z =
.738 and K =
188
to formula
(2)
results
in X = 4723.
This means
that 4723
additional studies averaging null results would have to be
filed away in researchers’
filedrawers to
bring the overall z score down to a 2-tailed nonsignificant level.
According
to
Rosenthal
(1984),
the number X
has
different meanings depending on
the
research
context.
In some
areas
of
research
(say
genetic
engineering),
perhaps
or
unpublished
or unretrieved
studies might be considered reasonable.
In other areas
(say
child
development), perhaps 200 to 500 filedrawer studies might be a reasonable estimate.
Rosenthal
(1984, p.110) proposes the following general guideline: “Perhaps we could regard as robust to the
file drawer problem any combined results for which the tolerance level (X) reaches 5 K +
10.”
Thus — not counting the Princeton data — since X
is more than 25 times larger than the
observed number of studies, we could
state, based on Rosenthal’s guideline,
that the observed
effect
is robust.
Indeed,
for this many unpublished or unretrieved studies to exist
it would have
required each
of
10 parapsychology
laboratories
to have
continuously produced nonsignificant
studies at the rate of 2.6 per month over the 15 years surveyed.
This is an unlikely scenario given
the limited number of researchers performing these experiments over the years and the time and
effort typically required to perform a single study.
If we apply the same procedure to the Princeton data of 144 studies, we find the mean z = .339,
overall z = 4.063, and p < 4.85 x 10 “5
.
Plugging these values into formula
2, we find we would
need X = 476 additional unpublished or unretrieved studies averaging null results.
But as previously
B-9
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5