Approved For Release 2U(MC&A<$SI$tfB)0789R002200390001-5
Psi Experiments with Random Number Generators:
Meta-Analysis Part 1
Dean
I. Radin
Edwin C. May
Martha
J. Thomson
SRI International
Menlo Park,
California
ABSTRACT: A meta-analysis of 332 psi experiments involving binary random number
generators
is described.
The combined binomial probability for data reported
in 56
references published from 1969-1984
is p
10“«.
A "fiiedrawer” analysis reveals
that over 4500 additional) nonsignificant, unpublished or unretrieved studies would be
required
to bring the overall
result down
to a nonsignificant
level.
Using a novel
approach, we estimate the actual size of the “fiiedrawer” to be 95 studies.
Adding the
equivalent of 95 nonsignificant studies to the existing data results in p ^ 10_1B
, while a
meta-analysis of 98 reported
control
studies
results
in p »
.78.
An
analysis
of
variance indicates that experimenters’ mean z scores are significantly different from
each
other.
We
discuss
an
approach
and
propose
criteria
for
performing
a
quality-weighted
analysis on
the
existing
data.
We conclude
that
the prima facie
evidence
supports
the
notion
that
observers’
intentions can
affect
the
statistical
properties of truly random number generators.
INTRODUCTION
This
is Part
1 of a two part meta-analysis of psi experiments involving truly random number
generators (RNG) published from 1969-1984.
This part describes the results of a
"first-pass”
analysis,
in
which
the
published
data
was
taken
at
face
value.
Part
will
report
on
a
quality-weighted analysis
in which the
results of each experiment
(in terms of z score)
will be
evaluated on each of a dozen criteria to produce an adjusted z score reflecting that experiment’s
overall quality.
Background: On the scent of a trail
When Albert Einstein was asked about his way of thinking, he reportedly replied, “All I have is
the stubbornness of a mule; no, that’s not quite
all,
I also have a nose”
(Bower,
1985, p.330).
What he meant was that he was not only extraordinarily obstinate in tracking down solutions to
problems, he was also able to sniff out when he was on the right track. The centennial anniversary
B-l
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5
^pVove^or'kelease 2(fc|bfojB©kA^SIFtE©0789R002200390001 -5
of the American Society for Psychical Research, celebrated this year (1985), clearly demonstrates
that parapsychologists have displayed Einstein’s stubbornness over the years.
One question we
might ask after 100 years, however, is whether the parapsychological nose has been sniffing along a
clearly defined
trail, and
if so,
is the
trail likely to grow more fragrant or more noxious as we
progress?
There
is evidence that the nose has not been shirking its duty.
This can be seen in the single
most predictable feature found in the parapsychological literature, that
is, the perennial call for a
replicable experiment.
The ideal experiment
is supposed to produce a significant result regardless
of the phase of the moon, the price of pork bellies, and the experimenter’s shoe size.
This Quest
for
replicable
experiments
is by no means unique
to
parapsychology,
however.
Social and
behavioral scientists in general have been acutely aware of the slow progress in the "softer” sciences
as compared to the natural sciences such as physics,
chemistry, and
biology.
In experimental
psychology, for example, Epstein (1980) has stated,
Psychological research
is rapidly approaching a crisis as the result of extremely
inefficient procedures for establishing replicable generalizations.
The traditional
solution of attempting to obtain a high degree of control in the laboratory
is often
ineffective because much human behavior
is so sensitive to incidental sources of
stimulation that adequate control cannot be achieved....
Not only are experimental
findings often difficult to replicate when there are the slightest alterations in
conditions, but even attempts at exact replication frequently
fail.
(p. 790)
Many observers of parapsychology (both within and outside the field) claim that the repeatable
parapsychological experiment does not exist.
For example, Beloff (1977) has written, "There
is
still no repeatable
[psi] experiment on the basis of which any competent investigator can verify a
given phenomenon for himself” (p.759).
Critics of the field have pointed to the lack of replicability
as perhaps
the
single most
serious problem
in parapsychology
(e.g.
Kurtz,
1981,
p.12).
In
response, proponents often point to significant psi studies involving ESP card-guessing (Honorton,
1975),
ganzfeld stimulation
(Honorton,
1978), remote perception
(Dunne,
Jahn, and Nelson,
1983), and RNGs (May, Hubbard and Humphrey, 1980) to indicate that there are some significant
replications.
The problem is that from different perspectives the proponents and critics are both right.
There
are indeed many psi experiments that have been repeated, but whether they are considered robust,
successful replications is the crux of the debate. One of the primary reasons for this debate, in our
opinion,
is because the traditional approach of assessing the results of a set of related studies
is by
descriptive literature review.
Within parapsychology there are many excellent examples of such
reviews
(e.g.
Carpenter,
1977; Palmer,
1982; Rush,
1982;
Schmeidler,
1984;
Stanford,
1977;
Stanford,
1984).
Unfortunately, what one has typically learned after studying such a review
is a
hodge-podge of variables, conditions, and p-values.
Rarely is one left with a quantitative statement
of the degree of significance obtained in the studies as a whole.
Addressing
this
issue
empirically,
Cooper and
Rosenthal
(1980)
demonstrated
that when
knowledgeable individuals are instructed to make judgments about the overall significance of a set
of studies based on their readine of a comprehensive, descriptive literature review,
it is possible for
B_2
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001 -5
Approved For Release U<N£loAS&lFJ>iD-00789R00l20W9b00i
a
-5
them to draw conclusions that are completely the opposite of the results obtained when the same
studies are summarized by more explicit, quantitative methods.
Given the difficulties in assessing evidence from existing psi studies,
is the replication trail likely
to be heading — to reinvoke our metaphor — towards a flowering meadow or something decidedly
less pleasant?
In
general, we believe that the prospects are aromatic.
In the
last few years,
quantitative techniques of combining and comparing research results in systematic ways have been
developed — called meta-analysis (Rosenthal, 1984) — that show great promise in demonstrating
that some areas of social science have been progressing much better than previously thought.
In
parapsychology,
initial meta-analyses applied to ganzfeld research
(Honorton,
1985), hypnotic
induction (Schechter, 1984), RNG studies (May, Hubbard and Humphrey, 1980; Nelson, Dunne
and Jahn, 1984; Tart,
1983), and remote viewing (Dunne, Jahn, and Nelson, 1983) have shown
that the overall evidence for these psi phenomena
is actually quite strong.
Because meta-analysis involves the aggregation of results of numerous studies, several criticisms
of this technique have been raised (Rosenthal, 1984, p.124-132).
Perhaps the three categories of
criticism most pertinant to review of parapsychological data are the following:
First, authors may
tend
to
report
only
the
studies
with
significant
results and
leave
the
nonsignificant
studies
unpublished (called the filedrawer problem) .
Second, the meta-analysis combines poorer quality
studies with better studies.
And third, meta-analysis may be comparing “apples and oranges” by
combining different experiments studying different variables.
The first two problems may inflate the estimate of an overall effect; the third criticism may make
the overall summary difficult or impossible to interpret.
In the present meta-analysis, however, we
actually are interested in whether these psi experiments have bome fruit, not whether they have
borne specific flavors of apples or oranges.
In other words, we are not concerned with whether
hypnotic induction, say, has an effect on RNG outputs, but whether there
is evidence for any psi
effect on RNG outputs.
Thus, in this investigation we have concentrated on the filedrawer issue
(in
this report) and the quality of studies
(to be described in Part 2 of this study).
OVERVIEW OF A TYPICAL RNG EXPERIMENT
The typical psi experiment with RNGs involves three main components:
An observer
(e.g. a
human,
goldfish,
cat or dog), a truly random number generator based on radioactive decay or
electronic noise, and an experimental task linking the observer with the device, such as a video
game, a set of instructions, a need to keep a heat lamp on or avoid a shock, and so on. The aim of
these experiments is to show that the instructions (when humans are involved) or the induced need
(when animals or plants are involved) are associated in some way — but not necessarily causally —
to the statistical output of the RNG.
For example, say an RNG was designed to produce 100 random bits at the press of a button. An
individual in this experiment might see a digital display of the number of l’s (called hits) produced
immediately after he or she pressed a button. The instructions in the experiment would typically be
to get as many hits as possible for each button press. The results of many presses, or trials, would
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001 -5