Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
Statistical Science
1991, Veil. 6, No.
4, 363-403
Replication and Meta-Analysis in
Parapsychology
Jessica Utts
Abstract. Parapsychology, the laboratory study of psychic phenomena,
has had
its
history interwoven with
that
of
statistics. Many
of the
controversies in parapsychology have focused on statistical issues, and
statistical models have
played an
integral
role
in
the experimental
work. Recently, parapsychologists have been using meta-analysis as a
tool
for
synthesizing
large
bodies
of work.
This paper
presents an
overview of the use of statistics in parapsychology and offers a summary
of the meta-analyses that have been conducted.
It begins with some
anecdotal information about the involvement of statistics and statisti-
cians with the early history of parapsychology. Next,
it is argued that
most nonstatisticians do not appreciate the connection between power
and “successful” replication of experimental effects. Returning to para-
psychology, a particular experimental regime is examined by summariz-
ing an extended debate over the interpretation of the results. A new set
of experiments designed to resolve the debate is then reviewed. Finally,
meta-analyses from several areas of parapsychology are summarized.
It
is concluded that the overall evidence indicates that there is an anoma-
lous effect in need of an explanation.
Key words and phrases: Effect size, psychic research, statistical contro-
versies, randomness, vote-counting.
1. INTRODUCTION
In a June
1990 Gallup
Poll, 49%
of the
1236
respondents claimed to believe in extrasensory per-
ception (ESP), and one in four claimed to have had
a personal experience involving telepathy (Gallup
and Newport,
1991).
Other
surveys have shown
even
higher
percentages;
the
University
of
Chicago’s
National
Opinion Research
Center
re-
cently surveyed 1473 adults, of which 67% claimed
that they had experienced ESP (Greeley, 1987).
Public opinion
is a poor arbiter of science, how-
ever, and experience
is a poor substitute
for the
scientific method. For more than a century, small
numbers of scientists have been conducting labora-
tory
experiments
to
study
phenomena
such
as
telepathy,
clairvoyance
and
precognition,
collec-
tively known
as
“psi”
abilities.
This paper
will
examine some of that work, as well as some of the
statistical controversies it has generated.
Parapsychology, as this field is called, has been a
source of controversy throughout its history. Strong
beliefs tend to be resistant to change even in the
face of data, and many people, scientists included,
seem to have made up their minds on the question
without examining any
empirical data
at
all. A
critic of parapsychology recently acknowledged that
“The level of the debate during the past 130 years
has been an embarrassment for anyone who would
like to believe that scholars and scientists adhere
to standards of rationality and fair play” (Hyman,
1985a, page 89). While much of the controversy has
focused on poor experimental design and potential
fraud, there have been attacks and defenses of the
statistical methods as well, sometimes calling into
question the very foundations
of probability and
statistical inference.
Most of the criticisms have been leveled by psy-
chologists. For example, a 1988 report of the U.S.
National Academy of Sciences concluded that “The
committee
finds
no
scientific
justification
from
research conducted over a period of 130 years for
the
existence
of
parapsychological
phenomena”
(Druckman and Swets, 1988, page 22). The chapter
Jessica
Utts
is
Associate
Professor,
Division
of
Statistics,
University
of California
at
Davis,
469
'{err Hall, Davis, Calfornm 95616.
nnnninoino
A
0IU^r^
S/
C^l°E,
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08
: CIA-RDP96-OO789R0O31OOO1OOO1-6
364
J. UTTS
chaired by
a
psychologist who had
published
a
similar conclusion prior to his appointment to the
committee (Hyman, 1985a, page 7). There were no
parapsychologists involved with the writing of the
report. Resulting accusations of bias (Palmer, Hon-
orton and Utts, 1989) led U.S. Senator Claiborne
Pell
to
request
that the
Congressional
Office
of
Technology Assessment (OTA) conduct an investi-
gation
with
a more
balanced
group. A
one-day
workshop was held on September 30, 1988, bring-
ing together parapsychologists, critics and experts
in some related fields (including the author of this
paper). The report concluded that parapsychology
needs “a fairer hearing across a broader spectrum
of the
scientific community,
so that emotionality
does not impede objective assessment of experimen-
tal
results”
(Office
of
Technology
Assessment,
1989).
It
is
in the
spirit
of the OTA report that
this
article
is written. After Section 2, which offers an
anecdotal account
of the
role of statisticians and
statistics in parapsychology, the discussion turns to
the more general question of replication of experi-
mental
results.
Section 3
illustrates how replica-
tion has been (mis)interpreted by scientists in many
fields. Returning to parapsychology in Section 4, a
particular experimental regime
called the “ganz-
feld”
is described, and an extended debate about
the
interpretation
of the experimental
results
is
discussed.
Section 5 examines a meta-analysis of
recent ganzfeld experiments designed to resolve the
debate. Finally, Section 6 contains a brief account
of meta-analyses that have been conducted in other
areas of parapsychology, and conclusions are given
in Section 7.
2. STATISTICS AND PARAPSYCHOLOGY
Parapsychology had its beginnings in the investi-
gation of purported mediums and other anecdotal
claims
in the
late
19th century. The Society
for
Psychical Research was founded in Britain in 1882,
and
its
American
counterpart
was
founded
in
Boston in 1884. While these organizations and their
members were primarily involved with investigat-
ing
anecdotal
material,
a
few
of the
early
re-
searchers were already conducting “forced-choice”
experiments such as card-guessing.
(Forced-choice
experiments are like multiple choice tests; on each
trial the subject must guess from a small, known
set
of
possibilities.)
Notable
among
these
was
Nobel Laureate Charles Richet, who
is generally
credited with being the first to recognize that prob-
ability
theory
could
be
applied
to
card-guessing
experiments (Rhine, 1977, page 26; Richet, 1884).
F. Y. Edgeworth, partly in response to what he
considered toApproved Fotf3ReJe&££s2Q9$/fi8/08
ments, offered one of the earliest treatises on the
statistical evaluation of forced-choice experiments
in two articles published in the Proceedings of the
Society for Psychical Research (Edgeworth,
1885,
1886).
Unfortunately, as noted by Mauskopf and
McVaugh (1979) in their historical account of the
period, Edgeworth’s papers were “perhaps too diffi-
cult for their immediate audience” (page 105).
Edgeworth began his [analysis by using Bayes’
theorem
to
derive
the formula
for
the
posterior
probability that chance [was operating, given the
data.
He
then
continued
with
an
argument
“savouring more of Bernoulli than Bayes” in which
“it is consonant, I submit, to experience, to put 1 /2
both for a and 0,” that is, for both the prior proba-
bility that chance
alone was
operating, and the
prior probability that “there should have been some
additional
agency.” He
then
reasoned
(using
a
Taylor
series
expansion
of
the
posterior
prob-
ability formula) that
if [there were a large
prob-
ability
of
observing
the
data
given
that
some
additional agency was
atj work, and a small objec-
tive probability of the data under chance, then the
latter (binomial) probability “may be taken as a
rough measure of the sought a posteriori probabil-
ity
in favour
of mere chance” (page
195).
Edge-
worth concluded his article by applying his method
to some
data
published’ previously
in
the same
journal. He found the probability against chance to
be 0.99996, which he said “may fairly be regarded
as physical certainty” (page 199). He concluded:
Such
is
the
evidence which
the
calculus
of
probabilities affords
asj to the existence of an
agency other than mere chance. The calculus is
silent as to the nature of that agency—whether
it
is more
likely to be vulgar illusion or ex-
traordinary
law.
That
is
a
question
to
be
decided, not by formulae and figures, but by
general
philosophy and common
sense
[page
199],
Both the
statistical arguments and the experi-
mental
controls
in
these] early experiments were
somewhat
loose. For example, Edgeworth treated
as binomial an experiment
in which one person
chose
a
string
of eight
[ letters and
another
at-
tempted to guess the string. Since it has long been
understood that people ard poor random number (or
letter) generators, there
js no statistical basis for
analyzing such an experiment. Nonetheless, Edge-
worth and his contemporaries set the stage for the
use of controlled experiments with statistical evalu-
ation in laboratory parapsychology. An interesting
historical account of Edgeworth’s involvement and
the role telepathy experiments played in the early
history of randomization and experimental design
lsQb^RiaR9640fl38^R0ffi5.1 0001 0001 -6