Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
One
of
the
first
American
researchers
to
use
statistical
methods
in
parapsychology
was
John Edgar Coover, who was the Thomas Welton
Stanford Psychical Research Fellow in the Psychol-
ogy Department at Stanford University from 1912
to 1937 (Dommeyer,
1975). In 1917, Coover pub-
lished
a
large
volume
summarizing
his
work
(Coover,
1917).
Coover
believed
that
his
results
were consistent with chance, but others have
ar-
gued that Coover’s
definition of significance was
too
strict (Dommeyer,
1975). For example, in one
evaluation
of
his
telepathy
experiments,
Coover
found a two-tailed p-value of 0.0062. He concluded,
“Since
this
value,
then,
lies within the
field
of
chance
deviation, although the
probability
of its
occurrence by chance
is
fairly
low,
it cannot be
accepted
as
a
decisive
indication
of some
cause
beyond chance which operated in favor of success in
guessing”
(Coover,
1917, page
82). On the next
page, he made
it explicit that he would require a
p-value
of 0.0000221
to
declare
that
something
other than chance was operating.
It was
during
the summer
of
1930,
with
the
card-guessing experiments of J. B. Rhine at Duke
University, that parapsychology began to take hold
as
a
laboratory
science.
Rhine’s
laboratory
still
exists under the name of the Foundation for Re-
search on the Nature of Man, housed at the edge of
the Duke University campus.
It
wasn’t
long
after Rhine
published
his
first
book,
Extrasensory
Perception
in
1934,
that
the
attacks on his methodology began. Since his claims
were wholly based on
statistical
analyses
of his
experiments, the
statistical methods were
closely
scrutinized by critics anxious to find a conventional
explanation for Rhine’s positive results.
The
most
persistent
critic was
a
psychologist
from
McGill
University named
Chester
Kellogg
(Mauskopf and McVaugh,
1979).
Kellogg’s main
argument was that Rhine was using the binomial
distribution (and normal approximation) on a
se-
ries of trials that were not independent. The experi-
ments
in
question
consisted
of having a
subject
guess the order of a deck of 25 cards, with five each
of five symbols, so technically Kellogg was correct.
By
1937,
several
mathematicians
and
statis-
ticians had come
to
Rhine’s
aid. Mauskopf and
McVaugh (1979) speculated that since statistics was
itself a young discipline, “a number of statisticians
were
equally
outraged by
Kellogg, whose
argu-
ments they saw as discrediting
their
profession”
(page
258). The major technical work, which
ac-
knowledged that Kellogg’s criticisms were accurate
but
did
little
to change
the
significance
of the
results,
was
conducted
by
Charles
Stuart
and
Joseph
volume
of the
Journal of Parapsychology
(Stuart
and Greenwood,
1937).
Stuart, who had been an
undergraduate in mathematics at Duke, was one of
Rhine’s early subjects and continued to work with
him as a researcher until Stuart’s death in 1947.
Greenwood was a Duke mathematician, who appar-
ently converted to a statistician at the urging of
Rhine.
Another prominent
figure who was
distressed
with
Kellogg’s
attack was
E.
V.
Huntington,
a
mathematician
at
Harvard.
After
corresponding
with Rhine, Huntington decided that, rather than
further confuse the public with a technical reply to
Kellogg’s arguments, a simple statement should be
made to the effect that the mathematical issues in
Rhine’s work had been resolved. Huntington must
have
successfully
convinced
his
former
student,
Burton Camp
of Wesleyan, that this was a wise
approach. Camp was the 1937 President of IMS.
When the annual meetings were held in December
of
1937
(jointly
with AMS
and AAAS), Camp
released a statement to the press that read:
Dr.
Rhine’s
investigations have two
aspects:
experimental
and
statistical. On
the
exper-
imental
side
mathematicians,
of
course,
have nothing to
say. On the
statistical
side,
however,
recent
mathematical
work
has
established the
fact
that, assuming
that
the
experiments
have
been
properly
performed,
the
statistical analysis
is essentially valid.
If
the Rhine investigation is to be fairly attacked,
it must be on other than mathematical grounds
[Camp, 1937].
One statistician who did emerge as a critic was
William Feller. In a talk at the Duke Mathemati-
cal Seminar on April 24, 1940, Feller raised three
criticisms to Rhine’s work (Feller, 1940). They had
been raised before by others (and continue to be
raised even today). The
first was that inadequate
shuffling of the cards resulted in additional infor-
mation from one series to the next. The second was
what
is now known
as
the
“file-drawer
effect,”
namely, that
if one combines the
results of pub-
lished studies only, there
is sure
to be a bias
in
favor of successful studies. The third was that the
results were enhanced by the use of optional stop-
ping, that is, by not specifying the number of trials
in advance. All three of these criticisms were ad-
dressed
in a rejoinder by Greenwood and Stuart
(1940), but Feller was never convinced. Even in its
third edition published in 1968, his book An Intro-
duction
to Probability Theory and Its Applications
still contains his conclusion about Greenwood and
Stuart: “Both
their arithmetic and
their
experi-
ments have a distinct tinge
of the supernatural”
Approved For Release 2000/08/08
: CIA-RDP96-00789RQ03100010001-6
366
j. UTTS
!
Feller was confused ... he seemed to have decided
the opposition was wrong and that was that.”
Several
statisticians
have
contributed
to
the
literature
in parapsychology
to
greater
or
lesser
degrees.
T.
N.
E.
Greville
developed
applicable
statistical methods for many of the experiments in
parapsychology and was
Statistical Editor
of the
Journal of Parapsychology (with J. A. Greenwood)
from its start in 1937 through Volume 31 in 1967;
Fisher (1924,
1929) addressed some
specific prob-
lems in card-guessing experiments; Wilks (1965a, b)
described various statistical methods for parapsy-
chology; Lindley (1957) presented a Bayesian anal-
ysis
of some
parapsychology
data; and
Diaconis
(1978) pointed out some problems with certain ex-
periments and presented a method
for analyzing
experiments when feedback is given.
Occasionally,
attacks
on
parapsychology
have
taken the form of attacks on statistical inference in
general,
at
least
as
it
is
applied
to
real
data.
Spencer-Brown (1957) attempted to show that true
randomness
is
impossible,
at
least
in
finite
se-
quences, and that this could be the explanation for
the results in parapsychology. That argument
re-
emerged in a recent debate on the role of random-
ness in parapsychology, initiated by psychologist J.
Barnard Gilmore (Gilmore, 1989, 1990; Utts, 1989;
Palmer, 1989, 1990). Gilmore stated that “The ag-
nostic
statistician,
advising
on
research
in
psi,
should take account of the possible inappropriate
ness of classical inferential statistics” (1989, page
338). In his second paper, Gilmore reviewed several
non-psi
studies showing purportedly random
sys-
tems
that
do
not behave
as
they
should under
randomness
(e.g.,
Iversen,
Longcor,
Mosteller,
Gilbert
and
Youtz,
1971;
Spencer-Brown,
1957).
Gilmore
concluded
that
“Anomalous
data
.
.
.
should
not
be
found
nearly
so
often
if
classical
statistics
offers a
valid model
of reality”
(1990,
page 54), thus rejecting the use of classical statisti-
cal inference for real-world applications in general.
3. REPLICATION
Implicit and explicit in the literature on parapsy-
chology
is the assumption that,
in order to truly
establish
itself, the
field needs
to
find
a
repeat-
able
experiment.
For
example,
Diaconis
(1978)
started the summary of his article in Science with
the words
“In
search
of repeatable ESP
experi-
ments, modern
investigators ...” (page
131). On
October 28-29, 1983, the 32nd International Con-
ference of the Parapsychology Foundation was held
in San Antonio, Texas, to address “The Repeatabil-
ity Problem
in Parapsychology.” The Conference
Proceedings
(Shapin and
Coly,
1985)
reflect
the
Approved For Release 2000/08/08
diverse views among parapsychologists on the na-
ture
of the
problem.
Ktonorton
(1985a) and Rao
(1985), for example, both argued that strict replica-
tion
is uncommon in most branches of science and
that parapsychology should not be singled out as
unique
in
this
regard.
: Other
authors
expressed
disappointment in the lgck of a single repeatable
experiment
in
parapsychology,
with
titles
such
as “Unrepeatability: Parapsychology ’s Only Find-
ing” (Blackmore, 1985), land “Research Strategies
for
Dealing
with
Unstable Phenomena”
(Beloff,
1985).
i
It
has never been
cigar, however, just
exactly
what would constitute acceptable evidence of a re-
peatable experiment. In the early days of investiga-
tion, the major
critics
‘-insisted that
it would be
sufficient
for Rhine and Soal to convince them of
ESP
if a parapsychologist could perform
success-
fully
a
single
‘fraud-proof experiment” (Hyman,
1985a, page 71). Howevejr, as soon as well-designed
experiments
showing
j statistical
significance
emerged, the
critics realized that a single experi-
ment
could
be
statistically
significant
just
by
chance. British psychologist C. E. M. Hansel quan-
tified
the new
expectation,
that
the
experiment
should be repeated a few. times, as follows:
If a
result
is significant at the
.01
level and
this result is not due to chance but to informa-
tion reaching the subject,
it may be expected
that by making two farther sets of trials the
antichance odds of one! hundred to one will be
increased to around a million to one, thus en-
abling the
effects of ESP— or whatever
is
re-
sponsible
for the original result— to manifest
itself to such an extentj that there will be little
doubt
that
the
result!
is
not
due
to
chance
[Hansel, 1980, page 2981.
In other words, three consecutive experiments at
p < 0.01
would
convince
Hansel
that something
other than chance was at: work.
This argument implies that if a particular experi-
ment produces a statistically significant result, but
subsequent replications
flail to attain significance,
then the original result was probably due to chance,
or at least remains unconvincing. The problem with
this
line
of reasoning
is: that there
is no
consid-
eration given
to sample
; size
or power.
Only an
experiment
with
extreniely
high
power
should
be
expected
to
be
“successful”
three
times
in
succession.
It
is perhaps a
failure! of the way
statistics
is
taught that many scientists do not understand the
importance of power in defining successful replica-
tion. To illustrate this point, psychologists Tversky
and Kahnemann (1982) distributed a questionnaire
: CIA-RDP96-OO789R0O31 0001 0001 -6
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
367
to their colleagues at a professional meeting, with
the question:
An investigator has reported a result that you
consider implausible. He ran 15 subjects, and
reported a significant value,
t = 2.46. Another
investigator has attempted to duplicate his pro-
cedure, and he obtained a nonsignificant value
of
t
with the same number
of subjects. The
direction was the same
in both
sets
of data.
You are reviewing the literature. What is the
highest value of
t in the second set of data that
you would describe as a
failure
to replicate?
[1982, page 28].
In
reporting
their
results,
Tversky
and Kahne-
mann stated:
The majority of our respondents regarded
t =
1.70 as a failure to replicate. If the data of two
such studies
(
t
= 2.46 and
t — 1.70) are pooled,
the value of
t
for the combined data
is about
3.00 (assuming equal variances). Thus, we are
faced with
a
paradoxical
state
of
affairs,
in
which the same data that would increase our
confidence in the finding when viewed as part
of
the
original
study,
shake
our
confidence
when viewed as an independent study
[1982,
page 28].
At a recent presentation to the History and Phi-
losophy
of Science Seminar
at the University
of
California at Davis, I asked the following question.
Two
scientists,
Professors A and B, each have
a
theory they would like to demonstrate. Each plans
to run a fixed number of Bernoulli trials and then
test H0 : p = 0.25 versus Ha : p > 0.25. Professor A
has
access
to
large
numbers
of
students
each
semester to use as subjects. In his first experiment,
he runs 100
subjects, and there are 33 successes
(p = 0.04, one-tailed). Knowing the importance of
replication, Professor A runs an additional 100 sub-
jects as a second experiment. He finds 36 successes
(p = 0.009, one-tailed).
Professor B
only
teaches
small
classes.
Each
quarter, she runs an experiment on her students to
test her theory. She
carries
out ten
studies
this
way, with the results in Table
1.
I
asked
the
audience by
a show
of hands
to
indicate whether or not they felt the scientists had
successfully demonstrated their theories. Professor
A’s
theory
received overwhelming
support,
with
approximately 20 votes, while Professor B’s theory
received only one vote.
If you aggregate the results of the experiments
for each professor, you will notice that each con-
ducted 200 trials, and Professor B actually demon-
strated a higher
level of success than Professor A,
Approved For Release 2000/08/08
with
as
opposed
to
successful
trials. The
one-tailed
p-values
for
the
combined
trials
are
0.0017 for Professor A and 0.0006 for Professor B.
To address the question of replication more ex-
plicitly,
I
also
posed
the
following
scenario.
In
December of 1987,
it was decided to prematurely
terminate a study on the effects of aspirin in reduc-
ing heart attacks because the data were so convinc-
ing
(see,
e.g., Greenhouse and Greenhouse,
1988;
Rosenthal, 1990a). The physician-subjects had been
randomly assigned
to take
aspirin
or a
placebo.
There were
104 heart attacks among the
11,037
subjects in the aspirin group, and 189 heart attacks
among
the
11,034
subjects
in
the
placebo
group
(chi-square = 25.01, p < 0.00001).
After showing the
results
of that study,
I pre-
sented the audience with two hypothetical experi-
ments conducted
to
try
to
replicate
the
original
result, with outcomes in Table 2.
I asked the audience to indicate which one they
thought was a more successful replication. The au-
dience chose the second one, as would most journal
editors,
because
of the
“significant
p-value.”
In
fact,
the
first
replication has almost exactly the
same proportion of heart attacks in the two groups
as the original study and is thus a very close repli-
cation
of that
result. The
second
replication has
Table 1
Attempted replciations for professor D
n
Number of successes
One-tailed p-value
0.22
0.15
0.23
0.17
0.20
0.18
0.14
0.08
0.31
0.21
Hypothetical
Table 2
replications of the aspirin / heart
attack study
Replication # 1
Heart attack
Replication #2
Heart attack
Yes
No
Yes
No
Aspirin
Placebo
Chi-square
1156
1090
2.596, p = 0.11
2314
2170
13.206, p = 0.0003
CIA-RDP96-00789R0031 0001 0001 -6
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
368
j. UTTS
very
different proportions, and in fact the relative
risk from the second study is not even contained in
a 95% confidence interval for relative risk from the
original
study. The
magnitude
of the
effect has
been much more closely matched by the “nonsig-
nificant” replication.
Fortunately, psychologists are beginning to no-
tice that
replication
is not as straightforward as
they were originally led to believe. A special issue
of the Journal of Social Behavior and Personality
was entirely devoted to the question of replication
(Neuliep,
1990).
In one of the
articles, Rosenthal
cautioned his colleagues: “Given the levels of sta-
tistical power
at which we normally
operate, we
have no right to expect the proportion of significant
results that we typically do expect, even
if in na-
ture there is a very real and very important effect”
(Rosenthal, 1990b, page 16).
Jacob
Cohen,
in
his
insightful
article
titled
“Things
I Have Learned (So Far),” identified an-
other misconception common among
social
scien-
tists:
“Despite widespread misconceptions
to
the
contrary, the rejection of a given null hypothesis
gives us no basis for estimating the probability that
a replication of the research
will again result
in
rejecting that null hypothesis” (Cohen, 1990, page
1307).
Cohen and Rosenthal both advocate the use
of
effect sizes as opposed to significance levels when
defining the strength of an experimental effect. In
general, effect sizes measure the amount by which
the data deviate from the null hypothesis in terms
of standardized units. For instance, the effect size
for a two-sample
£-test is usually defined to be the
difference in the two means, divided by the stan-
dard deviation for the control group. This measure
can be compared across studies without the depen-
dence on sample size inherent in significance lev-
els. (Of course there will still be variability in the
sample effect sizes, decreasing as a function of sam-
ple size.) Comparison of effect sizes across studies is
one of the major components of meta-analysis.
Similar arguments have recently been made
in
the medical literature. For example, Gardner and
Altman (1986) stated that the use of p-values “to
define two
alternative outcomes— significant and
not significant— is not helpful and encourages lazy
thinking” (page
746). They advocated the use
of
confidence intervals instead.
As discussed in the next section, the arguments
used to conclude that parapsychology has failed to
demonstrate a replicable effect hinge on these mis-
conceptions
of replication and
failure
to examine
power. A more appropriate analysis would compare
the effect sizes for similar experiments across ex-
perimenters and across time
to see
if there have
Approved For Release 2000/08/08
:
been
consistent
effects
of the same
magnitude.
Rosenthal also advocates this view of replication:
The traditional view! of replication focuses on
significance
level
asj
the
relevant summary
statistic of a study arid evaluates the success of
a
replication
in a dichotomous
fashion. The
newer, more useful view of replication focuses
on effect size as the more important summary
statistic of a study and evaluates the success of
a
replication not
in a dichotomous but
in a
continuous fashion [Rosenthal, 1990b, page 28],
The dichotomous view
of replication has been
used throughout the history of parapsychology, by
both parapsychologists rind critics (Utts, 1988). For
example, the National Academy of Sciences report
critically evaluated “significant” experiments, but
entirely ignored “nonsignificant” experiments.
In the next three sections, we will examine some
of the results in parapsychology using the broader,
more appropriate definition of replication. In doing
so, we
will show
that: the
results
are
far more
interesting than the critics would have us believe.
4. THE GANZFELD DEBATE IN
PARAPSYCHOLOGY
An extensive debate took place in the mid-1980s
between a parapsychologist and critic, questioning
whether or not a particular body of parapsychologi-
cal data had demonstrated psi abilities. The experi-
ments
in
question werg
all conducted using the
ganzfeld setting (described below). Several authors
were invited to write commentaries on the debate.
As a
result,
this data base has been more thor-
oughly
analyzed by
both
critics and
proponents
than any
other and
provides a good
source
for
studying replication in parapsychology.
The debate concluded with a detailed series of
recommendations for further experiments, and left
open the question of whether or not
psi
abilities
had been demonstrated, A new
series
of experi-
ments
that
followed
the
recommendations
were
conducted over the next few years. The results of
the new experiments will be presented in Section 5.
4.1
Free-Response Experiments
Recent experiments
ip parapsychology tend
to
use more complex target material than the cards
and dice used in the early investigations, partially
to alleviate boredom on the part of the subjects and
partially because they are thought to “more nearly
resemble the conditions
j of spontaneous psi occur-
rences” (Burdick and Kelly, 1977, page 109). These
experiments
fall
under
the
general
heading
of
“free-response” experiments, because the subject is
asked to give a verbal or: written description of the
CIA-RDP96-00789R0031 0001 0001 -6