Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
377
The correlation between extroversion scores and
ganzfeld rating scores was
r = 0.18, with a 95%
confidence interval from 0.05 to 0.30. This
is con-
sistent with the mean correlation of
r = 0.20
for
free-response
experiments,
determined
from
the
meta analysis. These correlations indicate that ex-
troverted
subjects
can
produce
higher
scores
in
free-response ESP tests.
7. CONCLUSIONS
Parapsychologists
often make
a
distinction
be-
tween
“proof-oriented
research”
and
“process-
oriented
research.” The
former
is
typically
con-
ducted to test the hypothesis that psi abilities exist,
while the
latter
is designed
to answer questions
about
how
psychic
functioning
works.
Proof-
oriented
research
has
dominated
the
literature
in
parapsychology.
Unfortunately, many
of the
studies
used
small
samples
and would
thus
be
nonsignificant
even
if
a
moderate-sized
effect
exists.
The
recent
focus on
meta-analysis
in parapsy-
chology
has
revealed
that
there
are
small
but
consistently nonzero effects across studies, experi-
menters and laboratories. The sizes of the effects in
forced-choice
studies appear
to be comparable
to
those reported
in some medical
studies that had
been heralded
as breakthroughs.
(See
Section
5;
also Honorton and Ferrari, 1989, page 301.) Free-
response
studies show
effect
sizes
of far
greater
magnitude.
A promising direction for future process-oriented
research
is
to examine
the
causes
of individual
differences
in
psychic
functioning.
The ESP/ex-
troversion meta-analysis is a step in that direction.
In keeping with
the
idea
of individual
differ-
ences, Bayes and empirical Bayes methods would
appear to make more sense than the classical infer-
ence methods commonly
used,
since
they would
allow individual abilities and beliefs to be modeled.
Jeffreys (1990) reported a Bayesian analysis of some
of the RNG experiments and showed that conclu-
sions were closely tied to prior beliefs even though
hundreds of thousands of trials were available.
It may be that the nonzero effects observed in the
meta-analyses can be explained by something other
than ESP, such as shortcomings in our understand-
ing of randomness and independence. Nonetheless,
there is an anomaly that needs an explanation. As
I have argued elsewhere (Utts,
1987), research in
parapsychology should receive more support from
the
scientific community.
If ESP does
not
exist,
there is little to be lost by erring in the direction of
further research, which may in fact uncover other
anomalies* If ESP does exist, there
is anuch, to be
lost by n&P&T&1/
:
much to be gained by discovering how to enhance
and
apply
these
abilities
to
important
world
problems.
ACKNOWLEDGMENTS
I would like to thank Deborah Delanoy, Charles
Honorton,
Wesley
Johnson,
Scott
Pious
and an
anonymous reviewer for their helpful comments on
an earlier draft of this paper, and Robert Rosenthal
and Charles Honorton for discussions that helped
clarify details.
REFERENCES
Atkinson, R. L., Atkinson, R. C., Smith, E. E. and Bem, D. J.
(1990). Introduction to Psychology, 10th ed. Harcourt Brace
Jovanovich, San Diego.
Beloff, J. (1985). Research strategies for dealing with unstable
phenomena. In The Repeatability Problem in Parapsychol-
ogy
(B. Shapin and
L.
Coly,
eds.)
1-21.
Parapsychology
Foundation, New York.
Blackmore, S. J. (1985). Unrepeatability: Parapsychology’s only
finding.
In
The Repeatability Problem
in Parapsychology
(B.
Shapin and
L.
Coly,
eds.)
183-206.
Parapsychology
Foundation, New York.
Burdick, D. S. and Kelly, E. F. (1977). Statistical methods in
parapsychological research. In Handbook of Parapsychology
(B. B. Wolman, ed.) 81-130. Van Nostrand Reinhold, New
York.
Camp, B. H.
(1937). (Statement in Notes Section.)
Journal of
Parapsychology 1 305.
Cohen,
J.
(1990).
Things
I have
learned
(so
far).
American
Psychologist 45 1304-1312.
Coover,
J.
E.
(1917).
Experiments
in
Psychical Research
at
Leland Stanford Junior University. Stanford Univ.
Dawes, R. M., Landman, J. and Williams, J. (1984). Reply to
Kurosawa. American Psychologist 39 74-75.
Diaconis, P. (1978). Statistical problems in ESP research.
Sci-
ence 201 131-136.
Dommeyer, F. C. (1975). Psychical research at Stanford Univer-
sity. Journal of Parapsychology 39 173-205.
Druckman, D. and Swets, J. A., eds. (1988) Enhancing Human
Performance:
Issues,
Theories,
and
Techniques.
National
Academy Press, Washington, D.C.
Edgeworth, F. Y. (1885). The calculus of probabilities applied
to
psychical
research.
In
Proceedings
of the
Society
for
Psychical Research 3 190-199.
Edgeworth, F. Y. (1886). The calculus of probabilities applied
to psychical research.
II. In
Proceedings of the Society for
Psychical Research 4 189-208.
Feller, W. K.
(1940).
Statistical
aspects of ESP.
Journal of
Parapsychology 4 271-297.
Feller, W.
K.
(1968). An Introduction
to Probability Theory
and Its Applications
1, 3rd ed. Wiley, New York.
Fisher, R. A. (1924). A method of scoring coincidences in tests
with playing cards. In Proceedings of the Society for Psychi-
cal Research 34 181-185.
Fisher, R.
A.
(1929). The
statistical method
in psychical
re-
search. In Proceedings of the Society for Psychical Research
39 189-192.
Gallup, G. H., Jr., and Newport, F. (1991). Belief in paranor-
mal phenomena among adult Americans. Skeptical Inquirer
15 137-146.
Gardner, M. J. and Altman, D. G. (1986). Confidence intervals
Approved For Release 2000/08/08
: CIA-RDP96-00789R0031 0001 0001 -6
378
J. UTTS
|
Gilmore,
J.
B.
(1989). Randomness and
the
search
for
psi.
the behavioral and social |sciences. Journal of Social Behav-
Journal of Parapsychology 53 309-340.
Gilmore,
J. B.
(1990). Anomalous significance
in pararandom
and psi-free domains. Journal of Parapsychology 54 53-58.
Greeley,
A.
(1987).
Mysticism
goes
mainstream.
American
Health 7 47-49.
Greenhouse, J. B. and Greenhouse, S. W. (1988). An aspirin a
day
.
.
. ? Chance 1 24-31.
Greenwood,
J. A. and Stuart, C.
E.
(1940). A review of Dr.
Feller’s critique. Journal of Parapsychology 4 299-319.
Hacking,
I.
(1988). Telepathy: Origins of randomization in ex-
perimental design.
Isis 79 427-451.
Hansel, C.
E. M.
(1980). ESP and Parapsychology: A
Critical
Re-evaluation. Prometheus Books, Buffalo, N.Y.
Harris, M.
J. and Rosenthal,
R.
(1988a).
Interpersonal
Ex-
pectancy
Effects and Human
Performance
Research.
Na-
tional Academy Press, Washington, D.C.
Harris, M. J. and Rosenthal, R. (1988b). Postscript to Interper-
sonal Expectancy Effects and Human Performance Research.
National Academy Press, Washington, D.C.
Hedges,
L.
V.
and Olkin,
I.
(1985).
Statistical Methods
for
Meta-Analysis. Academic, Orlando, Fla.
Honorton,
C.
(1977).
Psi
and
internal
attention
states.
In
Handbook of Parapsychology (B. B. Wolman, ed.) 435-472.
Van Nostrand Reinhold, New York.
Honorton, C. (1985a). How to evaluate and improve the repli-
cability
of parapsychological
effects.
In The
Repeatability
Problem
in Parapsychology
(B. Shapin and
L.
Coly,
eds.)
238-255. Parapsychology Foundation, New York.
Honorton, C. (1985b). Meta-analysis of psi ganzfeld research: A
response to Hyman.
Journal of Parapsychology 49 51-91.
Honorton,
C., Berger,
R.
E., Varvoglis, M.
P., Quant,
M.,
Derr,
P.,
Sciiechter,
E.
I.
and
Ferrari,
D.
C.
(1990).
Psi communication
in the
ganzfeld: Experiments with an
automated testing system and a comparison with
a meta-
analysis of earlier
studies.
Journal of Parapsychology 54
99-139.
Honorton, C. and Ferrari, D.
C.
(1989). “Future
telling”: A
meta-analysis
of
forced-choice
precognition
experiments,
1935-1987. Journal of Parapsychology 53 281-308.
Honorton, C., Ferrari, D. C. and Bem, D.
J. (1991). Extraver-
sion and ESP
performance: A
meta-analysis and
a new
confirmation. Research in Parapsychology 1990. The Scare-
crow Press, Metuchen, N.J. To appear.
Hyman, R. (1985a). A critical overview of parapsychology. In A
Skeptic's Handbook of Parapsychology (P. Kurtz, ed.) 1-96.
Prometheus Books, Buffalo, N.Y.
Hyman,
R.
(1985b). The
ganzfeld
psi
experiment: A
critical
appraisal. Journal of Parapsychology 49 3-49.
Hyman, R. and Honorton, C.
(1986). Joint communique: The
psi
ganzfeld
controversy.
Journal
of Parapsychology
351-364.
Iversen, G. R., Longcor, W. H., Mosteller, F., Gilbert, J.
P.
and Youtz, C. (1971). Bias and runs in dice throwing and
recording: A few million throws. Psychometrika 36 1-19.
Jeffreys, W.
H.
(1990).
Bayesian
analysis
of random
event
generator data. Journal of Scientific Exploration 4 153-169.
Lindley,
D.
V.
(1957). A
statistical
paradox.
Biometrika 44
187-192.
Mauskopf, S. H. and McVaugh, M. (1979). The Elusive Science:
Origins of Experimental Psychical Research. Johns Hopkins
Univ. Press.
McVaugh, M.
R. and Mauskopf,
S. H.
(1976).
J.
B. Rhine's
Extrasensory
Perception
and
its
background
in
psychical
research.
Isis 67 161-189.
Neuliep, J. W., ed. (1990). Handbook of replication research in
Approved For Release 2000/08/08
:
ior and Personality 5 (4) 1-510.
Office of Technology Assessment (1989). Report of a work-
shop on experimental parapsychology. Journal of the Amer-
ican Society for PsychicaliResearch 83 317-339.
Palmer, J. (1989). A reply to; Gilmore. Journal of Parapsychol-
ogy 53 341-344.
j
Palmer,
J.
(1990). Reply
to Gilmore: Round two.
Journal of
Parapsychology 54 59-61;
Palmer,
J. A., Honorton, C.jand Utts, J. (1989). Reply to the
National Research Council study on parapsychology.
Jour-
nal of the American Society for Psychical Research 83 31 -49.
Radin, D.
I. and Ferrari, D.
jC. (1991). Effects of consciousness
on the
fall of dice: A meta-analysis.
Journal of Scientific
Exploration 5 61-83.
Radin, D.
I. and Nelson, R.
|D.
(1989). Evidence for conscious-
ness-related anomalies
iri random physical systems.
Foun-
dations of Physics 19 1499- 1514.
Rao, K. R. (1985). Replication in conventional and controversial
sciences.
In The Repeatability Problem
in Parapsychology
(B. Shapin and L. Coly, ids.) 22-41. Parapsychology Foun-
dation, New York.
Rhine, J. B. (1934). Extrasensory Perception. Boston Society for
Psychical Research, Boston.
(Reprinted by Branden Press,
1964.)
Rhine,
J.
B.
(1977). History Of experimental studies.
In Hand-
book
of Parapsychology
{B.
B. Wolman,
ed.) 25-47. Van
Nostrand Reinhold, New York.
Riciiet, C. (1884). La suggestion mentale et le calcul des proba-
bility. Revue Philosophique 18 608-674.
Rosenthal, R.
(1984).
Meta-finalytic Procedures for Social Re-
search. Sage, Beverly Hills.
Rosenthal, R. (1986). Meta-analytic procedures and the nature
of replication: The ganzfeld debate. Journal of Parapsychol-
ogy 50 315-336.
Rosenthal, R.
(1990a). How; are we doing in
soft psychology?
American Psychologist 45 775-777
.
Rosenthal,
R.
(1990b).
Replication
in
behavioral
research.
Journal of Social Behavior and Personality 5 1-30.
Saunders, D. R. (1985). On Hyman’s factor analysis. Journal of
Parapsychology 49 86-88;
Shapin, B. and Coly, L., eds. (1985). The Repeatability Problem
in Parapsychology. Parapsychology Foundation, New York.
Spencer-Brown, G. (1957).
Probability and Scientific Inference.
Longmans Green, London and New York.
Stuart, C.
E. and Greenwood, J. A. (1937). A review of criti-
cisms of the mathematical evaluation of ESP data. Journal
of Parapsychology
1 295-304.
Tversky, A. and Kahneman,
D.
(1982).
Belief in the law
of
small numbers. In Judgment Under Uncertainty: Heuristics
and Biases (D. Kahnema'n, P. Slovic and A. Tversky, eds.)
23-31. Cambridge Univ. Press.
Utts,
J.
(1986). The
ganzfelijl debate: A statistician's perspec-
tive. Journal of Parapsychology 50 395-402.
Utts,
J.
(1987).
Psi,
statistics,
and
society.
Behavioral and
Brain Sciences 10 615— 6l|6.
Utts, J. (1988). Successful replication versus statistical signifi-
cance. Journal of Parapsychology 52 305-320.
Utts, J. (1989). Randomness pnd randomization tests: A reply to
Gilmore. Journal of Parapsychology 53 345-351.
Utts, J. (1991). Analyzing free-response data: A progress report.
In
Psi Research Methodology: A Re-examination
(L. Coly,
ed.). Parapsychology Foundation, New York. To appear.
Wilks,
S.
S.
(1965a).
Statistical
aspects
of expeirments
in
telepath. N.Y. Statistician 16 (6) 1-3.
Wilks,
S.
S.
(1965b).
Statistical
aspects
of experiments
in
telepathy. N.Y. Statistician 16 (7) 4-6.
CIA-RDP96-00789R0031 0001 0001 -6
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
J. UTTS
379
Comment
M. J. Bayarri and James Berger
1. INTRODUCTION
There are many fascinating issues discussed in
this paper.
Several concern parapsychology
itself
and the
interpretation
of statistical methodology
therein. We are not experts in parapsychology, and
so have only one comment concerning such mat-
ters:
In Section
3 we
briefly discuss the need to
switch from P-values to Bayes factors in discussing
evidence concerning parapsychology.
A more general issue raised in the paper is that
of replication.
It
is quite illuminating to consider
the
issue
of replication from a Bayesian perspec-
tive, and this is done in Section 2 of our discussion.
2. REPLICATION
Many insightful observations concerning replica-
tion are given in the article, and these spurred us
to determine
if they
could
be
quantified within
Bayesian reasoning. Quantification requires clear
delineation of the possible purposes of replication,
and
at least two are obvious. The
first
is simple
reduction of random
error, achieved by obtaining
more observations from the replication. The second
purpose is to search for possible bias in the original
experiment. We use “bias” in a loose sense here, to
refer to any of the huge number of ways in which
the effects being measured by the experiment can
differ from the
actual
effects
of interest. Thus a
clinical trial without a placebo can suffer a placebo
“bias”; a survey can
suffer a “bias” due
to
the
sampling
frame
being
unrepresentative
of
the
actual
population;
and
possible
sources
of
bias
in
parapsychological
experiments
have
been
extensively discussed.
Replication to Reduce Random Error
If the sole goal of replication of an experiment is
to reduce random error, matters are very straight-
forward. Reviewing the Bayesian way of studying
this
issue
is,
however,
useful and
will
be
done
through the following simple example.
M.
J. Bayarri
is
Titular Professor, Department of
Statistics and
Operations
Research,
University of
Valencia, Avenida Dr. Moliner 50, 46100 Burjassot,
Valencia, Spain. James Berger
is the Richard M.
Brumfield
Distinguished
Professor
of
Statistics,
Purdue
Example
1. Consider the example from Tversky
and Kahnemann
(1982),
in which an experiment
results in a standardized test statistic of z x = 2.46.
(We
will assume normality
to keep computations
trivial.) The question
is: What is the highest value
of z2 in a second set of data that would be consid-
ered
a
failure
to
replicate? Two
possible
precise
versions of this question are: Question
1: What
is
the probability of observing
z2
f°r which the null
hypothesis would be rejected in the replicated ex-
periment?
Question
2: What
value
of
z2
would
leave one's overall opinion about the null hypothe-
sis unchanged?
Consider the simple case where Z 1 ~ N(z 1
\ 0,
1)
and (independently) Z2 ~N(z 2 \0,
1), where
d
is
the mean and
is the standard deviation
of the
normal distribution. Note that we are considering
the case in which no experimental bias is suspected
and so the means for each experiment are assumed
to be the same.
Suppose that it is desired to test H0
: 6 < 0 versus
H x
: 0 > 0, and
suppose
that
initial
prior
opinion
about
can be described by the noninformative
prior
ir(6) =
1. We consider the one-sided testing
problem with a constant prior in this section, be-
cause
it is known that then the posterior probabil-
ity of H0
, to be denoted by P(H0 |data), equals the
P-value, allowing us to avoid complications arising
from
differences between Bayesian and
classical
answers.
After observing z x = 2.46, the posterior distribu-
tion of 0
is
tt(0
| z x ) = N(6
1 2 .46 , 1).
Question
1 then has the answer (using predictive
Bayesian reasoning)
P (rejecting at level a
\ z x )
-
/ 2 u 2 -0) 2
7r (0| Zl ) de dz2
= 1-4
°a ~ 2.46
where 4
is the standard normal cdf and
ca
is the
(one-sided) critical value corresponding to the level,
a, of the test. For instance,
if a = 0.05, then this
probability equals 0.7178, demonstrating that there
is a quite substantial probability that the second
experiment will
fail to reject.
If a
is chosen to be
the observed significance level from the first exper-
CIAtRtOP9&t00789R0i03i1t(I10(i)1.0G0i1k6 that the
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
380
J. UTTS
second experiment will
reject
is just
1/2. This
is
nothing but a statement of the well-known martin-
gale property of Bayesianism, that what you “ex-
pect”
to see in the future
is just what you know
today. In a sense, therefore, question
is exposed
as being uninteresting.
Question 2 more properly focuses on the fact that
the
stated
goal
of replication
here
is simply
to
reduce uncertainty in stated conclusions. The an-
swer to the question follows immediately from not-
ing
that
the
posterior
from
the
combined
data
(,zv z2 ) is
tt{6\zi, z2 ) = N(6\(z 1 + za )/2, 1//2),
so that
one
is very confident
(that the Xt
have mean
6.
Using normal approximations for convenience, the
data can be summarized as
~ /V(* x
|fl
f 4.82j,
X2 ~ N(x2 \6, 3.63)
with
actual
observations
x , = 7.704
and
x 9 =
13.07.
Consider now the bias issue. We assume that the
original experiment
is! somewhat suspect
in
this
regard, and we
will model
bias by
defining the
mean of Y to be
n t 0 + P,
where 0 is the unknown bias. Then the data in the
original experiment can be summarized by
P(tf0 |data) = 4>(-(z 1 + z2 )/v/2).
Y ~ N^y
17, 1 .54),
Setting this equal to P(H0
| z t ) and solving for
z2
yields
z2 = (\/2 -
1 )z 1 = 1.02. Any
value
of
z2
greater than this will increase the total evidence
against H0 , while any value smaller than 1.02 will
decrease the evidence.
Replication to Detect Bias
The aspirin example dramatically raises the
is-
sue of bias detection as a motive
for replication.
Professor
Utts
observes
that
replication
gives
results that are fully compatible with those of the
original study, which could be interpreted as sug-
gesting that there is no bias in the original study,
while replication 2 would raise serious concerns of
bias. We became very
interested
in the
implicit
suggestion that
replication 2 would thus
lead
to
less
overall evidence against the
null hypothesis
than would replication
1, even though in isolation
replication
2 was much more
“significant” than
was replication 1. In attempting to see if this is so,
we considered the Bayesian approach to study of
bias within the framework of the aspirin example.
with the actual observation being y = 7.707.
Bayesian analysis requires specification of a prior
distribution, ir(0), for the suspected amount of bias.
Of particular interest then are the posterior distri-
bution
of
0,
assuming
replication
i
has
been
performed, given by
*(01 y. *
t )
« 7r(/3)exp
| 1
2(1.54 2 + a 2
)
where
a,2
is the variance (4.82 or 3.63) from repli-
cation
i; and the posterior probability of H0
, given
by
P(H0 \y, x c )
1.54 \ZcFj2 + 1.54 2
1.54
(y - 0 )
<r, \/<7 2 + 1 .54 2
x
i
*(01 y, x t) d0.
Example 2. For simplicity in the aspiring exam-
ple, we reduce consideration to
6 = true difference in heart attack rates between
aspirin and placebo populations multiplied by
1000;
Y = difference in observed heart attack rates be-
tween aspirin and placebo groups in original
study multiplied by 1000;
X
( = difference in observed heart attack rates be-
tween aspirin and placebo groups in Replica-
tion
i multiplied by 1000.
We assume
that
the
replication
studies
are
ex-
tremely
well designed and implemented,
so that
Approved For Release 2000/08/08
Recall that our goal here was to see
if Bayesian
analysis can reproduce the intuition that the origi-
nal experiment could be trusted if replication 1 had
been done, while it could not be trusted (in spite of
its much larger sample size) had replication 2 been
performed.
Establishing
this
requires
finding
a
prior distribution
7r(/3)
j for which
ir(0
| y,
x
( ) has
little effect on P(H0
\ y,
Xj), but n(0
| y, x2 ) has a
large effect on P(H0
\ y,
x 2 ). To achieve the
first
objective,
w(0) must be tightly concentrated near
zero. To achieve the second, ir(0) must be such that
large
| y - x 2
|
, which suggests presence of a large
bias, can result in a substantial shift of posterior
mass for 0 away from zero.
CIA-RDP96-00789R0031 0001 0001 -6