Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
381
A sensible candidate for the prior density
7r(/3)
is the Cauchy (0, V) density
~
xv[i +
(e/vf
]
'
Flat-tailed densities, such as this, are well known
to have the property that when discordant data
is
observed (e.g., when
(| y - x2
|
is large), substan-
tial mass shifts away from the prior center towards
the likelihood center. It is easy to see that a normal
prior for
(3 can not have the desired behavior.
Our first surprise in consideration of these priors
was how small V needed to be chosen in order for
P(H0
\ y,
Xj)
to be
unaffected by
the
bias.
For
instance, even with V = 1.54/100 (recall that 1.54
was the standard deviation of Y from the original
experiment),
computation
yields P(H0
1 y,
x x ) =
4.3 x 10
, compared with the P-value (and poste-
rior probability from the
original experiment
as-
suming no bias)
of 2.8
x
~ 7
. There
is a clear
lesson here; even very small suspicions of bias can
drastically alter a small P-value. Note that replica-
tion
is very consistent with the presence of no
bias, and so the posterior distribution for the bias
remains
tightly
concentrated
near
zero;
for
in-
stance,
the mean
of the
posterior
for
(3
is then
7.2 x 10
' 6
, and the standard deviation is 0.25.
When we turned attention to replication
2, we
found
that
it
did
not
seriously change the
prior
perceptions of bias. Examination quickly revealed
the reason; even the maximum likelihood estimate
of the bias is no more than 1.4 standard deviations
from zero, which
is not enough to change strong
prior
beliefs.
We,
therefore,
considered
a
third
experiment,
defined
in Table
1. Transforming
to
approximate normality, as before, yields
X3 ~ N(x 3
| 6
, 3.48),
with x 3 = 22.72 being the actual observation. The
maximum likelihood estimate of bias
is now 3.95
standard deviations from zero, so there is potential
for a substantial change in opinion about the bias.
Sure
enough,
computation when V = 1.54/100
yields
that
E[f3\ y,
x 3 ] = -4.9
with
(posterior)
standard deviation equal to 6.62, which
is a dra-
matic shift from prior opinion (that
/3
is Cauchy (0,
Table 1
Frequency ofheart attacks in replication 3
Yes
No
1.54/100)). The effect of this is to essentially ignore
the original experiment in overall assessments of
evidence.
For
instance,
P(H0
| y,
x3) = 3.81 x
10“ n
, which
is very close to P(H0
| x 3 ) = 3.29 x
10" 11
. Note that,
if
/3 were set equal to zero, the
overall posterior probability of H0 (and
P-value)
would be 2.62 x 10“ 13
.
Thus Bayesian reasoning can reproduce the intu-
ition that replication which indicates bias can cast
considerable
doubt
on
the
original
experiment,
while
replication which
provides no
evidence
of
bias leaves evidence from the original experiment
intact. Such behavior seems only obtainable, how-
ever, with
flat-tailed priors
for bias (such as the
Cauchy) that are very concentrated (in comparison
with
the
experimental
standard
deviation)
near
zero.
3. P-VALUES OR BAYES FACTORS?
Parapsychology
experiments
usually
consider
testing of H0
: No parapsychological
effect
exists.
Such null hypotheses are often realistically repre-
sented as point nulls (see Berger and Delampady,
1987,
for the reason that care must be taken
in
such representation), in which case it is known that
there
is a large difference between
P-values and
posterior probabilities (see Berger and Delampady,
1987,
for review). The
article by
Jefferys
(1990)
dramatically illustrates this, showing that a very
small P-value can actually correspond to evidence
for H0 when considered from a Bayesian perspec-
tive. (This is very related to the famous “Jeffreys”
paradox.) The argument in favor of the Bayesian
approach here is very strong, since it can be shown
that the
conflict holds
for virtually any sensible
prior distribution; a Bayesian answer can be wrong
if the prior information turns out to be inaccurate,
but a Bayesian answer that holds for
all sensible
priors is unassailable.
Since P-values simply cannot be viewed as mean-
ingful in these situations, we found it of interest to
reconsider the example in Section 5 from a Bayes
factor perspective. We considered only analysis of
the overall totals, that
is, x = 122 successes out of
n = 355 trials. Assuming a simple Bernoulli
trial
model with success probability 6, the goal is to test
H0 :6 = 1/4 versus H^.d ± 1/4.
To determine the Bayes
factor
here, one must
specify
g(8), the conditional prior density on H 1
.
Consider choosing g to be uniform and symmetric,
that is,
Aspirin
Placebo
2309
2116
Gr (d) =
I—
,
for — - r<0< — + r
=
1 2r
0,
otherwise.
Appreveq-For Re lease 2000/08/08
: CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
J. UTTS
Crudely, r could be considered to be the maximum
change in success probability that one would expect
given that ESP exists. Also, these distributions are
the “extreme points” over the class of symmetric
unimodal conditional densities, so answers that hold
over this class are also representative of answers
over a much larger class. Note that here r < 0.25
(because 0 < 6 <
1); for the given data the 0 > 0.5
are
essentially
irrelevant, but
if
it were deemed
important to take them into account one could use
the more sophisticated binomial analysis in Berger
and Delampady (1987).
For gr , the Bayes factor of Hl to H0 , which is to
be interpreted as the relative odds for the hypothe-
ses provided by the data, is given by
(l/(2r))/,2
_
+;^
22 (l-g)
355 - 122 d6
(1/4)
122
(1 - 1/4)
355 - 122
(63.13)
( r - .0937
\
.0252
+ 4>
-(r + .0937)
.0252
This is graphed in Figure 1.
The P-value for this problem was 0.00005, indi-
cating overwhelming evidence against H0 from a
classical
perspective.
In contrast
to the
situation
studied by Jefferys
(1990), the Bayes factor here
does not completely reverse the conclusion, show-
ing that there are very reasonable values of r
for
which
the
evidence
against
jF/0
is
moderately
strong, for example 100/1 or 200/1. Of course, this
evidence
is probably not
of sufficient strength
to
overcome
strong
prior
opinions
against H0
(one
Fig.
1 .
The Bayes factor of H1
to H0
as a function of r,
the
maximum change
in succesk probability
that
is expected given
that ESP exists,
for the ganzfeld experiment.
obtains
final
posterior? odds by multiplying
prior
odds
by
the
Bayes
factor).
To
properly
assess
strength of evidence, vie feel that such Bayes factor
computations should become standard in parapsy-
chology.
As mentioned by Prpfessor Utts, Bayesian meth-
ods have additional potential in situations such as
this, by allowing unrealistic models of iid trials to
be replaced by hierarchical models reflecting differ-
ing abilities among subjects.
ACKNOWLEDGMENTS
M.
J. Bayarri’s research was supported
in part
by the Spanish Ministry of Education and Science
under DGICYT
Grant
BE91-038,
while
visiting
Purdue University. James Berger’s research was
supported by NSF Grant DMS-89-23071.
Comment
Ree Dawson
This paper offers readers interested in statistical
account of how both Resign and inferential aspects
science multiple views of the controversial history
of statistics have beert pivotal issues in evaluating
of
parapsychology
and
how
statistics
has
con-
the outcomes of experiments that study psi
abili-
tributed
to
its development.
It
first
provides an
ties.
It then emphasizes how the idea of science as
replication
has been
j key
in
this
field
in which
results have not been; conclusive or consistent and
thus meta-analysis has been
at the heart
of the
Ree Dawson
is Senior
Statistician, New England
literature in parapsychology. The author not only
Biomedical Research
Foundation,
and
Statistical
reviews past debate bn how to interpret repeated
Consultant, RFE/RL Research Institute. Her mail-
psi studies, but also provides very detailed informa -
ing address
is 177 Morrison Avenue,
Somerville,
tion on
the Honortop-Hyman
argument,
a
nice
Massachusetts 02144
illustration of the challenges of resolving such de-
Approved For Release 2000/08/08
: CIA-RDP96-OO789R0O31OOO1OOO1-6
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
383
bate. This debate
is also a good example
of how
statistical
criticism can be
part
of the
scientific
process and lead to better experiments and, in gen-
eral, better science.
The remainder of the paper addresses technical
issues
of meta-analysis, drawing upon recent
re-
search in parapsychology for an in-depth applica-
tion.
Through
a
series
of examples,
the
author
presents a convincing argument that power issues
cannot be overlooked in successive replications and
that comparison
of effect
sizes provides
a
richer
alternative to the dichotomous measure inherent in
the use of p-values. This
is particularly relevant
when
the
potential
effect
size
is
small and
re-
sources are limited, as seems to be the case for psi
studies.
The concluding section briefly mentions Bayesian
techniques. As noted by the author, Bayes (or em-
pirical Bayes) methodology seems to make sense for
research in parapsychology. This discussion exam-
ines possible Bayesian approaches to meta-analysis
in this field.
BAYES MODELS FOR PARAPSYCHOLOGY
The notion of repeatability maps well
into the
Bayesian set-up in which experiments, viewed as a
random sample from some superpopulation of ex-
periments, are assumed to be exchangeable. When
subjects can
also be viewed as an approximately
random sample from some population,
it
is appro-
priate to pool them across experiments. Otherwise,
analyses that partially pool information according
to experimental heterogeneity need
to be
consid-
ered.
Empirical and
hierarchical Bayes methods
offer a flexible modeling framework for such analy-
ses,
relying on empirical or subjective sources
to
determine the degree of pooling. These richer meth-
ods can be particularly useful to meta-analysis of
experiments
in
parapsychology
conducted
under
potentially diverse conditions.
For the recent ganzfeld
series, assuming them
to be independent binomially
distributed
as
dis-
cussed
in
Section
5,
the
data
can
be summed
(pooled)
across
series
to
estimate
a common
hit
rate. Honorton et al. (1990) assessed the homogene-
ity of effects across the 11 series using a chi-square
test
that
compares
individual
effect
sizes
to
the weighted mean effect. The chi-square statistic
Xi0
==
16.25,
not
statistically
significant
(p =
0.093), largely reflects the contribution of the last
“special”
series (contributes 9.2 units to the
Xio
value), and to a lesser extent the novice series with
a negative effect (contributes 2.5 units). The outlier
series can be dropped from the analysis to provide a
more conservative estimate of the presence of psi
Approved For Release 2000/08/08
:
effects for this data (this result
is reported in Sec-
tion 5). For the remaining 10 series, the chi-square
value
xf = 7.01
strongly favors homogeneity,
al-
though more than one-third of its value
is due to
the novice series (number 4 in Table
1). This pat-
tern points to the potential usefulness of a richer
model to accommodate series that may be distinct
from the others. For the earlier ganzfeld data ana-
lyzed by Honorton (1985b), the appeal of a Bayes or
other
model
that
recognizes
the
heterogeneity
across studies is clear cut: x %3 = 56.6, p = 0.0001,
where only those studies with common chance hit
rate have been included (see Table 2).
Historic reliance on voting-count approaches
to
determine the presence of psi effects makes it natu-
ral
to
consider Bayes
models
that
focus on
the
ensemble of experimental effects from parapsycho-
logical
studies, rather than
individual
estimates.
Recent work in parapsychology that compares
ef-
fect
sizes
across
studies,
rather than
estimating
separate study effects, reinforces the need to exam-
ine this type of model. Louis (1984) develops Bayes
and
empirical
Bayes methods
for problems
that
consider the ensemble
of parameter values
to be
the primary
goal,
for example,
multiple compar-
isons.
For
the
simple compound
normal
model,
7, ~ iV(0 if
1),
0; - iV(/z,
r
),
the
standard Bayes
estimates (posterior means)
T 2
9* =
V- + D{ 7; - n)
and
D =
1 + T
where the
represent experimental effects of in-
terest, are modified approximately to
0/-M+
when an ensemble
loss function
is assumed. The
new
estimates
adjust
the shrinkage
factor D
so
that
their sample mean and variance match the
posterior expectation and variance of the 0’s. Simi-
lar results are obtained when the model
is gener-
Table l
Recent ganzfeld series
Series type
N Trials
Hit rate
y,
Pilot
0.36
-0.58
0.44
Pilot
0.33
-0.71
0.71
Pilot
0.28
-0.94
0.37
Novice
0.24
-1.15
0.33
Novice
0.36
-0.58
0.30
Novice
0.30
-0.85
0.31
Novice
0.36
-0.58
0.30
Novice
0.67
0.71
0.87
Experienced
0.43
-0.28
0.76
Experienced
0.30
-0.85
0.31
Experienced
0.64
0.58
0.42
Overall
355
0.34
-RDP96-00789ROO31 0001 0001 -6
Approved For Release 2000/08/08
: CIA-RDP96-OO789R0O31OOO1OOO1-6
384
J. UTTS
i
Table 2
Earlier ganzfeld studies
N Trials
Hit rate
Y,
<7t
0.44
-0.24
0.36
0.86
1.82
1.09
0.43
-0.28
0.37
0.23
-1.21
0.43
0.10
-2.20
0.75
0.90
2.20
1.05
0.40
-0.41
0.65
0.29
-0.90
0.42
0.40
-0.41
0.65
0.35
-0.62
0.47
0.31
-0.80
0.42
0.45
-0.20
0.45
0.45
-0.20
0.45
0.53
0.12
0.37
0.33
-0.71
0.35
0.28
-0.94
0.39
0.28
-0.94
0.35
0.46
-0.16
0.39
0.60
0.41
0.46
100
0.41
-0.36
0.20
0.33
-0.71
0.34
0.41
-0.36
0.39
0.45
-0.20
0.26
0.21
-1.33
0.35
722
.38
alized
to
the
case
of unequal
variances,
Y* ~
N(6„ of).
For the above model, the fraction of $ above (or
below) a cut point C is a consistent estimate of the
fraction
of
t > C
(or
< C).
Thus,
the
use
of
ensemble,
rather than
component-wise,
loss
can
help
detect
when
individual
effects
are
above
a
specified
threshold by
chance.
For
the
meta-
analysis of ganzfeld experiments, the observed bi-
nomial
proportions transformed on
the
logit
(or
arcsinV) scale can be modeled in this framework.
Letting d
l and m i denote the number of direct hits
and misses respectively for the ith experiment, and
p,
as the corresponding population
proportion
of
direct hits, the Yt are the observed logits
Yi-logK/m,)
and
of,
estimated
by maximum
likelihood
as
1/dj + 1/m;,
is the variance of Y
; conditional on
0; = logit(p;). The threshold logit (0.25) = 1.10 can
be used to identify the number of experiments for
which the
proportion
of direct
hits exceeds
that
expected by chance.
Table
1 shows Y
t and
o
i
for
the
ganzfeld
series. All but one of the series are well above the
threshold; Y4
marginally
falls below
-1.10. Any
shrinkage toward a common hit rate will lead to an
estimate, 0* or 04 , above the threshold. The use of
ensemble
loss (with
its consistency property) pro-
vides more convincing Support that all
0; > - 1.10,
although
posterior
estimates
of uncertainty
are
needed
to
fully
calibrate
this.
For
the
earlier
ganzfeld data in Table; 2, ensemble loss can simi-
larly be used to determine the number of studies
with 0; < —1.10 and specifically whether the nega-
tive
effects
of
studies
and
(Y4 = -1.21
and Y24= “I .33) occurred as a result of chance
fluctuation.
Features of the ganzfeld data in Section 5, such
as the outlier series, suggest that further elabora-
tion of the basic Bayesijan set-up may be necessary
for some meta-analyses
j in parapsychology. Hierar-
chical models provide a:natural framework to spec-
ify
these
elaborations: and
explore how
results
change with the prior
j specification.
This type
of
sensitivity analysis can jexpose whether conclusions
are
closely
tied
to
prior
beliefs,
as
observed by
Jeffreys for RNG data (jsee Section 7). Quantifying
the influence of model; components deemed to be
more subjective or less certain is important to broad
acceptance of results as evidence of psi performance
(or lack thereof).
Consider the
initial model commonly used
for
Bayesian analysis of discrete data:
Yi\pi,n\~ B(pi,nf),
0 i ~N( fx,T 2
),
i = logit(pj),
with noninformative priors assumed for p and
t 2
(e.g., log
r locally uniform). The distinctiveness of
the last “special” series and, in general, the differ-
ent types of series (pilot versus formal, novice ver-
sus experienced) raises j;he question of whether the
experimental effects follow a normal distribution.
Weighted normal plots (Ryan and Dempster, 1984)
can be used to graphically diagnose the adequacy of
second-stage normality (see Dempster, Selwyn and
Weeks,
1983,
for examples with binary response
and normal superpopulation).
Alternatively,
if noniiormality
is suspected, the
model can be revised to include some sort of heavy-
tailed prior to accommodate possibly outlying
se-
ries or studies. West (1985) incorporates additional
scale parameters, one
for each component
of the
model (experiment), that flexibly adapt to a typi-
cal
0;
and
discount
their
influence on
posterior
estimates, thus avoiding under- or over-shrinkage
due
to
such
$ t
.
For
example,
the
second
stage
can specify the prior as a scale mixture of normals:
0; ~ N(n,T 2y~ l
),
k Ji ~ xl
This approach for the prior is similar to others for
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6