Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
385
maximum
likelihood estimation
that modify
the
sampling error distribution to yield estimates that
are “robust” against outlying observations.
Like its maximum likelihood counterparts, in ad-
dition to the robust effect estimates $*, the Bayes
model provides (posterior) scale estimates y *. These
can be interpreted as the weight given to the data
for each
in the analysis and are useful to diag-
nosing which model components (series or studies)
are unusual and how they influence the shrinkage.
When more complex groupings among the
t are
suspected,
for
example,
bimodal
distribution
of
studies from different sites or experimenters, other
mixture specifications can be used to further relax
the shrinkage toward a common value.
For
the
ganzfeld
series,
the
last
“outlier”
series,
quite
distinct from the others
(hit rate =
0.64),
is moderately precise (N = 25). Omitting
it
from the analysis causes the overall hit rate to drop
from 0.344 to 0.321. The scale mixture model
is a
compromise between these two values (on the logit
scale), discounting the influence of series 11 on the
estimated
posterior
common
hit
rate
used
for
shrinkage. The
scale
factor 7
*
lt an indication
of
how separate 0n is from the other parameters, also
causes 0*
x to be shrunk less toward the common hit
rate than other, more homogeneous 0,, giving more
weight to individual information for that series (see
West,
1985).
The
heterogeneity
of
the
earlier
ganzfeld data
is more pronounced, and studies are
taken from a variety of sources over time. For these
data, the y* can be used to explore atypical studies
(e.g., study 6, with hit rate = 0.90, contributes more
than 25%
to the x %3
value
for homogeneity) and
groupings among
effects,
as
well
as
protect the
analysis
from
misspecification
of
second-stage
normality.
Variation among ganzfeld series or studies and
the degree to which pooling or shrinking is appro-
priate can be investigated further by considering a
range of priors for r 2
. If the marginal likelihood of
r 2 dominates the prior
specification, then results
should not vary as the prior for r2 is varied. Other-
wise, it is important to identify the degree to which
subjective
information
about
interexperimental
variability
influences
the
conclusions.
This
sen-
sitivity
analysis
is
a
Bayesian
enrichment
of
the simpler
test
of homogeneity
directed toward
determining whether
or
not complete
pooling
is
appropriate.
To
assess how
well
heterogeneity among
his-
torical control groups
is determined by the data.
Dempster, Selwyn and Weeks (1983) propose three
priors for t 2 in the logistic-normal model. The prior
distributions range from strongly favoring individ-
ual estimates, p(t 2 )cIt oc t _1
, to the uniform refer-
ence prior p(t 2 )<It oc t'2
, flat on the log
r scale, to
strongly favoring complete pooling, p(t 2 )c/t oc T
~ 3
(the
latter forcing complete pooling
for the com-
pound normal model; see Morris, 1983). For their
two examples, the results (estimates of linear treat-
ment effects) are largely insensitive to variation in
the prior distribution, but the number of studies in
each example was large (70 and 19 studies avail-
able for pooling). For the 11 ganzfeld series, t 2 may
be less well determined by the data. The posterior
estimate of r 2 and its sensitivity to p(r 2 )o?r
will
also depend on whether individual scale parame-
ters are incorporated into the model. Discounting
the influence of the last series will both shift the
marginal
likelihood toward smaller values
of
r 2
and concentrate it more in that region.
The issue of objective assessment of experiment
results is one that extends well beyond the field of
parapsychology, and this paper provides insight into
issues surrounding the analysis and interpretation
of small effects from related studies. Bayes meth-
ods can contribute
to such meta-analyses
in two
ways. They permit experimental and subjective evi-
dence
to be formally combined
to determine the
presence or absence of effects that are not clear cut
or controversial
(e.g., psi abilities). They can also
help uncover sources and degree of uncertainty in
the scientific conclusions.
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
386
j. UTTS
Comment
Persi Diaconis
In my experience, parapsychologists use
statis-
tics extremely
carefully. The
plethora
of widely
significant p-values in the many thousands of pub-
lished parapsychological studies must give us pause
for thought. Either something spooky
is going on,
or
it
is possible
for a
field to
exist on error and
artifact for over 100 years. The present paper offers
a useful review by an expert and a glimpse at some
tantalizing new studies.
My
reaction
is
that
the
studies
are
crucially
flawed. Since my reasons are somewhat unusual,
I
will try to spell them out.
I have found it impossible to usefully judge what
actually went on
in a parapsychology
trial from
their published
record. Time
after time,
skeptics
have gone
to watch
trials and found
subtle and
not-so-subtle errors. Since the field has so far failed
to produce
a
replicable phenomena,
it seems
to
me that any
trial that asks us
to take
its
find-
ings seriously should include
full participation by
qualified
skeptics.
Without
a
magician
and/or
knowledgeable psychologist skilled at running ex-
periments with human
subjects,
I
don’t think
a
serious effort is being made.
I
recognize
that
this
is
an
unorthodox
set
of
requirements.
In
fact,
one
cannot
judge
what
“really goes on” in studies in most areas, and
it is
Persi Diaconis
is Professor of Mathematics at Har-
vard
University,
Science
Center,
Oxford
Street,
Cambridge, Massachusetts 02138.
impossible to demand \yide replicability in others.
Finally, defining “qualified skeptic” is difficult. In
defense, most areas have many
easily
replicable
experiments
and many
have
their
findings
ex-
plained and connected tjy unifying theories. It sim-
ply seems clear that when making claims at such
extraordinary variance
j with our daily experience,
claims that have been made and washed away so
often in the past, such extraordinary measures are
mandatory before one hhs the right to ask outsiders
to spend their time in review. The papers cited in
Section 5 do not actively involve qualified skeptics,
and
I do not feel they have earned the right to our
serious attention.
The points I have made above are not new. Many
appear in the present article. This does not dimin-
ish their utility nor applicability to the most recent
studies.
Parapsychology
is
wbrth
serious
study.
First,
there may be something there, and I marvel at the
patience and drive of pfeople like Jessica Utts and
Ray Hyman. Second, if it is wrong, it offers a truly
alarming massive case study of how statistics can
mislead and be misused. Third,
it offers marvelous
combinatorial
and
inferential
problems.
Chung,
Diaconis, Graham
and; Mallows
(1981),
Diaconis
and
Graham
(1981)
and
Samaniego
and
Utts
(1983) offer examples ndt cited in the text. Finally,
our budding statistics students are fascinated by its
claims;
the
present
paper
gives
a
responsible
overview providing background
for a
spectacular
classroom presentation,
j
Comment: Parapsychology —On the Margins
of Science?
Joel B. Greenhouse
Professor Utts reviews and synthesizes a large
body of experimental literature as well as the scien-
tific controversy involved in the attempt to estab-
Joel B. Greenhouse
is Associate Professor of Statis-
tics, Carnegie Mellon
University, Pittsburgh, Penn-
sylvania 15213-3890.
Approved For Release 2000/08/08
:
lish the existence of paranormal phenomena. The
organization and
clarity
of her presentation
are
noteworthy. Although
I
do
not believe
that
this
paper
will
necessarily change anyone’s views
re-
garding the existence of paranormal phenomena,
it
does raise very interesting questions about the pro-
cess by which new
idehs
are
either
accepted
or
rejected by the scientific community. As students of
science,
we
believe
that
scientific
discovery
CIA-RDP96-00789R0031 0001 0001 -6
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION OF PARAPSYCHOLOGY
387
advances methodically and objectively through the
accumulation of knowledge (or the rejection of false
knowledge) derived from the implementation of the
scientific method. But, as we will see, there is more
to the acceptance of new scientific discoveries than
the
systematic
accumulation
and
evaluation
of
facts. The recognition that there is a social process
involved with the acceptance or rejection of scien-
tific knowledge has been the
subject
of study
of
sociologists
for some time. The
scientific commu-
nity’s rejection of the existence of paranormal phe-
nomena
is an excellent case study of this process
(Allison, 1979; Collins and Pinch, 1979).
Implicit
in
Professor
Utts’
presentation
and
paramount to the acceptance of parapsychology as
a legitimate science are the description and docu-
mentation of the professionalization of the field of
parapsychology. It is true that many researchers in
the field have university appointments; there are
organized
professional
societies
for
the
advance-
ment
of parapsychology;
there
are journals with
rigorous standards for published research; the field
has received funding from
federal
agencies; and
parapsychology has received recognition from other
professional
societies,
such
as
the IMS and
the
American Association for the Advancement of Sci-
ence (Collins and Pinch, 1979). Nevertheless, most
readers
of
Statistical
Science
would
agree
that
parapsychology is not accepted as part of orthodox
science and
is considered by most of the scientific
community to be on the margins of science, at best
(Allison,
1979;
Collins and Pinch,
1979). Why
is
this
the
case?
Professor
Utts
believes
that
it
is
because people have not examined the data. She
states that “Strong beliefs tend to be resistant to
change even in the face of data, and many people,
scientists
included, seem
to have made up
their
minds on the question without examining any em-
pirical data at all.”
The history of science is replete with examples of
resistance by the established scientific community
to new discoveries. A challenging problem for
sci-
ence
is to understand the process by which a new
theory or discovery becomes accepted by the com-
munity of scientists and, likewise, to characterize
the nature of the resistance to new ideas. Barber
(1961)
suggests
that
there
are
many
different
sources of resistance to scientific discovery. In 1900,
for example, Karl Pearson met resistance to his use
of statistics in applications to biological problems,
illustrating a source of resistance due to the use of
a
particular methodology. The Royal
Society
in-
formed Pearson that future papers submitted to the
Society for publication must keep the mathematics
separate from the biological applications.
Another obvious source of resistance to new sci-
Approved For Release 2000/08/08
:
entific ideas, and the one referred to by Professor
Utts
above,
is
the
prevailing substantive
beliefs
and theories held by scientists at any given time.
Barber offers the opposition to Copernicus and his
heliocentric theory and
to Mendel’s theory of ge-
netic inheritance as examples
of how, because
of
preconceived ideas, theories and values, scientists
are not
as open-minded
to new advances as one
might think they should be.
It was R. A. Fisher
who said that each generation seems to have found
in Mendel’s paper only what it expected to find and
ignored what did not conform to
its own expecta-
tions (Fisher, 1936).
Pearson’s response to the antimathematical prej-
udice expressed by the Royal Society was to estab-
lish
with
Galton’s
support
a
new
journal,
Biometrika, to encourage the use of mathematics in
biology. Galton (1901) wrote an article for the first
issue
of the journal, explaining the need for this
new voice of “mutual encouragement and support”
for mathematics in biology and saying that “a new
science cannot depend on a welcome from the
fol-
lowers
of the
older
ones, and
[therefore] ... it
is
advisable to establish a special Journal for Biome-
try.” Lavoisier understood the role of preconceived
beliefs as a source of resistance when he wrote in
1785,
I do not expect my ideas to be adopted all at
once. The human mind gets creased into a way
of seeing
things.
Those who have envisaged
nature
according
to
a
certain
point
of view
during much
of their
career,
rise
only with
difficulty to new ideas. (Barber, 1961.)
I suspect that this paper by Professor Utts syn-
thesizing the accumulation of research results sup-
porting
the
existence
of paranormal phenomena
will continue to be received with skepticism by the
orthodox scientific community “even after examin-
ing the data.” In part, this resistance is due to the
popular perception of the association between para-
psychology and the occult (Allison, 1979) and due
to
the continued
suspicion and documentation
of
fraud in parapsychology (Diaconis, 1978). An addi-
tional and important
source
of resistance
to the
evidence presented by Professor Utts, however,
is
the
lack
of a model
to
explain
the phenomena.
Psychic phenomena are unexplainable by any cur-
rent
scientific
theory
and,
furthermore,
directly
contradict the laws
of physics. Acceptance
of psi
implies the rejection of a large body of accumulated
evidence
explaining
the
physical
and
biological
world as we know
it. Thus, even though the effect
size
for
a
relationship
between
aspirin and
the
prevention of heart attacks
is three times smaller
than the effect
size observed in the ganzfeld data
CIA-RDP96-00789R0031 0001 0001-6
Approved For Release 2000/08/08
: CIA-RDP96-00789RQ03100010001-6
388
J. UTTS
base,
it is the existence of a biological mechanism
tions about the role of publication bias. Publication
to
explain
the
effectiveness
of
aspirin
that
ac-
counts, in part, for acceptance of this relationship.
In evaluating the evidence in favor of the exis-
tence of paranormal phenomena,
it is necessary to
consider alternative explanations or hypotheses for
the results and, as noted by Cornfield (1959), “If
important
alternative hypotheses
are
compatible
with available evidence, then the question is unset-
tled, even
if the
evidence
is experimental”
(see
also Platt, 1964). Many of the experimental results
reported by Professor Utts need to be considered in
the context of explanations other than the
exist-
ence
of
paranormal
phenomena.
Consider
the
following examples:
bias or “the file-drawer problem” arises when only
statistically
significant
findings
get
published,
while statistically nonsignificant studies
sit unre-
ported
in
investigators’
file
drawers.
Typically,
Rosenthal’s method (1979) is used to calculate the
“fail-safe N,” that
is,; the number of unreported
studies that would have to be sitting in file-drawers
in order to negate
the! significant
effect. Iyengar
and Greenhouse (1988) describe a modification of
Rosenthal’s method, however, that gives a fail-safe
N that is often an order of magnitude smaller than
Rosenthal’s method, suggesting that the sensitivity
of the results of meta-analyses of psi experiments to
unpublished
negative
studies
is
greater than
is
currently believed.
(1) In the various psi experiments that Professor
Utts
discusses,
the
null
hypothesis
is
a
simple
chance model. However, as noted by Diaconis (1978)
in
a
critique
of parapsychological
research,
“In
complex,
badly
controlled
experiments
simple
chance models cannot be
seriously considered as
tenable explanations: hence, rejection of such mod-
els
is not of particular interest.” Diaconis shows
that the underlying probabilistic model in many of
these experiments (even those that are well con-
trolled) is much more complicated than chance.
(2) The role that experimenter expectancy plays
in the reporting and interpreting of results cannot
be underestimated. Rosenthal
(1966), based on a
meta-analysis of the
effects of experimenters’
ex-
pectancies on the results of their research, found
that experimenters tended to get the results they
expected
to
get.
Clearly this
is an important po-
tential confounder
in parapsychological research.
Professor
Utts
comments
on
a
debate
between
Honorton and Hyman, parapsychologist and critic,
respectively,
regarding
evidence
for
psi
abili-
ties, and, although not necessarily a result of ex-
perimenter
expectancy,
describes
how
“.
.
each
analyzed
the
results
of
all known
psi
ganzfeld
experiments to date, and reached strikingly differ-
ent conclusions.”
(3) What
is an acceptable response in these ex-
periments? What constitutes a direct hit? What
if
the response
is close, who decides whether or not
that constitutes a hit (see (2) above)? In an example
of a response of a Receiver in an automated ganzfeld
procedure, Professor Utts describes the “dream-like
quality of the mentation.” Someone must evaluate
these
stream-of-consciousness
responses
to
deter-
mine what
is a
hit. An important methodological
question is: How sensitive are the results to differ-
ent definitions of a hit?
(4) In describing the
results
of different meta-
analyses,
Professor Utts
is careful
to raise ques-
Approved For Release 2OOO/08/O8
Even
if parapsychology
is thought to be on the
margins
of science by
the
scientific community,
parapsychologists should not be helci to a different
standard of evidence to! support their findings than
orthodox
scientists, but
like other
Scientists they
must be concerned wiih spurious
effects and the
effects of extraneous variables. The experimental
results summarized by IProfessor Utts appear to be
sensitive to the effect of alternative hypotheses like
the
ones
described
above.
Sensitivity
analyses,
which question, for example, how large of an effect
due to experimenter expectancy there would have
to be to account for the effect sizes being reported
in
the
psi
experiment^,
are
not addressed
here.
Again, the ability to account for and eliminate the
role
of alternative
hypotheses
in
explaining
the
observed relationship between aspirin and the pre-
vention of heart attacks
is another reason for the
acceptance of these results.
A major new technology discussed by Professor
Utts in synthesizing thb experimental parapsychol-
ogy literature is meta-analysis. Until recently, the
quantitative
review and
synthesis
of a
research
literature, that is, meta-analysis, was considered by
many to be a questionable research tool (Wachter,
1988). Resistance by statisticians to meta-analysis
is
interesting
because,: historically, many promi-
nent statisticians found the combining of informa-
tion from independent studies to be an important
and
useful
methodologiy
(see,
e.g.,
Fisher,
1932;
Cochran, 1954; Mosteller and Bush, 1954; Mantel
and Haenszel, 1959). Peirhaps the more recent skep-
ticism about meta-analysis is because of its use as a
tool
to advance discoveries that themselves were
the
objects
of resistance,
such as the
efficacy
of
psychotherapy (Smith end
Glass,
1977) and now
the existence of paranormal phenomena.
It
is an
interesting problem
fort the
history
of science
to
explore why and when
in the development
of a
: CIA-RDP96-00789R0031 0001 0001 -6