Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
ited PRL for several days and was a subject in
Series 101” [pages 134-135].
Honorton has also informed me (personal communi-
cation, July 25, 1991) that several self-proclaimed
skeptics have visited his laboratory and received
demonstrations of the autoganzfeld procedure and
that no one expressed any concern with the secu-
rity arrangements.
This may not completely satisfy Professor Diaco-
nis’ objections, but
it does indicate a serious effort
on the part of the researchers to involve such peo-
ple.
Further,
the
original
publication
of the
re-
search in Section 5 followed the reporting criteria
established by Hyman and Honorton (1986), thus
providing much more detail for the reader than the
earlier
published
records
to
which
Professor
Diaconis alludes.
Points Raised by Greenhouse
cal question is the same. Under the null hypothe-
sis, since the target
is randomly selected from the
four
possibilities
presented,
the
probability
of a
direct hit is 0.25 regardless of who does the judg-
ing. Thus, the observed anomalous effects cannot
be
explained
by
assuming
there
was
an
over-
optimistic judge.
If Professor Greenhouse
is suggesting that the
source
of judging may be a moderating variable
that determines the magnitude of the demonstrated
anomalous
effect,
I
agree. The parapsychologists
have considered this issue in the context of whether
or not subjects should serve as judges for their own
sessions, with differing opinions in different labora-
tories. This is an example of an area that has been
suggested for further research.
Finally, Greenhouse raised the question of the
accuracy of the file-drawer estimates used in the
reported meta-analyses.
I agree that
it
is instruc-
tive to examine the file-drawer estimate using more
Greenhouse enumerated four items that offer al-
ternative explanations for the observed anomalous
effects. Three of these (items 2-4) will be addressed
in this
section by elaborating on the details pro-
vided in my paper. His item 1 will be addressed in
a later section.
Item 2 on his
list questioned the role of experi-
menter expectancy effects as a potential confounder
in parapsychological research. While the expecta-
tions of the experimenter may influence the report-
ing of results, the ganzfeld experiments (as well as
other psi experiments) are conducted in such a way
that experimenter expectancy cannot account
for
the results themselves. Rosenthal, who Greenhouse
cites as the expert in this area, addressed this in
his background paper
for the National Research
Council
(Harris and Rosenthal,
1988a) and
con-
cluded that the ganzfeld studies were adequately
controlled in this regard. He also visited the auto-
ganzfeld laboratory and was given a demonstration
of that procedure.
Greenhouse’s item 3, the question of what consti-
tutes a direct hit, was addressed in my paper but
perhaps needs elaboration. Although free-response
experiments do
generate
substantial amounts
of
subjective
data,
the
statistical
analysis
requires
that the results for each trial be condensed into a
single measure of whether or not a direct hit was
than one model. As an example, consider the 39
studies from the direct hit and autoganzfeld data
bases. Rosenthal’s fail-safe N estimates that there
would have to be 371 studies in the file-drawer to
account
for the
results.
In
contrast,
the method
proposed by Iyengar and Greenhouse gives a
file-
drawer estimate of 258 studies. Even this estimate
is unrealistically large for a discipline with as few
researchers as parapsychology. Given that the av-
erage number of trials per experiment
is 30, this
would represent almost 8000 unreported trials, and
at least that many hours of work.
There are pros and cons to any method of esti-
mating the number of unreported studies, and the
actual practices of the discipline in question should
be taken into account. Recognizing publication bias
as an issue, the Parapsychological Association has
had an official policy since 1975 against the selec-
tive reporting of positive
results. Of the original
ganzfeld studies reported in Section 4 of my paper,
less than half were significant, and it is a matter of
record that there are many nonsignificant studies
and “failed replications” published in all areas of
psi
research.
Further,
the
autoganzfeld database
reported in Section 5 has no file-drawer. Given the
publication practices and the size of the field, the
proposed
file-drawer
cannot
account
for
the
ob-
served effects.
achieved. This is done by presenting four choices to
pojnts Rajsed by Hyman
a judge (who of course does not know the correct
answer) and asking the judge to decide which of the
One of my
goals
in writing this paper was
to
four
best matches
the
subject’s
response.
If the
present a fair account of recent work and debate in
judge picks the target, a direct hit has occurred.
parapsychology.
Thus,
I was
disturbed
that Hy-
It is true that different judges may differ on their
man, who has devoted much of his career to the
opinions of whether or not there has been a direct
study
of parapsychology , and who had first-hand
hit on
ci^tS^-6b^%ft
5b
i
S¥o6b“i)oo
i-
r
#
port3 ’ be '
Approved For Release 2000/08/08
: CIA-RDP96-00789Rj)03100010001-6
398
J. UTTS
lieved that some of my statements were inaccurate
and
indicated
that
I had
not
carefully read
the
reports.
I will address some of his
specific
objec-
tions and show that, except where noted, the accu-
racy of my original statements can be verified by
further elaboration and clarification, with due apol-
ogy for whatever necessary details were lacking in
my original report.
Most
of
our
points
of
disagreement
concern
the National Academy
of Sciences (National Re-
search
Council)
report
Enhancing Human
Per-
formance
(Druckman
and
Swets,
1988).
This
report
evaluated
several
controversial
areas,
in-
cluding parapsychology. Professor Hyman chaired
the Parapsychology Subcommittee.
Several back-
ground papers were commissioned
to accompany
this
report,
available
from
the
“Publication
on
Demand
Program”
of
the
National
Academy
Press. One of the papers was written by Harris and
Rosenthal,
and
entitled
“Human
Performance
Research: An Overview.”
Professor Hyman alleged that “Utts mistakenly
asserts that my subcommittee on parapsychology
commissioned
Harris and Rosenthal
to
evaluate
parapsychology experiments
for us
.
.
. .”
I cannot
find
a statement
in my paper that
asserts
that
Harris and Rosenthal were commissioned by the
subcommittee,
nor
can
I
find
a
statement
that
asserts that they were asked to evaluate parapsy-
chology
experiments.
Nonetheless,
I
believe
our
substantive
disagreement
results
from
the
fact
that the work by Harris and Rosenthal was writ-
ten
in two
parts, both
of which
I referenced
in
my
paper.
They
were
written
several
months
apart,
but
published
together,
and
each
had
its own history.
The
first part (Harris and Rosenthal, 1988a)
is
the
one
to
which
I
referred
with
the
words
“Rosenthal
was
commissioned
by
the
National
Academy
of
Sciences
to
prepare
a
background
paper to accompany its 1988 report on parapsychol-
ogy”
(p.
372).
Accordingito Rosenthal
(personal
communication, July 23, 1991) he was asked to pre-
pare
a background
paper
to
address
evaluation
issues and experimenter effects to accompany the
report in five specific areas of research, including
parapsychology.
The second part was a “Postscript” to the com-
missioned paper (Harris and Rosenthal, 1988b), and
this
is the one to which
I referred on page 371 as
“requested by Hyman in his capacity as Chair of
the National Academy of Sciences’ Subcommittee
on Parapsychology.”
(It
is probably this wording
that led Professor Hyman to his erroneous allega-
tion.) The
postscript began with
the words “We
have been
y^tpe^^||^5(j0ffl(^8
Hyman, chair of the subcommittee on parapsychol-
ogy,
in which he raised questions about the pres-
ence and consequence
of methodological flaws
in
the ganzfeld studies
In reference to this postscript,
I stand corrected
on a technical point, because Hyman himself did
not request the response to his own letter. As noted
by Palmer, Honorton and Utts (1989), the postscript
was added because:
At one stage of the process, John Swets, Chair
of the Committee, actually phoned Rosenthal
and asked him to withdraw the parapsychology
section
of
his
[commissioned]
paper.
When
Rosenthal declined, Sivets and Druckman then
requested that Rosenthal respond to criticisms
that Hyman had included in a July 30, 1987
letter to Rosenthal [page 38].
A related issue on which I would like to elaborate
concerns the correlation between flaws and success
in
the
original
ganzfeld
data
base. Hyman has
misunderstood both my
j position and that of Harris
and Rosenthal. He beliejves that I implicitly denied
the importance
of the iflaws,
so
I
will make my
position
explicit.
I do not think there
is any
evi-
dence that the experimental results were due to the
identified flaws. The flaw analysis was clearly use-
ful
for
delineating
acceptable
criteria
for
future
experiments. Several experiments were conducted
using those criteria. The results were similar to the
original experiments.
I believe that this indicates
an anomaly in need of an explanation.
In discussing the papier and postscript by Harris
and Rosenthal, Hymanj
stated
that “The
alleged
contradictory conclusions [to the National Research
Council report] of Harris and Rosenthal are based
on a meta-analysis that supports Honorton’s posi-
tion when Honorton’s [flaw] ratings are used and
supports my position when my ratings are used.”
He believes that Harris bnd Rosenthal (and I) failed
to see this point because the low power of the test
associated with their analysis was not taken into
account.
The analysis in question was based on a canoni-
cal correlation between; flaw ratings and measures
of successful outcome for the ganzfeld studies. The
canonical correlation was 0.46, a value Hyman finds
to be impressive. What he has failed to take into
account however,
is
that
a
canonical
correlation
gives only the magnitude of the relationship, and
not the
direction. A careful reading of Harris and
Rosenthal (1988b) reveals that their analysis actu-
ally
contradicted
the
idea
that
the
flaws
could
account
for
the
successful
ganzfeld
results,
since
“Interestingly, three of the six flaw variables corre-
janonical
variable
1-6
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
399
and with the outcome canonical variable but three
correlated
negatively ”
(page
2,
italics
added).
Rosenthal (personal communication, July 23, 1991)
verified
that
this was
indeed
the
point he was
trying
to make.
Readers who
are
interested
in
drawing
their
own
conclusions
from
first-hand
analyses can find Hyman’s original flaw codings in
an Appendix
to
his paper (Hyman,
1985,
pages
44-49).
Finally, in my paper,
I stated that the parapsy-
chology chapter of the National Research Council
report critically evaluated
statistically significant
experiments, but not those
that were
nonsignifi-
cant. Professor Hyman “does not know how
[I] got
such an impression,” so
I will clarify by outlining
some of the material reviewed in that report. There
were surveys of three major areas of psi research:
remote viewing (a particular type of free-response
experiment),
experiments
with
random
number
generators, and the ganzfeld experiments. As an
example of where
I got the impression that they
evaluated only significant studies, consider the sec-
tion on remote viewing.
It began by referencing a
published
list of 28 studies. Fifteen of these were
immediately
discounted,
since
“only
.
.
. were
published under refereed auspices” (Druckman and
Swets, 1988, page 179). Four more were then dis-
missed,
since
“Of the
scientifically
reported
experiments, 9 are classified as successful” (page
179). The report continued by discussing these nine
experiments, never again mentioning any
of the
remaining
studies. The
other
sections
of the
report placed similar emphasis on significant stud-
ies.
I
did
not think
this was
a
valid
statistical
method for surveying a large body of research.
Minor Point Raised by Morris
The final clarification
I would like to offer con-
cerns the minor point raised by Professor Morris,
that “When Honorton omitted studies that did not
report direct hits as a measure, he may have biased
his
sample.”
This
possibility
was
explicitly
ad-
dressed by Honorton (1985, page 59). He examined
what would happen if z-scores of zero were inserted
for the 10 studies for which the number of direct
hits was not measured, but could have been. He
found that even with
this conservative
scenario,
the combined
z-score
only dropped from
6.60
to
5.67.
SATISFYING THE SKEPTICS
Parapsychology
is
probably
the
only
scientific
discipline
for which
there
is an
organization
of
skeptics trying to discredit its work. The Commit-
tee for the Scientific Investigation of Claims of the
Approved For Release 2000/08/08
:
Paranormal (CSICOP) was established in 1976 by
philosopher Paul
Kurtz and
sociologist
Marcello
Truzzi when
“Kurtz became
convinced
that
the
time was
ripe
for a more
active crusade against
parapsychology and other pseudo-scientists” (Pinch
and Collins, 1984, page 527). Truzzi resigned from
the organization the next year
(as
did Professor
Diaconis) “because of what he saw as the growing
danger of the committee’s excessive negative zeal
at the expense of responsible scholarship” (Collins
and
Pinch,
1982,
page
84).
In
an
advertising
brochure
for
their
publication
The
Skeptical
In-
quirer, CSICOP made clear its belief that paranor-
mal phenomena are worthy of scientific attention
only
to
the
extent
that
scientists
can
fight
the
growing interest in them. Part of the text of the
brochure read: “Why the sudden explosion of inter-
est, even among some otherwise sensible people, in
all sorts of paranormal ‘happenings’?
.
.
. Ten years
ago, scientists started to fight back. They set up an
organization—The Committee for the Scientific In-
vestigation of Claims of the Paranormal.”
During the
six years that
I have been working
with
parapsychologists, they have repeatedly
ex-
pressed their frustration with the unwillingness of
the skeptics to
specify what would constitute
ac-
ceptable evidence, or even to delineate criteria for
an acceptable experiment. The Hyman and Honor-
ton Joint Communique was seen as the first major
step in that direction, especially since Hyman was
the Chair of the Parapsychology Subcommittee of
CSICOP.
Hyman and Honorton (1986) devoted eight pages
to “Recommendations for Future Psi Experiments,”
carefully outlining details for how the experiments
should be conducted and reported. Honorton and
his
colleagues
then
conducted
several
hundred
trials using these specific criteria and found essen-
tially the same
effect
sizes as in earlier work for
both the overall
effect and effects with moderator
variables taken into account.
I would expect Profes-
sor Hyman to be very interested in the results of
these experiments he helped to create. While he did
acknowledge that they “have produced intriguing
results,”
it
is both
surprising and
disappointing
that he spent only a scant two paragraphs at the
end of his discussion on these results.
Instead, Hyman seems to be proposing yet an-
other
set
of requirements
to
be
satisfied
before
parapsychology should be taken seriously.
It is dif-
ficult to sort out what those requirements should be
from
his
account:
“[They
should]
specify,
in
ad-
vance, the complete sample space and the critical
region. When they get to the point where they can
specify this along with some boundary conditions
and make some reasonable predictions, then they
CIA-RDP96-00789R0031 0001 0001 -6
400
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
J. UTTS
will have demonstrated something worthy of our
attention.”
Diaconis
believes
that
psi
experiments
do
not
deserve serious attention unless they actively
in-
volve
skeptics. Presumably, he
is concerned with
subject or experimenter fraud, or with improperly
controlled experiments. There are numerous docu-
mented cases
of fraud and trickery
in purported
psychic phenomena. Some of these were observed
by Diaconis and reported in his article in Science.
Such cases have mainly been revealed when inves-
tigators attempted to verify the claims of individ-
ual psychic practitioners in quasi-experimental or
uncontrolled conditions. These instances have
re-
ceived considerable attention, probably because the
claims are so sensational, the fraud
is so easy to
detect by a skilled observer and they are an easy
target for skeptics looking for a way to discredit
psychic phenomena. As noted by Hansen
(1990),
“Parapsychology
has
long
been
tainted
by
the
fraudulent behavior of a few of those claiming psy-
chic abilities” (page 25).
Control against deception by subjects in the labo-
ratory has been discussed extensively in the para-
psychological literature (see, e.g., Morris, 1986, and
Hansen,
1990).
Properly
designed
experiments
should
preclude
the
possibility
of
such
fraud.
Hyman and Honorton (1986, page 355)
explicitly
discussed precautions to be taken in the ganzfeld
experiments, all of which were followed in the auto-
ganzfeld experiments. Further the controlled labo-
ratory experiments discussed in my paper usually
used a large number of subjects, a situation that
minimizes the possibility that the results were due
to fraud on the part of a few subjects. As for the
possibility of experimenter fraud, it is of course an
issue in all areas of science. There have been a few
such instances in parapsychology, but since para-
psychologists tend to be aware of this possibility,
they were generally detected and exposed by insid-
ers in the field.
It is not clear whether or not Diaconis is suggest-
ing that a magician or “qualified skeptic” needs to
be present at all times during a laboratory experi-
ment. I believe that it would be more productive for
such consultation to occur during the design phase,
and during the implementation of some pilot ses-
sions.
This
is essentially what was done
for the
autoganzfeld experiments, in which Professor Hy-
man, a skeptic as well as an accomplished magi-
cian,
participated
in
the
specification
of design
criteria, and mentalists Bern and Kross observed
experimental sessions. Bern is also a well-respected
experimental psychologist.
While
I
believe
that the
skeptics,
particularly
some
of
the
more
knowledgeable
members
of
CSICOP, have served
a! useful role in helping to
improve experiments, their counter-advocacy stance
is counterproductive.
If they are truly interested
in
resolving the
question
of whether
or not
psi
abilities
exist,
I would expect them to encourage
evaluation
and
experimentation
by
unbiased,
skilled
experimenters.
Instead,
they seem
to be
trying to discourage
su<j:h interest by providing a
moving target of requirements that must be satis-
fied first.
SHOULD PSI RESEARCH BE IGNORED BY THE
SCIENTIFIC 'COMMUNITY?
In the conclusion of my paper,
I argued that the
scientific community should pay more attention to
the experimental results in parapsychology.
I was
not suggesting that the Accumulated evidence con-
stitutes proof of psi
abilities, but rather
that
it
indicates that there is indeed an anomalous effect
that needs an explanation. Greenhouse noted that
my paper will not necessarily change anyone’s view
about the existence of paranormal phenomena, an
observation with which I agree. However, I hope it
will change some views about the importance
of
further investigation.
Mosteller and Diaconis both acknowledged that
there are reasons for statisticians to be interested
in studying
the anomalous
effects,
regardless
of
whether or not psi
is real. As noted by Mosteller,
“If there
is no ESP,
thjen we want to be able
to
carry out null experiments and get no effect, other-
wise we cannot put mu<jh belief in work on small
effects in non-ESP situations.” Diaconis concluded
that “Parapsychology
isj worthy of serious study”
partly because
“If
it
is wrong,
it
offers
a
truly
alarming massive case Study of how statistics can
mislead and be misused.”
Greenhouse noted several sociological reasons for
the resistance of the scientific community to accept-
ing parapsychological phenomena. One of these
is
that they
directly contradict the laws of physics.
However, this assertion! is not uniformly accepted
by physicists
(see,
e.g., Oteri,
1975), and some of
the
leading
parapsychological
researchers
hold
Ph.D.s in physics.
Another reason
cited by Greenhouse, and sup-
ported by Hyman,
is that psychic phenomena are
currently unexplainable jby a unified scientific the-
ory. But that is precisely! the reason for more inten-
sive
investigation.
The
history
of
science
and
medicine is replete with examples where empirical
departures from expectation led to important find-
ings or theoretical models. For example, the causal
connection between
cigarette
smoking and
lung
cancer was established only after years of statisti-
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6