Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
of Science
(AAAS)
in 1969,
signaling a tentative new status and recognition of
the
field.
In 1976
a group of philosophers',
magicians,
science writers,
scientists,
and others concerned about
a widespread increase
in popular interest
in the occult formed a Committee
for the Scientific Investigation of Claims of
the Paranormal
(CSICOP). ^
CSICOP and a number of affiliated local groups have
since become
a primary source of criticism and skepticism directed primarily
at
highly publicized claims
of the paranormal,
but often extending more generally
to
the field of parapsychology.
A highly charged and long-running debate has
emerged,
featuring charges
and countercharges between members of CSICOP and
the
scientific community,
on the one hand,
and members
of the parapsychological
community,
on the other.
One recent chapter of that debate appeared in the March 1985
issue
of The
Journal of Parapsychology
,
in which a leading parapsychology researcher
(Charles
Honorton)
and a leading psychologist and critic
(Ray Hyman)
presented their
views on
the nature of one category of parapsychology experiments.^
In the
December 1986
issue of the
same journal,
the
two published a "joint communique"
in which they explored in a constructive manner areas of agreement and
disagreement regarding experiment design,
documentation,
and data analysis.
(Both individuals participated
in the OTA workshop.)
Another landmark
publication appeared
in 1987:
a substantial portion of one
issue
of Behavioral
and Brain Sciences was devoted
to articles and letters
on the
subj ect
.
The most recent episode
in the continuing controversy over the field
appeared in the form of a December
1987
report by
the National Research Council
(NRC)
,
which evaluated parapsychology along with other possible mechanisms
for
enhancing the performance of U.S.
Army personnel.'
The NRC's evaluation of the
potential of parapsychology was generally negative,
prompting strenuous and
detailed claims of bias and unfairness
from the parapsychological community.**
PURPOSE OF THE OTA WORKSHOP
It was
in this context that
the OTA workshop took place.
OTA examined
parapsychology at the request of
its oversight body,
the Technology Assessment
Board,
which expressed interest
in the human potential aspect of
It should be noted that few scientists were involved
In the creation of
CSICOP.
Although today many of
its Fellows are scientists,
the organization
has always been led and operated by non- scientists
.
4 Hyman,
R.
The ganzfeld psi experiment:
A critical appraisal.
and Honorton,
C.
Meta-analysis of psi ganzfeld research:
A response
to Hyman.
The Journal
of Parapsychology
.
1985,
49(1):
3-91,
5 Hyman,
R.
&
C.
Honorton.
A joint communique:
The psi ganzfeld controversy.
The Journal of Parapsychology
.
1986,
50(4):
351-364.
Behavioral and Brain Sciences
.
1987,
10(4).
7 National Research Council.
Enhancing Human Performance:
Issues,
Theories,
and Techniques
.
Report of
the Committee on Techniques
for
the Enhancement of
Human Performance,
Commission on Behavioral and Social
Sciences and
Education.
Washington,
DC:
National Academy Press,
1988.
See,
for example:
Palmer,
J.A.,
C.
Honorton,
& J.
Utts.
"Reply
to
the
National Research Council Study on Parapsychology."
Special report prepared
for
the Board of Directors
of
the Parapsychological Association,
Inc.,
Research Triangle
Park,
N.C.,
1988.
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
parapsychological research.
This workshop and report are OTA's response
to
that
interest.
OTA's
intention in holding the workshop was not
to continue
the
debate described above,
but
to
illuminate
it by identifying
the main points
of
contention and the reasons
for
them.
For
that reason,
background materials were
researched to provide
a fuller elucidation of some issues not adequately aired
in the course of the one -day meeting.
Discussion was limited to
the area of laboratory parapsychology.
Participants represented three
groups:
researchers
in parapsychology,
constructive critics
of the
field,
and knowledgeable
individuals who follow the
field from a position of more neutral
interest or involvement.
Although the
meeting was
to some extent an occasion for spirited debate,
no attempt has been
made
to
"score”
that debate.
This report will describe
the controversies
only
to
the extent that
they bear on OTA’s
inquiry into
the status of the
field.
II.
DESIGN AND CONDUCT OF EXPERIMENTS
Controversy in the area of experiment design and conduct appears
to center
around the
two
issues
of flaws
in methodology and replicability of experiments.
METHODOLOGY
One of the most persistent and perhaps most damaging charges
leveled by the
critics
is
that various
flaws
in the design and conduct of nearly all
parapsychology experiments render their results scientifically less meaningful
than they are reported to be
--
and,
in fact,
often account for the results.
Parapsychologists dispute
this charge,
saying that most of the alleged flaws
have little or no effect on the experimental findings
that indicate
"psi,"
and
that much research in psychology
(the home discipline of many critics)
--
and
indeed in many other fields
of science
--
exhibits similar flaws without
encountering comparable criticism and questioning of the results.
They say the
question of flaws
is highly subjective, with individual views differing widely
over what distinguishes an experimental
"flower”
from an experimental
"weed."
Further,
the parapsychologists
claim,
this criticism
is
a red herring intended
to tarnish
the
image of parapsychology within the scientific community and
the
public and to impede broader awareness
of findings and progress
in the
field.
Those
at the workshop asserted strenuously that they and their colleagues make
every effort
to conduct their experiments
in the most rigorous manner possible.
The real
issue,
they
say,
is what constitutes
a
flaw.
The
types
of flaws
that
have been brought
into question include purported procedural flaws
such
as:
o
inadequate precautions against
"sensory leakage"
(for example,
allowing
subjects
in a telepathy experiment
to examine
the
same
"target",
such
as
a
photograph,
that had been handled earlier by a person serving as
"sender")
o
inadequate security provisions
(e.g.,
to prevent tampering with equipment)
o
improper randomization techniques
(such
as hand shuffling of cards)
o
feedback
(such
as
failure
to randomize
targets before presenting them to
the subject
for judging)
o
incomplete documentation of experimental procedures
o
inconsistency of conditions
and procedures used
in an experiment.
Also
at issue are various purported analytical and statistical flaws,
including:
o multiple testing or analysis
(i.e.
,
to
find the
test or analysis
that gives
the most positive
"effect size"
for psi)
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
o
underestimation of the effective error rate and overestimation of the
actual significance level
o
erroneous use of particular statistical procedures.
In their joint communique,
Honorton argued that there
is
"no significant
correlation between indices of study quality
[flaws]
and study outcome.".
Hyman
agreed in most cases,
but insisted that there
is
a positive correlation in the
case of poor randomization,
feedback,
and inadequate documentation.
(This
assertion was rebutted in a later analysis.) 10
Workshop participants generally
agreed that in any event
the
"flawless," perfect experiment
is very rare
in any
field,
if it exists
at all.
The point of the
issue,
critics
say,
is
that
the
presence of a flaw usually implies
inadequate controls
for error
.
The most
stringent among them believe that an apparently successful experiment cannot be
considered to have demonstrated a true anomaly
(i.e.,
a possible psi effect)
unless
it can be shown that the experiment
is completely flawless.
The
parapsychologists counter that any such flaws should be an issue only
if they
could have systematically influenced the result.
Another point made by parapsychologists
is
that,
by and large,
the critics
have done very little experimentation in parapsychology.
Therefore,
they have
little awareness of the difficulties encountered or of what
the
truly
significant flaws might be.
Indeed,
the parapsychologists note,
there could be
additional
flaws
that have not yet been considered and which critics could
identify through actual experimentation.
The NRC report distinguished three types
of criticisms relating
to
flaws
in
experiment design and conduct. 11
First
is what
is called the
"smoking gun,
in
which observed findings are said to be definitely attributable
to a specific
flaw or other factor
(such as deliberate
fraud)
that
is
shown to be present.
Second,
and more common,
is
the
"plausible alternative"
allegation that
a
particular
flaw
is present and could have produced the reported results
.
Third
is
the
"dirty test
tube"
concept,
which alleges not that
a particular discovered
flaw produced the positive result but simply that
the presence
of flaws
in the
experiment demonstrates
a general sloppiness which brings
the results
into
question.
In other words,
if results have been obtained under conditions
that
fail
to meet
"generally accepted standards"
for scientific research
(such
as
clean test
tubes)
,
that fact alone casts doubt on the accuracy and validity of
the results.
The authors
of the NRC report claimed that,
while
they could find
no instances of the
"smoking gun"
or "plausible alternative,"
even the best
parapsychology experiments exemplify the
"dirty test tube" problem
(an
allegation that has been heatedly disputed by parapsychologists)
.
Participants agreed that
the burden of proof in the
first
two categories
should be on the critic making the allegation,
as
it
is
in other sciences where
an experiment
is challenged on these grounds.
In the case of
the
dirty test
tube,"
the critics
feel
that
the burden of proof should be on the experimenter.
This position brings
from
the parapsychologists
the objection that
it
is non-
9 Hyman & Honorton,
op.
cit.,
p.
353.
Harris,
M.J.
& Rosenthal,
R.
"Interpersonal Expectancy Effects and the
Human Performance Research."
Washington,
DC:
National Academy Press,
1988.
(p.
3)
11 National Research Council,
op.
cit.,
pp.
199-200.
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
falsifiable
--
that
is,
that there
is no definitive defense against the charge
of
" eeneral sloppiness".
Without detailed criteria for what constitutes
aroeftable research
they
say,
criticism of an experiment cannot be anticipated
“ routed
In their vU„,
it
Is an example of the unfairly high standards
that
ar. 'ft for parapsychology research,
in which no amount of cars
is ever
M enough
.
Tlie critics present at
the workshop tended to agree
in principle with the
unfairness of -non-falsifiability,- but
they believe that the -dirty test
tube"
charge
is nevertheless valid because
the success of the experiments rests on
whether a small but significant departure
from some statistical norm or
°bs ®
norm has
in fact been seen,
making
it essential
to eliminate extr^®°U
J f
f^
-
Parapsychologists object that the charge itself
is
irrelevant
--
that
if
alleged flaw! or
"dirt" have no demonstrable connection with the experimental
outcome,
it
is capricious
to argue
that the experiment
is
invalid.
The litany of charge and countercharge could be detailed much further.
Suffice
it
to say that the controversy surrounding parapsychology
is nowhere
more heated and complex than in the area of experimental methodologies.
REPLICABILITY
For critics of parapsychology,
a key argument against the existence of psi
phenomena
is
the difficulty of replicating
(repeating]
>
^beha^orai'
a difficulty that also plagues many accepted areas of the social and behavior
sciences
If the same effect could be produced many times by different
experimenters using different equipment but
the
same procedures,
then the
likelihood that an extraneous variable
is producing the effect would be much
i
ar,H
the validitv level of the anomalous results would be higher.
However,
inmost eases maddeningly
(to
to replicate,
even with the same operator.
(Because
^lt to repiicate with
highly operator- specific,
results are even more difficult
to replicate witn
different operators or groups of operators.)
Parapsychologists attribute
this replicability problem to
the elusiveness
merely demonstrates
the effects
of randomness combined with multiple
flaws m
experiment design and conduct.
The parapsychologists are wall aware
that axparinantal flaws
both actual
and alleged, become less
Important
as
a phenomenon bacons
_
As
Therefore,
in recent years
they have placed gtea
P
success
or
failure
of
one noted in his presentation,
"The
final
° f
Q dgVel
procedures
that
I
my research program is
the extent
to which
I m ab
are ^ble
tQ obtain
can articulate with sufficient precision that other peop
similar results."
in the case of ESP-oriented experiments
(such as ““«
rP“"P““
r"
e„ts
" ganzfeld" psi experiments),
the number of these mor
termed
is still not
large.
There,
parapsychologists have
^Tatlstical
"meta- analysis"
of many studies,
taken as
a group,
g
tionale
j. s
that
"an
confirmation for
a pattern of positive results.
The
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2