Approved For Release 2000/08/08
: CIA-RDP96-00789R002200450001-8
ff
1 ® iflabl
f
i
that
is
’
that there
is no definitive defense against the charge
of
general sloppiness".
Without detailed criteria for what constitutes
re search,
they say,
criticism of an experiment cannot be anticipated
C
J
d *
In thei* 7
iew
’
ifc
is an example of the unfairly high standards
that
are
set for parapsychology research,
in which no amount of care
is ever
"enough."
cri
£
ias Present at the workshop tended to agree
in principle with the
nfaimess of
non- falsiflability, " but they believe that the
"dirty test tube"
charge
is nevertheless valid because the success of the experiments rests on
w e
er a sma
but significant departure from some statistical norm or observed
norm has
in fact been seen, making it essential
to eliminate extraneous
factors
Parapsychologists object that the charge itself
is irrelevant
--
that if the
alleged flaws
or
"dirt" have no demonstrable connection with the experimental
outcome,
it
is capricious
to argue that the experiment
is
invalid.
c
4r^, The
. litany of charSe and countercharge could be detailed much further.
:V°
!
ay thft tha controversy surrounding parapsychology is nowhere
more heated and complex than in the area of experimental methodologies.
REPLICABILITY
For critics of parapsychology,
a key argument against the existence of psi
ph®"°“®na
is
difficulty of replicating
(repeating)
successful experiments
--
a airriculty that also plagues many accepted areas of the social and behavioral
sciences.
If the same effect could be produced many times by different
USinS different equipment but the same procedures,
then the
likelihood that an extraneous variable
is producing the effect would be much
lower and the validity level of the anomalous results would be higher.
However
positive results are
in most cases maddeningly
(to parapsychologists)
difficult
to replicate,
even with the same operator.
(Because
the effects seem
to be
highly operator- specific,
results are even more difficult to replicate with
different operators or groups of operators
.
)
Parapsychologists attribute
this replicability problem to
the elusiveness
and weakness of the psi
"signal,”
and to
its vulnerability
to subtle subjective
factors
,
most of them unknown.
Critics contend that the near non- replicability
merely demonstrates
the effects of randomness combined with multiple flaws
in
experiment design and conduct.
The parapsychologists are well aware
that experimental flaws, both actual
and alleged,
become less important as
a phenomenon becomes more replicable.
Therefore,
in recent years they have placed great emphasis on replicability.
As
one noted in his presentation,
"The final criterion of the success or failure of
my research program is the extent
to which I'm able
to develop procedures
that
I
can articuiate with sufficient precision that other people are able
to obtain
similar results."
*n the case of ESP-oriented experiments
(such as remote perception or
ganzfeld" psi experiments),
the number of these more standardized experiments
is
still not large.
There,
parapsychologists have relied upon what
is
termed
meta-analysis"
of many studies,
taken as
a group,
to give statistical
confirmation for a pattern of positive
results.
The rationale
is
that
"an
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200450001-8
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200450001-8
experiment or an effect may be considered replicated if a series of replication
attempts provides statistically significant evidence for the original effect
when analyzed as
a series. 1,12
It
is also contended that this approach allows
the analyst
to assess methodological flaws
to determine their effect empirically
across
the data base.
*
y
Using the meta-analytic approach,
parapsychologists have calculated an
overall
"success"
rate,
in achieving statistically significant effects (e<- 05 ),
of 45% for ganzfeld ESP studies and 21% for random event generator
(REG)
studies.
However,
critics have objected strongly to these findings,
saying that
many of the studies included are not directly comparable due to differences
in
methodology.
They also claim that many unsuccessful experiments have not been
published and are therefore not in the data base.
This
"file drawer problem",
they assert,
artificially increases
the success ratio and the overall effect
size.
They also cite a corresponding "retrospective study" problem,
in which
small exploratory studies are included in the data base after the fact
if their
results turn out positive. 14
Furthermore,
they believe,
the proliferation of
flaws
in methodology simply perpetuates
the same erroneous results.
The debate over replicability,
particularly in the context of
meta- analysis,
has been heated and continues
to be
so.
However,
the nature of
the controversy
is currently being outlined in detail and areas of agreement are
being identified.
For example,
the recent joint communique by two of the
workshop participants refers
to
"the growing role we believe meta-analysis will
play, both in the evaluation of research quality and in the assessment of
moderating variables.
We urge parapsychological investigators
to plan and
report their experiments with the idea that their single experiment will
contribute
to a future meta-analysis." 1 ^
This
idea relies
in part on guidelines for greater uniformity in experiment
design and conduct which the two authors specified in their article.
Such
uniformity or consistency
is viewed as crucial by many moderate critics
,
who
bo
the fact that there has never been a successful parapsychology
experiment which any and all competent researchers,
following the same
procedure,
can replicate exactly.
As one of the workshop participants noted in
a recent article,
"Although one cannot set precise standards
that evidence of
psi must meet,
judgment should be suspended until there
is at least
some
consistency among research findings from a body of methodologically
irreproachable experiments
,
at least some of which are repeatable
[by any other
researcher]." 17
J
12 Rao,
K.R.
& J.
Palmer.
The anomaly called psi:
Recent research and
criticism.
Behavioral and Brain Sciences
.
1987,
10(4)
:
539-551
(p.
544).
It should be noted that at least one critic
(Hyman) has acknowledged the
file drawer problem to be
less evident in parapsychology than in other areas
of science.
(Hyman,
R.
&
C.
Honorton,
op.
cit.
,
p.359)
14 The selective reporting of "positive" results by either nonreporting or
retrospective reporting
is prohibited by Parapsychological Association
policy.
15 Hyman,
R.
,
op.
cit.,
pp.
8-15.
16 Hyman,
R.
& C.
Honorton,
op.
cit.,
p.
361.
17 Alcock,
J.E.
Parapsychology:
Science of the anomalous or search for the
soul?
Behavioral and Brain Sciences
.
1987,
10(4):
553-565
(p.
559).
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200450001-8
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200450001-8
CRITERIA FOR ACCEPTABLE EXPERIMENTS
One challenge that parapsychologists face
as
they strive
for greater
uniformity and more cleanly defined experimental methodologies
is
the variety of
(putative) phenomena and corresponding experiment types that are being pursued
lin the psi ganzfeld experimental regime, 18
for example
• -- which focuses on ESP
®“®cts
‘ '
the experimental setup,
equipment,
and methodology are quite
different from those employed in REG experiments,
which attempt to measure
psychokinesis.
Other types of remote viewing and PK experiments have still
other setups and protocols.
Within each category,
moreover,
different
researchers use different setups,
equipment,
and methodologies from each other
and over time.
This
is
true,
of course,
in any field of research;
but the
flexibility^
1117 appUed t0 ParaP sychology militates against the customary
.
current movement in the direction of greater uniformity and cleaner
methodologies
is
thus,
in large part,
an attempt to strengthen the case for
parapsychology in defense against its critics.
In their joint communique,
Hyman
and Honorton made several recommendations
for the design and conduct of future
psi ganzfeld experiments.
In general,
they said,
such experiments should strive
o
elimination of possible means of sensory leakage;
o
better procedures and documentation for the randomization of targets;
o
more thorough documentation of judging and feedback procedures'
o
avoidance of arbitrary multiple analyses;
o
avoidance of selective reporting of statistically significant results;
o
correct use of valid statistical tests and methods;
and
o
painstakingly thorough documentation of experimental procedures and other
Information on experimenters and subjects as well as certain raw data.
Detailed suggestions were made in each of these areas. «
The authors noted that
many of the recommendations would apply equally well
to other types of
parapsychology research besides ganzfeld.
During the workshop,
Robert Jahn described several criteria or "tactics"
that he and his colleagues had found to be
"useful
in helping us
to understand
the phenomena that we are examining."
While some were general suggestions for
improving the standing of the field as a whole,
others were specific
recommendations for strengthening parapsychology experiments,
particularly in
the PK area.
The latter are:
o
o
o
employ impeccable experimental equipment and technique,
using the highest
possible precision;
work with the
largest possible data base
(to separate the marginal effects
from background noise)
;
concentrate on operator- specific,
statistical replicability;
18 Ganzfeld,
or "whole field," ESP experiments place
the receiver in an
environment of reduced external stimuli for attempted receipt of a visual
image
"transmitted" telepathically by a sender located remotely.
19 Hyman,
R.
&
C.
Honorton,
op.
cit.
,
pp.
355-361.
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200450001-8