Approved For Release 2000/08/08
: CIA-RDP96-00789R002200450001-8
o
underestimation of the effective error rate and overestimation of the
actual significance level
o
erroneous use of particular statistical procedures.
In their joint communique,
Honorton argued that there
is
"no significant
correlation between indices of study quality
[flaws]
and study outcome."
Hyman
agreed in most cases,
but insisted that there
is
a positive correlation in the
case of poor randomization,
feedback,
and inadequate documentation. 0
(This
assertion was rebutted in a later analysis.) 10
Workshop participants generally
agreed that in any event the
"flawless," perfect experiment
is very rare
in any
ield,
if it exists at all.
The point of the
issue,
critics
say,
is
that the
presence of a flaw usually implies inadequate controls for error.
The most
stringent among them believe that an apparently successful experiment cannot be
considered to have demonstrated a true anomaly
(i.e.
,
a possible psi effect)
unless
it can be shown that the experiment
is completely flawless.
The
parapsychologists counter that any such flaws should be an issue only if they
could have systematically influenced the result.
Another point made by parapsychologists
is
that,
by and large,
the critics
have done very little experimentation in parapsychology.
Therefore,
they have
little awareness of the difficulties encountered or of what
the
truly
S
^f?if
iCa?
t
Jr5
laWS might be
*
Indeed,
the parapsychologists note,
there could be
f
ltl°nal
fl-aws
that have not yet been considered and which critics could
identify through actual experimentation.
The NRC report distinguished three types of criticisms relating
to flaws
in
experiment design and conduct. 11
First
is what
is called the
"smoking gun,"
in
which observed findings are said to be definitely attributable
to
a specific
flaw or other factor
(such as deliberate fraud)
that
is shown to be present.
Second,
and more common,
is
the
"plausible alternative" allegation that a
particular flaw is present and £flujd have produced the reported results.
Third
is
the
"dirty test tube" concept,
which alleges not that a particular discovered
flaw produced the positive result but simply that the presence of flaws
in the
experiment demonstrates
a general sloppiness which brings
the results
into
question.
In other words,
if results have been obtained under conditions
that
tail
to meet "generally accepted standards"
for scientific research
(such as
clean test tubes)
,
that fact alone casts doubt on the accuracy and validity of
the results.
The authors of the NRC report claimed that,
while they could find
no instances of the "smoking gun"
or "plausible alternative,"
even the best
parapsychology experiments exemplify the
"dirty test tube" problem
(an
allegation that has been heatedly disputed by parapsychologists).
Participants agreed that the burden of proof in the first two categories
should be on the critic making the allegation,
as
it
is
in other sciences where
an experiment
is challenged on these grounds.
In the case of the
"dirty test
tube,"
the critics feel that the burden of proof should be on the experimenter.
This position brings from the parapsychologists the objection that
it
is non-
9 Hyman & Honorton,
op.
cit.
,
p.
353.
Harris,
M.J.
& Rosenthal,
R.
"Interpersonal Expectancy Effects and the
Human) Performance Research."
Washington,
DC:
National Academy Press,
1988.
11 National Research Council,
op.
cit.,
pp.
199-200.
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200450001-8