Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
experiment or an effect may be considered replicated
if a series
of replication
attempts provides statistically significant evidence
for
the original effect
when analyzed as
a series."-^
It
is
also contended that this approach allows
the analyst to assess methodological flaws
to determine their effect empirically
across
the data base.
Using the meta-analytic approach,
parapsychologists have calculated an
overall
"success”
rate,
in achieving statistically significant effects
(p <.05),
of 45%
for ganzfeld ESP studies and 21%
for random event generator
(REG)
studies.
However,
critics have objected strongly
to these
findings,
saying that
many
of the studies
included are not directly comparable due
to differences
in
methodology.
They also claim that many unsuccessful experiments have not been
published and are therefore not
in the data base.
This
"file drawer problem",
they
assert,
artificially increases
the success ratio and the overall effect
size.
-*-3
They also cite
a corresponding "retrospective study" problem,
in which
small exploratory studies
are included in the data base after the
fact
if their
results
turn out positive. ^
Furthermore,
they believe,
the proliferation of
flaws
in methodology simply perpetuates
the same erroneous
results. 1^
The debate over replicability,
particularly
in the context of
meta- analysis
,
has been heated and continues
to be
so.
However,
the nature
of
the controversy
is currently being outlined
in detail and areas of agreement are
being identified.
For example,
the recent joint communique by two of
the
workshop participants refers
to
"the growing role we believe meta-analysis will
play, both
in the evaluation of research quality and in the assessment of
moderating variables.
We urge parapsychological investigators
to plan and
report their experiments with
the
idea that their single experiment will
contribute
to
a future meta-analysis
This
idea relies
in part on guidelines
for greater uniformity in experiment
design and conduct which
the
two authors specified in their article.
Such
uniformity or consistency
is viewed as crucial by many moderate critics,
who
point
to
the fact
that
there has never been
a successful parapsychology
experiment which any and all competent researchers,
following the
same
procedure,
can replicate exactly.
As
one of the workshop participants noted
in
a recent article,
"Although one cannot set precise standards
that evidence
of
psi must meet,
judgment should be suspended until there
is
at least some
consistency among research findings from a body of methodologically
irreproachable experiments,
at least some of which are repeatable
[by any other
researcher]
.
Rao,
K.R.
& J.
Palmer.
The anomaly called psi:
Recent research and
criticism.
Behavioral and Brain Sciences
.
1987,
10 (4)
:
539-551
(p.
544).
It should be noted that
at least one critic
(Hyman)
has acknowledged the
file drawer problem
to be
less evident
in parapsychology than
in other areas
of science.
(Hyman,
R.
&
C.
Honorton,
op.
cit.,
p.359)
14 The selective reporting of
"positive"
results by either nonreporting or
retrospective reporting
is prohibited by Parapsychological Association
policy.
15 Hyman,
R.
,
op.
cit.,
pp.
8-15.
16 Hyman,
R.
&
C.
Honorton,
op.
cit.,
p.
361.
17 Alcock,
J.E.
Parapsychology:
Science of the anomalous
or search for
the
soul?
Behavioral and Brain Sciences
.
1987,
10(4):
553-565
(p.
559).
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
CRITERIA FOR ACCEPTABLE EXPERIMENTS
One challenge that parapsychologists
face
as
they strive
for greater
uniformity and more cleanly defined experimental methodologies
is
the variety of
(putative)
phenomena and corresponding experiment types
that are being pursued.
In the psi ganzfeld experimental regime,
for example
-- which focuses on ESP
effects
--
the experimental
setup,
equipment,
and methodology are quite
different from those employed
in REG experiments,
which attempt
to measure
psychokinesis.
Other types
of remote viewing and PK experiments have
still
other setups and protocols.
Within each category,
moreover,
different
researchers use different setups,
equipment,
and methodologies from each other
and over
time.
This
is
true,
of course,
in any field of research; but
the
critical scrutiny applied to parapsychology militates against
the customary
flexibility.
The current movement
in the direction of greater uniformity and cleaner
methodologies
is
thus,
in large part,
an attempt
to strengthen the case
for
parapsychology in defense against
its critics.
In their joint communique,
Hyman
and Honorton made several recommendations
for the design and conduct of future
psi ganzfeld experiments.
In general,
they said,
such experiments should strive
for
:
o
elimination of possible means of sensory leakage;
o
better procedures and documentation for
the randomization of targets
;
o
more thorough documentation of judging and feedback procedures;
o
avoidance of arbitrary multiple analyses;
o
avoidance of selective reporting of statistically significant results;
o
correct use of valid statistical
tests
and methods;
and
o
painstakingly thorough documentation of experimental procedures
and other
information on experimenters and subjects
as well
as certain raw data.
Detailed suggestions were made
in each of these areas
.
The authors noted that
many of the recommendations would apply equally well
to other types of
parapsychology research besides ganzfeld.
During the workshop,
Robert Jahn described several criteria or "tactics"
that he and his colleagues had found to be
"useful
in helping us
to understand
the phenomena that we are examining."
While some were
general suggestions
for
improving the standing of the field as
a whole,
others were specific
recommendations
for strengthening parapsychology experiments,
particularly
in
the PK area.
The latter
are:
o
employ impeccable experimental equipment and technique,
using
the highest
possible precision;
o
work with the
largest possible data base
(to
separate
the marginal effects
from background noise)
;
o
concentrate on operator-specific,
statistical replicability;
18 Ganzfeld,
or
"whole
field,"
ESP experiments place
the receiver
in an
environment of reduced external stimuli
for attempted receipt of a visual
image
"transmitted"
telepathically by a sender located remotely.
19 Hyman,
R.
&
C.
Honorton,
op.
cit.
,
pp.
355-361.
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
o
use
a "tripolar protocol,"
i.e.,
collect data under three interspersed
conditions
(e.g.,
intention to raise the number of events,
intention to
lower the number of events,
and baseline);
and
o
within the experimental program, maintain equal respect for
the
aesthetic/impressionistic and rigorous/analytical aspects of the research on
this unusual
topic.
One additional point that was made repeatedly in the workshop regarded the
necessity of distinguishing between exploratory and confirmatory
(proof-
oriented)
experiments.
The former are generally shorter,
more broadly defined,
and not reported.
There was general agreement that
the explorational objectives
should be clear and the temptation to report positive results as
"proof"
of a
hypothesis should be resisted.
Instead,
exploratory studies should be used to
suggest and develop hypotheses
for later proof- oriented experiments which should
be highly defined and constrained.
Exploratory studies can also be used to
develop and refine experimental, procedures.
III.
ANALYSIS OF DATA
As was noted earlier,
the scientific evidence
for psi consists chiefly of
statistically significant deviations from a theoretical chance expectation or
experiential baseline.
Consequently,
the validity of statistical analyses
employed
is crucial
to the strength and acceptability of parapsychological
claims
.
According
to Jessica Utts,
a statistician participating in the OTA
workshop,
parapsychologists
in general tend to be more careful with statistical
methodology than researchers in other fields.
They are generally more
responsive
to criticism on this score than other researchers
are,
she
said.
There
is general understanding among them,
Utts
said,
that human responses
cannot be assumed to be statistically random.
It
is also well understood that
selection of targets with replacement
is preferable because
it allows
independent trials.
She also referred to
the formal policy of parapsychology
for reporting negative results.
However,
it was noted,
not all statistical
analysis by parapsychologists
(or by their critics)
is of high caliber.
Much of the discussion centered around a single broad topic:
how to
inject
the maximum robustness and power
into
the empirical data base.
It was
recommended that parapsychologists should confine themselves
to methodologies
that yield such robust data that
the outcome does not depend largely on subtle
statistical points.
There was general agreement on the various suggestions
that
were made
to this
end.
For example,
there was considerable discussion of the
traditional notion
that a "successful"
experiment has
a significance level
(p value)
of
.05
or
less
(meaning that there
is no more than a
5 percent chance
that the results are due
to chance).
Partly as
a result of this concept,
Utts noted,
nonsignificant
results
(that
is,
where p >.05)
are
too often interpreted erroneously by critics
as insignificant results.
An example was
given to demonstrate that sample
size
(n)
has a substantial
impact on the
statistical outcome;
if n
is
small
there
is
very little chance for a "successful"
outcome,
even if relatively high actual
probabilities
of success based on ESP are assumed.
Meta-analysis of all
the
forced-choice precognition experiments conducted from the
1930s on showed that
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
those with significant results had,
on average,
two and one-half times larger
sample sizes than those with nonsignificant results.
Thus,
large va
H
impart considerably greater "power"
to
the analysis.
Power
is,
essentially,
the
probability that the experiment
is going to succeed.
(This concept of power
is
now being Light at the graduate level in statistics.)
Table
illustrates how
power increases with increasing q.
At least one of the critics present argued that there
is no correlation
between sample
size and the probability of obtaining a significant outcome,
but
chat claim.
It was agreed that the spread of data -
for example,
one or
two "outliers"
(data points
far outside
the general cluster)
--
can
greatly weaken the correlation.
The statistician recommended that parapsychologists
stop focusing on the
arbitrary,
"sacrosanct" E <.05 and begin focusing,
instead,
on more meaningful
^es^uch as estimating the magnitude of the observed effect
looking for
consistency,
and studying patterns across many studies.
Meta-analysis
is
a
y
to obtain Lch larger sample sizes and "hits
In an analysis of ganzfeld psl
studies with hit probability of
.25 but which were nonsignificant
(and thus
"failures"),
combining the studies gave a p of
.02
( success
).
This
recommendation met with general agreement.
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
TABLE
"POWER” AS A FUNCTION. OF SAMPLE SIZE
FOR DIRECT HIT GANZFELD ANALYSIS
WITH CHANCE p-.25
Power
True
d
n-10
n—100
.30
.047
.29
.33
.073
.54
.40
.166
.94
.50
.377
.9998
(
-
.02)
(
.04)
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
It is well established that some
individuals achieve greater effects than
others
in psi experiments.
Figure
1 shows
the distribution of results across
operators
(subjects)
in a series of REG runs. 20
Given this correlation of
individual operator identities and characteristics with some of the alleged psi
phenomena,
it
is also
important
to understand how individual operator effects
are related to collective effects.
The point was made
that,
in seeking larger
data bases and greater
(collective)
significance levels,
researchers should not
lose
sight of individual effects by combining many operators.
Scientifically,
the individual and group effects pose
two distinct sets of questions.
Effect size
is another crucial
issue
in parapsychology.
From the
standpoint of practical significance,
the effect appears
to be quite
small.
In
the largest REG data base of over 750,000 trials
(150 million bits),
compiled at
the Princeton Engineering Anomalies Research Laboratory over the past decade,
the overall means were shifted in the
intended direction by only 0.035%.
Figure
2 plots graphically the
total cumulative deviations from baseline of all
operators
in the program (accounting for 250,000 trials
in each direction)
. 21
It can be seen that,
while
the deviation
is well beyond the
.05 significance
level,
it
is not an enormous effect
-- although
it
is said to be beyond the
tolerance
level of many contemporary microelectronics processors,
communications
discriminators
,
and dedicated computational arrays
Nevertheless,
parapsychologists point out that psi effects,
if indeed that
is what they are,
could be amplified by information theory techniques such as
"majority vote counting."
It
is said that
the effect size
in the best ganzfeld
psi research
is approximately equal
to
that found in the recent studies of
aspirin in heart attack prevention.
And small effect sizes
in physical
phenomena
(e.g.,
the bending of light by gravity) have been very important
in
the evolution of human knowledge
in the past.
Indeed,
the evolution of life
itself depends on genetic mutations which occur at very low rates.
In other
words,,
significance
(in the
"real world"
sense)
and effect size are
two separate
considerations.
It
is not the magnitude of an event,
but
its centrality and
relevance
to established beliefs
or processes,
that determines
its
importance.
Effect size means different things
in the context of different types of
experiments.
The average effect sizes found in psi ganzfeld experiments
(about
.17
in one series of experiments described at
the workshop)
are several times
larger than those found in the REG work.
The anomalies being displayed
(if
indeed they are
true anomalies) may well be
of fundamentally different
types.
The primary value of using effect sizes appears
to be
in compensating for
differences in sample size.
The key to satisfying proponents and critics of parapsychology may,
in the
end,
be replicability.
If a "recipe"
for replication of successful
parapsychology experiments could be developed,
it would render moot many of the
issues of statistical analysis.
However,
failure
to replicate does not disprove
the existence of psi;
and the criteria for replicability can be difficult
to
Durine,
B.J.,
et
al.
Individual operator contributions
in large data base
anomalies experiments
(Technical Note PEAR 88002).
Research Report.
Princeton Engineering Anomalies Research, July 1988.
21 Jahn,
R.G.
& B.J.
Dunne,
op.
cit.,
p.
110.
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
OVERALL
WEAK
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
100.3
100.3
100 .
:x
too
!g°
88.8
90.6
tO. 7
( a
)
HIGH
INTENTION
GRAND
.r.
.
4- + +
-
.
« a 84
+ 66*°
. +03
+ + + + - . 87
+ + +
+ 53 04 82
.
+60
+ +35
, + + +
81 59
+ &41 «
EXPECTED
WEAN
00 70
+ +
33 48
100. 3r
( b
)
BASELINE
100 . 21*
100. l\
100 h
°
88.81-
66 60
GRAND
WEAN
+ + +
54®« ea
+ +55 10 4t 62
+ + + --3U
+ + 20 53 53 *2 «
W
+
4- + *97 55 59
^ + + 29 49 70 M
19 33
—
EXPECTED WEAN
36 14
+
89. 6 1- Q0 83
88.
l
100.3
100.2
100. 1
100
89.0
88.6
88 .
( c
)
LOW
INTENTION
+ 83
EXPECTED
WEAN
+ r.w
, + + + m 20M«»
. + + + + + + + «S TO
+ +
-,-.97 9030
+ 44 70 59 33®*
+ + 10
SO 14
+ + + + K 3«49
M « 19 19~
+ + +
66 86 82
GRAND
WEAN
+ +
16 64
Z
NUMBER Of
OPERATORS
A&fiUlS Fir Reafl9fS^bSo«E6iA4S6TOi^F«Eft»a8oWNS
soh»^>h<wd
w
<
c
u
M
U
L
A
T
I
10000
5000
’5000
(BITS)
10000
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
50000
100000
150000
200000
250000
NUMBER
OF
TRIALS
FIGURE
2:
CUMULATIVE
REG
RESULTS,
ALL OPERATORS
(33),
ALL SERIES
(87)
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
.01
. 05
. 05
. 01
T
E
R
M
I
N-
A
L
P'
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
establish.
What
is needed is consensus among the various camps on the ground
rules
for replicability.
For example,
individuals must rigorously apply the
same criteria to themselves that they do to others,
and apply
to others
the
same
criteria that they themselves use elsewhere.
One approach that might move parapsychology out of the quagmire of
statistical significance levels could be to control for desired conditions
--
i.e.
for large effect size
-•
stripping away
in advance
the
"noise
injected by
those subjects with low capability.
It may be that replicability will hinge on
getting large effect sizes.
But even replicability in operators with large
effect sizes
is often problematical across time and changing experimental
conditions
In the end,
some parapsychologists believe,
it may be that replicability
will follow from the development of theory in parapsychology.
In this view,
the
priority should be
to pursue process -oriented research that can elucidate
the
mechanisms controlling these anomalies.
It will then be possible
to design
experiments and conditions
to produce the desired effects,
rather than relying
on experiments designed to produce statistical significance and satisfy a
critical audience.
(Critics,
if they are given the last word,
retort that
parapsychologists should not bother presenting their case
to
the scientific
establishment until
they have succeeded in developing an adequate
theory.)
IV.
THE MEANING OF EXPERIMENTAL RESULTS
ASSESSMENTS OF MEANING AND MEANINGFULNESS
Leading off the "Findings and Conclusions"
on parapsychology
in the recent
NRC report was a provocative assertion:
"The Committee
finds no scientific justification from research conducted over
a
period of 130 years for
the existence of parapsychological phenomena.
Although the treatment of parapsychology in the body of the report was genera
ly
more moderate
in tone,
this finding was taken by the media and many readers
to
be
the
"bottom line"
of the report with respect
to parapsychology.
It produced
considerable distress and resentment among parapsychologists and others,
who
felt that the field had been dealt an unwarranted blow.
However,
the statement did clearly delineate
the view of many of those who
monitor the
field.
Some critics are willing to concede that parapsyc
o ogy
research has demonstrated definite anomalies,
but they believe that
t ese
reflect statistical
or experimental artifacts,
not psi.
In their view,
t
e
22 National Research Council,
op.
cit.,
p.
22.
<5ee
for example:
Palmer,
J.A.,
C.
Honorton,
& J.
Utts
,
op.
cit.
24 This statement was based on a previously published assertion by
ay
yman,
nV, airman of the NRC subcommittee which authored the report
(Proc_.
,
nne
1986
74(6):
823-849).
At
the workshop,
Hyman said that the precise wor
ing
of
the""NRC's statement was not his.
Discussion ensued over the semant
c
jj ff|ir pnces between Hyman's
statement and the NRC report statement.
yman
^™he would have phrased the statement differently.
He later referred
.
differences between the wording he would have used and the wor mg
n
the report as
a matter of style,
not substance
(personal communication with
Dr.
Alan Shaw,
28 December
1988).
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2
anomalies are jas£ anomalies
-- mere
"puzzles"
--
and nothing more.
The more
conservative parapsychologists tend to agree that nothing more than anomalies
have been demonstrated; but they point out that the anomalies are structured
statistically in very interesting ways,
suggesting that something more
significant
is at work.
Critics,
in turn,
counter that
if a
successful
experiment has been cleanly run and the results are well above chance
then most
scientists would agree that something truly puzzling has been manifested.
That
is why methodology and replicability are so important for parapsychology
experiments
.
But! they conclude,
it
is not a puzzle that has high priority when
viewed against the other questions facing scientists.
They assume that the
explanation for the puzzle,
when found,
will not be very interesting after all.
One reason
(of several)
for the critics’
skepticism
is
the "experimenter
effect," which
is seen in the fact that certain investigators obtain
consistently positive experimental results while others
get only chance results.
So pervasive
is this phenomenon that
it has been called
the only con
rme
result of parapsychology." 25
The experimenter effect lends
force
to criticisms
of parapsychology,
seeming as
it does almost to imply fraud--
or at
®^st a
lack of objectivity.
Parapsychologists point out that the identity of the
experimenter
is also positively correlated with the placebo effect in clinical
medicine.
It may be,
they suggest,
a function of belief in the phenomenon under
investigation.
According to this view,
some of those who believe m psi are
somehow able
to elicit the phenomenon in their subjects.
But,
if true,
this
presents
a problem for
the experiments,
since
the
investigator can influence the
experiment subjectively and perhaps involuntarily.
In other words
it might
demonstrate the existence of psi while making
it even more difficult
to prove
scientifically
.
Stringent requirements for proof are placed on parapsychology.
The editor
of Science has stated that "extraordinary claims require extraordinary proof.
Some parapsychologists object that the burdens of proof in this field are
unique,
and unfairly
so.
In their view,
the critics of the field have an
arbitrarily high threshold for what constitutes scientifically acceptable
evidence,
and everything falling below that threshold
is dismissed as
insignificant.
Indeed,
the NRC committee recognized only two successful
experiments over
15 years
in all of parapsychology
--
one in remote viewing
and one using the random number generator 2®
--
as meeting
most of the minimal
25 Parker
A
A pilot study of the influence of experimenter expectancy on
ESP scores
in J.D.
Morris et al.
(Eds.) Researcb
In PaEdpsycholoKY 197.4 .
Z^Abelson ,
!
H^3
"A°s tepchild ' of science starts
to win friends."
U,S,
News
fr
zTschlltrl iYMc.
Remote viewing revisited:
An Intrasubject
replication
Journal of Parapsychology.
,
1984,
48-
•iy ' 4y
-
25Say
E.C.,
'B. S.
Humphrey.
&Vs. HuSard.
Electronic System Perturbation
Techniques
(Final Report).
Menlo Park,
CA:
SRI Internationa
,
Approved For Release 2003/09/09
: CIA-RDP96-00792R000100130001-2