Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
373
likely
to be
selected by
the
computer’s random
number generator than any of the others in the set.
The selection of the target by the computer is the
only source
of randomness in these experiments.
This
is an important point, and one that
is often
misunderstood. (See Utts, 1991, for elucidation.)
Eighty of the targets were “dynamic,” consisting
of scenes from movies, documentaries and cartoons;
80 were
“static,”
consisting
of photographs,
art
prints and advertisements. The four targets within
each set were all of the same type. Earlier studies
indicated that dynamic targets were more likely to
produce successful results, and one of the goals of
the new experiments was to test that theory.
The randomization procedure used to select the
target and the order of presentation for judging was
thoroughly
tested
before and during the
experi-
ments. A detailed description is given by Honorton
et al. (1990, pages 118-120).
Three of the 11 series were pilot series, five were
formal series with novice receivers, and three were
formal series with experienced receivers. The last
series with experienced receivers was the only one
that did not use the 160 targets. Instead,
it used
only one set of four dynamic targets in which one
target had previously received several
first place
ranks and one had
never
received
a
first
place
rank. The receivers, none of whom had had prior
exposure to that target pack, were not aware that
only one target pack was being used. They each
contributed one session only to the series. This will
be called the “special series” in what follows.
Except
for two
of the
pilot
series, numbers
of
trials were
planned
in advance
for each
series.
Unfortunately, three of the formal series were not
yet completed when the funding ran out, including
the special series, and one pilot study with advance
planning was terminated early when the experi-
menter relocated. There were no unreported trials
during the 6-year period under review, so there was
no “file drawer.”
Overall, there were 183 Rs who contributed only
one trial and 58 who contributed more than one, for
a total of 241 participants and 355 trials. Only 23
Rs had previously participated in ganzfeld experi-
ments, and 194 Rs (81%) had never participated in
any parapsychological research.
5.2 Results
While acknowledging that no probabilistic con-
clusions can be drawn from qualitative data, Hon-
orton
et
al.
(1990)
included
several examples
of
session excerpts that Rs identified as providing the
basis for their target rating. To give a flavor for the
dream-like quality of the mentation and the amount
of informa^j^^^ji^fo^^g^g 200^/08/08
rank,
the
first example
is reproduced
here. The
target was
a
painting
by
Salvador
Dali
called
“Christ Crucified.” The
correct target received a
first place rank. The part of the mentation R used
to make this assessment read:
... I think of guides, like spirit guides, leading
me and
I come into a court with a king.
It’s
quiet .... It’s like heaven. The king
is some-
thing like Jesus. Woman. Now I’m just sort of
summersaulting
through
heaven ....
Brooding .... Aztecs,
the
Sun
God .... High
priest
.
.
. .Fear .... Graves.
Woman.
Prayer .... Funeral .... Dark.
Death .... Souls .... Ten
Commandments.
Moses .... [Honorton et al., 1990].
Over all 11 series, there were 122 direct hits in
the 355 trials, for a hit rate of 34.4% (exact bino-
mial
p-value = 0.00005) when 25% were expected
by chance. Cohen’s A is 0.20, and a 95% confidence
interval for the overall hit rate is from 0.30 to 0.39.
This calculation assumes, of course, that the proba-
bility of a direct hit
is constant and independent
across trials, an assumption that may be question-
able
except under the
null
hypothesis
of no
psi
abilities.
Honorton et al. (1990) also calculated effect sizes
for each
of the
series and each
of the
eight
experimenters. All but one of the series (the
first
novice series) had positive effect sizes, as did all of
the experimenters.
The
special
series with experienced Rs had an
exceptionally high effect size with A = 0.81, corre-
sponding to 16 direct hits out of 25 trials (64%), but
the remaining
series and the experimenters had
relatively
homogeneous
effect
sizes
given
the
amount
of variability expected by chance.
If the
special
series
is removed, the
overall
hit
rate
is
32.1%, A = 0.16. Thus, the positive effects are not
due to just one series or one experimenter.
Of the 218 trials contributed by novices, 71 were
direct
hits
(32.5%,
A - 0.17), compared with
hits in the 137 trials by those with prior ganzfeld
experience (37%, A = 0.26). The hit rates and effect
sizes were 31% (A = 0.14)
for the combined pilot
series, 32.5% (A = 0.17)
for the combined formal
novice
series, and 41.5%
( A = 0.35)
for the com-
bined experienced series. The last figure drops to
31.6%
if the
outlier
series
is removed.
Finally,
without the outlier series the hit rate for the com-
bined series where
all of the planned trials were
completed was 31.2%
( A = 0.14), while it was 35%
(A = 0.22) for the combined series that were termi-
nated
early.
Thus,
optional
stopping
cannot
: eFA^^^ef^R^OOOl 0001 -6
374
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
J. UTTS
There were two interesting comparisons that had
been
suggested
by
earlier
work
and
were
pre-
planned
in
these
experiments. The
first was
to
compare
results
for
trials with dynamic
targets
with those
for
static targets.
In the 190 dynamic
target sessions there were 77 direct hits (40%, h =
0.32) and for the static targets there were 45 hits
in
165
trials
(27%,
h = 0.05),
thus
indicating
that dynamic targets produced far more successful
results.
The second comparison of interest was whether
or not the sender was a friend of the receiver. This
was a choice the receiver could make.
If he or she
did
not
bring
a
friend,
a
lab member
acted
as
sender.
There
were
211
trials
with
friends
as
senders (some of whom were also lab staff), result-
ing in 76
direct hits (36%, h = 0.24). Four trials
used
no
sender.
The
remaining
140
trials
used
nonfriend lab
staff as senders and resulted in 46
direct hits (33%, h = 0.18). Thus, trials with friends
as senders were slightly more successful than those
without.
Consonant with the definition of replication based
on consistent effect sizes,
it
is informative to com-
pare the autoganzfeld experiments with the direct
hit studies in the previous data base. The overall
success
rates
are extremely
similar. The
overall
direct hit rate was 34.4% for the autoganzfeld stud-
ies
and was 38%
for
the
comparable
direct
hit
studies
in
the
earlier
meta-analysis.
Rosenthal’s
(1986) adjustment for flaws had placed a more con-
servative
estimate
at
33%,
very
close
to
the
observed 34.4% in the new studies.
One
limitation
of this work
is
that
the
auto-
ganzfeld studies, while conducted by eight experi-
menters, all used the same equipment in the same
laboratory.
Unfortunately,
the
level
of
fund-
ing
available
in parapsychology and the
cost
in
time and equipment to conduct proper experiments
make
it
difficult
to amass large amounts of data
across
laboratories. Another autoganzfeld
labora-
tory
is currently being constructed at the Univer-
sity
of Edinburgh
in Scotland,
so interlaboratory
comparisons may be possible in the near future.
Based on the
effect size observed to date, large
samples are needed to achieve reasonable power. If
there is a constant effect across all trials, resulting
in 33% direct hits when 25% are expected by chance,
to achieve
a
one-tailed
significance
level
of 0.05
with 95% probability would require 345 sessions.
We end this section by returning to the aspirin
and heart attack example in Section 3 and expand-
ing
a
comparison
noted by
Atkinson,
Atkinson,
Smith and Bern (1990, page 237). Computing the
equivalent
of
Cohen’s
h
for
comparing
obser-
ved heartAp^Fbvett^rtRfeteaise
1 2
5G>£i(Eto87b
<8
:
groups results in h = 0.068. Thus, the effect size
observed
in the
ganzfeld data base
is
triple
the
much publicized effect pf aspirin on heart attacks.
6. OTHER META-ANALYSES IN
PARAPSYCHOLOGY
Four
additional
meta-analyses have been
con-
ducted in various areas of parapsychology since the
original
ganzfeld
meta-analyses
were
reported.
Three of the four analyses focused on evidence of
psi
abilities, while the fourth examined the
rela-
tionship between
extroversion and
psychic
func-
tioning.
In this section, each of the four analyses
will be briefly summarized.
There
are
only
a
handful
of English-language
journals
and
proceedings
in
parapsychology,
so
retrieval
of the
relevant
studies
in
each
of the
four cases was simple ;to accomplish by searching
those
sources
in
detail
and
by
searching
other
bibliographic data basejs for keywords.
Each analysis included an overall summary, an
analysis of the quality pf the studies versus the size
of the effect and a “filie-drawer” analysis to deter-
mine the
possible number
of unreported
studies.
Three of the four also contained comparisons across
various conditions.
6.1
Forced-Choice Precognition Experiments
Honorton
and
Ferrari
(1989)
analyzed
forced-
choice experiments conducted from 1935 to 1987, in
which the target material was randomly
selected
after the subject had attempted to predict what
it
would
be. The time delay
in
selecting the target
ranged from under a isecond
to one
year. Target
material
included
iterps
as
diverse
as ESP cards
and automated randopi number
generators. Two
investigators, S. G. Sopl and Walter J. Levy, were
not included because gome of their work has been
suspected to be fraudulent.
Overall
Results. There
were
309
studies
re-
ported by 62 senior authors, including more than
50,000 subjects and nearly two million individual
trials.
Honorton and Ferrari
used
z/v'TT
as the
measure of effect size (IES) for each study, where n
was the number
of Bernoulli
trials
in the
study.
They reported a mean ES
of 0.020, and a mean
z-score of 0.65 over all studies. They also reported a
combined
z
of 11.41, p = 6.3 x 10 -25
. Some 30%
(92) of the studies weife statistically significant at
a = 0.05. The mean Ep per investigator was 0.033,
and the significant results were not due to just a
few investigators.
Quality. Eight
dichotomous
quality
measures
CIW-
,RDPSfi-6d78
SrtO03‘toCf0
s(l)bClf:l«
possible
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
375
scores from zero for the lowest quality, to eight for
the
highest. They included features such as ade-
quate randomization, preplanned analysis and au-
tomated recording
of the
results. The correlation
between study quality and
effect
size was 0.081,
indicating
a
slight
tendency
for
higher
quality
studies to be more successful, contrary to claims by
critics that the opposite would be true. There was
a
clear relationship between quality and year
of
publication,
presumably
because
over
the
years
experimenters
in parapsychology have responded
to
suggestions
from
critics
for
improving
their
methodology.
File Drawer. Following Rosenthal
(1984),
the
authors calculated the “fail-safe N” indicating the
number of unreported studies that would have to be
sitting in file drawers in order to negate the signifi-
cant effect. They found N = 14,268, or a ratio of 46
unreported studies for each one reported. They also
followed
a
suggestion by
Dawes, Landman and
Williams (1984) and computed the mean
z
for
all
studies with
z > 1.65.
If such studies were a ran-
dom sample from the upper 5%
tail
of a N(0, 1)
distribution, the mean z would be 2.06. In this case
it was 3.61. They concluded that selective reporting
could not explain these results.
Comparisons. Four
variables
were
identified
that appeared to have a systematic relationship to
study outcome. The
first was that the 25 studies
using subjects
selected on the basis
of good past
performance were more
successful than
the
223
using unselected subjects, with mean effect sizes of
0.051 and 0.008, respectively. Second, the 97 stud-
ies testing subjects individually were more success-
ful than the 105 studies that used group testing;
mean
effect
sizes were
0.021
and
0.004,
respec-
tively. Timing of feedback was the third moderat-
ing variable, but information was only available for
104
studies. The
studies that never
told the
subjects what the targets were had a mean effect
size of -0.001. Feedback after each trial produced
the best results, the mean ES
for the 47 studies
was
0.035.
Feedback
after each
set
of trials
re-
sulted
in mean ES
of 0.023
(21
studies),
while
delayed feedback
(also 21 studies) yielded a mean
ES of only 0.009. There is a clear ordering; as the
gap
between
time
of feedback
and
time
of the
actual guesses decreased, effect sizes increased.
The
fourth
variable was the time
interval
be-
tween
the
subject’s
guess and
the
actual
target
selection, available for 144 studies. The best results
were for the 31 studies that generated targets less
than a second after the guess (mean ES = 0.045),
while
the worst were
for the seven
studies that
delayed target selection by at least a month (mean
trend,
decreasing
in
order
as
the
time
interval
increased from minutes to hours to days to weeks to
months.
6.2 Attempts to Influence Random Physical
Systems
Radin and Nelson
(1989) examined
studies
de-
signed to test the hypothesis that “The statistical
output of an electronic RNG [random number gen-
erator]
is correlated with observer intention in ac-
cordance
with
prespecified
instructions”
(page
1502). These experiments typically involve RNGs
based on radioactive decay, electronic noise or pseu-
dorandom number sequences seeded with true ran-
dom sources. Usually the
subject
is instructed to
try
to influence the
results of a string
of binary
trials by mental intention alone. A typical protocol
would ask a subject to press a button (thus stai'ting
the
collection
of a fixed-length sequence
of bits),
and then
try
to
influence
the random
source
to
produce more zeroes
or more
ones. A run might
consist of three successive button presses, one each
in which
the
desired
result was more
zeroes
or
more ones, and one as a control with no conscious
intention. A
z score would then be computed for
each button press.
The 832 studies in the analysis were conducted
from 1959 to 1987 and included 235 “control” stud-
ies, in which the output of the RNGs were recorded
but
there
was
no
conscious
intention
involved.
These were usually conducted before and during
the experimental series, as tests of the RNGs.
Results. The effect size measure used was again
z/v/n, where
z was
positive
if more
bits
of the
specified type were achieved. The mean effect size
for control studies was not significantly different
from
zero
(-1.0 x 10
“ 5
).
The mean
effect
size
for the experimental studies was also very small,
3.2 x 10
“ 4
, but it was significantly higher than the
mean ES for the control studies (z = 4.1).
Quality. Sixteen quality measures were defined
and assigned to each study, under the four general
categories
of procedures,
statistics,
data and
the
RNG
device. A
score
of 16
reflected the highest
quality. The authors regressed mean effect size on
mean
quality
for each
investigator and
found
a
slope
of 2.5 x 10“ 5
with standard error
of 3.2 x
“ 5
, indicating little relationship between quality
and outcome. They also calculated a weighted mean
effect
size,
using
quality
scores
as
weights, and
found that
it was very similar to the unweighted
mean
ES.
They
concluded
that
“differences
in
methodological
quality
are
not
significant
predictors of effect size” (page 1507).
File Drawer. Radin and Nelson
used
several
ES = 0.0
.nreported
Approved For Release 2000/08/08
: CIA-RDP96-OO789R0O31OOO1OOO1-6
376
J. UTTS
studies (pages 1508-1510). Their estimates ranged
from
200
to
1000
based
on
models
assuming
that
all
significant
studies were
reported.
They
calculated the fail-safe N to be 54,000.
6.3 Attempts to Influence Dice
Radin and Ferrari (1991) examined 148 studies,
published
from
1935
to
1987,
designed
to
test
whether
or
not
consciousness
can
influence
the
results of tossing
dice. They
also found 31
“con-
trol” studies in which no conscious intention was
involved.
Results. The
effect
size
measure
used
was
z / \fn
, where z was based on the number of throws
in which the die landed with the desired face
(or
faces) up, in n throws. The weighted mean ES for
the experimental studies was 0.0122 with a stan-
dard error of 0.00062;
for the control studies the
mean and standard error were 0.00093 and 0.00255,
respectively.
Weights
for
each
study
were
de-
termined by quality, giving more weight to high-
quality studies. Combined
z scores for the exper-
imental and control studies were reported by Radin
and Ferrari to be 18.2 and 0.18, respectively.
Quality. Eleven dichotomous
quality measures
were assigned, ranging from automated recording
to whether or not control studies were interspersed
with the experimental
studies. The
final
quality
score for each study combined these with informa-
tion on method of tossing the dice, and with source
of subject (defined below). A regression of quality
score versus effect size resulted in a slope of - 0.002,
with a standard error of 0.0011. However, when
effect sizes were weighted by sample size, there was
a significant relationship between quality and
ef-
fect
size,
leading Radin and Ferrari
to conclude
that higher-quality studies produced lower weighted
effect sizes.
File
Drawer. Radin
and
Ferrari
calculated
Rosenthal’s
fail-safe. N
for
this
analysis
to
be
17,974. Using the assumption that
all significant
studies were reported, they estimated the number
of unreported studies to be 1152. As a final assess-
ment, they compared studies published before and
after
1975, when the
Journal of Parapsychology
adopted an
official
policy
of publishing nonsigni-
ficant results. They concluded, based on that an-
alysis,
that
more
nonsignificant
studies
were
published
after
1975, and thus “We must
consi-
der the overall (1935-1987) data base as suspect
with respect to the filedrawer problem.”
Comparisons. Radin
and
Ferrari
noted
that
there was bias in both the experimental and control
studies across die face. Six was the face most likely
to come up, consistent with the observation that it
has the
suits for the subset of 69 studies in which targets
were evenly balanced among the
six
faces. They
still found a significant affect, with mean and stan-
dard error for effect si^e of 8.6 x 10 -3 and 1.1 x
10 -3
, respectively. The combined
z was 7.617 for
these studies.
They also compared effect
sizes across types
of
subjects used in the studies, categorizing them as
unselected, experimenter and other subjects, exper-
imenter as sole subject,; and specially selected sub-
jects. Like Honorton and Ferrari (1989), they found
the
highest mean ES
for
studies with
selected
subjects; it was approximately 0.02, more than twice
that for unselected subjects.
6.4 Extroversion and ESP Performance
Honorton, Ferrari and Bern (1991) conducted a
meta-analysis to examine the relationship between
scores
on
tests
of
extroversion
and
scores
on
psi-related
tasks.
They;
found 60
studies
by
investigators, conducted! from 1945 to 1983.
Results. The
effect
i^ize measure used
for
this
analysis was the correlation between each subject’s
extroversion
score
and: ESP
score. A
variety
of
measures had been used; for both scores across stud-
ies, so various correlation coefficients were used.
Nonetheless, a stem and leaf diagram of the corre-
lations showed an
approximate
bell
shape
with
mean and
standard
deviation
of 0.19 and
0.26,
respectively, and with an additional outlier at
r =
0.91. Honorton et al. reported that when weighted
by degrees of freedom, the weighted mean
r was
0.14, with a 95% confidence interval covering 0.10
to 0.19.
Forced-Choice
versus
Free-Response
Re-
sults. Because forced-chbice and free-response tests
differ qualitatively, Honorton et al. chose to exam-
ine
their relationship
to extroversion
separately.
They found that for free-response studies there was
a significant correlation: between extroversion and
ESP scores, with mean
rj = 0.20 and z = 4.46. Fur-
ther,
this
effect
was
homogeneous
across
both
investigators and extroversion scales.
For forced-choice studies, there was a significant
correlation between ESPj and extroversion, but only
for
those
studies
that
reported
the ESP
results
to
the
subjects
before
j measuring
extroversion.
Honorton
et
al.
speculated that the
relationship
was
an
artifact,
in
Which
extroversion
scores
were temporarily
inflated as a
result
of positive
feedback on ESP performance.
Confirmation with pJew Data Following
the
extroversion/ESP
meta-analysis, Honorton
et
al.
attempted
to
confirm
j the
relationship
using
the
autoganzfeld
data
base.
Extroversion
scores
based on the
Myers-Biiiggs Type
Indicator were
eaA^bP^6-^78§l4po31t)(W
l!j0d91^0
had
participated in autoganzfeld studies.