Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
369
target, rather than being forced to make a choice
from a small discrete
set of possibilities. Various
types of target material have been used, including
pictures, short segments of movies on video tapes,
actual locations and small objects.
Despite the more complex target material, the
statistical methods used
to analyze these experi-
ments are similar to those for forced-choice experi-
ments. A typical experiment proceeds as follows.
Before conducting any trials, a large pool of poten-
tial targets is assembled, usually in packets of four.
Similarity of targets within a packet
is kept to a
minimum,
for reasons made
clear below. At the
start of an experimental session, after the subject is
sequestered in an isolated room, a target is selected
at random from the
pool. A sender
is placed in
another room with the target. The subject is asked
to provide a verbal or written description of what
he or she thinks is in the target, knowing only that
it is a photograph, an object, etc.
After the subject’s description has been recorded
and secured against the potential
for later
alter-
ation, a judge (who may or may not be the subject)
is given a copy of the subject’s description and the
four possible targets that were in the packet with
the
correct
target. A
properly
conducted
experi-
ment either uses video tapes or has two identical
sets of target material and uses the duplicate
set
for
this part
of the process,
to ensure that clues
such as fingerprints don’t give away the answer.
Based on the subject’s description, and of course on
a blind basis, the judge is asked to either rank the
four choices from most to least likely to have been
the target, or to select the one from the four that
seems
to best match the
subject’s description.
If
ranks are used, the statistical analysis proceeds by
summing
the
ranks
over
a
series
of
trials and
comparing the sum to what would be expected by
chance.
If the selection method
is used, a “direct
hit” occurs if the correct target is chosen, and the
number
of
direct
hits
over
a
series
of
trials
is
compared
to the number expected
in a binomial
experiment with p = 0.25.
Note that the subjects’ responses cannot be con-
sidered to be “random” in any sense, so probability
assessments are based on the random selection of
the target and decoys. In a correctly designed ex-
periment, the probability of a direct hit by chance
is 0.25 on each trial, regardless of the response, and
the trials are independent. These and other issues
related to analyzing free-response experiments are
discussed by Utts (1991).
4.2 The Psi Ganzfeld Experiments
The ganzfeld procedure
is a particular kind
of
free-respfl^rOV^nRtf R&fffaWfe 200^/08^
:
isolation technique originally developed by Gestalt
psychologists
for
other
purposes.
Evidence
from
spontaneous case
studies and experimental work
had led parapsychologists to a model proposing that
psychic functioning may be masked by sensory in-
put and by inattention to internal states (Honorton,
1977). The ganzfeld procedure was specifically de-
signed to test whether or not reduction of external
“noise” would enhance psi performance.
In these experiments, the subject
is placed in a
comfortable
reclining
chair
in
an
acoustically
shielded room. To create a mild form of sensory
deprivation, the subject wears headphones through
which
white
noise
is
played, and
stares
into
a
constant
field
of red
light.
This
is
achieved by
taping halved translucent ping-pong balls over the
eyes and then illuminating the room with red light.
In the psi ganzfeld experiments, the subject speaks
into a microphone and attempts
to
describe
the
target material being observed by the sender in a
distant room.
At the 1982 Annual Meeting of the Parapsycho-
logical Association, a debate took place over the
degree
to which
the
results
of the
psi
ganzfeld
experiments constituted evidence
of psi
abilities.
Psychologist and
critic Ray Hyman and parapsy-
chologist Charles Honorton each analyzed the re-
sults of all known psi ganzfeld experiments to date,
and they reached strikingly
different conclusions
(Honorton, 1985b; Hyman, 1985b). The debate con-
tinued with the publication of their arguments in
separate
articles in the March 1985
issue
of the
Journal of Parapsychology . Finally, in the Decem-
ber 1986 issue of the
Journal of Parapsychology,
Hyman and Honorton (1986) wrote a joint article
in which they
highlighted their agreements and
disagreements and
outlined
detailed
criteria
for
future
experiments.
That
same
issue
contained
commentaries on the debate by 10 other authors.
The data base analyzed by Hyman and Honorton
(1986) consisted of results taken from 34 reports
written by a total of 47 authors. Honorton counted
42 separate experiments described in the reports, of
which 28 reported enough information to determine
the number of direct hits achieved. Twenty three of
the studies (55%) were classified by Honorton as
having achieved statistical significance at 0.05.
4.3 The Vote-Counting Debate
Vote-counting is the term commonly used for the
technique of drawing inferences about an experi-
mental effect by counting the number of significant
versus nonsignificant studies of the effect. Hedges
and Olkin (1985) give a detailed analysis of the
inadequacy of this method, showing that it is more
as the
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
370
J. UTTS
number
of
studies
increases.
While Hyman
ac-
that the number of significant studies (using his
knowledged that “vote-counting raises many prob-
definition
of a
study) only dropped from 55%
to
lems” (Hyman, 1985b, page 8), he nonetheless spent
45%.
Next, he proposed that a uniform index
of
half of his critique of the ganzfeld studies showing
success be applied to all studies. He used the num-
why Honorton’s count of 55% was wrong.
ber
of direct
hits,
sinqe
it was by
far the most
Hyman’s first complaint was that several of the
commonly reported measure and was the measure
studies contained multiple conditions, each of which
used in the
first published psi ganzfeld study. He
should be considered as a separate study. Using
then conducted a detailed analysis of the 28 studies
this definition he counted 80 studies (thus further
reporting direct hits and found that 43% were sig-
reducing the sample sizes of the individual studies),
niflcant at 0.05 on that ^measure alone. Further, he
of which 25 (31%) were
“successful.” Honorton’s
showed that significant effects were reported by six
response to this was to invite readers to examine
of the 10 independent investigators and thus were
the studies and decide for themselves if the varying
not due to just one or tvjo investigators or laborato-
conditions constituted separate experiments.
ries. He
also noted
thq.t
success rates were very
Hyman next postulated that there was selection
similar
for reports published in refereed journals
bias, so that significant studies were more likely to
and those published in ijinrefereed monographs and
be reported. He raised some important issues about
abstracts.
how
pilot studies may be terminated and not
re-
While Hyman’s arguments identified issues such
ported if they don’t show significant results, or may
as
selective reporting and optional stopping that
at
least be subject
to optional stopping, allowing
should be considered in any meta-analysis, the de-
the experimenter to determine the number of tri-
pendence of significance levels on sample size makes
als. He
also presented a chi-square analysis that
the vote-counting technique almost useless for as-
“suggests a tendency to report studies with a small
sessing the magnitude jof the
effect. Consider,
for
sample
only
if
they
have
significant
results”
example, the 24 studies where the direct hit meas-
(Hyman, 1985b, page 14), but I have questioned his
ure was reported and the chance probability of a
analysis elsewhere (Utts, 1986, page 397).
direct hit was 0.25, the most common type of study
Honorton refuted Hyman’s argument with four
in the data base. (There were four direct hit studies
rejoinders (Honorton, 1985b, page 66). In addition
with other chance probabilities and 14 that did not
to
reinterpreting
Hyman’s
chi-square
analysis,
report direct hits.) Of thje 24 studies, 13 (54%) were
Honorton pointed out that the
Parapsychological
“nonsignificant” at a
=j 0.05, one-tailed. But if the
Association has an
official policy encouraging the
367
trials
in these
“failed replications” are
corn-
publication of nonsignificant results in its journals
bined, there are 106 direct hits,
z = 1.66, and p =
and proceedings, that a large number of reported
0.0485,
one
tailed.
This
is
reminiscent
of
the
ganzfeld studies did not achieve statistical signifi-
dilemma of Professor B :in Section 3.
cance and that there would have to be 15 studies in
Power is typically very low for these studies. The
the “file-drawer” for every one reported to cancel
median sample size for the studies reporting direct
out the observed significant results.
hits was 28. If there is a real effect and it increases
The remainder of Hyman’s vote-counting analy-
the
success
probability from
the chance
0.25
to
sis consisted of showing that the effective error rate
an
actual
0.33
(a
valiie whose
rationale
will be
for each study was actually much higher than the
made clear below), the: power for a study with 28
nominal 5%. For example, each study could have
trials
is only 0.181
(Utts,
1986).
It should be no
been analyzed using the
direct
hit measure, the
surprise that there is
a: “repeatability” problem in
sum of ranks measure or one of two other measures
used for free-response analyses. Hyman carried out
a simulation study that showed the true error rate
would be 0.22
if “significance” was defined by re-
quiring
at
least
one
of these
four
measures
to
achieve the 0.05 level. He suggested several other
ways
in which
multiple
testing
could
occur and
concluded that the effective error rate in each ex-
periment was not the nominal 0.05, but rather was
probably close to the 31% he had determined to be
the actual success rate in his vote-count.
Honorton acknowledged that there was a multi
parapsychology.
j
4.4 Flaw Analysis and (future Recommendations
The second half of Homan’s paper consisted of a
“Meta-Analysis of Flawq and Successful Outcomes”
(1985b, page
30), designed to explore whether
or
not various measures
of success were
related
to
specific flaws in the experiments. While many crit-
ics have argued that the results in parapsychology
can be explained by experimental flaws, Hyman’s
analysis was the
first to attempt
to quantify the
relationship between flaws and significant results.
pie testing problem, but he had a two-fold response.
Hyman
identified
lj2
potential
flaws
in
the
anzfjeId experiments, such as inadequate random
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
ization, multiple tests used without adjusting the
significance
level
(thus
inflating the
significance
level from the nominal 5%) and failure
to use a
duplicate set of targets for the judging process (thus
allowing possible clues such as fingerprints). Using
cluster and
factor
analyses,
the
binary
flaw
variables were combined into three new variables,
which Hyman named General Security, Statistics
and Controls.
Several analyses were then conducted. The one
reported with the most detail
is a factor analysis
utilizing 17 variables for each of 36 studies. Four
factors emerged from
the
analysis. From
these,
Hyman concluded that security had increased over
the years, that the significance level tended to be
inflated the most for the most complex studies and
that both effect size and level of significance were
correlated with the existence of flaws.
Following his factor analysis, Hyman picked the
three flaws that seemed
to be most highly corre-
lated with success, which were inadequate atten-
tion to both randomization and documentation and
the potential for ordinary communication between
the sender and receiver. A regression equation was
then computed using each
of the
three
flaws
as
dummy variables, and the effect size for the experi-
ment as the dependent variable. From this equa-
tion, Hyman concluded that a study without these
three flaws would be predicted to have a hit rate of
27%. He concluded that
this
is “well within the
statistical neighborhood
of the 25% chance rate”
(1985b, page 37), and thus “the ganzfeld psi data
base, despite initial impressions,
is inadequate
ei-
ther to support the contention of a repeatable study
or to demonstrate the reality of psi” (page 38).
Honorton discounted both Hyman’s flaw classifi-
cation and his analysis. He did not deny that flaws
existed, but he objected that Hyman’s analysis was
faulty and impossible to interpret. Honorton asked
psychometrician David Saunders
to write an Ap-
pendix to his article, evaluating Hyman’s analysis.
Saunders
first
criticized Hyman’s use
of a factor
analysis with
variables (many
of which were
dichotomous) and only 36 cases and concluded that
“the
entire
analysis
is
meaningless”
(Saunders,
1985, page 87). He then noted that Hyman’s choice
of the three flaws to include in his regression anal-
ysis constituted a clear case
of multiple analysis,
since there were 84 possible sets of three that could
have been selected (out of nine potential flaws), and
Hyman chose the set most highly correlated with
effect
size.
Again, Saunders concluded that “any
interpretation drawn from [the regression analysis]
must be regarded as meaningless” (1985, page 88).
Hyman’s results were also contradicted by Harris
and Rosenthal (1988b) in an analysis requested by
Approved For Release 2000/08/08
:
371
Hyman
in
his capacity as Chair
of the National
Academy
of Sciences’ Subcommittee on
Parapsy-
chology. Using Hyman’s flaw classifications and a
multivariate
analysis, Harris and Rosenthal
con-
cluded that “Our analysis of the effects of flaws on
study outcome lends no support to the hypothesis
that
ganzfeld
research
results
are
a
significant
function
of
the
set
of
flaw
variables”
(1988b,
page 3).
Hyman
and Honorton
were
in
the
process
of
preparing papers for a second round of debate when
they were
invited
to lunch together
at the
1986
Meeting of the Parapsychological Association. They
discovered that they were in general agreement on
several major issues, and they decided to coauthor
a
“Joint Communique” (Hyman
and
Honorton,
1986).
It
is clear from their paper that they both
thought
it was more important to set the stage for
future experimentation than to continue the techni-
cal arguments over the current data base. In the
abstract to their paper, they wrote:
We agree that there
is an overall
significant
effect in this data base that cannot reasonably
be explained by selective reporting or multiple
analysis. We continue to differ over the degree
to which the effect constitutes evidence for psi,
but we agree that the final verdict awaits the
outcome of future experiments conducted by a
broader range of investigators and according to
more stringent standards [page 351],
The paper then outlined what these standards
should be. They included controls against any kind
of sensory leakage, thorough testing and documen-
tation
of randomization methods
used,
better
re-
porting of judging and feedback protocols,
control
for multiple analyses and advance specification of
number of trials and type
of experiment.
Indeed,
any
area
of research
could
benefit from
such
a
careful list of procedural recommendations.
4.5
Rosenthal’s Meta-Analysis
The same issue of the Journal of Parapsychology
in which the Joint Communique appeared also car-
ried commentaries on the debate by
separate
authors.
In
his commentary,
psychologist Robert
Rosenthal, one of the pioneers of meta-analysis in
psychology,
summarized
the
aspects
of Hyman’s
and Honorton’s work that would
typically be
in-
cluded
in a meta-analysis (Rosenthal,
1986).
It
is
worth reviewing Rosenthal’s results
so
that they
can be used as a basis of comparison for the more
recent psi ganzfeld studies reported in Section 5.
Rosenthal,
like Hyman and Honorton,
focused
only on the 28 studies
for which
direct hits were
known. He
chose
to
use
an
effect
size measure
CIA-RDP96-00789R0031 0001 0001 -6
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
372
J. UTTS
called Cohen’s
h, which
is the difference between
the
arcsin transformed proportions
of direct
hits
that were observed and expected:
h = 2 (arcsin yfp - arcsin \fp).
One advantage of this measure over the difference
in raw proportions is that it can be used to compare
experiments with different chance hit rates.
If the
observed and
expected numbers
of hits
were identical, the effect size would be zero. Of the
28 studies, 23 (82%) had effect sizes greater than
zero, with a median effect size of 0.32 and a mean
of 0.28. These correspond to direct hit rates of 0.40
and
0.38
respectively, when
0.25
is expected by
chance. A 95%
confidence
interval
for the
true
effect
size
is from 0.11
to 0.45, corresponding
to
direct hit rates of from 0.30 to 0.46 when chance is
0.25.
A common technique in meta-analysis is to calcu-
late a “combined
z ,” found by summing the indi-
vidual
z scores and dividing by the square root of
the number of studies. The result should have a
standard normal distribution if each
z score has a
standard
normal
distribution.
For
the
ganzfeld
studies, Rosenthal reported a combined
z of 6.60
with a p-value of 3.37 X 10 _n
. He also reiterated
Honorton’s
file-drawer assessment by
calculating
that there would have to be 423 studies unreported
to negate the significant effect in the 28 direct hit
studies.
Finally, Rosenthal acknowledged that, because of
the flaws in the data base and the potential for at
least a small
file-drawer
effect, the true average
effect size was probably closer to 0.18 than 0.28. He
concluded, “Thus, when the accuracy rate expected
under the null
is
1/4, we might estimate the ob-
tained accuracy rate to be about 1/3” (1986, page
333). This
is the value used for the earlier power
calculation.
It
is worth mentioning that Rosenthal was com-
missioned by the National Academy of Sciences to
prepare a background paper to accompany its 1988
report on parapsychology. That paper (Harris and
Rosenthal,
1988a)
contained much
of
the
same
analysis
as
his commentary summarized
above.
Ironically, the discussion of the ganzfeld work
in
the National Academy Report focused on Hyman’s
1985 analysis, but never mentioned the work it had
commissioned Rosenthal to perform, which contra-
dicted the final conclusion in the report.
5. A META-ANALYSIS OF RECENT GANZFELD
EXPERIMENTS
After
the
initial
exchange
with
Hyman
at
the
1982
Parapsychological
Association Meeting,
Approved For Release 2000/08/08
Honorton and
his
colleagues developed an
auto-
mated ganzfeld experiment that was designed
to
eliminate
the
methodological
flaws
identified by
Hyman. The execution and reporting of the experi-
ments followed the detailed guidelines agreed upon
by Hyman and Honorton.
Using this “autoganzfeld” experiment, 11 experi-
mental
series
were
conducted
by
eight
experi-
menters between February
1983
and September
1989, when the equipment had to be dismantled
due to lack of funding. In this section, the results
of these
experiments
are summarized and
com-
pared to the earlier ganzfeld studies. Much of the
information is derived from Honorton et al. (1990).
5.1 The Automated Ganjzfeld Procedure
Like earlier ganzfeld studies, the “autoganzfeld”
experiments require four participants. The first
is
the Receiver (R), who attempts to identify the tar-
get material being observed by the Sender (S). The
Experimenter (E) prepares R for the task,
elicits
the response from R and supervises R’s judging of
the
response
against
the
four
potential
targets.
(Judging is double blind; E does not know which is
the correct target.) The fourth participant is the lab
assistant (LA) whose ohly task
is to instruct the
computer
to randomly
: select
the
target. No one
involved in the experiment knows the identity of
the target.
Both R and S are sequestered in sound-isolated,
electrically
shielded ropms. R
is
prepared
as
in
earlier ganzfeld
studies, with white
noise and a
field of red light. In a nonadjacent room, S watches
the target material on a television and can hear R’s
target
description
(“mentation”)
as
it
is
being
given. The mentation is; also tape recorded.
The judging process takes place immediately af-
ter the 30-minute sending period. On a TV monitor
in the isolated room, R yiews the four choices from
the target pack that contains the actual target. R is
asked to rate each one According to how closely
it
matches the ganzfeld mentation. The ratings are
converted
to ranks
and,
if the
correct
target
is
ranked first, a direct hit: is scored. The entire proc-
ess is automatically recorded by the computer. The
computer then displays the correct choice to R as
feedback.
There were
160
preselected
targets, used with
replacement,
in
of the
series. They were
arranged in packets of four, and the decoys for a
given target were always the remaining three
in
the same set. Thus, even if a particular target in a
set were consistently favored by Rs, the probability
of a
direct
hit under the
null hypothesis would
remain at 1/4. Popular targets should be no more
CIA-RDP96-OO789R0O31 0001 0001 -6