Approved For Release 2000/08/08
: CIA-RDP96-00789R003200110001-4
DARYL
J. BEM AND CHARLES HONORTON
is scored as a “hit.” Thus, if the experiment uses judging sets
containing four stimuli (the target and three decoys or control
stimuli), the hit rate expected by chance is .25. The ratings can
also be analyzed in other ways; for example, they can be con-
verted to ranks or standardized scores within each set and ana-
lyzed parametrically across sessions. And, as with the dream
studies, the similarity ratings can also be made by outside judges
using transcripts ofthe receiver’s mentation report.
Meta-Analyses of the Ganzfeld Database
In 1985 and 1986, the Journal ofParapsychology devoted two
entire issues to a critical examination of the ganzfeld database.
The 1985 issue comprised two contributions: (a) a meta-analy-
sis and critique by Ray Hyman (1985), a cognitive psychologist
and skeptical critic of parapsychological research, and (b) a
competing meta-analysis and rejoinder by Charles Honorton
(1985), a parapsychologist and major contributor to the ganz-
feld database. The 1986 issue contained four commentaries on
the Hyman-Honorton exchange, a joint communique by Hy-
man and Honorton, and six additional commentaries on the
joint communique itself. We summarize the major issues and
conclusions here.
Replication Rates
Rates by study.
Hyman’s meta-analysis covered 42 psi ganz-
feld studies reported in 34 separate reports written or published
from 1974 through 1981. One of the first problems he discov-
ered in the database was multiple analysis. As noted earlier,
it
is possible to calculate several indexes of psi performance in a
ganzfeld experiment and, furthermore, to subject those indexes
to several kinds of statistical treatment. Many investigators re-
ported multiple indexes or applied multiple
statistical
tests
without adjusting the criterion significance level for the number
of tests conducted. Worse, some may have “shopped” among
the alternatives until finding one that yielded a significantly suc-
cessful outcome. Honorton agreed that this was a problem.
Accordingly, Honorton applied a uniform test on a common
index across all studies from which the pertinent datum could
be extracted, regardless of how the investigators had analyzed
the data in the original reports. He selected the proportion of
hits as the common index because it could be calculated for the
largest subset of studies: 28 of the 42 studies. The hit rate
is
also a conservative index because it discards most of the rating
information; a second place ranking—a “near miss”—receives
no more credit than a last place ranking. Honorton then calcu-
lated the exact binomial probability and its associated z score
for each study.
Of the 28 studies, 23 (82%) had positive z scores (p = 4.6 X
10~ 4
, exact binomial test with p = q - .5). Twelve of the studies
(43%) had z scores that were independently significant at the 5%
level (p = 3.5 X 1(T?
, binomial test with 28 studies, p = .05,
and q =
.95), and 7 of the studies (25%) were independently
significant at the 1%
level (p = 9.8 X
-9
). The composite
Stouffer z score across the 28 studies was 6.60 (p = 2. 1 X
1 0“ 1
).
A more conservative estimate of significance can be obtained
by including 10 additional studies that also used the relevant
judging procedure but did not report hit rates. If these studies
are assigned a mean z score of zero, the Stouffer z across all 38
studies becomes 5.67 (p = 7.3 X 10-9
).
Thus, whether one considers only the studies for which the
relevant information is available or includes a null estimate for
the additional studies for which the information is not available,
the aggregate results cannot reasonably be attributed to chance.
And, by design, the cumulative outcome reported here cannot
be attributed
to the
inflation of significance
levels through
multiple analysis.
Rates by laboratory.
One objection to estimates such as
those just described is that studies from a common laboratory
are not independent of one another (Parker, 1978). Thus, it is
possible for one or two investigators to be disproportionately
responsible for a high replication rate, whereas other, indepen-
dent investigators are unable to obtain the effect.
The ganzfeld database
is vulnerable to this possibility. The
28 studies providing hit rate information were conducted by
investigators in 10 different laboratories. One laboratory con-
tributed 9 of the studies, Honorton’s own laboratory contrib-
uted 5, 2 other laboratories contributed 3 each, 2 contributed 2
each, and the remaining 4 laboratories each contributed
1.
Thus, half of the studies were conducted by only 2 laboratories,
1 ofthem Honorton’s own.
Accordingly, Honorton calculated a separate Stouffer z score
for each laboratory. Significantly positive outcomes were re-
ported by 6 of the 10 laboratories, and the combined z score
across laboratories was 6.16 (p =
3.6 X
10~'°). Even if all of
the studies conducted by the 2 most prolific laboratories are
discarded from the analysis, the Stouffer z across the 8 other
laboratories remains significant (z = 3.67, p = 1.2 X 10~4
). Four
of these studies are significant at the 1% level (p = 9.2 X 10-6
,
binomial test with 14 studies, p = .01, and q =
.99), and each
was contributed by a different laboratory. Thus, even though
the total number of laboratories in this database is small, most
of them have reported significant studies, and the significance
of the overall effect docs not depend on just one or two of them.
Selective Reporting
In recent years, behavioral scientists have become increas-
ingly aware of the “file-drawer” problem: the likelihood that
successful studies are more likely to be published than unsuc-
cessful studies, which arc more likely to be consigned to the file
drawers of their disappointed investigators (Bozarth & Roberts,
1972; Sterling, 1959). Parapsychologists were among the first to
become sensitive to the problem, and, in 1975, the Parapsycho-
logical Association Council adopted a policy opposing the selec-
tive reporting of positive outcomes. As a consequence, negative
findings have been routinely reported at the association’s meet-
ings and in its affiliated publications for almost two decades. As
has already been shown, more than half of the ganzfeld studies
included in the meta-analysis yielded outcomes whose signifi-
cance falls short of the conventional .05 level.
A variant ofthe selective reporting problem arises from what
1 StoufFer’s z is computed by dividing the sum of the z scores for the
individual studies by the square root of the number of studies (Rosen-
thal, 1978).
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200110001-4
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200110001-4
ANOMALOUS INFORMATION TRANSFER
vman (1985) has termed the “retrospective study.” An inves-
tor conducts a small set of exploratory trials. If they yield
ull results, they remain exploratory and never become part of
ic official record; if they yield positive results, they are defined
j a study after the fact and are submitted for publication. In
jpport of this possibility, Hyman noted that there are more
gnificant studies in the database with fewer than 20 trials than
nc would expect under the assumption that, all other things
ting equal, statistical power should increase with the square
jot of the sample size. Although Honorton questioned the as-
umption that “all other things” are in fact equal across the
:udies and disagreed with Hyman’s particular statistical analy-
s, he agreed that there is an apparent clustering of significant
udics with fewer than 20 trials. (Of the complete ganzfeld da-
ibasc of 42 studies, 8 involved fewer than 20 trials, and 6 of
lose studies reported statistically significant results.)
Because
it is impossible, by definition, to know how many
inknown studies—exploratory or otherwise—are languishing
[1 file drawers, the major tool for estimating the seriousness of
elective reporting problems has become some variant of Ro-
enthal’s file-drawer statistic, an estimate of how many unre-
eled studies with z scores ofzero would be required to exactly
anccl out the significance of the known database (Rosenthal,
979). For the 28 direct-hit ganzfeld studies alone, this estimate
; 423 fugitive studies, a ratio of unreported-to-reported studies
>f approximately 15:1. When it is recalled that a single ganzfeld
cssion takes over an hour to conduct, it is not surprising that
lespite his concern with the retrospective study problem—Hy-
nan concurred with Honorton and other participants in the
jublished debate that selective reporting cannot plausibly ac-
count for the overall statistical significance of the psi ganzfeld
database (Hyman & Honorton, 1986).
Methodological Flaws
If the most frequent criticism of parapsychology is that it has
Hot produced a replicable psi effect, the second most frequent
Criticism is that many, if not most, psi experiments have inade-
quate controls and procedural safeguards. A frequent charge is
Ihat positive results emerge primarily from initial, poorly con-
trolled studies and then vanish as better controls and safeguards
Ire introduced.
Fortunately, meta-analysis provides a vehicle for empirically
evaluating the extent to which methodological flaws may have
contributed to artifactual positive outcomes across a set ofstud-
ies. First, ratings are assigned to each study that index the degree
to which particular methodological flaws are or are not present;
these ratings are then correlated with the studies’ outcomes.
Large
positive correlations constitute evidence
that the ob-
served effect may be artifactual.
In psi research, the most fatal flaws are those that might per-
mit a subject to obtain the target information in normal sensory
fashion, either inadvertently or through deliberate cheating.
Thu is called the problem of sensory leakage. Another poten-
hafi> serious flaw is inadequate randomization of target selec-
lKJQ.
Snuory leakage.
Because the ganzfeld is itself a perceptual
isolation procedure,
it goes a long way toward eliminating po-
tential sensory leakage during the ganzfeld portion of the ses-
sion. There are, however, potential channels of sensory leakage
after the ganzfeld period. For example, if the experimenter who
interacts with the receiver knows the identity of the target, he or
she could bias the receiver’s similarity ratings in favor ofcorrect
identification. Only one study in the database contained this
flaw, a study in which subjects actually performed slightly below
chance expectation. Second, if the stimulus set given to the re-
ceiver forjudging contains the actual physical target handled by
the sender during the sending period, there might be cues (e.g.,
fingerprints, smudges, or temperature differences) that could
differentiate the target from the decoys. Moreover, the process of
transferring the stimulus materials to the receiver’s room itself
opens up other potential channels of sensory leakage. Although
contemporary ganzfeld studies have eliminated both of these
possibilities by using duplicate stimulus sets, some ofthe earlier
studies did not.
Independent analyses by Hyman and Honorton agreed that
there was no
correlation between
inadequacies of security
against sensory leakage and study outcome. Honorton further
reported that if studies that failed to use duplicate stimulus sets
were discarded from the analysis, the remaining studies are still
highly significant (Stouffer z = 4.35, p = 6.8 X 10
).
Randomization.
In many
psi
experiments,
the
issue
of
target randomization is critical because systematic patterns in
inadequately randomized target sequences might be detected by
subjects during a session or might match subjects’ preexisting
response biases. In a ganzfeld study, however, randomization is
a much less critical issue because only one target is selected dur-
ing the session and most subjects serve in only one session. The
primary concern is simply that all the stimuli within each judg-
ing set be sampled uniformly over the course of the study. Sim-
ilar considerations govern the second randomization, which
takes place after the ganzfeld period and determines the se-
quence in which the target and decoys are presented to the re-
ceiver (or external judge) forjudging.
Nevertheless, Hyman and Honorton disagreed over the find-
ings here. Hyman claimed there was a correlation between flaws
of randomization and study outcome; Honorton claimed there
was not. The sources of this disagreement were in conflicting
definitions of flaw categories, in the coding and assignment of
flaw ratings to individual studies, and in the subsequent statisti-
cal treatment of those ratings.
Unfortunately, there have been no ratings of flaws by inde-
pendent
raters who were unaware of the
studies’ outcomes
(Morris,
1991). Nevertheless, none of the contributors to the
subsequent
debate
concurred
with
Hyman’s
conclusion,
whereas four nonparapsychologists—two statisticians and two
psychologists—explicitly concurred with Honorton’s conclu-
sion (Harris & Rosenthal, 1988b; Saunders, 1985; Utts, 1991a).
For example, Harris and Rosenthal (one of the pioneers in the
use of meta-analysis in psychology) used Hyman’s own flaw rat-
ings and failed to find any significant relationships between
flaws and study outcomes in each of two separate analyses:
2 A 1980 survey of parapsychologists uncovered only 19 completed
but unreported ganzfeld studies. Seven of these had achieved signifi-
cantly positive results, a proportion (.37) very similar to the proportion
of independently significant studies in the meta-analysis (.43) (Black-
more, 1980).
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200110001-4
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200110001-4
DARYL
J. BEM AND CHARLES HONORTON
“Our analysis of the effects of flaws on study outcome lends no
support to the hypothesis that Ganzfeld research results are a
significant function ofthe set offlaw variables” ( 1 988b, p. 3; for
a more recent exchange regarding Hyman’s analysis, see Hy-
man, 1991; Utts,
1 99 1 a,
1 99 1 b).
Effect Size
Some critics of parapsychology have argued that even if cur-
rent laboratory-produced psi effects turn out to be replicable
and nonartifactual, they are too small to be of theoretical inter-
est or practical importance. We do not believe this to be the case
for the psi ganzfeld effect.
In psi ganzfeld studies, the hit rate itself provides a straight-
forward descriptive measure of effect size, but this measure can-
not be compared directly across studies because they do not all
use a four-stimulus judging set and, hence, do not all have a
chance baseline of .25. The next most obvious candidate, the
difference in each study between the hit rate observed and the
hit rate expected under the null hypothesis,
is also intuitively
descriptive but is not appropriate for statistical analysis because
not
all
differences between
proportions
that
are equal
are
equally detectable (e.g., the power to detect the difference be-
tween
.55 and
.25
is different from the power to detect the
difference between .50 and .20).
To provide a scale of equal detectability, Cohen (1988) de-
vised the effect size index h, which involves an arcsine transfor-
mation on the proportions before calculation oftheir difference.
Cohen’s h is quite general and can assess the difference between
any two proportions drawn from independent samples or be-
tween a single proportion and any specified hypothetical value.
For the 28 studies examined in the meta-analyses, h was .28,
with a 95% confidence interval from .1
1 to .45.
But because values of h do not provide an intuitively descrip-
tive scale, Rosenthal and Rubin (1989; Rosenthal, 1991) have
recently suggested a new index,
tv, which applies specifically to
one-sample, multiple-choice data of the kind obtained in ganz-
feld experiments. In particular,
tv expresses all hit rates as the
proportion of hits that would have been obtained if there had
been only two equally likely alternatives—essentially a coin flip.
Thus,
tv ranges from 0 to
1, with
.5 expected under the null
hypothesis. The formula is
..
P(k-
1)
P(k- 2) + r
where P is the raw proportion of hits and k
is the number of
alternative choices available. Because w has such a straightfor-
ward intuitive interpretation, we use it (or its conversion back
to an equivalent four-alternative hit rate) throughout this article
whenever it is applicable.
For the 28 studies examined in the meta-analyses, the mean
value of tv was .62, with a 95% confidence interval from .55 to
.69. This corresponds to a four-alternative hit rate of 35%, with
a 95% confidence interval from 28% to 43%.
Cohen (1988,
1992) has also categorized
effect sizes into
small, medium, and large, with medium denoting an efTect size
that should be apparent to the naked eye of a careful observer.
For a statistic such as tv, which indexes the deviation of a pro-
portion from .5, Cohen considers .65 to be a medium effect size:
A statistically unaided observer should be able to detect the bias
of a coin that comes up heads on 65% ofthe trials. Thus, at .62,
the psi ganzfeld effect size falls just short of Cohen’s naked-eye
criterion. From
the phenomenology of the ganzfeld experi-
, menter, the corresponding hit rate of 35% implies that he or she
will see a subject obtain a hit approximately every third session
rather than every fourth.
It is also instructive to compare the psi ganzfeld effect with
the results of a recent medical study that sought to determine
whether aspirin can prevent heart attacks (Steering Committee
of the Physicians’ Health Study Research Group,
1988). The
study was discontinued after 6 years because it was already clear
that the aspirin treatment was effective {p < .00001) and it was
considered unethical to keep the control group on placebo med-
ication. The study was widely publicized as a major medical
breakthrough. But despite its undisputed reality and practical
importance, the size of the aspirin effect is quite small: Taking
aspirin reduces the probability of suffering a heart attack by
only .008. The corresponding effect size (h ) is .068, about one
third to one fourth the size of the psi ganzfeld effect (Atkinson
etal., 1993, p.236; Utts, 1991b).
In sum, we believe that the psi ganzfeld effect is large enough
to be of both theoretical interest and potential practical impor-
tance.
Experimental Correlates ofthe Psi Ganzfeld Effect
We showed earlier that the technique of correlating variables
with effect sizes across studies can help to assess whether meth-
odological flaws might have produced artifactual positive out-
comes. The same technique can be used more affirmatively to
explore whether an effect varies systematically with conceptu-
ally relevant variations in experimental procedure. The discov-
ery of such correlates can help to establish an effect as genuine;
suggest ways of increasing replication rates and effect sizes, and
enhance the chances of moving beyond the simple demonstra-
tion of an effect to its explanation. This strategy is only heuris-
tic,
however. Any correlates discovered must be considered
quite tentative, both because they emerge from post hoc explo-
ration and because they necessarily involve comparisons across
heterogeneous studies that differ simultaneously on many inter-
related variables, known and unknown. Two such correlates
emerged from the meta-analyses of the psi ganzfeld effect
Single-versus multiple-image targets.
Although most of the
28 studies in the meta-analysis used single pictures as targets 9
(conducted by three different investigators) used View Master
stereoscopic slide reels that presented multiple images focused
on a central theme. Studies using the View Master reels pro-
duced significantly higher hit rates than did studies using the
single-image targets (50% v*. 34%). ((26) - 2.22, p = .035, two-
tailed.
Sender-receiver pairing
In
17 of the 28 studies, partici-
pants were free to bring in friends to serve as senders. In 8 stud-
ies, only laboratory-assigned senders were used. (Three studies
used no sender.) Unfortunately, there is no record ofhow many
participants in the former studies actually brought in friends
Nevertheless, those
1 7 studies (conducted by six different inves-
tigators) had significantly higher hit rates than did the studies
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200110001-4
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200110001-4
anomalous information transfer
that used only laboratory-assigned senders (44% vs. 26%), 1(23)
= 2.39, p - -025, two-tailed.
The Joint Communique
After their published exchange in 1985, Hyman and Honor-
ton agreed to contribute a joint communique to the subsequent
discussion that was published in 1986. First, they set forth their
areas of agreement and disagreement:
We agree that there is an overall significant effect in this data base
that cannot reasonably be explained by
selective reporting or
multiple analysis. We continue to differ over the degree to which
the effect constitutes evidence for psi, but we agree that the final
verdict awaits the outcome of future experiments conducted by a
broader range of investigators and according to more stringent
standards. (Hyman & Honorton, 1986, p. 351)
They then spelled out in
detail the “more stringent stan-
dards” they believed should govern future experiments. These
standards included strict security precautions against sensory
leakage, testing and documentation of randomization methods
for selecting targets and sequencing the judging pool, statistical
correction for multiple analyses, advance specification of the
status of the experiment (e.g„ pilot study or confirmatory ex-
periment), and full documentation in the published report of
the experimental procedures and the status of statistical tests
(e.g., planned or post hoc).
The National Research Council Report
In 1988, the National Research Council (NRC) of the Na-
tional Academy of Sciences released a widely publicized report
commissioned by the U.S. Army that assessed several contro-
versial technologies for enhancing human performance, includ-
ing accelerated learning, neurolinguistic programming, mental
practice,
biofeedback,
and
parapsychology
(Druckman &
Swets, 1988; summarized in Swets & Bjork, 1990). The report’s
conclusion
concerning
parapsychology
was
quite
negative:
“The Committee finds no scientific justification from research
conducted over a period of 1 30 years for the existence of para-
psychological phenomena” (Druckman & Swets, 1988, p. 22).
An extended refutation strongly protesting the committee’s
treatment of parapsychology has been
published elsewhere
(Palmer et al.,
1989). The pertinent point here is simply that
the NRC’s evaluation ofthe ganzfeld studies does not reflect an
additional, independent examination of the ganzfeld database
but is based on the same meta-analysis conducted by Hyman
that we have discussed in this article.
Hyman chaired the NRC’s Subcommittee on Parapsychol-
ogy, and, although he had concurred with Honorton 2 years ear-
lier in their joint communique that “there is an overall signifi-
cant effect in this data base that cannot reasonably be explained
by selective reporting or multiple analysis” (p. 351) and that
“significant outcomes have been produced by a number of
different investigators” (p. 352), neither of these points is ac-
knowledged in the committee’s report.
The NRC also solicited a background report from Harris and
Rosenthal (1988a), which provided the committee with a com-
parative methodological analysis of the five controversial areas
just listed. Harris and Rosenthal noted that, of these areas,
“only the Ganzfeld ESP studies [the only psi studies they evalu-
ated] regularly meet the basic requirements of sound experi-
mental design” (p. 53), and they concluded that
it would be implausible to entertain the null given the combined p
from these 28 studies. Given the various problems or flaws pointed
out by Hyman and Honorton ... we might estimate the obtained
accuracy rate to be about 1/3
.
.
. when the accuracy rate expected
under the null is 1/4. (p. 5 1)
The Autoganzfeld Studies
In 1983, Honorton and his colleagues initiated a new series
of ganzfeld studies designed to avoid the methodological prob-
lems he and others had identified in earlier studies (Honorton,
1979; Kennedy, 1979). These studies complied with all of the
detailed guidelines that he and Hyman were to publish later in
their joint communique. The program continued until Septem-
ber 1989, when a loss of funding forced the laboratory to close.
The major innovations of the new studies were computer con-
trol of the experimental protocol—hence the name autoganz-
feld—and the introduction of videotaped film clips as target
stimuli.
Method
The basic design of the autoganzfeld studies was the same as that
described earlier4
: A receiver and sender were sequestered in separate,
acoustically isolated chambers. After a 14-min period of progressive
relaxation, the receiver underwent ganzfeld stimulation while describ-
ing his or her thoughts and images aloud for 30 min. Meanwhile, the
sender concentrated on a randomly selected target. At the end of the
ganzfeld period, the receiver was shown four stimuli and, without know-
ing which of the four had been the target, rated each stimulus for its
similarity to his or her mentation during the ganzfeld.
The targets consisted of 80 still pictures (static targets) and 80 short
video segments complete with soundtracks (dynamic targets),
all
re-
corded on videocassette. The static targets included art prints, pho-
tographs, and magazine advertisements; the dynamic targets included
excerpts of approximately 1-min duration from motion pictures, TV
shows, and cartoons. The 160 targets were arranged in judging sets of
four static or four dynamic targets each, constructed to minimize simi-
larities among targets within a set.
Target selection and presentation.
The VCR containing the taped
targets was interfaced to the controlling computer, which selected the
target and controlled its repeated presentation to the sender during the
ganzfeld period, thus eliminating the need for a second experimenter to
accompany the sender. After the ganzfeld period, the computer ran-
domly sequenced the. four-clip judging set and presented
it to the re-
ceiver on a TV monitor forjudging. The receiver used a computer game
paddle to make his or her ratings on a 40-point scale that appeared on
3 In a troubling development, the chair of the NRC Committee
phoned Rosenthal and asked him to delete the parapsychology section
of the paper (R. Rosenthal, personal communication, September
15,
1992). Although Rosenthal refused to do so, that section of the Harris-
Rosenthal paper is nowhere cited in the NRC report.
4 Because Honorton and his colleagues have complied with the Hy-
man-Honorton specification that experimental reports be sufficiently
complete to permit others to reconstruct the investigator’s procedures,
readers who wish to know more detail than we provide here are likely to
find whatever they need in the archival publication of these studies in
the Journal ofParapsychology (Honorton et al., 1990).
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200110001-4