^ppfigftfgfy^r Release 2000/08/10
: CIA-RDP96-00792R0001 001 30003-0 4
reliability of the phenomenon rather than simply being dismissed as a failure to replicate
(Glass, McGaw, & Smith, 1981; Rosenthal, 1984).
From this perspective, the ganzfeld studies provide impressive replicability: 23 (82%)
of the 28 studies obtain positive results (more direct hits than chance would predict), a result
whose probability of occurring by chance is less than 1 in a thousand.
The conventional criterion of replication further requires that any competent investigator
be able to reproduce the claimed phenomenon, not just one or two gifted experimenters. This is
often a difficult criterion to achieve in new areas of investigation because a number of
unsuspected variables might affect the outcome. In psychological experiments, the
experimenter is often an important social stimulus for the subject and hence a poorly controlled
source of variability. Even in such an established area as classical conditioning, investigators at
one university were obtaining positive results 94% of the time while other investigators could
do so only 62% of the time (Rosenthal, 1966; Spence, 1964). Nor is psychology alone here.
Similar replication difficulties have been reported in medical studies of placebo efficacy
(Moerman, 1981) and in the physical science area of laser technology (Collins, 1974).
This problem may be even more acute in parapsychology because psi effects may
legitimately depend upon the motivational atmosphere established by the experimenter. Some
parapsychologists further believe that the experimenter’s own psi abilities and/or attitudes can
affect the results.
Despite these potential difficulties, the replicability of the ganzfeld effect does not
appear to rest on the success of one or two investigators. Six of the 10 investigative teams
contributing to the 28 examined studies obtained statistically significant results; and, even if all
the studies from the two most successful laboratories are discarded from the analysis (half of
the studies), the results remain significant (Palmer, Honorton, & Utts, 1988).
The power of a particular experiment to replicate an effect also depends upon how
strong the effect is and how many observations are made. If an effect is weak, an experiment
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30003-0
-Apnroved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30003-0 _
PslFhenomena
with too few subjects, sessions, or observations will fail to detect it at a statistically significant
level-—even though the effect actually exists.
This is strikingly illustrated by a recent medical experiment designed to determine
whether aspirin can prevent heart attacks. The study was discontinued in 1987 because it was
already clear the answer was yes. After six years, the aspirin group had already suffered 45%
fewer heart attacks than a control group which received only placebo medication, a result that
would occur by chance less than one time out of a million (The Steering Committee of the
Physicians’ Health Study Research Group, 1988). With such impressive results, it was
considered unethical to keep the control group on placebo medication. The study was widely
publicized as a major medical breakthrough.
The pertinent point here is that the study included over 22,000 subjects. If it were to be
repeated with 3,000 subjects, a significant aspirin effect would be unlikely to emerge; the
experiment would fail to replicate. Despite its undisputed reality and its practical importance,
the aspirin effect is actually quite weak.
Now reconsider the ganzfeld effect. If the effect actually exists and has a true direct-hit
rate of 38%, then statistically we should expect ganzfeld studies with 30 sessions (the average
for the 28 studies) to obtain a significant psi effect only about one-third of the time (Utts,
1986). The ganzfeld effect is about 4 times stronger than the aspirin effect.
In short, it is unrealistic to demand that any “real” effect be replicable at any time by any
competent investigator. The replication issue is more complex than that, and meta-analysis is
proving to be a valuable tool for dealing empirically with some of those complexities.
Inadequate Controls The second major criticism of parapsychology is that many, if
not most, of the experiments have inadequate controls and safeguards. Flawed procedures that
could permit a subject to obtain the communicated information in normal sensory fashion either
inadvertently or through deliberate cheating are particularly fatal. This is called the problem of
“sensory leakage.” Inadequate procedures for randomizing (randomly selecting) target stimuli
are another common problem.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30003-0
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30003-0
Psi Phenomena
Methodological inadequacies plague all the sciences, but the history of parapsychology
is embarrassingly full of promising results that collapsed when the procedures were critically
examined (Akers, 1984). One common charge against parapsychology is that preliminary,
poorly controlled studies often obtain positive results, but that as soon as better controls and
safeguards are introduced, the results weaken or disappear.
Meta-analysis provides an empirical way of evaluating this charge, too. Once a flaw is
discovered in a completed experiment, there is no persuasive way of arguing that the flaw did
not contribute illegitimately to a positive outcome; the only remedy is to redo the experiment
correctly. In a database of several studies, however, it is possible to test whether the more
poorly controlled studies in the database did in fact obtain more positive results than the well
controlled studies. If there is a correlation between a procedural flaw and positive results across
the studies, then there is a problem. In the case of the ganzfeld database, both critic Hyman and
parapsychologist Honorton agree that flaws of inadequate security and possible sensory
leakage do not correlate with positive results. Hyman claimed to find a correlation between
flaws of randomization and positive results, but both Honorton’s analysis and two additional
analyses by nonparapsychologists dispute his conclusion (Harris & Rosenthal, 1988;
Saunders, 1985). Moreover, a series of 10 new studies designed to control for flaws identified
in the original database yielded results quite consistent with the original set of 28 studies
(Hariris & Rosenthal, 1988).
File-Drawer Problem Suppose that each of 20 investigators independently decides
to conduct a ganzfeld study. Even if there were no ganzfeld effect, there is a reasonable
probability that at least one of these investigators would obtain a significant result by pure
chance. That lucky investigator would then publish a report of the experiment, but the other 19
investigators—who obtained “null” results—are likely to become discouraged, put their data
into a file drawer, and move onto something more promising. As a result, the scientific
community would learn about the one successful study but have no knowledge of the 19 null
studies buried in the file drawers. The database of known studies would thus be seriously
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30003-0
I^f¥ifi
renomena0r Release 2000/08/10
: CIA-RDP96-00792R000100130003-0
biased toward positive studies, and any meta-analysis of that database would arrive at similarly
biased conclusions. This is called the file-drawer problem.
The problem is particularly tricky because it is impossible, by definition, to know how
many unknown studies are languishing in file drawers somewhere. Nevertheless,
parapsychologists offer two lines of defense against the charge that the file-drawer problem
seriously compromises their database.
First they point out that the Parapsychological Association may be the only professional
scientific society to have an official policy against selective reporting of positive findings and
that negative findings are frequently reported in the Association’s five affiliated journals.
Moreover, the community of parapsychologists is relatively small, and most investigators
know about ongoing work in the active laboratories around the world. When conducting meta-
analyses, they attempt to ferret out all unpublished studies through their personal networks.
But the major defense is statistical, and again meta-analysis provides an empirical
approach to the problem. By knowing the overall statistical significance of the known database,
it is possible to compute the number of studies with null results that would have to exist in file
drawers to cancel out that significance. In the case of the ganzfeld database, it is estimated that
there would have to be over 400 unreported ganzfeld studies with null results—the equivalent
of 12,000 sessions—to cancel out the statistical significance of the 28 studies analyzed in the
debate (Honorton, 1985). Not surprisingly, there is consensus that the overall significance of
the ganzfeld studies cannot reasonably be explained by the file-drawer effect (Hyman &
Honorton, 1986).
Rather than continuing their debate, Hyman and Honorton issued a joint communique
in which they set forth their areas of agreement and disagreement and made a series of
suggestions for the conduct of future ganzfeld studies (Hyman & Honorton, 1986). Their
debate and the subsequent discussion provide a valuable model for evaluating all disputed
domains of scientific inquiry.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30003-0