Approved For Release 2000/08/08
: CIA-RDP96-00789R003200100001-5
ANOMALY OR ARTIFACT
*
Consistency With Previous Parapsychological Findings
Bern and Honorton (1994) also claimed that “there are reli-
able relationships between successful psi performance and con-
ceptually relevant experimental and subject variables, relation-
ships that also replicate previous findings” (p.
13). They point
to three such “replications.” One is a small, but statistically sig-
nificant, correlation of .18 between a measure of extroversion
and “psi performance.” This is consistent with a tendency found
in previous psi studies. Second, they report the strong psi per-
formance of the Julliard students that they see as consistent with
psi studies that found a relation between psi abilities and cre-
ative and artistic abilities. This latter replication
is not so im-
pressive when one considers that only 20 students were involved
and that their performance was not significantly different from
the other participants in the two studies in which they partici-
pated (Fisher’s exact p =
.262, two-tailed).
In addition, as
point out below, the Julliard students were exposed to just those
conditions that favored high
hit rates—targets that were
re-
peated, a preponderance ofdynamic targets, and active prompt-
ing by the experimenter during judging. Thus,
it
is unclear
whether their high hit rate was a function of their creativity or a
function of the special targets and conditions with which they
happened to be associated.
The third correlate could not be demonstrated for the auto-
ganzfeld studies. Bern and Honorton (1994) pointed out that
the subjects in the autoganzfeld tended to believe in
psi, re-
ported psychic experiences, and had practiced meditation or
related techniques. These variables were previously reported as
correlates of psi. However,
I do not see how they can claim that
these attributes of their subject population are a replication of
previous findings. They report no correlations between these
variables and performance in the autoganzfeld studies. Indeed,
they cannot report any correlation because they did not have
subjects who lacked these properties. We do not know if nonbe-
lievers and people without psychic experiences would have per-
formed better or worse than the actual subjects.
In other words, they can justify only one of the correlates that
they use to claim consistency with previous psi studies. Even
here the relationship is weak and is just one of many previously
reported correlates that might have been found. At one time,
for example, parapsychologists claimed that the decline effect
was a pervasive and characteristic property of psi. However,
when no decline effect is found in a parupsychologicul study,
it
does not deter the experimenter from pointing to some other
significant departure from chance as evidence for psi. Note that
in the autoganzfeld studies, there is no decline effect.
Randomization and Claims of Psi
As 1 already stated, I agree that the autoganzfeld studies meet
most of the requirements that Honorton and
I specified in our
joint communique (Hyman & Honorton, 1986). One surprising
exception is the inadequate testing of the randomization proce-
dures. The issue of randomization was central to the debate con-
cerning the original ganzfeld findings (Hyman, 1985). Adequate
randomization procedures are critical for parapsychological re-
search because the evidence for psi is based on a low probability
value for a departure from a chance baseline. Such probability
values have meaning within an idealized statistical model ofthe
experimental situation. Whether this statistical model applies to
a given situation is an empirical matter that must be adequately
justified if the stated significance levels are to be taken seriously.
Appropriate randomization procedures are one way to help en-
sure that the statistical model applies to the experimental data.
With respect to the autoganzfeld studies, this would entail se-
lecting the targets for each trial and ordering the target and de-
coys during judging in a demonstrably random manner. In ad-
dition, following the practice of a few past researchers, the para-
psychologist can also provide some post hoc analyses to show
that the distributions of targets and judging orders are consis-
tent with the underlying probability model.
Unfortunately, the autoganzfeld studies fell short on this crit-
ical requirement. The tests for adequacy of randomization were
confined to showing a uniform distribution of outputs from
to 160 for target selection and a uniform distribution of the per-
mutations of all possible orderings during the judging proce-
dure. Emitting a uniform distribution of target choices is a nec-
essary but hardly sufficient requirement for an adequate ran-
dom generator.
These randomization procedures are critical because we can
expect strong systematic biases during the judging procedure.
The fact that the items to be judged have to be presented se-
quentially. when combined with what we know about subjective
validation (Marks & Kammann, 1980), would lead us to expect
a strong tendency to select the first or second items during the
judging
series. We would also expect strong response biases
within each target pool. Bern and Honorton show such a bias in
the target pool used for Study 302. Both these biases may be
strengthened by the fact that the experimenter interacts with
the receivcrduring the judging process. Although most receivers
participate
in one session, each experimenter participates in
several. The response biases of the experimenters can play an
important role, especially in those studies in which the experi-
menter deliberately prompts the receiver to choose a particular
item during the judging. Such active prompting occurred in 6
of the
1 studies (Honorton etal., 1990).
If the randomizing of the selection of targets and of the order-
ing of items during judging
is adequate, such response biases
should not affect the validity of the statistical tests. One way to
prevent response biases from distorting the hit rate is to use a
randomizing procedure that makes sure that each item within
a target pool occurs equally often. The simple randomizing pro-
cedure used in the autoganzfeld studies would guarantee that
each target occurred an equal proportion (not number) of times
only in the very long run.
In any finite number of trials, the
individual targets would occur with varying frequencies. Again,
if the randomization was adequate, this inequality of occur-
rence would not bias the hit rate. The items in some target pools
that occurred most frequently would be those that were favored
2 One referee suggested that
I make
it clear that
I am not claiming
that sensory leakage occurred because of experimenter prompting.
I
agree. The experimenter, according to the protocol, was ignorant of
which member of the target pool was the target during the judging pro-
cedure. The point is that by actively helping the subject to rate the mem-
bers of the target pool, the experimenter let his or her own subjective
biases enter the selection procedure.
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200100001-5
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200100001-5
RAY HYMAN
Table
Hit Rate as a Function ofthe Frequency ofOccurrence oj Targets
Variable
Frequency
Total
Hits
106
Misses
224
n
330
Hit rate
.250
.278
.381
.294
.346
.333
571
.750
.321
by the response bias. This would bias the hit rate upward. The
items in other target pools, however, that occurred most
fre-
quently would be those that were avoided by the response bias.
This would bias the hit rate downward. With adequate random-
ization, these two tendencies would balance each other.
Achieving adequate randomization is not easy. Much can go
wrong—as some parapsychologists, among others, have shown.
This is why it is disappointing that the autoganzfeld studies did
not show the same concern for randomizing that they showed
for other aspects of the methodology. This
is also why, in my
role of devil’s advocate,
l was interested in directly checking the
actual distribution of target positions among the decoys during
judging. Daryl Bern kindly agreed to supply me with this infor-
mation along with other data from the autoganzfeld database.
Unfortunately, the variable labeled position on the data sheet
turned out to be the original position of the target in its target
pool rather than its position during judging. This latter infor-
mation was unavailable to either Bern or me at the time of this
writing.
Hit Rate and Target Frequency
Because
I could not directly check the adequacy of the ran-
domization procedures,
1 tried to find some indirect indicators.
If randomizing was inadequate and targets occurred with vary-
ing frequency, possible biases might show up as differential hit
rates for targets occurring with various frequencies. For exam-
ple, if targets favored by response biases were also favored by a
deficient target selection procedure, then we would find a posi-
tive correlation between hit rate and target frequency.
It would
be possible, of course, for a deficient randomization procedure
to yield a negative correlation. To see
if actual repetitions of
targets had any observable consequences,
I tabulated the pro-
portion of hits as a function of how many times a target oc-
curred in this database.
As Table
shows, the relation between hit rate and target
frequency was strong. The test for a linear trend among the pro-
portions (Snedecor& Cochran, 1967, pp. 246-248) was positive
and significant, (z = 2.49, p= .013, two-tailed). An indication
of the strength of this trend is given by the Spearman rank order
correlation between the hit rate and target frequency, which was
.83. Another way to look at this relationship would be to com-
pare the hit rate of targets that occurred once or twice (.27) with
those that appeared three or more times (.36).
This pattern
exists separately
for the
static and dynamic
targets, although it is stronger among the dynamic targets. The
static targets that occurred once or twice had a hit rate of .22
compared with a hit rate of .31
for those that occurred more
than twice. The hit rate was .32 for those dynamic targets that
occurred once or twice us compared with a hit rate of .41
for
those that occurred three or more times.
Target Occurrence and Experimenter Prompting
What accounts for this peculiar relationship? Is the correla-
tion between target frequency and hit rate determined by which
particular targets get repeated? Or does replication itself some-
how increase the hit rate? If the relation is due to response bi-
ases,
I would expect experimenter prompting to affect the later
occurrences of targets rather than their first occurrences. With
these questions in mind,
I conducted a multinomial analysis of
variance (Woodward, Bonett, & Brecht, 1990). In this analysis,
hit rate was the dependent variable, and 3 two-level factors were
the independent variables: target type (static, dynamic), target
occurrence (first, later), and experimenter prompting (no, yes).
Of the interactions, only that between target occurrence and
experimenter prompting was significant, x
( I
, AT = 330) = 6.83,
p = .009. The two significant main effects were target type, x
( 1
,
N = 330) = 4.76, p = .030, and target occurrence, x
2U> N =
330)= 11.56,p <.001.
The difference between the hit rate for dynamic targets (.356)
and that for static targets (.249) does not interact with the other
two factors and will be ignored for the present discussion.
4 The
meaning of the
interaction
between
target occurrence and
prompting can be seen in the simple effects of target occurrence
within each level of prompting. With no experimenter prompt-
ing, the effects of target occurrence were minimal: The hit rate
for first occurrences of targets was .29 1 and that for later occur-
3 To be consistent with Bern and Honorton,
I treated the basic data-
base as the 330 sessions
it) Studies
1 through 301. Study 302, which
used a single target pool, was treated as a special case.
4 These hit rates are slightly different from those used by Honorton et
al. (1990) and Bern and Honorton (1994). This is because they com-
puted hit rates for any category by simply dividing the number of hits
by the total number in that category. The hit rate for dynamic targets
obtained with this approach is 6 1 / 1 64 = .372 and that for static targets
is 45/166 =
.271. These rates are means weighted by the number of
cases in the cells for each combination of levels of the factors. For the
purposes of additivity of effects,
I am using the unweighted means (each
cell of the design is weighted equally). This removes distortions and con-
founds that are due to unequal cell sizes. In the present case, the differ-
ences are small and inconsequential.
1 am supplying this footnote to
explain some discrepancies that might confuse the reader.
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200100001-5
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200100001-5
ANOMALY OR ARTIFACT
rences was .334, x
2(L (V = 181) = .396, p = .534. The effect of
target repetition combined with experimenter prompting, how-
ever, was very large. The hit rate for first occurrences of targets
with experimenter prompting was only .140. The hit rate for
later occurrences of targets when combined with experimenter
prompting jumped to .445. This gain was significant, x
(
I
, N -
149) =
14.702, p =
.0001. These results suggest that experi-
menter prompting depresses hit rates for first occurrences of
targets and enhances hit rates for subsequent occurrences of
targets.
Internal Checks on the Validity of This Pattern
Is this peculiar relation among hit rate, target frequency, and
experimenter prompting merely a fluke?
I broke the data into
subsets in several ways to see if the pattern was consistent in the
different subcategories.
I checked this pattern within the dy-
namic and static targets separately.
I compared Targets
1 to 80
with Targets 81 to
1 60. Likewise,
I looked for the pattern within
Studies
101 through 103 taken as a group as compared with
Studies 104 through 301 considered as a group.
I also checked
for this pattern for each of the five experimenters who contrib-
uted the most sessions. Although the numbers became small in
some of these comparisons, the hit rate was consistently larger
for later as opposed to first occurrences of a target.
I found just
one nonsignificant exception in the trials for one experimenter.
Likewise, wherever meaningful comparisons were possible, the
interaction
between
prompting and
target
occurrence
oc-
curred.
5 For this database, then, the dependence of hit rate on
frequency of target occurrence and experimenter prompting
was a robust effect.
Implications
As far as
I know, this dependence of hit rate on target occur-
rence and experimenter coaching has never been previously re-
ported in parapsychological research. One referee suggests that
the dependence of hit rate on target frequency and prompting
may reveal important moderator variables rather than artifacts.
The referee may be correct. The skeptic, however, might point
to the long history of alleged “moderator" variables in parapsy-
chology—such
as
the
decline
effect,
displacement
effects,
sheep-goat effects, and others. The problem is that when such
moderators are discovered in the data, they arc put forth as im-
portant indicators or characteristics of psi. The absence of such
characteristics in subsequent data, however, does not deter para-
psychologists from claiming evidence for psi if they find a sig-
nificant hit rate. This is the troublesome problem of boundary
conditions. The parapsychologists have been unable to specify
what would constitute the absence of psi.
The positive effect for repeated occurrences of a target may
eventually turn out to be an important property of psi— if psi
exists. However, the fact that first occurrences of a target pro-
duce a hit rate consistent with chance raises questions. All of
the positive effect in the ganzfeld experiments rests on those
targets that have occurred more than once. The prompting
effect
is even more curious. On
first occurrences of a target,
active coaching by the experimenter seems to depress the
hit
rate— .28 without prompting versus .15 with prompting. For
second or later occurrences of a target, active coaching appears
to enhance the hit rate— .33 without prompting versus .45 with
prompting. If the prompting by the experimenter is intended to
increase reliability by reminding the receiver of ganzfeld asso-
ciations that he or she might overlook during judging, why
should the effects of such prompting show up only for the sub-
sequent occurrences of a target?
That hit rate correlates with frequency of target occurrence
could mean that the “better" targets are somehow selected more
often by the randomizing procedure. Or it could mean that fre-
quency of occurrence, itself,
is the determinant of a higher hit
rate. The data suggest the latter possibility. The 48 targats that
occurred exactly once in the database had a hit rate of .22. The
first occurrence of the targets that occurred more than once had
a hit rate of .23. The combined hit rate for second or later oc-
currences of targets was .39. Another way of examining this re-
lation would be to look separately at the hit rates for first and
subsequent occurrences of targets that appeared exactly twice,
three times, four times, and so on. Only the targets that oc-
curred from two to five times could be used because only one or
two targets appeared with frequencies of six or more. In all these
comparisons, the first occurrences consistently had a lower hit
rate than subsequent occurrences of the same targets.
Whatever the source for this pattern, it raises questions about
interpretations of other findings in the database. For example,
Bern and Honorton (1994) pointed to the high rate of .50 for
the 20 Julliard students as evidence for the effect of artistic cre-
ativity on hit rate. However, all of the sessions in which the Jul-
liard students appeared were prompted, and
15 of the 20 used
second or later occurrences of a target. On the five targets that
were occurring for the first time, these students got one hit. Con-
sequently, we cannot tell if the hit rate for these students reflect
any special abilities or if they are due to whatever makes hit rate
a function of target frequency and coaching in this database.
Are these findings due to an artifact, or do they point to some
new, hitherto unrecognized property of psi? We cannot say. The
existence of this pattern in the database, however, strongly sup-
ports the need to replicate the findings before we can be confi-
dent that the parapsychologists have finally found a way to cap-
ture and tame their elusive quarry.
Conclusions
The autoganzfcld experiments are a praiseworthy improve-
ment in methodological sophistication and experimental rigor
over the previous ganzfeld experiments. Despite these improve-
ments, the experiments fall disappointingly short in the critical
area of justifying the randomization procedures. Even though
all but one of the individual studies produced a positive effect
size and the overall effect was significant, the autoganzfeld ex-
periments do not constitute a successful replication of the orig-
inal ganzfeld experiments.
5 As noted in Footnote 5 ofthe Bem-Honorton article, a recent review
of the original computer files uncovered a duplicate record in the auto-
ganzfeld database. This has now been eliminated, reducing by one the
number of sessions on which my analysis was based. Some experiment-
ers
contributed
only unprompted
sessions, and some
contributed
mainly prompted sessions.
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200100001-5
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200100001-5
RAY HYMAN
Although Bem and Honorton point to consistencies between
the autoganzfeld results and those of previous parapsychologi-
cal research, these consistencies are more apparent than real.
On the other hand, as
1 have argued, important inconsistencies
exist between the two databases.
Three robust elfects in the autoganzfeld database are the de-
pendence of hit rate on type of target (dynamic or static), target
occurrence (first or subsequent), and experimenter prompting
(yes or no). Although my looking for effects of the latter two
factors was motivated by my concern for possible randomiza-
tion deficiencies, their existence should interest both parapsy-
chologists and critics. This is because the existence of an effect
depends on these factors. The combination of dynamic targets,
repeated occurrences of a target, and experimenter prompting
produces a hit rate of .471 with 95% confidence limits from
.305 to .629. The combination of static targets, first occurrences
of a target, and no prompting yields a hit rate of
. 1 78 with 95%
confidence limits from .066 to .336.
We do not have enough information to know
if the depen-
dence of hit rate on target frequency and experimenter prompt-
ing involves response preferences for items within a target pool.
One way to ensure that such preferences do not bias the hit rate
is to present each member of a target pool equally often.
I tried
to get some idea what the hit rate might be if each member of a
target pool had occurred equally often.
I restricted myself just
to those target pools in which each member occurred at least
once.6 The hit rate for first occurrences ofa target in these target
pools was ,275 with 95% confidence limits from .167 to .399.
(The hit rate for the targets in these target pools that occurred
a second or later time was .427.) This finding does not prove
anything, but
it suggests that
if the targets within each target
pool had occurred equally often, the results might have been
consistent with chance.
The autoganzfeld studies failed to replicate key findings of
the original ganzfeld experiments, even though the power was
sufficient. The positive effect size and significance depended on
a new type of target whose presentation involves a new technol-
ogy and on target repetition and experimenter coaching. What-
ever their source, these effects are new to the psi literature. We
do not know how much of this is unique to this experimental
setup and laboratory. For these reasons, we have to wait for fu-
ture attempts at replication to see if a replicable psi effect is at
hand.
1 could not use higher frequency of occurrence because only three
target pools existed in this database that had at least two occurrences of
each of its members.
References
Bern, D.J., & Honorton, C. (1994). Does psi exist? Replicable evidence
for an anomalous process of information transfer. Psychological Bul-
letin, 115. 4-18.
Honorton, C. (1985). Meta-analysis of psi ganzfeld research: A response
to Hyman. Journal ofParapsychology, 49, 51-91.
Honorton, C„ Berger, R.E., Varvoglis, M.P., M., Derr, P., Schechter, E.I.,
& Ferrari, D.C. (1990). Psi communication in the ganzfeld: Experi-
ments with an automated testing system and a comparison with a
meta-analysis of earlier studies. Journal of Parapsychology, 54, 99-
139.
Hyman, R. (1985). The ganzfeld psi experiment: A critical appraisal.
Journal ofParapsychology, 49, 3-49.
Hyman, R„ & Honorton, C. ( 1 986). A joint communique: The psi ganz-
feld controversy. Journal ofParapsychology, 50, 351-364.
Marks, D., & Kammann, R. (1980). The psychology of the psychic.
Buffalo, NY: Prometheus Books.
McCrone, J.(1993, May
1). Roll up for the telepathy test. New Scientist,
pp. 29-33.
Snedecor, G.W., & Cochran, W.G. ( 1967). Statistical methods (6th ed.,
pp. 246-248). Ames: Iowa State University Press.
Woodward, J.A., Bonett, D.G., & Brecht, M. L. (1990). Introduction
to linear models and experimental design. San Diego, CA: Harcourt
Brace Jovanovich.
Received July 26, 1993
Accepted July 27, 1993
Approved For Release 2000/08/08
: CIA-RDP96-00789R003200100001-5