Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
401
cal studies, resulting from the observation by one
physician that his lung cancer patients who smoked
did not recover at the same rate as those who did
not. There are many medications in common use
for which there is
still no medical explanation for
their observed therapeutic
effectiveness, but that
does not prohibit their use.
There are also examples where a coherent theory
of a phenomenon was impossible because the
re-
quisite background information was missing. For
instance, the current theory of endorphins as an
explanation for the success of acupuncture would
have been impossible before the discovery of endor-
phins in the 1970s.
Hosteller’s observation that ESP will not replace
the telephone leads to the question of whether or
not psi abilities are of any use even if they do exist,
since the effects are relatively small. Again, a look
at history is instructive. For example, in 1938 For-
tune Magazine reported that “At present, few
sci-
entists
foresee
any
serious
or
practical
use
for
atomic energy.”
Greenhouse implied that I think parapsychology
is not accepted by more of the scientific community
only because they have not examined the data, but
this misses the main point
I was trying to make.
The point is that individual scientists are willing to
express an opinion without any reference to data.
The interesting sociological question
is why they
are so resistant to examining the data. One of the
major reasons is undoubtedly the perception identi-
fied by Greenhouse that there
is some connection
between parapsychology and the occult, or worse,
religious beliefs. Since religion is clearly not in the
realm
of science, the very thought that parapsy-
chology might be a science leads to what psychol-
ogists
call
“cognitive
dissonance.” As
noted
by
Griffin
(1988), “People
feel unpleasantly aroused
when two cognitions are dissonant—when they con-
tradict one another” (page 33). Griffin continued by
observing that there are also external reasons for
scientists to discount the evidence, since “It is gen-
erally easier to be a skeptic
in the face
of novel
evidence; skeptics may be overly conservative, but
they are rarely held up to ridicule” (page 34).
In summary, while
it may be
safer and more
consonant with their beliefs
for individual
scien-
tists to ignore the observed anomalous effects, the
scientific
community
should
be
concerned
with
finding an explanation. The explanations proposed
by Greenhouse and others are simply not tenable.
specify what should happen
if there
is no such
thing as ESP by using simple binomial models,
either to find p-values or Bayes factors. As noted
by Mosteller, if there is no ESP, or other nonstatis-
tical explanation for an effect, we should be able to
carry out null experiments and get no effect. Other-
wise, we should be worried about using these sim-
ple models for other applications.
Greenhouse, in his first alternative explanation
for the results, questioned the use of these simple
models, but his criticisms do not seem relevant to
the experiments discussed in Section 5 of my paper.
The experiments to which he referred were either
poorly controlled, in which case no statistical anal-
ysis could be valid, or were specifically designed to
incorporate
trial by trial feedback in such a way
that the analysis needed to account for the added
information. Models and analyses for such experi-
ments can be found in the references given at the
end of Diaconis’ discussion.
For the remainder of this discussion, I will con-
fine myself to models appropriate for experiments
such as the autoganzfeld described in Section 5.
It
is this scenario for which Bayarri and Berger com-
puted Bayes
factors, and
for which Dawson
dis-
cussed possible Bayesian models.
If ESP does exist, it is undoubtedly a gross over-
simplification
to use a simple
non-null binomial
model for these experiments. In addition to poten-
tial
differences
in
ability among
subjects,
there
were also observed differences due to dynamic ver-
sus static targets, whether or not the sender was a
friend, and how the receiver scored on measures of
extraversion. All of these differences were antici-
pated in advance and could be incorporated into
models as covariates.
It is nonetheless instructive to examine the Bayes
factor computed by
Bayarri and Berger
for
the
simple non-null binomial model. First, the observed
anomalous effects would be less interesting
if the
Bayes factor was small for reasonable values of r,
as it was for the random number generator experi-
ments analyzed by Jefferys (1990), most of which
purported to measure psychokinesis instead of ESP.
Second, the Bayes factor provides a rough measure
of the strength
of the evidence against the
null
hypothesis and is a much more sensible summary
than the
p-value. The Bayes factors provided by
Bayarri and Berger are probably more conserva-
tive, in the sense of favoring the null hypothesis,
than those that would result from priors
elicited
from parapsychologists, but are probably reason-
able
for those who know nothing about past
ob-
served effects.
I expect tht most parapsychologists
would not opt for a prior symmetric around chance,
but would
still choose one with some mass below
REPLICATION AND MODELING
Parapsychology
is one of the few areas where a
point null hypothesis makes some sense. We can
Approved For Release 2000/08/08
: CIA-RDP96-00789R0031 0001 0001 -6
402
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
J. UTTS
chance. The final reason
it
is instructive to exam-
ine these Bayes factors is that they provide a quan-
titative challenge to skeptics to be explicit about
their prior probabilities for the null and alternative
hypotheses.
Dawson
discussed
the
use
of
more
complex
Bayesian
models
for
the
analysis
of
the
auto-
ganzfeld data. She proposed a hierarchical model
where the number of successes for each experiment
followed a binomial distribution with hit rate
p,,
and logit( p,) came from a normal distribution with
noninformative priors for the mean and variance.
She then expanded this model to include heavier
tails by allowing an additional scale parameter for
each experiment. Her rationale for this expanded
model was that there were clear outlier series in
the data.
The hierarchical model proposed by Dawson is a
reasonable place to start given only that there were
several experiments trying
to measure the same
effect, conducted by different investigators. In the
autoganzfeld
database,
the
model
could
be
ex-
panded
to incorporate the
additional information
available.
Each
experiment
contained some
ses-
sions with
static targets and some with dynamic
targets, some
sessions
in which
the
sender and
receiver were
friends and
others
in which
they
were not and some information about the extraver-
sion score of the receiver. All of this information
could be included by defining the individual session
as the unit of analysis, and including a vector of
covariates
for each
session.
It would then make
sense to construct a logistic regression model with
a component
for each
experiment,
following
the
model
proposed by Dawson, and
a term
X{3
to
include the covariates. A prior distribution
for
/3
could
include
information
from
earlier
ganzfeld
studies. The advantage
of using a Bayesian
ap-
proach over
a
simple
logistic
regression
is
that
information could be continually updated. Some of
the recent work in Bayesian design could then be
incorporated so that future trials make use of the
best conditions.
Several of the discussants addressed the concept
of replication.
I agree with Mosteller’s implication
that it was unwise for the audience in my seminar
to respond to my replication questions so quickly,
and that was precisely my point. Most nonstatisti-
cians do not seem
to understand the complexity
of the replication question.
Parenthetically, when
I posed the same scenario to an audience of statis-
ticians,
very
few
were
willing
to
offer
a
quick
opinion.
Bayarri and Berger provided an
insightful
dis-
cussion of the purpose of replication, offering quan-
titative answers to questions that were implicit in
my discussion. Their analyses suggest some alter-
natives to power analysis that might be considered
when designing a new study to try to replicate a
questionable result.
Morris
addressed
thie
question
of what
con-
stitutes
a
replication
I of
a
meta-analysis.
He
distinguished between exact and conceptual
repli-
cations.
Using
his
distinction,
the
autoganzfeld
meta-analysis
could
bb
viewed
as
a
conceptual
replication
of the
earlijer ganzfeld
meta-analysis.
He noted that when sujch a conceptual replication
offers
results
similar
to
those
of
the
original
meta-analysis,
it lends
: legitimacy
to the
original
results,
as was
the
case
with
the
autoganzfeld
meta-analysis.
Greenhouse and Morris both noted the value of
meta-analysis as a method of comparing different
conditions,
and
I
endorse
that
view.
Conditions
found
to
produce
different
effects
in
one
meta-
analysis could be explicitly studied in a conceptual
replication. One
of the
intriguing
results
of the
autoganzfeld experiments was that they supported
the
distinction between
effect
sizes
for dynamic
versus static targets foimd in the earlier ganzfeld
work, and they supported the relationship between
ESP and extraversion found in the meta-analysis
by Honorton, Ferrari and Bern (1990).
Most modern parapsychologists, as indicated by
Morris, recognize that demonstrating the validity
of their preliminary findings will depend on identi-
fying and utilizing “moderator variables” in future
studies. The use of such; variables will require more
complicated statistical fnodels than the simple bi-
nomial models used
in: the past. Further, models
are needed for combining results from several
dif-
ferent experiments, that don’t oversimplify at the
expense of lost information.
In conclusion, the anomalous effect that persists
throughout the work reyiewed in my paper will be
better understood only after further experimenta-
tion that takes into account the complexity of the
system.
More
realistic,
and
thus more
complex,
models
will be
needed: to
analyze
the
results
of
those experiments. This: presents a challenge that I
hope will be welcomed tjy the statistics community.
ADDITIONAL REFERENCES
Allison, P.
(1979). Experimental parapsychology as a rejected
science. The Sociological Review Monograph 27 271-291.
Barber, B. (1961). Resistance by scientists to scientific discov-
ery. Science 134 596-602.
Berger,
J. O. and Delampapy, M.
(1987). Testing precise hy-
potheses (with discussion).
Statist. Sci. 2 317-352.
Chung, F.
R.
K., Diaconis, P., Graham, R.
L. and Mallows,
C.
L.
(1981). On
the permanents
of compliments
of the
direct sum of identity matrices. Adv. Appl. Math. 2 121-137.
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6
REPLICATION IN PARAPSYCHOLOGY
Cochran, W.
G.
(1954). The combination
of estimates from
different experiments. Biometrics 10 101-129.
Collins, H. and Pinch, T. (1979). The construction of the para-
normal: Nothing unscientific is happening. The Sociological
Review Monograph 27 237-270.
Collins, H. M. and Pinch, T. J. (1982). Frames of Meaning: The
Social Construction of Extraordinary Science. Routledge &
Kegan Paul, London.
Cornfield, J. (1959). Principles of research. American Journal
of Mental Deficiency 64 240-252.
Dempster,
A.
P., Selwyn, M.
R.
and Weeks,
B.
J.
(1983).
Combining historical and randomized controls for assessing
trends in proportions.
J. Amer.
Statist. Assoc. 78 221-227.
Diaconis, P. and Graham, R. L. (1981). The analysis of sequen-
tial experiments with feedback to subjects.
Ann. Statist. 9
236-244.
Fisher, R, A. (1932).
Statistical Methods for Research Workers,
4th ed. Oliver and Boyd, London.
Fisher,
R.
A.
(1935). Has Mendel’s work been
rediscovered?
Ann. ofSci. 1 116-137.
Galton, F. (1901-2). Biometry. Biometrika 1 7-10.
Greenhouse, J., Fromm, D., Iyengar, S., Dew, M. A., Holland,
A. and Kass, R. (1990). Case study: The effects of rehabili-
tation therapy for aphasia. In The Future of Meta-Analysis
(K. W. Wachter and M. L. Straf, eds.) 31-32. Russell Sage
Foundation, New York.
Griffin,
D.
(1988).
Intuitive judgment and the evaluation
of
evidence. In Enhancing Human Performance: Issues, Theo-
ries and Techniques Background Papers— Part
I. National
Academy Press, Washington, D.C.
Hansen, G. (1990). Deception by subjects in psi research. Jour-
nal of the American Society for Psychical Research 84 25-80.
Hunter, J. and Schmidt, F. (1990). Methods of Meta-Analysis.
Sage, London.
Iyengar, S. and Greenhouse,
J.
(1988). Selection models and
the
file drawer problem
(with
discussion).
Statist.
Sci. 3
109-135.
Louis,
T.
A.
(1984).
Estimating an ensemble
of parameters
using Bayes and empirical Bayes methods.
J. Amer. Statist.
Assoc. 79 393-398.
Mantel, N. and Haenszel, W. (1959). Statistical aspects of the
403
analysis of data from retrospective studies of disease. Jour-
nal of the National Cancer Institute 22 719-748.
Morris, C.
(1983). Parametric empirical Bayes inference: The-
ory and applications (rejoinder)
J. Amer.
Statist. Assoc. 78
47-65.
Morris, R. L. (1986). What psi
is not: The necessity for experi-
ments. In Foundations of Parapsychology (H. L. Edge, R. L.
Morris, J. H. Rush and J. Palmer, eds.) 70-110.
Routledge
& Kegan Paul, London.
Mosteller,
F.
and Bush R.
R.
(1954).
Selected quantitative
techniques. In Handbook of Social Psychology (G. Lindzey,
ed.) 1 289-334. Addison-Wesley, Cambridge, Mass.
Mosteller, F. and Chalmers, T. (1991). Progress and problems
in meta-analysis.
Statist. Sci. To appear.
Oteri,
L.,
ed.
(1975). Quantum Physics and Parapsychology.
Parapsychology Foundation, New York.
Pinch, T.
J. and Collins, H. M.
(1984).
Private science and
public knowledge: The Committee for the Scientific Investi-
gation
of Claims
of the Paranormal and
its
use
of the
literature. Social Studies of Science 14 521-546.
Platt, J. R. (1964). Strong inference. Science 146 347-353.
Rosenthal, R.
(1966).
Experimenter Effects
in Behavioral Re-
search. Appleton-Century-Crofts, New York.
Rosenthal, R. (1979). The “file drawer problem” and tolerance
for null results.
Psychological Bulletin 86 638-641.
Ryan,
L. M.
and Dempster,
A.
P.
(1984).
Weighted normal
plots.
Technical Report
394Z, Dana-Farber Cancer
Inst.,
Boston, Mass.
Samaniego, F.
J. and Utts,
J. (1983). Evaluating performance
in continuous experiments with feedback
to subjects.
Psy-
chometrika 48 195-209.
Smith, M. and Glass, G. (1977). Meta-analysis of psychotherapy
outcome studies. American Psychologist 32 752-760.
Waciiter, K.
(1988).
Disturbed by meta-analysis? Science 241
1407-1408.
West, M. (1985). Generalized linear models: Scale parameters,
outlier accommodation and prior distributions. In Bayesian
Statistics 2 (J. M. Bernardo, M. H. DeGroot, D. V. Lindley,
and A. F. M. Smith, eds.) 531-558. North-Holland Amster-
dam.
Approved For Release 2000/08/08
: CIA-RDP96-00789R003100010001-6