Approved For Release tJNGIPASStFtPD-oo789Ro6^b"(^§ti6b^
a
-5
A planned quality vs. z score correlational study
Once we perform the
quality
analysis and have a
list of raw z scores and
associated quality
weights, we plan on performing a correlation between these pairs of numbers.
If the correlation
is
significantly negative,
it would suggest that the better the quality of a study, the lower the z score.
This would be in accordance with what some
critics have claimed, namely that “there
is a strong
tendency for the rate of success to increase with the number of obvious defects” (Hyman,
1983,
p.23).
If a significant positive correlation
is seen, however, this criticism can be refuted.
CONCLUSION
In an
initial meta-analysis of psi experiments involving binary RNGs, we have identified 332
experiments published over the years 1969-1984 in 56 references.
Based on an analysis of 188 of
these
experiments
reported
in
parapsychological
journals, we
estimated
the
actual number
of
nonsignificant, unreported or unretrieved experiments to be 95. We found a total of 98 reported
control studies in 14 of these references. A summary of the meta-analytic results is shown in Table
7 (on the following page).
In agreement with a hypothesis of a “psi effect” on RNGs, the combined data indicate that, in
the aggregate, the experimental conditions resulted in anomalous statistical behavior of the RNG in
the
direction
of
effort
specified by the
task, and the
control
conditions
resulted
in expected
binomial
statistics for both mean z scores and standard deviations.
The combined data shows an interesting effect on the distribution of z scores. We find that in
the
experimental
condition
the mean
z
score
has been
increased
significantly
from
chance
expectation, which
in the present context
is in accord with the underlying hypothesis that the
z
score
will shift according to the direction of the subject’s effort.
In the control condition we find
the z mean shifted slightly, but not significantly so. We also find that the standard deviation of the
combined
distribution
of
experimental
z
scores
has become
significantly
fatter
than
chance
expectation, and that the combined control standard deviation is as expected.
Both of these effects
— a shifting of the mean and fattening of the standard deviation, are accounted for in a model
discussed by May, Radin et al (1985).
Part 2 of this study will report on a quality-weighted analysis of this same data.
By weighting
each
study according to
a
semi-objective
quality assessment
scale, we
will address
the major
criticisms of such experiments in a quantified way, and the overall experimental vs. control result
will provide a basis for discussion on whether or not this anomaly
is, in
fact,
real.
We urge readers to comment on and criticize the method described here, and especially on the
proposed weighting criteria presented in the Quality Analysis section above.
We plan to gain a
consensus
opinion among
informed
scientists
on
what
constitutes
an
agreeable,
conservative
weighting scheme
before we perform
the
quality
analysis.
In
this way,
the combined
results
observed
in the weighted data
will be
less subject to post hoc debate over the adequacy of the
B-25
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5
mA^M§3 M-feelease ^f(J|[8[0^:J^p^f|Pg^-00789R002200390001-5
analysis method, we
will avoid an enormous amount of pointless work, and we can proceed with
constructive discussion. We are especially interested in comparing the quality weights proposed by
parapsychologists,
critics of psi research, and “neutral”
scientists, as this may give us a clue as to
what
is considered
to be important
in
establishing consensus agreement among these
different
groups.
Table
7. Summary of RNG meta-analysis
Source
studies
(N)
z
Zz’s
Vn
p
(2-tail)
standard
deviation
of
z scores
variance test
against
a -
x
p
(2-tail)
SURVEY
Experiment
188
0.738
10.114
4.9 x 10
" 24m
568.5
- 47
4.9 x 10
Control
--0.0003
-0.002
0.999 m
44.0
0.62
PRINCETON
Experiment
0.339
4.063
4.9 x 10' 5
1.184
0.001
Control 19
0.050
0.375
0.708
.806
FILEDRAWER
ESTIMATE
Experiment
-0.084
-0.820
0.412
0.661
41.5
0.51
COMBINED
Experiment
427
0.420
8.684
3.9 x 10
' 18
1.414
853.7
5.9 x 10
' 34
Control
0.029
0.285
0.776
.905
80.3
0.22
*
This “too small" variance in the conlrol data
is compatible with a model proposed by May, Radin et al (1985)
and
is also discussed by Jahn, Nelson and Dunne (1985).
B-26
Approved For Release 2
00789R002200390001 -5