Meta-Analysis Part
Approved For Release fJf^0|O^:J^fpf|E>^-OO789ROO22OO39OOO1-5
To address
this problem in Part 2 of this meta-analysis, we
will
actually be performing two
separate meta-analyses. The first will take into account the minimum number of experiments that
we decide
is
a reasonable
partitioning, and the second meta-analysis
will be
for the maximum
number of experiments.
The two end results will be compared, and the more conservative of the
two
will be used as the overall result.
Deciding on a range of possible experiments allows us to form an “uncertainty” factor for each
reference.
If
a
reference’s maximum-minimum experiment range
is large
as compared to the
average observed
range, we must
consider
that the
quality
of that
reference,
at
least
for our
purposes,
is poor. We plan on presenting a breakdown of each reference’s uncertainly in the Part 2
meta-analysis to judge how clear each reference was in this study.
Example of reference source quality analysis
In Table
5 we present an example of a preliminary quality
analysis applied to the source of
reference.
We assigned
arbitrary weights according to our perception of the
quality of average
papers published
in each parapsychological reference source
(not counting the Princeton data).
Then, after making guesses for these weights, we calculated a combined z score contributed by each
journal and compared
it to a weighted z score according to equation
(5).
As seen in Table
5, the
original combined
z score dropped by
4 orders of magnitude
in significance, but the weighted
z
score
is
still
quite
significant.
We expect that the wider range
of quality
weights,
as we have
proposed above, will make a larger difference in a weighted analysis, but it would appear that most
of the reports would have to be extremely poor in quality to nullify the overall p value.
Table
5. Exploratory quality analysis of reference sources
Reference
Studies
Overall z
p (2-tail)
Assigned weight
Journal of Parapsychology
7.055
3.30 x 10
Proceedings of the PA
5.036
4.76 x 10" 7
Research in Parapsychology
4.052
5.08 x 10
" 5
Journal of the ASPR
4.105
4.04 x 10* 5
European Journal of Parapsychology
3.052
0.002
Journal of the SPR
1.692
0.091
Combined unweighted result =
10.27
p < 9.45 x 10
' 25
Combined weighted result =
9.53
p < 1.60 x 10
" 21
B-23
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5
Release |J(^^6j0^:^|^pf^-00789R002200390001-5
Example of chronological analysis
In Table
6 we show an analysis of variance of the 297 detailed experiments grouped according
to year of publication.
Table
6.
Chronological analysis of variance
SOURCE:
grand mean
year
N
MEAN
Z
SD
SE
297
0.5979
1.5823
0.0918
SOURCE
:
year
year
N
MEAN
Z
SD
SE
1970
0.8247
3.0969
1.3850
1971
0.6292
2.3180
0.9463
1972
1.3565
1.4253
0.4751
1973
4.1239
1.4665
0.5987
1974
1.1539
1.9879
0.6286
1975
1.5804
2.1841
0.7280
1976
0.7366
1.5784
0.3828
1977
0.5695
1.8333
0.3823
1978
-0.2520
1.1482
0.3827
1979
0.6012
1.2325
0.4658
1980
-1.1411
1.3480
0,6029
1981
2,3437
0.7836
0.2962
1982
164
0.3098
1.2595
0.0983
1983
1.7500
1984
0.8492
1.0779
0.2473
SOURCE
SS
df
MS
year
151 .2565
10. . 8040
error
589 ,8531
282
2.,0917
5.165
1.14
x 10-6
This ANOVA shows that mean z scores differ significantly from year to year. We then looked
for trends in z scores by performing a correlation between year and mean
z.
Results showed that
r = -0.205, t(13) = -0.756, p = 0.463,
i.e. there was no
significant correlation between year of
publication and mean z score observed for that year.
B-24
Approved For Release 200u/0
IA-RDP96-00789R002200390001 -5