Fe* Release ^(|^^|2^S^PKPFE^h00789R002200390001 ~5
Table
2:
Overall
z score per investigator
Principal
investigator
References
Experiments
z (overall)
Andre
2.413
Bierman
3.899
Braud
3.760
Broughton
-0.470
Debes
0.356
Dunne
i
144
4.063
Edge
i
0.369
Giesler
2.694
Heseltine
-0.386
Hill
2.950
Honorton
1.523
Houtkooper
0.981
Jungerman
2.332
Kelly
3.366
Matas
0.513
May 2
-2.384
Millar
-0.875
Morris
1.835
Morrison
1.342
Palmer
1.750
Pantas
i
1.525
Radin
4.343
Randall
-0.029
Schechter
-1.060
Schmeidler
-1.273
Schmidt
13.224
Shafer
-1.440
Winnett
-0.089
TOTAL
297
8.548
1 This
is the name of the first author as listed in the references.
2 The study by May, Humphrey, and Hubbard (1980)
is not included in this survey
because their sequential analysis data collection technique
is not amenable to z score analysis.
As seen in Table 2, the overall z scores for these investigators ranged between -2.384 to 13.224.
The grand total z score, obtained by summing the 28 z scores and dividing by V 28
is z = 8,548,
for an overall p < 1.27 x 10~17 (2-tailed).
If we remove Schmidt’s 30 studies, since he obtained the
largest overall z score and is responsible for the largest number of references in our survey, we find
the grand total z = 6.160, p < 7.31 x 10HO (2-tailed).
If we also remove the Princeton data, which
comprise nearly half of the reported experiments, we get a grand total z = 5.480,p<4.25x 10 “8
(2-tailed).
Thus,
after removing the two largest contributors to the database, we are left with a
fairly impressive overall result: Odds against chance of about
1 in 23,000,000.
In addition, we find
that 39%
(11/28)
of the experimenters obtained
overall
2-tailed
significance and 68%
(19/28)
obtained positive z scores.
Test for homogeneity of effect she
Do different experimenters tend to observe about the same
effects
in
their experiments?
Or are there some individuals who consistently obtain
significant
results and others do not?
In the present context, to test for homogeneity of effect size among
B-16
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5
Approved For Release
different experimenters, we believe
it makes more sense to test the individual z scores obtained in
each experiment rather than use effect sizes such as d, d’,
r, or so on, as discussed by Rosenthal
(1984) and others.
The reason
is the following: Effect size may be defined as
significance test =
[effect size] x
[size of study]
where “significance test” can be a z, t, r, chi-square, or any other statistical test.
In the studies we
found in the literature,
it
is clear that if the effect size were constant regardless of the size of the
study (say, N trials), we should be observing enormous z scores when N is even moderately large.
For example,
if an investigator ran a study with N = 100 and obtained a z score = 2.0, this would
imply that the effect size (defined as r = 2Ap = zA/n" for a binary RNG) would be
r - 2.0/V100 = 2. 0/10.0 =
.2.
If this effect size were constant, then
if we ran the same experiment
again but with N = 10000, the z score for this experiment would be z = (2Ap) VTOOOO = .2( 100) =
20.0. Z scores of this magnitude are simply not reported in individual experiments, thus our effect
size
is almost certainly n-dependent.
Indeed, this phenomenon has been observed repeatedly in a
variety of experiments and has been called a goal-directed effect (e.g. Kennedy, 1978; May, Radin
et al,
1985; Schmidt,
1974).
To take
the
effect
size n-dependence
into account, we must multiply the
effect
size by
a
function of the size of the study, which brings us back to a significance test, as noted above.
For
the sake of convenience, we can use the z score calculated for each experiment.
To see whether
different experimenters reported about the same magnitude z scores, we performed an analysis of
variance; the results are shown in Table
3 (on the next page).
It
is
clear from the
results
of the ANOVA
that
different experimenters do indeed
obtain
different mean z scores, although with 25% (7/28) of the principal investigators reporting mean z
scores greater than 2 or less than -2,
it
is not the case that only one or two experimenters have
obtained large mean z scores.
B-17
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5
^rovSSyor Release O^CCAS^fFFC®"00789^02200390001 '5
Table 3: Results of one-way analysis of variance
Grand mean
N
MEAN
Z
SD
SE
297
0.5979
1.5823
0.0918
Person
N
MEAN Z SD
SE
Andre
1.2065
1.9091
0.9546
Bierman
2.7570
1.3863
0.9803
Braud
1.8797
0.9373
0.4687
Broughten
-0.2347
0.3048
0.1524
Debes
0.1260
1.8205
0.6437
Dunne
144
0.3386
1.1842
0.0987
Edge
0.1166
2.0067
0.6346
Giesler
0.7778
0.8011
0.2313
Heseltine
-0.0885
1.7124
0.3928
Hill
2.9498
Honorton
0.4071
1.1328
0,3028
Houtkooper
0.4906
1.4944
0.7472
Jungerman
2.3322
Kelly
2.3799
0.3015
0.2132
Matas
0.3625
2.9522
2.0875
May
-2.3841
Millar
-0.6187
1.6406
1.1601
Morris
0.8206
1.0562
0.4723
Morrison
0.7746
0.4926
0.2844
Palmer
1.7500
Pantas
0.7625
2.4453
1.2226
Radin
2.1712
0.8822
0.4411
Randall
-0.0120
1.1753
0,4798
Schechter
-0,7496
4.2411
2.9989
Schmeidler
-1.2728
Schmidt
'
2.4144
2.0341
0.3714
Shafer
-1.0178
1.1158
0.7890
Winnett
-0.0396
0.5795
0.2592
SOURCE
ss
df
MS
F
p
person
197.9629
7.3320
3.631
2.78 x 10
error
543.1467
269
2.0191
B-18
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5