M^TMSf^dP^elease ©NStAS84FM5D-00789R002200390001 -5
mentioned
(vide supra), the Princeton lab has claimed that they have no unpublished
or
filed
studies, thus this estimate of the filedrawer size
is purely academic.
Another way of looking at the Princeton data
is shown in Figure
1.
This shows a histogram of
the absolute value of the observed z scores in light-colored bars, and a best Gaussian
fit in dark
bars.
As
is apparent from the
figure, the observed
z scores are
a good Gaussian
fit,
but the
standard deviation of the
fit
is not
1.0, as one would expect under the null hypothesis of z scores
chosen at random from a normal distribution, but rather the best fit Gaussian standard deviation
is
1.17. A variance test between these two variances results in z = 2.90, p < .004 (2-tailed).
Thus
the distribution of z scores
is significantly altered from that expected by chance.
This interesting
effect
is discussed in more detail by Jahn, Nelson and Dunne (1985) and May, Radin, Hubbard,
Humphrey, and Utts (1985).
Figure
1.
|z| score distribution for Princeton data
number
of
studies
Estimating the actual number offiledrawer studies
What if we wished to make an estimate of
the actual size of the filedrawer for the rest of the survey data? We would not be surprised to learn,
for example, that there are indeed some unpublished or unretrieved nonsignificant studies we may
have missed in our survey. To do this, we postulated what a z-score distribution might look like if
there were a filedrawer problem.
Figure 2 (next page) shows a histogram of the absolute value of
B-10
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5
Approved For Release y(^^|0j^:^^-^^-00789R0^O^38m^5
hypothetical z scores with a filedrawer problem.
Notice the discontinuity at the magic number z =
1.65
(p <
.05), which
is what one would expect
if nonsignificant studies remained unpublished.
Figure 2.
\z\ score distribution with filedrawer problem
number
of
studies
|z| scores
In Figure 3, we plot a histogram of z scores from the 188 survey studies. We also plot a double
Gaussian curve, assuming that the observed z-score curve is actually the sum of two Gaussians. The
resulting two-Gaussian curve
is a good
fit to the
data;
in
fact,
the sum
of two Gaussians
is
a
significantly better
fit than a single Gaussian curve
(zdiff = 1.718, p < .04,
1-tailed, determined by
transforming chi-square goodness-of-fit values for one
vs. two Gaussian
fits into
z
scores, and
comparing those two z scores.)
B-ll
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5
Approved' For Release y
'00789R002200390001 '5
Figure 3,
|z| score distribution for 186 survey experiments
number
of
studies
In Figure
4, we show how we estimated the
actual number of filedrawer
studies*
We have
assumed that the observed curve (Figure 3 above)
is the sum of two Gaussians (Figure 4b)
, shown
as two
separate curves
(1 and
2)
in
Figure
4a,
We obtained estimates
of the amplitude and
variance of these curves by allowing a computer-based curve-fitting routine the freedom to vary the
amplitude and variance of each curve so the obtained
fit to the curve shown in Figure 3 would be
the best possible.
Under these conditions, the standard deviation
(sd) of curve
1 was found to be
0*9256 and sd of curve 2 was 2.024,
B-12
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5