Approved For Release EO OLAR SGM BI6-00789R0022004¢ 6001-3. hypothetical z scores with a filedrawer problem. Notice the discontinuity at the magic number z = 1.65 (p < .05), which is what one would expect if nonsignificant studies remained unpublished. Figure 2. |z| score distribution with filedrawer problem 25 20 15 number of studies § 0 075 .675 1.275 1.875 2.475 3.075 3.675 4.275 » |z| scores In Figure 3, we plot a histogram of z scores from the 188 survey studies. We also plot a double Gaussian curve, assuming that the observed z-score curve is actually the sum of two Gaussians, The resulting two-Gaussian curve is a good fit to the data; in fact, the sum of two Gaussians is a significantly better fit than a single Gaussian curve (zaitt = 1.718, p < .04, 1-tailed, determined by transforming chi-square goodness-of-fit values for one vs. two Gaussian fits into z scores, and comparing those two z scores.) B-ll UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
Meta-Analysis Part 1 Approved For Release 4d ¢W Glo: S54 RHEB)-00789R002200400001-3 pa ek Figure 3. |z| score distribution for 186 survey experiments 25 Gaussian fit Data (186) number of studies 075 -675 1.275 1.875 2.475 3.075 3.675 4.275 |z| scores In Figure 4, we show how we estimated the actual number of filedrawer studies. We have assumed that the observed curve (Figure 3 above) is the sum of two Gaussians (Figure 4b), shown as two separate curves (1 and 2) in Figure 4a. We obtained estimates of the amplitude and variance of these curves by allowing a computer-based curve-fitting routine the freedom to vary the amplitude and variance of each curve so the obtained fit to the curve shown in Figure 3 would be the best possible. Under these conditions, the standard deviation (sd) of curve 1 was found to be 0.9256 and sd of curve 2 was 2.024, B-12 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
Approved For Release WOW GAA SGMFHEH6-00789R002200400001-3 Figure 4. Method of estimating filedrawer size (see text) (4b) y Now that we had a full description of curves 1 and 2, we assumed that the area labeled “b” in Figure 4a was the number of observed studies with jz] < 1.65 (188 - 76 = 112), that area “c + d” was composed of 76 observed studies with |z| = 1.65, and that the total area “at+b+c+d” was calculated at 283 studies §. Doing the subtraction 283 - 112 - 76 = 95, we estimate 95 unreported or unretrieved nonsignficant studies in the actual filedrawer. We believe that this number is a more realistic estimate than the 4700 studies determined by equation (2). In fact, 95 studies would require each of 10 parapsychology laboratories to have filed only about 0.6 studies per year over the 15 year survey period (as opposed to 2.6 per month, as 4700 studies would require). *” Now if we combine the 188 observed survey studies with 95 new, nonsignificant z scores (generated by Monte Carlo technique with z chosen at random from a normal distribution, and bounded between 10-25 and +£1.64), we find of the 283 resulting studies, mean z = .462, overall z = 7.768, andp< 8.03x 1075. Again applying formula (2) to the new values (for the sake of curiosity), we find X = 4078 additional nonsignificant studies needed to bring this overall p value down to p = .05, 2-tailed. Finally, combining all survey, newly estimated, and Princeton studies (188+144), we find that for the 425 total studies the mean z = .420, overall z = 8.684, and p < 3.9 x 1078. Applying formula (2), we find we would need 7778 additional nonsignificant studies in the filedrawer. Thus, from several different perspectives, it seems that the filedrawer issue is not as serious a problem as many have thought. 5. This calculation was based on the curve-fitted standard deviations for the two Gaussian curves and the observed number of studies in areas b and c + d. B-13 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
MesphaTett FUt Release BpPGICAOA SSAFIPHDS-00789R002200400001-3 Incidentally, testing the standard deviation of the z scores observed in these 425 studies (sd = 1.414) against the expected variance of 1.0 for a normal, unperturbed z distribution, results in a chi-square value of 853.7 (424 df), forap<5.9x 10-%. Table 1 (below) and 7 (at end of the paper) summarize these findings. Table 1. Summary of z score analyses variance test . y2’s against o = 1 Source of studies z score 2 reference N VN ¥p (2-tail) sd x p (2-tail) (N) -47 188 0.738 10.114 49x 107*4 1.739 568.5 4.9x 10 Princeton 144 0.339 4.063 49x 107° 1.184 201.9 0.001 Estimated (simulated) 95 -0.084 -0.820 0.412 filedrawer 425 0.420 8.684 3.9x 10778 | 1.414 853.7 5.9x10 4. Experiments are not replicable Occasional significant effects may be impressive, but the existence of the claimed anomaly cannot be established on the basis of results reported by only a few individuals. The same effect must be replicated by many others. Is it true, as Kurtz (1980) claims, that The basic problem ... is the lack of replicability by other experimenters. Apparently, some experimenters -- a relative few -- are able to get similar results, but most are unable to do so. (Italics in the original, p.12) In fact, of the 332 experiments we considered, 78.6% failed to reach significant levels. It is hardly surprising, then, that on the basis of examining individual experiments it is easy to reach the conclusion that the effect is elusive and non-replicable. At this failure rate, nearly 4 out of 5 experiments will fail to reject the null hypothesis. (Of course, if just chance were operating, 19 out of 20 experiments would fail to reject the null hypothesis.) 6. Actually, compared to experimental psychology, experimental parapsychology is in much better shape as far as replication rates go. Honorton (1975), for instance, describes a study by Bozarth and Roberts (1972), who, in a survey of 1334 articles from psychology journals, found only eight articles involving replications of previously published work. In this present meta-analysis alone, parapsychology is a factor of 40 ahead of psychology. B-14 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3