Approved For Release PACER SSIFPEDS-007s9Ro0zz0ess000 Fs: 1 Figure 4. Method of estimating filedrawer size (see text) (4b) Now that we had a full description of curves 1 and 2, we assumed that the area labeled “b” in Figure 4a was the number of observed studies with |z| < 1.65 (188 - 76 = 112), that area “c +d” was composed of 76 observed studies with |z| = 1.65, and that the total area “a +b + ¢ + d” was calculated at 283 studies §. Doing the subtraction 283 - 112 - 76 = 95, we estimate 95 unreported or unretrieved nonsignficant studies in the actual filedrawer. We believe that this number is a more realistic estimate than the 4700 studies determined by equation (2). In fact, 95 studies would require each of 10 parapsychology laboratories to have filed only about 0.6 studies per year over the 15 year survey period (as opposed to 2.6 per month, as 4700 studies would require). Now if we combine the 188 observed survey studies with 95 new, nonsignificant z scores (generated by Monte Carlo technique with z chosen at random from a normal distribution, and bounded between 10-25 and 11.64), we find of the 283 resulting studies, mean z = .462, overall z= 7,768, andp< 8.03x 10715. Again applying formula (2) to the new values (for the sake of curiosity), we find X = 4078 additional nonsignificant studies needed to bring this overall p value down to p = .05, 2-tailed. Finally, combining all survey, newly estimated, and Princeton studies (188+144), we find that for the 425 total studies the mean z = .420, overall z = 8.684, and p < 3.9 x 1078. Applying formula (2), we find we would need 7778 additional nonsignificant studies in the filedrawer. Thus, from several different perspectives, it seems that the filedrawer issue is not as serious a problem as many have thought. 5. This calculation was based on the curve-fitted standard deviations for the two Gaussian curves and the observed number of studies in areas b and c + d. B-13 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200390001-5
NpprovediFeatRelease CPRFE PS SSOTEPE BS 00789 R002200890001 -5 Incidentally, testing the standard deviation of the z scores observed in these 425 studies (sd = 1.414) against the expected variance of 1.0 for a normal, unperturbed z distribution, results in a chi-square value of 853.7 (424 df), fora p< 5.9 x 10-%. Table 1 (below) and 7 (at end of the paper) summarize these findings. Table 1. Summary of z score analyses variance test , against o=1 Source of studies 22's 2 6 reference (N) 3 VN p (2-tail) d x p (2-tail) - 47 Survey 188 0.738 10.114 4.9x 10-4 1.739 568.5 4.9x 10 144 0.339 4.063 49x 1075 1.184 201.9 0.001 Estimated (simulated) 95 -0.084 -0.820 0.412 0.661 41.5 0.51 filedrawer Combined 425 0.420 8.684 3.9x 10778 1.414 853.7 5.9x 10°" 4. Experiments are not replicable Occasional significant effects may be impressive, but the existence of the claimed anomaly cannot be established on the basis of results reported by only a few individuals.6 The same effect must be replicated by many others. Is it true, as Kurtz (1980) claims, that The basic problem ... is the Jack of replicability by other experimenters. Apparently, some experimenters —- a relative few -- are able to get similar results, but most are unable to do so. (Italics in the original, p.12) In fact, of the 332 experiments we considered, 78.6% failed to reach Significant levels. It is hardly surprising, then, that on the basis of examining individual experiments it is easy to reach the conclusion that the effect is elusive and non-replicable. At this failure rate, nearly 4 out of 5 experiments will fail to reject the null hypothesis. (Of course, if just chance were operating, 19 out of 20 experiments would fail to reject the null hypothesis.) 6. Actually, compared to experimental psychology, experimental parapsychology is in much better shape as far as replication rates go. Honorton (1975), for instance, describes a study by Bozarth and Roberts (1972), who, ina survey of 1334 articles from psychology journals, found only eight articles involving replications of previously published work. In this present meta-analysis alone, parapsychology is a factor of 40 ahead of psychology. B-14 Approved For Release HNGLA OEE oo7s9R002200390001-5