IFIED Meta-Analysis Part 1 Approved For Release YN Ae EEO oreaR002200400001-3 In summary, of the 332 experiments we considered (188 from the survey and 144 from Princeton), 71 were reported significant at p < .05, 2-tailed, for an overall binomial probability of p< 5.4 x 107. ADDRESSING CRITICISMS OF THE DATA Taken as prima facie evidence, one might think that this body of published data provides indisputable evidence that an anomaly exists. But there are numerous reasons why the data may be suspect. The main criticisms (Akers, 1984; Hansel, 1980; Hyman, 1985; Kurtz, 1981) include . Results are due to chance . Basic statistical assumptions are violated . Only significant studies are published . Experiments are not replicable . RNGs are nonrandom . Poorer studies are included with better studies DAwn bh wh Let us consider each of these six steps as successive filters for the reliability of the data. If each criticism can be satisfactorially refuted or countered, then a persuasive case for an anomalous effect can be made. 1. Results are due to chance In any one experiment we cannot establish the reality of a phenomenon, regardless of the significance level, unless strong theoretical predictions have preceeded the experiments. For example, the recent experiments suggesting that Bell’s inequality is violated (e.g. Aspect, Dalibard, and Roger, 1982; Aspect, Grangier and Roger, 1982) have been widely accepted within the physics community on the basis of only a few empirical studies despite its profound implications on our view of the nature of reality (cf. d’Espagnat, 1979; Mermin, 1985; Rohrlich, 1983). Parapsychology, however, has had the disadvantage of not having a firm theoretical base on which to stand. Thus the nature of the claim (any claimed psi effect) understandably requires extremely persuasive evidence. One wonders how statistically strong an effect must be to bring about a consensual agreement within the scientific community that a psi effect on RNGs is real. Would p < 10-4 be sufficient? If this figure were revised to take into account all of the criticisms noted above, and the end result were say, 1075, would that be sufficient? Clearly an overall p = .1 would not satisfy anyone, so there is a decision curve related to this question. This curve is probably different according to individual prejudices and predilictions, but the resolution of this question is beyond the scope of the present paper. Note that if an anomaly did exist, it would not necessarily imply that psi was the mediating factor. Such an anomaly may, for example, reveal some heretofore unknown statistical peculiarities about random numbers. UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
MaSproved For Release RAY CIOA:S-54-RHEBS-00789R002200400001-3 2. Basic statistical assumptions are violated This criticism incorporates such problems as the improper application of statistics to a particular experimental! design, violation of assumptions of independence, performing multiple analyses on the same data, and so on. In this meta-analysis, one of the reasons we only considered binary generators was to simplify the statistical assumptions to the point where we could avoid many such problems. Another reason was to avoid the “apples vs. oranges” comparison problem we mentioned earlier. Because we were interested only in RNG experiments that reported (or where we could calculate) the number of hits and trials, we were in fact comparing apples only with apples (actually bits with bits). While it is true that there were many different psychological and physiological conditions involved in these experiments, as well as human and non—hurhan subjects, the underlying question we asked was the same for each experiment: What was the behavior of the RNG as compared to the pre-specified direction of effort defined in the experimental task? The statistics in these RNG experiments are described by the well understood binomial distribution, and the central] limit theorem allows us to use the normal approximation to further simplify the statistical treatment for the range of trials observed in the data‘ (200 to 2 million trials in a single experiment). ’ Violation of the assumption of independence can be the downfall of an otherwise tightly controlled experiment. In the present case, however, the random events are based on sources that are quantum—mechanical (QM) in nature -- radioactive decay of alpha, beta, or gamma particles, or electronic noise from various semiconductor devices such as tunnel diodes. QM theory states that random numbers based on QM events are in principle indeterminant and therefore independent of each other, provided that the RNG device is properly designed and constructed.‘ In this meta-analysis, under the null hypothesis of no psi effect we can assume independence of random bits. Note that the assumption of independence among bits does not override proper concern about whether the RNGs used in the experiments produced bits with equal probabilities. It is entirely possible, for example, to produce bits that are completely independent, but with p(1) = .6 and p(0) = .4. This is addressed in point 5 below. 3. Only significant studies are published — the Filedrawer problem The filedrawer problem, in which only significant studies are reported and the nonsignificant studies languish in filedrawers, will inflate the results of a meta-analysis because there will be too many small p values (or equivalently, to many large z scores). To address this problem, we followed a procedure proposed by Rosenthal (1984, p. 108), in which the average z score for all combined studies is applied to the formula: 4, Note that some of the diodes used in noise-based RNGs are not QM in nature. RNGs that use avalanche . diodes, for example, derive their noise from fluctuations in charge carrier multiplication, which can be described by classical electromagnetic theory. B-8 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
Approved For Release babWhatoA SSUES oo7sorobsza03duides: K[KZ ?- 2.72] (1) 2.72 where K is the number of studies combined, Z is the mean Z obtained for the K studies, and X is the number of new, filed, or unretrieved studies averaging null results required to bring the new overall p level to a designated level. The value 2.72 in equation (1) is the square of 1.65, the z value for p = .05 (the p level that Rosenthal uses). To make our filedrawer estimate more conservative, we chose a 2-tailed p = .05, z= 1.96. Thus the formula we used was K[KZ ?- 3,92] xX = ———_*— (2) 3.92 / 4 We shall consider the Princeton studies separately from the rest of the survey because we have good reason to believe that all of the Princeton data was, in fact, published, thus their data has no filedrawer problem. [(Publishing all data is a part of the Princeton Laboratory’s philosophy (Jahn, 1982)]. In the 188 survey experiments, the mean z score = 0.738. A mean z of this value over 188 experiments produces an overall z = 10.114, for a 2-tailed p < 4.9 x 10-24 (See Table 1). Note that this method of estimating the overall probability is more accurate than determining the binomial probability of 71 successes out of 188 samples at p < .05, as described earlier in this paper. Applying Z = .738 and K = 188 to formula (2) results in X = 4723. This means that 4723 additional studies averaging null results would have to be filed away in researchers’ filedrawers to bring the overall z score down to a 2-tailed nonsignificant level. . According to Rosenthal (1984), the number X has different meanings depending on the research context. In some areas of research (say genetic engineering), perhaps 10 or 12 unpublished”or unretrieved studies might be considered reasonable. In other areas (say child development), perhaps 200 to 500 filedrawer studies might be a reasonable estimate. Rosenthal (1984, p.110) proposes the following general guideline: “Perhaps we could regard as robust to the file drawer problem any combined results for which the tolerance level (X) reaches 5 K + 10.” Thus -- not counting the Princeton data -- since X is more than 25 times larger than the observed number of studies, we could state, based on Rosenthal’s guideline, that the observed effect is robust. Indeed, for this many unpublished or unretrieved studies to exist it would have required each of 10 parapsychology laboratories to have continuously produced nonsignificant studies at the rate of 2.6 per month over the 15 years surveyed. This is an unlikely scenario given the limited number of researchers performing these experiments over the years and the time and effort typically required to perform a single study. If we apply the same procedure to the Princeton data of 144 studies, we find the mean z = .339, overall z = 4.063, and p < 4.85 x 1075, Plugging these values into formula 2, we find we would need X = 476 additional unpublished or unretrieved studies averaging null results. But as previously B~9 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
Mpproved For Release bdbWheko SSA RAGHO-00789R002200400001-3 mentioned (vide supra), the Princeton lab has claimed that they have no unpublished or filed studies, thus this estimate of the filedrawer size is purely academic. Another way of looking at the Princeton data is shown in Figure 1. This shows a histogram of the absolute value of the observed z scores in light-colored bars, and a best Gaussian fit in dark bars. As is apparent from the figure, the observed z scores are a good Gaussian fit, but the standard deviation of the fit is not 1.0, as one would expect under the null hypothesis of z scores chosen at random from a normal distribution, but rather the best fit Gaussian standard deviation is 1.17. A variance test between these two variances results in z = 2.90, p < .004 (2-tailed). Thus the distribution of z scores is significantly altered from that expected by chance. This interesting effect is discussed in more detail by Jahn, Nelson and Dunne (1985) and May, Radin, Hubbard, Humphrey, and Utts (1985). ‘ Figure 1. |z| score distribution for Princeton data number of studies 675 1.27 1.875 2.475 3.075 3.675 4.275 |z| scores Estimating the actual number of filedrawer studies | What if we wished to make an estimate of the actual size of the filedrawer for the rest of the survey data? We would not be surprised to learn, for example, that there are indeed some unpublished or unretrieved nonsignificant studies we may have missed in our survey. To do this, we postulated what a z-score distribution might look like if there were a filedrawer problem. Figure 2 (next page) shows a histogram of the absolute value of B-10 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3