MApproved For Release NICER SST RPEYS.00789R002200390001-5 2. Basic statistical assumptions are violated This criticism incorporates such problems as the improper application of statistics to a particular experimental! design, violation of assumptions of independence, performing multiple analyses on the same data, and so on. In this meta-analysis, one of the reasons we only considered binary generators was to simplify the statistical assumptions to the point where we could avoid many such problems. Another reason was to avoid the “apples vs. oranges” comparison problem we mentioned earlier. Because we were interested only in RNG experiments that reported (or where we could calculate) the number of hits and trials, we were in fact comparing apples only with apples (actually bits with bits). While it is true that there were many different psychological and physiological conditions involved in these experiments, as well as human and non-human subjects, the underlying question we asked was the same for each experiment: What was the behavior of the RNG as compared to the pre-specified direction of effort defined in the experimental task? The statistics in these RNG experiments are described by the well understood binomial distribution, and the central limit theorem allows us to use the normal approximation to further simplify the statistical treatment for the range of trials observed in the data (200 to 2 million trials in a single experiment). Violation of the assumption of independence can be the downfall of an otherwise tightly controlled experiment. In the present case, however, the random events are based on sources that are quantum-mechanical (QM) in nature —- radioactive decay of alpha, beta, or gamma particles, or electronic noise from various semiconductor devices such as tunnel diodes. QM theory states that random numbers based on QM events are in principle indeterminant and therefore independent of each other, provided that the RNG device is properly designed and constructed.‘ In this meta-analysis, under the null hypothesis of no psi effect we can assume independence of random bits. Note that the assumption of independence among bits does not override proper concern about whether the RNGs used in the experiments produced bits with equal probabilities. It is entirely possible, for example, to produce bits that are completely independent, but with p( 1) = .6 and p(0) = .4. This is addressed in point 5 below. 3. Only significant studies are published — the Filedrawer problem The filedrawer problem, in which only significant studies are reported and the nonsignificant studies languish in filedrawers, will inflate the results of a meta-analysis because there will be too many small p values (or equivalently, to many large z scores). To address this problem, we followed a procedure proposed by Rosenthal (1984, p. 108), in which the average z score for all combined studies is applied to the formula: 4. Note that some of the diodes used in noise-based RNGs are not QM in nature. RNGs thal use avalanche diodes, for example, derive their noise from fluctations in charge carrier multiplication, which can be described by classical electromagnetic theory. UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200390001-5
Approved For Release APRKCPA SERRE BY-00789Roaza00ca000 t6 1 52 X= K[KZ °- 2.72] (1) 2.72 where K is the number of studies combined, Z is the mean Z obtained for the K studies, and X is the number of new, filed, or unretrieved studies averaging null results required to bring the new overall p level to a designated level. The value 2.72 in equation (1) is the square of 1.65, the z value for p = .05 (the p level that Rosenthal uses). To make our filedrawer estimate more conservative, we chose a 2-tailed p = .05, z= 1.96. Thus the formula we used was K[KZ ?- 3.92] » rs (2) 3.92 We shall consider the Princeton studies separately from the rest of the survey because we have good reason to believe that all of the Princeton data was, in fact, published, thus their data has no filedrawer problem. [(Publishing all data is a part of the Princeton Laboratory’s philosophy (Jahn, 1982)]. In the 188 survey experiments, the mean z score = 0.738. A mean z of this value over 188 experiments produces an overall z = 10.114, for a 2-tailed p < 4.9 x 107@4 (see Table 1). Note that this method of estimating the overall probability is more accurate than determining the binomial probability of 71 successes out of 188 samples at p < .05, as described earlier in this paper. Applying Z = .738 and K = 188 to formula (2) results in X = 4723. This means that 4723 additional studies averaging null results would have to be filed away in researchers’ filedrawers to bring the overall z score down to a 2-tailed nonsignificant level. According to Rosenthal (1984), the number X has different meanings depending on the research context. In some areas of research (say genetic engineering), perhaps 10 or 12 unpublished or unretrieved studies might be considered reasonable. In other areas (say child development), perhaps 200 to 500 filedrawer studies might be a reasonable estimate. Rosenthal (1984, p.110) proposes the following general guideline: “Perhaps we could regard as robust to the file drawer problem any combined results for which the tolerance level (X) reaches 5 K + 10.” Thus -- not counting the Princeton data -- since X is more than 25 times larger than the observed number of studies, we could state, based on Rosenthal’s guideline, that the observed effect is robust. Indeed, for this many unpublished or unretrieved studies to exist it would have required each of 10 parapsychology laboratories to have continuously produced nonsignificant studies at the rate of 2.6 per month over the 15 years surveyed. This is an unlikely scenario given the limited number of researchers performing these experiments over the years and the time and effort typically required to perform a single study. If we apply the same procedure to the Princeton data of 144 studies, we find the mean z = .339, overall z = 4.063, and p < 4.85 x 10-5. Plugging these values into formula 2, we find we would need X = 476 additional unpublished or unretrieved studies averaging null results. But as previously BY UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200390001-5
Miproved FGF Release IGE pdt SBILRPEB-00789R002200390001-5 mentioned (vide supra), the Princeton lab has claimed that they have no unpublished or filed studies, thus this estimate of the filedrawer size is purely academic. Another way of looking at the Princeton data is shown in Figure 1. This shows a histogram of the absolute value of the observed z scores in light-colored bars, and a best Gaussian fit in dark bars. As is apparent from the figure, the observed z scores are a good Gaussian fit, but the standard deviation of the fit is not 1.0, as one would expect under the null hypothesis of z scores chosen at random from a normal distribution, but rather the best fit Gaussian standard deviation is 1.17. A variance test between these two variances results in z = 2.90, p < .004 (2-tailed). Thus the distribution of z scores is significantly altered from that expected by chance. This interesting effect is discussed in more detail by Jahn, Nelson and Dunne (1985) and May, Radin, Hubbard, Humphrey, and Utts (1985). Figure 1. |z| score distribution for Princeton data | | Gaussian fit Data (144) number of studies | | \ + i i { .075 -675 1.275 1.875 2.475 3.075 3.675 4.275 |z| scores Estimating the actual number of filedrawer studies What if we wished to make an estimate of the actual size of the filedrawer for the rest of the survey data? We would not be surprised to learn, for example, that there are indeed some unpublished or unretrieved nonsignificant studies we may have missed in our survey. To do this, we postulated what a z~score distribution might look like if there were a filedrawer problem. Figure 2 (next page) shows a histogram of the absolute value of B-10 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200390001-5