Meta-Analysis Part 1 Approved For Release 2b BNGba4 S'S HolEd).00789R002200400001-3 weight. If the criterion was not met, the weight assigned for that factor would be zero (0). The sum of the individual weights would be the overall weight per experiment, and the final overall weighted z score is then calculated as Wi Zi Vv Dwi? Thus the minimum weight per experiment would be 0 if there were no mention of control tests, no description indicating that data collection was protected in some way, no evidence that statistical tests were pre-planned, insufficient report on who the subjects were, and no report of results. The maximum weight would be 125 (sum of threé control weights, three data integrity weights, three Statistical integrity weights, use of ordinary subjects, and full report of data). Weighted Z = (5) Weighting the filedrawer estimate We propose to weight our estimated 95 nonsignificant filedrawer studies with the average weight found in the rest of the studies. This proposal has a potential criticism: however. Our means of estimating the filedrawer size depends on the observed z score distribution: Since the individual z scores depend on the weighting factors (which were in effect all 1’s in the analysis reported in this paper), the unweighted filedrawer estimate may be smaller than a similar estimate made with weighted z scores, thus inflating the final results. In response to this criticism, we would point out that the quality analysis is actually orthogonal to the filedrawer estimate because the actual magnitude of a z score does not change with our quality analysis, instead the importance of the z score is affected, and the importance of a z score is not considered in our filedrawer estimation method, only in the final estimate of overall Significance. In addition, by adding a group of nonsignificant studies (the filedrawer estimate by definition is composed of nonsignificant studies) into a pool of z scores that have already been weighted according to quality, we are in effect creating an u/tra-conservative test. A case could be made, for instance, on why a filedrawer estimate should not be added into a quality-weighted analysis at all, but to take the conservative approach given the nature of the claim, we will pool the 95 estimated Studies along with the quality-weighted z scores. Defining experiments in the Quality Analysis Although adequate for a first-pass analysis, the method of selecting experiments described above would be less than perfect for a quality-weighted analysis. The main objection that could be raised is that the decision on what constitutes the subjects’ “direction of effort” is dependent on the reviewer’s interpretation of the experimental procedure. In many articles, we took educated guesses to decide what were the actual conditions, what were the subjects’ intentions, did the authors in fact predict in advance the outcome, and so on. B-22 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
Meta-Analysis Part 1 Approved For Release AINGLAA GSERIED 007s9R002200400001 3 To address this problem in Part 2 of this meta-analysis, we will actually be performing two separate meta-analyses. The first will take into account the minimum number of experiments that we decide is a reasonable partitioning, and the second meta-analysis will be for the maximum number of experiments. The two end results will be compared, and the more conservative of the two will be used as the overall result. Deciding on a range of possible experiments allows us to form an “uncertainty” factor for each reference. If a reference’s maximum-minimum experiment range is large as compared to the average observed range, we must consider that the quality of that reference, at least for our purposes, is poor. We plan on presenting a breakdown of each reference’s uncertainly in the Part 2 meta-analysis to judge how clear each reference was in this study. ) - Example of reference source quality analysis In Table 5 we present an example of a preliminary quality analysis applied to the source of reference. We assigned arbitrary weights according to our perception of the quality of average papers published in each parapsychological reference source (not counting the Princeton data). Then, after making guesses for these weights, we calculated a combined z,score contributed by each journal and compared it to a weighted z score according to equation (5). AAs seen in Table 5, the original combined z score dropped by 4 orders of magnitude in significance, but the weighted z score is still quite significant. We expect that the wider range of quality weights, as we have proposed above, will make a larger difference in a weighted analysis, but it would appear that most of the reports would have to be extremely poor in quality to nullify the overall p value, Table 5. Exploratory quality analysis of reference sources Refer&nce Studies Overall z p(2-tail) Assigned weight Journal of Parapsychology . 3.30 x 107° Proceedings of the PA . 4.76 x 107’ Research in Parapsychology . 5.08 x 107° Journal of the ASPR ; 4.04x 1075 European Journal of Parapsychology . 0.002 Journal of the SPR . 0.091 Combined unweighted result = 27 p<9.45x 107% Combined weighted result= 9.53 p< 1.60x 1072? B-23 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
Meta-Analysis Part 1 Approved For Release AG bA § GIRSEGD-00789R002200400001-3 Example of chronological analysis In Table 6 we show an analysis of variance of the 297 detailed experiments grouped according to year of publication. Table 6. Chronological analysis of variance SOURCE: grand mean year N MEAN Z ‘ SD SE - 297 0.5979 1.5823 — 0.0918 ‘ SOURCE: year year N MEAN Z SD SE 1970 5 0.8247 3.0969 1.3850 1971 6 0.6292 2.3180 0.9463 1972 9 1.3565 1.4253 0.4751 1973 6 4.1239 1.4665 0.5987, ~ 1974 10 1.1539 1.9879 0.6286 ’ 1975 9 1.5804 2.1841 0.7280 1976 17 0.7366 1.5784 0.3828 1977 23 0.5695 1,8333 0.3823 1978 9 -0.2520 1.1482 0.3827 1979 7 0.6012 1,2325 0.4658 1980 5 -1.1411 1.3480 0.6029 1981 7 2.3437 0.7836 0.2962 1982 164 0.3098 1.2595 0.0983 1983 1 1.7500 4984 19 0.8492 1.0779 0.2473 SOURCE SS df MS F p year 151.2565 14 10.8040 5.165 1.14 x 1078 error 589.8531 282 2.0917 This ANOVA shows that mean z scores differ significantly from year to year. We then looked for trends in z scores by performing a correlation between year and mean z. Results showed that = -0.205, t(13) = -0.756, p = 0.463, i.e. there was no significant correlation between year of publication and mean z score observed for that year. B-24 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
Meta-Analysis Part 1 Approved For Release HMGbA §S1AES)-00789R002200400001-3 A planned quality vs. z score correlational study Once we perform the quality analysis and have a list of raw z scores and associated quality weights, we plan on performing a correlation between these pairs of numbers. If the correlation is significantly negative, it would suggest that the better the quality of a study, the lower the z score. This would be in accordance with what some critics have claimed, namely that “there is a strong tendency for the rate of success to increase with the number of obvious defects” (Hyman, 1983, p.23). Ifa significant positive correlation is seen, however, this criticism can be refuted. CONCLUSION ' In an initial meta-analysis of psi experiments involving binary RNGs, we have identified 332 experiments published over the years 1969-1984 in 56 references. Based on an analysis of 188 of these experiments reported in parapsychological journals, we estimated the actual number of nonsignificant, unreported or unretrieved experiments to be 95. We found a total of 98 reported control studies in 14 of these references. A summary of the meta-analytic results is shown in Table 7 (on the following page). ‘ In agreement with a hypothesis of a “psi effect” on RNGs, the combined data indicate that, in the aggregate, the experimental conditions resulted in anomalous statistical behavior of the RNG in the direction of effort specified by the task, and the control conditions resulted in expected binomial statistics for both mean z scores and standard deviations. The combined data shows an interesting effect on the distribution of z scores. We find that in the experimental condition the mean z score has been increased significantly from chance expectation,. which in the present context is in accord with the underlying hypothesis that the z score will shift according to the direction of the subject’s effort. In the control condition we find the z mean shifted slightly, but not significantly so. We also find that the standard deviation of the combined distribution of experimental z scores has become significantly fatter than chance expectation, and that the combined control standard deviation is as expected. Both of these effects -~ a shifting of the mean and fattening of the standard deviation, are accounted for in a model discussed by May, Radin et al (1985). Part 2 of this study will report on a quality-weighted analysis of this same data. By weighting each study according to a semi-objective quality assessment scale, we will address the major criticisms of such experiments in a quantified way, and the overall experimental vs. control result will provide a basis for discussion on whether or not this anomaly is, in fact, real. We urge readers to comment on and criticize the method described here, and especially on the proposed weighting criteria presented in the Quality Analysis section above. We plan to gain a consensus opinion among informed scientists on what constitutes an agreeable, conservative weighting scheme before we perform the quality analysis. In this way, the combined results observed in the weighted data will be less subject to post hoc debate over the adequacy of the B-25 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3