Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 374 J. UTTS There were two interesting comparisons that had been suggested by earlier work and were pre- planned in these experiments. The first was to compare results for trials with dynamic targets with those for static targets. In the 190 dynamic target sessions there were 77 direct hits (40%, h = 0.32) and for the static targets there were 45 hits in 165 trials (27%, A = 0.05), thus indicating that dynamic targets produced far more successful results. The second comparison of interest was whether or not the sender was a friend of the receiver. This was a choice the receiver could make. If he or she did not bring a friend, a lab member acted as sender. There were 211 trials with friends as senders (some of whom were also lab staff), result- ing in 76 direct hits (86%, h = 0.24). Four trials used no sender. The remaining 140 trials used nonfriend lab staff as senders and resulted in 46 direct hits (83%, A = 0.18). Thus, trials with friends as senders were slightly more successful than those without. Consonant with the definition of replication based on consistent effect sizes, it is informative to com- pare the autoganzfeld experiments with the direct hit studies in the previous data base. The overall success rates are extremely similar. The overall direct hit rate was 34.4% for the autoganzfeld stud- ies and was 38% for the comparable direct hit studies in the earlier meta-analysis. Rosenthal’s (1986) adjustment for flaws had placed a more con- servative estimate at 33%, very close to the observed 34.4% in the new studies. One limitation of this work is that the auto- ganzfeld studies, while conducted by eight experi- menters, all used the same equipment in the same laboratory. Unfortunately, the level of fund- ing available in parapsychology and the cost in time and equipment to conduct proper experiments make it difficult to amass large amounts of data across laboratories. Another autoganzfeld labora- tory is currently being constructed at the Univer- sity of Edinburgh in Scotland, so interlaboratory comparisons may be possible in the near future. Based on the effect size observed to date, large samples are needed to achieve reasonable power. If there is a constant effect across all trials, resulting in 33% direct hits when 25% are expected by chance, to achieve a one-tailed significance level of 0.05 with 95% probability would require 345 sessions. We end this section by returning to the aspirin and heart attack example in Section 3 and expand- ing a comparison noted by Atkinson, Atkinson, Smith and Bem (1990, page 237). Computing the equivalent of Cohen’s A for comparing obser- ved heartfaiproved FortRe ease 2060/08/08 groups results in Ah = 0.068. Thus, the effect size observed in the ganzfeld data base is triple the much publicized effect of aspirin on heart attacks. 6. OTHER META-ANALYSES IN PARAPSYCHOLOGY Four additional meta-analyses have been con- ducted in various areas of parapsychology since the original ganzfeld meta-analyses were reported. Three of the four analyses focused on evidence of psi abilities, while the fourth examined the rela- tionship between extroversion and psychic func- tioning. In this section, each of the four analyses will be briefly summarized. There are only a handful of English-language journals and proceedings in parapsychology, so retrieval of the relevant studies in each of the four cases was simple ;to accomplish by searching those sources in detail and by searching other bibliographic data bases for keywords. Each analysis included an overall summary, an analysis of the quality of the studies versus the size of the effect and a “file-drawer” analysis to deter- mine the possible number of unreported studies. Three of the four also contained comparisons across various conditions. . 6.1 Forced-Choice Precognition Experiments Honorton and Ferrari (1989) analyzed forced- choice experiments conducted from 1935 to 1987, in which the target material was randomly selected after the subject had attempted to predict what it would be. The time delay in selecting the target ranged from under a second to one year. Target material included items as diverse as ESP cards and automated random number generators. Two investigators, S. G. Soal and Walter J. Levy, were not included because some of their work has been suspected to be fraudulent. Overall Results. There were 309 studies re- ported by 62 senior authors, including more than 50,000 subjects and nearly two million individual trials. Honorton and Ferrari used z /V/n as the measure of effect size (ES) for each study, where n was the number of Bernoulli trials in the study. They reported a mean ES of 0.020, and a mean z-score of 0.65 over all studies. They also reported a combined z of 11.41, pp = 6.3 x 107°. Some 30% (92) of the studies were statistically significant at a = 0.05. The mean ES per investigator was 0.033, and the significant results were not due to just a few investigators. : Quality. Eight dichotomous quality measures : CHERDESELAd7 SOROUSTOdO TOGA 1B Possible
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION IN PARAPSYCHOLOGY 375 scores from zero for the lowest quality, to eight for the highest. They included features such as ade- quate randomization, preplanned analysis and au- tomated recording of the results. The correlation between study quality and effect size was 0.081, indicating a slight tendency for higher quality studies to be more successful, contrary to claims by critics that the opposite would be true. There was a clear relationship between quality and year of publication, presumably because over the years experimenters in parapsychology have responded to suggestions from critics for improving their methodology. File Drawer. Following Rosenthal (1984), the authors calculated the ‘‘fail-safe N”’ indicating the number of unreported studies that would have to be sitting in file drawers in order to negate the signifi- cant effect. They found N = 14,268, or a ratio of 46 unreported studies for each one reported. They also followed a suggestion by Dawes, Landman and Williams (1984) and computed the mean 2 for all studies with z > 1.65. If such studies were a ran- dom sample from the upper 5% tail of a N(0,1) distribution, the mean z would be 2.06. In this case it was 3.61. They concluded that selective reporting could not explain these results. Comparisons. Four variables were identified that appeared to have a systematic relationship to study outcome. The first was that the 25 studies using subjects selected on the basis of good past performance were more successful than the 223 using unselected subjects, with mean effect sizes of 0.051 and 0.008, respectively. Second, the 97 stud- ies testing subjects individually were more success- ful than the 105 studies that used group testing; mean effect sizes were 0.021 and 0.004, respec- tively. Timing of feedback was the third moderat- ing variable, but information was only available for 104 studies. The 15 studies that never told the subjects what the targets were had a mean effect size of —0.001. Feedback after each trial produced the best results, the mean ES for the 47 studies was 0.035. Feedback after each set of trials re- sulted in mean ES of 0.023 (21 studies), while delayed feedback (also 21 studies) yielded a mean ES of only 0.009. There is a clear ordering; as the gap between time of feedback and time of the actual guesses decreased, effect. sizes increased. The fourth variable was the time interval be- tween the subject’s guess and the actual target selection, available for 144 studies. The best results were for the 31 studies that generated targets less than a second after the guess (mean ES = 0.045), while the worst were for the seven studies that delayed target selection by at least a month (mean ES = 0.0R  FBVGE EST RE BAS E066 DSS trend, decreasing in order as the time interval increased from minutes to hours to days to weeks to months. 6.2 Attempts to Influence Random Physical Systems Radin and Nelson (1989) examined studies de- signed to test the hypothesis that “The statistical output of an electronic RNG [random number gen- erator] is correlated with observer intention in ac- cordance with prespecified instructions’ (page 1502). These experiments typically involve RNGs based on radioactive decay, electronic noise or pseu- dorandom number sequences seeded with true ran- dom sources. Usually the subject is instructed to try to influence the results of a string of binary trials by mental intention alone. A typical protocol would ask a subject to press a button (thus starting the collection of a fixed-length sequence of bits), and then try to influence the random source to produce more zeroes or more ones. A run might consist of three successive button presses, one each in which the desired result was more zeroes or more ones, and one as a control with no conscious intention. A z score would then be computed for each button press. The 832 studies in the analysis were conducted from 1959 to 1987 and included 235 “control’”’ stud- ies, in which the output of the RNGs were recorded but there was no conscious intention involved. These were usually conducted before and during the experimental series, as tests of the RNGs. Results. The effect size measure used was again z/Vvn, where z was positive if more bits of the specified type were achieved. The mean effect size for control studies was not significantly different from zero (—1.0 x 10~°). The mean effect size for the experimental studies was also very small, 3.2 x 1074, but it was significantly higher than the mean ES for the control studies (z = 4.1). Quality. Sixteen quality measures were defined and assigned to each study, under the four general categories of procedures, statistics, data and the RNG device. A score of 16 reflected the highest quality. The authors regressed mean effect size on mean quality for each investigator and found a slope of 2.5 x 10~° with standard error of 3.2 x 107°, indicating little relationship between quality and outcome. They also calculated a weighted mean effect size, using quality scores as weights, and found that it was very similar to the unweighted mean ES. They concluded that “differences in methodological quality are not significant predictors of effect size” (page 1507). File Drawer. Radin and Nelson used several : CIRROBSE-GO7BSRGOS FOBT Ob BP"
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 376 J. UTTS studies (pages 1508-1510). Their estimates ranged from 200 to 1000 based on models assuming that all significant studies were reported. They calculated the fail-safe N to be 54,000. 6.3 Attempts to Influence Dice Radin and Ferrari (1991) examined 148 studies, published from 1935 to 1987, designed to test whether or not consciousness can influence the results of tossing dice. They also found 31 “con trol” studies in which no conscious intention was involved. Results. The effect size measure used was z/vn, where z was based on the number of throws in which the die landed with the desired face (or faces) up, in n throws. The weighted mean ES for the experimental studies was 0.0122 with a stan- dard error of 0.00062; for the control studies the mean and standard error were 0.00093 and 0.00255, respectively. Weights for each study were de- termined by quality, giving more weight to high- quality studies. Combined z scores for the exper- imental and control studies were reported by Radin _and Ferrari to be 18.2 and 0.18, respectively. Quality. Eleven dichotomous quality measures were assigned, ranging from automated recording to whether or not control studies were interspersed with the experimental studies. The final quality score for each study combined these with informa- tion on method of tossing the dice, and with source of subject (defined below). A regression of quality score versus effect size resulted in a slope of — 0.002, with a standard error of 0.0011. However, when effect sizes were weighted by sample size, there was a significant relationship between quality and ef- fect size, leading Radin and Ferrari to conclude that higher-quality studies produced lower weighted effect sizes. File Drawer. Radin and Ferrari calculated Rosenthal’s fail-safe. N for this analysis to be 17,974. Using the assumption that all significant studies were reported, they estimated the number of unreported studies to be 1152. As a final assess- ment, they compared studies published before and after 1975, when the Journal of Parapsychology adopted an official policy of publishing nonsigni- ficant results. They concluded, based on that an- alysis, that more nonsignificant studies were published after 1975, and thus “We must consi- der the overall (1935-1987) data base as suspect. with respect to the filedrawer problem.” Comparisons. Radin and Ferrari noted that there was bias in both the experimental and control studies across die face. Six was the face most likely to come up, consistent with the cheaceaot that it has the leagt daesrr sults for tne set of 69 or Retes in sue targets were evenly balanced among the six faces. They still found a significant | effect, with mean and stan- dard error for effect size of 8.6 x 10-* and 1.1 x 10-3, respectively. The combined z was 7.617 for these studies. They also compared effect sizes across types of subjects used in the studies, categorizing them as unselected, experimenter and other subjects, exper- imenter as sole subject,'and specially selected sub- jects. Like Honorton and Ferrari (1989), they found the highest mean ES. for studies with selected subjects; it was approximately 0.02, more than twice that for unselected subjects. 6.4 Extroversion and ESP Performance Honorton, Ferrari and Bem (1991) conducted a meta-analysis to examine the relationship between scores on tests of extroversion and scores on psi-related tasks. They found 60 studies by 17 investigators, conducted; from 1945 to 1983. Results. The effect size measure used for this analysis was the correlation between each subject’s extroversion score and: ESP score. A variety of measures had been used for both scores across stud- ies, so various correlation coefficients were used. Nonetheless, a stem and leaf diagram of the corre- lations showed an approximate bell shape with mean and standard deviation of 0.19 and 0.26, respectively, and with an additional outlier at r = 0.91. Honorton et al. reported that when weighted by degrees of freedom, the weighted mean r was 0.14, with a 95% confidence interval covering 0.10 to 0.19. Forced-Choice versus Free-Response Re- sults. Because forced-choice and free-response tests differ qualitatively, Honorton et al. chose to exam- ine their relationship to extroversion separately. They found that for free-response studies there was a significant correlation between extroversion and ESP scores, with mean rv = 0.20 and z = 4.46. Fur- ther, this effect was homogeneous across both investigators and extroversion scales. For forced-choice studies, there was a significant correlation between ESP and extroversion, but only for those studies that reported the ESP results to the subjects before measuring extroversion. Honorton et al. speculated that the relationship was an artifact, in which extroversion scores were temporarily inflated as a result of positive feedback on ESP performance. Confirmation with New Data Following the extroversion/ESP meta-analysis, Honorton et al. attempted to confirm: the relationship using the autoganzfeld data base. Extroversion scores based on the Myers-Briggs Type Indicator were 8: CIALRDP RE - -B07 89 R034 bo0gtede1uG° had participated in autoganzteld studies.
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION IN PARAPSYCHOLOGY 377 The correlation between extroversion scores and ganzfeld rating scores was r= 0.18, with a 95% confidence interval from 0.05 to 0.30. This is con- sistent with the mean correlation of r = 0.20 for free-response experiments, determined from the meta-analysis. These correlations indicate that ex- troverted subjects can produce higher scores in free-response ESP tests. 7. CONCLUSIONS Parapsychologists often make a distinction be- tween ‘proof-oriented research” and ‘“process- oriented research.” The former is typically con- ducted to test the hypothesis that psi abilities exist, while the latter is designed to answer questions about how psychic functioning works. Proof- oriented research has dominated the literature in parapsychology. Unfortunately, many of the studies used small samples and would thus be nonsignificant even if a moderate-sized effect exists. The recent focus on meta-analysis in parapsy- chology has revealed that there are small but consistently nonzero effects across studies, experi- menters and laboratories. The sizes of the effects in forced-choice studies appear to be comparable to those reported in some medical studies that had been heralded as breakthroughs. (See Section 5; also Honorton and Ferrari, 1989, page 301.) Free- response studies show effect sizes of far greater magnitude. A promising direction for future process-oriented research is to examine the causes of individual differences in psychic functioning. The ESP/ex- troversion meta-analysis is a step in that direction. In keeping with the idea of individual differ- ences, Bayes and empirical Bayes methods would appear to make more sense than the classical infer- ence methods commonly used, since they would allow individual abilities and beliefs to be modeled. Jeffreys (1990) reported a Bayesian analysis of some of the RNG experiments and showed that conclu- sions were closely tied to prior beliefs even though hundreds of thousands of trials were available. It may be that the nonzero effects observed in the meta-analyses can be explained by something other than ESP, such as shortcomings in our understand- ing of randomness and independence. Nonetheless, there is an anomaly that needs an explanation. As I have argued elsewhere (Utts, 1987), research in parapsychology should receive more support from the scientific community. If ESP does not exist, there is little to be lost by erring in the direction of further research, which may in fact uncover other much to be gained by discovering how to enhance and apply these abilities to important world problems. ACKNOWLEDGMENTS I would like to thank Deborah Delanoy, Charles Honorton, Wesley Johnson, Scott Plous and an anonymous reviewer for their helpful comments on an earlier draft of this paper, and Robert Rosenthal and Charles Honorton for discussions that helped clarify details. REFERENCES Arxinson, R. L., ATKINSON, R. C., Smirs, E. E. and Bem, D. J. (1990). Introduction to Psychology, 10th ed. Harcourt Brace Jovanovich, San Diego. BE torr, J. (1985). Research strategies for dealing with unstable phenomena. In The Repeatability Problem in Parapsychol- ogy (B. Shapin and L. Coly, eds.) 1-21. Parapsychology Foundation, New York. BLACKMORE, S. J. (1985). Unrepeatability: Parapsychology’s only finding. In The Repeatability Problem in Parapsychology (B. Shapin and L. Coly, eds.) 183-206. Parapsychology Foundation, New York. Burpick, D. S. and KEtty, E. F. (1977). Statistical methods in parapsychological research. In Handbook of Parapsychology (B. B. Wolman, ed.) 81-130. Van Nostrand Reinhold, New York. Camp, B. H. (1937). (Statement in Notes Section.) Journal of Parapsychology 1 305. ConEn, J. (1990). Things I have learned (so far). American Psychologist 45 1304-1312. Coover, J. E. (1917). Experiments in Psychical Research at Leland Stanford Junior University, Stanford Univ. Dawes, R. M., LanpMan, J. and Wi.uias, J. (1984). Reply to Kurosawa. American Psychologist 39 74-75. Diaconis, P. (1978). Statistical problems in ESP research. Sci- ence 201 131-136. Dommeyer, F. C. (1975). Psychical research at Stanford Univer- sity. Journal of Parapsychology 39 173-205. Druckman, D. and Swets, J. A., eds. (1988) Enhancing Human Performance: Issues, Theories, and Techniques. National Academy Press, Washington, D.C. EpcewortH, F. Y. (1885). The calculus of probabilities applied to psychical research. In Proceedings of the Society for Psychical Research 3 190-199. Epceworth, F. Y. (1886). The calculus of probabilities applied to psychical research. II. In Proceedings of the Society for Psychical Research 4 189-208. Fetter, W. K. (1940). Statistical aspects of ESP. Journal of Parapsychology 4 271-297. FEeLLer, W. K. (1968). An Introduction to Probability Theory and Its Applications 1, 3rd ed. Wiley, New York. FisHeR, R. A. (1924). A method of scoring coincidences in tests with playing cards. In Proceedings of the Society for Psychi- cal Research 34 181-185. FisHer, R. A. (1929). The statistical method in psychical re- search. In Proceedings of the Society for Psychical Research 39 189-192. Ga.uup, G. H., Jr., and Newport, F. (1991). Belief in paranor- mal phenomena among adult Americans. Skeptical Inquirer 15 137-146. GARDNER, M. J. and ALTMAN, D. G. (1986). Confidence intervals ct by nAPRFAVES £6 HRlnae¢ ZONGIORA : C1A-RURSR NY THERON RI OS Ria 8