Approved For Release hanes Borel ArkRA6199792R000100130002-1 3 Table 1 Outcome by Study “ N N N % Effect size Stu Study type subjects trials hits hits u z 1 Pilot 19 2 Pilot 4 3 Pilot 25 101 Novice 50 102 Novice 50 103 Novice 50 104/105 (a) Novice 36 104/105 (b) Juilliard Sample 20 201 Experienced 3 301 Experienced 25 22 8 36 62 99 9 3 33 60 25 36 10 28 54 22 50 12 24 AT ~.30 50 18 36 63 1.60 50 15 30 55 67 36 12 33 60 97 20 10 50 75. 2.20 7 3 43 69 69 50 15 30 56 67 302 Experienced 25 a) 16 444 702 2.024 Overall (Studies 1-301) 24) 330 106 32 59 2.85 Note. Zscores are based on the exact binomial probability with p= .25 and q=.75. ®Corrected for maximum possible response bias. Hit rate actually observed was 64%. responds to an effect size (9 of .59, with a 95% confidence interval from .53 to .64. Table 1 also shows that when Studies 104 and 105 are combined and re-divided into the non-Juilliard and Juil- Hiard samples, 9 of the 10 studies yield positive effect sizes, with a mean effect size (2) of .61, ¢ (9) = 4.35, p = 0009, one-tailed. This effect size is equivalent to a four-al- ternative hit rate of 34%. Alternatively, if we retain Stud- jes 104 and 105 as separate studies, 9 of the 10 studies again yield positive effect sizes, with a mean effect size (7) of .62, t (9) = 3.67, p = .003, one-tailed. This effect size is equivalent to a four-alternative hit rate of 35% and is identical to that found across the 28 studies of the earlier meta-analysis.® Considered together, sessions with novice participants (Studies 101-105) yielded a statistically significant hit rate of 32.5% (p = .009), which is not significantly differ- ent from the 31.6% hit rate achieved by experienced par- ticipants in Studies 201 and 301. And finally, each of the 8 experimenters also achieved a positive effect size, with a mean 7 of .60, t (7)= 3.44, p = .005, one-tailed. 54s noted above, the laboratory was forced to close before three - of the formal studies could be completed. If we assume that the remaining trials in Studies 105 and 201 would have yielded only chance results, this would reduce the overall z for the first 10 autoganzteld studies from 2.85 to 2.73 (p = .003). Thus, inclusion of the two incomplete studies does not pose an optional stopping problem. The third incomplete study, Study 302, is discussed below. Approved For Release 2000/08/10 The Juilliard sample. There are several reports in the literature of a relationship between creativity or artistic ability and psi performance (Schmeidler, 1988). In order to explore this possibility in the ganzfeld setting, 10 male and 10 female undergraduates were recruited from the Juilliard School of Music in New York City. Of these, 8 were music students, 10 were drama students, and 2 were dance students. Each served as the receiver in a single session in Studies 104 or 105. As shown in Table 1, these students achieved a hit rate of 50% (p = .014), one of the five highest hit rates ever reported for a single sample in a ganzfeld study. The musicians were particularly suc- cessful: Six of the eight (75%) successfully identified their targeta (p = .004). Further details about this sample and their ganzfeld performance are reported in Schlitz and Honorton (1992). Study size and effect size. There is a significant negative correlation across the 10 studies listed in Table 1 between the number of sessions in a study and its effect size (x): r = —.64, ¢ (8) = 2.36, p < .05, two-tailed. This is reminiscent of Hyman’s discovery that the smaller studies in the orig- inal ganzfeld database were disproportionately likely to report statistically significant results. He interpreted this finding as evidence for a bias against the reporting of small studies that fail to achieve significant results. A similar interpretation cannot be applied to the auto- ganzfeld studies, however, because there are no unre- ported sessions. One reviewer of this article suggested that the negative correlation might reflect a decline effect in which earlier sessions of a study are more successful than later ses- : CIA-RDP96-00792R000100130002-1
Approved For Releag@,20008/40rdahhaRAR86:00292R000100130002-1 9 Table 2 Study 302: Proportion of Sessions in which Each Video Clip was Ranked First when it was a Target and wi when it was a Decoy Relative Ranked First Ranked First Fisher's Frequency when when Exact Video Clip as Target Target Decoy Difference p Tidal Wave .28 57 14 +.46 .032 (7/25) (4/7) (2/18) Snakes 12 .67 .05 +.62 .029 (3/25) (2/3) (1/22) High-Speed Sex 16 25 05 +.20 .300 (4/25) (1/4) (1/21) Bugs Bunny 44 82 36 +.46 .027 (11/25) (9/11) (5/14) Means 58 14 +.44 - sions. If there were such an effect, then studies with fewer sessions would show larger effect sizes because they would end before a decline could set in. To check this pos- sibility, we computed point-biserial correlations between hits (= 1) or misses (= 0) and the session number within each of the 10 studies. All of the correlations hovered around zero; six were positive, four were negative, and the overall mean was +.03, indicating that hits were actually slightly more likely to occur in later sessions of a study than in earlier ones. An inspection of Table 1 reveals that the negative corre- lation derives primarily from the two studies with the largest effect sizes: the 20 sessions with the Juilliard atu- dents and the 7 sessions of Study 201, the study specifi- cally designed to retest the most promising participants from the previous studies. Accordingly, we believe that the larger effect sizes of these two studies—and hence the significant negative correlation between number of ses- sions and effect size—reflects a genuinely higher psi abil- ity of participants in these two small but highly selected samples. Study 302. All of the studies except Study 302 randomly sampled from a pool of 160 static and dynamic targets. Study 302 sampled from a single, dynamic target set which contained one video clip that had been correctly identified several times in the previous ten studies—a scene of a tidal wave from the movie Clash of the Titans—— and one clip that had never been correctly identified—a high-speed sex scene from Clockwork Orange. The set also contained a scene of crawling snakes from a TV documen- tary and a scene from a Bugs Bunny cartoon. The experimental design called for this study to con- tinue until each of the clips had served as the target 15 times. Unfortunately, the premature termination of this study at 25 sessions left an imbalance in the frequency with which each clip had served as the target. This means that the chance expectation of .25 used as the baseline for evaluating the results in the other studies cannot be ap- plied to this study. More importantly, it means that the high hit rate observed (64%) could well be inflated by re- sponse biases. To illustrate, water imagery is frequently reported by receivers in ganzfeld sessions whereas sexual imagery is rarely reported. (It is reasonable to suppose that some participants might be reluctant both to report sexual im- agery and to give the highest rating to the sex-related clip.) If a video clip containing popular imagery (like wa- ter) happens to appear as a target more frequently than a clip containing unpopular imagery (like sex), a high hit rate might simply reflect the coincidence of those frequen- cies of occurrence with participants’ response biases. And, as the first column of Table 2 reveals, the tidal wave clip did in fact appear more frequently as the target than did the sex clip. The Bugs Bunny clip was even more frequent, appearing as the target in 11 of the 25 (44%) sessions. We can assess the damage in two ways. First, we can ask for each of the four clips whether it was selected as the target (ie., ranked in first place) significantly more frequently when it was the target than when it was one of the three control clips (decoys). This comparison, which controls for the baseline popularity of the themes and images within each clip, is shown in the remaining columns of Table 2. As can be seen, each of the four clips was selected as the target relatively more frequently when it was the target than when it was a decoy, a difference that is significant for three of the four clips. On average, a clip was identified as the target 58% of the time when it was the target compared with only 14% of the time when it was a decoy. In other words, there is an effect over and above that pro- duced by response bias. Next, we can calculate a conservative estimate of that remaining effect by imagining a hypothetical set of in- formed receivers who were able to maximally exploit the unequal target frequencies to achieve the highest possible hit rate. Their optimal (non-psi) strategy would be to iden- Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130002-1
Approved For Releas@R00R/08 ro RAcROR 9G007/97R000100130002-1 10 tify the most frequent clip (Bugs Bunny) as the target in every session, thereby achieving a hit rate of 44%. This figure thus represents the hit rate expected if response bi- _ ases were making their maximum possible contribution to the hit rate—a worst case baseline for evaluating any re- maining psi effect. When the observed hit rate of 64% is compared with a hypothetical hit rate of 44%, the effect size (2) is .40, As shown in Table 1, this is equivalent to a four-alternative hit rate of (coincidentally) 44% or a x value of .70 and is statistically significant (z = 2.02, p = .022). Dynamic versus static targets. The success of Study 302 raises the question of whether dynamic targets are, in general, more effective than static targets. This possibility was also suggested by the earlier meta-analysis, which found that studies using multiple-image targets (View Master stereoscopic slide reels) obtained significantly higher hit rates than did studies using single-image tar- gets. By adding motion and sound, the video clips might be thought of as high-tech versions of the View Master reels. The ten autoganzfeld studies that randomly sampled from both dynamic and static target pools yielded an even split of 165 sessions with each target type. As predicted, sessions using dynamic targets yielded significantly more hits than did sessions using static targets (37% vs. 27%), Fisher's exact p < .04. Sender/receiver pairing. The earlier meta-analysis found that studies in which participants were free to bring in friends to serve as senders produced significantly higher hit rates than studies that used only laboratory-as- signed senders. As noted, however, there is no record of how many of the participants in the former studies actu- ally did bring in friends. Whatever the case, sender/receiver pairing was not a significant correlate of psi performance in the auto- ganzfeld studies: The 198 sessions in which the sender and receiver. were friends did not yield a significantly higher proportion of hits than did the 132 sessions in which they were not (34% vs. 29%), Fisher’s exact p = .34. Correlations between receiver characteristics and psi performance. Most of the autoganzfeld participants were strong believers in psi: On a 7-point scale, where “1” indi- cates strong disbelief and “7” indicates atrong belief in psi, the mean was 6.2 (SD = 1.03); only two participants rated their belief in psi below the midpoint of the scale. In addi- tion, 88% of the participants reported personal experi- ences suggestive of psi and 80% had some training in meditation or other techniques involving internal focus of attention. All of these appear to be important variables. The corre- lation between belief in psi and psi performance is one of the most consistent findings in the parapsychological lit- erature (Palmer, 1978). And within the autoganzfeld stud- ies, successful performance of novice (first-time) partici- pants is significantly predicted by reported personal psi experiences, involvement with meditation or other mental disciplines, and high scores on the Feeling and Perception factors of the Myers-Briggs Type Inventory (Honorton, 1992; Honorton & Schechter, 1987). This recipe for success has now been independently replicated in another laboratory (Broughton, Kanthamani, & Khilji, 1990). The personality trait of extraversion is also associated with better psi performance. A meta-analysis of 60 inde- pendent studies with nearly 3,000 subjects revealed a Approved For Release 2000/08/10 small but reliable positive correlation between extraver- sion and psi performance—especially in studies that used free-response methods of the kind employed in the ganzfeld experiments (Honorton, Ferrari, & Bem, 1992). Across 14 free-response studies by 4 independent investi- gators, the correlation for 612 subjects was .20 (z = 4.82, p = 1.5 x 107). This correlation was replicated in the auto- ganzfeld studies, where extraversion scores were available for 221 of the 241 subjects: r = .18, ¢ (219) = 2.67, p = .004, one-tailed. And finally, there is the strong psi performance of the Juilliard students, discussed above, which is consistent with other studies in the parapsychological literature suggesting a relationship between successful psi perfor- mance and creativity or artistic ability. Discussion Earlier in this article we quoted from the abstract of the Hyman-Honorton communiqué: “We agree that the final verdict. awaita the outcome of future experiments con- ducted by a broader range of investigators and according to more stringent standards” (1986, p, 351). We believe that the “stringent standards” requirement has been met by the autoganzfeld studies. The results are statistically significant and consistent with those in the earlier database. The mean effect size is quite respectable when compared with other controversial research areas of human performance (Harris & Rosenthal, 1988a). And, there are reliable relationships between successful psi , performance and conceptually relevant experimental and subject variables—relationships that also replicate previ- ous findings. Hyman has also commented on the auto- ganzfeld studies: Honorton’s experiments have produced intriguing results. If,..independent laboratories can produce similar results with the same relationships and with the same attention to rigorous methodology, then parapsychology may indeed have finally captured its elusive quarry. (1991, p. 392) Issues of Replication As Hyman’s comment implies, the autoganzfeld studies by themselves cannot satisfy the requirement that repli- cations be conducted by a “broader range of investigators.” Accordingly, we hope the findings reported here will be sufficiently provocative to prompt others to try replicating the psi ganzfeld effect. We believe it is essential, however, that future studies comply with the methodological, statistical, and reporting standards set forth in the joint communiqué and achieved by the autoganzfeld studies. It is not necessary for studies to be as automated or as heavily inatrumented as the au- toganzfeld studies in order to satisfy the methodological guidelines, but they are still likely to be labor intensive and potentially expensive.® 8As the closing of the autoganzfeld laboratory exemplifies, it is also difficult to obtain funding for psi research. The traditional, peer-refereed sources of funding familiar to psychologists have almost never funded proposals for psi research. The widespread skepticism of psychologists toward psi is almost certainly a con- tributing factor. : CIA-RDP96-00792R000100130002-1
Approved For Relessenanornsit® oQlAIRRPS6A0G792R000100130002-1 Statistical Power and Replication Would-be replicators also need to be reminded of the power requirernents for replicating small effects. Although many academic psychologists do not believe in psi, many apparently do believe in miracles when it comes to repli- cation. Tversky and Kahneman (1971) posed the following problem to their colleagues at meetings of the Mathemati- cal Psychology Group and the American Psychological As- sociation: Suppose you have run an-experiment on 20 subjects and have obtained a significant result which confirms your the- ory (z= 2.23, p< .05, two-tailed). You now have cause to run an additional group of 10 subjects. What do you think the probability is that the results will be significant, by a one- tailed test, separately for this group? (p. 105) The median estimate was .85, with nine out of ten re- spondents giving an estimate greater than .60. The cor- rect answer is approximately .48. As Rosenthal (1990) has warned: “Given the levels of statistical power at which we normally operate, we have no right to expect the proportion of significant results that we typically do expect, even if in nature there is a very real and very important effect” (p. 16). In this regard, it is again instructive to consider the medical study that found a highly significant effect of as- pirin on the ineidence of heart attacks. The study moni- tored over 22,000 subjects. Had the investigators moni- tored 3,000 subjects, they would have had less than an even chance of finding a conventionally significant effect. Such is life with small effect sizes. Given its larger effect size, the prospects for success- fully replicating the psi ganzfeld effect are not quite so daunting, but they are probably still grimmer than intu- ition would suggest. If the true hit rate is in fact about 34% when 25% is expected by chance, then an experiment with 30 trials (the mean for the 28 studies in the original meta-analysis) has only about 1 chance in 6 of finding an effect significant at the .05 level with a one-tailed test. A 50 trial experiment boosts that to about 1 in 3. One must escalate to 100 trials in order to come close to the break even point—where one has a 50-50 chance of finding a statistically significant effect (Utta, 1986). (Recall that only 2 of the 11 autoganzfeld studies yielded results that were individually significant at the conventional .05 level.) Those who require that a psi effect be statistically significant every time before they will seriously entertain the possibility that an effect really exists know not what they ask. Significance us. Effect Size But the preceding discussion is unduly pessimistic be- cause it perpetuates the tradition of worshipping the sig- nificance level. Regular readers of this journal are likely to be familiar with recent arguments imploring behavioral scientists to overcome their slavish dependence on the significance level as the ultimate measure of virtue and to focus more of their attention on effect sizes instead: “Surely, God loves the .06 nearly as much as the .05” (Rosnow & Rosenthal, 1989, p. 1277). Accordingly, we suggest that achieving a respectable effect size with a methodologically tight ganzfeld study would be a perfectly welcome contribution to the replication effort—no matter how untenurable the p level renders the investigator. Approved For Release 2000/08/10 : Career consequences aside, this suggestion may seem quite counterintuitive. Again, Tversky and Kahneman (1971) provide an elegant demonstration. They asked sev- eral of their colleagues to consider an investigator who runs 15 subjects and obtains a significant ¢ value of 2.46. Another investigator attempts to duplicate the procedure with the same number of subjects and obtains a result in the same direction but with a nonsignificant value of ¢. Tversky and Kahneman then asked their colleagues to in- dicate the highest level of ¢ in the replication study they would describe as a failure to replicate. The majority of their colleagues regarded ¢ = 1.70 as a failure to replicate. But if the data from two such studies (¢ = 2.46 and# = 1.70) were pooled, the ¢ for the combined data would be about 3.00 (assuming equal variances): Thus, we are faced with a paradoxical state of affairs, in which the same data that would increase our confidence in the finding when viewed as part of the original study, shake our confidence when viewed as an independent study. (p. 108) Such is the iron grip of the arbitrary .05. Pooling the data, of course, is what meta-analysis is all about. Ac- cordingly, we suggest that two or more laboratories could collaborate in a ganzfeld replication effort by conducting independent studies and then pooling them in meta-ana- lytic fashion—what we might call real-time meta-analy- sis. (Each investigator could then claim the pooled p level for his or her own curriculum vitae.) Maximizing Effect Size Rather than buying or borrowing larger sample sizes, those who seek to replicate the psi ganzfeld effect might find it more intellectually satisfying to attempt to maxi- mize the effect size by attending to the variables associ- ated with successful outcomes. Thus researchers who wish to enhance the chances of successful replication should use dynamic rather than static targets. Similarly we ad- vise using participants with the characteristics we have reported to be correlated with successful psi performance. Random college sophomores enrolled in introductory psy- chology do not constitute the optimal subject pool. And finally, we urge, ganzfeld researchers to read care- fully the detailed description of the warm social ambiance that Honorton et al. (1990) sought to create in the auto- ganzfeld laboratory. We believe that the social climate created in psi experiments is a critical determinant of their success or failure. The Problem of “Other” Variables This caveat about the social climate of the ganzfeld ex- periment prompted one reviewer of this article to worry that this provided “an escape clause” which weakens the falsifiability of the psi hypothesis: “Until Bem and Hon- orton can provide operational criteria for creating a warm social ambiance, the failure of an experiment with otherwise adequate power can always be dismissed as due to a lack of warmth.” Alas, it’s true; we devoutly wish it were otherwise. But the operation of unknown variables in moderating the success of replications is a fact of life in all the sciences. Consider, for example, an earlier article in this journal by Spence (1964). He reviewed studies testing the straightforward derivation from Hullian learning theory that high anxiety subjects should condition more CIA-RDP96-00792R000100130002-1