agpebt & Approved For Release 2000/08/08 : CIA-RDP96-00789R003200100001-5 26 DARYL J. BEM I submit that Hyman is using a double standard here. If the successful replication of the relation between target type and psi performance is not analogous to the earlier finding with the View Master reels, then why is this near miss with a method- ologically cleaner assessment of the sender-receiver variable a “noteworthy inconsistency’? Hyman cannot have it both ways. If the heterogeneity of the original database and the methodological dissimilarities be- iween its variables and those in the autoganzfeld studies pre- clude strong claims of consistency, then these same factors pre- clude strong claims of inconsistency. Randomization As we noted in our article, the issue of target randomization is critical in many psi experiments because systematic patterns in inadequately randomized target sequences might be detected ‘by subjects during a session or might match their preexisting response biases. In a ganzfeld study, however, randomization is less problematic because only one target is selected during the session and most subjects serve in only one session. The primary concern is simply that all the stimuli within each judging set be sampled uniformly over the course of the study. Similar consid- erations govern the second randomization, which takes place after the ganzfeld period and determines the sequence in which the target and decoys are presented to the receiver for judging. In the 10 basic autoganzfeld experiments, 160 film clips were sampled for a total of 329 sessions; accordingly, a particular clip would be expected to appear as the target in only about 2 ses- sions. This low expected frequency means that it is not possible to statistically assess the randomness of the actual distribution observed. Accordingly, Honorton et al. (1990) ran several large- scale control series to test the output of the random number generator. These control series confirmed that it was providing a uniform distribution of values through the full target range. Statistical tests that could legitimately be performed on the ac- tual frequencies observed confirmed that targets were, on aver- age, selected uniformly from among the four film clips within each judging set and that the four possible judging sequences were uniformly distributed across the sessions. Nevertheless, Hyman remains legitimately concerned about the adequacy of the randomizations and their potential interac- tions with possible receiver or experimenter response biases. Two kinds of response bias are involved: differential preferences for video clips on the basis of their content and differential pref- erences for clips on the basis of their position in the judging sequence. Content-Related Response Bias Because the adequacy of target randomization cannot be sta- tistically assessed owing to the low expected frequencies, the possibility remains open that an unequal distribution of targets ‘could interact with receivers’ content preferences to produce artifactually high hit rates. As we reported in our article, Hon- orton and [ encountered this problem in an autoganzfeld study that used a single judging set for all sessions (Study 302), a prob- lem we dealt with in two ways. To respond to Hyman’s concerns, 1 have now performed the same two analyses on the remainder of the database. Both treat the four-clip judging set as the unit of analysis, and neither requires the assumption that the null baseline is fixed at 25% or at any other particular value. In the first analysis, the actual target frequencies observed are used in conjunction with receivers’ actual judgments to derive a new, empirical baseline for each judging set. In particular, I multiplied the proportion of times each clip in a set was the target by the proportion of times that a receiver rated it as the target. This product represents the probability that a receiver would score a hit if there were no psi effect. The sum of these products across the four clips in the set thus constitutes the em- pirical null baseline for that set. Next, [ computed Cohen’s mea- sure of effect size (4) on the difference between the overall hit rate observed within that set and this empirical baseline. For purposes of comparison, I then reconverted Cohen's h back to its equivalent hit rate for a uniformly distributed judging set in which the null baseline would, in fact, be 25%. Across the 40 sets, the mean unadjusted hit rate was 31.5%, significantly higher than 25%, one-sample (39) = 2.44, p = .O1, one-tailed. The new, bias-adjusted hit rate was virtually identi- cal (30.7%), (39) = 2.37, p = .O1, tain(39) = 0.85, p = .40, indi- cating that unequal target frequencies were not significantly in- flating the hit rate. The second analysis treats each film clip as its own control by comparing the proportion of times it was rated as the target when it actually was the target and the proportion of times it was tated as the target when it was one of the decoys. This procedure automatically cancels out any content-related target prefer- ences that receivers (or experimenters) might have. First, | cal- culated these two proportions for each clip and then averaged them across the four clips within each judging set. The results show that across the 40 judging sets, clips were rated as targets significantly more frequently when they were targets than when they were decoys (29% and 22%, respectively), paired (39) = 2.03, p = .025, one-tailed. Both of these analyses indicate that the observed psi effect cannot be attributed to the conjunction of unequal target distributions and content-related response bi- ases. Sequence-Related Response Bias Hyman is also concerned about the randomization of the judging sequence because we can expect strong systematic biases during the judging procedure. The fact that the items to be judged have to be presented sequentially, when combined with what we know about subjective validation. . . would lead us to expect a strong tendency to select the first or second items during the judging series. Hyman’s hypothesis is correct: As shown in Table 1, receivers do display a position bias in their judgments x7(3, N = 354) = 8.64, p < .05, tending to identify as targets clips appearing either first or last in the judging sequence. Moreover, the actual distri- bution of targets across the judging positions also departs sig- nificantly from a uniform distribution, x7(3, N = 354) = 7.83, p< .05, with targets appearing most frequently in the third po- sition. To determine whether the conjunction of these two unequal distributions might contribute artifactually to the hit rate, one Approved For Release 2000/08/08 : CIA-RDP96-00789R003200100001-5
Approved For Release 2000/08/08 : CIA-RDP96-00789R003200100001-5 RESPONSE TO HYMAN 27 can again combine the observed frequencies to derive an empir- ical null baseline. As shown in Table 1, each proportion in the second column can be multiplied by the corresponding propor- tion in the third column to yield the hit rate expected if there were no psi effect. As shown, the expected hit rate across all four judging positions is 24.7%. The pertinent fact here is that this is lower than the 25% that would have been obtained if the target positions had been uni- formly distributed across the sessions. In other words, the con- junction of receivers’ position biases with the imperfect ran- domization of target positions works against successful psi per- formance in these data. Again, inadequate randomization has not contributed artifactually to the hit rates. Alternative Randomizing Strategies? Hyman suggests that “one way to prevent response biases from distorting the hit rate is to use a randomizing procedure that makes sure that each item within a target pool occurs equally often.” Coming from a critic as sophisticated as Hyman, this is a very puzzling suggestion, because he appears to be sug- gesting some variant of sampling without replacement, a proce- dure that would virtually guarantee response-bias artifacts. For example, if receivers tend to avoid selecting targets that ap- peared in previous sessions, this response bias would coincide with the actual diminishing probabilities that a previously seen target would reappear. The experimenters—who participate in many sessions and discuss them with one another—are in an even better position to detect and possibly to exploit the dimin- ishing probabilities of target repetition. Sampling without re- placement is precisely what enables card counters to improve their odds at blackjack. Alternatively, perhaps Hyman is advocating a procedure in which the experiment continues until each clip within a judging set appears as a target a predesignated minimum number of times. For purposes of analysis, the investigator then randomly discards excess sessions until the target frequencies are equal- ized at that minimum number. This would solve the response- bias problem but would be enormously wasteful. Suppose, for _ example, that only 4 sessions from each judging set would have to be discarded, on average, to equalize the target frequencies. With 40 judging sets, the investigator would end up discarding 160 sessions, equal to nearly half of the sessions that took Honorton and his colleagues 6% years to collect! Only a study with many fewer judging sets could reasonably implement this strategy. Hit Rates as a Function of Target Repetition In his post hoc excursion through the autoganzfeld data, Hy- man uncovered an unexpected positive relationship between hit rates and the number of times targets had been targets in previ- ous sessions. (Ironically, Hyman has been one of the most out- spoken critics of parapsychologists who search through their data without specific hypotheses and then emerge with unex- pected ‘‘findings.’”) If this finding is reliable and not just a fluke of post hoc ex- ploration, then it is difficult to interpret because target repeti- tion is confounded with the chronological sequence of sessions: Table t Proportion of Sessions in Which Each Clip Was Selected as the Target and Proportion in Which It Appeared as the Target Position in Expected judging Selected Appeared hit rate sequence as target as target (%) 1 .30 25 75 2 .20 24 4.9 3 22 | 6.7 4 28 .20 5.6 Total 1.00 1.00 24.7 Note. N= 354 sessions. Higher repetitions of a target necessarily occur later in the se- quence than lower repetitions. In turn, the chronological se- quence of sessions is confounded with several other variables, including more experienced experimenters, more ‘‘talented” receivers (e.g., Juilliard students and receivers being retested be- cause of earlier successes), and methodological refinements in- troduced in the course of the program in an effort to enhance psi performance (e.g., experimenter “prompting”). Again, however, Hyman’s major concern is that this pattern might reflect an interaction between inadequate target random- ization and possible response biases on the part of those receiv- ers or experimenters who encounter the same judging set more than once. This seems highly unlikely. In the entire database, only 8 subjects saw the same judging set twice, and none of them performed better on the repetition than on the initial session. Similar arithmetic applies to experimenters: On average, each of the eight experimenters encountered a given judging set only 1.03 times. The worst case is an experimenter who encountered the same judging set 6 times over the 62 years of the program. These six sessions yielded three hits, two of them in the first two sessions. At the end of his discussion, Hyman wonders whether this relationship between target repetition and hit rates is “due to an artifact or [does it] point to some new, hitherto unrecognized property of psi?” If it should turn out to be the latter, then I believe it only appropriate that parapsychologists reward his serendipity by calling it the Hyman Effect. References Bem, D. J., & Honorton, C. (1994). Does psi exist? Replicable evidence for an anomalous process of information transfer. Psychological Bul- letin, 115, 4-18. Honorton, C., Berger, R. E., Varvoglis, M. P.. Quant, M., Derr, P., Schechter, E. 1., & Ferrari, D. C. (1990). Psi communication in the ganzfeld: Experiments with an automated testing system and a com- parison with a meta-analysis of earlier studies. Journal of Parapsy- chology, 54, 99-139. Hyman, R. (1994). Anomaly or artifact? Comments on Bem and Hon- orton. Pswhological Bulletin, 115, 19-24. Received August 9, 1993 Accepted August 9, 1993 Approved For Release 2000/08/08 : CIA-RDP96-00789R003200100001-5