Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 370 J. UTTS | number of studies increases. While Hyman ac- knowledged that ‘‘vote-counting raises many prob- lems” (Hyman, 1985b, page 8), he nonetheless spent half of his critique of the ganzfeld studies showing why Honorton’s count of 55% was wrong. Hyman’s first complaint was that several of the studies contained multiple conditions, each of which should be considered as a separate study. Using this definition he counted 80 studies (thus further reducing the sample sizes of the individual studies), of which 25 (81%) were “successful.” Honorton’s response to this was to invite readers to examine the studies and decide for themselves if the varying conditions constituted separate experiments. Hyman next postulated that there was selection bias, so that significant studies were more likely to be reported. He raised some important issues about how pilot studies may be terminated and not re- ported if they don’t show significant results, or may at least be subject to optional stopping, allowing the experimenter to determine the number of tri- als. He also presented a chi-square analysis that “suggests a tendency to report studies with a small sample only if they have significant results” (Hyman, 1985b, page 14), but I have questioned his analysis elsewhere (Utts, 1986, page 397). Honorton refuted Hyman’s argument with four rejoinders (Honorton, 1985b, page 66). In addition to reinterpreting Hyman’s chi-square analysis, Honorton pointed out that the Parapsychological Association has an official policy encouraging the publication of nonsignificant results in its journals and proceedings, that a large number of reported ganzfeld studies did not achieve statistical signifi- cance and that there would have to be 15 studies in the “‘file-drawer’”’ for every one reported to cancel out the observed significant results. The remainder of Hyman’s vote-counting analy- sis consisted of showing that the effective error rate for each study was actually much higher than the nominal 5%. For example, each study could have been analyzed using the direct hit measure, the sum of ranks measure or one of two other measures used for free-response analyses. Hyman carried out a simulation study that showed the true error rate would be 0.22 if “significance” was defined by re- quiring at least one of these four measures to achieve the 0.05 level. He suggested several other ways in which multiple testing could occur and concluded that the effective error rate in each ex- periment was not the nominal 0.05, but rather was probably close to the 31% he had determined to be the actual success rate in his vote-count. Honorton acknowledged that there was a multi- ple testing problem, but he had a two-fold response. First, he applied a Bonfpree pproved For that the number of significant studies (using his definition of a study) only dropped from 55% to 45%. Next, he proposed that a uniform index of success be applied to all studies. He used the num- ber of direct hits, since it was by far the most commonly reported measure and was the measure used in the first published psi ganzfeld study. He then conducted a detailed analysis of the 28 studies reporting direct hits and found that 43% were sig- nificant at 0.05 on that measure alone. Further, he showed that significant effects were reported by six of the 10 independent investigators and thus were not due to just one or two investigators or laborato- ries. He also noted that success rates were very similar for reports published in refereed journals and those published in unrefereed monographs and abstracts. While Hyman’s arguments identified issues such as selective reporting and optional stopping that should be considered in} any meta-analysis, the de- pendence of significance levels on sample size makes the vote-counting technique almost useless for as- sessing the magnitude of the effect. Consider, for example, the 24 studiesi where the direct hit meas- ure was reported and the chance probability of a direct hit was 0.25, the most common type of study in the data base. (There were four direct hit studies with other chance probabilities and 14 that did not report direct hits.) Of the 24 studies, 13 (54%) were “nonsignificant” at a = 0.05, one-tailed. But if the 367 trials in these “failed replications” are com- bined, there are 106 direct hits, z = 1.66, and p = 0.0485, one tailed. This is reminiscent of the dilemma of Professor B in Section 3. Power is typically very low for these studies. The median sample size for the studies reporting direct hits was 28. If there is a real effect and it increases the success probability from the chance 0.25 to an actual 0.33 (a value whose rationale wil] be made clear below), the power for a study with 28 trials is only 0.181 (Utts, 1986). It should be no surprise that there is a “repeatability” problem in parapsychology. 4.4 Flaw Analysis and Future Recommendations The second half of Hyman’s paper consisted of a “Meta-Analysis of Flaws and Successful Outcomes” (1985b, page 30), designed to explore whether or not various measures of success were related to specific flaws in the experiments. While many crit- ics have argued that the results in parapsychology can be explained by experimental flaws, Hyman’s analysis was the first to attempt to quantify the relationship between flaws and significant results. Hyman identified 12 potential flaws in the SSS DOd/UIbs : CIA-RHESS-0078SRG03100010001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION IN PARAPSYCHOLOGY 371 ization, multiple tests used without adjusting the significance level (thus inflating the significance level from the nominal 5%) and failure to use a duplicate set of targets for the judging process (thus allowing possible clues such as fingerprints). Using cluster and factor analyses, the 12 binary flaw variables were combined into three new variables, which Hyman named General Security, Statistics and Controls. Several analyses were then conducted. The one reported with the most detail is a factor analysis utilizing 17 variables for each of 36 studies. Four factors emerged from the analysis. From these, Hyman concluded that security had increased over the years, that the significance level tended to be inflated the most for the most complex studies and that both effect size and level of significance were correlated with the existence of flaws. Following his factor analysis, Hyman picked the three flaws that seemed to be most highly corre- lated with success, which were inadequate atten- tion to both randomization and documentation and the potential for ordinary communication between the sender and receiver. A regression equation was then computed using each of the three flaws as dummy variables, and the effect size for the experi- ment as the dependent variable. From this equa- tion, Hyman concluded that a study without these three flaws would be predicted to have a hit rate of 27%. He concluded that this is “well within the statistical neighborhood of the 25% chance rate” (1985b, page 37), and thus ‘“‘the ganzfeld psi data base, despite initial impressions, is inadequate ei- ther to support the contention of a repeatable study or to demonstrate the reality of psi” (page 38). Honorton discounted both Hyman’s flaw classifi- cation and his analysis. He did not deny that flaws existed, but he objected that Hyman’s analysis was faulty and impossible to interpret. Honorton asked psychometrician David Saunders to write an Ap- pendix to his article, evaluating Hyman’s analysis. Saunders first criticized Hyman’s use of a factor analysis with 17 variables (many of which were dichotomous) and only 36 cases and concluded that “the entire analysis is meaningless” (Saunders, 1985, page 87). He then noted that Hyman’s choice of the three flaws to include in his regression anal- ysis constituted a clear case of multiple analysis, since there were 84 possible sets of three that could have been selected (out of nine potential flaws), and Hyman chose the set most highly correlated with effect size. Again, Saunders concluded that “any interpretation drawn from [the regression analysis] must be regarded as meaningless” (1985, page 88). Hyman’s results were also contradicted by Harris and Rosenthal (1988b) in an analysis requested b Hyman in his capacity as Chair of the National Academy of Sciences’ Subcommittee on Parapsy- chology. Using Hyman’s flaw classifications and a multivariate analysis, Harris and Rosenthal con- cluded that “Our analysis of the effects of flaws on study outcome lends no support to the hypothesis that ganzfeld research results are a significant function of the set of flaw variables” (1988b, page 3). Hyman and Honorton were in the process of preparing papers for a second round of debate when they were invited to lunch together at the 1986 Meeting of the Parapsychological Association. They discovered that they were in general agreement on several major issues, and they decided to coauthor a “Joint Communiqué” (Hyman and Honorton, 1986). It is clear from their paper that they both thought it was more important to set the stage for future experimentation than to continue the techni- cal arguments over the current data base. In the abstract to their paper, they wrote: We agree that there is an overall significant effect in this data base that cannot reasonably be explained by selective reporting or multiple analysis. We continue to differ over the degree to which the effect constitutes evidence for psi, but we agree that the final verdict awaits the outcome of future experiments conducted by a broader range of investigators and according to more stringent standards [page 351]. The paper then outlined what these standards should be. They included controls against any kind of sensory leakage, thorough testing and documen- tation of randomization methods used, better re- porting of judging and feedback protocols, control for multiple analyses and advance specification of number of trials and type of experiment. Indeed, any area of research could benefit from such a careful list of procedural recommendations. 4.5 Rosenthal’s Meta-Analysis The same issue of the Journal of Parapsychology in which the Joint Communiqué appeared also car- ried commentaries on the debate by 10 separate authors. In his commentary, psychologist Robert Rosenthal, one of the pioneers of meta-analysis in psychology, summarized the aspects of Hyman’s and Honorton’s work that would typically be in- cluded in a meta-analysis (Rosenthal, 1986). It is worth reviewing Rosenthal’s results so that they can be used as a basis of comparison for the more recent psi ganzfeld studies reported in Section 5. Rosenthal, like Hyman and Honorton, focused only on the 28 studies for which direct hits were wn. He chose to use an effect size measure kno Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 372 J. UTTS called Cohen’s A, which is the difference between the arcsin transformed proportions of direct hits that were observed and expected: p — arcsin \/p). One advantage of this measure over the difference in raw proportions is that it can be used to compare experiments with different chance hit rates. If the observed and expected numbers of hits were identical, the effect size would be zero. Of the 28 studies, 23 (82%) had effect sizes greater than zero, with a median effect size of 0.32 and a mean of 0.28. These correspond to direct hit rates of 0.40 and 0.38 respectively, when 0.25 is expected by chance. A 95% confidence interval for the true effect size is from 0.11 to 0.45, corresponding to direct hit rates of from 0.30 to 0.46 when chance is 0.25. A common technique in meta-analysis is to calcu- late a “combined z,” found by summing the indi- vidual z scores and dividing by the square root of the number of studies. The result should have a standard normal distribution if each z score has a standard normal distribution. For the ganzfeld studies, Rosenthal reported a combined z of 6.60 with a p-value of 3.37 x 10-1. He also reiterated Honorton’s file-drawer assessment by calculating that there would have to be 423 studies unreported to negate the significant effect in the 28 direct hit studies. Finally, Rosenthal acknowledged that, because of the flaws in the data base and the potential for at least a small] file-drawer effect, the true average effect size was probably closer to 0.18 than 0.28. He concluded, ‘‘Thus, when the accuracy rate expected under the null is 1/4, we might estimate the ob- tained accuracy rate to be about 1/3” (1986, page 333). This is the value used for the earlier power calculation. It is worth mentioning that Rosenthal was com- missioned by the National Academy of Sciences to prepare a background paper to accompany its 1988 report on parapsychology. That paper (Harris and Rosenthal, 1988a) contained much of the same analysis as his commentary summarized above. Ironically, the discussion of the ganzfeld work in the National Academy Report focused on Hyman’s 1985 analysis, but never mentioned the work it had commissioned Rosenthal to perform, which contra- dicted the final conclusion in the report. h= 2(arcsin 5. A META-ANALYSIS OF RECENT GANZFELD EXPERIMENTS After the initial exchange with Hyman at the 1982 Farapsychologca! Association Meeting Approved For Release 2000/08/08 : Honorton and his colleagues developed an auto- mated ganzfeld experiment that was designed to eliminate the methodological flaws identified by Hyman. The execution and reporting of the experi- ments followed the detailed guidelines agreed upon by Hyman and Honorton. Using this “autoganzfeld” experiment, 11 experi- mental series were conducted by eight experi- menters between February 1983 and September 1989, when the equipment had to be dismantled due to lack of funding. In this section, the results of these experiments are summarized and com- pared to the earlier ganzfeld studies. Much of the information is derived from Honorton et al. (1990). 5.1 The Automated Ganzfeld Procedure Like earlier ganzfeld studies, the “autoganzfeld” experiments require four participants. The first is the Receiver (R), who attempts to identify the tar- get material being observed by the Sender (S). The Experimenter (E) prepares R for the task, elicits the response from R and supervises R’s judging of the response against the four potential targets. (Judging is double blind; E does not know which is the correct target.) The fourth participant is the lab assistant (LA) whose only task is to instruct the computer to randomly select the target. No one involved in the experiment knows the identity of the target. Both R and § are sequestered in sound-isolated, electrically shielded rooms. R is prepared as in earlier ganzfeld studies, with white noise and a field of red light. In a nonadjacent room, S watches the target material on a:television and can hear R’s target description (“mentation”) as it is being given. The mentation is also tape recorded. The judging process takes place immediately af- ter the 30-minute sending period. On a TV monitor in the isolated room, R yiews the four choices from the target pack that contains the actual target. R is asked to rate each one according to how closely it matches the ganzfeld mentation. The ratings are converted to ranks and, if the correct target is ranked first, a direct hit is scored. The entire proc- ess is automatically recorded by the computer. The computer then displays : the correct choice to R as feedback. There were 160 preselected targets, used with replacement, in 10 of the 11 series. They were arranged in packets of four, and the decoys for a given target were always the remaining three in the same set. Thus, even if a particular target in a set were consistently favored by Rs, the probability of a direct hit under the null hypothesis would remain at 1/4. Popular targets should be no more CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION IN PARAPSYCHOLOGY 373 likely to be selected by the computer’s random number generator than any of the others in the set. The selection of the target by the computer is the only source of randomness in these experiments. This is an important point, and one that is often misunderstood. (See Utts, 1991, for elucidation.) Eighty of the targets were “dynamic,” consisting of scenes from movies, documentaries and cartoons; 80 were “static,” consisting of photographs, art prints and advertisements. The four targets within each set were all of the same type. Earlier studies indicated that dynamic targets were more likely to produce successful results, and one of the goals of the new experiments was to test that theory. The randomization procedure used to select the target and the order of presentation for judging was thoroughly tested before and during the experi- ments. A detailed description is given by Honorton et al. (1990, pages 118-120), Three of the 11 series were pilot series, five were formal series with novice receivers, and three were formal series with experienced receivers. The last series with experienced receivers was the only one that did not use the 160 targets. Instead, it used only one set of four dynamic targets in which one target had previously received several first place ranks and one had never received a first place rank. The receivers, none of whom had had prior exposure to that target pack, were not aware that only one target pack was being used. They each contributed one session only to the series. This will be called the “special series” in what follows. Except for two of the pilot series, numbers of trials were planned in advance for each series. Unfortunately, three of the formal series were not yet completed when the funding ran out, including the special series, and one pilot study with advance planning was terminated early when the experi- menter relocated. There were no unreported trials during the 6-year period under review, so there was no “file drawer.” Overall, there were 183 Rs who contributed only one trial and 58 who contributed more than one, for a total of 241 participants and 355 trials. Only 23 Rs had previously participated in ganzfeld experi- ments, and 194 Rs (81%) had never participated in any parapsychological research. 5.2 Results While acknowledging that no probabilistic con- clusions can be drawn from qualitative data, Hon- orton et al. (1990) included several examples of session excerpts that Rs identified as providing the basis for their target rating. To give a flavor for the dream-like quality of the mentation and the amount rank, the first example is reproduced here. The target was a painting by. Salvador Dali called “Christ Crucified.” The correct target received a first place rank. The part of the mentation R used to make this assessment read: .. [think of guides, like spirit guides, leading me and I come into a court with a king. It’s quiet....It’s like heaven. The king is some- thing like Jesus. Woman. Now I’m just sort of summersaulting through heaven.... Brooding.... Aztecs, the Sun God....High priest....Fear....Graves. Woman. Prayer. . Funeral.... Dark. Death....Souls....Ten Commandments. Moses... .{Honorton et al., 19901. Over all 11 series, there were 122 direct hits in the 355 trials, for a hit rate of 34.4% (exact bino- mial p-value = 0.00005) when 25% were expected by chance. Cohen’s A is 0.20, and a 95% confidence interval for the overall hit rate is from 0.30 to 0.39. This calculation assumes, of course, that the proba- bility of a direct hit is constant and independent across trials, an assumption that may be question- able except under the null hypothesis of no psi abilities. Honorton et al. (1990) also calculated effect sizes for each of the 11 series and each of the eight experimenters. All but one of the series (the first novice series) had positive effect sizes, as did all of the experimenters. The special series with experienced Rs had an exceptionally high effect size with h = 0.81, corre- sponding to 16 direct hits out of 25 trials (64%), but the remaining series and the experimenters had relatively homogeneous effect sizes given the amount of variability expected by chance. If the special series is removed, the overall hit rate is 32.1%, h = 0.16. Thus, the positive effects are not due to just one series or one experimenter. Of the 218 trials contributed by novices, 71 were direct hits (82.5%, h = 0.17), compared with 51 hits in the 137 trials by those with prior ganzfeld experience (37%, h = 0.26). The hit rates and effect sizes were 31% (h = 0.14) for the combined pilot series, 32.5% (A = 0.17) for the combined formal novice series, and 41.5% (h = 0.35) for the com- bined experienced series. The last figure drops to 31.6% if the outlier series is removed. Finally, without the outlier series the hit rate for the com- bined series where all of the planned trials were completed was 31.2% (h = 0.14), while it was 35% (A = 0.22) for the combined series that were termi- nated early. Thus, optional stopping cannot of informataga tray Ee POPREEATS FUVOIDE HS : CFACRDPYBOB789RGE8T0001 0001-6