Approved For Release 2000/08/08 : CIA-RDP96- soreenbostcoo000 -6 398 J. UTTS lieved that some of my statements were inaccurate and indicated that I had not carefully read the reports. I will address some of his specific objec- tions and show that, except where noted, the accu- racy of my original statements can be verified by further elaboration and clarification, with due apol- ogy for whatever necessary details were lacking in my original report. Most of our points of disagreement concern the National Academy of Sciences (National Re- search Council) report Enhancing Human Per- formance (Druckman and Swets, 1988). This report evaluated several controversial areas, in- cluding parapsychology. Professor Hyman chaired the Parapsychology Subcommittee. Several back- ground papers were commissioned to accompany this report, available from the “Publication on Demand Program” of the National Academy Press. One of the papers was written by Harris and Rosenthal, and entitled ‘Human Performance Research: An Overview.” Professor Hyman alleged that “‘Utts mistakenly asserts that my subcommittee on parapsychology commissioned Harris and Rosenthal to evaluate parapsychology experiments for us....” I cannot find a statement in my paper that asserts that Harris and Rosenthal were commissioned by the subcommittee, nor can I find a statement that asserts that they were asked to evaluate parapsy- chology experiments. Nonetheless, I believe our substantive disagreement results from the fact that the work by Harris and Rosenthal was writ- ten in two parts, both of which I referenced in my paper. They were written several months apart, but published together, and each had its own history. The first part (Harris and Rosenthal, 1988a) is the one to which I referred with the words “Rosenthal was commissioned by the National Academy of Sciences to prepare a background paper to accompany its 1988 report on parapsychol- ogy” (p. 372). According:to Rosenthal (personal communication, July 23, 1991) he was asked to pre- pare a background paper to address evaluation issues and experimenter effects to accompany the report in five specific areas of research, including parapsychology. The second part was a “Postscript” to the com- missioned paper (Harris and Rosenthal, 1988b), and this is the one to which I referred on page 371 as “requested by Hyman in his capacity as Chair of the National Academy of Sciences’ Subcommittee on Parapsychology.” (It is probably this wording that led Professor Hyman to his erroneous allega- tion.) The postscript began with the words “We Hyman, chair of the subcommittee on parapsychol- ogy, in which he raiseg questions about the pres- ence and consequence of methodological flaws in the ganzfeld studies . 2 In reference to this postscript, I stand corrected on a technical point, because Hyman himself did not request the response to his own letter. As noted by Palmer, Honorton and Utts (1989), the postscript was added because: At one stage of the process, John Swets, Chair of the Committee, actually phoned Rosenthal and asked him to withdraw the parapsychology section of his [commissioned] paper. When Rosenthal declined, Swets and Druckman then requested that Rosenthal respond to criticisms that Hyman had included in a July 30, 1987 letter to Rosenthal [page 38]. A related issue on which I would like to elaborate concerns the correlation between flaws and success in the original ganzfeld data base. Hyman has misunderstood both my position and that of Harris and Rosenthal. He believes that I implicitly denied the importance of the flaws, so I will make my position explicit. I do not think there is any evi- dence that the experimental results were due to the identified flaws. The flaw analysis was clearly use- ful for delineating acceptable criteria for future experiments. Several experiments were conducted using those criteria. The results were similar to the original experiments. I believe that this indicates an anomaly in need of dn explanation. In discussing the paper and postscript by Harris and Rosenthal, Hyman stated that “The alleged contradictory conclusions [to the National Research Council report] of Harris and Rosenthal are based on a meta-analysis that supports Honorton’s posi- tion when Honorton’s [flaw] ratings are used and supports my position when my ratings are used.” He believes that Harris and Rosenthal (and J) failed to see this point because the low power of the test associated with their analysis was not taken into account. The analysis in question was based on a canoni- cal correlation between flaw ratings and measures of successful outcome for the ganzfeld studies. The canonical correlation was 0.46, a value Hyman finds to be impressive. What he has failed to take into account however, is that a canonical correlation gives only the magnitude of the relationship, and not the direction. A careful reading of Harris and Rosenthal (1988b) reveals that their analysis actu- ally contradicted the idea that the flaws could account for the successful ganzfeld results, since “Interestingly, three of the six flaw variables corre- have been asted otf Por Reads bdvUOBIOS ; CIRREBES HoVSERBDS {808 T00H1-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION IN PARAPSYCHOLOGY 399 and with the outcome canonical variable but three correlated negatively” (page 2, italics added). Rosenthal (personal communication, July 23, 1991) verified that this was indeed the point he was trying to make. Readers who are interested in drawing their own conclusions from first-hand analyses can find Hyman’s original flaw codings in an Appendix to his paper (Hyman, 1985, pages 44-.49), Finally, in my paper, I stated that the parapsy- chology chapter of the National Research Council report critically evaluated statistically significant experiments, but not those that were nonsignifi- cant. Professor Hyman “does not know how [I] got such an impression,” so I will clarify by outlining some of the material reviewed in that report. There were surveys of three major areas of psi research: remote viewing (a particular type of free-response experiment), experiments with random number generators, and the ganzfeld experiments. As an example of where I got the impression that they evaluated only significant studies, consider the sec- tion on remote viewing. It began by referencing a published list of 28 studies. Fifteen of these were immediately discounted, since “only 13... were published under refereed auspices” (Druckman and Swets, 1988, page 179). Four more were then dis- missed, since “Of the 13 scientifically reported experiments, 9 are classified as successful” (page 179). The report continued by discussing these nine experiments, never again mentioning any of the remaining 19 studies. The other sections of the report placed similar emphasis on significant stud- ies. I did not think this was a valid statistical method for surveying a large body of research. Minor Point Raised by Morris The final clarification I would like to offer con- cerns the minor point raised by Professor Morris, that “When Honorton omitted studies that did not report direct hits as a measure, he may have biased his sample.” This possibility was explicitly ad- dressed by Honorton (1985, page 59). He examined what would happen if z-scores of zero were inserted for the 10 studies for which the number of direct hits was not measured, but could have been. He found that even with this conservative scenario, the combined z-score only dropped from 6.60 to 5.67, SATISFYING THE SKEPTICS Parapsychology is probably the only scientific discipline for which there is an organization of skeptics trying to discredit its work. The Commit- tee for Re Scientific Investigation of Claims of the Approved For Release 2000/08/08 : CIA RDP S6- 00789R00310 Paranormal (CSICOP) was established in 1976 by philosopher Paul Kurtz and sociologist Marcello Truzzi when ‘‘Kurtz became convinced that the time was ripe for a more active crusade against parapsychology and other pseudo-scientists” (Pinch and Collins, 1984, page 527). Truzzi resigned from the organization the next year (as did Professor Diaconis) ‘“‘because of what he saw as the growing danger of the committee’s excessive negative zeal at the expense of responsible scholarship” (Collins and Pinch, 1982, page 84). In an advertising brochure for their publication The Skeptical In- quirer, CSICOP made clear its belief that paranor- mal phenomena are worthy of scientific attention only to the extent that scientists can fight the growing interest in them. Part of the text of the brochure read: ‘‘Why the sudden explosion of inter- est, even among some otherwise sensible people, in all sorts of paranormal ‘happenings’?... Ten years ago, scientists started to fight back. They set up an organization—The Committee for the Scientific In- vestigation of Claims of the Paranormal.” During the six years that I have been working with parapsychologists, they have repeatedly ex- pressed their frustration with the unwillingness of the skeptics to specify what would constitute ac- ceptable evidence, or even to delineate criteria for an acceptable experiment. The Hyman and Honor- ton Joint Communiqué was seen as the first major step in that direction, especially since Hyman was the Chair of the Parapsychology Subcommittee of CSICOP. Hyman and Honorton (1986) devoted eight pages to ‘“Recommendations for Future Psi Experiments,” carefully outlining details for how the experiments should be conducted and reported. Honorton and his colleagues then conducted several hundred trials using these specific criteria and found essen- tially the same effect sizes as in earlier work for both the overall effect and effects with moderator variables taken into account. I would expect Profes- sor Hyman to be very interested in the results of these experiments he helped to create. While he did acknowledge that they ‘have produced intriguing results,” it is both surprising and disappointing that he spent only a scant two paragraphs at the end of his discussion on these results. Instead, Hyman seems to be proposing yet an- other set of requirements to be satisfied before parapsychology should be taken seriously. It is dif- ficult to sort out what those requirements should be from his account: “[They should] specify, in ad- vance, the complete sample space and the critical region. When they get to the point where they can sperity this along with some boundary conditions make some reasonable Gb10004 6 then they 010001
400 will have demonstrated something worthy of our attention.” Diaconis believes that psi experiments do not deserve serious attention unless they actively in- volve skeptics. Presumably, he is concerned with subject or experimenter fraud, or with improperly controlled experiments. There are numerous docu- mented cases of fraud and trickery in purported psychic phenomena. Some of these were observed by Diaconis and reported in his article in Science. Such cases have mainly been revealed when inves- tigators attempted to verify the claims of individ- ual psychic practitioners in quasi-experimental or uncontrolled conditions. These instances have re- ceived considerable attention, probably because the claims are so sensational, the fraud is so easy to detect by a skilled observer and they are an easy target for skeptics looking for a way to discredit psychic phenomena. As noted by Hansen (1990), “Parapsychology has long been tainted by the fraudulent behavior of a few of those claiming psy- chic abilities” (page 25). Control against deception by subjects in the labo- ratory has been discussed extensively in the para- psychological literature (see, e.g., Morris, 1986, and Hansen, 1990). Properly designed experiments should preclude: the possibility of such fraud. Hyman and Honorton (1986, page 355) explicitly discussed precautions to be taken in the ganzfeld experiments, all of which were followed in the auto- ganzfeld experiments. Further the controlled labo- ratory experiments discussed in my paper usually used a large number of subjects, a situation that minimizes the possibility that the results were due to fraud on the part of a few subjects. As for the possibility of experimenter fraud, it is of course an issue in all areas of science. There have been a few such instances in parapsychology, but since para- psychologists tend to be aware of this possibility, they were generally detected and exposed by insid- ers in the field. It is not clear whether or not Diaconis is suggest- ing that a magician or “qualified skeptic” needs to be present at all times during a laboratory experi- ment. I believe that it would be more productive for such consultation to occur during the design phase, and during the implementation of some pilot ses- sions. This is essentially what was done for the autoganzfeld experiments, in which Professor Hy- man, a skeptic as well as an accomplished magi- cian, participated in the specification of design criteria, and mentalists Bem and Kross observed experimental sessions. Bem is also a well-respected experimental psychologist. While I believe that the skeptics, particularly some of the more knowledgeable members of Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 CSICOP, have served ai useful role in helping to improve experiments, their counter-advocacy stance is counterproductive. If they are truly interested in resolving the question of whether or not psi abilities exist, I would expect them to encourage evaluation and experimentation by unbiased, skilled experimenters. Instead, they seem to be trying to discourage such interest by providing a moving target of requirements that must be satis- fied first. SHOULD PSI RESEARCH BE IGNORED BY THE SCIENTIFIC COMMUNITY? In the conclusion of my paper, I argued that the scientific community should pay more attention to the experimental results in parapsychology. I was not suggesting that the accumulated evidence con- stitutes proof of psi abilities, but rather that it indicates that there is indeed an anomalous effect that needs an explanation. Greenhouse noted that my paper will not necessarily change anyone’s view about the existence of paranormal phenomena, an observation with which ] agree. However, I hope it will change some views about the importance of further investigation. Mosteller and Diaconis both acknowledged that there are reasons for statisticians to be interested in studying the anomalous effects, regardless of whether or not psi is real. As noted by Mosteller, “If there is no ESP, then we want to be able to carry out null experiments and get no effect, other- wise we cannot put much belief in work on small effects in non-ESP situations.” Diaconis concluded that “Parapsychology is worthy of serious study” partly because “If it ig wrong, it offers a truly alarming massive case study of how statistics can mislead and be misused.” Greenhouse noted several sociological reasons for the resistance of the scientific community to accept- ing parapsychological phenomena. One of these is that they directly contradict the laws of physics. However, this assertion is not uniformly accepted by physicists (see, e.g., Oteri, 1975), and some of the leading parapsychological researchers hold Ph.D.s in physics. Another reason cited by Greenhouse, and sup- ported by Hyman, is that psychic phenomena are currently unexplainable by a unified scientific the- ory. But that is precisely the reason for more inten- sive investigation. The history of science and medicine is replete with examples where empirical departures from expectation led to important find- ings or theoretical models. For example, the causal connection between cigarette smoking and lung cancer was established only after years of statisti- Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION IN PARAPSYCHOLOGY 401 cal studies, resulting from the observation by one physician that his lung cancer patients who smoked did not recover at the same rate as those who did not. There are many medications in common use for which there is still no medical explanation for their observed therapeutic effectiveness, but that does not prohibit their use. There are also examples where a coherent theory of a phenomenon was impossible because the re- quisite background information was missing. For instance, the current theory of endorphins as an explanation for the success of acupuncture would have been impossible before the discovery of endor- phins in the 1970s. Mosteller’s observation that ESP will not replace ‘the telephone leads to the question of whether or not psi abilities are of any use even if they do exist, since the effects are relatively small. Again, a look at history is instructive. For example, in 1938 For- tune Magazine reported that “‘At present, few sci- entists foresee any serious or practical use for atomic energy.” Greenhouse implied that I think parapsychology is not accepted by more of the scientific community only because they have not examined the data, but this misses the main point I was trying to make. The point is that individual scientists are willing to express an opinion without any reference to data. The interesting sociological question is why they are so resistant to examining the data. One of the major reasons is undoubtedly the perception identi- fied by Greenhouse that there is some connection between parapsychology and the occult, or worse, religious beliefs. Since religion is clearly not in the realm of science, the very thought that parapsy- chology might be a science leads to what psychol- ogists call “cognitive dissonance.” As noted by Griffin (1988), ‘“‘People feel unpleasantly aroused when two cognitions are dissonant—when they con- tradict one another” (page 33). Griffin continued by observing that there are also external reasons for scientists to discount the evidence, since “It is gen- erally easier to be a skeptic in the face of novel evidence; skeptics may be overly conservative, but they are rarely held up to ridicule” (page 34). In summary, while it may be safer and more consonant with their beliefs for individual scien- tists to ignore the observed anomalous effects, the scientific community should be concerned with finding an explanation. The explanations proposed by Greenhouse and others are simply not tenable. REPLICATION AND MODELING Parapsychology is one of the few areas ere a point null hypothesis makes some sense. We specify what should happen if there is no such thing as ESP by using simple binomial models, either to find p-values or Bayes factors. As noted by Mosteller, if there is no ESP, or other nonstatis- tical explanation for an effect, we should be able to carry out null experiments and get no effect. Other- wise, we should be worried about using these sim- ple models for other applications. Greenhouse, in his first alternative explanation for the results, questioned the use of these simple models, but his criticisms do not seem relevant to the experiments discussed in Section 5 of my paper. The experiments to which he referred were either poorly controlled, in which case no statistical anal- ysis could be valid, or were specifically designed to incorporate trial by trial feedback in such a way that the analysis needed to account for the added information. Models and analyses for such experi- ments can be found in the references given at the end of Diaconis’ discussion. For the remainder of this discussion, I will con- fine myself to models appropriate for experiments such as the autoganzfeld described in Section 5. It is this scenario for which Bayarri and Berger com- puted Bayes factors, and for which Dawson dis- cussed possible Bayesian models. If ESP does exist, it is undoubtedly a gross over- simplification to use a simple non-null binomial model for these experiments. In addition to poten- tial differences in ability among subjects, there were also observed differences due to dynamic ver- sus static targets, whether or not the sender was a friend, and how the receiver scored on measures of extraversion. All of these differences were antici- pated in advance and could be incorporated into models as covariates. It is nonetheless instructive to examine the Bayes factor computed by Bayarri and Berger for the simple non-null binomial model. First, the observed anomalous effects would be less interesting if the Bayes factor was small for reasonable values of r, as it was for the random number generator experi- ments analyzed by Jefferys (1990), most of which purported to measure psychokinesis instead of ESP. Second, the Bayes factor provides a rough measure of the strength of the evidence against the null hypothesis and is a much more sensible summary than the p-value. The Bayes factors provided by Bayarri and Berger are probably more conserva- tive, in the sense of favoring the null hypothesis, than those that would result from priors elicited from parapsychologists, but are probably reason- able for those who know nothing about past ob- served effects. I expect tht most parapsychologists would not opt for a prior symmetric around chance, but would still choose one with some mass below Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6