Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 394 J. UTTS evaluator. Certainly none of them cited any corre- lation values between evaluators, and the correla- tions between judges of research quality in other social sciences tend to be “at best around .50,” according to Hunter and Schmidt (1990, page 497). Although Honorton and Hyman reported a rela- tively high correlation of 0.77 between themselves, they were each doing their own study and their flaw analyses did reach somewhat different conclu- sions, as noted by Utts. Other than Hyman, the evaluators cited by Utts tend to be positively ori- ented toward parapsychology; roughly speaking, all evaluators doing flaw analyses found what they might hope to find, with the exception of the PK dice data base. Were evaluators blind as to study outcome when coding flaws? No comment is made on this aspect. The above studies need to be repli- cated, with multiple (and blind) evaluators and reported indices of evaluator agreement. Ideally, evaluator attitude should be assessed and taken into account as well. A study with all hostile evalu- ators may report very high evaluator correlations, yet be a less valid study than one that employs a range of evaluators and reports lower correlations among evaluators. But what constitutes a replication of a meta- analysis? As with experimental replications, it may be important to distinguish between exact and con- ceptual replications. In the former, a replicator would attempt to match all salient features of the initial analysis, from the selection of reports to the coding of features to the statistical tests employed, such as to verify that the stated original protocol had been followed faithfully and that a simi- lar outcome results. For conceptual replication, replicators would take the stated outcome of the meta-analysis and attempt their own independent analysis, with their own initial report selection criteria, coding criteria and strategy for statistical testing, to see if similar conclusions resulted. Con- ceptual replication allows more room for bias and resultant debate when findings differ, but when results are similar they can be assumed to have more legitimacy. Given the strong and surpris- ing (for many) conclusions reached in the meta- analysis reported by Utts, it is quite likely that others with strong views on parapsychology will attempt to replicate, hoping for clear confirma- tion or disconfirmation. The diversity of methods they are likely to employ and the resultant debates should provide a good opportunity for airing the many conceptual problems still present in meta- analysis. If results differ on moderator variables, there can come to be empirical resolution of the differences as further results unfold. With regard to flaw analysis, such cused attenti PFO egedr DER RGASE AAUUICBI08 : & dance of existing faults and how to avoid them. If results are as strong under well-controlled con- ditions as under sloppy ones, then additional research such as that done by Honorton and associ- ates under tight conditions should continue to pro- duce positive results. | In addition to the replication issue, there are some other problems that need to be addressed. So far, the assessment of moderator variables has been univariate, whereas a multivariate approach would seem more likely to produce a clearer picture. Mod- erator variables may covary, with each other or with flaws. For instance, in the dice data higher effect sizes were found for flawed studies and for studies with selected subjects. Did studies using special subjects use weaker procedures? Given the importance attached to effect size and incorporating estimates of effect size in designing ‘studies for power, we must be careful not to assume that effect size is independent of number of trials or subjects unless we have empirical reason to do so. Effect size may decrease with larger N if experi- menters are stressed or bored towards the end of a long study or if there are too many trials to be conducted within a short period of time and sub- jects are given less time to absorb their instructions or to complete their tasks. On one occasion there is presentation of an estimated “true average effect size,” (0.18 rather than 0.28) without also present- ing an estimate of effect size dispersal. Future investigators should have some sense of how the likelihood that they will obtain a hit rate of 1/3 (where 1/4 is expected) will vary in accordance with conditions. : There are a few additional quibbles with particu- lar points. In Utts’ example experiment with Pro- fessor A versus Professor B, sex of professor is a possible confounding variable. When Honorton omitted studies that did| not report direct hits as a measure, he may have biased his sample. Were there studies omitted that could have reported di- rect hits but declined to,do so, conceivably because they looked at that measure, saw no results and dropped it? This objection is only with regard to the initial meta-analysis and is not relevant for the later series of studies which all used direct hits. In Honorton’s meta-analysis of forced-choice precogni- tion experiments, the comparison variables of feed- back delay and time interval to target selection appear to be confounded. Studies delaying target selection cannot provide trial by trial feedback, for instance. Also, [am unsure about using an approxi- mation to Cohen’s A for assessing the effect size for the aspirin study. There would appear to be a very cia effect, with the aspirin condition heart Ch ARO BPaBpt/eeybes tony 9 GHGAzglacebo 4 O80 1 ion of
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION OF PARAPSYCHOLOGY 395 misses estimated; perhaps Cohen’s fA greatly un- derestimates effect size when very low probability events (less than 1 in 50 for heart attack in the placebo condition and less than 1 in a 100 for aspirin) are involved. I’m not a statistician and thus don’t know if there is a relevant literature on this point. Comment Frederick Mosteller Dr. Utts’s discussion stimulates me to offer some comments that bear on her topic but do not, in the main, fall into an agree-disagree mode. My refer- ences refer to her bibliography. Let me recommend J. Edgar Coover’s work to statisticians who would like to read about a pretty sequence of experiments developed and executed well before Fisher’s book on experimental design appeared. Most of the standard kinds of ESP exper- iments (though not the ganzfeld) are carried out and reported in this 1917 book. Coover even began looking into the amount of information contained in cues such as whispers. He also worked at expos- ing mediums. I found the book most impressive. As Utts says in her article, the question of significance level was a puzzling one, and one we still cannot solve even though some fields seem to have stan- dardized on 0.05. When Feller’s comments on Stuart and Green- wood’s sampling experiments came out in the first edition of his book, I was surprised. Feller devotes a problem to the results of generating 25 symbols from the set a, b, c, d and e (page 45, first edition) using random numbers with 0 and 1 corresponding to a, 2 and 3 to b, etc. He asks the student to find out how often the 25 produce 5 of each symbol. He asks the student to check the results using random number tables. The answer seems to be about 1 chance in 500. In a footnote Feller then says “They [random numbers] are occasionally extraordinarily obliging: c.f. J. A. Greenwood and E. E. Stuart, Review of Dr. Feller’s Critique, Journal of Para- Frederick Mosteller is Roger I. Lee Professor of Mathematical Statistics, Emeritus, at Harvard Uni- versity and Director of the Technology Assessment Group in the Harvard School of Public Health. His mailing address is Department of Statistics, Har- vard University, Science Center, 1 Oxford Street, The above objections should not detract from the overall value of the Utts survey. The findings she reports will need to be replicated; but even as is, they provide a challenge to some of the cherished arguments of counteradvocates, yet also challenge serious researchers to use these findings effectively as guidelines for future studies. psychology, vol. 4 (1940), pp. 298-319, in particular p. 306.” The 25 symbols of 5 kinds, 5 of each, correspond to the cards in a parapsychology deck. The point of page 306 is that Greenwood and Stuart on that page claim to have generated two random orders of such a deck using Tippett’s table — of random numbers. Apparently Feller thought that it would have taken them a long time to do it. If one assumes that Feller’s way of generating a ran- dom shuffle is required, then it would indeed be unreasonable to suppose that the experiments could be carried out quickly. I wondered then whether Feller thought this was the only way to produce a random order to such a deck of cards. If you happen to know how to shuffle a deck efficiently using random numbers, it is hard to believe that others do not know. I decided to test it out and so I proposed to a class of 90 people in mathematical statistics that we find a way of using random num- bers to shuffle a deck of cards. Although they were familiar with random numbers, they could not come up with a way of doing it, nor did anyone after class come in with a workable idea though several stu- dents made proposals. I concluded that inventing such a shuffling technique was a hard problem and that maybe Feller just did not know how at the time of writing the footnote. My face-to-face at- tempts to verify this failed because his response was evasive. I also recall Feller speaking at a scientific meeting where someone had complained about mistakes in published papers. He said essen- tially that we won’t have any literature if mistakes are disallowed and further claimed that he always had mistakes in his own papers, hard as he tried to avoid them. It was fun to hear him speak. Although I find Utts’s discussion of replication engaging as a problem in human perception, I do always feel that people should not be expected to carry out difficult mathematical exercises in their head, off the cuff, without computers, textbooks or advisors. The kind of problem treated requires Cambridkp proved BorsRelease 2000/08/08 : ClAcRDPSE-00*89RD0S7t00OT00Ghabysis. Even
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 396 J. UTTS after a careful analysis is completed, there can be vigorous reasonable arguments about the appropri- ateness of the formulation and its analysis. These investigations leave me reinforced with the belief that people cannot do hard mathematical problems in their heads, rather than with an attitude toward or against ESP investigations. When I first became aware of the work of Rhine and others, the concept seemed to me to be very important and I asked a psychologist friend why more psychologists didn’t study this field. He re- sponded that there were too many ways to do these experiments in a poorly controlled manner. At the time, I had just discovered that when viewed with light coming from a certain angle, I could read the Rejoinder Jessica Utts I would like to thank this distinguished group of discussants for their thought-provoking contribu- tions. They have raised many interesting and di- verse issues. Certain points, such as Professor Mosteller’s enlightening account of Feller’s posi- tion, require no further comment. Other points in- dicate the need for clarification and elaboration of my original material. Issues raised by Professors Diaconis and Hyman and subsequent conversations with Robert Rosenthal and Charles Honorten have led me to consider the topic of “Satisfying the Skeptics.” Since the conclusion in my paper was not that psychic phenomena have been proved, but rather that there is an anomalous effect that needs to be explained, comments by several of the discus- sants led me to address the question “Should Psi Research be Ignored by the Scientific Community?” Finally, each of the discussants addressed repli- cation and modeling issues. The last part of my rejoinder comments on some of these ideas and discusses them in the context of parapsychology. CLARIFICATION AND ELABORATION Since my paper was a survey of hundreds of experiments and many published reports, I could obviously not provide all of the details to accom- pany this overview. However, there were details lacking in my paper that have led to legitimate questions and misunderstandings from several of the discussants. In this section, I address specific points raised by Professors Diaconis, Greenhouse, backs of the cards of my parapsychology deck as clearly as the faces. While preparing these remarks in 1991, I found a note on page 305 of volume 1 of The Journal of Parapsychology (1937) indicating that imperfections in the cards precluded their use in unscreened situations, but that improvements were on the way. Thus I sympathize with Utts’s conclusion that much is to be gained by studying how to carry out such work well. If there is no ESP, then we want to be able to carry out null experi- ments and get no effect, otherwise we cannot put much belief in work or small effects in non-ESP situations. If there is ESP, that is exciting. How- ever, thus far it does not look as if it will replace the telephone. Hyman and Morris, by either clarifying my origi- nal statements or by adding more information from the original reports. Points Raised by Diaconls Diaconis raised the point that qualified skeptics and magicians should be active participants in parapsychology experinjents, I will discuss this general concept in the next section, but elaborate here on the steps that were taken in this regard for the autoganzfeld experiments described in Section 5 of my paper. As reported by Honorton et al. (1990): Two experts on the simulation of psi ability have examined the autoganzfeld system and protocol. Ford Kross has been a professional mentalist [a magician' who simulates psychic abilities] for over 20 years... Mr. Kross has provided us with the following statement: ‘In my professional capacity as a mentalist, I have reviewed Psychophysital Research Laborato- ries’ automated ganzfeld system and found it to provide excellent security against deception by subjects.” We have received similar comments from Daryl Bem, Professor of Psychology at Cornell University. Professor Bem is well known for his research in social and personal- ity psychology. He is: also a member of the Psychic Entertainers Association and has per- formed for many years as a mentalist. He vis- Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION IN PARAPSYCHOLOGY 397 ited PRL for several days and was a subject in Series 101” [pages 134-135]. Honorton has also informed me (personal communi- cation, July 25, 1991) that several self-proclaimed skeptics have visited his laboratory and received demonstrations of the autoganzfeld procedure and that no one expressed any concern with the secu- rity arrangements. This may not completely satisfy Professor Diaco- nis’ objections, but it does indicate a serious effort on the part of the researchers to involve such peo- ple. Further, the original publication of the re- search in Section 5 followed the reporting criteria established by Hyman and Honorton (1986), thus providing much more detail for the reader than the earlier published records to which Professor Diaconis alludes. Points Raised by Greenhouse Greenhouse enumerated four items that offer al- ternative explanations for the observed anomalous effects. Three of these (items 2-4) will be addressed in this section by elaborating on the details pro- vided in my paper. His item 1 will be addressed in a later section. Item 2 on his list questioned the role of experi- menter expectancy effects as a potential confounder in parapsychological research. While the expecta- tions of the experimenter may influence the report- ing of results, the ganzfeld experiments (as well as other psi experiments) are conducted in such a way that experimenter expectancy cannot account for the results themselves. Rosenthal, who Greenhouse cites as the expert in this area, addressed this in his background paper for the National Research Council (Harris and Rosenthal, 1988a) and con- cluded that the ganzfeld studies were adequately controlled in this regard. He also visited the auto- ganzfeld laboratory and was given a demonstration of that procedure. Greenhouse’s item 3, the question of what consti- tutes a direct hit, was addressed in my paper but perhaps needs elaboration. Although free-response experiments do generate substantial amounts of subjective data, the statistical analysis requires that the results for each trial be condensed into a single measure of whether or not a direct hit was achieved. This is done by presenting four choices to a judge (who of course does not know the correct answer) and asking the judge to decide which of the four best matches the subject’s response. If the judge picks the target, a direct hit has occurred. It is true that different judges may differ on their opinions of whether or not there has been a direct cal question is the same. Under the null hypothe- sis, since the target is randomly selected from the four possibilities presented, the probability of a direct hit is 0.25 regardless of who does the judg- ing. Thus, the observed anomalous effects cannot be explained by assuming there was an over- optimistic judge. If Professor Greenhouse is suggesting that the source of judging may be a moderating variable that determines the magnitude of the demonstrated anomalous effect, I agree. The parapsychologists have considered this issue in the context of whether or not subjects should serve as judges for their own sessions, with differing opinions in different labora- tories. This is an example of an area that has been suggested for further research. Finally, Greenhouse raised the question of the accuracy of the file-drawer estimates used in the reported meta-analyses. I agree that it is instruc- tive to examine the file-drawer estimate using more than one model. As an example, consider the 39 studies from the direct hit and autoganzfeld data bases. Rosenthal’s fail-safe N estimates that there would have to be 371 studies in the file-drawer to account for the results. In contrast, the method proposed by Iyengar and Greenhouse gives a file- drawer estimate of 258 studies. Even this estimate is unrealistically large for a discipline with as few researchers as parapsychology. Given that the av- erage number of trials per experiment is 30, this would represent almost 8000 unreported trials, and at least that many hours of work. There are pros and cons to any method of esti- mating the number of unreported studies, and the actual practices of the discipline in question should be taken into account. Recognizing publication bias as an issue, the Parapsychological Association has had an official policy since 1975 against the selec- tive reporting of positive results. Of the original ganzfeld studies reported in Section 4 of my paper, less than half were significant, and it is a matter of record that there are many nonsignificant studies and “failed replications” published in all areas of psi research. Further, the autoganzfeld database reported in Section 5 has no file-drawer. Given the publication practices and the size of the field, the proposed file-drawer cannot account for the ob- served effects. Points Raised by Hyman One of my goals in writing this paper was to present a fair account of recent work and debate in parapsychology. Thus, I was disturbed that Hy- man, who has devoted much of his career to the study of parapsychology, and who had first-hand hit on appBOV EEF OP RBIGASY 008 UBI0E : CIARBBUE.6o7SeRGES1oB010001-6—