Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 390 J. UTTS | the reported data at face value and focuses on the statistical interpretation of these data. Both the statistical interpretation of the results of an individual experiment and of the results of a meta-analysis are based on a model of an ideal world. In this ideal world, effect sizes have a tractable and known distribution and the points in the sample space are independent samples from a coherent population. The appropriateness of any statistical application in a given context is an em- pirical matter. That is why such issues as the adequacy of randomization, the non-independence of experiments in a meta-analysis and the over- analysis of data are central to the debate. The optimistic conclusions from the meta-analyses as- sume that the effect sizes are unbiased estimates from independent experiments and have nicely behaved distributional properties. Before my detailed assessment of all the avail- able ganzfeld experiments through 1981, I accepted the assertions by parapsychologists that their experiments were of high quality in terms of stat- istical and experimental! methodology. I was sur- prised to find that the ganzfeld experiments, widely heralded as the best exemplar of a suc- cessful research program in parapsychology, were characterized by obvious possibilities for sensory leakage, inadequate randomization, over-analysis and other departures from parapsychology’s own professed standards. One response was to argue that I had exaggerated the number of flaws. But even internal critics agreed that the rate of defects in the ganzfeld data base was too high. The other response, implicit in Utts’ discussion of the ganzfeld experiments and the meta-analyses, was to admit the existence of the flaws but to deny their importance. The parapsychologists doing the meta-analysis would rate each experiment for qual- ity on one or more attributes. Then, if the null hypothesis of no correlation between effect size and quality were upheld, the investigators concluded that the results could not be attributed to defects in methodology. This retrospective sanctification using statistical controls to compensate for inadequate experimental controls has many problems. The quality ratings are not blind. As the differences between myself and Honorton reveal, such ratings are highly sub- jective. Although I tried my best to restrict my ratings to what I thought were objective and ea- sily codeable indicators, my quality ratings pro- vide a different picture than do those of Honorton. Honorton, I am sure, believes he was just as objective in assigning his ratings as I believe I was. Another problem is the number of different prop- erties that are rated. Honorton’s rati ity omitted many attributes that I included in my ratings. Even in those cases where we used the same indicators to make our assessments, we differed because of our scaling. For example, on adequacy of randomization I used a simple dicho- tomy. Either the experimenter clearly indicated using an appropriate randomization procedure or he did not. Honorton converted this to a trichoto- mous scale. He distinguished between a clearly inadequate procedure such as hand-shuffling and failure to report how the randomization was done. He then assigned the lowest rating to failure to describe the randomization. In his scheme, clearly inadequate randomization was of higher quality than failure to describe the procedure. Although we agreed on which experiments had adequate ran- domization, inadequate randomization or inade- quate documentation, the different ways these were ordered produced important differences between us in how randomization related to effect size. These are just some of the reasons why the finding of no correlation between effect size and rated quality does not justify concluding that the observed flaws had no effect. I will now consider some of Utts’ assertions and hope that I can go into more detail in anoth- er forum. Utts discusses the conclusions of the National Research Council’s Committee on Techniques for the Enhancement of Human Per- formance. I was chairperson of that committee’s subcommittee on paranormal phenomena. She wrongly states that we restricted our evaluation only to significant studies. I do not know how she got such an impression since we based our analysis on meta-analyses whenever these were available. The two major inputs for the committee’s evalua- tion were a lengthy evaluation of contemporary parapsychology experiments by John Palmer and an independent assessment of these experiments by James Alcock. Our spdnsors, the Army Research Institute had commissioned the report from the parapsychologist John Palmer. They specifically asked our committee to provide a second opinion from a non-parapsychological perspective. They were most interested in'the experiments on remote viewing and random number generators. We de- cided to add the ganzfeld experiments. Alcock was instructed, in making ‘his evaluation, to restrict himself to the same experiments in these categories that Palmer had chosen. In this way, the experi- ments we evaluated, which included both signifi- cant and nonsignificant ones, were, in effect, selected for us by a prominent parapsychologist. Utts mistakenly asserts that my subcommittee on parapsychology commissioned Harris and Rosen- f ]- hal t luat isychol i ts fe Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION OF PARAPSYCHOLOGY 391 us. Harris and Rosenthal were commissioned by our evaluation subcommittee to write a paper on evaluation issues, especially those related to exper- imenter effects. On their own initiative, Harris and Rosenthal surveyed a number of data bases to illus- trate the application of methodological procedures such as meta-analysis. As one illustration, they included a meta-analysis of the subsample of ganzfeld experiments used by Honorton in his rebuttal to my critique. Because Harris and Rosenthal did not them- selves do a first-hand evaluation of the ganzfeld experiments, and because they used Honorton’s rat- ings for their illustration, I did not refer to their analysis when I wrote my draft for the chapter on the paranormal. Rosenthal told me, in 2 letter, that he had arbitrarily used Honorton’s ratings rather than mine because they were the most recent avail- able. I assumed that Harris and Rosenthal were using Honorton’s sample and ratings to illustrate meta-analytic procedures. I did not believe they were making a substantive contribution to the debate. Only after the committee’s complete report was in the hands of the editors did someone become concerned that Harris and Rosenthal had come to a conclusion on the ganzfeld experiments different from the committee. Apparently one or more com- mittee members contacted Rosenthal and asked him to explain why he and Harris were dissenting. Because some committee members believed that we should deal with this apparent discrepancy, I contacted Rosenthal and pointed out if he had used my ratings with the very same analysis he had applied to Honorton’s ratings, he would have reached a conclusion opposite to what Harris and he had asserted. I did this, not to suggest my ratings were necessarily more trustworthy than Honorton’s, but to point out how fragile any conclu- sions were based on this small and limited sample. Indeed, the data were so lacking in robustness that the difference between my rating and Honorton’s rating of one investigator (Sargent) on one at- tribute (randomization) sufficed to reverse the con- clusions Harris and Rosenthal made about the correlation between quality and effect size. Harris and Rosenthal responded by adding a foot- note to their paper. In this footnote, they repor- ted an analysis using my ratings rather than Honorton’s. This analysis, they concluded, still sup- ported the null hypothesis of no correlation be- tween quality and effect size. They used 6 of my 12 dichotomous ratings of flaws as predictors and the z score and effect size as criterion variables in both multiple regression and canonical correlation anal- yses. The reported an as pe canonical corre- pproved For lation between criterion variables and flaws of “only” 0.46. A true correlation of this magnitude would be impressive given the nature and split of the dichotomous variables. But, because it was not statistically significant, Harris and Rosenthal con- cluded that there was no relationship between quality and effect size. A canonical correlation on this sample of 28 nonindependent cases, of course, has virtually no chance of being significant, even if it were of much greater magnitude. What this amounts to is that the alleged contra- dictory conclusions of Harris and Rosenthal are based on a meta-analysis that supports Honorton’s position when Honorton’s ratings are used and supports my position when my ratings are used. Nothing substantive comes from this, and it is redundant with what Honorton and I have already published. Harris and Rosenthal’s footnote adds nothing because it supports the null hypothesis with a statistical test that has no power against a reasonably sized alternative. It is ironic that Utts, after emphasizing the importance of considering statistical power, places so much reliance on the outcome of a powerless test. (I should add that the recurrent charge that the NRC committee completely ignored Harris and Rosenthal’s conclusions is not strictly correct. I wrote a response to the Harris and Rosenthal paper that was included in the same supplementary volume that contains their commissioned paper.) Utts’ discussion of the ganzfeld debate, as I have indicated, also shows unfamiliarity with details. She cites my factor analysis and Saunders’ critique as if these somehow jeopardized the conclusions I drew. Again, the matter is too complex to discuss adequately in this forum. The “factor analysis’ she is talking about is discussed in a few pages of my critique. I introduced it as a convenient way to summarize my conclusions, none of which depended on this analysis. I agree with what Saunders has to say about the limitations of factor analysis in this context. Unfortunately, Saunders bases his criti- cism on wrong assumptions about what I did and why I did it. His dismissal of the results as “‘meaningless” is based on mistaken algebra. I in- cluded as dummy variables five experimenters in the factor analysis. Because an experimenter can only appear on one variable, this necessarily forces the average intercorrelation among the experi- menter variables to be negative. Saunders falsely asserts that this negative correlation must be —-1. If he were correct, this would make the results meaningless. But he could be correct only if there were just two investigators and that each one ac- counted for 50% of the experiments. In my case, as I made sure to check ahead of time, the use of five elease 2000/08/08 : CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 392 J. UTTS experimenters, each of whom contributed only a few studies to the data base, produced a mildly negative intercorrelation of —0.147. To make sure even that small correlation did not distort the re- sults, I did the factor analysis with and without the dummy variables. The same factors were obtained in both cases. However, I do not. wish to defend this factor analysis. None of my conclusions depend on it. I would agree with any editor who insisted that I omit it from the paper on the grounds of redun- dancy. I am discussing it here as another example that suggests that Utts is not familiar with some relevant details in literature she discusses. CONCLUSIONS Utts may be correct. There may indeed be an anomaly in the parapsychological findings. Anoma- lies may also exist in non-parapsychological do- mains. The question is when is an anomaly worth taking seriously. The anomaly that Utts has in mind, if it exists, can be described only as a depar- ture from a generalized statistical model. From the evidence she presents, we might conclude that we are dealing with a variety of different anomalies instead of one coherent phenomenon. Clearly, the reported effect sizes for the experiments with ran- dom number generators are orders of magnitude lower than those for the ganzfeld experiments. Even within the same experimental domain, the effect sizes do not come from the same population. The effects sizes obtained by Jahn are much smaller than those obtained by Schmidt with similar ex- periments on random number generators. In the ganzfeld experiments, experimenters differ significantly in the effect sizes each obtains. This problem of what effect sizes are and what they are measuring points to a problem for para- psychologists. In other fields of science such as astronomy, an “anomaly” is a very precisely speci- fied departure from a well-established substantive theory. When Leverrier discovered Neptune by studying the perturbations in the orbit of Uranus, he was able to characterize the anomaly as a very precise departure of a specific kind from the orbit expected on the basis of Newtonian mechanics. He knew exactly what he had to account for. The “anomaly” or “anomalies” that Utts talks about are different. We do not know what it is that we are asked to account for other than something that sometimes produces nonchance departures from a statistical mode], whose appropriateness is itself open to question. ° The case rests on a handful of meta-analyses that suggest effect sizes different from zero and uncorre- lated with some non-blindly determined indices of quality. For a variety of reasons, these retrospec- tive attempts to find evidence for paranormal phe- nomena are problematical. At best, they should provide the basis for parapsychologists designing prospective studies in which they can specify, in advance, the complete sample space and the critical region. When they get to the point where they can specify this along with; some boundary conditions and make some reasonable predictions, then they will have demonstrated something worthy of our attention. In this context, I agree with Utts that Honorton’s recent report of his automated ganzfeld experi- ments is a step in the right direction. He used the ganzfeld meta-analyses and the criticisms of the existing data base to design better experiments and make some predictions. Although he and Utts be- lieve that the findings of meaningful effect sizes in the dynamic targets and a lack of a nonzero effect size in the static targets are somehow consistent with previous ganzfeld results, I disagree. I believe the static targets are closer in spirit to the original data base. But this is a/minor criticism. Honorton’s experiments have produced intrigu- ing results. If, as Utts suggests, independent labo- ratories can produce similar results with the same relationships and with the same attention to rigor- ous methodology, then parapsychology may indeed have finally captured its elusive quarry. Of course, on several previous occasions in its century-plus history, parapsychology has felt it was on the threshold of a breakthrough. The breakthrough never materialized. We will have to patiently wait to see if the current situation is any different. Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION OF PARAPSYCHOLOGY 393 Comment Robert L. Morris Experimental sciences by their nature have found it relatively easy to deal with simple closed sys- tems. When they come to study more complex, open systems, however, they have more difficulty in gen- erating testable models, must rely more on multi- variate approaches, have more diversity from experiment to experiment (and thus more difficulty in constructing replication attempts), have more noise in the data, and more difficulty in construct- ing a linkage between concept and measurement. Data gatherers and other researchers are more likely to be part of the system themselves. Exam- ples include ecology, economics, social psychology and parapsychology. Parapsychology can be re- garded as the study of apparent new means of communication, or transfer of influence, between organism and environment. Any observer attempt- ing to decide whether or not such psychic communi- cation has taken place is one of several elements in a complex open system composed of an indefinite number of interactive features. The system can be modeled, as has been done elsewhere (e.g., Morris, 1986) such as to organise our understanding of how observers can be misled by themselves, or by delib- erate frauds. Parapsychologists designing experi- mental studies must take extreme care to ensure that the elements in the experimental system do not interact in unanticipated ways to produce arti- fact or encourage fraudulent procedures. When re- searchers follow up the findings of others, they must ensure that the new experimental system sufficiently resembles the earlier one, regarding its important components and their potential interac- tions. Specifying sufficient resemblance is more dif- ficult in complex and open systems, and in areas of research using novel methodologies. As a result, parapsychology and other such areas may well profit from the application of modern meta-analysis, and meta-analytic methods may in turn profit from being given a good stiff workout by controversial data bases, as suggested by Jessica Utts in her article. Parapsychology would appear to gain from meta-analytic techniques, in at least three important areas. First, in assessing the question of replication rate, the new focus on effect size and confidence Robert L. Morris occupies the Koestler Chair of Parapsychology in the Department of Psychology at the Uninnpron dd rot Rele see 2000 G8 Cl inburgh 9FZ, United King: intervals rather than arbitrarily chosen signifi- cance levels seems to indicate much greater consis- tency in the findings than has previously been claimed. Second, when one codes the individual studies for flaws and relates flaw abundance with effect size, there appears to be little correlation for all but one data base. This contradicts the frequent assertion that parapsychological results disappear when methodology is tightened. Additional evidence on this point is the series of studies by Honorton and associates using an automated ganzfeld procedure, apparently better conducted than any of the previ- ous research, which nevertheless obtained an effect size very similar to that of the earlier more diverse data base. Third, meta-analysis allows researchers to look at moderator variables, to build a clearer picture of the conditions that appear to produce the strongest effects. Research in any real scientific discipline must be cumulative, with later researchers build- ing on the work of those who preceded them. If our earlier successes and failures have meaning, they should help us obtain increasingly consistent, clearer results. If psychic ability exists and is suffi- ciently stable that it can be manifest in controlled experimental studies, then moderator variables should be present in groups of studies that would indicate conditions most favourable and least favourable to the production of large effect sizes. From the analyses presented by Utts, for instance, it seems evident that group studies tend to produce poor results and, however convenient it may be to conduct them, future researchers should apparently focus much more on individual testing. When doing ganzfeld studies, it appears best to work with dy- namic rather than static target material and with experienced participants rather than novices. If such results are valid, then future researchers who wish to get strong results now have a better idea of what procedures to select to increase the likelihood of so doing, what elements in the experimental system seem most relevant. The proportion of stud- ies obtaining positive results should therefore increase. However, the situation may be more complex than the somewhat ideal version painted above. As noted earlier, meta-analysis may learn from para- psychology as well as vice versa. Parapsychological data may well give meta-analytic techniques a good workout and will certainly pose some challenges. SRD PSR OUTS TRIO RTO BOOTS ase. or ASR