Approved For Release 2000/08/08 : CIA-RDP96- 00789R003100010001 -6 402 J. UTTS chance. The final reason it is instructive to exam- ine these Bayes factors is that they provide a quan- titative challenge to skeptics to be explicit about their prior probabilities for the null and alternative hypotheses. Dawson discussed the use of more complex Bayesian models for the analysis of the auto- ganzfeld data. She proposed a hierarchical model where the number of successes for each experiment followed a binomial distribution with hit rate p,, and logit( p;) came from a normal distribution with noninformative priors for the mean and variance. She then expanded this model to include heavier tails by allowing an additional scale parameter for each experiment. Her rationale for this expanded model was that there were clear outlier series in the data. The hierarchical model proposed by Dawson is a reasonable place to start given only that there were several experiments trying to measure the same effect, conducted by different investigators. In the autoganzfeld database, the model could be ex- panded to incorporate the additional information available. Each experiment contained some ses- sions with static targets and some with dynamic targets, some sessions in which the sender and receiver were friends and others in which they were not and some information about the extraver- sion score of the receiver. All of this information could be included by defining the individual session as the unit of analysis, and including a vector of covariates for each session. It would then make sense to construct a logistic regression model with a component for each experiment, following the model proposed by Dawson, and a term X86 to include the covariates. A prior distribution for 6 could include information from earlier ganzfeld studies. The advantage of using a Bayesian ap- proach over a simple logistic regression is that information could be continually updated. Some of the recent work in Bayesian design could then be incorporated so that future trials make use of the best conditions. Several of the discussants addressed the concept of replication. I agree with Mosteller’s implication that it was unwise for the audience in my seminar to respond to my replication questions so quickly, and that was precisely my point. Most nonstatisti- cians do not seem to understand the complexity of the replication question. Parenthetically, when I posed the same scenario to an audience of statis- ticians, very few were willing to offer a quick opinion. Bayarri and Berger provided an insightful dis- cussion of the purpose of replication, offering quan- titative answers to questions that were implicit in my discussion. Their analyses suggest some alter- natives to power analysis that might be considered when designing a new study to try to replicate a questionable result. Morris addressed the question of what con- stitutes a replication: of a meta-analysis. He distinguished between exact and conceptual repli- cations. Using his distinction, the autoganzfeld meta-analysis could be viewed as a conceptual replication of the earlier ganzfeld meta-analysis. He noted that when such a conceptual replication offers results similar to those of the original meta-analysis, it lends: legitimacy to the original results, as was the case with the autoganzfeld meta-analysis. Greenhouse and Mori is both noted the value of meta-analysis as a method of comparing different conditions, and I endorse that view. Conditions found to produce different effects in one meta- analysis could be explicitly studied in a conceptual replication. One of the intriguing results of the autoganzfeld experiments was that they supported the distinction between effect sizes for dynamic versus static targets found in the earlier ganzfeld work, and they supported the relationship between ESP and extraversion found in the meta-analysis by Honorton, Ferrari and Bem (1990). Most modern parapsychologists, as indicated by Morris, recognize that demonstrating the validity of their preliminary findings will depend on identi- fying and utilizing “moderator variables” in future studies. The use of such. variables will require more complicated statistical models than the simple bi- nomial models used in: the past. Further, models are needed for combining results from several dif- ferent experiments, that don’t oversimplify at the expense of lost information. In conclusion, the anomalous effect that persists throughout the work reviewed in my paper will be better understood only after further experimenta- tion that takes into account the complexity of the system. More realistic, and thus more complex, models will be needed, to analyze the results of those experiments. This presents a challenge that I hope will be welcomed hy the statistics community. ADDITIONAL REFERENCES ALLIson, P, (1979). Experimental parapsychology as a rejected science. The Sociological Review Monograph 27 271-291. Barser, B. (1961). Resistance by scientists to scientific discov- ery. Science 184 596-602. Bercer, J. O. and DELAMPaADY, M. (1987). Testing precise hy- potheses (with discussion). Statist, Sci. 2 317-352. Cuuna, F. R. K., Diaconis, P., Granam, R. L. and Mattows, C. L. (1981). On the pérmanents of compliments of the direct sum of identity matrices. Adv, Appl. Math. 2121-187. Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION IN PARAPSYCHOLOGY 403 Cocuran, W. G. (1954). The combination of estimates from different experiments. Biometrics 10 101-129. Co.iins, H. and Pincu, T. (1979). The construction of the para- normal: Nothing unscientific is happening. The Sociological Review Monograph 27 237-270. Cou.ins, H. M. and Pincu, T. J. (1982). Frames of Meaning: The Social Construction of Extraordinary Science. Routledge & Kegan Paul, London. CorNFIELD, J. (1959). Principles of research. American Journal of Mental Deficiency 64 240-252. Dempster, A. P., SeLwyn, M. R. and Weexs, B. J. (1983). Combining historical and randomized controls for assessing trends in proportions. J. Amer. Statist. Assoc. 78 221-227. Diaconis, P. and GRAHAM, R. L. (1981). The analysis of sequen- tial experiments with feedback to subjects. Ann. Statist. 9 236-244. FisHer, R, A. (1982). Statistical Methods for Research Workers, 4th ed. Oliver and Boyd, London. Fisuer, R. A. (1935). Has Mendel’s work been rediscovered? Ann. of Sci. 1 116-187. GaLTon, F. (1901-2). Biometry. Biometrika 1 7-10. GREENHOUSE, J., Fromm, D., Ivencar, S., Dew, M. A., HoLuanp, A. and Kass, R. (1990). Case study: The effects of rehabili- tation therapy for aphasia. In The Future of Meta-Analysis (K. W. Wachter and M. L. Straf, eds.) 31-32. Russell Sage Foundation, New York. GrirFin, D. (1988). Intuitive judgment and the evaluation of evidence. In Enhancing Human Performance: Issues, Theo- ries and Techniques Background Papers—Part I. National Academy Press, Washington, D.C. Hansen, G. (1990). Deception by subjects in psi research. Jour- nal of the American Society for Psychical Research 84 25-80. Hunter, J. and Scumipt, F. (1990). Methods of Meta-Analysis. Sage, London. Ivencar, S. and GREENHOUSE, J. (1988). Selection models and the file drawer problem (with discussion). Statist. Sci. 3 109-135. Louts, T. A. (1984). Estimating an ensemble of parameters using Bayes and empirical Bayes methods. J. Amer. Statist. Assoc. 79 393-398. ManTEL, N. and HAENSZEL, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Jour- nal of the National Cancer Institute 22 719-748. Morris, C. (1983). Parametric empirical Bayes inference: The- ory and applications (rejoinder) J. Amer. Statist. Assoc. 78 47-65. Morris, R. L. (1986). What psi is not: The necessity for experi- ments. In Foundations of Parapsychology (H. L. Edge, R. L. Morris, J. H. Rush and J. Palmer, eds.) 70-110. Routledge & Kegan Paul, London. MosTELLER, F. and Busy R. R. (1954). Selected quantitative techniques. In Handbook of Social Psychology (G. Lindzey, ed.) 1 289-334. Addison-Wesley, Cambridge, Mass. Mosreiter, F. and CHALMeErs, T. (1991). Progress and problems in meta-analysis. Statist. Sci. To appear. Oren, L., ed. (1975). Quantum Physics and Parapsychology. Parapsychology Foundation, New York. Pincu, T. J. and Couuins, H. M. (1984). Private science and public knowledge: The Committee for the Scientific Investi- gation of Claims of the Paranormal and its use of the literature. Social Studies of Science 14 521-546. Piatt, J. R. (1964). Strong inference. Science 146 347-353. RosENTHAL, R. (1966). Experimenter Effects in Behavioral Re- search, Appleton-Century-Crofts, New York. ROSENTHAL, R. (1979). The ‘‘file drawer problem” and tolerance for null results. Psychological Bulletin 86 638-641. Ryan, L. M. and Dempster, A. P. (1984). Weighted normal plots. Technical Report 394Z, Dana-Farber Cancer Inst., Boston, Mass. SAMANIEGO, F. J. and Urrts, J. (1983). Evaluating performance in continuous experiments with feedback to subjects. Psy- chometrika 48 195-209. Smitu, M. and Guass, G. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist 32 752-760. Wacuter, K. (1988). Disturbed by meta-analysis? Science 241 1407-1408. West, M. (1985). Generalized linear models: Scale parameters, outlier accommodation and prior distributions. In Bayesian Statistics 2 (J. M. Bernardo, M. H. DeGroot, D. V. Lindley, and A. F. M. Smith, eds.) 531-558. North-Holland Amster- dam. Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6