Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 ' 378 J. UTTS Gumore, J. B. (1989). Randomness and the search for psi. Journal of Parapsychology 63 309-340. GitmoreE, J. B. (1990). Anomalous significance in pararandom and psi-free domains. Journal of Parapsychology 54 53-58. GreeLey, A. (1987). Mysticism goes mainstream. American Health 7 47-49. GREENHOUSE, J. B. and GREENHOUSE, S. W. (1988). An aspirin a day...? Chance 1 24-31. GrEeenwoop, J. A. and Stuart, C. BE. (1940). A review of Dr. Feller’s critique. Journal of Parapsychology 4 299-319. Hackine, I. (1988). Telepathy: Origins of randomization in ex- perimental design. Isis 79 427-461. HansEL, C. E. M. (1980). ESP and Parapsychology: A Critical Re-evaluation. Prometheus Books, Buffalo, N.Y. Harris, M. J. and RosenTHAL, R. (1988a). Interpersonal Ex- pectancy Effects and Human Performance Research. Na- tional Academy Press, Washington, D.C. Harris, M. J. and Rosentuat, R. (1988b). Postscript to Interper- sonal Expectancy Effects and Human Performance. Research. National Academy Press, Washington, D.C. Hepces, L. V. and Ovkin, I. (1985). Statistical Methods for Meta-Analysis. Academic, Orlando, Fla. Honorton, C. (1977). Psi and internal attention states. In Handbook of Parapsychology (B. B. Wolman, ed.) 435-472. Van Nostrand Reinhold, New York. Honorton, C. (1985a). How to evaluate and improve the repli- cability of parapsychological effects. In The Repeatability Problem in Parapsychology (B. Shapin and L. Coly, eds.) 238-255. Parapsychology Foundation, New York. Honorton, C. (1985b). Meta-analysis of psi ganzfeld research: A response to Hyman. Journal of Parapsychology 49 51-91. Honorton, C., Bercer, R. E., VArvocuis, M. P., Quant, M., Derr, P., ScHECHTER, E. I. and Ferrari, D. C. (1990). Psi communication in the ganzfeld: Experiments with an automated testing system and a comparison with a meta- analysis of earlier studies. Journal of Parapsychology 54 99-139. HonortTon, C. and Ferrari, D. C. (1989). “Future telling’: A meta-analysis of forced-choice precognition experiments, 1935-1987. Journal of Parapsychology 53 281-308. Honorton, C., Ferrari, D. C. and Bem, D. J. (1991). Extraver- sion and ESP performance: A meta-analysis and a new confirmation. Research in Parapsychology 1990. The Scare- crow Press, Metuchen, N.J. To appear. Hyman, R. (1985a). A critical overview of parapsychology. In A Skeptic's Handbook of Parapsychology (P. Kurtz, ed.) 1-96. Prometheus Books, Buffalo, N.Y. Hyman, R. (1985b). The ganzfeld psi experiment: A critical appraisal. Journal of Parapsychology 49 3-49. Hyman, R. and Honorton, C. (1986). Joint communiqué: The psi ganzfeld controversy. Journal of Parapsychology 50 351-364. Iversen, G. R., Lonccor, W. H., Moste.ver, F., Giusert, J. P. and Youtz, C. (1971). Bias and runs in dice throwing and recording: A few million throws. Psychometrika 36 1-19. JEFFREYS, W. H. (1990). Bayesian analysis of random event generator data. Journal of Scientific Exploration 4 153-169. Linb.ey, D. V. (1957). A statistical paradox. Biometrika 44 187-192. Mauskopr, 8S. H. and McVauau, M. (1979). The Elusive Science: Origins of Experimental Psychical Research. Johns Hopkins Univ. Press. McVaueu, M. R. and Mauskopr, S. H. (1976). J. B. Rhine's Extrasensory Perception and its background in psychical research. Isis 67 161-189. Neuuiep, J. W., ed. (1990). Handbook of replication research i Approved For Release 2000/08/08 : CIA-R the behavioral and social sciences. Journal of Social Behav- ior and Personality 5 (4) 1-510. OFFICE OF TECHNOLOGY ASSESSMENT (1989). Report of a work- shop on experimental parapsychology. Journal of the Amer- ican Society for Psychical! Research 83 317-339. PALMER, J. (1989). A reply to} Gilmore. Journal of Parapsychol- ogy 53 341-344. : PALMER, J. (1990). Reply to ‘Gilmore: Round two. Journal of Parapsychology 54 59-61; Paumer, J. A., Honorton, C.. and Urrs, J. (1989). Reply to the National Research Council study on parapsychology. Jour- nal of the American Society for Psychical Research 83 31-49. Rapin, D. I. and Ferrari, D. C. (1991). Effects of consciousness on the fall of dice: A méta-analysis. Journal of Scientific Exploration & 61-83. : Rapin, D. I. and Netson, R. D. (1989). Evidence for conscious- ness-related anomalies in random physical systems. Foun- dations of Physics 19 1499-1514. Rao, K. R. (1985). Replication in conventional and controversial sciences. In The Repeatability Problem in Parapsychology (B. Shapin and L. Coly, dds.) 22-41. Parapsychology Foun- dation, New York. Rune, J. B. (1984). Extrasensory Perception. Boston Society for Psychical Research, Boston. (Reprinted by Branden Press, 1964.) Rune, J. B. (1977). History of experimental studies. In Hand- book of Parapsychology (B. B. Wolman, ed.) 25-47. Van Nostrand Reinhold, New :York. Ricuet, C. (1884). La suggestion mentale et le calcul des proba- bilités. Revue Philosophique 18 608-674. RosentHAL, R. (1984). Meta-Analytic Procedures for Social Re- search. Sage, Beverly Hills. RosentTua., R. (1986). Meta-analytic procedures and the nature of replication: The ganzfeld debate. Journal of Parapsychol- ogy 50 315-336. : RosenTua, R. (1990a). How: are we doing in soft psychology? American Psychologist 45 775-777. RosexTHAL, R, (1990b). Replication in behavioral research. Journal of Social Behavior and Personality 5 1-30. Sacunpers, D. R. (1985). On Hyman’s factor analysis. Journal of Parapsychology 49 86-88: Suapin, B. and Coty, L., eds.:(1985). The Repeatability Problem in Parapsychology. Parapsychology Foundation, New York. SpenceR-Brown, G. (1957). Probability and Scientific Inference. Longmans Green, London and New York. Stuart, C. E. and Greenwogp, J. A. (1937). A review of criti- cisms of the mathematical evaluation of ESP data. Journal of Parapsychology 1 295-304. Tversky, A. and KAHNEMAN, D. (1982). Belief in the law of small numbers. In Judgment Under Uncertainty: Heuristics and Biases (D. Kahneman, P. Slovic and A. Tversky, eds.) 23-31. Cambridge Univ. Press. Urts, J. (1986). The ganzfeld debate: A statistician’s perspec- tive. Journal of Parapsychology 50 395-402. Urrs, J. (1987). Psi, statistics, and society. Behavioral and Brain Sciences 10 615- 616. Urrts, J. (1988). Successful replication versus statistical signifi- cance. Journal of Parapsychology 52 305-320. Urts, J. (1989). Randomness and randomization tests: A reply to Gilmore. Journal of Parapsychology 53 345-351. Urts, J. (1991). Analyzing free-response data: A progress report. In Psi Research Methodology: A Re-examination (L. Coly, ed.). Parapsychology Foundation, New York. To appear. Wiiks, 8. S. (1965a). Statibtical aspects of expeirments in telepath. N.Y. Statistician 16 (6) 1-3. Wiixs, S. S. (1965b). Statistical aspects of experiments in DPS N.Y. Statistician 16 (7) 4-6. P96- 00789R003100010001 -6
_ Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 J. UTTS Comment M. J. Bayarri and James Berger 1. INTRODUCTION There are many fascinating issues discussed in this paper. Several concern parapsychology itself and the interpretation of statistical methodology therein. We are not experts in parapsychology, and so have only one comment concerning such mat- ters: In Section 3 we briefly discuss the need to switch from P-values to Bayes factors in discussing evidence concerning parapsychology. A more general issue raised in the paper is that of replication. It is quite illuminating to consider the issue of replication from a Bayesian perspec- tive, and this is done in Section 2 of our discussion. 2. REPLICATION Many insightful observations concerning replica- tion are given in the article, and these spurred us to determine if they could be quantified within Bayesian reasoning. Quantification requires clear delineation of the possible purposes of replication, and at least two are obvious. The first is simple reduction of random error, achieved by obtaining more observations from the replication. The second purpose is to search for possible bias in the original experiment. We use “bias” in a loose sense here, to refer to any of the huge number of ways in which the effects being measured by the experiment can differ from the actual effects of interest. Thus a clinical trial without a placebo can suffer a placebo “bias”; a survey can suffer a “bias” due to the sampling frame being unrepresentative of the actual population; and possible sources of bias in parapsychological experiments have been extensively discussed. Replication to Reduce Random Error If the sole goal of replication of an experiment is to reduce random error, matters are very straight- forward. Reviewing the Bayesian way of studying this issue is, however, useful and will be done through the following simple example. M. J. Bayarri is Titular Professor, Department of Statistics and Operations Research, University of Valencia, Avenida Dr. Moliner 50, 46100 Burjassot, Valencia, Spain. James Berger is the Richard M. Brumfield Distinguished Professor of Statistics, Purdue UNpproverdk— on/fRelease!2000/G8108 379 EXAMPLE 1. Consider the example from Tversky and Kahnemann (1982), in which an experiment results in a standardized test statistic of z, = 2.46. (We will assume normality to keep computations trivial.) The question is: What is the highest value of z, in a second set of data that would be consid- ered a failure to replicate? Two possible precise versions of this question are: Question 1: What is the probability of observing z,. for which the null hypothesis would be rejected in the replicated ex- periment? Question 2: What value of z, would leave one’s overall opinion about the null hypothe- sis unchanged? Consider the simple case where Z, ~ N(z,|@, 1) and (independently) Z, ~ N(z,|6, 1), where @ is the mean and 1 is the standard deviation of the normal distribution. Note that we are considering the case in which no experimental bias is suspected and so the means for each experiment are assumed to be the same. Suppose that it is desired to test Hy: 6 = 0 versus H,:6 > 0, and suppose that initial prior opinion about @ can be described by the noninformative prior 7(?) = 1. We consider the one-sided testing problem with a constant prior in this section, be- cause it is known that then the posterior probabil- ity of H,, to be denoted by P( H, |data), equals the P-value, allowing us to avoid complications arising from differences between Bayesian and classical answers. After observing z, = 2.46, the posterior distribu- tion of 6 is x(6| z,) = N(0|2.46, 1). Question 1 then has the answer (using predictive Bayesian reasoning) P(rejecting at level a| z,) ao a0 1 1 (| V2 ? where ® is the standard normal cdf and c, is the (one-sided) critical value corresponding to the level, a, of the test. For instance, if a = 0.05, then this probability equals 0.7178, demonstrating that there is a quite substantial probability that the second experiment will fail to reject. If a is chosen to be the observed significance level from the first exper- : ClAcRDR96200789R G0:5:1000 0100016 that the
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 380 second experiment will reject is just 1/2. This is nothing but a statement of the well-known martin- gale property of Bayesianism, that what you ‘‘ex pect” to see in the future is just what you know today. In a sense, therefore, question 1 is exposed as being uninteresting. Question 2 more properly focuses on the fact that the stated goal of replication here is simply to reduce uncertainty in stated conclusions. The an- swer to the question follows immediately from not- ing that the posterior from the combined data (2, 2) is N(8|(2, + 22)/2, 1/v2), (0 | 21, Z2) = so that P(H, |data) = 6(-(z, + z2)/V2). Setting this equal to P(H,| z,) and solving for z, yields z, = (V2 - Izy = 1.02. Any value of z, greater than this will increase the total evidence against Hy, while any value smaller than 1.02 will decrease the evidence. Replication to Detect Bias The aspirin example dramatically raises the is- sue of bias detection as a motive for replication. Professor Utts observes that replication 1 gives results that are fully compatible with those of the original study, which could be interpreted as sug- gesting that there is no bias in the original study, while replication 2 would raise serious concerns of bias. We became very interested in the implicit suggestion that replication 2 would thus lead to less overall evidence against the null hypothesis than would replication 1, even though in isolation replication 2 was much more “significant” than was replication 1. In attempting to see if this is so, we considered the Bayesian approach to study of bias within the framework of the aspirin example. EXAMPLE 2. For simplicity in the aspiring exam- ple, we reduce consideration to @ = true difference in heart attack rates between aspirin and placebo populations multiplied by 1000; Y = difference in observed heart attack rates be- tween aspirin and placebo groups in original study multiplied by 1000; X, = difference in observed heart attack rates be- tween aspirin and placebo groups in Replica- tion i multiplied by 1000. We assume that the replication studies are | ex tremely well designed and implemented, so J. UTTS one is very confident that the X; have mean 6. Using normal approximations for convenience, the data can be summarized as X,~ N(x,/6, 4.82), X, ~ N(x_|6, 3.63) with actual observations x, = 7.704 and x, = 13.07. Consider now the bias i issue. We assume that the original experiment is) somewhat suspect in this regard, and we will model bias by defining the mean of Y to be n=0+8, where @ is the unknown bias. Then the data in the original experiment can be summarized by Y ~ N(y|7, 1.54), with the actual observation being y = 7.707. Bayesian analysis requires specification of a prior distribution, +(8), for the suspected amount of bias. Of particular interest then are the posterior distri- bution of 6, assuming replication i has been performed, given by a(B| Yy> x;) w(B)ex) — 37 ga? +o?) where o? is the variance (4.82 or 3.63) from repli- cation 1; and the posterior probability of H,, given by P(Aoly, x;) oo Gg. = &| - ——_-——— (y- 8 i | 1.54 rise | ) 1.54 0, Vo? + 1.54? Recall that our goal here was to see if Bayesian analysis can reproduce the intuition that the origi- nal experiment could be trusted if replication 1 had been done, while it could not be trusted (in spite of its much larger sample ‘size) had replication 2 been performed. Establishing this requires finding a prior distribution 7(8) for which 7(8| y, x,) has little effect on P(Ho|y, x,), but 7(8| y, x2) has a large effect on P(H,| y, x2). To achieve the first objective, (8) must be tightly concentrated near zero. To achieve the second, 7(8) must be such that large | y — x,|, which suggests presence of a large bias, can result in a substantial shift of posterior r §@ away from [8 - (y - x,)]” ’ x;|7(8| y, x;) dB. Approved For Release 5000/08/08 : ClA- RDP96- 00789R003100010001 -6
' Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION IN PARAPSYCHOLOGY 381 A sensible candidate for the prior density 7(() is the Cauchy (0, V) density 1 rv(6) aV({1+(0/V)] Flat-tailed densities, such as this, are well known to have the property that when discordant data is observed (e.g., when (| y — x.| is large), substan- tial mass shifts away from the prior center towards the likelihood center. It is easy to see that a normal prior for 8 can not have the desired behavior. Our first surprise in consideration of these priors was how small V needed to be chosen in order for P(H,|y, *,) to be unaffected by the bias. For instance, even with V = 1.54/100 (recall that 1.54 was the standard deviation of Y from the original experiment), computation yields P(H)|y, x,) = 4.3 x 10~5, compared with the P-value (and poste- rior probability from the original experiment as- suming no bias) of 2.8 x 107-7. There is a clear lesson here; even very small suspicions of bias can drastically alter a small P-value. Note that replica- tion 1 is very consistent with the presence of no bias, and so the posterior distribution for the bias remains tightly concentrated near zero; for in- stance, the mean of the posterior for @ is then 7.2 x 107°, and the standard deviation is 0.25. When we turned attention to replication 2, we found that it did not seriously change the prior perceptions of bias. Examination quickly revealed the reason; even the maximum likelihood estimate of the bias is no more than 1.4 standard deviations from zero, which is not enough to change strong prior beliefs. We, therefore, considered a third experiment, defined in Table 1. Transforming to approximate normality, as before, yields X, ~ N(x3|6, 3.48), with x, = 22.72 being the actual observation. The maximum likelihood estimate of bias is now 3.95 standard deviations from zero, so there is potential for a substantial change in opinion about the bias. Sure enough, computation when V = 1.54/100 yields that E[6!¥y, x3} = -—4.9 with (posterior) standard deviation equal to 6.62, which is a dra- matic shift from prior opinion (that 8 is Cauchy (0, TABLE 1 Frequency of heart attacks in replication 3 Yes No Aspirin 5 2309 Placebo 54 2116 1.54/100)). The effect of this is to essentially ignore the original experiment in overall assessments of evidence. For instance, P(Hp|y, x3) = 3.81 x 10-11, which is very close to P(H,| x3) = 3.29 x 107". Note that, if 8 were set equal to zero, the overall posterior probability of H,) (and P-value) would be 2.62 x 10715. Thus Bayesian reasoning can reproduce the intu- ition that replication which indicates bias can cast considerable doubt on the original experiment, while replication which provides no evidence of bias leaves evidence from the original experiment intact. Such behavior seems only obtainable, how- ever, with flat-tailed priors for bias (such as the Cauchy) that are very concentrated (in comparison with the experimental standard deviation) near zero. 3. P-VALUES OR BAYES FACTORS? Parapsychology experiments usually consider testing of H,: No parapsychological effect exists. Such null hypotheses are often realistically repre- sented as point nulls (see Berger and Delampady, 1987, for the reason that care must be taken in such representation), in which case it is known that there is a large difference between P-values and posterior probabilities (see Berger and Delampady, 1987, for review). The article by Jefferys (1990) dramatically illustrates this, showing that a very small P-value can actually correspond to evidence for H) when considered from a Bayesian perspec- tive. (This is very related to the famous “Jeffreys” paradox.) The argument in favor of the Bayesian approach here is very strong, since it can be shown that the conflict holds for virtually any sensible prior distribution; a Bayesian answer can be wrong if the prior information turns out to be inaccurate, but a Bayesian answer that holds for all sensible priors is unassailable. Since P-values simply cannot be viewed as mean- ingful in these situations, we found it of interest to reconsider the example in Section 5 from a Bayes factor perspective. We considered only analysis of the overall totals, that is, x = 122 successes out of n = 3565 trials. Assuming a simple Bernoulli trial model with success probability 4, the goal is to test Hy:6 = 1/4 versus H,:0 # 1/4. To determine the Bayes factor here, one must specify g(6), the conditional prior density on H,. Consider choosing g to be uniform and symmetric, that is, 1 G.(6) = Br’ fe , 6 : —-rs@s—+r, T4 r 4 r otherwise. Approved For Release 2000/08/08 : CIA-RDP96- 00789R003100010001 -6