‘ Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 Statistical Science 1991, Vol. 6, No. 4, 363-403 Replication and Meta-Analysis in Parapsychology Jessica Utts Abstract. Parapsychology, the laboratory study of psychic phenomena, has had its history interwoven with that of statistics. Many of the controversies in parapsychology have focused on statistical issues, and statistical models have played an integral role in the experimental work. Recently, parapsychologists have been using meta-analysis as a tool for synthesizing large bodies of work. This paper presents an overview of the use of statistics in parapsychology and offers a summary of the meta-analyses that have been conducted. It begins with some anecdotal information about the involvement of statistics and statisti- cians with the early history of parapsychology. Next, it is argued that most nonstatisticians do not appreciate the connection between power and “successful” replication of experimental effects. Returning to para- psychology, a particular experimental regime is examined by summariz- ing an extended debate over the interpretation of the results. A new set of experiments designed to resolve the debate is then reviewed. Finally, meta-analyses from several areas of parapsychology are summarized. It is concluded that the overall evidence indicates that there is an anoma- lous effect in need of an explanation. Key words and phrases: Effect size, psychic research, statistical contro- versies, randomness, vote-counting. 1. INTRODUCTION In a June 1990 Gallup Poll, 49% of the 1236 respondents claimed to believe in extrasensory per- ception (ESP), and one in four claimed to have had a personal experience involving telepathy (Gallup and Newport, 1991). Other surveys have shown even higher percentages; the University of Chicago’s National Opinion Research Center re- cently surveyed 1473 adults, of which 67% claimed that they had experienced ESP (Greeley, 1987). Public opinion is a poor arbiter of science, how- ever, and experience is a poor substitute for the scientific method. For more than a century, small numbers of scientists have been conducting labora- tory experiments to study phenomena such as telepathy, clairvoyance and precognition, collec- tively known as “psi” abilities. This paper will examine some of that work, as well as some of the statistical controversies it has generated. Jessica Utts is Associate Professor, Division of Statistics, University of California at Davis, 469 Kerr Ha avis, C App Parapsychology, as this field is called, has been a source of controversy throughout its history. Strong beliefs tend to be resistant to change even in the face of data, and many people, scientists included, seem to have made up their minds on the question without examining any empirical data at all. A critic of parapsychology recently acknowledged that “The level of the debate during the past 130 years has been an embarrassment for anyone who would like to believe that scholars and scientists adhere to standards of rationality and fair play” (Hyman, 1985a, page 89). While much of the controversy has focused on poor experimental design and potential fraud, there have been attacks and defenses of the statistical methods as well, sometimes calling into question the very foundations of probability and statistical inference. Most of the criticisms have been leveled by psy- chologists. For example, a 1988 report of the U.S. National Academy of Sciences concluded that ‘“‘The committee finds no scientific justification from research conducted over a period of 130 years for the existence of parapsychological phenomena” (Druckman and Swets, 1988, page 22). The chapter sychology was written by a subcommittee I 9 Ved ESP RereaSe 2000/08/08 : CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 364 J. UTTS chaired by a psychologist who had published a similar conclusion prior to his appointment to the committee (Hyman, 1985a, page 7). There were no parapsychologists involved with the writing of the report. Resulting accusations of bias (Palmer, Hon- orton and Utts, 1989) led U.S. Senator Claiborne Pell to request that the Congressional Office of Technology Assessment (OTA) conduct an investi- gation with a more balanced group. A one-day workshop was held on September 30, 1988, bring- ing together parapsychologists, critics and experts in some related fields (including the author of this paper). The report concluded that parapsychology needs “a fairer hearing across a broader spectrum of the scientific community, so that emotionality does not impede objective assessment of experimen- tal results” (Office of Technology Assessment, 1989). It is in the spirit of the OTA report that this article is written. After Section 2, which offers an anecdotal account of the role of statisticians and statistics in parapsychology, the discussion turns to the more general question of replication of experi- mental results. Section 3 illustrates how replica- tion has been (mis)interpreted by scientists in many fields. Returning to parapsychology in Section 4, a particular experimental regime called the “ganz- feld’’ is described, and an extended debate about the interpretation of the experimental results is discussed. Section 5 examines a meta-analysis of recent ganzfeld experiments designed to resolve the debate. Finally, Section 6 contains a brief account of meta-analyses that have been conducted in other areas of parapsychology, and conclusions are given in Section 7. 2. STATISTICS AND PARAPSYCHOLOGY Parapsychology had its beginnings in the investi- gation of purported mediums and other anecdotal claims in the late 19th century. The Society for Psychical Research was founded in Britain in 1882, and its American counterpart was founded in Boston in 1884. While these organizations and their members were primarily involved with investigat- ing anecdotal material, a few of the early re- searchers were already conducting “forced-choice” experiments such as card-guessing. (Forced-choice experiments are like multiple choice tests; on each trial the subject must guess from a small, known set of possibilities.) Notable among these was Nobel Laureate Charles Richet, who is generally credited with being the first to recognize that prob- ability theory could be applied to card-guessing experiments (Rhine, 1977, page 26; Richet, 1884). F. Y. Edgeworth, partly in response to what he ments, offered one of the earliest treatises on the statistical evaluation of forced-choice experiments in two articles published i in the Proceedings of the Society for Psychical Research (Edgeworth, 1885, 1886). Unfortunately, as noted by Mauskopf and McVaugh (1979) in their historical account of the period, Edgeworth’s papers were “perhaps too diffi- cult for their immediate audience” (page 105). Edgeworth began his ‘analysis by using Bayes’ theorem to derive the formula for the posterior probability that chance was operating, given the data. He then continued with an argument “savouring more of Bernoulli than Bayes” in which “it is consonant, I submit, to experience, to put 1/2 both for a and @,” that is, for both the prior proba- bility that chance alone was operating, and the prior probability that “there should have been some additional agency.” He then reasoned (using a Taylor series expansion of the posterior prob- ability formula) that if there were a large prob- ability of observing the data given that some additional agency was at work, and a small objec- tive probability of the data under chance, then the latter (binomial) probability “may be taken as a rough measure of the sought a posteriori probabil- ity in favour of mere chance” (page 195). Edge- worth concluded his article by applying his method to some data published previously in the same journal. He found the probability against chance to be 0.99996, which he said “may fairly be regarded as physical certainty” (page 199). He concluded: Such is the evidence which the calculus of probabilities affords as to the existence of an agency other than mere chance. The calculus is silent as to the nature of that agency —whether it is more likely to be vulgar illusion or ex- traordinary law. That is a question to be decided, not by formulae and figures, but by general philosophy and common sense [page 199]. Both the statistical arguments and the experi- mental controls in these early experiments were somewhat loose. For example, Edgeworth treated as binomial an experiment in which one person chose a string of eight letters and another at- tempted to guess the string. Since it has long been understood that people are poor random number (or letter) generators, there is no statistical basis for andlyzing such an experiment. Nonetheless, Edge- worth and his contempordries set the stage for the use of controlled experiments with statistical evalu- ation in laboratory parapsychology. An interesting historical account of Edgeworth’s involvement and the role telepathy experiments played in the early history of randomization and experimental design considered to&pprowed horhsRel eases? 0060/08/08 1sGlArRERYEI100789 R9O3100010001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION IN PARAPSYCHOLOGY 365 One of the first American researchers to use statistical methods in parapsychology was John Edgar Coover, who was the Thomas Welton Stanford Psychical Research Fellow in the Psychol- ogy Department at Stanford University from 1912 to 1937 (Dommeyer, 1975). In 1917, Coover pub- lished a large volume summarizing his work (Coover, 1917). Coover believed that his results were consistent with chance, but others have ar- gued that Coover’s definition of significance was too strict (Dommeyer, 1975). For example, in one evaluation of his telepathy experiments, Coover found a two-tailed p-value of 0.0062. He concluded, “Since this value, then, lies within the field of chance deviation, although the probability of its occurrence by chance is fairly low, it cannot be accepted as a decisive indication of some cause beyond chance which operated in favor of success in guessing” (Coover, 1917, page 82). On the next page, he made it explicit that he would require a p-value of 0.0000221 to declare that something other than chance was operating. It was during the summer of 1930, with the card-guessing experiments of J. B. Rhine at Duke University, that parapsychology began to take hold as a laboratory science. Rhine’s laboratory still exists under the name of the Foundation for Re- search on the Nature of Man, housed at the edge of the Duke University campus. It wasn’t long after Rhine published his first book, Extrasensory Perception in 1934, that the attacks on his methodology began. Since his claims were wholly based on statistical analyses of his experiments, the statistical methods were closely scrutinized by critics anxious to find a conventional explanation for Rhine’s positive results. The most persistent critic was a psychologist from McGill University named Chester Kellogg (Mauskopf and McVaugh, 1979). Kellogg’s main argument was that Rhine was using the binomial distribution (and normal approximation) on a se- ries of trials that were not independent. The experi- ments in question consisted of having a subject guess the order of a deck of 25 cards, with five each of five symbols, so technically Kellogg was correct. By 1937, several mathematicians and statis- ticians had come to Rhine’s aid. Mauskopf and McVaugh (1979) speculated that since statistics was itself a young discipline, ‘‘a number of statisticians were equally outraged by Kellogg, whose argu- ments they saw as discrediting their profession” (page 258). The major technical work, which ac- knowledged that Kellogg’s criticisms were accurate but did little to change the significance of the results, was conducted by Charles Stuart and Joseph AnPpPOvER OP REESE 20 0108/08 : _ volume of the Journal of Parapsychology and Greenwood, 1937). Stuart, who had been an undergraduate in mathematics at Duke, was one of Rhine’s early subjects and continued to work with him as a researcher until Stuart’s death in 1947. Greenwood was a Duke mathematician, who appar- ently converted to a statistician at the urging of Rhine. Another prominent figure who was distressed with Kellogg’s attack was E. V. Huntington, a mathematician at Harvard. After corresponding with Rhine, Huntington decided that, rather than further confuse the public with a technical reply to Kellogg’s arguments, a simple statement should be made to the effect that the mathematical issues in Rhine’s work had been resolved. Huntington must have successfully convinced his former student, Burton Camp of Wesleyan, that this was a wise approach. Camp was the 1937 President of IMS. When the annual meetings were held in December of 1937 (jointly with AMS and AAAS), Camp released a statement to the press that read: Dr. Rhine’s investigations have two aspects: experimental and statistical. On the exper- imental side mathematicians, of course, have nothing to say. On the statistical side, however, recent mathematical work has established the fact that, assuming that the experiments have been properly performed, the statistical analysis is essentially valid. If the Rhine investigation is to be fairly attacked, it must be on other than mathematical grounds [Camp, 1937]. One statistician who did emerge as a critic was William Feller. In a talk at the Duke Mathemati- cal Seminar on April 24, 1940, Feller raised three criticisms to Rhine’s work (Feller, 1940). They had been raised before by others (and continue to be raised even today). The first was that inadequate shuffling of the cards resulted in additional infor- mation from one series to the next. The second was what is now known as the “file-drawer effect,” namely, that if one combines the results of pub- lished studies only, there is sure to be a bias in favor of successful studies. The third was that the results were enhanced by the use of optional stop- ping, that is, by not specifying the number of trials in advance. All three of these criticisms were ad- dressed in a rejoinder by Greenwood and Stuart (1940), but Feller was never convinced. Even in its third edition published in 1968, his book An Intro- duction to Probability Theory and Its Applications still contains his conclusion about Greenwood and Stuart: “Both their arithmetic and their experi- ments have a distinct tinge of the supernatural” CURBS DAHA A OU TBEN ST Lote
Approved For Release 2000/08/08 : CIA-RDP96- 00789R003100010001 -6 366 J. UTTS Feller was confused... he seemed to have decided the opposition was wrong and that was that.” Several statisticians have contributed to the literature in parapsychology to greater or lesser degrees. T. N. E. Greville developed applicable statistical methods for many of the experiments in parapsychology and was Statistical Editor of the Journal of Parapsychology (with J. A. Greenwood) from its start in 1937 through Volume 31 in 1967; Fisher (1924, 1929) addressed some specific prob- lems in card-guessing experiments; Wilks (1965a, b) described various statistical methods for parapsy- chology; Lindley (1957) presented a Bayesian anal- ysis of some parapsychology data; and Diaconis (1978) pointed out some problems with certain ex- periments and presented a method for analyzing experiments when feedback is given. Occasionally, attacks on parapsychology have taken the form of attacks on statistical inference in general, at least as it is applied to real data. Spencer-Brown (1957) attempted to show that true randomness is impossible, at least in finite se- quences, and that this could be the explanation for the results in parapsychology. That argument re- emerged in a recent debate on the role of random- ness in parapsychology, initiated by psychologist J. Barnard Gilmore (Gilmore, 1989, 1990; Utts, 1989; Palmer, 1989, 1990). Gilmore stated that “The ag- nostic statistician, advising on research in psi, should take account of the possible inappropriate- ness of classical inferential statistics” (1989, page 338). In his second paper, Gilmore reviewed several non-psi studies showing purportedly random sys- tems that do not behave as they should under randomness (e.g., Iversen, Longcor, Mosteller, Gilbert and Youtz, 1971; Spencer-Brown, 1957). Gilmore concluded that “Anomalous data. should not be found nearly so often if classical statistics offers a valid model of reality” (1990, page 54), thus rejecting the use of classical statisti- cal inference for real-world applications in general. 3. REPLICATION Implicit and explicit in the literature on parapsy- chology is the assumption that, in order to truly establish itself, the field needs to find a repeat- able experiment. For example, Diaconis (1978) started the summary of his article in Science with the words “In search of repeatable ESP experi- ments, modern investigators...” (page 131). On October 28-29, 1983, the 32nd International Con- ference of the Parapsychology Foundation was held in San Antonio, Texas, to address “The Repeatabil- ity Problem in Parapsychology.” The Conference Proceedings (Shapin and Coly, 1985) reflect the diverse views among parapsychologists on the na- ture of the problem. Honorton (1985a) and Rao (1985), for example, both argued that strict replica- tion is uncommon in mast branches of science and that parapsychology should not be singled out as unique in this regard. Other authors expressed disappointment in the lack of a single repeatable experiment in parapsychology, with titles such s “Unrepeatability: Parapsychology’s Only Find- ing” (Blackmore, 1985), and “Research Strategies for Dealing with Unstable Phenomena” (Beloff, 1985). It has never been clear, however, just exactly what would constitute acceptable evidence of a re- peatable experiment. In the early days of investiga- tion, the major critics ‘insisted that it would be sufficient for Rhine and Soal to convince them of ESP if a parapsychologist could perform success- fully a single ‘fraud-praof’ experiment” (Hyman, 1985a, page 71). However, as soon as well-designed experiments showing | statistical significance emerged, the critics realized that a single experi- ment could be statistically significant just by chance. British psychologist C. E. M. Hansel quan- tified the new expectation, that the experiment should be repeated a few:times, as follows: If a result is significant at the .01 level and this result is not due to chance but to informa- tion reaching the subject, it may be expected that by making two further sets of trials the antichance odds of one: hundred to one will be increased to around a million to one, thus en- abling the effects of ESP—or whatever is re- sponsible for the original result—to manifest itself to such an extent that there will be little doubt that the result! is not due to chance [Hansel, 1980, page 298]. In other words, three consecutive experiments at p = 0.01 would convince Hansel that something other than chance was at work. This argument implies that if a particular experi- ment produces a statistically significant result, but subsequent replications fail to attain significance, then the original result was probably due to chance, or at least remains unconvincing. The problem with this line of reasoning is; that there is no consid- eration given to sample size or power. Only an experiment with extremely high power should be expected to be “successful” three times in succession. : It is perhaps a failure of the way statistics is taught that many scientists do not understand the importance of power in defining successful replica- tion. To illustrate this point, psychologists Tversky and Kahnemann (1982) distributed a questionnaire Approved For Release 2000/08/08 : CIA-RDP96- 00789R00310001 001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION IN PARAPSYCHOLOGY 367 to their colleagues at a professional meeting, with the question: An investigator has reported a result that you consider implausible. He ran 15 subjects, and reported a significant value, ¢ = 2.46. Another investigator has attempted to duplicate his pro- cedure, and he obtained a nonsignificant value of ¢ with the same number of subjects. The direction was the same in both sets of data. You are reviewing the literature. What is the highest value of ¢ in the second set of data that you would describe as a failure to replicate? {1982, page 28]. In reporting their results, Tversky and Kahne- mann stated: The majority of our respondents regarded ¢ = 1.70 as a failure to replicate. If the data of two such studies (¢ = 2.46 and ¢ = 1.70) are pooled, the value of ¢ for the combined data is about 3.00 (assuming equal variances). Thus, we are faced with a paradoxical state of affairs, in which the same data that would increase our confidence in the finding when viewed as part of the original study, shake our confidence when viewed as an independent study [1982, page 28]. At a recent presentation to the History and Phi- losophy of Science Seminar at the University of California at Davis, I asked the following question. Two scientists, Professors A and B, each have a theory they would like to demonstrate. Each plans to run a fixed number of Bernoulli trials and then test Hy: p = 0.25 versus H,: p > 0.25. Professor A has access to large numbers of students each semester to use as subjects. In his first. experiment, he runs 100 subjects, and there are 33 successes (p = 0.04, one-tailed). Knowing the importance of replication, Professor A runs an additional 100 sub- jects as a second experiment. He finds 36 successes (p = 0.009, one-tailed). Professor B only teaches small classes. Each quarter, she runs an experiment on her students to test her. theory. She carries out ten studies this way, with the results in Table 1. I asked the audience by a show of hands to indicate whether or not they felt the scientists had successfully demonstrated their theories. Professor A’s theory received overwhelming support, with approximately 20 votes, while Professor B’s theory received only one vote. If you aggregate the results of the experiments for each professor, you will notice that each con- ducted 200 trials, and Professor B actually demon- strated a higher level_of success than Professor A Approved For Release 2000/08/08 with 71 as opposed to 69 successful trials. The one-tailed p-values for the combined trials are 0.0017 for Professor A and 0.0006 for Professor B. To address the question of replication more ex- plicitly, I also posed the following scenario. In December of 1987, it was decided to prematurely terminate a study on the effects of aspirin in reduc- ing heart attacks because the data were so convinc- ing (see, e.g., Greenhouse and Greenhouse, 1988; Rosenthal, 1990a). The physician-subjects had been randomly assigned to take aspirin or a placebo. There were 104 heart attacks among the 11,037 subjects in the aspirin group, and 189 heart attacks among the 11,034 subjects in the placebo group (chi-square = 25.01, p < 0.00001). After showing the results of that study, I pre- sented the audience with two hypothetical experi- ments conducted to try to replicate the original result, with outcomes in Table 2. I asked the audience to indicate which one they thought was a more successful replication. The au- dience chose the second one, as would most journal editors, because of the “significant p-value.” In fact, the first replication has almost exactly the same proportion of heart attacks in the two groups as the original study and is thus a very close repli- cation of that result. The second replication has TABLE 1 Aitempted repiciations for professor B n Number of successes One-tailed p-value 10 4 0.22 15 6 0.15 17 6 0.23 25 8 0.17 30 10 0.20 40 13 0.18 18 7 0.14 10 5 0.08 15 5 0.31 20 7 0.21 TABLE 2 Hypothetical replications of the aspirin / heart attack study Replication #1 Heart attack Replication #2 Heart attack Yes No ~ Yes No Aspirin 11 1156 20 2314 Placebo 19 1090 48 2170 Chi-square 2.596, p= 0.11 13.206, p = 0.00038 : CIA-RDP96-00789R003100010001-6
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 368 J. UTTS very different proportions, and in fact the relative risk from the second study is not even contained in a 95% confidence interval for relative risk from the original study. The magnitude of the effect has been much more closely matched by the “nonsig- nificant” replication. Fortunately, psychologists are beginning to no- tice that replication is not as straightforward as they were originally led to believe. A special issue of the Journal of Social Behavior and Personality was entirely devoted to the question of replication (Neuliep, 1990). In one of the articles, Rosenthal cautioned his colleagues: “Given the levels of sta- tistical power at which we normally operate, we have no right to expect the proportion of significant results that we typically do expect, even if in na- ture there is a very real and very important effect” (Rosenthal, 1990b, page 16). Jacob Cohen, in his insightful article titled “Things I Have Learned (So Far),” identified an- other misconception common among social scien- tists: “Despite widespread misconceptions to the contrary, the rejection of a given null hypothesis gives us no basis for estimating the probability that a replication of the research will again result in rejecting that null hypothesis” (Cohen, 1990, page 1307). Cohen and Rosenthal both advocate the use of effect. sizes as opposed to significance levels when defining the strength of an experimental effect. In general, effect sizes measure the amount by which the data deviate from the null hypothesis in terms of standardized units. For instance, the effect size for a two-sample ¢-test is usually defined to be the difference in the two means, divided by the stan- dard deviation for the control group. This measure can be compared across studies without the depen- dence on sample size inherent in significance lev- els. (Of course there will still be variability in the sample effect sizes, decreasing as a function of sam- ple size.) Comparison of effect sizes across studies is one of the major components of meta-analysis. Similar arguments have recently been made in the medical literature. For example, Gardner and Altman (1986) stated that the use of p-values “to define two alternative outcomes—significant and not significant—is not helpful and encourages lazy thinking” (page 746). They advocated the use of confidence intervals instead. As discussed in the next section, the arguments used to conclude that parapsychology has failed to demonstrate a replicable effect hinge on these mis- conceptions of replication and failure to examine power. A more appropriate analysis would compare the effect sizes for similar experiments across ex- perimenters and across time to see if there have been consistent effects of the same magnitude. Rosenthal also advocates this view of replication: The traditional view! of replication focuses on significance level as the relevant summary statistic of a study and evaluates the success of a replication in a dichotomous fashion. The newer, more useful view of replication focuses on effect size as the more important summary statistic of a study and evaluates the success of a replication not in ‘a dichotomous but in a continuous fashion [Rosenthal, 1990b, page 28]. The dichotomous view of replication has been used throughout the history of parapsychology, by both parapsychologists and critics (Utts, 1988). For example, the National Academy of Sciences report critically evaluated “significant” experiments, but entirely ignored “nonsignificant” experiments. In the next three sections, we will examine some of the results in parapsychology using the broader, more appropriate definition of replication. In doing so, we will show that the results are far more interesting than the critics would have us believe. 4. THE GANZFELD DEBATE IN PARAPSYCHOLOGY An extensive debate took place in the mid-1980s between a parapsychologist and critic, questioning whether or not a particular body of parapsychologi- cal data had demonstrated psi abilities. The experi- ments in question were all conducted using the ganzfeld setting (described below). Several authors were invited to write commentaries on the debate. As a result, this data base has-been more thor- oughly analyzed by both critics and proponents than any other and provides a good source for studying replication in parapsychology. The debate concluded with a detailed series of recommendations for further experiments, and left open the question of whether or not psi abilities had been demonstrated. A new series of experi- ments that followed the recommendations were conducted over the next, few years. The results of the new experiments will be presented in Section 5. 4.1 Free-Response Experiments Recent experiments in parapsychology tend to use more complex target material than the cards and dice used in the early investigations, partially to alleviate boredom on the part of the subjects and partially because they are thought to “more nearly resemble the conditions of spontaneous psi occur- rences” (Burdick and Kelly, 1977, page 109). These experiments fall under the general heading of “free-response” experiments, because the subject is asked to give a verbal or written descri bce of the Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001
Approved For Release 2000/08/08 : CIA-RDP96-00789R003100010001-6 REPLICATION IN PARAPSYCHOLOGY 369 target, rather than being forced to make a choice from a small discrete set of possibilities. Various types of target material have been used, including pictures, short segments of movies on video tapes, actual locations and small objects. Despite the more complex target material, the statistical methods used to analyze these experi- ments are similar to those for forced-choice experi- ments. A typical experiment proceeds as follows. Before conducting any trials, a large pool of poten- tial targets is assembled, usually in packets of four. Similarity of targets within a packet is kept to a minimum, for reasons made clear below. At the start of an experimental session, after the subject is sequestered in an isolated room, a target is selected at random from the pool. A sender is placed in another room with the target. The subject is asked to provide a verbal or written description of what he or she thinks is in the target, knowing only that it is a photograph, an object, etc. After the subject’s description has been recorded and secured against the potential for later alter- ation, a judge (who may or may not be the subject) is given a copy of the subject’s description and the four possible targets that were in the packet with the correct target. A properly conducted experi- ment either uses video tapes or has two identical sets of target material and uses the duplicate set for this part of the process, to ensure that clues such as fingerprints don’t give away the answer. Based on the subject’s description, and of course on a blind basis, the judge is asked to either rank the four choices from most to least likely to have been the target, or to select the one from the four that seems to best match the subject’s description. If ranks are used, the statistical analysis proceeds by summing the ranks over a series of trials and comparing the sum to what would be expected by chance. If the selection method is used, a “direct hit” occurs if the correct target is chosen, and the number of direct hits over a series of trials is compared to the number expected in a binomial experiment with p = 0.25. Note that the subjects’ responses cannot be con- sidered to be “random” in any sense, so probability assessments are based on the random selection of the target and decoys. In a correctly designed ex- periment, the probability of a direct hit by chance is 0.25 on each trial, regardless of the response, and the trials are independent. These and other issues related to analyzing free-response experiments are discussed by Utts (1991). 4.2 The Psi Ganzfeld Experiments The ganzfeld procedure is a particular kind of isolation technique originally developed by Gestalt psychologists for other purposes. Evidence from spontaneous case studies and experimental work had led parapsychologists to a model proposing that psychic functioning may be masked by sensory in- put and by inattention to internal states (Honorton, 1977). The ganzfeld procedure was specifically de- signed to test whether or not reduction of external ‘‘noise” would enhance psi performance. In these experiments, the subject is placed in a comfortable reclining chair in an acoustically shielded room. To create a mild form of sensory deprivation, the subject wears headphones through which white noise is played, and stares into a constant field of red light. This is achieved by taping halved translucent ping-pong balls over the eyes and then illuminating the room with red light. In the psi ganzfeld experiments, the subject speaks into a microphone and attempts to describe the target material being observed by the sender in a distant room. At the 1982 Annual Meeting of the Parapsycho- logical Association, a debate took place over the degree to which the results of the psi ganzfeld experiments constituted evidence of psi abilities. Psychologist and critic Ray Hyman and parapsy- chologist Charles Honorton each analyzed the re- sults of all known psi ganzfeld experiments to date, and they reached strikingly. different conclusions (Honorton, 1985b; Hyman, 1985b). The debate con- tinued with the publication of their arguments in separate articles in the March 1985 issue of the Journal of Parapsychology. Finally, in the Decem- ber 1986 issue of the Journal of Parapsychology, Hyman and Honorton (1986) wrote a joint article in which they highlighted their agreements and disagreements and outlined detailed criteria for future experiments. That same issue contained commentaries on the debate by 10 other authors. The data base analyzed by Hyman and Honorton (1986) consisted of results taken from 34 reports written by a total of 47 authors. Honorton counted 42 separate experiments described in the reports, of which 28 reported enough information to determine the number of direct hits achieved. Twenty three of the studies (55%) were classified by Honorton as having achieved statistical significance at 0.05. 4.3 The Vote-Counting Debate Vote-counting is the term commonly used for the technique of drawing inferences about an experi- mental effect by counting the number of significant versus nonsignificant studies of the effect. Hedges and Olkin (1985) give a detailed analysis of the inadequacy of this method, showing that it is more free-respiypprovedmRor Releawe 2000/0BIGE : CARERS HERO TAdETDOUTS™ 7M