Approved For Release 2000/08/08 : CIA-RDP96-00789R003100120001-4 CPYRGHT Journal of Communication, Winter 1975 108 able to the ESP hypothesis made 71.5 percent more errors of commission (increasing ESP scores), while those who were unfavorable to the ESP hypothesis made 100 percent more errors of omission (decreasing ESP scores), Murphy ($7) reported an analysis of 175,000 trials from experiments reporting positive evidence for ESP and found ‘only 175 errors (0.10 per- cent). Greenwood (12) reported only 90 recording errors in rechecking his 500,000-trial control study, of which 76 were errors of omission. Some critics also alleged that improper selection of data could account for experimental successes. This could be done in several ways: (a) selection of subjects; (b) selection of particular blocks of data out of larger samples; (c) selection of one of several forms of analysis; and (d) selective reporting of particular studies. The_questions raised haye sometimes been stated cynically in the form, “Parapsychologists must: run 100 subjects before they find one with ‘ESP’. As if in defense against this charge, a number of the reported studies specifically stated that all of the data collected were included in the analysis (see 43, pp. 118-124, Table 12). Concerning selection of subjects, Warner (76) suggested two criteria: first, results of “poor” subjects must be included up to the point when they are discontinued since it does not matter how many trials a given subject makes as long as all of the trials (for all subjects) are included; second, exclude all preliminary trials (for both “good” and “poor” sub- jects) and use preliminary screening studies to select “good” candidates for formal work. These criteria were generally endorsed by the chief critics of the period (e.g., 23). | The question of post hoc selection of analyses was not a point of serious concern in the period between 1934 and 1940, though it is relevant to the assessment of some of the process-oriented investigations reported more recently. The question of whether nonsignificant studies were withheld from publication involves an issue which is of great concern to the be- havioral sciences as a whole. (70, 81) and one which is difficult to accurately assess since there is no way of knowing how many studies may have been withheld from publication because their results failed to disconfirm the null hypothesis. Several studies of American Psychological Assaciation publication poli- cies (4, 70, 81) indicate that experimental studies in general are more likely to be published if the null hypothesis is rejected! at the conventional .05 and .01 alpha levels than if it is not rejected. These studies also indicate that a negligible proportion of published studies are replications. Bozarth and Roberts (4), in a survey of 1,334 articles from psychological journals, found that 94 percent of the articles involving statistical tests of significance reported rejection of specific null hypotheses; only eight articles (less than 1 percent) involved replications of previously published studies. With respect to the implications of such selection for the ESP hypothesis, there are two partial answers, First, considering the degree of critical inter- est which prevailed in the 1930s, it seems unlikely that nonsignificant find- ings would have been repressed during this period} second, the high levels Approved For Release 2000/08/08 : CIA-RDP96-00789R003100120001-4
ApprevecRitsyi Release 2000/08/08 : CIA-RDP96-00789R003100120001-4 Paranoryial Communication / “Error Se Place!” . of significance attached to some of the reported ESP investigations would necessitate postulating astronomical numbers of “chance’ trials in order to dilute the overall deviations to chance. To take one example, consider the Pearce-Pratt series of 1,850 trials which yielded p = 10-22, As Soal and Bateman (66) pointed out, it is difficult to believe that 10% (ten thousand million) sets of 1,850 trials could have possibly been carried out between 1934 and 1940 (or, for that matter, since 1940). But, as Soal and Bateman Suggest, “’... if we posit this absurd estimate as an upper limit [with overall chance totals], that would still give us odds of 101°... against the supposi- tion that the Pearce-Pratt results were a run of pure luck.” The possibility of obtaining significant “extrachance” results by stopping an experimental series at “favorable” points was also raised (9, 31). While this “optional stopping” hypothesis was generally agreed to be of significance only in cases of marginally significant’ results, it led to the adoption of several procedural modifications: specification of the total number of trials in advance of data collection, or accumulation of data in blocks of pre- determined size. The possibility was raised by several critics that hand-shuffled cards may display a tendency to stick together or otherwise produce patterns which could produce spurious results (24, 82). While the cross-check type of con- trol series, described earlier, failed to reveal any evidence of patterning, there was a general trend away from hand shuffling in the later published studies, which utilized tables of prepared random numbers as a basis for generating target sequences. There was—and is (e.g., 15)—a rather widespread belief that most of the evidence supporting the ESP hypothesis originated in the Duke Uni. versity studies and that most independent replications by other investigators were nonconfirmatory. A survey of the published literature between 1934 and 1940 fails to support this claim. Table 2 shows all the published experi- mental reports during this period which provided statistical treatment of the data. Inspection of this table reveals that a Majority (61 percent) of the outside replications report significant results (P < .01) and that the proportion of significant studies was not significantly greater for the Duke University group (x? = 1.70, 1 df). By 1940, the active methodological controversy was over. The issues raised were, for the most part, legitimate, and investigators modified their procedures to safeguard their results from methodological criticism, The major issues raised since 1940 center on alleged anomalies in probability theory and the hypothesis of widespread investigator fraud. Spencer Brown (68, 69) has suggested that statistically significant card- guessing studies provide evidence, not of extrasensory modes of communica- tion, but of fundamental defects in probability theory. He makes three criticisms of random number sequences: (a) published random number Approved For Release 2000/08/08 : CIA-RDP96-00789R003100120001-4
he OMe 8 ; Approved For Release 2000/08/08 : CIA-RDP96-00789R003100120001-4 CPYRGHT Journal of Communication, Winter 1975 Table 2: Breakdown of experimental ESP studies (1934-1939) N N studies reported (Studiesy significant (p < ;01) % signif. Duke group =>; «17 16 88 Non-Duke™ 33 20 61 Totat 50 35 70 * Includes all English-language studies involving assessment of statistical significance of data, 1934-1939 inclusive. : x! (Duke vs. non-Duke x significant.vs. nonsignificant) = 1.70 (1 df sequences have been “doctored” prior to publication in order to remove certain nonrandom features; this practice, according to Spencer Brown, makes such sequences nonrandom and invalidates the use of standard significance tests; (b) the source of some random number sequences involves randomizing machines which utilize the unpredictability of human be- havior when examined for. microscopic variation; such variation, says Spencer Brown, may be predictable enough to account for observed anom- alies in random sequences, as well as some of the significant results reported in ESP guessing experiments;- (c) Spencer Brown prdduces evidence to show that anomalous (significant) departures from probability theory can be obtained by matching columns of random numbers (39). A detailed examination of these points was undertaken by Scott (64). With respect to “doctored” sequences, Scott showed that the maximum error due to rejected (edited) sequences would not affect interpretations of results which are more than marginally significant and could, in fact, increase the likelihood of making a Type II error. On the hypothesis that ESP results are due to some kind of hyper-regulatity affecting both the target sequence and the response (guess) sequence simultaneously and similarly, Scott makes the point that this would lead to the expectation of similar results from matching any set of humanly! produced random se- quences. The cross-check type of control series and the Greenwood chance control series described earlier demonstrate that this is not the case. The anomalies reported by Spencer Brown (68), obtained by arbitrarily matching columns of random numbers, have been criticized on the basis of post hoc selection (40) and illustrate not that there are fundamental defects in prob. ability theory, but rather that significant deviations from chance can occur in any data where hypotheses and analyses are not specified in advance. The most recent phase of the ESP controversy centers.on the hypothesis of investigator fraud. This argument was most for efully presented in a lead article in Science, entitled “Science and the Supernatural,” by G. R. Price (47), who began with the following observations: | Believers in psychic phenomena . . . appear to have won a decisive victory and virtually silenced opposition. ... This victory is the result of an impressive amount of careful experimentation and intelligent argumentation. . . . Against all this evidence, almost the only defense Approved For Release 2000/08/08 : CIA-RDP96-00789R003100120001-4
A For Release 2000/08/08 : CIA-RDP96-00789R003100120001-4 PPreR GH | Paranormal Communication / “Error Some Place!” remaining to the skeptical scientist ts ignorance, ignorance concerning the work itself and concerning its implications. The typical scientist contents himself with retaining ... some criticism that at most applies toa small fraction of the published studies. But these findings (which challenge our very concepts of space and time) are—if valid—of enor: mous importance . .. so they ought not to be ignored. Following Hume's argument on miracles, Price asserted that ESP is “incompatible with current scientific theory,” and that it is therefore more parsimonious to believe that parapsychologists cheat than that ESP is a real phenomenon. He concluded, “My opinion concerning the findings of the parapsychologists is that many of them are dependent on clerical and Statistical errors and unintentional use of sensory clues, and that all extra- chance results not so explicable are dependent on deliberate fraud or mildly abnormal mental conditions” (47, p. 360). This extraordinary critique and the ensuing discussion in Science (5, 36, 48, 55, 56, 65) were widely reviewed. As Meehl and Scriven (46) pointed out, Price’s argument rests on two highly questionable assumptions, namely that contemporary scientific knowledge is complete, and that ESP necessarily conflicts with it. Seventeen years later, in an “Apology to Rhine and Soal,” Price retracted his accusations of investigator fraud (49). Very similar arguments have, however, been made more recently by the British parapsychological critic C. E. M. Hansel (14, 15), who began his examination of the ESP hypothesis by suggesting that “the a priori arguments . . . may even save time and effort in scrutinizing the [ESP] experiments. .. . In view of the a priori arguments against it we know in advance that telepathy, etc., cannot occur.” Because of the “a priort unlikelihood” of ESP, Hansel’s examination of the literature centered primarily on the possibility of fraud, by subjects or investigators. He reviewed in depth four experiments which he regarded as providing the best evidence of ESP: the already-mentioned Pearce-Pratt distance series (59); the Pratt-Woodruff (44) series, also conducted at Duke; and Soal’s work with Mrs. Stewart and Basil Shackleton (66), as well as a more recent series by Soal and Bowden (67). Hansel showed, in each case, how fraud could have been committed (by the experimenters in the Pratt-Woodruff and Soal-Bateman series, and by the subjects in the Pearce- Pratt and Soal-Bowden experiments). He gave no direct evidence that fraud was committed in these experiments but said, “If the result could have arisen through a trick, the experiment must be considered unsatisfactory proof of ESP, whether or not it is finally decided that such a trick was in fact used” (15, p. 18, italics mine). Hansel's argument is unclear, inasmuch as he quite properly insists that no single experiment can be conclusive, then proceeds to show that none is, given the theoretical possibility of fraud by subjects or investigators. Hansel’s only conclusion after more than 250 pages of careful scrutiny was that these experiments were not “fraud-proof” and therefore not con- Clusive proof of ESP. ql Approved For Release 2000/08/08 : CIA-RDP96-00789R003100120001-4