a For Release 2000/08/08 : CIA-RDP96-00789R002200450001-8 o underestimation of the effective error rate and overestimation of the actual significance level o erroneous use of particular statistical procedures. In their joint communique, Honorton argued that there is “no significant correlation between indices of study quality [flaws] and study outcome," Hyman agreed in most cases, but insisted that there is a positive correlation in the case of poor randomization, feedback, and inadequate documentation. ? (This assertion was rebutted in a later analysis, )10 Workshop participants generally agreed that in any event the "flawless," perfect experiment is very rare in any field, if it exists at all. The point of the issue, critics say, is that the presence of a flaw usually implies inadequate controls for error. The most stringent among them believe that an apparently successful experiment cannot be considered to have demonstrated a true anomaly (i.e., a possible psi effect) unless it can be shown that the experiment is completely flawless. The parapsychologists counter that any such flaws should be an issue only if they could have systematically influenced the result. Another point made by parapsychologists is that, by and large, the critics have done very little experimentation in parapsychology. Therefore, they have little awareness of the difficulties encountered or of what the truly significant flaws might be. Indeed, the parapsychologists note, there could be additional flaws that have not yet been considered and which critics could identify through actual experimentation. The NRC report distinguished three types of criticisms relating to flaws in experiment design and conduct.1! First is what is called the “smoking gun," in which observed findings are said to be definitely attributable to a specific flaw or other factor (such as deliberate fraud) that is shown to be present. Second, and more common, is the "plausible alternative" allegation that a particular flaw is present and could have produced the reported results. Third is the "dirty test tube" concept, which alleges not that a particular discovered flaw produced the positive result but simply that the presence of flaws in the experiment demonstrates a general sloppiness which brings the results into question. In other words, if results have been obtained under conditions that fail to meet "generally accepted standards" for scientific research (such as clean test tubes), that fact alone casts doubt on the accuracy and validity of the results. The authors of the NRC report claimed that, while they could find no instances of the "smoking gun" or “plausible alternative," even the best parapsychology experiments exemplify the “dirty test tube" problem (an allegation that has been heatedly disputed by parapsychologists). Participants agreed that the burden of proof in the first two categories should be on the critic making the allegation, as it is in other sciences where an experiment is challenged on these grounds. In the case of the "dirty test tube," the critics feel that the burden of proof should be on the experimenter. This position brings from the parapsychologists the objection that it is non- 9 Hyman & Honorton, op. cit., p. 353. 10 Harris, M.J. & Rosenthal, R. "Interpersonal Expectancy Effects and the Human Performance Research." Washington, DC: National Academy Press, 1988. (p. 3) 11 National Research Council, op. cit., pp. 199-200. Approved For Release 2000/08/08 : CIA-RDP96-00789R002200450001-8