Approved For Release 40d bA SSIRDEL-00789R00 95300390001-5 them to draw conclusions that are completely the opposite of the results obtained when the same studies are summarized by more explicit, quantitative methods. Given the difficulties in assessing evidence from existing psi studies, is the replication trail likely to be heading -- to reinvoke our metaphor -- towards a flowering meadow or something decidedly less pleasant? In general, we believe that the prospects are aromatic. In the last few years, quantitative techniques of combining and comparing research results in systematic ways have been developed —- called meta-analysis (Rosenthal, 1984) -— that show great promise in demonstrating that some areas of socia] science have been progressing much better than previously thought. In parapsychology, initial meta-analyses applied to ganzfeld research (Honorton, 1985), hypnotic induction (Schechter, 1984), RNG studies (May, Hubbard and Humphrey, 1980; Nelson, Dunne _and Jahn, 1984; Tart, 1983), and remote viewing (Dunne, Jahn, and Nelson, 1983) have shown that the overall evidence for these psi phenomena is actually quite strong. Because meta-analysis involves the aggregation of results of numerous studies, several criticisms of this technique have been raised (Rosenthal, 1984, p.124-132). Perhaps the three categories of criticism most pertinant to review of parapsychological data are the following: First, authors may tend to report only the studies with significant results and leave the nonsignificant studies unpublished (called the filedrawer problem). Second, the meta-analysis combines poorer quality studies with better studies. And third, meta-analysis may be comparing “apples and oranges” by combining different experiments studying different variables. The first two problems may inflate the estimate of an overall effect; the third criticism may make the overall summary difficult or impossible to interpret. In the present meta-analysis, however, we actually are interested in whether these psi experiments have borne fruit, not whether they have borne specific flavors of apples or oranges. In other words, we are not concerned with whether hypnotic induction, say, has an effect on RNG outputs, but whether there is evidence for any psi effect on RNG outputs. Thus, in this investigation we have concentrated on the filedrawer issue (in this report) and the quality of studies (to be described in Part 2 of this study). OVERVIEW OF A TYPICAL RNG EXPERIMENT The typical psi experiment with RNGs involves three main components: An observer (e.g. a human, goldfish, cat or dog), a truly random number generator based on radioactive decay or electronic noise, and an experimental task linking the observer with the device, such as a video game, a set of instructions, a need to keep a heat lamp on or avoid a shock, and so on. The aim of these experiments is to show that the instructions (when humans are involved) or the induced need (when animals or plants are involved) are associated in some way —- but not necessarily causally —- to the statistical output of the RNG. For example, say an RNG was designed to produce 100 random bits at the press of a button. An individual in this experiment might see a digital display of the number of 1’s (called hits) produced immediately after he or she pressed a button. The instructions in the experiment would typically be to get as many hits as possible for each button press. The results of many presses, or trials, would UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200390001-5
Approved For Release sod WheloAA Sd RUEES.00789R002200390001-5 then be evaluated statistically, where under the null hypothesis an average of 50 hits would be expected by chance. If the average number of hits over thousands of repetitions were say, 52, this deviation from chance would be interpreted as evidence of a psi effect (provided that the probability of observing this deviation was Jess than 1 in 20). PROCEDURE Because we were ultimately interested in testing among several different models of mechanisms possibly operating in these RNG experiments, in Part 1 of this meta-analysis (this paper) we surveyed the parapsychological literature with two goals in mind: First, we wanted to see whether the aggregated result of the RNG experiments showed evidence for an anomalous effect. And second, we needed the details of these experiments for use in evaluating a model of the underlying mechanism. [Our modeling effort is discussed in May, Radin, Hubbard, Humphrey and Utts (1985).] Source of references We searched through the five major English language parapsychological journals’ over the years 1969 to 1984. We also included the (refereed) Proceedings of Presented Papers for the Annual Parapsychological Association Conventions (1971 and 1984), and a report published by the Princeton Engineering Anomalies Research Laboratory (Nelson, Dunne and Jahn, 1984). The literature search was started in the year 1969 because that was the year Helmut Schmidt (1969) published the seminal RNG study that has since spawned many replications. Defining “an experiment” One of the difficulties faced in reviewing the articles for this meta-analysis was to decide what constituted an experiment. In most papers, authors analyze their data repeatedly in various ways, sometimes as a priori analyses, sometimes as post hoc afterthoughts. Even in cases of planned analyses, there are many ways of interpreting which of several conditions is the “real” experiment. How we decide what is an experiment is important to the meta-analysis for two main reasons: First, the meta-analytic statistical power depends on the number of experiments we find; and second, the z scores are different depending on how we break down the reported results. To illustrate the difficulty of deciding what an experiment is, consider this example. Say an author uses three different groups of 10 percipients each (e.g. meditators, truck drivers and athletes) and subjects each group to two different conditions (e.g. mental imagery vs. muscular tension) in a study on psi-conducive states. The results can be broken into one big, combined experiment, six experiments (3 groups x 2 conditions), two experiments (2 conditions), three 1. These are the Journal of Parapsychology, European Journal of Parapsychology, Research in Parapsychology, Journal of the Society for Psychical Research, Journal of the American Society for Psychical Research. UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200390001-5
Approved For Release 40¢N@bs SHIRIPED-00789R002200390001-5 experiments (3 groups), 30 experiments (subject by subject analysis), and so on. How do we decide what to use? We resolved this issue for this first-pass analysis in the following way: For cases where there were multiple hypotheses under test and multiple analyses of the data, we chose as the experimental unit the largest possible accumulation of data compatible with a single “direction of effort” assigned to the subjects. A clearly defined direction of effort meant that the experimental protocol required either more 1’s or more 0’s from the RNG to successfully complete the assigned task, regardless of whether or not the subjects actually knew their task in detail. Say, for example, a hypothesis predicted that group A would score higher than group B, and it was stated that “higher” meant more 1 bits. Then we would take this study as two experiments: Group A's and group B’s scores. In this particular case, since group A was predicted to score higher than group B, if in fact the difference between z(A) and z(B) were significant, then both z scores would be taken as positive, regardless of the reported z’s. Thus if z(A) = 1.5, z(B) = -1.0, then the z-score difference between them would be significant one-tailed with zaitt= 1.77. If the number of trials run in each case were 10000, then the number of hits assigned per experiment would be hits(A)= 5075 and hits(B) = 5050, which are both positive deviations; similarly the z scores would be recorded as z(A) = 1.5, z(B) = 1.0. If z(A) = 1.2 and z(B) = -1.0, the z scores would be recorded as originally reported since zaut is not significant. The same would be true if z(A) = -2.0 and z(B) = 2.0. (Fortunately, such problems of interpretation were not often encountered in the survey.) As another example, if groups A, B, and C all tried to influence an RNG in a particular way, and no predictions were made as to interactions, then their overall result would be combined as one experiment. In this way, we attempted to emphasize in the meta-analysis the underlying question of whether or not observers could influence or otherwise affect the statistical output of an RNG according to the stated intention of the experimenter. Results of literature review We found 73 pertinent references in the journals and reports.2 These references included 381 experiments contributed by 38 different principal investigators, representing about 10 different laboratories around the world. We say “about 10 laboratories” because over the years labs have come and gone, researchers have moved among different labs, and in many cases, one or two individuals at an academic or private research institution are considered a “laboratory.” Breakdown of experiments Of the 381 experiments found, 332 (in 56 remaining references) were described as using binary generators based on either radioactive decay or electronic noise. For this meta-analysis, we considered only studies using binary RNGs (or any study in which the hit rate was defined or could be interpreted as 50%) for three reasons: First, since 87% of the experiments (332/381) employed binary generators, we felt that this sample was representative of the entire RNG database; second, for the sake of simplicity; and, third because the test of a model 2. These references are listed under the heading “Meta-Analysis” in the references at the end of this paper. B-5 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200390001-5