Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Two: Evaluation Plan clearly be attributed to the operation of paranormal phenomena. Since then, the Principal Investigator, Dr. Edwin May, under formerly classified government contracts, has conducted a number of other studies not previously reviewed. These studies were expressly intended to address many of the criticisms raised in the initial NRC report. Because these studies might provide new evidence for the existence of the remote viewing phenomenon, its causal mechanisms, and its boundary conditions, a new review seemed called for. The Review Panel. With these issues in mind, a blue-ribbon review panel was commissioned, with the intent of ensuring a balanced and objective appraisal of the research. Two of the reviewers were scientists noted for their interest, expertise, and experience in parapsychological research. The first of these two expert reviewers, Dr. Jessica Utts, a Professor of Statistics at the University of California-Davis, is a nationally-recognized scholar who has made major contributions to the development and application of new statistical methods and techniques. Among many other positions and awards, Dr. Utts is an Associate Editor of the Journal of the American Statistical Association (Theory and Methods) and the Statistical Editor of the Journal of the American Society for Psychical Research, She has published several articles on the application of statistical methods to parapsychological research and has direct experience with the remote viewing research program. The second expert reviewer, Dr. Raymond Hyman, is a Professor of Psychology at the University of Oregon. Dr. Hyman has published over 200 articles in professional journals on perception, pattern recognition, creativity, problem solving, and critiques of the paranormal, He served on the original NRC Committee on Techniques for the Enhancement of Human Performance. Dr. Hyman serves as a resource to the media on topics related to the paranormal, and has testified as an expert witness in court cases involving paranormal claims. He is recognized as one of the most important and fair-minded skeptics working in this area, Curriculum Vitae for Dr. Utts and Dr. Hyman are included in Appendix A. In addition to these two experts, four other scientists were involved in the work of the review panel. Two senior behavioral scientists and experts in research methods at the American Institutes for Research, Dr. Michael Mumford and Dr. Andrew Rose, served both as members of and staff to the panel. Dr. Mumford holds a Ph.D. in Industrial/Organizational Psychology from the University of Georgia. He is a Fellow of the American Psychological Association's Division 5, Measurement, Evaluation, and Statistics. Dr. Rose is a cognitive DRAFT - American Institutes for Research 2-2 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Two: Evaluation Plan psychologist with a Ph.D. from the University of Michigan. He has over 22 years of experience in designing and conducting basic and applied behavic.al science research. Dr. Rose is Chief Scientist of the Washington Office of AIR. They were to bring to the panel a methodological perspective unbiased by prior work in the area of parapsychology. The third participant was Dr. Lincoln Moses, an Emeritus Professor of Statistics at Stanford University, who participated in the review as a resource with regard to various statistical issues. Finally, Dr. David A. Goslin, President of AIR, participated as both a reviewer and coordinator for the review panel. Research Content. Prior to convening the first meeting of the review panel, the CIA transferred to AIR all reports and documents relevant to the review. We organized and copied these documents. In addition, the Principal Investigator for the program, Dr. Edwin May, was asked to provide two other pieces of information for the panel. First, he was asked to list those studies which he believes provide the strongest evidence bearing on the nature and significance of the remote viewing phenomenon. Second, he was asked to identify all unique studies conducted since the initial NRC report that provide evidence bearing on the nature and significance of the phenomenon. Additionally, he was asked to participate in an interview with members of the review panel following its first meeting to clarify any ambiguities about these studies. The complete list of documents, including notations of the "strongest evidence" set and the "unique" set, is included in Appendix B.' Review Procedures. Remote viewing, like virtually all other parapsychological phenomena, represents one of the most controversial research areas in the social sciences (e.g., Bem & Honorton, 1994; Hyman, 1994). Therefore, any adequate review of the research program must take this controversy into account in such a way that the review procedures are likely to result in a fair and unbiased assessment of the research. To ensure a fair and comprehensive review, Drs, Utts and Hyman agreed to examine all program documents, In the course of this review it was agreed that all members of the review panel would carefully consider: 1One document pertaining to the program remained classified during the period of this review. One of the review panel (Dr. Mumford) examined this document and provided an unclassified synopsis to the review panel. DRAFT - American Institutes for Research 2-3 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
i Na ee ata ae Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Two: Evaluation Plan . those studies recommended by the Principal Investigator as providing compelling evidence for the phenomenon, and * those empirical studies conducted since the NRC review that might provide new evidence about the existence and nature of the phenomenon. The members of the review panel convened at the Palo Alto office of AIR to structure exactly how the review process would be carried out. To ensure that different perspectives on paranormal phenomena would be adequately represented, Drs. Utts and Hyman were asked to prepare independent reports based on their review. In this review, they were to cover four general topics: . Was there a statistically significant effect? ° Could the observed effect, if any, be attributed to a paranormal phenomenon? . What mechanisms, if any, might plausibly be used to account for any significant effects and what boundary conditions influence these effects? . What would the findings obtained in these studies indicate about the characteristics and potential applications of information obtained through the remote viewing process? After they had each completed their reports, they presented the reports to other members of the panel. After studying these reports, all members of the review panel (except Dr. Moses) participated in a series of conference calls, The primary purpose of these exchanges was to identify the conclusions on which the experts agreed and disagreed. Next, in areas where they disagreed, Drs. Utts and Hyman were asked to discuss the nature of the disagreements, determine why they disagreed, and if possible, attempt to resolve the disagreements. Both the initial reports and the dialogue associated with discussion of any disagreements were made a part of the written record. In fact, Dr. Hyman's opinions on areas of agreement and disagreement are included in his report; in addition to her initial report, Dr. Utts prepared a reply to Dr. Hyman's opinions of agreement and disagreement. This reply, in addition to their original reports, are included in Section 3 below. DRAFT - American Institutes for Research 2-4 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Two: Evaluation Plan If disagreements could not be resolved through this dialogue, then the other members of the review panel were to consider the remaining issues from a general methodological perspective. Subsequently, they were to provide an addendum to the dialogue indicating which of the two positions being presented seemed to be on firmer ground both substantively and methodologically. This addendum concludes Section 3 below. Intelligence Gathering: The Operational Program The Program. In addition to the research component, the program included two operational components. One of those components was "foreign assessment," or analysis of the paranormal research being conducted by other countries. This issue, however, is beyond the scope of the present review. The other component involved the use of remote viewing as a technique for gathering intelligence information. In the early 1970s, the CIA experimented with applications of remote viewing in intelligence gathering. Later in the decade, they abandoned the program. However, other government agencies, including the Department of Defense, used remote viewers to obtain intelligence information. The viewers were tasked with providing answers to questions posed by various intelligence agencies. These operations continued until the Spring of 1995, when the program was suspended. Although procedures varied somewhat during the history of the program, viewers typically were presented with a request for information about a target of interest to a particular agency. Multiple viewings were then obtained for the target. The results of the viewings then were summarized in a three- or four-page report and sent to the agency that had posed the original question. Starting in 1994, members of the agencies receiving the viewing reports were formally asked to evaluate their accuracy and value. Any comprehensive evaluation of the remote viewing program must consider how viewings were used by the intelligence community. One might demonstrate the existence of a statistically significant paranormal phenomenon in experiments conducted in the laboratory; however, the phenomenon could prove to be of limited operational value either because it does not occur consistently outside the laboratory setting or because the kind of information DRAFT - American Institutes for Research 2-5 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Two: Evaluation Plan provided is of limited value to the intelligence community. General Evaluation Procedures. No one piece of evidence provides unequivocal support for the usefulness of a program. Instead, a more accurate and comprehensive picture can be obtained by considering multiple sources of evidence (Messick, 1989). Three basic sources of information were used in evaluation of the intelligence gathering component:. ° Prior research studies . Interviews with program participants, and . Analyses of user feedback. Prior Research Studies. As noted above, one aspect of the laboratory research program was to identify those conditions that set bounds on the accuracy and success of the remote viewing process. Thus, one way to analytically evaluate potential applications in intelligence gathering is to enumerate the conditions under which viewers were assigned tasks and then examine the characteristics of the remote viewing paradigm as studied through experimentation in the laboratory. The conditions under which operational tasks occur — that is, the requirements imposed by intelligence gathering — could then provide an assessment of the applicability of the remote viewing process. Interviews. As part of the Star Gate program, the services of remote viewers were uséd to support operational activities in the intelligence community. This operational history provides an additional basis for evaluating the Star Gate program; ultimately, if the program is to be of any real value, it must be capable of serving the needs of the intelligence community. By examining how the remote viewing services have been used, it becomes possible to draw some initial, tentative conclusions about the potential value of the Star Gate program. Below, we describe how information bearing on intelligence applications of the remote viewing phenomenon was gathered. Later, in Section 4, we describe the results of this information-gathering activity and draw some conclusions from the information we obtained. Although a variety of techniques might be used to accrue retrospective information (questionnaires, interviews, diaries, etc.), the project team decided that structured interviews DRAPFT - American Institutes for Research 2-6 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
een SS SRA Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 3. Research Reviews In this section, we present the conclusions drawn by the two experts after reviewing the research studies bearing on remote viewing. We begin by presenting the review of Dr. Jessica Utts. Subsequently, a rejoinder is provided by Dr. Raymond Hyman. Finally, Dr. Utts presents a reply to Dr. Hyman. The major points of agreement and disagreement are noted in the final section, along with our conclusions. In conducting their reviews, both Dr. Hyman and Dr. Utts focused on the remote viewing research. However, additional material is provided as indicated by the need to clarify certain points being made. Furthermore, both reviewers provided unusually comprehensive reviews considering not only classified program research, but also a number of earlier studies having direct bearing on the nature and significance of the phenomenon. An Assessment of the Evidence for Psychic Functioning Dr. Jessica Utts Division of Statistics, University of California, Davis September 1, 1995 ABSTRACT Research on psychic functioning, conducted over a two decade period, is examined to determine whether or not the phenomenon has been scientifically established. A secondary question is whether or not it is useful for government purposes. The primary work examined in this report was government sponsored research conducted at Stanford Research Institute, later known as SRI International, and at Science Applications International Corporation, known as SAIC. Using the standards applied to any other area of science, it is concluded that psychic functioning has been well established. The statistical results of the studies examined are far beyond what is expected by chance. Arguments that these results could be due to methodological flaws in the experiments are soundly refuted. Effects of similar magnitude to those found in government-sponsored research at SRI and SAIC have been replicated at a number of laboratories across the world. Such consistency cannot be readily explained by claims of flaws or fraud. merican institutes for Researc 3 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews a The magnitude of psychic functioning exhibited appears to be in the range between what social scientists call a small and medium effect. That means that it is reliable enough to be replicated in properly conducted experiments, with sufficient trials to achieve the long-run statistical results needed for replicability. A number of other patterns have been found, suggestive of how to conduct more productive experiments and applied psychic functioning. For instance, it doesn't appear that a sender is needed. Precognition, in which the answer is known to no one until a future time, appears to work quite well. Recent experiments suggest that if there is a psychic sense then it works much like our other five senses, by detecting change. Given that physicists are currently grappling with an understanding of time, it may be that a psychic sense exists that scans the future for major change, much as our eyes scan the environment for visual change or out ears allow us to respond to sudden changes in sound. It is recommended that future experiments focus on understanding how this phenomenon works, and on how to make it as useful as possible. There is little benefit to continuing experiments designed to offer proof, since there is little more to be offered to anyone who does not accept the current collection of data. 1. INTRODUCTION The purpose of this report is to examine a body of evidence collected over the past few decades in an attempt to determine whether or not psychic functioning is possible. Secondary questions include whether or not such functioning can be used productively for government purposes, and whether or not the research to date provides any explanation for how it works. There is no reason to treat this area differently from any other area of science that relies on statistical methods. Any discussion based on belief should be limited to questions that are not data-driven, such as whether or not there are any methodological problems that could substantially alter the results. It is too often the case that people on both sides of the question debate the existence of psychic functioning on the basis of their personal belief systems rather than on an examination of the scientific data. American Institutes for Research 3-2 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 __.. Chapter Three: Research Reviews nnn One objective of this report is to provide a brief overview of recent data as well as the scientific tools necessary for a careful reader to reach his or her own conclusions based on that data. The tools consist of a rudimentary overview of how statistical evidence is typically evaluated, and a listing of methodological concerns particular to experiments of this type. Government-sponsored research in psychic functioning dates back to the early 1970s when a program was initiated at what was then the Stanford Research Institute, now called SRI International, That program was in existence until 1989, The following year, government sponsorship moved to a program at Science Applications International Corporation (SAIC) under the direction of Dr. Edwin May, who had been employed in the SRI program since the mid 1970s and had been Project Director from 1986 until the close of the program. This report will focus most closely on the most recent work, done by SAIC. Section 2 describes the basic statistical and methodological issues required to understand this work; Section 3 discusses the program at SRI; Section 4 covers the SAIC work (with some of the details in an Appendix); Section 5 is concerned with external validation by exploring related results from other laboratories; Section 6 includes a discussion of the usefulness of this capability for government purposes and Section 7 provides conclusions and recommendations. 2. SCIENCE NOTES 2.1 DEFINITIONS AND RESEARCH PROCEDURES There are two basic types of functioning that are generally considered under the broad heading of psychic or paranormal abilities. These are classically known as extrasensory perception (ESP), in which one acquires information through unexplainable means and psychokinesis, in which one physically manipulates the environment through unknown means. The SAIC laboratory uses more neutral terminology for these abilities; they refer to ESP as anomalous cognition (AC) and to psychokinesis as anomalous perturbation (AP). The vast majority of work at both SRI and SAIC investigated anomalous cognition rather than anomalous perturbation, although there was some work done on the latter. American Institutes for Research 3-3 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews A Anomalous cognition is further divided into categories based on the apparent source of the information. If it appears to come from another person, the ability is called telepathy, if it appears to come in real time but not from another person it is called clairvoyance and if the information could have only been obtained by knowledge of the future, it is called precognition. It is possible to identify apparent precognition by asking someone to describe something for which the correct answer isn't known until later in time. It is more difficult to rule out precognition in experiments attempting to test telepathy or clairvoyance, since it is almost impossible to be sure that subjects in such experiments never see the correct answer at some point in the future. These distinctions are important in the quest to identify an explanation for anomalous cognition, but do not bear on the existence issue. The vast majority of anomalous cognition experiments at both SRI and SAIC used a technique known as remote viewing. In these experiments, a viewer attempts to draw or describe (or both) a target location, photograph, object or short video segment. All known channels for receiving the information are blocked. Sometimes the viewer is assisted by a monitor who asks the viewer questions; of course in such cases the monitor is blind to the answer as well. Sometimes a sender is looking at the target during the session, but sometimes there is no sender, In most cases the viewer eventually receives feedback in which he or she learns the correct answer, thus making it difficult to rule - out precognition as the explanation for positive results, whether or not there was a sender. Most anomalous cognition experiments at SRI and SAIC were of the free-response type, in which viewers were simply asked to describe the target. In contrast, a forced-choice experiment is one in which there are a small number of known choices from which the viewer must choose. The latter may be easier to evaluate statistically but they have been traditionally less successful than free-response experiments. Some of the work done at SAIC addresses potential explanations for why that might be the case. 2.2 STATISTICAL ISSUES AND DEFINITIONS Few human capabilities are perfectly replicable on demand. For example, even the best hitters in the major baseball leagues cannot hit on demand. Nor can we predict when merican Institutes for Researe. 3-4 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews A someone will hit or when they will score a home run. In fact, we cannot even predict whether or not a horse run will occur in a particular game. That does not mean that home runs don't exist. . Scientific evidence in the statistical realm is based on replication of the same average performance or relationship over the long run. We would not expect a fair coin to result in five heads and five tails over each set of ten tosses, but we can expect the proportion of heads and tails to settle down to about one half over a very long series of tosses. Similarly, a good baseball hitter will not hit the ball exactly the same proportion of times in each game but should be relatively consistent over the long run. The same should be true of psychic functioning. Even if there truly is an effect, it may never be replicable on demand in the short run even if we understand how it works. However, over the long run in well-controlled laboratory experiments we should see a consistent level of functioning, above that expected by chance. The anticipated level of functioning may vary based on the individual players and the conditions, just as it does in baseball, but given players of similar ability tested under similar conditions the results should be replicable over the long run. In this report we will show that replicability in that sense has been achieved. 2.2.1 P-VALUES AND COMPARISON WITH CHANCE. In any area of science, evidence based on statistics comes from comparing what actually happened to what should have happened by chance. For instance, without any special interventions about 51 percent of births in the United States result in boys. Suppose someone claimed to have a method that enabled one to increase the chances of having a baby of the desired sex. We could study their method by comparing how often births resulted in a boy when that was the intended outcome. If that percentage was higher than the chance percentage of 51 percent over the long run, then the claim would have been supported by statistical evidence. Statisticians have developed numerical methods for comparing results to what is expected by chance. Upon observing the results of an experiment, the p-value is the answer to the following question: If chance alone is responsible for the results, how likely would we be to observe results this strong or stronger? If the answer to that question, i.e. the p-value is very small, then most researchers are willing to rule out chance as an explanation. In fact it is commonly accepted practice to say that if the p-value is 5 percent (0.05) or less, then we can American Institutes for Research 3-5 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 ~ Chapter Three: Research Reviews RE definition is somewhat technical it is given in Appendix 1. An intuitive explanation will be given in the next subsection. Here, we note that an effect size of 0 is consistent with chance, and social scientists have, by convention, declared an effect size of 0.2 as small, 0.5 as medium and 0.8 as large. A medium effect size is supposed to be visible to the naked eye of a careful observer, while a large effect size is supposed to be evident to any observer. 2.2.3 RANDOMNESS AND RANK-ORDER JUDGING. At the heart of any statistical method is a definition of what should happen "randomly" or “by chance.” Without a random mechanism, there can be no statistical evaluation. There is nothing random about the responses generated in anomalous cognition experiments; in other words, there is no way to define what they would look like "by chance." Therefore, the random mechanism in these experiments must be in the choice of the target. In that way, we can compare the response to the target and answer the question: "If chance alone is at work, what is the probability that a target would be chosen that matches this response as well as or better than does the actual target?" In order to accomplish this purpose, a properly conducted experiment uses a set of targets defined in advance. The target for each remote viewing is then selected randomly, in such a way that the probability of getting each possible target is known. The SAIC remote viewing experiments and all but the early ones at SRI used a statistical evaluation method known as rank-order judging. After the completion of a remote viewing, a judge who is blind to the true target (called a blind judge) is shown the response and five potential targets, one of which is the correct answer and the other four of which are "decoys.' Before the experiment is conducted each of those five choices must have had an equal chance of being selected as the actual target. The judge is asked to assign a rank to each of the possible targets, where a rank of one means it matches the response most closely, and a rank of five means it matches the least. The rank of the correct target is the numerical score for that remote viewing. By chance alone the actual target would receive each of the five ranks with equal likelihood, since despite what the response said the target matching it best would have the same chance of selection as the one matching it second best and so on. The average rank by chance would American Institutes for Research 3-7 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews ee be three. Evidence for anomalous cognition occurs when the average rank over a series of trials is significantly lower than three. (Notice that a rank of one is the best possible score for each viewing.) This scoring method is conservative in the sense that it gives no extra credit for an excellent match. A response that describes the target almost perfectly will achieve the same rank of one as a response that contains only enough information to pick the target as the best choice out of the five possible choices. One advantage of this method is that it is still valid even if the viewer knows the set of possible targets. The probability of a first place match by chance would still be only one in five. This is important because the later SRI and many of the SAIC experiments used the same large set of National Geographic photographs as targets. Therefore, the experienced viewers would eventually become familiar with the range of possibilities since they were usually shown the answer at the end of each remote viewing session. For technical reasons explained in Appendix 1, the effect size for a series of remote viewings using rank-order judging with five choices is (3.0 - average rank). Therefore, small, medium and large effect sizes (0. 1, 0. 5 and 0. 8) correspond to average ranks of 2.86, 2.29 and 1.87, respectively. Notice that the largest effect size possible using this method is 1.4, which would result if every remote viewing achieved a first place ranking. 2.3 METHODOLOGICAL ISSUES One of the challenges in designing a good experiment in any area of science is to close the loopholes that would allow explanations other than the intended one to account for the results. There are a number of places in remote viewing experiment where information could be conveyed by normal means if proper precautions are not taken. The early SRI experiments suffered from some of those problems, but the later SRI experiments and the SAIC work were done with reasonable methodological rigor, with some exceptions noted in the detailed descriptions of the SAIC experiments in Appendix 2. American Institutes for Research 3-8 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
er ls EE a A RNS Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews ne The following list of methodological issues shows the variety of concerns that must be addressed. It should be obvious that a well-designed experiment requires careful thought and planning: . No one who has knowledge of the specific target should have any contact with the viewer until after the response has been safely secured. ° No one who has knowledge of the specific target or even of whether or not the session was successful should have any contact with the judge until after that task has been completed. ° No one who has knowledge of the specific target should have access to the response until after the judging has been completed. ° Targets and decoys used in judging should be selected using a well-tested randomization device. ° Duplicate sets of targets photographs should be used, one during the experiment and one during the judging, so that no cues (like fingerprints) can be inserted onto the target that would help the judge recognize it. ° The criterion for stopping an experiment should be defined in advance so that it is not called to a halt when the results just happen to be favorable. Generally, that means specifying the number of trials in advance, but some statistical procedures require or allow other stopping rules. The important point is that the rule be defined in advance in such a way that there is no ambiguity about when to stop. . Reasons, if any; for excluding data must be defined in advance and followed consistently, and should not be dependent on the data. For example, a rule specifying that a trial could be aborted if the viewer felt ill would be legitimate, but only if the trial was aborted before anyone involved in that decision knew the correct target. American Institutes for Research 3-9 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
eee Seren Ae SS i SDAA Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 a Chapter Three: Research Reviews * Statistical analyses to be used must be planned in advance of collecting the data so that a method most favorable to the data isn't selected post hoc, If multiple methods of analysis are used the corresponding conclusions must recognize that fact. 2.4 PRIMA FACIE EVIDENCE According to Webster's Dictionary, in law prima facie evidence is “evidence having such a degree of probability that it must prevail unless the contrary be proved." There are a few examples of applied, non-laboratory remote viewings provided to the review team that would seem to meet that criterion for evidence. These are examples in which the sponsor or another government client asked for a single remote viewing of a site, known to the requester in real time or in the future, and the viewer provided details far beyond what could be taken as a ~ reasonable guess. Two such examples are given by May (1995) in which it appears that the results were so striking that they far exceed the phenomenon as observed in the laboratory. Using a post hoc analysis, Dr. May concluded that in one of the cases the remote viewer was able to describe a microwave generator with 80 percent accuracy, and that of what he said almost 70 percent of it was reliable. Laboratory remote viewings rarely show that level of correspondence. Notice that standard statistical methods cannot be used in these cases because there is no standard for probabilistic comparison. But evidence gained from applied remote viewing cannot be dismissed as inconsequential just because we cannot assign specific probabilities to the results. It is most important to ascertain whether or not the information was achievable in other standard ways. In Section 3 an example is given in which a remote viewer allegedly gave codewords from a secret facility that he should not have even known existed. Suppose the sponsors could be absolutely certain that the viewer could not have known about those codewords through normal means, Then even if we can't assign an exact probability to the fact that he guessed them correctly, we can agree that it would be very small. That would seem to constitute prima facie evidence unless an alternative explanation could be found. Similarly, the viewer who described the microwave generator allegedly knew only that the target was a technical site in the United States. Yet, he drew and described the microwave generator, including its function, its approximate size, how it was housed and that it had "a beam divergence angle of 30 degrees" (May, 1995, p. 15), American Institutes for Research 3-10 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 i ‘Chapter Three: Research Reviews Anecdotal reports of psychic functioning suffer from a similar problem in terms of their usefulness as proof They have the additional difficulty that the "response" isn't even well- defined in advance, unlike in applied remote viewing where the viewer provides a fixed set of information on request. For instance, if a few people each night happen to dream of plane crashes, then some will obviously do so on the night before a major plane crash. Those individuals may interpret the coincidental timing as meaningful. This is undoubtedly the reason many people think the reality of psychic functioning is a matter of belief rather than science, since they are more familiar with the provocative anecdotes than with the laboratory evidence. 3.1 THE SRI ERA 3.1 EARLY OPERATIONAL SUCCESSES AND EVALUATION According to Puthoff and Targ (1975) the scientific research endeavor at SRI may never have been supported had it not been for three apparent operational successes in the early days of the program. These are detailed by Puthoff and Targ (1975), although the level of the matches is not clearly delineated. One of the apparent successes concerned the "West Virginia Site" in which two remote viewers purportedly identified an underground secret facility. One of them apparently named codewords and personnel in this facility accurately enough that it set off a security investigation to determine how that information could have been leaked. Based only on the coordinates of the site, the viewer first described the above ground terrain, then proceeded to describe details of the hidden underground site. The same viewer then claimed that he could describe a similar Communist Bloc site and proceeded to do so for a site in the Urals. According to Puthoff and Targ "the two reports for the West Virginia Site, and the report for the Urals Site were verified by personnel in the sponsor organization as being substantially correct (p. 8)." American Insiltutes for Research mB Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews nn The third reported operational success concerned an accurate description of a large crane and other information at a site in Semipalatinsk, USSR. Again the viewer was provided with only the geographic coordinates of the site and was asked to describe what was there. Although some of the information in these examples was verified to be highly accurate, the evaluation of operational work remains difficult, in part because there is no chance baseline for comparison (as there is in controlled experiments) and in part because of differing expectations of different evaluators. For example, a government official who reviewed the Semipalatinsk work concluded that there was no way the remote viewer could have drawn the large gantry crane unless "he actually saw it through remote viewing, or he was informed of what to draw by someone knowledgeable of [the site]." Yet that same analyst concluded that "the remote viewing of [the site] by subject S1 proved to be unsuccessful" because "the only positive evidence of the rail-mounted gantry crane was far outweighed by the large amount of negative evidence noted in the body of this analysis." In other words, the analyst had the expectation that in order to be "successful" a remote viewing should contain accurate information only. Another problem with evaluating this operational work is that there is no way to know with certainty that the subject did not speak with someone who had knowledge of the site, however unlikely that possibility may appear. Finally, we do not know to what degree the results in the reports were selectively chosen because they were correct. These problems can all be avoided with well designed controlled experiments. 3.2 THE EARLY SCIENTIFIC EFFORT AT SRI During 1974 and early 1975 a number of controlled experiments were conducted to see if various types of target material could be successfully described with remote viewing. The results reported by Puthoff and Targ (1975) indicated success with a wide range of material, from "technical" targets like a xerox machine to natural settings, like a swimming pool. But these and some of the subsequent experiments were criticized on statistical and methodological grounds; we briefly describe one of the experiments and criticisms of it to show the kinds of problems that existed in the early scientific effort. merican Institutes for Researc. 3-12 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews SU The largest series during in the 1973 to 1975 time period involved remote viewing of natural sites. Sites were randomly selected for each trial from a set of 100 possibilities. They were selected "without replacement," meaning that sites were not reused once they had been selected. The series included eight viewers, including two supplied by the sponsor. Many of the descriptions showed a high degree of subjective correspondence, and the overall statistical results were quite striking for most of the viewers. Critics attacked these experiments on a number of issues, including the selection of sites without replacement and the statistical scoring method used. The results were scored by having a blind judge attempt to match the target material with the transcripts of the responses. A large fraction of the matches were successful. But critics noted that some successful matching could be attained just from cues contained in the transcripts of the material, like when a subject mentioned in one session what the target had been in the previous session. Because sites were selected without replacement, knowing what the answer was on one day would exclude that target site from being the answer on any other day. There was no way to determine the extent to which these problems influence the results. The criticisms of these and subsequent experiments, while perhaps unwelcome at the time, have resulted in substantially improved methodology in these experiments. 3.3 AN OVERALL ANALYSIS OF THE SRI EXPERIMENTS: 1973-1988 In 1988 an analysis was made of all of the experiments conducted at SRI from 1973 until that time (May et al, 1988). The analysis was based on all 154 experiments conducted during that era, consisting of over 26,000 individual trials. Of those, almost 20,000 were of the forced choice type and just over a thousand were laboratory remote viewings. There were a total of 227 subjects in all experiments. The statistical results were so overwhelming that results that extreme or more so would occur only about once in every 1020 such instances if chance alone is the explanation (i.e., the p-value was less than 10-'). Obviously some explanation other than chance must be found. Psychic functioning may not be the only possibility, especially since some of the earlier work contained methodological problems. However, the fact that the same level of functioning continued to hold in the later experiments, which did not contain those flaws, lends support to the idea that the methodological problems cannot account for the results, In fact, there was a merican Institutes for Researe. o+7 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews nna Ona talented group of subjects (labeled G1 in that report) for whom the effects were stronger than for the group at large. According to Dr. May, the majority of experiments with that group were conducted later in the program, when the methodology had been substantially improved. In addition to the statistical results, a number of other questions and patterns were examined. A summary of the results revealed the following: 1. "Free response" remote viewing, in which subjects describe a target, was much more successful than "forced choice" experiments, in which subjects were asked to choose from a small set of possibilities. 2, There was a group of six selected individuals whose performance far exceeded that of unselected subjects. The fact that these same selected individuals consistently performed better than others under a variety of protocols provides a type of replicability that helps substantiate the validity of the results. If methodological problems were responsible for the results, they should not have affected this group differently from others. 3. Mass-screening efforts found that about one percent of those who volunteered to be tested were consistently successful at remote viewing. This indicates that remote viewing is an ability that differs across individuals, much like athletic ability or musical talent. (Results of mass screenings were not included in the formal analysis because the conditions were not well-controlled, but the subsequent data from subjects found during mass-screening were included.) 4, Neither practice nor a variety of training techniques consistently worked to improve remote viewing ability. It appears that it is easier to find than to train good remote viewers. 5. It is not clear whether or not feedback (showing the subject the right answer) is necessary, but it does appear to provide a psychological boost that may increase performance. American Institutes for Research 3-14 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 ” Chapter Three: Research Reviews nnn 6. Distance between the target and the subject does not seem to impact the quality of the remote viewing. 7. Electromagnetic shielding does not appear to inhibit performance. 8. There is compelling evidence that precognition, in which the target is selected after the subject has given the description, is also successful. 9. There is no evidence to support anomalous perturbation (psychokinesis), ie. physical interaction with the environment by psychic means. 3.4 CONSISTENCY WITH OTHER LABORATORIES IN THE SAME ERA One of the hallmarks of a real phenomenon is that its magnitude is replicable by various researchers working under similar conditions. The results of the overall SRI analysis are consistent with results of similar experiments in other laboratories. For instance, an overview of forced choice precognition experiments (Honorton and Ferrari, 1989) found an average "effect size" per experimenter of 0.033, whereas all forced choice experiments at SRI resulted in a similar effect size of .052, The comparison is not ideal since the SRI forced choice experiments were not necessarily precognitive and they used different types of target material than the standard card-guessing experiments. Methodologically sound remote viewing has not been undertaken at other laboratories, but a similar regime called the ganzfeld (described in more detail in Section 5) has shown to be similarly successful. The largest collection of ganzfeld experiments was conducted from 1983 to 1989 at the Psychophysical Research Laboratories in Princeton, NJ. Those experiments were also reported by separating novices from experienced subjects. The overall effect size for novice remote viewing at SRI was 0. 164, while the effect size for novices in the ganzfeld at PRL was a very similar 0.17. For experienced remote viewers at SRI the overall effect size was 0.385; for experienced viewers in the ganzfeld experiments it was 0.35. These consistent results across laboratories help refute the idea that the successful experiments at any one lab are the result of fraud, sloppy protocols or some methodological problem and also provide an indication of what can be expected in future experiments. ne American institutes for Research 3-15 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 é i Chapter Three: Research Reviews 4, THE SAIC ERA 4.1 AN OVERVIEW The review team decided to focus more intensively on the experiments conducted at Science Applications International Corporation (SAIC), because they provide a manageable yet varied set to examine in detail. They were guided by a Scientific Oversight Committee consisting of experts in a variety of disciplines, including a winner of the Nobel Prize in Physics, internationally known professors of statistics, psychology, neuroscience and astronomy and a medical doctor who is a retired U.S. Army Major General. Further, we have access to the details for the full set of SAIC experiments, unlike for the set conducted at SRI, Whatever details may be missing from the written reports are obtainable from the principal investigator, Dr. Edwin May, to whom we have been given unlimited access. In a memorandum dated July 25, 1995, Dr. Edwin May listed the set of experiments conducted by SAIC. There were ten experiments, all designed to answer questions about psychic functioning, raised by the work at SRI and other laboratories, rather than just to provide additional proof of its cxistence. Some of the experiments were, of a similar format to the remote viewing experiments conducted at SRI and we can examine those to see whether or not they replicated the SRI results. We will also examine what new knowledge can be gained from the results of the SAIC work. 4.2 THE TEN EXPERIMENTS Of the ten experiments done at SAIC, six of them involved remote viewing and four did not. Rather than list the details in the body of this report, Appendix 2 gives a brief description of the experiments. What follows is a discussion of the methodology and results for the experiments as a whole. Because of the fundamental differences between remote viewing and the other types of experiments, we discuss them separately. In the memorandum of 25 July 1995, Dr. May provided the review team with details of the ten experiments, including a short title, number of trials, effect size and overall p-value for each one. His list was in time sequence. It is reproduced in Table 1, using his numbering system, with the experiments categorized by type, then sequentially within type. The effect American Institutes for Research 3-16 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 a Chapter Three: Research Reviews 4.4 CONSISTENCY AND REPLICABILITY OF THE REMOTE VIEWING RESULTS One of the most important hallmarks of science is replicability. A phenomenon with statistical variability, whether it is scoring home runs in baseball, curing a disease with chemotherapy or observing psychic functioning, should exhibit about the same level of success in the long run, over repeated experiments of a similar nature. The remote viewing experiments are no exception. Remember that such events should not replicate with any degree of precision in the short run because of statistical variability, just as we would not expect to always get five heads and five tails if we flip a coin ten times, or see the same batting averages in every game. The analysis of SRI experiments conducted in 1988 singled out the laboratory remote viewing sessions performed by six "expert" remote viewers, numbers 002, 009, 131, 372, 414 and 504. These six individuals contributed 196 sessions. The resulting effect size was 0.385 (May et al, 1988, p. 13). The SRI analysis does not include information individually by viewer, nor does it include information about how many of the 196 sessions used static versus dynamic targets. One report provided to the review team (May, Lantz and Piantineda) included an additional experiment conducted after the 1988 review was performed, in which Viewer 009 participated with 40 sessions. The effect size for Viewer 009 for those sessions was .363. None of the other six SRI experts were participants. The same subject identifying numbers were used at SAIC, so we can compare the performance for these individuals at SRI and SAIC. Of the six, three were specifically mentioned as participating in the SAIC remote viewing experiments. As can be seen in Table 2, viewers 009, 131 and 372 all participated in Experiments I and viewers 009 and 372 participated in Experiments 4, 9 and 10 as well. The overall effect sizes for two of the three, viewers 009 and 372, were very close to the SRI effect size of 0.385 for these subjects, at .35 and .34, respectively, and the .35 effect size for Viewer 009 was very similar to his .363 effect size in the report by May, Lantz and Piantineda (1994). Therefore, we see a repeated and, more importantly, hopefully a repeatable level of functioning above chance for these individuals. An effect of this size should be reliable enough to be sustained in any properly conducted experiment with cnough trials to obtain the long run statistical replicability required to rule out chance. American Institutes for Research 3-19 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews enn It is also important to notice that viewers 009 and 372 did well on the same experiments and poorly on the same experiments. In fact the correlation between their effect sizes across experiments is .901, which is very close to a perfect correlation of 1.0. This kind of consistency warrants investigation to determine whether it is the nature of the experiments, a statistical fluke or some methodological problems that led these two individuals to perform so closely to one another. If methodological problems are responsible, then they must be subtle indeed because the methodology was similar for many of the experiments, yet the results were not. For instance, procedures for the sessions with static and dynamic targets in Experiment 1 were almost identical to each other, yet the dynamic targets did not produce evidence of psychic functioning (p-value = .50) and the static targets did (P-value = .0073). Therefore, a methodological problem would have had to differentially effect results for the two types of targets, even though the assignment of target type was random across sessions. 4.5 METHODOLOGICAL ISSUES IN THE REMOTE VIEWING EXPERIMENTS AT SAIC As noted in Section 2.3, there are a number of methodological considerations needed to perform a careful remote viewing experiment. Information necessary to determine how well each of these were addressed is generally available in the reports, but in some instances ] consulted Dr. May for additional information. As an example of how the methodological issues in Section 2.3 were addressed, an explanation will be provided for Experiment 1. In this experiment the viewers all worked from their homes (in New York, Kansas, California, and Virginia). Dr. Nevin Lantz, who resided in Pennsylvania, was the principal investigator. After each session, viewers faxed their response to Dr. Lantz and mailed the original to SAIC. Upon receipt of the fax, Dr. Lantz mailed the correct answer to the viewer. The viewers were supposed to mail their original responses to SAIC immediately, after taxing them to Dr. Lantz. According to Dr. May, the taxed versions were later compared with the originals to make sure the originals were sent without any changes. Here are how the other methodological issues in Section 2.3 were handled: ° No one who has knowledge of the specific target should have any contact with the viewer until after the response has been safely secured. merican Institutes for Researe 3-20 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews nrc No one involved with the experiment had any contact with the viewers, since they were not in the vicinity of either SAIC or Dr. Lantz's home in Pennsylvania. ° No one who has knowledge of the specific target or even of whether or not the session was successful should have any contact with the judge until after that task has been completed. Dr. Lantz and the individual viewers were the only ones who knew the correct answers, but according to Dr. May, they did not have any contact with the judge during the period of this experiment. ° No one who has knowledge of the specific target should have access to the response until after the judging has been completed. Again, since only the viewers and Dr. Lantz knew the correct target, and since the responses were mailed to SAIC by the viewers before they received the answers, this condition appears to have been met. . Targets and decoys used in judging should be selected using a well-tested randomization device. This has been standard practice at both SRI and SAIC. ° Duplicate sets of targets photographs should be used, one during the experiment and : one during the judging, so that no cues (like fingerprints) can be inserted onto the target that would help the judge recognize it. This was done; Dr. Lantz maintained the set used during the experiment while the set used for judging was kept at SAIC in California. ° The criterion for stopping an experiment should be defined in advance so that it is not called to a halt when the results just happen to be favorable, Generally, that means specifying the number of trials in advance, but some statistical procedures require other stopping rules. The important point is that the rule be defined in advance in American Institutes for Research 3-27 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 I Chapter Three: Research Reviews such a way that there is no ambiguity about when io stop. In advance it was decided that each viewer would contribute 40 trials, ten under each of four conditions (all combinations of sender/no sender and static/dynamic). All sessions were completed. Reasons, if any, for excluding data must be defined in advance and followed consistently, and should not be dependent on the data. For example, a rule specifying that a trial could be aborted if the viewer felt ill would be legitimate, but only if the trial was aborted before anyone involved in that decision knew the correct target. No such reasons were given, nor was there any mention of any sessions being aborted or discarded. . Statistical analyses to be used must be planned in advance of collecting the data so that a method most favorable to the data isn't selected post hoc, If multiple methods of analysis are used the corresponding conclusions must recognize that fact. The standard rank-order judging had been planned, with results reported separately for each of the four conditions in the experiment for each viewer. Thus, 20 effect sizes were reported, four for each of the five viewers. 4.6 WAS ANYTHING LEARNED AT SAIC? 4.6.1 TARGET SELECTION. In addition to the question of whether or not psychic functioning is possible, the experiments at SAIC were designed to explore a number of hypotheses, Experiments I and 10 were both designed to see if there is a relationship between the "change in visual entropy" in the targets and the remote viewing performance. Each of the five senses with which we are familiar is a change detector. Our vision is most readily drawn to something that is moving, and in fact if our eyes are kept completely still, we cease to see at all. Similarly, we hear because of moving air, and our attention is drawn to sudden changes in sound levels. Other senses behave similarly. Thus, it is reasonable that if there really is a "psychic sense" then it would follow that same pattern. American Insututes for Research a2 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 “ Chapter Three: Research Reviews nena nnn rT Experiments 1 and 10 were designed to test whether or not remote viewing performance would be related to a particular type of change in the target material, namely the "change in visual entropy.” A target with a high degree of change would be one in which the colors changed considerably throughout the target. A detailed explanation can be found in the SAIC reports of this experiment, or in the article "Shannon Entropy: A Possible Intrinsic Target Property" by May, Spottiswoode and James, in the Journal of Parapsychology, December 1994. It was indeed found that there was a correlation between the change in entropy in the target and the remote viewing quality. This result was injtially shown in Experiment I and replicated in Experiment 10. A simulation study matching randomly chosen targets to response showed that this was unlikely to be an artifact of target complexity or other features. It is worth speculating on what this might mean for determining how psychic functioning works. Physicists are currently grappling with the concept of time, and -cannot rule out precognition as being consistent with current understanding. Perhaps it is the case that we do have a psychic sense, much like our other senses, and that it works by scanning the future for possibilities of major change much as our eyes scan the environment for visual change and our ears are responsive to auditory change. That idea is consistent with anecdotal reports of precognition, which are generally concerned with events involving major life change. Laboratory remote viewing may in part work by someone directing the viewer to focus on a particular point in the future, that in which he or she receives the feedback from the experiment. It may also be the case that this same sense can scan the environment in actual time and detect change as well. . Another hypothesis put forth at SAIC was that laboratory remote viewing experiments are most likely to be successful if the pool of potential targets is neither too narrow nor too wide in terms of the number of possible elements in the target. They called this feature the “target- pool bandwidth" and described it as the number of "differentiable cognitive elements." They reasoned that if the possible target set was too small, the viewer would see the entire set and be unable to distinguish that information from the psychic information. If the set was too broad, the viewer would not have any means for editing an extensive imagination. Combining these two results would indicate that a good target set would contain targets with high change in visual entropy, but that the set would contain a moderately-sized set of possibilities. The set of 100 National Geographic photographs used in the later days at SRI American Institutes for Research 3-23 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
a a Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 SCE Chapter Three: Research Reviews and at SAIC may have inadvertently displayed just those properties. _ 46.2 REMOTE STARING. Experiment 7, described in Appendix 2, provided results very different from the standard remote viewing work. That experiment was designed to test claims made in the Former Soviet Union and by some researchers in the United States, that individuals could influence the physiology of another individual from a remote location. The study was actually two separate replications of the same experiment, and both replications were successful from a traditional statistical perspective. In other words, it appeared that the physiology of one individual was activated when he or she was being watched by someone in a distant room. If these results are indeed sound, then they may substantiate the folklore indicating that people know when they are being observed from behind. 4.6.3 ENHANCED BINARY COMPUTER GUESSING. Experiment 2 was also very different from the standard remote viewing experiments, although it was still designed to test anomalous cognition. Three subjects attempted to use a statistical enhancement technique to increase the ability to guess forced choice targets with two choices. This clever computer experiment showed that for one subject, guessing was indeed enhanced from a raw rate of just above chance (51.6% instead of 50%) to an enhanced rate of 76 percent. The method was extremely inefficient, and it is difficult to imagine practical uses for this ability, if indeed it exists. 5. EXTERNAL VALIDATION: REPLICATIONS OF OTHER EXPERIMENTS 5.1 CONCEPTUAL SIMILARITY: GANZFELD EXPERIMENTS While remote viewing has been the primary activity at SRI and SAIC, other researchers have used a similar technique to test for anomalous cognition, called the ganzfeld. As noted in the SAIC Final Report of 29 Sept. 1994, the ganzfeld experiments differ from remote viewing in three fundamental ways. First, a "mild altered state is used," second, senders are [usually] used, so that telepathy is the primary mode, and third, the receivers (viewers) do their own judging just after the session, rather than having an independent judge. American Institutes for Research 3-24 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews a The ganzfeld experiments conducted at Psychophysical Research Laboratories (PRL) were already mentioned in Section 3.4. Since the time those results were reported, other laboratories have also been conducting ganzfeld experiments. At the 1995 Annual Meeting of the Parapsychological Association, three replications were reported, all published in the peer- reviewed Proceedings of the conference. The ganzfeld experiments differ in the preferred method of analysis as well, Rather than using the sum of the ranks across sessions, a simple count is made of how many first places matches resulted from a series. Four rather than five choices are given, so by chance there should be about 25% of the sessions resulting in first place matches. 5.2 GANZFELD RESULTS FROM FOUR LABORATORIES In publishing the ganzfeld results from PRL, Bem and Honorton (1994) excluded one of the studies from the general analysis for methodological reasons, and found that the remaining studies showed 106 hits out of 329 sessions, for a hit rate of 32.2 percent when 25 percent was expected by chance. The corresponding p-value was .002. As mentioned earlier, the hallmark of science is replication. This result has now been replicated by three additional laboratories. Bierman (1995) reported four series of experiments conducted at the University of Amsterdam. Overall, there were 124 sessions and 46 hits, for a hit rate of 37 percent. The hit rates for the four individual experiments were 34.3 percent, 37.5 percent, 40 percent and 36.1 percent, so the results are consistent across his four experiments. Morris, Dalton, Delanoy and Watt (1995) reported results of 97 sessions conducted at the University of Edinburgh in which there were 32 successes, for a hit rate of 33 percent. They conducted approximately equal numbers of sessions under each of three conditions, In one condition there was a known sender, and in the other two conditions it was randomly determined at the last minute (and unknown to the receiver) that there would either be a sender or not. Hit rates were 34 percent when there was a known sender and when there was no sender, and 28 percent when there was a sender but the receiver did not know whether or not there would be. They did discover post hoc that one experimenter was more successful than the other two at achieving successful sessions, but the result was not beyond what would American Institutes for Research 3-25 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 a Chapter Three: Research Reviews be expected by chance as a post hoc observation. Broughton and Alexander (1995) reported results from 100 sessions at the Institute for Parapsychology in North Carolina. They too found a similar hit rate, with 33 hits out of 100 sessions, or 33 percent hits. Results from the original ganzfeld work and these three replications are summarized in Table 3, along with the SRI and SAIC remote viewing results. The effect sizes for the ganzfeld replications are based on Cohen's h, which is similar in type to the effect size used for the remote viewing data. Both effect sizes measure the number of standard deviations the results fall above chance, using the standard deviation for a single session. TABLE 3: REMOTE VIEWING AND GANZFELD REPLICATIONS Laboratory Hit Rate Effect Size All Remote Viewing at SRI All Remote Viewing at SAIC University of Amsterdam, Netherlands 97 00 5.3 CONCLUSIONS ABOUT EXTERNAL REPLICATION Institute of Parapsychology, NC . The results shown in Table 3 show that remote viewing has been conceptually replicated across a number of laboratories, by various experimenters and in different cultures. This is a robust effect that, were it not in such an unusual domain, would no longer be questioned by science as a real phenomenon. It is unlikely that methodological problems could account for the remarkable consistency of results shown in Table 3. American Institutes for Research 3-26 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 i Chapter Three: Research Reviews 6. IS REMOTE VIEWING USEFUL? Even if we were all to agree that anomalous cognition is possible, there remains the question of whether or not it would have any practical use for government purposes. The answer to that question is beyond the scope of this report, but some speculations can be made about how to increase the usefulness. First, it appears that anomalous cognition is to some extent possible in the general population. None of the ganzfeld experiments used exclusively selected subjects. However, it also appears that certain individuals possess more talent than others, and that it is easier to find those individuals than to train people. It also appears to be the case that certain individuals are better at some tasks than others. For instance, Viewer 372 at SAIC appears to have a facility with describing technical sites. Second, if remote viewing is to be useful, the end users must be trained in what it can do and what it cannot. Given our current level of understanding, it is rarely 100 percent accurate, and there is no reliable way to learn what is accurate and what is not. The same is probably true of most sources of intelligence data. Third, what is useful for one purpose may not be useful for another. For instance, suppose a remote viewer could describe the setting in which a hostage is being held. That information may not be any use at all to those unfamiliar with the territory, but could be useful to those familiar with it. 7. CONCLUSIONS AND RECOMMENDATIONS It is clear to this author that anomalous cognition is possible and has been demonstrated. This conclusion is not based on belief, but rather on commonly accepted scientific criteria. The phenomenon has been replicated in a number of forms across laboratories and cultures. The various experiments in which it has been observed have been different enough that if some subtle methodological problems can explain the results, then there would have to be a different explanation for each type of experiment, yet the impact would have to be similar across experiments and laboratories. If fraud were responsible, similarly, it would require an American Institutes for Research 3-27 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 - a Chapter Three: Research Reviews equivalent amount of fraud on the part of a large number of experimenters or an even larger number of subjects. What is not so clear is that we have progressed very far in understanding the mechanism for anomalous cognition. Senders do not appear to be necessary at all; feedback of the correct answer may or may not be necessary. Distance in time and space do not seem to be an impediment. Beyond those conclusions, we know very little. I believe that it would be wasteful of valuable resources to continue to look for proof. No one who has examined all of the data across laboratories, taken as a collective whole, has been able to suggest methodological or statistical problems to explain the ever-increasing and consistent results to date. Resources should be directed to the pertinent questions about how this ability works. I am confident that the questions are no more elusive than any other questions in science dealing with small to medium sized effects, and that if appropriate resources are targeted to appropriate questions, we can have answers within the next decade. American institutes for Research 3-26 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 i Chapter Three: Research Reviews APPENDIX 1 EFFECT SIZE MEASURE USED WITH RANK ORDER JUDGING In general, effect sizes measure the number of standard deviation the true population value of interest falls from the value that would be true if chance alone were at work. The standard deviation used is for one subject, trial, etc., rather than being the standard error of the sample statistic used in the hypothesis test. In rank-order judging, let Ri be the rank for trial i. If the number of possible choices is N, then we find: E(R,) = (N+ D2 and 2 Var(R,)= “> Therefore, when N = 5, we find E(Rd 3 and Var(Rd = 2. The effect size is therefore: EffectSize (3.0 - Average Rank) F2 American Institutes for Research 3-30 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 a ce Chapter Three: Research Reviews APPENDIX 2 A BRIEF DESCRIPTION OF THE SAIC EXPERIMENTS EXPERIMENTS INVOLVING REMOTE VIEWING There were six experiments involving remote viewing, done for a variety of purposes. EXPERIMENT 1: TARGET AND SENDER DEPENDENCIES: PURPOSE: This expcriment was designed to test whether or not a sender is necessary for successful remote viewing and whether or not dynamic targets, consisting of short video clips, would result in more successful remote viewing than the standard National Geographic photographs used in most of the SRI experiments. METHOD: Five experienced remote viewers participated, three of whom (#s 009, 131 and 372) were included in the experienced group at SRI; their identification numbers were carried over to the SAIC experiments. Each viewer worked from his or her home and faxed the results of the sessions to the principal investigator, Nevin Lantz, located in Pennsylvania. Whether the target was static or dynamic and whether or not there was a sender was randomly determined and unknown to the viewer. Upon receiving the fax of the response, Dr. Lantz mailed the correct answer to the viewer. The original response was sent to SAIC in California, where the results were judged by an analyst blind to the correct target. Standard rank-order judging was used. | Since it is not explicitly stated, I asked Dr. May what measures were taken to make sure the viewer actually mailed the original response to SAIC before receiving the correct answer in the mail. He said that the original faxed responses were compared with the responses received by SAIC to make sure they were the same, and they all were. RESULTS: Each viewer contributed ten trials under each of the four possible conditions (sender/no sender and static/dynamic target), for a total of 40 trials per viewer. There was a American institutes for Research 3-37 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews cen moderate difference (effect size = .121, p = .08) between the static and dynamic targets, with the traditional National Geographic photographs faring better than the dynamic video clips. There was no noticeable difference based on whether or not a sender was involved, supporting the same conclusion reached in the overall analysis of the SRI work. Combined over all conditions and all viewers, the effect size was 0. 124 (p = .04); for the static targets alone it was .248 (exact p = .0073) while for the dynamic targets it was 0.00 (p = .50). DISCUSSION: The SAIC staff speculated that the dynamic targets were not successful because the possibilities were too broad. They chose a new set of dynamic targets to be more similar to the static targets and performed another experiment the following year to compare the static targets with the more similar set of dynamic ones. That experiment is described below (Experiment 10.) EXPERIMENT 4: ENHANCING DETECTION OF AC WITH BINARY CODING: PURPOSE: This experiment was designed to see if remote viewing could be used to develop a message-sending capability by focusing on the presence or absence of five specific features of a target. The target set was constructed in packets of four, with possible, combinations of the absence (0) or presence (1) of each of the five features chosen to correspond to the numbers 00000, 01 1 10, 10101, and I 101 1. This is standard practice in information theory when trying to send a two digit number (00, 01, 10 or 11); the remaining three bits are used for “error corrections." Different sets of five features were used for each of ten target packs. METHOD: Five viewers each contributed eight trials, but the same eight targets were used for all five viewers. There was no sender used, and viewers were told that each target would be in a fixed location for one week. They were to spend .15 minutes trying to draw the target, then fax their responses to SAIC in California. The results were blind-judged and the binary features were coded by both the viewers and an independent analyst. RESULTS: The results were unsuccessful in showing any evidence of psychic functioning. Neither standard rank-order judging nor analysis based on the binary guesses showed any promise that this method works to send messages. American Institutes for Research 3-32 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 a cee Chapter Three: Research Reviews EXPERIMENT 5: AC IN LUCID DREAMS (BASELINE): PURPOSE: Despite its name, this experiment did not involve lucid dreaming. Instead, it was used to test three novice remote viewers who were to participate in an experiment involving remote viewing while dreaming. This baseline experiment was designed to see if these individuals would be successful at standard laboratory remote viewing. METHOD: For this baseline experiment, each of the three viewers contributed eight trials using a standard protocol common in the SRI era, For each trial, a target was randomly chosen from the set of 100 National Geographic targets used at SRI and SAIC. The target was placed on a table (so no sender was used) while the viewer, in another room, was asked to provide a description. The response was later blind-judged by comparing it to the target and four decoys, and providing a rank-ordering of the five choices. RESULTS: Of the three novice viewers, one obtained a promising effect size of .265, although the result was not statistically significant due to the small number of trials (8). Individual results were not provided for the other two viewers, but the overall effect size was reported as 0.088 for the three viewers. EXPERIMENT 6: AC IN LUCID DREAMS (PILOT): PURPOSE: A lucid dream is a dream in which one becomes aware that he or she is dreaming, and can control subsequent events in the dream. This ability has apparently been successfully trained by Dr. Stephen LaBerge of the Lucidity Institute. He was the Principal Investigator for this experiment. The experiment was designed to see if remote viewing could be successfully employed while the viewer was having a lucid dream. METHOD: Seven remote viewers were used; four were experienced SAIC remote viewers and three were experienced lucid dreamers from the Lucidity Institute. The latter three were the novice viewers used in Experiment 5. The experienced SAIC remote viewers were given training in lucid dreaming. The number of trials contributed by each viewer could not be fixed in advance because of the difficulty of attaining the lucid dream state. A total of 21 trials were conducted, with the seven viewers contributing anywhere from one to seven trials each. The report did not mention whether or not the stopping criterion was fixed in advance, merican institutes for Researce 3-33 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
nn) Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 LN Chapter Three: Research Reviews but according to Dr. May the experiment was designed to proceed for a fixed time period and to include all sessions attained during that time period. Unlike with standard well-controlled protocols, the viewers were allowed to take the target material home with them. The targets, selected from the standard National Geographic pool, were sealed in opaque envelopes with covert threads to detect possible tampering (there were no indications of such tampering). Viewers were instructed to place the targets at bedside and to attempt a lucid dream in which the envelope was opened and the target viewed. Drawings and descriptions were then to be produced upon awakening. RESULTS: The results were blind-judged using the standard sum of ranks. Since the majority of viewers contributed only one or two trials, analysis by individual viewer would be meaningless. For the 21 trials combined, the effect size was 0.368 (p = .046). Information was not provided to differentiate the novice remote viewers from the experienced ones. EXPERIMENT 9: ERD AC BEHAVIOR: PURPOSE: The remote viewing in this experiment was conducted in conjunction with measurement of brain waves using an EEG. The purpose of the experiment was to see whether or not EEG activity would change when the target the person was attempting to describe was briefly displayed on a computer monitor in a distant room. Details of the EEG portion will be explained as experiment 8. Here, we summarize the remote viewing part of the study, METHOD: Three experienced remote viewers (#s 009, 372 and 389) participated. Because of the pilot nature of the experiment, the number of trials differed for each viewer based on availability, with viewers 009, 372 and 389 contributing 18, 24 and 28 trials, respectively. Although it is not good protocol to allow an unspecified number of trials, it does not appear that this problem can’ explain the results of this experiment. RESULTS: Responses were blind-judged using standard rank-order analysis. The effect sizes for the viewers 009, 372 and 389 were 0.432 (p = .033), 0.354 (p = .042) and 0. 177 (p .175), respectively. The overall effect size was 0.303 (p = 0.006). American Institutes for Research 3-34 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews ae EXPERIMENT 10: ENTROPY II: PURPOSE: This experiment was designed as an improved version of Experiment 1. After the unsuccessful showing ‘for the dynamic targets in Experiment 1, the SAIC team speculated that the 'target pool bandwidth” defined as the number of "cognitively differentiable elements" in the target pool might be an important factor. If the possible target material was extremely broad, viewers might have trouble filtering out extraneous noise. If the set of possibilities was too small, as in forced choice experiments, the viewer would see all choices at once and would have trouble filtering out that knowledge. An intermediate range of possibilities, too large to be considered all at once, was predicted to be ideal. The standard National Geographic pool seemed to fit that range. For this experiment, a pool of dynamic targets was created with a similar "band-width." In both Experiments (1 and 10) the researchers predicted that remote viewing success would correlate with the change in visual entropy of the target, as explained in Section 4.6.1. METHOD: Four of the five viewers from Experiment 1 were used (#s 009, 372, 389 and 518). They each contributed equal numbers of sessions with static and dynamic targets, with the viewers blind to which trials had which type. Senders were not used, and all sessions ,. were conducted at SAIC in California, unlike Experiment I in which the viewers worked at home. Viewer #372 contributed 15 of each type while the others each contributed 10 of each type. Standard rank-order judging was used. RESULTS: Table 4 shows the results for this experiment. Unlike in Experiment 1, the static and dynamic targets produced identical effect sizes, with both types producing very successful results. The combined effect size for all trials is .55, resulting in a z-score of 5.22. American institutes for Research 3-35 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews . TABLE 4: RESULTS FOR EXPERIMENT 10 ffs fe fe fe THE OTHER EXPERIMENTS AT SAIC There were four additional experiments at SAIC, not involving remote viewing. Two of them (experiments 3 and 8) involved trying to measure brain activity related to psychic functioning and will be described briefly. Experiment 3 used a magnetoenchephalograph (MEG) to attempt to detect anomalous signals in the brain when a remote stimulus was present. Due to the background noise in the brain measurements and the expected strength of the signal, the experimenters realized too late that they would not be able to detect a signal even if it existed. Experiment 8 utilized an EEG to try to detect the interruption of alpha waves when a remote viewing target was briefly displayed on a computer monitor in another room. The area‘of the brain tested was that corresponding to visual stimuli. No significant change in alpha was seen. . The remaining two experiments were replications of previous work measuring psychic functioning in areas other than remote viewing. They will be described in detail. EXPERIMENT 2: AC OF BINARY TARGETS: PURPOSE: This experiment attempted to replicate and enhance random number generator experiments conducted at SRI. In these types of experiments a computer randomly selects American Institutes for Research 3-36 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 ’ Chapter Three: Research Reviews a one of two choices to be the target, denoted as 0 or 1. The internal workings of the computer then rapidly oscillate between 0 and I and the subject pushes a mouse button when he or she thinks the internal choice matches the target choice. This process is repeated over many trials. The computer tabulates the results and the experiment is a success if the subject guesses the correct answer more often than would be expected by chance. The purpose is to see if humans can correctly guess computer-selected binary targets, and hopefully by extension, correctly solve binary choice problems in real situations. If that were to be the case, then real problems could be posed as binary ones (e.g. is the lost child still in this city or not) to narrow down possibilities. METHOD: This SAIC experiment was designed to enhance the accuracy of binary gucssing by using a statistical technique called sequential analysis. Rather than just one guess for each decision, the subject continues to guess until the computer ascertains that a decision has been reached. The computer keeps track of the number of times zero and one have each been guessed and announces a decision when one of the choices has clearly won out over the other, or when it is clear that it is essentially an ongoing tie. In the latter case, no decision is recorded. Three subjects participated (#s 007, 083 and 531) in this experiment. Subject #531 had been successful in similar experiments at SRI. RESULTS: Using this method for enhancing the accuracy of the guesses, subject #531, who had been successful in previous similar experiments, was able to achieve 76 correct answers out of 100 tries. This remarkable level of scoring for this type of experiment resulted in an effect size of .520 and a z-score of 5.20. The other two subjects did not differ from chance results, with 44 and 49 correct decisions out of 100 or 101. (One subject accidentally contributed an additional trial.) Although the result for subject 531 is remarkably successful, it does not represent a very efficient method of obtaining the decision. To reach the 100 decisions required a total of 21,337 guesses, i.e. over 200 guesses for each decision. Of the individual guesses, only 51.6 percent were correct, for an effect size of .032, similar to other forced choice experiments. Due to the large number of guesses, the corresponding z-score was 4.65, Combined over all three subjects, 56 percent of the 301 decisions were correct and the effect size was 0, 123. The combined results were still statistically significant, with p = 0. 0 17, as shown in Table 1. American institutes for Research 3-37 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews rt EXPERIMENT 7: REMOTE OBSERVATION: PURPOSE: It is often reported anecdotally that people know when they are being watched. Two experiments were conducted at SAIC to determine whether or not these anecdotes could be supported by a change in physiology when someone is being observed from a distance. The experimental design was essentially the same for the two experiments. This work was a conceptual replication of results reported by researchers in the Former Soviet Union (FSU), the United States and Scotland. The experiments in the FSU were interpreted to mean that the physiology of the recipient was being manipulated by the sender, an effect that if real could have frightening consequences. METHOD: The "observee" was seated in a room with a video camera focused on him or her, and with galvanic skin response measurements being recorded. In a distant room the "observer" attempted to influence the physiology of the observee at randomly spaced time intervals. During those time intervals, an image of the observee appeared on a computer monitor watched by the observer. During "control" periods, the video camera remained focused on the observee but the computer monitor did not display his or her image to the observer. There were 16 "influence" periods randomly interspersed with 16 "control" periods, each of 30 seconds, with blank periods of 0 to 5 seconds inserted to rule out patterns in physiology. RESULTS: To determine whether or not the galvanic skin response of the observees was activated while they were being watched, the response during the control periods was compared with the response during the "influence" periods for each subject. The results were then“averaged across subjects. In both experiments, there was greater activity during the periods of being watched than there was during the control periods. The results were statistically significant in each case (p = .036 and .014) and the effect sizes were similar, at 0.39 and 0.49. As preplanned, the results were combined, yielding an effect size of .39 (p = .005). As an interesting post hoc observation, it was noted that the effect was substantially stronger when the observer and observee were of opposite sexes than when they were of the same Sex. DISCUSSION: This experiment differs from the others conducted at SAIC since it involves interaction between two people rather than one person ascertaining information about the American Institutes for Research 3-38 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5| Chapter Three: Research Reviews environment or the future. It raises substantially more questions than it answers, since the mechanism for the shift in physiology is unknown. Possibilities range from the idea that the observee was able to know when the computer in the distant room was displaying his or her image, not unlike remote viewing, to the possibility that the observer actually did influence the physiology of the observee. Further experimentation as well as a review of similar past experiments may be able to shed light on this important question. American institutes for Researc 3- Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews Evaluation of Program on "Anomalous Mental Phenomena” Ray Hyman University of Oregon Department of Psychology Eugene, Oregon September 11, 1995 INTRODUCTION Professor Jessica Utts and I were given the task of evaluating the program on "Anomalous Mental Phenomena" carried out at SRI International (formerly the Stanford Research Institute) from 1973 through 1989 and continued at SAIC (Science Applications International Corporation) from 1992 through 1994. We were asked to evaluate this research in terms of its scientific value. We were also asked to comment on its potential utility for intelligence applications. The investigators use the term “Anomalous Mental Phenomena" to refer to what the parapsychologists label as psi. Psi includes both extrasensory perception (called Anomalous Cognition by the present investigators) and psychokinesis (called Anomalous Perturbation by the present investigators), The experimenters claim that their results support the existence of Anomalous Cognition--especially clairvoyance (information transmission from a target without the intervention of a human sender) and precognition. They found no evidence for the existence of Anomalous Perturbation. Our evaluation will focus on the 10 experiments conducted at SAIC. These are the most recent in the program as well as the only ones for which we have adequate documentation. The earlier SRI research on remote viewing suffered from methodological inadequacies. Another reason for concentrating upon this more recent set of experiments is the limited time frame allotted for this evaluation. American Institutes for Research 3-40 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 A Chapter Three: Research Reviews I will not ignore entirely the earlier SRI research. 1 will also consider some of the contemporary research in parapsychology at other laboratories. This is because a proper scientific evaluation of any research program has to place it in the context of the broader scientific community. In addition, some of this contemporary research was subcontracted by the SAIC investigators. Professor Utts has provided an historical overview of the SRI and SAIC programs as well as descriptions of the experiments under consideration. I will not duplicate what she has written on these topics. Instead, I will focus on her conclusions that: Using the standards applied to any other area of science, it is concluded that psychic functioning has been well established [Utts, Sept 1995, p I] Arguments that these results could be due to methodological flaws in the experiments are soundly refuted Effects of similar magnitude to those found in government- sponsored research at SRI and SAIC have been replicated at a number of laboratories across .he world. Such consistency cannot be readily explained by claims of flaws or fraud [Utts, Sept 1995, p I] Because my report will emphasize points of disagreement between Professor Utts and me, 1 want to state that we agree on many other points. We both agree that the SAIC experiments were free of the methodological weaknesses that plagued the early SRI research. We also agree that the SAIC experiments appear to be free of the more obvious and better known flaws that can invalidate the results of parapsychological investigations. We agree that the effect sizes reported in the SAIC experiments are too large and consistent to be dismissed as statistical flukes. L also believe that Jessica Utts and I agree on what the next steps should be. We disagree on key questions such as: 1, Do these apparently non-chance effects justify concluding that the existence of anomalous cognition has been established? American Institutes for Research 3-47 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews I 2. Has the possibility of methodological flaws been completely eliminated? 3. Are the SAIC results consistent with the contemporary findings in other parapsychological laboratories on remote viewing and the ganzfeld phenomenon? The remainder of this report will try to justify why I believe the answer to these three questions is "no". SCIENTIFIC STATUS OF THE PROGRAM Science is basically a communal activity. For any developed field of inquiry, a community of experts exist. This community provides the disciplinary matrix which determines what questions are worth asking, which issues are relevant, what variables matter and which can be safely ignored, and the criteria for judging the adequacy of observational data. The community provides checks and balances through the referee system, open criticism, and independent replications. Only those relationships that are reasonably Jawful and replicable across independent laboratories become part of the shared scientific store of "knowledge." An individual investigator or laboratory can contribute to this store. However, by itself, the output of a single investigator or laboratory does not constitute science. No matter how careful and competent the research, the findings of a single laboratory count for nothing unless they can be reliably replicated in other laboratories. This rule is true of ordinary claims. It holds true especially for claims that add something new or novel to the existing database. When an investigator, for example, announces the discovery of a new element, the claim is not accepted until the finding has been successfully replicated by several independent laboratories, Of course, this rule is enforced even more when the claim has revolutionary implications that challenge the fundamental principles underlying most sciences. GENERAL SCIENTIFIC HANDICAPS OF THE SAIC PROGRAM American Institutes for Research 3-42 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews enn The brief characterization of scientific inquiry in the preceding section alerts us to serious problems in trying to assess the scientific status of the SAIC research. The secrecy under which the SRI and SAIC programs was conducted necessarily cut them off from the communal aspects of scientific inquiry. The checks and balances that come from being an open part of the disciplinary matrix were absent. With the exception of the past year or so, none of the reports went through the all-important peer-review system. Worse, promising findings did not have the opportunity of being replicated in other laboratories. The commendable improvements in protocols, methodology, and data-gathering have not profited from the general shake-down and debugging that comes mainly from other laboratories trying to use the same improvements. Although the research program that started in 1973 continued for over twenty years, the secrecy and other constraints have produced only ten adequate experiments for consideration. Unfortunately, ten experiments--especially from one laboratory (considering the SAIC program as a continuation of the SRI program)--is far too few to establish reliable relationships in almost any area of inquiry. In the traditionally elusive quest for psi, ten experiments from one laboratory promise very little in the way of useful conclusions. The ten SAIC experiments suffer another handicap in their quest for scientific status. The principal investigator was not free to run the program to maximize scientific payoff. Instead, he had to do experiments and add variables to suit the desires of his sponsors. The result was an attempt to explore too many questions with too few resources. In other words, the scientific inquiry was spread too thin. The 10 experiments were asked to provide too many sorts of information. For these reasons, even before we get to the details (and remember the devil is usually in the details), the scientific contribution of this set of studies will necessarily be limited. PARAPSYCHOLOGY’S STATUS AS A SCIENCE American Institutes for Research 3-43 . Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
A Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews Parapsychology began its quest for scientific status in the mid-1800s, At that time it was known as Psychical research. The Society for Psychical Research was founded in London in 1882. Since that time, many investigators--including at least four Nobel laureates--have tried to establish parapsychology as a legitimate science. Beginning in the early 1930s, J.B. Rhine initiated an impressive program to distance parapsychology from its tainted beginnings in spiritualistic seances and turn it into an experimental science. He pulled together various ideas of his predecessors in an attempt to make the study of ESP and PK a rigorous discipline based on careful controls and statistical analysis. His first major publication caught the attention of the scientific community. Many were impressed with this display of a huge database, gathered under controlled conditions, and analyzed with the most modem statistical tools. Critics quickly attacked the statistical basis of the research. However, Burton Camp, the president of the Institute of Mathematical Statistics, came to the parapsychologists' defense in 1937. He issued a statement that if the critics were going to fault parapsychological research they could not do so on statistical grounds. The critics then turned their attention to methodological weaknesses. Here they had more success. What really turned scientists against parapsychological claims, however, was the fact that several scientists failed to replicate Rhine's results. This problem of replicability has plagued parapsychology ever since. The few, but well-publicized, cheating scandals that were uncovered also worked against parapsychology's acceptance into the general scientific community. Parapsychology shares with other sciences a number of features, The database comes from experiments using controlled procedures, double-blind techniques where applicable, the latest and most sophisticated apparatus, and sophisticated statistical analysis. In addition, the findings are reported at annual meetings and in refereed journals, Unfortunately, as I have pointed out elsewhere, parapsychology has other characteristics that make its status as a normal science problematic. Here I will list only a few. These are worth mentioning because they impinge upon the assessment of the scientific status of the SAIC program. Probably the most frequently discussed problem is the issue of replicability. Both critics and parapsychologists have agreed that the lack of consistently replicable results has American institutes for Researc 3-4 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 ——— Chapter Three: Research Reviews been a major reason for parapsychology's failure to achieve acceptance by the scientific establishment. Some parapsychologists have urged their colleagues to refrain from demanding such acceptance until they can put examples of replicable experiments before the scientific community. The late parapsychologist J.G.Pratt went further and argued that parapsychology would never develop a replicable experiment. He argued that psi was real but would forever elude deliberate control. More recently, the late Honorton claimed that the ganzfeld experiments had, indeed, achieved the status of a replicable paradigm. The title of the landmark paper in the January 1994 issue of the Psychological Bulletin by Bem and Honorton is "Does psi exist? Replicable evidence for an anomalous process of information transfer.?' In her position paper “Replication and meta-analysis in parapsychology" (Statistical Science, 1991, 6, pp. 363-403), Jessica Utts reviews the evidence from meat-analyses of parapsychological research to argue that replication has been demonstrated and “that the overall evidence indicates that there is an anomalous effect in need of explanation.” In evaluating the SAIC research, Utts points to the consistency of effect sizes produced by the expert viewers across experiments as well as the apparent consistency of average effect sizes of the SRI and SAIC experiments with those from other parapsychological laboratories. These consistencies in effect sizes across experiments and laboratories, in her opinion, justify the claim that anomalous mental phenomena can be reliably replicated with appropriately designed experiments. This is an important breakthrough for parapsychology, if it is true. However, to anticipate some of my later commentary, I wish to emphasize that simply replicating effect size is not the same thing as showing the repeated occurrence of anomalous mental phenomena. Effect size is nothing more than a standardized difference between an observed and an expected outcome hypothesized on the basis of an idealized probability model. An indefinite number of factors can cause departures from the idealized probability model. An investigator needs to go well beyond the mere demonstration that effect sizes are the same before he/she can legitimately claim that they are caused by the same underlying phenomenon, In my opinion, a more serious challenge to parapsychology's quest for scientific status is the lack of cumulativeness in its database. Only parapsychology, among the fields of inquiry merican Institutes for Researc 3- Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 a Chapter Three: Research Reviews claiming scientific status, lacks a cumulative database. Physics has changed dramatically since Newton conducted his famous experiment using prisms to show that white light contained all the colors of the spectrum. Yet, Newton's experiment is still valid and still yields the same results. Psychology has changed its ideas about the nature of memory since Ebbinghaus conducted his famous experiments on the curve of forgetting in the 1880s. We believe that memory is more dynamic and complicated than can be captured by Ebbinghaus’ ideas about a passive, rote memory system. Nevertheless, his findings still can be replicated and they form an important part of our database on memory. Parapsychology, unlike the other sciences, has a shifting database. Experimental data that one generation puts forth as rock-solid evidence for psi is discarded by later generations in favor of new data. When the Society for Psychical Research was founded in 1882, its first president Henry Sidgwick, pointed to the experiments with the Creery sisters as the evidence that should convince even the most hardened skeptic of the reality of psi. Soon, he and the other members of the Society argued that the data from Smith-Blackbum experiments provided the fraud-proof case for the reality of telepathy. The next generation of Psychical researchers, however, cast aside these cases as defective and we no longer hear about them. Instead, they turned to new data to argue their case. During the 1930s and 1940s, the results of Rhine's card guessing experiments were offered as the solid evidence for the reality of psi. The next generation dropped Rhine's data as being flawed and difficult to replicate and it hailed the Soal-Goldney experiments as the replicable and rock-solid basis for the existence of telepathy. Next came the Sheep-Goat experiments. Today, the Rhine data, the Sheep-Goats experiments, and the Soal-Goldney experiments no longer are used to argue the case for psi. Contemporary parapsychologists, instead, point to the ganzfeld experiments, the random-number gencrator experiments, and--with the declassifying of the SAIC experiments--the remote viewing experiments as their basis for insisting that psi exists. Professor Utts uses the ganzfeld data and the SAIC remote viewing results to assert that the existence of anomalous cognition has been proven. She does not completely discard earlier data. She cites meat-analyses of some of the earlier parapsychology experiments. Still, the cumulative database for anomalous mental phenomena does not exist. Most of the data accumulated by previous investigators has been discarded. In most cases the data have been American institutes for Research 3-46 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 a Chapter Three: Research Reviews discarded for good reasons. They were subsequently discovered to be seriously flawed in one or more ways that was not recognized by the original investigators. Yet, at the time they were part of the database, the parapsychologists were certain that they offered incontestable evidence for the reality of psi. . . How does this discussion relate to our present concerns with the scientific status of the SAIC program? This consideration of the shifting database of parapsychology offers a cautionary note to the use of contemporary research on the ganzfeld and remote viewing as solid evidence for anomalous mental phenomena. More than a century of parapsychological research teaches us that each generation of investigators was sure that it had found the "Holy Grail'--the indisputable evidence for psychic functioning. Each subsequent generation has abandoned their predecessors’ evidence as defective in one way or another. Instead, the new generation had its own version of the holy grail. Today, the parapsychologists offer us the ganzfeld experiments and, along with Jessica Utts, will presumably will include the SAIC remote viewing experiments as today's reasons for concluding that anomalous cognition has been demonstrated. Maybe this generation is correct. Maybe, this time the "indisputable" evidence will remain indisputable for subsequent generations. However, it is too soon to tell. Only history will reveal the answer. As E.G.Boring once wrote, when writing about the Soal-Goldney experiments, you cannot hurry history, Meanwhile, as I will point out later in this report, there are hints and suggestions that history may repeat itself Where Utts sees consistency and incontestable proof, I see inconsistency and hints that all is not as rock-solid as she implies. I can list other reasons to suggest that parapsychology's status as a science is shaky, at best. Some of these reasons will emerge as I discuss specific aspects of the SAIC results and their relation to other contemporary parapsychological research. THE CLAIM THAT ANOMALOUS COGNITION EXISTS American Institutes for Research 3-47 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews rl AR Professor Utts concludes "that psychic functioning has been well established." She bases this conclusion on three other claims: 1) the statistical results of the SAIC and other parapsychological experiments "are far beyond what is expected by chance", 2) "arguments that these results could be due to methodological flaws are soundly refuted"; and 3) "Effects of similar magnitude to those found in government-sponsored research at SRI and SAIC have been replicated at a number of laboratories across the world. Later, in this report, I will raise questions about her major conclusion and the three supporting claims. In this section, I want to unpack just what these claims entail. I will start with the statistical findings. Parapsychological is unique among the sciences in relying solely on significant departures from a chance baseline to establish the presence of its alleged phenomenon. In the other sciences the defining phenomena can be reliably observed and do not require indirect statistical measures to justify their existence. Indeed, each branch of science began with phenomena that could be observed directly. Gilbert began the study of magnetism by systematically studying a phenomenon that had been observed and was known to the ancients as well as his contemporaries. Modern physics began by becoming more systematic about moving objects and falling bodies. Psychology became a systematic science by looking for lawful relationships among sensory discriminations. Another starting point was the discovery of lawful relationships in the remembering and forgetting of verbal materials. Note that in none of these cases was the existence of the defining phenomena in question. No one required statistical tests and effect sizes to decide if magnetism was present or if a body had fallen. Psychophysicists did not need to reject a null hypothesis to decide if sensory processes were operating and memory researchers did not have to rely on reaching accepted levels of significance to know if recall or forgetting had occurred. Each of the major sciences began with phenomena whose presence was not in question. The existence of the primary phenomena was never in question. Each science began by finding systematic relationships among variations in the magnitudes of attributes of the central phenomena and the attributes of independent variables such as time, location, etc. The questions for the investigation of memory had to do with how best to describe the forgetting curve and what factors affected its parameters. No statistical tests or determination of effect sizes were required to decide if, in fact, forgetting was or was not present on any particular occasion. American Institutes for Research 3-4 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 a Chapter Three: Research Reviews Only parapsychology claims to be a science on the basis of phenomena (or a phenomenon) whose presence can be detected only by rejecting a null hypothesis. To be fair, parapsychologists also talk about doing process research where the emphasis is on finding systematic relationships between attributes of psi and variations in some independent variable. One conclusion from the SRI/SAIC project, for example, is that there is no relationship between the distance of the target from the viewer and the magnitude of the effect size for anomalous cognition. However, it is still the case that the effect size, and even the question of whether anomalous cognition was present in any experiment, is still a matter of deciding if a departure from a chance base line is non-accidental. At.this point I think it is worth emphasizing that the use of statistical inference to draw conclusions about the null hypothesis assumes that the underlying probability model adequately represents the distributions and variations in the real world situation. The underlying probability model is an idealization of the empirical situation for which it is being used. Whether or not the model is appropriate for any given application is an empirical matter and the adequacy of the model has to be justified for each new application. Empirical studies have shown that statistical models fit real world situations only approximately. The tails of real-world distributions, for example, almost always contain more cases than the standard statistics based on the normal curve assume. These departures from the idealized model do not have much practical import in many typical statistical applications because the statistical tests are robust. That is, the departures of the actual situation from the assumed probability model typically do not distort the outcome of the statistical test. However, when statistical tests are used in situations beyond their ordinary application, they can result in rejections of the null hypothesis for reasons other than a presumed departure from the expected chance value. Parapsychologists often complain that their results fail to replicate because of inadequate power. However, because the underlying probability models are only approximations, too much power can lead to rejections of the null hypothesis simply because the real world and the idealized statistical model are not exact matches, This discussion emphasizes that significant findings can arise for many reasons--including the simple fact that statistical inference is based on idealized models that mirror the real world only approximately. I agree with Jessica Utts that the effect sizes reported in the SAIC experiments and in the American Institutes for Research 3-49 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews recent ganzfeld studies probably cannot be dismissed as due to chance. Nor do they appear to be accounted for by multiple testing, file-drawer distortions, inappropriate statistical testing or other misuse of statistical inference. I do not rule out the possibility that some of this apparent departure from the null hypothesis might simply reflect the failure of the underlying model to be a truly adequate model of the experimental situation. However, I am willing to assume that the effect sizes represent true effects beyond inadequacies in the underlying model. Statistical effects, by themselves, do not justify claiming that anomalous cognition has been demonstrated--or, for that matter, that an anomaly of any kind has occurred, So, I accept Professor Utts' assertion that the statistical results of the SAIC and other parapsychological experiments "are far beyond what is expected by chance." Parapsychologists, of course, realize that the truth of this claim does not constitute proof of anomalous cognition. Numerous factors can produce significant statistical results. Operationally, the presence of anomalous cognition is detected by the elimination of all other possibilities. This reliance on a negative definition of its central phenomenon is another liability that parapsychology brings with its attempt to become a recognized science. Essentially, anomalous cognition is claimed to be present whenever statistically significant departures from the null hypothesis are observed under conditions that preclude the operation of all mundane causes of these departures. As Boring once observed, every success in parapsychological research is a failure. By this he meant that when the investigator or the critics succeed in finding a scientifically acceptable explanation for the significant effect the claim for ESP or anomalous cognition has failed. Having accepted the existence of non-chance effects, the focus now is upon whether these effects have normal causes. Since the beginning of Psychical research, each claim that psychic functioning had been demonstrated was countered by critics who suggested other reasons for the observed effects, Typical alternatives that have becn suggested to account for the effects have been fraud, statistical errors, and methodological artifacts. In the present discussion I am not considering fraud or statistical errors, This leaves only methodological oversight as the source for a plausible alternative to psychic functioning. Utts has concluded that “arguments that these results could be due to methodological flaws are soundly refuted.@!' If she is correct, then I would have to agree with her bottom line "that psychic functioning has been well established.?? American Institutes for Research 3-50 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 I Chapter Three: Research Reviews Obviously I do not agree that all possibilities for alternative explanations of the non-chance results have been eliminated. The SAIC experiments are well-designed and the investigators have taken pains to eliminate the known weaknesses in previous parapsychological research, In addition, I cannot provide suitable candidates for what flaws, if any, might be present. Just the same, it is impossible in principle to say that any particular experiment or experimental series is completely free from possible flaws. An experimenter cannot control for every possibility--especially for potential flaws that have not yet been discovered. At this point, a parapsychologist might protest that such "in principle?? arguments can always be raised against any findings, no matter how well conceived was the study from which they emerged. Such a response is understandable, but I believe my caution is reasonable in this particular case. Historically, many cases of evidence for psi were proffered on the grounds that they came from experiments of impeccable methodological design. Only subsequently, sometimes by fortunate accident, did the possibility of a serious flaw or alternative explanation of the results become available. The founders of the Society for Psychical Research believed that the Smith-Blackburn experiments afforded no alternative to the conclusion that telepathy was involved. They could conceive of no mundane explanation. Then Blackburn confessed and explained in detail just how he and Smith had tricked the investigators. The critics became suspicious of the Soal-Goldney findings not only because the results were too good, but also because Soal lost the original records under suspicious circumstances. Hansel, Scott, and Price each generated elaborate scenarios to explain how Soal might have cheated. Hansel and Scott reported finding peculiar patterns in the data. The scenarios, for accounting for these data, however, were extremely complicated and required the collusion of several individuals--some of whom were prominent statesmen and academics. The discovery of how Soal actually had cheated was made by the parapsychologist Betty Markwick. The finding came about through fortuitous circumstances. The method of cheating turned out to involve only one person and employed an ingenious, but simple, method that none of the critics had anticipated. During the first four years of the original ganzfeld-psi experiments, the investigators asserted that their findings demonstrated psi because the experimental design precluded any normal alternative. Only after I and a couple of parapsychologists independently pointed out how the American Institutes for Research 3-57 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews use of a single set of targets could provide a mundane alternative to psychic communication did the ganzfeld experimenters realize the existence of this flaw. After careful and lengthy scrutiny of the ganzfeld database, I was able to generate a lengthy list of potential flaws. Honorton and his colleagues devised the autoganzfeld experiments. These experiments were deliberately designed to preclude the flaws that I and others had eventually discovered in the original ganzfeld database. When the statistically significant results emerged from these latter experiments, they were proclaimed to be proof of anomalous communication because all alternative mundane explanations had been eliminated. When I was first confronted with these findings, I had to ad@t that the investigators had eliminated all but one of the flaws that I had listed for the original database. For some reason, Honorton and his colleagues did not seem to consider seriously the necessity of insuring that their randomization procedures were optimal. However, putting this one oversight aside, I] could find no obvious loopholes in the experiments as reported. When I was asked to comment on the paper that Daryl Bem and Charles Honorton wrote for the January 1994 issue of the Psychological Bulletin, I was able to get much of the raw data from Professor Bem. My analyses of that data revealed strong patterns that, to me, pointed to an artifact of some sort. One pattern, for example, was the finding that all the significant hitting above chance occurred only on the second or later occurrence of a target. All the first occurrences of a target yielded results consistent with chance. Although this was a post hoc finding, it was not the result of a fishing expedition. I deliberately looked for such a pattern as an indirect way of checking for the adequacy of the randomization procedures. The pattern was quite strong and persisted in every breakdown of the data that I tried--by separate investigator, by target type, by individual experiment, etc. The existence of this pattern by itself does not prove it is the result of an artifact. As expected, Professor Bem seized upon it as another peculiarity of psi. Subsequent to finding this pattern, I have learned about many other weaknesses in this experiment which could have compromised the results. Robert Morris and his colleagues at the University of Edinburgh took these flaws as well as some additional ones that they uncovered, into account when they designed the ganzfeld replication experiments. The point of this discussion is that it takes some time before we fully recognize the potential flaws in a newly designed experimental protocol. In some cases, the discovery of a serious American Institutes for Research : 52 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
eee SS py sy tS Se Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 i Chapter Three: Research Reviews flaw is the result of a fortuitous occurrence. In other cases, the uncovering of flaws came about only after the new protocol had been used for a while. Every new experimental design, as is the case for every new computer program, requires a shakedown period and debugging. The problems with any new method or design are not always apparent at first. Obvious flaws may be eliminated only to be replaced by more subtle ones. How does this apply to the SAIC experiments? These experiments were designed to eliminate the obvious flaws of the previous remote viewing experiments at SRI. Inspection of the protocol indicates that they succeeded in this respect. The new design and methodology, however, has not had a chance to be used in other laboratories or to be properly debugged. Many of the features that could be considered an asset also have possible down sides. I will return to this later in the report when I discuss the use of the same viewers and the same judge across the different experiments. For now, I just want to suggest some general grounds for caution in accepting the claim that all possible methodological flaws have been eliminated. The third warrant for Jessica Utts' conclusion that psi has been proven is that "Effects of similar magnitude to those found in government-sponsored research at SRI and SAIC have been replicated at a number of laboratories across the world." I will discuss this matter below. For now, I will point out that effects of similar magnitude can occur for several different reasons. Worse, the average effect size from different parapsychological research programs is typically a meaningless composite of arbitrary units. As such, these averages do not represent meaningful parameters in the real world. For example, Honorton claimed that the autoganzfeld experiments replicated the original ganzfeld experiments because the average effect size for both databases was approximately identical. This apparent similarity in average effect size is meaningless for many reasons. For one thing, the similarity in size depends upon which of many possible averages one considers. In the case under consideration the average effect size was obtained by adding up all the hits and trials for the 28 studies in the database. One experimenter contributed almost half to this total. Others contributed in greatly unequal numbers. The average will differ- if each experimenter's contribution is given equal weight. In addition, the heterogeneity of effect sizes among separate investigators is huge. All the effect sizes, for example, of one the investigators were negative. Another investigator American institutes for Research 3-53 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews contributed mostly moderately large effect sizes. If the first investigator had contributed more trials to the total, then the average would obviously have been lower. Similar problems exist for the average from the autoganzfeld experiments. In these latter experiments, the static targets--which most closely resembled the overwhelming majority of targets in the original database--yielded an effect size of zero. The dynamic targets yielded a highly significant and moderate effect size. Is the correct average effect size for these experiments based on a composite of the results of the static and dynamic targets or should it be based only the dynamic targets? THE SAIC PROGRAM As I have indicated, the SAIC experiments are an improvement on both the preceding SRI experiments as well as previous parapsychological investigations. The investigators seem to have taken pains to insure that randomization of targets for presentation and for judging was done properly. They have eliminated the major flaw in original SRI remote viewing experiments of non-independence in trials for a given viewer. Some of the other features can be considered as improvements but also as possible problems, in this category I would list the use of the same experienced viewers in many experiments and the use of the same target set across experiments. The major limitations that I see in these studies derive from their newness and their having been conducted in secrecy. The newness simply means that we have not had sufficient time to debug and to grasp fully both the strengths and weaknesses of this protocol. The secrecy aggravated this limitation by preventing other investigators from reviewing and criticizing the experiments from the beginning, and by making it impossible for independent laboratories to replicate the findings.' The fact that these experiments were conducted in the same laboratory, with the same basic protocol, using the same viewers across experiments, the same targets across experiments, and the same investigators aggravates, rather than alleviates, the problem of independent replication. If subtle, as-yet-undetected bias and flaws exist is the protocol, the very consistency of elements such as targets, viewers, investigators, and procedures across experiments enhances the possibility that these flaws will be compounded. Making matters even worse is the use of the same judge across all experiments. The judging of viewer responses is a critical factor in free-response remote viewing experiments. Ed May, American Institutes for Research 3-54 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews enn the principle investigator, as I understand it, has been the sole judge in all the free response experiments. May's rationale for this unusual procedure was that he is familiar with the response styles of the individual viewers. If a viewer, for example, talks about bridges, May- -from his Familiarity with this viewer--might realize that this viewer uses bridges to refer to any object that is on water. He could then interpret the response accordingly to make the appropriate match to a target. The SAIC experiments did benefit from the input of a distinguished oversight committee. But this still falls far short of what could have taken place in an open forum. Whatever merit this rationale has, it results in a methodological feature that violates some key principles of scientific credibility. One might argue that the judge, for example, should be blind not only about the correct target but also about who the viewer is. More important, the scientific community at large will be reluctant to accept evidence that depends upon the ability of one specific individual. In this regard, the reliance on the same judge for all free-response experiments is like the experimenter effect. To the extent that the results depend upon a particular investigator the question of scientific objectivity arises. Scientific proof depends upon the ability to generate evidence that, in principle, any serious and competent investigator--regardless of his or her personality--can observe. The use of the same judge across experiments further compounds the problem of non- independence of the experiments. Here, both Professor Utts and I agree. We believe it is important that the remote viewing results be obtainable with different judges. Again, the concern here is that the various factors that are similar across experiments, count against their separate findings as independent evidence for anomalous cognition. HAS ANOMALOUS COGNITION BEEN PROVEN? Obviously, I do not believe that the contemporary findings of parapsychology, including those from the SRI/SAIC program, justify concluding that anomalous mental phenomena have been proven. Professor Utts and some parapsychologists believe otherwise. I admit that the latest findings should make them optimistic. The case for psychic functioning seems better than it ever has been. The contemporary findings along with the output of the SRI/SAIC program do seem to indicate that something beyond odd statistical hiccups is taking place. I also have to admit that I do not have a ready explanation for these observed effects. Inexplicable statistical departures from chance, however, are a far cry from compelling evidence for American Institutes for Research 3-55 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 EEE eo Chapter Three: Research Reviews anomalous cognition. So what would be compelling evidence for the reality of anomalous cognition? Let's assume that the experimental results from the SAIC remote viewing experiments continue to hold up. Further assume that along with continued statistical significance no flaws or mundane alternative possibilities come to light. We would then want to ensure that similar results will occur with new viewers, new target pools, and several independent judges. Finally, to satisfy the normal standards of science, we would need to have the findings successfully replicated in independent laboratories by other parapsychologists as well as non-parapsychologists. If the parapsychologists could achieve this state of affairs, we are faced with a possible anomaly, but not necessarily anomalous cognition. As the parapsychologist John Palmer has recognized, parapsychologists will have to go beyond demonstrating the presence of a statistical anomaly before they can claim the presence of psychic functioning. This is because, among other things, the existence of a statistical anomaly is defined negatively. Something is occurring for which we have no obvious or ready explanation. This something may or may not turn out to be paranormal, According to Palmer, parapsychologists will have to devise a positive theory of the paranormal before they will be in a position to claim that the observed anomalies indicate paranormal functioning. Without such a positive theory, we have no way of specifying the boundary conditions for anomalous mental phenomena. Without such a theory we have no way of specifying when psi is present and when it is absent. Because psi or anomalous cognition is currently detected only by departures from a null hypothesis all kinds of problems beset the quest for the claim and pursuit of psychic functioning. For example, the decline effect, which was investigated in one of the SAIC experiments, was once used as an important sign for the presence of psi. J.B.Rhine discovered this effect not only in some of his data but in his re-analyses of data collected by earlier investigators. He attached great importance to his effect because it existed in data whose investigators neither knew of its existence nor had they been secking it. In addition, the decline effect helped Rhine to explain how seemingly null results really contained evidence for psi. This is because the decline effect often’ showed up as an excess of hitting in the early half of the experiment and as a deficit of hitting in the second half of the experiment. These two halves, when pooled together over the entire experiment, yielded an overall hit rate consistent with chance. merican institutes for Researc. 3-5 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 ° ee Chapter Threw: Research Reviews Although Rhine and other parapsychologists attached great importance to the decline effect as a reliable and often hidden sign of the presence of psychic functioning, the reliance on this indicator unwittingly emphasizes serious problems in the parapsychologist's quest. As the SAIC report on binary coding states, the decline effect is claimed for a bewildering variety of possibilities. Some investigators have found a decline effect going from the first quarter to the last quarter of each separate score sheet in their experiment, Other investigators have reported a decline effect as a decrease in hit rate from the first half to the second half of the total experiment. Still others find a decline effect across separate experiments. Indeed, almost any variation where the direction is from a higher hit rate to a lower hit rate has been offered as evidence for a decline effect. To confuse matters further, some investigators have claimed finding evidence for an incline effect. If the decline effect is a token for the presence of psi, what should one conclude when the data, as was the case in the SAIC experiment on binary coding, show a significant departure from the null hypothesis but no decline effect ? We know what the parapsychologist's conclude. As long as they get a significant effect, they do not interpret the absence of the decline effect as the absence of psychic functioning. This state of affairs holds as well for several other effects that have been put forth as tokens or signs of anomalous mental functioning. Several such signs are listed in the Handbook of parapsychology [!977, B.B. Wolman, Editor]. Typically, such signs are sought when the attempt to reject the ordinary null hypothesis fails. Displacement effects are frequently invoked. When his attempts to replicate Rhine's results failed, Soal was persuaded to re-analyze his data in terms of displacement effects. His retrospective analysis uncovered two subjects whose guesses significantly correlated with the target one or two places ahead of the intended target. In his subsequent experiments with these two subjects, one kept hitting on the symbol that came after the intended target while the other produced significant outcomes only when her guesses were matched against the symbol that occurred just before the intended target. Negative hitting, increased variability, and other types of departures from the underlying theoretical probability model have all been used as hidden signs of the presence of psychic functioning. What makes this search for hidden tokens of psi problematic is lack of constraints. Any time the original null hypothesis cannot be rejected, the eager investigator can search through the American Institutes for Research 3-57 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews data for one or more these markers. When one is found, the investigator has not hesitated in offering this as proof of the presence of psi. However, if the null hypothesis is rejected and none of these hidden signs of psi can be found in the data, the the investigator still claims the presence of psi. This creates the scientifically questionable situation where any significant departure from a probability model is used as proof of psi but the absence of these departures does not count as evidence against the presence of psi. So, acceptable evidence for the presence of anomalous cognition must be based on a positive theory that tells us when psi should and should not be present. Until we have such a theory, the claim that anomalous cognition has been demonstrated is empty. Without such a theory, we might just as well argue that what has been demonstrated is a set of effects--each one of which be the result of an entirely different cause. Professor Utts implicitly acknowledges some of the preceding argument by using consistency of findings with other laboratories as evidence that anomalous cognition has been demonstrated. I have already discussed why the apparent consistency in average effect size across experiments cannot be used as an argument for consistency of phenomena across these experiments. To be fair, parapsychologists who argue consistency of phenomena across experiments often go beyond simply pointing to consistency in effect sizes. One example is the claim that certain personality correlates replicate across experiments. May and his colleagues correctly point out, however, that these correlations tend to be low and inconsistent. Recently, parapsychologists have claimed that extroversion correlates positively with successful performance on anomalous cognition tasks. This was especially claimed to be true of the ganzfeld experiments. However, the apparently successful replication of the autoganzfeld experiments by the Edinburgh group (under subcontract to the SAIC program) found that the introverts, if anything, scored higher than the extroverts. The autoganzfeld experiments produced significant effects only for the dynamic targets. The static targets produced zero effect size. Yet the bulk of the targets in the original ganzfeld database were static and they produced an effect size that was significantly greater than the zero effect size of the autoganzfeld experiments (I was able to demonstrate that there was adequate power to detect an effect size of the appropriate magnitude for the static targets in the autoganzfeld experiments). Further indication of inconsistency is the SAIC experiment American Institutes for Research 3-58 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 ‘ Chapter Three: Research Reviews nn which found that the only the static targets produced a significant effect size, whereas the dynamic targets yielded a zero effect size. May and his colleagues speculated that the failure of the dynamic targets was due to a ‘bandwidth’ that was too wide. When they apparently narrowed the bandwidth of the dynamic targets in a second experiment, both dynamic and static targets did equally well. It is unclear whether this should be taken as evidence for consistency or inconsistency. Note that the hypothesis and claim for the autoganzfeld experiments is that dynamic targets should be significantly better than static ones. As far as I can tell the original dynamic targets of the ganzfeld experiments are consistent with an unlimited bandwidth. Other important inconsistencies exist among the contemporary databases. The raison d'etre for the ganzfeld experiments is the belief among some parapsychologists that an altered state facilitates picking up the psi signal because it lowers the noise-to-signal ratio from external sensory input. The touchstone of this protocol is the creation of an altered state in the receiver. This contrasts sharply with the remote viewing experiments in which the viewer is always in a normal state. More important is that the ganzfeld researchers believe that they get best results when each subject serves as his/her own judge. Those experiments in the ganzfeld database that employed both external judges and subjects as their own judges found that their results were more successful using subjects as their own judges. The reverse is true in the remote viewing experiments. The remote viewer experimenters believe that external judges provide much better hit rates than viewer-judges. This difference is even more extreme in the SAIC remote viewing where a single judge was used for all experiments. This judge, who was also the principal investigator, believed that he could achieve best results if he did the judging because of his familiarity with the response styles of the individual viewers. So even if the ganzfeld and the SAIC remote viewing experiments have achieved significant effects and average effect sizes of approximately the same magnitude, there is no compelling reason to assume they are dealing with the same phenomena or phenomenon. To make such a claim entails showing that the alleged effect shows the same pattern of relationships in each protocol. Almost certainly, a positive theory of anomalous mental phenomena that predicts lawful relationships of a recognizable type will be necessary before a serious claim can be made that the same phenomenon is present across different research laboratories and experiments. Such a positive theory will be necessary also to tell us when we are and when American Institutes for Researc 3-5 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews we are not in the presence of this alleged anomalous cognition. WHAT NEEDS TO BE EXPLAINED? Professor Utts and many parapsychologists argue that they have produced evidence of an anomaly that requires explanation. They assert that the statistical effects they have documented cannot be accounted for in terms of normal scientific principles or methodological artifact. After reviewing the results from the SAIC experiments in the context of other contemporary parapsychological research, Utts is confident that more than an anomaly has been demonstrated. She believes the evidence suffices to conclude that the anomaly establishes the existence of psychic functioning. This evidence for anomalous cognition, according to Utts and the parapsychologists, mects the standards employed by the other sciences. By this, ] think Professor Utts means that in many areas of scientific inquiry the decision that a real effect has occurred is based on rules of statistical inference. Only if the null hypothesis of no difference between two or more treatments is rejected can the investigator claim that the differences are real in the sense that they are greater than might be expected on the basis of some baseline variability. According to this standard, it seems that the SAIC experiments as well as the recent ganzfeld experiments have yielded effects that cannot be dismissed as the result of normal variability. While the rejection of the null hypothesis is typically a necessary step for claiming that an hypothesized effect or relationship has occurred, it is never sufficient. Indeed, because the underlying probability model is only an approximation, everyone realizes that the null hypothesis is rarely, if ever, strictly true. In practice, the investigator hopes that the statistical test is sufficiently robust that it will reject the null hypothesis only for meaningful departures from the null hypothesis. With sufficient power, the null hypothesis will almost certainly be rejected in most realistic situations. This is because effect sizes will rarely be exactly zero. Even if the true effect size is zero in a particular instance, sufficient power can result in the rejection of the null hypothesis because the assumed statistical model will depart from the real-world situation in other ways. For most applications of statistical inference, then, too much power can result in mistaken inferences as well as foo little power. American Institutes for Research 3-60 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews Here we encounter another way in which parapsychological inquiry differs from typical scientific inquiry. In those sciences that rely on statistical inference, they do so as an aid to weeding out effects that could be the result of chance variability. When effect sizes are very small or if the experimenter needs to use many more cases than is typical for the field to obtain significance, the conclusions are often suspect. This is because we know that with enough cases an investigator will get a significant result, regardless of whether it is meaningful or not. Parapsychologists are unique in postulating a null hypothesis that entails a true effect size of zero if psi is not operating. Any significant outcome, then, becomes evidence for psi. My concern here is that small effects and other departures from the statistical model can be expected to occur in the absence of psi. The statistical model is only an approximation. When power is sufficient and when the statistical test is pushed too far, rejections of the null hypothesis are bound to occur. This is another important reason why claiming the existence of an anomaly based solely on evidence from statistical inference is problematic. This is one concern about claiming the existence of an anomaly on the basis of statistical evidence. In the context of this report, I see it as a minor concern. As I have indicated, I am willing to grant Professor Utts' claim that the rejection of the null hypothesis is probably warranted in connection with the SAIC and the ganzfeld databases, I have other concerns. Both have to do with the fact that no other science, so far as I know, would draw conclusions about the existence of phenomena solely on the basis of statistical findings. Although it is consistent with scientific practice to use statistical inference to reject the null hypothesis, it is not consistent with such practice to postulate the existence of phenomena on this basis alone. Much more is required. I will discuss at least two additional requirements. Thomas Kuhn's classic characterization of normal and revolutionary science has served as the catalyst for many discussions about the nature of scientific inquiry. He popularized the idea that normal scientific inquiry is guided by what he called a paradigm. Later, in the face of criticisms, he admitted that he had used the term paradigm to cover several distinct and sometimes contradictory features of the scientific process. One of his key uses of the term paradigm was to refer to the store of exemplars or textbook cases of standard experiments that every field of scientific inquiry possesses. These exemplars are what enable members of a scientific community to quickly learn and share common principles, procedures, methods, and standards. These exemplars are also the basis for initiating new members into the American Institutes for Research - Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews rt community. New research is conducted by adapting one or more of the patterns in existing exemplars as guidelines about what constitutes acceptable research in the field under consideration. Every field of inquiry, including parapsychology, has its stock of exemplars. In parapsychology these would include the classic card guessing experiments of J.B. Rhine, the Sheep-Goat experiments, etc. What is critical here is the striking difference between the role of exemplars in parapsychology as contrasted with their role in all other fields of scientific inquiry. These exemplars not only serve as models of proper procedure, but they also are teaching tools. Students in a particular field of inquiry can be assigned the task of replicating some of these classic experiments. The instructor can make this assignment with the confident expectation that each student will obtain results consistent with the original findings. The physics instructor, for example, can ask novice students to try Newton's experiments with colors or Gilbert's experiments with magnets. The students who do so will get the expected results. The psychology instructor can ask novice students to repeat Ebbinghaus! experiments on forgetting or Peterson and Peterson's classic experiment on short- term memory and know that they will observe the same relationships as reported by the original experimenters. Parapsychology is the only field of scientific inquiry that does not have even one exemplar that can be assigned to students with the expectation that they will observe the original results! In every domain of scientific inquiry, with the exception of parapsychology, many core exemplars or paradigms exist that will reliably produce the expected, lawful relationships. This is another way of saying that the other domains of inquiry are based upon robust, lawful phenomena whose conditions of occurrence can be specified in such a way that even novices will be able to observe and/or produce them. Parapsychologists do not possess even one exemplar for which they can confidently specify conditions that will enable anyone —let alone a novice—to reliably witness the phenomenon. The situation is worse than I have so far described. The phenomena that can be observed with the standard exemplars do not require sensitive statistical rejections of the null hypothesis based on many trials to announce their presence. The exemplar in which the student uses a prism to break white light into its component colors requires no statistics or complicated inference at all. The forgetting curve in the Ebbinghaus experiment, requires American institutes for Research 3-62 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews i nothing more than plotting proportion recalled against trial number. Yet, to the extent that parapsychology is approaching the day when it will possess at least one exemplar of this sort, the "observation?F of the “phenomenon’ will presumably depend upon the indirect use of statistical inference to document its presence. In the standard domains of science, this problem of having not a single exemplar for reliably observing its alleged phenomenon, would be taken as a sign that the domain has no central phenomena, When Soviet scientists announced the discovery of mitogenetic radiation, some western scientists attempted to replicate the findings. Some reported success; others reported mixed results; and many failed entirely to observe the effect. Eventually scientists, including the Soviets, abandoned the quest for mitogenetic radiation. Because no one, including the original discover, could specify conditions under which the phenomenon--if there be one-- could be observed, the scientific community decided that there was nothing to explain other than as-yet-undetected artifacts. The same story can be told about N-Rays, Polywater, and other candidate phenomena that could not be reliably observed or produced. We cannot explain something for which we do not have at least some conditions under which we can confidently say it occurs. Even this is not enough. The alleged phenomenon not only must reliably occur at least under some conditions but it also must reliably vary in magnitude or other attributes as a function of other variables, Without this minimal amount of lawfulness, the idea that there is something to explain is senseless. Yet, at best, parapsychology's current claim to having demonstrated a form of anomalous cognition rests on the possibility that it can generate significant differences from the null hypothesis under conditions that are still not reliably specified. I will suggest one more reason for my belief that it is premature to try to account for what the SAIC and the ganzfeld experiments have so far put before us. On the basis of these experiments, contemporary parapsychologists claim that they have demonstrated the existence of an "anomaly". I will grant them that they have apparently demonstrated that the SAIC and the ganzfeld experiments have generated significant effect sizes beyond what we should expect from chance variations. I will further admit that, at this writing, I cannot suggest obvious methodological flaws to account for these significant effects. As I have previously mentioned, this admission does not mean that these experiments are free from subtle biases and potential bugs. The experimental paradigms are too recent and insufficiently evaluated to know for sure. I can point to departures from optimality that might harbor potential flaws-- mierican Institutes for Heseare. 3-63 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 a Chapter Three: Research Reviews such as the use of a single judge across the remote viewing experiments, the active coaching of viewers by the experimenter during judging procedures in the ganzfeld, my discovery of peculiar patterns of scoring in the ganzfeld experiments, etc. Having granted that significant effects do occur in these experiments, I hasten to add that without further evidence, I do not think we can conclude that these effects are all due to the same cause—let alone that they result from a single phenomenon that is paranormal in origin. The additional reason for concern is the difference in the use of ‘anomaly’ ;n this context and how the term ‘anomaly’ is used in other sciences. In the present context, the parapsychologists are using the term ‘anomaly’ to refer to apparently inexplicable departures from the null hypothesis. These departures are considered inexplicable in the sense that apparently all normal reasons for such departures from the null hypothesis have been excluded. But these departures are not lawful in the sense that the effect sizes are consistent. The effect sizes differ among viewers and subjects; they also differ for different experimenters; they come and go in inexplicable ways within the same subject. Possibly some of these variations in effect size will be found to exhibit some lawfulness in the sense that they will correlate with other variables. The SAIC investigators, for example, hope they have found such correlates in the entropy and bandwidth of targets. At the moment this is just a hope. The term "anomaly" is used in a much more restricted sense in the other sciences. Typically an anomaly refers to a lawful and precise departure from a theoretical baseline. As such it is something the requires explaining. Astronomers were faced with a possible anomaly when discrepancies from Newtonian theory were reported in the orbit of Uranus. In the middle 1800s, Urban Leverrier decided to investigate this problem. He reviewed all the data on previous sightings of Uranus--both before and after it had been discovered as new planet. On the basis of the previous sightings, he laboriously recalculated the orbital path based on Newtonian theory and the reported coordinates. Sure enough, he found errors in the original calculations. When he corrected for these crrors, the apparent discrepancy in Uranus' orbit was much reduced. But the newly revised orbit was still discrepant from where it should be on Newtonian theory. With this careful work, Leverrier had transformed a potential anomaly into an actual anomaly. Anomaly in this sense meant a precise and lawful departure from a well-defined theory. It was only after the precise nature, direction, and magnitude of this discrepancy was carefully specified did Leverrier and the scientific community decide that American Institutes for Research 3-6: Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
ese Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews here was an anomaly that required explanation. What had to explain was quite precise. What was needed was an explanation that exactly accounted for this specific departure from the currently accepted theory. Leverrier's solution was to postulate a new planet beyond the orbit of Uranus. This was no easy task because it involved the relatively unconstrained and difficult problem of inverse perturbations. Leverrier had to decide on a size, orbit, location, and other attributes of a hitherto unknown body whose characteristics would be just those to produce the observed effects on Uranus without affecting the known orbit of Saturn. Leverrier's calculations resulted in his predicting the location of this hitherto unknown planet and the astronomer Galle located this new planet, Neptune, close to where Leverrier had said it would be. The point of this story is to emphasize the distinction between the parapsychologists’ use of anomaly from that of other scientists’. Anomalies in most domains of scientific inquiry are carefully specified deviations from a formal theory. What needs to be explained or accounted for is precisely described. The anomalies that parapsychologists are currently talking about differ from this standard meaning in that the departures are from the general statistical model and are far from having the status of carefully specified and precise deviations from a theoretical baseline, In this latter case we do not know what it is that we are being asked to explain. Under what conditions can we reliably observe it? What theoretical baselines are the results a departure from? How much and in what direction and form do the departures exist? What specifically must our explanation account for? Finally, I should add that some parapsychologists, at least in the recent past, have agreed with my position that parapsychological results are not: yet ready to be placed before the scientific community. Parapsychologists such as Beloff, Martin Johnson, Gardner Murphy, J.G. Pratt and others have complained that parapsychological data are volatile and messy. Some of these investigators have urged their colleagues to first get their house in order before they ask the scientific community at large to take them seriously. Martin Johnson, especially, has urged his colleagues to refrain from asking the scientific community to accept their findings until they can tame them and produce lawful results under specified conditions. Clearly, parapsychology has still not reached this desired state. At best, the results of the SAIC experiments combined with other contemporary findings offer hope that the parapsychologists may be getting closer to the day when they can put something before the scientific American institutes for Research 3-65 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews community and challenge it to provide an explanation. POTENTIALS FOR OPERATIONAL APPLICATIONS It may seem obvious that the utility of remote viewing for intelligence gathering should depend upon its scientific validity. If the scientific research cannot confirm the existence of a remote viewing ability, then it would seem to be pointless to try an use this non-existent ability for any practical application. However, the matter is not this simple. If the scientific research confirms the existence of anomalous cognition, this does not guarantee that this ability would have useful applications. Ed May, in his presentation to the evaluation panel, gave several reasons why remote viewing could be real and, yet, not helpful for intelligence gathering. In his opinion, approximately 20 percent of the information supplied by a viewer is accurate. Unfortunately, at the time the remote viewer is generating the information, we have no way of deciding which portion is likely to be the accurate one. Another problem is that the viewer's information could be accurate, yet not relevant for the intelligence analyst's purposes. This question is related to the problem of boundary conditions which I discussed earlier in this report. From both a scientific and an operational viewpoint the claim that anomalous cognition exists is not very credible until we have ways to specify when and when it is not present. So far, parapsychology seems to have concentrated only in finding ways to document the existence of anomalous cognition. The result is a patchwork quilt of markers that, when present, are offered as evidence for the presence of psi. These markers or indicators include the decline cffect, negative hitting as well as positive hitting, displacement hitting, the incline effect, increased variability, decreased variability and just about any other way a discrepancy from a probability model can occur. A cynic will note that the absence of any or most of these markers is not used as evidence for the absence of psi. This lack of way to distinguish between the presence and absence of anomalous cognition creates many challenges for parapsychology, some of which I have already discussed. So, even if remote viewing is a real ability possessed by some individuals, its usefulness for intelligence gathering is questionable. If May is correct, then 80% of the all the information supplied by this talented viewer will be crroncous, Without any way to tell which statements American Institutes for Research - Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
LT A Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews a of the views are reliable and which are not, the use of this information may make matters worse rather than better. Can remote viewing have utility for information gathering even if it cannot be scientifically validated? I can imagine some possibilities for remote viewing to be an asset to the intelligence analyst even when the viewer possesses no valid paranormal powers. The viewer might be a person of uncommonly good sense or have a background that enables him or her to provide helpful information even ‘f it does not come from a paranormal source. Another possibility is that the viewer, even though lacking in any truly accurate intelligence information, might say things or open up new ways of dealing with the analyst's problem. In this latter scenario the remote viewer is a catalyst that may open up new ways of looking at an intelligence situation much like programs for problem solving and creative thinking stimulate new ways of looking at a situation. However, if the usefulness of the remote viewer reduces to a matter of injecting common sense or new perspectives into the situation, I believe that we can accomplish the same purpose in more efficient ways. In considering potential utility, 1 am most concerned about separation of the operational program in remote viewing from the research and development phase. By default, the assessment of the usefulness of the remote viewing in the operational arena is decided entirely by subjective validation or what May and Utts call prima facie evidence. Granted it is difficult to assess adequately the effectiveness of remote viewing in the operational domain. Nevertheless, better ways can be devised than have apparently been used up to now. In our current attempt to get an initial idea about the effectiveness of the current operational use of remote viewing, we have simply been asking individuals and agencies who have used the services of the remote viewers, if the information they received was accurate and useful. Whatever information we get from this survey is extremely limited for the purposes of judging the utility of remote viewing in the operational domain. Even psychologists who should know better underrate the power of subjective validation. Anyone who relies on prima facie evidence as a basis for affirming the validity of remote viewing should carefully read that portion of Marks and Kamman's The psychology of the psychic [1981] in which they discuss the SRI and their own experiments on remote viewing. In the early stages of their attempt to replicate the SRI remote viewing experiments, they were astonished at the high quality of their subject's protocols and the apparent accuracy of merican Institutes for Researc. 3-67 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews eereerwe rte A the viewing. After each session, the experimenters and the subject (viewer) would visit the target site and compare the verbal protocol with the actual site. The specific details of the viewers' responses appeared to match specific objects in the target site with uncanny accuracy. When they gave the verbal protocols to the judge, a distinguished professor, to blindly match against the actual target sites, he was astonished at how well what he considered the closest matching protocol for each site matched actual details of the target. He had no doubt that the viewers had demonstrated strong remote viewing abilities. So, both the viewers and the judge quickly became convinced of the reality of remote viewing on the basis of the uncanny matches between the verbal descriptions and the actual target sites. The experimenters received a rude awakening when they discovered that, despite the striking matches observed between target and verbal description, the judge had matched the verbal protocols to the wrong target sites. When all parties were given the results the subjects could not understand how the judge could have matched any but the actual target site to their descriptions. For them the match was so obvious that it would be impossible for the judge to have missed it. The judge, on the other hand, could not accept that any but the matches he made could be paired with the actual target sites. This phenomenon of subjective validation is pervasive, compelling and powerful. Psychologists have demonstrated it in a variety of settings. I have demonstrated it and written about in the context of the psychic reading. In the present context, subjective validation comes about when a person evaluates the similarity between a relatively rich verbal description and an actual target or situation. Inevitably, many matches will be found. Once the verbal description has been judged to be a good match to a given target, the description gets locked in and it becomes virtually impossible for the judge to see the description as fitting any but the original target. Unfortunately, all the so-called prima facie evidence put before us is tainted by subjective validation. We are told that the many details supplied by the viewers were indeed inaccurate. But some details were uncannily correct and even, in one case, hidden code words were correctly revealed. Such accounts do indeed seem compelling. They have to be put in the context, however, of all such operational attempts. We have to know the general background and expectations of the viewers, the questioners, etc. Obviously, the targets selected for the American Institutes for Research 3-68 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 — Chapter Three: Research Reviews viewers in the operational setting will have military and intelligence relevance. If the viewer [some of the viewers have intelligence backgrounds] suspects the general nature of the target, then previous background knowledge might very well make the presence, say of a gantry, highly likely. In addition, the interactions and questioning of the viewers in these settings appear to be highly suggestive and leading. I can imagine that the preceding paragraph might strike a reader as being unreasonable. Even allowing for subjective validation, the possibility that a viewer might accurately come up with secret code words and a detailed description of particular gantry is quite remote on the basis of common sense and sophisticated guessing. I understand the complaint and I realize the reluctance to dismiss such evidence out of hand. However, I have had experience with similarly compelling prima facie evidence for more than a chance match between a description and a target. In the cases I have in mind, however, the double blind controls were used to pair descriptions with the true as well as with the wrong target sites. In all these test cases with which I am familiar, the unwitting subjects found the matches between their descriptions and the presumed target equally compelling regardless of whether the presumed target was the actual or the wrong one. What this says about operational effectiveness, is that, for evaluation purposes, half of the time the viewers and the judges should be mislead about the what was the actual target. In these cases, both the interrogator and the viewer, as well as the judge, have to be blind to the actual targets. Under such conditions, if the judges and the others find the matches between the verbal descriptions and the actual targets consistently better than the matches between the verbal descriptions and the decoy targets, then this would constitute some evidence for the effectiveness of remote viewing. I can confidently predict, regardless of the outcome of such an evaluation, that many of the verbal descriptions when matched with decoy targets will be judged to be uncanny matches. SUGGESTIONS: WHAT NEXT? I have played the devil's advocate in this report, I have argued that the case for the existence of anomalous cognition is still shaky, at best. On the other hand, I want to state that I believe that the SAIC experiments as well as the contemporary ganzfeld experiments display methodological and statistical sophistication well above previous parapsychological research. American institutes for Research 3-69 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews Despite better controls and careful use of statistical inference, the investigators seem to be getting significant results that do not appear to derive from the more obvious flaws of previous research. I have argued that this does not justify concluding that anomalous cognition has been demonstrated. However, it does suggest that it might be worthwhile to allocate some resources toward seeing whether these findings can be independently replicated. If so, then it will be time to reassess if it is worth pursuing the task of determining if these effects do indeed reflect the operation of anomalous cognition. This latter quest will involve finding lawful relationships between attributes of this hypothesized phenomenon and different independent variables. Both the scientific and operational value of such an alleged phenomenon will depend upon how well the conditions for its occurrence can be specified and how well its functioning can be brought under control. Both Professor Utts and I agree that the very first consideration is to see if the SAIC remote viewing results will still be significant when independent judges are used. I understand Ed May's desire to use a judge who is very familiar with the response styles of the experienced viewers. However, if remote viewing is real, then conscientious judges, who are blind to the actual targets, should still be able to match the verbal descriptions to the actual targets better than chance. If this cannot be done, the viability of the case for remote viewing becomes problematical. On the other hand, assuming that independent judges can match the descriptions to the correct targets reasonably well, then it becomes worthwhile to try to independently replicate the SAIC experiments. At this point we face some interesting questions. Should we try to replicate the remote viewing studies by using the same viewers, the same targets, and the same protocol? Perhaps change only the experimenters, the judge, and the laboratory? At some point we would also want to change the targets. For completeness, we would also want to search for new, talented viewers. If independent replications confirm the SAIC findings, we still have a long way to go. However, at this stage in the proceedings, the scientific community at large might be willing to acknowledge that an anomaly of some sort has been demonstrated. Before the scientific community will go beyond this acknowledgment, the parapsychologists will have to devise a positive theory of anomalous communication from which they can make testable predictions about relationships between anomalous communication and other variables. American Institutes for Research 3-70 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews CONCLUSIONS THE SCIENTIFIC STATUS OF THE SAIC RESEARCH PROGRAM 1. The SAIC experiments on anomalous mental phenomena are statistically and methodologically superior to the earlier SRI remote viewing research as well as to previous parapsychological studies. In particular, the experiments avoided the major flaw of non-independent trials for a given viewer. The investigators also made sure to avoid the problems of multiple stastical testing that was characteristic of much previous parapsychological research. From a scientific viewpoint, the SAIC program was hampered by its secrecy and the multiple demands placed upon it. The secrecy kept the program from benefiting from the checks and balances that comes from doing research in a public forum. Scrutiny by peers and replication in other laboratories would accelerated the scientific contributions from the program. The multiple demands placed on the program meant that too many things were being investigated with too few resources. As a result, no particular finding was followed up in sufficient detail to pin it down scientifically. Ten experiments, no matter how well conducted, are insufficient to fully resolve one important question, let alone the several that were posed to the SAIC investigators. Although, I cannot point to any obvious flaws in the experiments, the experimental program is too recent and insufficiently evaluated to be sure that flaws and biases have been eliminated. Historically, each new paradigm in parapsychology has appeared to its designers and contemporary critics as relatively flawless. Only subsequently did previously unrecognized drawbacks come to light. Just as new computer programs require a shakedown period before hidden bugs come to light, each new scientific program requires scrutiny over time in the public arena before its defects emerge. Some possible sources of problems for the SAIC program are its reliance on experienced viewers, and the use of the same judge--one who is familtar to the viewers, for all the remote viewing. The statistical departures from chance appear to be too large and consistent to attribute to statistical flukes of any sort. Although I cannot dismiss the possibility that these American institutes for Research 3-71 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 a Chapter Three: Research Reviews rejections of the null hypothesis might reflect limitations in the statistical model as an approximation of the experimental situation, I tend to agree with Professor Utts that real effects are occurring in these experiments. Something other than chance departures from the null hypothesis has occurred in these experiments. 5. However, the occurrence of statistical effects does not warrant the conclusion that psychic functioning has been demonstrated. Significant departures from the null hypothesis can occur for several reasons. Without a positive theory of anomalous cognition, we cannot say that these effects are due to a single cause, let alone claim they reflect anomalous cognition. We do not yet know how replicable these results will be, especially in terms of showing consistent relations to other variables. The investigators report findings that they believe show that the degree of anomalous cognition varies with target entropy and the ‘bandwidth’ of the target set. These findings are preliminary and only suggestive at this time, Parapsychologists, in the past, have reported finding other correlates of psychic functioning such as extroversion, sheep/goats, altered states only to find that later studies could not replicate them. 6. Professor Utts and the investigators point to what they see as consistencies between the outcome of contemporary ganzfeld experiments and the SAIC results. The major consistency is similarity of average effect sizes across experiments. Such consistency is problematical because these average effect sizes, in each case, are the result of arbitrary combinations from different investigators and conditions. None of these averages can be justified as estimating a meaningful parameter. Effect size, by itself, says nothing about its origin. Where parapsychologists see consistency, I see inconsistency. The ganzfeld studies are premised on the idea that viewers must be in altered state for successful results. The remote viewing studies use viewers in a normal state. The ganzfeld experimenters believe that the viewers should judge the match between their ideation and the target for best results; the remote viewers believe that independent judges provide better evidence for psi than viewers judging their own responses. The recent autoganzfeld studics found successful hitting only with dynamic targets and only chance results with static targets. The SAIC investigators, in one study, found hitting with static targets and not with dynamic ones. In a subsequent study they found hitting for both types of targets. They suggest that they may have Ameriéan Institutes for Research a-72 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews solution to this apparent inconsistency in terms of their concept of bandwidth. At this time, this is only suggestive. 7. The challenge to parapsychology, if it hopes to convincingly claim the discover, of anomalous cognition, is to go beyond the demonstration of significant effects. The parapsychologists need to achieve the ability to specify conditions under which one can reliably witness their alleged phenomenon. They have to show that they can generate lawful relationships between attributes of this alleged phenomenon and independent variables. They have to be able to specify boundary conditions that will enable us to detect when anomalous cognition is and is nof present. SUGGESTIONS FOR FUTURE RESEARCH 1. Both Professor Utts and I agree that the first step should be to have the SAIC protocols rejudged by independent judges who are blind to the actual target. 2. Assuming that such independent judging confirms the extra-chance matchings, the findings should be replicated in independent laboratories. Replication could take several forms. Some of the original viewers from the SAIC experiments could be used. However, it seems desirable to use a new target set and several independent judges. OPERATIONAL IMPLICATIONS 1, - Tthe current default assessment of the operational effectiveness of remote viewing is fraught with hazards. Subjective validation is well known to generate compelling, but false, convictions that a description matches a target in striking ways. Better, double blind, ways of assessing operational effectiveness can be used. I suggest at least one way in the report. 2. The ultimate assessment of the potential utility of remote viewing for intelligence gathering cannot be separated from the findings of laboratory research. American institutes for Research 3-73 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 «Chapter Three: Research Reviews THE REPLY Response to Ray Hyman’s Report of September 11, 1995 "Evaluation of Program in Anomalous Mental Phenomena" Jessica Utts Division of Statistics University of California, Davis Ray Hyman's report of September 11, 1995, written partially in response to my written report of September 1, 1995 elucidates the issues on which he and I agree and disagree. I basically concur with his assessment, but there are three issues he raises with regard to the scientific status of parapsychology to which I would like to respond. 1, "Only parapsychology, among the fields of inquiry claiming scientific status, lacks a cumulative database (p.6).” It is simply not true that parapsychology lacks a cumulative database. In fact, the accumulated database is truly impressive for a science that has had so few resources. While critics are fond of relating, as Professor Hyman does in his report, that there has been "more than a century of parapsychological research (p. 7)" psychologist Sybo Schouten (1993, p.316) has noted that the total human and financial resources devoted to parapsychology since 1882 is at best equivalent to the expenditures devoted to fewer than two months of research in conventional psychology in the United States. On pages 4 and 5 of their September 29, 1994 SAIC final report, May, Luke and James summarize four reports that do precisely what Professor Hyman claims is not don in parapsychology; they put forth the accumulated evidence for anomalous cognition in a variety of formats. Rather than dismissing the former experiments, parapsychologists build on them. As in any area of science, it is of course the most recent experiments that receive the most attention, but that does not mean that the field would divorce itself from past work. Quite to the contrary, past experimental results and methodological weaknesses are used to design better and more efficient experiments. merican institutes for Researc 3-7. Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews cee As an exatnple of the normal progress of inquiry expected in any area of science, the autoganzfeld experiments currently conducted by parapyscholgists did not simply spring out of thin air. The original ganzfeld experiments followed from Honorton's observation at Maimondies Medical Center, that anomalous cognition seemed to work well in dreams. He investigated ways in which a similar state could be achieved in normal waking hours, and found the ganzfeld regime in another area of psychology. The automated ganzfeld followed from a critical evaluation of the earlier ganzfeld experiments, and a set of conditions agreed upon by Honorton and Professor Hyman. The current use of dynamic targets in autoganzfeld experiments follows from the observation that they were more successful than static targets in the initial experiments. The investigation of entropy at SAIC follows from this observation as well. This is just one example of how current experiments are built from past results, 2. "Only parapsychology claims to be a science on the basis of phenomena (or a phenomenon) whose presence can be detected only by rejecting a null hypothesis (p.8)." While it is true that parapsychology has not figured out all the answers, it does not differ from normal science in this regard. It is the norm of scientific progress to make observations first, and then to attempt to explain them. Before quantum mechanics was developed there were a number of anomalies observed in physics that could not be explained. There are many observations in physics and in the social and medical sciences that can be observed, either statistically or deterministically, but which cannot be explained. As a more recent example, consider the impact of electromagnetic fields on health. An article in Science (Vol. 269, 18 August, p. 911) reported that "After spending nearly a decade reviewing the literature on electromagnetic fields (EMFs), a panel of the National Council on Radiation Protection and Measurements (NCRP) has produced a draft report concluding that some health effects linked to EMFs--such as cancer and immune deficiencies--appear real and warrant steps to reduce EMF exposure... Biologists have failed to pinpoint a convincing mechanism of action." In other words, a statistical effect has been convincingly established and it is not the responsibility of science to attempt to establish its mechanism, just as in parapsychology. American Institutes for Research 3-75 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews en As yet another example, consider learning and memory, which have long been studied in psychology. We know they exist, but brain researchers are just beginning to understand how they work by using sophisticated brain imaging techniques. Psychologists do not understand these simple human capabilities, and they certainly do not understand other observable human phenomena such as what causes people to fall in love. Yet, no one would deny the existence of these phenomena just because we do not understand them. In any area involving the natural inherent in humans, science progresses by first observing a statistical difference and then attempting to explain it. At this stage, I believe parapsychology has convincingly demonstrated that an effect is present, and future research attempts should be directed at finding an explanation. In this regard parapsychology in on par with scientific questions like the impact of electromagnetic fields on health, or the cross-cultural differences in memory that have been observed by psychologists. 3. “Parapsychology is the only field of scientific inquiry that does not have even one exemplar that can be assigned to students with the expectation that they will observe the original results (p. 18)." I disagree with this statement for two reasons. - First, I can name other phenomena for which students could not be expected to do a simple experiment and observe a result, such as the connection between taking aspirin and preventing heart attacks or the connection between smoking and getting lung cancer. What differentiates these phenomena from simple experiments like splitting light with a prism is that the effects are statistical in nature and are not expected to occur every single time. Not everyone who smokes gets lung cancer, but we can ‘predict the proportion who will. Not everyone who attempts anomalous cognition will be successful, but I think we can predict the proportion of time success should be achieved. Since I believe the probability of success has been established in the autoganzfeld experiments, I would offer them as the exemplar Professor Hyman requests. The problem is that to be relatively assured of a successful outcome requires several hundred trials, and no student has the resources to commit to this experiment. As I have repeatedly tried to explain to Professor Hyman and others, when dealing with a small to medium effect it takes hundreds or sometimes thousands of trials to establish "statistical significance." In fact, the Physicians Health Study that initially established the link between taking aspirin and reducing heart American Institutes for Research 3-76 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 ane Chapter Three: Research Reviews attacks studies over 22,000 men. Had it been conducted on only 2,200 men with the same reduction in heart attacks, it would not have achieved statistical significance. Should students be required to recruit 22,000 participants and conduct such an experiment before we believe the connection between aspirin and heart attacks is real? Despite Professor Hyman's continued protests about parapsychology lacking repeatability, I have never seen a skeptic attempt to perform an experiment with enough trials to even come close to insuring success. The parapsychologists who have recently been willing to take on this challenge have indeed found success in their experiments, as described in my original report. REFERENCE: Schouten, Sybo (1993). "Are we making progress?" In Psi Research Methodology: A Re- examination, Proceedings of an International Conference, Oct 29-30, 1988, edited by L. Coly and J. McMahon, NY: Parapsychology Foundation, Inc., pgs. 295-322. American Institutes for Research 3-77 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews Points of Agreement and Disagreement ‘The exchanges between Dr. Utts and Dr. Hyman has converged upon a number of issues bearing on the remote viewing research, and parapsychological research in general. In this section of the report, we summarize the major points of agreement and disagreement particularly as they pertain to the remote viewing laboratory experiments conducted as part of the current program. By adopting this more narrow focus, we are intentionally bypassing the evidence for remote viewing or other paranormal phenomena that have arguably been demonstrated in other paradigms (e.g., the autoganzfeld studies) or prior to the present set of laboratory studies. Our position is that the charter for the present evaluation is to examine the current program; evidence obtained by other researchers in other laboratories is not a direct component of this program. Before beginning this discussion, we feel it important to recognize that, in our opinion, both review papers are of exceptional quality. One of Dr. Hyman's first comments about Dr, Utts' review was that he considered it perhaps the best defense of parapsychological research he had come across, We concur; likewise, we feel that Dr. Hyman's paper represents one of the clearest expressions of the skeptic position we have seen. At the outset, it should be noted that the two reviewers agreed far more than they disagreed. One central point of agreement concerns the existence of a statistically significant effect: Both reviewers note that the evidence accrued to date in the experimental laboratory studies of remote viewing indicate that a statistically significant effect has been obtained. Likewise, they agree that the current (e.g., post-NRC review) experimental procedures contain significant improvements in methodology and experimental control. The major remaining area of disagreement concerns attribution of causality. It is Dr. Hyman's position that existence of a statistically significant effect does not allow one to infer that these laboratory experiments have provided an unequivocal demonstration of remote viewing. A statistically significant effect might arise from many sources; more simply stated, the results could have occurred for many different reasons, Until the reasons for the results can be pinpointed, one can not say that fully adequate evidence has been obtained for the existence of any phenomenon, including paranormal phenomena such as remote viewing. Anierican institutes for Research 3-78 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Chapter Three: Research Reviews cr Typically, the process of identifying causes involves both eliminating competing explanations and developing and testing, in Dr, Hyman's terms, “positive” hypotheses. Dr. Hyman states that these competing explanations have not been adequately eliminated; Dr. Utts believes that they have. Our conclusion is that, although the remote viewing research has made substantial methodological progress in recent years, a central problem remains in the laboratory experiments conducted as part of the present program. The problem is that most if not all significant findings have been obtained by using the same remote viewers, the same judge, the same target set, and the same scoring procedures. This characteristic of the remote viewing research in the program raises the possibility of what has been termed "monomethod bias" — a factor limiting the confidence we can have in drawing conclusions about the generality utility of the phenomenon. Basically, the concept here is that a specific method could have some very subtle influences on performance that only appear over many repetitions or trials; these influences could have nothing to do withh the phenomenon being investigated. Recognition of this point is why both reviewers stress the need for independent replication, Another consequence of this experimental situation (the same remote viewers, etc.) is that it makes attribution of causation difficult. For example, one interpretation of the significant outcomes obtained is that the judge has the ability to influence the results of the remote viewing. As unlikely as this seems, this interpretation cannot be ruled out unless different judges are used. Although this general problem was of some concern to the reviewers, both were more concerned about one specific aspect of the problem: The methods used in the current program to study remote viewing involve having a judge assess the degree of correspondence between the viewings and a set of targets. In the current studies, this judge has typically been the Principal Investigator, a person who has substantial familiarity with both the viewers and the research paradigm. In addition to the potential confounding noted above, this kind of procedure can potentially to result in "criterion contamination"— some unconscious, nondeliberate influence on the results in a particular direction. As a result, both reviewers stress the point that cross-judge agreement needs to be demonstrated: the studies need to be replicated using independent judges before strong statements can be made about the existence American institutes for Research 3-79 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5