A Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 Journal of Parapsychology, Vol. 54, September 1990 ‘ ADVANCES IN REMOTE-VIEWING ANALYSIS By EpwIN C. May, JEssicA M. UTTS, BEVERLY 5. HUMPHREY, Wanpa L. W. LUKE, THANE J. FRIVOLD, AND VIRGINIA V. TRASK ABSTRACT: Fuzzy set technology is applied to the ongoing research question of how to automate the analysis of remote-viewing data. Fuzzy sets were invented to describe, in a formal way, the subjectivity inherent in human reasoning. Applied to remote-viewing analysis, the technique involves a quantitative encoding of target and response material and provides a formal comparison. In this progress report, the accuracy of a response is defined as the percent of the intended target material that is described correctly. The reliability is defined as the percent of the response that was correct. The assessment of the remote-viewing quality is defined as the product of accuracy and reliability, called the figure of merit. The procedure is applied to a test set of six remote-viewing trials. A comparison of the figures of merit with the subjective assessments of 37 independent analysts shows good agreement. The fuzzy set technology is also used to provide a quantitative defini- tion of target orthogonality. i Human analysts are commonly used to evaluate free-response data. Although there are many variations, the basic idea is that an analyst, who is blind to the actual result, is presented with a re- sponse and a number of target possibilities, one of which is the in- tended target. The analyst’s task is to decide what is the best re- sponse/target match, and frequently includes rank-ordering the targets from best to worst correspondence with the response. It is beyond the scope of this report to provide a critical review of the extensive literature on this topic. One aspect, however, of this type of evaluation is that analysts are required to make global judgments about the overall match be- tween a complex target (e.g., a photograph of a natural scene) and an equally complex response (e.g., written words and drawings). In a recent book, Dawes (1988) has discussed various decision algo- rithms in general and the difficulty with global techniques, such as those used in rank-order evaluation, in particular.’ According to Dawes, the research results suggest that global decisions of this type are not as good as those based on smaller subelements that are later 1We are indebted to Professor D. Bem, Cornell University, for directing us to this valuable source of information. Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
‘Approved, For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 The Journal of Parapsychology ? combined. (See Dawes, 1988, chap. 10, for references to the re- search.) Humans appear to be capable of deciding what the appro- priate variables should be in complex decision processes, but they have proved to be unreliable at combining these variables to arrive at a single decision. Linear algorithms are consistently better at this latter task. Therefore, it seems prudent to develop evaluation tech- niques that are less sensitive to global decision processes and rely on combinations of more restrictive decisions. Honorton (1975) has pointed out an additional difficulty inher- ent in a global rank-order approach. Asking an analyst to rank- order a small set of target possibilities converts the free-response experiment into a forced-choice one, at least on the part of the an- alyst. It is obvious that in doing so, much quantitative information is lost. For example, a near perfect correspondence between re- sponse and target will receive only as much “credit” as one that just barely allowed an analyst to discriminate among the possibilities. If multiple analysts are used, addition problems arise concerning interanalyst reliability. If an individual analyst judges a number of responses in a series, within-analyst consistency becomes an individ- ual problem. To address these difficulties, various computer-automated pro- cedures have been suggested in an attempt to reduce the inter- analyst reliability while increasing within-analyst consistency. For ex- amples, see Honorton (1975), Humphrey, May, Trask, and Thomson (1986), Humphrey, May, and Utts (1988), Jahn, Dunne, and Jahn (1980), May (1983), May, Humphrey, and Mathews (1985), and Targ, Puthoff, and May (1977). In this paper we present the current status of an ongoing re- search topic. We are not yet ready to propose that the techniques described here be used for free-response analysis; however, we hope to inspire the community to develop a proper set of subvariables so that the problems inherent in global decision processes can be avoided. Finally, we present a successful application of the mathematical techniques for quantifying target orthogonality for a complex target pool. Background Substantial progress has been made in methods for evaluating | remote-viewing experiments since the publication of the initial re- mote-viewing (RV) effort at SRI International (Puthoff & Targ, Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
Approved For Release 2001/03/07 : CIA-RDP96- 00791R000200100003- 3 dvances in Remote-Viewing Analysis 1976). This paper outlines some of the progress and presents the details for one particular method.” Two basic questions are inherent in the analysis of any remote- viewing data, namely, how is the target defined, and how is the re- sponse defined. In a typical outbound RV experiment, definitions of target and response are particularly difficult to achieve. The protocol for such an experiment dictates that an experimenter travel to some ran- domly chosen location at a prearranged time; a viewer’s task is to describe that location. One method of trying to assess the quality of the RV descriptions in a series of trials is to require that an analyst visit each of the sites and attempt to match responses to them. While standing at a site, the analyst has to determine not only the bounds of the site, but also the site details that are to be included in the analysis. For example, if the target location was the Golden Gate Bridge, the analyst would have to determine whether the buildings of downtown San Francisco, which are clearly and prominently vis- ible from the bridge, were to be considered part of the target. The RV response to the Golden Gate Bridge target could be equally troublesome, because responses of this sort are typically 15 pages of dream-like free associations. A reasonable description of the bridge might be contained in the response; it might be obfuscated, how- ever, by a large amount of unrelated material. How is an analyst to approach this problem of response definition? The first attempt at SRI at quantitatively definmg an RV re- sponse involved reducing the raw transcript to a series of declarative statements called concepts (Targ et al., 1977). Initially, it was de- cided that a coherent concept should not be reduced to its compo- nent parts. For example, a small red VW car would be considered a single concept rather than four separate concepts, small, red, VW, and car. Once a transcript had been “conceptualized,” the list of concepts constituted, by definition, the RV response. The analyst rated the concept lists against the sites. Although the response was well defined by this method, no attempt was made to define the tar- get site. In 1982, a procedure was developed to define both the target and response material (May, 1983). It became evident that before a site can be qualified, the overall remote-viewing goal must be clearly defined. If the goal is simply to demonstrate the existence of the ® Although the term remote viewing is used throughout this paper, the analysis techniques can easily be applied to any free-response data. Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
‘Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 196 The Journal of Parapsychology , RV phenomenon, then anything that is perceived at the site is im- portant. But if the goal is to gain specific information about the RV process, then possibly specific items at the site are important whereas others remain insignificant. | In 1984, work began on a computerized evaluation procedure (May et al., 1985), which underwent significant expansion and re- finement during 1986 (Humphrey et al., 1986). The mathematical formalism underlying this procedure is known as the “figure of merit” (FM) analysis. This method is predicated on descriptor list technology, which represented a significant improvement over ear- lier “conceptual analysis” techniques, both in terms of “objectifying” | the analysis of RV data and in increasing the speed and efficiency with which evaluation can be accomplished. Humphrey’s technique, which was based on the pioneering work of Honorton (1975) and its expansion by Jahn, Dunne, and Jahn (1980), was to encode tar- get and response material in accordance with the presence or ab- sence of specific elements. It became increasingly evident, however, that this particular ap- plication of descriptor lists was inadequate in providing discrimina- tors that were “fine” enough to describe a complex target accurately, and unable to exploit fully the more subtle or abstract information content of the RV response. To decrease the granularity of the RV evaluation system, therefore, a new technology would have to allow the analyst a gradation of judgment about target and response fea- tures rather than the hard-edged (and rather imprecise) all-or-noth- ing binary determinations. Requiring an analyst to restrict subjective judgment to single elements rather than to complete responses is consistent with the research reported by Dawes (1988). A preliminary survey of various disciplines and their evaluation methods (spanning such diverse fields as artificial intelligence, lin- guistics, and environmental psychology) revealed a branch of math- ematics, known as “fuzzy set theory.” Fuzzy Set Concepis | Fuzzy set theory was chosen as the focal point of the RV analyt- | ical techniques because it provides a mathematical framework for modeling situations that are inherently imprecise. Because it is such an important component in the analysis, a brief tutorial will be pre- sented to highlight its major concepts. ° We wish to thank S. James P. Spottiswoode and D. Graff, CE, for directing us to the fuzzy set literature and for many helpful discussions. ‘Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 Advances in Remote-Viewing Analysis 197 TEST CITY Membership Value 1.0 0.8 0.5 0.3 0.2 0.1 0.01 0.0 1 2 3 4 5 6 7 8 9 10 eee 15 ee 30 Population in 100 Thousands Figure 1. The fuzzy set “kind-of-small” cities. In traditional set theory (i.e., crisp sets), an element either is or is not a member of a set. For example, the crisp set of cities with population equal to or greater than 1,000,000 includes New York City, but not San Francisco. This set would also not include a city with a population of 999,999. The problem is obvious. There is no real difference between cities with populations of 1,000,000 and 999,999, yet one is in the set and the other is not. Humans do not reason this way; therefore, something other than crisp sets is re- quired to capture the subjectivity inherent in RV analysis. Fuzzy set theory introduces the concept of degree of membership. Herein lies the essence of its applicability to the modeling of impre- cise concepts. For example, if we consider the size of a city, we might define certain fuzzy sets, such as very small cities or kind-of-small cities. Using kind-of-small cities as a fuzzy set example, we might sub- jectively assert that a city with a population of 100,000 is definitely such a city, but a city with a population of 400,000 is only a little bit like a kind-of-small city. As depicted in Figure 1, fuzzy set theory al- lows us to assign a membership value between 0 and 1 that repre- sents our best subjective estimate as to how much each of the pos- sible city populations embodies the concept kind-of-small. In this example, a population of 700,000 assigned a membership value of Clearly, a different set of membership values would be assigned to the populations for the fuzzy sets very small cities, medium cities, large cities, and so forth; a population of 100,000 might receive a value of 0.2 for very small cities, but a value for 1.0 for kind-of- small cities, depending on context, consensus, and the particular Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 198 The Journal of Parapsychology , application. These membership values can be obtained through con- sensus opinion, a mathematical formula, or by several other means. Crisp sets are special cases of fuzzy sets, in which all membership values are either zero or one. By using membership values, we are able to provide manipulatable numerical values for imprecise natu- ral language expressions; in addition, we are no longer forced into making inaccurate binary decisions such as, “Is the city of San Fran- cisco large—yes or no?” In this example, the crisp set of all cities defines the universal set ; of elements (USE). The crisp set of cities with populations of one million or more is a subset of USE. The fuzzy sets very small, kind- of-small, medium, and large cities are fuzzy subsets of USE. Universal Set of Elements Since targets and the responses will be defined as fuzzy sets, we must specify a USE. The universal set of elements can be quite gen- eral and include all aspects of a given target pool, or it can be tai- lored to a specific experiment to test a given concept (e.g., include only geometric shapes). Since the method of fuzzy set analysis crit- ically depends on the choice of USE, we provide one example that was derived from a target pool used in earlier experiments. What follows is only an example of how one might construct a USE. The one we use is not generally applicable to other target pools or other experiments. We constructed our USE by including a list of features present in photographs from the National Geographic magazine with ele- ments obtained from the RV responses in earlier experiments. This USE is presented in Appendix A as the actual coding forms. For the target features, we focused on direct visual elements. (In the general case, other perceptual dimensions can be considered.) In the case of the RV response-derived elements, an effort was made to preserve the vocabulary used by the viewers. Some of the ele- ments, therefore, are either response-dependent or target-depen- dent or both, whereas others, particularly at the more abstract lev- els, appear to be more universal across possible USEs. This universal set of elements is structured in levels, ranging from the relatively abstract, information poor (such as vertical lines), to the relatively complex, information rich (such as churches). The current system is structured into seven primary and three secondary levels of elernents; the main intent of this structure is to serve as a heuristic device for guiding the analyst into making judicious con- Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 vances in Remote-Viewing Analysis 199 crete element assignments based on rather abstract commentary. The use of levels is advantageous in that each element level can be weighted separately and used or not, as the case may be. This ena- bles various combinations of levels to be deployed to identify the optimal mix of concrete versus abstract elements. Of course, any such weighting scheme must be determined in advance of any ex- periment. The determination as to which elements belonged on which level was made after consideration of two primary factors: (1) the appar- ent ability of the viewers to be able to resolve certain features, cou- pled with (2) the amount of pure information thought to be con- tained in any given element. Some of these “factor one” determinations were based on the combined anecdotal experience of analysts and monitors in the course of either analyzing or con- ducting numerous RV experiments; some were determined empir- ically from post hoc analyses of viewers’ abilities to perceive various elements in previous experiments. The “factor two” determinations were made primarily by arrang- ing the elements such that an element at any given level represents the sum of its constituent elements at lower levels. For example, a port element (Level 7) could be considered to include canal (Level 6) and partially bounded expanse of water (Level 5). The world is not a very crisp place and not all its elements are amenable to hierarchical structuring. Certain violations of the “factor two” rule appear, therefore, throughout the USE example. It should be noted, how- ever, that some of the more glaring violations were largely driven by the “factor one” determinations (i.e., the viewers’ abilities to dis- cern certain elements) enumerated above. To emphasize once again, it is very important to realize that this universal set of elements was constructed to match our particular special targets, viewers, and requirements. They are shown here to illustrate the procedure. Any particular application of fuzzy set tech- nology to the analysis of free-response material requires an a priori construction of an individualized, and improved, USE specific to the target pool and the goals of the experiment. Target Fuzzy Sets Each target is defined as a fuzzy set constructed by assigning a membership value to each of the elements in the USE (see Appen- dix A). In general, membership values can vary continuously on the interval [0,1]. In this application they represent human judgment Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
‘Approved For Release 2001/03/07 : CIA-RDP96-00791 R000200100003-3 200 The Journal of Parapsychology , and, thus, were constrained to vary in steps of 0.1. In addition, they must represent the perceptual dimension used to construct the USE. In our example, membership values were assigned to each element for each of the targets, according to a consensus (on an element-by- element basis) reached by three analysts. This approach was used to mitigate the potential influence of any single coder’s biases and idio- syncrasies. A numerical assignment, p (0 < p < 1, in steps of 0.1), was made for each element in response to the following question: How visually important is this element to this photograph? Encoded by this method, the fuzzy sets served as a formal defi- nition of the targets for the analysis. It should be noted that our USE defined targets in terms of visual importance.’ If other dimen- sions are of interest (e.g., conceptual, functional, allegorical), the USE would have to be revised to incorporate them. In an actual experimental series, it is critical that the target fuzzy sets be defined by analysts before the series begins. Because of the potential information leakage owing to bias on the part of the ana- lyst, it is an obvious mistake to attempt to define the target fuzzy set on a target-by-target basis in real time or post hoc, Response Fuzzy Sets To define RV response fuzzy sets, membership values ». are as- signed for each element in the USE by asking: To what degree am I (the analyst) convinced that this element is represented in this re- sponse? For example, if a response explicitly states “water,” then the membership value for the water-element should be 1. If, however, the response is a rough sketch of what might be waves, then the membership value for the water-element might be only 0.3, depend- ing on the specificity of the drawing. This definition of membership value is quite general and can be used in most applications. In our example, responses were coded according to this defini- tion (but still using the USE in Appendix A). The assigned p’s for the targets and responses were one-digit fuzzy numbers on the in- terval [0,1] (e.g., 0.1, 0.2, 0.3, etc.). In some rare cases, two-digit assignments (€.g., 0.05, 0.15, 0.25, 0.35, etc.) were made; any finer assignments, however, were deemed to be meaningless. Thus, the response was defined as its fuzzy subset of the USE. 4 Implied visual importance was ignored. For example, in a photograph of the Grand Canyon that did not show the Colorado River, water, river, and so on would be scored as zero. By definition the target was only what was visible in the photo- graph. Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 Advances in Remote-Viewing Analysis 201 In an actual experimental series, each response fuzzy set is cre- ated by analysts who are blind to the intended target. Fuzzy Set Definition of Figure of Merit Once the fuzzy sets that define the target and the response have been specified, the comparison between them to provide a figure of merit (FM) is straightforward. In previous work (Humphrey et al., 1986), we have defined accuracy as the percent of the target material that was described correctly by a response. Likewise, we have de- fined reliability (of the viewer) as the percent of the response that was correct. The FM is the product of the two; to obtain a high FM, a response must be a comprehensive description of the target and be devoid of inaccuracies. The mathematical definitions for accuracy and reliability for the jth target/response pair are as follows. Let v(R;) and y,(T;) be the membership values for the kth element in USE for the ith response and the jth target, respectively. Then the accuracy and reliability for the ith response applied to the jth target are given by: 2,Wamin{u(R,) maT} accuracy, = ay = 7 ae I k 2,Wymin{n(R,),pa(T)} reliability, = ry = WandR) f) med 3 4, h where the sum over & is called the sigma count in fuzzy set terminol- ogy, and is defined as the sum of the membership values. We have allowed for the possibility of weighting the membership values with weights W, in order to examine various level/element contributions to the FM. The index, k, ranges over the entire USE. For the above calculation to be meaningful, the w’s for the tar- gets must be similar in meaning to the w’s for the responses. As we noted above, in our definition of the membership values, this is not the case. The target .’s represent the visual importance of the ele- ment relative to the scene, and the response y’s represent the de- gree to which an analyst is convinced that the element is repre- sented in the response regardless of its relevance to that response. With advanced viewers it might be possible to change the defi- nition of the response y’s to match the definition of the target ps. In that case, the viewer must not only recognize that an element is Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
‘Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 202 The Journal of Parapsychology ; | present in the target, but must also provide information as to how visually important it is. This ability is currently beyond the skill of most novice viewers. Alternatively, we have opted to modify the tar- get ». definition by using the fuzzy set technique of a-cuts. In our example, an a-cut is a way to set a threshold for visual importance. All target elements possessing that threshold value or higher are considered to be full members of the target set. In fuzzy set par- lance, an a-cut converts a fuzzy set to a crisp one. The result is that the target set is now devoid of detailed visual information: a poten- tial target element is either present or absent in the target set, re- gardless of its actual visual importance. Even with this conceptual change in the target definition, the FM formalism described above ; remains applicable, because a crisp set can be considered as a fuzzy set with all membership values equal to 0 or 1. It is important to recognize that the a-cut is only applied to the target set; the re- sponse set remains fuzzy. Assessment of Quality of the Remote Viewing It is difficult to arrive at a general assessment of how well a given response matches a specified target. The ideal situation is to obtain some absolute measure of goodness of match. Although the FM is an approximation to this measure, it is impossible to assess the like- lihood of a particular FM value because it requires knowledge of the viewer's specific response bias for the session. It is possible to deter- mine general response biases (May et al., 1985), but that knowledge is only useful on the average. For example, a viewer may love rock climbing and may spend most of his free time involved in that ac- tivity. Thus, the general response bias would probably entail aspects of mountains, rocks, ropes, and so forth. Suppose, however, that the viewer spent the evening previous to a given RV session on a ro- mantic moonlight sail on San Francisco Bay. For this specific RV session, the response bias might include romantic images of the moonlit water, lights of the city, and bridges. The current solution to the problem is to provide a relative as- sessment of FM likelihood. A relative assessment addresses the fol- lowing question: “How good is the response matched against its in- tended target, when compared to all possible targets that could have been chosen for the session?” This is not ideal, since the answer de- pends on the nature of the remaining targets in the pool. An ex- ample of the worst-case scenario illustrates the problem. Suppose Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
a a er Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 Advances in Remote-Viewing Analysis 203 that the target pool consisted of 100 photographs of waterfalls, and the viewer gave a near-perfect description of a waterfall. (We as- sume that this description is not fortuitous.) An absolute assessment of the resulting FM should be good, whereas a relative assessment will be low. The worst-case scenario can be avoided, to a large de- gree, by carefully selecting the target pool. (See the later section “A Quantitative Definition of Target Orthogonality.”) To provide a relative assessment of the likelihood of a given FM, we define the score for one session to be the number of targets, n, out of a total, N, that have an FM equal to or higher than the FM achieved by the correct match.* The answer to the question: “Given this response, what is the probability of selecting a target that would match it as well as or better than the target selected?” is n/N. Consecutive RV responses by the same viewer are not statistically independent, nor can the responses be considered to be random in any sense. The statistically independent random element in the ses- sion is the target. Since targets are selected with replacement, under the null hypothesis of no psi, the collection of scores derived over a series of m trials constitutes a set of independent random variables, each with a discrete uniform distribution. Under the null hypothe- sis, the mean chance expectation for the score in each session is given by (N + 1)/2 and the variance is given by (N? — 1)/12. If K is the sum of scores from a series of remote viewings, then the prob- ability of K, under the null hypothesis, can be obtained from the exact distribution for the sum of ranks given by Solfvin, Kelly, and Burdick (1978): PK or less) = > deus 7 ON ') (1) =e If m is large, then the sum-of-ranks distribution is approximately normal and K/m has a mean of (N + 1)/2 and a variance of (N* — 1)/12m. Thus, a z score can be computed from: 0.5(N + 1) - * 2(K or less) = 2 Ne 1 °) 12m °N must be the size of the target pool from which each target was randomly selected, and for this theoretical discussion, we assume no ties, Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
d For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 Approve 204 The Journal of Parapsychology | Ground Truth To determine whether the new analytical approach was effective, a standard had to be developed against which it could be measured. It was determined that this standard—known as “ground truth”— : should consist of a “real-world” normalized consensus about the de- gree of correspondence between RV responses and their intended targets. To achieve this objective, we presented analysts (chosen from the general SRI staff) with the same test case of six remote-viewing re- sponses and their associated targets. The test case was the data from a single viewer (177) taken from an experimental series in a 1986 photomultiplier tube experiment (Hubbard, May, & Frivold, 1987). The responses (i.e., two to five pages of rudimentary drawings with some associated descriptive words) were fairly typical of novice viewer output and represented a broad range of response quality. The targets consisted of six photographs of outdoor scenes selected from a National Geographic magazine target pool of 200. Thus, this data set was ideally suited for an analysis testbed. Appendix B con- tains the “best” and “worst” trials (Sessions 9005 and 9004, respec- tively) from this series in the form of their responses, their intended targets, and their fuzzy set encodings (see the next section). Each analyst was asked individually for his subjective judgment about the degree of correspondence between the remote-viewing re- sponses and their respective intended targets. The “degree of cor- respondence” was purposely undefined, the analysts had to formu- late their own criteria. The only information provided was that responses typically begin with small bits of information and even- tually culminate in a composite drawing at the end. Appendix C contains the coding form that was used to obtain “ground truth.” Each analyst was instructed to examine all of the responses and their intended targets. Then, on a session-by-session basis, he was asked: (1) to assess the degree of correspondence between the remote-viewing response and its intended target, and (2) to register this correspondence assessment by making a vertical hash mark across a 10-cm scale ranging from “none” to “complete.” | To perform the ground truth analysis, distance measurements were taken from the left end point of each scale to the vertical slash mark for each assessment. Let the distance obtained for the Ath ses- Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
Approved For Release 2001/03/07 : CIA-RDP96-00791 R000200100003-s vances in Remote-Viewing Analysis sion from the jth analyst be given by x;,. To account for analysts’ biases, the x,, were normalized by a z transformation, where 1; and a; are the mean and standard deviation of the jth an- alyst’s distance scores, x,,. The effect of this transformation is to convert an analyst’s absolute subjective opinion to a relative one. For the jth analyst, the largest z,, indicates that the degree of corre- spondence for response/target k is higher than any other pair in the series. It does not indicate overall quality. This type of transforma- tion was necessary since we wished to combine the assessments from a number of different analysts. To combine the assessments across analysts, we computed the mean z score for each response/target pair, k, as: Mz ZR = ihe “ N, j=l where N, is the number of analysts. The number of analysts was determined by the data. For the best response/target pair (i.e., ses- sion 9005, k = 5) we computed the percent change of z, for every additional analyst. When the addition of two new analysts produced consecutive changes of less than 2%, the process was considered complete. For this data set, 37 analysts were required before this condition was met. Figure 2 shows the normalized mean for each target/response pair, and represents a relative assessment of remote- viewing quality. These means constitute the basis for the ground truth against which the fuzzy set technique was measured. We re- cognize that this definition of ground truth is based on global deci- sions and may not be most optimal (Dawes, 1988). Results of the Fuzzy Set Analysis To effect a meaningful comparison between ground truth and the figure of merit analysis, we also analyzed the same RV series that served as the ground truth set by the fuzzy set figure of merit method. The fuzzy set membership values (w’s) for the six targets and six responses were consensus coded by five analysts ranging from expert to novice. A typical spread of p assignments was + 0.1 with an occasional outlier. Some of the elements were vigorously de- bated until a consensus was reached. Accuracies, reliabilities, and Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
Approved For Release 206 Normalized Mean 1.5 9001 9002 9003 9004 9005 9006 Session Number Figure 2. Normalized mean for each target/response pair. figures of merit were calculated for each target/response pair (Table 1). It should be noted that the encoding was a post hoc exercise, but because the assignment for each element in the USE had to be de- fended before a consensus was reached, the FMs shown in Table 1 constitute reasonable estimates of their “blind” equivalents. Appen- dix B shows the target and response elements that were scored from the universal set (see Appendix A) for Sessions 9004 and 9005. As an example of the fuzzy calculation, Appendix B also shows the re- Figure of Merit 0.500 Fuzzy Sets CL] Ground Truth | 0.250. ~~~ 9001 9002 9003 9004 9005 9006 Session Number Figure 3. Comparison with ground truth. Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 2001/03/07 : CIA-RDP96-00791R0002001000 The Journal of Parapsychology 03-3 qd
Approved For Release 2001/03/07 : CIA-RDP96-00791 R000200100003-3 vances in Remote-Viewing Analysis TABLE 1 Fuzzy SET QUANTITIES FOR “GROUND TRUTH” SERIES eee Figure of Fractional Session Accuracy Reliability merit Rank rank 9001 317 484 153 80 .403 9002 273 477 130 103 515 9003 358 571 .205 31 155 9004 212 379 .080 142 713 9005 573 594 340 3 015 9006 .298 555 165 13 068 TT eee sults of the target a-cut, the fuzzy intersection, and the accuracy, reliability, and figure of merit for Session 9005. Table 1 also shows the absolute and relative ranks from a target pool of 200. To deter- mine the absolute rank for each session, we calculated figures of merit for all 200 targets in the pool and placed them in numerical order from the largest to the smallest. The absolute rank is just the position (from the top) of the FM corresponding to the intended target. Ties were resolved by choosing the next larger integer rank number to the centroid of the ties. The fractional rank number can be considered a p value for an individual session and is equal to the absolute rank/200. Using Equation 1, the overall p value for the combined six trials is .052 (V = 200, K = 372, m = 6). Using the approximation (Equation 2), we compute z = 1.633, Pp = .05, to demonstrate that for six trials, the approximation in reasonable. For completeness, we compute the effect size (r = 0.67). To compare the results of the fuzzy set analysis with those of the ground truth, we linearly renormalized the ground truth figures to be within the interval [0,1] and to possess the same maximum and minimum. As can be seen from Figure 3, the results from the fuzzy set analysis system parallel those obtained by a consensus of the 37 analysts each making a subjective assessment of the matches. These results imply that the combination of (1) the structure of the USE (i.e., the linguistic hierarchical structure), (2) the fuzzy set mathematics, and (3) a consensus approach to assessing the fuzzy sets themselves provided a reasonable representation of the subjec- tive scoring of the same data by a large number of individuals. A Quantitative Definition of Target Orthogonality It is often of interest to define how similar or dissimilar targets are to each other. For example, free-response experiments like the Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
: Approv Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 ed For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 208 The Journal of Parapsychology ganzfeld often use target packets, with the unselected targets in a packet serving as decoys for judging. Assigning potential targets to packets would be easier with some measure of target orthogonality. Target definition for the purposes of this mode of analysis is ex- actly the same as the one described (Le., a given target is defined by its fuzzy subset of the USE, which has been coded to reflect the vis- ual importance of each target element). The average number of ele- ments, of the total of 131, that was assigned a nonzero value for the targets in our pool of 200 was approximately 37, indicating that the fuzzy set representation of the target pool is rich in visual infor- mation. We used this information to determine the degree to which the target set contains visually similar targets. It is beyond the scope of this paper to describe the extensive work in the literature seeking to find algorithmic techniques that mimic human assessments of visual similarity. One recent article de- scribes techniques similar to the one we used (Zick, Carlstein, & Bu- descu, 1987). We begin by defining the simlarity between target i and target 7 to be a normalized fuzzy set intersection between the two target sets: (Swninoucrd.t7)) 8; = ——o—————— j 2 Wapa(T) 2, War(T) where the index f ranges over the entire USE. We have allowed for the possibility of weighting the membership values with weights W, to examine various level/element contributions to the target similar- ities. For N targets, there are N(N— 1)/2 unique values (19,900 for N = 200) of S,. The values ¢ and j that correspond to the largest value of S, represent the two targets that “look” most similar. Suppose another target m is chosen and S,,; and S,,; are computed. If both of these values are larger than S,,,, (for all n not equal to i or j), then target m is assessed to be most similar to the pair ij. The process of grouping targets based on these similarities is called cluster analysis. Using this process, 200 targets were grouped into 19 clusters, such that the targets are similar within a cluster, and dissimilar be- tween clusters. Table 2 provides an overview of the 19 clusters found from the total analysis of the 200 targets. Some of the names appear to be quite similar, but, in fact, these sets are visually quite distinctive. Figure 4 shows the graphic output of a single cluster in
Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 Advances in Remote-Viewing Analysts 209 TABLE 2 NAMES OF THE 19 CLUSTERS No. Name No. Name 1 Flat towns 11 Cities with prominent geometries 2 Waterfalls 12 Snowy mountains 3 Mountain towns 13 Valleys with rivers 4 Cities with prominent structure 14 Meandering rivers 5 Cities on water 15 Alpine scenes 6 Desert/water interfaces 16 Outposts in snowy mountains 7 Deserts 17 Islands 8 Dry ruins 18 Verdant ruins 9 Towns on water 19 Agricultural scenes 10 Outposts on water detail. A much more complex—and visually difficult to under- stand—graph is generated for the full cluster analysis and is not included here; this smaller subset, therefore, has been chosen to be illustrative of the whole analysis. All targets in this particular sample cluster are islands; the island in each photograph is visible in its en- tirety. Except for one outlier (i.e., a hexagonal building covering an island), the islands fall into two main groups (i.e., with and without . Numb Linear Geometries ast (e.g., Runways) Many Structures (e.g., Town) Flat And Verdant Sand Bars Mountains 0.0 0.2 0.4 0.6 0.8 1.0 Figure 4. Cluster analysis of island targets. Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
‘Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 210 The Journal of Parapsychology ; manmade elements). The natural islands include three similar mountain islands, two sandbars, and two flat verdant islands. Using cluster analysis in conjunction with fuzzy set analysis pro- vides for a quantitative definition of sets of targets that are similar to each other within a cluster, but visually different across clusters. Orthogonal clusters can be used to provide visual decoy targets for traditional rank-order judging. Recommendations and Conelusions To apply the analysis in its present form to a long RV series is quite labor intensive and, from the results shown in Figure 3, is most likely not justified since this fuzzy set technique approximates human assessment. As we stated in the introduction, however, we are providing only a progress report of ongoing research. Because of the decision concepts described in Dawes (1988) and the obvious benefits of an automated evaluation system, the effort to improve what was described in this paper is certainly justified. The proce- dure can be used “as is” to improve and quantify target orthogo- nality. Several future research areas are suggested to improve the tech- niques described in this paper. The use of both inter- and intra-level weighting factors needs to be examined systematically. In the analy- sis described above, all levels and elements were accorded equal weight. The ideal goal would be to determine the optimal weighted mix of abstract versus concrete elements, as a means to achieving the following objectives: 1. Refinement of the cluster analysis for targets, in an effort to simulate, as closely as possible, what is meant by “visual simi- larities” between targets. 2. Refinement of the analysis of responses, in an effort to achieve even greater correlations between the fuzzy set figure of merit analysis and various forms of ground truth. Another area that requires examination in some detail is the - USE and the hierarchical nature of its structure. It is probable that some elements are more appropriate than others; furthermore, they mS might be more effectively structured in a semantic network as op- posed to a true hierarchy. If a hierarchical structure is retained, then some attention must be paid to the formulation of logical con- sistency rules that govern element use. This would include numeri- Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 Advances in Remote-Viewing Analysis 211 cal relationships governing the membership values (1’s) of higher- order elements (e.g., port) vis-a-vis the combined value of their con- stituent parts (e.g., city, river, boats, jetties, commercial). One inadequacy of the system is that it atomizes conceptual “units.” For example, if the response element is red box, information is lost in separating red from box. Current research in fuzzy set the- ory indicates that fuzzy aggregates of fuzzy elements—“fuzzy sets of fuzzy sets”—are mathematically complex but possible. Some effort should be made to determine whether this technology could be im- plemented as a means to capturing the information content of the RV response with greater accuracy. For the visual analysis, research into visual similarities between pictures of natural scenes may serve as a potential refinement tool. The aim here would be to enhance the visual orthogonality of rank- order analysis decoy targets as much as possible. Experiments in normal perception of similarities would assist in determining whether scenes are perceived as similar because of their low-level geometries, concrete elements, or some combination of factors. The ultimate aim would be to refine the target cluster analysis such that it closely simulates ground truth representations of orthogonality. APPENDIX A CODING FORMS FOR THE UNIVERSAL SET OF ELEMENTS The following coding forms illustrate the use of a universal set of elements (USE) that matched our particular special targets, viewers, and requirements, We constructed our USE by including a list of features present in photographs from the National Geographic with elements obtained from the remote- viewing responses in earlier experiments. Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
‘Approved For Release 2001/03/07 : CIA-RDP96- 00791R000200100003- 3 212 The Journal of Parapsychology : wl i fe) _ a x S) vn fu Qo fi oa o z re) cS) 2 3 4 ([_7] church (other religious buikdings, monastery) 5 6 Figure Al. Coding form. Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
213 Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 , Advances in Remote-Viewing Analysis "za? (= ll STSAZ7 YOLdIHOSAG ALSHONOD Figure A2. Coding form. Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
‘Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 ) wos as a cress me Col* — “eyattese! [7] SadALAHOUV piu : ot eed ; ‘pesennis) palsobue a oe wy [__]# tpoows [Jor tergooyen) Auys [7] te moyod =: The Journal of Parapsychology i STAAR7 HOLdIHDSAd LOVELSAaV 214 Figure A3. Coding form. Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3 215 Advances in Remote-Viewing Analysis SWYOF SWYOI yvinoayHyl ¥YVANTTIANND SaluLAWOID G-€ ¥ -z il STSA31 HOLdINDS3G LOVYLsay Figure A4. Coding form. Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
|Approved For Release 2001/03/07 : CIA-RDP96- 00791R000200100003- 3 216 The Journal of Parapsychology APPENDIX B Fuzzy SET ANALYSIS TESTBED The following pages show the targets, responses, and analysis for two remote-viewing trials. Figure B1. Target for Session 9004. Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3
Approved For Release 2001/03/07 : CIA-RDP96- 00791R000200100003- 3 The Journal of Parapsychology Text “ ree ee tty fex ture [] Figure B3. Page two of the response (Session 9004, Target 1094), [Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003- $
Approved For Release 20 FNS OT ee DE 9600791 R0002001 00003-3 vewing Analys Yauged 4 i a mo |? Inucteres sgh good \t) a Fear Ew 2 Yess -ayv Figure B4, Page three of the response (Session 9004, Target 1094). Approved For Release 2001/03/07 : CIA-RDP96-00791R000200100003-3