Approved For Release JUDIGKA Sa LESER 00789R002200390001-5 A FIGURE OF MERIT ANALYSIS FOR FREE-RESPONSE MATERIAL by E. C. May B. S. Humphrey C. Mathews SRI International, Menlo Park, CA ABSTRACT: A simplified automated procedure is suggested for the analysis of free-response material. As in earlier similar procedures, the target and response materials are coded as yes/no answers to a set of questions (descriptors). By definition, this coding defines the complete target and response information. The accuracy of the response is defined as the percent of the target material that is correctly described (i.e., number of correct response bits divided by the number of target bits = 1). The reliability of the response is defined as the percent of the response that is correct (i.e. the number of correct response bits divided by the total number of response bits = 1). The figure of merit is the product of the accuracy and reliability. The advantages and weaknesses of the figure of merit are discussed with examples. INTRODUCTION With the increased use of computers in parapsychology laboratories, it has become possible to consider more complex methods of analysis to provide deeper insight into the mechanisms of the phenomena. The Engineering Anomalies Research Laboratory, Princeton University, provided a major advancement in the analysis of free-response material (Jahn, Dunne and Jahn, 1980). THE PRINCETON EVALUATION PROCEDURE (PEP) - A BRIEF REVIEW In general, the Princeton Evaluation Procedure (PEP) is based on comparing a priori, quantitatively-defined target information with similarly quantitatively-defined response information. So defined, the PEP applies various methods of mathematical comparisons to arrive at a meaningful assessment score for remote viewing responses. A-1 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200390001-5
Approved For Release PII PA SSF PEB-00789R002200390001-5 Target Information The definition of a particular target site (usually outdoor sites in and around Princeton, New Jersey) is contained in the yes/no answers to a set of questions called descriptors. These descriptors are designed in such a way as to characterize the typical Princeton target. Each descriptor bit is weighted by its a priori probability of occurrence in a large target pool. By definition, the only target information that is to be considered for analysis, is that which is contained completely in the yes/no answers to the descriptor questions (with their associated set of descriptor weights) for the site in question. For example, one descriptor from the Princeton list, “Are any animals, birds, fish, major insects, or figures of these significant in the scene?” defines the animal content of the site. The question would be answered “yes” for a zoo and a pet store target, but “no” in all probability for a typical campus building target. Similarly, a set of yes/no responses (30 for the PEP) constitutes the target information. Response Definition The descriptor list for the target sites is used as a definition of the response as well. For a given remote viewing session, the remote viewer (or an analyst who is blind to the target site) attempts to answer the 30 questions on the basis of that single response only. In the example above, it would be necessary for a viewer (or analyst) to decide whether or not a particular verbal passage or a quick sketch could be interpreted as depicting animals. For some responses this might be an easy task, e.g. “I get a picture of a cow.” Most responses, however, are somewhat ambiguous and require a judgment, e.g, “I see a farm.” Nonetheless, the yes/no answers to the 30 questions constitute the only response information that are used in the analysis. Analysis For a given response/target combination, the information is contained exclusively in the yes/no answers to the descriptors. Two binary numbers (30 bits long each for PEP) are constructed, one for the target and one for the response descriptor questions, respectively. A “yes” answer is considered a binary “1,” while a “no” answer is considered a binary “0.” The resulting two, 30-bit binary numbers can then be compared by a variety of mathematical techniques involving use of the weighting factors, to form a score for that specific remote viewing session. For a series of sessions, a quantitative assessment is made by comparing a given response (matched to its corresponding target site) against the scores that are computed by matching the response to all other targets used in the series. This procedure has the added advantage of a built-in, within-group control. In other words, this assessment determines the uniqueness of the target/response match as compared with all other possible matches for the series. Advantages of the PEP There are a number of obvious and proven advantages (Dunne, Jahn, and Nelson 1983) of the Princeton Evaluation Procedure: A-2 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200390001-5
Approved For Release -UNCL : SOT EB oR Onzoosgo001- UNCLA @ Automation - Rapid and accurate analysis of a large number of free-response sessions can be accomplished with ease. @ Archives — With the aid of computer database management, large numbers of free-response sessions can be organized and maintained in a usable manner. ® Control - The cross-target scoring procedure provides a powerful built-in within-group control. e Use ~ PEP is widely distributed and provides a commonalty of analysis procedure across laboratories. Disadvantages of the PEP There are actually very few disadvantages to PEP. A common problem that has been observed before (Dunne, Jahn, 1982) arises in the “granularity” of the descriptor list. With any finite list of binary—-type descriptors, it is always possible that a response will appear to be correct with “analogue” analysis procedures but will be evaluated as incorrect with the “digital” approach. Another disadvantage of PEP (also noted above, op cit) is that any given descriptor list is likely to be applicable only to a given target pool type (i.e., Princeton area natural sites, National Geographic magazine photographs, etc.). Lastly, one of PEP’s strong points--namely, the cross-match, built-in, within-group control—-is also potentially one of its weaknesses. Since nearly all of the various PEP scoring algorithms involve bit-by—bit weighting, which is based upon relative probability of occurrences, a given response/target score depends not only upon the correctness of the response, but also upon the nature of the remaining targets in the pool. Thus, a score for a given session depends upon the quality of response and the target pool. The following hypothetical example illustrates this dependency: a given target has 10 of 30 bits present; furthermore, a few bits (e.g. 3) are particularly rare when compared to the remaining bits (i.e. they possess comparatively large weighting factors). Let us assume that two different viewers provide responses to this target and that each asserts 8 descriptors in the response, 6 of which are correct. If the first viewer’s response contains only one of the rare bits, while the other viewer’s response contains all three, the second viewer’s score will be considerably larger as a consequence of the weighting factors. Such a scoring discrepancy forces us to define what the purpose of the remote viewing session is. If the goal is to demonstrate the existence of psi phenomena, then the PEP is a perfectly adequate system of analysis, and it exhibits all of the advantages described above. If the goal, however, is to demonstrate correlation effects (e.g., correlation of free-response material with personality, physiology, environment, etc.), then the scoring difficulties described above confound the correlation measurement. To summarize, a target pool dependent scoring procedure provides an important measure of a viewer’s ability to discriminate from among a number of possible targets. (The A-3 Approved For Release 200th Siok AcS:Sd bt Ed30789R002200390001-5