For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8 Final Report December 1982 REVISED JULY 1983 A REMOTE VIEWING EVALUATION PROTOCOL (U) By: EDWIN C. MAY Prepared for: DEFENSE INTELLIGENCE AGENCY WASHINGTON, D.C. 20301 Attention: i 8 sci DT-5A CONTRACT MDA908-82-C-0034 SPECIAL ACCESS PROGRAM FOR GRILL FLAME. RESTRICT DISSEMINATION TO ONLY INDIVIDUALS WITH VERIFIED ACCESS. Copy No. .....! ! devaeee This document consists of 48 pages. CLASSIFIED BY: DT-5A REVIEW ON: 31 December 2002 SRI/GF-0247 SIRI] International 333 Ravenswood Ave. « Menlo Park, California 94025 NOT RELEASABLE TO i (415) 326-6200 wai hran it © TWX: 910-373-1246 gh NATIONALS ¥etoved For Release 2000/0 “EWE RDP96-00788R00180005000
Approved For Release 200PQROB ACOG] FH Fp 0788R001800050001-8 CONTENTS (U) LIST OF ILLUSTRATIONS (U). . . 2. 2 2 2 2 we ee we we ew we ew ew ww LIST OF TABLES (U) . I OBJECTIVE (U) . II INTRODUCTION (U) e 8 © © © © © © © © © © @ ee © © 8© © © 8# @ @ * 2© © © «© © © © &© 8© © © @© 8 © © @ es @ © © & III ANALYSIS PROTOCOL (U) . 2... 6 ee ew ew we tw we we ew IV. CONCLUSIONS (U) Appendix A--EVALUATION OF FOUR COORDINATE REMOTE VIEWINGS (U). . Appendix B--SUMMARY OF EVALUATION TECHNIQUE (U). ........ REFERENCES (U. .. . e ee = © © © © © &@® © e@ © © © @ © © ss BP 8 @& «& ii UNCLASSIFIED ll 12 43 44 Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 20bs AGL ASSH5HED\0788R001800050001-8 An~4 A-5 A-6 A-7 FIGURES (U) Sample RV Assessment Form (U) oe e © e@ @ @ © © @ @ @ @ © © @ @ Transcript 1 (U) Transcript 2 (U) oo #¢ e© e#® © © © @ @¢ © @© e @ # #8 # # #8 @© 8 @ @ Transcript 3 (U) Transcript 4 (U) Target Site Hango (U) Target Site Fernando de Noronha (U). .....++.6-e eee Target Site Inverness (U) Target Site Punkaharju (U) Lit UNCLASSIFIED 14 15 16 17 19 20 21 22 Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 200 NGL ASSEBLEB)788R001800050001-8 TABLES (U) 1 Task-Defined Relevance Scale for Target Elements (U) .... 6 2 Quality Assessment Scale (U) ..., Pr 8 3 Numerical Score Conversion Table (U) a 0 eo © oe 8 A-1 Target Element Relevance Ratings for Hangé (U)....... 23 A-2 Target Element Relevance Ratings for Inverness (U) se 23 A-3 Target Element Relevance Ratings for Fernando de Noronha (U). . 2. 1. 1 2 ee we ee we ww wt tt tt we 24 A-4 Target Element Relevance Ratings for Punkaharju (U). .... 24 A-5 RV Assessment Forms for Calibration of Remote Viewing (U). . 25 A-6 A Rank Ordering of Weighted Averages (U) oo ee we we ew Al iv UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release o0bd NGd-AS SHH &D0788R001800050001-8 I OBJECTIVE (U) (U) The objective of this task was to develop an evaluation procedure to assess the relative quality of a set of different remote viewing (RV) responses, UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 2000hkob AcSSRBHGDb0788R001800050001-8 III ANALYSIS PROTOCOL (U) (U) To quantify the analysis procedure, we have divided the task into four separate areas: subject response, target/task definition, quality assessment, and numerical analysis. Figure 1 is a sample RV Assessment Form that has been designed to emphasize the separation of the analysis tasks, Each of the parts of the form are described below. (U) The subject's response should be prepared for analysis without any knowledge of either the target site or the overall task. The aim of this method of response preparation is to reduce a possibly redundant, rambling response to a coherent set of concepts. To meet this requirement we have developed a set of initial guidelines to the conceptualization procedure, (U) A concept is defined as a paraphrase of a single idea that has been expressed in the RV verbal or drawing response. That coherent idea should not be fragmented into component parts. For example, a response might be of the form, "I see a large, textured, gray building." The single concept that expresses this idea should be "large, textured, gray building,” rather than four separate concepts--one for each word in the phrase. Each concept should be entered under the "Transcript Concept" column in the RV Assessment Form, (U) For this initial evaluation technique, a particular concept should be used only once in the analysis. (Some weighting factor propor- tional to concept frequency could be utilized, but, for the initial attempt, only unique concepts are used.) If in the construction of the transcript concept list a concept later in the transcript is a duplicate of an earlier one, it should be so noted by placing the concept number of the original concept in the "D" column. UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Viewer: Approved For Release 200d GLAS SHHE[00788R001800050001-8 Las] 25 Weighted Average Date: Class: iii i NOILVMIVAT IdIMOSNVUL Element of Target Method of Targeting: Session: Target: Transcript Concept n ~ a o o f=} ° oO Lm) ° * Length of Session: Begin Time: End Time UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8 (U) SAMPLE RV ASSESSMENT FORM FIGURE 1
Approved For Release coded Gel. A,$ SHFLEI2b7ssR001800050001-8 (U) To utilize an analysis procedure that is capable of quantitative assessment, it is necessary to define, in advance, what the goals of the assessment-are, In the RV Assessment Form, columns "Element of Target” and "Relevance" are provided to clearly define the goal of the analysis. In the ideal situation, an RV target should be completely specified in advance, A target typically consists of a number of target elements, each of which may have varying relevance with regard to the overall RV task, For any given target, an independent list of target elements should be prepared, The selection of what constitutes a target element is left completely to the discretion of the task coordinator. The target element must be selected with little regard to task relevance (target element relevance is accounted for later), Because an RV target consists, in principle, of an essentially infinite number of possible elements, discre- tion needs to be exercised in the selection process, (U) For each target element identified for the site, the task coordinator must define a relevance rating. This rating allows the coordinator to tailor the analysis to the task requirements. Table 1 shows the scale that is used for the target element relevance rating. Table 1 (U) TASK-DEFINED RELEVANCE SCALE FOR TARGET ELEMENTS (U) Relevance Scale A target element trivial interest A target element minor interest A target element intermediate interest A target element major interest A target element key interest 6 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 200 NGLASSHEBLEB788R001800050001-8 (U) For each transcript concept on the RV Assessment Form (Figure 1), the analyst should attempt to find the element on the list of target elements that he/she considers to be the best match. The analyst should be quite liberal in the concept/element matching (i.e., the quality of the match should be considered at this point in the analysis). If he/she is able to identify a target element that might be considered a match to the given concept, a 1 is placed in the "p"” (present) column on the assessment form. If no element can be identified, a 0 is placed in the "p" column, After making a target element identification, the selected target element, and its corresponding overall relevance rating should be entered in the appropriate columns on the assessment form, (U) Having identified a corresponding target element for each con- cept, it is now appropriate to assess the quality of the match. The quality assessment is done on the basis of how well the single concept in question matches the selected target element. The judgement is to be made without regard to any other issues, such as importance of the concept to the transcript, or importance (relevance) of the target element to the target. Table 2 shows the quality assessment scale that is used for this part of the analysis, The appropriate quality score from Table 2 is entered in the "Quality" column on the RV Assessment Form for each concept for which a matching target element has been identified. (U) An intermediate numerical score is computed for each concept from the relevance and quality (Tables 1 and 2) evaluation as follows: S’=PyxRxQ , tt where P is the value in the p' column (0 or 1); R is the relevance evaluation; and 0 is the quality assessment. S’ can assume values ranging between 0 and 25, Table 3 demonstrates how to determine the final score, S, for a given value of 8’ for each concept. The conversion table is 7 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 2000/08 1GSEe PREP 96-00788R001800050001-8 (U) the transcripts in order of the best to least match for each of the n targets, A simple numerical counting procedure is then used to estimate the likelihood that the judge's transcript/target matches are by chance alone,” This early technique contained little systematic structure for determining the final order of matches, (S) The first step toward systematizing the rank order judging procedure was to preprocess the raw data in the transcript by "concep- tualizing" both the verbal and the pictorial responses. Conceptualizing a transcript requires an analyst to paraphrase the transcript into a list of coherent statements. This concept list is then compared and scored concept-by-concept to each of the targets in the experiment, The resulting scores are averaged for each response, and all responses are rank~ordered on the basis of these scores,° This improved analysis procedure was applied to a number of experiments within the Technology Transfer Task for INSCOM in FY 1980, (U) The problem with the above technique is that there are no guide- lines as to how the analyst should paraphrase the transcript; furthermore, the method in which the concepts are to be assessed against the targets remains undefined. The purpose of the Evaluation Task in FY 1982 was to identify a procedure that corrected these deficiencies. 3 SECRET Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 2obbibifob A&Sibd§20788R001800050001-8 Table 2 (U). QUALITY ASSESSMENT SCALE (U) Discrimination Scale Poor description; only one or two aspects of the Material match. Fair description; a few aspects of the material match, but a large ambiguity exists. Reasonable description; many aspects of the material match, but there remains some ambiguity. Good description; a large number of aspects of the material matches, but it is possible to conceive of material that would be a better match. Excellent description; all or nearly all aspects of the material match. Table 3 (U) NUMERICAL SCORE CONVERSION TABLE (U) Normalized Score 0 1 2 3 4 5 6 8 9 Oo Own DBD uw fF WN Ff © NN FP FP eB mw onan wn © UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 2000 D804 A & Stirdel00788R001800050001-8 (U) used to eliminate the nonuniformly-distributed gaps in scoring numbers that occur if one simply uses the product S’. Thus, the final score for each concept ranges from 0 to a maximum of 14. This conversion table is provided as part of the RV Assessment Form. (U) If an assessment of an individual concept is required, the final score for each concept/target-element match can be related to the quality assessment scale by using the conversions shown in the third column of Table 3 and on the assessment sheet. It should be noted, however, that the integer scores are used to simplify the remaining calculations. (U) To determine a final evaluation of the complete transcript assessed against a given target, a weighted average of concept scores is computed. To assist in the calculation of the weighted average, a tally box score is provided at the bottom of the RV Assessment Form. For each of the possible scores, 0 through 14, the number of concepts that attained that particular score are counted. For example, if 3 concepts were evaluated with a score of 12, a 3 is entered in the box below the 12 score. If the frequency of occurrence of score Si is Ea then the final weighted average is computed by AY = Ef, vf, X S./Z £, vf, : J J J J J Ay, 0.357 Ay (U) The normalized, weighted average score, AL» is entered in the weighted average box on the assessment sheet. The weighted average score has been normalized to be within the range OK< A, < 5.0 . To aid in the interpretation of the result, the quality assessment scale (Table 2) can be used to assess quality of the match between the whole RV response and the given target site. (U) At this point in the evaluation protocol, the following options are available, depending on the task requirement: 9 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 2D) NDEIE8A 9S HEHEB)00788R001800050001-8 (U) e An n Xn rank ordering on the basis of the weighted averages. e A simple selection of the best match. e A statistical evaluation on a concept-by-concept basis. 10 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 2000/08/88 : OE Rppes.007asR001s00050001-8 IV CONCLUSIONS (U) (S) A protocol has been developed to address the relative evaluation portion of the overall RV transcript assessment problem. As a demonstra- tion of the technique, we provide in Appendix A an analysis of a series of four remote viewings that were performed as calibrations during operational sessions at DIA in December 1981. In this series the remote viewing products were of relatively high quality, but nonetheless require a sensitive technique to differentiate because of the similarity of the targets and, hence, of the descriptions. (The series was chosen primarily for that reason.) Application of the assessment technique resulted in the correct blind matching (highest scoring in matches versus cross matches) of three of the four. (U) Appendix B is a one page, step-by-step procedure for the application of this evaluation technique. (U) The material in this document thus constitutes an instruction manual or protocol for application of a step-by-step procedure for quan- titative assessment of the relative target/transcript correlations of a series of transcripts matched into a series of targets. 11 SECRET Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 2Qgq§@0B ACSSEB FED)0788R001800050001-8 Appendix A EVALUATION OF FOUR COORDINATE REMOTE VIEWINGS (U) 12 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 2000 : -RDP96-00788R001800050001-8 (S) On 14 December 1981, four coordinate remote viewings were con- ducted as calibrations during an operational remote viewing session tar- geted on client-designated sites of interest. The calibrations and the operational RV session were conducted at the client's organization and were witnessed by client representatives, These four calibrations were chosen as a test bed for the evaluation procedure for the following reasons, (1) they were conducted in an operational setting, and (2) the targets had many similar features, and would thus provide a sensitive test of the protocol, (S) Figures A-1 through A-4 are the transcripts that were presented to the analyst. They are exactly as they were when collected, except that the coordinates have been removed, Figures A-5 through A-8 are the National Geographic magazine targets that were used during the calibration sessions. Finally, the task coordinator provided Tables A-1 through A-4 as target element relevance scales for the four targets in Figures A-5 through A-8, This completes the information that was given to the analyst, and thus the analysis was carried out blind as to the matching target/ transcript pairs. (U) Table A-5 is a compilation of the completed work sheets that were used by the analyst in this evaluation. They are shown in groups by session number, and alphabetized on the four targets. (The task coordinator first randomized the transcript order then assigned the session number used above.) For each of the transcripts, the analyst simply included all phrases and all drawings as concepts. For example, seven concepts were found during Session 2. (U) All concepts were then analyzed as described in the text. The matching target element, its relevance rating, and the computed score are shown for all possible combinations of transcript/target pairs in Table A-5. The score distributions and their resulting weighted averages are also shown in Table A-5. 13 SECRET” Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release obs bbGLASSIBHEDi0788R001800050001-8 IY Qeol TM 8 Bnvelo: }aere FIGURE A-1 TRANSCRIPT 1 (U) UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 2d¢N&ida AGS Hlifel)00788R001800050001-8 /Y ,2¢ee §] Co2 is 3.23 Plobfonclonns M1) / (Breolo a US wellny «. cll FIGURE A-2 TRANSCRIPT 2 (U) UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 200th LASSESER7s82001800050001-8 (U) (U) Table A-1 TARGET ELEMENT RELEVANCE RATINGS FOR HANGO (U) Town 5 Cold 4 Peninsula 5 Rocky 3 Vegetation 2 Bay 3 Table A-2 TARGET ELEMENT RELEVANCE RATINGS FOR INVERNESS (U) Target Element City Relevance River 5 4 Bridge 3 2 River banks Vegetation 1 23 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 200RDMG LASSIFIEDD788R001800050001-8 Table A=-3 (U) TARGET ELEMENT RELEVANCE RATINGS FOR FERNANDO DE NORONHA (U) Target Element Island Relevance Surf Hills Uninhabited Mountain peak Temperate climate Vegetation Ocean Table A-4 (U) TARGET ELEMENT RELEVANCE RATINGS FOR PUNKAHARJU (U) Target Element Relevance Connect lakes Town Bridges Cold Vegetation Islands 24 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8 SS ee bain fa: | (MQ) ONIMAIA YLOWHA JO NOILVYAITVO YOA SWHOA LTNYWSSASSV Ad G-V¥ Wave aselsay pelystay wfelede fete ep fe Lebo] a comer |e a eee A A, a eS *s88TD idoTsseg sdoTsseg jo 44803] I861T 29qu=D9q FT 1azeq $9} BUT P100) 3 Sutzesrey, Jo poy ey TOURL Puy Zoo# TLOMATA BSIUIAATI 2302318, OF ZT loMTL UpTzeq PET ET sm NOLLVOTVAZ IAIYOSNVEL 25 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 200 GLA SRSELED 7ssR001800050001-8 Afe| [=] -[-[-[-[-[-[-[s[a[s[s[ = {HIEECCEEEEPEEE EET EERGn #002 14 December 1981 Weighted Average Viewer: . * et 7/0 rere | nents (een a inn RN EE ER ens aed eee eee eee nd 9 | le F a a Fs ro] i # 8 g et a 1] *8, oe FI 2 al] 3 ; & e . Fog 2 38 sa pe 3 Hx Rw 8 9 a [o} Q » i=" z i) a ¢ fi < e N ve eA a ° val a . 4 a ey 2 . of age: a oO a oo # ray auf 8 a @ & 7 NOLLYMIVAN Ld IYOSNVAL ° tH 26 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8 Table A-5 (continued)
Approved For Release oGLASSLELED, o7ser00180005000 ye [a'8} a SI M4 g << for} ' 0 feat Z +) g| § rl @| 3 nl oa ‘et & aad ® = Ai ee oO ee n n B23 ee ee Se Ee oes | od nm fo} wi] 3 Na Bl § 6 a ol Z| 2 3 aq et wl o s = et ) et n em) ® eI 5 B ee SE Se a] x Ht tt i% o ms 4 o | § : fe | | o 8 o Bag 8 peo & 2 3 3 | 4 ne SO 1 1 a an an mE an | |e | | | || ee 4 8 mo : ; ns s 3 wey ny q a | ow & 2 3 H a B ° £ g i i is = 3 F be 4 Nn bat ea] wt al 3 rt 5 a a 2 tl 4% o 3 4 a z ha ol 3 . 8 a a oS a el 3 5. OU 4 5 A > g 2» do. 4 e oo et ° Oo vw} at of eis Si at At al oa ws F=| » 1 e Ed a8 8 aw a = NOLLVQIVAD LEIMDSNVEL 27 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8 Table A-5 (continued)