Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Appendix D: Stargate Operational User Intertviews response. However, until 1994, the results from this tasking were not evaluated by the tasking organizations by any numerical method that would identify the accuracy and value of the provided information (for a few cases in prior years narrative comments were provided by some organizations). In 1994, this situation changed when the Program Office developed a methodology for obtaining numerical evaluations from the tasking organizations of the Star Gate inputs; this methodology is described briefly in Section 3.0. By May 1, 1995, 40 tasks assigned by five operational organizations had been evaluated under this process.’ Section 4.0 describes the numerical evaluations performed by evaluators from the tasking organizations. The descriptions presented below regarding the tasking and the related targets refer principally to the operational tasks that were numerically evaluated. The process for a typical tasking, RV response and subsequent evaluation is as follows: - The tasking organization provides information to the Star Gate Program Manager (PM) describing the problem to be addressed. - The PM provides a Tasking Form delineating only the most rudimentary information to one or more of the three Star-Gate RV's? for their use during the RV session (a typical Tasking Form is presented in Figure 2-1). In addition, the RV's are appraised of the identity of the tasking organization. - ° Subsequently the RV's hold individual "viewing" sessions recording their comments, observations, feelings, etc. and including line drawings or sketches of things, places, or other items "observed" during the session. - The individual RV inputs are collected and provided to the tasking organization for their review with a request for completing a numerical evaluation of the individual RV inputs for accuracy and for value. - Finally, for those organization who comply with the request, the evaluation scores are returned to the Star Gate Program Office. ‘Evaluation of additional 1994-95 tasks continued after 5/1/95; three tasks since evaluated were reviewed. They caused only insignificant changes to the statistical information provided in Table 4-1 and did not alter any of the Conclusions and Recommendations in Section 7.0. 7 (U) All three RV's were full time government employees. American Institutes for Research D-2 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Te ee Este: Stn aL RT TA aS DN settnein ae Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 a . Appeniiix D: Stargate Operational User Intertviews FIGURE 2-1 TASKING SHEET SOURCE 140: 079 DATE: JS Jul 94 SUSPENSE: 18 Jul 94 1600 Hrs 1. PROJECT NUMBER: 94-252-0 2. METHOD/TECHNIQUE: Method of Choice 3. BACKGROUND: 4, ESSENTIAL ELEMENTS OF INFORMATION: Access and describe target, 5, COMMENTS: ‘American Institutes for Research ~—SC~C~C=“‘—s<‘;73273}SRTté<i‘i‘<2 2 23wW!”!”!”:tS” ns 2) Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
LL Ss erases RHIDTOSNRNNSNNENRNnArer nn a Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Appendix D: Stargate Operational User Intertviews Twenty-six (26) of the 40 operational tasks originated from DIA in support of two joint Task Forces, Org. B and Org. C, (see Section 4.0). Typical tasking targets for these organizations comprised the name of a person or thing (e.g., vessel) with a peneric request to describe the target, his/herAts activities, location, associations, etc. as appropriate. No specific information (e.g., what is the height/weight/age of the target?) was requested in the tasking. As noted above, the identity of the supported organizations also was provided. For these tasks that identification provides the RV's with knowledge regarding the specific operational interests of these organizations. Thus, any information provided by the RV's which describes or relates to those interests "could be" relevant; and, therefore, could be interpreted by the evaluators as having some level of "accuracy" and "value" depending upon the information described and the evaluator’s interests and beliefs. The tasking provided by the organization denoted as Org. A comprised targets that were “places” visited by "beacons", i.e., an individual from Org. A who visited and ,'viewed" the site of interest to assist the RV in "visualizing" and describing the site. Targets could be a general vista in or around a particular location, a particular facility at a selected location or, perhaps, a particular item at a location (in the one case where this type of target was used, the item was a particular kind of boat). Usually, no specifics regarding the type of target or its focation were provided. Tasking by Org. D comprised two generic types of targets that related to military interests/concerns current at the time of the tasking, e.g., North Korean (NK) capabilities and leadership. The first type of target focused upon then-current military concerns while the second type required "precognitive" (predictive) capabilities since it required a prognoses of future intentions and actions.’ The tasking from Org. E was similar in scope, albeit quite different in context, from the tasks noted earlier for Org. B and Org. C, ie., describe a person, his activities, location, ete. As mentioned at the beginning of this section, the descriptions noted above relate to operational tasks that were numerically scored. During the summer/fall period of 1993, eight operational tasks were levied on the program pertaining to North Korean (NK) tunnels. The target information provided to the RV's typically comprised a map of a large section of NK with a request to identify tunnels within the map area. Evaluation of the results from these tasks was in narrative form only; discussion regarding this narration is presented at the end of Section 3.0. *Some operational tasks from the period Oct. 1990 to Jan 1991 regarding Middle East issues were of a similar types, albeit these were not numerically evaluated. They would provide some data for an after- the-fact check of the accuracy of the RV predictions - see Section 7.0 for a discussion of this possibility. American Institutes for Research BFE Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
TL ee ae ea A a RAS taeerer= ese Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Appendix D: Stargate Operational User Intertviews 3.0 EVALUATION MEASURES The numerical evaluation measures that were given to the evaluators of the tasking organizations to score the accuracy and value of the Star Gate inputs were extracted from the Defense Intelligence Agency Manual (DIAM) 58-13. These measures are shown in Table 3- 1. Most of the stipulated measures include modifiers such as "may", "possibly', “high”, "low", etc. which are subjective and open to individual interpretation by each evaluator. The DIAM 58-13 definitions for the ratings under "Value" are presented in Table 3-2; whether the individual evaluators reviewed these definitions prior to their scoring is unknown. There was no clarification of what was intended by the generic headings of “Accuracy” and "Value", e.g., in the evaluator's estimation how much of the RV's response to the tasking had to qualify for a particular measure, 1 10%, 90%, to be granted the related score? Table 3-1 Numerical Evaluation Measures Category Score Accuracy - Is the information accurate? Yes (true) May be true Possible true No Possibly not true* Unsure Value - what is the value of the sources’ information? Major significance 1 High Value 2 Of Value 3 Low Value 4 No Value 5 as | anh oh As noted in Section 2.0, one series of tasks were evaluated by a narrative discussion only. While much of the final narrative evaluation for this series was complimentary, it lacked any real specifics regarding the usefulness or relevance of the Star Gate inputs and much of the narrative was replete with modifiers and other hedges. A sanitized extract from the final evaluation report for these tasks is presented in Appendix A illustrating the subjective, “uncertain” nature of the comments. 4Note that Accuracy scores 5 and 6 actually rank "higher" than 4 since both imply that there may be something accurate in the information. Changing the scoring order to accommodate this observation causes insignificant changes to both the averages and the standard deviations shown on Table 4-1. American Institutes for Research D-5 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
LL aS aan nscA Rianne ) Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Appendix D: Stargate Operational User intertviews TABLE 3-2 - VALUE RATING DEFINITIONS FROM DIAM 58-13 MA.JOR SIGNIFICANCE - Intelligence Information Report (IIR) provided information which will alter or significantly influence national policy, perceptions, or analysis; or provided unique or timely indications and warning of impending significant foreign military or political actions having a national impact. HIGH VALUE - IIR(s) was best report to date or first report on this important topic, but did not significantly influence policy or change analyses. OF VALUE - IIR(s) provided information which supplements, updates, confirms, or aids in the interpretation of information in data bases, intelligence production, policy research and analysis, or military operations and plans; most DoD HUMINT System reporting falls into this category. LOW VALUE - IIR was not a good report because the information was not reported in a timely manner, or was of poor quaility/of little substance. Nevertheless, it satisfied some of the consumer's informational needs. NO VALUE - IIR provided no worthwhile information to support data base maintenance, intelligence production, policy research and analysis, or military operations and planning; or its information had no utility, was erroneous, or misleading. 4.0 EVALUATION SUMMARY AND COMMENTS Thirty-nine (39) of the 40 numerically evaluated, operational tasks were performed in 1994 and one in 1995, The information provided by the Star Gate RV's for each task was evaluated by staff of the tasking organization. The complete compilation of evaluated scores is presented in Table 4-1 which includes a designation of the tasking organization and, where known, a numerical designator for the individual from that organization who signed the response to the evaluation request (in some instances, this was also an evaluator). Also presented are the individual and collective scores for Accuracy (A) and Value (V) for each of the three RV's and the related average and standard deviations for the compiled scores. (Note that the total number of scoring entries for either Accuracy or Value is not equal to the maximum of 120, i-e., 3x40, since all three RV's did not participated in all tasks). Table 4-2 presents the same scoring data by tasking organization. Histograms of the scores from Table 4-1 are shown below. Note that "Accuracy" scores tend to cluster around 2's and 3's (55 of the 99 entries) while "Value" scores cluster around 3's and 4's (80 of the 1 00 entries). This is not too surprising as the nonspecific, nebulous nature of the individual task)target requests permits the RV to "free associate" and permits the evaluator to pick and choose from the RV commentary anything that he thinks "may” or "possibly" is related to his problem (and score accordingly) regardless of how much of the RVcommentary American Institutes for Research D-6 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 cy a =i vo] o>) SULT oO 2 ° ° ° elojojelolo sjelolojojols t 3 N oF (st cD [9 [OO wv ON FLO — 12 gjejejejojlejojo 2 elojojole wm pom IN aoe fem fem [ON Le oD OLIN ze hal io sd je Qjo|ojo ° e[e] jojojo]} jolojc 2 oO} jo Bs oF 109 OD 1D [09 OD ON sei ho CONTIN 69 (00 109 isp) wT oy oO OD 73 10D OWN aii Q|o 2 2] jo Qo o Qjajo} jejejejejelojojojo} joj jojo Ps NEON Ye) ie iN ONE ION ae NE [on JON () CO TN OD WLS jOD FLED Le CO 1ON tad CT PD 69 70 laj ° isd jo Sjojo|ejojo DlQJOJQlo/o lO ja lo lojolalajalajolalajalajalojol- ery sr [ OF) [UL [e9 ed 19 Lael LD [ON o> OD (O93 [TIN [09 [ sr pst [st ish poo foo wt | ivf 2A | 2v | 3A | Appendix D: Stargate Operational User Intertviews OPERATIONAL TASKS THAT HAVE Remote Viewer & Scores NUMERICAL EVALUATIONS . 5 isd ° 2 1n|O IS bed bcd bad bac alo a\o OS. lett] + wT Law a Lied bcd co OTN [a NUON — [x EN [ret (09 IN Jo w TEL lel! (8 mn = fon fers foo 60 foo fet ip ber feo st = {co In st fst ft [ot SS ili} 12) fy Oio <— c << Slt hd i = wH Lo ie] - Shs hie = E/S| ols 2 << [0 jeo¥ feo (ee) C9 jO iO {CIO 10 FO IO FT ETE EO 1O 10 10 IO [Oj |O JO jus LU LL 3 13 | 1 01813) ror? fd . 1 . . 2 - ok os ry ry ef . a . . * oo . ce a oo ol os a Ce . of os . wo 51S | 2] SAS Sy Hl | SH) SH Sal Sh] SH} SH) Gy) Gi] SH Oh] BH] OH) SD] SH} Sh] | 1) SD] BB} i] 1} 1] HB) DI OSE) IS IP OIO|O|OjO|Ojo B1G1O1B [5/5/50 JO} /5/O|S|G Ojo [O/OjO|5 [5 jojolojojojojojojojo|sis|/zi2 <r st pr it ist I ist pict is ba wt bt bo oe be le be be ee ee tit a go | |S |S] 1S5 |S Ss 1S 1 1S |S IS 1 1S IS Se Ss SS [|S 1D |S 1B 1D |B [SB 1S 1D |B 1S |S |B [SID |F [OO ® S| P1S1S O21 11 lO sR S/x SPOS FSS ERISA ISR SSeS © Q i — bod an = [S TI ST es (= hams = iN i. Pe JOR FT LP PE I J 1S STR PS FS IS JIS Fa fess [Je LS |S FS FS Les [05 [053 fo Le SS PSL [GS I IS [ed 5 +>) 2 SD [SP [OC fx 1 [- AT 100 IM 10 [OD LOIN Ix Fc St IO 1O IO HO [tT 109 (OD [OO FR. [0 {LO EN ICO I. be I 10 [OD IN aa 4 CO Ee [I 1c: it~ LS LO 1 <P Tt LO ILO it FP 1S 1D bt 10D JD 109 LEONEAN INT ICN 109 1 [09 Et It ICO ILO KO 100 pwd NUTONE TEN TNT ION ION: TNE TONTION JON TNE ENETONETON ION END TEND TENE TNT TENE TENG TENE FOND FONE TENT [ONT FON FCN TONS FONT FESO IONE TONNE H c ° 8 lou jeo IN ee TEN [09 [SP ILD [60 IB 100 10D 1D Ie FCN I wlellelalatclolalskslobsieigigtslelekgtele ~ pom free fom pm pee pee fem bem fem rom JON FONT ICE FONT ONE FONT FONG TONE TON FO [OD $099 FD 109 ED OD JOD OD TOD Pt eh ee IS et i = << Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 Appendix D: Stargate Operational User Intertviews D- ! ! oO re] wo i i - inp) iy} O10 h [st ep t [st [> pd Bs fg Th lA iy lew] fot bead Fs ajojojolojojojo glejelelojg!” jo] je co} | TiO XP PST LO OF 10 FON [LO erp wT N vv i [CO So ES olololojojalo ° slolojojeim ° ~| i oO LNSEIEN [em fe 109 er) OLIN Iw LO w O wT om oO re ° alo ejojolo al" fe BOO les] Iv en col ty Tir wt cy ory OF 109 109 [em ~* ofc eal? a S 2} | jelelolojololojo ol [2 ol jelel fai Ol ww ery CD [ON LOD co > o> ie “Tt OO - foo] io = S S\ololojo SJojolojololsla|s ole < io oD eo} [oo cD ery tr CO he [OO Ti NI (uf fo Ele wo Lu | |S ajolo Q}alolojolojajalololojolololai|a {eo ° o (‘a [0 [NN ed Prem JON FD LN FD TN fe 109 0 TN NID |= [ON ee c « ae re << : 3 St tis hs 3] boa feo Jor fer fer fon foo ——— 2 0% ja " * Els = |2\5 OHOfO/OlO JOJO j/O lO lOO lo jojo lo [ojo solo 00 {co |oo |e fro [=a fc > \2/0 21D) P| Di) D1) D} D>) D) Dy} DH} Dy} Oy} Oh} D>} DH] Sy} Dy} | DD] Dy Sy | DD = 0101010 10 [Oj 10 10 10 10 fO [0 jO [O10 0 10/0 CHO jO{O jO lO lo a SISISIZISISISISISISIS a| | lel isisieieisie B1B|& a SEINE |< |& =|Sis 10 [OO Jr jr iE PSP et IF JOD f09 JOS fo [ed | fod IST iN viii @ loo jr. ao tey Peaited pce NIN 1 {CO IN ex IN NIN te Viewer & Scores VALUATIONS Cc valuato 4(?) Score sums' No. of entries: Score suma® No. of entries= Avg scores Table 4-2 /18/9 114/94 129/94 24194 122/04 (21194 115/94 123/94 / OPERATIONAL TASKS THAT HAVE (3/95 Pd — _ — 11/3/94 5/25/9. 5/17/99. 242 24 23 23 22 merican institutes for Researc Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 —— Appendix D: Stargate Operational User Intertviews s m/s ° Pf | N fem | o . ey (a = 7) N the iS ale ‘fen 2lolols Ol [es iN Sed Ge) N oi st] isd joj is | a 2} Jololol*la| | lslelolelala|™la 2} |els () 3) IN UAT IN fe fad = (re) = 2 st P| Slelelojolg]” le ed 2Jololola}* le iS =f —~ NIN N IN 1 fer) | = Ww ajalajala Sd idu didn uu fous fuss ju DP) Bl Sy lS BD} 2] 1 Do OS |SOjOjo OlSIO}ajo ehislielis Oo 1° Tory (2) b |__ No, [1A] izati a: |S |_| 10/14 10/11 Score sums= No. of entriese No. of eniries= Avg scores Avg score» Avg scores NUMERICAL EVALUATIONS OPERATIONAL TASKS THAT HAVE American institutes for Research D-9 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5
‘oa orn OT ee na maaan ae serene earec RE Rpe ene = Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5 ee Appendix D: Stargate Operational User Intertviews may satisfy the particular measure. If the Accuracy of the information is somewhat uncertain, its Value must be vaguer still, i.e., scored lower. This presumption is supported by review of the scored "pairs" for all cases, e.g., 1 A and 1 V; only rarely does the "V" score equal or exceed the "A" score for a specific RV and target. Note further that of the 1 00 "V" scores shown on Table 4-1, there are no "1" scores*, while the 99 "A" scores include 13 "I's". Regarding the latter, a detailed review of the evaluator comments and/or the tasking suggests that the importance of these I's is less than the score would imply in all but four cases since: - the evaluator of Document 243 stated that the RV 3A score "...though vague, is probably correct." - the tasking and targets for Documents 245,247, 248, 249 and 265° concern topics widely publicized in the open media during the same. period, hence the "source" of the RV | A and 3A comments, intended or not, is suspect, and - for Documents 230, 239 and 244, the evaluator's supporting narrative is between the RV(s) and the evaluator: ~ has a very narrow information bandwidth, i.e., the RV-derived information cannot be embellished by a dialogue with the evaluator without substantially telegraphing the evaluator's needs and interests, thereby biasing any RV information subsequently derived , and - is extremely "noisy” as a result of the unidentifiable beliefs, intentions, knowledge, biases, etc. that reside in the subconsciousness of the RV(s) and/or the evaluator . As a result, the potential for self-deception on the part of the evaluator exists, i.e, he/she "reads" into the RV information a degree of validity that in truth is based upon fragmentary, generalized information and which may have little real applicability to his/her problem. The relevant question in the overall evaluation process is who and what is being evaluated, i.e., is the score a measure of the RV's paranormal capabilities or of the evaluators views, beliefs and concepts? *The significance of this omission is further enhanced if one assumes that the evaluators were familiar with the definitions in Table 3-2 since even those 11 instances scored as #2 ("High value") merely require that the input be the "best report to date or first report on this important topic, but [it] did not significantly influence policy or change analyses." “(U) The evaluation of Document 265 is actually a second evaluation of the same RV inputs provided many months after the first evaluation of Document 248 and probably done by a different evaluator. American Institutes for Research ~— D-10 Approved For Release 2002/05/22 : CIA-RDP96-00791R000200180005-5