= Approved For Release 2000/08/10 : CIA-RDP96-00789R003800020001-8 — “A %y Seehija B. (S/NF/SS-2) EVALUATION PROCEDURES: 1. (S/NF/SS-2) Operational Phase: (S/NF/SS-2) The Project Manager will assess the value of the operational phase of the SUN STREAK program by requiring al? data consumers (i.e., operational task originators) to furnish him with data accuracy and data utility assessments. Table 1 illustrates the general format and assessment scale (0-3) for this evaluation. Specific data categories may vary depending on the nature of the task or on specific objectives; however, the overall approach will remain the same. (S/NF/SS-2) The Project Manager will enter results of the accuracy and utility assessment into the project data base. He will make periodic reviews (quarterly) of this data to assess overall effectiveness. (S/NF/SS-2) The Project Manager, in coordination with the task originators and the Intelligence Community Task Coordinating Group, wil] develop an additional measure of program value. This measurement (overall program value) is designed to measure the net worth or total contribution of the task in relation to other tasks and to the overall] intelligence mission. Data accuracy and utility may be high for a particular consumer, but the results may not have a significant overall] impact. This program value scale is similar to the utility scale; however it is based on specific measures of over-all benefit. This value scale is as follows: “ Program Value Scale 0 - Little or no over-all program value, 1 - Some program value (e.g., helped refine estimates), 2 - Moderate value (e.g., identified new data, narrowed down possibilities), 3 - High value (e.g., led to significant cost savings, identified critical high priority S&T or operational activities), 4 - Exceptional value (e.g., predicted major events/activities, located lost/missing resources or personnel). The value evaluation will be used in conjunction with other program evaluation parameters (accuracy, utility) for determining program accom- plishments and for defining future program directions. art NOT RELEASABLE TO FOREIGN NATIONALS 3 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800020001-8 ht-,
I dTav. *UOFJUUOJUT JIVA4UdUT Y1IITT . Al eazsavlol pus syuswoye eTQvuo jeu anbyun snond}queun yQ IA vouspuddsorIOs poop * * * 7 8 8 8 Fe ee WI [vIxd = € VOFIeuUAOZJUT YIaIIOIuUy suos jang ‘HZuzpysj eu squawara TBA9AV8 YATRA adUspUuddsaaaOD pooy °° * * * * & © & © © ew poo) - Zz *a}}s J9Z1eQ ayy passacoe A] qeqoad sey adsanos 338d} put 8I[NG9l poxyw 0} Jawiozy 242 JO YBnoua Ynq ‘squawata J28IAOSUT PUB 3393103 JO BANAXPW °° * * * * * * YQTmH YDBQUOD 4_JIS -od *AlOquUUTdXa JTIVGS * * ° * * wdouepUuodseda0d 01IITT -<8 9TBI8 ADGANDIV 9Yyy AOZ BUdT TUT; -3p aq Jouus) A 2 I 0 oS N - o AVFLFIM 11¥AMO (s ce © --o- (°330 ‘eufd Cc) Cj CJ Cj (_} , | ‘savItiqisuodsaz ‘suoy ope ‘guoyjd}assep 12} s4GQ) UOT QeWwIojsUy 43} Bus $d (Ss (*9]29° ‘Sup10770Qu INI1qd ‘adei0j6 mad ‘Suz aly attsstw ‘suzy 3eay avaqzoAd) SIFIJAPIOV DPJ Jods auena fou (s (°930 ‘3up 98a] suodeaa ‘Aqyay Ge aye ‘Ayyaypqoe [easu ‘squew -asow door ‘3382036 ‘uo}3 622 -syTuywpe ‘uoyjonpoad ‘yoages 02) adueTque yadivqy [v1s@ (Ss =) (*a9a ‘saazyjo ‘saT Tsu ‘syueq ‘siaqgndwoa ‘seuua qe) sjuowea[s speuueW 91 B98- 11 MaS (Ss (aaa ‘spratzage S‘sau}[ peoat}ea ‘sx20p ‘sopys ‘sduypy yng ‘sa73fS) Suaua,e apewugu a[eos-a83g71 (S LL CoO Cf U 0 0 U CIA-RDP96-00789R008800020001-8 U0 U OU U0 U Uv C) a) CI (990 ‘sojuM ‘upuadey) aga -d}aosep aTes0, pwoyydeasy (s fe) CO) U roved For Release 2000/08/10 -O} O U U “OH O U U -O OU DU U CI Appr @1qQ82} ] ddy uAoUXUL quat[aoxg poo S{[Nsey poxIW 9 uapuodsea1109 ION yata OTIIF1 Syoweyuod) 2215 A a OVENIOV “TEyAa eM Psaqyuqns 242 yo AdBvaNddB |YA 02 SB saxog BuzyMOTIOJ |YQ YdaY4I Yseoy{d ‘uo; AENnTerna AreuNS oy 104 (A. rs a er Aemenee re teen far
F - Approved For Release 2000/08/10 : CIA-RDP96-00789R003800020001-8 =sn6nnt= 2. (S/NF/SS-2) Training and Operational Qualifying Phase: (C) There are two other aspects of this program that will require evalu- ation by the Project Manager or by his representative; the basic training phase and the operational qualifying (or operational training) phase. (S/NF/SS-2) The basic training phase consists of a large variety of easy-to-verify training tasks that are designed for various stages of proficiency. Overall evaluations will be based on the accuracy scale used for operational projects and on an additional quantitative procedure. The accuracy scale will permit assessment of an individual's progress through the six training stages. The quantitative procedure (i.e., concept/element analysis) will allow assessments of degree of accuracy and will provide the basis for determining what general classes of operational projects a viewer can best perform. (S/NF/SS-2) The operational qualifying phase follows satisfactory completion of the training phase. This phase consists of tasks that simulate operational projects but differ in that ground truth can be readily determined, and there are no operational consumers. The main purpose of this phase is to systematically quantify the operational readiness of each viewer for a wide variety of potential operational projects. Data from this phase will also help identify the best types of operational tasks for remote viewing (RV) applications. (S/NF/SS-2) Each RV source will perform several hundred operational qualifying tasks per year. These tasks will include representative examples from DIA's intelligence production codes and from other potential operational requirements. Evaluation will be performed by the Project Manager or his representative and will be based on the accuracy scale used for operational projects. In addition, accurate estimates of operational reliability and utility will be made using quantitative methods (i.e., concept/element analysis, pattern evaluation). These evaluation procedures will require extensive use of a dedicated data base management system for storing large amounts of information and for performing appropriate analyses. (S/NF/SS-2) It is anticipated that a limited operational capability will be achieved by mid-1987. Some operational projects will be initiated in FY 1986; however, the type of task accepted will depend on the nature of the proposed task and on capability levels of available RV sources. =gnehit= NOT RELEASABLE TO FOREIGN NATIONALS 5 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800020001-8 —————————— hl
= r , | Approved For Release 2onPPH OTe HTD P 96-00789R003800020001-8 - 24 hr. availability of people who can perform operational tasks (viewers, interviewers); - Resource allocation authority (people required, specific assignments, travel); - Procedures for interface with the user community (for intelligence and outside); - Procedures for interface with others (i.e., VO, HUMINT Committee); - Methods/procedures for transmitting RY data to users (i.e., specfal courfer); - Procedures for preliminary evaluation of data and decision-making for follow-on actions; - Procedures for maintaining security controls, and safeguarding viewer identity; These and others would be constantly refined to improve over-all responsiveness and effectiveness of operational activities. Approved For Release 2000/QERER SHR TOS-00789R003800020001-8
Approved For Release sooth a compte ss-00789R003800020001-8 (2) (S/NF/WN) Methodology: A consistent and scientifically-valid approach to methods and procedures for active conduct of operational sessfons will be developed. These will be established as firm project requirements, and any exception (4,e., due to error, unforseen circumstances) will be clearly noted. The user community personnel involved in operational tasks (f.e., as observers, or for data analysis) will -be informed of al] methods/procedures in advance of the conduct of any operational task. This is necessary for operational consistency, for data evaluation, and for making decisions on how best to apply the RV data. Specific methods/procedures will be devel oped -for each operational task in accordance with the nature of the task (i.e., QRT, long-term), type of targeting methods required, and the type of training program used for developing the viewers’ capability. (3) Records (U): (S/NF) Detailed records will be maintained for a1] R¥ operational sessions. Specific data will include people involved, chronology of events, data provided to the viewer, targeting methods used, degree of knowledge of , target information by those involved in the session, or any other issue pertinent to performing the session and evaluating the results. All these factors will be identified and clear documentation procedures will be established prior to initiation of any operational project. The actual RV data will also be carefully recorded and controlled, and will be entered into the appropriate data base. Data format will include verbal records (tape recordings) as well as raw data (written description, sketches). These records will be stored in such a way as to protect identify of the user and the viewers involved, and will be subject to all other security controls estanpisnsdeen ear prerladeASGORIOBYPO"AR CREEP SE S00 TASK cSabOdId801-8 avatlable to any authorized individual or project review group as necessary,
epemeeemnnnneninanerenpennnsmnene eg ‘Approved For Release 200K FEE £4 FEAP96-00789R003800020001-8 | c. Evaluation of Data (U): * (S/NF/WN) A data package with pertinent raw RV data, and a summarized version performed by project personnel, will be returned to the task originator and the IC Task Coordinating Group via DIA/DT. All session methods and conditions will be included with this data package. The task originator wilt provide an evaluation of the data according to pre-established criteria. The originator will return this evaluation to DIA/DT along with sufficient background and feed-back data so that the basis of the evaluation can be clearly understood. Any difference of opinion will be discussed with the task originator and noted in the project records. In cases where evaluation fs only partial (i.e., for tasks that can't be immediately evaluated) the file will be kept active until such e time when sufficient data becomes avaflable. The task originator will also indicate how the data could be (or was) utilized, and will describe the impact such data had on resolution of the specific problem of interest. Additional data analysis will be performed on the raw data in the growing data base to evaluate trends, patterns, statistical implications, operational constraints, reliability/utility, and other aspects pertinent to data utlization. d. Reporting of Results (U): (S/NF) In addition to data reports provided to the users, over-all] project reports will also be prepared on a periodic basis. They will contain general information on types of tasks, over-all results, summaries of task originator evaluations, statisitics/trends, and other relevant data as Approved For Release 2000/08/10 : Spore /eeRobesnoozoeet®
s Approved For Release 200 RATE 47FpP96-00789R003800020001-8 required by DIA/DT. These reports would not contain sensitive data; however such data could be provided separately on a strict need-to-know basis pending appropriate data release approval. 7. Process Flow Chart (U): ~ (C/NF) All project procedures and operational activity identified in te ae the section for tratning~and for operational projects will be developed in detail in a Project Operations Manual. A process flow chart will also be prepared. - Approved For Release 200M REI OE MER b> 96-00789R003800020001-8