Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9 REVIEW OF PROJECT PROCEDURES Date of Fublication 5 April 1991 This is a Department of Defense Intelligence Document prepared by the Technology Assessment and Support Office, Directorate for Scientific and Technical Intelligence, Defense Intelligence Agency. FREFARED BY Technology Assessment and Support Office (DT-S) NOT RELEASABLE TO FOREIGN NATIONALS REPRODUCTION REQUIRES APPROVAL OF ORIGINATOR OR HIGHER DOD AUTHORITY. FURTHER DISSEMINATION ONLY AS DIRECTED BY DT OR HIGHER DOD AUTHORITY. CLASSIFIED BY: DIA/DT DECLASSIFY ON: QADR NOFORN LIMDIS Approved For oe) |'\\0 1: | aiaailcaaiaaeaali
Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9 UNCLASSIFIED Page I LO 1 TY, SCOPE. Cece s ewer eee e ene nese erateeutneunnveeeuunan 1 TIT. SUMMARY «cscs seen an nan seueneasasuaveuwenunnunn « il TV, BACKGROUND. cc eee ece een essceuneacectervuuuenannccs i V. SHORTFALLS/REVISIONS. 00... cc cue ceceesecucueueucuce 2 APPENDIX A. EVALUATION FLOW CHART BE. REPORT FORMAT C. SUMMARY OF INFORMATION FORMAT UNCLASSIFIED Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9
Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9 UNCLASSIFIED T. (WU) PURPOSE: CU) Furpose af this report is to review initial STAR GATE procedures and to provide recommendations for revisions based on recent experience including DESERT STORM tasking. (S/NF This report examines data development, evaluation and assessment procedures for data generated by STAR GATE sources. TIT. (WU) SUMMARY: A. (8/NF) Lessons learned from recent DESERT STORM tasking have led to several immadiate changes in STAR GATE procedures as discussed in Section IV, Some recommendations for future revisions are also identified. B, (S/NF} Evaluations of future STAR GATE reporting will be prepared in accordance with the criteria and provisions of DIAM 58-13, Subject: The Department of Defense HUMINT Management System, dated 28 March 1988, TV. (U) BACKGROUND: A. (S/NF) In December 1990 and January 1991, initial efforts were made to provide an effective and viable system to evaluate STAR GATE reporting. These procedures area illustrated in Figure 1. The ~ INCLASSIFIED -l- Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9
Approved For Release 2002/05/42 : GLA RDPI6-00789R002900290001-9 initial proposals resulted in the implementation of procedures that attempted to assess both the utility and the value of STAR GATE reporting. RB. (S/NF) During the course of an off-site conference held on 7 and 8 March 1991 at Fort Dietrich, senior analysts from DT-SA, DT-3k, and DT-S, identified shortfalls in current evaluation procedures and prescribed revisions that would allow for the implementation of an objective and workable system. The senior analysts believed that assessments relating to the accuracy of STAR GATE reporting could best be addressed by applying applicable scientific methodolagies to STAR GATE material. Procedures already exist and can be implemented available project ADF methods. Specific accuracy assessments would be determined by intelligence analysts in conjunction with appropriate outside scientific review. The same analysts were of the opinion that the intelligence value or utility of STAR GATE reporting could best be determined by select intelligence analysts directly involved with specific task areas. Cc. (U) What follows are the results of a comprehensive group effort conducted to identify procedural shortfalls and recommend the implementation of the following revisions. V. (S/NF/SG/LIMDIS) SHORTFALLS AND REVISTONS: ~ SHORTFALL: Need to identify action officers with whom DT-3B can deal directly to expedite the evaluation process and insure that such evaluations are accomplished in the mast successful manner. einen (0 FORN/LIMDIS STAR GATE , Approved For Release 2002/05/17% CIA-RDP96-00789R002900290001-9
Approved For Release 20Qgé@éiddistalgibdigRDP96-00789R002900290001-9 SG1J analytical evaluation process. Direct involvement with these (Cand other) scientific analysts should help in better task definition and in identifying ways for improving potential utility. ~ SHORTFALL: Procedures to date appear burdensome and need streamlining. ~ REVISION: Proceduras developed by the R&D community should be used for assessing the accuracy of project reporting. The intelligence value of STAR GATE reporting must be determined by intelligence community analysts working from an all source intelligence perspective and environment. This should be based on an agreed-upon definition of utility and what target data would be required to satisfy the utility criteria. ~ SHORTFALL: Need to assess whether our quantifiable answer categories would be expanded to reflect the range of uncertainty when one is evaluating a DT-S report. ~ REVISTON: Specificity in the judicious selection of mission requirements will go a lang way towards streamlining the process. Target selection must be based on targets against which STAR GATE has Quimpuieeeta NOFORN/LIMDIS STAR GATE -3- Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9
Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9 eo a favorable track record. To this extent, limits to STAR GATE data (or project type), need to be provided in advance to the tasking elements. ~ SHORTFALL: Need to re-accomplish certain original tasking, updating it as necessary to the current situation. - REVISTON: Tasking should be levied within specific time windows and not be left open-ended. Requirement originators should be reminded of their responsibility to update requirements and changes of collection emphasis. ~ SHORTFALL Senior analysts need ta insure that the intelligence avaluations (ia, utility assessments) of project reporting are timely, objective, substantive and prepared in consonance with established intelligence community procedures. ~ REVISION: STAR GATE reporting needs to be evaluated in consonance with the DIA HUMINT evaluation scheme paraphrased substantially as follows: RATING VALUE Of Major Significance Having Impact on National Policy and Decision-Making Of High Value Best Effort - First Report in a Field of Endeavor Of Value Supplements ~ Updates - Confirms Known Data cememienieninh|()F ORN/LIMDIS STAR GATE -4- Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9
Approved For Release abe paldininie A-RDP96-00789R002900290001 -9 Of Low Value . Foor Quality - Not Timely Of No Value Erroneous ~ Misleading There is another consideration. Proper assessment procedures for either accuracy or utility require that criteria for evaluation be specific in advance. Consequently, it should be appropriate for a candidate user (or task originator) to clearly define what type of data would have potential utility, and how it would be used if provided and how it would be evaluated. This type of data could include specific essential elements of information (EET) and required timing. & troublesome problem may be encountered if the specific EEI (if posed as a question) provides too much background information for the source that could adversely influence session results. To avoid such difficulty, it may be necessary to conduct the session in saver al phases, time permitting. This issue is not easy to resolve and remains as a future action item. (A basic review of this overall issue can be found in DT-S-1010-5, 13 Dec 1990, Evaluation Methods). - SHORTFALL: Need to reconsider acceptance of tasking based on availability and timeliness of known ground truth. - REVISION: The STAR GATE collection management system must be galactive in its acceptance of requirements for which there is a likelihood of learning ground truth within a reasonable amount of time. A procedure needs to be established that allows for rapid follow-up for task refinement. emg ORN/LIMDIS STAR GATE -5- Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9
Approved For Release 2002/0 ot : CIA BDP96-00789R002900290001-9 - SHORTFALL: Multiple source reporting in a combined summary format 15 confusing precludes affixing responsibility to the reporting. ~ REVISION: The principle of “one source one report" must be strictly adhered to so that reporting may be attributable toa a specific source. Consequently, data from any source will be reported separately or in @asily identifiable and separated sections of an averall summary report. ~ SHORTFALL: Current STAR GATE reporting is inconsistent with other established HUMINT reporting formats. - REVISION: Reports need to be prepared in accordance with the criteria used in the preparation of Intelligence Information Reports (IIR‘s). Each report will bear a numerical source designator that will protect the identity of the source. An example of the reporting format is at Figure &. A Summary of Information (SOI) will be prepared by the source. It will reflect aproject number, source number, session number, and date of report. The SOI will be attached as an addendum to the formal report. An example of an SOI is at Figure 3 -~ SHORTFALL: STAR GATE predictions are rarely accurate. ~ REVISTON: Predictive reporting is indeed the project's weakest feature. The results of searches are conducted at a slightly higher rate of success. Historically, the project’s track record equmeupnny |) FORN/LIMDIS STAR GATE -6- Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9
Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9 aqui reveals are@latively high rate of success when mission tasking is targeted against fixed, static targets located in past or current time Windows, Consequently, project tasking should be prioritized according to estimated success probability. ~ SHORTFALL: STAR GATE project reporting should nat stand alone, - REVISION: Senior analysts need to insure that STAR GATE is an integral part of the all-source collection management system, a key tip-off player acting in concert with other intelligence disciplines. ~ SHORTFALL: Highlights of singularly outstanding reporting should be recorded and made readily retrievable. ~ REVISION: 4 “Kudo File" will be established to facilitate the retrieval of examples of high quality raporting. Factors that may have contributed to these results will be identified whenever possible. eiunniniemiontinn|()FORN/LIMDIS STAR’ GATE -7~ Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9
T ounbl4 WY XIONIddY & © & € Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9 | SCOT ER DATA DEVELOPMENT, EVALUATION AND ASSESSMENT SANS OOPS iad | O N See cen, C6487.06UP Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9
Approved For Release 2002/05/1% sClA;RIPP96-00789R002900290001-9 PROT E STAR GATE SOURCE NCI EEK DATE OF SESSION: DATE OF REROPR Ts SESSION START s SS TON END: 1. (S/NF OY BACK GRQUIAD s ‘ew (S/NFY ORIGINATOR REQUIREMENT s ac (S/NF > SOURCE TAGE s 4 CE / NF D OFS CORP TCER COPMENTS a Gddress incleamenci@s. Include any info not contained in Summary of Information (sor) pravided by source. a (S/N) EVALUATION: Erie mumer ical Lcnmator TAW DIAM Sa- bw (S/NF) SEARCH EVALUATION: Not Releasable to for@ign nationals (ED BYs DTA/DT JECLASSTIEY ONG OADIN Limited distribution anime |) (3 OPO AL TMD ES APPENDIX Q Figure 2 Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9
Approved For Release 20a FoclA-RDP96-00789R002900290001-9 X SUMMARY THE SOI 19 ATTACHED TC PRCT EL SOURCE s OF INFORMAT LCN JET OF APPENDIX C; Figure 3 Ts re & PARLOR Sh Cho mRMA Cel cL heels Approved For Release 2002/05/17 : CIA-RDP96-00789R002900290001-9