Approved For Release PT Loe iOVRRDP96-00789R000700260002-5 4 oe a n deja wr Wore calling Ate they are known x However, y are included in +he=deta. YWhre Y evaluation for this project since they tend to provide confidence that other (as yet unknown) details in the data may be correct. (S/NF/SS-2) <A more sophisticated analysis methodology has been recently developed by SRI for use in evaluating complex projects. This methods’ main advantage is that it allows quantified ds wx FO. stated 1 An RWhr estimates to be made for each and every data element ,with respect to both actual target existence and importance. This technique is currently being examined for use in the SUN STREAK program and has been applied to a few projects. However, it is a labor intensive technique that will probably he used only for select high interest projects in the future. (S/NF/SS-2) Most of the other SUN STREAK project types do not require a complex analysis methodology. For example, due to the nature of what type of data is desired (and availability of collection assets that can be cued), most of the CTy, CN or predictive projects where ground truth is known can be evaluated in a “black or white” manner. The sata even if not acted upon a , either correlated with the subsequent location of the fugitive or ship, or it oe The event predicted did, or did not, happen. Thus, overall results for many of these projects are simply a matter of counting hits and misses. Hit ratios or percentages of hits/misses form the basis of overall data correlations made in this report for these type of projects. Additional data analysis is jot course performed to determine how eer Approved For Release 2000/08/08 : CIA-RDP96-00789R000700260002-5
Approved For Release 2000/08/08 : CIA-RDP96-00789R000700260002-5 close to ground truth the data actually was. This may be of prodects of er aku - eareh—or—predictiocna. value in understanding how to conduct future s 3. EVALUATION RESULTS (U) (S/NF/SS-3) Overall data correlations for all SUN STREAK operational and operational simulation projects performed since ~ 1986 are shown in figure 5. These results were obtained by averaging the data entered on the summary data evaluation sheeta for each project primarily for_two data categories (i.e. large ob avd function) , scale and small acaley. The top lines on the bar charts reflect data averages obtained from the proven or experienced viewers, For some projects, especially some of the CN and CI projects, the distinction between large scale and small scale is not clearcut ; Lurthemeore ead this differentiation may not be too important. For the , prck , predictive category and most of the CN a, data correlations were based on a hit/mias calculation. (S/NF/SS-3) Figure 5 indicates that, on the average, data from proven SUN STREAK viewers for S/T projects will tend to have a 20 percent to 30 percent correlation with ground truth for small scale targets, and a 30 percent to 50 percent correlation with ground truth for large scale target features. Likewise, for CT or CN projects, about 20 percent to 50 percent of SUN STREAK data would be expected to correlate with ground truth. Caution must be exercised in interpreting this data, however, since the data Joon ap YWser Cote gomtey . base with mary ground truth is quite low, oe a 9 Approved For Release 2000/08/08 : CIA-RDP96-00789R000700260002-5
Approved For Release 2000/08/Q2se@m@RE? 96 -00789R000700260002-5 pase—tecetaccemall Yor CI and Document (reading projecta, SUN STREAK data shows a 40 {2 perce nt to 60 percent correlation level with ground truth: , Sethe projects athe Pocument Reading nN projects were, however, carefully isolated and Ya a known or designated location. Predictive data of the complex event type wore (e.g., political/military situations) ommbeng—term)- shows a plow data correlation (i.e., reliability) of about 10 percent or less. (S/NF/SS-3) If all SUN STREAK projects are averaged together, as shown in figure 6, data correlation would range from about 20 percent for small scale aspects to about 40 percent for large scale aspects. While “averaging” such data may indicate overall results in the long run, such averaging tends to washout those results that have singular high merit, such as the identification several months in advance of a specific area in the U.S. where a fugitive was later found. In this case, SUN STREAK data was not angus rthelere, wr Ante. Fatimanmpeere! CAO dina Ye acted upon; fortunately, the fugitive was pabducted there | nec. Ly BS ner Ao cet Aour serena fsci (S/NF/SS-3) Another way of considering overall SUN STREAK project data correlation is to consider only the proven viewers. This data is shown in figure ig for times when these experienced viewers received a 2 or 2+ in the numerical ratings assigned to their data correlations. Only two types of data are presented here; S/T, and Rersonality data L€ obtained from the various CT, CN and CI projects. For S/T projects, proven viewers would he date Contledt tow expected to receive a high (i.e. approx. 70 percent) rating on asin 10 Approved For Release 2000/08/08 : CIA-RDP96-00789R000700260002-5
Approved For Release 2000/08/08 : CIA-RDP96-00789R000700260002-5 about 20 percent of the S/T projects attempted. For Bersonality pods damm (i.e., background, state-of-the-health, specific activities), around 50 percent to 60 percent of the projects would yield high results. Essentially, this chart indicates certain strengths/weaknesses of the present SUN STREAK staff and Suggests more projects foreign or CI target personalities are warranted. ll \ Approved For Release 2000/08/08 : CIA-RDP96-00789R000700260002-5
Rae Approved For Release 2000/08/08 : CiitRbPse-00789R000700260002-5 Lot FINDINGS (VU) peu (S/NF/SS-3) “fhe overall data correlations provided in the A nae previous section, .ekekemeh in some instances bgge a low overall phe AranDis org ? average, ave, unique enough to warrant further attention and continudtes SUN STREAK activity. Even in the lowest reliability case (i.e., predictive), identification of even one important future event out of ten could in fact be highly significant ta jet ul U cost or life saving. ThetS averages arse do not do justice fer tO the single unique cases that cost little to act upon, as in the case of the fugitive location cited in section 3. (S/NF/SS-3) Specific findings that resulted from in-depth review of the entire SUN STREAK data base include: \ fs) ndividuals performance correlates with project type. This observation has already assisted in better task/person matching, and overall data correlations would be expected to improve in the future. o SUN STREAK has a distinct potential for direct contribution to certain CI, CN and CT cases, as born out by De hl lt ie i Ren re Semen aaa specific instances over the past two years. o Predictive data is promising under certain conditions such as near term events or situations that do not involve complex interactions. ore Approved For Release 2000/08/08 : CIA-RDP96-00789R000700260002-5