Approved For Release 2000/6WEsL Ai $ SHPEE 1b788R001800050001-8 (penutzuoo) G-y eTaeL CPL EP LEP EEE v2 2 weet ere bt Epo jor a | a da a ps for Soa a Od ne nd ce ee es a a a Oa a a er dry ye te pep ree Oa a a ae De Oe a a a a PE) Ee re er leet ae fe | Zz°Z | eseisay pe ysitey NOLLVIIVAD Id TYOSNVEL qdeoucg yd frosussy q ?8SBID uo Tsses tuotsses fo 44UZue7 T86T Joqueseq FT 1983Bq S93 BUT pio0cg $ Sut es1BL_ Jo poyzoR TOWEL pug Zoo# i1OMOTA BYUCION ap opusuts, 2 qearey, OP:2T TeUTL Upseq 39 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 204g/ Dd CL: AS SHFHS61D0788R001800050001-8 (Pepaptsuos) = G-V eTqBL Tey Peet ET PEEP LE] erceeee 2 aa agersay peiystay . m fer fer [rx for fs fe fe ]o [s | > fe] ze] re] o} os Oa a ee pete} (ete pep rrr ete} { ff po ete} (spe pep mer ete} (ste pep mre OE ete; P RP pep etet gps pepe FS eR ee NOILVMTIVAZ LATYOSNVEL a SSBID tuo Tssag :uo;Tsseg Fo 773ue7 I86I 1aquesaq FT 1:97 eq S9}BUTPIceD 3 3utz.eBrwy Jo poyzon LeutL pug z00# TLOMOTA nfreyequng $7e21By oF: 21 [amTL up~zeg 40 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 20d G4 AG SafHds100788R001800050001-8 (U) Two types of overall assessment were chosen to emphasize the versatility of the evaluation procedure, (1) a simple rank ordering based on weighted average scores, and (2) a concept-by-concept, non-parametric, statistical technique. Table A-6 shows the results of the first method, the rank ordering. For convenience, the correct matches are underlined. Table A-6 (U) A RANK ORDERING OF WEIGHTED AVERAGES (U) Fernando Session/Target Inverness Hangé de Noronha Punkahar ju te ” Scores computed with non-uniform target relevance factors. (U) From Table A-6, we see that there were 3 first-place matches and 1 fourth-place match. The probability of obtaining 3 of 4 possible first-place matches from chance fluctuations alone are less than 0.051. The point spread between the best match (Inverness) and the worst match (Punkaharju) are in qualitative agreement with a subjective "first look" at the quality of the transcripts as well. (U) The second analysis determines the significance of the difference between the correct concept/target matches and a control set of matches. All concept/target matches that are not the correct matches act as an internal control set. To avoid any invalid assumptions as to the correct parent distribution, a non-parametric statistical test, the Mann-Whitney U-Test, was chosen for the analysis.’ 41 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release 20d INGd-AS SAF E20788R001800050001-8 (U) It is beyond the scope of this report to review the details of the Mann-Witney U-Test; thus, only the results are quoted here. The probability that the set of correct concept/target matches is statistically indistinguishable from the control concept/target matches is less than 0.071, (U) There are a number of additional statistical procedures that could be used to analyze the results of this evaluation technique. The two cited above, however, represent a spread in complexity that demonstrates the internal consistency of the basic evaluation procedure. With only four similar RV sessions, the evaluation technique nearly reached the 0.05 level of statistical significance with each of the two statistical procedures, a result indicating a successful outcome with regard to the overall assessment procedure, 42 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8
Approved For Release sot GL ARSE o7ssR001800050001-8 Appendix B (U) SUMMARY OF EVALUATION TECHNIQUE (U) Step Action 1 Task coordinator defines the evaluation goal. He/she identifies target elements and assigns target element relevance factors as appropriate. 2 Analyst conceptualizes responses and prepares an RV assessment sheet for each response. 3 Repeated concepts are noted in the "D' column. 4 Copies of the sheets from Item 2 are made; one for each possible target used in the analysis. FOR EACH POSSIBLE RESPONSE/TARGET COMBINATION: 5 Identify a target element for each concept not marked in the "D4 column; mark a 1 in the "p't column and write the target element and its relevance factor from Step 1 in the appropriate columns. (Write O and blanks if no element can be found.) 6 Using Table 3, assign a quality rating for all present (p = 1) concept/element combinations. 7 Compute the score as follows: a. Calculate relevance (R) X quality (Q) b. Convert R X Q to an integer between O and 14 using the conversion table provided. 8 Enter the number of concepts that obtained each possible score in the space provided. 9 Calculate the weighted average using: A. = 0.357 [ £. VE. s./znf, VF, J J J J J where: S, is the score and f, is the number of concepts that obtained score 4 = 0,1, 2, «e+, 14 10 For each response, rank order the weighted averages. 43 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8