',/ 

FAA-A.."!-82-2 

A GENERfC ~!O:>EL FOR EVALUATIG~l OF THE FEDER.IIL AVU.TIO:t( 

AD~!I~ISTRATION AIR TR..'\FFIC CC:.O"TROL SPECIALIST TRAINlNG PROGRA!-:S 

J~mes () ... Boon~ 

Civil Al:OT')$edical Inst.it.ut.£ 

Federal Aviation Ad~inistration 

Ok~chcmi! Cit:y, Oklaho!:l;: 

Y.arch 1982 

Do-:ur..r-nt is •. n.".a:U;:;b1c tc the p< olic t'h.rougb the 

~:.; t :on2~ Tecl"-.n:: c.?l Info~.a-r ion Sen.~ :icc., 

Springfield, Virgini~ 22161 

Prepared for 

t;.s. DEPAR!!'!D<"T OF TRANSPORTA"i'!VN 

Fede~al Aviztion ACminis~rotion 

Office of Aviation Y~rlicine 

Yashington, D.c. 20591 
NOTICE 

This document is disseminated under the sponsorsnlp of th~ 

Departcent of Transportation in the interest of info~atiorr 

exchange. Tne United States Government assumes no lizbility 

for its contents or use thereof. 
T echnica\ Report Documentation P":lge 

2. Government Acce""'on No. Rec:.,.ent's ~ota!og No. 

I 4. T,t\e a!"'d Subtitle Repo•t Dc•e: 

March 1982 JA GE:Ol""ERIC MODEL FOR EVALUATION OF THE FEDERAL 

iAVlATION ADMINISTRATION AIR TRAFFIC CO~~ROL SPECIALIST 

,TRAINING PROGRAMS 

I 6. 

r;--;:-:;-:-:;-:;------------------------------il8. l~~~~~~~4 Performing Orqan•:ot;on Repoct No. 

i 7. Author/~} 

Performing Orgt:tH:ZOt!On Code 

IJames 0. Boone 

! 9. Performing Organi zction Nome and Address 

!FAA Civil Aeromedical Institu~e 

IP.O. Box 25082 

10klahoma City, Oklahoma 73125 

110. Wock Unot No. (TRA!S~ 

I 111. Cor,tro;:et o~ Grunt No. 

t 13. Type <;!f Rtoport ond Pet~od Cco.vered 

J 

~~------~--~~------------------__, 12. Sponsoring A;ency Nome and Address 

]Office of Aviation Medicine 

!Federal Aviation Administration 

!soo Independence Avenue, s.w. 

!washington, D.C. 20591 

115. Supplementory Notes 

I 

!Work was performed under Tasks !L~-C-80/81-PSY-87. 

' I 

l 16. A~srraet 

114 ~oonsori .. g Agency Code 

' 

'j. The Systems Analysis Research Unit at the Civil Aeromedical Ir..stitute (CA..lwfi) has 

developed a generic model for Federal Aviation Administration (F~~) Academy training 

jprogram evaluation. The model will serve as a basis for integrating the total data 

base into a common format across all training programs. The model consists of four 

components: (1) design, (2) implementation, (3) formative, and (4) sumnative 

evaluation. Design evaluation is an assessment of the co~preheneive i~plementatio~ 

plan; implementation evaluation is a determination that the plan is completely and 

accurately implewented according to prescription; formative evaluaticn is ~ continual 

~onitoring of the program to keep the process reliabl~, stable, and on track; and 

summative evaluation monitors the product of the trai~iug program. The design 

evaluation relies on the task, knowledge, and skills analysis anC the docLunents in 

the impleme~tation plan~ The implementation evaluation makes use of the data from 

!frequent status studies. Formative and summative evaluations ~~e use of stetistics 

\and mathematical modeling, primarily linear regression models, to ~onitcr the ~recess and products of the programs and to estimate and determine the impact of cb~itges nade 

to the programs. 

17. KeyWords 

Progr~ Evaluation 

Math !1odeling 

Training 

Air Traffic Control 

~ iS. Oistribu~ion Statement I Document is available to the public 

i through the National Tech~ical 

Service, Springfield, Virginia 

Information 

22161. 

hl;-;9;-. -s;-~-:--u-:,,:-,.-,C,-Io-:•-•"if-:. ;-{o-;f-:,.,;-,-:,-:,.,._=:::;;)----;-~ c;:ro;;_<s.'===.,:=:,-:c=;l-:o:-••:,i.,-f.-:(-:o;-f :;,.,:-:,:-,-:p::oge::::;)----,, 'i2co:,_-N;;;o;:-_-:o:;fc;p;::oges 22. ?riee 

iUnclassified Unclassified I 29 

Form DOT F 1700.7 iS-72l Res>roduction of eomJ)leteci P'age euthoriz.ed 

i 
J\CL"lOI-ILEDGENENTS 

Acknowledgements are given to Lelnnd Page for provicing Figures 2, 3. 

4, 5, and 6, to Allan VanDeventer and Linda RitchiP for prO'!iding the r"port 

fonnz.ts in Appendices A and B, and to Jo Ann Steen for thE> prep3ration of 

this manuscript. 

ii 
A GENERIC MODEL FOR EVALUATION OF THE FEDERAL AVIATION 

ADMINISTRATION AIR TRAFFIC CONTROL SPECIALIST TRAINING PROGRAHS 

I. Introduction. 

In a large training institution such as the Federal Aviation 

Administration (FAA) Academy, several independent training programs operate 

simultaneously. As new technology becomes available for training, 

especially in the computer field, new training methods are frequer.tly 

implemented. The new simulation facility for radar Air Traffic Control 

Specialist (ATCS) training and the new PLATO computer-based instructiQn 

system are examples of these advances. It is redundant and incoherent to 

develop a new program evaluation for each new development in ATCS training 

methods. Consequently the Systems Analysis Research Unit at the Civil 

Aeromedical Institute (CAM!) has developed a generic program evaluation 

monel for Academy training programs. While the ATCS training programs were 

the primary aim of the model, it is appropriate for Airway Facility or 

Flight Standards training programs. The generic model allows research at 

CAM! on Academy programs to be integrated into our total systems approach by 

making specific application of the generic model to any new Academy 

development. By consistent application of the generic model, the data 

colle~ted on programs will be ~ompatible with our continuing data base and 

offer a means of expanding our total picture of Academy training programs in 

an integrated fashion. 

II. Description of the Program Evaluation Model Comocnents. 

Program evaluation is designed to accomplish several tasks. These 

tasks "re to (i) definE' exactly what the program is, its purposes and goals, 

(ii) document the exact structure of the program, (iii) define the process 

in the program (a logjcal step-by-step explanation) that achieves the goals, 

(iv) monitor the process to insure that any breakdown in the program during 

implementation or operation can be identified, (v) measure the outcomes of 

the program to determine if it is accompljshing its goals, and {vi) define 

and document any program revisions made to change the process, including the 

basis for the change and how this alters the structure and oaths to produce 

the desired =esults. This paper describes a generic model for ATCS training 

program evaluation. The four components of the modal are (i) design 

evaluation, (ii) implementation evaluation, (iii) f~rmative evaluation, and 

(iv) summative evaluation (3). 

Program design and imp;ementation evaluations, as the terms imply, occur 

at the beginning of the program. Formative and summative evaluations occur 

simultaneously and serve to evaluate the process and course of the program 

as well as its products. Each of these evaluation components uses the 

techniques of statistics, math modeling, and various reporting systems. 

Design Evaluation. Program design evaluation involves insuring the 

proper development of several tasks that make up the program implementation 

l 
plan. First, the overall objectives of the program must be clearly defined. 

Every expected outcome of the program should be listed. The outcomes should 

be organized by broad categories and related to the objectives of the 

program. All curricula objectives, student assessment techniques and 

instruments, and teaching/training lesson plans ~ be based firmly on 

thorough task, knowledge, and skills analyses. A task analysis is a careful 

documentation of all the tasks performed in controlling air traffic and 

their relative importance and interaction. A knowledge and skills analysis 

is a determination of the knowledge and skills and knowledge and skill 

levels required to perform each task. Consequently, the task, knowledge, 

and skills aualyses serve as thE' precise and clear job sample.on which the 

student curricula, assessment, and teaching/training le>sson plans are based. 

This is a very crucial and important step• 

Next the teaching/training and assessment proce>ss or methods must be 

operationally defined. This involves a logically connected step-by-step 

explanation of ~he methods to be employed in accomplishing each of the 

outcomes and measuring the accomplishment of each of the objectives. This 

should include the use of any teaching equipment or aids. Flowcharts, PERT, 

tables, GANTT charts, and graphs should be used as appropriate in defining 

the process. Careful documentaticn of every step should be made during this 

evaluation phase by the evaluation staff vith regular reports to the 

responsible supervisor on the progress of the design. The completed 

implementation plan should be clear enough that any competent educational 

expert could carry out the design. Figure 1 illustrates the process of 

specifying the program design requirements. 

Curricula 

Task 11 objectives 1\ Operational 

knowledge definition 

and skills 

~ Student 1\ of teoc hing I 

analysis Traceability "' assessment training process 

I documentation \ •ooho~"'V and methods·· 

I PERT I 

I J instruments 

Programs Gantt 

Tables objectives Teaching I Graphs 

tro1ning Flowcharts 

I esson plans 

F!Gm\.E 1. SPEClFICATIOtl OF :'ROGRAM DESIGN REQUIREHENTS. 

In the case of automated ATCS training systems, the design phase has 

several additional components. First, the operational requirements (Figure 

1) from the task, knowledge, and skills analysis are stated in terms of the 

functional products that a training system must produce. This is a clear 

description of the visible workings/outcomes of the needed training system. 

The functicnal requirements should contain only the essentials necessary to 

2 
simulate thP required operational activity for the detC'rmined level of 

training. Figure 2 describes this st~p. 

OPERATIONAL 

REQUIREMENTS 

FUNCTIONAL 

REQUIREMENTS 

FIGL'RE 2. FUNCTIONAL REQUIREMENTS. 

Which details of the Real Operational 

ATC environment MUST be Simulated? 

o Specify Essentials 

o Eliminate "Oeslrements" 

• • Freeze" for Duration 

• Program Manager has Decision 

on Future Changes 

It is at this stage that computer-derived measures to assess student 

performance are stated as a fcnctional requirement. Particular ca1e must b€ 

taken in this phase to ei!minate any unnecessary requirements. As pointed 

out by Page (2), the opti~al, cost-efficient point on the complexity 

function is the mi.nimal system required to satisfy the needed functional 

activity (see Figure 3). 

Cost 

to 

Develop 

I 

I I 

L----1-------------

System Complexity 

FIGURE 3. THE COMPLEXITY TRADEOFF. 

The next design phase concerns the engineer more than the educational 

technologist; however, the educational technologist is involved in this 

stage and should be aware of the proc~ss. This ph~se is the design 

OPERATIONAL 

REQUIREMENT J. 

FUNCTIONAL 

REQUIREMENT ~ 

HARD 8 SOFTWARE 

DESIGN APPROACH 

FIGURE 4. THE DESIGN APPROACH. 

3 

Competitive Design : 

• Select System Architecture 

• Types and Size of Computers 

• Software Approach, Language, 

Architecture 

• Determines Growth t"otentlal, 

Flexibility, Maintainability, 

Rellabilily, Etc. 
approach. As Figur~ 4 points out, this step includes the selection of the 

most reliable and cost-efficient minimal system architectur~. The 

educational technologist acts as a consultant to the system engineer to 

insure that the selected syst~ performance will satisfy the functional 

requirements. 

Page (2) points out several reascns why it is very important to make 

correct judgments about the system during the design approach phase: 

(i) 

(ii) 

(iii) 

(iv) 

The developer has to me~e the corrections; 

The impact on progr~• cost is less; 

The cost to th~ ~ser after system delivery {maintenance) 

is much less; and 

The system Will be less troublesome to use early in the 

operational phase. 

Figure 5 further illustrates the impact on cost of making errors that 

must be corrected later in the development process. The two lines on the 

graph show the relative cost for making a large number of errors versus many 

feWPr errors. 

Rl!latiVI! 

Cost to 

Corrl!ct 

Error 

50 r1 

20 1 

I 

10 t1 

s r 

2 I r 

I 

I 

I 

I 

I 

I 

I 

L-----1-----r----r-----r-----r----J 

Preliminary Detailed Code + lnti!Qrate Validate Operation 

Dl!sign Design Dl!bug 

FIGURE 5. ERROR VERSUS COST. 

The rem~inder of the dEvelopment process during the design evaluation 

consists of the detailed design, hardware and software development, and 

system te£ting. The detailed design and hardware and software development 

are engineering tasks; however, the educational technologist again ac~s as 

a consultent to insure that the product satisfies the needs of the training 

re~uirements. Figure 6 depicts the entire process. The system testing 

phase is particularly important to the educational technologist, since this 

is a demonstration of the system's ability to perform the functional 

requirements as specified. Care ~hould be taken to insure that the test is 

a valid demonstration, covering all aspects of the functional reqcirements 

4 
OPERATIONAL 

REQU!REii!ENT t--..., 

FUNCTIONAL 

ltEQUii'!EIIENTS 

~------~~~~~---~ HARDWARE AND 

SOFTWARE 

DESIGN APPROACH 

FIGURE f>. THE DEVELOPMENT PROCESS. 

DETAILED ..... ---. DESIGN 

~----~~---Z~--~ 

HAIIDWIIRE& 

SOFTWARE 

DE'!t:LOPIIENT 

with thP stated system reliability. Corrections after the system is sc•t 

intact can be very costly. The system test should secrve as tho> final 

checkpoint to catch all remaining bugs in the system and incongruencics with 

the functional specifications. 

Implementation Ev<Jluation. The implementation eYalu•tion phase 

monitoLs program implementation and insur0s and documents th2t the program 

was implemented strictly according to thQ design. Any changes made to the 

design during implementation should be carefully documented and the design 

revised. The implementation evaluation stage insures that the stated 

process is operational, inta~t, and stable. This evaluation is generally 

accomplished by means of frequent statos studies during the implementation 

stage. Data is collected (usually by surveying the responsible personnel) 

on Pach aspect of the process and a determination made about the state of 

implementation. Direct observations shou1d also be made on a periodic 

schedule. The status studies are generally made into a report for 

decision-makers \-...-fth suggestions to improve or expedite implementation. 

Shortcomin~s in implementation are noted in ~ach report. Figure 7 is n 

flowchart depicting th~ process of implem~ntation ~valuation. 

COLLECTION 

OBSERVATION 

STATUS 

REPORT 

y 

CONTINUE 

PROTOTYPE 

TRAINING 

FIGURE 7. FLOWC!Y1.RT OF INPLENENTATION EVALUATION PF.ASE. 

5 
Formative Evaluation. Whetl the program is d"termined to be 

operational, intact, and suffici~ntly stabl~, formative and summative 

eva) uations begin. Formative evaluatic-n is an ongoing process that insures 

that the program remains on target. It is the process of continually 

collecting data and ~tatistics related to training criteria, i.e., how w~~~ 

students are doing in training. This is a monitoring process to gauge the 

operational stability of the program and the quality of students coming into 

the program. It is also a method for ~onitoring compliance Yith Equ~l 

Employment Opportunity Commission (EEOC) guidelines. 

The data base for formative evaluation should be extensive. 1t should 

contain information for each htdividual on the <oucrent EEOC and Office of 

Personnel ~mnagement (OPM) minority status code, all pertinent attitude 

information such as expectation and the set/information given to them prior 

to coming to the Academy, individu3l and composite scores for selection 

tests, other information used for points in selection such as education, 

experience, and veteran's preference, pass/fail Information, and all 

training scores for academic and lab phases. Item responses for all tests 

during the training phase should also be maintained. 

On a periodic basis, statistics and reports should b~ summarized for 

research purposes and for transmittal to decision-makers. Statistics should 

include sample size, means, standard deviations, intercorrclations, 

pass/fail rates, reliabilities on tests and labs, tests for parallelism on 

different forms of the same measur~, and item parameters. i~e., item 

difficulty, item discrimination, .1nd th(' vc:lidation of par~1lel laboratory 

problems and new items for parallel tests. These statistics should be 

maintained on record in both computer backup files and hard copy. Fu~ther, 

the statistics should be calculatP.d by input and be c~ulative up to and 

including the most recent input. Administration formative evaluation 

reports should include sample size, means, and intercorrelations on all 

relevaut measures, and pass/fail rates stratified by t:linority status, sex, 

prior experience, pred~velopmental/noncompetitive entry, veteran's 

preference, educational level, option, and region. Appendix A contains 

sample reports for formative evaluation. 

~ben, based on the formative summary data, there appears to be a 

problem in how the training program is running, the evaluator has the 

resonsibility to aler" the appropr~ate administrative personnel and prepare 

a concise report identifying the problem areas. Isolzting the exact aren of 

concern may require some mathematical modeling. The attitude information, 

where appropriate, should be employed as a covariate in the modeling. 

Modeling will be discussed in detail later. 

Summatjve ~valuation. S~ativc evaluation is a continual assesscent 

of the quality of the products of the program. While formative evaluation 

is sammarized on an input-by-input basis and serves as an immediate feedback 

loop for ongoing program revisions if need~d, summative evaluation occurs on 

a larger scale across a longer time span (e.g., on a yearly basis). 

Formative evaluation is concerned with internal program accuracy and 

6 
stability, program reliability, and content andi cr concurrent demonstrations 

of validity.(For example, are the measures reliable? Are the objectives 

well matched with curricula content? Do the pass/fail rates remain 

stable?) Summative evaluation, however, is a check on the quality of the 

output from the stabilized program. The summative evaluation is a test of 

predictive or criterion validity. It is a measure of the on-the-job success 

of those who pass the Academy training, and the relationship of how well the 

candidates performed in the Academy compared to how well they performed 

on-th~-job. The so-called validity coefficient is the measure of this 

relationship. 

The summative data bas~ should consist of several components. It is a 

comprehensive tracking of the career progression of every successful Academy 

candid<1te. It should contain data for every individual on types of 

facilities where the person has been ,·mployed, measures of job performance 

at each of these sites (criterion measures), type of attrition and why, 

whether a person changed options and ,.'hy, whether a person was maintained by 

the agency in a non-2152 (ATCS) position, and as much attitude and 

demographic information as possible (e.g., divorce, aspects of the job the 

person likes or dislikes, etc.). 

Statistics and reports shouid be summarized from the summative data 

base on a regular schedu1c for research and as information for 

decision-making. Statistics should include sample sizes, mc3ns, standard 

deviations, intercorrelations, validity coefficients, attrition rates, and 

Data 

Collection 

y 

Administrativ~ 

Summary 

Reports 

Statistical 

Research 

Files and 

Reports 

Model the 

Change 

N 

N 

y 

Select 

Change 

Strategy 

FIGURE 8. FLOWCHART OF THE GENERAL PROCESS FOR BOTE FCil'iATIVE 

A,'.'D SID~~ATIVE EVALUATION. 

7 
mathematical modeling, using the attitude information as a covariate. 

Administrative summative reports should i~c!ude sample size, means, 

intercorrelations, and validity coefficients on all relevant data, and 

attrition data should be stratified by minority status, sex, prior 

experience, predevelopmental/non-competitive entry, veteran~s preference, 

edu~ational level, reasons for attrition, 2152/non-2152 attrition, option, 

and region. 

If the summative data base demonstrates a problem in the prograc, a 

need for a major program revision may be indi~ated. The data should be 

reviewed very carefully, employing mathema~ical modeling to isolat~ the 

source of the problem. As in the formative evaluation, the decision-makers 

should be alerted to the problem but, in addition, in the case of the 

summative data, policymakers and Acadecy officials should be alerted. Major 

program revisions require c~reful planning and more detailed ?.ttention than 

revisions based on fo~ative data. ft~pendix B contains an example of tha 

summative reports. Figure 8 flowcharts the general process for both 

fo~ative and summative evaluation. 

The interaction and dynamic nature of the program evaluation 

components. Figure 9 contains a summary of the four components of the ATCS 

progra~aluation model. The descriptions in Figure 9 imply an internction 

bet~en the formative and summative evaluations. Th£ fo~?.tive evaluation 

is designed, through constant analyses and feedback mechtinism~: to serve cs 

a guidance system ir. keeping the program on track toward mceti~g t~e stated 

curricular objectives. It serves to stabilize the methods employed in 

teaching and training the curricuiar objectives. Tne summativ~ e~~lu~tion 

is designed to inform policymakers as to ~~ether the ~etho~s employed in 

meeting the curricular objectives snd/or ~~ether teaching to these s.ated 

objectives actually produces a s·uccessful ATCS. If the tr~ining oethods are 

not stable or the curriculsr training objectives are not met, and/or these 

shortcO!!lings are not detected and corrected vit.hin a very short. time period 

by the formative evaluation~ it is iepossibl<' fo:!' the Stn:!mative evaluation 

to determine •~ether the present training methods being employed and/or the 

present curricular objectives are producing the product being viewed. Tne 

interaction bet~en forma~iv~ and suomative evaluation is depicted in Figure 

10. 

This interaction bet~een suc~tive and formative evaluation has several 

implications: \i) It implies that a progrzm should be ve~y curefully 

designed and implemented initially. F£ previously mentioned, this means 

performing thorough task, knowledge a~d skills analyses and a careful 

matchjng between the job samples taken from the analyses 3nd t~~icular 

objectives, assessment techniques, and trai~ing methods. (ii) The 

interaction also implies that the sue~tive evaluati~r. assesses how 

successfully the formative evaluation is working. An unstable program 

produces confusing and inconsistent sum~tive deta. (iii) The l2st 

implication relates to p~ogram ch2nges: ~~en can progr2m revisions be made; 

how large a chang~ can be made based or. formative anc summative data; and 

what type of evaluation is required given that a change is made. 

8 
DESIGN 

EVALUATION 

IMPLEMENTATION 

EVALUA'!ION 

FOR..V..ATIVE 

EVALJ.:ATION 

S~'ATIVE 

EVALJ.:ATION 

--------------------------------------------------------------------------- !Tc state goals; !To insure that 

!develop, define,!the design was 

!and document !fully and corlcurricula, ob- lrectly impl~ 

Jjectives, mod- !mented and i& 

lules and assess-iintac~ and 

lment tools. De-lstable. 

OBJECTIVE ivelop and define! 

!logically con- I 

lnected step-by- l 

!step process to I 

!achieve goals. ! 

!Automated trai- ! 

lning systems re-I 

!quire menitoringl 

lthe system l 

I development. 

!To insure th2t iTo neasure the 

ithe program fquzlity of the 

!stays on target (final program 

land to add end (product. ~nat 

)evaluate refine-!is the p~ogram 

lments changes tolpayoff? More 

!the program in a~conce:ncd ~ith 

!systematic man- !program validlner. ~:ore con- llty. 

lcerned w~"h pro-1 

lgrem reli- i 

lability. 

' I 

~!ETEOD 

!Careful documen-IFreque~t statu3l}~intenance ot 

ltation and d~s- !studies with !an ongoing data 

lcription of !Geta indicating!basc that m~almajor systems !the extent of !sures progr~ 

J and subsystems i implementation ! stability and 

!of the program; !for each area !collection of 

!use of flow- lof the proc~ss.idata that ~uld 

!charts, PERT, !Regular sta~us lbe sensitive to 

!tables, graphs, !reports are !any chdnge inland general sys-lissued. ltrnduced. Regu-

!tems analysis l !lar reports to 

!technology. l !management. 

!Collection 

iand analysis 

(of date oc 

!the success 

l of gr~duates 

!of the p~o­

!gr=· Perijodie rep:>rts 

!on the data 

!analysis. 

---------------------------------------------------------------------------

RELATION 

TO 

DECISIONMAKING 

!Determining rhc !Status reports 

!best plan of )offer suggeslacticn to accoc-ltions for imlplish the pro- fprovement and 

!gram objectives.liPdicate where 

i I implementation 

l lis falling 

I !short. 

! I 

; l 

! l 

iOffers informa- !Offers inforItion on progr~ }mation en the 

!stability iquality oi 

:through regular !the program 

!reports and in- fproduct that 

!formation o~ thelis especially 

!effects of any !useful for 

!program change. llor~-tero 

IAlso offers in- iplanning an~ 

ifo~ation on ineeCed 

IIEOC compliance.\change. 

---------------------------------------------------------------------------

FIGURE 9. SUM!1ARY OF TiiE FOUR CCMP0!<"E1."TS OF THF A'!:CS PROGRA.'! 

EVALU_<.TION HODEL. 

9 
SUMMATIVE EVALUATION KEEPS F'ROGRAM ON 

TRACK TOWARO UEC:TING PROGRAM 013.JECT!VES. 

FORMATIVE EVALUATION KEEPS PROGRAM I 

ON TRAC;< TOWARD ACCOMPliSHING 

CURRICULA OBJECTIVES. 

P~OGRAM '1'RACV. 

PROGRAM 

<:&IECTI\IES 

FIGURE 10. UlTE&\CTION BE':.'I<EEN FOR!-!ATJVE A~:D SUI'rMATIVE EVALUATION. 

Program changes can be classified as (i) progr&m adjustm2nts, (ji) 

changing a program component, (iii) adding or subtr,;cting 2 program 

component, and {iv) a najor program r~structuring. 

Program adjustments ;;rc changes that affect a common elemC'nt across 

sevPral program components. They are small or medium chong~s. LargP 

program adjustments vould fa 11 undf:'·r t .1e c:::~tcgory of mr:j cr rest rue turing • 

Program adjustments usually tBke the form of changes in presentation of 

1t?sson DlCit~rial, sm.?.ll C'urricu1D ;1djustrnents, moCificntions in thf." typ0s of 

ass<'ssmcnt devices, or changing the it<'m fon:?.t in t<:st._. Generally, the 

formative t..:valuntion process can offer sufficiPnt inform8tion to ~valuate 

such a change. In a small number of cases a medium program adjustm~nt may 

require summative dara to evnluate the chAnge. 

Changing a program component can v2.ry from ::1 small to a l2rgc- cbangc. 

Small changes vould include cha.1ging j tf'mS in a component test, reordering 

the sequence of test items, or minor codifications in a program co~ponent 

curriculum. Small program compon~nt changes can be sufficiently cvuluated 

by the formative Pvaluation data. }~cdi.um component ch<mgf's would inclu2£' 

changing the sequ'3nC£> of th.:.: component in t'ne program or n medium curricu11..II:l 

changP in th-r- component. }1cdium changes require formntivP evaluation dnta 

and usually also require summative evaluation drt:!. A larp,e change is a 

major revamping of the compon~nt ~nd requires design, implementation, 

formative, and sum.m;_tive evaluation. 

Adding or subtracting a program component, ~ven in F- conserv~tive 

sense, represents a medium or (usually) a large chang<'. In either case, 

data required to evaluate the <'ffect of c;dding or subtracting a compor.l'nt 

include design, implementation, form2tiv0, and summative cvalust~on. 

A major restructuring of the program essentially requires the same 

process as a beginning program and involves df>:sign, implemE-r.tation, 

formA.t ive and summative evaluation. Eval u:1tion of e major rcstructur ing 

should place more emphasis on the design and implem('ntation evaluation than 

10 
any of the other types of change. Program changes and the required types of 

evaluation are s~rized in Figure 11. 

---------------~-------------------------------

ACTHITY SMALL MEDIUM LARGE 

-------------------------------------------------- I Program I Formativ,;, 

! Adjustments l 

I Formative 

l Summative 

N/A 

------------------------------------------------------------------- Chang£> a 

Program 

Component 

I Formative 

I 

I 

I 

Formative 

Summative 

I Design 

I Implementation 

! Formative 

! Summative 

--------------·-------------------------------------------- Add/Subtract 

a Program 

Component 

Major 

Program 

Restruct• .. ring 

N/A 

N/A 

Design 

Implementation 

Formative 

Summative 

N/A 

Desig:> 

Implementation 

Formative 

Summative 

Design 

Implementation 

Formative 

Summative 

-----------------------------------------------·-----~------------

FIGL"RE 11. TYPE OF EVALUATION REQUIRED FOR PROGRA."l CHANGES. 

Tne operation of the total model is dynamic and interactive. Each 

component is dependent on the correct accomplishment of the other 

components. The dependency, while overlapping, is somewhat linear. 

Accurate summative evaluation depends on accurat~ formative ev2luation, 

accurate program implementation, and accurate design. Accurate formative 

evaluation depends on accurate implementation and d~sign, and so forth back 

to the task, knowledge and skills analysis used in the design evaluation. 

Figure 12 is a schematic path diagram of the interaction and dependency 

among the four components. 

Design Implementation 

Eva ' uatton . r Evaluation 

.... 

Formative 

~ 

Summative 

"""' Evaluation Evaluation 

FIGURE 12. THE INTERACTIVE CLOSED LOOP PROGRA}1 EVALUATION STRUCTL"RE. 

ll 
III. ¥~thematical Models 1n Formative and Sumnative Evaluation. 

~ linear model and intervening variables. The mathematical models 

used in formative and su=nmative evaluation center on principles of linear 

regression. The most common phenomenon of interest is hoY each measure is 

related to another, i.e., the regression of one ~easure on another. The 

simple equation for linear regression is: 

1 

Y z a+ bX, (1) 

where X = independent measures, a ~ the value where the regression line 

1 

intercepts the Y axis, b = the regression coefficient and Y = the predicted 

dependent scores. TWo sets of measures (X and Y) can be plotted as in 

y 

:r 

4 

3 

2 

0 

I 2 

r • 

X 

3 4 

.90 

'! = i~O+ .90X 

I 

tor X •3, Y•l.0+(.90)3 

I 

y E 3.7 

FIGURE 13. PLOT OF TviO SETS OF ~~EASCRES. 

Figure 13. As vi,wed in the plot, the regression line intercepts theY axis 

at 1 and the s.Lope of the regressiou line :o. • 90. Th.e slope indicates t.he 

predicted change in Y for each unit change in X. Conse~uently, it is easily 

seen how linear regression offers a me2ns to predict or estimate values of 

on~ measure frOtJ. values of another measure. The: closer the data points on 

the graph arE' to "- straig:C.t line, the bett"r the prediction. A good exanple 

is plotting Academy scores {X) by a success ~easure on the job. 

To explicate, suppose a researcher ~~rz to take measures of the same 

phenomena tw~ times and then plot the two occasions. Identical measurement 

processes and conditions would yield a graph likr Figure 14. The slope 

Yould be 45 degrees, placing the intercept through the origin at a ~ 0 and 

the slope, the increment in Y for a UJit ch~nge in X, at b = 1.0. Tnis 

result, as noted above, is contingent on t~o factors in th~ process being 

identical: (i) perfectly accurate measures on both occ~sions, and (ii) 

l2 
identical relevant conditions. If e~ther of these factors were altered, the 

regression slope and/or intercept most probably would chang~. 

y 

4 / YzO + I.OX 

3 

0 2 3 4 

FIGURE 14. PLOT OF TWO IDENTICAL !1Et .. SCRES. 

The factors that would alter the regression line in the above example 

are referred to as: (i) neasurement error and (ii) intervening variables. 

Figure 15 illustrates the concept. 

I MEASURE # I 1---· 

INTERVENING 

VARIABLES 

ANO 

MEASURE ME NT 

ERROR 

-1 MEASURE #21 

FIGL'RE 15. TEE CONCEP'I OF INTERVENING VARIABLES. 

Measurement error is usually assumed to be symmetrically distributed about 

the true value and consequently, when summed, equals zero and has no effect 

<>n the analyses. 

One of the major uses of the linear model in program evaluation is the 

identification of intervening variables and determining their impact on 

dependent variables. For example, what intervening variables affect the 

relationship bet~~en Academy scores and field success {the validity 

coefficient)? In exper~ental design the intervenir~ variables are 

generally viewed as independent variables and the measures affected as 

dependent variables. Suppose one wanted to determine the impact of 

motivation (independent variable} on Academy success {dependent variable). 

Several methods are employed to identify intervening variables and their 

impact on dependent measures. 

13 
l'ethods for identifying .Jm2_ determining the impact of intervening 

variabl~s. If one can identify intervening variables and th~ir impact on 

dependent measures, then a transformation equation can be generated to map X 

on to Y and the relationship between X and Y can be mathematically explained 

and quantified. The ~ethods employed to accomplish this depend on the 

nature of the variables involved and the assumptions one employs in the 

model. 

Assuming the linear model, the process can be explained as follows. 

Suppose, for example, ve have two measures of the same variable, and a plot 

of the measures appears as in Figure 16. 

y 

Measure #2 

Measure# I 

0 2 3 4 5 

FIGURE 16. PLOT ON HF.ASlJRE ill, }!E.,\SiJRE 1!2, AND REGRESSION LINES. 

Further, suppose ve suspect that an intervening variable Z was the 

reason for that difference. It ~~uld then stand to logic that if the 

measures (Y ) were adjusted to acco~T.t for the influence of Z then the 

i 

r~gression lines should be equal. Returning to the linear Pquation \1) 

l 

Y = a+ bX~ (l) 

1 

we can sec that the estimated Y~ {Y ), contains so~e error of estioation 

unless all the X and Y points lie on a straight line. So, 

1 

y Y m Y{E) • (2) 

i i i 

This is the error made in estimating Y for the elem~nt i. we c~n nov state 

an equation for Y using the linear ncdel. 

i 

Y = a + bX + Y(E) (3) 

1 i i 

14 
From equations (2) and (3) it is also easily seen that Y(E) is uncorrelated 

with X since Y(E) is that component in Y which is unexplained by X. 

The aim of the adjustment of the measures (Y ) is to remove the 

i 

influence (or the relationship) of Z on the measures (Y). Thus applying the 

linear equation to estimate Y from Z, 

1 

y ~ A + bZ, (4) 

we can produce 

1 

y - y ~ Y{E) . (5) 

i i i 

Y(E) is now a Y value minus the influence of z. If we plot the Y(E) 

values separately for ~1111 and ~f-2 as in Figure 15, we have a picture of the 

plot without the influencE' of z. If the regression lines fc..c 11111 and Mf/2 

are identical after removing the influence of Z, then Z can be used to 

explain the differe~cc in the =egressior. lines fer MOl and M~2 and Z can be 

used to adjust Y to equate Mfl and !fi!2 regression lines. This procedure 

obviously assumes parall<'l regression lines since it is the intercept that 

is being adjusted. Consequently, the adjustment to Y can be stated in the 

linear equation as, 

Y(A) -y b(Z -a) + Y(E) • (6) 

i i i i 

where Y(a) m the adjusted y for element i. o~. the estimated adjusted 

Y is, 

1 

Y(A) •Y b(Z -a). (7) 

i i i 

The method just explained for removing the influence of Z is a 

univariate process, checking one variable at a time. Often, intervening 

variables that account for the differences in regression lines may be 

correlated. Consequently, the effects they account for are not additive, as 

seen in Figure 17. 

Univariate analyses can help select variables in which their total 

effects (unique effect+ shared effect) are shown; ho~ver, if two or more 

intervening variables share, to a large degree, in their effect on the Y 

regression lines under two "eparate conditions, }lffl and M#Z, then all the 

variables are not needed to explain the difference nor to adjust Y. One 

must employ an analysis that utilizes the unique contribution of variables 

in producing the differences in Mil and Ilf!Z without the spurious addition of 

overlapping effects shared by one or more variables. 

15 
I L_ 

FIGURE 17. OVERLAP OF INTERVENING VARIABLES Z 

1 

AND THEMSELVES. 

AND Z WITH Y 

2 

Multiple regression. l!ultiple linear regression is a method of 

analyzing the shared and unique contributions of more than one independent 

variable X (i=l ••• k) to the variation of one independent measure, Y 

i 

(Kerlinger, 1973). By variation woa mean how the measures in Y a;' different 

from each other. For example, if all Academy candidates were equally 

successful on-the-job regardless of their Academy scores or selection test 

scores~ then the variance in the success measure is zero and we have no need 

to analyze the measure. 

The model for multiple regression is an extension of our previous 

1 

single variable regression of X on Y. For an estimated Y, Y , 

1 

y = a+ b X + b X + ••• +b X > (8) 

1 1 2 2 k k 

and for Y, 

Y=a+ b X + b X + . . . +b X + e • (9) 

1 1 2 2 k k 

The procedures in multiple regression a<~ an extension of the previous 

dicussion on adjusting Y for the influence of a third variable, Z, prior to 

correlating X and Y. In multiple regression the variance in Y that can be 

explained by the first variable is partialed out. The remaining variance in 

Y that can be explained by the second variable ~~thout duplicating or 

overlapping that expressed by the first variable is then partialed out. 

This process continues until all the independent variables, X , have been 

considered. The relationship of X 

i 

i 

and Y without duplication or overlap 

16 
2 

among X is termed the multiple R. The multiple R squared, R , expresses 

i 

the proportion of variation in Y explained by X • If all X were 

i i 

2 

uncorrelated, not duplicative, then the multiple R would be the simple sum 

of all the squared correlations, r (see Figure 13 to review "r"}, of X and 

i 

Y. 

2 

R 

y.l2 ••• k 

2 

m r 

yl 

2 

+ r + 

y2 

2 

+ r 

yk 

(10} 

2 2 

However, if the X are correlated, then R is the sum of all the r of X 

i i 

and Y with the duplication and overlap partialed out. 

2 2 

R e r 

y.l2 ••• k yl 

2 

2 

+ r 

y(2.i) 

+ ••• + 

2 

r 

y(k.l2 ••• k-1) 

• (ll) 

where r is read, the correlation of variable 2 and Y with the effects 

y(2 .1) 

of variable 1 partialed out. 

Discriminant analysis. Since the Academy programs are pass/fail, it is 

a common question to ask which measures best discriminate between passing 

students and failing students. As an example, suppose we had measures on 

motivation, level of education, prior experience, and Academy scores and •Me 

wanted to know which of these best discriminated between students who pass 

X 

II 

I 

I 

I 

I 

I 

I 

I 

I 

L---------- x2 

FIGURE 18. PLOTTED DATA FOR THE T".JO VARIABLES, TWO GROlJP CASE. 

17 
field training to full performance level (FPL) and those ~uo fail. Then we 

would perform a discrimin.,nt analysis. To explain this procedur<', '""' go to 

the most simple case. Suppose we have the simple case of two suspected 

intervening variables and two groups. If the data for ~he t~o groups ~~re 

expressed on a graph where the axes were the t~~ predictor variables x and 

x , the data could be shown as coordin.'!tes of the two variables. 

2 

1 

Forming the weighted sum of the intervening variables would create a 

new variable, Y. 

1 

Y • V X + V X ~ (12) 

l l 2 2 

where v • the weights employed. This may be recognized as another linear 

equation similar to those previously discussed under "multiple regression." 

The question is how to express the measure Y on our graph, or, more 

accurately, how can a Y ~xis be indicated in the way x and x arc? The 

1 2 

answer is, the desired axis can be 

represented by the two weights, v 

demonstrated by locating the coordinates 

and v , and drawing a line from this 

point to the origin of the x 

1 

1 2 

and x axis (sec Figure 19). 

2 

The data 

coordinates can now be projec•ed onto the new Y axis as separ te 

distributions for each group. 

The following is a representation of the scheme described above for 

four different weighted sums of x and x • 

1 2 

y "' 

I . 8 x, - .6x2 

y = 2 .97x1- .24x2 

Y. • 3 .9sx1 - . 18x2 

y = 4 • 71 x, - . 7tx2 

FIGURE l 9. PROJECTIONS OF TWO GROUPS ON FOUR AXES REPRESEtiT:NG 

LINEAR COMBINATIOUS OF THE ORIGINAL VARIABLES. 

18 
It can be noted from the representation that the projected 

distributions on the Y axes are separat~d differently. Some of the 

projected distributions overlap mryre than others. The problem, then, is to 

nefine the Y axis in such a manner that the projected distributions overlap 

the least. Obviously, in order to do that, a means to measure the overlap 

must be determined. 

One means to define the overlap might be to subtract the means of 

Y and Y and divide that d.ifference by the standard deviation of one 

1 2 

of the groups. 

y - y 

1 2 

s 

y 

(13) 

However, this would express the difference only in terms of one of the 

standard deviations. 

A more equitable way to do this is to pool the within groups standard 

deviation. This is accomplished in the following manner. 

I 2 2 

(n -1 )S - (n -l)S 

J 

1 y1 2 y2 

s = -----------------------' (14) 

y(w) n + n - 2 

1 2 

where S is the pooled within groups standard deviation, n is the 

y(w) 

2 

group sample size, and S is the variance for each group. 

y 

Now, a more stable and equitable measure of overlap can be expressed 

2 

(Y - y ) 

2 1 2 

f s ------------, 

2 

s 

y(w) 

where f is the measure of overlap in the two distributions. 

To extend this measure to more than two groups, 

19 

(15) 
where 

with 

and 

2 VAR(Y) 

f - -------. k 2 

s 

y(w) 

k 2 

E (Y -Y.) 

gal g 

0 --------------, 

k-1 

y + y + ••• 

1 2 

+Y 

k 

y. - --------------------, k 

k 2 

E {n-l)S 

2 ll""l g y(g) 

s - -----------------, y(w) N-k 

this being the within groups mean square, ~$ 

(16) 

(17) 

(18) 

(19) 

( "'' 

In order to take unequal n into accou~t in the numerator above, 

k 2 

E n (Y -Y.) 

g-1 g (g) 

MS • ------------------ (20) 

(b) k-1 

where 

En Y 

g (g) 

Y. • ---------, (2:) 

N 

and Y. is the grand mean of Y in the total sample in all k groups. 

20 
Collecting things together, Ye have, 

2 

f = 

1< 

vith MS 

(b) 

'MS 

(b) 

-------• l!S 

(w) 

= mean squares between and MS 

(w) 

This formula is generally expressed as, 

ss /(k-1) ss 

2 (b) (b) N-k 

f c 

------------- .:: -------X • 

k ss /(N-k) ss k-1 

('W) h>) 

{22) 

e mean squares within. 

(23) 

and, since the multipliers (N-k)/(k-1), are constant for any given problem, 

they can be omitted, yielding 

ss 

(b) 

h = -------• (24) 

ss 

'"'' 

where SS c sum of squares between and SS = sum of squares within. 

(b) (w) 

k 2 

ss = E n (Y -Y.) • (25) 

(b) g=l g {g) 

and 

k 2 

ss E E (Y -Y.) 

(w) g=l (g) 

n 

k g ') 

~ 

= E E (Y -Y ) • (26) 

gal i=l (g)i (g) 

Y being the Y score of the ith individual in the gth group. The 

(g):i 

quantity h is termed the criterion. 

21 
The problem, then, in performing this analysis is to express the 

criterion, h, as a function of the weights v , v , ···~ v , and to 

1 2 p 

determine by differential calculus the set of weights which maxi~ize h. The 

weights then express the relative contribution of each intervening variable 

in explaining the differences in the two groups. 

Summary of mathematical models. Basically three examples of linear 

models were described, as well as the gen~ral notion of linear regression. 

The three examples are by no means inclusivP of all linear models; however, 

the ones presented are the most frequently used in the ATCS program 

evaluation mod0l• ~inear regression models arc particul~rly useful in 

program evaluation since the major function of any screening progrnm is ~o 

best predict on-the-job success and to determine the most efficient subset 

of measures that can be used to do that. Linear regression is also very 

usefu' for estimating the impact of various proposed changes to the program. 

Withot•i: mathematical models, program <>valuation would be extremely difficult 

at bet t:. 

IV. Summary. 

The Systems Analysis Research Cnit at C&~I has developed a generic 

model for Academy training program evaluation. The model will serve as a 

basis for 5ntegrating the total data base into a common format across all 

training programs. The model consists of four components; (i) design, (ii) 

implementation, (iii) formative, and (jv) sumrnatiVP evaluation. Design 

ev2luation is an assessment of the cocprehcnsive implementation plan; 

implementation ev21uation is a detercination that the plan is completely and 

accurately implemented according to prescription; formative evaluation is a 

continual monitoring of the program to keep the process reliable, stable, 

and on track; and summative evaluation monitors the product of the ~raining 

program. The design evaluation relies on the task, kno~ledge, and skills 

analysis and on the documents in the i~p]ementation plan. ThP 

implementation evaluation mak~s use of dat~ from frequent status sturliesv 

Formative and summative evcduations make usc of stat-:::stics and mathematic.e.l 

modeling, primarily Jjnear regression mode1s, ~o monitor the process 2nd 

products of the programs and to estimate an~ d~t~rmine the impact of changes 

made to the programs. 

22 
REFERENCES 

1. Kerlinger 9 F. N.: Multiple Regression in Beh~Tioral Research. New 

York: Holt, Rhinehart, a!'.d 1-linston, 1973. 

2. Page, L. F.: Technology in Air Traffic Control Training and Simulation. 

PROCEEDINGS OF TRE TPlRD INTERNATIONAL LEARNING T£CEKOLOGY C>JKGRESS Mil 

EXPOSITION, 2:212-216, 1980. 

3. Stufflebeam, D. L., et al.: Educational Evaluation and Decision Y~king. 

Itasca~ 111.: p. E. Peacock Publishers, Inc., l9il. 

23 
.Q'-\ 

CAlli. SYSTEHS ANALYSIS RESEARCH 1111\T 

9-.MI-81 REPORT NUIIB£R &.lRR£12 

PERFORIIAHCE BY REGION FOR 1990 EHROIJTE IIII'IJT 12 STARTING 10-FEB-81 12 

s "' ..., 

.... 

,. REGION H PASS FAIL IIJTHMAU 1!0 511111 RAilttR COO'OSIT£ .. _.,. ________ --------- ........................... ............... -...... ------------------ 0 

• % • % • % • % AVERAGE SD H .... 

:co 

ALASKA 0 0 o.oo 0 o.oo 0 o.oo 0 o.oo o.oo o.oo 0 "' ..., 

0 

EASTERN 5 4 90.00 1 20.00 0 o.oo 0 o.oo 75.20 10.43 5 .. 

... > .. CENTRAL 3 75.00 1 25.00 o.oo 0 o.oo 71.50 3.32 4 "' "' 4 0 ... l'1 

0 2: 

GREAT LAKES 11 6 54.55 5 45.45 0 o.oo 0 o.oo 71.82 5.19 11 .. t:l 

.... '<I >< 

HEll EHSLAHD 1 I 100.00 0 o.oo 0 o.oo 0 o.oo 71.00 o.oo I 0 

~ > 

I H!JITIIIEST 0 0 o.oo 0 o.oo 0 o.oo 0 o.oo o.oo o.oo 0 "' ... 

.... PACIFIC-ASIA I 0 o.oo I 100.00 0 o.oo 0 o.oo 52.00 o.oo 1 < 

.. ROCKY IIOUTAIN 4 4 100.00 0 o.oo 0 o.oo 0 o.oo n.oo 5.94 4 ~ 

SOOTHERH "' 0 0 o.oo 0 o.oo 0 o.oo 0 o.oo o.oo .... o.oo 0 c: 

SOUTIIIEST 0 0 o.oo 0 o.oo "' 0 o.oo 0 o.oo o.oo o.oo 0 ... 

.... 

0 

UESTERH 3 3 100.00 0 o.oo 0 o.oo 0 o.oo 77.00 6.00 3 "' 

TOTAL 29 21 72.41 8 27.59 0 o.oo 0 o.oo 72.90 7.37 29 
CAH!t SYSTEHS ANALYSIS RESEARCH UHIT 

9-JUN-81 REPORT NUMBER 80RXE12 

HEANS BY CLASS FOR 1980 ENROUTE INPUT 12 STAf:TING 10-F£8-81 

8!05 8106 B107 8108 TOTAl INPUT --------------- -----------· -~- -- -------·--~ -- ---... ·----·----- _____________ ... 

H HEAN SD N HEA.~ SD II MfAtl s~ II HEAII SD tl MEAN SD 

ltl.OCK TEST 1 9 94.00 4.58 7 SB.U s.~z 5 92 .41

j 3.58 B 35,5(; 6.21 29 90.14 6.02 BLOCK TEST 2 9 89,78 4.63 7 84.86 6.09 5 rs.4o Z.6l 8 S6.0C 5.45 29 87 ,JJ 5.15 RAJIAR CPT 9 as. 56 4.77 7 83.71 5. 71 5 85.60 6.~3 8 tl2,2S 9.41 29 84.21 o.sa 

PRO& 1 TECHNICAL ASSESSHT 9 72.11 13.38 7 60.86 13.64 5 6l.60 24.03 8 72.()•) 16.39 29 67.72 16.39 PROB I INSTRUCTOR ASSESHT 9 86. :a 8.48 7 66.29 15.93 5 73.60 24.85 8 97.00 9.12 29 79.62 16.34 PROB 1 LAB SCI"lE 9 79.33 10.07 7 63.71 13.92 5 68.00 24.29 8 79.50 12.57 29 73.66 15.65 

PROB 2 TA 9 56.11 18.85 7 58.71 16.13 5 48.60 21.85 8 69.13 16.93 29 59.03 18.64 

• PROB 2 IA 9 79.00 12.36 7 69.71 17.90 5 70.20 25.40 B 83,75 11.29 2Y 76.55 16.49 PROB 2LS 9 67.67 13.75 7 64.43 16.18 5 ~.9.60 !9.42 a 76.50 12.44 29 67.93 15,47 

PROB J TA 9 54.67 21.26 7 5~.86 12,7o ~. 59 ,9(1 ;9, ~·B s 66.2~ 11.3~ ~· 58.31 18.74 

PROD 3 IA 9 73,99 21.1B 7 78.29 10.67 5 72.~0 26,JS 8 85.50 6.9? 29 77,97 !7.00 

PROB HS 9 64,44 20.96 7 65.4~ 11.37 s 6t.,2fo 27.55 8 76.0(l 8.83 29 68.17 17.48 

PROD 4 TA 9 55.11 11.04 7 58.14 15.87 S 63.4V 15.85 8 62.13 21.17 29 ~9.21 15.75 PROB 4 IA 9 ao.oo 12.87 7 78.57 11.57 5 71.40 13.87 B SI.SS 11.64 29 78.69 12.26 PROB 4 LS 9 67.56 10.55 7 68.29 12.94 5 67.60 14.57 8 72.13 16.13 2g 69.00 12.92 

PROD 5 TA 9 54,33 n.14 7 56.14 23.19 5 46.80 ]7,49 8 61.50 21.!1 29 55.45 J2,J4 PROB 5 IA 9 71.78 16.45 7 72.!4 18,71 ~. 69.0~ 19.54 8 83.13 8.90 29 74,52 16.03 

PROD 5 LS 9 63.22 18.83 7 o4.00 20.77 5 58.00 !3.02 8 72,25 IM9 29 65.0~ 18.67 

AU£RASE TA 9 58.33 6.75 7 57.14 4.71 5 56.20 19.66 8 66.00 12.24 29 59,1~ 11.23 AU£RA6E IA 9 79.33 4.39 7 73.00 6.2~ 5 71.4? 2(),07 8 84.25 6.89 29 77,48 10.38 LAB AU£RAGE (All Sl 9 66.78 4.59 7 65.43 6.40 5 63.60 19.37 8 74.38 8.99 29 68.00 10.2F 

RADAR CST 9 85.11 5.59 7 80.57 6.19 5 84.40 7.13 8 81.25 5.65 29 92.83 6.r4 LAB AU£RAGE (BEST 41 9 73.33 5.70 7 69.00 4.U 5 66.80 18.75 8 77.75 8.99 29 72,Je 1o.o9 LAB COHPOSITE SCORE 9 73.11 4.08 7 70.57 s.19 5 70.40 14.64 8 76.25 5.47 29 72.90 7.J7 

015 
:>c.. 

CAHI• SYSTEHS ANALYSIS RESEARCH UNIT 

9-JUH-81 REPORT NUHB£R 80REEI2 

RADAR LAB STATUS BY ENTRY TYPE AND BY HINORITY CATEGORY FOR 1980 ENROUTE INPUT 12 STARTING 1G-FEB-81 

---------------------- PASS ---------------------- -------------------- FAll. ---------------------- % = t PASSED I t ENTERED % = t FAILED I t [HTERED ----·--------------------------------------------- --------------------------~----------------------- All COHPETITIIJE PREDEV CEP ALL CllltPETITIVE PREDEII CEP 

----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- • % • % • X • % • % • % • % • X 

IIO:Il 21. 77.78 20. 80.00 1. 100.00 o. o.oo 6. 22.22 5. 20.00 o. o.oo 1, 100.00 

IIOHEN o. o.oc o. o.oo o. o.oo o. o.oo 2. 100.00 2. 100.00 o. o.oo o. o.oo 

IIIHORITIES 2. 66.67 1. 100.00 1. 100.00 o. o.oo 1. 33.33 o. o.oo o. o.oo 1. 100.00 

NOHitiNORITIES 19. 73.08 !9. 73.08 o. o.oo o. o.oo 7. 26.92 7. 26.92 o. o.oo o. o.oo 

liEN HIN 2. 66.61 I. 100.00 1. 1oo.oo o. o.oo 1. 33,33 o. o.oo o. o.oo 1. 100.00 

liEN HOHIIIN 19. 79.17 19. 79.17 o. o.oo o. o.oo 5. 20.83 5. 20.83 o. o.oo o. o.oo 

WOllEN IIIN o. o.oo o. o.oo o. o.oo o. o.oo o • o.oo o. o.oo o. o.oo o. o.oo 

• UOIIEH HOHitiH o. o.oo o. o.oo o. o.oo o. o.oo 2. 100.00 2. 100.00 o. o.oo o. o.oo 

TIJTAl 21. 72.41 20. 74.07 1. 100.00 o. o.oo 8. 27.59 7. 25.93 o. o.oo 1. 100.00 

--------------- UITHDRAIIIHO SHOU --------------··- ------------------ All ENTRIES ------------------- % = t liD OR NS I tENT£RED % = I ENTERED I TOTAL t ENTERED -------------------------------------------------- -------------------------------------------------- All COIIPETITIIIE I'll EMil CEP All COitPETITIIIE f'llEI(U CEP ----·------ ----------- ----------- ----------- ----------- ----------- ---------~- ----------- • % • % • % 0 % • % • % • % • % 

IIEH o. o.oo o. o.oo o. o.oo o. o.oo 27. 93.10 25. 92.59 1. 100.00 1. 100.00 

IIOIIEN o. o.oo o. o.oo o. o.oo o. o.oo 2. 6.90 2. 7.41 o. o.oo o. o.oo 

IIIIIOIUTIES o. o.oo o. o.oo o. o.oo o. o.oo 3. 10.34 ,, 3.70 1. 1oo.oo 1. 100.00 

HOHIIIHORITIES o. o.oo o. o.oo o. o.oo o. o.oo 26. 89.66 26. 96.30 o. o.oo o. o.oo 

lf£H IIIN o. o.oo o. o.oo o. o.oo o. o.oo 3. 10.34 1. 3.70 1. 100.00 1. 100.00 

IIEH HOIIIIIH o. o.oo o. o.oo o. o.oo o. o.oo 24. 82.76 24. 88.89 o. o.oo o. o.oo 

IIOIIEH HIH o. o.oo o. o.oo o. o.oo o. o.oo o. o.oo o. o.oo o. o.oo o. o.oo 

IIOHEH M1HIIIN o. o.oo o. o.oo o. o.oo o. o.oo 2. 6.90 2. 7.41 o. o.oo o. o.oo 

TOTAL o. o.oo o. o.oo o. o.oo o. o.oo 29. 1oo.oo 27. 100.00 1. 100.00 1. 100.00 
CAKI, SYSTEHS ANAlYSIS RESEARCH UNIT 

9-JUN-81 REPOFT Hli~P[R SORF£12 

4 OF 5 SCORING IHPACT OH RA!IAR LAB FOR 19BQ ENROUTE !NP•JT l~ ST~RTlHG 10-m-81 

PASS FAIL ~DINS TOTAL 

------------------------ ------------------------ --------~·-

50F5 4 OF 5 5 OF 5 4 or s ----------- ----------- ---------··- ----... ·----··-

• % • % • • • • • • t • h • 

• HEN 21 77.78 24 88.89 6 22.22 3 11.11 0 o.oo 27 

IIOHEH 0 o.oo 2 100.00 2 100.00 0 v.oo 0 o.oo • L 

HIHORITIES 2 66.67 J 100.00 1 33.33 0 o.oo 0 o.oo 3 

HOHHIHORITIES 19 73.08 23 88.46 7 26.92 3 11.54 0 o.oo 26 

COitPETITIIIE 20 74.07 24 88.89 7 25.93 J u.u 0 o.oo 27 

PRED£VELOPHEHTAL I 100,00 I 100,00 ~ o.oo 0 o.oo 0 o.oo 1 

COOPERATIVE ED 0 o.oo 1 !00.00 1 100.00 0 o.oo 0 o.oo I 

TOTAL 21 n.u 26 89 •. !6 8 27.59 J !O,J4 0 o.oo 29 

c:li 
d1 

CAIIIo SYSTEIIS AIIAl YSIS R£S£Mal llllT 

9-.QI-81 R£1'0RT IUIJ£R 80R&E12 

RADM LAB 6RMIE RAIIfiS ff SliB6RtJWS FOR 1980 ENROOTF. IIII'IIT 12 STARTIII6 ICHU-81 

GRADE TOTAL 6RP liEN IIOIIEH "IHORITY IIOIIliiN II£N HIH IIEll IIOIIIIH IQtEII I!IH MIIIIOIII 

RAHG£ ---------- ·-------- __ .,_,. ___ ---·----- ------·-- --------- t % - --- -- • % t % t % t % • % • % • % • % 

9&-100 0 o.oo 0 o.oo 0 o.oo 0 o.co 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 o.oo 

80-89 J 10.34 J u. t1 0 o.oo I l3,3J 2 7.69 I 3J,3J 2 8.33 0 o.oo 0 o.oo 

75-79 9 31.03 9 33.33 0 o.oo 1 33.33 8 30.77 1 33.33 8 33.33 0 o.oo 0 o.oo 

70-74 9 31.03 9 33.33 0 o.oo 0 o.oo 9 34.62 0 o.oo 9 37.50 0 o.oo 0 o.oo 

65-69 6 20.69 4 14.81 2 100.00 I 33,33 5 19.23 I 33,33 ~ 12.50 0 o.oo 2 100.00 

60-64 I 3.45 I 3.70 0 o.oo 0 o.oo I 3.85 0 o.oo I 4.17 0 o.oo 0 o.oo 

50-59 I 3.45 I 3.70 0 o.oo 0 o.oo I 3.85 0 o.oo I 4.17 0 o.oo 0 o.oo 

• LT50 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 1),00 0 o,oo 0 o.oo 0 o.oo 

TOTALS 29 100.00 21 100.00 2 100.00 3 too.oo 2~ 100.00 3 100.90 24 100.00 0 o.oo :l 100.00 

GRADE COift:TITIU PREDEV C£1' ATC EXP HO ATC EXP VET PIIEF 1.'11 VET l'fi:F Ctll N:GW: IKI DEGREE 

RAHGE ------ ---------- -·------- -~-------- --------- ----~ ... - ---- ------ ------- t % • % • % ~ % • % • % • % • % • % 

90-100 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 o.oo 

80-89 3 u.u 0 o.oo 0 o.oo 3 20.00 0 o.oo 3 18.75 0 o.oo 0 o.oo 3 16.67 

75-79 8 29.63 1 100.00 0 o.oo 5 n.33 3 23.08 6 37.50 3 25.00 J JO,oo 5 27.78 

70-74 9 33.33 0 o.oo 0 o.oo 5 33.33 4 30.77 6 37.50 2 16.67 2 20.00 7 38.89 

65-69 5 18.52 0 o.oo 1 100.00 2 13.33 4 3Q.77 I 6.25 5 41.67 3 JO,oo 3 16.67 

60-64 1 3.70 0 o.oo 0 o.oo 0 o.oo I 7.69 0 o.oo 1 8.33 I 10.00 0 o.oo 

50-59 I 3.70 0 o.oo 0 o.oo 0 o.oo I 7.69 0 o.oo I 0.33 1 10.00 0 o.oo 

LT 50 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 o.oo 0 o.oo 

TOTALS 27 100.00 1 100.00 1 100.00 15 100.00 t3 100.00 16 100.00 12 100.00 10 100.00 18 100.00 
APPENDIX !I 

EXM!PLE OF A SID!MATIVE REPORT 

EN ROUTE TRACKING STUDY TEmliNAL TRACKING STUDY 

-------------------------------------- -------------------------------------- CROUP SIZE PERCENT GROUP SIZE PERCENT 

PASS ACADEMY 

FAIL ACADEMY 

NO SHOW OR WITHDRAI! 

TOTAL 

STILL ACTI\~ IN 2152 

{OF PASSES) 

NOT ACTIVE lN 2152 

(OF PASSES) 

TO'l'AL PAS~ES 

TOTAL STILL ACTIVE 

TOTAL NOT ACTIVE 

TOTAL DE\~LOPMESTALS 

!'24 

80 

32 

936 

698 

126 

F24 

691' 

23f 

<;16 

88.0% 

8.5% 

3. 4/: 

lCO.C~. 

84. n: 

15.37 

100.0% 

74.65:. 

25.4~ 

100. Ci. 

FOR irlOSE STILL ACTIVE IN 2152 OPTION 

(N=698): 

L..o\B PHASE Cm!POSITE SCORE 

SUPERVISOR RATING SCORE 

= 81.7 

= 5.1 

CORRELATION OF LAB CQ}!POSITE WITH 

SUPERVISOR R!<TING = .212 

CORRECTED FOR RESTRICTION = .295 

<>US. GOVERNMENT PR!~ING OFFICE: ! 982 369-606j-;'4l 1-3 

29 

PASS ACADEMY 

FAIL ACADEMY 

NO SHOW OR WITHDRAW 

TOTAL 

STILL ACTIVE IN 2152 

(OF PASSES) 

NOT ACTIVE IN 2152 

(OF PASSES) 

TOTAL PASSES 

TOTAL STILL 4.CTI\~ 

TOTAL NOT ACTIVE 

TOTAL !lEVELOPHENTALS 

868 

~5 

19 

972 

789 

79 

86!' 

7!'9 

183 

972 

89.37 

8.7% 

2.0% 

100. o:<: 

90.9% 

9.17 

100.()~, 

81.2% 

18.8% 

100.0% 

FOR THOSE STILL ACTIVE IN 2i52 OPTIU~l 

(N~7f9): 

LAB P~o\SE COMPOSITE SCORE = 81.2 

SUPERVIS0R RATING SCORE = 5.~ 

CORRELATION OF L..4.B COMPOSITE WITH 

SL'1'ERVISOR RATING • 245 

CORRECTED FOR RESTRICTION .334