ED 206 661

_AUTHOR

TITLE

INSTITUTION

SPONS AGENCY,

PUB DATE

NOTE

AVAILABLE FROli

EDRS PRICE

DESCRIPTORS

DOCUNENT RESUME

CE 029 969

Stallings, David; McDonnell, Melinda

Evaluative Research in Local Youth.Programming--Using

the Tools Of Knowledge Development. Knowledge

Development Framework. youth Knowledge Development

Report 1.5.

Osoro and Assoc., Bellingham, Wash.

Employment and Training Administration (DOL),

Washington D.C. Office Of Youth Programs,

May-60

107p.; For related cloCumnts see note of CE 029-

966;

Superintendent of DocuMents, U.S. Governmgnt Printing

Office, Washington, DC 20402 (Stock No.

029-014-00120-1, $4.25).

MF01/PC05 Pldg Postage..

Annotated-Bibliographies; Case Studies; Demonstration

Programs; Educational Assegsment; *Educational

Research; Employment Programs; *Evaluation Methods;

Evaluation Needs; Federal Programs; Guidelines; Job

Training; Models; Reeds Assessment; *(rogram

Development;.*Program Evaluation; *School Districts;

Youth Employment; *Youth Prograis

IDENTIFDERS *Portland Career Survey Project; Youth Employment and

Demonstration Projects. Act

, ABSTRACT

Designeefor use in programs implemented under the

mandate of the Youth Employment-an@ Demonstration Projects Act

(YEDPA) of 1976, this,guidesinelhdes gurdelines and models for,

conducting the evaluation, research, and demonstration effoits that

local youth staff are expected to carry on in their own local YEDPA

programs. The evaluation, research, and demonstration activities . called for under YEDPA'are discussed, and guidelines are set forth

for their implementation. Presented next is a model local knowledge

development case study--the Portland Career Survey Project A

developmental sketch for conducting program.evaluation research,

guidelines for applying an evaluative research framework the

knowledge development process, and comments on the techniqu s of

evaluative research.are provided. Discussed in a-section on -1 understanding national knowledge development are the learni g focus

of YEDPA and an examples of discretionary project rationale. An

annotated bibliography follows. (Related youth kn6wledge and

development reports are available separately through ERIC - -see note.)

(MN)

aL e ***************************************************4**********

Reproductiong supplied by EDRS are the best that can be ma

from the original -document. ***************************************************************

4
.

O

.

YOUTH KNOWLEDGE DEVELOPMENT REPORT 1.5

I

42

EVALUATIVE RESEARCH IN r\

LOCAL YOUTH PROGRAMMING -

USING THE TOOLS OF KNOWLEDGE

DEVELOPMENT

David Stallings

Melinda McDonnell

OsorO and Adsopiates

April 1980

For sole b2 the Superintendent of Documents, U.S. Government Printing Office

D (' 20402

US. DEPARTMENT OF EDUCATION NA ONAL INSTITUTE OF EDUCATION

EDU ATIONAL REpURCES INFORMATION

CENTER (ERIC)

This document has been reproduced as

received from the person or organization

onginating

[ Minor changes have been made to improve

reproduction quality

Points of view or opinions stated in this docu

ment do not necossanty represent official NIE

position or policy
c

A

U.S. Department of Labor

Ray Marshall, Secretary

Employment and Training Administration

Ernest G. Green, Assistant Secretary for

Ehiployment. and Training Administration

Office of Youth Programs

I

-.. Material contained in this publication is

in the public dcfrnains and may be

-reproduced, fully or partially, without

permission of the Federal Government.

Source credit is requested but not _ required. Permission is required only 4

-to reproduce any copyrighted material

contained herein.

r \

.

\

,.

. ..

*

I

>

r)t)

,

i.

I

..

I

4

.

I--
CONTENTS

OVERVIEWS

.

WHAT IS KNOWLEDGE DEVELOPMENT? /-

1

Understanding the KD Challenge

Toward a Working Understandi/g of'the Knowledge Development Process

A LOCAL KNOWLEDGE DEVELOPMENT CASE STUDY': THE PORTLAND

CAREER SURVEY PROJECT

18

EVALUATIVE RESEARCH AS A KNOWLEDGE DEVELOPMENT TOOL 3 5

Program Evaluation Research: A Developmental Sketch

Applying an Evaluative Research Framework to the Knowledge Development Process

Comments on the _Techniques of Evaluative Research

UNDERSTANDING NATIONAL KNOWLEDGE DEVELOPMENT 74

The Learning Focus of YELIPA

Ancxample of Discretionary Project Rationale

SELECTED REFERENCES

Is

Cr.

88
/ OVERVIEW

4

In A "'Planning Charter for the Youth Employment and Demonstration. Projects Act of 1977, the Office 9f Youth Programs articulated the general principlqs involved iri-the implement'ation

Of the new youth initiatives: The first principle was as

follOTT:

'!Knowledge development is a primary aim of the new

youth.programs. At every decisiOnmaking level, an

effort must be made to try out promising ideas, to

support ongoing innovations and to assess performance

a :rigorously as possible. Resources should beconr

aprlying tapas

can be given a reasonable test. Hypotheses and

questions should be determined at the outset, with an

evaluation methodology built in. This does not mean

4 tha-rvery youth-program must be experimental, but

rather tht 'we should move as far as possible in this

and locally."

Nationll ]snowledge development objectives, supported by the psi nificant dis6retionary authority under YEDPA,. were outlihed in

a series

rected

issues

meat

that

of k owledge development plans. The centrally dinowle ge development activities ,focused on those

requiring multi-site demonstrations, rigorous assessnd national data bases. However, .it.was anticipated

CETA primp sponsors, withthe new resources provided

and /r YEDPA, would try,dif ent approaches in their lodal Con -

te bsand would more careful y monitor and evaluate existing

pr grams in order to see hol. they could be improved: Rigoro s experimentation at the local level wag not anticipated

ecause of the difficulties of experimentation

in an operational setting and the-lack of resources locally

. to achieve adequate sample 'sizes. It was assumed, however,

that prime sponsors could increase their research and evaluation activity as well as trying out some new ideas.,

/ . ',To achieve this aill:71 the grant package and regulations for

Youth Employment afid Training Programs required prime sponsors,

to identify previous knowledge development activities and to

outline plans for new ones. End-of-the-year narrative reports

were requested tp summarize the findings of these activities.

* An evaluation of local prime sponsor YEDPA performance by the

NatiOnal Council on Employment Policy concluded that some sig-',

nificant activity was generated by this mandate:

A

,...1010
1'N

N r "Under thegun to do something, most sponsors responded..

Some repackaged evaluation work they-had'aIready' been A doing and called it knowledge development. -MoSt made

an attempt at some kind of explicit evaluatiOn experience;

some were quite' elaborate. A few got away with doing

nothing. On the balance, though, theresearch aid

apalysis mandate produced a.surprising amount.of new

local activity.... WhAt is especially romisin 'is the

fact that in th second year of the yob progr ms, local

sponsors', still lacking specific patio ffice,guidance,

took the initia ive in judging the mev. of their knowledge development projects and frequently completely

rerped them."

The NCEP evaluation noted a1,50._ hat____t_prime_sponsors-have been--- almost entirely on their owft when it has'come to developing

research, demonstration or evIaluatiQn initiatives, or testing

innovative prograffi%ideas.", There was very little guidance - from the Office of YouthPrograms on this issue_ in contrast

to the carefully structured nationally-directed knowledge

development efforts.

1

Evaluation Research In Local Youth Program Development provides

such directidn for prime sponsor activities. It is, in essence,

aframework for local knowledge development planning and.impleorientation. Concurrently, there has been an evaluation of local

knowledge development.activities to identify models, and an

of ort to estabp.sh technical assistance capacity to provide

de ailed hands-on,guidance to interested prime sponsors. This

monograph desdribes not only what prime sponsors can initiate.

locally, but hock they can utilize the products of nationallydirected knowledge development activities.

This volume is, itself, one of the many products of the "knowledge development" effort implemented under the mandate of the

Youth Employment and Demonstration projects Act..of 1977. The

knowledge development effort consists of hundreds of separate 1

. research,, evaluation and demonstratiOn activities which will

result in literally thousands of written,products. The'activities have been structured from the outset so that each is

self-standing butalso interrelated with a host of other activ-,

itiei. The framepork is presented inA Knowledge Development

Plan for the Youth ,Employment and Demonstration Projects Act of

1977, A Knowledge Development Plan far the Youth Initiatives

-fTgEal 1979and Completing the Youth Agenda: A Plan for Knowledqe'Development, Dissemination and Application in Fiscal 1980.

-*
I'

Information is available or will be coming available from. the

various knowledge development activities to help resolve an

almost limitless array of issues, but answers to policy'

questiOns will usually require integratiOn and synthesis

from-a number of separate products, which, in turn, will

depend on knbwledge and availability of these products. A

major shortcoming of past research; evaluation and demonstration activity has been the failure to Organize and disgeminate the products adequately to assure the full exploitation

of the findings. The magnitude and structure of the youth

knowledge development effort puts a premium on organization

and dissemination.

As part of its knowledge development mandate, therefore, the,

Office of Youth Programs of the Department of Labdi will organize, publishland dissevinate the written products of all major

research, evaluation and demonstr'ation activities supported

directly by or mounted in conjunction with the

development effort. Some' of the same products may also be

pbblished and disSeminated through other channels, but they

will be included in the structured series of Youth Knowledge

Development Reports in order to facilitate access and integration.

The Youth Knowledge Development Reports, of which this is'e,

are diyided into twelve bra categories:

1. Knowledge Development Framework: The products in this

, category are conceied with the structure of knowledge AeN'ielopment activities, the assessment methodologiesNwhich are employed,

validation of measurement instruments, the translation of knowledge into policy, and the strategy for disseininating findings.

2.' Research on Youth-Employment and Employability' Development: The products in this category represent nalys s of existdata, presentation of findings from new da a sources, /

special studies of youth labor market pFoblems'and policy

analyses.

3. Program Evaluations: Tite products in this category' . include impact, process. and benefit-caSt evaluations of youth

programs including-the Summer YOuth Employment Program, Job

Corps, the Young Adult Conservation Corps,' Youth Effiployinent and

Training Programs, Youth Community Conservation and Improvement

Projects, and the Targeted Jobs Tax Credit. .4 4. Service and Participant Mix: ,The evaluations and

qemanstrations summarized in this category concern the matching,,

of different types pf youth with different service contination..

This involveS experiments with work vs. work plua remediatibn

vs. straight remediation as treatment options. It also4includes

attempts -to mix disadvantaged and more affluent partigipants,

as well as,youth with older workers.

le - r--t
fo

5: Education and Traininv Approaches: The products in

this category present the findings of structured experiments

-to test the impact and effectiveness of various education and

vocational training approaches including specific education

_methodologies for the disadvantaged, alternative education,

approaches and advanced career raining.

6. lire- Employment and Transition Services: The products

in this category present the findings of structured experiments

to test the impact anal, effectiveness of school-to-work tran--

.sition activities, vocational exploration, job-search assistance and other efforts to better prepare youth for labor market

success.

/.. Youth Work Experience: .The productS in this -category

address the organization of mirk activities, their output, productive roles for yOuth and the impacts of various employment

approaches.

8. Imp3,ementation Issues: This category includes crosscutting analyses of the prpctical lessons concerning "how-toISSues such as learning curves, replication processes\

and 'programmatic "batting averages" will be addressed under

this category, as well as the comparative advantages of alternative delivery agents.

9. Design and Organizational Alternatives: The products

in this category represent assessments of,.demonstrations of

alternative program and delivery arrangements such as consoll.-

dation, year - round, preparation for summer programming, the upe

of ihcentives and multi-year tracking of individuals.

10. Special Needs'GroupA: The products in this category

present findings on the ipecial problems of and adaptations

needed'for signifiCant segmentsincluding minorities; young

mothers, troubled youth, Indochinese refugees and the handicapped. . 11.. Innovative Approaches: The products in this category

present the'findings Of those aotiiyities designed to explore

new approaches. The subjects coveted include the You

Incentive Entitlement Pilot,Projects, .privat, sedtor initiatives, the national youth'service experiment, and energy

initiatives in weatheriztioh, low-head hydroelectric dam

restorattOni windpower and the like.

,

. N 12. Institutional Linkages: The products in this category ''_,: will include studies of institutional arrangements'and linkages

'-'''* as well as'assessments of demonstration activities to encourage

such linkages with education, volunteer groups, drug abuse

. agencies and the like. . 1,

iv

-s
In each of4 these knowledge development cdegories, there will

be'a range of discrete demonstration, resdarch and evaluation

activities, focused on different policy, program 'and analytical

) issues. For instance, all experimental demonstration projects

have both process and impact evaluations, frequently undertaken

by different evaluation agents. Findings will be published as

they becoMe available so that there willcutually be a series

of reports as evidence accumulates. To organize these pro-

'ducts, each publication is classified in one of the twelve

broad knowledger development categories, described in terms of

'the more Specifi6 issu#', activity or cluster of activities to,

which it is addressed, with an identifier of thq product and

what it represents relative to other products in the demonstration. Balca, the multiple products-undera-knowledgedelielopment activity are closely interrelated and the activ-

. ities in each broad cluster have si4hificant interconnections.

This framework for local knowledge development activity is

closely related to aTl of the products in the'"knowledge

development framework" category. It'should'also be

assessed in conjunction with Youth and the Local Employment

Agenda in the "program evaluation" category as, well as

Local Mixed Income Testing in the "service and participant

mix category" which assesses the success and problems prime

sponsors have had in pursuing the knowledge development

mandate.

6

ROBERT TAGGART

Administrator

Office of Youth Programs

- v
1

Nib

JP

,

a

CHAPTER I

WHAT IS KNOWLEDGE DEVELOPMENT?

I fl

0

s.
CHAPTER I

WHAT IS KNOWLEDGE, DEVELOPMENT?

C Understanding the KD Challenge

A primary emphasis of the youth employment and demonstration Projects is on

I

learning about the ipature of employment problerrv, among youth, and about the

appropriateness and effectiveness of a

problems. The hope of DOL4 is that

evaluation, and demonstration projedis

more effective in the long run.

variety of strategies for dealing with these

a few years of well-documented research,

wild yield informatiOn _for making programs

This learning rpOnent of the 'Youth Employment 'Demonstration Projects Act X

(Yt'DPA) was dubbed "knowledge development" at the inception of the Act. The first

of terpprincipl s in the YEDPA planning charter stated that a principal objective of

YEDPA is to develop knowledge from the youth initiatives about how to best assist

youth. 40. I

.11 It is crucial for local youth staff to realize that KD has a dual focus: the national \-

level and the local level. At the national level, the Office of Youth-Programs (within

the.Enwloyment and Training Administration) is overseeing a broad, structured array

of multi-site demonstration projects, large scale evaluatioris, and .coordinated research

efforts, all funded with discretionary funds. 4

The largest of these discretionary projectt is the Youth Incentive EXtitlement Pilot

Project (YIEPP), for which 17 prime sponsors across the country were 'selected to

operate every strudtured pilot projects testing the effects of guaranteed jobs on school

6

e 1
1

dropout rates and youth, employment behavior. Although directly funded and directed

by the national office, most Of the over 200 demonstration and evaluation activities

were conducted, at local prime sponsor sites. The results of these findings will be of',

great importance to prime sponsor youth staff. As of FY'80, a, principal focus of the

Office of Youth Iograms is to synthesize and disseminate the results of discretionarily funded projects to prime sponsors. National knowledge dehlopment organization is

further discussed in Chapter 4.

The other aspect of,knowledge de,velopment activities, that which is emphasized_in

[thiS guide, is the evaluation, research, and demonstration effort all local youth staff

are expected to carry on in their own local youth programs. DOL hopes, on the one

hand, that these KD findings will benefit individual local youth staff immediately and

directly by increasing their understlnCling of the youth employment problems and

programs in their respective areas. On the otherhand, it is also hoped that these local

knowledge developFnent efforts will encourage local administrators to institutionalize

processes foa4seSsing youth programs and utililing evaluation findingsIKnew local

policy formulation. However, it is important to note that these hopes for local

knowledge de,velopment efforts stem prirriarily from the federal perspective of what

prime sponsors should be doing. In fact, local youth staffs' inter pftations of the

value, functions, and purpose of local knowledge dev4ment have often varied

considerably from that of the national office. A brief examination of prime sponsor

reactions to, and experiences with, KD in FlY '78 and FY '79 may assist the reader in

grasping the principal pitfalls, as well as promises, of local knowledge development.

Two of the most important sets :of realizations.about local conduct of knowledge

developrper4 activities Concern making methodOlogical and locally relevant decisions.

As a general rule, less elaborate, basic approaches to youth prpgram assessment often \ ,

2

04
.provide more fruitfUl results than the use of more sophisticated approaches. In FY '78,

many prime sponsors and subcontractors interpreted 'knowledge. development as a

mandate to do elaborate social experiMentation. Such experimental social design

often included experimental Bind control groups of participating- and non-participating

youth. Such app\-oaches often included the10% mixed inome test, which examines

, the notion that mixed income participant backgrounds provides for improved youth

outcomes. Geherally speaking, these more sophisticated experimental- approaches to

assessing youth program outcomes Were not successful. This is not to say that some

good 'ideas were not, Being, ifispected, Alitt the manner of inspectioyr Was often

.

ineicropriate and did not 4/ork. For example, youth staff foUnd that inAlecentralized

balance of state prime sponsorships, it was not practical to setup tight experimental

condition's% More dibtralized prime sponsors, on the other hand, discovered the

difficulty of maintaining sufficient co trol group size. Youth staff also found that

experimental models were of ten.awkward and unrealistic because staffs were not able.'

to control, or even adequately measure, non-income variables. Then _too, youth

planners found that findings from the experimental approach were often too late to-be

r

politically useful.

L.< Learning from their mistakes, many primes and subdcontrActors took simpler, more

basic approaches to youth program assessment in FY 79. Less elaborate approaches

often proved to be more manageable, more likely to provide locally useful information,w

and to serve as building blocks f or expanded and/or slightly more sophisticated

projects.

A second, related lesson of knowledge development for local youth staff in FY'78 and

FY '79, is the importance of focusing local youth prbgram evaluation interests on

. variables under the control of the local prith sponsorship. A local learning focus

e 3
,.1

provides year to year continuity and cumulative progress in local knOwledge development effOrts. By -contrast, less fruitful result's were provided by local knowledge

development efforts investigating questions el mainly academic interest or broad

policy issues, which could principally be settled only at the federal level (e.g., 10%

mixed income tests).

'.14

Benefiting from these lessons, this guide emphasizes the utilization of relatively more

straightforward approaches to youth program assessment and evaluation, although it

also contains ;advice and instruction in Conductingmore sophisticated experimentation.

In particular, the guide presumes that the principal 'reason for local youth staff to

conduct knowledge development activities is for the purpose of supporting locality

initiated youth program policy change. The guide's message is equally applicable for

youth employmerft and -training program, staffs in CETA prime sponsors). school

districts, CBOs, various LEA and CETA subcontractors,and similar agencies.

The model approach to knowledge development, discussed in the following section, is

based on many observations, of how local youth staff have successfully approached the

learning cycle of youth program planning and design, program delivery, assessment,

and redesign. The general approach to KD described here places higpest priority, on

the development of realistic and useable information for rocal policy purposes, and

suggests that the choice, Of specific 'evaluation tools and approaches be largely

/

dependent on how much is currently known about-the specific local program beirig

assessed, as well as on what is Actiial with given resources and expertise. I

1 .

-4,

4
e

11* Toward A Working UMerstariding of the Knowledge Development Process

A general model which syntheSizes And summarizes the approach that many local

youth staffs have suc*sfully used is presented and discussed in this section. Ah

understanding of thismodel at the outset should give the reader a general background

of the knowledge ,development process, as well as a feel for ,key decision points along

the way. A case study example'of a prime sponsor's utilization of this knowledge

development model through several program development cycles of a particular youth

program is4ubsequently presented in Chapter 2. Some readers may wish to consider

this soecific"case study prior` to examining the more general presentation of the basic

KD approach as discussed in this section. In either case, it should be recalled that the

model presented here is based on analysis and observation of the ways many loCal

youth staffs have sticcessfully.approached learning from their youth programs. The

basic approach which emerges may be utilized on many different levels, from the

simplest and most straightforward youth program development efforts of very small

rural prime sponsors or subcontractors to the most sophisticated and research-oriented

ones of large urban primes, LEAs and CBbs. The basic logic of the approach remains

the same.

N.

At the ':-.. level, knowledge development is probably best understood as belffg-Z/-1

, .. synonymous with what most youth coordinators think okas "youth program development." As shown in Figure 1 -1, the basic program development process usually

,consists of:

1. Planning. and 1esign: Identifying the problem, setting goals for problem

solutiorfs, examining alternative solutions, and designing a program of

r.

youth services-:-

2. Implementation: Implementing the program of youth services

5I
:,

FEEDBACK:

Application

of

Findings

PROGRAM PLANNING' AND DESIGN

1. Prbblem Identification

2. Goal Setting.

3. Examination of

alternatives

4. .Program Design

t

t

, PROGRAM ASSESSMENT:.

M onjtoring \,and ."

Evaluation

(

1

PROGRAM , \

>IMPLEMENTATION ,

.

FIGURE 1-1: THE PROGRAM DEVELOPMENT PROCESS

.

-s6

. s

0 r

if

,

\
3. Assessment: Monitoring and evaluating the Ozlementation and results of

4

the youth services program

, 4. Feedback: Applying the results of monitoring and evaluation findings back

into the original problem identification and allowing for subsequent_modification. The' cycle may then be repeated, though at a more developed level

of experience (or knowledge base): Learning has occurred. Y

In knowledge development, considerable emphasis is often placed on the 'program

evaluation component of the development cycle. Even so, it is crucially important to

understand that all the elements of the local youth program development ccle must

fit together as parts of a w1lole. Thus, the program development elements must link

and interact in a cOmpatikle way. .

-- For example, it. would not normally be wise to design and demonstrate a new career

awareness program, then narrowly evaluate the implementation phase by experimental

comparison of precise partic ant test scores of some sort'. In such a case, our

evaluative approach would be "out of sync" with the developmental level of our

program, planning design and implementation. Instead, we might prefer to evaluate the

demonstration implementation by using simple quantitative measures (planned vs.'

actual enrollments, terminations, plicement rates, expenditure rates, etc.) along with

qualitative indicators (interviews with counselors, participants, others).

T cumulative learning approach to youth program development modeleq in Figure 1-

-2 emphasizes the ifflportance of fociising on locally relevant program questions, as well

as appropriately tailoring assessment style to the stage of program development.

Youth planners, typically, are immersed in an ongoing system of program development.

7 -72
Information

Network

Programs

I

Identification

of

KD Focus A

flow much do we currently know

about this KO Focus .area? (State

of local knowledge base) Select

principal learning level.

' A LITTLE ... ... A LOT

4 4

2 3 4. 5 6 7 8

Program Assessment Asseisment Process In-Program Outcome of Outcome/F.U. Impact eval- Model of Program of Program Evaluation Assessment participant Evaluation uation of identifi- Model De- ModePAtodi- of Model of Model benef ts.fit of longer effects on

cation or monstratlon ficatioti program? term parti- youth Creation , -

termination cipant benefits

community

Local Youth,

Labor Market

Analysis (are

programs addressing current reeds

Current

Youth Program

Assessment

Start over;

Continue

1

I

PROCESS FOCUS

YOUTH PROGRAM ASSESSMENT

(EV4LUATIVE RESEARCH: SEE FIGURE 3-2)

OUTCOME/IMPACT FOCUS

More

Quail tative

Subjective ,4

Judgmental

(METHODS SPECTRUM' ' More

Quantitative

r"ii3Objective Statistically

Reliable

Youth Program

Findings

FIGURE I-2:, A MODELTOF THE I..0C/}L KID PROCESS

1
However, for purposes of understanding the model KD approach, we can break into this

continuous cycle of program development at the point labeled in Figure 2-1 as

"Current Youth Program Assessment." Here we are examining the findings of how

Well our present y.,cuth programs are operating, their cost effectiveness, how well they i

are being received by youth participants, an, as far as we can determine, the impact

these programs are having on participants.

Next, we analyze the local youth labor market, asking ourselves such questions as:

Are the youth programs being offered addressing current youth needs in'the community? Are changes in the youth community, regulatory changhs in target group

identification, or other circumstances creating new needs and, perhaps, new unknowns?

Usually problems or questions we need more understanding about Aril' emerge from this

effort to clarify and focus our youth program offerings. Here is the raw material -tint

of which we can begin ,to identify the knowledge development focus which is "right" ti

for our jurisdiction. Althotigh the focus area stems from our assessment of current

Offerings and needs in local youth programs, we will, of course, bring together a broad

. array of external tools and resources to meet this challenge. For example, we may

pull -upon past experience in our own staff for youth programming.. Eachexperienced

you-th program staff member involved in the plInning effort will most likely draw upon

experiences and lessons learned ip somewhat similar situations elsewhere. Then too,

s .. there is the ever changing, ongoing sense of understlanding abut what works and what

,.. -,., . , does not work in youth programming which is shared by all employment and training

r youth program staffers. Although this "invisible colge" or body of prolesgonal 1 understanding thrives and is communicated in.a number of ways, pro ly the n?ost

important method utilized by youth staffers in obtaining this information is informal,

ti
communication with colleagues in other jurisdications or Ibcations. A "Knowledge

P. Development Interest Network" now exists to help energize this ongo ng, evolutionary_

network of understanding about what does and does not work in youth programming.

By tapping into this body of professional understanding when identifying our own local

youth program needs and knowledg4 'dev'elopment focus, we can avoid re- inventing the

youth programs wheel.

In reviewing, prioritizing, and selecting our specific KD foc>'we may find it useful to

distinguish between questions of "program" focus and "variable" focus. That is, are we

e

principally interested in the purely practical outcomes Of a specific program at a

specific time (program questions), or are we more interested in singling out specgic

5 components of the program as more generalizable stimuli ?. In other words, if, we can

- focus upon learning about the genrral variables underlying a program (rather. than upon

fhe ffectiyeness of the programilas a whole), we may produce findings of greater local

significance. This point is especially relevant tq learning from demonstration projects.

An example May help illustrate this point. Suppose we wish to learn about a youth

career info?mation center demonstrated in a local high school. Measuring the _number

of center users (for example') will be necessary' for reporting purposes, but may help is

learn only about this specific center. However, if the centeris set up to test some

specific principle or variable in youth career center establishment, say, the relative

effectiveness of personal versus formal appeals for attendance; our 'results may have

greater transferability o other local career center situations*.

*Thus emerges a possib e difference between "Knowledge Development" and program

'evaluation". The development of relevant local knowledge -tends to stress ontinuity

and cumulativeness of youth program findings, while purely conventional program

evaluation is often a one-shot proposition. The difference between the two is found in

the, rationale, focus, and definition of what we wish to learn, rather than in the

specific methods and techniques of study.

10

a
11. A Assuming the. identification of a realistic and locally relevant learning focus for our

youth programi, we must consider how much we actually know aboytt this particular

local KD concern. This importknt consideration will obviOusly have a great impact on

the way we approach designing the program of youth services to be offered, as well as

, the way these services are actually delivered. For example, if we know very little

about a new CETA-LgA linkage we would like to establish, it is likely that our first'

order Of business will be to select a program ,model or approach to the possibility of

this new linkage. The source of this program model might be our unique local insight

into local needs, orit might represent an "imported" idea or variation of an idea

' borrowed and adapted from another prime sponsor:* Next, we would probably design

and implement a model demonstration of the CETA-LEA connection we wish to try

out.

It may be that we have already gone through an original program development .cycle

demonstrating this new institutional linkage, in which case, it is likely that unless the

004 demOnstration was clearly unsuccessful, we will be interested in running a modified

version of the original model or approgch. This-modification will incorporate what we

0 have Mimed from the initial model.demonstration. The planing-delivery-feedbackmodification cycle Will continue, each cycle adding to our understanding, of the

evolving CETA -LEA approach that works best for Us. As our unddrstandin becomes

more and more complete, we are able tb a-sk

---and answer more and more -sPecific (---7"7- -. - --

.< . 0 a /'

questions about theina re of what works in our CeTA-LEA coordination eff or; and

.

. . what doeS not.

,.. .. ia. .:

, . - d . . ,

. *E.xportation and inipbrtation may occur in vaelous ways: e.g., through the Knowledge ' Development Interest Networ , through DOL Regional federal repFesentftiyes or

youth coordinators, through t e Office of Youth Programs, through, prime sponsor

, associations, through the youth program "information broker" described in Chapter 4,,

or through more informal professional exchange.

,

......... , ,

11

21

C

IrN
V

;Thus, at each program development stage, we consolidate and apply ,what was formally

and informally learned frcom the last development round, finding ourselves at a sort of

frontier or margin of understanding about the particular situation. The, next bite, or

increment, of unknown area to be explored at this frontier' cbntains future Candidates

for knowledge 'development which will be considered and prioritized for our next round

of program ilevelopment. We move from a relatively _general to an increasingly

particular and complete development of understanding about the particular youth

program being considered.

Identifying the type of knowledge development inquiry which is appropriate for a

prime's specific yoUth program demands an accurate understanding, of how much is

collectively known locally aboUt 'the particular program- approach at the .time. The

more we-know about the program, the mgre specific and precise we car be in the

nature of the questions we ask and answer about, the pwgram. In some ways, this

question fitting Process is comparable to deciding how far but ?n the ice you-should

-walk on a newly frozen pond. As you walk closer to the middle of the Pond, you leave

more solidly frozen ice and risk a dunking. So too, if w&'attempt to ask and answer;

. overly specifid questions about a youth program prior to developing a firm, understand

ing of how the process works, we risl procedural failure in Opr analysis or, at least, the

./

possibly of collecting information which is erroneous or 4eless.*

) 6

In Figure 1-20 eight sequential levels of evaluative focus are identified which

correspond or are parallel with a general spectrum of prograin development. Although

. the sequencNy be reorderd to fit specific local circumstances, the progression

shown depicts a logical,- essentially chronological order of assessment foci. For a

given developmental stage of a progt1m, it is likely that a local youth staff will find it

appropriate to orient its assessment around some point along thii spectrum. Positions
4.4 111 It

,

.. i t

shown in Figure 1-2 include:

.0'

)

1. Program identification or creation. Assessment ,of d model program

r ,. approach which has either been internally generated or modified from an

.4.

approach used elsewhere. . 2. Program Model Demonstration. Process evaluation of initial model demonstration.

3. Program ModerModification. Predominantly a process oriented evaluation

i

. of a modified version of the original demonstration model. Some sample

outcome measures may begin to be used at this point. /

4. Process Evaluatipn of the Model. Possibly done separately from Step 3, if

/ ,

I

circumstances warrant a general progfam implementation assessment. If

J .-.. included _outcome measurement is still simple and de-emphasized; but it is d,_outcome

V likely to be of increasing importance and credibility. )

* A process study provides information about the meanuanployed-iti a program to

produce specified outcomes. Especially important during the developmental phases of

a model or program component, process studies are often, employed before the

program as a whole can be described. Thus, process stualei of program.implementa -

tion are com,rnon, examining, for example, attendance ratei, component completions,

staff-participant ratios, rates or types of attitudinal change, termination by reason,,

etc. Process studies can not only help develop the program, but they also help to

conceptualize what the,program is and how it works. s

An outcome study describes the results of the 'program process on program participants. Outcome studies are typically with producing a summary statement.

about the effectiveness of the program or some phase of it. Both the audienve and

titling involved in outcome evaluations will be different thdn in a process study.

..' v Impact studies try to measure the overall effect of a program, including the

achievement of goals in behalf of the target group, as well as the difficult-to-measure

side effects of the program on the general population or community. Some evaluator's

terminology doesil't distinguish between "impact" and "outcome" measures, typically

lumping both under the term "impact."

. . An evaluation' study often blends outcome and process. It may be useful to haveis

process data to assist in the interpretation of outcome results; so, too, it may be

appropriate to incorporate an outcome measure into a' predominantly process oriented

study. Nonetheleis, the basic orientation of an evaluative study as a whole is like to

fall into one category or the'other

.

4,

I'

Me

4

Y.

Nik

/

""l
0

5. In- Program Assessment of ih(Model. Evaluation of the interactional

effects beteeen the study progrik component or program and other, ongoing, components or programs. May be part of Steps 3 or '4. Example: A

new intake approaclitras been tested "in isolation" in earlier steps of the

sequence; now we concentrate on the "ecological" effects of the new

approach in the total piogram environment.

6. Outcome Evaluation of Participant Benefits At Prfgram Termination:

Ideally, we now, have a rich ';,process II undei-standing of the -program

. , approach as it has developed. We now begin to focus predominantly(on the

measurement of outcome effects of. the program on its participants, though

we continue to make,process observations. Our process/outcome balance

has reversed its position from that in Steps 1 and 2. Rather than just

drawing inferences, we may now wish to move toward answering program

outcome effectiveness questions in terms of how sure we are that our

.?rogram actually causes the outcomes we claim.

7. Qutcome , Erluation of Participant Benefits Longer" Teerelcenefits:

Similar to Step 6, but involving follow-up of participants over some pertod.

Impact Evaluation of Effects on the Youth Community: Our focus turns to

, estimating or measuring the "ripple effects" of our program on the larger

local youth community or- the local geographic community of which the

program participants are part.

Within a -single program development cycle, youth staff may choose to focus ,their

learning effotts on more than one of these levels, simultaneously or sequentially. An

obvious. limitation on the scale of the learning effort is resources time, money, and

1

expeitise. However, it is important to realize that an inherent, "horse before the

cart" logic acts as a delimiter as well. For example, it generally doesn't make sense to . narrowly measure career guidance outcomes prior to first answering such questions as

14
wi!

"Does our new program include career guidance at all?"

The rate of progression through the learning/assessment spectrum w111 also b,e

delimited by local youth staff resources and by the practical length of a program

Ldeyelopment cycle. For example, two-week cycles of intense remedial reading may

A

pass through, multiple developmentnearning cycles' a more elaborate program of

services passes through only one cyc17.

Just as clarification of our program's stage of development helps determine the proper

1 assessment styls(aleng,the process-oUtdome evaluation spectrum), so, too, sloes this

./

clarification have implications on our choice of assessment methods.

As will be discussed more completely in Chapter 3, process oriented levels of inquiry

are often most appropriatley approached using predominantly qualitative, subjectil/e,

And often judgmental methods of analysis. However, this is not to suggest that simple

quantitative data'should not be collected at this early stage. For example, quarterly

characteristics summaries and basic MIS data will always be collected for any program

and will, typically, be of great use in process evaluations. More precise types of

quantitative measures (such as carious test results) are necessary to Answer outconlb

evaluative questions* Thus, kl those levels of inquiry where our principal focus is on

outcome or impact measurement, we will predlorninantly be interested in collecting

preciseyquantitative information although even then we may continue to have an

interest in collecting at least some qualitative information to provide a cor4ext,of

understanding for_our more precise quantitative findings.

47

Regardless of which particular integration of program development stage; assess-

. ment/learning focus, and method orientation makes sense for a given youth program,

youth staff will generally apply a basic evaluative research approach to come up with

15

tiv
L ry

.6

their findings. This evaluative research logic, further discussed in Chapter 5, involves

goal analysis, methodologcal considerations, a research plan or design, and documen-

. tation.

The key to the entire local knowledge development `effort is in the Utilization of

. , . results. If assessment res Its are not utilized, there is very little reason to obtain

them. Much of the remain er of this guide more specifically explores the applicatioh

of assessment<4methods tio the systematic aspircts of learning from local youth

programs.

16

A.

4

a

I

t

a
.

.............,

.

,

*

CHAPTER II . A LOCAL KNOWLEDGE DEVELOPMENT CASE STUDY:

THE PORTLAND CAREER SURVEY PROJECT

/

) ,e

27

c

)

,,.

,

..-

.),

* ,

a

ti
CHAPTER 2

A LOCAL KNOWLEDGE DEVELOPMENT CASE STUDY:

THE PORTLAND CAREER SURVEY PROJECT,

This case study was selected to illustr.ate how the knowledge development process

occurs./ Althgh the composite of locally generated knowledge development subject

areas is vast, this case study represents one prime sponsor's attempt to focus local

learning efforts on a single program category to determine the issues4to concentrate

on, design an approach to deal with the identified needs, and demonstrate, assess, and

continually refine, the approach.

The case study is representative of the youth program development process because it

illustrates how the insights and knowledge gained from each pr:ogram cycle are used to

continually modify the prOgram approach so that it better meets the needs of the

target population, and because it adds to the local youth staffs' understanding of which

approaches work best for which groups of youth. The case study illustrates how a

prime sponsor, in its attempt to upgrade programming and be responsive to the,

dynamics of youth employment and training works'through the knowledge development

pi;otess (Figure 1.2: A MODEL OF THE LOCAL KD PROCESS).

The rationale for utilizing a 'case study is based upon the reality that specific,

observable processes and stages of program develOpment do ,occur: in the real world of

youth employmgnt and training, and that those processes occur systematically and can

be captured, documented, and utilized to further enhance the effectiveness of local

youth programming.

'The case study depicts a chronological progression in program development and scope

and describes the various phases of the local knowledge development process. Specific

18 25
emphasis is placed on the elements of youth program assessment. The importance of

focusing on rocess-oriented questions during the early stages of program development

and implementation is exemplified, as is the appropriateness of focusing on outcome

and impact questions during the latter stagies of program development. The usefulness

of qualitative and quantitative assessment techniques in seeking answers to "how a

program works and why" sand "what types of benefits or impact this program has on thee

youth participants" is highlighted. The pu'rpose is to show how low-cost, simple, and

effective assessment methods can be combined with easily gathered numerical data to

tell us what is working well within a program and which program elements are not

bringing about the desired results. The case study also underscores the vital role of

"goals analysi" in moving through the knowledge development cycle in an orderly and

systematic .way.

The Portland Career Survey Project was selected because of the availability of

information and documentation with which to track the knOwledge development

process; from the program's, inception (1974) to its current state. In addition, the

program has passed through several iterations of the KD Modely Processand has been

replicated in similar settings. Moreover, the Portland Career Survey Project represents a prime sponsor's practical, although imperfect, effort to learn from, develop,

and refine youth programming strategy in a low-cost, effective manner.

PROJECT PROFILE

The Career Survey Project is a career awareness program designed_ for 14 and 15 yearold economically disadvantaged youth.. The Project, designed by the City of Portland

Oregon, Prime Sponsor provides hands-on career experiences and career ,information

through, direct exposure to 10 career cluster areas: office occupations, small engine

repair, electronics, data processing, communications /broadcasting, food services,

patient care, graphic arts,.construction technology, and engineering. The project is

19

2D
1/4 operated during the summer in a community college setting.

The project serves youth who normally do riot take advantage of the career ethlation

training available in public high schools. By identifying these youth and giving them

the opportunity to earn money while being exposed to aspects of ethployment in 10

career areas, it is anticipated that they will be better able' to make career and

educational decisions affecting their futures.

Fektures of the Career Survey Project

o Allows a prime sponsor to create well-structured, substantive suMmer career

exploration experiences for 14-15yeir=old youth

o Youth like it

o Low -cost; easy'to administer

o Provides youth with career experiences fulfilling state And local academic credit

requirements

0

e

o Effective for youth with short attention spans and basic skill deficiencies /

o Easily replicated (the ProjeCt has been replicated at two community colleges;

1 . ... -

the project, design was used as the basis for -a university -based knowledgei, t-e , development project (Un.iversity of Oregon Medical Careers Exploration Project)

o Lends itself to replication in industrial and business environments

o

4

May be adapted to serve specific client groups

o Project structure _easily accommodates joint sponsorship across prime sponsoF

jurisdictions (Summer Youth Employment Program (SYEP) youth from two prime

nsors areas participate in the project on a cost-share basis.)

o High school counselors are supportive of the project; recruitment and academic

credit provision are facilitated by,sudft endorsement

I

el 2

20
1/4

Prime Sponsor Description: City of Portland, Oregon

Portland is a city of 385,000,, the center of a metropolitan area containing 1.1 'rnillidn

people. CETA is administered by the City Human Resources Bureau; Youth CETA

services .are administered through a network of youth career training centers. These

City operated training centers provide comprehensive services for tut-of-school youth,

and administer programs for in-school youth. Services to VS youth are provided via a

J

non-financial agreement with Portland; Public Schools. Formula youth CETA dollars

allocated in FY'79 amounted to 3,245,000.

Logistics of Project operation

The Project is operated via contract between the City of 'Portland Prime Sponsor,

Portland Community College, and Portland Public Schools. Portland Community

College conducts the project;. Portland Public School assumes advisory and youth

referral responsibilities.

The, project, operative since 1974, has undergone five modification cycles, each cycle

providing a slightly different focuss and level of refinement. However, the core design

involves the rotation of youth through career awareness and exploratory classes in 10-

. career cluster areas. Community College instructors design the curriculum around the

career information and participatory (hands-on) focus of the project, and youth

(grouped in teams) rotate from one cluster area to another on a weekly basks under the

guidance of a team leader. Each youth spends 3 hours per day in career exploratory

classes and 3 hours'per day in supervised work activities.

The adult eam leader serves multiple functions. The team leader monitors the team's

flow through the stern, supervises team work activities, acts as an instructional aid

to the career cluster instructor, maintains participant time and attendance reports,

and provides counseling to youth-participants.

21

,

'OP
t

Initi Ily, in 1 74, the project served 50 youth. As time progressed, the Prime Sponsor

and

and

nu

chool District recognized the value of the summer' career exploration concept

took measures to refine the project and extend the experience to increased

ers of SYEP youth. The following table (2-1) reflects the growth factor (increase

in participants served)

'reflect) the-cost of the

as well as the charac' teristits of the participants. Figure 2-2

project's operation.

TABLE 2-1 CAREER SURVEY PROJECT PARTICIPANTS: NUMBERS SERVED/

CHARACTERISTICS

2-la Participants Characteristics (based upoli 1978 project operation):

AGE % OF TOTAL RACE % OF TOTAL

14 32% Black 42%

15 .43% White 44%

16 21% Asian 12%

17 2% Native Americari 2%

18 1%

19 1% SEX

Male -58%

Female, 42%

HANDICAPPED 13%

I

Note: Although the project served youth 14-19 years of age, the typical participant

was 14-15 years old, economically disadvantaged (70% LLSIL), a freshman or sophomore, and deficient in basic skill areas.

2-lb Number of CSP Youth Participants

1974: 50

1975: 108

1976: 24011-

22

(r)
OP

1977: 240*

1978: 240*

1979: 240*

r

e.

TABLE 2-2 CAREER SURVEY PROJECT COST (BASED UPON 1978 PROJECT

OPERATION: ONE SITE)

i

o Funding Source: Summer Youth Enyloyment Program

o Total Project Cost: $19,726 (Exclusive of participant wages)

o City of Portland Prime Sponsor Co'it: $16,171 (83.3% of Total)

IP

o Multnomah-Washington CETA Consortium Cost: $2,955 (16.7% of Total)

o Cost includes: to half-time instructors, 1 full-time coordinator, materials and

supplies, mileage

o . Number of participants: 120

.0 Project Cost per Participant: $164

o Participant Salaries; $509 per youth (192 hobrs x $2.65)

o Total Youth Participartt Wages: $61,080

4

I --, *The 240 figure reflects project operation at tw,o community colleges: 120 youth per

site.

. . Note:,J The City of Portland Prime Sponsor contracts with Portland Community

, : t

. . College, the project operator: School District responsibilities, identification/-,.

. referral of youth and academic wedit provision, are included in a non-financial

agreement, negotiated ,between the City of Portland ,and Portland Publid

Schools. A financial agreement negotiated between the City of Portland and

Multnomah-Washington CETA Consortium, the neighboring prime sponsor

providessf or participation scf youth from the 2 prithe sponsorships on a cost-

_share basis.

. n

3

23

3

r

/
The Program Development Process --AR J

An analys of the Career Survey Project's evolution can be pictorially illustrated in a

general modeP(Figure 1-1),, as well as in a more detailed, phase-specific mode (Figure

1-2). A Model of the Local Knowledge Development Process merely provides a more

in-depth view of the "Program Development Proce.ss."

The following analysis_ attempts to describe how we

referencing key stages in the Career Survey Project's

the systematic, orderly flow of program 'development

and use the learnings gleaned. -- Current Youth Program Assessment: Policy Analksis and Needs Assessment

The "current youth programa.s.senent" process emanated from an informal series of

discussions between prime sponsor and scFia district representatives. The principal

learn about youth programs by

developmertt* and by capturing -, AI processes to better understand

issues that emerged includedthe quality of work experien-Ce- placements made within

the summer youth employment program, and the extremely low enrollnient of yotith'in

vocational/technical courses offered during the academic year within local high

schools.

An analysis of the two issues began. Prime sponsor and school districtrepresentatives

agreed to approach the examination of each issue by isolating the questions for which.,

ti they sought answers:

\ , 1. Why had previous -efflrts to increase the number of summer work sites

a providing quality supervision and training been less fruitful than anticipated?

24

a

r'
I

1

o

L 2. What dynamics contributed to the Placement of the more mature 16 and 17

year-old SYEP populatiOn in the most substantive jab sites?

3. Conversely, what dynamics contribmted to the recurrent appearance of a 14

and 15 year-old "left-over pool ", the majority of whom Were placed in Jess

attractive, lackluster work sites, which offered little substantive 'training?

4. How could the provision ofavell-structured career experiences-be intreased

. for the 14-15 year-old group?,,

CC. 5. Why, after local high schools had recently increased the quantity and

quality of vocational course. offerings, were enrollment levels in such

courses frightfully low?

6. Why w re high school students, already severely deficient in basic- skills,

opting for college preparation courso, failing, and dropping out?

8 7. Why was the enrollm , ent iri comniunity colleges offering high school

. .. ; completion programs with vocational training options increasing so rapidly,

r, while high school dropout rates were higher than ever? t b

o The Prime Sponsor/Schoot District team began to gather data to assist them in

-,, . concurrently analyzing poky- and

. assessing needs. Simple, quantitatively-oriented"

surveys were conduCted to determine the relationshi en high-schTnal enrollment

levels and drop-out rates of local youth for GED completion in technical training

courses. Interviews with sch061 district career education staff and high school

students,- a- qualitative approach, was used to gain a better understanding of yita

enrollment in high school vocational/technical courses was so low.

25

.t.4"

a

a

Q

4,f
.

/'

. The data and information gathering process enabl4d the Prime Sponsor/School District

team to conclude that:

.11111 A substantial number of high school studentS who had "dropped out" subsequently enrolled in GED completion /vocational training courses

The majority of high school youth had, at most, a limited understanding of the

career education/vocational training options available to them

o Enrollment in school district' sponsored career education courses did not carry

1,4

sufficient status to entice students who may have benefited from such training

o Youth needed to be exposured' to the realities of the job market

o More effective methods of providing labor market and career information to

. SYEP participants needed to be developed

o Well-structured career experiences for 14 and 15 year-old SYEP participants

needed to be generated

It began to be obvious that SYEP resources should be-used to address thesIssues. The

problem identification process enabled the Prime Sponsor/Staff District Team to sets

goals. Then they formulated a statement of purpose:',

"To encourage the student -population to make earlier career ,plans and take

°4Tha advantage of .vocational training opportunities that-may be ommensurate with

their career interest areas."

J. The statement of purpose evolved intoa goal statement: "TA) allow §tudelos to

make better informed career decisions through hands-on exposure to

. ..,

. al- and technical careers and career information dissemination by , summer exploratory career workshops."

The knowledge development focal area had been identified.

26 3 C

vocatior*

attendl'hg

ti
Selection or Creaton of a Model Phase

The PS/SD-team conducted a local search for existing modelsough no such nibdel

existed within. the Portland Public School 'at that time which could be

replicated salient features of distributive education and cooperative alternative

"Acation approaches were identified. A survey of the local comunity college system

indiated that career exploration activities of the nature sought were not in existence, z nor were community colleges serving econornicalry disadvantaged 14 and 15 year -olds.

A poll taken of neighboring prime sponsors confirmed the opinion that similarly

focused projects had not been devised. Although the PS/SD team recognized that

prime sponsors having similar target populations and geographical characteristics most

likely had identified similar programmatic needs, the team did not consider tapping

out-of-state prime sponsors.

In retrospect, a significant amount of planning/design time could have been saved had

similar program models been "imported" for replication or adaptation.

, The PS/SD team elected to create a model to meet the specific needs of their locale.

--Project Goals Analysis

The PS/SD team initiated the Model Design phase by integrating general systems goals

identified by the Prime Sponsor for program development with the needs statements

generated by the school district staff. A critical juncture in the program development

process occurred when a "goal statement" consensus was reached. Specific objectives

and prOductivity indicator statements were delineated which, the team felt, would be

indicative of goal attainment.

Figure 23-Career Survey Project Objectives and Prtductivity Indicators, and Figure

2 -4- Statement of Activities/Timeline for Each Objective, reflect theoutcome of the

27
.r

.

N

FIGURE 2-3:._

CAREER SURVEY PROJECT OBJECTIVES AND PRODUCTIVITY INDICATORS

OBJECTIVES

P 1. To increase 120 participants' under-,

standing of vocational/technical

careers by providing exploratory

education classed in 10 vocational

areas.

.,

L

6

. To increase 120 participants' utili- zation of vocational/technical resources within the high schools by

fifteen (15) percent.

a,

../

3. To increase 120 participants' understanding of their obligations as employees by requiring attendance at sixty (60) percent of the sessions

i over the length of the project.

4. To increase the employment opportunities for 120 economically disadvantaged youth by employing

them during the summer in Super- vised work and instruction activities.

, 5. To maintain effective .leadership

and administrative management for

the Career Survey. Project during

Summer, 1978.

A

28

1.

..:

PRODUCTIVITY *INDICATORS

- Students knowledge of careers as

measured by pre-test and post-test.

- Degree of difference between pretest and post-test.

- Increase 120 participants' exposure

to job opportunities in ten (10) voca-..

tional clusters.

- Increase of 120 participants' knowledge of- skills, aptitudes, and interests required for success in ten (10)

vocational clusters.

. . - Number . of vocational cluster

areas\attended by each participant.

- Average number of hours spent per

participant in each vocational cluster.

- Range of hours attended" ,by 120

participants. - Number of youth enrolled at the

end of the project.

- Number and types of "hands-on"

experience provided in each cluster.

2. - Number of participants enrolled in

vocational/technical courses during

the academic years prior to 1978-

1979. . - Number of students enrolling in

vocational/technical classes after

participation in the Career Survey

Project.

3. - Attendance records.

- Range of hours attended per participant.

4._ - Total dollars paid to participants.

- Averge allowance amounts- earned

per participant.

- Hours of work performed.

- Total number of youth employed.

5. - Number and dates of activities

listed under activities 5.1, 5.2, 5.3,

etc.

3 r' V

),

...".......
4 (

FIGURE 2-4: STATEMENT cri, ACTIVITIES/TIMELINE FOR EACH OBJECTIVE

Objective 01:pg--SaIe objective heft)

To increase 120 participants' understanding of

classes in 10 vocational areas.

No. Activity

1.1 Develop curriculum

1.2 Hire 10 instructors and

secure facilities for

classes to meet.

1.3 Develop recruitment criteria for participants

1.4 Recruit participants fro!rn

eligible SPED\ clients

4 1.5 Assign instructors

1.6 Assign participants

e) n

vv

1.7 Recruit team leader's

1.8 Assign team leaders to

teams of participants

1

Develop orientation f r

team leaders

Completion

Date

5-31-78

5 -15 -78

5-01-78

6-02-78

I

5-15-78

6- i 5=78

6-02-78

6-15-78

vocational /technological careers by

Measure of completion

of activity

Written course outline

and cepirse description`

of each instructiolal

component submitted by

each of the 10- instructors

to the City. Courses provided as planned.

Contract with PCC

Written criteria

Participants recruited ,

and enrolled

Schedule for instructors

complete

Participants assigned to

teams

Appicants interviewed and

9 team leaders hired

4t

Team'leaders assigned to

teams of participants

Orientation agenda developed

providing explo

Other Amancies

Involved \

PCC, HRB

HRB, PCC

HRB

PPS, HRB

PCC

HRB, PCC

HRB, PCC

PCC, HRB

PCC, IJRB

.4%cation

4n

.

4
goal setting effort. The figures provide a sampling of the goal analysis results, which

were developed for' the 4th cycle of the project (1978). The program operator

(Portland Community College), as its operational expertise increased, assumed an

active role in suggesting structural modifications.

The, objectives, productivity indicators, and activity/timeline statements reflect a'

much more specific, extensive, and quantitative/design than those developed the first

year. Most of the productivity indicator over-hauling occurred during the second year.

Formally defined activities and timelines were developed the third year.

The refinement of these items was an outgrowth of program experience as the project

evolved .through the Demonstration, Modification, Process Evaluation, In-Program

Analysis, and Evaluation of Participant Benefits at Program Termination Outcome

stages. Goals, objectives, and productivity indicator's were continually reassessed, re.

focused, and re-written for -the succeeding program cycle.

Learnings from each modification cycle contributed to an increasingly specific 'and

more thorough knowledge of which program approaches work best.

The goal setting and refinement process was crtitical; it allowed planning, implemen-

-tation, and project assessment to occur, in an orderly, systematic way. It insured that

the concept of the project was clarified in atKrance, and that activities were

operationally defined.

Model Demonstration

The model was implemented via contract with Portland Comminity College (Summer,

1974). The majority of administrative energy focused on insuring that the program
was initiated and completed (more commonly referred to in youth programming circles

as "get, the program up and rolling"). The assessment was based largely upon a

qualitative format, but included some quantitative data. The principal instruments

included instructor questionnaires (narrative responses); program coordinators' project-

,. end reports (identification of program strengthsAveaknesses/recommendations for

change) and team leader project critique ,sessions. The rationale for the use of such

qualitatively-oriented instruments is based on the "process evaluation" thinking

discussed in Chapter.1 and, in more detail, in Chapter 3.

In the early stages of program implementation, it is appropriate to concentrate on

assessing the general workings of the program (the macro-apprOach) rather than on the

measurement of outcomes. The focus of the first cycle's assessment was, therefore,

to develop a general understanding of if and how the program achieved its goal. The

first cyCle's assessment was positive.

In spite' of innumerable "rough edges," it was agreed that the concept of summer

career awareness/exploratory sessions far jhe 14 -15 year-old group was sound.

Process Evaluation of the Model: Second Cycle ,

. As. the PS/SD team began seriously to consider the long-term utility of the project and

the transfeiability of the rotational career awareness concept to other projects, the

heed to fine-tune the total program prOcess was recognized.

9 It was agreed that an adequate lead-time for project planning modification and

assessment design had to be provided and that suggestions for program modification

should be solicited from a broader range of education and employment and training

personnel. r

31

4"4
0

The rante of assessment instruments and indicators included: , Quantitative

o Number of Vocationaltlusters

attended by each participant

o Average number'of hours

spent by each participant

in each vocational cluster

o Range of hours attended

by the 120 participants

o Number of yoUth enrolled

at the end of the project

o Analysis of participant

characteristics.

o Pre/Post Testing

o

o Participant data Cross-tabulation (to check for sex and race bias)

Qualitative

o Team Leader Summary Reports

o Instructor Reports

o Debriefing Sessions

o Project Coordinator Reports

o Counseldr Recommendations

o Participant Interviews

(

o Work Experience Coordinator Interviews

Outcome Assessment Efforts (Evaluation of Participant Benefits at Program

Termination/Impact Evaluation of the Effect on the Youth Community)

I

To date, outcome evaluations of the project have been limited to very simple, nonexperimental indications of participant change. at programeermination and follow-up.

Participant pre-post tests measured career information/awareness gains. Participants

were polled to determine if project participation encouraged enrollment in high school

vocational/technical courses. A six-month follow-up of participants was conducted to

determine how many youth had actually enrolled in vocationally oriented courses upon

-,. returning to school in the Fall. c

32
4

Although it is normally very difficult to subistankiate the impact of a specific project

on the youth community, institutional change gradually occurred; the community

college setting is now recognized by the prime sponsor, chool district, and community

college network as a valuable and vlable resource o provide services for 1/S

disadvantaged youth. -Moreover, the rotational career' exploratory concept was

0

incorporated the design of a year-round YEDPA knowledge development project:

, 1

The University of Oregon Medical Careers Exploration Project. The University

Atmplex is now engaged in providing 'careeripploratory services to 1/S and 0/S youth.'

33'

A
EVALUATIVE RESEARCH AS A KNOWLEDGE DEVELOPMENT TOOL
CHAPTER 3

EVALUATIVE RESEARCH AS A

KNOWLEDGE DEVELOPMENT TOOL

Program evaluation research is concerned with systematically finding out how well

Social action programs such aslYEDPA'work. The basic logic of program eyaluation

research evolved in part to address specific questions concerning social program

development.

It is useful to make a distinction between "evaluation" and "evaluation research."

"Evaluation" is used in general sense to refer to the social process of making

judgments of worth. It is a process 'which is basic to most forms of social behavior,

whether that of a single individual or of a complex organization. Usually an evaluation

implies a logical or rational basis for making a judgment, but it does not necessarily

involve the use of a systematic procedure far bringing together and presenting

_objective evidence to support the jUdgment. Thus, the term "evaluation," in its

A

common sense, refers simply to the getieral process of assessment or appraisal.

s i /

.

. ,

. / "Evaluation research," on the ot$er hand, refers to the use of social science research

methods and techniques for th-e" purpose of making an evaluation. Thus, "evaluation"

becomes an adjective specifying a type Of research, and the main emphasis' is placed

on the noun "research." So,. evaluate (or "evaluative") research refers to procedures

. , -

,

.. ,. _. . for collecting, anNlyzing, and utilizing infbrinatIon which increases the possibility of

proving rather than asserting the worth of a youth program. A great deal of variation

in the techniques and subjects of focus is possible within program evaluation research.

For exam .although much evaluative research* foclises on outcomes or results,

. evaluations can also be erected toward program goals and objectives, plahs or designs

c

35

. .4G
for carrying out the goals, or program activities and processes. As will be shown, the

wide range of evaluation methodology allows for adaptability to type1s and stages of

youth program development, resources available, and other unique local requirements.

-Despite this methodological range, a basic frameWork, or model, of the evaluation7----

process can be described. In the following two sections, this logical process is

introduced, then integrated and illustrated in the broader context of the knowledge

be development model introduced in Chapter 1.

t-- Program Evaluation Research: A Developmental Sketch

, \

The earliest program evaluation approach emerged from the education field. this

approach Was typically concerned with narrowly determining the degree . to which .

. program objectives were achieveq. The rationale was that programs should be

evaluated by comparing the attainment of objectives) usually involving participant

..

, performance, with clearly specified behavioral outcomes established for the program.

From such a process, it could be determined if the program was achieving its goals. If

it was not, prqggram modifications could be made*. This early apprtach has been

. , extremely influential in contemporary social program evaluations.

6

T is approach also has several implications for what program ditectors and staff must

know and do. The most important of these is the need for clarity a d consensus on t agency and prograM goals, on methods of assessing participant outcomes, and on what

k

constitutes' acceptable or desirable, participant outcomes.

.... _ Much of current good practice in program evaluation simply expands on this methodology: Perhaps the most important development is the increased concern for process

1

*This is basically R.,Tyler's "Goal AttainmentModel," first described in 1949.

36

4'''6

.

4

1
4

""` 1

assessment in the overall evaluation of particular program. Process evaluative

research assesses the qulity of a pfogram during its implementation., and has grown' to

be seen as particularly important during the formative early, and intermediAte stages

of a program. One of the principal. roles of process evaluative research during this

formative period is to help discover problems, limitations, and successes in the

workings of the program and its implementation.

Outcome /Impact evaltiation is more terminal in nature and is aimed at determining the

value of the completely developed program - usually in terms of program-related

changes in program participants. A complete developmental cycle of program

evaluation ideally synthesizes both types of evaluation.

Reflecting

evaluators

evaluation

the evolution of approach touched upon above, many contemporary program

now subscribe to a 4level evaluative process which associates a set of

types with a parallel set of needs for planning decisions*:'

a

Four planning categories may be identifieciftie

1. Planning decisions - to set goals and objective,

2. Programming decisions - to"design program operations

3.

4.

Implementing, decisions - to monitor, evaluate, and improve program activities

Recycling decisions - to judge, learn from, and react to outcomes of the program

(continue, modify, or terminate)

For each clasi of decisions, a parallel type of evaluativekresearch activity can be

identified:

*The following discussion draws from Daniel Stufflebeam (1971).

37-

4 J

)
e?

1. Cctext Evaluation provides the basis for determining goals and-objectives.

a. Isolate the problems, needs, and unmet opportunities the program can

address.

b. Search outside the immediate. system ,to locate useful values, needs and

c.

approaches. o Explore research findings,

o Visit other programs.

o Assess community priorities

Examine existing program(s): What-are the. policies, the organizations, and

the goals? Are goals being achieved as intended?

d. Should goals be changed or expanded?

2. . Input Evaluation provides the basis for selecting prqgram activities to meet goals

and-objectives. How are the objectives best achieved?,

a. Identify alternative strategies.

b. Determine the limitatidns and advantages of each.

, c. Identify the likely effects and side effects of each approach:

d. Determine what is required to administer and monitor each approach.

3. Process Evaluation provides "feedpack'about program activities. It. has three

major functions:

a. Detect or predict problems.

6. Provide informational needs for management /staff decisions.

c. Record the procedures.

Information is obtained and reported as often as the program, staff requires it.

4. Product Evaluation provides a measure and interpretation of achievemicas_as

often as necessary. This form o evaluative research is similar to that identified

earlier as outcome/impact evaluative research. However product. evaluation

may also assess intermediate attainments within the program.

V 38
0

.In summary, this approach is designed to assist the, decision-maker at all steps of

. ,

program management. Thus, evaluative research is the process of determining -the

kinds of decins to be made and lessons to be learned, then selecting),Oolleeting, ands

interpreting the needed, information in Making these decisions and learning these

lessons.

a

Five evaluative research phases, shown in Fig. 3-1, provide a simple framework for the

decisiop.making, lesson learning process.

4

,/

Applying An Evaluative Research .Framework to the Knowledge Development Process

The principal reason for introducing a "state of the art" framework of contempo'ary

evaluative research is that the logic .of this framework can easily be adapted' to fit

neatly into, the original model -of the.19,cal Kg process introduced in Chapter 1.

This adaptation is rpade eledrer- an d more workable by taking an intermediate. step, i ; , '

< . whereby the evaluative resekah framewlittk (Fig. 3-) is restated as a flOW of ;decisions

- ,. g and activities. This restatement is Yawn- as,Fiik 3-2, which contains all of the original

> ,i-

. framework elements except the "Needs Acssessment, bloc The reason for this

e

. omission is that, within the KD model (Figure 1-2,), We.. ve ;already concerned

,

. , ,,.. ourselves with a need/policy assessment earlier in the program development cycle.

..

-. A key feature of the evaluative r earch model (Fig; 3-2), which we are now inserting

,

, "-I a,-, - into the overall KD process, is.t at. we may work our way through the logical flow in

. . several different ways. This'flexibility will meet our. earlier ,criterion of allowing the ,

/ suitate adaptation of evaluation approach (methodolog pe of data, study design,

data analysis techniques, etc.) for each stage csf youth Program development.

39
A

NEEDS

ASSESSMENT

PROGRAM PLANNING

SPECIFYING:

GOALS-OBJECTIVESACTIVITIES-OUTCOME

INDICATORS .

MANAGEMENT

INFORMATION

SYSTEM

PROCESS

EVALUATION

OUTCOME dc FOLLOW-UP

EVALUATION

Needs Assessment involves stating the

objectives. This information is used to

identify program needs.

program' planning, evaluative research can

provide the project director with tools to4help

make planning decisions. It can also build

procedures into the program, that will be

needed assess whether or not the

program is operating as planned and how well

it is achieving its objectives.

A Management Information System is essential to the monitoring process and to determine

the extent to which the elements of the

program.are being implemented as planned.

Process evaluative research provides

information about the activities and

interactions of the program's components in

meeting program objectives. This information

is used to make modifications where

necessary.

.,"" L

Outcome and Impact Evaluative Research

provide information about the success of the

entire program. This information can support a decision to maintain, modify, expand, or

discontinue the program.

f,

FIGURE 3-1: AN EVALUATIVE RESEARCH AMEWORK

. (Adapted from a model developed by the UCLA Center of Evaluation.

See Morris and Fitz-Gibbon (1978).)

40
t

(

nnICONF.IPRAtt

r

nni1111.

r

qUANNIAMF 1

0111T.ONt Olt IN

IMPACT

E7MAIn11011 II

SIOOT IS

rrIncinnn MANNINO

COAIS Ann

nn. 1: ivrs

1._ commit

t

nnitniqi On

IPNInt 1

EtALVAMF

WONT

(1ttinht

Q11/1111%

..

A

rit ottss

i

QUAL.

W1411111131 QUAN111 A t Tr F.. QUAlltAIIVF

niitnnf All rIttlYASS , ei ranctss

111111C I IYALIIASION Ivnutnitoll

t VALIfit lion SWOT II CIIII1Y

STU!) Y S )

1

Q

en NETWORK

tIntint 3 7! An IVA! IlAtivE ntstAwcti nott

z

-t%

rnOstss

EVALUAIIVF

lir Mutt

A

..
Let's see hoour evaluative research model works when we consider how best to

assess and learn from a LEA-private sector approach. Assume that, within the broader

KD process, we have already:

(a) assessed currently pffered youth programs,

additio I unmet youth needs,

(c) tentatively ident eD:focus area, and

(d) determined the proper "learning level," based on how much we know about

. the KD focus area. We have a new LEA-private sector program modes to

fry but, and are at learning level 2.

e

74,

Nowle are ready to demonstrate the model and, at the same time, analyze what

happened during our demonstration. At this point ih the overall KD model; we enter

the logic of the evaluative research model (Fig. 3-2). Notice that we could have

entered the evaluative research model from any of a number of other learning levels;

but, had we done so, it is most likely that we would work our way through the logical

flow in a different, fashion than exhibited here, where we:

Needs Assessmeht involves stating the objectives. This information is used to identify

program needs.

1. Plan the model demonstration, being certain to articulate our ,-goals and

objectives aGlearly. Clear delineation of program goals is crucial to good

program evaluation..

:14'7 r

2. Decide on the most appropriate evaluative research approach suggested by our

goals and objectives. At this rather exploratory level, we might well decide that

42
L our LEA-private sector link aPproach.is.best examined via &process evaluation,

since at this early stage, we are most likely to be interested in dapturing a sense

of the major characteristics of the new linkage's implementation.

.3. Next we will decide what blend of methods to,use within the process evaluative

research approach. At this early stage it is likely that many of the questions we

have about our new LEA-private sector linkage idea may best be answered by

using qualitative methods. Thus, techniques such as intensive interviewing may

be used to explore key questions and then verify their relevancy. For example,

whit are th&rrarious "agendas" of LEA and private sector personnel? 'Did

program participants enjoy the experience? What unanticipated problems or

benefits were associated with the demonstration? And so on. Of course, at the

very ,least, our monitoring system will also collect certain bottom line' quantitative data on youth participant characteristics and services delivered within the

program segment affected by our new model. Tiiis;and perhaps other quantitative :information, will allow us to report later' on the delivery of the actual

"service units" that occurred during the demonstration: for example, the number

of conferences held between LEA and private sector personnel, thenumber of

_-`--ti,

private sector Work exposure situations available to youth participants, as well

A

as participation rates, costs, and.so on. 7,

4. We would then prepare a final evaluative report (and perhaps a brief intern

report or two) combining quantitative and qualitative findings.- This report

serves to dcicument at least the highlights of what we have actually done in

implementing our new linkage idea, and what were some.-of the effects and

reactions associated' with our new approach. .At this point, we leave the logic df

the evaluative research model and retion to that of the broader KD model.

43
5. The vfinal evaluative regort_he mes part of the decision and policy making

11

. - process of the local jurisdiction. I addition, the result of our demonstration

becomes available to.others interested in KD. If the results of our evaluation

mare not utilized, there is little reason to perform the evaluation in the first

place.

.6. Depending on policy makers' decisions, we may next decide to modify the model

and .perorm further, more detailed, and still predominantly process-oriented

evaluations (learning levels 3 and 4 on the K Flowchart). Or, if the initial

demonstration was judged completely unsatisfactory to meet our needs, we may

alter our idea or discard it alibgether and look for another program approach to

demonstrate.

As mentioned, different pathways through the evaluative model will be appropriate at

different learning levels of the KD framework. Generally,, in learning levels 6-8 we

will tip the methods' balance in favor of quantitatively oriented. outcome evaluations,

which may involve experimental designs using control or comparison groups. However,

in such a case, some qualitative material will continue be desirable to' serve as in

interpretive background for understanding quantitative results.

It is important to note that as we proceed along the learning levels of- the K'D Model,

we accumulate information about the program approach we wish to locally implement

or continue to modify. Indeed, it is The continuous accumulation of information which

enables Lis to progres's along the spectrum of learning levels. Of course, more readily

available and generalizable findings from youth prpgrams elsewhere, can greatly assist

us in this learning journey..

The remaining sections of this chapter highlight some ol the more sppcjfic principles

44 vu
and procedures of the evaluative model as it applies to the youth program knowledge

development process.

Comments on the Techniques of Evaluative Research

. c, ,

0

We have examined the general strategy of knowledge development and inspected the

role played by the logic of evaluative research in KD.. It now seems appropriate to

offer further insight into some of the specific techniques of evaluative research.

Space is limited, and for detailed, procedures the reader is referred to Mangum et al.

(1976), Sum et al. (1978), and Morris and Fitz-Gibbon (1979). The following section

emphasizes goal analysis, discusses the balance of qualitative and- quantitative

methods in youth program evaluation, and offers some useful rules of. thumb in

research design, data collection/analysis, and documentation foiloyouth program

evaluation.

Goal Analysis

Clarifying goals enables us to establish a common understanding about what it is we

want a program to achieve and then to deterMine how and if the program is achieving

. it. In program planning, ope tions, and assessment, goal analysis plays an initial role.

In large part this is because o, the systematic thought process analysis necessarily ,.. Y. . , subjects us to as individuals and, even more importantly, as a group, of individuals with

different perceptions about the many aspects of youth programming.

From an assessment standpoint, we can be assured that if we reach no basicconsensus

about program goals prior to operating a program, no consensus is likely to exist by the*.

end of the program 7 particularly if the program had proglems. Thus,,from a learning,

knowledge development standpoint, it is,crucial that operations, planning, and manage-

. ment be involved in ascertaining what it is that we are going to do in the program and

45

1
(

what it is that we wish to learn from the program. Although additional program

actions and learnings may occur, we will have a bottom line agreement against which

to judge and orient ourselves, both during and at the completion of the program cycle.

Techniques of Goa,i1rAnalysis

Goal analysis is the systematic process_ of moving from the general to the increasingly

specific.* At each step, there occurs a further honing and refinement of the subelements of the original goal. Deal with goal analysis as you would the writing of a

report: first drafts are for getting down, not for getting good. Goal analysis

eventgally, lead to polished statements which, in addition to assessment purposes, are

also suitable for grant applications and public consumption. I

Five basic ,steps comprise goal analysis:

1. Establish the goal. Use whatever words are comfortable, though they may

be fuzzy and broad. The idea is to "get it down," to get started in the

overall process. Additionally, It will be easier to achieve an initial

consensus on the importance of more vaguely stated intentions.

Check to make sure the goal describes an outcome rather than a process,

so you don't get stuck with the problem of means, and ends at the very

start. For example, make the goal "understand..." rather than "develop an

untierstanding of..."

*The approach to goal analysis discussed here is that of Robert F. Mager (1972). ,

46

5"4

r",

4
2. Jot_down actions that define the goal. Use words and phrases, making no

attempt to. keep things tidy. Don't try 'to write objectives - that's for later.

Two strategi s may help you to describe the meaning of the goal:

a) use the word "by" after the goal. For example:

GOAL: Increase youth participant services

BY

1. Training staff in youth counseling

2. Contacting schools and juverag-agencies to identify. potent aLpartItipants

3. Expanding facilities

4. Planning new intake procedures

As you try to think of actions to reach the goal,

b) Consider whichever of the folloIving seems most helpful:

1. Answer the question, "What would I take as evidence for

the achievement of the goal?"

2. Imagine someone else with the responsibility of defining

the goal. How would that person recognize the goal's

attainment?

3. Go back over your list of actions and tidy it up. Eliminate duplicatigns and

items that, upon reconsideration, are not what you want to say. Check for

abstractions (fuzzies), which can be either discarded or analyzed fUrther.

You may find making reference to a list of more precise verbs helpful in

formulating concrete statements. (See Table 3-1).

The following statements illustrate the distinction between "fuzzies" and

47 .ter

t_r

3
C

"concretes".

4

FUZZIES

iv ---

' t

CONCRETES

1. Be a good counselor 1. Name ten counseling competencies

2. Defend confidentiality 2, Train staff

3. Appreciate other people 3. Smile when you say hello

4. Understand particip is 4. Identify fuzzies

Each of the "concretes" is an action. Naming, reciting, selecting, smiling,

and identifying are all actions. The "fugies," however, are fuzzy because

they are abstractions rather than actions. "Being a good counselor" does

not tell what the individual is doing when s/he,,is being a good counselor.

,. . The same is true for "defending confidentiality"; how is the individual

defining confidentiality?

4. Write a complete statement for each action. Note the manner anchor

I

extent of accomplishment that you would insist be achieved before .,

agreeing that the goal had been attained.

- , This complete statement will become an ob'ective. Fully developed

, - objectives contain the following features:

. . a. n Action - A good objective always states what is expected to be done.

b. Conditions - A good objective always describes the important conditions (if any) under which the_action is to occur.

) 5

48
TABLE 34: A categorized "snooping-list" of verbs useful for making objectives m ore precise

SIMPLE TASKS

attend

:noose

collect

co m Olete

cooy

count

define

cescbe

:es gnat!

detect

differentiate

dtsc1minate

distinguisn

distribute

duo licate

find

identify

m state

indicate

isolate

label

list

nark

p atch

name note

pcrcer

olace

point

Provide

recall

repeat

select

state

:ally

tall

.; no erli ne

STUDY SKILLS

arrange

atte m:t

cateaorce

char:

cite

circle

classify

:omolle

consiaer--

di agra m

document

find

follow

form ulate

gather

.ncuce

7-.e. cc ate

eo organize

cuote

record

-slate

"eturi

search

signify

sort

suggest

su000rt

underline

volunteer

ANALYSIS

tri a lyz e

appraise

corn oine

compare

conclude

contrast

crticize

ceauCe

cefine

evaluate e

explain

form ulate

generate

Induce

infer

paraphrase

Plan

Present save

shorten

structure

switch

SYNTHESIS

SKILLS

alter

Sch an ge

ze vel op

discover

expand

extend

generalize

;nocify oaraonrase ore cict

propose

cuestions

et ar-ange

recoil tine

reconst.ruct

regroup

rename

reorganize

reorcer

-ebnrase

-estate

-estructutv

retell

^° write

signify

sim plify

synthesize

syste mite

GENERAL

APoLIC ATIONS

Arts & Craft

asse m pie

plena

guild

:rusn

carve

color

construct

crush

Cut aaD cot

draw

orll

finish it

fix

old

form

frame

grna

la m m er

nandle

neat

illustrate

m ake m en m end

mix

m old

n ail

Point

paste

pate

Position

pour

ores

procedure

roll

rub

sane

saw

SCU lot sew

snake

sharpen

sk mon

sm Mtn

stamp

stick

stir

:race

dice

',rap

Drama

act

clasp

correct cross direct'

display

enter

exit

express

leave move

oanto m ine DaSS oer-fcr

praceec

respond

snow

start

. turn

Language

abbreviate

accert

alphabetize

argue

, articulate

49

capitalize

edit

hyonenate

indent

outline

print

pronounce

punctuate

read

recite

sneak

spell

state

summarize

syllabicate

translate

type

verbalize

write

add

bisect

calculate

check

tom oound

count

derive

civice

estimate e

extra bol ate,

extract

graph

grouo

rtte grate

interpolate measure Z m oer uiticlY

plot orove

-eaUCe

solve

sou are suotract

taou late

verify

Music

blow

bow

ciao

cam Pose

conduct

'anger It

^ arm onize

mute

clay

clack

oractice

sing

strum Social

tao

whistle accept

, agree

Physical aid

allow

arch answer bat ,buy

bend 11 communicate

carry

catch

cnase climb

coach

coordinate

oritioue

Boat

giro

let

hop

jump

kick knock

lift

m arch

perform

Ritch

ssckozer

' to

ski

skip

so m ersault

stand

stretch

strike

swim

swing

thro w

toss

Science

calibrate

compound . connect

convert

decrease

onsonte

3153 graft

grow

ncrease

nsert

lengthen

tight

lim

manipulate

nurture

operate

Plant

prepare

reduce

remove

reolace,

report

reset

set

SPeC-`./

saigrten

P-ansfer

weigh

fit'

compliment

contribute

:000erate

disagree

discuss excuse forgive

greet

guise

nelo

-nform

interact

invite

join

laugh'

lend

m eat

offer

Participate

permit

praise

react

relate

serve

snare

smile

supply

talk

thank

volunteer

vote
c. Criterion - Whenever possible, a good objective describes the criterion necessary for the objective to be considered completed.

)

For example: Client services are considered to be increased when the

/ program demonstrates it has been able to: - Train two new counselors by June.

"N. 2. Plan hew intake procedures, which will fulfill federal guidelines and

are considered acceptable to a test group of participants.

3. Indrease the client caseload 50% bylune.

4. Expaild facilities, provided funding is obtainable, for 25% increased

program usage.

5.. Identify participard Populations not previously served, which match

the results of an independent research survey.

5. Finally, modify these 'statements until you can answer yes to the test

question!' "If something was achieved according to these statements, would

I be willing to say the goal has been accomplished?" When you can answer

yes, the goal analysis is complete.

Goal analysis is an'important part of evaluative research regardleis of the KD learning

level. As is to be expected, the specific learning level will strongly influence the

nature of the goals we produce. For. example, the goal. statements of an initial

50 r
1

program demonstration will be quite general, while the refined goal statements of a 11142 participant outcomes study might actually lake the 'form of a research hypothesis.

The main point here is that well - written goals and objectives should point the way4to

the types*Of procedures most appropriate #oC evaluating the program. This amounts to

saying that problem statements and solutions should be appropriately matched.

Quantitative aid Qualitative Methodology in Evaluative Research

In evaluative research, the process of ,"problernsolution" or "types of procedures" used

is called the "methodology." Regardless of the specific methodology chosen, several

basic elements shou 6 present:

1. Selecting the basic evaluative approach based on the KD learning level, the

type of questions asked, the nature of program goals, and who the users of

the research findings are likely to be.

2. Developing the evaluative research design.

3. Collecting materials and data about program events and participants as . called for in the research design.

4. 'Analyzing and interpreting the findings.

5. Presenting and interpreting. study results.

In deciding what sort of methodological approach to use, it is important to realize that

an almost organic interconnectedness exists between the nature of the knowledge

deyelopment question itself, the method used for obtaining information about the KD

question, and the data collected according to procedures outlined within the methodology. These three elements interrelate in a systematic way, such that a change in one

of the three necessitates a change in the other two.

ti

A poor choice of KD methodology can severely limit the kinds of questions we might

ask and the types of problems studied. Thus, -according to the unique loCal needs and

51
...

--\

l

circumstanaces of youth programs, we should select a research approach which is

based on the unique characteristics apd goals of !peal programs, not an automatic

evaluation approach using standardized criteria applied across

(the

board. The task,

Sok then, is to combine the validity of the question with the validity of the answer.

For example, if a question.has to do with determining which one of t'Wo youth work

configurations caused specific changes in the youth participants, a precise answer to

this outcome evaluation research question will most likely, call for a rather precise

methodology, possibly involving an experimental research design utilizing a Comparison

or controrgroup. Data collected within this method would typically be uant tative

data that is numerical results based on test results, questionnaire responses, or some

... other sort of quantitatively measurable response. On the other hand, if the question

, deals with the nature of participant characteristics and reactions in a demonstration

' program, out- choice Of method might involve intensive interviewing, participant

observation,. or some other such qualitative methodological approach.* In this case,

--.,

. the resulting data from our qualitative Methodological approach would-tend to be more

4 narrative and less numerical in natur*

Likewise, the. way in which data is analyzed may+ differ considerably from problem to

. , problem .and methodology t methodology. Using qua'nti*ve approaches, for

example, we. typically becom involved in measures of _statistical description and

statistical inference. However, in the aniaysis of qualitative data, we seek to ilk

establish themes and patterns based upon the unfolding results of interviews and other

material collection Kocedures.'

l

*For detailed discussion of qualitative methodology see: Patton (1980); Bogdan and

Taylor (1975); Lofland (1971); Cook and Reichardt (1979).
It is important to realize that a range of methodological approaches to.KD exist.

Currently, the dominant evaluative research approach, accepted almost without

question by many youth planners and eVitrd-tors, involves the use of a natural science

based, deductive, hypothesis'-oriented methodology, which involves quantitative meai-:

urement, experimental design,and presumes statistical analysis to be the epitome of

1

"good" procedure. Incontrast, the alternative (qualitative) end of the methodological

scale is derived from the Itadition of anthropological field study.

The qualitative approach is induttive and holistic; it aims at understanding program

phenomena ' through the use of qualitative data, holistic analysis, and detailed

description derived from close contact with program participants. Although the

qualitative approach is sometimes erroneously regarded as being "soft," "fuzzy," and

generally less desirable than a "harder" quantitative approach, a more reasonable

position ,is simply that different kinds of problenis KD learning levels) require

different types of methodological research approaches. No position on the methodological spectrum is intrinsically better than another. Rather, these are alternatives

z from which the we can choose the solution which best matches our particular researt

problem.

It is often said that the distinction between qualitative and quantitative approaches is

that of "knowing" something, as opposed to merely "knowing about" something. Thissuggests that ,qualitative approaches, by which one means those research strategies

AO. which involve participant observation, ,ih-depth interviewing, fieldwork, and so ori,

allow the researcher to obtain first hand knowledge about the youth employment and

training programs in question. Qualitative data collection strategies, then, are, aimed

at describink and understanding istic nature of the program component from the

participants' points of view

53 6
Using a quantitative approach to youth program teaming involves an attempt to

establish reliable generalizations about the 'frequencies of program events and about

associations among these events. By using a qualitative approach, we can acquire a

systematic overview of the tissue and fabric,Of program activities, events, or settings.

It can sometimes be argued that a strongly ckaratative approach may cause us to

draw our attention away from the major features.of the ongoing setting itself. We can

trap ourselves into an exclusive concern with very. small problems simply because

these problems are subject to quantification. On the other hand, a quantitptive

c approach is allsessary to provide more conclusive answers to highly specific questions i

about program operations. Because of the rigor and 'control necessary for many

quantitative evaluative research approaches, the detailed view of the .4suantitative"

researcher may be compared to that of a viewer with a Isma11 magnifying glass. The

qualitative researcher, however, has a macroscopic though less detailed, view of the

total context of the program under consideration. Thus, three important of

, . distinction emerge when the tv,lo techniques ace compared:

1. The scale or scope of program operations which may be identified and

studied;

2. The perspective of the analyst (the "insider's II view of the qualitative

analyst vs. the "outsider's" vieof the quantitative analyst);

3. The meth ds and data of the two approaches.

N.

How do we decide which approach to use? When our research focuses on program or

participant characteristics, we may well choose a predominatly qualitative approach.

On the other hand, if our focus is more concerned with the establishment of causes or

consequences, we May be predominantly interested in a more quantitative approach.

For example, in process evaluations of experimental or demonstration youth programs,

54
0.

a more qualitative approaCh is' often appropriate. Here the agumer4, is that not

enough is yet known about the programm ,iic approach, selection of target yOuth

_communities, and delivery methods to appropriately apply highly specific quantitative

approaches to evaluate the program. The more macroscopic approach of qualitative

procedures may be inuch more in order.

D

For outcome or impact evaluations of youth programs which may have been estab-

..

PV lished in a prime sponsorship for a longer 'period of time and about which a

considerable amount of information about program delivery methods, problems, etc., is

already known, quantitative approaches to answering very specific questions may be

most appropriate.

Usually, some balance of qualitative and qUantitative methods should be incorporated

throughout the course of an evaluation. Each method can provide a cross check'on the

other. Ideally, evaluative research should tap multiple perspectives through the use of

multiple research methods in order to 'capture the most comprehensive view of the

$ youth program. Thus, qualitative methods can provide the context within which the

quantitative findings can be interpreted and understood. It is q ite common to obtain

progr:am evaluation results-that don't fit with what vs expected. Wondering. about

rival explanations of outcomes an become very frustrating when there is a lack of

ti contextual understanding which surrounds the quantitative assessment. What could

have precipitated such puzzling quantitative results? Was the measaVnent instrument okay? Did the questions asked correctly tap the area being evaluated? Did thee,

youth participants percieve these questions in a different manner than was anticipated? Such questions can go on and on, and often do; qualitative data that can provide

the framework for comprehending the quantitative portion of the evaluative findings

can often provide answers about the larger context. In short, qualitative methods

often provide the basis for understanding the significance of statistical associations.

55

(j
Notes on Selecting an Evaluative Research Design --psEree,

Most simply stated, a "research design" is a plan which states how, by whom, when,'

and, from whom measurements will be gathered while conducting a knowledge

development pr jeCt. The initial and most obvious reason for using an evaluative

research design is to insure ajwell-Organized study, with all the right people taking

part in the evaluative research at the right time;*. A good design, however, provides us

with something more useful than just keeping the collection of data on schedule. Most

basically, it is a way of gathering comparative information about program elements or

participants or some other set of variables so that results frdm the program being

evaluated can be placed in a context that will enable us to judge the program's worth.

In selecting a ,speciffc research evaluation design, the KD analyst should ideally

consider a number of factors, such as the reliability and validity of outcomes, the

trade-offs in reliability and vailidity which must be rripde due to limited time and

resouies, the specific nature of the youth program or programs being studied, , -..

Perhaps dne of the Jnost important criteria in the selection of a" study design, however,

is the "face validity" of the research design and of the instruments used within the

research design:. Face validity concerns the extent*to whic an entire study or an

:individual measurement instrument looks as if it measure what it is,jritenEled to

measure. A research design and the instrument used within it have face validity if

youth program decision-makers and information-users can look at the design and the

t

items measured by the instruments, and understand wht is being measured.

For example, if ke,, youth program decision-makers and informatOn-users have

chance to 'review. the items on a questionnaire to be used in our knowle

development stud , well listen to an explanation about what the questionnaire items

were supposed to indicate, they may then be -asked: "If we administer these

56

a
4

t.

questionnaires with these-items measuring these, factors, will that tell is,swat we

want to know? Does this approach make sense? Would we believe the results if they

came out negative?': It is difficult to get decis*n-makers and information-users to

look at designg and instrumentation carefully in advance. But the more this sort of

review tak&place, the greater the insurance that the study results will be accepted as

being reasonable by the decision-makers and information -tiers and actually be

utilized. Thus, face validity-is extremely important in research design and subsequent

data analysis, data interpretation, and data utulization.

As with the face validity of research instruments and the credibility of the .research

design, so, too, consensus on the relevance of research definitions and the units of

analysis.4.re also factors that affect the credibility and usefulness ef Kb. evaluation

data. Furthermore, activeiy4involving information-users in making decisions about

these issues means more than a one-point-in-time approval of a KD research proposal.

Ideally, decision-makers and information-users shoUld be involved in all KD research

decisions as' they occur. Typically, there will be a slippage between the methods as

originally proposed and ths methods as actually implemented in the KD study.

decision-makers and information-users are involved only at the stage of approving

research proposals, they are- likely to be gurprised when they see a final 'report. Th, us

the making o/ decisions about research methods is a continuous process that ideally

involves checking out changes as they 4e made.

"\=

The research design of a KD study, will vary considerably depending onwhether -a

;.,

quantitative or qualitative approach to analyzing,the youth program is used. Quantitative design approaches involve the specific determination of research interests, goals,

data collection procedures, and hypotheses well in advance of actually conductng,the

study. Qualitative research studies /end to remain quite flexible from the beginning

and well ihto the research. Although qualitative researchers have a methodology to

follow, the specificS of their approach evolve-as they proceed.

57

SU L.1
tl

KD projects may be quantitatively tudied in very simple but effective ways, as well

as in more complex and powerful ways:

1. Descriptive evaluation research is concerned mostly with "what is!'

and "how much" and involves the use of descriptive statistics (e.g.,

frequency distributions., measures of central tendency, measures of

dispersion) to describe the framework or characteristics of the youth

program population and en ironment. It is necessary to establish a

conceptual frame of ref e nce, e.g., the particular new component

being studied and its rela ionship to other youth offerings. Workable definitions of measures to be used must also be agreed upon an

exercise which should be eased by the presence of well-written goals

and objectives.

At the developmental stage of a KD project, it will probably be appropriate to devote the most' energy to descriptive strategies before moving to the next stages of quantitative study. Thus, the

descriptive approach will be especially appropriate at the earlier

learning levels of the KD model.

2. The correlational or comparative level of evaluation is concerned

with relationships among measures developed at the descriptive level.

For example, we may determine that work groups comprised of mixed

income participants do,.indeed, seemtocompare or correlate with

higher levels of performance (or attit64nal change, .etc.) than do

work groups comprised only of disadvantaged yo,ofh.

Such critical examination of relationships arkong measures describing

a youth program may provide valuable insights -into what goes on in

the program. These insights ma then raise meaningful questions,

which can be_answered by subsequ6nt experimental studies.

3. The experimental approach to quantitative evaluation research gen-, erally requires an experimental design in which experimental and

control groups are compared, and to which program-participants have

been randomly assigned. The change in some outcome variable of

interest (such as reading improvement or completion rates) is contrasted between experimental and control groups.

Experimental designs are often inporrectly assumed to be the best or

even the only really acceptable type of evaluation methodology. It is

true, that questions :about pr gram effectiveness (e.g., at learning

levels 6-81n-the KD model) a often best 'answered. by experimental,

desigri. Descriptive and correlational approaches may certainly be

utilized in such analysis, though less conclusive outcomes will result.

- In contrast wh quantitative design methodology, qualitative research approaches

. ...' -Zs- . .

involve a gradual unfolding of the specifics as they procede. Challenging research 6

design decisions, which mu§t occur attthe outset of -the quantitative approach, occur

. .

cr

. 58
later in the qualitative approach. Though different in nature from quantitative design,

the discovery and creation of the design within the qualitative approach can be equally

challenging.

A major methogiological consequence of the qualitative study of people in a given

location is the process of discovery. Of necessity, it is a process of learning what is

happening. Since a major part of what is happening is provided by people in their own

terms*, we must find out about, those terms rather than impose a preconceived or

outsider's scheme. 'It is our, the observers', task to find out what is fundamental 'or

essential to the people or enviornment in the youth program under observation.

The same range of design types that exist in quantitative research (descriptive,

correlation-al, experimentalYcan occur in qualitative research approaches. The real

strength of qualitative methodology lies in its facilitation of discovery. Although

conjecture regakfing causes and consequences of. program events may emerge from

qualitatiye research, the principal focus of this approach will usually be upon

des,c,./bing the characteristics of progranioevents and participants.

Nonetheless, qualitative approaches may be incorporated into, for example, an

experimental study design. The forthcoming qualitative results would not be statistically testable as in the case of quantitative results, but the breadth; comprehensiveness, and sensitivity of the qualitatiye resu'ts might well offset this consideration.

Ideally, an experimental research approach would include both a quantitative and a

qualitative dimension. The two approaches can complement one another in a very.

desirable way.

In qualitative prOgram an' alysis, we will, of course, have some general questions in

59
mind when we begin our research. These typically fall into .one of two broad'

categories.

1. The first category includes questions related to specific substantive issues

in a specific program setting. For example, in studying a central youth

employment center, one might examine how youth participants view the

physical and/or social environment of 'that center.

2. The second category of interest involves asking questions about -toilics,

such as the nature of a certain kind of organization or the dynamics of

small groups. Thus, for example, a youth participant's experience interacting with fellow workers in a community home improvement work team

might be examined.-

It may be helpful to consider different categories which the qualitative researcher can

examine. Ranging from the small scale to the large scale, one qualitative methodolo!

gist has recommended the following sp trum:*

1. ACTS. Action.in a situation that is terripora,ry and brief, co. nsuming only a

few minutes or hours.

2. ACTIVITIES. Action in a setting of some duration: days, weeks., months,

constituting significant elements of pergons' involvements.

3. MEANINGS. The verbal products of participants that define and direct

action. I

4. PARTICIPATION. participant's total involvement with or adaptation to a

situation or setting under study.

5. RELATIONSHIPS., Interrelationships among several persons considered

sinveltaneously within the program.

*Lofland (1971).

60
A

6. SETTINGS. The entire setting of a program under study, conceived here as

the unit of analysis.'

Further, we may consider within each of these six scale units either a static depiction

of the category under study or phases (or sequences) through which the category being

studied passes over the course of time. This is somewhat like the' difference between

a photograph and a motion picture.

What distinguishes qualitative methods from other methodologies is that the qualitative researcher's questions are framed in general terms. Therefore, most users of the

,methodology try to begin a study without specific hypotheses or preconceived notions.

To enter a setting with a set of specific hypotheses is to impose preconditions and

perhaps misconceptions on the setting.'

During the first days of a qualitative study, we may find that our ideas and areas of

interest do not fit the program setting. Our questions may nat be relevant to the

concerns and behavior of the part\cipants. We will then begin to formulate a new

research design or new tactics and begin to ask different questions.*

or. Notes on Data Collection Procedures'

Once the basic approach of a knowledge deVelopment project has been determined and

a research design chosen, data must then e collected. Data collection is sometimes

considered to be a dirty job in program evalu. ion (or in any type of analysis for that

*An excellent example of qualita ve research in CETA youth prograrh' is Bonnie

Snedeker's Getting There, a monogr ph published by the U.S. Department of Labor in

1980. A procursor to this study as included as part of course training materials in

the Knowledge Development W rkshop series sponsored by the Office of Youth

Programs.

61
matter). For both the qualitative and quantitative approaches, the collection of data

probably consumes the greatest amount of time and effort in the evaluative project.

It is important that we purposefully decide what to measure. We could, in a youth

employment and training program, decide to measure an infinite number of things:

smiles per second, time scheduled for counseling, types of job placements, number of

participants. bringing extra materials to classroom activities, self-concept, etc. "To

measure" is used here in its broadest sense-to record inorder to summarize and report.

Clearly, we must maintain an intelligent selectfyity so that we can concentrate on

data items of demonstrable value.

Conducting a KD evaluation is a matter of gathering evidence to demonstrate the

attributes and effects of the program or one of its subcomponents. The program's

learning level position on the KID model, program goals, and the nature of our audience

will help us make gross decisions about what program variables to look at. For

example, we might measure some mix of:

a. Characteristics of the program context or environment (e. program

organization, time frame, physical environment-, class size, counseling

style,tietc.).

b. Participant charac tics (e.g., age, sex, socioceconomic status, language

ability, skill levels, attitudes, etc.)

c. Characteristics of cprogram implementation (e.g., activities, interTeiationships, administrative arrangements, principal materials, etc.)

d. Program outcomes Here, consideration,of program goals and objectives is

paramount. Be alert, however, to the results and types of outcomes totally

unanticipated by planners. a

62
0'

t

Beyond such general guidelines, decisions about precisely what information to collect

will be unique to each study situation. Every program has unique goals and generates ..

unique kinds,. of data. Though there is no easy way to decide what data to collect or

what variables to examine, here are some rules of thumb which may help:1

1. Focus data collection where you are most likely to uncover program

effects, if any occur.

/

2. Try to collect a variety of information.

3. Try to think of clever (and credible) ways to detect achievement of

program objectives (or other participant change).

4. Focus on the type of information you think members of your audience will

... look for when they receive your report.

5. Try to measure things that will enable you to further develop the youth

program approach_with which you are working, and'tha,t will advance local

understanding of youth programming in general.

The process of collecting data is considerably different depending on whether we are

using a qualitative or a quantitative design. For example, although interviews may be

.e. used in both types of research approaches, we will probably conduct the interview in a

very different way, depending on the methodology we' are using. As developed in

quantitative research, the activity of interviewing has increasingly been narrowed to

force a choice between rigidly formulated alternative answers attached to rigidly

formulated questions. ,Here is an example:

1 Morris and Fitz-Gibbon (1978), p. 20.

. 63

1

I

i

At
How interested are you in discovering new vocational opportunities? For

instance, how often do you use your school's career resource center?

A. Very often

B. , fairly often

C. not very often

An interview made up of such questions and utilizing pre-formed categories of answers

is often called a "structured interview." This more rigidly structured data collection

activity, so typical in quantitative studies, necessarily assams knowledge of what the

important questions are in the KD study and, more importantly, what the main kinds of

answers to these questions can be. To the extent tha/ we want to impose our questions

on others and/or know what is happening with the people we are interviewing, this is a

legitimate strategy. But to the extent that we do not want to make such an

imposition, nor assume in advance that we -already know a great deal about our

respondents' lives,. a different strategy of interviewing is, required a flexible

tr- strategy of discovery. Qualitative approaches to data collection are typically more

open and flexible.: An example of this iS*.the "unstructured interview" in which the

objeCt is not to elicit choices. between alternative answers to pre-formed questions,

but instead, to elicit from the interviewee what the researcher considers are

important questios relative to the KD project. Questions uncovered in this fashion

may theti ultimately provide the basis for a more structured interview or questionnaire.

Data for evaluative research can come from a great range of sources and be gathered

by an arsenal of collection techniques. We are limited only by our ingenuity and

imagination and, of course, time and money. Some possible data sources are:

I. Questionnaires

2. Interviews

64
3.

4.

5.

Obs.efvation

Psychometric tests of values, attitudes, personality preferences, norms,

beliefs

Rating by peers, staff, or experts

6. Existing plans, records, contracts, Monitoring reports, MIS reports, and

other statistics within our agency . A

q$ 7. Data collected by other agencies (for example, health records, employment

Y security records,retc.)

8. Tests of information, interpretations, skillsapplication of knowledge

9. Projective tests'

10. Diary or other per'sonal records

11. Situational tests. presenting the respondent with simulated life situations

12. Physical evidence - 13. Documents (such as the minutes-of board meetings,-news kaper accounts of

program actions, and so.forth)

4

14. Correspondence

A lot of quantitative evaluative research dependg on asking people for information.

Such an aipprdach relies on interviews and questionnaires to gather information about

youth program partkipants who they are, what it is they do in the'program, and

. what their behaviors and,agitudes are before and after participation in the particular

program. Frequently the staff can be queried.

u

. In educationally .oriented KD programs, tests are 'commonly used; they can provide

important data on knowledge and learning.

Sometimes we can find ways of collecting relevant data by "unobtrusive" methods that

do not involve asking anybody anything. Thus, for example, the recorded number of,

-4- 1 65

(
student visits to a career resource center over a period of weeks or months might be

correlated with actions to encourage such behavior. Sometimes counselor records can

be used for this. 7An excellent guide, full of imaginative examples and suggestions, is

Eugene Webb'sassic little book, Unobtrusive Measures. Observation can be a very

important tool for collecting data on both pre-and post program indicators. To

increase reliability, we should record observations immediately; if observations lend

themselves to easy classifications, they should be coded on the spot.

Youth program records -and other prime sponsor or subcontractor files' are sometimes a

very good source of evaluative data. Unfortunately, such records-are rarely as useful

as they should be. Often a prime sponsor's record keeping and its transfer of intake

and service information to permanent records tends to be fairly haphazard. Records

are often inaccurate, out of date, and the definitions and categories used by the prime

may not be appropriate for youth program evaluation purposes. MIS reports can

sometimes be very useful, though incompleteness and inaccuracies plague many MIS

systems. Indeed, record keeping procedures are occasionally changed; and if this

happens during the period under study, it can torpedo all attempts at before and after

comparisons,.

On the other hand, there are compensations for using the prime sponsor's or

subcontractor's records. One is saving the money and time that collecting original

data requires. Another is the advantage of continuity. Typically, the prime has been

4k, receiving a continual influx of information on the gram. Prime sponsor record

keeping or MIS systems may not be usable as is

accuracy and completeness. We may also attempt to

collection procedures within our organization, or int

knowledge development requirements.

66

- we should check carefully for

change certain standardized data

roduce new items more suited to
Intensive interviewing and participant observation are the data collection procedures

which are most common to qualitative program evaluation. Both of ;these approaches

involve 4/ in a more gradual unfolding of an outlook and understanding of the

participants in a particular youth program.*

A Comment on Data Analysis and Interpretation

The nature and organization of the data collected within the research design usually

dictates the level or complexity of data analysis which is appropriate in determining

the results of the study. Considerably different analytical tools are used in the

qualitative approach as opposed to the quantitative approach. Furthermore, within

each of these approaches a wide range of possible analytical methods is possible.

The data analysis and interpretation phases of a knowledge development project are

particularly critical in producing results which are likely to be well-received and

utilized. It is at this stage that we look at the data and try to make sense outof it. It

is important to separate analysis and evaluative interpretation,. For exa9ple, in the

case of quantitative analysis, we are involved with, organizing the data, constructing

appropriate statistical tables, and arranging for the data to be displayed in a orderly,

usable format. On the other hand, interpretation involves making judgment about

what these data mean, establishing the implications of the findings, and applying the

results of our investigation to possible future action. By separating out analysis from

interpretation, it is possible for a variety of users to look a; the data analysis alone,

come to their own conclusions, and make interpretations apart from the judgments of

the original evaluator.

*Two excellent guides to intensive interviewin participant observation, and other

qualitative research techniques are Lofland (197 ), and Bogdan and Taylor (1975).

67
Some, program evaluators strongly advocate the _evaluator's responsibility to draw

conclusions from data and make judgments about the evaluation results. Others argue

that the evaluator's job is merely to support the dita, and that the decision-maker

must make the judgments. (In many local youth staffi, of course, the evaluator may

also be-a decision-maker). An ideal*approach, which is a compromise between these

positions, involves the early presentation of relatively raw results to decision-Makers

and infOrmation users, giving them an opportunity to study and react to the data (

analysis without the intrusion of the evaluator's interpretation. In this manner, the

judgments and findings of key decision-makers and information us rs may become a

part of our own later interpretation of the data. In this way, th process of _local

utilization of youth program !earnings may get under way well before the final report

is distributed.

1

Generally speaking, there are simply no hard and fast rules for interpreting the results

of data analyses in youth program evaluation nor are there any rules foc drawing

conclusions from these results. If we have followed a well planned and relatively

objective evaluation procedure, certain findings may lle themselves to obvious

unarguable interpretations. However, other findings may fall into a less clear cut area

where alternative and often divergent interpretations are possible. At this point, the

experience of the analyst becomes very important. In many cases, we must rely on our

personal knowledge of the specific prom and components which are being evaluated

and come to some conclusions. Ideally, we will confer with program personnel or

"significant others" in order to clarify findings. In any case, we should always be able

to justify aid defend interpretations and conclusions on the basis of ,our data. In

addition, it is important to resist having our findings distorted in order to "document"

the preconceived notions of program or other personnel.

Three additional rules of thumb for analyzing and interpreting youth program

t8
evaluation data include:

1. Clear, readable, and simplified statistical tables are neceSfary to enable decision-makers and information-users to understand and' interpret the

data. Sophisticated analyses should be reworded and presented in simplified and tabular form.

. 2. Data must be "worked with" in order to.,be really understood. We must spend some time getting inside the data. Busy information-users and

decision-makers will be unwilling or unable to spend days at such a task,

but a fev.w hOurs of structured time spent by, an analyst can pay off

tremendously in a greater understanding of, and commitment. to, the

evaluation data.

3. Evaluators can learn a lot from decision makers' and information users'

interpretations of the data by remaining open and listening to what people

knowledgeable about the program have to say.

Documenting. Knowledge Development Findings

A critical missing link in the youth program learning efforts of many local youth staffs

is simply that of writing down a cogent description of the evaluative study and its

results. Considerable credit and advantage may be foregone if this last step in

methodology is overlooked or discarded. Although we distribute "problem" results

more selectively, there also exists a real need among youth staffs to learn what

appears not to work, as well as what appears to work well.

A glance at an outline of audiences interested in knowledge development findings

'should provide 'a sense of .the vahle of placing study results into a distributable,

discussable form:

1. Prime Sponsor (Staff and Adv ry Bodies):. o CETA Prime Sponsor Administrators

Program Planners /Evaluators

Program Vperators/In-House

Counseling Staff

Contractors (LEAs afrd other)

Advisory Council

Cit unpil, County Commissioners

7 S. ,

AM.

s.
2. Regional: . Regional Youth Coordinator

Federal Representative

3. National:

Office of Youth Pro§rams

Research/Evaluation Teams

.1

4. CETA Prime Sponsors:

, Primes within the State network

Primes within the National network

5. Public:

,--1 Elected Officials

Community-Based Organizations

Special Interest Groups

Press

Private Sector Employers

Unions

z

t

1'

Here are ways that youth program !earnings, if well-documented, can be used by each

N

of these groups:

1. Prime Sponsor (Staff and Advisory Bodies)

a. To assess the effectiveness of the youth program concept being

tested in terms, of its benefits to program participants and cost

effectiveness.

--' b. To inform administrators, program planners program operators, and

counselOrs of yOutti programming progress and outc mes.

/--

c. To present research project status reports to th theory Council.

d. To present a comprehensive assessment of the learning effort to the

. Advisory Council.

, 70

1 81

,
.....(.:--V:

e. Tp develop a substantive data base upon which to determine what

t 4 0 I

works best for whom under what circumstances; to re-direct pn-going

and future youth programming. - . . f. To develop a substantive data b4se to use in determining which youth

progt:am ptrojectt to delete, modify, expand.

CTo help further refine our knowledge base re: modification of the

overall delivery system that would result in. better provisions of , services to the client population.

h. TO insure that K'D assessment and evaluation data is utilized.

i. To provide a data base to analyze the idelivety capabilities of

°

(;) 'cbnteactors and program operators.

j. To determine what service mix is most effective in dealing with

signific\nt gtoup segments.

2. Regional Office .

a. To meet Regional reporting requirements.

b. To . inform the Regional Offi,c6 ofie status of youth program

learning efforts.

c. ''To satisfy the Regional Office Knowledge Development Report

requirements.

d. To insure that written documentation of the knowledge development

9

experience is completed to meet Regional Asfssment Reviews.

3. . National Office 8;

a. To inform the. National Office of successful program models and'

innovations. 4 b. To provide the National Office with a more comprehensive under- -.

standing of the dynamics of local youth program learning in .relation

to the overalkpr ogramming ef, fortsimiti4 local iiriSAtions:,--..., c. To,inform the Nationak,Office of .the barriers/obstacles encountered ;.!

,# , ... . f i * -hi carrying out the challenge of youth programming.

- . 1-

4
d. To develop a`local knowledge base that will assist the Natiorial Office, °-

in on-going national policy formulation.

e. Preparation of Annual Prime Sponior Report on Youth Programs:

,Knowledge Development.

. CETA/Prime Sponsor/Program Operatorl

a. To establish an informational base upon which to share youth program

successes and"non-successes with other CETA -Prime Sponsors.

b. c. TO provide local' contractors, program operators, and LEAs with

sRitrstantive information on local knowledge development efforts.

5. Public Consurnptioh

a. To inform City Council members, County Commissioners, and other

elected officials Of youth program evaluation efforts and outcomes.

b. To satisfy the increasing demand.f or accountability prompted by the

surge of press devoted to CETA.

c. To use for general public relations (presentations to parent groups,

ao butiness associations,-community-based organizations, news releases,

etc.):

d. Tdo use in soliciting the cooperltion of local public ,interest groups in

establishing a solid base orcommunity involvement-and support for

local youth injtiatives.

72

I 744

r

f 4

)
fiA I

E.

CHAPTER IV 0'

UNDERSTANDING NATIONAL KNOWLEDGE DEVELOPMENT

je

s

a

a

8,;
I

CHAPTER 4:

UNDERSTANDING AND USING NATIONAL KNOWLEDGE DEVELOPMENT FINDINGS

As explained at the outset, this guide is principally intended as a primer on methods of

knowledge development for local youth staff. Nonetheless, the guide would be

incomplete without an overview of the national KD effort, which was instigated by the

passage of the .Youth Employment and Demonstration, Projects 'Act of 1977. Many of

the findings and results of the nationarl ort are of direct use in planning and

operating locatyouth programs.

A detailed history of the Office of Youth Program's knowledge development scher:ne

44

and corresponding schedule of discretionarily funded projects is beyond our present

,..

. .. ., . . . scope, but we can note the basic rationale and organization of the national KD

approach. We can also identify the most direct means for local youth staff to explore

and, gain access to this burgeoning stockpile of useful youth program findings and ,-

results.

The Learning Focus of YEDPA s Learning about youth programs is the central theme ofaYEDPA. The Act was designed

to "try a variety of solutions to yoUth employment,problems," as well, as to research,

e ., demonstrate, explore, examine, study, and evaluate the solutions tried.

* The Off ic of Youth Programs (OYP) of the Employment and Training Administration

was, created, in pal-t;*to_design and oversee a number of discretionary projects in order

..

. . d

to answer specific questions generated by theVassage of YEDPA. It was OYP which

1

. ,. . . coined the term knowledge development - "a comprehensive systematic effort to \

identify, capture, 4nd replicate the lessons learned about the contributions of different

74 , -

,

O
services, participation incentiOes, and program management approiches aimed at

increasing the career potentials of young Americans."

It bears noting that DOL- sponsored efforti to learn more about the employment

problems of young people are not new. Although attention to youth employment

problems lessened in the early 70's, such efforts were a major DOL priority in the

1960's.*

As spelled.eout in a series of knowledge development plans, OYP's learning strategy

comprised a broad array of more than 200 discretionary projects that systematically

addressed critical youth employment issues. For the most part, these projects were

relatively complex, often involving sophistickted research methodologies, multi-site

defigns, and other considerations well out of reach of most local evaluation. staffs.

, Depending on how much was known about a ,iarticular youth employment issue, OYPIs

. - ' \ .

discretiorTy

,project( were,one (or a comblilatioa*of three possible sorts: \ , 1. Demonstration Projects - an array of activities and prcigram approach configurations having tcO dO, for example, with in-school programs, private 'sector

. .

. initiatives, service mix experiMents, educational vouchers, work and education

. . councils.

A spec example of a aemOnstratn project is the Ventures in demonstration CorrimUnity

Impro, mem Demonstration.- Among other things, this 2 year projec sought to. . rev whether it makes .sense to replicate a model community improvement

project in' widely different areas, This prbject, successful' in POrtlan.d, Oregon,,

was tried in 12 U.S. cities.

*For a synthesis and. review of earlier youth prograin efforts, see Mangum and Wa!

(1978), a1 Walther (1976).

75 ;1> 8 G

'1.1
<-

2. Special Research Studies,- 'designed to prov,ide very basic information and

analysis .conceming youth employment problems and ;o assess the performance

of programs which seek to better the circumstances of youth.

The Youth Labor Market Experience Study-is an example of such a research

project. This study analyzes youths' labor market behavior 4 to 5(Years after

graduation from high school.

3. Program Evaluation - primarily prows and outcome evaluations of programs

already in place, typically as formula-funded youth programs.

4

The Process Evaluation of Youth Programs, performed by the National Council

on Employment Policy (NCEP), is an example of ea .national KD evaluation

project. It assessed the planning, start-up, and operational experience of YCCIt

and YETP in relation to other employment and training programs for youth.

OYP initiated ,projects addressing critical youth employment issges- in a manner which

was thought. to systematically focus new knowledge development'efforts on areas that

t needed attention, building on what was known, and focusing on the next appropriate

1 level of inquiry.

7

.3 The logical elements of OYP's systematic approach to focus and organize each

discretionary project are already familiar to readers of this guide as the steps

identified in the local KD model (Figure 1-2). Of course, OYP applied the logic of the

KD model on a national, rather- than a local, scale. When OYP accomplished such a

step in thp model as "Current Youth Program Assessment"' or "KnOwledge Base

v ,e Assessment," it was with reference to a generalized, aggregate piCture of the youth

employment and raining prqgram needs of the entire nation. 'When.the "learning

4
levels" assessment within the KD model was applied, often to a multi-site demonstration design, the implications of the selection of an appropriate learning, level held

firm. For example, although OYP might simultaneously explore a new program odel

at a number of different sites, the effort' was still typically conceived s a

"demonstration" program, and process-oriented assessment techniques were empl yeti:

Correspondingly, outcome/impact research and evaluation techniques wereemployed

in as.ses§ing national projects about which more was originally known.' t

What is significant for local youth staff is that the admittedly awesome OYP

4.) /

discretionary effort can be understood in terms of its many individual projeCts, each of

which is examined according to the same fundarnintal KD process described in

Chapter 1 of this guide. Thcbroad rang,of issues examined by OYP virtually assures

A that the findings of at least certain national projects will be of use to local youth

staff. Indeed, because of the resources t5YP was able to invest in most of its

discretionaryefforts, a given set of national findings may prove to\be among the most

valuable of "outside resource" considerations made by a local youth staff in conducting

their van local KD effort.' For example, OYP. has typically examined. specific youth

program models in a relatively controlled multi-site manner. This approach improves

both the likelihood of aribing, causality to program events, as well as the chances of

, \

the model program having been demonstrated within a jurisdiction' which more closely

1,

\ . approaches that of interested local yoUth staff. Of course, we can never assume that

\ ..... "irriported" findings will per\fectly fit specific local circumstances, but a cloPhr fit may

:_ provide much greater insight in del/eloping.local programs..

A Discretionary Study a Voc4tional Exploration

-

4. ( u 1 e

' t

4

e .

I

.

,... Exalnination of the Vocational \Exploration Demohratlon Project (VEDP) may provide

. *2 Q . i -,,

a sense of the scope and rationale of national knowledge development. projects. VEDP

77

411
is a multi-site test of alternative approaches for conducting a local vocational- exploration progr

Vocational exploration is an approach to youth prOgramming which is familiar to most

youth employment and training staff. Generally stated, vocational exploration is an

educational and motivational experience through which youth learn about the world of

work, look at different occupations,' jobs, and career opportunities, gain experience

usefUt in making decisions about future' directions, and develop a firm behavioral

41.

foUndation from which to proceed.

However', many specific variations and refinements to the basic approach have

occurred since the eprliest vocational exploration efforts in 1971. Originally little

0

more than a work/experience program,,various cognitive and affective enrichMents

have come to be seen as essential to the approach. Typically utilized in summer

programming, some youth staff saw year-round value in the vocational exploration

.approach. OYP's' decision to systematically modify and replicate a number of

vocational exploration models was, thus, a national outgrowth-of the hodgepodge of

t

Sometimes conflicting information which had accumulated around the basic vocational

-4-exploration approach.

Teaming up with the National Alliance of 8.1sineSs, the Human

Resource Development Institute jof° the AFL-CIO), And the St.

Louis University Center for Urban Programs', OYP, designed. VEDP

to test -alterntive vocational 1-xplora'tion ,approaches of varying.

duratibn and with different' Varget groups,

O 78 8'
s

l46

A The Vocational Exploration Demonstratio Project operates in 16 cities

throughout th'country and tests five (5) basic program models which employ

different strategies for providing a vocational experience for youth. Each

of the I§ program operators implementoa specific program design based on

ohe'of the following

- fivesmodels:

1. Onsite Exposure is occupational exploration through actual

placement at a private or public sector employer's place of

business for worker shadowing, job/task observation and

rotation, and limited practical "hands-on" experience.

2. Vocational Exposure Laboratory is occ upational exploration

through classroom activities, '#survey style's vocational

training, lima skill instruction and simulated, "laboratory" mock-up work-conducted, at union trade instruction

institutions, vocational schools, skill training centers,

, or community colleges.

----- 3. Eclectic Exposure is occupational exploration through a multiple

process approach which combines several basic elements into a planned program intervention strategy. .These activities include

but are not limited to field trips and tours of businesses and

industr'es, films and seminars given by.various employers and

businessmen, presentation of occupational and vocational infor- °'-s,,

.4 oration, sessions on survival skills, and youth projects such as

small enterprises.

,

. t 0,

.. 4. Multi Modal Exposure provides for the oper4tion of various mixtures

of the three basic types'described above.

5. Extension Model Exposure provides Onsite Exposure or Multi Modal.

,

--..\---7.'"4. Exposure in a gequence4of activities that extend over a twelve-

,

a

AOnthveriod of time.

, )

-7 9

4
e length of the experience that h9.the most impact on participants, and the

types of labor market conditions that are best suited to,the five basic

All of the local programs have three c onents--one serving in-school

you h, another serving out-of-school yout and a third, offered during

the summer, for both in-school and 'out-of-school youth. Within each of

th three components, all Youth receive the same mix, of activities and

se ices called for by the specific program model beinglbperated. There

ar= four major areas that, will be investigated through VEDP. These include:

examining the relationships, in a variety of vocational exploration

models, beOreen the people served, the program activities and

services, the results end the environmental factors.

comparing summer and n,onsummer vocational exploration efforts.

comparing the effects of exposure to vocational exploration

strategies for a 120month period to similar activities and

services offered for shorter periods of time.

investigating the effects of vocational exploration programs

upon the attitudes and institution)) behavior of business and

organized\labor.

This national demonstration will further indicate the types of youths that

most b-&nefit from the vocational exploration experience, the specific ,

r

program model that produces the best results for' certain types oybuths,the

program models. Local progralt operators will be able to detetmine the

amount of programatic and supportive services that are required to assist

in the transition to the world of work. The findings will further indicate

the kinds of implementation strategies that are needed to work withle

private sector in VEDP as it relates to size and-typesof employers and

the role of the supervisor. Finally, inforMation gn the ease with which

linkages with L 's and'other local actors are obtained in implementing a

I

.4 (

b..

0

3:

L1
program, as well as the influence of linkages on the transitional readiness

of participants will be available. In addition to the evaluations, the

products will include technical assistance guides, monograpi and o,

model management information system. A system is being developed.which

will provide for a built-in technical assistance capacity through program

committees made up of experienced VEDP operators and specialists from

the Natidnal Alliance of Business and the AFL-CIO Human Resources Develop-

. ment Institute. These committees would provide informatiomon every aspect

of the program including the design, planning, marketing, implementation,

administration and research.

Quite clearly, these results will be important for prime sponsors and

delivery agents who are considering or have decided on implementing

vocational exploration activities. The aggregate findings will suggest

whether and to what extent vocational exploration can change awarenesses

441. and outcomes for youth. The vocational exploration approach will be

more effective in some settings than others. Prime sponsors and delivery

agei1ts should 'pay particular attention to the relative effectiveness

demonstration projett sites with siMillar characteristics to their

This will suggest whether and what model to impleMent. NeXt, the reN' Its

will indicate what groups benefit most from VEPs. The findings can be ,-

used to establish target group priorities for the projects that are mounted.

The process studies and how- to -do -it guides, will be useful inthe'implementtien phase. Prime sponsors or delivery agents might ava. themselves

of the technical assistance network established under VEDP, tappipg the

expe-tise partidularly of nearby sites. The evaluations and instruments

used in the national demonstration might tie adopted locally to refeence,

pgrformance to national norms. In other words, by. careful utilization- of

41 114i

4
4

O

the n'htional demonstration findings, local decisionmakers can determine

0 whether to implement VEPs, the best. type and target group priorities, as

well as getting help .in lemention and assessment.

Finding Out About National KD Results

A range of forlal and a informal youth proAgm informatiOn sources

have been mentioned'in this guide. Short of accidentally happening . upon someone who has experience with.precisely the type of program

approach, jurisdiction, and clientele as our own, by far the most

efficient avenue to youth programming information and assistance in the

new "information brokerage" established under the auspices of OYP's

Technical Assistance and Training Division. . The function of the youth' information broker is to act as an intermediary in matching

youth program people, who have questions about a particular youth issue area with

e 4

I ir8 2

O

4

6

.11

\

r

r 11
I

et However, ,to r duce the number of research variables, a number of program elements

are maintained ip common at all sites; such controls include program philosophy and

purpose, clien't eligibility and selection, program reporting formats; and a number of

other elements.

-I

. Many answers about.vocatioh ekplbratidn'programming are emerging from VEDP; in

some cases, these questions are taking the form of more refined questions, that

*quire additional examination. 'While While such findings are of value to national youth

.°'. . planners in, for example, changing and clarifying youth regulations, they are also 43f

enormous potential benefit to lbcal_youth_staff in their efforts to develop and

implement workable youth programs. The question arises, however, of how local youth

staff, typically short of time 'sand moray, can gain access to such findings as ace

emerging from VEDP. If this question is not 'effectively addressed, there is little

chance that local youth staff will directly profit from national knowledge development

findings.

Fin5ling Out About National KD

A range of formal and informal youth program information Sources have been

mentioned in this guide. Short of accidentlly" happening upon ,sonieorie who 'has

experience with precisely the type of program approach, jurisdiction, and clientete as

our dom, by far the most efficient avenue to youth programming information and

assistance is the new "information brokerage" established under the auspices of.OYP's

TechnicalAssisfance and Training Division.

-. The funttion of the youth information brokers is to act as an intermedia6/ in, matching

I , . youth program pi'bple who have questions ,about a Rarticulhr y uth issue i eta ,with

$. , . .

,. /

. t

. ,

83

9e

94

0 0,
a

youth program people who have some answers and experience in that issue area. For

example, let's assume that our assessment of-ilocal youth employment and training in

our jurisdiction suggests that a ,vocational exploration program would be `worthwhile.

Exciting as the prospect may be, the often conflicting details of how to actually tg,

begin to implement an approach will quickly convince us that some experienced advice

would be invaluable. We decide to make use of the youth information broker.

Contact with the OYP youth information broker is initiated through an authorization

request to the DOL regional office. After obtaining the necessary authorization, local

youth staff may telephone the brokerage service directory. Brokerage staff will listen

to our request, perhaps assisting us in clarifying or refining the typp of question we

have asked. In a reasonably short time,we may-expect three types of replies to our

request:-

. , a) the results of a bibliographic search through educational andJamploytnent

I and training information banks--including, of course, the latest available

findings from OYF's discretlonarily funded KD projects.

b), a listing incoming natimial, regional, and local training, events that are

licable to our Est.

A

4 delineation of several negotiable technical assistance oRtions tailored to

oue,specific.41eds. ,

, . 4

' : , /11 V ' / *,, 1 ^ .* . 1 "4, t° ew , 44' \ la

to' ,

fr A part'ofSthe broker's function is guide u§ to the most useful'aids'among Ivai wealth

r.41 * ' 4. w. ' ;Iv If 4511fr. ' ,,, liP . 1 :fr 2,,. 1 ; . -

, fr , : w

, ..,

v

fr of iesourdes. that ar" ava atzte. i§of aided, wed may qu'ickly. ,discoyer sbcoe of tile,'' <ii' , 4).. r r

.

, 4

, . . r4 1 : 4, ,,,, answers , to 'our' more., obvious program imprsrhentation ,questions. aboyt vo at,ionil ,, ,..,yA

0

. , '% .

'1,, ,.. . . , ,. 1 ., 1 v4, .., N., xexploration; such as what basic progliagn knOciels ,exist, s; Ping, budgeting, avid it n. :,e-44

;

1--e'f

it, I ., 1 .. - 4 ' e.) Howeyer, we may,. ilsi..be alerted fo a variety of ,issues arii questOoDs we May' not yet.

/

* A ' V.!" : . % -' .l. have asked,about vocational explbratiorr programs. What ..,abut. the intricacies of

4 W. s - .

4. Vi. '. establishing local, linkages with LEAs ark CBOs".' What about the Issue of

r

\

;

16.
n

remediation among youth participants .with widely varied skill levels? bo the needs of

in- school and -of-school youths. differ, such that different types of vocational

exploration progra approaches are called for in each instance? What about

expe-drttions of private sector employers participating

the best methods f or marketing the program concept to

L. job site 'development, and how much:rOation is it fair w employers as well as participants? What are the key iss

coping skilfs cutriculdm design? And so on. it \

Utilizing. National Findings Locally

in the, program, And wiiat are,

these People? Haw difficult is

3- '4t.

to expct,of cer,tain :types of ,

ues and cholei in 61assroom or .0;

e, 1

t r.

,

, . Our own local circumstances,will, of course, be unique. Nonetheless, in implementing

.4. . .. ir." , -- I, . . v our vocational exploration program, 'we may clearly,benefit fl'om the 'hard. won adviceo , ... .

. ., ,

.

. of others involved with similar projects. -,,,, / A.

.> "

A , .- ,

A I ..

4

1/4, A. 4, 44. .

1 V. 4 .. ; ' : 01 ," 4 : o. 0

4

Ina, siftint and .sorting national f. ins, ye are 4t. one of the4impontant jruqtipns the't

... A

04 ' i . l'ci ; av .

, ,

'',. ii ; \

v

I ' yt.

connects 'aid ihtertNytnes and natiortil'kng*fedge development, effo-A-.

N

b

f

impprte, ition pit..,,

itei.

delYel 'ent Oocess44,..p

'..Q ..

.,,,

, We may "ben444 loc

raw mateeial

S

considiingfare, 1-1;mlelV44,pduCtS pi the knowledge"

Rued learninetevqls approPril otherst.circumaandes:

nor vets -

y frovrese findings, but only to the 'extent that we use them as'

in.our own version of the same KD process, and at the learni g level that

is afpropriate -cor us. Asdiscussed earlier, the particular way in which we respond in,/

4 _

our own program developn(ent cycle depends on a host of circumstances which deflq

'otir unique labor marketebvironment, clientele, staff, jurisdicition and so oh.' We may

conduct a simple model demonstrationol a design' based on an elaborate replication of

f a multi-dimension, model tried successfully elsewhere, We may design and deliver

these services with our own staff, or subcontract them to. a CBO or\other community

- or institutional bod accordpig to our specifications. ', These are 'Unique actions, but,

85

9 1;

O
viewed more-broadly, they are actions taken within the overall knowledge development

process; the fundamental logic Of the process holds, regardless of scale (national or

local), Specific learning level, or the sophistication of staff or type of jurisdiction.

The ND process is a powerlul tool; as with any tool, one must learn how to use it with

purpose and care,.

.

NM.

9

I 9"

ti

A

J

CHAPTER 5

SELECTED REFERENCES O

The following selection of references is not meant to be complete. Instea

represents a thoufhtful culling through the enormous applied social resea (literature, which may be of some help to busy, often harried, youth4planne evaluators. In

particular, the literature of social program evaluation has blosso (1 in the past dozen

I years as the field has come o age. However, muCti of thi itecature-is

and an iittempt has'been made to eliminate redundancy.

The'references are broken into four categorizes'

I. Knowledge Development /Youth Programs

IL, Program Evaluation: Employ eryt Related

III. Program Evaluation: Gen= ral and Methodological

IV. Social Research Metho s: Gener 1

/

I. Knowledge

1: A Gui

2.

metix/Y th Pro rams

, Development. Prepared for U.S. Department of

Labor mp oym t and Training Administration-, Office of Youth Pro-

'gram 1979. ,

lengthly re

a series of

of Y Uth

N/ ...,.. ur e guide for local prime sponsors; originally prepared for

hops on Knowledge Development sponsored by the Office

Br ndia ni

ms.),

ersity Center for Public Service. Youth Programs: Knowledge

t Through the Youth Initiatives.

ful, irregularly published newsletter.)

88

p 9:j
t

3. "Completing the Youth Agenda: Plan for Knowledge Development,

Disemination.and Application for Fiscal 1980." U.S. Department of Labor,

Employment and Training Administration, Office of Youth Programs,

October, 1979.

(A compilation of the final national KZ philosophy and discretionary'

program thrusts of the Office of Youth Programs. Also see earlier plans

for FY'78-and FY'79.) 1'

4. Mangum, Garth, and Walsh. Employment and Training Programs for Youth:

What Works Best for Whom? 'Washington, D.C.: U.S. Department of Labor,

Employment and-Training Administration, Office of Youth Programs, M.ay,

.1978.

(An extremely useful review \and synthesis of 17 years of public experimentation with employment and training programs for youth.)

'5.. Publication List, Office of Youth Programs Library, 601 "D" Street, N.W.,

Room 7108, Washington, D.C. 20213.

(A complete, annotated listing of all current publications of the USDOL/

ETA, Office of Youth Programs. Write for the most current listiag.) 4.0

6. Snedeker, Bonnie. Getting There: A Case Study Report on ,the Lives, c Employment Preparation, and Prospects of YEDPA Participants. Washing-

. tdn, D.C., U.S. Department of Labor, Employment' and Training Admini- \

stration, Office of Youth Programs, 1980. f 5 (A clew illustration of the use of intensive interviewing methods in youth

program assessmentl.this excellent work provides eloquent testimony to

' . the use and value of qualitative assessment techniques in youth progrkns.

I

89

.' 1

9."

T.
tg

to

IL

7. rational Foundation, Inc. You and Youth. "I'

(A monthly newsletter devoted ;'What American business is cloing to

employ and train young people. ")

a

Walther, Regis H. Analysis and Synthesis of DOL Experience in Youth in

Transition to Work Programs. Wa.srangton, D.C.: U.S. Department of

Labor, Employment and Training Administration, 1976.

(Good, though now dated, overview of DOL experience' with compbnents of

employment and training programs for youth.

9. "Youth Program Models and Innovations." U.S. Department of Labor,

..gmployment and Training Administration, Office of Youth Programs.

(A monthly series of descriptions of local program innovations distributed

10/78 - 10/79.)

Program Evaluation: Employment Related

1. , Bennett, E.C., and Weisinger,' M.

Handbook for Vocational Rehabilitation. New

and Research Center, 1974.

Program Evaluation: k, Resource

York: ICD Rehabilitation

(A manual intended for vocational rehabilitatiOn program evaluators, which

is of value to prime sponsor evaluators.

A

, 1.

. ,

. -,, 2. Borus, Michael, E. Measuring the' Impact of Employthent-Related Social , _ ,

,F. 1 , : . Piograms. Kalamazoo, Michigan: W.E. Upjohn Ins;itutefor Employment

. . ' >. Research, 1979.

(Devoted entirely to a discussion of dutcdme, rather thin proces§, evaluatom)

90 U
3. "CETA: Comprehensive Program Review and Assessment Training." A

technical training package developedifor the U.S. Department of Labor,

Employment and Training Administration, 1977.

(A loose leaf publication which includes many exercises dealing with a fullA

'range range of prime sponsor assessment *techniques: - orientation, planning

assessments, monitoring, evaluation, and utilization of assessment information.)

4. Mangum, Garth, et al. Self Evaluation of CETA Manpower Programs: A

Guile for Prime Sponsors. ,CETA Technical Assistance Document. Wash-

, ington, D.C: U.S. Department of..Labor, Employment and Training Administration, 1975.

(A practical systematic guide to local monitoring and evaluatiOn of all

CETA 'titles.)

5. Sum, Andrew, et. al. Evaluating the Performance of Employment and

Training Programs at the Local Level' (Vols. 1-3). Boston: U.S. Dept. of

'54

Labor,, and Training Administration, Region I, 1978.

(A very complete guide to evaluation at the primkponsor level with a

complete discussion and examples of process, outcome, and benefit-cost

evaluation techniques. An entire volume is dedicated to conducting and

using the results' of follow-up studies.)

, .

III. Programs Evaluation: General and Methodological

1. Cook, Thomas D.; and Reichardt, Charles S., editors. Qualitative and

Quantitative Methods in Evaluation Research. Beverly Hills: Sage

Publications, 1979.

(A collection of artiles which deVelop the case for 'using a' blend of both

qualitative and quantitative techniques in program evaluation.)
s

2. Epstein, Irwin, and Tripodi, Tony. Research Techniques for Program

Planning, Monitoring, and Evaluation. York: Columbia University

Press, 1977.

(A good quick overview of, research concepts and techniques for administrators.)

3. FyaluatisuLaadPmgram__Planning._ ___New_ "__York:--P-ergarnmon-Press,---Pub1 1.

lished quarterly.

o4S

ti

4. 'Evaluation Review: A Journal of Applied Social Research. Beverly Hills:

Sage. Publications., published six times yearly.

(One of the principal, MultiClisciplinaryforums of evaluatidn researchers.) ft.

5. Isaac, S. and W.B. Michael. \Handbook in Research and Evaluation. San

Diego: Edits, 1971.

(A very handy, educationally oriented digest of principles, methods, and

strategies useful in all phases of evaluation research.

6. Levitan, Sar A., and Wbrzburg, Gregory. .--Evaluat ng ,Federal Social

Programs: An Uncertain Art. Kalamazoo Michigan: The W. E. Upjohn

it

Institute for Employment Research,11979.

(A broad scale assessment of the n ture and value of DOL and HEW social'

program evaluation)

-7. Morris, Lynn Lyons,'' and C.T. Fi z-Gibbon. Program Evaluation

Beverly Hills: Sage Publications, 197. Includes the following books:

a. Evaluator's Handbook

92

4
b. How. peal with Goals and Objectives D

C. How to Design a Program Evaluation

. d. How to Measure Program Implementation

e. How to Measure Achievement

f. How to Calculate Statistics

g. Hpw to Present an Evaluation Report

h.

(An e,xcellent set of books, rich in procedure and example; examples are

primarily from the field of education.)

8. Patton, Michael Q. Qualitative Evaluation Methods. 'Beverly Hills: Sage

Publications, 1980.

(An in -depth exploration of strategies for generating valid, useful, and

credible qualitative information for decision-making.)

9. Patton, Michael Quinn. Utilization Focused Evaluation. Beverly Hills:

Sage Publications, ,1978.

(Combining the theoretical and the practical, Patton examines how and

why to conduct evaluations. His principal concern, is for enhancing the

likelihoodfor actual utilization of evaluation findings.)

10. Rossi, Peter H.et. al: Evaluation: A Systemmatic Approach. ,Beverly

Frills: Sage Publicatiops, 1979. 4

(A comprehensive text dealing with Planning, designi4, and implementing

evaluations of both established and innovative social programs and pro-

, jects.. 'Includes detailed treatment of Cost benefit and cost effectiveness

techniques.)

14 93. 1 CL,4`
1

o

II. Struening,. Elmer L. and Mar cia Guttentag. Handbook of Eva luation

Research. (Vols. 1 & 2). Beverly Hills: Sage Publications, 1976. , .(A techni61 but readable compendium of the methods required at every

step of the evaluation process. rrobably. the 'easiest way to scan the

classic, articles in the field.)

12 a_ Sififfleb-eam, Daniel L. -IrE\iItnatrob as Enlightenment for DecitIon Making." -An address delivered at the Working Conference on Assessment'

Theory,'The Association for Supervision and Curriculum Development,

Sarasota', Florida, January,,1968.,

13. Suchman, E.A. Evaluative .Research. New York: Russell Sage, 1967.

(Fairly technical and-academic, this is one of the early classics of the

11, program evaluation literature.)

14. Weiss, Carol H. Evaluation Research. Englewood Cliffs: Prentice -Hall,

1972.

(A good basic overview of the purpose and politics of social program

evaluation.)

IV. Social Research Methodology: General

1. Blalock, H.M. Social .Statistics (3rd edition). New York: McGraw-Mill,

(

1979.

(An excellent text, one of the current standards of statistical methods , courses in sociology.)

2. Bogdan, R., and Taylor, Steven. Introduction to- Qualitative Research

Methods. New York: John Wiley 1975. , 94 ,
(Good rationale and illustration of .how to conduct and report qualitative

research.)

3. Campbell, Donald T:, and ,Stanley, 3.C.. Experimental and Quasi-ExpC

erimental Designs for Research. Ricago: 'Rand McNally, 1963:

(Academic, kthe classic statement of "quasi-experimental" methods which

may possibly be used when prografn conditions will not permit "true"

excieriMental designs.)

. 4. Eckhardt, K.W.'arki Ermann,-M.D. Social Research. Methods: Perspective,

. 'Theory, and Analysis New York: Random House, 1477.

(A basic, comprehensive research methods text.)

5. Gay, L.R. Educational Research: Cornpptencies for Analysis and Appl-

. ication. Columbus, Ohio: Charles E. Merrill Publishing Company, 1976.

(A relatiltely ,comprehensive introductory text on the methods of educational research.)

6. Gay, L.R. Educational Research: Models for Analysis and Application.

Columbus, Ohio: Charles E. Merrill Publishing Company, 1976.

(A very basic, self-paced introduction to planning, conducting; and reporting simple educational research. This manual is designed to ac,company

Gays Educational. Research: Competencies for Analysis and Application.)

7. Hays, W. Statistics. New York: Holt, Rhinehart, and ,Winston: 1963.

(There are many statistics texts, but this one remains an academic classic;

examples are principally from psychology.)

8. Lofland, John. Analyzing Social settings: A Guide to Qualitative

Observation and Analysis. Belmont, CA:VadswOrth, 1971.

95 . UG

I
VW,

(A readable guide to qualitative methodology. Strong on techniques of

observation and analysis of qualitative rmaterial)

9. Mager, Robert F. Goal Analysis. Belmbnt, CA: Fearon, 1\972. %

(A short; humorous, and very readable discourse on goal ;stablishrnent and . clarification.)

10. Nie, N.H. et. al. Statistical Package for the Social Sciences (SPSS) (2nd

edition). New York: McGraw -Hill, 1975.

(A tremendous tool to know about SPSS. Is probably tile best of the easily

'used "canned" statistical programs.)

11. Stevenson, W. Quantative Analysis in Human Resources Research. Salt

Lake City: Human Resource Institute, 1.976.

9

.

1 (On e few introductory statistics texts specifically oriented toward

'--) employm t and training practitioners.)

12. Webb, E.,J. et. al Unobtrusive 'Measures: Non-Reactive Research in the

Social Sciences. Chicago: 'Rand McNally, 1966.

(A unique explanation of how physical evidence, secondary data, and simple

observation may be used as alternatives or supplements to interviews' and

questionnaires in social science r'search.)

/-

13. Wright, Sonia R. Quantitative Methods and Statistics. Beverly Hills: Sage

. Publications, 1979.

(A "survival 'Manual" approach intended to serve as a quick and straightforward overview of the principles and techniques of 9uanititative social

research.) e"

96

FN. 1

4