Design, Implementation, and Evaluation

of the Work First Profiling Pilot Project

Prepared for the 

Employment and Training Administration

U.S. Department of Labor

by

Randall W. Eberts

W. E. Upjohn Institute for Employment Research

Kalamazoo, Michigan

March 8, 2002

The author acknowledges the valuable assistance of Stephen Wandner, Ronald Putz, Jon Messenger,

and Janet Javar of the U.S. Department of Labor; and Timothy Bartik, Christopher O’Leary, Lillian

Vesic-Petrovic, Radika Rajachar, Kris Kracker, Robert Straits, Craig Schreuder, Phyllis Molhoek,

and Claire Black of the Upjohn Institute.
i

Abstract

This report describes a new assessment and referral system that was designed to assist local

welfare-to-work program staff in targeting employment services more effectively in order to help

welfare recipients find jobs. The motivation for the development of this system was the potential effects

of targeting services to meet the specific needs of customers. The system is based on statistical

methods and uses administrative data typically collected by welfare-to-work agencies. The

Kalamazoo-St. Joseph Workforce Development Board piloted the new system by integrating it within

the existing Work First program that it administers for the local workforce development area. The pilot

was conducted from January 1998 through March 2000, during which time more than 6,000 welfare

recipients participated in the program and used the assessment and referral tools. 

At the time of enrollment in the Work First program, staff used the statistical tool to make an

initial assessment, referred to as an employability score, of each participant’s ability to find and retain a

job. The staff then used the individual employability scores to refer customers to service providers that

offered the set of services and pursued an approach to delivering services that best met their needs. 

An evaluation of the pilot, based on a random assignment design, found that referring

participants to service providers according to their employability assessment increased the overall

effectiveness of the program. Using a job retention rate of 90 consecutive days as the employment

outcome, the optimal referral pattern based on the statistical assessment tool yielded retention rates that

were 25 percent higher than if participants were randomly assigned to providers. The analysis also

found that the difference in retention rates between the best and worst referral combinations was 56

percent. Using earnings as a measure of the additional benefits to participants of the new system, the

benefit-to-cost ratio ranged between 3.25 and 5.8, depending upon assumptions regarding the length of

time the earnings differential between the treatment and control groups persisted. The system was

designed to be integrated into most existing welfare-to-work programs and once operational to require

minimal (if any) additional staff. The W. E. Upjohn Institute developed the system, with funding from

the Employment and Training Administration of the U.S. Department of Labor. 
ii

Contents

Executive Summary ............................................................. v

I. Introduction ................................................................. 1

A. Purpose of the Work First Profiling Pilot ...................................... 1

B. Welfare Reform and Michigan’s Work First Program ............................ 4

C. Targeting Services to Welfare-to-Work Participants ............................. 7

D. The Work First Pilot Project’s Statistical Assessment and Referral System ............ 10

II. Statistical Assessment and Referral Model ......................................... 13

A. Overview ........................................................... 13

B. Designing the Optimal Allocation Mechanism ................................. 13

C. Data Requirements and Availability ........................................ 17

D. Characteristics of Work First Participants .................................... 19

E. Reasons for Leaving Work First ........................................... 21

F. Work First Activities ................................................... 23

G. Differences in Activities among Providers .................................... 24

H. Employment Outcomes ................................................. 26

I. Estimating the Statistical Assessment Model................................... 27

J. Assigning Probabilities on a Weekly Basis .................................... 37

K. Alternative Specifications ............................................... 39

III. Implementation of the Statistical Assessment and Referral Work First Operating System ....... 41

A. Assignment of Providers to Serve Customers with Specific Scores .................. 41

B. Intake Software Development ............................................ 43

C. Intake Procedure ...................................................... 45

IV. Evaluation of the Kalamazoo/St. Joseph Work First Pilot Project ......................... 47

A. Purpose and Design of the Evaluation ....................................... 47

B. Random Assignment Procedure ........................................... 49

C. Program Outcome: Retention Rates ........................................ 53

D. Variation in the Outcomes of Different Combinations of Providers .................. 57

E. Benefit/Cost Analysis of the Statistical Assessment and Referral System ............. 64

V. Replicating the Pilot in Other Areas: Broward County, Florida .......................... 70

VI. Conclusion ................................................................ 72

A. Statistical Assessment and Referral System .................................. 72

B. Evaluation Findings .................................................... 72

C. Extensions to Other Sites and Programs ..................................... 74

References ................................................................... 76

Appendix 

Description of the Automated Database Implemented by

Kalamazoo/St. Joseph’s Work First Profiling Pilot Project ........................... 78
iii

List of Tables

Table 1 Job Retention Rates by Provider and Participant’s Education ................... 15

Table 2 Outcomes of Two Hypothetical Assignment Rules .......................... 16

Table 3 Variables Used in the Work First Statistical Assessment Model ................. 20

Table 4 Termination Types ................................................. 22

Table 5 Selected Activities of Work First Programs ............................... 23

Table 6 Distribution of Hours Engaged in Assessment and Employability Planning .......... 25

Table 7 Distribution of Hours Engaged in Job Search Activities ....................... 26

Table 8 Logit Estimates of the Basic Statistical Assessment Model .................... 28

Table 9 Characteristics of Participants with Low and High Probabilities ................. 32

Table 10 Characteristics by Quintiles of Predicted Probabilities ........................ 36

Table 11 Relationship Between Actual and Classified Events ......................... 37

Table 12 Assigned Predicted Probabilities by Weekly Intake .......................... 39

Table 13 Logit Model Specifications ............................................ 40

Table 14 Correlations of the Predicted Probabilities of Different Model Specifications ........ 40

Table 15 Estimated Employability Scores by Employability Groups ...................... 53

Table 16 Years of Education and Age by Provider and Employability Group ............... 54

Table 17 Retention Rates by Provider and Employability Group ........................ 54

Table 18 Upper and Lower Bounds of 95 Percent Confidence Intervals for the

Retention Rates of Each Provider ................................ 55

Table 19 Comparison of Retention Rates from Single and Multiple Appearance Samples ...... 56

Table 20 Number of Participants Assigned to Each Provider .......................... 57

Table 21 Number of Participants Employed 90 Consecutive Days by

 Combination of Providers ...................................... 58

Table 22 Differences in Retention Rates Between Pairs of Combinations of Providers ....... 60

Table 23 Estimated Retention Rates Using Logit Estimates Controlling for Admission Date .... 62

Table 24 Retention Rates by Combination of Providers .............................. 62

Table 25 Differences in Retention Rates Between Pairs of Combinations of Providers

 Using Logit Estimates Controlling for Admission Date ................. 63

Table 26 Differences in Retention Rates Between Pairs of Combinations of Providers Using

Combined Treatment and Control Groups ........................... 63
iv

Table 27 Hourly Wages and Weekly Hours of Participants Working 90 Consecutive Days .... 66

Table 28 Average Weekly Earnings by Different Combinations of Providers .............. 66

Table 29 Difference in Earnings between Treatment and Control Groups and Benefit-to Cost

Ratio of the System, Assuming Retention Rates Remain Constant ......... 67

Table 30 Difference in Earnings between Treatment and Control Groups and Benefit-to-Cost

Ratio of the System, Assuming Retention Rates Converge ............... 68

List of Figures

Figure 1 Predicted Probability of 90-day Employment ............................... 30

Figure 2 Actual and Predicted Probability of 90-day Employment ...................... 31

Figure 3 Grade 12 Completion and Predicted Probability of 90-day Employment ............ 33

Figure 4 GED and Predicted Probability of 90-day Employment ....................... 34

Figure 5 Prior Employment and Predicted Probability of 90-day Employment .............. 34

Figure 6 Noncompliance and Predicted Probability of 90-day Employment ................ 35

Figure 7 Referral of Participants to Providers Based on Employability Score .............. 38

Figure 8 Random Assignment Procedure ........................................ 43
v

Executive Summary 

Purpose of the Work First Profiling Pilot Project

This report describes the development, implementation, and evaluation of a new assessment and

referral system designed to assist staff of local welfare-to-work programs in targeting employment

services more effectively to welfare recipients. The motivation for the development of this system is the

potential benefit to program participants of addressing their specific needs rather than providing all

customers with the same set of services, which has been the approach of most welfare-to-work

programs. The assessment and referral system includes administrative tools that provide staff with a

quick and efficient means to assess the needs of participants as they enroll in welfare-to-work programs

and then to use the assessment to refer participants to service providers that are best suited to meet their

needs. The assessment tool is based on a statistical method that uses administrative data to estimate a

participant’s level of employability. The employability estimate is then used to refer participants to

appropriate service providers. The assessment and referral system is designed to be integrated into an

existing intake process, to require minimal (if any) additional operations staff, and to comply with the

procedures and practices of the various welfare-to-work programs administered by the states. 

The Kalamazoo-St. Joseph Workforce Development Board (WDB) piloted the system by

incorporating it into the welfare-to-work program, referred to as Work First, that it administers for a

two-county area in southwestern Michigan. The W.E. Upjohn Institute for Employment Research

developed the system, and the Employment and Training Administration of the U.S. Department of

Labor sponsored the project. The pilot took place from January 1998 through March 2000, during

which time more than 6,000 welfare recipients participated in the program. 

The purpose of the pilot was to develop a statistical assessment and referral system and to 

determine the efficacy of integrating it into the operations of an existing welfare-to-work program. The

Work First program administered by the Kalamazoo-St. Joseph WDB had been in operation for several
vi

years before the pilot was initiated. The major difference in the operation of the Work First program

during the pilot from its operation before was the use of the statistical assessment tool and the targeted

referral. Before the pilot, all participants were treated the same. They attended the same orientation and

were randomly assigned to one of three service providers. Random assignment was used because staff

had insufficient information at the time of enrollment to identify the barriers to employment that

participant’s faced or the type of services that would best meet their needs. During the pilot, participants

continued to attend the same orientation as before but were referred to service providers according to

an assessment of their employability based on statistical methods and a determination of the comparative

advantage of each provider in serving participants with different employment capabilities. Although

each provider offered the same basic set of services as required under Michigan’s Work First program,

they varied in their approach in providing these services, which were seen as more effective for some

participants than for others. 

The pilot explored the following issues: 1) can statistical tools provide a reliable initial assessment

of a participant’s ability to find and retain a job, 2) can this assessment be used to refer participants to

appropriate service providers, 3) are some providers actually better at meeting the needs of some

participants than others (as predicted prior to the start of the pilot), and 4) can such a system improve

outcomes? 

Need for an Efficient Assessment and Referral System

Most welfare-to-work programs provide the same initial set of services to all welfare recipients,

regardless of their needs and past work history. The skills, aptitudes, and motivations of welfare

recipients vary widely, however. Many clients possess job-ready skills and significant work experience

and thus need little assistance in obtaining a job. The harder-to-serve customers are confronted with

multiple barriers to employment and stand to benefit from more intensive services. Only after clients

have tried to find a job, but have failed to do so, do most local offices provide them with more than a
vii

minimal set of services. Yet, studies have shown that work-first type programs are more effective when

services are targeted to meet individual needs. 

In order to target employment services more effectively, local offices must first assess the needs

of clients and determine which services are most appropriate in meeting their needs. Limited program

dollars, however, often preclude hiring enough case workers to assess the employment barriers faced by

welfare recipients. Thus, there is a need for a low-cost assessment and referral system that can be

integrated into existing operating systems of welfare-to-work programs. This pilot is an attempt to fulfill

that need. 

The Concept of Statistical Assessment and Referral 

Central to the statistical assessment and referral system is an administrative tool that relates an

individual’s attributes to his or her employment outcomes. For instance, research shows that individuals

with a post-high school education and prior work experience are more likely to find and retain a job than

people who have less-than-a-high-school education and little work experience. Other factors, such as

tenure on a previous job, also contribute to the predicted probability of employment, which we refer to

as the employability score. Several factors were included in the statistical model. The relative

contributions, or weights, of these factors were estimated using the administrative records of participants

who recently participated in the local Work First program. Using these weights, an employability score

was calculated for each individual who enrolled in the program. A person with attributes similar to past

Work First participants who were successful in finding a job was assigned a high employability score,

while a person with attributes similar to past Work First participants who were not successful was given

a low score. 

Since the predicted probability of employment reflects the extent to which an individual faces

barriers to employment, the score can be used to determine the level and type of employment services

that may help the individual find employment. Those with high employability scores are expected to
viii

need relatively little assistance in finding a job, while those with low scores are expected to require

significantly more assistance. 

Operation of the Pilot

The Kalamazoo-St. Joseph WDB pilot used the statistical assessment tool to provide an initial

appraisal of the needs of welfare recipients as they enrolled in the Work First program. Employability

scores were estimated for each enrollee by using administrative data collected by the Work First agency. 

Local agency staff entered the information onto a laptop computer, which contained a database program

and the statistical assessment algorithm. While the enrollees attended an orientation session, staff used

the algorithm to compute an employability score for each participant and then used the scores to refer

them to one of three service providers. 

Prior to conducting the pilot, WDB staff determined which of the three service providers already

under contract with the WDB would most benefit participants who have certain employability scores. 

The determination was based on analysis of past administrative data and the opinions of WDB staff

regarding each service organization. Employability scores were separated into three groups. 

Participants with employability scores in roughly the lowest third of the distribution were assigned to one

provider, those in the middle group were assigned to another provider, and those at the high end of the

distribution were referred to a third subcontractor. 

Evaluation of the Pilot

 The pilot was evaluated using a random assignment design. Participants within each of the three

groups of employability scores were randomly assigned to a control group and a treatment group. 

Those within the control groups were randomly assigned to each of the three subcontractors. Those in

the treatment group were assigned to the provider that a priori was determined to be most beneficial to

participants in each of the three groups. The treatment or intervention in the pilot is different from typical

demonstrations. Instead of testing the effectiveness of receiving a specific service versus not receiving
ix

the service, this pilot demonstrated the relative effectiveness of referring participants with specific

employability scores (and thus specific needs) to various service providers. In order to examine the

effectiveness of these different combinations of referrals, participants in the control group were randomly

assigned to each of the three subcontractors, which yielded six combinations of service providers and

employability groups (more combinations are possible, but we were constrained by the requirement that

enrollees would be assigned to each of the three subcontractors). No participant was denied services. 

Each participant was referred to a provider that offered roughly the same services, albeit delivered in

slightly different ways. It was the difference in styles and the focus on certain services over others that

constituted the treatment in this pilot. Therefore, unlike typical evaluations, the purpose of this

evaluation is not to compare the outcome of the treatment group with that of the control group. Rather,

the purpose is to determine whether the different combinations of referrals yield significantly different

outcomes. If they do, then the next step is to see whether or not the combination that yielded the highest

outcome, in this case the highest job retention rate, is the same combination that was expected to yield

the highest outcome (i.e., to best meet the needs of customers in the three groups) before the pilot

began. That is, we check to see whether the combination found to yield the greatest outcome in the

control group is the same combination as was chosen a priori for the treatment group. If so, then the

evaluation suggests that targeting makes a difference and that the optimal combination can be determined

beforehand. 

The evaluation found that referring participants to service providers according to their

employability score increased the overall effectiveness of the program. Using a job retention rate of 90

consecutive days as the employment outcome, the results showed that:

• The statistical assessment tool was successful in distinguishing among participants with

respect to their likelihood of employment and retention. 
x

• The optimal referral pattern based on the statistical assessment tool yielded retention

rates that were 25 percent higher than retention rates of participants who were randomly

assigned to providers.

• The difference in retention rates between the best and worst referral combinations was

56 percent. 

Average weekly earnings of those who retained their jobs for 90 consecutive days were used to

account for the benefits of the pilot system. The net present value of the difference in earnings between

the treatment group (generated from the optimal assignment rule) and the control group was used to

estimate the net impact of the program. The net present value, assuming that the earnings differentials

persisted for 8 quarters, ranged from $471,000 to $841,000. Combining these estimates with the total

cost of designing, implementing and operating the program of $145,000 yielded a benefit-to-cost ratio

that ranged from 3.25 to 5.8. 

Extension to Other Sites and Programs

This pilot expands upon the techniques used in the Worker Profiling and Reemployment

Services (WPRS) system in which the likelihood that UI claimants will exhaust benefits are identified

through statistical means. Evaluations of WPRS based on two states show that it yields expected

benefits. The evaluation of the Work First pilot provides evidence that a statistical assessment and

referral system can be integrated into an existing system of delivering employment services and that it can

improve the effectiveness of the program. 

At the writing of this report, another site has adopted a system similar to the statistical

assessment and referral system used in the Kalamazoo-St. Joseph WDB pilot. The Broward County

(Florida) Workforce Development Board, in cooperation with the Florida Institute for Career and

Employment Training affiliated with Florida Atlantic University, piloted a similar system. They modeled
xi

their approach after the one used in the pilot reported here. They referred customers to different

services, not necessarily different providers, based on an employability score. An evaluation of this

project has not yet been performed. 

Another application using statistical means to identify needs and refer clients to services is being

developed for One-Stop Centers. The Upjohn Institute, with funding from the Employment and Training

Administration of the U.S. Department of Labor, is also developing a statistical assessment and referral

system for the many services provided within a one-stop environment. These administrative tools,

referred to as the Frontline Decision Support System (FDSS), are being developed to help front-line

staff quickly assess the needs of customers and then to refer them to services that better meet their

needs. FDSS includes new tools to 1) help customers conduct a systematic search for jobs that offer

the best employment match and to set a realistic wage goal, and 2) assist staff in determining which onestop center services are likely to be effective in meeting the needs of specific customers in becoming

employed. The FDSS tools are designed to be used within the current data retrieval and display

systems implemented by states for their One-Stop Centers. 
1

There are few rigorous evaluations of the welfare-to-work programs implemented under PWRORA. Most

evidence related to the effectiveness of programs with an emphasis on employment have been gleaned from

programs that resulted from waivers granted to states. Other evidence comes from evaluations of the Job

Opportunities and Basic Skills Training (JOBS) program. There are several papers that synthesize the results from

the evaluations of these studies. One such study, entitled National Evaluation of Welfare-to-Work Strategies, was

prepared for the U.S. Department of Health and Human Services by the Manpower Demonstration Research

Corporation. This report concludes that the programs they studied did not improve upon the already high rate of job

finding of welfare recipients who did not participate in the programs, but “nearly all programs helped single parents

work during more quarters of the follow-up and earn more than they would have in the absence of a program” (ES-1). 

2

The study, National Evaluation of Welfare-to-Work Strategies, prepared by the Manpower

Demonstration Research Corporation, reports that evidence from evaluations suggest that a “‘mixed’ approach–one

that blends employment search and education or training–might be the most effective” (ES-3).

1

I. Introduction

A. Purpose of the Work First Profiling Pilot

The welfare reform movement of the 1990s, marked by the passage of the Personal

Responsibility and Work Opportunity Reconciliation Act (PWRORA) in 1996, has focused on placing

welfare recipients into jobs as quickly as possible. Underlying this reform is the premise that experience

on the job is as valuable, if not more so, as classroom training or other forms of formal job preparation

and skill development. In addition, actual work experience provides self-esteem, self-discipline, and job

know-how skills, and it starts welfare recipients on the path of future career advancement and economic

self-sufficiency.1 

To make this approach successful, local welfare-to-work agencies and their providers are faced

with the challenge of providing the appropriate level and mix of employment services that will quickly

move welfare recipients into jobs and will also equip them with sufficient skills to retain those jobs.2

Most welfare-to-work programs offer a minimal level of instruction in job-readiness skills and in job

search techniques. Furthermore, most programs provide the same initial set of services to all welfare-towork participants regardless of their needs and past history, even though their skills, aptitudes, work

experience, and motivations vary widely. Yet, studies have shown that welfare-to-work programs are
3

For example, Gueron and Pauly (1991), from their evaluations of welfare-to-work demonstrations,

suggested that increased service intensity improves employment rates of clients and that spreading resources too

thinly reduces program effectiveness. In addition, evaluations of programs such as California GAINS (1996) and the

JOBS (1997) have suggested the importance of assessment in placing welfare recipients into jobs.

2

more effective when services are targeted to meet the specific needs of individual participants.3 While

welfare caseloads have decreased significantly since the implementation of welfare reform under

PWRORA, welfare-to-work programs have come under increased pressure to find jobs for the harder

to serve and to find ways to assist former welfare recipients with retaining their jobs. Both challenges

prove to be increasingly difficult in a less robust economy. 

Therefore, targeting employment services to meet the specific needs of participants offers a

promising avenue for improving the effectiveness of welfare-to-work programs and thus for achieving the

overall goal of welfare reform of placing welfare recipients into jobs as quickly as possible. Targeting

services requires first an assessment of the needs of clients and second an evaluation of which services

are most appropriate in meeting their specific needs. Traditional means of assessment and referral

require extensive use of staff to perform these functions. Limited program dollars under PWRORA for

that purpose, however, often precludes hiring enough case workers to assess the needs of welfare

recipients at the time of enrollment. Statistical assessment and referral methods potentially use fewer

resources, and their use in programs such as the Worker Profiling and Reemployment Services (WPRS)

system demonstrates their ability to target resources to clients who need additional assistance. 

The purpose of this pilot project is to explore more cost-effective ways of targeting services to

participants of welfare-to-work programs. In 1997, the Employment and Training Administration of the

U.S. Department of Labor contracted with the W.E. Upjohn Institute for Employment Research to

develop and implement an assessment and referral system based on statistical methods. The

Kalamazoo-St. Joseph Workforce Development Board, which is responsible for administering

Michigan’s welfare-to-work program (referred to as Work First) for a two-county area in southwest

Michigan, piloted the project. The pilot operated from January 1998 through March 2000, during which
4

The W.E. Upjohn Institute for Employment Research is an independent, not-for-profit organization that

conducts research on a variety of employment issues. In addition, the Institute is the administrative entity for the

Kalamazoo-St. Joseph Workforce Development Board. Having both research and operations within the same

organization offered a unique opportunity to develop and conduct such a pilot. 

5At the start of the project, we used the term “profiling” to refer to the statistical assessment tool. As the

project progressed and the tool was further developed, we found that the term “statistical assessment and referral

system” better described the process. We use this latter term throughout the report. The title of this report still

includes the initial terminology in order to be consistent with the original project title.

3

time more than 6,000 welfare recipients participated in the program.4 Central to this system is an

administrative tool that uses statistical methods to quickly assess the needs of participants during initial

enrollment and then to use the assessment to refer participants to service providers that are best suited to

meet their needs. The assessment and referral system is designed to be integrated into existing intake

processes, to use existing administrative data, to require minimal, if any, additional staff, and to comply

with the procedures and practices of existing welfare-to-work programs.

The Kalamazoo-St. Joseph pilot addressed the following issues: 1) can statistical tools provide a

reliable initial assessment of a participant’s ability to find and retain a job, 2) can this assessment be used

to refer participants to appropriate service providers, 3) are some providers actually better at meeting

the needs of some participants than others (as predicted prior to the start of the pilot), and 4) can such a

system improve outcomes without significantly increasing daily operating costs? 

This report describes the development, implementation, and evaluation of the Kalamazoo-St.

Joseph WDB profiling pilot project and is intended to reach several groups of readers.5 First, it is

intended to inform administrators at the local, state, and federal levels about the basic concept of this

assessment and referral system and to demonstrate the benefits of adopting such a system. Second, the

report provides a detailed description of the statistical underpinnings of the system, so that researchers

can scrutinize the methodology and so that local and state agencies may have sufficient information if they

are interested in adopting this system for their own use. Third, the report presents a rigorous evaluation

of the system, based on a random assignment design, in order to demonstrate the merit of the system and

in order that future users may understand how the benefits were derived. 
4

B. Welfare Reform and Michigan’s Work First Program

The welfare reform movement of the 1990s transformed the existing 60-year-old system based

on cash assistance entitlements to the needy into one predicated on moving the needy into jobs. With

the passage of the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) in

1996, Congress formally replaced the Aid to Families with Dependent Children (AFDC) entitlement

program with the Temporary Assistance for Needy Families (TANF) program. TANF ended needy

families’ entitlement to cash welfare payments and gave states broad flexibility in how they could use the

federal funds allocated to them. Congress also barred states from using TANF funds to assist families

for more than 60 months over their life times and required states to put in placed incentives and sanctions

in order to ensure that a larger portion of welfare recipients were working or engaged in work-related

activities. 

Most of the reform initiatives contained in PRWORA originated with the states, not with the

federal government. Between 1992 and 1996, more than 40 states requested and were granted waivers

from federal requirements under AFDC. Many of these states used their waivers to impose tougher

work requirements on adult welfare recipients, including mothers with very young children, and to set

limits on the receipt of cash welfare recipients. Some states embarked on programs that were designed

to improve the financial and material conditions of needy families through encouraging work and by

making employment more financially attractive. These states extended their earned income disregards

which allowed welfare recipients to keep some of the cash assistance (and other assistance such as

medical care and food stamps) if they worked. Some states also created or expanded their earned

income tax credit programs, following the federal government’s lead in expanding the Earned Income

Credit (Bloom and Michalopoulos, 2001). 

Michigan was one of the first states to use waivers to reform their state welfare programs and is

often regarded as one of the leaders in establishing a work-based welfare system. Beginning in 1992,

they requested and received waivers in order to encourage work through a system of rewards and
6

For a more detailed description of the nature and history of Michigan’s welfare reform see Seefeldt, et. al.

(2001).

5

stricter sanctions. By 1994, the state had established its Work First program. As the name suggests,

Michigan’s Work First program stresses the placement of welfare recipients into jobs as quickly as

possible with a major emphasis on job search but with limited opportunities for education and training. 

The program offers instruction in the proper techniques for writing resumes, completing applications, and

interviewing for jobs. It also provides assistance in searching for a job. All enrollees receive similar

services regardless of their needs and must find qualified employment within four weeks of enrollment. 

More intensive skill training is available only to those who hold a job or to those who have repeatedly

failed to find employment.6

At the time of the pilot, Michigan’s Work First program was administered locally by the 23

Service Delivery Areas (SDAs), which were originally created by the Job Training Partnership Act to

administer and deliver job training programs to dislocated adults and economically disadvantaged

individuals. With the enactment of the Workforce Investment Act (WIA) in 1998, the SDAs were

reorganized as Workforce Development Boards. The Kalamazoo/St. Joseph SDA changed its name to

the Kalamazoo/St. Joseph Workforce Development Board but maintained responsibility for the same

geographical area as it had before the transition. Throughout this report, the local Kalamazoo/St. Joseph

office will be referred to as a Workforce Development Board, even though it was still designated as an

SDA during the first several months of the pilot. 

Michigan’s local Workforce Development Boards (WDBs) work closely with the local offices of

the state’s social service agency, the Family Independence Agency (FIA), to administer the Work First

program. FIA determines welfare eligibility, issues welfare payments, and refers welfare recipients to

Work First programs, while the Work First agency provides welfare recipients with employment

services, through intermediaries. At the time of the pilot, FIA referred all applicants for public assistance

to Work First, with the following exceptions: 1) persons less than 16 or older than 65 years of age; 2)
7

The initial assessment that was performed at orientation before the pilot began was minimal, and staff at

the Kalamazoo-St. Joseph WDB did not consider this assessment adequate to be used as a basis of referring

customers to service providers. Therefore, prior to the pilot customers were randomly assigned to providers. 

6

the mother of a child under the age of three months; 3) one parent or other caregiver of a child with a

disability; 4) a person who is the full-time caregiver for his or her spouse suffering from a disability; 5) a

child 16-17 years of age who is a full-time student in elementary school or high school; 6) a minor parent

attending full-time elementary school or high school; 7) an individual employed or self-employed 20

hours or more a week at the minimum federal wage; and 8) a person suffering from a long-term physical

or mental disability.

FIA refers all qualified applicants to Work First within 10 days of their applying for cash

assistance. Applicants are notified of the date and time they are to enroll in the program and attend

orientation. Orientation includes an introduction to the Work First program, specification of the roles

and responsibilities of the program and client, and a brief assessment of the client’s situation and

immediate needs, including the possible need for supportive services.7 In-depth assessment and

counseling are offered only to those in considerable need. In most cases, all those referred by FIA to

Work First are required to participate in the same job search and job readiness workshops regardless of

their past work histories or qualifications. Job search/job club workshops provide training in

appropriate skills in seeking, locating, applying for, and obtaining employment. Job search training is

typically conducted in group settings. Each person is expected to develop a résumé and to understand

the proper techniques for completing applications and interviewing for jobs. 

After clients complete the core services, they are required to search intensively for work and to

accept offers that provide at least 20 hours of work per week at or above the minimum wage. More

extensive assessment and skill training are available through the local WIA program, but only for those

who have extreme difficulty finding a job. Participants are expected to obtain a job within 90 days or

risk a reduction in benefits. For example, if a single parent does not participate 20 hours per week in a

Work First activity or employment, then sanctions are imposed by reducing welfare benefits and food
8

The two-parent family program was not included in this pilot.

9

The requirements described here are those in effect during the time the pilot was in operation. Some

requirements have changed since that time. 

7

stamps. Two-parent families are subject to similar requirements and sanctions.8 As an incentive for

finding work, participants are allowed to keep the first $200 earned each month and 20 percent over

that amount without reducing benefits. Participants also receive transportation, child care, and Medicaid

for a limited time. Allowable work activities include 1) unsubsidized employment; 2) subsidized private

sector employment; 3) subsidized public sector employment; 4) on-the-job training; 5) job search and

job readiness training and activities up to six weeks; 6) community service programs, and 7) no more

than 12 months of vocational educational training. Customers employed for 90 consecutive days in a

qualified job are considered to have achieved a successful outcome, at which time they are terminated

from the program.9

In Michigan, intermediaries, not the local WDBs, provide employment services to Work First

participants. During the time the pilot was conducted, the Kalamazoo/St. Joseph WDB subcontracted

with three local service organizations to provide employment services to participants included in the pilot. 

The formal agreements required that the organizations provide a set of basic services. Nevertheless, the

number of hours in which customers participated in these activities varied, and in some cases, service

providers offered additional services beyond those prescribed by the contract or they provided

assistance using different approaches. Therefore, the level and intensity of services varied across service

providers, although they all were in compliance with their contracts and state requirements. 

C. Targeting Services to Welfare-to-Work Participants

Michigan’s Work First program as well as many other welfare-to-work programs provide basic

instruction in job search techniques and minimal assistance in contacting employers. All Work First

participants, regardless of their qualifications and work experience, are required to participate in these

services. Research, however, has shown that the benefits from these basic services vary across the
10See Bloom and Michalopoulos (2001) for a synthesis of research on the effectiveness of welfare-to-work

programs.

8

welfare population and that this variation depends to a large extent on an individual’s characteristics, past

work experience, and welfare dependence.10 Therefore, targeting services to the specific needs of

participants instead of pursuing a one-size-fits-all approach opens the possibility of improving the

effectiveness of welfare-to-work programs and of helping states make more efficient use of their

resources. 

The idea of targeting services to would-be welfare recipients who can benefit most from the

assistance predates the current welfare reform movement. Ellwood (1986) explored the possibility of

using statistical means to identify individuals who are most likely to be long-term welfare recipients. He

estimated recidivism rates and exit rates using the characteristics of individuals and their previous

employment and welfare histories as predictors. Based upon his ability to identify those at risk of

becoming long-term welfare recipients, he concluded that the effectiveness of welfare programs (in

particular AFDC) could be enhanced by targeting services to welfare recipients with specific

characteristics. Support for targeting is further found in the evaluations of previous programs and

demonstrations that targeted the welfare population. Gueron and Pauly (1991) reviewed the evaluations

of a host of programs, both broad-coverage and small and selective voluntary programs, in order to

discern whether the effectiveness of the service components within these programs vary among

participants. They concluded that the impacts do vary among participants and that they are larger for

more disadvantaged recipients. For example, they cited an analysis of the Supported Work program

that shows that services were more effective for women who had never worked and had been on

welfare longer. The same pattern emerged from a reanalysis of Supported Work and the quasiexperimental studies of WIN and CETA on AFDC recipients in which welfare recipients with little or no

recent work experience benefitted substantially more than did those with some recent work experience

(Grossman, Maynard, and Roberts, 1985). Friedlander (1988), in an analysis of five selected welfare

employment programs, identified additional characteristics that affected program effectiveness, including
9

marital status, education, and the number and ages of children. While still important, they are less

strongly related to future employment and welfare receipt than past employment and welfare experience. 

Once it is established that the effectiveness of programs varies by participants, the next issue

regarding targeting services is the method by which various subgroups are identified. Identification

methods vary widely, from using the subjective evaluations of staff to a much more objective assessment

based on statistical techniques. The effectiveness of these approaches vary as well. Gueron and Pauly

(1991) cited two studies that used the perceptions and knowledge of staff about their clients to refer

them to services. The first case was a study of AFDC recipients in Louisville who were randomly

assigned to participate in job club activities. Before they entered the job club, staff counselors rated

them on job-readiness based upon their perception of the client’s motivation and skills. The study

analyzed the relationship between these initial ratings and participants’ performance in the program, such

as attending job club or dropping out, and finding a job during job club. The study found that there was

no relationship between the job-readiness ratings and those measures of performance. In a second

study cited by Gueron and Pauly, intake workers in the Homemaker-Home Health Aide Demonstrations

rated the job readiness of clients by supplementing their own perceptions with quantitative information,

such as a client’s education, work experience, and other personal attributes. Even though this additional

information was considered, the study found that the staff based their ratings primarily on perceptions,

with only a weak relationship between a client’s intake information and the ratings. The study further

reported that although the ratings were correlated with post-program performance outcomes they did

not help to distinguish the success of program participants from those in the control group. 

More recent identification methods have relied exclusively on objective ratings. In particular, the

Worker Profiling and Reemployment Services (WPRS) system uses statistical methods to relate

individual attributes and local market conditions to the likelihood that a person will exhaust his or her

benefits. In 1993, Congress mandated that all states implement a profiling system within their

Unemployment Insurance system. The purpose of WPRS is to encourage UI claimants to receive
10

reemployment services and search actively for jobs sooner than later during their unemployment spell. 

The program was established in response to the widespread observation that many UI claimants were

waiting until they had nearly exhausted their 26 weeks of benefits before actively searching for

employment and taking advantage of the reemployment services available to them through the

employment service. Each state has successfully integrated this statistical tool into the daily operations of

their UI programs. An evaluation of the program, based on the experience of two states, New Jersey

and Kentucky, found positive outcomes. Those who were profiled and referred to services, compared

with those who were not, spent less time on unemployment insurance, had lower rates of benefit

exhaustion, enjoyed increased earnings, and increased the amount of reemployment services they

received (U.S. Department of Labor, 1997). 

D. The Work First Pilot Project’s Statistical Assessment and Referral System

The assessment and referral system developed for the Kalamazoo/St. Joseph Work First pilot

uses a statistical method similar to that used by WPRS and proposed by Ellwood (1986). The

assessment tool identifies welfare recipients who may have difficulty finding and retaining a job. Those

with a low probability of employment presumably face a significant number of barriers to work while

those with a high probability have fewer barriers. The probability is derived from a statistical model that

uses information commonly collected at enrollment interviews. It estimates the relationship between an

individual’s propensity to find and hold a job and that person’s personal attributes, work and welfare

histories, and the local labor market conditions. The model is based on information about the experience

of welfare recipients who previously entered the local Work First program.

The same critical elements that prompted the creation of WPRS and contributed to its successful

implementation are present in welfare-to-work programs. First, participants in welfare-to-work

programs vary widely in their dependency on welfare and in their response to reemployment services. 

Second, not enough funds are available to provide sufficient levels of reemployment services to all

welfare recipients who could benefit from the programs. Third, the methodologies are available to
11

identify those individuals most likely to benefit from reemployment services. Fourth, the goals of the

welfare reform program can be better met by targeting services to the specific needs of individuals, and

thus incentives are present for state and local service providers to pursue a more targeted delivery of

services.

Therefore, the statistical assessment and referral system developed for the pilot builds upon the

methodology of previous programs, particularly WPRS. Unlike WPRS, however, the Kalamazoo/St.

Joseph Work First pilot’s assessment and referral system is used to determine the set of services that is

most appropriate for the specific needs of groups of Work First participants. In contrast, profiling in the

WPRS is used to determine who will receive services or not. All Work First participants in the pilot

receive services, but the services differ by intensity and mix. In this respect, the task of the statistical

assessment and referral system in the Work First pilot is more complex than that of WPRS. Not only

must it identify the employability of participants, but it also must determine the set of services most

appropriate for the customer. 

The Kalamazoo-St. Joseph pilot incorporated the statistical assessment and referral system into

the initial intake and orientation process. Each welfare recipient who enrolled in Work First was

immediately assigned a score indicating his or her probability of finding employment. The score

provided an assessment of each participant’s need for services, based upon the past experience of local

Work First participants like themselves. A high score indicated that a person had little need of services,

since past participants with the same set of characteristics had a high probability of finding a job. Those

with a low score required more services, since past recipients with similar attributes had less success in

finding and retaining employment. Each participant was then referred to one of three subcontractors

based on their employability scores. 

The service providers under contract during the pilot differed in their approach to delivering

services, while still complying with the requirements of the state’s Work First program. Local office staff
12

determined before the pilot began which service provider was most appropriate in meeting the needs of

participants whose employability scores fell within specific ranges. The participants were not informed

of their scores; neither were the providers. Prior to the pilot, participants were assigned to the three

service providers on a random basis, since staff had no meaningful way to assign customers. 

The pilot’s assessment and referral system provided an efficient way to target services to meet a

customer’s needs without requiring substantial increases in staff to make assessments and referrals. The

only additional staff required to operate the system on a daily basis was a part-time employee who

operated the computerized intake system. Consequently, local office staff was better able to meet the

needs of its customers without incurring significant additional operating costs. 
13

II. Statistical Assessment and Referral Model

A. Overview

The purpose of the statistical assessment model is to use information commonly collected during

the enrollment process to identify Work First participants according to their need for assistance in finding

and maintaining employment. The model generates employability scores for each participant when they

enroll in the program. It uses a statistical technique, referred to as logit analysis, that relates an

individual’s personal characteristics, welfare and work histories and local labor market conditions to

whether or not they retained a job for 90 consecutive days, which is the successful outcome of the

program. The data used to estimate the model are obtained from administrative data recorded by the

local Work First program. 

B. Designing the Optimal Allocation Mechanism

The statistical assessment and referral system designed for this pilot is an allocation mechanism

that assigns participants to providers based on one or more identifiable and measurable characteristics of

the participant. The goal of this assignment mechanism is to improve the outcomes of the local Work

First system. For this pilot, that means to increase the overall job retention rate of the program. This

section describes the basic principles of an allocation mechanism and illustrates how it can be used to

assign participants to services (or providers) and how it can increase the overall effectiveness of the

program. Three papers are particularly helpful in formalizing the general framework for the allocation

mechanism. Manski (1999, 2001) has developed a formal framework for understanding and assessing

various allocation mechanisms, and Berger, Black, and Smith (2000) have applied this general

framework to evaluate profiling under WPRS as a means of allocating government programs. Although

these papers appeared after the allocation mechanism and the evaluation design for this pilot were

developed, they are still helpful in understanding the principles underlying the allocation mechanism. In

particular, Berger, Black, and Smith’s notation and presentation is used here to frame the allocation

problem.
14

The premise for developing an allocation mechanism based on individual characteristics is that

responses to services (or treatments) may vary across program participants. The task in designing an

allocation mechanism is to be able to identify characteristics that are associated with the variation in

outcomes of each participant or group of participants across the various services. For example, if a

participant with less than a high school education responds better (in terms of retaining a job) to services

offered by provider A than by provider B, participants with less than a high school education should be

assigned to provider A and vice versa. (Here we use providers instead of services to represent

different types of treatments, since in the pilot the choice is between providers and not specific services.) 

More formally, the deterministic allocation mechanism is:

M(X): X 6T,

where X is the set of identifying characteristics, T is the set of providers, and M symbolizes the allocation

mechanism that assigns a person with a specific characteristic (Xi) to a specific provider (T=A, B, ...).

Suppose for now that the only characteristic included in the set of characteristics, X, is education

(denoted by X1); that is, education is the only characteristic that is associated with the variation in

responses. Furthermore, suppose that a participant’s education level is measured as either attaining a

high school education or higher (X1=1) or not (X1=0). Using one characteristic to assign individuals to

services is often referred to as a characteristics screen.

With two classifications of a single characteristic and two providers, there are two possible

assignment rules if participants are assigned to each provider and no one provider receives all

participants regardless of educational attainment. The two possible assignment rules are: 

1) those with X1=0 are assigned to provider A and those with X1=1 are assigned to provider B, and 2)

those with X1=1 are assigned to provider A and those with X1=0 are assigned to provider B. Since the

allocation mechanism is based on only one characteristic, and it is assumed that participants with one

dimension of that characteristic do better with one provider than with the other, it makes no sense to split
15

participants with the same dimension of the characteristic among the two providers. This is what would

happen if the participants were randomly assigned to the two providers. 

To determine which assignment rule is optimal, the outcomes of the individuals are evaluated

based on the goal of the program, which in this case is to maximize the job retention rate of the

participants. Suppose that through observing the outcomes of participants who were randomly assigned

to the two providers, which presumably occurred before the allocation mechanism was adopted,

outcomes were recorded as shown in Table 1. Participants with less than a high school education

(X1=0) who received services from provider A exhibited a retention rate of 30 percent. That is, for

every 100 people who received services from provider A, 30 retained their jobs. Those with less than a

high school education who received services from provider B exhibited a retention rate of 20 percent. 

Since individuals with the same level of education were randomly assigned to the two providers, one can

assume that the variation in outcomes was the result of differences in the services offered by the two

providers and not the result of differences in the ability of participants to retain jobs. For those with a

high school education or more (X1=1), suppose that the retention rate was 40 percent if they received

services from provider A and 60 percent if they received services from provider B. 

Table 1: Job Retention Rates by Provider and Participant’s Education 

Providers

A B

Education X=0 30% 20%

X=1 40% 60%

The outcomes of the two assignment rules can be determined from the information in the table

and the assumption that each provider has the capacity for 100 participants. The first assignment rule

assigns X1=0 to A and X1=1 to B with an outcome of S1; the second assignment rule assigns X=0 to A

and X=1 to B with an outcome of S2. Table 2 exhibits the outcomes from the two assignment rules.
16

Table 2: Outcomes of Two Hypothetical Assignment Rules

Assignment

Rules

Education Provider Total

Participants

Number who

Retained Jobs

System Total

Number Rate

1

X=0 A 100 30 90 45%

X=1 B 100 60

2

X=0 B 100 20 60 30%

X=1 A 100 40

From Table 2, it can be seen that assignment rule 1 is optimal. It yields a system-wide job

retention rate of 45 percent compared with a retention rate of 30 percent for assignment rule 2. Clearly,

the difference in the outcomes of the two assignment rules shows that providers have a comparative

advantage in serving one group over another. 

The allocation mechanism based on one characteristic is relatively simple and straightforward. It

is rarely the case, however, that responses to services vary with only one characteristic. For example, in

addition to education, it is conceivable that retention rates may vary across services by past work

history, marital status, age, and welfare dependency. Multiple characteristics complicate the allocation

mechanism. One method in which more than one characteristic can be incorporated into an allocation

mechanism is to use the characteristics to predict an outcome that is either the goal of the system or one

that is closely aligned with that goal. The profiling system used in the WPRS is an example of such an

allocation mechanism in which the assignment is based on a predicted outcome. Under WPRS the

outcome is the probability of exhausting benefits, which is estimated using a set of personal

characteristics (as well as local economic conditions). Following the presentation by Berger, Black and

Smith (2000), denote the predicted value of the outcome for the individual participant by Q’(X), where

X is the set of characteristics used to predict Q. The allocation mechanism using the predicted value is

similar to the one that used only one characteristic:

S(Q’(X)):Q’(X)6T.
17

Once the predictive model is estimated, the allocation mechanism based on the predictive value

of the outcome becomes a deterministic function of X. The distinction between the two mechanisms

centers on the relationship between the Xs and the predicted value Q’. Instead of using a single

characteristic, such as education, to assign a participant to a provider, a set of characteristics is used. 

The relative importance of each characteristic in making the assignment is determined by the estimated

coefficients of the model used to predict the individual outcome Q.

An assessment of the optimal assignment rule under the statistical assessment allocation

mechanism is similar to the process previously used to illustrate the deterministic allocation mechanism. 

Since the statistical allocation mechanism is similar to the one used in the pilot, we will use the actual data

generated from the pilot to describe this mechanism. This will be provided in a subsequent section. The

next issue is the estimation of the predictive model. 

C. Data Requirements and Availability

The predictive model (that is, Q’(X)) for this pilot is the statistical assessment model that uses

personal characteristics to explain the likelihood that a person will retain a job for 90 consecutive days. 

We refer to the statistical assessment tool as an employability score, and the two terms are

interchangeable throughout the remainder of the report. Data were obtained from the intake forms that

were maintained by the Kalamazoo-St. Joseph WDB in order to administer its Work First program. 

The intake process took place after the Family Independence Agency (FIA) referred welfare recipients

to Work First and before those enrollees reported to the subcontractors for services. FIA collected

additional information about the client, such as health problems that may limit work or training activities,

current drug treatment, or prior convictions. This information, however, was not necessarily shared with

the WDB and thus was not available for use in developing the statistical assessment tool.

For each participant, we constructed a file that chronicled their activities before, during, and after

their most recent enrollment in the Work First program. Activities prior to their most recent enrollment
18

included employment status and the number of months on welfare. For those who had participated in

Work First more than once, we also included their reason for leaving the program at that time. Work

First activities during their most recent enrollment included the type of activity, the number of hours

engaged in each activity, and the starting and ending dates of each activity. We also included their

reason for leaving the program, which included among other possibilities success in finding a job. 

Finally, whether or not they retained a job for 90 consecutive days after they left the program was

included in their record. 

The purpose of the employability score is to use information that is available at the time of

enrollment to predict a participant’s likelihood of finding a job. Therefore, not all of the information

pieced together for each participant can be used to predict future employment. At the time of

enrollment, we know a person’s characteristics, such as age and education, her prior work history, past

enrollment in a Work First program, length of time on welfare. We do not know, however, the activities

in which they are about to participate. These activities occur after they enroll and thus after we predict

their employability scores. The following information is used in the statistical assessment model to

predict future employment:

• Age

• Parental status

• Educational attainment

• AFDC history

• Target group (long-term welfare recipient, older children, little or no work experience

or education)

• Subcontractor

• Employment prior to first assignment

• Compliance history in previous Work First enrollments.
19

As previously mentioned, during the operation of the pilot, the Work First program defined a

successful outcome as a participant working in a qualified job for 90 consecutive days (with a grace

period of no longer than a week if they changed jobs). A qualified job must offer a single parent at least

a minimum wage and 20 hours a week of work. When this outcome was achieved, the person

terminated from the program. Other reasons for termination included exemption for reasons such as

health or medical problems (occurring or revealed after intake), family care responsibilities, no child

care, noncompliance, and inappropriate referral. For those who participated in the program, a detailed

log was kept of their activities and dates. These activities included job readiness training, job

development and/or job placement services, assessment and employability planning, longer-term training,

and unsubsidized employment. Wages and hours worked per week were recorded for each

employment spell included in the files. The records also included information about each participant’s

unsubsidized employment immediately prior to their first enrollment in a Work First program. Those

participants in this category were working in an unsubsidized job at the time they were referred to Work

First, or obtained unsubsidized employment prior to reporting to the first activity offered by the service

provider. Individuals who entered the program more than once had two or more employment spells

included in the files. Hourly wages and hours worked were also recorded for each employment spell.

D. Characteristics of Work First Participants

The data used to estimate the statistical assessment model included Work First participants who

entered the program during 1996. The next several sections describe the characteristics of these

participants, the activities in which they participated, their employment outcomes, and differences in

services offered by the three subcontractors during this time period. 

As shown in Table 3, the typical participant during this time period was slightly under the age of

30, had completed 12th grade, and had received welfare payments for at least 36 of the last 60 months. 

Some of the participants had completed a GED, but few had vocational training. About a fifth of the

participants had unsubsidized employment prior to enrolling in Work First. For example, while 39 
20

Table 3. Variables Used in the Work First Statistical Assessment Model

Name Description Mean

single parent =1 if single parent 0.827

age age at time of enrollment 29.7

no schooling no formal schooling 0.038

less than 9th grade education

9

th grade education

10th grade education

11th grade education

12th grade education

grade level completed less than 9th grade

completed 9th grade

completed 10th grade

completed 11th grade

completed 12th grade (omitted from analysis, thus reference)

0.056

0.056

0.089

0.191

0.387

1 year post-secondary education

2 years post-secondary education

3 years post-secondary education

GED

completed one year of post-secondary

completed two years of post-secondary

completed three years of post-secondary

earned graduate equivalent certification

0.012

0.016

0.004

0.161

YOU

Goodwill

Behavioral Foundation

referred to Youth Opportunities Unlimited 

referred to Goodwill Industries

referred to Behavioral Foundation

0.189

0.179

0.303

vocational education attended postsecondary vocational education program 0.014

not a target group not a target group, which includes AFDC received any 36 of

preceding 60 months, youngest child 16-18, or custodial

parent under 24 and who has not completed high school or

with little or no work experience

0.528

AFDC36 received AFDC any 36 of preceding 60 months 0.343

employed prior to enrollment

employed prior to previous 

enrollment

qualified unsubsidized employment prior to assignment

qualified unsubsidized employment prior to assignment in

previous enrollment 

0.190

0.003

noncompliance

employed

terminated as noncompliant in previous enrollment 

terminated as employed in qualified unsubsidized job

0.057

0.427

No. of observations 1546

percent completed high school, slightly over 4 percent went on to receive any education after high

school, including those who attended post secondary vocational education programs. For those not

completing high school, nearly 15 percent did not go beyond the 9th grade, and another 30 percent

dropped out before completing their senior year. Yet, 16 percent did earn a graduate equivalent

certification (GED), which when combined with those completing 12th grade puts the percentage of high
11The retention rates shown here are different from those shown in reports submitted to the states because

of a different denominator. We used the number of participants who entered the program as the denominator and

then followed that cohort until their determination. The reports use the number of participants placed in jobs that

expected to lead to long-term employment as the denominator. As a result, the reports to the state show a much

higher retention rate than shown here. 

21

school graduates or equivalents at 55 percent. The Work First program targeted participants with

characteristics that program staff believed created significant barriers to employment. Labeled “not a

target group,” in Table 3, this group included participants with a lengthy history of receiving welfare, little

work experience, and child-rearing responsibilities. Slightly fewer than half of the participants were

included in this category(the variable is entered in the table as not included in the group, so the

percentage in the group is 1-0.528). An individual’s AFDC history during the past 5 years is included

separately from the targeted group. About a third of the participants received AFDC payments for three

or more years during that five-year period. Only 19 percent held a job immediately prior to enrolling in

the Work First program. 

E. Reasons for Leaving Work First

Table 4 lists various reasons for leaving the program. Twenty-six percent found employment

for 90 consecutive days.11 Roughly 6 percent were terminated because of personal issues such as health

problems or family responsibilities. Another 5 percent of the participants had their case closed by the

FIA because they earned too much money to be eligible after working or they did not fill out the

appropriate paperwork on time. Twelve percent were found to be ineligible, or were referred

inappropriately to Work First. Thirty-two percent terminated the program either as a no-show, a

noncompliant, or one who attended orientation only. Some of the participants whose reason for leaving

was recorded as other (code 55) may have been no-shows or minimal participants, but because those

individuals did not participate in Work First or left without an exit interview, no specific termination type

is recorded in their files. For these four groups, we do not know the activities, if any, in which they

participated, and we do not know whether they had unsubsidized employment prior to the first

assignment. The latter information is critical, since prior employment is considered important to work

force attachment and future labor market success. The importance of prior employment to the
22

predictive power of the model is an empirical issue, and we wish to estimate the assessment model both

ways. The need to have prior employment history, which is included in a participant’s activities file,

dictates that we use only those individuals with activity information to estimate the assessment model.

Table 4. Termination Types

Code Termination type Frequency Percent Cumulative

 40 Employed 90 days 1030 26.18 26.18

 50 Institutionalized 9 0.23 26.40

 51 Health/medical 202 5.13 31.54

 52 Family care 43 1.09 32.63

 53 Lacks transport 21 0.53 33.16

 54 Cannot locate 84 2.13 35.30

 55 Other 517 13.14 48.44

 57 No child care 12 0.30 48.74

59 Attended orientation only 287 7.29 56.04

 60 No-show 430 10.93 66.96

 61 Noncompliance 543 13.80 80.76

 64 Out of county 57 1.45 82.21

 65 Case closure 188 4.78 86.99

 66 Inappropriate referral 218 5.54 92.53

 67 Ineligible 277 7.04 99.57

 70 Other parent excused 17 0.43 100.00

 Total 3,935 100.00

Noncompliance (termination code 61) is different from codes 59 and 60 in that some

individuals who terminated as such did have activities while enrolled. The reason is that a person can be

considered out of compliance for three reasons: 1) disruptive behavior; 2) the client threatened or

physically abused FIA/Michigan Works Agency staff; and 3) the client quit or was dismissed from a job. 

Those separating from a job held while enrolled in Work First will have other activities recorded (such

as employment in an unsubsidized job). Unfortunately, we do not know which of the three reasons

actually pertained to a person being terminated as noncompliant. However, we do know that 63

percent of those terminated as noncompliant were recorded as having an unsubsidized job while enrolled

(code 01), which suggests that the same percentage was recorded as noncompliant because they quit or

were dismissed from that job.
23

F. Work First Activities

Work First participants engaged in a variety of activities as part of their requirement for

successfully participating in Work First and, consequently, for receiving cash assistance. Most

participants began with assessment and employability planning (code 12). As shown in Table 5, 83

percent of all participants received this service in 1996. The percentage was higher, about 90 percent,

for those who were not employed prior to entering Work First. About half the participants engaged in

group or individual job-search assistance, which includes counseling, job-seeking skills training, and may

include support on a one-to-one basis (code 13). These activities were designed to help participants

become familiar with general workplace expectations and learn about behavior and attitudes necessary

to compete successfully in the labor market (Glossary of Terms and Definitions, Work First

Management Information Guide, Issued 2/97). Fifty-three percent were employed in a qualified job

(code 1) that paid minimum wage or more and offered at least 20 hours of work per week (or 35 hours

if a working spouse). Another 6 percent were employed in unsubsidized employment that offered

minimum wage but less than 20 hours per week. Nineteen percent of the participants were in

unsubsidized employment when referred, obtained subsidized employment meeting the requirements of

code 01 prior to reporting, or obtained the appropriate employment prior to reporting to the first activity. 

Only a few participants (2 percent) were referred to community service programs or vocational

educational training.

Table 5. Selected Activities of Work First Programs 

Activity Code Mean

Standard

deviation Minimum Maximum

Unsubsidized employment 01 0.53 0.50 0 1

Job readiness 10 0.09 0.28 0 1

Assessment and employability planning 12 0.83 0.37 0 1

Job search 13 0.55 0.50 0 1

Part-time employment 19 0.06 0.24 0 1

Employment prior to assignment 20 0.19 0.39 0 1

Community service 33 0.01 0.11 0 1

Vocational education training 34 0.01 0.09 0 1
24

G. Differences in Activities among Providers

The Kalamazoo/St. Joseph WDB contracted with three organizations to provide employment

services to participants of the Work First program. The providers delivered services that met state and

federal requirements regarding content and duration. Nevertheless, there was some flexibility within the

requirements. WDB staff observed that providers differed in their style and philosophies in delivering

services and in the number of hours in which participants were engaged in specific activities. These

observed differences were critical to the pilot project by providing the opportunity to refer participants

to the provider, and thus the mix and style of services, that best met their needs.

 The length of time that Work First enrollees engaged in activities varied by type of activity and by

subcontractor. For example, as shown in Table 6, 38.1 percent of the participants spent two hours in

the assessment and employability planning activity (code 12), while 39.6 percent spent 20 hours in the

same activity. Of the three subcontractors within the Kalamazoo area, participants at YOU averaged

7.3 hours, those at Behavioral Foundation averaged 11.2 hours, and those at Goodwill averaged 16.0

hours in this activity. The higher average for Goodwill results from a much larger percentage of

participants spending time engaged in this activity than those assigned to other providers. Over threequarters of those going to Goodwill spent 20 hours in this service. Only 27 percent of the participants

receiving services from either YOU or Behavioral Foundation received 20 hours of this service. For

those going to YOU, two-thirds of the participants received two hours or less of assessment and

employability planning. Hours spent in this activity for those receiving services from Behavioral

Foundation were split between 2, 15, 16, and 20 hours. Hours spent in group or individual job-search

activities were much more uniform (Table 7). Ninety-seven percent of the participants spent 20 hours,

and there was no significant difference in the amount of hours the three subcontractors devoted to this

activity.
25

Table 6. Distribution of Hours Engaged in Assessment and Employability Planning

No. of Hours Percent

All Foundation Goodwill YOU

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

5.9

38.1

0.2

0.4

0.1

0.1

0.0

0.1

0.0

0.1

0.7

0.2

0.0

0.2

4.8

9.3

0.0

0.0

0.0

39.6

1.9

38.3

0.5

0.5

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.5

0.0

0.5

11.1

19.6

0.0

0.0

0.0

26.9

1.9

19.0

0.0

0.5

0.5

0.0

0.0

0.0

0.0

0.0

0.5

0.0

0.0

0.0

0.0

0.9

0.0

0.0

0.0

76.8

14.6

52.8

0.0

0.0

0.0

0.4

0.0

0.4

0.0

0.4

1.9

0.0

0.0

0.0

0.0

1.9

0.0

0.0

0.0

27.7
26

Table 7. Distribution of Hours Engaged in Job Search Activities

No. of Hours Percent

All Foundation Goodwill YOU

1

2

3

4

5

6

7

8

10

12

15

20

35

0.4

0.2

0.2

0.4

0.2

0.0

0.2

0.0

0.5

0.2

0.2

96.9

0.7

0.3

0.3

0.3

0.7

0.3

0.0

0.3

0.0

0.0

0.0

0.0

97.6

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.3

0.0

0.0

0.0

0.0

95.2

2.4

0.8

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.8

0.0

0.0

97.0

0.8

The service providers also differed to some degree in their approach to delivering services. 

For instance, Behavioral Foundation stressed a goal-oriented approach to job search, requiring that

participants contact a given number of employers each day until they find a job. Goodwill offered more

assistance to customers in conducting phone inquiries and in interviewing for jobs. Their staff would

work directly with customers to show them how to find employment postings and telephone numbers,

how to inquire about job postings, and how to present themselves during interviews. This same

organization would also provide more intensive training to those who were not able to find a job during

their initial several weeks in the program. 

H. Employment Outcomes

The goal of Work First is to move welfare recipients off the welfare rolls and into jobs so that

they can become economically self-sufficient. Employment success can be measured in several ways:

whether or not a participant holds a job, the length of time a participant holds a job, the hours worked,
12Another outcome of the Work First program, used by the state to measure effectiveness, is the placement

rate. The placement rate is defined as whether or not the participant held any unsubsidized employment during the

time they were enrolled (activity code 01). This measure is less of a hurdle to overcome, but it does show some

attachment to the workforce even if it does not terminate in 90 consecutive days of employment. Since the goal of

Work First is for welfare recipients to achieve economic self-sufficiency, we found the retention rate to be more

consistent with this goal.

27

or the hourly wage received. The positive outcome for Work First is for a participant to obtain

unsubsidized employment in a qualified job and to remain employed for 90 consecutive working days

(with a short grace period of no more than a week between jobs if they change jobs).12 Some enrollees

experienced periods of unemployment. Others had a job when they entered the program and continued

with that job throughout the program, ending with a successful termination.

I. Estimating the Statistical Assessment Model

A logit statistical technique was used to estimate the relationship between a Work First

participant’s personal characteristics and the likelihood of finding and retaining qualified employment for

90 consecutive days. The dependent variable in this statistical model is discrete, taking on the value of 1

(if employed) or 0 (if not employed). A logit estimation procedure transforms the discrete event into a

smooth functional form bounded by 0 and 1 and estimates the effect of specified variables on the

probability of employment. 

Estimates were based on a sample of Work First participants from the Kalamazoo-St. Joseph

WDB who enrolled in the program during 1996. The 1996 period is used because all who enrolled in

Work First during that time had completed the program before the start of the pilot and thus their

outcomes were known. Individuals can and do enroll in Work First several times. However, only about

8 percent of those who enrolled during 1996 enrolled more than once. We included each enrollee only

once in the sample and included their latest appearance so that we could use any previous history in the

analysis. The variable definitions and sample means were displayed in Table 3.
28

Results of the logit estimation are shown in Table 8. Focusing on the signs of the statistically

significant coefficients, Work First participants were more likely to complete 90 consecutive days of

employment if they had completed 12th grade (the omitted variable in the equation), were older, were

employed prior to first assignment, enrolled in the program earlier in the year rather than later, and were

not out of compliance if they had previously enrolled in Work First.

Table 8. Logit Estimates of the Basic Statistical Assessment Model

Employed Coefficient Std. Err. z P>|z| [95% Conf. Interval]

single parent 0.223 0.156 1.429 0.153 -0.083 0.528

age

age squared

0.115*

-0.002*

0.041

0.001

2.790

-2.602

0.005

0.009

0.034

-0.003

0.196

-0.000

no schooling

less than 9th grade education

9

th grade education

10th grade education

11th grade education

-1.801*

-0.454

-0.167

-0.775*

-0.431*

0.555

0.304

0.252

0.218

0.157

-3.244

-1.495

-0.662

-3.553

-2.744

0.001

0.135

0.508

0.000

0.006

-2.889

-1.049

-0.661

-1.203

-0.739

-0.713

0.141

0.327

-0.348

-0.123

GED

vocational education

1 year postsecondary education

2 years postsecondary education

3 years postsecondary education

0.174

-0.591

0.079

0.162

0.011

0.162

0.487

0.501

0.438

0.884

1.074

-1.212

0.159

0.371

0.013

0.283

0.225

0.874

0.711

0.990

-0.143

-1.546

-0.903

-0.695

-1.721

0.492

0.364

1.062

1.020

1.744

Goodwill

Behavioral Foundation 

 -0.463*

-0.560*

0.187

0.164

-2.485

-3.406

0.013

0.001

-0.829

-0.883

-0.098

-0.238

not a target group 0.064 0.116 0.555 0.579 -0.163 0.292

enrollment date -0.003* 0.001 -5.424 0.000 -0.004 -0.002

employed prior to enrollment

employed prior to previous enroll

1.107*

-0.393

0.144

1.055

7.683

-0.373

0.000

0.709

0.825

-2.46

1.390

1.674

noncompliance -0.750* 0.281 -2.672 0.008 -1.301 -0.200

constant 36.921* 7.260 5.086 0.000 22.693 51.150

Note: Number of obs=1,546, chi2(23)=213.10, prob > chi2=0.00, pseudo R2=0.10, and Log Likelihood = -948.47. (*)

denotes statistical significance at the 0.05 level. YOU is the omitted provider.
13These results are consistent with previous studies that examine the employment prospects of welfare

recipients. Estimates based on the national SIPP survey found that education and prior employment history were

important determinants of the likelihood of leaving welfare for employment (see appendix in Eberts 1997). A study

for the state of Texas also found these factors to be important (Schexnayder, King, and Olson 1991). The Texas

study also found that the number of children, the age of the welfare recipient, the duration on welfare, and the use of

the employment service and participation in job training programs also affected the likelihood of employment in the

expected direction. The employment- and training-related results from Texas are consistent with our results from

Work First that prior employment and compliance with previous Work First enrollment positively affect the

likelihood of qualified employment.

29

The only variable that may need an explanation for its inclusion in the model is the date of

enrollment into Work First. The coefficient on this variable is negative and statistically significant, which

indicates that those who enrolled in Work First in more recent periods experienced a lower probability

of finding and maintaining employment for 90 consecutive days. The percentage of participants who

achieved this outcome steadily declined during the operation of the pilot. During the first and second

quarters of 1996, 53 percent of participants in the sample were employed for 90 days, after which the

percentage dropped to 50 percent during the third quarter, 31 percent during the fourth quarter, and 24

percent during the first quarter of 1997. The admission date variable can be interpreted as a proxy for

attributes of Work First participants that are not captured in the characteristics included in the model. 

Work First staff observed that as the pool of welfare recipients going through the program diminished,

enrollees were increasingly less qualified to find and hold jobs. The variable may also capture changes in

the program and changes in local labor market conditions over time.13

Applying the estimated coefficients to the characteristics associated with each Work First

participant yields predictions of the probabilities of employment for each individual. Consequently, each

Work First enrollee can be ranked according to this estimated probability. One criterion for judging the

utility of the model is its ability to distinguish among Work First participants as to their likelihood of

finding employment. This ability can be measured in two ways: 1) the relative steepness of the

distribution of each individual’s employment probabilities, and 2) the width of the confidence intervals. If

the function is flat throughout the range of individual probabilities, then its ability to differentiate among

participants is minimal. On the other hand, if the function increases throughout the range of individuals,

then its ability to distinguish between participants with different employment propensities is greater.
30

Figure 1.

Predicted Probability of 90-day Employment

(95% confidence intervals)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

147

1013161922252831343740434649

50 groups

Predicted Probability

predicted probability

upper conf. interval

lower conf. interval

For heuristic purposes, one can view a plot of employability scores as representing participants

lined up to enter the Work First program according to their probabilities of finding employment. If the

door of the intake facility is envisioned to be on the left side of the graph in Figure 1, then participants

with the least propensity to find a job are at the front of the line and those with the highest propensity are

at the end on the right. For presentation purposes, the 1,546 observations included in this analysis have

been collapsed into 50 groups of about 30 people each. According to our model, the estimated

probabilities of employment range from a low of 0.02 to a high of 0.90. Therefore, the person at the

head of the line has almost no chance of finding a job and would need considerably more assistance than

the person at the end of the line, who is almost certain to find employment without much help. 

Also shown in Figure 1 are the 95 percent confidence intervals for each point on the logit

function, as represented by the 50 groups. A confidence interval shows the range of probabilities that

are statistically indistinguishable. The wider the confidence interval for any point on the logit function, the

less able the model is to differentiate among participants with any degree of statistical confidence. The

band is relatively tight along most of the curve, with the narrowest part of the band at the steepest

segment to the far right. Accordingly, an individual with a 70 percent probability (0.7 in the figure) of
31

Figure 2.

Actual and Predicted Probability of 90-day Employment

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

147

1013161922252831343740434649

 50 groups

Probability

predicted probability

actual

finding employment is indistinguishable from the 60 people to her right in the queue and the 210 people

to her left. A confidence interval of roughly 210 individuals on either side of a specific person is

maintained throughout much of the graph, except at the two tails. 

Figure 2 shows the relatively close relationship between the predicted probability and the

percentage of participants who are employed. Observations are within each of the 50 groups of 30

people. If the sample were larger within each group, the percentage employed would be tighter and

closer to the average predicted probabilities for each group.

Table 9 illustrates how the estimated coefficients are combined with an individual's specific

characteristics to generate a predicted probability of employment. Note that most of the explanatory

variables are binary, that is, the value of 1 is recorded when the characteristic describes the recipient and

0 otherwise. Three examples are given in Table 9. The first person described (Person A) is a single
32

parent with no formal schooling and no employment immediately prior to being assigned to Work First. 

She entered the Work First program in mid October 1996. She also was enrolled in Work First

previously, but left because of noncompliance. She was not employed for 90 days, and her probability

of finding a job was estimated to be 3.5 percent. Persons B and C, on the other hand, are single parents

in their mid-thirties. One has a 12th grade education and the other her GED. Both were employed prior

to assignment to Work First, and neither were terminated from previous enrollment in Work First as

noncompliant. They entered the program in the first quarter of 1996, and both had been employed for at

least 90 days. Each has a probability of employment of 88 percent.

Table 9. Characteristics of Participants with Low and High Employability Scores

Characteristics Person A Person B Person C

single parent (=1) 1 1 1

age 22 35 38

no formal schooling

9

th grade or less

9

th grade

10th grade

11th grade

12th grade

one year post-secondary

two years post-secondary

graduate equivalent degree

vocational education

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

1

0

0

0

0

notarget 1 0 0

prior employment 0 1 1

noncompliance 1 0 0

enrollment date 10/17/96 3/5/96 1/17/96

predicted probability of employment 0.035 0.884 0.880

employed 90 consecutive days (=1) 0 1 1
33

Figure 3.

Grade 12 Completion and Predicted Probability of 90-day Employment

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

147

1013161922252831343740434649

1546 individuals in 50 groups

Predicted Probability

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Fraction Completing 12th Grade

gr12

pemp

The statistical assessment model assigns a probability of employment to each individual participant. 

Thus, the probabilities, when ranked by size, create a continuum bounded by 0 and 1. In order to

examine the characteristics of individuals along the continuum, we divided the distribution of 1,546

participants into five groups, or quintiles, of equal numbers of participants. Table 10 shows the

relationships between education (particularly completing 12th grade and obtaining a GED), prior

employment, and noncompliance on the predicted probability and the percentage employed. Figures 3,

4, 5, and 6 show these relationships graphically relative to the predicted probability, but by using 50

groups instead of five.
34

Figure 4.

GED and Predicted Probability of 90-day Employment

0

0.1

0.2

0.3

0.4

0.5

0.6

147

1013161922252831343740434649

1546 individuals in 50 groups

Predicted Probability

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

% with GED

ged

pemp

Figure 5.

Prior Employment and Predicted Probability of 90-day Employment

0

0.2

0.4

0.6

0.8

1

1.2

147

1013161922252831343740434649

1546 individuals in 50 groups

Predicted Probability

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

% prior employment

code20_1

pemp
35

Figure 6.

Noncompliance and Predicted Probability of 90-day Employment

0

0.1

0.2

0.3

0.4

0.5

0.6

147

1013161922252831343740434649

1546 individuals in 50 groups

Predicted Probability

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

% non compliant

nocmpl

pemp

Table 10 also shows that the average predicted probability of employment generated by the model is

reasonably close to the percentage employed for each of the five groups, with perhaps the exception of

the first group. Only by dividing the distribution into groups can we discern the relationship between the

predicted probability and the actual probability. As shown in Table 11, the model classifies 66.2

percent of the cases correctly. To determine the percentage correct, a participant is classified as

“employed” and the variable is given a value of 1 if the predicted probability is greater than or equal to

0.5. Based on this criterion, 79.1 percent of the cases in which employment does not occur are

classified accordingly, whereas 48.9 percent of the cases that are true are classified as such. These

percentages can be changed depending upon the cutoff level chosen for classifying the event. 
36

Table 10. Characteristics by Quintiles of Employability Scores

Characteristics Group 1 Group 2 Group 3 Group 4 Group 5

single parent (=1) 0.789 0.787 0.830 0.86 0.862

age 28.2 28.8 30.7 29.6 31.0

no formal schooling

9

th grade or less

9

th grade

10th grade

11th grade

12th grade

one year post-secondary

two years post-secondary

graduate equivalent degree

vocational education

0.120

0.077

0.070

0.244

0.268

0.144

0.010

0.007

0.013

0.040

0.003

0.050

0.077

0.067

0.217

0.453

0.020

0.020

0.053

0.007

0

0.023

0.057

0.083

0.183

0.437

0.017

0.030

0.140

0.013

0

0.033

0.033

0.043

0.203

0.423

0.013

0.007

0.233

0.01

0

0.012

0.043

0.017

0.098

0.464

0.003

0.017

0.337

0.003

notarget 0.448 0.530 0.567 0.537 0.553

prior employment 0.033 0.013 0.060 0.147 0.660

noncompliance 0.227 0.027 0.020 0.017 0.003

predicted probability of 

employment

0.035 0.316 0.398 0.506 0.683

employed 0.167 0.330 0.397 0.543 0.660
14For evaluation purposes, half of the participants were assigned to the treatment group and the other half

were assigned to a control group. This will be discussed later in the report. For operational purposes, when an

evaluation is not called for, all would be assigned to providers according to this method.

37

Table 11. Relationship Between Actual and Classified Events

Dependent Variable: 90-day Employment=1

Classified

True

Total Employed (D) Not Employed (~D)

Employed (+) 323 185 508

Not Employed (-) 337 701 1038

Total 660 886 1546

Classified as employed if predicted Pr(Employed) >= 0.5

True Event “employed” defined as employed ~= 0

Sensitivity Pr ( +| D) 48.94%

Specificity Pr ( - |~D) 79.12%

Positive predictive value Pr ( D|+) 63.58%

Negative predictive value Pr ( ~D| -) 67.53%

False + rate for true ~D Pr ( +|~D) 20.88%

False - rate for true D Pr ( -| D) 51.06%

False + rate for classified + Pr ( ~D|+) 36.42%

False - rate for classified - Pr ( D| -) 32.47%

Correctly classified 66.24%

J. Assigning Probabilities on a Weekly Basis

The purpose of the model is to distinguish between participants according to their likelihood of

finding and holding a job. Participants in the treatment group were assigned to the three providers

according to their employability score, as shown in Figure 7.14 The distribution of employability scores

was divided into three groups, which roughly approximated the capacity of the three providers to

accommodate the participants. Actually, YOU was able to accommodate only 20 percent of the
38

Figure 7.

Referral of Participants to Providers

Based on Employability Score

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

50 groups

Predicted Probability

employability

score

Goodwill Y.O.U. Foundation

workers. The cutoff points in the distribution that determined the assignments (as shown by the vertical

lines in the figure) could be moved depending upon changes in the capacity of providers. 

In practice, probabilities were assigned to individuals as they enrolled in Work First twice 

weekly. Using the model estimated from the sample of 1996 Work First participants, Table 12 displays

the range of predicted probabilities by week starting in April 1997 and ending in June. For the weekly

assignment to approximate the assignment that would have occurred if everyone who participated in the

program over a year’s time were assigned at one time, the weekly and annual distributions need to be

similar. Table 12 shows that the means and the end points of the weekly distributions are similar to the

overall distribution and that the weekly distributions are fairly constant over time. This provides some

confidence that the weekly intake process is assigning participants in a consistent manner. Analysis of

the three employability groups in the evaluation section of this report gives additional support to the fact

that this occurred. 
39

Table 12. Assigned Predicted Probabilities by Weekly Intake

Week

Number

of persons 

Predicted probability

Mean Minimum Maximum

1 55 0.280 0.071 0.642

2 73 0.270 0.072 0.603

3 77 0.247 0.054 0.571

4 52 0.246 0.045 0.619

5 71 0.256 0.026 0.596

6 60 0.244 0.038 0.597

7 59 0.244 0.029 0.667

8 62 0.229 0.030 0.583

9 51 0.218 0.032 0.506

10 61 0.247 0.030 0.534

11 63 0.243 0.052 0.552

12 113 0.212 0.023 0.590

13 85 0.223 0.034 0.584

K. Alternative Specifications

Several variations of the model used to estimate employability scores were tried. The first

variation excluded the prior employment variables--prior to current enrollment and prior to previous

enrollment. The results are in column A of Table 13. As indicated by the pseudo R

2

, the percentage of

the variation explained in the occurrence of employment is smaller without the prior employment

variable. Another issue was whether different model specifications would change the ranking of

individuals according to the predicted probability. Table 14 displays the correlations of the predicted

probabilities from the various models. The high correlations indicate that the ranking of participants using

the various specifications is similar. The correlation of the predicted probabilities derived from the model

with the prior employment variables (labeled “pempq4") and without the prior employment variables

(labeled pempq4b) is 0.84. Since including age in the statistical assessment model may be a concern,

we have excluded it from the basic model but left in the prior employment variables. The correlation

between the two probabilities is 0.98, suggesting that excluding age may not change the ranking of the

employability scores to any significant extent. 
40

Table 13. Logit Model Specifications

Basic model

Basic model minus prior

employment

Basic model

 minus age

coef. t-ratio coef. t-ratio coef. t-ratio

single parent 0.223 1.49 0.171 1.13 0.209 1.35

age 0.115* 2.79 0.115 2.66

age2

no schooling

-0.002*

-1.80*

2.60

3.24

-0.0016

-1.74

2.47

3.18 -1.80 3.25

grlt9

gr9

gr10

gr11

ged 

voc ed

post1

post2

post3

-0.454

-0.167

-0.775*

-0.431*

0.174

-0.591

0.079

0.162

0.011

1.50

0.66

3.55

2.74

1.07

1.21

0.159

0.371

0.013

-0.497

-0.156

-0.743

-0.429

0.185

-0.643

-0.017

0.248

-0.218

1.67

0.635

3.48

2.79

1.17

1.34

0.034

0.587

0.246

-0.481

-0.190

-0.780

-0.475

0.209

-0.539

0.115

0.202

0.051

1.61

0.756

3.59

3.04

1.30

0.486

0.230

0.460

0.058

goodwill

foundation

-0.463*

-0.560*

2.49

3.41

-0.480

-0.642

2.65

3.98

-0.467

-0.569

2.51

3.47

notarget 0.064 0.055 0.071 0.625 0.090 0.782

addate -0.003* 5.42 -0.003 5.38 -0.003 5.60

code20_1

code20_2

1.11*

-0.394

7.68

0.373

1.10

-0.294

7.65

0.278

nocompl -0.750* 2.67 -0.804 2.92 -0.721 2.57

constant 36.92* 7.26 36.05 7.09 39.97 7.17

R

2

0.101 0.072 0.096

Note: (*) denotes statistical significance at the 0.05 level. YOU is the omitted service provider.

Table 14. Correlations of the Predicted Probabilities of Different Model Specifications

pnoage pempq4 pempq4b

pnoage (no age variables) 1.0000

pempq4 (basic model) 0.9787 1.0000

pempq4b (no prior employment variables) 0.8141 0.8440 1.0000
41

III. Implementation of the Statistical Assessment and Referral Work First

Operating System

A. Assignment of Providers to Serve Customers with Specific Scores

The purpose of the Kalamazoo/St. Joseph Work First pilot was to demonstrate the usefulness of

statistical tools in assessing the needs of customers and in referring them to services that are more

appropriate in meeting their needs. The desired outcome was an improvement in the retention rate (as

measured by 90 consecutive days of employment) of welfare recipients. The pilot was designed to

comply with state regulations for administering and delivering services and to work within the existing

operating system of the local Work First program.

The pilot was launched in January 1998. The staff of the Kalamazoo/St. Joseph WDB operated

the pilot, in cooperation with staff of the three organizations providing services to Work First

participants. The pilot focused on Work First participants residing in the greater Kalamazoo area. This

area was served by three providers: Goodwill Industries, Behavioral Foundation, and Youth

Opportunities Unlimited (YOU). The three subcontractors provided employment services to welfare

recipients in accordance with state guidelines for Work First. Before the pilot was initiated, participants

were randomly assigned to the three providers, because there was no way to assess the needs of

individual customers. 

Under the pilot, Work First participants were referred to one of the three providers depending

upon their employability score. WDB staff determined through their own observations and through the

development of the statistical assessment tool that each provider differed sufficiently in their approach

and philosophy toward delivering the prescribed services. Staff also observed that different providers

met the needs of some customers better than the needs of others. This tendency was supported by the

statistical analysis. The staff determined that Goodwill should serve customers with the lowest predicted

probability of employment, since it provided more hours of assessment and employability and since its

approach to services was more conducive to helping those with fewer job-ready skills. The staff also
15As discussed below, it turned out that throughout much of the pilot YOU could accommodate only 20

percent of the participants, leaving Goodwill and Behavioral Foundation to split evenly the remaining 80 percent. 

16Later on in the paper when the evaluation of the program is discussed, the term “treatment group” and

“control group” is used in a different way. Here, treatment refers to the assignment made before the pilot began, and

the control group is randomly assigned to providers. Later on in the paper, the treatment group refers to the optimal

assignment derived from different combinations of referrals to providers of those in the control group, whereas the

control group refers to the group derived from randomly assigning participants to providers. Additional explanation

will be offered in the evaluation section. 

42

determined that Behavioral Foundation should serve those participants with the highest employability

scores, since their philosophy of delivering services was more self-directed and self-paced. These two

decisions left YOU to provide services to the middle group.

The staff’s determination as to which provider should service which target group was supported

by analysis of administrative data prior to launching the pilot. After the assessment tool was estimated,

additional administrative data were used to estimate the relative effectiveness of the three subcontractors

in contributing to the outcomes of the three groups of clients. This test was based on the performance of

the three providers at a time clients were randomly assigned to each of them. The random assignment

helped to provide a valid test of the providers’ effectiveness. If this were not the case, then econometric

methodologies to adjust for selection bias would have had to be considered.

Since each of the three subcontractors could accommodate roughly a third of the participants,

we divided the distribution into thirds. If the capacity of any of the three providers changed, the

distribution was divided accordingly.15 Figure 8 shows how participants in the treatment group were

assigned. Those participants with scores in the first third of the distribution (low scores) were assigned

to Goodwill; those in the middle third to YOU; and those in the top third (high scores) to Behavioral

Foundation. 

For the purpose of conducting an evaluation of the pilot, participants were split into two

groups–a treatment and control group–within each of the three segments of the employability score

distribution.16 The treatment group was assigned to providers according to the criteria described above. 
43

Figure 8: Random Assignment Procedure

Referrals

Middle

Treatment

Y

Control

G F

High

Employability Groups

F

Control Treatment

Low

Control Treatment

G F Y G

Note: Dashed lines indicate that participants were randomly assigned to the next 

stage of the process. G refers to Goodwill, F to Foundation, Y to Youth 

Opportunities Unlimited. 

Y G F Y

The purpose of the treatment group is to determine whether the assignment of participants to providers

before the pilot began is consistent with the optimal assignment determined after the evaluation of the

pilot. The control group was randomly assigned to the three subcontractors within each of the three

groupings of the distribution. The purpose of the control group is to determine if different combinations

of referrals for each of the three groups of participants yielded different employment outcomes. If so,

this would support the premise that targeted referrals, based on a statistical assessment tool, could

improve the outcomes of the Work First system.

B. Intake Software Development

After the statistical assessment tool was estimated and the assignment of providers was

determined, the next step in implementing the pilot was to design a computerized intake process.

Employability scores were assigned to each participant during the intake process and those scores were

used to assign participants to providers, as described above. For purposes of evaluation, the intake

process also randomly assigned participants to a treatment and control group and assigned those within
44

the control group to the three providers. An overarching criterion for the computerized intake process

was for it to be easily understood, for a staff person to be able to operate it easily, and for it not to

encumber the intake process.

The computerized intake process was developed so that one staff member could operate it on a

laptop computer using a readily available database software package, in this case Lotus Approach. The

system was developed by research staff from the Upjohn Institute in collaboration with WDB staff. A

detailed description of the computerized intake process is included in the appendix. A key component

of the process was the ability to enter into the laptop’s database the relevant information about each

participant who was scheduled to report for orientation on a specific day. Having much of their

information already loaded onto the laptop reduced the amount of time required to the time of

enrollment. The software program was designed to visually flag variables that were missing from the

downloaded data so that the participant could supply the missing information as they enrolled in the

program.

Two staff members from the WDB were trained to operate the computerized intake process. 

Only one part-time staff person was needed to operate the laptop during the actual intake process, but

an additional person was trained in order that someone was available to fill in if needed. The training

took about half a day, with some updating of procedures as changes in the reporting of some information

occurred during the time the pilot was in operation. It was also necessary to coordinate activities with

the data entry staff from the WDB, who maintained the administrative database for the Work First

program. Coordination was required because data had to be downloaded from the Work First master

administrative database to the laptop. Data from the laptop were not uploaded to the master files. Any

information that was supplied by the participant and generated for the pilot program, such as who was

assigned to a control or treatment group, was maintained on a standalone file.
45

C. Intake Procedure

The intake process took place as follows. Individuals who were granted cash assistance from

the Family Independent Agency (FIA) and met certain requirements were referred to Work First. 

Those individuals were then notified to report to orientation at a specific time. Orientation typically took

place once a week. Prior to the day of orientation, data for those participants who were scheduled to

report for orientation that day were downloaded to the laptop computer from the state master file. The

staff person responsible for the intake process verified that the data were for the correct date, that the

printed sheets reflected the correct total participants, and that all the relevant information present in the

master file was properly downloaded into the spreadsheet.

As individuals entered the facility where the orientation was held, they were handed a sheet that

listed the information needed to generate their employability scores, such as prior employment,

education, and so forth. They were asked to fill in any missing information that was not already

downloaded from the master file. As they entered the orientation session, they returned the sheets to the

staff person operating the laptop computer. During orientation, which typically lasted up to two hours,

the laptop computer operator entered the missing information supplied by the participants. For those

who were on the schedule, the operator had only to enter their social security number and the predownloaded data appeared. If information was missing, the operator entered it into the computer

database from the sheets filled out by the participants. For those who reported to the program but were

not scheduled to enroll that day, a new record was created so long as their information was found in the

master file. 

Once all the participants’ information was entered into the database, the program automatically

computed employability scores for each individual and assigned participants to providers. The program

printed a form for each participant that included the name of the provider to whom they were to report,

the time and day they were expected to report, and the provider’s address and telephone number. The

appropriate form was distributed to participants as they exited the orientation session. For the few
46

individuals who did not want to report to their assigned provider, perhaps because of a prior experience,

they were assigned to an alternative provider, which was duly recorded in the database. Very few

participants requested a provider other than the one to which they were originally assigned.

The rest of the Work First program remained the same as it was before the pilot. Participants

reported to their assigned provider and received the services that they offered. The WDB tracked and

recorded the activities of the enrollees while they participated in the program. WDB staff contacted

welfare recipients who left the program with a job after 90 days to determine their employment status. 

Those employed for 90 consecutive days were considered to have achieved a successful outcome.
17During the last six months of the two-year pilot, the state replaced the retention rate, defined as 90

consecutive days of employment, with a slightly different criterion. This criterion was based on the wage level and

weekly hours. However, success rates using the new criterion compared favorably with the success rates based on

the previous one, so we used the new criteria for the last six months without concern about biasing the results in any

significant way. 

47

IV. Evaluation of the Kalamazoo/St. Joseph Work First Pilot Project

A. Purpose and Design of the Evaluation

The Kalamazoo/St. Joseph Work First pilot was evaluated using a random assignment

methodology. The goal of the pilot was to determine whether a statistical assessment tool can help

identify the needs of clients and then whether using that tool to refer participants to service providers

increases the effectiveness of the Work First program. The principal measure of effectiveness was the

retention rate, which is defined as the percentage of participants who were employed 90 consecutive

days in a qualified job.17 A qualified job was one that was unsubsidized, offered a minimum wage, and

at which the participant worked more than 20 hours per week. The state designated the retention rate

as one of the performance criteria for Work First. The other performance criterion was the placement

rate, which was defined as the percentage of participants who engaged in full time or part time work in

an unsubsidized qualified job. The evaluation focused on retention rates as the performance outcome. 

Placement rates were not considered in the evaluation, because the statistical assessment tool used to

separate participants into the three employability groups was based on retention rates and not placement

rates. Hourly wages and weekly hours worked, however, were used along with the cost of the pilot to

compute a benefit-to-cost ratio.

The evaluation included participants who entered the program from March 1998 to March

2000. Although the pilot began in January 1998, the evaluation was delayed a few months in order for

the staff to become acquainted with the program and to minimize any difficulties associated with the

computerized intake process. During the two-year period, nearly 3,600 welfare recipients who were

single parents were assigned to the three providers serving the Kalamazoo area. Two-parent families

were also served, but they were not included in the pilot nor in the evaluation. About half the
18One could argue that including the same person more than once in the evaluation overweights that

person’s experience relative to those who entered the program only once. More will be said about this approach in a

subsequent section. 

48

participants went through the program at least twice. For purposes of the evaluation, if a person

appeared more than once in the program, only their last appearance was included. This approach was

adopted to avoid biasing the evaluation towards multiple enrollees.18

In order for the pilot to be successful, three components had to be effective. First, the statistical

assessment tool had to predict the employability of welfare recipients with sufficient precision to separate

the participants into three distinct groups. Second, providers had to offer services in a sufficiently

different way so that services could be targeted to meet the particular needs of the various groups. 

Third, one had to determine which provider could best meet the needs of each group of welfare

recipients. The first condition–the precision of the statistical assessment tool--was discussed in a

previous section, and it was shown that it could separate the welfare recipients into three relatively

distinct groups, as indicated by the confidence intervals. The second condition was supported by

opinions of the WDB staff that the three providers had different philosophies and approaches to

providing employment services to welfare recipients and that those differences could serve one group

better than another. The statistical analysis also suggested that the participants spent varying amounts of

time engaged in at least one service activity, depending upon to which service provider they were

assigned.

The third condition requires that the proper assignment of welfare recipients to providers is

known before they enroll in the program. The proper assignment is one that yields a higher retention rate

among the possible combinations of providers and groups of welfare recipients than would occur from

simply randomly assigning participants to providers, as was done prior to the pilot project. Three

methods are possible. One is to use the opinions of staff to determine the proper assignment. The

second is to analyze the outcomes of participants who had enrolled in Work First prior to the pilot. The

third approach combines the two, which was the method used in the pilot to assign participants in the
19Three providers and three groups of participants can be sorted into nine distinct combinations. Only six

are used in the evaluation, however, since participants had to be referred to each provider, according to the

contractual arrangements. 

49

treatment group to the various providers. Those in the low employability group were assigned to

Goodwill Industries, those in the middle group to YOU, and those in the high group were assigned to

Behavioral Foundation. Of course, other combinations of referrals are possible. Manski (2001)

suggests that the best approach of determining the proper assignment is to run a small random

assignment experiment before the targeting program begins and then to use the results of this experiment

to assign participants in the treatment group to service providers.

B. Random Assignment Procedure

The treatment in this pilot project was different from that typically found in other demonstrations. 

Instead of testing the effectiveness of receiving a specific service versus not receiving the service, the

purpose of the pilot was to demonstrate the relative effectiveness of referring participants with specific

employability scores (and thus specific needs) to various service providers. In order to examine the

effectiveness of these different combinations of referrals, participants in the control group were randomly

assigned to each of the three subcontractors. The relative effectiveness of the six combinations of

referrals was evaluated.19 By comparing the retention rates of the combinations from the treatment

group that correspond to those in the comparison, we can then determine the relative performance of the

treatment group. Since the referrals to the treatment and control groups and to the various combinations

within the control group were randomly assigned, the combinations of referrals in the treatment and

control groups should closely match. 

The random assignment procedure, integrated into the computerized intake process, took place

in three steps. As shown in Figure 8, participants were first divided into one of three groups, depending

upon their employability score. Assignment of participants to the three employability groups was based

on their relative ranking in the distribution of employability scores of those who enrolled in Work First at

that session: it was not based on a pre-determined cutoff value. Those participants with employability
50

scores in the lowest 40 percent of the distribution were assigned to the low employability group (L), the

next 20 percent were assigned to the middle group (M), and the highest 40 percent were assigned to the

high group (H). Second, those within each of the three groups were divided equally into control and

treatment groups. Third, enrollees in the control group were randomly assigned to one of the three

providers. Those in the treatment group were assigned to a pre-determined provider that was

considered to be most effective for those in each of the three employability groups. The middle group

included only 20 percent of the participants because the treatment provider for that group, YOU, was

limited by its capacity to accommodate only that percentage of enrollees. 

The actual assignment of employability scores was different from the way in which the statistical

assessment model was originally estimated. The model was estimated based on the entire set of

individuals who participated in and completed the program during a year’s time. The computation of

employability scores, based on the coefficients from the model, was done at each intake and orientation

session, which took place twice a week. Obviously, only a small number of people who participated in

the program attended each session. During the pilot, 30 to 40 people would typically show up for each

session. When we developed the pilot, we anticipated that there might be a difference in the distribution

of participants with respect to employability scores at each session. We found, however, that the

distribution was roughly the same each week, for a sample of several weeks, and that the range was

comparable to the range of the full sample of participants enrolling in the program during an entire year,

as shown in Table 12. 

The actual intake process was confounded by other factors. First, not all participants who

showed for orientation were included in the pilot. Of the 30 to 40 people who showed up, only about

half were included. Some were excluded because their records were not found in the master file. 

Without those records, participants could not be assigned an employability score, and without being part

of the master file their activities and their employment outcome could not be tracked. In addition, others

were dropped from the evaluation because they were enrolled in the two-parent program, which was
51

not part of this evaluation (even though they received services alongside those in the single-parent

program). 

Because of the small number of participants at each session, it may be the case that individuals in

attendance on any given day were not fully representative of the Work First population. In examining

the distribution of employability scores for each session, we found that on some days the employability

scores would cluster on the high side, while on other days they would center on the low side of the

distribution. Since the cutoffs were determined by dividing the distribution of scores of individuals who

showed up on a given day, it is possible that individuals with lower-than-average employability scores on

one orientation day were assigned to the “high” employability group, while on another day individuals

with higher-than-average employability scores were assigned to the “low” employability group. It

depended upon who was referred to a particular session. 

Another difference between the employability scores as originally estimated and those assigned

to participants during the pilot was the magnitude of the score. We noticed that the employability scores

declined over the year in which the statistical assessment model was estimated. This relationship was

consistent with the general observation by the WDB staff that as an increasing number of Work First

participants found jobs, those remaining had lower skills and were harder to place into jobs and more

difficult to serve. To account for this trend, we included in the model the date that the participant

enrolled in the program. The coefficient on this variable (enrollment date), as shown in Table 8, was

relatively large and highly statistically significant. The value of the coefficient (-0.003) was large relative

to the mean of the variable (approximately 14460, which is the date expressed in machine language). 

Nevertheless, it turned out that, as time increased from the date in which the model was

estimated to when it was used to assign the employability scores, the coefficient played a much larger

role in determining the size of the predicted value. The mean value of the employability score fell from

about 0.30 in the original model to 0.05 in the evaluation. Most of the difference was due to the more
20In hindsight, however, it would have been better to specify the admission date in another way, perhaps as

a quadratic, so that the time effect would not be accentuated so much. The quadratic specification would allow the

data to determine whether or not the retention variable linearly related to the admission date. 

52

advanced date. When the date is rolled back to its average value during the period in which the model

was estimated, the mean employability score for the sample used in the evaluation increases to 0.46. 

Further investigation shows that the rank ordering of employability scores computed with and

without the adjustment for the time is highly correlated. The correlation coefficient of the actual

employability score assigned to participants during the evaluation and the hypothetical one when the date

of enrollment is rolled back by two years is 0.82. Therefore, since the assignment of a participant to a

provider is based on the relative position of the individual in the distribution and not on a specific cutoff

point, it is reasonable to conclude that the referrals would be roughly comparable regardless of whether

the actual score or the time-adjusted score was used.20 

Despite these confounding factors, the employability score still serves as a way to provide a

relative ranking of the likelihood that an individual will find a job. Table 15 shows the means and ranges

of employability scores by the three employability groups. With respect to the means, the three groups

have the employability scores that one would expect. The scores rank individuals from low to high

probability of employment. Although the overall means of the score are lower than the actual

employment rates (0.049 vs. 0.157), there is some spread in scores between the three groups. The

mean score for the middle group is 52 percent higher than the mean score for the low group, and the

mean score for the high group is 65 percent higher than the middle group’s average. The range does not

follow this pattern as neatly, however. While the person with the highest employability score was

assigned to the high employability group, so was the person with the lowest employability score. This

assignment presumably resulted from the small number of participants in each session and the

nonrandom nature of the referrals by FIA to each session at various times. 
53

Table 15. Estimated Employability Scores by Employability Groups

Mean

Standard

Deviation Minimum Maximum Number

Low 0.028 0.026 0.003 0.148 783

Middle 0.043 0.035 0.005 0.142 377

High 0.071 0.071 0.003 0.587 826

All 0.049 0.055 0.003 0.587 1986

The characteristics of participants in each of the three employability groups were also consistent

with sorting individuals according to their likelihood of employment. Consider educational attainment

and age, displayed in Table 16, both of which have been shown to be positively correlated with the

prospects of employment. With respect to educational attainment, those in the high employability group

have roughly 1.2 more years of schooling than those in the middle group, and those in the middle group

have slightly more than one additional year of schooling than those in the low group. Within each group, 

educational attainment is roughly the same across providers, although YOU deviates slightly from the

norm in the middle and high groups. A participant’s age, another variable positively related to

employability, also exhibits a consistent pattern across employability groups. Members in the high

employability group are two years older than those in the middle group, and those in the middle group

are about a year older than those in the low employability group. 

C. Program Outcome: Retention Rates

The primary outcome measure for the evaluation is the retention rate–whether the participant

was employed 90 consecutive days. Table 17 shows the retention rates of those in the control and

treatment groups by employability group and provider. In this case, there is considerable variation both

across groups and within groups. Note that the actual retention rate averaged for each group increases 

from the lowest employability group to the highest. For the control group, it increases from 11.6 percent

for the lowest group to 21.7 percent for the highest employability group. The treatment group also

follows the pattern of increasing retention rates from low to high employability groups. The same
21The overlap is not as great between the low and middle employability groups as it is between the middle

and high groups. The difference in the average retention rates for the low and middle employability groups is

statistically significant at the 95 percent significance level. On the other hand, the difference in the average retention

rates for the middle and high employability groups is not. 

54

monotonic increase is exhibited for each provider, except for YOU. Nevertheless, as shown in Table

18, the upper and lower bounds of the 95 percent confidence intervals overlap across the various

groups.21

Table 16. Years of Education and Age by Provider and Employability Group

 Education

Employability Group

Low Middle High

Provider Control Treatment Control Treatment Control Treatment

Goodwill 10.67 10.76 11.80 13.24

Foundation 10.89 11.76 13.10 12.76

YOU 10.71 12.48 12.60 12.57

Average 10.78 11.88 13.09

Age of Participant

Employability Group

Low Middle High

Provider Control Treatment Control Treatment Control Treatment

Goodwill 25.76 26.44 27.51 28.28

Foundation 25.93 26.52 29.87 29.73

YOU 25.93 28.37 28.07 29.69

Average 25.87 27.19 29.24

Table 17. Retention Rates by Provider and Employability Group

Employability Group

Low Middle High

Provider Control Treatment Control Treatment Control Treatment

Goodwill 0.153 0.154 0.219 0.226

Foundation 0.079 0.145 0.223 0.234

YOU 0.136 0.370 0.170 0.167

Average 0.116 0.208 0.217
22This difference in retention rates will be discussed later. 

55

Table 18. Upper and Lower Bounds of 95 Percent Confidence Intervals for the Retention Rates of Each 

Provider

A. Control

Group

Employability Groups

Low Middle High

Lower Mean Upper Lower Mean Upper Lower Mean Upper

Goodwill 0.094 0.153 0.212 0.124 0.219 0.314 0.162 0.226 0.290

Foundation 0.039 0.079 0.119 0.069 0.145 0.221 0.167 0.223 0.279

YOU 0.049 0.136 0.223 0.188 0.370 0.552 0.068 0.167 0.266

B. Treatment

Group

Employability Groups

Low Middle High

Lower Mean Upper Lower Mean Upper Lower Mean Upper

0.119 0.154 0.189 0.117 0.170 0.223 0.193 0.234 0.275

For YOU, the retention rate is much higher for the middle employability group than it is for the

high employability group.22 While the retention rates increase for each provider from low to high

employability groups, the difference in retention rates varies from group to group for each provider. This

variation, along with differences in retention rates among providers, indicates a comparative advantage

among providers in serving participants of various abilities. This will be discussed later as well.

For those who entered the program more than once between March 1998 and March 2000, we

included only their last appearance in the evaluation. Table 19 compares the retention rates of the

sample that includes only that last appearance to the retention rates of the sample that includes multiple

appearances. The retention rate of the sample with multiple appearances is slightly lower, 15.7 percent

versus 18.4 percent. This difference reflects the possibility that those who enrolled in Work First more

than once have had greater difficulty finding and retaining a job. It is also consistent with the view that by

including their last appearance only, the sample contains people who have had more experience with the

Work First program and with searching for employment. Nonetheless, there is little difference in the

observed comparative advantage of each provider in serving their assigned customers, as measured by

the ranking of providers according to their effectiveness in promoting retention. The ranking of the

effectiveness of each provider within each group is the same, with one exception: within the low
56

employability group, the retention rates for those receiving services from Behavioral Foundation and

YOU are virtually identical. The differences in the retention rates, however, are more pronounced in the

sample with single appearances than the one with multiple appearances. While recognizing that

differences in the retention rates across different combinations of providers and employability groups

may not be as large, we base the rest of the evaluation on the sample that includes last appearances only.

Table 19: Comparison of Retention Rates from Single and Multiple Appearance Samples

Low Middle High

Multiple Once Multiple Once Multiple Once

Goodwill 0.133 0.153 0.202 0.219 0.203 0.226

Foundation 0.104 0.079 0.143 0.145 0.171 0.223

YOU 0.103 0.136 0.265 0.370 0.149 0.167

All 0.113 0.116 0.183 0.208 0.180 0.217

As noted earlier, participants were randomly assigned to a control group and to a treatment

group, and the two groups included roughly the same number of participants. Another check of the

validity of the random assignment technique used at intake is to compare the retention rates of those

within the control group who were randomly assigned to a provider with those in the treatment group

who were purposely assigned, or targeted, to a provider. Since enrollees are randomly assigned to the

control group or treatment group, this comparison is instructive. As shown in Table 17, the retention

rates are similar for the low and high employability groups. In the low group, the treatment group was

assigned to Goodwill. This group had a retention rate of 15.4 percent, which is virtually identical to the

retention rate of 15.3 for those in the control group assigned to Goodwill. In the high group, the

treatment group was referred to Behavioral Foundation. This group had nearly the same retention rate

as those in the control group who were referred to Behavioral Foundation, 23.4 percent versus 22.3

percent. Results of a t-test show that the retention rates are not statistically significantly different at the

95 percent confidence level.

The only statistically significant difference in retention rates between the treatment and control

groups for the same combination was with the middle group. The treatment group, which was assigned
23One difference between YOU and the other two providers was that YOU served two-parent families at the

same time it served single parent families. The programs are different, but there is no apparent reason why this

would have yielded the high retention rates for the control group.

57

to YOU, had a retention rate of 17.0 percent. The difference in the two rates is much larger than

expected, and the rate for the control group is out of line with those of the retention rates of YOU

participants in the low and high employability groups. One explanation for this difference is the small

sample size of the control group referred to YOU. As shown in Table 20, the control group assigned to

YOU in the middle employability group included only 26 participants, compared with 194 for the

treatment group. The assignment to a control group and treatment group was blind to both the

participants and the staff of YOU. Participants from both groups attended the same services provided

by YOU. Because the assignment was blind, there is no reason to believe that the staff treated the two

groups differently.23

Table 20: Number of Participants Assigned to Each Provider

Employability Group

Low Middle High Total

Provider Control Treatment Control Treatment Control Treatment Control Treatment

Goodwill 144 402 73 164 381 402

Foundation 177 83 211 402 471 402

YOU 59 26 194 54 140 194

Total 380 402 183 194 429 402 992 998

D. Variation in the Outcomes of Different Combinations of Providers

In order to determine whether different combinations of assignments of employability groups to

service providers yield different outcomes, we examined six combinations that occurred during the study

period. More than six combinations are possible with three providers and three employability groups by

assigning more than one employability group to a provider. Nevertheless, we adhered to the WDB’s

contractual arrangement during the pilot that all three providers would deliver services. Therefore, we

eliminated from consideration combinations that assigned two or three groups to one service provider.
58

The effects of the various combinations are measured by computing the number of participants

belonging to each employability group who retained their jobs, assuming everyone in that group received

services from the same provider. To illustrate this approach, consider the first combination listed in

Table 21. The designation “gyk” refers to the combination in which all participants in the low

employability group (the left-most group in Table 17) are assigned to Goodwill (g); all participants in the

middle employability group are assigned to YOU (y); and all participants in the high employability group

are assigned to Behavioral Foundation (k). Since participants in the control group were randomly

assigned to each of the providers within each of the three employability groups, using the subgroup

assigned to a particular subcontractor to represent the effects for everyone in that employability group is

a sound approach. 

Table 21: Number of Participants Employed 90 Consecutive Days by Combination of Providers

Employability Group

Combination of Providers Low Middle High Total Ranking

1 gyk 58 68 96 222 1

2 gky 58 26 72 156 5

3 ygk 52 40 96 188 3

4 ykg 52 26 97 175 4

5 kyg 30 68 97 195 2

6 kgy 30 40 72 142 6

Note: Providers are designated as letters: “g” Goodwill; “k” foundation; and “y” YOU. The combination “gyk” refers

to low employability group assigned to Goodwill, the middle employability group to YOU, and the high employability

group to Behavioral Foundation.

Using this approach, the appropriate retention rate for each employability group (obtained from

Table 17) is multiplied by the total number of participants in the control group in order to compute the

number of participants within that group who retained their jobs for 90 consecutive days. For instance,

for the first combination, the retention rate of 0.153 for Goodwill is multiplied by 380, the size of the

control for the low employment group. This yields 58, which indicates that 58 participants (out of a total

of 380) in the low employability group of the control group would have retained their jobs if all were

assigned to Goodwill. The same calculation is performed for the middle group, multiplying 0.370 by 183

which yields 68, and for the high group, multiplying 0.223 by 429 gives 96. Summing these three figures
59

yields the total number of participants in the three control groups who retained their jobs. Dividing by

the total number of participants in the control groups (992) results in the hypothetical retention rate if the

combination “gyk” were used to assign participants.

Performing these calculations for all six combinations provides a convenient measure of the

effectiveness of the various combinations. As shown in Table 21, the number of retentions ranges from a

high of 222 for the combination “gyk” to a low of 142 for “kgy”. The difference between the highest and

lowest is 80 retentions, or 56 percent. The difference between the highest number and the average is

47, or 27 percent. That is, the optimal combination of providers “gyk” yields a 27 percent higher

retention rate than if the participants were randomly assigned to the providers. The results indicate that

using the statistical tool to assess and refer Work First participants to providers increases the

effectiveness of the program without significantly increasing staffing or changing the nature of the

services.

Most of the differences between any of the various pairs of combinations are statistically

significant at the 95 percent significance level. Table 22 displays the difference in the retention rates and

the t-statistics for each pair of combinations. For instance, the difference between the retention rate for

combination “gyk” and combination “gky” is 0.066 (e.g., 65/992). The t-statistic for this pair is 5.26,

which is much greater than the critical value of 1.96 for a 95 percent significance level. Note that 10 out

of the possible 15 pairs are statistically significant. Only those with differences in the retention rates of

less than two percentage points (approximately 20 participants out of 992) are not statistically significant.

Based upon the analysis of the effectiveness of the combinations of providers, it appears that

Goodwill had a comparative advantage in serving low employability participants, YOU in serving middle

employability participants, and Behavioral Foundation in serving high employability customers. This

combination of assignments was the same as the treatment group, which was determined by staff

knowledge of the approaches taken by each provider and an analysis of welfare recipients who had
60

pˆ qˆ

1

n

1

%

1

n

2

participated in the program before the pilot began. It was beyond the scope of the pilot, however, to

determine the specific aspects of each provider’s approach that led to this outcome. 

Table 22. Differences in Retention Rates between Pairs of Combinations of Providers

A. Differences in Retention Rates

Provider

Combination

1 2 3 4 5 6

1 gyk - 0.066* 0.034* 0.046* 0.026* 0.080*

2 gky - -0.031* -0.019 -0.039* 0.014

3 ygk - 0.012 -0.008 0.045*

4 ykg - -0.020 0.033*

5 kyg - 0.053*

6 kgy -

B. t-Test of Difference in Retention Rates

1 2 3 4 5 6

1 gyk - 5.260 2.671 3.654 2.028 6.487

2 gky - -2.603 -1.618 -3.245 1.244

3 ygk - 0.986 -0.644 3.842

4 ykg - -1.630 2.860

5 kyg - 4.481

6 kgy -

Note: Standard deviation derived according to the following formula:

where ; ; and x1, x2 are number of successes in the samples of size n1 and n2

pˆ ' . t-statistics greater

x1% x2

n1% n2

qˆ ' 1&pˆ

than 1.96 indicate that the null hypothesis of no difference in retention rates can be rejected at the 95 percent

significance level, denoted by asterisk (*).

As previously noted, the retention rate for those in the middle employability control group

assigned to YOU is higher than the rate for the treatment group assigned to YOU. If, as intended,

individuals were randomly assigned to the treatment and control groups and then those within the control

group were randomly assigned to the providers, one would expect the two retention rates to be similar. 

We tried two alternative approaches of deriving retention estimates for the different combinations that
61

may mitigate the problem. The first approach controls for factors that could be responsible for the

significant difference between the treatment and control groups assigned to a specific provider. One

possible factor is the date in which participants enter the program. It could be the case that, because of

the small number enrolled during each session and the nonrandom nature of referrals from FIA, the time

of enrollment may lead to these differences. The second method was to combine the outcomes of both

control and treatment groups. In this way, we reduce the effect of the timing of enrollment by

considering outcomes from both groups.

The first approach entailed estimating the retention rates for each grouping by controlling for the

date of enrollment. Logit estimation was used to relate the event that a person was employed for 90

consecutive days (a binary variable) to the date of enrollment. In addition, nine binary variables, one for

each of the nine cells created by the intersection of the employability group and providers, as shown

back in Table 16, were generated and included in the estimation. For instance, a binary variable for

assignment to Goodwill within the low employability group took the value of 1 if an individual was

assigned to that cell and 0 otherwise. The same procedure was followed for the other eight cells. One

binary variable was omitted for the estimation. Only the control group was used in the estimation.

The results (not shown) exhibit a statistically significant negative relationship between the

probability of employment and the date of enrollment. However, the adjustment did not change the

retention rates significantly. In fact, as shown in Table 23, the adjusted retention rate for those in the

middle employability group assigned to Goodwill was identical to the actual rate. When these rates are

used to compute the retention rate of the various combinations of providers, the ordering is generally the

same as it is for the actual rates. As shown in Table 24, the adjusted rates maintain the same general

ordering, except for the order of the second and third place, which is virtually a tie. In addition,

differences in retention rates are statistically significant for the same pairs of providers as were found for

the unadjusted rates, with only two exceptions (Tables 25 and 26). These exceptions did not alter the

total number of statistically significant pairs.
62

Table 23. Estimated Retention Rates Using Logit Estimates Controlling for Admission Date

Provider

Employability Group

Low Middle High

Estimated Actual Estimated Actual Estimated Actual

g 0.153 0.153 0.239 0.219 0.235 0.226

k 0.078 0.079 0.158 0.145 0.240 0.223

y 0.150 0.136 0.369 0.370 0.120 0.167

Table 24. Retention Rates by Combination of Providers

Provider

Combination

Employability Group

A B C

Actual Rank Adjusted Rank Combined Rank

gyk 0.223 1 0.231 1 0.206 1

gky 0.157 5 0.171 5 0.157 5

ygk 0.189 3 0.205 2 0.202 2

ykg 0.176 4 0.188 4 0.177 4

kyg 0.197 2 0.200 3 0.192 3

kgy 0.143 6 0.157 6 0.143 6

Note: “Actual” refers to all means from random assignment; “adjusted” refers to logit estimates using admission date;

and “combined” refers to including both control and treatment groups to compute all means.
63

Table 25. Differences in Retention Rates between Pairs of Combinations of Providers Using Logit Estimates

Controlling for Admission Date

A. Differences in Retention Rates

Provider Combination 1 2 3 4 5 6

1 gyk - 0.060* 0.026* 0.043* 0.031* 0.074*

2 gky - -0.034* -0.017 -0.029* 0.014

3 ygk - 0.017 0.005 0.048*

4 ykg - -0.012 0.031*

5 kyg - 0.043*

6 kgy -

B. t-Statistic of Difference in Retention Rates

1 2 3 4 5 6

1 gyk - 4.726 1.986 3.334 2.379 5.905

2 gky - -2.745 -1.397 -2.355 1.193

3 ygk - 1.349 0.392 3.931

4 ykg - -0.957 2.588

5 kyg - 3.550

6 kgy -

Note: t-statistics greater than 1.96 indicate that the null hypothesis of no difference in retention rates can be rejected

at the 95 percent significance level, denoted by asterisk (*). See Table 22 note for the standard deviation formula. 

Table 26. Differences in Retention Rates Between Pairs of Combinations of Providers Using Combined Treatment

and Control Groups

A. Combined Treatment and Control

1 2 3 4 5 6

1 gyk - 0.050* 0.004* 0.030* 0.014 0.063*

2 gky - -0.045* -0.020 -0.035* 0.014

3 ygk - 0.025* 0.010 0.059*

4 ykg - -0.015 0.034*

5 kyg - 0.049*

6 kgy -

B. t-Statistic of Differences in Retention Rates

1 2 3 4 5 6

1 gyk - 4.893 0.442 2.885 1.423 6.301

2 gky - -4.418 -1.865 -3.364 1.319

3 ygk - 2.226 0.900 5.339

4 ykg - -1.307 2.897

5 kyg - 4.315

6 kgy -

Note: t-statistics greater than 1.96 indicate that the null hypothesis of no difference in retention rates can be rejected

as the 95 percent significance level, denoted by asterisk(*). See Table 22 note for the standard deviation formula.
24I wish to thank Kevin Hollenbeck and Jeff Smith for suggestions and guidance on conducting the

benefit/cost analysis.

25The social value of the new system may be less than the value computed here because of displacement

effects among the welfare population. It is conceivable that the additional retention by participants of the program

with the new system may displace other welfare recipients from their existing jobs or preclude new Work First

participants from finding jobs since the additional retentions reduce the job vacancies. Bartik (2001) estimates that

the displacement effect among low-wage workers ranges from 20 to 60 percent. That is, for every ten additional

Work First participants who find a retain a job, two to six jobs are lost by other less-educated workers. Therefore,

the social value of the additional placements and retained jobs by Work First participants is less than the value

64

The second approach combined the outcomes of both control and treatment groups. The

retention rates for the various combinations of providers are also included in Table 24. Note that the

ordering is the same as it is for the adjusted rates, although there is a virtual tie for first place.

The range in outcomes from the various combinations of service providers indicates the

difference in effectiveness of the providers in meeting the needs of Work First participants. It also

indicates the ability of the employability assessment tool to distinguish among participants. If the tool

was not an adequate predictor of a participant’s needs, there would not have been the systematic

difference in retention rates across employability groups. Furthermore, the differences in outcomes

across combinations would not have been as great.

E. Benefit/Cost Analysis of the Statistical Assessment and Referral System24

The benefits of using the statistical assessment and referral system can be valued by taking into

account the earnings received by those additional participants who retained their jobs. As described in

the previous section, the optimal assignment rule yielded a net increase of 47 participants who retained

their jobs 90 consecutive days over the number in the group created by random assignment. 

Consequently, the net effect of the statistical assessment and referral system can be computed by

comparing the earnings of those referred by the optimal assignment rule with the earnings of those

assigned to the randomly assigned group. A benefit-to-cost ratio is then calculated by first using the net

effect to measure the benefit of the system and then dividing it by the cost of the pilot. The benefit-tocost ratio measures the return to society of implementing and operating the program.25

 
computed for the program itself. It should also be mentioned that while society (and most directly participants) may

benefit from the additional earnings, current funding arrangements for Michigan’s Work First program do not

provide additional revenue to local workforce developments boards that generate higher earnings. 

26Eight quarters may be a conservative estimate, and there is no way to determine from our data the average

length of time a participant remains employed nor is it possible to determine the length of time the earnings

difference persists. Evaluations of the pre-PRWORA welfare-to-work programs show that the impact on earnings

was as strong in the third year after a participant left the program as it was during the first two years (U.S.

Department of Health and Human Services and U.S. Department of Education, January 2001, p. 61).

65

To compute the net effect of the system, the first step is to calculate the earnings for both the

treatment group (the optimal referral design) and the control group (the group that would result from

randomly assigning participants to providers). Let BT denote the earnings of the treatment group, and

BC denote the earnings of the control group. The earnings are made up of two components: the number

of participants who retained their jobs (R) and the average weekly earnings of each participant in that

group during the 90 days (calculated here as 13 weeks) of employment (E). Therefore, the net effect is

the difference in the earnings of the treatment group (BT) and the earnings of the control group (BC). 

This difference can be decomposed in the following way, using the control group as the base of

comparison:

BT - BC = [(RT - RC) * EC] + [(ET - EC) * RC] + [(RT - RC) * (ET - EC)]

This decomposition yields the net effect in terms of additional earnings to program participants as a result

of the statistical assessment and referral system. Although data are available only at the time of the 90-

day followup (these are the values included in Table 27), more than likely the earnings difference

continues for several quarters. It is assumed here that the earnings difference continues for eight

quarters, with two possible scenarios considered.26 The first scenario assumes that the difference in the

number of participants retaining their jobs for 90 days persists throughout the 8 quarters. The second

scenario assumes that the difference in job retention narrows throughout the eight-quarter period until

they are equal. In both scenarios, wages are assumed to grow by 3 percent per year, and a 10 percent

annual discount rate is used when computing the net present value of the earnings streams. 
66

The value for the number of participants in the treatment group who retained employment for

90 days is found in Table 21, and the number of retainees for the control group is computed from the

retention rates in Table 17. Average weekly earnings are computed from the hourly wages and weekly

hours displayed in Table 27. Table 28 shows average weekly earnings for the various assignment rules

and for the control group. 

Table 27: Hourly Wages and Weekly Hours of Participants Working 90 Consecutive Days

Employability Group

Low Middle High

wage hours wage hours wage hours

Goodwill 7.02 32.95 6.08 25.94 6.02 28.22

Foundation 5.04 24.64 5.14 25.83 7.43 32.17

YOU 7.03 31.88 6.23 32.00 7.21 32.33

Weighted Average 6.39 30.11 5.82 27.50 6.85 30.61

Table 28. Average Weekly Earnings by Different Combinations of Providers

Combination of Providers Average Weekly Earnings

gyk (treatment group) 192

gky 211

ygk 181

ykg 175

kyg 165

kgy 189

Randomly assigned (control group) 195

Note: Providers are designated by letters: “g” Goodwill; “k” Foundation; and “y” YOU. The combination “gyk” refers

to low employability group assigned to Goodwill, the middle employability group to YOU, and the high employability

group to Foundation. 

Using these figures, the difference in the earnings streams over 8 quarters is computed and

displayed in Tables 29 and 30. The results in Table 29 are based on the assumption that the retention

rates remain constant during the 8 quarters after the participant leaves the program. As one can see

from the table, the total number of participants in the treatment group who retained their jobs 90

consecutive days remained the same, as did the number retaining their jobs in the control group. It is
67

assumed, however, that the earnings of the two groups converge, as shown in the last two columns of

the table. Based on these assumptions, the net present value of the earnings differential equals

$840,827. The results shown in Table 30 are similar to those in Table 29 except that the difference in

the number of retained jobs converges until they are equal after 8 quarters. In this case, the net present

value of the earnings differential between the two groups is $471,054. 

Table 29: Difference in Earnings between Treatment and Control Groups and Benefit-to-Cost Ratio of the

 System, Assuming Retention Rates Remain Constant

Quarters after leaving

program

BT - BC RT RC ET EC

1 $112,179 222 175 $192.00 $195.00

2 $113,666 222 175 $193.44 $196.08

3 $115,165 222 175 $194.89 $197.18

4 $116,675 222 175 $196.35 $198.28

5 $118,197 222 175 $197.83 $199.39

6 $119,730 222 175 $199.31 $200.51

7 $121,274 222 175 $200.80 $201.63

8 $122,830 222 175 $202.31 $202.77

Net present value $840,827

Program Cost $145,000

Benefit-to-cost Ratio 5.8

Note: This calculation of net impact and benefit-to-cost ratio assumes that the retention rates remain the same

throughout the eight-quarter period, while the average weekly earnings converge. Wages are assumed to increase 3

percent per year and a 10 percent discount rate is assumed for the net present value calculation. 
27The development of the system was funded by the U.S. Department of Labor. Therefore, including the

entire cost of development when calculating the benefits and costs of the system to an individual program

overstates the costs. Replicating this system in other Work First programs would conceivably cost considerably

less, since these other programs can adopt the general design and the basic structure of the system already

developed for this pilot program. 

68

Table 30: Difference in Earnings between Treatment and Control Groups and Benefit-to-Cost Ratio of the 

 System, Assuming Retention Rates Converge

Quarters after leaving the

program

BT - BC RT RC ET EC

1 $112,179 222 175 $192.00 $195.00

2 $98,706 216 175 $193.44 $196.08

3 $85,073 210 175 $194.89 $197.18

4 $71,279 204 175 $196.35 $198.28

5 $57,321 198 175 $197.83 $199.39

6 $43,197 193 175 $199.31 $200.51

7 $28,906 187 175 $200.80 $201.63

8 $14,445 181 175 $202.31 $202.77

Net present value $471,054

Program Cost $145,000

Benefit-to-cost Ratio 3.25

Note: This calculation of net impact and benefit-to-cost ratio assumes that the retention rates and the average weekly

earnings converge during the eight-quarter period. Wages are assumed to increase 3 percent per year and a 10

percent discount rate is assumed for the net present value calculation. 

The additional costs incurred to develop and operate the statistical assessment and referral

system for the two-year life of the pilot totaled $145,000. This expense included designing and

integrating the system into the existing Work First program, which cost roughly $105,000, and hiring a

part-time person to administer the system during the intake and orientation process, which amounted to

another $40,000 during the two-year period.27 Operating this system requires hiring a part-time staff

person who downloads the data from the master files prior to the weekly orientation, enters missing data

during the orientation, and runs the programs that generates the employability score for each participant

and refers them to providers. These tasks took no more than 6 hours a week for the pilot project. The
69

amount of staff time required to operate the system in other programs depends upon the number of

participants going through the program and the frequency of the orientation sessions. Obviously, if the

retention rates are roughly the same regardless of the size of the program, those programs with more

participants will also have proportionally more participants retaining their jobs. Therefore, the additional

earnings due to the system will also be proportionally larger and is likely to exceed the additional costs.

Based on the difference in earnings computed above and the total cost of designing,

implementing and operating the system, the benefit-to-cost ratio ranges from 3.25 to 5.8, as shown in the

bottom rows of Tables 29 and 30. Assuming longer streams of earnings differentials between the two

groups would obviously increase the return to this program. Nevertheless, the more conservative

estimates of earnings used here demonstrate that the benefits of the system exceed its costs. 
70

V. Replicating the Pilot in Other Areas: Broward County, Florida 

As part of its responsibilities under a grant from the U.S. Department of Labor to conduct the

Work First pilot, the Upjohn Institute was asked to provide technical assistance to states and local

employment service agencies that were interested in developing a similar assessment and referral tool. 

The Florida Institute for Career and Employment Training (FICET), located in Broward County,

Florida, expressed an interest in the tools developed by the Institute. FICET is affiliated with Florida

Atlantic University and is sanctioned under the Florida Board of Regents with responsibilities in welfare

reform and workforce development, among other employment-related activities. Members of their staff,

including the executive director, Dr. Phillip Rokicki, and the assistant director, Jorge Zumaeta, attended

a technical workshop that the Institute held in August 1998. FICET subsequently applied for and

received a grant from the U.S. Department of Labor to develop similar tools for use by Broward’s

County Work First staff. 

 FICET developed tools that were similar to those designed by the Upjohn Institute. The

overall objective of the pilot was to use the information typically collected at intake to shape a

participant’s particular mix of programs and services in order to offer the best chance for success. The

statistical tools were expected to perform three functions: 1) calculate success ratios based on client

characteristics and past performance, 2) estimate the effect of various job training programs/services on

employment, and 3) evaluate the program on an on-going basis. These objectives were very similar to

those of the Upjohn Institute Work First pilot.

The statistical model was based on a logit estimation in which employment outcome was related

to demographic information, a person’s work history, and employment services offered by the Work

First program. The demographic variables included age, number of children under the age of 13,

gender, ethnicity, education, and work experience. The services included employment preparation,

GED preparation or some high school, vocational education training, unsubsidized employment,

assessment, English as a second language, and other counseling. The option to choose among a set of
71

services, depending upon the customer’s needs, was one of the features that distinguished the Florida

program from the Michigan program. In Michigan, each customer received the same set of services

regardless of need, with few exceptions. Therefore, in the Michigan pilot, the variation in services was

achieved by referring customers to different providers, which offered the same basic services but

delivered them in slightly different ways that appeared to benefit some clients more than others. In

Florida, the variation of services was more direct by actually referring clients to different services.

Data to estimate the logit model were derived from the TANF database maintained by the

Florida Department of Children and Families, which determines eligibility, and from the Work Activity

Database, which records activities of TANF clients. The initial estimates were based on more than

7,000 records of Work First participants in Broward County. Broward County has a population of

approximately 1.4 million with a labor force of 650,000. The county’s unemployment rate during the

initial phase of the pilot was about 4.8 percent. The pilot began with about 3,800 active TANF cases.

Work First participants had the following characteristics: 1) the average age was 31, with a

range from 16 to 68; 2) 31 percent of the population had completed the 12th grade; 3) 43 percent were

classified as high school dropouts, 4) 27 percent graduated from high school or received a GED; and 5)

28 percent had not worked at any job in the past two years.

The statistical model was used to estimate an employability score for each customer. Based

upon that score, customers were placed in one of two tracks. Track I included customers with minimal

barriers to employment. The cutoff value of the success ratio for this track was set at 70 percent. 

Customers assigned to Track II had moderate barriers and had a success ratio of less than 70 percent. 

Once assigned to a specific track, customers then received prescribed services. For instance, those in

Track I received core services as described under the Workforce Investment Act. Those in Track II

could participate in intensive services, such as job skills training, vocational training, and GED prep,

among others. The tool was integrated into the Broward County’s MIS system and was used during the

intake process. 
72

VI. Conclusion

The purpose of the Work First pilot project is to determine the benefits of targeting employment

services to meet the individual needs of Work First participants. The pilot developed a statistical

assessment tool to identify the employability of welfare recipients and then used the tool to refer them to

service providers that were determined beforehand to best meet their needs. The pilot was conducted

at the Kalamazoo/St. Joseph WDB, and its design, implementation, and evaluation was carried out by

the W.E. Upjohn Institute for Employment Research. The pilot operated within the budget and program

requirements of Michigan’s Work First program.

A. Statistical Assessment and Referral System

The statistical assessment tool related personal characteristics and work history of former Work

First participants to the event that they worked for 90 consecutive days. Estimates were based on

administrative records of welfare recipients who had participated in the Work First program prior to the

time of the pilot. The assessment tool was incorporated into an automated intake process that used the

estimated coefficients corresponding to the various personal characteristics and other factors included in

the model to compute a score, indicating the probability of working 90 consecutive days. A score was

computed for each person who enrolled in the Work First program. The score was then used to assign

new enrollees to a service provider that was predetermined to best meet their needs.

B. Evaluation Findings

The pilot was evaluated using a random assignment design. At the time of enrollment,

participants were randomly assigned to a control group and a treatment group, and within the control

group to each of the three providers. Success of the pilot depended upon three components: 1) the

ability of the statistical assessment tool to predict with adequate precision the employability of individual

participants; 2) sufficient differences in the type of services and the methods of delivering services among

the three subcontractors so that some participants may benefit more than others from these packages of
73

services; and 3) the ability to determine before the referral is made as to which provider is better suited

to assist participants as identified by their employability score.

The evaluation yielded the following results. First, the statistical model exhibited sufficient

precision to distinguish among participants according to their likelihood of working 90 consecutive days. 

Second, there was considerable variation in the retention rates among the various combinations of

providers offering services to participants in the three employability groups, as identified by the

assessment tool. The retention rate of the combination of providers that yielded the highest rate was 56

percent higher than the combination yielding the lowest rate, and 27 percent higher than the average. 

Third, the predetermined assignment of participants to providers in the treatment group, as determined

by the judgement of the staff and by statistical analysis, was the same combination that yielded the

highest retention rate according to the random assignment experiment.

While the large difference in outcomes associated with the various combinations of referrals

demonstrates the effectiveness of targeting services, it also underscores the importance of properly

aligning the statistical assessment tool with the goals of the program and in accurately estimating the

statistical assessment model. Results showed a 56 percent difference in job retention rates between the

most effective and least effective combination of referrals. If the statistical assessment model is not

properly specified and estimated or the model is not aligned with the goals of the program, then the

targeting procedure could yield a suboptimal outcome, and perhaps lead to an outcome that is worse

than simply randomly referring clients to providers. Consequently, care has to be exercised in specifying

the statistical assessment model and in estimating the predicted outcomes, such as the estimation of job

retention by the employability score used in the Kalamazoo-St. Joseph Work First pilot program. As

shown in the pilot program, successful implementation of such a procedure can increase the effectiveness

of a welfare-to-work program, or other similar employment service, in meeting its objectives. 
74

The net present value of the difference in earnings between the treatment group (generated from

the optimal assignment rule) and the control group was used to estimate the net impact of the program. 

The net present value, assuming that the earnings differentials persisted for 8 quarters, ranged from

$471,000 to $841,000. Combining these estimates with the total cost of designing, implementing and

operating the program of $145,000 yielded a benefit-to-cost ratio that ranged from 3.25 to 5.8. 

Therefore, the benefits of the statistical and referral system sufficiently covered the operating expenses

and the fixed cost of designing and implementing the pilot.

C. Extensions to Other Sites and Programs

The assessment and referral system developed for the Kalamazoo/St. Joseph Work First pilot

has been adopted elsewhere. The Broward County (Florida) WIB, in cooperation with the Florida

Institute for Career and Employment Training, has adopted an approach that is similar to the one in

Kalamazoo. It has been successfully integrated into the WIB’s operating system. Employability scores

are assigned to individuals entering the program and referral to specific sets of services are based on a

participant’s score. 

The success of the Kalamazoo-St. Joseph Work First pilot in increasing job retention and the

successful implementation of the Broward County WIB program open the possibility of adopting similar

approaches in other areas and for other programs. The computerized intake process developed for the

Kalamazoo/St. Joseph pilot is easily adaptable to other systems. Its standalone nature presents few

compatibility problems when integrating it into an existing operating system. 

Within Work First programs, the statistical assessment and referral system may take on added

value as staff is faced with customers who have fewer skills and thus are harder to serve. As an

increasing percentage of Work First customers are having difficulty finding and holding a job, they end

up cycling through the program, which increases their frustration in finding a job and uses resources that

are proving not to be effective. The evaluation of the Kalamazoo/St. Joseph pilot provides evidence that
75

targeting services may increase the effectiveness of Work First programs for those who are harder to

serve. 

The Upjohn Institute, with funding from the Employment and Training Administration of the U.S.

Department of Labor, is extending the techniques used in the Work First pilot to develop a set of

administrative tools for use within One-Stop Centers. Referred to as the Frontline Decision Support

System (FDSS), the purpose of this system is to assist staff in quickly assessing the needs of customers

and in referring customers to services that better meet their needs. FDSS includes new tools to 1) help

customers conduct a systematic search for jobs that offer the best employment match and to set a

realistic wage goal, and 2) assist staff in determining which One-Stop Center services are likely to be

effective in meeting the needs of specific customers in becoming employed. The FDSS tools are

designed to be used within the current data retrieval and display systems implemented by the states for

their One-Stop Centers. These tools have the flexibility to interface with existing operating systems and

visual displays of various One-Stop Centers.

The results of the Kalamazoo/St. Joseph Work First pilot and efforts elsewhere provide

evidence that the statistical assessment and referral system can be successful in identifying needs and in

targeting services to help customers find jobs. The Work First pilot demonstrates that integrating the

system into an existing welfare-to-work program can increase the retention rate of participants at little

additional cost to the program. The pilot opens the possibility for statistical tools to be used to help

improve the effectiveness and efficiency of other employment programs and service delivery systems. 
76

References

Bartik, Timothy J. Jobs for the Poor: Can Labor Demand Policies Help? New York, NY: Russell

Sage Foundation and Kalamazoo, MI: W.E. Upjohn Institute for Employment Research, 2001.

Berger, Mark C., Dan Black, and Jeffrey Smith. “Evaluating Profiling as a Means of Allocating

Government Services,” University of Western Ontario Working Paper (#200018), September

2000.

Bloom, Dan and Charles Michalopoulos. How Welfare and Work Policies Affect Employment and

Income: A Synthesis of Research, New York, NY: Manpower Demonstration and Research

Corporation, May 2001.

Eberts, Randall W. The Use of Profiling to Target Services in State Welfare-to-Work Programs:

An Example of Process and Implementation, Kalamazoo, MI: W. E. Upjohn Institute for

Employment Research, WP00098-052, 1997. 

Eberts, Randall W., Christopher J. O’Leary, and Kelly J. DeRango. “Frontline Decision Support

System for One-Stop Centers,” paper delivered at the APPAM Annual Meeting, Seattle, WA:

November 2000.

Freedman, S., Friedlander, D., Lin, W. and Schweder, A. The GAIN Evaluation: Five Year Impacts

of Employment, Earnings, and AFDC Receipt. New York, NY: Manpower Demonstration

Research Corporation, July 1996.

Friedlander, Daniel. Subgroup Impacts and Performance Indicators for Selected Welfare

Employment Programs. New York, NY: Manpower Demonstration and Research

Corporation, 1988.

Grossman, Jean, Rebecca Maynard, and Judith Roberts. Reanalysis of the Effects of Selected

Employment and Training Programs for Welfare Recipients. Princeton, NJ: Mathematic

Policy Research, Inc., 1985.

Gueron, Judith and Edward Pauly. From Welfare to Work, New York, NY: Russell Sage Foundation,

1991.

Manski, Charles F. “Identification Problems and Decisions Under Ambiguity: Empirical Analysis of

Treatment Response and Normative Analysis of Treatment Choice.” Journal of Econometrics.

95(2) (2001), pp. 415-442.
77

Manski, Charles F. “Statistical Treatment Rules for Heterogeneous Populations.” Unpublished

manuscript. Evanston, IL: Department of Economics and Institute for Policy Research,

Northwestern University, October 2001.

Schexnayder, Deanna, Christopher King, and Jerome Olson. “A Baseline Analysis of the Factors

Influencing AFDC Duration and Labor Market Outcomes,” report to the Center for the Study of

Human Resources and the Bureau of Business Research, The University of Texas at Austin,

1991.

U.S. Department of Health and Human Services and U.S. Department of Education. Evaluating

Two Welfare-to-Work Program Approaches: Two-Year Findings on the Labor Force

Attachment and Human Capital Development Programs in Three Sites. New York, NY:

Manpower Demonstration Research Corporation, December 1997.

U.S. Department of Labor. Evaluation of Workers Profiling and Reemployment Services System,

Employment and Training Administration, 1997. 

U.S. Department of Labor. National Evaluation of Welfare-to-Work Strategies: How Effective Are

Different Welfare-to-Work Approaches? Five-Year Adult and Child Impacts for Eleven

Programs. New York, NY: Manpower Demonstration Research Corporation, November

2001. 

U.S. Department of Health and Human Services and U.S. Department of Education. National

Evaluation of Welfare-to-Work Strategies: What Works Best for Whom: Impacts of 20

Welfare-to-Work Programs by Subgroup. New York, NY: Manpower Demonstration

Research Corporation, January 2001.
78

Appendix 

Description of the Automated Database Implemented by

Kalamazoo/St. Joseph’s Work First Profiling Pilot Project

W. E. Upjohn Institute for Employment Research

Kalamazoo, Michigan

Kris Kracker

Ljiljana Vesic-Petrovic
79

I. Introduction: The Purpose of Statistical Assessment and the Role of the Automated

Database

The Upjohn Institute designed, tested, and implemented a statistical assessment and referral

system, which was integrated into the daily operations of the Work First program that the Institute

administers for the Kalamazoo-St. Joseph WDB. As part of this effort, the Institute developed an

automated data base system to perform the task of calculating the probability of success in finding and

retaining employment for Work First clients, and thus to allocate resources according to levels of need. 

This appendix includes a description of the automated intake process and a set of instructions that were

prepared to train staff on the use of the system. The system, written as an APPROACH database

program, is available free of charge to interested parties. 

The intake system performed four basic functions. First, it retrieved data on each participant

that was scheduled to enroll in Work First on a particular day. The data were obtained from the master

files of the Family Independence Agency and downloaded onto the laptop computer which was used

during the intake process. Second, the system computed the employability scores, based on the

downloaded data. Third, the system assigned participants to the various treatment and control groups

and to the various providers, based on the employability score. Fourth, it stored the relevant data that

were necessary to evaluate the pilot. The purpose of this appendix is to provide a brief description of

the automated database and computerized intake system.

II. The Database: Structure, Record Storage, and Access

 

A. Structure and Storage

The Work First database used for the assessment and referral system actually consisted of two

databases: a “temporary” one for running calculations in the assessment process, and a permanent one

to hold the results. Such a composition allowed for a database that stores information with greater

integrity. The database was written in Lotus Approach, but other database programs could be used.
80

With that structure, data for expected orientation participants were imported into the temporary

database each day or for each orientation within a day. The data remained in the first database through

the completion of calculations. At that point, a flag marking the current orientation’s records was set. 

When the database was closed, the records were then imported into the permanent database, because

upon opening the temporary one again, all records with a flag would be deleted.

Since the temporary database contained only records of those individuals expected to enroll on

a particular day, there was no possibility that when the processor searched for a person’s record it

would find or harm a record that was stored from a previous orientation. All such records were

maintained in the permanent database for tracking and other purposes.

The temporary database was used primarily by the processor who checked records and

completed the statistical assessment for clients present at a given orientation. It was the processor’s

responsibility to import the current orientation’s records into the permanent database. The main menu of

the second database contained a button to initialize the importation of data. The computer requested a

confirmation for importing, after which the step was completed. At that time, the day’s results could be

viewed if necessary, but normally a Work First Assignment Sheet for each person would have been

printed.

Other than importing and viewing the day’s results, the processor had no responsibilities nor

privileges within the permanent database. That database had been constructed for the purpose of

storing past records and was intended to be used by management or other persons who may be tracking

the results and conducting statistical analyses. Certain measures had been taken that allowed for a

distinction among users and therefore a distinction of different user levels of access.
81
82

B. Levels of Access

Unique privileges had been assigned to distinct database users for the purpose of maintaining quality

data. Through the use of passwords, different types of database users could gain different levels of access.

For instance, since the processor worked primarily with the statistical assessment in the temporary database,

he or she had privileges only to import and view data in the permanent database. The screen in which the

day’s results could be viewed was protected by read-only access in this case.

The manager or other persons who used mostly the permanent database had a separate password

with greater privileges. While the main menu appeared the same, the pop-up menu in the lower right corner

of the status bar allowed access to other spreadsheets or reports he or she created. In addition, the pallet

of menus along the top of the screen allowed for many more options.

III. Using the Database: The Statistical Assessment and Referral Process

This section of the Appendix provides instructions for operating the statistical assessment and referral

process. In order to use the statistical tool to assess the employability of the welfare recipients referred to

Work First, there are four steps in completing the process:

1. Importing new data for each orientation’s expected participants.

2. Editing expected participants’ records and adding new records for those not scheduled to

report at that session.

3. Computing and assigning participants to program destinations.

4. Distributing referral sheets.

While the database program is designed to guide the processor through each of those steps with ease, a

walk-through of the process is described below.
83

A. Importing New Data

The statistical assessment process begins when the Approach file is opened. By default, the Main

Menu is the first view that appears on the computer screen. This view contains four options, the first of

which is to import new data. Before importation, data for the day’s possible participants are extracted from

the master database for the Work First program and then stored in a spreadsheet file. With a click of a

button, importing the spreadsheet is automatic, except for confirming that you do want to go ahead with the

selection action.

B. Editing and Adding Records

After the day’s data have been imported, the processor then moves along to the Participant Record

view in order to edit or add new records for today’s participants. Upon entering this view, the screen is

equipped to search for a record simply by entering a unique identifier, such as a social security number or

first and last name, and then pressing <Enter>.
84

To type a person’s social security number or name into a field and find a record, use either the

<Tab> key or the mouse. Tabbing is consistent throughout the database and may be a faster means of

entering data. 

Since search mode is the default when entering the Participant Record view, in order to return to

the main menu or do anything aside from searching for a record, one must first hit <Enter> to deactivate the

mode.

Upon finding a participant’s record, missing information is completed and the orientation date added.

While there is an automatic check that the age, marital status, education level, target and code 20 are

completed for each record, there is no check for the orientation date. It is particularly important that the

processor enter the date for participants in attendance. Without completion of the orientation date, the

record and therefore the person will receive neither a group assignment, nor a Work First destination

assignment. The process is the same for each person that is present.
85

If any of the fields for age, marital status, education level, target or code 20 is left blank, a message

will appear. The optional responses will be “OK” and “Help.” Since “OK” is highlighted, pressing <Enter>

will allow the processor to go back and enter the missing age. In order for each of the present participants

to be profiled and referred to receivers, these fields must be completed for each of their corresponding

records.

When the last name field is used to search for a record, since it is not guaranteed to be unique, as

in the case of “Smith” or “Brown,” the computer may find more than one record that corresponds to the

name entered. The portion of the status bar in the bottom left corner of the screen will indicate if this should

occur. In the case that more than one record is found, the processor may forward to the next or previous

record by clicking on the arrows in the status bar.

If a person’s record is not found or does not exist, he or she was not expected to attend the

orientation that day and his or her data were not imported. However, a new record may be added simply

by clicking on the New Record button and the person’s information entered at the time they enter

orientation.

When all participants have been checked in, the processor must then return to the main menu in

order to assign participants to a Work First site.

C. Computing Calculations and Assignments

On the main menu there is a button entitled “Assign Work Locations.” By clicking on this button,

the algorithm calculates the employability scores for each participant and assigns the participant to the

appropriate provider. A sheet is printed that contains Work First destination information for each

participant. To receive the output, however, the algorithm goes through several steps.

First, since the data that was imported at the beginning of the intake process contained records for

all of the day’s possible orientation attendees, those not in attendance (or whose records were not found

and the dates were not added) are thus deleted. In addition, for the duration of the calculations, people with
86

a Work First Referral code equal to two are disregarded since they are referred to orientation only and not

assigned to a Work First program location.

Next, from the remaining records, participants from a two-parent household (that is PRG=2) are

automatically assigned to the destination YOU. These participants are neither part of the treatment group

nor the control group for this project. While two-parent families are included in the same intake and

orientation session, they are not profiled in the pilot. Only single parents (PRG=1) are targeted for profiling.

The probability of success for each participant is calculated automatically when the data are

imported, so the computer needs only to sort the records (where PRG=1) in descending order of probability

after which it assigns a rank to each one. Using a random number generating function, each of the single

parent participants are placed randomly in either the treatment or the control group. Members of the control

group are then randomly assigned, within the bounds of subcontractor capacity, to one of the three possible

Work First sites in the pilot model. Capacity is represented in the calculation process as a weight for the

percent of total participants that each location may handle.

The remaining members of the treatment group are assigned to destinations according to their

probability of success and based upon their ranking among the entire sample of single-parent participants

in both the control and treatment groups. Once again, this assignment depends on the available capacity at

each of the locations.

Finally, the destination results are printed for each individual.

D. Distributing Referral Sheets

On the Work First Assignment Sheet printed for each individual, only information relevant to the

participants is disclosed. Each person receives a print-out containing his or her name, social security

number, the orientation date and the place of assignment along with its address. Even the date and time to

report are printed according to each individual’s Work First site.
87

Work First Assignment

Soc Sec #: 

First Name: 

Last Name: 

Today's Date: 

Work First Site: 

Work First Site Address: 

Phone Number: 

Date to Report: 

Time to Report: 

Contact: 

 The Michigan Family Independence Agency has assigned you to the Work First program. You are 

 scheduled to report to the following agency for services:

 The primary purpose of the Work First program is to assist you in locating and obtaining gainful 

 employment. We will begin the assessment of your needs at the above location and time. We do 

 not have daycare on site. You will need to find child care for your child/children. If you have any

 questions, cannot make the appointment, or are currently working, please contact the above person

 immediately.

If you fail or refuse to appear for this scheduled appointment, action may be taken to reduce 

 or cancel you FIA grant.

 In accordance with the Americans with Disabilities Act, this information will be made available in

 alternative format upon request. Michigan Relay Center, 1-800-649-3777 (Voice and TDD).

Address: 

Case Number:

For tracking purposes, calculation results such as the probability of success, ranking, group,

destination, etc. are stored in a separate and permanent database. If there should be a need to view the

profiling results of the current day, there is easy access to the information in the second database.