Quasi-Experimental Impact Study of NFWS/SIF

Workforce Partnership Programs

Evidence on the Effectiveness of Workforce Partnership

Programs in Ohio and Wisconsin

Final	Report	–	August 2016

Authors

Marios Michaelides

Project Director and co-Principal Investigator

(University of Cyprus and	IMPAQ	International)

Peter Mueser

Co-Principal Investigator

(University of Missouri and	IMPAQ	International)

Scott	Davis	

(IMPAQ International)

Kassim	Mbwana

(IMPAQ International)

Prepared and Submitted by:

IMPAQ	International, LLC

10420 Little Patuxent Parkway

Columbia, MD 21044

Submitted to:

National Fund for Workforce Solutions

88 Broad Street

Boston,	MA	02110
TABLE OF CONTENTS

Page

Executive Summary ...................................................................................................................	i

1. Introduction.......................................................................................................................	1

2. Background........................................................................................................................	3

2.1 The National Fund	for Workforce Solutions......................................................................	3

2.2 Quasi-experimental Impact Study Design........................................................................	4

2.2.1 Key	Research	Questions.............................................................................................	5	

2.2.2 Program	Selection ......................................................................................................	6	

2.2.3 Data	Sources...............................................................................................................	9	

2.2.4 Quasi-experimental Approach .................................................................................11

3. Program Descriptions .................................................................................................... 17

3.1 Partners for	a	Competitive Workforce Collaborative......................................................17

3.1.1 Health Careers Collaborative	of Greater Cincinnati.................................................17

3.1.2 Advanced	Manufacturing Partnership .....................................................................21

3.1.3 Construction Sector Partnership ..............................................................................23

3.1.4 Participant Characteristics........................................................................................24

3.1.5 Services	Received .....................................................................................................29

3.2 The Milwaukee	Area	Workforce	Alliance ........................................................................31

3.2.1 Wisconsin Regional	Training	Partnership.................................................................	33

3.2.2 Milwaukee Area Healthcare Alliance .......................................................................34

3.2.3 Participant Characteristics........................................................................................34

3.2.4 Services	Received .....................................................................................................38

4. Quasi-experimental Impact Study Results ..................................................................... 39

4.1 Ohio Impact Study Results...............................................................................................39

4.1.1 Matching	Results......................................................................................................41

4.1.2 Impact	Results..........................................................................................................47

4.1.3 Discussion of the Results..........................................................................................58

4.2 Wisconsin Impact Study	Results	.................................................................................... 64
4.2.1 Matching	Results...................................................................................................... 66

4.2.2 Impact	Results.......................................................................................................... 70

4.2.3 Discussion of the Results.......................................................................................... 80

5. Conclusion........................................................................................................................87

References ..............................................................................................................................90

Appendix.................................................................................................................................	92

Disclaimer .............................................................................................................................104
IMPAQ International, LLC Page i NFWS/SIF Quasi-Experimental Impact Study

Executive Summary

Since	its	establishment in 2007, the	National Fund for Workforce Solutions	(NFWS)	has	invested

in	 Regional	 Funding Collaboratives that	 match	NFWS funds with funds from other sources	 to	

support the development of	 local	 workforce partnerships. These partnerships identify the

workforce needs of local employers and design	 and	 administer programs	 to	 help workers	 to	

obtain the skills needed	 to	 meet those needs. By 2010, NFWS was supporting 30 local

workforce partnerships with	active	training programs in six states.	In	2010, NFWS was awarded

a	two-year, $7.7 million	grant	by the	Social Innovation Fund	(SIF)	to	expand	these	programs	and	

to	support the creation	of new programs.

In	2011, NFWS contracted with	IMPAQ	International, LLC (IMPAQ)	to	evaluate	the	effectiveness

of	the	30 NFWS/SIF-funded	workforce partnership programs. The evaluation consisted of:

1) an outcomes study to	examine	program	participation, services provided, and participant

outcomes in the period of	the SIF funding (January 2010	–	February 2012);	and	

2) a	quasi-experimental impact study to	estimate the effects of selected NFWS/SIF-funded

programs	 on	 the	 labor	 market	 outcomes of individuals who entered those	 programs

during the	SIF funding period.

This report presents the findings of the quasi-experimental impact study for six NFWS/SIF-	

funded programs	 –	 three based in Ohio (Healthcare Careers Collaborative,	 Advanced

Manufacturing Partnership, and Construction Sector	Partnership)	and three based in Wisconsin

(Wisconsin	Regional	Training Partnership Manufacturing Pathway, Wisconsin	Regional	Training

Partnership Construction Pathway, and Milwaukee Area Health	 Alliance).	 These six	 programs

provided training and other services	 to	 individuals interested in obtaining jobs and advancing

their	 careers in healthcare, advanced manufacturing, and construction. The study relies on	 a	

quasi-experimental approach to	 estimate	 program impacts for participants who were

unemployed at program	entry	by: (1) using the propensity score matching method to identify
IMPAQ International, LLC Page 2 NFWS/SIF Quasi-Experimental Impact Study

matched comparison groups consisting	 	 of unemployed non-participants	 	 who were

observationally equivalent to unemployed NFWS/SIF participants,	 and	(2) estimating program

impacts	by comparing the	labor	market outcomes of NFWS/SIF participants with the outcomes

of unemployed	individuals in	the	matched comparison	groups.

A. Program Descriptions

A.1 Ohio-based Programs

The three Ohio-based programs were supported by	 the	Partners for	a	Competitive Workforce

collaborative,	a	regional	partnership in the Cincinnati	area.	This collaborative	used NFWS funds,

combined with funds from numerous private	 and	 public	 organizations,	 to	 help the three

partnerships design and implement sustainable workforce strategies	 to	 promote the

employment of low-income	individuals in their	respective focus industries. Below is summary of

each of	the	three programs.

Health Careers Collaborative of Greater Cincinnati. This	 program	 focused on helping

unemployed workers obtain the skills needed	to	access healthcare jobs. The program provided

a	wide	range	of services, including job readiness training,	assistance	in obtaining employability

and	training credentials, industry-focused training,	and	job search assistance. During the study

period	(January	2010	–	February	2012),	the	program served 992 unemployed participants, who

were primarily women,	 had	more than	a	high school education, were under	 age	35,	 and	had

prior work experience.

Advanced Manufacturing Partnership. This program focused on promoting the employment

and	 career advancement of low-skill workers in	 advanced	 manufacturing jobs. The	 program

used	 an	 incremental approach in promoting	 participants’ employment and educational

advancement, which included job readiness training, assistance in obtaining employability

credentials, enrollment in college coursework and specialized apprenticeships, and receipt of

job		search		assistance.		During		the		study		period,		the		program		recruited		684		unemployed
IMPAQ International, LLC Page 3 NFWS/SIF Quasi-Experimental Impact Study

participants	–	the	majority of participants were men, were nonwhite, had no more than	a	high

school education, were	under	age	35, and had	limited work experience.

Construction Sector Partnership. This program focused on	 creating	 career pathways in

construction for low-skill workers. The program’s pathways model was	 based	 on enrolling

participants in pre-apprenticeship	 programs	 to help them obtain construction skills	 and	

providing job search services. During	the	study period, the program recruited 379 unemployed

participants	–	the	majority of participants were men, were nonwhite, with no more than	a	high

school education, under	age	35, and with	limited work experience.

A.2 Wisconsin-BasedPrograms

The three Wisconsin-based	 programs	 were supported by	 the	 Milwaukee Area Workforce

Alliance,	which operates in the	Milwaukee	area. This collaborative used NFWS	funds,	combined

with funds from private	and	public organizations,	 to	support training programs	 that	help low-	

income workers	 to	 obtain in-demand jobs in construction, manufacturing, and	 	 healthcare.

Below is summary of the three	programs.	

Wisconsin Regional Training Partnership (WRTP) Construction Pathways and Manufacturing

Pathways. WRTP focused on brokering relations between employers, unions, and workers to

promote employment of low-skill workers in construction and manufacturing,	and	to	help	local	

employers	 to	 recruit	 a	 diversified and qualified workforce. The partnership supported two

programs	 –	 WRTP	 Construction Pathways	 and	 WRTP	 Manufacturing Pathways	 –	 that	 offered

participants pre-apprenticeship training, assistance in obtaining occupational credentials,

career advancement	training,	and	job search services. During	the	study period, the construction

and	manufacturing programs served 1,103 and 88 unemployed participants, respectively	–	the

majority	were men,	were	nonwhite,	and with no more than	a	high school	education.	

Milwaukee Area Healthcare	 Alliance.	 This program worked with healthcare employers to

identify sought-after skills	 and	 to	 provide training	 and	 other services to low-skill workers	 to	
IMPAQ International, LLC Page 4 NFWS/SIF Quasi-Experimental Impact Study

meet those needs.	 It	 provided participants with occupational training	 to	 obtain in-demand	

healthcare jobs (as identified by their employer partners), on-the-job training	 to	help advance

their	 careers, and job search services to identify suitable jobs. During the study period, the

program	served 306 unemployed workers, the majority were women, black,	and	under	age	35.

B. Impact	Study Results

The objective of the study is to examine	 the	 impacts of each program on the labor market

outcomes of unemployed participants, including: employment, employment in	 the	 program’s

focus industry, job retention, and earnings. IMPAQ developed	 a	 quasi-experimental approach

based	on the propensity score matching method, which involved	the	following steps:

§ Step 1: Merge data –	 Merge	 NFWS/SIF	 data	 on unemployed program participants

(treatment group) with	 state	 Employer Service	 data	 on unemployed workers who

sought state services during	the	same period	as	NFWS/SIF-funded program participants

(comparison group).

§ Step 2: Produce propensity score	–	Apply	a	logit model on the merged data	to	estimate

the	 probability of NFWS/SIF	 program	 participation based on individual characteristics

and	employment history,	and	use	the	results	to	produce	the propensity score (predicted

probability of NFWS/SIF participation) for treatment and comparison cases.

§ Step 3: Use propensity score	to	construct sample weight	–	Weigh each comparison case

by the odds ratio of the predicted	 propensity	score, so that the weighted comparison

sample matches the characteristics distribution of the treatment sample.

§ Step 4: Compare treatment and weighted matched comparison	 sample	 –	 Conduct

statistical	 tests	 to	 verify	 that	the treatment and matched	 comparison	 groups	 are	truly

matched in their characteristics.

The	above	approach was successfully implemented, producing	a	matched	comparison	group for

each NFWS/SIF-funded	program.	It	consisted	of non-participants who sought state employment

services during the same period,	had	similar characteristics, and resided in the same area as
IMPAQ International, LLC Page 5 NFWS/SIF Quasi-Experimental Impact Study

unemployed NFWS/SIF participants. State Unemployment Insurance (UI) wage records	 data	

were used to produce common labor market outcomes for treatment and matched comparison

cases	in	the six-quarter	period following	program	entry,	including: employment, employment in

the	program’s focus industry, job retention, and earnings.	Program	impacts were estimated by

comparing	 the	mean outcomes between	 the	treatment and the matched	 comparison	 group;

results for each program	are	summarized below.

B.1 Ohio-basedNFWS/SIF-funded programs

Health Careers Collaborative of Greater Cincinnati

§ The program had	large	effects on employment.	In	the	six quarters after	program	entry,

participants’ employment	 rates	 were 57.6-64.8 percent, exceeding the	 rates	 of the

matchedcomparisongroupby14.1-17.0percentagepoints(29-37percent).

§ The program was effective in helping participants	to	obtain healthcare jobs.	In	the	four

quarters after entry, 33.4-34.3 percent of participants were employed in healthcare,

exceeding the healthcare employment	 rates	 of	 the	 matched comparison group by

24.0-25.3 percentage points (23.3-30.4 percent).

§ The program was very	effective	in improving participant job retention	rates.	About 35.2

percent of participants found employment in quarter	1	and remained employed in each

of	 the	six quarters after	entry,	compared	 to	only 22.4 percent of matched comparison

group members	–	a	12.8 percentage point (57 percent) difference.

§ The	 program	 had large effects on earnings.	 In	 the six-quarter follow-up period,

participants	 had	 $5,517 (52 percent) higher	 earnings	 than those in	 the	 matched

comparison	group.

Advanced Manufacturing Partnership

§ The	program	had	large	effects on employment. In the six quarters after entry, 42.2-52.6

percent of participants were employed, exceeding	 the	 employment rates of matched

comparison	groupmembersby8.2-14.3percentagepoints(24-38percent).
IMPAQ International, LLC Page 6 NFWS/SIF Quasi-Experimental Impact Study

§ The	 program	 had modest effects on manufacturing employment; no more than 4.2

percent of participants were employed in manufacturing in the four quarters after

entry.	

§ The program had positive effects on job retention. About 19.9 percent of participants

found	a	job in quarter	1	and	remained employed in each of	the	six	quarters	after entry,

compared	 to	 15.9 percent of matched comparison group members,	 a	 25 percent

difference.

§ The program had positive effects on earnings.	 In	 the	 six-quarter	 follow-up period,

participants	 had	 $2,635 (31 percent) higher	 earnings	 than those in the matched

comparison	group. These effects were lower than those of the Health Careers program.

Construction Sector Partnership

§ The program	 had	 modest effects on employment. Between 38.8-45.9 percent of

participants were employed in quarters	 1-6	 after	 program	 entry, which exceeded the

employment	 rates	 of matched comparison group members by 3.2-6.1 percentage

points (9-16 percent).

§ The	 program	 had	 modest effects on employment in construction; no more than 4.8

percent of participants were employed in construction in	the	four quarters after entry.

§ The	program	had no effects on job retention and modest effects on earnings.

B.2 Wisconsin-basedNFWS/SIF-funded programs

WRTP Construction Pathways

§ The program had	 large	 effects on employment.	 In	 the	six quarters	 after	 program	 entry,

67.2-72.7 percent of participants were employed, exceeding the employment	 rates	 of

matchedcomparisongroupmembersby8.8-21.7percentagepoints(14-43percent).

§ The	program	was not effective in helping participants	to	obtain construction jobs.	In	the six

quarters after entry, fewer than three percent of participants were in	construction jobs.

§ The	program had	large	positive effects	on job retention. About 44.5 percent of participants

found employment in quarter	1	and	remained employed in each of the six	quarters	after
IMPAQ International, LLC Page 7 NFWS/SIF Quasi-Experimental Impact Study

entry,	 compared to only 29.4 percent of matched comparison group members,	 a	 15.1

percentage point (51 percent) difference.

§ The	program	had	large effects on earnings.	In	the	six-quarter	follow-up period, participants

had	$11,237 (56 percent) higher earnings than the matched comparison group.

WRTP Manufacturing Pathways

§ The program had	 large	 effects on employment.	 In	 the	six quarters after	 program	 entry,

66.3-72.1 percent of participants were employed, exceeding the employment	 rates	 of

matchedcomparisongroupmembersby11.1-28.6percentagepoints(21-68percent).

§ The program was effective in helping participants	to	obtain manufacturing jobs.	In	the six

quarters after entry, 32.6-40.7 percent of participants were employed in	manufacturing,

exceeding	 the	 rates	 of	 the	 matched comparison group by 23.7-29.6 percentage points

(202-535percent).

§ The	program	had large effects on job retention. About 43.0 percent of participants	found	a	

job in quarter	1	and	remained employed in each of the six quarters after	entry,	compared

to	19.8 percent of matched comparison group members,	a	118 percent difference.

§ The	program	had	large effects on earnings.	In	the	six-quarter	follow-up period, participants

had	$16,661 (134 percent) higher earnings than the matched comparison group.

Milwaukee Healthcare Alliance

§ The program	 had	 substantial effects on employment.	 In	 the	 six quarters after program

entry,	57.1-77.0 percent of participants were	employed, exceeding	the	employment rates

ofmatchedcomparisongroupmembersby16.5-19.4percentagepoints(28-40percent).

§ The program was effective in helping participants to obtain healthcare jobs.	 In	 the six

quarters after entry, 27.9-47.4 percent of participants were employed in healthcare,

exceeding the healthcare employment rates of	 the	 matched comparison group by

18.1-29.6 percentage	points (155-196	percent).

§ The	program	had	large	effects	on job retention. About 40.1	percent of participants	found	a	

job in quarter	1	and remained employed in each of	the	six	quarters	after	entry,	compared
IMPAQ International, LLC Page 8 NFWS/SIF Quasi-Experimental Impact Study

to	22.3 percent of matched	comparison group members, an 80 percent difference.

§ The	 program		 had		positive		 effects		on	 earnings.		In		 the		six-quarter		follow-up		period,

participants had $5,418 (55	percent) higher earnings	than	the	matched	comparison group.

C. Conclusions

The results of	 the	 	quasi-experimental impact study provide promising evidence	 	 about the

effectiveness of NFWS/SIF-funded workforce training	 programs	 in healthcare, manufacturing,

and	construction. Results for the Ohio-based	programs show that the Health Careers program

was very effective in placing participants in healthcare jobs, leading to positive effects on

overall	 employment, job retention,	 and	 earnings. The Advanced Manufacturing and

Construction Partnership	programs	were not effective in	placing	participants in	their	respective

focus industries. The Advanced Manufacturing program had positive effects on employment,

job retention, and earnings, which were lower than the effects of the Health Careers programs,

while the Construction Partnership program	had	modest effects on employment	and	earnings.

All three Wisconsin-based programs had important positive effects on participants’ labor

market outcomes. The	 WRTP	 Manufacturing	 program	 was the most effective of the six

programs,	since it placed participants in manufacturing jobs, leading	 to	substantial effects on

overall	 employment, job retention, and earnings. The Healthcare Alliance program was

effective in helping participants find jobs in healthcare, leading to improved employment, job

retention, and earnings. The program’s effects were	 similar	 to	the effects of the Ohio-based	

Health Careers program. The	 WRTP	 Construction program was ineffective in	 helping

participants obtain jobs in construction, but	 –	unlike	 the	Ohio-based Construction Partnership

program	–	it	led	to	positive effects on employment, job retention, and earnings.

Overall, these results provide important insights on the effectiveness of NFWS/SIF-funded

programs. Programs focusing on the healthcare industry can help participants obtain jobs in

the	 industry and improve their labor market outcomes. While we get mixed results on

manufacturing	programs’	effectiveness in helping participants obtain jobs	in the industry, these
IMPAQ International, LLC Page ix NFWS/SIF Quasi-Experimental Impact Study

programs	 often lead	 to	 substantively important improvements	 	 in overall employment, job

retention, and earnings. Finally, it appears that construction programs	 are	 unlikely	 to	 help

participants obtain construction jobs, and are likely	 to	 have lower	 overall	 impacts on

participants’ labor market outcomesthan healthcare and manufacturing programs.
IMPAQ International, LLC Page 1 NFWS/SIF Quasi-Experimental Impact Study

1. Introduction

The National Fund for Workforce Solutions (NFWS),	 a	 collaboration of national	 foundations,

was	established	in 2007	 to	promote the employment and career advancement of low-income

individuals and ensure that employers obtain the skilled workforce needed	 to	 compete in the

modern economy. To achieve these	 goals,	 NFWS	 invests	 in	 Regional	 Funding Collaboratives,

which	 match	NFWS contributions with public and private funding	 to	support	 local	 workforce

partnerships composed of employers, community-based organizations, and service providers.

These partnerships use the funding	to	identify employers’ workforce needs in their	local	areas

and	then design and administer training programs	and	other services	to	low-income workers to

prepare	them	to meet those needs.

In	 2010, NFWS was	 awarded	 a	 two-year, $7.7 million Social Innovation Fund (SIF) grant to

expand	 their model. NFWS used the grant to support existing collaboratives	 committed	 to	

extend	the operations	of their programs and	to	help develop	new collaboratives.	In	2011, NFWS

selected IMPAQ International, LLC (IMPAQ)	 to	 evaluate	 the	 30 NFWS/SIF-funded workforce

partnerships supported by eight scale-up collaboratives. The evaluation consisted of two

components:

1) An outcomes study of	 all	 30	 scale-up programs	 to	 examine	 program participation,

services provided, and participant outcomes in	 the	 period January 2010	 –	 February

2012. This study was completed in	February 2013 (Michaelides et al.,	2013).

2) A	 quasi-experimental study of	 the	 impacts of selected NFWS/SIF-funded workforce

partnership programs on the labor market outcomes of individuals who	entered	these

programs	from January 2010 through February 2012.

This report presents the findings of	 the	 quasi-experimental impact study for six workforce

partnership programs, focusing on helping unemployed workers	 to	 obtain jobs	 and	 access

better careers	in healthcare,	manufacturing, and construction. Three of the	programs	examined

in this study	–	the Healthcare Careers Collaborative, the Advanced Manufacturing Partnership,

and	 the	 Construction Sector Partnership	 –	 are supported by the Ohio-based Partners for	 a	
IMPAQ International, LLC Page 2 NFWS/SIF Quasi-Experimental Impact Study

Competitive Workforce Collaborative (formerly called	 Greater	 Cincinnati Workforce Network).

The other three study programs	 –	the	WRTP	Manufacturing Pathway, the	WRTP	Construction

Pathway, and the Milwaukee Area Health Alliance	 –	 are supported by the Wisconsin-based

Milwaukee Area Workforce	Alliance.	These six programs were	among	the first	to	receive NFWS

support and	leverage	NFWS funds with funds from	other	sources	to	develop training programs

focusing on in-demand jobs in healthcare, manufacturing,	 and	 construction.	 In	 the	 period of

January	 2010 through February 2012, the six programs provided training, employment, and

other services	to	3,552 unemployed workers.

This study	 examines	 the	 impact	 of the six NFWS/SIF-funded	 programs	 on the	 labor	 market

outcomes of participants who were unemployed	at	program entry. To estimate the programs’

impacts,	 a	 quasi-experimental approach was implemented.	 It	 relied on NFWS/SIF-funded

program	data and state	administrative	data	to:	(1)	apply	the	propensity	score matching method

to	 identify matched comparison groups consisting of unemployed non-participants	 who

registered	 to	 receive	 state	 employment services	 and	 were observationally equivalent to

NFWS/SIF-funded program participants, and (2) estimate program impacts by comparing the

labor	market outcomes of unemployed NFWS/SIF-funded program participants (employment,

employment in	a	program’s focus industry, job retention, and earnings) with the outcomes of

the	matched comparison group. Essentially, this study	answers the question: Are the NFWS/SIF-	

funded programs	 effective	 in helping unemployed workers achieve	 better	 labor market

outcomes than they would achieve if they were left	 to	 find jobs without access	 to	 these

programs?	

The remainder of the report is organized	as	follows. Section	 2	provides	an	overview of NFWS

and	a	discussion of the design of the quasi-experimental impact study. Section	3	describes the

six partnership programs, including partnership objectives, program services provided	 to	their

target	 populations, and	 the	 characteristics of	 all	 unemployed individuals who participated in

each program during the study period. Section	4	presents	the	results of the quasi-experimental

impact	study for	each	program.	Section	5	provides	our conclusions	based	on	the	study findings.
IMPAQ International, LLC Page 3 NFWS/SIF Quasi-Experimental Impact Study

2. Background

2.1 The National Fund for Workforce Solutions

NFWS	 has	 created	 a	 model of developing sustainable workforce strategies, based on the

principle that	local	entities	are	best suited	to	identify	local	workforce needs	and	to	leverage	the

funding needed	 to	develop workforce	programs	 to	address those needs. Based on	 this	model,

NFWS invests in	 regional	collaboratives responsible for: (1) matching NFWS contributions with

public and private funding, and (2) using	 the	 leveraged	 funding to build	 local	 workforce

partnershipsthat are developing or implementing promising strategies.

Local workforce partnerships are typically composed of employers,	 community-based	

organizations, and training/service providers.	These partnerships identify employers’	workforce

needs in their	local	areas,	often in specific industries, and fund the training	and	provide other

resources	 to	 low-skill workers	 to	prepare them	 to	meet those needs. By	 January	2010, NFWS

was funding eight regional collaboratives that supported 30	active	workforce programs:1

§ Pennsylvania Fund for Workforce Solutions (Pennsylvania,	5	programs)	

§ Philadelphia Job Opportunity Investment Network (Pennsylvania,	4	 programs)

§ Milwaukee Area Workforce Funding Alliance (Wisconsin,	3	programs)

§ Workforce Central Funders Collaborative (Wisconsin, 3	programs)

§ Partnersfor	a	Competitive Workforce (Ohio,	3	programs)

§ Baltimore	Workforce Funders Collaborative (Maryland, 3	programs)

§ SkillWorks(Massachusetts,	4	programs)

§ SkillUp Washington (Washington,	5	programs)	

Of	 the	 30 partnership	 programs,	 26 focused on helping workers	 to	 obtain skills needed to

access		careers		in		a		specific		industry,		while		the		remaining		four	 provided		services		to		help

1	NFWS was funding 14 additional collaboratives that supported partnerships, which, at that time, had not started

implementing their programs.
IMPAQ International, LLC Page 4 NFWS/SIF Quasi-Experimental Impact Study

participants obtain employment in	a	broad	array	of industries. The majority of	the	26 industry-	

specific programs emphasized three sectors: healthcare (eight programs),	 manufacturing	(five

programs),	 and construction (four programs). Other focus industries included hospitality,

utilities, landscaping, transportation, information technology, financial services, and

biotechnology.2

In	 2010, SIF,	 an	 initiative of	 the Corporation for National and Community	 	 Service (CNCS),

awarded	NFWS	a	two-year, $7.7 million grant	to	expand	its	model. These funds extended	grants

to	 the	 eight collaboratives listed above, which committed	 to	 use the funding	 to	 expand the

operations of their 30 workforce partnership	 programs	 through service enhancement and

increasing recruitment. The remaining funds supported	 the	 efforts of collaboratives that had

not started implementing their	 programs.	 The funds	 also	 supported the creation of

collaboratives. These collaboratives were expected to use the funding	 to	adopt successful and

sustainable workforce strategies	to	address employers’ needs in their region.

Due in	 part	 to	the SIF	 grant,	NFWS	 has	expanded substantially since 2010. It	 has	grown from

eight collaboratives that supported 30 workforce partnership programs in 2010	 to	 29

collaboratives that supported 96	 programs	 in 2012.3	 These 96 programs focused	 mainly	 on

three	sectors	–	healthcare (38 programs), manufacturing	(17),	and	construction (16). According

to	NFWS, through 2012, NFWS partnerships	had	received $41 million in federal funds, and	had	

leveraged	approximately $192 million from 476 private sources.4	NFWS also	reports	that these

funds	 engaged	 4,064 employers and provided training	 and	 related services	 to	 42,299

individuals.

2.2 Quasi-Experimental Impact Study Design

One of	the	key components of	IMPAQ’s	evaluation was	to	examine	the	impacts	of NFWS/SIF-	

2	For more details, see Michaelides et al.	(2013). 3	 Source: Building on Success: Five Years of Impact 2007-2012. National Fund for Workforce Solutions, 2013.

(http://www.nfwsolutions.org/tools/building-success). 4	 Source: Building on Success: Five Years of Impact 2007-2012. National Fund for Workforce Solutions, 2013.

(http://www.nfwsolutions.org/tools/building-success).
IMPAQ International, LLC Page 5 NFWS/SIF Quasi-Experimental Impact Study

funded programs5	on the	labor	market	outcomes	of individuals who entered	those	programs in

the	period January 2010	 –	 February	 2012. IMPAQ developed	 a	quasi-experimental evaluation

approach, in which program	impacts	were estimated by	comparing	the	outcomes of NFWS/SIF

participants (treatment group) with the outcomes of non-participants	 	 who	 	 were

observationally equivalent	to	participants (matched comparison group). This approach included

the	following components:

§ Apply matching methods to construct matched comparison groups for NFWS/SIF

participants, using NFWS/SIF-funded program data merged	 to	administrative	data	from

the	state in	which the program	operates.	

§ Use	 state	 administrative	 data	 to	 construct common labor market outcomes for

treatment and matched	comparison	group casesin the	six-quarter	follow-up period.

§ Estimate program impacts by comparing the labor market outcomes between the

treatment and the matched comparison group.

Most NFWS/SIF program participants were unemployed workers, but	 many	 programs	 also

served workers who were employed	 at	 program	 entry.	 However, due	 to	 lack	 of appropriate

data	for	constructing matched comparison groups for employed participants, the study	focused

on estimating impacts only for unemployed workers. Below, we discuss	 the	 evaluation

approach, including key research questions, the process for selecting programs for inclusion in

the	study,	data	sources,	and	the methodology for the quasi-experimental impact study.

2.2.1 Key Research Questions

The objective of the NFWS/SIF quasi-experimental impact study was	 to	 address key research

questions about	 the	efficacy of	 the	NFWS/SIF-funded	programs	in improving	 the	labor market

outcomes of unemployed participants:

§ Were these	 programs	 effective in helping unemployed participants to obtain

employment after program entry?

5			Hereafter,		the term “NFWS/SIF programs”	 refers to	 the 30	 workforce partnership		programs active at	 the

beginning of 2010 and received SIF funding to expand their operations.
IMPAQ International, LLC Page 6 NFWS/SIF Quasi-Experimental Impact Study

§ Did these	 programs	 help unemployed participants to obtain jobs in	 the	 industries

related	to	the	training	received?

§ Were	 these	 programs successful in helping unemployed participants	 to	 obtain

employment and remain employed for longer periods than they would	 have	 in the

absence of the	program?	

§ Were these	 programs	 effective in helping unemployed participants to	 achieve	 higher

earnings	than they would	have	in the	absence of	the	program?	

Addressing these questions provides substantial insight into the effectiveness of NFWS/SIF-	

funded	 programs	 in promoting	 the	 employment and	 career	 advancement of low-income

unemployed individuals. Further,	 this	study provides information on whether certain	 types	of

programs	–	based on the focus industry and services provided	–	are	likely	to	be more effective

than others in improving participant	labor	market outcomes.

2.2.2 Program Selection

During the study period, NFWS funded eight collaboratives, which supported 30 partnerships

with	active	workforce programs. Of these, 17 programs provided training	and	other services to

low-income workers	 to	 access	 jobs in healthcare, manufacturing, and construction, while the

remaining	 programs	 focused on other industries. An	 important	 aspect	 of implementing the

impact	study was	to	identify which programs were suitable for inclusion in the study. Selection

criteria	included:

§ Exemplary Implementation of the NFWS Model –	 Select sites that successfully

implemented the NFWS model, based		on: 1) their effectiveness in matching NFWS

funds, 2) the strength of	their	partnership with employers and service providers, and 3)

the	 types	 and	 quality of training/services they provided. This criterion ensured that

impact	 findings would represent the programs effectively implementing the NFWS

model.

§ Number		of	 Participants		–		Select		programs		that		served		at		least		approximately		100
IMPAQ International, LLC Page 7 NFWS/SIF Quasi-Experimental Impact Study

unemployed participants,	to	ensure sufficiently	large	samples of participants so that the

study could detect substantively meaningful impacts.

§ Data Availability –	As outlined in Section 2.2.3, below, the study relied on two types of

data:	 (1) NFWS/SIF-funded program data, with information on participants’

characteristics	and	personal identifiers to link participants	 to	state administrative data,

and	 (2) state administrative	 data,	 with information on	 the	 characteristics of non-	

participants	 and	 on the	 labor	 market outcomes of both participants	 and	 non-	

participants.	 It	 was, therefore, critical that programs included in the study were

collecting	 high-quality data on participants (including personal identifiers)	 and	 were

operating in states willing	 to	provide their administrative labor market data	 to	support

the	study.

Nearly	 all	30 NFWS/SIF partnership	 programs	successfully	 implemented	 the	NFWS model and

collected participants’ data that provided the information listed	above.6	However, 14 programs

served much fewer than 100 participants during	 the	study period, so were excluded from the

study because they were	 too	small to	 allow	 the	 identification of matched comparison groups

and	to	produce meaningful impact estimates.	In	addition, although the five SkillUp Washington

programs	served 1,285 participants,	a	sample size that was sufficient	to	implement	the	impact

study,	 they	were excluded because	 they	 could not provide participants’ identifiers, which are

critical for state administrative	data.	

Using	 those	 criteria, only 11	 remaining	programs were deemed eligible for the study.	Table	 1

presents	a	summary of these programs, including	their	focus industry and the total participants

served during	the	study period.

6	For	a	detailed discussion of each program and their data availability, see Michaelides et al.	(2013).
IMPAQ International, LLC Page 8 NFWS/SIF Quasi-Experimental Impact Study

Table 1: NFWS/SIF-funded Programs Eligible for the Quasi-experimental Impact Study

Number of Unemployed Collaborative/PartnershipName FocusIndustry Participants

Partnersfor	a	Competitive Workforce

Health Careers Collaborative Healthcare 992

Advanced Manufacturing Partnership Manufacturing 684

Construction Sector Partnership Construction 379

Philadelphia Job Opportunity Investment Network

Pennsylvania Partnership for Direct Care Workers Healthcare 172

Pennsylvania Fund for Workforce Solutions

Lancaster-SACAPartnership VariousIndustries 315

Building Services, Labor Management Clearinghouse 111 Hospitality

Keystone Utilities Partnership Utilities 136

Reading Regional Construction Partnership Construction 191

Milwaukee Area Workforce Funding Alliance

WRTP Construction Pathways Construction 1,103

WRTP Manufacturing Pathways Manufacturing 88

Milwaukee Area Healthcare Alliance Healthcare 306

These 11 programs were supported by four collaboratives, operating in three states	 –	 Ohio

(three	programs),	Pennsylvania (five	programs),	and	Wisconsin	(three	programs).	Table	1	shows

that eight of these 11	 programs	 focused on three industries: healthcare	 (three	 programs),

manufacturing (two programs),	and	construction	(three	programs). During	the	study period, the

11 programs served 4,477 unemployed participants.

A	 critical	 aspect	 of implementing the quasi-experimental impact study was	 the	 availability of

state	administrative data	 to	 identify appropriate matched comparison groups	 and	to produce

labor	market outcome measures for both treatment and matched	comparison	group cases (see

Section 2.2.3). To date,	 IMPAQ	 and NFWS	 have	secured state administrative data from Ohio

and	Wisconsin, but have not been	able	to	secure Pennsylvania data.	In	 the	remainder of this
IMPAQ International, LLC Page 9 NFWS/SIF Quasi-Experimental Impact Study

report, we present the impact results for	the	three	Ohio and	the	three Wisconsin	programs.	

We believe that	 the	impact study for the six programs in Ohio and Wisconsin can be used to

draw	 inferences about the efficacy of	 the	remaining programs supported by NFWS/SIF. First,

the	 three programs focused on healthcare, manufacturing, and construction	 –	 the focus

industries of most NFWS/SIF-funded	programs.	Second,	the	six	programs	offered	a	wide range

of training and employment services	to	participants, which	are	comparable	to	the training and

services provided by nearly	all	NFWS/SIF-funded programs.7

2.2.3 Data Sources

The quasi-experimental impact study	relies on	participants’	data gathered by each	program	and

on state administrative data. Below, we provide an overview of	these	data	sources.

NFWS/SIF-funded program Data. The NFWS/SIF-funded program data provide information on

all	 unemployed individuals who entered NFWS/SIF-funded programs from January 2010

through February 2012. The data include the following:

§ Participants’ socioeconomic characteristics	 at	 program entry, including gender,	 race,

age,	education,	date of program	entry, and	the	Local Workforce	Investment Area (LWIA)

area	in which the participant resided.

§ Personal	identifiers to match	to	state administrative	data.	

Also,	 the	 data of	 the	 three		Ohio programs provided information on	 the	 types of services

received by participants. This information was	 unavailable	for	 the	three Wisconsin programs.

Notably, the data provided by the six programs were mostly complete with very few cases of

missing data. In particular,	 the	 three Ohio programs had 2,055 participants, with only two

missing	 values	for	 race,	23 missing values for age,	 and	25 missing values for area. The three

Wisconsin		programs		had		no	 missing	 values		on		individual		characteristics.				In		the		analyses	

7	 For	 a	 discussion of the focus industry, services provided, and participant characteristics for each NFWS/SIF

program, see Michaelides et al.	(2013).
IMPAQ International, LLC Page 10 NFWS/SIF Quasi-Experimental Impact Study

described below, no attempts were	made	to impute	missing values;	instead, we	treated	missing

values	as	a	separate category.

Employment Service (ES) Data. ES	 data	 provide information on	 all	 unemployed workers who

sought employment and training services with the state employment exchange	 agency	 from

January	2010 through February 2012 and were residing in the same LWIAs	as	NFWS/SIF-funded

program	participants.8	In	particular,	 the	data provide information similar	 to	 those	reported in

the	NFWS/SIF data, including socioeconomic characteristics (gender,	race,	age,	education, date

of	 program	entry, and LWIA of residence) and personal identifiers (name, address, and Social

Security number). IMPAQ obtained	 a	 de-identified dataset for each state that provided

information on	all	ES participants in the period January 2010	–	February 2012. The Ohio data

were provided by	the	Ohio State University’s Center for Human Resource	Research	(CHRR);	the

Wisconsin data were provided by the Wisconsin Department of Workforce	Development.	 The

ES	data	used in this study were mostly complete	–	the	Ohio ES data were missing	race	for 12

percent of	 the	 cases and	 age	 for less than	 1	 percent of the cases; no missing	 values	 in the

Wisconsin ES	 data.	In	 our	 analyses,	 we treated missing values in	 the	 Ohio	 data	 as	separate

categories.	

UI Wage Records. UI	 Wage	 Records	 provide quarterly earnings information on	 all	 NFWS/SIF

participants who entered the program during the study period and on all ES non-participants	

who sought state services during that period.9	 The data include	 the	 year/quarter of

employment, total earnings,	employer industry,	and	identifiers	for matching with NFWS/SIF and

ES	data.	IMPAQ	has	obtained de-identified Ohio	and	Wisconsin UI	Wage	Records	from the first

quarter of 2009 through the fourth quarter of 2012. These data were used	 to	 construct

employment outcomes for six quarters after program	entry	for NFWS/SIF	and	ES participants

8	ES data typically include	information on unemployed workers who applied for UI benefits	and/or registered with

the state’s labor exchange agency to find	a	job. 9	UI Wage	Records include earnings from jobs	with employers located within the state and covered by the state UI

system. The data do not report individual earnings earned within the state (such as from Federal government jobs,

including the military, and self-employment) or individual earningsfrom out-of-state employers.

IMPAQ International, LLC Page 12 NFWS/SIF Quasi-Experimental Impact Study

participants can be	 an	 appropriate matched comparison group for estimating	 the	 programs’

impacts.

Matching methods provide credible impact estimates when	 the	 data	 include	 large	samples of

non-participants	 and	 matching	 is	 based	 on rich information on	 participants’	 and	 non-	

participants’ characteristics and prior work history.	 In	 particular, it is critically important that

the	set of covariates include	 demographic	 characteristics and measures of prior employment

experience,	 and	 that the participant and comparison groups include individuals in	 the	same

local	labor	market. Outcome measures need	to	be consistent for treated	and	untreated	cases;	if

possible,	 they	 should be derived from the same or strictly comparable sources. These

conditions are satisfied in the analyses performed here (Mueser et al., 2007;	 Heinrich	 et al.,

2013).

The treatment group in this study includes unemployed individuals who participated in each of

the	six NFWS/SIF-funded	programs	during the study period. To construct matched comparison

groups for each program, we relied on ES	 data,	 which provide rich information on the

characteristics of unemployed individuals who sought state employment services, combined

with UI	 Wage	 Records, which provide detailed information on	 individuals’ employment

histories. ES	 data	are	particularly appropriate	 to	use for	 this	purpose. First, they include large

samples of unemployed non-participants who: (1) were residing in	 the	 same LWIAs	 as	

NFWS/SIF participants, and (2) sought state employment and training	services	at	the	same	time

the	 NFWS/SIF participants entered	 their	 programs. Second,	 the	 ES data report similar

information	 to	 that reported in the NFWS/SIF-funded	 program	 data,	 which facilitates the

matching	process. These	properties ensure that, if matching is done correctly, we can identify	a	

matched comparison sample in the same labor market and	 nearly	 identical	 to	the treatment

sample in personal characteristics and prior employment history.

To construct matched comparison groups in this study, we use	 the	propensity score matching

(PSM) method. Since	the	six NFWS/SIF-funded programs included in	this	study differed in their



IMPAQ International, LLC Page 16 NFWS/SIF Quasi-Experimental Impact Study

This approach estimates	 the	 impact	 of each of the six programs on	 the	 average	 program

participant. There is, of course, variation in the	 program’s effect	 across	 participants,	 and	the

estimate	is subject	to	uncertainty because of random	factors	that	may	affect individual program

success. For	that	reason,	it is	important	to	calculate	the	statistical significance of	the	estimates,

using standard errors	that capture statistical factors that influence	a	program’s success. For	the

type	 of matching process used in	 this	study, bootstrapping is	 the	 best method	 to	 calculate

standard errors that capture such statistical	 factors.13		Bootstrap standard errors were used to

calculate	t-tests	to	assess whether	the	estimated	program	impacts	are	statistically significant.

The analyses below report	tests	of statistical significance for	a	large number of program effect

estimates.	 If	true program effects were zero, we would expect	 to	 nonetheless find statistical

significance for some	 estimates	 due	 to	 chance. There	 are	many approaches	 to	 guard	 against

making	 false	 inferences from results produced by	 multiple	 tests	 (e.g., Holm–Bonferroni). We

have	 chosen not to pursue such methods because	 many	 of our tests	 are	 related	 –	 both

statistically and substantively	 –	 and	 such approaches	 fail	 to consider	 this.	 The analyses we

report consider programs’ effects on employment, industry of employment, employment

stability,	 and	 earnings, each measured for up	 to	 six quarters following	 initial	 program

participation, and providing for up	 to	two dozen	estimates	for each	program.	The hypotheses

considered are not tied to	a	single estimate; rather, inferences are drawn from the patterns of

results	 across	 these outcomes	 and	 over time. As	 a	 practical matter, estimated effects that

border on statistical significance	 are	 often so small	 that	 they	 are	 of little substantive

importance, whereas the important conclusions depend on estimates where sampling error is

relatively	small	and statistical	significance	–	however gauged	–	is not open	to	question.

13 Bootstrapping involves re-estimating the impact multiple times,	using	samples formed by random sampling	with

replacement from the treatment and matched comparison samples. The bootstrap standard errors reported here

are based on 20 replications. For	a	discussion of bootstrapping in PSM models, see Caliendo and Kopeing (2008).
IMPAQ International, LLC Page 17 NFWS/SIF Quasi-Experimental Impact Study

3. Program Descriptions

3.1 Partnersfor	a	Competitive Workforce Collaborative

The Partners for	 a	 Competitive Workforce collaborative (originally named Greater Cincinnati

Workforce Network) was established in 2008	as	a	regional	partnership in	the	greater Cincinnati

area	in southwest Ohio,	and	consists of	area	Workforce Investment Boards (WIBs), employers,

community colleges,	 	 service	 	 providers, and other	 community-based	 organizations.	 	 The

collaborative	had	three	main	objectives:

1) Connect regional employers with qualified workers by coordinating the	efforts	of area

WIBs	to	create	a	common system, which can be used by workers	to	access information

on available jobs and by employers	to	access information on jobseekers.

2) Improve	 job readiness of low-income individuals by providing them with counseling

services	and	training to improve their core work competencies and	basic	skills.

3) Align training/education with employer needs by creating industry-driven training

programs	that create career pathways for low-skill workers.

To	achieve	its	objectives, the collaborative	leveraged	funding from foundations and numerous

private	and public organizations, including NFWS.	In	fact,	the	$450,000	start-up grant provided

by NFWS in 2008 was instrumental in	the	collaborative’s	inception.14 The leveraged funds were

invested in three workforce partnership programs	to	provide training	and other services	to	low-	

income individuals to help them access jobs in	 their	 respective focus industries: (1) Health

Careers Collaborative of Greater Cincinnati; (2)	 Advanced	 Manufacturing	 Partnership;	 and (3)

Construction Sector Partnership.

The	 collaborative		obtained		$22.5		million		from		public	 and		private		sources,	 including		the

$450,000 NFWS start-up	grant	in 2008 and $600,000 from	the	NFWS/SIF	grant	in 2010.15 These

14 See: Spence C., Elvery J., and Stacy L. (2009). Greater Cincinnati Workforce Network, Annual Evaluation Report. 15		The collaborative secured funds from multiple sources, including $4.5 million from philanthropic funds, $8

million from state and Federal grants, and $10 million in aligned training funds from the region's public workforce

system:							 http://www.nfwsolutions.org/regional-collaboratives/partners-for-competitive-workforce.
IMPAQ International, LLC Page 18 NFWS/SIF Quasi-Experimental Impact Study

funds were partly used to expand the operations of the three programs and enhance

recruitment. Detailed descriptions of the three programs	 are	 provided below, followed by an

overview of the characteristics of participants in each program during	the	study period and the

types	of services participants received.	A	summary of each	program	is provided	in Box 1.

3.1.1 Health Careers Collaborative of Greater Cincinnati

The Health Careers Collaborative of Greater Cincinnati (Health Careers)	was	established	in 2003

as	 a	 partnership between	 the	 Cincinnati	 State	 Technical	 and	 Community College,	 Great	Oaks

Institute of Technology,	 and	 the Cincinnati Children’s Hospital Medical	 Center	 in Greater

Cincinnati	 to	 address	 a	serious shortage of skilled healthcare workers. The partnership’s key

objective was	to	create	a	training	program	to provide workers with	the	skills needed to access

in-demand healthcare jobs.	 A	secondary objective was	 to	 increase	 the	 diversity of healthcare

workers by	recruiting	and training	minorities.	Over	time,	the partnership grew	to	involve many

healthcare employers, education and training providers, and	community-based	organizations.

The partnership leveraged the funds from the NFWS grant	 to	the Partners for	 a	 Competitive

Workforce with funding from numerous additional sources, including the U.S.	 Department	 of

Labor and the Bill	 and	 Melinda	 Gates	 Foundation.		These funds helped	 to	 implement their

healthcare-focused training program and	 to	 extend operations by expanding	 to	 other area

hospitals, recruiting additional educational institutions, and expanding the breadth of training

provided.
IMPAQ International, LLC Page 19 NFWS/SIF Quasi-Experimental Impact Study

BOX 1: SUMMARY OF	OHIO-BASED NFWS/SIF PROGRAMS

Health	Careers Collaborative	of Greater Cincinnati

Objective. A	workforce partnership focused on addressing shortages of skilled healthcare workers

by assisting: (1) low-skill unemployed workers to obtain the skills needed to access healthcare

jobs, and (2) incumbent entry-level healthcare workers to promote their careers.

Services. The program offered participants	 a	 wide range of	 	 services, including job readiness

training, assistance in obtaining employability and training credentials, industry-focused training,

and	job search	services.

Participants. The program recruited 992 unemployed participants during the study period

(January 2010	 –	 February 2012). The majority of unemployed participants were women	 (90	

percent), had more than	 a	 high	 	school	 	 education	 	(54 percent), and	 	 were under age 35	 (65	

percent). Large proportions were white (50 percent), had prior work experience (67 percent), and

had	prior work experience in	healthcare (28 percent).

Advanced	Manufacturing Partnership

Objective. A	 workforce partnership focusing on promoting the employment and career

advancement of low-skill workers	 –	 particularly unemployed workers	 –	 in advanced

manufacturing	jobs

Services. The program used an incremental approach	in promoting participants’ employment and

educational advancement. Upon program entry, participants were offered job readiness training,

followed by assistance in obtaining national employability credentials. Participants	 who

completed these steps could enroll in college coursework or engage in specialized

apprenticeships. Participants were also	offered	job	search	services.

Participants. The program recruited 684 unemployed participants during the study period. The

majority of unemployed participants were men (66 percent) and nonwhite (79 percent), had no

more than	 a	 high school education (69 percent), and were under age 35 (51 percent).	 A	 large

proportion had no prior work experience	 (47	 percent), and most had no experience in

manufacturing	(95 percent).

Construction	Sector Partnership

Objective. A	workforce partnership focused on creating career pathways for low-skill workers to

meet regional construction workforce needs.

Services. The program’s career pathways model was based on providing participants with: 1) the

opportunity to enroll in construction pre-apprenticeship programs and receive on-the-job training

to help them obtain the skills needed to access construction jobs, and 2) services to help them

find	jobs that suited their	skills.

Participants. The program recruited 379 unemployed participants during the study period. The

majority of unemployed participants were men (52 percent) and nonwhite (81 percent), had no

more than	 a	 high school education (72 percent), and were under age 35 (60 percent).	 A	 large

proportion had no work experience (56 percent), and most had no experience in construction	(98	

percent).
IMPAQ International, LLC Page 20 NFWS/SIF Quasi-Experimental Impact Study

The partnership used the funding	 to	 develop	 a	 career pathways program in nursing, allied

health, rehabilitation, health information technology (IT),	 and	 biotechnology. The program

targeted	 workers interested in obtaining	 the	 skills needed	 to	 access jobs in the healthcare

industry, particularly in	 the	sectors	 listed	 above.	 Recruitment	 was supported by	 community-	

based	 organizations and partnership service providers through referrals of jobseekers who

expressed interest in the program.	 In	 addition, employer partners	 played	 a	 key role in

recruitment by referring	entry-level healthcare workers	to	the	program who were interested in

accessing mid-level careers. As	 a	 result,	 the	 program	 primarily attracted two	 types	 of

participants: low-skill unemployed workers interested in healthcare jobs,	 and	 incumbent

workers in entry-level healthcare jobs interested in promoting their	careers.

Upon program entry, participants were paired with	 a	 qualified career pathways	 advisor	from

partner OhioMeansJobs Centers or other training/services providers. The	advisor	was	assigned

to	help the participant identify	 the	 types	of program services best suited	 to	 individual needs.

Key program servicesincluded:

§ Job readiness training.	The purpose of	this	training was	to	provide participants with the

basic	skills needed	 to	pursue, obtain, and retain	a	rewarding career in healthcare. The

training included	 the	 following components: (1) workplace professionalism	 guidance

(including dressing for success); (2)	computer	literacy	training,	to	learn the	basic	use of

computers and the Internet; (3) financial and life skills	 training;	and (4) introduction to

avenues	for accessing public benefits and other available public services.

§ Obtain National Career Readiness Certificate (NCRC). The	 program	offered participants

assistance	 in obtaining	 an	NCRC, which demonstrates	 to	 potential employers that the

individual possesses basic skills in applied mathematics, locating information, and

reading	 for information.16 In	 addition, participants	 had	 access	 to	 the School	 at	 Work

program,	 a	 healthcare-focused career development and	 academic	 readiness course

designed	to	help incumbent workers advance their careers and education. Participants

16		 For		more		information	 on	 NCRC,		see	 http://www.act.org/products/workforce-act-national-career-readinesscertificate.
IMPAQ International, LLC Page 21 NFWS/SIF Quasi-Experimental Impact Study

also	had	access	to	programs	to	assist	them in earning	a	GED diploma and preparing for

post-secondary education.

§ Industry-focused training. Service providers worked closely with employers	 to	 develop

curricula with customized training courses	 to	 help participants acquire the skills and

knowledge	 to	address employer needs. Participants received	a	training credential upon

completion that they could use	 to	 obtain immediate employment with the partner

employers that supported the training. To enhance recruitment and retention in these

programs, partner employers offered tuition reimbursement	 to	 	 participants and, in

some cases, even prepaid tuition.

§ Job search assistance.	 Participants could receive personalized services	 to	 help them

access jobs	 compatible	 with	 their	 skills.	 In	 particular,	 participants had face-to-face

consultations with advisors	 and	 other qualified workforce staff from OhioMeansJobs

Centers or other service providers, in which they received (1)	 an	 assessment	 to	 help

identify their skills	and	work experience, (2)	assistance	to	develop	a	résumé	to	highlight

their	 skills	 and	 work experience, (3) job application	 assistance,	 including mock

interviews,	and	(4) referrals	to	job openings	at	partner employers.

The advisor was responsible for helping participants identify which services best suited their

needs, and working	 	with participants	 as	they progressed through the	 program	 to identify

additional services to	 advance	their educational/employment	 goals.	 A	 2011 study found that

the	 program successfully served	 its	 target	 population and provided participants with the

training needed	to	access healthcare jobs (Elvery	and	Spence, 2011).

3.1.2 Advanced Manufacturing Partnership

The Advanced Manufacturing Partnership (Advanced Manufacturing) was	created	in 2009, with

the	 objective of promoting the employment and career advancement of low-skill workers in

advanced	manufacturing jobs. The partnership includes nearly 40 employers, eight educational

institutions, and eight community-based organizations. Advanced Manufacturing used funds

from NFWS		and other sources		to develop		a	 program that		creates		educational		and		career
IMPAQ International, LLC Page 22 NFWS/SIF Quasi-Experimental Impact Study

pathways for in-demand advanced manufacturing jobs, including team	 assembler,	 electro-	

mechanical	maintenance technician, welder, computer numerical control (CNC) operator, and

bioscience/ pharmaceutical	 technician.	 The program	 primarily	 targeted unemployed workers

interested in advanced manufacturing jobs, including new	 labor	 force entrants.	 A	secondary

target	population was	entry-level incumbent workers interested in accessing high-skill jobs and

advancing	their careers.

The Advanced Manufacturing program used	 a	service delivery model that was different from

that of	the	Health Careers partnership.	 In	particular,	the	training was primarily short-term,	with

a	 career pathways framework promoting incremental employment and educational

advancement. The following services were offered:

§ Job readiness training.	 Participants were offered training	 to	 improve	 personal	 and

professional skills. Once individuals	 completed	the training, the program attempted to

place	 them in entry-level internships or part-time jobs in manufacturing	 to	 gain	

workplace experience.

§ Obtain NCRC and Manufacturing Standard Skills Council (MSSC) certifications.

Participants who completed the job readiness training and retained entry-level jobs for

some	 time	 were offered	 assistance	 in obtaining	 the	 NCRC.	 Participants	 were	 also	

encouraged	 to	 enroll in	 the	 MSSC Certified Production Technician program	 to	 earn

certification of their readiness for high-skill manufacturing jobs.17 These certifications

were expected	 to	 promote evidence-based hiring of program participants in mid-level

and,	potentially, high-skill manufacturing jobs.

§ Academic and career advancement services. Participants successful in earning

certificates and obtaining mid-level and high-skill	 jobs could (1) enroll in college-level

coursework	and	to obtain	an	associate	degree with one of the partner colleges, and (2)

engage	 in specialized apprenticeships with partner employers	 to	 help	 advance	 their

careers.

17 For more details,see http://www.msscusa.org/production-certification-cpt.
IMPAQ International, LLC Page 23 NFWS/SIF Quasi-Experimental Impact Study

§ Job search assistance.	 Throughout the program,	 participants	 were offered job search

assistance	 to help connect	 to	 employers with workforce needs. These services were

expected	to	be most valuable for participants who went through	the	program	and	were

able	to	earn	both NCRC	and	MSSC certifications, which opened pathways	to	in-demand,

high-skill jobs with partner employers. These services were	 available	 even	 	 to	

participants who did not	earn	credentials.

The program was structured so that, over time, motivated	 participants	 could obtain	 all	 the

credentials and work experience needed	 to	 access high-skill manufacturing jobs. To ensure

success, participants were	advised	by case managers who supported them from enrollment to

finding	a	job.

3.1.3 Construction Sector Partnership

The Construction Sector Partnership (Construction Partnership), which was formed in 2009, is

to	 improve existing construction career pathways	 and	 design new ones to meet regional

workforce needs. The partnership is composed of employers, community colleges, vocational

schools,	 the	Associated Builders and Contractors,	 the	 Independent Electrical Contractors, and

the	 Greater Cincinnati Apprenticeship Council.	 The partnership’s program emphasizes the

development of educational pathways for	the	region’s in-demand construction occupations, as

identified by partner employers, including carpentry; electrical; plumbing/pipefitting; heating,

ventilation,	air	conditioning, and refrigeration (HVAC/R);	and	laborers.

The program	primarily	focuses on recruiting low-skill jobseekers, including new entrants in	the

labor	market and inexperienced workers. Special	emphasis	is	placed	on recruiting women and

minorities,	 to	 increase	 diversity	 in the construction workforce. The Construction Partnership

program	differs from the	 two	programs	described earlier in that its career pathways model is

based	 on pre-apprenticeship	 programs	 and on-the-job training. The	 program	 enrolls

participants in pre-apprenticeship programs sponsored by	its	partners, which enables them to

receive on-the-job training	and	acquire the skills needed to access	entry-level construction	jobs.
IMPAQ International, LLC Page 24 NFWS/SIF Quasi-Experimental Impact Study

The program is similar to the other two programs in offering participants job readiness training

and	job search assistance	to	ensure high job placement and retention	rates.	

3.1.4 Participant Characteristics

All three programs collected rich information on	 all	 participants who entered their	 programs	

during the study period	 (January	2010	–	February 2012). Table	2	provides descriptive analyses

of the characteristics of unemployed participants in each of	 the	 three	 programs.	 As shown,

during	this	period, the Health Careers program served 992 unemployed participants, making it

the	largest of the three	programs.	The Advanced Manufacturing	and	Construction Partnership

programs	served 684 and 379 participants, respectively.18

The	figures in Table	2	also show	that	the	three programs attracted different populations. About

90	percent of		unemployed Healthcare Careers participants were women,		 compared	 to	 34

percent and 48 percent for the Advanced Manufacturing and Construction Partnership

programs, respectively. This is not surprising, given that, historically, women	 have	 long been

overrepresented in healthcare relative	to	men, while the opposite is true for manufacturing and

construction.19 However, we should note that Advanced	 Manufacturing,	 and	 particularly the

Construction Partnership	program,	recruited	a	high number of women relative	to	employment

in their focus industries	–	a	key program recruitment objective.

Although half of the unemployed participants in the Health Careers program were white, the

program	 recruited	 a	 large	 proportion of black participants (40 percent). The Advanced

Manufacturing and Construction Partnership programs recruited primarily	black	and	other race

participants. This shows that the latter two programs were very effective in recruiting racial

minorities, which was one of their objectives.

18 Note that, during the study period, the three programs also served employed participants. In particular, Health

Careers served 978 employed participants, Advanced Manufacturing served 82 employed participants, and

Construction Pathways served 85 employed participants. Since employed participants are not included in the

quasi-experimentalstudy, we present only analyses of the characteristics of unemployed participants. 19 See Michaelides and Mueser (2013) for	a	discussion of the industry workforce composition in the United States

by gender, race, and ethnicity.
IMPAQ International, LLC Page 25 NFWS/SIF Quasi-Experimental Impact Study

Table 2: Characteristics of Unemployed Program Participants

Health Advanced Construction

Careers Manufacturing Partnership

Unemployed	Participants 992 (100%) 684 (100%) 379 (100%)

Gender

Men 97 (10%) 449 (66%) 197 (52%)

Women 895 (90%) 235 (34%) 182 (48%)

Race

White 499 (50%) 142 (21%) 70 (18%)

Black 397 (40%) 517 (76%) 296 (78%)

Other Race 95 (10%) 25 (3%) 12 (3%)

Missing 1	(0%) --	 1	(0%)

Education

No High School Diploma 80 (8%) 199 (29%) 63 (17%)

High School Diploma 391 (38%) 273 (40%) 210 (55%)

Associate Degree, Some College 454 (46%) 187 (27%) 93 (25%)

College Degree 77 (8%) 25 (4%) 13 (3%)

Age

Less than 25 Years 311 (31%) 145 (21%) 118 (31%)

25-34 Years 333 (34%) 206 (30%) 111 (29%)

35-44 Years 147 (15%) 146 (21%) 90 (24%)

45-54 Years 124 (13%) 127 (19%) 48 (13%)

55-64 Years 54 (5%) 53 (8%) 8	(2%)

65+ Years 5	(1%) 5	(1%) 1	(0%)

Missing 18 (2%) 2	(0%) 3	(1%)

Local Workforce Investment Area

Area 12 (Butler County) 184 (19%) 81 (12%) 53 (14%)

Area 13 (Hamilton County) 788 (79%) 598 (87%) 326 (86%)

Other Areas 20 (2%) 5	(1%) --	

Program Entry

Quarter 1, 2010 234 (24%) 12 (2%) 59 (16%)

Quarter 2, 2010 76 (8%) 29 (4%) 77 (20%)

Quarter 3, 2010 96 (10%) 52 (8%) 73 (19%)

Quarter 4, 2010 93 (9%) 80 (12%) 75 (20%)

Quarter 1, 2011 87 (9%) 147 (21%) 26 (7%)

Quarter 2, 2011 127 (13%) 123 (18%) 25 (7%)

Quarter 3, 2011 150 (15%) 87 (13%) 20 (5%)

Quarter 4, 2011 129 (13%) 154 (23%) 24 (6%)

Note: Reported is the number of participants with sample proportion in parentheses.
IMPAQ International, LLC Page 26 NFWS/SIF Quasi-Experimental Impact Study

The Health Careers program	 attracted	 individuals with higher educational attainment than

those	attracted by the other two	programs.	About 46 percent of Health	Careers’	unemployed

participants had an associate degree or some college education, and	 8	percent had	 a	 college

degree. On	 the	 other	 hand,	 only 31 and 28 percent of participants in the Advanced

Manufacturing and Construction Partnership programs, respectively, had more than	 a	 high

school education.

The age distribution of participants did not vary much	 across	 programs. All three	 programs

attracted high proportions of unemployed participants under	 age	 35 (65 percent for Health

Careers, 51	percent for Advanced	Manufacturing,	and	60 percent for	Construction Partnership),

while only	 9	 percent of participants were age 55 or older in each	 program.	 Finally, the	 vast

majority	of unemployed participants in all three programs resided in Ohio’s LWIAs 12 (Butler

County) and 13 (Hamilton	County).

Using the merged Ohio UI	Wage	Records	and	the NFWS/SIF-funded program data, we examined

the	employment history of unemployed program participants in	 the	eight-quarter	period prior

to	program	entry. The following prior employment measures	were constructed:

§ Employment in prior	eight quarters	–	Participant	had positive earnings in the	quarter, for

each prior	quarter.	

§ Prior	employment in	both	quarters	1–2	–	Participant	had positive	earnings in each of the

two quarters prior	to	program entry.

§ Prior employment in all quarters	1–4	–	Participant had positive earnings in each of	the

four quarters prior	to	program	entry.	

§ No prior employment in quarters	1–2	–	Participant	had	zero earnings in each of the two

quarters prior	to	program entry.

§ No prior employment in quarters	1–4	–	Participant	had	zero earnings in each of the four

quarters prior	to	program entry.
IMPAQ International, LLC Page 27 NFWS/SIF Quasi-Experimental Impact Study

§ Prior employment in	 the	 focus industry, quarter	 1	 –	 Participant	 had	 positive earnings

from	an	employer in the	program’s	focus industry	in quarter	1	prior to program entry.

§ Prior employment in the focus industry, quarter	1–4	–	Participant	had	positive earnings

from	 an	employer in the	 program’s	focus industry in	 any	of	 the	four quarters prior	 to	

program	entry.

§ Earnings amount in the prior eight quarters	–	Earnings	amount in each quarter prior	to	

program	entry.	

Table	3	presents these	measures for	the	three programs. As	shown, unemployed participants	in

the	 Health Careers program were more likely than those in the other two	 programs	 to	 be

employed prior	 to	 program entry.	 In	 particular, 45 percent of unemployed Health Careers

participants were employed in quarter	 1	 prior to program entry, compared	 to	 29 percent of

Advanced	Manufacturing participants and 30 percent of Construction Partnership participants.

These proportions were similar	in the entire eight-quarter	period prior	to	program	entry.

Comparing the remaining	measures	of prior employment history	across	programs, we find that

390 (39 percent) Health Careers participants were employed in both quarters	1	and	2	prior	to	

program	entry,	and 309 (31 percent) were employed	in	all	four quarters	prior to	program entry.

These proportions greatly exceeded those of	 the	 other two programs, which shows	 that

unemployed participants in Health Careers were more likely	 to	be employed prior	 to	program

entry	than participants in the other two programs	 and	much more likely	 to	 have	 continuous

employment. Moreover, only 33 percent of Health Careers’	 unemployed participants had no

earnings	 in the entire four-quarter period prior to	 program	entry, compared with 50 percent

and	 44 percent for Advanced Manufacturing’s	 and	 Construction Partnership’s unemployed

participants,respectively.

Another interesting difference	across	the three programs is that	a	relatively high proportion of

Health	Careers’	participants had prior work experience in the healthcare industry. As shown in

Table	3, 20 percent of unemployed participants in this program	had	positive earnings from	a	
IMPAQ International, LLC Page 28 NFWS/SIF Quasi-Experimental Impact Study

healthcare employer in	 the	 quarter immediately prior to program entry,	 and	 28 percent had

positive earnings from	 a	 healthcare employer in	 at	 least one of	 the	 four quarters prior	 to	

program	 entry. By	 comparison,	 only	 5	 percent of Advanced Manufacturing unemployed

participants,	 and	 fewer than	 3	 percent of Construction Partnership unemployed participants

had	experience working in their	program’s focus industry in the four quarters prior	to	entry.

Table 3: Employment History of Unemployed Program Participants

Health

Careers

Advanced

Manufacturing

Construction

Partnership

Unemployed	Participants 992 (100%) 684 (100%) 379 (100%)

Employment

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

1	

2	

3	

4	

5	

6	

7	

8	

451

487

511

531

522

518

549

551

(45%)

(49%)

(52%)

(54%)

(53%)

(52%)

(55%)

(56%)

196

199

227

221

234

257

260

292

(29%)

(29%)

(33%)

(32%)

(34%)

(38%)

(38%)

(42%)

113

131

122

130

151

166

171

170

(30%)

(35%)

(32%)

(34%)

(40%)

(44%)

(45%)

(45%)

Prior Employment

In Both Quarters	1-2	

In All Quarters	1-4	

390

309

(39%)

(31%)

131

95 (

(19%)

14%)

80

49

(21%)

(13%)

No Prior Employment

In Quarters	1-2	

In Quarters	1-4	

444

323

(45%)

(33%)

420

340

(61%)

(50%)

215

168

(57%)

(44%)

Prior Employment

In Quarter	1	

In Quarter	1-4	

in FocusIndustry

200

280

(20%)

(28%)

19

34

(3%)

(5%)

<10

<10

(<3%)

(<3%)

Earnings Amount

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

($)

1	

2	

3	

4	

5	

6	

7	

8	

1,902

1,905

2,328

2,216

2,380

2,348

2,354

2,364

(5,241)

(3,560)

(5,560)

(4,456)

(5,285)

(4,547)

(3,718)

(4,506)

785 (2,469)

1,073 (3,322)

1,352 (3,358)

1,464 (3,452)

1,617 (3,744)

1,679 (3,519)

1,983 (5,218)

2,001 (3,680)

1,028 (5,549)

824 (2,046)

935 (2,369)

1,095 (2,677)

1,172 (2,448)

1,627 (5,800)

1,460 (2,770)

1,451 (2,855)

Note: Reported is the number of participants with sample proportion in parentheses; for prior earnings, reported

is the sample mean with standard deviation in parentheses.
IMPAQ International, LLC Page 29 NFWS/SIF Quasi-Experimental Impact Study

Overall,	the	figures presented in Tables	2	and	3	are	compatible with the recruitment strategies

and	 objectives of the three programs. The Health Careers program targeted unemployed

workers interested in accessing healthcare jobs, so it primarily attracted women, individuals

with more than	a	high school diploma, younger people,	and	those with prior work experience,

particularly in the healthcare industry. The	 Advanced	 Manufacturing	 and	 Construction

Partnership programs primarily focused on low-skill jobseekers, including new	 labor	 market

entrants and inexperienced unemployed workers. Most unemployed participants in these two

programs	were	 male,	 nonwhite, with no more than	 a	 high school	 education,	 young workers,

and	unemployed workers with limited work experience. Importantly, both	 programs	recruited

relatively	high proportions of women and	 racial	minorities. Based on these results, it appears

that	all	three programs reached their target populations.

3.1.5 Services Received

Program	data	 also	provide information on	 the	types of services received by participants. This

information cannot be	used	to	identify	all	the services participants received in	a	program,	but it

can identify whether participants received: (1) job readiness training, (2) occupational training

(includes industry-focused training, NCRC	 and	 other certificate preparation assistance, and

participation in pre-apprenticeship	 programs),	 and	(3) employment services. The data	 also	

report whether	 a	participant earned	 a	 credential (occupational skills credential or NCRC)	 as	 a	

result of	 program	 participation. Using this information, we	 examined	 the	 types	 of services

received by unemployed participants in each program.

Table	 4	 presents the services received by unemployed participants in each	 program.	 In	 the

Health Careers program, only 27 percent of unemployed participants received job readiness

training. The majority of Health Careers participants received occupational training (including

industry-focused training	 and	NCRC preparation	 assistance)	 to	 improve	 their	 employability in

healthcare, and	 a	 little over	a	third received employment services. Of	 the	676 Health Careers

participants who received occupational training, 26	 also	 received job readiness	 training,	 and

110 received employment services. About 14 percent of	 all	 participants received	 all	 three
IMPAQ International, LLC Page 30 NFWS/SIF Quasi-Experimental Impact Study

services,	and	only 20 percent received no services. Finally,	nearly	two thirds of Health Careers

participants earned	 the	 NCRC or another training/occupational credential following their

participation.

By	 comparison,	 higher proportions of Advanced Manufacturing participants received job

readiness training (73 percent) and employment services (82 percent). Interestingly, only 23

percent received occupational training, which included NCRC	and	MSSC preparation assistance,

enrollment in undergraduate coursework, and participation in specialized pre-apprenticeship

programs.	 Of	 the	 501	 Advanced	 Manufacturing participants who received job readiness

training, 373	 also	 received employment services. We	 also	 find that 18 percent of all

unemployed participants received	all	three types of services, and only 14 percent received no

services	 at	 all.	 These figures suggest that most Advanced Manufacturing participants were

more interested in improving their employability skills	and	looking for	a	job, and less interested

in receiving manufacturing-focused training. As	 a	 result, only eight percent of unemployed

participants earned	a	credential, much lower than in	the	Health Careers program. This disparity

may	 be due	 to	 several	 factors,	 including Advanced Manufacturing’s attracting	 	 larger

proportions	of inexperienced workers who perhaps needed	basic	training	before they would be

ready	to	work toward earning	a	credential.

Table 4: Services Received	by Unemployed Participants, Health Careers

Advanced Construction Health Careers Manufacturing Partnership

UnemployedParticipants 992 (100%) 684 (100%) 379 (100%)

Job readiness Training 266 (27%) 501 (73%) 235 (62%)

Occupational Training 676 (68%) 157 (23%) 211 (56%)

Employment Services 352 (35%) 559 (82%) 236 (62%)

Job readiness	&	Occupational Training 26 (3%) --	 26 (7%)

Job readiness	&	Employment Services 76 (8%) 373 (55%) 79 (21%)

Occupational Training	&	Employment Services 110 (11%) 2	(2%) 15 (4%)

All Services 142 (14%) 124 (18%) 127 (34%)

No Services 194 (20%) 99 (14%) 71 (19%)

Earned Credential 626 (64%) 57 (8%) 212 (56%)

Note: Reported is the number of participants with sample proportion in parentheses.
IMPAQ International, LLC Page 31 NFWS/SIF Quasi-Experimental Impact Study

The Construction Partnership figures show that 62 percent of participants received job

readiness training, 56 percent received occupational training (which included on-the-job

training obtained through participation in pre-apprenticeship programs),	 and	 62 percent

received employment services. Interestingly, more than	a	third of participants received	all	three

services offered by the	program,	greatly	exceeding	the	proportions in the other two programs.

Finally, 56 percent of Construction Partnership participants earned	 a	 credential	 (including

training credential and apprenticeship certificate),	 a	 slightly lower proportion than for the

Health Careers proportion, but much higher than	for Advanced Manufacturing.

3.2 The Milwaukee Area Workforce Alliance

The Milwaukee Area Workforce Funding Alliance	 –	 a	 consortium of 13 philanthropic

organizations, six public	 agencies,	 and	 seven employers	 –	 was founded in 2008 with the

objective of providing	 a	 skilled workforce that meets the	 demands	 of employers, while

providing good jobs with	family	supporting wages. The collaborative is administered under the

Donors Forum of Wisconsin. Its activities	 are	 implemented by Urban	 Strategies,	 a	 research,

facilitation, training, and project management firm focusing on community and workforce

development.

The collaborative	 aims	 to	 enhance the workforce development system	 and	 improve the

coordination of public	 and	 private	 ventures to enhance	 regional	 competitiveness by: (1)

leveraging	 local	 investments; (2) building	 capacity	of public and private workforce system; (3)

creating	 career advancement opportunities for low-income individuals; (4) helping employers

get	the	skilled	workers they	need; and (5) advocating for policies that improve the effectiveness

of regional workforce partnerships. The collaborative	 leverages	 funds from NFWS	 and	 other

sources	 to	support local workforce partnerships that provide industry-focused training	 to	low-	

skill unemployed and incumbent workers.

Between October 2008	 and	 September 2009, the collaborative obtained approximately $15
IMPAQ International, LLC Page 32 NFWS/SIF Quasi-Experimental Impact Study

million from public and	private	sources, including $600,000 from the NFWS/SIF	grant.20 These

funds were primarily used	 to	support two partnerships:	 the	 WRTP	 and the Milwaukee Area

Healthcare	Alliance.	These partnerships provide training and other services	to	help low-income

individuals find jobs in manufacturing and construction (in the case of WRTP)	and	in healthcare

(in	the	case of	the	Milwaukee Area Healthcare	Alliance).	Below, we present each partnership’s

objectives and	a	description of	the	characteristics of unemployed participants in each	program.

Box	2	provides	a	summary of	the	two partnerships.

BOX 2: SUMMARY OF	WISCONSIN-BASED NFWS/SIF PROGRAMS

Wisconsin	Regional Training Partnership (WRTP)

Objective. A	 workforce partnership focused on brokering relationships between employers,

unions, and workers, with the objective to: (1) offer training and	 	 other services to low-skill

workers to promote their employment in construction and manufacturing; and (2) help local

builders and	manufacturers recruit a	diverse, qualified	workforce.

Services. The WRTP Construction Pathways and Manufacturing Pathways programs offered

participants	a	wide	range of services, including	pre-apprenticeship	training, assistance in	obtaining

occupational credentials, career advancement training, and job search	services.

Participants. The construction program recruited 1,103 unemployed participants during the study

period (January 2010	 –	 February 2012). The majority of participants were men (90 percent),

nonwhite (62 percent), and had no more than	 a	 high school education (97 percent). The

manufacturing program recruited 88 unemployed participants, of whom 84 percent were men,	69

percent were nonwhite and 97 percent had no more than	 a	high school education. In both

programs, the majority of participants did not have continuous employment prior to program

entry,	but many with	prior employment had experience in the program’s focus industry.

Milwaukee	Area	Healthcare	Alliance

Objective. A	workforce partnership that works with healthcare employers to identify sought-after

skills and provide training	and	other services to low-skill	workers to help	them	meet those needs.

Services. The program provided participants with occupational training to help them obtain jobs

in healthcare, on-the-job training	to help them advance their careers, and	job	search	services.

Participants. The program served 306 unemployed participants during the study period, with 94

percent women, 74 percent black, and 72 percent under age 35. Fewer than half the participants

had continuous employment prior to program entry, and 31 percent had prior experience in

healthcare jobs.

20 See: Holm, R. Aligning for Impact – The Milwaukee Area Workforce Funding Alliance, May 2013.
IMPAQ International, LLC Page 33 NFWS/SIF Quasi-Experimental Impact Study

3.2.1		 Wisconsin Regional Training Partnership

The Wisconsin Regional Training Partnership (WRTP) was established in 1992	 to	 broker

relationships between employers, unions, and workers, with the objective to enhance the

ability	 of employers in the manufacturing and construction sectors	 to	 recruit and develop	 a	

diverse, qualified workforce. To	achieve	this,	the	partnership offers training	and	other services

to	low-skill individuals to promote their employment and career advancement in	career-ladder,

union-supported jobs. The partnership offers two	 programs,	 Manufacturing Pathways	 and	

Construction Pathways, which offer the following services:

§ Individual needs assessment. Participants	 are	evaluated to	 assess	 their	workforce skills

and	job readiness	and,	based	on	this	assessment,	are	referred	to	appropriate services.

§ Pre-apprenticeship training.	Participants	are	offered tutoring	to	improve their	academic	

skills	and	prepare for apprenticeship	exams.	While receiving these services, participants

are	also	referred to industry	partners	to engage in pre-apprenticeship training.

§ Occupational certifications. Participants	 deemed	 ready	 to	 obtain entry-level jobs	 are

offered the opportunity	 to	 receive pre-employment training certificates in skilled

manufacturing trades (e.g., machinists, operating engineers, sheet metal workers,

steamfitters, and welders)	 and	 in skilled construction trades (e.g., bricklayers,

carpenters, electricians,	 laborers,	 and painters).	 WRTP	 Construction Pathways	 also	

offers	 the	 City of Milwaukee Residential Preference	 Program	 certification, which

promotes local workers’	hiring in city construction projects.

§ Career advancement training. Participants	who obtain	an	occupational certification	and	

find mid-level jobs in construction or	manufacturing	are	given	the	opportunity to	engage

in specialized apprenticeship programs provided by	 the	 partner employers	 to	 help

improve their skills and advance their careers.

§ Job search	services.	Participants	are	offered job search services	to	help them connect to

partner or other employers with workforce needs	compatible with participant	skills.

The	 partnership		works		closely		with		several		regional		partners,		including		construction		trade
IMPAQ International, LLC Page 34 NFWS/SIF Quasi-Experimental Impact Study

associations,	such	as	the Construction Labor Management Council; government organizations,

such	 as	 the	 state	 of Wisconsin Joint Apprenticeship Board and	 the	 	 Milwaukee Workforce

Investment	Board;	community organizations, such	as	 the	Milwaukee Community Service Corps

and	the Milwaukee YWCA;	and	training providers, such	as	the	Milwaukee Christian Center and

the	Northcott Neighborhood House. Moreover,	 to	 identify employers’ workforce and industry

training needs,	the	partnership works with several employer partners, including Pieper Electric,

Inc.; Michaels;	CD Smith Construction;	and	Langer Roofing and	Sheet	Metal.	

3.2.2 Milwaukee Area Healthcare Alliance

Formed in 2009, the Milwaukee Area Healthcare	Alliance	is	a	workforce intermediary led by the

YWCA of Greater Milwaukee, which works with healthcare employers to identify sought-after

skills	and	 to	develop	 a	workforce possessing those skills. The partnership provides specialized

training that prepares low-skilled workers, particularly African Americans, for entry in mid-level

skill positions in the healthcare industry. The	 program	 provides: (1)	 a	 training curriculum

designed	 to	 help unemployed workers	 to	 obtain jobs in healthcare, (2) on-the-job training to

help incumbent workers	advance	their careers, and (3) job search services.

To	 achieve	 its	 objectives,	 the	 YWCA of Greater Milwaukee is partnered with the Milwaukee

Area Health Education	 Center,	 which brings access	 to	 healthcare employers	 and	 educational

institutions; the Milwaukee Area Technical College, which brings expertise in developing

training curricula;	 and	 the	 Milwaukee Area Workforce Investment	 Board,	 which provides

employment services to participants through its	American	Job Centers.

3.2.3 Participant Characteristics

Table	5	provides descriptive	analyses	of	the	characteristics of unemployed participants in each

of	the	three programs. As shown, from January 2010 through March 2012,	WRTP	Construction

Pathways served 1,103 unemployed participants, compared with	 WRTP	 Manufacturing

Pathways, which served only 88 participants. The Milwaukee Area Healthcare	Alliance	program

served 306 unemployed participants	during the same period.
IMPAQ International, LLC Page 35 NFWS/SIF Quasi-Experimental Impact Study

Table 5: Characteristics of Unemployed Program Participants

WRTP WRTP Healthcare

Construction Manufacturing Alliance

Unemployed	Participants 1,103	(100%) 88 (100%) 306 (100%)

Gender

Men 998 (90%) 74 (84%) 17 (6%)

Women 105 (10%) 14 (16%) 289 (94%)

Race/Ethnicity

White 424 (38%) 27 (31%) 18 (6%)

Black 448 (41%) 50 (57%) 227 (74%)

Other Race 72 (7%) 4	(5%) 32 (10%)

Hispanic 159 (14%) 7	(8%) 5	(2%)

Education

No High School Diploma 197 (18%) 18 (20%) --	

High School Diploma 873 (79%) 68 (77%) --	

Associate Degree, Some College 22 (2%) 2	(2%) --	

College Degree 11 (1%) --	 --	

Age

Less than 25 Years 214 (19%) 11 (13%) 119 (39%)

25-34 Years 363 (33%) 24 (27%) 101 (33%)

35-44 Years 251 (23%) 19 (22%) 29 (9%)

45-54 Years 191 (17%) 22 (25%) 26 (9%)

55-64 Years 70 (6%) 10 (11%) 11 (4%)

65+ Years 14 (1%) 2	(2%) 20 (7%)

Program Entry

Quarter 1, 2010 7	(1%) --	 --	

Quarter 2, 2010 22 (2%) --	 --	

Quarter 3, 2010 11 (1%) 2	(2%) --	

Quarter 4, 2010 6	(1%) 3	(3%) --	

Quarter 1, 2011 206 (19%) 24 (27%) --	

Quarter 2, 2011 225 (20%) 24 (27%) 41 (13%)

Quarter 3, 2011 389 (35%) 19 (22%) 123 (40%)

Quarter 4, 2011 237 (21%) 16 (18%) 94 (31%)

Quarter 1, 2012 --	 --	 48 (16%)

Note: Reported is the number of participants with sample proportion in parentheses.

There were important differences in participant characteristics	across	the three programs. The

two	 WRTP	 programs	 primarily served	 male	 participants,	 while healthcare participants were

primarily	female. Moreover,	the	majority of participants in the three programs were nonwhite,
IMPAQ International, LLC Page 36 NFWS/SIF Quasi-Experimental Impact Study

which shows	 that	the three	programs	were effective in recruiting minorities.	More	than three

quarters of participants in the construction and manufacturing programs	 had	no more than	 a	

high school education. Information on education levels was not available for the healthcare

program.	While	all	three	programs	attracted younger participants, the healthcare program had

a	 higher proportion of participants under	 age	 35 (72 percent) relative	 to	 the other two

programs.	

Table	 6	 presents prior employment outcomes of participants in the three programs using

Wisconsin UI	Wage	Records. As shown, participants were	about equally likely	to	be employed in

each of the	eight quarters prior	to	program entry. About 60 percent	of unemployed participants

in the construction and healthcare program were employed in each of the eight quarters prior

to	program	entry. Prior employment rates were slightly lower for manufacturing participants.

In	the	construction and healthcare programs, about 38 percent of participants were employed

in each of the two quarters prior	to	entry,	and approximately 42 percent were employed in	all	

four quarters prior	 to	 entry.	 At the same time, about	 a	 quarter of participants in these two

programs	 had no employment in any of the four quarters prior to	 program	 entry. By

comparison,	manufacturing participants were less likely	 to	 have	 been continuously employed

prior	 to	 program entry,	 but		had		about		the		same proportion		of participants with no prior

employment.

Interestingly,	a	relatively high proportion of participants in each	program	had	prior experience

in the relevant focus industry. As shown in Table 6, 39 percent of construction participants had

experience working in	 the	 construction sector in	 the	 four-quarter	 period prior	 to	 program

entry.	 Similarly, 26 percent of manufacturing participants and 31 percent of healthcare

participants worked in their	 programs’	 focus industry	 prior	 to	 entering the program. These

figures show that the Wisconsin construction	 and	 manufacturing programs	 had	 much higher

proportions of participants with prior experience in	the	program’s focus industry than the Ohio

construction and manufacturing	 programs	 (3 and	 5	 percent, respectively). Prior experience
IMPAQ International, LLC Page 37 NFWS/SIF Quasi-Experimental Impact Study

proportions were similar between the Ohio and the Wisconsin	healthcare	programs.	

Table 6: Employment History of Unemployed Program Participants

WRTP

Construction

WRTP

Manufacturing

Healthcare

Alliance

Unemployed	Participants 1,103	(100%) 88 (100%) 306 (100%)

Employment

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

1	

2	

3	

4	

5	

6	

7	

666

664

664

655

622

646

665

(60%)

(60%)

(60%)

(59%)

(56%)

(59%)

(60%)

44

50

49

44

48

48

50

(50%)

(57%)

(56%)

(50%)

(55%)

(55%)

(57%)

168

184

181

184

185

187

189

(55%)

(60%)

(59%)

(60%)

(61%)

(61%)

(62%)

In Prior Quarter 8	 669 (61%) 51 (58%) 169 (59%)

Prior Employment

In Both Quarters	1-2	 569 (48%) 38 (43%) 148 (48%)

In All Quarters	1-4	 456 (41%) 26 (30%) 127 (42%)

No Prior Employment

In Quarters	1-2	 342 (31%) 32 (36%) 102 (33%)

In Quarters	1-4	 258 (23%) 23 (26%) 75 (25%)

Prior Employment

In Quarter	1	

in FocusIndustry

305 (28%) 15 (17%) 70 (23%)

In Quarter	1-4	 432 (39%) 23 (26%) 96 (31%)

Earnings Amount

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

In Prior Quarter

($)

1	

2	

3	

4	

5	

6	

7	

3,790

4,168

4,384

4,424

4,451

4,647

4,965

(4,981)

(5,398)

(5,559)

(5,912)

(6,138)

(6,364)

(6,462)

2,234

2,565

2,941

2,526

2,578

2,840

3,494

(3,129)

(4,045)

(4,585)

(3,829)

(3,634)

(4,061)

(5,197)

1,577

1,792

2,031

1,968

2,077

2,101

2,102

(2,249)

(2,277)

(2,523)

(2,477)

(2,631)

(2,606)

(2,622)

In Prior Quarter 8	 4,884 (6,203) 3,811 (5,350) 2,046 (2,802)

Note: Reported is the number of participants with sample proportion in parentheses; for prior earnings, reported

is the sample mean with standard deviation in parentheses.

Finally, construction participants had relatively higher average prior earnings in	 	 the eight-	

quarter period prior	 to	 program	 entry than participants in	 the	manufacturing and healthcare

programs. Because participants in the three programs	had	similar prior employment	rates,	the
IMPAQ International, LLC Page 38 NFWS/SIF Quasi-Experimental Impact Study

earnings	differences suggest that	construction participants	were	more likely	to	be	employed for

longer periods or more hours within each quarter or had higher hourly	 earnings	 relative	 to	

those in the other programs.

Overall,	the	above descriptive	analyses	highlight the	important differences	in the characteristics

and	prior employment of unemployed participants	across	 the	three programs. The healthcare

program	attracted unemployed participants who were primarily female, black,	and	under age

35. The construction	and	manufacturing	programs	primarily	served	male,	minority workers with

no more than	a	high school education. While most participants in each of the three programs

did not	have	continuous employment in the four-quarter period prior	to	program	entry, many

had	prior experience working in the program’s focus industry.

3.2.4 Services Received

Program	 data for the three partnerships did not provide complete information on services

received. Thus, it is not feasible to	 assess	 which program services unemployed participants

received during	the	study period.
IMPAQ International, LLC Page 39 NFWS/SIF Quasi-Experimental Impact Study

4. Quasi-Experimental Impact Study Results

IMPAQ	implemented the quasi-experimental approach (described in Section 2.2.4)	 to	estimate

the	 impacts of each of	 the	six NFWS/SIF-funded	 programs	 on the labor market outcomes of

unemployed participants. Due	 to	 differences	 across	 programs	 in their focus	 industries,

participant characteristics,	 and	services provided, we implemented	 the	matching	process and

estimated	program impacts separately for each program. Below, we present	 the	results of the

impact	 study, starting with the results for the three Ohio-based programs, followed by the

results of three Wisconsin-based	programs.	

4.1 Ohio Results

This section describes the quasi-experimental impact study results for the three Ohio-based

programs,	 including the	 matching	 process and	 the	 estimated	 impacts	 for each program.	 A

summary of	the	results is provided in	Box 3.
IMPAQ International, LLC Page 40 NFWS/SIF Quasi-Experimental Impact Study

BOX 3: SUMMARY OF	QUASI-EXPERIMENTAL IMPACT RESULTS IN OHIO

Health	Careers Collaborative	of Greater Cincinnati

§ The program had large effects on employment. In the six quarters after program entry,

participant employment rates were 57.6-64.8 percent, exceeding the employment rates of

thematchedcomparisongroupby14.1-17.0percentagepoints(29-37percent).

§ The program was effective in helping participants to obtain healthcare	 	 jobs. In the four

quarters after entry, 33.4-34.3 percent of participants were employed in healthcare,

exceeding the healthcare employment rates of the matched comparison group by 24.0-25.3

percentage points (233-304 percent).

§ The program was very effective in helping participants to improve their job retention rates.

About 35.2 percent of participants found employment in quarter	1	and remained employed in

each of the six quarters after entry, compared to only 22.4 percent of matched comparison

group	members,	a	12.8 percentage point (57 percent) difference.

§ The program had large effects on earnings. In the six-quarter follow-up period, participants

had	$5,517 (52 percent) higher earnings than	those in the matched comparison	group.

Advanced	Manufacturing Partnership

§ The program had large effects on employment. In the six quarters after entry, 42.2-52.6

percent of participants were employed, exceeding the employment rates of matched

comparisongroupmembersby8.2-14.3percentagepoints(24-38percent).

§ The program had modest effects on employment in manufacturing; no more than 4.2 percent

of	participants were employed in manufacturing	in	the four quarters after entry.

§ The program had	positive effects on job retention. About	19.9 percent	of	participants found	a

job in quarter	1	and remained employed in each of the six quarters after entry, compared to

15.9 percent of matched comparison	group members, a	25 percent difference.

§ The program had positive effects on earnings. In the six-quarter follow-up period, participants

had $2,635 (31 percent) higher earnings than those in the matched comparison group. These

effects were lower than those of the	Health	Careers program.

Construction	Sector Partnership

§ The program had modest effects on	employment.	Between 38.8-45.9	percent of participants

were employed in quarters	 1-6	after program entry, which exceeded the employment rates

ofmatched comparison groupmembers by 3.2-6.1 percentage points(9-16 percent).

§ The program had	modest effects on employment	in	construction; no more	than 4.8 percent of

participants were employed	in	construction	in the four quarters after entry.

§ The program had	no effects on job	retention	and modest	effects	on earnings.
IMPAQ International, LLC Page 41 NFWS/SIF Quasi-Experimental Impact Study

4.1.1 Matching	Results

To construct appropriate	 matched	 comparison groups for unemployed participants in each of

the	three Ohio NFWS/SIF-funded	 programs,	 we used the Ohio ES	 population,	 which includes

unemployed workers who sought state employment and training services during the study

period.	Table	 7	summarizes the characteristics of unemployed participants in each NFWS/SIF-	

funded program	and	of unemployed ES	participants in Ohio.

As shown in Table 7, there were notable differences between the unemployed participants in

each program and the unemployed ES population. For example,	 the	 Health Careers program

attracted unemployed participants who were much more likely	 to	 be female, nonwhite,

educated	beyond	high school,	and	under	age	45,	as	compared	to	unemployed workers in the ES

population. Similarly, the	Advanced	Manufacturing	program	attracted unemployed participants

who were more likely	 to	 be	 male	 and nonwhite	 than	 ES participants, while	 the	 Construction

Partnership program attracted	a	relatively higher proportion of nonwhites, individuals with no

more than	 a	 high school diploma, and younger workers. These disparities show that the

unemployed ES population differs in important ways from	 the	unemployed participants in the

three programs, and, therefore, we could not use	 the	 ES	data	 in their		original		form		as	 a	

comparison	group for the impact study.

It	 is	 also	 important to	 note	 the disparities in the characteristics of unemployed participants

across	 the	 three NFWS/SIF-funded	 programs.	 Relative	 to	 the other two	 programs,	 Health

Careers attracted	larger	proportions of unemployed participants who were	 female,	white,	and	

had	 more than	 a	 high school education. Moreover, the Advanced Manufacturing program

attracted	 larger	 proportions of unemployed participants who were male,	 had	 no high school

diploma, and	were	at	least	age	45.
IMPAQ International, LLC Page 42 NFWS/SIF Quasi-Experimental Impact Study

Table 7: Characteristics of Unemployed NFWS/SIF and ES Participants in Ohio

Health Advanced Construction ES

Careers Manufacturing Partnership

Total Number of Participants 992 (100%) 684 (100%) 379 (100%) 55,754 (100%)

Gender

Men 97 (10%) 449 (66%) 197 (52%) 30,645 (55%)

Women 895 (90%) 235 (34%) 182 (48%) 25,109 (45%)

Race

White 499 (50%) 142 (21%) 70 (18%) 37,302 (67%)

Black 397 (40%) 517 (76%) 296 (78%) 9,072 (16%)

Other Race 95 (10%) 25 (3%) 12 (3%) 2,445 (4%)

Missing 1	(0%) --	 1	(0%) 6,935 (12%)

Education

No High School Diploma 80 (8%) 199 (29%) 63 (17%) 8,219 (15%)

High School Diploma 391 (38%) 273 (40%) 210 (55%) 24,230 (43%)

Associate Degree, Some College 454 (46%) 187 (27%) 93 (25%) 13,164 (24%)

College Degree 77 (8%) 25 (4%) 13 (3%) 10,141 (18%)

Age

Less than 25 Years 311 (31%) 145 (21%) 118 (31%) 9,584 (17%)

25-34 Years 333 (34%) 206 (30%) 111 (29%) 13,533 (24%)

35-44 Years 147 (15%) 146 (21%) 90 (24%) 11,197 (20%)

45-54 Years 124 (13%) 127 (19%) 48 (13%) 11,752 (21%)

55-64 Years 54 (5%) 53 (8%) 8	(2%) 7,603 (14%)

65+ Years 5	(1%) 5	(1%) 1	(0%) 1,885 (3%)

Missing 18 (2%) 2	(0%) 3	(1%) 200 (0%)

Local Workforce Investment Area

Area 12 (Butler County) 184 (19%) 81 (12%) 53 (14%) 23,882 (43%)

Area 13 (Hamilton County) 788 (79%) 598 (87%) 326 (86%) 27,609 (50%)

Other Areas 20 (2%) 5	(1%) --	 4,263 (7%)

Program Entry

Quarter 1, 2010 234 (24%) 12 (2%) 59 (16%) 8,604 (15%)

Quarter 2, 2010 76 (8%) 29 (4%) 77 (20%) 7,581 (14%)

Quarter 3, 2010 96 (10%) 52 (8%) 73 (19%) 8,311 (15%)

Quarter 4, 2010 93 (9%) 80 (12%) 75 (20%) 6,761 (12%)

Quarter 1, 2011 87 (9%) 147 (21%) 26 (7%) 7,128 (13%)

Quarter 2, 2011 127 (13%) 123 (18%) 25 (7%) 6,605 (12%)

Quarter 3, 2011 150 (15%) 87 (13%) 20 (5%) 5,621 (10%)

Quarter 4, 2011 129 (13%) 154 (23%) 24 (6%) 5,143 (9%)

Note: Reported is the number of participants with sample proportion in parentheses.
IMPAQ International, LLC Page 43 NFWS/SIF Quasi-Experimental Impact Study

Table	 8	summarizes the employment history of unemployed participants in each NFWS/SIF-	

funded program and of ES unemployed participants, and confirms the important disparities

between the four populations on these measures. All three NFWS/SIF-funded programs

attracted unemployed participants with weak employment history	 relative	 to	 	 the	 	 ES

population. As shown in	Table	8, NFWS/SIF participants in	 the	three programs were less likely

than unemployed ES	participants	to	be employed	in each of the eight	quarters	prior to program

entry	and	to have continuous employment in the	four quarters prior	to	program entry. Notably,

Health Careers participants were more likely than participants in the other two	programs	to be

employed and	 to	 have	 continuous employment prior	 to	 program entry. Health Careers

participants were much more likely than ES participants to	 have	 prior employment in the

program’s	focus industry, while	the	opposite is true for participants in the	other	two NFWS/SIF-	

funded programs. Participants in	 the	three NFWS/SIF-funded programs	 had	much lower prior

earnings	in the eight quarters prior	to	program entry compared with ES participants.

The disparities in characteristics and prior employment measures show that the three

NFWS/SIF-funded programs attracted different	 types	 of unemployed	 participants.	 This

suggests that participation in	a	given	program	is strongly correlated with	certain	characteristics

that do not necessarily influence participation in	 the	 other two programs. Furthermore,	 the

three programs focused on different industries	 and	 provided different services. For these

reasons,	the three programs were considered separately in the impact study.
IMPAQ International, LLC Page 44 NFWS/SIF Quasi-Experimental Impact Study

Table 8: Employment History of Unemployed NFWS/SIF and

Health Advanced

Careers Manufacturing

Total Number of Participants 992 (100%) 684 (100%)

Employment

In Prior Quarter	1	 451 (45%) 196 (29%)

ES Participants in Ohio

Construction ES Partnership

379 (100%) 55,754 (100%)

113 (30%) 42,891 (77%)

In Prior Quarter 2	 487 (49%) 199 (29%) 131 (35%) 43,705 (78%)

In Prior Quarter 3	 511 (52%) 227 (33%) 122 (32%) 42,895 (77%)

In Prior Quarter

In Prior Quarter

4	

5	

531

522

(54%) 221

(53%) 234

(32%)

(34%)

130

151

(34%) 41,855

(40%) 41,288

(75%)

(74%)

In Prior Quarter 6	 518 (52%) 257 (38%) 166 (44%) 40,883 (73%)

In Prior Quarter 7	 549 (55%) 260 (38%) 171 (45%) 40,295 (72%)

In Prior Quarter	8	

Prior Employment

In Both Quarters	1-2

In All Quarters	1-4

No Prior Employment

InQuarters	1-2

InQuarters	1-4

Prior Employment in FocusIndustry

In Quarter	1	(Healthcare)

551

390

309

444

323

200

(56%) 292

(39%) 131

(31%) 95 (

(45%) 420

(33%) 340

(20%)

(42%)

(19%)

14%)

(61%)

(50%)

--	

170

80 (

49 (

215

168

(45%) 39,802 (71%)

21%) 40,598 (73%)

13%) 35,580 (64%)

(57%) 9,756 (18%)

(44%) 7,579 (14%)

--	 5,283 (9%)

In Quarter	1-4	(Healthcare) 280 (28%) --	 --	 6,301 (11%)

In Quarter	1	(Manufacturing) --	 19 (3%) --	 5,301 (10%)

In	Quarter	1-4	(Manufacturing)

In Quarter	1	(Construction)

--	 34

--	

(5%)

--	 <10

--	 6,058 (11%)

(<3%) 2,873 (5%)

In Quarter	1-4	(

Earnings Amount

In Prior Quarter

Construction)

($)

1	 1,902

--	 --	

(5,241) 785 (2,469)

<10 (<3%) 3,558

1,028 (5,549) 6,681

(6%)

(8,468)

In Prior Quarter 2	 1,905 (3,560) 1,073 (3,322) 824 (2,046) 6,882 (8,396)

In Prior Quarter

In Prior Quarter

3	

4	

2,328

2,216

(5,560) 1,352 (3,358)

(4,456) 1,464 (3,452)

935 (2,369) 6,925

1,095 (2,677) 6,787

(8,896)

(9,027)

In Prior Quarter 5	 2,380 (5,285) 1,617 (3,744) 1,172 (2,448) 6,646 (8,238)

In Prior Quarter 6	 2,348 (4,547) 1,679 (3,519) 1,627 (5,800) 6,608 (8,327)

In Prior Quarter 7	 2,354 (3,718) 1,983 (5,218) 1,460 (2,770) 6,488 (8,202)

In Prior Quarter	8	 2,364 (4,506) 2,001 (3,680) 1,451 (2,855) 6,386 (8,147)

Note: Reported		is the number		of participants,		with		sample proportion	 in	 parentheses;		for		prior earnings,

reported is the sample mean, with standard deviation in parentheses.
IMPAQ International, LLC Page 45 NFWS/SIF Quasi-Experimental Impact Study

Our matching methods,	 as	 described in Section 2.2.4, were designed	 to	 re-weigh the

comparison	sample of ES participants	to	remove the differences in characteristics for the three

program	groups observed	in	Tables	7	and	8. The PSM methods	were	applied separately	for each

program.	Although the	 reader	 is referred to the	 earlier	section for	 a	discussion of the formal

structure underlying the	matching	process, we restate the process here, as applied	to	the Ohio

program:	

§ Step 1: Merge data –	We merged the NFWS/SIF	data	for each program with ES and UI

Wage	Record data using participant personal identifiers.

§ Step 2: Produce propensity score –	We used	a	logit model to estimate the likelihood of

program	 participation based on individual characteristics, employment history

measures, and interactions between these. Using	 the	 results, we produced the

propensity score for each participant and non-participant in the	data.	At	 this	point, we

omitted cases from each sample that were off	 the	 common support of	 the	propensity

score;21		these		were		cases		whose		characteristics		were		such		that		they	 could		not		be

matched.22		 We	 then		reran		the		logit		model		on		the		remaining		sample		to		produce		a	

propensity score for	all	cases on the common support.

§ Step	3: Use propensity score	to	construct sample weight	–	We weighed each	comparison	

case by	the	odds	ratio	of	the	predicted propensity score, which	assures	—	in theory	—	

that	 the	 weighted	 comparison sample will	 have	 the	 same distribution on	 all	 control

variables	as	the treated	sample.	

21 In practice, the common support includes all cases with predicted propensity scores between the smallest

propensity score observed for the treatment group and the largest propensity score observed for the comparison

group. Propensity scores outside this range are based on extrapolation and, therefore, may be subject to serious

bias(Caliendo and Kopeing, 2008). 22 In matching applications, it is common to omit	a	large number of comparison cases that do not provide useful

matches for any treatment case. In implementing the matching process for the Health Careers program, 9,054 of

the 55,754 ES participants were omitted because they failed to match program participants. Similarly, in the

matching process of the Advanced Manufacturing and Construction Partnership, 13,361 and 18,895 ES cases were

omitted, respectively. Such omissions do not create	 a	 bias in our estimates because our focus is on estimating

impacts for participants. On the other hand, if many treatment cases are omitted, the true impact of the full

population of participants may	not correspond to the estimated impact. Fortunately, given	the large sample size of

the ES comparison sample, it was not necessary to omit many treatment cases. None of the participants in Health

Careers or in Construction Partnership were omitted, and only two participants in Advanced Manufacturing were

omitted. Hence, there is essentially no bias due to failure to match participants to comparison cases.
IMPAQ International, LLC Page 46 NFWS/SIF Quasi-Experimental Impact Study

§ Step 4: Compare treatment	and weighted	matched	comparison sample (balancing	test)	–	

We employed	 a	 balancing	 test	 based on	 a	 t-statistic for the differences in mean

characteristics between	the	treatment	and the	matched	comparison groups.	If	matching

was successful,	the	t-tests should yield no or very few statistically significant differences

between	 the	treatment	and	the matched comparison groups.	If	such differences were

found, more complex specifications were considered (normally by including interaction

terms), with the process repeated until matching was successful.23

This process, implemented separately for each	 program,	 allowed us to construct	 a	set of

weights for the remaining ES sample so that the weighted ES sample	 had	 the same

characteristics	and	prior employment measures	as	 the	treatment sample. This	means	that the

only difference between	the	treatment and the matched comparison samples is	that	individuals

in the treatment sample	participated in the NFWS/SIF-funded program. Thus, differences in the

labor	 market outcomes between the treatment and the matched comparison	 samples	

constitute reliable estimates	of	the	program’s impacts.

The results of the balancing	tests	are	presented in	Tables	A,	B,	and	C	in the Appendix. As	noted,

above,	these results test whether the matched	 comparison	 group for each	 program	 has	the

same distribution of variables	as	the	sample of program	participants.	In	each	table,	we see that

the	matched comparison group is similar	to	the	corresponding	program	sample.

Appendix Table	A	lists	variable	means for	unemployed Health Careers program participants and

the	 matched comparison	 sample,	 indicating the success of	 matching.	 In	 50 comparisons of

23 At the conclusion of this process, the following control variables were used in the logit model: 1) individual

characteristics	 –	 gender,	 	race,	 	education,	 	 age, LWIA,	 	 and quarter of program entry; 2) employment	 	history

measures	–	employment in each prior quarter	1-8, prior employment in both quarters	1-2, prior employment in

all quarters	 1-4, no prior employment in both quarters	 1-2, no prior employment in any quarter	 1-4, prior

employment in focus industry in prior quarter 1, prior employment in focus industry in prior quarters	1-4, and

earnings in prior quarters	 1-8; and 3) interactions between gender and race, gender and age, gender and

education, gender and quarter of entry, gender and prior earnings, gender and prior employment in focus industry,

race and age,	race and education, race and quarter of entry,	race and prior earnings,	race and prior employment	in

focus industry, education and age, education and quarter of entry, education and prior earnings, education, and

prior employment in focusindustry.
IMPAQ International, LLC Page 47 NFWS/SIF Quasi-Experimental Impact Study

means,	we found that only one is	statistically significant; among participants, approximately 1.8

percent were coded	 as	 missing	 age,	 whereas the proportion is only 0.1 percent among the

matched comparison group. Although statistically significant,	 this	 difference is so small and

affects so few cases	 that	any	resulting	bias	will be	 trivial.	The comparable	comparison	for the

Advance	 Manufacturing program and its matched comparison group similarly reveals 49

statistically insignificant differences,	and	only one statistically significant difference, in program

entry	 in quarter 4, 2011.	Again,	 the	 difference is very small,	 and	 it is unlikely	 to	 cause any

important	 bias.	 Finally, in	 the	 case of the Construction	 Partnership	program, we observe only

one statistically significant difference. That difference is in the	 age	missing group, where 0.8

percent of treatment group members and no matched comparison group members	are	coded

as	missing	age.	This difference is very small	so that	any	resulting	bias	is	negligible.

The balancing	 tests	showed that, with only one exception in each program, treatment group

cases	 are	 observationally equivalent	 to	 matched	 comparison	 group	 cases.	 Nevertheless, we

wanted	 to	 ensure that	 the	 few differences in characteristics remaining after	 the	 matching

would not influence our results. Therefore, we applied	 a	 bias adjustment method	 to	 our

estimates	 of program	 impacts.	This approach	 fits	 a	 linear regression model	 to	 the	 matched

comparison	 sample for each outcome of interest	 and	 uses the results	 to	 adjust the impact

estimate	 to	 account for differences in characteristics between treatment and	 the	 matched

comparison	 group. We found that bias-adjusted impacts were equivalent to	 the	 impact

estimates	 reported in the following section,		which further supports our		confidence in the

validity	of	the	matching implementation.

4.1.2 Impact Results

Program	 impacts for each of the three programs were estimated by calculating mean

differences in employment and earnings between the treatment group and	 the	 matched

comparison	group. To	assess	the	statistical significance of the impact estimates, we produced	t-	

tests	 based on bootstrap standard errors. The results of	 the	 impact	 analyses,	 below,	 are

presented separately for each	program.	
IMPAQ International, LLC Page 48 NFWS/SIF Quasi-Experimental Impact Study

Health Careers. Table	 9	 presents the	 impact	results for	 overall	 employment in quarters	 1-6

after	program entry and employment in the healthcare industry in	quarters	1-4	after program

entry.24		The two left-hand columns of Table	 8	present	 the	means and standard deviations of

these	outcomes for	the	treatment and	the	matched comparison groups, respectively. The right-	

hand	 column presents	 the	 treatment matched comparison group difference, which is the

estimated	 impact of the	 program	 on the outcome. The estimated	 impact	 expressed	 as	 a	

percentage	of the matched	comparison group mean appears in brackets.

Starting with the outcomes of program participants (treatment group), we see	 that	 	 57.6

percent of participants were employed in quarter	 1	 after	 program	 entry.	 This proportion

increased slightly in quarters	1-6	after entry, peaking	at	64.8 percent in quarter 3. We also find

that about one third of	program	participants were employed in healthcare jobs after program

entry.	Dividing participant healthcare employment rate by the participant	overall	employment,

we find that over half of participants who found employment after program entry were

employed in healthcare.

Comparing employment	 rates	 between the treatment and	 the	 matched comparison groups

shows that the	program	led	 to	positive effects on overall employment rates in each of	 the	six

quarters	after	program	entry. For	example,	57.6	percent of treatment group members	and 43.5

percent of matched comparison group members	 had	 positive earnings in quarter	 1	 after

program	 entry. As the right-hand column shows,	 the	 difference is .141 and statistically

significant at	 the	 1	 percent level. This means that the Health Careers program led	 to	 a	 14.1

percentage-point increase in employment for unemployed	participants.	If	we divide this impact

by	 the	 weighted matched comparison group	 mean,	 we find that	 the	 program	 led	 to	 a	 32

percent increase in quarter	 1	 employment. This	 impact	was sustained through the entire six-	

quarter follow-up period	 –	 program	 participants were 29-37 percent more likely	 to	 be

employed in quarters	2-6	than individualsin the matched comparison group.

24 As noted, employer industry was not available for quarters	3-4, 2012; thus, employment in the program’sfocus

industry was constructed only for up to four quarters after program entry.
IMPAQ International, LLC Page 49 NFWS/SIF Quasi-Experimental Impact Study

Table 9: Program Impacts on Employment, Health Careers

Treatment Matched Comparison

Group Group [% Total Number of Participants 992 46,701

Employed

Program Impact

Comparison Group]

In Quarter 1	 .576 (.495) .435 (.496) .141 (.012)***

[+32%]

In Quarter 2	 .628 (.484) .458 (.498) .170 (.016)***

[+37%]

In Quarter 3	 .648 (.478) .481 (.500) .167 (.015)***

[+35%]

In Quarter 4	 .634 (.481) .484 (.500) .150 (.019)***

[+31%]

In Quarter 5	 .641 (.480) .491 (.500) .150 (.019)***

[+31%]

In Quarter

Employed in

6	 .628

Healthcare

(.484) .486 (.500) .142 (.021)***

[+29%]

In Quarter	1	 .343 (.475) .103 (.303) .240 (.012)***

[+233%]

In Quarter	2	 .338 (.473) .087 (.282) .250 (.018)***

[+287%]

In Quarter	3	 .338 (.473) .085 (.279) .253 (.020)***

[+297%]

.253 (.015)*** In Quarter	4	 .334 (.472) .083 (.275) [+304%]

Note: The two left-hand columns report the mean and standard deviation for the treatment and the matched

comparison groups. The right-hand column reports the estimated program impact, with bootstrap standard

errors in parentheses; in brackets is the program impact as	 a	 percentage of the matched comparison group

mean. Statistical significance: ***	=	at	1	percent level.

Table	 9	 also	shows	 that	 the	program had positive	 effects	on healthcare employment	 rates.	In	

quarter 1, 34.3 percent of unemployed participants were employed in healthcare, compared	 to	

only 10.3 percent of the matched comparison group. This shows that the program led	to	a	24.0

percentage-point (233 percent) increase in healthcare employment rates	 at	 quarter	 1	 after

entry.	The	 percentage	impact increased over time,	as	 the	proportion of participants who were

employed in healthcare remained relatively constant in quarters	 2-4, while the proportion of

matched comparison group members employed in healthcare declined slightly. These results

show that the Health Careers program was very effective in promoting participant employment

in the healthcare industry, which was	 a	 main	 driver of the	 large	 program effects on overall

employment.
IMPAQ International, LLC Page 50 NFWS/SIF Quasi-Experimental Impact Study

Table	10 presents the program’s	impacts	on job retention	and	earnings.	Starting	with	 the	 job

retention	 rates	 of program participants, we find that 51.2 percent of participants found

employment in quarter	 1	 and	 remained employed in quarter	 2	 after	 entry.	 	 Job retention

declined over	 time	 to	 35.2 percent in quarter	 6	 –	 meaning	that 35.2 percent of participants

found employment in quarter	1	and remained employed in each of the six quarters after entry.

Comparing the job retention rates between the treatment and the matched comparison

groups, we find that the	 program	 led	 to	 large	 impacts	 on job retention in each of the six

quarters after	program	entry,	ranging from 43 percent in quarter	1	to	57 percent in quarter 6.

These results show that the program was not only effective in helping participants obtain

employment, but	also in helping participants to	sustain their jobs for	long periods after program

entry.	

The program led	 to	 large	 impacts	on employment, so we would expect that participants had

significantly higher earnings when compared with unemployed workers in	 the	 matched

comparison	 group. Table 10 confirms this expectation.	 In	 quarter 1, the	 average	 participant

earned $2,094,	an	amount that gradually increased,	reaching	$3,217 in quarter 6. Dividing the

average	 earnings amount by the	 overall	 employment rate (see	 Table	 10), we find that the

average	participant who was employed in quarter	1	earned	$3,635, which	gradually	increased

to	 $5,123 in quarter 6. As shown in the right-hand column of	 Table	10,	 program	participants

had	 significantly higher earnings than the matched comparison group throughout	 the	 six-	

quarter follow-up period.	 In	 quarter 1, treatment group members earned $868 (71 percent)

higher earnings relative	 to	the matched comparison group, an effect that gradually increased

over time, reaching $1,104 (52 percent) in quarter 6.
IMPAQ International, LLC Page 51 NFWS/SIF Quasi-Experimental Impact Study

Table 10: Program Impacts on Job Retention and Earnings, Health Careers

Treatment Matched Comparison Program Impact Group Group [% Comparison Group] Total Number of Participants 992 46,701

Job Retention

.153 (.018)*** .512 (.500) .359 (.480) Employed inQ1-2 [+43%]

.159 (.018)*** .472 (.499) .314 (.464) Employed inQ1-3 [+51%]

.153 (.020)*** .426 (.495) .273 (.446) Employed inQ1-4 [+56%]

.152 (.020)*** .400 (.490) .248 (.432) Employed inQ1-5 [+61%]

.128 (.017)*** .352 (.478) .224 (.417) Employed inQ1-6 [+57%]

Earnings($)

868 (146)*** In Quarter	1	 2,094 (3,822) 1,226 (2,362) [+71%]

851 (115)*** In Quarter	2	 2,473 (3,319) 1,622 (2,707) [+52%]

906 (162)*** In Quarter	3	 2,749 (3,692) 1,843 (2,966) [+49%]

870 (150)*** In Quarter	4	 2,760 (3,858) 1,889 (3,026) [+46%]

918 (186)*** In Quarter	5	 2,938 (4,269) 2,019 (3,065) [+46%]

1,104 (277)*** In Quarter	6	 3,217 (6,121) 2,113 (3,170) [+52%]

Note: The two left-hand columns report	the mean and	standard deviation for the treatment	and	the matched

comparison groups. The right-hand column reports the estimated program impact, with bootstrap standard

errors in parentheses; in brackets is the program impact as	a	percentage of the matched comparison group

mean. Statistical significance: ***	=	at	1	percent level.

Overall,	 these	 results show	 that	 the Health Careers	 program	 was very	 effective	 in helping

participants to obtain healthcare jobs, leading to	 large	 effects on overall employment rates.

Moreover, our results show that	the	program helped participants not only	to	obtain jobs soon

after	program	entry but	also	to	sustain them throughout the six-quarter	follow-up period. As	a	

result of the program’s	 large	 effects on employment and job retention,	 program	 participants

earned significantly higher	earnings than they would	have	earned	in	the	program’s absence.
IMPAQ International, LLC Page 52 NFWS/SIF Quasi-Experimental Impact Study

Advanced Manufacturing. Table 11 presents the impact results for overall employment and

employment in manufacturing. As shown,	42.2	percent	of	program	participants were employed

in quarter	 1	 after entry,	 a	 proportion that	 gradually	 increased over time	 to	 52.2 percent in

quarter 6. At the same	 time,	 we find that no more than 4.1 percent of participants were

employed in manufacturing, the	 program’s	 focus industry, in the four quarters after entry.

Dividing the manufacturing employment rate by	the	overall	employment	 rate	in quarters	1-4,

we find that only about	6	 to	10 percent of participants who found employment after program

entry	were employed in	the	program’s focus industry.

The impact results in	 Table	 11 show that the	 program	 led	 to	 positive effects on overall

employment.	 The program’s	 impact	 on employment in quarter	 1	 was 8.2	 percentage points,

which translates to	 a	 24 percent impact on employment. This impact grew over time and

remained statistically significant through	 quarter	 6	 following program entry, when treatment

group members were 12.2 percentage points (30 percent) more likely to be employed relative

to	 their matched comparison group peers.	 Table	 11	 also	 shows	 that,	 despite very few

participants obtaining jobs in manufacturing, the participant manufacturing employment	 rate

exceeded that of the matched comparison group by 2.0-2.8 percentage points (147-215

percent) over the four-quarter follow-up period. These results suggest that the largest portion

of program effects on	 overall	 employment is not attributable	 to	 program	 effects on

manufacturing employment.
IMPAQ International, LLC Page 53 NFWS/SIF Quasi-Experimental Impact Study

Table 11: Program Impacts on Employment,	Advanced Manufacturing

Treatment Matched

Program Impact Group ComparisonGroup [% Comparison Group] Total Number of Participants 682 42,293

Employed

.082 (.020)*** In Quarter	1	 .422 (.494) .340 (.474) [+24%]

.117 (.017)*** In Quarter	2	 .478 (.500) .362 (.481) [+32%]

.141 (.023)*** In Quarter	3	 .509 (.500) .369 (.483) [+38%]

.124 (.021)*** In Quarter	4	 .508 (.500) .384 (.486) [+32%]

.143 (.027)*** In Quarter	5	 .526 (.500) .382 (.486) [+37%]

.122 (.024)*** In Quarter	6	 .522 (.500) .401 (.490) [+30%]

Employed in Manufacturing

.025 (.007)*** In Quarter	1	 .041 (.199) .016 (.124) [+156%]

.028 (.004)*** In Quarter	2	 .041 (.199) .013 (.113) [+215%]

.025 (.006)*** In Quarter	3	 .042 (.200) .017 (.128) [+147%]

.020 (.007)*** In Quarter	4	 .032 (.176) .012 (.107) [+167%]

Note: The two left-hand columns report the mean and standard deviation for the treatment and the matched

comparison groups. The right-hand column reports the estimated program impact, with bootstrap standard

errors in parentheses; in brackets is the program impact as	a	percentage of the matched comparison group

mean. Statistical significance: ***	=	at	1	percent level.

Table	12 presents program	impacts	on job retention and earnings. As shown, 34.6 percent of

participants found	 a	 job in quarter	 1	 after entry	 and	 remained employed in quarter 2. Job

retention declined	to	19.9 percent in quarter 6, which shows that one in every five participants

found	a	job in quarter	1	and	remained employed throughout the six-quarter	follow-up period.

The right-hand column of Table 12 shows that participant job retention rates were higher than

the	 rates	 of the matched comparison group. For	 example,	 program participants were 4.0

percentage points (25 percent) more likely	 than	 matched	 comparison group members	 to	 be

continuously employed in	all	six quarters after program entry.

The program’s positive effects on employment	 and	 job retention yielded positive effects on
IMPAQ International, LLC Page 54 NFWS/SIF Quasi-Experimental Impact Study

earnings.	As shown in Table 12, program participants earned $1,146 in quarter 1, with average

earnings	 increasing in each quarter after entry,	 reaching	 $2,442 in quarter 6. Dividing these

amounts by the	overall	employment rate (see Table 11) shows that the average participant	who

was employed in quarter	 1	 earned $2,716, an amount that gradually increased	 to	 $4,678 in

quarter 6. Program effects, reported in the right column of Table 12, show that participants

earned $220 (24 percent) higher earnings in quarter	 1	 than matched comparison group

members. This difference grew over	time	and,	by	quarters	5	and 6, treatment group members

had	47	and	37 percent higher earnings, respectively.

Table 12: Program Impacts on Job Retention and Earnings, Advanced Manufacturing

Treatment Matched

Program Impact Group ComparisonGroup [% Comparison Group] Total Number of Participants 682 42,293

Job Retention

.076 (.019)*** .346 (.477) .269 (.444) Employed inQ1-2 [+28%]

.064 (.020)*** .288 (.453) .225 (.418) Employed inQ1-3 [+28%]

.048 (.019)*** .245 (.431) .198 (.398) Employed inQ1-4 [+24%]

.043 (.016)*** .217 (.413) .175 (.380) Employed inQ1-5 [+24%]

.040 (.016)** .199 (.400) .159 (.365) Employed inQ1-6 [+25%]

Earnings

220 (84)*** In Quarter	1	 1,146 (2,512) 925 (2,190) [+24%]

331 (130)** In Quarter	2	 1,580 (3,354) 1,249 (2,640) [+27%]

261 (166) In Quarter	3	 1,707 (3,141) 1,447 (3,028) [+18%]

409 (164)** In Quarter	4	 1,958 (3,324) 1,549 (3,063) [+26%]

752 (152)*** In Quarter	5	 2,354 (3,736) 1,602 (3,108) [+47%]

662 (155)*** In Quarter	6	 2,442 (3,956) 1,780 (3,357) [+37%]

Note: The two left-hand columns report	the mean and	standard deviation for the treatment	and	the matched

comparison groups. The right-hand column reports the estimated program impact, with bootstrap standard

errors in parentheses; in brackets is the program impact as	a	percentage of the weighted matched comparison

group mean. Statistical	significance: **	=	at	5	percent level; ***	=	at	1	percent level.
IMPAQ International, LLC Page 55 NFWS/SIF Quasi-Experimental Impact Study

The results for the Advanced Manufacturing program show that the program was effective in

helping participants obtain employment following	program	entry, with only	a	small proportion

of the employment effects attributable to manufacturing	employment.	The program	also	led to

positive effects on job retention, helping some participants	to	retain their jobs for long periods

after	 program entry. As	 a	 result, program participants had higher	 earnings than matched

comparison	group members throughout the entire six-quarter follow-up period.

Construction Partnership. Impact	 estimates for	 the	 Construction Partnership program on

employment	are	presented in Table 13. As	shown, 38.8	of program participants were employed

in quarter	 1	after entry, which increased over time, peaking	at	45.9 percent in quarter 4. We

also	 find that no more than 4.8 percent of participants were employed in construction, the

program’s	 focus industry.	 In	 fact,	 comparing	 construction employment rates	 to	 overall

employment rates, we find that only about one in every 10 participants who found

employment were employed in construction in quarters	1-4	after program entry.

The impact results in the right-hand column of	 Table	 13 show that	 the	 program	 had	 minor

positive effects on employment in quarters	2-4	after	program entry.	In	particular, the program

increased	 the	employment rates of participants by 5.5-6.1 percentage points (14-16 percent)

during that period. Although positive, point	 estimates	 of effects	 are	 appreciably smaller in

quarters	 1	 and	 5	 and	 are	 not statistically significant. Our	 analyses	 also	show	 that,	 although

construction employment rates in quarters	 1-4	 did not exceed 4.8 percent for program

participants, these	 rates	 were 3.3-3.7 percentage points (275-336 percent) higher than	 the

rates	of the matched comparison group.	These results	suggest that the program was somewhat

effective in	improving	overall	employment rates, which was driven by helping	a	few participants

obtain	jobs	in construction.
IMPAQ International, LLC Page 56 NFWS/SIF Quasi-Experimental Impact Study

Table 13: Program Impacts on Employment, Construction Partnership

Treatment Matched

Program Impact Group ComparisonGroup [% Comparison Group] Total Number of Participants 379 36,859

Employed

.032 (.029) In Quarter	1	 .388 (.488) .356 (.479) [+9%]

.061 (.028)** In Quarter	2	 .441 (.497) .380 (.485) [+16%]

.055 (.025)** In Quarter	3	 .449 (.498) .394 (.489) [+14%]

.057 (.027)** In Quarter	4	 .459 (.499) .402 (.490) [+14%]

.038 (.031) In Quarter	5	 .446 (.498) .408 (.491) [+9%]

.040 (.027) In Quarter	6	 .453 (.498) .414 (.493) [+9%]

Employed in Construction

.037 (.008)*** In Quarter	1	 .047 (.213) .011 (.104) [+336%]

.033 (.010)*** In Quarter	2	 .045 (.207) .012 (.109) [+275%]

.035 (.009)*** In Quarter	3	 .048 (.214) .013 (.111) [+269%]

.034 (.012)*** In Quarter	4	 .048 (.214) .011 (.104) [+309%]

Note: The two left-hand columns	report the mean and	standard deviation for the treatment and	the matched

comparison groups. The right-hand column reports the estimated program impact, with bootstrap standard

errors in parentheses; in brackets is the program impact as	a	percentage of the matched comparison group

mean. Statistical significance:	*	=	at 10 percent level; **	=	at	5	percent level; ***	=	at	1	percent level.

Table	14 presents the program’s effects on job retention and earnings. Only 29.0 percent of

participants were employed in quarter	 1	after entry and remained employed in quarter 2.

Job retention rates declined over time, with only 16.1 percent of participants finding

employment in quarter	 1	 and	 remaining employed throughout	 the	 six-quarter	 follow-up

period. Comparisons in the right-hand column of	Table	14 show that the job retention rates

of the treatment group were statistically equivalent	 to	those of the matched comparison

group.
IMPAQ International, LLC Page 57 NFWS/SIF Quasi-Experimental Impact Study

Table 14: Program Impacts on Job Retention and Earnings, Construction Partnership

Treatment Matched

Program Impact Group ComparisonGroup [% Comparison Group] Total Number of Participants 379 36,859

Job Retention

.009 (.026) .290 (.454) .281 (.449) Employed inQ1-2 [+3%]

.010 (.020) .251 (.434) .240 (.427) Employed inQ1-3 [+5%]

-.011 (.020) .200 (.400) .211 (.408) Employed inQ1-4 [-5%]

.006 (.020) .188 (.391) .181 (.385) Employed inQ1-5 [+4%]

-.002 (.024) .161 (.368) .162 (.369) Employed inQ1-6 [-1%]

Earnings

244 (121)** In Quarter	1	 1,070 (2,289) 830 (1,880) [+29%]

119 (151) In Quarter	2	 1,277 (2,487) 1,160 (2,304) [+10%]

104 (93) In Quarter	3	 1,381 (2,524) 1,280 (2,485) [+8%]

135 (90) In Quarter	4	 1,454 (2,464) 1,320 (2,495) [+10%]

81 (28)*** In Quarter	5	 1,518 (2,547) 1,434 (2,619) [+6%]

93 (126) In Quarter	6	 1,617 (2,665) 1,524 (2,665) [+6%]

Note: The two left-hand columns report	the mean and	standard deviation	for the treatment	and	the matched

comparison groups. The right-hand column reports the estimated program impact, with bootstrap standard

errors in parentheses; in brackets is the program impact as	a	percentage of the weighted matched comparison

group mean. **	=	statistically significant at	5	percent level.

Table	 14 shows that the	 average	 program	 participant earned $1,070 in quarter 1, which

increased over	 time	 to	$1,617 in quarter 6. Dividing	average	earnings amounts by	 the	overall

employment	 rate	 (see	 Table	 13), we find that	 average	 earnings	 for employed participants

increased from $2,758 in quarter	 1	 to	$3,570 in quarter 6. Program effects on earnings were

small, with program participants earning $244 and $81 higher than the matched comparison

group members in quarters	1	and	5; although positive, effects on earnings	were not statistically

significant	in the other four quarters of the	follow-up period.
IMPAQ International, LLC Page 58 NFWS/SIF Quasi-Experimental Impact Study

Overall,	 the	results show	 that	the Construction Partnership	 program	had minor effects on

employment, which were driven by placing	a	few participants in construction jobs. But the

program	was	ineffective in promoting job retention	and had modest impacts on earnings.

4.1.3 Discussion

In	this	section, we summarize	and	compare	the	impact	study results for the three	programs and

attempt to	 examine	 the underlying	 factors	 that	 may	 help interpret the results and	 explain	

variation	 in program effectiveness. Table 15 summarizes the impact results on employment.

The Health Careers and	Advanced	Manufacturing	program	led to significant positive effects on

overall	 employment, increasing employment rates by 29-37 percent	 and	 24-38 percent,

respectively, in the	six	quarters	following program	entry. The	Construction Partnership program

had	minor positive	effects on overall employment in quarters	2-4, but statistically insignificant

effects in the remaining quarters.

A	 key objective of	 the	 three	 programs	 was to promote the employment of unemployed

participants in their respective focus industries.	Results	show that the	Health	Careers program

was very	effective	in helping participants to obtain healthcare jobs	 –	on	average,	participants

were 24.0-25.3 percentage points (233-304 percent) more likely than matched comparison

group members	to	be employed in healthcare in quarters	1-4	after entry.	In	contrast,	although

statistically significant, the differentials for the	Advanced	Manufacturing	and	the Construction

Partnership	 programs	were	 2	to	 4	percentage points, and in each case less than	 5	percent of

participants	found employment	in the	program’s	focus industry.

These results suggest that	 the	Health Careers large	effects	on	 overall	employment	 rates	were

largely	 driven by	 its	 effectiveness in promoting participant employment in healthcare.	 In	

contrast, only	 a	 small portion of the	 large	 Advanced	 Manufacturing effects on overall

employment can be attributed	 to	manufacturing employment. This suggests that the program

provided participants with skills that were transferrable	 to	 non-manufacturing jobs		and/or

helped		 participants		 conduct		 a		 wider		 job		search,		 effectively		 helping		 them		 to		find		 non-	
IMPAQ International, LLC Page 59 NFWS/SIF Quasi-Experimental Impact Study

manufacturing jobs at	a	higher	 rate	than	 the	matched comparison group. By	 the	same token,

the	small Construction Pathways’	effects on overall employment were driven	largely	by	the	fact

that	 the	program	placed	a	few	participants	in construction jobs. However, it	appears	that the

skills obtained by	 the	program provided little help for participants in finding non-construction

jobs.

Table 15: Summary of Program Impacts on Employment

Health Advanced Construction

Careers Manufacturing Partnership

Employed

.141 (.012)*** .082 (.020)*** .032 (.029) In Quarter	1	 [+32%] [+24%] [+9%]

.170 (.016)*** .117 (.017)*** .061 (.028)** In Quarter	2	 [+37%] [+32%] [+16%]

.167 (.015)*** .141 (.023)*** .055 (.025)** In Quarter	3	 [+35%] [+38%] [+14%]

.150 (.019)*** .124 (.021)*** .057 (.027)** In Quarter	4	 [+31%] [+32%] [+14%]

.150 (.019)*** .143 (.027)*** .038 (.031) In Quarter	5	 [+31%] [+37%] [+9%]

.142 (.021)*** .122 (.024)*** .040 (.027) In Quarter	6	 [+29%] [+30%] [+9%]

Employed in FocusIndustry

.240 (.012)*** .025 (.007)*** .037 (.008)*** In Quarter	1	 [+233%] [+156%] [+336%]

.250 (.018)*** .028 (.004)*** .033 (.010)*** In Quarter	2	 [+287%] [+215%] [+275%]

.253 (.020)*** .025 (.006)*** .035 (.009)*** In Quarter	3	 [+297%] [+147%] [+269%]

.253 (.015)*** .020 (.007)*** .034 (.012)*** In Quarter	4	 [+304%] [+167%] [+309%]

Note: Reported is the	estimated program impact with bootstrap standard	errors in parentheses; in brackets

is the program impact		as a	 percentage of the weighted	 matched		comparison		group		mean.		Statistical

significance: **	=	at	5	percent	level; ***	= at	1	percent level.

Table	 16 summarizes program effects on job retention and earnings. The Health Careers

program	was very effective in improving job retention, with program participants being 43-61

percent more likely than those in the matched comparison group to obtain employment in

quarter	1	after entry	and	remain	employed in subsequent	quarters.	These	results show that the

program	was not only effective in helping participants	 to	obtain healthcare jobs, but	 also	to
IMPAQ International, LLC Page 60 NFWS/SIF Quasi-Experimental Impact Study

remain	employed for long periods	after program	entry.	Advanced	Manufacturing led	to	positive

effects on job retention (24-28 percent), although these were lower than the effects of	 the

Health Careers program.	This implies that, while	the	vast	majority of participants were not	able	

to	 obtain manufacturing jobs, the skills obtained from the program helped	 them	 find

sustainable non-manufacturing jobs.	In	contrast,	the	Construction Partnership program did not

affect job retention, which is consistent with	 the	 view that the program did not provide

participants with transferrable skills that helped them access non-construction jobs.

Table 16: Summary of Program Impacts on Job Retention and Earnings

Health Advanced Construction

Careers Manufacturing Partnership

Job Retention

.153 (.018)*** .076 (.019)*** .009 (.026)

Employed inQ1-2 [+43%] [+28%] [+3%]

.159 (.018)*** .064 (.020)*** .010 (.020)

Employed inQ1-3 [+51%] [+28%] [+5%]

.153 (.020)*** .048 (.019)*** -.011 (.020)

Employed inQ1-4 [+56%] [+24%] [-5%]

.152 (.020)*** .043 (.016)*** .006 (.020)

Employed inQ1-5 [+61%] [+24%] [+4%]

.128 (.017)*** .040 (.016)** -.002 (.024)

Employed inQ1-6 [+57%] [+25%] [-1%]

Earnings

868 (146)*** 220 (84)*** 244 (121)** In Quarter	1	 [+71%] [+24%] [+29%]

851 (115)*** 331 (130)** 119 (151) In Quarter	2	 [+52%] [+27%] [+10%]

906 (162)*** 261 (166) 104 (93) In Quarter	3	 [+49%] [+18%] [+8%]

870 (150)*** 409 (164)** 135 (90) In Quarter	4	 [+46%] [+26%] [+10%]

918 (186)*** 752 (152)*** 81 (28)*** In Quarter	5	 [+46%] [+47%] [+6%]

1,104 (277)*** 662 (155)*** 93 (126) In Quarter	6	 [+52%] [+37%] [+6%]

Note: Reported is the program estimated	impact, with bootstrap standard errors in parentheses; in brackets

is the program’s impact as	 a	 percentage of the weighted matched comparison group mean. Statistical

significance: **	=	at	5	percent	level; ***	= at	1	percent level.

Finally,	as	shown in	Table	16, the effects on earnings reflected an individual program’s success

in improving employment and job retention. The Health Careers program	 –	 which had the
IMPAQ International, LLC Page 61 NFWS/SIF Quasi-Experimental Impact Study

largest effects on	overall	employment, employment in focus industry, and	job retention	–	led	to

substantial effects on earnings throughout the six-quarter	 follow-up period. The Advanced

Manufacturing	program	also	had	positive effects on earnings, reflecting the program’s success

in helping participants to improve their overall employment and job retention rates. But the

program’s	effects on earnings were lower than the Health	Careers	program’s	effects,	likely tied

to	Advanced Manufacturing’s being less successful in promoting participant employment in	its	

focus industry	and	improving job retention.	Finally,	 the	Construction Partnership	program	had

very small effects on	 earnings,	 which is consistent with	 its	 limited success in improving

participant employment	and	job retention.

A	few underlying factors	may	help	explain	differences in	a	program’s success. First,	there	were

notable differences in the service delivery process. Health Careers offered	 a	 wide	 range	 of

services, including job readiness training, NCRC preparation assistance, healthcare-focused

training, and job search	assistance.	The program’s staff worked individually with participants to

help	 them	 assess	 which services would best help	 achieve	 their goals. Participants with low

education and limited work experience could participate in job readiness training	and	receive

NCRC	assistance	 to	 improve their employability before	engaging	in industry-specific training.25

Participants with higher levels of education and more stable employment histories (particularly

those with experience in healthcare) could participate in industry-focused training upon

program	 entry	 to	 help them access mid-level healthcare careers.26 This design ensured that

services	 catered	 to	 individual needs in: (1) helping low-skill, inexperienced	 	 participants	 	 to

obtain skills	 to	 improve	 their	employability	and	(2) helping high-skill, experienced participants

to	obtain skills	to	access careers in healthcare.

The Advanced Manufacturing program used	an	 incremental approach	 to	providing services, in

which	all	program participants were first offered job readiness training and, upon	completion,

NCRC	 and	 MSSC certification training. Those successful in obtaining these certificates were

25 As shown in Table 4, 27 percent of participants received job readiness training. 26		As shown in Table 4, 68 percent of participants received healthcare-specific occupational training; consistent

with program design, only	3	percent of participants received both job readiness	and occupational training.
IMPAQ International, LLC Page 62 NFWS/SIF Quasi-Experimental Impact Study

offered academic	 and	 career advancement services. The	 program’s participants could receive

job search	 assistance	services	at	any	 time	during	 the	program period. As discussed in Section

3.1.5, 73 percent of participants received job readiness training,	 and	 82 percent received

employment services; only 23 percent received occupational training. These figures show that

participants were most interested in improving	 their	 employability skills	 and	 finding	 a	 job,

rather than receiving training	 to	 find jobs in manufacturing. This is consistent with	 the

program’s results discussed above.

The Construction Partnership	 program	 used	 a	 different	 approach.	 Its	 career pathways model

was	 primarily	 based	 on pre-apprenticeship programs	 and	 on-the-job training.	 In	 addition,

similar to	the	other two	programs,	Construction Partnership offered job search assistance. The

program’s limited effects on employment, job retention,	and	earnings	suggest that its relatively

narrow	focus had	a	minor effect on participant employability in construction and did not offer

them	skills	to	obtain	jobs in other	sectors.

Another	distinguishing	characteristic	of the Health Careers	program	is that	its	partners	included

large	employers (i.e.,	large	hospital systems) with very specific occupational-training needs and

a	well-defined job demand. These employers could hire	many	appropriately	trained	individuals

graduating from the program. Although the	 Advanced	 Manufacturing	 and	 Construction

Partnership	programs	partnered with	a	large pool of employers with specific workforce needs,

many	 were small employers lacking the capacity	 to	 hire	 many	 program graduates. These

differences	may	have	had	a	bearing on	 the	limited effectiveness of these	programs	in helping

participants access and retain high-quality jobs in their focus industries.

Third, disparities in program success	may	be	partly	attributed	to	the three	programs	focused	on

three distinct industries.	 Healthcare	 jobs are	 typically	 less volatile	 and	 have	 lower

unemployment rates than manufacturing and construction jobs, particularly during periods of

high unemployment. As shown in Table 17,	the	unemployment	rate	for the healthcare industry

in Ohio was 5.1 percent in 2010	–	much lower than the overall unemployment	rate	in	the	state
IMPAQ International, LLC Page 63 NFWS/SIF Quasi-Experimental Impact Study

(10.7 percent). The manufacturing unemployment	rate	was 11.7 percent, one percentage	point

higher than	 the	statewide rate, and more than twice	 the	healthcare rate. The unemployment

rate	in construction was 19.8 percent,	nearly	four times higher than for healthcare and nearly

twice than for manufacturing.27

Table 17: Ohio Unemployment	Rates	and Employment Projections

Healthcare Manufacturing Construction Overall

Unemployment Rate, 2010 5.1% 11.7% 19.8% 10.7%

Total Employment, 2010 726,730 620,450 168,660 5,368,900

(% of overall employment in state) (14%) (12%) (3%) (100%)

Employment Projections, 2010-20 +183,150 –2,840 +38,880 +498,100

(% change over 2010) (25%) (<1%) (23%) (+9%)

Unionization Rate, 2010 4.8% 13.8% 12.3% 11.0%

Note: Unemployment and unionization rates are based on authors’ tabulations of the 2010 American Community

Survey. Total employment and employment projections are	from	the Ohio Job Outlook of	the Ohio Department of

Job and Family Services	(http://ohiolmi.com/proj/OhioJobOutlook.htm).

Table	17	also	illustrates	the	importance of	 the	three sectors in the Ohio workforce. As shown,

there were 5.4 million workers in Ohio in 2010, of whom 726,730 (14 percent) were employed

in healthcare, 620,450 (12 percent) in manufacturing, and 168,660 (3 percent) in construction.

Employment projections of the Ohio Department of Job and Family Services show that the

healthcare industry was expected	 to	grow by 183,150 workers by 2020,	a	25 percent increase

from 2010. The same projections show that manufacturing was expected	 to	shrink by about

one percent	and	construction to grow by 23 percent.	These figures indicate that	healthcare	and

construction in Ohio were expected	 to	 have	higher than average growth in employment from

2010 through 2020, while manufacturing employment was expected	to	remain about	the	same.

It	 is	 also	 important to note that manufacturing	 and	 healthcare were more	 heavily	 unionized

than healthcare jobs. As shown in Table 17, only 4.8 percent of healthcare workers in Ohio

were represented by unions, compared with 13.8 percent	 and	12.3 percent of manufacturing

and		construction		workers,		respectively.		These	 figures	 suggest		that		it		may		be	 easier		for

27 All unemployment	rates indicated here and in Table 17 are unemployment	rates for the experienced labor force.

In 2010, the national unemployment rate was 10.1 percent, including 5.3 percent for healthcare, 10.7 percent for

manufacturing, and 17.9 percent for construction.
IMPAQ International, LLC Page 64 NFWS/SIF Quasi-Experimental Impact Study

unemployed workers to access healthcare jobs	 than	 manufacturing or construction jobs in

Ohio, even	if	they	have	the appropriate	skills and credentials for those	jobs.

Based on these figures,	the	greater impacts estimated for the Health Careers program relative

to	 the	 other two programs	 may	 be partly attributable	 to	 differences in the nature of the

programs’	 focus industries. Health	 Careers	 focused on	 an	 industry that: (1)	 had	 very low

unemployment; (2) employed	a	relatively	large	share of the state’s workforce; (3) was	expected	

to	 grow substantially; and (4) was less unionized.	 In	 contrast, Advanced Manufacturing and

Construction Partnership	programs	were targeting industries with much higher unemployment

rates	and	a	much stronger union presence.

Finally, disparities in program success	may	be attributed to Health Careers	attracting	different

types	of participants. For instance, Health Careers attracted higher	proportions	of participants

with more than	a	high school	education,	extensive	prior work experience,	and	experience in	the

program’s	 focus industry. The other two programs	 primarily	 attracted participants with no

more than	a	high school education, less stable work experience, and limited experience in the

program’s	focus	industry.	It	is possible that	highly skilled, experienced workers were more likely

to	 benefit from industry-focused training than low-skill workers	 with	 weak prior work

experience.

4.2 Wisconsin Results

This section		presents the		quasi-experimental impact results for		the three		Wisconsin-based

NFWS/SIF-funded programs. The section begins with	 a	 discussion	 of	 the	 matching results,

followed by	 a	 presentation of the impact results	 and	 a	 discussion of the findings and their

interpretation.	A	summary of the	results	is provided	in	Box 4.
IMPAQ International, LLC Page 65 NFWS/SIF Quasi-Experimental Impact Study

BOX 4: SUMMARY OF	QUASI-EXPERIMENTAL IMPACT RESULTS IN WISCONSIN

WRTP Construction	Pathways

§ The program had large effects on	 	 employment.	 	 In the six quarters after program entry,

67.2-72.7 percent of participants were employed, exceeding the employment rates of

matchedcomparisongroupmembersby8.8-21.7percentagepoints(14-43percent).

§ The program was not effective in helping participants to obtain construction jobs. In the six

quarters after entry, fewer than three percent of participants worked in construction.

§ The program had large effects on job retention rates. About 44.5 percent of participants

found employment in quarter	 1	 and remained employed in each of the six quarters after

entry, compared to only 29.4 percent of matched comparison group members,	 a	 15.1	

percentage point (51 percent) difference.

§ The program had large effects on earnings. In the six-quarter follow-up period, participants

had	$11,237 (56 percent) higher earnings than	the	matched comparison	group.

WRTP	Manufacturing Pathways

§ The program had large effects on	 	 employment.	 	 In the six quarters after program entry,

66.3-72.1 percent of participants were employed, exceeding the employment rates of

matchedcomparisongroupmembersby11.1-28.6percentagepoints(21-68percent).

§ The program helped participants to obtain manufacturing jobs. In the six quarters after entry,

32.6-40.7 percent of participants were employed in manufacturing, exceeding the

manufacturing employment rates of the matched comparison group by 23.7-29.6 percentage

points (202-535 percent).

§ The program had large effects on job retention. About 43.0 percent of participants found	 a	

job in quarter	1	and remained employed in each of the six quarters after entry, compared to

19.8 percent of matched comparison	group members, a	118 percent	difference.

§ The program had substantial effects on earnings. In the six-quarter follow-up period,

participants had	$16,661 (134 percent) higher earnings than	the matched comparison	group.

Milwaukee	Healthcare Alliance

§ The program had large effects on	 	 employment.	 	 In the six quarters after program entry,

57.1-77.0 percent of participants were employed, exceeding the employment rates of

matchedcomparisongroupmembersby16.5-19.4percentagepoints(28-40percent).

§ The program helped participants to obtain healthcare jobs. In the six quarters after entry,

27.9-47.4 percent of participants were employed in healthcare, exceeding the healthcare

employment rates of the matched comparison group by 18.1-29.6 percentage points

(155-196 percent).

§ The program had large effects on job retention. About 40.1 percent of participants found	 a	

job in quarter	1	and remained employed in each of the six quarters after entry, compared to

22.3 percent of matched comparison	group members, an	80 percent difference.

§ The program had positive effects on earnings. In the	six-quarter follow-up period, participants

had	$5,418 (55 percent) higher earnings than	the	matched	comparison	group.
IMPAQ International, LLC Page 66 NFWS/SIF Quasi-Experimental Impact Study

4.2.1 Matching Results

Similar to	the	Ohio study, matched comparison groups for each of	the	three NFWS/SIF-funded

programs	were constructed	using Wisconsin ES data.	Table	18 summarizes	the	characteristics	of

unemployed NFWS/SIF participants and of unemployed workers in	the	ES data.

Table 18: Characteristics of Unemployed NFWS/SIF and ES Participants in Wisconsin

WRTP WRTP Healthcare ES Construction Manufacturing Alliance

Unemployed	Participants 1,103	(100%) 88 (100%) 306 (100%) 35,482	(100%)

Gender

Men 998 (90%) 74 (84%) 17 (6%) 18,495 (52%)

Women 105 (10%) 14 (16%) 289 (94%) 16,987 (48%)

Race/Ethnicity

White 424 (38%) 27 (31%) 18 (6%) 22,807 (64%)

Black 448 (41%) 50 (57%) 227 (74%) 8,059 (23%)

Other Race 72 (7%) 4	(5%) 32 (10%) 2,224 (6%)

Hispanic 159 (14%) 7	(8%) 5	(2%) 2,392 (7%)

Education

No High School Diploma 197 (18%) 18 (20%) --	 4,014 (11%)

High School Diploma 873 (79%) 68 (77%) --	 16,870 (48%)

Associate Degree, Some College 22 (2%) 2	(2%) --	 9,102 (26%)

College Degree 11 (1%) 0	(0%) --	 5,496 (15%)

Age

Less than 25 Years 214 (19%) 11 (13%) 119 (39%) 5,243 (15%)

25-34 Years 363 (33%) 24 (27%) 101 (33%) 9,293 (26%)

35-44 Years 251 (23%) 19 (22%) 29 (9%) 7,264 (20%)

45-54 Years 191 (17%) 22 (25%) 26 (9%) 7,711 (22%)

55-64 Years 70 (6%) 10 (11%) 11 (4%) 5,069 (14%)

65+ Years 14 (1%) 2	(2%) 20 (7%) 902 (3%)

Veteran 38 (3%) 2	(2%) --	 1,869 (5%)

Program Entry --	

Quarter 1, 2010 7	(1%) --	 --	 3,930 (11%)

Quarter 2, 2010 22 (2%) --	 --	 3,955 (11%)

Quarter 3, 2010 11 (1%) 2	(2%) --	 3,792 (11%)

Quarter 4, 2010 6	(1%) 3	(3%) --	 3,675 (10%)

Quarter 1, 2011 206 (19%) 24 (27%) --	 4,568 (13%)

Quarter 2, 2011 225 (20%) 24 (27%) 41 (13%) 4,340 (12%)

Quarter 3, 2011 389 (35%) 19 (22%) 123 (40%) 4,088 (12%)

Quarter 4, 2011 237 (21%) 16 (18%) 94 (31%) 4,332 (12%)

Quarter 1, 2012 --	 --	 48 (16%) 2,802 (8%)

Note: Reported is the number of participants with sample proportion in parentheses.
IMPAQ International, LLC Page 67 NFWS/SIF Quasi-Experimental Impact Study

Important differences in characteristics exist between each NFWS/SIF-funded program and ES.

WRTP	 Manufacturing had much higher proportions of	 male,	 black,	 and	 Hispanic participants

than the ES, and much lower proportions of participants with more than	 a	 high school

education and	 age	 45 or older. Similarly, compared	 to	 the ES population, much higher

proportions of	WRTP	Manufacturing participants were	 male,	 black, and with no more than	 a	

high school education. At the same time, women, nonwhites, and individuals younger than age

35 were overrepresented in	 the	 Health Careers participant population relative	 to	 the ES

population.

Table	 19 presents the employment	 history	 of unemployed workers in	 the	 three NFWS/SIF-	

funded programs and ES. As shown, employment rates in prior quarters for ES participants

were similar	 to	 those of	 WRTP	 Construction and of Health Careers participants, but slightly

higher than those of WRTP Manufacturing participants. Also, ES	 participants	were	as	 likely	 as	

WRTP	Construction and Health Careers participants, and more likely than	WRTP	Manufacturing

participants, to have continuous employment prior to program entry.

The most notable differences between NFWS/SIF participants and ES participants were in prior

employment in focus industry and earnings. Participants in each of the three NFWS/SIF-funded

programs	 were significantly	 	 more likely	 to	 have prior	 employment	 in	 	 the	 program’s focus

industry than were ES participants. Moreover,	 WRTP	 Manufacturing	 and	 Health Careers

participants had lower prior earnings than ES participants over	 the	entire eight-quarter	period

prior to program entry. Overall,	 the	 disparities in socioeconomic and prior employment

characteristics	between participants	in each NFWS/SIF-funded	program	and ES participants	–	as	

presented in	Tables	18 and 19	–	suggest that the ES sample	cannot	be used in	its	original	form

as	a	comparison group for	any	of the three programs.
IMPAQ International, LLC Page 68 NFWS/SIF Quasi-Experimental Impact Study

Table 19: Employment History of Unemployed NFWS/SIF and ES Participants in Wisconsin

WRTP WRTP Healthcare ES Construction Manufacturing Alliance

Total Number of Participants 1,103	(100%) 88 (100%) 306 (100%) 35,482	(100%)

Employment

In Prior Quarter	1	 666 (60%) 44 (50%) 168 (55%) 19,404 (55%)

In Prior Quarter	2	 664 (60%) 50 (57%) 184 (60%) 21,187 (60%)

In Prior Quarter	3	 664 (60%) 49 (56%) 181 (59%) 22,011 (62%)

In Prior Quarter	4	 655 (59%) 44 (50%) 184 (60%) 22,721 (64%)

In Prior Quarter	5	 622 (56%) 48 (55%) 185 (61%) 23,243 (66%)

In Prior Quarter	6	 646 (59%) 48 (55%) 187 (61%) 23,553 (66%)

In Prior Quarter	7	 665 (60%) 50 (57%) 189 (62%) 23,732 (67%

In Prior Quarter	8	 669 (61%) 51 (58%) 169 (59%) 23,992 (68%)

Prior Employment

In Both Quarters	1-2	 569 (48%) 38 (43%) 148 (48%) 17,675 (50%)

In All Quarters	1-4	 456 (41%) 26 (30%) 127 (42%) 14,736 (42%)

No Prior Employment

In Quarters	1-2	 342 (31%) 32 (36%) 102 (33%) 12,566 (35%)

In Quarters	1-4	 258 (23%) 23 (26%) 75 (25%) 8,357 (24%)

Prior Employment in FocusIndustry

In Quarter	1	(Healthcare) --	 --	 70 (23%) 2,537 (7%)

In Quarter	1-4	(Healthcare) --	 --	 96 (31%) 3,898 (11%)

In Quarter	1	(Manufacturing) --	 15 (17%) --	 2,611 (7%)

In Quarter	1-4	(Manufacturing) --	 23 (26%) --	 3,211 (9%)

In Quarter	1	(Construction) 305 (28%) --	 --	 692 (2%)

In Quarter	1-4	(Construction) 432 (39%) --	 --	 929 (3%)

Earnings Amount ($)

In Prior Quarter	1	 3,790 (4,981) 2,234 (3,129) 1,577 (2,249) 3,306 (5,579)

In Prior Quarter	2	 4,168 (5,398) 2,565 (4,045) 1,792 (2,277) 3,813 (6,196)

In Prior Quarter	3	 4,384 (5,559) 2,941 (4,585) 2,031 (2,523) 4,053 (6,401)

In Prior Quarter	4	 4,424 (5,912) 2,526 (3,829) 1,968 (2,477) 4,265 (8,171)

In Prior Quarter	5	 4,451 (6,138) 2,578 (3,634) 2,077 (2,631) 4,449 (6,280)

In Prior Quarter	6	 4,647 (6,364) 2,840 (4,061) 2,101 (2,606) 4,519 (6,531)

In Prior Quarter	7	 4,965 (6,462) 3,494 (5,197) 2,102 (2,622) 4,580 (5,982)

In Prior Quarter	8	 4,884 (6,203) 3,811 (5,350) 2,046 (2,802) 4,658 (6,108)

Note:		Reported		is		the	 number	 of		participants	 with		sample	 proportion	 in	 parentheses;		for		prior	 earnings,

reported is the sample mean with standard deviation in parentheses.

To construct	an	appropriate, matched comparison group for each program, we used the same

matching	approach used in	the	Ohio study. The formal details of	this	approach	are	provided in

Section 2.2.4.	 Using	a	logistic model, we estimated the likelihood of program participation and
IMPAQ International, LLC Page 69 NFWS/SIF Quasi-Experimental Impact Study

produced	the	propensity score for each participant and non-participant in the data. Cases	that

were	off the	common	support	of	the	propensity score	were	omitted,	and	the	logistic	model	was

re-estimated on	 the	remaining sample	 to	produce the propensity score for	all	cases that were

on	 the	 common support.28 We then weighted each comparison case by the odds	 ratio	of the

propensity score, so that	the	weighted comparison sample had the same characteristics	as	the

treatment sample. Balancing	 tests	 were used	 to	 compare	 the treatment and the matched

comparison	 sample; where differences were detected, we repeated the above process

modifying the specification	of	the	logistic model until the matching was successful.29

The results of the balancing	tests	(see	Tables	D, E,	and	F	in	the	Appendix) show	that	for each of

the	three programs, the matched comparison group was similar in observable characteristics

and	 prior employment	 to	 the	 treatment group. Balancing tests for	 WRTP	 Construction

(Appendix Table D) show	 that	 of 45 characteristics, only three had statistically	 significant

differences (two	at	the 10 percent level,	one	at the 5	percent level) between the treatment and

matched	comparison	groups (45-54	years;	program	entry	in quarter	1, 2010; and	employment

in prior quarter 3). Balancing	tests	for	WRTP	Manufacturing	and	Healthcare	Alliance	(Appendix

Tables	 E	 and F, respectively)	 reveal	 no statistically significant differences between treatment

and	 matched comparison cases. These	 tests	 show equivalence between treatment and

matched comparison cases in	 all	three programs, with	 a	few exceptions regarding the WRTP

Construction program. We wanted to ensure that these differences do not inject bias in our

28 In implementing the matching process for each program, we omitted: (1) 12 treatment and 5,991 comparison

cases for WRTP Construction; (2)	2	treatment and 15,638 comparison cases for WRTP Manufacturing; and (3) 19

treatment and 26,816 comparison cases for Healthcare Alliance. The small number of treatment cases omitted

ensures that bias due to failure to match participants to comparison cases is negligible. Omission of comparison

cases in the matching process does not induce bias, given that our focus is on program impacts for participants. 29		At the conclusion of this process, the following control variables were used in the logit model: 1) individual

characteristics: gender, race, education, age, and quarter of program entry; 2) employment history measures:

employment in each prior quarter	1-8, prior employment in both quarters	1-2, prior employment in all quarters

1-4, no prior employment in both quarters	1-2, no prior employment in any quarter	1-4, prior employment in

focus industry in prior quarter 1, prior employment in focus industry in prior quarters	1-4, and earnings in prior

quarters	1-8; and 3) interactions between gender and race, gender and age, gender and education, gender and

quarter of entry, gender and prior earnings, gender and prior employment in focus industry, race and age, race and

education, race and quarter of entry, race and prior earnings, race and prior employment in focus industry,

education and age, education and quarter of entry, education and prior earnings, education, and prior

employment in focusindustry.
IMPAQ International, LLC Page 70 NFWS/SIF Quasi-Experimental Impact Study

results. Therefore, in addition	 to	 using outcome	 means	 comparisons	 to	 estimate program

effects, we applied the same bias-adjustment method used in the Ohio	 analyses.	 The bias-	

adjusted impacts were equivalent	to	the impact	estimates	reported below, supporting	the view

that the few differences in characteristics for WRTP Construction did not affect the validity of

the	matching process.

4.2.2 Impact Results

Program	 impacts were	 estimated	 by comparing	 the	 mean outcomes between the treatment

and	 the matched comparison groups. To	 assess	 their	statistical significance, we produced	 t-	

tests	 based on bootstrap standard errors. Impact results for each program	 are	 presented

below.

WRTP Construction Program	 Impacts.	 Table	 20 presents the impact results on employment

and	employment in construction. We	find that 72.7 percent of	program	participants (treatment

group) were employed	in quarter	1	after entry,	a	proportion that declined slightly but remained

in the 67.2-70.7 percent range in quarters	 2-6. Comparisons with the matched comparison

group show that	 the	 program was effective in increasing employment	 rates	 throughout the

follow-up period. In quarter	 1	 after entry, 72.7 percent of treatment group members were

employed, compared with 51.0 percent of matched comparison group members. As the right-	

hand	 column shows, the 21.7 percentage-point difference was statistically significant	 at	 the	 1

percent level, which means that participants were 43 percent more likely	 than	those in the

matched comparison group	to	be employed	at	quarter 1. The	program’s	effect on employment

declined over time, but remained statistically and substantively important	through	quarter 6.
IMPAQ International, LLC Page 71 NFWS/SIF Quasi-Experimental Impact Study

Table 20: Program Impacts on Employment,	WRTP Construction P

Treatment Matched Comparison

Group Group [% Total Number of Participants 1,091 29,491

Employed

athways

Program Impact

Comparison Group]

In Quarter 1	 .727 (.446) .510 (.500) .217 (.027)***

[+43%]

In Quarter 2	 .705 (.456) .521 (.500) .184 (.028)***

[+35%]

In Quarter 3	 .672 (.470) .571 (.495) .101 (.028)***

[+18%]

In Quarter 4	 .700 (.458) .590 (.492) .110 (.026)***

[+19%]

In Quarter 5	 .706 (.456) .618 (.486) .088 (.025)***

[+14%]

In Quarter

Employed in

6	 .707

Construction

(.455) .600 (.490) .107 (.023)***

[+18%]

In Quarter	1	 .027 (.161) .024 (.152) .003 (.005)

[+12%]

In Quarter	2	 .024 (.153) .029 (.167) -.005 (.006)

[-17%]

In Quarter	3	 .019 (.137) .029 (.168) -.010 (.005)*

[-34%]

In Quarter	4	 .022 (.147) .034 (.180) -.012 (.005)**

[-34%]

In Quarter	5	 .029 (.169) .035 (.183) -.006 [.007]

[-17%]

-.004 [.007] In Quarter	6	 .031 (.174) .035 (.183) [-11%]

Note: The two left-hand columns report the mean and standard deviation for the treatment and the matched

comparison group. The right-hand column reports the estimated program impact, with bootstrap standard

errors in parentheses; in brackets is the program impact as	 a	 percentage of the matched comparison group

mean. Statistical significance: ***	=	at	1	percent level.

The same	table	shows that very	few participants found construction jobs, with the construction

employment	 rate	 not exceeding 3.1 percent in	 the	 six-quarter	 follow-up period. Comparing

these	to	the matched comparison group, we find that the	program	did not lead	to	any	positive

effects on construction employment. These results show that, while the program effectively

promoted	overall	employment rates, these effects	are	not	attributable	to	the	program’s placing

participants in construction	jobs.
IMPAQ International, LLC Page 72 NFWS/SIF Quasi-Experimental Impact Study

Table	21 presents program	 impacts	on job retention and earnings. The	 table	shows that 64.3

percent of participants were employed in both quarter	 1	 and	 quarter 2. Job retention rates

declined over time, with 44.5 percent of participants finding	a	job in quarter	 1	and remaining

employed in each quarter throughout the six-quarter follow-up period. Compared	 to	 the

matched comparison group, we find that the program helped participants	obtain	employment

in quarter	 1	 after program	 entry	 and	 remain employed in multiple quarters thereafter. For

example,	64.3 percent of treatment group members were employed in both quarters	 1	and	 2

after	entry, compared	to	42.3 percent of matched comparison group members. The difference

was statistically significant, which shows that treatment group members were 52 percent more

likely	to	find	and	sustain employment for two quarters after entry. The program’s effect on job

retention was retained	 through	 quarter 6; 44.5 percent of treatment group members were

employed in	 all	 six quarters after	 program	 entry,	 compared	 with 29.4 percent of matched

comparison	group members,	a	51 percent increase. These results show that the program was

not only effective in helping participants obtain jobs, but	 also	 able	 to	retain the jobs for long

periods after entry.

The program’s positive effects on employment and job retention led to large	positive	effects on

earnings.	 The average participant earned $5,087 in quarter	1	after entry, with	average	earnings

increasing over time, reaching $5,677 in quarter 6. Dividing average earnings by the overall

employment rate (see Table 20), we find that the	 average	 participant who was employed in

quarter	 1	 earned $6,997, increasing	 to	 $8,029 by	 quarter	 6. Compared	 to	 the matched

comparison	group,	the	average participant	had	$2,137	(72 percent)	higher earnings	in quarter	1

after	program entry. The program’s effect on earnings was positive and	statistically	significant

in quarters	2–6, ranging from 32 to 78 percent.
IMPAQ International, LLC Page 73 NFWS/SIF Quasi-Experimental Impact Study

Table 21: Program Impacts on Job Retention and Earnings, WRTP Constru

Treatment Matched Comparison

Group Group [% Total Number of Participants 1,091 29,491

Job Retention

ction Pathways

Program Impact

Comparison Group]

Employed inQ1-2 .643 (.479) .423 (.494) .220 (.029)***

[+52%]

Employed inQ1-3 .555 (.497) .367 (.482) .188 (.029)***

[+51%]

Employed inQ1-4 .503 (.500) .334 (.472) .169 (.033)***

[+51%]

Employed inQ1-5 .467 (.499) .315 (.465) .152 (.035)***

[+48%]

Employed inQ1-6

Earnings($)

.445 (.497) .294 (.455) .151 (.032)***

[+51%]

In Quarter 1	 5,087 (5,551) 2,950 (5,234) 2,137 (485)***

[+72%]

In Quarter 2	 4,999 (5,477) 2,810 (3,975) 2,189 (248)***

[+78%]

In Quarter 3	 4,746 (5,591) 3,586 (4,725) 1,161 (417)***

[+32%]

In Quarter 4	 5,282 (5,682) 3,640 (5,008) 1,641 (419)***

[+45%]

In Quarter 5	 5,641 (5,907) 3,616 (4,766) 2,024 (342)***

[+56]

2,085 (311) In Quarter	6	 5,677 (5,876) 3,592 (4,588) [+58%]

Note: The two left-hand columns report the mean and standard deviation for the treatment and the matched

comparison groups. The right-hand column reports the estimated program impact, with bootstrap standard

errors in parentheses; in brackets is the program impact as	 a	 percentage of the matched comparison group

mean. Statistical significance: ***	=	at	1	percent level.

These results show that, despite	 WRTP	 Construction’s ineffectiveness in helping participants

obtain employment in construction,	 the	program	 had	positive effects on overall employment.

The program	also	helped	participants obtain sustainable jobs,	as	suggested	by	the	job retention

rates	through	the	entire six-quarter follow-up period. As	a	result, participants	had	much higher

earnings	than non-participants in the matched	comparison	group.

WRTP Manufacturing Program	 Impacts.	 Program	 impacts	 on employment	 are	 presented in

Table	22. About 70.9 percent of program participants were employed in quarter	1	after entry.

Participant		employment		rates		were		relatively		steady		thereafter,		at		66.3-72.1		percent		in
IMPAQ International, LLC Page 74 NFWS/SIF Quasi-Experimental Impact Study

quarters	 2–6. Table 22	 also	shows	 that	 many	 participants were employed in healthcare jobs

after	program entry.	In	quarters	1-2	after entry, 32.6 percent of participants were employed in

manufacturing jobs,	 a	 proportion that increased slightly over	 time,	 reaching	 as	 high	 as	 40.7

percent in quarter 5. Dividing the manufacturing employment	 rate	by	 the	overall	employment

rate,	we find that about	 half	the treatment group participants who found jobs after program

entry	were	employed	in	the	program’s focus industry.

Comparisons with the matched comparison group show that the program	 had	 large	 positive

effects on employment in each of	the	six quarters	after	program entry.	In	quarter	1	after entry,

program	 participants were 28.6 percentage	 points	 (68 percent) more likely	 to	 be employed

than unemployed workers in	the	matched comparison group. The program’s	effects	on	overall	

employment declined in each subsequent quarter, but remained large	 and	 statistically

significant.

The program	 also	 had	 significant effects on manufacturing employment. For	 example,	 32.6

percent of participants found employment in manufacturing in quarter 1, compared with just

5.1 percent of matched	 comparison	 group members. The 25.3	 percentage-point difference

implies that program	 	 participants were more than five times more likely	 to	 find	 a	

manufacturing job relative	 to	 non-participants. Throughout the follow-up period, treatment

matched comparison group differences in manufacturing employment rates remained large and

statistically	significant, with program participants	at	least three	times	more likely	than	matched

comparison	 group members	 to	 be employed in manufacturing.	 In	 fact,	 the program’s

percentage-point effects on manufacturing employment were similar or exceeded the	 effects	

on	 overall	 employment,	 suggesting	 that the program’s effects on employment were	 largely	

driven by	the	program’s effectiveness in promoting employment in manufacturing.
IMPAQ International, LLC Page 75 NFWS/SIF Quasi-Experimental Impact Study

Table 22: Program Impacts on Employment,	WRTP Manufactu

Treatment Matched Comparison

Group Group [% Total Number of Participants 86 19,844

Employed

ring

Program Impact

Comparison Group]

In Quarter 1	 .709 (.457) .423 (.494) .286 (.054)***

[+68%]

In Quarter 2	 .663 (.476) .466 (.499) .199 (.046)***

[+42%]

In Quarter 3	 .698 (.462) .478 (.500) .212 (.056)***

[+46%]

In Quarter 4	 .721 (.451) .511 (.500) .208 (.041)***

[+41%]

In Quarter 5	 .698 (.462) .526 (.499) .173 (.048)***

[+33%]

In Quarter

Employed in

6	 .663

Manufacturing

(.476) .548 (.498) .111 (.062)*

[+21%]

In Quarter	1	 .326 (.471) .051 (.220) .253 (.052)***

[+535%]

In Quarter	2	 .326 (.471) .067 (.250) .237 (.054)***

[+386%]

In Quarter	3	 .349 (.479) .069 (.254) .258 (.056)***

[+403%]

In Quarter	4	 .384 (.489) .089 (.284) .275 (.069)***

[+333%]

In Quarter	5	 .407 (.494) .111 (.314) .296 (.067)***

[+267%]

.249 (.058)*** In Quarter	6	 .372 (.486) .123 (.328) [+202%]

Note: The two left-hand columns report the mean and standard deviation for the treatment and the matched

comparison groups. The right-hand column reports the estimated program impact, with bootstrap standard

errors in parentheses; in brackets is the program impact as	 a	 percentage of the matched comparison group

mean. Statistical significance: ***	=	at	1	percent level.

Table	23	presents program	impacts	on job	retention	and	earnings. Contrary	to	the	results of the

WRTP	Construction	 program,	WRTP	Manufacturing led to significant effects on job	 retention.	

About 61.6 percent of treatment group members were employed in quarters	 1	 and	 2	 after

entry.	 Job retention	 rates	 remained relatively high throughout the follow-up period	 –	 43.0

percent of participants were employed in each of the six quarters after program entry.

Participants’ job retention rates were 81-126 percent higher	than	the job retention rates of the
IMPAQ International, LLC Page 76 NFWS/SIF Quasi-Experimental Impact Study

matched		comparison		group.	 That		shows		that		the		program		was		very		effective		in		helping

participants find sustainable jobs.

Table 23: Program Impacts on Job Retention and Earnings, WRTP Manufacturing

Treatment Matched Comparison Program Impact Group Group [% Comparison Group] Total Number of Participants 86 19,844

Job Retention

.277 (.058)*** .616 (.489) .341 (.474) Employed inQ1-2 [+81%]

.282 (.060)*** .547 (.501) .264 (.441) Employed inQ1-3 [+107%]

.243 (.052)*** .477 (.502) .232 (.422) Employed inQ1-4 [+106%]

.265 (.051)*** .477 (.502) .211 (.408) Employed inQ1-5 [+126%]

.231 (.054)*** .430 (.498) .198 (.398) Employed inQ1-6 [+118%]

Earnings($)

3,434 (482)*** In Quarter	1	 4,783 (4,818) 1,336 (2,531) [+258%]

3,062 (487)*** In Quarter	2	 4,846 (4,637) 1,768 (4,846) [+174%]

2,690 (456)*** In Quarter	3	 4,724 (4,473) 2,022 (3,035) [+134%]

2,584 (497)*** In Quarter	4	 4,796 (4,425) 2,203 (3,331) [+118%]

2,679 (542)*** In Quarter	5	 5,138 (4,630) 2,484 (3,460) [+107%]

2,212 (456)*** In Quarter	6	 4,802 (4,565) 2,590 (3,524) [+85%]

Note: The two left-hand columns report the mean and standard deviation for the treatment and the matched

comparison group. The right-hand column reports the estimated program impact, with bootstrap standard

errors in parentheses; in brackets is the program impact as	 a	 percentage of the matched comparison group

mean. Statistical significance: ***	=	at	1	percent level.

Finally, the program led	to	substantial effects on	earnings.	In quarter 1, the	average	participant

earned $4,783,	an	amount that remained	relatively steady throughout the six-quarter	follow-up

period. Dividing	average	earnings	by	the	overall	employment	rates	(from Table 22), we find that

participants who found employment after program entry	had	average	quarterly earnings in the

$6,746-$7,361 range. Compared	 to	 the matched comparison group, participants had

substantially higher earnings in each quarter after entry.	 In	 quarter 1,	 the	program led	 to	 a	
IMPAQ International, LLC Page 77 NFWS/SIF Quasi-Experimental Impact Study

$3,434 increase in earnings, which	 means	 that participants	 had	 more	 than	 2.5 times the

earnings	 of those in the matched comparison group. The program’s	 effects	 on earnings

followed	a	declining patter over	time	that was	similar	to	the program’s effects on employment.

Nevertheless, effects on	earnings	remained substantial and statistically significant in the entire

six-quarter follow-up period, with treatment group members earning $2,212 (85	 	 percent)

higher earnings in quarter	6	relative to their peers.

Overall,	 these	 results show that the	 WRTP	 Manufacturing program was very successful in

helping participants	 to	 obtain employment in manufacturing, leading	 to	 positive effects on

overall	 employment rates. Moreover, the program	 had	 large effects on job retention, which

shows	that	the	program	helped participants obtain jobs that were sustainable for long periods

after	program entry. The program’s effectiveness in promoting employment and job retention

yielded	 large	positive effects on earnings; in fact,	 the	 program’s	effects on earnings were the

largest	across	the	six programs examined	in	this	study.

Healthcare Alliance Program	 Impacts.	 Table 24 presents the program’s	 impacts	 on

employment. We observe that 57.1 percent of program participants were employed in quarter

1, with employment	 rates	 increasing in each subsequent quarter, reaching 76.0 percent in

quarter 6. Large proportions of participants were employed in healthcare.	 In	 quarter	 1	 after

entry,	 27.9 percent of participants were employed in healthcare, which indicates that nearly

half	of	all	employed treatment group members in quarter	 1	were employed in	 the	program’s

focus industry. Healthcare	employment rates	and	their	importance	in overall	employment rates

gradually	increased over	time,	reaching 47.4 percent in quarter 6.

The right-hand column of	 Table	 24 shows that	 the	 program led	 to	 positive	 and	 significant

effects on overall employment in each of the six quarters after	program	entry.	 The program’s

effect on employment was 16.5 percentage points (40 percent) and 19.4 percentage points (41

percent) in quarters	 1	 and	 2	 after entry, respectively. The program’s effects on employment

rates	remained statistically	and	substantively important in quarters	3-6: 28-34 percent. Table
IMPAQ International, LLC Page 78 NFWS/SIF Quasi-Experimental Impact Study

24	 also	shows that	 the	 program	 was very effective in helping participants	 to	 obtain jobs in

healthcare. Compared with matched comparison group members, treatment group members

were	 at	 least 155 percent more likely	 to	 be employed in	 a	 healthcare job in each of the six

quarters after entry. These results suggest that	 the	 program’s success in improving	 overall	

employment is tied	to	the	program’s helping participants	to	obtain jobs in healthcare.

Table 24: Program Impacts on Employment, Healthcare Alliance

Treatment Matched Comparison

Group Group [% Total Number of Participants 287 8,666

Employed

Program Impact

Comparison Group]

In Quarter 1	 .571 (.500) .407 (.491) .165 (.032)***

[+40%]

In Quarter 2	 .669 (.471) .475 (.500) .194 (.031)***

[+41%]

In Quarter 3	 .679 (.468) .530 (.499) .149 (.024)***

[+28%]

In Quarter 4	 .753 (.432) .561 (.496) .192 (.024)***

[+34%]

In Quarter 5	 .770 (.422) .587 (.492) .183 (.026)***

[+31%]

In Quarter

Employed in

6	 .760

Healthcare

(.428) .594 (.491) .166 (.026)***

[28%]

In Quarter	1	 .279 (.449) .098 (.297) .181 (.022)***

[185%]

In Quarter	2	 .376 (.485) .127 (.333) .249 (.033)***

[+196%]

In Quarter	3	 .383 (.487) .137 (.344) .246 (.032)***

[+180%]

In Quarter	4	 .449 (.498) .170 (.375) .280 (.030)***

[+165%]

In Quarter	5	 .470 (.500) .174 (.379) .296 (.027)***

[+170%]

.288 (.030)*** In Quarter	6	 .474 (.500) .186 (.389) [+155%]

Note: The two left-hand columns report the mean and standard deviation for the treatment and the matched

comparison group. The right-hand column reports the estimated program’s impact, with bootstrap standard

errors in parentheses; in brackets is the program impact as	 a	 percentage of the matched comparison group

mean. Statistical significance: ***	=	at	1	percent level.
IMPAQ International, LLC Page 79 NFWS/SIF Quasi-Experimental Impact Study

The program’s effects on job retention	and	earnings	are	summarized in	Table	25. We see that

52.6 percent of participants found employment in quarter	 1	 after	 entry	 and remained

employed in quarter 2. Job retention rates remained	 large	 throughout the entire follow-up

period. Comparing job retention rates between the treatment and the matched comparison

group, we find that		the		program led	 to	 positive	 and		statistically significant effects on		job

retention throughout the six-quarter follow-up period. Case in	point,	40.1 percent of treatment

group	members were employed in	each of	the six quarters after program entry, compared with

22.3 percent of matched comparison group members, an 80 percent difference.

Table 25: Program Impacts on Job Retention and Earnings, Healthcare

Treatment Matched Comparison

Group Group [% Total Number of Participants 287 8,666

Job Retention

Alliance

Program Impact

Comparison Group]

Employed inQ1-2 .526 (.500) .340 (.474) .186 (.033)***

[55%]

Employed inQ1-3 .467 (.500) .282 (.450) .185 (.029)***

[66%]

Employed inQ1-4 .443 (.498) .255 (.436) .188 (.032)***

[+74%]

Employed inQ1-5 .429 (.496) .236 (.425) .192 (.033)***

[+82%]

Employed inQ1-6

Earnings($)

.401 (.491) .223 (.416) .177 (.037)***

[+80%]

In Quarter 1	 1,690 (2,160) 947 (1,754) 743 (122)***

[79%]

In Quarter 2	 2,159 (2,306) 1,335 (2,135) 825 (157)***

[+62%]

In Quarter 3	 2,301 (2,464) 1,550 (2,291) 750 (149)***

[+48%]

In Quarter 4	 3,003 (2,795) 1,829 (2,566) 1,174 (203)***

[+64%]

In Quarter 5	 2,982 (2,666) 2,023 (2,666) 960 (176)***

[+47%]

966 (171)*** In Quarter	6	 3,106 (2,811) 2,140 (2,732) [+45%]

Note: The two left-hand columns report the mean and standard deviation for the treatment and the matched

comparison groups. The right-hand column reports the estimated program impact, with bootstrap standard

errors in parentheses; in brackets is the program impact as	 a	 percentage of the matched comparison group

mean. Statistical significance: ***	=	at	1	percent level.
IMPAQ International, LLC Page 80 NFWS/SIF Quasi-Experimental Impact Study

The program led	 to	 positive	 effects	 on earnings in each of	 the	six quarters after entry. The

average	participant earned $1,690 in quarter	1	after entry, which gradually increased	to	$3,106

by quarter 6. Compared	to	matched comparison group members, program participants earned

significantly higher earnings in each of the six quarters after entry.	 In	particular, the program

led	 to	a	$743 (79 percent) increase in average earnings in quarter	1	after entry, an effect that

was	statistically	significant at the	1	percent level. The	program’s	effect	on earnings was	at	least

as	great	and	as	statistically significant	in	all	subsequent quarters.

Overall,	 the	 above results show that the Healthcare	 Alliance	 program	 was very	 effective	 in

helping participants to obtain jobs in healthcare, leading	 to	 positive	 effects on	 overall

employment. Importantly,	the	jobs that programs participants were	able	to	obtain	as	a	result	of

program	participation were sustainable. As	a	result, the average participant	 had	earnings that

exceeded the earnings of unemployed workers in	the	matched comparison group.

4.2.3		 Discussion

This section summarizes	 and	 compares	 the	 results	 	 for	 the three programs, attempting to

examine	underlying factors	 to	explain	differences in	program	results.	Table	26 summarizes the

program	 effects on	 employment.	 All three programs	 had	 large	 positive	 effects	 on	 overall

employment rates in the six quarters after program entry.	 WRTP	 Manufacturing had the

greatest effects on employment, leading	 to	 a	 68 percent increase in quarter	 1	 and	 a	 33-41

percent	 increase	 in quarters	 2-6.	 WRTP	 Construction	 and	 Healthcare	 Alliance	 raised

employment rates by 43	and	40	percent,	respectively, in quarter	1	after entry. Program effects

declined over time, but remained substantively important, with	 the	 WRTP Construction	 and	

Healthcare Alliance programs increasing employment rates by 14-35 percent and 28-41

percent,	respectively,inquarters	2-6.

The bottom	panel	of	Table	26	also	shows	that	the	WRTP	Manufacturing and	Healthcare	Alliance

programs	 were very effective in promoting participant employment in	 their	focus industries.

WRTP	 Manufacturing increased manufacturing employment	 rates	 by 23.7-29.6 percentage
IMPAQ International, LLC Page 81 NFWS/SIF Quasi-Experimental Impact Study

points in the six quarters	after	program	entry,	meaning that participants were	at	least twice	as	

likely	as	their	comparison group peers	to	find manufacturing jobs. Similarly, Healthcare Alliance

increased healthcare employment rates by 18.1-29.6 percentage points, meaning that

participants were	at	least 155 percent more likely	 to	be employed in healthcare than those in

the	 matched comparison group.	 In	 contrast,	 WRTP	 Construction was	ineffective	 in helping

participants to	find jobs in construction.

Table 26: Summary of Program Impacts on Employment

WRTP WRTP Healthcare

Construction Manufacturing Alliance

Employed

.217 (.027)*** .286 (.054)*** .165 (.032)*** In Quarter	1	 [+43%] [+68%] [+40%]

.184 (.028)*** .199 (.046)*** .194 (.031)*** In Quarter	2	 [+35%] [+42%] [+41%]

.101 (.028)*** .212 (.056)*** .149 (.024)*** In Quarter	3	 [+18%] [+46%] [+28%]

.110 (.026)*** .208 (.041)*** .192 (.024)*** In Quarter	4	 [+19%] [+41%] [+34%]

.088 (.025)*** .173 (.048)*** .183 (.026)*** In Quarter	5	 [+14%] [+33%] [+31%]

.107 (.023)*** .111 (.062)* .166 (.026)*** In Quarter	6	 [+18%] [+21%] [28%]

Employed in FocusIndustry

.003 (.005) .253 (.052)*** .181 (.022)*** In Quarter	1	 [+12%] [+535%] [185%]

-.005 (.006) .237 (.054)*** .249 (.033)*** In Quarter	2	 [-17%] [+386%] [+196%]

-.010 (.005)* .258 (.056)*** .246 (.032)*** In Quarter	3	 [-34%] [+403%] [+180%]

-.012 (.005)** .275 (.069)*** .280 (.030)*** In Quarter	4	 [-34%] [+333%] [+165%]

-.006 [.007] .296 (.067)*** .296 (.027)*** In Quarter	5	 [-17%] [+267%] [+170%]

-.004 [.007] .249 (.058)*** .288 (.030)*** In Quarter	6	 [-11%] [+202%] [+155%]

Note: Reported is the estimated	program impact	with	bootstrap	standard errors in	parentheses; in	brackets

is the program impact as	 	 a percentage of the weighted matched comparison group mean. Statistical

significance: **	=	at	5	percent	level; ***	= at	1	percent level.
IMPAQ International, LLC Page 82 NFWS/SIF Quasi-Experimental Impact Study

These results indicate that the effects on overall employment produced by the WRTP

Manufacturing and Healthcare Alliance programs were	 largely	 attributable	 to	 the two

programs’	 effectiveness in	 placing	 participants in jobs in their respective focus industries. On

the	 other hand,	 WRTP	 Construction	 having	 no effect on employment in construction implies

that the program’s positive effects on	overall	employment were driven by	participants’	finding

non-construction jobs at	 a	 higher rate than non-participants. This indicates	 that	 the services

offered by	WRTP	Construction provided participants with transferrable skills	and/or	 assistance

in conducting an effective	search for non-construction jobs.

Table	 27 summarizes the	 impacts	 on job retention	 and	 earnings.	 All three	 programs	 were

effective in helping participants	to	find jobs soon	after	program	entry and retain those jobs for

long periods.	WRTP	Manufacturing participants were 81-126 percent more	likely than those	in

the	matched comparison group	to	find	a	job in quarter	1	and remain employed in subsequent

quarters. These results imply that	the	program did not just	help	participants find manufacturing

jobs	and	improve their overall employment, but helped them find sustainable jobs, ensuring	a	

more stable employment path. Similarly,	WRTP	Construction	and	Healthcare	Alliance	increased

participants’ job retention rates by 48-52 and 55-82 percent, respectively. For Healthcare

Alliance,	 these results imply that the	 program	 achieved	 its	 goal	 of placing participants in

promising, sustainable jobs in the healthcare industry.	WRTP	 Construction’s effects on job

retention imply that the skills obtained by participants helped	 them	 obtain non-construction	

jobs more sustainable than the jobs they	would have obtained in the program’s absence.

Finally, Table 27 shows that	all	three	programs	had positive effects on average earnings in the

entire six-quarter follow-up period.	WRTP	Manufacturing	–	which	had	greater effects	on overall

employment, focus industry employment, and	job retention than the other two	programs	–	had

substantial effects on earnings. Driven by	 its	 positive effects on overall employment and job

retention, WRTP Construction	 also	 had positive effects on earnings. This implies that the

program’s participants got better	 paying	 non-construction jobs than those in	 the	 matched

comparison			group.		 At		 the			same		 time,		 the		 Healthcare			Alliance			had			much			lower,			but
IMPAQ International, LLC Page 83 NFWS/SIF Quasi-Experimental Impact Study

substantively important, effects on earnings than the other two programs, reflecting the

program’s	effectiveness in placing participants in jobs earlier than if they left	to	search for	a	job

without receiving program services.

Table 27: Summary of Program Impacts on Job Retention and Earnings, Wisconsin

WRTP WRTP Healthcare

Construction Manufacturing Alliance

Job Retention

.220 (.029)*** .277 (.058)*** .186 (.033)***

Employed in Q1-2 [+52%] [+81%] [55%]

.188 (.029)*** .282 (.060)*** .185 (.029)***

Employed in Q1-3 [+51%] [+107%] [66%]

.169 (.033)*** .243 (.052)*** .188 (.032)*** Employed in Q1-4 [+51%] [+106%] [+74%]

.152 (.035)*** .265 (.051)*** .192 (.033)***

Employed in Q1-5 [+48%] [+126%] [+82%]

.151 (.032)*** .231 (.054)*** .177 (.037)***

Employed in Q1-6 [+51%] [+118%] [+80%]

Earnings

2,137 (485)*** 3,434 (482)*** 743 (122)*** In Quarter	1	 [+72%] [+258%] [79%]

2,189 (248)*** 3,062 (487)*** 825 (157)*** In Quarter	2	 [+78%] [+174%] [+62%]

1,161 (417)*** 2,690 (456)*** 750 (149)*** In Quarter	3	 [+32%] [+134%] [+48%]

1,641 (419)*** 2,584 (497)*** 1,174 (203)*** In Quarter	4	 [+45%] [+118%] [+64%]

2,024 (342)*** 2,679 (542)*** 960 (176)*** In Quarter	5	 [+56] [+107%] [+47%]

2,085 (311) 2,212 (456)*** 966 (171)*** In Quarter	6	 [+58%] [+85%] [+45%]

Note: Reported is the estimated program impact, with bootstrap standard errors in parentheses; in brackets

is the program impact as	 a	 percentage of the weighted matched comparison group mean. Statistical

significance: **	=	at	5	percent	level; ***	= at	1	percent level.

The Wisconsin NFWS/SIF-funded	 program	 results show that	 all	 three	 programs	 were very

effective in helping participants	 to	obtain sustainable jobs following program entry, leading	 to	

positive effects on earnings. There	 are	two notable differences in program results. First, the

success of	 WRTP	 Manufacturing	 and	 Healthcare	 Alliance	 was	 tied	 to	 the	 	 programs’

effectiveness in	 placing	 participants in manufacturing	 	 and healthcare jobs, respectively.	 In	

contrast,	WRTP	Construction was not effective in placing participants in construction jobs, but
IMPAQ International, LLC Page 84 NFWS/SIF Quasi-Experimental Impact Study

did help participants to	 obtain	 non-construction jobs	 at	 a	 higher rate than they would in	 the

program’s	 absence. Second, WRTP Manufacturing had greater effects on	 all	 outcomes of

interest than did the other two programs.

Interestingly, differences in	 program	 effects cannot be directly tied	 to	 the service delivery

process.	 In	 fact,	 WRTP	 Construction and WRTP Manufacturing followed the same model,

consisting of: (1) pre-apprenticeship training	 to	 help low-skill, inexperienced workers to	 gain	

on-the-job training; (2)	 assistance	 to	 job-ready, experienced workers	 to	 obtain certificates in

skilled trades; and (3) career advancement training	to	those who	are	successful in finding	a	job.

These services	are	tied	with the programs’ success in promoting	overall	employment rates, job

retention, and earnings, but cannot explain why the	WRTP	 Manufacturing had higher overall

impacts.	The Healthcare	Alliance	program offered unemployed participants the opportunity to

receive industry-specific classroom	 training	 and job search services,	 aimed	 at	 helping them

obtain healthcare jobs. This differed from	 the	design of the two	WRTP	programs, but it is not

clear why the Healthcare Alliance	 program’s design led	 to	markedly better results than WRTP

Construction	and	to	slightly weaker effects than	WRTP	Manufacturing in	placing	participants in

jobs	in the program’s focus industry.

Perhaps	a	more plausible explanation for WRTP Manufacturing and Healthcare	 Alliance	having

greater	 effects than WRTP Construction in placing workers in jobs in	 their	 respective focus

industries is their relationship with their employer partners. The former two programs

cooperated closely with	their	employer partners	to	develop curriculums that catered	to	specific

employer needs and	to	create	a	pipeline for placing participants who	completed	the	training	in

available	jobs. Employer partners in	WRTP	Construction had	a	less intensive involvement in the

program’s design and participant job placement.

In	thinking about	 variations	 in program results, we	 note	the potential role of the	 nature	 and

structure of the programs’ focus industries, and the particular economic environment	 at	the

time.	 Similar	 to	 Ohio, the healthcare unemployment	 rate	 was 4.3 percent, about half the
IMPAQ International, LLC Page 85 NFWS/SIF Quasi-Experimental Impact Study

manufacturing rate (8.6 percent) and more than four	 times	 lower than the	 construction	 rate

(18.0 percent). These figures	 may	 explain	 why	 the	 construction program was unsuccessful in

placing	 participants in construction jobs. The very high unemployment	 rates	 in construction

reflect the fact that construction employment declined	 substantially	during	 the	recession	 and	

that	 many	 construction workers were experiencing long-term	 unemployment. This suggests

that	 WRTP	 Construction participants would have	 to	 compete with many former construction

workers in Wisconsin for	a	limited number of jobs.

Although manufacturing was	the	largest of the three sectors in the ratio of total employment in

the	 state, it had the lowest projected growth.	 In	 particular,	 453,620 (15 percent) of the

3,051,328 workers in Wisconsin were employed in manufacturing in 2012. Employment

projections of the Wisconsin Department of Workforce Development show	that	manufacturing

is expected	 to	 grow by two percent through 2022.	 In	 contrast, healthcare accounted for	 a	

slightly lower proportion of	the	workforce in 2012, but was expected to grow by 15 percent by

2022	 –	 more than double the projected		average workforce growth for	 the	 entire state (7

percent). Construction accounted for only three percent of the workforce in 2012, but	had	the

highest projected job growth (18 percent).

Table 28: Wisconsin Unemployment Rates and Employment Projections

Healthcare Manufacturing Construction Overall

Unemployment Rate, 2010 4.3% 8.6% 18.0% 8.7%

Total

(% of

Employment, 2012

overall employment in state)

375,019

(12%)

453,620

(15%)

93,197

(3%)

3,051,328

(100%)

Employment Projections,

(% change over 2012)

2012-2022 430,649

(+15%)

462,784

(+2%)

110,310

(+18%)

3,269,173

(+7%)

Unionization Rate, 2010 12.6% 19.0% 14.0% 16.6%

Note: Unemployment and unionization rates are based on authors’ tabulations of the 2010 American Community

Survey. Total employment and employment projections are provided by the Wisconsin Department of Workforce

Development					 (http://wisconsinjobcenter.org/labormarketinfo/).

Table	 28 shows	 that	 healthcare	 and	 construction were less heavily unionized than

manufacturing. Since the most successful program focused on the industry with the highest
IMPAQ International, LLC Page 86 NFWS/SIF Quasi-Experimental Impact Study

unionization rate, it would appear that	any	union limitations on	access	to jobs does not	play	an	

important role in explaining differences	in relative success	across	programs.

Comparing	the	characteristics of unemployed participants in the	three programs shows	that	the

demographic profiles of the	 WRTP	 Construction	 and	 WRTP Manufacturing participants were

roughly similar	–	the two	programs	attracted large proportions of participants who were	male,

were nonwhite, had no more than	 a	 high school education,	 and	 were under	 age	 45.

Participants in the Healthcare	 Alliance	 program were	 almost	 exclusively women (94 percent),

but similar	 to	the	other two programs, the majority of participants were nonwhites and under

age	 45. Comparing the prior employment histories of participants, we find that WRTP

Manufacturing participants had	a	relatively	less	stable	employment history	and	were less likely

to	 have	 experience than those in the other two	 programs.	Overall, there is	 little	 evidence	 to	

suggest that differences in industry structure	and	in participants’ characteristics can be used to

explain differences in the programs’ success.
IMPAQ International, LLC Page 87 NFWS/SIF Quasi-Experimental Impact Study

5. Conclusion

Since	its	establishment in 2007, NFWS	has	supported	a	wide range of programs that promote

the	 employment and career advancement of low-skill	 individuals	 in in-demand industries. At

the	 beginning of 2010, NFWS was supporting 30	 active	 programs	 that were responsible for

identifying employer workforce needs in their local	areas	and developing workforce programs

providing	 training	 and	 other	services	 to	 low-income workers	 to	 prepare	 them	to meet those

needs. NFWS efforts were enhanced by	 a	 two-year $7.7 million SIF	 grant	 awarded in 2010,

which was partly used	to	support scaling up the operations of those 30 programs.

This report presented the results of	a	quasi-experimental	impact	study of six NFWS/SIF-funded

programs	 –	 the Health Careers, Advanced Manufacturing, and Construction Partnership

programs	 in Ohio, and the	WRTP	Construction,	WRTP	Manufacturing, and Healthcare Alliance

programs	 in Wisconsin.	 Results	 for the three Ohio-based	 programs	 show that the	 	 Health

Careers	program	was very	effective	in placing participants in healthcare jobs,	leading	to	positive

effects on	overall	employment	rates,	job retention,	and	earnings. The Advanced Manufacturing

and	 Construction Partnership programs helped very few participants to obtain jobs in their

respective focus industries. Nevertheless, Advanced Manufacturing	 positively	 affected overall

employment rates, job retention, and earnings.	In	contrast, Construction Partnership had small

effects on overall employment	and	earnings.

Results	for the three Wisconsin-based	programs	show that	all	three programs were effective in

improving participant outcomes following program	 entry.	 WRTP	Manufacturing was the most

effective of the six programs. It was very successful in	 placing	 participants in manufacturing

jobs, substantially affecting	overall	employment rates, job retention, and	earnings.	 The results

of	 the	WRTP	Manufacturing program in Wisconsin were markedly different from those of the

Advanced	 Manufacturing	 program	 in Ohio, with the latter being not effective in helping

participants	find manufacturing	jobs	and having relatively lower effects on	overall	employment,

job retention, and earnings. The Healthcare Alliance program was found	 to	 be effective in

helping participants obtain healthcare jobs, and	thus	improved	overall	employment rates, job
IMPAQ International, LLC Page 88 NFWS/SIF Quasi-Experimental Impact Study

retention rates, and earnings.	In	fact, the results of this program were similar	 to	the	results of

the	Health Careers program in Ohio. As with	the	Construction Partnership	program,	the WRTP

Construction program	was not effective in placing	participants in construction jobs. Still, we find

that the Wisconsin program led	 to	positive effects on	 overall	employment, job retention, and

earnings	–	substantially	greater	than the effects observed for	the	Ohio program.

Finally, it is	natural	to	ask	how	these	results	compare	with those obtained in other high-quality

studies of training programs. Because of their focus on particular industries, these programs

differ from most other programs studied	 to	date. However, like the programs examined here,

other	 training programs studied attempt	 to	 provide disadvantaged workers with	 training	 for

jobs	expected	to	be	in greatest demand. The bulk	of	training	programs’	effects	estimates come

from U.S.	program	evaluations	of the Workforce Investment Act programs	and	its	predecessor,

the	 Job Training Partnership Act. The estimates of employment effects	 are	 5–29 percentage

points (measured monthly or	quarterly),	with some differences observed between women and

men, and by specific training	type	and time following	program	entry (Card et al., 2010; Decker,

2011; Heinrich	et al., 2013).

The six programs we	studied	are	in	this	range. Construction Partnership produced increments in

employment in the range of five percentage points, whereas the Ohio Health Careers and

Advanced	Manufacturing	programs	produced improvements in the range of 10–15 percentage

points. Similarly,	 the	 WRTP	 Construction and	 WRTP	 Manufacturing	 programs	 showed

increments exceeding 20	 percentage	points, but the effect declined to around 10 percentage

points	 at	 the	 end of	 the	 six-quarter follow-up period. The Healthcare	 Alliance	 showed

increments in	the	range of 15–20 percentage points.

Studies examining the effects of training programs on	earnings	suggest benefits that range from

about $300–$900 per quarter for participants,	 based	 on 2006	 dollars	(Andersson et al.	 2013;

Bloom et al., 2003; Decker, 2011; Heinrich et al., 2013). Some studies also translate earnings

effects into percentage terms, with estimated effects (earnings increases) of training programs
IMPAQ International, LLC Page 89 NFWS/SIF Quasi-Experimental Impact Study

in	the	United States and	abroad	ranging from about	5–26 percent (Bloom et al., 2003; Decker,

2011; Greenberg et al., 2005; Haelermans and	 Borghans,	 2011; Heinrich et al., 2013).	 In	the

analyses reported here, Ohio’s Construction Partnership showed very modest gains, generally

less than $200 per quarter, which were statistically significant in only two of	 the	six quarters.

The two other Ohio programs showed effects	ranging	from $200–$1,100 per quarter. Although

Wisconsin’s Healthcare	 Alliance	 showed earnings	 effects	 in the	 same	 range,	 the two	 WRTP	

programs	 raised	 earnings by	 appreciably	more, often exceeding $2,000 in	 a	 quarter, in some

cases implying that earnings more than doubled.

Overall, the results of this study provide important insights on	 the	effectiveness of NFWS/SIF-	

funded programs focusing on healthcare, manufacturing, and construction.	 	 Healthcare

programs	are	likely	to	be effective in helping participants obtain jobs in healthcare,	an	industry

with specific workforce needs, relatively low unemployment rates, and high growth potential.

At the same	 time,	training	 programs	focusing on manufacturing	 are	found	 to	 be effective in

helping participants find sustainable jobs and improve their earnings. However, we get mixed

results on the effectiveness of such programs in promoting employment in the manufacturing

sector. Training programs focusing on construction jobs	 are	 found	 to	 be ineffective in

promoting participants’ employment in construction, and	 have	 lower	 overall	 effects on

employment, job retention,	 and	 earnings than programs focusing on healthcare and

manufacturing. Finally, it appears that program	 estimates	 of	 overall	 employment rates and

earnings	for all but one of the six NFWS/SIF-funded programs studied here	–	particularly for the

two	 WRTP	 programs	 in Wisconsin	 –	 compared	 favorably	 with estimates obtained for other

training	programs	studied to date.
IMPAQ International, LLC Page 90 NFWS/SIF Quasi-Experimental Impact Study

References

Andersson F., Holzer H.J., Lane J.L., Rosenblum D.,	and	Smith J. (2013). Does Federally-Funded

Job Training Work? Nonexperimental Estimates of WIA Training	 Impacts	 Using Longitudinal

Data on Workers and Firms.	NBER Working	Paper	No. w19446.

Angrist J.	 and	 Pischke J. Mostly Harmless Econometrics. Princeton, NJ:	 Princeton	 University

Press, 2009, pp. 80–83

Bloom H., Orr L., Cave G., Bell S.,	 and	 Doolittle F. (2003). The National JTPA Study:	 Title	 IIA

Impacts on Earnings and Employment	at 18 Months.	Bethesda,	MD:	Abt Associates.

Caliendo	 M.	 and Kopeing S. (2008). Some Practical Guidance for the Implementation of

Propensity Score Matching. Journal	of Economic Surveys, Vol. 22, No. 1, pp. 21-72.

Card D., Kluve J.,	and	Weber A. (2010). Active Labor Market	Policy	Evaluations:	A	Meta-Analysis.

NBER Working Paper No. 16173.

Decker P. Ten Years of WIA Research.	 In	 The Workforce Investment	 Act:	 Implementation

Experiences and Evaluation Findings, edited by D. Besharov and P. Cottingham. Kalamazoo,

Michigan: W.E. Upjohn Institute, 2011.

Elvery J. and Spence C. Health Careers Collaborative of Greater Cincinnati Return on Investment

Report, 2011.	Report prepared by The New Growth Group.

Greenberg D.H., Cebulla A., and Bouchet S. (2005). Report on	 a	 Meta-Analysis of Welfare-to-	

Work Programs. Institute for	Research	on Poverty Discussion Paper No. 1312-05.
IMPAQ International, LLC Page 91 NFWS/SIF Quasi-Experimental Impact Study

Haelermans	C.	and	Borghans	L. (2011). Wage Effects of On-the-Job Training:	A	Meta-Analysis.

IZA	Discussion Paper No. 6077.

Heinrich	 C.J.,	 Mueser	 P.R., Troske K.R., Jeon K.S.,	 and	 Kahvecioglu D.C. (2013). Do Public

Employment and Training Programs Work? IZA	Journal	of	labor	Economics, Vol. 2, No. 6, pp.	1–	

23.

Michaelides	 M.	 and	 Mueser	 P. The Role of Industry and Occupation in	 Recent	 U.S.

Unemployment Differentials by Gender, Race, and Ethnicity. Eastern Economic Journal, Vol. 39,

No. 3, 2013, pp. 358–386

Michaelides M., Poe-Yamagata E., and Mbwana K. (2013). Outcomes Assessment Study of

NFWS/SIF-funded Programs. Boston,	MA:	National Fund for Workforce Solutions.

Mueser P., Troske K.,	and	Gorislavsky A. (2007). Using State	 Administrative	Data	 to	Measure

Program Performance.	Review of Economics and Statistics, Vol. 89, No. 4, pp. 761–783.
IMPAQ International, LLC Page 92 NFWS/SIF Quasi-Experimental Impact Study

Appendix

Table A: Characteristics of	Treatment and	Matched	Comparison	Cases, Health	Careers

Treatment Matched

Group ComparisonGroup Difference

Total Number of Participants 992 46,701

Gender

Men .098 (.297) .100 (.300) -.002 [.005]

Women .902 (.297) .900 (.300) .002 [.005]

Race

White .503 (.500) .506 (.500) -.003 [.008]

Black .400 (.490) .397 (.489) .003 [.008]

Other Race .096 (.294) .096 (.294) .000 [.005]

Missing .001 (.032) .001 (.032) .000 [.001]

Education

No High School Diploma .081 (.272) .077 (.267) .004 [.004]

High School Diploma .384 (.487) .384 (.486) .000 [.008]

Associate Degree, Some College .458 (.498) .458 (.498) .000 [.008]

College Degree .078 (.268) .081 (.272) -.003 [.004]

Age

Less than 25 Years .314 (.464) .321 (.467) -.007 [.007]

25-34 Years .336 (.472) .337 (.473) -.001 [.007]

35-44 Years .148 (.355) .151 (.358) -.003 [.006]

45-54 Years .125 (.331) .129 (.335) -.004 [.005]

55-64 Years .054 (.227) .056 (.231) -.002 [.004]

65+ Years .005 (.071) .005 (.071) .000 [.001]

Missing .018 (.134) .001 (.030) .017 [.002]***

Local Workforce Investment Area

Area 12 (Butler County) .185 (.389) .187 (.390) -.002 [.006]

Area 13 (Hamilton County) .794 (.404) .793 (.405) .001 [.006]

Other Areas .020 (.141) .021 (.143) -.001 [.002]

Program Entry

Quarter 1, 2010 .236 (.424) .228 (.420) .008 [.007]

Quarter 2, 2010 .077 (.266) .077 (.266) .000 [.004]

Quarter 3, 2010 .097 (.296) .100 (.300) -.003 [.005]

Quarter 4, 2010 .094 (.292) .096 (.294) -.002 [.005]

Quarter 1, 2011 .088 (.283) .090 (.286) -.002 [.004]

Quarter 2, 2011 .128 (.334) .128 (.334) .000 [.005]

Quarter 3, 2011 .151 (.358) .157 (.364) -.006 [.006]

Quarter 4, 2011 .130 (.337) .124 (.330) .005 [.005]

(Table continues on next page)
IMPAQ International, LLC Page 93 NFWS/SIF Quasi-Experimental Impact Study

(Table A, continued from previous page)

Treatment Matched

Group ComparisonGroup Difference

Total Number of Participants 992 46,701

Employment

In Prior Quarter 1	 .455 (.498) .454 (.498) .001 [.008]

In Prior Quarter 2	 .491 (.500) .487 (.500) .004 [.008]

In Prior Quarter 3	 .515 (.500) .509 (.500) .006 [.008]

In Prior Quarter 4	 .535 (.499) .532 (.499) .003 [.008]

In Prior Quarter 5	 .526 (.500) .523 (.499) .003 [.008]

In Prior Quarter 6	 .522 (.500) .524 (.499) -.002 [.008]

In Prior Quarter 7	 .553 (.497) .555 (.497) -.002 [.008]

In Prior Quarter 8	 .555 (.489) .562 (.496) -.007 [.008]

Prior Employment

In Both Quarters	1-2	 .393 (.489) .391 (488) .001 [.008]

In All Quarters	1-4	 .311 (.463) .309 (.462) .002 [.007]

No Prior Employment

In Quarters	1-2	 .448 (.497) .450 (.497) -.002 [.008]

In Quarters	1-4	 .326 (.469) .329 (.470) -.003 [.007]

Prior Employment

In Quarter	1	

in Healthcare

.202 (.401) .202 (.402) .000 [.006]

In Quarter	1-4	 .282 (.450) .279 (.449) .032 [.007]

Earnings Amount

In Prior Quarter 1	 1,902 (5,241) 1,878 (5,251) 24 [86]

In Prior Quarter 2	 1,905 (3,560) 1,897 (3,558) 8	[71]

In Prior Quarter 3	 2,328 (5,560) 2,377 (5,907) -49 [90]

In Prior Quarter 4	 2,216 (4,456) 2,198 (4,320) 18 [79]

In Prior Quarter 5	 2,380 (5,285) 2,320 (4,582) 60 [84]

In Prior Quarter 6	 2,348 (4,547) 2,357 (4,509) -9	[80]

In Prior Quarter 7	 2,354 (3,718) 2,365 (3,771) -11 [70]

In Prior Quarter 8	 2,364 (4,506) 2,364 (3,987) 0	[78]

Note: Left-hand and middle	columns report mean, with standard deviation in	parentheses. The	right-hand column

reports the treatment-matched comparison group difference, with standard error in brackets. Statistical

significance: ***	=	at the	1	percent level.
IMPAQ International, LLC Page 94 NFWS/SIF Quasi-Experimental Impact Study

Table B: Characteristics of Treatment and	Matched	Comparison	Cases, Advanced	Manufacturing

Treatment Matched

Group ComparisonGroup Difference

Total Number of Participants 682 42,293

Gender

Men .657 (.475) .661 (.473) -.004 [.008]

Women .343 (.475) .339 (.473) .004 [.008]

Race

White .208 (.406) .212 (.409) -.004 [.006]

Black .757 (.429) .752 (.432) .005 [.007]

Other Race .035 (.184) .036 (.185) -.001 [.003]

Missing --	 --	

Education

No High School Diploma .292 (.455) .303 (.459) -.011 [.007]

High School Diploma .399 (.490) .391 (.488) .008 [.008]

Associate Degree, Some College .273 (.446) .271 (.444) .002 [.007]

College Degree .037 (.188) .036 (.444) .001 [.001]

Age

Less than 25 Years .213 (.409) .209 (.407) .004 [.006]

25-34 Years .302 (.459) .311 (.463) -.009 [.007]

35-44 Years .214 (.410) .209 (.406) .005 [.006]

45-54 Years .186 (.390) .184 (.387) .002 [.006]

55-64 Years .078 (.268) .080 (.271) -.002 [.044]

65+ Years .007 (.085) .007 (.084) .000 [.001]

Missing --	 --	 --

Local Workforce Investment Area

Area 12 (Butler County) .119 (.324) .126 (.331) -.007 [.005]

Area 13 (Hamilton County) .874 (.332) .867 (.340) .007 [.005]

Other Areas .007 (.085) .008 (.087) -.001 [.001]

Program Entry

Quarter 1, 2010 .018 (.132) .018 (.133) .000 [.002]

Quarter 2, 2010 .043 (.202) .043 (.202) .000 [.003]

Quarter 3, 2010 .076 (.266) .081 (.272) -.005 [.004]

Quarter 4, 2010 .117 (.322) .116 (.321) .001 [.005]

Quarter 1, 2011 .213 (.409) .214 (.410) -.001 [.006]

Quarter 2, 2011 .180 (.385) .182 (.386) -.002 [.006]

Quarter 3, 2011 .128 (.334) .135 (.342) -.006 [.005]

Quarter 4, 2011 .226 (.418) .212 (.408) .014 [.007]**

(Table continues on next page)
IMPAQ International, LLC Page 95 NFWS/SIF Quasi-Experimental Impact Study

(Table B, continued from previous page)

Treatment Matched

Group ComparisonGroup Difference

Total Number of Participants 682 42,393

Employment

In Prior Quarter	1	 .287 (.453) .288 (.453) -.001 [.007]

In Prior Quarter	2	 .292 (.455) .297 (.457) -.005 [.007]

In Prior Quarter	3	 .331 (.471) .334 (.472) -.003 [.007]

In Prior Quarter	4	 .323 (.467) .331 (.471) -.008 [.007]

In Prior Quarter	5	 .342 (.475) .347 (.476) -.005 [.008]

In Prior Quarter	6	 .375 (.485) .384 (.486) -.009 [.008]

In Prior Quarter	7	 .380 (.486) .388 (.487) -.008 [.008]

In Prior Quarter	8	 .427 (.495) .428 (.495) -.001 [.008]

Prior Employment

In Both Quarters	1-2	 .192 (.394) .195 (.396) -.003 [.006]

In All Quarters	1-4	 .139 (.347) .142 (.350) -.003 [.005]

No Prior Employment

In Quarters	1-2	 .613 (.487) .609 (.488) .004 [.008]

In Quarters	1-4	 .497 (.500) .492 (.500) .005 [.008]

Prior Employment in Manufacturing

In Quarter	1	 .028 (.165) .028 (.166) .000 [.003]

In Quarter	1-4	 .050 (.218) .053 (.223) -.003 [.003]

Earnings Amount

In Prior Quarter	1	 787 (2,472) 817 (2,514) -30 [59]

In Prior Quarter	2	 1,076 (3,326) 1,127 (3,436) -51 [69]

In Prior Quarter	3	 1,348 (3,358) 1,370 (3,336) -22 [60]

In Prior Quarter	4	 1,460 (3,452) 1,508 (3,466) -48 [70]

In Prior Quarter	5	 1,612 (3,743) 1,634 (3,535) -22 [72]

In Prior Quarter	6	 1,672 (3,515) 1,754 (3,757) -82 [72]

In Prior Quarter	7	 1,979 (5,222) 1,948 (4,263) 31 [82]

In Prior Quarter	8	 1,996 (3,679) 2,067 (3,747) -71 [72]

Note: Left-hand and middle columns report mean with standard deviation in parentheses. The right-hand column

reports the treatment-matched comparison group difference, with standard error in brackets. Statistical

significance: **	=	at the	5	percent level.
IMPAQ International, LLC Page 96 NFWS/SIF Quasi-Experimental Impact Study

Table C: Characteristics of	Treatment and	Matched	Comparison	Cases, Construction	Partnership

Treatment Matched

Group ComparisonGroup Difference

Total Number of Participants 379 36,859

Gender

Men .520 (.500) .519 (.500) .001 [.008]

Women .480 (.500) .481 (.500) -.001 [.008]

Race

White .185 (.389) .187 (.390) -.002 [.006]

Black .781 (.414) .778 (.416) .003 [.007]

Other Race .032 (.175) .032 (.177) .000 [.003]

Missing .003 (.051) .003 (.052) .000 [.001]

Education

No High School Diploma .166 (.373) .167 (.373) -.001 [.006]

High School Diploma .554 (.498) .552 (.497) .002 [.008]

Associate Degree, Some College .245 (.431) .246 (.430) -.001 [.007]

College Degree .034 (.182) .035 (.184) -.001 [.003]

Age

Less than 25 Years .311 (.464) .314 (.464) -.003 [.007]

25-34 Years .293 (.457) .294 (.456) -.001 [.007]

35-44 Years .237 (.426) .241 (.428) -.004 [.007]

45-54 Years .127 (.333) .127 (.333) .000 [.005]

55-64 Years .021 (.144) .022 (.145) -.001 [.002]

65+ Years .003 (.051) .003 (.052) .000 [.001]

Missing .008 (.089) .000 (.000) .008 [.001]***

Local Workforce Investment Area

Area 12 (Butler County) .140 (.347) .143 (.350) -.003 [.006]

Area 13 (Hamilton County) .860 (.347) .857 (.350) .003 [.006]

Other Areas --	 --	 --

Program Entry

Quarter 1, 2010 .156 (.363) .155 (.362) .001 [.006]

Quarter 2, 2010 .203 (.403) .204 (.395) -.001 [.006]

Quarter 3, 2010 .193 (.395) .194 (.395) -.001 [.006]

Quarter 4, 2010 .198 (.399) .199 (.399) -.001 [.006]

Quarter 1, 2011 .069 (.253) .068 (.252) .001 [.004]

Quarter 2, 2011 .065 (.249) .063 (.244) .002 [.004]

Quarter 3, 2011 .053 (.224) .054 (.225) -.001 [.004]

Quarter 4, 2011 .063 (.244) .063 (.244) .000 [.004]

(Table continues on next page)
IMPAQ International, LLC Page 97 NFWS/SIF Quasi-Experimental Impact Study

(Table C, continued from previous page)

Treatment Matched

Group ComparisonGroup Difference

Total Number of Participants 379 36,859

Employment

In Prior Quarter	1	 .298 (.458) .302 (.459) -.004 [.007]

In Prior Quarter	2	 .346 (.476) .344 (.475) .002 [.008]

In Prior Quarter	3	 .322 (.468) .323 (.468) -.001 [.007]

In Prior Quarter	4	 .343 (.475) 345 (.475) -.002 [.008]

In Prior Quarter	5	 .398 (.490) .397 (.489) .001 [.008]

In Prior Quarter	6	 .438 (.497) .432 (.495) .006 [.008]

In Prior Quarter	7	 .451 (.498) .445 (.497) .006 [.008]

In Prior Quarter	8	 .449 (.498) .446 (.497) .003 [.008]

Prior Employment

In Both Quarters	1-2	 .211 (.409) .214 (.410) -.003 [.006]

In All Quarters	1-4	 .129 (.336) .131 (.337) -.002 [.005]

No Prior Employment

In Quarters	1-2	 .567 (.496) .568 (.495) -.001 [.008]

In Quarters	1-4	 .443 (.497) .445 (.497) -.002 [.008]

Prior Employment in Construction

In Quarter	1	 .013 (.114) .014 (.116) -.001 [.002]

In Quarter	1-4	 .018 (.135) .016 (.127) .002 [.002]

Earnings Amount

In Prior Quarter	1	 1,028 (5,549) 1,045 (5,573) -17 [89]

In Prior Quarter	2	 824 (2,046) 836 (2,063) -12 [54]

In Prior Quarter	3	 935 (2,369) 952 (2,392) -17 [58]

In Prior Quarter	4	 1,095 (2,677) 1,109 (2,704) -14 [62]

In Prior Quarter	5	 1,172 (2,448) 1,186 (2,476) -14 [59]

In Prior Quarter	6	 1,627 (5,800) 1,645 (5,966) -18 [91]

In Prior Quarter	7	 1,460 (2,770) 1,459 (2,908) 1	[63]

In Prior Quarter	8	 1,451 (2,855) 1,456 (2,867) -5	[64]

Note: Left-hand and middle columns report mean with standard deviation in parentheses. The right-hand column

reports the treatment-matched comparison group difference, with standard error in brackets. Statistical

significance: ***	=	at the	1	percent level.
IMPAQ International, LLC Page 98 NFWS/SIF Quasi-Experimental Impact Study

Table D: Characteristics of	Treatment and	Matched	Comparison	Cases,	WRTP	Construction

Treatment Matched

Group ComparisonGroup Difference

Total Number of Participants 1,091 29,491

Gender

Men .905 (.294) .912 (.284) -.007 [.011]

Women .095 (.294) .088 (.284) .007 [.011]

Race

White .385 (.487) .358 (.479) .027 [.029]

Black .405 (.491) .389 (.487) .017 [.030]

Other Race .065 (.247) .077 (.266) -.012 [.020]

Education

No High School Diploma .179 (.383) .169 (.375) .010 [.023]

High School Diploma .791 (.407) .800 (.401) -.008 [.024]

Associate Degree, Some College .020 (.141) .020 (.141) .000 [.005]

College Degree .010 (.100) .012 (.107) -.001 [.004]

Age

Less than 25 Years .196 (.397) .165 (.372) .031 [.019]

25-34 Years .333 (.471) .356 (.479) -.023 [.031]

35-44 Years .230 (.421) .262 (.439) -.032 [.033]

45-54 Years .175 (.380) .144 (.351) .031 [.017]*

55-64 Years .064 (.245) .070 (.255) -.006 [.013]

65+ Years .002 (.043) .003 (.057) -.001 [.001]

Program Entry

Quarter 1, 2010 .006 (.080) .006 (.076) .001 [.003]

Quarter 2, 2010 .020 (.141) .017 (.129) .003 [.005]

Quarter 3, 2010 .010 (.100) .009 (.095) .001 [.003]

Quarter 4, 2010 .005 (.074) .005 (.073) .000 [.002]

Quarter 1, 2011 .188 (.391) .154 (.361) .034 [.018]*

Quarter 2, 2011 .204 (.403) .184 (.388) .020 [.020]

Quarter 3, 2011 .351 (.478) .380 (.485) -.029 [.033]

Quarter 4, 2011 .214 (.411) .245 (.430) -.031 [.029]

(Table continues on next page)
IMPAQ International, LLC Page 99 NFWS/SIF Quasi-Experimental Impact Study

(Table D,	continued from previous page)

Treatment Matched

Group ComparisonGroup Difference

Total Number of Participants 1,091 29,491

Employment

In Prior Quarter 1	 .605 (.489) .644 (.479) -.039 [.026]

In Prior Quarter 2	 .606 (.489) .624 (.484) -.018 [.029]

In Prior Quarter 3	 .604 (.489) .658 (.474) -.054 [.025]**

In Prior Quarter 4	 .597 (.491) .634 (.482) -.037 [.026]

In Prior Quarter 5	 .564 (.496) .592 (.492) -.028 [.029]

In Prior Quarter 6	 .586 (.493) .564 (.496) .022 [.031]

In Prior Quarter 7	 .604 (.489) .582 (.493) .022 [.031]

In Prior Quarter 8	 .607 (.489) .593 (.491) .014 [.030]

Prior Employment

In Both Quarters	1-2	 .520 (.500) .548 (.498) -.029 [.031]

In All Quarters	1-4	 .416 (.493) .457 (.498) -.041 [.033]

No Prior Employment

In Quarters	1-2	 .309 (.462) .280 (.449) .029 [.022]

In Quarters	1-4	 .232 (.422) .212 (.409) .020 [.018]

Prior Employment

In Quarter	1	

in Construction

.279 (.449) .323 (.468) -.045 [.036]

In Quarter	1-4	 .393 (.489) .419 (.493) -.026 [.035]

Earnings Amount

In Prior Quarter 1	 3,820 (4,995) 3,977 (4,869) -158 [326]

In Prior Quarter 2	 4,201 (5,412) 4,025 (4,924) 176 [312]

In Prior Quarter 3	 4,410 (5,572) 4,472 (5,257) -62 [341]

In Prior Quarter 4	 4,440 (5,919) 4,510 (5,519) -71 [402]

In Prior Quarter 5	 4,462 (6,142) 4,387 (5,583) 75 [436]

In Prior Quarter 6	 4,656 (6,356) 4,173 (5,722) 483 [446]

In Prior Quarter 7	 4,986 (6,480) 4,831 (6,204) 155 [540]

In Prior Quarter 8	 4,895 (6,217) 4,717 (6,166) 179 [556]

Note: Left-hand and middle columns report mean with standard deviation in parentheses. The right-hand column

reports the treatment-matched comparison group difference, with standard error in brackets. Statistical

significance:	*	=	at the 10 percent level; **	=	at the	5	percent level.
IMPAQ International, LLC Page 100 NFWS/SIF Quasi-Experimental Impact Study

Table E: Characteristics of	Treatment and	Matched	Comparison	Cases,	WRTP	Manufacturing

Treatment Matched

Group ComparisonGroup Difference

Total Number of Participants 86 19,844

Gender

Men .837 (.371) .836 (.370) .001 (.040)

Women .163 (.371) .164 (.370) -.001 (.040)

Race

White .291 (.457) .293 (.455) -.002 (.049)

Black .581 (.496) .577 (.494) .004 (.054)

Other Race .047 (.212) .048 (.214) -.001 (.023)

Education

No High School Diploma .209 (.409) .206 (.405) .003 (.045)

High School Diploma .767 (.425) .770 (.421) -.003 (.046)

Associate Degree, Some College .023 (.152) .023 (.151) -.000 (.016)

College Degree -	 -	 -	

Age

Less than 25 Years .128 (.336) .134 (.340) -.006 (.036)

25-34 Years .279 (.451) .280 (.449) -.001 (.049)

35-44 Years .221 (.417) .258 (.438) -.038 (.046)

45-54 Years .256 (.439) .223 (.416) .033 (.048)

55-64 Years .116 (.322) .097 (.297) .019 (.035)

65+ Years -	 -	 -	

Program Entry

Quarter 1, 2010 .023 (.152) .023 (.151) -.000 (.016)

Quarter 2, 2010 .035 (.185) .035 (.185) -.000 (.020)

Quarter 3, 2010 .267 (.445) .269 (.444) -.002 (.049)

Quarter 4, 2010 .279 (.451) .276 (.447) .003 (.049)

Quarter 1, 2011 .221 (.417) .218 (.413) .003 (.045)

Quarter 2, 2011 .174 (.382) .178 (.382) -.003 (.042)

(Table continues on next page)
IMPAQ International, LLC Page 101 NFWS/SIF Quasi-Experimental Impact Study

(Table E,	continued from previous page)

Treatment Matched

Group ComparisonGroup Difference

Total Number of Participants 86 19,844

Employment

In Prior Quarter 1	 .488 (.503) .487 (.500) .001 (.055)

In Prior Quarter 2	 .558 (.500) .557 (.497) .001 (.054)

In Prior Quarter 3	 .547 (.501) .550 (.497) -.004 (.054)

In Prior Quarter 4	 .488 (.503) .492 (.500) -.003 (.055)

In Prior Quarter 5	 .535 (.502) .540 (.498) -.005 (.055)

In Prior Quarter 6	 .535 (.502) .533 (.499) .002 (.055)

In Prior Quarter 7	 .558 (.500) .558 (.497) .000 (.054)

In Prior Quarter 8	 .570 (.498) .570 (.495) .000 (.054)

Prior Employment

In Both Quarters	1-2	 .419 (.496) .418 (.493) .001 (.054)

In All Quarters	1-4	 .279 (.451) .282 (.450) -.003 (.049)

No Prior Employment

In Quarters	1-2	 .372 (.486) .373 (.484) -.001 (.053)

In Quarters	1-4	 .267 (.445) .272 (.445) -.005 (.048)

Prior Employment

In Quarter	1	

in Manufacturing

.174 (.382) .176 (.381) -.002 (.042)

In Quarter	1-4	 .267 (.445) .241 (.428) .026 (.049)

Earnings Amount

In Prior Quarter 1	 2,131 (3,058) 2,136 (3,044) -5	[333]

In Prior Quarter 2	 2,503 (4,054) 2,484 (4,009) 19 [440]

In Prior Quarter 3	 2,919 (4,631) 2,945 (4,651) -27 [511]

In Prior Quarter 4	 2,486 (3,858) 2,508 (3,840) -22 [420]

In Prior Quarter 5	 2,573 (3,676) 2,588 (3,652) -14 [401]

In Prior Quarter 6	 2,806 (4,094) 2,787 (4,054) 20 [447]

In Prior Quarter 7	 3,436 (5,220) 3,313 (4,904) 124 [570]

In Prior Quarter 8	 3,838 (5,400) 3,755 (5,231) 83 [594]

Note:	 Left-hand and middle columns report mean with standard deviation in parentheses. The

reportsthe treatment-matched comparison group difference, with standard error in brackets.

right-hand column
IMPAQ International, LLC Page 102 NFWS/SIF Quasi-Experimental Impact Study

Table F:	Characteristics of	Treatment and	Matched	Comparison	Cases, Healthcare	Alliance

Treatment Matched

Group ComparisonGroup Difference

Total Number of Participants 287 8,666

Gender

Men .056 (.230) .056 (.230) -.000 (.014)

Women .944 (.230) .944 (.230) .000 (.014)

Race

White .063 (.243) .063 (.243) -.000 (.014)

Black .777 (.417) .776 (.417) .001 (.026)

Other Race .143 (.351) .143 (.350) -.000 (.022)

Age

Less than 25 Years .415 (.494) .402 (.490) .012 (.032)

25-34 Years .352 (.478) .343 (.475) .009 (.030)

35-44 Years .101 (.302) .141 (.348) -.040 (.019)**

45-54 Years .091 (.288) .076 (.265) .015 (.017)

55-64 Years .038 (.192) .032 (.175) .007 (.012)

65+ Years .003 (.059) .006 (.079) -.003 (.004)

Program Entry

Quarter 1, 2010 .143 (.351) .146 (.353) -.003 (.022)

Quarter 2, 2010 .369 (.483) .370 (.483) -.001 (.031)

Quarter 3, 2010 .324 (.469) .323 (.468) .001 (.030)

Quarter 4, 2010 .164 (.371) .161 (.367) .002 (.024)

(Table continues on next page)
IMPAQ International, LLC Page 103 NFWS/SIF Quasi-Experimental Impact Study

(Table F, continued from previous page)

Treatment Matched

Group ComparisonGroup Difference

Total Number of Participants 287 8,666

Employment

In Prior Quarter 1	 .530 (.500) .530 (.499) -.000 (.032)

In Prior Quarter 2	 .589 (.493) .589 (.492) .000 (.031)

In Prior Quarter 3	 .578 (.495) .579 (.494) -.000 (.031)

In Prior Quarter 4	 .589 (.493) .589 (.492) -.001 (.031)

In Prior Quarter 5	 .592 (.492) .594 (.491) -.002 (.031)

In Prior Quarter 6	 .599 (.491) .600 (.490) -.000 (.031)

In Prior Quarter 7	 .606 (.489) .608 (.488) -.001 (.031)

In Prior Quarter 8	 .578 (.495) .581 (.493) -.003 (.032)

Prior Employment

In Both Quarters	1-2	 .463 (.500) .463 (.499) .000 (.032)

In All Quarters	1-4	 .390 (.489) .389 (.488) .001 (.031)

No Prior Employment

In Quarters	1-2	 .345 (.476) .345 (.475) .000 (.030)

In Quarters	1-4	 .251 (.434) .264 (.441) -.013 (.027)

Prior Employment

In Quarter	1	

in Healthcare

.199 (.400) .204 (.403) -.006 (.026)

In Quarter	1-4	 .289 (.454) .278 (.448) .012 (.030)

Earnings Amount

In Prior Quarter 1	 1,308 (1,875) 1,303 (1,864) 5	[116]

In Prior Quarter 2	 1,598 (2,086) 1,609 (2,087) -11 [130]

In Prior Quarter 3	 1,807 (2,295) 1,818 (2,299) -11 [143]

In Prior Quarter 4	 1,793 (2,352) 1,815 (2,364) -22 [146]

In Prior Quarter 5	 1,878 (2,441) 1,898 (2,444) -20 [152]

In Prior Quarter 6	 1,962 (2,532) 1,979 (2,536) -18 [157]

In Prior Quarter 7	 1,984 (2,461) 1,939 (2,468) -15 [152]

In Prior Quarter 8	 1,923 (2,735) 1,947 (2,748) -24 [169]

Note:	 Left-hand and middle columns report mean with standard deviation in parentheses. The

reportsthe treatment-matched comparison group difference, with standard error in brackets.

right-hand column
IMPAQ International, LLC Page 104 NFWS/SIF Quasi-Experimental Impact Study

Disclaimer

This study	 has	 been partly funded by	 a	 Social Innovation Fund (SIF)	 grant	 awarded by the

Corporation for National	 and	 Community Service (CNCS)	 to	the National Fund for Workforce

Solutions (NFWS). The study was produced by	 IMPAQ	 International, LLC and does not

necessarily	reflect the views, policies, or official position of CNCS or other government	agencies,

nor does the mention of trade names, commercial products, or organizations	 	 imply

endorsement of same by	 the	 U.S. government. The study uses cross-matched,	 longitudinal

state	administrative data from the Ohio Longitudinal Data Archive (OLDA), which is	managed by

the	 Ohio State University’s	 Center	 for Human	 Resource	 Research	 (CHRR). This workforce

solution was funded by	 a	 grant	 awarded	 to	the U.S.	 Department	 of Labor's Employment and

Training Administration. This solution was created by	 the	 CHRR on behalf of the Ohio

Department of Job and Family Services	and	does not	necessarily	reflect	 the	official position of

the	U.S. Department of Labor. The Department of Labor makes no guarantees, warranties, or

assurances	 of	 any	 kind, express or implied, with respect	 to	such information, including	 any

information on linked sites	and	including, but not limited to,	accuracy	of	the	information or its

completeness,	 timeliness, usefulness, adequacy, continued availability, or ownership. This

solution	is copyrighted by	the	institution	that created	it.	 Internal use, by	an	organization	and/or

personal use by	 an	 individual for non-commercial purposes, is permissible. All other uses

require	the	prior authorization of the copyright owner.