Quasi-Experimental Impact Study of NFWS/SIF 

Workforce Partnership Programs 

Evidence on the Effectiveness of Workforce Partnership 

Programs in Ohio and Wisconsin

Final Report – August 2016

Executive Summary

Authors 

Marios Michaelides 

Project Director and co-Principal Investigator 

(University of Cyprus and IMPAQ International)

Peter Mueser 

Co-Principal Investigator

(University of Missouri and IMPAQ International) 

Scott Davis 

(IMPAQ International) 

Kassim Mbwana

(IMPAQ International) 

Prepared and Submitted by:

IMPAQ International, LLC

10420 Little Patuxent Parkway

Columbia, MD 21044

Submitted to:

National Fund for Workforce Solutions

88 Broad Street

Boston, MA 02110
TABLE OF CONTENTS 

Page

Executive Summary ................................................................................................................ i

1. Introduction ................................................................................................................... 1

2. Background .................................................................................................................... 3

2.1 The National Fund for Workforce Solutions .................................................................... 3

2.2 Quasi-experimental Impact Study Design ........................................................................ 4

2.2.1 Key Research Questions ............................................................................................ 5

2.2.2 Program Selection ..................................................................................................... 6

2.2.3 Data Sources ............................................................................................................. 9

2.2.4 Quasi-experimental Approach ................................................................................ 11

3. Program Descriptions ................................................................................................... 17

3.1 Partners for a Competitive Workforce Collaborative .................................................... 17

3.1.1 Health Careers Collaborative of Greater Cincinnati ............................................... 17

3.1.2 Advanced Manufacturing Partnership .................................................................... 21

3.1.3 Construction Sector Partnership ............................................................................. 23

3.1.4 Participant Characteristics ...................................................................................... 24

3.1.5 Services Received .................................................................................................... 29

3.2 The Milwaukee Area Workforce Alliance ....................................................................... 31

3.2.1 Wisconsin Regional Training Partnership ............................................................... 33

3.2.2 Milwaukee Area Healthcare Alliance ...................................................................... 34

3.2.3 Participant Characteristics ...................................................................................... 34

3.2.4 Services Received .................................................................................................... 38

4. Quasi-experimental Impact Study Results ..................................................................... 39

4.1 Ohio Impact Study Results ............................................................................................. 39

4.1.1 Matching Results ..................................................................................................... 41

4.1.2 Impact Results ......................................................................................................... 47

4.1.3 Discussion of the Results ........................................................................................ 58

4.2 Wisconsin Impact Study Results..................................................................................... 64
4.2.1 Matching Results..................................................................................................... 66

4.2.2 Impact Results......................................................................................................... 70

4.2.3 Discussion of the Results ........................................................................................ 80

5. Conclusion.................................................................................................................... 87

References.......................................................................................................................... 90

Appendix ............................................................................................................................ 92

Disclaimer..........................................................................................................................104
IMPAQ International, LLC Page i NFWS/SIF Quasi-Experimental Impact Study

Executive Summary 

Since its establishment in 2007, the National Fund for Workforce Solutions (NFWS) has invested 

in Regional Funding Collaboratives that match NFWS funds with funds from other sources to 

support the development of local workforce partnerships. These partnerships identify the 

workforce needs of local employers and design and administer programs to help workers to 

obtain the skills needed to meet those needs. By 2010, NFWS was supporting 30 local 

workforce partnerships with active training programs in six states. In 2010, NFWS was awarded 

a two-year, $7.7 million grant by the Social Innovation Fund (SIF) to expand these programs and 

to support the creation of new programs. 

In 2011, NFWS contracted with IMPAQ International, LLC (IMPAQ) to evaluate the effectiveness 

of the 30 NFWS/SIF-funded workforce partnership programs. The evaluation consisted of: 

1) an outcomes study to examine program participation, services provided, and participant 

outcomes in the period of the SIF funding (January 2010 – February 2012); and 

2) a quasi-experimental impact study to estimate the effects of selected NFWS/SIF-funded 

programs on the labor market outcomes of individuals who entered those programs 

during the SIF funding period. 

This report presents the findings of the quasi-experimental impact study for six NFWS/SIFfunded programs – three based in Ohio (Healthcare Careers Collaborative, Advanced 

Manufacturing Partnership, and Construction Sector Partnership) and three based in Wisconsin 

(Wisconsin Regional Training Partnership Manufacturing Pathway, Wisconsin Regional Training 

Partnership Construction Pathway, and Milwaukee Area Health Alliance). These six programs 

provided training and other services to individuals interested in obtaining jobs and advancing 

their careers in healthcare, advanced manufacturing, and construction. The study relies on a 

quasi-experimental approach to estimate program impacts for participants who were 

unemployed at program entry by: (1) using the propensity score matching method to identify 
IMPAQ International, LLC Page ii NFWS/SIF Quasi-Experimental Impact Study

matched comparison groups consisting of unemployed non-participants who were 

observationally equivalent to unemployed NFWS/SIF participants, and (2) estimating program 

impacts by comparing the labor market outcomes of NFWS/SIF participants with the outcomes 

of unemployed individuals in the matched comparison groups. 

A. Program Descriptions 

A.1 Ohio-based Programs 

The three Ohio-based programs were supported by the Partners for a Competitive Workforce 

collaborative, a regional partnership in the Cincinnati area. This collaborative used NFWS funds, 

combined with funds from numerous private and public organizations, to help the three 

partnerships design and implement sustainable workforce strategies to promote the 

employment of low-income individuals in their respective focus industries. Below is summary of 

each of the three programs. 

Health Careers Collaborative of Greater Cincinnati. This program focused on helping 

unemployed workers obtain the skills needed to access healthcare jobs. The program provided 

a wide range of services, including job readiness training, assistance in obtaining employability 

and training credentials, industry-focused training, and job search assistance. During the study 

period (January 2010 – February 2012), the program served 992 unemployed participants, who 

were primarily women, had more than a high school education, were under age 35, and had 

prior work experience. 

Advanced Manufacturing Partnership. This program focused on promoting the employment 

and career advancement of low-skill workers in advanced manufacturing jobs. The program 

used an incremental approach in promoting participants’ employment and educational 

advancement, which included job readiness training, assistance in obtaining employability 

credentials, enrollment in college coursework and specialized apprenticeships, and receipt of 

job search assistance. During the study period, the program recruited 684 unemployed 
IMPAQ International, LLC Page iii NFWS/SIF Quasi-Experimental Impact Study

participants – the majority of participants were men, were nonwhite, had no more than a high 

school education, were under age 35, and had limited work experience.

Construction Sector Partnership. This program focused on creating career pathways in 

construction for low-skill workers. The program’s pathways model was based on enrolling 

participants in pre-apprenticeship programs to help them obtain construction skills and 

providing job search services. During the study period, the program recruited 379 unemployed 

participants – the majority of participants were men, were nonwhite, with no more than a high 

school education, under age 35, and with limited work experience.

A.2 Wisconsin-Based Programs

The three Wisconsin-based programs were supported by the Milwaukee Area Workforce 

Alliance, which operates in the Milwaukee area. This collaborative used NFWS funds, combined 

with funds from private and public organizations, to support training programs that help lowincome workers to obtain in-demand jobs in construction, manufacturing, and healthcare. 

Below is summary of the three programs.

Wisconsin Regional Training Partnership (WRTP) Construction Pathways and Manufacturing 

Pathways. WRTP focused on brokering relations between employers, unions, and workers to 

promote employment of low-skill workers in construction and manufacturing, and to help local 

employers to recruit a diversified and qualified workforce. The partnership supported two 

programs – WRTP Construction Pathways and WRTP Manufacturing Pathways – that offered 

participants pre-apprenticeship training, assistance in obtaining occupational credentials, 

career advancement training, and job search services. During the study period, the construction 

and manufacturing programs served 1,103 and 88 unemployed participants, respectively – the 

majority were men, were nonwhite, and with no more than a high school education.

Milwaukee Area Healthcare Alliance. This program worked with healthcare employers to 

identify sought-after skills and to provide training and other services to low-skill workers to 
IMPAQ International, LLC Page iv NFWS/SIF Quasi-Experimental Impact Study

meet those needs. It provided participants with occupational training to obtain in-demand 

healthcare jobs (as identified by their employer partners), on-the-job training to help advance 

their careers, and job search services to identify suitable jobs. During the study period, the 

program served 306 unemployed workers, the majority were women, black, and under age 35. 

B. Impact Study Results 

The objective of the study is to examine the impacts of each program on the labor market 

outcomes of unemployed participants, including: employment, employment in the program’s 

focus industry, job retention, and earnings. IMPAQ developed a quasi-experimental approach 

based on the propensity score matching method, which involved the following steps: 

 Step 1: Merge data – Merge NFWS/SIF data on unemployed program participants 

(treatment group) with state Employer Service data on unemployed workers who 

sought state services during the same period as NFWS/SIF-funded program participants 

(comparison group). 

 Step 2: Produce propensity score – Apply a logit model on the merged data to estimate 

the probability of NFWS/SIF program participation based on individual characteristics 

and employment history, and use the results to produce the propensity score (predicted 

probability of NFWS/SIF participation) for treatment and comparison cases. 

 Step 3: Use propensity score to construct sample weight – Weigh each comparison case 

by the odds ratio of the predicted propensity score, so that the weighted comparison 

sample matches the characteristics distribution of the treatment sample. 

 Step 4: Compare treatment and weighted matched comparison sample – Conduct 

statistical tests to verify that the treatment and matched comparison groups are truly 

matched in their characteristics.

The above approach was successfully implemented, producing a matched comparison group for 

each NFWS/SIF-funded program. It consisted of non-participants who sought state employment 

services during the same period, had similar characteristics, and resided in the same area as 
IMPAQ International, LLC Page v NFWS/SIF Quasi-Experimental Impact Study

unemployed NFWS/SIF participants. State Unemployment Insurance (UI) wage records data 

were used to produce common labor market outcomes for treatment and matched comparison 

cases in the six-quarter period following program entry, including: employment, employment in 

the program’s focus industry, job retention, and earnings. Program impacts were estimated by 

comparing the mean outcomes between the treatment and the matched comparison group; 

results for each program are summarized below.

B.1 Ohio-based NFWS/SIF-funded programs

Health Careers Collaborative of Greater Cincinnati

 The program had large effects on employment. In the six quarters after program entry, 

participants’ employment rates were 57.664.8 percent, exceeding the rates of the 

matched comparison group by 14.117.0 percentage points (2937 percent).

 The program was effective in helping participants to obtain healthcare jobs. In the four 

quarters after entry, 33.434.3 percent of participants were employed in healthcare, 

exceeding the healthcare employment rates of the matched comparison group by 

24.025.3 percentage points (23.330.4 percent).

 The program was very effective in improving participant job retention rates. About 35.2 

percent of participants found employment in quarter 1 and remained employed in each 

of the six quarters after entry, compared to only 22.4 percent of matched comparison

group members – a 12.8 percentage point (57 percent) difference.

 The program had large effects on earnings. In the six-quarter follow-up period, 

participants had $5,517 (52 percent) higher earnings than those in the matched 

comparison group.

Advanced Manufacturing Partnership

 The program had large effects on employment. In the six quarters after entry, 42.252.6 

percent of participants were employed, exceeding the employment rates of matched 

comparison group members by 8.214.3 percentage points (2438 percent).
IMPAQ International, LLC Page vi NFWS/SIF Quasi-Experimental Impact Study

 The program had modest effects on manufacturing employment; no more than 4.2 

percent of participants were employed in manufacturing in the four quarters after 

entry. 

 The program had positive effects on job retention. About 19.9 percent of participants 

found a job in quarter 1 and remained employed in each of the six quarters after entry, 

compared to 15.9 percent of matched comparison group members, a 25 percent 

difference.

 The program had positive effects on earnings. In the six-quarter follow-up period, 

participants had $2,635 (31 percent) higher earnings than those in the matched 

comparison group. These effects were lower than those of the Health Careers program.

Construction Sector Partnership

 The program had modest effects on employment. Between 38.845.9 percent of 

participants were employed in quarters 16 after program entry, which exceeded the 

employment rates of matched comparison group members by 3.26.1 percentage 

points (916 percent).

 The program had modest effects on employment in construction; no more than 4.8 

percent of participants were employed in construction in the four quarters after entry.

 The program had no effects on job retention and modest effects on earnings.

B.2 Wisconsin-based NFWS/SIF-funded programs

WRTP Construction Pathways

 The program had large effects on employment. In the six quarters after program entry, 

67.272.7 percent of participants were employed, exceeding the employment rates of 

matched comparison group members by 8.821.7 percentage points (1443 percent).

 The program was not effective in helping participants to obtain construction jobs. In the six 

quarters after entry, fewer than three percent of participants were in construction jobs.

 The program had large positive effects on job retention. About 44.5 percent of participants 

found employment in quarter 1 and remained employed in each of the six quarters after 
IMPAQ International, LLC Page vii NFWS/SIF Quasi-Experimental Impact Study

entry, compared to only 29.4 percent of matched comparison group members, a 15.1 

percentage point (51 percent) difference.

 The program had large effects on earnings. In the six-quarter follow-up period, participants 

had $11,237 (56 percent) higher earnings than the matched comparison group.

WRTP Manufacturing Pathways

 The program had large effects on employment. In the six quarters after program entry, 

66.372.1 percent of participants were employed, exceeding the employment rates of 

matched comparison group members by 11.128.6 percentage points (2168 percent).

 The program was effective in helping participants to obtain manufacturing jobs. In the six 

quarters after entry, 32.640.7 percent of participants were employed in manufacturing, 

exceeding the rates of the matched comparison group by 23.729.6 percentage points 

(202535 percent).

 The program had large effects on job retention. About 43.0 percent of participants found a 

job in quarter 1 and remained employed in each of the six quarters after entry, compared 

to 19.8 percent of matched comparison group members, a 118 percent difference.

 The program had large effects on earnings. In the six-quarter follow-up period, participants 

had $16,661 (134 percent) higher earnings than the matched comparison group.

Milwaukee Healthcare Alliance

 The program had substantial effects on employment. In the six quarters after program 

entry, 57.177.0 percent of participants were employed, exceeding the employment rates 

of matched comparison group members by 16.519.4 percentage points (2840 percent).

 The program was effective in helping participants to obtain healthcare jobs. In the six 

quarters after entry, 27.947.4 percent of participants were employed in healthcare, 

exceeding the healthcare employment rates of the matched comparison group by 

18.129.6 percentage points (155196 percent).

 The program had large effects on job retention. About 40.1 percent of participants found a 

job in quarter 1 and remained employed in each of the six quarters after entry, compared 
IMPAQ International, LLC Page viii NFWS/SIF Quasi-Experimental Impact Study

to 22.3 percent of matched comparison group members, an 80 percent difference. 

 The program had positive effects on earnings. In the six-quarter follow-up period, 

participants had $5,418 (55 percent) higher earnings than the matched comparison group. 

C. Conclusions 

The results of the quasi-experimental impact study provide promising evidence about the 

effectiveness of NFWS/SIF-funded workforce training programs in healthcare, manufacturing, 

and construction. Results for the Ohio-based programs show that the Health Careers program 

was very effective in placing participants in healthcare jobs, leading to positive effects on 

overall employment, job retention, and earnings. The Advanced Manufacturing and 

Construction Partnership programs were not effective in placing participants in their respective 

focus industries. The Advanced Manufacturing program had positive effects on employment, 

job retention, and earnings, which were lower than the effects of the Health Careers programs, 

while the Construction Partnership program had modest effects on employment and earnings. 

All three Wisconsin-based programs had important positive effects on participants’ labor 

market outcomes. The WRTP Manufacturing program was the most effective of the six 

programs, since it placed participants in manufacturing jobs, leading to substantial effects on 

overall employment, job retention, and earnings. The Healthcare Alliance program was 

effective in helping participants find jobs in healthcare, leading to improved employment, job 

retention, and earnings. The program’s effects were similar to the effects of the Ohio-based 

Health Careers program. The WRTP Construction program was ineffective in helping 

participants obtain jobs in construction, but – unlike the Ohio-based Construction Partnership 

program – it led to positive effects on employment, job retention, and earnings. 

Overall, these results provide important insights on the effectiveness of NFWS/SIF-funded 

programs. Programs focusing on the healthcare industry can help participants obtain jobs in 

the industry and improve their labor market outcomes. While we get mixed results on 

manufacturing programs’ effectiveness in helping participants obtain jobs in the industry, these 
IMPAQ International, LLC Page ix NFWS/SIF Quasi-Experimental Impact Study

programs often lead to substantively important improvements in overall employment, job 

retention, and earnings. Finally, it appears that construction programs are unlikely to help 

participants obtain construction jobs, and are likely to have lower overall impacts on 

participants’ labor market outcomes than healthcare and manufacturing programs.