Preview Summary

Encouraging Evidence on a 

Sector-Focused Advancement Strategy 

d

Two-Year Impacts 

from the WorkAdvance 

Demonstration

Richard Hendra

David H. Greenberg

Gayle Hamilton

Ari Oppenheim

Alexandra Pennington

Kelsey Schaberg

Betsy L. Tessler

August 2016

Encouraging Evidence on a Sector-Focused 

Advancement Strategy 

Two-Year Impacts from the WorkAdvance Demonstration

Richard Hendra

David H. Greenberg

Gayle Hamilton

Ari Oppenheim

Alexandra Pennington

Kelsey Schaberg

Betsy L. Tessler

August 2016
WorkAdvance is one of five evidence-based programs that were implemented as part of the 2010 

Social Innovation Fund (SIF) grant to the Mayor’s Fund to Advance New York City and the 

Center for Economic Opportunity. The SIF unites public and private resources to evaluate and 

grow innovative community-based solutions with evidence of results. The SIF is a program of the 

Corporation for National and Community Service (CNCS), a federal agency that engages more 

than 5 million Americans in service through its AmeriCorps, Senior Corps, Social Innovation 

Fund, and Volunteer Generation Fund programs, and leads the President’s national call to 

service initiative, United We Serve. 

Matching funds for WorkAdvance have been provided by Bloomberg Philanthropies, Open 

Society Foundations, The Rockefeller Foundation, Altman Foundation, Common Bond Foundation, Ford Foundation, The Fund for Our Economic Future, George Kaiser Family Foundation, 

The Ira W. DeCamp Foundation, JPMorgan Chase Foundation, The Kresge Foundation, Robin 

Hood Foundation, Surdna Foundation, Tiger Foundation, Tulsa Area United Way, Tulsa 

Community Foundation, and The Harry and Jeanette Weinberg Foundation, Inc. 

Dissemination of MDRC publications is supported by the following funders that help finance 

MDRC’s public policy outreach and expanding efforts to communicate the results and implications of our work to policymakers, practitioners, and others: The Annie E. Casey Foundation, 

Charles and Lynn Schusterman Family Foundation, The Edna McConnell Clark Foundation, Ford 

Foundation, The George Gund Foundation, Daniel and Corinne Goldman, The Harry and Jeanette 

Weinberg Foundation, Inc., The JBP Foundation, The Joyce Foundation, The Kresge Foundation, 

Laura and John Arnold Foundation, Sandler Foundation, and The Starr Foundation.

 

In addition, earnings from the MDRC Endowment help sustain our dissemination efforts. Contributors to the MDRC Endowment include Alcoa Foundation, The Ambrose Monell Foundation, 

Anheuser-Busch Foundation, Bristol-Myers Squibb Foundation, Charles Stewart Mott Foundation, Ford Foundation, The George Gund Foundation, The Grable Foundation, The Lizabeth and 

Frank Newman Charitable Foundation, The New York Times Company Foundation, Jan Nicholson, Paul H. O’Neill Charitable Foundation, John S. Reed, Sandler Foundation, and The Stupski 

Family Fund, as well as other individual contributors.

This material is based upon work supported by CNCS under Grant No. 10SIHNY002. Opinions 

or points of view expressed in this document are those of the authors and do not necessarily 

reflect the official position of, or a position that is endorsed by, CNCS or the Social Innovation 

Fund. 

The findings and conclusions in this report do not necessarily represent the official positions or 

policies of the funders, The Mayor’s Fund to Advance New York City, or the New York City 

Center for Economic Opportunity.

For information about MDRC and copies of our publications, see our website: www.mdrc.org. 

Copyright © 2016 by MDRC®. All rights reserved.
CEO Response to MDRC’s Report on WorkAdvance: “Encouraging Evidence on a Sector-Focused 

Advancement Strategy”

August 2016 

In the aftermath of the Great Recession, policymakers have recognized that federal, state, and local workforce 

programs often lack the tools needed to connect low-income and unemployed jobseekers with true economic 

security. Since the mid-1990s era of welfare reform, government policy has favored rapid attachment to jobs 

over job quality. While this strategy can be effective in the short-term to connect program participants to 

jobs, it has not succeeded in creating long-term pathways out of poverty. The evidence in this report shows 

that the WorkAdvance approach offers an alternative that raises earnings for participants, including those 

who have been out of the labor market for a long time. 

WorkAdvance, a program of CEO and the Mayor’s Fund to Advance New York City, is designed to boost 

the earnings of unemployed and low-wage working adults by helping them obtain quality jobs in targeted 

sectors (such as healthcare or technology) with opportunities for career growth. WorkAdvance prepares, 

trains, and places unemployed and low-wage workers in good, high quality jobs with established career tracks. 

After placement, the program continues to assist participants to help them advance in their chosen careers. 

WorkAdvance, which was supported by the federal Social Innovation Fund, operates in New York City, 

northeast Ohio, and Tulsa, Oklahoma. 

WorkAdvance is the most recent iteration of a series of evaluations showing the potential of both sectorbased strategies and advancement-focused approaches. Public/Private Ventures’ Sectoral Employment 

Impact Study1 showed that sector-focused programs could increase earnings for participants. This evidence 

informed CEO’s work with the NYC Department of Small Business Services (SBS) to launch sector-focused 

career centers, which developed important additional evidence about the value of a sector-focused approach 

in general and training in particular.2 CEO developed WorkAdvance in partnership with MDRC to build on 

the existing evidence base by adding an explicit focus on career advancement, and examining the program 

effectiveness in a different labor market and in diverse cities. MDRC conducted a rigorous Randomized 

Controlled Trial impact evaluation of WorkAdvance and CEO is pleased to note that the results, which are 

detailed in the following report, show that this strategy works. In particular: 

 By the second year after entering the program, participants at three out of four sites saw annual 

earnings increase significantly over the control group. 

1 Maguire, Sheila, et al. “Job Training that Works: Findings from the Sectoral Employment Impact Study.” 

Public/Private Ventures, 2009. 

http://knowledgecenter.completionbydesign.org/sites/default/files/33%20Maguire%202009.pdf

2 Gasper, Joseph and Kathryn Henderson. “Sector-Focused Career Centers Evaluation: Effects on Employment and 

Earnings After One Year.” Westat, 2014. http://www.nyc.gov/html/ceo/downloads/pdf/CEOSector_Based_Approaches_Evaluation_Report-2014_final.pdf

 
 WorkAdvance was able to significantly increase earnings and employment rates for the long-term 

unemployed, a group that has proved resistant to many other policy interventions. 

 At the two sites that were newer to this approach, the program had much larger impacts on earnings 

and employment for those who began WorkAdvance later in the study period, showing that the 

model takes time to fully implement. 

 WorkAdvance significantly increased employment in the target sector at all providers. 

These findings, and others in the report, show that WorkAdvance succeeded in improving participants’ 

earnings and improving outcomes across a variety of job quality measures. In particular, it is heartening that 

the program made a significant difference for the long-term unemployed. The WorkAdvance evaluation 

comes at an opportune time, since the federal government recently began implementation of the Workforce 

Innovation and Opportunity Act (WIOA), which is now the governing law for most federal workforce 

funding nationwide. WIOA specifically calls on states and municipalities to utilize sector-based strategies and 

a career pathways approach, and these results show the potential of this approach to move the needle.

At the same time, the evaluation also shows that sector-based strategies take time to implement and are not a 

silver bullet to meet the needs of every jobseeker. As noted above, the impacts at newer providers were much 

stronger for those who enrolled after the program had been operating for a while. An earlier report on 

WorkAdvance documented a range of key implementation lessons from the project that will be helpful to 

others looking to replicate this work.3 

In addition, because WorkAdvance was explicitly employer-driven, providers were not always able to enroll 

potential participants with barriers that would have prevented them from getting jobs in the target sector 

during this time period. For instance, some potential participants’ literacy and/or numeracy levels were too 

low at the program outset and would have required remedial intervention that WorkAdvance was unable to 

provide during the program timeframe. These challenges point to the need for a more comprehensive 

workforce development system that provides a range of approaches for a variety of populations. CEO is 

participating in just such a revision of New York City’s workforce development system right now through 

Mayor de Blasio’s Career Pathways initiative.

4

Despite these challenges, WorkAdvance improved earnings for low-income and unemployed participants and 

this report can provide guidance to workforce providers and policymakers nationwide. CEO is pleased that 

MDRC plans to conduct ongoing research on the WorkAdvance participants over the next few years. As the 

workforce community moves towards a greater focus on sector-focused programs and career pathways, CEO 

is committed to continuing to inform this conversation. 

David Berman 

Director of Programs and Evaluation 

Patrick Hart 

Senior Advisor 

 

3

Tessler, Betsy, et al. “Meeting the Needs of Workers and Employers,” MDRC, 2014. 

http://www.mdrc.org/publication/meeting-needs-workers-and-employers

4

City of New York. “Career Pathways: One City Working Together.” 2014. 

http://www1.nyc.gov/assets/careerpathways/downloads/pdf/career-pathways-full-report.pdf
iii

Overview

This report summarizes the two-year findings of a rigorous random assignment evaluation of the 

WorkAdvance model, a sectoral training and advancement initiative. Launched in 2011, WorkAdvance goes beyond the previous generation of employment programs by introducing demand-driven

skills training and a focus on jobs that have career pathways. The model is heavily influenced by the 

positive findings from the Sectoral Employment Impact Study (SEIS) completed in 2010. A major 

component of the WorkAdvance model, in common with the programs studied in the SEIS, is 

formal training offering industry-recognized certifications, reflecting the hypothesis that skills 

acquisition is necessary for advancement. The model also requires providers to be far more employerfacing than traditional training programs, taking into account multiple employers’ changing skill 

requirements, employee assessment practices, and personnel needs. This report presents the implementation, cost, participation, and two-year economic impacts of WorkAdvance. The economic 

results are based on unemployment insurance earnings records and a second-year follow-up survey. 

The WorkAdvance program operations and evaluation are funded through the federal Social 

Innovation Fund (SIF), a public-private partnership administered by the Corporation for National 

and Community Service. This SIF project is led by the Mayor’s Fund to Advance New York City 

and the NYC Center for Economic Opportunity in collaboration with MDRC. 

Key Findings

• All providers translated the WorkAdvance model into a set of concrete services, but it took time 

— more than a year for some components and providers — and a substantial amount of technical assistance and support. As a result, at some sites, later study enrollees were more likely 

than earlier ones to experience a fully implemented and “mature” WorkAdvance program.

• Overall, WorkAdvance resulted in large increases in participation in every category of services, 

as well as in training completion, credential acquisition, and employment in the targeted sector,

compared with what would have happened in the absence of the program. Expenditures for the 

operation of WorkAdvance fell between $5,200 and $6,700 per participant at the four providers

delivering the program.

• WorkAdvance providers increased earnings, with variation in results that closely matched the 

providers’ experience in running sector-based programs and the extent to which the services 

they offered were demand driven. The most experienced sectoral provider, Per Scholas, had 

large and consistent impacts on both primary and secondary outcomes. Madison Strategies 

Group and Towards Employment, providers new to sectoral training, had promising but less 

consistent results that grew stronger for later enrollees. One provider, St. Nicks Alliance, did not 

produce positive impacts. The results did not differ dramatically across subgroups, though encouragingly, WorkAdvance was able to increase earnings among the long-term unemployed. 

The evaluation as a whole provides important information for workforce development providers 

interested in pursuing a sector strategy. The analysis considers the role played by providers’ sectorspecific training and preparation and the role played by the nature of the sectors themselves. Future 

priorities that emerge from the results are (1) understanding how to help the more disadvantaged access

the programs and (2) learning how to build service capacity, given how complex the model is to run.

v

Contents

Overview iii

List of Exhibits vii

Preface xv

Acknowledgments xvii 

Executive Summary ES-1 

Chapter

1 Introduction 1

The WorkAdvance Project 1 

The WorkAdvance Evaluation Design 14 

The Organization of This Report 25 

2 Implementation of the WorkAdvance Program 27

Chapter Highlights 27 

Introduction 27 

Key Findings 28 

Providers’ Backgrounds and Experience 30 

WorkAdvance’s Services 33 

Changes in Components and Participation Patterns over Time 57 

Conclusion 59 

3 WorkAdvance-Control Group Differences in Received Services 63

Chapter Highlights 63 

Introduction 63 

Summary of Participation Impacts 64 

Impacts on Participation in Career Readiness Services, by Site 69 

Impacts on Participation in Education and Training, by Site 73 

Impacts on Job Search Activities, by Site 83

Impacts on Postemployment Service Help Received, by Site 85 

Participation Impacts by Enrollment Cohort Varied 87 

Conclusion 89 

4 The Cost of the WorkAdvance Program 93

Chapter Highlights 93 

Introduction 93 

Key Concepts 94 

Data Sources and Methods 96 

Findings 98 

Conclusion 113 
vi

5 Economic Impacts of WorkAdvance 115

Chapter Highlights 115 

Introduction 115 

Impacts on Employment and Earnings 119 

Impacts on Wage Growth and Advancement 131 

Impacts on Job Benefits and Work Schedule 136 

Impacts on Total Income 139 

Impacts on Life Satisfaction and Material Hardship 142 

Conclusion 144 

6 Exploring Sources of Variation in the Economic Impacts 145

Introduction 145 

Were the Impacts Driven by the Targeted Sector? 145 

Did the Impacts Change over Time? 156 

Were the Impacts Affected by the Characteristics of Participants? 163 

Conclusion 170 

7 Conclusion 173

Summary of Findings 174 

A Comparison of WorkAdvance and SEIS Findings 176 

Overall Assessment 181 

Considerations for Policymakers and Providers Designing Future Sectoral 

Employment Programs 181 

Revisiting the WorkAdvance Theory of Change 183 

Conclusion 184 

Appendix

A Data Collection, Analysis Methods, and Methodological Issues 187

B Supplementary Exhibits for Chapter 1 251 

C Supplementary Exhibits for Chapter 2 259 

D Supplementary Exhibit for Chapter 3 269

E Net Cost Thought Experiment 273

F Supplementary Exhibits for Chapter 5 277 

G Further Analysis of Sources of Variation in the Economic Impacts 299 

References 321
vii

List of Exhibits

Table

ES.1 Key Details of WorkAdvance Service Providers ES-2

1.1 Labor Market Obstacles and WorkAdvance Responses 8

1.2 Institutional Structure of Providers 15

1.3 Data Sources 17

1.4 Selected Characteristics of Research Sample Members at Baseline and 

Comparison to National Low-Wage Workers, Cross-Site 22

2.1 Summary of Indicators of Participation in Program Group Activities 

Within 18 Months of Random Assignment, Cross-Site 30

2.2 Career Readiness Services, by Provider 38

2.3 Participation in Career Readiness Activities and Supportive Services 

Within 18 Months of Random Assignment, Cross-Site 39

2.4 Occupational Skills Training, by Provider 42

2.5 Participation in Skills Training Activities Within 18 Months of Random 

Assignment, Cross-Site 44

2.6 Postemployment Services, by Provider 52

2.7 Percentage of WorkAdvance Group Members Who Attended an 

Advancement Coaching Meeting, Cross-Site 55

2.8 Mode and Content of Advancement Coaching Meetings for 

WorkAdvance Group Members Who Attended a Meeting, Cross-Site 56

2.9 Content of Advancement Coaching Meetings for WorkAdvance Group 

Members Who Attended a Meeting While Employed, Cross-Site 58

2.10 Indicators of Participation in Program Group Activities Within 18 Months 

After Random Assignment, by Random Assignment Cohort, Cross-Site 60

3.1 Year 2 Impacts on Participation in Career Readiness Services, by Site 71
viii

Table

3.2 Year 2 Impacts on Participation in Education and Training, by Site 74

3.3 Year 2 Impacts on Participation in Primary Occupational Skills Training 

Program, by Site 78

3.4 Year 2 Impacts on Participation in Job Search Activities, by Site 84

3.5 Year 2 Impacts on Participation in Postemployment Services, by Site 86

3.6 Selected Participation Characteristics of Survey Respondents, by Site and 

Cohort 89

4.1 Aggregate Gross Operating Costs (Years 1 to 3) 99

4.2 Steady-State Gross Operating Costs (per Participant) 105

4.3 Gross and Net In-Program and Out-of-Program Costs (per Individual) 108

5.1 Year 2 Impacts on Employment and Wages, by Site 120

5.2 Quarters 2 to 10 Impacts on Unemployment Insurance-Covered 

Employment and Earnings, by Site 124

5.3 Impacts on Confirmatory Outcomes, by Site 132

5.4 Year 2 Impacts on Wage Growth and Advancement in Current or Most 

Recent Job, by Site 134

5.5 Year 2 Impacts on Current or Most Recent Job Characteristics, by Site 137

5.6 Year 2 Impacts on Income, by Site 141

5.7 Year 2 Impacts on Life Satisfaction and Material Hardship, by Site 143

6.1 Common Job Titles and Employers for WorkAdvance Respondents 

Working in the Targeted Sectors 147

6.2 Comparison of Characteristics of Targeted Sector Jobs and Nonsector

Jobs Among WorkAdvance Group Survey Respondents Employed

Since Random Assignment 155

6.3 Impacts for Subgroups Defined Based on Random Assignment Cohort, 

by Site 158

6.4 Impacts on Subgroups Defined by Baseline Labor Market Attachment, 

All Sites Combined 164
ix

Table

A.1 Selected Characteristics of Research Sample Members at Baseline, by 

Random Assignment Cohort and Site 195

A.2 Estimated Regression Coefficients for the Probability of Being in the 

Early Cohort 197

A.3 Comparison of MIS and Survey Reports of Participation in Targeted 

Sector Training Since Random Assignment Among WorkAdvance Group 

Survey Respondents, by Site 207

A.4 Year 2 Survey Response Rates, by Site 209

A.5 Estimated Regression Coefficients for the Probability of Being a 

Respondent to the WorkAdvance Year 2 Survey, by Site 214

A.6 Estimated Regression Coefficients for the Probability of Being a 

WorkAdvance Group Respondent to the WorkAdvance Year 2 Survey, 

by Site 216

A.7 Quarters 2 to 7 Impacts on Employment and Earnings for the Full and 

Survey Respondent Samples, by Site 218

A.8 Multiply Imputed Year 2 Impacts on Employment, Earnings, and Income, 

by Site 221

A.9 Minimum Detectable Effects, by Sample 228

A.10 Confidence Intervals, P-Values, Standard Errors, and Effect Sizes for Key 

Impacts, by Site 231

A.11 Comparison of Impacts on Income, Wages, Hours, Debt, and Earnings, 

by Level of Exclusion of Outlier Values 237

A.12 Adjusted Versus Unadjusted Impacts on Employment and Earnings, by 

Site 240

A.13 Percentage Missing on Key Outcomes, by Site 243

A.14 Employment in a UI-Covered Job Compared with Survey-Reported 

Employment at the Time of the Year 2 Survey Interview, Among Survey 

Respondents, by Site 248

B.1 Additional Characteristics of Research Sample Members at Baseline 253

C.1 Percentage Distribution of Applicant Recruitment Sources During Select 

Time Periods 261
x 

Table

C.2 Criteria for Study and Program Eligibility, by Provider, During Select 

Time Periods 262

C.3 Description of Program-Tracking Metrics, by Site 264

C.4 Percentage Distribution of Reasons for Drop-Off Among Applicants Who 

Attended Orientation 267

D.1 Year 2 Impacts on Educational Attainment, Internships, On-the-Job 

Training, and Employment Services, by Site 271

F.1 Year 2 Impacts on Current Job Characteristics, by Site 279

F.2 Additional Year 2 Impacts on Employment and Material Hardship, by 

Site 281

F.3 Year 2 Impacts on Household Composition, Housing Arrangement, and 

Health Care Coverage, by Site 284

F.4 Impacts on Employment and Earnings, by Random Assignment Sector, at 

Towards Employment 285

F.5 Quarters 2 to 10 Impacts on Employment Stability and Duration, by Site 287

G.1 Targeted Sector Job Characteristics, Among Control Group Survey 

Respondents Employed Since Random Assignment 306

G.2 Adjusted Versus Unadjusted Differences in Targeted Sector Versus NonTargeted Sector Employment and Advancement, Among WorkAdvance 

Group Survey Respondents Employed Since Random Assignment 307

G.3 Targeted Sector Job Characteristics, Among WorkAdvance Group 

Survey Respondents Employed Since Random Assignment with High 

Predicted Earnings 308

G.4 Targeted Sector Job Characteristics Among WorkAdvance Group Survey 

Respondents Employed Since Random Assignment with Low Predicted 

Earnings 309

G.5 Conditional Subgroup Impacts on Year 2 Earnings 310

G.6 Quarters 2 to 10 Impacts on Unemployment Insurance-Covered 

Employment and Earnings for Sample Members Demographically 

Similar to Per Scholas Sample Members 312

G.7 Impacts on Subgroups Defined by Age, All Sites Combined 313
xi

Table

G.8 Impacts on Subgroups Defined by Level of Education, All Sites 

Combined 314

G.9 Impacts on Subgroups Defined by Predicted Earnings in First Quarter of 

Year 3, All Sites Combined 315

G.10 Impacts on Subgroups Defined by Level of Earnings in the Three Years 

Before Study, All Sites Combined 316

G.11 Impacts on Subgroups Defined by Previous Conviction or Incarceration 

Status, All Sites Combined 317

G.12 Impacts on Subgroups Defined by Race/Ethnicity, All Sites Combined 318

G.13 Impacts for Subgroups Defined by Random Assignment Cohort, Per 

Scholas and St. Nicks Alliance (Training-First Only Sites) Combined 319

G.14 Impacts for Subgroups Defined by Random Assignment Cohort, Towards 

Employment and Madison Strategies Group (Dual-Track Sites) 

Combined 320

Figure

ES.1 Completion of Training in the Targeted Sector, by Site ES-8

ES.2 Employment in the Targeted Sector at Current or Most Recent Job, by 

Site ES-9

ES.3 Impacts on Earnings in Year 2, by Site ES-11

ES.4 Impacts on Earnings in Year 2, by Site and Random Assignment Cohort ES-12

ES.5 Impacts on Earnings in Year 2 for All Sites Pooled, by Prior Attachment 

to the Labor Market ES-15

1.1 Program Logic Model 5

1.2 Random Assignment Period and Report Sample Sizes, by Site, Analysis 

Sample, and Research Group 20

2.1 Percentage of Applicants Remaining, by Step in Intake Process 35

3.1 Summary of Year 2 Impacts on Participation in Employment Services 

and Training, by Site 66

3.2 Percentage of WorkAdvance and Control Group Members Participating 

in Occupational Skills Training, by Month Relative to Random 

Assignment and Site 76
xii

Figure

4.1 Percentage Distribution of Gross Operating Costs (Year 2) 101

5.1 Quarters 1 to 10 Impacts on Percentage Employed in an Unemployment 

Insurance-Covered Job, by Relative Quarter and Site 126

5.2 Quarters 1 to 10 Impacts on Mean Earnings from an Unemployment 

Insurance-Covered Job, by Relative Quarter and Site 128

6.1 Impacts on Employment Rates in Targeted Sector at Current or Most 

Recent Job, by Site 148

6.2 Activity Flow in Targeted Sector, Among WorkAdvance Group 

Respondents, by Site 149

6.3 Comparison of Characteristics of Targeted Sector Jobs and Nonsector 

Jobs Among WorkAdvance Group Survey Respondents Employed Since 

Random Assignment 153

A.1 Classification and Regression Tree (CART) Results, by Site 220

B.1 Annual Unemployment Rates 2011-2015, Nationally and by Metropolitan 

Area 255

B.2 Sector Employment in Quarter 3, 2011, and Quarter 3, 2013, by 

Metropolitan Area 256

F.1 Year 2 Impacts on Percentage Employed, by Month Relative to Random 

Assignment and Site 288

F.2 Year 2 Impacts on Average Monthly Earnings, by Month Relative to 

Random Assignment and Site 290

F.3 Year 2 Impacts on Percentage with Hourly Wage of $12 or More, by 

Month Relative to Random Assignment and Site 292

F.4 Impacts on Percentage Employed in an Unemployment InsuranceCovered Job in the Targeted Sector, by Relative Quarter, Site, and 

Random Assignment Sector 294

F.5 Quarters 1 to 10 Impacts on Percentage Receiving Unemployment 

Insurance Benefits, by Relative Quarter and Site 296

G.1 Activity Flow in Targeted Sector, Among Control Group Respondents, 

by Site 304

Box

2.1 Employer Perspectives on WorkAdvance 50

3.1 How to Read the Tables in This Report 65

3.2 To What Extent and Where Did Control Group Members Receive 

WorkAdvance-Like Services? 67
xiii

Box

4.1 Definitions of the Cost Categories 100

5.1 Confirmatory Economic Outcomes 133

6.1 Analyzing the Results of Figure 6.2 from Another Perspective 152

6.2 Why Have Impacts Failed to Emerge So Far at St. Nicks Alliance? 162

7.1 How Long Does It Take New Sector Programs to Hit Their Stride? 179

7.2 A Collaborative Way to Set Up a Sectoral Initiative: The Northeast Ohio 

Example 186

A.1 Sample Interview Questions for Participant Focus Groups and One-onOne Participant Interviews 199

A.2 Sample Interview Questions for Employers 202

A.3 Selected Interview Questions for Provider Staff Members Regarding 

Postemployment Component 204
xv

Preface

In recent years, the workforce development field has increasingly recognized that programs

must work closely with employers to more effectively serve the needs of job seekers. The 

sectoral programs that have emerged bring together employers, job seekers, community 

colleges, and other stakeholders to adopt coordinated regional economic and workforce development strategies that target industries and occupations with a growing demand for workers as 

well as the career ladders that afford opportunities for upward mobility. The aim of this coordination is both to promote upward mobility and to spur economic development by increasing the 

skills of the local workforce, in turn attracting and retaining employers. 

While the sectoral strategy has been around for many years, rigorous evidence on its 

potential benefits has emerged only recently, most notably in Public/Private Ventures’ Sectoral 

Employment Impact Study, completed in 2010. The study excited policymakers by showing

that the sectoral strategy can increase earnings. Yet it left many questions unanswered; the 

samples were small and the data sources relatively limited. Moreover, the providers in the study 

were well established, mature organizations. Could those same results be achieved with a 

broader range of providers and in a different economy? 

From this convergence of evidence and policy came WorkAdvance, which combined

the key elements of the Sectoral Employment Impact programs and the best of what was known

about advancement programs. The results, shown in this report, are encouraging. Together the 

four WorkAdvance sites helped participants earn an average of 14 percent (or nearly $2,000 in 

annual income) more than they otherwise would have earned two years after they entered the 

program. The effects differed by site, ranging from no earnings gain at one provider to a 26 

percent increase at the most effective provider. These results confirm that sectoral programs, 

when well implemented, can increase earnings among low-income individuals. 

It is important to note that the benefits to participants can take at least a year to emerge 

for experienced providers; for those new to the sectoral strategy, it will take longer. The results 

also show that sector programs must continually monitor developments in the industries they 

target and adapt as demand changes. An important priority for the field should be developing

guidance to help providers adjust to a dual-customer, demand-driven approach. 

As local Workforce Investment Boards and city and state leaders work to implement

the Workforce Innovation and Opportunity Act (WIOA), with its strong sectoral focus, they 

should consider these findings and draw upon the lessons herein. The challenge will be how to 

incorporate the WorkAdvance approach into larger workforce systems without losing the local

focus inherent in sector programs.

Gordon L. Berlin 

President

xvii

Acknowledgments

The writing of this report would not have been possible without the dedication and energy of the 

administrators and staff at all the provider organizations. The WorkAdvance program was made 

a reality under the guidance and leadership of Dale Grant, Doug Cotter, Chris Bernhardt, and 

Karen Pennington at Madison Strategies Group; Plinio Ayala, Angie Kamath, Kelly Richardson, Linda Quinones-Lopez, Venicia El-Amin, and Craig Rosenberg at Per Scholas; Jill Rizika, 

Rebecca Kusner, Chelsea Mills, and Sam Awad at Towards Employment; and Michael Rochford, Carolann Johns, and Janice Tosto at St. Nicks Alliance. In particular, we also want to 

thank all the career coaches, job developers, data managers, technical and career readiness instructors, and other staff members involved with WorkAdvance at the provider organizations 

for their positive attitude and tireless energy in bringing the WorkAdvance program to life.

Several people provided help with securing the necessary data: Vickie Maddux at the 

Ohio Department of Job and Family Services; John Miley, Jane Stout, Melissa Copenhaver, and 

Gina Antipov at the Oklahoma Employment Security Commission; Jeffrey Sorensen, Brian 

Close, and Shannon Hesnor at the New York State Department of Labor; Sofia Agosto at Per 

Scholas; Rachel Griffin at Madison Strategies Group; Jeff Stupiansky at Towards Employment;

Enrique Vasquez at St. Nicks Alliance; and Lenin Williams, Sylvia Epps, Monica Schneider, 

and Jim Cooper at Decision Information Resources. 

Many others contributed greatly to the implementation of the WorkAdvance program 

through technical assistance and consulting. We thank Mary Myrick, who assisted in the development of orientation and recruitment materials; her colleague Sandino Thompson, who was 

instrumental in guiding job development efforts; Jodie Sue Kelly, who led postemployment 

coaching trainings; Tirza Barnes-Griffith, who guided efforts to strengthen career readiness instruction; and Chris Spence, who assisted MDRC in fieldwork and analysis and provided meaningful input on best practices across many of the program’s components.

In addition to the generous funding of key partners who are acknowledged at the front 

of this report, we give thanks to Jim East, Bethia Burke, Brad Whitehead, and Kurt Karakul for 

helping build local support for the program. 

We are especially grateful to our colleagues at the New York City Center for Economic 

Opportunity (CEO), who managed the entire New York City Social Innovation Fund (SIF) project, worked closely with the providers to raise the SIF matching funds, brought providers together to share best practices and experiences through the SIF Learning Network, and monitored providers’ performance. In particular, we thank David Berman, Patrick Hart, Sinead Keegan, and Kristin Morse for their commitment to WorkAdvance.
xviii

At MDRC, Caroline Schultz, Donna Wharton-Fields, Stephanie Rubino, James Healy, 

Kristin Cahill Garcia, Vanessa Martin, Hannah Dalporto, and Herbert Collado all served as 

liaisons to the WorkAdvance providers. In addition to the coauthors, Mark van Dok and Anastasia Korolkova assisted in acquiring, processing, and analyzing the quantitative data. Alla 

Chaplygina helped with the exchange, logging, and archiving of data files for the project, and 

the random assignment system and data room staff, led by Joel Gordon, managed the study entry data and document collection process. Johanna Walter provided invaluable assistance in 

checking the cost study. Lee Robeson and Jo Anna Hunter served as liaisons with the survey 

firm. In addition, Gordon Berlin, Jean Grossman, Gayle Hamilton, James Riccio, Cynthia Miller, Frieda Molina, and Sonya Williams all provided feedback on report drafts, and Barbara

Goldman provided indispensable guidance throughout nearly all stages of the project to date. 

External reviewers Larry Katz and Sheila Maguire provided invaluable advice. Keith 

Olejniczak assisted with program operations and qualitative data collection and analysis, and 

Emma Saltzberg conducted special analyses and led the production of this report. Jennie Kaufman edited the report, and Carolyn Thomas and Ann Kottner prepared it for publication. Frieda 

Molina was the project director. 

Finally, our deepest gratitude goes to the individuals who contributed to the study by 

participating in the WorkAdvance demonstration, who have allowed us to learn from their experiences.

The Authors
ES-1 

Executive Summary 

Even in good economic times, many adults — particularly individuals with no more than a high 

school education — struggle to obtain and maintain jobs that pay enough to support their families and permit upward mobility. At the same time, some employers report difficulty finding 

employees with the right skills to meet their needs. This report summarizes the two-year effectiveness results of WorkAdvance — a workforce development program that seeks to meet the 

needs of participants and employers equally. For unemployed and low-wage working adults, the 

program provides occupational skills training in targeted sectors that have good quality jobs and 

room for advancement within established career pathways. The program’s strategy is a sectorbased one, in which program management and staff members also seek to fully understand — 

and fulfill — the skill requirements and other needs of employers in the targeted sectors.

In brief, the two-year WorkAdvance effectiveness findings are encouraging: A rigorous randomized controlled trial indicates that sectoral programs can increase earnings among 

low-income individuals. One provider had large and consistent impacts on both primary and 

secondary outcomes. Two providers, both of which were new to the sectoral strategy and 

gained experience over time, had promising, though less consistent, results that grew stronger 

for later entrants into the program. One provider did not produce positive impacts. The results 

also show that these types of programs are hard to run well, and, even when they are implemented well, impacts take time to emerge. The evaluation as a whole provides important information for workforce development providers interested in pursuing a sector strategy. The 

analysis considers the role played in the results by providers preparing participants for jobs in 

specific sectors and the role played by the nature of the sectors themselves.

WorkAdvance has been implemented in diverse settings by four providers specializing 

in specific sectors in which they have sought to develop relationships with employers and indepth industry knowledge: Per Scholas (in New York City) targeted the information technology 

(IT) sector; St. Nicks Alliance (also in New York City) focused on environmental remediation; 

Madison Strategies Group (in Tulsa, Oklahoma) focused on transportation and, later, manufacturing; and Towards Employment (in northeast Ohio) targeted health care and manufacturing. 

Table ES.1 presents information on the types of training provided at each site. The providers 

differed in a variety of ways. Two of the providers (Per Scholas and Madison Strategies Group) 

were solely focused on WorkAdvance and other sectoral training initiatives. By contrast, St. 

Nicks Alliance is a large multiservice organization with a small workforce division. Towards 

Employment runs a comprehensive employment program in addition to the WorkAdvance program, providing job placement and other services. The providers also differ significantly with 

regard to their experience. Per Scholas has been operating an IT sector program since 1998, and
ES-2 

The WorkAdvance Study

Table ES.1

Key Details of WorkAdvance Service Providers

Site 

(Location)

Per Scholas 

(Bronx, NY)

St. Nicks Alliance 

(Brooklyn, NY)

Madison Strategies 

Group

 (Tulsa, OK)

Towards Employment

(Cleveland, OH)

Targeted 

sector(s) Information technology Environmental 

remediation

Transportation, 

manufacturing

Health care, 

manufacturing

Study 

sample size 690 479 697 698

Structure Single program Division of large 

multiservice program Single program Multiple employment programs

Approach Training-first Training-first

Mixed until fall 2012 

(training-first and 

placement-first) and 

later predominantly 

training-first

Mixed until fall 2012 

(training-first and 

placement-first) and 

later predominantly 

training-first

 Occupational skills training

Length 15 weeks 5 to 12 weeks 4 to 32 weeks 2 to 17 weeks

Location On-site

On-site for pest control 

training; mix of on- and 

off-site at private 

schools for other 

training courses

Off-site at private or 

technical schools or 

community colleges

Off-site at private or 

technical schools or 

community colleges

Offerings A Plus, Network Plus

Environmental 

Remediation, 

Commercial Driver's 

License B with hazmat 

endorsement, Pest 

Control

Aviation 

Manufacturing, 

Commercial Driver's 

License A and B, 

Computerized 

Numerical Control 

(CNC) Machining, 

Diesel Mechanic, 

Welding, Supervisory 

Leadership

Computerized 

Numerical Control 

(CNC) Machining, 

Welding, Phlebotomy, 

Certified Health Care 

Access Associate, 

Patient Care Assistant, 

State-Tested Nurse 

Assistant, Medical 

Billing and Coding

SOURCES: MDRC sample size counts from the WorkAdvance baseline information form and other information 

from documentation supplied by providers and interviews with provider staff members. 
ES-3 

St. Nicks Alliance has given training in environmental remediation since 2001. By contrast, 

Madison Strategies Group was operating in a new city and Towards Employment was operating 

in a new sector (manufacturing). 

WorkAdvance’s programming and evaluation have been conducted under the auspices 

of the Social Innovation Fund (SIF). Administered by the Corporation for National and Community Service, the SIF is a public-private partnership designed to identify and expand effective 

solutions to critical social challenges. WorkAdvance is part of the New York City Center for 

Economic Opportunity (CEO) 2010 SIF project, which is led by CEO and the Mayor’s Fund to 

Advance New York City in collaboration with MDRC. MDRC is leading the WorkAdvance 

evaluation; has provided technical assistance to the providers; and, jointly with CEO, has monitored providers’ operations. Funding for the WorkAdvance program and evaluation came from 

the SIF and a broad array of local funding partners.

The WorkAdvance Program Model

The WorkAdvance model goes beyond the previous generation of employment programs by 

introducing sector-based skills training and a focus on jobs that have career ladders or pathways, 

rather than just seeking “any job” or an initial credential for participants. The model is heavily 

influenced by the positive findings from the Sectoral Employment Impact Study (SEIS) completed in 2010.1 A major component of the WorkAdvance model, in common with the programs 

studied in the SEIS, is formal training offering industry-recognized certifications, reflecting the 

hypothesis that skills acquisition is necessary for advancement. Additionally, the WorkAdvance 

model requires providers to be far more employer-facing than traditional training programs, taking into account multiple employers’ changing skill requirements, employee assessment practices, and human resource needs. 

The essential theory behind WorkAdvance is that strategic, demand-guided upgrades in 

human capital — that is, education and employment-related skills and experience — will eventually lead to advancement in the labor market. This theory informs the key components of the 

WorkAdvance model:

1. Intensive screening of program applicants before enrollment — a practice 

not always found in employment programs offered to low-income individuals — intended to ensure that the program providers select participants who 

can take advantage of the skills training in the sector and be qualified for 

specific occupations within it. 

 1

Sheila Maguire, Joshua Freely, Carol Clymer, Maureen Conway, and Deena Schwartz, Tuning In to 

Local Labor Markets: Findings from the Sectoral Employment Study (Philadelphia: Public/Private Ventures, 2010).
ES-4 

2. Sector-appropriate preemployment and career readiness services, consisting of an orientation to the sector, career readiness training (which stresses 

how to search for work in the sector and how to comport oneself on the job), 

individualized career coaching, and limited supportive services to sustain 

program engagement and assist participants to complete their training and 

find employment.

3. Sector-specific occupational skills training, intended to impart skills and 

lead to credentials that will substantially enhance workers’ employment opportunities. Training is geared toward current job openings in specific sectors 

and occupations, and training offerings are to be adapted to changes in available jobs. (In the first half of the program enrollment period, the goal was to 

place approximately half the enrollees at two providers into a placement-first 

track, in which they skipped the initial training phase, so the model’s training 

component was not offered to all program enrollees.)

4. Sector-specific job development and placement services, intended to facilitate entry into positions for which participants have been trained and for 

which there are thought to be genuine opportunities for continued skill development and career advancement. The providers’ job developers are to 

maintain strong relationships with employers who hire individuals with the 

kinds of skills the program imparts. 

5. Postemployment retention and advancement services, provided to assist 

participants beyond the placement stage. Providers are to maintain close contact with workers and employers to assess performance, offer coaching to 

address any complicating life situations that might arise for workers, help 

identify next-step job opportunities and skills training to enable participants 

to move up career ladders over time, and assist with rapid reemployment if 

workers lose their jobs. It should be noted that the full effects of these services may not yet be seen in this report given that only slightly over two 

years of follow-up data are available. 

Evaluation Design and Study Enrollee Characteristics 

The WorkAdvance programs at the four sites were studied using a random assignment design, a 

methodology often referred to as the “gold standard” in evaluation research, which allows practitioners and policymakers to have a very high degree of confidence in the results. From June 

2011 to June 2013, individuals who met the WorkAdvance eligibility criteria, as well as the requirements for the specific sector programs, were assigned at random to the WorkAdvance 
ES-5

group or to the control group. Members of the WorkAdvance group were eligible to receive 

WorkAdvance services, while those in the control group were not eligible for these services but 

could get other services and support available in the community. Both research groups were 

tracked over time. The random assignment process ensured that, when individuals entered the 

study, there were no systematic differences in sample members’ characteristics, measured or 

unmeasured, between the two research groups. Thus, any differences between the groups that 

emerge after random assignment can be attributed to the WorkAdvance program.

During the evaluation period, WorkAdvance targeted unemployed and low-wage working adults with a monthly family income below 200 percent of the federal poverty level. Adults 

who were working at the time they entered the study were required to be earning less than $15 

per hour. Sector-specific requirements might include drug testing, literacy or numeracy testing, 

or screens for criminal history, and in many cases individuals had to demonstrate that they could 

engage in training full time and that they had a high degree of interest in a career in the sector. 

Across the sites, 2,564 individuals were enrolled in the study and randomly assigned to either 

the WorkAdvance group (1,293) or the control group (1,271). Site-level sample sizes range 

from 479 at the St. Nicks Alliance site up to 698 at Towards Employment.2

Across the sites, the average WorkAdvance sample member was 34 years old, and the 

majority of sample members were male (73 percent) and single (67 percent). (These averages 

mask important variation by site and sector. For example, the health care sector at Towards 

Employment mostly enrolled females.) Half the participants were black/African-American, 18 

percent were white, and 17 percent were Latino/Hispanic. Almost all sample members had at 

least a high school diploma or equivalent, and over half the sample had at least some college 

education, though most did not have a college degree of any type. Only one in five were working at the time they entered the study, and more than one-third of sample members had been 

unemployed for at least seven months immediately preceding study entry. Overall, 15 percent of 

sample members had work experience in their targeted industry. At study entry, less than half 

the enrollees were covered by health insurance, more than one-third were receiving Supplemental Nutrition Assistance Program (SNAP) benefits (food stamps), 16 percent were receiving 

unemployment insurance benefits (even though most were unemployed), and 6 percent were 

receiving Temporary Assistance for Needy Families (TANF). Despite fairly high levels of education and work experience, many sample members still faced substantial barriers to employment: One-quarter of enrollees, for example, had a criminal conviction, and even higher rates 

were seen among transportation and manufacturing sector enrollees.

 2

Attrition from the study was minimal: Only 20 individuals (less than 0.8 percent of the original sample) 

withdrew from the study at any point following their random assignment.
ES-6 

This report extends the evaluation findings of two previous documents,3 addressing a 

number of research questions: How did the four providers implement the program model, and 

what adjustments did they make over time? To what extent did WorkAdvance increase engagement in employment and training services, compared with what would have happened in 

the absence of the program (the control group)? What does it cost to operate the WorkAdvance 

program? To what extent did WorkAdvance, over a two-year follow-up period, improve individuals’ employment, earnings, income (including public assistance), and life satisfaction? And 

what helps explain the patterns in WorkAdvance’s economic effects — for example, what is the 

influence of program maturity? 

Sources of data analyzed in this report to determine WorkAdvance’s effectiveness include a follow-up survey, in which the average survey respondent was interviewed 22 months 

following his or her random assignment (the “Year 2 Survey,” for which about 80 percent of the 

entire study sample was interviewed), and unemployment insurance data, available for nine

quarters (2.25 years) following random assignment (obtained for all sample enrollees). Thus, all 

study enrollees have approximately two years of common follow-up, in most cases from both 

the survey and the unemployment insurance administrative records. 

Key Findings 

Analyses in this report yielded the following key findings. 

Implementation and Participation Findings

• Translating the WorkAdvance model into a set of concrete services took 

time — more than a year for some components and providers — and a 

substantial amount of technical assistance and related support; some 

providers required more time and assistance than others. As a result, at 

some sites, a fully implemented and “mature” WorkAdvance program 

was more likely to be experienced by later study enrollees than by early 

study enrollees.

All four providers eventually implemented all the WorkAdvance model components, 

with postemployment services the last to be fully developed and implemented. The engagement 

of WorkAdvance group members in career readiness services, occupational skills training, and 

postemployment retention and advancement services was high for all four providers, in part re-

 3

Betsy Tessler, WorkAdvance: Testing a New Approach to Increase Employment Advancement for LowSkilled Adults (New York: MDRC, 2013), and Betsy L. Tessler, Michael Bangser, Alexandra Pennington, 

Kelsey Schaberg, and Hannah Dalporto, Meeting the Needs of Workers and Employers: Implementation of a 

Sector-Focused Career Advancement Model for Low-Skilled Adults (New York: MDRC, 2014).
ES-7

flecting the pre-random assignment intake process that screened out individuals who were not 

motivated or able to complete the intake process, who did not have the literacy levels needed to 

take full advantage of the training, or who had other impediments to being hired in the sector.

While all WorkAdvance providers delivered the model components, differences in implementation across the providers were apparent. Per Scholas’s history of providing information 

technology training and its long-standing connections with employers in that sector gave it a 

head start on the other providers. Because it did not outsource skills training but provided it onsite, Per Scholas staff members also had more opportunities to interact with program participants, deliver career readiness services, and reinforce workplace behaviors. Madison Strategies 

Group and Towards Employment initially operated a dual-track approach, in which some enrollees were first placed in jobs, before being offered training opportunities. This resulted in early 

enrollees at those two sites being less likely than enrollees at the other two sites to ever start and 

complete occupational skills training. These two providers were also the newest to sectoral 

training. For these reasons, later program enrollees (the “late cohort”) probably experienced a 

stronger program at those two sites. Finally, while all providers had ties to employers in their 

targeted sectors, some providers had more experience being demand driven than others or had 

closer ties to employers. Per Scholas had the most experience delivering demand-driven services (which means that employer input is evident in all program components). 

• Across all sites, WorkAdvance resulted in very large increases in participation in every category of services, as well as in training completion 

and credential acquisition, compared with what would have happened 

in the absence of the program.

On their own initiative the control group members obtained a range of employment services, including (in some cases) training in the targeted sector, from community colleges and 

other providers. It is not surprising that the control group members found their way to services,

given that they had to be quite motivated to persist through the WorkAdvance screening process. WorkAdvance, however, resulted in many more individuals participating in training, as 

well as in career readiness, job search, and postemployment services, over a two-year follow-up 

period, and the evaluation is thus in a good position to measure the economic value that these 

activities and services can add. For example, WorkAdvance increased participation in vocational training in the targeted sector by approximately 40 percentage points or more at every site,

relative to control group members’ levels (which ranged from 13 percent to 21 percent depending on the site). WorkAdvance also increased the likelihood of completing such training by 31 

percentage points or more (as shown in Figure ES.1) and increased the likelihood of attaining a 

vocational training credential in the targeted sector by 25 percentage points or more. These results are impressive, particularly given that two of the providers — Madison Strategies Group 
ES-8

and Towards Employment — placed approximately half their early enrollees in a placementfirst track. MDRC calculations indicate that expenditures on the operation of WorkAdvance 

were within the range of $5,200 to $6,700 per program group study participant at the four providers delivering the program.

Economic Impact Findings

• The WorkAdvance programs at all providers increased participants’ 

employment in the targeted sector, but the size of the impact varied substantially across the sites. 

Figure ES.2 shows that at all four sites WorkAdvance increased employment in the targeted sector, relative to the experiences of control group members. Of all the sites, Per Scholas

Figure ES.1

Completion of Training in the Targeted Sector, by Site

The WorkAdvance Study

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA group = WorkAdvance (program) group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

57.9

51.8

39.3

47.9

12.0

20.2

8.1 10.8

0

10

20

30

40

50

60

70

80

90

100

***

Per Scholas

***

St. Nicks Alliance

***

Madison Strategies

Group

***

Towards Employment

WA group Control group

Percentage
ES-9 

produced by far the largest impact on employment in the targeted sector: 61 percent of WorkAdvance group members, when surveyed, reported working in the information technology sector at their current or most recent job, a striking 41 percentage points higher than was the case 

for control group members. St. Nicks Alliance had the lowest rate of WorkAdvance group 

members who reported current or recent work in the targeted sector — 32 percent — but its impact of 12 percentage points on this measure is still statistically significant. Madison Strategies

Group and Towards Employment both increased current or recent targeted sector employment 

by a little over 16 percentage points, even with relatively high control group rates.

The WorkAdvance Study

Figure ES.2

Employment in the Targeted Sector at Current or Most Recent Job, by Site

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA group = WorkAdvance (program) group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

61.1

31.8

64.6

50.1

20.4 19.9

48.2

33.6

0

10

20

30

40

50

60

70

80

90

100

***

Per Scholas

***

St. Nicks Alliance

***

Madison Strategies

Group

***

Towards Employment

WA group Control group

Percentage
ES-10

• Impacts on earnings — from any type of job in any sector — varied 

across the sites, in a pattern that closely matched the providers’ experience in running sector-based programs and the extent to which the services they offered were demand driven. 

Figure ES.3 shows impacts on earnings in Year 2 of the follow-up period. Early in the 

follow-up period, WorkAdvance group members’ earnings levels were lower than those of the 

control group owing to the well-known labor market opportunity cost of training participation. 

This situation, however, changed for three of the four providers in Year 2. At Per Scholas 

(which has long operated an information technology sector training program), WorkAdvance

increased earnings by over $3,700 (or 26 percent) above the control group level in Year 2. At 

Madison Strategies Group (which was a spin-off from an experienced provider that sought to 

operate programs in a new area of the country) and Towards Employment (an organization 

relatively new to working in one of the sectors in which it operated for WorkAdvance), statistically significant impacts on earnings began to emerge in Year 2, particularly toward the end 

of that year. At St. Nicks Alliance, statistically significant impacts on earnings had not yet 

emerged by Year 2.4

• At the two sites where many early enrollees were channeled into a 

placement-first track, impacts were stronger for the late enrollee cohort 

than for the early enrollee cohort, as the providers matured in their ability to deliver services and switched to a mostly training-first model.

Figure ES.4 shows the impacts of WorkAdvance by random assignment cohort — an 

analysis that in the evaluation’s planning stage was seen as critically important. Two of the programs (Madison Strategies Group and Towards Employment) both initially implemented a 

mixed model, in which the goal was to place half the program participants into jobs in the targeted sectors before training. As shown in Figure ES.4, at both providers impacts were dramatically stronger for individuals who enrolled in the second half of the intake period than for individuals who enrolled in the first half. It is difficult to determine reliably the extent to which the 

better results for the later cohort were due to a greater emphasis on training versus an overall 

maturation of the program at these providers, both of which were new to sectoral programming.

 4

As discussed in the report, two measures were prespecified as “confirmatory” measures: earnings in the 

last quarter available for the full sample (which originally was planned as Quarter 7, but was changed to Quarter 10 after data collection was extended) and employment (in any sector) at the time of the survey. Only the 

Per Scholas program produced statistically significant impacts if just these two measures are considered. Madison Strategies Group and Towards Employment produced impacts that are not quite statistically significant on 

these measures, though the impacts on earnings in Quarter 10 are statistically significant when those two sites 

are pooled. St. Nicks Alliance did not produce impacts on either measure.
ES-11

The WorkAdvance Study

Figure ES.3

Impacts on Earnings in Year 2, by Site

18,217

14,420

16,641

13,223

14,471 14,229 14,804

11,602

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

18,000

20,000

***

Per Scholas St. Nicks Alliance

*

Madison Strategies

Group

*

Towards Employment

WA group Control group

Earnings ($)

The cohort analysis is the least relevant to Per Scholas, as it was the most mature provider coming into the study. However, the results indicate that program effects were somewhat 

weaker for the later cohort at that site than for the early cohort; the reasons are unclear, but these 

results could relate to the strengthening local economy during the study period (and perhaps to 

other factors as well). At St. Nicks Alliance, no improvement in impacts was evident for the 

later cohort relative to the early cohort. 

SOURCES: MDRC calculations from unemployment insurance administrative records provided by New York 

State Department of Labor for Per Scholas and St. Nicks Alliance sample members; Ohio Department of Job 

and Family Services for Towards Employment sample members; and Oklahoma Employment Security 

Commission for Madison Strategies Group sample members. 

NOTES: WA group = WorkAdvance (program) group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.
ES-12

The WorkAdvance Study

Figure ES.4

Impacts on Earnings in Year 2, by Site and Random Assignment Cohort

17,318

12,482

13,825

9,690

12,351 12,722 13,902

9,559

0

5,000

10,000

15,000

20,000

***

Per Scholas St. Nicks

Alliance

Madison

Strategies

Group

Towards

Employment

Early Cohort

WA group

Control group

Earnings ($)

18,809

16,697

19,549

16,598 17,466

15,945 15,424

13,602

0

5,000

10,000

15,000

20,000

Per Scholas St. Nicks

Alliance

***

Madison

Strategies

**

Towards

Employment

WA group

Control group

Late Cohort

Group

SOURCES: MDRC calculations from unemployment insurance administrative records provided by New York State Department of Labor for Per Scholas 

and St. Nicks Alliance sample members; Ohio Department of Job and Family Services for Towards Employment sample members; and Oklahoma 

Employment Security Commission for Madison Strategies Group sample members. 

NOTES: WA group = WorkAdvance (program) group.

The early cohort includes sample members randomly assigned through Quarter 3, 2012. The late cohort includes all sample members randomly assigned 

in and after Quarter 4, 2012. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. (continued)
ES-13

• The program at Per Scholas produced impacts on the widest array of 

secondary employment outcomes. However, most providers produced 

impacts on some indicators of career advancement or advancement

potential. 

The WorkAdvance program at Per Scholas increased hourly wages and weekly earnings, and the Madison Strategies Group program increased the likelihood of participants working at jobs with higher wages. WorkAdvance had no effect on average wages or earnings in the 

current or most recent job (as of the survey administration) at the other sites and did not affect 

the likelihood of working full time at any of the sites. 

At all sites except for St. Nicks Alliance, WorkAdvance group members were more 

likely than their control group counterparts to report that the job they held currently or most recently as of the survey administration offered many opportunities for career advancement. 

WorkAdvance also improved other aspects of job quality and advancement. Notably, WorkAdvance increased the proportion of individuals who received several types of employment-related 

benefits at Madison Strategies Group; improved job satisfaction at both that site and Per Scholas; and increased the proportion of people who reported working in a regular permanent job, as 

opposed to a temporary one, at Towards Employment. 

The impact analysis also examined a range of measures related to income, material 

hardship, and overall well-being. Only WorkAdvance at Per Scholas produced impacts in all 

these domains. The Per Scholas program increased income, reduced material hardship, reduced 

public assistance usage, and increased overall life satisfaction. It is unusual to see such a consistent pattern of impacts across so many domains. At the other sites, only a few impacts on 

such measures are statistically significant.

• The extent to which WorkAdvance increased employment in targeted 

sector jobs was the critical factor in explaining the pattern of impacts 

across the sites. At all the sites, jobs in the targeted sector were generally 

of higher quality than jobs outside the targeted sector.

For a sectoral program to produce employment and earnings impacts, the program 

needs to increase employment in the targeted sector, and the targeted sector jobs have to have 

better characteristics than jobs outside the targeted sector (that is, there needs to be a “sector 

premium”). In terms of increasing targeted sector employment, Per Scholas was by far the most 

successful. As shown in Figure ES.2, the 41 percentage point impact on targeted sector employment at that site was more than twice as large as the impact at any of the other sites and almost four times as large as the impact at St. Nicks Alliance. This differential in targeted sector 

employment appears to explain why Per Scholas produced the strongest impacts and why St. 

Nicks Alliance produced the weakest impacts. 
ES-14

At most sites, an examination of the jobs held by WorkAdvance group members who 

worked following random assignment suggests that targeted sector jobs were better than jobs 

outside the targeted sector. At Per Scholas, St. Nicks Alliance, and Madison Strategies Group, 

targeted sector jobs paid more than other jobs that WorkAdvance group members found. In addition, at most sites, the typical number of hours required by the jobs was higher, or other job 

characteristics, such as offered benefits, tended to be better, for jobs WorkAdvance group 

members held in the targeted sector than for jobs they held outside the targeted sector. Thus, 

there was no notable variation among the sites in the “premium” added for targeted sector jobs 

versus non-targeted sector jobs. Instead, the overriding factor for why the impacts varied from 

site to site appears to be differences in the extent to which the sites increased placements in the 

targeted sector. 

• The results also highlight that sector programs can be hard to run well. 

The WorkAdvance model was intended to go well beyond traditional workforce development models by incorporating a sectoral and advancement emphasis. It will take time for 

many providers to develop these capacities. In particular, the St. Nicks Alliance WorkAdvance 

program confronted numerous difficulties in adapting its more traditional vocational training 

program to the Work Advance model, which may explain why impacts have not emerged, at 

least through this report’s follow-up period. St. Nicks Alliance is a highly experienced community-based multiservice provider with a relatively small workforce division. The WorkAdvance 

program at St. Nicks Alliance experienced a collapse in the demand for environmental remediation work early in the program period and faced challenges in responding to these changes. A 

more effective response would have required a more proactive approach with employers than 

St. Nicks had previously used. 

• Pooling the results from the four providers, the economic impacts of 

WorkAdvance are positive and statistically significant, but this masks 

considerable variation at the site level. 

During the evaluation’s analysis planning phase, a decision was made to focus the impact analysis at the site level,5 given the providers’ widely varying experience with WorkAdvance-like services and sectoral strategies. Still, it is instructive for policymakers to understand 

how a strategy like WorkAdvance might perform overall, across a range of providers and locations. Figure ES.5 shows, for all the sites pooled together (left set of columns), WorkAdvance’s 

impact on earnings in the post-training period (Year 2). As a whole, WorkAdvance increased 

 5

Initially, the analysis plan called for an examination of impacts by sector. This changed after a site was 

dropped from the study, because the sector results would have lacked sufficient statistical power. The decision 

to examine impacts by site was made well before impact results were available.
ES-15

The WorkAdvance Study

Figure ES.5

Impacts on Earnings in Year 2 for All Sites Pooled,

by Prior Attachment to the Labor Market

15,717

18,540

16,188

13,440 13,772

17,540

13,815

11,093

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

18,000

20,000

***

Full sample Fully attached

**

Semiattached

***

Long-term

unemployed

WA group Control group

Earnings ($)

SOURCES: MDRC calculations from unemployment insurance administrative records provided by New York 

State Department of Labor for Per Scholas and St. Nicks Alliance sample members; Ohio Department of Job 

and Family Services for Towards Employment sample members; and Oklahoma Employment Security 

Commission for Madison Strategies Group sample members. 

NOTES: WA group = WorkAdvance (program) group.

The fully attached subgroup consists of sample members who were working at random assignment or who 

were unemployed for less than one month before random assignment. The semiattached subgroup consists of 

sample members who were unemployed for one to six months before random assignment. The long-term 

unemployed subgroup consists of sample members who have never been employed or who were unemployed 

for seven or more months before random assignment.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.
ES-16

earnings by about $1,945, or about 14 percent, above the control group level in Year 2. The results show that with a configuration of sites similar to those evaluated in WorkAdvance, policymakers can expect positive impacts on earnings overall. It is important, however, to recognize 

that this overall effect will mask considerable variation at the site level. For example, without 

Per Scholas, the pooled impacts are less than half as large, though still statistically significant.

• WorkAdvance produced positive impacts for the long-term unemployed.

WorkAdvance operated during the long wake of the Great Recession of 2007-2009. 

During this period, the number of people who qualified as long-term unemployed increased 

markedly, and there was significant concern about the likelihood of reengaging this group in the 

labor market. Given this concern, the analysis examined WorkAdvance’s effects for subgroups

defined by individuals’ prior level of attachment to the labor market (Figure ES.5). This analysis found that WorkAdvance produced statistically significant impacts on employment and 

earnings for both the long-term unemployed and those who were semiattached to the labor 

market.6

Conclusions 

• The WorkAdvance results show that sectoral programs can increase 

earnings among low-income individuals. But even when a program is 

well implemented, the benefits take time to emerge, for providers that 

have lengthy experience in sectoral training as well as for providers that 

are new to the strategy.

The Per Scholas program has now been shown to produce large impacts in two separate 

studies: in this study as well as in the Sector Employment Impact Study (SEIS).7 The WorkAdvance results thus provide important validation that a mature and highly functioning sectoral 

training provider can produce large impacts on a consistent basis. More generally, the results 

provide confirmation of the SEIS results, which showed that three separate providers (including 

Per Scholas) produced large effects on low-income individuals’ earnings. The WorkAdvance 

 6

For this analysis, long-term unemployment was defined as never having been employed or having been 

unemployed for seven months or more immediately preceding study entry. Note that the more positive results 

among the long-term unemployed are related to the fact that sites that served more long-term unemployed 

workers had larger impacts in general. Therefore, while the results show that WorkAdvance can work for the 

long-term unemployed, the results do not necessarily imply that the program will always work better for that 

group, nor do they demonstrate that the program will be less effective among the recently or currently employed. In general, a variety of analyses suggest that the characteristics of sample members in each site did not 

explain the variation in impacts across the sites. 7

Maguire et al. (2010).
ES-17

impacts at Per Scholas are evident using both survey data and unemployment insurance records 

and across a large number of primary and secondary outcomes. Moreover, the size and consistency of the impacts of Per Scholas’s sector training program are unusual in the history of 

employment and training studies. 

The WorkAdvance results also highlight that in sectoral training programs, impacts take 

time to emerge, for two reasons. First, for participants, it takes time for the training and advancement services to be completed and for them to translate into job placements and work advancement. Even for individuals enrolled at the experienced Per Scholas, impacts did not 

emerge until the second year of follow-up. Second, for providers new to a sector, it takes time 

for program staff relationships with employers and staff specialization in sector practices to take 

root and to result in strong and tailored training programs. Both these factors were clearly in 

play at Madison Strategies Group and Towards Employment, two providers that implemented 

WorkAdvance well, but which were new to their sector or location. At these two sites, impacts 

on participants’ earnings emerged only in the second year of follow-up. In addition, the impacts 

were much stronger for the late-enrolling cohorts, who experienced stronger programs. 

It is likely that the wide range of experience and backgrounds among the WorkAdvance 

providers is typical of the landscape of workforce development providers across the nation, and 

thus the WorkAdvance findings have important implications for Workforce Innovation and Opportunity Act (WIOA) implementation and workforce practice in general. (Many of these implications are discussed in the concluding chapter of the report.) In this sense, the WorkAdvance 

evaluation provides the field with a reliable indication of how the sectoral strategy will work 

among a representative range of providers. When all sites are pooled together, the results of 

WorkAdvance are positive and statistically significant. This implies that, on average, the programs can be effective. But the report results also show that the size and particularly the timing 

of impacts may vary critically based on where providers start from in terms of their experience 

in the sector. The sectoral strategy requires highly capable providers, and the report results made 

it clear that not all providers will produce positive impacts via the sectoral training approach. 

The results also highlight that for the sectoral and postemployment advancement strategy featured in WorkAdvance to translate into earnings gains, providers need to focus on increasing 

targeted sector employment in jobs that have higher wages and better characteristics than the 

jobs participants can find on their own in other sectors. This, in turn, will happen only if program services are truly demand driven, which requires that program leaders and staff members 

at all levels understand many aspects of their targeted sector — its language, needs, metrics, and 

skill requirements. 

* * * 
ES-18

While this report presents the final WorkAdvance impact estimates under the project’s 

Social Innovation Fund grant, additional reports targeted to specific audiences are planned. Future publications will consider the implications of the WorkAdvance experience for practitioners and for the career pathways field in general. Furthermore, longer follow-up data will be collected at the three- and five-year points. The results will provide important insights into whether 

impacts grow over time and about the effects of postemployment advancement services. A full 

benefit-cost analysis will also be conducted. 
1

Chapter 1

Introduction

The WorkAdvance Project

Even in good economic times, many adults in the United States have difficulty obtaining jobs 

and advancing in careers that pay enough to support their families. In particular, individuals 

with no more than a high school education have seen their wages remain flat in real terms for 

decades, and their employment is often unsteady.1 Training programs for low-skilled adults often fail to prepare participants for sustained employment and upward mobility, especially if the 

programs do not lead to a marketable credential,

2 or if they do not focus on jobs in high-demand 

occupations with genuine advancement opportunities. At the same time, some employers report 

difficulty finding people with the right skills to meet their needs, even in periods of high unemployment.3

Amid much debate about how workforce policy should address these concerns, there is 

a continuing need for clearer evidence on the best ways to promote upward mobility. The 

WorkAdvance evaluation seeks to fill the gap in evidence by testing the effectiveness of a model that builds on previous research and practitioners’ experience in two especially important 

areas of workforce policy: sectoral training and strategies for job retention and career advancement. 

Sectoral training focuses on preparing individuals for quality jobs that employers are 

seeking to fill in specific high-demand industries or occupational clusters. A key element of this 

approach is to address the needs of employers and workers simultaneously. Although variations 

of sectoral strategies have been used for some time, increased interest was sparked by the results 

of a rigorous random assignment evaluation4 — Public/Private Ventures’ Sectoral Employment 

Impact Study (SEIS),5 completed in 2010 — as well as those of other tests that used less rigorous research designs, including promising results from a study of sector centers sponsored by 

the New York City Center for Economic Opportunity (CEO) that directly influenced the design 

 1

Mishel, Bernstein, and Shierholz (2009).

2

Hamilton and Scrivener (2012b).

3

Morrison et al. (2011); Holzer (2013). 

4

In a random assignment evaluation, eligible individuals who apply for a program are assigned at random 

either to receive the program’s services or not to receive them. If sample sizes are large enough, the difference 

between the two groups’ outcomes — referred to as “impacts” — can be attributed to the program, since the 

two groups were statistically alike at baseline and the only difference between them is that one group received 

program services and the other did not. A random assignment study (also known as a randomized controlled 

trial) is widely held to be the most reliable way to study a program’s effectiveness, providing the highest level 

of evidence. 5

Maguire et al. (2010). 
2

of WorkAdvance.6 The SEIS, which examined three small programs operated by organizations 

with experience in sector-focused efforts, found substantial improvements in individuals’ employment, earnings, and wage rates over a two-year follow-up period.7

A key distinction between sectoral strategies and traditional employment and training 

programs is that sector programs adopt a dual-customer approach. For example, a training provider that trains in a specific field but does not have strong relationships with employers or industry associations in that field would not be considered a sectoral provider under this definition. To qualify as a sector program, an initiative must bring together multiple employers in a 

given field to collaborate on developing a qualified workforce.8 Many training programs focus 

more on the participants and work with employers only during the job placement phase. A sector program works with employers at every stage of programming and often invites employers 

on-site for mock interviews, to consult about curriculum design, or even to provide hands-on 

training.

WorkAdvance also draws on lessons from efforts to improve the job retention and 

career advancement of low-skilled workers after initial job placement. Retention-andadvancement programs have had mixed results, but much has been learned about what is likely 

to be effective and, equally important, ineffective. Particularly relevant for WorkAdvance is the 

hypothesis that concrete postemployment support — such as coaching tied to specific career 

paths and proactive reemployment services when a participant loses a job — could help individuals not only maintain their employment, but also continue to increase their earnings over 

time.9 By integrating the most promising features of sectoral and retention and advancement 

strategies, the designers of WorkAdvance were hopeful that this combination of services would 

produce larger and longer-lasting effects on employment, earnings, and career paths than either 

strategy might have done on its own. While other training programs have incorporated retention 

and advancement services, the WorkAdvance study provides the first rigorous test of this combination of services. 

The WorkAdvance program operations and evaluation have been conducted under the 

auspices of the Social Innovation Fund (SIF). Administered by the Corporation for National and 

Community Service, SIF is a public-private partnership designed to identify and expand effec-

 6

A quasi-experimental evaluation of New York City’s Workforce Transportation Career Center, which 

provides sector-focused job placement, training, and support to low-income individuals interested in new or 

higher-wage positions in the transportation sector, showed that participants were three times more likely to be 

placed in jobs than the comparison group and earned $1.90 more per hour (Henderson, MacAllum, and Karakus, 2010).

7

Because the study did not follow the research sample beyond two years, it is not clear how much longer 

these positive results would have lasted and the extent to which workers would have advanced along career 

pathways. 8

Woolsey and Groves (2013); Conway, Blair, Dawson, and Dworak-Munoz (2007). 9

Hamilton and Scrivener (2012a).
3

tive solutions to critical social challenges. WorkAdvance has been part of the New York City 

CEO SIF project, which is led by CEO and the Mayor’s Fund to Advance New York City in 

collaboration with MDRC. MDRC is leading the WorkAdvance evaluation; has provided technical assistance to the local providers; and, jointly with CEO, has monitored providers’ operations. Funding for the WorkAdvance program and evaluation came from the SIF and a broad 

array of local partners that matched the SIF funding.10

This report provides policymakers, practitioners, and funders with important information on the feasibility, two-year effects (or “impacts”), and cost of expanding and replicating 

a model of this type for low-income populations in various local contexts. WorkAdvance has 

been delivered by organizations with varying missions and experience. Notably, two of the four 

providers had no previous experience operating sector programs, and although some had job 

retention services in place, none of them had ever incorporated substantial postemployment advancement services into their programs. The WorkAdvance providers operated in three different 

locations (New York City, northeast Ohio, and Tulsa, Oklahoma) and across multiple sectors 

(transportation, information technology, environmental remediation, health care, and manufacturing). Consistent with the SIF goal of testing programs on a larger scale, the local WorkAdvance providers recruited twice the number of people that they could serve (since half the people they recruited were assigned to the control group), and the number enrolled in the program 

group was twice the number enrolled in the SEIS. The WorkAdvance providers began operations during an especially poor economy, when low-skilled workers were experiencing extended periods of unemployment or underemployment. In contrast, the programs that participated in 

the SEIS operated primarily in a stronger economy.

It is important to note that two years of follow-up is probably not long enough to measure the full benefits of the WorkAdvance programs. It may take time for the benefits of a sectorprogram placement to translate into higher wages and career advancement. Current plans call 

for the collection of additional follow-up so that WorkAdvance’s economic outcomes can be 

further measured at three and five years after each participant entered the program. Assuming 

these data can be collected, this report should be viewed as showing the interim rather than final 

impacts of the WorkAdvance model. 

The rest of this chapter provides more detail on the WorkAdvance model, the experimental sites that were chosen, the research design, the economic conditions, and the target populations. 

 10The Mayor’s Fund to Advance New York City and CEO led fundraising efforts for the SIF initiative, 

identifying a diverse set of national and local funders to match the federal funds for support with program operations, evaluation, and oversight activities. The WorkAdvance providers worked closely with CEO to raise 

matching funds to support program operations and to meet SIF match requirements.
4

The WorkAdvance Model

A fundamental focus on employer input and long-term career advancement is reflected 

in each of the five WorkAdvance program elements shown in Figure 1.1 (including screening, 

which precedes enrollment). WorkAdvance is a workforce development model designed to help 

unemployed and low-wage working adults increase their employment and earnings by finding 

good quality jobs in selected sectors that have room for advancement within established career 

pathways. The essential theory behind WorkAdvance is that strategic upgrades in human capital

— that is, education and employment-related skills and experience — will lead to advancement 

in the labor market, but only if training and job preparation are directly aligned with specific job 

openings. This theoretical pathway applies to most program participants in the evaluation. For 

those who did not attend training (because they already had the requisite skills and experience) 

the theory is that strong connections to the labor market will help facilitate access to jobs in the 

selected industry. The theory also assumes that well-regarded providers with strong labor market connections will have a spillover effect: They will help individuals who would otherwise not 

be as well represented in the labor market. For example, sector programs can advocate for individuals who have trouble entering the labor market because of discrimination by race or gender. 

A critical part of the sectoral vision is that employers will come to trust organizations that prepare trainees, and these organizations will become labor market intermediaries that over time 

become valuable to employers as a source of recruitment for qualified job applicants. Eventually such providers also might be able to advocate on behalf of employees for better wages and 

working conditions. 

As displayed in Figure 1.1, the program includes the following key components:

1. Intensive screening of program applicants before enrollment — a practice 

not always common in employment programs offered to low-income individuals — is intended to ensure that program providers select participants 

who can take advantage of the skills training for the sector and occupations. 

The expectation is that screening closely aligns with employer requirements 

for the targeted jobs. This was identified as one of the key elements of success in the SEIS. 

2. Sector-appropriate preemployment and career readiness services comprise an orientation to the sector, career readiness training, individualized career coaching, and limited supportive services to sustain engagement and assist participants to complete their training and find employment.
5
6 

3. Sector-specific occupational skills training is intended to impart skills and 

lead to credentials that will substantially enhance workers’ employment opportunities. Providers offer training tailored to current job openings in specific sectors and occupations. In addition, providers are to adapt these offerings 

to changes in available jobs. This step did not apply to all enrollees in this 

evaluation. Early in the sample enrollment period approximately half the enrollees at two of the four WorkAdvance providers were in a placement-first 

track in which they skipped the training phase (discussed further below). 

4. Sector-specific job development and placement services are intended to 

facilitate entry into positions for which the participants have been trained and 

for which there are thought to be genuine opportunities for continued skill 

development and career advancement. The providers’ job developers are expected to maintain strong relationships with employers who hire individuals 

with the kinds of skills the program has imparted. 

5. Postemployment retention and advancement services are meant to assist 

participants beyond the placement stage. Providers are expected to maintain 

close contact with workers and employers to assess performance, offer 

coaching to address any complicating life situations that might arise for 

workers, help identify next-step job opportunities and skills training to enable

participants to move up career ladders over time, and assist with rapid 

reemployment if workers lose their jobs. By working closely with employers 

and understanding the career trajectories in their workplaces, providers 

should be able to help guide a participant’s career path.

The Two-Track Model

Although all the WorkAdvance providers eventually emphasized training before job 

placement, two of them (in northeast Ohio and Tulsa) implemented the program model with 

two separate tracks. This program design was suggested in the site selection process, based on 

previous workforce experience in New York City. One track emphasized gaining skills first 

through training (similar to most other sector-based programs), and the other sought to place 

people into jobs first; the goal for the two dual-track providers was to designate at least 50 percent of their participants to go into training first, while the remainder could be placed first. The 

placement-first track was intended to be a less expensive but still effective route to advancement 

by enabling enrollees to gain experience and sector-specific skills (such as through on-the-job 

training) without participating in formal training first. Another rationale for making placements 

right away was that it helped the providers offer and deliver a more immediate service to employers while participants in the other track were still going through training and provider staff
7 

members were building relationships with employers. However, at MDRC’s urging, both the 

providers eventually shifted mostly to the training-first approach, since providers were not going to be able to reach the goal of enrolling 50 percent of participants in training if so many 

people continued to be placed first. Additionally, preliminary evidence suggested that placement first too often resulted in participants entering low-wage jobs that, in practice, did not lead 

to on-the-job acquisition of skills.

The WorkAdvance Model in the Current Policy Context

Why is a program like WorkAdvance needed? It is well documented that low-wage 

workers often experience unstable employment and modest wage increases.11 WorkAdvance is 

designed to confront a number of obstacles that impede many low-wage workers (and programs 

serving them) from achieving labor market progression. These obstacles, and how WorkAdvance attempts to address them, are summarized in Table 1.1.

In theory, a program like WorkAdvance seems to have all the right elements for a successful employment program — so why is it even necessary to rigorously test it? In fact, there 

are several reasons why WorkAdvance might not be effective. First, participants, many of 

whom have limited resources, might not be able to complete the training. In addition, the training might not meet employer needs, or the employers might not take the training as a serious 

signal of preparedness. There have been several instances of training programs that did not lead 

to advancement.12 There are also some risks in the sectoral strategy. By focusing so much on a 

specialized segment of the labor market, providers and participants are particularly exposed 

when demand shifts. Finally, participants may get jobs in the sector, but low-wage workers often face obstacles to keeping jobs. Life complications may interfere with them performing well 

on the job, which can lead to job loss. Therefore, while it seems obvious that employers need 

trained employees to work at jobs that require specialized skills, it is not obvious that programs 

like WorkAdvance will succeed in producing the right kind of labor supply for those jobs or that 

such programs will be an effective route to economic mobility for low-income people. 

The study is timely. There had been a number of sectoral programs even before the 

SEIS study,13 but the promising findings of that study led to even greater interest in sectoral initiatives,14 including a major focus on such programs as part of the recent Workforce Innovation

 11Mishel, Bernstein, and Allegretto (2007); Schochet and Rangarajan (2004); Acs and Loprest (2004). 12For one example, see Hendra et al. (2010). 

13Project Quest in San Antonio (Osterman and Lautsch, 1996) and Focus: HOPE in Detroit (Hollenbeck, 

2007) are notable examples. 14In a survey, the National Network of Sector Partners found that 47 percent of sectoral initiatives profiled 

were less than 5 years old (Mangat, 2010). Prominent initiatives founded between about 2005 and 2010 include,
8 

The WorkAdvance Study

Table 1.1 

Labor Market Obstacles and WorkAdvance Responses

Obstacle WorkAdvance Element That Addresses Obstacle

Human capital deficitsa • Occupational skills training

• Career readiness training

• Placement in jobs with skill acquisition potential

Information and social capital deficitsb

 

• Brokering with employers and intermediation 

role in job development/placement 

• Preemployment services to acclimate participant to sector 

Costs of advancement • Free occupational skills training 

• Modest financial assistance to help participants engage in program

Lack of career opportunities • Advancement services/coaching for career 

navigation

Employer hesitation to hire • Effective screening 

• Employer involvement in program design and 

adaptation 

• Career readiness services

Misalignment of employment programs with 

labor market 

• Focus of program staff on specific sectors 

and the involvement of employers

NOTES: aHuman capital is generally defined as the skills, knowledge, and experience possessed

by an individual or population, viewed in terms of their value or cost to an organization or 

country.

 b

Social capital is generally defined as the networks of relationships among people who live and 

work in a particular society, enabling that society to function effectively.

 

among many others, Skills2Compete Maryland, Pueblo Manufacturing Consortium in Colorado, and the South 

Central Pennsylvania Food Manufacturers’ Training Consortium.
9 

and Opportunity Act (WIOA). WIOA was passed with bipartisan support in July 2014, reauthorizing the Workforce Investment Act (WIA) for the years 2015 to 2020. The bill specifically 

requires states to implement industry or sector partnerships and career pathways.15 The WorkAdvance findings, therefore, have direct relevance to WIOA, and the study’s lessons concerning

postemployment and advancement services will have important implications as the new legislation is implemented. In addition, WIOA includes a provision on service priorities that will ensure that workforce efforts focus on a low-income population similar to the WorkAdvance sample. Thus it is critical to understand these types of programs better, to confirm that they are effective, to determine how they perform in different conditions and on a larger scale, and to 

measure how much they cost to operate. 

The design of WorkAdvance took into account several considerations, as discussed 

below.

• WorkAdvance was specifically designed to be more than simply a job 

placement program because substantial evidence has accumulated and 

established that initial job placement alone is not sufficient to foster longterm advancement. 

The WorkAdvance model was built on several generations of rigorous random assignment evaluations of welfare-to-work and workforce programs that have sought to improve the 

employment outcomes of low-income people. The first strand of evidence comes from evaluations of programs designed to help people make the transition from welfare to work. 

The welfare-to-work studies yielded substantial knowledge about how to help lowincome individuals prepare for and find jobs. The evaluations showed how numerous strategies, 

particularly those focused on getting participants into the labor market quickly, were quite effective in increasing employment and reducing welfare.16 However, many participants in the programs that successfully boosted employment over a five-year follow-up period still ended up in 

unstable, low-paying jobs. Thus, the research suggested a need to focus on ways to effectively 

increase employment stability and wage progression. By the mid- to late 1990s, the federal government and states focused squarely on the problem of employment retention and advancement. 

An initial multisite randomized controlled trial, the Post-Employment Services Demonstration 

(PESD), operated in the mid-1990s. It examined the effectiveness of offering certain services, 

such as counseling and support, frequent and flexible payments for work-related expenses, and 

other services, to newly employed welfare recipients.17 The programs studied in the PESD, 

 15Workforce Innovation and Opportunity Act, Pub. L. No. 113-128, 128 Stat. 1425. 16Gueron and Rolston (2013). 17Rangarajan and Novak (1999).
10

however, had little effect on employment or earnings. This led to another, more ambitious evaluation called the Employment Retention and Advancement project (ERA). 

ERA tried to go beyond the PESD models and tested a wide variety of strategies, which 

mostly focused on coaching, counseling, and case management. The models did not generally 

focus on skills training. The results of the ERA trials highlighted the difficulty of achieving upward mobility through simple strategic placement of people into jobs and generic on-the-job 

coaching alone. Of the 12 relevant programs studied in the ERA project,18 only three were 

found to be effective at increasing earnings for participants. The nine unsuccessful programs 

offered guidance and advice after people found jobs (that is, postemployment), but little 

else.19 These findings suggested that more needed to be done than simply helping participants 

navigate the labor market better.20 One implication of the results was a refocus on skills training 

as a route to advancement, given the ERA evidence that counseling and coaching (as a supplement to work-first approaches) were insufficient to lead to advancement for most participants. 

As results from the PESD and ERA evaluations unfolded, some programs began to incorporate more job training, acknowledging that some kind of vocational skill-building was 

needed in order to increase wages for low-wage workers. One initiative that attempted this was 

studied as part of the United Kingdom’s Employment Retention and Advancement project (UK 

ERA). The UK ERA results supported a long-standing lesson in the field of employment and 

training: Training does not work if it is not aligned with employer demand. The UK ERA program boosted training engagement, but labor market benefits attributable to training were not 

found, suggesting that there was a mismatch between the training undertaken and the labor 

market demand for individuals with that training.21 The leading explanation for this result related to the program staff’s capacity. The UK ERA advisory staff members functioned as employment “generalists” — they offered participants general advice and guidance on adapting to 

work, encouraged them to consider seeking full-time work, helped them address issues of balancing work and family life, advised them on seeking promotions and finding better jobs, and 

urged them to enroll in training courses in whatever areas interested them. But staff members 

did not have in-depth knowledge of particular occupations or industries, or expertise on the ca-

 18The 12 programs exclude those targeted to “harder to employ” enrollees, such as individuals with substance abuse issues. 19The programs were built upon a variety of hypotheses about what might be advantageous. For example,

maintaining small caseloads; offering services at individuals’ workplaces; providing services through a collaboration among welfare, WIA, and community college staff members; and continuing counseling relationships 

from pre- to post-job placement. None of these features produced sustained positive impacts on earnings, in 

and of themselves. (While the counseling and coaching produced a low yield on their own, researchers concluded that it was possible that these services could be very valuable when combined with other, more concrete 

services.) 20Hendra et al. (2010). 21Hendra et al. (2011).
11

reer ladders and training requirements for jobs in those areas. Nor did they steer participants 

assertively toward particular occupations known to offer real advancement opportunities. They 

were not positioned to connect participants who had trained in particular occupational areas 

with relevant employers who were hiring people with the new skills those participants had acquired. It is likely that these limitations undermined the benefits of the extra participation in 

training that UK ERA produced. The findings point toward the value of providing career advice 

that is sector specific and more narrowly focused on opportunities available in the local labor 

market.

A subsequent test of an approach with a more deliberate demand-driven focus occurred 

in the late 2000s, in the Work Advancement and Support Center (WASC) demonstration. The 

programs examined in WASC aimed to increase the incomes of low-wage workers by stabilizing employment, building skills, increasing earnings, and easing access to work supports, such 

as child care subsidies and the Earned Income Tax Credit. One of the central hypotheses of 

WASC was that providing training through WIA One-Stop Career Centers would result in better alignment between training and work. Two of the WASC programs increased participation 

in education and training and also increased earnings in the third follow-up year.22 In one program, these effects faded somewhat in the subsequent follow-up year; in the other, longer-term 

follow-up was not available. In both programs, the level of staff capacity to provide employerinformed advice was lower than anticipated. Still, because funding for training was mainly 

through WIA, there were conditions in place to try to ensure that training was in high-demand 

fields. In particular, in one of the programs, many of the training vouchers were used to pay for 

training in the rapidly growing health care field. These results suggested the promise of focusing 

training in high-demand areas, but doing so requires program staff members with deep 

knowledge of labor market niches: a central aspect of sector-based programs.

• WorkAdvance was designed as a sectoral job training program that

aligned skills training with job opportunities.

One of the clearest findings from past research is that training works only when it corresponds to the labor needs of employers.23 The sectoral approach is an effort to improve the 

alignment between training and employment opportunities. Beginning in the late 1980s, community-based organizations across the United States pioneered workforce development programs using a “sector strategies” approach.24

While early work by the Aspen Institute and CEO’s Workforce Innovations study of 

sector-focused programs found some encouraging evidence on the benefits of the sectoral ap-

 22Miller, van Dok, Tessler, and Pennington (2012). 23Melendez (1996).

24Mangat (2007).
12

proach,25 these studies were not as scientifically rigorous as randomized controlled trials; rather, 

the most powerful evidence to date comes from the aforementioned SEIS.

26 That study, which 

was a randomized controlled trial, found that sector-focused training programs for low-income 

workers and job seekers increased their earnings, employment, job stability, and access to benefits over a two-year period. Participants’ earnings over two years were $4,500 (or 18 percent) 

higher than earnings for the control group.27 Earnings in the year after training were 29 percent 

higher than the control group average. Impacts from job training programs are usually much 

more modest, so these findings led to enthusiasm about sectoral programs. 

Key elements of the SEIS programs included the providers’ experience with sectoral 

programs; their strong relationships with local employers (“brokering”); provision of job readiness training in addition to occupational skills training; a stringent screening and intake process; 

and provision of individually tailored services. Although they aimed to place workers in “good” 

jobs — jobs that are higher-paying and more stable — the programs did not have an explicit 

advancement component. 

Building on the SEIS findings, it is also important to understand how well the sectoral 

strategy works with a wider range of providers. The three programs in the SEIS were carefully 

selected, and arguably represented unusually strong implementations of the sectoral approach. 

Notably, and in contrast with WorkAdvance, as this report explains, all three SEIS providers 

had been operating the program that was evaluated for at least three years before the study. Unlike in WorkAdvance, the providers were not asked to run a program model that had similar, 

specific components; rather, they were asked only to continue to run their programs “as is.”

Therefore, one of the key contributions of WorkAdvance is to help shed light on whether a 

broader range of providers, less experienced with this particular strategy, can produce impacts 

similar to those measured in the SEIS.

Finally, it is important to recognize that WorkAdvance sought to go beyond traditional 

sector programs. Traditional sector programs are built on the assumption that if providers find a 

sector in which there is room for advancement, workers will advance on their own given the 

right start. The explicit focus on advancement assistance as a means to upward mobility was an 

important enhancement introduced in the WorkAdvance model. Because the providers may not 

have been able to place participants directly into “middle-skill” jobs in all cases, the advancement coaching component of the model was hypothesized to help participants make the transition from an entry-level job to a middle-skill job after they gained some additional experience.

 25Zandniapour and Conway (2002); Henderson, MacAllum, and Karakus (2010). 26Maguire et al. (2010).

27Maguire et al. (2010).
13

Selection of WorkAdvance Program Providers

Under the auspices of the federal SIF grant, the WorkAdvance managing consortium

set out to competitively select local providers in three geographic areas — New York City; 

northeast Ohio (Greater Cleveland and Youngstown); and Tulsa, Oklahoma — rather than to 

conduct a broader national search.28 A principal consideration in selecting the geographic areas 

was demonstrated local interest, as represented by a commitment to help raise the match funding needed to satisfy the SIF requirements. A second important factor was that there be a diversity of local economies and industry sectors, so that the WorkAdvance demonstration could 

draw on a range of local conditions to inform potential replication and expansion of similar programs.

The four providers that were selected for WorkAdvance had a range of experiences and 

backgrounds. Ultimately, this benefited the study, as it provided the opportunity to learn, among 

other things, whether an array of providers — including some that were less mature in their delivery of sectoral programs than those included in the SEIS or that had no sectoral experience at 

all — could successfully implement WorkAdvance. 

The WorkAdvance providers were selected by CEO and the Mayor’s Fund to Advance 

New York City through a competitive process conducted within each of the identified geographic areas, with input from MDRC and local stakeholders from each region, including representatives of government and philanthropy. A primary factor in selection decisions was whether 

a provider could demonstrate that it was currently, or had the capability to be, firmly grounded 

in a targeted sector; this included in-depth knowledge of and strong relationships with employers who provided letters of support. Applicants had to demonstrate current or potential capacity

to operate at the intended scale, to carry out an advancement-focused approach, and to work 

with a range of lower-income individuals — rather than only those who would be easiest to 

place in jobs. Additional selection criteria included overall organizational capabilities (including 

appropriate fiscal and data management capacity and the ability to comply with federal funding 

requirements), clear commitment to the program model, and a willingness and ability to participate in a random assignment study and to help raise matching local funds. All the selected providers demonstrated the commitment of the agency’s leadership to the requirements of the 

WorkAdvance demonstration.

 28The WorkAdvance managing consortium members are as follows: The Mayor’s Fund to Advance New 

York City was the grantee of the SIF; the New York City Center for Economic Opportunity (CEO) managed 

the contracts with the providers (called “subgrantees”) and was accountable to the funders for program implementation and progress, and for operating a Learning Network for the project; and MDRC provided technical 

assistance to the providers in the implementation of WorkAdvance, monitored their performance (along with 

CEO), and led the evaluation.
14

CEO and the Mayor’s Fund did not specify the targeted sectors in each geographic area; 

instead, providers proposed and had to justify the sector and range of occupations, based on 

their experience, local labor market information, and the advancement potential of the targeted 

jobs. 

The four organizations selected to operate WorkAdvance are as follows: 

• Per Scholas in the Bronx, New York, which focused on the information 

technology (IT) sector

• St. Nicks Alliance in Brooklyn, New York, which focused on environmental 

remediation and related occupations

• Madison Strategies Group in Tulsa, Oklahoma, which focused on the transportation and manufacturing sectors

• Towards Employment in northeast Ohio, which focused on the health care 

and manufacturing sectors 

Table 1.2 provides some background on the four WorkAdvance providers. Chapter 2 provides 

more details, particularly with regard to the organizations’ levels of experience in operating sectoral training programs. All four programs continue to provide services similar to those offered 

under WorkAdvance. 

The WorkAdvance Evaluation Design

Random Assignment

The WorkAdvance programs were studied using a random assignment design, a methodology that allows practitioners and policymakers to have a high degree of confidence in the 

results. Random assignment designs are known as the “gold standard” because the randomization process removes potential sources of bias, creating two groups that are expected to be 

equivalent at baseline. The WorkAdvance study also has high external validity — or the extent 

to which the results can be expected to apply to other settings — because the intervention was 

tested in four diverse sites and across several sectors. For these reasons this study provides the 

highest level of evidence, according to SIF criteria.29

From June 2011 through June 2013, individuals who met the WorkAdvance and sectorspecific eligibility criteria were assigned at random to the WorkAdvance group or to the control

 29The SIF defines an example of the highest level of evidence as a large, well-designed and wellimplemented, randomized controlled multisite trial that supports the effectiveness of the practice, strategy, or 

program. This study meets those criteria.
The WorkAdvance Study

Table 1.2

Institutional Structure of Providers

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Location Bronx, New York Brooklyn, New York Tulsa, Oklahoma Cleveland, Ohio

Year founded 1995 1975 2011 1976

Sector(s) Information technology Environmental remediation Transportation and manufacturing Health care and manufacturing

Key 

organizat

character

ional 

istics

Provides in-house information 

technology training and job 

placement services. 

Large social service agency 

offering housing, health care, 

workforce and economic 

development, and youth and 

education services. WorkAdvance 

operates within a broader 

workforce development unit at St. 

Nicks Alliance. 

New workforce nonprofit 

operating only WorkAdvance, 

built as the offshoot of a for-profit 

workforce agency operating in 

New York City. 

Employment-focused organization 

offering career readiness training, 

supportive services, access to 

occupational skills training, and 

employer partnerships.

Approach Training-first for all enrollees Training-first for all enrollees Mixed enrollment to training- and 

placement-first

Mixed enrollment to training- and 

placement-first

SOURCES: Documentation provided by providers.

15
16

group.30 Members of the WorkAdvance group were offered WorkAdvance services, while those 

in the control group were not eligible for WorkAdvance services but were eligible for other services available in the community.31 The first six months of random assignment consisted of a 

pilot phase, designed to provide the organizations with time to work out implementation issues 

and to learn to perform research procedures (such as collecting baseline data, explaining informed consent, and conducting random assignment). A pilot assessment was conducted to determine how well the program was implemented during the pilot phase. The assessment concluded that the program was sufficiently well implemented to enable the pilot sample to be incorporated in the research sample.

Both the WorkAdvance group and the control group were tracked over time. The random assignment process ensured that, when individuals entered the study, there were no systematic differences in sample members’ characteristics, measured or unmeasured, between the 

two research groups. Thus, any differences between them that emerge after random assignment 

can be attributed to the location’s WorkAdvance program.

The Role of Technical Assistance 

WorkAdvance was a multicomponent program that required all providers to go beyond 

their normal operations. All providers received extensive technical assistance during the study 

period, to ensure that research procedures were being followed and to ensure that the providers 

were delivering the strongest program possible, according to the way it was designed. MDRC 

and several consultants provided technical assistance.

Components of the Study and Data Sources

This section lays out the components and the key questions addressed in each analysis 

and the key data sources. Table 1.3 provides more detail on the sources. 

The implementation analysis assesses how each provider put the key components of the 

WorkAdvance model into operation. The analysis also examines what it takes to operate the 

 30Random assignment began at Per Scholas first. As Figure 1.2 shows, random assignment began later in 

2011 at the other three sites. The programs continued to provide sectoral training and employment services 

after the random assignment period concluded. 31MDRC conducted random assignment using a secure, online web-based random assignment system that 

has been used in many MDRC studies. The random assignment algorithm has been extensively checked and 

random assignment was monitored by analyzing the percentage of assignments to both groups. Extensive statistical tests were conducted to ensure that random assignment produced two similar groups; that is, that baseline characteristics did not vary systematically across the research groups. These procedures were the same at 

each site and are described in further detail in Appendix A. See Appendix Table C.2 for more information on 

intake and the point of random assignment.
17

The WorkAdvance Study

Table 1.3

Data Sources

Data Source Description

Baseline data Baseline data were collected by MDRC at the time of random assignment using an online 

module. Data include information on demographic characteristics, prior employment, and 

various other characteristics. This information is used to describe the study samples, to create 

subgroups, and to refine statistical estimates. 

Field research Field research is the main data source for the analysis of program features and implementation. 

The field research includes MDRC’s observations of and interviews conducted with the 

providers’ staff members, observations of participants, review of program materials (for 

example, curricula for career readiness), and operational experience. Further information 

collected from five focus groups of between 9 and 12 WorkAdvance group members each and 

individual interviews conducted with 20 WorkAdvance group members at two points in their 

program tenure is used to provide examples of how WorkAdvance was implemented across 

the providers. Two special analyses were conducted: a “funnel analysis,” which provides data 

on recruitment and the flow of applicants through the study intake process, and an analysis that 

documented the frequency and content of advancement coaching sessions. (These analyses are 

described in more detail in Appendix A and Chapter 2, respectively.)

Program Program tracking data are used as a source of information for WorkAdvance group activities, 

tracking data supportive services receipt, education and training funds (where applicable), operating costs 

(administrative and staff), and job placement activities. These data are used to estimate costs of the 

program and to describe the program services (as a supplement to the field research data). These data 

were provided over the course of the study by the providers using a template created by MDRC. 

These data were collected only for the WorkAdvance group.

Administrative Effects on employment and earnings are computed using unemployment insurance (UI) wage 

records records data provided by the state employment agencies during the course of the study. It is a estimated that UI data cover approximately 90 percent of all jobs. All unreported employment is 

missing. In addition, UI benefits data, where available, are used to measure whether increases in 

employment via WorkAdvance translate into reductions in the take-up of UI benefits. The same data 

were collected for both research groups.

Survey data Information about sample members’ experiences with program operations and services, as 

well as their employment outcomes, was collected through a survey administered to 

WorkAdvance and control group members approximately 18 to 24 months after their random 

b assignment. The survey achieved a response rate of 81 percent. The survey focused on 

contact, message and help received (for example, for job placement), participation in a job 

search and education and training, and characteristics of employment, such as the number of 

months employed, hours worked, hourly and monthly wages, and industry and occupation. 

The survey also captured various secondary outcomes.

NOTES: aKornfeld and Bloom (1999). A WorkAdvance-specific estimate of UI coverage is presented in 

Appendix A.

b A survey response analysis is shown in Appendix A. While there are some minor differences between survey 

respondents and nonrespondents, there are no major concerns about the representativeness of the survey data.
18

WorkAdvance model well, what organizational and other factors influenced operations, and 

how the programs evolved over time. The analysis relies on data collected through a combination of site visits, interviews, focus groups, program records, and study-specific site data collection efforts.

The participation analysis measures the differences in services received by the WorkAdvance group and the control group, using survey data. Services measured include not only

those provided by WorkAdvance, but also other services generally available in the community. 

By establishing the service contrast that WorkAdvance produced, the analysis provides critical 

context for interpreting the results from the economic impact analysis.

The cost analysis estimates the average per person cost of operating the WorkAdvance 

program. It first estimates the gross cost per person by looking at the full costs of services delivered to the WorkAdvance group divided by the number of WorkAdvance group members. It 

then estimates WorkAdvance’s net cost per person by subtracting the average costs of the services that the control group received from the average gross cost of providing services to the 

WorkAdvance group. 

The economic impact analysis focuses on measuring the extent to which WorkAdvance 

affected labor market outcomes, such as employment, employment retention, earnings, wage 

rates, hours worked, employer-provided benefits, and advancement, beyond the levels attained 

by control group members over the follow-up period. These outcomes are measured by Year 2 

survey data and (in the case of earnings and employment) unemployment insurance (UI) wage 

data. The analysis also examines the effects the program had on secondary measures such as life 

satisfaction and material hardship.

Key Samples and Follow-Up Periods

Given the differences in starting points and organizational emphases across the four 

providers involved in WorkAdvance, a decision was made early in the project to give priority to

analyzing impacts by site.32 The main exception is for the subgroup analysis, which focuses on 

the pooled sample. This exception is necessary because sample sizes for subgroups are not large 

enough at the site level to enable precise estimates.

 32In the Subgrantee Evaluation Plan prepared for WorkAdvance as part of the SIF, the original plan was to 

conduct the analysis by sector. Because most sites focused on a unique sector, the only difference between 

analyzing impacts at the site level and at the sector level is in Ohio. Because one of the original Ohio programs 

(Compass) was excluded (see Tessler et al., 2014), there was insufficient sample size to analyze the Ohio sectors separately. Appendix A discusses this in more detail.
19

Figure 1.2 shows the sample size for each site. Sample sizes range from 479 at St. 

Nicks Alliance up to 698 at Towards Employment. The pooled sample size is 2,564.33 Survey 

sample sizes are roughly 80 percent of these totals. The overall study sample is more than twice 

as large as the sample enrolled in the SEIS, and the average WorkAdvance site sample is more 

than twice as large as the average SEIS site sample. Therefore, in addition to enlisting a more 

diverse set of providers (in terms of their experience operating sectoral training programs), the 

WorkAdvance study also built on the SEIS by providing estimates of how these programs 

would perform on a larger scale. 

The full sample has approximately two years of common follow-up from both administrative records and the follow-up survey. The average survey respondent was interviewed in 

Month 22.34 Two and one-quarter years of UI wage data are available for the full sample. 

It is well established in the job training literature that at least two years of follow-up is 

needed to measure the impacts of training programs, because it takes time for participants to 

complete their training, find a job, and advance in the job. For example, in the SEIS, positive 

impacts did not emerge until late in Year 1; impacts were negative early in Year 1 due to the 

opportunity cost of time spent in training instead of work. In other studies, impacts have taken 

as long as three years to emerge. Therefore, the amount of follow-up analyzed in this report may 

not be enough to capture WorkAdvance’s long-term impacts (particularly the impacts of advancement services). As previously noted, plans are in place to collect and analyze longer-term 

follow-up data for the study sample. 

Characteristics of the WorkAdvance Sample

WorkAdvance targeted unemployed and low-wage working adults with a monthly family income below 200 percent of the federal poverty level. As a proxy for the targeted low-wage

 33The target sample size was originally 3,850. Recruiting a sample that large within the confines of the SIF 

grant period proved infeasible. A statistical power analysis, discussed further in Appendix A, verified that the 

study is more than adequately powered at the sample size that was achieved. All but 48 individuals recruited 

for the study are included in the research sample. Attrition from the study was minimal: A total of 20 individuals (0.8 percent) withdrew from the study (10 from the Per Scholas site, 3 from the St. Nicks Alliance site, 2 

from the Madison Strategies Group site, and 5 from the Towards Employment site). Two of the withdrawals 

were WorkAdvance group members, and 18 were control group members. Apart from these withdrawals, participants were excluded from the impact analysis for a few reasons: 22 were being served under another funding stream, 1 was missing a signed consent form, and 5 were enrolled before the official start of random assignment. Of the 2,564 individuals in the research sample, about half (1,293) were assigned to the program 

group and were eligible to receive WorkAdvance program services. The vast majority (95 percent) of program 

group members received at least one program service within the first 18 months of follow-up. (See Chapter 3 

for more details on service take-up.) 34According to the original Subgrantee Evaluation Plan, the survey was to be launched 18 months after 

baseline, but it was moved to the two-year point to extend the follow-up period.
The WorkAdvance Study

Figure 1.2

Random Assignment Period and Report Sample Sizes, by Site, Analysis Sample, and Research Group

Random Assignment Period

2011 2012 2013 Sample Size

Analysis sample J J A S O N D J F M A M J J A S O N D J F M A M J WA Group C Group Total

Per Scholas

Report sample 349 341 690

Year 2 Survey respondent sample 287 265 552

St. Nicks Alliance

Report sample 242 237 479

Year 2 Survey respondent sample 205 179 384

Madison Strategies Group

Report sample 353 344 697

Year 2 Survey respondent sample 297 263 560

Towards Employment

Report sample 349 349 698

Year 2 Survey respondent sample 286 276 562

SOURCE: MDRC calculations from WorkAdvance baseline information form.

NOTE: WA group = WorkAdvance (program) group; C group = control group.

20
21

worker population, adults who were working at the time that they entered the study were also 

required to be earning less than $15 per hour.35 

Table 1.4 presents selected characteristics at baseline of the WorkAdvance research 

sample by provider and overall.36 The variation in baseline characteristics that is seen across 

providers is likely to reflect provider-specific eligibility criteria, the sectors selected, and the city 

in which the provider is located. Various tests for baseline equivalence between the WorkAdvance group and the control group are presented and discussed in Appendix A. As expected (by 

virtue of the random assignment research design) there are no meaningful or systematic differences between the research groups. 

The average WorkAdvance sample member is 34 years old, black/African-American, 

and single, with some variation seen across the sectors. Notably, the Per Scholas sample members are younger and much less likely to have children. The majority of WorkAdvance sample 

members are male, with the exception of the health care sector at Towards Employment (not 

shown separately), in which more than 92 percent of sample members are female. Almost all 

sample members had at least a high school diploma or General Educational Development 

(GED) certificate, and over half the sample had at least some college education — a much higher rate than seen in the SEIS. Individuals at the St. Nicks Alliance site had lower levels of education than at the other sites.

Early in the follow-up period in particular, the sample composition seemed heavily affected by the Great Recession and the accompanying labor market displacement that took place 

in 2008-2009. These events pushed many previously stably employed individuals into unemployment and led to declines in the economic circumstances of many formerly “middle-class” 

individuals. Almost all sample members had previous work experience, but only one in five 

were working at the time that they entered the study. In addition, more than 36 percent of sample members had been unemployed for at least seven months before study entry. Overall, 15 

percent of sample members had previous work experience in their targeted industry; this rate 

varied from 2 percent at St. Nicks Alliance to 31 percent at Towards Employment, which focused partly on health care, an industry which employs a large number of workers and which 

has low barriers to entry-level jobs, such as home health aides and nursing assistants. Despite 

fairly high levels of education and work experience, many sample members still faced substantial barriers to employment, and many were receiving public benefits. One-quarter of the overall 

 35In addition to study eligibility criteria, some of the providers also imposed eligibility criteria related to a 

given sector and the occupations for which their training programs prepared participants. See Appendix Table 

C.2 for details. 36Additional sample characteristics are presented in Table B.1.
22

 The WorkAdvance Study 

Table 1.4

Selected Characteristics of Research Sample Members at Baseline and Comparison

to National Low-Wage Workers, Cross-Site

WA United

Characteristic PS SNA MSG TE Overall States

WorkAdvance group (%) 50.6 50.5 50.6 50.0 50.4

Demographic characteristics

Female (%) 13.2 14.5 15.9 58.6 26.6 57.0

Average age (years) 31 35 35 35 34 38

Age (%)

18-24 years 31.4 16.3 22.4 22.9 23.8

25-34 years 39.6 37.2 33.9 29.2 34.8

35-44 years 16.1 25.7 20.4 21.2 20.4

45-59 years 12.8 19.8 22.2 24.9 20.0

60 years or more 0.1 1.0 1.1 1.7 1.0

Race/ethnicity (%)

Latino/Hispanic 36.0 22.8 6.1 4.6 16.8 27.6

White 5.2 6.9 39.2 18.4 18.4 45.7

Black/African-American 44.5 62.7 28.4 70.8 50.6 18.9 a Other race 14.3 7.5 26.4 6.2 14.1 7.8

Born in United States (%) 71.9 76.9 95.4 97.9 86.3 75.2

Family status (%)

Marital status

Single, never married 76.7 70.9 50.6 72.7 67.4 46.8

Married and living with spouse 11.9 14.3 22.8 9.6 14.7 31.0

Married but living apart from spouse 6.1 4.0 6.1 6.2 5.7 1.9

Legally separated, divorced, or widowed 5.4 10.9 20.6 11.5 12.2 20.3

Parent of one or more children 26.2 45.3 51.8 51.8 43.7 44.3

Education level (%)

Highest level of education attainment

Less than GED certificate or high school diploma 0.1 11.9 6.0 5.9 5.5 27.3

GED certificate/high school diploma 37.1 44.5 35.7 37.1 38.1 35.9

Some college 32.5 26.5 48.1 47.0 39.5 20.3

Associate's degree/2-year college 9.9 7.5 5.7 4.9 6.9 6.2

4-year college degree or more 20.4 9.6 4.4 5.2 9.9 10.3

Already has a license/certificate in targeted industry 3.6 1.9 13.5 24.9 11.8

Employment status

Ever employed (%) 96.4 97.9 99.3 97.3 97.7

(continued)
23

Table 1.4 (continued)

Characteristic PS SNA MSG TE

WA

Overall

United

States

Number of months of current unemployment spell (%)

Never employed

Currently employed

12 months or less

6 months or less

7-12 months

More than 12 months

Number of months in current or most recent job (%)

b 12 months or less

More than 12 months

Is or has been employed in targeted industry (%)

Average hourly wage at current or most recent jobc ($)

Among those currently working

Worked full time (35 or more hours per week)d (%)

Among those currently working

Average weekly wage at current or most recent jobc ($)

Circumstances that may affect job change or retention (%)

Previously convicted of a crime

Previously incarcerated

Income and medical coverage

Average monthly family income ($)

Income sources (%)

Food stamps/SNAP

Welfare/TANF

Unemployment insurance benefits

Covered by health insurance plan (%)

3.6

13.0

56.1

41.3

14.8

27.2

55.5

44.5

7.9

11.91

10.05

58.6

12.2

408

10.4

6.2

647

17.4

6.2

24.3

54.3

2.1

10.5

55.9

41.2

14.7

31.5

54.0

46.0

2.1

13.00

10.84

65.1

28.0

454

19.9

18.2

695

41.9

13.7

24.6

54.3

0.7

26.7

58.0

49.9

8.0

14.6

65.9

34.1

14.7

10.32

9.54

71.4

46.8

400

39.7

34.2

804

34.5

0.7

7.5

28.3

2.7

26.6

44.0

32.7

11.3

26.6

58.3

41.7

30.6

9.88

9.46

61.1

43.5

342

25.4

12.0

600

55.2

3.9

10.5

48.1

2.3

20.0

53.3

41.3

12.0

24.4

58.8

41.2

14.9

11.12

9.72

64.0

37.7

396

24.2

17.6

686

36.9

5.5

16.0

45.6

3.4

45.9

10.5

40.1

9.77

29.8

Sample size 690 479 697 698 2,564 8,946

SOURCES: MDRC calculations from the WorkAdvance (WA) baseline information form and March 2012 Current 

Population Survey.

NOTES: PS = Per Scholas; SNA = St. Nicks Alliance; MSG = Madison Strategies Group; TE = Towards 

Employment; WA = WorkAdvance group; GED = General Educational Development.

Sample sizes may vary because of missing values.

Italics indicate the metric is not among the full sample shown in the table.

Low-wage workers for the U.S. sample are defined as individuals at least 18 years old, with a family income of 

less than 200 percent of the federal poverty level, not retired, and earning less than $15 per hour if currently 

employed. The definitions of "family" differed across data sources. a"Other race" includes sample members who identify as non-Hispanic and listed "Asian," "American Indian," or 

"other" as their race, including sample members who answered as "multiracial." 

bPercentage includes sample members who have never been employed.

c

Wages for sample members who have never been employed are counted as $0.

d"Worked full time" does not include the 2.3 percent of sample members who have never been employed.
24

sample had a previous criminal conviction, and even higher rates were seen within the transportation and manufacturing sectors. At study entry, less than half the sample members were covered by health insurance, 6 percent were receiving Temporary Assistance for Needy Families

(welfare), 16 percent were receiving unemployment insurance benefits, and over one-third were 

receiving food stamps.

Table 1.4 also presents a column showing selected baseline characteristics for a national 

sample of low-wage workers. While WorkAdvance targeted all unemployed and low-wage 

workers, the sample ultimately enrolled in the study reflects the providers’ choice of sectors and 

recruitment methods, and the study and provider screening criteria. There are a few notable differences between the WorkAdvance sample and the national low-wage worker sample. Over 

half of the national sample is female, compared with only 27 percent of the WorkAdvance sample. This is probably due in part to the sectors targeted by the WorkAdvance providers; all of 

the sectors, with the exception of health care, tend to be male dominated. The WorkAdvance 

sample had relatively higher levels of education: Over one-quarter of the national low-wage 

worker sample does not have a GED or high school diploma, compared with only 6 percent of 

the WorkAdvance sample. Few individuals in either sample lack work experience, but the national sample has higher proportions of individuals who are currently employed and individuals 

who have been out of work for over a year (not shown). 

Economic Conditions During the Study Period 

WorkAdvance began operations in June 2011, random assignment concluded in June 

2013, and this report covers outcomes through September 2015. The WorkAdvance study coincided with a time of fairly steady economic growth despite occurring during the slow recovery 

period following the Great Recession — a period when even relatively experienced and skilled 

workers struggled to find work. The early part of the recovery was notable for its lack of job 

creation and earnings growth. This was compounded by the fact that the period up to 2007 was 

sometimes called the “jobless recovery.” Thus, low-wage workers confronted an extended period of labor market stagnation.37 Recent studies indicate that employers have responded to this 

increased supply of unemployed workers by being especially selective about whom they hire, 

particularly in relation to recent work experience. Those who have been out of the labor market 

for six months or longer are much less likely to receive calls for job interviews, even when applicants have extensive relevant experience.38 Despite these challenging economic conditions, 

during the study period unemployment fell by 2.7 percentage points nationally — from 8.9 percent in 2011 to 5.3 percent in 2015. There was some important economic variation by site location, as shown in Appendix B. Notably, the economy was especially strong during the study 

 37Kolesnikova and Liu (2011).

38Ghayad (2013); Kroft, Lange, and Notowidigdo (2013).
25

period in Tulsa due to a local economic boom related to the natural gas and oil industries. While 

the economy in New York City was improving throughout the evaluation period, the environmental remediation industry (which the St. Nicks Alliance program focused on) was quite slack 

early in the follow-up period. In Ohio, the economy was fairly steady.

The Organization of This Report

The report is organized according to five primary questions outlined in the WorkAdvance Subgrantee Evaluation Plan (SEP) prepared as part of the SIF:39 

• How did the different providers implement the program model, and what adjustments did they make over time? (Chapter 2)

• To what extent did WorkAdvance increase engagement in employment and 

training services, compared with what would have happened in the absence 

of the program (represented by the control group)? (Chapter 3)

• What does it cost, on average, to operate the WorkAdvance program? (Chapter 4)

• To what extent did WorkAdvance improve primary economic outcomes 

(employment and earnings) and secondary outcomes (such as income and 

life satisfaction) over the follow-up period? (Chapter 5)

• What helps explain the economic impact results — the targeted sector? The 

characteristics of the study participants? Did these impacts change over time 

as providers matured? (Chapter 6)

• The conclusion (Chapter 7) describes the WorkAdvance results within the 

broader framework of other employment and training studies.

 39Deviations from the original questions included in the SEP were minor. As mentioned above, one shift 

was a focus on impacts by sector instead of by site (which affected only the Ohio site). In addition, the SEP 

specified that the evaluation would attempt to measure impacts on skills. This effort was abandoned, except to 

the extent that impacts on credentials are a proxy for impacts on skills. Finally, the WorkAdvance SEP raised 

the possibility of conducting a full benefit-cost analysis. Current plans are to conduct this analysis after five 

years of follow-up data have been obtained, to obtain a more reliable estimate of benefits. 

27 

Chapter 2

Implementation of the WorkAdvance Program

Chapter Highlights 

• The WorkAdvance intake screening process (which took place before random assignment occurred) was a key part of the program.

• Engagement of program enrollees in program services, particularly occupational skills training, was high across all four providers.

• Postemployment advancement coaching was the last program component to 

be implemented, but it was in place during the latter half of the service delivery period.

Introduction

The degree to which a program like WorkAdvance is implemented as intended can be influenced by a wide variety of factors: the population served, the organizational history and approach of the provider, staff capacity, local economic conditions, and the strengths or limitations 

of the chosen sector. The ways in which WorkAdvance was implemented, and how implementation differed across providers, is the focus of this chapter. An important point is that all major 

WorkAdvance program components were eventually implemented at all four providers. 

In addition to updating the implementation findings presented in the previous

port,1 the chapter describes the implementation of the program components that were offered to 

participants for up to two years after enrollment: preemployment and career readiness services, 

occupational skills training, job development and placement, and postemployment retention and 

advancement services. The chapter particularly focuses on retention and advancement services, 

which were the last set of WorkAdvance services to be implemented at all providers. It also updates participation statistics relative to the previous report, providing a total of 18 months of 

follow-up for the full program group sample — that is, for the 1,293 unemployed and low-wage 

working adults who were randomly assigned to the WorkAdvance group from June 2011 (the 

start of the WorkAdvance program) to June 2013.2

 

 1

Tessler et al. (2014).

2

Another 1,271 individuals were randomly assigned to the control group during this period, resulting in a 

total, cross-site sample size of 2,564 individuals.
28 

Analyses in the chapter draw on a variety of data sources. Qualitative data come from 

interviews with staff, participants, and employers; observations of participant-staff advancement 

coaching meetings; reviews of enrollee case files; and focus groups with program participants. 

Quantitative data on the recruitment and the flow of participants through the screening and enrollment process come from the providers’ management information systems (MIS), and data

that documented the frequency and content of advancement coaching sessions come from a 

special study. The qualitative data cover program operations through June 2014; the quantitative 

data used in this chapter go through December 2014.

Both qualitative and quantitative data were analyzed to understand several program implementation dimensions:

• What organizational and other factors influenced the operations of WorkAdvance at each site? 

• How closely did program operations hew to the original WorkAdvance program design, and how did they evolve over time? 

• To what extent did participants engage in the various WorkAdvance program 

components? 

• How well did program staff members deliver services, particularly postemployment services, to identify and support advancement opportunities? 

• To what extent did participants complete training and obtain credentials? 

• What were participants’ views of the program? 

In the analysis, qualitative data were coded into themes and trends and then compiled 

across codes and analyzed by theme. Quantitative data were generally analyzed for set followup periods relative to each program enrollee’s random assignment date. See Appendix A for 

more details on data collection and analysis.

Key Findings

• The WorkAdvance intake and screening process, which occurred before 

random assignment, resulted in many interested individuals being identified as not academically qualified or self-selecting out of the process. 

This process increased the recruitment burden on the sites but may have 

contributed to the high program activity participation and completion 

rates. 
29 

Only about 20 percent of the individuals who initially expressed interest in WorkAdvance were both still interested in and qualified for enrollment by the end of the program intake 

process, before random assignment was conducted. The process included several steps, and interested individuals were asked to report to the provider on multiple occasions. This thorough 

process was in itself a screen for individuals’ motivation to participate in the program. Most 

people who did not eventually enroll in the program either withdrew of their own accord or 

failed to achieve the required score on the academic assessments, which reflected sector and 

occupation standards. While the rigorous intake process was a burden on the providers, it probably contributed to the high participation and completion rates discussed below. 

• Translating the WorkAdvance model into a set of concrete, implemented services took time — more than a year for some components and 

providers — and all four providers required a substantial amount of 

technical assistance and related support, although some required more

time and assistance than others. 

The program model was designed specifically for the WorkAdvance demonstration, 

and each provider had little or no experience with at least one of the WorkAdvance components, especially the postemployment emphasis on career advancement rather than just job retention. The model also demanded that providers have a strong capability to work with employers, training partners, and program participants. In addition, preparing for and carrying out 

random assignment was labor intensive and impinged on program operations at times. The providers’ varying degrees of experience in each of these areas influenced their implementation of 

the program in different ways, as described in the chapter. 

• All four providers eventually delivered services across all the WorkAdvance model components, with postemployment services the last to be 

fully developed and implemented. The engagement of program enrollees 

in key program components was high across all four providers.

As illustrated in Table 2.1, which summarizes key findings on participants’ engagement 

in program services, close to 95 percent of WorkAdvance group members engaged in career 

readiness activities. Within 18 months of enrollment, around 76 percent started occupational 

skills training, and about 63 percent of all WorkAdvance group members completed it. In most 

cases, completion of occupational skills training led to the earning of a nationally or locally recognized credential (or both) — a critical first step toward getting a job in a sector. Across all 

providers, a high proportion of all WorkAdvance enrollees — 55 percent — earned such licenses or certificates. 
30 

The WorkAdvance Study

Table 2.1

Summary of Indicators of Participation in Program Group Activities Within

18 Months of Random Assignment, Cross-Site

Per St. Nicks Madison Towards

Participation in program activity since RA (%) Scholas Alliance Strategies Group Employment Overall

a Ever participated in any career readiness activity 95.7 82.6 98.3 98.3 94.7

Ever started skills training 95.7 82.6 62.0 65.3 75.9

Ever completed skills training 78.5 78.9 48.7 50.1 62.8

Ever obtained a credentialb in targeted sector 72.5 73.6 48.7 31.8c 55.2

Sample size 349 242 353 349 1,293

SOURCES: Program tracking systems managed by Per Scholas (PS), St. Nicks Alliance (SNA), Madison 

Strategies Group (MSG), and Towards Employment (TE).

NOTES: RA = random assignment.

Refer to Appendix Table C.3 for site-specific details regarding how each metric is defined. aThe first program activity at PS and SNA is skills training, which is offered in combination with career 

readiness training and includes help with developing a career plan, résumés, and interview preparation. At MSG, 

the first progam activity is career readiness training. At TE, the first program activity is an initial assessment 

wherein career goals and barriers to employment are discussed. 

bCredentials in the targeted sector are recognized locally or nationally, or both. There is cross-site variation in 

reporting of locally recognized credentials obtained in the targeted sector. cTE’s program tracking system captures only nationally recognized credentials. Therefore, participants who 

obtained the locally recognized computer numerical control (CNC) machining credential are not counted as ever 

having obtained a credential.

Postemployment advancement coaching did not occur in earnest until after many of the 

other WorkAdvance program components were put in place, and for many providers not until 

sample recruitment was complete — roughly two years after WorkAdvance started. Still, in the 

later part of the follow-up period it appears that this service was offered more systematically.

Providers’ Backgrounds and Experience

The four providers selected for WorkAdvance had a range of backgrounds. Ultimately, this 

benefited the study, as it allowed the study to learn, among other things, whether a variety of 

providers — including some that were less experienced in their delivery of sectoral programs 
31 

than those included in the earlier Sectoral Employment Impact Study (SEIS)3 or that had no sectoral experience at all — could successfully implement WorkAdvance. As described in Chapter 

1, the following four organizations were selected to operate WorkAdvance:

• Per Scholas in the Bronx, New York, which focused on the information 

technology sector

• St. Nicks Alliance in Brooklyn, New York, which focused on environmental 

remediation and related occupations

• Madison Strategies Group in Tulsa, Oklahoma, which focused on the transportation and (later) manufacturing sectors

• Towards Employment in northeast Ohio, which focused on the health care 

and manufacturing sectors

This section provides more information about the providers, with an emphasis on their experience running sector programs within the location and sector that they operated for WorkAdvance.4

 

Per Scholas

Per Scholas is a nonprofit information technology training and employment services 

provider. Launching and operating the WorkAdvance program components took the least effort 

for Per Scholas, which already had substantial experience with sectoral programs and evaluations, including in the SEIS. Per Scholas had essentially already operated all of the main components of WorkAdvance since 1998, with the exception of career advancement and postemployment services. The content of technical assistance there primarily concerned adding 

these two domains to what Per Scholas was already doing.5

 

When it came to establishing strong employer relationships, Per Scholas had a head 

start, as it had already been operating a sector program in information technology and had important employer relationships in place, though it also developed new ones for WorkAdvance. 

However, Per Scholas’s emphasis had primarily been on getting its graduates into entry-level 

information technology jobs, not assisting with career advancement. Regardless of this shift in 

emphasis, Per Scholas maintained a strong business-facing focus; never was there a reference to 

Per Scholas graduates being “program participants” or “low income.” 

 3

Maguire et al. (2010); see Chapter 1 of the present report. 4

For more extensive background on the providers, as well as more information on the early implementation of WorkAdvance, see Tessler et al. (2014). 5

For all providers, technical assistance was provided by MDRC, and sometimes by contractors working 

closely with MDRC.
32 

St. Nicks Alliance

St. Nicks Alliance is a large multiservice agency housed in several locations in Williamsburg, Brooklyn, offering help with affordable housing and health care, workforce development services, and youth services, among other social programs; WorkAdvance was a small 

part of the organization and was largely staffed by individuals already working within St. Nicks 

Alliance. This provider had to make the biggest shifts in order to implement the WorkAdvance 

model. Although St. Nicks Alliance had operated specific vocational training programs for 

more than 10 years, including one in environmental remediation, it had to develop a sectoral 

strategy for its skills training program; diversify its sectoral effort to include related occupations, 

such as pest control; and add a focus on advancement and on working with employers in more 

of a demand-driven way. These changes took time. Additionally, the staff members managing 

WorkAdvance were balancing its needs with the need to attend to St. Nicks Alliance’s other 

workforce development programs.

St. Nicks Alliance had developed some fruitful relationships with employers within the 

environmental remediation sector before WorkAdvance, but it was more accustomed to providing case management to address participants’ employment barriers and less accustomed to treating employers as a second customer. Staff members working on WorkAdvance initially were 

not well connected to employers’ needs and relied on a small network of employers for placements, rather than attempting to expand that network. Eventually, with technical assistance, the 

number of these connections started to grow somewhat, but St. Nicks Alliance’s overall orientation toward workers rather than employers did not change much over time.

Madison Strategies Group

Madison Strategies Group is a nonprofit spinoff of Grant Associates, a for-profit workforce development company that operates a variety of programs in New York City, including 

sector strategies. Madison Strategies Group transferred its parent organization’s considerable 

sectoral experience from New York City to Tulsa, where it had never operated. Madison Strategies Group initially offered services only in the transportation sector, including training for 

transportation-related manufacturing; however, the manufacturing focus gradually became distinct from transportation as it became clear that someone who is trained to manufacture transportation-related parts has the requisite skills to work in manufacturing more generally. It is 

now most accurate to say that Madison Strategies Group focuses on both the transportation and 

manufacturing sectors.

While its leaders’ sector experience helped Madison Strategies Group understand what 

it would take to launch and operate WorkAdvance, it was at a disadvantage relative to the other 

providers in that it was developing completely new operations in a new location, including hiring a staff, making connections with training providers, and developing brand new employer 
33 

relationships. Additionally, while its leaders had a deep understanding of the transportation sector from their work in New York City, when Madison Strategies Group expanded into manufacturing it had to develop knowledge of that sector and forge new relationships with employers — 

in addition to learning the Tulsa-specific nuances of the transportation sector. That said, because 

Madison Strategies Group was singularly focused on WorkAdvance and had sector experience 

through its relationship with Grant Associates, it was able to establish WorkAdvance services 

more efficiently than would have been likely if it had also been operating other programs or was 

new to sector programs in general. 

Towards Employment

Towards Employment is a community-based organization that provides a broad range 

of employment services for low-income populations in Greater Cleveland. Towards Employment focused on the health care and manufacturing sectors in WorkAdvance, and it had its own

challenges in implementing the program. While it was an established organization in its community and had extensive experience with workforce and work readiness programs, particularly 

in the health care sector, it had not operated sector programs before, and its experience was limited primarily to entry-level nursing assistant and other low-skill long-term care positions. In 

addition, many existing staff members had backgrounds primarily in general employment issues 

and the provision of social services. Finally, it had to learn about the manufacturing sector and 

had to launch and develop WorkAdvance services in two sectors and in two locations.6

 

In an arrangement probably typical of many sector programs, Towards Employment 

managed multiagency collaborative partnerships in both locations with organizations that were 

responsible for delivering different aspects of the WorkAdvance services, allowing Towards 

Employment to draw on, and not duplicate, existing service expertise in the area. In addition, 

Towards Employment affiliated with industry intermediary groups that brought their own industry knowledge and relationships with targeted employers to the delivery of WorkAdvance 

services. The organization managed these complex relationships while operating other programs at the same time. Even under these circumstances, Towards Employment put all the 

WorkAdvance pieces in place. 

WorkAdvance’s Services 

This section describes the recruitment, screening, and enrollment of the WorkAdvance research 

sample and the implementation of each component of the WorkAdvance model. It also describes the extent to which enrollees actually engaged in the services offered. Broad descrip-

 6

As mentioned in Chapter 1, the Compass site in Ohio was not included in the evaluation’s analysis. (See 

Tessler et al., 2014.)
34 

tions are provided; site-specific practices are mentioned only when a site’s practice differed significantly from those of the other sites, or as examples.

Recruitment and Screening

• WorkAdvance had fairly intensive screening of program applicants. Individuals who eventually enrolled in the program, however, still faced 

substantial barriers to employment.

The WorkAdvance recruitment and screening process sought to identify candidates 

who had the ability to complete training successfully and be attractive to employers, while not 

being so qualified that they could find appropriate employment on their own and thus would not 

need the program’s services. In addition, unlike many other workforce programs that may have 

only basic eligibility requirements such as income guidelines, WorkAdvance required that candidates demonstrate an interest in and commitment to the particular sector, have the ability (as 

perceived by WorkAdvance staff members) to work in that sector, and have the motivation 

(again, in the view of the program staff) to complete the required training.

As a result, the WorkAdvance providers screened applicants fairly intensively. Program 

administrators used a combination of objective criteria (such as income guidelines and reading 

and math literacy test scores) and subjective criteria (such as staff views of applicants’ attitudes, 

interest in training and likelihood of completing it, and desire for advancement) to screen 

WorkAdvance applicants for program suitability. Nevertheless, many enrolled individuals faced 

substantial barriers to employment; for example, long-term unemployment, previous criminal 

convictions, unresolved legal matters, and unstable housing.

• About one in five program applicants enrolled in the study and were 

subsequently randomly assigned. The vast majority of applicants who 

were screened out did not meet objective academic criteria or failed to 

attend enrollment-required meetings.

Among individuals who initially expressed interest in WorkAdvance, a minority — 

from about 35 percent at St. Nicks Alliance to only about 8 percent at Madison Strategies Group

— were both still interested in and qualified for enrollment by the end of the intake process. 

(See Figure 2.1.) It is important to note, however, that relatively few applicants were screened 

out as inappropriate by more subjective criteria at the discretion of the providers’ staff. Most of 

the individuals who did not eventually enroll in the program either withdrew on their own accord or failed to achieve the required score on assessments of their academic level. The thorough intake process, which could span several days, turned out to be a key feature of program 

implementation, despite occurring before random assignment. For example, the up-front
35 

screening for applicants’ motivation is probably one of the reasons for the high rates of participation in program activities, discussed below. 

Since it was understood from the outset that some individuals would be screened out 

before random assignment, the providers sought to make recruitment efforts as efficient as possible. Efficiency was especially important because the providers were filling class cohorts to 

start training together; the longer it took to recruit eligible and qualified individuals, the longer 

those who were enrolled and ready to start training had to wait for their classes to begin. 

Sources that brought in the largest number of applicants for WorkAdvance — for example, 

Percentage of Applicants Remaining, by Step in Intake Process

Figure 2.1

The WorkAdvance Study

0

10

20

30

40

50

60

70

80

90

100

All applicants Orientation Assessment(s) Interview(s) Internal case

conference

Random

assignment

Percentage

Per Scholas (N = 70)

St. Nicks Alliance (N = 145)

Madison Strategies Group (N = 432)

Towards Employment (N = 1,136)

Intake step

SOURCES: MDRC calculations from a recruitment questionnaire administered at Per Scholas and St. Nicks 

Alliance, a report based on program tracking data provided by Madison Strategies Group, and program tracking 

data provided by Towards Employment.

NOTE: Refer to Appendix Tables C.1 and C.2 for provider-specific details regarding the length and timing of the 

data coverage period and the definition of an applicant, as there is variation across providers.
36 

friends and family members and the Internet — were not always the same sources that brought 

in the largest pool of eligible and qualified applicants, depending on the provider, and an awareness of this distinction helped providers direct their outreach efforts more productively. (See 

Appendix Table C.1.)

• Recruitment and screening procedures did not differ substantially 

across the providers.

Each provider customized recruitment and screening nuances to its specific targeted 

sectors, but no provider stood out from the others in level of intensity of scrutiny. (See Appendix Table C.2 for the intake procedures by provider.) All providers refined their screening 

processes over time, based on employer feedback. For instance, Per Scholas eliminated some 

questions from the Test of Adult Basic Education that assessed skills that were not of high interest to information technology employers. In another example, St. Nicks Alliance added a 

criminal background check to the environmental remediation technician training program intake 

process, because potential employers were eventually screening out candidates with such a 

background. 

Informally, WorkAdvance program managers across all four providers mentioned that 

toward the “end” of the study enrollment period, which each defined differently (anywhere 

from the last six months to the last month or so of enrollment), they relaxed their screening 

standards somewhat in order to get the required number of people into the study by the sample 

enrollment deadline.7 While conversations with program managers suggested that they allowed 

less “qualified” individuals into the program than they would have preferred — for example, 

people who may not have showed the desired commitment or motivation or may not have presented themselves in the best light — a comparison of demographic data for enrollees who entered WorkAdvance in the last few months of the study enrollment period with data for enrollees who entered WorkAdvance earlier did not reveal stark differences, and differences that did 

emerge did not go in a consistent direction: Some comparisons suggested that late enrollees 

might have been more qualified than earlier ones; others suggested the opposite. The program 

managers’ impressions, however, could still be true, as the demographic data do not shed light 

on enrollees’ relative levels of commitment, motivation, or other more subjective factors.

Once a potential program participant passed through the screening process, he or she 

was randomly assigned to either the WorkAdvance group or the control group. WorkAdvance 

group members’ subsequent flow through the WorkAdvance program varied by provider. A 

principal reason for this variance is that, as discussed in Chapter 1, Towards Employment and 

 7

As a result of the difficulties that the providers encountered with recruitment, study sample goals were 

revised downward several times. The total sample goal across providers was originally 3,850, to be enrolled 

over a two-year period (mid-2011 to mid-2013); in the end, it was reduced to 2,600, or between 500 and 700 

individuals per provider.
37 

Madison Strategies Group initially had two different program tracks: one that offered training 

first and another that placed people directly into jobs first, with formal or informal training to 

follow. In contrast, all WorkAdvance group members at Per Scholas and St. Nicks Alliance 

went into training first. Individuals assigned to the control group, regardless of provider, were 

given information about the local workforce system, including the American Job Centers network (One-Stop Career Centers), where they could seek information about training opportunities. 

Preemployment and Career Readiness Services

In programs such as WorkAdvance, many participants need more than just vocational

training to find a job and succeed in it. The WorkAdvance model therefore included a career 

readiness component, which included classes to teach WorkAdvance participants about the designated sectors and help them acquire the résumé and the interview skills needed to be hired and 

the “soft skills” critical to success in the sectors, as well as preemployment coaching to help 

participants set and follow through with career advancement goals (Table 2.2).8

 

• Almost all WorkAdvance group members engaged in career readiness 

activities, and the basic content of such activities was similar across providers.

As Table 2.3 indicates, WorkAdvance group members’ engagement in career readiness 

activities was impressively high: Combining all the providers, close to 95 percent of WorkAdvance group members engaged in a career readiness activity.

All providers covered such topics as an introduction to the sector; how to get a job in 

the sector, including how to develop a résumé and cover letter; job search ideas and how to prepare for job interviews (one provider, for example, had all WorkAdvance participants prepare 

and perfect over time a one-minute pitch to employers); and the development of individualized 

career plans (ICPs)9

 — although some emphasized certain topics more than others.10 Some also 

 8

“Soft skills” refers to the general habits and competencies that make for an effective employee, such as 

showing up to work on time and working cooperatively with others. 9

While seemingly a good tool for first-time planning, the ICP in practice was not the “living document” 

that it was intended to be; it was mostly not updated after it was initially completed. However, staff members 

did use the ICPs sometimes to reengage participants and restart conversations with those whose contact with 

their coach had lapsed. 10Despite attempts to customize some of the curricula to the targeted sector, feedback from participant interviews suggests that career readiness classes largely covered general topics. Preparation for work in a given 

sector appeared to be covered more thoroughly and more intensively in orientations that the providers offered 

— sometimes as stand-alone activities and sometimes as the first session of career readiness classes — than in 

the classes themselves. 
38

SOURCES: Documentation supplied by providers and interviews with provider staff.

NOTE: CRT = career readiness training; OST = occupational skills training.

The WorkAdvance Study

Table 2.2 

Career Readiness Services, by Provider

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Career readiness services

Sector 

orientation

Stand-alone 1-day kickoff; 

also integrated into CRT Integrated into CRT Integrated into CRT Integrated into CRT

Career 

readiness 

training 

Concurrent with OST Concurrent with OST

Typically provided for 1 week 

immediately following 

enrollment and before start of 

OST

Placement-first: typically 

provided for 2 weeks before

start of job search; trainingfirst: typically provided 

concurrent with OST

Hours of CRT 12 sessions at 7 hours each 9 sessions at 4 hours each 5 sessions at 6 hours each 10 sessions at 6 hours each 

Transportation 

assistance Need-based Need-based

Bus passes or gas cards are 

provided to all training 

participants; otherwise needbased

Need-based

Preemployment 

coaching

One-on-one sessions during 

OST. Coaches follow up with 

e-mails or in person during job 

search.

One-on-one sessions during 

OST. Coaches are expected to 

follow up 2-3 times per week

during job search. 

One-on-one sessions during 

CRT. Coaches visit weekly 

with participants as a group 

during OST. One-on-one 

coaching during OST as 

needed.

Coaches introduce career plan 

during CRT. Coaches meet 

with participants in groups

and one on one up to 3 times 

per week during job search.
39 

focused more than others on workplace skills such as punctuality, teamwork, and problemsolving. All the providers had to adjust to the WorkAdvance model idea that messages about 

planning for advancement had to be infused from the beginning of enrollees’ engagement with 

WorkAdvance and not treated as an add-on after enrollees became employed. It took a while 

before providers addressed advancement from the start of service delivery — for example, by 

having the occupational skills instructors talk about the different types of skills that would be 

required as someone advanced within a particular industry, or by having instructors work with 

participants to think about career ladders rather than just a first job and incorporate these advancement goals into their ICPs.

Per St. Nicks Madison Towards 

Career readiness activity since RA (%) Scholas Alliance Strategies Group Employment Overall

a Ever participated in any career readiness activity 95.7 82.6 98.3 98.3 94.7

b Started classroom-based career readiness training 95.7 82.6 98.3 70.2 87.1

Completed classroom-based career readiness

 training 78.5 78.9 92.1 62.8 78.0

c Received help obtaining supportive services 49.0 NA 72.2 71.4 64.2

Sample size 349 242 353 349 1,293

The WorkAdvance Study

Table 2.3

Participation in Career Readiness Activities and Supportive Services

Within 18 Months of Random Assignment, Cross-Site

SOURCES: Program tracking systems managed by Per Scholas (PS), St. Nicks Alliance (SNA), Madison 

Strategies Group (MSG), and Towards Employment (TE). 

NOTES: RA = random assignment; NA = not applicable.

Refer to Appendix Table C.3 for site-specific details regarding how each metric is defined.

Only career readiness activities before first placement are shown in the table. These might include attending 

post-RA orientation or assessment, starting career readiness training, developing an individualized career plan, 

receiving help with résumé development and job applications, and interview scheduling or preparation. a

The first program activity at PS and SNA is skills training, which is offered in combination with career 

readiness training and includes help with developing a career plan, résumés, and interview preparation. At MSG, 

the first program activity is career readiness training. At TE, the first program activity is an initial assessment 

wherein career goals and barriers to employment are discussed. 

bSome participants may engage in career readiness training with the provider outside a classroom setting. At 

TE, these participants are counted as ever having participated in any career readiness activity. cInformation on supportive service take-up was not included in program tracking data provided by SNA. The 

overall rate for this measure is among participants from PS, MSG, and TE. 
40 

Preemployment coaching — one aspect of the career readiness activities — was focused on a range of issues, some more technical in nature and some more emotional or behavioral. Activities and discussion topics included reviewing résumés and preparing for job interviews; discussing anxiety about going back to work; helping participants weigh pay from 

employment against unemployment benefits; addressing unrealistic expectations of wage rates; 

reminding participants to take into account commuting time and costs when considering job 

offers; encouraging discouraged job seekers; preparing individuals to retake certification exams; 

and identifying how to sell “transferable” skills to potential employers.

• Providers differed in the schedules and length of their career readiness 

training, the curriculum used, the instructors’ experience, and the 

amount of advancement coaching provided, as well as in the degree to 

which workplace norms were modeled by the WorkAdvance staff.

The provider that approached this program component most distinctively was Per Scholas. Relative to the other providers, Per Scholas placed more emphasis on staff teaching and 

modeling workplace norms and holding WorkAdvance participants accountable to an expected 

set of behaviors.11 It also had a longer series of career readiness training classes and more career 

coach contact with WorkAdvance participants; its career readiness classes were concurrent with 

occupational skills training and conducted in the same location as other WorkAdvance activities, allowing more opportunities for staff members to interact with participants.

• Employer partners influenced and were engaged in career readiness activities in a variety of ways: via counseling to providers on curricula and 

sector trends, as guest speakers in classes, as mock interviewers of participants, and as hosts for participant workplace visits. 

Madison Strategies Group and Per Scholas, for example, used advisory groups to counsel them on curricula for career readiness classes as well as on occupational skills training and 

the latest trends in the sector. Madison Strategies Group also hired transportation industry consultants who came in every week to speak with WorkAdvance participants about the industry. 

Towards Employment had relationships with individual employers and developed relationships 

with existing business intermediary groups in the health care and manufacturing sectors, which 

carried out similar functions. St. Nicks Alliance relied primarily on relationships with individual 

employers in the environmental remediation sector to get input about its career readiness activities. The providers had a range of employer partners who came to the organizations’ offices to 

talk about their companies or conduct mock interviews, and employers also hosted visits that 

gave program participants firsthand exposure to the work environment that they could expect to 

 11For example, Per Scholas enforces a “zero tolerance” policy for the first three weeks, in which a participant can be dismissed from the program for arriving late to class.
41 

join. As expected, all the providers adjusted their career readiness services over time, based at 

least in part on input from employers. Changes included Per Scholas’s addition of a customer 

service component, Madison Strategies Group’s use of actual employer interview questions to 

reinforce best practices in applying for a job, and Towards Employment’s integrated examples 

of real job descriptions and performance reviews. 

Occupational Skills Training

Occupational skills training in WorkAdvance was intended to meet the needs of both 

program enrollees — by helping them to obtain industry-recognized credentials and develop 

technical skills relevant to the targeted sectors and occupations within those sectors — and particular local employers. The providers selected their training offerings based on local industry 

demand, staff and institutional knowledge of the industry, and the characteristics of targeted 

occupations — including job entry requirements, pay rates, benefits, and opportunities for advancement. Table 2.4 summarizes the key features of the training courses offered by the WorkAdvance providers. 

• The length, location (at the WorkAdvance site or off-site), and breadth 

of the occupational skills training options differed by provider.

The length of training, by design, ranged from 2 weeks to 32 weeks, depending on the 

occupation, and was, for the most part, “hands-on” in nature. All four WorkAdvance providers 

generally offered training in cohorts — that is, groups of participants went through training cycles together. All the providers offered full-time training during regular business hours, though 

Towards Employment and Madison Strategies Group eventually also offered evening and parttime classes (both in cohorts and in “mixed” classes), in part to accommodate participants who 

worked during the day. 

In addition to a range of training course durations, providers varied in terms of where 

training was held (on-site or through a contract with an off-site training provider) and the 

breadth of the offerings. Per Scholas was unique in offering all of its training on-site and with its 

own instructors. St. Nicks Alliance had a blend of off-site and on-site trainings (with independently contracted instructors), and Towards Employment and Madison Strategies Group 

partnered with off-site training institutes, including community and technical colleges, with 

some of the courses specifically tailored at the request of these providers (Table 2.4). While the 

evaluation did not explore indicators of the quality of training or whether training quality differed in terms of the location, training provided on-site (at the location where other WorkAdvance services were provided) may have had some “spillover” advantages: allowing for more 

contact between the WorkAdvance program staff (including career advancement coaches) and 

the participants, and making in-house instructors generally more aware of the overall mission of 

WorkAdvance, which could have reinforced messages about the importance of planning for 
The WorkAdvance Study

 

Table 2.4 

Occupational Skills Training, by Provider

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Occupational skills training

Duration 15 weeks 5 to 12 weeks 4 to 32 weeks 2 to 17 weeks

Training

courses

offered

A Plus, Network Plus

Environmental Remediation 

Training, Commercial Driver's 

License B (CDL-B) with 

hazmat endorsement, Pest 

Control Technician

Aviation Manufacturing, 

Commercial Driver's License 

A and B (CDL-A, CDL-B), 

Computerized Numerical 

Control (CNC) Machining, 

Diesel Mechanic, Welding, 

Supervisory Leadership

MF: Computerized Numerical 

Control (CNC) Machining, 

Welding

HC: Phlebotomy, Certified 

Health Care Access Associate, 

Patient Care Assistant, StateTested Nurse Assistant, 

Medical Billing and Coding

Target 

occupations

Help Desk Technician, IT 

Field Technician

Environmental Remediation 

Technician, Tanker or Hazmat 

CDL Driver, Pest Control 

Technician

Aviation Manufacturing 

Assistants, Semi- and Truck 

Driver, CNC Operator, Diesel 

Maintenance Technician, 

Welder

MF: CNC Operator, Welder

HC: Phlebotomist, Patient 

Access Specialist, Patient Care 

Assistant and State-Tested 

Nurse Assistant, Certified 

Professional Coder

On-site/offsite On-site

On-site for pest control

training; mix of on- and offsite at private schools for all 

other training courses

Off-site 

schools

at private or technical 

or community colleges

Off-site 

schools

at private or technical 

or community colleges

 

SOURCES: Documentation supplied by providers and interviews with provider staff.

NOTES: MF = manufacturing; HC = health care, IT = information technology.

42
43 

career advancement that participants were hearing from their career readiness instructors. It was 

also probably easier to customize the training according to feedback from employers when the 

instructors were directly employed by the program; with contracted training, customization required more negotiation.

The providers also used a number of approaches, and invested substantial resources, to 

ensure training completion. Their efforts, which varied by provider, included offering paid internships, covering transportation costs, adjusting training schedules to allow for evening classes so that participants could work during the day, and providing activities to get families involved and foster support among them. (Specifically, Madison Strategies Group convened open 

houses at training locations so families could see the equipment their family members would be 

using, understand what they would be doing, and feel more involved and supportive.) Providers 

also checked in regularly with participants and instructors to track progress, including visiting 

off-site training locations on a regular basis. Reasons participants dropped out of training, 

gleaned from case notes, included, for example, an immediate need for income because unemployment insurance ran out sooner than expected; work schedule changes that no longer allowed the participant to attend training; and behavioral or mental health setbacks, such as debilitating episodes of depression. The providers tried to work around these issues by rescheduling 

participants for later cohorts or helping them to find employment in the sector.

• Rates of participation in and completion of training were high across the 

providers, and over half of all enrollees received a nationally or locally

recognized training credential — a critical first step toward getting a job 

in a sector.

Within 18 months of random assignment, 87 percent of all WorkAdvance group members had been scheduled for occupational skills training; about 76 percent of all WorkAdvance 

group members actually started the training; and, among those who started training, few — only 

16 percent — dropped out (Table 2.5). These high training participation and completion rates 

probably reflect a number of factors: the extensive upfront screening done in WorkAdvance,12

in part to gauge motivation but also to enroll individuals who had the academic skills to handle 

the training; enrollees’ relatively high levels of pretraining education; the peer support offered 

through WorkAdvance’s cohort-based model; appealing instructors and engaging material; and 

the level of effort that staff members put into supporting participants while they were in skills 

training.

 12Full-time training courses meant that participants had to find ways to support themselves financially. At 

Per Scholas, the screening process required applicants to have financial support that would enable them to 

complete a 15-week, full-time training cycle. As noted above, Towards Employment and Madison Strategies 

Group offered some evening and part-time classes, which allowed participants to work while also participating 

in training.
44 

The WorkAdvance Study

Table 2.5

Participation in Skills Training Activities Within

18 Months of Random Assignment, Cross-Site

Per St. Nicks Madison Towards

Skills training activity since RA Scholas Alliance Strategies Group Employment Overall

Ever scheduled to start skills training (%) 100.0 100.0 78.2 73.4 86.9

Ever started skills training 95.7 82.6 62.0 65.3 75.9

Ever completed skills training 78.5 78.9 48.7 50.1 62.8

Among participants who started skills training 82.0 95.5 78.5 76.8 82.8

Ever dropped out of skills training 16.6 3.7 12.5 12.9 12.1

Among participants who started skills training 17.4 4.5 20.1 19.7 15.9

Enrolled in skills training at 18 months after RA 0.6 0.0 0.9 2.3 1.0

Among participants who started skills training 0.6 0.0 1.4 3.5 1.3

a Ever obtained a credential in targeted sector 72.5 73.6 48.7 31.8b 55.2

Average number of weeks in skills trainingc (#) 13.8 6.8 9.4 9.1 10.0

Among participants who started skills training 14.4 8.2 15.1 13.9 13.2

Sample size 349 242 353 349 1,293

SOURCES: Program tracking systems managed by Per Scholas (PS), St. Nicks Alliance (SNA), Madison Strategies 

Group (MSG), and Towards Employment (TE).

NOTES: RA = random assignment. 

Italics indicate the metric is not among the full sample shown in the table.

Skills training activity refers to enrollment in an occupational skills training course in the targeted sector or 

industry. Refer to Appendix Table C.3 for site-specific details regarding how each metric is defined.

The results in this table are derived from providers' management information system data. Other measures of 

training participation and completion, shown elsewhere in this report, are based on survey data and present different 

estimates. It is common for self-reports from survey data to differ from program records for numerous reasons, 

including problems recalling the timing of events or activities. aCredentials in the targeted sector are recognized locally or nationally, or both. There is cross-site variation in 

reporting of locally recognized credentials obtained in the targeted sector. 

bTE’s program tracking system captures only nationally recognized credentials. Therefore, participants who 

obtained the locally recognized computer numerical control (CNC) machining credential are not counted as ever 

having obtained a credential. cThe scheduled lengths of training varied across sites, industries, and types of training.

According to the providers’ MIS data, those who started training generally began it 

within a few months of random assignment and generally completed it within a year of random 

assignment. Across all providers, individuals who started skills training participated in it for an 

average of over 13 weeks (commonly full-time) during the follow-up period (Table 2.5).
45 

Provider-specific rates of starting and completing skills training are not clearly correlated with any of the differences in training setup described above; that is, such factors as length or 

location of training do not seem to be correlated with rates of starting or completing training. 

Rather, differences across providers in these rates are primarily driven by the two-track approach used by two providers (Madison Strategies Group and Towards Employment) before

summer 2012, which allowed more WorkAdvance participants to be placed directly into jobs 

rather than engaging in training first. Skills training participation and completion rates at these 

two providers are much higher for the late cohort — after the providers began to direct more 

participants to training before employment — than they were for the early cohort, as discussed 

further below.

13 

In most cases, completion of occupational skills training led to earning either a nationally or a locally recognized credential (or both). (See Table 2.5.) Across all sites, a high proportion of WorkAdvance group members — 55 percent — earned such licenses or certificates. 

Again, differences across providers in these statistics primarily reflect the two-track approach 

used early on by Madison Strategies Group and Towards Employment, as well as a lack of data 

on locally recognized credentials for one of the providers.14

• Overall, the WorkAdvance occupational skills training component was 

informed by and adaptive to local labor markets.

All providers adapted their skills training curricula at some point, based on labor market 

and employer input, and most altered their course offerings as well. Madison Strategies Group, 

for example, added a week to its computerized numerical control (CNC) machining course because employers felt that the participants needed specific additional skills, and it dropped several components from the aviation training, based on employer feedback that those skills would 

not be needed. Per Scholas developed a unit that trained on the repair and programming of 

handheld devices, such as tablets and smartphones, and St. Nicks Alliance changed its environmental remediation certificate offerings in response to high demand for mold remediation skills 

following Hurricane Sandy. Towards Employment added an online learning module to its basic 

 13Statistics on the proportion of program enrollees scheduled for training can illustrate the influence of the 

placement-first track, which was primarily in place for individuals who enrolled in the early cohort, on the likelihood of individuals being directed to training: At Madison Strategies Group, 64 percent of the early cohort 

were scheduled for training within 18 months of their random assignment, compared with 92 percent of the 

later cohort. At Towards Employment, 60 percent of the early cohort were scheduled for training within an 18-

month follow-up period, compared with 86 percent of the later cohort. 14The credentials obtained through the diesel mechanic and CNC machinist training sessions at Madison 

Strategies Group and Towards Employment were unique to the local industry’s employer needs and were created by the WorkAdvance providers, in concert with employers or training providers or both, to abbreviate and 

adapt manufacturing training courses and offer credentials accepted by local employers. Madison Strategies 

Group included the credentials obtained upon completion of this locally recognized course in its rates measuring the earning of a credential, while Towards Employment did not, though the courses were very similar.
46 

machining course, in order to incorporate additional content that was preferred by manufacturing employers but was not typically offered in shorter machining classes.

Over time, most of the WorkAdvance providers offered new areas of training and 

dropped others. Madison Strategies Group, for example, began CNC machining training in June 

2012 and welding training in February 2013 — both in response to employers’ and industry 

experts’ input. Likewise, it suspended enrollment for a second training session in aviation and 

aerospace in August 2012 in order to better assess the market need, and subsequently held just 

one more session before discontinuing it. Per Scholas responded to employer feedback by adding a separate training course for program graduates that focused on software testing. St. Nicks 

Alliance added pest control technician training in August 2012, when the demand for environmental remediation technicians was decreasing and demand for pest control technicians was 

confirmed. Finally, Towards Employment adjusted the types of welding credentials that were 

targeted, based on employer needs, and stopped recruiting for health care patient navigators, 

since the occupation was not growing at the pace initially projected. 

Job Development and Placement Services

• WorkAdvance job developers served as a crucial link between the program and employers, to maximize the chances that program participants would find jobs that offered good pay, security, and advancement 

opportunities.

Identifying jobs that offer good wages, employment security, and opportunities for advancement requires an understanding of local labor markets and the specific needs of employers.15 Moreover, prior workforce studies suggest that sector-based training programs that have 

direct relationships with employers have clear advantages.16 Job developers thus serve an important role as conduits through which program managers receive timely feedback from employers and local labor market information so that they can prepare program enrollees for the 

best jobs available. 

In WorkAdvance, MDRC provided substantial technical assistance in some cases to 

help job developers establish closer relationships with employers and industry associations.17 At 

Madison Strategies Group and Towards Employment, the technical assistance was “lighter 

touch” and focused on early development of local sector relationships. Conversely, St. Nicks 

 15D’Amico (2006).

16Maguire et al. (2010); Henderson, MacAllum, and Karakus (2010); Bloom, Hendra, and Page (2006).

17While this did not prompt the technical assistance, several program enrollees, when asked in “openended” questions in the Year 2 survey about how WorkAdvance could be improved, mentioned a need for 

stronger job placement services, and some expressed frustration that they had completed training and certifications without finding a job in the sector. 
47 

Alliance received extensive, ongoing technical assistance to address low placement numbers 

and strengthen job developers’ skills and placement capacity. There, the goal was to increase

staff capacity, improve internal reporting mechanisms, provide accountability and general advice, and improve the provider’s ability to gather and analyze systematic labor market information. While Per Scholas received technical assistance on other program aspects, it did not 

receive any focused on job development or placement.

• WorkAdvance providers differed in how they staffed the job development role, with one seeking to hire people with sales experience and another contracting with people at local industry associations to play this 

role. Additionally, some providers directly matched program participants and employers, while others recommended multiple candidates to 

employers for any given job.

Notably, when initially hiring staff members, Madison Strategies Group put greater 

emphasis than the other providers did on selecting a job developer with sales experience, plus 

experience in hiring, recruiting, and managing others, as this was seen as important to demonstrate credibility and encourage confidence among employers.18 While Per Scholas, St. Nicks 

Alliance, and Madison Strategies Group had full-time on-site staff members devoted to job development and placement, Towards Employment contracted with local industry associations 

and industry-specific workforce intermediaries in Greater Cleveland to establish relationships 

with both the manufacturing and the health care sectors, and career coaches then handled the 

relationships between particular WorkAdvance participants and employers. The goal was to 

make use of these associations’ existing relationships to allow for more rapid acceptance of 

WorkAdvance participants, and the use of a partnership was part of a strategy to encourage collaboration and the spread of WorkAdvance practices across key workforce partners, for the sake 

of sustainability. For the manufacturing sector, this relationship endured, but in early 2012, the 

health care industry association’s priorities changed and that contract was terminated. In fall 

2012, Towards Employment added a different health care industry association as a partner, but 

that relationship also ended. Towards Employment eventually began working directly with 

large hospitals in Cleveland.

Per Scholas and Madison Strategies Group had strong direct employer relationships and 

conducted individualized job matching between participants and employers. St. Nicks Alliance 

matched individuals to jobs but also identified and recommended multiple candidates for any 

given job. Towards Employment relied on intermediary organizations to match participants 

with job leads. In none of the WorkAdvance sites did providers or participants expect to have a 

job “guaranteed” to those who completed WorkAdvance training.

 18Madison Strategies Group, unlike the other providers, had a performance-based salary structure tied to 

employment outcomes. For job developers with sales experience, this was a familiar way of operating.
48 

• WorkAdvance providers dealt with employers with different characteristics, in line with the sectors they targeted, which may have influenced 

the likelihood of placing program participants into jobs.

While the WorkAdvance study was not set up to systematically analyze the depth, 

breadth, or “quality” of the providers’ relationships with employers, there were some notable 

differences in the employers with whom the providers worked. For example, most employers of 

information technology workers are large, “corporate,” and “professional” in nature; they include technology and telecommunications firms but also banks, universities, hospital systems, 

and other large corporations that have an information technology unit. This may have made job 

placement efforts somewhat more efficient at Per Scholas, in that many people could be placed 

at the same company (although Per Scholas did not work exclusively with large employers). It 

is also important to note that Per Scholas has a long history in the sector and is known by many 

employers (in contrast to some of the other providers who had to develop these relationships).19

Strong relationships between Per Scholas and a number of these large firms may have opened 

doors to opportunities that participants might not have been able to get on their own — assisted 

by the fact that Per Scholas offered a package of training and career readiness services in information technology that was not easily found elsewhere. In the health care sector, the primary 

employers were hospitals; while these were also large, “professional” employers, it is possible 

that Towards Employment was competing with many other training programs focused on health 

care jobs. In addition, hospital jobs are generally more accessible to low-income individuals 

seeking employment in lower-level jobs, compared, for example, with jobs at large financial 

corporations.

Manufacturing and environmental remediation employers (such as those served by St. 

Nicks Alliance and Towards Employment) tended to be smaller to midsize shops, some of 

which were family-run businesses, which had fewer available job openings on an ongoing basis. 

Thus, these sectors required more intensive job development efforts. Given that the smaller employers did not have as many positions, and thus job openings were not as frequent, opportunities probably were not as common for ongoing or long-lasting relationships to develop between 

the WorkAdvance providers and the employers, compared with the situation with large employers. That said, an advantage for providers working with small- to medium-sized employers was 

that these employers often lacked large human resource departments, and thus the employers 

may have more readily perceived a need for WorkAdvance’s services. It may also have been 

 19This chapter does not present statistics on employment, as data from the Year 2 Survey provide a much 

more complete and accurate picture of employment than statistics based on the reporting of jobs to WorkAdvance staff. Using Year 2 Survey data, Chapter 5 details the types of jobs WorkAdvance group members 

found, as well as the extent to which they were employed over an 18-month follow-up period. The analysis in 

Chapter 5 also indicates the extent to which the jobs found by WorkAdvance enrollees were in the sector in 

which they had trained and offered potential for advancement.
49 

easier for WorkAdvance providers to adjust their emphases in response to labor market shifts 

when working with more, and smaller, employers — the closing of a few smaller employers 

would not have required as big a shift for the WorkAdvance provider as the closing of very 

large employers. 

Employers’ reactions to the WorkAdvance model were hard to gauge. See Box 2.1, 

however, for the views expressed in a small number of employer interviews regarding employers’ perceived value of the program.

Postemployment Retention and Advancement Services 

Simply helping workers find employment is often not enough to improve earnings in 

the long run; in many cases, the kinds of jobs people find (generally low-wage ones) do not offer opportunities for hard-skill acquisition, and these skills are often needed in order for earnings 

to increase.20 In light of this, programs that have been studied have experimented with different 

ways to provide services to workers after they obtain employment to help them advance over 

time.21 To help workers figure out the best way to advance — for example, whether and how to 

try to move up at a current employer, apply for a higher-level position at a different employer, 

or obtain additional credentials first — it stands to reason that coaches should provide concrete 

guidance, including setting specific goals, sharing feedback from employers on performance, 

and getting permission from participants to hold them accountable for their commitments. 

Drawing on suggestions from previous studies about what appears to work and what does not to 

help low-wage workers advance, the postemployment services in WorkAdvance were designed 

to include a follow-up plan to contact and communicate with the employee at strategic points 

after the start of employment; updating of the employee’s individual career plan at least twice 

during the first year of employment, to foster a focus on career advancement rather than only 

job retention; and maintenance of regular contact with the employer.

• Postemployment services were the last set of services to be completely 

developed and implemented. Once they were fully in place, the WorkAdvance providers differed in how often the coaching services were offered, in whether incentives were offered to staff members for connecting with program participants, and in their advice regarding how 

quickly or the best ways in which to seek advancement opportunities 

(which differed by sector).

 20Hendra et al. (2011); Gladden and Taber (2009); Card, Michalopoulos, and Robins (2001); Miller, 

Tessler, and van Dok (2009). 21Hamilton and Scrivener (2012b).
50 

Box 2.1

Employer Perspectives on WorkAdvance

As part of the WorkAdvance evaluation, a limited number of interviews were conducted with employers who worked with WorkAdvance providers and had hired WorkAdvance participants. 

These interviews suggested, as illustrated below, that employers particularly valued several aspects of the program: its identification of individuals who showed a commitment to a career in a 

particular field, its emphasis (at one provider) on modeling workplace behavioral norms, its ability and willingness to adapt skills training to employer needs, and its prescreening of potential job 

applicants.

Employers appreciated applicants’ interest in a career, rather than just a job. As one employer in 

northeast Ohio stated, “The message that resonated with me was the fact that Towards Employment had the intention of getting people training who are interested in growing in the manufacturing industry and who are planning on making a career in manufacturing. ... We are always looking for people who want to grow within the industry and are not interested in those who just want 

to get any job.” 

Modeling workplace norms and holding participants accountable to an expected set of behaviors 

was emphasized at one WorkAdvance provider, Per Scholas, and this practice seemed to make a 

good impression on employers. As one employer said, “The school’s always clean, everyone’s 

professional and nice. They get on their students. If anyone is late to class, or not dressing appropriately, they get on them. It’s that that really takes individuals from the community and takes 

them to the next level.”

All WorkAdvance providers adapted their skills training curricula at some point based on labor 

market and employer input. Both Towards Employment and Madison Strategies Group, for example, worked with employers or training providers to abbreviate or expand manufacturing 

courses. Madison Strategies Group added a week to a machining class because employers felt the 

participants needed specific additional skills, and dropped several components from its aviation 

training based on employer feedback that those skills would not be needed. One employer who 

works with Madison Strategies Group noted in an interview that he appreciated that he “could 

help frame the curriculum to be in line with what we are looking for” and that the provider could 

“tailor make the product we need to fill the voids” in positions that they have had difficulty filling. Similarly, one employer that works with Per Scholas noted that “Per Scholas is always eager 

to get feedback to help develop the students’ skill sets and interviewing. ... Positive or negative, 

they always want to hear it.”

Interviewed employers also noted WorkAdvance’s value in prescreening job applicants and referring 

individuals who already had the required certifications. One employer said he needs to spend a week 

training new employees from other recruitment sources, whereas, by working with St. Nicks Alliance, 

“it probably saves us a couple thousand dollars [for] each guy.” Similarly, an employer in Tulsa 

(continued) 
51 

Box 2.1 (continued)

reported that Madison Strategies Group’s services helped his company “reduce time to hire, reduce turnover, troubleshoot with new hires, reduce safety infractions, and fill skilled positions beyond entry level,” the latter of which had the biggest impact on the company. And one employer 

in northeast Ohio reported that, compared with other recruitment sources he used, Towards Employment was more interested in learning exactly what the employer was looking for in a candidate, while other recruitment sources “simply give us anybody” because those other entities “really only care about making money for themselves.”

Several employers did have criticisms. For example, when making hiring decisions, they valued 

“softer” skills, or what one employer called “essential skills” — such as interview skills, good attitudes, and a willingness to work — and they noted that some of the WorkAdvance participants 

they had hired (as well as hires from other sources) had shortcomings in these areas. 

Providers were preoccupied for much of the first year or two of program operations 

with recruiting and enrolling individuals into the study and developing, launching, and adapting 

the other core program components. But to some extent, the delay in fully rolling out the 

postemployment advancement services also reflected, for at least some of the staff, a skepticism 

of the value of the postemployment services and the need to change the culture of their organizations. The WorkAdvance model required staff members, even after program participants 

found employment, to put a priority on continued service delivery and to value an ongoing relationship with the participant — something not typical in workforce programs — and it took 

some time for this perspective to develop among some staff members. Perhaps just as important, it also took time and effort, and substantial technical assistance, for staff members to 

gain knowledge of what types of advancement services to provide and how to provide them. 

Once the postemployment component did take hold across the WorkAdvance sites, 

WorkAdvance staff members at two of the providers normally contacted participants during 

their first month on the job and followed up monthly thereafter. Per Scholas was unusual in that 

it followed up less frequently (quarterly) in the postemployment period, although the coaches’ 

preemployment contact with participants was more frequent than that of the other providers. At 

Madison Strategies Group, staff members contacted participants in their first week of employment and then monthly thereafter. WorkAdvance staff members also initiated contact with employers to see how program graduates were doing on the job. As Table 2.6 indicates, this outreach to employers was frequent at three sites and rare at Per Scholas. 
 

.

52

 

SOURCES: Documentation supplied by providers and interviews with provider staff.

The WorkAdvance Study

Table 2.6 

Postemployment Services, by Provider 

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

1 full-time career coach with 4 

Staffing

additional coaches/career 

readiness training instructors 

(serving WorkAdvance and

non-WorkAdvance Per Scholas

1 senior coach with 2 additional 

career coaches/case managers

2 career coaches 4 career coaches (2 for each 

sector)

Postemployment services

clients)

Initial 

postemployment 

follow-up

Coaches follow up with 

participants during the first 90 

days of employment. Job 

developers follow up with 

employers in the same period.

Job developers follow up with

participants weekly during the 

first 30 days of employment.

Job developer or career coach

follows up with participant and 

employer within the first week 

of employment.

Coaches follow up with 

participants weekly for the

first 30 days after placement.

Ongoing followup

Coaches follow up with 

participants quarterly, either 

over the phone or in person one 

on one. Participants are also 

invited to workshops on-site.

Coaches follow up with 

participants monthly, either 

over the phone, in person one

on one, or in a workshop 

setting.

Coaches follow up with 

participants monthly, either 

over the phone, in person one

on one, or in a workshop 

setting.

Coaches follow up with 

participants bimonthly 

through 90 days, monthly 

until 180 days, and quarterly 

thereafter.
53 

Madison Strategies Group was unique among providers for having staff compensation 

tied to performance benchmarks for WorkAdvance staff members in the area of postemployment services (and, as discussed above, in the area of employment outcomes). There, staff 

members were expected to try to contact a minimum of 60 percent of active clients and at least 

80 percent of their total caseloads, and compensation varied accordingly. Attempts to reach clients were recorded in Madison Strategies Group’s MIS, and the program manager used the MIS 

to monitor performance. The other providers’ program managers also monitored staff efforts to 

contact participants via the providers’ MIS, and set clear expectations for the staff, but there 

were no financial incentives to meet the expectations.

In the first three months that WorkAdvance participants were on the job, WorkAdvance 

staff members across all providers tended to focus on retention issues while still working with 

program enrollees to set them up for advancement in the future. Providers agreed that an appropriate amount of time had to pass after employment began before enrollees would be open to 

thinking beyond retention toward advancement, and before employers would be willing even to 

discuss moving an employee up. After the first three months on the job, the theme of the interactions tended to become more focused on advancement-related topics.

Some aspects of advancement advice given by WorkAdvance staff members varied by 

sector. Generally, providers advised clients to stay on the job for at least six months, though the 

advice they gave regarding the recommended time before looking elsewhere varied by provider

and sector. In the manufacturing and environmental remediation sectors, staff members reported 

that changing jobs in pursuit of advancement is more common or acceptable. In information 

technology and health care, where job hopping is seen by employers as a more notable blemish

on a résumé, WorkAdvance providers recommended that clients stay on the job for at least a 

year. 

After a considerable amount of technical assistance for some providers (and considerable effort by program staff members), all the providers had a much greater emphasis on coaching (defined as working with participants toward their goals by building on their strengths) than 

on case management (defined as focusing on participants’ barriers and the services needed to 

overcome them) — even at providers where staff members were originally trained as case managers. Three of the providers maintained this distinction by typically referring participants to 

other organizations for services to address barriers; meanwhile, the coaches worked on building 

on participants’ strengths and focused on advancement. 

• A special study suggested that advancement coaching occurred fairly 

frequently, and that most WorkAdvance group members who were engaged in coaching meetings were employed. Advancement-related topics 

most commonly discussed included opportunities for advancement at a 
54 

current job or at a new job, and additional education or training opportunities.

To get a clearer sense of how advancement coaching was delivered, MDRC conducted 

a special study that documented the details of advancement advice and suggestions delivered 

during in-person and phone contacts with individual WorkAdvance participants that occurred 

over a specific three-month period. All four WorkAdvance providers documented, from midApril 2014 to mid-July 2014, the mode of advancement coaching, its frequency, and topics discussed during all advancement coaching meetings (Tables 2.7, 2.8, and 2.9). 

The advancement coaching study results suggest that advancement coaching occurred 

for around one-third of WorkAdvance participants during this three-month period (Table 2.7), 

and that the majority (or 76 percent) of enrollees who were engaged in a coaching meeting during this three-month period attended at least one coaching session while employed (Table 2.9). 

As Table 2.8 shows, among all individuals who had a coaching meeting during this period, phone meetings were somewhat more prevalent than in-person meetings, and these phone 

discussions were mostly initiated by WorkAdvance staff members, as opposed to WorkAdvance participants. Staff interviews and observations suggested that advancement-related advice 

was conveyed in other ways as well, including via individual emails, e-mail blasts, LinkedIn 

groups, and, at Per Scholas, in-person group “new hire” workshop sessions. 22

The advancement coaching study also suggested that in coaching interactions, topics related to job retention and advancement were more commonly discussed (at least during the special study period) among employed than among unemployed meeting attendees. The most 

common retention-related topic discussed during the special study period was verifying employment, which was a requirement of the Social Innovation Fund grant that funded WorkAdvance (Table 2.8).23 Advancement-related topics most commonly discussed included jobspecific opportunities for advancement and additional education or training opportunities. Case 

note reviews provided more specific examples of topics discussed during advancement coaching sessions, including (out of a large variety of topics) trying to unearth problems employees 

were possibly having on the job (difficulty handling job responsibilities or tasks, or problems 

with coworkers); encouraging employees to get an employee handbook to learn about employers’ expectations and policies — for example, the number of acceptable absences in the first 90 

 22While Per Scholas had fewer one-on-one meetings between graduates and coaches than the other providers, it engaged graduates in a variety of other ways following employment: It convened alumni events, 

formed affinity groups on LinkedIn and Facebook, held advance-level certification classes for alumni, and sent 

e-mails to graduates about new job opportunities and new certification classes. 23Multiple topics were discussed in most coaching meetings, as is implied by Table 2.8. In very few 

coaching interactions was verifying employment/retention the only topic discussed. 
55 

days, the performance review process, and the pay raise schedule; encouraging employees to get 

information about the responsibilities of different jobs; and practicing how to negotiate for 

higher wages. 

• At three of the providers (not Per Scholas), WorkAdvance staff members had direct discussions with employers about particular employees.

Staff members at three providers engaged “partner” employers — those with whom 

they had a close relationship — in discussions about the advancement of particular participants. 

Examples of this kind of engagement included advocating for higher wages, for example, if a 

participant received an additional certification but the pay was not adjusted accordingly; making 

plans to advance incumbent employees and backfill the positions they would vacate; and intervening to encourage a discussion between an employee and a supervisor to discuss advancement. At Per Scholas and Madison Strategies Group, it was the job developers who talked with 

employers about particular participants. Per Scholas managers indicated that the job developers 

had the relationships with the employers, and they felt it would be “bad for business” for the 

coaches to reach out to the employers, because it could inadvertently send the message that Per 

Scholas graduates had issues that the program staff needed to check up on (and that employers 

should worry about). 

Madison

Per St. Nicks Strategies Towards

Activity (%) Scholas Alliance Group Employment Overall

Attended an advancement coaching meeting 10.9 22.3 51.0 42.7 32.6

Sample size 349 242 353 349 1,293

The WorkAdvance Study

Table 2.7

Percentage of WorkAdvance Group Members Who Attended an Advancement

Coaching Meeting, Cross-Site

SOURCES: MDRC calculations from special data collected on advancement coaching, recorded on paper by 

two sites and in program tracking systems by two sites.

NOTE: The special data on advancement coaching was collected between April 7, 2014, and July 18, 2014. 

Provider staff members documented all phone and in-person advancement coaching meetings with participants 

during that time period.
56 

 

Activity

Per

Scholas

St. Nicks

Alliance

Madison

Strategies

Group

Towards

Employment Overall

Average number of advancement coaching

meetings attendeda

By phone

Initiated by provider staff member

Initiated by participant

In person

Attended a meeting by phone (%)

Attended a meeting in person (%)

Discussed job search-related topic (%)

Updating individual career plan

Updating résumé

Identifying skills and career interests

Job search activities

Interviewing

Discussed retention-related topic (%)

Verifying employment/retention

Acclimating to job

Passing probationary period

Getting an employee handbook/job description

Managing problems with coworkers/supervisors

Backup plans for transportation or child care

Feedback from employer on job performance

Discussed advancement-related topic (%)

Opportunities for advancement at current job

Asking for a raise, promotion, or more hours

Acquiring job benefits

Improving work schedule or shift

Feedback from employer on advancement

opportunities

Opportunities for advancement at new job

Additional education or training opportunities

Other advancement topic

Discussed other topic (%)

Financial goals

Work-life balance

Participant invited to alumni event

Other topic (none of the abovementioned)

1.0

0.6

0.5

0.1

0.3

60.5

34.2

36.8

15.8

18.4

5.3

15.8

5.3

23.7

7.9

15.8

15.8

10.5

2.6

2.6

2.6

68.4

21.1

21.1

5.3

5.3

0.0

15.8

15.8

29.0

42.1

18.4

13.2

18.4

10.5

1.7

1.0

0.8

0.2

0.7

74.1

55.6

48.2

11.1

7.4

3.7

37.0

0.0

42.6

42.6

5.6

0.0

0.0

0.0

0.0

0.0

25.9

9.3

3.7

1.9

3.7

0.0

13.0

9.3

7.4

46.3

1.9

0.0

35.2

13.0

1.8

1.0

0.7

0.3

0.8

72.8

50.0

48.3

2.8

20.6

21.1

41.1

7.8

63.3

58.3

17.2

8.9

5.0

7.2

4.4

2.2

46.7

13.3

3.9

2.8

6.1

0.0

12.8

18.3

7.2

26.1

0.0

0.0

0.0

26.1

2.9

2.2

1.4

0.9

0.7

92.0

38.3

64.4

28.2

28.2

35.6

47.7

8.7

52.4

42.3

16.8

11.4

4.0

9.4

10.1

9.4

65.1

32.2

7.4

8.1

18.1

4.7

37.6

26.9

22.8

72.5

9.4

28.9

7.4

63.8

2.1

1.4

1.0

0.5

0.7

78.6

45.1

53.0

14.0

21.4

22.6

40.6

6.9

53.2

46.1

15.4

9.3

4.5

6.7

5.7

4.5

52.5

20.2

6.7

4.8

10.0

1.7

21.9

20.0

14.7

46.6

5.2

11.4

8.8

36.3

Sample size 38 54 180 149 421

(continued)

The WorkAdvance Study

Table 2.8

Mode and Content of Advancement Coaching Meetings for WorkAdvance Group

Members Who Attended a Meeting, Cross-Site
57 

Table 2.8 (continued)

SOURCES: MDRC calculations from special data collected on advancement coaching, recorded on paper by two 

sites and in program tracking systems by two sites.

NOTES: The special data on advancement coaching was collected between April 7, 2014, and July 18, 2014. 

Provider staff members documented all phone and in-person advancement coaching meetings with participants 

during that time period.

WorkAdvance group members may attend more than one meeting during the data collection period. 

Some statistics may not add to 100 percent due to missing values. 

a

Only one meeting per person and day is counted. Two participants appear to have met with staff members 

twice on the same day. 

WorkAdvance staff members reported that the ways they saw WorkAdvance participants advancing most quickly or commonly often differed by sector, but staff members held 

different opinions, even regarding the same sector, about whether changing jobs or advocating 

for better pay at one’s current employer was the most expeditious route to advancement. At the 

same time, they noted a number of services that they provided as being effective ways to help 

people advance. These included motivational coaching, teaching negotiation tactics, instructing 

participants to log their on-the-job accomplishments to formally demonstrate their value to their 

employer, and providing general encouragement and moral support.

When asked if some types of WorkAdvance participants were more likely to advance 

than others, WorkAdvance staff members emphasized the role of perceived attitudes. As examples, they indicated that WorkAdvance participants who took ownership of their goals, believed 

in themselves, were committed to attending training, and communicated more frequently with 

staff members were more likely to advance in their jobs. In contrast, WorkAdvance participants 

who were either overconfident or lacked self-confidence were viewed as likely to have trouble 

advancing. In addition, staff members thought that participants with mental health problems or 

criminal backgrounds might have insurmountable challenges to advancing.

Changes in Components and Participation Patterns over Time

• WorkAdvance providers adjusted the delivery of various program components over time, as staff members learned what seemed to work best 

in practice. 

The WorkAdvance programs did, in general, mature over time — that is, the providers 

changed the delivery of some components as they gained operational experience, just as they

adjusted the content of their career readiness and occupational skills training classes. For example, Per Scholas and St. Nicks Alliance initially did not have designated career coaches to lead 
58 

pre- and postemployment advancement coaching; preemployment coaching fell primarily to the 

career readiness instructors. After about a year of program operations, both providers hired a 

designated career coach to deliver pre- and postemployment advancement coaching. Then, in 

fall 2013, Per Scholas broadened the roles of its career readiness instructors to include career 

coaching with a focus on advancement. As another example, Towards Employment initially had 

participants start their interaction with the program by meeting with a case manager, who conducted a needs assessment focused on participants’ barriers to employment; the organization

eventually began to use a different kind of assessment tool that emphasized goal-setting rather 

than barriers. 

The WorkAdvance Study

Table 2.9

Content of Advancement Coaching Meetings for WorkAdvance Group Members

Who Attended a Meeting While Employed, Cross-Site

Activity (%)

Per

Scholas

St. Nicks

Alliance

Madison

Strategies

Group

Towards

Employment Overall

Among WorkAdvance group members who attended a meeting

Ever engaged in an advancement coaching meeting 

while employed 68.4

Average number of advancement coaching meetings

that took place while the participant was:a

Employed 0.7

Not employed 0.1

Sample size 38

70.4

1.1

0.6

54

79.4

1.3

0.5

180

75.8

1.8

1.1

149

76.0

1.4

0.7

421

Among WorkAdvance group members employed at time of meeting

Discussed job search-related topic 38.5

Discussed retention-related topic 34.6

Discussed advancement-related topic 65.4

Discussed other topic 50.0

Sample size 26

34.2

60.5

36.8

52.6

38

40.6

74.8

55.2

26.6

143

59.3

68.1

79.7

69.0

113

46.3

67.5

62.5

46.6

320

SOURCES: MDRC calculations from special data collected on advancement coaching, recorded on paper by two 

sites and in program tracking systems by two sites.

NOTES: The special data on advancement coaching was collected between April 7, 2014, and July 18, 2014. 

Provider staff members documented all phone and in-person advancement coaching meetings with participants 

during that time period.

WorkAdvance group members may attend more than one meeting during the data collection period. 

Some statistics may not add to 100 percent due to missing values. 

aOnly one meeting per person and day is counted. Two participants appear to have met with staff members 

twice on the same day. 
59 

• A “mature” and “fully implemented” WorkAdvance program was more 

likely to have been experienced by individuals who enrolled in the program later in the random assignment period (the “late cohort”) than by 

individuals who enrolled in the program earlier (the “early cohort”).

Notably, many WorkAdvance group members had already made their way through a 

significant part of the WorkAdvance model before mid-2013. As a result, coaching truly focused on advancement was more likely to have been experienced by individuals who enrolled 

in WorkAdvance in the late cohort than by individuals who enrolled earlier.

While theoretically maturation might also be indicated by changes in participation patterns for later program enrollees compared with early program enrollees — if program staff

members were better able to engage and support participants in their activities as the staff 

learned, through practice, what techniques work best operationally — in reality participation 

patterns did not differ much between early and late enrollees within each site. As Table 2.10

shows, skills training participation and completion rates were high for both the early and the late

cohorts at Per Scholas and St. Nicks Alliance (providers that consistently operated “training 

first” WorkAdvance models). Rates of starting and completing skills training did change from 

the early to the late cohort at Madison Strategies Group and Towards Employment, but this 

change primarily reflects those providers’ initial two-track approach to WorkAdvance, which 

allowed some participants to be placed directly into jobs rather than engaging in training first. 

Once those providers began to direct more participants to training before employment, their 

training start and completion rates became higher and much closer to those of the other two 

providers. This suggests that program “maturation” was probably evident more in the quality of 

the services provided or in the smoothness of program operations than in the likelihood of individuals receiving specific services.

Conclusion

Key differences in implementation across the WorkAdvance providers were apparent and centered on a number of factors. First, the providers had different starting points. Per Scholas had a 

sizable head start, in its delivery of sector training in information technology for more than 10 

years before WorkAdvance and in its previous participation in a random assignment research 

study. Second, Towards Employment’s operational complexity — stemming from its administrative structure, which required collaboration across a number of institutions to deliver services 

in two completely distinctive sector programs, one of which was in a sector relatively new to it 

— set it apart from the other providers. Third, two sites (Madison Strategies Group and Towards Employment) initially operated a dual-track approach to WorkAdvance, which resulted 

in early enrollees at those two sites being less likely than enrollees at the other two sites to ever 
60 

The WorkAdvance Study

Table 2.10

Indicators of Participation in Program Group Activities 

Within 18 Months After Random Assignment, 

by Random Assignment Cohort, Cross-Site

Per St. Nicks Madison Towards

Participation in program activity (%) Scholas Alliance Strategies Group Employment Overall

Among enrollees in the early cohort a Ever participated in any career readiness activity 95.2 83.5 99.4 98.8 95.0

Ever started skills training 95.2 83.5 51.5 53.6 70.8

Ever completed skills training 79.4 79.5 39.3 42.3 59.4

b Ever obtained a credential in targeted sector 70.9 76.4 38.7 26.8c 52.2

Sample size 189 127 173 168 657

Among enrollees in the late cohort a Ever participated in any career readiness activity 96.3 81.7 97.2 97.8 94.3

Ever started skills training 96.3 81.7 72.2 76.2 81.1

Ever completed skills training 77.5 78.3 57.8 57.5 66.4

b Ever obtained a credential in targeted sector 74.4 70.4 58.3 36.5c 58.3

Sample size 160 115 180 181 636

SOURCES: Program tracking systems managed by Per Scholas (PS), St. Nicks Alliance (SNA), Madison 

Strategies Group (MSG), and Towards Employment (TE).

NOTES: The early cohort includes all sample members randomly assigned through Quarter 3, 2012. The late 

cohort includes all sample members randomly assigned in or after Quarter 4, 2012.

Refer to Appendix Table C.3 for site-specific details regarding how each metric is defined. aThe first program activity at PS and SNA is skills training, which is offered in combination with career 

readiness training and includes help with developing a career plan, résumés, and interview preparation. At MSG, 

the first progam activity is career readiness training. At TE, the first program activity is an initial assessment 

wherein career goals and barriers to employment are discussed. 

bCredentials in the targeted sector are recognized locally or nationally, or both. There is cross-site variation in 

reporting of locally recognized credentials obtained in the targeted sector. cTE’s program tracking system captures only nationally recognized credentials. Therefore, participants who 

obtained the locally recognized computer numerical control (CNC) machining credential are not counted as ever 

having obtained a credential.
61 

start and complete occupational skills training. Fourth, the providers differed in their delivery of 

career readiness and postemployment services. Per Scholas, by virtue of having all its training 

on-site and therefore having more access to program participants, delivered more frequent career readiness services, and Madison Strategies Group took an innovative incentive approach to 

the delivery of postemployment services. Fifth, Per Scholas took an approach to postemployment services that did not quite follow the model: One-on-one, individualized postemployment 

career advancement coaching was less frequent than called for by the model and received less 

emphasis than did group activities focused on advancement, such as the advanced certification 

classes for alumni, which were starting in earnest around the time that field research for the 

evaluation was wrapping up. Finally, the nature of the different sectors targeted by each provider influenced the providers’ emphasis on how participants could best advance. In the information technology and health care sectors, more emphasis was generally placed on working in a 

professional environment and staying on the job longer; in the manufacturing and environmental remediation sectors, more emphasis was placed on refining skills on the job and taking on 

additional responsibilities that could be made use of during subsequent job searches or employer negotiations. 

Most of this chapter examined the implementation of WorkAdvance in light of the degree to which the program’s components were advancement focused and employer driven. Implementation of the WorkAdvance model also can be examined in light of similarities to or differences from the features of the programs in the Sectoral Employment Impact Study (SEIS). 

The researchers in that study suggested that the features that made those programs successful 

were “a good understanding of and connection to industry needs, careful screening to identify 

appropriate clients, a sector-focused approach to training, individualized support services and 

the organizational capacity to put all of these ingredients together.”24 Many of these features 

were exhibited by the WorkAdvance providers, though not all of them and not equally across 

the providers. Most important, perhaps, is that the organizational capacities of the provider organizations varied — from each other, and also from those in the SEIS. To be selected for the 

SEIS, programs had to be operating for at least three years and were asked to simply continue 

running the programs they already were operating, as opposed to implementing features of a 

specific model. A key difference between the two sets of programs, therefore — with the exception of Per Scholas — was their experience operating sector programs and the degree to which 

they had deep connections with employers and intermediaries in the sector. These connections 

take time to develop, even for an organization with experience in the sector if it establishes itself 

in a new location, such as Madison Strategies Group did in Tulsa. 

The next chapter continues a discussion of participation in WorkAdvance but focuses 

on the treatment differential between the WorkAdvance group members and control group 

 24Maguire et al. (2010), p. 48.
62 

members. Information on participation in the next chapter also comes from a different data 

source — it is based on survey data, which captures training and other activities that occurred 

within WorkAdvance, as well as outside of WorkAdvance, and covers a slightly longer followup period.
63

Chapter 3

WorkAdvance-Control Group Differences in

Services Received

Chapter Highlights

• Across all sites, WorkAdvance had high participation in services, producing 

large impacts in all the key service domains despite high control group service receipt levels. 

• WorkAdvance participants completed training and earned credentials at a 

much higher rate than control group members. 

• Impacts were particularly large for postemployment services — from about 

40 to 70 percentage points, depending on the site. 

Introduction

The previous chapter described the implementation of the WorkAdvance program by each of 

the providers and indicated the percentage of WorkAdvance group members who, according to 

the programs’ management information system (MIS) data, participated in WorkAdvance services within the first 18 months of enrollment. The data shown in the previous chapter indicate 

that participation in employment services and training among WorkAdvance group members 

was high, but how much higher was it relative to participation among control group members? 

This chapter focuses on the differences in services received by the WorkAdvance group and the 

control group at each of the four sites. Using data from the Year 2 Survey, which was administered to both groups, this chapter examines the extent to which WorkAdvance increased participation, compared with what would have happened in the absence of the program, in the following types of activities: 

• career advice and advancement help

• education and training

• job search assistance

• postemployment services
64

The Year 2 Survey measures participation over the first 18 to 30 months after random 

assignment. It asked respondents whether they participated in employment-related services and 

education and training activities and, if so, the duration, type, and provider of the service.1

 

Comparing the Year 2 Survey responses regarding participation by the WorkAdvance 

group members and by the control group members reveals WorkAdvance’s estimated “impacts” 

on service receipt and training participation. (Box 3.1 explains how to read the impact exhibits 

in this report.) In other words, these results measure the “treatment contrast,” that is, the differences in service receipt that were produced by WorkAdvance. The underlying hypothesis of the 

WorkAdvance model is that helping low-income adults improve both their career readiness and 

their technical skills, through sector-focused training and close relationships with employers,

will substantially enhance their opportunities for long-term employment. Together with the 

findings from Chapter 2, which discussed the extent to which WorkAdvance services were focused on advancement and reflective of employer needs, knowledge of the results presented in 

this chapter is central to understanding the employment and other economic impacts of WorkAdvance that are presented in Chapter 5. 

Following a summary of WorkAdvance’s participation impacts, the chapter discusses 

findings for each of the abovementioned service areas in the order of the WorkAdvance program flow. Thereafter, participation impacts by time of enrollment are presented to examine 

whether programmatic changes implemented during and after the fall of 2012 (discussed in 

Chapter 2) made a difference in the size of the participation impacts. Finally, the chapter ends 

with key points about variation in these impacts by site and time of enrollment. 

Summary of Participation Impacts

WorkAdvance produced substantial increases in training and employment services at all four 

sites. This is illustrated in Figure 3.1, which presents program and control group rates of participation in the core services that the WorkAdvance providers were expected to deliver. These 

impacts are noteworthy given the large proportions of individuals who would have participated 

in similar services without WorkAdvance — as indicated by the proportions of control group 

members who reported engaging in training and employment services. (Box 3.2 presents more 

information on the services received by control group members.)

2

 1

As discussed in detail in Appendix A, the overall response rate for the Year 2 Survey was about 80 percent. This rate did not differ significantly between WorkAdvance group members and control group members. 2

To help determine, near the start of the WorkAdvance evaluation, whether there was a sufficient difference between the services received by the program group and by the control group, a Control Services Survey 

(CSS) was fielded to a very small sample of early enrollees about six months after they entered the study. 

While statistically significant differences between the groups in the take-up of employment- and trainingrelated services were found in this early check, the CSS clearly showed that some control group members were 
65

 

finding their way to a variety of local services that were similar to those the WorkAdvance group was to receive. Using data from the Year 2 Survey, Box 3.2 describes where control group members found these services — reports that were consistent with those made by control group members in the CSS. 

Box 3.1

How to Read the Tables in This Report 

Most tables in this report use a similar format, illustrated below. The table shows key participation outcomes for the WorkAdvance and control group Year 2 Survey respondents at 

the Per Scholas site — whether they obtained an employment service and what type. (The 

survey question was designed to capture both WorkAdvance services for program group 

members and services from any employment counselors or programs that might be available to members of either research group.) For example, the table shows that 93 percent of 

the site’s WorkAdvance respondents reported obtaining an employment service, compared 

with 65 percent of control group respondents. 

Because study participants were assigned randomly to either the WorkAdvance group or 

the control group, the effects of WorkAdvance can be estimated by the difference in outcomes between the two groups. The “Difference” column in the table shows the WorkAdvance group’s rate of service receipt minus the control group’s rate — that is, the program’s impact on service receipt at that site. For example, the impact on obtaining an employment service is calculated by subtracting 65 from 93, yielding 28 percentage points. 

Differences marked with asterisks are statistically significant, meaning that it is quite unlikely that the differences arose by chance. The number of asterisks indicates whether the 

impact is statistically significant at the 1 percent, 5 percent, or 10 percent level (the lower 

the level, the less likely that the impact is due to chance). For example, as shown below, 

WorkAdvance had a statistically significant impact of 28 percentage points on the proportion of participants who reported obtaining an employment service. Three asterisks indicate that this impact is statistically significant at the 1 percent level. 

Summary of Year 2 Impacts on Participation in 

Employment Services and Training, Per Scholas

Difference

Outcome (%) WA C (Impact)

Obtained an employment service 93.3 65.4 27.9 ***

 Career readiness service 89.2 60.3 28.9 ***

 Job search service 81.7 36.2 45.5 ***

 Postemployment service 64.3 25.5 38.9 ***

NOTE: WA = WorkAdvance group; C = control group.
(continued)

The WorkAdvance Study

Figure 3.1

Summary of Year 2 Impacts on Participation in Employment Services and Training, by Site

81.7

75.1 76.8 72.5

36.2 44.8

24.9

33.4

0

20

40

60

80

100

***

Per Scholas

***

St. Nicks

Alliance

***

Madison

Strategies Group

***

Towards

Employment

Job search servicesb

WA group Control group

Percentage

89.2 90.3 90.1 87.2

60.3 64.2

43.8

52.2

0

20

40

60

80

100

***

Per Scholas

***

St. Nicks

Alliance

***

Madison

Strategies Group

***

Towards

Employment

Percentage

Career advice and advancement servicesa

WA group Control group

78.4 75.1

66.8 69.9

46.8 45.2

30.0

38.5

0

20

40

60

80

100

***

Per Scholas

***

St. Nicks

Alliance

***

Madison

Strategies Group

***

Towards

Employment

Started skills training

WA group Control group

Percentage

64.3 69.2

85.8

68.8

25.5 23.6

14.3

23.3

0

20

40

60

80

100

***

Per Scholas

***

St. Nicks

Alliance

***

Madison

Strategies Group

***

Towards

Employment

Postemployment servicesc

WA group Control group

Percentage

66
67

Figure 3.1 (continued)

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 

percent.

Sample sizes may vary because of missing values.

a

This measure includes career planning, job readiness, and services related to keeping or 

advancing on a job.

bThis measure includes finding a specific job lead in the target field, finding a specific job 

opportunity to apply for, and scheduling a job interview. c

Postemployment services include those received while an individual is employed: Respondent 

obtained help keeping current or most recent job, or employment service providers contacted the 

employer about how the respondent could keep the job or advance. 

Box 3.2

To What Extent and Where Did Control Group Members Receive 

WorkAdvance-Like Services?

How many control group respondents received services that were the same as or similar to 

the ones that WorkAdvance providers offered to WorkAdvance group members? And who 

offered these services? The table below helps to answer these questions. It provides siteby-site control group participation rates in employment and training services by provider. 

Overall, between 64 percent and 76 percent of control group members reported in the 

Year 2 Survey that they received employment and occupational skills training services in 

their communities. 

• Between 51 percent and 68 percent of control group members, depending on the site, 

received an “employment service”: that is, help preparing for a career, help searching 

for a job, or postemployment assistance, from an employment or government agency 

or a community-based organization.

• Between 13 percent and 21 percent of control group members participated in occupational skills training in the targeted sector. Though control group members found their 

way to a broad range of training providers in the targeted sectors, community colleges 

and for-profit training providers were the most common ones. At each site, the most 

common types of training providers used by control group members were roughly the 

same for occupational skills training in and outside the targeted sector. 

• A special analysis of verbatim responses to a survey question about the type of occupational skills training programs that control group members attended showed some 

site-level variation. The most common nontargeted sectors in which control group 

members sought skills training were health care, transportation, and security (Per 

Scholas); health care and transportation (St. Nicks Alliance); health care and security 

(Madison Strategies Group); and customer service, cosmetology, and hospitality (Towards Employment).

(continued) 
68

Box 3.2 (continued)

Control Group Participation Rates in Employment Services and 

Occupational Skills Training, by Site

Outcome (%)

Per

 Scholas

St. Nicks

Alliance

Madison

Strategies Towards

Group Employment

Ever received any employment service a or participated in skills training

Ever received any employment service through an

employment or government agency or CBOb

Ever participated in skills training

In targeted sector

Community/2-year/technical college

For-profit training provider

High school/community school

One-Stop Career Center/government agency c Other

d In other sector

Community/2-year/technical college

For-profit training provider

High school/community school

One-Stop Career Center/government agency c Other

76.2

65.4

46.8

20.3

6.7

1.1

7.0

2.8

3.4

18.0

5.6

2.0

4.2

1.6

5.5

76.1

67.8

45.2

21.3

2.9

0.5

9.6

3.6

4.8

18.7

5.7

0.0

7.0

0.6

8.5

63.9

50.7

30.0

13.3

9.1

0.0

2.3

0.0

2.1

12.3

7.4

0.0

1.7

0.4

2.8

68.5

58.7

38.5

15.3

8.4

0.0

3.9

0.8

2.9

13.5

5.2

0.7

1.8

1.2

5.9

Sample size (total = 983) 265 179 263 276

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: Sample sizes may vary because of missing values. Respondents were allowed to report

on up to five occupational skills training programs. aThis outcome includes job-specific skills training, job search assistance, career readiness 

assistance, and postemployment assistance. 

bRespondents were asked to report on employment services received from an employment 

agency, government agency, or community-based organization (CBO). They were not 

asked to specify where the service was provided. cOther occupational skills training providers include nonprofit organizations, four-year 

colleges, the respondent's workplace, and nonspecified providers. 

dThis outcome includes participation in any occupational skills training programs geared 

toward occupations in sectors other than those targeted by the WorkAdvance provider. 
69

The fact that so many control group members found and participated in these services 

indicates that WorkAdvance had a high bar to meet and exceed, in terms of delivering “additional” employment and training services to WorkAdvance group members.

• Across all sites, WorkAdvance resulted in very large increases in participation in every category of services, compared with what would have 

happened in the absence of the program.

Figure 3.1 shows that despite the high control group participation levels, WorkAdvance 

produced large increases in service receipt. The estimated impacts on the receipt of career readiness services ranged from 26 percentage points at St. Nicks Alliance to 46 percentage points at 

Madison Strategies Group. Similarly, the increases in the receipt of job search services ranged 

from 31 percentage points at St. Nicks Alliance to 52 percentage points at Madison Strategies 

Group. WorkAdvance also increased the receipt of postemployment services, which was the 

last of all core service components to be fully developed, by roughly 40 percentage points or 

more at every site. Notably, Madison Strategies Group yielded a 72 percentage point increase in 

receipt of this type of service, with a take-up rate of 14 percent for the control group and an impressive 86 percent for the WorkAdvance group. 

WorkAdvance increased participation in occupational skills training (vocational training) by 30 percentage points or more at every site, and increased the occupational skills training 

completion rate by 27 percentage points or more. (Impacts are shown in the following sections.) 

The impacts on participating in training in the sectors targeted by the WorkAdvance programs 

are especially pronounced, ranging from 38 to 49 percentage points by site. 

The sections that follow review site-by-site impacts on participation and related outcomes, by WorkAdvance model component. The findings demonstrate that the WorkAdvance 

model received a good test, in terms of providing a much higher amount of employment- and 

training-related help than would have been received in the absence of the program. 

Impacts on Participation in Career Readiness Services, by Site 

As discussed in Chapter 2, as well as in the previous WorkAdvance report,3 the career advice 

and advancement component, referred to as “career readiness,” covers a wide range of services, 

including career planning, job readiness, and preemployment job retention and advancementrelated help. 

• WorkAdvance group members at all four sites were much more likely 

than control group members to report receiving career readiness ser-

 3

Tessler et al. (2014).
70

vices, with the impacts ranging from about 29 percentage points for Per 

Scholas to 46 percentage points at Madison Strategies Group. 

Table 3.1 captures the effects of WorkAdvance on the receipt of career readiness help. 

The Year 2 Survey included multiple questions about the nature of career readiness help that 

study participants received from staff members of various types of organizations — employment, government, and community-based — regardless of whether this help was offered as part 

of WorkAdvance. While the survey did not inquire about the frequency or quality of the career 

readiness help received, respondents were asked to assess the overall importance of any type of 

employment service they received. 

Developing a career plan is one of the initial activities built into the WorkAdvance 

model, and each of the providers made it a priority to offer this help in the earliest stage of service delivery. Therefore, it is not surprising that WorkAdvance group members were much 

more likely than control group members to report that they received help in making a career 

plan. Impacts in reporting receiving such help ranged from 34 percentage points for St. Nicks 

Alliance to 55 percentage points for Madison Strategies Group. The relatively low incidence of 

receiving help in this area among control group members indicates that this population does not 

ordinarily receive these types of services in the community at large; this is especially true at the

Madison Strategies Group site, where only 11 percent of control group members reported receiving help making a career plan.

Similarly, as described in the previous chapter, job readiness training was also offered 

in the early phase of WorkAdvance service delivery, and therefore participation levels for the 

WorkAdvance group were expected to be high. Between 70 percent (for Towards Employment) 

and 82 percent (for Per Scholas) of WorkAdvance group members reported receiving help obtaining training on how to be a good or better employee. These rates are much higher than the 

respective control group rates, yielding statistically significant increases in the receipt of this 

type of help, ranging from 39 percentage points at St. Nicks Alliance (where overall participation levels in career readiness activities among WorkAdvance group members were somewhat 

lower than at the other providers) to 61 percentage points at Madison Strategies Group (which 

was the only provider to enroll WorkAdvance group members in this type of training immediately following entry into the program). 

• Considering the three types of career readiness services measured via 

the Year 2 Survey — help with career planning, help with job readiness, 

and help keeping or advancing in a job — WorkAdvance produced the 

largest increases in providing individuals with help keeping or advancing in a job (Table 3.1). 
71

The WorkAdvance Study

Table 3.1

Year 2 Impacts on Participation in Career Readiness Services, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome (%) WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

a Received a career readiness service 89.2 60.3

Employment services in which respondent participated

b Career planning 76.0 44.9

Finding a suitable career 61.0 33.4

Making a career plan 60.7 26.6

Finding skills training 66.2 35.5

c Job readiness 88.0 53.7

Preparing a résumé 79.5 39.2

Completing a job application 49.2 20.7

Appearing professional 78.0 33.3

Enrolling in job readiness training 82.2 29.9

Getting a job recommendation 65.8 27.4

Practicing for a job interview 80.5 29.4

Keeping or advancing on a jobd 78.8 30.2

Advice on keeping a job 75.1 24.4

Managing problems with 

coworkers 71.8 26.1

28.9

31.1

27.6

34.0

30.7

34.3

40.2

28.5

44.8

52.3

38.4

51.1

48.6

50.7

45.7

***

***

***

***

***

***

***

***

***

***

***

***

***

***

***

90.3

75.6

51.9

58.1

62.2

84.0

67.8

47.9

65.7

71.8

57.5

70.3

68.5

62.6

62.1

64.2

48.4

27.4

24.3

42.2

55.2

45.7

19.7

31.9

32.5

22.9

30.5

34.3

28.6

27.7

26.2

27.2

24.5

33.8

20.0

28.9

22.1

28.2

33.8

39.2

34.6

39.7

34.2

34.0

34.3

***

***

***

***

***

***

***

***

***

***

***

***

***

***

***

90.1

81.6

61.5

66.2

69.7

86.7

78.1

57.7

76.1

76.7

53.9

76.6

72.3

67.5

68.2

43.8

28.1

18.3

11.4

18.2

38.0

28.6

11.4

16.1

15.7

13.6

11.3

15.9

13.5

12.8

46.3

53.5

43.3

54.9

51.5

48.8

49.5

46.3

60.0

61.1

40.3

65.3

56.5

54.0

55.5

***

***

***

***

***

***

***

***

***

***

***

***

***

***

***

87.2

76.0

61.5

65.3

68.9

83.5

73.9

53.9

71.5

70.1

55.7

70.0

70.7

65.6

66.8

52.2

37.0

26.2

20.5

26.5

43.4

34.8

19.5

27.8

26.2

20.6

27.6

29.0

23.6

26.0

35.0

39.0

35.3

44.8

42.4

40.1

39.1

34.4

43.7

43.9

35.0

42.5

41.6

42.0

40.9

***

***

***

***

***

***

***

***

***

***

***

***

***

***

***

(continued)
72

Table 3.1 (continued)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome (%) WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Respondent employed since RA and discussed:

Passing probation at new job 28.3

Obtaining an employee handbook 27.3

Understanding job duties 41.1

Understanding how to advance 42.2

Making child care or transportation

plans 21.9

Respondent said contact with staff is

important for making progress

toward career goals 75.2

7.0

9.7

15.5

13.8

7.7

43.9

21.3 ***

17.6 ***

25.6 ***

28.4 ***

14.1 ***

31.3 ***

21.7

25.0

34.7

33.2

27.9

80.7

7.8

12.9

12.8

15.2

9.0

54.8

13.9

12.1

21.8

18.0

18.9

25.9

***

***

***

***

***

***

32.9

43.2

50.7

51.6

37.9

84.2

4.7

10.0

12.8

10.3

5.3

44.9

28.3

33.2

37.9

41.2

32.7

39.3

***

***

***

***

***

***

26.5

30.0

39.7

41.2

40.1

82.4

6.6

10.7

12.8

14.2

12.4

59.0

19.9

19.3

26.9

27.0

27.7

23.4

***

***

***

***

***

***

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group; RA = random assignment.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Sample sizes may vary because of missing values.

a

Category includes career planning, job readiness, and services related to keeping or advancing on a job (as defined in footnotes b through d).

bCategory includes identifying skills and interest in a suitable career, making a career plan, and advice on getting training. cCategory includes preparing a résumé, filling out a job application, help presenting oneself professionally, obtaining training on how to be a good or 

employee, getting a recommendation, practicing for a job interview.

dCategory includes services related to keeping or advancing in a job that might be received before a job is actually obtained.

better 
73

The impacts of WorkAdvance on help with keeping or advancing in a job ranged from 

34 percentage points (St. Nicks Alliance) to 57 percentage points (Madison Strategies Group). 

Control group rates of help received in this employment service area are similar across sites, 

except at the Madison Strategies Group site, where control group receipt rates were lower. The 

size of these increases is driven mostly by high receipt rates among WorkAdvance group members, as opposed to low receipt rates among control group members. For example, the proportion of WorkAdvance group respondents reporting that they received help with keeping or advancing in a job is notably high at Per Scholas, where approximately 79 percent reported that 

they obtained either advice on keeping a job or help managing problems with coworkers. One 

factor that may help explain the high receipt rate at this site is that these topics were covered at 

the same time that Per Scholas provided its occupational skills training, as discussed in Chapter 

2. WorkAdvance also produced large and positive impacts, across all sites, on the receipt of 

other advancement-related help, especially with regard to having discussions with staff members about how to advance in a job.

WorkAdvance also appeared to increase people’s perception that it is important to get 

help with career readiness. At all sites there were very large impacts on the proportion of sample 

members who said that contact with staff members was important to making progress on career 

goals, especially at Per Scholas and Madison Strategies Group, where increases above the control group exceeded 30 percentage points. As noted in Chapter 2, these two providers had strong 

employer relationships and conducted individualized job matching between participants and 

employers.

Impacts on Participation in Education and Training, by Site 

The impacts of WorkAdvance on education and training can be examined from multiple angles. 

First, Table 3.2 presents the impacts of WorkAdvance on participation in any type of education 

and training. This includes measures of take-up, persistence, length, type, completion, and credential acquisition. Next, Figure 3.2 illustrates the percentage of WorkAdvance and control 

group members who participated in any occupational skills training program in each month following random assignment. Finally, Table 3.3 focuses on the effects of the program on participation in a primary occupational skills training program. This primary training table also covers 

any differences between the two groups in how sample members assessed these training programs and in the amounts they paid for the training. 

• WorkAdvance significantly increased participation in education and 

training, most notably in the form of occupational skills training, at all 

four sites (Table 3.2).
The WorkAdvance Study

Table 3.2

Year 2 Impacts on Participation in Education and Training, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome (%) WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Participated in education or training a Occupational skills training

In targeted sector

College courses 

ABE/GED/HS diploma preparation

Short-term classes (1-5 days)

Completed any education or training 

Obtained a degree or credentialb

Average number of skills training

programs started 

Completed a skills training

program

Dropped out of a skills training

program

Average number of months in skills c training

Among those who started

Obtained a credential 

In targeted sector

84.4

78.4

69.4

27.5

2.7

35.0

66.9

66.2

1.1

62.0

7.9

4.3

5.9

58.1

54.1

63.0

46.8

20.3

28.7

2.0

23.9

36.1

30.1

0.5

25.3

5.1

2.1

5.9

19.0

7.9

21.4 ***

31.6 ***

49.1 ***

-1.2

0.8

11.1 ***

30.8 ***

36.1 ***

0.6 ***

36.8 ***

2.8

2.2 ***

0.0

39.1 ***

46.1 ***

81.2

75.1

60.0

14.7

7.3

36.5

63.7

62.1

1.0

59.9

6.3

3.2

4.9

55.5

48.3

60.6

45.2

21.3

15.8

6.2

27.7

42.7

37.8

0.6

32.5

1.3

1.8

5.4

26.0

17.6

20.6 ***

29.9 ***

38.7 ***

-1.0

1.0

8.7 *

21.0 ***

24.3 ***

0.4 ***

27.4 ***

5.0 **

1.4 ***

-0.5

29.5 ***

30.7 ***

76.1

66.8

55.3

17.6

5.4

30.3

47.8

49.4

0.8

43.0

9.6

3.7

6.2

36.3

32.1

49.5

30.0

13.3

18.9

5.3

24.9

25.2

21.6

0.3

16.4

5.2

1.0

4.4

13.7

6.9

26.6 ***

36.8 ***

42.1 ***

-1.3

0.1

5.4

22.6 ***

27.8 ***

0.4 ***

26.6 ***

4.4 *

2.6 ***

1.7

22.6 ***

25.2 ***

80.1

69.9

61.1

23.2

5.3

42.5

55.2

51.2

1.0

49.6

9.0

4.1

6.6

39.9

36.5

54.5

38.5

15.3

22.6

7.5

25.1

28.8

25.5

0.4

18.8

3.1

1.5

5.6

15.0

9.6

25.6 ***

31.4 ***

45.8 ***

0.6

-2.2

17.5 ***

26.4 ***

25.7 ***

0.6 ***

30.9 ***

5.9 ***

2.6 ***

1.0

24.9 ***

26.9 ***

(continued)

74
Table 3.2 (continued)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome (%) WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Occupational skills training provider 

WorkAdvance provider

Community/2-year/technical college

4-year college

For-profit training provider

HS/community school

One-Stop Career Center/unemployment

office/government agency

Nonprofit organization/church

Workplace

Other

68.0

4.3

0.3

1.7

0.4

0.7

0.9

1.7

2.4

0.0

12.1

3.1

10.6

3.4

0.4

4.4

1.5

3.5

68.3 ***

-7.8 ***

-2.8 **

-8.9 ***

-3.0 **

0.3

-3.5 **

0.2

-1.1

54.6

6.8

0.1

7.8

0.9

3.3

3.2

3.0

4.4

1.7

9.2

0.5

15.4

1.8

0.8

4.3

5.7

4.0

53.0 ***

-2.5

-0.4

-7.6 **

-0.8

2.5 *

-1.0

-2.7

0.4

49.9

12.0

0.3

2.6

1.3

0.0

0.3

1.1

0.6

0.1

16.0

0.4

4.3

1.6

0.4

0.4

2.3

0.5

49.9 ***

-4.0

-0.1

-1.7

-0.4

-0.4

-0.1

-1.2

0.2

54.8

9.8

0.0

6.6

1.7

0.0

4.1

2.9

1.9

1.6

13.6

0.7

5.6

1.6

0.4

2.0

2.1

2.4

53.2 ***

-3.7

-0.7

0.9

0.1

-0.4

2.1

0.8

-0.5

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. Table Svy.1 (continued)

NOTES: WA = WorkAdvance (program) group; C = control group; ABE = adult basic education; GED = general educational development; HS = high school.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Sample sizes may vary because of missing values.

Italic type indicates that the measure is nonexperimental; thus, statistical tests were not performed. 

a

Includes all reports of occupational skills training from sections A, D, and E of the survey. All subsequent occupational skills training measures include 

only skills training programs recorded in section E. For example, 113 people reported participating in skills training in the survey but not in section E.

bMeasure includes completing high school; obtaining a GED certificate; obtaining an associate's, bachelor's, or professional degree; and obtaining a 

credential as part of a skills training program. c

Measure includes only participation in occupational skills training within 18 months following each respondent's date of random assignment (the common 

follow-up period).

75
(continued)

The WorkAdvance Study

Figure 3.2

Percentage of WorkAdvance and Control Group Members Participating in Occupational Skills Training,

by Month Relative to Random Assignment and Site

0

20

40

60

80

100

RA ***2 ***3 ***4 ***5 ***6 ***789101112131415161718

Percentage

Month relative to random assignment

Per Scholas

WA group

Control group

0

20

40

60

80

100

RA ***2 ***3 ***4 ***5 ***6 **7 **89101112131415161718

Percentage

Month relative to random assignment

St. Nicks Alliance WA group

Control group

76
Figure 3.2 (continued)

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: RA = random assignment; WA group = WorkAdvance (program) group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Sample sizes may vary because of missing values.

0

20

40

60

80

100

RA ***2 ***3 ***4 ***5 ***6 ***7 ***8 ***9 ***10 ***11 ***12 **13 ***14 **15 **16 **17 **18 ***

Percentage

Month relative to random assignment

Madison Strategies Group WA group

Control group

0

20

40

60

80

100

RA ***2 ***3 ***4 ***5 ***6 ***7 ***8 ***9 ***10 ***11 **12 **13 **14 ***15 **16 ***17 ***18 ***

Percentage

Month relative to random assignment

Towards Employment WA group

Control group

77
The WorkAdvance Study

Table 3.3

Year 2 Impacts on Participation in Primary Occupational Skills Training Program, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Difference Difference Difference Difference

Outcome (%) WA C (Impact) WA C (Impact) WA C (Impact) WA C (Impact)

Started primary training 71.6 37.1 34.5 *** 66.2 35.9 30.4 *** 60.2 25.2 34.9 *** 64.0 27.6 36.4 ***

Completed primary training 60.5 24.5 35.9 *** 55.7 30.6 25.1 *** 41.0 16.1 24.9 *** 45.6 17.8 27.9 ***

Currently in primary training 4.7 7.8 -3.2 3.9 4.5 -0.5 10.7 3.9 6.8 *** 10.3 6.7 3.6

Dropped out of primary training 6.4 4.8 1.6 6.3 0.7 5.6 *** 8.5 5.2 3.3 7.8 2.8 5.0 ***

Reasons for not completing primary training

Found a job 52.4 19.8 32.5 33.3 0.0 33.3 14.8 2.3 12.5 20.1 0.0 22.8

Not enough money to continue 6.5 6.5 0.0 33.3 0.0 33.3 0.0 40.1 -44.9 6.7 20.0 -13.3

Needed to work 22.5 0.0 22.8 33.3 0.0 33.3 14.7 33.2 -18.5 0.0 15.2 -16.3

Other 15.1 86.8 -71.7 41.7 100.0 -58.3 75.6 39.3 36.3 76.5 74.3 2.2

a Characteristics of primary training classes

Included examples/assignments 

relevant to career interests 77.6 59.3 18.3 67.1 72.4 -5.3 75.7 62.4 13.2 71.3 63.1 8.1

Organized and clear expectations 81.3 74.1 7.2 83.6 81.7 1.9 75.4 75.5 -0.1 73.6 78.6 -5.0

Involved hands-on experience 83.9 63.4 20.5 77.3 72.9 4.5 81.3 77.5 3.7 72.5 72.1 0.4

Used skills learned in primary training

on the job 59.9 20.0 39.9 *** 44.3 23.0 21.3 *** 45.3 14.8 30.6 *** 39.3 14.1 25.2 ***

Participated in primary training while 

employed 16.5 15.4 1.2 12.4 14.4 -2.1 32.3 16.2 16.1 *** 22.9 11.6 11.3 ***

Employed and received pay for time in

b primary training 4.5 5.3 -0.8 3.6 3.7 -0.2 11.4 7.0 4.5 * 5.9 4.5 1.4

Paid for primary training out of

pocket or with loans 4.5 17.8 -13.4 *** 8.3 16.2 -7.9 ** 3.6 12.7 -9.1 *** 6.2 13.7 -7.5 ***

(continued)

78
Table 3.3 (continued)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Difference

Outcome (%) WA WA C (Impact)

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Average amount spent on or borrowed

for primary training ($)

Amount spent or borrowed for

primary training (%)

None, never in primary training

None,didnot pay for primarytraining

Under $5,000

$5,000 or more

Primary training financed by loans

Received a reimbursement from an

employer for primary training cost

Received help paying for primary training c from an outside organization

110

28.4

67.2

3.9

0.5

0.6

1.0

39.3

623

62.9

19.3

13.5

4.3

6.2

0.4

13.4

-512 ***

†††

-34.5 ***

47.9 ***

-9.5 ***

-3.8 ***

-5.6 ***

0.6

25.9 ***

227

33.8

57.9

6.9

1.4

1.5

1.0

41.3

411

64.5

19.3

14.0

2.3

3.4

0.0

9.2

-184

†††

-30.7 ***

38.6 ***

-7.1 **

-0.8

-1.9

1.1

32.1 ***

279

40.1

56.4

1.8

1.7

1.8

0.0

46.4

562

75.2

12.7

8.3

3.9

5.3

0.9

7.8

-283

†††

-35.1 ***

43.8 ***

-6.5 ***

-2.2

-3.5 **

-0.9 *

38.6 ***

193

36.2

57.7

4.7

1.5

4.0

0.0

47.1

813

73.0

13.4

8.3

5.4

6.5

0.3

8.6

-619 ***

†††

-36.8 ***

44.3 ***

-3.6 *

-4.0 **

-2.5

-0.3

38.5 ***

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey. Table Svy.1 (continued)

NOTES: WA = WorkAdvance (program) group; C = control group.

Primary occupational skills training for program group respondents is, for the most part, the skills training offered through WorkAdvance; for control group 

respondents, it is the first skills training after random assignment. All measures in this table relate to the primary skills training only.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall-Young adjusted p-values were used for 

categorical measures.

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 

follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Sample sizes may vary because of missing values.

Italics indicate the metric is not among the full sample shown in the table. Therefore, the measure is nonexperimental and statistical tests were not 

performed. a

Percentage represents respondents who indicated they "strongly agree" with the statements.

bMeasure includes participants who were paid by an employer; an employment, government, or community-based program; or in some other way. c

Outside organizations include the WA provider and employment, government, and community-based organizations.

79
80

WorkAdvance yielded double-digit, statistically significant impacts at all sites on participation in any education and training, ranging from 21 percentage points at St. Nicks Alliance 

to 27 percentage points at Madison Strategies Group. This overall measure includes the following types of educational activities: Adult Basic Education (ABE), General Educational Development (GED) classes, occupational skills training, college courses, and short-term classes. 

Overall, these impacts are noteworthy, given that a substantial proportion of control 

group members (between 50 percent and 63 percent) participated in education or training activities. For example, at Towards Employment, WorkAdvance increased participation in education 

and training by approximately 26 percentage points above the control group level of 55 percent. 

As expected, these effects were mostly concentrated in occupational skills training, particularly 

training in the sectors targeted by WorkAdvance.4 As shown in Box 3.2, community colleges 

and for-profit training providers were the most common places where control group members 

received training; the sector in which control group training occurred most commonly was 

health care.

In general, WorkAdvance increased participation in occupational skills training by 30 

percentage points or more, depending on the site. The skills training completion and credential 

attainment rates for WorkAdvance group members were also significantly higher than those of 

control group members at each of the sites. Impacts in the likelihood of attaining occupational 

skills training credentials range from 23 percentage points for Madison Strategies Group to 39 

percentage points at Per Scholas, which offered its program enrollees an opportunity to acquire 

multiple information technology sector credentials. 

WorkAdvance also increased individuals’ participation in short-term classes at three 

sites (such as cardiopulmonary resuscitation [CPR], how to operate a forklift, certification in a 

computer software like Microsoft Word, or basic computer skills) that may have overlapped 

somewhat with the occupational skills classes. It is important to note that WorkAdvance did not 

appear to divert people who would have gone to college into vocational training instead. This 

would be evident if there were a large negative impact on participation in college courses

Because WorkAdvance group members were more likely to have participated in an occupational skills training program, it is not surprising that they were also significantly more likely to have spent more months, on average, in such programs. Among just those WorkAdvance 

 4

Rates of participation in targeted sector training among WorkAdvance group members, as measured via 

the Year 2 Survey (Table 3.2), are lower than those measured via the providers’ MIS records (Table 2.5). Much 

of the discrepancy is probably due to recall error in the survey — a phenomenon that would have been present 

for both WorkAdvance and control group members. Appendix Table A.3 shows a sensitivity analysis that 

measures the extent to which training participation rates varied across data sources. Results show that the patterns found in providers’ MIS training records are largely consistent with those in WorkAdvance group members’ responses to the Year 2 Survey question about their participation in targeted sector training.
81

and control group members who attended occupational skills training, however, differences in 

the length of time spent in training were more modest. Among individuals who attended such

training, control group respondents at the Per Scholas and St. Nicks Alliance sites spent about 

the same amount of time in training as WorkAdvance group respondents. WorkAdvance group 

training participants at Madison Strategies Group and Towards Employment spent one to two

more months, on average, in occupational skills training than control group members who participated in training.

Despite the fact that the length of time spent in training was roughly comparable for 

WorkAdvance and control group respondent training participants, there were some differences 

in when individuals engaged in training. Specifically, control group respondents generally did 

not enroll in an occupational skills training program immediately after random assignment. This 

is illustrated in Figure 3.2, which shows that the percentage of WorkAdvance group respondents participating in occupational skills training was higher than that of control group respondents in the first six months after random assignment. 

Participation impacts on occupational skills training lasted later into the follow-up period for Madison Strategies Group and Towards Employment, the two providers that operated

dual-track programs until late 2012. Both Madison Strategies Group and Towards Employment’s WorkAdvance group respondents were more likely than their control group counterparts 

to be enrolled in a training program through the first 18 months after random assignment (Figure 3.2), suggesting that these providers engaged at least some program enrollees in training 

relatively late in the follow-up period.5

More detailed survey questions were asked of sample members about their primary occupational skills training; these findings are presented in Table 3.3.6 Given the very large and 

statistically significant differences between the WorkAdvance group and the control group in 

participation in occupational skills training, the significant effects of WorkAdvance on primary 

training participation and completion are not surprising. These more detailed questions shed 

light on how the WorkAdvance training might have been different (or not) from training that 

control group members accessed. For example, among just those individuals who did participate 

in training, WorkAdvance group members, compared with control group members, were more 

 5

A supplementary analysis of training participation in Months 10 through 18 after random assignment 

found that some early program enrollees at Madison Strategies Group and Towards Employment were attending a WorkAdvance training program at that stage. This finding suggests that these providers reengaged at least 

some early program enrollees after making the transition to a training-first approach. It is also possible that 

WorkAdvance group members were engaged in a lengthy occupational skills training program (for example, 

diesel mechanics training offered at the Madison Strategies Group site, which spanned eight months). 6

This refers to the occupational skills training offered through the WorkAdvance provider for WorkAdvance group members and the first occupational skills training program after random assignment for control 

group members. 
82

likely to “strongly agree” that their training included examples or assignments relevant to their 

career interests (at all sites but St. Nicks Alliance) and, at Per Scholas, involved hands-on experience.

In addition, at each of the sites, WorkAdvance group respondents were much more likely than control group respondents to have used skills learned in their primary training on a job. 

(See Table 3.3.) Furthermore, at Madison Strategies Group and Towards Employment, WorkAdvance group respondents were more likely to have participated in the primary training while 

employed. This finding may be partially explained by the fact that WorkAdvance group respondents at these two sites were also more likely than their respective control group respondents to be in training at the time of the survey interview (when many individuals were employed). Again, both Madison Strategies Group and Towards Employment made a transition in 

the last couple of years of WorkAdvance implementation to a training-first approach, which

may have resulted in some program enrollees participating in training programs later into their 

respective service periods.

• WorkAdvance (which offered training and other services free of charge 

to enrollees) reduced sample members’ expenditures for training. 

Publicly funded resources for training — made available for job seekers through the 

Workforce Investment Act (WIA) — were significantly reduced during the study period, having 

declined nearly 60 percent from 2000 to 2010.7 Perhaps for this reason, control group members, 

relative to WorkAdvance group members, were much more likely to have spent or borrowed 

money for training — of any type — at all sites. In addition, WorkAdvance significantly reduced the amount survey respondents spent or borrowed for training. WorkAdvance reduced 

the occurrence of education-related spending or borrowing by a range of 8 percentage points at

Towards Employment and St. Nicks Alliance to 13 percentage points at Per Scholas. These effects are statistically significant across all the sites (Table 3.3). Per Scholas and Madison Strategies Group control group respondents, in particular, were more likely to have financed their 

primary training with loans. Among control group respondents who paid for their primary training, the average amount spent or borrowed ranged from $2,300 at St. Nicks Alliance to $5,882 

at Towards Employment (not shown).

8

 

 7

Hilliard (2013). 8

Appendix Table D.1 shows impacts on a broader array of education and training measures.
83

Impacts on Job Search Activities, by Site

• Relative to control group members, WorkAdvance group members at 

each site were not more likely to have looked for a job, but they were 

much more likely to report receiving help with a job search at some 

point during the follow-up period (Table 3.4). 

Table 3.4 presents impacts on job search and work-based learning activities, as measured using responses to the Year 2 Survey. WorkAdvance did not significantly affect the proportion of people who looked for a job during the follow-up period.

9 WorkAdvance did, however, produce statistically significant effects on participation in work-based learning:

10 WorkAdvance group members at three of four sites were much more likely than their control group 

counterparts to participate in on-the-job training and were also more likely to participate in an 

internship. In the wake of the Great Recession, employers were very selective about hiring individuals with relevant experience. WorkAdvance providers reacted to this by developing internships to build up participants’ résumés with sector-relevant experience.

The survey also asked about the receipt of help related to a job search, in three forms: 

finding a job in the targeted sector, finding a specific job opening, and scheduling a job interview. While WorkAdvance group members were much more likely than control group members to say that they received help in each of these areas, increases were largest for receiving 

help finding a job in the targeted sector, with impacts ranging from 42 percentage points at Towards Employment — which relied on intermediary organizations that had existing relationships with sector employers to match program enrollees with job leads — to about 56 percentage points at Per Scholas, which used its strong relationships with large firms to open doors that 

training participants might not have had access to in the absence of the program. This finding 

supports the WorkAdvance providers’ common goal of getting WorkAdvance group members 

into jobs in the sectors they were targeting, and not just into “any job.” 

Generally, between 20 percent and 42 percent of WorkAdvance group members reported that they found at least one job with help from the WorkAdvance program. In contrast, control group members reported that they found at least one job with help from another source, especially friends or relatives.

 9

While Table 3.4 shows that WorkAdvance members at the Madison Strategies Group site are less likely 

than their respective control group members to have ever looked for a job, this 7 percentage point impact represents a difference of only about six respondents. 10Work-based learning activities include on-the-job training and internships. 
The WorkAdvance Study

Table 3.4

Year 2 Impacts on Participation in Job Search Activities, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome (%) WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Ever looked for a joba 83.6 86.7

Individual job search 77.3 79.4

Not working and has looked for

b a job in the past four weeks 21.1 28.1

Ever participated in an internship 31.0 15.1

Ever participated in on-the-job training 4.0 7.8

Job search activities in which respondent received help

Job search activities 81.7 36.2

 Finding a job in targeted sector 70.9 14.6

 Finding a specific job opening 69.8 29.2

 Scheduling a job interview 64.4 20.3

Obtained a job with help from:

 WorkAdvance provider 42.8 0.5

 Employment agency or CBOc 19.0 20.7

 Own job search 38.8 45.1

 Social media 3.8 3.0

d Other 34.1 51.1

-3.1

-2.1

-7.1 *

15.9 ***

-3.8 *

45.5 ***

56.3 ***

40.6 ***

44.2 ***

42.3 ***

-1.7

-6.4

0.8

-17.1 ***

84.8

73.5

25.4

11.8

5.9

75.1

52.7

60.3

55.8

23.6

24.2

45.5

5.9

39.2

81.5

75.2

27.3

7.2

3.3

44.8

11.1

34.2

24.6

0.0

27.7

47.5

2.2

43.2

3.3

-1.6

-1.9

4.7

2.6

30.3 ***

41.6 ***

26.0 ***

31.2 ***

24.2 ***

-3.5

-2.0

3.7 *

-3.9

75.2

68.5

15.5

11.0

6.1

76.8

61.0

65.8

53.5

31.4

24.9

50.5

6.5

38.8

82.7

73.4

17.2

4.0

2.2

24.9

8.2

19.4

7.9

0.7

34.6

55.8

5.6

42.4

-7.6 **

-5.0

-1.7

7.0 ***

3.9 **

51.9 ***

52.8 ***

46.4 ***

45.5 ***

30.7 ***

-9.8 **

-5.4

0.9

-3.7

82.4

75.2

24.9

23.6

6.8

72.5

59.7

61.0

51.4

21.2

17.6

51.4

1.4

36.7

82.4

76.6

26.5

4.9

2.0

33.4

18.2

26.0

17.1

1.2

21.2

57.1

3.3

40.7

0.0

-1.4

-1.6

18.7 ***

4.8 ***

39.1 ***

41.5 ***

35.0 ***

34.3 ***

20.1 ***

-3.6

-5.6

-1.9

-4.0

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

(continued)

84
85

Table 3.4 (continued)

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group; CBO = community-based 

organization.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 

percent.

Sample sizes may vary because of missing values.

a"Looked for a job" may include group or individual job search.

bJob search activities within the last four weeks include contacting employers, contacting a 

public employment agency, contacting a private employment agency, contacting a school or 

university employment center, contacting friends or relatives, attending or enrolling in a program 

or course, attending job training, interviewing for a job, sending out a résumé or filling out an 

application, checking union or professional registers, looking at job advertisements, placing an ad 

or answering an ad, and other job search activities. cCategory includes a One-Stop Career Center, a staffing or temporary employment agency, or 

some other employment agency or program.

d"Other" includes friends, relatives, and the response "Other."

Impacts on Postemployment Help Received, by Site

• WorkAdvance group members at each site were much more likely than 

control group members to report receiving postemployment services, 

with the impacts ranging from approximately 40 percentage points at 

Per Scholas to more than 70 percentage points at Madison Strategies 

Group (Table 3.5). 

Table 3.5 presents effects on postemployment service help received from staff members

at employment, government, or community-based organizations. It is somewhat surprising that 

WorkAdvance significantly increased the receipt of postemployment help, given the relatively 

late start (discussed in Chapter 2) that providers had in putting this program component into 

place. Although there are statistically significant increases across all the sites, Madison Strategies Group’s increase is impressively large. The result for this site may be explained in part by 

the provider’s performance-based financial compensation for WorkAdvance staff members who 

were responsible for delivering postemployment services, also described in Chapter 2. 

The Year 2 Survey asked a series of questions about the nature of postemployment services received, including help that staff provided through direct contact with the respondent, as 

well as help provided through direct staff contact with the respondent’s employer, to the extent 

that the respondent was aware of such contact. Among WorkAdvance group respondents, a little less than 50 percent at Per Scholas and more than 60 percent at St. Nicks Alliance, Madison 
The WorkAdvance Study

Table 3.5

Year 2 Impacts on Participation in Postemployment Services, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome (%) WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

a Received a postemployment service 64.3 25.5 38.9 ***

Postemployment service areas in which respondent received help 

Keeping current or most recent job 31.3 5.0 26.3 ***

Discussing how job is going 46.9 12.9 34.0 ***

Discussing how to handle problems 

with coworkers or supervisors 36.3 9.4 26.8 ***

Respondent aware that employer was contacted to:

Discuss problems with respondent's

performance on the job 10.6 4.8 5.8 **

Collect feedback on respondent's 

performance/promotion possibilities 10.8 8.3 2.6

Learn more about opportunities for 

advancement 12.7 7.8 4.9 *

Help respondent keep job 8.4 5.5 2.8

Number of times respondent was aware that employer was contacted

to discuss respondent

Never 80.8 87.0 -6.2

Once or twice 15.1 11.0 4.1

More than 2 times 4.1 2.0 2.1

69.2

20.3

62.1

38.5

13.1

15.7

17.3

13.7

73.6

21.1

5.3

23.6

3.0

11.5

8.4

4.8

5.7

7.9

4.8

88.2

8.8

3.0

45.6 ***

17.4 ***

50.6 ***

30.1 ***

8.4 ***

10.1 ***

9.4 ***

9.0 ***

†††

-14.6 ***

12.4 ***

2.2

85.8

31.3

75.2

49.2

22.9

32.1

27.0

17.8

58.4

32.8

8.8

14.3

4.5

7.9

2.8

1.5

2.2

3.4

2.5

94.4

4.7

0.9

71.6 ***

26.8 ***

67.3 ***

46.4 ***

21.3 ***

29.9 ***

23.7 ***

15.3 ***

†††

-36.0 ***

28.2 ***

7.8 ***

68.8

27.1

61.1

44.3

14.9

17.9

14.7

10.9

73.3

19.6

7.1

23.3

3.0

13.3

8.9

4.5

6.0

5.7

5.1

90.4

8.1

1.5

45.6 ***

24.1 ***

47.8 ***

35.4 ***

10.5 ***

11.9 ***

9.0 ***

5.8 **

†††

-17.1 ***

11.5 ***

5.6 ***

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

(continued)

86
87

Strategies Group, and Towards Employment reported that they ever received help via check-ins 

about “how their job was going.” Though relatively few respondents reported that a staff member from the program had ever contacted their employer to discuss their employment, WorkAdvance group respondents were still more likely than control group respondents to indicate that 

such contact occurred. At St. Nicks Alliance, Madison Strategies Group, and Towards Employment, WorkAdvance group respondents were much more likely than control group respondents to report that a staff person contacted their employers specifically to discuss job performance problems, as well as to discuss how to help the respondent keep his or her job. As discussed in Chapter 2, Per Scholas had reservations about reaching out to employers in this way 

as it might signal that the participant was potentially a “problem employee.”

The number of times staff members contacted employers also tended to differ between 

WorkAdvance and control group members. For example, at both the Madison Strategies Group 

and Towards Employment sites, WorkAdvance group respondents were more likely than their 

respective control group respondents to report more than two occasions when their employer 

was contacted.

Participation Impacts by Enrollment Cohort Varied

As mentioned in Chapter 1, a number of changes were implemented during and after the fall of 

2012 across all sites in an effort to improve service delivery, most notably a greater focus on 

advancement coaching (getting the postemployment service component into place) and, for two 

of the providers (Madison Strategies Group and Towards Employment), moving to a trainingfirst approach, which meant that most, if not all, new WorkAdvance group members were routed toward occupational skills training before beginning a job search. Given these changes in the 

scope and, perhaps, quality of services that the WorkAdvance providers implemented, a natural 

Table 3.5 (continued)

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group.

Sample sizes may vary because of missing values.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 

10 percent. The Westfall-Young adjusted p-values were used for categorical measures.

F-tests were also used to assess differences in the distribution of categorical measures across 

research groups. Statistical significance levels are indicated as follows: ††† = 1 percent; †† = 5 

percent; † = 10 percent. aPostemployment services include those received while an individual is employed: 

Respondent obtained help keeping current or most recent job, or employment service providers 

contacted the employer about how the respondent could keep the job or advance. 
88

question is whether there was any subsequent change in the pattern of participation impacts for 

those who enrolled earlier (the “early cohort”) compared with those who enrolled later (the “late 

cohort”). (As will be seen in the analyses in the following chapters, the economic impacts are 

different for the two cohorts in some sites.) Table 3.6 helps answer this question by showing 

two-year participation impacts for individuals who enrolled in the study between June 2011 and 

September 2012 and for those who enrolled between October 2012 and June 2013. 

• At least some of the WorkAdvance participation impacts are larger for 

the late cohort, compared with the early cohort, but these differences — 

in terms of their magnitude — vary by site (Table 3.6). 

For Per Scholas, WorkAdvance’s impacts on employment service receipt are about the 

same for the early and late cohorts, including impacts on ever receiving postemployment services. There is a somewhat larger impact, however, for the late cohort on the likelihood of starting training in the targeted sector: The impact was 43 percentage points for the early cohort and 

56 percentage points for the late cohort.11 

For the St. Nicks Alliance site, there are larger impacts for the late cohort on the receipt 

of help with a job search: The difference between the WorkAdvance group and the control 

group in receiving job search help is 20 percentage points for the early cohort and 37 percentage 

points for the late cohort. Participation impacts on other activities are very similar for the early 

and late cohorts. 

While Madison Strategies Group’s impacts on training participation were larger for the 

late cohort than for the early cohort, it was surprising that the difference in the two cohorts’ 

training impacts was not statistically significant, given the provider’s transition over time to the 

training-first approach for all enrollees. One partial explanation might be that some WorkAdvance group members in the early cohort participated in training later in their respective service 

periods (Figure 3.2). In contrast to Madison Strategies Group, late cohort impacts at Towards 

Employment — another provider that was initially operating a two-track program — are much 

larger than early cohort impacts regarding participation in employment services, especially 

those related to career planning and job readiness (not shown), finding a job in a particular field, 

and postemployment services. 

 11For this cohort analysis, unless noted otherwise, impacts across cohorts are discussed only if they are statistically different from one another. 
89

The WorkAdvance Study

Table 3.6

Selected Participation Characteristics of Survey Respondents, by Site and Cohort

Outcome (%)

Early Cohort Late Cohort

WA Sig.

Difference

C (Impact) WA

Difference

C (Impact)

Per Scholas a Career readiness service

b Job search service c Postemployment service

Started a skills training program

in targeted sector

Completed a skills training program

in targeted sector

Obtained a credential from a training program

in targeted sector

Sample size (total = 552)

88.6

80.9

62.0

67.9

57.5

53.6

152

57.1

32.6

23.2

24.8

17.2

11.2

132

31.6 ***

48.3 ***

38.9 ***

43.1 ***

40.3 ***

42.4 ***

88.8

81.2

65.5

71.6

58.5

54.6

135

64.5

41.2

29.2

15.3

6.8

4.7

133

24.3 ***

40.1 ***

36.3 ***

56.2 ***

51.6 ***

49.9 ***

†

St. Nicks Alliance a Career readiness service

b Job search service c Postemployment service

Started a skills training program

in targeted sector

Completed a skills training program

in targeted sector

Obtained a credential from a training program

in targeted sector

Sample size (total = 384)

89.1

72.7

70.7

57.4

51.8

49.3

106

63.1

52.5

25.1

20.7

17.9

13.8

88

26.0 ***

20.2 ***

45.5 ***

36.7 ***

33.9 ***

35.5 ***

88.8

76.0

64.5

61.7

50.6

46.2

99

68.2

39.2

25.5

23.3

23.8

22.2

91

20.6 ***

36.7 ***

39.0 ***

38.4 ***

26.9 ***

24.0 ***

(continued)

Conclusion

As demonstrated throughout this chapter, across all sites WorkAdvance substantially increased 

the take-up of employment services and occupational skills training participation, completion, 

and credential attainment. The effects of WorkAdvance on skills training participation are, as 

expected, most evident for training related to the targeted sector. Below are some key sitespecific findings about participation impacts to keep in mind when reviewing the economic impacts in Chapter 5. 
90

Table 3.6 (continued)

Outcome (%)

Early Cohort Late Cohort

WA Sig.

Difference

C (Impact) WA

Difference

C (Impact)

Madison Strategies Group a Career readiness service

b Job search service c Postemployment service

Started a skills training program

in targeted sector

Completed a skills training program

in targeted sector

Obtained a credential from a training program

in targeted sector

Sample size (total = 560)

89.6

74.1

82.5

50.9

37.8

28.4

138

40.6

24.3

12.3

15.4

9.5

7.7

115

48.9 ***

49.8 ***

70.3 ***

35.6 ***

28.3 ***

20.6 ***

90.7

79.1

88.7

59.2

41.1

35.7

159

46.1

25.5

15.7

11.5

6.6

5.9

148

44.6 ***

53.6 ***

73.0 ***

47.7 ***

34.5 ***

29.9 ***

Towards Employment a Career readiness service

b Job search service c Postemployment service

Started a skills training program

in targeted sector

Completed a skills training program

in targeted sector

Obtained a credential from a training program

in targeted sector

Sample size (total = 562)

87.5

71.5

64.8

58.5

44.8

32.7

144

61.9

44.4

29.2

16.7

12.8

12.4

142

25.5 ***

27.1 ***

35.7 ***

41.8 ***

32.1 ***

20.3 ***

87.2

73.9

72.6

63.2

49.7

39.3

142

41.6

21.3

17.4

14.5

10.0

7.9

134

45.6 ***

52.5 ***

55.2 ***

48.7 ***

39.7 ***

31.4 ***

†††

†††

††

SOURCE: MDRC calculations from the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group.

The early cohort includes all sample members randomly assigned through Quarter 3, 2012. The late cohort 

includes all sample members randomly assigned in or after Quarter 4, 2012.

Sample sizes may vary because of missing values.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Differences across subgroups were tested for statistical significance. Statistical significance levels are indicated 

as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent. a

Services include career planning, job readiness, and services related to keeping or advancing on a job.

bServices include help finding a job in the target field, finding a specific job opportunity to apply for, and 

scheduling a job interview. cPostemployment services include those received while an individual is employed: Respondent obtained help 

keeping current or most recent job, or employment service providers contacted the employer about how the 

respondent could keep the job or advance. 
91

Per Scholas, which focused on the information technology (IT) sector in New York 

City, yielded the largest impacts on occupational skills training starting, completion, and credential attainment. Also, relative to the other sites, Per Scholas produced the largest increases in 

people reporting that they received help in finding a job in the targeted sector, which may reflect 

Per Scholas’s many years of experience preparing individuals for careers in the IT sector in 

New York City. 

St. Nicks Alliance, which focused on the environmental remediation and pest control 

sectors in New York City, increased participation in every major category of employment and 

training services (career advice, starting occupational skills training, job search, and postemployment), but most of its impacts in these categories were the smallest of the four WorkAdvance sites. This is, at least in part, due to relatively high control group service participation 

rates at this site, especially with respect to skills training in the targeted sector. This suggests 

that the WorkAdvance program at St. Nicks Alliance had a higher bar to meet in delivering program services with a value beyond what enrollees could obtain elsewhere, relative to the other 

three sites. 

Madison Strategies Group, which focused on the transportation and (later) the manufacturing sectors in Tulsa, produced the largest impact in every category of employment services. At least two factors may have contributed to these large effect sizes: (1) Relative to the 

other three sites, control group receipt rates of these services were lower at this site, and (2) 

Madison Strategies Group employed a performance-based financial compensation model for its 

staff. WorkAdvance group members at this site were also more likely than their respective control group members to be enrolled in training later in the follow-up period — 10 to 18 months 

after random assignment. This may reflect the provider’s transition to a “training-first for all” 

approach, which may have induced some WorkAdvance group members in the early cohort to 

begin training late in their service periods. 

Towards Employment, which focused on health care and manufacturing in the Greater 

Cleveland area, was the other provider that transitioned to a “training-first for all” approach in 

fall 2012. At Towards Employment, WorkAdvance group members were also more likely than 

their respective control group members to be enrolled in training 10 to 18 months after random 

assignment. The participation impacts on several types of employment services at Towards 

Employment were much larger for its late cohort than for its early cohort. 

Many employment and training programs that have been studied have produced only 

small or no impacts on participation in services. It is not always easy for program enrollees to 

attend program activities, and it is certainly often difficult for them to finish long activities like 

training. The bottom line in this chapter is a very positive one: Across all sites, compared with

what would have happened in the absence of the program, WorkAdvance resulted in very large 
92

and statistically significant increases in participation in every category of measured WorkAdvance-type services. This provides the field with an excellent opportunity to understand the 

effectiveness of these services. But how much did it cost to provide all these services to the typical WorkAdvance participant, and how does this compare with the costs of the services that 

individuals would have received without WorkAdvance? This topic, which is another indication 

of the difference in treatment provided by WorkAdvance, is discussed in the next chapter. 
93

Chapter 4

The Cost of the WorkAdvance Program

Chapter Highlights

• Expenditures on the operation of WorkAdvance ran between $5,200 and 

$6,700 per participant for the four providers delivering the program and were 

between $5,200 and $5,800 at three of the programs. 

• The largest share of gross program operating expenditures (roughly half) for

each of the four providers went toward providing preemployment activities 

and occupational skills training.

• Net costs — computed by subtracting the costs that would have occurred in 

the absence of WorkAdvance from the costs of operating the program — are 

smaller than operating costs, but only the net costs of Per Scholas appear to 

be significantly smaller. (More Per Scholas control group members enrolled 

in training and they generally spent more hours in training than did control 

group members at the other sites.) Net costs are about $3,500 per participant 

for Per Scholas, and in the range of $4,900 to $5,900 at the other three sites.

• Differences between gross in-program operating costs and net costs are almost entirely attributable to control group members enrolling in community 

colleges and for-profit providers, and in the case of Per Scholas, in four-year 

colleges as well.

Introduction

The previous chapter presents information on the receipt of services by WorkAdvance group 

members, which it compares with the receipt of other available services by control group members. This chapter examines a different aspect of the contrast between the two research groups: 

what it cost to provide WorkAdvance’s services, and how this compares with the cost of what 

would have happened in the absence of WorkAdvance. This cost analysis does not take into 

account the economic benefits produced by WorkAdvance (discussed in the following chapter). 

A full benefit-cost analysis, using follow-up data from a much longer period than that available 

for this report, is planned for the future.

The following section of this chapter defines the key cost terms used in the analysis and discusses a few issues that arose in the attempt to estimate the costs of WorkAdvance. The next 

section describes the sources of the data and the methods used to measure costs. Findings from 

the cost analysis are then presented, followed by a brief conclusion. 
94

Key Concepts

Gross Costs Versus Net Costs

It is important to distinguish between two measures of costs: gross costs and net costs. 

Because these measures serve different purposes, estimates of both are reported in this chapter.

Gross costs are simply the outlays required to operate a program. An estimate of gross 

costs is needed to plan a budget and is obviously essential for providers that are already operating WorkAdvance and want to continue the program. A city without WorkAdvance that is considering introducing the program should find estimates of the cost of operating similar programs 

elsewhere to be of value. In addition, gross cost estimates can be used to examine how program 

resources are allocated among the program’s critical components, such as engaging with employers, recruiting program participants, screening potential participants, and then providing 

soft skills training, occupational skills training, interview training, and postemployment retention and advancement activities. 

Net costs are the changes in costs that result from operating a program. Stated slightly 

differently, they are costs that would not be incurred in the absence of the program. The net 

costs of WorkAdvance differ from the program’s gross costs because if the program did not 

exist, some program participants would have received at least a portion of the services provided 

by the program from other providers. For example, individuals may have found soft skills training or occupational skills training on their own elsewhere, thereby engendering costs. Net costs 

are estimated as the difference between the cost of services received by a typical WorkAdvance 

participant and the cost of similar services received by an average control group member. It follows that net costs for WorkAdvance (but not gross costs) can appropriately be compared with

the program’s net economic impacts, because both measure changes that would not have occurred without the program.

Steady-State Costs

One objective of the cost analysis is to determine what the future cost of WorkAdvance 

would be if it were to become an ongoing program — specifically, what the program would 

cost once it reached a “steady state” of operation. Because WorkAdvance was implemented as a 

demonstration program that is being evaluated by random assignment, certain research costs 

were incurred that are excluded from the cost estimates presented here because they would not 

normally exist in an ongoing version of the program. Such excluded costs encompass payments 

to the evaluation firm (MDRC) for such things as technical assistance, costs incurred that relate 

to performing random assignment (such as time spent explaining the details of research and 

random assignment to those randomly assigned), and efforts associated with recruiting and 

screening the control group.
95

The data used in the cost analysis were taken from a three-year period and divided into 

three sequential 12-month periods corresponding to the first three years of the program’s operation: April 1, 2011, through March 31, 2012 (Year 1); April 1, 2012, through March 31, 2013 

(Year 2); and April 1, 2013, through March 31, 2014 (Year 3). This report emphasizes findings 

for Year 2, although some findings for Years 1 and 3 are also reported.

Year 2 of the program was selected as the steady-state period for the WorkAdvance 

cost analysis. By Year 2, many of the program’s start-up costs, related to its staff members

learning how to operate most effectively and efficiently and establishing relationships with employers in the targeted occupational sectors, had substantially diminished. At the same time, 

during Year 2 many program participants graduated from occupational skills training and 

reached the postemployment stage of WorkAdvance. Thus, costs during the year reflect both 

the pre- and postemployment stages of program operations. It should be noted that Year 2 does 

not represent an ideal steady state, but it is the best available option to serve as one.1

Aggregate Costs and Costs per Program Participant

Some of the estimates presented in this chapter are reported as costs per program participant. More specifically, aggregate costs for Year 2 are divided by the number of program entrants in that year. As demonstrated in Appendix E, this provides an estimate of costs per participant for those entering WorkAdvance during Year 2 over their entire 24 months of program 

eligibility. Essentially, this occurs because a consistent flow of individuals entering and exiting 

the program balances out in a steady state. While entrants during Year 2 incurred only 12 or 

fewer months of costs during that year, some program participants who entered during Year 1 

continued to incur costs during Year 2. As detailed in the appendix, the costs that these earlier 

entrants incurred in Year 2 represent (and can substitute for) the costs that Year 2 entrants incurred after their year of entry ended.

Of course, some WorkAdvance entrants incurred more costs than others because they 

took greater advantage of the services offered by the program, and some individuals who entered the program dropped out of the program before incurring many WorkAdvance costs. 

Thus, the estimates should be viewed as the cost of serving an average or typical program participant.

The division of aggregate net costs by the number of individuals potentially eligible to 

incur them (WorkAdvance group members) allows for an appropriate comparison with WorkAdvance’s economic impacts, which are estimated as averages for all individuals who were 

randomly assigned to the WorkAdvance group and thus potentially eligible to receive program 

services (regardless of the extent to which they actually took advantage of them). 

 1

Potential disadvantages to the selection of Year 2 as a steady state are described in detail in Appendix A.
96

The division of aggregate gross costs by the number of individuals potentially eligible 

to incur them (WorkAdvance group members) enables comparisons of costs among the four 

WorkAdvance sites, which differ in scale. The gross cost measure also facilitates planning for 

WorkAdvance programs in places other than the four evaluation sites because it indicates the 

costs of operating the program for each individual who is expected to participate. A rough prediction of aggregate costs for a new provider at a given scale can be obtained by simply multiplying the measure by the anticipated number of participants.

In-Program Versus Out-of-Program Costs

Another way of breaking down costs that programs can engender is to sort them into inprogram costs and out-of-program costs. In-program costs are those required to operate a program and are paid for directly out of program funds. Out-of-program costs are paid for through 

other sources. For example, out-of-program costs can arise if a program sends participants to 

outside organizations and those organizations absorb any resulting costs, or if the program uses 

volunteers or donated resources. Other than occasionally prevailing on employers to talk to participants about job opportunities and needed skills in their respective sectors, WorkAdvance 

engendered almost no costs of this sort. However, as seen in Chapter 3, some control group 

members (as well as some members of the WorkAdvance group) enrolled in vocational training 

on their own and received pre- and postemployment services from providers other than WorkAdvance. The costs that these activities led to are taken into account in estimating net costs.

Data Sources and Methods

Calculating the costs of WorkAdvance required that cost data from the four sites (Per Scholas, 

St. Nicks Alliance, Madison Strategies Group, and Towards Employment) first be collected and 

sorted appropriately. Once pertinent cost data were obtained, they were then categorized, filtered, and distributed so that they could be analyzed. Aggregate costs were then broken down 

further into per participant measures for each site.

Data Sources 

As discussed above, gross costs are a fairly straightforward tally of the resources expended in the delivery of a program. In the case of WorkAdvance, virtually all program costs 

corresponded to the set of expenditures authorized in the providers’ official program budgets, 

which were established before each program year. Each provider was required to submit periodic financial reports throughout the life of the program, enumerating all monetary expenditures 

incurred in line with its respective WorkAdvance budget.2 Conversations with the WorkAd-

 2

Per Scholas, St. Nicks Alliance, and Towards Employment submitted reports quarterly, while Madison 

Strategies Group submitted reports monthly.
97

vance program director at each site also took place to assess whether there were any costs unaccounted for in the financial reports (such as unbudgeted expenses or services delivered by outside providers).3

Net costs were more difficult to capture. They are estimated by removing from gross 

costs any costs that would have existed in the absence of WorkAdvance. These were estimated 

as the costs of preemployment services and support, occupational skills training, and postemployment retention and advancement services that were received by members of the control

group.4

 Net cost estimates were then developed by subtracting the costs of services received by 

a typical control group member from the gross costs of a typical WorkAdvance participant.

The number of WorkAdvance participants at the different sites was drawn from files 

provided by each site. The files were used to determine the number of participants at each site in 

Year 2. This number was used as a denominator in the computation of per participant costs. 

Cost Categories

After gross costs were ascertained, they were categorized to facilitate further analysis. 

Gross costs were classified on two separate fronts — by primary program components and by 

key budget descriptors. WorkAdvance program activities were broken down into six core functions: recruitment; job development and employer engagement; screening; preemployment services and support; occupational skills training; and postemployment retention and advancement. 

Because certain costs may straddle one or more of those functions or may not be easily attributable to any single one (such as supervisory costs, office space, and so forth), additional categories were added for management and for costs of an undefined nature. The categorization of 

recorded costs into these eight categories was done in consultation with the program directors at 

each site and MDRC implementation staff members. From a practical operational perspective, it 

is also useful to be able to view costs in terms of salary expenditures, fringe benefits, consultant 

services, overhead, and other expenses, so costs were also broken down in that manner.

5

 3

Per Scholas was the only provider to indicate that significant costs had not been incorporated into its financial reports. It identified two full-time staff members who had not been budgeted but who spent all their 

time on WorkAdvance, as well as additional overhead costs that had not been budgeted and that were therefore 

left out of its financial reports. 4

Some members of the WorkAdvance group also participated in these activities outside the program, resulting in out-of-program costs. These costs are taken into account in estimating net costs. 5

Certain costs could not easily be associated with any specific category. As an example, fringe benefits 

were reported by providers as a total amount incurred during a defined reporting period rather than as exact 

amounts corresponding to the various salary lines reported in that period. Therefore, fringe benefits were simply prorated to the different cost categories on the basis of the percentage of total salary costs accounted for by 

each category. Similarly, all costs categorized as “undefined” were ultimately prorated to the seven remaining 

categories after all the identified costs had been distributed. Indirect costs, which are an institutional overhead 

rate built into a program budget, were similarly prorated across the same seven categories.
98

Findings

Aggregate Gross Costs

Table 4.1 presents estimates of the aggregate in-program gross costs — gross costs that 

have not yet been divided by the number of program participants — that were incurred in operating WorkAdvance at each of the program’s four sites during each of the three years for which 

data are available. Total dollar costs are reported in the top row of the panel for each site. These 

aggregate dollar values are indicated in percentage terms for seven categories that capture all the 

program activities in WorkAdvance. The seven categories are defined in Box 4.1.

The aggregate dollar values are not usefully compared across the four sites because they 

reflect the differences among the providers both in costs per program participant and in the 

number of participants. But they can be compared across program years for each site, and it is 

appropriate to compare the percentage breakdown of how program resources were allocated 

among the seven categories across sites. 

• Aggregate costs tended to be larger in Year 2 than in either Year 1 or 

Year 3.

Table 4.1 indicates that aggregate gross costs in Year 2 were considerably larger than in 

Year 1 for all four sites, reflecting the fact that the programs did not begin operating until part of 

the way through Year 1 and had relatively few participants once they did start. Costs were also 

larger in Year 2 than in Year 3 at three of the four sites and were almost as large at the fourth 

site (Towards Employment). This is probably due to the break in enrollment of new program 

participants that followed the end of random assignment in June 2013, which lasted from four to 

eight months at the different sites. This is a major factor in the selection of Year 2 costs as being 

most representative of steady-state program operations. For that reason, the remainder of this 

chapter focuses mainly on that year.

• Occupational skills training accounted for the largest share of operating 

costs. Preemployment activities, which like skills training promoted job 

readiness, were also significant.

Table 4.1 reveals that with relatively few exceptions (discussed below, as the findings 

for each cost category are described), the four WorkAdvance providers distributed their resources similarly across the cost categories during Year 2. This can be seen clearly in Figure

4.1, which uses a bar graph to show the percentage of aggregate costs that each provider spent

on the respective program activities. Notably, all four providers expended more on occupational 
99

 

The WorkAdvance Study

Table 4.1

Aggregate Gross Operating Costs (Years 1 to 3)

Internal costs Year 1 Year 2 Year 3 Total

Per Scholas

Total cost ($) 584,665 861,429 573,344 2,019,437

Program components (%)

Management 7.8 9.5 19.2 11.8

Recruitment 7.0 17.0 7.1 11.3

Job development and employer engagement 14.4 10.0 14.0 12.4

Screening 13.3 10.4 4.1 9.5

Preemployment services and support 24.3 16.9 24.4 21.2

Occupational skills training 33.2 29.0 19.0 27.4

Postemployment retention and advancement 0.0 7.2 12.1 6.5

St. Nicks Alliance

Total cost ($) 388,240 790,829 688,514 1,867,583

Program components (%)

Management 8.7 6.7 12.3 9.2

Recruitment 0.8 6.6 5.9 5.2

Job development and employer engagement 5.8 5.9 5.2 5.6

Screening 27.0 12.8 8.3 14.1

Preemployment services and support 20.8 23.1 27.1 24.1

Occupational skills training 24.9 32.7 22.9 27.5

Postemployment retention and advancement 11.9 12.2 18.3 14.4

Madison Strategies Group 

Total cost ($) 635,891 976,794 767,998 2,380,682

Program components (%)

Management 32.5 14.5 23.7 22.3

Recruitment 6.0 6.4 8.7 7.0

Job development and employer engagement 10.5 6.9 13.4 10.0

Screening 13.0 10.6 8.5 10.6

Preemployment services and support 10.2 15.4 15.8 14.1

Occupational skills training 22.4 37.4 19.3 27.5

Postemployment retention and advancement 5.5 8.8 10.6 8.5

Towards Employment 

Total cost ($) 628,469 972,850 1,027,074 2,628,393

Program components (%)

Management 30.4 15.6 10.0 16.9

Recruitment 4.3 5.1 3.8 4.4

Job development and employer engagement 28.0 15.2 14.4 18.0

Screening 2.3 3.4 3.5 3.2

Preemployment services and support 27.2 22.9 24.0 24.4

Occupational skills training 7.3 33.7 38.4 29.3

Postemployment retention and advancement 0.5 4.0 5.9 3.9

SOURCES: Financial reports submitted to The Mayor's Fund to Advance New York City through the Grants 

Management System, and additional estimates by Per Scholas of uncaptured expenditures.

NOTES: Costs attributable to Compass (Towards Employment's partner in the administration of WorkAdvance) 

and operations in Youngstown, Ohio, have been excluded. Research-related costs have been excluded for all 

sites.
100

 

Box 4.1

Definitions of the Cost Categories

Management 

Costs associated with program administration, oversight, and staff supervision. (Costs attributable 

to management of the research are excluded.)

Recruitment

Cost of activities and materials necessary to attract eligible individuals to participate in 

WorkAdvance. (Costs attributable to the recruitment of the control group are excluded.)

Job development and employer engagement

Costs relevant to determining where jobs are available and where they should be sought for 

program participants; specific types of training necessary to fill those jobs; and the development of relationships with employers that can provide job opportunities to program participants. Costs span preemployment and postemployment periods and encompass relationships 

with employers and trade-specific organizations.

Screening

Costs that result from determining whether individuals qualify for participation in a site’s 

WorkAdvance program (for example, testing, assessing work history, and so on). (Costs attributable to the screening of the control group are excluded.)

Preemployment services and support

Cost of providing program participants with general skills and other capabilities needed to 

obtain and keep a job (for example, soft skills training, interview coaching, financial assistance for employment- and training-related expenses, and so on).

Occupational skills training

Cost of providing program participants with the sector-specific skills necessary to perform 

jobs targeted by the respective WorkAdvance program.

Postemployment retention and advancement

Costs that arise from follow-up once program participants have obtained employment (for example, coaching participants through challenges and advancement opportunities in their employment, interventions with the employers of participants when necessary, and so on).
101

The WorkAdvance Study

Figure 4.1

Percentage Distribution of Gross Operating Costs (Year 2)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Management Recruitment Job development

and employer

engagement

Screening Pre-employment

services and

support

Occupational skills

training

Postemployment

retention and

advancement

Share of costs (percentage)

Program components

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

SOURCES: Financial reports submitted to The Mayor's Fund to Advance New York City through the Grants Management System, and additional estimates 

by Per Scholas of uncaptured expenditures.

NOTES: Costs attributable to Compass (Towards Employment's partner in the administration of WorkAdvance) and operations in Youngstown, Ohio, have 

been excluded. Research-related costs have been excluded for all sites.
102

skills training — the program’s centerpiece — than on any other activity. Specifically, occupational skills training accounted for roughly one-third of total operating costs at the four sites, 

ranging between 29 percent and 37 percent. In addition, preemployment services and support, 

which involved soft skills training and job interview coaching aimed at getting program participants into the sorts of jobs for which they were being trained, accounted for 15 percent to 23 

percent of total gross expenditures at the four sites. Thus, over half of all expenditures at three 

of the four sites, and 46 percent at the remaining site (Per Scholas), were devoted to what might 

be termed “job readiness activities.” 

Per Scholas, the information technology (IT) training provider, which had operated a 

program similar to WorkAdvance for a number of years, is the only provider that spent well 

under a third of its budget on occupational skills training. Yet Per Scholas also enrolled a higher 

percentage of its participants in skills training, and those enrolled received more hours of training. One plausible explanation for these differences is that Per Scholas is the only provider to 

have used its own employees exclusively to conduct training. It is conceivable that in-house 

training allowed Per Scholas to keep the per participant cost of occupational skills training relatively low, perhaps by avoiding various provider fees or allowing for greater flexibility among 

its staff members to multitask. The other three providers, in effect, absorbed the infrastructure 

costs of other institutions by paying outside organizations to provide training. In addition, Per 

Scholas had more years of experience in providing the type of training that it delivered in 

WorkAdvance than the other providers, which may have allowed it to use its resources more 

efficiently. 

As discussed in Chapter 1, a major objective of WorkAdvance was to help participants 

both maintain and advance in employment. In contrast to the expenditures on job readiness activities, however, postemployment retention and advancement never accounted for more than 

12 percent of total Year 2 gross expenditures. This probably stems from the fact that fewer 

WorkAdvance participants graduated from occupational skills training and were therefore potential recipients of postemployment retention and advancement aid during Year 2 than would 

have been the case in a true steady state.6 Consequently, expenditures on this service are probably smaller than would have occurred in an actual steady state. These numbers can be interpreted in the context of postemployment expenditures in Year 3, which probably overrepresents 

such expenditures relative to the steady state.7 Even in Year 3, postemployment services never 

 6

The relatively low number of graduates from occupational skills training can be attributed to the less robust enrollment in Year 1 that is typical of program start-up. 7

Appendix E indicates why postemployment costs in Year 2 are probably underrepresented and those in 

Year 3 are probably overrepresented. Overrepresentation of postemployment costs in Year 3 relative to Year 2 

implies that the percentage of total expenditures devoted to postemployment retention and advancement in 

Year 3 can be viewed as an upper bound.
103

account for more than 18 percent of total costs, so any underrepresentation in Year 2 is probably 

relatively minor.

As discussed in Chapter 2, it took time to recruit a sufficient number of program participants. Even so, less than 10 percent of total provider costs were expended on recruitment during each of the three program years,8 with an exception at Per Scholas (which made a large, 

one-time-only purchase of “outreach materials” in Year 2). As Table 4.1 indicates, recruitment 

costs at Per Scholas are similar to those at other providers in Years 1 and 3. 

Following recruitment, those who applied to WorkAdvance had to be screened by each 

provider to determine whether they qualified for the program and the study. The cost of this effort was nontrivial at three of the four sites, where it accounted for between 10 percent and 13 

percent of total operating costs in Year 2.9 In contrast, the Year 2 cost of screening made up only 4.2 percent of costs at Towards Employment. 

As described Chapter 2, part of the WorkAdvance program model involved establishing 

relationships with employers, both to determine the sort of training that should be provided and 

to place and retain program participants in the types of jobs for which they were trained. Towards Employment was the only provider to enlist the services of outside entities to perform 

this task, which may have contributed to the higher cost that it incurred for this activity relative 

to the other providers (see Figure 4.1).

Although WorkAdvance is a fairly complex program, Table 4.1 indicates that the cost 

of managing it was kept relatively modest. The variation in the magnitude of management costs 

across providers is probably due to a degree of uncertainty on the part of the program directors 

as to which activities constitute management activities, which may have led to some inconsistency across sites. Moreover, in measuring management costs, program directors attempted

to net out time devoted to managing the research effort (as opposed to the program itself), and 

staff judgments about this probably also varied across sites. Still, it is evident that management 

costs, while not large, are a nontrivial fraction of total gross operating costs.

 8

Control group members, as well as those assigned to the WorkAdvance group, must be recruited before 

random assignment. Costs for recruiting control group members are excluded from Table 4.1, however, as well 

as from all subsequent tables. 9

Because screening took place before random assignment, persons ultimately assigned to the control group 

as well as those assigned to the WorkAdvance group were screened. Costs for screening control group members are removed from the cost estimates.
104

Gross Costs per WorkAdvance Group Member

Year 2 gross in-program costs per WorkAdvance group member (“per participant”) for 

each provider appear in Table 4.2.10 These costs are broken down into the seven program components that were defined above. While they are reported as dollars per person instead of in percentage terms, the pattern of expenditures is largely similar to the Year 2 cost information conveyed in Table 4.1 and Figure 4.1, so costs by program component are not discussed further in 

this section. However, Table 4.2 also breaks down costs into five budget components — salary, 

fringe benefits, overhead, consultants, and other program expenditures — that were not included in Table 4.1 and Figure 4.1. “Overhead” refers to expenses that support an organization or 

provide general resources or infrastructure for the program, such as rent, office supplies, equipment, and auditing and payroll services. “Consultants” are entities not directly employed by 

WorkAdvance organizations that provide services typically outlined in some sort of written 

agreement. In the context of WorkAdvance, consultants were most often used to provide skills 

training. “Other program expenditures” is a catch-all term for the remaining expenditures that 

are not directly related to personnel costs, which are captured by “salary” and “fringe benefits.” 

As previously noted, by far the most important expenditure engendered by WorkAdvance was providing occupational skills training. Towards Employment covered this cost 

through tuition payments to schools, categorized as “other program expenditures,” while Per 

Scholas used its own employees as trainers and Madison Strategies and St. Nicks Alliance opted to provide training by using subcontractors. Because Per Scholas depended so heavily on its 

own staff, its salary costs account for a much larger share of its total WorkAdvance budget than 

was the case across the other three providers (70 percent versus less than 40 percent).11

• Total gross costs per WorkAdvance group member in Year 2 are quite 

similar among three of the four WorkAdvance sites, ranging between 

$5,200 and $5,750. The fourth site — St. Nicks Alliance — had gross 

costs in excess of $6,650 per WorkAdvance group member, which is 

$900 more than the next most expensive site.

The cost of a similar training program and the cost of the same number of hours of 

training at a community college can put these WorkAdvance cost estimates into perspective. On 

 10As discussed in Appendix E, postemployment costs in Year 2 are likely to be understated relative to 

what they would be were WorkAdvance in a true steady state. Thus, in computing Year 2 costs per participant, 

the cost of postemployment services in Year 2 was replaced by the Year 3 cost of these services, which should 

bring the measure closer to an accurate approximation of the steady state. 11Per Scholas’s fringe benefits were similar to those of the other providers because its fringe benefits as a 

percentage of salary were lower than those of the other providers (12 percent compared with 15 percent, 24 

percent, and 25 percent).
The WorkAdvance Study

Table 4.2

Steady-State Gross Operating Costs (per Participant)

Internal costs ($)

Budget Components

Salary Total

Fringe

Benefits Overhead Consultants

Other Program

Expenditures

Per Scholas

Program components

Management

Recruitment

Job development and employer engagement

Screening

Preemployment services and support

Occupational skills training

Postemployment retention and advancement

Total

St. Nicks Alliance

Program components

Management

Recruitment

Job development and employer engagement

Screening

Preemployment services and support

Occupational skills training

Postemployment retention and advancement

Total

446

364

467

485

791

1,003

359

3,915

285

245

249

524

780

24

668

2,775

52

43

55

57

93

117

52

468

70

60

61

128

191

6

151

667

46

81

48

50

81

400

50

756

77

76

67

146

264

374

203

1,206

0

0

0

0

0

0

0

0

0

0

0

0

0

1,676

0

1,676

0

480

0

0

0

135

0

616

0

45

0

22

252

22

0

342

544

968

570

592

965

1,655

461

5,754

431

426

377

820

1,487

2,103

1,022

6,666

(continued)

105
Table 4.2 (continued)

Internal costs ($)

Budget Components

Salary Total

Fringe

Benefits Overhead Consultants

Other Program

Expenditures

Madison Strategies Group 

Program components

Management

Recruitment

Job development and employer engagement

Screening

Preemployment services and support

Occupational skills training

Postemployment retention and advancement

Total

Towards Employment 

Program components

Management

Recruitment

Job development and employer engagement

Screening

Preemployment services and support

Occupational skills training

Postemployment retention and advancement

Total

403

210

254

415

484

77

222

2,066

503

103

137

104

665

118

219

1,849

53

27

33

54

63

10

39

280

119

24

32

25

157

28

49

434

219

50

53

82

119

289

89

901

64

21

62

14

94

138

51

445

77

0

19

0

45

1,267

0

1,409

119

44

553

2

118

0

0

836

4

49

2

3

94

308

87

547

0

69

0

32

145

1,453

0

1,699

757

336

362

555

805

1,951

437

5,203

804

261

784

177

1,179

1,737

320

5,262

SOURCES: Financial reports submitted to The Mayor's Fund to Advance New York City through the Grants Management System, and 

additional estimates by Per Scholas of uncaptured expenditures.

NOTES: Costs attributable to Compass (Towards Employment's partner in the administration of WorkAdvance) and operations in Youngstown, 

Ohio, have been excluded. Research-related costs have been excluded for all sites.

106
107

one hand, the cost per participant for the adult programs funded under the Job Training Partnership Act (JTPA) in the 1980s (and evaluated using an experimental evaluation similar to that for 

WorkAdvance) was about $4,500 in today’s dollars.12 On the other hand, the annual expenses 

incurred by a community college that provides full-time students with approximately 450 hours 

of classroom instruction over the course of a year are generally above $10,000.13 A typical fulltime college student, like the typical WorkAdvance participant, also receives services in addition to classroom instruction (such as assessment, advising, and help with course work). Per 

Scholas participants received on the order of 450 hours of classroom instruction, while WorkAdvance participants at the other three sites received somewhat less.

Relatively high gross costs per WorkAdvance group member at St. Nicks Alliance are 

reflected by St. Nicks being the most expensive provider in four of the seven core categories 

listed in Table 4.2 (screening, preemployment services and support, occupational skills training, 

and postemployment retention and advancement). These high costs appear to be attributable to 

multiple factors. For example, St. Nicks Alliance had substantially fewer participants than any 

of the other providers in Year 2 — less than two-thirds as many as Madison Strategies or Towards Employment, and approximately four-fifths as many as Per Scholas — and thus operated 

at a smaller scale (which may have translated into excess capacity, for example if each career 

coach at St. Nicks served fewer participants than coaches at the other sites). Madison Strategies 

Group and Towards Employment also placed some participants directly into employment, resulting in lower costs than St. Nicks for any participants who did not receive occupational skills

training or other WorkAdvance services. In addition, management costs per participant at St. 

Nicks were lower than those at the other three providers, possibly resulting in a less efficient 

operation. Finally, St. Nicks Alliance operated in a high-cost environment (New York City). 

These factors are discussed in greater detail in Appendix A.

Net Costs per WorkAdvance Group Member

Table 4.3 displays the net costs for WorkAdvance. Gross costs per WorkAdvance 

group member are shown in the table’s first column, and gross costs associated with control 

group members are shown in the second. Estimates of WorkAdvance’s net costs per person are 

computed by subtracting the second column from the first and are reported in the third column. 

Table 4.3 also shows “in-program” and “out-of-program” costs separately for three categories 

of activities: preemployment services and support, occupational skills training, and post- 

 12Bloom et al. (1997), Table 1. Costs were originally estimated in 1987-1989 dollars; the $4,500 estimate 

reflects inflation adjustment into 2012-2013 dollars. 13See Appendix A for information on data sources.
108

 

The WorkAdvance Study

Table 4.3

Gross and Net In-Program and Out-of-Program Costs (per Individual)

Per person costs ($)

Gross Costs

WA C Net Costs

Per Scholas

Program components

Management

Recruitment

Job development and employer engagement

Screening

Preemployment services and support

In-program

Out-of-program

Occupational skills training

In-program

Out-of-program

Postemployment retention and advancement

In-program

Out-of-program

Total costs

St. Nicks Alliance

Program components

Management

Recruitment

Job development and employer engagement

Screening

Preemployment services and support

In-program

Out-of-program

Occupational skills training

In-program

Out-of-program

Postemployment retention and advancement

In-program

Out-of-program

Total costs

544

968

570

592

965

34

1,655

391

461

35

6,214

431

426

377

820

1,487

39

2,103

898

1,022

93

7,696

0

0

0

0

0

34

0

2,626

0

79

2,739

0

0

0

0

0

39

0

1,625

0

164

1,828

544

968

570

592

965

0

1,655

-2,234

461

-44

3,476

431

426

377

820

1,487

0

2,103

-727

1,022

-71

5,868

(continued)
109

 

Table 4.3 (continued)

Per person costs ($)

Gross Costs

WA C Net Costs

Madison Strategies Group 

Program components

Management

Recruitment

Job development and employer engagement

Screening

Preemployment services and support

In-program

Out-of-program

Occupational skills training

In-program

Out-of-program

Postemployment retention and advancement

In-program

Out-of-program

Total costs

Towards Employment 

Program components

Management

Recruitment

Job development and employer engagement

Screening

Preemployment services and support

In-program

Out-of-program

Occupational skills training

In-program

Out-of-program

Postemployment retention and advancement

In-program

Out-of-program

Total costs

757

336

362

555

805

28

1,951

781

437

36

6,048

804

261

784

177

1,179

26

1,737

1,145

320

11

6,443

0

0

0

0

0

39

0

982

0

68

1,089

0

0

0

0

0

30

0

1,522

0

37

1,589

757

336

362

555

805

-11

1,951

-201

437

-32

4,959

804

261

784

177

1,179

-4

1,737

-378

320

-26

4,855

SOURCES: In-program costs are derived from financial reports submitted to The Mayor's Fund to 

Advance New York City through the Grants Management System and from additional Per Scholas 

estimates of uncaptured expenditures. Out-of-program costs are based on estimates developed by MDRC 

using figures from: (1) the Integrated Postsecondary Education Data System, hosted by the U.S. 

Department of Education's National Center for Education Studies; (2) U.S. Department of Labor 

Employment and Training Administration Program Year 2013 Final Allotments and employment 

outcomes; and (3) official budgets issued to WorkAdvance Social Innovation Fund subgrantees.

NOTES: WA = WorkAdvance (program) group; C = control group.

Costs attributable to Compass (Towards Employment's partner in the administration of 

WorkAdvance) and operations in Youngstown, Ohio, have been excluded. Research-related costs have 

been excluded for all sites.
110

employment retention and advancement. In-program expenditures for these activities were incurred only by the WorkAdvance group and were paid from the program’s budget for WorkAdvance.14 In contrast, both the WorkAdvance group and the control group incurred out-ofprogram costs, which were paid by sources other than the WorkAdvance budget. For example, 

costs may have been covered by community colleges, for-profit training organizations, or state 

employment services, or paid out of pocket by those participating in the relevant activities. All 

costs incurred in the other four categories listed in the table were in-program. In-program costs 

also appear in Table 4.2, but out-of-program costs appear only in Table 4.3.

The key finding from Table 4.3 is that net costs are smaller than the gross in-program 

operating cost figures that appear in Table 4.2, as expected; but with the rather striking exception of Per Scholas, they are not much smaller. This is partly because out-of-program occupational skills training costs for control group members at sites other than Per Scholas were substantially offset by out-of-program costs for the WorkAdvance groups at the respective sites. As 

a consequence, net costs for three of the four sites are relatively similar to gross costs, falling 

only 5 percent to 12 percent below gross costs. Per Scholas, in contrast, had net costs that fell 40 

percent below its gross costs. In absolute terms, net costs per individual for Per Scholas are 

around $3,500, which is over $1,300 lower than for the next lowest site (Towards Employment). 

• Per Scholas had substantially lower net costs than any other provider, in 

part because its control group members incurred considerably more 

out-of-program training costs than those at any other site.

As discussed in greater detail below, Per Scholas’ relatively low net costs arise because 

the cost of out-of-program occupational skills training at Per Scholas is much larger for the control group than for the WorkAdvance group. Indeed, Table 4.3 shows that out-of-program costs 

for control group members at Per Scholas are substantially higher than they are at the other 

three sites and out-of-program costs for the members of the WorkAdvance group are substantially lower.

Table 4.3 indicates that two of the categories of out-of-program costs — preemployment services and support, and postemployment retention and advancement — are of 

relatively minor importance. Their magnitude is small for both the WorkAdvance group and the 

control group at each site. Because the cost for the control group is partially offset by the cost 

for the WorkAdvance group, their effect on net costs is also slight. 

 14Management, recruitment, and screening costs were incurred for the control group, but these costs were 

entirely for research purposes — they would not have existed in the absence of an evaluation. Therefore, they 

are not included in the costs reported in Table 4.3. 
111

Out-of-program preemployment service and support costs were incurred when members of the WorkAdvance and control groups obtained a job through employment agencies or 

community-based organizations (CBOs), and, as indicated by Table 3.4, the fraction who obtained a job in this way was relatively small (usually under one-quarter) and not much larger for 

control group members than for the WorkAdvance group members. Moreover, the expenditures 

that are incurred when jobs are obtained through employment agencies are often fairly small. 

State employment offices, for example, rely heavily on computerized lists and other measures 

that help bring costs down. 

Out-of-program postemployment retention and advancement costs resulted from members of the WorkAdvance and control groups receiving help from an employment agency or 

CBO to retain a job. Relatively few did: Never more than 5 percent of the control groups and an 

even smaller percentage of the WorkAdvance groups received this sort of assistance.

Costs associated with out-of-program occupational skills training arise from courses 

taken at high schools, community colleges, four-year colleges, for-profit providers, and nonprofit organizations. Of these, community colleges and for-profit providers account for most of 

the measured cost of out-of-program training received by members of the WorkAdvance research sample. Combined, these two categories account for 84 percent of the control group 

training costs at St. Nicks Alliance, 95 percent of such costs at Madison Strategies Group, 86 

percent of such costs at Towards Employment, and 66 percent of such costs at Per Scholas. 

Four-year colleges also played an important role among control group members at Per Scholas, 

accounting for 20 percent of the out-of-program training costs.

Why Are Per Scholas’s Net Costs So Low?

The costs of out-of-program training reported in Table 4.3 are a reflection of several 

factors: the percentage of the WorkAdvance and control group members that enroll in courses; 

whether enrollees are full-time or part-time students; the number of months that those who enroll remain in class; and the per student expenses borne by the local educational institutions in 

which they enroll (for example, the cost of providing instruction, institutional support, and student services). Together, these factors contributed to Per Scholas’s relatively low net costs per 

participant.

The percentage of each group that enrolls is of particular importance. Table 3.2 indicates that the percentages of WorkAdvance group members who enrolled at community colleges and at for-profit training providers were lower at Per Scholas than at the other three sites (often considerably so), perhaps because a higher proportion of Per Scholas participants took skills 

training provided by WorkAdvance and this training was longer and more intense than at the 

other three providers. It is possible that as a consequence, more program participants at Per 
112

Scholas than at the other sites became convinced that they had received the training they needed 

to improve their lives and therefore did not seek it elsewhere. 

In contrast to the Per Scholas WorkAdvance group, Per Scholas control group members, when taken as a whole, enrolled in occupational skills training and college courses at a 

higher rate than at any of the other sites. Furthermore, those who enrolled generally spent more 

hours in training. This may be attributable to the lower age and the higher level of education of 

Per Scholas control group members relative to control group members at the other sites (Table 

1.4). Although the percentages of control group members enrolling in community colleges and 

at for-profit providers were not the highest at Per Scholas, they were relatively high among the 

four sites. Moreover, Per Scholas control group members were the only group among which 

more than a minuscule percentage enrolled in four-year colleges. The expenditures resulting 

from this enrollment in four-year colleges account for roughly a third of the differences between 

the control group out-of-program costs at Per Scholas and those at Madison Strategies and Towards Employment, and for almost half the difference between Per Scholas and St. Nicks Alliance. Most of the remaining differences between Per Scholas out-of-program control group 

costs and the corresponding costs at the other three sites can be attributed to expenses resulting 

from courses taken at community colleges and for-profit providers.

As shown in Table 3.2, the differences in favor of the control group (versus the WorkAdvance group) in enrollment at community colleges and for-profit training providers were 

much larger at Per Scholas than at the other three sites. In addition, results from the Year 2 Survey indicate that among those taking training at community colleges or for-profit providers, Per 

Scholas control group members enrolled for more months than their counterpart WorkAdvance 

group members, a gap in outcomes that was considerably larger than the corresponding gaps at 

the other three sites. These gaps are reflected in the difference between the out-of-program occupational skills training costs that were incurred by Per Scholas control group members and by 

Per Scholas WorkAdvance group members — a difference that is much larger than at any of the 

other three sites. 

Per student educational expenditures — especially at community colleges — were 

somewhat larger in the Bronx, where Per Scholas is located, than at the locations of the other 

three WorkAdvance programs. This tended to increase out-of-program educational costs for 

control group members in the area relative to their counterpart, the Per Scholas WorkAdvance 

group members, thereby decreasing Per Scholas’s estimated net cost. However, when average 

per student expenditures at the other three sites are substituted for per student expenditures in 

the Bronx, Per Scholas’ net costs increase by only $286, suggesting that higher local educational costs in the Bronx account for little of Per Scholas’s relatively small estimated net costs.
113

Conclusion

One key finding from the cost analysis is that gross program operating costs per WorkAdvance 

group member (which include only in-program costs) ranged from $5,200 to $5,800 at three of 

the four sites, and were $900 higher at the fourth site (St. Nicks Alliance). The narrow range at 

three of the sites suggests that similar WorkAdvance programs run elsewhere are likely to incur 

costs within this general range, and the higher costs at the fourth site might be viewed as a potential upper bound. Moreover, considering that the programs at all four sites allocated roughly 

half their operating expenditures to providing preemployment activities and occupational skills

training, it would be reasonable for any programs replicating WorkAdvance as it is implemented here to incorporate a similar distribution of costs into their budgets.

Net cost figures are the extra cost society spends letting eligible individuals attend 

WorkAdvance rather than what they would have done otherwise, in the absence of WorkAdvance. Because WorkAdvance encompasses activities that individuals targeted by the program 

would have engaged in on their own — as the control group members did — net costs (which 

include both in-program and out-of-program costs) are smaller than gross in-program operating 

costs. However, with the exception of Per Scholas, they do not appear to be much smaller. The 

differences between gross in-program operating costs and net costs are almost entirely attributable to control group members enrolling in community colleges and for-profit institutions, and, 

in the case of Per Scholas, in four-year colleges. Members of the WorkAdvance group also enrolled in these institutions, but to a lesser extent than control group members, especially at Per 

Scholas. 

The next chapter examines the economic effects of the difference in services received 

by WorkAdvance and control group members (as well as the attendant differences in cost). 

Thus, the next chapter’s focus is on WorkAdvance’s effects on such outcomes as employment 

in the targeted sector and total earnings and income.

115

Chapter 5

Economic Impacts of WorkAdvance

Chapter Highlights

• The employment and earnings effects of the WorkAdvance programs varied 

across the providers, but overall, the findings show that well-implemented 

sector-based training programs can improve participants’ economic outcomes.

• Per Scholas produced large impacts on employment and earnings throughout 

most of the follow-up period. These impacts were substantial enough to 

translate into positive impacts on income, public benefit receipt, and life satisfaction.

• The WorkAdvance program at St. Nicks Alliance had little to no effect on 

employment and earnings, and only a few impacts on secondary economic 

outcomes.

• Towards Employment increased employment and earnings, primarily late in 

the follow-up period. The program produced impacts on a few measures of 

advancement but had little effect on other secondary outcomes.

• The WorkAdvance program at Madison Strategies Group increased earnings 

late in the follow-up period but did not have a systematic detectable effect on 

employment. The program produced impacts on several secondary outcomes,

including nonfinancial advancement measures and receiving employerprovided benefits.

Introduction

Chapter 3 shows large differentials in service receipt between WorkAdvance group members 

and control group members at all four sites. Using data from the Year 2 Survey and unemployment insurance wage records, this chapter discusses whether these large differentials in service 

receipt led to impacts on economic outcomes. As in Chapters 2 and 3, the results in this chapter 

are discussed by site within domains.1 The first domain — employment and earnings — is con-

 1

In the original Subgrantee Evaluation Plan, the intention was to analyze impacts by sector. Because most 

sites focused on a unique sector, the only difference between analyzing impacts at the site level and at the sector level is in Ohio. But since the Compass site in Ohio was eliminated from the study, the statistical power was 

insufficient to reliably estimate impacts by sector. Therefore, a decision was made in the analysis planning 
116

sidered primary. This domain includes employment and earnings, as measured by both data 

sources, as well as various exploratory measures from the Year 2 Survey. The other domains 

include income, life satisfaction, and material hardship, and are considered secondary. 

Data Sources and Samples

The findings on employment, earnings, and other economic outcomes presented in this 

chapter come from two data sources: the WorkAdvance Year 2 Survey and unemployment insurance wage records. While both data sources provide valuable information, there are some 

important differences and trade-offs between the two sources. Administrative data are available 

for the full sample, but as discussed in Chapter 1, at least 10 percent of jobs are not covered and 

these records cover employment only within the state. Survey data cover all jobs, but survey 

participants often do not recall the details of employment accurately, and approximately 20 percent of the sample did not respond to the survey. Taking into account both data sources enables 

the most reliable, composite assessment of impacts. For the most part, the findings are consistent across the two data sources.2

The WorkAdvance Year 2 Survey asked sample members about their participation in 

workforce services (these findings are presented in Chapter 3), as well as their employment and 

earnings history since study entry. Sample members were interviewed for the survey between 

18 and 30 months after they entered the study; the average sample member was interviewed 22 

months after entering the study. Overall, the survey achieved an 80 percent response rate; the 

response rate for the WorkAdvance group (83 percent) was slightly higher than for the control 

group (77 percent).3 Most survey-based measures presented in this chapter cover the full followup period for each respondent (from the date of study entry through the date of the survey interview), regardless of the month the respondent was interviewed. Measures that are reported in 

specific time intervals (for example, average number of months in current or most recent job) 

cover only the first 18 months relative to each respondent’s month of random assignment. This 

18-month coverage period is referred to as the common follow-up period.

 

stage to estimate impacts at the site level. Ultimately, this affects only the Ohio site because every other site 

focused on a single and different sector (broadly defined). Findings are shown by site based on the results of a 

power analysis conducted during the analysis planning phase. Additional information on the power analysis 

can be found in Appendix A. Pooled impacts and impacts by cohort and subgroup are shown in Chapter 6. 2

While the results are generally consistent between the two data sources, in some cases the results are statistically significant for the larger administrative records sample but not for the survey. For example, this pattern occurs at the Towards Employment site for measures of employment. See Appendix A for the survey response analysis and more details on the comparison between survey-based and unemployment insurance-based 

employment findings. 3

The response rates by site are Per Scholas, 80 percent; St. Nicks Alliance, 80 percent; Madison Strategies 

Group, 80 percent; and Towards Employment, 81 percent. Appendix A contains more details on the Year 2 

Survey response rates.
117

Findings based on unemployment insurance records data are presented for the full research sample. Nine quarters of follow-up data are available for all sample members.4 Unemployment insurance records are available quarterly, and findings are presented relative to each 

sample member’s quarter of random assignment. For example, “Year 1” refers to the first four 

quarters following each sample member’s quarter of random assignment.

The Expected Impacts of the WorkAdvance Programs

As mentioned in Chapter 1, only one in five sample members were employed at the 

time they entered the study. This was expected given that the services offered by the program 

— career readiness and occupational skills training — required a substantial time commitment. 

The WorkAdvance model was designed to increase employment and earnings by offering these 

services along with job placement and postemployment services that would help participants 

obtain jobs with opportunities for career advancement. Based on the model, it was hypothesized 

that the WorkAdvance programs would first increase employment, primarily in the targeted sector, and then increase earnings.

As noted earlier, two of the providers — Madison Strategies Group and Towards Employment — initially offered a placement-first track in addition to the training-first track. For 

participants in the placement-first track,5

 it was thought that employment and earnings impacts 

would begin to emerge shortly after study entry, as participants did not engage in occupational 

skills training and went directly into job search after career readiness training. However, because these participants did not go to skills training (at least initially), they may not have received as intensive a set of services as did the participants in the training-first track. The treatment contrast for this group may not have been enough for them to obtain better jobs than their 

control group counterparts.6 

For participants in the training-first track (the majority of participants), it was expected 

that WorkAdvance group members would have lower employment rates, and therefore earnings, than the control group in the first few months following study entry owing to the opportunity cost of training. It was thought that not all control group members would engage in occupational skills training on their own due to the often high costs of training and the limited avail-

 4

Ten quarters of data were collected altogether. Quarter 1 was the quarter of random assignment and is 

therefore not included in the follow-up data. 5

Sample members were not randomly assigned into one of the two tracks, and therefore the exact number 

of sample members in the placement-first track is not known. 6

The placement-first track was phased out for the most part in fall 2012 due to concerns about participants 

getting “stuck” in dead-end jobs. However, having some initial placements may have helped these providers

(each of which was dealing with a new sector or a new location) to establish relationships with employers more 

quickly than they would have had they started with a pure training program.
118

ability of public funding for it.

7 Instead, most control group members would probably seek 

more immediate employment. Therefore, only after WorkAdvance group members completed 

occupational skills training and looked for and obtained a job would employment and earnings 

for that group begin to increase. 

Past studies of workforce programs have shown it can take up to two or even three

years before economic impacts begin to emerge.8 After accounting for the time spent in occupational skills training, only a little over one year of follow-up data are available.9 It was projected 

that in the Year 2 Survey data (which include information on the receipt of postemployment 

services and on advancement outcomes), only initial job placements would be seen for most 

sample members, and that any advancement gains stemming from postemployment services (a 

key and distinguishing component of the WorkAdvance model) would not yet have materialized. The unemployment insurance data presented in this report cover a longer follow-up period 

than the Year 2 Survey data (nine quarters), and so it was thought that these data might capture 

subsequent jobs and initial advancement gains. Plans are also being made to collect longer-term 

data beyond the follow-up period covered by this report to capture future advancement.

The Control Group Benchmark

Control group members were not eligible to receive WorkAdvance services, and therefore their outcomes can be used as a benchmark against which to measure the impacts of the 

WorkAdvance programs. Employment and earnings outcomes for the control group represent 

what would have happened in the absence of WorkAdvance. Control group members were free 

to seek out any non-WorkAdvance employment services and training opportunities in their local 

communities. 

• Employment rates reported in both the Year 2 Survey and in unemployment insurance wage records are high (between 84 percent and 95 

percent) for the control groups at all four sites. Earnings levels for the 

control group increased over time.

The amount of screening required by applicants may have contributed to their high employment rates.10 Madison Strategies Group control group members had the highest employment rates in both the Year 2 Survey (95 percent) and in unemployment insurance wage records 

 7

National Skills Coalition (2011). 8

Card, Kluve, and Weber (2010); D’Amico (2006). 9

At Madison Strategies Group and Towards Employment, control group members ended primary skills 

training significantly earlier (between two and three months earlier) than WorkAdvance group members. 10As discussed previously, only one in five applicants made it through the intake process. The control 

group appeared to be motivated and fairly likely to seek out other, similar services on their own (see Chapter 

3).
119

(93 percent). During much of the follow-up period there was an economic boom in the Tulsa 

region related to the oil and gas sector, which made it very easy for control group members to 

find employment and which provided a higher bar to clear for the program to produce impacts. 

Among the control groups, Towards Employment had the lowest employment rate as reported 

by the survey (88 percent), and St. Nicks Alliance had the lowest employment rate from unemployment insurance wage records (84 percent). From Year 1 to Year 2, employment rates for 

control group members increased somewhat at two sites and decreased somewhat at the other 

two sites. However, earnings levels increased at all four sites during the same time period, suggesting that some control group members who stayed employed were able to increase their 

earnings. Control group members at Madison Strategies Group had the highest average earnings 

in Year 2 ($14,804) and the highest rate of control group members earning $20,000 or more (31 

percent). 

Impacts on Employment and Earnings

This section presents employment and earnings impacts separately for the two data sources: the 

Year 2 Survey and unemployment insurance wage records.11

Employment and Earnings Impacts from the Year 2 Survey

Table 5.1 presents impacts on employment and wages as measured by the Year 2 Survey. Survey respondents were allowed to report up to eight jobs that they had held since random 

assignment. The first four measures in Table 5.1 on employment status and the amount of time 

employed look across all the jobs reported by survey respondents. The remaining measures in 

the table represent only survey respondents’ current or most recent jobs at the time of the survey 

interview.12 Most of the measures in the table cover the entire follow-up period, except for the 

 11Appendix F includes supplementary exhibits showing employment and earnings impacts. All estimates 

presented in this chapter were regression adjusted using ordinary least squares, controlling for pre-random assignment characteristics of sample members (referred to as covariates). The covariates differ somewhat by site, 

but in general include random assignment year, gender, race/ethnicity, age, education attainment, number of 

children, whether born in the state, previous license or certification in the targeted industry, employment status 

at baseline, monthly family income, and employment and earnings history before random assignment. Additional information on covariate selection can be found in Appendix A. 12For sample members who were employed at the time of the survey interview, these measures capture the 

job in which they were working at that time (their “current job”). If sample members were working two or 

more jobs at the time of the survey interview, the job where they worked the most hours was considered their 

“current job.” For sample members who had worked since random assignment but were not working at the 

time of the survey interview, these measures capture the job that ended most recently before the time of the 

survey interview (their “most recent job”). Impacts on characteristics of the current job are shown in Appendix 

Table F.1.
The WorkAdvance Study

Table 5.1

Year 2 Impacts on Employment and Wages, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Difference Difference Difference Difference

Outcome (%) WA C (Impact) WA C (Impact) WA C (Impact) WA C (Impact)

All jobs

Ever employed 

Currently employed

94.5

74.4

90.1

64.4

4.4 *

10.0 ***

90.0

65.3

90.8

65.4

-0.8

-0.1

96.1

77.6

94.6

72.0

1.5

5.6

89.2

64.5

87.5

58.2

1.7

6.3

Percentage of months employeda 57.0 53.2 3.8 53.7 52.7 0.9 65.7 67.6 -2.0 52.5 57.5 -5.0 *

a Employed 6 or more consecutive months 73.1 65.7 7.5 * 68.7 61.6 7.1 77.1 79.7 -2.6 62.1 66.4 -4.3

Current or most recent job

Employed in targeted sector 61.1 20.4 40.7 *** 31.8 19.9 11.9 *** 64.6 48.2 16.5 *** 50.1 33.6 16.4 ***

Earnings

Average total earnings per week ($) 561 463 98 *** 520 504 16 551 525 27 379 360 19

Average hourly wage ($) 15.23 12.85 2.38 *** 13.07 13.34 -0.27 12.47 12.01 0.46 10.11 9.78 0.33

Employed and hourly wage above $12.00

Employed and hourly wage above $15.00

63.1

46.0

46.8

29.8

16.3 ***

16.2 ***

53.3

27.7

50.9

26.9

2.4

0.8

53.2

23.6

41.9

17.0

11.2 ***

6.6 *

28.5

8.7

26.7

11.5

1.8

-2.8

Hours

Average hours worked per week (#) 34.3 33.2 1.1 36.0 34.1 1.9 42.0 40.5 1.4 33.1 31.6 1.5

b Employed full time 68.1 62.6 5.5 70.1 66.3 3.8 82.3 79.4 2.9 60.4 53.7 6.7

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group.

Sample sizes may vary because of missing values.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. aMeasures reported in time intervals cover only the first 18 months following each sample member's month 

follow-up period).

bFull-time employment is defined as working 35 hours or more in a week.

of random assignment (the common 

120
121

percentage of months employed and whether the respondent was employed for six or more consecutive months; for consistency, these two measures cover only the common follow-up period 

— the first 18 months following each sample member’s month of random assignment.

Targeted Sector Employment from the Year 2 Survey

• At all four providers, WorkAdvance increased employment in the targeted sector relative to the control groups.

Among the providers, Per Scholas produced by far the largest impact on employment in 

the targeted sector: 61 percent of WorkAdvance group members reported working in the information technology (IT) sector at their current or most recent job, a striking 41 percentage points 

higher than the control group. St. Nicks Alliance had the lowest rate of WorkAdvance group 

members who reported working in the targeted sector (environmental remediation) at 32 percent, but the impact of 12 percentage points is still statistically significant.13 Almost two-thirds 

of WorkAdvance group members at Madison Strategies Group and half of WorkAdvance group 

members at Towards Employment were employed in the targeted sector, compared with 48 

percent and 34 percent, respectively, of control group members (Table 5.1). The high control 

group level at these two sites might be due to lower barriers to entry into those sectors, compared with the barriers to entry into the IT and environmental remediation sectors.

While it is impressive that all providers were able to move a significant proportion of 

participants into jobs in the targeted sectors, this represents a necessary but not sufficient condition for impacts on overall employment, earnings, and advancement. For these effects to have 

occurred, wages and benefits at jobs in the targeted sector would have to be better than at jobs 

outside the targeted sector. The rest of this chapter shows whether there are impacts on employment, earnings, job characteristics (such as schedules and job satisfaction), and advancement, and the next chapter provides an indication of whether targeted sector jobs have better 

characteristics (wages, hours, and so on) than jobs outside the targeted sector.

Overall Employment and Current Employment from the Year 2 Survey

The first two measures in Table 5.1 show the percentage of survey respondents ever

employed since random assignment and the percentage of respondents employed at the time of 

the survey interview (this is, on average, Month 22). The latter provides an indication of employment retention.

 13It is important to note that the measure used here indicates whether employment in the current or most 

recent job is in the targeted sector. Because St. Nicks often places individuals in short-term project work and 

individuals might move in and out of the sector, it is perhaps more likely that this measure might underestimate 

targeted sector employment at that site. 
122

• Only the WorkAdvance program at Per Scholas produced a statistically 

significant increase in employment since random assignment and in current employment at the time of the Year 2 Survey, relative to the control 

group. Towards Employment and Madison Strategies Group produced 

positive, although not statistically significant, impacts on current employment.

Table 5.1 shows that around 90 percent of control group members at all four sites were 

ever employed since random assignment. Despite the high control group employment rate, Per 

Scholas WorkAdvance group members were 4 percentage points more likely to have ever 

worked than control group members. Almost 95 percent of Per Scholas WorkAdvance group 

members worked since random assignment compared with 90 percent of control group members. Per Scholas was the only provider to produce a statistically significant impact on overall 

employment measured by the survey.

At all four sites, around 20 percent to 25 percent of respondents in both research groups 

lost a job following study entry and did not obtain a new job by the time of the survey interview 

(measured as the difference between “ever employed” and “currently employed”). Per Scholas 

was the only provider to produce an impact on the current employment measure — 74 percent 

of WorkAdvance group members at the site were currently employed at the time of the survey 

interview, an increase of 10 percentage points over the control group. The WorkAdvance programs at both Madison Strategies Group and Towards Employment produced increases in current employment relative to the control group, and while the estimated impacts were not statistically significant, both are just above the cutoff for statistical significance.14

Earnings and Hours from the Year 2 Survey

• The WorkAdvance program at Per Scholas produced impacts on hourly 

wages and weekly earnings. At Madison Strategies Group, the WorkAdvance program increased the percentage of respondents working and 

earning more than $12 per hour. WorkAdvance had no effect on wages 

or earnings in the current or most recent job at the other two sites.

The estimated weekly earnings for Per Scholas WorkAdvance group members were 

$561, a statistically significant increase of $98, or 21 percent higher than the control group average. WorkAdvance group members at the site also reported significantly higher hourly wages 

($15.23) compared with the control group ($12.85) and were 16 percentage points more likely 

than control group members to be employed with a wage above $15 per hour (Table 5.1). 

 14The cutoff for statistical significance used in this report is 0.100. The p-value on current employment at 

Madison Strategies Group is 0.122 and at Towards Employment is 0.109. With slightly larger sample sizes, 

both of these estimated impacts would have been statistically significant, assuming constant effect sizes.
123

Madison Strategies Group did not have an effect on weekly earnings or average hourly 

wages,15 but the provider did increase the percentage of respondents employed since random 

assignment who earned above $12 per hour and who earned above $15 per hour. Around 53 

percent of WorkAdvance group members at Madison Strategies Group found a job with a wage 

above $12 per hour, compared with 42 percent of control group members, a statistically significant increase of 11 percentage points.

At Towards Employment and St. Nicks Alliance, WorkAdvance did not produce statistically significant impacts on wages or weekly earnings. Average hourly wages and weekly 

earnings were lowest for both research groups at Towards Employment, perhaps reflecting the 

economic conditions in the local labor market of the site — Cleveland, Ohio — which has experienced a prolonged period of deindustrialization. WorkAdvance at Towards Employment did 

increase the likelihood of working at wages above $10 per hour by 4 percentage points (not 

shown), but this increase was not statistically significant. WorkAdvance group members at Towards Employment earned an average of $10.11 per hour at their current or most recent job, 

while control group members earned an average of $9.78. 

• WorkAdvance did not produce statistically significant impacts on work 

hours at any of the sites.

The majority of respondents at all four sites worked full time (at least 35 hours per 

week). Respondents at Madison Strategies Group worked the most hours per week across the 

sites — an average of 42 hours per week for WorkAdvance group members and an average of 

41 hours per week for control group members (Table 5.1).

Employment and Earnings Impacts from Unemployment Insurance 

Wage Data 

This section presents employment and earnings impacts based on unemployment insurance wage data. These findings differ slightly from the findings from the Year 2 Survey because 

of differences in samples and jobs covered, but overall, the findings are consistent. Table 5.2 

presents impacts over several time periods. The first set of measures captures the full follow-up 

period — Quarters 2 to 10. Because, as discussed earlier, part of the follow-up period includes 

 15When measured based on current or most recent jobs, which was the prespecified outcome of primary 

interest in this report, these effects are not statistically significant. However, looking only at current jobs (held 

at the time of the survey interview), the WorkAdvance program at Madison Strategies Group did increase 

weekly earnings by a statistically significant amount (as shown in Appendix Table F.1).
115 

124

The WorkAdvance Study

Table 5.2

Quarters 2 to 10 Impacts on Unemployment Insurance-Covered Employment and Earnings, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Quarters 2-10

Ever employed (%)

Average quarterly employment (%)

Earnings ($)

Year 1 (Quarters 2-5)

Ever employed (%)

Average quarterly employment (%)

Employed in all 4 quarters (%)

Earnings ($)

Earned $20,000 or more (%)

Year 2 (Quarters 6-9)

Ever employed (%)

Average quarterly employment (%)

Employed in all 4 quarters (%)

Earnings ($)

Earned $20,000 or more (%)

Quarter 10

Ever employed (%)

Earnings ($)

89.0 86.1

61.5 57.4

32,470 27,284

77.1 71.0

53.3 51.1

24.5 28.0

8,868 8,718

13.2 13.5

82.4 76.4

67.5 61.8

50.4 43.4

18,217 14,471

37.8 29.3

70.4 64.3

5,385 4,096

2.9

4.2 *

5,186 **

6.1 *

2.2

-3.5

150

-0.3

6.0 **

5.7 *

7.0 *

3,747 ***

8.5 **

6.0 *

1,289 ***

87.6 83.5

56.7 55.2

27,782 27,982

78.2 70.4

52.8 50.6

20.8 27.7

9,395 9,648

13.7 17.7

74.3 72.2

59.8 57.8

44.7 39.2

14,420 14,229

32.2 27.4

60.4 63.3

3,966 4,105

4.1

1.5

-201

7.7 **

2.2

-6.9 *

-253

-4.0

2.1

2.0

5.5

191

4.8

-2.9

-139

92.8 92.8

69.9 67.5

33,759 31,207

89.3 86.5

72.6 68.8

50.1 48.8

13,261 12,933

26.2 23.4

80.3 82.8

68.4 67.6

55.3 52.5

16,641 14,804

40.5 31.1

64.8 61.7

3,857 3,470

0.0

2.4

2,552

2.8

3.8

1.3

328

2.8

-2.5

0.8

2.8

1,837 *

9.4 ***

3.1

387

92.5 86.6

65.3 62.2

26,036 24,227

83.6 78.8

61.9 61.4

38.1 42.4

9,495 9,483

14.2 17.0

79.1 73.7

67.6 62.5

53.0 48.1

13,223 11,602

28.0 22.5

68.9 64.0

3,318 3,142

5.9 ***

3.1

1,809

4.8 *

0.5

-4.3

12

-2.8

5.4 *

5.1 *

4.9

1,621 *

5.5 *

4.9

176

Sample size (total = 2,564) 349 341 242 237 353 344 349 349

SOURCES: MDRC calculations from unemployment insurance administrative records provided by New York State Department of Labor for Per Scholas and 

St. Nicks Alliance sample members; Ohio Department of Job and Family Services for Towards Employment sample members; and Oklahoma Employment 

Security Commission for Madison Strategies Group sample members. Table Svy.1 (continued)

NOTES: WA = WorkAdvance (program) group; C = control group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Sample sizes may vary because of missing values.
125

the time participants spent in occupational skills training, these measures may not fully represent the effects of the WorkAdvance programs. The remaining sets of measures break the follow-up period up into pieces — Year 1 (Quarters 2 to 5),16 Year 2 (Quarters 6 to 9), and the first 

quarter of Year 3 (Quarter 10) — to help isolate any impacts observed in the post-occupational 

skills training period.17 Figure 5.1 and Figure 5.2 present impacts on employment and earnings, 

respectively, in unemployment insurance-covered jobs by quarter relative to random assignment 

for each site. 

• Consistent with the findings from the Year 2 Survey, the WorkAdvance 

program at Per Scholas produced impacts on employment in and earnings from unemployment insurance-covered jobs. The impact on earnings grew stronger over time.

In the quarter of random assignment and the quarter following random assignment 

(Quarter 2), more control group members at Per Scholas were employed than WorkAdvance 

group members (see Figure 5.1). This was expected, given that many WorkAdvance group 

members were enrolled in occupational skills training in those quarters. By Quarter 3, though, 

WorkAdvance group members were more likely to be employed (although the difference is not 

statistically significant). Looking at Year 1 overall, 77 percent of Per Scholas WorkAdvance 

group members were ever employed, compared with 71 percent of control group members, a 

statistically significant increase of 6 percentage points (Table 5.2). This effect on employment 

was similar in Year 2 and the first quarter of Year 3.

In the full follow-up period, WorkAdvance group members at Per Scholas earned 

$32,470 compared with average earnings of $27,284 for control group members, a statistically 

significant increase of $5,186 (Table 5.2). Most of this increase in earnings came later in the 

follow-up period. Figure 5.2 shows that at Per Scholas, WorkAdvance group members did not 

begin earning more than control group members until Quarter 4 — slightly later than the trend 

seen for employment. After Quarter 4, earnings increased for the most part for both research 

groups, but WorkAdvance group members earned more in all four quarters of Year 2 and in the 

first quarter of Year 3. In the first quarter of Year 3, WorkAdvance group members earned 

$1,289, or 31 percent, more than the control group. Combining this with the trend seen on employment, it appears Per Scholas WorkAdvance group members had higher earnings than control group members because they had higher wages (as the survey suggests), and not just because more of them were working.

 16Quarter 1 is the quarter in which sample members were randomly assigned. Depending on when in the 

quarter a sample member was randomly assigned, some employment and earnings in that quarter may precede 

the sample member’s entry into the study. For that reason, Year 1 measures do not include Quarter 1. 17An alternative way to break up the follow-up period would be into Year 1 (Quarters 2 to 5) and the postvocational training period (Quarters 6 to 10). The Quarters 6 to 10 measures are not shown in the report, but in 

general, they follow a similar pattern to that seen in the Year 2 impact estimates.
126

(continued)

The WorkAdvance Study

Figure 5.1

Quarters 1 to 10 Impacts on Percentage Employed in an Unemployment Insurance-Covered Job,

 by Relative Quarter and Site

0

20

40

60

80

100

RA 2

**

3 4 5

***

6 7 8 9

**

10

*

Quarter relative to random assignment

Per Scholas

WA group

Control group

0

20

40

60

80

100

RA

***

2

*

3 4 5 6 7 8 9 10

Quarter relative to random assignment

St. Nicks Alliance

WA group

Control group

Percentage

0

20

40

60

80

100

RA 2 3 4

**

5 6 7 8 9 10

Quarter relative to random assignment

Madison Strategies Group

WA group

Control group

Percentage

0

20

40

60

80

100

RA

**

2 3 4 5 6 7 8

**

9

**

10

Quarter relative to random assignment

Towards Employment

WA group

Control group

Percentage

Percentage
127

• WorkAdvance group members at St. Nicks Alliance were more likely 

than control group members to have worked in an unemployment insurance-covered job in Year 1, but the effect faded by Year 2. No statistically significant effects were measured on earnings.

In Year 1, WorkAdvance group members at St. Nicks Alliance were more likely to be 

working than control group members — 78 percent of WorkAdvance group members compared with 70 percent of control group members were employed. A negative impact is seen on 

employment stability during that time period. Only 21 percent of WorkAdvance group members were employed in all four quarters of Year 1 compared with 28 percent of control group 

members. This may relate to the project-based nature of environmental remediation jobs. Workers in the sector often hold shorter-term jobs and have to seek employment on a new project or 

with a new employer after one job ends. By Year 2, the estimated effects on both employment 

and employment stability at St. Nicks Alliance had faded to statistical insignificance (Table 

5.2).

There is no statistically significant difference on earnings over the full follow-up period 

at St. Nicks Alliance. WorkAdvance group members earned an average of $27,782, $201 less 

than control group members (Table 5.2). Figure 5.2 shows that the two research groups had 

similar earnings throughout most of the follow-up period. 

• At Madison Strategies Group (one of the providers that initially offered 

the placement-first track), the WorkAdvance program produced impacts on earnings in Year 2. The program had little effect on employment, perhaps because the control group had the highest rate of unemployment insurance-covered employment in the full follow-up period 

across the sites. 

Figure 5.1 (continued)

SOURCES: MDRC calculations from unemployment insurance administrative records provided by 

New York State Department of Labor for Per Scholas and St. Nicks Alliance sample members; 

Ohio Department of Job and Family Services for Towards Employment sample members; and 

Oklahoma Employment Security Commission for Madison Strategies Group sample members. 

NOTES: RA = random assignment; WA group = WorkAdvance (program) group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 

percent.

Sample sizes may vary because of missing values.
(continued)

Mean earnings ($)

Per Scholas St. Nicks Alliance

6,000 6,000

5,000 5,000

WA group

4,000 4,000

3,000 3,000

Control group Control group 2,000 2,000

1,000 1,000 WA group

0 0 Mean earnings ($)

RA 2 3 4 5 6 7 8 9 10 RA 2 3 4 5 6 7 8 9 10

***   **  *** *** *** **

Quarter relative to random assignment Quarter relative to random assignment

6,000 Madison Strategies Group Towards Employment

6,000

 s ($)

5,000

WA group

5,000

 s ($)

gin

4,000 4,000 Control group gin

a earn 3,000 Control group 3,000 a earn

2,000 2,000

Me n

1,000

Me n

WA group 1,000

0 0

RA 2 3 4 5 6 7 8 9 10 RA 2 3 4 5 6 7 8 9 10

* * * ** * **

Quarter relative to random assignment Quarter relative to random assignment

The WorkAdvance Study

Figure 5.2

Quarters 1 to 10 Impacts on Mean Earnings from an Unemployment Insurance-Covered Job, by Relative Quarter and Site

128
129

The WorkAdvance program at Madison Strategies Group did not produce an impact on 

employment in any of the follow-up periods shown in Table 5.2. Figure 5.1 shows that the percentage of WorkAdvance group members employed peaked in Quarter 4 and then began to decrease through Quarter 10. The employment rate for control group members remained fairly 

consistent throughout the follow-up period, and in most quarters, fewer (although the differences are not statistically significant) control group members were working than WorkAdvance 

group members.18 These high control group employment rates are probably a reflection of the 

strong economic conditions in Tulsa during the study period (see Appendix Figure B.1).

Although there was no effect on employment, the story on earnings is more positive. 

The WorkAdvance program at Madison Strategies Group increased earnings by $2,552 over the 

control group average during the full follow-up period, although this increase is not statistically 

significant. WorkAdvance group members earned an average of $33,759 in this time period, 

compared with earnings of $31,207 for control group members. When looking just at Year 2 

(the period after occupational skills training), however, WorkAdvance produced statistically 

significant impacts on earnings and earnings above $20,000. WorkAdvance group members 

earned an average of $16,641 in Year 2, an increase of $1,837 (or 12 percent) over the control 

group average. An impact of this size is on par with impacts measured in older job training 

evaluations, such as the Job Training Partnership Act, but at this point in the follow-up period is 

far weaker than the impacts measured in the Sectoral Employment Impact Study.

19 Figure 5.2 

shows that WorkAdvance group members earned significantly more than control group members in Quarters 8 and 9.

 18Appendix Figure F.4 shows impacts on unemployment insurance-covered employment in the targeted 

sectors — manufacturing and transportation — at Madison Strategies Group. Beginning in the quarter of random assignment, employment in the targeted sector increased for both research groups, but WorkAdvance 

group members were significantly more likely to be employed in the targeted sector than control group members throughout the follow-up period. 19Bloom et al. (1997); Maguire et al. (2010).

Figure 5.2 (continued)

SOURCES: MDRC calculations from unemployment insurance administrative records provided by 

New York State Department of Labor for Per Scholas and St. Nicks Alliance sample members; Ohio 

Department of Job and Family Services for Towards Employment sample members; and Oklahoma 

Employment Security Commission for Madison Strategies Group sample members. 

NOTES: RA = random assignment; WA group = WorkAdvance (program) group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 

percent.

Sample sizes may vary because of missing values.
130 

• The WorkAdvance program at Towards Employment (the other provider that initially offered the placement-first track) produced an impact 

on employment in Year 1. This effect on employment grew stronger in 

Year 2, when the provider also produced impacts on earnings. 

At Towards Employment, WorkAdvance produced an increase in the percentage of 

WorkAdvance group members who worked in Quarters 2 to 10 compared with the control 

group: 93 percent of WorkAdvance group members were employed compared with 87 percent 

of control group members (see Table 5.2).20 Figure 5.1 shows that the employment rate was 

fairly consistent across research groups through Quarter 6. After that, the employment rate for 

the control group dropped somewhat, while the employment rate rose for the WorkAdvance

group. WorkAdvance group members worked at a significantly higher rate than control group 

members in Quarters 8 and 9.

The effects on earnings follow a similar pattern. Figure 5.2 shows that in Quarters 7 and 

8, Towards Employment WorkAdvance group members earned significantly more than control 

group members.21 In Year 2 overall, Towards Employment produced an impact on earnings and 

high earnings (Table 5.2). Around 28 percent of WorkAdvance group members earned $20,000 

or more in Year 2, a statistically significant increase of 6 percentage points over the control 

group.

Overall Assessment

This section shows that the impacts on employment and earnings varied across the sites. 

Whenever there are mixed findings across multiple outcomes it can be difficult to assess the 

overall results. The results at this stage for Per Scholas and St. Nicks Alliance are clear — the 

former produced large impacts on employment, wages, and earnings, while the latter produced 

no impacts on employment or earnings. However, the results at Madison Strategies Group and 

Towards Employment are mixed, with evidence of stronger impacts emerging in Year 2. 

One very conservative approach to assessing the overall results is to use what are 

known as “confirmatory outcomes.” Confirmatory analyses involve the estimation of impacts 

on a limited set of prespecified outcomes, which are subjected to multiple comparisons adjust-

 20Appendix Figure F.4 shows that throughout the follow-up period, the impacts on employment at Towards Employment for the manufacturing sector are larger and more consistent than the impacts for the health 

care sector. WorkAdvance increased employment in the manufacturing sector (among sample members randomly assigned into the manufacturing sector program) from Quarter 3 to Quarter 10. 21Appendix Table F.4 shows impacts on employment and earnings by random assignment sector at Towards Employment. Among sample members randomly assigned in the manufacturing sector, WorkAdvance 

increased employment in an unemployment insurance-covered job in Year 2. In the first quarter of Year 3, the 

estimated impact on employment among sample members randomly assigned in the health care sector is statistically significant.
131

ments.22 This type of analysis measures the overall success of a program at a very high standard 

of evidence. Table 5.3 shows the impacts on the confirmatory outcomes prespecified in WorkAdvance, and Box 5.1 provides further explanation of the analysis and the results. From this 

perspective, and at this interim stage, only Per Scholas passed the confirmatory impacts test. 

Still, despite not producing impacts on the confirmatory measures, the evidence above shows 

that Madison Strategies Group and Towards Employment did increase employment and earnings, particularly late in the follow-up period. The rest of this chapter will show that these two 

sites also produced impacts on measures of other job characteristics and advancement.

Impacts on Wage Growth and Advancement

One of the key innovations of the WorkAdvance model was the addition of advancementfocused postemployment services to sectoral training. Many low-wage workers report that their 

jobs offer few opportunities for advancement and will not help them meet their career goals.23

To combat this, the postemployment services were designed to provide coaching to help employed participants retain and advance in their jobs.24 Table 5.4 presents impacts on financial 

and nonfinancial advancement at sample members’ current or most recent jobs. The WorkAdvance evaluation as a whole was set up to test the full array of WorkAdvance services, and not 

the effectiveness of postemployment services or advancement coaching on their own. Therefore, it is hard to say with confidence that any effects are due specifically to the postemployment services. 

The primary measure of advancement in Table 5.4 is the change in sample members’ 

hourly wages since study entry. This measure compares survey respondents’ hourly wages at 

the time of the survey interview to their most recent hourly wages reported at the time they entered the study. Changes in hourly wages were measured only for respondents who had worked 

within the two years preceding study entry and who were currently employed at the time of the 

survey interview.

The other advancement measures presented in Table 5.4 are based on nonfinancial job 

characteristics and are related to changes in job title and increases in job responsibilities and 

skills. As discussed earlier, it was hypothesized that little, if any, advancement in terms of wage 

growth would occur in the short follow-up period covered by the Year 2 Survey. The nonfinancial advancement measures, however, represent early indicators of future wage growth or signs 

of higher-quality employers and jobs. One of the main potential benefits of sectoral programs, 

such as WorkAdvance, is their connection with higher-quality employers. Participants who are 

 22See Olken (2015) for a discussion. 23Tompson, Benz, Agiesta, and Junius (2013). 24This coaching was supposed to occur throughout the entire WorkAdvance program.
132

The WorkAdvance Study

Table 5.3

Impacts on Confirmatory Outcomes, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Difference Difference Difference Difference

Outcome WA C (Impact) WA C (Impact) WA C (Impact) WA C (Impact)

Among respondents to the Year 2 Survey

Currently employed (%) 74.4 64.3 10.1 *** 65.2 65.6 -0.5 77.5 72.1 5.4 64.4 58.3 6.1

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

Among full research sample

Earnings in first quarter of Year 3 ($) 5,416 4,064 1,351 *** 3,938 4,133 -195 3,863 3,463 400 3,314 3,146 167

Sample size (total = 2,564) 349 341 242 237 353 344 349 349

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance administrative records from New 

York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. Gray shading indicates that the statistical test 

withstood the Westfall-Young adjustment for multiple comparisons. After the adjustment, the impact on current employment at Per Scholas is significant at the 

10 percent level and the impact on earnings in the first quarter of Year 3 is significant at the 1 percent level.

Sample sizes may vary because of missing values.
133

Box 5.1

Confirmatory Economic Outcomes

In recent years, the issue of multiple test bias has become more prominent in both the academic literature and the field of program evaluation more generally. Every time one estimates an impact on 

an outcome, there is a precisely defined probability (conventionally 10 percent in such studies as

WorkAdvance) of concluding that a program has had an impact when the observed difference is 

simply due to chance. Since researchers typically examine many outcomes, the probability that at 

least one estimate will be statistically significant, simply by chance, can get very high.

One approach to this problem is to conduct fewer impact estimates and to state in advance which 

tests will be conducted.* The WorkAdvance research team followed this approach, prespecifying 

two measures in the analysis planning phase — current employment at the time of the Year 2 Survey and earnings in an unemployment insurance-covered job using the last quarter of available data 

— as the most likely to be affected if the program were successful.† Impact estimates on these 

measures were also subjected to Westfall-Young multiple comparisons correction. (See Appendix 

A for more details.) 

Statistically significant impacts on these “confirmatory” measures represent the highest level of evidence of the success of the programs with the available amount of follow-up data. If the program 

does not pass the confirmatory test but produces statistically significant impacts on other measures, 

it does not mean that the program was unsuccessful or that it will not lead to increases in employment or earnings in the future. It simply means that the program passed a lower standard of evidence given the available amount of follow-up data. 

Table 5.3 shows that at the Per Scholas site, 74 percent of WorkAdvance group respondents were 

currently employed at the time of the Year 2 Survey compared with 64 percent of control group respondents, a statistically significant increase of 10 percentage points. Further, WorkAdvance increased earnings in an unemployment-insurance covered job in the first quarter of Year 3 by 33 

percent over the control group average.‡ WorkAdvance group members earned an average of 

$5,416 in the first quarter of Year 3, compared with average earnings of $4,064 for control group 

members. Both of these impacts withstood the Westfall-Young adjustment.§

At Madison Strategies Group and Towards Employment, the WorkAdvance programs did increase 

both employment and earnings, although not by a statistically significant amount. Negative differences are seen on both confirmatory outcomes at St. Nicks Alliance.

__________________________ 

*

Olken (2015).

†

These measures were selected because employment and earnings are the key outcomes in the WorkAdvance 

logic model and the selected points allow for the most time to pass after sample members engaged in occupational

skills training. Initially, the earnings measure chosen was for Quarter 7. However, as more follow-up data became 

available, it was decided to stick with the concept of using the latest quarter of UI wage data as the confirmatory 

measure.

‡

The results in Table 5.3 differ slightly from the results presented elsewhere in this chapter. In order to perform 

the Westfall-Young correction, a different set of common, cross-site covariates was used to obtain the impact estimates in Table 5.3. The other tables in this chapter used site-specific covariates to estimate the site-specific impacts.

§

After the adjustment, the impact on current employment is significant at the 10 percent level and the impact 

on earnings in the first quarter of Year 3 is significant at the 1 percent level.
The WorkAdvance Study

Table 5.4

Year 2 Impacts on Wage Growth and Advancement in Current or Most Recent Job, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Difference

Outcome (%) WA WA C (Impact)

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Change in hourly wage since RA

Did not have a job before RA and/or a not employed at survey interview

Had a job before RA and employed

at survey interview and:

Hourly wage increased by:

$8.00 or more

$5.00-$7.99

$2.00-$4.99

Hourly wage within $2.00 of 

baseline wage

Hourly wage decreased

Job skills at current or most recent job 

Scope of work has increased

Job title changed since job started

Offered many opportunities for career 

advancement

Obtained new skills while working job 

37.9

16.4

11.3

8.9

15.9

9.7

50.0

19.4

66.1

72.3

44.9

8.4

8.1

11.3

15.0

12.4

39.7

15.2

53.1

61.2

††

-7.0

8.0 **

3.2

-2.4

0.9

-2.7

10.3 **

4.2

13.0 ***

11.2 ***

47.5

8.1

7.1

13.0

12.3

12.1

37.6

15.0

61.1

65.0

43.0

8.6

3.7

11.2

17.0

16.6

40.4

14.8

59.3

56.0

4.5

-0.5

3.3

1.9

-4.7

-4.5

-2.8

0.2

1.8

9.0 *

28.2

10.0

11.3

20.9

22.7

6.8

47.0

19.6

66.1

67.4

31.9

6.3

9.2

17.4

25.2

10.0

49.2

25.3

55.4

59.4

-3.7

3.7

2.1

3.5

-2.5

-3.2

-2.3

-5.8

10.7 ***

8.0 **

46.8

1.5

4.3

16.1

24.9

6.3

42.7

15.0

55.6

56.2

49.2

2.5

5.7

7.8

25.3

9.6

33.7

16.2

47.2

56.0

†

-2.4

-1.0

-1.4

8.3 **

-0.3

-3.2

9.0 **

-1.2

8.4 **

0.2

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group; RA = random assignment.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall-Young adjusted p-values were used 

for categorical measures.

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 

follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Sample sizes may vary because of missing values.

a

Sample members who did not work within the two years preceding RA or who were not employed at the time of the survey interview are included in this 

category.

134
135

placed in jobs — even if the jobs are entry level or initially low wage — with employers who 

offer advancement opportunities have the potential for earnings gains in the future. It should be 

noted that any impacts seen on these measures may have occurred either because the jobs obtained by WorkAdvance group members did offer more opportunities for advancement or because WorkAdvance group members were more aware of and likely to seek out advancement 

opportunities than control group members since they were trained by WorkAdvance providers 

to do so. Either way, the impacts are noteworthy.

• WorkAdvance at Per Scholas resulted in more WorkAdvance group 

members than control group members having increased their wages and 

holding jobs with opportunities for advancement. 

Over one-third of Per Scholas WorkAdvance group members reported having a higher 

wage at the time of the survey interview than the most recent wage they had at study entry. Specifically, the provider produced an impact of 8 percentage points on wage increases of $8 or 

more per hour over the control group average — 16 percent of WorkAdvance group members 

reported a wage increase at or above this level. Some of this may be due to WorkAdvance 

group members breaking into the IT sector and the associated high wages of jobs in that sector 

(evidence presented in Chapter 6 indicates that IT sector jobs paid more than non-IT sector jobs 

for the employed Per Scholas WorkAdvance respondent sample). Before entering the study, 

only 8 percent of sample members at Per Scholas had previous employment in the IT sector. 

The sector employment rate increased dramatically for the WorkAdvance group during the follow-up period — 61 percent of WorkAdvance group members were employed in the IT sector 

at their current or most recent job, compared with 19 percent of control group members (as 

shown in Table 5.2). 

Additionally, WorkAdvance at Per Scholas had statistically significant impacts on several measures of nonfinancial advancement. WorkAdvance group members were 10 percentage 

points more likely to work a job in which their scope of work had increased and 11 percentage 

points more likely to have obtained new skills while working compared with control group 

members. Further, 66 percent of WorkAdvance group members believed their job offered many 

opportunities for career advancement, an increase of 13 percentage points over the control 

group average. 

• The WorkAdvance programs at St. Nicks Alliance, Madison Strategies 

Group, and Towards Employment had little to no impact on wage 

growth relative to the control groups. However, the WorkAdvance programs at all three sites produced impacts on at least one of the nonfinancial measures of advancement. 
136

At Towards Employment, WorkAdvance group members were significantly more likely to have had a wage increase of between $2.00 and $4.99 per hour since study entry compared 

with control group members. The WorkAdvance program at Towards Employment also produced impacts of 9 and 8 percentage points, respectively, on the likelihood that WorkAdvance 

group members agreed they worked at jobs where their scope of work increased and at jobs that 

offered opportunities for career advancement.

WorkAdvance group members at St. Nicks Alliance and Madison Strategies Group 

were more likely to report that they obtained new skills while working at their current or most 

recent job, with increases of 9 and 8 percentage points, respectively, over the control group averages. In addition, Madison Strategies Group WorkAdvance group members were 11 percentage points more likely to report that their jobs offered many opportunities for career advancement compared with the control group. These nonfinancial advancement measures are characteristics of higher-quality jobs. 

Impacts on Job Benefits and Work Schedule

Although wages are often thought of as the primary indicator of high-quality jobs, other job 

characteristics, including employer-offered benefits and standard or stable work schedules, are 

also important determinants of job quality. Low-wage workers are offered employer-provided 

benefits at a lower rate than workers with higher wages. These benefits include paid leave, retirement plans, and health insurance coverage, among others.25 Further, the number of “underemployed” workers — individuals who work part time but are available to work full time — 

increased substantially during and following the Great Recession.26 Recent reports and legislation have highlighted the challenge of irregular job schedules and shifts faced by these underemployed, and often low-wage, workers.27 These types of job characteristics are part of what 

determines the quality of a job, as well as affecting workers’ assessments of job satisfaction. 

Table 5.5 presents impacts on job characteristics of survey respondents’ current or most recent 

jobs.

• At Madison Strategies Group, WorkAdvance group members reported 

more employer-offered benefits, higher levels of overall job compensation, and higher levels of job satisfaction than control group members.

 25Bureau of Labor Statistics (2014). 26Mayer (2014). 27See Golden (2015). The Schedules That Work Act was introduced in Congress in 2014 but remains in 

committee (see U.S. Congress, 2015).
137

Table 5.5

Year 2 Impacts on Current or Most Recent Job Characteristics, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Difference

Outcome (%) WA WA C (Impact)

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Very or somewhat satisfied with current

or most recent job

Employer-provided benefits 

Sick days with full pay

Paid vacation

Paid holidays

Dental benefits

Retirement plan

Health plan or medical insurance

Enrolled 

Tuition reimbursement

Hourly wage above $12 and

employer-provided health insurance

Work schedule 

Regular shifta

Rotating, split, or irregular shift

b Job type

Work for "temp" agency

Work for staffing agency

Occasional, odd job

Work for a friend or family member

Regular permanent job

75.6

40.7

37.6

41.5

38.5

36.1

42.5

70.2

20.4

33.4

73.6

19.8

14.9

19.7

14.0

5.2

71.4

63.7

36.9

34.9

36.8

35.9

28.9

39.8

59.6

18.5

27.9

63.6

25.0

9.2

8.8

15.2

5.4

64.1

11.9 ***

3.8

2.8

4.7

2.7

7.3 *

2.7

10.6

2.0

5.5

10.0 **

-5.2

5.7 **

10.9 ***

-1.2

-0.2

7.3 *

68.8

43.6

40.2

46.5

32.1

29.8

37.8

60.2

14.4

27.2

71.0

18.5

8.0

13.4

12.5

5.5

70.8

67.6

42.2

41.5

42.9

33.4

33.2

40.9

56.0

17.7

26.6

63.5

24.8

9.0

9.1

15.6

6.0

66.3

1.2

1.4

-1.2

3.6

-1.3

-3.5

-3.1

4.3

-3.3

0.6

7.5

-6.3

-1.0

4.3

-3.1

-0.6

4.5

77.7

46.8

60.3

60.7

57.4

48.1

65.2

67.8

29.0

40.3

76.9

15.9

12.5

12.8

6.7

8.5

86.3

69.2

42.0

51.6

53.7

50.9

41.5

56.4

67.9

21.1

30.4

78.4

15.2

18.0

18.7

9.6

6.5

82.0

8.6 **

4.8

8.7 **

7.0 *

6.5

6.6

8.8 **

-0.2

7.9 **

9.9 **

-1.5

0.7

-5.5 *

-6.0 *

-3.0

2.0

4.4

60.8

35.0

41.1

48.3

40.0

39.3

43.0

62.6

22.2

20.7

71.7

16.5

12.8

14.5

10.3

6.6

75.6

65.6

32.4

39.4

41.9

36.6

35.8

40.9

61.2

22.0

16.7

64.0

20.9

11.8

10.7

15.6

6.4

68.1

-4.8

2.7

1.8

6.4

3.5

3.5

2.1

1.4

0.2

4.0

7.7 *

-4.5

1.0

3.8

-5.3 *

0.2

7.5 **

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

(continued)
138

WorkAdvance group members at Madison Strategies Group reported significantly more 

employer-offered benefits than the control group, including increases of between 7 and 9 percentage points on the likelihood that respondents’ employers offered paid vacation, paid holidays, health insurance,28 and tuition reimbursement. Madison Strategies Group directly targeted 

employers and jobs that offered benefits. The provider was focused on increasing total compensation, rather than just hourly wages, and sometimes placed participants in jobs with slightly 

lower hourly wages if the job offered benefits. This is seen in the impact of almost 10 percentage points on the likelihood that respondents had a job with an hourly wage above $12 per hour 

and were offered health insurance by their employer.

WorkAdvance group members at Madison Strategies Group also reported significantly 

higher levels of job satisfaction than control group members. Around 78 percent of WorkAdvance group members reported being very or somewhat satisfied with their current or most recent job, an increase of 9 percentage points over the control group average. 

• Per Scholas WorkAdvance group members reported higher levels of job 

satisfaction and more stable work schedules than control group members. The program at Towards Employment produced increases in 

regular permanent employment (versus occasional or odd jobs).

Per Scholas WorkAdvance group members reported higher levels of job satisfaction (76 

percent) than control group members (64 percent), an impact of nearly 12 percentage points. 

 28Around 65 percent of WorkAdvance group members at Madison Strategies Group reported being offered health insurance through their employer. This is notable given that less than 30 percent of sample members at that provider reported being covered by health insurance at baseline. In another section of the Year 2 

Survey, respondents were asked whether they were covered by health insurance in the past month and if so, 

what type of health insurance they were covered by (see Appendix Table F.3). Fewer respondents in both research groups at Madison Strategies Group reported being covered by employer-based health insurance in the 

prior month than the rates shown in Table 5.5. The estimated effect is not statistically significant.

Table 5.5 (continued)

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group.

Sample sizes may vary because of missing values.

Italics indicate the metric is not among the full sample shown in the table. Therefore, the 

measure is nonexperimental and statistical tests were not performed.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 

percent. a

A regular shift includes those worked in the daytime, evening, or nighttime.

bRespondents could select more than one job type, so percentages may sum to more than 100 

percent.
139

Additionally, 74 percent of WorkAdvance group members reported having a job with a regular 

shift and 20 percent reported having a job with a rotating, split, or irregular shift (compared with 

64 percent and 25 percent of control group members, respectively). The estimated difference on 

working a regular shift is statistically significant. As mentioned, the issue of low-wage workers 

having jobs with irregular hours and shifts has been a major concern for policymakers lately,making this impact especially noteworthy.

The Per Scholas program had little effect on employer-provided benefits, with the exception of retirement plans. This may be related to the fact that entry-level jobs in the IT industry are often offered through “temp” or staffing agencies (the impacts on both of these measures 

are statistically significant). Jobs obtained through these types of agencies do not usually offer 

benefits. 

The Towards Employment program, while not affecting job satisfaction or the receipt 

of employer-offered benefits, did affect individuals’ work schedules and job type: Towards 

Employment WorkAdvance group members were 5 percentage points less likely to report that 

their job is an occasional or odd job and 8 percentage points more likely to report their job is a 

regular permanent job, compared with control group members.

• The WorkAdvance programs at St. Nicks Alliance produced no statistically significant impacts on job satisfaction, employer-provided benefits, 

and job schedules and types. 

WorkAdvance and control group members at St. Nicks Alliance reported similar levels 

of job satisfaction (69 percent and 68 percent, respectively) and employer-offered benefits.29 

Impacts on Total Income

Increases in employment and job earnings do not always translate into higher income levels for 

low-wage workers.30 As individuals make the transition from unemployment to employment or 

to employment with higher wages, they often lose eligibility for public benefits.31 In that case, 

job earnings may merely replace previous income from public assistance, rather than increase 

 29Notably, though, St. Nicks Alliance had the highest rate of respondents employed in a unionized job 

across the four sites — 21 percent of WorkAdvance group members and 20 percent of control group members 

(see Appendix Table F.2). While the difference across research groups is not statistically significant, it is still 

important to note that WorkAdvance group members were obtaining unionized jobs, as this is a sign of job 

security. This is probably a reflection of the provider’s focus on the environmental remediation sector. 30Total income includes income from job earnings, Supplemental Nutrition Assistance Program (food 

stamps), Social Security Disability Income or Social Security Income, Temporary Assistance for Needy Families (welfare), retirement funds or Social Security, unemployment insurance benefits, child support, and other 

sources. 31Bloom and Michalopoulos (2001).
140

total income. While there can be a societal benefit to a reduction in the use of public assistance 

programs, individuals are not always better off financially. Table 5.6 presents WorkAdvance’s 

impacts on income and sources of income in the month before the survey interview. Survey respondents were asked to report both their own and their household’s previous month’s income, 

as well as the sources of their household’s income.32

• At Per Scholas, WorkAdvance increased the average prior month’s income by 33 percent over the control group average. WorkAdvance also 

reduced use of several of the most common public assistance programs.

The average monthly income for Per Scholas WorkAdvance group members was 

$1,580, a large and statistically significant increase of $396 over the control group average of 

$1,185. The bulk of this income appears to stem from job earnings: 80 percent of Per Scholas 

WorkAdvance group members reported having job earnings in the prior month, an increase of 

12 percentage points over the control group average. Further, WorkAdvance at Per Scholas decreased respondents’ use in the prior month of three of the most common public assistance programs — Supplemental Nutrition Assistance Program (SNAP) benefits (food stamps), Temporary Assistance for Needy Families (TANF), and unemployment insurance benefits33 — by between 4 and 11 percentage points. These findings highlight that the WorkAdvance program at 

Per Scholas was able to increase income while also reducing use of public assistance programs, 

a feat only rarely accomplished by employment and training programs.34

• The WorkAdvance programs at St. Nicks Alliance, Madison Strategies 

Group, and Towards Employment had little to no effect on total income.

The income levels for both research groups at Towards Employment are much lower 

than those at the other three sites. WorkAdvance group members at Towards Employment reported an average income of $1,002 in the month before the survey interview, $17 less than 

control group members, but this effect is not statistically significant. WorkAdvance did increase 

 32The WorkAdvance Year 2 Survey defined a household as the survey respondent and the survey respondent’s live-in spouse or partner, if applicable. 33Unemployment insurance benefits data were collected, in addition to the wage data, from the three state 

agencies. Appendix Figure F.5 presents impacts on receipt of unemployment insurance benefits using administrative data by quarter relative to random assignment. In the quarter of random assignment, 26 percent of Per 

Scholas WorkAdvance group members and 28 percent of control group members received unemployment 

insurance benefits (the estimated impact is not statistically significant). The percentage of sample members 

receiving unemployment insurance benefits decreased for both groups over time, and in Quarters 6 through 9 

Per Scholas WorkAdvance group members were significantly less likely to receive unemployment insurance 

benefits than control group members. 34These findings also suggest, if present trends continue, that a full benefit-cost analysis might find quite 

positive results from the Per Scholas program.
The WorkAdvance Study

Table 5.6

Year 2 Impacts on Income, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Average total respondent income in 

prior month ($)

Average total household income in prior 

montha

 ($)

Household income source (%)

Job earnings

Food stamps/SNAP

SSI/SSDI 

Welfare/TANF

Housing assistance 

Unemployment insurance 

Other 

1,580

2,166

79.7

12.8

4.1

4.0

8.9

4.5

10.8

1,185

1,754

67.6

23.8

1.6

7.9

11.3

8.1

14.9

396 ***

412 ***

12.2 ***

-11.0 ***

2.4 *

-3.9 *

-2.3

-3.6 *

-4.1

1,515

1,957

70.1

26.3

3.8

8.9

8.9

7.8

14.4

1,425

2,024

73.6

29.1

6.5

7.3

10.3

5.2

16.3

90

-68

-3.5

-2.8

-2.7

1.7

-1.4

2.6

-1.9

1,664

2,147

80.6

21.4

7.9

1.3

5.2

2.2

20.1

1,664

2,270

78.9

24.8

6.1

1.6

5.7

2.6

21.4

-1

-123

1.7

-3.5

1.8

-0.4

-0.5

-0.4

-1.3

1,002

1,406

71.8

43.3

12.4

3.7

15.4

4.3

40.6

1,019

1,417

62.5

47.1

10.5

6.0

13.8

4.7

36.5

-17

-11

9.4 **

-3.8

1.8

-2.3

1.6

-0.4

4.1

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group; SNAP = Supplemental Nutrition Assistance Program; SSI = Supplemental Security 

Income; SSDI = Social Security Disability Income; TANF = Temporary Assistance for Needy Families.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Sample sizes may vary because of missing values.

a

Household income includes income from the survey respondent and, if applicable, income from the respondent's live-in spouse or partner.

141
142 

the percentage of Towards Employment respondents receiving job earnings in the prior month, 

by 9 percentage points. This impact on income from job earnings differs from Towards Employment’s impact on current employment reported earlier in this chapter (about 6 percentage 

points, but the effect is not statistically significant). This difference may be because the income 

sources shown in Table 5.6, including job earnings, cover both the respondent and any live-in 

spouse or partner, while the current employment measure reflects only earnings from the sample member.

Neither the WorkAdvance program at St. Nicks Alliance nor the WorkAdvance program at Madison Strategies Group had a statistically significant effect on income levels. In addition, neither program reduced WorkAdvance group members’ use of public assistance programs.35 

Impacts on Life Satisfaction and Material Hardship

Many low-wage workers report feeling pessimistic about their financial situation and worry 

about being able to meet daily living expenses.36 These living expenses include rent or mortgage 

payments, bill payments, and medical expenses, among others. Table 5.7 presents impacts on 

the secondary economic outcomes of life satisfaction and material hardship. These outcomes are 

considered secondary because they are thought to be affected only after individuals first find

jobs and increase their earnings.

• WorkAdvance group members at Per Scholas reported higher levels of 

life satisfaction and of having an improved financial situation than control group members. WorkAdvance also decreased the number of material hardships respondents at Per Scholas faced in the past year.

Approximately 65 percent of Per Scholas WorkAdvance group members reported being satisfied with their lives compared with 44 percent of control group members, a statistically significant impact of 21 percentage points.37 Per Scholas WorkAdvance group members were also less 

likely than control group members to have reported facing more than five material hardships in 

 35Appendix Figure F.5 (using administrative records data) shows that at St. Nicks Alliance and Madison 

Strategies Group, the two research groups had similar levels of unemployment insurance benefits receipt. In 

Quarter 2, however, WorkAdvance group members at St. Nicks Alliance were more likely than control group 

members to receive unemployment insurance benefits. It is common that workers in the environmental remediation field apply for unemployment insurance benefits when they are between projects. 36Tompson, Benz, Agiesta, and Junius (2013). 37Survey respondents were asked on a scale from 1 to 10 — with 1 being completely dissatisfied and 10 

being completed satisfied — how satisfied they were with their lives. Responses of 7 to 10 were counted as 

“satisfied.”
143

The WorkAdvance Study

Table 5.7

Year 2 Impacts on Life Satisfaction and Material Hardship, by Site

Outcome (%) WA

Per Scholas St. Nicks Alliance Madison Strategies Group

Difference

WA C (Impact)

Towards Employment

Difference

WA C (Impact)

Difference

C (Impact)

Difference

WA C (Impact)

a Life satisfaction

Dissatisfied 

Neither dissatisfied nor satisfied 

Satisfied 

Improved financial situation

Number of material hardships in the past

b 12 months

0

1-2

3-4

5 or more

Worry often about being able to meet

normal monthly living expenses

6.6

28.3

65.1

76.0

51.7

39.7

7.8

0.8

31.1

13.9

42.3

43.8

61.6

47.6

37.4

10.1

4.9

40.8

†††

-7.4 ***

-14.0 ***

21.3 ***

14.4 ***

††

4.2

2.3

-2.3

-4.1 **

-9.7 **

10.1

32.6

57.3

70.7

46.8

33.5

14.7

4.9

48.1

11.7

39.8

48.5

63.4

43.9

35.9

15.2

5.0

46.2

-1.5

-7.2

8.8

7.3

2.9

-2.4

-0.5

-0.1

1.9

11.0

31.4

57.6

73.8

39.2

35.6

16.2

9.0

32.6

12.1

29.7

58.3

69.8

38.4

36.4

16.2

9.0

37.5

-1.0

1.7

-0.7

4.0

0.8

-0.8

0.1

0.0

-4.8

13.8

37.2

49.0

61.3

38.5

35.3

16.5

9.6

49.2

14.8

38.3

46.9

58.4

33.0

42.5

19.2

5.3

49.2

-1.0

-1.1

2.2

2.9

5.5

-7.2

-2.7

4.4

0.0

†

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group.

Sample sizes may vary because of missing values.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall-Young adjusted p-values were used 

for categorical measures.

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 

follows: ††† = 1 percent; †† = 5 percent; † = 10 percent. aRespondents were asked to report their life satisfaction on a scale from 1 to 10, with 1 being completely dissatisfied and 10 being completely satisfied. 

Respondents who answered 1-3 were counted as dissatisfied; respondents who answered 4-6 were counted as neither dissatisfied nor satisfied; and respondents 

who answered 7-10 were counted as satisfied.

bMaterial hardships include not paying the full amount of rent or mortgage due; being evicted from home or apartment; not paying the full amount due for 

gas, oil, or electricity; having gas, electrical, or oil services turned off; having phone services disconnected because payments were not made; moving in with 

family and friends; and being unable to afford a visit to a doctor or dentist.119 
144

the past year. These large impacts on secondary economic measures reaffirm the overall effectiveness of the WorkAdvance program at Per Scholas.

• Few to no significant effects were seen on life satisfaction and material 

hardship for the other three WorkAdvance programs.

At St. Nicks Alliance and Madison Strategies Group, the WorkAdvance programs produced no statistically significant effects on life satisfaction or material hardship. WorkAdvance 

group members at both sites reported having an improved financial situation compared with the 

control group, although neither provider produced a statistically significant impact on that 

measure.

The effects of the WorkAdvance program at Towards Employment on material hardship are mixed. The number of material hardships respondents faced in the past 12 months differed significantly across research groups. More WorkAdvance group members reported having 

no (39 percent) and five or more (10 percent) material hardships in the past year than control 

group members (33 percent and 5 percent, respectively). 

Conclusion

This chapter shows that the impacts on employment and earnings produced by the WorkAdvance programs varied across the sites. The story at two sites is fairly straightforward. The Per 

Scholas WorkAdvance program was successful at increasing both employment and earnings 

(including both confirmatory outcomes), as well as producing impacts on several secondary 

economic outcomes. St. Nicks Alliance had little effect on economic outcomes throughout the 

follow-up period. The story at Towards Employment and Madison Strategies Group (the two 

sites that initially operated the “placement-first track”) is somewhat more complicated. There is 

some indication that the WorkAdvance programs at these two sites increased employment, 

earnings, or both for at least part of the follow-up period. Madison Strategies Group increased 

wages above the $12 and $15 per hour thresholds, and both providers also produced improvements in job characteristics. More follow-up data is needed to clarify the longer-term pattern of 

effects at these two sites.

These findings lead to a few questions. How was Per Scholas able to produce impacts 

on employment and earnings? Why were the other three providers not able to produce as large 

or as consistent impacts? What other factors may have led to this variation in impacts? What 

was the role of the sector in explaining the variation in impacts across the sites? Did the impacts 

grow as the newer providers improved their service offerings and as the two “mixed” sites 

moved to a predominantly training-first model? Did the impacts vary across the sites because 

providers targeted different populations? The next chapter will try to answer these questions by 

examining some possible explanations for what led to the variation in impacts by sector, site, 

time, and subgroup.
145

Chapter 6

Exploring Sources of Variation in the Economic Impacts

Introduction

Chapter 5 described the impacts of WorkAdvance on economic outcomes and presented evidence that strongly suggested that the impacts varied by site: The Per Scholas program produced large impacts over several domains; the Madison Strategies Group and Towards Employment programs produced much smaller impacts that were less consistent across time and 

outcomes; and St. Nicks Alliance produced few statistically significant impacts on the economic 

outcomes. This chapter highlights several analyses that were conducted to better understand the 

mechanisms that generated these impacts. In other words, this chapter addresses the “how” and 

“why” questions behind the economic impacts. Understanding these mechanisms provides insights into the causes behind the site-by-site variation. For the WorkAdvance theory of change 

to translate into impacts, providers had to target the right sectors, recruit the right participants, 

and implement the model well. This chapter focuses on three key questions that assess whether 

variation on any of these elements might help explain site variation in economic impacts:

1. Were the impacts driven by the targeted sector? That is, were they driven by 

either the proportion of participants placed in targeted sector jobs or the characteristics of those jobs, or both? 

2. Did the impacts change over time as programs gained more experience and 

as two of the providers moved to a predominantly training-first approach?

3. Were the impacts affected by the differing characteristics of participants at 

each site?

While none of the analyses in this chapter provides a definitive explanation for why the

impacts varied as they did, the chapter gives rise to a number of hypotheses that can be investigated in future studies.

Were the Impacts Driven by the Targeted Sector?

• The weight of the evidence suggests that both the extent and the quality 

of targeted sector placements were important factors in explaining the 

pattern of impacts by site. 

A key element of the theory of change in sector programs is that by concentrating on 

specific sectors with strong labor market demand and good prospects for advancement, program 
146

staff members can develop deep relationships with employers, allowing them to place participants in better jobs than they could without this specialization. As discussed in Chapter 1, however, this mechanism can break down at several points. First, participants must complete the 

training and obtain any requisite certifications. If participants do not complete (or in some cases 

even start) training, it is unlikely that the training-first pathway of the program will achieve its 

goals.1 The theory of change could also break down if participants complete the training, but 

providers prove unable to place participants in the targeted sector. This could happen because of

a misreading of the demand in the sector or a provider’s inability to implement the aggressive 

job development — that is, establishing relationships with employers and matching participants 

with positions — that is needed to take advantage of the jobs that do exist. Finally, it is possible 

that all these elements could be in place, but the program could still fail to produce impacts if 

the jobs in the targeted sector are no better than the jobs that individuals would have found in 

other sectors. (Table 6.1 shows the typical job titles in the sectors targeted by the WorkAdvance 

programs.) This section analyzes these dimensions in turn and suggests that several of these factors were central to how the varying impacts of WorkAdvance were produced.

Impacts on Employment in the Targeted Sector 

Figure 6.1 shows the extent to which the WorkAdvance providers increased employment in the targeted sector. There is important variation by site, which parallels the overall pattern of impacts shown in Chapter 5. Per Scholas increased employment in the targeted sector by 

over 40 percentage points. The increase at St. Nicks Alliance is 12 percentage points. Both 

Madison Strategies Group and Towards Employment increased targeted sector employment by

16 percentage points.2

 

Activity Flow in Targeted Sector

Figure 6.2 drills deeper by examining the linkages between training and placement rates 

within the targeted sector for the WorkAdvance group. The leftmost boxes within each panel 

depict the expected pathway (indicated with arrows) of WorkAdvance group members. All percentages are for activities within the targeted sector and are out of the full WorkAdvance group 

sample at each site. The focus of this figure is better understanding the various pathways and 

drop-offs between starting services and obtaining a job in the sector. 

 1

Training completion rates have been low in some past programs. See Gueron and Hamilton (2002). 2

The targeted sector placement impacts varied over time at Towards Employment. Among the early cohort 

(roughly the first half of enrollees) the impact on employment in the targeted sector was less than half as large 

as it was among later enrollees. The impact on targeted sector employment was more similar between the cohorts at the other sites. 
The WorkAdvance Study

Table 6.1 

Common Job Titles and Employers for WorkAdvance Respondents Working in the Targeted Sectors 

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Sector Information technology Environmental remediation 

and related fields

Transportation and manufacturing Health care Manufacturing

Common 

job titles

•

•

•

Technical support/help 

desk

Field technician

Computer technician

•

•

•

•

•

Pesticide technician

Exterminator technician

Asbestos handler

Manhole inspector

Truck driver/ 

equipment cleanup

•

•

•

Truck driver

Computerized numerical 

control (CNC) 

machinist/CNC operator

Diesel mechanic

•

•

•

•

•

Patient care representative

Medical billing and 

patient access 

specialist

Patient care assistant

Medical assistant

Phlebotomist

•

•

•

Machine operator

CNC machinist/CNC 

operator

Welder

Common 

employer 

types

•

•

•

•

•

Information technology staffing firms

Banks

Financial firms

Cable companies

Educational institutions

•

•

•

Pest control firms

Environmental 

remediation firms

Firms specializing in 

asbestos removal, mold 

removal, and demolition

•

•

•

•

Trucking companies

Food and beverage 

manufacturers

Welding supply companies

Package delivery services

•

•

•

Hospitals

Nursing homes/

assisted living 

facilities

Home health care 

agencies

• Manufacturers of:

o Metal parts and 

items

o Automotive parts

o Plastics

o Ceramics

SOURCE: MDRC analysis of responses to the Year 2 Survey.

147
148

 

The WorkAdvance Study

Figure 6.1

Impacts on Employment Rates in Targeted Sector 

at Current or Most Recent Job, by Site

0 10 20 30 40 50

 Towards

Employment

***

 Madison

Strategies Group

***

St. Nicks Alliance

***

Per Scholas

***

Percentage point change in employment in targeted sector

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTE: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 

percent.

3

As discussed in Chapter 3, the rates of participation reported in the management information system data 

are somewhat higher, suggesting that some of the survey-based estimates are lower due to lack of recall or other reporting issues. 

The first two panels show Per Scholas and St. Nicks Alliance — the two providers that

focused on training throughout the entire evaluation period. Nearly 70 percent of WorkAdvance 

group members at Per Scholas reported starting training in the information technology (IT) sector.3 At St. Nicks Alliance, a somewhat lower rate of WorkAdvance group members started 

training in the environmental remediation sector (59 percent). At both providers, there was relatively little drop-off between starting training in the targeted sector and obtaining a credential in 

the targeted sector (15 percent of respondents at Per Scholas and 11 percent of respondents at 
149

69%

54%

40%

The WorkAdvance Study

Figure 6.2 

Activity Flow in Targeted Sector, Among WorkAdvance Group Respondents, 

by Site

(continued)

Per Scholas (N = 287)

31%

15%

8%

Obtained job in 

targeted sectora

13%

Obtained job in 

targeted sectora

59%

48%

21%

41%

11%

2%

Obtained job in 

targeted sectora

8%

Obtained job in 

targeted sectora

St. Nicks Alliance (N = 205) 

Did not start training in targeted 

sector 

Did not obtain 

targeted sector 

credential 

Did not start training in targeted sector 

Did not obtain 

targeted sector 

credential

Obtained job in targeted 

sectora

Started training in targeted sector 

Obtained targeted sector credential

Obtained job in targeted sectora

Started training in targeted sector

Obtained targeted sector credential
150

Figure 6.2 (continued)

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: Sample sizes may vary because of missing values. Measures may not sum to 100 percent because of 

missing values.

 a

Jobs in targeted sector are WorkAdvance group respondents’ current or most recent jobs only.

60%

36%

22%

40%

24%

10%

Obtained job in 

targeted sectora

18%

Obtained job in 

targeted sectora

Did not start training in targeted sector

Did not obtain 

targeted sector credential

Obtained job in targeted 

sectora

Obtained targeted sector credential

Towards Employment (N = 286) 

55%

32%

25%

45%

23%

12%

Obtained job in 

targeted sectora

26%

Obtained job in targeted sectora

Madison Strategies Group (N = 297) 

Did not start training in targeted sector

Did not obtain 

targeted sector credential

Obtained job in targeted 

sectora

Obtained targeted sector credential

Started training in targeted sector

Started training in targeted sector
151

St. Nicks Alliance fell off the expected path at this point). Figure 6.2 reveals, however, that credentials were much more likely to translate into targeted sector jobs at Per Scholas. At St. Nicks 

Alliance, 48 percent of WorkAdvance group members started targeted sector training and obtained a credential in the targeted sector, but only 21 percent started training, got a targeted sector credential, and also found a job in the sector (compared with 54 percent and 40 percent, respectively, at Per Scholas). These results suggest that job development and placement strategies 

were much more successful in getting participants into targeted sector jobs at Per Scholas — 

either because jobs were more plentiful in the IT sector or because Per Scholas was better able 

to make use of employer relationships to place individuals once they were credentialed, or a 

combination of the two. One possible reason for the low targeted sector employment rates at St. 

Nicks Alliance is that participants in the environmental remediation training had to wait for 

weeks or even months before receiving their certifications to work in New York. Some individuals were probably unable to wait so long and may have opted to work in another field. As discussed below, another factor may have been the more limited network of employers that St. 

Nick’s Alliance had to work with compared with the other providers. 

The last two panels show Madison Strategies Group and Towards Employment. Despite starting as mixed training-first and placement-first providers, rates of starting training in 

the targeted sector at Madison Strategies Group and Towards Employment were similar to the 

rates at St. Nicks Alliance (but lower than at Per Scholas). However, the rates of those who both 

started training and received a targeted sector credential at those two providers were lower than 

the rates at both Per Scholas and St. Nicks Alliance. Finally, more than three-quarters of those at 

Madison Strategies Group and almost two-thirds of those at Towards Employment who started 

training and obtained a credential found a targeted sector job. (This result is obtained by dividing the percentage who stayed on the expected pathway until obtaining a targeted sector job by 

the percentage who stayed on the expected pathway only until obtaining a targeted sector credential.) Because these providers also featured a placement-first track, a substantial proportion 

of those who never started training in the targeted sector found work in that sector, unlike at Per 

Scholas and St. Nicks Alliance.4 Overall, the pattern at these providers is that they were able to 

place more individuals into the targeted sector than St. Nicks Alliance, and they used more diverse pathways to do so (sometimes with training and sometimes without). See Box 6.1 for a 

discussion of the results in Figure 6.2 from another perspective.

 4

Appendix Figure G.1 shows these same results for the control group. In general, control group members 

at all sites who found targeted sector jobs were less likely to have received a credential than not. 
152

Targeted Sector Job Characteristics

While all the WorkAdvance providers produced impacts on employment in the targeted 

sector, those jobs have to be better than jobs in other sectors to generate impacts on the primary 

economic outcomes. Figure 6.3 compares targeted sector and non-targeted sector jobs on two 

key dimensions of job quality: wages (as measured by the percentage of employed respondents

earning above $15 per hour — a commonly used threshold for “living wage” jobs) and hours 

(as measured by the percentage working full time). These comparisons are only among WorkAdvance group members who had worked since random assignment. It should be noted that 

since the goal of the WorkAdvance programs was to place individuals into jobs in the targeted 

sector, it is likely that participants who were the most job-ready at each site were more apt to get 

these jobs. This may explain some of the differences in outcomes between the targeted sector 

jobs and the others. Given this probable “selection bias” into targeted sector jobs, these results 

should be viewed as descriptive rather than representing the causal effects of the sectoral 

placements themselves. To help account for selection bias, outcomes are regression adjusted 

Box 6.1

Analyzing the Results of Figure 6.2 from Another Perspective

Besides viewing Figure 6.2 as a comparison of how the providers moved participants 

along the pathway, one can look at it from the perspective of how these programs might be 

improved. Figure 6.2 shows that even at Per Scholas, which produced such impressive impacts, almost one-third of WorkAdvance group respondents reported that they never started training in the targeted sector, and almost 40 percent of respondents started the program 

with the expectation of getting an information technology (IT) job but failed to do so.* It is 

important to keep these findings about drop-off in mind when considering how programs 

can be improved in the future. Improvements depend on understanding why some individuals never went to training and why many who completed training were unable to find a 

job in the targeted sector. No training program is ever going to achieve a perfect placement rate, but maximizing the percentage of individuals who successfully complete a sectoral program and get a job in the targeted sector is crucial.

__________________________ 

*

It should be noted that the percentages shown in Figure 6.2 are from responses to the Year 2 Survey. 

The rates of starting training within the targeted sector as reported by the providers’ own program tracking data are much higher (see Table 2.5), but this information was not available from every provider. 

Some of the differences in rates may be due to recall issues in the survey responses. 
153

The WorkAdvance Study

Figure 6.3

Comparison of Characteristics of Targeted Sector Jobs and Nonsector Jobs

Among WorkAdvance Group Survey Respondents Employed Since Random Assignment

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: Statistics are among WorkAdvance group respondents who indicated that they had worked for pay since random assignment. Job characteristics refer 

to the current or most recent job.

Full time is considered working 35 hours or more per week.

"Nonsector job" is any job that is not in the sector(s) targeted by the WorkAdvance provider.

0

20

40

60

80

100

Hourly wage $15.01 or

higher (%)

Working full time (%)

Percentage

Madison Strategies Group

Targeted

sector job

(N=189)

Nonsector

job (N=96)

0

20

40

60

80

100

Hourly wage $15.01 or

higher (%)

Working full time (%)

Percentage

Towards Employment Targeted

sector job

(N=143)

Nonsector

job

(N=109)

0

20

40

60

80

100

Hourly wage $15.01 or

higher (%)

Working full time (%)

Percentage

St. Nicks Alliance

Targeted

sector job

(N=66)

Nonsector

job

(N=117)

0

20

40

60

80

100

Hourly wage $15.01 or

higher (%)

Working full time (%)

Percentage

Per Scholas

Targeted

sector job

(N=177)

Nonsector

job (N=95)
154 

based on a range of observable baseline characteristics, but it is likely that unobservable factors 

that are not adjusted for influence these outcomes.5

At Per Scholas and St. Nicks Alliance, there is a clear difference in the percentage of 

targeted sector jobs (compared with non-targeted sector jobs) that paid above $15 per hour. At 

Madison Strategies Group (which focused on transportation and manufacturing in Tulsa), there 

is a smaller difference, and at Towards Employment (which targeted health care and manufacturing in northeast Ohio), there is almost no difference in the likelihood of making above $15 

per hour. While $15 per hour is a key policy-relevant threshold, it is important to note that the 

overall cost of living is lower in Tulsa and northeast Ohio than in New York City, and in both 

locations, WorkAdvance group members who worked in targeted sector jobs were substantially 

more likely to work at wages of $12.01 to $15.00 an hour than those working outside the sector. Looking simply at the average wage premium (again, comparing targeted sector jobs with

other jobs obtained by WorkAdvance group members), the targeted sector jobs paid $1.76 per 

hour more at Per Scholas, $3.77 per hour more at St. Nicks Alliance, $2.23 per hour more at the 

Madison Strategies Group, and only $0.86 per hour more at Towards Employment. Thus, from 

this perspective, only at Towards Employment were targeted sector jobs associated with a relatively small wage premium. 

At all four sites, individuals who obtained targeted sector jobs were more likely to work 

full time at these jobs compared with individuals who obtained non-targeted sector jobs. This 

difference is particularly large at Madison Strategies Group and Towards Employment.

While the primary outcome of WorkAdvance is earnings, the program sought to improve other job characteristics as well. Table 6.2 shows that, generally, targeted sector jobs tend 

to have better characteristics across a range of measures compared with non-targeted sector 

jobs.6 It is important to remember that these comparisons refer to the characteristics of targeted 

sector versus non-targeted sector jobs and do not necessarily indicate the effects of the program. 

Differences in having employer-offered health insurance on the job were especially large at the 

Madison Strategies Group site (where there was also a statistically significant impact on this 

measure, as discussed in Chapter 5). WorkAdvance group members who worked in the targeted 

 5

Appendix Table G.2 shows the adjusted and unadjusted differences in job characteristics between targeted sector and non-targeted sector jobs. In addition, as a sensitivity check, the analysis was reproduced using 

propensity score matching. For this analysis, a model was built that predicted having top quartile earnings in 

Quarter 10. This model was fit to the whole WorkAdvance group, which was then stratified by propensity 

score into two groups, above or below the median. More refined stratification would be preferred, but sample 

sizes were limited. The results, shown in Appendix Tables G.3 and G.4, are similar to those shown here, but 

this analysis still does not rule out the possibility that unobservable factors (such as participant motivation) 

might drive the associations described in this section. 6

Appendix Table G.1 makes the same comparison with control group survey respondents.
152 

155

The WorkAdvance Study

Table 6.2

Comparison of Characteristics of Targeted Sector Jobs and Nonsector Jobs

Among WorkAdvance Group Survey Respondents Employed Since Random Assignment

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome (%)

Targeted Nonsector

Sector Job Joba

Targeted Nonsector

Sector Job Joba

Targeted Nonsector

Sector Job Joba

Targeted Nonsector

Sector Job Joba

Job characteristics

Offered health plan or medical insurance 

through employer

Working regular shift b

Very or somewhat satisfied with job

Advancement

Had a job before RA, employed at time

of survey interview, and hourly wage

increased by $8.00 or more

Job skills at current or most recent job

Scope of work increased

Offered many opportunities for career

advancement

Obtained new skills while working job

50.8

80.6

85.5

30.7

59.7

71.9

82.9

35.2

73.8

71.9

18.6

42.5

67.9

65.5

45.1

81.4

93.5

32.2

54.0

72.4

84.4

39.7

77.8

66.9

7.5

34.7

64.4

65.2

79.8

84.3

84.4

14.2

56.1

77.8

77.2

43.1

72.6

74.1

13.6

35.5

48.8

56.0

58.7

87.9

69.6

3.7

58.6

68.0

69.3

33.7

69.8

66.8

1.5

31.4

55.0

54.6

Sample size (total = 992) 177 95 66 117 189 96 143 109

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: RA = random assignment.

Italics indicate the statistics are not from the full sample. Statistics are among WorkAdvance group respondents who indicated that they had worked for pay 

since random assignment. Job characteristics refer to the current or most recent job.

Sample sizes may vary because of missing values. a"Nonsector job" is any job that is not in the sector(s) targeted by theWorkAdvance provider.

bA regular shift includes those worked in the daytime, evening, or nighttime. 
156

sector at the St. Nicks Alliance site reported much higher levels of job satisfaction compared 

with WorkAdvance group members who worked outside the targeted sector. At the Madison 

Strategies Group and Towards Employment sites, targeted sector jobs offered more opportunities for advancement. And Per Scholas and St. Nicks Alliance respondents working in the targeted sector saw particularly large growth from baseline wage levels, which are consistent with 

the wage differences shown in Figure 6.3. 

Overall Assessment

• The results suggest that the economic impacts of WorkAdvance depended critically on increasing targeted sector employment relative to the 

control group.

WorkAdvance produced larger wage impacts at Per Scholas by placing many more participants into the better-paying IT sector. At the other three sites, targeted sector jobs had 

somewhat higher wages or hours than non-targeted sector jobs. WorkAdvance group members 

who worked in the targeted sector at St. Nicks Alliance earned much higher wages and worked 

more hours than WorkAdvance group members in non-targeted sector jobs, and wages at targeted sector jobs were also higher than at other jobs at Madison Strategies Group. At Towards 

Employment,7 there was a smaller wage premium associated with working in the targeted sector 

(at least initially), but respondents who worked in the targeted sector worked more hours. But 

all three providers increased targeted sector employment at a rate much lower than Per Scholas, 

so impacts were insufficient to translate into large economic impacts. It is important to note, 

however, that WorkAdvance group members employed at targeted sector jobs reported that 

these jobs had more potential for advancement compared with non-targeted sector jobs, and it 

may take time for the wage and earnings premiums to emerge, which highlights the need for 

longer-term follow-up data. 

Did the Impacts Change over Time? 

• At the two providers that were running programs in sectors they were 

new to, the impacts strengthened for the late cohort, as the providers 

gained more experience and switched to a predominantly training-first 

model. These results suggest that the impacts of WorkAdvance at these 

sites might have been stronger had they been given more time to mature 

before the evaluation started.

 7

Recall that Towards Employment targeted two sectors: health care and manufacturing. Both sectors are 

reflected in targeted sector employment. This analysis does not distinguish the sectors because of sample size 

limitations.
157

A cohort analysis was prespecified as likely to be a key source of variation during the 

analysis planning phase. As mentioned in Chapter 1, Madison Strategies Group was new to the 

Tulsa area, and Towards Employment had limited experience in the manufacturing sector. It 

was thought that it would take time for these two providers to develop deep relationships with 

employers. In addition, these two providers (Towards Employment and Madison Strategies 

Group) were operating programs that featured a mix of job placement and training first in the 

early intake period and only evolved to a mostly training-focused approach during the second 

half of the intake period. For these reasons, it was expected that the impacts of WorkAdvance 

would strengthen over time. Because of this, it is especially important to analyze whether the 

impacts of WorkAdvance vary by the stage in program development when sample members 

were randomly assigned (that is, by cohort). 

Employment and earnings impacts for the early and late cohorts are shown in Table 6.3. 

The early cohort includes sample members who were randomly assigned before Quarter 4, 

2012, and the late cohort includes the remaining sample members. The sample sizes in Table 

6.3 are relatively small. Therefore, this analysis has statistical power to detect only large differences between cohorts. Several patterns in Table 6.3 are noteworthy. 

On the second page, Table 6.3 shows the impacts for the two providers that operated 

mixed training and placement models early in the intake period — Madison Strategies Group 

and Towards Employment. At both sites, there is evidence of stronger effects for the late cohort, 

as expected. At Madison Strategies Group, there are positive and statistically significant impacts 

on both of the survey measures for the late cohort, whereas impacts on these measures were 

close to zero for the early cohort. The estimated difference in the impacts between the cohorts is 

statistically significant for weekly earnings (as indicated by the daggers). In the administrative 

records data, impacts on Year 2 earnings are positive and statistically significant for the late cohort. For the late cohort, Madison Strategies Group increased Year 2 earnings by an impressive 

$4,125 (or 26 percent) above the control group level. In the first quarter of Year 3, the impact is 

no longer statistically significant, though it is still rather large in percentage terms (19 percent) 

and is quite close to the threshold for statistical significance. 

At Towards Employment, there is no clear pattern of cohort differences in the survey 

data, but there is clear evidence of stronger impacts for the late cohort in the administrative records outcomes. Despite control group gains in both outcomes for the late cohort compared with 

the early cohort, Towards Employment produced impacts on both employment and earnings for 

the late cohort. For the late cohort, employment levels in Year 2 were 9 percentage points higher for the WorkAdvance group and earnings levels were nearly $3,000 higher compared with 

the control group. 
158

The WorkAdvance Study

Table 6.3

Impacts for Subgroups Defined Based on Random Assignment Cohort, by Site

Early Cohort Late Cohort

Difference Difference

Outcome WA C (Impact) WA C (Impact) Sig.

Per Scholas

Among Year 2 Survey respondents

Currently employed (%) 71.9 58.2 13.7 ** 77.5 70.2 7.4

Weekly earnings ($) 564 465 99 ** 549 471 78 *

Sample size (total = 552) 152 132 135 133

Among full research sample, UI-covered jobs

Year 1

Ever employed (%) 74.9 67.3

Earnings ($) 8,894 7,410

7.6 *

1,485

79.8 75.2

8,768 10,340

4.6

-1,572 †

Year 2

Ever employed (%) 82.0 74.6

Earnings ($) 17,318 12,351

7.5 *

4,967 ***

82.5 78.9

18,809 17,466

3.6

1,343

First quarter of Year 3

Ever employed (%) 70.9 59.0

Earnings ($) 5,198 3,433

11.9 **

1,765 ***

68.7 71.9

5,469 5,022

-3.2

447

††

†

Sample size (total = 690) 189 185 160 156

St. Nicks Alliance

Among Year 2 Survey respondents

Currently employed (%) 65.5 68.9 -3.4 63.3 64.1 -0.9

Weekly earnings ($) 547 486 62 492 519 -27

Sample size (total = 384) 106 88 99 91

Among full research sample, UI-covered jobs

Year 1

Ever employed (%) 73.2 65.7

Earnings ($) 8,141 8,250

7.6

-110

83.6 76.2

10,720 11,442

7.4

-721

Year 2

Ever employed (%) 72.5 69.5

Earnings ($) 12,482 12,722

3.0

-240

76.6 75.4

16,697 15,945

1.2

752

First quarter of Year 3

Ever employed (%) 56.6 61.9

Earnings ($) 3,143 3,619

-5.3

-476

64.4 65.0

4,910 4,668

-0.6

241

Sample size (total = 479) 127 131 115 106

(continued)
159

Table 6.3 (continued)

Early Cohort Late Cohort

Difference Difference

Outcome WA C (Impact) WA C (Impact) Sig.

Madison Strategies Group

Among Year 2 Survey respondents

Currently employed (%) 71.2 69.1 2.0 82.9 74.5 8.4 *

Weekly earnings ($) 488 526 -38 605 525 80 ** ††

Sample size (total = 560) 138 115 159 148

Among full research sample, UI-covered jobs

Year 1

Ever employed (%) 88.9 86.1

Earnings ($) 12,219 12,514

2.9

-295

89.7 86.9

14,286 13,293

2.8

992

Year 2

Ever employed (%) 78.5 79.4

Earnings ($) 13,825 13,902

-0.9

-77

82.3 85.5

19,549 15,424

-3.2

4,125 *** ††

First quarter of Year 3

Ever employed (%) 59.5 61.0

Earnings ($) 3,302 3,224

-1.6

77

70.2 62.0

4,396 3,688

8.2

708

Sample size (total = 697) 173 164 180 180

Towards Employment

Among Year 2 Survey respondents

Currently employed (%) 58.5 50.7 7.9 71.0 65.6 5.4

Weekly earnings ($) 325 334 -9 435 385 50

Sample size (total = 562) 144 142 142 134

Among full research sample, UI-covered jobs

Year 1

Ever employed (%)

Earnings ($)

80.8

7,537

75.8

7,320

5.0

217

86.6 81.6

11,464 11,549

5.0

-86

Year 2

Ever employed (%) 72.8 70.5 2.3 85.5 76.3 9.2 **

Earnings ($) 9,690 9,559 132 16,598 13,602 2,996 ** †

First quarter of Year 3

Ever employed (%)

Earnings ($)

61.4

2,438

59.8

2,670

1.6

-232

76.6 67.6

4,168 3,593

9.0 *

575 †

Sample size (total = 698) 168 177 181 172

(continued)
160

These findings might reflect somewhat larger participation differences on occupational 

skills training for the late cohort, which occurred as these providers evolved toward a focus on 

training first and away from a significant placement-first orientation (as discussed in Chapter 

3).8 Although the sample sizes are relatively small when sites are divided into cohorts (which 

suggests some caution in interpreting these results), the evidence for stronger impacts for the 

later cohort at both Madison Strategies Group and Towards Employment is consistent with expectations.9

Another result of interest in Table 6.3 is that the impacts in Per Scholas are much weaker in the late cohort. For example, impacts on Year 1 earnings for the early cohort are positive 

and quite substantial. These patterns are very different from the negative (though not statistically significant) Year 1 impacts for the late cohort. Impacts on earnings in Year 2 are very large 

($4,967) for the early cohort, but only a third as large (and not statistically significant) for the 

late cohort. It is important to point out that the variation in impacts by cohort at Per Scholas is 

much more obvious in the administrative records than in the survey data. Though the overall 

pattern is the same, much less pronounced differences are evident in the smaller survey sample.

To understand what might be happening to the late cohort in Per Scholas, it is helpful to 

note that while WorkAdvance group earnings and employment levels do not vary substantially 

between cohorts, the control group employment and earnings levels for the late cohort are much 

higher than for the early cohort. This provides some preliminary evidence that the impacts may 

 8

It is not possible to reliably separate out the impact for individuals who were enrolled in the placementfirst track, because the decision about whether an individual went into that track was made after random assignment. 9

When the results at Towards Employment and Madison Strategies Group are pooled, the impacts on both 

survey measures, earnings in Year 2, and earnings and employment in the first quarter of Year 3 are statistically significant in the late cohort (shown in Appendix Table G.14).

Table 6.3 (continued)

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment 

insurance administrative records from New York State Department of Labor, Ohio Department of Jobs and 

Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group.

The early cohort includes all sample members randomly assigned through Quarter 3, 2012. The late cohort 

includes all sample members randomly assigned in or after Quarter 4, 2012.

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are 

indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Sample sizes may vary because of missing values.
161

have changed because of the improving local economy or owing to changes in the composition 

of the sample at that site over time in terms of unobservable characteristics.10 The economy 

probably explains at least part of the difference in control group levels on employment outcomes. The New York City economy was rebounding from the Great Recession throughout the 

follow-up period. (This can also be seen at St. Nicks Alliance, another New York City site, 

where control group levels increased by similar magnitudes.) But this difference in control 

group levels doesn’t explain why the WorkAdvance group levels did not increase correspondingly. An analysis of baseline characteristics across these cohorts shows that the late cohort is 

more likely than the early cohort to be female (18 percent versus 9 percent), is less likely to 

have been currently employed when they entered the study (8 percent versus 17 percent), is less 

likely to have been previously incarcerated (4 percent versus 8 percent), and is less likely to 

have been on SNAP (food stamps) (11 percent versus 22 percent).11 But changes in the composition of the sample do not appear to drive the patterns in outcomes, as shown in Appendix Table G.5.12 An analysis of participation data (discussed in Chapter 3) found that, if anything, 

training completion and credentialing differentials were larger in the late cohort. 

The weight of the available data suggests that the economy may have been the main 

factor, but it is difficult to know for sure and other factors may have played a role as well. The 

results may suggest that Per Scholas works better in a weaker economy or that for some reason 

the program was less able to make job placements in the late cohort. Prior research has found

larger impacts for training programs in weak economies compared with stronger ones.13 In addition, as a mature provider, Per Scholas may not have strengthened its program as much during 

the follow-up period, compared with the other providers. Thus, the impacts at Per Scholas might 

be more sensitive to this countercyclical economic effect. 

The impacts at St. Nicks Alliance are largely the same for both cohorts, despite sustained and intensive technical assistance at that site. Box 6.2 suggests that St. Nicks Alliance 

may have chosen the wrong sector. The environmental remediation sector has many short-term 

jobs with limited advancement opportunities, and there are long built-in delays between ending 

training and being able to work. In addition, the staff at St. Nicks Alliance never truly committed to serving employers as much as the participants. For example, meeting face to face with 

employers is known as a “best practice” for job placement, and St. Nicks Alliance job developers seldom were able to get out into the field to do so. 

 10Differences in the observable characteristics of the cohorts do not fully explain the cohort differences. In 

addition, there are no differences in the baseline characteristics of the WorkAdvance and control groups within 

the cohort, so the “bad draw” explanation has been ruled out. 11These results are shown in Appendix Tables A.1 and A.2. 

12Appendix Table G.5 shows the results of a regression analysis that interacted treatment status with sites, 

cohorts, and several other characteristics. The results show that the effects for the late cohort at Per Scholas are 

still insignificant after adjusting for several demographic characteristics. 13Card, Kluve, and Weber (2015).
162

Box 6.2

Why Have Impacts Failed to Emerge So Far at St. Nicks Alliance?

St. Nicks Alliance produced large increases in participation in WorkAdvance services. 

They have operated an environmental remediation training program for over a decade. Unlike the Towards Employment and Madison Strategies Group sites, they never operated a 

placement-first model. So why haven’t statistically significant impacts emerged at this site 

so far?

Several factors may explain the weaker impacts at St. Nicks Alliance. First, demand conditions in the environmental remediation field were quite weak early in the follow-up period, which required St. Nicks Alliance to shift eventually to related industries, such as pest 

control. But demand conditions did pick up in the environmental remediation field later in 

the follow-up period, and impacts are still not evident for the late cohort. The first WorkAdvance implementation report points to several operational factors that could explain the 

lack of impacts at St. Nicks Alliance.*

 

One factor could be organizational structure and background. Unlike most other WorkAdvance providers, the St. Nicks Alliance program operated in the context of a large, multifaceted organization in which workforce development was a relatively small component. 

Senior leaders were not as tuned in to the program, and St. Nicks Alliance had a new director, who started just as the evaluation was launching. There were also staffing challenges: At several points in the evaluation, career navigator positions were understaffed and 

attention was split between WorkAdvance and other programs. Finally, the first report 

notes that St. Nicks Alliance, as a human services organization, was more participant focused (and less employer focused) than the other sites.

St. Nicks Alliance also experienced issues related to their training program, which could 

have reduced the flow of participants through the full program, as shown in Figure 6.2. 

First, because individuals were randomly assigned into cohorts, they often had to wait for 

several weeks before training began. Then the participants who completed environmental 

remediation training to work with asbestos had to wait to take their certification test and 

then wait another 45 days to obtain a “hard card,” which provides city and state authorization to work in the field. That is a long time to wait for a low-income population, many of 

whom may have opted to work elsewhere. 

The first implementation report also notes that job developers at St. Nicks Alliance struggled to find permanent jobs with advancement potential in environmental remediation, 

which has a heavy concentration of short-term project-based work. St. Nicks Alliance received extensive, ongoing technical assistance to address low placement numbers and 

strengthen job developers’ skills. Meetings with consultants took place weekly, and then 

every other week, for more than a year. For example, job developers often stayed in the 

office rather than meeting with employers in the field. MDRC and consultants worked to 

encourage more proactive job development with stronger direct employer relationships, 

(continued) 
163

Were the Impacts Affected by the Characteristics of Participants?

• While some subgroups experienced larger impacts than others, the variation in impacts of WorkAdvance from site to site does not appear to be 

explained by variation in participants’ characteristics. 

For the theory of change in WorkAdvance to lead to impacts, the services provided by 

the programs had to target the right population. One possibility that could explain the variation 

in site impacts is that the providers served somewhat different target groups, and it may be that 

some subgroups are more responsive to the program than others. First, a series of subgroup 

analyses looked at the question of whether the impacts of WorkAdvance varied across key subgroups. Then an analysis is discussed that suggests that differences in the compositions of the 

samples from site to site were not a key driver of the variation in impacts. 

Pooled Sample Impacts

Due to concerns about statistical power at the site level, a decision was made during the 

analysis planning stage to run subgroups using the pooled WorkAdvance sample. Findings for 

the pooled sample are shown in the left panel of Table 6.4. Combining all sites, WorkAdvance 

increased current employment and weekly earnings among survey respondents. WorkAdvance 

increased Year 1 employment, based on unemployment insurance records, by 5 percentage 

points above the control group average of 77 percent. This effect was somewhat surprising given that Year 1 was when many WorkAdvance group members participated in occupational 

Box 6.2 (continued)

such as skill matching between individual participants and job opportunities. St. Nicks Alliance also received assistance in gathering and analyzing systematic labor market information. 

Ultimately there could be many reasons why the impacts were weaker at St. Nicks Alliance. It is clear that the targeted sector experienced weak demand, and the nature of jobs 

and certifications in that sector may not have enabled longer-term advancement to emerge 

thus far. It is also possible (though the research does not indicate that this is likely, based 

on measurable characteristics, at least) that the weaker impacts so far relate to the characteristics of the individuals served by the program.

__________________________ 

*

Tessler (2013). 
164

The WorkAdvance Study

Table 6.4

Impacts on Subgroups Defined by Baseline Labor Market Attachment, All Sites Combined

Full Samplea Fully Attached

Difference

Semiattached Long-Term Unemployed

Difference Difference Difference

Outcome (%) WA C (Impact) WA C (Impact) WA C (Impact) WA C (Impact) Sig.

Among respondents to the Year 2 Survey

Currently employed 70.6 65.3 5.3 *** 82.4 76.9 5.5 68.6 69.2 -0.7 64.6 53.0 11.6 *** ††

Weekly earnings ($) 502 459 42 *** 515 492 23 530 466 63 *** 467 429 38

Sample size (total = 2,058) 1,075 983 254 287 373 344 445 348

Among full research sample

Year 1

Ever employed in a UI-covered job 82.4 77.4 5.0 *** 93.9 91.0 2.9 82.9 79.5 3.3 74.8 65.5 9.3 ***

UI earnings ($) 10,299 10,304 -5 14,492 14,611 -119 10,429 10,233 195 7,533 7,303 230

Year 2

Ever employed in a UI-covered job 

UI earnings ($)

79.5 76.7

15,717 13,772

2.8 *

1,946 ***

85.8 89.0

18,540 17,540

-3.2

1,000

81.2 79.5

16,188 13,815

1.7

2,373 **

73.5 65.4

13,440 11,093

8.1 *** ††

2,347 ***

First quarter of Year 3

Ever employed in a UI-covered job 

UI earnings ($)

66.7 63.4

4,144 3,675

3.3 *

469 ***

73.7 73.6

4,457 4,441

0.1

16

67.0 64.0

4,386 3,671

2.9

715 **

61.8 55.5

3,683 3,171

6.3 **

512 **

Sample size (total = 2,564) 1,293 1,271 308 356 456 449 529 463

(continued)
165

skills training, but recall that two of the providers operated a mixed placement-first and trainingfirst model during the early phase of the demonstration.14 During the second year of follow-up, 

the impact on employment was somewhat smaller. However, the program produced a statistically significant $1,946 increase in earnings in Year 2. The same patterns are evident in the first 

quarter of Year 3. 

It is important to note that the pooled impact estimates mask the substantial variation in 

the site impacts. Without Per Scholas, the pooled impacts would be weaker. Without St. Nicks 

Alliance, the pooled impacts would be stronger. Assuming that the WorkAdvance sites represent a generalizable set of providers, the pooled impacts provide a good estimate of the impacts 

from an “average” provider.15 

 14Placement programs typically have impacts more quickly than training programs. See Hamilton et al. 

(2001) for a discussion. 15A sensitivity check examined the pooled impacts with and without the Per Scholas site. The results suggest that Per Scholas drives most, though not quite all, of the positive impacts of WorkAdvance in the pooled 

sample. These results, shown in Appendix Table G.6, indicate that the other three providers produced increases 

in earnings and employment that were much smaller and less consistent across time and measures than the 

impacts at Per Scholas. Excluding Per Scholas, the Year 1 impact on employment is nearly 5 percentage points 

(again, recall that the two placement-first providers are included here). However, the estimated impact on employment is statistically significant in neither Year 2 nor the first quarter of Year 3. The Year 1 impact on earnings is not statistically significant, but the Year 2 impact on earnings is $1,196 and statistically significant at the 

10 percent level. The estimated effect on earnings in the first quarter of Year 3 is not statistically significant 

when Per Scholas is excluded.

Table 6.4 (continued)

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from 

unemployment insurance (UI) administrative records from New York State Department of Labor, 

Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group.

The fully attached subgroup consists of sample members who were working at random 

assignment or who were unemployed for less than one month before random assignment. The 

semiattached subgroup consists of sample members who were unemployed for one to six months 

before random assignment. The long-term unemployed subgroup consists of sample members who 

have never been employed or who were unemployed for seven or more months before random 

assignment.

A two-tailed t-test was applied to the differences between outcomes for the program and control 

groups. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 

10 percent.

Differences across subgroups were tested for statistical significance. Statistical significance 

levels (Sig.) are indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Sample sizes may vary because of missing values.

a

Full sample includes all sample members included in one of the subgroups. Three sample 

members are missing a labor market attachment status and are therefore not included in the full 

sample.
166

Confirmatory Subgroup Analysis

• The impacts of WorkAdvance were concentrated among the mediumterm and long-term unemployed. Those who entered the program already working or who lost their job only within the past month did not 

experience statistically significant impacts on the key economic outcomes. At least part of this pattern is due to larger impacts at sites that 

were more likely to serve individuals who had been unemployed longer. 

In WorkAdvance, one confirmatory subgroup was prespecified and is based on sample 

members’ pre-random assignment level of attachment to the labor market.16 Three groups were 

identified: (1) those who were “fully attached” to the labor market (defined as those currently 

employed or unemployed for less than one month before the study), (2) those who were “semiattached” (defined as those unemployed for one to six months before the study), and (3) the 

“long-term unemployed” (defined as those never employed or unemployed for seven or more 

months before the study). The groups were defined this way to be consistent with the categories 

commonly used by labor economists. Generally speaking, the long-term unemployed group was 

more distinctive in terms of participant characteristics than the semiattached and fully attached 

groups.17

The expectation was that WorkAdvance would be most effective for the semiattached 

group. This expectation was rooted in the results of previous studies, which showed that employment programs can be most effective for those who are at a “tipping point” in their employment trajectories.18 This hypothesis asserts that the fully attached might not benefit from 

WorkAdvance because their outcome levels are relatively high (leaving less room for im-

 16In recent years, the program evaluation field has become more sensitive to the need to limit the number 

of subgroups analyzed in order to reduce the number of “false positives” that result when one makes too many 

statistical comparisons. (See Box 5.1.) In order to manage this risk, methodologists have recommended prespecification of a limited number of “confirmatory” subgroups that theory and experience suggest might moderate the impacts of a program (see Bloom and Michalopoulos, 2010). In WorkAdvance, a decision was made 

during the analysis planning phase to focus on subgroups based on prior attachment to the labor market because previous studies have found this to be an important subgroup (for example, Friedlander, 1988; Hamilton 

and Scrivener, 2012b). 17The long-term unemployed were older and more likely to be African-American. By definition they had 

lower earnings in the past year, but they also had clearly lower earnings in the three years before random assignment. Per Scholas and St. Nicks Alliance are overrepresented among the long-term unemployed group. 

The semiattached are more likely to be Hispanic compared with the attached and much less likely than the attached to have entered the program with credentials in the targeted sector. Sample members from Towards 

Employment and Madison Strategies Group are overrepresented among the fully attached group because both 

of these providers were more likely to serve incumbent workers. In addition, the fully attached group had higher educational levels than the semiattached or long-term unemployed. Interestingly, the long-term unemployed 

sample was more likely to be in the late cohorts, perhaps owing to loosening of entry criteria at several sites 

late in the intake period. 18Hamilton and Scrivener (2012b); Friedlander (1988).
167

provement) and they may need less assistance from the program than the semiattached and 

long-term unemployed groups. At the other extreme, the long-term unemployed would be expected to have more barriers to employment (particularly based on the selectivity of employers 

in the wake of the Great Recession and the resulting scarring from a long duration of unemployment),19 which a training-focused program like WorkAdvance might not be equipped to 

resolve.

As shown in Table 6.4, the impacts of Work Advance tended to be stronger among the 

semiattached and long-term unemployed groups. For the survey outcomes, WorkAdvance produced a large and statistically significant impact on current employment for the long-term unemployed group and increased weekly earnings among the semiattached group. The estimated 

variation in impacts across subgroups (indicated by the daggers) is statistically significant for 

the current employment measure, but not for the weekly earnings measure. 

The same pattern is evident in the administrative records: WorkAdvance had no statistically significant impacts on any of the measures shown in Table 6.4 for the fully attached group. 

Among the semiattached group, WorkAdvance increased earnings by a statistically significant 

margin in Year 2 and the first quarter of Year 3 but did not significantly increase employment in 

either time period. Among the long-term unemployed group, WorkAdvance produced large effects on employment and earnings in Years 1 and 2 and the first quarter of Year 3. The estimated variation in impacts across subgroups (indicated by the daggers) is statistically significant for 

Year 2 employment.

Since the sites were pooled together for this analysis, a natural question is the extent to 

which these patterns are driven by the site-specific impacts rather than by the long-term unemployed group. To check this possibility, a regression analysis tested whether controlling for site 

weakens or eliminates the impacts among the long-term unemployed. The results of this test, 

shown in Appendix Table G.5, show that the stronger effects among the long-term unemployed 

are eliminated when controlling for site. This suggests that the impacts for this group are at least 

partly due to program-related factors at specific sites. Therefore, the results imply that WorkAdvance can produce impacts among the long-term unemployed, but WorkAdvance does not 

necessarily produce stronger impacts for that subgroup (nor do the results imply that WorkAdvance should not be targeted to the fully attached subgroup). 

Impacts for Exploratory Subgroups

This section provides a brief summary of the findings from several other subgroup 

analyses. Because these subgroups were not prespecified based on strong theory, these analyses 

should be viewed as exploratory. While these exploratory findings have less standing in the 

 19Kroft, Lange, and Notowidigdo (2013). 
168

analysis, they could generate hypotheses for future studies. The exploratory subgroup analyses 

examined the extent to which the effects of WorkAdvance vary by other participant characteristics, such as age, prior conviction or incarceration status, prior earnings, and education.

Before reviewing these subgroup results, it is important to highlight that some of these 

findings, like the labor market attachment subgroup findings, could be explained by site-level 

factors. Notably, Per Scholas, which produced larger impacts than the other providers, has a 

younger and more Hispanic sample. In order to help disentangle site-level versus person-level 

characteristics, a compositional analysis was also conducted. This analysis used a variety of interaction terms and weighting strategies to examine the extent to which the variation in impacts 

of WorkAdvance was attributable to the characteristics of sample members at each site. The 

overall assessment from this analysis, shown and discussed in Appendix G, is that the sample 

composition was not an important factor in explaining the variation in impacts across the sites 

(Appendix Tables G.5 and G.6).20 Put differently, other aspects of the sites, such as the quality 

of implementation or the economic context in which the programs operated, are more likely to 

be drivers of the variation in impacts.21

The overall impression from the exploratory subgroup analysis is that the impacts of 

WorkAdvance did not vary greatly across many subsamples. While there is some variation 

(noted below), in most cases it is not statistically significant. 

• Age: In recent years, there has been increasing concern about the labor market performance of two age groups in particular. First, employment rates 

among young adults (those aged 18 to 24) have been dropping, leading to 

concerns about “disconnected youth.”22 Another group of concern is older 

workers, many of whom were dislocated by the Great Recession and have 

had trouble regaining a foothold in the labor market.23 WorkAdvance produced statistically significant impacts across all age groups. If anything, impacts were generally larger among the 18-to-24 age group, but this is probably an artifact of Per Scholas enrolling more individuals in that age range. 

The variation in impacts across age subgroups is statistically significant for 

 20There is some limited evidence that impacts were stronger for a few measures among sample members 

at the other three sites who have characteristics similar to those of the Per Scholas sample, but results are inconsistent across measures. 21This conclusion is evident in two ways. First, in Appendix Table G.5, it is clear that site interactions (and 

site-by-cohort interactions) are robust to the inclusion of demographic interactions. In Appendix Table G.6, 

compositional alignment of samples does not seem to coherently explain why Per Scholas’s impacts were larger or why the other providers’ impacts were smaller. See Appendix G for more discussion. 22See Holzer (2010) for a discussion. 23See Van Horn, Krepcio, and Heidkamp (2011) for a discussion.
169

only one measure (employment in the first quarter of Year 3).

24 (See Appendix Table G.7.)

• Education: While the WorkAdvance programs did a fair amount of screening, individuals entered WorkAdvance with varying levels of education. This 

might raise concerns that some entered with insufficient preparation and others with too much preparation for the level of training offered. An analysis of 

the impacts by education levels, however, suggests that there is no pattern evident across these subgroups: WorkAdvance produced positive impacts for 

those with lower and higher educational attainment (Appendix Table G.8).

• Earnings levels: Individuals entered WorkAdvance with different levels of 

prior earnings. Some individuals just barely qualified for the program due to 

higher earnings, whereas others were well below the eligibility cutoff. In 

terms of prior earnings and predicted earnings levels, the estimated impacts 

varied across the subgroups, but not in a consistent way from measure to 

measure (Appendix Tables G.9 and G.10). 

• Prior conviction or incarceration status: One target group of major concern 

is individuals who have a history of criminal convictions or incarceration. 

These individuals often have difficulty obtaining employment, as employers 

in some sectors (such as health care and pest control) are quite reluctant to 

hire individuals with a criminal background. The results bear this out; the 

impacts are somewhat larger among individuals who were not previously 

convicted or incarcerated. Late in the follow-up period, there is statistically 

significant variation in impacts across subgroups based on conviction or incarceration status at study entry (Appendix Table G.11).

• Race: One of the goals of sector programs is to help individuals who have 

historically had trouble entering a sector, either due to discrimination or fewer social ties, and to improve diversity in sectors where minorities are underrepresented. For example, early sector projects focused on helping women 

into construction jobs. In WorkAdvance, generally speaking, impacts are 

larger among Hispanic and black sample members than among white sample 

members. The variation in impacts is statistically significant for earnings in 

Year 1 and employment in Year 2 (Appendix Table G.12).

 24There is some evidence, presented in Appendix G, that the effects of age are more important when controlling for other characteristics, such as site and race. The estimate discussed here shows the overall “gross” 

effect of age rather than the “net effect,” which is obtained after one controls for other characteristics.
170

Conclusion

This chapter has examined three questions to help shed light on the mechanisms behind the impacts presented in Chapter 5: 

1. Were the impacts driven by either the proportion of participants placed 

in targeted sector jobs or the characteristics of those jobs? 

The weight of the evidence suggests the extent and quality of targeted sector placements were important factors in explaining the pattern of impacts by site. Per Scholas was by 

far the most successful provider at placing participants in targeted sector jobs. In addition, nonexperimental evidence suggests that those targeted sector jobs had higher wages than nontargeted sector jobs. By contrast, targeted sector jobs at St. Nicks Alliance and Madison Strategies Group also paid more, but the impacts on getting individuals into those jobs were much 

smaller than those at Per Scholas. At Towards Employment, targeted sector jobs had better 

hours and other non-wage characteristics, but wages were not substantially higher than nontargeted sector jobs, and the impacts on getting a targeted sector job were much smaller than at 

Per Scholas. These findings highlight the importance of two factors in strong sectoral programs: 

(1) good job development in the targeted sector and (2) ensuring that targeted sector jobs come 

with a wage premium.

2. Did the impacts change over time as providers gained more experience 

and moved to a predominantly training-first approach?

Chapter 5 shows that the overall impacts at Madison Strategies Group and Towards 

Employment were mixed, with some positive impacts on earnings and employment emerging 

late in the follow-up period and some impacts on job characteristics. This chapter gives clear 

indications that the impacts at those sites were stronger for the late cohort, which may reflect the 

movement toward a training emphasis for that cohort and may also indicate that the impacts of 

WorkAdvance improved with the maturity of the programs at those providers (both of which

were new to sectoral training). It is not possible to reliably disentangle which factor was most 

important. These factors are least relevant to Per Scholas because it was the most mature provider coming into the study and had a training-first emphasis in both of its cohorts. In fact, the 

results indicate some weakening in effects for the late cohort at Per Scholas, for reasons that are 

unclear but which may relate to the improving economy. At St. Nicks Alliance, no improvement in impacts was evident for the late cohort. 

3. Were the impacts affected by the characteristics of participants?

While there are some subgroups that experienced larger impacts than others, the variation in who was served does not seem to explain the pattern of impacts across the sites. One im-
171

portant lesson for the field, however, is that WorkAdvance has shown evidence of improving 

outcomes for those who came into the program already medium- to long-term unemployed. 

This chapter examined some of the possible key drivers behind the variation in impacts 

across the sites. The final chapter will put these results in a wider context by situating WorkAdvance within the literature on demand-driven training programs.

173

Chapter 7

Conclusion

The WorkAdvance study provides the first rigorous evidence on whether the promising results 

from the Sectoral Employment Impact Study (SEIS) can be replicated in a different economy, 

scaled up to a wider set of providers, and strengthened through the inclusion of advancement 

services. Since the SEIS results were made public,1 the sectoral strategy has been implemented 

widely throughout the country and is a key component of the federal Workforce Innovation and 

Opportunity Act passed in 2014. The WorkAdvance evaluation, begun in 2011 and still in progress, has already contributed a great deal of knowledge about how to implement a sectoral 

program with a provider that has not run one before and about how to implement advancement 

services within a sector program.2

A year and a half to two years after people applied to WorkAdvance, the impacts of the 

program — the main topic of this report — are promising. Not surprisingly, the most mature 

program, at Per Scholas, a provider with well over a decade of experience operating its sectoral 

program (and which was the only WorkAdvance site also included in the SEIS), had large and 

impressive impacts. Impacts from two of the other three newly established sectoral programs, at 

Towards Employment and the Madison Strategies Group, were less strong, but there were some 

positive impacts on labor market outcomes, particularly in Year 2 and for participants enrolling

during the second half of the demonstration, who may have experienced a stronger program. 

The remaining WorkAdvance program, at St. Nicks Alliance, shows little evidence of positive 

impacts so far. Participants in that program had the least success in getting jobs in the targeted 

sector. This variation in impacts provides the field with a lot of insights into the conditions 

needed to operate an effective sectoral program.

In general, the findings suggest that sectoral training programs can produce large impacts on employment and earnings, but (like any policy intervention) they are not going to be 

effective under all circumstances. Certain conditions need to be met for the strategy to increase 

the employment and earnings of low-income populations, and this chapter aims to shed light on 

what those conditions are. The chapter first summarizes the findings to date from the WorkAdvance evaluation and then places these findings in the context of findings from the SEIS. The 

chapter also discusses lessons from the WorkAdvance study for policymakers and practitioners. 

It concludes with a discussion of what the findings suggest for future tests in this area and outlines some critical unanswered questions for the field. 

 1

Maguire et al. (2010).

2

See Tessler et al. (2014).
174

Summary of Findings

The implementation study shows that mastering new components — whether employerresponsive sectoral training, a dual-customer focus, or advancement services — took a substantial amount of time for all four providers, and for some more than others. The program model 

was designed specifically for the WorkAdvance demonstration, and each provider lacked experience with at least one of the WorkAdvance program components, especially postemployment 

services that focused on career advancement rather than only job retention. The providers’ varying degrees of experience in each of these areas influenced their implementation of the program 

in different ways. Per Scholas had a sizable head start with its delivery of sector training in information technology for more than 15 years before WorkAdvance.

Despite start-up issues, documented in the implementation report,

3 all four providers 

eventually delivered services in all the WorkAdvance model components, with postemployment 

services the last to be developed and implemented. The engagement of participants in key program components — career readiness training and occupational skills training — was high in all 

four providers. In most cases, completion of occupational skills training led to the earning of 

either a nationally or a locally recognized credential, or both. The high participation and completion rates may have been, in part, a consequence of the rigorous screening the programs conducted before enrolling individuals in WorkAdvance (which was also part of the value of the 

programs for employers).

Notably, WorkAdvance increased participation in occupational skills training at every 

site over what the participants would have gotten on their own, with especially large impacts on 

participation in occupational skills training in the targeted sector — between 38 and 49 percentage points. Many employment and training programs have produced only small or no impacts 

on participation in services. It is not always easy for participants to show up for services, and it 

is certainly difficult for them to finish training programs, yet the combination of WorkAdvance 

services enabled the participants to persevere longer than they would have elsewhere. 

WorkAdvance services cost between $5,200 and $6,700 per participant across the four 

providers. For three of the providers, the range was quite narrow, $5,200 to $5,800, suggesting 

perhaps that many organizations could operate programs similar to WorkAdvance for around 

$5,500 per participant. Roughly half the providers’ operating expenditures were devoted to 

providing preemployment activities and occupational skills training. From a societal perspective, what matters is how much more WorkAdvance costs than what society would have spent 

on the participants otherwise. This net cost, which varied from site to site, is what should be 

weighed against the cumulative net change in outcomes, or impacts, when considering the cost-

 3

Tessler et al. (2014).
175

effectiveness of the program. A full benefit-cost analysis is planned after five years of follow-up 

data are available for analysis.

As discussed above, the impact evidence suggests that an advancement-oriented sectoral training program, such as WorkAdvance, is a promising approach to increasing earnings. 

The impacts here and in the SEIS leave little doubt that the Per Scholas program has large, even 

transformative, effects on participants. Because these impressive results have been measured in 

two rigorous evaluations, policymakers and program operators should carefully review Per 

Scholas’s approach as a model of effective workforce development.4

The performance of the other organizations shows that these types of strong results are 

not guaranteed, but the experience of Madison Strategies Group and Towards Employment 

shows how program maturation and a shift toward training can help produce economic impacts. 

Trying to understand the impacts across the sites gave rise to several insights. First, the weight 

of evidence suggests that a critical factor in having an impact was an organization’s ability to 

get a significant proportion of its participants employed in the targeted sector. Second, those 

jobs needed to be good jobs. Providing sector-specific training seemed to be associated with 

being able to secure these better jobs. Given the slow process of developing credibility with 

employers by placing well-prepared individuals in jobs, it is not surprising that these types of 

advancement-oriented sectoral programs seem to take time to reach their potential.

Per Scholas was by far the most successful site in placing participants in targeted sector 

jobs. In addition, nonexperimental evidence suggests that IT jobs had higher wages than nontargeted sector jobs obtained by WorkAdvance group members. By contrast, perhaps due to 

slack demand conditions, St. Nicks Alliance was the least effective site in placing participants in 

the targeted sector, though the jobs in that sector did seem to have higher wages.5 The pattern at 

the other two sites (like the impacts) was more mixed. Towards Employment and the Madison 

Strategies Group had smaller impacts on placing individuals into sector jobs compared with Per 

Scholas (though these effects were somewhat larger than at St. Nicks Alliance). For Madison 

Strategies Group enrollees, targeted sector jobs paid better and had better benefits and hours. 

For Towards Employment enrollees, targeted sector jobs did not pay much better, but they had 

better hours and advancement characteristics. The overriding factor for site variation in impacts 

 4

The Per Scholas impact on earnings was somewhat smaller for participants enrolling in the second half of 

the study, when the economy was stronger, consistent with a commonly seen trend (Card, Kluve, and Weber, 

2015). Control group members enrolling in the later months of the study had much stronger outcomes than 

earlier control group enrollees. It cannot be ruled out that the weaker impacts in the late cohort are due to other 

factors, such as sampling variation. 5

As mentioned in Chapter 5, targeted sector employment rates were based on the current or most recent 

job. It is possible that the short-term nature of work in St. Nicks Alliance’s targeted sector, environmental remediation, could have led to an underestimate of the percentage of sample members who worked in that sector.
176

appeared to be differences across the sites in the extent to which they increased placements in 

the targeted sector. 

In the middle of demonstration, two of the programs, Madison Strategies Group and 

Towards Employment, switched from a mixed placement and training model to a training-first 

model. An impact analysis was conducted by cohort — comparing those who enrolled in the 

first half of the study with those who enrolled later — in part to examine the hypothesis that 

sending more individuals initially through training would improve impacts. The impacts, in fact, 

were stronger for the late cohort, but it is unclear whether this was due to the greater trainingfirst orientation for the late cohort or to program maturity. The cohort results do not appear to be 

due to changes in sample composition. No improvement through maturity was evident at St. 

Nicks Alliance, which had always implemented a training-first model.

Finally, while there are some groups of study enrollees who experienced larger impacts 

than others, variation in who was enrolled does not seem to explain the pattern of impacts 

across the sites. Particularly encouraging were results that suggested that WorkAdvance had 

success placing the long-term unemployed back into the labor market — a group that is of major policy concern in the wake of the Great Recession. WorkAdvance produced both employment and earnings impacts for this group. The model seems particularly effective for the medium- or long-term unemployed, but it is unclear whether it is as effective among incumbent 

workers. This finding echoes other studies of training programs.

6

 Because it is hard to disentangle site effects from demographic characteristics, the program’s effectiveness for these different 

groups should be explored more in the future. 

Where do these results leave the workforce development field? Before coming to some 

preliminary conclusions, it is useful to consider the WorkAdvance findings in light of the findings for the SEIS, which motivated some core aspects of the WorkAdvance model.

A Comparison of WorkAdvance and SEIS Findings

The implementation of the WorkAdvance model can usefully be examined in light of similarities to or differences from the implementation features of the sector programs in the SEIS. In the 

SEIS, all three providers produced large impacts on earnings. Overall, the pooled impacts of 

WorkAdvance, while positive, are approximately half as large as the impacts of the SEIS. The 

researchers who conducted the SEIS (Public/Private Ventures, or P/PV) suggested that key elements of the programs included the providers’ experience with sectoral programs, their strong 

relationships with local employers (which they called the “brokering” function), provision of 

job readiness training in addition to occupational skills training, a stringent screening and intake 

 6

Card, Kluve, and Weber (2015).
177

process, and provision of individualized services.7 Many of these features were exhibited by the 

WorkAdvance providers, though not all of them and not equally across the providers.

Provider Experience

The most important factor in explaining the variation in impacts, perhaps, is that the organizational experience of the WorkAdvance providers varied, both among themselves and 

compared with those of the SEIS providers. To be selected for the SEIS, providers had to be 

operating the same program that was eventually evaluated for at least three years. A key difference between the two sets of studies, therefore, was providers’ previous experience operating 

sector programs (with the exception of Per Scholas, which participated in both studies) and the 

degree to which they had deep connections in the sector. These connections take time to develop, even for an experienced organization if it is in a new location, such as Madison Strategies 

Group was in Tulsa.8 For its part, Towards Employment was new to the manufacturing sector 

and had the compounded challenge of focusing on two very different sectors while coordinating 

program operations in two separate locations.

Thus, from the perspective of provider experience, the WorkAdvance results are not 

that complicated. The site that was already up and running a sector program (as the SEIS programs were) replicated its results well. The three other sites — that were either new to a sector 

approach or new to their locations — had a longer way to go. This “head start” hypothesis asserts that since Per Scholas was running the core elements of the program for a long time and 

already had experience with a random assignment study, it had an insurmountable advantage. 

The results at the other sites suggest that sectoral programs are hard to get off the ground. While 

these results probably explain the impacts to some extent, they don’t explain why a provider 

such as St. Nicks Alliance was successful in producing large participation differences and could 

train people for years in the same field but nevertheless has been unable to produce statistically 

significant impacts on employment so far. (See Box 6.2 for a discussion of why the impacts 

may have been weaker thus far at St. Nicks Alliance.) The value of experience, however, is not 

solely related to running the training program. The benefit comes from working with employers 

in a different way for several years and gaining their trust and respect. The sector “experience” 

hypothesis is given some support by the clear evidence of stronger effects for the late cohort at 

the Towards Employment and Madison Strategies Group sites. 

In order to provide more information to practitioners in this field who are planning on 

starting sector programs, the WorkAdvance research team spoke with leaders at the Per Scholas 

 7

Although they aimed to place workers in “good” jobs — jobs that are higher-paying and more stable — 

the SEIS programs did not have an explicit advancement component (Maguire et al., 2010, p. 48). 8

Leaders at Per Scholas report that it takes two to three years before a new Per Scholas affiliate can take 

root and reach full operation in a new location. (See Box 7.1 for more discussion.) 
178

and Madison Strategies Group programs. These programs were chosen because both of these 

providers have recent experience in starting sector programs in new locations. The highlights of 

these interviews are discussed in Box 7.1.

Picking the Right Sector

Chapter 6 presented nonexperimental evidence that Per Scholas’s impacts were largely 

generated by its impressive success in getting people into the relatively high-wage information 

technology (IT) sector. In fact, this sector has been booming for some time, and while Per Scholas has had to adapt to changing technology, the sector has not been characterized by the ups 

and downs experienced by, for example, the transportation sector, which is very sensitive to 

cyclical economic factors, or the environmental remediation sector, which is characterized by 

“boom and bust” periods. But this theory — that it was the choice of sector that was critical — 

does not explain why the other sites in the SEIS evaluation also had large wage effects. In addition to IT, the SEIS providers worked in many of the same sectors targeted in the WorkAdvance 

evaluation, such as the health care sector (like Towards Employment); manufacturing (like Towards Employment and, later, Madison Strategies Group); and construction (like St. Nicks Alliance).9 In addition, the health care sector has been a growth area for some time and has arguably 

been more recession-proof than IT. So if sector matters, how can it be that the SEIS providers 

had larger impacts than the WorkAdvance providers in many of the same sectors? 

It may be that a more textured understanding of what sectors are and how they are defined is required, one that takes into account providers’ skills and experience and the nature of 

the relationships that providers have with employers in the sector. Broadly defined sectors such 

as “health care,” “IT,” and “manufacturing” cover large swaths of jobs, which range from very 

low-wage jobs with little advancement potential to highly paid jobs at the other end of the wage 

distribution. It may be that the SEIS providers, due to their extensive experience in their sectors 

(including relationships with unions, in one case), were able to find the right subsectors within 

these broader sectors in which to place individuals most advantageously, and to place more individuals into those subsectors. The analysis in Chapter 6 found that it was providers’ success in 

placing individuals in targeted sector jobs, combined with those jobs having better characteristics, that explained site variation in impacts. Long experience in a sector may enable deeper 

webs of relationships to form, which can help providers achieve higher sectoral placement rates 

and may enable providers to find the best niches (or subsectors) within overall sectors. 

In other words, compared with the SEIS providers, the WorkAdvance providers, particularly the ones that were newer to the sectors, may have been less able to find the best jobs or 

employers (subsectors) in the sectors in which they operated, particularly early in the demonstration.

 9

Many environmental remediation jobs are in the construction industry.
179

Box 7.1

How Long Does It Take New Sector Programs to Hit Their Stride?

One of the main themes of this report is that it takes time for sectoral programs to reach 

maturity and be capable of producing value for program participants beyond what they 

could receive elsewhere. But what is involved in starting up a sector program? In late 2015 

to early 2016, MDRC WorkAdvance evaluators interviewed leaders at Per Scholas and 

Grant Associates, two organizations that have sought to replicate their original sector programs in new locations. (Per Scholas has replicated its program in five U.S. cities. Grant 

Associates founded Madison Strategies Group, which operated the Tulsa WorkAdvance 

program, and Madison Strategies Group’s initial leaders had run sector programs in New 

York City.) Even with extensive experience running such programs, leaders of the two organizations indicated that it takes between 18 months and three years to get a sector program in a new location to run at full capacity and for participants to have a program experience similar to what they would have received in the original program. Presumably, organizations new to the sector approach may require even more time. The start-up process 

may go more quickly, however, if an experienced sector program organization already operates in the location where it starts a new sector program. 

Leaders at Per Scholas, which has run sector programs for 20 years, estimated that it takes 

two to three years before a new provider in a new location is running at the capacity of 

their flagship New York City program. As one leader put it: “You can’t turn on a switch 

and expect everything to be perfect. It takes time to become a known quantity in the community.” Grant Associates leaders estimated this time frame to be 18 months. (In WorkAdvance, this would roughly correspond to the middle of program enrollment for the late 

cohort.) Interviewees emphasized that the goal is not simply to replicate an approach; because sectoral programs are inherently local, every program will be different. They also 

spoke of the importance of having local support on the ground to jump-start the relationships that are so critical to sector programs, and of hiring respected and knowledgeable local staff members to help avoid giving a “carpetbagger” impression. Leaders of both providers described a highly systematic expansion approach, driven by both numbers and relationships, that balances numerous factors, such as local capacity, funding, local need, 

and a highly detailed analysis of the local labor market. As an example, Per Scholas estimates the target population in a potential new location by conducting an analysis of the 

number of individuals with a high school degree but without postsecondary credentials. 

Leaders stressed the need to take local conditions into account in designing new sector 

programs: adjusting curricula, adding classes in growing subsectors (for example, in data 

security), tracking the number of positions open in different highly specific occupations, 

and understanding the geographic relationships between neighborhoods where potential 

program enrollees live and the locations of the relevant job openings. 

(continued) 
180

Connection with Employers

The SEIS researchers heavily emphasized the role of employer relationships and brokering as keys to the impressive impacts in that study. As in the SEIS, it was clear that the services at Per Scholas were greatly influenced by employers. Per Scholas has a strong reputation 

and deep relationships with large employers in New York City’s financial and technology sector, built up over years. While other providers, notably Madison Strategies Group and Towards 

Employment, worked closely with employers without seeing the same level of impacts, the impacts for the late cohort indicate that substantial improvement occurred over time. It may be that 

connections with employers and deep specialization in a field take more time to develop than 

the amount of time available in this study. Recall that in the SEIS, each provider had been operating for at least three years before the study started. All these threads point in the direction of 

program maturation and evolution as the key driver of site variation in impacts. 

A Focus on Training Rather Than Placement First

All three providers in the SEIS focused on training as the core intervention. It is possible that the placement-first track that was initially operated by Madison Strategies Group and 

Towards Employment did not work well and “depressed” the impacts for these two sites. There 

is clear evidence of strengthening impacts in the late cohorts (which were made up almost exclusively of training-first participants). In recent discussions, staff members at the Madison 

Strategies Group site emphasized that the early focus on placements was very important because the provider was new to Tulsa and had to establish credibility with employers through 

some “early wins.” However, the impacts at these two providers provide clear evidence that 

after the providers moved to a training focus, the impacts were much larger. This is not to say 

that training in and of itself will ensure impacts. The job training field has known for decades 

that training, in order to produce impacts, needs to be demand driven. 

Box 7.1 (continued)

The overall impression left by the interviews confirms that a sector program takes time to 

bear fruit. In addition to the considerations above, interviewees noted that success in a new 

community depends on bringing together a long list of actors: local funders, communitybased organizations, workforce boards, community colleges and training providers, community leaders, and, crucially, businesses. Grant Associates and Madison Strategies Group 

leaders felt strongly that the WorkAdvance placement-first track served a critical role early on in building credibility and currency with local employers, although the research suggests it did not yield strong impacts in the study.
181

Overall Assessment

So where do these various threads lead us? It is evident that sectoral placement matters; experience and evolution seem to matter; and a focus on training is important. One key ingredient underlying these findings seems to be the nimbleness of the provider. While important, a focus on 

training does not seem sufficient in itself; a provider has to provide demand-driven training that 

improves over time according to evolving relationships with employers. As discussed in Chapter 1, many training programs have failed to produce impacts when the training provided was 

not nimble and proactive about labor market conditions. Longevity in the field is helpful only if 

an organization is continuously “tapped in” to local demand conditions.

Given these findings, it is hard to see how a new organization can immediately produce 

impacts. Even the most effective providers need time to learn the sector, to understand how to get 

ahead of trends in the sector, and to calibrate services to meet employers’ evolving needs. The 

impacts of WorkAdvance provide a clear message to the field: With time, providers that are truly 

nimble and demand driven can be expected to produce impacts on earnings — sometimes quite 

large impacts. But these qualities are not to be found in every provider and take time to refine. A 

high priority for the field should be to provide curricula so that more providers can truly internalize the dual-customer, demand-driven emphasis that research suggests is the active ingredient. 

Considerations for Policymakers and Providers Designing Future 

Sectoral Employment Programs 

The WorkAdvance study highlights that it takes time and a very specific disposition and approach to operate an effective sector program. The Per Scholas results show the impressive benefits that can be attained once a sectoral provider is well embedded in a labor market and functioning in its prime. The findings here suggest several considerations for designing future sectoral employment programs:

• Plan for a long start-up period (up to two years) with minimal economic results. New sectoral programs require a shift from the rapid job placement approach many organizations use for workforce development. Maturity matters, particularly for allowing deeper relationships with employers. 

• Carefully study the local labor market at the subsector level. Find niches in

demand by local employers that are attainable (with training) for participants 

and that offer upward advancement. It is important to verify on an ongoing 

basis that there are jobs available for all graduates of training, and it is also 

important to find niches that most participants cannot access on their own. A 

key difference between sectoral training providers and traditional training 

providers is the ability to stay ahead of, or at least on top of, shifts in demand.
182

• Make sure that these subsectors truly have better wages and advancement 

prospects, compared with what individuals can find on their own.

• Do not underestimate the level of effort necessary to help individuals through 

training. The WorkAdvance training completion numbers were relatively 

high, but this was probably due at least in part to the substantial effort expended by the providers to keep individuals in training. The initial screening 

is also likely to be an important factor. 

• Focus on the funnel through the program, starting with the number of individuals who initially walk through the front door. Even at the provider with 

the largest impacts, many either didn’t complete training or completed training without finding a job in the targeted sector.

• Focus on the “wage premium” of the jobs targeted for training. Some providers were quite effective at getting individuals into the targeted sector, but the 

evidence suggests that, so far, jobs in some of these sectors had wages that 

were no higher than jobs that WorkAdvance group members found in other 

sectors (though the jobs did have some other beneficial characteristics, such 

as better hours or benefits).

• Plan for change. This can be changes in employer demand, the local economy, or other factors. Both the WorkAdvance evaluation and the SEIS found 

that demand shifted at every site.

• Do not underestimate the importance of soft (or “essential”) skill instruction. 

Employers and participants alike continually highlight the need for and importance of instruction in these types of skills. 

• Plan to employ staff members who possess the skills needed to operate a 

successful advancement program — strong job developers who get out from 

behind their desks and have close relationships with key employers; staff

members skilled in career coaching (rather than barrier removal); staff members who have expertise in assessing strong training programs (including 

hands-on ones) and are prepared to adapt curricula as the labor market changes, and who have very close ties to the needs of employers as well as experience in the industry or a willingness to gain it; and leaders who can integrate 

the members of teams, focus on key goals, use metrics to measure progress 

and success, and proactively take corrective action when they see problems.

A practitioner brief, scheduled to be released in late 2016, will discuss these lessons for practitioners in more detail. 
183

Revisiting the WorkAdvance Theory of Change

In Chapter 1, several reasons are mentioned for why WorkAdvance might not produce impacts. 

It is useful to return to those when making an overall assessment of results. The first concern 

was that individuals might not be able to finish training due to a lack of financial support. In 

general, this did not turn out to be an important issue. Training completion rates were high 

across all the sites, and the WorkAdvance group members at every site were much more likely 

to complete training and earn credentials than their control group counterparts. There is still 

room for improvement, as a sizable percentage of sample members did not finish training, but 

low training completion rates do not appear to explain variation in the impacts. 

Another possible breakdown in the theory of change would be if individuals completed 

training but were unable to be placed in the targeted sector. This factor does seem to be central 

to explaining the variation in impacts across sites. Rates of targeted sector placement were highest in Per Scholas (the site with the strongest impacts) and lowest at St. Nicks Alliance (the site 

with the weakest impacts thus far). To some extent this variation is due to different demand 

conditions in the two sectors targeted by these providers, but another reason is likely to be the 

strong relationships Per Scholas had developed with many large employers over years of working with them.

Another factor raised in the program theory discussion was the “sector premium,” that 

is, the expectation that targeted sector jobs would have better characteristics than jobs outside

the targeted sector. For all four organizations, there were some indications of better job quality 

for targeted sector jobs. However, with the exception of Per Scholas, either the quality advantage of the in-sector jobs was not great enough or (more often the case) the organization was 

unable to place enough participants into the targeted sector jobs to translate into large impacts. 

An open question, which will await longer-term follow-up, is whether the gains produced by WorkAdvance will grow or decay over time. The theory of WorkAdvance holds that 

the placement of people into sectors with viable career pathways as well as the provision of 

postemployment services will lead to a strengthening of impacts over time. In addition, at Madison Strategies Group and Towards Employment, training was more commonly experienced by 

the late cohorts than by the early cohorts, and some individuals were still in training at the end 

of follow-up for this report. Thus, more time is needed to come to a final assessment of the impacts of WorkAdvance. Longer-term follow-up may also shed light on whether targeted jobs 

need to have higher wages (and not just more opportunities for full-time rather than part-time 

employment) to generate long-term impacts.10 

 10If WorkAdvance participants (including the control group) are the type of motivated individuals who 

will eventually find full-time employment, sectoral wage premiums could well prove to be key to long-time 

impacts.
184

Conclusion 

Evidence suggests that skill-building in occupations with better jobs can be a means of increasing earnings in the long run for low-wage workers, as long as training is well aligned with the 

needs of employers, and participants can find employment in the sector for which they were 

trained. Several generations of experiments have also made clear, however, that there are limits 

as to what can be done on the employee side of the equation. Sector programs, in contrast to 

many programs from the past, are heavily demand driven, enabling them to bring workers and 

employers together in ways that solve local and regional economic challenges. The evidence 

suggests that future programs and evaluations should continue to include and examine this 

promising demand-side focus. WorkAdvance has provided the field with important evidence on 

both the potential and the challenges of this approach. The evaluation has highlighted what can 

be accomplished when a provider is deeply embedded in a field and constantly evolving, while 

at the same time underscoring the amount of time and the type of disposition a provider needs to 

have for the strategy to bear fruit. 

WorkAdvance is not the only program currently under evaluation that is designed to use 

a demand-driven skills acquisition approach as a means to advancement for low-income individuals. Several programs in the Pathways for Advancing Careers and Education (PACE)

demonstration use a broadly similar strategy.11 In addition, evaluations are under way of some 

programs funded through Health Programs Opportunities Grants that use a demand-driven 

training approach to help recipients of Temporary Assistance for Needy Families (TANF) and 

other low-income workers advance in the health care sector.12 Finally, some programs undergoing evaluation in the Department of Labor’s Workforce Innovation Fund portfolio use a similar 

strategy.13The fact that so many agencies and foundations are operating or supporting programs 

that have evolved in this direction suggests an emerging consensus about the need for a truly 

dual-customer approach to employment programs. In coming years there should be much more 

evidence available to determine the critical features of a successful sectoral program. Is the sector placement rate the key to impacts? How much better do the targeted jobs have to be and 

along what dimensions? Are impacts always smaller for incumbent or recently unemployed 

workers, or in stronger economies? Are there other key features not revealed by this study that 

critically affect impacts? Can the strategy work for workers with lower levels of literacy or numeracy? These ongoing projects have a strong potential to inform workforce policy about the 

scalability of this type of advancement-oriented, demand-driven skills-building approach.

 11Martinson and Gardiner (2014). 12Lower-Basch and Ridley (2013). 13See U.S. Department of Labor (2015).
185

Wider Policy Implications

Even if the final results of the abovementioned studies are positive, however, the difficulty of implementing successful sector-based interventions, coupled with the small size and 

specific focus of some of the models, raises questions about scalability. WorkAdvance, in particular, is a challenging model because individual providers have to implement several components simultaneously on their own. An alternative approach, which might aid scalability, would 

be to have different organizations coordinate to implement different components of the model 

— for example, taking advantage of the ability of the community college system to provide 

some program components. (Box 7.2 offers a look at such an approach.) Two sites in the SEIS 

evaluation outsourced training, similar to what was done in three of the WorkAdvance sites, 

suggesting that multiple approaches can be successful; still, careful thought needs to be given to 

how the outsourced training is better than what motivated workers would find for themselves.

And the Per Scholas results in WorkAdvance suggest caution about this strategy; its ability to 

control all aspects of service provision arguably led to an integrated service experience that may 

partly explain why their program is so successful. Similarly, a smaller provider may be more 

nimble in adjusting to changing employer needs.

Another challenge with expanding the scale of this strategy is that sector programs are 

inherently small because of the specialization that is necessary to truly understand the highdemand niches of the local labor market and to match appropriate individuals to job openings. 

But small programs can become important components of sectoral systems (or “career pathways” systems). In some cities and some labor markets, sector-based programs have been embedded in much broader initiatives, which also take advantage of feeder systems from “bridge” 

programs to enable a diverse segment of disadvantaged workers to enter the initiative. Project 

Quest,14 the initiatives implemented by the Instituto del Progreso Latino in Chicago,15 and the 

Career Pathways system in New York City are examples of programs that apply sector-based 

strategies on a larger scale and in some cases for a more disadvantaged set of workers. 

The WorkAdvance findings have particular relevance to implementation of the Workforce Innovation and Opportunity Act (WIOA). The legislation incorporates a strong sectoral 

focus and an emphasis on career pathways, as well as a provision to focus on a disadvantaged 

population similar to that targeted by WorkAdvance. Local Workforce Investment Boards and 

city and state government leaders should consider these findings when putting WIOA into practice, and draw upon the lessons herein as they craft requests for proposals and decide how best 

to oversee local workforce providers.

 14Osterman and Lautsch (1996). 15Martinson and Gardiner (2014). 
186

* * * 

These results are not the final word on the WorkAdvance evaluation. Further follow-up is 

planned, which would provide impact results at the three-year and five-year points.

Box 7.2

A Collaborative Way to Set Up a Sectoral Initiative:

The Northeast Ohio Example

As noted in Chapter 2, Towards Employment is an established community-based organization that provides a broad range of employment services for low-income individuals in 

the greater Cleveland area. While it had extensive experience with workforce and work 

readiness programs, it had not operated a sector program before. To operate WorkAdvance, Towards Employment, in contrast to the other three WorkAdvance providers, set 

up a multiagency partnership. Experienced organizations were responsible for delivering 

various aspects of the WorkAdvance services, reflecting the partners’ areas of expertise. 

This allowed Towards Employment to act as a “convener,” and draw on, rather than duplicate, existing service expertise in the Cleveland area. The involvement of multiple partners 

and providers was also intended to help foster the program’s long-term sustainability. 

Regarding training, for example, Towards Employment “purchased” instruction from local 

community colleges or a private technical school partner to be delivered to cohorts of 

WorkAdvance enrollees, in this way controlling the curriculum content. This arrangement 

also allowed Towards Employment to switch training partners if a provider turned out to 

be more or less cooperative with the WorkAdvance model’s approach or had better or 

worse relationships with employers. As another example, Towards Employment’s use of 

an industry association and a local manufacturing workforce intermediary — as opposed 

to relationships with individual employers — helped it to build sector relationships in a 

new area (manufacturing) more quickly than otherwise would have been possible. These 

intermediaries also provided industry-specific information in such areas as manufacturing 

skills pathways, state and national industry trends, credentialing, labor markets, employer 

engagement, and employer performance appraisal processes. The manufacturing intermediary also helped to identify job openings and match program participants with them. In 

addition to these “employer-facing” partners, Towards Employment contracted with other 

agencies to provide participants with a range of services, including social services and extra academic support when needed.

An arrangement similar to the one used by Towards Employment is probably typical of 

many sector programs around the United States, where there is existing knowledge and 

experience in certain areas and a desire to braid resources and services across different entities. This type of arrangement, however, is complex to set up and manage, and it requires 

the convener to monitor partners’ performance to ensure they operate in line with the program’s approaches and remain accountable for meeting the program’s specific goals.
Appendix A 

Data Collection, Analysis Methods, and 

Methodological Issues

189

This technical appendix describes the collection and analysis approach for the various types of 

data used in this report. The appendix largely follows the structure of the WorkAdvance 

program and the report. It starts with a discussion of recruitment and baseline data. This is 

followed by sections describing the qualitative, program-tracking, survey, cost, and unemployment insurance (UI) wage and benefits data. The appendix concludes with a section on the 

impact estimation approach. The remaining appendixes provide supplementary materials for 

Chapters 1 to 6 of the report.

Recruitment Data

About a year into the sample enrollment period, MDRC collected data on recruitment and 

intake from each provider.

1 These data were collected to help providers assess their recruitment 

strategies, to help evaluators understand the populations to whom the research results can apply, 

and to understand the process by which participants were identified as eligible for WorkAdvance services.

Data Collection

The first step in the process of study enrollment and random assignment was to market 

WorkAdvance and encourage individuals to participate. Each provider collected data on all 

applicants who expressed interest, either on site or by phone, in the WorkAdvance program. 

Staff members recorded how each individual had first heard about WorkAdvance and whether 

he or she was eventually enrolled in the study. If an individual was not enrolled, the provider’s 

staff recorded the step in the intake process during which he or she dropped out or was screened 

out.2 These data were collected, for each applicant, either in the provider’s management 

information system (MIS) or on a separate paper tracking tool developed by MDRC. Because 

some applicants did not enter the study, the providers shared copies of deidentified data by 

sending them to MDRC via FedEx or by uploading data files to the project’s secure file transfer 

protocol (FTP) website. Upon file retrieval, MDRC stored these files on a secure network which 

restricts access to a limited number of project staff members who manage and work with the 

data.

Data Analysis Methods

Using the abovementioned data, MDRC researchers conducted a “funnel analysis” to 

help identify (1) the recruitment sources that yielded the most applicants eligible for the study 

and (2) at what point applicants were selecting or being screened out of study eligibility.

 1

Please refer to notes in Appendix Table C.1 for further details on the data collection period, by site. 2

See Appendix Table C.2 for more site-level detail on the intake process, which varied by WorkAdvance 

provider.
190

The analysis tracked interested applicants from their first contact with the program until 

they either dropped out of the intake process or were randomly assigned in the WorkAdvance 

study.3 Applicants were categorized by their recruitment source and the stage of the intake 

process they were in when they left.

In preparation for this analysis, the data were securely transmitted to MDRC and 

checked for quality. Data issues identified varied by provider (due at least in part to site-level 

variation in the intake and data collection process):

• The recruitment data were checked for duplicates (the expected structure of 

the data file was one record per applicant), outliers, consistency across responses, and out-of-range dates. No major issues were identified, but MDRC 

made some minor changes to illogical responses (for example, dates with an 

out-of-range year were changed to the year in which the analysis was completed).

• Recruitment data from Towards Employment were available only in aggregate form, and the data covered a window of time rather than a cohort of applicants. To address these issues, MDRC made some adjustments to align the 

counts across intake steps and excluded applicants who were thought still to 

be in the intake process at the time the data report was generated. Also, only 

applicants from the 11 most common recruitment sources were included in 

the analysis.

• Per Scholas began collecting recruitment data at the time applicants attended 

the initial orientation, while the other three sites began collecting recruitment 

data at an earlier point in the intake process. When doing the cross-site analysis, MDRC did not report on drop-off before orientation for Per Scholas applicants.

• The intake process varied somewhat across the sites, and only common steps 

in the intake process were included in the cross-site analysis.

 3

The first point of contact with the WorkAdvance program varied by site. At Per Scholas, the first point of 

contact was attending the initial orientation. At St. Nicks Alliance and Towards Employment, the first point of 

contact was calling or showing up in person at the site. At Madison Strategies Group, the first point of contact 

was being scheduled for an orientation. After the first point of contact, data were collected to track the points 

where recruitment numbers dropped off (by site, where available; see Appendix Table C.4), assessment results 

(including eligibility screening), interview results, internal case conference results, and random assignment 

status. Please refer to Chapter 3 of the WorkAdvance implementation report (Tessler et al., 2014) for further 

details on analysis results.
191

Because this analysis was primarily focused on helping providers address recruitment 

challenges and because data were collected for only a small portion of the sample enrollment 

period and were deidentified, these data were not merged with other data used for the study. 

Moreover, no weights were applied to the data.

Random Assignment and Baseline Data

Random Assignment Process

This section describes the procedures WorkAdvance staff members performed when 

enrolling eligible applicants into the study. Random assignment occurred after the recruitment 

and screening process, described in the previous section, was carried out.

Once interested applicants were deemed eligible for WorkAdvance, staff members explained to them in detail the goals of the study, the random assignment process, the data that the 

research team would collect, and the benefits and risks of participation. Next, prospective study 

participants received a WorkAdvance research document packet that contained two forms: (1) 

the WorkAdvance Study Informed Consent Form and (2) the state waiver authorizing the 

release of UI wage and benefits data for research purposes.4 Prospective study participants also 

received a verbal explanation from WorkAdvance staff members about the content of each 

form. Study participants needed to sign all consent forms and undergo random assignment to 

have a chance at receiving WorkAdvance employment and training services.

After the consent forms were signed, a WorkAdvance staff member logged into 

MDRC’s online random assignment application and entered the study participant’s identifying 

information and selected (self-reported) measures about the participant’s demographic, education, and employment history. Key identifiers (like name and Social Security number) were 

entered twice into the database. The application recognized discrepancies in data entry of these 

fields and required the WorkAdvance staff member to correct one or both versions before

submitting the record. Finally, the application randomly assigned the study participant to either 

to the WorkAdvance group or the control group.5

 4

This release form was administered only at Per Scholas, St. Nicks Alliance, and Madison Strategies 

Group, as the Ohio administrative records agency (which provided UI wage and benefits data for Towards 

Employment sample members) did not require the research study to obtain study participant and notary 

signatures authorizing the release of wage and benefits data. 5

At the Per Scholas site, for the last 12 months of the sample build-up period, three-way random assignment was used to help accommodate the provider’s efforts to fill training slots in line with multiple funding

streams. Per Scholas had a separate grant to serve Supplemental Nutrition Assistance Program (SNAP) 

recipients with WorkAdvance-like services. To ensure that Per Scholas satisfied the sample size goals for the 

(continued)
192

MDRC’s random assignment process uses the widely recommended technique of 

“permuted blocks” in which a block of sample members consists of a given number of control 

and program assignments — usually the number of control assignments equals the number of 

program assignments in each block, for example 10 of each. This is done to ensure an even flow 

of program and control participants, which makes it easier for staff members to manage their 

programs. The order of assignments is sorted by a unique random number associated with each 

assignment. To create a project’s random assignment order several different block sizes are used 

to keep the assignments as unpredictable as possible. Because blocks are constructed of equal 

numbers of program and control assignments, the overall sample will be made up of equal 

program (P) and control (C) assignments — or nearly so, if the study ends in the middle of a 

block. This method can, in principle, be used for uneven P:C ratios (for example, 60 percent Ps, 

40 percent Cs) as well as designs requiring more than two research categories (such as one-third 

each of P, Q, and C assignments).

The computer programs that carry out random assignment are implemented in secure 

online applications. Sites are generally provided with their own sequences so that each will stay 

close to the desired P:C ratio. In addition, the occasional site may require smaller block sizes 

than other sites for operational reasons.

Immediately following random assignment, WorkAdvance staff members recorded the 

assignment of each WorkAdvance group member into their MIS, which all staff members 

consulted upon engaging individuals in services, to help ensure that control group members 

were not served by WorkAdvance for the full three-year service embargo period. The Year 2 

Survey results, as explained in the main report, suggest that the random assignment process 

worked as expected from June 2011 through June 2013, when the study sample was enrolled. 

The treatment (access to WorkAdvance services) and counterfactual (no access) conditions 

were implemented as originally intended by the designers of the evaluation.

Data Collection

As described above, MDRC collected a variety of demographic and employment history information on each study participant at the time of random assignment. Baseline data 

collection and random assignment both ended on June 30, 2013, as expected.

 

funders of the SNAP grant as well as the goals of this evaluation, a three-way random assignment process was 

implemented on July 5, 2012. Individuals who qualified for both WorkAdvance and the SNAP contract were 

randomly assigned to one of three groups: the WorkAdvance program group, the WorkAdvance control group, 

or the “original Per Scholas program group,” whose members were offered services similar to WorkAdvance 

services (minus postemployment support), funded by the SNAP grant.
193

To collect this information, provider staff members primarily used an MDRC online 

baseline data collection tool. To help ensure data quality, the online baseline collection tool 

imposed certain rules for accepted values, controlled skip patterns (where a response might lead 

to additional questions), and prompted users with various messages if they reported potential 

errors or did not answer all the questions. If the online random assignment system was not 

functioning for any reason, staff members collected baseline information on a paper form that 

was later sent to MDRC for data processing. In this scenario, provider staff members were 

instructed to review the baseline forms to check for the types of issues the online tool controlled 

for automatically, but the manual process was somewhat more error prone. In total, around 98 

percent of the study sample members had their data entered into the online tool, with the 

remaining 2 percent having data recorded via the paper tool. The quality of these baseline data 

was high overall but was somewhat lower for baseline data collected via the paper form.

Data Analysis Methods

In general, the strict validation rules built into the baseline data collection system greatly reduced the number of data analysis issues encountered. Still, MDRC routinely ran quality 

control procedures on the baseline data, to check for the following: 

• Reasons to exclude enrolled participants from the impact analysis.

6 Aside 

from withdrawing from the study,

7 participants could be excluded from the 

impact analysis for a few reasons: being served under another funding stream 

(N = 22), missing a signed consent form (N = 1), or having been enrolled before the official start of random assignment (N = 5).

• Systematic problems with out-of-range, missing, and outlier response values. 

Only minor issues of this type were found, which appeared to be driven primarily by keystroke errors and misunderstood questions. MDRC periodically 

provided technical assistance to the provider staff members, to help ensure 

the quality of responses over time.

• Logical and consistent item response patterns across research groups. No 

major issues of this type were found.

MDRC conducted a special analysis to check for statistically significant differences in 

select baseline characteristics by research group. As expected, only a few statistically significant 

differences in baseline characteristics were found. In addition, a logistic regression was run to 

 6

All sample members enrolled through Compass, a provider in northeast Ohio, were excluded from the 

impact sample when the site was dropped from the study. 7

A total of 20 participants withdrew from the study after random assignment. 
194

test whether key baseline characteristics could predict whether a participant was in the program 

group, both for the full sample and at each site. The model included 21 covariates that were 

regressed on a program group indicator (P = 1 = program group; P = 0 = control group). Neither 

the cross-site model (p-value = 0.755) nor any of the site-specific models are statistically 

significant, indicating that program and control group members do not differ significantly 

across the key baseline characteristics.

MDRC also conducted a special analysis to check for statistically significant differences 

in select baseline characteristics by random assignment cohort (that is, when participants were 

enrolled). The results of this analysis, which are presented in Appendix Tables A.1 and A.2, 

suggest that — for each of the four WorkAdvance sites — study participants who enrolled in 

the first half of the sample build-up period are different from those who enrolled in the second 

half.8

Qualitative Data

Qualitative Data Sources

Focus Group and Participant Interview Data

Focus groups were conducted once for each sector in the program: one focus group 

each at Per Scholas, St. Nicks Alliance, and Madison Strategies Group, and two focus groups at 

Towards Employment for the health care and manufacturing sectors. The focus groups were 

conducted in Quarter 3, 2013, two years after sample enrollment began at Per Scholas, 20 

months after sample enrollment began at St. Nicks Alliance, and 19 months after sample 

enrollment began at Madison Strategies Group and Towards Employment. The focus groups 

were made up of 8 to 12 WorkAdvance group members who were interested in participating, 

with preference given to individuals in a shared training cohort who were close to the end of 

their occupational skills training. Participants were informed of their option to participate in 

advance. Participants in the focus groups were each served food and given $25 cash as a token 

of appreciation for taking the time to speak with MDRC researchers. They were asked about 

their motivations for applying to the program, their experiences with WorkAdvance skills 

training and other services, and their general impressions of the program.

 8

Study entry year is included as a covariate. More details on covariates can be found in the last section of 

this appendix.
The WorkAdvance Study

Appendix Table A.1

Selected Characteristics of Research Sample Members at Baseline, by Random Assignment Cohort and Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Early Late Early Late Early Late Early Late

Characteristic Cohort Cohort Cohort Cohort Cohort Cohort Cohort Cohort

Demographic characteristics

Female (%) 9.1 18.0 *** 14.8 14.1 18.1 13.9 64.6 52.7 ***

Average age (years) 31 31 35 36 37 33 *** 37 34 ***

Race/ethnicity (%)

Latino/Hispanic 33.5 38.9 19.5 26.8 * 4.8 7.2 4.1 5.1 **

White 6.2 4.1 7.2 6.6 37.9 40.4 14.2 22.5

Black/African-American 45.4 43.4 67.7 56.8 29.6 27.3 75.3 66.4 a Other race 14.9 13.6 5.6 9.9 27.8 25.1 6.4 6.0

Parent of one or more children (%) 27.8 24.4 51.6 38.0 *** 51.6 51.9 49.6 54.0

Education level (%)

Highest level of educational attainment

Less than GED certificate or high school diploma 0.3 0.0 15.9 7.2 *** 6.8 5.3 *** 11.9 0.0 ***

GED certificate/high school diploma 38.5 35.4 41.1 48.4 30.9 40.3 34.5 39.7

Some college 34.0 30.7 27.5 25.3 48.7 47.5 41.4 52.4

Associate's degree/2-year college 8.3 11.7 5.0 10.4 7.1 4.4 6.1 3.7

4-year college degree or more 19.0 22.2 10.5 8.6 6.5 2.5 6.1 4.2

Already has a license/certificate in targeted sector 5.1 1.9 ** 3.5 0.0 [***] 17.0 10.3 *** 23.8 26.1

Employment status

Number of months of current unemployment spell (%)

Never employed 4.0 3.2 *** 2.7 1.4 0.6 0.8 * 2.9 2.5 ***

Currently employed 17.4 7.9 9.8 11.3 28.5 25.0 17.7 35.4

12 months or less 50.8 62.3 54.1 57.9 53.1 62.5 43.8 44.2

More than 12 months 27.8 26.6 33.3 29.4 17.8 11.7 35.7 17.8

(continued)

195
Appendix Table A.1 (continued)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Early Late Early Late Early Late Early Late

Characteristic Cohort Cohort Cohort Cohort Cohort Cohort Cohort Cohort

Number of months in current or most recent job (%)

b 12 months or less 57.0 53.8 58.0 49.3 * 60.8 70.6 *** 55.4 61.2

More than 12 months 43.0 46.2 42.0 50.7 39.2 29.4 44.6 38.8

Is or has been employed in targeted sector (%) 7.8 8.0 2.4 1.9 [ ] 17.0 12.6 24.3 36.7 ***

Average hourly wage at current or most recent jobc

 ($) 11.99 11.80 12.59 13.47 10.31 10.33 9.68 10.07

Worked full time (35 or more hours per week)d

 (%) 58.5 58.8 64.7 65.6 70.7 72.0 61.5 60.8

Circumstances that may affect job change or retention (%)

Previously convicted of a crime 10.7 10.1 26.1 12.7 *** 39.1 40.3 22.9 27.8

Previously incarcerated 8.0 4.1 ** 25.6 9.5 *** 36.3 32.2 9.3 14.7 **

Income

Average monthly family income ($) 714 568 ** 665 730 860 753 471 727 ***

Income sources (%)

Food stamps/SNAP 22.5 11.4 *** 39.5 44.8 35.9 33.1 64.1 46.5 ***

Welfare/TANF 6.2 6.3 15.0 12.3 0.6 0.8 [ ] 5.2 2.5 *

Unemployment insurance benefits 25.4 23.1 27.8 20.9 * 9.2 5.8 * 11.9 9.1

Sample size (total = 2,564) 374 316 258 221 337 360 345 353

SOURCE: MDRC calculations from the WorkAdvance baseline information form.

NOTES: GED = General Educational Development; SNAP = Supplemental Nutrition Assistance Program; TANF = Temporary Assistance for Needy Families.

The early cohort includes all sample members randomly assigned through Quarter 3, 2012. The late cohort includes all sample members randomly assigned in 

or after Quarter 4, 2012.

Chi-square tests were used for categorical variables and t-tests were used for continuous variables to assess the difference in average characteristics between 

program and control group members. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. Square brackets 

indicate the chi-square test may not be valid due to small sample sizes.

Sample sizes may vary because of missing values.

a

"Other race" includes sample members who identify as non-Hispanic and listed "Asian," "American Indian," or "Other" as their race, including sample 

members who answered "multiracial." 

bMeasure includes sample members who have never been employed.

cWages for sample members who have never been employed are counted as $0.

d"Worked full time" does not include sample members who have never been employed. 

196
 The WorkAdvance Study 

Appendix Table A.2

Estimated Regression Coefficients for the Probability of Being in the Early Cohort

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Characteristic Parameter Estimate Parameter Estimate Parameter Estimate Parameter Estimate

Female -0.745 *** 0.257 0.228 0.610 **

Black -0.111 0.376 0.183 0.417 **

Hispanic -0.384 * 0.332 -0.339 0.097

Age -0.012 -0.019 0.040 *** 0.031 ***

Under age 24 -0.267 -0.812 ** 0.436 * 0.302

Number of children living with respondent 0.285 ** 0.143 -0.027 -0.080

Born in state 0.427 * 0.051 0.117 0.135

Highest level of education -0.132 ** NA NA NA

Highest level of education is GED or less NA 0.511 ** NA NA

Highest level of education is 2- or 4-year degree or more NA NA 0.679 ** NA

Highest level of education is high school, GED, or less NA NA NA 0.335 *

Already has a license/certificate in targeted sector 0.779 14.840 0.337 -0.053

Currently employed at baseline 1.085 *** -0.219 0.161 -0.408 *

Monthly family income 0.000 0.000 0.000 0.000

Employed in at least 1 quarter in year prior to RA 0.153 -0.664 ** -0.334 -0.323

Employed in all 4 quarters in year prior to RA -0.275 -0.647 * -0.232 -0.489 *

Number of quarters employed in 3 years prior to RA -0.043 0.166 *** 0.008 -0.038

Total earnings in quarter prior to RA 0.000 0.000 0.000 0.000

Total earnings in 3 years prior to RA 0.000 * 0.000 0.000 0.000

Received unemployment insurance benefits in 

3 years prior to RA -0.242 -0.568 ** -0.098 NA

Randomly assigned to environmental remediation sector NA 2.094 *** NA NA

Randomly assigned to health care sector NA NA NA -0.171

R-squared 0.095 0.264 0.080 0.123

Chi-square statistic 68.549 146.823 58.443 91.849

P-value of chi-square statistic 0.000 *** 0.000 *** 0.000 *** 0.000 ***

Sample size (total = 2,564) 690 479 697 698

(continued)

197
198

During the focus groups, a total of 20 interested participants were randomly selected to 

be interviewed individually as well: four each from Per Scholas, St. Nicks Alliance, and 

Madison Strategies Group; and four each from Towards Employment’s manufacturing and 

health care programs. MDRC researchers interviewed all these participants in Quarter 3, 2013, 

and contacted these same participants in Quarter 4, 2013, for follow-up interviews. In addition, 

informal case reviews were conducted for each of these participants with the staff members

most familiar with them.

Ultimately, 18 of the original 20 participants were able to be interviewed in person 

again three months later (Quarter 4, 2013). In-person interviewees were given $25 cash as a 

thank-you; phone interviewees were sent $25 gift cards after the interviews. Interviewees were 

asked about their motivation for applying to the program, interest in the sector, plans for the 

future, and experiences with WorkAdvance services, and for feedback about the program.

Sample interview questions for participant focus groups and participant one-on-one interviews are included in Appendix Box A.1. The convenience sampling and small sample sizes 

for focus groups and participant interviews limit the generalizability of findings from these 

sources.

Employer Interviews

Phone interviews with employers of WorkAdvance group members were conducted in 

Quarter 4, 2013. MDRC researchers worked with WorkAdvance providers to identify individual staff members at employers who could comment on the relationship between WorkAdvance 

and their firms. MDRC interviewed a total of 21 employers: four with connections to Per 

Scholas, six with connections to St. Nicks Alliance, five with connections to Madison Strategies 

Group, and six with connections to Towards Employment — three health care employers and 

Appendix Table A.2 (continued)

SOURCES: MDRC calculations from the WorkAdvance baseline information form and 

unemployment insurance administrative records from New York State Department of Labor, Ohio 

Department of Jobs and Family Services, and Oklahoma Employment Security Commission.

NOTES: NA = not applicable; GED = General Educational Development certificate; RA = random 

assignment.

The early cohort includes sample members randomly assigned through Quarter 3, 2012. The late 

cohort includes sample members randomly assigned in or after Quarter 4, 2012.

A chi-square test was applied to differences between outcomes for the early and late cohorts in the 

analysis for the full research sample. Statistical significance levels are indicated as follows: *** = 1 

percent; ** = 5 percent; * = 10 percent.
199

Appendix Box A.1

Sample Interview Questions for Participant Focus Groups

and One-on-One Participant Interviews

I. Interest and Expectations

1. Why did you sign up for this program/training? 

a. What did you hear about [WorkAdvance program] that got your attention? What 

attracted you to the program? (Probe: Was the fact that it was free the main thing 

that stood out? Did you hear that you could earn more money by participating in 

this program? Did you hear you could establish yourself in a new career?)

b. What did you hope/expect to get from the program? We are not asking what you 

hope the result will be of your participation in this program, but what kinds of 

skills or services did you hope to get from the program. Please be specific. 

(Probes: specific occupational skills, help with your résumé, job leads, career 

coaching.) 

c. Did you consider other ways of getting these skills or services? If so, what ways 

or where?

2. Were you specifically looking to work in the __________ industry when you first applied? Why? 

a. What interests you about this sector? 

b. What did you know about the _______ industry and the types of jobs that are 

available in that sector before you started training? Were there specific jobs you 

were hoping to be able to get? Which ones? 

c. For those of you who were not specifically interested in __________, what interested you about this training/program enough to sign up anyway?

3. This program tries to help people not only find a job, but also plan to move ahead in 

their jobs or careers later on. What does “moving ahead” or “moving up” in a job or 

career mean to you? (Probes: Getting a better-sounding title? Earning more money? 

Getting better hours? Getting better benefits? Becoming a supervisor?)

II. Occupational Skills Training

1. Please describe the occupational skills training you are participating in. We are not referring here to general life skills or job readiness skills training, but to the sectorspecific skills training.

(continued) 
200

Appendix Box A.1 (continued)

a. What type of training program was/is it? What were/are you being trained to do? 

What type of job are you expecting to look for?

b. Please describe a typical day in this training class. 

i. How did you spend your time? 

(Probes: Did you spend most of your time in or outside of a classroom? 

Did you spend most of your time sitting at a desk and listening to a teacher 

or working hands-on to learn new skills — for example, taking apart a 

machine? Did you work individually, or were there group projects? Was 

there a lot of reading? Did you feel comfortable asking questions?) 

ii. What did you like or dislike about how the training content was delivered?

c. Did you visit any employers or work sites? Did employers come to visit your 

class and talk about what types of jobs might be available? What kinds of things 

did you learn from those visits?

d. We understand that you were in training with (read whichever applies): other 

members of this program (including others in this focus group) / other people in 

the general population — not necessarily with people from [provider] / a mix of 

people, some from [provider] and some not. What did you like or dislike about 

this arrangement?

e. While you were in training, was there any discussion about career or job planning 

beyond getting your first job in the _______ sector? Please describe who started 

and participated in those discussions and what you talked about.

f. During the occupational skills training, were you in touch with the program staff 

from [provider]? In what ways? (Did they visit the training site? Did you receive 

phone calls or e-mails? How frequently? Did you visit the program offices?)

g. Besides paying for training, what other forms of assistance did the program provide? (Probes: transportation vouchers, tools, equipment, books, licensing fees, as 

well as other supports from the staff.) 

h. Were there times during your training program when it was difficult to participate? What made it difficult? How did you handle those situations? Did you ever 

consider dropping out altogether? If yes, what made you feel that way? How did 

you handle that and stay in the training?

i. [High Priority] Do you feel that the training you received through [provider] was 

different from the kind of training you could have received through another program? Or if you have ever participated in other training programs, was this training different? If yes, how?

j. What would you change about the occupational skills training, if anything? What 

do you think worked well?
201

three manufacturing employers. The interviewees were asked how they learned about WorkAdvance, their experiences with WorkAdvance participants, and their relationships with the 

WorkAdvance providers. Sample interview questions for employers are included in Appendix 

Box A.2.

Staff Interviews and Program Observations

Face-to-face and phone interviews with program staff members were conducted at various points over the study period. MDRC researchers collected information on staff members’ 

understanding of the goals and purposes of WorkAdvance; recruitment and enrollment; development and delivery of the specifics of the coaching, skills training, and career readiness 

training components; and participant motivation and engagement. As an example, Appendix 

Box A.3 presents a list of selected staff interview questions that were asked regarding the 

postemployment component of WorkAdvance.

MDRC researchers and consultants conducted site visits and observations of components of the program throughout the study. This research included observations of intake 

processes and orientations at Per Scholas, St. Nicks Alliance, and Madison Strategies Group in 

Quarters 2 and 3, 2011; career readiness training at Towards Employment and Madison Strategies Group in Quarter 3, 2013; career coaching at Per Scholas in Quarter 1, 2014, at Madison 

Strategies Group in Quarter 2, 2014, and at Towards Employment in Quarter 3, 2014; and 

postemployment alumni workshops at Per Scholas in Quarter 1, 2014.

Provider-Created Program Materials

During site visits, MDRC researchers collected samples of materials used by the providers in their implementation of WorkAdvance. These materials included curriculum documents for career readiness training at all four sites, recruitment materials such as flyers, and 

scripts for career coaching sessions. These materials were analyzed to help MDRC researchers 

understand the content, length, and intensity of the career readiness training; methods and 

messaging for recruitment; and the content of the career coaching component of the program.

CEO Social Innovation Fund Quarterly Progress Reports

MDRC researchers drew some descriptive information from quarterly grant reports that 

the WorkAdvance providers submitted to New York City’s Center for Economic Opportunity 

(CEO).
202

Appendix Box A.2

Sample Interview Questions for Employers

I. Questions Geared Primarily Toward More-Involved Employers

1. How did you first hear about (PROVIDER)? Did someone refer you to (PROVIDER) 

or did (PROVIDER) staff reach out to you directly? What message did you first hear 

about (PROVIDER) that made you think it might be worth pursuing a relationship?

2. What were you hoping to achieve by working with (PROVIDER)?

3. A primary goal of (PROVIDER) is to be sure that the training and other services they 

provide to eventual job applicants in the _____ sector are aligned closely with the 

needs of local employers in the sector. 

a. Has staff at (PROVIDER) ever actively sought your company’s input or 

guidance about the training or other activities they conduct — for example, 

the way they screen applicants for their training program, the technical or 

soft skills curriculum they use, etc.? If yes, which activities have they sought 

input on? 

b. Has your company ever had the opportunity to provide this kind of input or support to (PROVIDER)? If yes, please describe.

[Probe, if necessary]: Other types of guidance or input they might seek could include:

i. Selection of targeted occupations and description of career paths

ii. Placement services

o Visit training classroom to describe jobs and workplace environment

o Employer site tours

o Mock interviews

o Work experiences such as internships or mentoring

4. What kinds of services have you received from (PROVIDER)? 

5. Did (PROVIDER’s) services help you and your company in any of the following 

ways? 

i. Reduced time to hire

ii. Reduced turnover

iii. Troubleshooting with new hires — for example, assistance to:

o Improve attendance

o Improve job performance

iv. Reduced safety infractions

v. Help filling skilled positions beyond entry level

6. [If employer has hired a relatively large number of WA graduates]: Have any of these 

services saved your business money or helped your business grow or expand?

(continued) 
203

Qualitative Data Analysis

Qualitative data collected for the WorkAdvance evaluation were analyzed and are presented throughout the implementation and impact reports to give context to the program and 

describe how WorkAdvance was implemented. The interview and focus group data provide rich 

information that added to the quantitative baseline and participation analyses. Interview transcriptions and notes were coded into cross-cutting themes and trends. Codes were developed 

based on key research questions, emerging themes and lessons, and other topics of inquiry.

Appendix Box A.2 (continued)

7. Did (PROVIDER’s) services to your company meet your company’s expectations? 

8. How does the experience of working with (PROVIDER) compare with the experience 

of working with your other recruitment sources or other staffing agencies?

a. Is (PROVIDER) an important source of recruitment for your company when 

you are trying to fill jobs in the ____ industry/occupation? 

b. How do you evaluate if these recruitment sources are effective in terms of 

providing qualified workers? 

c. Do you expect to hire again from this provider?

d. Would you recommend them to other employers?

9. Are there any other ways in which your company benefited from the services it received from (PROVIDER)?

10. How could (PROVIDER) have improved your company’s experience with them?

a. [if provide feedback:] Did you share any of that feedback with (PROVIDER)?

11. Are there any services (PROVIDER) could offer to your company to be helpful in 

meeting your needs as a (COMPANY INDUSTRY [ex: trucking provider])? What 

kinds of services?

a. [if yes:] Have you ever shared that feedback with (PROVIDER)? 

12. Are there any ways that (PROVIDER) could better prepare individuals for jobs with 

your company or in your industry more broadly? If yes, how?

a. [if yes:] Have you ever shared that feedback with (PROVIDER)? 

13. WRAP-UP QUESTION: Are there any other ways that you’ve interacted with 

(PROVIDER) that we haven’t touched on here today?
204

Sample codes included basic program features such as “career readiness” or “occupational skills 

training” and implementation processes such as “recruitment,” while other codes explored 

trends, such as “interest in sector” and “institutional context.” Coded interview excerpts were 

then aggregated and analyzed accordingly.

Appendix Box A.3

Selected Interview Questions for Provider Staff Members

Regarding Postemployment Component

1. In your interactions that involve individualized communication, what do you usually 

discuss? 

a. Can you give me an example of some of the points you touch on, or how a typical 

post-employment conversation might flow? 

b. What do you do to prepare for those conversations, if anything? 

c. Do you discuss or review the Individual Career Plan (ICP)? 

d. Do those conversations produce action items or next steps? What are they? 

e. How does the medium of conversation (phone, email, etc.) change the content of the 

discussion? 

f. After those conversations, do you send any written record of what was discussed? Is 

the ICP updated in each conversation with the participant? If so, does the participant 

receive a copy of the updated ICP? 

2. How is follow-up conducted after advancement conversations? 

a. What are some of the challenges participants face in following through with next 

steps and advancement action items?

b. Do you use incentives to motivate participants to follow through on the next steps or 

to reach specific milestones?

3. Does anyone else reach out to or stay in touch with participants postemployment that you 

haven’t yet mentioned? 

a. To that end, how do staff coordinate and communicate their respective participant interactions?
205

Program-Tracking Data

Quantitative Program-Tracking Data Collection

Staff members at all four providers recorded information for each WorkAdvance group 

member on his or her participation in the program between June 2011 and December 2014.9

Staff members tracked participation in activities in their own MIS or using paper tools MDRC 

designed for special analyses. These data were securely transferred (and stored on a restricted 

access network) to MDRC every four to six months. The program-tracking data presented in 

this report cover WorkAdvance group participation in services and activities within the first 18

months of program entry. Provider staff members recorded data on career readiness services, 

supportive services, occupational skills training, placement (not shown in this report), and 

career advancement coaching services.

Before collecting these data, MDRC conducted an extensive MIS assessment with each 

of the providers to evaluate system capacity to reliably track participants’ engagement in 

WorkAdvance services and activities. This assessment comprised a series of discussions with 

program staff members and data system administrators about the WorkAdvance service flow 

and the procedures for recording and managing information about the providers’ engagement 

with participants along the way. Data systems and paper case files were also reviewed to check 

for consistency and completeness across information sources.

Data Analysis Methods

Using program-tracking data, MDRC analyzed program participation for all WorkAdvance group members. The analysis covered program activities that occurred between June 

2011 and December 2014.

Upon receipt of the data files, MDRC performed its standard data quality procedures, 

which included checking for missing, out-of-range, outlier, and overwritten data. Provider staff

members routinely helped MDRC resolve data issues identified in these files to help facilitate 

an accurate interpretation of the participation in WorkAdvance services. Further, for the purpose 

of making an overall assessment of how well these data represent the services program participants received, MDRC checked each provider’s data with a few questions in mind:

 9

The WorkAdvance sites began enrolling individuals in the study in different months: Per Scholas began 

in June 2011; St. Nicks Alliance began in September 2011; and Madison Strategies Group and Towards 

Employment began in October 2011. All sites completed study enrollment at the end of June 2013.
206

• Are the activity trends reasonable relative to the provider’s planned service 

sequence? For this purpose, where possible, MDRC compared activity rates 

among participants with different lengths of follow-up.10

• Are activity rates the same as or similar to the provider’s monthly performance monitoring reports? This is a measure of whether the providers and 

MDRC had the same concept of how to define and measure participation in a 

WorkAdvance activity.

• Is the provider’s account (as recorded in the MIS) the same as or similar to 

the report made by the participant in the follow-up survey regarding what 

she or he did in the program? There are often some differences between 

what a program-tracking system will show and what an individual may recall 

or think of during a survey interview when asked specific questions about 

what she or he did. Additionally, WorkAdvance participants may have received services outside the WorkAdvance program. Because of this, program-tracking data and survey data are not expected to match completely, 

but this type of comparison does provide some measure of the extent to 

which provider staff members use their systems to record participant activities. Appendix Table A.3, which illustrates the extent to which training participation rates varied between the MIS and survey data, shows that the providers’ training records are largely consistent with WorkAdvance group 

members’ responses to the question about their participation in sector training. For example, 83 percent of Towards Employment WorkAdvance group 

respondents indicated that they either did (56 percent) or did not (27 percent) 

participate in training in the targeted sector, and the provider’s MIS data support these responses. The remaining 17 percent of the site’s WorkAdvance 

group respondents did not have consistent records across the data sources, 

and the results suggest that these respondents had challenges recalling when 

they participated in targeted sector training.

Although the abovementioned data quality checks suggest that the providers’ data systems contained a reliable record of program participation among WorkAdvance group members, this data source is subject to limitations. First, the estimated rates of participation in 

employment and training services or activities are more representative of participants who were 

actively engaged in WorkAdvance services than of those who were not engaged. For instance, 

employment service data for participants who had frequent spells of disengagement are likely

 10For an example, see Table 4.1 in Tessler et al. (2014). 
207

to be less complete than data for participants who remained engaged throughout the service 

period, since it is not always easy for providers to verify participants’ employment status 

without first establishing direct contact with those participants.

Second, some providers do not systematically track certain activities in their MIS systems, so the research team had to create customized data collection instruments (as an example, 

MDRC designed a career advancement coaching form and asked the providers to complete the

forms for a three-month period during coaching sessions with WorkAdvance participants), and 

staff members within the same program may record information differently. As such, engagement in some services was not tracked for the full service period (for example, advancement 

coaching). In other cases, MDRC made reasonable adjustments to data that were collected 

throughout the service period and were sometimes inconsistent or incomplete. (For example, 

most providers were not able to report the date that participants dropped out of training, so

MDRC used the midpoint of the training cycle as the participant’s last day in training.) Where 

this was not possible, MDRC relied on other data sources to help measure participation in 

WorkAdvance services (for example, the Year 2 Survey).

The WorkAdvance Study

Appendix Table A.3

Comparison of MIS and Survey Reports of Participation in

Targeted Sector Training Since Random Assignment

 Among WorkAdvance Group Survey Respondents, by Site

Madison Towards

Unadjusted outcome

Per Scholas St. Nicks Alliance

N %

Strategies Group

N %

Employment

N % N %

Not in training 11 3.8 25 12.3 85 28.6 77 27.0

MIS-reported

training only 77 26.8 58 28.3 48 16.2 35 12.2

Survey-reported

training only 0 0.0 12 5.9 22 7.4 14 4.9

MIS and survey-reported

training 199 69.3 109 53.2 142 47.8 159 55.6

Sample size 287 205 297 286

SOURCES: MDRC calculations from program tracking data managed by Per Scholas, St. Nicks Alliance, 

Strategies Group, and Towards Employment, and from responses to the WorkAdvance Year 2 Survey.

Madison 

NOTES: N = number (sample size); MIS = management information system.

Sample sizes may vary due to missing values.
208

Summary participation outcome measures for employment services, education, and 

training were calculated for all study participants, as well as by site. Measures at 18 months are 

shown in the main report, and site-level definitions for measures are shown in Appendix Table 

C.3.

Tests for statistical differences across sites were not performed.

Year 2 Survey

This section answers the following questions about the Year 2 Survey: (1) How many sample 

members responded to the survey? (2) How were these response rates achieved? (3) What 

procedures were followed to help ensure data quality? and (4) To what extent are survey 

respondents representative of the full sample?

How Many Sample Members Responded to the Survey?

MDRC contracted with Decision Information Resources (DIR) to design and administer the Year 2 Survey. MDRC has worked with DIR on surveys for other studies, including 

evaluations of career advancement initiatives for low-income individuals and households. DIR 

staff members used their Computer-Assisted Telephone Interviewing (CATI) call center and 

database system to conduct all interviews. Per agreement with MDRC, DIR sought to interview 

at least 80 percent of both WorkAdvance group members and control group members at each of 

the four sites. As illustrated in Appendix Table A.4, DIR met this goal for the WorkAdvance 

groups and came close for the control groups at each WorkAdvance site.

How Were These Response Rates Achieved? 

DIR put in place multiple efforts to achieve high survey response rates:

Respondent location efforts. DIR used standard locator databases to find additional 

contact information, beyond what was collected at the baseline interview. 

Marketing efforts. MDRC worked with members of the DIR survey team to create 

marketing materials (including letters, flyers, and e-mail messages) and scripts for marketing 

phone calls to encourage participants to complete survey interviews.

Financial incentives. Respondents received a gift card after completing an interview. 

Notice of the gift cards was included in marketing materials. Throughout the fielding period, 

MDRC and DIR team members monitored survey response rates and at strategic points in time 

(for example, when response rates had dropped for several weeks in a row) increased the value 
209

of the gift card. About 48 percent of respondents received a Visa gift card for $25 or $40, 35 

percent received either a $50 or $60 gift card, and 17 percent received a $100 gift card. In 

addition, unconditional $15 gift cards were included with the abovementioned marketing 

materials for approximately 40 percent of the fielding sample in an effort to help improve

response rates.

Field locators. DIR employed and trained a group of field locators and assigned them 

to personally contact study participants who had not yet completed an interview. Field locators 

set up appointments for interviews with DIR’s call center but did not interview respondents on 

site. DIR monitored the success rates of each field locator weekly. In some instances, locators 

were reassigned to different WorkAdvance locations to help boost response rates. DIR also 

replaced locators who were not successful in contacting study participants.

Monitoring responses. During the fielding period, DIR prepared and shared with

MDRC weekly reports on survey response rates, organized by WorkAdvance site and research 

group. Members of the DIR and MDRC teams reviewed these reports weekly and made 

adjustments to fielding efforts within WorkAdvance sites (for example, allocation of field 

locators) in response to identification of low response rates or relatively large differences in 

response rates by research group.

Madison Towards

Per Scholas

WA C Total

Fielded sample (#) 349 341 690

St. Nicks Alliance

WA C Total

242 237 479

Strategies Group

WA C Total

353 344 697

Employment

WA C Total

349 349 698

Respondent

sample (#) 287 265 552 205 179 384 297 263 560 286 276 562

Response rate (%) 82.2 77.7 80.0 84.7 75.5 80.2 84.1 76.5 80.3 81.9 79.1 80.5

The WorkAdvance Study

Appendix Table A.4

Year 2 Survey Response Rates, by Site

SOURCES: MDRC calculations from WorkAdvance baseline information form and responses to the Year 2 

Survey.

NOTE: WA = WorkAdvance (program) group; C = control group.
210

What Procedures Were Followed to Help Ensure Data Quality? 

Survey design. Most questions included in the survey had been used in previous surveys on employment and training, including surveys administered by DIR for previous MDRC 

studies. The survey questionnaire was reviewed by a team that included experts in survey 

design and researchers in training initiatives.

CATI system testing. Before fielding the survey, members of MDRC’s research team 

and corporate Survey Unit and the DIR survey team tested the programming of DIR’s CATI 

system. Team members completed several rounds of review of technical issues in the CATI 

system and modified the questionnaire and CATI system accordingly. 

Monitoring interviewer training. Before fielding the survey, members of MDRC’s 

research team and corporate Survey Unit reviewed DIR’s interviewer training materials and 

scripts and attended a training session for interviewers.

Identifying survey respondents. MDRC shared with DIR respondents’ dates of birth 

and the last four digits of their Social Security numbers. At the start of each interview, respondents were asked to provide this information to the interviewer to verify their identity. No proxy 

interviews by third parties were allowed.

Interviewing respondents. All interviews were conducted by phone with interviewers 

at DIR’s call center. Field locators did not interview respondents.

DIR review of survey responses. DIR recorded all survey interviews and made the recordings available for review. For a sample of respondents, DIR supervisors monitored interviews in real time. In addition, DIR conducted verification calls on 10 percent of each CATI 

interviewer’s completed interviews and on 10 percent of each field locator’s completed contacts 

that led to an interview. Finally, for a sample of respondents, DIR supervisors reviewed every 

recorded response and compared the response with the value recorded in the CATI system.11

MDRC review of survey responses. During the first months of survey fielding, DIR

transmitted to MDRC two test files of survey responses. Members of MDRC’s research team 

processed the data in these test files and ran quality control checks on the data. Team members 

also checked item response rates, skip patterns, missing responses, outlier responses, and 

responses that appeared to be inconsistent with responses to related questions, with the audio 

 11Due to anomalous data values that were discovered during their routine quality checks, the survey firm 

launched an investigation to review all attempted and completed interviews that were conducted within the 

timeframe that these anomalous values were generated. They verified that records for three participants were 

affected and dropped these records from survey data files that were submitted to MDRC for processing and 

analysis.
211

recording, or both. MDRC performed quality control checks on individual items and composite 

measures (for example, total weekly earnings at interview). MDRC shared examples of problem 

responses with DIR. DIR team members then reviewed each issue and, where necessary, 

reviewed the recorded verbal response to determine whether the interviewer had entered the 

response incorrectly in DIR’s CATI system. MDRC and DIR repeated these procedures 

following secure transmission to MDRC of each survey file. DIR corrected responses as needed 

based on these reviews.

To What Extent Are Survey Respondents Representative of the 

Full Sample?

The WorkAdvance Year 2 Survey provided data on research sample members’ participation in services, employment, and various measures of overall well-being. Survey interviews 

were conducted between 18 and 30 months after each individual entered the study. The survey 

nonresponse analysis presented in this appendix examines whether the survey-based outcomes 

presented in the main report (which cover only Year 2 Survey respondents) can be generalized 

to the full research sample. Findings are presented for each site individually. Five main analyses 

were performed:

Comparison of respondents and nonrespondents. Because the full WorkAdvance 

sample was contacted for the survey, respondents are expected to have characteristics similar to 

nonrespondents.12 A logistic regression was run to test whether key baseline characteristics 

could predict whether an individual was a respondent to the Year 2 Survey at each site. The 

models included several characteristics that were regressed on a dichotomous survey response 

indicator (1 = survey respondent; 0 = nonrespondent).13 If this model is statistically significant it 

indicates that survey respondents have different background characteristics from nonrespondents. This would not imply that the causal effects of WorkAdvance are biased. It would simply 

imply that those causal effects are measured for a slightly different sample than was used for the 

administrative records analysis.

Comparison of WorkAdvance and control group respondents. Because of random 

assignment, WorkAdvance group members are expected to be similar to control group members, and thus respondents should also be similar across research groups. A logistic regression 

analysis was conducted to test whether key baseline characteristics could predict whether a 

 12For the purpose of the nonresponse analyses, nonrespondents are defined as sample members who did 

not respond to the survey between 18 and 30 months of study entry. This includes sample members who did 

not respond at all to the Year 2 Survey. 13The baseline characteristics used in the logistic regression models differed somewhat by site. The full list 

of baseline characteristics for each site can be found in the tables. See Appendix Table A.5 for a list of baseline 

characteristics that were used in each site’s regression model.
212

respondent to the Year 2 Survey was in the WorkAdvance group. The model included several 

characteristics that were regressed on a dichotomous WorkAdvance group indicator (1 = 

WorkAdvance group; 0 = control group). If this model is statistically significant the implications are potentially more serious. It would imply that differential attrition, due to the survey 

response process, has (to some extent) undermined the comparability of the research groups and 

thereby the internal validity of the experiment. Such results would require further exploration to 

measure how large any differences between the groups are and how correlated those characteristics are with the key outcomes of the experiment.

Comparison of employment and earnings outcomes, through the first half of Year 

2, between the research sample and the respondent sample. Individuals across the research 

and respondent samples are expected to show similar levels of and impacts on UI-covered 

employment and earnings. This test examines whether that is the case. 

Classification tree analysis. In order to assess for nonresponse bias more deeply, a 

classification and decision tree analysis (a form of data mining) was used to check whether any 

specific group or groups of study participants were more or less likely to have responded to the 

Year 2 Survey.

Multiple imputation. Missing data can be especially troublesome and can lead to bias 

when reporting and interpreting outcomes. Multiple imputation was used as a final check of the 

implications of survey nonresponse on the validity of the survey results. This approach, recommended as a best practice by the Institute of Education Sciences,14 involves simulating what the 

survey results would have looked like had the survey response rate been 100 percent. Multiple 

imputation was used as a sensitivity check of the Year 2 Survey results. Missing values were 

imputed for a range of survey-based measures to check for bias due to both survey and item

nonresponse. The main check here determines whether the findings from the imputed survey 

results look similar to the pattern of findings from the actual survey results. If so, this would 

provide strong evidence of the validity of the survey findings.

Below is a summary of findings from the five-part analysis for each of the four WorkAdvance sites. Overall, no serious nonresponse bias issues were found among the respondent 

sample, so no adjustments for nonresponse were imposed.

Per Scholas

Overall, there is no serious nonresponse bias among the sample enrolled at Per Scholas.

 14Puma, Olsen, Bell, and Price (2009). 
213

• Appendix Table A.5 shows that survey respondents at Per Scholas are similar to nonrespondents across the measured baseline characteristics; the logistic regression model is not statistically significant (p-value = 0.194). Nevertheless, the results do show that Per Scholas respondents are more likely 

than nonrespondents to be black, less likely to have received UI benefits before study entry, and especially more likely to have been randomly assigned 

after 2011.

• The statistically insignificant results (p-value = 0.146) of the logistic regression model in Appendix Table A.6 show that WorkAdvance group respondents at Per Scholas are similar, on average, to control group respondents.

• Appendix Table A.7 shows that the research group means and impacts on UI

employment and earnings measures for the research and respondent samples 

at Per Scholas are fairly consistent. For two of three measures, the impacts 

are somewhat larger for the respondent sample compared with the research 

sample, but the estimated impacts are statistically insignificant for both the 

research and respondent samples on all three measures. Using the “ever employed” outcome as an example, there is an impact of approximately 4 percentage points among the research sample and an impact of approximately 4 

percentage points among the respondent sample. Note these impacts are statistically insignificant for both samples.

• The classification tree analysis did not reveal any noteworthy groups of Per 

Scholas sample members who were more or less likely to respond to the 

Year 2 Survey (Appendix Figure A.1). This is consistent with the abovementioned logit model results.

• In Appendix Table A.8, multiple imputation was used to assign values to 

measures with missing values due to both survey and item nonresponse. The 

multiply imputed impacts are consistent with the impacts shown in the main 

report for Per Scholas. For example, Table 5.1 shows a statistically significant 10 percentage point impact on current employment at this site; in Appendix Table A.8, the multiply imputed 13 percentage point impact on this 

outcome is also statistically significant.

St. Nicks Alliance

Overall, while there are some differences between respondents and nonrespondents, 

there is no serious nonresponse bias among the sample enrolled at St. Nicks Alliance.
 The WorkAdvance Study 

Appendix Table A.5

Estimated Regression Coefficients for the Probability of Being a Respondent to the WorkAdvance Year 2 Survey, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Characteristic Parameter Estimate Parameter Estimate Parameter Estimate Parameter Estimate

WorkAdvance group 0.258 0.753 *** 0.494 ** 0.242

Female -0.287 0.488 -0.180 0.545 *

Black 0.431 * 0.871 ** 0.186 -0.208

Hispanic 0.330 0.304 0.052 -0.688

Age -0.008 -0.006 0.009 -0.012

Under age 24 0.019 0.406 0.386 -0.497

Number of children living with respondent 0.032 0.142 0.171 * 0.005

Born in state -0.010 1.385 *** 0.398 * -0.768 **

Highest level of education -0.008 NA NA NA

Highest level of education is GED or less NA -0.167 NA NA

Highest level of education is 2- or 4-year degree or more NA NA 0.907 ** NA

Highest level of education is high school, GED, or less NA NA NA -0.208

Already has a license/certificate in targeted sector 0.023 -0.833 -0.710 ** 0.107

Currently employed at baseline -0.272 0.362 -0.195 0.565 *

Monthly family income 0.000 0.000 0.000 0.000

Employed in at least 1 quarter in year prior to RA -0.338 0.230 0.169 -0.318

Employed in all 4 quarters in year prior to RA 0.070 1.068 * 0.501 -0.791 **

Number of quarters employed in 3 years prior to RA 0.082 -0.021 -0.012 0.044

Total earnings in quarter prior to RA 0.000 0.000 0.000 0.000

Total earnings in 3 years prior to RA 0.000 0.000 0.000 0.000

Received unemployment insurance benefits in 

3 years prior to RA -0.464 * 0.780 ** 0.249 NA

Randomly assigned to environmental remediation sector NA -0.260 NA NA

Randomly assigned to health care sector NA NA NA 0.155

Randomly assigned in 2011 -0.763 *** -1.445 *** -0.915 *** 0.182

Randomly assigned in 2012 -0.267 -0.457 -0.104 0.489 **

R-squared 0.041 0.152 0.088 0.058

Chi-square statistic 28.602 78.827 63.859 41.908

P-value of chi-square statistic 0.194 0.000 *** 0.000 *** 0.009 ***

Sample size (total = 2,564) 690 479 697 698

(continued)

214
Appendix Table A.5 (continued)

SOURCE: MDRC calculations from the WorkAdvance baseline information form and unemployment insurance administrative records from New 

York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission.

NOTES: NA = not applicable; GED = General Education Development certificate; RA = random assignment.

A chi-square test was applied to differences between outcomes for the early and late cohorts in the analysis for the full research sample. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

215
 The WorkAdvance Study 

Appendix Table A.6

Estimated Regression Coefficients for the Probability of Being a WorkAdvance

Group Respondent to the WorkAdvance Year 2 Survey, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Characteristic Parameter Estimate Parameter Estimate Parameter Estimate Parameter Estimate

Female -0.239 0.126 0.371 -0.057

Black 0.218 -0.300 -0.013 0.403 *

Hispanic 0.157 -0.417 -0.757 * -0.112

Age -0.017 0.029 ** -0.003 -0.003

Under age 24 0.283 0.483 0.288 -0.160

Number of children living with respondent -0.002 -0.119 0.104 0.060

Born in state 0.089 -0.175 -0.235 0.054 a Highest level of education 0.097 0.119 -0.095 0.175

Already has a license/certificate in targeted industry -0.294 0.917 0.206 0.480 **

Currently employed at baseline -0.661 ** -0.080 -0.078 -0.232

Monthly family income 0.000 0.000 0.000 0.000

Employed in at least 1 quarter in year prior to RA -0.175 0.146 0.274 -0.066

Employed in all 4 quarters in year prior to RA 0.027 0.001 0.330 0.559 *

Number of quarters employed in 3 years prior to RA -0.025 0.003 -0.036 -0.061

Total earnings in quarter prior to RA 0.000 0.000 0.000 0.000

Total earnings in 3 years prior to RA 0.000 0.000 0.000 0.000 *

Received unemployment insurance benefits in 

3 years prior to RA 0.508 ** -0.332 0.335 * NA

Randomly assigned in 2011 0.205 0.594 0.121 -0.015

Randomly assigned in 2012 0.005 -0.002 -0.047 0.264

Randomly assigned to environmental remediation sector NA -0.056 NA NA

Randomly assigned to health care sector NA NA NA -0.151

R-squared 0.049 0.047 0.043 0.046

Chi-square statistic 27.813 18.374 24.525 26.738

P-value of chi-square statistic 0.146 0.826 0.489 0.221

Sample size 552 384 560 562

(continued)

216
217

• Appendix Table A.5 shows that survey respondents at St. Nicks Alliance differ from nonrespondents across the measured baseline characteristics; the logistic regression model is statistically significant (p-value < 0.001). One of 

the key drivers of the differences between respondents and nonrespondents is 

that respondents were more likely to be WorkAdvance group members and 

less likely to have been randomly assigned in 2011. This is consistent with 

DIR’s early challenges in completing survey interviews with control group 

members for this site.

• The average WorkAdvance group respondent at St. Nicks Alliance is not different from the average control group respondent (shown in Appendix Table 

A.6). Overall, the logistic regression model is not statistically significant (pvalue = 0.826).

• Appendix Table A.7 shows that the research group means and impacts on UI

employment and earnings measures for the research and respondent samples 

at St. Nicks Alliance are fairly consistent. The impacts are somewhat larger 

for the research sample compared with the respondent sample, but none of 

the estimated impacts are statistically significant for either sample. Using the 

“ever employed” outcome as an example, there is a 5 percentage point impact among the research sample but only a 1 percentage point impact among 

the respondent sample, but these impacts are statistically insignificant for 

both samples.

• The classification tree analysis confirmed that at St. Nicks Alliance, control 

group members who were randomly assigned in 2011 were less likely to respond to the survey (Appendix Figure A.1). This is consistent with the very 

large program-control group differentials for the first few cohorts of survey 

respondents.

Appendix Table A.6 (continued)

SOURCES: MDRC calculations from the WorkAdvance baseline information form and from 

unemployment insurance wage records from the New York State Department of Labor, Ohio 

Department of Jobs and Family Services, and Oklahoma Employment Security Commission.

NOTES: RA = random assignment; NA = not applicable.

A chi-square test was applied to differences between outcomes for the WorkAdvance and control 

groups in the analysis for the full research sample. Statistical significance levels are indicated as follows: 

*** = 1 percent; ** = 5 percent; * = 10 percent. a

Highest level of education measures used in the regression model varied by site. 
218

 The WorkAdvance Study 

Appendix Table A.7

Quarters 2 to 7 Impacts on Employment and Earnings for the Full and

Survey Respondent Samples, by Site

Outcomes

WorkAdvance Group

Average N

Control Group

Average N

Difference

(Impact)

Percentage

Difference P-Value

Per Scholas

Ever employed (%)

Research sample

Respondent sample

Average quarterly employment (%)

Research sample

Respondent sample

Total earnings ($)

Research sample

Respondent sample

St. Nicks Alliance

Ever employed (%)

Research sample

Respondent sample

Average quarterly employment (%)

Research sample

Respondent sample

Total earnings ($)

Research sample

Respondent sample

Madison Strategies Group

Ever employed (%)

Research sample

Respondent sample

Average quarterly employment (%)

Research sample

Respondent sample

Total earnings ($)

Research sample

Respondent sample

83.9

85.3

57.8

59.1

17,144

17,919

83.7

85.0

55.2

58.0

16,348

17,543

91.4

93.4

71.6

74.6

21,602

23,467

349

288

349

288

349

288

242

205

242

205

242

205

353

297

353

297

353

297

80.4

81.0

54.6

56.2

15,595

15,983

78.7

83.7

53.0

56.5

16,437

17,621

89.4

90.4

69.2

71.8

20,461

21,301

341

266

341

266

341

266

237

179

237

179

237

179

344

263

344

263

344

263

3.5

4.3

3.2

2.9

1,549

1,936

5.0

1.3

2.1

1.5

-89

-79

2.0

3.0

2.4

2.8

1,141

2,166

4.3

5.2

5.8

5.2

9.9

12.1

6.3

1.6

4.0

2.7

-0.5

-0.4

2.3

3.3

3.5

3.9

5.6

10.2

0.221

0.170

0.235

0.314

0.214

0.171

0.158

0.717

0.502

0.655

0.955

0.966

0.355

0.185

0.325

0.285

0.364

0.122

(continued)
219

• The multiply imputed impacts in Appendix Table A.8 are consistent with the 

impacts shown in the main report for St. Nicks Alliance. For example, Table 

5.1 shows an insignificant, negative 1 percentage point impact on ever being 

employed at this site; in Table A.8, the multiply imputed negative 2 percentage point impact on this outcome is also statistically insignificant.

Madison Strategies Group

Overall, while there are some differences between respondents and nonrespondents, 

there is no serious nonresponse bias among the sample enrolled at Madison Strategies Group.

• Appendix Table A.5 shows that survey respondents at Madison Strategies 

Group differ from nonrespondents across the measured baseline characteristics; the logistic regression model is statistically significant (p-value < 0.001). 

Here, too, one of the key drivers of the differences between respondents and 

nonrespondent is the study entry year, with responses being less likely 

among those randomly assigned in 2011.

Appendix Table A.7 (continued)

WorkAdvance Group Control Group Difference Percentage

Outcomes Average N Average N (Impact) Difference P-Value

Towards Employment

Ever employed (%)

Research sample 87.6 349 83.1 349 4.5 * 5.4 0.080

Respondent sample 87.0 287 81.6 280 5.4 * 6.6 0.066

Average quarterly employment (%)

Research sample 63.4 349 62.0 349 1.4 2.2 0.572

Respondent sample 63.3 287 60.2 280 3.1 5.2 0.251

Total earnings ($)

Research sample 15,867 349 15,153 349 714 4.7 0.452

Respondent sample 16,010 287 14,106 280 1,903 * 13.5 0.068

SOURCES: MDRC calculations from the WorkAdvance baseline information form and from unemployment 

insurance administrative records provided by New York State Department of Labor, Ohio Department of Jobs and 

Family Services, and Oklahoma Employment Security Commission.

NOTES: N = number (sample size).

A two-tailed t-test was applied to the difference between outcomes for the program and control groups in the 

analysis for the full research sample. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 

5 percent; * = 10 percent.
220

The WorkAdvance Study 

Appendix Figure A.1 

Classification and Regression Tree (CART) Results, by Site

Per Scholas

 Node ID: 1

 

 

 

 0: 19.71%

 1: 80.29%

 Count: 690

St. Nicks Alliance

 Node ID: 1

 

 

 

 0: 20.04%

 1: 79.96%

 Count: 479

1/0 RAd in 2011

Madison Str

 Node ID: 

 

 

 

ategies Group

1

 0: 19.23%

 1: 80.77%

 Count: 697

Towards Employment

 Node ID: 

 

 

 

1

 0: 18.77%

 1: 81.23%

 Count: 698

0 or Missing

 

 

 

 

Node ID: 2

 0: 17.37%

 1: 82.63%

 Count: 426

 Node ID: 

 

 

 

1

3

 0: 41.51%

 1: 58.49%

 Count: 53

BIF: 1/0 Program group

0 or Missing

 Node ID: 

 

 

 

7

 0: 59.26%

 1: 40.74%

 Count: 27

 Node ID: 

 

 

 

1

6

 0: 23.08%

 1: 76.92%

 Count: 26

SOURCES: MDRC calculations from the WorkAdvance baseline information form, survey administration data from 

Decision Information Resources, and unemployment insurance administrative records from New York State Department

of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission.

NOTES: RAd = randomly assigned; BIF = baseline information form. 

 Percentages in the “1” category represent all survey respondents, including a few sample members who 

responded outside the 18- to 30-month follow-up window that was used for the Year 2 Survey analysis. As such, 

these percentages may differ from response rates given in Appendix Table A.4.

 The CART technique was used to predict the probability of survey interview response by classifying sample 

members based on their employment and wage history, their baseline characteristics, the number of attempts made 

by the survey firm to reach the sample member for the interview, and the mode in which the interview was 

completed. This classification process is performed separately for each site. 

 The gray shading of boxes denotes a greater proportion of survey respondents in the indicated group. The darker 

the shading, the larger the proportion of survey respondents in that group.

 For Per Scholas, Madison Strategies Group, and Towards Employment, sample members’ baseline and surveyrelated characteristics did not predict likelihood of survey response. This is depicted in the above illustration by no 

“tree” growth. At St. Nicks Alliance, two characteristics were identified as predictive of survey response: year of 

random assignment and research group. The tree growth for this site is interpreted as follows: (1) sample members 

randomly assigned after 2011 were more likely to respond to the survey than those randomly assigned in 2011, and 

(2) program group members who were randomly assigned in 2011 were more likely to respond to the survey than 

control group members who were randomly assigned in the same year. 
The WorkAdvance Study

Appendix Table A.8

Multiply Imputed Year 2 Impacts on Employment, Earnings, and Income, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Difference

Outcome WA WA C (Impact)

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Ever employed (%)

Currently employed

Currently employed

in targeted sector (%)

Average number of hours worked per 

week at current or most recent job

Not employed since RA (%)

Employed part-timea Employed full-time

Average hourly wage at current or

most recent job ($)

Percentage of months employed

Offered opportunities for career 

advancement (%)

Household income in prior month ($)

Job earnings (%)

Unemployment insurance (%)

94.5

74.8

46.9

34.1

4.6

31.7

63.7

15.28

56.9

67.3

2,129

79.8

4.1

88.5

62.0

15.3

32.5

8.9

34.5

56.5

12.84

50.7

49.1

1,650

65.7

7.9

6.0 ***

12.7 ***

31.6 ***

1.6

-4.4 *

-2.8

7.2

2.44 ***

6.2 **

18.1 ***

479 ***

14.1 ***

-3.8

89.1

64.7

23.4

36.2

9.3

22.7

68.0

13.13

51.7

59.0

2,018

69.1

6.9

90.6

67.1

14.2

34.5

7.1

30.8

62.1

13.16

51.2

58.1

2,319

72.5

3.3

-1.5

-2.3

9.2 *

1.7

2.2

-8.1 *

5.9

-0.03

0.5

0.9

-301

-3.4

3.6

95.7

75.5

54.7

41.8

3.4

17.3

79.3

12.48

65.2

64.7

2,053

79.6

2.4

92.7

70.6

37.4

39.6

4.4

22.7

72.9

11.81

65.7

55.7

2,106

76.8

2.5

3.0

5.0

17.3 ***

2.1 *

-1.1

-5.3 *

6.4 *

0.66

-0.5

8.9 **

-53

2.8

-0.1

89.7

65.2

37.4

33.2

9.3

32.7

58.0

10.18

52.9

58.2

1,446

71.0

3.5

88.1

61.8

24.0

31.6

10.6

38.0

51.4

10.29

58.4

46.9

1,557

64.1

3.9

1.6

3.5

13.4 ***

1.5

-1.3

-5.3

6.6

-0.11

-5.5 *

11.3 ***

-111

6.9

-0.4

Sample size (total = 2,564) 349 341 242 237 353 344 349 349

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group; RA = random assignment.

Multiple imputation was used to assign values to measures with missing values due to both survey and item nonresponse. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Sample sizes may vary because of missing values.

a

Full-time employment is defined as working 35 or more hours in a week.

221
222

• The statistically insignificant results (p-value = 0.489) of the logistic regression model in Appendix Table A.6 show that WorkAdvance group respondents at Madison Strategies Group are similar, on average, to control group respondents.

• Appendix Table A.7 shows that the research group means and impacts on UI

employment and earnings measures for the research and respondent samples 

at Madison Strategies Group are fairly consistent. The earnings impacts are 

somewhat larger for the respondent sample (at $2,166) compared with the research sample (at $1,141), but these estimated differences are not statistically 

significant for either sample.

• The classification tree analysis did not reveal any noteworthy groups of Madison Strategies Group sample members who were more or less likely to respond to the Year 2 Survey (Appendix Figure A.1). This is consistent with 

the results above.

• The multiply imputed impacts in Appendix Table A.8 are consistent with the 

impacts shown in the main report for Madison Strategies Group. For example, Table 5.1 shows an insignificant 2 percentage point impact on ever being 

employed at this site among the respondent sample; this is similar to the multiply imputed and insignificant 3 percentage point impact shown in Table A.8

for the research sample.

Towards Employment

• Appendix Table A.5 shows that survey respondents at Towards Employment 

differ from nonrespondents across the measured baseline characteristics; the 

logistic regression model is statistically significant (p-value = 0.009). Some 

of the key drivers of the differences between respondents and nonrespondents include gender, birth state, employment history and status at study entry, 

and year of study entry.

• The average WorkAdvance group respondent at Towards Employment is not 

different from the average control group respondent (shown in Appendix Table A.6). Overall, the logistic regression model is not statistically significant 

(p-value = 0.221).

• Appendix Table A.7 shows that the research group means and impacts on UI

employment and earnings measures for the research and respondent samples 

at Towards Employment are fairly consistent. Still, the earnings impacts (at 
223

$1,903) are somewhat elevated and statistically significant for the respondent 

sample compared with the research sample (at $714), suggesting some bias 

due to survey nonresponse.

• The classification tree analysis did not reveal any noteworthy groups of Towards Employment sample members who were more or less likely to respond to the Year 2 Survey (Appendix Figure A.1). This is consistent with 

the abovementioned logit model results.

• The multiply imputed impacts in Appendix Table A.8 are consistent with the 

impacts shown in the main report for Towards Employment. For example, 

Table 5.1 shows an insignificant 2 percentage point impact on ever being 

employed at this site among the respondent sample; this is similar to the multiply imputed and insignificant 2 percentage point impact shown in Table A.8 

for the research sample.

Cost Analysis Data and Findings

Selection of Year 2 as the Steady-State Period

As stated in Chapter 4, Year 2 of the WorkAdvance program was selected as the 

steady-state period for the cost analysis. Year 3 might appear at first to be a better choice, being

even less subject to the inclusion of start-up costs than Year 2; moreover, by Year 3 postemployment services had been given more opportunity to mature. But random assignment ended 

for WorkAdvance at the end of June 2013, only three months after Year 3 had begun. While 

new participants were enrolled into the program after random assignment ended, there was a 

gap of four to eight months (depending on the site) between the end of random assignment and 

the start of new enrollment into WorkAdvance.

15 As a consequence, Year 3 is not sufficiently 

representative of a steady state or even of normal program operations. Thus, in reporting the 

cost findings, costs for Year 2 are given greater prominence than those for Year 3.

Still, Year 2 falls short of accurately reflecting the steady state because the entry of new 

participants into WorkAdvance during Year 1 was slower than program entry during Year 2 — 

considerably slower in some sites. Moreover, the entry period in Year 1 varied by site and did 

not begin until well after April 2011 at any of them. As a result, in Year 2 there were fewer 

 15While all sites suspended enrollment into WorkAdvance during this gap, they continued to run other 

programs. Per Scholas was unique among the sites in that it continued to enroll new participants for training 

and services resembling WorkAdvance. However, those activities did not fall under the umbrella of WorkAdvance.
224

persons far enough along in their two-year period of eligibility to have graduated from occupational skills training than there would have been in a true steady state, and, as a consequence, 

the Year 2 cost of services received after graduating is probably understated relative to costs 

incurred earlier. (During Year 3, in contrast, because of the long break in enrollment there were 

fewer participants who had not yet graduated from occupational skills training than there would 

have been in the steady state, so pregraduation costs incurred that year are probably slightly 

understated relative to costs incurred after graduation.) Based on an assessment of costs tied to 

postemployment services at different stages of the program, MDRC estimates that gross costs in 

Year 2 were probably understated on the order of 1 percent to 2 percent at the different sites. 

The sensitivity of the cost findings to Year 2 conditions was examined, and postemployment 

costs from Year 3 are substituted for those of Year 2 to account for the Year 2 understatement 

of such costs.

Sources of Cost Data

Financial reports detailing expenditures, when submitted to the Mayor’s Fund to Advance New York City, included a certification statement and were substantiated through 

additional fiscal oversight. They were submitted using an online grants management system 

(GMS) that included the budgets for each program year, and expenditures in each report were 

listed as total amounts spent against a line item during the designated reporting period. Reported 

information was exported from the GMS to Excel for further analysis.

For purposes of determining net costs, specific survey questions were selected as key 

indicators of the receipt of preemployment services and support, occupational skills training, 

and postemployment retention and advancement services by control group members. These 

questions were used to isolate costs that would have existed in the absence of WorkAdvance. 

Costs associated with the activities mentioned above were established for each of the three 

relevant cities based on financial data collected and made public by the U.S. Department of 

Education16 and the U.S. Department of Labor,17 and using estimates developed by MDRC for 

services not captured in the available data.

 16The National Center for Education Statistics, housed within the Institute of Education Sciences, publishes core expenses per full-time equivalent enrollment at public and private for-profit and not-for-profit educational institutions, derived from data that each institution is required to report to the Department of Education 

as part of their participation in federal student financial aid programs. Expense data for fiscal year 2013 was 

used, drawn from the Integrated Postsecondary Education Data System, which is available online. 17Employment and Training Administration Employment Service (Wagner-Peyser) Program Year 2013 

final allotments are available online.
225

Variation in the Gross Cost Estimates

The two New York providers, St. Nicks Alliance and Per Scholas, are at the higher end 

of the narrow range of total per participant costs in Year 2, with Madison Strategies Group and 

Towards Employment at the lower end. There are at least three possible explanations for this. 

The first is simply that salaries and other operating costs tend to be higher in New York City 

than in Tulsa or Cleveland. A second potential explanation is tied to the dual track operated 

initially by Madison Strategies Group and Towards Employment, in which, as discussed in the 

main text of the report, some WorkAdvance enrollees were placed directly into jobs while 

others received the full suite of training and supportive services. While the percentage of 

individuals at each of the two dual-track sites given training from the start was significantly 

larger than the share directed into jobs, fewer resources were expended on those placed directly 

into jobs, and the difference is substantial enough to have an effect on costs. Consequently, 

when aggregate costs are divided by the number of WorkAdvance participants at each site, the 

larger numbers of individuals receiving less costly service at Madison Strategies Group and 

Towards Employment may be bringing down the gross costs per participant at those two sites.

A third possibility arises from the fact that St. Nicks Alliance and Per Scholas had fewer participants than Madison Strategies Group or Towards Employment. During Year 2, for 

example, St. Nicks Alliance had only 68 percent as many entrants as either Madison Strategies 

Group or Towards Employment, yet its aggregate gross costs in Year 2 were 81 percent as large 

as either of those two sites. Similarly, but less dramatically, Per Scholas had only about 80 

percent as many Year 2 entrants as Madison Strategies Group or Towards Employment, but had 

aggregate Year 2 gross costs that were 88 percent as large. Thus, the numbers of participants at 

St. Nicks Alliance and Per Scholas were smaller relative to aggregate costs than they were at 

Madison Strategies Group and Towards Employment, suggesting that economies of scale might 

have existed in the operation of WorkAdvance. For example, the number of participants 

assigned to each of the job developers and career coaches at St. Nicks Alliance and Per Scholas 

may have been smaller than at the other two sites even though each had the capacity to serve 

more.

Unemployment Insurance Wage and Benefits Data

Data Collection

MDRC negotiated separate data sharing agreements with three state agencies (the New 

York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma 

Employment Security Commission) to facilitate the collection of UI wage and benefits data to 
226

measure WorkAdvance’s effect on employment and earnings. These data were collected every 

6 to 12 months.18

Each shipment included the most recent quarter of data available and at least five quarters of historical data. This allowed for an “overlap” across data files, and records were checked 

for any updated information from one file to the next.

Procedures used to ensure data quality include:

• Checks for duplicate records across and within data files.

o Duplicates by person, quarter, and employer: One record was dropped 

and the other record was updated to report the two records’ average earnings amount.

o Duplicates by person, quarter, and earnings amount: If the two records 

were thought to be from the same employer (that is, the employer ID had 

been updated from one file to the next), the record from the earlier file 

was dropped.

• Checks for outliers in quarterly earnings and benefits amounts. (More information on how outliers were handled can be found in the next section.)

• Checks for incomplete quarters of data.

• Checks of average quarterly earnings and benefits amounts and average quarterly employment by calendar quarter and relative quarter.

o Where data were available, checks of average quarterly earnings and average quarterly employment within the targeted sector(s) were performed 

as well.19

Estimating Effects of WorkAdvance

Before estimating the effects of the WorkAdvance programs, MDRC conducted an analysis 

planning process. All the analytic decisions discussed in this section — including sample 

definitions, outcome measure and subgroup definitions, statistical procedures, and other data 

analysis issues — were prespecified during that planning process. Prespecification is a key 

 18Because of the limited amount of historical UI benefits data available in Ohio, MDRC did not collect UI 

benefits data for Towards Employment sample members. 19Sector employment information was not available in New York UI records.
227

means of safeguarding a statistical study from drawing false conclusions. By prespecifying, 

analysts are limited in their ability to “search for impacts” when they may not be present.

Power Analysis

In the Subgrantee Evaluation Plan, MDRC indicated the analysis would be conducted at 

the sector level. Each WorkAdvance provider targeted only one sector, with the exception of 

Towards Employment, which targeted two sectors — health care and manufacturing. Roughly 

half that provider’s sample was randomly assigned within the health care program and half 

within the manufacturing program. In the original study design, Towards Employment partnered with another provider in northeast Ohio (Compass), which also targeted the same two 

sectors. The original intention was to pool the samples within sectors across these two sites. 

However, because the Compass sample was dropped from the study, the sample sizes within the 

health care and manufacturing sectors were diminished. A power analysis performed by MDRC 

found these sample sizes to be too small to detect any impacts at a commonly accepted level.20

For this reason, a decision was made, before impact data were available, to switch from the 

sector to the site level of analysis. This decision affected only the Ohio site, because at the other 

three sites the sector is coterminous with the site.

Appendix Table A.9 shows the minimum detectable effects (or MDEs) calculated during the power analysis. MDEs are a key measure of statistical power. Conventionally, an MDE 

is the smallest true effect that has an 80 percent chance of being statistically significant at the 10 

percent level. MDEs are commonly expressed in effect size units (specifically, in terms of 

standard deviations) to permit comparisons across outcomes with different units. This expression of an MDE is referred to as a minimum detectable effect size (or MDES). A common rule 

of thumb is to ensure studies have sufficient power to detect impacts at or below an MDES of

0.2, which is a common threshold for a “small” effect size.21

As shown in the top half of Appendix Table A.9, the MDESs for the administrative 

records sample at all four providers are between 0.167 (at Towards Employment) and 0.204 (at 

St. Nicks Alliance). These are all below or close to the 0.2 threshold. Assuming 50 percent of 

the control group was employed (that is, the standard deviation is 0.5),22 these MDESs translate 

into MDEs of between 8.4 and 10.2 for percentage measures.

 20See Cohen (1992) and Hill, Bloom, Black, and Lipsey (2007). 21The 0.2 rule of thumb comes from Cohen. Cohen defined an effect size of 0.2 as “small,” 0.5 as “medium,” and 0.8 as “large.” Lipsey, another prominent researcher, sets the threshold lower. To Lipsey, an effect 

size of 0.15 or lower is small. See Cohen (1992) and Hill, Bloom, Black, and Lipsey (2007). 22This assumption is the worst-case scenario. The point of maximum variance for a percentage measure is 

0.5 (a control group level of 50 percent). At that point, an MDES of 0.2 translates into an MDE of 10 percent-

(continued)
228

The bottom half of Appendix Table A.9 shows the MDESs and MDEs for the Year 2 

Survey sample. Because the sample sizes here are smaller (as mentioned, around 80 percent of 

the full research sample responded to the Year 2 Survey at each site), the MDESs are larger than 

those for the administrative records sample. The MDESs here range from 0.192 (at Towards 

Employment) to 0.241 (at St. Nicks Alliance). For a continuous measure, such as earnings, with 

a standard deviation of $3,500, these MDESs translate into MDEs of $672 and $844, respectively.

 

age points. The further the variance is from 0.5, the smaller the MDE. For example, if the control group level 

for a measure is 20 percent, the MDE for a study powered at 80 percent would be 8 percentage points.

The WorkAdvance Study

Appendix Table A.9

Minimum Detectable Effects, by Sample

Sample

Administrative records sample

Sample

Size R-squareda MDES

Employment (%)

SD = 0.4 SD = 0.5

MDEb

Earnings ($)

SD = 2,000 SD = 3,500

Per Scholas 690 0.183 0.171 6.8 8.6 342 599

St. Nicks Alliance 479 0.193 0.204 8.2 10.2 408 714

Madison Strategies Group 697 0.142 0.175 7.0 8.8 350 613

Towards Employment 698 0.219 0.167 6.7 8.4 334 585

Health care 351 0.231 0.233 9.3 11.7 466 816

Manufacturing 347 0.278 0.227 9.1 11.4 454 795

Year 2 Survey sample

Per Scholas 552 0.118 0.199 8.0 10.0 398 697

St. Nicks Alliance 384 0.101 0.241 9.6 12.1 482 844

Madison Strategies Group 560 0.113 0.198 7.9 9.9 396 693

Towards Employment 562 0.161 0.192 7.7 9.6 384 672

Health care 295 0.167 0.265 10.6 13.3 530 928

Manufacturing 267 0.201 0.273 10.9 13.7 546 956

SOURCE: MDRC calculations using PowerUP! tool.

NOTES: MDE = minimum detectable effect; MDES = minimum detectable effect size; SD = standard deviation. aR-squared values differ by sample and site. For the administrative records sample, R-squared values are from 

the models for unemployment insurance earnings in Quarter 10. For the Year 2 Survey sample, R-squared values 

are from the models for current employment.

bMDEs are for a two-tailed test at the 10 percent significance level with 80 percent power.
229

While Towards Employment has the lowest MDES of the sites for both the administrative records and Year 2 Survey sample, if its sample were to be split into the two targeted 

sectors, the MDES would increase to well above the 0.2 threshold. For the Year 2 Survey 

sample, the MDES is 0.265 for the health care sector sample and 0.273 for the manufacturing 

sector sample. Those MDESs translate into MDEs of 13.3 and 13.7, respectively, for percentage 

measures, assuming a 50 percent control group level. It would be more difficult to detect any 

statistically significant impacts at this level. Because of this, the research team decided not to do 

the analysis at Towards Employment at the sector level but instead to present the results at the 

site level.

Unit of Analysis

Based on the results of the power analysis described in the previous section, a decision 

was made that WorkAdvance impacts would be presented primarily at the site level. This 

decision was also based on the substantial variation in various provider-level factors. In the 

subgroup analyses, however, the decision was to pool the sites since there is not enough 

statistical power to support such analyses at the site level.

Follow-Up Period

The job training literature makes it clear that extended follow-up is necessary to capture 

the long-term impacts of training programs.23 In this report, the entire sample as a whole has 

2.25 years of follow-up from UI wage data. The WorkAdvance Year 2 Survey covers 18 

months of common follow-up for the full respondent sample. For these reasons, the impacts 

presented in this report should be viewed as “interim” in nature. Efforts are under way to collect 

longer-term follow-up data. The data acquisition contracts and informed consent forms were set 

up to enable the collection of at least five years of common follow-up from UI wage data, funds 

permitting.

Estimation Strategy

Results presented in WorkAdvance were based on “intent-to-treat” impact estimates. 

That is, the impacts were calculated by comparing all individuals in the WorkAdvance group 

with all individuals in the control group, regardless of whether or how long individuals were 

engaged in WorkAdvance services. The impact estimates were regression adjusted using 

background characteristics of the sample, including prior earnings, age, race or ethnicity, 

education, household characteristics, and year of random assignment. (Covariate selection is 

described in more detail below.) 

 23See Card, Kluve, and Weber (2010). 
230

For impacts on binary (0/1) and continuous outcomes, such as earnings during a period,

MDRC used estimated regression models of the following form, using ordinary least squares:24

Yi = α + βPi + δXi + εi

where:

YYii = outcome measure for sample member i; 

PPii = indicator of program group membership for sample member i (1 = program 

group member; 0 = control group member);

Xi = a set of background characteristics for sample member i; 

εεii = random error for sample member i; 

ββ = the estimate of the impact of the program on the average value of the outcome;

α = the intercept of the regression; and

δ = the set of regression coefficients for the background characteristics.

Regression adjustment can increase the statistical precision of the estimated effects, 

helping to distinguish normal variation in outcomes from the effects of WorkAdvance group 

members’ sole access to the WorkAdvance training and employment services.

Confidence Intervals, P-Values, Standard Errors, and Effect Sizes of 

Program Impacts

Appendix Table A.10 displays more detailed statistical data on MDRC’s impact estimates of key employment and earnings outcomes by site. These details are included to provide 

more information on the uncertainty associated with specific impact estimates; they may be 

useful to meta-analysts who are interested in including the WorkAdvance findings. For each 

measure, the first two columns of each site panel show the lower and upper bounds of 90 

percent confidence intervals around the average impact estimate. The third column shows the 

level of statistical significance (p-value), using the results shown in the tables in Chapter 5, and 

the fourth column shows the standard error for each measure. The rightmost column in each site 

 24As sensitivity tests for several previous studies, MDRC estimated outcomes and impacts on binary (0/1) 

measures using ordinary least squares regression and again using logistic regression. MDRC found that the 

results were nearly identical.
The WorkAdvance Study

Appendix Table A.10

Confidence Intervals, P-Values, Standard Errors, and Effect Sizes for Key Impacts, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome

90 Percent

Confidence PInterval ValueEffect

SE Size

90 Percent

Confidence P- Effect

Interval Value SE Size

90 Percent

Confidence P- Effect

Interval Value SE Size

90 Percent

Confidence P- Effect

Interval Value SE Size

Among Year 2 Survey respondents

All jobs (%)

Ever employed 0.7 8.2 0.051

Currently employed 3.7 16.4 0.010

Percentage of months

employeda -0.8 8.4 0.169

Employed 6 or more

consecutive monthsa 1.0 13.9 0.058

Current or most recent job (%)

Employed in targeted sector 34.4 47.0 0.000

Earnings

Average total earnings

per week ($) 48 148 0.001

Average hourly wage ($) 1.21 3.55 0.001

Employed and hourly wage

above $12.00 (%) 9.3 23.3 0.000

Employed and hourly wage

above $15.00 (%) 9.5 22.8 0.000

Hours

Average hours worked

per week (#) -1.0 3.2 0.399

b Employed full-time (%) -1.2 12.3 0.178

Sample size (total = 2,058) 552

2.3

3.9

2.8

3.9

3.8

30

0.71

4.3

4.1

1.3

4.1

0.17

0.22

0.12

0.16

0.83

0.28

0.28

0.33

0.33

0.07

0.12

-5.8

-8.2

-5.1

-1.1

4.5

-46

-1.67

-6.2

-6.8

-0.9

-4.2

384

4.2 0.791

8.0 0.984

7.0 0.797

15.2 0.155

19.3 0.008

77 0.679

1.12 0.749

11.1 0.645

8.4 0.860

4.7 0.276

11.9 0.436

3.1

4.9

3.7

5.0

4.5

37

0.85

5.3

4.6

1.7

4.9

0.03

0.00

0.03

0.15

0.27

0.04

0.03

0.05

0.02

0.11

0.08

-1.5

-0.4

-6.7

-8.5

9.7

-16

-0.36

4.2

0.9

-0.7

-2.6

560

4.5 0.412

11.6 0.122

2.7 0.491

3.3 0.470

23.3 0.000

69 0.303

1.27 0.358

18.3 0.009

12.2 0.059

3.6 0.268

8.4 0.387

1.8

3.6

2.9

3.6

4.1

26

0.50

4.3

3.5

1.3

3.4

0.07

0.13

0.06

0.06

0.33

0.09

0.08

0.22

0.16

0.09

0.07

-2.8

-0.2

-10.0

-10.7

9.6

-18

-0.49

-4.4

-7.1

-0.9

-0.1

562

6.2 0.530

12.8 0.109

-0.1 0.096

2.2 0.279

23.2 0.000

56 0.393

1.16 0.505

7.9 0.636

1.4 0.274

3.9 0.304

13.5 0.103

2.7

3.9

3.0

3.9

4.1

23

0.50

3.7

2.6

1.4

4.1

0.05

0.13

0.13

0.09

0.33

0.07

0.06

0.04

0.09

0.09

0.14

(continued)

231
Appendix Table A.10 (continued)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome

90 Percent

Confidence PInterval ValueEffect

SE Size

90 Percent

Confidence P- Effect

Interval Value SE Size

90 Percent

Confidence P- Effect

Interval Value SE Size

90 Percent

Confidence P- Effect

Interval Value SE Size

Among full research sample

Quarter 2 to 10

Ever employed (%) -1.2 6.9 0.247 2.5

Average quarterly

employment 0.1 8.2 0.096 2.5

Earnings ($) 1,830 8,542 0.011 2,040

Year 1 (Quarters 2 to 5)

Ever employed (%) 0.8 11.4 0.058 3.2

Average quarterly

employment -2.5 6.8 0.443 2.8

Employed in all 4 quarters -8.8 1.8 0.281 3.2

Earnings ($) -1,119 1,420 0.846 772

Earned $20,000 or more -4.5 3.8 0.896 2.5

Year 2 (Quarters 6 to 9)

Ever employed (%) 1.0 11.0 0.050 3.1

Average quarterly

employment 0.8 10.6 0.058 3.0

Employed in all 4 quarters 0.9 13.1 0.062 3.7

Earnings ($) 1,780 5,714 0.002 1,196

Earned $20,000 or more 2.9 14.2 0.014 3.5

0.09

0.12

0.18

0.14

0.06

0.08

0.01

0.01

0.15

0.14

0.14

0.22

0.18

-1.1 9.3 0.198 3.2

-3.5 6.5 0.619 3.0

-4,349 3,948 0.937 2,522

1.4 14.0 0.044 3.8

-3.2 7.6 0.509 3.3

-13.1 -0.6 0.070 3.8

-1,908 1,402 0.802 1,006

-9.2 1.3 0.215 3.2

-4.5 8.6 0.600 4.0

-4.2 8.1 0.596 3.7

-1.7 12.6 0.207 4.3

-2,128 2,510 0.892 1,410

-1.7 11.3 0.224 3.9

0.12

0.04

0.01

0.18

0.06

0.16

0.02

0.11

0.05

0.05

0.11

0.01

0.10

-3.2 3.1 0.993 1.9

-1.6 6.3 0.319 2.4

-612 5,717 0.185 1,924

-1.2 6.8 0.255 2.4

-0.4 8.1 0.140 2.6

-4.7 7.3 0.719 3.6

-990 1,647 0.682 802

-2.1 7.7 0.342 3.0

-7.2 2.2 0.384 2.9

-4.0 5.6 0.789 2.9

-3.2 8.8 0.443 3.7

101 3,574 0.082 1,056

3.7 15.1 0.007 3.5

0.00

0.07

0.09

0.09

0.11

0.03

0.03

0.07

0.06

0.02

0.06

0.12

0.19

2.2 9.5 0.009 2.2

-0.8 6.9 0.189 2.3

-599 4,217 0.217 1,464

0.2 9.5 0.089 2.8

-3.7 4.7 0.838 2.6

-9.5 0.9 0.177 3.2

-997 1,022 0.984 614

-6.8 1.1 0.241 2.4

0.4 10.4 0.076 3.0

0.4 9.8 0.075 2.9

-0.8 10.6 0.161 3.5

238 3,004 0.054 841

0.6 10.5 0.066 3.0

0.19

0.09

0.08

0.12

0.01

0.09

0.00

0.08

0.13

0.12

0.10

0.13

0.13

(continued)

232
Appendix Table A.10 (continued)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome

90 Percent

Confidence PInterval ValueEffect

SE Size

90 Percent

Confidence P- Effect

Interval Value SE Size

90 Percent

Confidence P- Effect

Interval Value SE Size

90 Percent

Confidence P- Effect

Interval Value SE Size

First quarter of Year 3

Ever employed (%)

Earnings ($)

Sample size (total = 2,564)

0.3 11.8 0.084

693 1,885 0.000

690

3.5

362

0.13

0.25

-10.2

-851

479

4.3

573

0.508

0.748

4.4

433

0.06

0.03

-2.8

-110

697

9.1

883

0.383

0.200

3.6

302

0.07

0.09

-0.8

-215

698

10.5 0.156

567 0.460

3.4

238

0.10

0.05

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance administrative records from New York 

State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: SE = standard error.

This table shows the upper and lower bounds of 90 percent confidence intervals around the average impact and the p-values, standard errors, and effect sizes 

of the impact estimates. Effect sizes are shown as absolute values and were calculated by dividing the impact estimate by the standard deviation of the pooled 

sample of sample members from both research groups.

Sample sizes may vary because of missing values.

aMeasures reported in time intervals cover only the first 18 months following each sample member's month of random assignment (the common follow-up 

period).

bFull-time employment is defined as working 35 hours or more per week.

233
234

panel displays the effect sizes in absolute values. For each measure, the effect size was calculated by dividing the impact estimate by the standard deviation for the pooled sample. As discussed above, effect sizes standardize impact estimates for comparison with impact estimates 

from other studies.

For example, the table shows that the 90 percent confidence interval around the impact 

on hourly wages in Per Scholas ranges from a lower bound of $1.21 to an upper bound of $3.55 

per hour. Because both of these bounds are on the same side of zero (in other words, even the 

lower bound shows a positive increase), the estimate is statistically significant. This is also 

shown by the p-value, which indicates the probability of measuring an impact of this size or 

larger if there really is no true impact at the Per Scholas site. Because this probability is very 

low (p-value = 0.001), it can be concluded that the estimated effect is due to the true impact of 

Per Scholas rather than to chance. The standard error reflects the statistical uncertainty associated with this estimate, factoring in the sample size, the standard deviation, and the units of 

measurement. The final element, the effect size (0.28), indicates that this is a moderate-sized 

impact based on statistical literature on effect sizes.25

Assessment of Possible Effects of Multiple Comparisons

In recent years, the issue of multiple test bias has become more prominent in both the 

academic literature and the field of program evaluation more generally. The basic issue is well 

known and not new. Every time one estimates an impact on an outcome there is a precisely 

defined probability (conventionally 10 percent in such studies as WorkAdvance) of concluding

that a program has had a true impact when the observed difference is simply due to chance. 

Since researchers typically examine many outcomes, the probability that at least one estimate

will be statistically significant simply by chance can get very high.

In the past, this issue was dealt with through researcher discretion. Impact analysts 

would assess patterns of findings and were trained not to emphasize small, sporadic effects even 

if they were statistically significant. As the technology used to produce outcome estimates has 

improved, however, it has become possible to produce thousands of estimates for a given report 

and to “cut the data” in innumerable ways. In addition, there is a thriving literature about how 

some studies have abused the statistical inference process by either “cherry picking” significant 

results or overemphasizing results that are probably due to chance. (This is sometimes called 

“p-value hacking.”) So there is broad consensus that something needs to be done to discipline 

the process. There is substantial debate, however, about what should be done.

 25See Hill, Bloom, Black, and Lipsey (2007). As many authors have noted, effect sizes are domain dependent. This is a rather large effect size compared with that of other employment programs.
235

Confirmatory Measures

One approach to dealing with this problem is to conduct fewer impact estimates and to 

state in advance which tests will be conducted. This was one strategy used in WorkAdvance to 

address the multiplicity problem. Evaluations such as WorkAdvance collect rich data that can 

be useful to diverse audiences. A common means of navigating this issue is to choose a select 

group of “confirmatory” outcomes, which are featured in the report and are highly influential in 

driving the determination of program effectiveness. Other measures (sometimes called “exploratory” measures) are shown in the main report but do not have as much weight in defining 

“success.” Thus, the study can preserve the richness of the data while addressing the multiple 

comparisons problem. However, this approach requires that one think carefully about confirmatory measures well in advance of measuring impacts.

One consideration in choosing the confirmatory measures was a desire to capture the 

key goals of WorkAdvance programs: employment and earnings. Recent evidence has highlighted the importance of measuring employment and earnings with multiple data sources.26

Another consideration is that with four sites, multiple comparisons adjustments can quickly get 

very conservative. Thus only two confirmatory measures were chosen in WorkAdvance. This 

results in eight statistical tests. In order to correct the p-values for an inflated familywise error 

rate,27 the Westfall-Young correction was applied to the p-values. This correction was favored 

because of its performance with correlated outcomes.

Taking into account all these considerations, in WorkAdvance, one confirmatory outcome was chosen from the survey and one from administrative records. The measures chosen 

were participants’ current employment at the time of the Year 2 Survey interview and UI 

earnings in the first quarter of Year 3.28 The Westfall-Young adjusted p-values are shown in 

Table 5.3 and are discussed in the main report. All other measures presented in the main report 

are exploratory measures. If there were no statistically significant estimated impacts on the 

primary measures, but there were impacts on some of the exploratory measures, this would 

imply that there is some preliminary evidence of the effectiveness of Work Advance.

 26See Barnow and Greenberg (2015). 27The familywise error rate is the probability of obtaining at least one statistically significant test across 

several measures due to chance (rather than the true impact of the program). 28Initially the earnings measure chosen was earnings in Quarter 7. However, as more follow-up data became available, it was decided to stick with the concept of using the latest quarter of UI wage data as the 

confirmatory measure.
236

Categorical Measures

Categorical measures can also exhibit the multiple comparisons problem when two apparent impacts represent the same actual effect. Because categorical measures are mutually 

exclusive, it follows that if more people are in one category, then fewer people must be in 

another category. For example, using the categorical version of hourly wages, if WorkAdvance 

increased the number of people with wages at the level of more than $15 per hour, then it also 

reduced the number of people at another wage level (for example, hourly wages of $8 or less). 

Impacts may be present in both of these levels simply due to the nature of the measures.

To deal with this issue, two types of tests were run. First, the Westfall-Young correction 

was applied to the p-values for each level of categorical measures. Second, seemingly unrelated 

regression (SUR) models were run to test for differences in the distribution of categorical 

measures across research groups, and the F-test is used to test whether all program impacts are 

zero across the categories. For almost all categorical measures, the impacts were consistent 

across the two types of tests.

Outliers

To improve precision, when estimating program effects on the key continuous measures 

in the report (those that would be most affected by outliers, such as dollar-value measures), 

extreme values were identified as outliers, and for some measures, outliers were recoded to the 

value at the 99th percentile (for example, in the case of hourly wages and weekly earnings from 

the survey) or set to zero (for example, in the case of UI earnings and benefits). Appendix Table

A.11 compares the impacts on income, wages, debt, and earnings measures by the level of 

exclusion of outlier values. The first set of columns shows the impacts with all values included. 

The second set of columns shows the impacts with extreme outliers removed (these are the 

impacts presented in the main body of the report). Not all measures shown in the table had 

extreme outliers, and for those measures, the impacts are the same in the first two sets of 

columns. (The details on changes made to outlier values are included in the footnotes of the 

table.) The third set of columns shows the impacts after excluding the top 1 percent of values.

The table shows that the findings in this report are robust to outliers. If anything, the 

impacts are slightly larger after outliers are excluded. For example, the impact on Year 1 

earnings is –$39 for all responses, $11 once extreme outliers are removed,29 and $299 after 

excluding the top 1 percent of values.

 29UI earnings above $50,000 per quarter were set to $0.
237

The WorkAdvance Study

Appendix Table A.11

Comparison of Impacts on Income, Wages, Hours, Debt, and Earnings, by Level of Exclusion of Outlier Values

Outcome N

All Responses Extreme Outliers Removeda Top 1 Percent Excluded

SE WA C

Difference

(Impact) N

Difference

SE WA C (Impact) N

Difference

SE WA C (Impact)

Year 2 Survey

Respondent income in

prior month ($)

Household income in

prior month ($)

Weekly earningsb ($)

Hourly wagec

 ($)

Hours worked per week (#)

Amount spent/borrowed for

primary training ($)

Unemployment insurance ($)

d Year 1 earnings

d Year 2 earnings

d First quarter of Year 3 earnings

e Quarter 2-10 UI benefits

1,889

1,883

1,957

1,960

2,040

2,051

2,564

2,564

2,564

1,866

58 1,435 1,315

72 1,909 1,860

15 506 461

0.36 12.82 11.96

0.7 36.3 35.0

100 198 624

395 10,293 10,332

559 15,715 13,816

163 4,143 3,667

155 1,754 1,695

119 **

49

45 ***

0.86 **

1.4 **

-426 ***

-39

1,899 ***

476 ***

59

1,889

1,883

1,957

1,960

2,040

2,051

2,564

2,564

2,564

1,866

58 1,435 1,315

72 1,909 1,860

14 502 459

0.31 12.68 11.87

0.7 36.3 35.0

100 198 624

393 10,295 10,284

553 15,713 13,744

163 4,143 3,667

155 1,746 1,681

119 **

49

43 ***

0.81 ***

1.4 **

-426 ***

11

1,969 ***

476 ***

65

1,861

1,859

1,937

1,941

2,020

2,029

2,539

2,539

2,539

1,848

46 1,390 1,168

61 1,846 1,724

13 490 442

0.27 12.38 11.45

0.7 36.0 34.4

51 119 296

361 10,009 9,710

516 15,296 13,004

153 3,999 3,480

133 1,588 1,455

222 ***

122 **

48 ***

0.94 ***

1.5 **

-178 ***

299

2,292 ***

519 ***

134

(continued)
238

Covariates

Covariate selection for WorkAdvance was based on both theory and modeling. During 

covariate modeling, a series of regressions were run with earnings in Quarter 2 (from UI wage 

data) as the dependent variable. The LASSO (least absolute shrinkage and selection operator) 

method was used in the regressions to help protect against including spurious variables.

30 The 

variables tested were drawn from baseline and UI wage and benefits data. In addition to testing 

standard variables from these sources, several transformations were tested, including interaction 

terms, indexes, logs, and propensity scores. Variables with statistically significant coefficients 

and other variables correlated with the outcome were included in the final covariate models.

The covariates were tested using the pooled WorkAdvance sample, and this sample was 

used to make decisions about the final covariate model. In the analysis, the impacts are presented by site, and the covariate model was adjusted somewhat for each individual site. The site 

covariate models were adjusted to include additional covariates where data were available for 

only some sites (for example, UI benefits data) and to eliminate unneeded covariates (such as

the site dummies). Following is a list of the baseline characteristics that MDRC used as covariates in the regression models:

• Random assignment year

• WorkAdvance site

 30The LASSO method is an alternative estimation method for linear regression models. The method minimizes the residual sum of squares relative to a constant that is the sum of the absolute value of the coefficients. 

See Tibshirani (1996) for more information on the LASSO method.

Appendix Table A.11 (continued)

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from 

unemployment insurance (UI) administrative records from New York State Department of Labor, Ohio 

Department of Jobs and Family Services, and Oklahoma Employment Security Commission.

NOTES: N = number (sample size); SE = standard error; WA = WorkAdvance (program) group; C = 

control group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 

percent. aThe impacts presented in the tables in the main body of the report are the impacts with extreme 

outliers removed. No extreme outliers were identified or removed from measures except as indicated.

bExtreme weekly earnings outliers are defined as earnings over $2,000 per week. These outlier values 

were recoded to $2,000. cExtreme hourly wage outliers are defined as wages over $50 per hour. These outlier values were 

recoded to $50.

dExtreme UI earnings outliers are defined as earnings over $50,000 per quarter. These outlier values 

were recoded to $0. e

Extreme UI benefits outliers are defined as benefits over $6,500 per quarter. These outlier values 

were recoded to $0.
239

• Sector31

• Gender

• Race/ethnicity

• Age

• Educational attainment (varied by site)

• Number of children living with respondent

• Born in state

• Previous license or certificate in targeted industry

• Employment status at baseline

• Monthly family income

• Employment and earnings history before random assignment

• UI benefits history before random assignment32

To test the sensitivity of the regression adjustment, MDRC compared the adjusted and 

unadjusted research group means and differences (impacts) for key outcome measures (see 

Appendix Table A.12). As shown, the adjusted and unadjusted estimates are very similar. These 

findings help confirm that the random assignment process resulted in the creation of research 

groups with similar characteristics and that the effort to field the WorkAdvance Year 2 Survey 

did not bias the results.

Missing Data

Covariates. Fifteen of the 21 covariates in the cross-site regression model for estimating program impacts had nonmissing values for all study participants. The other six measures 

were collected from responses to the WorkAdvance baseline information form. These measures 

concern study participants’ gender, race/ethnicity, number of dependent children, previous 

receipt of credentials in the targeted sector, and family income. Among the full research sample, 

the proportion of missing responses for these measures ranged from about 0.1 percent (for 

family income) to about 1.5 percent (for race/ethnicity). In response, missing values for covariates were imputed using the full sample’s mean, and dummy variables were added to the model 

to indicate missing status for each of the six covariates with missing values.

 31A dummy variable for being randomly assigned in the health care sector was used as a covariate in Towards Employment’s site-specific model. A dummy variable for being randomly assigned in the environmental 

remediation sector was used in St. Nicks Alliance’s site-specific model. 32UI benefits data were not available for sample members at Towards Employment, so this variable was 

not used as a covariate in Towards Employment’s site-specific model.
The WorkAdvance Study

 Appendix Table A.12

Adjusted Versus Unadjusted Impacts on Employment and Earnings, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome Adjusted Unadjusted Adjusted Unadjusted Adjusted Unadjusted Adjusted Unadjusted

Among respondents to Year 2 Survey

Ever employed (%)

Currently employed

Percentage of months employeda

a Employed 6 or more consecutive months

Current or most recent job

Employed in targeted sector (%)

Average total earnings per week ($)

Average hourly wage ($)

Employed and hourly wage 

above $12.00 (%)

Employed and hourly wage

above $15.00 (%)

Average hours worked per week (#)

b Employed full-time (%)

Sample size (total = 2,058)

4.4 *

10.0 ***

3.8

7.5 *

40.7 ***

98 ***

2.38 ***

16.3 ***

16.2 ***

1.1

5.5

552

5.0 **

9.0 **

2.6

6.8 *

42.0 ***

118 ***

2.76 ***

17.6 ***

16.0 ***

1.8

7.2 *

-0.8

-0.1

0.9

7.1

11.9 ***

16

-0.27

2.4

0.8

1.9

3.8

384

-2.4

-2.1

-1.1

5.2

13.1 ***

18

-0.22

1.9

1.6

1.2

2.8

1.5

5.6

-2.0

-2.6

16.5 ***

27

0.46

11.2 ***

6.6 *

1.4

2.9

560

1.3

4.6

-2.5

-2.7

14.8 ***

18

0.34

8.9 **

5.5

1.1

2.3

1.7

6.3

-5.0 *

-4.3

16.4 ***

19

0.33

1.8

-2.8

1.5

6.7

562

1.4

5.3

-6.2 *

-4.8

17.7 ***

22

0.44

2.6

-1.6

1.2

6.1

Among full research sample

Quarters 2 to 10

Ever employed (%)

Average quarterly employment (%)

Earnings ($)

2.9

4.2 *

5,186 **

3.8

3.9

5,741 ***

4.1

1.5

-201

4.1

1.2

-430

0.0

2.4

2,552

-0.4

1.5

1,093

5.9 ***

3.1

1,809

6.6 ***

3.7

2,454

(continued)

240
Appendix Table A.12 (continued)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome Adjusted Unadjusted Adjusted Unadjusted Adjusted Unadjusted Adjusted Unadjusted

Year 1 (Quarters 2 to 5)

Ever employed (%)

Average quarterly employment

Employed in all 4 quarters

Earnings ($)

Earned $20,000 or more

Year 2 (Quarters 6 to 9)

Ever employed (%)

Average quarterly employment

Employed in all 4 quarters

Earnings ($)

Earned $20,000 or more

First quarter of Year 3

Ever employed (%)

Earnings ($)

Sample size (total = 2,564)

6.1 *

2.2

-3.5

150

-0.3

6.0 **

5.7 *

7.0 *

3,747 ***

8.5 **

6.0 *

1,289 ***

690

6.1 *

0.9

-6.1 *

121

-0.3

6.9 **

6.0 **

7.0 *

4,168 ***

9.7 ***

7.4 **

1,451 ***

7.7 **

2.2

-6.9 *

-253

-4.0

2.1

2.0

5.5

191

4.8

-2.9

-139

479

7.6 *

1.7

-8.0 **

-448

-4.9

2.2

1.7

4.6

172

4.0

-3.0

-154

2.8

3.8

1.3

328

2.8

-2.5

0.8

2.8

1,837 *

9.4 ***

3.1

387

697

2.6

3.0

0.2

-306

0.8

-3.3

-0.3

1.5

1,151

7.1 **

2.7

248

4.8 *

0.5

-4.3

12

-2.8

5.4 *

5.1 *

4.9

1,621 *

5.5 *

4.9

176

698

5.4 *

1.2

-3.7

307

-1.4

6.0 *

5.8 *

5.4

1,920 **

6.3 *

5.2

226

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance administrative records from New York 

State Deparment of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: Sample sizes may vary because of missing values. aMeasures reported in time intervals cover only the first 18 months following each sample member's month of random assignment (the common follow-up 

period).

bFull-time employment is defined as working 35 or more hours in a week.

241
242

Outcomes. Sample members with missing values for dependent variables (outcomes) 

were excluded from the impact estimates. Appendix Table A.13 shows the percentage of 

sample members with missing values on key outcomes from both the Year 2 Survey and 

administrative records (there are no missing values for administrative records data). Procedures 

for assessing the effect of missing survey data are described above.

Subgroup Analysis

Impacts were calculated for key subgroups to better understand what works best for 

whom. In recent years, the standards for subgroup analysis have tightened. Leading methodologists have argued for prespecification of subgroups and limiting the number of subgroups 

tested.33 Following that lead, on the WorkAdvance project, MDRC selected two “confirmatory” 

subgroups: random assignment cohort and labor market attachment (level of prior employment). Additional “exploratory” subgroups were examined as well but are not highlighted in the 

report. The following text is from the WorkAdvance analysis plan, which was completed well 

before impacts were estimated.34

Random Assignment Cohort

MDRC chose random assignment cohort as one key subgroup, because start-up of the 

WorkAdvance program was slow across all the providers and the programs matured over time. 

Thus, the quantity and quality of services provided to participants who entered the program 

earlier compared with those who entered the program later likely differed. It was hypothesized 

that WorkAdvance would be most effective for the late cohort due to program maturation. The 

decision in choosing a cutoff point between the early and late cohorts was influenced by several 

factors:

• The sites changed some eligibility criteria and recruitment strategies over 

time; this may have influenced the baseline characteristics of study participants enrolling at a specific time.

• In fall 2012, Towards Employment and the Madison Strategies Group both 

shifted their recruitment efforts and programs from a focus on the placementfirst track to a focus on the training-first track.

• St. Nicks Alliance added new training programs in hazard materials transportation and pest control in mid-2012 due to labor market demand.

 33Bloom and Michalopoulos (2010). 34The text was edited for clarity (for example, to spell out acronyms).
243

The WorkAdvance Study

Appendix Table A.13

Percentage Missing on Key Outcomes, by Site

Per Scholas St. Nicks Alliance

Madison 

Strategies Group

Towards

Employment

Outcome (%) WA C WA C WA C WA C

Among respondents to Year 2 Survey

Ever employed

Currently employed

Employed in targeted sector

Average hourly wage at current or

most recent job

Currently or most recently working 

with hourly wages $15.01 or higher

Employed full-time at current or most

recent job

Offered many opportunities for career 

advancement

Sample size (total = 2,058)

0.0

0.0

0.0

2.1

2.1

0.3

0.3

287

0.0

0.0

0.4

9.1

9.1

1.5

1.1

265

0.0

0.0

0.5

4.4

4.4

0.5

1.5

205

0.0

0.0

1.1

7.3

7.3

0.6

0.6

179

0.0

0.0

0.3

2.4

2.4

0.7

0.7

297

0.4

0.4

0.8

4.9

4.9

0.8

1.1

263

1.0

1.0

1.4

5.6

5.6

1.7

2.4

286

0.0

0.0

1.1

3.6

3.6

0.7

0.4

276

Among full research sample

Quarters 2 to 10

Ever employed

Average quarterly employment

Earnings

Year 1 (Quarters 2 to 5)

Ever employed

Average quarterly employment

Employed in all 4 quarters

Earnings

Earned $20,000 or more

Year 2 (Quarters 6 to 9)

Ever employed

Average quarterly employment

Employed in all 4 quarters

Earnings

Earned $20,000 or more

First quarter of Year 3

Ever employed

Earnings

Sample size (total = 2,564)

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

349

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

341

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

242

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

237

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

353

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

344

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

349

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

349

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from 

insurance administrative records from New York State Department of Labor, Ohio Department 

Services, and Oklahoma Employment Security Commission.

NOTE: WA = WorkAdvance (program) group; C = control group.

unemployment 

of Jobs and Family 
244

• Most of the sites did not begin implementing postemployment services until 

late 2012 or early 2013.

• There was a desire for a close to 50/50 split between the two groups.

Accounting for all the above factors, the research team decided on a cutoff point of October 2012 for the cohort definition. The early cohort includes all participants randomly assigned through September 2012, and the late cohort includes all participants randomly assigned 

in or after October 2012.

Labor Market Attachment

Various MDRC studies have used participant level of disadvantage as a subgroup construct, and variation in impacts is often measured in this domain. The Employment Retention 

and Advancement Study (ERA) focused specifically on labor market experience, and the ERA 

research team selected prior employment status as a confirmatory subgroup.35 ERA found 

substantial variation in impacts by the level of participants’ prior employment experience. 

Impact analyses showed that the programs produced positive economic effects (albeit relatively 

modest) for participants who entered ERA with a medium level of labor force attachment 

(individuals who had worked in two or three quarters of the prior year or had earnings between 

$3,000 and $10,000 that year), while no impacts were found for participants with a low or high 

level of labor force attachment.36

This second key subgroup stems from the hypothesis that WorkAdvance programs will 

be more effective for those with at least semiattachment to the labor market, as was seen in 

ERA. The positive effects seen in this middle group may have been because individuals who 

had not worked recently (those in the low recent labor force attachment subgroup) had too many 

barriers to employment to benefit from program services, while those same services offered too 

little value to individuals with extensive recent employment (those in the high recent labor force 

attachment subgroup). Additionally, with the recent recession, there has been a concern about 

the effects on reentry into the labor market for the long-term unemployed.

The prior employment subgroup analysis breaks the sample into the following groups:

• Unattached or long-term unemployed — sample members who have never 

been employed or who were not employed for at least seven months before

random assignment

 35Hamilton and Scrivener (2012b).

36Sample members from all the ERA model tests were combined and, based on their employment and 

earnings in the previous year, were put into three subgroups: high, medium, or low recent labor force attachment.
245

• Semiattached — sample members not employed for one to six months before

random assignment

• Fully attached — sample members working at the time of random assignment or not employed for less than one month before random assignment

Conditional Subgroup Analysis

An additional technique used for exploratory analysis is called “conditional subgroup 

analysis.” In this type of analysis, interaction terms are used to control for the moderating 

effects of other baseline characteristics when estimating the relationship between a particular 

subgroup and program effects. For example, in estimating whether WorkAdvance has a larger 

effect for the early or late cohort sample, it might be important to control for work experience in 

the sector or prior education.37

The conditional subgroup model specifies earnings and advancement outcomes as follows:

YYii = ∝ + ββPPii + �δδkk

kk

CCCCkk + �γγkk

kk

CCCCkk PPii + εεii

where participant characteristics are measured in terms of deviations from the grand 

mean and:

YYii = outcome for sample member i

PPii = indicator of program group membership for sample member i (1 = program 

group member; 0 = control group member)

CCCCkk = client characteristics k for sample member i

∝ = mean outcome for typical control group member

ββ = program impact for typical program group member

δδkk = effect of client characteristic k on control group mean outcome

γγkk = effect of client characteristic k on program impacts

εεii = random error for sample member i

 37Controlling for other factors, including economic conditions, may also be important here, but it was not 

possible in this analysis.
246

Comparison of Survey and Unemployment Insurance 

Employment Impacts

There are several reasons why estimates from UI administrative records may differ 

from estimates from survey data. This section first describes the strengths and weaknesses of 

each data source and then describes the results of a comparison between findings from the UI 

wage administrative data and the survey data in WorkAdvance.

Administrative records have a number of advantages:

• They cover the full sample, which enhances statistical power.

• They contain a history of events. Administrative data can be used to produce 

time trends of employment, earnings, and public assistance — often at a fine 

level of granularity.

• Measures created from administrative records are not affected by recall error, 

such as would be expected if a survey participant is asked to remember distant past events.

• They can often be collected from one centralized source, rather than by tracking thousands of individuals. Once data collection procedures are in place, 

further data collection has low marginal costs.

The primary disadvantage of administrative records is that they typically do not cover 

all jobs. It is estimated that UI records cover approximately 90 percent of jobs, though rates 

might be lower for low-wage workers.38 For example, administrative records will not have 

information on jobs in the informal sector or jobs with the federal government.39 Furthermore, 

state administrative records do not have information on what happens outside the state. While 

the conventional wisdom suggests that undercoverage in administrative records should be 

equivalent between research groups in randomized controlled trials (RCTs), it is easy to imagine 

cases where undercoverage could interact with intervention strategies to produce bias.40 Differing coverage rates could produce (or shroud) estimated impacts.41

 38Kornfeld and Bloom (1999); Hotz and Scholz (2001). 39Kornfeld and Bloom (1999). 40Barnow and Greenberg (2015). 41Riccio et al. (2013). This type of scenario at least partly accounted for the difference between surveyand administrative records-based estimates of employment in the New York City Conditional Cash Transfer 

experiment. In that study, there was evidence that program group members were disproportionately likely to 

seek employment in the informal sector, where jobs that qualified for program-related incentives were more 

easily found (such as babysitting). Such jobs are not typically covered by UI records. Therefore, the survey-

(continued)
247

This may be especially troublesome in sectoral employment programs, including 

WorkAdvance, which by definition focus on tightly defined segments of the labor market.42 If a 

sectoral program targets a sector with a different coverage rate in UI data compared with the 

overall labor market, there is the potential for biased impact estimates. As one example, within 

the transportation sector (the sector targeted by Madison Strategies Group), large trucking 

companies are often headquartered in one state but have employees working in many states. If 

more WorkAdvance group members end up in jobs within the transportation sector compared 

with control group members, it is possible that these in-sector, out-of-state jobs may be missed 

by administrative records. Moreover, a targeted sector might be more or less likely to have 

individuals who are self-employed, whose employment and earnings histories would not be 

included in administrative records.

Surveys are also an important data source for many RCTs because they provide information that administrative records and other data fail to capture. Without survey data, it would 

be difficult to quantify program “dosage,” or the extent to which a person actually engaged with 

a program.43 Survey data also provide valuable insight on certain behaviors, beliefs, program 

experiences, participant or household characteristics, and other issues that may influence 

outcomes observed in administrative records. In addition, summarized earnings data from 

administrative records can be better understood with survey data, which provide information 

about work schedules, rates of pay, and job changes. Finally, in many domains, administrative 

records are not available, and some evaluations have to depend almost completely on surveys. 

(For example, the research team was interested in the impacts of WorkAdvance on various 

measures of overall well-being, which are not captured in administrative records.) 

Unlike administrative records — where data are obtained for the full study sample — 

survey data are relatively expensive to collect and often are obtained for only a subset of the full 

sample, with the expectation that they will represent the full sample, which in turn is expected to 

represent a larger population. When a survey fails to be representative, it is considered biased. 

The nonresponse analysis, described above, found little evidence that the survey results in 

WorkAdvance were biased.

To summarize, both UI wage and survey data have advantages and disadvantages, and 

it is not clear which source is superior. Different results can be obtained based on the coverage 

and biases inherent in each data source and how these interact with the program and population 

 

based estimates of impacts were positive, whereas estimated impacts from administrative records were not 

statistically significant.

42Maguire et al. (2010). 43While most programs collect program participation data in management information systems, these data 

are typically not available for the control group.
The WorkAdvance Study

Appendix Table A.14

Employment in a UI-Covere

Su rvey Respondents, 

Outcome (%)

Currently employed according to survey

d Job Comp

rvey Intervi

WA

74.4

ared with Surv

ew, Among Su

Per Scholas

C Difference

64.4 10.0 ***

ey-Reported Emplo

St. Nicks Alliance

WA C Difference

65.3 65.4 -0.1

yment at the Time of t

by Site

Madison Strateiges Group

WA C Difference

77.6 72.0 5.6

he Year 2 

Towards Employment

WA C Difference

64.5 58.2 6.3

Employed in quarter of survey interview

according to UI 72.1 61.9 10.2 ** 63.7 59.5 4.2 72.0 71.2 0.8 70.8 62.1 8.7 **

Employment status coverage

Same employment status 

according to survey and UIa 77.5 77.2 0.3 76.0 77.3 -1.3 77.3 75.7 1.6 81.5 74.7 6.8 *

Currently employed only according to UI 10.1 10.2 -0.1 11.2 8.4 2.8 8.6 11.7 -3.1 12.5 14.7 -2.2

Currently employed only according to survey 12.4 12.6 -0.2 12.8 14.3 -1.5 14.1 12.6 1.5 6.0 10.6 -4.6 *

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from New 

York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission.

NOTES: WA = WorkAdvance (program) group; C = control group.

Survey employment is measured by month relative to each sample member's date of random assignment, while UI employment is measured in relative 

quarters. The UI employment measures are for the relative quarter in which the respondent was interviewed. This ranges from Quarter 6 to Quarter 11.

Four survey respondents were missing a current employment status and are not included in the first measure in the table.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

aThis measure includes both (1) sample members unemployed according to both the survey and UI and (2) sample members employed according to both the 

survey and UI.

248
249

under study. Therefore, it is important to compare UI wage and survey data. Appendix Table

A.14 compares the employment rates of survey respondents in UI-covered jobs with the surveyreported employment rates at the time of the Year 2 Survey.

This analysis, shown in Appendix Table A.14, helps answer a couple of questions. 

First, are the employment levels and effects consistent across these data sources? Second, for 

each survey respondent, do these data sources capture consistent employment information (that 

is, if a respondent said he was working at the time of the survey interview, was this reflected in 

his UI wage records)? The first two rows of the table answer the first question: Here the results 

show that, for three of four sites, the levels and effects on employment at the time of the survey 

interview are fairly consistent across data sources (though the effect sizes — which are not 

statistically significant — vary for St. Nicks Alliance and Madison Strategies Group). For the 

fourth site, Towards Employment, only the effects on UI-based employment are statistically 

significant. In addition, for this site, more employment (at the time of survey interview) is 

captured via UI wage records than from the survey.

The remaining rows of results provide insight on the second question. Here again, for 

three sites, the data sources reflect consistent employment information on about three-fourths of 

the respondent samples. In the case of Towards Employment, it appears that WorkAdvance 

may have led to an increase in employment that is covered by UI wage records in Ohio. Overall, 

the results of this analysis suggest that, for the most part, the UI wage and survey data sources 

tell similar stories at each of the four sites.

Appendix B 

Supplementary Exhibits for Chapter 1

253

 The WorkAdvance Study 

Appendix Table B.1

Additional Characteristics of Research Sample Members at Baseline

Characteristic PS SNA MSG TE Overall

Demographic characteristics (%)

Citizenship

Born in United States

Naturalized

Noncitizen

Veteran

Family status

Living with a partnera (%)

Primary caregiverb (%)

Single caregiver

Noncustodial parentc

Average number of children living with study participantd

Age of primary caregiver's youngest child (%)

Less than 6 years old

6 to 9 years old

10 to 15 years old

16 years or older

Not a primary caregiver for any children

Currently enrolled in education or training program (%)

English as a second language

Adult Basic Education

High school/GED preparation course

Occupational skills training

College course toward associate's/two-year degree

College course toward bachelor's/four-year degree

Other

Employment status (%)

Average hourly wage at current or most recent jobe

Never employed

$7.24 or less

$7.25-$8.00

$8.01-$9.00

$9.01-$10.00

$10.01-$12.50

$12.51 - $15.00

$15.01 or higher

Among those unemployed for more than 12 monthsf

71.9

15.7

12.5

2.2

8.1

17.7

7.4

9.1

0.3

9.4

3.8

2.8

1.7

82.3

4.3

0.0

0.0

0.0

0.7

2.3

1.9

0.1

3.6

5.9

11.3

12.3

13.2

17.1

18.4

18.1

40.0

76.9

11.9

11.1

6.4

8.4

24.8

11.2

22.9

0.4

13.1

4.2

4.6

2.7

75.4

3.0

0.2

0.0

0.6

0.8

0.8

0.4

0.2

2.1

5.0

14.0

9.0

17.2

12.6

15.5

24.7

44.9

95.4

1.1

3.4

11.7

17.0

38.4

13.4

17.1

0.8

24.4

5.5

7.3

1.1

61.6

4.6

0.3

0.1

1.0

0.9

1.1

1.1

0.3

0.7

5.5

13.8

14.5

20.9

25.7

13.9

5.0

25.7

97.9

2.0

0.1

4.6

8.3

43.3

31.1

10.3

0.9

24.5

7.6

7.5

3.7

56.7

7.6

0.0

0.3

1.7

1.1

3.7

0.6

0.3

2.7

6.7

21.2

20.9

16.5

16.9

8.0

7.0

38.8

86.3

7.3

6.4

6.2

10.6

31.6

16.2

14.2

0.6

18.3

5.4

5.6

2.3

68.4

5.1

0.1

0.1

0.9

0.9

2.1

1.1

0.2

2.3

5.9

15.2

14.6

16.9

18.5

13.8

12.8

40.1

(continued)
254

Appendix Table B.1 (continued)

Characteristic PS SNA MSG TE Overall

Average hours worked per week at current

 or most recent job

Among those currently working

Circumstances that may affect job change or retention (%)

Physical or mental health problem that limits work 

Previously convicted of a crime and incarcerated

Other income sources (%)

Earnings from spouse or partner

Medical coverage, among those with coverage (%)

Type of health insurance plan

Employer-provided

Other

Publicly funded coverage

g Parents with publicly funded coverage for their children

Housing arrangements (%)

Public housing

Section 8 housing vouchers

Reduced rent

Group shelter

Homeless

32.0

22.6

0.6

4.8

9.9

1.9

24.9

73.2

53.5

13.8

6.1

2.9

1.5

0.0

34.4

25.8

0.8

15.9

12.6

0.4

7.1

92.5

38.5

9.9

3.6

3.8

5.5

0.2

37.7

30.6

1.9

26.8

22.3

16.5

51.0

32.5

68.4

4.9

3.2

1.3

1.9

0.6

33.1

28.6

1.0

11.9

8.8

9.3

18.2

72.5

74.2

11.7

5.0

3.3

1.7

1.1

34.3

28.0

1.1

14.8

13.5

6.2

23.5

70.4

62.3

10.1

4.5

2.7

2.4

0.5

Sample size 690 479 697 698 2,564

SOURCE: MDRC calculations from the WorkAdvance baseline information form.

NOTES: PS = Per Scholas; SNA = St. Nicks Alliance; MSG = Madison Strategies Group; TE = 

Towards Employment; GED = General Educational Development certificate.

Italic type indicates that the metric is not among the full sample shown in the table.

Sample sizes may vary because of missing values.

a

This does not include sample members who responded as "married and living with spouse."

bA primary caregiver is a parent who has at least one child living with him or her more than half the 

time. c

While there is a legal definition for a noncustodial parent, in this analysis a noncustodial parent is 

defined as a parent who has at least one child not living with him or her for more than half the time. 

dThis is the average number of children living with the sample member for more than half the time. 

The estimate is based on how many people are in the family minus the sample member and another 

adult if he or she is married and living with the spouse or is not married but living with a partner. e

Wages for sample members who have never been employed are counted as $0.

f

This is restricted to sample members unemployed for more than 12 months whose wage at their 

most recent job was higher than $15 per hour.

gThe percentage of parents with publicly funded health care coverage for their children is measured 

among sample members with children.
255

The WorkAdvance Study

Figure B.1

Annual Unemployment Rates 2011-2015, Nationally and by Metropolitan Area

0

2

4

6

8

10

2011 2012 2013 2014 2015

New York City

Tulsa, OK

Cleveland, OH

United States

Percentage

SOURCE: Local Area Unemployment Statistics data from the Bureau of Labor Statistics.

NOTE: Unemployment rates are shown by metropolitan area and for the United States. The Cleveland

metropolitan area includes Elyria and Mentor, Ohio. The New York metropolitan area includes northern New 

Jersey and Long Island, New York.
256

The WorkAdvance Study

Figure B.2

Sector Employment in Quarter 3, 2011, and Quarter 3, 2013, by Metropolitan Area

122,571 123,170 142,391 183,007

0

100,000

200,000

300,000

400,000

Manufacturing Health care

Cleveland, Ohio

Q3 2011 Q3 2013

315,599

119,292

334,475

125,702

0

100,000

200,000

300,000

400,000

Environmental remediation and related industries Information technology

New York City

Q3 2011 Q3 2013

64,415 68,411

0

100,000

200,000

300,000

400,000

Transportation and manufacturing

Tulsa, Oklahoma

Q3 2011 Q3 2013

(continued)
257

Figure B.2 (continued)

SOURCE: U.S. Census Bureau, Quarterly Workforce Indicators from the Longitudinal Employer-Household 

Dynamics extraction tool.

NOTES: Sectors and industries are defined by the North American Industry Classification System. Manufacturing 

includes codes starting with 31-33; health care includes codes starting with 62; environmental remediation includes 

codes starting with 236, 2371, 2373, 2379, 238, 5617, and 562; information technology includes codes starting 

with 5112, 517, 518, 5415, and 8112; and transportation includes codes starting with 48-49.

The employment numbers presented are the total number of industry jobs on the first day of the quarter.

Appendix C 

Supplementary Exhibits for Chapter 2 

261

The WorkAdvance Study

Appendix Table C.1

Percentage Distribution of Applicant Recruitment Sources During Select Time Periods

Recruitment Source a Per Scholas

St. Nicksa AllianceMadison

Strategies Group

Towards

Employment

Among all applicants (%)

Friend or family member

Internet

Flyer, poster, newsprint

Job/career fair

Another program or organization

Radio/TV

Walk-in

Employer

b Other/unknown

Applicants (total = 1,783)

32.9

22.9

12.9

8.6

24.3

1.4

NA

1.4

0.0

70

31.7

17.2

13.8

10.3

21.4

0.7

4.1

1.4

2.1

145

5.8

47.9

NA

1.4

17.8

0.5

NA

0.5

26.2

432

35.0

13.6

11.0

NA

14.6

5.7

5.4

NA

14.6

1,136

Among applicants randomly assigned (%)

Friend or family member

Internet

Flyer, poster, newsprint

Job/career fair

Another program or organization

Radio/TV

Walk-in

Employer

b Other/unknown

Applicants (total = 342)

28.6

14.3

0.0

21.4

21.4

7.1

NA

7.1

0.0

14

28.0

30.0

20.0

2.0

14.0

0.0

8.0

0.0

0.0

50

8.6

65.7

0.0

0.0

20.0

0.0

0.0

0.0

5.7

35

33.3

11.1

5.3

0.0

9.9

6.2

5.3

0.0

28.8

243

SOURCES: MDRC calculations from a recruitment questionnaire administered at Per Scholas (PS) and St. Nicks 

Alliance (SNA), a report based on program tracking data provided by Madison Strategies Group (MSG), and 

program tracking data provided by Towards Employment (TE).

NOTES: NA= not applicable. 

At PS, the recruitment funnel analysis covered 3 weeks in July 2012, and applicants are individuals who 

attended orientation.

At SNA, two rounds of the recruitment funnel analysis were completed in mid-2012 covering 15 total weeks. 

The findings from both rounds were combined in this analysis. Applicants are individuals who expressed an interest 

in WorkAdvance.

At MSG, the recruitment funnel analysis covered 6 weeks in mid-2012, and applicants are individuals who were 

scheduled for orientation.

At TE, the recruitment funnel analysis covered 64 weeks from mid-2011 to late 2012. Applicants are individuals 

from the 11 most common recruitment sources who expressed an interest in WorkAdvance.

See Appendix Table C.2 for details regarding the length and timing of the data coverage period and the 

definition of an applicant. a

Applicants were allowed to select more than one recruitment source at PS and in the second round of the 

recruitment funnel at SNA, so percentages may sum to more than 100 percent.

bApplicants at TE in the category of "other/unknown" did not specify how they heard about WorkAdvance. At 

MSG, this category includes applicants referred by training programs and job placements and applicants who did 

not specify how they heard about WorkAdvance.
The WorkAdvance Study

Appendix Table C.2

Criteria for Study and Program Eligibility, by Provider, During Select Time Periods

Study intake

• 18 years or older

Eligibility • Legally allowed to work in the United States

requirements • Monthly family income less than 200 percent of the federal poverty level

• Earning less than $15 per hour, if employed at intake

Program intake

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Assessment 

score

• TABE: 10th-grade level 

(lowered to 9.5 grade level 

in July 2012)

• TABE: 9th-grade level 

(later lowered to 8thgrade level)

•

•

•

Prove It! Math and Reading: 

8th-grade level

Mechanical aptitude test

Behavioral assessment

• TABE Locator: 6th- to 

10th-grade level, depending on track

OST-specific

criteria

• Driver’s license (HCDL 

only)

•

•

Manual dexterity test 

Driver’s license (CDL only)

•

•

Criminal background 

check (health care only)

Sector screening 

questionnaire

Additional 

eligibility a requirements

•

•

Not eligible for another 

contract (e.g., veterans)

High school diploma/GED

•

•

Not trained by St. 

Alliance in past 2 

Drug screen

Nicks 

years • Drug screen

(continued)

262
Appendix Table C.2 (continued)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Intake 

process

Over 1-7 days or longer:

• Potential applicants attend 

orientation. Staff determines basic eligibility.

• Eligible applicants take the 

TABE. Staff interviews 

those who pass.

• Applicants are interviewed 

again. Those who remain 

fill out paperwork and 

enter random assignment.

Over 1-2 days or longer:

• Staff schedules eligible 

applicants for orientation.

• At orientation, applicants 

take the TABE. Those 

who pass are interviewed 

and fill out paperwork.

• Applicants complete drug 

screening.

• Staff conducts case 

conferences and schedules eligible applicants 

for random assignment at 

a later date.

Over 2-3 days or longer:

• Applicants complete 

paperwork and tests. Those 

who pass receive a blank 

work history template.

• Applicants return with draft 

work history. Staff interviews applicants and conducts case conferences. 

• Eligible applicants enter 

random assignment.

Over 2 days or longer:

• Staff checks eligibility via 

phone and schedules 

candidates for orientation.

• At orientation, applicants 

complete all assessments 

and are interviewed by the 

staff.

• Applicants complete drug 

screening (48 hours).

• Those who pass fill out 

paperwork and enter 

random assignment.

NOTES: TABE = Tests of Adult Basic Education; OST = occupational skills training; HCDL = hazmat commercial driver’s license; CDL = 

commercial driver’s license; GED = General Educational Development certificate. 

 a

Some additional eligibility requirements changed over time across all the providers.

263
Metric Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

 1

Ever participated in 

any career a readiness activity

Participant attended first day of 

skills training program, which 

includes career readiness 

training 

Participant attended first day of 

skills training program, which 

includes career readiness 

training 

Participant attended at least 1 

day of career readiness training

Participant completed initial 

assessment to discuss career 

goals and employment barriers 

Ever started skills b training

Participant attended first day of 

skills training program

Participant attended first day of 

skills training program

Participant attended first day of 

skills training program

Participant attended first day of 

skills training program

Participant is reported by Participant is reported by Participant is reported by Participant is reported by 

le 2.

Ever completed training instructor to have training instructor to have training instructor to have training instructor to have 

Tab

b skills training completed all requirements of completed all requirements of completed all requirements of completed all requirements of 

training program training program training program training program

Ever obtained a 

credential in 

targeted sector

Participant passed required 

exam(s) for at least 1 of the 

credentials offered through the 

provider

Participant passed required 

exam(s) for at least 1 of the 

credentials offered through the 

provider

Participant passed required 

exam(s) for at least 1 of the 

credentials offered through the 

provider

Participant passed required 

exam(s) for at least 1 of the 

credentials offered through the 

provider

Table 2.3

Ever participated in 

any career a readiness activity

Participant attended first day of 

skills training program, which 

includes career readiness 

training 

Participant attended first day of 

skills training program, which 

includes career readiness 

training 

Participant attended at least 1 

day of career readiness training

Participant completed initial 

assessment to discuss career 

goals and employment barriers 

Ever started 

classroom-based

career readine

training

ss 

Participant attended first day of 

skills training program, which 

includes career readiness 

training 

Participant attended first day of 

skills training program, which 

includes career readiness 

training 

Participant attended at least 1 

day of career readiness training

Participant attended first day of 

contextualized career readiness 

training workshop

Ever completed 

classroom-based 

career readiness 

training

Participant attended last day of 

skills training program, which 

includes career readiness 

training

Participant attended last day of 

skills training program, which 

includes career readiness 

training

Participant completed at least 4 

days of career readiness training

Participant successfully 

completed contextualized career 

readiness training workshop

Ever received help 

obtaining 

supportive services

Participant received a referral to 

an outside organization (Not measured)

Participant received a referral to 

an outside organization, a 

transportation voucher, or a 

training voucher

Participant met with staff to 

discuss barrier removal

(continued)

264

The WorkAdvance Study

Appendix Table C.3

Description of Program-Tracking Metrics, by Site
265

Appendix Table C.3 (continued)

Metric Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Table 2.5b

Ever scheduled to 

start skills training All participants All participants Participant was designated prior 

to RA as "on the training track"

Participant was signed up for 

training

Ever started skills 

training

Participant attended first day of 

skills training program

Participant attended first day of 

skills training program

Participant attended first day of 

skills training program

Participant attended first day of 

skills training program

Ever completed 

skills training

Participant is reported by 

training instructor to have 

completed all requirements of 

training program

Participant is reported by 

training instructor to have 

completed all requirements of 

training program

Participant is reported by 

training instructor to have 

completed all requirements of 

training program

Participant is reported by 

training instructor to have 

completed all requirements of 

training program

Ever dropped out of 

skills training

Participant started skills training 

and is reported by training 

instructor to have withdrawn or 

been dismissed from training

Participant started skills 

training, did not complete 

training within 18 months of 

RA, and the participant's 

training cycle is complete

Participant started skills training 

and is reported by training 

instructor to have withdrawn or 

been dismissed from training

Participant started skills training 

and is reported by training 

instructor to have withdrawn or 

been dismissed from training

Enrolled in skills 

training at 18 

months after RA

Participant started skills 

training, did not complete skills 

training, and did not drop out of 

skills training within the first 18 

months of enrollment

Participant started skills 

training, did not complete skills 

training, and did not drop out of 

skills training within the first 18 

months of enrollment

Participant started skills 

training, did not complete skills 

training, and did not drop out of 

skills training within the first 18 

months of enrollment

Participant started skills 

training, did not complete skills 

training, and did not drop out of 

skills training within the first 18 

months of enrollment

Ever obtained a 

credential in 

targeted sector

Participant passed required 

exam(s) for at least 1 of the 

credentials offered through the 

provider

Participant passed required 

exam(s) for at least 1 of the 

credentials offered through the 

provider

Participant passed required 

exam(s) for at least 1 of the 

credentials offered through the 

provider

Participant passed required 

exam(s) for at least 1 of the 

credentials offered through the 

provider

(continued)
266

Appendix Table C.3 (continued)

Metric Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Table 2.5 (continued)b

Average number of 

weeks in training

For participants who completed 

training, the average number of 

weeks between training start and 

end dates; for participants who 

dropped out of training, the 

average number of weeks 

between training start and 

halfway point of training cycle; 

for participants currently 

enrolled in training, average 

number of weeks between 

training start and 18 months 

after enrollment; for participants 

who never started training, 0 

weeks

For participants who completed 

training, the average number of 

weeks between training start and 

end dates; for participants who 

dropped out of training, either 

the average number of weeks 

between training start and 

halfway point of training cycle 

or average number of weeks 

between training start and 

dropout date, where dropout 

date is recorded; for participants 

currently enrolled in training, 

average number of weeks 

between training start and 18 

months after enrollment; for 

participants who never started 

skills training, 0 weeks

For participants who completed 

training, the average number of 

weeks between training start and 

end dates; for participants who 

dropped out of training, the 

average number of weeks 

between training start and 

dropout date; for participants 

currently enrolled in training, 

average number of weeks 

between training start and 18 

months after enrollment; for 

participants who never started 

training, 0 weeks

For participants who completed 

training, the average number of 

weeks between training start and 

end dates; for participants who 

dropped out of training, the 

average number of weeks 

between training start and 

dropout date; for participants 

currently enrolled in training, 

average number of weeks 

between training start and 18 

months after enrollment; for 

participants who never started 

training, 0 weeks

SOURCES: Program tracking systems managed by Per Scholas, St. Nicks Alliance, Madison Strategies Group, and Towards Employment.

NOTES: RA = random assignment.

All metrics include only activities that started and occurred after random assignment. 

The metrics included are among participants and cover the first 18 months of program activity.

a

The career readiness activity metric captures the first activity after random assignment.

bAll skills training-related measures include only skills training programs offered through the provider.
267

The WorkAdvance Study

Appendix Table C.4

Percentage Distribution of Reasons for Drop-Off Among Applicants

Who Attended Orientation

Description (%) Per Scholas

St. Nicks

Alliance

Madison

Strategies Group

Towards

Employment

Reason for drop-off after orientation

Did not meet eligibility requirementsa

Left after orientation

Did not pass assessment test(s)

b Did not attend staff interview

Did not pass internal case conference

Did not attend random assignment appointment

15.7

12.9

35.7

7.1

8.6

0.0

0.0

12.4

22.7

1.0

10.3

2.1

0.0

14.5

18.8

19.8

28.5

1.4

0.0

7.7

21.1

NA

12.3

NA

Applicants who attended orientation (total = 787) 70 97 207 413

SOURCES: MDRC calculations from a recruitment questionnaire administered at Per Scholas (PS) and St. Nicks 

Alliance (SNA); a report based on program tracking data provided by Madison Strategies Group (MSG); and 

program tracking data provided by Towards Employment (TE).

NOTES: NA = not applicable.

Refer to the notes to Appendix Table C.1 and to Appendix Table C.2 for provider-specific details regarding the 

length and timing of the data coverage period and the definition of an applicant, as there is variation across 

providers.

Screening processes varied over time at all providers.

a

Initial checks for applicant eligibility were performed before orientation at SNA, MSG, and TE, and eligibility 

was checked after orientation at PS. All providers verified applicants' eligibility after orientation.

b"Did not attend staff interview" was not used as a drop-off point in the analysis at TE because its data did not 

include this level of detail.

Appendix D 

Supplementary Exhibit for Chapter 3 

The WorkAdvance Study

Appendix Table D.1

Year 2 Impacts on Educational Attainment, Internships, On-the-Job Training, and Employment Services, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome (%) WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Educational attainment

Less than a high school diploma

High school diploma 

GED

Some college credit but less than

 1 year

1 or more years of college, but

 no degree

Associate's or 2-year college degree

Bachelor's degree or above

Participated in any E&T or obtained a an employment service

b Obtained an employment service

Ever participated in an internship

Currently participating in an internship

Ever paid as part of an internship

Ever worked in an on-the-job 

training position

Currently working in an on-the-job

training position

0.4

19.6

9.0

11.9

21.7

14.7

22.8

95.9

93.3

31.0

2.6

12.9

4.0

0.4

0.7

16.2

10.3

10.6

25.6

12.1

24.4

76.2

65.4

15.1

3.3

7.9

7.8

0.4

-0.4

3.3

-1.3

1.3

-3.9

2.5

-1.6

19.7 ***

27.9 ***

15.9 ***

-0.7

4.9 *

-3.8 *

0.0

8.0

25.6

15.5

9.9

20.3

8.4

12.4

97.9

96.5

11.8

1.5

4.6

5.9

3.0

8.2

25.6

15.9

12.2

18.8

10.0

9.3

76.1

67.8

7.2

2.2

4.2

3.3

0.5

-0.2

-0.1

-0.5

-2.3

1.4

-1.7

3.1

21.8 ***

28.7 ***

4.7

-0.7

0.4

2.6

2.4 *

7.7

32.1

12.5

10.5

23.4

7.9

6.0

99.0

98.0

11.0

2.2

8.9

6.1

0.7

7.7

30.0

11.5

11.4

22.0

12.8

4.7

63.9

50.7

4.0

0.2

3.3

2.2

1.1

0.0

2.1

1.0

-0.9

1.4

-4.9

1.3

35.1 ***

47.3 ***

7.0 ***

1.9 **

5.7 ***

3.9 **

-0.4

6.0

30.4

10.0

12.9

24.5

8.5

7.7

93.9

90.9

23.6

0.7

1.6

6.8

0.4

5.4

32.6

11.4

15.6

21.0

9.4

4.7

68.5

58.7

4.9

1.1

2.3

2.0

1.1

0.6

-2.3

-1.4

-2.7

3.6

-0.9

3.1

25.4 ***

32.1 ***

18.7 ***

-0.5

-0.7

4.8 ***

-0.8

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

(continued)

271
 

272

Appendix Table D.1 (continued)

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group; GED = General Educational Development certificate; E&T = education and training.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall-Young adjusted p-values were used for 

categorical measures.

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 

follows: ††† = 1 percent; †† = 5 percent; † = 10 percent. No statistically significant differences were found.

Sample sizes may vary because of missing values.

aMeasure includes job training services such as job-specific skills training, career readiness help, job search assistance, postemployment help, training in 

basic reading and math skills, college courses, short-term classes, internships, on-the-job training, and GED classes.

bMeasure includes career readiness services, job search assistance, and postemployment help. 
Appendix E 

Net Cost Thought Experiment

275

This appendix uses a thought experiment to demonstrate the implications of dividing aggregate 

costs in Year 2 of the WorkAdvance program by the number of Year 2 program entrants. It is 

based on assuming that WorkAdvance was in a steady state not only in Year 2, but also in the

two prior years. The exercise has two important implications:

1. If the assumption about the steady state were valid, dividing gross aggregate costs 

in Year 2 by the number of program entrants during Year 2 would provide an appropriate estimate of gross costs per participant for those entering during Year 2, 

including the program costs they incurred after Year 2.

2. Because the assumption is invalid — Year 1 began in April 2011, but program entry did not begin until well after that date and program start-up was slow — aggregate costs in Year 2 are understated relative to what they would be under a true 

steady state. As implied by the thought experiment, however, most of this understatement would occur in the estimated cost of postemployment retention and advancement because that activity occurs late in the program process. To adjust for 

this understatement, the estimated Year 2 cost of postemployment services is replaced by the Year 3 cost of these services, as explained below. 

The thought experiment is as follows: 

If WorkAdvance were in the steady state not only in Year 2 but also in the two prior 

years, someone entering WorkAdvance in the first month of Year 2 could potentially incur costs 

during every month of Year 2 plus every month of Year 3, given the 24 months of eligibility. 

Year 2 costs represent only the first 12 months of the 24. However, someone else who entered 

WorkAdvance in the first month of Year 1 could potentially incur costs during all 12 months of 

Year 2, and these costs would be included as part of Year 2 costs. (Notice that because WorkAdvance started after the first month of Year 1, this second person does not actually exist. This 

causes Year 2 costs to understate costs incurred during the last 12 months of the 24 months of

eligibility. These missing costs should mainly be the cost of postemployment services.)

Now, look at someone entering WorkAdvance in the second month of Year 2. That person could potentially incur costs for 11 months of Year 2 plus 13 months after Year 2. Costs in 

Year 2 do not capture these 13 months of costs. However, someone who entered WorkAdvance 

in the second month of Year 1 could potentially incur costs during all months of Year 2, and 

someone entering in the second month of Year 0 (the hypothetical year before Year 1) could 

potentially incur costs for the first month of Year 2. This would account for all 13 months of the 

costs that are missed for the person entering in Month 2 of Year 2.

Obviously, this exercise could be continued for a person entering in the third month of 

Year 2, entering in the fourth month, and so forth. Consider just one more possibility: a person 
276

who enters WorkAdvance in the twelfth month of Year 2. This person would incur costs for 

only one month during Year 2, but he could potentially incur costs for 23 months after Year 2. 

However, given the steady-state assumption, the potential Year 2 costs for 12 of these months 

would be accounted for by an individual entering in the twelfth month of Year 1, and the 

additional 11 months would be accounted for by another individual entering in the twelfth 

month of Year 0. (Notice that the second of these individuals does not exist, but the first may 

because WorkAdvance was well under way by the twelfth month of Year 1. Thus, most of the 

missing costs would be incurred late in the 24 months of eligibility and, consequently, would 

mainly consist of postemployment services.)

Under the assumption about the steady state, therefore, total aggregate costs in Year 2 

would account for all the costs incurred by all those entering WorkAdvance in that year, 

including the costs they incurred after the end of Year 2. Thus, if the assumption were valid, 

dividing aggregate costs for Year 2 by entrants in Year 2 would provide an estimate of costs per 

participant for those entering WorkAdvance during Year 2 over their 24 months of eligibility.

But in fact, Year 1 was nowhere near the steady state and Year 0 does not exist. Because the assumption is invalid, total costs in Year 2 are understated somewhat, particularly 

postemployment costs relative to the steady state. As mentioned in Chapter 4 and Appendix A, 

to adjust for this, Year 3 postemployment costs are used instead of Year 2 postemployment 

costs in computing costs per participant in the steady state. Year 3 is not used to represent the 

steady state because there was a break in program recruitment after random assignment ended 

in that year, leading to the understatement of other costs relative to the steady state. 
Appendix F 

Supplementary Exhibits for Chapter 5 

The WorkAdvance Study

Appendix Table F.1

Year 2 Impacts on Current Job Characteristics, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Employment status

Currently employed (%)

In targeted sector

Average number of months in 

current job

Earnings

Average total earnings per week ($)

Average hourly wage ($)

Currently employed and hourly wage

above $12 (%)

Currently employed and hourly wage

above $15 (%)

Hours

Average hours worked per week 

a Working full-time (%)

Work schedule and benefits

Average number of employer-provided

benefits offered 

Offered health plan or medical insurance 

through employer (%)

Obtained a unionized job (%)

Working regular shiftb (%)

74.4

48.2

5.6

460

12.26

51.9

38.9

27.7

56.7

2.3

39.3

10.5

59.9

64.4

16.2

5.1

341

9.25

35.2

24.8

23.8

46.1

1.9

33.6

9.9

46.7

10.0 ***

32.1 ***

0.5

119 ***

3.01 ***

16.6 ***

14.1 ***

3.8 **

10.5 **

0.4 *

5.6

0.6

13.2 ***

65.3

21.7

5.0

389

9.53

37.1

21.0

27.0

53.0

2.0

34.3

17.0

51.7

65.4

12.7

4.6

376

9.89

39.3

21.9

24.9

49.3

2.1

35.6

16.3

47.5

-0.1

9.1 **

0.4

13

-0.36

-2.2

-0.9

2.1

3.7

-0.1

-1.3

0.7

4.2

77.6

55.5

6.2

469

10.41

46.7

21.8

34.9

69.6

3.1

58.3

2.6

62.7

72.0

36.5

5.9

404

9.41

35.6

14.5

30.7

60.5

2.6

48.9

2.8

60.6

5.6

19.0 ***

0.3

64 **

1.00

11.1 ***

7.2 **

4.2 **

9.1 **

0.4 *

9.3 **

-0.1

2.2

64.5

37.8

4.8

275

7.16

22.7

5.4

24.3

45.4

2.0

35.6

6.9

52.9

58.2

23.1

5.3

249

6.40

17.7

7.9

21.7

38.3

1.8

33.0

8.3

41.4

6.3

14.7 ***

-0.5

26

0.76

5.0

-2.5

2.5

7.1 *

0.2

2.6

-1.4

11.6 ***

(continued)

279
Appendix Table F.1 (continued)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome (%) WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Very or somewhat satisfied with current

job 61.6

Advancement

Had a job before RA and employed at time

of survey interview and hourly wage

increased by $8.00 or more 16.1

Job skills at current job

Scope of work increased 42.2

Job title changed since job started 16.0

Offered many opportunities for career

advancement 56.4

Obtained new skills while working job 59.9

48.3

7.8

29.0

11.7

41.8

46.7

13.3 ***

8.3 ***

13.2 ***

4.2

14.6 ***

13.2 ***

53.8

7.8

27.3

12.0

49.3

49.6

51.8

7.9

29.7

13.7

47.8

38.3

2.1

-0.1

-2.5

-1.8

1.5

11.3 **

66.5

9.8

41.4

18.4

59.1

56.2

54.7

6.1

39.0

22.3

48.6

46.9

11.8 ***

3.7

2.5

-3.9

10.5 **

9.3 **

46.6

1.5

33.6

12.4

43.9

43.4

46.0

2.5

23.3

12.7

34.6

37.1

0.6

-1.0

10.3 ***

-0.3

9.2 **

6.3

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group; RA = random assignment.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall-Young adjusted p-values were used for 

categorical adjustment.

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 

follows: ††† = 1 percent; †† = 5 percent; † = 10 percent. No statistically significant differences were found.

Sample sizes may vary because of missing values.

a

Full-time employment is defined as working 35 hours per week or more.

bA regular shift includes those worked in the daytime, evening, or nighttime. 

280
The WorkAdvance Study

Appendix Table F.2

Additional Year 2 Impacts on Employment and Material Hardship, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Average number of months employed

or in a skills training program

All jobs

Average number of jobs obtained, among

those employed since RA 

One job (%)

Two jobs

Three jobs

Four or more jobs

Reasons for not working currently or

not having worked since RA (%)

Injury, illness, or disability

Pregnancy, childbirth, or family 

responsibilities

Going to school/attending training

Can't find a job, or looking for a

 job/better job

Transportation problems

Laid off

Incarcerated

Temporary, seasonal, or intermittent 

job completed

Never had a job

Other

Current or most recent job

Average number of months at job

Work out-of-state (%)

13.7

2.0

42.9

27.1

16.0

8.5

7.6

12.2

4.9

34.3

0.0

8.7

0.0

9.4

-0.1

23.1

6.8

8.9

12.3

1.9

38.5

34.5

13.4

4.0

8.1

1.1

24.6

33.7

0.0

9.4

0.0

11.0

1.2

10.9

6.8

7.7

1.4 ***

0.1

4.4

-7.4

2.7

4.5

-0.6

11.1

-19.7

0.6

0.0

-0.7

0.0

-1.6

-1.3

12.1

0.0

1.2

13.3

2.2

35.1

25.5

20.2

9.3

6.5

14.1

7.5

28.8

0.0

9.8

2.1

12.4

0.0

18.9

6.6

7.1

12.1

1.9

38.3

32.5

14.9

5.6

10.5

6.4

4.3

40.5

0.0

14.9

0.9

3.5

0.0

17.3

6.5

7.5

1.2 **

0.2

-3.2

-7.0

5.4

3.7

-4.0

7.7

3.2

-11.7

0.0

-5.0

1.2

8.9

0.0

1.6

0.1

-0.4

14.0

2.0

38.1

32.2

18.7

7.8

13.1

16.8

6.2

21.6

4.7

14.8

2.8

2.4

0.0

17.5

7.2

8.9

13.5

2.2

33.5

34.4

14.5

12.6

12.8

6.6

9.6

35.4

1.2

11.2

1.6

6.1

0.0

15.6

7.2

5.5

0.5

-0.2

4.6

-2.2

4.2

-4.7

0.4

10.2

-3.4

-13.8

3.6

3.5

1.3

-3.6

0.0

1.9

0.0

3.4

13.2

2.0

36.8

31.1

14.2

7.4

17.7

9.4

8.2

36.3

2.1

10.8

0.9

3.3

0.0

11.3

5.9

2.0

12.9

2.0

38.0

28.6

11.6

9.8

16.9

8.4

8.6

30.4

0.8

16.0

0.1

4.1

0.0

14.7

7.0

2.6

0.3

0.0

-1.1

2.4

2.6

-2.4

0.8

1.0

-0.4

5.9

1.3

-5.3

0.9

-0.8

0.0

-3.4

-1.0 *

-0.6

(continued)

281
Appendix Table F.2 (continued)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Employer size (%)

1-9 employees 14.4

10-49 employees 24.0

50-99 employees 10.5

100-499 employees 25.0

500 or more employees 26.2

Change in hours worked per week since RA (%)

Did not have a job before RA and/or a not employed at survey interview 37.4

Had a job before RA and employed

at survey interview and:

Hours worked increased by:

20 hours or more 11.9

10 to 19 hours 10.6

5 to 9 hours 5.4

Hours worked were within 5 hours 

of hours worked at baseline 20.7

Hours worked decreased 14.2

Benefits at current or most recent job

Employed at a unionized job (%) 10.6

Average number of employer-provided 

benefits offered 2.5

Main reason for not enrolling in the 

employer-provided health plan (%)

Covered by Medicaid 4.7

Covered by another insurance plan 36.8

The cost was too high 34.8

Had not worked long enough at 

employer 12.6

Some other reason 11.1

18.6

37.8

10.3

18.2

15.1

42.4

8.8

11.8

5.9

18.7

12.4

15.1

2.2

16.4

31.6

20.5

18.9

12.6

-4.2

-13.8

0.1

6.8

11.1

-5.1

3.0

-1.2

-0.5

2.0

1.7

-4.5

0.3

-11.7

5.2

14.3

-6.3

-1.4

16.3

23.2

25.0

23.9

11.6

45.6

13.8

8.2

7.7

16.8

7.9

20.5

2.2

31.1

11.2

31.1

25.9

0.7

16.4

31.4

11.9

23.2

17.1

40.6

5.6

10.4

7.1

19.3

17.0

19.6

2.3

32.2

28.3

12.9

20.9

5.8

0.0

-8.2

13.0

0.7

-5.5

††

5.0

8.3 **

-2.2

0.6

-2.6

-9.1 **

0.9

-0.1

-1.0

-17.1

18.2

5.0

-5.0

14.4

31.6

16.0

20.4

17.6

27.8

17.6

14.2

9.7

18.0

12.7

2.6

3.4

2.2

27.4

35.9

26.1

8.3

17.9

27.7

9.7

25.2

19.5

31.0

14.6

13.7

6.2

16.4

18.1

3.2

3.0

1.4

33.1

26.5

19.9

19.2

-3.5

3.9

6.3

-4.8

-1.9

-3.2

3.0

0.5

3.5

1.6

-5.4

-0.5

0.4

0.8

-5.7

9.5

6.3

-10.9

17.2

28.8

9.9

25.8

18.4

45.2

10.4

10.6

4.2

17.4

12.2

7.9

2.4

21.4

30.1

17.4

21.6

9.5

19.4

32.8

9.4

21.7

16.8

48.3

6.9

11.0

4.4

14.6

14.8

9.5

2.2

31.1

18.6

26.0

10.9

13.4

-2.2

-4.0

0.5

4.2

1.6

-3.1

3.5

-0.4

-0.2

2.8

-2.6

-1.6

0.2

-9.7

11.5

-8.6

10.7

-3.9

(continued)

282
Appendix Table F.2 (continued)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Material hardship (%)

Experienced the following material 

hardships:b

Did not pay the full amount of rent 

or mortgage

Evicted from home or apartment

Did not pay the full amount for gas, 

oil, or electricity

Gas, electrical, or oil services turned 

off or not delivered 

Phone service disconnected because 

payments were not made

Moved in with family or friends

Unable to afford visit to a doctor or 

dentist

19.1

0.9

9.2

1.9

10.9

16.7

24.7

23.2

2.8

16.9

4.0

18.4

14.2

28.3

-4.1

-1.9

-7.7 ***

-2.1

-7.5 **

2.5

-3.7

30.3

6.9

15.1

5.0

21.5

21.0

23.4

34.7

7.4

18.9

3.9

21.3

17.0

27.9

-4.4

-0.5

-3.8

1.0

0.2

4.0

-4.4

28.0

4.8

26.0

13.8

19.7

20.4

37.5

29.0

5.3

24.9

11.7

20.8

19.6

44.6

-1.0

-0.5

1.1

2.2

-1.1

0.8

-7.1 *

32.2

6.7

33.3

15.4

23.7

14.5

32.5

32.8

4.7

34.0

9.3

23.0

19.8

37.0

-0.5

2.1

-0.7

6.1 **

0.7

-5.3

-4.5

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group; RA = random assignment.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall-Young adjusted p-values were used 

for categorical measures.

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 

follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Sample sizes may vary because of missing values.

Italic type indicates that the metric is not among the full sample shown in the table. Therefore, the measure is nonexperimental and statistical tests were not 

performed. aSample members who did not work within two years before RA or who were not employed at the time of the survey interview are included in this 

category.

bWith the exception of respondents who indicated being unable to afford visit to a doctor or dentist, all material hardships were reported as having taken 

place within 12 months of the survey interview. 

283
The WorkAdvance Study

Appendix Table F.3

Year 2 Impacts on Household Composition, Housing Arrangement, and Health Care Coverage, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome (%) WA

Difference

C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Difference

WA C (Impact)

Marital status

Single, never married

Married and living with spouse

Married but living apart from spouse

Legally separated, divorced, or 

widowed

Living with a partner

Housing arrangement in prior month

Public housing

Section 8 housing vouchers

Reduced rent

Group shelter or homeless

Respondent had health care coverage 

in prior montha

Public

Employer-based

Other

75.2

14.5

2.8

7.6

14.2

5.9

3.2

46.4

0.3

65.5

29.5

21.1

16.1

74.4

17.4

2.7

5.5

11.3

8.0

4.7

42.6

1.2

71.1

39.5

16.3

16.3

0.8

-2.9

0.1

2.1

2.9

-2.1

-1.5

3.8

-0.9

-5.6

-10.1 **

4.7

-0.2

62.7

21.4

4.2

11.7

13.7

7.9

5.1

24.9

4.8

61.9

34.6

14.5

13.7

68.2

15.3

5.4

11.1

20.5

12.4

3.7

33.9

3.1

66.5

40.0

17.1

10.6

-5.5

6.1

-1.2

0.7

-6.8 *

-4.6

1.3

-9.0 *

1.7

-4.6

-5.4

-2.6

3.1

47.9

27.9

3.8

20.5

17.0

2.8

3.2

29.0

0.7

55.8

11.8

29.3

15.7

50.9

25.6

2.7

20.9

20.6

2.3

4.1

25.7

2.3

50.9

11.6

26.5

13.4

-3.0

2.4

1.0

-0.4

-3.6

0.5

-0.9

3.3

-1.6

4.9

0.2

2.8

2.3

67.6

14.6

3.0

14.9

17.7

4.2

10.7

20.1

2.3

73.5

44.4

17.4

13.4

70.4

13.4

4.2

12.0

11.6

7.0

7.4

29.3

1.0

71.0

45.0

14.0

13.3

-2.8

1.1

-1.2

2.9

6.1 **

-2.8

3.3

-9.2 ***

1.3

2.5

-0.6

3.4

0.1

Sample size (total = 2,058) 287 265 205 179 297 263 286 276

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance (program) group; C = control group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall-Young adjusted p-values were used 

for categorical measures.

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 

follows: ††† = 1 percent; †† = 5 percent; † = 10 percent. No statistically significant differences were found.

Sample sizes may vary because of missing values.

aRespondents were allowed to select being covered by more than one type of health insurance, so percentages may sum to more than 100 percent.

284
285

The WorkAdvance Study

Appendix Table F.4

Impacts on Employment and Earnings, by Random Assignment Sector,

at Toward

Full Sample

Difference

s Employment

Health Care

Difference

Manufacturing

Difference

Outcome WA C

Among respondents to the Year 2 Survey

(Impact) WA C (Impact) WA C (Impact) Sig.

All jobs (%)

Ever employed 89.2 87.5 1.6 89.6 87.9 1.7 88.5 87.3 1.3

Currently employed 64.4 58.2 6.2 66.0 59.8 6.2 63.0 56.1 6.9

Percentage of months

employeda 52.5 57.5 -5.1 * 50.2 56.5 -6.3 55.3 58.3 -3.0

Employed 6 or more

consecutive monthsa 65.2 68.3 -3.1 64.0 70.0 -6.0 66.8 66.1 0.7

Current or most recent job

Employed in

targeted sector (%) 50.1 33.7 16.4 *** 49.1 37.5 11.6 ** 50.7 29.8 20.9 ***

Average total earnings

per week ($) 379 360 20 357 315 42 402 412 -10

Average hourly wage ($) 10.11 9.78 0.33 10.39 9.19 1.21 9.81 10.45 -0.64 †

Average hours worked

per week 33.1 31.6 1.5 31.0 29.7 1.3 35.2 34.0 1.2

Sample size (total = 562) 286 276

Among full research sample, UI-covered jobs

149 146 137 130

Year 1

Ever employed (%) 83.6 78.8 4.8 * 82.1 80.4 1.7 84.2 78.2 6.0

Average quarterly

employment (%) 61.9 61.4 0.5 60.7 60.27 0.4 62.7 63.1 -0.3

Earnings ($) 9,500 9,478 23 8,228 8,218 9 10,597 10,948 -351

Year 2

Ever employed (%) 79.1 73.7 5.4 * 75.8 74.1 1.7 82.1 73.4 8.6 **

Average quarterly

employment (%) 67.6 62.5 5.1 * 64.8 65.41 -0.6 69.9 60.1 9.8 ** †

Earnings ($) 13,228 11,596 1,632 * 11,472 10,526 946 14,771 12,900 1,871

First quarter of Year 3

Ever employed (%) 68.9 64.1 4.8 72.3 61.8 10.5 ** 65.1 66.9 -1.8 †

Earnings ($) 3,318 3,142 176 3,180 2,777 403 3,448 3,528 -81

Sample size (total = 698) 349 349 173 178 176 171

(continued)
286

Appendix Table F.4 (continued)

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment 

insurance (UI) administrative records from the Ohio Department of Jobs and Family Services. 

NOTES: WA = WorkAdvance (program) group; C = control group.

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are 

indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Sample sizes may vary because of missing values.

a

Measures reported in time intervals cover only the first 18 months following each sample member's month of 

random assignment (the common follow-up period).
287

The WorkAdvance Study

Appendix Table F.5

Quarters 2 to 10 Impacts on Employment Stability and Duration, by Site

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Difference Difference Difference Difference

Outcome WA C (Impact) WA C (Impact) WA C (Impact) WA C (Impact)

Number of quarters worked 5.5 5.2 0.4 * 5.1 5.0 0.1 6.3 6.1 0.2 5.9 5.6 0.3

Employed entire follow-up period (%) 18.9 19.9 -1.0 15.1 20.9 -5.9 * 36.8 34.4 2.4 31.8 31.2 0.6

Average number of employers 2.2 2.0 0.2 * 2.6 2.2 0.4 ** 3.1 3.2 0.0 2.9 2.8 0.1

Number of employers (%)

Never employed

1 to 2

11.4

56.2

14.2

56.3

-2.8

-0.1

12.4

42.0

16.5

48.3

-4.1

-6.4

7.2

38.7

7.2

39.0

0.0

-0.3

7.5

48.1

13.4

41.4

††

-5.9 **

6.7

3 to 4 24.3 23.2 1.1 32.7 24.4 8.4 30.5 31.2 -0.7 23.7 25.6 -2.0

5 or more 8.1 6.3 1.8 12.9 10.8 2.1 23.6 22.6 1.0 20.8 19.6 1.1

Average number of quarters in first

employment spell a 5.5 5.1 0.3 5.0 5.0 -0.1 6.0 5.6 0.3 5.5 5.7 -0.2

Average first quarter of employment a (#) 2.5 2.5 0.0 2.3 2.4 -0.1 1.5 1.7 -0.2 2.0 2.0 0.1

Sample size (total = 2,564) 349 341 242 237 353 344 349 349

SOURCES: MDRC calculations from unemployment insurance (UI) administrative records provided by New York State Department of Labor for Per Scholas 

and St. Nicks Alliance sample members; Ohio Department of Job and Family Services for Towards Employment sample members; and Oklahoma 

Employment Security Commission for Madison Strategies Group sample members. 

NOTES: WA = WorkAdvance (program) group; C = control group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. The Westfall-Young adjusted p-values were used for 

categorical measures.

F-tests were also used to assess differences in the distribution of categorical measures across research groups. Statistical significance levels are indicated as 

follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Sample sizes may vary because of missing values.

aMeasures are among participants who worked in a UI-covered job within 10 quarters of random assignment. These measures are nonexperimental and 

statistical tests were not performed.
(continued)

The WorkAdvance Study

Appendix Figure F.1

Year 2 Impacts on Percentage Employed, by Month Relative to Random Assignment and Site

-20

0

20

40

60

80

100

RA2 *3 ***4 ***5 *67891011 **12 **13 ***14 **15 ***16 ***17 ***18 ***

Percentage

Month relative to random assignment

Per Scholas WA group

Control group

Impact

-20

0

20

40

60

80

100

RA *2 **34567891011 **12131415161718

Percentage

Month relative to random assignment

St. Nicks Alliance WA group

Control group

Impact

288
Appendix Figure F.1 (continued)

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: RA = random assignment; WA group = WorkAdvance (program) group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

-20

0

20

40

60

80

100

RA2345 *6 **789101112131415161718

Percentage

Month relative to random assignment

Madison Strategies Group WA group

Control group

Impact

-20

0

20

40

60

80

100

RA23 **45678910 *11121314 **15161718

Percentage

Month relative to random assignment

Towards Employment WA group

Control group

Impact

289
The WorkAdvance Study

Appendix Figure F.2

Year 2 Impacts on Average Monthly Earnings, by Month Relative to Random Assignment and Site

(continued)

-1,000

0

1,000

2,000

3,000

RA *2 *3 ***4 ***5 **67891011 **12 **13 **14 ***15 ***16 ***17 ***18 ***

Monthly earnings ($)

Month relative to random assignment

Per Scholas WA group

Control group

Impact

-1,000

0

1,000

2,000

3,000

RA234567891011 **12 *13 *14 *15 *16 *17 *18

Monthly earnings ($)

Month relative to random assignment

St. Nicks Alliance WA group

Control group

Impact

290
Appendix Figure F.2 (continued)

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: RA = random assignment; WA group = WorkAdvance (program) group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

-1,000

0

1,000

2,000

3,000

RA23456 *789101112131415161718

Monthly earnings ($)

Month relative to random assignment

Madison Strategies Group WA group

Control group

Impact

-1,000

0

1,000

2,000

3,000

RA23456789101112131415161718

Monthly earnings ($)

Month relative to random assignment

Towards Employment WA group

Control group

Impact

291
The WorkAdvance Study

Appendix Figure F.3

Year 2 Impacts on Percentage with Hourly Wage of $12 or More, by Month Relative to Random Assignment and Site

-20

0

20

40

60

80

100

RA23456 **7 ***8 ***9 ***10 ***11 ***12 ***13 ***14 ***15 ***16 ***17 ***18 ***

Percentage

Month relative to random assignment

Per Scholas WA group

Control group

Impact

-20

0

20

40

60

80

100

RA2345678910 **11 **12 *131415161718

Percentage

Month relative to random assignment

St. Nicks Alliance

WA group

Control group

Impact

292

(continued)
Appendix Figure F.3 (continued)

-20

0

20

40

60

80

100

RA23456789101112131415161718

Percentage

Month relative to random assignment

Madison Strategies Group

WA group

Control group

Impact

-20

0

20

40

60

80

100

RA23456789101112131415161718

Percentage

Month relative to random assignment

Towards Employment

WA group

Control group

Impact

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: RA = random assignment; WA group = WorkAdvance (program) group.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

293
294

The WorkAdvance Study

Appendix Figure F.4

in the Targeted Sector, by Relative Quarter, Site, and Random Assignment Sector

Impacts on Percentage Employed in an Unemployment Insurance-Covered Job

0

20

40

60

80

100

-4 -3 -2 -1 RA

**

2

***

3

***

4

***

5

***

6

***

7

***

8

***

9

***

10

***

Quarter relative to random assignment

Madison Strategies Group - Transportation and Manufacturing

WA group

Control group

0

20

40

60

80

100

-4 -3 -2 -1 RA 2 3 4 5 6 7 8 9 10

* Quarter relative to random assignment

Towards Employment - Health Care

WA group

Control group

Percentage

0

20

40

60

80

100

-4 -3 -2 -1 RA 2 3

***

4

***

5

***

6

***

7

***

8

***

9

**

10

** Quarter relative to random assignment

Towards Employment - Manufacturing

WA group Control group

Percentage Percentage

(continued)
295

Appendix Figure F.4 (continued)

SOURCES: MDRC calculations from unemployment insurance (UI) administrative records provided by Ohio 

Department of Job and Family Services for Towards Employment sample members and Oklahoma Employment 

Security Commission for Madison Strategies Group sample members. 

NOTES: RA = random assignment; WA group = WorkAdvance (program) group.

Sectors are defined by the North American Industry Classification System (NAICS) and are linked to 

employers. NAICS codes are not available in the UI records provided for sample members at Per Scholas and St. 

Nicks Alliance.

Transportation includes NAICS codes starting with 48-49, manufacturing includes NAICS codes starting with 

31-33, and health care includes NAICS codes starting with 62.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.
296

The WorkAdvance Study

Appendix Figure F.5

Quarters 1 to 10 Impacts on Percentage Receiving Unemployment Insurance Benefits,

by Relative Quarter and Site

0

20

40

60

80

100

RA 2 3 4 5 6

*

7

**

8

*

9

*

10

Quarter relative to random assignment

Per Scholas

WA group

Control group

0

20

40

60

80

100

RA 2

**

3 4 5 6 7 8 9 10

Quarter relative to random assignment

St. Nicks Alliance

WA group

Control group

Percentage

0

20

40

60

80

100

RA 2 3 4 5 6 7 8 9 10

Quarter relative to random assignment

Madison Strategies Group

WA group Control group

Percentage Percentage

(continued)
297

Appendix Figure F.5 (continued)

SOURCES: MDRC calculations from unemployment insurance (UI) benefits records provided by New York State 

Department of Labor for Per Scholas and St. Nicks Alliance sample members and Oklahoma Employment 

Security Commission for Madison Strategies Group sample members. 

NOTES: RA = random assignment; WA group = WorkAdvance (program) group.

UI benefits data are not available for Towards Employment.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Appendix G 

Further Analysis of Sources of Variation in the 

Economic Impacts 

301

In seeking reasons for the variation of WorkAdvance effects by site, Chapter 6 explores 

employment in the targeted sector and characteristics of those jobs, differences in impacts 

between the early and late cohorts of enrollees, and differences among the participants across 

the sites. Appendix Figure G.1 and Appendix Tables G.1 through G. 4 support the discussion of 

differences in participants across the sites. 

In terms of sample composition, the analysis in Chapter 6 found that the impacts of 

WorkAdvance are weaker for those who entered the program with more recent employment. 

Based on this result, if one provider served more participants who were recently employed, it 

might be expected that the impacts at that site would be weaker. This appendix explains two 

analyses that were conducted to understand the extent to which the variation in impacts across 

the sites is due to the characteristics of the individuals who were targeted. 

Conditional Subgroup Analysis

First, a regression model was estimated, which tried to assess whether the variation in site 

impacts is explained by variation in other characteristics.1

 This analysis focused on earnings in 

Year 2 as a key outcome; the results are shown in Appendix Table G.5. The site impacts were 

conditioned on a series of control variables, which assessed whether the impacts varied by labor 

market attachment status, age, gender, previous incarceration or conviction status, and education. The first set of terms in Appendix Table G.5 show relationships between various baseline 

measures and Year 2 earnings. These are not the main estimates of interest in this scenario. The 

key terms of interest in Appendix Table G.5 are the interaction terms in the second part of the 

table, which show how various baseline characteristics moderate the effects of WorkAdvance. 

The first several interaction terms show the site and cohort impacts. These are followed by 

interactions that control for how the impacts of WorkAdvance varied by the characteristics 

mentioned above. The key issue in this analysis is whether site or background characteristics are 

significant after controlling for both factors simultaneously. If the site impacts are significant, it 

suggests that program factors (that is, implementation factors or other aspects of the site that are 

not controlled for in the analysis) are the dominant source of variation. If the background 

characteristics are significant, but the site interactions are insignificant, this implies that the 

composition of the sample, rather than implementation factors, is the key driver of the impacts.

The results of Appendix Table G.5 indicate clearly that it is the site factors (that is, implementation factors) that are dominant. After controlling for multiple interactions between 

baseline characteristics and being a WorkAdvance group member, the impacts at Per Scholas 

 1

This type of analysis is known as a conditional subgroup analysis. This analysis involves a series of regressions that test for “interactions” between whether someone was assigned to the WorkAdvance group and 

other baseline characteristics (such as age). 
302

and for the late cohorts at Madison Strategies Group and Towards Employment are statistically 

significant. The stronger impacts for the semiattached and long-term unemployed seen in 

Chapter 6 are reduced to statistical insignificance, suggesting that those effects are due to site 

factors, rather than simply long-term unemployment. As discussed in the report, this result does 

not take away from the fact that WorkAdvance was able to increase earnings and employment 

for this group, but it does imply that the stronger effects for this group are related to the fact that 

there were more long-term unemployed at the sites with larger impacts.

One troubling result in Appendix Table G.5 is that after adjusting for site and other 

baseline characteristics, the impacts of WorkAdvance are significantly weaker among the 

previously convicted or incarcerated. Helping this population into work is a priority for workforce development professionals, and it is disappointing that WorkAdvance was, on average,

unable to do so.

Compositional Alignment Analysis

A second analysis tried to align the characteristics of the samples across the sites using weights. 

For this exercise, Per Scholas was chosen as the benchmark site because of the larger impacts 

produced by that program. Sample members at the Per Scholas site were more likely to be male, 

Hispanic, and younger, to have no children, to have more education, and to have higher earnings in the three years before study entry compared with the other sites. Per Scholas sample 

members were also less likely to have been employed at study entry, to have had a previous 

certification in the targeted sector, and to have worked steadily in the three years before study 

entry.

This analysis used matching methods in order to identify the individuals at Per Scholas 

who had the most similar characteristics to sample members at the other three sites.2

 Impacts 

were then estimated for this group, suggesting what the impacts of WorkAdvance would have 

been had Per Scholas participants shared baseline characteristics more similar to participants at 

the other sites. The same exercise was then run in reverse: A model was built to estimate 

 2

This analysis involved several steps. First, a probability model was developed that used baseline variables 

to predict being a participant at St. Nicks Alliance, Madison Strategies Group, or Towards Employment. This 

model was then applied to the Per Scholas site in order to identify the participants who looked most similar to 

participants from the other three sites. The top third of the sample at Per Scholas (in terms of similarity to the 

sample from the other three sites) was selected. This can also be thought of as the top third most “non-Per 

Scholas-like” participants at Per Scholas. Impacts were then estimated for this subgroup. This analysis is 

experimental because it uses only information from baseline survey and administrative records data, which are

exogenous to the treatment. Further analysis used the assignment probability as a weight and found similar 

results (not shown). The analysis implicitly assumes, however, that the role of characteristics in moderating the 

impacts is the same across sites, which is not necessarily the case.
303

impacts on participants at the other three sites who were most similar on baseline characteristics

to Per Scholas sample members. The results of this analysis are shown in Appendix Table G.6. 

Overall, there is no clear pattern to the results. The estimated effects on earnings and employment are not statistically significant among Per Scholas sample members who are more similar 

to the samples at the other three sites. There is some limited evidence that impacts are stronger 

(although not always statistically significant) for a few measures among sample members at the 

other three sites who have characteristics similar to those at Per Scholas, but the results are 

inconsistent.

* * *

The overall assessment from these two analyses is that the differences in impacts across the sites 

were not due to differences in the composition of the samples, but rather to differences in 

implementation or other factors.
304

The WorkAdvance Study

Appendix Figure G.1

20%

8%

80%

Did not start training in targeted sector

12%

Did not obtain 

targeted sector 

credential

Activity Flow in Targeted Sector, Among Control Group Respondents, by Site

4% 4% 11%

Per Scholas (N = 265)

Started training in 

targeted sector

Obtained 

targeted 

sector 

credential

Obtained 

job in 

targeted 

sectora

Obtained 

job in 

targeted 

sectora

Obtained job 

in targeted 

sectora

St. Nicks Alliance (N = 179)

22%

18%

78%

Did not start training in targeted sector

6% 12%

Started training in 

targeted sector

Obtained 

job in 

targeted 

sectora

Obtained job in 

targeted sectora

3%

Did not obtain 

targeted sector 

credential

Obtained 

targeted 

sector 

credential 

(continued)
305

Appendix Figure G.1 (continued)

Madison Strategies Group (N = 263)

13%

7%

86%

Did not start training in targeted sector

6%

Did not 

obtain 

targeted 

sector 

credential

6% 39%

Started 

training in 

targeted sector

Obtained 

targeted 

sector 

credential

Obtained 

job in 

targeted 

sectora

Obtained 

job in 

targeted 

sectora

Obtained job in targeted sectora

3%

Towards Employment (N = 276)

16%

10%

83%

Did not start training in targeted sector

6%

Did not obtain 

targeted sector 

credential

6% 25%

Started 

training in 

targeted sector

Obtained 

targeted 

sector 

credential

Obtained job in targeted 

sectora

2%

Obtained 

job in 

targeted 

sectora

Obtained 

job in 

targeted 

sectora

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: Sample sizes vary because of missing values. Measures may not sum to 100 percent because of missing 

values.

 a

Jobs in targeted sector are control group members’ current or most recent jobs only.
306

The WorkAdvance Study

Appendix Table G.1

Targeted Sector Job Characteristics,

Among Control Group Survey Respondents Employed Since Random Assignment

Outcome (%)

Per Scholas

Targeted Nonsector

Sector Job Joba

St. Nicks Alliance Madison Strategies Group

Targeted Nonsector

Sector Job Joba

Towards Employment

Targeted Nonsector

Sector Job Joba

Targeted Nonsector

Sector Job Joba

Job characteristics

Hourly wage above $15

b Working full time

Offered health plan or medical insurance 

through employer

Working regular shift c

Very or somewhat satisfied with job

Advancement

Had a job before RA and employed at time

of survey interview and hourly wage

increased by $8.00 or more

Job skills at current or most recent job

Scope of work increased

Offered many opportunities for career

advancement

Obtained new skills while working job

49.8

73.5

42.1

74.6

81.1

27.0

52.8

62.7

69.8

29.0

67.5

44.3

69.0

66.6

11.5

40.5

57.0

66.9

31.6

82.3

44.0

59.4

69.2

12.6

42.8

55.7

62.2

28.5

70.6

46.1

72.7

75.7

15.1

45.2

69.1

61.9

22.4

87.5

68.7

81.7

74.0

12.6

54.5

60.1

68.6

14.6

80.7

50.9

83.7

72.6

5.4

49.0

58.6

56.9

13.8

76.3

58.6

81.0

79.6

5.4

37.7

53.7

66.7

11.7

52.9

40.6

69.0

72.0

4.9

40.7

54.1

62.5

Sample size (total = 892) 52 186 34 130 128 120 90 152

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: RA = random assignment.

Statistics are among control group respondents who indicated that they had worked for pay since random assignment. Job characteristics refer to the current 

or most recent job.

Sample sizes may vary because of missing values. a

Nonsector job is any job that is not in the sector(s) targeted by the WorkAdvance provider.

bFull time is considered working 35 hours or more per week. 

c

A regular shift includes those worked in the daytime, evening, or nighttime. 
307

The WorkAdvance Study

Appendix Table G.2

Adjusted Versus Unadjusted Differences in Targeted Sector Versus Non-Targeted Sector Employment

 and Advancement, Among WorkAdvance Group Survey Respondents Employed Since Random Assignment

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Outcome (%) Adjusted Unadjusted Adjusted Unadjusted Adjusted Unadjusted Adjusted Unadjusted

Job characteristics

Hourly wage above $15 a Working full time

Offered health plan or medical insurance 

through employer

Working regular shift b

Very or somewhat satisfied with job

Advancement

Had a job before RA and employed at time

of survey interview and hourly wage

increased by $8.00 or more

Job skills at current or most recent job

Scope of work increased

Offered many opportunities for career

advancement

Obtained new skills while working job

22.1

15.9

15.6

6.8

13.6

12.2

17.2

3.9

17.5

25.0

16.3

15.7

9.0

14.6

12.1

19.4

6.4

22.6

21.2

14.3

5.4

3.6

26.6

24.7

19.4

8.1

19.1

19.2

13.1

-0.2

1.9

22.5

20.5

16.7

2.4

19.9

10.3

28.5

36.7

11.6

10.3

0.6

20.6

29.0

21.2

15.2

30.5

36.6

9.6

12.0

-1.4

19.2

28.8

19.9

2.7

27.0

25.1

18.1

2.7

2.1

27.2

13.0

14.7

3.0

27.7

26.7

18.3

1.9

1.9

25.9

12.6

15.6

Sample size (total = 992) 272 183 285 252

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: RA = random assignment.

Statistics are among WorkAdvance (program) group survey respondents who indicated that they had worked for pay since random assignment. Job 

characteristics refer to the current or most recent job.

Sample sizes may vary because of missing values. a

Full time is considered working 35 hours or more per week. 

bA regular shift includes those worked in the daytime, evening, or nighttime. 
308

The WorkAdvance Study

Appendix Table G.3

Targeted Sector Job Characteristics,

Among WorkAdvance Group Survey Respondents Employed Since Random Assignment with High Predicted Earnings

Outcome (%)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Targeted Nonsector

Sector Job Joba

Targeted Nonsector

Sector Job Joba

Targeted Nonsector

Sector Job Joba

Targeted Nonsector

Sector Job Joba

Job characteristics

Hourly wage above $15

b Working full time

Offered health plan or medical insurance 

through employer

Working regular shift c

Very or somewhat satisfied with job

Advancement

Had a job before RA and employed at time

of survey interview and hourly wage

increased by $8.00 or more

Job skills at current or most recent job

Scope of work increased

Offered many opportunities for career

advancement

Obtained new skills while working job

62.7

79.2

54.4

84.0

88.4

32.1

63.7

75.9

85.1

40.3

61.4

34.8

76.8

72.7

21.2

52.3

69.7

70.0

48.2

88.1

64.4

71.6

96.6

59.7

61.4

73.4

84.5

19.7

80.0

46.1

81.5

68.7

-0.5

38.8

60.4

71.6

32.5

95.3

87.2

85.1

85.9

14.2

53.8

79.5

76.7

29.5

70.2

53.4

64.3

87.1

17.8

43.0

55.8

65.4

16.0

82.0

69.4

88.2

75.7

2.0

64.2

75.4

66.4

14.4

61.8

47.7

72.9

68.1

-0.3

17.2

52.6

60.7

Sample size (total = 522) 132 61 27 45 101 39 70 47

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance; RA = random assignment.

Statistics are among WorkAdvance group respondents who indicated that they had worked for pay since random assignment with predicted earnings in the 

first quarter of Year 3 above the 50th percentile. Job characteristics refer to the current or most recent job.

Sample sizes may vary because of missing values. a

Nonsector job is any job that is not in the sector(s) targeted by the WorkAdvance service provider.

bFull time is considered working 35 hours or more per week. 

c

A regular shift includes those worked in the daytime, evening, or nighttime. 
309

The WorkAdvance Study

Appendix Table G.4

Targeted Sector Job Characteristics

Among WorkAdvance Group Survey Respondents Employed Since Random Assignment with Low Predicted Earnings

Outcome (%)

Per Scholas St. Nicks Alliance Madison Strategies Group Towards Employment

Targeted Nonsector

Sector Job Joba

Targeted Nonsector

Sector Job Joba

Targeted Nonsector

Sector Job Joba

Targeted Nonsector

Sector Job Joba

Job characteristics

Hourly wage above $15

b Working full time

Offered health plan or medical insurance 

through employer

Working regular shift c

Very or somewhat satisfied with job

Advancement

Had a job before RA and employed at time

of survey interview and hourly wage

increased by $8.00 or more

Job skills at current or most recent job

Scope of work increased

Offered many opportunities for career

advancement

Obtained new skills while working job

42.5

78.9

48.4

78.3

76.2

25.2

50.8

63.6

80.5

16.0

59.7

25.0

58.1

71.8

10.1

21.0

59.9

52.3

48.9

88.0

33.4

90.6

90.9

15.6

49.9

76.0

84.2

23.0

67.6

35.2

74.2

66.0

9.6

31.5

64.6

61.3

20.7

96.7

72.7

84.4

83.9

11.6

61.1

77.5

78.3

11.0

61.3

33.9

76.8

63.2

14.3

26.7

41.5

48.4

9.4

79.3

51.0

86.6

63.5

11.5

55.1

62.1

71.9

2.5

42.2

19.7

68.8

66.1

-0.9

40.0

55.4

50.4

Sample size (total = 470) 45 34 39 72 88 57 73 62

SOURCE: MDRC calculations from responses to the WorkAdvance Year 2 Survey.

NOTES: WA = WorkAdvance; RA = random assignment.

Statistics are among WorkAdvance group respondents who indicated that they had worked for pay since random assignment with predicted earnings in the 

first quarter of Year 3 below or equal to the 50th percentile. Job characteristics refer to the current or most recent job.

Sample sizes may vary because of missing values. aNonsector job is any job that is not in the sector(s) targeted by the WorkAdvance service provider.

bFull time is considered working 35 hours or more per week. 

cA regular shift includes those worked in the daytime, evening, or nighttime. 
310

The WorkAdvance Study

Appendix Table G.5

Conditional Subgroup Impacts on Year 2 Earnings

Characteristic Parameter Estimate ($)

Baseline covariates

Female

Black 

Hispanic

Age

Under age 24

Number of children living with respondent

Born in state

Highest level of education

Already has a license/certificate in targeted sector

Currently employed at baseline

Monthly family income

Employed in at least 1 quarter in year prior to RA

Employed in all 4 quarters in year prior to RA

Number of quarters employed in 3 years prior to RA

Total earnings in quarter prior to RA

Total earnings in 3 years prior to RA

Randomly assigned in 2011

Randomly assigned in 2012

Sample member at Per Scholas

Sample member at Towards Employment

Sample member at Madison Strategies Group

Interactions with research group status

WorkAdvance group member at Per Scholas

WorkAdvance group member at Towards Employment

WorkAdvance group member at Madison Strategies Group

WorkAdvance group member in late cohort at Per Scholas

WorkAdvance group member in late cohort at Towards Employment

WorkAdvance group member in late cohort at Madison Strategies Group

WorkAdvance group member and long-term unemployed

WorkAdvance group member and semiattached to labor market

WorkAdvance group member and female

WorkAdvance group member and under age 24

WorkAdvance group member and previous convicted or incarcerated

WorkAdvance group member and at least some college

R-squared

F statistic

P-value of F statistic

-2,714.10 ***

-1,676.82 **

-16.07

-122.51 ***

-1,296.70

-164.60

606.13

852.25 ***

877.14

2,786.69 ***

1.54

1,096.10

101.45

-213.53

0.17

0.15

-3,317.77 ***

-1,793.81 ***

-1,071.84

-1,177.76

-64.10

3,260.68 **

-309.50

451.61

924.56

3,081.64 *

3,141.66 **

810.47

581.73

240.80

-887.16

-2,204.29 **

577.82

0.1705

14.81

< 0.0001

(continued)
311

Appendix Table G.5 (continued)

SOURCE: MDRC calculations from the WorkAdvance baseline information form and unemployment 

insurance administrative records from New York State Department of Labor, Ohio Department of Jobs 

and Family Services, and Oklahoma Employment Security Commission.

NOTES: RA = random assignment.

The late cohort includes sample members randomly assigned in or after Quarter 4, 2012.

The semiattached subgroup consists of sample members who were unemployed for one to six months 

before random assignment. The long-term unemployed subgroup consists of sample members who have 

never been employed or who were unemployed for seven or more months before random assignment.

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.
32712

The WorkAdvance Study

Appendix Table G.6

Quarters 2 to 10 Impacts on Unemployment Insurance-Covered Employment and Earnings 

for Sample Members Demographically Similar to Per Scholas Sample Members

Per Scholas Sample Members All Sample Members Non-Per Scholas Sample

All Per Scholas Most Similar to Sample Except Those Members Most Similar to

a b Sample Members Members at Other Sites at Per Scholas Per Scholas Sample Members

Difference Difference Difference Difference

Outcome WA C (Impact) WA C (Impact) WA C (Impact) WA C (Impact)

Year 1 (Quarters 2 to 5)

Ever employed (%) 77.1 71.0 6.1 * 79.0 69.6 9.4 84.4 79.5 4.8 *** 85.7 76.3 9.4 ***

UI earnings ($) 8,868 8,718 150 7,577 9,095 -1,518 10,813 10,867 -55 11,464 10,429 1,035

Year 2 (Quarters 6 to 9)

Ever employed (%) 82.4 76.4 6.0 ** 79.7 76.9 2.8 78.3 76.7 1.7 77.8 76.1 1.7

UI earnings ($) 18,217 14,471 3,747 *** 16,288 14,093 2,194 14,731 13,534 1,196 * 15,888 13,987 1,901

First quarter of Year 3

Ever employed (%) 70.4 64.3 6.0 * 64.0 57.8 6.2 65.1 63.1 2.1 61.9 62.4 -0.4

UI earnings ($) 5,385 4,096 1,289 *** 4,516 3,673 844 3,661 3,534 128 3,864 3,842 22

Sample size 349 341 108 126 944 930 309 328

SOURCES: MDRC calculations from unemployment insurance (UI) administrative records from New York State Department of Labor, Ohio Department 

of Jobs and Family Services, and Oklahoma Employment Security Commission.

NOTES: WA = WorkAdvance (program) group; C = control group.

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. Statistical significance levels are indicated as 

follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Sample sizes may vary because of missing values.

aThis panel shows impacts among the top third of the sample in terms of similarity with sample members at Madison Strategies Group, Towards 

Employment, and St. Nicks Alliance. 

bThis panel shows impacts among the top third of the sample in terms of similarity with Per Scholas sample members.
313

The WorkAdvance Study

Appendix Table G.7

Impacts on Subgroups Defined by Age, All Sites Combined

Full sample 18 to 24 25 to 34 35 or older

Difference Difference Difference Difference

Outcome WA C (Impact) WA C (Impact) WA C (Impact) WA C (Impact) Sig.

Among respondents to the Year 2 Survey

Currently employed (%) 70.6 65.2 5.3 *** 72.2 67.8 4.5 75.9 66.3 9.6 *** 65.2 62.9 2.3

Weekly earnings ($) 502 459 43 *** 464 436 28 547 477 69 *** 482 464 18

Sample size (total = 2,058) 1,075 983 265 228 361 366 449 389

Among full research sample

Year 1

Ever employed in a UI-covered job (%) 82.4 77.2 5.2 *** 88.3 82.6 5.7 ** 82.7 78.1 4.6 * 78.7 73.4 5.3 **

UI earnings ($) 10,295 10,284 11 9,535 9,144 392 10,931 10,932 -2 10,180 10,406 -226

Year 2

Ever employed in a UI-covered job (%) 79.5 76.5 2.9 * 85.8 81.6 4.2 79.7 80.5 -0.8 75.6 70.1 5.4 **

UI earnings ($) 15,713 13,744 1,969 *** 13,940 12,296 1,644 * 17,792 14,750 3,042 *** 14,931 13,838 1,093

First quarter of Year 3

Ever employed in a UI-covered job (%) 66.7 63.3 3.4 * 72.8 63.2 9.6 ** 66.4 67.6 -1.2 63.5 59.2 4.3 †

UI earnings ($) 4,143 3,667 476 *** 3,933 3,098 835 *** 4,645 4,017 628 ** 3,848 3,696 152

Sample size (total = 2,564) 1,293 1,271 320 291 433 458 540 522

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from New 

York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group.

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. Statistical significance levels are indicated as 

follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: ††† = 1 percent; †† = 5 

percent; † = 10 percent.

Sample sizes may vary because of missing values.
314

The WorkAdvance Study

Appendix Table G.8

Impacts on Subgroups Defined by Level of Education, All Sites Combined

Full sample High school, GED, or less Some college or more

Difference Difference Difference

Outcome WA C (Impact) WA C (Impact) WA C (Impact) Sig.

Among respondents to the Year 2 Survey

Currently employed (%) 70.6 65.2 5.3 *** 66.9 61.6 5.3 * 73.3 68.1 5.2 **

Weekly earnings ($) 502 459 43 *** 467 435 32 528 478 50 ***

Sample size (total = 2,058) 1,075 983 462 429 610 553

Among full research sample

Year 1

Ever employed in a UI-covered job (%) 82.4 77.2 5.2 *** 81.7 74.4 7.4 *** 83.0 79.4 3.6 * 

UI earnings ($) 10,295 10,284 11 9,414 9,102 312 10,948 11,225 -277

Year 2

Ever employed in a UI-covered job (%) 79.5 76.5 2.9 * 77.7 75.6 2.0 81.0 77.0 4.0 * 

UI earnings ($) 15,713 13,744 1,969 *** 13,694 12,426 1,268 17,221 14,816 2,405 ***

First quarter of Year 3

Ever employed in a UI-covered job (%) 66.7 63.3 3.4 * 63.5 61.4 2.1 69.1 64.8 4.3 * 

UI earnings ($) 4,143 3,667 476 *** 3,594 3,392 202 4,570 3,879 691 ***

Sample size (total = 2,564) 1,293 1,271 564 554 729 717

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from New 

York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group; GED = General Educational Development certificate.

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. Statistical significance levels are indicated as 

follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: ††† = 1 percent; †† = 5 

percent; † = 10 percent.

Sample sizes may vary because of missing values.
315

The WorkAdvance Study

Appendix Table G.9

Impacts on Subgroups Defined by Predicted Earnings in First Quarter of Year 3, All Sites Combined

Full sample Less than 25th percentile

Difference

25th to 75th percentile

Difference

Greater than 75th percentile

Difference Difference

Outcome WA C (Impact) WA C (Impact) WA C (Impact) WA C (Impact) Sig.

Among respondents to the Year 2 Survey

Currently employed (%) 70.6 65.2 5.3 *** 54.9 48.1 6.9 71.6 65.2 6.3 ** 83.9 81.1 2.7

Weekly earnings ($) 502 459 43 *** 358 367 -9 501 446 55 *** 641 574 66 **

Sample size (total = 2,058) 1,075 983 274 228 526 502 275 253

Among full research sample

Year 1

Ever employed in a UI-covered job (%) 82.4 77.2 5.2 *** 75.2 68.1 7.1 ** 81.2 77.6 3.7 * 92.2 85.4 6.8 ***

UI earnings ($) 10,295 10,284 11 5,997 5,885 113 9,616 9,572 43 16,147 15,905 242

Year 2

Ever employed in a UI-covered job (%) 79.5 76.5 2.9 * 70.7 60.9 9.8 *** 78.8 79.4 -0.7 90.2 85.4 4.8 * ††

UI earnings ($) 15,713 13,744 1,969 *** 8,256 7,833 423 14,527 12,784 1,743 ** 25,733 21,449 4,284 *** †

First quarter of Year 3

Ever employed in a UI-covered job (%) 66.7 63.3 3.4 * 57.2 48.5 8.7 ** 64.5 64.4 0.1 81.0 75.0 6.0 *

UI earnings ($) 4,143 3,667 476 *** 2,157 1,966 191 3,832 3,428 403 * 6,729 5,881 848 **

Sample size (total = 2,564) 1,293 1,271 334 307 639 643 320 321

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from New 

York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group.

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. Statistical significance levels are indicated as 

follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: ††† = 1 percent; †† = 5 

percent; † = 10 percent.

Sample sizes may vary because of missing values.
316

The WorkAdvance Study

Appendix Table G.10

Impacts on Subgroups Defined by Level of Earnings in Three Years Before Study, All Sites Combined

Full sample Less than $10,000 $10,000-30,000 $30,000 or more

Difference Difference Difference Difference

Outcome WA C (Impact) WA C (Impact) WA C (Impact) WA C (Impact) Sig.

Among respondents to the Year 2 Survey

Currently employed (%) 70.6 65.2 5.3 *** 63.1 53.5 9.6 *** 71.9 63.9 8.0 * 79.6 82.4 -2.8 ††

Weekly earnings ($) 502 459 43 *** 434 420 14 501 429 72 *** 595 535 60 **

Sample size (total = 2,058) 1,075 983 463 422 262 253 347 307

Among full research sample

Year 1

Ever employed in a UI-covered job (%) 82.4 77.2 5.2 *** 75.1 65.7 9.4 *** 84.5 85.2 -0.7 91.4 87.1 4.3 ** ††

UI earnings ($) 10,295 10,284 11 6,649 6,524 125 10,589 10,116 473 15,235 15,835 -600

Year 2

Ever employed in a UI-covered job (%) 79.5 76.5 2.9 * 69.5 66.4 3.1 84.3 82.6 1.8 89.9 86.1 3.8

UI earnings ($) 15,713 13,744 1,969 *** 10,795 9,448 1,347 * 16,011 12,902 3,109 *** 22,588 20,479 2,109 *

First quarter of Year 3

Ever employed in a UI-covered job (%) 66.7 63.3 3.4 * 56.8 50.3 6.5 ** 70.1 69.6 0.5 78.4 76.3 2.1

UI earnings ($) 4,143 3,667 476 *** 2,909 2,428 480 ** 4,034 3,569 465 6,031 5,481 550

Sample size (total = 2,564) 1,293 1,271 570 562 320 320 403 389

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from New 

York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group.

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. Statistical significance levels are indicated as 

follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: ††† = 1 percent; †† = 5 

percent; † = 10 percent.

Sample sizes may vary because of missing values.
317

The WorkAdvance Study

Appendix Table G.11

Impacts on Subgroups Defined by Previous Conviction or Incarceration Status, All Sites Combined

Full samplea Previously convicted or incarcerated

Difference

Not previously convicted or incarcerated

Difference Difference

Outcome WA C (Impact) WA C (Impact) WA C (Impact) Sig.

Among respondents to the Year 2 Survey

Currently employed (%) 70.5 65.3 5.2 *** 66.4 62.0 4.4 71.8 66.6 5.2 **

Weekly earnings ($) 501 459 42 *** 515 480 35 495 454 41 **

Sample size (total = 2,058) 1,074 983 266 269 808 714

Among full research sample

Year 1

Ever employed in a UI-covered job (%) 82.5 77.3 5.2 *** 79.7 75.2 4.4 83.4 78.2 5.2 ***

UI earnings ($) 10,302 10,292 9 9,264 8,892 373 10,634 10,877 -242

Year 2

Ever employed in a UI-covered job (%) 79.5 76.6 2.9 * 75.1 73.4 1.6 81.1 77.8 3.3 *

UI earnings ($) 15,713 13,759 1,954 *** 13,140 12,137 1,003 16,566 14,451 2,115 ***

First quarter of Year 3

Ever employed in a UI-covered job (%) 66.7 63.3 3.4 * 56.5 59.0 -2.5 70.3 65.0 5.3 ** †

UI earnings ($) 4,139 3,672 467 *** 3,157 3,231 -74 4,472 3,857 615 *** †

Sample size (total = 2,564) 1,291 1,270 335 359 956 911

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from New 

York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group.

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. Statistical significance levels are indicated as 

follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: ††† = 1 percent; †† = 5 

percent; † = 10 percent.

Sample sizes may vary because of missing values.

aFull sample includes all sample members included in one of the subgroups. Three sample members are missing a previous conviction or incarceration status 

and therefore are not included in the full sample.
318

The WorkAdvance Study

Appendix Table G.12

Impacts on Subgroups Defined by Race/Ethnicity, All Sites Combined

Latino/Hispanic White Black/African-American

Difference Difference Difference

Outcome WA C (Impact) WA C (Impact) WA C (Impact) Sig.

Among respondents to the Year 2 Survey

Currently employed (%)

Weekly earnings ($)

77.7

526

67.2

463

10.5 **

63 *

75.2

545

72.8

536

2.3

9

67.5

460

61.5

427

6.0 **

33 * 

Sample size (total = 1,762) 168 172 192 176 557 497

Among full research sample

Year 1

Ever employed in a UI-covered job (%)

UI earnings ($)

81.6 72.1

11,309 9,267

9.5 **

2,042 *

83.3 82.5

12,079 12,469

0.9

-389

80.9 76.4

8,626 9,156

4.5 **

-530 †

Year 2

Ever employed in a UI-covered job (%)

UI earnings ($)

83.6 76.7

18,835 14,688

7.0 *

4,147 ***

78.2 81.7

17,307 15,145

-3.5

2,162 *

79.6 73.6

13,863 12,185

6.0 ***

1,678 **

†

First quarter of Year 3

Ever employed in a UI-covered job (%)

UI earnings ($)

71.4 66.4

5,170 4,135

5.1

1,035 **

64.2 64.8

4,048 3,858

-0.6

190

67.4 61.7

3,824 3,252

5.6 **

572 ***

Sample size (total = 2,180) 211 216 235 233 655 630

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment insurance (UI) administrative records from 

New York State Department of Labor, Ohio Department of Jobs and Family Services, and Oklahoma Employment Security Commission.

NOTES: WA = WorkAdvance (program) group; C = control group.

Fifteen percent of sample members (1) identified as non-Hispanic and listed "Asian," "American Indian," or "Other" as their race; (2) answered as 

"multiracial"; or (3) did not provide a race; these sample members are not shown in this table.

A two-tailed t-test was applied to the differences between outcomes for the WorkAdvance and control groups. Statistical significance levels are indicated 

as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are indicated as follows: ††† = 1 percent; †† = 5 

percent; † = 10 percent.

Sample sizes may vary because of missing values.
319

The WorkAdvance Study

Appendix Table G.13

Impacts for Subgroups Defined by Random Assignment Cohort,

Per Scholas and St. Nicks Alliance (Training-First Only Sites) Combined

Full sample Early cohort Late cohort

Difference Difference Difference

Outcome WA C (Impact) WA C (Impact) WA C (Impact) Sig.

Among respondents to the Year 2 Survey

Currently employed (%) 70.2 65.3 4.9 68.6 63.2 5.5 71.5 67.8 3.7

Weekly earnings ($) 544 480 65 *** 559 471 87 ** 524 493 31

Sample size (total = 936) 492 444 258 220 234 224

Among full research sample

Year 1

Ever employed in a

UI-covered job (%) 77.5 70.8 6.7 *** 74.4 66.5 7.9 ** 81.1 76.0 5.0

UI earnings ($) 9,074 9,110 -36 8,605 7,744 861 9,588 10,782 -1,194 †

Year 2

Ever employed in a

UI-covered job (%) 79.2 74.6 4.6 * 78.2 72.4 5.9 * 80.1 77.4 2.6

UI earnings ($) 16,701 14,333 2,368 *** 15,433 12,447 2,986 ** 17,996 16,777 1,220

First quarter of Year 3

Ever employed in a

UI-covered job (%) 66.3 63.8 2.5 65.3 60.0 5.2 66.9 69.1 -2.2

UI earnings ($) 4,808 4,095 713 ** 4,395 3,486 909 ** 5,214 4,901 314

Sample size (total = 1,169) 591 578 316 316 275 262

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment 

insurance (UI) administrative records from the New York State Department of Labor. 

NOTES: WA = WorkAdvance (program) group; C = control group.

The early cohort includes all sample members randomly assigned through Quarter 3, 2012. The late cohort 

includes all sample members randomly assigned in or after Quarter 4, 2012.

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are 

indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Sample sizes may vary because of missing values.
320

The WorkAdvance Study

Appendix Table G.14

Impacts for Subgroups Defined by Random Assignment Cohort,

Towards Employment and Madison Strategies Group (Dual-Track Sites) Combined

Full sample Early cohort Late cohort

Difference Difference Difference

Outcome WA C (Impact) WA C (Impact) WA C (Impact) Sig.

Among respondents to the Year 2 Survey

Currently employed (%) 70.9 65.2 5.6 ** 64.3 59.4 4.9 77.1 70.5 6.7 *

Weekly earnings ($) 466 442 24 400 426 -26 525 459 66 *** †††

Sample size (total = 1,122) 583 539 282 257 301 282

Among full research sample

Year 1

Ever employed in a

UI-covered job (%) 86.5 82.6 3.9 ** 84.7 80.9 3.8 88.3 84.2 4.1

UI earnings ($) 11,347 11,238 109 9,746 9,984 -238 12,936 12,375 561

Year 2

Ever employed in a

UI-covered job (%) 79.6 78.2 1.4 75.2 75.2 0.0 83.9 81.0 2.9

UI earnings ($) 14,876 13,258 1,618 ** 11,633 11,803 -170 18,055 14,548 3,507 *** †††

First quarter of Year 3

Ever employed in a

UI-covered job (%) 66.7 63.0 3.7 60.1 60.7 -0.6 73.2 65.0 8.2 ** †

UI earnings ($) 3,577 3,317 260 2,863 2,950 -87 4,279 3,644 636 ** †

Sample size (total = 1,395) 702 693 341 341 361 352

SOURCES: MDRC calculations from responses to the WorkAdvance Year 2 Survey and from unemployment 

insurance (UI) administrative records from the Ohio Department of Jobs and Family Services and Oklahoma 

Employment Security Commission. 

NOTES: WA = WorkAdvance (program) group; C = control group.

The early cohort includes all sample members randomly assigned through Quarter 3, 2012. The late cohort 

includes all sample members randomly assigned in or after Quarter 4, 2012.

A two-tailed t-test was applied to the differences between outcomes for the program and control groups. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

Differences across subgroups were tested for statistical significance. Statistical significance levels (Sig.) are 

indicated as follows: ††† = 1 percent; †† = 5 percent; † = 10 percent.

Sample sizes may vary because of missing values.
321

References

Acs, Gregory, and Pamela Loprest. 2004. “Leaving Welfare: Challenges in a New Economy.” 

Employment Research 11, 4: 5-6. 

Barnow, Burt S., and David Greenberg. 2015. “Do Estimated Impacts on Earnings Depend on 

the Source of the Data Used to Measure Them? Evidence from Previous Social 

Experiments.” Evaluation Review 39, 2: 179-228. 

Bloom, Dan, Richard Hendra, and Jocelyn Page. 2006. The Employment Retention and 

Advancement Project: Results from the Chicago ERA Site. New York: MDRC.

Bloom, Dan, and Charles Michalopoulos. 2001. How Welfare and Work Policies Affect 

Employment and Income: A Synthesis of Research. New York: MDRC.

Bloom, Howard S., and Charles Michalopoulos. 2010. When Is the Story in the Subgroups? 

Strategies for Interpreting and Reporting Intervention Effects on Subgroups. New York: 

MDRC.

Bloom, Howard S., Larry L. Orr, Stephen H. Bell, George Cave, Fred Doolittle, Winston Lin, 

and Johannes M. Bos. 1997. “The Benefits and Costs of JTPA Title II-A Programs: Key 

Findings from the National Job Training Partnership Act Study.” Journal of Human 

Resources 32, 3: 549-576. 

Bureau of Labor Statistics. 2014. Employee Benefits in the United States — March 2014. 

Washington, DC: Bureau of Labor Statistics, U.S. Department of Labor. 

Card, David, Jochen Kluve, and Andrea Weber. 2010. “Active Labour Market Policy 

Evaluations: A Meta‐Analysis.” Economic Journal 120, 548: F452-F477. 

Card, David, Jochen Kluve, and Andrea Weber. 2015. “What Works? A Meta Analysis of 

Recent Active Labor Market Program Evaluations.” Institute for the Study of Labor (IZA) 

Discussion Paper No. 9236. Bonn, Germany: IZA.

Card, David, Charles Michalopoulos, and Philip K. Robins. 2001. “The Limits to Wage 

Growth: Measuring the Growth Rate of Wages for Recent Welfare Leavers.” Working 

Paper 8444, National Bureau of Economic Research. 

Cohen, Jacob. 1992. “A Power Primer.” Psychological Bulletin 112, 1: 155.

Conway, Maureen, Amy Blair, Steven L. Dawson, and Linda Dworak-Munoz. 2007. Sectoral 

Strategies for Low-Income Workers: Lessons From The Field. Washington, DC: Aspen 

Institute.

D’Amico, Ronald. 2006. What’s Known About the Effects of Publicly-Funded Employment and 

Training Programs. Washington, DC: Social Policy Research Associates.

Friedlander, Daniel. 1988. “Subgroup Impacts and Performance Indicators for Selected Welfare 

Employment Programs.” New York: MDRC. 
322

Ghayad, Rand. 2013. “The Jobless Trap.” Working paper. Boston: Federal Reserve Bank of 

Boston and Northeastern University. 

Gladden, Tricia, and Christopher Taber. 2009. “The Relationship Between Wage Growth and 

Wage Levels.” Journal of Applied Econometrics 24, 6: 914-932. 

Golden, Lonnie. 2015. Irregular Work Scheduling and Its Consequences. Washington, DC: 

Economic Policy Institute. 

Gueron, Judith M., and Gayle Hamilton. 2002. The Role of Education and Training in Welfare 

Reform. New York: MDRC.

Gueron, Judith M., and Howard Rolston. 2013. Fighting for Reliable Evidence. New York: 

Russell Sage Foundation. 

Hamilton, Gayle, Stephen Freedman, Lisa Gennetian, Charles Michalopoulos, Johanna Walter, 

Diana Adams-Ciardullo, Anna Gassman-Pines, Sharon McGroder, Martha Zaslow, Surjeet 

Ahluwalia, and Jennifer Brooks. 2001. How Effective Are Different Welfare-to-Work 

Approaches? Five-Year Adult and Child Impacts for Eleven Programs. New York: MDRC.

Hamilton, Gayle, and Susan Scrivener. 2012a. Facilitating Postsecondary Education and 

Training for TANF Recipients. Brief No. 07. Washington, DC: Urban Institute. 

Hamilton, Gayle, and Susan Scrivener. 2012b. Increasing Employment Stability and Earnings 

for Low-Wage Workers: Lessons from the Employment Retention and Advancement (ERA) 

Project. OPRE Report 2012-19. Washington, DC: Office of Planning, Research and 

Evaluation, Administration for Children and Families, U.S. Department of Health and 

Human Services. 

Henderson, Kathryn, Crystal MacAllum, and Mustafa Karakus. 2010. Workforce Innovations: 

Outcome Analysis of Outreach, Career Advancement and Sector-Focused Programs. 

Rockville, MD: Westat.

Hendra, Richard, Keri-Nicole Dillman, Gayle Hamilton, Erika Lundquist, Karin Martinson, and 

Melissa Wavelet. 2010. How Effective Are Different Approaches Aiming to Increase 

Employment Retention and Advancement? Final Impacts for Twelve Models. New York: 

MDRC.

Hendra, Richard, James A. Riccio, Richard Dorsett, David H. Greenberg, Genevieve Knight, 

Joan Phillips, Philip K. Robins, Sandra Vegeris, and Johanna Walter. 2011. Breaking the 

Low-Pay, No-Pay Cycle: Final Evidence from the UK Employment Retention and 

Advancement (ERA) Demonstration. New York: MDRC.

Hill, Carolyn J., Howard S. Bloom, Alison Rebeck Black, and Mark W. Lipsey. 2007. 

Empirical Benchmarks for Interpreting Effect Sizes in Research. New York: MDRC.

Hilliard, Thomas. 2013. “Building the American Workforce.” Working paper. New York: 

Council on Foreign Relations. 
323

Hollenbeck, Kevin. 2007. Process and Net Impact Evaluations of the Focus: HOPE Adult 

Training Programs and Student Loan Fund. Upjohn Institute Technical Report No. 07-

023. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research. 

Holzer, Harry J. 2010. “Avoiding a Lost Generation: How to Minimize the Impact of the Great 

Recession on Young Workers.” Testimony before the Joint Economic Committee of the 

U.S. Congress. Washington, DC: Urban Institute. 

Holzer, Harry J. 2013. “Skill Mismatches in Contemporary Labor Markets: How Real? And 

What Remedies?” Conference paper. Washington, DC: Atlantic Council and University of 

Maryland School of Public Policy.

Hotz, V. Joseph, and John Karl Scholz. 2001. “Measuring Employment and Income for LowIncome Populations with Administrative and Survey Data.” Discussion Paper 1224-01. 

Madison, WI: Institute for Research on Poverty.

Kolesnikova, Natalia, and Yang Liu. 2011. “Jobless Recoveries: Causes and Consequences.” 

Regional Economist, April. Website: www.stlouisfed.org/publications/regionaleconomist. 

Kornfeld, Robert, and Howard S. Bloom. 1999. “Measuring Program Impacts on Earnings and 

Employment: Do Unemployment Insurance Wage Reports from Employers Agree with 

Surveys of Individuals?” Journal of Labor Economics 17, 1: 168-197. 

Kroft, Kory, Fabian Lange, and Matthew J. Notowidigdo. 2013. “Duration Dependence and 

Labor Market Conditions: Evidence from a Field Experiment.” Quarterly Journal of 

Economics 128, 1123-1167. 

Lower-Basch, Elizabeth, and Neil Ridley. 2013. Navigating Federal Programs to Build 

Sustainable Career Pathways in the Health Professions: A Guide for HPOG Programs. 

Washington, DC: Center for Law and Social Policy.

Maguire, Sheila, Joshua Freely, Carol Clymer, Maureen Conway, and Deena Schwartz. 2010. 

Tuning In to Local Labor Markets: Findings from the Sectoral Employment Study. 

Philadelphia: Public/Private Ventures.

Mangat, Ravinder. 2007. Sector Snapshot: A Profile of Sector Initiatives, 2007. Oakland, CA: 

National Network of Sector Partners.

Mangat, Ravinder. 2010. Sector Snapshot: A Profile of Sector Initiatives, 2010. Oakland, CA: 

National Network of Sector Partners.

Martinson, Karin, and Karen Gardiner. 2014. Improving the Economic Prospects of LowIncome Individuals Through Career Pathways Programs: The Innovative Strategies for 

Increasing Self-Sufficiency Evaluation. OPRE Report 2014-17. Washington, DC: Office of 

Planning, Research and Evaluation, Administration for Children and Families, U.S. 

Department of Health and Human Services.

Mayer, Gerald. 2014. “The Increased Supply of Underutilized Labor from 2006 to 2014.” 

Monthly Labor Review, November. Website: www.bls.gov/opub/mlr. 
324

Melendez, Edwin. 1996. Working on Jobs: The Center for Employment Training. Boston: 

Mauricio Gastón Institute.

Miller, Cynthia, Betsy L. Tessler, and Mark van Dok. 2009. Implementation and Early Impacts 

of the Work Advancement and Support Center (WASC) Demonstration. New York: MDRC.

Miller, Cynthia, Mark van Dok, Betsy L. Tessler, and Alexandra Pennington. 2012. Strategies 

to Help Low-Wage Workers Advance: Implementation and Final Impacts of the Work 

Advancement and Support Center (WASC) Demonstration. New York: MDRC.

Mishel, Lawrence, Jared Bernstein, and Sylvia Allegretto. 2007. The State of Working America 

2006/2007: An Economic Policy Institute Book. Ithaca, NY: ILR Press.

Mishel, Lawrence, Jared Bernstein, and Heidi Shierholz. 2009. The State of Working America:

2008/2009. Ithaca, NY: ILR Press.

Morrison, Tom, Bob Maciejewski, Craig Giffi, Emily Stover DeRocco, Jennifer McNelly, and

Gardner Carrick. 2011. Boiling Point? The Skills Gap in U.S. Manufacturing. Washington,

DC: Deloitte and The Manufacturing Institute.

National Skills Coalition. 2011. Workforce Investment Act, Title I: Workforce Investment 

Systems for Adults, Dislocated Workers, and Youth. Washington, DC: National Skills 

Coalition. Website: www.nationalskillscoalition.org/InBrief.

Olken, Benjamin. 2015. “Promises and Perils of Pre-Analysis Plans.” Journal of Economic 

Perspectives 29, 3: 61-80. 

Osterman, Paul, and Brenda Lautsch. 1996. Project QUEST: A Report to the Ford Foundation. 

Cambridge: Massachusetts Institute of Technology, Sloan School of Management.

Puma, Michael J., Robert B. Olsen, Stephen H. Bell, and Cristofer Price. 2009. What to Do 

When Data Are Missing in Group Randomized Controlled Trials (NCEE 2009-0049). 

Washington, DC: National Center for Education Evaluation and Regional Assistance, 

Institute of Education Sciences, U.S. Department of Education. 

Rangarajan, Anu, and Tim Novak. 1999. The Struggle to Sustain Employment: The 

Effectiveness of the Postemployment Services Demonstration. Springfield: Illinois 

Department of Human Services.

Riccio, James A., Nadine Dechausay, Cynthia Miller, Stephen Nuñez, Nandita Verma, and 

Edith Yang. 2013. Conditional Cash Transfers in New York City: The Continuing Story of 

the Opportunity NYC-Family Rewards Demonstration. New York: MDRC.

Schochet, Peter, and Anu Rangarajan. 2004. Characteristics of Low-Wage Workers and Their 

Labor Market Experiences: Evidence from the Mid- to Late 1990s. Washington, DC: U.S. 

Department of Health and Human Services.

Tessler, Betsy. 2013. WorkAdvance: Testing a New Approach to Increase Employment 

Advancement for Low-Skilled Adults. New York: MDRC.
325

Tessler, Betsy L., Michael Bangser, Alexandra Pennington, Kelsey Schaberg, and Hannah 

Dalporto. 2014. Meeting the Needs of Workers and Employers: Implementation of a 

Sector-Focused Career Advancement Model for Low-Skilled Adults. New York: MDRC.

Tibshirani, Robert. 1996. “Regression Shrinkage and Selection via the Lasso.” Journal of the 

Royal Statistical Society, Series B (Methodological) 58, 1: 267-288. 

Tompson, Trevor, Jennifer Benz, Jennifer Agiesta, and Dennis Junius. 2013. America’s LowerWage Workforce: Employer and Worker Perspectives. Chicago: Associated Press-NORC 

Center for Public Affairs Research.

U.S. Congress. 2015. “S.1772 - Schedules That Work Act.” Website: 

https://www.congress.gov/bill/114th-congress/senate-bill/1772. 

U.S. Department of Labor. 2015. “What Is the Workforce Innovation Fund?” Website: 

http://www.doleta.gov/workforce_innovation/. 

Van Horn, Carl, Kathy Krepcio, and Maria Heidkamp. 2011. Research Report: Improving 

Education and Training for Older Workers. Washington, DC: AARP Public Policy 

Institute.

Woolsey, Lindsey, and Garrett Groves. 2013. State Sector Strategies Coming of Age: 

Implications for State Workforce Policymakers. Washington, DC: National Governors 

Association Center for Best Practices.

Zandniapour, Lily, and Maureen Conway. 2002. Gaining Ground: The Labor Market Progress 

of Participants of Sectoral Employment Development Programs. Sectoral Development 

Learning Project (SEDLP) Research Report No. 3. Washington, DC: Workforce Strategies 

Institute at the Aspen Institute.
About MDRC

MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 

to learning what works to improve the well-being of low-income people. Through its research 

and the active communication of its findings, MDRC seeks to enhance the effectiveness of social and education policies and programs.

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 

for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 

Its projects are a mix of demonstrations (field tests of promising new program approaches) and 

evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 

combination of research and organizational experience to their work, providing expertise on the 

latest in qualitative and quantitative methods and on program design, development, implementation, and management. MDRC seeks to learn not just whether a program is effective but also 

how and why the program’s effects occur. In addition, it tries to place each project’s findings in 

the broader context of related research — in order to build knowledge about what works across 

the social and education policy fields. MDRC’s findings, lessons, and best practices are proactively shared with a broad audience in the policy and practitioner community as well as with the 

general public and the media.

Over the years, MDRC has brought its unique approach to an ever-growing range of policy areas and target populations. Once known primarily for evaluations of state welfare-to-work programs, today MDRC is also studying public school reforms, employment programs for exoffenders and people with disabilities, and programs to help low-income students succeed in 

college. MDRC’s projects are organized into five areas:

• Promoting Family Well-Being and Children’s Development

• Improving Public Education

• Raising Academic Achievement and Persistence in College

• Supporting Low-Wage Workers and Communities

• Overcoming Barriers to Employment

Working in almost every state, all of the nation’s largest cities, and Canada and the United 

Kingdom, MDRC conducts its projects in partnership with national, state, and local governments, public school systems, community organizations, and numerous private philanthropies.