CONTRACT NO. DOLJ061A20363 

TASK ORDER #2 

IMPLEMENTING EFFICIENCY MEASURES FOR 

EMPLOYMENT AND TRAINING PROGRAMS 

Final Report 

MAY 2010

Prepared for: 

U.S. DEPARTMENT OF LABOR 

EMPLOYMENT AND TRAINING ADMINISTRATION

Prepared by: 

John Trutko Burt S. Barnow 

Capital Research Corporation Johns Hopkins University 

1910 N. Stafford Street Institute for Policy Studies 

Arlington, VA 22207 3400 N. Charles Street 

 Baltimore, MD 21218
Final Report – Implementing Efficiency Measures for ETA Programs Page i

ABSTRACT 

In 2007, the Office of Management and Budget (OMB) required the development of 

efficiency measures for all federal government programs as part of the effort to improve federal 

government program performance. As a result of Program Assessment Rating Tool reviews, 

OMB asked the Employment and Training Administration of the U.S. Department of Labor to 

develop and implementing an outcome-based measure or measures of efficiency for employment 

and training programs administered by the agency. In response to this OMB directive, in May 

2008, ETA initiated a study to identify outcome-based efficiency measures for implementation 

by 11 ETA-administered programs: Workforce Investment Act (WIA) Adult Program; WIA 

Dislocated Worker Program; WIA Youth Activities Program; WIA National Emergency Grants 

Program; Trade Adjustment Assistance Program; Wagner-Peyser/Employment Service (ES) 

Program; Senior Community Service Employment Program National Farmworker Jobs Program 

Indian and Native American Program Work Incentive Grant Program; and Apprenticeship 

Program. 

A key lesson that emerges from this study is that it is critical in selecting measures, 

standards, rewards, and sanctions to anticipate the behavioral changes that are likely to be 

induced by the performance management policies adopted and to structure the system so that the 

presence of efficiency measures does not result in undesirable behavior by programs, states, and 

grantees. To be implemented within three years, the study recommends that efficiency measures 

should be closely tied to the current outcome performance measures in effect under ETA’s 

Common Measures framework. Though the report highlights some of the challenges of 

comparing efficiency measure results across programs, the Common Measures provide common 

definitions for outcome measures and thus increase the potential for making meaningful 

comparisons of efficiency measure results within individual programs (e.g., across 

states/subgrantees) and across at least some of the ETA programs of interest. This report also 

recommends use of program expenditures (rather than appropriations or obligations) as the 

measure of program costs in efficiency measures. Among the efficiency measures recommended 

for consideration in this report are cost per entered employment, cost per retained in 

employment, cost divided by post-program (average) earnings, and cost divided by change in 

earnings. The report concludes with a series of recommendations concerning the specific 

efficiency measures that should (and should not) be considered for implementation by each of 

the 11 ETA programs that are the focus of this study and, if adopted, how these measures should 

be used to monitor and enhance program performance. 

 
Final Report – Implementing Efficiency Measures for ETA Programs Page ii

ACKNOWLEDGEMENTS 

The following local and state workforce agency officials, U.S. Department of Labor 

regional and national staff, and Office of Management and Budget (OMB) officials significantly 

contributed to the development of this report by providing comments on reports, program data 

and other documents, and/or other input to help understand program operations and potential 

effects of efficiency measures: Erik Anderson, Randee Chafkin, Terry Clark, James Conley, 

Traci DiMartini, Denise Dombeck, Christina Eckenroth, Clint Flowers, Bernard Foster, Judith 

Gilbert, Wayne Gordon, Duane Hall, Anita Harvey, Matthew Heaney, Joseph Holicky, O’Shell 

Howell, Alexandra Kielty, Carole Kitti, Kenneth Leung, Ernesto Lirag, Richard Little, Richard 

Moore, Beverly Odom, Donald Palmer, Kimberly Powell, Michael Qualter, Steven Reenstra, 

Gail Robinson, Evan Rosenberg, Keith Rowe, Gloria Salas-Kos, Brad Sickles, LaTonya 

Torrence, Steven Urquhart, Stephen Wandner, Maria Weidmark, and Susan Worden. The 

authors would particularly like to thank Bernard Foster, Gloria Salas-Kos, and Stephen Wandner 

(of the Employment and Training Administration, ETA) for their thoughtful ongoing guidance, 

help with coordinating data collection among various ETA program offices, and for their many 

hours reviewing and commenting on interim and final reports developed over the course of this 

research project. 

Other significant input on the use of efficiency measures in workforce programs in other 

countries was provided by: Frank Fedyk, Associate Assistant Deputy Minister for Policy 

Research; Serge Bertrand, Director General for Evaluation; and Stephen Johnson, Director 

General for Corporate Planning and Accountability, Human Resources and Social Development 

Canada; Bill Wells, Department of Business Enterprise and Regulatory Reform, United 

Kingdom; and John Martin and David Grubb, Organization for Economic Cooperation and 

Development (OECD). 

To ensure quality, the report included guidance and review from notable experts in the 

field of performance measurement. The following individuals served on a peer review panel: 

Carolyn Heinrich (University of Wisconsin), Jeffrey Smith (University of Michigan), 

Christopher King (University of Texas), and Beryl Radin (American University). 

Finally, Jonathan Pollak (of Johns Hopkins University) played a key role in developing 

cost and outcome databases for ETA programs, as well as analyzing efficiency measure results 

by program and at the state-level, and developing regression models used in this report. 
TABLE OF CONTENTS 

Final Report – Implementing Efficiency Measures for ETA Programs Page iii

ABSTRACT..................................................................................................................................................i

ACKNOWLEDGEMENTS........................................................................................................................ ii

EXECUTIVE SUMMARY................................................................................................................... ES­1

CHAPTER 1:  INTRODUCTION............................................................................................................ 1

A. STUDY BACKGROUND, OBJECTIVES, AND METHODOLOGY............................................ 1

B. DEFINITION AND CRITERIA FOR SELECTION OF EFFICIENCY MEASURES................ 7

C. CANDIDATE EFFICIENCY MEASURES ....................................................................................10

CHAPTER 2:  REVIEW OF LITERATURE ON EFFICIENCY MEASUREMENT IN

EMPLOYMENT AND TRAINING PROGRAMS................................................................................17

A. THE IMPACT OF PERFORMANCE STANDARDS ON WHO IS SERVED ..........................18

B. EFFECTS OF PERFORMANCE INCENTIVES ON SERVICES PROVIDED .........................22

C. RELATIONSHIP BETWEEN PERFORMANCE MEASURES AND                            

PROGRAM IMPACT......................................................................................................................23

D. STRATEGIC RESPONSES TO PERFORMANCE INCENTIVES ............................................24

E. USING RETURN ON INVESTMENT (ROI) AS A PERFORMANCE MEASURE.................29

F. EMPLOYMENT AND TRAINING ADMINISTRATION EXPERIENCE                              

WITH EFFICIENCY MEASURES .................................................................................................36

G. STATE/LOCAL WORKFORCE AGENCY EXPERIENCE WITH EFFICIENCY         

MEASURES......................................................................................................................................37

H. CONCLUSIONS...............................................................................................................................43

CHAPTER 3:  USE OF EFFICIENCY MEASURES FOR MONITORING PERFORMANCE IN

DOMESTIC AND FOREIGN WORKFORCE DEVELOPMENT AND OTHER HUMAN  

SERVICE PROGRAMS...........................................................................................................................46

A. USE OF EFFICIENCY MEASURES IN OTHER FEDERAL AGENCIES.................................46

B. USE OF EFFICIENCY MEASURES IN OTHER INDUSTRIALIZED COUNTRIES .............50
TABLE OF CONTENTS 

Final Report – Implementing Efficiency Measures for ETA Programs Page iv

CHAPTER 4:  STATE WORKFORCE AGENCIES’ VIEWS ON USE OF EFFICIENCY

MEASURES TO MONITOR PROGRAM PERFORMANCE.............................................................58

A. FINDINGS FROM AN EARLY ROUND OF INTERVIEWS WITH SIX STATES.................59

B. FINDINGS FROM A SECOND (LATE) ROUND OF INTERVIEWS WITH                         

FIVE STATES..................................................................................................................................67

CHAPTER 5:  OPTIONS AVAILABLE, ANALYSIS, AND CHALLENGES TO MEASURING

COSTS IN CALCULATING EFFICIENCY MEASURES.....................................................................78

A. OPTIONS AVAILABLE, ANALYSIS, AND RECOMMENDATIONS FOR MEASURING

PROGRAM COSTS.........................................................................................................................78

B. CHALLENGES INVOLVED IN PRODUCING ACCURATE AND TIMELY      

EXPENDITURE DATA TO PRODUCE VALID AND RELIABLE EFFICIENCY     

MEASURE RESULTS ACROSS STATES....................................................................................83

C. CONCLUSIONS ..............................................................................................................................90

CHAPTER 6:  OPTIONS AVAILABLE, ANALYSIS, AND CHALLENGES TO       

CALCULATING EFFICIENCY MEASURES........................................................................................92

A. OUTCOME/PERFORMANCE MEASURES USED BY ETA PROGRAMS............................92

B. ANALYSES OF EFFICIENCY MEASURES RESULTS BASED ON THE COMMON

MEASURE OUTCOMES AT THE NATIONAL LEVEL ............................................................98

C. ANALYSES OF EFFICIENCY MEASURES RESULTS AT THE STATE LEVEL................ 110

D. CONCLUSIONS............................................................................................................................ 126

CHAPTER 7: POTENTIAL APPROACHES TO SETTING AND ADJUSTING           

STANDARDS FOR EFFICIENCY MEASURES ............................................................................... 128

A. OVERVIEW OF ETA’S EXPERIENCE WITH SETTING AND ADJUSTING PERFORMANCE

STANDARDS FOR EMPLOYMENT AND TRAINING PROGRAMS ................................................129

B. OPTIONS FOR SETTING EFFICIENCY MEASURE STANDARDS FOR ETA    

PROGRAMS ................................................................................................................................. 135

C. APPROACHES TO MAKING STATISTICAL ADJUSTMENTS ........................................... 141

D. ADJUSTING EFFICIENCY MEASURES USING STATISTICAL MODELS:       

PRELIMINARY MODELS AND RESULTS FOR THE WIA ADULT, DISLOCATED

WORKER, AND YOUTH PROGRAMS ................................................................................... 145

E. CONCLUSIONS ............................................................................................................................ 162
TABLE OF CONTENTS 

Final Report – Implementing Efficiency Measures for ETA Programs Page v

CHAPTER 8:  STUDY RECOMMENDATIONS AND CONCLUSIONS....................................... 165

A. OVERVIEW OF EXPERT PANEL VIEWS ON IMPLEMENTATION OF OUTCOMEBASED EFFICIENCY MEASURES BY ETA PROGRAMS .................................................... 165

B. STUDY RECOMMENDATIONS ON IMPLEMENTATION OF OUTCOME­BASED

EFFICIENCY MEASURES FOR ETA PROGRAMS................................................................ 168

APPENDICES:

APPENDIX A:  DISCUSSION GUIDES FOR INTERVIEWS WITH STATE WORKFORCE

AGENCIES............................................................................................................................................. 184

APPENDIX B:  REFERENCES........................................................................................................... 189

APPENDIX C:  REGRESSION RESULTS FOR FOUR ADDITIONAL EFFICIENCY       

MEASURE OUTCOMES ..................................................................................................................... 195

APPENDIX D:  WRITTEN COMMENTS PROVIDED BY EXPERT PANEL FOLLOWING

NOVEMBER 2009 PANEL MEETING............................................................................................ 200

APPENDIX E:  WRITTEN COMMENTS PROVIDED BY EMPLOYMENT AND TRAINING

PROGRAM OFFICES IN RESPONSE TO THE DRAFT FINAL REPORT AND AUTHORS’

RESPONSES TO COMMENTS PROVIDED.................................................................................... 212
Final Report – Implementing Efficiency Measures for ETA Programs Page ES-1

EXECUTIVE SUMMARY 

INTRODUCTION

In 2007, the Office of Management and Budget (OMB) required the development of 

efficiency measures for all federal government programs as part of the effort to improve federal 

government program performance. As a result of Program Assessment Rating Tool (PART) 

reviews, OMB asked the Employment and Training Administration (ETA) to develop and 

implement an outcome-based measure or measures of efficiency for employment and training 

programs administered by the agency.1

 In response to this OMB directive, in May 2008, ETA 

initiated a study to identify outcome-based efficiency measures for implementation by 11 ETAadministered programs: Workforce Investment Act (WIA) Adult Program; WIA Dislocated 

Worker Program; WIA Youth Activities Program; WIA National Emergency Grants (NEG) 

Program; Trade Adjustment Assistance (TAA) Program; Wagner-Peyser/Employment Service 

(ES) Program; Senior Community Service Employment Program (SCSEP); National 

Farmworker Jobs Program (NFJP); Indian and Native American Program (INAP); Work 

Incentive Grant (WIG) Program; and Apprenticeship Program.2

 

 While the definition of efficiency measures is relatively straightforward, there are a 

number of serious challenges to identifying appropriate and feasible measures for a single 

program. These challenges multiply when consideration is given to applying an efficiency 

measure or measures across more than one program. The efficiency measure or measures 

applied to one or more programs should at a minimum meet the following four criteria: 

• be fair to the programs being judged; 

• encourage desired service delivery and program outcomes; 

• discourage undesired strategies and behaviors; and 

• maintain program quality, integrity, and fiscal responsibility. 

In addition, the efficiency measures recommended should be feasible and cost-effective to 

implement using participant, outcome, and cost data that are currently collected or that could 

potentially be collected in the future at a reasonable cost (such as data collected and reported as 

part of the Common Measures). 

 

1

Executive Office of the President, Office of Management and Budget, Program Assessment Rating Tool Guidance 

No. 2007-7: Improving the Quality of PART Performance and Efficiency Goals, issued December 12, 2007 

(available at: http://georgewbush-whitehouse.archives.gov/omb/performance/guidance/part_guid_2007-07.pdf). 

The Program Assessment Rating Tool Guidance No. 2007-7 called for a government-wide initiative “to increase the 

government and public’s ability to assess and improve a program’s effectiveness and efficiency.” 

2

Job Corps was not included because that program was not part of ETA during the period of the study. 
Final Report – Implementing Efficiency Measures for ETA Programs Page ES-2

STUDY FINDINGS

Findings from the Review of the Literature on Efficiency Measurement

The research shows that programs pay careful attention to the performance standards they 

face, and that the presence of performance standards can have important effects on who is served 

and the services received. Unfortunately, the effects of performance standards have not always 

been in the desired direction: 

• The most significant problems that appear in the literature are that performance standards 

can encourage “cream skimming,” where the programs are more likely to enroll 

individuals who would do well even without the programs, and “gaming,” where 

programs spend resources manipulating their measured performance by strategically 

enrolling and terminating individuals in a manner that makes the program look most 

effective. 

• Studies of Job Training Partnership Act (JTPA) and Job Corps indicate that ranking on 

performance measures is not related to program impact on key outcomes. Thus, it is 

important that measured performance not be misinterpreted as measuring program 

impact. 

A key lesson from the literature, as well as ETA’s and state/local experiences with 

efficiency measures, is that it is critical in selecting measures, standards, rewards, and sanctions 

to anticipate the behavioral changes that are likely to be induced by the performance 

management policies adopted and to structure the system so that the presence of efficiency 

measures does not result in undesirable behavior by states and grantees. In addition, careful 

thought should be given as to whether outcome-based efficiency measures should be applied 

only at the national level or whether they should be “drilled-down” from the federal to states 

and/or local workforce investment areas. 

Findings on Use of Efficiency Measurement in the United States and Other 

Industrialized Countries 

The primary lesson that emerged from a review of domestic and foreign government 

utilization of efficiency measures to assess public sector program performance is that 

government agencies have interpreted the concept of efficiency quite broadly. Within the 

United States, a review was conducted of efficiency measures used by five federal cabinet-level 

agencies: the U.S. Department of Agriculture (USDA), the Education Department (ED), the 

Department of Health and Human Services (DHHS), the Department of Housing and Urban 

Development (HUD), and the Department of Veterans’ Affairs (VA). Within these departments, 

this assessment focused on programs that provide activities and services related to employment 

and training. Many federal agencies in the United States have more than one efficiency measure, 

and agencies have tailored the measures broadly to reflect their concerns about what aspects of 

their programs can be made more efficient. The most commonly used efficiency measure among 

programs reviewed were cost per participant, outcome- and output-based efficiency measures 
Final Report – Implementing Efficiency Measures for ETA Programs Page ES-3

timeliness of government decisions and service delivery measures, accuracy of payments or 

determinations, and costs per service provided. 

Based on the interviews conducted and review of background documentation, the United 

Kingdom and Canada (along with other Organisation of Economic Cooperation and 

Development [OECD] countries) assess the performance of workforce development programs 

both on an ongoing basis and through periodic evaluations. While workforce agencies utilize 

performance monitoring systems to track program participation levels, outcomes, and program 

costs, they do not employ outcome-based efficiency measures (linking outcomes such as job 

placement to program expenditures). Some programs utilize processing/output efficiency 

measures as part of ongoing performance monitoring systems, such as measurements of accuracy 

and timeliness of payments to participants (but these measures are not tied to program costs). 

There is some concern over the potential for “gaming” if, as part of ongoing performance 

monitoring, program expenditures are directly linked to outcomes – mostly, the concern is that 

service delivery units (e.g., job centers) may target services on the less difficult to serve to 

reduce per-participant costs. Overall, major industrialized countries, as yet, have not 

implemented the types of outcome-based efficiency measures (contemplated by ETA) for 

ongoing performance management system, though some periodic evaluation efforts have 

rigorously examined cost-effectiveness and return on investment for workforce programs. 

Findings from Interviews with State Workforce Agency Officials

 Two rounds of interviews were conducted of state workforce agencies. State 

administrators offered several recommendations on developing and implementing efficiency 

measures for the 11 ETA programs: 

 

• To the extent possible, develop efficiency measures that rely upon information already 

being collected by states – this will reduce the cost and burden of data collection for 

states/local areas. 

• Some programs may do well on one efficiency measure but not another – so ETA should 

consider implementing more than one efficiency measure. 

• Be careful in selecting and implementing efficiency measures so that you do not 

discourage sharing of funds and co-enrollment across partners and building integrated 

systems. 

• Be cautious about making comparisons across states and local workforce areas on 

efficiency measures because there are many factors that affect program participation, 

outcomes, and costs. Making such comparisons (and setting of performance standards) 

for states/local areas could potentially create strong incentives for workforce programs to 

provide services that are least expensive (such as labor exchange services) and not 

tailored to the specific needs (and best long-term outcomes) of customers served. 

State workforce agency officials observed that efficiency measures need to be carefully tailored 

to the goals of each ETA program, noting that special attention is particularly needed in 
Final Report – Implementing Efficiency Measures for ETA Programs Page ES-4

determining and applying efficiency measures for the WIA Youth, WIG, and Apprenticeship 

programs. State workforce officials agreed that outcome-based efficiency measures could be a 

useful tool for monitoring program performance, but they stated that great care is needed in 

selecting the specific measures and determining how they will be applied over time to monitor 

program performance. Several state administrators worried that if efficiency measures are 

implemented as part of regular performance measurement they would likely drive states/local 

areas toward providing less costly labor exchange services (such as those provided under 

Wagner-Peyser) – and that this would work against states and local areas providing training to 

enable workers to upgrade skills in order to fill higher skilled/wage jobs. 

Findings from Analyses of Expenditure and Outcome Data to Produce Preliminary 

Efficiency Measure Results for ETA Programs

The quantitative analyses of available expenditure and outcome data (to produce 

estimates on select efficiency measures by program for a three-year period (PY 2005-2007) 

demonstrated the feasibility of producing outcome-based efficiency measure results for most of 

the 11 ETA programs, but also highlighted some of the concerns that have been expressed by 

ETA and state-level program officials, as well as the Expert Panel. In particular, the very 

substantial variation across programs in efficiency measure results points to the widely varying 

cost structures for programs that provide intensive assistance and training services (such as the 

TAA and WIA programs) versus programs such as the Wagner-Peyser program providing less 

customer intensive, labor exchange-type services. The often sizable differences between the 

highest and lowest states on efficiency measure results demonstrates how efficiency measure 

results can dramatically differ across states, as well as suggests that data submitted by states on 

either expenditures or outcomes may be based on inconsistencies in data collection or erroneous 

data. The analyses included in the report also examines the prospects for taking a step beyond 

implementing efficiency measures to implementing performance standards with or without 

statistical adjustments for each efficiency measure. Such performance standards, if adopted, 

would parallel the standards used for outcomes under Job Training Partnership Act (using 

regression-based adjustment models) and currently under WIA (using negotiations). The 

modeling efforts conducted for this report, which focuses on the three WIA programs, suggested 

that great caution and several additional years of results are required before ETA should consider 

implementing performance standards for states/grantees on the recommended efficiency 

measures.

STUDY RECOMMENDATIONS

Recommendation #1: Use Program Expenditures Rather than Appropriations or 

Obligations as the Measure of Program Costs in Efficiency Measures

Use of expenditures rather than appropriations, allocations, or obligations in calculating 

efficiency measures is recommended because (1) expenditures can vary substantially from what 

is initially appropriated/allocated, especially at the state level (because of transfers, rescissions, 

and unexpended funds); and (2) expenditures reflect what is actually spent on delivery of 

services and capture the underlying notion of efficiency. States interviewed, ETA program 
Final Report – Implementing Efficiency Measures for ETA Programs Page ES-5

offices, and the Expert Panel endorsed the use of expenditures over the other available measures 

of costs.

Recommendation #2: Use Common Measures as Starting Point for Measuring 

Program Outcomes in Efficiency Measures 

If they were to be implemented within three years, efficiency measures should be closely 

tied to the current outcome performance measures in effect under ETA’s Common Measures 

framework. Data is already being collected at the state and grantee levels on these outcomes, so 

the performance data needed to generate efficiency measure results would already be available 

(for most programs) – reducing costs and start-up time. In addition, though the report highlights 

some of the challenges of comparing efficiency measure results across programs and there is 

considerable variability across programs in terms of data quality and comparability, the Common 

Measures provide common definitions for outcome measures and thus increase the potential for 

making meaningful comparisons of efficiency measure results within individual programs (e.g., 

across states/subgrantees) and across at least some of the ETA programs of interest. 

• Recommendation 2a: Cost per entered employment should be tracked (for monitoring 

purposes initially) as an efficiency measure for 8 of the 11 ETA programs: WIA Adult, 

WIA Dislocated Worker, WIA NEG, Wagner-Peyser, TAA, SCSEP, INA, and NFJP. 

The WIA Youth program could use cost per placement in employment or education (as 

an alternative to cost per entered employment). The WIG and Apprenticeship 

programs should be excluded from implementing this measure. The main rationale for 

recommending cost per entered employment (and using cost per placement in education 

or training for the WIA Youth program) as an efficiency measure is as follows: (1) 

employment is a high priority for all programs (except WIA younger youth); (2) entered 

employment is the simplest and most direct way to assess whether programs are 

achieving their goals; (3) data are already being collected on the number of entered 

employments under the Common Measures (making this measure feasible and relatively 

inexpensive to implement); and (4) in comparison to other measures, data are available 

sooner for entered employment than for post-program earnings and job retention rates. 

• Recommendation 2b: Cost per retained in employment should be tracked (for 

monitoring purposes initially) as an efficiency measure for 8 of the 11 ETA programs: 

WIA Adult, WIA Dislocated Worker, WIA NEG, Wagner-Peyser, TAA, SCSEP, INA, 

and NFJP. The WIA Youth, Apprenticeship, and WIG programs should be excluded 

from implementing this measure. The main rationale for recommending cost per 

retained in employment as an efficiency measure is similar to cost per entered 

employment and is as follows: (1) job retention is a high priority for all programs (except 

WIA younger youth); (2) participants who are employed at the time of entry into the 

program are included in this measure (unlike entered employment rate); and (3) data are 

already being collected on the number of retained employments under the Common 

Measures (making this measure feasible and relatively inexpensive to implement). In 

comparison to entered employment rate, data on this measure is available later, but this 

indicator provides a downstream measure (of job retention and longer-term employment) 

of the effects of training and other employment services. Additionally, this measure is 
Final Report – Implementing Efficiency Measures for ETA Programs Page ES-6

appropriate for incumbent workers, a source of concern that some states expressed about 

the cost per entered employment efficiency measure, which (as defined under Common 

Measures) excludes individuals who were working at the time of enrollment. 

• Recommendation #2c: Cost divided by post-program (average) earnings should be 

tracked (for monitoring purposes initially) as an efficiency measure in 8 of 11 ETA 

programs: WIA Adult, WIA Dislocated Worker, WIA NEG, Wagner-Peyser, TAA, 

SCSEP, INA, and NFJP. The WIA Youth, Apprenticeship, and WIG programs should 

be excluded from implementing this measure. The rationale for recommending cost 

divided by post-program earnings as an efficiency measure for implementation by ETA 

programs is as follows: (1) effective programs should increase earnings as well as 

employment; (2) omitting earnings might encourage focus on inexpensive labor exchange 

or core services rather than intensive services and training; (3) participants who are 

employed at the time of entry into the program are included in this measure (unlike 

entered employment rate); and (4) data are already being collected on the pre- and postearnings under the Common Measures (making this measure feasible and relatively 

inexpensive to implement). In comparison to the entered employment rate, data on this 

measure is available later, but this indicator provides a downstream measure of the 

earnings effects of training and other employment services that result in job retention and 

longer-term employment. 

• Recommendation #2d: Cost divided by change in earnings should be tracked (for 

monitoring purposes initially) as an efficiency measure in 8 of the 11 ETA programs: 

WIA Adult, WIA Dislocated Worker, WIA NEG, Wagner-Peyser, TAA, SCSEP, INA, 

and NFJP. The WIA Youth, Apprenticeship, and WIG programs should be excluded 

from implementing this measure. The rationale for recommending cost divided by 

change in earnings as an efficiency measure for implementation by ETA programs is as 

follows: (1) effective human capital building programs should increase earnings as well 

as employment; (2) omitting earnings might encourage focus on inexpensive labor 

exchange or core services rather than intensive services and training; (3) by looking at 

pre/post earnings change (versus average post-program earnings), programs face fewer 

incentives for “creaming” those individuals who are likely to have the highest postprogram earnings; and (4) although an earnings change measure is not currently used 

under the Common Measures, data are already being collected on the pre- and postearnings under the Common Measures (making this measure feasible and relatively 

inexpensive to implement). 

Recommendation #3: Carefully Consider Programmatic Differences Before 

Implementing Efficiency Measures – Among the 11 ETA Programs, WIG, 

Apprenticeship, and WIA Youth Programs Will Likely Require a Different Set of 

Efficiency Measures

An often recurring message of ETA program administrators and state program operators 

– reinforced by the Expert Panel and findings from the literature – is that the 11 ETA programs 

have varying goals/objectives, target and serve different at-risk populations, offer widely varying 
Final Report – Implementing Efficiency Measures for ETA Programs Page ES-7

types and intensities of services, and have widely differing costs. As a result, efficiency 

measures need to be cautiously developed and tailored to what programs are attempting to 

achieve – and great care is needed in comparing results on such measures within programs (i.e., 

across states, grantees, and local jurisdictions) and across programs. For Apprenticeship, WIG, 

and the WIA Youth programs, ETA should be cautious in applying the measures identified in 

Recommendation #2 and should consider alternative measures as follows: 

• Recommendation #3a: ETA should consider alternative efficiency measures for the 

Apprenticeship Program linked to the goals of the program and what federal funds are 

being spent on – for example, increasing the number of apprenticeships offered, 

building the quality of apprenticeship programs, and registering and monitoring of 

Apprenticeship programs accurately and in a timely manner. Therefore, ETA should 

consider applying the following alternative efficiency measures to the Apprenticeship 

program (all at the national level): cost per additional apprenticeship program 

registered and timeliness of registration decisions. Specific measures should be 

adopted after appropriate dialogue and analysis is undertaken. 

• Recommendation #3b: ETA should consider alternative efficiency measures for the 

WIG Program that reflect the training and technical assistance goals of this program. 

In particular, efficiency of this program should be aimed at measuring how the 

services of Disability Navigators increase the numbers of disabled individuals served by 

the One-Stop system (and various ETA programs operating out of the One-Stop 

system), as well as improvements in identifying individuals served by the workforce 

system and enhancements to the quality of services provided to disabled individuals. 

Therefore, ETA should consider applying the following alternative efficiency measure 

to the WIG program: cost per change in the number of One-Stop customers served 

with disabilities. Efficiency measures for WIG should be adopted after appropriate 

dialogue and analysis. Because the WIG program may be terminated, it may be 

appropriate not to develop efficiency measures for this program. 

• Recommendation #3c: The efficiency measure that should be applied to the WIA 

Youth Program is cost per placement in employment or education. In the longer term, 

the possibility of collecting separate cost and customer data for in-school and out-ofschool youth should be investigated so that appropriate separate outcome and 

efficiency measures can be developed for these disparate groups. For example, if 

separate cost data were available for these two groups, it would feasible and perhaps 

appropriate to apply the four measures recommended for the WIA Adult and 

Dislocated Worker Programs to the older youth served by the WIA Youth Program (i.e., 

cost per entered employment, cost per retained in employment, cost divided by postprogram earnings, and cost divided by pre/post-program earnings). 
Final Report – Implementing Efficiency Measures for ETA Programs Page ES-8

Recommendation #4: Performance Standards for States/Grantees on Recommended 

Efficiency Measures Should Be Considered Exploratory at This Time -- Do Not 

Reward or Sanction States/Grantees for Performance on the Recommended Efficiency 

Measures 

It is recommended that ETA track efficiency measure results for ETA programs of 

interest for several program years for program monitoring purposes only. Several years of 

experience are needed with the efficiency measures (perhaps three or more years) to determine if 

it is appropriate to set standards and apply rewards and sanctions to states and grantees on the 

efficiency measures (as is currently done for outcome measures). The additional time is needed 

to identify definitional problems (particularly with respect to expenditures to be included), allow 

for co-enrollment patterns to stabilize, analyze variation in performance on the efficiency 

measures across states, and determine if and how the standards should be adjusted to take into 

account various factors. It is also important to assess factors that account for variation across 

states/grantees on efficiency outcomes, as well. With the rapid changes in co-enrollment 

patterns underway in many states, it will be possible to assess how the large increases in 

participant and exiter counts affect the outcome-based efficiency measures (and outcomes for 

other Common Measures) over the next few years. 

Recommendation #5: Improve Consistency and Quality of Cost, Customer 

Characteristics, and Outcome Data

This study has attempted to establish a baseline of efficiency measure results for the 11 

ETA programs -- an effort that has not always resulted in success. For some programs (notably 

the WIA, ES, and TAA programs), there was success in obtaining both the cost and outcome 

data to generate three years of efficiency measure results both at the national and state levels. 

Other programs struggled with providing data for the full three-year period and some could not 

generate the outcome data needed for even one year (in part, because they may have recently 

transitioned to the Common Measures or were in the process of making this transition). The 

efficiency measure results for the programs able to provide three years of data at the state or 

grantee level revealed substantial variation in results within states from year to year and among 

states/grantees within a single program year. These variations were sometimes very large, and it 

is not clear that this variation is a true reflection of the “efficiency” of programs or of other 

factors – including substantial cross-state differences in co-enrollment patterns and in the ways 

in which states collect and report on both program costs and outcomes, as well as simply 

erroneous data. Given some difficulties in obtaining cost and outcome data for some programs at 

the federal and state/grantee levels – and the variable nature of efficiency measure results 

generated from the data provided -- we suggest that ETA carefully study variation in the ways in 

which expenditure and outcome data are being collected within programs (across states) and 

across programs, and require that states and grantees use common definitions and procedures to 

report data to ETA and that they be required to assure the quality of the data reported. 
Final Report – Implementing Efficiency Measures for ETA Programs Page ES-9

Recommendation #6: ETA Should Explore Developing Efficiency Models by Activity 

for Programs That Offer a Range of Activities

Some programs, such as WIA Adult and Dislocated Worker programs, offer a wide range 

of activities in terms of the cost per customer. For example, assisted core services may cost well 

under $100 per customer, but occupational training sometimes costs $10,000 or more per 

customer. The concept of cost per customer loses much of its meaning when services vary so 

much within and across states and grantees. More meaningful results are likely to be obtained 

for programs with a range of activities if separate regression models could be estimated for each 

major activity. Also, regression results vary over time as service strategies or population served 

changes. In WIA, for example, separate efficiency models could be estimated for each of the 

three tiers of service or for training and core assisted and intensive services combined. 

Unfortunately, it is not a simple matter to develop such models because cost data are currently 

not collected by activity in most states. To develop statistical models of efficiency measures, 

ETA would have to mandate that states/grantees and, most likely substate grantees as well, 

collect cost data by activity. To assure that the data are consistent across states, ETA would also 

have to require that a specific method be used to allocate joint costs and to deal with coenrollment. We recognize that this would be a considerable burden, so our recommendation at 

this time is that ETA explores this issue further rather than immediately begin collecting such 

data. 

Recommendation #7: Adjustment Models for Efficiency Measures Are Not Likely to 

Be Useful at the State-Level for Many Years; They Potentially Could be Useful and 

Valid at the Local/Grantee Level for Some Programs

 The modeling efforts conducted for this report, which focused on the WIA Adult and 

Dislocated Worker programs (using WIASRD data), suggest that great caution and several 

additional years of results are required before ETA should consider implementing state-level 

regression models to adjust performance standards for states/grantees on the recommended 

efficiency measures. Overall, the regression models tested for the WIA Adult and Dislocated 

Worker programs did not provide sensible magnitudes and statistically significant coefficients 

for the regression coefficients in the models. It is possible that once the co-enrollment pattern 

stabilizes and several years of data after that are available, regression modeling might produce 

useful results. Other factors that could make such modeling useful include requiring states (and 

local areas where appropriate) and grantees to use consistently measured and high-quality data 

(see Recommendation 5) and that activity-level cost data be collected (see Recommendation 6). 

Recommendation #8: Estimate Return on Investment (ROI) in Conjunction with 

Impact Studies but Not as Regular Performance Measurement

ROI and the closely related concept of cost-benefit analysis are essential to assess if a 

program is a worthwhile investment and to compare alternative investments -- we encourage the 

Department of Labor and states to conduct such analyses on a regular basis. However, it would 

be very expensive to measure ROI on an annual basis as a performance measure. Other 
Final Report – Implementing Efficiency Measures for ETA Programs Page ES-10

challenges to using ROI on a regular basis as a performance measure are (1) a lack of consensus 

about the best methods that should be used to generate appropriate comparison group data (e.g., 

some analysts believe that the matching methods that are widely used do not provide good 

comparison groups); (2) the time required to generate reasonable post-program data is too long to 

provide measures that are useful for annual performance measures; (3) there is a lack of 

consensus among economists and government agencies about the appropriate discount rate to 

use; and (4) for programs that provide earnings gains, it is difficult to make reasonable 

assumptions about how long observed earnings gains will persist. Thus, it is recommended that 

ROI and cost-benefit analysis should be considered an important tool for periodic program 

evaluation rather than annual performance assessment. 

Recommendation #9: Further Study Is Needed on Several Topics Related to and Likely 

to Affect Efficiency Measure Results, Including Co-Enrollment and Cost Sharing

Through our conversations with ETA officials and states, we identified a number of 

important issues that must be resolved so that ETA’s efficiency measures are consistent, valid, 

and reliable. These issues involve policy considerations that can only be made by ETA officials. 

For example, some programs, such as SCSEP and TAA, provide stipends or other cash payments 

on a regular basis; whether expenditures on stipends are included as an expenditure item would 

have a major effect on how costs compare across programs. Examples of such issues include: 

• Should stipends, support services, and need-based payments be counted as 

expenditures in computing efficiency measures?

• How should co-enrollment be accounted for in efficiency measures?

• How should shared costs across programs, such as One-Stop Career Center 

infrastructure costs, be dealt with in efficiency measures?

• How should the efficiency measures deal with customers who remain in the program 

for more than one program year?

• How should self-service customers be dealt with in programs such as WIA Adult and 

Dislocated Worker programs where such customers are not counted for outcome 

measures?

• How should incumbent worker customers be dealt with in programs such as WIA 

Adult and Dislocated Worker programs where such customers are not counted for 

outcome measures?

Resolution of these issues is important for making sure that the efficiency measures are 

consistent within and across programs and to assure that programs and policy officials 

understand what is and is not being captured by the efficiency measures. As noted above, these 

issues generally require value judgments and some of them would have major cost implications 

for data collection. 
Final Report – Implementing Efficiency Measures for ETA Programs Page 1 

CHAPTER 1: 

INTRODUCTION 

A. STUDY BACKGROUND, OBJECTIVES, AND METHODOLOGY 

 In 2007, the Office of Management and Budget (OMB) required development of 

efficiency measures for all federal government programs as part of the effort to improve federal 

government program performance. As a result of ongoing Program Assessment Rating Tool 

(PART) reviews, OMB charged the Employment and Training Administration (ETA) with the 

responsibility of developing and implementing an outcome-based measure or measures of 

efficiency for employment and training programs administered by the agency.3 Apart from 

OMB’s interest in the development of efficiency measures, ETA has a long-term interest in 

improving program efficiency so that both taxpayers and customers can be better served. In 

response to this OMB directive, in May 2008, ETA initiated a study aimed at identifying 

outcome-based efficiency measures for implementation by 11 ETA-administered programs: 

• Workforce Investment Act (WIA) Adult Program; 

• WIA Dislocated Worker Program; 

• WIA Youth Activities Program; 

• WIA National Emergency Grants (NEG) Program; 

• Trade Adjustment Assistance (TAA) Program; 

• Wagner-Peyser/Employment Service (ES) Program; 

• Senior Community Service Employment Program (SCSEP); 

• National Farmworker Jobs Program (NFJP); 

• Indian and Native American Program (INAP); 

• Work Incentive Grant (WIG) Program; and 

• Apprenticeship Program. 

 

3

Executive Office of the President, Office of Management and Budget, Program Assessment Rating Tool Guidance 

No. 2007-7: Improving the Quality of PART Performance and Efficiency Goals, issued December 12, 2007 

(available at: http://georgewbush-whitehouse.archives.gov/omb/performance/guidance/part_guid_2007-07.pdf). 

The Program Assessment Rating Tool Guidance No. 2007-7 called for a government-wide initiative “to increase the 

government and public’s ability to assess and improve a program’s effectiveness and efficiency.” 
Final Report – Implementing Efficiency Measures for ETA Programs Page 2 

To support this intensive effort, ETA formed an internal work group composed of 

program officials representing each of the above-referenced programs and each of the DOL 

regional offices.4

 The research effort was conducted in two phases – (1) an initial study phase 

(conducted from June 2008 through December 2008), which was primarily aimed at identifying 

one or more outcome-based efficiency measures (based primarily on the Common Measures) 

that could be uniformly implemented across the 11 ETA programs and a plan for implementing 

selected measures; and (2) a second study phase (conducted from January 2009 through 

December 2009, during a period of transition within the White House and DOL) which stressed a 

more cautious and flexible approach to identifying and implementing efficiency measures that 

would reflect the varying goals, types of populations served, and services provided by each of 

the 11 ETA programs of interest. The initial study phase, with a focus on implementation of one 

or more outcome-based efficiency measures across the 11 ETA programs (starting as early as 

June 2009), was executed in three stages:5

 

• Stage I--Identification of efficiency measure options for consideration and additional 

analysis (completed June 30, 2008 and presented in Interim Report #1);6

• Stage II--Analysis and preliminary recommendations of an efficiency measure or 

measures for implementation by ETA programs starting as early as June 2009 (completed 

September 30, 2008 and presented in Interim Report #2);7

 

4

 To support this effort, an independent team of researchers (from Capital Research Corporation and the Johns 

Hopkins University Institute for Policy Studies) was contracted to identify and analyze efficiency measure options 

and recommendations. An Expert Panel (of leading academicians in the field of performance measurement) also 

provided an independent external review of interim reports, and this final report. 

5

 For additional details about the purpose and scope of the research effort during study Phase I, as well as planned 

data collection, analysis, and report preparation, see: Burt Barnow and John Trutko, Improving Measures of 

Efficiency for Employment and Training Programs: Project Workplan, prepared for the U.S. Department of Labor, 

Employment and Training Administration, prepared by Capital Research Corporation and Johns Hopkins 

University’s Institute for Policy Studies, June 2008. 

6

Burt Barnow and John Trutko, Improving Measures of Efficiency for Employment and Training Programs: Interim 

Report #1: Efficiency Measure Options for Selected ETA Programs, prepared for the U.S. Department of Labor, 

Employment and Training Administration, prepared by Capital Research Corporation and Johns Hopkins 

University’s Institute for Policy Studies, June 2008. 

7

 Burt Barnow and John Trutko, Implementing Efficiency Measures for Employment and Training Programs: Final 

(Year 1) Report: Recommended Efficiency Measures for Selected ETA Programs, prepared for the U.S. Department 
Final Report – Implementing Efficiency Measures for ETA Programs Page 3 

• Stage III-- Preparation of a First Year report, which included two recommended 

efficiency measures for implementation starting as early as July 2009 and an assessment 

of potential implementation challenges (completed in December 2008 and presented in 

the First Year Final Report).8

 

Phase II of this study was initiated in January 2009 during a time of change in 

administrations (from the George W. Bush to the Barack Obama Administration) and featured a 

shift in approach to efficiency measurement across the 11 program of interest. In early 2009, 

during the transition period, the research team conducting this study met with ETA officials to 

discuss several important shifts that had occurred in underlying thinking about efficiency 

measures and their potential application to the ETA programs, including the following: 

• Recognition that application of a single, uniform efficiency measure, such as cost per 

entered employment or cost divided by post-program earnings, may not be desirable 

across all 11 programs. ETA was now more open to greater flexibility and tailoring of 

efficiency measures (and definitions) to individual program goals, types of individuals 

served, and data availability. In particular, alternative efficiency measures and 

definitions might be appropriate for the WIA Youth, WIG, SCSEP, and Apprenticeship 

programs. In addition, there may be additional alternative measures that would be 

appropriate for implementation by one or more other programs. 

• Within ETA/OMB, there was emerging interest in the feasibility of implementing a 

Return on Investment (ROI) measure. While such a measure would depend upon having 

reliable impact estimates for individual programs on an annual basis, ETA officials noted 

that ROI could potentially be very useful (if feasible) for systematically linking overall 

program costs and benefits. As a result, ETA officials suggested that an assessment was 

needed of the potential for applying ROI measures to specific programs, particularly 

those such as WIA where participant impacts have been measured in the past. Also, an 

assessment was needed of when it might be feasible to apply an ROI measure (e.g., as a 

result of periodic studies) and what steps individual programs would need to undertake to 

effectively implement a ROI-type measure. 

• There was also increased interest in whether it might be feasible to disaggregated costs by 

service level (e.g., unassisted, intensive, and training services) in programs such as the 

 of Labor, Employment and Training Administration, prepared by Capital Research Corporation and Johns Hopkins 

University’s Institute for Policy Studies, June 2008. 

8

 Burt Barnow and John Trutko, Improving Measures of Efficiency for Employment and Training Programs: 

Interim Report #1: Efficiency Measure Options for Selected ETA Programs, prepared for the U.S. Department of 

Labor, Employment and Training Administration, prepared by Capital Research Corporation and Johns Hopkins 

University’s Institute for Policy Studies, March 2009. 
Final Report – Implementing Efficiency Measures for ETA Programs Page 4 

Workforce Investment Act – and then potentially, link efficiency measures to service 

levels. 

• ETA officials expressed heightened concern for a range of challenges that could 

potentially stand in the way of equitably implementing efficiency measures (and 

performance standards associated with those measures). In particular, ETA officials 

pointed to challenges such as cost sharing and co-enrollment – hallmarks of the One-Stop 

Career Center system – that could pose formidable barriers to implementing efficiency 

measures that equitably take into consideration costs of providing services and the 

numbers of customers served by programs. ETA officials indicated the need to carefully 

assess these challenges – and other potential issues – that could affect efficiency measure 

results national, at the state/grantee levels, and the local levels. 

ETA officials indicated that as a result of these underlying shifts in views on efficiency 

measures, the schedule for implementation of efficiency measures by ETA programs was likely 

to be pushed back, and when (and if) implementation of such measures occurred, it was not 

necessarily optimal to have efficiency measures uniformly applied across the 11 programs. 

Rather, a more nuanced approach was emphasized – one that would more carefully tailor 

efficiency measures to individual programs and take into consideration both potential benefits 

and adverse effects of efficiency measures (such as potentially decreasing emphasis on more 

costly intensive/training services). Finally, because of the likelihood that decisions concerning 

implementation of efficiency measures by individual ETA programs, ETA officials indicated it 

would be important for the final report to fully document the results of study activities and serve 

as a resource for assisting ETA in making informed decisions concerning future selection and 

implementation of efficiency measures for some or all of the 11 ETA programs. 

The study findings and recommendations offered in this report are based on the following 

data collection and analysis activities, conducted over the full length of this study (from June 

2008 through the end of the study in December 2009): 

• a review of background literature on performance measurement and management, with a 

focus on efficiency measurement in employment and training programs; 
Final Report – Implementing Efficiency Measures for ETA Programs Page 5 

• an initial round of telephone interviews of officials at workforce agencies in six states 

(California, Florida, Missouri, New York, Utah, and Virginia) conducted in the early 

stages of this study (July/August 2008), which focused on performance measurement 

systems and use of efficiency measures in employment and training programs; 

• telephone interviews with officials from the Organization for Economic Cooperation and 

Development (OECD), the United Kingdom, and Canada that explored use of efficiency 

measures for monitoring employment and training program performance in other 

developed countries; 

• analysis of efficiency measures used in other federal agencies (including the Department 

of Health and Human Services (DHHS), Department of Education (DOE), Department of 

Housing and Urban Development (HUD), U.S. Department of Agriculture (USDA), and 

the Department of Veterans Affairs (VA); 

• a second round of telephone interviews with officials at workforce agencies in five states 

-- California, Maryland, Ohio, New York, and Washington -- conducted near the end of 

the study (September/October 2009), aimed at gaining qualitative input on setting of 

efficiency measure standards, as well as potential effects of co-enrollment, One-Stop selfservice customers, cost sharing, and other selected issues on efficiency measure results of 

states, grantees, and local workforce areas. 

• collection/analysis of aggregate cost and outcome data for 11 ETA programs, intended to 

produce national and state/grantee level results on a range of efficiency measures, as well 

as to support multivariate regression analysis of potential adjustment models that could 

be used in setting/adjusting performance standards on efficiency measures; and 

• ongoing review and comment by ETA program offices, ETA regional offices, an ETA 

internal Cross-Functional Performance workgroup, and an external Expert Panel.9

The purposes of these data collection and analyses activities were to assess the feasibility and 

appropriateness of the efficiency measures for implementation by the 11 ETA programs of 

interest, as well as to identify and carefully assess the potential challenges associated with 

implementation of efficiency measures and the likely short- and long-term effects (both 

beneficial and adverse) on ETA programs. The study was also aimed at producing a set of 

recommendations for implementation of one or more efficiency measures by the 11 ETA 

programs for consideration by ETA. 

 

9

 The Expert Panel and ETA staff are identified in the acknowledgements page of this report. 
Final Report – Implementing Efficiency Measures for ETA Programs Page 6 

This report is organized into eight chapters. This first chapter is intended to provide 

background on the study purpose, scope, and methodology. In addition, the remainder of this 

chapter provides some basic background on efficiency measurement (including definitions and a 

possible criteria for selection of efficiency measures appropriate to ETA workforce programs), 

followed by a listing of candidate efficiency measures that could be considered for 

implementation across the 11 ETA programs and an initial assessment of potential advantages 

and disadvantages of each of the candidate measures. Chapter 2 reviews the literature on 

performance measurement, as well as the Employment and Training Administration’s prior use 

of and experience with efficiency measures/standards in ongoing monitoring of workforce 

investment programs. Chapter 3 provides a preliminary assessment of the types of measures 

that have been used by government agencies in the United States and in several other countries, 

including lessons learned from use of such measures. Chapter 4 provides a summary of 

perspectives of state workforce officials on prior experience with efficiency measures, as well as 

views on a range of implementation issues associated with potential future adoption by ETA of 

outcome-based efficiency measures. The results of two rounds of interviews with states 

workforce agency officials are discussed in this chapter, including state administrators’ 

perspectives on the appropriateness and feasibility of implementing efficiency measures for the 

11 ETA programs, specific implementation challenges, and likely impacts should efficiency 

measures be applied to some or all of the ETA programs. 

Chapter 5 focuses on the cost portion of the efficiency measure calculation, examining 

the three options available for measuring costs, providing a recommendation with regard to 

which cost type should be used, and detailing challenges to appropriately and consistently 

capturing costs when measuring program efficiency. Chapter 6 examines a range of outcome 
Final Report – Implementing Efficiency Measures for ETA Programs Page 7 

measures that could potentially be coupled with cost data to produce outcome-based efficiency 

measures for implementation by some or all of the ETA programs. The chapter then presents 

quantitative and qualitative analyses on a core set of the outcome-based efficiency measures with 

the aim of narrowing and focusing study recommendations on a set of feasible and relevant 

efficiency measures for possible implementation by the 11 ETA programs. This chapter also 

presents efficiency measure results for the past three program years for most of the 11 programs, 

both at the national level and at the state/grantee levels. 

Chapter 7 presents alternative approaches to setting standards for ETA programs and 

alternative approaches to making adjustments for factors that may affect outcomes and costs of 

ETA programs. This chapter highlights ETA’s historical experience with performance 

measurement, focusing on setting performance standards and making adjustments. The chapter 

then reports on the results of modeling efforts conducted under this study, focusing on the three 

WIA programs (using WIASRD data). 

The final chapter (Chapter 8) provides a set of study recommendations for 

implementation of specific outcome-based efficiency measures for the 11 ETA programs. 

B. DEFINITION AND CRITERIA FOR SELECTION OF EFFICIENCY 

MEASURES 

In its 2007 guidance letter to federal agencies, OMB noted that the “Government’s ability 

to determine a program’s effectiveness, and to direct attention to genuinely desired outcomes, is 

largely dependent upon the quality of the program’s performance and efficiency goals, i.e., their 

measures and targets.” OMB also provided a basic definition of and purpose for generating such 

efficiency measures – terming such measures as “efforts to provide the most benefits (outcomes 

and outputs) for the taxpayer dollars spent.” Finally, this OMB guidance letter underscored the 
Final Report – Implementing Efficiency Measures for ETA Programs Page 8 

urgency for federal agencies to refocus their attention on improving the types of efficiency 

measures used to assess ongoing program performance: “...a substantial portion of PART 

[Program Assessment and Rating Tool] efficiency measures need to be revised in order to meet 

current PART guidance.”10 

Frequently, in employment and training programs (and more broadly for all government 

human service programs), efficiency measures use unit costs tied to one or more of the 

following: (1) participation (such as “cost per participant served or exited”); (2) delivery of 

services (such as “cost per participant trained”); or (3) outcomes (such as “cost per participant 

entering or retaining employment”).11 An underlying concept for many efficiency measures is 

that a unit cost for a particular time period (usually a program or fiscal year) is produced by 

dividing program costs by the number of participants served/exited, the number of 

participants/exiters receiving a particular service, or the number of participants/exiters achieving 

a certain outcome. For example, if a local workforce investment board spent $100,000 on an 

employment and training initiative in a program year and was able to place 50 of the individuals 

served (or exited) into employment during that same program year, the “cost per entered 

employment” for the program year would be $2,000 ($100,000 divided by 50). Some potential 

uses of such efficiency measures for programs (such as the 11 ETA programs that are part of this 

study) include the following: 

• to measure performance on outcomes relative to costs; 

• to compare efficiency over time for a single program; 

 

10Executive Office of the President, Office of Management and Budget, Program Assessment Rating Tool Guidance 

No. 2007-7: Improving the Quality of PART Performance and Efficiency Goals, issued December 12, 2007 

(available at: http://georgewbush-whitehouse.archives.gov/omb/performance/guidance/part_guid_2007-07.pdf). 11For example, GAO identifies several potential types of performance measures, making a distinction between three 

types of potential measures: “Performance measures may address the type or level of program activities conducted 

(process), the direct products and services delivered by a program (outputs), and /or the results of those products and 

services (outcomes).” United States General Accounting Office, Glossary: Performance Measurement and 

Evaluation -- Definitions and Relationships, GAO/GGD-98-26, April 1998 (available at: 

http://www.gao.gov/special.pubs/gg98026.pdf.).
Final Report – Implementing Efficiency Measures for ETA Programs Page 9 

• to compare efficiency across programs or services; 

• to compare efficiency across states or grantees (including identifying outliers for 

possible corrective actions); 

• to promote continuous improvement in program efficiency; 

• to help inform or support decisions about how to best to allocate or target scarce 

resources within or across programs; and 

• to meet external requirements (from OMB, Congress, and the public) for ongoing 

performance measurement and accountability. 

 While the definition of efficiency measures is relatively straightforward, there are a 

number of serious challenges to identifying appropriate and feasible measures for a single 

program – and these challenges multiply when consideration is given to applying an efficiency 

measure or measures across more than one program. The efficiency measure or measures 

applied to one or more programs should at a minimum meet the following four criteria: 

• be fair to the programs being judged; 

• encourage desired service delivery and program outcomes; 

• discourage undesired strategies and behaviors; and 

• maintain program quality, integrity, and fiscal responsibility. 

In addition, the efficiency measures recommended for a specific program should be feasible and 

cost-effective to implement using participant, outcome, and cost data that are currently collected 

or that could potentially be collected in the future at a reasonable cost (such as data collected and 

reported as part of the Common Measures).12 

Although discussed in much greater detail later in this report, among the challenges to 

developing and implementing efficiency measures that are feasible, useful, and appropriate 

across the 11 ETA programs are the following: 

• Programs have differing objectives – In addition to performance goal indicators for 

employment, retention, and improving earnings – which are objectives typically shared 

across most employment and training programs – some programs emphasize other 

objectives. For example, some programs place more emphases on attaining educational 

 

12For additional background on the Common Measures, see Training and Employment Guidance Letter No. 17-05, 

“Common Measures Policy for the Employment and Training Administration’s (ETA) Performance Accountability 

System and Related Performance Issues” (available at: http://wdr.doleta.gov/directives/corr_doc.cfm?DOCN=2195). 
Final Report – Implementing Efficiency Measures for ETA Programs Page 10 

degrees or other recognized credentials, such as achieving a high school diploma or GED 

(as is the case in the WIA Youth program) or journeyman status (as is the case in 

Apprenticeship programs).13

• Measuring some outcomes is difficult – While entered employment and increases in 

earnings may be relatively straightforward to measure, other outcomes are more difficult 

to measure or impose costly (or burdensome) data collection on reporting entities (such 

as states and local program grantees).14 For example, under the Employment 

Service/Wagner-Peyser program, reduction in job vacancy time for employers and 

shortening duration of unemployment spells for workers are important, but also difficult 

to measure outcomes. 

• Measuring efficiency in a One-Stop environment is complicated – One-Stop Career 

Centers emphasize leveraging resources, sharing space and other costs, and co-enrollment 

of participants with a focus on services rather than programs. These types of strategies 

complicate tracking and applying costs to services received and outcomes for co-enrolled 

participants. Methodologies for estimating costs of concurrent or sequential services by 

programs are needed where there are high degrees of service integration.15

In determining whether (and which) outcome-based efficiency measures should be implemented 

by specific ETA programs, it is essential to take into account both the criteria suggested above 

(i.e., appropriateness, fairness, cost to implement, etc), as well as the challenges likely to be 

faced by each program in implementing selected measures. 

C. CANDIDATE EFFICIENCY MEASURES 

In the early stages of this study, the research team identified and considered a range of 

candidate process and outcome measures that could potentially be combined with cost measures 

to produce one or more efficiency measures to be tracked across the 11 ETA programs. The 

 

13For in-school youth, ironically employment may be an “unsuccessful” outcome, at least in the short run, as it may 

reduce attainment of important credentials. 

14For training programs, for example, it would be preferable (though potentially burdensome) to measure attainment 

of employment or earnings changes due to the program (e.g., using experimental design). 

15An added challenge for implementation of efficiency measures within ETA programs is setting of performance 

standards (i.e., requirements for achieving satisfactory performance on the measure), and adjustments on each 

selected efficiency standard for states/grantees, and/or local workforce areas to take into account underlying factors 

such as environmental/economic conditions, case mix (e.g., demographic characteristics of individuals served), and 

the mix of services provided (e.g., intensity and types of services provided). If efficiency measures and standards 

are implemented by ETA programs, it will be important to monitor and assess the effects of such measures on 

program operations and performance, particularly related to intended and unintended consequences. 
Final Report – Implementing Efficiency Measures for ETA Programs Page 11 

efficiency measures identified and recommended for consideration for implementation by the 11 

ETA programs were identified based on the following activities: (1) review of the literature on 

performance measurement, particularly related to monitoring employment and training 

programs; (2) initial discussion with members of the ETA Work Group (which included 

representatives from each of the 11 ETA programs, as well as ETA national and regional office 

staff); (3) review of current and past performance measurement requirements for each of the 

ETA programs; and (4) review of data systems and data elements collected by each of the ETA 

programs (particularly relating to program participation, services received, outcomes, and 

costs/expenditures). Exhibit 1-1 provides an overview of candidate efficiency measures the 

research team identified for consideration by ETA and OMB, broken down by measures for 

“short-term” and “long-term” consideration. Efficiency measures for short-term consideration – 

that is, measures that ETA was already collecting and potentially could be implemented within 

one to three years across some or all of the 11 ETA programs -- were the following: 

• Cost per participant; 

• Cost per exiter; 

• Cost per entered employment; 

• Cost per retained employment; 

• Cost divided by increase in earnings (or cost per $1 increase in earnings); 

• Cost divided post-program earnings (or cost per $1 post-program earnings); 

Efficiency measures identified in the early stages of the research study for longer-term 

consideration (which could be phased in over time and applied to one or more ETA programs) 

were the following: 

• Cost per exiter or participant receiving a particular service (such as an intensive or 

training service); 

• Cost per positive outcome (e.g., placed in a job, the military, a registered apprenticeship 

program, education program, or advanced training program); 

• Cost per recognized credential received; and 

• Return on Investment (ROI) 
EXHIBIT 1-1: CANDIDATE EFFICIENCY MEASURES FOR POSSIBLE IMPLEMENTATION BY 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 12 

Potential Efficiency Measures 

Definition of Measure 

(i.e., Numerator/ 

Denominator) 

Arguments for and Against Using Measure Applicability/Use of M

A. EFFICIENCY MEASURE ALTERNATIVES -- FOR SHORT-TERM CONSIDERATION Cost per Participant 

Total Program Cost/  # of Participants Served 

-Applicable to most/all programs. -Data is readily available -Easy to understand -No lags in data (i.e., inter-temporal issues)– can be immediately generated at the end of each year -Not costly or burdensome to produce -Is not an “outcome-based” efficiency measure – so of limited use in assessing program effectiveness- 

-Applicability: Measure is potprograms (with possible except-Use: Of only limited use, beca

outcome. Also, measure does economic conditions – which coprograms and programs targetisubpopulations and services. 

Cost per Exiter 

Total Program Cost/  # of Exiters 

-Applicable to most/all programs. -Data is readily available -Easy to understand -No lags in data (i.e., inter-temporal issues)– can be immediately generated at the end of each year -Not costly or burdensome to produce -Is not an “outcome-based” efficiency measure – so of limited use in assessing program effectiveness 

-Applicability: Measure is pot(with exception of programs wh-Use: Of only limited use, beca

outcome. Also, measure does economic conditions – which coprograms and programs targetisubpopulations and services. 

Cost per Entered Employment 

Total Program Cost/ # of Exiters or Participants Entering Employment in the 1st

Quarter Following Exit 

-Potentially applicable to most/all programs -Focuses on a key outcome of interest in most workforce programs – whether a participant obtains employment -Data is readily available -Relatively easy to understand -Short lags in data (i.e., inter-temporal issues)– can be generated (using UI wage record data) about two quarters after the end of program year -Relatively low cost and low burden to produce (based on existing administrative data which can be matched by Social Security Number) -Measure is an outcome-based efficiency measure -- so is of substantial use to understanding program effectiveness 

-Applicability: Is potentially athough may not be as appropriprograms where individuals are and Apprenticeship); aimed at scapacity building (NEG and WI(WIA-Youth and Apprenticeship(WIG). -Use: Substantial use for unde

cost-effectiveness because cosdoes not control for case mix, swhich could result in unfair comtargeting services on less costl

Cost per Retained Employment 

Total Program Cost/ # of Exiters or Participants Retained in Employment for Two Quarters Following Exit 

-Potentially applicable to most programs -Data is readily available -Focuses on a key outcome of interest in most workforce programs – whether a participant obtains employment -Relatively easy to understand -Lengthier lags in data (i.e., inter-temporal issues)– can be generated (using UI wage record data) several quarters after the end of program year 

-Applicability: Is potentially athough may not be as appropriprograms where individuals are Apprenticeship), aimed at subsbuilding (NEG and WIG), have Youth and Apprenticeship), or d-Use: Substantial use for unde
EXHIBIT 1-1: CANDIDATE EFFICIENCY MEASURES FOR POSSIBLE IMPLEMENTATION BY 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 13 

Potential Efficiency Measures 

Definition of Measure 

(i.e., Numerator/ 

Denominator) 

Arguments for and Against Using Measure Applicability/Use of M

(depending on post-program period) -Relatively low cost and low burden to produce (based on existing administrative data which can be matched by SSN) -Measure is an outcome-based efficiency measure -- so is of substantial use to understanding program effectiveness and costs 

cost-effectiveness because cosdoes not control for case mix, swhich could result in unfair comtargeting services on less costl

Cost Divided by Increase in Earnings 

Total Program Cost/ Total Earnings Change from 2nd and 3rd

Preprogram Quarters to 2nd and 3rd Post-program 

Quarters for Participants or Exiters 

-Potentially applicable to most programs -Data is readily available -Focuses on a key outcome of interest in most workforce programs – whether a participant improves earnings (compared to earnings prior to participation) 

-Somewhat more difficult to understand than other potential measures -Lengthier lags in data (i.e., inter-temporal issues)– can be generated (using UI wage record data) several quarters after the end of program year (depending on post-program period) -Relatively low cost and low burden to produce (based on existing administrative data which can be matched by SSN) -Measure is an outcome-based efficiency measure -- so is of substantial use to understanding program effectiveness and costs 

-Applicability: Is potentially athough may not be as appropriprograms aimed at subsidized building (NEG and WIG), have Youth and Apprenticeship), or d-Use: Substantial use for unde

cost-effectiveness because cosdoes not control for case mix, swhich could result in unfair comtargeting services on less costl

Cost per Divided by Post-Program Earnings 

Total Program Cost/ Total Earnings in 2nd and 

3rd Post-program 

Quarters for Participants or Exiters 

-Potentially applicable to most programs -Data is readily available -Focuses on a key outcome of interest in most workforce programs – stream of earnings after exit from the program -Somewhat more difficult to understand -Lengthier lags in data (i.e., inter-temporal issues)– can be generated (using 

UI wage record data) several quarters after the end of program year (depending on post-program period) -Relatively low cost and low burden to produce (based on existing administrative data which can be matched by SSN) -Measure is an outcome-based efficiency measure -- so is of substantial use to understanding program effectiveness and costs 

-Applicability: Is potentially athough may not be as appropriprograms aimed at subsidized building (NEG and WIG), have Youth and Apprenticeship), or d-Use: Substantial use for unde

cost-effectiveness because cosdoes not control for case mix, swhich could result in unfair comtargeting services on less costl
EXHIBIT 1-1: CANDIDATE EFFICIENCY MEASURES FOR POSSIBLE IMPLEMENTATION BY 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 14 

Potential Efficiency Measures 

Definition of Measure 

(i.e., Numerator/ 

Denominator) 

Arguments for and Against Using Measure Applicability/Use of M

B. EFFICIENCY MEASURE ALTERNATIVES -- FOR LONG-TERM CONSIDERATIONCost per Exiter or Participant Receiving a Particular Service 

Total Program Cost (on Particular Service)/  # of Exiters or Participants Receiving a Particular Service 

-Only applicable to programs that distinguish types of services (e.g., intensive versus non-intensive services or training versus non-training) -Data is readily available for some programs, but not all -Relatively easy to understand – though types of services vary across programs (e.g., intensive or training services may or may not be offered by a program or may not be consistently defined across programs) -No lags in data (i.e., inter-temporal issues)– can be immediately generate at the end of each year -May be burdensome to generate because labor and other costs have to be allocated to particular service and whether individuals receive a particular services (e.g., intensive or training service) must be documented -Is not an “outcome-based” efficiency measure – so of limited use in assessing program effectiveness 

-Applicability: Potentially appservice would likely be differenservices was the service measto WIA Adult, WIA-DW, WIA-YoSCSEP, NFJP, INAP; not appli-Use: Of only limited use, beca

outcome. Measure does adjusnot control for case mix or econunfair comparisons across progon less costly subpopulations. 

Cost per Placement in Employment or Education 

Total Program Cost/ # of Participants or Exiters in Employment (Including Military) or Enrolled in PostSecondary Education and/or Advanced Training/Occupational Skills Training in the 1st

Quarter After Exit 

-Limited applicability – primarily to WIA-Youth program, though could be applied to other programs providing training related services -Data is readily available for WIA-Youth program and could potentially be generated by other programs providing training -Relatively easy to understand -Short lags in data (i.e., inter-temporal issues)– employment data can be generated (using UI wage record data) about two quarters after the end of program year; data on enrollment in education/training potentially immediately available at end of first quarter after exit -Relatively low cost and low burden to produce for WIA-Youth program (based on existing administrative data); need to gather and combine employment information with education/training and military service data if applied to other ETA programs -Measure is an outcome-based efficiency measure -- so is of substantial use to understanding program effectiveness 

-Applicability: Likely applicab

-Use: Substantial use for unde

cost-effectiveness of youth train

to an outcome. Measure does conditions – which could result and programs targeting serviceservices. 

Cost per Recognized Credential Received 

Total Program Cost/ # of Participants or Exiters Receiving a Training Service Attaining a Recognized Credential During Participation or by the End of the 3rd Quarter 

-Only applicable to programs that provide training services and identify individuals as receiving training and types of credentialing may differ across programs (e.g., under WIA Adult credentials include: HS Diploma/GED; AA/AS Diploma/Degree; BA/BS Diploma/Degree; Occupation Skills Licensure, Certificate or Credential; Other Recognized Educational or Occupational Skills Certificate/Credential) -Data is readily available for some programs, but not all 

-Applicability: Likely applicabWIA-DW, WIA-Youth, TAA, ApNEG, SCSEP, NFJP, INAP; no-Use: Substantial use for unde

cost-effectiveness because cosdoes not control for case mix oresult in unfair comparisons ac
EXHIBIT 1-1: CANDIDATE EFFICIENCY MEASURES FOR POSSIBLE IMPLEMENTATION BY 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 15 

Potential Efficiency Measures 

Definition of Measure 

(i.e., Numerator/ 

Denominator) 

Arguments for and Against Using Measure Applicability/Use of M

After Exit -Somewhat more difficult to understand – definition of “training” must be 

understood and intensity/duration/cost of training could be quite different across programs -Potentially lengthy lags in data (i.e., inter-temporal issues), though can be immediately generate at the end of each year for exiters (participants could be involved in training in programs such as Apprenticeship for 3 or 4 years) -Some burden to produce because credentialing must be tracked -Measure is an outcome-based efficiency measure -- so is of substantial use to understanding program effectiveness and costs 

services on less costly subpopu

Return on Investment (ROI) 

Solve equation for r: Impact yr 3/(1+r)3 + 

impact yr 2/(1+r)2 + 

impact yr 3/(1+r) – cost per participant = 016

-Potentially applicable to most programs -Data is very difficult and costly to produce – impact studies (with experimental design) would be needed to determine impacts of training and other treatments on participants -Difficult to understand -Lengthy lags in data (i.e., inter-temporal issues) – ROI can only be calculated after impacts are determined (which with respect to employment and earnings could take several years) -Measure is an impact-based efficiency measure, which controls for factors that could potentially influence/bias results -- so is of the greatest utility in understanding program cost-effectiveness -ROI Controls for difficulty or cost of serving different populations (e.g., hardto-serve), service mix, and economic conditions 

-Applicability: Is potentially avery costly and time-consuming impact results needed for ROI -Use: Substantial use for unde

cost-effectiveness because of ecase mix, service mix, and ecoresults in fair comparisons acro

services on less costly subpopu

16 Ideally, the ROI calculation should be based on impacts for all post-program years. The use of 3 years in the exhibit is for illustrative 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 16 

The measures identified were intended as a comprehensive listing of possible measures 

from which one or several measures could be selected and applied across some or all of the 11 

ETA programs of interest. The chapters that follow examine in much greater detail based on 

both qualitative and quantitative analyses these candidate measures, culminating in Chapter 8 in 

specific recommendations for implementing outcome-based efficiency measures for the 11 ETA 

programs of interest. The next chapter provides a review of the literature on efficiency 

measurement in employment and training programs, as a backdrop for better understanding the 

analyses, key findings, and recommendations that follow in later chapters of this report. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 17 

CHAPTER 2: 

REVIEW OF LITERATURE ON EFFICIENCY MEASUREMENT 

IN EMPLOYMENT AND TRAINING PROGRAMS 

The Employment and Training Administration has had substantial experience with 

performance standards, and a number of studies have been conducted on the impacts of 

performance management on participants served, activities, costs, and program impacts.17 While 

most analysts note the strong rationale for developing performance measures for government 

programs, there has been considerable controversy in the literature regarding the benefits of 

performance management systems, particularly as they have been applied since enactment of the 

Government Performance and Results Act (GPRA) in 1993. This section of the report reviews 

the literature on performance standards for workforce programs; most of the research was 

conducted on the performance standards system used under the Job Training Partnership Act 

(JTPA), WIA’s predecessor.18 Although much of the literature on performance management 

points to its salutary effects, there is little doubt from the literature that instituting performance 

standards can have a strong impact on program behavior, and not always in the desired direction. 

This section of the report summarizes the literature on performance standards in employment and 

training programs in several key areas: (1) the impact of performance standards on who is 

served, (2) the impact of performance standards on the services provided, (3) the relationship 

between performance measures and program impacts, (4) strategic responses by state and local 

programs to performance standards, (5) use of Return on Investment (ROI) for measuring 

 

17See Appendix B for references for the literature review included in this section of the report. 18For a more in-depth review of the literature on performance standards in workforce programs, see Burt S. Barnow 

and Jeffrey A. Smith (2004). “Performance Management of U.S. Job Training Programs: Lessons from the Job 

Training Partnership Act.” 4(3): 247-287. Most of this section is based on Barnow and Smith (2004). For a critical 

review of the performance management movement, see Beryl A. Radin (2006). Challenging the Performance 

Movement: Accountability, Complexity, and Democratic Values. Washington, DC: Georgetown University Press. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 18 

effectiveness and efficiency of workforce development programs, and (6) lessons learned by 

ETA and states/localities on the use and effects of efficiency measures/standards. 

A. THE IMPACT OF PERFORMANCE STANDARDS ON WHO IS SERVED 

The majority of the employment and training literature on performance incentives 

addresses the question of their effect on who gets served. Under JTPA, local service delivery 

areas (SDAs) had strong incentives to serve persons likely to have good labor market outcomes, 

regardless of whether those outcomes were due to JTPA. Similar incentives guide the WIA 

program. In fact, the absence of a regression model to adjust standards for serving individuals 

with labor market barriers should make these incentives stronger under WIA than they were 

under JTPA. 

The literature divides this issue into two parts. First, do SDAs (WIBs under WIA) 

respond to these incentives by differentially serving persons likely to have good outcomes, 

whether or not those good outcomes result from the effects of the program? This is the literature 

on “cream skimming.” Second, if there is cream skimming, what are its impact effects? Taking 

the best among the eligible could be economically efficient if the types of services offered by 

these programs have their largest net impacts for this group. In what follows, we review the 

literature on each of these two questions in turn. 

Do Employment and Training Programs “Cream Skim?” 

A few papers, all about the JTPA program, examine whether or not program staff cream 

skim in response to the incentives provided by the JTPA performance system. The key issue in 

this literature is the counterfactual: to what group of non-participants should the participants be 

compared in order to determine whether or not cream skimming has occurred? In all cases, the 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 19 

studies proceed by comparing observable characteristics correlated with outcomes, such as 

education levels or participation in transfer programs such as Aid to Families with Dependent 

Children (AFDC) or Temporary Aid to Needy Families (TANF). A finding that participants 

have “better” characteristics relative to non-participants in the form of higher mean years of 

schooling or lower average pre-program transfer receipt, is interpreted as evidence of cream 

skimming.

Anderson et al. (1992, 1993) compare the characteristics of JTPA enrollees in Tennessee 

in 1987 with the characteristics of a sample of JTPA eligibles in the same state constructed from 

the Current Population Survey. The literature suggests that less than five percent of the eligible 

population participated in JTPA in each year (see the discussion in Heckman and Smith, 1999), 

which allows wide scope for cream skimming. Both papers find modest evidence of cream 

skimming. In particular, the Anderson et al. (1993) analysis of program participation and postprogram job placement suggests that if eligible persons participated at random, the placement 

rate would have been 61.6 percent rather than 70.7 percent, a fall of 9.1 percentage points. 

Heckman and Smith (2004) address the issue of self-selection versus selection by 

program staff using data from the Survey of Income and Program Participation (SIPP) on JTPA 

eligibles combined with data from the National JTPA Study. They break the participation 

process for JTPA into a series of stages – eligibility, awareness, application and acceptance, and 

participation – and look at the observed determinants of going from each stage to the next. They 

find that some differences between program eligibles and participants result primarily from selfselection at stages of the participation process, such as awareness, over which program staff have 

little or no control. The evidence in Heckman and Smith (2004) suggests that while cream 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 20 

skimming may be empirically relevant, comparing the eligible population as a whole to 

participants likely overstates its extent, and misses a lot of substantive and policy-relevant detail. 

The paper by Heckman, Smith, and Taber, (1996) presents a contrasting view. They use 

data from the Corpus Christi, Texas SDA, the only SDA in the National JTPA Study for which 

reliable data on all program applicants are available for the period during the experiment. In 

their empirical work, they examine whether those applicants who reach random assignment (i.e., 

were selected to participate in the program) differ from those who do not in terms of both 

predicted outcome levels (earnings in the 18 months after random assignment) and predicted 

program impacts (projected into the future and discounted). Heckman, Smith, and Taber (1996) 

argue that it is this stage over which program staff have the greatest control, although even here 

applicants may wander off if they find employment elsewhere, get in trouble with the law, and so 

on. The authors find strong evidence of negative selection on levels combined with weak 

evidence for positive selection on impacts. They attribute the former to a strong “social worker 

mentality” toward helping the hard-to-serve among the eligible that was evident in interactions 

with program staff at the Corpus Christi site. The Workforce Investment Act (WIA) program 

offers an interesting contrast to JTPA because the WIA performance standards are not adjusted 

by a regression model, and they therefore do not hold programs harmless for the characteristics 

of their participants. Because programs now have stronger incentives to enroll individuals with 

few barriers to employment, we would expect to observe enrollment shift toward this group. An 

internal (U.S. Department of Labor, 2002) study finds that this is precisely what appears to be 

occurring, at least in the area scrutinized: 

A brief survey of States by our Chicago Regional Office indicated that WIA registrations 

were occurring at only half the level of enrollment achieved by JTPA. While some of this 

may be due to start up issues, there are indications that the reduced registration levels 

are due to a reluctance in local areas to officially register people in WIA because of 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 21 

concerns about their ability to meet performance goals, especially the “earnings gain” 

measure. It appears that local areas in these States are selective in whom they will be 

accountable for. Some local areas are basing their decisions to register a person on the 

likelihood of success, rather than on an individual’s need for services. 

A study by the U.S. General Accounting Office (GAO, 2002) confirms these problems. The 

GAO report, based on a survey of 50 states, indicated “many states reported that the need to meet 

performance levels may be the driving factor in deciding who receives WIA-funded services at 

the local level.” 

Overall, the literature provides modest evidence that program staff responded to the 

incentives provided by the JTPA performance standards system to choose participants likely to 

improve their measured performance whether or not they benefited from program services, and 

studies of the implementation of WIA indicate that, if anything, the situation has been 

exacerbated by the new program. At the same time, the evidence from the Corpus Christi SDA 

indicates that staff concerns about serving the hard-to-serve could trump the performance 

incentives in some contexts. 

What Are the Impact Implications of “Cream Skimming?”

A number of studies have examined the efficiency implications of cream skimming by 

estimating the correlation between performance measures and program impacts. Barnow and 

Smith (2004) summarize the evidence from the seven studies that comprise this literature. The 

seven papers examine a variety of different programs, ranging from the Manpower Development 

and Training Act (MDTA) program of the 1960s to Job Corps programs of today. Most rely on 

experimental data for their impact estimates. With one exception (Zornitsky et al., 1988), the 

findings are negative or mixed regarding the relationship between outcome-based performance 

measures of the type typically used in employment and training programs and program impacts. 

The (Zornitsky et al., 1988) findings refer to a program, the AFDC Homemaker-Home Health 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 22 

Aide Demonstration, which differs from programs such as JTPA and WIA in that it provided a 

homogeneous treatment to a relatively homogeneous population. Taken together, the literature 

clearly indicates that, in the context of employment and training programs, commonly used 

performance measures do not improve program impact by inducing service to those who will 

benefit most. At the same time, the literature indicates that cream skimming likely has a very 

small effect, if any, on program earnings impact. 

B. EFFECTS OF PERFORMANCE INCENTIVES ON SERVICES PROVIDED 

 

At least two papers examine the effect of performance incentives on the types and 

duration of services offered in an employment and training program, holding constant the 

characteristics of persons served. Marschke’s (2002) analysis uses the variation in performance 

incentives facing the training centers in the National JTPA Study to identify the effects of 

performance incentives on the types of services received by JTPA participants. Marschke (2002) 

finds evidence that changes in the performance measures employed in JTPA led SDAs to alter 

the mix of services provided in ways that would improve their performance relative to the altered 

incentives they faced. In some cases, these changes led to increases in efficiency, but in others 

they did not. Marschke (2002) interprets his evidence as indicating that SDAs’ service choices 

are responsive at the margin, but that existing performance measures do a poor job of capturing 

program goals such as maximizing the (net) impacts of the services provided. 

More recently, Courty and Marschke (2004) demonstrate that the JTPA performance 

management system affects the duration of training for some participants because program 

managers manipulate the duration of services for some participants in order to be able to count 

them on their performance measures for a specific program year. Courty and Marschke (2004) 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 23 

find that these manipulations reduced the overall mean impact of the employment and training 

services provided by JTPA. 

C. RELATIONSHIP BETWEEN PERFORMANCE MEASURES AND PROGRAM 

IMPACT 

Performance measures for a program may be of intrinsic interest, or they may be a proxy 

for some underlying factor of interest that is not easy to measure on a relatively quick and 

inexpensive manner. For example, Blalock and Barnow (2001) note that programs may wish to 

use program impact as a performance measure, but accurately measuring impact requires many 

years and the presence of a randomly assigned control group. Because this is not generally 

compatible with obtaining quick, inexpensive measures, programs often rely on proxy measures 

such as post-program earnings or the pre-post change in earnings. If the goal is to have 

performance measures serve as a proxy for impact, then it is necessary to assess how well the 

types of measures that are practical and have been used for the JTPA and WIA programs 

correspond with program impact. 

Two studies have explored this issue for JTPA in recent years, and another study looked 

at Job Corps programs. The studies by Barnow (2000) and by Heckman, Heinrich, and Smith 

(2002) both made use of the fact that the National JTPA Study provided experimental impact 

findings in 16 local areas and included the data needed to construct performance measures 

similar to those used by ETA. However, the approach used to measure performance does not 

include a control group, so it is not surprising that the performance measures used are at best 

weakly correlated with program impact.19

 

19A related problem is that performance measures must use short-term post-program earnings to measure 

performance, but the impact of a program is best measured over a longer period. Barnow and Smith (2004) review 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 24 

The recent evaluation of Job Corps that was based on a classical experimental design 

provided Schochet and Burghardt (2008) with an opportunity to analyze how closely Job Corps’ 

performance standards track the program’s impacts. Job Corps is a primarily residential program 

for highly disadvantaged out-of-school youth. Schochet and Burghardt indicate that during the 

evaluation period, program years 1994 through 1996, the performance measures included eight 

measures in three broad areas: (1) program achievement (reading and math gains, GED 

attainment rate, and vocational completion rate), (2) placement measures (placement rate, 

average wage at placement, and the percentage of placements related to training), and (3) 

quality/compliance measures (ratings of federal monitors). Because of the random assignment 

used to assign treatment status, impact can be estimated as the difference between treatment and 

control group values on the outcome measures. Schochet and Burghardt (2008) compared 

program impacts for Job Corps centers ranked in each third of the performance distribution. 

They concluded “Our results indicate that at the time of the National Job Corps Study, measured 

center performance was not associated with impacts on key education, crime, and earnings 

outcomes.” 

D. STRATEGIC RESPONSES TO PERFORMANCE INCENTIVES 

In addition to the substantive responses to performance incentives considered above, in 

which training centers changed what they actually did, local training programs can also attempt 

to change their measured performance without changing their actual performance. This behavior 

is referred to as a strategic response, or as “gaming” the performance system. Regardless of their 

differing goals, all types of organizations have an incentive to respond strategically to 

 the literature on the relationship between short-term earnings impacts and long-term impacts, and they find that most 

studies find a very weak relationship between the two. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 25 

performance incentives, provided the cost is low, as doing so yields additional resources to 

further their own goals. The literature provides clear evidence of such gaming behavior under 

JTPA. 

One important form of strategic behavior under JTPA was the manipulation of whether or 

not participants were formally enrolled. Under the JTPA incentive system, only persons 

formally enrolled counted towards site performance. In addition, for the first decade of JTPA’s 

existence, training centers had substantial flexibility in regard to when someone became formally 

enrolled. Clever SDAs improved their performance by basing enrollments on job placements 

rather than the initiation of services. For example, some SDAs boosted performance by 

providing job search assistance without formally enrolling those receiving it in the program. 

Then, if an individual found a job, the person would be enrolled, counted as a placement, and 

terminated, all in quick succession. Similarly, SDAs would send potential trainees to employers 

to see if the employer would approve them for an on-the-job training slot; enrollment would not 

take place until a willing employer was found. 

There are several pieces of evidence regarding the empirical importance of this 

phenomenon. The first is indirect, and consists of the fact that DOL found it enough of a 

problem to change the regulations. Specifically, in 1992, the Department of Labor required that 

individuals become enrolled once they received objective assessment and that they count as a 

participant for performance standards purposes once they received any substantive service, 

including job search assistance. 

Other evidence comes from the National JTPA Study. As part of their process analysis of 

the treatments provided at the 16 SDAs in the study, Kemple, Doolittle, and Wallace (1993) 

conducted interviews of non-enrolled members of the experimental treatment group at 12 of the 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 26 

16 sites. These results (available on Table 3.2 of their report) show that 53 percent of nonenrolled treatment group members received services, most often referrals to employers for 

possible on-the-job training (36 percent of all non-enrollees) and job search assistance (20 

percent of all non-enrollees). They report that: “... most of the study sites enrolled individuals in 

classroom training when they attended their first class or in OJT when they worked their first 

day.” There is also evidence that this type of behavior has continued under WIA. The U.S. 

General Accounting Office (2002, p. 14) notes that: “All the states we visited told us that local 

areas are not registering many WIA participants, largely attributing the low number of WIA 

participants to concerns by local staff about meeting performance levels.” 

The flexibility of JTPA also allowed strategic manipulation of the termination decision. 

Because performance standards in JTPA were based on terminees, SDAs had no incentive to 

terminate individuals from the program who were not successfully placed in a job. By keeping 

them on the rolls, the person’s lack of success would never be recognized and used against the 

SDA in measuring its performance. As the Department of Labor explains in one of its guidance 

letters, “Without some policy on termination, performance standards create strong incentives for 

local programs to avoid terminating failures even when individuals no longer have any contact 

with the program.”20 

 Problems with local programs retaining participants on the rolls long after they stopped 

receiving services go back to the days of JTPA’s predecessor, the Comprehensive Employment 

and Training Act (CETA). In one of their guidance letters, the Department of Labor observed 

that “monitors and auditors found that some participants continued to be carried in an ‘active’ or 

‘inactive’ status for two or three years after last contact with these programs.” For Title II-A of 

 

20See TEIN 5-93 available at: http://wdr.doleta.gov/directives/corr_doc.cfm?DOCN=770
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 27 

JTPA, DOL limited the period of inactivity to 90 days, although some commentators suggested 

periods of 180 days or more. 

Courty and Marschke (1996, 1997, 2004) provide additional evidence on the strategic 

manipulation of termination dates using data from the National JTPA Study. The first type of 

evidence consists of the timing of termination relative to the end of services as a function of the 

employment status of the trainee as of the end of services. Assuming that the timing of 

termination responds mainly to the employment at termination standard in place during the time 

their data were collected (rather than the wage rate or cost standards, which would be more 

difficult to game), they argue that sites should immediately terminate participants who are 

employed when their services end. In contrast, they should not terminate participants who are 

not employed at the end of their services; instead, they should wait and see if they later become 

employed, at which point they should then terminate them from the program. Not surprisingly, 

Courty and Marschke (1996, 1997, 2004) find that the sites in the National JTPA Study did 

exactly this. For example, Courty and Marschke (1997) found a spike in terminations at the end 

of services for employed participants, and a spike in terminations at the end of the mandatory 90 

days after the end of services for participants not employed at the end of services.21 Their 

analysis likely understates the full extent of sites’ strategic behavior, as it takes the date of the 

end of services as given, when in fact sites had some control over this as well. For example, a 

participant without a job at the end of classroom training could be assigned to a job club in the 

hope that employment would soon follow. 

Courty and Marschke (1997) interviewed 11 of the 16 sites in the National JTPA Study 

regarding their responses to the switch from measuring employment at termination to measuring 

it 90 days after termination. They report that: 

 

21See Exhibit 1 in Courty and Marschke (1997). 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 28 

...[m]ost administrators indicated that ... case managers began tracking terminees until 

the follow-up period expired. To increase the chances that an employment match lasted 

until the third month, some SDAs reported that they offered special services between 

termination and follow-up, such as child-care, transportation and clothing allowances. 

Case managers also attempted to influence employers to keep their clients until the third 

month. 

Moreover, “training administrators reported that after the third month, they did not contact the 

client again.” While these follow-up services may add value, their sudden termination at 90 

days, and their sudden use after the change in performance standards, suggests motives other 

than impact maximization. 

The second type of evidence from the National JTPA Study reported in Courty and 

Marschke (1996, 1997, 2004) concerns the timing of terminations relative to the end of the 

program year. In JTPA, performance was measured over the program year from July 1 to June 

30. For SDAs in states where there were no marginal rewards for performance above the 

standard, this leads to an incentive to wait on termination until the end of the program year when 

possible, and then to strategically terminate each participant in the program year in which his or 

her marginal value is highest. 

Courty and Marschke (2004) builds on the analyses in earlier works (Courty and 

Marschke, 1996 and 1997) by embedding them in an econometric framework and by examining 

whether the manipulation of the termination dates is merely an accounting phenomenon or 

whether it has efficiency costs. To do this, they look at non-experimental differences in mean 

impacts between persons terminated at training centers that appear to engage in more gaming 

(based on measures of the average waiting time to termination after the conclusion of training), 

at differences in mean impacts for trainees terminated in June (at the end of the program year) 

relative to other trainees, and at whether or not trainees are more likely to have their training 

truncated at the end of the program year. The impacts at the end of the training year are also 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 29 

interacted with how close the center is to its performance standards for the year. All of their 

analyses indicate an apparent (and surprisingly large) efficiency cost to the gaming behavior. 

E. USING RETURN ON INVESTMENT (ROI) AS A PERFORMANCE MEASURE 

In making investment decisions, businesses must determine the best or most profitable 

use of scarce funds. In general terms, the metric that is used to compare potential investments is 

called return on investment, often referred to by its initials as ROI. Economists who try to 

determine the best use of public or private funds have developed a very similar concept that they 

refer to as cost-benefit analysis. Although the two concepts are closely related, an examination 

of several books on cost-benefit analysis and return on investment analysis yielded no cross 

references. Our interpretation is that return on investment analysis tends to focus more on the 

rate of return to investments, while cost-benefit analysis generally focuses more on net present 

values. (These terms are defined and described and below.) As described in more detail below, 

we believe that cost-benefit analysis is generally the preferred approach for comparing 

alternative investments. As public training programs have tried to learn from business practices, 

many analysts have advocated developing ROI or cost-benefit measures that can be used to judge 

how good investments in workforce programs are. The concept can be used to determine if a 

single program or project is worth carrying out, or it can be used to compare alternative projects 

or programs to rank them or to see if they are performing adequately. This section describes the 

concepts of ROI and cost-benefit analysis, provides some examples of how ROI and cost-benefit 

analysis have been applied to workforce development programs, and explores pros and cons of 

using ROI as a performance measure for workforce development programs. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 30 

Perhaps surprisingly, education and training were not explicitly recognized as 

investments with an expected return on the investment until the middle of the twentieth century 

when economists such as Gary Becker and T. W. Schultz (1961) began describing education and 

training as investments that could be characterized as “human capital.” Although workforce 

development programs have implicitly viewed training as an investment, the Job Training 

Partnership Act (JTPA) explicitly recognized this in Section 106, which states: 

The Congress recognizes that job training is an investment in human capital and not an 

expense. In order to determine whether that investment has been productive, the 

Congress finds that -- (1) it is essential that criteria for measuring the return on this 

investment be developed; and (2) the basic return on the investment is to be measured by 

long-term economic self-sufficiency, increased employment and earnings, reductions in 

welfare dependency, and increased educational attainment and occupational skills. 

The concept of ROI is quite straightforward—an ROI measure should relate the benefits 

of a project to its costs. In a book describing how to conduct ROI analyses for training 

programs, Jack Phillips (2003, p. 21) offers two definitions. He first suggests using a benefitcost ratio, BCR, defined as BCR = Program Benefits ÷ Program Costs. Alternatively, he 

suggests that ROI can be measured as a percentage: ROI = (Net Program Benefits ÷ Program 

Costs) X 100. In practice, however, measuring ROI involves many additional decisions and 

assumptions. The decisions that must be made to compute the ROI of a project include both 

technical issues (e.g., How do we account for nonmonetary benefits such as improved health? 

Should we compute an annual rate of return or the overall rate?) and philosophical and 

judgmental issues (e.g., From whose perspective are the returns calculated? What time period 

following the investment should be included in the calculations?). Moreover, there is 

disagreement among experts on the most appropriate way to measure ROI. While Phillips (2003, 

p. 21) states that computing BCR is acceptable, economists generally shun the use of benefit-cost 

ratios; one leading text (Boardman et al. 2006) concludes that “Thus, benefit cost ratios are 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 31 

subject to manipulation. For these reasons, we recommend that analysts avoid using benefit cost 

ratios...” 

To help determine if ROI is a good performance measure, it is instructive to follow the 

steps that must be undertaken to compute the ROI. To understand what is involved, we use the 

steps outlined by Boardman et al. (2006) in their cost-benefit analysis text. Phillips (2003) 

covers much the same ground in his ROI volume. Rather than cover all the steps, we focus here 

on the three steps that are most likely to cause difficulty in using ROI as a performance measure. 

Specify the alternatives. When calculating ROI, we estimate the costs and benefits for 

the investment, e.g., receiving WIA services, versus an alternative situation, usually referred to 

as the counterfactual. In the case of workforce development programs, the programs are 

typically assessed with a counterfactual of whatever programs and services participants would 

have obtained in the absence of the program; in some instances, the counterfactual is receiving 

no services. This is the approach used in key evaluations of workforce programs such as the Job 

Training Partnership Act (JTPA) and Job Corps in recent years.22

Decide whose costs and benefits count. In computing ROI, the results can differ 

markedly depending upon whose viewpoint is adopted; sometimes this issue is referred to as 

“who has standing for the analysis.” Businesses typically want to know the value of a project 

from the perspective of stockholders, but for workforce programs, ROI can be computed from 

the perspective of participants, the government, or all the members of society. Cost-benefit 

evaluations, such as the ones cited above for JTPA and Job Corps, typically calculate the returns 

 

22 See Larry L. Orr, Howard S. Bloom, Stephen H. Bell, Fred Doolittle, Winston Lin, and George Cave (1996). 

Does Training for the Disadvantaged Work? Evidence from the National JTPA Study. Washington, DC: The 

Urban Institute Press, and Peter Z. Schochet, John Burghardt, and Sheena McConnell (2006). National Job Corps 

Study and Longer-Term Follow-Up Study: Impact and Benefit-Cost Findings Using Survey and Summary Earnings 

Records Data. Princeton, NJ: Mathematica Policy Research. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 32 

from all three perspectives, but a performance management system could use only one or two of 

these perspectives. 

Predict the impacts quantitatively over the life of the project. This is one of the most 

difficult steps in a program evaluation, and it is more difficult in a performance management 

perspective. Benefits from a labor exchange program, whose main goal is to help customers find 

a new job more quickly, may occur over a relatively short period such as a month or a year at 

most. Because such programs generally have low costs for each customer, this type of program 

can still be a worthwhile investment. Training programs, whose goal is to increase skill levels 

and earnings for many years, may require many years of post-program earnings data to capture 

all the benefits to customers. This is one of the key problems in computing ROI for workforce 

programs: we must either use or project earnings gains for many years after a program occurs, or 

risk potentially greatly underestimating program benefits for investments with a long payoff 

period. 

A second important issue in estimating program benefits is that we must generally have a 

control group or comparison group of people similar to the customers who did not participate in 

the program. In a workforce program, it is generally not enough to observe what customers earn 

before and after participation—some and often most of their post-program earnings would have 

accrued without the program. Thus, the best impact evaluations rely on a randomly selected 

control group of individuals who applied to the program or a comparison group of individuals 

who are similar to the participants but who did not receive the training. It is widely accepted that 

having a randomly assigned control group is the best method of creating a group that is as similar 

as possible to the treatment group. Unfortunately, incorporating random assignment into a 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 33 

workforce program is expensive, time consuming, and sometimes considered unethical. This is 

particularly true if the program is to be evaluated each year. 

The alternative of creating a comparison group has generated considerable controversy in 

recent years. Although sophisticated matching strategies, often based on propensity scores (the 

estimated probability of enrolling in the program given the person’s characteristics), have been 

advocated by some researchers as being a reasonable alternative to random assignment, studies 

that have compared results from classical experiments with comparison groups generated by 

propensity score matching have frequently questioned how well the matching strategies achieve 

their goal of generating comparable groups and unbiased impact estimates.23 Thus, the choices 

in creating an appropriate group for comparing the participants may boil down to a strategy that 

is infeasible or one that may lead to biased estimates of the impact. 

A further complication of using ROI as a performance measure is that to make 

comparisons across states or local areas, the services received by the control or comparison 

group must be the same across all areas. Even random assignment is insufficient to solve this 

problem if control group members are permitted to enroll in other available training 

opportunities (such as courses available at the community college). If some states have more 

alternative training programs available and the control or comparison group is not barred from 

enrolling in these other programs, the counterfactual will not be the same across states, and the 

ROI estimated in different states are not comparable. 

 

23 See Howard S. Bloom, Charles Michalopoulos, Carolyn J. Hill, and Y. Lei (2002). Can Nonexperimental 

Comparison Group Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-toWork Programs? New York, NY: Manpower Demonstration Research Corporation; Jeffrey Smith and Petra Todd 

(2005). “Does Matching Overcome LaLonde’s Critique of Nonexperimental Estimators?” Journal of 

Econometrics 125: 305-353; and Elizabeth T. Wilde and Robinson Hollister (2007). “How Close Is Close 

Enough? Testing Nonexperimental Estimates of Impact Against Experimental Estimates of Impact with Education 

Test Scores as Outcomes.” Journal of Policy Analysis and Management. 26(3): 455-477. For a more optimistic 

appraisal, see Thomas D. Cook, William R. Shadish, and Vivian C, Wong (2008). “Three Conditions under Which 

Experiments and Observational Studies Produce Comparable Causal Estimates.” Journal of Policy Analysis and 

Management. 27(4): 724-750. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 34 

It should be noted that some analysts believe that ROI can be a useful performance 

measure for workforce development programs. Most notably, the Integrated Performance 

Information (IPI) Project, which included teams from six states, developed a set of performance 

measures for workforce development programs.24 The IPI report included two ROI measures 

among the eight measures recommended.25 The two ROI measures recommended in the IPI 

report are measures of the return to taxpayers and participants over the five years following 

program exit. For the taxpayer measure, the report suggests capturing the return to taxpayers as 

the change in tax revenue and social welfare payments due to the program divided by the cost of 

the services provided. For the participant measure, the report suggests using the net impact on 

participant earnings and employer-provided benefits divided by program costs. To measure 

differences in earnings, fringe benefits, tax revenues, and social welfare payments, the IPI report 

recommends using unemployment insurance wage record data to generate comparison groups 

and earnings data, and accessing other administrative data to obtain tax and welfare information. 

The IPI report does not gloss over the difficulties in estimating and using ROI as a 

performance measure. The report states “The return on investment measures have a long lag 

between the time of service and the time the results are available, are expensive to measure on a 

frequent basis, and the methodology is not sufficiently rigorous to accurately measure small 

changes over time.” (p. 21). For these reasons, the report suggests that ROI be used as a 

“performance indicator” rather than an “accountability measure.” The IPI report specifically 

recommends that ROI not be used for “setting targets and applying consequences.” Thus, 

 

24 The six states are Florida, Michigan, Montana, Oregon, Texas, and Washington. See Washington State Training 

and Education Coordinating Board. (2005). Integrated Performance Information for Workforce Development: A 

Blueprint for States. Olympia, WA: Washington State Training and Education Coordinating Board. 

25 In addition to two ROI measures, the IPI report recommended the following six additional measures: short-term 

employment rate, long-term employment rate, median earnings in the second quarter after exit, credential 

completion rate, repeat employer customer rate, and employer market penetration rate. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 35 

although the IPI report suggests using ROI measures, they do not recommend using ROI as an 

annual performance measure with associated rewards and sanctions. 

The IPI report’s recommendations on the use of ROI are useful and apply to use of ROI 

in evaluating efficiency and effectiveness of ETA programs.26 The application of these 

techniques for ROI and the closely related concept of cost-benefit analysis are essential to assess 

if a program is a worthwhile investment and to compare alternative investments. Hence, it 

would valuable for the Department of Labor and states to conduct such analyses on a regular 

basis. However, it would also be very expensive to measure ROI on an annual basis as a 

performance measure. In addition, there is a lack of consensus about the best methods that 

should be used to generate comparison group data. Some analysts believe that the matching 

methods that are widely used do not provide good comparison groups, and the time required to 

generate reasonable post-program data is too long to provide measures that are useful for annual 

performance measures.27 Thus, the research on ROI and experiences of government agencies in 

using ROI to evaluate program performance suggest that while ROI and cost-benefit analysis 

should be considered an important tool for periodic evaluation of program performance, the 

methodology is problematic and of limited utility for annual performance reporting/monitoring. 

 

26 It should be noted that the ROI measures suggested by the IPI report could be improved upon in several ways. 

First, when possible random assignment should be used rather than developing a comparison group through 

matching procedures. Second, the return should be computed from the perspectives of society as a whole and from 

the perspective of the participants; neither proposed ROI measure calculates whether the program is a good 

investment from the perspective of society or potential participants. Third, limiting the post-program period to five 

years may bias the findings toward short-term investments, so a longer perspective, with varying assumptions about 

the decay of the impact, is more appropriate. Fourth, the IPI report appears to ignore the time value of money; we 

suggest that computing the internal rate of return is more appropriate. Finally, rather than just computing an ROI 

measure, evaluations should compute the net present value of the investment using a range of discount rates; this is 

the procedure that was used in the cost-benefit analyses of the JTPA program and Job Corps. 27 One of our Expert Panel members recommended that random assignment be used in all states on a continuous 

basis to provide benefit measures, but this approach may not be feasible, particularly in the short run. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 36 

F. EMPLOYMENT AND TRAINING ADMINISTRATION EXPERIENCE WITH 

EFFICIENCY MEASURES 

The Employment and Training Administration also has previous experience with 

efficiency standards under WIA’s predecessor, the Job Training Partnership Act (JTPA). Under 

JTPA, Section 106(b)(4) required that efficiency measures be prescribed for the JTPA Adult 

Program and that the efficiency measures be related to the outcome measures used. The National 

Commission for Employment Policy (NCEP) sponsored an evaluation of the effects of JTPA 

performance standards on participants, services, and costs.28 The study included quantitative 

statistical analysis of JTPA Annual Status Report (JASR) data linked to data on the 

characteristics of local program areas, as well as qualitative analysis based on interviews with 30 

local programs and 87 service providers in eight states. 

For the most part, the study found that the JTPA performance standards had the desired 

effects of holding programs harmless for differences in participant characteristics and local 

economic conditions. However, the study found that the cost standards had intrinsic problems 

and created some undesirable effects on participants served: 

This evaluation found that the federal standards for the entered employment rate and 

wage rate for adults generally did not have unintended effects on clients or services. 

...The federal cost standards, however, had the most unintended effects and were the 

least comparably measured of all the federal performance measures. The evaluation 

found that SDAs in states that placed more weight on the federal cost standard tended to 

serve fewer hard-to-serve clients and that [local areas] concerned about exceeding the 

cost standards tended to design less intensive services. At the same time, this evaluation 

found serious measurement problems with the cost standards. We found large 

differences in the extent to which [local programs] were leveraging JTPA funds, either by 

using funds from other programs to help fund JTPA Title II-A programs or by using 

service providers that had alternative funding sources. As a result, it is difficult to 

compare the cost of services received by JTPA participants across [local programs]. 

(Dickinson, et al, p. 5). 

 

28See Katherine Dickinson et al. (1988). Evaluation of the Effects of JTPA Performance Standards on Clients, 

Services, and Costs. Washington, DC: National Commission for Employment Policy Research Report No. 88-15. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 37 

Based on their findings from both the quantitative and qualitative components of the study, the 

authors recommended that alternatives to the cost measures be explored. The authors noted that 

as a result of concern about the unintended impacts of the cost standards, ETA set more lenient 

cost standards in PY 1988, but they concluded that this policy change would not eliminate the 

disincentive problems in states that emphasize exceeding rather than meeting standards. 

In response to the research findings, the NCEP made a number of recommendations for 

changing the statutory provisions of JTPA dealing with performance standards. Taking note of 

the study’s findings regarding the undesirable incentives and comparability of cost issues, the 

Commission’s first recommendation was: 

The Commission recommends that Section 106(b)(4), which requires the Secretary [of 

Labor] to prescribe performance standards relating gross program expenditures to 

various performance measures, be amended to direct that cost-efficiency be monitored by 

states. 

In August 1992, the JTPA statute was amended, and the amendments repealed the federal 

requirement for efficiency standards and prohibited Governors from using efficiency standards in 

making awards to local areas. 

G. STATE/LOCAL WORKFORCE AGENCY EXPERIENCE WITH EFFICIENCY 

MEASURES 

A search of the literature reveals that some states and local areas have experimented with 

use of a variety of efficiency measures for better understanding costs of service delivery – and in 

some instances, attempted to link costs to outcomes.29 Three examples of such efforts – which 

demonstrate the feasibility of efficiency measurement and possible measures that could be 

 

29See Chapter 4 of this report for additional views from interviews conducted with six states on previous experience 

with efficiency measures and recommendations on how efficiency measures should be implemented. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 38 

adopted, as well as limitations and potential for unintended consequences of such measures – are 

discussed below. 

Chicago Workforce Board – Use of a Volume and a “Positive Program 

Outcome” Efficiency Measure30 

The Chicago Workforce Board (CWB) contracted with Workforce Enterprise Services 

Inc. (WES) to conduct a study to compare the cost of serving participants through Chicago’s 

Workforce Investment Act (WIA) Title I adult, youth, and dislocated worker service providers to 

other comparable providers. A goal of the study was to help the Board and other stakeholders 

analyze the cost of operating WIA programs to see if the costs typically incurred by Chicago 

service providers are reasonable and comparable with the cost of operating such programs 

elsewhere. The project was divided into two phases: 

• The goal of the first phase was to determine if it is feasible to conduct a comparative 

analysis of cost-per-outcome for Chicago service providers, and if so, identify a 

recommended analytic approach. 

• During the second phase, WES applied the analytic approach developed during phase one 

to real-world cost and outcome information for Chicago WIA service providers. 

During phase one, WES investigated whether or not local Workforce Investment Areas 

(LWIAs) elsewhere in the nation could be identified that are comparable to Chicago in terms of 

the characteristics of the registrants served and the mix of services provided. Initially, the 

Boards overseeing WIA services in four comparison sites (New York, Milwaukee, Philadelphia, 

and Indianapolis) agreed to participate in the project and supply the requested data. However 

several issues and problems arose as efforts continued to obtain data. Due to various information 

systems and resource limitations, the Boards of the comparison LWIAs were unable to provide 

complete data on a timely basis. As a result of these efforts, it was concluded that it would only 

 

30Workforce Enterprise Services, Inc., “Chicago Workforce Board Cost per Participant Study Final Report; Cost Per 

Participant Comparison Study - Phase Two; Final Report,” July 15, 2007. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 39 

be possible to obtain provider level data for Chicago providers. The Phase One Report discussed 

the traditional workforce development cost measures - cost per registrant (or participant) and 

cost per exiter (or terminee). The report found that these measures were straightforward to 

develop, but had weaknesses that limited their usefulness. The primary weakness identified was 

that these traditional cost measures do not incorporate program outcomes and are more correctly 

characterized as “volume” measures rather than cost efficiency measures. 

The researchers noted, “since cost efficiency means little without effectiveness, a true 

cost efficiency measure must be designed that take into account the intended outcomes of WIA.” 

As a result, WES explored alternative cost-related measures that would incorporate WIA 

outcomes and could be considered true cost efficiency measures. After reviewing several 

alternatives, a new measure was proposed that combined the achievement of federally-required 

programmatic outcomes and contractor expenditures. This statistic measured the ratio between 

the share of positive program outcomes achieved by a contractor (as a proportion of total 

outcomes achieved by all providers) and the contractor’s share of total program expenditures. 

Ultimately, WES recommended that the CWB adopt a set of cost-related measures that are 

sensitive to both program volume indicators (e.g., rates of registration and program exit) and 

longer-term outcomes (e.g., employed at follow-up and earnings). The three measures 

recommended were: (1) cost per registrant, (2) cost per exiter, and (3) the share measure. 

Finally, the research contractor (WES) warned against the possibility of “unanticipated 

consequences” if the measures selected were not carefully selected and implemented: 

From the outset of the project, WES understood that the Board did not want to adopt a 

cost measurement system that might encourage short-term outcomes at the expense of 

long-term outcomes or discourage the provision of training. The Board wanted to avoid 

adopting a system that might discourage the provision of services to hard-to-serve 

populations. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 40 

To address this issue, WES recommended that the three cost-related measures not be used in 

isolation. WES recommended that a “cost profile” be developed for each provider that presented 

the Board with contextual information that would help the Board (and other stakeholders) 

interpret the cost measures. Along with the three cost-related measures, the proposed profiles 

included data showing: 

• Budgeted and actual contractor expenditures at the line item level, 

• Related expenditures for individual training accounts and on-the-job training, 

• Contractor performance on federal WIA performance measures, 

• The number of full-time equivalent (FTE) staff billed to the contract, and 

• The number and percent of exiters from hard-to-serve populations. 

Further, WES recommended that the profiles be used by the Service Delivery Committee (SDC) 

to select a relatively small sample of providers each year for a more intensive cost review. 

Contractors would be initially selected based on information presented in the cost profiles. The 

SDC would then examine the components of the contractor’s costs in detail vis-à-vis the 

programmatic model used, the characteristics of the population served, and the results achieved. 

This more in-depth review by the SDC would be designed to yield a judgment regarding whether 

or not the contractor’s cost were justified given these factors or if costs are too high given the 

results achieved. 

California – Use of Per Client, Visit, and Several Outcome-Based Efficiency 

Measures31 

 In 2006, the California Workforce Investment Board’s (CWIB) Accountability 

Committee called for a cost study of the One-Stop Career System. To complete a cost study of 

One-Stops, the CWIB contracted with a team of researchers in the College of Business and 

 

31Richard Moore, Phillip Gorman, and Andrew Wilson, California One-Stop System Cost Study Report, prepared for 

the California Workforce Investment Board, prepared by California State University at Northridge, October 18, 

2007. The comprehensive report of study and appendices are available on the CWIB Website at 

http://www.calwia.org. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 41 

Economics at California State University, Northridge to analyze the costs and operations of 

California’s One-Stop Career Centers. Working with staff from the State of California’s 

Employment Development Department (EDD) and the CWIB, the Cal State Northridge team 

developed the following research questions to guide the study: 

• What resources do California One-Stops have and where do they spend them? 

• What do partners contribute to the operation of the One-Stop and how does the pattern 

vary between sites? 

• How much and what types of services do One-Stops produce, and how do sites vary from 

each other? 

• What do different One-Stop services cost to produce and how do costs vary between 

sites? 

• Can standard measures or service units and costs be developed and applied across OneStops? 

The study had two phases. In Phase I, the research team conducted four in-depth case studies. 

The goal of the case studies was to answer the research questions and develop methods that 

could be replicated in a statewide survey of One-Stops. In Phase II of the study, the team 

conducted a survey of 18 comprehensive One-Stops, which when added to the case study data 

provided a study sample of 22 full-service One-Stops. The study used the Activity Based Cost 

(ABC) accounting model to conduct its analysis. The purpose of the ABC Accounting Model 

was to better understand the real costs of producing a product or service. The ABC accounting 

model begins by mapping the processes and activities that generate costs and produce services. 

Case studies revealed that the One-Stops were built around four processes: 

• A universal services process, where a person can walk in to the One-Stop and use 

services to find a job or training opportunities; 

• An enrolled services process, where eligible clients formally enroll in programs such as 

the WIA Adult program, the Trade Adjustment Assistance Act program, or CalWorks; 

• A business services process, which are services to business ranging from workshops on 

writing a business plan to rapid response services for companies facing a layoff or 

closure; 

• A youth services process, where services to eligible youth are provided ranging from 

academic support to case management, to help finding a summer job. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 42 

Using this framework, the study estimated the costs of processes and the activities within them, 

and then estimated the cost-per-unit of services produced. It also estimated the costs incurred by 

partners within the One-Stops associated with each process. In the analysis, the research team 

developed a wide array of cost per-unit of service measures, which could potentially serve as 

management tools. From the case studies, the team developed a standard set of services OneStops produced and then estimated the per-unit cost of producing those services, by dividing the 

total costs of a service by the number of units produced at each One-Stop. In the final report, the 

researchers presented “what we believe are the most valuable ‘cost-per’ measures for 

understanding One-Stop operations by process”: 

• Universal Services: Cost per Universal Client, Cost per Universal Visit, and Cost per 

Coaching and One-on-One Assistance Event;

• Enrolled Service: Cost per Enrolled Client, Cost per Client Receiving Case 

Management, and Cost per Enrolled Client Placed; 

• Business Services: Cost per Rapid Response Employer Assisted; Cost per Rapid 

Response Employee Assisted; and Cost per Mass Hire Event; and 

• Youth: Cost per Youth Served and Cost per Youth Placed in Employment. 

This project was the first Activity Based Cost accounting analysis of One-Stop Career Centers 

and their finances, and as such, raised as many questions as it answered. Researchers concluded 

that One-Stops tend to structure their processes around the WIA program, so the four processes 

uncovered appeared to be fairly consistent across sites. Further, One-Stop services were 

customized to local needs and this customizing accounted in large part for the differences in 

costs between sites. Finally, the researchers recommended that the State Board take the lead in 

developing a voluntary system of standard measures of costs and services for the One-Stops. 

The researchers noted “such a system can evolve over time into a powerful method for 

improving the performance of One-Stops. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 43 

 Idaho – Use of a Return on Investment Methodology32 

 The State of Idaho has employed a “Return on Investment” methodology to analyze the 

state’s workforce investment activities relative to the effects and costs of the activities. 

According to the state, reviewing the level of investment (taxpayers’ dollars) versus the return on 

that investment (participant gains in wages, taxes, and reduced public assistance) provides the 

state with “another look at the success of the programs beyond the required performance 

standards.” For example, in its ROI analysis, for individuals enrolled in the Adult program in 

PY 2006, $5.24 was returned to the community for each dollar spent and the investment was 

returned by the participant in little more than six months; for dislocated workers, the investment 

was returned in six and one-half months with those leaving the program returning $5.59 to the 

community for each dollar invested. The Taxpayer Return on Investment represented the rate of 

return of taxpayer dollars, through increased tax contributions and decreased welfare costs. To 

calculate the Taxpayer Return on Investment, the Total Annual Benefit was divided by twelve to 

produce a Monthly Taxpayer Benefit. The program cost was then divided by the Monthly 

Taxpayer Benefit to calculate the number of months it takes to pay back the taxpayer investment 

in WIA for the year in question. When calculating the Impact of Investment (a measure of the 

program’s overall benefit to the community), net earnings of the participants were included, as 

well as the increased tax contributions and decreased welfare costs. 

H. CONCLUSIONS 

 

Because workforce programs have a long history of using performance standards and the 

ETA has had substantial data collected, there is a substantial research record on the use of 

 

32 State of Idaho, “Workforce Investment Act – Annual Report for PY 2006,” October 2007, available at: 

http://labor.idaho.gov/wia1/PY2006.pdf. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 44 

performance standards for workforce programs. The research shows that programs pay careful 

attention to the performance standards they face, and that the presence of performance standards 

can have important effects on who is served and the services received. Unfortunately, the effects 

of performance standards have not always been in the desired direction. The most significant 

problems that appear in the literature are that performance standards can encourage “cream 

skimming,” where the programs are more likely to enroll individuals who would do well even 

without the programs, and “gaming,” where programs spend resources manipulating their 

measured performance by strategically enrolling and terminating individuals in a manner that 

makes the program look most effective. In addition, studies of JTPA and Job Corps indicate that 

ranking on performance measures is not related to program impact on key outcomes. Thus, 

although performance management systems can be used to encourage desired behavior, care 

must be taken to assure that undesired behavior is minimized. In their overview of lessons 

learned from performance standards in employment and training programs, Courty and Marschke 

(2007) conclude with some optimism that federal officials can monitor the misalignment and 

gaming behavior of state and local programs and take corrective actions. 

The lesson from ETA’s and state/local experiences with efficiency measures is not that it 

is a bad idea to monitor program efficiency. Rather, the inference that should be drawn is that it 

is important in selecting measures, standards, rewards, and sanctions to anticipate the behavioral 

changes that are likely to be induced by the performance management policies adopted and to 

structure the system so that the presence of efficiency measures does not result in undesirable 

behavior by states and grantees. In addition, careful thought should be given as to whether 

outcome-based efficiency measures should be applied only at the national level or whether they 

should be “drilled-down” from the federal to states and/or local workforce investment areas. The 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 45 

application of efficiency measures (and setting of standards on such measures) across states or 

local levels could, for example, lead to efforts by states and localities to reduce per participant 

costs by either providing less costly services (e.g., reducing the amounts of more costly staffassisted and training services, in favor of self-service labor exchange services) or bolstering the 

numbers of individuals enrolled (and co-enrolled) in programs. With regard to ROI, the overall 

finding from this review is that ROI should be used periodically as a tool for evaluating program 

efficiency and effectiveness, but that it is not practical to use ROI for ongoing performance 

monitoring. Studies featuring ROI typically require lengthy period of time to conduct and net 

impacts based on experimental research designs (which are costly).
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 46 

CHAPTER 3: 

USE OF EFFICIENCY MEASURES FOR MONITORING PERFORMANCE IN 

DOMESTIC AND FOREIGN WORKFORCE DEVELOPMENT AND OTHER HUMAN 

SERVICE PROGRAMS 

Other federal agencies, as well as government agencies in other countries, have to 

varying degrees used efficiency measures to monitor and assess performance of workforce 

development and other human services programs. This chapter provides an overview of the 

types of measures that have been tested by government agencies in the United States and in 

several other countries, including lessons learned from use of such measures. 

A. USE OF EFFICIENCY MEASURES IN OTHER FEDERAL AGENCIES 

Other federal departments and agencies have implemented efficiency measures to track 

ongoing program performance. This section of the report reviews efficiency measures used by 

five other federal cabinet-level agencies: The U.S. Department of Agriculture (USDA), the 

Education Department (ED), the Department of Health and Human Services (DHHS), the 

Department of Housing and Urban Development (HUD), and the Department of Veterans’ 

Affairs (VA). Within these departments, this assessment focuses on programs that provide 

activities and services related to employment and training.33 

Before presenting findings from this review, several caveats should be kept in mind. 

First, the OMB website where we obtained our data only includes programs with performance 

measures. There may be programs that have no measures but we did not know were missing 

from the website. Second, federal agencies are constantly considering whether they should add, 

drop, or modify their performance measures for strategic planning and accountability purposes, 

 

33 The source for this information is the OMB Internet site www.expectmore.gov as of May 14, 2010. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 47 

so the research may not reflect changes to the efficiency measures that are under consideration: 

for example, several measures were added and dropped since the exhibit was first prepared 

roughly one year prior to the development of the final version. 

The efficiency measures for the 21 programs identified from the OMB website plus the 

Food Stamp Employment and Training Program are summarized in Exhibit 3-1. Most of the 

programs had one or two efficiency measures, but one program (Food Stamp Employment and 

Training) had no efficiency measures and the 21st Century Community Learning Centers 

Program in the Department of Education had five efficiency measures. 

The two most commonly used types of efficiency measures are cost per participant, and 

outcome- and output-based efficiency measures with seven and eight measures observed, 

respectively. Two programs have efficiency measures based on administrative costs; for 

example the Food Stamp Nutrition Education program has the ratio of administrative costs to 

delivery costs as an efficiency measure. 

The programs reviewed clearly considered timeliness of government decisions and 

service delivery to be an important aspect of efficiency, and there were three programs with nine 

timeliness measures identified. Other concepts of efficiency reflected in the measures include 

penetration rates and accuracy of determinations. For example, the USDA’s Food Stamp 

Nutrition Education (FSNE) program uses “Ratio of FSNE Direct Education Participants 

Receiving Food Stamps to All FSNE Direct Education Participants” (a penetration-type 

efficiency measure).
EXHIBIT 3-1: EFFICIENCY MEASURES FOR SELECTED FEDERAL PROGRAMS 

(AS OF MAY 2010)

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 48 

AGENCY PROGRAM MEASURE 

Ratio of FSNE Administrative Costs to FSNE 

Delivery Costs 

Ratio of FSNE Direct Education Participants 

Receiving Food Stamps to All FSNE Direct 

Education Participants 

Food Stamp 

Nutrition Education 

Ratio of Persons Reached by FSNE Social 

Marketing Who Receive Food Stamps to All Persons 

Reached by FSNE Social Marketing 

Food Stamp Employment and 

Training None identified 

The Average Number Of Days for USDA to Make a 

Petition Determination Is 40 Days or Less. 

USDA 

Trade Adjustment Assistance 

for Farmers Timely Certification or De-Certification of Petitions 

Prior to the Beginning of a Second or Subsequent 

Year 

The Percentage of SEAs That Submit Complete and 

Accurate Data on 21st Century Community Learning 

Centers Performance Measures in a Timely Manner 

Will Increase 

The Time It Takes SEAs to Draw Funds Down to 

Reimburse Grantees Will Decrease. 

The Percentage of State Educational Agencies that 

Submit Complete Data on 21st Century Program 

Performance Measures by the Deadline. 

The Average Number of Days It Takes the 

Department to Submit the Final Monitoring Report 

to an SEA after the Conclusion of a Site Visit. 

21st Century Community 

Learning Centers 

The Average Number of Weeks a State Takes to 

Resolve Compliance Findings in a Monitoring Visit 

Report. 

The Annual Cost per Participant 

Adult Education State Grants The Cost per Adult Education Participant Who 

Advanced One or More Educational Levels or 

Earned a High School Diploma or GED 

Percentage of AIVRS Projects that Demonstrate an 

Average Annual Cost per Employment Outcome of 

No More than $35,000 

Percentage of AIVRS Projects that Demonstrate an 

Average Annual Cost per Participant of No More 

than $10,000 

American Indian Vocational 

Rehabilitation Services 

Cost per Participant 

The Annual Cost per Limited English Proficient 

English Language Acquisition Student Attaining English Language Proficiency. 

State Grants The Average Number of Days States Receiving Title 

III Funds Take to Make Subgrants to Subgrantees. 

Federal Pell Grants Administrative Unit Cost 

Education 

High School Equivalency 

The Cost per Training for HEP Participants Who 

Earn a GED 
EXHIBIT 3-1: EFFICIENCY MEASURES FOR SELECTED FEDERAL PROGRAMS 

(AS OF MAY 2010)

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 49 

AGENCY PROGRAM MEASURE 

Migrant and Seasonal 

Farmworker 

Percentage of Projects that Report an Average 

Annual Cost per Employment Outcome within a 

Specified Range (to Be Determined). 

The Percentage of Projects Whose Annual Cost Per 

Placement Is No More than $11,000 

Annual Cost per Participant Projects with Industry for 

People with Disabilities The Percentage of Projects with Industry Projects 

Who Demonstrate an Average Annual Cost per 

Participant of No More than $4,500 

Tech-Prep Education State 

Grants Cost per Secondary Student 

Tribally Controlled 

Postsecondary Vocational and 

Technical Institutions 

Cost Per Participant 

Vocational Education State Cost per Secondary Student 

Grants Cost Per Postsecondary Student. 

Vocational Rehabilitation 

Demonstration and Training 

Programs 

The Percentage of Projects that Met Their Goals and 

Objectives as Established in Their Original 

Applications. 

Education 

Vocational Rehabilitation 

Training 

The Federal Cost per RSA Supported Rehabilitation 

Counseling Graduate at the Masters-Level 

Promote Efficient Use of Chafee Foster Care 

Independence Program (CFCIP) Funds by 1) 

Increasing the Number of Jurisdictions that 

Completely Expend Their Allocations within the 2-

Year Expenditure Period and 2) Decreasing the 

Total Amount of Funds that Remain Unexpended by 

States at the End of the Prescribed Period 

Independent Living Program 

Increase the Percentages of FCIP Youth Who Avoid 

High-Risk Behaviors which Might Otherwise Lead 

to Criminal Investigations and Incarceration 

Refugee Social Services 

For Refugees Receiving Temporary Assistance for 

Needy Families (TANF) or Other Forms of Federal 

Cash Assistance, Shorten the Length of Time from 

Arrival in the U.S. to Achievement to Self 

Sufficiency. 

Health and 

Human Services 

Temporary Assistance for 

Needy Families 

Increase the percentage of state work participation 

rates that meet or exceed requirements 

Community Development Block 

Grants 

Cost per Single Family Rehabilitated Unit Will 

Remain Consistent Adjusted for Inflation Factors. 

Public Housing 

Each Year, the Federal Cost of Public Housing per 

Occupied Unit Will Remain Less that the Federal 

Cost of Voucher per Occupied Unit. 

Housing and 

Urban 

Development 

_____________ 

Veteran Affairs Vocational Rehabilitation and 

Employment Program Average Cost of Placing Participant in Employment. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 50 

The Vocational Education State Grants Program shares an important feature with the 

WIA Youth Program—the participants vary considerably in age, and hence, the use of a single 

efficiency measure could make comparisons across states difficult to interpret. To make their 

cost per participant figures more meaningful, the Education Department has separate efficiency 

measures of cost per participant for secondary and postsecondary participants. 

The primary lesson from this review is that agencies have interpreted the concept of 

efficiency quite broadly. Many agencies have more than one efficiency measure, and agencies 

have tailored the measures broadly to reflect their concerns about what aspects of their programs 

can be made more efficient. The most commonly used efficiency measure among programs 

reviewed was cost per participant; outcome- and output-based efficiency measures were also 

common, focusing on timeliness of government decisions and service delivery, accuracy of 

payments or determinations, and costs per service provided. 

B. USE OF EFFICIENCY MEASURES IN OTHER INDUSTRIALIZED 

COUNTRIES 

The project team conducted telephone interviews with officials at the Organisation for 

Economic Co-operation and Development (OECD), Canada, and the United Kingdom (UK) to 

discuss use of efficiency measures in administering employment and training programs in major 

industrialized nations. Discussions focused on the use of efficiency measures (tying outcomes 

to costs), as well as other performance measures, as part of regular performance monitoring 

activities and periodically as part of longer-term evaluation efforts. It is important to note that 

while there are important lessons to be learned from the experiences of other industrialized 

countries with regard to performance measurement, that the structure, objectives, and available 

resources of workforce development programs in other countries can be quite different from that 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 51 

found in the United States – and so, caution is needed in applying the experiences of other 

nations to the workforce development system in the United States. Below, key findings from 

these interviews with officials from the OECD, as well as the Canadian and UK government 

agencies responsible for administering workforce programs are highlighted, with particular 

emphasis on the extent of use of efficiency measures and other performance measures in ongoing 

monitoring and periodic evaluation efforts. 

Organisation for Economic Co-operation and Development (OECD). While the 

OECD member countries examine a range of participation and outcome measures and have 

conducted periodic studies examining return on investment (ROI) of workforce investment 

programs, OECD officials interviewed were not aware of ongoing performance monitoring 

systems used by member countries to track outcome-based efficiency measures of large-scale 

employment and training programs. Several countries were cited as having conducted special 

studies, some of which examined the costs and benefits of training and other services. For 

example, the Scandinavian countries (including Norway and Sweden) include detailed 

registration data, which allows for monitoring participant characteristics, services received, and 

employment outcomes (with an emphasis on employment outcomes rather than earnings); 

Ireland (the Employment, Enterprise, and Trade Agency) initiated a number of cost-benefit 

studies that have focused on employment and training initiatives. The studies conducted on 

performance of workforce programs by OECD countries tend to be one-time studies and focus 

mostly on short-term employment outcomes (such as initial job placement and retention at six 

months). Use of earnings histories (by matching to administrative earnings data collected for 

taxes or social insurance) is not widespread in OECD countries – though there is increasing 

interest in doing so. Finally, according to OECD officials, there is increasing interest in 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 52 

applying random assignment (and experimental design) to employment and training initiatives. 

France, the United Kingdom, Sweden, and Germany have launched such studies; though again, 

such studies are periodic and not part of ongoing performance measurement efforts. 

Canada. Canadian provincial governments are mostly responsible for structuring and 

underwriting job training costs, though the central government provides some funding and 

ongoing oversight. The two levels of government – central and provincial governments – work 

together to evaluate/monitor the effectiveness and cost-effectiveness of the various types of 

training provided (e.g., classroom, industry, apprenticeship, and other training). Officials from 

Canada’s main employment and training agency – Human Resources and Social Development 

Canada – highlighted the main features of current efforts to assess the performance of key 

programs of this vast department both on an ongoing and periodic basis: 

o Departmental Annual Performance Reporting. Part of the Canadian government’s 

responsibility is to make certain there are performance measures for programs 

administered by the agency – and that data are collected and analyzed on performance 

through regular monitoring activities and periodic evaluation efforts. Each fall, a 

Departmental performance report is produced.34 This annual report includes a 

“scorecard” on a number performance indicators for the (1) general population served 

(e.g., all persons qualifying for unemployment insurance payments) and (2) for several 

population subgroups (e.g., aboriginals, youth). There are several broad measures of 

program success used for the general population to gauge outcomes and cost savings – 

such as the percentage of unemployed individuals that return to work within 13 weeks, 

and savings accrued to the public purse as a result of the early return to work of 

unemployed workers receiving training and other assistance. Analyses and performance 

measures are also tailored to some extent to special populations – for example, one 

performance measure used is savings to the social services account. 

 

34The Human Resources and Social Development Canada 2006-07 performance report is available via the Treasury 

Board of Canada Secretariat website at: http://www.tbs-sct.gc.ca/dpr-rmr/2006-2007/inst/csd/csd00-eng.asp. Other 

reports on the Department’s website are the following: (1) a 2007 Employment Insurance Monitoring and 

Assessment report, available at http://www.hrsdc.gc.ca/en/employment/ei/reports/eimar_2007/toc.shtml; (2) a report 

on the Summative Evaluation of Employment Benefits and Support Measures under the Terms of the Canada/British 

Columbia Labour Market Development Agreement, available at: 

http://www.hrsdc.gc.ca/en/publications_resources/evaluation/2007/sp_ah_666_04_04e/sp_ah_666_04_04e.pdf; and 

(3) a 2008-2009 Report on Departmental Plans and Priorities, available at http://www.tbs-sct.gc.ca/rpp/2008-

2009/inst/csd/csd00-eng.asp 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 53 

o Individual Program Annual Reporting and Ongoing Performance Monitoring. 

There are also annual reports prepared on individual programs operated by Human 

Resources and Social Development Canada, which are more detailed than the annual 

Departmental report discussed above. For example, the Department prepares an annual 

Employment Insurance report (required by law), which draws on operational reports 

submitted by provinces and assesses both program outcomes and costs. Analyses are 

provided for Canada and at the provincial level. For example, one focus of measurement 

is on return to and retention in work of individuals receiving unemployment insurance. 

Some examples of processing efficiency measures (though not tied to program 

expenditures) used by the Department include the following (note: the 2008-2009 target 

is in parenthesis): 

 Percentage of access to automated telephone information services with no busy 

signals (2008-09 target: 95%) 

 Percentage of general inquiry calls answered by an agent within 18 seconds (85%) 

 Percentage of specialized calls answered by agents within 180 seconds (80%) 

 24/7 Availability of Service Canada Internet - information and transaction (98%) 

 Percentage of Canadians with access to a Service Canada point of service within 

50 kilometers of where they live (95%) 

 Percentage of notifications sent within seven days of receipt of applications (80%) 

 Percentage of Employment Insurance benefit payment or non-payment 

notification issued within 28 days of filing (80%) 

 Percentage of retirement benefit payment or non-payment notification issued 

within first month of entitlement (85%) 

 Percentage of Old Age Security (OAS) basic benefit payment or non-payment 

notification issued within first month of entitlement (90%) 

 Percentage of Apprenticeship Incentive Grant payments issued within 28 days of 

filing (85%) 

 Payment Accuracy of Employment Insurance (95%) 

 Payment Accuracy of Old Age Security (95%) 

 Client satisfaction in relation to services provided (80%).35

o Periodic Evaluation Efforts. All major programs administered by the Department are 

evaluated on a five-year schedule.36 These more in-depth programmatic evaluations 

(typically, planned by the Department, but contracted out to private firms) examine 

program performance and costs in a more in-depth and controlled way than the ongoing 

performance monitoring efforts. Use of employment and earnings over a two- or threeyear period (using both survey and administrative data) is incorporated into periodic 

evaluation efforts that focus on individual programs. In longer-term evaluation efforts, 

the Department has examined participant earnings for each dollar spent (relying on 

 

35Human Resources and Social Development Canada Report Card, available at: http://www.tbs-sct.gc.ca/dprrmr/2006-2007/inst/csd/csd00-eng.asp. 

36While there is not an OMB-like agency that oversees Departmental performance, there is Cabinet-level review of 

individual programs every four years (this group performs management oversight, looking in a more arm’s length 

way than OMB would at program performance). In addition, new programs require a plan for future evaluation to 

assess program performance. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 54 

administrative data). For example, the Department has calculated costs of participation in 

terms of direct program expenditures (including overhead costs for third party deliverers, 

individual out-of-pocket costs, and foregone earnings during the participation period). 

The Department is able to examine overall cost, such as the average cost of skills upgrade 

training. The Department has found that detailed breakdowns of program costs are often 

more difficult to obtain than output or outcome performance. For example, the 

Department has found that it is onerous for staff to complete weekly time sheets (to 

provide a breakdown of staff costs by intervention), and as a result, has periodically 

conducted short-term “time and motion” surveys to allocate staff time across function and 

program funding sources. 

The results of performance monitoring and periodic evaluation efforts are not used by the 

Department to punish or reward provinces. The Department is very careful in making crossprovincial comparisons because of substantial variation in population and environmental 

characteristics across provinces. Where problems with performance are uncovered, the 

Department typically “initiates a conversation” to discuss the problem and potential solutions. 

Departmental officials observed that with respect to implementing efficiency measures at the 

provincial or local service delivery level, care is needed in applying the measures so as to avoid 

“gaming” – for example, local program managers could potentially target easier-to-serve 

individuals to achieve lower (per participant) costs of service delivery and better outcomes (such 

as job placement and retention). 

United Kingdom (UK). According to officials at the Department of Business Enterprise 

and Regulatory Reform (BERR), the UK workforce system features primarily a “work first” 

approach to delivery of services for unemployed individuals. The system operates in a more 

centralized manner than the U.S. workforce system, offering basically the same structure of 

benefits and services (with emphasis on job search assistance) across its estimated 800 Jobcentre 

Plus offices (i.e., there is not the kind of variability in types of services and partners that is a 

hallmark of One-Stop Career Centers in the United States). Jobcentre Plus's main performance 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 55 

measurement targets, published in an annual business plan, are designed to improve overall 

productivity, efficiency, and effectiveness. The performance measures are the following: 37

• Helping people into work - Job Outcome Target -- To achieve a total points score of 11.1 

million based on the job outcomes Jobcentre Plus achieves. 

• Helping employers to recruit - Employer Engagement Target -- To achieve 92% in the 

delivery of our services to employers according to the standards we have set for our 

business. 

• Delivering a good service to our customers - Customer Service Target -- To achieve 86% 

in the delivery of services according to the standards we have set for our business. 

• Providing work-focused support at the right time - Interventions Delivery Target -- To 

ensure that Jobcentre Plus work-focused support is given to customers at the correct time 

in 86% of cases measured. 

• Timely processing of benefit claims - Average Actual Clearance Time -- To process 

customer claims for Incapacity Benefit, Income Support, and Jobseeker's Allowance 

within an average number of working days: Incapacity Benefit, 15 days; Income Support, 

10 days; and Jobseeker's Allowance, 11.5 days. 

• Cutting fraud and error - Fraud and Error Target -- To deliver a total volume of 2 

million counter-fraud and error activities by March 2009.38 

Each year, performance in each of these areas is assessed, but direct linkages are not made 

between these measures and expended funds. Budgets are set for the Jobcentre Plus program on 

a three-year basis. The Department considers outcomes in light of budgets and may make 

adjustments from year-to-year in program budgets based on outcomes and performance. Hence, 

in the UK there is an overall objective of getting more outcomes for a fixed budget, but 

performance on outcomes and decisions about an appropriate budget for a program are 

considered separately. 

Conclusions. Based on the interviews conducted and review of background 

documentation, the UK and Canada (along with other OECD countries) assess the performance 

 

37Performance of local centers is measured in part based on a “points” system, which adjusts to some extent 

performance in light of the severity of employment barriers faced by those served. Different types of individuals 

served get different “points” in measuring performance depending on the degree of difficulty in serving and placing 

the individual into work. Success in placement of harder-to-serve individuals into employment gets awarded more 

points (e.g., a centre might get 12 points if a single parent is moved into employment, while they may get only one 

point if a job-ready individual is placed). 38Source: From the Department of Work and Pensions website, available at: 

http://www.jobcentreplus.gov.uk/JCP/Aboutus/Jobcentreplusperformance/2008_-_9_Targets/index.html 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 56 

of workforce development programs both on an ongoing basis and through periodic evaluations. 

While workforce agencies utilize performance monitoring systems to track (typically on an 

annual basis and often quarterly) program participation levels, outcomes, and program costs, 

they do not employ outcome-based efficiency measures (linking outcomes such as job placement 

to program expenditures). Some programs utilize processing/output efficiency measures as part 

of ongoing performance monitoring systems, such as measurements of accuracy and timeliness 

of payments to participants (but these measures are not tied to program costs). There is some 

concern over the potential for “gaming” if, as part of ongoing performance monitoring, program 

expenditures are directly linked to outcomes – mostly, the concern is that service delivery units 

(e.g., job centers) may target services on the less difficult to serve to reduce per-participant costs. 

The UK, Canada, and OECD countries – similar to the United States – also conduct 

periodic evaluations of program performance. For example, the Canadian government schedules 

comprehensive evaluations of major workforce programs at least every five years. These 

evaluation studies examine program participation, outcomes (including employment and 

earnings based on tax records), and overall program costs in a more exhaustive, controlled, and 

systematic manner than the ongoing performance monitoring systems. In some cases, impact 

studies have been funded to examine return on investment (ROI) and statistically link 

employment and earnings to costs of providing services. UK and Canadian officials indicated 

that they do not directly reward or punish regions/provinces/localities based on outcomes or 

efficiencies (e.g., provide incentive bonuses tied to performance), but as part of the annual 

budgeting process examined outcomes in light of program budgets and, if appropriate, discuss 

shortcoming in performance. Overall, major industrialized countries, as yet, have not 

implemented the types of outcome-based efficiency measures (contemplated by ETA) for 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 57 

ongoing performance management system, though some periodic evaluation efforts have 

rigorously examined cost-effectiveness and return on investment for workforce programs. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 58 

CHAPTER 4: 

STATE WORKFORCE AGENCIES’ VIEWS ON USE OF EFFICIENCY MEASURES 

TO MONITOR PROGRAM PERFORMANCE 

 

Two rounds of interviews with state workforce agency officials provided an opportunity 

to probe state administrators about their general views on the appropriateness and feasibility of 

implementing efficiency measures for the 11 ETA programs, as well to obtain direct input on 

candidate measures, specific implementation challenges, and likely impacts should efficiency 

measures be applied to some or all of the ETA programs. Research staff conducted interviews 

with six state workforce agencies in the early stages of the study and then a second round of 

interviews in the concluding months of this study with five states, as follows: 

• An initial round of telephone interviews of officials at workforce agencies in six states 

(California, Florida, Missouri, New York, Utah, and Virginia) was conducted in the early 

stages of this study (July/August 2008), with a focus on gathering information on 

performance measurement systems and efficiency measures being used to monitor 

workforce programs, views on possible outcome-based efficiency measures, and 

perspectives on possible uses and implementation challenges of such measures. 

• A second round of telephone interviews with officials at workforce agencies in five states 

(California, Maryland, Ohio, New York, and Washington) was conducted near the end of 

the study (September/October 2009), aimed at gaining qualitative input on setting of 

efficiency measure targets/standards, as well as potential effects of co-enrollment, OneStop self-service customers, cost sharing, and other selected issues related to 

implementation and use of efficiency measure results. 

Because of the timing of these interviews and use of different interview instruments (see 

Appendix A for a copy of the questionnaires) with substantially different questions, the results of 

these interviews are reported in two sections below. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 59 

A. FINDINGS FROM AN EARLY ROUND OF INTERVIEWS WITH SIX STATES 

The six states involved in the early round of interviews were purposively selected, with 

input from the ETA national and regional offices, which nominated states that were particularly 

active in the performance management area. While state administrators from these states 

generally viewed efficiency measures as potentially helpful for monitoring performance of 

employment and training programs, they expressed serious reservations about the use of such 

measures to compare states and establish standards on which states would be rewarded or 

sanctioned. Key findings from these interviews are highlighted below. 

 Existing Use of Efficiency Measures. Most of the six states interviewed during the 

initial round have experimented to some extent with efficiency measures in the past – mostly 

cost per exiter or cost per participant efficiency measures. A few states have linked costs to 

outcomes in such measures, though this is the exception more than the rule. Several states have 

initiatives to expand the use of efficiency measures in the near future. There have also been some 

initiatives within states by local workforce areas to track program efficiency and measure return 

on investment (ROI). Some efficiency measures used or being considered for future 

implementation in the six states interviewed are: 

• cost per participant; 

• cost per exiter; 

• cost per registrant; 

• customers served per full-time equivalent (FTE) staff; 

• total cost per universal (core services) customer; 

• total cost per universal (core services) visit; 

• cost per service; 

• cost per entered employment; 

• cost per business served; and 

• cost per one-on-one coaching event. 

States (with the exception of one state) generally do not sanction or distribute incentives 

to local workforce areas based on efficiency measures; typically, efficiency measures are used to 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 60 

monitor performance and identify outliers. For example, in one state, if performance is lagging 

on a particular efficiency measure for a given substate area, the state typically uses the area’s 

performance on the measure as the basis for discussing performance and possible changes in 

service delivery strategies to enhance performance. A second state uses what it terms a “Job 

Stat” process, in which the state meets quarterly with local workforce agency staff to review 

performance before submitting the state’s performance reports to USDOL. In a third state, 

efficiency measures have aided in better program planning; for example, where there have been 

exceptionally high costs per customer, local boards have been notified and given an opportunity 

to assess factors that may be contributing to high costs; from the state’s perspective, the use of 

efficiency measures has helped some local WIBs to rein in costs. 

When states have employed multiple efficiency measures, they have not formally 

weighted the measures, though some efficiency measures are considered to be more important 

than others from the state’s perspective. Efficiency measures are generally not considered by 

state/local areas to be as important as outcome measures (especially when compared to 

performance measures reported under Common Measures, such as entered employment or job 

retention rates). States have not set standards/benchmarks for efficiency measures they use, and 

they are unsure whether such standards can be applied fairly across local workforce areas 

because of differences in service delivery approaches, characteristics of the population served, 

and environmental conditions (such as unemployment rate). One state uses cost per exiter as one 

of four criteria for determining if local areas qualify for WIA state incentive funds. 

 Challenges to Gathering Data on Outcomes. States from the early round of interviews 

identified co-enrollment as potentially a huge problem that could make it difficult (or 

impossible) to make cross-state and cross-workforce area comparisons unless all states adopt 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 61 

identical service strategy and co-enrollment policies.39 Variation in policies concerning coenrollment was evident both across states and within states (across local workforce areas). For 

example, among the six states interviewed: 

• One state is a “co-enrollment state” which means that virtually all WIA and ES 

participants are co-enrolled.40 As a result of high rates of co-enrollment, state per 

participant and exiter costs are very low but may not offer an effective way of 

distinguishing cost efficiency of training and other more intensive services. Because of 

co-enrollment across WIA and ES programs, self-service (unassisted) clients swamp 

those receiving staff-assisted intensive and training services – driving the system to a low 

cost per participant served. 

• In a second state, co-enrollment polices vary across local areas, which makes comparison 

of efficiency measures across local workforce areas (within the state) very difficult. The 

state has encouraged co-enrollment for a long time, and as a result, about one-quarter of 

the Workforce Investment Boards (WIBs) in the state are co-enrolling all Wagner-Peyser 

and WIA participants. 

• In a third state, everyone coming through the One-Stop Career Centers is initially 

enrolled under Wagner-Peyser, but only small numbers of these customers go on to enroll 

in WIA. Wagner-Peyser funds pay for core services and part of the intensive services 

received by WIA customers. There is also some co-enrollment of WIA Dislocated 

Workers in the state-funded dislocated worker program, but when average cost per 

participant and exiter are calculated, they are based only on the WIA Dislocated Worker 

program costs (i.e., per participant costs exclude the contribution made by the statefunded dislocated worker program) Unlike some other states, co-enrollment is the 

exception, that is, relatively few Wagner-Peyser participants are co-enrolled in WIA 

(though all WIA participants are co-enrolled in Wagner-Peyser). 

Officials in several states noted that with regard to efficiency measures, the numbers served (and 

outcomes) depend to a large extent on the types of individuals served and the types of services 

provided. For example, many more individuals can be served for a given amount of funds in 

labor exchange-type programs than in training programs. It is possible that if performance of 

 

39Challenges in accounting for co-enrollment in applying efficiency measures across states and local areas was a key 

focus of the second round of interviews with states conducted near the end of this project and are covered in Section 

B of this chapter. 

40Discussion with ETA officials indicated that several states have moved recently or are planning to move in the 

direction of co-enrolling all or substantial numbers of Wagner-Peyser, WIA, and TAA participants across programs. 

A new data system – the Workforce Investment Streamlined Performance Reporting (WISPR) approved by OMB 

(1205-0469) –could improve tracking of all participants in these programs. Additional background is available on 

the U.S. Department of Labor website, at the Performance and Results page, available at: 

http://www.doleta.gov/performance/guidance/wia.cfm?CFID=23589588&CFTOKEN=55168650. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 62 

states and local workforce areas is based solely on an outcome-based efficiency measure such as 

cost per entered employment, that over time states will gravitate away from providing more 

costly intensive and training services toward emphasizing labor exchange services. 

 Views on Efficiency Measures for Certain ETA Programs. The Work Incentive Grant 

(WIG) program and WIA Youth (particularly, the program component serving younger youth 

were singled out in interviews with state workforce officials as requiring special attention with 

regard to developing and applying efficiency measures: 

• WIA Youth Program - One state administrator noted that “Youth is a different ball game – 

you have older/younger; in-school/out of school youth; it is really different – you have to 

look at the desired outcomes for the program you are dealing with.” In particular, while 

employment, job retention, and earnings change are important goals for older youth 

served under the program, such goals are longer-term objectives for younger youth. 

Employment may, in fact, be counterproductive to younger participants compared to 

completing school and earning education credentials. As discussed later, efficiency 

measures that capture employment and education/credentialing would more appropriately 

connect to the underlying multiple purposes of the WIA Youth program component. An 

additional challenge with regard to the WIA Youth program is that expenditure data are 

only available at the national level for the program as a whole – and, unlike several 

performance measures applied to the program, WIA Youth expenditure data are not 

available broken down for expenditures on “younger” versus “older” youth served (or for 

in- versus out-of-school youth). 

• Work Incentive Grant Program – Unlike the other ETA programs, the WIG program does 

not directly enroll or serve individual customers. Rather, the Disability Navigators (DNs) 

funded under this initiative are responsible for building the capacity of state and local 

workforce agencies to conduct outreach to and more effectively serve individuals with 

disabilities. In one state, for example, state administrators observed: “WIG is an add-on 

program – it would be hard to look at this program the same as WIA or others. Funding 

goes for building an infrastructure, not delivery of services to specific participants.” In a 

second state, administrators noted that it would not make sense to apply efficiency 

measures in the same way for WIG as for other ETA programs (such as the WIA or 

Wagner-Peyser programs) because Disability Navigators funded under the program have 

no caseload and do not provide direct client services. The primary role of DNs is to train 

staff at the local workforce level on disability issues and effective service delivery for 

disabled individuals. The WIG program currently assesses program performance using 

the Common Measures (including entered employment, employment retention, and 

average earnings), but limits analysis to the workforce areas receiving Disability Program 

Navigator (DPN) cooperative grants. The DPN grants issued under the WIG program are 

aimed at systemic change – and it is important to note that Navigators funded under the 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 63 

grants provide training to staff that is expected to yield improvements in access and 

quality of services to disabled individuals over the length of the grant period (and even 

after the grant is concluded). Measuring the full effects of the WIG program is also 

constrained by limiting performance reporting to analysis of participation levels and 

outcomes for WIA participants (i.e., the WIG reporting system relies on the WIASRD 

data) – which means that the effects of DPN grants are missed for other (non-WIA 

enrolled) customers of the One-Stop system (e.g., Wagner-Peyser/ES registrants). 

Similar to the Apprenticeship and WIA Youth programs, reasonable alternatives for 

measuring efficiency are needed for the WIG program, which take into consideration 

special circumstances with regard to program goals, types of individuals targeted and 

served, and services delivered. 

In addition, state agency officials were concerned that One-Stop Career Center selfservice customers and incumbent workers could pose problems with respect to calculating valid 

efficiency measure results for the WIA program. A problem with self-service customers is that 

while WIA funds are expended to assist such customers, they are not counted for performance 

outcomes (such as entered employment or post-program earnings). Therefore, in using an 

efficiency measure such as cost per entered employment, costs of providing services allows selfservice customers to be counted in the numerator of the efficiency measure, and excludes the 

outcomes for self-service customers in the denominator of the measure. One potential solution 

factors out WIA expenditures for those self-service customers that do not enroll in WIA 

intensive services – though this may prove difficult because of a lack of data breaking out WIA 

costs by service level. 

Similarly, there is a potential problem with using an outcome-based efficiency measure, 

such as cost per entered employment, because it excludes exiters who are employed at the time 

of registration. The effect of not being counted for performance purposes (i.e., because the 

individual is employed at the point of registration) is that funds are expended on provision of 

services to these individuals, and yet, there is no possibility for receiving credit with respect to 

entered employment. This is a particular problem for incumbent workers who receive WIA 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 64 

intensive or training services, whose costs would be included in the efficiency measure 

calculation, but whose outcomes (e.g., entered employment) would not be. There are several 

possible approaches to dealing with this problem, including making adjustments after-the-fact to 

efficiency measure results by subtracting out costs associated with serving incumbent workers 

(in the numerator of the efficiency measure), adding in the number of incumbent workers exiting 

the program (who were excluded from the denominator), or adjusting the standard for acceptable 

performance. 

 Views of States on Future Implementation of Efficiency Measures. States are open to 

the idea of implementing efficiency measures and believe they can have value in terms of better 

managing programs and resources, but they are concerned that ETA (and others) will make 

inappropriate comparisons of efficiency measure results across states. The main problem cited 

by state administrators is that programs are administered differently, especially with regard to 

policies governing co-enrollment, sharing of costs (within the One-Stop Career Center system 

and across funding streams), types of services delivered, and methodologies used in capturing 

and allocating program costs. These differences can make comparisons across states (and local 

workforce areas) problematic. Fears of inappropriate and unfair cross-state comparisons were 

expressed by most of the state administrators interviewed. For example, administrators in one 

state worried that the federal government will make cross-state comparisons when there is so 

much diversity across states – and even go a step further and set identical standards or 

benchmarks for all states: “We don’t see a level playing field across states that would permit 

setting of standards across states. We are worried about the possibility of the federal 

government making comparisons of our state with other states. Efficiency measure data are 

useful to analyze and provide context, but efficiency measure data are not useful as a report card 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 65 

to compare states.” In another state, administrators noted that a program such as Wagner-Peyser 

would “love” to implement cost per participant, exiter, or entered employment measures, 

because costs would be very low compared to other programs such as WIA, TAA, and SCSEP. 

Several state administrators noted that some programs provide more costly services, because the 

goal of such programs is to enhance work-related skills and capabilities, as evidenced in the 

comment of one such state administrator: “Obviously, Wagner-Peyser may look better on cost 

per participant, exiter, and entered employment than some of the other programs that provide 

more intensive/training services. If data are collected and measures are available, comparisons 

will inevitably be made.”

 Officials in several states also observed that it is important to consider long-term effects 

of efficiency measures on programs, and especially, the incentives that efficiency measures 

create which might be counterproductive to providing high quality and intensive services tailored 

to customer need. State and local agencies will, according to several state administrators 

interviewed, respond quickly and ingeniously to what they are being measured on – which could 

have adverse consequences on providing services that are carefully tailored to meeting 

customers’ needs. Program officials in one state pointed out that under a performance system 

used over a decade ago for the Wagner-Peyser program, the state used several efficiency 

measures that focused on outputs per staff-year worked, e.g., obtained employment per staff-year 

worked and job orders per staff-year worked. The state had mixed results with efficiency 

measures based on staffing levels (which are no longer in use): “Managers got sophisticated 

fast – for example, they told us ‘please don’t fill that staff vacancy’...whatever they are being 

measured on drives what they do...we did get their attention on what were our priorities...it is 

amazing how you can change behavior with incentives...people get very smart, very fast.” 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 66 

 Finally, state administrators offered several other recommendations on developing and 

implementing efficiency measures for ETA programs: 

• To the extent possible, develop efficiency measures that rely upon information already 

being collected by states – this will reduce the cost and burden of data collection for 

states/local areas. 

• Some programs may do well on one efficiency measure but not another – so ETA should 

consider implementing more than one efficiency measure. 

 

• Be careful in selecting and implementing efficiency measures so that you do not 

discourage sharing of funds and co-enrollment across partners and building integrated 

systems. 

• Be cautious about making comparisons across states and local workforce areas on 

efficiency measures because there are many factors that affect program participation, 

outcomes, and costs. Making such comparisons (and setting of performance standards) 

for states/local areas could potentially create strong incentives for workforce programs to 

provide services that are least expensive (such as labor exchange services) and not 

tailored to the specific needs (and best long-term outcomes) of customers served. In 

addition, the 11 ETA programs each have somewhat (and sometimes substantially) 

different goals and objectives for targeted customers. 

Overall, based on the initial round of interviews with state workforce agency officials, as 

well as discussion with ETA representatives, if implemented, efficiency measures need to be 

carefully constructed and tailored to the goals of each ETA programs. Special attention is 

needed in determining and applying efficiency measures for the WIA Youth, WIG, and 

Apprenticeship programs. State workforce officials agreed that outcome-based efficiency 

measures could be a useful tool for monitoring program performance, but they stated that great 

care is needed in selecting the specific measures and determining how they will be applied over 

time to monitor program performance. In particular, there was concern that state and local 

programs could potentially be unfairly compared with one another (e.g., both in terms of 

comparisons of results across states for an individual program or simplistic comparisons of 

efficiency measure results across all of the programs at the national or state levels). 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 67 

Additionally, if efficiency measures are eventually applied to states and local workforce areas, 

there was concern about adverse long-term consequences for the types of persons served and 

services provided. Many of the concerns and challenges to establishing efficiency measures 

identified in the first round of interviews with state workforce agencies were echoed (and 

elaborated on) during the second round of states interviews, the results of which are summarized 

in the next section. 

 

B. FINDINGS FROM A SECOND (LATE) ROUND OF INTERVIEWS WITH FIVE 

STATES 

The aim of the second round of telephone interviews with five states conducted near the 

end of the project was to gain more qualitative input on candidate efficiency measures and the 

setting of efficiency measure targets/standards, as well as potential effects of co-enrollment, 

enrollment of One-Stop self-service customers, and cost sharing on efficiency measure results. 

Initially, the team worked with PROTECH to identify states appropriate for these interviews, 

giving consideration to factors such as size and location of the state, numbers of WIA/ES/TAA 

co-enrollments, and numbers of One-Stop self-service customers being enrolled under WIA. 

Five states were selected for interviews: California, Maryland, Ohio, New York, and 

Washington. The discussion guide attached in Appendix A guided interviews, which were 

conducted in September and October 2009. Key findings from interviews conducted with the 

five state agencies are highlighted below, with a particular focus on some of the specific 

challenges to implementing efficiency measures across the 11 ETA programs. 

Co-enrollment Policies/Practices. The co-enrollment policies and practices across the 

five states interviewed varied considerably. All the states allow for co-enrollment, but some 

states are much more aggressive in their promotion of co-enrollment across programs. In four of 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 68 

the five states (excluding New York), administrators noted that there is variation across local 

workforce areas in co-enrollment practices. At one end of the range is New York, which 

strongly encourages local workforce areas to co-enroll individuals across WIA, ES, and TAA. 

New York was the only state of the five interviewed that has policies that result in the automatic 

co-enrollment of staff-assisted Wagner-Peyser participants into WIA. New York’s interpretation 

is that federal regulations (issued in a DOL TEGL) require the state to enroll individuals into a 

particular program if the services they receive are a staff-assisted service that is wholly or 

partially funded by a particular funding source. The state’s policy is to enroll One-Stop 

customers into a particular program (such as WIA or ES) once they have receive a staff-assisted 

service. Customers coming into One-Stops are normally assessed (during their first or second 

visit), which is considered to be a staff-assisted service – and are co-enrolled in both the WagnerPeyser and WIA Programs. Further, if an individual is enrolled in TAA, he or she would also be 

co-enrolled under the WIA-Dislocated Worker and Wagner-Peyser programs. As a result of the 

state’s policy, there has been a very large increase in the number of Wagner-Peyser participants 

co-enrolled into WIA (and, overall, WIA participation has surged in the last several years).41 

At the other end of the spectrum among the five states interviewed is Maryland, which 

allows co-enrollment, but cautions local areas that they will still be held accountable for meeting 

performance standards for each program (and that those co-enrolled will be included in 

performance measure calculations for each program in which they are enrolled). Maryland 

officials noted that in the past, some local areas that co-enrolled WIA Adult customers into the 

WIA-Dislocated Worker program have experienced some difficulties in meeting WIA Dislocated 

Worker performance standards – in particular, meeting the earnings performance measure. The 

 

41 See Chapter 7 for data on the surge of enrollments under WIA during the past three program years in New York 

(in comparison to other states). 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 69 

state has a policy that One-Stop customers receiving WIA Adult services are to also be coenrolled under Wagner-Peyser. In contrast, local areas are not encouraged to enroll WagnerPeyser participants into WIA (though they are allowed to do so). The longer average training 

period for the WIA Dislocated Worker and TAA programs conflicts with a performance measure 

applied to the Wagner-Peyser program – namely, that Wagner-Peyser participants obtain 

employment within six months of being registered under the Wagner-Peyser program. Hence, 

the state does not encourage local areas to co-enroll either WIA Dislocated Workers or TAA 

participants into the Wagner-Peyser program. 

 Ohio, and California’s policies regarding co-enrollment are in the middle ground between 

Maryland and New York – both states encouraged co-enrollment (though do not mandate it) and 

had special co-enrollment initiatives involving some local workforce areas. Generally, 

California encourages local workforce areas to co-enroll One-Stop customers under WIA, ES, 

and TAA (when appropriate) so that customers receive the full range of services that are needed. 

Since July 2008, the state has been piloting a co-enrollment initiative in 12 of its 49 Local 

Workforce Investment Areas (LWIAs). In local workforce areas involved in the pilot, all OneStop customers are first enrolled in the Wagner-Peyser program. Most Wagner-Peyser 

participants are then co-enrolled under WIA once they receive a staff-assisted service (which 

includes an initial assessment) and are counted for performance purposes under both the WIA 

and Wagener-Peyser Programs. TAA participants in pilot sites are all co-enrolled in the WIA 

Dislocated Worker program; if a WIA Dislocated Worker is TAA-eligible, he or she is coenrolled under TAA. Because there are several large LWIAs involved in the pilot, the state has 

experienced an increase in recent years in the number of co-enrolled customers across the state 

(particularly, in the number of Wagner-Peyser participants co-enrolled under WIA). 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 70 

In Ohio, co-enrollment is encouraged between the WIA, TAA, and Wagner-Peyser 

programs, but there is no mandate to do so. The state has what it terms “integrated” local areas, 

which are seven local areas and part of an eighth local area (of the state’s 20 LWIAs) that are 

working under a mandate that all TAA participants also be co-enrolled under the WIA 

Dislocated Worker Program. The state is in the process of expanding this integration project 

throughout the state. The state began this integration project in the areas with the highest 

concentration of trade-affected workers. Within Ohio, there is no mandate for co-enrollment of 

Wagner-Peyser participants in WIA. However, there is considerable co-enrollment between the 

Wagner-Peyser and WIA programs. 

Finally, the State of Washington is similarly in the middle ground with Ohio and 

California. The state has no formal policy on co-enrollment, but has what it terms an “integrated 

framework.” 42 Generally, the state encourages co-enrollment across workforce development 

programs when it is appropriate, but it is not required. The state wants local areas to take a 

holistic approach to serving customers – the states encourages local workforce areas to co-enroll 

across workforce programs when it benefits the customer, but it does not encourage coenrollment for the sake of co-enrollment. As a result of the state’s flexibility, there is some 

variation across local areas in co-enrollment across local workforce areas in the state. In 

particular, two local workforce investment areas (the Southwest and Spokane WIBs) have been 

co-enrolling quite a few Wagner-Peyser participants under WIA. State officials, however, 

observed that these two local areas had scaled back on co-enrollment recently because they had 

found that co-enrollment had lowered their WIA performance outcomes relative to other areas as 

their participant cohorts took on Wagner-Peyser characteristics. 

 

42 See state’s web site for additional details (at http://www.wa.gov/esd/1stop/).
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 71 

Change in the Numbers and Proportion of Co-Enrolled Customers. Three of the five 

states interviewed indicated that there had been an increase in co-enrollment – with New York 

indicating that there has been a very substantial increase in co-enrollment. New York’s policy 

regarding co-enrollment across WIA, ES, and TAA went into effect in July 2006 – and as a 

result, there have been dramatic increases in both the number and percentage of individuals coenrolled across these programs. In contrast, Maryland has seen some increase in the numbers of 

WIA Adults co-enrolled in ES (but not the reverse), and there has been a decrease in coenrollment between WIA Adult and WIA Dislocated Worker Programs because of fear that WIA 

Adults could potentially drag down performance on the WIA Dislocated Workers earnings 

performance measure. California and Ohio have experienced somewhat of an increase in 

numbers of co-enrolled individuals in recent years, in part due to state encouragement for coenrollment (when it is appropriate) and because of co-enrollment pilot projects involving local 

areas. Washington reported there have not been shifts in co-enrollment patterns across programs 

in recent years. 

Views on How to Deal with Co-Enrollment in Measuring Program Efficiency. States 

emphasized that ETA should take great care in implementing efficiency measures. In particular, 

because of the potential effects of co-enrollment, cost sharing, and other cross-state differences 

on efficiency measure results, administrators in all four states indicated that ETA should not 

implement state-level performance standards with regard to efficiency measures and states 

should not be rewarded or sanctioned based on efficiency measure results. There is much 

concern that efficiency measures could potentially drive states in the direction of offering lower 

cost services, at a time when ETA is pushing states to provide more training. One state 

administrator observed that: 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 72 

“Low cost is not an indicator of a quality program – efficiency does not measure quality 

of services provided. The purpose of performance measures is to drive a program toward 

meeting certain objectives – it would be a big mistake to put a measure out there without 

knowing how you want to drive the program. It would be bad idea to have efficiency 

measures unless you want to drive the program toward provision of cheap services. ETA 

is pushing a training agenda right now, which is not consistent with providing low-cost 

services (which are the likely result of implementing efficiency measures).” 

In another state, an administrator was worried about ETA eventually moving toward 

adoption of “system-wide” efficiency and outcome measures: “Because of its emphasis on 

automated service and job placement – and the much larger number of customers served under 

Wagner-Peyser – efficiency measure results (such as cost per entered employment) on systemwide measures would be largely driven by the ES (and be very low compared to similar 

measures for WIA and TAA participants who receive much more expensive training services).” 

Accounting for Program Costs. All five states interviewed receive WIA expenditure 

data from local areas and could potentially provide this data to ETA at minimal costs if the 

content and format of the data are the same as that submitted by local areas. There was some 

concern expressed about ETA creating additional paperwork burden for states and local areas -- 

for example, one state administrator observed that states are very busy and do not necessarily 

have time and resources to format and send the data to the federal government; an administrator 

from a second state observed that if the federal government introduced a new form and the 

formatting required different breakdowns from what is currently being provided by local areas, it 

could be very expensive for the state to provide the local area expenditure data to the federal 

government. A third state noted that a challenge to collecting and reporting from local 

workforce areas in the state is that each local area has its own financial system, so the content 

and format of expenditure data varies. This also has implications for the state (or federal 

government) changing requirements for content or format of expenditure data submitted by local 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 73 

workforce areas – namely, that each area in the state would need to implement changes to 

collection and/or reporting of expenditure data. 

Four of the five states do not receive expenditure data from local areas disaggregated by 

service level. The fifth state – New York - just started to require local areas (as of March 2009) 

to provide additional expenditure breakdown (each month) on training costs for three types of 

training: Individual Training Accounts (ITAs), On-the-Job Training (OJT), and customized 

training. The training breakdown is for direct costs of training (i.e., does not include staff or 

infrastructure costs). It has not been difficult for LWIAs to provide direct training costs, but 

New York state administrators noted that it would be difficult if LWIAs were required to 

disaggregate and report indirect-type costs (such as staff costs associated with intensive services, 

which result in an individual being referred to training). States were primarily concerned with 

the costs associated with requiring states to submit service level expenditure breakouts and 

difficulties with breaking down non-training related costs (especially indirect costs). For 

example one state administrator observed that the state and localities are facing serious resource 

constraints right now: 

“The state and local areas are strapped for cash—they would need additional resources to 

cover extra reporting costs. The state’s IT [Information Technology] staff are facing a lot 

of demands on their time, especially with making system changes associated with 

extended UI benefits and TAA program changes; stimulus-related changes seem to be 

coming down each day. In addition, local governments are struggling with layoffs 

because of a loss of local tax revenues – many One-Stops in the state, for example, are 

cutting back staff right now because of budget cutbacks.” 

Finally, all five states interviewed indicated that within a One-Stop environment there is 

extensive cost sharing, especially between WIA and the Wagner-Peyser program. State 

administrators were in agreement that it would be very difficult to account for shared costs 

across programs and had no specific suggestions on how to account for shared costs in 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 74 

calculating efficiency measure results for coordinated programs such as WIA, TAA, and the 

Wagner-Peyser programs. 

Accounting for Self-Service Customers. Receipt of staff-assisted services typically 

transitioned One-Stop customers from being “self-service” customers to being enrolled in a 

particular program (such as WIA) and counted for performance purposes. For example, one state 

administrator observed: “The state and LWIAs use TEGL definitions (i.e., “significant staff 

assistance”) to determine when customers are counted for performance purposes.” States 

acknowledged that it is somewhat of a problem to exclude such customers from efficiency 

measure results on the outcome side (in the denominator of the efficiency measure), while 

counting costs associated with serving self-service customers (in the numerator of the efficiency 

measure results). The states indicated that costs associated with self-service customers are not 

all that great (especially when compared to intensive and training services). The states did not 

have specific recommendations on accounting for self-service customers in calculating efficiency 

measure results – though because costs are not that significant for this group, several states 

indicated that it is probably best to leave this group out of the outcomes but include costs 

associated with this group in the efficiency measure result (as long as this is done uniformly 

across states). It would be difficult for states to break out costs associated with serving selfservice customers in One-Stops. 

Accounting for Incumbent Workers. With regard to incumbent workers, states are in 

agreement that not counting incumbent workers in certain outcomes and for efficiency measure 

results is problematic, but states are divided on how to deal with this issue. One state that served 

relatively few incumbent workers (Maryland) indicated that it might be better to look at 

incumbent workers separately – that is, look at funds given to companies for incumbent worker 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 75 

training and the numbers of incumbent workers served (and their outcomes). For incumbent 

workers, this state indicated it would be important to look at wage gain (as an outcome) and 

wage gain in relation to expenditures (as an efficiency measure). A second state (California) 

thought incumbent workers should be included in efficiency measure results. A state 

administrator noted that if job retention (rather than entered employment) is used as the basis for 

the efficiency measure, incumbent workers would be included in both outcome and efficiency 

measure results. Because of this, this administrator felt that job retention might be a better 

measure to use than entered employment (which excludes exiters who are employed the quarter 

prior to enrollment) as the basis for measuring program efficiency. A third state (New York) 

indicated that it served some incumbent workers and advocated with regard to efficiency 

measures such as cost per entered employment that incumbent workers should be included. In 

Washington ,the numbers of incumbent workers served varies in cycles (i.e., from year to year), 

and there is also considerable variability across localities. State administrators noted that one 

way to incorporate incumbent workers into performance measurement would be to switch to 

employment rate rather than entered employment rate (which under WIA performance standards, 

excludes workers who are employed the quarter before entry into WIA). Finally, Ohio state 

workforce officials noted that incumbent workers served in the WIA Adult and Dislocated 

Worker Programs are typically served under waivers. As such, they do not count for outcome 

measures, and yet, they are served with WIA funds and so the costs of serving such individuals 

would potentially count on the cost side but not on the outcome side in an efficiency measure 

such as cost per entered employment. Administrators in this state (and other states) felt there 

was no good way to adjust for incumbent workers in efficiency measure results. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 76 

Accounting for Costs Spread Over More than One Year. This was an issue for each 

of the five states, and it is particularly, likely to affect efficiency measure results for programs 

such as WIA, Apprenticeship, and TAA that offer longer-term training. For example, one state 

(Ohio) estimated that 70 percent of WIA participants are engaged in training that is spread across 

two or more program years. Two states (California and Maryland) indicated that the only way to 

really solve this problem is to track expenditures by participant, but that states and local areas do 

not do this and it would be costly to do so. Washington state workforce officials noted that some 

costs that are spread across multiple program years are more difficult than others – for example, 

while it is possible to track and adjust for costs such as individual training account (ITA) costs, it 

is difficult to track non-training related costs (such as costs related to the provision of intensive 

services). 

Return on Investment as a Methodology for Assessing Program Efficiency and 

Effectiveness. Four of the five states interviewed (with the exception of Washington) have not 

used ROI in recent years to assess program performance or efficiency (though some local areas 

within states have attempted to implement ROI studies). Washington state workforce officials 

indicated that cost-benefit analysis is conducted on workforce programs every four years. These 

officials acknowledged that cost-benefit and ROI-type studies are “kind of expensive and there is 

variability in methodology to generate consistent numbers...it would be difficult to estimate ROI 

every year.” State workforce officials in all five states agreed that ROI studies can be expensive 

and complicated – and that results are not always timely – for example, according one state 

administrator, “Timeliness is an issue with ROI – it may be 10 years before you get the ROI 

results. It is important to look to see if you can get data you need and get it quickly enough to 

make ROI results useful.” Overall, officials in the five states agreed that it would be impractical 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 77 

to use ROI for regular/ongoing performance monitoring, but that it could be a useful tool for 

periodically evaluating program performance. 

Overall Views on Implementation of Efficiency Measures. Overall, the five states 

interviewed are in general agreement that efficiency measures that regularly examine program 

costs and efficiency are important, but that efficiency measures should not be incorporated into 

the annual Common Measure performance measurement process whereby performance standards 

are set for individual measures and states are reward or sanctioned based on whether standards 

are met. Several state administrators worried that if efficiency measures are implemented as part 

of regular performance measurement they would likely drive states/local areas toward providing 

less costly labor exchange services (such as under Wagner-Peyser) – and that this would work 

against states and local areas providing training to enable workers to upgrade skills in order to 

fill higher skilled/wage jobs. Reflecting underlying concerns in several of the states interviewed, 

one state official observed: “It is easy to get carried away with efficiency and this could create 

wrong incentives to ignore certain subpopulations on equity and efficiency measures. Efficiency 

measures may also create perverse incentives to avoid more effective treatments...be careful on 

incentives—they can come back to bite you – and be careful on how you set standards.” 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 78 

CHAPTER 5: 

OPTIONS AVAILABLE, ANALYSIS, AND CHALLENGES 

TO MEASURING COSTS IN CALCULATING EFFICIENCY MEASURES 

As discussed earlier in Chapter 1, in order to generate an efficiency or unit cost measure, 

it is necessary to account for costs in the numerator of the measure and a unit of participation, 

service delivery, or outcome in the denominator of the measure. This chapter focuses on the cost 

portion of the efficiency measure calculation, examining the three options available for 

measuring costs, providing a recommendation on which cost type should be used, and detailing 

challenges to appropriately and consistently capturing costs when measuring program efficiency. 

A. OPTIONS AVAILABLE, ANALYSIS, AND RECOMMENDATIONS FOR 

MEASURING PROGRAM COSTS 

Based on discussions with ETA officials (and a review of the literature and available data 

sources within ETA), three potential types of “cost” data could be considered for the numerator 

of each efficiency measure: 

• Appropriations/Allotments – “Appropriations” are defined by the Government 

Accountability Office as “Authority given to federal agencies to incur obligations and to 

make payments from Treasury for specified purposes.”

43 Appropriations are generally the 

amount of funding made available by Congress for spending on a given program (such as 

the TAA program) during a fiscal year. Appropriations have been used by ETA in 

calculating “cost per participant” for programs such as WIA and Wagner-Peyser. Such 

measures have been used primarily as part of the planning process prior to the start of a 

program year to estimate the numbers of individuals who can likely be served for a given 

funding level. Allotments are the amount of appropriated funds distributed to a state or 

grantee based upon a legislative or regulatory formula.44 Allotments exclude amounts 

retained by the federal government to administer programs. In programs that do not have 

 

43 See U.S. Government Accountability Office (2004). Principles of Federal Appropriations Law, Third Edition, 

Volume 1. Washington, DC: U.S. Government Accountability Office Report GAO-04-261, SP, p. 2-5. 44Although the term “allocation” is used quite often in an interchangeable manner with allotment, in the context of 

formula programs the appropriate term is allotment. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 79 

a formula distribution, this term may be used to denote the discretionary amount planned 

to be distributed to a grantee/contractor. 

• Obligations – According to the General Accounting Office (GAO), “...obligations 

reflect orders placed, contracts awarded, and other similar transactions during a fiscal 

year. As an expression of an agency’s total financial commitments for a given period, 

gross obligations portray the relative size of an organization, without regard to the type 

of underlying budgetary resource or when resulting outlays may occur.”45 Hence, 

obligations are funds that have been committed through contracts, grants, and other 

vehicles. 

• Expenditures – Expenditures are funds paid or the amount of funds due (depending on 

whether a cash or accrual basis is used) for provision of goods or services pursuant to a 

grant or contract agreement. With regard to accounting for various program 

expenditures, ETA’s Office of Financial and Administrative Management (OFAM) noted 

(in an electronic response to a question posed by the research team) that “as far as the 

accounting process for the various levels, the process varies depending on what type of 

program you look at and how it operates. The general process is that DOL obligates 

federal funds to direct grantees, and requires those grantees to provide financial reports 

which include cash transactions, obligations for lower level grants or contracts, and cost 

reporting. There could be several levels of grants/contracts below DOL, but the direct 

grantees are responsible for summarizing all financial data for those sub-levels when 

reporting back to DOL.”46 

Telephone interviews with six states and discussions with ETA program officials, as well 

as analyses of data on individual programs, confirmed that expenditure data should be used when 

calculating efficiency measures for ETA programs. Expenditures are preferred to allotments or 

obligations because they are based on what is actually expended on providing services for 

program participants, taking into consideration rescissions, transfers between programs, and 

expenditure of program funds across more than one program year. Exhibit 5-1 provides an 

illustration of how the use of expenditures versus allotments can make a difference when 

calculating cost per entered employment for a program such as the WIA Adult program. 

 

45United States General Accounting Office (GAO), Federal Budget: Agency Obligations by Budget Function and 

Object Classification for Fiscal Year 2003, GAO-04-834, June 2004 (available at 

http://www.gao.gov/new.items/d04834.pdf). 46Also according to OFAM, the term “drawdown or payment” is sometimes used to reflect the “transfer of cash to a 

grantee/contractor based on grantee requests/contractor invoices to reimburse the grantee/contractor for expenditures 

on a valid grant/contract. Source of quotations in footnote and text is OFAM electronic response to a question. 
EXHIBIT 5-1: VARIATION IN COSTS (WIA ADULT, PY 2005)

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 80 

 

 

State 

 

PY 2005 

Allotments 

 

PY 2005 

Expenditures 

Difference 

(Expenditures - 

Allotments) 

 

Percentage 

Difference 

 Total $972,406,996 $989,481,877 $17,074,881 1.8% 

New Mexico $6,460,982 $9,547,783 $3,086,801 47.8% 

Delaware $1,978,186 $2,505,938 $527,752 26.7% 

Nevada $4,643,187 $5,729,486 $1,086,299 23.4% 

Mississippi $12,201,673 $15,006,538 $2,804,865 23.0% 

Hawaii $3,026,432 $3,686,765 $660,333 21.8% 

New Jersey $21,742,322 $25,210,314 $3,467,992 16.0% 

Kentucky $17,150,395 $18,785,452 $1,635,057 9.5% 

North Carolina $30,650,748 $33,559,283 $2,908,535 9.5% 

Louisiana $20,206,337 $22,087,656 $1,881,319 9.3% 

Washington $22,810,203 $24,588,371 $1,778,168 7.8% 

Florida $42,534,930 $45,228,422 $2,693,492 6.3% 

Puerto Rico $35,811,897 $38,032,686 $2,220,789 6.2% 

Utah $5,186,709 $5,452,807 $266,098 5.1% 

Oregon $16,023,659 $16,644,227 $620,568 3.9% 

Pennsylvania $33,565,397 $34,786,979 $1,221,582 3.6% 

Michigan $41,989,813 $43,393,133 $1,403,320 3.3% 

Texas $88,060,741 $90,920,855 $2,860,114 3.2% 

Arizona $16,629,687 $17,133,496 $503,809 3.0% 

Ohio $40,994,031 $41,977,559 $983,528 2.4% 

New York $77,930,704 $79,595,396 $1,664,692 2.1% 

Wisconsin $11,542,384 $11,772,235 $229,851 2.0% 

Wyoming $2,640,294 $2,676,683 $36,389 1.4% 

Alabama $17,442,093 $17,665,302 $223,209 1.3% 

Illinois $44,436,912 $44,847,898 $410,986 0.9% 

Idaho $2,801,747 $2,815,094 $13,347 0.5% 

Minnesota $9,435,871 $9,412,227 -$23,644 -0.3% 

California $137,225,360 $136,577,973 -$647,387 -0.5% 

Montana $2,561,631 $2,540,090 -$21,541 -0.8% 

Maryland $9,919,836 $9,778,360 -$141,476 -1.4% 

Nebraska $2,499,885 $2,460,076 -$39,809 -1.6% 

Missouri $15,743,342 $15,480,533 -$262,809 -1.7% 

Virginia $12,535,527 $12,322,012 -$213,515 -1.7% 

New Hampshire $1,900,876 $1,857,842 -$43,034 -2.3% 

Kansas $6,673,042 $6,497,283 -$175,759 -2.6% 

Rhode Island $2,824,329 $2,714,604 -$109,725 -3.9% 

Tennessee $19,176,233 $18,420,971 -$755,262 -3.9% 

Massachusetts $14,684,986 $14,037,922 -$647,064 -4.4% 

Georgia $16,958,731 $16,091,106 -$867,625 -5.1% 

Alaska $3,029,756 $2,858,082 -$171,674 -5.7% 

Maine $3,069,783 $2,869,752 -$200,031 -6.5% 

North Dakota $2,218,186 $2,064,034 -$154,152 -6.9% 

Connecticut $7,592,184 $6,955,344 -$636,840 -8.4% 

West Virginia $6,286,348 $5,747,333 -$539,015 -8.6% 

Arkansas $9,574,708 $8,710,671 -$864,037 -9.0% 

Iowa $4,265,476 $3,854,824 -$410,652 -9.6% 

Oklahoma $9,753,985 $8,582,068 -$1,171,917 -12.0% 

District of Columbia $3,876,655 $3,388,419 -$488,236 -12.6% 

Colorado $13,187,525 $11,452,728 -$1,734,797 -13.2% 

South Dakota $2,385,238 $2,020,786 -$364,452 -15.3% 

Vermont $2,249,082 $1,859,946 -$389,136 -17.3% 

Indiana $15,987,777 $12,916,277 -$3,071,500 -19.2% 

South Carolina $18,329,181 $14,360,256 -$3,968,925 -21.7% 

Note: Total excludes funds not distributed to states; data provided by OFAM.
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 81 

As shown in the exhibit, for the United States overall, in PY 2005, expenditures in the WIA 

Adult program were slightly higher (about $17 million or 1.8 percent) than allotments. Despite 

the relatively small difference at the national level, as shown in Exhibit 5-1, there can be 

substantial differences between allotments and expenditures at the state level; for example, in 13 

states, the percentage difference between expenditures and allocations is plus or minus 10 

percent or more, and in one state, the difference was nearly 50 percent. 

Actual expenditure patterns can be quite different from allotments and obligations 

because of transfers of funds, rescissions, and allotted funds not expended. Within a state (or at 

the local workforce level), patterns of expenditures can change based on priorities and needs of 

states and local workforce areas (based on changing economic conditions and needs of the 

individuals served by each program). In WIA, for example, funds can be transferred to some 

extent among the Adult, Dislocated Worker, and Youth programs; and Governors can move 

funds from one local workforce investment area to another if the funds are not being spent. 

Original allotments can also be affected by rescissions and supplemental appropriations. As 

another example, in PY 2008, the Department of Labor is required to apply three different 

rescissions, which result in an across-the-board cut of allotted funds to states.47 Such rescissions 

result in cutbacks of training and other services (thus, affecting overall expenditure levels). 

 

47 The rescissions are to be applied to various WIA funds during FY 2008 as follows: (a) to the FY 2007 Advance 

fund year source, a 1 percent across-the-board rescission; (b) to the FY 2008 Advance fund year source 

(appropriated in advance in the FY 2007 appropriation act), a 1.747 percent across-the-board rescission; and (c) to 

unexpended balances of formula funds appropriated for PY 2005 and PY 2006 which includes fund year sources PY 

2005, FY 2006, PY 2006, and FY 2007, a rescission of $250 million, required by P.L. 110-161, applicable to the 

WIA Adult, Dislocated Worker and Youth formula programs only. For additional details, see U.S. Department of 

Labor, Training and Employment Guidance Letter No. 24-07, “Rescissions During Fiscal Year 2008 for Workforce 

Investment Act (WIA) Programs, issued March 26, 2008, available at: 

http://wdr.doleta.gov/directives/corr_doc.cfm?docn=2620. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 82 

EXHIBIT 5-2: ARGUMENTS FOR AND AGAINST 

VARIOUS MEASURES OF COSTS 

Types of Costs PROS CONS 

Appropriations/ 

Allotments 

• Available at beginning of 

program/fiscal year 

• Not subject to revision due to 

audits 

• Does not reflect resources used to produce 

outcomes—funds can be used from prior 

years, carried over to future years, be 

reallocated, or be rescinded 

• For longer programs, particularly programs 

that last more than one year, 

appropriations/allotments in a single year fail 

to capture all the resources spent on exiters for 

that year 

Obligations • Available sooner than 

expenditures 

• Closer than appropriations to 

concept of resources used to 

produce services 

• Does not reflect resources used to produce 

outcomes in particular year 

• More difficult to track than 

appropriations/allotments and expenditures 

• For longer programs, particularly programs 

that last more than one year, obligations in a 

single year may fail to capture all the resources 

spent on exiters for that year 

Expenditures • Measures resources used to 

produce outcomes, which is goal 

of efficiency measure 

• Expenditures audited for 

accuracy 

• Viewed as fairest measure by 

states and program offices 

• Can be revised due to audits 

• Require more time to gather data than other 

cost measures 

• For longer programs, particularly programs 

that last more than one year, expenditures in a 

single year fail to capture all the resources 

spent on exiters for that year 

Exhibit 5-2 (above) provides an overview of the arguments for and against each of the measures 

of cost for use in calculating efficiency measures. It is recommended that ETA use actual 

expenditures (rather than appropriations, allocations, or obligations) in calculating efficiency 

measures because (1) expenditures can vary substantially from what is initially 

appropriated/allocated, especially at the state level (e.g., because of transfers, rescissions, and 

unexpended funds unexpended funds); and (2) expenditures reflect what is actually spent on 

delivery of services and capture the underlying notion of efficiency. States interviewed, ETA 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 83 

program offices, and the Expert Panel endorsed the use of expenditures over the other available 

measures of costs. 

B. CHALLENGES INVOLVED IN PRODUCING ACCURATE AND TIMELY 

EXPENDITURE DATA TO PRODUCE VALID AND RELIABLE EFFICIENCY 

MEASURE RESULTS ACROSS STATES 

Even though use of actual expenditure data is recommended for calculating efficiency 

measures, there are a number of challenges and issues that arise in using expenditure data. 

Among the main challenges, most of which were identified in discussions with state 

administrators, are the following:

 Challenge #1 - Varying Cost Allocation Methods Used Across States and Local 

Areas. The ways in which states and local workforce areas collect, allocate, and report on 

expenditures vary substantially. For example, an official in one state workforce agency 

observed: “There are different ways that states account for costs – how do you standardize across 

states? There are also differences in accounting procedures below the state level (at the local 

level). Within our state, for example, Wagner-Peyser pays for some of the core and intensive 

services for WIA enrollees.” Two particular problems with respect to allocating and reporting 

expenditures were cited in discussions with states and ETA program officials, which could affect 

efficiency measure results (and undercut comparability of results across states and local areas). 

These two issues are discussed below.

Sharing of Costs across Programs and in the One-Stop Career Center 

System Clouds Calculating Efficiency Measures by Funding Stream

 Over the past decade, there has been a mandate to build a comprehensive One-Stop 

Career Center system that brings together various programs within a single physical location and 

shares (to the extent possible and appropriate) the costs of serving individuals. States have 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 84 

created One-Stop delivery systems supported by Memoranda of Understanding (MOUs) and cost 

allocation plans in local areas with the intent of leveraging resources from Wagner-Peyser, WIA, 

TAA, and VETS, as well as other local partners. The range of other funding sources/partner 

programs in One-Stop Career Centers is designed meet needs of customers served by the 

workforce investment system. In such a system, individuals may receive services by one or 

more programs (either at the same time or sequentially), and various funding streams may be 

brought together to pay for services. For example, an individual enrolled in the WIA Dislocated 

Worker program may also be co-enrolled and simultaneously (or sequentially) receive services 

paid for by the WIA Adult, TAA, Wagner-Peyser, and/or non-DOL programs. Core unassisted 

services—when a participant first comes into the One-Stop Career Center—may have been paid 

for with Wagner-Peyser, WIA, and/or other funding sources (e.g., county funding), while 

training services may be reimbursed by a combination of WIA Dislocated Worker and TAA 

funding (perhaps combined with and further subsidized with funds from a Pell Grant, statefunded training assistance program, or subsidized training at a community college.) Hence, the 

sharing of funds and co-enrollment of participants, which can greatly affect numbers of 

individuals served across programs, makes it complicated (at times nearly impossible) to isolate 

costs of serving a particular individual with a particular funding stream. One state administrator 

warned that if ETA was not careful in developing and applying efficiency measures that there 

could be an adverse effect on the One-Stop Career System: 

“Why is DOL looking to develop efficiency measures for individual programs/funding 

streams versus the system as a whole? For the last 9 or 10 years, we have been talking 

about creating a ‘system.’ If this is the case, we don’t know if calculating cost per 

participant, etc. by program (funding stream) is the best thing. Our state has undertaken 

serious efforts to build a workforce development system. A fundamental question I 

would ask is what are we doing with individual program measures? There are system 

costs for all of these programs operating in a One-Stop environment. There are also 

shared costs. The whole idea is to share performance across programs/funding streams. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 85 

It is important to look at what performance looks like in a One-Stop Career Center where 

you leverage performance and cost. It is confusing to states to ask them to develop 

systems on the one hand, but be accountable for individual programs (and funding 

streams).” 

 One possibility would be to develop a “composite efficiency measure” for the One-Stop 

system as a whole. One of the challenges to developing a composite measure for the One-Stop 

system is that some of the 11 ETA programs are not located at the One-Stop (e.g., in some states, 

there still exists stand-alone Wagner-Peyser/ES offices that operate in tandem with One-Stops). 

A One-Stop is a place – not a program – that offers a varying constellation of voluntary and 

mandatory partners from one state to another (and even within states, sometimes there is 

variation within and across localities as well). As a result, comparing cost per placement or other 

outcome-based efficiency measures is complex. Further, the cost of supporting One-Stop 

operations comes from a variety of sources outside of ETA programs that may be difficult or 

impossible to track (e.g., county government or TANF programs may be contributing to paying 

the operating costs of the One-Stop, but are not included in the 11 ETA programs). An 

alternative to a composite measure for the One-Stop system would be a composite “system” 

measure that would assess efficiency across the 11 ETA programs, or possibly a broader array of 

programs including TANF, Food Stamps Employment and Training, and similar programs. An 

example would be to calculate cost per placement using all expenditures across the 11 ETA 

programs in the denominator and an unduplicated count of job placements across the same ETA 

programs (taking into consideration co-enrollments across WIA, TAA, and other programs by 

counting co-enrolled individuals only once). 

Complexities of Calculating Staff Time by Program Activity

 Allocating staff time by activity is a particular challenge should DOL institute a cost per 

activity-type efficiency measure. Staff time is a major cost item for all of the programs. Some 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 86 

states use time sheets or random moment time studies to allocate staff time across funding 

streams. For example, among the states interviewed, one state uses random moment time 

sampling to allocate staff time across programs – WIA, Wagner-Peyser, etc. – at the state and 

local levels; however, within a program such as WIA, the state does not track staff time devoted 

to core, intensive, or training services. A second state uses a case study approach to break down 

costs by activity: under a special study focusing on a small number of local workforce areas in 

the state, a team of university researchers conducted site visits to One-Stop Career Centers to 

develop estimates of staff and other expenditures by program activity within the One-Stop 

setting.48 This proved to be a very time intensive and the researchers uncovered idiosyncrasies in 

co-enrollment, sharing of costs, and accounting for costs that made it difficult to make valid 

comparisons across local areas.49 

Challenge #2 - Need to Account for Expenditures of Funds Received Over Multiple 

Years. After programs receive an initial allotment of funding in a program, states and local 

workforce areas have several years to expend funds (for example, under the WIA program, states 

have up to three years to expend program funds after they have been allocated for a given 

program year). As a result, when collecting and analyzing expenditure data to support efficiency 

measure calculations, it is necessary to gather expenditure amounts from allotments for the 

current program year and past program years expended during the program year of focus. For 

example, for WIA, this means that to calculate cost per entered employment for PY 2006, 

expenditures that occurred during the program year (July 1, 2006 through June 30, 2007) of 

 

48Richard Moore, Phillip Gorman, and Andrew Wilson, California One-Stop System Cost Study Report, prepared for 

the California Workforce Investment Board, prepared by California State University at Northridge, October 18, 

2007. 

49An alternative to collecting cost data by activity would be to take into account the activity mix of participants (e.g., 

the percentage of individuals trained) when setting performance standards for efficiency measures). 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 87 

program funds received for the current program year (PY 2006), as well as the two previous 

program years (PY 2004 and PY 2005) would have to be gathered. 

Challenge #3 – Expenditures for Individuals Served by the 11 Programs May Occur 

over One or Several Years. Customers are enrolled in programs for varying periods, depending 

upon the types of services delivered. For a labor exchange program, such as the Wagner-Peyser 

program, expenditures typically occur over a relatively short period of time – several weeks to 

several months. Such expenditures typically would be confined to one or two program years, 

depending upon when an individual enters the program (e.g., if an individual enrolled near the 

end of the program year, then participation could easily stretch across more than one program 

year). At the other extreme are programs that provide substantive training, such as the 

Apprenticeship, WIA, and TAA programs. For example, individuals enrolled in WIA and TAA 

may participate in training that extends over two or three program years (using funding 

stretching across multiple program years). Expenditures on apprentices may go on for up to five 

years, depending on the length of the apprenticeship program. To further complicate matters, the 

funding for training costs for apprentices comes from states, unions, and/or employers (rather 

than the federal government). The duration of program participation and expenditures for other 

ETA programs also varies considerably – for example, an older worker may be engaged in 

subsidized work under the SCSEP program for one year or much longer. The overlapping of 

expenditures across two or more program years is not a big problem, but because an individual is 

counted as exiting or entering employment only for the last year of participation in the program, 

it requires a simplifying “steady state” assumption (i.e., that although costs of serving an 

individual may lap across more than one year, unless there are extraordinary changes in patterns 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 88 

of expenditures or exiting, costs per outcome will even out over the long run and from year to 

year).

Challenge #4 – Lags Occur in Reporting on Expenditures at Various Levels of 

Government and Final Expenditure Data May Not Become Available for Several Years 

After the Fact. For programs with several administrative levels, there may be a significant time 

lag between when money is spent at the lowest program level (such as a service provider or 

contractor) and when it is reported to the local workforce area or grantee, the state, and finally, 

the federal government). States submit final annual expenditure data for the ETA programs 

generally within 90 days of the end of each program year. However, the expenditure data 

submitted by states can be revised after it is submitted to ETA – for example, financial audits 

may result in disallowance or modification of program expenditures for a year or even longer 

after the close out of a program year.

Challenge #5 – Inclusion or Exclusion of Subsidized Wages, Need-Based Payments, 

and Stipends Can Substantially Affect Performance on Efficiency Measures. With regard to 

programs such as SCSEP (where about three-quarters of program expenditures are devoted to 

wage subsidies for program participants), TAA (where substantial amounts of program funds are 

expended on Trade Readjustment Allowances [TRA] or income supports), and NEG (where in 

the case of NEGs in response to disasters a large portion of fund are expended on wages for 

temporary workers rather than training), the inclusion or exclusion of such payments can very 

significantly alter outcomes for efficiency measures such as cost per entered employment.50 

 

50The Federal Trade Act provides special benefits under the Trade Adjustment Assistance (TAA) program to those 

who were laid off or had hours reduced because their employer was adversely affected by increased imports from 

other countries. These benefits include paid training for a new job, financial help in making a job search in other 

areas, or relocation to an area where jobs are more plentiful. Those who qualify may be entitled to weekly Trade 

Adjustment Allowances (TRA) after their unemployment compensation is exhausted. Additional background on 

TAA and TRA is available on the U.S. Department of Labor website, at 

http://workforcesecurity.doleta.gov/unemploy/tra.asp. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 89 

Moreover, whether a customer receives TRA or unemployment compensation (which would not 

be included as a TAA cost) depends in part on factors such as the state’s unemployment rate. 

ETA officials overseeing SCSEP and TAA observed that definitions of the cost items included in 

the numerator of efficiency measures need to be carefully defined – and in the case of these two 

programs, should exclude wage subsidies and income supports.

 Challenge #6 – Burden/Costs Likely to Be Imposed on States and Localities to 

Ensure Standardized Cost Data Are Collected; Technical Assistance and Training Likely 

to Be Necessary. Given the variation in practices across states and localities, collection of 

expenditure data in a form that is comparable across states/grantees and local programs would 

likely impose substantial burden and costs on reporting units (i.e., states, grantees, and 

localities), particularly if such cost data was furnished by program and by discreet activity. It 

will likely be necessary to provide guidance, training, technical assistance, and ongoing 

monitoring to ensure that expenditure data are provided in an appropriate format. Also, to avoid 

gaming by grantees, ETA may find it necessary to standardize some aspects of cost reporting so 

that measures are consistent across grantees and so that grantees do not make changes to their 

allocation of costs simply to improve measured performance.

 Challenge #7 – Cost-of-Living Differences, Along with Differing Practices with 

Regard to Co-Enrollment, Sharing of Costs Across Programs, and Accounting Procedures, 

Make States Anxious About Comparisons of Efficiency Measure Results Across States and 

Localities. Cost-of-living differences across states and localities (especially staff and facility 

cost differences, which are major cost elements) could affect performance on efficiency 

measures. One state official, for example, cautioned against ETA making what might be unfair 

comparisons across states: “We would have concerns if you start comparing states or setting 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 90 

standards because costs vary from state to state. Costs are so significantly different across states, 

for example, compared to (say) Kansas, our state has much higher rents and staff costs. If 

comparisons were made across states, you might need a cost-of-living adjustment. It is tough to 

compare a state with among the highest cost–of-living on an efficiency measure to say the 49th or 

50th state (in terms of cost-of-living). There is a fairness issue – how much it costs to do 

business in one state versus another state is a factor.”51 Different policies with regard to coenrollment and cost sharing across states and local WIBs can have substantial effects on 

efficiency measure results. Most of the states interviewed for this study expressed concerns over 

the potential for inappropriate and unfair comparisons being made once efficiency measure 

results became available.

C. CONCLUSIONS

The main recommendation that emerges from this chapter is that ETA should use actual 

expenditures in calculating efficiency measures because (1) expenditures can vary substantially 

from what is initially appropriated/allocated, especially at the state level (e.g., because of 

transfers, rescissions, and unexpended funds unexpended funds); and (2) expenditures reflect 

what is actually spent on delivery of services and capture the underlying notion of efficiency. 

Despite this overall recommendation, there are serious challenges documented in this chapter to 

collecting and utilizing expenditure data that can complicate uniform implementation of 

efficiency measures and cloud interpretation of the results nationally and across grantees/states, 

including: (1) substantial variation in the ways in which states and local workforce areas collect, 

allocate, and report on expenditures; (2) complexities of accounting for cost sharing and co-

 

51One potentially complicating factor for making adjustments for cost-of-living differences is that there is not a 

federal government cost-of-living index to adjust costs across states. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 91 

enrollment across closely linked workforce development programs (particularly within a OneStop system that emphasizes integration across programs and sharing of funds to serve often coenrolled individuals); (3) accounting for funds that are allocated in one year, but often spent over 

the course of several years; (4) accounting for expenditures on individuals (especially for 

apprentices and others receiving long-term training) that could be spread over several years (e.g., 

as many as five years for apprentices); (5) lags in reporting of expenditures to the federal 

government from local workforce areas and states/grantees, as well as adjustments that may 

occur many years later due to audits; (6) complexities associated with determining which 

expenditures should be include in efficiency measure calculations, particularly related to highcost items such as needs-based payments, stipends, and subsidized wages (that can rival or 

exceed training and other employment service costs in programs such TAA, NEG, and SCSEP); 

and (7) variations across states and localities in cost-of-living. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 92 

CHAPTER 6: 

OPTIONS AVAILABLE, ANALYSIS, AND CHALLENGES 

TO CALCULATING EFFICIENCY MEASURES 

 

As observed earlier in this report, to generate an efficiency measure, it is necessary to 

account for costs in the numerator of the measure and a unit of participation, service delivery, or 

outcome in the denominator of the measure. The preceding chapter focused on the cost side of 

the equation, and this chapter focuses on the denominator – primarily outcomes – that could be 

used to produce one or more efficiency measures that could be implemented across the 11 ETA 

programs of interest. It first examines a range of outcome measures that could be coupled with 

cost data to produce outcome-based efficiency measures for implementation by some or all the 

ETA programs. The chapter then presents quantitative and qualitative analyses on a core set of 

the outcome-based efficiency measures with the aim of narrowing to a set of feasible and 

relevant efficiency measures for possible implementation. 

A. OUTCOME/PERFORMANCE MEASURES USED BY ETA PROGRAMS 

 One key recommendation of state workforce administrators and ETA program offices 

was that, at least in the short run, ETA efficiency measures be tied to existing 

outcome/performance measures, as well as data currently being collected by each of the 

programs. Use of existing outcome measures – particularly those used under Common 

Measures, would make it much less burdensome for states and local areas to implement such 

efficiency measures. In addition, data on employment outcomes for ETA programs (e.g., 

employment, retention, and earnings) comes from other administrative systems – for example, 

the Unemployment Insurance wage record system – that are fixed and unlikely to change. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 93 

Exhibit 6-1 compares the types and definitions of outcome measures used by the 11 ETA 

programs. As shown in this exhibit, 9 of the 11 ETA programs are currently (as of December 

2009) using the same three outcome measures under what are termed the “Common Measures”: 

WIA Adult, WIA Dislocated Worker; NEG, TAA, Wagner-Peyser/ES; SCSEP, NFJP, WIG, and 

INA programs. The three outcome measures commonly in use across these programs are entered 

employment rate, employment retention rate, and average (post-program) earnings (see the 

exhibit for how each of these measures is defined).52 Each of the ETA programs covered by the 

Common Measures also have management information systems that support the collection and 

analysis of participant-level data required for calculating outcomes reported under Common 

Measures. 

The WIA Youth program is also covered by the Common Measures, but the performance 

measures used are somewhat differently defined for WIA Youth. For example, as shown in 

Exhibit 6-1, a WIA Youth exiter is viewed as a having a “positive” outcome if he or she is placed 

in employment or education: “of those who are not in postsecondary education or employment 

(including the military) at the date of participation, the number of youth participants who are in 

employment (including the military) or enrolled in postsecondary education and/or advanced 

training/occupational skills training in the first quarter after the exit quarter divided by the 

number of youth participants who exit during the quarter.” The WIA Youth program 

performance is also assessed in terms of the program’s capacity to help youth 

 

52In addition, in TEGL 17-05, ETA defines a set of parameters for each of these measures; for example, with regard 

to the entered employment rate: “Individuals who are employed at the date of participation are excluded from this 

measure (i.e., programs will not be held accountable for these individuals under this measure). Individuals who, 

although employed at the date of participation, have either received a notice of termination of employment or whose 

employer has issued a Worker Adjustment and Retraining Notification (WARN) or other notice that the facility or 

enterprise will close, or who are transitioning service members are considered not employed at the date of 

participation and are included in the performance measure. Employment at the date of participation is based on 

information collected from the individual, not from wage records. 
EXHIBIT 6-1: DEFINITIONS AND USE OF OUTCOME MEASURES ACROSS ETA PRO

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 94 

PROGRAM EMPLOYMENT MEASURES EMPLOYMENT RETENTION MEASURES EARNINGS MEASURES EDU

ATTAMEA

WIA (Excluding Youth), Wagner-Peyser, TAA, SCSEP, INA, WIG, and NFJP. (Note: All of these programs use Common Measures.) 

Entered Employment: Of those who are not employed at the date of participation, the number of participants who are employed in the first quarter after the exit quarter divided by the number of 

participants who exit during the quarter. 

Employment Retention: Of those who are employed in the first quarter after the exit quarter, the number of participants who are employed in both the second and third 

quarters after the exit quarter divided by the number of 

participants who exit during the quarter. 

Average Earnings: Of those participants who are employed in the first, second, and third quarters after the exit quarter, total earnings in the second quarter plus total earnings in the third quarter after the exit quarter divided by the number of participants who exit during the quarter. 

WIA Youth (Note: The Youth program is covered by Common Measures, but uses difference measures.) 

Placement in Employment or Education: Of those who are not in 

postsecondary education or employment (including the military) at the date of participation, the number of youth participants who are in employment (including the military) or enrolled in postsecondary education and/or advanced training/occupational skills training in the first quarter after the exit quarter divided by the number of youth participants who exit during the quarter. 

None None Attai

Certieducaparticthe prparticGED, the thquarteyouth the quLiterof thoare banumbincreafunctinumbhave progrdate oplus tparticcomp
EXHIBIT 6-1: DEFINITIONS AND USE OF OUTCOME MEASURES ACROSS ETA PRO

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 95 

PROGRAM EMPLOYMENT MEASURES EMPLOYMENT RETENTION MEASURES EARNINGS MEASURES EDU

ATTAMEA

Apprenticeship Program (Apprenticeship is transitioning from these measures to the Common Measures) 

None Employment Retention: The 

number of apprentices employed nine months after registration divided by the number of apprentices registered in the first quarter of the fiscal year. 

Earnings Gain: The difference 

between the average of the current wage of the total number of entrants still employed nine months later and the average of the starting wage of the total number of entrants registered in the first quarter of the fiscal year. 

AveraAppr

allocafedera(appre

Note: The aim of Common Measures is to have common performance measures for programs with similar goals. Originalcollaborated to define Common Measures, and as of July 1, 2005, these measures were implemented for the WIA Title IB a

Peyser/Employment Services and VETS programs; TAA programs implemented the Common Measures on October 1, 200

NFJP transitioned to the Common Measures reporting system in PY 2006. Common Measure outcomes are calculated for WIASRD data -- for each measure, the result is calculated for WIA enrolled individuals with disabilities for the workforce Disability Program Navigator (DPN) cooperative agreements. The Apprenticeship is making the transition to the Common (from the performance measures shown in the table). Source: U.S. Department of Labor website for the Glossary of Performance. More detailed discussion of each of these me

in Training and Employment Guidance Letter No. 17-05, “Common Measures Policy for the Employment and Training Ad

Performance Accountability System and Related Performance Issues,” issued by DOL/ETA, February 17, 2006,and a chan

guidance.
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 96 

attain additional educational degrees/certifications and improvements in numeracy and literacy 

proficiency. 

The WIG program is also covered by the Common Measures displayed in Exhibit 6-1, 

but because the WIG program has no enrolled customers, the measures must be implemented 

somewhat differently.53 For each measure, the rate is calculated for WIA enrolled exiters with 

disabilities served in workforce areas receiving Disability Program Navigator (DPN) grants. 

This means that other customers of the One-Stop system not enrolled in WIA, such as WagnerPeyser exiters (i.e., not co-enrolled in WIA), go uncounted under the Common Measures for the 

WIG program.54 When interpreting performance and efficiency measures for the WIG program, 

it is important to take into consideration that the program is primarily a capacity building 

program aimed at improving access and service delivery for individuals with disabilities within 

the workforce development system. Unlike the other Common Measure programs, individuals 

are not formally enrolled in the WIG program or served directly by Disability Navigators.55 

Apprenticeship is in the process of transitioning from the measures shown in Exhibit 6-1 

to the Common Measures. During FY 2009, the Office of Apprenticeship (OA) is collecting 

baseline data to support the transition to the Common Measures. Three quarters of data collected 

through Common Reporting Information System (CRIS) will be reviewed and analyzed. Targets 

for entered employment, retention, and average earnings will be set using the baseline results for 

 

53The program has set annual performance goals for each of the three Common Measures, as well as for a fourth 

measure – percentage of exiters with disabilities in the workforce areas receiving a DPN cooperative agreement. 54In a February 11, 2008 Memorandum to Round One and Two DPN states, notes: “Although the Wagner-Peyser 

program registers many more individuals than those identified in the WIASRD, ETA is unable to derive data at the 

local workforce area level to identify impacts of Navigators located there.” 55To further complicate matters, disability status is not always disclosed by individuals served under WIA, ES, 

TAA, and other programs – and there is considerably variability across programs in the percentage of individuals 

who in fact disclose whether they have a disability. Individuals with disabilities are often co-enrolled in a number of 

ETA programs and a range of other programs (particularly Vocational Rehabilitation) – so accounting for 

participation and tying participation to costs is complicated. Finally, with regard to cost, the budget for WIG is a 

fraction of the budget of programs such as WIA and ES, and costs of the WIG program (particularly related to 

supporting the more than 500 Disability Navigators across the country) is supplemented by funds from the Social 

Security Administration and states (e.g., in New York, an estimated half the costs of Disability Navigators is paid by 

the state). Finally, funding for the WIG program has not been renewed and the program will end in the coming year. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 97 

apprentices who complete training programs during subsequent reporting years.56 As shown in 

Exhibit 6-1, the performance measures used in the past by Apprenticeship, while including 

measures of employment and earnings, are somewhat different from those mandated under the 

Common Measures. For example, the Apprenticeship Program has not in the past used a 

measure of entered employment. The Apprenticeship program has used the concept of retention 

in employment, but the definition and way in which “retention” has been calculated are different 

from that of the Common Measures, i.e., retention in the Apprenticeship program has been 

defined as “the number of apprentices employed nine months after registration divided by the 

number of apprentices registered in the first quarter of the fiscal year.” 

As also shown in the exhibit, while the Apprenticeship program has used an earnings 

measure, unlike the Common Measures, the earnings measure has been one of “earnings gain” 

and is substantially different in nature from the Common Measures’ post-program “average 

earnings” measure, i.e., earnings gain in the Apprenticeship program has in the past been defined 

as “the difference between the average of the current wage of the total number of entrants still 

employed nine months later and the average of the starting wage of the total number of entrants 

registered in the first quarter of the fiscal year.”57 There are also three other issues (discussed 

earlier in this report) regarding Apprenticeship that make the program different from the other 

ETA programs and should be taken into consideration in establishing efficiency measures 

appropriate to this program: (1) generally, costs of training and serving apprentices are paid by 

the state, unions, employers, and/or participants (i.e., so federal costs of the program are low and 

generally are targeted on promoting Apprenticeship and helping to ensure that programs offered 

are of high quality); (2) apprentices are enrolled for up to five years, so costs in the year of exit 

 

56Specifically, OA received the first set of data from the CRIS; two additional quarters of data are expected 

December 2008 and January 2009; and a draft of the targets is expected to be available by June 2009. 57 In prior years the Common Measures used earnings gain rather than post-program earnings. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 98 

may be much less representative of the total cost than in other programs; and (3) some states 

administer their apprenticeship program themselves, so there are no federal costs. 

Overall, the current measures of participation and outcome collected by the 11 ETA 

programs suggest that at least in the short run, a feasible and potentially cost-effective strategy 

for implementing efficiency measures would be adopting efficiency measures that build on the 

structure of the outcome Common Measures being collected and reported by ETA programs. As 

will be discussed later in this report in the study’s recommendations (see Chapter 8), 

considerable caution is required and there are a number of serious methodological and data 

constraints that will need to be resolved before valid and reliable efficiency measurement can be 

implemented by the 11 ETA programs. 

B. ANALYSES OF EFFICIENCY MEASURES RESULTS BASED ON THE 

COMMON MEASURE OUTCOMES AT THE NATIONAL LEVEL 

ETA program offices provided data on Common Measure performance outcomes, and the 

Office of Financial and Administrative Management (OFAM) provided data on program 

expenditures to support development of estimates of efficiency measure results for PY/FY 2007. 

The estimates provided in this section and the next section (on state-level results) are preliminary 

and (as discussed throughout this report) great care should be taken in making cross program 

comparisons. For several program it was also possible to collect cost and outcome data that 

permitted three years of analyses of efficiency measure results.58 Exhibit 6-2 provides 

preliminary analysis of results for efficiency measure results for candidate efficiency measures 

(based on the Common Measures) at the national level for PY/FY 2007. 

 

58 OFAM provided three years (PY 2005 through PY 2007) of expenditure data for the 11 ETA programs; Common 

Measure outcome results for three years were provided for the WIA (Adults, Dislocated Worker, Youth, and NEG), 

Wagner-Peyser, and TAA programs. NFJP was able to provide PY 2007 outcome data, but not for earlier because 

the program had only started to track Common Measure outcomes in PY 2007. Outcome data was not available for 

INAP, WIG, or the Apprenticeship program. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 99 

EXHIBIT 6-2: ESTIMATED RESULTS FOR FOUR OUTCOME-BASED EFFICIENCY 

MEASURES FOR ETA PROGRAMS, NATIONAL RESULTS, PY/FY 2007 

Notes: Figures are based on data provided by OFAM on expenditures and ETA program offices for outcomes. 

*For WIA-Youth, cost per placement in education or employment was used in place of cost per entered 

employment. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 100 

EXHIBIT 6-2: ESTIMATED RESULTS ON FOUR OUTCOME-BASED EFFICIENCY 

MEASURES, FOR ETA PROGRAMS, NATIONAL RESULTS, PY/FY 2007 

Notes: Figures are based on data provided by OFAM on expenditures and ETA program offices for outcomes. 

*For WIA-Youth, cost per placement in education or employment was used in place of cost per entered 

employment. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 101 

The four charts shown in Exhibit 6-2 demonstrate the substantial variability in cost across 

programs. Below, the results are highlighted for each of these four measures: 

• Cost per Exiter: The term “exiter” is defined as follows under the Common Measures: 

“An exiter is a participant who has not received a service funded by the program or 

funded by a partner program for 90 consecutive calendar days, and is not scheduled for 

future services. The exit date is the last date of service.”59 The “cost per exiter” 

efficiency measure is calculated by taking total program costs (in terms of expenditures) 

and dividing by the number of exiters terminating the program during the year by the 

particular program. Cost per exiter in PY/FY 2007 ranged from in excess of $15,000 in 

the TAA program ($15,199) to less than $1,000 for the WIA Adult Program ($468) and 

Wagner-Peyser/ES Program ($54). As also shown in the exhibit, when only training 

costs are considered under the TAA program (i.e., Trade Adjustment Allowances [TRA] 

are excluded) cost per exiter decreases by about two-thirds, from $15,199 to $4,775.60

• Cost per Entered Employment. This outcome-based efficiency measure is calculated 

by taking total program costs (in terms of expenditures) and dividing by the number of 

exiters (who were not employed at program entry) entering employment in the first 

quarter following exit from the particular program. As shown in the exhibit, in PY/FY 

2007, cost per entered employment ran in excess of $20,000 for the TAA program 

($23,546) to just $113 in the Wagner-Peyser program. The costs shown for the WIA 

Youth Program -- $15,769 -- are for cost per placement in employment or education. This 

very substantial variation points to the widely varying cost structure for programs that 

provide intensive assistance and training services (such as TAA) versus those providing 

less customer intensive, labor exchange-type services. Even when only training costs are 

considered for the TAA program, costs are much higher than the Wagner-Peyser Program 

and similar to the WIA-Dislocated Worker Program. 

• Cost per Retained Employment. This efficiency measure is calculated by taking total 

program costs (in terms of expenditures) and dividing by the number of exiters employed 

in the first quarter after exit and who are employed in both the second and third quarters 

after the exit quarter. Exhibit 6-2 shows how cost per retained employment, based on 

expenditures, varies significantly across ETA programs for PY/FY 2007. As shown in 

the exhibit, cost per retention in PY/FY 2007 were as high as $24,206 for the TAA 

Program ($7,604 for TAA when only training costs are included) to less $3,000 for the 

 

59Training and Employment Guidance Letter No. 17-05, “Common Measures Policy for the Employment and 

Training Administration’s (ETA) Performance Accountability System and Related Performance Issues,” issued by 

DOL/ETA, February 17, 2006 and a change to this original guidance. 60Cost per exiter can be affected significantly at the national and particularly state levels by policies related to coenrollment and where the gateway for “participation” or “enrollment” occurs for individuals entering the One-Stop 

system. For example, even though expenditures for the WIA Adult Program have remained virtually unchanged (at 

just under $1 billion), the number of program exiters increased from 626,225 to 1,984,923 between PY 2005 and PY 

2007. This change in the number of exiters (a 217 percent increase between PY 2005 and PY 2007) is reflected in a 

steep decrease in cost per exiter – from $940 in PY 2005 to $331 in PY 2007 (a 64.8 percent decrease). The 

precipitous increase in exiters (and fall in per exiter costs) is the result (at least in part) of changes in policies in 

some states governing enrollment of self-service One-Stop customers and co-enrollment of Wagner-Peyser 

participants into WIA. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 102 

WIA Adult Program ($2,558), WIA NEG ($1,779), and the Wagner-Peyser Program 

($116). 

• Cost Divided by Post-Program Earnings. This efficiency measure is calculated by 

taking total program costs (in terms of expenditures) and dividing by total earnings in the 

second quarter plus total earnings in the third quarter after the exit quarter. Exhibit 6-2 

shows that cost divided by post-program earnings range as high as $2.05 for TAA and 

$1.48 for NFJP, to less than $0.20 for the WIA Adult ($0.19), NEG ($0.15) and WagnerPeyser (less than $0.01). 

At the national level, not only is there considerable variation across programs, but within 

programs there is substantial year-to-year variation in efficiency measure results. Exhibit 6-3 

provides a series of charts that compare efficiency measure results for several years (usually 

three years) for the WIA Adult, WIA Dislocated Worker, WIA Youth, Wagner-Peyser, and 

Trade Assistance Act Programs. The top portion of each exhibit graphically shows efficiency 

measure results by program year on three efficiency measures – cost per exiter, cost per entered 

employment, and cost per retained in employment. The bottom portion of each exhibit provides 

more detailed data in a table, including data on factors that determine each of the efficiency 

measures, as well as efficiency measure results. For example, for the WIA Adult Program, the 

table provides data on expenditures, number of exiters, number entering employment, number 

retained in employment, and post-program earnings for PY 2005-07. In addition, the table 

provides data on one additional efficiency measure not included in the graph – cost divided by 

post program earnings. The table also shows the change for each data item between 2005 and 

2007, which is helpful in identifying particular factors that may be changing rapidly and 

potentially driving efficiency measure results (e.g., increases in exiters versus expenditures). 

 Across the five programs shown in Exhibit 6-3, the program displaying the greatest 

change over the three-year period is the WIA Adult program. As shown in the first of the 

exhibits (Exhibit 6-3a), while expenditures declined slightly (6.2 percent) over the three-year 

period (PY 2005-07) for the WIA Adult Program, there was more than a tripling of the number 
EXHIBIT 6-3a: EFFICIENCY MEASURE RESULTS 

FOR SELECTED PROGRAMS, 2005-07

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 103 

WIA Adult Program PY 2005 PY 2006 PY 2007 

Change PY 

2005-07 

Expenditures $989,481,877 $972,812,239 $928,304,645 -6.2% 

# Exiters 626,182 1,081,243 1,984,923 217.0% 

# Entering Employment 133,982 219,701 436,207 225.6% 

# Retained in Employment 148,470 208,540 362,920 144.4% 

Post Program Earnings N/A $2,349,832,107 $4,784,807,604 N/A 

Cost per Exiter $1,580 $900 $468 -70.4% 

Cost per Entered Employment $7,385 $4,428 $2,128 -71.2% 

Cost per Retained in Employment $6,665 $4,665 $2,558 -61.6% 

Cost Divided by Post Program Earnings N/A $0.41 $0.19 N/A 

Note: Figures are based on data provided by OFAM on expenditures and ETA program offices for outcomes.
EXHIBIT 6-3b: EFFICIENCY MEASURE RESULTS 

FOR SELECTED PROGRAMS, 2005-07

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 104 

WIA Dislocated Worker PY 2005 PY 2006 PY 2007 

Change PY 

2005-07 

Expenditures $1,073,768,048 $1,061,829,731 $1,062,881,904 -1.0% 

# Exiters 152,350 208,906 233,845 53.5% 

# Entering Employment 120,415 118,946 153,352 27.4% 

# Retained in Employment 122,279 111,208 133,012 8.8% 

Post Program Earnings N/A $1,488,444,765 $1,925,846,839 N/A 

Cost per Exiter $7,048 $5,083 $4,545 -35.5% 

Cost per Entered Employment $8,917 $8,927 $6,931 -22.3% 

Cost per Retained in Employment $8,781 $9,548 $7,991 -9.0% 

Cost Divided by Post Program Earnings N/A $0.71 $0.55 N/A 

Note: Figures are based on data provided by OFAM on expenditures and ETA program offices for outcomes.
EXHIBIT 6-3c: EFFICIENCY MEASURE RESULTS 

FOR SELECTED PROGRAMS, 2005-07

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 105 

WIA Youth PY 2005 PY 2006 PY 2007 

Change PY 

2005-07 

Expenditures $964,935,524 $906,348,036 $897,344,260 -7.0% 

# Exiters 149,210 119,155 111,905 -25.0% 

# Placed in Employment or Education N/A 63,721 56,906 N/A 

Cost per Exiter $6,467 $7,606 $8,019 24.0% 

Cost per Placed in Employment or 

Education N/A $14,224 $15,769 N/A 

Note: Figures are based on data provided by OFAM on expenditures and ETA program offices for outcomes. Data 

was not available for PY 2005 on the number placed into employment or education.
EXHIBIT 6-3d: EFFICIENCY MEASURE RESULTS 

FOR SELECTED PROGRAMS, 2005-07

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 106 

Wagner-Peyser/ES PY 2005 PY 2006 PY 2007 

Change PY 

2005-07 

Expenditures $733,372,773 $741,538,970 $716,291,359 -2.3% 

# Exiters 12,099,618 12,317,667 13,167,927 8.8% 

# Entering Employment 6,292,364 6,025,247 6,337,438 0.7% 

# Retained in Employment 6,428,226 6,350,889 6,173,210 -4.0% 

Post Program Earnings $1,828,165,860,512 $1,706,491,491,655 $1,421,133,516,337 -22.3% 

Cost per Exiter $61 $60 $54 -10.3% 

Cost per Entered Employment $117 $123 $113 -3.0% 

Cost per Retained in Employment $114 $117 $116 1.7% 

Cost Divided by Post Program 

Earnings $0.0004 $0.0004 $0.0005 25.6% 

Note: Figures are based on data provided by OFAM on expenditures and ETA program offices for outcomes.
EXHIBIT 6-3e: EFFICIENCY MEASURE RESULTS 

FOR SELECTED PROGRAMS, 2005-07

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 107 

TAA-Training and Benefits PY 2006 PY 2007 

Change PY 

2006-07 

Expenditures $713,360,351 $788,352,845 10.5% 

# Exiters 55,482 51,870 -6.5% 

# Entering Employment 29,489 33,482 13.5% 

# Retained in Employment 34,735 32,569 -6.2% 

Post Program Earnings $490,009,891 $384,218,859 -21.6% 

Cost per Exiter $12,858 $15,199 18.2% 

Cost per Entered Employment $24,191 $23,546 -2.7% 

Cost per Retained in Employment $20,537 $24,206 17.9% 

Cost Divided by Post Program Earnings $1.46 $2.05 40.9% 

 

TAA -Training Expenses Only PY 2006 PY 2007 

Change PY 

2006-07 

Expenditures $193,456,107 $247,659,201 28.0% 

Exiters 55,482 51,870 -6.5% 

# Entering Employment 29,489 33,482 13.5% 

# Retained in Employment 34,735 32,569 -6.2% 

Post Program Earnings $490,009,891 $384,218,859 -21.6% 

Cost per Exiter $3,487 $4,775 36.9% 

Cost per Entered Employment $6,560 $7,397 12.8% 

Cost per Retained in Employment $5,569 $7,604 36.5% 

Cost Divided by Post Program Earnings $0.39 $0.64 63.3% 

Note: Figures are based on data provided by OFAM on expenditures and ETA program offices for outcomes.
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 108 

of exiters (217 percent increase, from 626,182 to nearly 2 million exiters, as shown in the table). 

As a result of the surge in the number of exiters, there was also very substantial percentage 

increases in the number of exiters entering employment (226 percent increase) and exiters 

retained in employment (144 percent increase). The increases in numbers of individuals flowing 

through the WIA Adult Program finding and retaining jobs are reflected in steep decreases in 

efficiency measure results between PY 2005 and PY 2007 – a 70 percent decrease in the cost per 

exiter, a 71 percent decrease in cost per exiter entering employment, and a 62 percent decrease in 

cost per exiter retained in employment. As discuss earlier in this report, some states have 

aggressively moved in the direction of co-enrolling large numbers of Wagner-Peyser customers 

(and in some cases other One-Stop customers as well) into the WIA program, which accounts for 

some (and perhaps much) of the surge in enrollment in the WIA program in recent years. 

 The patterns of change as shown in the other four exhibits that make up Exhibit 6-3, 

while not showing nearly the volatility of the WIA Adult Program, demonstrate the considerable 

year-to-year fluctuation in efficiency measure results (and the underlying factors that determine 

efficiency measure results). Generally, with the exception of the TAA program (for which only 

two years of results were available), there was a relatively slight decrease in program 

expenditures (e.g., -1 percent for the WIA Dislocated Worker). The TAA Program for the two 

years for which data were available experienced about a 10 percent increase in overall 

expenditures (and a 28 percent increase for training expenses, as shown in Exhibit 6-3e). 

Patterns for efficiency measure results across programs over the three-year period examined 

varied in both direction (i.e., increases and decreases) and in extent of change. The efficiency 

measure results for the WIA Dislocated Worker Program (shown in Exhibit 6-3b), similar to the 

WIA Adult Program, displayed a significant downward trend, though not as dramatic as for the 

WIA Adult Program – for example, cost per entered employment decreased by 22 percent from 

$8,917 in PY 2005 to $6,931 in PY 2007. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 109 

Data available for the WIA Youth Program (Exhibit 6-3c) shows a slight decrease in 

expenditures (a 7 percent decrease), accompanied by a one-quarter decline in exiters (from 

149,210 exiters in PY 2005 to 111,905 exiters in PY 2007). Cost per exiter increased by about 

the same percentage as the loss in the number of exiters (24 percent). Data on cost per exiter 

placed in employment or education (the Common Measure outcome for the WIA Youth 

Program, which was available for only two years), increased slightly (from $14,244 in PY 2006 

to $15,769 in PY 2007). 

 With regard to the Wagner-Peyser Program (see Exhibit 6-3d), change in the efficiency 

measure results for the three years was relatively modest. The cost per entered employment, for 

example, decreased slightly over the three-year period (by 3 percent) from $117 in PY 2005 to 

$113 in PY 2007. As discussed earlier in this chapter, given its emphasis on less intensive labor 

exchange services, the efficiency measure results (in dollar terms) for the Wagner-Peyser 

Program are only a fraction of those in other programs (e.g., cost per entered employment was 

slightly above $100 in each of the three-years observed, compared to costs per entered 

employment cost per entered employment in excess of $20,000 for the TAA program, when 

training and benefit costs are included). 

Finally, data was available on efficiency measure results for the TAA Program for two 

years – and those two years, demonstrate just how much efficiency measure results (and 

underlying factors such as expenditures, can change from one year to the next. In addition, the 

TAA results shown in the final exhibit in the series (in Exhibit 6-3e) show how in a program like 

TAA efficiency measure results can be quite different when cash benefit costs (i.e., Trade 

Adjustment Allowances, TRA) are included in or excluded from program expenditures. As 

shown in the exhibit, unlike the other programs, expenditures increased by a fairly significant 

margin over the two years for which data was available – training and benefit expenditures 

together increased by 10.5 percent between FY 2006 and FY 2007 and TAA training 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 110 

expenditures alone increased by nearly one-quarter (23.7 percent). The table shows that while 

cost per entered employment decreased slightly (by -3 percent, from $24,191 to $23,546) for the 

two-year period when training and benefit costs are taken into consideration, when only training 

costs are considered, cost per entered employment increased by 13 percent (from $6,560 to 

$7,397). 

C. ANALYSES OF EFFICIENCY MEASURES RESULTS AT THE STATE LEVEL 

Several exhibits follow that show the considerable variation that exists in efficiency 

measure results across states (for programs that collect data at the state level and were able to 

furnish outcome results that could be matched with expenditures to produce efficiency measure 

results). The estimates provided in this section of the report (on state-level results) are 

preliminary and (as discussed throughout this report) great care should be taken in making crossprogram and cross-state comparisons. In addition, there are critical differences in the ways in 

which programs are structured and operate, as well as the ways in which expenditures are 

accounted for that may explain the often highly variable efficiency measure results across states 

(and even within states, across program years). The analyses presented in this section are 

intended to begin to establish a baseline of data on efficiency measure results by state, as well as 

to highlight how variable and volatile patterns can be within programs across states (and within 

states, from year to year). Some key trends with regard to state-level results on efficiency 

measures are highlighted below: 

• There is substantial variation across programs at state levels in efficiency 

measure results. Exhibit 6-4 show state level patterns for PY 2007 for the WIA 

Adult, WIA Dislocated Worker, WIA Youth, Wagner-Peyser, TAA, and National 

Emergency Grants Programs on four Common Measure defined efficiency measures: 

cost per exiter (Exhibit 6-4a), cost per entered employment (6-4b), cost per retained 

in employment (Exhibit 6-4c), and cost divided by post program earnings (Exhibit 6-

4d). At the bottom of each table is the average across all states, as well as a summary 
EXHIBIT 6-4a: COST PER EXITER BY PROGRAM, PY 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 111 

 State WIAAdult 

WIADW WIAYouth WagnerPeyser 

TAA 

(Training 

Only) 

TAA 

(Training and 

Benefits) 

NEG 

Alabama $4,512 $5,902 $9,755 $34 $19,923 $44,986 N/A 

Alaska $6,814 $18,074 $5,081 $82 $8,546 $9,106 $489 

Arizona $768 $6,025 $11,013 $58 $3,137 $4,995 N/A 

Arkansas $407 $34,589 $9,911 $38 $8,296 $23,085 $965 

California $3,877 $10,458 $9,489 $59 $11,463 $23,956 $12,262 

Colorado $5,409 $9,053 $8,782 $53 $4,877 $15,276 N/A 

Connecticut $271 $14,168 $14,017 $57 $4,640 $13,652 $1,979 

Delaware $1,516 $9,056 $11,530 $61 N/A N/A N/A 

DC N/A $189,049 $23,308 $158 N/A N/A N/A 

Florida $2,453 $7,165 $7,428 $48 $4,584 $9,018 $3,979 

Georgia $7,178 $13,557 $7,904 $42 N/A N/A $1,366 

Hawaii $2,200 $6,359 $8,660 $57 $129 $212 $1,693 

Idaho $155 $5,172 $5,864 $32 $4,618 $8,222 $11,531 

Illinois $418 $8,661 $10,588 $68 $6,582 $20,086 $8,442 

Indiana $361 $5,488 $6,962 $46 $2,581 $12,686 N/A 

Iowa $7,939 $6,717 $9,519 $36 $24,434 $38,357 $4,382 

Kansas $153 $36,138 $9,145 $94 $4,908 $13,052 $31,923 

Kentucky $6,258 $6,650 $9,336 $45 $7,549 $26,333 $186 

Louisiana $269 $9,585 $10,925 $60 $6,265 $12,835 $1,235 

Maine $8,200 $6,663 $8,539 $114 $1,931 $4,267 $3,717 

Maryland $87 $12,002 $9,837 $95 $2,809 $13,151 $73,966 

Massachusetts $2,707 $5,505 $10,892 $85 $7,945 $17,536 $2,012 

Michigan $108 $14,135 $8,750 $53 $7,540 $33,045 N/A 

Minnesota $5,432 $4,175 $4,946 $67 $7,179 $16,300 $47,901 

Mississippi $547 $688 $4,317 $24 $1,169 $3,272 $1,408 

Missouri $4,349 $7,358 $8,175 $81 $14,698 $26,613 $5,980 

Montana $66 $8,227 $11,142 $66 $7,261 $9,770 $3,410 

Nebraska $7,009 $10,515 $11,482 $90 $6,063 $15,296 $23 

Nevada $870 $5,475 $10,620 $56 $1,612 $4,868 N/A 

New Hampshire $51 $5,697 $7,224 $62 $1,829 $5,453 $5,071 

New Jersey $7,767 $7,247 $7,826 $96 $5,285 $24,126 N/A 

New Mexico $5,104 $20,578 $6,004 $56 $4,025 $6,000 N/A 

New York $146 $658 $10,124 $72 $1,097 $4,239 N/A 

North Carolina $973 $10,329 $10,960 $33 $4,050 $22,458 $724 

North Dakota $160 $8,623 $5,572 $90 $19,944 $37,424 N/A 

Ohio $5,506 $16,010 $10,680 $74 $2,400 $7,801 $10,504 

Oklahoma $123 $12,685 $15,049 $59 $7,009 $20,876 $2,483 

Oregon $6,433 $9,104 $9,274 $30 $11,786 $28,772 $3,016 

Pennsylvania $6,405 $6,944 $9,543 $84 N/A N/A $264 

Puerto Rico $7,407 $26,507 $3,244 $194 $62 $87 N/A 

Rhode Island $3,050 $11,078 $10,431 $108 $4,314 $17,436 $13,879 

South Carolina $2,200 $5,659 $7,983 $28 $10,122 $35,344 N/A 

South Dakota $2,851 $3,667 $4,807 $80 $3,160 $5,752 N/A 

Tennessee $2,172 $7,902 $5,663 $36 $1,034 $8,698 N/A 

Texas $3,559 $9,852 $6,803 $35 $2,848 $6,410 $986 

Utah $34 $5,373 $6,560 $52 $4,801 $11,890 N/A 
EXHIBIT 6-4a: COST PER EXITER BY PROGRAM, PY 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 112 

 State WIAAdult 

WIADW WIAYouth WagnerPeyser 

TAA 

(Training 

Only) 

TAA 

(Training and 

Benefits) 

NEG 

Vermont $11,506 $12,412 $9,719 $132 $8,190 $25,664 N/A 

Virginia $401 $5,963 $7,466 $60 $4,195 $14,547 $106,481 

Washington $438 $7,363 $9,529 $53 $3,396 $8,221 $44,190 

West Virginia $3,966 $5,903 $9,954 $43 $10,880 $26,620 $5,114 

Wisconsin $5,047 $5,882 $7,457 $176 $4,464 $12,377 $3,650 

Wyoming $151 $5,100 $14,081 $53 $31,445 $31,445 N/A 

 

Total $468 $4,545 $8,106 $54 $4,775 $15,199 $2,018 

 High State $11,506 $189,049 $23,308 $194 $31,445 $31,445 $106,481 

 Low State $34 $658 $4,317 $24 $62 $87 $23 

Note: Figures are based on data provided by OFAM on expenditures and ETA program offices for outcomes.
EXHIBIT 6-4b: COST PER ENTERED EMPLOYMENT BY PROGRAM, PY 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 113 

State WIA-Adult WIA-DW WIAYouth* 

WagnerPeyser 

TAA 

(Training 

Only) 

TAA 

(Training and 

Benefits) 

NEG 

Alabama $6,831 $7,198 $19,464 $67 $29,538 $66,696 N/A 

Alaska $30,694 $54,410 $8,910 $244 $16,798 $17,898 $1,613 

Arizona $4,437 $8,064 $19,808 $288 $3,872 $6,165 N/A 

Arkansas $18,119 $51,783 $17,592 $68 $15,136 $42,121 $633 

California $9,035 $14,905 $15,677 $157 $16,028 $33,496 $16,390 

Colorado $10,619 $10,415 $15,171 $97 $6,868 $21,510 N/A 

Connecticut $14,915 $16,846 $23,362 $104 $8,539 $25,121 $3,383 

Delaware $11,031 $10,140 $18,105 $114 N/A N/A N/A 

DC N/A $129,971 $19,771 $288 N/A N/A N/A 

Florida $10,189 $8,855 $15,768 $106 $8,337 $16,403 $4,248 

Georgia $11,912 $18,067 $13,090 $72 N/A N/A $1,401 

Hawaii $9,968 $7,214 $16,702 $170 $178 $294 $2,389 

Idaho $8,516 $5,536 $9,617 $121 $6,978 $12,424 $14,706 

Illinois $13,506 $12,039 $20,684 $164 $8,558 $26,119 $6,030 

Indiana $5,415 $8,311 $16,350 $97 $3,952 $19,424 N/A 

Iowa $10,826 $8,361 $14,669 $74 $31,320 $49,166 $8,170 

Kansas $7,556 $40,351 $23,105 $141 $6,515 $17,327 $282,744 

Kentucky $10,443 $7,708 $16,358 $73 $12,385 $43,205 $302 

Louisiana $559 $9,920 $21,253 $138 $21,302 $43,638 $793 

Maine $14,380 $8,368 $16,933 $203 $3,113 $6,877 $5,156 

Maryland $10,265 $14,988 $17,968 $201 $3,873 $18,131 $92,457 

Massachusetts $15,409 $6,823 $17,049 $163 $9,742 $21,502 $3,228 

Michigan $11,952 $15,644 $17,802 $123 $11,646 $51,043 N/A 

Minnesota $8,169 $7,667 $7,877 $162 N/A N/A $93,520 

Mississippi $947 $876 $6,761 $36 $1,923 $5,384 $1,809 

Missouri $7,417 $10,050 $15,482 $152 $22,331 $40,433 $6,108 

Montana $11,266 $10,479 $17,095 $97 $635,339 $854,867 $2,747 

Nebraska $13,246 $10,952 $17,368 $155 $6,729 $16,975 $16 

Nevada $7,550 $7,872 $22,702 $100 $4,192 $12,656 N/A 

New Hampshire $8,530 $6,607 $15,241 $107 $2,780 $8,290 N/A 

New Jersey $9,280 $8,184 $14,780 $169 $10,030 $45,788 $330,208 

New Mexico $10,658 $16,414 $12,665 $126 $5,154 $7,685 N/A 

New York $296 $1,337 $15,688 $135 $1,885 $7,282 N/A 

North Carolina $1,659 $11,275 $18,162 $65 $5,361 $29,730 $640 

North Dakota $7,014 $10,607 $10,757 $248 $24,930 $46,780 N/A 

Ohio $9,836 $18,315 $19,263 $113 $3,733 $12,130 $11,024 

Oklahoma $333 $17,715 $23,272 $99 $8,488 $25,282 $1,943 

Oregon $9,813 $10,388 $17,236 $66 $33,234 $81,133 $3,787 

Pennsylvania $13,372 $9,791 $20,480 $250 N/A N/A $309 

Puerto Rico $12,012 $37,003 $13,488 $2,434 $92 $129 N/A 

Rhode Island $5,117 $13,467 $26,853 $152 $6,176 $24,963 N/A 

South Carolina $9,284 $9,329 $16,224 $43 $14,674 $51,237 N/A 

South Dakota $6,477 $3,686 $9,615 $185 $4,235 $7,708 N/A 

Tennessee $5,947 $10,258 $11,807 $59 $1,893 $15,925 N/A 

Texas $6,100 $13,369 $14,629 $85 $4,687 $10,547 $1,536 

Utah $153 $10,313 $14,100 $87 $7,387 $18,292 N/A 

Vermont $15,589 $11,128 $25,484 $285 $10,374 $32,507 N/A 
EXHIBIT 6-4b: COST PER ENTERED EMPLOYMENT BY PROGRAM, PY 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 114 

State WIA-Adult WIA-DW WIAYouth* 

WagnerPeyser 

TAA 

(Training 

Only) 

TAA 

(Training and 

Benefits) 

NEG 

Virginia $8,492 $9,309 $13,229 $126 $6,324 $21,928 N/A 

Washington $8,527 $7,811 $15,123 $100 $5,106 $12,362 $28,775 

West Virginia $7,062 $5,444 $15,751 $87 $15,808 $38,677 $4,845 

Wisconsin $8,256 $6,567 $14,055 $375 $7,141 $19,799 $4,834 

Wyoming $15,034 $5,525 $22,338 $113 $47,168 $47,168 N/A 

 

Total $2,128 $6,931 $15,769 $113 $7,397 $23,546 $2,532 

 High State $30,694 $129,971 $26,853 $2,434 $635,339 $854,867 $330,208 

 Low State $153 $876 $6,761 $36 $178 $294 $302 

Note: Figures are based on data provided by OFAM on expenditures and ETA program offices for 

outcomes. * Cost per placement into Employment or Education is used for the WIA Youth Program. 
EXHIBIT 6-4c: COST PER RETAINED IN EMPLOYMENT BY PROGRAM, PY 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 115 

 State 

WIAAdult WIA-DW 

WagnerPeyser 

TAA 

(Training 

Only) 

TAA 

(Training and 

Benefits) NEG 

Alabama $5,290 $6,348 $53 $34,925 $78,858 N/A 

Alaska $16,842 $34,952 $183 $18,736 $19,963 $474 

Arizona $4,549 $12,284 $225 $4,653 $7,410 N/A 

Arkansas $11,667 $51,783 $71 $13,703 $38,132 $412 

California $6,983 $16,619 $137 $20,200 $42,215 $14,002 

Colorado $8,526 $8,978 $99 $7,181 $22,491 N/A 

Connecticut $12,508 $18,573 $127 $8,815 $25,934 $1,968 

Delaware $7,256 $9,230 $120 N/A N/A N/A 

DC N/A $116,828 $348 N/A N/A N/A 

Florida $3,366 $7,908 $111 $9,833 $19,347 $982 

Georgia $11,121 $18,718 $78 N/A N/A $1,498 

Hawaii $9,929 $8,175 $151 $194 $320 $10,258 

Idaho $5,095 $4,508 $55 $5,924 $10,549 $10,250 

Illinois $11,660 $13,449 $115 $9,061 $27,652 $6,253 

Indiana $7,156 $7,806 $100 $3,919 $19,260 N/A 

Iowa $7,171 $8,864 $60 $34,263 $53,786 $15,379 

Kansas $5,680 $47,055 $107 $7,184 $19,106 N/A 

Kentucky $7,209 $7,805 $71 $13,240 $46,187 $250 

Louisiana $960 $14,741 $111 $22,824 $46,755 $970 

Maine $10,638 $7,895 $151 $2,976 $6,574 $3,329 

Maryland $8,780 $13,767 $182 $3,892 $18,219 $10,567 

Massachusetts $13,937 $6,918 $188 $11,209 $24,740 $3,886 

Michigan $11,799 $16,739 $125 $12,566 $55,075 N/A 

Minnesota $6,426 $7,644 $107 $11,077 $25,148 $63,352 

Mississippi $900 $1,033 $53 $2,158 $6,042 $803 

Missouri $6,879 $11,094 $159 $23,659 $42,838 $3,797 

Montana $9,022 $8,848 $81 $11,345 $15,265 $1,848 

Nebraska $8,630 $9,621 $157 $6,897 $17,399 $4 

Nevada $6,328 $10,201 $113 $2,994 $9,040 N/A 

New Hampshire $7,978 $6,051 $121 $2,987 $8,908 $53,244 

New Jersey $10,137 $8,384 $201 $10,987 $50,156 $330,208 

New Mexico $7,265 $15,096 $138 $5,650 $8,424 N/A 

New York $487 $1,997 $179 $1,854 $7,162 N/A 

North Carolina $963 $9,961 $68 $6,062 $33,620 $430 

North Dakota $7,736 $8,159 $220 $24,930 $46,780 N/A 

Ohio $7,353 $18,761 $163 $4,227 $13,735 $17,959 

Oklahoma $806 $12,618 $97 $9,543 $28,424 $1,114 

Oregon $8,758 $11,127 $55 $16,883 $41,216 $3,553 

Pennsylvania $13,200 $9,560 $367 N/A N/A $795 

Puerto Rico $10,286 $31,285 N/A $91 $128 N/A 

Rhode Island $3,547 $13,755 $161 $6,841 $27,648 $26,602 

South Carolina $8,663 $10,841 $51 $16,574 $57,872 N/A 

South Dakota $6,127 $3,245 $121 $4,849 $8,826 N/A 

Tennessee $4,321 $10,590 $70 $2,109 $17,743 N/A 

Texas $6,502 $12,716 $129 $4,334 $9,754 $929 

Utah $89 $8,813 $73 $7,816 $19,356 N/A 

Vermont $13,808 $8,722 $339 $10,732 $33,628 N/A 
EXHIBIT 6-4c: COST PER RETAINED IN EMPLOYMENT BY PROGRAM, PY 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 116 

 State 

WIAAdult WIA-DW 

WagnerPeyser 

TAA 

(Training 

Only) 

TAA 

(Training and 

Benefits) NEG 

Virginia $5,433 $8,562 $105 $7,075 $24,534 N/A 

Washington $7,739 $7,815 $88 $5,097 $12,339 $10,575 

West Virginia $5,269 $5,429 $95 $18,380 $44,970 $7,464 

Wisconsin $5,822 $4,609 $304 $6,676 $18,510 $2,131 

Wyoming $10,467 $2,455 $104 $94,336 $94,336 N/A 

 

Total $2,558 $7,991 $116 $7,604 $24,206 $1,779 

 High State $16,842 $116,828 $367 $94,336 $94,336 $4 

 Low State $89 $1,033 $51 $91 $128 $330,208 

Note: Figures are based on data provided by OFAM on expenditures and ETA program offices for outcomes.
EXHIBIT 6-4d: COST DIVIDED BY POST PROGRAM EARNINGS, 

BY PROGRAM, PY 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 117 

State WIA-Adult WIA-DW 

WagnerPeyser TAA (Training 

Only) 

TAA (Training 

and Benefits) NEG 

Alabama $0.477 $0.429 $0.000 $3.183 $7.188 N/A 

Alaska $1.147 $1.750 $0.006 $1.974 $2.103 $0.026 

Arizona $0.407 $0.902 $0.006 $0.404 $0.644 N/A 

Arkansas $1.015 $4.011 $0.001 $1.789 $4.978 $0.047 

California $0.436 $1.041 $0.000 $1.355 $2.833 $1.157 

Colorado $0.672 $0.630 $0.001 $0.460 $1.442 N/A 

Connecticut $1.158 $1.152 $0.002 $0.540 $1.590 $0.089 

Delaware $0.738 $0.704 $0.007 N/A N/A N/A 

DC N/A $7.504 $0.017 N/A N/A N/A 

Florida $0.180 $0.513 $0.000 $1.741 $3.426 $0.096 

Georgia $0.927 $1.280 $0.000 N/A N/A $0.132 

Hawaii $0.820 $0.558 $0.010 $0.020 $0.033 $0.786 

Idaho $0.482 $0.343 $0.001 $0.536 $0.955 $0.555 

Illinois $1.039 $0.857 $0.001 $0.814 $2.485 $0.385 

Indiana $0.637 $0.552 $0.001 $0.311 $1.531 N/A 

Iowa $0.865 $0.773 $0.001 $2.594 $4.071 $1.361 

Kansas $0.455 $3.198 $0.003 $0.406 $1.079 N/A 

Kentucky $0.483 $0.643 $0.001 $1.178 $4.109 $0.021 

Louisiana $0.077 $1.031 $0.001 $9.375 $19.204 $0.069 

Maine $1.102 $0.663 $0.008 $0.249 $0.551 $0.193 

Maryland $0.758 $0.932 $0.003 $0.317 $1.482 $0.805 

Massachusetts $1.520 $0.445 $0.002 $0.870 $1.921 $0.191 

Michigan $1.413 $1.432 $0.001 $1.048 $4.591 N/A 

Minnesota $0.527 $0.451 $0.001 $3.856 $8.754 $3.326 

Mississippi $0.094 $0.095 $0.000 $0.266 $0.745 $0.071 

Missouri $0.763 $0.870 $0.002 $2.015 $3.649 $0.209 

Montana $0.687 $0.548 $0.002 $0.795 $1.070 $0.117 

Nebraska $0.904 $0.780 $0.004 $0.702 $1.772 N/A 

Nevada $0.530 $0.644 $0.002 $0.176 $0.532 N/A 

New Hampshire $0.866 $0.381 $0.004 $0.246 $0.734 $3.258 

New Jersey $0.902 $0.570 $0.002 $1.230 $5.617 $15.396 

New Mexico $0.713 $1.281 $0.003 $0.335 $0.500 N/A 

New York $0.032 $0.115 $0.001 $0.148 $0.571 N/A 

North Carolina $0.085 $0.752 $0.000 $0.677 $3.756 $0.038 

North Dakota $0.745 $0.688 $0.010 $1.725 $3.237 N/A 

Ohio $0.844 $1.459 $0.001 $0.307 $0.997 $1.211 

Oklahoma $0.071 $0.955 $0.001 $3.877 $11.547 $0.090 

Oregon $0.883 $0.845 $0.000 $1.106 $2.701 $0.280 

Pennsylvania $1.059 $0.642 $0.003 N/A N/A $0.081 

Puerto Rico $3.127 $8.273 $0.000 $0.007 $0.010 N/A 

Rhode Island $0.286 $0.986 $0.009 $0.505 $2.039 $3.638 

South Carolina $1.018 $0.966 $0.000 $1.600 $5.586 N/A 

South Dakota $0.570 $0.246 $0.004 $0.966 $1.757 N/A 

Tennessee $0.346 $0.867 $0.000 $0.231 $1.947 N/A 

Texas $0.540 $0.901 $0.000 $0.306 $0.689 $0.084 
EXHIBIT 6-4d: COST DIVIDED BY POST PROGRAM EARNINGS, 

BY PROGRAM, PY 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 118 

Utah $0.007 $0.610 $0.001 $0.574 $1.421 N/A 

Vermont $1.092 $0.563 $0.039 $0.835 $2.618 N/A 

Virginia $0.557 $0.654 $0.001 $0.674 $2.338 N/A 

Washington $0.639 $0.439 $0.001 $0.315 $0.764 $0.510 

West Virginia $0.530 $0.386 $0.002 $1.282 $3.138 $0.323 

Wisconsin $0.619 $0.322 $0.009 $0.582 $1.613 $0.167 

Wyoming $0.911 $0.175 $0.003 $30.943 $30.943 N/A 

 

Total $0.194 $0.552 $0.001 $0.645 $2.052 $0.147 

 High State $3.127 $8.273 $0.039 $30.943 $30.943 $15.396 

 Low State $0.007 $0.175 $0.000 $0.020 $0.033 $0.021 

Note: Figures are based on data provided by OFAM on expenditures and ETA program offices for outcomes. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 119 

of the high and low state value for each program on each measure. One of the most 

interesting findings of this analysis is how significant the difference is between the 

high and low state. For example, in Exhibit 6-4b, for the WIA Adult Program, while 

the average cost per entered employment was $2,128, the range between the highest 

state (Vermont, which recorded costs per entered employment of $11,506) and the 

lowest state (Utah, $34) was substantial. The often extreme differences between the 

highest and lowest state demonstrate how efficiency measure results can dramatically 

differ across states. In addition, the extremes suggest that data submitted by states on 

either expenditures our outcomes may be based on inconsistencies in data collection 

or simply erroneous data. 

• Within programs there are substantial year-to-year fluctuations on efficiency 

measure results. On a given measure, such as cost per entered employment, there is 

substantial change in efficiency measure results from year-to-year at the state level. 

Overall, as shown in Exhibit 6-5a, the national average cost per entered employment 

for the WIA Adult Program decreased from $7,385 in PY 2005 to $2,128 in PY 2007 

(a 71 percent decrease). Exhibit 6-5a, which is sorted by state in descending order by 

percentage change in cost per entered employment between PY 2005 and PY 2007, 

shows that for some states the cost per entered employment for the WIA Adult 

Program more than doubled over the 3-year period (Georgia, 155 percent; Alaska, 

141 percent, and Nevada, 125 percent), while for four states cost per entered 

employment decreased by more than 90 percent between PY 2005 and PY 2007 (New 

York, -92 percent; Louisiana, -96 percent; Oklahoma, -97 percent; and Utah, -99 

percent). 

• Underlying trends point to volatility in number of exiters terminated rather than 

expenditure patterns as a key factor in determining efficiency measure results, 

especially for the WIA Adult Program. Exhibits 6-5b and 6-5c illustrate how 

underlying patterns of expenditure and participation can drive efficiency measure 

results nationally and at the state level. Exhibit 6-5b show extremely large increases 

in the numbers of WIA Adult exiters at the national level entering employment (a 226 

percent increase, from 133,982 entering employment in PY 2005 to 436,207 entering 

employment in PY 2007). Several states accounted for a large share of this increase, 

especially New York, with the number of exiters entering employment increasing by 

902 percent, from 20,963 to 210,049. Utah (7,107 percent increase), Oklahoma 

(3,245 percent increase), and Louisiana (2,837 percent increase) had even higher 

percentage increases in the number of exiters entering employment (though the actual 

increase in numbers of exiters entering employment were far less than those recorded 

in New York). Exhibit 6-6c shows a modest decrease in expenditures at the national 

level for the WIA Adult program from PY 2005 to PY 2007 (a -6.2 percent decrease), 

but fairly substantial swings in expenditures (both positive and negative) when 

patterns are observed at the state level. For example, expenditures increased by over 

50 percent for the WIA Adult Program over the three-year period in three states (76 

percent in South Carolina, 54 percent in Mississippi, and 51 percent in Colorado), 

while falling by in excess of 25 percent in five states.
EXHIBIT 6-5a: CHANGE IN COST PER ENTERED EMPLOYMENT, 

WIA ADULT PROGRAM, PY 2005 TO 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 120 

State PY 2005 PY 2006 PY 2007 Change 2005 

to 2007 

Georgia $4,676 $7,419 $11,912 154.7% 

Alaska $12,759 $11,307 $30,694 140.6% 

Nevada $3,360 $4,333 $7,550 124.7% 

Maine $7,927 $9,576 $14,380 81.4% 

Colorado $6,597 $9,461 $10,619 61.0% 

South Dakota $4,091 $7,382 $6,477 58.3% 

South Carolina $6,739 $11,103 $9,284 37.8% 

Michigan $8,883 $10,831 $11,952 34.5% 

Arkansas $13,696 $15,249 $18,119 32.3% 

Missouri $5,700 $7,261 $7,417 30.1% 

Oregon $7,656 $8,354 $9,813 28.2% 

Alabama $5,574 $9,229 $6,831 22.5% 

Florida $8,342 $8,680 $10,189 22.1% 

Hawaii $8,322 $14,012 $9,968 19.8% 

Nebraska $11,081 $10,212 $13,246 19.5% 

New Hampshire $7,201 $8,990 $8,530 18.5% 

Massachusetts $13,642 $10,532 $15,409 12.9% 

Wisconsin $7,585 $9,436 $8,256 8.8% 

Maryland $9,503 $8,740 $10,265 8.0% 

North Dakota $6,637 $7,166 $7,014 5.7% 

Mississippi $899 $878 $947 5.3% 

Ohio $9,733 $11,163 $9,836 1.1% 

Tennessee $5,958 $6,156 $5,947 -0.2% 

Illinois $14,090 $13,358 $13,506 -4.1% 

Iowa $11,304 $10,952 $10,826 -4.2% 

California $9,825 $9,681 $9,035 -8.0% 

Puerto Rico $13,321 $14,149 $12,012 -9.8% 

Virginia $9,464 $10,223 $8,492 -10.3% 

Washington $9,654 $8,657 $8,527 -11.7% 

Delaware $12,720 $9,010 $11,031 -13.3% 

West Virginia $8,210 $12,645 $7,062 -14.0% 

Montana $13,161 $16,652 $11,266 -14.4% 

Minnesota $9,835 $9,832 $8,169 -16.9% 

Wyoming $18,209 $17,753 $15,034 -17.4% 

New Jersey $11,305 $10,404 $9,280 -17.9% 

Pennsylvania $16,637 $14,996 $13,372 -19.6% 

Connecticut $18,900 $14,361 $14,915 -21.1% 

Texas $7,811 $7,161 $6,100 -21.9% 

Indiana $7,176 $7,413 $5,415 -24.5% 

New Mexico $14,208 $12,168 $10,658 -25.0% 

Idaho $12,681 $8,854 $8,516 -32.8% 

Vermont $23,544 $18,471 $15,589 -33.8% 

Kentucky $17,266 $12,889 $10,443 -39.5% 

Kansas $13,707 $19,406 $7,556 -44.9% 
EXHIBIT 6-5a: CHANGE IN COST PER ENTERED EMPLOYMENT, 

WIA ADULT PROGRAM, PY 2005 TO 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 121 

State PY 2005 PY 2006 PY 2007 Change 2005 

to 2007 

Arizona $8,178 $7,404 $4,437 -45.7% 

Rhode Island $10,687 $6,153 $5,117 -52.1% 

North Carolina $14,849 $10,387 $1,659 -88.8% 

New York $3,797 $1,573 $296 -92.2% 

Louisiana $13,077 $3,415 $559 -95.7% 

Oklahoma $9,763 $5,259 $333 -96.6% 

Utah $12,449 $79 $153 -98.8% 

DC $8,801 $7,263 N/A N/A 

 

Total $7,385 $4,428 $2,128 -71.2% 

Notes: Figures are based on data provided by OFAM on costs and ETA program offices for outcomes.
EXHIBIT 6-5b: CHANGE IN NUMBER ENTERING EMPLOYMENT, 

WIA ADULT PROGRAM, PY 2005 TO 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 122 

 State PY 2005 PY 2006 PY 2007 Change 2005 

to 2007 

Utah 438 64,698 31,565 7106.6% 

Oklahoma 879 1,319 29,400 3244.7% 

Louisiana 1,689 6,348 49,598 2836.5% 

New York 20,963 40,860 210,049 902.0% 

Kentucky 1,088 1,300 1,899 74.5% 

Indiana 1,800 2,050 2,853 58.5% 

Vermont 79 99 124 57.0% 

Mississippi 16,688 16,884 24,442 46.5% 

Kansas 474 437 657 38.6% 

Arizona 2,095 2,048 2,734 30.5% 

South Carolina 2,131 1,981 2,720 27.6% 

Delaware 197 244 246 24.9% 

Idaho 222 299 277 24.8% 

Texas 11,640 12,146 14,366 23.4% 

Rhode Island 254 357 298 17.3% 

Connecticut 368 475 421 14.4% 

Pennsylvania 2,091 2,385 2,379 13.8% 

Wyoming 147 156 165 12.2% 

Tennessee 3,092 3,238 3,463 12.0% 

New Jersey 2,230 2,190 2,484 11.4% 

Ohio 4,313 4,541 4,794 11.2% 

Iowa 341 362 363 6.5% 

New Mexico 672 807 713 6.1% 

Massachusetts 1,029 1,123 1,089 5.8% 

Montana 193 149 193 0.0% 

Missouri 2,716 2,680 2,671 -1.7% 

West Virginia 700 496 679 -3.0% 

North Dakota 311 260 300 -3.5% 

Illinois 3,183 3,340 3,050 -4.2% 

New Hampshire 258 227 246 -4.7% 

Minnesota 957 906 903 -5.6% 

Colorado 1,736 1,480 1,629 -6.2% 

Washington 2,547 2,533 2,387 -6.3% 

California 13,901 13,460 12,954 -6.8% 

Michigan 4,885 4,313 4,549 -6.9% 

Nebraska 222 279 200 -9.9% 

Virginia 1,302 1,262 1,158 -11.1% 

North Carolina 2,260 2,660 1,938 -14.2% 

Maryland 1,029 1,015 875 -15.0% 

Puerto Rico 2,855 2,466 2,395 -16.1% 

Arkansas 636 516 528 -17.0% 

Florida 5,422 5,939 4,447 -18.0% 

South Dakota 494 320 368 -25.5% 

DC 385 421 277 -28.1% 

Oregon 2,174 1,837 1,553 -28.6% 

Wisconsin 1,552 1,229 1,093 -29.6% 
EXHIBIT 6-5b: CHANGE IN NUMBER ENTERING EMPLOYMENT, 

WIA ADULT PROGRAM, PY 2005 TO 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 123 

 State PY 2005 PY 2006 PY 2007 Change 2005 

to 2007 

Alabama 3,169 1,788 1,891 -40.3% 

Hawaii 443 228 254 -42.7% 

Maine 362 298 199 -45.0% 

Georgia 3,441 2,152 1,689 -50.9% 

Alaska 224 200 107 -52.2% 

Nevada 1,705 900 575 -66.3% 

 

Total 133,982 219,701 436,207 225.6% 

Notes: Figures are based on data provided by OFAM on costs and ETA program offices for outcomes.
EXHIBIT 6-5c: CHANGE IN PROGRAM EXPENDITURES, 

WIA ADULT PROGRAM, PY 2005 TO 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 124 

State PY 2005 PY 2006 PY 2007 Change 2005 

to 2007 

South Carolina $14,360,256 $21,995,034 $25,251,355 75.8% 

Mississippi $15,006,538 $14,830,793 $23,148,499 54.3% 

Colorado $11,452,728 $14,001,577 $17,298,946 51.0% 

Missouri $15,480,533 $19,458,293 $19,810,617 28.0% 

Louisiana $22,087,656 $21,677,197 $27,744,974 25.6% 

Michigan $43,393,133 $46,715,168 $54,368,281 25.3% 

Georgia $16,091,106 $15,966,403 $20,118,760 25.0% 

Indiana $12,916,277 $15,196,059 $15,449,667 19.6% 

Massachusetts $14,037,922 $11,827,175 $16,780,295 19.5% 

South Dakota $2,020,786 $2,362,352 $2,383,442 17.9% 

Alaska $2,858,082 $2,261,426 $3,284,220 14.9% 

Oklahoma $8,582,068 $6,936,044 $9,780,427 14.0% 

New Hampshire $1,857,842 $2,040,803 $2,098,339 12.9% 

Ohio $41,977,559 $50,691,722 $47,155,811 12.3% 

Tennessee $18,420,971 $19,932,239 $20,595,096 11.8% 

Arkansas $8,710,671 $7,868,303 $9,566,922 9.8% 

Delaware $2,505,938 $2,198,398 $2,713,681 8.3% 

Nebraska $2,460,076 $2,849,267 $2,649,259 7.7% 

Kentucky $18,785,452 $16,756,060 $19,830,730 5.6% 

Vermont $1,859,946 $1,828,584 $1,933,057 3.9% 

North Dakota $2,064,034 $1,863,074 $2,104,255 1.9% 

Iowa $3,854,824 $3,964,688 $3,929,696 1.9% 

Florida $45,228,422 $51,550,344 $45,310,101 0.2% 

Maine $2,869,752 $2,853,751 $2,861,647 -0.3% 

Texas $90,920,855 $86,983,125 $87,627,892 -3.6% 

Wyoming $2,676,683 $2,769,487 $2,480,679 -7.3% 

Illinois $44,847,898 $44,615,193 $41,194,133 -8.1% 

Maryland $9,778,360 $8,871,352 $8,981,576 -8.1% 

Oregon $16,644,227 $15,347,195 $15,239,406 -8.4% 

Pennsylvania $34,786,979 $35,766,601 $31,812,213 -8.6% 

New Jersey $25,210,314 $22,784,831 $23,050,998 -8.6% 

Connecticut $6,955,344 $6,821,389 $6,279,168 -9.7% 

Utah $5,452,807 $5,095,187 $4,836,487 -11.3% 

California $136,577,973 $130,301,074 $117,033,956 -14.3% 

Montana $2,540,090 $2,481,122 $2,174,267 -14.4% 

Idaho $2,815,094 $2,647,200 $2,358,925 -16.2% 

West Virginia $5,747,333 $6,271,884 $4,794,828 -16.6% 

Washington $24,588,371 $21,927,581 $20,353,995 -17.2% 

Virginia $12,322,012 $12,901,625 $9,833,971 -20.2% 

New Mexico $9,547,783 $9,819,766 $7,599,500 -20.4% 

Minnesota $9,412,227 $8,908,141 $7,376,734 -21.6% 

New York $79,595,396 $64,264,267 $62,131,044 -21.9% 

Wisconsin $11,772,235 $11,597,449 $9,024,091 -23.3% 

Kansas $6,497,283 $8,480,620 $4,964,203 -23.6% 

Nevada $5,729,486 $3,899,799 $4,341,042 -24.2% 
EXHIBIT 6-5c: CHANGE IN PROGRAM EXPENDITURES, 

WIA ADULT PROGRAM, PY 2005 TO 2007 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 125 

State PY 2005 PY 2006 PY 2007 Change 2005 

to 2007 

Puerto Rico $38,032,686 $34,890,210 $28,769,937 -24.4% 

Alabama $17,665,302 $16,501,484 $12,917,607 -26.9% 

Arizona $17,133,496 $15,162,653 $12,130,943 -29.2% 

Hawaii $3,686,765 $3,194,766 $2,531,942 -31.3% 

Rhode Island $2,714,604 $2,196,710 $1,525,004 -43.8% 

North Carolina $33,559,283 $27,628,947 $3,214,475 -90.4% 

DC $3,388,419 $3,057,827 N/A N/A 

 

Total $989,481,877 $972,812,239 $928,304,645 -6.2% 

Notes: Figures are based on data provided by OFAM on costs and ETA program offices for outcomes.
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 126 

The often extreme differences between the highest and lowest state demonstrate how 

efficiency measure results can dramatically differ across states. In addition, the 

extremes suggest that data submitted by states on either expenditures our outcomes 

may be based on inconsistencies in data collection or simply erroneous data. 

• Within programs there is a lot of variation from year-to-year in efficiency 

measure results. On a given measure, such as cost per entered employment, there is 

substantial change in efficiency measure results from year-to-year at the state level. 

Overall, as shown in Exhibit 6-5a, the national average cost per entered employment 

for the WIA Adult Program decreased from $7,385 in PY 2005 to $2,128 in PY 2005 

(a 71 percent decrease). Exhibit 6-5b, which is sorted in descending order, shows that 

for some states the cost per entered employment for the WIA Adult Program more 

than doubled over the 3-year period (Georgia, 155 percent; Alaska, 141 percent; and 

Nevada, 125 percent), while for four states cost per entered employment decreased by 

more than 90 percent between PY 2005 and PY 2007 (New York, -92 percent; 

Louisiana, -96 percent; Oklahoma, -97 percent; and Utah, -99 percent). 

• Underlying trends point to volatility in number of participant served rather than 

expenditure patterns as a key factor in determining efficiency measure results, 

especially for the WIA Adult Program. Exhibit 6-5b and c illustrate how 

underlying patterns of expenditure and participation can drive efficiency measure 

results nationally and at the state level. Exhibit 6-5b show extremely large increases 

in the numbers of WIA Adult exiters at the national level entering employment (a 226 

percent increase, from 133,982 entering employment in PY 2005 to 436,207 entering 

employment in PY 2007). Several states accounted for a large share of this increase, 

especially New York, with the number entering employment increasing by 902 

percent, from 20,963 to 210,049. Utah (7,107 percent increase), Oklahoma (3,245 

percent increase), and Louisiana (2,837 percent increase) had even high percentage 

increases in numbers entering employment. Exhibit 6-6c shows a modest decrease in 

expenditures at the national level for the WIA Adult program from PY 2005 to PY 

2007 (a -6.2 percent decrease), but fairly substantial swings in expenditures (both 

positive and negative) when patterns are observed at the state level. For example, 

expenditures increased by over 50 percent for the WIA Adult program over the threeyear period in three states (76 percent in South Carolina, 54 percent in Mississippi, 

and 51 percent in Colorado), while falling by in excess of 25 percent in 5 states.

D. CONCLUSIONS 

The analyses in this section – particularly the quantitative analyses – suggest that 

additional time and caution are needed before implementation of one or more efficiency measure 

by the 11 ETA programs that are the focus of this study. As discussed earlier in this chapter, the 

Common Measures provide a structure that could potentially facilitate cost-effective 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 127 

implementation of outcome-based efficiency measures across some or all of the 11 ETA 

programs. The analyses included in this chapter for recent years (2005-07) demonstrate both the 

feasibility of producing outcome-based efficiency measure results for most of the 11 ETA 

programs, but also highlight some of the concerns that have been expressed by ETA and statelevel program officials, as well as the Expert Panel. In particular, the very substantial variation 

across programs in efficiency measure results on a measure such as cost per entered employment 

points to the widely varying cost structure for programs that provide intensive assistance and 

training services (such as the TAA and WIA Programs) versus programs such as the WagnerPeyser program providing less customer intensive, labor exchange-type services. The often 

sizable differences between the highest and lowest state demonstrate how efficiency measure 

results can dramatically differ across states, as well as suggest that data submitted by states on 

either expenditures our outcomes may be based on inconsistencies in data collection or simply 

erroneous data. The next chapter addresses the possible setting of standards (through regression 

models) should ETA implement efficiency measures across (all or some) of the 11 programs. 

Similar to this chapter, the results of this modeling effort suggest that ETA should approach the 

implementation of efficiency measure – and holding states/grantees for performance on such 

measures – with extreme caution. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 128 

CHAPTER 7: 

POTENTIAL APPROACHES TO SETTING AND ADJUSTING STANDARDS FOR 

EFFICIENCY MEASURES 

Performance measurement systems for employment and training program (and other 

public sector programs) – which include standards or expectations – are typically aimed at one or 

more of the following: 

• Encouraging units to maximize some outcome, input, or process of interest; 

• Providing feedback to government monitors and the public on how programs are doing 

(“performing”); 

• Fostering continuous improvement; 

• Assessing if a program is worth maintaining; and 

• Comparing various programs to judge their relative value. 

Designing a performance measurement system for public sector programs typically involves the 

following basic steps. First, one establishes a consensus on specific measurable program goals. 

Second, one defines empirical measures to use in quantitatively assessing performance toward 

those goals (e.g., outcome and efficiency measures). Third, most programs or organizations also 

set expectations for progress toward performance goals, that is, targets for performance 

improvements to be achieved in a given timeframe. In many public sector performance 

measurement systems, these targets are annual, and increasingly, they also may incorporate 

expectations for “continuous performance improvements,” a “total quality management” (TQM) 

principle (Deming, 1986). 61 It is this third step – approaches to establishment of standards (for 

recommended efficiency measures) and the factors that could potentially be applied to adjusting 

 

61 Yet, few public programs opt to undertake the final step of developing formal procedures to regularly adjust 

performance expectations for unanticipated or uncontrollable factors that might thwart progress toward the goals. In 

this regard, they neglect a corresponding tenet of TQM that advocates the use of statistical analysis to adjust for 

factors outside managers’ control in evaluating and managing performance (Deming, 1986). 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 129 

performance from year-to-year and across states/localities – that is the principal focus of this 

chapter. 

 The sections that follow focus on (1) alternative approaches to setting standards for ETA 

programs and (2) alternative approaches to making adjustments to factors that may affect 

outcomes and costs of ETA programs. Prior to this discussion, we highlight ETA’s historical 

experience with performance measurement, with a particular focus on setting performance 

standards and making adjustments. ETA’s prior experience provides some important lessons 

that can help inform decisions about whether and how to set performance standards for 

employment and training programs. 

A. OVERVIEW OF ETA’S EXPERIENCE WITH SETTING AND ADJUSTING 

PERFORMANCE STANDARDS FOR EMPLOYMENT AND TRAINING 

PROGRAMS 62

Performance management for workforce investment programs began on an exploratory 

basis in the later years of the Comprehensive Employment and Training Act (CETA), the 

nation’s major workforce program from 1973 through 1982.63 Economists working for the 

Assistant Secretary for Policy, Evaluation, and Research (ASPER) in the late 1970s advanced the 

idea of holding local CETA programs accountable for their performance as measured by the 

impact of the programs on earnings and employment, but they also recognized the unfairness of 

setting the same earnings or employment rate standard for local programs that served very 

different populations in varying local economic conditions. This laid the groundwork for the 

formalization of performance measures under the Job Training Partnership Act (JTPA), the 

 

62 Portions of this section of the report are based on an unpublished paper co-authored by one of the authors of this 

report (Burt Barnow): Burt Barnow and Carolyn Heinrich, “One Size Fits All? The Pros and Cons of Performance 

Standard Adjustments,” Johns Hopkins University and University of Wisconsin-Madison, Draft (January 23, 2009). 

See the end of the report for a bibliography of articles cited in this section. 

63 Christopher King provided background for this section on the history of performance management under CETA 

and JTPA. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 130 

nation’s primary workforce development program from 1982 through July 2000.64 The 

performance measures used for adults, youth, and dislocated workers, which evolved over time, 

are shown in Exhibit 7-1 for program years 1998 and 1999, the final years that JTPA operated. 

The table also shows the adjustment factors and regression-determined adjustments for the adult 

follow-up employment rate. If, for example, a local program increased the proportion of adult 

terminees who were women by 10 percentage points, the program’s performance standard for the 

adult follow-up employment rate would decline by 0.5 percentage points, if all other 

characteristics remained the same. 

In establishing national performance measures and standards, the federal government 

used data on past experience to set targets that they expected approximately 75 percent of the 

local service delivery areas (SDAs) would be able to meet or exceed; that is, performance 

standards were set at the performance level of the SDA at the 25th percentile in the prior period.65 

Governors were also empowered to select additional measures, set standards for acceptable 

performance for each measure that differed from the Secretary’s standards, determine how 

performance on the individual standards was to be combined to provide an overall measure of 

performance, adjust standards for the state’s SDAs to reflect differences in participant 

characteristics and local economic conditions, and determine sanctions and rewards, subject to 

federal requirements. Although the particular variables and adjustment parameters varied from 

year to year, the basic approach remained the same; the bottom half of Exhibit 7-1 shows the 

variables in the PY 1998-1999 models for the WIA Adult program. Local programs that failed 

 

64 States distributed 78 percent of the federal funds to approximately 600 local units of government and consortia of 

local units of government that were referred to as service delivery areas (SDAs) and were responsible for service 

provision. JTPA program activities, including vocational or basic skills classroom training, on-the-job training 

(OJT), job search assistance, and work experience, were sometimes provided by SDAs themselves but were more 

commonly delivered through contracts with community colleges and other nonprofit and for-profit training 

organizations. 

65 Service delivery areas under JTPA were analogous to local workforce investment areas under WIA. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 131 

EXHIBIT 7-1: 

JTPA PERFORMANCE MEASURES, NATIONAL STANDARDS AND FACTORS IN 

THE NATIONAL ADJUSTMENT MODEL FOR JTPA ADULT FOLLOW-UP RATE 

PERFORMANCE STANDARDS IN PROGRAM YEARS 1998 AND 1999

Performance Measures National Standards 

Title II-A Adult 

 Adult follow-up employment rate 60% 

 Adult weekly earnings at follow-up $289 

 Welfare follow-up employment rate 52% 

 Welfare weekly earnings at follow-up $255 

Title II-C Youth 

 Youth entered employment rate 45% 

 Youth employability enhancement rate 40% 

Title III Dislocated Workers

 Entered employment rate 73% 

Factors and Adjustments in National Model for Adult Follow-Up Employment Rate 

Terminee Characteristics (Percent) Regression 

Adjustment

Local Economic 

Conditions 

Regression 

Adjustment 

 Female 

 Age 55 or more 

 Not a high school graduate 

 Post high school (including college) 

 Dropout under age 30 

 Black (not Hispanic) 

 Minority male 

 Cash welfare recipient 

 Long-term TANF recipient 

 Supplemental Security Income (SSI) recipient 

 Basic skills deficient 

 Individual with a disability 

 Lacks significant work history 

 Homeless 

 Viet Nam era veteran 

 Not in labor force 

 Unemployed 15 or more weeks 

 Unemployment insurance claimant or exhaustee 

-0.050 

-0.130 

-0.066 

0.008 

-0.015 

-0.027 

-0.026 

-0.031 

-0.018 

-0.133 

-0.037 

-0.096 

-0.055 

-0.043 

-0.081 

-0.108 

-0.073 

0.022 

Unemployment rate 

Three-year growth in 

earnings in trade 

Annual earnings in 

trade 

Families with income 

below poverty level 

(percent) 

-0.608 

0.245 

-0.539 

-0.211

Source: Social Policy Research Associates (1999). Guide to JTPA Performance Standards for Program 

Years 1998 and 1999. Menlo Park, CA: Social Policy Research Associates, p. I-3 and II-10. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 132 

to meet half or more of the core JTPA standards were ineligible to receive performance awards, 

and if an SDA failed to meet half or more of the standards in two consecutive years, the governor 

was required to implement a reorganization plan for the SDA. Thus, it is important to note that 

the JTPA performance management system had stronger reward and sanction provisions than 

were later required by the Government Performance and Results Act (GPRA)—organizations 

that did well could obtain substantial additional resources, and those that did poorly could lose 

their right to operate the program. 

Although at the onset, many states simply used the Secretary of Labor’s standards 

without adjustments, by the early 1990s, a majority recognized that failure to adjust standards for 

local economic conditions and participant characteristics was generating incentives to enroll 

individuals who would do best in the labor market regardless of the impact of the program 

(referred to as “cream skimming” or “creaming”). As a result, more governors opted to use the 

Secretary’s adjustment model, and following the 1992 JTPA amendments, governors were 

required to adjust performance standards, either using the optional DOL regression models or 

some alternative approach that was approved by DOL (Barnow, 1992).66 Yet despite the DOL’s 

efforts to “level the playing field” and reduce creaming, there is still ample evidence of gaming 

of the performance management system by local programs to improve their measured 

performance, through actions such as the strategic enrollment and timing of customers’ entry and 

exit from the program (Courty and Marschke, 1996, 2004). A review of research in this area 

 

66 In fact, the criteria for using adjustments other than the DOL regression model were quite strict and discouraged 

states from developing their own adjustment procedures. The Department of Labor’s guide to performance 

standards (Social Policy Research Associates, 1999) stated that the adjustment procedures had to meet the following 

criteria: (1) procedures for adjusting performance standards must be responsive to the Act, consistently applied 

among the SDAs and substate areas, objective and equitable throughout the state, and in conformance with widely 

accepted statistical criteria; (2) source data must be of public use quality and available upon request; (3) results must 

be documented and reproducible, and (4) adjustment factors must be limited to economic factors, labor market 

conditions, geographic factors, characteristics of the population to be served, demonstrated difficulties in serving the 

population, and type of services to be provided.
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 133 

(Barnow and Smith, 2004) concluded that the JTPA system was highly susceptible to 

manipulation by local program operators. At the same time, recent research contrasting the 

JTPA and WIA systems suggests that the regression model adjustments likely tempered such 

problems (Heinrich, 2004); without them, both incentives and means for local programs to 

engage in cream skimming behavior are greater. 

This research also suggests some aspects of the JTPA performance standards adjustment 

procedure that one might question.67 For example, in using data from the prior program year to 

develop regression adjustment models, only statistically significant variables of the expected sign 

and a reasonable magnitude were retained in the model. Thus, if the coefficient for percent of 

participants who were black in the entered employment rate model was either positive (counterintuitively) or not statistically significant, that variable would be excluded from the model for 

that year.68 Although Barnow (1996) showed inconsistent treatment of people with disabilities 

because of this feature, changes in the adjustment procedures did not follow.69 And in other 

ways, the implementation of the performance standards adjustments may have been too rigid. If 

a governor decided to use the Secretary’s adjustment model, he or she could modify the target 

level of performance, but governors could not change the regression coefficients themselves 

(e.g., to encourage enrollment of certain groups beyond simply holding local areas harmless for 

serving that group), nor could they add adjustments for characteristics not in the models (e.g., to 

 

67 An issue we do not discuss here is whether it was correct to estimate the adjustment models at the local workforce 

area level rather than the individual level. There are arguments for and against both approaches, and sometimes 

grouping data can lead to large changes in the estimated relationships (Blalock, 1961). 

68 In addition to the two criteria noted in the text, five other criteria were also used to decide which variables were 

included in the adjustment models (Social Policy Research Associates, 1999, p. III-8). 69 For example, data from a number of years could have been pooled, which would likely have led to fewer changes 

in the variables included in the models and to smaller changes in the magnitude of the adjustments. Another 

alternative would have been to limit the magnitude of the change in the adjustments so that the incentives would not 

vary so much from year to year. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 134 

give credit for serving refugees or the disabled in years when such characteristics were not 

included in the model). 

Overall, despite the deficiencies, the use of the regression adjustment models was largely 

accepted by the various parties and appears to have contributed to leveling the playing field and 

to producing more valuable information for policymakers to use both for program management 

and accountability purposes. By the time the JTPA program ended, the national regressionbased adjustments were the default case and were generally perceived as a fair and appropriate 

way to set performance standards. 

When the Workforce Investment Act became operational in July 2000, the performance 

management system was modified in several significant ways, both in terms of the measures 

used and the performance standards adopted. Standards are now set for states as well as local 

areas, and the standards are “negotiated” rather than set by a regression model. No automatic 

adjustments are made to take account of economic conditions or participant characteristics, but 

states may petition to the Department of Labor if circumstances have changed. The WIA 

legislation did not require dropping the model-based performance management system used 

under JTPA, so the switch was based on preferences rather than necessity.70 There were several 

reasons for substituting a negotiated standards system for a model-based system. First, ETA 

wanted to signal that WIA was going to be different than JTPA, so change was considered good 

in its own right. Second, the group charged with developing the performance management 

system felt that under JTPA the system was “looking back,” and they believed that a negotiated 

standards system was prospective in nature rather than retrospective. Finally, a model-based 

 

70 The lack of a regression adjustment model is not based on statutory language. While not requiring the use of a 

regression model, the statute states that the state-level standards are supposed to be set “taking into account factors 

including differences in economic conditions, the characteristics of participants when the participants entered the 

program, and the services to be provided.” 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 135 

system, by definition, requires that data for the regression models be collected. States indicated 

to ETA that they found the JTPA data collection requirements to be onerous, and they urged that 

the data collection be reduced under WIA (Barnow and Smith, 2004).71

B. OPTIONS FOR SETTING EFFICIENCY MEASURE STANDARDS FOR ETA 

PROGRAMS 

In the three-step process of establishing a performance measurement system discussed 

above, once agreement is attained on performance measures to be adopted, most programs or 

organizations set expectations for progress toward performance goals, that is, targets for 

performance (possibly including improvements) to be achieved in a given timeframe. 

Because WIA, like JTPA and CETA before it, is a multi-level program, measures can be 

applied at the national, state, and/or local levels. Each level of the program is responsible to the 

level directly above it, and in the case of the U.S. Department of Labor, DOL is responsible to 

OMB. OMB has indicated that it is establishing outcome-based efficiency measures for all 

programs, so the primary task for ETA is determining how best to assure that the states have 

incentives to help ETA meet the standards imposed by OMB. Under WIA, establishing 

performance measures and standards for the Local Workforce Investment Areas (LWIAs) is the 

responsibility of the states, so that issue is not discussed here. Thus, the primary issue addressed 

in this report is how ETA should establish efficiency measures and standards for states. 

There are at least three basic approaches available to ETA with regard to setting 

performance standards for the recommended efficiency measures (i.e., cost per entered 

employment and cost divided by post-program earnings): (1) set no standards, (2) set standards 

without adjustments; or (3) set standards with adjustments. These options are discussed below. 

 

71 For the specific performance measures used and further discussion of the WIA and JTPA performance 

measurement system, see Barnow and Smith (2004). 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 136 

 Alternative #1: Set No Standard. Setting no standard is a strategy that essentially 

means that the performance measures are for primarily informational/monitoring purposes -- and 

no incentive payments are awarded or sanctions imposed as a result of how well or poorly an 

operating unit performs. A similar strategy, which we include under this broad heading, is to 

establish standards that have no rewards or sanctions. Such a strategy would make states 

conscious of their efficiency but provide less incentive to meet or exceed standards because there 

is no gain or loss from exceeding or falling short of the standard. 

If no standards for performance are promulgated, the federal government, states, and 

local grantees/operators, may use the measures to track how programs are achieving goals over 

time and across local operating units. The federal government, states, and local operating units 

may make adjustments in their program operations as appropriate to enhance prospects for future 

performance. For example, to improve performance on a measure such as cost per entered 

employment, a local program operator might change staffing levels, alter client flow through 

services, change the mix of services delivered, or enhance linkages to other service providers. 

Such changes might be undertaken to either reduce costs or improve outcomes. The option of 

adopting no standards might be an appropriate initial strategy when a new system of measures is 

introduced because it gives time to develop a track record of results upon which standards can 

later be set. It also provides time for reporting units (e.g., states, grantees, and local operators) to 

troubleshoot potential glitches in data collection/reporting. Finally, and as discussed in greater 

detail later in this paper, using performance results for informational purposes only (and not 

setting standards or rewarding performance) reduces incentives for distorting or “gaming” 

performance results. If specific standards are set or performance is rewarded or punished based 

on performance standards, states, grantees, or local operating units may institute 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 137 

strategies/procedures that do not promote overall program objectives for purposes of appearing 

to perform statistically better on what they are being measured (e.g., “creaming” the most likely 

to succeed applicants).72 

 Alternative #2: Set Standards Without Adjustments. The second option open to ETA 

with regard to setting standards in relation to efficiency measures is to set standards without 

making adjustments across programs or geographic area. On an efficiency measure such as cost 

per entered employment, the same standard could be established across each program. 

Arguments for this approach are that it is easy for program operators to understand and 

administratively straightforward – without the need for costly analyses of factors potentially 

underlying performance—and that it could be deemed appropriate for all states to meet the same 

standard.73 

Setting the same standard for a given efficiency measure across programs (or applying 

the same standard within a program across states or local jurisdictions) may be appropriate if the 

goals, service delivery approaches, costs, and types of individuals served are similar across 

programs. The pitfall to setting the same standard is that in the case of the 11 ETA programs of 

interest, that while employment and improving earnings is a common emphasis, the populations 

served and mix of services provided are quite varied. For example, the SCSEP program, which 

focuses services on older workers, provides a different mix of services than the WIA Youth 

program (which targets in-school and out-of-school youth). In comparison to the SCSEP or 

WIA Adult and Dislocated Worker programs, the Wagner-Peyser program typically provides a 

less costly mix of services, and participants may be involved in such services for only a matter of 

days or weeks (until they find a job). As will be seen in the analyses section later in this report, 

 

72 See below for more discussion on how instituting measures and standards can lead to gaming. 73See Barnow and Heinrich (2009) for a discussion of arguments for and against adjustments to standards. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 138 

there are very substantial differences in results on measures such as cost per entered employment 

across the 11 ETA programs, which suggest that setting one standard across all programs would 

in most likelihood be undesirable. 

 Alternative #3: Set Standards with Adjustments. The third, and most complicated 

and costly option available, is to set and adjust standards for performance measures across 

programs -- and within programs, across states, grantees, and local program operators. The 

adjustments to performance standards may be based on the following: 

• Past performance. ETA could use prior performance as a starting point and base the 

next year’s standard on what is considered to be reasonable continuous improvement. 

• Relative performance. In some situations there may be a “ceiling effect,” where 

states that are already performing very efficiently cannot be expected to improve as 

much as those not performing as well. For example, a program in the top 10 percent 

of the efficiency distribution may not be able to achieve as great an improvement as 

one in middle of the distribution, so expected performance should take this into 

account. 

• Statistical adjustments. In this approach, regression analysis or a similar approach 

is used to take account of how expected performance varies with factors such as 

participant characteristics, the economic environment, activities and services used, 

and the cost of providing services. This is the approach that was used under JTPA to 

“level the playing field” across grantees facing different environments and to hold 

them harmless from these variations. 

• Consensus agreement. This is the approach currently used for WIA. Under this 

approach, standards for individual units are negotiated between ETA and the states. 

The concepts of fairness and equity have been set forth to argue both for and against the 

use of performance adjustments. 74 The most often cited reason for adjusting standards is to 

“level the playing field,” or to make performance management systems as fair as possible by 

establishing expectations that take into account different demographic, economic, and other 

conditions or circumstances outside of public managers’ control that influence performance. It 

 

74 Pro- and con- arguments presented in this section are based on Burt Barnow and Carolyn Heinrich, “One Size Fits 

All? The Pros and Cons of Performance Standard Adjustments, Johns Hopkins University and University of 

Wisconsin-Madison, Draft (January 23, 2009), forthcoming. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 139 

has also been argued, however, that it is not acceptable to set lower expectations for some 

programs than others, even if they serve more disadvantaged populations or operate in more 

difficult circumstances. For example, do we perpetuate inequities in education if less rigorous 

standards for reading and math performance are established for schools serving poorer children? 

Or if a single standard is set for all, could governments instead direct more resources to those 

programs that face more difficult conditions or disadvantaged populations to help put them on a 

more level playing field? 

Another argument of those advocating performance adjustments is that they better 

approximate the value-added of programs (rather than gross outcome levels or change). For 

policy makers or program managers, having a better understanding of the contributions of 

program activities to performance (net of factors that are not influenced by the production or 

service processes) may contribute to more effective use of the performance information to 

improve program operations and management. The use of adjusted performance measures is 

also more likely to discourage (if not eliminate) “gaming” responses, in which program 

managers attempt to influence measured performance in ways that do not increase impacts (e.g., 

by altering who is served and how). A system that adjusts for population characteristics and 

other such factors will reduce the efficacy of these gaming strategies and the misspent effort and 

resources associated with them. Of course, these benefits may be contingent on program 

managers understanding and having confidence in the adjustment mechanisms. Regressionbased performance adjustment models (discussed below in greater detail) have been criticized for 

having low explanatory power (as measured by R2

) and flawed specifications, suggesting that 

sometimes adjustments may be biased or unreliable.75 

 

75 The argument that a low R2 implies that the statistical model is not useful is in most cases false. A low R2

 means 

that there is a lot of noise in predicting the overall level of the dependent variable, not necessarily that the results are 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 140 

Some scholars have criticized these more (technically) rigorous approaches to 

performance measurement as too positivist or elitist, arguing that they may put performance 

analysis and results out of reach of practitioners and the public (i.e., limiting their transparency 

and usefulness for accountability purposes) (Shulock, 1999). In WIA, the U.S. Department of 

Labor discontinued use of the regression-based performance standards adjustment procedures 

and replaced them with a system of negotiated standards with the goal of promoting “shared 

accountability” (U.S. DOL-ETA, 2001). It was suggested that factors outside managers’ control 

that affected program outcomes (some immeasurable) might be more easily conveyed and 

weighed in negotiations.76 

In other cases, the use of performance standards adjustments may be viewed as 

incompatible with the objective of motivating particular individual or organizational responses to 

performance requirements. For example, some performance measurement system designers 

intentionally set a more ambitious standard—also known as a “stretch target”—which is not 

adjusted in order to motivate laggards to change their ways and aspire to achieve higher 

standards of performance. In the TANF high performance bonus system, states that in the past 

had invested little to help clients achieve self-sufficiency had to work harder to meet 

performance requirements for client work participation, job entry, retention, and earnings gains. 

A related argument for not developing or using performance standard adjustments is to 

promote equity in outcomes, that is, to hold up the same standard for all individuals or 

organizations, despite the greater challenges that may be involved in achieving the minimum 

 unreliable. Indeed, one may obtain statistically significant coefficients for the adjustment factors even with a low 

R2, implying that there are important factors that have a strong effect on predicted performance and should be 

accounted for in measuring performance. Based on similar reasoning, Rubenstein et al. (2003) argue that 

performance standard adjustments should also be attempted even when the number of organizations available for 

comparison is small. 

76 The use of negotiated standards under WIA has been very unpopular. See Social Policy Research Associates 

(2004) and Barnow and King (2005). 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 141 

level of performance for some. The recent education reforms that require states to set standards 

for reading and mathematics proficiency and to ensure that all children achieve these minimum 

levels within a specified timeframe, regardless of their backgrounds, special needs, school 

resources, etc., are an example of this approach. In fact, the U.S. Department of Education 

describes its requirements for “challenging state standards” and student testing as one of the 

“pillars” of NCLB that is intended to strengthen “accountability for results” (Radin, 2006: 62-

63). In addition to requiring states to measure and report “adequate yearly progress” under strict 

timelines and in compliance with federal guidelines, NCLB also established a uniform target for 

schools to have 100% of their students proficient in reading and mathematics within 12 years. 

Some states have responded accordingly by developing their own performance-based funding 

incentive systems, in which school districts, schools, and even principals receive incentive 

payments for schools or students who “demonstrate progress” or exceed performance 

expectations. Texas and California, for example, are funding incentive awards at approximately 

one-half billion dollars per year, and a number of states (including Arizona, Colorado, Florida, 

North Carolina, Ohio, and Tennessee) are using value-added statistical models to measure 

teachers’ contributions to learning and to give teachers credit based on how much better (than 

expected) their students perform on tests compared to peers (House Research Organization, 

Texas House of Representatives, 2004). 

C. APPROACHES TO MAKING STATISTICAL ADJUSTMENTS 

The objective of producing accurate knowledge of program impacts or the value-added of 

government activities through statistical modeling (e.g., regression analysis) has been strongly 

advocated by numerous coalitions organized to promote “evidence-based policy making,” that is, 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 142 

government policies and practices based on or guided by scientifically rigorous evidence of their 

effectiveness.77 If accepted as the principal objective of a performance measurement system, it is 

an imperative (in the absence of an experimental evaluation) to model statistically the 

relationship of government activities (i.e., the technology of public production) to performance 

outcomes, while adjusting for factors that influence outcomes but are not (or should not be) 

controlled by public managers. In effect, by adjusting performance expectations for factors that 

are not controlled in production, the estimates of performance are more likely to accurately (and 

usefully) reflect the contribution (or value-added) of public managers and program activities to 

any changes in performance. 

Although there are a number of alternatives to statistically adjusting performance 

expectations, in light of the limitations associated with informal procedures and the lack of 

transparency with their use in practice, we focus primarily in this interim report on formal 

statistical methods for performance standards adjustments. One such technique involves 

adjusting a common performance standard or target (Ps) to which an individual or organization’s 

measured performance (Pm) is compared. In public employment and training programs, for 

example, federal and state officials may establish target performance goals (e.g., a minimum 

entered employment rate or cost per entered employment) for states/local agencies by using a 

regression model to adjust for client demographic characteristics (X) and other uncontrollable 

factors (Z) that may influence performance (e.g., local area unemployment rates). Typically, 

 

77 For example, there is a Center for Evidence-Based Policy (at Oregon Health and Science University), a 

National Coalition for Evidence-Based Policy, the Cochrane Collaboration and the Evidence for Policy 

and Practice Information and Coordinating Centre (both established in the United Kingdom), evidencebased policy networks, evidence-based journals and journal clubs, and evidence-based policy making 

newsletters and bulletins that review and disseminate current research findings on the effectiveness of 

interventions. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 143 

baseline data and/or data on past performance (P0) and the vectors X and Z of factors influencing 

performance are pooled across units and used to estimate a model such as: 

P0 = α + β1X + β2Z + ε. 

The estimates of β1 and β2 are subsequently used as weights for the influence of these factors in 

adjusting the common standard (Ps) to derive unit (e.g., agency-specific) performance targets for 

a given performance measure. Performance is then judged not by comparing actual performance 

(Pm) across units and/or time, but by comparing the differential between a unit’s target (Psi) and 

its measured performance (Pmi). 78

 There are a number of factors to consider when developing statistical (regression) models 

to adjust for outcomes and costs (which are the key components of efficiency measures, such as 

cost per entered employment). With regard to outcomes (the denominator in efficiency 

measures), among the key factors to control for in such adjustment models are the following: 

• Participant characteristics – Characteristics of individuals enrolling or exiting 

programs can significantly affect how well programs perform. For example, among 

the factors used in the JTPA national model for the adult follow-up employment rate 

(discussed earlier) were gender, age, education, race/ethnicity, and receipt of cash 

assistance. Example of other factors that could be considered for such models are: 

veteran status, employment status at registration, limited English proficiency, basic 

skill deficiencies, homeless status, disability status, and offender status. 

• Economic factors – Local economic conditions have a substantial effect on the 

ability of program participants to secure and retain employment, as well as the wages 

which they are likely to receive. For example, among the factors used in the JTPA 

national model for the adult follow-up employment rate (discussed earlier) were 

unemployment rate, three-year growth in earnings in trade, annual earnings in trade, 

and families with income below poverty.

• Program activities – The types of program activities and intensity of such activities 

can affect how well participants/exiters perform in terms of outcomes such as 

employment and earnings both over the short- and long-term. For example, the 

attainment of a degree (such as an Associates or BA degree) or certification can affect 

 

78 Later in this chapter, we provide a preliminary model and utilize data from the WIA program to analyze various 

factors (e.g., participant characteristics, environmental factors, services received, etc.) that may explain variation 

across states in results on efficiency measures such as cost per entered employment. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 144 

how much an exiter from a training program makes in his/her new job. In a program 

such as WIA or TAA, for example, there are several levels of service that could be 

considered within a model – receipt of core unassisted services, receipt of intensive 

services, or receipt training services. 

• Co-enrollment – With emphasis in recent years being placed on partnering and 

coordination of services, as well as leveraging of resources, to bring the full array of 

services to bear on behalf of program participants, it is important when developing 

models to take into account co-enrollment. For example, WIA participants may be 

enrolled at the same time in more than one WIA program (e.g., the WIA Adult and 

Youth programs or the Adult and Dislocated Worker programs, or even all three of 

these programs at once or sequentially). Additional, WIA participants may be 

referred to (and enrolled in) a host of other programs to both stretch resources and 

maximize chances for positive outcomes, including Wagner-Peyser, TAA, the Indian 

and Native American Program, and others. Policies with regard to co-enrollment and 

the extent to which co-enrollment is tracked in various information systems can vary 

substantially across states and, within states, across local workforce areas. To the 

extent possible – and it can be a problem if information systems are not linked across 

programs – it is important to take into account patterns of co-enrollment both because 

they may affect outcomes and alter costs. 

 With regard to the cost side of the equation (i.e., the numerator in an efficiency measure), 

it is important to consider the following that can drive up or down costs of providing services: 

• Shared/joint cost across programs -- Particularly within a One-Stop system and 

programs that emphasize strong linkages to other programs and co-enrollment to 

leverage resources/services, there can be significant sharing of costs across programs 

that makes it difficult to fully account for the costs of serving a particular individual. 

For example, costs for the WIA program can be significantly affected by whether 

other programs (such as Wagner-Peyser, rehabilitation services, welfare programs) 

pay none, some portion, or all of the costs of One-Stop operations. Tuition costs in 

TAA or WIA can be substantially stretched if training participants use Pell Grants; 

support service costs such as transportation or purchase of work clothing/tools may be 

reimbursed by welfare programs or other programs. This is (and will be) a 

challenging dimension of designing adjustment models for programs.

• Cost-of-living differences (across states/local areas) – Cost of staff salaries, tuition 

for participants, office space, and a host of other items can substantially vary across 

states and local jurisdictions. With regard to efficiency measures, such cost 

differentials can have serious impacts on a measure such as cost per entered 

employment; for example, it may be much more costly to provide services in urban 

areas such as the New York metropolitan area, compared to small cities, towns, and 

rural areas. Factors such as average weekly wage rate for retail trade or average 

weekly wage rate for wholesale trade might be incorporated into adjustment models 

to reflect such differentials across states or localities.
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 145 

• Inflation (over time) – Making comparisons of efficiency measure results across 

program or fiscal years necessitates use of inflation adjusters (such as the consumer 

price index (CPI) or producer price index (PPI)) or possible change in average wage 

rates.

The next section of this report explores the potential for taking into consideration some of these 

factors in building a statistical adjustment model and present preliminary results of adjustment 

models applied to the WIA Adult, Dislocated Worker, and Youth programs. 

D. ADJUSTING EFFICIENCY MEASURES USING STATISTICAL MODELS: 

PRELIMINARY MODELS AND RESULTS FOR THE WIA ADULT, 

DISLOCATED WORKER, AND YOUTH PROGRAMS 

This section explores the potential for using regression analysis to statistically adjust 

standards for the recommended efficiency measures. Overall, while it is technically possible to 

build such models, when applied to the WIA program the various models did not generally yield 

statistically significant coefficients that could explain the wide variation in efficiency measure 

results across states. This analysis suggests that while adjustment models have been useful in the 

past for explaining variation in outcomes for exiters from employment and training programs 

(such as those served under JTPA), that for various reasons explained below, such models (at 

least at the state level) do not appear promising for explaining variation in efficiency measure 

results across states for the 11 ETA programs. 

Background. A number of regression models were constructed to estimate the extent to 

which variation in efficiency measure results might be explained by a set of characteristics of the 

population served, environmental factors, and various levels of program services received by 

participants. This exploratory analysis focuses on exiters from the WIA Adult, Dislocated 

Worker, and Youth programs with the goals of (1) determining if such models can be built, (2) 

defining the types of variables that can and should be included in such models, and (3) initially 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 146 

testing whether such models generate statistically significant and reasonable coefficients that are 

helpful in explaining variation in efficiency measure results across states. We have focused on 

the three WIA programs because (1) its predecessor program (JTPA) had a considerable track 

record using regression models to adjusts performance results for outcome measures (across 

local workforce areas) and (2) the WIA program collects detailed data on individual participant 

characteristics, service utilization, and outcomes (i.e., exiters of the program) that can be readily 

matched with state-level expenditure data. By focusing on the WIA program, this exploratory 

analysis provides a case study of the potential feasibility and effectiveness of such adjustment 

modeling, including identifying potential pitfalls in applying such adjustment models to 

employment and training programs. If such models are feasible and useful for adjusting 

performance results for the WIA program, they might also be useful for other programs (with 

perhaps a somewhat different set of explanatory variables), such as the Wagner-Peyser and TAA 

Programs. 

Several data sources were used for this exploratory analysis: (1) the Workforce 

Investment Act Standardized Record Data (WIASRD) covering exiters for a three-year period 

(PY 2004-06) was used to generate data on participant characteristics (e.g., gender, age at 

registration, race/ethnicity, etc.), services received (e.g., whether intensive or training services 

were received), and outcomes (such as whether an exiter entered and retained employment, postprogram earnings);79 (2) Bureau of Labor Statistics (BLS) and Census Bureau data were used to 

generate several key environmental variables at the state level (e.g., unemployment rate, average 

wage rates, etc.); and (3) WIA Program expenditure data (provided by OFAM) was aggregated 

to the state level for each of the three program years of interest. Because cost data for the WIA 

 

79 For documentation of this dataset, see Social Policy Research Associates, PY 2005 WIASRD Data File Public Use 

Including Data Quality Revisions, Record Layout, March 23, 2007. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 147 

program is only available at the state level (i.e., WIA program expenditures by local workforce 

investment areas are not reported to USDOL), participant, services receipt, and outcome data 

were aggregated to the state level and the models were estimated at the state level.80 81

In specifying the independent and dependent variables to be included in the various 

regression models to be tested, we first considered and built on the main variables that were used 

in the original JTPA adjustment models (see the earlier discussion in Section A of this chapter 

and Exhibit 7-1 for the specific independent and dependent variables included in the JTPA 

model). We broadened our modeling effort to include some additional independent variables 

that were not included in the JTPA model, which we felt could be reasonably expected to explain 

variation in results on the efficiency (dependent) variables. We discuss the specific dependent 

and independent variables included in our models below (and they are shown in the tables that 

show our regression results). It is important to note that shortcomings in several of the variables 

found within the WIASRD precluded their use in these models. For instance, for the WIA Adult 

and Dislocated Worker Programs, because of substantial amounts of missing data from 

individual exiter records, information on unemployment compensation receipt, low-income 

status, homeless status, and offender status, among others, could not be used. Similarly, for WIA 

Youth, foster care and low-income status could not be used. 

With regard to understanding potential effects of co-enrollment on efficiency measure 

results across states, there were also clear limitations. Whether a WIA participant was coenrolled in other programs was not consistently entered on exiter records by local workforce 

 

80 Expenditure data at the Local Workforce Investment Area (LWIA) level are not submitted to the U.S. Department 

of Labor by local areas or the states; hence, at this time it is not possible to run the various cost models at the local 

level which would allow for many more observations on an annual basis (e.g., about 600 observations if the analysis 

was at the LWIA level versus 51 observations at the state level, with the inclusion of the District of Columbia). As 

is discussed later in this section, the small number of observations is a serious limitation to producing models with 

explanatory power. 

81 This is unlike the regression models used under JTPA, which were used to adjust performance at the local 

workforce area – referred at the time as the Service Delivery Area (SDA). 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 148 

areas. Hence, co-enrollment in other programs could not be included in the models (though it 

was possible to include co-enrollment of WIA exiters in other WIA-funded programs and the 

Wagner-Peyser program as variables in the models). 

Models were tested using one year of data (PY 2006), as well as for two and three years 

(PY 2004-06). Given that any statistical model for a single year, run at the state level, would 

have a relatively small sample size (51 observations) – and therefore, be less likely to generate 

statistically significant coefficients for independent variables of interest than a sample with many 

more observations – we report on models with WIASRD data pooled for three years in the 

exhibits.82 Using three years of pooled data boosted the number of observations from 51 

observations to 153 observations – though as will soon be shown, this pooling across three years 

did not result in the models having much in the way of statistically significant coefficients. 

Overview of the Models Tested. Two sets of models were constructed for all dependent 

variables: (1) Pooled Ordinary Least Squares (OLS) and (2) Fixed Effects. Each of these 

models is briefly discussed below, including their relative advantages and their limitations. The 

two models were used because each approach has limitations. The Pooled OLS Model is 

generally defined as: 

where: aggregated state (i) level observations are pooled across the three program years (t), Yit = 

Total expendituresit/outcomeit, X is a vector of individual characteristics of the exiter population, 

 

82It should be noted that all exiters for the WIA Adult and Dislocated Worker programs who exited in the fourth 

quarter of program year 2006, are excluded from the Employment Retention and Post Program Earnings common 

measures and the efficiency measures based on them. This is because at the time of the release of the 2007 WIASRD 

data, there had not been sufficient time to allow for a third quarter post-program follow up. As a result, if we assume 

steady state programs, then one-twelfth of the customer observations are missing. The only viable solution to this 

problem other than waiting until the data become available is to limit the analysis of these two efficiency measures 

to PY2004 and PY2005. These models were estimated, and the results are available upon request. However, very 

few explanatory measures had statistically significant regression coefficients, which is not surprising given the 

limited sample size of only 102. This is not an issue for the Entered Employment Rate and the efficiency measure 

based on its underlying outcomes. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 149 

L is a vector of local area economic characteristics, P is a vector of programmatic/service level 

characteristics, and C is a vector of co-enrollment characteristics. The OLS regression model is 

advantageous in that it is widely used, well understood, and accepted in a performance 

management context for workforce investment programs. Moreover, it provides easily 

interpretable coefficients describing the relationship of a single characteristic to the efficiency 

measures, holding the other characteristics constant. The pooled nature of the model also has the 

benefit of estimating coefficients from both differences between states and within states over 

time. Moreover, using multiple years should reduce the likelihood of having adjustment factors 

vary a great deal from year to year, as sometimes occurred when annual models were estimated 

to adjust performance standards under JTPA. 

There are, however, some serious potential drawbacks to using a pooled OLS regression 

model with geographically aggregated observations over time. The theory that allows for the use 

of pooled OLS over several years requires that the samples for each year be independent of one 

another. Generally, this can only be achieved through drawing a random sample from a large 

population in each time period. Specifically, for the coefficient estimates to be unbiased and 

standard errors to be valid, not only must the unobserved factors that affect the outcome 

measures be uncorrelated with the explanatory variables within a single time period, as would be 

required for avoiding classic omitted variable bias for OLS in a single program year, they must 

also be completely independent across time periods.83 

 

83 In other words, if there is an unobserved constant characteristic of a state that affects the outcome measure (e.g. a 

state specific “fixed effect”) that is omitted from the model, not only will the coefficients be biased if that 

characteristic is correlated with one or more of the explanatory variables, tests of significance and our inference 

assumptions will also be invalid regardless of whether or not the unobserved state “fixed effects” bias the 

coefficients through correlation with the explanatory variables. Simply the fact that there are unobserved constant 

characteristics of states/local populations/WIA programs that impact outcomes (and are omitted from the models), 

requires viewing pooled OLS results in this setting with some skepticism. Moreover, given limited data availability 

and often very different programs within states, assuming that there are not unobserved state “fixed effects” that 

impact outcomes may be a very strong assumption to make. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 150 

Overcoming an omitted variable bias problem is possible. While not ideal, if we assume 

that the relationship of the omitted variable is consistent across all states, the use of the 

coefficient for the adjustment of standards would simply adjust for both the observed variable 

and the unobserved variable. However, drawing valid inferences from pooled cross sections 

does require eliminating the fixed state effects, should they exist. This is what the Fixed Effects 

Model – the second model tested -- is designed to accomplish. The Fixed Effects Model is 

generally defined as: 

This model, also called the “within transformation,” is computationally identical to pooled OLS, 

except that it utilizes time-demeaned observations, instead of the original 

observations. Simply put, time-demeaned observations result from subtracting the cross-year 

averages from individual year observations. 

Without going into detail as to why this is the case,84 what is important to note is that this 

transformation drops out any unobserved time constant characteristics that affect the outcomes 

and may or may not be correlated with the other explanatory variables, including the state “fixed 

effects.”85 It should also be noted that, while the reported coefficients are interpreted in exactly 

the same way as those reported for the pooled OLS model, the Fixed Effects model is 

specifically looking at the extent to which the time variation in the dependent variable can be 

 

84 See, for instance, Wooldridge, Introductory Econometrics: A Modern Approach, 3rd edition (2006) chapter 14. 85 Also note that the model intercept and any explanatory variables that are constant over all time periods, or have a 

constant rate of change, are also dropped. However, because the data was aggregated at the state level there were no 

time constant variable that would be dropped by the transformation, as would be the case for any participant 

characteristics, for instance, in an individual level panel. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 151 

explained by changes in the explanatory variables within states over time (hence, the term 

“within transformation”).86 

The Fixed Effects Model is not without its drawbacks, however. While the transformation 

does remove the unobserved time constant “fixed effects,” it does not remove unobserved timevarying effects that relate to issues such as unobserved programmatic or co-enrollment changes. 

As a result, to the extent that we are unable to account for the non steady-state nature of these 

programs during the period studied though the explanatory variables, especially in relation to coenrollment and external financial support, there is a potential for bias. Another significant issue 

is that the Fixed Effects Model requires large changes in the variables over time to draw proper 

inferences. If an explanatory variable does not change over time (a) it will be completely 

dropped by the “demeaning” of the data, and (b) if it only varies slightly over time, statistically 

valid effects cannot be estimated with high likelihood. Also, the Fixed Effects Model is far more 

susceptible to bias in the estimated coefficients due to classic measurement error in the 

underlying data than OLS estimates are. There is reason to believe that at least some of the data 

used for this study suffers from substantial measurement error.87

Dependent Variables (Efficiency Measures) and Independent Variables Included in 

the Adjustment Models Tested. The dependent variables used in the models are efficiency 

measures, including the two measures recommended in our earlier report for implementation by 

the 11 ETA programs – cost per entered employment and cost divided by post-program earnings. 

 

86 For readers more familiar with the somewhat (historically) better known “least squares dummy variable (LSDV) 

model” where dummy variables are added for all but one state, it should be noted that this model produces identical 

results to LSDV and holds the same benefits of dropping out the individual state “fixed effects.” 

87 Apart from the many known measurement problems, either addressed or not, that are listed in the record layout 

documents that accompany each version of the public release WIASRD data, careful examination of the dataset 

reveals some unlikely observations and/or missing data where it should not be missing. Moreover, it appears as 

though certain variables are inconsistently reported across states (e.g., being served by a Wagner-Peyser funding 

stream appears to be one) and there is a tendency for individuals receiving higher levels of service to have better 

reporting, even beyond the fact that certain questions are only asked for exiters with higher levels of service. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 152 

In recommending these efficiency measures for implementation for WIA (and the other ETA 

programs), an effort was made to align the constructed efficiency measures with the already 

established Common Measures because states are already tracking the Common Measure 

outcomes. For the WIA Adult and Dislocated Worker Programs, the main dependent variables 

included in the models are defined as follows: 

1) Cost per Entered Employment equals [Total State Expenditures] / [the number of 

individuals counted as “entering employment” by the common performance 

measure’s definition of entering employment]; 

2) Cost per Employment Retention equals [Total State Expenditures] / [the number of 

individuals counted as “retaining employment” by the common performance 

measure’s definition of employment retention]; and 

3) Cost divided by Total Post-Program Earnings = [Total State Expenditures] / [the 

aggregate 2nd Quarter + 3rd Quarter post exit earnings that are used to calculate the 

Adult and Dislocated Worker Average Earnings Common Measure].88 

The WIA Youth Program required construction of a different dependent variable because the 

three dependent variables calculated for the Adult and Dislocated Worker programs are only 

applicable to older youth (and not younger youth served by the program). Cost data for the WIA 

Youth Program is only available at the state level for both older and younger youth (combined). 

Thus, to use the available cost data, it was necessary to come up with a composite efficiency 

measure that would be applicable to both older and younger youth. The efficiency measure 

recommended for the WIA Youth Program – and used as the dependent variable in the 

regression model -- is Cost per Positive Outcome, where the numerator is total expenditures 

across the state for the WIA Youth Program, and the denominator is the number of exiters that 

are counted as either “yes” for Placement in Employment or Education (WIASRD Item 966) or 

“yes” for Attainment of Degree or Certificate (WIASRD Item 967). 

 

88 For documentation of how each of the variables is defined see Social Policy Research Associates, PY 2005 

WIASRD Data File Public Use Including Data Quality Revisions, Record Layout, March 23, 2007. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 153 

 The independent variables used in the models were the same for the WIA Adult and 

Dislocated Worker programs, but differed (by necessity) for the WIA Youth program. As noted 

earlier, we reviewed the earlier JTPA adjustment model in determining which variables to 

include in the models. We tried to be inclusive in terms of adding variables not included in the 

JTPA model that we hypothesized could explain variation in efficiency measure results. As 

noted earlier, we were unable to include some potential variables in the WIASRD data set 

because of missing data. With the exception of the economic factors and program expenditure 

data included in the model, we were limited to the variables reported in the WIASRD. The 

independent variables included in the regression models tested for the WIA Adult and Dislocated 

Worker programs were as follows: 

• Age -- percent under 22, 22-29, 30-44, 45-54, and 55 year of age or older 

• Gender -- percent female 

• Race/Ethnicity -- percent white, black, Hispanic, other or multiple races 

• Veteran status (percent veteran) 

• Percent working at registration 

• Education -- percent not a high school graduate 

• Percent with limited English proficiency 

• Percent with Temporary Assistance for Needy Families (TANF) receipt within 6 months 

• Percent enrolled in multiple WIA Programs 

• Percent funded by Wagner-Peyser 

• Type of WIA services received – percent receiving training, percent receiving intensive 

services, percent receiving core services 

• Average weekly wage rate for retail trade workers (for state) 

• Average weekly wage for production workers in manufacturing (for state) 

• Percent below poverty (for state) 

• Average unemployment rate, non-seasonally adjusted (for state) 

The independent variables included in the regression models tested for the WIA Youth programs 

were as follows: 

• Percent younger youth 

• Gender -- percent female 

• Race/Ethnicity -- percent white, black, Hispanic, other or multiple races 

• Percent pregnant or parenting 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 154 

• Percent working at registration 

• Education -- percent in-school, dropout, not a high school graduate 

• Percent with Temporary Assistance for Needy Families (TANF) receipt within 6 months 

• Percent enrolled in multiple WIA Programs 

• Percent funded by Wagner-Peyser 

• Percent in need of additional assistance 

• Percent homeless 

• Percent offender 

• Average weekly wage rate for retail trade workers (for state) 

• Average weekly wage for production workers in manufacturing (for state) 

• Percent below poverty (for state) 

• Average unemployment rate, non-seasonally adjusted (for state). 

Regression Analysis Results. Exhibits 7-2 through 7-5 show the coefficients and 

significance levels for the Ordinary Lease Squares (OLS) and Fixed Effects Models tested for the 

WIA Adult, Dislocated Worker, and Youth Programs.89 

• Exhibit 7-2 shows results for the OLS and the Fixed Effect Models for cost per 

entered employment for the WIA Adult and Dislocated Worker Programs. 

• Exhibit 7-3 shows results for the OLS and the Fixed Effect Models for cost per 

reemployment retention for the WIA Adult and Dislocated Worker Programs. 

• Exhibit 7-4 shows results for the OLS and the Fixed Effect Models for cost divided 

by total program earnings for the WIA Adult and Dislocated Worker Programs. 

• Exhibit 7-5 shows results for the OLS and the Fixed Effect Models for cost per 

positive outcome (employment or educational) for WIA Youth. 

Overall, examination of the regression results reported on each of these four exhibits reveals very 

few coefficients that are statistically significant, indicating that knowledge of the values of the 

independent variables does not help much in predicting the efficiency measure results. We 

would expect to see some significant coefficients simply by random chance, given the number of 

 

89 An earlier paper prepared under this study contained four additional regressions conducted for alternative 

efficiency measures. See Burt Barnow, Jonathon Pollak, and John Trutko, Implementing Efficiency Measures for 

Employment and Training Programs: Interim Report #3 – Potential Methodologies and Factors for Establishing 

and Adjusting Efficiency Measure Standards for ETA Programs, prepared for the U.S. Department of Labor, 

Employment and Training Administration, prepared by Capital Research Corporation and Johns Hopkins 

University’s Institute for Policy Studies, June 2009. 
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 155 

EXHIBIT 7-2: 

 

OLS * Fixed Effects ** OLS * Fixed Effects **

Average Exiters Per WIB -0.2410507 (0.332) -0.3926693 (0.344) -2.11783 (0.025) -0.5714805 (0.613)

Percent Training 3027.881 (0.342) -381.4137 (0.920) 11642.46 (0.003) 22847.42 (0.000)

Percent Intensive 9296.053 (0.003) 7647.355 (0.062) -7361.732 (0.337) -475.3166 (0.938)

Percent Core (Reference Group) ////

Percent Enrolled in Multiple WIA Programs -9946.479 (0.107) -13881.74 (0.100) -531.4047 (0.895) -8268.414 (0.070)

Percent Funded by Wagner Peyser Funding 

Sourse -157.344 (0.836) -541.7541 (0.741) -1002 (0.391) -2443.783 (0.299)

Percent Age 0 to 21 -6064.433 (0.659) -9991.389 (0.586) -77746.79 (0.348) -13120.87 (0.820)

Percent Age 22 to 29 (Reference Group) ////

Percent Age 30 to 44 -11113.63 (0.503) -16700.98 (0.270) 10668.43 (0.692) 63820.09 (0.016)

Percent Age 45 to 54 -5590.327 (0.695) -9038.783 (0.622) -51822.36 (0.082) 28791.33 (0.206)

Percent Age 55 Plus 11522.61 (0.505) 32690.41 (0.137) 40533.22 (0.121) 58642.68 (0.060)

Percent White (Reference Group) ////

Percent Black 303.6727 (0.861) 6219.461 (0.477) 6768.007 (0.129) -21733.44 (0.261)

Percent Hispanic -665.5727 (0.843) 2886.601 (0.815) 4542.773 (0.450) -23767.59 (0.338)

Percent Other or Multiple Race 3904.474 (0.215) -21944.76 (0.201) 10802.28 (0.012) -37719.23 (0.102)

Percent Female -3328.627 (0.457) 2942.052 (0.706) -3983.639 (0.648) -9390.187 (0.427)

Percent Veteran 27150.06 (0.122) -16716.49 (0.529) 5000.08 (0.809) 3944.235 (0.863)

Percent Working at Registration 7401.894 (0.058) 13777.68 (0.017) 23513.48 (0.004) 6751.104 (0.594)

Percent Non High School Grad -10026.62 (0.122) -12819.15 (0.160) -34688.17 (0.003) -14352.28 (0.368)

Percent Limited English Proficiency 2358.936 (0.695) 5320.54 (0.678) 13552.74 (0.097) 9434.484 (0.426)

Percent TANF Receipt within 6 months 9222.582 (0.044) 6023.326 (0.614) / /

Average Unemployment Rate (nonseasonally adjusted) 135.4183 (0.677) -50.55264 (0.947) 980.2279 (0.021) -2943.66 (0.019)

Percent Below Poverty Line -66.97503 (0.622) 466.0286 (0.334) 591.3577 (0.023) 1117.926 (0.130)

Average Weekly Retail Wage Rate -1.068796 (0.908) -25.12525 (0.432) 23.24225 (0.186) 6.483937 (0.897)

Average Wkly Wage Production Workers in 

Manufacturing 5.790297 (0.095) 5.948339 (0.566) 4.050348 (0.474) 21.55965 (0.163)

Constant 1350.745 (0.893) 10051.07 (0.607) -5506.852 (0.821) -46478.33 (0.139)

Observations 153 153 153 153

R-squared / Within State R-squared 0.4973 0.4456 0.4164 0.4121

* P-values from Huber/White Variance Estimator (Robust)

** Standard Fixed Effects P-Value

Source: Combined 2005, 2006, and 2007 WIASRD Public Use Data Files, Authors' Calculations

WIA Adult WIA Dislocated Worker

Exhibit C-5: Estimated Adjustment Coefficients for Cost per Entered Employment from Common Measure
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 156 

EXHIBIT 7-3: 

OLS * Fixed Effects ** OLS * Fixed Effects **

Average Exiters Per WIB -0.1361559 (0.640) -0.3924325 (0.418) -2.627389 (0.017) -1.06479 (0.528)

Percent Training 726.866 (0.861) -6540.853 (0.143) 13110.33 (0.005) 29917 (0.001)

Percent Intensive 11822.02 (0.003) 16316.52 (0.001) -11474.46 (0.236) -4800.615 (0.600)

Percent Core (Reference Group) ////

Percent Enrolled in Multiple WIA Programs -10204.53 (0.085) -17522.89 (0.076) -1809.781 (0.722) -8964.194 (0.186)

Percent Funded by Wagner Peyser Funding 

Sourse 126.9771 (0.881) -404.4047 (0.833) -1353.204 (0.351) -1273.917 (0.716)

Percent Age 0 to 21 -6437.677 (0.648) -20123.53 (0.349) -73058.75 (0.547) -54838.89 (0.524)

Percent Age 22 to 29 (Reference Group) ////

Percent Age 30 to 44 -20241.81 (0.211) -34263.24 (0.055) 10469.37 (0.733) 90690.44 (0.022)

Percent Age 45 to 54 10890.43 (0.464) -6866.509 (0.749) -46984.53 (0.189) 69422.57 (0.042)

Percent Age 55 Plus -6131.53 (0.743) 14305.58 (0.575) 32796.01 (0.317) 87575.6 (0.060)

Percent White (Reference Group) ////

Percent Black 652.1862 (0.721) 1518.185 (0.882) 7327.739 (0.185) -26529.47 (0.357)

Percent Hispanic -3778.893 (0.298) 4487.134 (0.755) 3006.308 (0.712) -34668.96 (0.348)

Percent Other or Multiple Race 2940.826 (0.382) -9840.811 (0.622) 11695.66 (0.010) -22355.86 (0.513)

Percent Female -5764.002 (0.204) -2854.158 (0.754) -2073.045 (0.832) -21353.73 (0.227)

Percent Veteran 21068.76 (0.243) -10154.43 (0.744) 2833.824 (0.916) -3321.681 (0.922)

Percent Working at Registration -4701.025 (0.216) -1555.156 (0.814) 11564.89 (0.233) -11383.2 (0.547)

Percent Non High School Grad -12086.02 (0.119) -8430.672 (0.427) -36678.97 (0.011) -24406.63 (0.305)

Percent Limited English Proficiency 4520.876 (0.534) 8382.215 (0.576) 20693.93 (0.079) 16165.4 (0.360)

Percent TANF Receipt within 6 months 11204.54 (0.021) -2780.214 (0.842) / /

Average Unemployment Rate (nonseasonally adjusted) -433.8928 (0.169) -1243.517 (0.167) 253.1024 (0.626) -4558.503 (0.015)

Percent Below Poverty Line 79.07242 (0.573) -173.9185 (0.757) 950.3115 (0.005) 809.2602 (0.459)

Average Weekly Retail Wage Rate 10.09833 (0.315) 36.53447 (0.329) 39.25501 (0.091) 101.1983 (0.178)

Average Wkly Wage Production Workers in 

Manufacturing 8.453969 (0.028) 12.50665 (0.303) 9.879961 (0.197) 39.40166 (0.088)

Constant 808.6201 (0.940) 2319.972 (0.919) -13464.71 (0.637) -105981 (0.025)

Observations 153 153 153 153

R-squared / Within State R-squared 0.4079 0.4484 0.348 0.4492

* P-values from Huber/White Variance Estimator (Robust)

** Standard Fixed Effects P-Value

Source: Combined 2005, 2006, and 2007 WIASRD Public Use Data Files, Authors' Calculations

Exhibit C-6: Estimated Adjustment Coefficients for Cost per Employment Retention from Common Measure

WIA Adult WIA Dislocated Worker
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 157 

EXHIBIT 7-4: 

OLS * Fixed Effects ** OLS * Fixed Effects **

Average Exiters Per WIB -0.0000279 (0.281) -0.000021 (0.651) -0.0002113 (0.010) -0.000071 (0.544)

Percent Training 0.1378646 (0.700) -0.7119615 (0.098) 1.054287 (0.002) 2.107511 (0.001)

Percent Intensive 0.8753246 (0.013) 1.700843 (0.000) -0.8399887 (0.202) -0.1892001 (0.765)

Percent Core (Reference Group) ////

Percent Enrolled in Multiple WIA Programs -1.320245 (0.015) -2.12791 (0.026) -0.291317 (0.409) -0.6976112 (0.138)

Percent Funded by Wagner Peyser Funding 

Sourse 0.0527718 (0.502) -0.0875856 (0.635) -0.0947984 (0.371) -0.0681167 (0.779)

Percent Age 0 to 21 -0.1965093 (0.886) -1.643755 (0.426) -9.406894 (0.274) -5.710135 (0.339)

Percent Age 22 to 29 (Reference Group) ////

Percent Age 30 to 44 -1.334989 (0.379) -3.122848 (0.068) -0.8740823 (0.693) 5.046404 (0.064)

Percent Age 45 to 54 0.9373433 (0.508) -0.7798212 (0.705) -5.093376 (0.050) 3.276781 (0.164)

Percent Age 55 Plus -0.0332746 (0.985) 2.184205 (0.374) 1.852173 (0.420) 5.665247 (0.079)

Percent White (Reference Group) ////

Percent Black 0.0830954 (0.613) 0.0396065 (0.968) 0.5338182 (0.176) -1.123689 (0.573)

Percent Hispanic -0.0684893 (0.842) 0.3334352 (0.810) 0.3247373 (0.602) -2.243995 (0.381)

Percent Other or Multiple Race 0.0981963 (0.738) -0.8774224 (0.648) 0.9283059 (0.006) -0.8762359 (0.711)

Percent Female -0.0861321 (0.829) 0.2950104 (0.736) 0.2778206 (0.690) -1.475122 (0.229)

Percent Veteran 1.153726 (0.489) 0.6478198 (0.828) 0.5817059 (0.764) -0.2343555 (0.921)

Percent Working at Registration -0.876296 (0.009) -0.4488301 (0.481) 0.4027768 (0.558) -0.9289011 (0.478)

Percent Non High School Grad -0.7719485 (0.260) -0.5423891 (0.595) -2.353014 (0.020) -1.504677 (0.361)

Percent Limited English Proficiency 0.4322359 (0.523) 0.7408043 (0.607) 1.408453 (0.079) 1.184009 (0.334)

Percent TANF Receipt within 6 months 1.119706 (0.010) 0.1398994 (0.917) / /

Average Unemployment Rate (nonseasonally adjusted) -0.0231249 (0.404) -0.1087115 (0.209) 0.0407248 (0.292) -0.3198244 (0.014)

Percent Below Poverty Line 0.0021129 (0.869) -0.0163412 (0.762) 0.0721937 (0.003) 0.0966868 (0.204)

Average Weekly Retail Wage Rate -0.0008236 (0.386) 0.0020394 (0.570) 0.0011286 (0.495) 0.0049983 (0.337)

Average Wkly Wage Production Workers in 

Manufacturing 0.0010748 (0.005) 0.0007152 (0.539) 0.000765 (0.180) 0.0022067 (0.167)

Constant 0.3998158 (0.704) 0.5906916 (0.788) 0.6859695 (0.730) -5.790276 (0.075)

Observations 153 153 153 153

R-squared / Within State R-squared 0.4329 0.3918 0.3711 0.4293

* P-values from Huber/White Variance Estimator (Robust)

** Standard Fixed Effects P-Value

Source: Combined 2005, 2006, and 2007 WIASRD Public Use Data Files, Authors' Calculations

Exhibit C-7: Estimated Adjustment Coefficients for Cost per Total Post Program Earnings from Common Measure

WIA Adult WIA Dislocated Worker
Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 158 

EXHIBIT 7-5: 

OLS * Fixed Effects **

Average Exiters Per WIB -8.465314 (0.004) -9.212684 (0.025)

Percent Younger Youth 744.1997 (0.961) -32380.28 (0.091)

Percent Enrolled in Multiple WIA Programs 5777.041 (0.289) -1253.009 (0.870)

Percent Funded by Wagner Peyser Funding Sourse -2025.758 (0.158) 417.9339 (0.892)

Percent White (Reference Group) / /

Percent Black 925.854 (0.725) -16665.42 (0.242)

Percent Hispanic -354.7738 (0.946) 6725.881 (0.765)

Percent Other or Multiple Race -2426.525 (0.519) 12553.79 (0.708)

Percent Female 975.4109 (0.952) 16277.3 (0.366)

Percent Pregnant or Parenting -12167.97 (0.370) -26612.15 (0.349)

Percent Working at Registration -3308.572 (0.801) -12726.73 (0.447)

Percent in School -34871.56 (0.123) 43267.99 (0.395)

Percent Dropout -34183.96 (0.169) 22582.18 (0.662)

Percent Non High School Grad 19353.36 (0.437) -3584.162 (0.938)

Percent Limited English Proficiency -24987.5 (0.495) -35004.12 (0.468)

Percent that Need Additional Assistance 4677.243 (0.025) 2942.471 (0.630)

Percent TANF Receipt within 6 months 39341.05 (0.012) -10997.05 (0.652)

Percent Homeless -17133.56 (0.257) 12218.57 (0.750)

Percent Offender 10702.54 (0.282) 11168.99 (0.583)

Average Unemployment Rate (non-seasonally adjusted) -490.9724 (0.224) -184.0144 (0.889)

Percent Below Poverty Line 334.3872 (0.254) 187.5362 (0.828)

Average Weekly Retail Wage Rate 21.1578 (0.231) 3.805398 (0.948)

Average Wkly Wage Production Workers in Manufacturing -1.193129 (0.842) 17.02219 (0.368)

Constant 12456.77 (0.481) -4664.059 (0.883)

Observations 153 153

R-squared / Within State R-squared 0.3751 0.2166

* P-values from Huber/White Variance Estimator (Robust)

** Standard Fixed Effects P-Value

Source: Combined 2005, 2006, and 2007 WIASRD Public Use Data Files, Authors' Calculations

Exhibit C-8: Estimated Adjustment Coefficients for Cost per 

Youth Positive Outcome
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 159 

variables and models tested.90 Because of the small sample size and several other limitations 

discussed below, extreme caution is required in interpreting the results of each model. In short, 

the results of the models tested are not useful from the standpoint of identifying variables that 

statistically are associated with higher or lower efficiency measure results across states. These 

results suggest that applying a regression adjustment model -- at least one applied at the state 

level to the WIA program – is not likely to be very useful for setting or adjusting standards for 

the recommended efficiency measures. It is not known whether the models would be more 

useful in explaining variation if they were applied at the local workforce investment level (where 

many more observations would be available) – however, as discussed earlier this would not be 

possible without expenditure data being reported from the local workforce area to U.S. DOL. 

These regression results also suggest that statistical adjustment models applied at the state-level 

for the other programs would similarly be unlikely to explain much in the way of variation in 

efficiency measure results across states – and therefore, probably not be that useful for setting or 

adjusting standards.

Additional Discussion of Key Factors Limiting Regression Results. Among the 

critical factors that constrain the models that were tested for the WIA Program are the following: 

small sample size, “noisy” cost data, the non-steady-state nature of the programs, and exclusions 

of exiters from Common Measure outcomes. Apart from recommending analyzing data for more 

years, or collecting cost data for, and running adjustment models for, smaller areas such as at the 

local workforce investment areas (which would provide many more observations within a oneyear period), there is not much more to be said about the sample size issue. The other issues 

merit some further discussion. 

 

90 Some variable, such as service mix and co-enrollment are more consistent across models and are sometimes of the 

expected sign. However, most are un-interpretable, and others, such as percent not a high school graduate, display a 

sign counter to expectation. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 160 

“Noisy” Cost Data. There are two main issues in this category that cause problems for 

the efficiency measure regression analysis: programmatic linkages and co-enrollment. The first 

issue arises from multiple programs collaborating and, in some instances, being housed under 

one roof, as in the case of a One-Stop Career Center. When programs are linked and overhead is 

shared, it can be very hard to say what funds went to support which programs, and thus, to 

support which participants. As a result of these linkages and collaboration, the amounts spent to 

support WIA program participants may not be fully captured or WIA may be paying for costs of 

serving some customers (e.g., within the One-Stop) but may not have the opportunity to capture 

these individuals’ outcomes. 

The reverse side of this is the problem of co-enrollment. Co-enrollment means that a 

WIA customer is also served by another program, either within WIA or elsewhere (e.g., a 

dislocated worker customer is co-enrolled in the WIA Adult or Wagner-Peyser Programs). Coenrollment will make the amount of money spent on an exiter to achieve a particular outcome 

appear smaller than it actually is. This discrepancy could be controlled for given sufficient data, 

but local areas are inconsistent in recording co-enrollment of WIA participants into other 

programs. 

Non-Steady-State Programs. Vast programmatic changes can and do occur from year to 

year in different states, and this can greatly affect efficiency measure results. The most 

important instance in the data analyzed was the decision of several states to co-enroll all or most 

of their Wagner-Peyser customers in WIA. For example, WIA Adult Program exiters in New 

York surged by nearly a factor of 10, from 31,990 to 305,423 in just one year (from PY 2005 to 

PY 2006), while total expenditures actually decreased. This was the direct result of the state 

automatically enrolling Wagner-Peyser customers in WIA in PY 2006. The result, with regard to 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 161 

estimating efficiency measure results (and regression adjustment models) are profound -- for 

instance, in New York in the space of just one program year, the cost per entered employment 

for WIA Adult exiters plunged by nearly a factor of 10. Moreover, this change in the cost per 

entered employment is almost entirely driven by the fact that a large number of exiters are 

actually being mainly served by another program and not likely as a result of increase in the 

actual efficiency of the WIA program. Unless states develop relatively consistent policies on coenrollment that remain stable over several years, we believe it will be very difficult to estimate 

good efficiency measure regression models. 

Exclusions of Some Exiters from Common Measure Outcomes. A critical conceptual 

issue is that while the cost data is for all served within a state, Common Measures outcomes 

(because of the way in which they are defined) often exclude significant groups of exiters, and 

the number and proportion of exiters excluded, vary significantly across states. Exclusions, such 

as excluding exiters who were employed at the time of registration from the entered employment 

rate, are intended to measure performance based on the extent to which unemployed customers 

obtain jobs. However, when costs are being counted for all individuals served (regardless of 

whether they were or were not employed at the time of registration), the cost per entered 

employment measure overstates how much a state serving incumbent workers spends to have its 

customers obtain employment. Because the proportion of incumbent workers served varies 

significantly across states, the cost per entered employment measure will tend to overstate costs 

in states with a greater share of incumbent workers. For instance, overall, just over 21 percent of 

all WIA Adult exiters were excluded from the calculation of the entered employment rate from 

PY 2004 through PY 2006. However, this magnitude varies greatly from state to state, with 

exclusion rates ranging from 8.4 percent to 65.3 percent of exiters. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 162 

E. CONCLUSIONS 

 This chapter has focused on the prospects for taking a step beyond implementing 

efficiency measures to implementing performance standards with or without statistical 

adjustments for each efficiency measure. Such performance standards, if adopted, would parallel 

the standards used for outcomes under JTPA (using regression-based adjustment models) and 

currently under WIA (using negotiations). The modeling efforts conducted for this report 

focusing on the three WIA programs (using WIASRD data) – combined with analyses of 

aggregate national results on efficiency measures for WIA and the other ETA programs – 

suggests that great caution and several additional years of results are required before ETA should 

consider implementing performance standards for states/grantees on the recommended efficiency 

measures. For reasons discussed in the preceding section and highlighted below, it is unlikely 

that even with additional years of data that using an adjustment model (similar to what was used 

under JTPA) to set and adjust state/grantee-level standards for the recommended efficiency 

measures will be useful. This is not to say such modeling efforts should not be undertaken for 

other reasons – such as monitoring program performance or informing understanding of factors 

that are associated with higher or lower costs of serving program participants – or that such 

modeling efforts would not be successful in generating valid adjustments at a substate level (e.g., 

adjustment models may provide more significant coefficients for independent variables if 

conducted at the local workforce investment area level, where many more observations would be 

available). To summarize, based on experiences to date in applying an adjustment model at the 

state-level to efficiency measures for the WIA program, the following was found: 

• The WIASRD data set provides a rich source of data on participant characteristics, 

services received, and outcomes to make building such a model feasible. It is necessary 

to add state-level economic data, as well as cost data, to this dataset to be able to create a 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 163 

model to potentially adjust efficiency measure results. Similarly, other programs – such 

as TAA, SCSEP, INA, and others – collect individual participant data that could be used 

to develop and estimate similar adjustment models. 

• While such modeling is feasible, when actually run at the state level for a program, such 

as for the three WIA programs explored in this report, it does not necessarily yield 

statistically significant results for most of the independent variables tested (e.g., 

participant characteristics, services, and environmental factors). Overall, the regression 

models do not explain (with statistically significant coefficients) variation in the various 

efficiency measure results across states (such as cost per entered employment) in a way 

comparable to the models that have been used in the past to understand and adjust for 

variation on outcome measures (such as entered employment rate) in employment and 

training programs. 

• There are several reasons why in comparison to adjustment models used in the past to 

explain variance in outcomes (i.e., such as the adjustment models used under JTPA for 

many years), that the regression models applied to efficiency measures (at least under 

WIA using WIASRD data) fail to adequately identify factors that account for variation in 

results across states: 

• One important explanation is the number of observations (about 50 per year, 

slightly more if the Virgin Islands and Puerto Rico are included) is fairly 

small to yield statistically significant estimates.91 It would be preferable to 

apply the model at the local workforce investment area level, which would 

generate many more observations (about 600 observations per year for the 

WIA program), but cost data are only provided to the DOL at the state level at 

this time. 

• Second, there are a host of underlying factors that are difficult or impossible 

to take into account, given the current structure of programs and their 

reporting systems, especially with regard to accounting for cost sharing and 

co-enrollment across programs. It may not be feasible or cost-effective to 

collect the data needed to fully incorporate the underlying factors needed in 

the model – e.g., it may be necessary to introduce time sheets to accurately 

account for cost sharing related to staff within a One-Stop system. 

• Third, there is a need to account for customers who result in costs but are not 

counted in outcomes. For example, under the WIA Adult and Dislocated 

Worker programs, individuals who do not receive staff-assisted services but 

still may use One-Stop resources are not counted for outcomes; similarly, 

those who are employed at the time of registration for WIA are not included in 

entered employment counts (yet may consume considerable resources). 

• Finally, the workforce development system is in a state of flux regarding coenrollment of customers. Over the period analyzed, some states co-enrolled 

 

91 With regard to outcomes, it is possible to estimate models at the individual participant level, workforce investment 

level, or state level. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 164 

substantial numbers of Wagner-Peyser customers in WIA, while others did 

not. Adding Wagner-Peyser customers to the WIA customer base can 

increase the number of WIA customers by a factor of 10 or more. Until the 

system stabilizes, regression analysis cannot meaningfully be used to estimate 

relationships that are not stable. 

While use of an adjustment model for the recommended efficiency measures cannot be ruled out 

for other programs based solely on experiences to date in testing such adjustment models for 

WIA, the implications are that such adjustment models are likely to suffer from some or all of 

the same problems – particularly low numbers of observations (if models are calculated at the 

state level), difficulties accounting for shared costs, and complications related to accounting for 

co-enrollment. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 165 

CHAPTER 8: 

STUDY RECOMMENDATIONS AND CONCLUSIONS 

The final chapter provides a set of study recommendations for implementation of specific 

outcome-based efficiency measures for the 11 ETA programs. It begins with an overview of the 

views and concerns expressed by the Expert Panel on a range of issues related to identifying and 

implementing efficiency measures. Building off of the perspectives of the members of Expert 

Panel and key findings from earlier chapters of this report, nine study recommendations are 

offered and discussed. 

A. OVERVIEW OF EXPERT PANEL VIEWS ON IMPLEMENTATION OF 

OUTCOME-BASED EFFICIENCY MEASURES BY ETA PROGRAMS 

The Expert Panel provided many useful comments at a meeting held November 9, 2009, 

and the panel also provided written comments, which are contained in Appendix D. The primary 

message conveyed by the panel is that based on the limited data available, difficult issues on how 

to deal with factors such as co-enrollment and shared costs, and ETA’s prior experience with 

efficiency measures (where an evaluation indicated that the efficiency measures led to creaming 

applicants and more use of less expensive services than was warranted), ETA should move very 

cautiously in adopting efficiency measures. Instead, the panel suggested that ETA track a variety 

of potential measures before establishing any efficiency measures associated with rewards or 

sanctions for states. 

Dr. Beryl Radin’s comments stem from her perspective as a political scientist who has 

studied and written about performance measurement for many years. Radin’s central point is 

that the change in administrations in Washington has changed the views of OMB managers and 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 166 

others on how performance management and program evaluation should be implemented. The 

changes in philosophy have not yet fully worked their way down to agency-specific measures, so 

she urges caution in continuing to apply the principles established by the prior administration. 

Moreover, implementation of the stimulus package has changed priorities about the goals of 

programs, leading to more focus on job creation than was previously the case. Finally, Radin 

suggests that ETA consider the basic objectives of the performance management system, and she 

wonders if it is prudent to use the same measures across programs with diverse services and 

customers. 

Dr. Christopher King, a professor and economist who has been involved in performance 

management issues for workforce investment programs from their inception in the 1970s,, also 

urges ETA to be cautious in implementing efficiency measures for its programs. He notes that 

when efficiency measures were first employed, they led to serious problems and resulted in 

amendments prohibiting their use under JTPA. King also notes that the world is much more 

complex than it was when efficiency measures were first used because of co-enrollment, 

leveraged resources, and loose measurement of performance data. He concludes that unintended 

consequences are all but guaranteed in such an environment. King urges that efficiency 

measures with rewards and sanctions not be instituted at this time, and he argues that several 

programs, Apprenticeship and Work Incentive Grants, are so different that they should be 

excluded from the measures. King makes several other points, and he suggests that while Return 

on Investment (ROI) be tracked, it should be looked at periodically through evaluations rather 

than through the performance management system on an annual basis. 

Dr. Carolyn Heinrich, a public policy professor, provided detailed comments on all the 

issues raised for the meeting. While recognizing that it is important for ETA to explore the use 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 167 

of outcome-based efficiency measures, Heinrich notes that the literature is clear that 

organizations respond to performance measures, often in unintended ways. She notes that 

conducting analyses at the state level hides much of the variation in outcomes and explanatory 

variables, so she recommends that data be collected and analyzed at the sub-state level. Like the 

other panel members, Heinrich stresses the importance of moving slowly before implementing 

efficiency standards. Heinrich emphasizes the importance of collecting uniform and accurate 

data on participants and also trying to collect efficiency data by activity. 

Dr. Jeffrey Smith, an economics professor, advised ETA to be careful in interpreting 

efficiency measures of the type being considered. Smith notes that available research indicates 

that the outcome measures currently in use are measures of gross outcome only and are weakly, 

at best, related to program impact. Smith suggests that, in the long run, ETA should substantially 

modify the performance management system so that random assignment based impact estimates 

be computed for each state and that program impact be used in place of gross outcomes in 

performance measures. Smith is optimistic, based on his experience with Canadian workforce 

programs, that reasonable impact measures can be developed. Like the other panel members, 

Smith cautions that the WIG and Apprenticeship programs are quite different from the other nine 

programs and should have different efficiency measures. Smith provided detailed responses to 

the questions posed to the panel, and he urged ETA to develop accurate, consistent data as an 

important step in improving the performance management system. 

In sum, the panel encourages ETA to move cautiously in adopting outcome-based 

efficiency standards. The panel encourages ETA to improve the quality of the data and to 

explore the used of sub-state data and activity-based data. In conjunction with these 

recommendations, the panel suggests that ETA track a number of measures to better assess 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 168 

which measures provide the best efficiency measures. The panel members also offered 

suggestions on how to deal with thorny issues such as co-enrollment, shared costs, and services 

to customers who are not counted in the outcome measures. 

B. STUDY RECOMMENDATIONS ON IMPLEMENTATION OF OUTCOMEBASED EFFICIENCY MEASURES FOR ETA PROGRAMS 

This section of the report provides recommendations regarding implementation of 

outcome-based efficiency measure for the 11 ETA programs that are the focus of this study. 

These recommendations are based on analysis of available data, as well as input from members 

of the Expert Panel, interviews with state workforce agencies, ETA program administrators/staff, 

and the ETA internal work group.92 

1. Recommendation #1: Use Program Expenditures Rather than 

Appropriations or Obligations as the Measure of Program Costs in 

Efficiency Measures 

To generate an efficiency or unit cost measure, it is necessary to account for costs in the 

numerator of the measure and a unit of participation, service delivery, or outcome in the 

denominator of the measure. Based on discussions with ETA officials (and a review of the 

literature and available data sources within ETA), three potential types of “cost” data could be 

considered for the numerator of each efficiency measure: 

• Appropriations/Allotments – “Appropriations” are defined by the Government 

Accountability Office as “Authority given to federal agencies to incur obligations and to 

make payments from Treasury for specified purposes.”

93 Appropriations are generally 

the amount of funding made available by Congress for spending on a given program 

(such as the TAA program) during a fiscal year. Allotments are the amount of 

 

92 The recommendations in this report differ to some extent to the recommendations in our earlier reports. The 

changes are due to additional data analyses and suggestions from ETA staff and the Expert Panel. 93 See U.S. Government Accountability Office (2004). Principles of Federal Appropriations Law, Third Edition, 

Volume 1. Washington, DC: U.S. Government Accountability Office Report GAO-04-261, SP, p. 2-5. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 169 

appropriated funds distributed to a state or grantee based upon a legislative or regulatory 

formula.94 Allotments exclude amounts retained by the federal government to administer 

programs. 

• Obligations – According to the General Accounting Office (GAO), “...obligations 

reflect orders placed, contracts awarded, and other similar transactions during a fiscal 

year. As an expression of an agency’s total financial commitments for a given period, 

gross obligations portray the relative size of an organization, without regard to the type 

of underlying budgetary resource or when resulting outlays may occur.”95 Hence, 

obligations are funds that have been committed through contracts, grants, and other 

vehicles.

• Expenditures – Expenditures are funds paid or the amount of funds due (depending on 

whether a cash or accrual basis is used) for provision of goods or services pursuant to a 

grant or contract agreement. 

Use of expenditures rather than appropriations, allocations, or obligations in calculating 

efficiency measures is recommended because (1) expenditures can vary substantially from what 

is initially appropriated/allocated, especially at the state level (because of transfers, rescissions, 

and unexpended funds); and (2) expenditures reflect what is actually spent on delivery of 

services and capture the underlying notion of efficiency. States interviewed, ETA program 

offices, and the Expert Panel endorsed the use of expenditures over the other available measures 

of costs. 

2. Recommendation #2: Use Common Measures as Starting Point for 

Measuring Program Outcomes in Efficiency Measures 

If they were to be implemented in the short run (i.e., within three years), efficiency 

measures should be closely tied to the current outcome performance measures in effect under 

ETA’s Common Measures framework. Data is already being collected at the state and grantee 

 

94Although the term “allocation” is used quite often in an interchangeable manner with allotment, in the context of 

formula programs the appropriate term is allotment. 95United States General Accounting Office (GAO), Federal Budget: Agency Obligations by Budget Function and 

Object Classification for Fiscal Year 2003, GAO-04-834, June 2004 (available at 

http://www.gao.gov/new.items/d04834.pdf). 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 170 

levels on these outcomes, so the performance data needed to generate efficiency measure results 

would already be available (for most programs) – reducing costs and start-up time. In addition, 

though the report highlights some of the challenges of comparing efficiency measure results 

across programs and there is considerable variability across programs in terms of data quality 

and comparability, the Common Measures provide common definitions for outcome measures 

and thus increase the potential for making meaningful comparisons of efficiency measure results 

within individual programs (e.g., across states/subgrantees) and across at least some of the ETA 

programs of interest. As discussed in more detail in the report, the following are key features of 

the current Common Measures performance measurement system across the 11 ETA programs 

of interest: 

• Ten (of the 11) ETA programs currently use the Common Measures: Workforce 

Investment Act (WIA) Adult, WIA Youth, WIA Dislocate Worker, WIA National 

Emergency Grants (NEG), Wagner-Peyser/Employment Service, Trade Assistance 

Act (TAA), Senior Community Service Employment Program (SCSEP), Indian and 

Native American (INA), National Farmworker Jobs Program (NFJP), and Work 

Incentive Grant (WIG) Programs. The Apprenticeship program (the 11th program), 

which has in the past measured program performance in terms of retention and 

earnings gains for apprentices, is in the process of shifting to tracking and reporting 

on the three main Common Measure performance outcomes. 

• For Common Measures programs (except WIA Youth), outcome measures are 

entered employment, retained employment, and post-program earnings.96 

• For WIA Youth, the Common Measures are placement in education or training, 

credential attainment, and literacy/numeracy. 

It is essential that the outcome-based efficiency measures adopted should be sensitive to goals 

and objectives of each of the 11 programs. There are three potential efficiency measures the 11 

ETA program could implement over the short run based on the Common Measure outcomes: 

• Cost per entered employment (for the WIA Youth program - cost per placement in 

employment or education); 

 

96 See Chapter 6 for detailed definitions of each of these measures. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 171 

• Cost per retained employment; and 

• Cost divided by post-program earnings. 

In addition, although not currently used, cost divided by change in post-program earnings could 

be used, as this data is being collected by most of the programs currently (as a result of the 

Common Measures post-program earnings measure) and was formerly a measure that was used 

by the WIA Adult and Dislocated Worker programs. Because the proposed efficiency measures 

are based on cost and outcome data already being submitted to ETA (or soon to be reported in 

the case of the Apprenticeship program) through annual expenditure reports and Common 

Measures reporting, states and grantees will not have to develop new data systems or submit new 

cost or outcome data. We recommend that ETA initially focus on monitoring results nationally 

and at the state/grantee levels on these four outcome-based efficiency measures for several years 

for most of the ETA programs, as described below. See Exhibit 7-1 for overview of programs 

for which the four recommended efficiency measures highlighted below should be considered: 

• Recommendation 2a: Cost per entered employment should be tracked (for monitoring 

purposes initially) as an efficiency measure for 8 of the 11 ETA programs: WIA Adult, 

WIA Dislocated Worker, WIA NEG, Wagner-Peyser, TAA, SCSEP, INA, and NFJP. 

The WIA Youth program could use cost per placement in employment or education (as 

an alternative to cost per entered employment). The WIG program should be excluded 

from implementing this measure because it does not enroll participants; the 

Apprenticeship program should be excluded because federal funding is primarily used 

to promote and monitor apprenticeship programs (i.e., other sources of funding 

primarily pay for training-related costs) and other factors discussed earlier in this 

report and summarized in the next recommendation (Recommendation #3).97 The main 

rationale for recommending cost per entered employment (and using cost per placement 

in education or training for the WIA Youth program) as an efficiency measure is as 

follows: (1) employment is a high priority for all programs except WIA younger youth; 

(2) entered employment is the simplest and most direct way to assess whether programs 

are achieving their goals; (3) data are already being collected on the number of entered 

employments under the Common Measures 

 

97 In addition, the WIG program’s funding is scheduled to be eliminated in the coming program year, and so, by the 

time the measure was implemented the program will have ended (see below, Recommendation #3 for additional 

discussion of WIG and potential suitable efficiency measures). 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 172 

EXHIBIT 7-1: OVERVIEW OF RECOMMENDED OUTCOME-BASED 

EFFICIENCY MEASURES FOR TRACKING BY ETA PROGRAMS 

PROGRAM COST PER 

ENTERED 

EMPLOYMENT 

COST PER 

RETAINED IN 

EMPLOYMENT

COST DIVIDED BY 

POST-PROGRAM 

EARNINGS 

COST DIVIDED 

BY CHANGE IN 

EARNINGS 

WIA Adult √ √ √ √

WIA DW √ √ √ √

WIA Youth √

(Use cost per 

placement in 

employment or 

education) 

X X X 

NEG √ √ √ √

TAA √ √ √ √

Wagner-Peyser √ √ √ √

SCSEP √ √ √ √

NFJP √ √ √ √

INA √ √ √ √

WIG X X X X 

Apprenticeship X X X X 

Note: “√” denotes that the measure is recommended for implementation as a performance monitoring 

efficiency measure; “X” indicates measure should not be adopted by the program. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 173 

(making this measure feasible and relatively inexpensive to implement); and (4) in 

comparison to other measures, data are available sooner for entered employment than for 

post-program earnings and job retention rates. 

• Recommendation 2b: Cost per retained in employment should be tracked (for 

monitoring purposes initially) as an efficiency measure for 8 of the 11 ETA programs: 

WIA Adult, WIA Dislocated Worker, WIA NEG, Wagner-Peyser, TAA, SCSEP, INA, 

and NFJP. The WIA Youth, Apprenticeship, and WIG programs should be excluded 

from implementing this measure. This is not a Common Measure collected for WIA 

Youth (and is not applicable to younger youth served by the program) and, should not 

be applied to the WIG and Apprenticeship program for reasons discussed in 

Recommendation 2a (note: see Recommendation #3 for additional details on why it is 

not applicable and potential alternative measures for these two programs). The main 

rationale for recommending cost per retained in employment as an efficiency measure is 

similar to cost per entered employment and is as follows: (1) job retention is a high 

priority for all programs except WIA younger youth; (2) participants who are employed 

at the time of entry into the program are included in this measure (unlike entered 

employment rate); (3) data are already being collected on the number of retained 

employments under the Common Measures (making this measure feasible and relatively 

inexpensive to implement). In comparison to entered employment rate, data on this 

measure is available later, but this indicator provides a downstream measure (of job 

retention and longer-term employment) of the effects of training and other employment 

services. Additionally, this measure is appropriate for incumbent workers, a source of 

concern that some states expressed about the cost per entered employment efficiency 

measure, which (as defined under Common Measures) excludes individuals who were 

working at the time of enrollment. 

• Recommendation #2c: Cost divided by post-program earnings should be tracked (for 

monitoring purposes initially) as an efficiency measure in 8 of 11 ETA programs: WIA 

Adult, WIA Dislocated Worker, WIA NEG, Wagner-Peyser, TAA, SCSEP, INA, and 

NFJP. The WIA Youth, Apprenticeship, and WIG programs should be excluded from 

implementing this measure. This is not a Common Measure collected for WIA Youth 

(and it is not applicable to younger youth) and, should not be applied to the WIG and 

Apprenticeship program for reasons discussed in Recommendation 2a (note: see 

Recommendation #3 for additional details on why it is not applicable and potential 

alternative measures for these two programs). The rationale for recommending cost 

divided by post-program earnings as an efficiency measure for implementation by ETA 

programs is as follows: (1) effective programs should increase earnings as well as 

employment; (2) omitting earnings might encourage focus on inexpensive labor exchange 

or core services rather than intensive services and training; (3) participants who are 

employed at the time of entry into the program are included in this measure (unlike 

entered employment rate); (4) data are already being collected on the pre- and postearnings under the Common Measures (making this measure feasible and relatively 

inexpensive to implement). In comparison to the entered employment rate, data on this 

measure is available later, but this indicator provides a downstream measure of the 

earnings effects of training and other employment services that result in job retention and 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 174 

longer-term employment. 

• Recommendation #2d: Cost divided by change in earnings should be tracked (for 

monitoring purposes initially) as an efficiency measure in 8 of the 11 ETA programs: 

WIA Adult, WIA Dislocated Worker, WIA NEG, Wagner-Peyser, TAA, SCSEP, INA, 

and NFJP. The WIA Youth, Apprenticeship, and WIG programs should be excluded 

from implementing this measure. This is not a Common Measure collected for WIA 

Youth (and it not applicable to younger youth, and it should not be applied to the WIG 

and Apprenticeship programs for reasons discussed in Recommendation 2a (note: see 

Recommendation #3 for additional details on why it is not applicable and potential 

alternative measures for these two programs). The rationale for recommending cost 

divided by change in earnings as an efficiency measure for implementation by ETA 

programs is as follows: (1) effective human capital building programs should increase 

earnings as well as employment; (2) omitting earnings might encourage focus on 

inexpensive labor exchange or core services rather than intensive services and training; 

(3) by looking at pre/post earnings change (versus average post-program earnings), 

programs face fewer incentives for “creaming” those individuals who are likely to have 

the highest post-program earnings; and (4) although an earnings change measure is not 

currently used under the Common Measures, data are already being collected on the preand post-earnings under the Common Measures (making this measure feasible and 

relatively inexpensive to implement). 

3. Recommendation #3: Carefully Consider Programmatic Differences Before 

Implementing Efficiency Measures – Among the 11 ETA Programs, WIG, 

Apprenticeship, and WIA Youth Programs Will Likely Require a Different 

Set of Efficiency Measures 

An often recurring message of ETA program administrators and state program operators 

– reinforced by the Expert Panel and findings from the literature – is that the 11 ETA programs 

have varying goals/objectives, target and serve different at-risk populations, offer widely varying 

types and intensities of services, and have widely differing costs. As a result, efficiency 

measures need to be cautiously developed and tailored to what programs are attempting to 

achieve – and great care is needed in comparing results on such measures within programs (i.e., 

across states, grantees, and local jurisdictions) and across programs. The quantitative analyses of 

efficiency measure results conducted as part of this study, showed substantial differences on 

measures such as cost per entered employment both within programs (across states/grantees) and 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 175 

between programs. Program officials at the state and local levels were concerned about unfair 

comparisons being made between programs and the possibility that higher cost programs 

(particularly those providing training services and serving more disadvantaged populations) 

would be seen in a poor light (e.g., have much higher cost per entered enrollment) – and perhaps, 

as a result, eventually see funding and services gravitate toward lower intensity, less costly 

services or have their budgets inappropriately reduced. 

As alluded to in Recommendation #2, among the 11 programs included in this study, 

three programs in particular – the Apprenticeship, WIG, and the WIA Youth programs -- stand 

out as qualitatively different from the other programs. With regard to these three programs, ETA 

should be cautious in applying the measures identified in Recommendation #2 and should 

consider alternative measures carefully tailored to the goals/operations of these programs: 

• Apprenticeship Program – Several operational aspects of the Apprenticeship program 

make it quite different from the other 10 ETA programs. First, federal funding is 

used to support promotion, assessment, and registration of Apprenticeship programs 

across the country, but federal funding does not pay directly for delivery of services 

or training costs. The costs of training and serving apprentices are paid by the state, 

unions, employers, and/or participants. Second, in roughly half the states, these 

functions are undertaken by the states themselves at their own expense rather than 

ETA’s, and so in those states, there is virtually no federal cost. In addition, the 

Apprenticeship program encourages sponsors and apprentices to use WIA and other 

funding sources to cover the cost of training where appropriate – which means that 

costs of delivery of training is not fully reflected in Apprenticeship expenditures. 

Finally, because apprentices can be enrolled for up to five years, the costs in the year 

of completion may be less representative of the total costs when compared with other 

ETA programs. 

• Work Incentive Grant Program – Unlike the other ETA programs, the WIG program 

does not directly enroll or serve individual customers. Rather, the Disability 

Navigators (DNs) funded under this initiative are responsible for building the 

capacity of state and local workforce agencies to outreach to and more effectively 

serve individuals with disabilities. The primary role of DNs is to train staff at the 

local workforce level on disability issues and effective service delivery for disabled 

individuals. The grants issued under the WIG program are aimed at systemic change 

– Disability Navigators funded under the grants provide training to staff that is 

expected to yield improvements in access and quality of services to disabled 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 176 

individuals over the length of the grant period (and even after the grant is concluded). 

There are several other factors that complicate the implementation of measures 

recommended under Recommendation #2 – (1) funding for this program is scheduled 

to end shortly – and so it is likely that measures could not be fully implemented prior 

to the end of the program, (2) the current WIG reporting system relies on the 

WIASRD data, which only covers WIA customers – which means that the effects of 

grants are missed for other (non-WIA enrolled) customers of the One-Stop system 

(e.g., Wagner-Peyser/ES registrants); and (3) the disability status of individuals 

served under WIA and other ETA programs is not fully captured because some 

individuals may not view themselves as disabled and/or are reluctant to reveal their 

disability status. 

• WIA Youth Program - While employment, job retention, and earnings change are 

important goals for older youth served under the program, such goals are longer-term 

objectives for younger youth. Employment may, in fact, be counterproductive to 

younger participants compared to completing school and earning education 

credentials. Efficiency measures that capture employment and 

education/credentialing are more appropriately connected to the underlying multiple 

purposes of the WIA Youth program component. An additional challenge with 

regard to the WIA Youth program is that expenditure data are only available at the 

national level for all youth combined (i.e., so it is not possible to calculate a separate 

efficiency measure for younger and older youth served by the program). 

Reasonable alternatives for measuring efficiency are needed for the these three programs 

that take into consideration special circumstances with regard to program goals, types of 

individuals served, and services delivered. While it is important to hold these three programs 

accountable for their “efficiency,” the measures employed need to be carefully linked to 

purposes and activities on which the federal funding is expended. Alternative efficiency 

measures for these three programs are the following: 

• Recommendation #3a: ETA should consider alternative efficiency measures for the 

Apprenticeship Program linked to the goals of the program and what federal funds are 

being spent on – for example, increasing the number of apprenticeships offered, 

building the quality of apprenticeship programs, and registering and monitoring of 

Apprenticeship programs accurately and in a timely manner. Therefore, ETA should 

consider applying the following alternative efficiency measures to the Apprenticeship 

program (all at the national level): cost per additional apprenticeship program 

registered and timeliness of registration decisions. Specific measures should be 

adopted after appropriate dialogue and analysis is undertaken. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 177 

• Recommendation #3b: ETA should consider alternative efficiency measures for the 

WIG Program that reflect the training and technical assistance goals of this program. 

In particular, efficiency of this program should be aimed at measuring how the 

services of Disability Navigators increase the numbers of disabled individuals served by 

the One-Stop system (and various ETA programs operating out of the One-Stop 

system), as well as improvements in identifying individuals served by the workforce 

system and enhancements to the quality of services provided to disabled individuals. 

Therefore, ETA should consider applying the following alternative efficiency measure 

to the WIG program): cost per change in the number of One-Stop customers served 

with disabilities. As for the Apprenticeship program, efficiency measures for WIG 

should be adopted after appropriate dialogue and analysis. Because the WIG program 

may be terminated, it may be appropriate not to develop efficiency measures for this 

program. 

• Recommendation #3c: As discussed in Recommendation #2 above, the efficiency 

measure that should be applied to the WIA Youth Program is cost per placement in 

employment or education. In the longer term, the possibility of collecting separate cost 

and customer data for in-school and out-of-school youth should be investigated so that 

appropriate separate outcome and efficiency measures can be developed for these 

disparate groups. For example, if separate cost data were available for these two 

groups, it would feasible and perhaps appropriate to apply the four measures 

recommended for the WIA Adult and Dislocated Worker Programs to the older youth 

served by the WIA Youth Program (i.e., cost per entered employment, cost per retained 

in employment, cost divided by post-program earnings, and cost divided by pre/postprogram earnings). 

There are distinctive characteristics of each program that make it difficult to 

appropriately compare and contrast efficiency measure results across programs – and hence, 

ETA should be cautious in making such comparisons, and when direct comparisons are made, 

care should be taken to note that there are critical differences in program goals, types of 

individuals served, and types/intensity of services delivered under the various programs. 

4. Recommendation #4: Performance Standards for States/Grantees on 

Recommended Efficiency Measures Should Be Considered Exploratory at 

This Time -- Do Not Reward or Sanction States/Grantees for Performance on 

the Recommended Efficiency Measures 

It is recommended that ETA track efficiency measure results for ETA programs of 

interest for several program years for program monitoring purposes only. Several years of 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 178 

experience are needed with the efficiency measures (perhaps three or more years) to determine if 

it is appropriate to set standards and apply rewards and sanctions to states and grantees on the 

efficiency measures (as is currently done for outcome measures). The additional time is needed 

to identify definitional problems (particularly with respect to expenditures to be included), allow 

for co-enrollment patterns to stabilize, analyze variation in performance on the efficiency 

measures across states, and determine if and how the standards should be adjusted to take 

account of various factors. It is also important to assess factors that account for variation across 

states/grantees on efficiency outcomes, as well. With the rapid changes in co-enrollment 

patterns underway in many states, it will be possible to assess how the large increases in 

participant and exiter counts affect the outcome-based efficiency measures (and outcomes for 

other Common Measures) over the next few years. 

A key finding from this study (based on a careful review of the literature and input from 

the Expert Panel and our interviews with federal/state program officials) is that it is important in 

selecting measures, standards, rewards, and sanctions to anticipate the behavioral changes that 

are likely to be induced by the performance management policies adopted and to structure the 

system so that the presence of efficiency measures does not result in undesirable behavior by 

states and grantees.98 In addition, careful thought needs to be given as to whether outcome-based 

efficiency measures should be applied only at the national level or whether they should be 

“drilled-down” from the federal level to states/grantees and/or local workforce investment areas. 

The application of efficiency measures (and setting of standards on such measures) across 

states/grantees or local levels could, for example, lead to efforts by states and localities to reduce 

per-participant costs by either providing less costly services (e.g., reducing the amounts of more 

 

98Unlike WIA, targeted programs such as NFJP and INA, have no mandates for sanctions or rewards in their 

legislation and, thus, it is not clear that sanctions/rewards can and should be considered for programs without such 

mandates. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 179 

costly staff-assisted and training services, in favor of self-service labor exchange services) or 

bolstering the numbers of individuals enrolled (and co-enrolled) in programs. Therefore, ETA 

should initially use the efficiency measures recommended above for program monitoring 

purposes – and only after at least several years of observation consider whether to apply 

standards (and rewards/sanctions) to some or all of the measures recommended. 

5. Recommendation #5: Improve Consistency and Quality of Cost, Customer 

Characteristics, and Outcome Data 

This study has attempted to establish a baseline of efficiency measure results for the 11 

ETA programs -- an effort that has not always resulted in success. For some programs (notably 

the WIA, ES, and TAA programs), there was success in obtaining both the cost and outcome 

data to generate three years of efficiency measure results both at the national and state levels. 

Other programs struggled with providing data for the full three-year period and some could not 

generate the outcome data needed for even one year (in part, because they may have recently 

transitioned to the Common Measures or were in the process of making this transition). The 

efficiency measure results for programs that were able to provide three years of data at the state 

or grantee level revealed substantial variation in results within states from year to year and 

among states/grantees within a single program year. These variations were sometimes very 

large, and it is not clear that this variation is a true reflection of the “efficiency” of programs or 

of other factors – including substantial cross-state differences in co-enrollment patterns and in 

the ways in which states collect and report on both program costs and outcomes, as well as 

simply erroneous data. Given the difficulty in obtaining cost and outcome data for programs at 

the federal and state/grantee levels – and the variable nature of efficiency measure results we 

suggest that ETA carefully study variation in the ways in which costs and outcome data are being 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 180 

collected within programs (across states) and across programs and, if consistent with other 

priorities, require that states and grantees use common definitions, and procedures to report data 

to ETA and that they be required to assure the quality of the data reported . 

6. Recommendation #6: ETA Should Explore Developing Efficiency Models by 

Activity for Programs That Offer a Range of Activities. 

Some programs, such as WIA Adult and Dislocated Worker programs, offer a wide range 

of activities in terms of the cost per customer. For example, assisted core services may cost well 

under $100 per customer, but occupational training sometimes costs $10,000 or more per 

customer. The concept of cost per customer loses much of its meaning when services vary so 

much within and across states and grantees. More meaningful results are likely to be obtained 

for programs with a range of activities if separate regression models could be estimated for each 

major activity. In WIA, for example, separate efficiency models could be estimated for each of 

the three tiers of service or for training and core assisted and intensive services combined. 

Unfortunately, it is not a simple matter to develop such models because cost data are currently 

not collected by activity in most states. In an earlier report, the authors identified only two states 

that collect costs by activity, Florida and Michigan, and these states do not use the same methods 

and sub-state areas also use a variety of approaches within the states.99 To develop statistical 

models of efficiency measures, ETA would have to mandate that states/grantees and, most likely 

substate grantees as well, collect cost data by activity. To assure that the data are consistent 

across states, ETA would also have to require that a specific method be used to allocate joint 

costs and to deal with co-enrollment. We recognize that this would be a considerable burden, so 

 

99 Heinberg, John, and J. Trutko, B. Barnow, M. Farrell, and A. Glosser. 2005. Unit Costs of Intensive 

And Training Services for WIA Adults and Dislocated Workers: An Exploratory Study of Methodologies 

and Estimates in Selected States and Localities: Final Report. Capital Research Corporation: prepared 

for the U.S. Department of Labor, Employment and Training Administration.
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 181 

our recommendation at this time is that ETA explores this issue further rather than immediately 

begin collecting such data. 

7. Recommendation #7: Adjustment Models for Efficiency Measures Are Not 

Likely to Be Useful at the State-Level for Many Years; They Potentially 

Could be Useful and Valid at the Local/Grantee Level for Some Programs 

 The modeling efforts conducted for this report, which focused on the WIA Adult and 

Dislocated Worker programs (using WIASRD data), suggested that great caution and several 

additional years of results are required before ETA should consider implementing state-level 

regression models to adjust performance standards for states/grantees on the recommended 

efficiency measures. Overall, the regression models tested for the WIA Adult and Dislocated 

Worker programs did not provide sensible magnitudes and statistically significant coefficients 

for the regression coefficients in the models. It is possible that once the co-enrollment pattern 

stabilizes and several years of data after that are available, regression modeling might produce 

useful results. Other factors that could make such modeling useful include requiring states (and 

local areas where appropriate) and grantees to use consistently measured and high-quality data 

(see Recommendation 5) and that activity-level cost data be collected (see Recommendation 6). 

8. Recommendation #8: Estimate Return on Investment (ROI) in Conjunction 

with Impact Studies but Not as Regular Performance Measurement 

ROI and the closely related concept of cost-benefit analysis are essential to assess if a 

program is a worthwhile investment and to compare alternative investments -- we encourage the 

Department of Labor and states to conduct such analyses on a regular basis. However, it would 

be very expensive to measure ROI on an annual basis as a performance measure. Other 

challenges to using ROI on a regular basis as a performance measure are (1) a lack of consensus 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 182 

about the best methods that should be used to generate appropriate comparison group data (e.g., 

some analysts believe that the matching methods that are widely used do not provide good 

comparison groups), (2) the time required to generate reasonable post-program data is too long to 

provide measures that are useful for annual performance measures, (3) there is a lack of 

consensus among economists and government agencies about the appropriate discount rate to 

use, and (4) for programs that provide earnings gains, it is difficult to make reasonable 

assumptions about how long observed earnings gains will persist. Thus, it is recommended that 

ROI and cost-benefit analysis should be considered an important tool for periodic program 

evaluation rather than annual performance assessment. 

9. Recommendation #9: Further Study Is Needed on Several Topics Areas 

Related to and Likely to Affect Efficiency Measure Results – Including CoEnrollment and Cost Sharing 

Through our conversations with ETA officials and states, we identified a number of 

important issues that must be resolved so that ETA’s efficiency measures are consistent. These 

issues involve policy considerations that can only be made by ETA officials. For example, some 

programs, such as SCSEP and TAA, provide stipends or other cash payments on a regular basis; 

whether expenditures on stipends are included as an expenditure would have a major effect on 

how costs compare across programs. Examples of such issues include: 

• Should stipends, supportive services, and need-based payments be counted as 

expenditures in computing efficiency measures?

• How should co-enrollment be accounted for in efficiency measures?

• How should shared costs such across programs, such as One-Stop Career Center 

infrastructure costs, be dealt with in efficiency measures?

• How should the efficiency measures deal with customers who remain in the 

program for more than one program year?
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 183 

• How should self-service customers be dealt with in programs such as WIA Adult 

and Dislocated Worker programs where such customers are not counted for 

outcome measures?

• How should self-service customers be dealt with in programs such as WIA Adult 

and Dislocated Worker programs where such customers are not counted for 

outcome measures?

• How should incumbent worker customers be dealt with in programs such as WIA 

Adult and Dislocated Worker programs where such customers are not counted for 

outcome measures?

Resolution of these issues is important for making sure that the efficiency measures are 

consistent within and across programs and to assure that programs and policy officials 

understand what is and is not being captured by the efficiency measures. As noted above, these 

issues generally require value judgments and some of them would have major cost implications 

for data collection. 

 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 184 

APPENDIX A: 

DISCUSSION GUIDES FOR INTERVIEWS WITH 

STATE WORKFORCE AGENCIES 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 185 

DISCUSSION GUIDE FOR EARLY ROUND OF 

INTERVIEWS WITH SIX STATES 

1. Do you have a performance measurement/management system? If so, please briefly describe 

this system and the measures that are included in the system. If available, please provide 

additional written documentation about this system. 

2. What types of efficiency measures are currently being used by your agency, especially with 

regard to employment and training programs administered by your agency? By efficiency 

measures, we mean a performance measure that includes some measure of program cost or 

resources used; examples include cost per participant, cost per entered employment, and 

return on investment. With regard to each efficiency measure in use: 

 a. How is the measure defined? 

 b. Why was this measure selected? 

 c. What were the challenges to developing and implementing this measure? 

 d. At what program level or levels (federal, state, or local/program levels) is the efficiency 

measure applied? If the efficiency measure is applied at the state or local/program levels, 

is the same standard or varying standards applied across states or local/program areas? If 

varying standards are used, how are the standards developed (e.g., statistical models or 

through negotiations)? 

3. What types of data and data systems are used to generate the necessary participant and cost 

data required to generate valid, reliable, and appropriate efficiency measures? How 

burdensome do you think data collection is at the federal, state, and/or local/program levels? 

Are there types of data that have been particularly difficult to obtain (e.g., costs broken down 

by discrete service)? 

4. How has your agency used efficiency measures to monitor and enhance program 

performance? Please explain any rewards and/or sanctions that are applied because of 

performance on the efficiency measures you use. 

5. If more than one efficiency measure is used, are all of the measures given the same weight in 

assessing performance or is greater weight given to certain measures? If different weights 

are applied, which measures are given more weight and why? 

6. How important are the efficiency measures compared to other performance measures you 

use? Are all of the measures given the same weight in assessing performance or is greater 

weight given to certain measures? If different weights are applied, which measures are given 

more weight and why? 

7. What have been the effects/consequences of the efficiency measures on the number of people 

served, they way they are served, and program outcomes, both intended and unintended? 

Please explain. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 186 

8. Are there efficiency measures that your agency used in the past, but are no longer used? If 

so, what were these earlier efficiency measures and why were they discontinued? 

9. Does your agency have plans to implement additional efficiency measures in the future? If 

yes, what types of measures are being considered and why are they under consideration? 

10. Does you agency provide technical assistance, training, and ongoing monitoring to ensure 

effective implementation of efficiency measures at the state, local, or programmatic levels? 

If so, what types of technical assistance and training is provided? 

11. Does your agency have any additional views about efficiency measures or recommendations 

with regard to development or implementation of efficiency measures for employment, 

training, or other human service programs? 

12. Please provide any background reports or documents that you have concerning the efficiency 

measures that your agency has implemented. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 187 

DISCUSSION GUIDE FOR LATE ROUND OF 

INTERVIEWS WITH FIVE STATES 

A. Co-Enrollment Policies and Practices 

1. What are your state’s policies with regard to co-enrollment of participants of ETA-funded 

programs (particularly policies with regard to WIA, Wagner-Peyser/ES, and TAA 

participants)? 

2. Do co-enrollment policies/practices vary within the state by local workforce area? If yes, 

to what extent and why? 

3. How reliable is your state’s data system in capturing patterns of co-enrollment (e.g., for 

WIA, Wagner-Peyser/ES, and TAA participants)? If there are challenges to tracking 

co-enrollment, why is this the case? 

4. For the past 3 years, have there been any major changes in the number or percentage of 

participants that are co-enrolled between the programs listed below? And if so, how 

and why? (Note: if available, please provide actual counts or estimates of the 

numbers/percentages of participants co-enrolled in each program.)

a. WIA Adults co-enrolled in WIA Dislocated Worker, WIA Youth, WagnerPeyser/ES, or TAA programs 

b. WIA Dislocated Workers co-enrolled in WIA Adult, WIA Youth, WagnerPeyser/ES, or TAA programs 

c. WIA Youth co-enrolled in WIA Dislocated Worker program, WIA Adult, 

Wagner-Peyser/ES, or TAA programs 

d. ES/Wagner-Peyser participants co-enrolled in WIA Adult, WIA Dislocated 

Worker, WIA Youth, or, TAA programs 

e. TAA participants co-enrolled in WIA Adult, WIA Dislocated Worker, WIA 

Youth, or, ES/Wagner-Peyser programs 

5. Do you anticipate that patterns of co-enrollment will change for any of these programs 

over the next three years? 

6. How do you suggest that ETA deal with co-enrollment in its outcome and efficiency 

performance measures? 

B. Accounting for Program Costs 

1. Does the state receive WIA expenditure data from local workforce areas? If so, could 

this data be provided to the federal government with little additional cost to the state? 

If not, please explain. 

2. At the local workforce level, do LWIAs collect and disaggregate WIA costs by service 

level (e.g., core/unassisted, staff-assisted, intensive, and training services under 

WIA)? 

a. If so, how are costs disaggregated (by what service categories)? 

b. Do LWIAs across the state use the same methodology for disaggregating costs? 

c. Is it challenging for LWIAs to generate this disaggregated data? If so, what 

makes it difficult? 

d. Are these disaggregated costs reported to the state? And if yes, is the data useful? 

3. If disaggregated cost data (by service level) are not collected for the WIA program at the 

LWIB level, how difficult would it be to do so? What would be the challenges? 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 188 

4. To what extent are there shared costs across workforce programs, including WIA, 

Wagner-Peyser/ES, and TAA (e.g., where training costs for an individual are shared 

by WIA and TAA or where Wagner-Peyser/ES and WIA share the costs of One-Stop 

operations at the local level)? 

a. For which programs are there the most shared costs? 

b. Do local areas or the state collect any data on the sharing of costs between 

programs? If yes, please explain. 

c. How do you suggest that ETA deal with shared costs in its efficiency performance 

measures? 

C. Other Issues/Conclusion 

 

1. At what point do self-service customers of the One-Stop Career Center system get 

enrolled for performance purposes? 

a. Are procedures the same across the state or do they vary by local workforce area 

or One-Stop Center? 

b. How should these self-service customers be taken into account with regard to 

outcome and efficiency measures? 

2. Does your state have any issues with regard to exclusion of incumbent workers from 

outcome measures (such as entered employment rate) or efficiency measures (such as 

cost per entered employment)? Do you have any suggestions on how to deal with 

incumbent workers in measuring program outcomes and efficiency? 

3. In some instances (particularly in programs such as WIA and TAA), program participants 

are served over more than one program year (e.g., a WIA participant’s expenditures 

on training may be spread across two program years), while for performance and 

efficiency measurement purposes they are counted as an exiter for just one program 

year (i.e., the year they exit from the program). Does the spreading of costs across 

multiple program years pose any challenges with regard to measuring program 

efficiency? 

4. Has your state agency used Return on Investment (ROI) to measure the cost-effectiveness 

of WIA, Wagner-Peyser/ES, TAA, or other employment and training programs? If 

yes – 

a. Please discuss your use of and experiences with ROI – for which programs, how 

were impact results obtained, and who conducted the ROI study? 

b. Do you use ROI measures as part of the annual performance measurement process 

or in program evaluations? 

c. If your state uses ROI, have the ROI results been useful? If yes, how have the 

ROI results been used? 

5. Do you have any addition thoughts you would like to express about the use of efficiency 

measures to monitor employment and training programs – 

a. Challenges to gathering outcome or expenditure data to generate efficiency 

measure results 

b. Potential for agencies being monitored “gaming” results to the detriment of 

program goals/objectives 

c. Issues concerning comparing performance on efficiency measures across states, 

local workforce areas, and employment and training programs 

d. Potential benefits of efficiency measures in terms of enhancing program 

operations or outcomes over the long-term
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 189 

APPENDIX B: 

REFERENCES
APPENDIX B: REFERENCES 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 190 

Anderson, Kathryn, Richard Burkhauser, Jennie Raymond, and Clifford Russell. 1992. “Mixed 

Signals in the Job Training Partnership Act.” Growth and Change. 22(3): pp. 32-48. 

Anderson, Kathryn, Richard Burkhauser, and Jennie Raymond. 1993. “The Effect of Creaming 

on Placement Rates under the Job Training Partnership Act.” Industrial and Labor Relations 

Review. 46(4): pp. 613-624. 

Barnow, Burt S. 1992. “The Effects of Performance Standards on State and Local Programs: 

Lessons for the Job Opportunities and Basic Skills Programs.” In C. Manski & I. Garfinkel 

(Eds.), Evaluating Welfare and Training Programs (pp. 277-309). Cambridge, MA: Harvard 

University Press. 

Barnow, Burt S. 1996. "Policies for People with Disabilities in U.S. Employment and Training 

Programs," in Disabilities, Cash Benefits, and Work, Jerry L. Mashaw, Virginia Reno, Richard 

Burkhauser, and Monroe Berkowitz Eds. Kalamazoo, Michigan: Upjohn Institute for 

Employment Research. 

Barnow, Burt. 2000. “Exploring the Relationship between Performance Management and 

Program Impact: A Case Study of the Job Training Partnership Act.” Journal of Policy Analysis 

and Management. 19(1): pp. 118-141. 

Barnow, Burt S. and Jeffrey A. Smith. 2004. “Performance Management of U.S. Job Training 

Programs: Lessons from the Job Training Partnership Act.” Public Finance and Management. 

4(3): 247-287. 

Barnow, Burt S. and Christopher T. King. 2005. The Workforce Investment Act in Eight States. 

Albany, NY: The Nelson A. Rockefeller Institute of Government. 

Barnow, Burt S. and Carolyn Heinrich. 2009. “One Size Fits All? The Pros and Cons of 

Performance Standard Adjustments.” Johns Hopkins University and University of WisconsinMadison. 

Becker , Gary S. 1964. “Human Capital: A Theoretical and Empirical Analysis with Special 

Reference to Education.” New York: National Bureau of Economic Research. 

Blalock, Ann and Burt Barnow. 2001. “Is the New Obsession with ‘Performance Management’ 

Masking the Truth About Social Programs?” In Quicker, Better, Cheaper: Managing 

Performance in American Government. Ed. Dall Forsythe. Albany, NY: Rockefeller Institute 

Press, pp. 485-517. 

Blalock, Hubert M. Jr. 1961. Causal Inferences in Nonexperimental Research. Chapel Hill, NC: 

University of North Carolina Press. 

Bloom, Howard S., Charles Michalopoulos, Carolyn J. Hill, and Y. Lei. 2002. “Can 

Nonexperimental Comparison Group Methods Match the Findings from a Random Assignment 
APPENDIX B: REFERENCES 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 191 

Evaluation of Mandatory Welfare-to-Work Programs?” New York, NY: Manpower 

Demonstration Research Corporation 

Cook, Thomas D., William R. Shadish, and Vivian C, Wong. 2008. “Three Conditions under 

Which Experiments and Observational Studies Produce Comparable Causal Estimates.” Journal 

of Policy Analysis and Management. 27(4): 724-750. 

Courty, Pascal, and Gerald Marschke. 1996. “Moral Hazard under Incentive Systems: The Case 

of a Federal Bureaucracy.” In Advances in the Study of Entrepreneurship, Innovation and 

Economic Growth, Volume 7. Ed. Gary Libecap. Greenwich, CT: JAI Press, pp. 157-190. 

Courty, Pascal, and Gerald Marschke. 1997. “Measuring Government Performance: Lessons 

from a Federal Job-Training Program.” American Economic Review. 87(2): pp. 383-388. 

Courty, Pascal, and Gerald Marschke. 2004. “An Empirical Investigation of Gaming Responses 

to Explicit Performance Incentives.” Journal of Labor Economics. 22(1): pp. forthcoming22-56. 

Courty, Pascal, and Gerald Marschke. 2007. “Making Government Accountable: Lessons from a 

Federal Job Training Program.” Public Administration review 67(5): pp. 904-916. 

Deming, W. Edwards 1986. Out of the Crisis. Cambridge, MA: MIT Institute for Advanced 

Engineering Study. 

Dickinson, Katherin, et al. 1988. Evaluation of the Effects of JTPA Performance Standards on 

Clients, Services, and Costs. Washington, DC: National Commission for Employment Policy 

Research Report No. 88-15. 

Heckman, James, Carolyn Heinrich and Jeffrey Smith. 2002. “The Performance of Performance 

Standards.” Journal of Human Resources. 37(4): pp. 778-811. 

 

Heckman, James, and Jeffrey Smith. 2004. “The Determinants of Participation in a Social 

Program: Evidence from JTPA.” Journal of Labor Economics, forthcoming. 22(2): pp. 243-298. 

Heckman, James, Jeffrey Smith and Christopher Taber. 1996. “What Do Bureaucrats Do? The 

Effects of Performance Standards and Bureaucratic Preferences on Acceptance into the JTPA 

Program.” In Advances in the Study of Entrepreneurship, Innovation and Economic Growth, 

Volume 7. Ed. Gary Libecap. Greenwich, CT: JAI Press, pp. 191-218. 

Heinberg, John, J. Trutko, B. Barnow, M. Farrell, and A. Glosser. 2005. Unit Costs of Intensive 

And Training Services for WIA Adults and Dislocated Workers: An Exploratory Study of 

Methodologies and Estimates in Selected States and Localities: Final Report. Arlington, VA: 

Capital Research Corporation, prepared for the U.S. Department of Labor, Employment and 

Training Administration. 

Heinrich, Carolyn J. 2004. Improving Public-Sector Performance Management: One Step 

Forward, Two Steps Back? Public Finance and Management, 4(3), 317-351. 
APPENDIX B: REFERENCES 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 192 

House Research Organization, Texas House of Representatives. 2004. “Examining Teacher 

Performance Incentives.” Focus Report Number 78-17. 

Idaho. 2007. “Workforce Investment Act – Annual Report for PY 2006.” October 2007. 

Kemple, James, Fred Doolittle, and John Wallace. 1993. The National JTPA Study: Site 

Characteristics and Participation Patterns. New York, NY: Manpower Demonstration Research 

Corporation. 

Marschke, Gerald. 2002. “Performance Incentives and Bureaucratic Behavior: Evidence from a 

Federal Bureaucracy.” Working paper, Department of Economics, SUNY Albany, Albany, NY. 

Moore, Richard, Phillip Gorman, and Andrew Wilson. 2007. “California One-Stop System Cost 

Study Report.” Prepared for the California Workforce Investment Board by The California State 

University – Northridge. 

Orr, Larry L., Howard S. Bloom, Stephen H. Bell, Fred Doolittle, Winston Lin, and George 

Cave. 1996. Does Training for the Disadvantaged Work? Evidence from the National JTPA 

Study. Washington, DC: The Urban Institute Press. 

Radin, Beryl 2006. Challenging the Performance Movement. Washington, D.C.: Georgetown 

University Press. 

Phillips, Jack J. 2003. Return on Investment in Training and Performance Improvement 

Programs, Second Edition. New York: Butterworth Heinemann. 

Roderick, Melissa, Brian Jacob, and Anthony S. Bryk. 2000. “Evaluating Chicago’s Efforts to 

End Social Promotion.” In Governance and Performance: New Perspectives, eds. Lynn, L. and 

Heinrich, C. Washington, D.C.: Georgetown University Press. 

Rubenstein, Ross, Leanna Stiefel and Amy Ellen Schwartz. 2003. “Better than Raw: A Guide to 

Measuring Organizational Performance with Adjusted Performance Measures.” Public 

Administration Review 63(5): 607-615.

Schochet, Peter Z., John Burghardt, and Sheena McConnell. 2006. National Job Corps Study 

and Longer-Term Follow-Up Study: Impact and Benefit-Cost Findings Using Survey and 

Summary Earnings Records Data. Princeton, NJ: Mathematica Policy Research. 

Schochet, Peter Z. and John Burghardt. 2008. “Do Job Corps Performance Measures Track 

Program Impacts?” Journal of Policy Analysis and Management. 27(3): pp. 556-576. 

Shulock, Nancy B. 1999. The paradox of policy analysis: if it is not used, why do we produce 

so much of it? Journal of Policy Analysis and Management 18(2): 226-244. 

Schultz , T. W. 1961. “Investment in Human Capital.” American Economic Review. 51(1): 1-

17. 
APPENDIX B: REFERENCES 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 193 

Smith, Jeffrey and Petra Todd. 2005. “Does Matching Overcome LaLonde’s Critique of 

Nonexperimental Estimators?” Journal of Econometrics 125: 305-353. 

Social Policy Research Associates. 2004. The Workforce Investment Act after Five Years: 

Results from the National Evaluation of the Implementation of WIA. Oakland, CA: Social 

Policy Research Associates. 

Social Policy Research Associates. 1999. Guide to JTPA Performance Standards for Program 

Years 1998 and 1999. Menlo Park, CA: Social Policy Research Associates. 

U.S. Department of Labor, Office of Job Corps. 2001. Policy and Requirements Handbook: Job 

Corps. Washington, DC: U.S. Department of Labor, Office of Job Corps. Revised July 2007. 

U.S. Department of Labor, Employment and Training Administration. 2001. 2002 Annual 

Performance Plan for Committee on Appropriations. 

U.S. Department of Labor. 2002. Employment and Training Administration. Training and 

Employment Guidance Letter No. 11-01. “Guidance on Revising Workforce Investment Act 

(WIA) State Negotiated Levels of Performance.” Washington, DC: U.S. Department of Labor, 

Employment and Training Administration. 

U.S. Department of Labor. 2002. “Summary Report on WIA Implementation.” Washington, DC: 

Employment and Training Administration. 

U.S. Department of Labor. 2007. Employment and Training Administration. Training and 

Employment Guidance Letter No. 19-06. “Negotiating Performance Goals for the Workforce 

Investment Act Title 1B Programs and Wagner-Peyser Act Program for Program Years 2007 and 

2008.” Washington, DC: U.S. Department of Labor, Employment and Training Administration. 

U.S. General Accounting Office. 2002. Workforce Investment Act: Improvements Needed in 

Performance Measures to Provide a More Accurate Picture of WIA’s Effectiveness. 

Washington, DC: U.S. General Accounting Office. 

Washington State Training and Education Coordinating Board. 2005. Integrated Performance 

Information for Workforce Development: A Blueprint for States. Olympia, WA: Washington 

State Training and Education Coordinating Board. 

Workforce Enterprise Services, Inc. 2007. “Chicago Workforce Board Cost per Participant 

Study Final Report; Cost Per Participant Comparison Study - Phase Two; Final Report.” 

Wilde, Elizabeth T., and Robinson Hollister. 2007. “How Close Is Close Enough? Testing 

Nonexperimental Estimates of Impact Against Experimental Estimates of Impact with Education 

Test Scores as Outcomes.” Journal of Policy Analysis and Management. 26(3): 455-477. 
APPENDIX B: REFERENCES 

 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 194 

Zornitsky, Jeffrey, Mary Rubin, Stephen Bell, and William Martin. 1988. “Establishing a 

Performance Management System for Targeted Welfare Programs.” National Commission for 

Employment Policy Research Report 88-14. Washington, DC: National Commission for 

Employment Policy. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 195 

APPENDIX C: 

REGRESSION RESULTS FOR 

FOUR ADDITIONAL EFFICIENCY MEASURE OUTCOMES 

 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 196 

OLS * Fixed Effects ** OLS * Fixed Effects **

Average Exiters Per WIB -0.1450731 (0.449) -0.2359341 (0.387) -1.892358 (0.028) -0.6512025 (0.530)

Percent Training 2052.685 (0.384) -1359.554 (0.587) 10483.16 (0.003) 20328.4 (0.000)

Percent Intensive 7226.694 (0.002) 6963.288 (0.011) -8360.84 (0.251) -2935.347 (0.602)

Percent Core (Reference Group) ////

Percent Enrolled in Multiple WIA 

Programs -7512.523 (0.090) -10273.69 (0.065) 137.6309 (0.972) -7783.778 (0.063)

Percent Funded by Wagner Peyser 

Funding Sourse -1.725425 (0.997) -235.6306 (0.827) -1030.882 (0.343) -1882.616 (0.382)

Percent Age 0 to 21 640.0206 (0.945) -2619.409 (0.828) -83382.72 (0.269) -12737.61 (0.809)

Percent Age 22 to 29 (Reference Group) ////

Percent Age 30 to 44 -1714.479 (0.867) -7534.465 (0.449) 12146.99 (0.630) 60607.23 (0.013)

Percent Age 45 to 54 -828.56 (0.936) -5743.269 (0.634) -41435.46 (0.143) 34227.58 (0.102)

Percent Age 55 Plus 11225.34 (0.372) 26900.95 (0.064) 34001.46 (0.154) 49524.88 (0.083)

Percent White (Reference Group) ////

Percent Black 447.2234 (0.699) 7859.619 (0.174) 7126.506 (0.084) -22469.57 (0.205)

Percent Hispanic -777.1583 (0.743) 2273.36 (0.779) 4718.668 (0.402) -16126.61 (0.477)

Percent Other or Multiple Race 2086.279 (0.373) -10865.68 (0.335) 10158.98 (0.008) -22236.07 (0.290)

Percent Female -2657.746 (0.404) 3704.951 (0.470) -4934.869 (0.528) -11580.92 (0.286)

Percent Veteran 14906.98 (0.220) -12651.13 (0.470) 5600.369 (0.763) 5183.127 (0.805)

Percent Working at Registration -5431.899 (0.034) -490.7754 (0.895) 8694.082 (0.219) -5513.92 (0.635)

Percent Non High School Grad -7114.708 (0.124) -10268.56 (0.088) -30348.54 (0.005) -10626.48 (0.467)

Percent Limited English Proficiency 3849.221 (0.410) 4926.454 (0.560) 12949.24 (0.082) 9130.558 (0.400)

Percent TANF Receipt within 6 months 6994.897 (0.034) 3609.694 (0.646) / /

Average Unemployment Rate 

(non-seasonally adjusted) -13.8872 (0.948) -339.7294 (0.501) 739.249 (0.046) -3214.073 (0.006)

Percent Below Poverty Line -35.49007 (0.702) 277.5333 (0.381) 551.3832 (0.023) 977.0194 (0.148)

Average Weekly Retail Wage Rate -2.606886 (0.680) -11.34553 (0.590) 18.41033 (0.250) -0.1377437 (0.998)

Average Wkly Wage Production Workers 

in Manufacturing 3.344418 (0.156) -0.5937875 (0.931) 3.270214 (0.527) 11.54599 (0.413)

Constant 1476.348 (0.831) 6508.582 (0.613) -2290.841 (0.920) -30354.89 (0.290)

Observations 153 153 153 153

R-squared / Within State R-squared 0.4508 0.4024 0.398 0.3738

* P-values from Huber/White Variance Estimator (Robust)

** Standard Fixed Effects P-Value

Source: Combined 2005, 2006, and 2007 WIASRD Public Use Data Files, Authors' Calculations

Exhibit A-1: Estimated Adjustment Coefficients for Cost per Employed Exiter

WIA Adult WIA Dislocated Worker
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 197 

OLS * Fixed Effects ** OLS * Fixed Effects **

Average Exiters Per WIB -0.1496385 (0.412) -0.2393657 (0.365) -1.858962 (0.027) -0.6565914 (0.516)

Percent Training 1641.056 (0.471) -1284.741 (0.596) 10266.15 (0.003) 20012.81 (0.000)

Percent Intensive 7038.39 (0.002) 6928.979 (0.009) -8232.783 (0.252) -2661.057 (0.627)

Percent Core (Reference Group) ////

Percent Enrolled in Multiple WIA 

Programs -7008.421 (0.085) -10837.75 (0.044) 381.9988 (0.921) -7542.773 (0.064)

Percent Funded by Wagner Peyser 

Funding Sourse 14.33322 (0.978) -103.1592 (0.921) -999.7519 (0.343) -1975.971 (0.347)

Percent Age 0 to 21 -1012.787 (0.910) -733.1558 (0.950) -75841.06 (0.303) -3275.234 (0.949)

Percent Age 22 to 29 (Reference Group) ////

Percent Age 30 to 44 -2633.594 (0.791) -7864.054 (0.414) 15255.6 (0.536) 61363.01 (0.010)

Percent Age 45 to 54 -2758.183 (0.775) -3436.303 (0.769) -39607.2 (0.151) 33886.25 (0.097)

Percent Age 55 Plus 8121.733 (0.490) 23183.46 (0.098) 38531.77 (0.103) 51708.19 (0.064)

Percent White (Reference Group) ////

Percent Black 597.2213 (0.602) 7562.116 (0.176) 7331.326 (0.068) -20615.97 (0.233)

Percent Hispanic -272.7357 (0.902) 1715.885 (0.827) 4646.275 (0.395) -16482.6 (0.456)

Percent Other or Multiple Race 2323.307 (0.295) -11397.72 (0.296) 9871.427 (0.008) -21829.1 (0.287)

Percent Female -2457.743 (0.404) 3874.544 (0.436) -5572.689 (0.464) -12486.51 (0.238)

Percent Veteran 13768.41 (0.232) -12011.25 (0.478) 4480.081 (0.806) 3835.389 (0.851)

Percent Working at Registration -4423.167 (0.077) 135.8916 (0.970) 9005.33 (0.194) -4641.471 (0.682)

Percent Non High School Grad -7233.268 (0.106) -9954.624 (0.087) -29816.42 (0.005) -10064.74 (0.479)

Percent Limited English Proficiency 2692.948 (0.528) 5208.405 (0.524) 12903.99 (0.080) 9480.308 (0.370)

Percent TANF Receipt within 6 months 6536.823 (0.041) 3110.306 (0.683) / /

Average Unemployment Rate 

(non-seasonally adjusted) -0.8387973 (0.997) -315.9036 (0.518) 686.4986 (0.058) -3266.112 (0.004)

Percent Below Poverty Line -40.0346 (0.656) 268.0356 (0.382) 541.9747 (0.021) 896.4433 (0.173)

Average Weekly Retail Wage Rate -1.894882 (0.754) -10.79258 (0.596) 17.12215 (0.273) -0.9207573 (0.984)

Average Wkly Wage Production Workers 

in Manufacturing 3.558209 (0.117) -0.3296395 (0.960) 3.572832 (0.479) 11.36247 (0.409)

Constant 2038.036 (0.762) 5609.81 (0.652) -3969.568 (0.859) -29337.14 (0.294)

Observations 153 153 153 153

R-squared / Within State R-squared 0.4537 0.4148 0.4012 0.377

* P-values from Huber/White Variance Estimator (Robust)

** Standard Fixed Effects P-Value

Source: Combined 2005, 2006, and 2007 WIASRD Public Use Data Files, Author's Calculations

Exhibit A-2: Estimated Adjustment Coefficients for Adjusted Cost per Entered Employment 

from Common Measure

WIA Adult WIA Dislocated Worker
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 198 

OLS * Fixed Effects ** OLS * Fixed Effects **

Average Exiters Per WIB -0.2105902 (0.357) -0.3236129 (0.340) -2.456632 (0.011) -0.9343323 (0.422)

Percent Training 3038.862 (0.293) -1124.925 (0.717) 11723.83 (0.002) 23209.08 (0.000)

Percent Intensive 8560.389 (0.003) 8051.29 (0.017) -11115.67 (0.179) -5116.314 (0.418)

Percent Core (Reference Group) ////

Percent Enrolled in Multiple WIA 

Programs -10473.07 (0.059) -12962.21 (0.061) 586.6261 (0.893) -7725.39 (0.099)

Percent Funded by Wagner Peyser 

Funding Sourse -0.3273392 (1.000) -183.097 (0.892) -1082.723 (0.354) -1914.306 (0.428)

Percent Age 0 to 21 5143.251 (0.648) -2379.268 (0.874) -73882.11 (0.359) -1084.459 (0.985)

Percent Age 22 to 29 (Reference Group) ////

Percent Age 30 to 44 -909.1746 (0.939) -8054.669 (0.515) 10063.79 (0.717) 68144.74 (0.013)

Percent Age 45 to 54 -277.3029 (0.983) -7372.969 (0.623) -43127.02 (0.162) 39949.59 (0.089)

Percent Age 55 Plus 20997.73 (0.187) 44793.45 (0.014) 36962.47 (0.140) 54650.68 (0.088)

Percent White (Reference Group) ////

Percent Black 1425.821 (0.289) 10875.51 (0.131) 9270.387 (0.038) -22228.27 (0.263)

Percent Hispanic -1512.532 (0.600) 2363.521 (0.815) 5562.095 (0.360) -18687.96 (0.463)

Percent Other or Multiple Race 2738.486 (0.340) -13523.63 (0.335) 11985.57 (0.005) -27116.07 (0.250)

Percent Female -5244.098 (0.195) 6097.48 (0.340) -4662.428 (0.580) -9304.186 (0.444)

Percent Veteran 19671.51 (0.182) -14232.17 (0.513) 7407.905 (0.709) 6416.627 (0.785)

Percent Working at Registration -7454.713 (0.018) -1283.888 (0.782) 8518.633 (0.268) -10226.2 (0.433)

Percent Non High School Grad -6567.596 (0.228) -12436.72 (0.096) -32318.37 (0.007) -11168.98 (0.495)

Percent Limited English Proficiency 5788.697 (0.329) 7165.305 (0.495) 15757.79 (0.060) 9385.012 (0.440)

Percent TANF Receipt within 6 months 9822.337 (0.016) 6499.958 (0.506) / /

Average Unemployment Rate 

(non-seasonally adjusted) 34.42382 (0.895) -430.7413 (0.493) 872.0695 (0.031) -3458.574 (0.008)

Percent Below Poverty Line -76.236 (0.479) 352.3973 (0.371) 584.1576 (0.025) 1141.866 (0.132)

Average Weekly Retail Wage Rate -3.301931 (0.658) -18.66472 (0.476) 22.11306 (0.197) -10.40512 (0.840)

Average Wkly Wage Production Workers 

in Manufacturing 3.017922 (0.279) -3.552083 (0.675) 3.729429 (0.497) 12.63215 (0.424)

Constant 1682.387 (0.835) 9209.127 (0.565) -2383.203 (0.925) -32092.94 (0.318)

Observations 153 153 153 153

R-squared / Within State R-squared 0.4567 0.4078 0.4265 0.3622

* P-values from Huber/White Variance Estimator (Robust)

** Standard Fixed Effects P-Value

Source: Combined 2005, 2006, and 2007 WIASRD Public Use Data Files, Authors' Calculations

Exhibit A-3: Estimated Adjustment Coefficients for Cost per Employment Retention 

Alternative Measure

WIA Adult WIA Dislocated Worker
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 199 

OLS * Fixed Effects ** OLS * Fixed Effects **

Average Exiters Per WIB -0.0000223 (0.352) -0.0000192 (0.652) -0.000183 (0.016) -0.0000605 (0.580)

Percent Training 0.0902101 (0.785) -0.6395018 (0.104) 0.9557113 (0.002) 1.953429 (0.001)

Percent Intensive 0.8320392 (0.010) 1.502651 (0.001) -0.7713247 (0.215) -0.1629988 (0.783)

Percent Core (Reference Group) ////

Percent Enrolled in Multiple WIA 

Programs -1.176685 (0.019) -1.903192 (0.029) -0.2606286 (0.432) -0.7723066 (0.079)

Percent Funded by Wagner Peyser 

Funding Sourse 0.0286506 (0.696) -0.1033871 (0.540) -0.111083 (0.267) -0.103475 (0.648)

Percent Age 0 to 21 -0.2725275 (0.833) -1.515343 (0.422) -9.095301 (0.274) -4.75703 (0.393)

Percent Age 22 to 29 (Reference Group) ////

Percent Age 30 to 44 -1.062277 (0.451) -2.832725 (0.070) -0.2962484 (0.887) 4.885902 (0.055)

Percent Age 45 to 54 0.7442227 (0.569) -0.8749846 (0.643) -4.539023 (0.066) 3.205812 (0.145)

Percent Age 55 Plus -0.028188 (0.986) 2.022709 (0.368) 1.754984 (0.422) 5.081829 (0.091)

Percent White (Reference Group) ////

Percent Black 0.0448645 (0.770) 0.0398775 (0.965) 0.4348713 (0.242) -1.309235 (0.482)

Percent Hispanic -0.1090298 (0.728) 0.2869463 (0.821) 0.222541 (0.698) -2.202599 (0.357)

Percent Other or Multiple Race 0.0873886 (0.752) -0.751808 (0.669) 0.8409738 (0.008) -0.5931302 (0.788)

Percent Female -0.0688709 (0.851) 0.1790668 (0.823) 0.2984367 (0.648) -1.615071 (0.159)

Percent Veteran 0.8938205 (0.560) 0.0318955 (0.991) 0.4164445 (0.824) -0.4140949 (0.851)

Percent Working at Registration -0.7928779 (0.010) -0.441624 (0.448) 0.4718814 (0.468) -0.5453322 (0.655)

Percent Non High School Grad -0.7251118 (0.255) -0.5317346 (0.568) -2.186094 (0.022) -1.396955 (0.364)

Percent Limited English Proficiency 0.4715261 (0.450) 0.6066981 (0.645) 1.415817 (0.062) 1.250804 (0.275)

Percent TANF Receipt within 6 months 0.9336879 (0.021) 0.0245711 (0.984) / /

Average Unemployment Rate 

(non-seasonally adjusted) -0.0288937 (0.256) -0.1072351 (0.176) 0.0319428 (0.370) -0.3189647 (0.009)

Percent Below Poverty Line 0.0024626 (0.837) -0.0177057 (0.720) 0.0667353 (0.004) 0.0753364 (0.288)

Average Weekly Retail Wage Rate -0.0006688 (0.447) 0.0019258 (0.558) 0.0012666 (0.419) 0.0050897 (0.295)

Average Wkly Wage Production Workers 

in Manufacturing 0.0009119 (0.009) 0.0006445 (0.545) 0.000609 (0.266) 0.0019896 (0.182)

Constant 0.3975948 (0.685) 0.7697051 (0.701) 0.3825446 (0.838) -5.181015 (0.088)

Observations 153 153 153 153

R-squared / Within State R-squared 0.4224 0.3895 0.3512 0.4277

* P-values from Huber/White Variance Estimator (Robust)

** Standard Fixed Effects P-Value

Source: Combined 2005, 2006, and 2007 WIASRD Public Use Data Files, Authors' Calculations

Exhibit A-4: Estimated Adjustment Coefficients for Cost per Total Post Program Earnings 

Alternative Measure

WIA Adult WIA Dislocated Worker
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 200 

APPENDIX D: 

WRITTEN COMMENTS PROVIDED 

BY EXPERT PANEL FOLLOWING 

NOVEMBER 2009 PANEL MEETING 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 201 

Beryl Radin 

American University 

Written Comments 

ETA Expert Panel Meeting 

November 9, 2009

Unlike Chris and Carolyn, I have limited expertise on the detailed process and 

expectations of creating performance measures in this set of programs. 

However, I am somewhat familiar with the pattern of modifications that have occurred in 

this policy area since the 1960s and the reality that the design of programs has been changed and 

priorities shifted as changes have occurred in the larger political and policy environment. 

It seems to me that this evaluation is facing such a transition reality. Its important to 

acknowledge this because the programs may find themselves trapped in a very different policy 

context and political expectations particularly from OMB. 

First, the policy shifts. We are in the midst of a very different economic climate today 

than when the project began. I don’t think it is productive to proceed with the assumption that 

things are the same today as when the project started. I’m especially concerned about the 

relationship between these programs and a range of programs within the stimulus package. A 

new set of programs and agencies are now charged with job creation. Thus I think that the 

performance measures should find a way to link the DOL programs with those in other agencies. 

Actually that linkage may make the 11 programs in this study more visible and relevant. 

 

Second are the shifts in OMB and the current administration. This occurs in two ways. 

The first deals with policy expectations that may change as a result of the change of 

administration. Some of the programs (particularly the apprenticeship programs) are likely to 

provide a different set of values related to the balance between the private sector and unions. 

There also may be a different approach to the concept of efficiency measures as opposed to 

effectiveness and equity measures. In addition, there may be more emphasis on the differences 

between the client groups that are expected to be served by the various programs. 

The second deals with the OMB expectations about performance measurement. It is not 

clear to me how many changes will occur from the past; there are different signals from Orszag 

and Zients. My sense is that whatever is created is likely to provide more discretion to agencies 

and programs to define measures that are useful to program managers and to more clearly 

acknowledge that managers do not always have control over the achievement of real outcomes. 

There is likely to be less control over details by OMB than occurred in the past 

administration. Also I don’t know whether there may be more acknowledgement of the 

importance of outputs and processes as well as outcomes. Is it possible now to emphasize 

different levels of analysis -- assessment of performance by individuals as well as applicants? I 

think OMB is less likely to impose a national model as was communicated by PART. 

Information about the international experience in the report indicates that the past 

administration’s emphasis on efficiency measures is not the international norm. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 202 

There are several other issues. It is important to acknowledge differences among the 

programs (both in terms of clients as well as delivery mechanisms). Should we assume that the 

same delivery structures are effective across programs? I have been concerned about the 

incompatibility between some of the performance expectations and the reality of federalism and 

differences across jurisdictions. Population groups experience problems differently in different 

political cultures. 

Finally, the purpose of the performance measurement activities is not clear to me. The 

determination of who we expect to use the measures should be a first order issue. Is it 

envisioned as a way to kill programs? To shift annual budgetary processes? To provide 

information to congressional oversight efforts? Or to give federal managers (or state and local 

managers) better information about ways that they can modify their practices? There are 

different uses depending on the level of government involved. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 203 

Christopher T. King 

Ray Marshall Center, University of Texas at Austin 

Written Comments 

ETA Expert Panel Meeting 

November 9, 2009

Per your request, I’m offering brief written comments in addition to the ones I provided 

in yesterday’s Expert Panel Meeting in Washington, D.C. My comments are organized into 

three areas: general comments, recommendations, and suggestions for future research/analysis. 

GENERAL COMMENTS

As I mentioned in the Expert Panel Meeting, I think the context for implementing efficiency 

measures should be carefully considered whether USDOL decides to move forward with 

standards with “teeth”, simple measures rather than standards, or just information guidelines. A 

few aspects in particular bear noting: 

• It has been years since any type of efficiency standard or measure has been used in US 

workforce development programs, I believe since the 1992 JTPA Amendments that 

responded to the adverse impact findings from the 1988 SRI study. Implementing them 

would necessitate a considerable lead time in order to be done right and to avoid large 

unintended consequences. 

• The environment for US workforce development programs has become far more varied 

and complex since the mid-to-late 1990s, as described in Barnow and King (2005), 

Grubb et al. (2005) and others. In some states, due to legislative reforms, local WIBS 

operate comprehensive, highly integrated programs that can offer participants services 

from WIA, ES and TAA, as well as from TANF, FSE&T and even childcare. In others, 

WIBS can only offer what WIA provides and supports. Co-enrollment, leveraging and, 

of course, accurate measurement of all key inputs is far more likely in the former than the 

latter. This is dicey territory for implementing any type of efficiency measure. 

Unintended consequences are all but guaranteed. 

• WIA also presents an environment that is far more susceptible to “creaming” than JTPA 

for several reasons, including: universal services eligibility, far more ambiguous data 

definitions (e.g., of participant, of exiter, of intensive v. training services), coupled with 

greater consequences for enrolling participants in some services (i.e., staff-assisted core, 

intensive and training) v. others (unassisted core services). 

• In recent decades, US policy in both welfare-employment and workforce development 

has over-emphasized on-the-cheap, work-first strategies over longer-term or more 

intensive skill-building (human capital) ones. Funding for these programs has also been 

on the decline for much of the last decade. Recently, the American Recovery and 

Reinvestment Act (ARRA) began funding and encouraging states and localities to 

support longer-term training for participants. Given what happened in the much simpler 

JTPA program under the efficiency measure (with adjustment models and better data 

collection and reporting), implementing an efficiency standard (measure) now would 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 204 

almost certainly lead to both greater “creaming” and reduced emphasis on more costly 

training strategies, directly countering the current policy thrust/intent. 

RECOMMENDATIONS

My recommendations on the various topics of interest are as follows: 

• Efficiency standards should not be implemented for any of USDOL’s workforce 

programs. If something must be implemented, I recommend going with efficiency 

measures or even guidelines, with an ample supply of caveats and an extended discussion 

of program context as referenced above. 

• Expenditures should serve as the basis for any efficiency standard, measure or guideline. 

Neither appropriations or obligations accurately and consistently reflect the actual 

resources invested in a participant; expenditures do. 

• USDOL should review its data definitions, data collection and reporting procedures 

thoroughly for its workforce development programs with an eye to instituting far more 

consistent, tighter data systems generally. Despite the assurances of some who have 

examined and help to create these data, even a cursory review indicates that the problems 

with these data are considerable. The range of values, the within- and between-year 

variation and other aspects of the reported data defy rational explanation. Real 

improvements need to be made with greater involvement of policy and program as well 

as the usual technical/IT staff. 

• If efficiency standards/measures/guidelines are to be instituted, several of USDOL’s 

“program” line-up should be explicitly excluded; namely, Workforce Incentive Grants, 

National Emergency Grants and Apprenticeship. For various reasons, none of these 

“programs” would be appropriate for an efficiency measure of any sort. WIGs lack 

participant services altogether. NEGs are episodic. Apprenticeship is more a marketing 

effort by USDOL/States and doesn’t invest in training services. 

• Return-on-investment (ROI) measures should certainly be considered but only as 

periodic, evaluative performance measures, not standards. The IPI Blueprint Report 

(Wilson, 2006) was clear about this as is Washington State’s legislation that has required 

gross outcome measurement for all of its workforce training and education programs 

since the early 1990s and cost-effectiveness (ROI) analysis based on quasi-experimental 

impact estimation every four years (see Hollenbeck and Huang 2006). This has worked 

very well in Washington and gained considerable credibility across their system in ways 

not seen anywhere else. In Texas, the state association of workforce boards, not the state 

workforce agency, pushed for and supported the ROI analysis, which we conducted at the 

state and WIB level (King et al. 2008, available on http://www.utexas.edu/research/cshr ). 

SUGGESTIONS FOR FUTURE RESEARCH

Areas suggested for future research include: 

• USDOL should conduct a study of comprehensive/portfolio v. siloed workforce 

development policy and program contexts and outline their implications for performance 

measurement and management. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 205 

• USDOL, along with USDOE, USHHS and other affected federal agencies, should also 

work with OMB to thoroughly revisit the “common measures.” The OMB measures 

were created in a tops-down environment after which they were shared with states for 

comments. The process would be greatly improved by following the IPI approach of 

working directly with the states and localities from the outset and consulting with the 

National Governors Association, Washington State (Bryan Wilson) and others including 

myself. The Ray Marshall Center was a consultant to the IPI effort from 2003-2006. 

Several of the issue papers we prepared for NGA and the participating states are still 

available on our website. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 206 

Comments on Reports and Discussion of Implementing Outcomes-Based 

Efficiency Measures for Employment and Training Programs 

Carolyn J. Heinrich, Expert Panel Member 

The comments provided below reflect both my review of the reports (interim reports #3 

and #4 and the draft final report) and the discussion at the November 9, 2009 meeting at the 

USDOL. I begin with a set of general comments on the various issues that were raised in the 

reports and the November 9 meeting, and I subsequently list some more specific comments and 

recommendations for next steps in the process of designing efficiency measures for USDOL 

programs. 

General comments 

First, let me state up front how important I think it is that the USDOL is exploring 

outcomes-based efficiency measures, at a minimum for internal use. It is clear that the public is 

interested in accountability for program costs in conjunction with program performance. For 

example, very large average labor market impacts could likely be produced if employment and 

training program managers spent their budgets on a very limited number of clients. In this 

regard, efficiency will always be an important consideration in the effective use of public 

program resources. 

It is also apparent from the November 9 discussion and the wide range of complex issues 

addressed in the interim and final draft reports that the design and use of efficiency measures 

should be approached with caution and a thorough, careful investigation and analysis of the 

existing administrative data and performance measurement systems. This includes both current 

systems and past performance measurement efforts. For example, a cost per placement measure 

was used under JTPA, and there is published research and other studies that document the 

challenges with the use of such an outcomes-based efficiency measure. 

One unambiguous generalization that we can make from the existing research is that 

employment and training program managers and staff will respond to performance measures 

once they are established by the USDOL, regardless of whether sanctions or rewards are attached 

to meeting them. “What matters is what is measured” has been the mantra of performance 

measurement systems across a variety of public program domains. Unfortunately, once a 

performance measure is activated, those under pressure to meet it will frequently use all means 

that they can to influence measured performance, even if those actions do not benefit the clients. 

For example, with the introduction of the JTPA cost per placement measure, some local service 

delivery areas shifted away from providing more costly intensive services or those less likely to 

produce quick job placements in order to minimize costs and maximize costs per placement. As 

it is easier for program managers to control costs (than to affect the wage or earnings per entered 

employment), they are even more likely to manipulate an efficiency measure. 

In addition, we know from the analyses conducted by Trutko and Barnow in their reports 

that there is considerable variation in how program costs are currently calculated at both state 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 207 

and local service delivery levels. This variation and/or the lack of consistency in how costs are 

recorded and calculated leaves greater room for states and localities to manipulate an efficiency 

measure. I would suggest (and I think there was some consensus on this point in the November 

9 meeting) that there is considerable more work to be done in determining how states and 

Workforce Investment Boards (WIBs) are calculating costs and in developing guidelines or 

directives that would ensure greater consistency in the reporting of program costs. I will outline 

some specific suggestions on this issue below. 

Another important issue to consider in a potential move to the use and reporting of 

efficiency measures is the appropriate level of analysis. Currently, WIA performance outcomes 

are reported at the state level, but we know that particularly in the case of efficiency measures 

(factoring in program costs), there is important and valuable variation in the data that gets lost in 

aggregation. How does the USDOL want to use these data? There may be less buy-in or interest 

in doing this right on the part of the states if they see this as just one more reporting requirement. 

However, for this information to be useful for program management and performance 

improvement, it is almost essential to look at program costs closer to the point of service delivery 

(at the WIB level). 

In the initial examination of prospective efficiency measures conducted by Trutko and 

Barnow, we see substantial variation across states in measured (efficiency) performance, and 

there is also variation across states in the relationship of one measure to another. For example, 

for some states, there is very little difference in the average cost per entered employment and the 

average cost per retained employment, while for others, the cost per retained employment is 

twice or more the cost per entered employment. What explains these variations—are they 

accounting differences or real differences in the cost of placing people into jobs vs. retaining 

them in employment in some states? At this time, we do not have the data and analysis in hand 

that can help us understand these differences in a way that would allow the USDOL to move 

forward with fair and accurate efficiency measures. 

In the November 9 discussion, there was a clear consensus that costs should be measured 

in the form of expenditures (rather than appropriations, allocations or obligations). Exactly 

which expenditures should be counted and how they could be accurately accounted for (e.g., by 

type of service, fixed vs. variable costs, over time, etc.) was a subject of debate (on which I 

further comment below). However, there was also important discussion about whether the 

current common performance measures provide the best numerator for these measures across 

programs, and thus, this should also be given further consideration as the USDOL explores the 

design and use of efficiency measures. 

In my final general comments, I would like to emphasize how important I think it will be 

for the USDOL to have an adequate period of exploration and learning about possible efficiency 

measures before moving to implementation across states and/or WIBs. We will get better 

information on the true program costs and their relationship to other program activity and 

outcome measures before any performance measures are activated. And even more importantly, 

before any efficiency measures are placed into use, it will be very important to have the 

components measured and reported consistently across states and WIBs. The reports by Trutko 

and Barnow demonstrate that there are currently important inconsistencies and vagaries in the 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 208 

measures for which we do not have clear explanations, and going forward with such noisy/messy 

data in efficiency measure calculations could only exacerbate the difficulties already noted 

above. 

Specific comments in response to the reports and November 9 discussion 

I am organizing these specific (brief) comments according to the 15 questions that John Trutko 

and Burt Barnow set forth for discussion at the November 9 meeting. 

1. Should ETA implement efficiency measures for all 11 ETA programs? 

In the meeting, we discussed why the application of efficiency measures may be different for the 

various programs, particularly WIG, SCSEP, NEG and Apprenticeship programs. However, at 

the exploratory stage in which the measurement of costs is further considered, I would 

recommend looking at these programs, even if it is later determined that efficiency measures for 

them will tell us little. 

2. What specific measures should be implemented and should they be the same or different 

across the 11 programs? 

In principle, fewer measures and common measures may be most desirable. However, if the 

efficiency measures created are to be of value to program managers and most informative for the 

USDOL, it may be necessary to have different measures for some of the programs. For example, 

we discussed the possibility of using earnings change measures for incumbent worker programs 

in addition to the other common measures. 

3. Is it feasible/practical/desirable/cost-effective to calculate efficiency measures at the service 

activity level? 

I think there was some consensus at the meeting that it would be very desirable for program 

managers and the USDOL to have efficiency measures by program activity, particularly for core, 

intensive and training activities in the adult program. The feasibility of doing this will require 

more investigation. Separating out training would be the easiest, given the voucher cost. We 

also discussed the desirability of measures for self-services. As self-services are frequently not 

counted by WIBs, the fixed costs of providing these services are distributed across the other 

programs, inflating those service costs and the overall program cost. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 209 

4. What definition of cost should be used in calculating efficiency measures? 

There was strong consensus at the meeting that expenditures (rather than appropriations, 

allocations, or obligations) should be used. Both fixed expenditures (e.g., of the One-Stop 

Center infrastructure) and variable costs (e.g., personnel time) need to be considered. 

5. Should cost of items such as need-based payments, stipends, allowances and subsidized wages 

be counted as costs when calculating efficiency measures? 

Yes, these items are real costs associated with producing outcomes and should be included in 

efficiency measure calculations. For example, if providing a stipend makes the difference in 

whether a client completes training and gets a job, it needs to be measured and included. It will 

be of interest for the USDOL to use such information on these costs (and how they vary across 

states/WIBs) to assess the value added of these supports to training services and outcomes. 

6. How should the ETA account for co-enrollment in implementing efficiency measures? 

The analysis by Trutko and Barnow clearly showed that differences across states in coenrollment practices have important implications for measured program efficiency. The USDOL 

should conduct further research to better understand the reasons underlying these differences—

e.g., whether they are mainly accounting or related to real substantive differences in service 

approaches—and also wait for these practices to stabilize across states in consistent patterns or 

approaches. 

7. How should the ETA deal with shared costs across programs? 

Shared costs are important to factor in as well if the USDOL is to get more accurate estimates of 

program efficiency. The memorandums of understanding (MOUs) established in the WIBs 

would provide possibly the best guidance for assessing cost sharing, although this would be at 

the WIB level, and some of this useful information would be lost in a state-level measure. If the 

WIB’s cost-sharing practices are not in line with the MOUs, this might provide an impetus for 

modifications to more accurately reflect program cost-sharing. 

8. How should ETA deal with the issue of customers being served by programs for longer than 

one program year? 

If efficiency measures are computed annually, the costs contributed by each participant should be 

incorporated in a given year, even though they won’t enter the numerator (of the efficiency 

measure) until the year the outcome is measured. As long there is not substantial year-to-year 

variation in these relationships (between the timing of expenditures and outcome measurement), 

this shouldn’t matter greatly in the aggregate. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 210 

9. How should self-service customers be taken into account in efficiency measures? 

A separate measure for self-services would be useful to reflect the infrastructure investments that 

also benefit One-Stop and on-line system users (as this group of users is primarily accessing 

fixed-cost program resources). This would lower the overall program costs and more accurately 

reflect the services/benefits that are provided through the system. 

10. How should ETA deal with incumbent workers in calculating efficiency measures? 

A separate earnings change measure should be added to the cost per entered employment 

efficiency measure (instead of the earnings divided by costs measure). This should more 

accurately reflect the value added of these programs, but also the tradeoff of allocating resources 

to those who are already employed (by keeping cost per entered employment). 

11. Should statistical models be developed and used to adjust efficiency standards? 

Statistical models for making performance adjustments will only be useful if applied to WIBlevel data. There simply isn’t sufficient variation at the state level to identify the relationships of 

client, service and environmental factors to efficiency outcomes. In addition, once efficiency 

measures are activated, these relationships may become stronger as program managers and staff 

change behavior in response to them. Therefore, one would have to expect to modify these 

models over time as these relationships change. 

12. Are there alternative efficiency measures that ETA should consider? 

An earnings change measure that used the fifth quarter of earnings prior to program entry as the 

pre-program (baseline) measure would be a valuable addition to the set of common measures that 

are intended to approximate program impacts. 

13. Should a return on investment measure be considered a suitable outcome-based efficiency 

measure? 

A ROI measure is really only plausible/justifiable with a rigorous program impact measure, that 

is, one generated by a randomized experimental evaluation. I would recommend that this be 

calculated with the currently planned WIA random assignment impact evaluation, and that this 

be brought to the attention of those evaluation designers now if this has not already been done. 

14. Should efficiency measures have some incentive/sanction policies as outcome measures? 

There was a strong consensus in the November 9 meeting that we are not ready for incentives or 

sanctions to be tied to efficiency measures, and for some of the reasons noted above (e.g., the 

ease with which these measures can be gamed and some of the challenges in accurately 

accounting for costs), this may never be desirable. There is considerable more research and 

investigation to be done before further considering policies that would tie consequences to these 

measures. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 211 

15. What additional research should be conducted to develop efficiency measure policies? 

My suggestions for additional research have largely been incorporated in the above discussion 

and include the following: 

• How to separately account for costs by program activity; 

• Distinguishing fixed, variable and shared (infrastructure and service) costs and taking 

into account self-service costs; 

• What underlies differences in co-enrollment practices (e.g., accounting vs. substantive 

practices); 

• Improving common measures (that are the numerator of efficiency measures), such as a 

new and improved earnings change measure; 

• Performance adjustment models with WIB-level data and applied to a refined set of 

efficiency measures, and 

• Exploration of approaches to encouraging state and WIB “buy-in” and greater use of 

efficiency measures for program improvements. 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 212 

APPENDIX E: 

WRITTEN COMMENTS PROVIDED 

BY EMPLOYMENT AND TRAINING PROGRAM OFFICES 

IN RESPONSE TO THE DRAFT FINAL REPORT AND 

AUTHORS’ RESPONSES TO COMMENTS PROVIDED
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 213 

APPENDIX E: 

WRITTEN COMMENTS PROVIDED 

BY EMPLOYMENT AND TRAINING PROGRAM OFFICES 

IN RESPONSE TO THE DRAFT FINAL REPORT AND 

AUTHORS’ RESPONSES TO COMMENTS PROVIDED100

PROGRAM ETA PROGRAM OFFICE COMMENTS AUTHORS’ RESPONSE 

Division of 

Workforce System 

Support 

The Division of Workforce System Support has 

reviewed the document. It appears that the document is 

primarily concerned with recipients of evaluated 

programs and discusses cost per participant around 

training. The discussed efficiency measures 

recommended for consideration in this report (cost per 

entered employment, cost per retained in employment, 

cost divided by post-program (average) earnings, and 

cost divided by change in earnings) do not seem to 

apply to the work we do in our Division. 

No change necessary to the 

Final Report. 

Quick comments: 

 

There is one recommendation specifically regarding 

apprenticeship. OA suggest minor technical edits to 

this recommendation. See correction below: 

 

p. 7 -Recommendation #3a: ETA should consider 

alternative efficiency measures for the Apprenticeship 

Program linked to the goals of the program and what 

federal funds are being spent on – for example, 

increasing the number of apprenticeships offered, 

building the quality of apprenticeship programs, and 

registering and monitoring of Apprenticeship programs 

accurately and in a timely manner. Therefore, ETA 

should consider applying the following alternative 

efficiency measures to the Apprenticeship program (all 

at the national level): cost per additional 

apprenticeship program registered and timeliness of 

registration decisions. Specific measures should be 

adopted after appropriate dialogue and analysis is 

undertaken. 

 

Recommendation #3a in the 

Executive Summary and 

Chapter 8 of the Final Report 

was slightly revised as 

suggested, replacing certifying 

with “registering.” 

Office of 

Apprenticeship 

Three other recommendations also pertain to 

apprenticeship: p. 5-6 - Recommendations to exclude 

Apprenticeship from the cost per entered employment, 

cost per employment retention, and cost per postprogram average earnings measures. OA agrees with 

No change necessary to the 

Final Report. 

 

100Italics indicates comments that resulted in revisions made to the Final Report (in column 2) and specific revisions 

that were made to the report (in column 3). 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 214 

PROGRAM ETA PROGRAM OFFICE COMMENTS AUTHORS’ RESPONSE 

this recommendation. 

Youth DYS has reviewed the paper and has no specific 

comments on the paper itself. It was very thorough and 

informative. Our one comment, which we have made 

in the past, is that we do not agree with the 

recommendation to use an outcome based efficiency 

measure for the WIA Youth program. Given the longterm nature of services for the WIA youth program, a 

cost per placement measure does not take into account 

that youth often participate in the program for greater 

than one year and does not provide an accurate picture 

of the program. If this measure were ever implemented 

beyond the federal level, it would definitely lead to 

creaming and shorter-term services. A cost per 

participant measure is a better measure for the WIA 

youth program. 

No change necessary to the 

Final Report. We 

recommended using cost per 

placement in education or 

training as Youth measure 

(which is based on the 

Common Measure outcome 

applicable to the Youth 

program). We talk a lot in the 

report about the potential 

problem of “creaming” and 

incentives to provide shortterm and less costly services. 

We do not think cost per 

participant is a better measure 

– it is a measure that could be 

collected, but is not outcomebased. 

ONR agrees with an incremental approach to 

introducing efficiency measures, introduced to grantees 

first as only a monitoring tool rather than an outcome 

measure. 

No change necessary to the 

Final Report. 

ONR shares the concern about 

unanticipated/unintended behavioral changes on the 

part of grantees seeking to improve their efficiency 

measures. NEG participants may include hard-to-serve 

populations, and for Disaster NEG projects may 

include long-term unemployed, so there could be a 

disincentive to provide services to these populations 

with an efficiency-based measure. Also, because NEGs 

primarily serve dislocated workers retraining is often 

required, which is a more costly service than WagnerPeyser labor exchange or WIA core services. Outcome 

efficiency measures could influence NEG project 

operators to provide less costly services than the more 

intensive and training services NEG participants may 

require. In addition, to earlier points, Disaster NEGs 

are not primarily training grants. A large portion of 

these funds are used for wages for temporary workers, 

not training, and should be differentiated from 

traditional training grants so as not to skew unit costs.

The authors agree with 

comment with regard to 

concerns about efficiency 

measures potentially resulting 

in creaming and providing less 

costly services – which is 

noted throughout report and 

reflected in recommendations. 

A footnote was added in 

Chapter 5 of the Final Report 

acknowledging that a large 

portion of NEG funds in 

Disaster NEGS are expended 

on training and consideration 

should be given to excluding 

such costs in calculating 

efficiency measures for NEG. 

ONR 

ONR agrees with recommendations on the use of 

expenditures rather than appropriations, and on the use 

of cost per entered employment and cost per retained in 

employment. However, ONR has concerns with the 

cost per average earnings, and cost per change in 

earnings as metrics, at least for NEGs, as the NEG 

No change necessary to the 

Final Report. The authors 

agree that NEG participants 

may not do well on the cost 

per earnings change measure 

because of the reasons stated 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 215 

PROGRAM ETA PROGRAM OFFICE COMMENTS AUTHORS’ RESPONSE 

participants are often well-compensated individuals 

that were laid-off from declining industries. NEG 

service interventions result in the program putting 

people on a new career path, which means they are 

usually starting over at entry-level positions. As with 

the broader dislocated worker program, NEGs would 

probably not look particularly efficient under these 

efficiency measures, as the individual may not be able 

to recoup their previous earnings at the point the 

measurement is taken. Despite this, it may have been a 

good outcome for the individual, who has better 

longer-term prospects and/or is better than the 

alternative of no earnings from being long-term 

unemployed. 

(that is in part why more than 

one measure has been 

recommended – some 

programs will do better on 

some measures than others). 

This is more of a standard 

setting issue (and is one that 

also a potential issue for the 

WIA dislocated worker 

program, which is pointed out 

in the comment). The authors 

also caution against making 

cross-program and cross-state 

comparisons of efficiency 

measure results because 

programs have differing 

objectives, serve different 

populations, provide varying 

services, and a host of other 

underlying factors (differing 

patterns of cost sharing, coenrollment, etc). 

ONR agrees with the recommendation to look at 

separate efficiency measures for each type of service 

provided. ONR also shares concerns over additional 

information collection burdens that may be required by 

implementing such an outcome efficiency measure; 

NEG grantees already submit 4 reports to track 

progress, measure performance, and report fiscal 

expenditures. 

No change necessary to the 

Final Report. 

Authors share concerns about 

information collection burdens 

(one of the reasons that the 

Common Measures are at the 

heart of the recommended 

measures is to reduce data 

collection burden on 

states/local areas. 

Similar to the presentation slides on Improving 

Measures of Efficiency for Employment and Training 

Programs, NEG is listed as a program that would 

require a different set of efficiency measures. 

However, the explanatory text does not discuss this 

rationale for NEGs. When we inquired about the 

discrepancy to John Trutko, he said that this was a 

typographical error and NEG was not supposed to be 

included in that list. Instead, the program listed should 

have been WIA Youth. Please remove “NEG” from 

Recommendation #3 in the report if it is indeed a typo. 

The text should have read 

“Youth” program instead of 

“NEG.”A revision was made 

to the Final Report (in the 

Executive Summary and 

Chapter 8) to reflect this 

change. 

OWI Attached are OWI/Adult Services’ comments on the 

Efficiency Measures Study. Let me know if you have 

any questions. Thanks for the chance to comment. 

 

The issue of cost comparisons. A higher unit cost in 

No change necessary to the 

Final Report. 

Authors agree with concerns 

about cost comparisons across 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 216 

PROGRAM ETA PROGRAM OFFICE COMMENTS AUTHORS’ RESPONSE 

one year compared to the previous year does not 

necessarily denote a less efficient program, nor does a 

lower unit cost necessarily suggest a more efficient use 

of funds. I say this because we are at a time when 

education and training costs are sky rocketing and the 

model has yet to discuss target setting – do we assume 

that there is a ‘right’ cost per entered employment, 

retention, etc. 

programs and highlight this 

concern in various places in 

the report. There is no 

assumed “right” cost. 

The Administration is focused on serving low-skill, 

low-income and the most disconnected jobseekers. 

Additional focus has also been given to the benefits of 

longer term training, support services and increasing 

credential attainment. These services cost more. After 

JTPA, we moved away from so-called efficiency 

targeting because they seemed to discourage providing 

higher cost services. Measuring ‘performance’ in these 

terms puts our performance system at odds with our 

program design. 

No change necessary to the 

Final Report. Authors agree 

with concerns and highlight 

issues throughout report on 

creaming and being careful not 

to discourage training where it 

is appropriate. 

Related to this, there was mention that we would be 

communicating with states to let them know what the 

national efficiency goal was and recommend against 

this. As you know, one of the goals of WIA is to 

encourage flexibility in serving the various populations 

of jobseekers that come through the one-stop career 

centers. Again, having states operate under the 

umbrella of a cost per (employment or earnings, etc.) 

may drive perverse performance by providing a 

disincentive to serve the hardest-to-serve populations. 

No change necessary to the 

Final Report. Authors agree 

with concerns about having 

perverse incentives to not 

serve hardest to serve and 

potentially cutting down 

flexibility to provide services 

needed most by participants. 

We argue for great caution in 

making cross-state and crossprogram comparisons and in 

setting either national or stateby-state performance 

standards. We also suggest 

that work on statistical 

adjustment models be pursued. 

We feel strongly that the unit of analysis for tracking 

results of an efficiency measure should remain at the 

national level – this should not be a measure imposed 

on states/grantees. However, the current study 

recommendations leave open the possibility of having 

states collect and report these results. We suggest 

striking that idea as a possibility. “The modeling effort 

conducted for this report, which focuses on the three 

WIA programs (emphasis added), suggested that great 

caution and several additional years of results are 

required before ETA should consider implementing 

performance standards for states/grantees on the 

recommended efficiency measures.”] 

No change necessary to the 

Final Report. The authors 

wanted to provide ETA with 

some flexibility in determining 

if and when to set standards on 

measures for states – but at the 

same time warn great caution 

in setting sub-national 

standards and suggest not 

doing this for at least several 

years. We concur that 

standards probably should not 

in the end be set, but wanted to 

preserve flexibility for ETA if 

it wanted in the future to 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 217 

PROGRAM ETA PROGRAM OFFICE COMMENTS AUTHORS’ RESPONSE 

perhaps apply efficiency 

measures to a subset of the 

programs considered in this 

study. 

How will the information be used if we collect and 

calculate it? During the previous administration, OMB 

wanted to show reduced training costs over time which 

was not feasible. Do we have new direction on 

efficiency measures which should be factored into the 

report? We feel that it would be better to have more 

focus on interim measures (e.g., credentials) rather than 

efficiency. If this is not a priority area of measurement, 

it is questionable to continuing to fund this project. 

No change necessary to the 

Final Report. The authors 

concur that it would be 

problematic to show reduced 

training costs over time and 

that given ETA’s emphasis on 

providing training services that 

it would be even more 

problematic. The report does 

not make an argument one 

way or another about 

increasing/decreasing training 

cost over time. The authors 

have linked efficiency 

measures to the Common 

Measure outcomes and do not 

recommend use across 

programs of interim measures 

– though individual programs 

could develop supplemental 

interim efficiency measures if 

they thought they had merit. 

One of the suggested measures was Cost per Entered 

Employment. We mentioned during the meeting that 

some of our programs have lower placement rates due 

to their circumstances (for example, Indian reservations 

with significant systemic unemployment). In this way, 

the measure may not be the best way to gauge 

‘efficiency’ since the measure is unable to take into 

account the differences in the populations being served 

among different programs. 

No change necessary to the 

Final Report. We highlight 

throughout the need for care in 

comparing across programs 

and within programs across 

states/grantees and over time. 

How were individual programs’ data used to determine 

the efficiency measure recommendation? Did the 

researchers take into account that some populations are 

harder to serve than others? 

No change necessary to the 

Final Report. Throughout the 

report we highlight that care 

has to be taken with regard to 

use of efficiency measures and 

not discouraging services to 

the hardest to serve. When we 

did regressions on the WIA 

Adult and Dislocated Worker 

programs, we did take into 

account some population 

characteristics; we did not take 

population characteristics into 

account in analyzing results 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 218 

PROGRAM ETA PROGRAM OFFICE COMMENTS AUTHORS’ RESPONSE 

for other programs, but did 

highlight that it is important to 

be very careful in making 

cross-program comparisons of 

efficiency measure results. 

The study’s authors state in Recommendation #3 that 

programmatic differences should be carefully 

considered before implementing efficiency measures – 

then single out WIG, Apprenticeship, and NEG as 

requiring different measures. How did they come to 

the conclusion that the other programs, particularly 

NFJP, INA, and SCSEP should be considered as 

similar to WIA Adults, Dislocated Workers, Wagner 

Peyser, etc.? 

No change necessary to the 

Final Report. The authors 

singled out WIG, 

Apprenticeship, and Youth as 

qualitatively different and in 

need of a set of different 

measures. We recognize that 

there are programmatic 

differences among the other 

programs, but these programs 

are all covered by the 

Common Measures – so these 

same problems of comparison 

exist with regard to the 

existing performance 

measurement system. We 

urge in our recommendations 

not to make simplistic 

comparisons of efficiency 

measure results across 

programs. 

Recommendation #4 suggests that caution should be 

taken before applying sanctions and/or rewards. 

Targeted programs such as the NFJP and INA, have no 

mandates for sanctions or rewards in the legislation; 

how are they suggesting this be developed or applied?

We have cautioned strongly 

about applying 

sanctions/awards and this 

applies to all programs, 

including NFJP and INA. A 

footnote was added to 

Recommendation Chapter 8 of 

the Final Report indicating 

that the NFJP/INA do not 

having mandates for 

sanctions/rewards.. 

Recommendation #9 states that further study is needed 

on topics related to and likely to affect efficiency 

measures results, and then lists a series of topics/issues 

that ETA should resolve before applying efficiency 

measures. One omission from the list is the provision 

of supportive services. For the NFJP, these 

wraparound services are part of an individual’s IEP; 

its impact on the program (or other programs that 

routinely provide these services) should be considered 

as a policy issue that must be resolved. 

Recommendation #9 in the 

Final Report (in the Executive 

Summary and concluding 

chapter) has been revised to 

include supportive services as 

a possible expenditure item to 

be excluded in costs along 

with needs-based payments 

and stipends (the bullet point 

now read: “Should stipends, 

supportive services, and needbased payments be counted as 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 219 

PROGRAM ETA PROGRAM OFFICE COMMENTS AUTHORS’ RESPONSE 

expenditures in computing 

efficiency measures?) 

TAA TAA provided a series of comments to the Executive 

Summary) 

The TAA program is fully supportive of this summary 

of serious challenges for establishing an efficiency 

measure which would apply to services and benefits 

provided under TAA. In particular, TAA is concerned 

with the unintended consequences of reviewing state 

outcomes in the context of an arbitrary “efficiency 

measure”. 

No change necessary to the 

Final Report. 

The report covers potential for 

“unintended consequences” 

and cautions about applying 

measures/standards across 

states. 

 The TAA program concurs with the cautions expressed 

by state workforce officials. In particular, TAA notes 

that it is a program designed to provide costly 

retraining for trade affected individuals to return them 

not just to a job, but to a sustainable job. As such, 

short term training is not the goal, resulting in a high 

cost per placement – a cost that does not indicate 

inefficiency in the program design or administration; 

but rather reflects congressional intent for the 

population served. Any efficiency measure established 

for the TAA program must reflect the congressional 

intent inherent in the TAA mission. 

In addition, training costs vary greatly by state making 

cross state comparisons of little use. Efficient program 

administration in one state can not be used as a model 

to establish an efficiency benchmark in another state. 

TAA tends to look at average cost for training in each 

state to gauge the amount of funding that is needed per 

certified individual, and recommends that some 

consideration be given to developing a model based on 

a like approach to reflect state differences in training 

costs. 

No change necessary to the 

Final Report. The authors 

concur with concerns and 

emphasis of TAA on providing 

training and returning workers 

to sustainable jobs. The report 

cautions about setting 

standards for programs/states/ 

grantees; and warns about 

making simplistic comparisons 

of results across programs. 

Authors agree that training 

costs can vary greatly across 

states (as well as participants). 

 The TAA program concurs that expenditure data is the 

correct measure for quantitative analysis, as 

expenditure data reflect actually cost, while obligations 

reflect anticipated costs that may or may not 

materialize. With regard to TAA program data, 

however, we are pleased to note that the newly 

instituted TAPR reporting system for the first quarter 

of FY 2010 is data upon which we believe we can 

readily rely. 

No change necessary to the 

Final Report 

Recommendation #1: Use Program Expenditures 

Rather than Appropriations or Obligations as the 

No change necessary to the 

Final Report 
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 220 

PROGRAM ETA PROGRAM OFFICE COMMENTS AUTHORS’ RESPONSE 

Measure of Program Costs in Efficiency Measures

OTAA concurs – see above 

Recommendation #2: Use Common Measures as 

Starting Point for Measuring Program Outcomes in 

Efficiency Measures 

OTAA concurs. 

No change necessary to the 

Final Report 

Recommendation 2a: Cost per entered employment 

should be tracked (for monitoring purposes 

initially) as an efficiency measure for 8 of the 11 

ETA programs: WIA Adult, WIA Dislocated 

Worker, WIA NEG, Wagner-Peyser, TAA, SCSEP, 

INA, and NFJP. 

OTAA concurs – however, benchmarks established 

must reflect the high cost of the TAA program which 

reflects its congressional mission as well as a series of 

differing goals in the states which reflects the variation 

required for successful performance in a given state. 

No change necessary to the 

Final Report. 

Authors agree with OTAA 

concern, and the report 

cautions against making 

simplistic cross-program and 

cross-state (within program) 

efficiency measure results. 

Clearly costs of TAA will be 

much higher than other 

programs because of the 

emphasis on long-term 

training. 

Recommendation 2b: Cost per retained in 

employment should be tracked (for monitoring 

purposes initially) as an efficiency measure for 8 of 

the 11 ETA programs: WIA Adult, WIA Dislocated 

Worker, WIA NEG, Wagner-Peyser, TAA, SCSEP, 

INA, and NFJP. 

OTAA concurs – however, benchmarks established 

must reflect the high cost of the TAA program which 

reflects its congressional mission as well as a series of 

differing benchmarks in the various states which 

reflects the varying costs required for successful 

performance in different states. 

No change necessary to the 

Final Report. 

Authors agree with OTAA 

concern and the report 

cautions against making 

simplistic cross-program and 

cross-state (within program) 

efficiency measure results. 

Clearly costs of TAA will be 

much higher than other 

programs because of the 

emphasis on long-term 

training. 

Recommendation #2c: Cost divided by postprogram (average) earnings should be tracked (for 

monitoring purposes initially) as an efficiency 

measure in 8 of 11 ETA programs: WIA Adult, 

WIA Dislocated Worker, WIA NEG, WagnerPeyser, TAA, SCSEP, INA, and NFJP. 

OTAA does not believe that post-program average 

earnings have any particular relevance to the effective 

delivery of services under the TAA program. Rather, 

OTAA believes that an efficiency measure reflecting 

the goals of the TAA program (returning workers to 

sustainable employment) should look at the cost 

associated with returning trade affected workers to 

employment that approaches pre program earnings for 

those trainees. 

No change necessary to the 

Final Report. Authors agree 

that care is needed in using 

post-program average earnings 

and that this measure should 

be only one of four measures 

examined. Also, average postprogram earnings is a measure 

currently reported on by TAA 

as part of the Common 

Measures. Cost per earnings 

change was intended as a 

second measure that would in 

some ways better reward 

training efforts (than post-
 

Final Report - Recommended Efficiency Measures for Selected ETA Programs Page 221 

PROGRAM ETA PROGRAM OFFICE COMMENTS AUTHORS’ RESPONSE 

program earnings), though 

authors acknowledge that 

trade-affected workers are 

often not likely to achieve 

their pre-program earnings. 

Recommendation #2d: Cost divided by change in 

earnings should be tracked (for monitoring 

purposes initially) as an efficiency measure in 8 of 

the 11 ETA programs: WIA Adult, WIA Dislocated 

Worker, WIA NEG, Wagner-Peyser, TAA, SCSEP, 

INA, and NFJP. 

OTAA concurs. 

No change necessary to the 

Final Report 

Recommendation #3: Carefully Consider 

Programmatic Differences Before Implementing 

Efficiency Measures – Among the 11 ETA 

Programs, WIG, Apprenticeship, and NEG 

Programs Will Likely Require a Different Set of 

Efficiency Measures

OTAA strongly concurs. TAA is a program designed to 

provide costly retraining for trade affected individuals 

to return them not just to a job, but to a sustainable job. 

As such, short term training is not the goal, resulting in 

a high cost per placement – a cost that does not indicate 

inefficiency in the program design or administration; 

but rather reflects congressional intent for the 

population served. Any efficiency measure established 

for the TAA program must reflect the congressional 

intent inherent in the TAA mission. 

No change necessary to the 

Final Report 

Recommendation #4: Performance Standards for 

States/Grantees on Recommended Efficiency 

Measures Should Be Considered Exploratory at 

This Time -- Do Not Reward or Sanction 

States/Grantees for Performance on the 

Recommended Efficiency Measures

OTAA strongly concurs. 

No change necessary to the 

Final Report 

Recommendation #5: Improve Consistency and 

Quality of Cost, Customer Characteristics, and 

Outcome Data

Under the provisions of the new TAPR reporting, 

effective for first quarter of FY 2010, OTAA believes 

that reliable data exists for the TAA program.

No change necessary to the 

Final Report