1330 Broadway, Suite 1426 

Oakland, CA 94612 

Tel: (510) 763-1499 

Fax: (510) 763-1599 

www.spra.com 

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •

S O CIA L P O L ICY R ESEA R C H

A S S O CIATE S

Prepared for: 

U.S. Department of Labor 

Employment and Training 

Administration 

200 Constitution Ave. NW 

Room N-5637 

Washington, D.C. 20210 

Contract # AF-12555-02-30 

Project No. 1146

Evaluation of the WIA 

Performance Measurement 

System 

Survey Report 

August 2005 

Prepared by: 

Social Policy Research Associates 
This page intentionally left blank.

Insert blank page here when making double-sided copies
1

DISCLAIMER 

This report was prepared for the U.S. Department of Labor, Employment and Training 

Administration, Office of Policy Development and Research, by Social Policy Research 

Associates. Since contractors conducting research and evaluation projects under government 

sponsorship are encouraged to express their own judgment freely, this report does not necessarily 

represent the official opinion or policy of the U.S. Department of Labor. 
This page intentionally left blank.

Insert blank page here when making double-sided copies
Acknowledgements and Attributions 

This report was conducted by Social Policy Research Associates. The project was directed by 

Andrew Wiegand of SPR, but depended for its success on the strong contributions of the 

numerous field staff and analysts who worked on this project during its five years, including the 

primary authors of this survey report, Jessica Pearlman, Jared Rodecker, and Richard West. 

We wish to acknowledge as well the contributions and support of DOL staff, especially our 

project officers, Alberta Baker, Lekesha Campbell, and Dan Ryan. Their advice and steadfast 

support are much appreciated. Thanks, too, to the invaluable contributions of the Local 

Workforce Investment Area staff, whose responses to the evaluation survey form the heart of this 

report. 
This page intentionally left blank.

Insert blank page here when making double-sided copies
i

CONTENTS 

EXECUTIVE SUMMARY .......................................................................................... ES-1

I. ANALYSIS OF THE LOCAL AREA PERFORMANCE 

MEASURES SURVEY .......................................................................................... I-1

Introduction .....................................................................................................................I-1

Survey Administration.................................................................................................I-1

Comparison of Survey Sample with Non-Respondents..............................................I-2

Topics Addressed by the Local Area Survey of the WIA 

Performance Measurement System ...........................................................................I-5

Implementation of the Performance Measurement System: State 

Policies and Local Response ..........................................................................................I-5

Negotiation of Performance Levels.............................................................................I-6

Incentives..................................................................................................................I-13

Sanctions..................................................................................................................I-13

Technical Assistance................................................................................................I-15

Implementation of the Performance Measurement System: Measures 

and Data........................................................................................................................I-17

Operationalization of Measures................................................................................I-17

Measures Emphasized .............................................................................................I-19

Supplemental Data ...................................................................................................I-20

Performance and Relationships with Providers and Partners.......................................I-21

Types of Providers Used for Adult and Dislocated Worker Services........................I-22

Strategies Used to Promote Performance: Eligible Training Provider 

List ............................................................................................................................I-22

Types of Youth Service Providers ............................................................................I-25

Strategies Used to Promote Performance: Selection and Contracts 

with Youth Providers................................................................................................I-27

Strategies Used to Promote Performance: Incentives and Sanctions 

in Contracts with Service Providers ..........................................................................I-31

Availability of Adult and Youth Providers: Additional Considerations .......................I-34

Performance Measures and Relationships with WIA Partners .................................I-35

Local Area Perception of the Impact of the Performance Measurement 

System ..........................................................................................................................I-40
 ii

Customer Focused Strategies ..................................................................................I-41

Service Design..........................................................................................................I-46

Program Costs..........................................................................................................I-49

Use of WIA Performance Measures and Alternative Measures to 

Inform Program Management and Improve the Quality of Service ...............................I-50

Use of WIA Measures for Program Management and to Improve 

Program Quality........................................................................................................I-50

Additional Local Performance Measures..................................................................I-54

Formal (Explicit) Continuous Improvement Programs ..............................................I-58

Overall Reaction to the Performance Measurement System ........................................I-59

Unanticipated Benefits..............................................................................................I-59

Unanticipated Negative Impacts...............................................................................I-61

Summary.......................................................................................................................I-62

II. EFFECTS OF STATE AND LOCAL PERFORMANCE POLICIES 

ON CUSTOMERS, SERVICES, AND OUTCOMES............................................. II-1

How We Coded State Policies .......................................................................................II-2

How We Measured Local Policies................................................................................II-10

Overall Service Design ............................................................................................II-10

Relationships with Providers....................................................................................II-12

Methodological Approach and Data Sources...............................................................II-14

Unit of Analysis........................................................................................................II-15

Dependent Variables ...............................................................................................II-15

Exogenous Variables...............................................................................................II-16

Intervening Variables...............................................................................................II-18

Effects of State Policies ...............................................................................................II-18

Effects of State Policies on Customers....................................................................II-19

Effects of State Policies on Services .......................................................................II-25

Effects of State Policies on Outcomes.....................................................................II-28

Effects of Local Policies ...............................................................................................II-30

Effects of Local Policies on Customers ...................................................................II-30

Effects of Local Policies on Services .......................................................................II-30

Effects of Local Policies on Outcomes ....................................................................II-33

Effects of State Policies on Local Policies ...............................................................II-34

Summary and Implications for the National Performance Measurement 

System .........................................................................................................................II-36 
iii

EXHIBITS 

Exhibit I-1: Characteristics of Survey Respondents and Non-Responders............................I-3

Exhibit I-2: LWIA Perception of the Fairness of Initial Levels.................................................I-7

Exhibit I-3: Perception of Fairness of Initial Levels by Items Considered in 

Initial Negotiation Process....................................................................................I-7

Exhibit I-4: Perceived Fairness of Initial Levels by Number of Items 

Considered in the Initial Negotiation Process ......................................................I-8

Exhibit I-5: Percentage of LWIAs who Feel that Initial Negotiated Levels 

Are Attainable by Items Considered in the Negotiation Process........................I-10

Exhibit I-6: Factors that Make Initial Levels Unattainable ....................................................I-11

Exhibit I-7: Factors Associated with LWIA Requests to Revise Initial Levels ......................I-12

Exhibit I-8: Reasons LWIAs Received Performance Related Technical 

Assistance..........................................................................................................I-15

Exhibit I-9: Measures Addressed by Technical Assistance Sessions ..................................I-16

Exhibit I-10: Types of Locally Recognized Credentials Developed by Local 

Areas..................................................................................................................I-18

Exhibit I-11: Number of LWIAs Emphasizing Specific Performance Measures .....................I-20

Exhibit I-12: Sources of Supplemental Data Used .................................................................I-21

Exhibit I-13: Aspects of the Performance Measurement System that 

Discourage Training Providers from Applying for Eligibility................................I-23

Exhibit I-14: Availability of Adult Service Providers................................................................I-24

Exhibit I-15: Types of Organizations Used to Provide Youth Services ..................................I-25

Exhibit I-16: Average Number of Youth Providers and Number of Different 

Types of Youth Providers by Size of Local Area................................................I-26

Exhibit I-17: Use of Explicit Performance Requirements in Contracts with 

Youth Providers by Size of Local Area...............................................................I-28

Exhibit I-18: Aspects of the Performance Measurement System that 

Discourage Youth Providers from Submitting Proposals ...................................I-29

Exhibit I-19: Availability of Youth Service Providers...............................................................I-30

Exhibit I-20: Forms of Incentives and Sanctions Used in Contracts with 

Service Providers...............................................................................................I-31

Exhibit I-21: Use of Incentives/Sanctions in Contracts with Service Providers 

by Extent of Urbanization...................................................................................I-33

Exhibit I-22: Use of Incentives/Sanctions in Contracts with Service Providers 

by Size of Local Area .........................................................................................I-33
iv

Exhibit I-23: Availability of Adult Service Providers by Urbanization of Local 

Area....................................................................................................................I-34

Exhibit I-24: Models Used by Local Areas to Provide Core and Intensive 

Services .............................................................................................................I-36

Exhibit I-25: WIA Staff Members’ Willingness to Co-Enroll Customers..................................I-39

Exhibit I-26: Extent of Co-Enrollment Between ES and WIA by Program(s) 

Used to Provide Core Services..........................................................................I-40

Exhibit I-27: Customers Targeted by Local Areas..................................................................I-41

Exhibit I-28: Influent of the Performance Measures on Local Area Targeting 

of Customer Groups...........................................................................................I-42

Exhibit I-29: Performance Related Factors that Influence the WIA 

Registration Process..........................................................................................I-43

Exhibit I-30: Performance Related Criteria Used to Determine the Time of 

Exit from WIA .....................................................................................................I-45

Exhibit I-31: Types of Customer Focused Strategies Used by Local Areas to 

Improve Performance.........................................................................................I-46

Exhibit I-32: Impact of the Performance Measurement System on Services 

Offered by One-Stop Centers and Service Providers ........................................I-47

Exhibit I-33: Impact of the Performance Measurement System on WIA 

Program Costs ...................................................................................................I-49

Exhibit I-34: Centrality and Helpfulness of the WIA Performance Measures in 

Overall Efforts to Improve Program Quality .......................................................I-51

Exhibit I-35: Contributions of the WIA Performance Measures for Program 

Management and Improvement Efforts..............................................................I-52

Exhibit I-36: Factors that Limit the Usefulness of the WIA Performance 

Measures for Program Management and Improvement Efforts .........................I-53

Exhibit I-37: Additional Performance Measures Used by Local Areas...................................I-54

Exhibit I-38: Use of Additional Local Performance Measures by Perceived 

Helpfulness of the WIA Measures in Improving Program Quality ......................I-55

Exhibit I-39: Use of Additional Measures by Size of Local Area ............................................I-57

Exhibit I-40: Use of WIA Performance Measures and Explicit Continuous 

Improvement Programs to Achieve Continuous Improvement...........................I-58

Exhibit I-41: Elements of Local Area Formal Continuous Improvement 

Programs............................................................................................................I-59

Exhibit I-42: Unanticipated Benefits of the Performance Measurement 

System ...............................................................................................................I-60

Exhibit I-43: Unanticipated Negative Impacts of the Performance 

Measurement System ........................................................................................I-61
v

Exhibit II-1 Methods States Use to Set Performance Goals .................................................II-4

Exhibit II-2: Provision of Performance Incentives to Local Areas...........................................II-5

Exhibit II-3: Criteria States Use to Qualify Local Areas for Incentive Awards ........................II-6

Exhibit II-4: How States Determine Incentive Awards............................................................II-7

Exhibit II-5: Sanction Criteria Used by States ........................................................................II-8

Exhibit II-6: Flexibility of Credential Definitions ......................................................................II-9

Exhibit II-7: Conceptual Framework for Quantitative Analysis .............................................II-14

Exhibit II-8: Effects of State Policies on the Characteristics of Adults..................................II-20

Exhibit II-9: Effects of State Policies on the Characteristics of Youth ..................................II-24

Exhibit II-10: Effects of State Policies on the Services Received by Adults 

and Dislocated Workers....................................................................................II-26

Exhibit II-11: Effects of State Policies on the Services Received by Youth ...........................II-28

Exhibit II-12: Effects of State Policies on Outcomes..............................................................II-29

Exhibit II-13: Effects of Local Policies on the Characteristics of Adults .................................II-31

Exhibit II-14: Effects of Local Policies on Services ................................................................II-32

Exhibit II-15: Effects of Local Policies on Outcomes..............................................................II-34

Exhibit II-16: Effects of State Polices on Local Policies .........................................................II-35
This page intentionally left blank.

Insert blank page here when making double-sided copies
ES-1

EXECUTIVE SUMMARY 

This report includes the results of two separate sets of quantitative analyses for our Evaluation of 

the WIA Performance Measurement System. In the first chapter, we present results from an 

analysis of the local area survey, which was mailed in the fall of 2004 to all LWIAs throughout 

the nation. These results detail the specific policies and procedures that local areas have adopted 

in response to and furtherance of the WIA performance measurement system. In addition, this 

chapter discusses local area perceptions of the system and their perceptions of the state’s role in 

implementing it. The second chapter of this report details how state-level policies, as coded 

from the written plans and guidance developed by each state, and the local-level policies are 

associated with the characteristics of customers served by WIA, the services they receive, and 

the outcomes they obtain. This chapter thus examines how the various policies and procedures 

developed by states and local areas affect who is being served under WIA and how they are 

being served. 

Local Area Survey of the Performance Measurement System 

We received completed responses from 455 of the 572 local areas that were mailed a survey, 

which represents a response rate of 80%. Several key findings emerge from analysis of this 

survey. These findings were categorized into several sections, including those concerning: the 

initial negotiation process, incentives, sanctions and technical assistance, relationships with 

providers and partners, impacts on who is served and the services provided, and the use of 

measures in program management and improvement. 

Initial Negotiation Process 

Significant diversity exists among local areas in terms of their feelings about the quality of the 

initial negotiation process. The majority of local areas felt that the process involved some 

negotiation but that more attention could have been given to local concerns. Approximately one 

third of local areas felt that the initial negotiation process was extremely minimal or nonexistent. A small percentage of local areas were completely satisfied with the initial negotiation 

process. 
ES-2

In spite of the fact that most local areas felt the initial negotiation process could have involved 

more consideration to local issues, most local areas reported that their initial levels were 

attainable. This may be due to the fact that many local areas have adapted their programs in order 

to facilitate positive performance outcomes. 

Incentives, Sanctions and Technical Assistance 

Most states have implemented policies to award incentives and impose sanctions upon local 

areas for exemplary and poor performance, respectively. The possibility of receiving sanctions 

has impacted some aspect of WIA program design in approximately two-thirds of local areas. 

The possibility of receiving incentives has influenced WIA programs in a somewhat smaller 

percentage; just over one-half of local areas. 

Technical assistance is also very common. The great majority of local areas have received 

performance related technical assistance. Performance TA tends to focus on all of the measures 

and is frequently provided to local areas regardless of whether they have failed a measure or are 

in danger of doing so. 

Relationships with Providers and Partners 

The majority of local areas attempt to ensure performance through relationships with eligible 

training providers and adult and youth contracted service providers. In more than 60% of local 

areas, all eligible training providers are required to achieve local performance levels, and in an 

additional 25% of local areas at least some providers are required to meet these levels. In 

addition, nearly all local areas hold youth service providers contractually responsible for 

achieving local performance targets. Finally, two-thirds of local areas reported that they use 

incentives and/or sanctions in relationships with service providers. 

In most local areas, the performance measurement system does not have a substantial impact on 

the availability of eligible training providers or youth providers. However, a significant minority 

of local areas (30%) reported that the performance measurement system serves to discourage 

numerous youth service providers and/or eligible training providers from submitting applications 

to serve WIA customers. 

Most local areas make substantial use of WIA partners, especially in the provision of core 

services. However, the performance measures have a negative impact on coordination among 

WIA partners in a subset of local areas. Approximately 30% local areas reported that the 

performance measures decrease partner willingness to be actively involved in the One-Stop 

system and one half of local areas reported that the performance measures decrease willingness 

to co-enroll clients. 
ES-3

Impact on Who is Served and What Services are Provided 

According to local area responses, the performance measurement system has had a substantial 

impact both on which customers are served and what services are offered. Key points are 

highlighted below: 

• More than 80% of local areas make strategic decisions regarding the time of exit 

in order to improve the probability of achieving positive performance outcomes. 

• Most local areas also use strategies such as targeting or the time of registration in 

order to select customers who are likely to improve performance. Evaluation of 

customer motivation is an important factor in the selection process. However, we 

found little evidence that local areas are choosing not to serve traditionally hardto-serve customers populations. Rather, targeting tends to focus on populations 

with barriers and low-income customers. 

• Significant diversity exists among local areas as to perspectives on whether the 

performance measures have served to expand, limit or have had no impact on the 

services offered by the One Stop Centers and providers. 

• Two-thirds of local areas have made the decision to alter the relative emphasis 

placed on the service tiers (core, intensive and training). Typically, these local 

areas have made the decision to place more emphasis on intensive and/or training 

services. 

Use of WIA Measures and Alternative Measures for Program Management and to 

Improve Program Quality 

Nearly all local areas (82%) report that they use the performance measures to assess continuous 

improvement. Local areas report that the performance requirements provide an overall goal and 

the outcomes enable them to determine what services are effective. However, the use of the 

WIA performance measures in improvement efforts is constrained by the time lag necessary to 

obtain outcomes using UI wage records outcomes. Primarily due to this time lag, only 46% of 

local areas reported that the performance measures are helpful in informing day-to-day 

management decisions. 

Nearly 60% of local areas have chosen to adopt additional measures of program performance. 

The most commonly used measures track enrollments, job placements, customer satisfaction, 

program costs, and services to employers. 

Effects of State and Local Performance Policies on 

Customers, Services, and Outcomes 

The results from the local area survey are those self-reported by local areas and represent these 

areas’ perceptions of the performance measurement system. The second chapter of this report 

uses statistical analysis to formally examine how the policies developed by states and locals have 
ES-4

actually affected who is served under WIA and how they are served. This analysis relied on our 

coding of written state policies and guidance collected as part of this evaluation, and the coding 

of the local area survey. These policies were included in regression models that also included 

local area economic and population characteristics and individual-level data drawn from the 

WIASRD file, which is a comprehensive record of WIA exiters. 

Overall, many of the results in the chapter are not conclusive and do not suggest that variations 

in state policies have substantial impact on the customers who are served under WIA, the 

services they receive, or the outcomes they obtain. There is, however, one significant and 

important conclusion that can be drawn from the data. The provision by states to local areas of 

incentive awards based on performance seems to increase service to a variety of customer groups 

that tend to be less job-ready and have poorer work histories. This result is confirmed by similar 

results for states with a single local workforce area, which are subject to federal incentive 

policies. It is also confirmed by similar results for the provision of incentives to providers by 

local areas. 

This result suggests that fears that the WIA performance system might induce local areas to 

focus services on better-prepared individuals are unfounded. Rather, the analysis in this report 

suggests that the performance system spurs service to those with poorer work histories and, thus, 

poorer future employment prospects. For example, local areas in states that provide incentives 

are more likely to serve less job-ready individuals with poor work histories. Although this and 

similar findings may seem counterintuitive, they may, in fact, be very calculated decisions that 

reflect a desire to serve those who stand to gain the most from WIA and, as a result, show the 

most substantial improvement on the performance measures. In other words, individuals with 

low or no prior earnings can show far more significant earnings gains than can those who have 

an established and recent work history. Similarly, serving the unemployed allows local areas to 

earn credit on the entered employment measure, while serving those already employed would not 

do so. Thus, these results may simply reflect local areas’ decisions about which customers will 

gain the most from WIA services, both for the customers’ sake and for the sake of positive 

performance outcomes. 

In this respect, then, it would seem that the presence of the earnings change and replacement 

measures offsets the influence of the other measures in affecting which customers are served. 

But we cannot say conclusively that this is true, especially because we also uncovered some 

contradictory evidence. If local areas in states with performance incentives were following a 

strategy to increase earnings change and replacement, one would expect to see an increase in the 

earnings change outcomes and, perhaps, a decrease in the other outcomes in those states. While 

we found some significant relationships between state policies and outcomes, we did not find 
ES-5

that provision of incentives increased the earnings change measures. To the contrary, we found 

that more stringent incentive criteria seemed to decrease adult earnings change. Thus, decisions 

about whom to serve may be made with the earnings change or replacement measures in mind, 

but, if so, these decisions have little affect on the actual performance on these measures. 

Ultimately, then, much of the analysis concerning the effects of state and local policies is 

inconclusive, in part because states have stuck fairly closely with the system as promulgated at 

the federal level. Thus, there is relatively little variation across these states that would enable us 

to examine more fully how these variations are associated with different customer 

characteristics, services, and outcomes. Importantly, however, in each of our analyses, we found 

little to suggest that those most in need of WIA services have a difficult time receiving those 

services because of any state or local concern for achieving high performance. Rather, states and 

locals, while focused on achieving strong performance, also seem focused on providing needed 

workforce services to those who can most benefit from them. In this sense, then, the 

performance measurement system has achieved one of its purposes by establishing benchmarks 

and standards for states and local areas to achieve without altering the intent of the workforce 

system to provide services to those who can benefit from them. 
This page intentionally left blank.

Insert blank page here when making double-sided copies
I-1

I. ANALYSIS OF THE LOCAL AREA PERFORMANCE 

MEASURES SURVEY 

Introduction 

The first round of site visits, conducted in the spring of 2003, provided us with detailed 

information on the implementation of the WIA performance measurement system in eight states, 

as well as the impact of this system on WIA service design and delivery. In order to develop an 

overall picture of how the performance measurement system operates in local areas across the 

nation, a survey was administered to all local areas in the summer of 2004. In this chapter, we 

discuss the findings from this survey. 

Survey Administration 

The data collection process took place over a six-month period during the summer and fall of 

2004. In July of 2004, a paper survey was mailed to each of the 572 local areas in the 50 states 

and the District of Columbia.1 Instructions enclosed with the survey also directed the local area 

to a website where the respondent might complete an online version of the survey, if he/she 

preferred to do so. A second copy of the survey was mailed to all local areas in August of 2004 

and a reminder post-card was sent in early September. Over the next two months, all local areas 

who had not completed the survey received two email reminders and at least one follow-up 

phone call by SPR staff. In December, Department of Labor Regional Program Officers 

conducted a final round of phone calls to local areas who had not submitted a survey at that 

point. By the end of December 2004, we had received 455 surveys, covering 80% of local areas 

 

1

 Two local areas were not asked to complete the survey. One of these local areas serves the Navajo Nation in the 

state of Arizona. While this local area submits performance data to the WIASRD, their relationship to the 

Department Of Labor is different in several respects from other local areas nationwide. Therefore they were not 

included in the study. The second local area, formerly Region 7 in Ohio, divided into several different local 

areas at the start of PY04, the time at which the survey was originally administered. Due to the fact that the 

former local area no longer existed and the several new local areas had not yet had time to develop performance 

policies, upon consultation with former and current WIA directors in Ohio, it was determined that neither the old 

nor the new local areas should be asked to complete the survey. 
I-2

nationwide. The findings presented in this chapter are based on survey responses from these 455 

local areas. 

Comparison of Survey Sample with Non-Respondents 

Factors such as state policies, local economic conditions, and customer characteristics can have a 

substantial influence on the manner in which local areas choose to implement the performance 

measurement system. Therefore, it is important to ensure that, in terms of these factors, the 

survey sample is representative of local areas nationwide. 

Our survey sample includes strong representation from nearly all of the 50 states and the District 

of Columbia. With the exception of Alaska, the survey sample includes at least 50% of the local 

areas in each state and the District of Columbia. In 47 states and the District of Columbia at least 

60% of local areas completed surveys and in 40 states and the District of Columbia, at least 70% 

of local areas were survey respondents. 

We also conducted analysis to compare the local economic conditions and demographic 

characteristics of the survey sample with those of local areas that did not complete the survey. 

Data for this analysis were drawn from the Bureau of Labor Statistics Current Population Survey 

(unemployment rates), the United States 2000 Census (all other economic and local population 

characteristics), and the Workforce Investment Act Standardized Record Data system 

(WIASRD) (characteristics of WIA customers). Our analysis includes demographic indicators of 

the local population in addition to customer characteristics, because both types of characteristics 

can influence local area planning and policy implementation. 
I-3

Exhibit I-1: 

Characteristics of Survey Respondents and Non-Responders 

Characteristic 

Average for 

Survey 

Respondents 

(n=455) 

Average for Local 

Areas that did not 

Complete the 

Survey (n=118)2

Economic Conditions and Related Characteristics 

Unemployment Rate for PY03 5.9% 6.0% 

Average Earnings (1999) $29,189 $31,308 

Poverty Rate (2000) 12.5% 11.9% 

Population Size3 335,381 307,327 

Percentage of the Population living in Urban Areas 69% 71% 

Demographic Characteristics of the Local Population4 

Percentage of Adults Without a High School Diploma 16% 16% 

Percentage of Adults with a Bachelors Degree 23% 25% 

Percentage of Youth that are High School Dropouts 10% 10% 

Percentage of the Local Population that Receives Public Assistance 3% 3% 

Percentage of Unemployed Adults that are not White 35% 36% 

Percentage of Low Income Youth that are not White 44% 46% 

Percentage of Unemployed Adults that are African American 16% 20% 

Percentage of Low Income Youth that are African American 18% 23% 

Percentage of Unemployed Adults that are Latino 13% 10% 

Percentage of Low Income Youth that are Latino 17% 16% 

Percentage of Unemployed Adults that are Asian American 2% 2% 

Percentage of Low Income Youth that are Asian American 3% 3% 

Percentage of Adults that are Limited English Proficient 3% 3% 

Percentage of Youth that are Limited English Proficient 2% 2% 

 

2 The former Region 7 in Ohio is included in this analysis as a non-respondent. Data are used from the old rather 

than the new local areas as the available WIASRD data reflect the LWIA boundaries prior to PY04. 

3

 For population size, medians are presented rather than averages, as the averages may be skewed by a few very 

large or very small local areas. 

4

 When this information was available, characteristics are reported for the segment of the adult and youth 

populations that is most likely to receive WIA services: unemployed adults and low income youth. For some 

characteristics, information is only available for the overall population of adults and youth (education level, 

English proficiency) or the entire population at large (public assistance). 
I-4

Exhibit I-1 Continued: 

Characteristics of Survey Respondents and Non-Responders 

Characteristic 

Average for 

Survey 

Respondents 

(n=455) 

Average for Local 

Areas that did not 

Complete the 

Survey (n=118)2

Characteristics of WIA Customers for PY035 

Total Number of Adults and Dislocated Workers Served6 324 283 

Total Number of Youth Served 145 138 

Percentage of Adults/Dislocated Workers Without a High School Diploma 21% 20% 

Percentage of Adults/Dislocated Workers with a Bachelors Degree 10% 10% 

Percentage of Youth that are High School Dropouts 23% 23% 

Percentage of Adults that Receive Public Assistance7 13% 16% 

Percentage of Adults that are Low Income8 72% 70% 

Percentage of Adults/Dislocated Workers that are Not White 36% 39% 

Percentage of Youth that are Not White 54% 58% 

Percentage of Adults/Dislocated Workers that are African American 20% 24% 

Percentage of Youth that are African American 33% 39% 

Percentage of Adults/Dislocated Workers that are Latino 11% 9% 

Percentage of Youth that are Latino 16% 13% 

Percentage of Adults/Dislocated Workers that are Asian American 3% 4% 

Percentage of Youth that are Asian American 2% 3% 

Percentage of Adults/Dislocated Workers that are Limited English Proficient 3% 3% 

Percentage of Youth that are Limited English Proficient 2% 2% 

We see from Exhibit I-1, that there is little variation between survey respondents and nonresponders in terms of local economic conditions, local population characteristics and 

characteristics of WIA customers served. Thus, there is little reason to consider these survey 

respondents unrepresentative of local areas nationally. 

 

5

 Individuals exiting from WIA services between April 1, 2003-March 31, 2004, the latest year of data currently 

available in the WIASRD. 

6

 For number of customers served, medians are presented rather than averages as the averages may be skewed by a 

few very large or very small local areas. 

7

 Information on public assistance is not collected for dislocated workers. 

8

 Information on poverty status is not collected for dislocated workers. 
I-5

Topics Addressed by the Local Area Survey of the WIA Performance 

Measurement System 

The survey included questions focused on several major topics related to the implementation of 

the performance measurement system and the impact of this system on local area programs, 

including: 

• Implementation of the performance measurement system 

• The quality of the initial negotiation process between the state and local area to 

set performance levels 

• State policies designed to promote high performance, including incentives, 

sanctions and technical assistance, and the impact of these policies on local 

programs 

• State and local policies regarding the operationalization of measures and the use 

of supplemental data to track outcomes 

• The impact of the performance measurement system on relationships with 

contracted service providers, eligible training providers and WIA partners 

• The manner in which the performance measurement system influences which 

customers are served 

• The impact of the performance measurement system on service design and 

delivery, including program costs 

• The contribution of the performance measures to local efforts to improve 

management operations and program quality 

• Additional performance measures used by local areas 

• Local perspectives regarding the overall benefits and negative impacts of the 

performance measurement system 

This chapter will discuss findings pertaining to these topics. While we will focus on local area 

perspectives gathered from survey responses, where relevant, we will also draw upon findings 

from our analysis of state performance policy documents. The analysis of state policies is 

discussed in more detail in Chapter II. 

Implementation of the Performance Measurement 

System: State Policies and Local Response 

Implementation of the performance measurement system involves the development of several 

key state policies and practices. Most importantly, states establish the performance levels that 

local areas must meet. In order to increase LWIA interest in achieving these levels, states may 

create policies to provide incentive funding and impose sanctions for exemplary and poor 

performance respectively. In addition, states may also provide technical assistance to local areas 
I-6

focusing on strategies to meet performance goals. This section will discuss the performance 

related policies developed by states, local area perception of these policies and the impact of 

state policies on local program design. 

Negotiation of Performance Levels 

The local area survey asked LWIAs to describe their perceptions of the process of negotiating 

initial performance levels with their state, including the quality of negotiation, factors 

considered, the fairness and attainability of their initial negotiated levels, and any adjustments 

that were made to these levels. States with only one local area were excluded from this analysis. 

Quality of Negotiation 

The consensus of local areas is that the initial negotiation process with their state was neither 

extensive nor adequate. Nearly one third of local areas surveyed (29%) stated that the process 

involved no negotiation at all. This is not surprising given that our analysis of federal and state 

policy documents indicated that 13 of the 42 states with more than one local area simply 

assigned the statewide levels to all local areas. An additional 55% of local areas felt that the 

process involved only minimal negotiation. Given the minimal level of negotiation reported by 

nearly all LWIAs, it is not surprising that two-thirds (66%) of local areas did not feel that the 

state gave adequate attention to the issues raised by the local representatives during the 

negotiation process. Overall perceptions regarding the limited nature of the initial negotiation 

process generally were in line with findings from our first round of site visits. 

Factors Considered in Negotiation 

Six states use formal adjustment models to determine performance levels for the local areas. 

More common is informal analysis of factors such as local economic conditions and the 

characteristics of customers served. Even so, only 50% of local areas reported that states 

consider local economic conditions when determining performance levels. An even smaller 

percentage of local areas reported that states considered the characteristics of their customers 

(36%) or the services they provide (24%). Finally, 47% of local areas stated that the negotiation 

considered JTPA or other historical data. 

Fairness of Initial Negotiated Levels 

Overall, nearly two-thirds of local areas (66%) reported that their initial negotiated levels were 

‘somewhat fair’. Not surprisingly, perceptions of fairness were related to the extent that local 

areas felt the states considered issues of importance to them in the negotiation process. For 

instance, as shown in Exhibit I-1, 29% of local areas who felt that the state gave adequate 

consideration to local issues reported that their initial levels are ‘completely fair,’ in comparison 

to only 2% of local areas who felt the state did not give adequate consideration to local issues. 
I-7

Exhibit I-2: 

LWIA Perception of the Fairness of Initial Levels 

Fairness of Initial 

Levels All LWIAs 

LWIAs who feel that the 

State gave Adequate 

Consideration to Local 

Issues in the Negotiation 

Process 

LWIAs who feel that the 

State did not give 

Adequate Consideration 

to Local Issues in the 

Negotiation Process 

Not at All Fair 24% 4% 33% 

Somewhat Fair 66% 67% 65% 

Completely Fair 11% 29% 2% 

Perceptions of fairness are also related to whether specific factors were considered in the 

negotiation process. Exhibit I-3 displays LWIA perceptions about the fairness of their initial 

levels. Local areas are grouped according to whether local economic conditions, customer 

characteristics and services were considered in the negotiation process. 

Exhibit I-3: 

Perception of Fairness of Initial Levels by Items Considered in Initial Negotiation Process 

36%

11%

31%

9%

27%

13%

57%

73%

61%

73%

64%

70%

6%

16%

7%

17%

9%

18%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Conditions Not

Local Economic

Considered

Local Economic

Considered

Conditions

Customer

Characteristics Not

Considered

Customer

Characteristics

Considered

Considered

Services Not

Services

Considered

Percentage of LWIAs

Initial Levels are Completely Fair

Initial Levels are Somewhat Fair

Initial Levels are Not At All Fair
I-8

Consideration of each item (economic conditions, customer characteristics or services) is 

associated with an increase in the percentage of local areas who report their levels as being 

‘completely fair’. Correspondingly, the percentage of local areas who report that their levels are 

‘not fair at all’ is higher when each of the three items was not considered in the initial negotiation 

process. 

Nearly two-fifths of LWIAs (37%) reported that more than one of the items shown above were 

considered in their initial negotiation process. The overlap in terms of which items were 

considered in the negotiation process makes it difficult to isolate the relationship between each 

specific factor and local perception regarding the attainability of performance levels. However, 

we conducted additional analyses to examine whether the number of factors (economic 

conditions, customer characteristics and services) considered in the initial negotiation process is 

related to LWIA perception regarding the fairness of the initial levels. The results of this 

analysis are shown in Exhibit I-4 below. 

Exhibit I-4: 

Perceived Fairness of Initial Levels by Number of Items Considered in the Initial 

Negotiation Process 

37%

17%

9% 10%

58%

70%

76% 71%

5%

13% 15% 19%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 items 1 item 2 items 3 items

Number of Items Considered

Percentage of LWIAs

Initial Levels are Completely Fair

Initial Levels are Somewhat Fair

Initial Levels are Not at All Fair
I-9

We see from Exhibit I-4 that, when at least one of the three items is considered, local areas are 

significantly more likely to believe that the initial negotiations were somewhat or completely 

fair. However, the consideration of additional items is not associated with a significant increase 

in the perceptions of fairness. 

Attainability of Initial Levels 

In spite of the fact that nearly all LWIAs felt that the initial negotiation process was not 

substantial, 75% of the LWIAs reported that their initial negotiated levels are most likely 

attainable9. This may be due to the fact that, as we will discuss in subsequent sections, LWIAs 

have developed policies and programs that are designed to facilitate positive performance 

outcomes. 

Opinions regarding the attainability of initial levels did vary according to the extent that the state 

considered local issues in the negotiation process: 

• Of LWIAs who reported that the state gave adequate consideration to local issues, 

88% feel that their initial levels are attainable. 

• Of LWIAs who reported that the state did not give adequate consideration to local 

issues, 69% feel that their initial levels are attainable. 

More specifically, as shown in Exhibit I-5, perceptions regarding the likelihood that performance 

levels can be achieved vary according to whether economic conditions, customer characteristics 

and services provided were considered in the negotiation process. Consideration of each of these 

items is associated with an increase of at least 10 percentage points in the number of LWIAs who 

believe their initial levels are attainable. 

 

9

 This includes a very small percentage of LWIAs (2%) who reported that their levels are definitely attainable. 
I-10

Exhibit I-5: 

Percentage of LWIAs who Feel that Initial Negotiated Levels Are Attainable by Items 

Considered in the Negotiation Process 

84% 83%

86%

68%

73% 73%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Local Economic

Conditions

Customer

Characteristics

Services Provided

Percentage of LWIAs who feel that Initial Negotiated 

Levels are Attainable

Item Considered in Initial

Negotiation Process

Item Not Considered in

Initial Negotiation Process

As with perceptions of fairness, we found that the significant factor associated with an increase 

in the percentage of local areas who believe their initial levels are attainable is the consideration 

of at least one of the aforementioned items (economic conditions, customer characteristics, 

services provided). When at least one of these items was considered, 84% of local areas reported 

that their initial levels are attainable as compared to only 66% of local areas whose initial 

negotiations addressed none of these issues. The consideration of additional items is not 

associated with an increase in the likelihood that the LWIA will believe its initial negotiated 

levels are attainable. 

LWIAs who reported that their initial levels are not attainable were asked to describe the factors 

that make their levels unattainable. The most common responses are shown in Exhibit I-6 

below. 
I-11

Exhibit I-6: 

Factors that Make Initial Levels Unattainable 

Number of LWIAs 

Local Economic Conditions 54 

High Wage Jobs (Manufacturing) Replaced 

by Low Wage Jobs (Service Sector) 

20 

Participant Characteristics 16 

Insufficient Negotiation with State 11 

Use of JTPA Data in Setting Levels 8 

Total Number of Survey Respondents 

Reporting that their Levels are Probably or 

Definitely Not Attainable 

106 

By far the most common factor mentioned was difficult local economic conditions such as 

layoffs and unemployment rates that have remained high. Of those who listed local economic 

conditions as an issue, a substantial number provided additional clarification, describing a labor 

market in which relatively high wage manufacturing jobs are being replaced by lower wage jobs 

in the service sector. This condition can make it difficult to achieve the earnings replacement 

measure in particular. Smaller numbers of LWIAs reported that the characteristics of the 

customer population, such as low educational levels or multiple employment barriers, contribute 

to difficulties in meeting performance levels. Some of these LWIAs commented that 

characteristics of dislocated workers are also problematic as these customers frequently have 

substantial work histories but few transferable skills. 

Finally, 19 LWIAs felt that the negotiation process contributed to performance levels that are 

difficult to attain. Eleven LWIAs felt that the process involved insufficient negotiation, whereas 

eight LWIAs found the use of JTPA data problematic given the differences between that 

program and WIA. 

Adjustment of Initial Levels 

Although most local areas describe the initial negotiation process as minimal, the state not 

infrequently responded to these negotiations by lowering the LWIA performance levels. Overall, 

44% of LWIAs reported that the initial negotiations resulted in a decrease in their performance 

levels. While 30% of LWIAs reported that these decreases were ‘slight’, the remaining 14% 

reported that the decrease was ‘moderate’ or ‘substantial’. 

Finally, a small percentage of LWIAs (13%) reported that the initial negotiation process resulted 

in an increase in their performance levels. 
I-12

Requests for Revisions of Initial Levels 

Thus far, we have focused on the initial process used by states and LWIAs to negotiate 

performance levels for PY 2000-2003. LWIAs also had the option during the three-year period 

to ask the state to revise their initial levels. Approximately half of LWIAs (51%) asked the state 

to revise the initial negotiated levels. As shown in Exhibit I-7, the frequency with which LWIAs 

asked states to make revisions to the negotiated levels was associated with two factors: the 

amount of initial negotiation with the state and the perception of the attainability of the initial 

negotiated levels. 

Exhibit I-7: 

Factors Associated with LWIA Requests to Revise Initial Levels 

Type of Local Workforce Investment Area 

Percentage of LWIAs Requesting 

Revisions to Initial Performance Levels 

All Local Workforce Investment Areas 51% 

At Least ‘Minor’ Initial Negotiation with the State 56% 

No Initial Negotiation with the State 40% 

Initial Levels Perceived as Attainable 48% 

Initial Levels Perceived as not Attainable 63% 

Not surprisingly, LWIAs were more likely to ask the state to revise the initial performance levels 

when their levels were perceived as unattainable. The fact that LWIAs were more likely to 

request revisions when the initial process of setting the levels had involved at least minor 

negotiation suggests that one factor promoting requests for further negotiation is a belief on the 

part of the LWIA that the state will be willing to negotiate. Nevertheless, requests for revisions 

were made by 40% of LWIAs whose initial negotiation process was reportedly non-existent. 

One possible explanation for this is that these LWIAs felt that the state would be more 

responsive the second time. Indeed, LWIAs interviewed as part of the site visits expressed the 

view that the states had become more willing to engage in negotiation during the years since the 

initial implementation of WIA. 

Initial Negotiation Process: Variation Among Types of Local Areas 

Local perceptions of the initial negotiation process did not vary by local area size, extent of 

urbanization or local economic conditions. 
I-13

Incentives 

One important way in which states can potentially increase local interest in performance 

outcomes is by offering incentive funds to local areas who achieve exemplary performance 

outcomes. By the fall of 2004, of the 42 states with more than one local area, 35 had policies in 

place to provide incentive funds to local areas based on exemplary performance.10 In states that 

provide incentives for performance, 73% of LWIAs reported that they have received such an 

incentive award.11 Not surprisingly, there seems to be some association between the existence of 

state performance incentives and the likelihood that LWIAs will place special importance on 

performance outcomes: In states with formal incentive policies, 57% of LWIAs reported that it 

is ‘very important’ to exceed their performance levels, as compared to only 45% of LWIAs in 

states that do not have policies to provide performance incentives.12 

Overall, of local areas who reported that their state provides incentives for exemplary 

performance, 56% indicated that the possibility of receiving incentive funding has had an impact 

on LWIA policies and programs. Forty percent reported that the possibility of receiving 

incentives has impacted some aspect of their service design and 37% stated that incentives have 

impacted which customers they choose to serve.13 

Sanctions 

As the counterpart to incentive funding, states may also impose sanctions upon LWIAs that do 

not achieve a minimal level of performance. By the fall of 2004, of the 41 states with more than 

one local area from which we could obtain information,14 36 had developed formal policies that 

outline the process for imposing sanctions upon LWIAs for poor performance. Two of the 

remaining states are in the process of finalizing a formal sanction policy. Of the 36 state 

sanction policies, 11 include explicit references to financial sanctions. For some states, this takes 

the form of a reduction in LWIA allocation; in others, local areas are ineligible to compete for 

state incentive funds while the sanction is in effect. 

 

10 We analyzed state policies obtained from 48 states. For the remaining two states (Oklahoma and Georgia), 

information was drawn from the Local Area Survey of the WIA Performance Measurement System. 

11 This translates into 62% of all survey respondents. 

12 This difference was not quite statistically significant (p = .08). 

13 Of local areas in states with formal incentive policies, 92% reported that their state provided incentives for 

performance. The slight discrepancy may be due to the difference between a formal policy and the actual 

provision of incentives to local areas; newer local area staff may also not be aware of state incentive policies. 

14 We were not able to obtain information on Oklahoma’s policies. 
I-14

Although nearly all states have sanction policies, and over 90% of survey respondents have 

failed at least one measure since the beginning of WIA,15 only 33% of LWIAs reported 

receiving any form of sanction, including technical assistance for failing a measure and 

corrective action plans.16 Financial sanctions are typically reserved for local areas that fail the 

same measure two years in a row. While 42% of survey respondents have failed the same 

measure two years in a row,17 a very small number of local areas (13) reported receiving financial 

sanctions, such as being ineligible for incentive funds (7 LWIAs) or receiving a reduction in 

local allocation (6 LWIAs). 

Although few LWIAs have actually received sanctions and nearly none have received financial 

sanctions, the possibility of receiving sanctions has nevertheless had an impact on LWIA policies 

and programs. Of local areas who reported that their state imposes sanctions for poor 

performance, 70% indicated that the possibility of receiving sanctions has had an impact on 

customer selection and/or service design. Just over one half (54%) stated that the possibility of 

sanctions impacts which customers they choose to serve and 51% reported that potential 

sanctions have impacted some aspect of their service design.18

The existence of a state sanction policy that specifies financial sanctions is also associated with a 

small increase in the likelihood that LWIAs will use incentives and/or sanctions in contracts with 

service providers. In states that have a sanction policy outlining financial sanctions for poor 

performance, 75% of LWIAs use incentives and/or sanctions in relationships with service 

providers, as compared with 64% of LWIAs in states that do not have a policy explicitly 

mentioning financial sanctions for poor performance. 

 

15 This figure includes survey respondents from states with more than one LWIA. Through PY02, 90% of these 

survey respondents had failed at least one measure, when PY03 is included this figure increases to 91%. It is 

unlikely that local areas would have provided technical assistance or issued sanctions based on PY03 outcomes 

by the time the surveys were completed in the summer and fall of 2004. 

16 33% of LWIAs in states with more than one local area. Not surprisingly, no local areas reported receiving 

sanctions when their state did not have a policy to impose them. 

17 For PY 00-02 including survey respondents from states with more than one local area. See Footnote 5 for 

explanation of selected time frame. When PY03 data are included, 53% of survey respondents from multiple 

LWIA states had failed the same measure two years in a row. 

18 Of local areas in states with formal sanction policies, 83% reported that their state imposes sanctions. The 

discrepancy may have to do with varying local area definitions of what constitutes a sanction (e.g. financial 

reductions only vs. all forms of sanctions?), as well as the difference between a state sanction policy and the 

actual implementation of sanctions by the state. This analysis was restricted to the sample of local areas who 

reported that their state imposes sanctions because local areas who indicated otherwise were asked to skip the 

question on the impact of the possibility of receiving sanctions. 
I-15

Technical Assistance 

States also provide technical assistance to local areas in order to assist them with meeting 

performance goals. Nearly four-fifths (79%) of local areas report that they have received 

technical assistance from the state focusing on some aspect of WIA performance.19 Technical 

Assistance may be provided in several forms. The most common of these is a technical training 

session, which was provided to 76% of LWIAs. Smaller percentages of LWIAs received written 

guidance (44%) and targeted telephone assistance (24%). 

States provide performance related technical assistance for several reasons. Exhibit I-8 shows 

common reasons for receiving TA. Several local areas reported receiving TA for more than one 

reason. 

Exhibit I-8: 

Reasons LWIAs Received Performance Related Technical Assistance 

Reason for Receiving Performance Related 

Technical Assistance Percentage of LWIAs 

General TA on Performance Measures 57% 

Local Request for TA 36% 

Failure on One or More Measures 31% 

State Failed a Measure 2% 

We see from Exhibit I-8 that the most common reason that states provide technical assistance is 

to give LWIAs general guidance regarding the performance measurement system, including 

methods for improving performance outcomes. In addition, local areas are as likely to request 

TA as the state is to require it due to poor performance by local areas. These findings suggest 

that performance related TA is typically provided in a proactive manner, to facilitate positive 

performance and prevent problems before they occur. 

Local areas were also asked to describe which performance measures were covered by the 

technical assistance they received. Exhibit I-9 shows the measures most frequently addressed 

by technical assistance. 

 

19 Single local area states are excluded from analysis. 
I-16

Exhibit I-9: 

Measures Addressed by Technical Assistance Sessions 

Measure 

Number of LWIAS who Received 

Technical Assistance Addressing the 

Measure 

All Measures 172 

In addition to those LWIAs listing “All” 

Youth Measures 51 

Earnings Change/Earnings Replacement Measures 39 

Credential Measures 27 

Younger Youth Skill Attainment 8 

General Issues Only, e.g. data management, service 

design to promote performance 

32 

Total Number of LWIAs responding to the question20 298 

We see from Exhibit I-9 that it is most common for technical assistance to address all of the WIA 

performance measures. When specific measures or groups or measures were targeted, the focus 

was commonly placed on youth measures, earnings change/earnings replacement measures or 

credential measures. This is not surprising as these are the measures that local areas have 

expressed the most concern about their ability to meet.21 

Finally, a small percentage of LWIAs reported that the TA they received did not focus on the 

measures per se, but rather on general issues such as data collection and management and service 

design to ensure positive performance. LWIAs were not explicitly asked whether these general 

topics were addressed in their TA session. Some LWIAs included this information by way of 

explanation that their technical assistance did not focus on specific measures. It is very likely 

that many LWIAs whose technical assistance focused on all or specific groups of measures also 

received guidance on general information on data management and service design. 

 

20 Of survey respondents, 348 reported receiving technical assistance. Fifty of these LWIAs did not respond to the 

question asking which measures were addressed. 

21 From interviews conducted in the first round of site visits and scattered references throughout open-ended survey 

questions. 
I-17

Implementation of the Performance Measurement 

System: Measures and Data 

Implementation of the performance measurement system also involves several decisions 

regarding the specific measures and data used to capture them that may have a significant impact 

on performance outcomes. First, states and in some cases local areas must make decisions as to 

the definition of two measures: credential (for adults, dislocated workers and older youth) and 

younger youth skill attainment. Local areas also must determine whether special emphasis will 

be placed on achieving particular WIA measures. Finally, states and local areas must decide 

whether supplemental data will be used to provide documentation of customer outcomes. 

Operationalization of Measures 

Credential 

Federal WIA policy does not provide a strict definition regarding what types of certificates 

should qualify as credentials for completing training. Rather, the federal policy encourages 

states to work with local workforce investment boards to develop local credentials for the 

completion of training that equips individuals to enter or retain employment. Given the 

flexibility of the federal policy, there may be significant variation among state and local 

credential policies. These differences in policy may contribute to differences in outcomes on the 

credential measures. 

We analyzed state level policies regarding the definition of credential for 49 states.22 Of the 41 

states with more than one local area, six have implemented credential policies that allow less 

local flexibility than the federal policy, e.g. restricting the use of locally recognized credentials to 

the completion of OJT or customized training only. Given the looseness of the federal guidelines 

for the definition of credential,23 the remaining 35 states allow for considerable local flexibility in 

terms of determining what will count as a credential. However, only 62% of LWIAs in these 

states reported that their state allows for local flexibility in the definition of credential. The 

remaining LWIAs reported that a uniform state-mandated definition is required. This suggests 

that many local areas are not aware of the flexibility they apparently have in terms of deciding 

 

22 We were not able to gather information on Oklahoma’s policies. 

23 The definition of credential as outlined in TEGL 7-99 includes: nationally recognized degree or certificates or 

state/locally recognized credentials, including but not limited to high school diplomas, GEDs, post-secondary 

degrees/certificates, recognized skill standards, licensure and industry-recognized certificates, state education 

agency recognized certificates and certificates for the completion of training services that are designed to equip 

individuals to enter or re-enter employment, retain employment or advance into better employment. 
I-18

what constitutes a credential, or that states’ written policy is not always reflective of the informal 

guidance they have provided.. 

Local areas who reported that they are allowed to develop a customized definition of credential 

were asked to provide information on what types of credentials are accepted. Most local areas 

gave descriptions that mirrored the federal definition of credential outlined in TEGL 7-99.24 

However, as shown in Exhibit I-10 a small number of local areas provided additional detail 

regarding locally recognized credentials they have developed in conjunction with their workforce 

investment boards. 

Exhibit I-10: 

Types of Locally Recognized Credentials Developed by Local Areas 

Type of Locally Recognized Credential 

Number of Local Areas Using this 

Type of Credential 

Completion of Job Readiness Workshops 24 

Achievement of Specific Competencies and Skill Sets that are not 

formally recognized on state or national level 

18 

Specific Skills Valued by Local Employers 17 

The types of credentials shown in Exhibit I-10 extend the traditional concept of an occupational 

certificate to include overall job readiness and the needs of specific employers. In the first round 

of site visits, we found that locally defined credentials such as those listed in Exhibit I-10 were 

frequently used by local areas. This suggests that additional local areas may also have developed 

these or similar types of locally recognized credentials. However, the majority of local area 

responses did not provide sufficient detail for this to be conclusively determined. 

Younger Youth Skill Attainment 

Younger youth skill attainment consists of three types of goals: basic skills goals, occupational 

skills goals, and work readiness goals. As with credentials, the federal government provides 

only broad guidelines as to how each of these goals should be defined. Although we could not 

conclusively examine state-level policies related to younger youth skill attainment, survey 

responses indicate that 46% of local areas25 believe that their state allows local flexibility in the 

definition of younger youth skill attainment goals. Unfortunately, there appears to be significant 

disagreement among LWIA respondents within specific states as to the whether such flexibility 

 

24 See footnote 23. 

25 From states with multiple local areas. 
I-19

is permitted. In 33 of the 42 states with more than one local area, some LWIAs reported that the 

state provides a mandated definition of younger youth skill attainment whereas other LWIAs 

stated that local customization of the measure is allowed. This suggests that significant 

confusion exists among local areas as to state policies regarding local customization of this 

measure. 

Local areas who reported that their state allows for local customization of the younger youth skill 

measure were asked to provide their local definition of this measure. The great majority of the 

185 local areas who responded to the question gave general descriptions that did not vary 

significantly from the federal definition of younger youth skill attainment goals outlined in 

TEGL 7-9926. However, a small number of local areas gave evidence of having developed 

stricter definitions of what constitutes a basic skill goal. Twenty-eight local areas reported that 

to achieve a basic skill goal, youth must achieve an increase in basic skills equivalent to 1⁄2 - one 

grade level. It is likely that additional local areas have operationalized basic skill goals in the 

manner described above. However, the majority of LWIA responses did not provide sufficient 

detail for this to be conclusively determined. 

Measures Emphasized 

Nearly 90% of local areas reported that they emphasize all measures equally. However, a small 

number of LWIAs reported that they placed emphasis on specific measures within one or more 

of the funding streams. Adult, Dislocated Worker and Youth measures that receive special 

emphasis are shown in Exhibit I-11 below. 

 

26 The federal definition of younger youth skill attainment goals includes three types of goals: 1. A measurable 

increase in basic educational skills (basic skills goal); 2. Skills necessary for proficiency in specific occupations 

(occupational skills goal); and 3. Overall readiness for the world of work including job search techniques, 

appropriate workplace attitudes and behaviors and daily living skills (work readiness skills goal). 
I-20

Exhibit I-11: 

Number of LWIAs Emphasizing Specific Performance Measures 

Measure 

Number of LWIAs 

Emphasizing the 

Measure for 

Adults 

Number of LWIAs 

Emphasizing the 

Measure for 

Dislocated Workers 

Number of LWIAs 

Emphasizing the 

Measure for 

Youth 

Entered Employment 21 16 8 

Retention (Adult, Dislocated Worker, 

Older Youth) 

9 6 5 

Earnings Change/Replacement 11 12 8 

Credential 3 3 9 

Diploma 6 

Skill Attainment 4 

Retention (younger youth) 2 

Total Number of Survey Respondents 

that Emphasize Specific Measure(s) 

within the Funding Stream 

38 34 31 

Of the extremely small number of LWIAs who place emphasis on specific measures, entered 

employment is the measure most likely to be emphasized followed by earnings change/earnings 

replacement and credential for older youth. 

Supplemental Data 

Entered Employment and Retention outcomes are primarily tracked using the Unemployment 

Insurance wage records system. However, this system does not capture all individuals, such as 

individuals who are self-employed, federal employees and railroad and postal workers. 

Therefore, some states and local areas choose to use additional data to supplement the outcomes 

captured by UI wage records. Responses from the survey indicate that 74% of LWIAs use some 

form of supplemental data. Common sources of supplemental data are shown in Exhibit I-12 

below. 
I-21

Exhibit I-12: 

Sources of Supplemental Data Used 

Type of Supplemental Data 

Number of Local Areas Reporting Use 

of this Type of Supplemental Data 

Employer Verification 169 

Pay Stub, Tax Form, W-2 94 

Client Contact 75 

Partner Data and Administrative Records 16 

Business License 6 

Total Number of Survey Respondents that Use 

Supplemental Data 

332 

We see that the most common type of supplemental data used is verification provided by the 

employer. This may include conversations with employers and/or written documentation. 

Official documents such as pay stubs, W-2 forms and tax forms are the second primary source of 

supplemental data that is used. In addition, many LWIAs use follow up contact with clients as a 

method of documenting employment and retention outcomes. Most respondents did not specify 

as to the format of these client self-attestations, e.g. written form, telephone contact, etc. A small 

number of local areas use additional sources of data such as administrative databases or business 

licenses (for the self-employed). 

Although the great majority of local areas chose to use supplemental data, these local areas were 

divided as to their opinions of the impact these data have on their employment and retention 

outcomes. Only 40% of LWIAs who use supplemental data feel that these data have a 

substantial impact on the outcomes. The majority of LWIAs (56%) reported that the additional 

data have only a slight impact on their performance outcomes. The remaining 4% of local areas 

felt that the supplemental data do not impact their performance outcomes at all. 

Local areas that choose not to use supplemental data were asked to explain the rationale behind 

this decision. By far the most frequent response was that the state does not use supplemental 

data when determining outcomes. Therefore, some LWIAs felt that there was not sufficient 

benefit to collecting supplemental data. In addition, a small number of LWIAs reported that the 

cost and time required to collect supplemental data prohibit its use. 

Performance and Relationships with Providers and Partners 

Local areas may also take steps to ensure positive performance outcomes through relationships 

with eligible training providers, adult and youth contracted service providers and WIA partners. 
I-22

This section will examine the types of relationships that exist, the strategies that are used to 

promote performance, and the broader impacts on the WIA service delivery system. 

Types of Providers Used for Adult and Dislocated Worker Services 

For 77% of local areas, Individual Training Accounts (ITAs) are the primary method used to 

provide training services to adult and dislocated workers.27 A small percentage of local areas 

(10%) favor the use of contract service providers and the remaining 13% of local areas place 

equal emphasis on contract training and ITAs. Local areas with high unemployment (above 

6.5%) are slightly more likely to use contract training providers: 16% of local areas with 

unemployment rate above 6.5% emphasize use of contract training providers as compared to 8% 

of local areas with unemployment rates below 6.5%. Anecdotal information from the first round 

of site visits suggests that some local areas with high unemployment favor the use of contract 

training such as OJT and customized training because the employer frequently commits to hiring 

the customer at the completion of training. 

Strategies Used to Promote Performance: Eligible Training Provider List 

Customers may use ITAs to attend training programs on the Eligible Training Provider List 

(ETPL). Therefore, one key manner in which local areas can facilitate positive performance 

outcomes is by ensuring that the ETPL includes only providers with a proven track record of 

training individuals who have achieved positive employment outcomes. Over two-thirds (68%) 

of local areas review (or have the state eligibility board review) the past performance of 

providers when determining their initial eligibility for the ETPL. 

The state plays a key role in establishing ongoing performance requirements for eligible training 

providers. Most local areas have not chosen to set performance levels higher than those outlined 

by the state. While 60% of states allow local areas to establish higher levels, only 19% of local 

areas have chosen to do so28. This may be because many local areas may feel that the levels set 

by the state are sufficient to ensure positive outcomes. Indeed, 63% percent of local areas 

reported that performance requirements for eligible training providers are equal to or higher than 

local performance targets.29 An additional 25% of local areas reported that ETPL performance 

 

27 This is not surprising given that WIA legislation requires ITAs as the primary method for training delivery (WIA 

§134.d.4.G1. 

28 Between 59-61% of local areas; some responses indicating ‘other’ were unclear. 

29 Performance requirements for providers’ WIA students. This percentage is the same regardless of whether or not 

the local area set performance requirements higher than the state. 
I-23

requirements vary by the specific measure (20%) and/or population served by the provider (9%). 

For these local areas, ETPL performance requirements are in line with LWIA targets some 

percentage of the time. 

Impact of the Performance Measurement System on the Availability of Eligible 

Training Providers 

While ETPL performance requirements can help to ensure that only qualified institutions are 

used to provide ITAs, a potential risk is that the performance and corresponding reporting 

requirements might discourage even qualified providers from applying, thereby unnecessarily 

limiting customer choice. However, the performance measurement system is not currently a 

major deterrent for training providers in most local areas. Only 21% of local areas reported that 

the performance measurement system discourages numerous providers from applying for 

eligibility. A larger percentage (40%) reported that the system discourages a few providers from 

applying, and the remaining 39% indicated that the performance requirements have no impact on 

provider interest in being on the ETPL. 

Local areas who reported that the performance requirements discourage providers from applying 

for eligibility were asked to describe the reason(s) why this is the case. Exhibit I-13 shows the 

most common responses given. 

Exhibit I-13: 

Aspects of the Performance Measurement System that Discourage Training Providers 

from Applying for Eligibility 

Aspect that Discourages Potential Applicants Number of Local Areas 

Time and Cost Required to Track Outcomes 154 

Performance Measures are Difficult to Achieve 29 

WIA Funds only a Small Number of Students 29 

Total Number of Survey Respondents Reporting that the 

Performance Measurement System Discourages Providers 

from Applying for Eligibility 

273 

We see that by far the most common reason given is the reporting requirements associated with 

the performance accountability system. A substantial number of local areas reported that at least 

a few providers are deterred by the investment of time and money needed to track outcomes for 

all WIA participants. By contrast, only a small number of LWIAs reported that the actual 

performance levels discourage providers from applying. A small number of local areas offered 

the additional perspective that some providers feel that the burden associated with the 
I-24

performance measurement system is not worth their while given the fact that WIA only provides 

funding to a small number of participants. 

Although this was not specifically mentioned by local areas, the requirement that ETPL 

applicants submit historical performance information also appears to be a factor that discourages 

potential eligible training providers. Among local areas that require historical performance 

information, 65% reported that the performance accountability system discourages at least a few 

potential ETPL applicants, as compared to 54% of local areas that do not require historical 

performance information. 

Although the performance measurement system discourages numerous providers from applying 

for eligibility in only 21% of local areas, it is important to examine whether this has an adverse 

effect on the ability of this subset of local areas to provide training to customers. Exhibit I-14 

shows the percentage of LWIAs who reported that there are a sufficient number of providers for 

adult services in their local area who meet eligibility requirements and are willing to participate 

in WIA, grouped by the extent to which the performance measurement system discourages 

providers from applying for eligibility. 

Exhibit I-14: 

Availability of Adult Service Providers 

Extent to Which Performance Measurement System Discourages 

Providers from Applying for Eligibility 

Percentage of LWIAs 

who Reported that there 

are a Sufficient Number 

of Providers for Adult 

Services 

Performance Measurement System Does Not Discourage Providers 85% 

Performance Measurement System Discourages a Few Providers 80% 

Performance Measurement System Discourages Numerous Providers 65% 

All Local Areas 79% 

We see from Exhibit I-14 there is a definite relationship between the extent that the performance 

measurement system discourages service providers from applying for eligibility and the 

availability of adult service providers. When numerous providers are discouraged from applying 

for eligibility, only 65% of local areas report that there are a sufficient number of adult service 

providers, as compared to 85% when no providers are discouraged. This suggests that the 

performance measurement system can have the unintended adverse effect of limiting the number 

of available adult providers with the consequence that there are not sufficient providers to meet 

customer needs. 
I-25

This is not to suggest that the performance measurement system is inappropriately weeding out 

adult providers. It may be that the providers who choose not to apply for eligibility do not have 

the capability to successfully serve WIA participants. In this case, the problem is not the 

performance accountability system, but rather the lack of appropriate providers. However, as 

was shown in Exhibit I-13 , it is not the performance outcomes themselves, but the extensive 

WIA reporting requirements that are most frequently listed as a factor that discourages providers 

from applying for eligibility. If the reporting requirements were scaled back and the 

administrative burden on providers lessened, local areas might find that additional qualified 

providers would be willing to apply for eligibility and subsequently serve WIA customers.30 

Types of Youth Service Providers 

While ITAs may be occasionally used to provide training to older youth, youth programs are 

typically offered by contract service providers. These programs are frequently provided by 

organizations with expertise in youth development and may operate quite independently from the 

rest of the One-Stop system. Exhibit I-15 shows common types of organizations with whom 

local areas contract to provide services and programs for youth. 

Exhibit I-15: 

Types of Organizations Used to Provide Youth Services 

Type of Provider 

Percentage of Local Areas 

that Use this Type of 

Provider 

Average Number of 

Providers Used (for local 

areas that use this type of 

provider) 

Community-based Organizations 77% 4.0 

Secondary Schools (e.g. High Schools) 59% 3.4 

Post-Secondary Schools (e.g. Community 

Colleges, Universities) 

37% 2.5 

For-profit Providers 34% 2.1 

Other Types of Organizations 22% 2.7 

We see from Exhibit I-15 that community-based organizations are the type of agency most 

frequently used to provide youth services. CBOs are used by more than 75% of local areas, the 

largest percentage for any type of provider. In addition, the average number of CBOs used is 

 

30 For a further discussion of these issues, please see a recent report (completed for DOL, ETA), D’Amico and 

Salzman, An Evaluation of the Individual Training Account/Eligible Training Provider Demonstration: Final 

Report (2004). 
I-26

higher than for other types of agencies. Secondary schools are the second most frequently used 

type of provider, receiving contracts from nearly 60% of local areas. Nearly one fourth of local 

areas contract with other types of organizations to provide youth services, most of which are 

state or local government agencies. 

There is significant variation in the total number of youth providers used by local areas. While 

10% of local areas use only one provider, 11% of local areas use 15 providers or more. The 

most providers used by any survey respondent is 13331. Not surprisingly, both the overall 

number of providers and the number of different types of providers vary by the size of the local 

area. Exhibit I-16 shows the average number of youth providers and provider types, grouped by 

local population size. 

Exhibit I-16: 

Average Number of Youth Providers and Number of Different Types of Youth Providers 

by Size of Local Area 

Population of Local Area 

Average Number of Youth 

Providers 

Average Number of Different 

Types of Providers32

Less than 200,000 4.4 1.8 

200,000-400,000 6.3 2.3 

400,000-1,000,000 8.6 2.5 

More than 1,000,000 19.8 2.9 

All Local Areas 7.5 2.3 

As would be expected, larger local areas contract with a greater number of providers, 

representing a more extensive variety of provider types33. However, the average increase of 15 

providers between the largest and smallest local areas only results on average in one additional 

type of provider. It is not surprising that a local area that contracts with only four providers 

would only use two types of providers. However, one might expect that a local area contracting 

with 20 providers would use four or five different types of providers. These findings suggest 

 

31 The median number of youth providers per local area is 5 and the median number of different types of providers 

used is 2. Providers listed as ‘other’ are considered to be one type for the purpose of this analysis. 

32 Providers listed as ‘other’ are considered to be one type for the purpose of this analysis. 

33 The overall associations between population size and both number of providers and type of providers are 

statistically significant (using OLS regression). Using chi square, for number of providers, local areas with 

populations over one million are significantly different from the other local areas; for number of types of 

providers, local areas with populations under 200,000 are significantly different from other local areas. 
I-27

that larger local areas tend to contract with multiple providers of the same type rather using the 

entire spectrum of available types of providers. 

Strategies Used to Promote Performance: Selection and Contracts 

with Youth Providers 

Similar to eligible training providers, a variety of strategies are frequently used by local areas to 

promote positive performance among youth service providers: 

• In order to gauge provider ability to achieve desired outcomes, 82% of local 

areas require youth provider applicants to submit historical performance 

information. 

• Once youth providers are selected, 84% of local areas use contracts that include 

explicit performance requirements. 

Performance requirements for youth service providers are strongly linked to local area 

performance targets. Of local areas that include explicit performance requirements, 66% require 

that providers achieve outcomes equal to the LWIA performance levels. An additional 23% of 

local areas set performance targets for youth service providers higher than local area targets. The 

remaining 11% local areas that include explicit performance requirements determine the 

performance levels on a program by program basis; only one local area specified that 

requirements for youth providers are unilaterally below the LWIA’s performance levels. 

The likelihood that local areas use explicit performance requirements in contracts with youth 

service providers varies according to both the local poverty rate and the size of the local area. 

Over 90% (92%) of local areas with poverty rates above 15% use explicit performance 

requirements in youth contracts, as compared to 81% of local areas with poverty rates below 

15%. While use of performance requirements in youth contracts is very common among all 

LWIAs, local areas in high poverty areas may want to be especially careful that their service 

providers are committed to achieving positive employment outcomes for the low income, often 

high risk youth served by WIA. 

Exhibit II-17 shows the percentage of local areas using explicit performance requirements in 

relationships with youth service providers, grouped by the size of the local area population. 
I-28

Exhibit I-17: 

Use of Explicit Performance Requirements in Contracts with Youth Providers by Size of 

Local Area 

Size of Local Area 

Percentage of Local Areas that Use Explicit Performance 

Requirements in Contracts with Youth Service Providers 

Less than 200,000 75% 

200,000-400,000 83% 

400,000-1,000,000 88% 

More than 1,000,000 91% 

From Exhibit 17, we see an overall statistically significant association between population size 

and the probability that the local area will use explicit performance requirements in contracts 

with youth service providers.34 Because large local areas contribute substantially to their state’s 

overall performance outcomes, they may feel extra pressure to achieve their performance targets. 

Holding youth providers accountable to the same targets is one way to ensure that these levels 

will be reached. 

Impact of the Performance Measurement System on the Availability of Youth Providers 

As with the ETPL, while the aforementioned strategies may ensure that only the most qualified 

providers serve WIA youth, the requirements may also have the unintended effect of decreasing 

the number of qualified providers that are interested and willing to participate in WIA programs. 

Fortunately, only 16% of local areas reported that the performance accountability system 

discourages numerous youth providers from responding to requests for proposals. An additional 

one-half of local areas (46%) did report that the performance accountability system discourages a 

few youth providers from applying. 

Local areas who reported that the performance measurement system discourages youth providers 

from submitting proposals were asked to explain why this is the case. Common responses are 

shown in Exhibit I-18 below. 

 

34 There is an overall statistically significant association using logistic regression. Using chi-square, the only 

statistically significant difference among pairs of groups is between local areas with less than 200,000 and those 

with populations between 400,000 and 1,000,000. The number of LWIAs with populations larger than 1,000,000 

is very small, rendering the difference between this group and those with populations less than 200,000 not quite 

statistically significant (.06). 
I-29

Exhibit I-18: 

Aspects of the Performance Measurement System that Discourage Youth Providers from 

Submitting Proposals 

Aspect that Discourages Youth Providers 

Number of Local 

Areas 

Time and Cost Required to Track Outcomes 76 

Concern about Achieving Performance Levels 66 

Too Many Reporting and Performance Requirements for the Level of 

Funding Provided 

16 

Performance/Tracking Requirements are Especially Difficult for New 

and/or Small Providers 

10 

Total Number of Survey Respondents Reporting that the 

Performance Measurement System Discourages Youth Providers 

from Submitting Proposals 

268 

We see from Exhibit I-18 that, similar to ETPL applicants, the extensive reporting requirements 

are the aspect of the performance accountability system that most frequently discourages youth 

provider applicants. However, a significant number of local areas also reported that concern 

about the performance standards themselves also discourages potential youth service providers. 

A few local areas specifically mentioned that the use of performance based contracts (where 

payment is withheld if specified outcomes are not achieved) contributes to this concern. A 

smaller number of local areas provided the additional response that the performance 

requirements are viewed as excessive given the level of funding that is available through WIA. 

Finally, a few local areas mentioned that the performance and/or reporting requirements are 

especially difficult for recently established and small providers and serve to deter these providers 

from submitting proposals.35

Although local areas were not specifically asked to comment on non-performance related aspects 

of WIA that discourage youth providers, some local areas chose to provide this information. 

Twenty three local areas commented that WIA requirements such as the strict eligibility criteria 

and the 10 required program elements serve to discourage youth providers from submitting 

proposals. Nine local areas also noted that the minimal amount of WIA funds also serves to 

deter providers, irrespective of the performance measures. 

 

35 We did not find that either the use of explicit performance requirements in youth contracts or the requirement 

that youth applicants include historical performance information when submitting proposals were associated 

with an increased likelihood that any or numerous youth providers are discouraged from submitting proposals. 

This is not surprising due to the fact that these policies are used by nearly all local areas and are in essence 

standard elements of the performance accountability system; therefore an adequate control sample does not truly 

exist. 
I-30

As we have mentioned, 62% of local areas reported that the performance measurement system 

discourages at least a few providers, although three-fourths of these local areas did not feel that 

the accountability system discourages numerous providers. The question thus arises as to 

whether the fact that the performance measurements system deters some number of providers in 

most local areas has an impact on the overall availability of service providers in those local 

areas. Exhibit I-19 shows the percentage of LWIAs who reported that there are a sufficient 

number of youth service providers in their local area who meet eligibility requirements and are 

willing to participate in WIA, grouped by the extent to which the performance measurement 

system discourages providers from submitting proposals. 

Exhibit I-19: 

Availability of Youth Service Providers 

Extent to Which Performance Measurement System 

Discourages Youth Providers from Submitting 

Proposals 

Percentage of LWIAs who Reported 

that there are a Sufficient Number of 

Youth Service Providers 

Performance Measurement System Does Not Discourage 

Youth Providers 

80% 

Performance Measurement System Discourages a Few 

Youth Providers 

67% 

Performance Measurement System Discourages Numerous 

Youth Providers 

34% 

All Local Areas 67% 

We see from Exhibit I-19 that there is a very strong relationship between the extent to which the 

performance accountability system discourages potential youth providers and the availability of 

youth providers. Local area responses to open-ended questions concur with these findings. 

When local areas who report that there are not a sufficient number of youth service providers 

were asked what factors contribute to this situation, the second most common response was WIA 

regulations, performance and reporting requirements. 

Of specific concern is the finding that, among local areas who report that the performance 

measurement system discourages numerous providers, only 34% believe there are a sufficient 

number of youth providers who are willing and eligible to participate in WIA. While this applies 

to a small percentage of local areas (only 16% of LWIAs feel that the performance measures 

discourage numerous applicants), the findings nevertheless highlight the issue that when many 

youth providers in a local area have difficulty with the performance accountability system, 

recruiting and maintaining enough providers becomes a challenge. 
I-31

As was discussed in the section on eligible training providers, these findings do not necessarily 

suggest that the performance measurement system is inappropriately screening out youth 

providers. It may be that the providers who choose not to apply do not have the ability to 

implement a successful WIA program. In this case, the problem is not the performance 

accountability system, but rather the lack of capable providers. However, as was shown in 

Exhibit I-18, the most frequent reason youth providers are discouraged by the performance 

measurement system is the extensive reporting requirements. If the reporting requirements were 

decreased, local areas might find that additional qualified providers would be more inclined to 

submit proposals to serve WIA youth. 

Strategies Used to Promote Performance: Incentives and Sanctions 

in Contracts with Service Providers 

Local areas also promote performance by including performance-based incentives and/or 

sanctions in contracts with providers. Two thirds of local areas reported that they use incentives 

and/or sanctions in their contracts with service providers.36 Common forms of incentives and 

sanctions are listed in Exhibit I-20 below. 

Exhibit I-20: 

Forms of Incentives and Sanctions Used in Contracts with Service Providers 

Type of Incentive/Sanction Number of LWIAs 

Decision whether to Renew Contract is based on Performance 

Outcomes 

170 

Termination of Contracts for Poor Performance Outcomes 54 

Payment Increase or Decrease based on Performance Outcomes 112 

Corrective Action Plan 36 

Share in Incentive Funding Awarded to Local Area 13 

Total Number of Survey Respondents that use Incentives/Sanctions in 

Contracts with Providers 

298 

We see from Exhibit I-20 that local areas define incentives and sanctions broadly. The most 

common form of incentive/sanction is the possibility of renewal at the conclusion of the contract. 

These local areas reported that they would only issue additional contracts to providers that have 

achieved satisfactory performance outcomes. Approximately one third of these local areas take 

 

36 We found that what some local areas consider to be incentives, others consider as sanctions, and vice versa. 

Therefore these two strategies were combined for the purpose of tabulation. 
I-32

the additional step of terminating the relationship mid-contract if the provider is not meeting 

performance targets. The second frequent type of incentive/sanction is payment based on 

performance outcomes. In such cases, a portion of the contract payment is typically paid upon 

the achievement of outcomes specified in the contract. If outcomes are not achieved, this 

payment is withheld. 

When asking about selection of providers and use of incentives and sanctions, the survey did not 

distinguish between traditional contracted service providers and eligible training providers. 

Open ended survey responses as well as interviews from the first round of site visits indicate that 

some local areas do use a type of sanction in relationships with providers that receive payment 

through ITAs. In such situations the ITA is the equivalent of a performance based contract to 

serve one individual; payment is withheld until the customer has completed training and 

achieved positive employment outcomes. 

Two findings from the first round of site visits provide additional context in understanding the 

use of incentives and sanctions in contracts with service providers. First, we found that 

performance based contracts typically include a broad spectrum of performance measures that 

include the WIA performance measures and other measures to assess customer flow and real 

time performance outcomes. Following this model, the survey did not restrict responses to 

incentives/sanctions that are used specifically to promote positive outcomes on the WIA 

performance measures. Secondly, we found that existence of sanctions does not always indicate 

that they will be enforced. 

The use of incentives/sanctions in relationships with service providers varies according to the 

size and geographic location (urban vs. rural) of the LWIA. Exhibit I-21 shows the percentage 

of survey respondents that use incentives/sanctions in contracts with service providers, grouped 

by the percentage of the local population that lives in urban areas.37 

 

37 The U.S. Census definition of urban, which includes urbanized areas and urban places is used throughout this 

report. To provide some context, nationwide, 79% of individuals live in urban areas. Approximately 60% of 

local areas have populations that are at least 79% urban. 
I-33

Exhibit I-21: 

Use of Incentives/Sanctions in Contracts with Service Providers by Extent of 

Urbanization 

Percentage of Population 

that lives in an Urban Area 

Percentage of Local Areas that Use Incentives/Sanctions 

in Contracts with Service Providers 

Less than 50% 52% 

50%-90% 69% 

More than 90% 75% 

We see from Exhibit I-21 that a greater degree of urbanization is associated with an increased 

likelihood that the local area will use incentives/sanctions in relationships with contracted service 

providers. Local areas serving predominantly rural populations stand out as being substantially 

less likely to use incentives/sanctions than other local areas38. 

Exhibit I-22 shows the percentage of local areas that use incentives/sanctions in contracts with 

service providers grouped by the size of the local area. Here we see that population size is also 

associated with an increased likelihood that the local area will use incentives/sanctions in 

relationships with contracted service providers. This pattern is most dramatic for local areas 

serving populations greater than one million, who are particularly likely to use 

incentives/sanctions in contracts with service providers.39

Exhibit I-22: 

Use of Incentives/Sanctions in Contracts with Service Providers by Size of Local Area 

Size of Local Area 

Percentage of Local Areas that Use Incentives/Sanctions 

in Contracts with Service Providers 

Less than 200,000 56% 

200,000-400,000 66% 

400,000-1,000,000 68% 

More than 1,000,000 94% 

 

38 This overall association is based on logistic regression analysis. Using chi-square tests, local areas that serve 

populations that are under 50% urban are significantly different from those with populations that are more than 

50% urban. 

39 This overall association is based on logistic regression analysis. Chi-square tests reveal that local areas serving 

populations greater than one million are significantly different from smaller local areas. 
I-34

The information from Exhibit I-21 and Exhibit I-22 together form a cohesive pattern that 

indicates that use of incentives and sanctions is least common in the smallest and most rural local 

areas, and is most common in very large LWIAs, which are typically located in the largest 

metropolitan areas. This pattern is likely the result of two factors. First, because the 

performance of very large local areas has a very strong impact on the overall performance of the 

state, these local areas may feel additional pressure to achieve performance outcomes. 

Incentives and sanctions are one way to ensure provider cooperation with efforts to achieve 

performance targets. Secondly, rural areas, who have fewer providers to choose from, may feel 

reluctant to sanction and potentially alienate the local providers that are willing to participate in 

LWIA. 

Impact of Incentives and Sanctions on the Availability of Service Providers 

We did not find any significant relationships between the use of incentives/sanctions and the 

availability of adult or youth providers or the likelihood that eligible training providers or youth 

providers are discouraged from participating in WIA. 

Availability of Adult and Youth Providers: Additional Considerations 

Thus far we have focused on the relationship between the performance measurement system and 

the availability of adult and youth service providers. However, the geographic location of the 

local area also appears to contribute to the availability of adult and youth service providers. 

Exhibit I-23 shows the percentage of LWIAs reporting that there are a sufficient number of adult 

service providers and youth service providers grouped by the percentage of the population that 

lives in urban areas. 

Exhibit I-23: 

Availability of Adult Service Providers by Urbanization of Local Area 

Percentage of 

Population that lives in 

an Urban Area 

Percentage of LWIAs Reporting that 

there are Sufficient Number of Adult 

Service Providers 

Percentage of LWIAs Reporting 

that there are Sufficient Number 

of Youth Service Providers 

Less than 50% 79% 68% 

50%-90% 74% 61% 

More than 90% 90% 78% 

We see from Exhibit I-23 that the LWIAs with the largest proportion of urban residents are also 

the most likely to report that there are a sufficient number of adult and youth service providers. 

This suggests that the most densely populated areas also have the most available service 

providers per capita. We see a slight trend that indicates that predominantly rural areas have 
I-35

more available providers than areas that are predominantly urban but with a significant rural 

population. However, differences among these groups are not statistically significant.40 

Open ended responses support the analysis presented above. LWIAs that reported that there are 

not a sufficient number of adult and/or youth service providers were asked what factors 

contribute to the lack of providers in their area. The most common response was that the rural 

location of the LWIA makes it difficult to locate a sufficient number of service providers. 

Analyses we have presented thus far indicate that complex relationships exist among the 

strategies designed to enhance performance outcomes, provider responses to these policies, the 

size of and level of urbanization of the local area and the availability of service providers. In 

Chapter II of this report, we will conduct multivariate analysis to determine the impact of these 

specific factors on customer outcomes. 

Performance Measures and Relationships with WIA Partners 

WIA partners also play a key role in the implementation of WIA Partners help WIA staff to 

provide services to customers in two primary ways. First, the responsibility of providing core 

and intensive services may be shared among partner agencies. Second, WIA registrants may be 

co-enrolled with other partners who provide services that customers need. Strong collaboration 

and appropriate use of partners can increase the likelihood of achieving performance outcomes. 

Role of Partners in Providing Core and Intensive Services 

There is considerable diversity among local areas in how they have allocated the responsibility 

for providing core and intensive services. Most local areas (67%) share the responsibility of 

providing core services with multiple partners including the Employment Service (ES). WIA is 

primarily responsible for providing core services in only 22% of local areas. Interestingly, an 

even smaller percentage of local areas (10%) reported that ES alone is primarily responsible for 

providing core services. 

WIA resources are used more heavily to provide intensive services. Two-thirds of local areas 

reported that WIA is primarily responsible for providing intensive services, including 28% of 

local areas who reported that intensive services are exclusively provided by WIA. The most 

urbanized local areas are even more likely to use WIA to provide intensive services; 76% of 

 

40 In interpreting these results, it should be noted that census definitions of ‘urban’ and ‘rural’ do not completely 

coincide with local WIA staff members’ more casual use of these terms. For instance, Tulare County staff 

describe the community as a ‘rural’, although according to census definitions, 81% of the population lives in 

urban areas. It is not inappropriate to consider all local areas whose population is less than approximately 90% 

urban as facing some of the same issues as the most rural areas. 
I-36

these areas report that intensive services are primarily provided by WIA. The second common 

model, used by 31% of local areas, is to use provide intensive services through multiple partners 

including ES. 

The strategies used to provide core and intensive services described above can be combined into 

five general models of service delivery. Exhibit I-24 shows the percentage of local areas that use 

each of these models. 

Exhibit I-24: 

Models Used by Local Areas to Provide Core and Intensive Services 

ES provides Core 

and Intensive

2%

ES provides Core, 

WIA provides 

Intensive

6%

WIA provides Core 

and Intensive

22%

Multiple Partners 

provide Core, WIA 

provides Intensive

37%

Other Model

5%

Multiple Partners 

provide Core and 

Intensive

28%

 

We see from Exhibit I-24 that partners play a substantial role in the provision of WIA core and 

intensive services. Two of the three most common models of service delivery heavily involve 

multiple partners, either as providers of core and intensive services or as providers of core 

services only. Under the third frequently used model, WIA has primary responsibility of 

providing both core and intensive services. However this model is the only design that does not 

involve partners in the delivery of WIA services and is used by less than 25% of local areas. 

Because partners play a significant role in providing WIA core and intensive services, partner 
I-37

responses to and involvement with the performance measurement system are particularly 

important and can substantially impact customer outcomes. 

Impact of the Performance Measurement System on Partner Involvement in the 

One-Stop System 

Survey respondents were asked to describe the extent to which the performance measures impact 

partner willingness to be actively involved in the One-Stop system. The majority of local areas 

(62%) reported that the performance measures do not impact partner willingness to be involved 

in the One-Stop system. In addition, 10% of local areas reported that the performance measures 

actually increase partner willingness to be involved in the One Stop. However, 28% of local 

areas reported that the performance measures reduce partner willingness to be actively involved 

in the One Stop system. In sum, the performance measurement system does not appear to have a 

negative impact on partner integration in most local areas, although a significant subset of local 

areas do struggle with this issue.41

One major factor that can influence partner response to the performance measurement system is 

the extent to which they see the performance measures as aligned with their agency goals and 

objectives. The common measures will certainly serve to increase the relevance that the 

performance measures have for WIA partners and can facilitate further partner integration into 

the One-stop system. In addition, some states have implemented ‘common’ performance 

measures that apply to some or all WIA partners, and some local areas have developed One-Stop 

system measures for which all partners are responsible. Survey responses and findings from the 

first round of site visits indicate that, when these measures exist, they tend to focus on customer 

flow and services provided rather than customer outcomes. 

However a small percentage of survey respondents (6%) did report that many WIA partners have 

the same measures as those specified in WIA. An additional 24% reported that a few of the 

partners have the same measures. Because so few local areas reported that many partners have 

the same measures as WIA, it is not possible to determine with statistical accuracy the impact 

that this has on partner attitudes toward the performance measurement system. However, the 

trends in the data do suggest partners are more accepting of the performance measurement 

system when they share the same measures as WIA: 

• When many partners have the same performance measures as WIA, only 17% of 

local areas reported that the performance measures reduce partner willingness to 

be actively involved in the One-Stop system 

 

41 76 local areas gave a response of ‘don’t know’ to this question; these responses were treated as missing data. 
I-38

• When no or a few partners have the same performance measures as WIA, 32% of 

local areas reported that the performance measures reduce partner willingness to 

be actively involved in the One-Stop system42. 

These trends suggest that the implementation of the federal common measures will increase 

partner willingness to be actively involved in the One-Stop system. 

Impact of the Performance Measurement System on Willingness to Co-Enroll Clients 

In addition to the WIA core and intensive services described above, partners also provide 

additional services and programs that may not be available through WIA, such as special 

programs for older workers and disabled customers and low-cost childcare and transportation for 

TANF clients. WIA staff can incorporate these services into the customer’s individual service 

plan through co-enrolling them in a partner program. As these additional services frequently 

make it easier for customers to achieve their employment goals, co-enrollment can increase the 

chance that customers will achieve positive employment outcomes. 

Although in principle co-enrolling customers can help to improve customer outcomes, coenrollment can also create problematic situations from a performance standpoint. As we have 

seen, partners are not typically responsible the same performance measures as WIA. Rather, 

these programs have their own goals and standards that may not align well with the WIA 

performance standards. Understandably, WIA staff may be reluctant to co-enroll their customers 

in programs whose immediate goals run counter to the WIA performance measures. In most 

local areas, this concern appears to override any beliefs about the benefits of co-enrollment for 

performance43. 

• More than half (53%) of local areas reported that the performance measures 

reduce WIA staff willingness to co-enroll clients. 

• A much smaller number of local areas (20%) reported that the performance 

measures increase WIA staff willingness to co-enroll clients 

• The remaining 27% of local areas reported that the performance measures do not 

impact WIA staff willingness to co-enroll clients. 

Decisions about co-enrollment are thus somewhat of a ‘catch 22’ situation. While the services 

provided by partner programs can help to improve customer outcomes, the disparate 

performance goals may also hinder efforts to achieve positive outcomes. The common measures 

 

42 Percentages do not differ whether a few or no partners share the WIA performance measures. 

43 77 local areas indicated that they ‘did not know’ whether the performance measures impact staff willingness to 

co-enroll customers. These responses were treated as missing data. 
I-39

will help to solve this dilemma by establishing a single set of performance goals for which all 

partner workforce programs are responsible. 

As we have mentioned, in a small percentage of local areas, many partners already have the same 

performance measures as WIA. These local areas have in effect served as informal pilot sites for 

the implementation of common performance measures and can illustrate what the likely impact 

of the common measures will be. Not surprisingly, these local areas are less likely to report that 

the performance measures decrease willingness to co-enroll clients and are more likely to report 

that the performance measures increase willingness to co-enroll clients. Exhibit I-25 shows local 

area perception of the impact of the performance measurement system on willingness to coenroll clients, grouped according to whether partners have the same performance measures as 

WIA. 

Exhibit I-25: 

WIA Staff Members’ Willingness to Co-Enroll Customers 

33%

57%

25%

25%

42%

18%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Many Partners have the Same

Performance Measures as WIA

Few/No Partners have the Same

Performance Measures as WIA

Percentage of LWIAs

Performance Measures Increase

Willingness to Co-enroll Customers

Performance Measures Do Not Impact

Willingness to Co-enroll Customers

Performance Measures Reduce

Willingness to Co-enroll Customers

We see from Exhibit I-25 that substantial differences exist between local areas where many 

partners share the WIA performance measures and those where few or no partners share the WIA 

performance measures. These findings provide additional support for the idea that providing a 
I-40

common set of performance measures will increase One-Stop integration and willingness to coenroll customers. 

Co-Enrollment Between ES and WIA: Additional Factors 

Two additional factors also influence the extent of co-enrollment between ES and WIA. These 

are the service design used to provide core services and the size of the local area. Exhibit I-26 

shows the percentage of local areas reporting that there is substantial co-enrollment between ES 

and WIA grouped by the program(s) used to provide core services to customers. 

Exhibit I-26: 

Extent of Co-Enrollment Between ES and WIA by Program(s) Used to Provide Core 

Services 

Program(s) That Provide Core Services 

Percentage of Local Areas reporting that there is 

Substantial Co-enrollment between ES and WIA 

ES (primarily or exclusively) 59% 

Multiple Partners including ES and WIA 59% 

WIA (primarily or exclusively) 33% 

All Local Areas 53% 

Not surprisingly, we see from Exhibit I-26 that substantial co-enrollment between ES and WIA 

is much more likely when ES plays a significant role in providing core services to WIA 

customers. 

Local Area Perception of the Impact of the Performance 

Measurement System 

Thus far we have discussed several strategies that local areas can use to help to ensure positive 

performance outcomes, including requests for technical assistance, supplemental data, ETPL 

requirements and performance based contracts with providers. This next section examines 

perhaps the two most important steps local areas can take to improve performance. These are the 

selection of customers that are likely to achieve positive outcomes and the use of services that 

are effective in helping customers to achieve them. While these strategies can help to improve 

performance outcomes, they may also serve to restrict the types of customers who receive WIA 

services and/or the types of services that are provided. This section will explore the impact of 

the performance measurement system on which customers are served and what services are 

provided. 
I-41

Customer Focused Strategies 

Three primary strategies may be used to focus on customers who are likely to achieve positive 

performance outcomes. Each strategy takes place at a different point in the service delivery 

process. The first of these is to target customers that are likely to achieve positive performance 

outcomes. The second strategy is to register customers who have demonstrated that they have 

the ability to achieve positive employment outcomes. The final strategy is based on decisions to 

exit customers who will achieve positive performance outcomes. 

Targeting 

Local areas may choose to target one or more specific groups of customers for service, such as 

low income individuals, older workers or displaced homemakers. This targeting may or may not 

be related to the performance measures. Overall, 55% of local areas report that they target 

particular groups for services. Common types of customers targeted for service are shown in 

Exhibit I-27. 

Exhibit I-27: 

Customers Targeted by Local Areas 

Type of Customer 

Number of Local Areas 

Targeting this Type of 

Customer 

Low Income Individuals 127 

Public Assistance Recipients 55 

Limited Education (high school dropouts, individuals 

with low literacy or limited basic skills) 

52 

Disabled Individuals 48 

Ex-Offenders 38 

Veterans 36 

Mature Workers 27 

Out of School Youth 29 

Limited English Proficient Individuals 21 

Specific Ethnic Groups 19 

Teen Parents and/or Single Parents 16 

Youth in Foster Care 15 

Total Number of Survey Respondents that Target 

Specific Customer Groups 

248 
I-42

We see from Exhibit I-27 that there is considerable variety in the types of individuals targeted by 

local areas. However, a general pattern emerges in that local areas tend to target individuals with 

specific barriers to employments, such as disability or limited English proficiency. The most 

frequently targeted customer groups, low income customers and individuals receiving public 

assistance, frequently face multiple barriers such as low education levels and limited 

occupational skills. Interestingly, customers with employment barriers are typically considered 

to be risky in terms of performance outcomes. One possibility is that the performance measures 

do not impact decisions regarding which customer groups are formally targeted. Another 

possibility is that targeting is primarily used to improve outcomes on the earnings change and 

earnings replacement measures, because, when they are able to find work, low income customers 

might be expected to have high post-program, relative to pre-program earnings. 

Local areas that target specific customer groups were asked the extent to which the performance 

measures influence their targeting practices. Results are shown in Exhibit I-28. 

Exhibit I-28: 

Influent of the Performance Measures on Local Area Targeting of Customer Groups 

Extent to Which the Performance Measures Influence 

Targeting 

Percentage of Local Areas 

(Sample: Local Areas that 

Target Customers) 

Performance Measures Do Not Influence Targeting 33% 

Performance Measures Somewhat Influence Targeting 39% 

Performance Measures Substantially Influence Targeting 28% 

We see a fairly even distribution in terms of the influence that the performance measures have on 

the targeting of specific customer groups. While 33% of local areas that target specific types of 

customers do so without regard to the performance measures, 28% report that the performance 

measures have considerable influence on which customers they choose to target for services. 

Given that most of specific groups targeted could be considered hard-to-serve customers, it is 

somewhat surprising that two thirds of local areas report that the performance measures 

influence their targeting practices. One explanation is that described above, namely that low 

income customers are targeted for the purpose of improving outcomes on the earnings change 

and earnings replacement measures. Indeed, of the local areas who indicated that the 
I-43

performance measures influence targeting, 80% reported that they target low income customers.44 

Equally plausible is that local areas target but in more complex ways than could be captured by 

the survey. In other words, it is possible that these areas target low-income customers, but only 

those who they think will get jobs (i.e., they target within the specific target groups identified in 

the survey). Unfortunately, we could not capture this level of detail within the survey to discern 

if this is, indeed, happening in a number of local areas. 

Registration 

The desire to achieve positive performance outcomes can also impact which customers are 

registered into WIA. Just over 40% of local areas (41%) reported that the performance 

measurement system influences decisions regarding when an individual is eligible to register in 

WIA. These respondents were asked to describe the specific manner in which the performance 

measures influence registration decisions. Common responses are shown in Exhibit I-29 below. 

Exhibit I-29: 

Performance Related Factors that Influence the WIA Registration Process 

Number of Local 

Areas 

Registration of Customers Likely to Achieve Positive Performance Outcomes 82 

Overall Ability to Succeed 21 

Commitment Level/Motivation to Succeed 19 

Work Readiness/Employment Barriers have been Addressed 15 

Completed Activities to Demonstrate Motivation 10 

Consider Impact of Prior Wages on Performance 18 

Time (Date) of Registration is Structured to Achieve Positive Performance 

Outcome (e.g. quarter of registration) 

21 

Registration Postponed to decrease Pre-Program Wages 14 

Prior Earnings Impact Decisions (no details provided) 11 

Total Number of Survey Respondents that Answered the Question45 122 

 

44 We calculated this figure by combining two separate questions, including whether they target low income or 

public assistance customers and the performance measures influence targeting, and whether the performance 

measures increase the likelihood that customers with low or no prior earnings will be targeted. 

45 Although 184 local areas reported that the performance measures influence decisions regarding when an 

individual is eligible to register for WIA, several chose not to answer the open-ended question that followed. In 

addition, approximately 30 local areas provided responses that were difficult to interpret or did not seem to 

directly apply to the question. 
I-44

We see from Exhibit I-29 that the performance measures impact eligibility for WIA registration 

in two general ways. First and more commonly, performance measures contribute to decisions 

regarding which customers will be registered. Customers are selected for registration in WIA 

based (at least in part) on the likelihood that they will achieve positive performance outcomes. 

Several local areas provided further clarification on specific criteria that are considered; those 

most commonly mentioned were the overall ability to achieve successful employment outcomes, 

commitment to training and/or finding employment, and work readiness. Several of the local 

areas who indicated that work readiness is a consideration stated that registration would simply 

be delayed until employment barriers have been addressed. In addition, a small number of local 

areas require customers interested in WIA to complete activities designed to demonstrate work 

readiness and/or commitment level. Finally, some local areas reported that they are less likely to 

register customers with high prior wages who would likely have a negative impact on the 

earnings change or earnings replacement measures. 

The desire to achieve positive performance outcomes may impact the time at which a customer is 

registered in a small number of local areas. Of the 21 local areas who reported that the date of 

registration is selected to achieve positive performance outcomes, the majority specifically 

mentioned that this is done to decrease the level of pre-program wages and thereby improve 

outcomes on the earnings change and/or earnings replacement measures. An additional 11 local 

areas reported that prior wages influence the registration process but did not specify whether the 

impact is in terms of which customers are registered or the date at which they are registered. 

The survey question examining the impact of the performance measures on the registration 

process was broadly worded to incorporate both the time of registration as well as criteria used in 

the selection process. In addition, many local areas chose not to answer the open-ended question 

describing the specific impact of the performance measures on registration processes. Therefore 

it is difficult to quantify exactly how many local areas screen out customers based on their ability 

to achieve positive performance outcomes. In general however, findings appear to support our 

conclusions from the first round of site visits: While ability to complete training and motivation 

to find employment are often considered before registering a customer, there is not evidence that 

local areas frequently screen out ‘hard-to-serve’ customers. 

Exit 

The final customer focused strategy that may be used to facilitate positive performance outcomes 

is strategic decision making regarding the time of exit from WIA. Nearly all local areas (81%) 

reported that the performance measures influence decisions regarding when to exit a customer 

from WIA. These local areas were asked what performance related criteria they use to determine 

the appropriate exit time. Many local areas gave very general responses to this question. While 
I-45

indicating that the exit point is selected to maximize positive performance outcomes, they did not 

specify the decision making criteria used. However, some local areas gave more specific 

responses; common themes are shown in Exhibit I-30 below. 

Exhibit I-30: 

Performance Related Criteria Used to Determine the Time of Exit from WIA 

Criteria 

Number of Local 

Areas 

Employed 106 

Employed long enough that the placement appears stable and secure 22 

Employed with Earnings Gain/Earnings Replacement 33 

Obtained Credential 15 

Achieved Positive Performance Outcome(s) (not specified) 40 

End of Quarter 40 

We see from Exhibit I-30, that by far the most common method used is to delay exit until the 

customer has been placed into a job. By delaying exit until the customer is employed, the local 

area increases the likelihood that a positive ‘entered employment’ outcome will be achieved. A 

subset of these local areas delay exit until the customer has been employed for a long enough 

period that the job placement appears to be secure and job retention seems likely. In addition, a 

smaller number of local areas delay exit until a credential has been obtained and/or a wage 

increase/replacement has been received. Finally, 40 local areas reported that they would exit a 

customer when positive performance outcomes have been achieved, but did not specify which 

measures are considered when making this determination. 

An additional 40 local areas reported that the performance measures influence the time of exit in 

a slightly different manner. These local areas stated that exits are made toward the end of the 

quarter in order to increase the likelihood of positive outcomes. It seems plausible that, in order 

for such a strategy to be successful, exit must also be delayed until the customer has obtained 

employment, though local areas did not explicitly cite this as a factor in waiting until the end of a 

quarter to exit their customers.. 

Use of Multiple Customer Focused Strategies: Targeting, Registration and Exit 

Thus far, we have discussed three customer focused strategies that local areas may use to 

facilitate positive performance outcomes. Local areas may opt to use none of these strategies, 

one of these strategies or multiple strategies together. Exhibit I-31 displays the frequency with 

which various combinations of customer focused strategies are used. Local areas who reported 
I-46

that the performance measures influence their targeting practices are considered to use targeting 

as a performance enhancement strategy. 

Exhibit I-31: 

Types of Customer Focused Strategies Used by Local Areas to Improve Performance 

Registration only

1%

Exit only

25%

Targeting and 

Registration

1%

Targeting and Exit

16%

Targeting only

3%

No Strategies 

14%

Registration and Exit

22%

All Strategies

18%

We see from Exhibit I-31 that considerable variation exists among local areas in terms of the mix 

of customer-focused strategies that are used. Overall, the largest percentage (39%) of local areas 

have chosen to use two customer selection strategies, which nearly always includes exit and 

either targeting or registration. In addition, 29% of local areas use one strategy and 18% use all 

three strategies. A small percentage of local areas (14%) do not use any customer-focused 

strategies to improve performance outcomes. 

Service Design 

In addition to using the customer-focused strategies described above, local areas may also choose 

to provide services that have proven to be particularly effective in terms of performance 

outcomes. This applies to One-Stop Center services as well as programs offered by contracted 

and eligible training providers. Local areas were asked to report the extent to which the 
I-47

performance measurement system has influenced One-Stop services and provider services. In 

addition, they were asked whether the result has been to expand or limit the services that are 

provided. Exhibit I-32 shows the influence of the performance measurement system on OneStop and provider services. 

Exhibit I-32: 

Impact of the Performance Measurement System on Services Offered by One-Stop 

Centers and Service Providers 

8%

17%

38%

28%

9%

13%

17%

38%

23%

9%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Substantially

Limits the

Services

Offered

Somewhat

Limits the

Services

Offered

No Influence Somewhat

Expands the

Services

Offered

Substantially

Expands the

Services

Offered

Percentage of LWIAS

Impact on One-Stop

Center Services

Impact on Services

Providers Deliver

 

We see from Exhibit I-32 that, across local areas, the overall influence of the performance 

measurement system on One-Stop services is very similar to the influence on services offered by 

providers. In addition, for each type of service, there is considerable variation among local areas 

in terms of their perception of the impact of the performance measurement system on services 

provided: 

• For both types of services, nearly 40% of local areas reported that the 

performance measures have had no impact at all on what services are offered. 

• Local areas were more likely to report that the performance measurement system 

has led to an expansion of One-Stop services (37%), as compared to limiting these 

services (25%). 
I-48

• For services offered by providers, local areas were equally likely to report that the 

performance measurement system has resulted in providers expanding (32%) as 

opposed to limiting services offered (30%) . 

In sum, there are clearly multiple local perspectives regarding the impact that performance 

accountability has on the quality of services provided to customers. Significant percentages of 

local areas hold the contrasting opinions that the performance measures lead to more services, 

result in fewer services, and have no impact on services at all46. This perspective varies 

somewhat according to the poverty rate of the local area. Forty percent of local areas with high 

poverty rates (above 15%) reported that the performance measurement system has had an impact 

on services provided by One Stop Centers, as compared to only 27% of local areas with poverty 

rates below 15%. Local areas whose customer base is predominantly a low income population 

with multiple barriers to employment may have made special efforts to provide One-Stop 

services that will enable hard-to-serve populations to achieve positive outcomes47. 

Finally, it is important to consider the fact that 25 and 30% of local areas believe that the 

performance measures serve to limit the services offered by One-Stop Centers and providers 

respectively. This suggests that service designs that are selected to enhance performance 

outcomes are not necessarily in the best interests of WIA customers. While these service designs 

can lead to improved performance, they may at times be in conflict with the WIA goal of 

providing a comprehensive menu of employment services that allows sufficient choice and meets 

the needs of all customers. 

Influence of the Performance Measures on the Relative Emphasis Given to Core, 

Intensive and Training Services 

The relative emphasis placed on core, intensive and training services is one specific aspect of 

service design that has been adjusted by many local areas in order to achieve performance goals. 

Overall, 67% of local areas reported that the performance measurement system has had at least 

some impact on the relative emphasis given to the three service tiers. When the performance 

measures lead local areas to adjust the relative emphasis given to the specific service tiers, this 

typically results in greater emphasis being placed on intensive services and/or training. Of local 

 

46 Overall, the impact of the performance measures on service design is somewhat more extensive than what we 

found in the first round of site visits, where respondents stressed that there was minimal impact except in terms 

of increased attention to follow-up services and more focus on specific types of training. 

47 This impact may expand or limit services. 
I-49

areas who reported that the performance measures influence the relative emphasis given to the 

three service tiers:48 

• 73% place additional emphasis on Training 

• 75% place additional emphasis on Intensive Services. 

• 56% place additional emphasis on both Intensive Services and Training 

• 27% place additional emphasis on Core Services 

The majority of local areas appear to believe that the counseling, assessment and occupational 

skill development provided through intensive and training services will lead to stronger 

performance outcomes than the light touch job placement services that are offered during core 

services. Due to the fact that intensive services are frequently seen as a bridge to training 

services, it is not surprising that additional emphasis is frequently given to both of these service 

tiers. 

Program Costs 

A final important element of local area WIA programs that can be impacted by the performance 

measurement system is the cost of operating these programs. Local areas were asked to assess 

the impact that the performance measurement system has had on several types of administrative 

and service delivery costs. Exhibit I-33 summarizes these responses. 

Exhibit I-33: 

Impact of the Performance Measurement System on WIA Program Costs 

Type of Cost 

Percentage of Local Areas Reporting that the Performance 

Measurement System Increases this Type of Cost 

Documenting Outcomes 79% 

Follow-up Services 77% 

Case Management 73% 

Providing Services 63% 

Administrative Costs 45% 

Registration Process 42% 

Selecting Contractors 40% 

 

48 Local areas who reported placing emphasis on two of the three service tiers were also included in this analysis. 

Local areas who indicated that, due to the performance measures they place greater emphasis on all service tiers 

(56), were excluded from analysis. 
I-50

We see from Exhibit I-33, that the performance measurement system frequently leads to an 

increase in costs associated with documenting outcomes and direct service provision, such as 

case management and follow up. This is not surprising, given that follow-up services and 

outcome tracking activities are frequently implemented or enhanced for the purpose of achieving 

performance goals.49 By contrast, less than half of local areas reported that the performance 

measures increase more general operating costs, such as administrative costs and the cost of 

selecting contractors.50

Use of WIA Performance Measures and Alternative 

Measures to Inform Program Management and Improve 

the Quality of Service 

Performance outcomes provide a barometer of the success of program operations and the quality 

of services offered to customers. As such, the measures serve as a source of data for local areas, 

which can indicate what aspects of service delivery are effective and any changes that should be 

made. In this final section we will focus on the factors that influence local area use of the 

performance measures to achieve local goals, such as improving the quality of services provided 

to customers. We will also examine state and local use of additional measures of program 

performance and continuous improvement programs. 

Use of WIA Measures for Program Management and to Improve 

Program Quality 

Performance outcomes can be used to improve service quality in two key ways. First, the 

outcomes can inform ongoing, day-to-day decisions regarding program management and 

operations. This requires a relatively quick analysis and application of the data. Performance 

measures can also be used to assess the extent to which outcomes improve over time and can 

help to determine what long-term programmatic adjustments should be made. 

Nearly all local areas (82%) reported that they use the performance measures to assess 

continuous improvement. This suggests that most local areas believe the WIA performance 

measures are useful indicators of overall program success. However, less than half of survey 

respondents (46%) reported that the performance measures are helpful in managing the day-today functioning of programs. The difference in responses is likely a result of the time lag 

 

49 Interestingly, the use of supplemental data is not associated with greater likelihood that these costs will be 

increased. 

50 For each type of cost a very small percentage of local areas reported that the performance measurement system 

leads to a decrease in that type of cost (less than 2% for all costs except selecting contractors [6%] ).
I-51

associated with UI data. When local areas were asked to describe reasons why the performance 

measures are not helpful for informing day-to-day management practices, by far the most 

common response was that the data are not provided in a timely enough manner to be useful in 

making real-time decisions. 

Local areas were also asked the extent to which the performance measures are central to the 

success of overall efforts to improve program quality, as well as the degree to which the 

performance measures have been helpful in these efforts. Reponses are summarized in Exhibit 

I-34 below. 

Exhibit I-34: 

Centrality and Helpfulness of the WIA Performance Measures in Overall Efforts to 

Improve Program Quality 

5%

15%

39%

42%

12%

27%

47%

13%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Not at all Minimally Somewhat Very

Degree of Centrality/Helpfulness of WIA Measures

Percentage of LWIAs

Centrality of the WIA Measures in

Efforts to Improve Program Quality

Helpfulness of the WIA Measures to

Efforts to Improve Program Quality

We see from Exhibit I-34 that the majority of local areas believe that the WIA performance 

measures are at least somewhat helpful (60%) and central (81%) to their overall efforts to 

improve program quality. However, the measures are much more likely to play a very central 

role in improvement efforts than they are to be very helpful to these efforts. Open ended 

responses provide some indication to why this is the case. When local areas were asked to 

discuss the reasons why the performance measures play a central role in guiding overall 

improvement efforts, one of the most common responses was that the performance outcomes are 

very important to the Department of Labor. In other words, as some local areas explicitly stated, 
I-52

they must achieve their performance goals if they want to have continued access to financial 

resources. The implication is that, for some local areas, the measures are the focus of 

improvement efforts because they are mandated by law, not because they help local staff to 

improve the quality of their programs. 

Local areas were given several opportunities to discuss the ways in which the performance 

measures are helpful for day-to-day program management, continuous improvement activities 

and overall efforts to improve program quality. They were also asked to describe the factors that 

serve to limit the use of the performance measures for these activities. Common responses are 

presented in Exhibit I-35 and Exhibit I-36. 

Exhibit I-35: 

Contributions of the WIA Performance Measures for Program Management and 

Improvement Efforts 

Benefit Number of Local Areas 

Performance Outcomes Indicate whether Services are Effective 90 

Performance Levels Provide a Goal or Target 74 

Performance Outcome Requirements Result in Improved Services 67 

Performance Outcome Requirements Encourage Enrollment of the 

Most Appropriate Customers 

16 

As shown in Exhibit I-35, local areas report that the performance measures help to improve 

program quality in four key ways. First, the performance levels serve as targets to strive for and 

as such provide local areas with a set of overall goals. Several local areas reported that these 

goals provide needed focus to their activities and programs. Secondly, in order to achieve the 

requirements (levels) set by the state, local areas have improved the quality of services provided, 

frequently offering additional case management and follow-up services. In addition, a small 

number of local areas reported that these requirements have also resulted in a more rigorous 

screening process to select customers for WIA services. Rather than viewing this process as 

restricting access to services, these local areas believe that the process leads to the selection of 

customers who are the most suitable and will benefit the most from WIA services. Finally, and 

most frequently cited, performance outcomes provide an indication of whether services are 

effective in helping customers achieve their employment and training goals. If performance 

outcomes are low, local areas can make the appropriate adjustments to service design. 
I-53

Exhibit I-36: 

Factors that Limit the Usefulness of the WIA Performance Measures for Program 

Management and Improvement Efforts 

Reason 

Number of 

Local Areas 

Data Issues 

Time Lag in Receiving Data 151 

Other Data Issues (not accessible, not reliable) 7 

Scope and Definition(s) of the Measures 

Do not Apply to WIA Partners 25 

Measures are Too Numerous and Complex 19 

Overall Focus of the Measures 

Measures Provide Information that is not Relevant or Necessary to Achieve Local Goals of 

Providing Quality Services that Meet Customer Needs 

118 

Local Area Focuses on Providing Services that Customers Need, not Performance 51 

Measures Discourage Service to the Hard-to Serve Customers 28 

Unfortunately, the Performance Measures Drive Program Design Too Much 17 

In contrast, we see from Exhibit I-36 that three types of issues serve to limit the usefulness of the 

performance measures for program management and improvement efforts. The first of these 

includes problems with the performance outcome data. The most frequently mentioned concern 

was the time lag associated with UI wage records, which limits local ability to analyze outcomes 

and implement appropriate changes in a timely manner. The second type of issue relates to the 

parameters of the performance measures themselves. A relatively small number of local areas 

expressed frustration that there are too many measures, that the definitions are complex and 

difficult to understand and that the measures only apply to WIA and are not applicable to the 

entire workforce development system. The new common measures will address this second set 

of concerns. 

Finally, many local areas reported that performance outcome data is not relevant or necessary for 

the local area to provide quality services that meet customer needs. Some local areas provided 

further clarification, stating that performance outcomes do not necessarily reflect the quality of 

services that were provided. However, many of these local areas simply reported that their focus 

is on providing the services that their customers need rather than on ‘the performance numbers 

game’. Other local areas offered the more critical view that the performance requirements are 

actually in conflict with local area goals, as they discourage serving customers who are a 

performance risk. Finally, a small number of local areas admitted that they give attention to 
I-54

performance outcomes that they feel would be more appropriately directed toward meeting 

customer needs. 

Additional Local Performance Measures 

In order to improve service quality and program operations, local areas may also choose to 

develop their own performance measures. Overall, 59% of local areas use their own measures as 

additional indicators of program performance. Another 7% of local areas have considered 

adopting additional performance measures, but have not implemented these measures as of yet. 

Common types of additional performance measures are shown in Exhibit I-37. 

Exhibit I-37: 

Additional Performance Measures Used by Local Areas 

Type of Measure Number of Local Areas 

Volume Measures: 

Number of Customers Enrolled 58 

Number of Customers Exited 12 

Outcome (Proxy) Measures 

Number of Job Placements 55 

Earnings Change 12 

Proxy for WIA Measures (to counteract time lag in data) 6 

Services to Customers 

Customer Satisfaction 50 

Services to Employers 41 

Cost Measures 

Measures of Program Cost 54 

Total Number of Survey Respondents that have Implemented 

Additional Performance Measures 

270 

The additional performance measures used by local areas fall into four general categories. The 

first of these are volume measures, which capture the number of customers that are served. The 

measure most frequently used to capture customer volume is the number of enrollees; a smaller 

number of local areas track customer exits. The second type of additional measure are proxy 

indicators for WIA performance outcomes. The most commonly used proxy measure captures 

the number of customers who were placed into jobs. Typically, this measures employment status 
I-55

at the time of exit from WIA.51 In addition, a small number of local areas track changes in 

customer earnings. The third type of additional measure looks more closely at the quality of 

services provided to job-seeking and employer customers. One method used to capture this 

information is through customer satisfaction surveys. Finally, local areas have implemented 

measures to determine the cost of providing services. Some of these local areas have developed 

measures that assess the return on investment for providing WIA services to customers. 

Factors that Influence the Use of Additional Measures: Attitudes Regarding the 

WIA Performance Measures 

Given that the WIA performance measures are quite extensive in scope, the question arises as to 

what factors lead to the decision to also adopt additional performance measures. The conclusion 

that immediately comes to mind is that local areas implement additional performance measures 

to address what they perceive as the limitations of the WIA performance measures. However, 

examination of survey responses indicates that the decision to adopt additional measures is not 

simply a reaction to perceived inadequacies in the WIA performance measures, but is based on a 

more complex set of factors. Exhibit I-38 shows the percentage of local areas who have 

implemented or considered adopting additional performance measures, grouped by local 

perception as to the helpfulness of the WIA performance measures in efforts to improve program 

quality. 

Exhibit I-38: 

Use of Additional Local Performance Measures by Perceived Helpfulness of the WIA 

Measures in Improving Program Quality 

Helpfulness of the WIA Performance Measures to 

Overall Efforts to Improve Program Quality 

Percentage of Local Areas that have Adopted 

or have Considered Adopting Additional 

Performance Measures 

Not at All Helpful/Minimally Helpful 60% 

Somewhat Helpful 69% 

Very Helpful 75% 

All Local Areas 67% 

Exhibit I-38 shows a pattern indicating that, when local areas believe the WIA performance 

measures are helpful for efforts to improve program quality, the local area is more likely to 

consider implementing additional measures of program performance. Although, at first glance 

 

51 Many local areas did not provide detail as to the time frame of job placement measures, those that did nearly 

always indicated that these measures are captured at time of exit. 
I-56

this pattern appears counter-intuitive, it is not illogical if we consider the concept of performance 

measures more broadly. In order to consider adopting additional measures of program 

performance, local areas must believe that performance outcomes in general are helpful to the 

process of making improvements to WIA programs. It is not surprising that local areas who 

value performance measurement in general would also be more likely to find the WIA 

performance measures helpful for making program improvements. 

However, it is likely that the specific focus and parameters of additional measures are guided by 

a desire to expand on the information that is available through the WIA performance measures. 

This idea is supported by the relationship between adoption/consideration of additional measures 

and attitudes regarding the helpfulness of the WIA measures for day-to-day program 

management: 

• Of local areas that believe the WIA measures are helpful for day-to day 

management, 63% have adopted or considered adopting additional performance 

measures. 

• Of local areas that believe the WIA measures are not helpful for day-to day 

management, 71% have adopted or considered adopting additional performance 

measures. 

The difference between the two groups of local areas is not quite statistically significant (p=.07). 

Nevertheless, the trend provides some support for the notion that the specific additional measures 

are selected in order to address the limitations of the WIA performance measures. For instance, 

additional measures often track outcomes at exit because the time lag in receiving WIA 

performance outcomes makes those measures difficult to use for day-to-day program 

management. 

Factors that Influence the Use of Additional Measures: Local Area Characteristics 

The adoption of additional performance measures also varies according to the size and 

geographic location of the local area. Exhibit I-39 shows the percentage of local areas that have 

adopted additional performance measures, grouped by the size of the local area. 
I-57

Exhibit I-39: 

Use of Additional Measures by Size of Local Area 

Population of Local Area 

Percentage of Local Areas that have Adopted 

Additional Measures 

Less than 200,000 52% 

200,000-400,000 57% 

400,000-1,000,000 63% 

More than 1,000,000 81% 

This exhibit reveals that the largest local areas are especially likely to have implemented 

additional performance measures.52 Two factors may make large local areas more likely to 

implement additional performance measures. First, due to the fact that large local areas serve 

customers through extensive networks of providers, they may feel a greater need to have formal 

proxy indicators that provide reasonable estimates of what their performance outcomes will be. 

In contrast, small local areas may be able to quickly gather the same information through 

informal discussions with the few service providers that they use. Second, large local areas have 

more extensive administrative structures, which provides them with additional flexibility in 

terms of allocating resources and assigning staff responsibilities. This flexibility is most likely 

helpful for establishing ancillary processes such as the selection, tracking and analysis of 

additional measures. 

Factors that Prevent the Adoption of Additional Measures 

As has been mentioned, 7% of local areas have considered adopting additional measures but 

have not implemented any additional measures as of yet. In addition, 14% of local areas have 

adopted only a portion of the additional measures they initially considered. These local areas 

were asked to describe the factors that lead to decisions to not adopt some or all of the additional 

measures that were considered. The response most frequently given was that implementation of 

additional measures requires significant financial resources and/or staff time. While these local 

areas understand the value of additional performance measures, they do not believe that the 

benefit is worth the investment of resources required to track and analyze the additional 

information. 

 

52 Overall differences due to population size are significant using logistic regression. Using chi-square, the only 

significant difference is between the largest areas and all other groups. 
I-58

Formal (Explicit) Continuous Improvement Programs 

Finally, local areas were asked if they have set up an explicit continuous improvement program. 

Slightly over one half of local areas (58%) have established a formal continuous improvement 

program; the majority of these programs use WIA measures to assess continuous improvement. 

Exhibit I-40 shows local use of formal continuous improvement programs as well as the use of 

WIA measures to assess continuous improvement. 

Exhibit I-40: 

Use of WIA Performance Measures and Explicit Continuous Improvement Programs to 

Achieve Continuous Improvement 

Formal Continuous 

Improvement 

Program that Does 

Not Use WIA 

Measures

9%

Formal Continuous 

Improvement 

Program that Uses 

WIA Measures

49%

 No Formal 

Continuous 

Improvement 

Program but Use of 

WIA Measures to 

Assess Continuous 

Improvement 

33%

Neither Formal 

Continuous 

Improvement 

Program nor Use of 

WIA Measures to 

Assess Continuous 

Improvement

9%

We see from Exhibit I-40 that nearly one half of local areas have established an explicit 

continuous improvement program incorporating the WIA performance measures. In addition, 

33% of local areas use WIA performance measures to assess continuous improvement but have 

not established a formal program to do so. Formal continuous improvement programs that do 

not include the WIA performance measures are uncommon. 

Local areas that have established formal continuous improvement programs were asked to 

describe the format of these programs. Common program elements are shown in Exhibit I-41 

below. 
I-59

Exhibit I-41: 

Elements of Local Area Formal Continuous Improvement Programs 

Program Element Number of Local Areas 

Use of WIA Measures to Assess Continuous Improvement 213 

Mystery Shopper Program 40 

Customer Satisfaction Surveys 36 

Use Baldrige Criteria 27 

Provide Training/Technical Assistance to Staff 13 

Focus on Employer Customers 8 

Total Number of Survey Respondents with Explicit Continuous 

Improvement Programs 

255 

We see from Exhibit I-41 that, aside from the WIA measures, the most commonly used methods 

to collect data on program quality are customer satisfaction surveys and mystery shopper 

programs. Whereas customer satisfaction surveys allow local areas to incorporate the customer 

point of view, mystery shopper programs provide an opportunity to gather detailed first-hand 

data on interactions between customers and front-line staff. Finally, a small number of local 

areas have chosen to base their continuous improvement on the Baldrige Criteria. It should be 

noted that many local areas gave very general responses when asked to describe their continuous 

improvement programs, often merely stating that there is a continual quality improvement team 

that sets goals and monitors program activities. Therefore, it is likely that continual 

improvement program elements listed in Exhibit I-41 above are in fact used more frequently than 

the data would suggest. 

Overall Reaction to the Performance Measurement 

System 

Finally, local areas were asked to provide an overall assessment of the impact of the performance 

measurement system on their WIA program. More specifically, they were asked to describe any 

unanticipated benefits and unanticipated negative impacts that have occurred as a result of the 

performance accountability system. 

Unanticipated Benefits 

A small but significant portion of local areas (22%) of local areas reported that implementation 

of the performance measurement system has brought benefits that were not anticipated at the 

outset. A wide variety of different types of benefits were described; those most frequently 

mentioned are listed in Exhibit I-42. 
I-60

Exhibit I-42: 

Unanticipated Benefits of the Performance Measurement System 

Unanticipated Benefit Number of Local Areas 

Better Service Quality 24 

More Follow-up Services 9 

More focus on job retention and career advancement 3 

Increased Availability of Reliable Data 9 

More Focus on Outcomes 8 

Having an Accountability System is Good for WIA’s Public 

Image/Public Relations 

6 

Local Area Received Incentive Funds 4 

The Measures Help to Set and Overall Goal and Focus 3 

Partners are More Integrated 3 

More Focus on Customer Satisfaction 2 

Better pre-screening of Customers 2 

Total Number of Survey Respondents Reporting that there have been 

Unanticipated Benefits 

95 

We see from Exhibit I-42 that the most frequently mentioned unanticipated benefit of the 

performance accountability system is that it has resulted in higher quality services being offered 

to customers. One way in which this has been manifested is in additional follow-up services that 

are being provided to customers. Another aspect of service quality that was highlighted is 

increased focus on long term job retention - on placing customers ‘in a career not just a job’. 

From local area responses, it appears that the retention and earnings change measures may have 

inspired these improvements in service quality. 

As Exhibit I-42 shows, numerous additional types of unanticipated benefits were mentioned. 

One that is particularly interesting is the benefit in terms of public relations. While local areas 

did not provide details, the comments imply that the public tends to have the perception that 

government programs are not held accountable to taxpayers, the business community or the 

customers they serve. In the opinion of these local areas, the WIA performance measurement 

system is serving to counteract this perception and thereby enable local areas to build stronger 

relationships with their stakeholders and the community at large. This in turn can help to bring 

in additional resources, enhance WIA services and ultimately result in improved performance 

outcomes. 
I-61

Unanticipated Negative Impacts 

Nearly all local areas (78%) reported the implementation of the performance measurement 

system has had negative impacts that were not anticipated at the outset. Specific items that were 

frequently mentioned are shown in Exhibit I-43. While local areas were asked to list 

unanticipated negative consequences, it appears that many local areas simply described more 

what they believe to be negative impacts of the performance measurement system, regardless of 

whether these were anticipated from the start of WIA. 

Exhibit I-43: 

Unanticipated Negative Impacts of the Performance Measurement System 

Unanticipated Negative Impact 

Number of Local 

Areas 

Customers with Employment Barriers and/or Individuals with High Prior Wages 

are Less Likely to be Served 

52 

Increased Costs and Staff Time 25 

Impacts Service Design and Quality 22 

Performance Measures are Confusing, especially for WIB members 17 

The Focus is on Performance rather than the Customer 5 

Total Number of Local Areas who reported that there have been Unanticipated 

Negative Impacts due to the Performance Measurement System 

25853

The themes shown in Exhibit I-43 echo local area perspectives presented throughout this report. 

Local area concerns regarding the negative consequences of the performance measurement 

system focus around two key issues. The first is that the need to achieve performance goals can 

conflict with the desire to provide quality services to all customers who need them. In order to 

meet performance standards, local areas make decisions about which customers to provide and 

what services to offer. As we have shown, at times this results in certain types of customers 

being less likely to be served. In addition, local areas may implement service designs that are 

effective in terms of performance outcomes but may not meet the needs of all customers. The 

second key issue highlighted in Exhibit I-43 addresses the fact that significant staff time and/or 

financial resources must be devoted to staff activities that support performance but do not 

directly benefit customers, e.g. documenting outcomes. Finally, although this is not a negative 

consequence per se, several local areas expressed concern that the complex nature of the 

 

53 When asked to describe the unanticipated negative impacts, many local areas simply mentioned that the 

measures are unrealistic and difficult to achieve or repeated responses to earlier questions e.g. data is too old to 

be useful, poor economy makes measures challenging, etc. 
I-62

performance measures makes them difficult for WIB members and other stakeholders to 

understand, which decreases the utility of performance outcomes for the larger community. 

Summary 

In this chapter, we have described the key results derived from our survey of local areas, which 

was administered in the fall of 2004. We received 455 responses to this survey, for an 80% 

response rate. These responses yielded several key findings to our understanding of how the 

performance measurement system has been implemented at the local level, including findings 

concerning the initial negotiations, incentives and sanctions, relationships with providers and 

partners, impacts on who is served and what services are provided, and the use of WIA and 

alternative measures to monitor and improve program quality. 

For example, the majority of local areas felt that the initial negotiation process involved some 

negotiation but that more attention could have been given to local concerns. Approximately one 

third of local areas felt that the initial negotiation process was extremely minimal or non-existent 

and a very small percentage of local areas were completely satisfied with the initial negotiation 

process. Despite this, most local areas reported that their initial levels are likely attainable. 

Additionally, most states have implemented policies to award incentives and impose sanctions 

upon local areas for exemplary and poor performance, respectively. The possibility of receiving 

sanctions has impacted some aspect of WIA program design in approximately two-thirds of local 

areas, while the possibility of receiving incentives has influenced WIA programs in a somewhat 

smaller percentage. Further, the great majority of local areas have received performance related 

technical assistance. This technical assistance tends to focus on all of the measures and is 

frequently provided to local areas regardless of whether they have failed a measure or are in 

danger of doing so. 

We also found that in more than 60% of local areas, eligible training providers are required to 

achieve local performance levels and in an additional 25% of local areas at least some providers 

are required to meet these levels. In addition, nearly all local areas hold youth service providers 

contractually responsible for achieving local performance targets. Further, two-thirds of local 

areas reported that they use incentives and/or sanctions in relationships with service providers. 

Despite this, according to most local areas, the performance measurement system does not have a 

substantial impact on the availability of eligible training providers or youth providers. However, 

a significant minority of local areas (30%) reported that the performance measurement system 

serves to discourage numerous youth service providers and/or eligible training providers from 

submitting applications to serve WIA customers. 
I-63

Most local areas make substantial use of WIA partners, especially in the provision of core 

services. However, approximately 30% of local areas reported that the performance measures 

decrease partner willingness to be actively involved in the One-Stop system and one-half of local 

areas reported that the performance measures decrease willingness to co-enroll clients. 

Responses to the survey also suggest that the performance measurement system has had a 

substantial impact on both which customers are served and what services are offered. For 

example, more than 80% of local areas make strategic decisions regarding the time of exit in 

order to improve the probability of achieving positive performance outcomes. Additionally, 

most local areas also use strategies such as targeting or the time of registration in order to select 

customers who are likely to improve performance, often focusing on customer motivation as an 

important factor in the selection process. However, we found little evidence that local areas are 

choosing not to serve traditionally hard-to-serve customers populations. Rather, targeting tends 

to focus on populations with barriers and low-income customers. Finally, two-thirds of local 

areas have made the decision to alter the relative emphasis placed on the service tiers (core, 

intensive and training), typically by placing more emphasis on intensive and/or training services. 

The vast majority of local areas (82%) report that they use the performance measures to assess 

continuous improvement, both by using the performance requirements as an overall goal by 

analyzing the outcomes to determine which services are effective. However, the use of the WIA 

performance measures in program improvement efforts is constrained by the time lag necessary 

to obtain outcomes using UI wage records outcomes. Primarily due to this time lag, only 46% of 

local areas reported that the performance measures are helpful in informing day-to-day 

management decisions. Perhaps as a result, nearly 60% of local areas have chosen to adopt 

additional measures of program performance. The most commonly used measures track 

enrollments, job placements, customer satisfaction, program costs, and services to employers. 

Finally, very large local areas are more likely to have implemented policies to ensure positive 

performance outcomes. The largest local areas are more likely to use incentives and /or 

sanctions in relationships with service providers, to make explicit contractual requirements for 

youth providers, to have implemented additional performance measures and to have established 

formal continuous improvement programs. Because these local areas frequently account for a 

substantial proportion of their states performance outcomes, they may feel additional pressure to 

achieve performance targets. 

In the next chapter, we examine how the local policies and approaches described in this chapter 

may have been impacted by state-level policies. Additionally, we explore whether variations in 

state- and local-level policies impact the customers who are served under WIA, the services they 

receive, and the outcomes they obtain. 
This page intentionally left blank.

Insert blank page here when making double-sided copies
II-1

II. EFFECTS OF STATE AND LOCAL PERFORMANCE 

POLICIES ON CUSTOMERS, SERVICES, AND OUTCOMES 

In this chapter, we attempt to assess quantitatively the effectiveness of the WIA performance 

accountability system. The major challenge to a quantitative analysis of the effectiveness of the 

WIA performance accountability system is that the system promulgated by ETA is the same 

everywhere. Therefore, there is no natural "experiment" that would allow us to compare some 

local areas operating under the WIA performance system and others operating a different system. 

Thus, a more subtle approach is called for. 

Our approach takes advantage of the fact that, although the system put forth by ETA is identical 

for all states, there is critical variation in state performance policies. Under WIA, states have 

broad flexibility in developing their own performance accountability policies, including policies 

regarding how performance levels are set and how incentives and sanctions are applied to local 

areas. By incorporating these varying factors into a quantitative analysis, we can assess how 

variations in state performance policy affect who is served, the services provided, and the 

outcomes attained. From the results of this analysis, we can infer the effects of the Federal 

performance-accountability system. 

For example, although most states provide financial incentives to local areas based on 

performance, some do not. We might find that outcomes are higher in states that provide 

incentives based on local performance. We could then conclude that performance incentives 

furthered the WIA objective of increasing outcomes. Thus, our analysis aims to identify how 

specific policies influence outcomes by examining the effect of variations in state approaches 

and procedures. 

In the remainder of this chapter, we first describe how we coded state performance policies. 

Next, we look at how we coded local policies. We then describe our methodological approach 

and data sources. Finally, we present our quantitative analysis of the effects of state and local 

performance policies on customers, services, and outcomes. 
II-2

How We Coded State Policies 

State policies are the key variables in this quantitative analysis. To develop these variables we 

obtained and coded state policies related to performance issues. 

We used a three-pronged approach to obtaining state policies. First, some states make their WIA 

policies available on a public website. For these states, we downloaded copies of the policies to 

minimize the burden placed on state staff. Second, in other states we telephoned appropriate 

state staff and requested copies of the policies. Finally, in some states, some written policies 

were not available. In these states, we conducted an unstructured interview with state staff to 

learn about their key policies. 

We requested WIA policies covering the following performance-related issues: 

• Policies for negotiating performance goals with local areas 

− If and how they take into account customer characteristics, services 

and economic conditions. For example, do they use a model? 

− Who initially proposes goals (the state or the local workforce areas) to 

start negotiation? 

• Incentives 

− Whether local workforce areas can earn incentives based on 

performance. 

− Criteria for determining which local workforce areas are eligible for 

incentives. 

− How the size of a local workforce area's incentive is determined? 

− The size of the incentive pool relative to the total allocation of local 

areas. 

• Sanctions 

− The rule for determining which local workforce areas fall into 

sanction modes for the first and second year. 

− The nature of first year sanctions. 

− The nature of second year sanctions. 

− Does the extent of sanction depend on the degree of failure? 

− Can local areas lose allocation as part of sanction? 

• Other policies that might affect characteristics of, services received by, and 

outcomes obtained by customers including: 

− Registration point. 
II-3

− Policies regarding the priority for low-income individuals. 

− Other policies regarding priorities of service. 

− Policies regarding how they define credentials. 

A senior staff person with extensive experience with performance policies was responsible for 

coding the state policies. This individual read all state policies and coded significant variations 

among states. 

Most state policies largely mimicked the existing federal policies. For example, incentive and 

sanction policies for local workforce areas often mirrored the federal incentive and sanction 

policies for states. Therefore, in coding the policies we strived to identify significant differences 

between the state’s policy and the corresponding federal policy. 

After initially reviewing state policies, we identified areas where there was a significant variation 

among states for potential use in our analysis. We coded these policies to group states with 

similar policies. These policy groupings included: 

• Single workforce area states. 

• Setting performance goals. 

• Incentive policies. 

• Sanction criteria. 

• Definition of credential. 

The specific policies we coded in each of these areas are discussed below. 

Single Workforce Area States 

Single workforce area states are subject to federal incentive and sanction policies and usually do 

not have their own incentive and sanction policies. Therefore, we did not code incentive and 

sanction policies for these states. Instead, we included a variable for single workforce area states 

in our models. There are nine states with just a single local workforce area. 

Setting performance goals 

WIA calls for states to negotiate goals with local areas, just as ETA negotiates goals with states. 

The nature and extent of this negotiation varied among states. We identified several modes for 

the negotiations. 

• State proposes local goals. States differ in whether the state or local area first 

proposes the local goals to begin negotiations. Although we coded this policy, we 

decided that the coding was not accurate due to ambiguities in the policies and 
II-4

discrepancies between how we read the state policy and what local areas in the 

state told us in the local survey. 

• State goals used for all local areas. In some states, goals did not vary among local 

areas; rather the goal that the state negotiated with ETA was applied to all local 

areas. In some states, the same goal was applied across all local areas for each of 

the performance measures; in other states, the same goal was used for selected 

performance measures, often the employment and credential rate and younger 

youth skill attainment. We identified states that used the same goal for all local 

areas using the negotiated goals reported in the states’ annual reports. 

• Adjustment model used to set initial goals or available to assist negotiations. In a 

few states, models that adjusted goals for customer characteristics or local 

economic conditions were used to help set goals. 

The pie chart in Exhibit II-1 shows the distribution of states among these modes. States with a 

single workforce area are excluded from the graph. Thirteen states (31%) set the same 

performance goals for all local areas in the state. Another 4 states (10%) set the same goals for 

at least one customer group. Just six states (14%) use a model to help set initial standards. The 

remaining 19 states (45%) use the standard negotiation process. 

Exhibit II-1 

Methods States Use to Set Performance Goals1

Same Goals for 

Some Groups

4

10%

Standard 

Negotiations

19

45%

Adjustment 

Model

6

14%

Same Goals for 

All areas

13

31%

 

1

 Excludes states with a single local workforce area. 
II-5

Incentive policies 

WIA allows, but does not require, states to provide performance incentives to local areas. 

Therefore, one of the incentive policies we coded is: 

• State provides monetary incentives to local areas based on performance. 

As shown in Exhibit II-2, we found that 35 states or 83% of the states with more than one local 

area provided performance incentives to local areas. 

Exhibit II-2: 

Provision of Performance Incentives to Local Areas2

No Incentives

7

17%

Performance

Incentives

35

83%

States that provide incentives to local areas must also develop criteria for determining which 

local areas qualify for incentives. Most states model their incentive criteria on the criteria that 

DOL uses to award incentives to states. For a state to receive incentives from DOL, average 

performance relative to goal must exceed 100% for all three customer groups (adults, dislocated 

workers, and youth) and for the two customer satisfaction measures.3 Most states model their 

incentive criteria after these federal criteria. However, some states established incentive criteria 

that were either more or less stringent than the federal criteria. Therefore, we created a variable 

that coded the stringency of incentive criteria: 

 

2

 Excludes states with a single local workforce area. 

3

 The state must also meet performance criteria for WIA Title II Adult Education and Literacy programs and Title 

I of the Carl. D. Perkins Vocational and Technical Education Act. However, these other criteria are not used by 

states when awarding incentives to local workforce areas. 
II-6

• Criteria for incentives are more/less stringent than federal criteria. This variable 

was coded as -1 for states with less stringent criteria, 0 for states that used the 

federal criteria, and +1 for states with more stringent criteria. 

Exhibit II-3: 

Criteria States Use to Qualify Local Areas for Incentive Awards4

Federal Criteria

11

31%

Less Stringent

21

60%

More Stringent

3

9%

As shown in Exhibit II-3, 11 states–or 31% of the 35 states providing incentive awards–qualify 

local areas for incentives using the federal criteria. The majority (21 states or 60%) of states 

awarding incentives use less stringent criteria. Less stringent criteria include, for example, 

policies that allow a local area to receive an incentive for one performance measure even if 

performance on other measures is below the goal. Just 3 states (9%) use more stringent criteria. 

More stringent criteria include, for example, policies that require average performance relative to 

goal to exceed 100% for all measures and not just on average for each customer group. 

Once states decide which local areas are qualified for incentives, they must also determine the 

size of the incentives. We looked at two aspects of the formulas states use to determine 

incentives: whether incentives are based on the size of the local area and whether incentives vary 

according to the extent by which performance goals are exceeded. Therefore, we coded two 

aspects of the states incentive formula: 

• Performance incentive proportional to allocation. 

• Performance incentive depends on the extent to which the goal is exceeded 

 

4

 Includes only the 35 states giving incentive awards for performance. 
II-7

In some states, local areas that qualified for incentives received awards based only on their 

performance, so that two local areas with the same performance would receive the same award, 

regardless of their size. In this case, receiving incentives would be much more important to 

small local areas than to large local areas. In other states, the award was proportional to the local 

area’s allocation. In this case, receiving an incentive would be equally important to all local 

areas, regardless of size. As shown in Exhibit II-4, 22 states, or 63% of states awarding 

incentives, give the same incentive award to all qualifying local areas; the remaining 13 states 

give larger awards to large local areas. 

Exhibit II-4: 

How States Determine Incentive Awards5

Same Awards 

All Areas

22

63%

Awards Based 

on Size

13

37%

In most states, the incentive award a local area receives does not depend on the extent to which 

the performance goal is exceeded—exceeding the goal qualifies that local area for the full 

incentives. In some states, the incentive is based, at least in part, on how far above the goal the 

local area’s performance is. Twenty-three states (66%) give incentive award to all qualifying 

local areas that do not depend on how much performance exceeds the goal. However, in some 

cases separate awards are made for each performance measure. The remaining 12 states give 

awards that increase as performance increases beyond the goal. 

Nearly half of states giving awards based on the size of the local area also gave awards that 

depended on performance beyond the goal (6 of 13 states). Only 6 of the 22 states that gave the 

same award to all local areas gave awards that were based on performance beyond the goal. 

Therefore, 16 of the 35 states giving incentives gave the same award to all qualifying local areas 

without regard to either the size of the local area or performance beyond the goal. 

 

5

 Includes only the 35 states giving incentive awards for performance. 

Awards Based 

Only on Qualifying

23

66%

Awards Based on 

Performance 

Beyond Goal

12

34%
II-8

Sanction Criteria 

According to federal criteria, a state fails performance overall if performance on one or more 

measures falls below 80% of the goal. States that fail for a first year are given technical 

assistance. States that fail for a second year based on the same performance measure are subject 

to a sanction providing for a reduction in allocation. Most states apply the same criteria for 

determining which local areas are subject to sanctions, but a few apply less stringent criteria. 

Less stringent criteria include, for example, only applying sanctions when performance on 

several measures is less than 80% of the goal. A few states use different criteria for the first and 

second years. In coding sanction criteria, we focused on criteria used to determine a second year 

of failure and coded: 

• Less stringent sanction criteria. 

As shown in Exhibit II-5, 21 states (50% of states with more than one local area) use criteria that 

are less stringent than the federal criteria. The remaining 21 states use the federal sanction 

criteria 

Exhibit II-5: 

Sanction Criteria Used by States6

Less Stringent 

Criteria

21

50%

Federal Sanction 

Criteria

21

50%

There seems to be a relationship between the use of less stringent sanction criteria and the use of 

less stringent incentive criteria. Almost all states that use less stringent sanction criteria also 

 

6

 Excludes states with a single local workforce area. 
II-9

used less stringent incentive criteria (6 of 7 states). Only 15 of the 28 states with both federal 

sanction policies and incentive awards used less stringent incentive criteria. 

Definition of Credential 

Federal guidelines give states and local areas considerable flexibility in defining credentials. 

Most states allow local areas to define credential within these federal guidelines. However, some 

states have issued guidance that provides less flexibility than the federal criteria. A few states 

have definitions that are more flexible than intended by the federal criteria. We coded: 

• Less flexible definition of credential. In addition to credentials granted by 

accredited educational institutions, Federal criteria allow local areas to award 

certificates based on “successful completion of training services ... that are 

designed to equip individuals to enter or re-enter employment, retain employment, 

or advance into better employment.” Some states limited these certificates to onthe-job and customized training or to other specific services. 

• More flexible credential criteria. A few states adopted policies that, in effect, 

stated that becoming employed was sufficient evidence that a trainee qualified for 

a credential. 

As shown in Exhibit II-6, 9 states used credential criteria that were more flexible than the federal 

criteria and 4 states used less flexible criteria. The remaining 39 states followed the federal 

criteria. 

Exhibit II-6: 

Flexibility of Credential Definitions 

Federal 

Definition

39

75%

Less Flexible

4

8%

More Flexible

9

17%
II-10

How We Measured Local Policies 

We also created variables to capture performance policies implemented at the local area level. 

We used factor analysis to develop indexes of local performance policies, based on responses to 

questions drawn from the Local Area Survey of the Performance Measurement System, 

described in the previous chapter.7

 For this analysis, sixteen survey questions were grouped into 

two categories, each of which covers an aspect of local performance policy. These two 

categories are: 

• Overall service design. 

• Relationships with providers. 

For both categories listed above, survey questions were combined into a smaller number of index 

variables, using factor analysis.8

 The local policy variables used in our models are discussed 

below. 

Overall Service Design 

Nine survey questions were used to measure the overall service design of local areas. These 

questions capture information about the adequacy of the provider network and other features 

relating to One-Stop operation.9 Factor analysis produced four factors from this set of variables. 

These factors are listed below, along with the survey questions that contribute significantly to the 

value of the factor:10

Service Design Factor 1: Use of WIA to Provide Core and Intensive Services 

This factor contains contributions from the following three survey questions: 

 

7

 In exploratory factor analysis, we also examined factors relating to several other dimensions of local policies, 

including their initial negotiation process, use of supplemental data, impact of incentives and sanctions on local 

WIA programs, customer selection and service design, and the use of additional performance measures. These 

additional factors were dropped from our final analysis, however, for one of two reasons. First, variables were 

removed if analysis uncovered a high degree of correlation with other variables in the model. Second, indices 

that had little or no significant impact on customer characteristics, services or outcomes were eliminated from 

the analysis 

8

 From the initial extract, a subset of factors were selected based on the following criteria. First, factors with 

Eigenvalues greater than 1 were selected for further analysis. Second, a sufficient number of factors were 

selected to ensure that at least 50% of the variance would be explained. Finally, the selected factors were rotated 

once in order to facilitate meaningful interpretation of the content of these factors. 

9

 The actual survey questions can be found in Appendix A. 

10 The cutoff for substantial contribution is a factor loading of .6; the cutoff for some contribution is a factor 

loading of .3. 
II-11

• The extent to which WIA is used to provide core services (substantial 

contribution). 

• The extent to which WIA is used to provide intensive services (substantial 

contribution). 

• Lack of co-enrollment between ES and WIA (some contribution). 

This factor primarily captures the extent to which WIA staff are involved with providing core 

and intensive services. As was discussed in the previous chapter, WIA is the primary provider of 

intensive services in two-thirds of local areas. WIA staff are also involved with providing core 

services in nearly all local areas; however for two thirds of local areas this responsibility is 

shared with the ES and other partners. The extent of co-enrollment between ES and WIA also 

contributes somewhat to the overall factor score. Findings from the local area survey indicate 

that extensive co-enrollment is least frequent among local areas who rely primarily on WIA to 

provide core and intensive services. 

Service Design Factor 2: Performance Measurement System Restricts the Pool of 

Available Providers 

The following three survey questions contribute to this factor: 

• Extent to which the performance measurement system discourages youth service 

providers from submitting applications (substantial contribution). 

• Extent to which the performance measurement system discourages eligible 

training providers from submitting proposals (substantial contribution). 

• Lack of sufficient number of youth service providers (some contribution). 

This factor measures the extent to which the performance measurement system discourages 

providers from applying to participate in WIA and thereby restricts the pool of available 

providers. Fewer than one-third of local areas reported that the performance measurement 

system discourages numerous providers from submitting applications and/or proposals. 

Service Design Factor 3: Emphasis on Partnerships: 

The following three survey questions contributed to this factor. 

• Extent to which partners share the same performance measures as WIA 

(substantial contribution) 

• Emphasis on use of contracted providers (as opposed to ITAs) to provide services 

to adults and dislocated workers (substantial contribution). 

• Degree of co-enrollment between ES and WIA (some contribution). 

This factor measures three aspects of service design that reflect an emphasis on formal 

relationships with WIA partners and service providers. First, the factor captures the number of 
II-12

local partners who share the same performance measures as WIA. Second, the factor measures 

the extent to which the local area uses contracted providers to offer services to adults and 

dislocated workers. As was discussed in the analysis of the local area survey, neither shared 

performance measures nor use of contracted training providers are common among local areas 

nationwide. Only 10% of local areas emphasize the use of contracted service providers over 

ITAs,11 and a miniscule 6% of local areas reported that many partners share the same 

performance measures as WIA. Third, the degree of co-enrollment between ES and WIA, also 

contributes somewhat to this factor. Survey findings indicate that co-enrollment is more likely 

among local areas who report that at least some partners (e.g., ES) share the same performance 

measures as WIA. 

A fourth service design factor (Insufficient Number of Service Providers) was initially developed 

as 41% of LWIAs reported that they lack access to a sufficient number of adult and youth 

providers who are interested in participating in WIA. However, this factor ultimately was not 

included in the regression models because it did not have a significant impact on the dependent 

variables. 

Relationships with Providers 

Seven survey questions were used to capture the role of performance in relationships between the 

local area and service providers, including eligible training providers, contracted training 

providers and providers of youth services.12 Factor analysis produced two factors from these 

seven questions. These factors are shown below along with the survey questions that contribute 

significantly to the value of the factor:13

Relationships with Providers Factor 1: Performance Related Requirements for 

Providers and Applicants: 

The following three survey questions contributed substantially to this factor: 

• Use of explicit performance requirements in contracts with youth service 

providers (substantial contribution). 

• Requirements that eligible training provider applicants submit historical 

performance information (substantial contribution). 

 

11 For adults and dislocated workers. Contracted service providers are typically used to provider services for youth. 

12 Actual survey questions can be found in Appendix A. 

13 As with the earlier examples, the cutoff for substantial contribution is a factor weight of 0.6; the cutoff for some 

contribution is a coefficient of 0.3. 
II-13

• Requirements that youth service provider applicants submit historical 

performance information (substantial contribution). 

This factor captures two types of requirements that are implemented by local areas to ensure that 

provider services will lead to positive performance outcomes. First, some local areas hold youth 

providers contractually responsible for performance targets, which are typically equivalent to the 

local area’s performance levels. Second, a review of each applicant’s performance history 

allows local areas to determine if the provider is capable of achieving WIA performance 

standards. More than two thirds of local areas require eligible training providers to submit 

historical performance information. Each of the policies that substantially contribute to this 

factor have been implemented by at least two-thirds of local areas. 

Relationships with Providers Factor 2: The Impact of Performance on Providers: 

Selection, Incentives and Sanctions 

Four survey questions contributed to this factor: 

• Extent to which performance measures influence which service providers are used 

(substantial contribution). 

• Establishment of local performance requirements for eligible training providers 

that are higher than those set by the state (substantial contribution). 

• Use of incentives and/or sanctions in relationships with providers (substantial 

contribution). 

• Use of explicit performance requirements in contracts with youth service 

providers (some contribution). 

This factor captures two additional methods used by local areas to ensure that providers will 

achieve performance goals. Whereas the previous factor measures the processes required of 

applicants and providers, this factor captures policies that translate these requirements into 

positive and negative consequences for the providers. The first method is to select new and 

returning providers on the basis of their performance outcomes. One specific way in which this 

is accomplished is by establishing rigorous performance standards for eligible training providers. 

The second method is to build incentives and sanctions into contracts with service providers, and 

on occasion into contracts with eligible training providers. Both of these methods are frequently 

used by local areas. More than 60% of local areas reported that the performance measures have 

a substantial influence on which providers are selected and two-thirds of local areas offer 

incentives and/or impose sanctions for exemplary and poor performance respectively.14 The use 

 

14 Fewer than 20% of local areas have established ETPL performance requirements that are higher than the state. 

However, as was discussed in the previous chapter, this may be a result of the state having already established 

rigorous performance requirements. 
II-14

of explicit performance requirements in contracts with youth service providers also contributes 

somewhat to this factor, illustrating that there is some degree of overlap between the policies 

captured by this factor and the previous factor. 

Methodological Approach and Data Sources 

The methodological approach used in the quantitative analysis is outlined below. In the most 

general sense, we seek to investigate the quantitative relationships between customer 

characteristics, services used, performance outcomes achieved, state and local policies, eligible 

population characteristics, and local geographic and economic conditions. Exhibit II-7 presents 

the conceptual framework for our analysis. 

Exhibit II-7: 

Conceptual Framework for Quantitative Analysis 

Exogenous Variables

State Policies

 – Performance

 – Other

Eligible Population

 – Characteristics

Economic and

Geographic Conditions

Intervening Variables

Local Policies and

Practices

 – Performance

 – Service Priority

 – Provider Relationships

 – Partnerships

Dependent Variables

Client

Characteristics

Outcomes

Services

Received

The right-hand box of Exhibit II-7 shows the dependent variables of the quantitative analysis: 

client characteristics, services received, and outcomes. The left-hand box shows the exogenous 

variables: state policies, characteristics of the eligible population, and economic and geographic 

conditions. The middle box shows intervening variables, which include local policies and 

practices. The hypothesized relationships among these variables are illustrated by the arrows 

connecting the boxes. For example, the exogenous variables affect the dependent variables 

directly, but also have effects on the intervening variables, which in turn affect the dependent 

variables. 15 

 

15 We estimated three other variants of this more general model as part of this analysis – those to estimate the total 

effects of state policies, those to estimate the effects of local policies and procedures and those to explain the 

intervening variables. The first of these is a reduced form model that relates the dependent variables to the 

exogenous variables. These models have client characteristics, services, and outcomes as the dependent 

variables. The independent variables are state performance accountability policies and other factors that affect 
II-15

Unit of Analysis 

For all the models described above the local area is the unit of analysis. All variables related to 

individual customer characteristics, service usage, and performance outcomes are aggregated to 

the local area level. Local areas are defined according to boundaries set forth in accordance with 

the designation of local service areas under the Workforce Investment Act. 

Dependent Variables 

The dependent variables in the analysis include LWIA-level indicators16 of customer 

characteristics,17 the services they received,18 and the outcomes they attained.19 

All data for customer characteristics, services received, and performance outcomes are drawn 

from the Workforce Investment Act Standardized Record Data (WIASRD). The data for models 

with customer characteristics and services received as the dependent variable include program 

 client characteristics, services and outcomes, including local area economic conditions and the characteristics of 

the eligible population. The results from these models allow us to determine the total effects of state 

performance accountability policies on clients, services, and outcomes. Most of the analyses presented below is 

based on these models of total effects. The second variant is conceptually similar to the primary model 

displayed in Exhibit II-7. The purposes of these models are to determine the effects of local program design 

elements and to examine which program design elements are potential mechanisms through which state and 

federal performance policies affect clients, services, and outcomes. In addition, we estimated models to explain 

the intervening variables. These models are different in that the dependent variables are local program design 

elements while the independent variables include state performance policies and local economic conditions and 

population characteristics. The purposes of this model are to examine how the elements of local program design 

are affected by state policies and thereby can act as mechanisms through which state policies have influence on 

clients, services, and outcomes. 

16 Because the unit of analysis is the local area, each of these dependent variables is measured as the percentage of 

individuals served by the given LWIA that exhibited a particular characteristic, received a service, or obtained an 

outcome. 

17 Among the customer characteristics we included in our analysis were: gender, ethnicity and race, age categories, 

individual with a disability, education level, limited English-language proficiency, low income (adults and youth 

only), employment status at registration, veteran (adults and dislocated workers only), public assistance receipt 

(adults and youth only), single parent, not attending school (youth only), offenders (youth only), and preprogram 

earnings. 

18 For adults and dislocated workers, we included the following service variables: received core services only, 

received training, issuance of ITA, type of training (i.e., adult education and basic skills, on-the-job training, 

other occupational training), length of participation, and cost per participant. For youth, we included: received 

educational achievement services, received employment services, received summer employment opportunities, 

received additional support of youth services, received leadership development opportunities, received follow-up 

services, length of participation, and cost per participant. 

19 For adults, dislocated workers and older youth, we included the following outcomes in the analysis: entry into 

employment, retention in employment, earnings change, earnings replacement (dislocated workers), credential 

attainment. For younger youth, we included: attainment of goals, high school diploma attainment, placement in 

education, training or employment, and retention in education, training or employment. 
II-16

exiters between April 1, 2003 and March 31, 2004, while data for models with performance 

outcomes as the dependent variables include program exiters between April 1, 2002 and March 

31, 2003. These timing conventions are adopted to provide the most recent annual cohort of 

exiters for which data are readily available for each class of dependent variables. 

When customer characteristics and services received are used as independent variables, missing 

values for these variables are set equal to the national average value for that particular 

characteristic or service. Individual records are then aggregated to the local area level so that all 

values represent local area averages for their respective variables. 

Exogenous Variables 

Exogenous variables include state performance and other WIA policies, characteristics of the 

eligible population, and local economic conditions, each of which is described below. 

State Performance Policies 

State policies are the key independent variables in the analysis. In essence the state policies 

"stand in" for federal performance policies in our analysis. It is by determining how variations in 

state performance policies affect client characteristics, services, and outcomes that we will be 

able to infer the effects of federal performance policies. While a richer set of state policies were 

explored and are described above, the state policy variables used in the analysis include:

• State provides performance incentives for local areas. 

• For states that provide performance incentives, the extent to which criteria are 

more or less stringent that federal criteria. 

• Less stringent sanction criteria for local areas compared to federal criteria for 

states. 

• Single area states for whom the above policies relating to managing local 

performance are not relevant. 

• Less flexible definitions of credentials. 

Characteristics of the Eligible Population 

The characteristics of the eligible population are likely the primary determinants of the 

characteristics of clients served in a local area. Thus, we included several variables in the 

analysis to control for these characteristics of the eligible population at the local area level.20

 

20 Among these variables, we included gender, ethnicity and race, age categories, disability, education level, 

limited English-language proficiency, veteran (adults and dislocated workers only), and single parent status. 
II-17

These data are drawn from the 2000 United States Census, Summary File 4. Data for limited 

English-language proficiency and veteran status are drawn from the population of all individuals 

ages 18-64; data for single parent status are drawn from the population of all individuals ages 15-

64. Only data covering all individuals is readily available for these variables. Data for all other 

characteristics of the eligible population are drawn from the unemployed and/or low-income 

population as this population is most relevant for analyzing the characteristics of customers 

accessing the WIA program. Data used in adult and dislocated worker models for gender, 

ethnicity and race, and age are drawn from the unemployed population ages 20 and up; data for 

education level are drawn from the unemployed population ages 18-64; data for disability status 

are drawn from the unemployed population ages 21-64. Data used in models for older and 

younger youth for gender, ethnicity and race, and age are drawn from low-income population 

ages 15-20; data for education level are drawn from the low-income population ages 18-24; data 

for disability status are drawn from the low-income youth population ages 16-20. All of these 

variables are mapped to the local area level so that data represent local area averages. 

For models with customer characteristics as the dependent variables, only the variable 

corresponding to the particular customer characteristic of interest is included in any particular 

model (e.g., in models for which veteran status is the dependent variable, the variable controlling 

for the percentage of the eligible population that is a veteran is the only control variable used). 

For models with services received and performance outcomes as the dependent variables, the full 

set of customer characteristic control variables is included in all models. 

Local Economic and Geographic Conditions 

Local economic and geographic conditions are important determinants of participant outcomes. 

As a result, we included several factors, including the unemployment rate, average annual 

earnings, rural location, poverty rate, and population density, in all models to control for these 

factors. 

Unemployment rate data are drawn from the Local Area Unemployment Statistics (LAUS) 

generated by the Bureau of Labor Statistics, while data for the other variables are drawn from the 

2000 United States Census, Summary File 4. Census data are mapped to local areas so that data 

for average annual earnings, the percent of the population living in rural areas, and the poverty 

rate represent averages at the local area level. Unemployment rates are also mapped to their 

corresponding local area. 
II-18

Intervening Variables 

The intervening variables, included in some models, represent local program policies, practices, 

and service design. The process by which these variables were constructed and selected is 

described above. Local policies are represented by five indices that capture: 

• The extent to which WIA is used to provide core and intensive services. 

• The degree to which the performance measurement system restricts the pool of 

available providers. 

• The extent to which partners share the WIA performance measures and the use of 

contracted providers to offer services to adults and dislocated workers. 

• Performance related requirements for providers and applicants. 

• The impact of performance on providers: selection, incentives and sanctions. 

Effects of State Policies 

In this section, we look at how state performance policies affect the types of customers served by 

WIA. Historically, there has been concern that performance systems may induce states and local 

areas to focus their services on individuals who are considered more likely to be “successes.” 

Because most performance outcomes measure gross outcomes, such as employment, individuals 

most likely to be successes typically are those with stronger labor market histories—those people 

who are likely to do well even without WIA services. Thus, program operators might be tempted 

to encourage those with strong labor market histories to move on to services beyond 

informational and self-service core services and to discourage those with poor work histories 

from doing so. Because only those individuals who move beyond informational and self-service 

core services are registered in WIA and included in the performance measures system, this 

strategy would tend to result in higher performance on gross outcomes. 

In WIA’s predecessor program, JTPA, all the performance measures were gross outcomes. The 

situation is somewhat more complex in WIA. For adults, dislocated workers, and older youth, 

three of the four performance measures are gross outcomes: entered employment, retention, and 

credential attainment. However, the fourth performance measure, earnings change for adults and 

older youth and earnings replacement for dislocated workers, compares postprogram earnings to 

preprogram earnings. Thus, individuals with low preprogram earnings may do well because they 

have a lower preprogram earnings comparison. For example, if an adult with no prior earnings 

gets a minimum wage job, that adult’s six-month earnings change would be approximately 

$5,356, far above the average of about $3,249. Thus, to maximize earnings change, program 

operators might focus on individuals with low preprogram earnings. This strategy might 

increase performance on the earnings change measure, but might also reduce performance on the 
II-19

entered employment, retention, and credential measures. Thus, the incentives provided to 

program operators are mixed. Depending on whether local areas are concerned about earnings 

change or the other measures, the performance system might induce them to serve either those 

with poor work histories or those with strong work histories. 

Effects of State Policies on Customers 

We begin by looking at the overall effects of state policies on customers without controlling for 

local policies. Because state policies are likely to influence local policies, controlling for local 

policies would result in only partial estimates of the effects of state policies. 

In developing our estimates, we began by estimating models containing all the state policies that 

we coded. However, we ultimately restricted the state policies included in our models for 

several reasons. We eliminated policies that were adopted only in a few states and used 

summary measures for policies that were highly correlated. These restrictions did not affect our 

main findings. 

The results of our analysis are presented in Exhibit II-8 to Exhibit II-16. Because we are 

conducting a large number of statistical tests, we focus on findings that show a consistent pattern 

of effects and ignore occasional effects that are marginally significant. 

Adults 

We begin by focusing on the results for adults, shown in Exhibit II-8. As shown in the exhibit, 

states that give monetary incentives for performance show a strong and consistent pattern of 

serving more individuals with characteristics normally considered indicative of poorer work 

histories. Groups with greater service in states with performance incentives include those who 

are: 

• Not high school graduates. 

• Females. 

• Limited English-language proficient. 
II-20

Exhibit II-8: 

Effects of State Policies on the Characteristics of Adults 

State Provides 

Performance 

Incentives 

Single Area 

States 

Stringency of 

Incentive 

Criteria 

Less Stringent 

Sanction 

Criteria 

Less Flexible 

Credential 

Definitions 

Female 4.6** 1.2 -1.2 3.9* 3.4* 

Race and Ethnicity 

Latino 1.5 0.0 1.5** 1.7 0.2 

African American 0.7 -0.3 -0.9 2.4* 0.2 

Other Minority 1.1 -1.6 -0.4 -0.7 0.3 

Age 

Under 18 -0.1 0.2 0.0 0.1* 0.0 

18 to 19 0.7 1.9 -0.4 -0.2 -0.3 

20 to 34 0.2 -0.9 -0.9 4.9** 1.5 

35 to 54 0.5 -0.9 0.9 -4.4** -0.8 

55 and over -1.0 -0.2 0.2 -0.8 -0.6 

Education 

No High School Diploma 3.3** 7.3* 0.9 0.2 -0.5 

High School Graduate -2.5 -3.2 -0.3 1.4 3.5** 

College Graduate -1.0 -3.9 -0.6 -1.3 -2.8** 

Limited English-language 

Proficiency 1.7* 1.6 0.5 0.4 0.1 

With a Disability 1.0 4.9* -0.2 0.1 -0.8 

Low Income 26.5** 35.0** -0.5 3.9 -1.5 

Not Employed at 

Registration 3.7* 4.3 2.6** 0.1 -0.4 

Veteran -0.8 -0.3 0.8* -0.6 -0.6 

Public Assistance Recipient 3.9* 8.6 -0.5 5.1** -4.3* 

Single Parent 7.7** 9.6 3.1* 12.6** 1.7 

Earnings 2nd and 3rd 

Quarters Before Registration 

Average+ -$1,714** -$2,340** $18 -$647* -$112 

$0-$4,999 8.8** 13.0** -0.7 3.4* 0.4 

$5,000- $9,999 0.0 -1.6 0.5 0.3 -0.7 

$10,000 or more -8.7** -11.3** 0.2 -3.7** 0.3 

 ** Statistically significant at the 1% level. 

 * Statistically significant at the 5% level. 

 + Coefficient denotes the effect on the average value of this continuous variable. 
II-21

• Low income. 

• Low preprogram earnings. 

• Public assistance recipients. 

• Single parents. 

• Not employed at registration 

Single workforce area states also show greater service to groups that tend to have poorer work 

histories, although not as consistently. Groups with greater service in single area states include 

those who: 

• Are not high school graduates. 

• Have a disability. 

• Are low income. 

• Have low preprogram earnings. 

This result can be viewed as confirming the result for states with incentive awards because single 

workforce area states can receive incentives from ETA. 

The stringency of the criteria for incentives awards tends to have effects on customers that are 

similar to, but less consistent than, the effects of providing incentives. The more stringent the 

incentive criteria, the greater service is to Latinos, single parents, and those not employed at 

registration. The one exception is that the stringency of incentive criteria leads to a small 

reduction in the percentage of customers with no prior earnings (not shown in the exhibit). 

Thus, for adults we can conclude that providing performance incentives seems to increase 

service to groups that historically have had poorer work histories. Further, all the groups 

mentioned above, except for females, have lower than average outcomes on the entered 

employment and retention measures. In contrast, except for high school dropouts, they have 

higher average earnings change.21 Thus, initially, these results might seem counterintuitive in 

that they seem to suggest that providing incentives lead local areas to serve customers whose 

outcomes are likely to be lower. However, this approach makes sense when considering the poor 

prior earnings, because serving these customers substantially increases one’s likelihood of 

achieving positive and sizable earnings increases. Similarly, serving those not employed at 

registration also increases the chance that a local area will be credited with a positive outcome, 

because these individuals will count toward the entered employment rate if they do find work. 

Thus, the incentives seem to focus local areas on serving those who are likely to give them credit 
II-22

on the performance measures, even though these are the individuals who have the poorest work 

histories. In this way, then, the performance measurement system seems to motivate local areas 

to serve those who can most benefit from the public workforce system. 

The estimated effects of having less stringent criteria for determining which local areas are 

subject to sanctions are contradictory to the effects of incentives. Like having incentives and 

having more stringent incentive criteria, having less stringent sanction criteria also tends to 

increase service to some groups with historically poorer work histories. In particular, states with 

less stringent sanction criteria tend to serve more: 

• Individuals under age 35. 

• African Americans. 

• Females. 

• Individuals with low preprogram earnings. 

• Public assistance recipients. 

• Single parents. 

For most characteristics, the effects of providing incentives are larger than the effects of using 

less stringent sanction criteria. Thus, a combination of providing incentives and sanction criteria 

like the federal criteria or even less stringent would seem to have a positive effect on serving 

individuals with poorer work histories. As noted above, serving those with low prior earnings is 

entirely sensible in that these individuals can most easily observe strong positive earnings gains 

if they are able to find employment. 

The last state policy we examine is having a less flexible credential definition. States with less 

flexible credential definitions tended to serve more high school graduates and females, and fewer 

public assistance recipients. High school graduates and females may be viewed as more likely to 

complete formal training programs that lead to generally recognized credentials. 

Dislocated Workers 

Although we do not show the table, the results on the effects of state policies on customer 

characteristics for dislocated workers are remarkably similar to the results for adults. For 

example, states that provide performance incentives to local workforce areas tend to have greater 

service to other ethnicities (not Latino, black, or white), individuals with lower preprogram 

earnings, single parents, and those not employed at registration. Similarly, providing incentives 

leads to greater service to individuals with poorer work histories. Further, states with a single 

 

21 PY 2003 WIASRD Data Book. 
II-23

local workforce area tend to serve more single parents. Also, the estimated effects for having 

less stringent sanction policies generally work in the opposite direction than the effects of 

incentives. Less stringent sanction criteria lead to reduced service to other ethnicities and 

individuals with low preprogram earnings. However, in contrast to the findings for adults, less 

stringent sanction criteria do not lead to increased services to harder-to-serve dislocated workers, 

such as single parents. 

The stringency of states’ incentive criteria has somewhat mixed effects for dislocated workers. 

As for adults, more stringent incentive policies tend to lead to greater service to high school 

dropouts. They also lead to increased service to older workers (age 55 and over), who often have 

difficulty obtaining reemployment. However, in contrast to the results for adults, they also tend 

to reduce service to dislocated workers with low preprogram earnings. 

On an overall basis, however, providing incentives appears to increase service to groups with 

poorer work histories—either the effects of providing incentives and the effects of stringent 

incentive criteria are reinforcing or the net effect leads to greater service to those with poorer 

work histories. In particular, the negative effect of providing incentives on average preprogram 

earnings more than offsets the positive effect of stringent incentive criteria. 

Thus, we can conclude that, for dislocated workers, a combination of providing incentives, using 

stringent criteria to determine eligibility for incentives, and having sanction criteria as stringent 

as the federal criteria generally leads to greater service to groups with poorer work histories. 

Youth 

The results for youth are presented in Exhibit II-9. These results continue the pattern we saw for 

adults and dislocated workers. States that provide performance incentives are more likely to 

serve youth with significant employment challenges, including youth with disabilities, offenders, 

pregnant and parenting youth, and single parents. However, there is increased service to youth 

who are employed at registration. There is also increased service to younger youth. 
II-24

Exhibit II-9: 

Effects of State Policies on the Characteristics of Youth 

State Provides 

Performance 

Incentives 

Single Area 

States 

Stringency of 

Incentive Criteria 

Less Stringent 

Sanction Criteria 

Female 1.6 -3.7 -1.7* 1.2 

Race and ethnicity 

Latino 2.6 1.4 1.1 -2.4 

African American -2.7 -9.5 0.5 3.3* 

Other Minority 0.0 0.9 -0.2 -0.2 

Age 

Under 18 -6.2** 1.2 -0.5 0.1 

18 to 19 3.0 -1.1 0.0 -0.8 

20 to 21 3.2** 0.6 -0.1 0.7 

Limited English-language 

Proficiency 

0.2 1.7 -0.7 0.0 

With a Disability 5.4* 14.5* 1.0 -1.3 

Not Employed at Registration -3.6** -2.3 0.2 -0.1 

Receiving Public Assistance -1.2 -2.9 -1.6 -0.3 

Single Parent 5.7** 1.6 1.2 1.1 

Pregnant or Parenting Youth 4.1* -1.1 0.6 3.0* 

Not Attending School 2.6 2.6 -0.9 0.9 

High School Dropout -0.1 3.2 -0.4 4.5* 

High School Graduate 2.7 -1.7 -0.6 -3.6* 

Youth Offender 4.9** 9.7* 0.1 -0.2 

** Statistically significant at the 1% level. 

 * Statistically significant at the 5% level. 
II-25

States with a single local workforce area also tend to have greater service to some of these same 

groups, including youth with disabilities and offenders. 

The stringency of incentive criteria has little effect on the types of youth served. The only 

significant effect is a reduction in service to female youth. 

As for adults and dislocated workers, the effects of less stringent sanction criteria are similar to 

effect of providing incentives, in that service to less job ready individuals is increased. States 

that apply sanction criteria less stringent than the federal criteria tend to serve more high school 

dropouts and fewer graduates than other states. 

Summary of Effects on Customers 

Overall, we see a very consistent pattern in how state performance policies affect the types of 

customers served by WIA. Both providing performance incentives to local areas and the 

stringency of the incentive criteria are associated with increased service to less job-ready 

individuals and those with poorer work histories. States with single local workforce areas are 

also associated with increased service to many of these same groups. Because these states are 

subject to federal incentive criteria, this result can be viewed as confirming the results for state 

incentive policies. In contrast, sanction criteria seem to have opposite effects. Less stringent 

sanction criteria are associated with increased service to some of the same groups. Overall, 

however, the results suggest that a combination of incentive and sanction policies increases 

service to less job-ready individuals. 

These results thus provide little support for the oft-mentioned concern that a heightened focus on 

performance policies, and an emphasis on incentives and sanctions based on performance, would 

lead areas to cream the customer pool and select only those who had the highest likelihood of 

achieving successful outcomes. Indeed, to the extent that local areas are enrolling some 

customers more than others, they seem to be doing so in the opposite way, by selecting those 

who have a lower likelihood of achieving successful outcomes. 

Effects of State Policies on Services 

We now shift to examining the effects of state performance policies on the types of services 

offered. Because the performance system measures the outcomes of the people who are served 

and does not give any “credit” for the number of people served, one strategy that local areas 

might use to attain high performance is to provide more costly services, such as intensive 

services and training, to fewer people. Thus, the mix of services might be skewed more towards 

training and the cost of services might be increased. 
II-26

Adults and Dislocated Workers 

Exhibit II-10 shows estimates of the relationship between state performance policies and services 

for adults and dislocated workers. These results show that there is little effect of state policies on 

services for adults. 

Exhibit II-10: 

Effects of State Policies on the Services Received by Adults and Dislocated Workers 

State Provides 

Performance 

Incentives 

Single 

Area 

States 

Stringency of 

Incentive Criteria 

Less Stringent 

Sanction 

Criteria 

Less Flexible 

Credential 

Definitions 

Adults 

Core Services Only -2.9 -3.7 1.0 -3.4 -4.2 

Training -6.3 -3.2 -4.7* -0.7 12.7**

ITA Account -1.3 9.4 0.8 -2.4 10.6*

Adult Basic Education 0.1 3.9 -1.5 0.3 0.8 

On-the-Job Training -3.8 2.4 -1.4 -2.4 -3.7 

Occupational Training 3.2 6.3 1.8 4.0 1.7 

Average Duration of 

Training (Days)+

 15.4 -33.6 -54.0** 17.1 13.9 

Average Cost per 

Participant+ -$460 -$977 -$480 -$128 -$316 

Dislocated Workers 

Core Service Only -3.4 -3.8 1.8 0.3 -0.8 

Training -5.5 -0.1 -4.3* -7.8* 6.7*

ITA Account -7.7 4.9 0.3 -6.7 5.7 

Adult Basic Education 0.1 3.7 -0.5 -1.7 1.9 

On-the-Job Training -1.8 1.4 -0.9 -2.4 -4.1*

Occupational Training 1.5 5.6 1.1 4.2* 2.6 

Average Duration of 

Training (Days)+

 8.7 -17.1 -42.0** 10.4 41.7*

Average Cost per 

Participant+ -$758 -$99 -$1,253 -$1,356 -$1,410 

^ These results hold whether customer characteristics are included as predictors in the model or not. 

** Statistically significant at the 1% level. 

 * Statistically significant at the 5% level. 

 + Coefficient denotes the effect on the average value of this continuous variable. 
II-27

While the provision of performance incentives has no effect on the types of services received by 

WIA participants, states with more stringent criteria for awarding such incentives provide less 

training and have lower average lengths of participation. This finding contradicts our initial 

hypothesis that the performance system may lead to services that are more intensive. 

States with less flexible definitions of credential provide more training and are more likely to use 

ITAs. The greater use of ITAs may result because ITAs are more commonly used for accredited 

institutions that award recognized credentials. 

The results for dislocated workers in Exhibit II-10 are very similar to the results for adults. 

States with more stringent incentive criteria provide less training and have shorter average 

lengths of participation. 

In contrast to adults, however, less stringent sanction policies seem to have an effect for 

dislocated workers. Less stringent sanction policies seem to reduce the amount of training, but 

also increase the prevalence of occupational training other than OJT among those who receive 

training. However, the combination of these two effects leads to reduced other occupational 

training overall. 

As was the case for adults, states with less flexible credential definitions tend to provide more 

training. They also provide less OJT, perhaps because it is more difficult to meet the credential 

criteria with OJT, and have longer lengths of participation. 

Youth 

Exhibit II-11 shows the estimated effects of state performance policies on the services received 

by youth. The most consistent results seem to be that both the stringency of incentive criteria 

and the use of less stringent sanction criteria seem to increase the provision of educational 

achievement services, summer employment activities, and follow-up services. For most states, 

these effects offset so that there is little effect on youth services. However, for the common 

combination of less stringent incentive criteria and federal sanction criteria, the net effect is a 

reduction in these services. The use of less stringent sanction criteria also appears to increase the 

length of participation and the cost of services. 

It should be noted that the reporting of youth services changed with the advent of WIA and 

reporting in some states and local areas appears to be inconsistent. For example, many youth are 

reported as receiving no services at all (8.6% of exiters from April 2003 through July 2004). 

Thus, it seems best not to draw strong conclusions from these results. 
II-28

Exhibit II-11: 

Effects of State Policies on the Services Received by Youth 

State Provides 

Performance 

Incentives 

Single 

Area 

States 

Stringency of 

Incentive 

Criteria 

Less Stringent 

Sanction 

Criteria 

Educational Achievement 8.5 10.0 12.1** 12.4**

Employment 1.7 12.5 -1.2 -7.4 

Summer Employment -4.6 12.0 9.8** 11.9**

Additional Support 11.8* -3.7 2.3 7.3 

Leadership Development -1.5 22.4* -2.1 7.5 

Follow-Up -9.7 13.5 23.1** 13.4*

Average Length of Participation (Days)+ -47.3 -60.2 -16.0 77.2**

Average Cost Per Participant+ -$1,077 -$2,181 $177 $2,144*

** Statistically significant at the 1% level. 

 * Statistically significant at the 5% level. 

 + Coefficient denotes the effect on the average value of this continuous variable. 

Effects of State Policies on Outcomes 

One would expect that performance incentives and sanctions would work to increase 

performance outcomes. As shown in Exhibit II-12, however, our analysis shows no consistent 

effects of state performance policies on outcomes. These results are for measured outcomes. 

However, we obtained very similar results when we looked at the difference between the actual 

outcome and both local and state performance goals. 
II-29

Exhibit II-12: 

Effects of State Policies on Outcomes 

State Provides 

Performance 

Incentives 

Single 

Area 

States 

Stringency of 

Incentive 

Criteria 

Less Stringent 

Sanction 

Criteria 

Less flexible 

Credential 

Definitions 

Adults 

Entered Employment -1.6 -8.6* -0.7 0.3 2.3 

Retention -2.6** -6.8** -0.8 0.3 3.6** 

Credential -1.8 -9.0 -1.5 0.5 -0.2 

Earnings Change $44 -$367 -$338* $664** $278 

Dislocated Workers 

Entered Employment -0.7 -4.9 0.4 2.1* 3.4** 

Retention -1.0 -4.1** 0.4 0.6 1.8** 

Credential -2.4 -2.8 -0.2 2.3 0.5 

Earnings Change $139 -$614 -$303 -$313 $563 

Wage Replacement Rate -0.7 -6.6 -2.8 -1.4 3.2 

Older Youth 

Entered Employment -3.0 -4.9 -3.9** -0.9 2.6 

Retention 4.8* 1.4 -1.0 -3.1 2.8 

Credential -2.0 -2.6 -6.2** -4.1 0.7 

Earnings Change -$81 -$375 -$273 $81 -$111 

Younger Youth 

Retention 0.7 2.2 -0.7 -0.9 -- 

Diploma Attainment 2.6 0.9 0.1 4.5 -- 

Skill Attainment 4.4* 0.6 -2.3* -3.4* -- 

1 Coefficients for ‘More Stringent Definition of Credentials’ policy not estimated for younger youth exiters 

because the credential definition does not apply to younger youth. 

** Statistically significant at the 1% level. 

 * Statistically significant at the 5% level. 

The provision of performance incentives seems to reduce adult retention, increase older youth 

retention, and increase younger youth skill attainment. More stringent incentive criteria seem to 

reduce adult earnings change, older youth credential attainment, and older youth entered 

employment. Less stringent sanction policies seem to increase adult earnings change and 

dislocated worker entered employment, but to reduce younger youth skill attainment. 

Having a less flexible credential definition has no effect on attainment of credentials, but seems 

to increase adult and dislocated worker retention and increase dislocated worker entered 
II-30

employment. Perhaps focusing on recognized credentials places credential recipients in a better 

position to find and keep jobs. 

Overall, these scattered results regarding the effects of state performance policies on outcomes 

are not sufficient to lead to a strong conclusion. 

Effects of Local Policies 

Effects of Local Policies on Customers 

Exhibit II-13 shows the estimated effects of local policies on customer characteristics for 

adults.22 Local areas that give performance incentives or sanctions to providers tend to serve 

more younger individuals, fewer college graduates, and more individuals with lower preprogram 

wages for both adults and dislocated workers. They also serve more adult public assistance 

recipients and more Latino dislocated workers.23 

These results are similar to the effects of state policies that provide performance incentives to 

local areas. Both providing incentives to local workforce areas and offering incentives to 

providers result in increased service to groups with poorer work histories. 

Local areas in which the performance system is seen as limiting the availability of providers tend 

to serve more job-ready individuals, especially for adults. Such local areas tend to serve fewer 

younger adults, more better-educated adults, and more adults with higher preprogram wages. 

They also tend to serve more middle-aged dislocated workers. There is little effect for youth. 

While there is no clear explanation for this effect, it is possible that training is effectively limited 

to individuals with prior skills because providers are able to establish prerequisites for training. 

Effects of Local Policies on Services 

Exhibit II-14 shows the estimated effects for local policies on services. Here we see two results 

that are fairly consistent. 

 

22 We estimated a similar model for dislocated workers. Although the results we describe in the text are stronger 

for adults, in general results for both adults and dislocated workers are highly similar. 

23 Although it is not shown, an additional model examining the effects of local policies on the characteristics of 

youth served revealed only a few, relatively scattered findings. For example, in local areas that emphasize 

performance requirements for providers and applicants , there seems to be less service to a number of somewhat 

disparate groups among youth, including youth aged 18 to 20, employed youth, single parents, pregnant and 

parenting youth, high school graduates, and offenders. The one commonality among these groups would seem to 

be that they are likely to be concentrated among older youth. 
II-31

Exhibit II-13: 

Effects of Local Policies on the Characteristics of Adults 

Performance 

Requirements 

for Providers 

Performance Emphasis 

on Providers' 

Selection, Incentives & 

Sanctions 

Use of WIA 

for Core & 

Intensive 

Services 

Performance 

Measurement 

System Restricts 

Providers 

Emphasis 

on 

Partnerships 

Female -1.1 0.0 -0.9 -0.9 -0.1 

Race and Ethnicity 

Latino 0.7* 0.0 -0.6 -0.6* 0.3 

African American 0.6 0.8 0.6 -0.4 0.2 

Other Minority 0.0 -0.4* -0.5* 0.5* 0.2 

Age 

Under 18 0.0 0.0 0.0 0.0 0.0 

18 to 19 0.3 0.5** 0.0 -0.4* 0.1 

20 to 34 0.0 1.3* -0.3 -1.7** -0.7 

35 to 54 -0.1 -1.3** 0.3 1.7** 0.4 

55 and Over -0.3 -0.5* 0.1 0.3 0.4 

Education 

No High School Diploma -0.1 0.2 -0.3 0.1 -0.1 

High School Graduate 0.7 0.6 0.3 -1.2** -0.3 

College Graduate -0.6* -0.9** 0.1 1.1** 0.4 

Limited English-language 

Proficiency 

0.0 -0.2 0.0 0.4 0.1 

With a Disability 0.0 -0.1 0.0 -0.1 -0.2 

Low Income 0.8 1.6 0.4 -2.1 -0.3 

Not Employed at 

Registration 

0.8 0.1 -0.5 0.9 0.5 

Veteran 0.3 -0.3 0.3 0.3 0.2 

Public Assistance Recipient 0.2 1.2* 0.2 -0.9 1.2* 

Single Parent 0.1 0.5 -1.0 -1.1 1.6* 

Earnings 2nd and 3rd 

Quarters Before 

Registration 

 

Average+ -$69 -$225* -$109 $296** -$61 

$0 to $4,999 0.2 1.2* 0.8 -1.6** 0.5 

$5,000 to $9,999 0.0 -0.2 -0.2 0.1 -0.3 

$10,000 or more -0.3 -1.1* -0.6 1.5** -0.2 

** Statistically significant at the 1% level. 

 * Statistically significant at the 5% level. 

 + Coefficient denotes the effect on the average value of this continuous variable. 
II-32

Exhibit II-14: 

Effects of Local Policies on Services 

Performance 

Requirements 

for Providers 

Performance 

Emphasis on 

Providers' Selection, 

Incentives & Sanctions 

Use of WIA 

for Core & 

Intensive 

Services 

Performance 

Measurement 

System Restricts 

Providers 

Emphasis 

on 

Partnerships 

Adults 

Core Service Only -0.7 0.1 2.6** 0.3 3.2** 

Training -1.0 1.4 -3.7** -1.7 -6.2** 

ITA Account 1.4 2.2 -2.7* -2.7* -5.8** 

On-the-Job Training 0.0 0.9 0.5 -0.4 1.1 

Adult Basic Education -0.7 -1.0 -0.7 0.6 0.8 

Occupational Training -0.7 0.1 -0.2 0.0 -1.9* 

Average Duration of 

Training (Days)+

-7.7 3.6 -1.7 -8.8 -18.9** 

Average Cost Per 

Participant+

-$369 -$320 $122 -$162 -$535* 

Dislocated Workers 

Core only -0.1 -0.3 1.4* 0.9 1.7* 

Received Training -1.2 0.4 -2.9** -2.0 -4.2** 

ITA 0.8 1.2 -1.6 -3.4* -3.6** 

OJT -0.1 0.4 -0.5 0.7 0.1 

Adult Basic Skills 0.4 -0.2 -0.5 0.2 0.7 

Occupational Training -0.8 -0.1 1.4* -0.6 -0.8 

Average Duration of 

Training (Days)+

-12.5 3.1 4.4 -5.0 -7.7 

Average Cost Per 

Participant+

$138 -$882 $844 $85 -$623 

Youth 

Educational 

Achievement 

0.4 0.3 1.6 0.4 0.6 

Employment -1.0 2.6 2.7 -3.8** 2.6 

Summer Employment 1.3 -2.7* 0.5 0.5 -3.1* 

Additional Support -0.2 -1.9 3.5 1.5 0.7 

Leadership Development 2.1 0.5 -1.5 4.1** 0.8 

Follow-Up 1.5 -1.9 -3.3 -0.3 -2.6 

Average Duration of 

Training (Days)+

9.6 7.5 7.2 -6.7 3.3 

Average Cost per 

participant+

$5 -$549 $314 -$283 -$287 

** Statistically significant at the 1% level. 

 * Statistically significant at the 5% level. 
II-33

Not surprisingly, in local areas where WIA is heavily involved with providing core and intensive 

services, both adults and dislocated workers are more likely to receive only core services and less 

likely to receive training. This result is almost tautological. 

Second, in local areas where there is a greater emphasis on partnerships, adults and dislocated 

workers are also more likely to receive core services and less likely to receive training. They are 

also less likely to receive ITAs. Adults in these areas also seem to receive less other 

occupational training and have shorter lengths of WIA participation. Further, the emphasis on 

partnerships seems to reduce the average cost per adult exiter perhaps because these are sharing 

cost with WIA. We do not see similar effects for youth. 

Effects of Local Policies on Outcomes 

Exhibit II-15 shows the estimated effects of local policies on outcomes. What is most striking 

about this exhibit is the lack of any impacts of local policies. Although there are some scattered 

statistically significant results, there are about the same number that would be expected by 

chance even if there were no effects—just 5 significant out of 85 estimates. Thus, while it may 

be that the use of performance requirements for providers does lead to lower earnings change 

among older youth or that the use of WIA as the primary provider for core and intensive services 

leads to slightly lower retention rates among dislocated workers, there is no clear explanation for 

these effects, and they are so sporadic as to possibly be the result of random chance alone. The 

lack of association is important in itself, however, as it suggests little relationship between local 

area decisions and their customers’ outcomes. Obviously, local areas vary widely in the 

outcomes their customers obtain, but these variations do not seem to be associated with who 

provides core and intensive services, whether local areas emphasize performance to and build in 

specific requirements for their providers, or the emphasis they place on partnerships. Of course, 

it is entirely possible that other local policies and decisions have an impact on customers’ 

outcomes. Indeed, the impacts noted above on customer characteristics and services received 

carry through and have a resultant effect on outcomes. But the policies have little discernible 

impact on customer outcomes themselves. 
II-34

Exhibit II-15: 

Effects of Local Policies on Outcomes 

Performance 

Requirements 

for Providers 

Performance 

Emphasis on 

Providers' Selection, 

Incentives& Sanctions 

Use of WIA 

for Core & 

Intensive 

Services 

Performance 

Measurement 

System Restricts 

Providers 

Emphasis 

on 

Partnerships 

Adults 

Entered 

Employment -0.4 0.4 -0.8 0.1 0.4 

Retention 0.0 -0.5* 0.1 -0.1 -0.1 

Credential -0.8 0.2 -1.4 1.3 1.1 

Earnings Change -$19 $94 -$29 -$48 -$153 

Dislocated Workers 

Entered 

Employment -0.2 -0.1 -0.1 0.2 0.0 

Retention -0.2 -0.2 -0.5* 0.0 -0.2 

Credential -0.1 -0.2 -0.9 1.4* 0.2 

Earnings Change -$39 $89 $37 $91 $108 

Earnings 

Replacement -0.3 0.8 0.6 0.3 0.9 

Older Youth 

Entered 

Employment 0.0 -1.0 0.4 0.6 1.3 

Retention -0.4 -0.7 -0.2 -0.4 0.0 

Credential -0.4 -0.5 -0.9 -0.9 1.5 

Earnings Change -$240* $26 $18 $2 $11 

Younger Youth 

Retention -0.1 -0.8 -0.4 -0.1 0.8 

Diploma 

Attainment 0.1 -0.8 -1.0 -2.2* -0.1 

Skill Attainment 0.9 0.1 -1.0 -0.2 0.3 

** Statistically significant at the 1% level. 

 * Statistically significant at the 5% level. 

Effects of State Policies on Local Policies 

Local performance policies are not developed in a vacuum; instead, they are developed in the 

context of federal and state performance policies. Thus, it is interesting to examine how state 

policies influence local policies. 
II-35

Exhibit II-16 shows our estimates of how state policies influence the local policies in our model. 

Of the possible associations we examine between state and local policies, there are a few 

significant results. 

Exhibit II-16: 

Effects of State Polices on Local Policies 

State Provides 

Performance 

Incentives 

Single 

Area 

States 

Stringency of 

Incentive 

Criteria 

Less Stringent 

Sanction 

Criteria 

Less Flexible 

Credential 

Definitions 

Performance Related 

Requirements for Providers 

and Applicants 

1.8 7.3 12.9 -4.0 -18.8 

Performance Emphasis on 

Providers' Selection, 

Incentives and Sanctions 

-10.6 -6.4 -4.4 -9.1 -15.3 

Use of WIA to provide Core 

and Intensive Services 11.7 -89.1* -28.3** -6.2 -16.0 

Performance Measurement 

System Restricts the Pool of 

Available Providers 

-4.3 11.6 18.8* 5.2 -28.7 

Emphasis on Partnerships 2.8 -39.4 10.8 57.0** -5.1 

** Statistically significant at the 1% level. 

 * Statistically significant at the 5% level. 

In states that use stringent criteria to award incentives, WIA is less likely to be the primary 

provider of core and intensive services and local areas are more likely to see the availability of 

providers as limited by performance policies. 

In states with a single local workforce area, WIA tends to have a smaller role in providing core 

and intensive services. This result possibly occurs because it is easier to integrate the stateoperated ES and locally –operated WIA in states where the local area is the entire state. 

In states that apply sanction criteria that are less stringent than the federal criteria, local areas are 

more likely to emphasize partnerships. 

Despite these few associations, however, overall, these results do not show a compelling 

relationship between the state and local performance policies that we measured. As a result, 

when we estimate models that include both state and local policies as independent variables (not 

shown), the effects of state policies are very similar to the estimates in models that do not include 

local policies. Thus, the effects of state policies we discussed above do not work through the 

local policies we measured to affect customer characteristics and services. 
II-36

Summary and Implications for the National Performance 

Measurement System 

Overall, many of the results described in this chapter are inconclusive and do not suggest that 

variations in state policies have substantial impact on the customers who are served under WIA, 

the services they receive, or the outcomes they obtain. However, the analysis does lead to one 

significant and important conclusion. The provision by states to local areas of incentive awards 

based on performance seems to increase service to a variety of customer groups that tend to be 

less job-ready and have poorer work histories. This result is confirmed by similar results for 

states with a single local workforce area, which are subject to federal incentive policies. It is 

also confirmed by similar results for the provision of incentives to providers by local areas. 

This result suggests that fears that the WIA performance system might induce local areas to 

focus services on better-prepared individuals are unfounded. One might have expected that the 

desire to do well on the performance measures would lead local areas to focus services on those 

who are most likely to achieve high outcomes because the WIA performance system is primarily 

based on total outcomes, such as entered employment, retention, and credential attainment. The 

individuals most likely to achieve high results on these outcomes are those who have strong 

work histories—just those who are least in need of WIA services. 

However, we do not see any evidence that state policies that emphasize performance by 

providing incentives have this effect. Indeed, just the opposite occurs. Local areas in states that 

provide incentives are more likely to serve less job-ready individuals with poor work histories. 

One potential explanation of this result is the use of a few performance measures that measure 

change rather than the total outcome: adult and older youth earnings change and dislocated 

worker earnings replacement. Because these measures compare postprogram earnings to 

preprogram earnings, one way to succeed on these measures might be to focus services on 

individuals with low preprogram earnings, which is one of the results we found. 

Although we suspect that the presence of the earnings change and replacement measures more 

than offsets the influence of the other measures, we cannot say conclusively that this is true. 

Indeed, there is some contradictory evidence. If local areas in states with performance incentives 

were following a strategy to increase earnings change and replacement, one would expect to see 

an increase in the earnings change outcomes and, perhaps, a decrease in the other outcomes in 

those states. While we found some significant relationships between state policies and 

outcomes, we did not find that provision of incentives increased the earnings change measures. 

To the contrary, we found that more stringent incentive criteria seemed to decrease adult 

earnings change. 
II-37

Ultimately, then, much of the analysis in this chapter is inconclusive, in part because states and 

local areas have stuck fairly closely with the system as promulgated at the federal level. 

Importantly, however, we found little to suggest that those most in need of WIA services have a 

difficult time receiving those services because of any state or local concern for achieving high 

performance. Rather, states and locals, while focused on achieving strong performance, also 

seem focused on providing needed workforce services to those who can most benefit from them.