1330 Broadway, Suite 1426 

Oakland, CA 94612 

Tel: (510) 763-1499 

Fax: (510) 763-1599 

www.spra.com 

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •

S O CIA L P O L ICY R ESEA R C H

A S S O CIATE S

Prepared for: 

U.S. Department of Labor 

Employment and Training 

Administration 

Office of Policy Development and 

Research 

200 Constitution Ave., N.W. 

Washington, DC 20210

Evaluation of the WIA 

Performance Measurement 

System 

Final Report 

May 2006 

Prepared by: 

Kate Dunham 

Melissa Mack 

Jeff Salzman 

Andrew Wiegand 
This page intentionally left blank.

Insert blank page here when making double-sided copies
1

DISCLAIMER 

This report was prepared for the U.S. Department of Labor, Employment and Training 

Administration, Office of Policy Development and Research, by Social Policy Research 

Associates. Since contractors conducting research and evaluation projects under government 

sponsorship are encouraged to express their own judgment freely, this report does not necessarily 

represent the official opinion or policy of the U.S. Department of Labor. 
This page intentionally left blank.

Insert blank page here when making double-sided copies
Acknowledgements and Attributions 

This report was conducted by Social Policy Research Associates. The project was directed by 

Andrew Wiegand of SPR, but depended for its success on the strong contributions of the 

numerous field staff and analysts who worked on this project during its five years, including the 

primary authors of this final report: Kate Dunham, Melissa Mack, Jeff Salzman, and Dr. 

Wiegand. 

We wish to acknowledge as well the contributions and support of DOL staff, especially our 

project officers, Alberta Baker, Lekesha Campbell, and Dan Ryan. Their advice and steadfast 

support are much appreciated. 

Finally, we are enormously grateful to the administrators and staff from the 15 states and 25 

local areas who were very gracious in accommodating our site visits and very frankly described 

to us their performance measurement practices and challenges. These states and local areas are: 

State LWIA(s) 

Alabama Mobile 

Alabama Workforce Investment Area 

California City of Los Angeles 

Tulare County 

Colorado Weld County 

Idaho Idaho Works 

Maine Coastal Counties Workforce Board 

Michigan City of Detroit 

Gennessee-Shiawassee Counties 

New Jersey Balance of Hudson Co. Less Jersey City 

Cumberland County 

New York Monroe County 

Suffolk County 

North Carolina Region L Job Training Consortium 

Cape Fear SDA Job Training Consortium 

North Dakota North Dakota Consortium (Single WIA State) 

Ohio Ohio Option Area Southwest Ohio (Cincinnati/Hamilton County) 

City of Cleveland 

Oregon The Oregon Consortium and Oregon Workforce Alliance 

Mid-Willamette Workforce Network 

Washington Seattle/King County 

North Central Washington/Columbia Basin 

Wisconsin Milwaukee County 

Bay Area (Green Bay) 

West Virginia Kanawha County 
This page intentionally left blank.

Insert blank page here when making double-sided copies
i

TABLE OF CONTENTS 

I. INTRODUCTION................................................................................................... I-1

Overview of WIA and Key Themes........................................................................I-2

Key Principles of WIA........................................................................................I-4

The Original WIA Performance Measurement System..........................................I-5

Original Performance Measures........................................................................I-6

Negotiations ......................................................................................................I-9

Incentives and Sanctions ..................................................................................I-9

Continuous Improvement ................................................................................I-10

Implementation of the Common Measures..........................................................I-12

Methodology and Key Issues...............................................................................I-15

Data Used in the Final Report.........................................................................I-15

Additional Study Reports.................................................................................I-17

Road Map for the Remainder of the Report.........................................................I-19 

II. ELEMENTS OF THE PERFORMANCE ACCOUNTABILITY 

SYSTEM............................................................................................................... II-1

Use of Performance Measures .............................................................................II-1

Use of Additional Measures .............................................................................II-2

Effects of Common Measures on Focus on Performance................................II-4

Negotiating Standards ..........................................................................................II-5

When Sites Exceed Negotiated Goals .............................................................II-7

When Sites Fail to Meet Negotiated Goals ....................................................II-10

Changes to Negotiations Resulting from the Common 

Measures........................................................................................................II-11

Documenting Performance .................................................................................II-14
ii

MIS Architecture.............................................................................................II-14

Changes to MIS/Reporting Due to the Common Measures...........................II-16

Using Data to Monitor Performance...............................................................II-17

Mechanisms to Encourage Strong Performance ................................................II-19

Incentives .......................................................................................................II-19

Sanctions........................................................................................................II-22

Continuous Improvement ...............................................................................II-24

Summary ............................................................................................................II-26 

III. COMMUNICATING SYSTEM ELEMENTS TO THOSE WHO 

OPERATE THE SYSTEM ................................................................................... III-1

WIA Accountability and One-Stop Operations.....................................................III-1

Effects of WIA Accountability on One-Stop Design.........................................III-1

Effects of Common Measures on One-Stop Operations.................................III-2

One-Stop Performance Measures...................................................................III-5

Conveying Accountability in the WIA Adult and Dislocated 

Worker Program ..................................................................................................III-5

General Understanding of WIA Performance..................................................III-5

Understanding of Common Measures for Adults and Dislocated 

Workers ...............................................................................................................III-6

Methods of Conveying Accountability: Training Adult/Dislocated 

Worker Staff on Performance..............................................................................III-7

Conveying Accountability in the WIA Youth Program..........................................III-7

General Understanding of WIA Youth Performance .......................................III-8

Understanding and Implications of Common Measures for Youth ......................III-9

Methods of Conveying Accountability in the Youth Program ........................III-10

Comparison Between Interim and Final Samples in Types of 

Measures Used .................................................................................................III-13

Conclusion.........................................................................................................III-15

IV. SERVICE DESIGN AND DELIVERY ..................................................................IV-1

Managing Participation: Controlling Program Entry............................................ IV-2

Managing When Customers Become Participants......................................... IV-2
iii

Managing Who Becomes a Participant .......................................................... IV-6

Service Design and Delivery............................................................................. IV-14

Effect of Original WIA Measures on Service Design and 

Delivery......................................................................................................... IV-15

Effect of Common Measures on Service Design and Delivery..................... IV-23

Managing Exit ................................................................................................... IV-28

Adult and Dislocated Worker Exit Processes............................................... IV-28

Youth Exit Processes ................................................................................... IV-30

When Participants are Typically Exited........................................................ IV-31

Conclusion........................................................................................................ IV-33

V. SUMMARY...........................................................................................................V-1
This page intentionally left blank.

Insert blank page here when making double-sided copies
I-1

I. INTRODUCTION 

Social Policy Research Associates (SPR) was awarded a contract in June 2002 by the U.S. 

Department of Labor (DOL) to conduct an Evaluation of the WIA Performance Measurement 

system. As part of this evaluation, we have already submitted an Interim Report that 

summarized our findings across an initial round of site visits conducted in the spring of 2003 to 

eight states and fourteen local areas.1

 In addition, we submitted a Survey Report that 

summarized findings from a survey of all local areas operating under WIA and also included 

quantitative findings linking the state and local performance policies to the characteristics of 

customers served, services received and outcomes obtained by these customers under WIA.2 

During the course of the evaluation, new measures were proposed by the Office of 

Management and Budget (OMB) to assess performance under WIA, as well as a number of other 

workforce development programs. These were introduced as part of an effort to make 

comparable the measures across varying programs with relatively similar employment and 

earnings aims. Known as the common measures, these indicators of performance were in 

discussion by 2003, and it became clear that they could have a significant impact on the 

performance measurement system under WIA. As a result, although initially we had planned to 

complete this evaluation by the end of 2004, in consultation with DOL we agreed to substantially 

delay our final round of site visits until the common measures were formally implemented so 

that we could assess the extent to which their implementation affected the performance 

measurement system. This report, the last to be completed for this evaluation, summarizes our 

findings from this final round of site visits to seven states and eleven local areas and identifies 

how these findings are similar to or different from our earlier results. Before detailing these 

 

1

 “Evaluation of the WIA Performance Measurement System: Interim Report,” Social Policy Research 

Associates, 2004. 

2

 “Evaluation of the WIA Performance Measurement System: Survey Report,” Social Policy Research Associates, 

2005. 
I-2

findings, however, in this chapter, we provide an overview of the performance measurement 

system, including how the common measures change this system, and of our evaluation of it. 

The Workforce Investment Act (WIA) established a performance accountability system 

whose explicit purpose is "to assess the effectiveness of States and local areas in achieving 

continuous improvement of workforce investment activities."3

 In addition, WIA identifies other 

broader goals for the accountability system, including: “to increase the employment, retention, 

and earnings of participants, and increase occupational skill attainment by participants, and, as a 

result, improve the quality of the workforce, reduce welfare dependency, and enhance the 

productivity and competitiveness of the Nation."4

 

Although this evaluation has many specific research questions that are of interest, we have 

been guided in this final round of data collection by two more general goals. First, we intended 

to assess the effectiveness of the current system, including the common measures as they were 

implemented across the states and local areas in our sample. Broadly interpreted, this requires 

assessing how the accountability system is meeting all the explicit and implicit goals described 

above. Our second general goal is to identify any ways in which the system has evolved over 

time, including by incorporating the common measures. 

In this chapter, we first present the context for the evaluation by providing a brief overview 

of WIA and identifying its key themes. Following this, we describe the WIA accountability 

system by presenting the various original performance measures adopted as part of this system 

and discussing how performance goals are established for each state for each of these measures. 

Finally, we detail how the common measures alter this system and introduce new ways of 

assessing performance under WIA. The chapter concludes with a roadmap for the rest of this 

Final Report. 

Overview of WIA and Key Themes 

When enacted in 1998, WIA represented an important overhaul of the nation’s structure for 

delivering employment and training services that in many ways represented a fundamental 

departure from past practices. Among its key principles, WIA allows for substantial state and 

local flexibility, so that these workforce systems can best meet state and local needs. Within this 

framework, the legislation calls on states and local areas to establish streamlined One-Stop 

 

3

 Workforce Investment Act, Sec. 136(a). 

4

 Ibid, Sec. 106. 
I-3

service delivery systems that integrate resources available from a multitude of separately funded 

programs. In this way, customers can benefit from a service delivery structure that is less 

bureaucratic and more oriented to their needs. WIA also emphasizes universal access to 

employment and job-seeking services among adults, so that anyone who wants assistance 

relating to job search or career planning should be able to receive it. This approach stands in 

marked contrast to the Job Training Partnership Act (JTPA), the program that WIA replaced, 

which generally required both youth and adults to be certified as income eligible before receiving 

services. 

Another key tenet that underlies the WIA approach to services is empowering individuals 

to take greater control over their own career planning. Thus, customers are given more 

responsibility for establishing their own career goals and have new options with respect to the 

services they access, the training they want to undertake, and the vendors available to provide 

training to them. Staff members, conversely, are placed in the support role of providing 

guidance and information that will enable customers to make wise choices, and ensuring that 

appropriate resources are made available to them so that their goals can be realized. 

To promote these objectives, DOL formulated new regulations, developed in consultation 

with states and local areas, that were designed to be much more flexible and empowering than 

the regulations they replace and, in keeping with WIA, called for the establishment of new 

administrative and service delivery structures. For example, although the Federal government 

and states share the role of providing guidance to the local areas, local areas for their part retain 

substantial discretion to shape service designs and practices to meet local community needs. 

WIA also mandated that the service delivery structure be built on One-Stop Career Centers, so 

that customers can access a broad range of services to meet diverse needs at a single location. 

In addition, WIA established a tiered approach to adult services, whereby services begin 

with help in finding employment and increase in scope and intensity as the need for additional 

services is documented. This tiered, hierarchical system includes core, intensive, and training 

services, each of which generally must be accessed sequentially. Core services include many 

self-directed services, including conducting job searches or obtaining labor market information, 

and informational or minimal staff-assisted services. Intensive services typically require greater 

staff involvement, and include activities such as specialized assessments of skill levels, 

counseling, or case management. Training services are the highest tier, and include activities 

such as occupational skills training or on-the-job training (OJT). Generally speaking, one must 

receive at least one core service before receiving an intensive service, which they must access 
I-4

before receiving any training services, and services become more intensive as they progress 

through the tiers. 

Currently, individuals who receive only self-directed core services (or, as noted above, 

informational or minimal staff-assisted services) are not officially registered under WIA, while 

those who receive core services requiring significant staff assistance, as well as those receiving 

intensive and training services, are registered. Upon registration into WIA, individuals are 

included in the performance-measurement system, as described below. 

Key Principles of WIA 

WIA was designed to address the issues identified above, among others. In doing so, DOL 

identified seven key principles that underlie the legislation. 

• Streamlining services through integration. WIA built upon the implementation 

of a One-Stop delivery system so that multiple employment and training 

programs show a common face to the customer. Towards this end, the legislation 

identified a wide range of required partners who must make services available 

through One-Stop Career Centers, including programs authorized under Title I of 

WIA, Employment Service programs authorized under the Wagner-Peyser Act, 

adult education and literacy programs authorized under Title II of WIA, 

vocational rehabilitation programs authorized under Title I of the Rehabilitation 

Act, and others. 

• Empowering individuals. WIA allows customers meaningful choices about when 

and where they receive services and how much staff guidance they need or want. 

It seeks to achieve this goal by providing for self-access services and by 

transforming the delivery of training services to adults and dislocated workers 

through the use of Individual Training Accounts, or ITAs. 

• Universal access. Through the One-Stop system, every adult has access to 

employment-related services, including Wagner-Peyser services and core WIA 

services. These core services are supplemented with intensive and training 

services, which can be provided to adults and dislocated workers who cannot 

obtain employment or achieve self-sufficiency through services on lower tiers. 

• Increased accountability. WIA introduced an enhanced focus on accountability, 

customer satisfaction, and continuous improvement, each of which is intended to 

hold states accountable for the performance of their workforce investment 

systems. 

• Strengthened role for Local Boards. Local Workforce Investment Boards are the 

primary governing body of the local workforce investment system. 

Responsibilities of the Local Board include developing the local plan in 

cooperation with the chief elected official, selecting One-Stop operators and 

youth providers, and identifying eligible providers of training services and of 

intensive services. 
I-5

• State and local flexibility. WIA substantially enhances state and local flexibility 

in designing and operating workforce development programs. Several provisions 

of WIA promote this, such as by providing greater flexibility in how funds are 

used and expanding the availability of waivers. 

• Improved youth programs. To improve youth services, WIA established Youth 

Councils as an adjunct to the Local Boards, which are responsible for developing 

plans for and coordinating youth activities. WIA also called on states and local 

areas to enhance the effectiveness of youth services by requiring strong linkages 

between academics and occupational learning, adult mentoring, and follow-up 

services. 

Of these principles, the focus on increased accountability and the establishment of the WIA 

performance accountability system is the most critical for this evaluation. In the next section, we 

describe this accountability system. 

The Original WIA Performance Measurement 

System 

Performance and accountability have long been established as essential in assessing 

programs, particularly those in workforce development. Under the predecessor to WIA, the Job 

Training Partnership Act (JTPA), DOL had established a system of performance measures that 

applied to the various Service Delivery Areas (SDAs) across the nation. The accountability 

system under JTPA was somewhat narrower in scope than that established by WIA. Although 

several performance measures were in place under JTPA, including many very similar to those 

adopted under WIA, as described below the authorization of WIA added several new measures, 

in large part to broaden the scope of outcomes measured. In addition, the performance 

measurement system under WIA was also intended to achieve several other objectives. For 

example, WIA stipulates that states should use UI wage records to document customer outcomes, 

rather than conduct a follow-up directly with customers to assess whether they obtained a job, 

retained employment over time, or had gains in their earnings. These UI records are a more 

objective source of outcomes than are individual customer reports of their work experiences, and 

can be captured more consistently across states or local areas. Additionally, WIA sought to 

refine the system in place under JTPA by promoting continuous improvement, enabling DOL to 

provide incentives or impose sanctions on states not meeting their performance targets (which 

substantially increases states’ responsibility for performance), and introducing negotiations into 

procedures to set performance levels, rather than relying heavily upon adjustment models to set 

these targets. Each of these changes is discussed in greater detail below. 

While the JTPA performance-measurement system was in operation, the implementation of 

the Government Performance and Results Act (GPRA) in 1993 focused increased attention on the 
I-6

critical role of accountability in providing federal services to customers. Specifically, this Act 

was intended to hold federal agencies accountable for achieving program results “by promoting a 

new focus on results, service quality, and customer satisfaction,” and to help improve service 

delivery by requiring that programs plan for meeting their objectives, and by providing them with 

information about program results and service quality. Further, it was intended to improve 

legislative decisions by providing objective information on the relative effectiveness and 

efficiency of federal programs and spending. This Act required each federal agency to establish 

performance goals and to define the level of performance to be achieved by the program. It also 

required that these goals be objective, quantifiable and measurable. Further, each agency must 

establish the specific performance indicators they will use to measure or assess the services or 

outcomes of each program activity, and provide a basis of comparing actual program results with 

established performance measures. GPRA, then, built upon an already existing framework of 

performance measurement in the workforce field and provided yet more incentive to develop 

measures by which one can assess program performance 

Original Performance Measures 

Into this context, WIA was passed in 1998. Following the requirements in Section 136 of 

the Act, DOL originally established 17 performance outcomes covering Title I-B programs. 

Separate measures we established for 4 groups of participants: adults, dislocated workers, 

younger youth (14 to 18), and older youth (19 to 21). Individuals are classified into these groups 

based on the funding source from which they are served. Thus, individuals served with adult 

funds are counted in the adult performance outcomes, individuals served with dislocated worker 

funds are counted in the dislocated worker performance outcomes, and individuals served with 

youth funds are counted in either the younger or older youth outcomes, depending on their age 

when they register for WIA. In addition, DOL established two customer satisfaction outcomes. 

The original seventeen performance measures are: 

• Adults 

− Entered Employment Rate: the percentage of adults who are employed 

in the quarter after exit from the program, excluding adults who were 

employed at entry. 

− Employment Retention Rate: among adults employed in the quarter 

after exit from the program, the percentage who are employed in the 

3rd quarter after exit. 
I-7

− Earnings Change: among adults employed in the quarter after exit, the 

change in earnings from the 2nd and 3rd quarters before entry into the 

program to the 2nd and 3rd quarters after exit from the program. 

− Employment and Credential Rate: Among adults who received 

training services, the percentage who are employed in the quarter after 

exit and received a credential. 

• Dislocated Workers 

− Entered Employment Rate: the percentage of dislocated workers who 

are employed in the quarter after exit from the program. 

− Employment Retention Rate: among dislocated workers employed in 

the quarter after exit from the program, the percentage who are 

employed in the 3rd quarter after exit. 

− Earnings Replacement Rate: among dislocated workers employed in 

the quarter after exit, earnings in the 2nd and 3rd quarters after exit 

from the program as a percentage of earnings in the 2nd and 3rd

quarters before entry into the program. 

− Employment and Credential Rate: Among dislocated workers who 

received training services, the percentage who are employed in the 

quarter after exit and received a credential. 

• Older Youth 

− Entered Employment Rate: the percentage of older youth who are 

employed in the quarter after exit from the program, excluding those 

who were employed at entry and youth who were in postsecondary 

education or advanced training (but not employed) in the quarter after 

exit. 

− Employment Retention Rate: among older youth employed in the 

quarter after exit from the program, the percentage who are employed 

in the 3rd quarter after exit, excluding those who were in 

postsecondary education or advanced training (but not employed) in 

the 3rd quarter after exit. 

− Earnings Change: among older youth employed in the quarter after 

exit, the change in earnings from the 2nd and 3rd quarters before entry 

into the program to the 2nd and 3rd quarters after exit from the 

program, excluding those who were in postsecondary education or 

advanced training (but not employed) in the 3rd quarter after exit. 
I-8

− Credential Rate: Among older youth, the percentage who are 

employed or in advanced training or postsecondary education in the 

quarter after exit and received a credential by the 3rd quarter after exit. 

• Younger Youth 

− Diploma Attainment Rate: among youth without a secondary school 

diploma or equivalent at entry into the program, the percentage who 

attain a diploma or equivalent either during participation or within a 

quarter after exit from the program, excluding youth still attending 

secondary school at exit. 

− Retention Rate: the percentage of younger youth who are employed, 

in the military, in a qualified apprenticeship, attending postsecondary 

education, or in advanced training in the 3rd quarter after exit, 

excluding those still attending secondary school at exit. 

− Goal Attainment Rate: the percentage of skill attainment goals met 

within a year after being set. Goals in the areas of basic skills, work 

readiness skills, and occupational skills are set for all in-school youth 

and for out-of-school youth assessed to be in need of skills in any of 

the three skill areas. 

• Customer Satisfaction 

− Participant Customer Satisfaction: measured using the American 

Customer Satisfaction Index (ACSI) and based on a telephone survey 

conducted after exit from the program. 

− Employer Customer Satisfaction: measured using the American 

Customer Satisfaction Index (ACSI) and based on a telephone survey 

conducted after services are completed. 

Broadly speaking, the employment, retention, and earnings measures were designed 

around the fact that states are to use UI wage records as the primary data source. As a result, 

each of these measures focuses on exiters within each calendar quarter, because these data are 

obtained in quarterly increments. 

As noted above, individuals receiving only informational and self- (or minimal staffassisted) service core services are excluded from all performance outcomes. Also excluded are 

individuals who were institutionalized or deceased when they exited from WIA or were unable to 

complete services for health reasons. 

Beyond these seventeen measures mandated by the legislation, WIA also explicitly allows 

states to establish additional measures of performance, at their option. As will be discussed in 
I-9

detail in subsequent chapters, several states in our sample have done so, as have some of the 

local areas. 

Negotiations 

As noted above, under JTPA, adjustment models were widely used in an effort to account 

for the different characteristics in the type of customers being served by local areas. Under WIA, 

these adjustment models are not used.5 Instead, DOL negotiates goals (referred to as "negotiated 

levels") for the performance outcomes with each state, and states, in turn, negotiate with their 

local areas to set local goals. These negotiations are intended to incorporate differences between 

areas, including differences in economic conditions, the characteristics of participants when the 

participants entered the program, and the services to be provided. Further, these negotiations are 

intended to factor in the extent to which the levels set will enable states and local areas to obtain 

high levels of customer satisfaction, as well as promote continuous improvement in their efforts. 

In large part, the switch from adjustment models to negotiations was intended to simplify 

the process of setting performance levels, because these negotiations would no longer rely on 

complicated mathematical procedures but, rather, on reasoned decisions about how serving 

customers with greater challenges to achieving successful outcomes would impact the outcomes 

obtained by these areas. Enabling states, and subsequently local areas, to present data in support 

of their case concerning appropriate levels of performance is consistent with the general thrust of 

WIA, in that much of the control is transferred away from the federal level and toward states and 

local areas. Through a negotiation process, states’ expected levels of performance would be set 

and, subsequent to this, the levels for local areas within each states would be set using a similar 

process. In Chapter II, we discuss these negotiation processes, detailing how they proceeded 

both at the state and local area levels for the sites in our sample. 

Incentives and Sanctions 

The establishment of performance goals that states and local areas view as reasonable takes 

on increased importance in light of WIA’s stipulation that states and local areas are to be 

awarded incentives for exceeding their performance goals, but, conversely, should be subject to 

sanctions for a persistent inability to meet them. To receive an incentive award, states must 

exceed their performance goals on average within each of 4 groups of performance outcomes: 

 

5

 Some states do use models for setting performance targets for their local areas, and for negotiating their targets 

with DOL. However, the use of such models is far more sporadic and less uniform than under JTPA. 
I-10

adults, dislocated workers, youth, and customer satisfaction.6 In addition, states must exceed the 

performance goals established for Title II of WIA (adult literacy programs) and for vocational 

education programs under the Perkins Act, thereby broadening the scope necessary for strong 

performance beyond only the agency that has authority over WIA Title I. To obtain such 

incentives, then, multiple agencies must work together, or at the very least all be operating strong 

performing programs. These incentive grants can range from $750,000 to $3 million. 

States that fail to meet performance goals may be subject to sanctions. Unlike incentives, 

sanctions depend only on WIA Title I outcomes. For the first year of failure, the state can 

request technical assistance from DOL. If a state fails to meet performance goals for two 

consecutive years, that state can lose up to 5 percent of its WIA allocation. The amount of the 

penalty is to be based on the state's degree of failure to meet negotiated levels. 

Local performance relative to their negotiated levels can also lead to incentives or 

sanctions. States have broad discretion in determining their policies regarding negotiating 

performance levels, determining incentives, and applying sanctions, and in the subsequent 

chapter we identify the various policies concerning incentives and sanctions the states in our 

sample have adopted. 

Continuous Improvement 

Another key aspect of the performance accountability system is the desire to foster 

continuous improvement. There are, though, many ways of conceptualizing the notion of 

continuous improvement. One approach is to view continuous improvement as synonymous 

with increases in program outcomes. DOL’s Employment and Training Administration (ETA) 

and others, however, have put forth an alternative view. A consultation paper published by ETA 

in the Federal Register in 1999 stated: 

According to leading Baldrige experts, continuous improvement is the systematic 

and ongoing improvement of products, programs, services, and processes by small 

increments and major breakthroughs. Continuous improvement is the process of 

building dynamic, high achieving systems within every organization, and becomes 

embedded in the way the organization conducts its daily activities. 

Thus, continuous improvement can be viewed as more than just increases in measured outcomes. 

Rather it can be viewed as a process by which methods to enable improvements become 

institutionalized. 

 

6

 To receive an incentive grant, states must achieve an average score of 100 percent or more in each Title I area (adults, dislocated workers, 

and youth) and for customer satisfaction, and at least 80 percent of the targeted performance level on each of the 17 measures.
I-11

By this view of continuous improvement, the contribution of the accountability system can 

be measured not just in terms of increases in the measured outcomes, but also in terms of the 

usefulness of the performance outcomes in the continuous improvement process. The Baldrige 

quality model incorporates a process whereby "business results" (that is, outcomes, under WIA) 

are measured, processes are changed with the intent of improving outcomes, and outcomes are 

measured again to assess the effectiveness of the changes. It would be natural for the 

performance outcomes to be used in this feedback loop to determine the effectiveness of process 

changes. 

However, many of the WIA performance outcomes may be inappropriate for this purpose, 

because of the long time lag between when processes are changed, when outcomes occur, and, 

finally, when data measuring outcomes are available. Consider, for example, the original entered 

employment rate. To use the entered employment rate in its continuous improvement process, a 

local area would need to make changes in its service process, say by adopting an improved 

assessment instrument. Participants then experience the new process, probably early in their 

participation, and complete all services. Entered employment, as measured in the accountability 

system, occurs in the quarter after exit. Because unemployment insurance (UI) wage records are 

the primary data source for documenting employment under the current performancemeasurement system, data on entered employment is typically not available for yet another two 

quarters. Allowing 3 months for participation after experiencing the revised assessment, data on 

entered employment outcomes would not be available for at least a year after the process was 

changed. Data on the retention and earnings outcomes would not be available for another 6 

months, which clearly limit use of these measures in continuous improvement. 

Despite these difficulties, fostering continuous improvement is an important goal for states 

and local areas under WIA. As such, in this report we examine whether the states and local areas 

in our sample have developed continuous improvement processes, and the extent to which sites 

in our second round are further along in the development of these systems. However, in light of 

the limitations described above, we also explore whether areas that have adopted continuous 

improvement strategies use the performance measures in this process, or whether they have 

developed other outcome indicators for this purpose, and for carrying out program oversight and 

supporting program management more generally. These strategies include the use of additional 

performance measures other than the ones required by law or other procedures and practices 

useful in achieving continuous improvement. 
I-12

Implementation of the Common Measures 

In 2001, President Bush announced his Management Agenda aimed at improving the 

management and performance of the Federal government. As part of this agenda, he targeted budget 

and performance integration, focusing on the need to make consistent benchmarks for performance 

across programs with similar goals and objectives. In response to this, the Office of Management and 

Budget (OMB), in cooperation with other agencies, developed consistent performance measures for 

programs with broadly similar goals, including those programs, such as WIA, that focus on job 

training and employment. These measures, known as the common measures since they intend to 

make performance accountability measured consistently across common programs, quickly became a 

focus for DOL in its assessment of WIA performance. In 2003, DOL issued its first comprehensive 

guidance on these common measures through TEGL 15-03. This guidance was subsequently revised 

in TEGL 28-04, which was the operative guidance during the time of our site visits.7

 Through this 

guidance, DOL clarified that the common measures were to be implemented for adults beginning in 

Program Year (PY) 2005, which began July 1, 2005, and for youth in PY 2006 (beginning July 1, 

2006). 

Although the common measures are conceptually similar to several of the core measures used to 

assess performance under WIA, they are calculated somewhat differently. Further, because the core 

measures were specified as part of the initial WIA legislation and regulations, the common measures 

do not simply replace all the core measures in the accountability system but are, instead, an additional 

means of documenting performance.8

 Although WIA reauthorization may result in the elimination of 

all the core measures that are not included in the common measures, to date such reauthorization has 

not been passed.9 Thus, during the course of our final round of visits, states and local areas were 

subject both to the original performance measures identified above, as well as the common measures 

(for adults and dislocated workers). 

 

7

 Even more recently, DOL has issued further guidance through TEGL 17-05 which expands upon earlier 

guidance, provides clarification in a number of areas, and describes changes to the common measures that will 

be enacted for PY 2006. This TEGL, however, was issued on February 17, 2006 and, thus, came after our 

final round of site visits had been completed. Thus, for this report, we focus primarily on the guidance in 

TEGL 28-05, as this was the relevant guidance during the time of our visits. 

8

 The common measures that are conceptually similar to adult and dislocated worker core measures do, in fact, 

replace these core measures (i.e., the common measure of entered employment replaces the identically named 

core measure; the same is true for the employment retention and earnings increase measures). 

9

 In fact, TEGL 28-04 notes that, “as ETA programs are reauthorized, [they] will work to ensure that statutory 

performance measures are consistent with [the] common measures.” (TEGL 28-04, p. 2) 
I-13

There are three common measures for adults (and dislocated workers) and three for 

youth.10 The three adult measures include: entered employment, employment retention, and 

earnings increase.11 Each of these measures is described below: 

• Entered Employment: Of those who are not employed at the date of participation, 

the number of participants who are employed in the first quarter after the exit 

quarter divided by the number of participants who exit during the quarter. This is 

identical to the core entered employment measure.12

• Employment Retention: Of those employed in the first quarter after the exit 

quarter, the number of participants who are employed in both the second and 

third quarters after the exit quarter, divided by the number of participants who 

exit during the quarter.13 This extends the original retention measure to include a 

requirement that individuals also be employed during the second quarter after 

exit (the core measure required only employment in the 3rd quarter after exit). 

• Six Months Earnings Increase: Of those who are employed in the first quarter 

after the exit quarter, the total earnings in the second and third quarters after the 

exit quarter minus the total earnings in the second and third quarters prior to the 

quarter in which the individual began participation. This total is then divided by 

the number of participants who exit during the quarter (and are employed in the 

first quarter after the exit quarter).14 This measure, too, is identical to the core 

 

10 A fourth measure, program efficiency, is part of the common measures, but programs do not calculate or report on 

this measure; instead it is calculated by DOL at a national level using existing program management data. Thus, 

states and local areas have no control over this measure and, in fact, will not be evaluated individually on it. 

11 TEGL 17-05 changes the earnings increase measure to a post-program only measure of average earnings, 

effective starting in PY 2006. As noted above, however, this guidance was issued well after our site visits 

and, therefore, in discussing this measure we focus on earnings increase, which in any case is the relevant 

performance indicator for PY 2005. 

12 TEGL 17-05, the most recent guidance on the common measures, provides an identical definition, though it 

specifies adult participants, rather than simply participants. 

13 This measure, too, is defined identically in TEGL 17-05, with the exception that it now specifies adult

participants rather than simply participants. 

14 This measure was changed substantially by the guidance issued in TEGL 17-05. Under this latest guidance, as 

of PY 2006, the measure is now called Average Earnings, and is now, of those adult participants employed in 

the first, second and third quarters after exit, simply the sum of earnings in the second and third quarters after 

exit, divided by the number of adult participants who exit in the quarter. Thus, the most recent guidance 

issued removes pre-program earnings as a consideration in this measure. This guidance was not issued, 

however, until well after our final round of site visits had been completed. As a result, we collected data on 

the six month earnings increase measure that was in place at the time of our visits. 
I-14

earnings change measure for adults (though somewhat different from that for 

dislocated workers)15. 

The three youth measures, in contrast, are very different from the core measures described 

above. The first difference is that there is no longer a distinction made in measures between 

older and younger youth. Similarly, the measures themselves are substantially different than the 

core measures. And each of the common measures tends to place less emphasis on serving inschool youth, as two of the measures specifically exclude these individuals from the calculations, 

and the third includes them as a negative outcome unless the youth has attained their diploma or 

certificate. Each of the three youth measures (to be implemented in PY 2006) is described 

below: 

• Placement in Employment or Education: Of those who are not in post-secondary 

education, employment, or the military at the date of participation, the number of 

participants who are in employment or the military or enrolled in post-secondary 

education and/or advanced training/occupational skills training in the first quarter 

after the exit quarter divided by the number of participants who exit during the 

quarter.16 This measure, in effect, uses aspects of the original entered 

employment calculation among older youth but adds placement in education as 

an additional positive outcome. Removed from this calculation, however, are 

those youth who were in post-secondary school at the date of participation (who 

are captured in the original younger youth retention measure), thereby reflecting 

DOL’s renewed emphasis on serving primarily out-of-school youth. 

• Attainment of a Degree or Certificate: Of those enrolled in education (at the date 

of participation or at any point during the program), the number of participants 

who attain a diploma, GED, or certificate by the end of the third quarter after the 

exit quarter divided by the number of participants who exit during the quarter. 

This is conceptually similar to the original younger youth diploma attainment 

measure, but adds an additional two quarters following exit to attain the 

certificate. Additionally, this measure, unlike the original diploma attainment 

one, includes those still enrolled in secondary school at exit, meaning these 

individuals will count in the performance calculations. 

• Literacy and Numeracy Gains: Of those out-of-school youth who are basic skills 

deficient, the number of participants who increase one or more educational 

 

15 For dislocated workers, the core earnings measure was the earnings replacement rate, which was calculated as 

the earnings in the second and third quarters after exit divided by the earnings in the second and third quarters 

prior to their dislocation. Thus, rather than an average of earnings, the core measure was instead an average 

of the proportion of their prior earnings dislocated workers earned following program exit. 

16 For this and the remaining two youth measures, TEGL 17-05 provides an identical methodology for calculating 

the measures, except that, similar to the adult measures, the guidance refers to youth participants rather than 

simply participants. 
I-15

functioning levels divided by the number of participants who have completed a 

year in the program plus the number of participants who exit before completing a 

year in the program. This measure has little parallel with any of the core 

measures (although the younger youth goal attainment rate could be similar for 

some youth, depending on the specific goals set). 

As noted above, the adult measures were implemented on July 1, 2005, and, although the 

youth measures were to be implemented in PY 2006, the guidance for them had been issued prior to 

July 1, 2005. Thus, our final round of visits, which occurred in the late summer of 2005, took place 

at a time when the common measure were just being implemented and information and technical 

assistance concerning them was being disseminated. As such, we focused heavily on how these 

new measures were affecting programs, including the customers they serve, the services they 

provide, and the outcomes they expected to obtain with these customers. 

Methodology and Key Issues 

Having established the federal framework and conceptual underpinnings of our evaluation, in 

this section we describe the methodology used for this evaluation (and this report). 

We designed our technical approach to the evaluation in an effort to provide a 

comprehensive picture of the performance measurement system and to examine the intended and 

unintended consequences for participant selection, and service design and delivery. Further, our 

methodology enables us to gauge the current system’s ability to further WIA’s objectives and 

adequacy in supporting program improvements, and to discuss the implications of the findings 

from our data collection for the recent implementation of the common measures, and for issues 

of WIA reauthorization. 

Data Used in the Final Report 

Our overall evaluation approach was designed to weave together information from multiple 

data sources to provide both the breadth and depth of information needed to fully address the 

study’s chief research questions. Although this Final Report relies heavily upon data collected 

from our final round of site visits, conducted in the late summer and early fall of 2005, we also 

draw throughout on data we have already presented in our earlier Interim and Survey Reports.17 

 

17 The Interim Report summarized our data collection and analytic efforts from an initial round of site visits, 

conducted in 2003, and was submitted in 2004. The Survey Report presented results from a local area survey 

and quantitative analysis of all local workforce investment areas (and corresponding state policies), and was 

submitted in 2005. For more detail, please see these earlier reports, although we do provide a brief 

description of each below. 
I-16

As noted, however, our chief source of data on which the findings of this Final Report are 

based are visits conducted in seven separate states, and eleven local areas within those states.18 

These site visits employ a case study design that provides detailed information about the 

influences of performance accountability systems, at different levels of aggregation and in a 

variety of different contexts. 

Our sample selection strategy to select the sites which we visited was detailed in the 

Interim Report. As such, we will only briefly summarize our approach here. There were three 

key elements in our sampling strategy. First, we used a stratified sampling approach, stratifying 

by DOL regions, and used random selection within these clusters. Second, because states, local 

areas, and service providers all play important yet unique roles in the performance accountability 

system, we utilized a nested sampling plan, whereby states were first sampled, and local areas 

were next sampled within the states. In each case, we selected areas randomly with their 

probability of selection being proportional to their size (in WIA funding). Third, we scheduled 

the visits so that the site visits occur in two separate program years—for the Interim Report we 

completed visits to eight states and fourteen local areas during PY 2002. For this Final Report, 

we delayed our visits until the common measures had been implemented. Thus, these visits 

occurred in early PY 2005. For further description of the sample selection methodology, please 

see Chapter I of the Interim Report for this evaluation. 

 Our final sample of states and local areas is shown in Table I-1, which also displays the 

timing of these visits. Although the initial sample was selected so that it would be broadly 

representative of states and local areas, as described above, the division between those visited in 

the initial and final rounds was not done randomly. Instead, this division was created largely 

based on a determination of which states and local areas could host a visit during our initial 

round of data collection.19 Thus, while the overall sample can be taken to be reflective of states 

and local areas, data from each round taken separately may well be biased in unknown ways as a 

result of this distinction. As a result, although this report focuses heavily on data collected from 

 

18 Over the life of the project, we conducted site visits to a total of 15 states and 25 local areas. 

19 Initially, we had planned to cover much the same material in the second round of site visits as we had in our 

initial visits. As such, selecting sites to visit in the initial round non-randomly would have been a less 

important issue, as the overall sample would have been representative. Unfortunately for this 

representativeness, however, we received the request to focus heavily on the common measures for our second 

round of visits after completing the initial visits. In retrospect, given this knowledge we would have divided 

the two samples in a more scientific way. 
I-17

the final round of site visits, it also includes comparisons with and discusses similarities to the 

data collected earlier. 

Further, we made one substitution in the final round of site visits. Our original sample 

included Arkansas as one of the states. Unfortunately, despite repeated efforts to include them in 

the study, we could not garner their cooperation and, thus, were forced to replace this state. We 

selected, at random, a state from the same DOL region (Region 4) as a replacement. Once we 

had selected this replacement state (Colorado), we selected a local area within it using the 

procedures described above. 

Data from the final round of site visits form the core of this Final Report, and are discussed 

fully in the subsequent chapters. 

Additional Study Reports 

In addition to this Final Report, we have already submitted as part of this project an Interim 

Report, focusing on results obtained from our initial round of site visits and from data collected and 

summarized through a series of workgroups on workforce performance accountability that had been 

conducted over the past two decades. Additionally, at the request of DOL we completed a Survey 

Report that presented the results of data collected through a survey administered to each of the 

approximately 600 workforce investment areas.20 This report summarized descriptive information 

about the full range of areas’ policies and practices that might be related to performance 

accountability, including what performance goals and measures have been established and what tools 

are used for purposes of performance management. This Survey Report also included a quantitative 

analysis of data from the WIASRD and data on state performance policies. This analysis examined 

how policies and practices relating to performance accountability implicitly relate to decisions about 

what types of customers are served and with what services and outcomes. 

 

20 Initially, the survey data and corresponding quantitative analysis were to be included in the Final Report. 

However, because we delayed the final round of site visits (and, hence, the Final Report) until the common 

measures had been implemented, at DOL’s request, DOL asked that we compile a separate report using these 

data. This report was submitted in 2005. 
I-18

Table I-1 

Sample of State and Local Areas for Case Studies 

States 

Visit 

Occurred LWIA(s) 

California PY 2002

PY 2002 

City of Los Angeles 

Tulare County 

Michigan PY 2002

PY 2002 

City of Detroit

Gennessee-Shiawassee Counties 

New York PY 2002

PY 2002 

Monroe County

Suffolk County 

North Dakota PY 2002 North Dakota Consortium (Single WIA State) 

Ohio PY 2002

PY 2002 

Ohio Option Area Southwest Ohio 

(Cincinnati/Hamilton County)

City of Cleveland 

Washington PY 2002

PY 2002 

Seattle/King County

North Central Washington/Columbia Basin 

Wisconsin PY 2002

PY 2002 

Milwaukee County

Bay Area (Green Bay) 

West Virginia PY 2002 Kanawha County 

Alabama PY 2005

PY 2005 

Mobile

Alabama Workforce Investment Area 

Colorado PY 2005 Weld County 

Idaho PY 2005 Idaho Works 

Maine PY 2005 Coastal Counties Workforce Board 

New Jersey PY 2005

PY 2005 

Balance of Hudson Co. Less Jersey City

Cumberland County 

North Carolina PY 2005

PY 2005 

Region L Job Training Consortium

Cape Fear SDA Job Training Consortium 

Oregon PY 2005

PY 2005 

The Oregon Consortium and Oregon 

Workforce Alliance

Mid-Willamette Workforce Network 
I-19

This multi-pronged approach combining a variety of types of data enables us to address a 

wide variety of issues of interest to DOL, including, most prominently, how the performancemeasurement system currently is operating, as well as how the recent implementation of the 

common measures has been received and is affecting states and local areas. Further, the data can 

contribute to the extensive discussions taking place surrounding the reauthorization of WIA. We 

will discuss each of these issues in turn. 

Road Map for the Remainder of the Report 

In the remainder of this report, we provide a summary of our findings, primarily from the 

final round of site visits conducted in PY 2005. The results presented in this report represent 

cross-site analyses from these localities, and describe the general themes and trends discovered 

as part of our data collection. Generally, we present the findings across all sites, although, where 

there are aspects that are highly unique to a single state or local area, we highlight this area in the 

text. This Final Report is intended to parallel our presentation in the Interim Report, thereby 

updating our findings from the earlier report, and making for easy comparison between the two 

sets of data (drawn from the separate rounds of site visits). Hence, the structure of the chapters 

described below generally mirrors that presented in the Interim Report. One critical difference 

between the reports, however, is that, in each chapter of this Final Report, we not only describe 

how the original performance measurement system has impact on those operating WIA, we also 

focus on how the recently implemented common measures are affecting this system. Thus, 

throughout each chapter, we identify the ways in which states and local areas have been affected 

by the implementation of the common measures. 

In Chapter II, we provide a summary of the elements of the performance system in the sites 

we visited. Included in this discussion are descriptions of the negotiation process—both between 

states and the Federal DOL, as well as between states and their local areas—that took place to 

establish performance levels, the specific features of the accountability systems, mechanisms and 

operational definitions established to help operate the system, information on the data collection 

and reporting systems that produce performance reports, and the extent to which the 

implementation of the common measures is likely to change any of these. 

Chapter III describes how the performance system has been communicated to various 

levels of the system, including partners and contractors or service providers. This chapter 

focuses on the extent to which the performance-measurement system has been an impediment or 

incentive to establishing partnerships with agencies beyond WIA, and how the system has 

influenced those partnerships that have been established. Additionally, we discuss the extent to 

which contractors and service providers have been made aware of the performance-measurement 
I-20

system, and whether their performance or activities are explicitly tied to this system. In addition, 

we explore how the implementation of the common measures may affect the communication of 

performance issues throughout the system. 

In Chapter IV, we detail the effects of the performance-measurement system on the 

participants and services under WIA. In this chapter, we describe the degree to which the 

customers who are being served under WIA, at least in the states and local areas we visited, are 

impacted by the performance-measurement system. Additionally, we focus on whether the 

number, types, or sequence of services provided to WIA customers is affected by this 

measurement system. Further, we examine how states and local areas attempt to manage their 

programs and services to achieve good outcomes, and how these efforts are influenced by the 

measurement system, including by selectively focusing on particular customers, or gearing 

service design toward achieving high levels of performance. We also describe the implications 

of the move toward common measures will have on who is served, what services they receive, 

and what outcomes they obtain. 

Finally, in Chapter V, we summarize the key findings from our final round of site visits. 

For those seeking a concise summary of the primary findings from this report, this chapter serves 

this role. 
II-1

II. ELEMENTS OF THE PERFORMANCE ACCOUNTABILITY 

SYSTEM 

While the performance measures themselves—the 17 required measures and the common 

measures—may be at the center of the accountability system, there are numerous other features 

of this system that impact states’ and local areas’ ability to perform successfully on these 

measures. These other aspects of the accountability system include a negotiation process to 

arrive at agreed-upon performance standards for states to meet; and a system of incentives and 

sanctions to encourage high performance, and technical assistance to help states to correct low 

performance. Similarly, DOL encourages (but does not require) states to use continuous 

improvement systems to provide its customers ever more effective services, for better and better 

results. Partly in an effort to document and calculate WIA performance measures, states are 

required to collect data about the registered customers they serve and the outcomes they achieve. 

Indeed, under the common measures, they are now encouraged to track the volume, 

characteristics, and outcomes over time of nearly all individuals served by the universal 

workforce system. Thus, this data collection and management can also impact states’ and local 

areas’ ability to perform successfully on the required measures. Finally, WIA programs are 

generally co-located with and work in conjunction with other service delivery systems that 

receive funding from other federal departments, and from state-funded programs and initiatives. 

In light of this interaction, WIA performance often co-exists with other state goals, which may 

well impact the ability to meet performance goals. 

In this chapter, we explore each of these features of the performance accountability system. We 

pay special attention to the ways in which states and local areas in this round of visits—

compared to the visits conducted for the interim report—use the elements of the system to 

encourage and achieve good performance. 

Use of Performance Measures 

Most of the seven states we visited in our final round of site visits were almost exclusively 

focused on WIA-specific, performance-oriented policy objectives, and use the performance 

measures as a means to emphasize specific aspects of their program or services or establish 

bigger-picture accountability goals. Thus, several states and local areas emphasize certain of the 

performance measures, highlight specific types of services, target specific populations, or focus 

on the volume of customers. These areas have embraced the core measures, but have seized 

upon certain of them as a way to focus their service design and delivery and highlight specific 

aspects of their program. For example, Alabama LWIA emphasizes retention, while Oregon has 
II-2

focused special attention on the wage measures, as has Mobile. These states and local areas 

therefore use the WIA accountability system not only to document their success, but also to 

focus their efforts and services on areas of particular interest to them or their Boards. 

While most states, and numerous local areas use the performance measures to establish biggerpicture accountability goals, a relative few articulate a philosophy that can be summed up as, 

“Provide good services to customers, and performance will take care of itself,” thus diminishing 

the importance of the performance measures themselves in focusing services or establishing 

system-wide goals.1

 North Carolina and the Alabama LWIA and Cape Fear local area all 

espoused this philosophy. In these states and local areas, the performance measures, while 

important, are not used to focus services or program design. 

Use of Additional Measures 

In addition to the core measures under WIA, and the common measures implemented in July, 

2005, we also examined whether states and local areas have adopted additional measures to 

assess their performance and target their service design or delivery. In our Interim Report, we 

had found so many additional measures being used that we created a system to classify these 

additional measures, identifying proxy, system, and alternative measures.2 In our Interim Report, 

we found that all eight states, and a majority of local areas (8 of 13) used performance measures 

in addition to the required WIA measures to help the state or local area achieve better 

performance and other goals. Somewhat in contrast, in this round of site visits, hardly any of the 

states or local areas we visited were using additional measures as formal benchmarks, though 

several continued to use additional measures informally to help gauge their success or to direct 

their attention to areas of specific emphasis for the program(s). 

For example, at the time of our visit, Maine was in the process of developing a work plan for 

“access to services” for special populations, including women, the disabled, older workers, lowincome workers, and minorities. Similarly, Mobile is attempting to increase its service to out-ofschool youth.3

 In addition to using the WIA performance measures to help focus their programs, 

two states – North Carolina and Oregon – use broad cross-program performance initiatives that 

 

1

 This is also an argument for diminishing the importance of the WIA required measures, something we talk more 

about in Chapter III. 

2

 Proxy measures were intended as a substitute for the federally required measures, but measure the same general 

outcome. System measures evaluated the performance of an entire system, not just WIA programs. Alternative 

measures were used to estimate and evaluate services or processes that were not measured by the required 

measures and may reflect additional policy objectives at the state or local levels. 

3

 Numerous states and local areas are actively shifting the focus of their services to out-of-school youth in 

response to DOL policy and in anticipation of WIA reauthorization and the common measures. We will talk 

more about the connection to the common measures in Chapter IV.
II-3

include WIA and numerous other state systems, such as TANF, vocational rehabilitation, K-12 

education and the community colleges. In these two states, the WIA measures are simply part of 

a larger set of benchmarks, and reflect the concept that these programs can use performance 

measures to facilitate working together to effectively serve customers. In general, the strength of 

these initiatives is that they approach accountability from a perspective that is broader than 

discrete programs, thereby facilitating partnerships between programs and smoothing transitions 

for customers accessing services from different programs. Indeed, Oregon explicitly identifies 

itself as being a “performance-driven” state, and is the only state in our second round sample that 

has established a system of additional measures, which it tracks on its state-designed PRISM 

system. These measures are employment placement, employment retention, wage gain, welfare 

caseload reduction, and welfare recidivism. Given that many WIA participants do not receive 

welfare, it is clear from the use of these measures that they are designed for a broader set of 

programs than just WIA. 

In addition, both Oregon and New Jersey have expressed an interest in the Integrated 

Performance Information (IPI) system of measures that grew out of a six–state project funded by 

DOL.4

 The IPI project developed measures that are conceptually similar to WIA measures (but 

are calculated differently) and also added several new measures. The IPI measures are shortterm employment (employed in the second quarter after exit), long-term employment (employed 

in the fourth quarter after exit), earnings level (median earnings during the second quarter after 

exit), credential completion rate (within one year of exit), and repeat employer customers 

(percentage of employers who return to the same program for service within one year). 

Additional indicators are employer market penetration, taxpayer return on investment, and 

participant return on investment. 

Similar to the data presented above for states, only two of the 11 local areas we visited use 

additional measures of performance, and in both cases these additional measures are used 

primarily to enhance the likelihood that the areas will meet their targets on the core measures. 

The Alabama LWIA has 1) established a method of evaluating its staff that uses the original 

WIA measures. Under this approach, their goal is that 100% of customers will meet three 

measures: entered employment, employment retention, and earnings gains; and 2) developed 

two proxy, or intermediate process measures that they believe are predictors of success for the 

WIA measures. These two measures are percentage of customers completing training and the 

percentage obtaining a credential. For each of these measures, too, the area has established a 

benchmark of 100%. 

 

4

 See Washington State Workforce Training and Education Coordinating Board, “Integrated Performance 

Information for Workforce Development: A Blueprint for States,” 2005. 
II-4

Establishing a goal of 100% completion provides a strong incentive for local areas to ensure 

that customers achieve positive outcomes on the credential measure, though there are no 

sanctions applied if the area does not meet this goal. Training is also associated with increased 

wages, and thus this goal would also encourage high performance on the wage gain measure. 

Establishing the goal that all customers obtain a credential is yet further support for customer 

success on the credential measure, assuring that staff does not approve training programs that 

do not lead to a DOL-recognized credential. 

At the time of our visit, Coastal Counties—the second LWIA with additional measures—was 

in the process of implementing two One-Stop measures they think will support their WIA 

performance outcomes. These measures are: 1) the number of individuals that come into the 

Career Centers, and 2) the number of individuals who received referrals. Coastal Counties had 

not yet set goals for these measures, but they are part of a broader effort to improve 

performance in the area, largely in response to a corrective action plan required by the state 

because of the areas’ multi-year failures of one measure. Thus, both the local areas that have 

adopted additional measures use them (or plan to, in the case of Coastal Counties) as a means 

to improve their performance on the core measures. 

Finally, while not implementing any formal additional measures, numerous respondents have 

established interim indicators (i.e., employed at exit, hourly wage and hours per week worked) 

to gauge potential performance on the core or common measures, or track per-participant costs 

and local area or provider expenditures to monitor their progress.5

 Moreover, similar to the 

method employed by Alabama LWIA in using the original WIA measures as a basis for 

additional measures, Idaho LWIA simply took as its goal to achieve one or two percent higher 

on each WIA goal set for it by the state. 

Effects of Common Measures on Focus on Performance 

Nearly all the states we visited believe the common measures will have little effect on their 

overall emphasis on performance. Alabama feels that the common measures are in line with 

the state’s general focus on accountability, and thus will not cause any major changes.6 

Maine, on the other hand, has started tracking self-services in response to their perception that 

the common measures move toward counting more customers as participants, but feels this is 

 

5

 These interim indicators are similar to what we referred to as proxy measures in our Interim Report. 

6 The governor is interested in developing some common measures for government programs. In comparison with 

other state government programs, the workforce development system is much farther ahead. 
II-5

simply an extension of the already-existing focus they had on achieving strong performance.7 

Both Maine and Oregon note that the common measures compel even greater coordination 

between One-Stop system partners, especially for coordination between data systems, because 

several of the partners are now subject to the same performance measures. According to 

respondents in these states, prior to the common measures’ implementation, agencies that were 

trying to work together may well have had competing goals (i.e., finding work for customers 

quickly versus investing in training to ensure high-paying and stable employment). Oregon 

also has a longstanding interest in creating cross-program measures and enhanced flexibility 

from regulatory constraints through regulatory waivers and Work-Flex. Thus, although the 

common measures are not the cause of these activities, the state recognizes the importance of 

its efforts in light of the new accountability system. 

One partner identified an unwanted impact that the common measures are having on its 

overarching policy goals, and especially on the types of customers it serves. This local area 

suggests that the common measures are forcing a greater emphasis on youth academics and 

college-preparation to the detriment of vocational education and the trades, a critical option for 

the youth who access the WIA Youth program. This belief derived from a concern about the 

literacy and numeracy gains measure that will be implemented for youth in 2006. Because outof-school youth will be subject to this measure, the partner believes that the emphasis must now 

be on increasing these youth’s educational skills rather than focusing on learning a trade so they 

can obtain employment. This partner acknowledged that this is speculative at this point, since 

the measure has yet to be enacted, but their concern is that, once implemented, programs will 

need to attempt to convince many out-of-school youth to focus on academics rather than on 

obtaining employment in order to meet their target on this measure. 

Negotiating Standards 

In this section, we examine trends in the negotiation process—both federal with state and state 

with local—and the effect of these trends on the sites we visited. 

In our visits for the final report, we found that states still view the negotiation process negatively, 

as did the vast majority of those states and local areas in our first round of visits. Six of the 

states we visited for the Final Report identified the negotiation process as being “a negotiation in 

 

7

 Under the common measures, for the adult and dislocated worker programs, a person is considered a participant 

(and thus, included in the performance measures) when he or she is determined eligible and receives a service 

funded by the program in a physical location. This is slightly more encompassing than the definition of a 

participant under the original WIA measures, in which an individual must be registered when they are 

determined eligible and have received a service funded by the program that entails significant staff involvement. 

In addition, as part of WIA reauthorization, the performance measures may apply to all individuals—including 

those who participate exclusively in self-service or informational activities. 
II-6

name only.” On the other hand, three of those states are resigned to meeting whatever standards 

are set for them, despite what they perceive as a lack of true negotiation. Further, one state 

reported that the negotiation rules changed several times in the middle of the process, and 

expressed frustration at not knowing what the important features of the negotiation were. In 

contrast, one state expressed appreciation that DOL had offered a mid-course renegotiation for 

PY 05 and 06 standards. 

We observed an interesting phenomenon between this round of site visits and the last. In our 

interim report, we found that states were unhappy with the level of sophistication that DOL 

expected to be brought to the negotiation process, a sophistication states say they were 

unprepared to demonstrate. However, in this round, four states say they prepared economic and 

customer characteristic data in preparation for the negotiation process—several ran regression 

analyses which they used in their negotiations with the federal government and with their local 

areas—and yet the state-proposed goals were rejected. The general impression of the states is 

that DOL expects ever-increasing levels of performance, an expectation one state says will 

eventually become unattainable. 

The local areas we visited mirrored their respective states’ opinions about the negotiation 

process, although less uniformly: chiefly, that “negotiation” is a misnomer. Seven of the 

eleven local areas hold this view. This view makes sense in light of the fact that of the seven 

states we visited, four states (or 57%) pass the same federally assigned performance standards 

down to each LWIA . This represents a higher incidence of this behavior than happens 

nationally. We noted in our survey report that only 30 percent of states pass state goals down 

to the local areas with no further negotiation. However, while experiencing frustration, some 

local areas understand that once the states receive their performance targets from DOL, the 

level of flexibility for negotiating with local areas is limited, given the targets for which the 

state is held accountable. Three local areas say that the state tries to be fair, but that, in the 

most recent round of negotiations (for PY 05 and 06 rates), DOL’s negotiations with the state 

required the state to re-negotiate its goals with local areas. One local area started PY 05 

without standards, because the state had yet to negotiate with DOL. 

In three of the states we visited, local areas explained that the state used to give them the 

opportunity to negotiate with each other. For example, one area could negotiate to perform 

higher on one measure in exchange for performing lower on another measure, with the second 

local area striving for a lower goal and a higher goal, respectively, on the two measures. 

However, in all three states, the local areas were not enthusiastic about this type of negotiation, 

and the practice soon ceased. 
II-7

In order to identify trends in the negotiation process, and to interpret state perceptions of a 

general lack of true negotiations, we examined state goals and performance from PY 00 through 

PY 03, and looked at PY 03 and PY 05 negotiated goals. We posed two questions related to 

state (and local) negotiated targets and actual performance: 

• In the event of a state or local area exceeding its negotiated rates by more than ten 

percent in a given year, do negotiated rates go up by ten percent or more in their 

next round of negotiations?8

• In the event of a state or local area failure (of at least ten percent) in a given year, 

did the negotiated rate fall in subsequent negotiations? 

When Sites Exceed Negotiated Goals 

Exhibit II-1 displays the results of our analysis, which includes two comparisons: whether 

exceeding performance goals by at least 10% in PYs 2000 or 2001 is associated with a 

subsequent negotiated goal for PY 2003 that is at least 10% greater; and whether similar 

excesses in PYs 2002 and 2003 are associated with an increased goal in PY 2005. As can be 

seen in this table, in PYs 2000 and 2001, states exceeded their goals by at least 10% 360 

times across all measures.9

 The seven states we visited exceeded their goals in these years 

52 times across all measures (or, about the same number of times, on average, as the overall 

population of states). In the 360 instances in which all states met this criterion, these states’ 

negotiated goals for PY 2003 increased by at least 10% a total of 150 times. In other words, 

in 41.7% of the cases when a state exceeded their PY 2000 or PY 2001 goal by at least ten 

percent, their subsequent negotiations led to PY 2003 goals that were at least ten percent 

higher. Similarly, this occurred 23 times in the states in our sample (44.2% of the time). 

In contrast, the second set of columns in Exhibit II-1 displays comparable performance data 

for PYs 2002 and 2003, with negotiated goals for PY 2005. As seen in the exhibit, states 

exceeded their targets by at least ten percent a total of 306 times across all measures during 

 

8

 Note that the time lag in calculating actual performance means the data will not be available at the time the next 

year’s goals are negotiated. Thus, performance in one year cannot be meaningfully compared with goals in the 

next year; instead negotiated goals must be compared with earlier years’ actual performance. Additionally, 

DOL negotiated the first three years of goals in one round of negotiations, the subsequent two years (PYs 2003 

and 2004) in a second round of negotiations, and goals for PYs 2005 and 2006 in a third round of negotiations 

with states. Thus, data for PYs 00 and 01 were used as the comparison for PY 03 negotiated rates (PY 02 

actual performance data were unavailable by the time of these negotiations), and data from PYs 02 and 03 were 

used as the comparison for negotiated goals for PY 2005. In addition, we selected 10 percent as the threshold 

for comparing performance to subsequent negotiated goals primarily because it is large enough to winnow the 

number of sites achieving it, but small enough to be attainable—i.e. so that we were likely to see some results. 

9

 This number reflects the number of times a given state exceeded a particular goal in either PY 2000 OR PY 2001. 

In other words, if a given state exceeded their adult entered employment measure by at least 10% in both 2000 

and 2001, they are counted only one time for this analysis. 
II-8

PYs 2002 and 2003, fairly similar to the total for PYs 2000 and 2001. The states we visited 

exceeded their goals by at least ten percent 65 times across all measures during this period. In 

fairly stark contrast to the earlier negotiations, however, the negotiated rates for states that 

exceeded their goals by at least ten percent (in PYs 02 and 03) were much more likely to be 

established at least ten percent higher for PY 2005. As can be seen in Exhibit II-1, of the 306 

instances of states exceeding their performance goals by at least ten percent, in 236 cases their 

PY 2005 goals were set at least 10% higher. Thus, the negotiated rates were set at least ten 

percent higher in PY 2005 in 77.1% of the cases, a sizable increase over the earlier period 

described above. Similarly, this occurred 46 times for the states in our sample (or 70.8% of 

the time). 

This tells us that in negotiating rates for PY 03, DOL responded to exceptional state performance 

in earlier years with exceptional demands less than half the time. Yet, the results from 

negotiations for PY 05 goals were quite different. For PY 05, DOL was much more likely to 

have negotiated much higher standards for states that exceeded their performance by a “large 

amount” (i.e. at least 10%). This pattern is carried out for nearly all the individual measures, as 

well, and across each of the four the different WIA programs in our analysis. 

We examined the extent to which this pattern was mimicked in the state-local negotiations. 

For this analysis, we only included those states and local areas that we visited during our final 

round of data collection, because the data for all local areas nationally were unavailable. The 

local areas in our sample exceeded their negotiated goals by at least ten percent 60 times 

during PYs 00 and 01, as shown in Exhibit II-2. In these 60 cases, states subsequently 

negotiated a PY 03 goal at least ten percent higher than their PY 01 goal 23 times (or, 38.3% 

of the time). In PYs 02 and 03, local areas exceeded their negotiated goals by at least ten 

percent 81 times. In response, states negotiated a ten percent or higher raise in their PY 05 

targets 57 times, or 70.3 percent of the time. Thus, the pattern established during federal to 

state negotiations is quite closely matched in the state to local negotiations. Local staff 

resignation about the negotiation process and actual goals may be reflected in this finding: 

local areas understand that if DOL is negotiating consistently higher goals with the states, then 

local areas themselves can expect no less. However, these higher goals mean the bar is set 

especially high for PY 05; on top of common measures implementation and any changes to 

system and program design that entails, LWIAs may need to attend more diligently to the 

incremental attainment of goals in order to meet or exceed the measures at the end of the 

program year. 
II-9

Exhibit II-1:

Incidence of States Surpassing a Negotiated Performance Level by 10% Resulting in a 

10% Increase in Performance Levels the Following Negotiation 

 

 PYs 00 and 01 PYs 02 and 03 

 

# of Times 

States 

Exceeded by 

at Least 10% 

Increase of at 

Least 10% in 

PY 2003 Goal 

# of Times 

States 

Exceeded by at 

Least 10% 

Increase of at 

Least 10% in 

PY 2005 

Goal 

All Measures 360 150 306 236 

Adult 

Entered Employment 30 8 21 13 

Retention 13 5 11 3 

Earnings Change 20 6 21 11 

Credential 29 15 34 29 

Dislocated Workers 

Entered Employment 28 8 20 14 

Retention 5 3 2 2 

Earnings Change 20 3 N/A N/A10

Credential 36 19 35 27 

Older Youth 

Entered Employment 26 9 22 11 

Retention 13 5 16 9 

Earnings Change 22 10 33 23 

Credential 19 10 31 26 

Younger Youth 

Skill Attainment 40 23 29 22 

Diploma Attainment 29 10 36 27 

Retention 30 16 30 19 

 

10 We do not include dislocated worker earnings replacement, because that measure changed in PY 05 to 

dislocated worker earnings change, or, from a percentage to a dollar figure. Thus, the negotiated goals from 

previous years are not comparable. 
II-10

Exhibit II-2: 

Incidence of Local Areas Surpassing a Negotiated Performance Level by 10% Resulting 

in a 10% Increase in Negotiated Performance Levels the Following Year* 

 

 PYs 00 and 01 PYs 02 and 03 

 

# of Times 

Local Areas 

Exceeded by 

at Least 10% 

Increase of at 

Least 10% in 

PY 2003 Goal 

# of Times 

Local Areas 

Exceeded by at 

Least 10% 

Increase of at 

Least 10% in 

PY 2005 

Goal 

All Measures 60 23 81 57 

Adult 

Entered Employment 5 2 6 4 

Retention 4 1 6 4 

Earnings Change 5 2 4 3 

Credential 4 2 6 4 

Dislocated Workers 

Entered Employment 6 2 5 3 

Retention 2 0 3 2 

Earnings Change 3 1 N/A N/A11

Credential 4 2 6 4 

Older Youth 

Entered Employment 7 3 8 5 

Retention 2 1 4 3 

Earnings Change 3 1 5 3 

Credential 2 0 4 2 

Younger Youth 

Skill Attainment 8 4 6 4 

Diploma Attainment 5 2 8 5 

Retention 0 0 10 6 

When Sites Fail to Meet Negotiated Goals 

As demonstrated in Exhibit II-3, the negotiations in response to poor performance by a state 

appear to be less responsive, although in large part this analysis must be considered inconclusive 

because of the relatively few number of states and local areas that fail a given measure by at least 

ten percent, especially during PYs 2002 and 2003. Although states performed at least ten 

percent worse than their goal in 126 instances in PYs 2000 and 2001, their PY 03 negotiated 

rates went down by at least ten percent in only 41 cases (32.5%), across all the performance 

 

11 See footnote 10 for an explanation of why this measure was not calculated for PY 2005.. 
II-11

measures. In contrast, states performed at least ten percent below their goal in only 38 instances 

in PYs 2002 and 2003. In response, their negotiated rates were at least ten percent lower for PY 

05 in only five cases (13.2%). 

Local area perception of state responsiveness on this issue is mixed, but our review of local area 

failures to meet 90 percent of the negotiated goal in one year and a lowered goal in the following 

year suggests that states are somewhat responsive to local areas that miss measures. But this 

flexibility decreased between PY 03 negotiated goals and PY 05 negotiations. As shown in 

Exhibit II-4, the percentage of times in which states lowered local areas’ goals (by at least ten 

percent) in response to poor performance in prior years dropped from 53.8% in PY 03 to 33.3% 

in PY 05. 

Changes to Negotiations Resulting from the Common Measures 

By the time of our visits, states had already held one round of negotiations to establish new 

performance targets for at least the adult measures under the newly implemented common 

measures. Thus, states were able to identify how the transition to the common measures has 

impacted their negotiations with the federal government for these measures, though it was still 

unclear whether this would differ for negotiations for the new youth measures. Generally 

speaking, states felt that the negotiation process for the adult measures, and the targets that they 

set as a result of it, changed relatively little as compared to their earlier negotiations. For most 

states, the common measures for adults are highly similar to the core measures documenting 

employment, retention, and earnings. Thus, the negotiations were over similar concepts and 

were little changed from earlier rounds of negotiation. There was one significant change to the 

negotiation of outcomes in that, in response to dislocated workers now being held to the common 

measure earnings change rather than the previous earnings replacement, most states negotiated a 

negative goal. This was a reflection of the fact that the former ratio (of less than one) of earnings 

replacement implies that dislocated workers were expected to earn less following their 

participation. Given that the new measure tracks earnings change, rather than replacement, the 

negative values reflect the fact that DOL was being responsive to the likelihood that these 

workers would, indeed, earn less following participation.12

 

12 As of PY 2006, this earnings measure will shift to be simply the average earnings in the 2nd and 3rd quarters 

following exit. Thus, the “negative” earnings gains goals will be replaced by a positive value of targeted average 

earnings. 
II-12

Exhibit II-3: 

Incidence of state failure to meet 90% of negotiated goal resulting in a decrease in the 

negotiated rate the following year 

 

 PYs 00 and 01 PYs 02 and 03 

 

# of Times 

States 

Missed by at 

Least 10% 

Decrease of at 

Least 10% in 

PY 2003 Goal 

# of Times 

States Missed 

by at Least 10% 

Decrease of 

at Least 10% 

in PY 2005 

Goal 

All Measures 126 41 38 5 

Adult 

Entered Employment 30 8 21 13 

Retention 13 5 11 3 

Earnings Change 20 6 21 11 

Credential 29 15 34 29 

Dislocated Workers 

Entered Employment 28 8 20 14 

Retention 5 3 2 2 

Earnings Change 20 3 N/A N/A13

Credential 36 19 35 27 

Older Youth 

Entered Employment 26 9 22 11 

Retention 13 5 16 9 

Earnings Change 22 10 33 23 

Credential 19 10 31 26 

Younger Youth 

Skill Attainment 40 23 29 22 

Diploma Attainment 29 10 36 27 

Retention 30 16 30 19 

 

13 We do not include dislocated worker earnings replacement, because that measure changed in PY 05 to 

dislocated worker earnings change, or, from a percentage to a dollar figure. Thus, the negotiated goals from 

previous years are not comparable. 
II-13

Exhibit II-4: 

Incidence of local area failure to meet 90% of negotiated goal resulting in a decrease in 

the negotiated rate the following year 

 

 PYs 00 and 01 PYs 02 and 03 

 

# of Times 

Local Areas 

Missed by at 

Least 10% 

Decrease of at 

Least 10% in 

PY 2003 Goal 

# of Times 

Local Areas 

Missed by at 

Least 10% 

Decrease of 

at Least 10% 

in PY 2005 

Goal 

All Measures 26 14 12 4 

Adult 

Entered Employment 0 0 0 0 

Retention 0 0 0 0 

Earnings Change 6 2 2 0 

Credential 3 2 2 2 

Dislocated Workers 

Entered Employment 0 0 0 0 

Retention 0 0 0 0 

Earnings Change 1 0 N/A N/A14

Credential 4 3 2 1 

Older Youth 

Entered Employment 2 1 0 0 

Retention 0 0 0 0 

Earnings Change 5 3 4 0 

Credential 3 2 2 1 

Younger Youth 

Skill Attainment 0 0 0 0 

Diploma Attainment 1 1 0 0 

Retention 1 0 0 0 

One fairly minimal concern was raised by only one state. Because states and local areas are now 

encouraged to track outcomes for all those who receive services, whether or not they register in 

WIA, this state expressed concern that these individuals would soon become subject to the 

common measures. Specifically, representatives within this state raised the concern that, with 

the point of registration (or, point of participation, as it is now known) growing closer to 

applying to a more universal customer, states and local areas may struggle to meet performance 

standards for a larger group that has received less scrutiny to determine their suitability for WIA 

 

14 See footnote 10 for an explanation of why this measure was not calculated for PY 2005. 
II-14

services. This concern is speculative, however, as there has not been any requirement 

established that these individuals actually be included in the measures, just the encouragement 

that they be tracked.15 

Documenting Performance 

Data tracking and reporting is a cornerstone of the WIA accountability system. States are 

required to submit quarterly and annual reports to DOL with calculations of the seventeen 

required measures.16 In addition, states must provide data about two of the three new Youth 

common measures and information about total numbers of customers served and any 

characteristics collected. 

MIS Architecture 

In this round of visits, six of the seven states have a statewide system into which most local areas 

enter data (only Oregon does not have a state system). Two of the states have adopted variations 

of the One-Stop Operating System (OSOS)17 and the other four are a mix of state-designed or 

private-contractor-designed systems. Some of the state systems are highly integrated. For 

example, Colorado’s system is designed to house data from WIA, ES, TAA, Veterans programs, 

and TANF welfare-to-work. Staff from any of these partner programs can see whether a 

customer is receiving services from any other program. Similarly, in Maine, WIA, ES, and TAA 

share a system. 

As noted above, Oregon does not have a statewide system. Instead, local areas each have their 

own systems, and submit reports separately to the state. Due to a recent law requiring a data 

warehouse for the K-12 and post-secondary education systems, Oregon may eventually provide 

an interface to link the various workforce investment programs’ data with these educational data 

systems. In other states, WIA, ES, and other partners each maintain their own stand-alone 

systems. Two states, Maine and New Jersey, also make available to local areas an interface 

within this system that is specifically designed to enable them to produce reports and manage 

performance. The Alabama WIA system is not linked to the ES database, but the state plans to 

link the two systems in PY 2006. 

 

15 Although the concern is only speculative at this point, language in bills reauthorizing WIA would make this 

concern more concrete, as states and local areas would be forced to calculate performance on those who receive 

virtually any service. 

16 As of PY 05, states must also submit information for adults and dislocated workers on the common measures. 

However, for these programs, the three common measures simply replace the conceptually similar core measures 

that had been in effect until PY 05. 

17 This system was designed by a consortium of states using funding provided by ETA. 
II-15

State and Local Satisfaction with the MIS and the Availability of Outcome Data 

In the interim report, concerns about data systems were some of the most significant issues in the 

performance measurement system; however, in this round of site visits states express a greater 

degree of confidence in their MIS. Four of our seven states reported satisfaction with their 

systems. Similarly, at least four local areas explicitly reported satisfaction and confidence in 

their state (or their local) systems. In fact, only a few states and local areas expressed serious 

concerns or frustrations with their MIS in this round. 

In the interim report, we found that local and provider access to state MIS was an important 

predictor of local satisfaction with the system (i.e. the more access, the better the approval 

rating). The ability to extract reports was also important. Access overall is quite high in the sites 

we visited this round. In fact, as we discuss in a later sub-section of this chapter, states and local 

areas in our sample use their MIS in a variety of ways to manage performance. 

All seven states can access state UI wage records without trouble, and local areas generally have 

access to UI wage records as well. By contrast, in our interim report we found that only three of 

the eight states we visited provided their local areas access to UI wage data. Only two of the 11 

local areas we visited this round feel that state UI wage data is inaccurate and thus, suspect. 

However, states and local areas continue to feel, as in the interim report, that the primary 

disadvantage to UI wage data is the time lag associated with collecting it, and the fact that some 

employment is not covered by the state UI systems, such as agriculture in some states, selfemployment, and the federal workforce. 

In addition, many respondents express concern that those to whom they provide services may 

obtain employment in a different state and, thus, not appear in their state’s UI records. In order 

to compensate for this perceived disadvantage, multiple states collect data from other systems. 

Five of seven states say they access data from the Wage Record Interchange System (WRIS) to 

identify customers who become employed out of state. One state was even giving its local areas 

access to WRIS data, until that practice came under fire due to other states’ privacy laws. Two 

states—Idaho and New Jersey—find WRIS data less useful because one or some of their border 

states do not participate in the WRIS. Several other states access additional employment records 

for customers from other systems. For example, Maine is piloting obtaining data from FEDES, 

the system that collects employment data on federal employees. New Jersey is thinking about 

joining a six-state consortium for data sharing. Finally, Colorado is extracting data from the 

United States Postal Service and the military to augment state UI wage data and WRIS data. 

In addition to accessing data from systems other than the state’s own UI wage records, states 

may also choose to collect supplemental data. Supplemental data is an acceptable form of 

documentation of employment for the WIA entered employment and retention measures, but not 

for the earnings change measures. In our interim report, we found that only half the states we 
II-16

visited used supplemental data. In this round of visits, six out of seven states accept 

supplemental data as an alternative to wage records as a demonstration of customer employment. 

Only New Jersey does not accept supplemental data as an alternative form of documentation. In 

general, local areas collect supplemental data through follow-up contact with customers. 

However, two states noted that they would request supplemental data specifically from local 

areas for individuals not appearing in the wage records. 

Changes to MIS/Reporting Due to the Common Measures 

States (and the few local areas with their own MIS) are making a number of changes to their MIS 

in response to the common measures, but most sites feel confident that their MIS will be able to 

capture the necessary changes. For example, both Colorado and Alabama had completed their 

proposed changes at the time of our visit. Two local areas and three states we visited had either 

just started or were still in the middle of making changes to their MIS. One state had submitted a 

request to its MIS provider for 80 changes to the system, but most other sites noted far fewer 

changes. 

At least two states have started producing reports showing performance on the common 

measures in order for both states and local areas to begin managing performance according to 

these new measures. New Jersey has started using a reporting system that interfaces with its case 

management system, FutureWorks, to provide local areas with common measures reports. Idaho 

has begun to include new screens in the MIS system so that the state can run reports using the 

new common measures rather than, or in addition to, the WIA measures. 

The sites we visited have some anxiety about the common measures’ impact on their data 

systems. Lack of knowledge of customer activity in partner programs is among the most 

frequently cited concerns. This concern arises primarily because, as the common measures are 

implemented, there is greater incentive for different workforce systems to collaborate in serving 

customers. Without greater attention to how to manage and track these individuals across 

programs, however, respondents worry that this incentive will lead to difficulty in managing and 

documenting outcomes for these customers. Three states mentioned this concern, some because 

ES and WIA do not share the same state system, and one because there is confusion about which 

program “gets credit” for customer outcomes.18 A related concern is that the “common exit” 

described in TEGL 28-04 will require a much greater degree of data sharing than most WIA 

 

18 This concern may be somewhat unfounded. According to TEGL 28-04, each program a customer is receiving 

services from takes credit for customer outcomes, though the specifics of this may depend on how costs are 

allocated or shared between the programs. 
II-17

partner programs currently conduct.19 In some states, partner programs use different identifiers 

for customers, preventing customer matching. DOL guidance to the Trade Adjustment 

Assistance (TAA) program has specified that customer identifiers be common between TAA and 

WIA, but other programs may not follow this trend. Other states mentioned the onset of the 

DOL’s EMILE common reporting system20 as being ideal in terms of data sharing across 

programs, but suggest that the cost of implementing the system will be insurmountable for many 

states. Others say they are more focused on data validation efforts, and choose to continue to 

focus on this rather than worry about common measures. 

Using Data to Monitor Performance 

States and local areas use their MIS to monitor performance in several ways: 1) by producing 

post-exit reports that primarily note the status of customers after they exit, 2) by producing inprogram reports that provide information about customers while they are still enrolled in the 

program, 3) by installing reminders or “ticklers” for staff to remind them of the points at which 

they need to follow up with customers, and 4) by producing reports by local area, One-Stop 

Career Center/provider, or case manager to assess the performance by different units of 

measurement, or levels within the state system. We will explore each method in turn. 

Post-exit Reports 

Post-exit reports are the most clearly tied to the WIA performance measures, because they are 

intended to identify customers who have exited and will soon be or already are subject to the 

performance measures. State and local MISs offer a variety of “canned” reports, and some 

systems allow ad hoc reports to be created by the user. Four local areas and four states described 

using specially designed post-exit reports to keep track of customer outcomes.21 First, there are 

basic “exit” reports such as the “number of exited participants,” “total quarterly exiters,” “reason 

for exit,” and “employment at exit” (as a proxy for employment in the first quarter after exit). In 

the states we visited, for example, reports on exited customers include “number currently 

employed and unemployed” or “placements” to assist staff in monitoring the employment and 

education measures. Second, some reports are measure-specific, such as Alabama and North 

 

19 This definition of exit suggests that an individual is exited when they have not received a service funded by the 

program or by a partner for 90 consecutive calendar days. This implies that an individual can remain in a 

program without receiving a service provided by that program so long as they do receive one from a partner 

program. This definition remains consistent in TEGL 17-05. 

20 EMILE has now been replaced in discussions with a system that is considerably reduced in scope and is now 

referred to as the WISPR (Workforce Investment Streamlined Performance Reporting) system, but we use 

EMILE since that was the reporting system under consideration at the time of our visits. 

21 These reports differ from the quarterly and annual reports that states submit to DOL, as they are designed for use 

by program managers to monitor the outcomes and customers on which those outcomes are based. 
II-18

Carolina’s reports on Younger Youth Diploma/Equivalent Attainment or Idaho’s report listing 

customers who have attained credentials. A third type of exit report looks for the absence of 

information as evidence of missed or about-to-be-missed outcomes. For example, Mobile uses a 

report entitled “No wages in quarters 1 and 3” and the Oregon Consortium looks at post-program 

wages (through customer self-report) as a proxy for the formal wage record calculation of the 

Earnings Change measures. One local area expressed concern because they recently discovered 

that the state system’s capacity to generate Earnings Change reports was gone. Finally, post-exit 

reports may also be used to prevent negative outcomes. For example, North Carolina sends local 

areas “troubleshooting” reports, such as a list of all in-school youth who exited before they 

obtained a diploma. 

In-program Reports 

In-program reports serve a different purpose: generally, their purpose is to monitor the progress 

or status of customers, often to ensure that customers who have not yet exited from WIA do not 

do so until they are ready to achieve positive outcomes on performance measures. One major 

type of in-program report helps to prevent “soft exits” (one of the two types of exit under the 17 

required measures, and the only type of exit under the new common measures). These reports 

include Mobile’s “youth with no services,” Cape Fear’s soft exit report, and the Oregon 

Consortium’s reminders to case managers to make contact with customers every 30 days to 

prevent 90 days going by without a service, which would trigger a soft exit. Colorado’s MIS can 

generate reports on current participants and new enrollments, service use, and participant 

characteristics. 

Reminders 

Reminders to case managers can also serve a variety of purposes, such as preventing soft exits, 

as noted above. Weld County uses reminders for staff to meet deadlines for customer 

registration and to conduct timely follow-up. Allowing customers to soft-exit before they are 

well situated to successfully obtain and retain well-paying employment increases the likelihood 

that program performance will suffer and that customers will not achieve their goals. Timely 

follow-up is critical for staff to determine customer status and, if necessary, provide assistance so 

that customers can achieve positive outcomes. 

Reports for Different Units or Levels 

Finally, several of the local areas we visited use the MIS to assess One-Stop Center and 

individual case manager performance. For example, Weld County, Mid-Willamette, and Cape 

Fear all practice this type of monitoring. Cape Fear routinely monitors caseloads, to make sure 

they are not too high for case managers to serve their customers effectively. The local area also 

monitors case notes to make sure case managers are keeping track of what services customers 
II-19

receive and what needs they have. Mid-Willamette monitors customer contact information, to 

make sure customers can be reached during the critical follow-up period. Both Mid-Willamette 

and Weld County look at customer outcomes by case manager, and use these reports as a way of 

assessing case manager performance and facilitating case manager improvement, and Weld 

County further produces monthly reports that examine the originally required measures by local 

area and by individual Career Centers. 

Mechanisms to Encourage Strong Performance 

The workforce investment accountability system is intended to encourage improved 

performance. Specifically, the act requires states to provide sanctions for poor performance and 

encourages them to provide incentives for good performance. In addition, language is built into 

WIA and later guidance that encourages states and local areas to develop systems of continuous 

improvement. In this section, we discuss state and local systems of continuous improvement, 

and incentives and sanctions. 

In our interim report, the states and local areas that we visited had not done much to develop 

incentives, sanctions, and continuous improvement policies and systems. Only half of the states 

we visited had incentive policies, and only two of eight states had sanction policies. Only three 

states had developed formal continuous improvement programs, and only one of those used the 

required WIA measures as the basis of quality measurement within their program. In general, it 

was agreed that the WIA measures were not appropriate for assessing continuous improvement 

due to time lag and the fact that the measures are outcome- rather than process-oriented. In this 

round of site visits, we found that more states had developed incentive and sanction policies, but 

that continuous improvement programs continue to be sporadic and informal rather than uniform 

and formal. 

Incentives 

In this round of visits, six of the seven states we visited have developed incentive policies.22 The 

one state that did not have a policy reported that they wrote a policy two years ago, but that it 

was never released or approved. 

The Survey Report found that the possibility of receiving incentives has influenced WIA 

program design in half the local areas in the nation. Oregon state staff supported the view that 

incentives make a difference. According to one staff person, “philosophically, incentives and 

 

22 Idaho is included here, although the state recently changed to a single area state, thus eliminating the need for a 

policy regarding distributing incentives to local areas. 
II-20

sanctions make a difference in a performance system.” But, as noted in the quantitative analysis 

of our Survey Report, incentive awards provided by states to local areas based on performance 

seem to increase service to a variety of customer groups that tend to be less job-ready and have 

poorer work histories. This result is confirmed in the Survey Report by similar results for states 

with a single local workforce area, which are subject to federal incentive policies. We found this 

as well in our examination of the provision of incentives to providers by local areas – in those 

areas in which incentives are offered to providers based on their performance, the providers often 

suggested that they serve a broader range of individuals than do those who are not eligible for 

incentives. Why this is so is unclear, but one provider suggested that by serving a pool of 

customers who have poorer work histories, they are better able to target a range of occupations 

and they need not focus only on very specific and potentially difficult to enter employment areas. 

This result suggests that fears that the WIA performance system might induce local areas to 

focus on services to better-prepared individuals are unfounded. 

Five of the states we visited have an incentive policy that is somewhat less intensive than the 

federal policy.23 For example, if Colorado wins incentives from DOL, it gives each local area a 

portion of that award, regardless of LWIA performance. However, in general (for example, in 

years when the state does not win federal incentives), the policy is the same as the federal policy. 

The other states’ policies match the requirements of the federal policy in order for local areas to 

qualify for an incentive. In five states, these incentives are competitive among those LWIAs that 

have met the requirements. Thus, in these states, the more local areas that qualify, the smaller 

the total amount any qualifying LWIA will receive. 

Only two states provide incentives that are proportional to the local area allocation. In doing so, 

these states target the majority of their incentive funds to those areas that have been designated 

as having the greatest “need”—areas with larger populations, and that have higher rates of 

unemployment and poverty. Viewed another way, a proportional incentive structure recognizes 

that areas with larger allocations are likely to need more resources to adopt system-wide 

initiatives or changes with incentive funds. Thus, in the states that do not provide incentives that 

are proportional to their allocation, in effect they are favoring smaller local areas or areas with 

less identified need by providing them incentives on an equal basis with larger areas. 

Only one state, Alabama, bases its incentives on the extent to which the local area exceeds the 

measure. In Alabama, each eligible LWIA whose actual performance exceeds its goals receives 

awards based on established percentages. These differ depending on the measure that has been 

 

23 The federal incentive policy stipulates that the state achieve an average of 100% for each funding stream, and 

not less than 80% on any individual measure. The state must also meet performance criteria for WIA Title II 

Adult Education and Literacy programs and Title I of the Carl D. Perkins Vocational and Technical Education 

Act. However, these other criteria are not used by states when awarding incentives to local workforce areas. 
II-21

exceeded. For example, for exceeding the earnings change measure by zero to 5 percent, local 

areas receive up to $500; for exceeding between 5 and 10 percent, LWIAs can receive from $500 

to $1,000; for exceeding by more than 10 percent, local areas can receive more than $1,000. In 

this way, the state rewards those areas more the better they perform, a motivation for LWIAs to 

strive for excellent, rather than just good, performance. 

In their specifics, the state policies reflect the thinking states have put into this aspect of the 

accountability system: a couple of states have created new iterations of policies to encourage 

outstanding performance. For example, until PY 04, New Jersey incentives were based only on 

performance; local areas exceeding or meeting 15 or more of their performance measures shared 

in the incentive pool. Areas received “shares” based on how many measures they exceeded or 

met—the more measures met at or above 15, the more shares. As of PY 04, however, local areas 

in New Jersey will be measured on the number of customers they serve and will have to pass 

through “gateways” to qualify for receiving incentive funds. The gateways include: 

1. To be eligible for an incentive in any area, an area must meet the two customer 

satisfaction measures, and exceed at least 13 of 15 remaining measures. 

2. To be eligible for an incentive in a specific program area, the local area must have 

exceeded all measures in that area. 

3. The local area must also have obligated 80% of its allocation plus carry-in dollars 

from the previous years. 

4. The local area’s cost per exiter must not exceed by a specified percentage the 

State average for exiters of that program area or $2,000 greater than the state 

average, whichever is less. (The state average is 125% Adult/Dislocated Worker; 

140% Older Youth/Younger Youth). The program area pot will be divided 

equally by the local areas qualifying for an award in that area. 

Four of the states we visited have won incentives from DOL, and 8 of the 11 local areas have 

won incentives from their state. Some of these areas are in states that did not receive incentive 

awards from DOL because states also use portions of their state set-aside to fund incentive pools. 

States choose to reserve different amounts of money for the incentive pool, either by setting 

aside a specific dollar figure or a by establishing a particular percentage of the state’s set aside 

funding to be used for incentives. For example, North Carolina made available one million 

dollars in incentives for PY 04, while Oregon and Idaho both reserve about ten percent of the 

state’s set aside for incentives. 

The amount of incentive funds available is one of the reasons for local area dissatisfaction with 

the state’s incentive system. One state and three local areas note that the amount of incentive 

awards (two of the local areas, who are from the same state, said a common incentive award is 

between $30,000 and $40,000) is too low to make a real difference in the program, but that the 

awards are seen as public recognition of good performance. 
II-22

Other reasons for local area dissatisfaction have to do with who receives the funds. In a couple 

of cases, local areas say the incentive funds the local area qualified for were not given to the 

local area leadership, but rather to the education community or to individual providers at the 

behest of the state, not by the decision of the local area. 

In addition, two states have never received incentive dollars from DOL because their Perkins Act 

and Adult Education programs have not met their performance targets. These states are 

frustrated with this arrangement, yet continue to provide incentives to their local areas. In one 

local area, it is not dissatisfaction that is expressed, but ignorance. The local area leadership was 

not aware whether the state or the local area had received any incentives. 

What are incentives used for? 

In addition to providing incentive funds directly to local areas that have met the requirements for 

these funds, states also use incentive funds for state initiatives. For example, Colorado is using a 

portion of its incentive money to establish a computerized training program for customers and an 

online resource called E-Colorado, a knowledge portal with information and links for job 

seekers, employers, and workforce development professionals. Local areas often use incentive 

funds to send staff to trainings, hire consultants, or purchase equipment to improve the quality of 

the One-Stop Centers. 

Sanctions 

As noted, in the interim report, we found that only two of eight states visited in the first round 

had formal sanction policies, and that those policies were less strict than the federal sanction 

policy. However, most states did have a method for dealing with local areas that failed 

measures. In this round of visits, we found something similar: whether or not states have what 

they identify as a “formal” sanction policy, all states have a plan for what to do when local areas 

are approaching failure, fail one or more measures once, or fail one or more measures more than 

once. 

Seven of the eleven local areas we visited had failed at least one measure once since the 

beginning of WIA, and would thus have reason to have received technical assistance. Only 

Alabama LWIA, Mobile, Cumberland, and the Oregon Consortium have never failed a measure. 

Coastal Counties is the only local area to fail the same measure two years in a row (though they 

received technical assistance, rather than a sanction, for this failure). The most commonly failed 

measure in our sample was Older Youth Earnings Change (5 of the 7 local areas that failed 

anything failed this measure at least once). 
II-23

New Jersey and Idaho follow the federal sanction policy (except for the provision that repeated 

failures may result in a loss of part of the state’s allocation): if a local area falls below 80 percent 

on any measure in year one, they will submit a performance improvement plan and the state will 

provide technical assistance. If a local area fails the same measure for a second year, the state 

will take corrective action that may include appointing a new board, or prohibit the area from 

using particular provider or One-Stop operator or partner. Alabama’s policy includes the 

specific stipulation that a local area may lose up to five percent of its allocation for failing the 

same measure two years in a row. Other states are unwilling to extend their sanction policies to 

include financial sanctions. Maine suggests that sanctions beyond corrective action plans shall 

not be imposed when good faith efforts are made to meet performance standards. Similarly, 

New Jersey does not want to take away money from areas like economically distressed Newark 

who need the funds and services the most, implicitly recognizing that factors such as the local 

economy or participant characteristics play a role in outcomes. 

These states and others also focus on preventive measures. In Idaho, if the state observes 

through its monitoring of performance reports that a local-service unit or individual One-Stop 

center is approaching failure on a measure, the state telephones the area or center and offers 

technical assistance, either via phone or in person. In addition, the state encourages local areas 

to send their new case managers to the state trainings in WIA, MIS, reporting, contracts, and 

customer selection procedures. In Maine, if monitoring findings suggest a local area, One-Stop 

center, or provider is failing one or more measures this issue will be raised in managers’ 

meetings, which provide a regular venue for the state and local areas to talk about performance. 

New Jersey also has monitors that track performance throughout the year so that local areas can 

receive technical assistance before they fail a measure. In North Carolina, the state performance 

staff person and MIS staff pay close attention to each local area’s performance at mid-year so 

that they can warn the designated MIS or performance point persons at the local level, try to 

figure out what is going wrong, and implement changes to improve or fix errors. The MIS 

person will print reports to help local areas track possible problems; for example, he will print a 

report showing all in-school youth who exited before they got their diploma. 

At least one state also uses its monitors to assess local areas’ mid-year progress toward attaining 

negotiated standards. In North Carolina, if the MIS staff person identifies that a local area is 

struggling to meet its performance goals, monitoring team members might look more closely at 

the small section in the monitoring document that deals with performance. The monitoring visit 

generally results in a memo sent to the chief elected official/grant recipient that gives general 

information about the visit and a more detailed “entrance and exit notes” to the local area 

directors. Detailed information is given to that official only if an area has failed to comply with 

some aspect of the system two or three years in a row. 
II-24

Considerations about Sanction Policies 

Like the considerations states are putting into their incentive policies, some states are thinking 

critically about how to use their sanction policies to encourage improved performance. Oregon 

is one state in our sample that has recently discussed its sanction policy. Rather than 

“rewarding” poorly performing LWIAs with additional funds through technical assistance, the 

state is considering aligning its policy more closely to the federal policy of taking away a portion 

of a local area’s allocation if it fails the same measure two years in a row. The state is also 

considering the inequity that areas meeting, but not exceeding their measures, are not being 

rewarded at all, while those that fail are being “rewarded” with technical assistance funds. 

Similarly, a criticism some state and LWIA officials have made of the policy is that although 

both high and poor performers receive funding, those areas that neither fail nor exceed, but pass 

by performing above 80 percent of their target get nothing. These critics say that it seems unfair 

that these middle performers get nothing. However, other state staff argue that the current policy 

is correct because it takes money to implement PIPs and make improvements. State staff said 

that they are likely to adjust the sanctions policy somewhat based on these considerations. In 

general, Oregon staff believes that sanctions are effective because they get LWIBs’ attention. 

Typically, just being informed that an area is subject to sanctions is enough to spur the area into 

making changes. 

Continuous Improvement 

In the interim report, our respondents found the concept of continuous improvement appealing, 

but had made little progress in implementing formal systems of continuous improvement. In this 

round, we found that five states use the language of continuous improvement to describe their 

efforts to manage and improve performance. In this section, we will explore states’ efforts to 

increase their performance attainments through a variety of techniques. 

Some states manage their continuous improvement efforts through their MIS. For example, both 

state and local administrators in Colorado routinely print reports to track local area progress on 

performance measures throughout the program year. Idaho uses its MIS to monitor local area 

progress on interim indicators that it terms a key component of its continuous improvement 

effort. 

In some states, “continuous improvement” means not only achieving higher numbers on their 

performance levels, but also increasing the number of registrations and exits, serving different 

populations (such as at-risk youth and TANF recipients) and improving services. New Jersey 

considers all these efforts part of its continuous improvement effort, and also incorporates cost 

efficiency to their efforts. The data systems and the reports it generates are used for continuous 

improvement (e.g., reports from the FutureWorks MIS system are shared with local area 

managers monthly). 
II-25

Two states use particularly formal continuous improvement systems. For example, Maine uses a 

system called “Bend the Curve” that focuses on cost-savings and six basic goals that are based 

on the Lean Manufacturing management technique. Each of the goals in Bend the Curve is 

aimed at questioning why specific tasks are accomplished, whether there are alternative ways of 

accomplishing these tasks or even if they need to be accomplished at all, and developing 

thoughtful new approaches to achieving program objectives. At regular Bend the Curve 

meetings, updates are provided on progress toward meeting goals and efforts to streamline the 

service delivery process. Additionally, Maine is currently conducting “value stream mapping” to 

look at which steps in each programs’ process add value, whether to the customer, or at another 

step in the service delivery process. Ultimately, the state’s purpose is to save $9 million by 

reducing lead-time, staff, services, and so forth. North Carolina has instituted a career center 

“chartering” process based on Baldrige criteria; the system has been evolving rapidly in recent 

months. Originally in the hands of the local boards, two years ago the state took over the process 

with the intention to make it more meaningful. Currently, the state feels the application is too 

lengthy (it is 38 pages), and revisions are ongoing. The current categories are as follows: to 

charter a “comprehensive” JobLink Career Center, there must be full-time ES and WIA staff; for 

a “non-comprehensive center” there must be 16 hours per week of concurrent ES and WIA staff. 

The state has not instituted explicit performance goals as part of certification, but they do look at 

the number of job-seeking and employer customers served, and at the numbers and make-up of 

staff. The state says the chartering system is successfully evaluating process, but not quality. 

This is part of the impetus for the revision effort. In addition, North Carolina hopes to use the 

data in the state WIB’s new system, described above, as a continuous improvement tool—for 

example, to explore what combination of services seem to work best for different cohorts of 

customers. 

Even when states do maintain what they term continuous improvement efforts, these efforts do 

not always impact the local areas in the state. For example, Maine’s “Bend the Curve” system is 

relevant at the state level, but has not yet become important to the local areas, although the state 

is considering applying the concepts to the Career Centers. 

One local area, Alabama LWIA, practices continuous improvement using a Management-byObjective approach. Like some of the state approaches, Alabama LWIA uses their MIS to 

facilitate their continuous improvement process. On a monthly basis, the MIS tracks all 

participants with their expected services, benchmarks, unit cost, and outcomes achieved. 

Similarly, in response to missing their target on the adult earnings measure, Mid-Willamette 

began a comprehensive effort to incorporate continuous improvement in their program focus on 

retention, which they felt also would coincide with earnings. Mid-Willamette’s retention effort, 

laid out in its performance improvement plan, involved the creation of a retention workgroup 

that met for a year to develop ways the area could improve retention. It also involved bringing in 
II-26

national experts on retention and performance. In Oregon, LWIA liaisons follow-up on 

performance improvement plans rigorously and make sure that timeframes are adhered to and 

performance improves. 

Most of the remaining states and local areas in our sample suggest that they do focus on 

continuous improvement, but had difficulty delineating any formal mechanism by which they 

attempt to do so. Only two states explicitly explained they have no continuous improvement 

system. However, both these states did identify that they had undertaken major efforts to 

improve their performance each year. Thus, continuous improvement continues to be a 

somewhat opaque concept, as few states or local areas have implemented a highly formal 

process, yet most express a strong desire to improve their performance and effort each year. 

Effect of Common Measures on Continuous Improvement/Performance 

Management Efforts 

In general, the advent of the common measures does not appear to have changed states’ approach 

to continuous improvement. According to Colorado, the common measures are simply a 

continuation of the culture of performance management that has always guided Colorado’s 

workforce system. Some states recognize that the common measures will bring significant 

changes to WIA program design, but suggest their approach to continuous improvement and 

performance management in general will remain the same. In fact, North Carolina state WIB 

staff hopes that the common measures will make continuous improvement easier than it has been 

under the WIA measures. Their reasoning is that the commonality of performance across 

programs will better enable the various programs to work together to promote strong 

performance and improvement. Given that many of the co-located partners will now have a 

common objective for assessing their performance, their aims will be better aligned and, thus, the 

strengths of each can be brought to bear in attempts to improve performance. At the time of our 

visit, of course, this was simply speculation, but respondents were fairly consistent in this hope 

and belief. 

Summary 

States and local areas we visited for the final report are on the whole savvier about using the 

many elements of the performance system to promote and achieve good performance. Sites use 

these elements—the negotiation process, MIS, incentives, sanctions, systems of continuous 

improvement, and performance measures themselves—to meet or exceed the WIA required 

measures and the common measures, and to design and implement accountability goals of their 

own. 

First and foremost, states must meet their targets on the DOL-required measures (and, generally, 

LWIAs must meet the targets negotiated with their states). If they do not exceed these measures, 
II-27

they will not receive incentive funds, which can be boon for funding state initiatives like 

Colorado’s E-Colorado knowledge portal, or for supporting staff training. If they fail, they face 

corrective action, or worse, a reduction in their federal allocation. While the negotiation of 

required measures continues to be fraught with challenges, by and large, states have been able to 

meet measures; often—especially in PY 05 rates—DOL has responded by negotiating 

substantially higher goals when states perform exceptionally, although they appear to have been 

somewhat less likely to negotiate lower goals when states perform below their goals. 

Second, states and local areas may establish additional goals for their WIA programs, workforce 

development systems, or other partner systems. They may also choose to develop goals for 

individual Career Centers, providers, or contractors. All the sites in our sample have some 

additional goals they seek to accomplish, although additional measures––while still used 

informally by several states and local areas–were far less common as formal benchmarks in this 

round than in the first round of visits. The additional goals this round include: 

5. Cross-system goals that include workforce development, health and human 

services, K-12, post-secondary education in Oregon and North Carolina 

6. Proxy/process measures that help sites predict progress toward attainment of 

required WIA measures in the Alabama LWIA and Idaho LWIA 

7. One-Stop measures in Coastal Counties 

8. Multiple instances of tracking participant costs or provider expenditures to ensure 

efficient delivery of services to as many customers as possible 

Perhaps more important than explicit goals are the reports the sites use to monitor performance. 

In contrast to the last round of visits, the sites in this sample are making thorough use of their 

MIS for these purposes, perhaps because even sites that bought or developed new MIS after WIA 

passed have now had their systems in place long enough to learn and take advantage of the 

reporting and management capacity. 

Sites in our sample commonly make use of post-exit and in-program reports to track participant 

progress toward intermediate and outcome goals. For example, numerous sites monitor youth 

goal attainment, training completion, credential attainment, employment at exit and in ensuing 

quarters, as well as wages at placement and in the following months. Sites do this through 

follow-up calls and customer self-report, and also by using UI wage records and supplemental 

sources of data. Several sites also use some type of soft exit report to prevent customers from 

exiting the program before they have achieved their goals and are ready to attain positive 

outcomes. In addition, three LWIAs in our sample monitor case manager performance and other 

aspects of the case management system, such as case notes and caseloads. Thus, compared to 

the sites in our first round of visits, the trend is toward using all the tools of the accountability 

system to encourage or even guarantee excellent outcomes. 
II-28

Sanctions and incentives are commonly used to some degree, though how they are applied varies 

significantly. For example, incentives are used both to provide additional funds–either 

competitively, proportional to allocation, or based on the extent to which the area exceeded the 

goal– for areas that exceed their performance targets and to fund other state initiatives. 

Sanctions, on the other hand, tend to be focused on preventive measures so that states provide 

technical assistance to those areas struggling to meet their targets, though they do maintain that 

local areas can lose a portion of their subsequent allocation with repeated failures. Additionally, 

given our survey report findings that the possibility of receiving sanctions and incentives has 

influenced WIA program design in half or more of the local areas in the nation, these tools can 

be viewed as important tools that shape WIA programs to ensure successful customer outcomes. 

Finally, while only two states have implemented formal continuous improvement systems, most 

use their MIS to monitor progress toward achieving their performance goals, including both the 

performance measures as well as the numbers and types of customers served. Thus, the concept 

of continuous improvement remains somewhat vague for most states and local areas in our 

sample, but they do emphasize the need to consistently produce strong performance and attempt 

to improve their performance over time. 
III-1

III. COMMUNICATING SYSTEM ELEMENTS TO THOSE 

WHO OPERATE THE SYSTEM 

The previous chapter described the building blocks of the WIA performance system. In this 

chapter, we detail how that design is conveyed to the staff and service providers who deliver 

services within the framework of an accountability system. We start with linkages to the OneStop system. This is particularly salient now that two workforce programs—WIA Title I and 

Wagner-Peyser—began using the common measures in PY 2005. The chapter then moves on to 

discuss, in separate sections, how the accountability system is conveyed to the staff and service 

providers who deliver adult and dislocated worker and youth services. The latter section focuses 

on contractual relationships with service providers. The chapter concludes by comparing the 

types of measures used in One-Stop and WIA operations and summarizes the key findings and 

trends from the second round. 

WIA Accountability and One-Stop Operations 

Effects of WIA Accountability on One-Stop Design 

One-Stop design reflects the influence of the WIA accountability system in that most of the local 

area sites carefully insulate core services from the effects of the accountability system. Four of 

the sites rely entirely on Wagner-Peyser staff to provide core services. While the other seven 

sites use WIA staff to deliver at least some core services, these services are configured to ensure 

that WIA staff provide limited levels of staff assistance so that they do not trigger WIA 

registration. This design is so widely and readily accepted that most respondents in our sample 

feel that the WIA accountability system in general does not have any significant effect on the 

One-Stop design. Rather, most contend that the division of labor recognizes that joint services 

are better for the customer because each partner can emphasize the functional roles that it does 

best. This pattern is very similar to the findings from the sites in the interim report, the results of 
III-2

our survey of all LWIAs, and our experience in visiting One-Stop centers in other research 

projects. 1

Effects of Common Measures on One-Stop Operations 

While the intent of the common measures is to provide the same performance yardsticks for all 

federally funded workforce programs, the adoption of the common measures has thus far been 

limited to DOL programs. Wagner-Peyser programs adopted common measures for PY 05 at the 

same time as WIA. Because Wagner-Peyser provides core services either by itself or together 

with WIA Title I in all sites visited, its adoption of common measures has significant potential to 

affect One-Stop operation and the WIA program. No other One-Stop partners had adopted 

common measures by the time of our visits. Thus, this section focuses on Wagner-Peyser and 

has only a limited discussion about the other programs. 

Adoption of Common Measures by Wagner-Peyser 

State Level. Each of the state Wagner-Peyser programs is well on its way to adopting the 

common measures, but no state indicated that they were making any significant change in 

the way they operated the program in the field. Thus, none expected any change in the way 

core services would be delivered or the relationship to WIA. This pattern holds, regardless 

of whether core services are delivered by ES alone or by ES in partnership with WIA. 

Only ES respondents in one state believe that common measures will have a profound effect on 

Wagner-Peyser operations, but even this state was not yet planning any operational changes. 

The ES respondents generally noted that PY 05 would be a dry run with no standards that they 

would have to meet. One state was deliberately withholding detailed information on the new 

measures to reduce any possibility that the staff would actually change program operation. Two 

states–New Jersey and Maine–thought that they needed to improve the initial customer 

experience in the One-Stop centers in order to help with later follow-up contact with these 

customers to get employment and retention information to supplement the UI records for 

outcomes. Only Colorado expected to try to manage ES exits to improve performance. 

However, none of these states had yet developed any specific instructions to staff to implement 

these strategies. 

 

1

 The Survey Report did not find any significant relationships between the organization of core services and WIA 

outcomes, except in that in those areas in which WIA provided core services there were reduced average costs 

per customer. This is not surprising, given that areas in which WIA does not provide core services must instead 

provide the more expensive intensive and training services, thereby increasing overall average cost. 
III-3

Another factor that minimizes the impact of common measures is the long-term downward 

trend in real dollars in the Wagner-Peyser appropriation. One state had just gone through a 

substantial reduction in staff and closing of several local offices that had served as One-Stop 

centers. Its leaders felt that common measures, regardless of their ultimate impact, are 

insignificant compared to the long-term, downward pressure from the reduced real 

appropriations. 

All of the states indicated that the new measures required substantial changes in their MIS, 

although at least two of the states, Alabama and Oregon, were making a major change in their 

entire labor exchange system anyway so the necessary programming requirements were 

marginal. 

Local Level. At the local level, most local ES staff in the sites visited are familiar with the 

common measures, but they too generally are not greatly concerned about the effect on their 

program. No local site reported that it made any significant change in the way it operated. Only 

two local ES managers are attempting to fine-tune their core services in response to common 

measures. The ES manager and staff at the Mobile One-Stop had decided to streamline the 

initial customer assessment to determine more quickly whether a new customer was interested 

and could benefit from WIA training. They planned to refer customers like high-wage dislocated 

workers immediately to WIA rather than work with them for a while to test the labor market. 

The ES manager felt that this procedure would improve service to customers and result in better 

outcomes for both programs. 

Adoption of Common Measures by Other Partners 

Vocational Rehabilitation. State officials were generally aware of the common measures but 

had received no instructions on any future implementation. However, several noted that if 

implementation would not sub-divide clients by types of disabilities or otherwise control for the 

type and severity of disability, the focus on overall employment and earnings outcomes would 

likely lead to greater program emphasis on less disabled persons. For example, New Jersey 

officials felt that the earnings gain measure would stifle service to well-paid professionals who 

generally will not achieve a positive earnings gain, though recent guidance issued by DOL that 

changes this to a measure of post-program earnings alleviates this concern. Two states were 

undergoing substantial reorganizations and thus were little concerned about their existing 

performance system, much less a future one. 

At the local level, most vocational rehabilitation staff had never heard of common measures. 

Several noted, however, that the existing WIA accountability system was not a significant 

obstacle, so that they did not expect that the adoption of the common measures by WIA would 
III-4

affect the partnership or service to customers.2 For example, vocational rehabilitation staff in 

Montgomery noted that integration preceded establishment of the One-Stop system, and actually 

dates back to CETA. Although co-enrollment is not common in the Montgomery One-Stop 

center, vocational rehabilitation staff use the center resources all the time. Further, this 

respondent and other local vocational rehabilitation staff noted that their own performance 

measures—how many disabled and severely disabled participants are placed in jobs, how many 

get degrees/certificates—are similar to WIA measures. 

TANF. Although several states that we visited required TANF to be a One-Stop partner, few 

states indicated that there was extensive co-location of the TANF program at One-Stop centers 

or co-enrollment of TANF customers in WIA intensive services or training. In Alabama, for 

example, TANF is a mandatory partner but is not co-located or functionally active. The ES 

program managers indicated that the state had a more active partnership when the One-Stop 

system was first created, but the partnership did not work out and TANF essentially has left 

the One-Stop system. Most of the sites visited have little or no direct involvement with 

serving TANF customers through WIA. Most WIA respondents noted that serving TANF 

customers posed substantial suitability questions (discussed in Chapter IV) under the old WIA 

measures, and that the common measures would not change this situation. In any event, 

respondents from both WIA and TANF in several sites felt that the common measures would 

be difficult to adapt to TANF, with its emphasis on participation rates. 

TANF has much more extensive interactions with the Wagner-Peyser program, and this 

involvement theoretically poses an increased challenge for Wagner-Peyser now that they are 

subject to the common measures. Nearly all the states visited require some or all TANF 

recipients (or applicants in some cases) to register for Wagner-Peyser. Nevertheless, despite 

the fact that all parties understand that the barriers to employment for many TANF recipients 

would hamper good outcomes, neither TANF nor Wagner-Peyser respondents expressed 

concern that their inclusion in Wagner-Peyser common measures outcomes would create any 

problems. This is very likely because neither organization expects any required levels to be 

unreasonable and because the day of reckoning is in the indefinite future. 

Adult basic education. We talked with only one local representative of adult basic education. 

This respondent indicated that the common measures were inappropriate for an education 

 

2

 Most LWIAs report that customers tend to be enrolled sequentially in WIA and vocational rehabilitation 

services. There is relatively little co-enrollment. Discussions with WIA staff generally suggest that a similar 

pattern holds for adult basic education. 
III-5

program, a position that has been adopted by the National Council of State Directors of Adult 

Education. 

Since none of these partners had actually adopted the common measures, there were no changes 

to partnership structure, service delivery, or the extent and intensity of referral or co-enrollment 

because of the implementation of the common measures. 

One-Stop Performance Measures 

We did not observe much change between the first- and second-round sites in the level of 

emphasis on One-Stop measures. About half of each sample is tracking the total volume of 

customers served—as seemingly will be required should the current legislation being proposed 

for WIA reauthorization be passed. However, the first-round sites were generally more 

interested in measuring internal processes though at least one site had an outcome wage measure 

(a minimum wage at placement that is effectively a proxy for the wage gain) for the universal 

customers. In the second round, interest in other One-Stop measurement is largely prospective, 

by which we mean that these are used only when considering providing a charter to operate a 

One-Stop center rather than apply them retrospectively as measures of performance. Two 

second-round states–Maine and North Carolina–consider One-Stop accomplishments in their 

chartering review criteria, but none of these criteria contain any outcome measures. Officials in 

several states did comment that some local boards are interested in developing such measures. 

Only New Jersey state staff explicitly advocated One-Stop measurement now that the common 

measures have started. The limited extent of One-Stop measurement is somewhat surprising in 

that the advent of common measures was expected to facilitate this type of measurement. This 

limited decline in interest in One-Stop measurement, to the extent that it is not coincidental, may 

be attributable to wider use of ES or integrated systems, several of which have evolved to 

capture the universal customers. 

Conveying Accountability in the WIA Adult and 

Dislocated Worker Program 

General Understanding of WIA Performance 

Front-line WIA adult and dislocated worker staff generally understand the WIA performance 

measurement system. In contrast to the findings of the interim report, where we found that quite 

a few front-line staff knew little about the accountability system, nearly all front-line respondents 

in this sample were aware of the overall importance of obtaining good performance and most 

know a good deal about the accountability system and its role. Within this group, however, we 

still see a range of knowledge. Staff in at least six sites consider the performance implications at 
III-6

virtually every step in working with customers. These staff typically are aware of judging 

participant suitability, the relative value of longer-term and more intensive services, implications 

of exit timing, and the importance of follow-up from both service and data collection 

perspectives, even if most are less knowledgeable about the details of measurements and the 

methods by which they are calculated. 

For example, Weld County staff said that performance generally infuses all of their daily 

activities, but they do not know about the implications of enrolling a low-wage dislocated worker 

compared to one who may have had greater earnings before dislocation. Cape Fear staff talked 

more specifically about exiting when it was appropriate and how different kinds of customers 

would contribute to or detract from their wage gain measures. In most of these sites, the 

extensive knowledge allows managers to achieve performance goals without having to include 

management controls such as higher-level approvals for registration and exit. Mobile’s staff, 

while generally knowledgeable about suitability and service questions, needed to know a little 

less detail about the performance implications of any customer selection or service decision 

because that LWIA created a very rigorous process, including explicit managerial review of 

registration and exit decisions that embedded key performance principles. In several other sites, 

the staff are generally less informed about performance, believing that they should focus 

primarily on a good service process. These few sites are more consistent with those sites visited 

in our first round, in which there was a much greater emphasis on effectively serving customers 

no matter the performance implications. Finally, a few respondents also adopted the traditional 

fear that too great a focus on performance will exclude some of the neediest customers from 

WIA services.3

 

Understanding of Common Measures for Adults and Dislocated 

Workers 

In general, the level of understanding of the common measures follows that of the WIA 

accountability system in general. The staff in those areas who best understand general WIA 

accountability also know the most about the common measures. In only one of those areas with 

a solid understanding of WIA accountability was the staff not at all familiar with common 

measures. Staff in the other areas had already been trained on the common measures and were 

aware of potential implications for service activity, although some still have some questions that 

the state training did not address. 

 

3

 This point was also made by a few vocational rehabilitation respondents about the common measures. 
III-7

The local areas with little interest in, or understanding of, the common measures paid less 

attention to the new measures for a variety of reasons. The Idaho Panhandle area had only 

recently trained its new front-line staff, state ES employees, after taking over WIA operations. 

They are still learning the basics of WIA accountability and had not gotten to the common 

measures. The other areas, however, generally feel that the common measures were not going to 

affect their operations greatly. Mobile’s leaders, who feel that their area has a very well defined 

process covering registration, exit, and service delivery, felt the area would not have to change 

its procedures under the common measures, so learning about those measures is a lower priority 

in this area. It is likely that similar beliefs are present in the other areas [Montgomery, Region L, 

Salem] in which front-line staff are generally unaware that the WIA measures changed at the 

beginning of PY 05. 

Methods of Conveying Accountability: Training Adult/Dislocated 

Worker Staff on Performance 

Seven of the eleven LWIAs carry out the WIA adult/dislocated worker program with in-house 

staff. In the remaining four sites, the contract providers are all long-standing deliverers of 

WIA—and JTPA before that—services. Thus, the issue of communicating WIA performance 

requirements to front-line staff is essentially a matter of staff training. However, to the extent 

that most of these sites have relatively experienced staff, we found that frequent training is a 

lower priority for the second-round sites. In fact, many of the sites rely on informal 

collaboration among staff to ensure that the particulars of WIA accountability are observed by 

experienced staff and informal on-the-job-training is used whenever new staff come on board. 

Several sites appear to have alternatives to training. In Mobile, for instance, training is relatively 

less important because performance is embedded in the LWIA’s procedures and managerial 

controls at the critical registration and exit points. For Rocky Mount, the assistant to the LWIA 

executive director is known in the area and the state for her performance expertise. She 

regularly discusses the importance of performance with the front-line staff, who are quite 

cognizant of the performance implications of various aspects of customer service, but they leave 

the calculations entirely to the expert. 

Conveying Accountability in the WIA Youth Program 

The youth program differs structurally from its adult and dislocated worker counterpart in that 

most of the services are contracted out to a variety of school districts, community-based 

organizations, and even a few for-profit firms. In nearly all the second-round sites, the youth 

program is conducted outside the One-Stop system. As a consequence, conveying accountability 

for youth programs is a two-stage process. First, as with the adult and dislocated worker 
III-8

programs discussed above, in-house staff need to understand accountability. This was relatively 

simple in nearly all of our sites because they typically had only a single professional staff 

member who served as a coordinator. Thus, we discuss this topic briefly. The second stage, 

however, is more complex, involving procurement and contractor oversight. 

General Understanding of WIA Youth Performance 

Staff. As with the adult program staff, the LWIA youth staff—often a single individual in each 

LWIA—had a solid understanding of the accountability system within the context of the WIA 

youth program. The coordinator’s role is especially critical in the accountability system because 

that individual is typically responsible for determining or approving eligibility and program 

suitability before registration, exiting participants, and overseeing contractor operations. Here 

too, the general level of understanding is significantly higher than among the sample of LWIAs 

that we examined for the interim report. Many of the second-round coordinators had been in the 

WIA and JTPA youth programs for a long time, so it is hardly surprising that they had a strong 

understanding of accountability. The only area in which the youth coordinator’s understanding 

of the WIA accountability system was relatively weaker than the others is in the Idaho 

panhandle, the one area that had a major reorganization of service, and that coordinator was new 

on the job at the time of the visit. 

Contract Service Providers. Although specific levels of knowledge varied from place to place, 

service provider staff at our sites were also well versed in the WIA performance system. A few 

sites, however, noted differences among the types of staff. Administrators, who are responsible 

for summarizing and reporting their organization’s results tended to be more aware of the 

importance of performance than front-line staff, who often are sympathetic towards the neediest 

youth and less willing to make adverse suitability decisions. 

Many respondents consider that knowledge of the performance system is a function of 

experience. Nearly all of our youth program respondents, both staff and providers, noted that 

the contractors have been with the program for a long time, typically going back to JTPA 

days. One contract administrator felt that it takes about three years for a contractor to get 

familiar with how the measures are calculated and how their operations affect those 

measures. Mobile’s youth providers (for older youth) were instructed to use calculators to 

refigure their wage gain when considering whether to exit an individual, but it took several 

years of prodding during provider-training sessions before front-line staff actually began to 

make the calculations. Environmental factors also increased the awareness of performance 

among those who previously thought that an emphasis on performance might adversely affect 

program quality or the ability to serve the neediest. For example, reduced allocations to the 
III-9

Alabama LWIAs forced the acquisition process to emphasize outcomes even more, and the 

providers responded to that pressure. 

The level of understanding concerning the WIA performance measurement system among 

both staff and contract service providers is considerably greater than we observed during the 

first round of site visits. At that time, the predominant view among staff in general, 

including youth staff, is that service-provider staff were aware of the accountability system 

only in a very general way. That is, they understood that it was important to consider future 

outcomes for an individual when considering whether to register, but most respondents knew 

less about how their specific interactions with customers affect outcomes. Although many 

respondents talked more about the importance of delivering good service and its effect on 

performance than in the first round, the second-round group still are only modestly aware of 

how specific actions, like establishing reachable goals or timing of exits, could affect 

measured outcomes. 

Understanding and Implications of Common Measures for Youth 

Staff. The level of understanding of common measures among staff was lower than their 

knowledge of the WIA accountability system in general. While most staff were aware of the 

common measures in general, especially the current change to the literacy/numeracy measure, 

many were unaware that other youth measures may change with WIA reauthorization. Few 

understand the method of calculation of the various measures, the shift to a soft exit, or the likely 

implications of these changes. 

Service Providers. The service providers know less about common measures than their WIA 

staff counterparts. Overall, the evidence from staff and service provider interviews suggest that 

only two of the 11 local areas had already achieved a strong understanding of the common 

measures. In two other areas, training of service providers on the new measures was proceeding 

but was not complete at the time of the site visit. Several LWIAs (or the state) had provided 

training, but staff believed that the understanding of the measures was modest and required 

additional training. 

The generally low level of understanding of the common measures was not seen by any 

respondents as a problem, largely because 10 of the 11 sites concluded that the PY 05 changes 

in measures would have no significant effect on their program. One was preoccupied with a 

major organizational change shifting case management from staff to providers. Only two sites 

indicated that they will revise service design. Cumberland County, which is highly attuned to 

achieving good outcomes, is upset about the loss of the goal-attainment measure because it 

helps to identify those who are suitable for services and was a great way of tracking 
III-10

incremental advancement among the youngest of the younger youth, especially because no 

high school degree is expected within the typical short WIA enrollment period for these youth. 

The staff and providers both expect to reduce service to younger youth who have more 

difficulty showing gains within one or two quarters. Weld County staff indicated that they 

will be less likely to spend program resources sending youth to potentially valuable but 

unrecognized credentialing programs (for example, CPR or First Aid certification). 

As most respondents were unaware of the future changes expected with reauthorization, there 

was little concern with those potential changes. 

Methods of Conveying Accountability in the Youth Program 

LWIAs have two general methods of conveying information about the accountability system to 

youth providers: the procurement system and training. Procurement offers LWIAs the 

theoretical advantage of an enforceable relationship. However, as we discuss in this section, 

there are a number of practical constraints that limit enforcement. Thus, training, as in the adult 

program, is also a primary method to convey performance goals to youth providers. 

Procuring Youth Providers through Contracting for Services 

The bulk of the sites visited use competitive procedures to procure youth services. Each LWIA 

in this group uses a request-for-proposals in its efforts to acquire high-quality services at 

reasonable cost. A one-year contract, with one or two annual options, is the most common type 

of contract awarded. However, only one of the sites uses specific performance measures as the 

basis for payment in a performance-based contract. 

Of the 11 sites, 7 use cost reimbursable contracts with specific youth providers, providing 

payment to contractors as they meet expenditure and enrollment targets. At least one site also 

has exit volume requirements. As we noted in the interim report, enrollment targets are nearly 

universal because they ensure that contractors will spread the services to increase the program’s 

reach. All the sites have specific monetary amounts that cap the overall amount paid to a given 

provider. While together with the enrollment requirements these effectively provide an 

efficiency measure, no site uses their funding amounts to measure actual efficiency. Rather, 

budget proposals are typically assessed for the cost reasonableness of the services provided to a 

particular target group. Each of the sites embeds the required levels of performance on the 

mandatory measures as explicit performance goals in the contracts, but none of the costreimbursable sites uses performance benchmarks as a basis for payment. 

In contrast to the cost-reimbursable sites, Hudson County is the only site that uses 

performance-based contracts for youth services. For out-of-school-youth, the contractor is 
III-11

paid based upon how many high school credits their youth earn. For in-school-youth, a 

contractor serving ninth-grade youth is paid on a prorated basis for reaching two benchmarks 

for each youth: attendance and participation in a community-service project. A third 

benchmark is a threshold, promotion to tenth grade. If the student drops out of school before 

the final promotion payment, all payments are forfeited. The contractor must provide report 

cards and progress reports to verify these payment points. However, the summer program is 

run on a cost reimbursable basis. 

The remaining sites deliver a substantial portion of youth services themselves through 

partnership arrangements. In these cases, the successful bidders are consortia of agencies that 

include several youth providers, including school districts, community-based organizations, and 

YMCAs. Thus, youth services are part of a broader contract environment. These sites have 

memoranda of understanding to establish the respective duties of each party. Weld County, 

which uses county public employees to deliver most youth services, also uses memoranda of 

understanding to obtain services from other public entities such as other county agencies, the 

community college, and local school districts. 

Although performance on required measures is not used strictly as the basis for contract award, 

most of the sites indicated that they are considering performance more in the contract-award 

process than they did at the beginning of WIA. As an example, our interim report findings 

noted that most of the LWIAs merely considered the required outcomes to be one of several 

important measures. In contrast, the state of North Carolina, in the final-report sample, has 

recently directed each of its LWIAs to require contractors to include in their proposals how 

their services will help the LWIA and the state meet their performance requirements. Some of 

the LWIAs use a similar approach, such as Mid-Willamette, whose RFP requires bidders to 

demonstrate how their service design will improve required outcomes. The Mobile LWIA does 

not have an explicit requirement in its RFP, but the staff emphasize the importance of 

performance during its bidder’s conference with a detailed explication of WIA performance 

requirements. Mobile’s executives believe that the contractors have gotten the message. At 

least one LWIA’s enhanced concern with performance is driven by a reduction in its allocation. 

In the first two years of WIA, with larger allocations and reduced expenditures (from 

uncertainty about the new program) leading to plenty of carry-in funds in subsequent years, the 

Alabama LWIA had a fairly relaxed attitude about contracting. Most proposals were funded at 

the requested rates. Additionally, the local board was interested in innovative programs, and 

thus it favored a wider variety of awards. However, as allocations have steadily tightened in 

recent years, the LWIA’s contract review panels increasingly rely on performance on the 

required measures to determine the amount of funds awarded to each applicant. Thus, panels 
III-12

have reduced the amounts of some contracts, but they have not yet denied any qualified bidder 

an award. 

Enforcing Contract Provisions on Performance 

While nearly all youth contracts have provisions embedding the required performance levels, not 

a single LWIA explicitly enforces these provisions during the term of the contract. Several sites 

indicated that they use the same initial-sanction approach with their vendors that the states and 

federal government use. Most will give training and technical assistance and work with their 

contractors to improve performance when a contractor does poorly, whether it is on the required 

outcome measures or other aspects of the contract. At least two sites, Alabama LWIA and 

Region L, explicitly reduce funding in the next term. Performance on required measures appears 

to be more important in contract re-competitions and exercise of subsequent-year options. In 

that most of the sites use annual contracts with option years, it is important to note that several 

sites use performance as an important, and often essential, test for whether to exercise the 

options in the second or third year of the contracts. 

There are several underlying constraints on rigorous enforcement of contractors meeting their 

targeted WIA performance measures. 4 There is the well-understood problem that the delays 

built into the measures make it difficult to use these measures as management tools. But there 

are also some specific constraints in the youth-service environment that make sanctions on any 

basis difficult to use. First, the competitive environment is limited. Even in more populous 

areas, there is often no obvious youth provider that can take the place of an ousted contractor, 

especially for specific targeted groups. Second, although contracting theoretically establishes 

an arms-length relationship between two parties, most of the local sites had very collaborative 

relationships with their vendors. These vendors have typically been providing youth services 

going back to JTPA, and their leaders sometimes come from the same social milieu as the 

LWIA leaders. Finally, some organizations have significant access and acceptability 

advantages that may not be readily acquired by a substitute because the organization is deeply 

rooted in the community it serves. Other vendors from the public sector—school districts, 

probation agencies, foster care, and the like—may have unique access and even institutional 

control over the youth in their jurisdiction, and they may have no substitutes at all. Thus, 

 

4

 The LWIA survey report noted that while only a few LWIAs reported that the performance accountability system 

discourages numerous youth providers from responding to requests for proposals, almost half of local areas 

(46%) reported that the performance accountability system discourages at least a few youth providers from 

applying. Given this finding, it is perhaps not surprising that enforcement of performance targets is not 

stringent. 
III-13

absent the rare instances of such disastrous performance that the LWIA staff must act, most 

contract-enforcement tools tend to be too blunt to use with these vendors. 

Training Youth Providers 

If, in fact, contract enforcement has limited practical reach, training becomes increasingly 

important as a method to convey the requirements of the accountability system. However, only 

about half of the LWIAs regularly provide training to their youth contractors on accountability. 

Several sites that provide regular training to front-line staff use it as an opportunity to convey 

directly expectations on participant suitability, exit timing, and related issues of performance 

management. In contrast, other sites tend to emphasize reporting issues, that is, understanding 

the local or state MIS requirements and providing accurate reports. Timing may play a role in 

the reduced incidence of training in that several LWIAs indicated that they are planning future 

training, once their own staff fully understood the new common measures and would be in a 

better position to provide good information to contractors. 

Comparison Between Interim and Final Samples in 

Types of Measures Used 

In the interim report, we classified various types of performance measures and used this as the 

basis for organizing the chapter about communicating performance requirements to those who 

actually operate the workforce development system. While we did not use this classification to 

organize this chapter, it provides a useful mechanism to summarize our findings. Using a 

slightly revised classification scheme, Exhibit III-1 compares the use of different types of 

measures in the two groups of LWIAs visited. 

Exhibit III-1: 

Types of Performance Measures in the Interim and Final Samples 

Scope Interim Sample Final Sample 

Process measures. Process measures are measures of the system itself. 

• Entry 

measures 

These are volume measures that count the 

flow of individuals into the program to 

receive services, thus ensuring that these 

resources are widely distributed, or at least 

to the stipulated number of individuals, 

typically an important consideration for a 

public program. Additionally, they also 

reflect efficiency objectives because, in a 

contract, they must be linked to an amount 

of money. 

Entry measures were 

used in nearly all 14 

sites and were central 

to the procurement 

process. 

These measures 

remain central to the 

contract process in all 

the new sites. 
III-14

Scope Interim Sample Final Sample 

• Internal 

procedural 

measures 

These measures, often used in quality 

management and continuous improvement 

are intermediate measures of the system 

itself. For example, some programs 

measure the amount of time an individual 

spends in a service tier. Such measures 

may be used to allocate resources or to 

analyze processes for continuous 

improvement 

Five sites used these 

measures, primarily to 

manage the flow 

between service tiers 

and accurate 

reporting. 

Not used 

• Exit 

measures 

These are measures of the number of 

participants exiting a program and entering 

employment. As with volume at entry, these

may reflect the objective of serving a broad 

distribution of customers and efficiency 

objectives (when linked to contract 

reimbursement points). 

Exit measures were 

used in about half of 

the sites and were 

central to the 

procurement process. 

At least one site 

required contractors to 

exit a stipulated 

number of 

participants. 

Outcome measures. Outcome measures are those that describe the accomplishment of a program 

participant at the end of program participation or the program itself. These include: 

• Required 

measures 

The required measures occur, by definition, 

at or after exit (except for younger youth 

skill and diploma attainments). These 

compare a condition at program end to its 

comparable state before the program 

began.5

About half the sites 

used core measures 

in contracts 

All of the sites 

embedded the 

required WIA 

measures in contracts 

with youth providers, 

but in each case, the 

measures were 

general goals of the 

contract rather than 

benchmarks or 

payment points. Only 

one site used the 

required measures as 

part of performancebased contracting. 

• Job quality 

measures 

These measures, usually taken at exit, set a

particular standard for the quality of an 

entered employment. Examples include 

obtaining specified wage levels or fringebenefit receipt. 

Three sites used job 

quality measures, 

such as minimum 

wage at placement, 

variable payments to 

contractors based on 

wage at placement, 

and placement into 

jobs with health 

benefits. 

No sites had any job 

quality measures, 

although at least one 

contractor set its 

internal minimum 

wage at placement for 

older youth to help 

promote its wage gain 

outcome. 

 

5

 We do not consider alternative or proxy measures here because none of the local areas used them in contracts 

with service providers. See Chapter II for several statewide alternative measures. 
III-15

As we can see from the table above, there was a considerable degree of experimentation with a 

wide variety of measures during the earlier period. By the time of the final report, we see a 

much narrower range in the types and use of measures. However, as we have discussed 

throughout this chapter, there is a generally stronger focus on the required measures. Thus, we 

likely see some evidence in maturation of the accountability system. However, any such 

conclusions may be attributable to random variation in sample selection.6

Conclusion 

In this chapter, we describe the level of understanding of the performance accountability system 

and how this understanding is conveyed throughout the system, including to staff and service 

providers. As in our Interim Report, we found the effects of the performance measurement 

system to have wide-ranging influence, but also significant variation in how this influence played 

out across the states and local areas we visited. 

A common theme is that One-Stop design reflects the influence of the WIA accountability 

system in that most of the local area sites carefully insulate core services from the effects of the 

accountability system. Indeed, this design is so widely and readily accepted that most 

respondents in our sample feel that the WIA accountability system in general does not have any 

significant effect on the One-Stop design. Rather, most contend that the division of labor 

recognizes that joint services are better for the customer because each partner can emphasize the 

functional roles that it does best. 

We found the influence of the common measures to be relatively minimal. For example, 

although each of the state Wagner-Peyser programs is well on its way to adopting the common 

measures, no state indicated that they were making any significant change in the way they 

operated the program in the field, with the exception that all of the states indicated that the new 

measures required substantial changes in their MIS. Further, although we investigated the 

impact of the common measures on several other WIA partners, since none of these partners had 

actually adopted the common measures, there were no changes to partnership structure, service 

delivery, or the extent and intensity of referral or co-enrollment because of the implementation of 

the common measures. 

 

6

 As noted in Chapter 1, although our overall sample was selected randomly (stratified by region and with 

probability of selection proportional to WIA allocation) to be representative, the distinction between sites visited 

in our earlier round and in the final round of visits was made more arbitrarily. Thus, differences observed 

between the sites in the two samples cannot be taken to be representative of the larger performance measurement 

system. 
III-16

In addition, we found that front-line WIA adult and dislocated worker staff generally understand 

the WIA performance measurement system. In contrast to the findings of the interim report, 

nearly all front-line respondents in our final sample were aware of the overall importance of 

obtaining good performance and most know a good deal about the accountability system and its 

role. Staff in at least six sites consider the performance implications at virtually every step in 

working with customers. In several other sites, however, the staff are generally less concerned 

about performance, believing that they should focus primarily on a good service process. 

In general, the level of understanding of the common measures mirrors that of the WIA 

accountability system. The staff in those areas who best understand general WIA accountability 

also know the most about the common measures. In fact, many of the sites rely on informal 

collaboration among staff to ensure that the particulars of WIA accountability are observed by 

experienced staff and informal on-the-job-training is used whenever new staff are hired. 

As with the adult program staff, the LWIA youth staff—often a single individual in each 

LWIA—had a solid understanding of the accountability system within the context of the WIA 

youth program. However, the level of understanding of common measures among staff was 

lower than their knowledge of the WIA accountability system in general. The service providers 

know less about common measures than their WIA staff counterparts do, but this was rarely seen 

as a problem because representatives in 10 of the 11 sites had concluded that the changes due to 

common measures would have little to no impact on their service design and delivery. 

Because the youth program differs structurally from its adult and dislocated worker counterpart 

in that most of the services are contracted out to a variety of school districts, community-based 

organizations, and even a few for-profit firms, conveying accountability for youth programs is 

done using two methods: the procurement system and training. 

The bulk of the sites visited use competitive procedures to procure youth services. However, 

only one of the sites uses specific performance measures as the basis for payment in a 

performance-based contract. While nearly all youth contracts have provisions embedding the 

required performance levels, not a single LWIA explicitly enforces these provisions during the 

term of the contract due to one or more of several underlying constraints. First, the competitive 

environment is limited. Second, although contracting theoretically establishes a formal 

relationship between two parties, most of the local sites had very informal and collaborative 

relationships with their vendors. Finally, some organizations have significant access and 

acceptability advantages that may not be readily acquired by a substitute because the 

organization is deeply rooted in the community it serves. 
III-17

Given that contract enforcement tends to have limited practical reach, training becomes 

increasingly important as a method to convey the requirements of the accountability system. 

However, only about half of the LWIAs regularly provide training to their youth contractors on 

accountability. Admittedly, however, this may be an artifact of the timing of our visits, as 

several LWIAs indicated that they are planning future training, once their own staff fully 

understood the new common measures and would be in a better position to provide good 

information to contractors. 

Finally, we examined the types of measures being used by states and local areas to assess their 

performance. In our Interim Report, there was a considerable degree of experimentation with a 

wide variety of measures. In contrast, by the time of the final visits we saw a much narrower 

range in the types and use of measures. However, as was a common theme throughout the 

chapter, there is a generally stronger focus on the required measures. This may be taken as 

providing some evidence in the maturation of the accountability system. 
This page intentionally left blank.

Insert blank page here when making double-sided copies
IV-1

IV. SERVICE DESIGN AND DELIVERY 

Numerous state and local level respondents from this round of site visits described their efforts to 

ensure that their programs perform well on the WIA performance measures as well as their 

commitment to providing customers with the assistance they need to be successful. The reason 

why these respondents could say that they are both trying to succeed on the performance 

measures and helping customers to be successful is that in most cases what customers seek from 

WIA programs leads to good performance and vice versa. For example, being employed at a 

high wage, earning a certificate or diploma, and being enrolled in college or an advanced training 

program—all of which are goals of the performance system—are also common goals of WIA 

customers. Consequently, programs typically said that even in the absence of the performance 

system, they would still do the same things they do now to help customers to be successful. 

However, there are some cases where programs do things that have little or nothing to do with 

providing services that help customers to achieve their career or educational goals, but are 

related only to helping the program attain better results on the WIA performance measures. 

Typically, these actions or processes where programs attempt to “game the system,” such as by 

waiting to exit customers until the end of the quarter, have little impact on customers, although 

occasionally they do limit customer choice in services or employment. 

This chapter explores the impact of both the original WIA performance measures and the new 

common measures on the management of program entry, service design and delivery, and exit 

among WIA adult, dislocated worker and youth programs. It examines impacts on services and 

processes that are good for customers and for performance, as well as the few occasions when 

programs do things only to improve performance system results. The chapter begins by 

exploring the critical decisions programs make at program entry by exploring how and to what 

extent WIA programs manage both when and which individuals become participants who receive 

services that require inclusion in performance. The chapter then describes the impact of the 

performance system on service design and delivery. Finally, the chapter concludes by discussing 

how, why and to what extent programs manage exit. 
IV-2

Managing Participation: Controlling Program Entry 

A critical way that states, local areas and providers attempt to manage their service delivery to 

achieve high performance is by managing both who and when eligible customers become WIA 

participants who are counted in the performance measures. 1 This section of the chapter first 

describes how programs attempt to manage when individuals become participants. Following 

this discussion of the timing of participation, the section then describes how programs also try to 

manage which customers become program participants. 

Managing When Customers Become Participants 

One way that programs manage participation is by controlling when individuals become 

participants. Typically this entails delaying when customers begin receiving services that require 

inclusion in performance. Based on our recent site visits, we found this to be much more 

common among adult and dislocated worker programs than youth programs. We also found that 

the common measures are expected to have almost no impact on how programs manage the 

timing of participation. 

How Programs Delay Participation 

There are two primary ways that WIA programs delay providing customers with services that 

require inclusion in performance. The first and most common method is to stretch out the 

services and interactions with staff customers can have before they receive services that require 

formal enrollment in the program and inclusion in performance. Among adult and dislocated 

worker programs, this is often accomplished through more expansive definitions of what services 

are considered self-service and informational, as customers who receive only those services are 

not required to be included in performance. 

Because the guidance provided in the WIA Final Rule and TEGL 7-99 about how to 

operationalize self-services and informational services was fairly broad, the way that these types 

 

1

 Currently, all individuals who become program “participants” by being determined eligible and receiving a 

service funded by the program in a physical location are required to be included in the performance calculations. 

For WIA adult and dislocated worker programs, a notable exception to this rule is that “individuals who 

participate exclusively in self-service or informational activities” are excluded by law from the performance 

measures. 

Henceforth, for brevity, we will use the term “participant” to refer only to those individuals who are included in 

performance, rather than for all individuals who are determined eligible and receive a service funded by the 

program, as defined in TEGLs 28-04 and 17-05. Consequently, for discussions related to the adult and 

dislocated worker programs, a “participant” will be an individual determined eligible and who has received more 

than self service or informational activities funded by WIA. For individuals served by the WIA youth program, 

“participant” will have the same meaning as in TEGLs 28-04 and 17-05. 
IV-3

of services are defined in different states and local areas varies considerably. In some local 

areas, these definitions have been specifically developed to allow One-Stop staff to provide more 

services and work with customers more extensively before customers have to be included in the 

performance measures. Consequently, while a service in one state or local area is defined as a 

“staff-assisted” core or an intensive service requiring the customer to be included in 

performance, in another locale the same service is defined as informational or self-service. For 

example, in North Carolina, customers who participate in Job Clubs are included in performance, 

while in the Oregon Consortium, Job Clubs are not considered a service requiring inclusion in 

performance. Similarly, the amount of time customers can spend meeting individually with WIA 

staff without being counted as having received more than self-service or informational activities, 

varies considerably. For example, in New Jersey, anytime a customer meets individually with a 

WIA staff person, the customer is considered to have received a “staff-assisted” service and must 

be included in performance. By contrast, in Oregon’s Mid-Willamette LWIA, WIA staff have to 

spend more than two hours in any one workweek assisting individual customers before the 

customer has to be included in performance. 

The other way that adult and dislocated worker programs can stretch out the number of services 

potential customers can receive prior to participation is by allowing customers to receive more 

services from the state employment service. This is an effective way of delaying when 

customers are included in performance, because customers only have to be included in the WIA 

performance measures when they receive services funded by WIA. 

WIA youth programs have much less flexibility than adult and dislocated workers programs to 

stretch out the interaction between program counselors and potential customers. This is due 

partly to the fact that the exclusion of customers who receive only self-service or informational 

activities from performance applies only to WIA adult and dislocated worker programs. In 

addition, there is less likely to be another readily available youth program that can provide 

similar services as WIA, as Wagner-Peyser programs can for WIA adult and dislocated worker 

customers. Consequently, WIA youth programs can only stretch out the interaction between 

counselors and potential customers by lengthening the number of steps youth have to complete 

during eligibility determination and the assessment of a youth’s appropriateness for the program. 

As will be explained below, programs typically do this by requiring multiple meetings between 

youth applicants and counselors to collect required eligibility paperwork and describe program 

requirements and activities in an effort to gauge the youth’s motivation and willingness to follow 

through with program activities. 

The other principal way in which programs delay participation is by staff referring customers to 

partners for services prior to those customers receiving WIA services. This method is used 
IV-4

equally by adult, dislocated worker and youth programs. Typically customers are instructed to 

complete partner services and then return to receive WIA services. We discuss this further in the 

next section. 

Reasons for Delaying Participation 

Delaying when customers become participants is done for a variety of reasons, and was noted as 

occurring in a majority of sites in our sample. The primary reasons noted by respondents include 

to change the period measured for pre-program wages, to give customers time to gain skills 

outside of WIA that are necessary for success on the performance measures, and to give staff 

more time to effectively screen customers for a variety of factors to be discussed below. 

One of the clearest and most purely performance-related reasons programs sometimes delay 

registration is to change the period used to calculate adult, dislocated worker and older youth 

customers’ pre-program earnings. This finding is supported by our recent survey of LWIAs that 

similarly found that a number of local areas reported trying to delay inclusion in performance 

until a customer’s second and third quarters prior to participation reflected lower earnings, 

usually due to a longer period of unemployment for the customer.2

 Counselors said that when 

they do this, they simply stretch out delivery of informational, self-service or Employment 

Service (ES) activities such as job search, until enough time has passed. Although counselors 

said they do not think that stretching out non-registered core or ES activities harms these 

customers, it could be argued that this practice might have a potentially negative effect on some 

customers. For example, by forcing customers to be unemployed longer before receiving more 

substantial services such as training, counselors are making it more likely that customers will 

feel forced to drop out of training and take a job (probably at a lower than desired pay rate) 

simply because they can not afford to be unemployed for so long. 

Adult program respondents identified trying to change the period measured for pre-program 

wages as a reason for delaying participation more often than did youth program respondents. 

Respondents attributed this difference to the fact that older youth are less likely to have 

significant pre-program wages and because programs have less flexibility to provide services 

without having to include youth in performance. With the common measures’ change in the 

definition of which quarters are used to calculate pre-program wages for dislocated workers, 

 

2

 Younger youth do not have an earnings measure, so their pre-program wages have no bearing on performance. 
IV-5

some respondents also mentioned that they may be more likely to try to lower pre-program 

earnings for dislocated workers. 3 

Another common reason for delaying participation is when customers’ skills are not up to the 

level where LWIAs feel they can serve those customers effectively. While many respondents 

stated that they do this primarily for the good of the customers, most also admitted that it is also 

important for performance reasons. The most common set of skills that affects this consideration 

are basic skills, particularly reading skills. Typically, LWIAs asserted that they have to limit 

their services primarily to customers with higher basic skills because of the similar requirements 

of many training providers and employers. For example, in the Alabama LWIA, adults and 

dislocated workers are required to score at or above a 9th grade reading level before being 

enrolled, as that is the level required by most of the LWIA’s training providers. When applicants 

are assessed below the required level, they are typically referred to a local adult education 

partner and told to return when their reading scores have improved. Most LWIAs asserted that 

they do not have the resources to pay for lengthy basic skills remediation in addition to training, 

which is why they must refer these customers to partners for services. 4

Some respondents also noted that they sometimes delay participation when customers have 

pressing needs that the WIA program cannot appropriately address. For example, customers 

with substance abuse problems or who are homeless need to deal with those problems before 

they will be able to focus on looking for work or before an employer will hire them, according to 

these respondents. Consequently, WIA programs often refer these applicants to partner 

organizations for services and instruct them to return when the services are completed. For 

example, a youth provider in Mobile, Alabama, requires potential youth customers to pass a drug 

test before enrollment. If youth fail the drug test, they are then referred to a substance-abuse 

provider and told they have 30 days to clean up and retake the test. 

A final reason cited by numerous respondents for delaying participation is to allow staff more 

time to screen and select out customers who lack the motivation, goals or skills to be successful 

in the WIA program and to meet the performance measures. These screening criteria are 

discussed extensively in the next section. 

 

3

 Under the common measures, pre-program wages for dislocated workers are the second and third quarters prior 

to registration rather than the second and third quarters prior to dislocation as under the previous measures. 

Effective July 1, 2006, however, the common measures will remove pre-program earnings as a component and 

focus solely on earnings post-program participation. This may well alleviate the perceived need for this delay 

tactic. 

4

 WIA Sec. 134(d)(4)(D)(vii) requires that adult education and literacy activities be provided in combination with 

other training services, except customized training. 
IV-6

Managing Who Becomes a Participant 

In addition to managing when individuals become participants, programs also manage 

participation by controlling who becomes a participant. This appears to be widespread in the 

local areas we visited, particularly in adult and dislocated worker programs. It also appears to be 

more common than we found in our first round of site visits. 

Adults and Dislocated Workers 

Respondents in every local area visited during this round of site visits indicated that they screen 

potential adult and dislocated worker customers to try to manage which customers are allowed to 

become registered participants. These qualitative findings are supported by our recent survey of 

LWIA’s, which found that 41 percent of local areas said they take performance into 

consideration when registering customers in WIA. 

In addition to helping them to achieve good performance, respondents gave a number of other 

reasons why they try to control who becomes a participant. For example, the Mobile LWIA’s 

executive director said that he would like to see greater selectivity in his program to ensure that 

the WIA programs refer only high quality job seekers to local businesses. A number of other 

respondents said that screening and selectivity help them ensure that their limited resources are 

used efficiently and that WIA is the most appropriate program for customers. In addition, some 

respondents noted that the information collected from assessments conducted during preparticipation screening are used by counselors to improve service planning. 

Indeed, some selectivity about which individuals become participants is specifically allowed and 

even required by the Workforce Investment Act. For example, adult and dislocated worker 

programs are specifically instructed in the Act to carefully assess those customers who are to 

receive intensive and training services, to ensure that customers are in need of such services “to 

obtain or retain employment that allows for self sufficiency.”5 For training services, programs 

are required to be even more selective, and are allowed to provide services only to those 

customers who “have the skills and qualifications to successfully participate in the selected 

program of training services.”6

 Finally, all WIA Title I programs are directed to “provide 

employment and training opportunities to those who can benefit from, and who are in most need 

of, such opportunities.”7 This last legislative justification is undoubtedly why several 

 

5

 WIA, Sec.134 (d)(3)(B). 

6

 WIA, Sec.134 (d)(4)(A)(ii). 

7

 WIA, Sec. 195 (1). 
IV-7

respondents said that their screening programs are aimed at determining customers’ “ability to 

benefit” from WIA services. 

Characteristics Used by Programs to Screen Customers 

There are a number of characteristics on which adult and dislocated worker programs screen 

potential customers. Some of the most common characteristics include: motivation and 

commitment, skills and qualifications related to the customer’s ability to be successful in 

completing training and finding employment, and pre-program wages. Our recent survey of 

local areas found nearly identically that the most common factors local areas consider when 

registering customers for WIA (and thus including them in performance calculations) included 

customer motivation, work readiness or employment barriers and the impact of prior wages. 

Similarly, in our prior round of site visits, we found that many programs screened adults and 

dislocated workers into WIA on the basis of commitment to the program, while fewer screened 

on the basis customers’ likelihood of being successful on the measures or pre-program wages. 

While all of these characteristics relate at least somewhat to both success on the performance 

measures and the ability of the program to serve customers well, screening for pre-program 

wages is primarily driven by the likelihood of achieving success on the earnings gains 

performance measure.8

One of the most prevalent characteristics used by locals—both formally and informally—to 

determine whether to allow customers to become participants is customer motivation and 

commitment to the program. To assess commitment and motivation, many local programs have 

developed elaborate formal processes that customers must go through prior to becoming a 

participant. For example, in the Montgomery One-Stop center, potential customers are required 

to participate in an orientation and attend at least one workshop before they are allowed to 

receive more extensive services. Customers who do not show up as required for these activities 

are not allowed to become registered participants. 

Other local programs use more informal methods to test motivation and commitment. For 

example, in Weld County, Colorado, counselors who determine in an initial assessment (not 

requiring the customer to be counted in performance) that a customer is in need of additional 

services often schedule the customer’s first service requiring inclusion in performance (a 

 

8

 Although this may sound negative, in some sense it may serve as an offset to the incentive to cream the best 

candidates for inclusion into WIA because, under this example, the motivation is to identify those who have low 

pre-program earnings. As noted above, though, this consideration may well be removed by recent guidance from 

DOL suggesting that, as of July 1, 2006, preprogram earnings will no longer be included as a component of the 

earnings measure. 
IV-8

comprehensive assessment) for another day. Those who do not show up for the second 

appointment are thus not included in the local area’s performance. This allows counselors to 

“weed out” those who are not serious about following through with the program’s requirements. 

Another way that some programs screen for customer commitment is by assessing whether 

customers’ interests are congruent with performance goals. For example, Maine has provided 

guidance to its local areas that programs should “limit the number of enrolled customers who 

only want a job and will not work toward a credential.” Counselors in a local area in another 

state also said that if a customer tells them he or she is looking for only part-time employment, 

counselors are unlikely to provide this individual with more than core services that do not require 

inclusion in performance. In these situations, counselors indicated that they would refer these 

customers to the Employment Service for assistance. 

Another common set of characteristics used by programs when selecting participants are 

customer skills and qualifications. These characteristics are assessed both to ensure that 

customers have the qualifications to be successful in the program as well as to ensure that WIA 

programs are the best and most appropriate to serve customers successfully. As noted above, 

one set of skills, for which those potential customers are commonly assessed, are basic skills. If 

customers’ basic skills are too low for the WIA program to serve effectively, customers are 

referred to other partners such as Adult Education who are better suited to serving those who are 

learning disabled. 

In addition to assessing basic skills, some programs assess customers’ likelihood of success in 

the labor market and thus success on the performance measures. For example, they review 

customers’ past work histories, life situations, job interests, and support systems when 

determining who is likely to meet the measures. For those with serious barriers such as 

substance abuse problems, homelessness or a disability, adult and dislocated worker programs 

often choose to refer customers to partner programs rather than continue to provide services 

through WIA. In some states, such as New Jersey, customers with serious barriers are purposely 

enrolled in programs with less stringent performance standards, such as a state welfare-to-work 

program rather than WIA. 

Finally, a number of local programs screen customers for their pre-program earnings. While this 

type of screening is sometimes used to screen out customers with high pre-program earnings, 

respondents also said that it is used to help counselors with service planning to make it more 

likely that customers will experience a significant gain in earnings after participating in the 

program. To screen customers for pre-program earnings, counselors are instructed to ask 

potential customers about their earnings in the second and third quarters prior to participation 
IV-9

and use that information in making selection and service-planning decisions. For example, in 

Maine, the state developed a Performance Management Guide that includes an “Entrance 

Checklist” that directs case managers to determine pre-program earnings and calculate an 

earnings change goal. In Mid-Willamette, adult and dislocated worker program case managers 

have been provided with an electronic Performance Measure Calculation Worksheet to use for 

this purpose. 

Extent of Selectivity 

Despite the existence of these screening processes, respondents indicated that they do not 

exclude all individuals who are unlikely to achieve high performance. Instead many respondents 

said that they often use the screening processes to collect performance-related information on 

participants. This allows programs not only to conduct more effective service planning with 

customers, but also helps them try to find other participants who can “balance” the 

characteristics of customers with barriers or characteristics that make it difficult for them to be 

successful on the performance measures. For example, in one local area, counselors said that 

even though they do not like to, they will typically provide extensive services to customers with 

high pre-program earnings, but will then seek to find other customers with lower earnings to 

“balance” their likelihood of reaching their performance goal on the earnings measure. 

Similarly, case managers in another area noted that they often struggle with whether to enroll 

customers with serious barriers or refer them to partners such as vocational rehabilitation, adult 

education, homeless services or drug and alcohol services. Particularly in a semi-rural area, 

these kinds of decisions are particularly difficult as “its not just some person...This is your 

neighbor. This is your brother. This could be you.” [Coastal Counties] In the words of one 

local area director, programs want to find a “middle ground” between getting positive outcomes 

and being forced to turn people away. 

Indeed, a number of respondents made it clear that they sometimes choose to allow some 

customers to receive more than self-service or informational services even when they know the 

customer is unlikely to help the area reach its performance goals. For example, in North 

Carolina’s Region L, some local providers indicated that they will allow even customers with 

many barriers to participate in the program, because, as one respondent put it, “slow progress is 

still progress.” In Coastal Counties, counselors said that they will often choose to register 

participants who are likely to become self-employed even though those customers will be 

excluded from the earnings increase measure and the state has a non-WIA program expressly for 

customers interested in self-employment. 

In many areas, this willingness to enroll customers who will not contribute positively to 

performance is partly due to the aforementioned attempts by local areas to balance the negative 
IV-10

performance of a few customers with the positive performance outcomes earned by most other 

customers. As one respondent in Idaho put it, local areas are “willing to take a hit” with a few 

individuals as long as they have a solid number of other customers who are likely to be 

successful on the measures. 

In addition, it does not appear that adult and dislocated worker programs are extensively 

screening out low income customers Instead, many respondents at both the state and local 

levels indicated that many adult and dislocated worker programs actively recruit individuals with 

low or zero pre-program earnings who are typically low income. For example, in New Jersey, 

respondents said that because of the earnings measure, they try to enroll long-term welfare 

recipients who are work-ready.9

 

Effect of Common Measures 

All of the states and local areas visited for this study said that they plan to continue the screening 

processes they used during the prior measures under common measures. In fact, because they 

expect that the performance standards they have to achieve will continue to rise, in two local 

areas, respondents indicated that believed that under the common measures they will become 

even more selective than in the past. For example in one state, local providers and case 

managers said they believe that ever increasing standards will likely discourage them from 

enrolling individuals with serious barriers such as disabled adults. 

Youth 

As we found in our Interim Report, we encountered less evidence of screening by WIA youth 

programs than by adult and dislocated worker programs. This may be due to the fact that WIA 

youth programs do not have “non-registered core services” and must register all youth who are 

provided with any of the 10 elements. However, because youth programs do need to determine 

“ability to benefit,” eligibility, and to ensure that all youth who meet the minimum income 

criteria are referred “to appropriate training and educational programs”10 prior to registration, 

programs do have some opportunities for screening. In fact, in some local areas, the paperwork 

and discussions with staff needed to determine eligibility and the appropriateness of youth for the 

program can involve multiple steps and appointments with counselors. This section discusses 

both the most common factors programs screen for and the rigor of their screening mechanisms. 

 

9

 The change in the earnings measure for PY 2006, which focuses solely on post-program wages, may well provide 

an opposite incentive in that any low-income customer may be unlikely to achieve high outcomes, regardless of 

their work-readiness. 

10 WIA Sec. 129 (c)(3)(A)(ii). 
IV-11

Youth Program Screening Characteristics and Processes 

We found that the most common screening factors used by local youth programs include 

motivation, ability of the program to effectively serve youth, and the likelihood that youth will 

succeed in the program and achieve performance goals. As with adult and dislocated worker 

programs, the most common screening characteristic used by WIA youth programs is motivation 

and commitment to the program. These factors are seen as important in ensuring good WIA 

performance since youth who lack these characteristics are more likely to drop out before 

completion and fare poorly on the performance measures. 

Local programs assess youths’ motivation and commitment in different ways. Oregon’s MidWillamette youth program requires case managers to directly assess youth motivation, as well as 

punctuality, attendance and paperwork completion before they are allowed to receive WIA 

services. In Cape Fear, North Carolina, the YWCA requires youth to follow-through on four 

different appointments before they are enrolled in the WIA youth program. The WIA youth 

program in Montgomery Alabama also tests whether the families of potential youth customers 

are committed to the program by requiring that parents accompany youth to the eligibility 

determination session at the One-Stop center. The program director said that the One-Stop 

center added this requirement because of the critical role of parents in helping youth succeed, 

and because it made it much more likely that parents would supply the income verification 

documents required for eligibility. 

In addition to assessing motivation, several respondents noted that their WIA youth programs 

also assess the likelihood that youth will succeed in attaining both their own goals and the goals 

of the program. For example, one of Mobile’s main youth providers, DESI, rigorously screens 

youth to ensure that they are above the 3rd grade level on the TABE (Test of Adult Basic 

Education), can pass a drug test, and have a high probability of remaining on a job long enough 

to meet the retention measures. In Oregon’s Mid-Willamette workforce area, WIA youth 

programs assess youth on numerous skill categories, including problem-solving, communication, 

teamwork, work readiness skills (such as appearance, hygiene, and ability to be supervised), and 

stress management. 

A few programs also screen for what they term as their ability to serve youth appropriately. For 

example, in Mobile, one in-school provider only enrolls youth who are assessed as one grade 

level behind or less because the provider does not have the capacity to assist youth who have 

greater basic skills problems. 

Despite the use of these screening processes, most youth programs indicated that they are not 

especially selective. For example, youth providers for the Montgomery One-Stop center stated 
IV-12

that they only exclude active drug users and violent youth. A respondent in Mobile noted that 

local youth providers are more interested in estimating income eligibility to meet enrollment 

targets than in figuring out who is likely to attain positive performance outcomes. 

However, other LWIAs that have failed to attain some of their youth goals have made their 

screening and selectivity processes more rigorous. For example, one LWIA, which missed 

attainment of its older youth earnings change goal in PY 2002, brought in national performanceimprovement consultants who advised the area on how to adjust their screening processes to 

improve performance.11 Consequently, this area’s youth counselors now are using a formal 

“Readiness for WIA” evaluation form that assesses youth attendance, punctuality, appearance, 

motivation and effort, paperwork completion, personal and family coping skills, problem-solving 

skills, communication skills and stress-management skills. Youth who score high on this 

evaluation are enrolled, while those who score very low are referred to other programs. 

Separately, case managers also assess pre-program wages for older youth. The manager of youth 

programs for the LWIA said she feels this process has begun to positively impact the LWIA’s 

youth performance, and since PY 02, the LWIA has attained at least 80 percent of its negotiated 

goal. 

A second LWIA, which has faced similar youth performance problems, is also stepping up its 

screening and moving to become more selective. For example, the LWIA is planning to stop 

serving youth with disabilities due to pressure from the state to improve performance. The 

LWIA also indicated in its 2005 performance improvement plan that it plans to set up a referral 

process to increase its pool of older youth who lack work experience and are likely to have zero 

or limited pre-program earnings. These changes may be at least partially in response to a 

“Performance Management Guide” developed by state staff that advised local youth programs to 

assess older youth pre-program earnings and “balance” enrollment of high pre-program earnings 

customers with low or zero pre-program earnings due to performance considerations. Although 

there was nothing specific about disabilities in this guide, the local area believes these youth are 

too unlikely to achieve strong performance and, thus, feels they need to refer them to other 

programs. 

 

11 Although we did not ask the respondents to identify the consultants, we did confirm that they were not part of 

SPR’s Performance Enhancement Project, which provides technical assistance and training to states and local 

areas to assist in achieving strong performance. 
IV-13

Expected Impact of Common Measures 

In contrast to the expected minor impact of the common measures on screening and selectivity 

for adult and dislocated worker programs, most WIA youth program respondents reported that 

they expect the common measures to have a significant effect on their selectivity. Although 

some sites still had too little understanding of the new measures to understand their implications, 

most of those that did indicated that they believe the common measures are likely to substantially 

increase selectivity, preventing them from serving “hard-to-serve” youth such as those with 

disabilities. 

Both the Literacy and Numeracy and the Attainment of a Degree or Certificate measures were 

highlighted by respondents as reasons why they think they will have to become more selective 

under the common measures. Specifically, respondents said that they do not think youth facing 

significant barriers such as learning disabilities will be able to achieve the goals required for 

success on these measures. Consequently, they believe that they will be unable to continue to 

serve many disabled or “hard-to-serve” youth or they will likely have trouble meeting their 

performance goals. For example, officials in several local areas said that they could not see how 

youth with developmental disabilities would be able to increase one or more functioning levels in 

one year as required by the Literacy and Numeracy Gains measure. A youth program manager 

in another local area asserted that her area was likely to stop serving disabled youth because 

these youth would have great difficulty achieving a certificate that qualified under the new, 

“harder” definitions. 12 

Another likely impact on selectivity that was noted by numerous states and local areas is that the 

common measures are likely to result in WIA youth programs choosing to serve fewer 14 and 

15-year olds. Respondents said that one of the main reasons they are likely to exclude these 

younger youth is because they are unlikely to attain the new Placement in Employment or 

Education measure unless they are kept enrolled in the program until they are old enough to be 

placed in unsubsidized employment, the military, or enrolled in advanced training, occupational 

skills training, or post-secondary education. However, this would likely require 14 and 15-yearolds to participate in the WIA program for more than a year, which is considered to be too 

difficult and resource-intensive for many programs. 

 

12 The definition of a certificate under the common measures is much more rigorous than under WIA’s original 

Older Youth Employment and Credential measures, which allowed states and local areas to establish locally 

recognized credentials that could be issued by local WIBs. These local credentials will not count under the 

common measures. 
IV-14

Several respondents also noted that the new Attainment of a Degree or Certificate measure 

similarly will push programs away from serving 14 and 15-year-olds. They said that because 

youth in secondary school at exit are not excluded from the measure, youth who are too young to 

earn a diploma or GED will have to attain some type of certificate on top of their regular school 

studies to avoid a negative outcome, and they do not feel that this is realistic. Otherwise, these 

younger youth will have to be kept enrolled in the program until they are able to complete a GED 

or diploma, in one to three years. 

In addition to serving fewer 14 and 15-year olds, some respondents also said that the common 

measures will push WIA youth programs into serving far fewer in-school youth. They explained 

that the fact that youth in secondary school at exit will not be excluded from either the new 

Placement in Employment or Education measure or the Attainment of a Degree or Certificate 

measure makes it harder for youth programs to serve younger in-school youth and succeed on 

these measures. This impact would be in line with DOL’s vision to focus on serving out-ofschool youth that was described in TEGL 3-04 in July, 2004. 

Service Design and Delivery 

As discussed at the outset of this chapter, many respondents said that performance implications 

infuse their decisions about service design and delivery on a daily basis. However, a number of 

respondents from the same states or even local areas were also emphatic that the performance 

measures do not drive their service delivery, but rather the needs of customers drive services. 

Despite the apparent disagreement between these findings, a number of respondents pointed out 

that both of these approaches often have the same result, because, very often, what is good for 

customers (such as finding a job at a high wage) is also good for program performance. Indeed, 

we found that only rarely do programs change their service delivery or design simply to improve 

their performance outcomes. And, even when they do, respondents assured us that they make 

sure that these performance-related changes will not harm customers. In fact, a number of 

respondents also specifically pointed out situations where staff knowingly sacrifice 

performance—such as by approving training programs leading to low earnings—to do what is in 

the best interest of customers. 

This section first describes changes to service design and delivery related to the original WIA 

measures on adult and dislocated worker programs and youth programs. The section then moves 

on to discuss the impact of the common measures on the three programs. Because the transition 

to common measures began only shortly before we conducted this round of site visits, only a few 

respondents were knowledgeable enough to talk in detail about their impact on service design 
IV-15

and delivery. Consequently, the findings we present related to the common measures should be 

treated as only suggestive. 

Effect of Original WIA Measures on Service Design and Delivery 

Among the sites we visited for this round, we found that all had made some changes to their 

service design and delivery that respondents said were at least partly due to their desire to 

improve their success on the performance measures. Perhaps because of their focus on the new 

common measures, youth program respondents noted fewer impacts than did adult and 

dislocated worker respondents. In the remainder of this section, we discuss impacts of 

performance on service design and delivery noted by respondents in all three program streams, 

beginning with adults and dislocated workers. 

Adult and Dislocated Worker Programs 

For adult and dislocated worker programs, respondents described a number of impacts on service 

design and delivery related to the performance system, including a greater emphasis on case 

management, training services, placement services and follow-up and retention services. These 

results are somewhat similar to what we found in the earlier round of site visits where training 

and follow-up services were impacted by performance. 

Training 

One of the main ways that respondents reported the performance system has had an impact on 

services is in the area of training. Numerous respondents said that they believe training services 

are critical to good performance, especially on the earnings measures. This finding is supported 

by our recent survey of LWIAs, which found that about three-fourths (73%) of local areas who 

reported performance having an effect on the service tiers indicated that they had placed 

additional emphasis on training services due to performance. This is also similar to what we 

found in our first round of site visits. 

The training-related changes reported by states and local areas include investing in higher-cost 

and higher-quality training programs, trying to increase the use of On-the-Job Training (OJT), 

and tightening up requirements for customers to enter training. 

Many LWIAs and states said that they have changed their service design and delivery due to the 

performance system by investing more resources in funding higher-quality and higher cost 

training programs for customers. Respondents said that they were particularly willing to pay for 

higher cost training programs for occupations such as registered nurse or truck driver that will 

result in customers being qualified for high paying jobs that will positively impact their 
IV-16

performance on the earnings measures. In the Alabama LWIA, respondents said that they are 

particularly likely to approve longer-term, higher cost training for dislocated workers who have 

to experience a significant earnings gain to meet the local area’s earnings goal. 

Because of this trend toward approving more high cost training programs, several respondents 

reported higher average training costs per customer. Because their budgets have stayed the same 

or even decreased in recent years, these sites also reported being able to serve fewer customers. 

Another impact of the performance system on training services is that several LWIAs we visited 

are emphasizing OJT as a way to improve their performance outcomes. These areas emphasized 

the performance benefits of training programs that have a guaranteed job at the end of the 

process. For example, Coastal Counties, as a result of its recent performance improvement plan, 

is now requiring 30 percent of all training for adults and dislocated workers to be in the form of 

OJT. Mobile is also emphasizing OJT and is hoping to have 50 percent of all training be in the 

form of OJT. Respondents in Weld County Colorado, also noted the performance benefits of 

OJT, although they complained about the time-intensive nature of setting up OJT contracts. 

This increased emphasis on OJTs may be easier now that the adult and dislocated worker 

Employment and Credential measures are likely to be eliminated. For example, Cumberland 

County, New Jersey indicated that it would like to offer more OJT because of OJT’s positive 

performance impacts and cost efficiency, but have chosen not to do so thus far because of the 

adult and dislocated worker Employment and Credential measures. Guidance provided by 

Maine also cautions local areas about using OJT too extensively because of the Employment and 

Credential measure, unless the LWIA has established a local credential that allows customers to 

earn a credential from OJT. 

In addition to emphasizing higher cost training and OJT, several LWIAs also noted that the 

performance system has caused them to tighten up their training plan approval processes. One 

common way that programs have done this is by requiring customers to conduct additional labor 

market research to ensure that customers understand what jobs are available and the wages they 

are likely to earn upon completion of training in a given occupation. Some local areas, such as 

Region L, use this labor-market information to actively discourage customers from choosing 

certain occupations, like Certified Nursing Assistant or child care provider, that pay low wages. 

In other areas, like Weld County, when labor market research shows that the customers are going 

to have a difficult time finding employment in a chosen occupation, customers are required to 

bring in evidence, such as current job postings, indicating that jobs in the occupation are 

available in the local area before they are allowed to enter training. 
IV-17

Some LWIAs have also begun requiring customers to conduct other types of research before a 

training plan is approved to make sure customers have good information with which to make 

decisions and are serious about training. For example, Mid-Willamette requires potential 

training customers to visit with both potential employers and training providers, while Idaho 

LWIA counselors often require potential training recipients to shadow someone in their proposed 

occupation. 

Although these LWIAs said that this research is beneficial to customers, several also explicitly 

linked these requirements to performance concerns. For example, Region L began requiring 

additional labor market research after a number of dislocated workers who were trained to be 

child care providers found jobs with low earnings, hurting the LWIA’s earning performance. 

Follow-up Services 

In addition to training, follow-up services are another common service area that has received 

additional emphasis due to performance. Several respondents noted that they have stepped up 

their provision of follow-up activities and services—both tracking of supplemental data and 

provision of re-employment and retention services—in the past few years due to the importance 

of follow-up to achieving success on the performance measures. Our survey of LWIAs reported 

a similar finding in that more than three-fourths of all local areas reported that the performance 

system has resulted in increased follow-up costs. 

Most respondents felt that the collection of supplemental data is the element of follow-up that is 

most important for performance purposes. Similarly, our survey of LWIAs reported that 96 

percent of local areas felt that supplemental data has at least some impact on their performance. 

By contrast, far fewer respondents from this round of site visits thought that actual reemployment or advancement services are critical, since few customers actually partake of these 

latter services. 

While the typical follow-up services reported by states and locals consist primarily of quarterly 

communication with exited customers to collect supplemental data on employment outcomes and 

offer additional services, a few of the local areas we visited provide much more substantial 

services. For example, primarily to overcome performance problems, Mid-Willamette in Oregon 

has implemented an extensive system of follow-up services. This policy, called a “Retention 

Services Strategy,” requires that each One-Stop center have a specific retention specialist and 

requires extensive contact between retention specialists or counselors and newly employed 

customers. For example, the LWIA requires that staff check in with customers within three to 

five days after the customer becomes employed to find out how the job is going, whether there 

are any immediate retention issues, and to introduce an “income improvement plan” for the 
IV-18

customer. Throughout the exit quarter and the first three quarters after exit, retention specialists 

are also required to contact exited customers every 30 days, trying to reach them at different 

times of day or evening if necessary. The LWIA also ensures clear communication by requiring 

counselors, who work with customers during enrollment, to share customer files with retention 

specialists at exit. From program entry, the LWIA emphasizes the importance of follow-up 

services, repeatedly informing customers prior to exit of available follow-up services both 

through in-person meetings with counselors or retention specialists and letters from the One-Stop 

center. 

Despite the strong belief of many states and local areas that follow-up services are critical for 

good performance, other respondents were not sure that follow-up services really improve 

performance. These respondents commented on the large amount of time counselors spend 

trying to track down customers to offer them follow-up services and the small number of 

customers who actually use the proffered services. A few respondents said that they believe 

resources used on follow-up would be much better spent on in-program services such as case 

management or training. A handful of these respondents went even further, calling follow-up 

“intrusive” and based on a “welfare model” that involves checking up on people. Staff from one 

local area said that exited customers often do not want to be reminded of WIA because they see 

their enrollment in the program as a time in their lives when they needed help. As one local area 

put it, “By and large, people don’t want us in their lives [after completing services.]” 

Respondents who disagreed with the need for programs to be aggressive about regularly 

contacting customers to check-in with them and offer follow-up services typically asserted that 

customers would contact the One-Stop center on their own if they needed help finding a new job. 

However, this fact was disputed by other respondents who said they think retention services are 

the only way to end the “revolving door” for customers who quickly lose their jobs and return to 

the program. 

Case Management 

Related to the impact of the performance system on training and follow-up services is the 

increased focus this system demands local areas place on case management. This finding is 

supported by our survey of LWIAs where about two-thirds (63%) of all LWIAs reported 

increased costs for case management due to the performance system. 

One of the main impacts of performance on case management has been to generally increase 

contact, monitoring and support for WIA customers to help make it more likely that customers 

will successfully complete services and attain performance goals. For example, guidance 

provided by Maine state WIA staff recommends that local programs try to “establish strong 
IV-19

personal bonds between the client and Career Center staff as a prelude to providing follow-up 

services and as a complement to training and job placement.”13 This guidance further advises 

programs to “assign case managers on whom participants can rely for guidance, good example, 

or friendship—staff with a ‘whatever it takes’ approach to keeping individuals at work and 

focused on advancement.” In the Alabama LWIA, the local MIS has a place for counselors to 

document the emotional state of customers, so that staff will know to follow up more regularly 

with customers who are despondent or depressed about being unemployed or laid off, since those 

customers may have more difficulty completing program goals or finding a job than others. Case 

managers in this same local area also reported that they try to work closest with customers who 

are judged least likely to make the LWIA’s earning goals to try and help those customers achieve 

better outcomes. 

One specific area where several local areas and states reported greater emphasis on case 

management is on providing greater support to customers to help them complete training. For 

example, in Mid-Willamette, WIA staff are now required to check-in with customers in training 

every 30 days to help prevent customers from dropping out. Similarly, customers receiving 

training are required to come in to the Montgomery One-Stop center monthly to review their 

progress with a case manager. Montgomery One-Stop WIA staff also check in weekly with local 

community college staff to monitor customer progress in training. 

Another performance-related goal of the intensified case management being provided by WIA 

adult and dislocated worker programs is to ensure that relationships with staff are strong before 

customers are exited. Several respondents noted that, without the development of strong 

relationships during program participation, customers are unlikely to communicate with staff 

during follow-up, thus making the collection of supplemental data or provision of reemployment or advancement services very difficult. 

Placement 

Another service delivery area where performance has had an impact is placement services. As a 

result of the performance system, respondents reported improving their programs’ capacity to 

effectively place applicants, limiting the number of customers who are placed in seasonal 

employment and attempting to discourage employment in certain industries and occupations that 

are not covered or are only partially covered by the UI system. Among these impacts, the first 

 

13 “Managing Performance: A CareerCenter Approach,” presented by Stephen Duval, Maine Department of Labor. 

Eastern Seaboard Apprenticeship Conference. June 17, 2004 Portland, ME. 
IV-20

two relate both to good performance and quality service delivery, while the latter is almost 

purely aimed at improving performance, even at the expense of customer wishes. 

Numerous respondents explained how “good initial placements” and connecting “customers to 

‘well-paying’ jobs”14 are important in achieving positive performance outcomes. Consequently, 

some of the local areas we visited for this round of site visits said that they have recently beefed 

up their placement services to improve performance. For example, the Mobile LWIA recently 

added full-time job developers to its staff. Respondents in Coastal Counties, Maine, also 

described how the local area how is planning to increase its employer outreach to improve 

placement. 

A few respondents said that the performance system has forced them to try to limit the number of 

customers placed in occupations or industries that are seasonal, such as agriculture or tourism. 

Instead, counselors try to find customers placements in permanent jobs so that customers are 

likely to be employed throughout the first three quarters after the exit quarter when performance 

is measured. Respondents said that they try to limit seasonal placements partly because of the 

performance system, but also because permanent jobs are better for customers. 

In a few other LWIAs, the performance system has resulted in WIA staff trying to discourage 

employment in certain industries or occupations that are not covered, or are only partially 

covered by UI, even when those occupations or industries are preferred by customers. For 

example, when customers come in to the WIA program in one local area and tell counselors they 

want to become self-employed medical transcriptionists, staff try to convince these customers to 

first find employment in an office setting to build up their experience. While this is generally 

good advice for the customer since self-employment is risky, it also helps the LWIA since these 

initial office jobs are likely to be covered by UI. Similarly, but purely to improve performance, 

another local area has decreased the number of placements it makes in industries where tipping is 

prevalent, such as casinos and restaurants, because tips are typically not reported to UI. 

Respondents from the same area also said that they somewhat discourage truck driving training 

because of the fact that truck drivers get paid a fee per haul that also doesn’t show up in UI and 

thus lessens the earnings that the area can document for the performance measures. 

Youth 

Based on our recent round of site visits, we also found that service design and delivery in WIA 

youth programs have also been impacted by the performance system. As with adult and 

 

14 Performance training provided by the North Carolina Division of Employment and Training, August 2005, 

pp.26-27. 
IV-21

dislocated worker programs, the biggest impact of the performance system appears to be on 

follow-up services, which have been intensified. Respondents also noted some impacts of the 

performance system on assessment, case management, in-program monitoring and incentives 

policies. Perhaps because many respondents are more focused on the new common measures, 

youth program respondents reported fewer impacts on service design and delivery from the 

original WIA measures than did adult and dislocated worker respondents. 

Follow-up 

One of the primary ways that services for youth have been affected by the WIA performance 

system is in the area of follow-up services. As noted above, this finding is supported by our 

recent survey of local areas that found that 77 percent of local areas reported increased follow-up 

costs due to the performance measures. Because follow-up services are a required element of the 

WIA youth program, local areas all have to provide some form of follow-up to youth for 12 

months after exit. However, several local areas said they provide more intensive follow-up than 

required because of its beneficial effect on performance outcomes. 

As with adults, in a majority of sites, follow-up consists primarily of the collection of data on 

outcomes obtained by youth. However, in several local areas we visited, youth programs have 

developed or are developing much more extensive follow-up systems. For example, in Cape 

Fear, North Carolina, the local youth contractor provides a number of workshops and retreats 

related to employment and retention for youth exiters and is trying to set up an alumni advisory 

group as a way to keep youth involved. The LWIA also provides food at the events as a further 

way of attracting exited youth. In the Idaho LWIA, one youth provider meets in person with all 

youth in follow-up once a month and calls youth every two weeks. Idaho LWIA staff also 

provide one-on-one advancement coaching to youth during follow-up, helping them with issues 

such as how to get a raise. 

One of the most extensive follow-up systems for youth we encountered is in Mid-Willamette, 

Oregon. First, the local area tries to make youth more likely to use follow-up services by making 

all in-program services available to youth after exit. Program staff also do not tell youth that 

they are “exited,” but rather inform them they have been “promoted,” to the placement phase of 

the program. Mid-Willamette’s youth program tried to use “retention specialists” in the same 

way that the LWIA’s adult and dislocated worker program does, but it found that retention 

specialists were unsuccessful in maintaining contact with youth. Respondents attributed the lack 

of success of retention specialists to the fact that youth are much less likely than adult customers 

to be responsive to staff with whom they don’t have a well-developed relationship. 

Consequently, most follow-up and retention services are now provided by regular case managers 

who work with youth from assessment through the end of follow-up. 
IV-22

Despite the emphasis placed on follow-up services for youth, many areas also noted the 

challenges of providing follow-up to youth, particularly out-of-school and older youth. For 

example, Cumberland County contrasted the ease of conducting follow-up with in-school youth, 

who are easy to find, with the difficulties they face in maintaining contact with out-of-school 

youth. Unfortunately, WIA youth staff often have to handle very large case loads, and with all of 

their other duties, respondents noted that despite their good intentions, follow-up services 

sometimes “fall to the bottom of their list” of tasks. 

Because of the time intensiveness of follow-up for youth and limited staff resources, youth 

providers in two local areas use triage systems for youth follow-up. These systems are design to 

help providers assess which youth are in need of more intensive follow-up so that staff can focus 

their attention on the neediest youth and provide only “light-touch” services to others, thus 

lessening the burden on case managers. These triage systems require case managers to assess 

exiting youth based on certain risk factors and then assign youth to particular follow-up services 

categories based on the number of risk factors youth present. For example, Mid-Willamette’s 

triage system includes three risk categories into which youth can be placed: low, moderate and 

high. According to this system, low risk youth are characterized by having stable families, good 

housing, access to transportation, adequate day care for their children, and no history of medical 

or substance abuse problems, while high risk youth are those with “significant problems in one 

or more” of those same areas. While youth exiters who fall in the “low risk” category receive 

only monthly contact from a case manager, high risk youth receive bi-monthly home visits, biweekly phone contact, monthly contact with the youth’s employer, and enrollment in peer 

support groups related to problems or employment assistance programs. 

Another local area has simply decided to limit youth enrollment to allow case managers time to 

provide the kinds of extensive follow-up services required to see better outcomes, including 

income advancement. 

Other Effects on Service Delivery 

The original performance measures have also had an impact on a few other types of services 

provided to youth. One service area that was mentioned by both state and local respondents is 

assessment. Several respondents mentioned that they now place great emphasis on the need for 

counselors to conduct thorough and accurate assessments of youth at program entry to correctly 

identify barriers and come up with training and supportive service plans that will allow youth to 

meet performance goals. 

Related to the need to provide good initial assessments, several respondents also talked about the 

increased importance they now place on case management. These respondents indicated that 
IV-23

they have found effective case management to be critical to making sure that youth are on track 

to achieve performance goals and are unlikely to drop out of the program or activities. 

In addition to this enhanced case management, some respondents also described other types of 

in-program monitoring that are necessary to make sure youth are likely to achieve goals. For 

example, as described above, DESI, a WIA youth services provider in Mobile, regularly 

monitors youth participants for drug use to ensure that youth will be able to pass employer drug 

tests. 

Incentives are another service strategy that at least two local programs use to encourage youth to 

stay in the program and achieve performance goals. While in Coastal Counties, Maine, these 

incentives are special events such as a t-shirt and logo contest, Mobile, Alabama has a formal 

program of cash incentives. Mobile’s incentive program involves cash payments of under $25 

up to a total of $300 per year for attainment of skill goals and credentials, perfect attendance or 

drug testing. Only a few youth ever earn the full $300. The LWIA is currently studying the 

impact of this incentive system and its effectiveness in improving outcomes, and the WIB is 

planning to develop a campaign in which local businesses would match the local area’s 

incentives. 

Finally, just as with adult and dislocated worker programs, the performance measures have 

caused local areas to focus on ensuring that youth are enrolled in training or other activities that 

lead to a credential and/or employment at good wages. For example, Coastal Counties, Maine 

has implemented a requirement that 30 percent of older youth training funds must be spent on 

OJT because of the local area’s belief that OJT leads to better performance outcomes. To make 

achievement of this OJT requirement possible, the local area is also undertaking increased 

employer outreach to convince more employers to agree to use OJT. 

Effect of Common Measures on Service Design and Delivery 

While most respondents do not expect the common measures to greatly affect adult and 

dislocated worker service design and delivery, many do expect them to have a significant effect 

on WIA youth program services. 15 Respondents in at least six local areas asserted that the youth 

measures are the biggest challenge of the common measures and the new measures will force 

 

15 Because the youth common measures are not scheduled to go into affect until PY 06, most respondents, 

particularly at the local level had limited understanding of these measures, particularly the Literacy and 

Numeracy Gains measure, which states are not even required to collect data on until PY 06. Alabama is the only 

state we visited that is actually collecting data on the Literacy and Numeracy Gains measure this program year. 
IV-24

them to re-design their youth programs and provide services that are more intensive and longterm, causing costs to rise and the number of youth served to decrease. Respondents’ specific 

expectations of how each of the new common measures is likely to affect youth program service 

delivery are discussed in this section, beginning with the Literacy and Numeracy gains measure. 

Because a number of youth program respondents had only limited knowledge of these new 

measures at the time of our site visits, these findings should be regarded as only preliminary. 

Literacy and Numeracy Gains 

The measure that was cited by most respondents as likely to require the biggest service design 

and delivery changes is the Literacy and Numeracy Gains measure. This measure requires basic 

skills deficient out-of school youth to increase one or more National Reporting System (NRS) 

educational functioning levels16 within a year of program entry. 

A number of respondents indicated that they are likely to need to make major changes to their 

basic skills remediation programming to be successful on the measure. They believe these 

changes will typically involve beefing up their basic skills services and making them more 

attractive to youth. For example, one local contractor commented that she is likely to subcontract with a formal basic skills remediation program, although that will also force her to cut 

back on other services. The LWIA youth program director from Mid-Willamette said that she 

expects that her contractors will need to build basic skills emphases into other program 

components such as community service. Mid-Willamette’s youth program is also planning on 

bringing in a literacy expert to assist the LWIA in developing more effective literacy strategies 

and providing training on literacy to local contractors and staff. It is also considering trying to 

enlist business people to act as basic skills tutors. 

In Idaho, local staff think the measure will force them to focus more heavily on tutoring services 

and registering youth who are in educational programs that will allow them top achieve success 

on this measure. However, they are concerned that providing tutoring is not the problem, but 

getting youth to make use of those tutoring services is the greater challenge. For example, as one 

case manager said, “You can provide the service, but they don’t go if they don’t want to go.” 

This case manager said the area has had this problem with youth under the original measures 

and has had to exit youth without a GED because these youth refuse to go to tutoring. 

 The state opted to begin implementation this year because state staff wanted locals to gain experience with the 

measure while outcomes from it still do not count. 

16 The NRS educational functioning levels were created by the U.S. Department of Education’s Division of Adult 

Education and Literacy for an accountability system for federally funded adult education programs under WIA 

Title II. 
IV-25

Other respondents commented that because their current basic skills services are geared toward 

helping youth earn GEDs or high school diplomas, they will have to re-design their programs to 

suit older youth who are basic skills deficient and thus subject to the measure, but have already 

attained their GED or diploma. Local respondents from one area also said they think it will be a 

challenge just to convince these youth to keep studying when a diploma or GED is all most 

employers require. 

Another major implementation and service design issue related to this measure is the testing it 

requires. Many states and locals noted a number of challenges and concerns related to these 

testing requirements. One of the most common concerns is that out-of-school youth will be 

unwilling to take required pre and post-tests. Respondents said that one of the reasons these 

youth have dropped out of school is that they do poorly on tests and therefore they hate to take 

them. Some respondents are particularly unhappy with recent guidance that programs will have 

to test even out-of-school youth who have already attained a GED or high school diploma. They 

feel that these youth will be particularly unwilling to be tested because they feel that they have 

already achieved their educational goal. Thus, they feel that many of these youth will probably 

choose not to enroll in the WIA program to avoid the required testing. 

A number of respondents thought that it would be even more difficult for programs to convince 

youth to come in and be re-tested at the end of a year. Indeed, Weld County staff said that they 

are afraid that the local area will not be able to meet performance standards related to this 

measure simply because of youth’s unwillingness to come in and be re-tested. Consequently, at 

the behest of local areas, the state of Colorado is petitioning USDOL to allow attainment of a 

GED/high school diploma as a qualifying post-test. Officials in Colorado argue that 

achievement of a GED or diploma is a much more useful standard than simply showing 

improvement on a test like the TABE. Because of their concerns about convincing youth to be 

re-tested, several respondents are planning to offer incentives to youth who are re-tested. 

Another design question raised by a local area manager in the Oregon Consortium is how often 

programs should optimally re-test youth. Although reliability concerns arise when youth are 

tested too often (as youth may become too familiar with some sections of the test), regular retesting would help programs to know how far youth have advanced toward their goal and how 

far they need to go. 

There were also a number of respondents who reported challenges related to the use of specific 

standardized tests. For example, because of the requirement to use only basic skills tests that are 

crosswalked directly with the Department of Education’s National Reporting System, some 

states and local areas reported challenges related to training staff on use of the new tests. 
IV-26

Another local respondent took issue with the entire measure because it does not include a 

computer component, which he thinks is essential for labor market success. 

One local area manager asserted that this measure would drive up WIA youth program costs. He 

said that this cost increase would be due to the cost of purchasing or, more importantly, 

administering more basic skills tests coupled with the added cost of providing more intensive 

basic skills services. 

One state performance staffer said she thought the exclusion of in-school-youth and youth who 

are not basic skills deficient makes the measure too complex. She thinks this complexity will 

make it much more difficult for local programs to operationalize the measure. She argued that a 

better and more simplified measure would hold all youth enrolled in a WIA youth program 

accountable for this measure. 

Only a handful of respondents in seem to be unconcerned about this measure and feel that they 

will have no problem succeeding on it. In one area, this was because local youth programs 

already require basic skills deficient, out-of-school youth to improve one grade level over a year 

period and so feel that they are already used to helping youth experience an increase similar to 

the one required by the measure. This area also feels that its basic skills services are 

appropriately intensive to succeed on the measure. 

Attainment of a Degree or Certificate 

A number of respondents also said that they expect the common measures Attainment of a 

Degree or Certificate measure to necessitate significant programmatic changes. They expect the 

biggest programmatic changes resulting from this measure to be related to the types of 

certificates that are countable under this measure. Under the original older youth Employment 

and Credential measure, states and local areas have the flexibility to establish locally recognized 

or developed credentials that can be issued for completion of a variety of activities, including 

work readiness skills, soft skills, CPR or first aid training. However, these types of certificates 

will not be allowed to count under the new Common Measure, which uses a strict definition of a 

certificate. 

Consequently, a number of respondents said that their programs will need to change to ensure 

that youth enrolled in an education program as defined by the measure are able to achieve either 

a degree or a certificate that is countable. Staff at one local program noted that, once this 

measure is implemented, they will be less likely to use scarce program resources to send youth to 

training programs such as CPR or first aid that don’t result in a countable certificate, even though 

these types of training are often needed to secure employment in a health-related field. Another 

local area that has developed an employer-approved soft skill assessment and training program is 
IV-27

planning to re-evaluate how this program fits in to its service design since the credential youth 

earn from this program will no longer be countable. A contractor in a third local area is seeking 

to become accredited by the state as a private career school to allow the certificates youth earn in 

its occupational training programs to count under this measure. 

Despite these adjustment challenges, some respondents welcome the stricter definition of a 

certificate under this new measure. Although they developed local credentials to meet the 

performance standards, some think that these credentials are basically worthless in the labor 

market, or, at best, are useful in only very narrow labor markets. However, other local 

respondents are sorry to lose the incentive to work with employers to develop locally recognized 

certificates. 

Placement in Employment or Education 

The only program change (mentioned by three respondents) regarding the new youth Placement 

in Employment or Education measure is that programs will have to keep youth, particularly 

younger youth, enrolled longer to ensure that youth achieve an acceptable placement. This is 

due to the fact that youth who exit while still in secondary school will not be excluded from this 

measure as they are now excluded from the younger youth retention measure. Currently, 

programs can exit youth who are still far away from graduation or placement in unsubsidized 

employment and, if those youth are still in secondary school, the program is not penalized. 

However, under the new Common Measure, programs would be penalized for youth who exit 

while still in secondary school who are not employed, in the military, or enrolled in postsecondary education, advanced training or occupational skills training. According to several 

respondents, this means that, essentially, programs will need to keep youth registered until they 

graduate. 

Effects of the Common Measures on Adult and Dislocated Worker Programs 

In contrast to the expected impacts on the WIA youth program, thus far, the common measures 

have had very little impact on service delivery in adult and dislocated worker programs. The 

only changes we found were small impacts on follow-up services in the two LWIAs in Oregon. 

Staff from these local areas said that as a result of the new retention measure requiring 

employment in the second as well as the third quarters after exit, staff have increased their 

follow-up efforts during second quarter after exit to ensure that supplemental data are being 

collected and services being offered during that time. 
IV-28

Managing Exit 

During this round of site visits, we found that nearly all local areas said they attempt to manage 

their exits to improve performance. This finding is supported by our recent survey of LWIAs 

that found that 81 percent of all local areas reported that the performance measures influence the 

decisions they make about when to exit customers, and is also similar to our findings from our 

first round of site visits. 

Exit management is seen by most respondents as critical to performance because, for all but one 

of both the original and common measures, customers are not reported in performance until they 

are exited from the program, and specific outcomes are not measured until the first, second, or 

third quarters after exit. 

In addition, in contrast to services and participation, exit management is much more closely 

related to the performance system than it is to providing good services to customers. This is 

because at present, exit is primarily an administrative or reporting distinction that does not have 

to impact services. Under current WIA regulations, programs are allowed to provide most of the 

same services that are available prior to exit—including limited amounts of training—during the 

post-exit follow-up period as well. For this reason, as noted above, Mid-Willamette’s WIA 

youth program does not even inform youth that they have been exited, but simply tells them they 

have been promoted to a new program phase. However, because many programs continue to be 

hesitant about providing substantial services after exit, some respondents said that in practice, 

exit continues to have important service delivery ramifications and careful decisions about exit 

are important to providing good services. 

The process used by many areas for determining when to exit customers is a complex one, 

requiring both front-line counselors and managers to assess whether customers are ready to attain 

the performance goals and to determine the best timing to promote better outcomes. In this 

section, we first describe the exit processes used by adult and dislocated worker programs and 

then by youth programs. We then discuss when programs typically exit customers and the 

effects of the common measures on these practices. 

Adult and Dislocated Worker Exit Processes 

Nearly all adult and dislocated worker programs that we visited have established some type of 

exit process in which counselors and their managers determine whether it is appropriate to exit a 

customer. Typically, local areas encourage counselors to consider both the goals and needs of 

customers as well as the requirements of the performance measurement system in making exit 

decisions. 
IV-29

Usually, the first step in the exiting process is for counselors to check in with customers 

regarding their employment status, such has whether they are employed and the quality and 

permanence of the jobs. Typically, this involves a call or an e-mail from a counselor to a 

customer, although in one area it is done through the mail. 

Once counselors have collected information about a customer’s employment status, they use that 

information to make a decision about exit. To make it more likely that these exiting decisions 

are appropriate for both customers and performance, many areas now require exiting decisions to 

be approved by supervisors, discussed with peers, or some combination of the two. For example, 

in Mid-Willamette, the counselor, retention specialist and One-Stop supervisor are all required to 

hold a joint staffing conference to discuss a customer’s exit and the transfer of his or her file 

from the in-program counselor to the retention specialist. In Mobile, before a customer can be 

exited, both a staff discussion about exiting and review by a supervisor—including both a file 

review and an oral interview—are required. 

In some places, exit reviews are coupled with data validation-related file reviews. To fulfill new 

data validation requirements about required documentation in customer files, some areas now 

require that all files must be reviewed at certain key points. In Cape Fear and the Oregon 

Consortium, for example, this type of review occurs at least at exit. Because of the important 

performance implications of exiting, a discussion of performance issues is also included in these 

data validation exit reviews. Because of the primacy of data issues to these sessions, MIS staff 

are typically the prime reviewers rather than staff supervisors. For example, in Cape Fear, the 

MIS manager serves as the “exit gatekeeper” and meets with all counselors to go over the files of 

exiting customers and discuss performance issues. 

In other areas, there is less rigorous review of exits, and supervisors only get involved in more 

complicated cases or simply receive notification of exits via e-mail. For example, at MTC 

Works in the Oregon Consortium, respondents said that counselors only discuss “difficult” 

exits—such as when the counselor has lost contact with a customer—with supervisors. 

In some areas, counselors have to complete “exit checklists” before exiting customers. For 

example, in Mid-Willamette, an “Exit Consideration Checklist” helps adult and dislocated 

worker counselors carefully consider the performance implications of exiting a customer. The 

checklist includes questions about whether it is the end of the quarter, whether the customer is 

employed, the quality of the job (e.g. temporary or permanent, how well the customer likes it, 

etc.), length of employment, salary or wage, as well as other questions related to performance. 

There is even a place for the employer’s opinion of the customer’s job performance. This 
IV-30

information can then be used to make a determination of whether to provide further services or to 

exit the customer, based on their likelihood for achieving positive performance results. 

In its “Performance Management Guide,” Maine also has exit checklists for adult and dislocated 

worker programs that staff can use to help make exiting decisions. These checklists query staff 

about whether the customer is being exited because he or she is employed, and, if not, identify 

whether every service option has been exhausted before an exit is taken. The checklists also 

document whether contact information has been collected and follow-up services discussed. The 

checklist also includes a series of questions aimed at helping staff to understand what measures 

customers will be counted in and assessing how likely it is that the customer will attain all of the 

performance outcome targets. 

Because several of these exit review processes are extremely time intensive, some counselors 

expressed frustration about having to complete all of these steps. These respondents also 

questioned the value of such resource-intensive processes, especially for routine exits made by 

experienced staff. 

Youth Exit Processes 

Exit processes for youth are fairly similar to those for adults and dislocated workers. For 

example, youth counselors also typically check with youth about their service and placement 

status, typically by phone or e-mail although occasionally in person, just prior to exit. For 

example, in Cape Fear, youth case managers are required to have an in-person exit interview 

with youth to review the “youth Toolkit” that includes the goals set at program entry and discuss 

exit. 

Following the collection of information from youth, counselors again use that information to 

make exit decisions. As it has done for adults and dislocated workers, Maine has provided 

guidance in the form of exit checklists to help youth counselors make decisions about when to 

exit older and younger youth. Maine’s older youth exit checklist is very similar to the adult and 

dislocated worker checklists described above, while the younger youth exit checklist asks about 

skill attainment goals, diploma or GED attainment, and whether youth are likely to remain in one 

of the five countable placements in the third quarter after exit. 

As with adult and dislocated worker programs, some areas require or encourage supervisor or 

peer reviews of youth exits. However, perhaps because services are usually provided by 

contracted providers rather than LWIA staff, exit reviews for youth appear to be somewhat less 

common or rigorous than for adults and dislocated workers. For example, although the 

Montgomery One-Stop youth coordinator holds discussions with contracted service providers 
IV-31

about the best strategies for exiting, she allows them to make actual exit decisions on their own. 

In Mid-Willamette, although the LWIA’s preferred exiting model is to have case managers 

discuss their possible exits with colleagues at weekly staff meetings, this is not absolutely 

required and is practiced by only some providers. 

When Participants are Typically Exited 

The main objective in the exit processes described above is to determine when it is best to exit 

participants from the WIA program. This decision is critical in terms of performance, because 

an exit triggers inclusion in performance reporting and sets individuals on a timeline for all the 

measures. Programs do not want to begin the measurement process for customers who not yet 

ready to attain performance goals, but do need to exit at least some customers in every quarter in 

order to have some base rate for their performance. 

Because of the importance for performance of exit timing, nearly all of the local areas in this 

round of site visits have developed preferred practices and guidance for line staff about when to 

exit customers. The most common practice among local programs is to exit participants only 

when they are employed or in some other countable placement. In addition to helping programs 

with performance, respondents in nearly every local area also said that this practice is in the best 

interests of participants. They explained that participants who are not in a positive placement 

such as employment, post-secondary education or advanced training, are usually in need of 

additional services and it is easier to serve these participants while they are still enrolled. Thus, 

for example, local staff will provide unemployed adults and dislocated workers with additional 

one-on-one counseling, intensive job search, resume assistance, and job readiness assistance. 

For youth who are not in a positive placement, programs may also provide employment-related 

assistance, additional basic skills remediation, or assistance enrolling in post-secondary 

education or advanced training. Counselors may also choose to enroll participants in need of 

services that are not available from WIA into partner programs, and delay exit until completion 

of those partner services. 

To make it more likely that individuals will maintain these positive placements into the first, 

second and third quarters after exit (as required for positive performance in many measures), 

some areas also require that counselors ensure that the placement is stable and permanent. For 

example, the Montgomery One-Stop center teaches staff to differentiate between temporary jobs 

that participants are using to tide themselves over and permanent jobs. In Region L, youth 

counselors assess whether jobs are only temporary or “dead-end,” and, if the latter, will keep 

youth enrolled to see if they can help find a better placement. 
IV-32

For the same reason, many other LWIAs go further and require counselors to hold participants 

for a period beyond employment or enrollment in a positive placement. For example, several 

local areas have their counselors “hold” all exits until the end of the exit quarter, so that in only a 

day or two, exited participants are in their first quarter after exit when their placement will be 

counted as a positive in the performance calculations. In a few other areas, counselors are 

instructed to delay exit for as long as six months by providing fairly light-touch services 

designed to ensure that participants’ jobs are stable.17 Some of these local areas will suspend 

these delaying practices and allow immediate exits for participants who appear stable and seem 

to be in no danger of losing their jobs. 

Many WIA programs also wait to exit customers until participants have achieved their 

credentials. For example, some youth programs noted waiting to exit younger youth until 

graduation, or at least until youth are within one quarter of earning their diploma or GED, since 

programs can get credit on the original Younger Youth Diploma or Equivalent Attainment 

measure for diplomas or GEDs earned within the first quarter after exit. 

Finally, in at least one local area, counselors who have many participants who become employed 

in seasonal industries try to time exits to maximize participant employment during key 

performance measurement periods. For example, counselors with participants employed in 

seasonal agriculture might try to exit these participants at the end of December so that they are 

employed only briefly at the end of the first quarter after exit (i.e., in March), but employment 

during the two quarters measured for earnings are maximized. These counselors said they would 

only do this if it would not negatively affect the participant. 

These practices involving delaying exits, particularly that of holding customers until the end of 

the quarter and timing the exits of customers employed in seasonal industries, are clearly aimed 

primarily at achieving good performance, since WIA regulations clearly allow exited customers 

with access to most of the same services they can receive while still enrolled. However, 

counselors in one local area asserted that delaying exits until customers are in stable placements 

and have achieved most of their goals is good for participants because it ensures that they have 

all the tools and services they need before being exited. One reason for the continued 

significance of exits, despite the flexibility in WIA regulations, is that many programs continue 

to be cautious about allowing counselors to provide substantial services to customers after exit. 

 

17 We investigated with each local area whether there was any evidence of providing merely “token” services to 

extend the exit date beyond what might be considered their date of last “real” service. No respondents indicated 

they were doing this. 
IV-33

Effect of Common Measures on Exit 

We encountered a great deal of confusion related to the change in the definition of exit 

implemented as part of common measures. Consequently, few respondents were clear on how 

this change in definition might affect their exit management strategies, particularly the common 

practice of “holding” exits until the end of the quarter. In only two local areas did staff seem to 

understand that the change to the new definition would make managing the timing of exits 

somewhat more complex since LWIAs would no longer be able to “hard exit” individuals at a 

certain time. However, many other respondents were concerned about the changes and wanted 

to request additional training on this issue. 

Because of the difficulty many local staff have in understanding the new exit definition, one state 

is planning to allow locals to continue taking hard exits on the date of a participant’s last service, 

even though the exit will not officially be final until 90 days later. This state said it was doing 

this even though it would implement the federal requirement at the state level because it did not 

want to add another layer of confusion for counselors by forcing the federal definition on locals. 

They said that before they made this decision, counselors had already begun complaining about 

the loss of control they felt due to the fact that the federal definition does not allow exits to 

become final until 90 days after the last service. 

Finally, DOL’s clarification that calling a participant to collect outcome data or offer additional 

services is not itself a service has also had an effect on some local areas. For example, one state 

expects this clarification will force some areas to exit numerous participants much sooner than 

they might have done prior to the common measures. 

Conclusion 

This chapter has reviewed the impact of both the original WIA measures and the new common 

measures on participation, service design and delivery, and exit. Overall, we found that 

programs are much more knowledgeable about the WIA performance system—particularly the 

original 17 measures—than in the past. Consequently we found many more instances where 

programs made changes to their management of program entry, service design and delivery and 

exit aimed at both improving their performance system outcomes and improving customer 

success. We also found more examples—particularly related to the earnings measures—where 

programs attempt to “game” the performance system and carry out actions or processes that 

either have no impact on customer success or even limit customer choice or customer access to 

timely services. Finally, we found that the implementation of the common measures was causing 

few changes to the management of participation, service design and delivery or exit among adult 
IV-34

and dislocated worker programs, but are expected to cause major changes to the management of 

participation and service design and delivery among WIA youth programs. 

Regarding the management of participation, we found widespread evidence that programs clearly 

attempt to manage both who becomes a participant and when individuals become participants. 

Programs typically try to control when individuals become participants both by stretching out the 

services and interactions with staff that customers can have before they become participants, and 

by referring customers to partner services prior to enrollment. The main reasons respondents 

gave for delaying participation include changing the period measured for pre-program wages, 

giving individuals time to gain skills outside of WIA that are necessary for success on the 

performance measures, and allowing time for more effective screening. Of these three reasons, 

the one that may be considered most directly “gaming the system” and possibly harmful to 

participants, is changing the period measured for pre-program wages, because this practice could 

make it more difficult for customers to complete intensive or training services. Interestingly, we 

found that the common measures are expected to have almost no impact on how programs 

manage participation. 

In addition to managing when individuals become participants, we also found that programs also 

are somewhat selective in determining who becomes a participant. Among adult and dislocated 

worker programs, we found that many programs screen potential customers for their motivation 

and commitment, skills and qualifications related to being successful in completing training and 

finding employment, and pre-program wages. For youth programs, many areas similarly screen 

youth on factors such as motivation, ability of the program to effectively serve them, and the 

likelihood that youth will succeed in the program and achieve performance goals. Respondents 

asserted that the reason for these screening programs is not only to ensure that their programs 

perform well, but also to ensure that customers are served by the most appropriate programs, 

counselors have the information to effectively plan for services, and that scarce program 

resources are used as effectively as possible. 

Although we found these screening mechanisms to be widespread, we also found that few 

programs are particularly selective. Rather than always trying to exclude individuals, program 

staff said that they typically try to balance individuals they think will perform poorly with those 

they estimate will do better. Consequently, it does not appear that these LWIAs are making 

many decisions about who to serve purely for performance reasons, although on the few 

occasions programs exclude customers with high pre-program wages, this may well be seen by 

outsiders as an effort to “game” the system in a way that can prevents these customers from 

accessing services from which they may well benefit. 
IV-35

The common measures are not expected to have much effect on adult and dislocated worker 

program management of who is served, but many respondents said that they expect them to have 

a significant effect on youth participation, forcing youth programs to be much more selective 

about who they serve. They attribute this primarily to the new Literacy and Numeracy Gains and 

Attainment of a Degree or Certificate measures, which they think will be too difficult for youth 

with significant barriers, such as out-of-school youth or disabled youth. They also expect to 

serve even fewer 14 and 15-year old youth due to both the Placement in Employment or 

Education measure and Attainment of a Degree or Certificate measure. 

According to respondents, the performance system has had a major impact on service design and 

delivery in all WIA Title IB program streams: adult, dislocated worker and youth. For example, 

many WIA programs have increased the amount and intensity of their follow-up and case 

management services. The performance system has also had a strong impact on training services 

provided by adult and dislocated worker programs. For example, a number of local areas 

reported investing in higher cost training programs and encouraging OJT. Some adult and 

dislocated worker respondents also noted that the performance system has pushed them to beef 

up their placement services. A number of youth programs similarly reported making changes to 

beef up their services, especially in the area of follow-up services. Because of the higher cost of 

many of these service changes, some programs reported being able to serve fewer participants. 

While nearly all of these service design and delivery changes both help participants be more 

successful and programs to attain good performance, we did find one example where adult and 

dislocated worker programs carry out processes or actions that are related only to success on the 

performance measures. Regarding placement and training services, respondents from at least 

two local areas said that they discourage customers from going into training or finding 

employment in industries or occupations that are not covered, or are only partially covered, by 

their state’s UI system. Although these jobs may be excellent opportunities otherwise, the fact 

that they will not be documented in the primary data source for WIA outcomes leads to 

avoidance of them in these areas. 

For adult and dislocated worker programs, these service delivery changes are expected to 

continue under the common measures with few changes. However, for WIA youth programs, a 

number of respondents said they expected the common measures to cause several significant 

programmatic changes. Most of these changes are expected to be related to improving basic 

skills programs and implementing the testing requirements of the Literacy and Numeracy Gains 

Measure. Some respondents also said they expected to have to change their programs due to the 

Attainment of a Degree or Certificate measure. Respondents generally agreed that these service 

changes will result in more intensive and costly services, forcing programs to cut back on the 
IV-36

number of youth they are able to serve. Some respondents also noted that the strict definition of 

a certificate under the new common measures may decrease the current practice by some local 

areas of granting youth credentials that are not widely recognized as having labor market value 

and may be awarded solely for the purpose of attaining higher performance. 

Finally, we found that respondents in all but one state have developed processes to manage exits 

to improve performance. Usually, these processes involve the collection of information from 

customers, exit decision-making by counselors based on local policies or guidance, and review 

of exit decisions by peers and supervisors. The most common practices and policies regarding 

when to exit customers involve delaying exits until customers are in stable placements and have 

achieved required credentials. 

Overall, we found that exit management is much more closely tied to the performance system 

than the management of participation or most service design and delivery activities. We also 

found two fairly clear examples of “gaming” related to exit management. The most common of 

these, practiced in at least five local areas, is to “hold” all exits until the end of the quarter. 

Another example practiced by at least one other local area is to try and time exits for customers 

who become employed in seasonal industries such as agriculture or tourism. While neither of 

these examples of exit “gaming” is obviously harmful to customers, they also seem to have 

nothing to do with customer success. 

Because our visits occurred shortly after the implementation of the common measures, few 

respondents had a clear enough understanding of the new definition of exit to understand firmly 

how it would affect their exit management. However, respondents from two local areas did 

comment that the new definition will make exit management more complicated, and one state 

indicated that it will not change the local definition to make it easier for line staff to understand. 

In particular, exit practices related to “gaming,” such as holding customers for exit at the end of a 

quarter, may become more difficult and, thus, this definitional change may greatly reduce the 

ability of programs to manipulate exit solely for performance reasons. 

Although it is clear that programs have made many changes to their management of 

participation, service design and delivery and exit that are aimed at ensuring success on the 

performance measures, it is likely too early to tell what impact these changes have had on 

program performance. This is because many of these changes have been implemented in only 

the last year or two. It is clear, though, that programs are highly cognizant of both the original 

measures and the newly implemented common measures, and work to the best of their ability to 

coordinate participation, service design and delivery, exits, and follow-up services to promote 

strong performance on these measures. Although it is likely too early to determine specifically 
IV-37

how the common measures will impact these efforts, it seems clear that programs will continue 

to attempt to provide high-quality services to customers while promoting strong performance as 

captured by the performance measurement system. 
This page intentionally left blank.

Insert blank page here when making double-sided copies
V-1

V. SUMMARY 

This report summarizes our findings from our three-year evaluation of the WIA performance 

measurement system, focusing heavily on our final round of site visits conducted in PY 2005. 

The results thus represent cross-site analyses from the seven states and eleven local areas, and 

describe the general themes and trends discovered as part of our data collection. In addition, we 

include a summary of how these more recent findings are consistent with or differ from results 

we described as part of our Survey Report for this evaluation and those obtained in our initial site 

visits and detailed in our Interim Report for this evaluation. Included in this analysis is a 

discussion of the elements of the performance measurement system, how those elements are 

communicated to those who operate the system, and service design and delivery under WIA 

including how this design and delivery is impacted by the performance measurement system. 

In chapter II, we described the elements of the performance accountability system, including the 

negotiation process—both between states and DOL, as well as between states and their local 

areas—that took place to establish performance levels, the specific features of the accountability 

systems, operational definitions established to help operate the system, and information on the 

data collection and reporting systems. The states and local areas we visited for the final report 

were generally more savvy in using these elements to promote and achieve good performance, 

and to design and implement accountability goals of their own. 

While the negotiation of performance targets for the original measures continues to be fraught 

with challenges, by and large states have been able to meet measures; DOL has often responded 

by negotiating much higher goals when states perform exceptionally, and, to a much lesser 

extent, negotiating lower goals when states fail. 

Additionally, states and local areas often establish additional goals for their WIA programs, 

workforce development systems, or other partner systems. They may also choose to develop 

goals for individual Career Centers, providers, or contractors. All the sites in our sample had 

some additional goals they seek to accomplish, although formal additional measures were far 

less common in this round than in the first round of visits. 
V-2

Perhaps more important than explicit goals are the reports the sites use to monitor performance. 

In contrast to our earlier visits, the sites in our final sample are making thorough use of their MIS 

for these purposes, perhaps because even sites that bought or developed new MIS after WIA 

passed have now had their systems in place long enough to learn and take advantage of the 

reporting and management capacity. Sites commonly make use of post-exit and in-program 

reports to track participant progress toward intermediate and outcome goals through follow-up 

calls and customer self-report, and also by using UI wage records and supplemental sources of 

data. Several sites also use some type of soft exit report to prevent customers from exiting the 

program before they have achieved their goals and are ready to attain positive outcomes. 

Sanctions and incentives are commonly used to some degree, though how they are applied varies 

significantly. Incentives are used both to provide additional funds for areas that exceed their 

performance targets and to fund other state initiatives. Sanctions tend to be focused on 

preventive measures so that states provide technical assistance to those areas struggling to meet 

their targets, though they do maintain that local areas can lose a portion of their subsequent 

allocation with repeated failures. 

Finally, the concept of continuous improvement remains somewhat vague for most states and 

local areas in our sample, but most use their MIS to monitor progress toward achieving their 

performance goals, including both the performance measures as well as the numbers and types of 

customers served. 

Chapter III described how the performance system has been communicated to various levels of 

the system, including partners and contractors or service providers. This chapter focused on the 

extent to which the performance-measurement system has been an impediment or incentive to 

establishing partnerships with agencies beyond WIA, and how the system has influenced those 

partnerships that have been established. Additionally, we discussed the extent to which 

contractors and service providers have been made aware of the performance-measurement 

system, and whether their performance or activities have been explicitly tied to this system. 

One-Stop design reflects the influence of the WIA accountability system in that most of the local 

area sites carefully insulate core services from the effects of the accountability system. Four of 

the sites rely entirely on Wagner-Peyser staff to provide core services and, in the other seven 

sites, these services are configured to ensure that WIA staff provide limited levels of staff 

assistance so that they do not trigger WIA registration. 

Each of the state Wagner-Peyser programs is well on its way to adopting the common measures, 

but no state or local area indicated that they were making any significant change in the way they 

operated the program in the field. Thus, none expected any change in the way core services 
V-3

would be delivered or the relationship to WIA. This pattern holds, regardless of whether core 

services are delivered by ES alone or by ES in partnership with WIA. 

Front-line WIA adult and dislocated worker staff generally understand the WIA performance 

measurement system quite well, and nearly all front-line respondents in this sample were aware 

of the overall importance of obtaining good performance and know a good deal about the 

accountability system and its role. In general, the level of understanding of the common 

measures follows that of the WIA accountability system in general. The staff in those areas who 

best understand general WIA accountability also know the most about the common measures. 

Seven of the eleven LWIAs carry out the WIA adult/dislocated worker program with in-house 

staff. In the remaining four sites, the contract providers are all long-standing deliverers of 

WIA—and JTPA before that—services. Thus, the issue of communicating WIA performance 

requirements to front-line staff is essentially a matter of staff training. The youth program differs 

structurally from its adult and dislocated worker counterpart in that most of the services are 

contracted out to a variety of school districts, community-based organizations, and even a few 

for-profit firms. In nearly all the second-round sites, the youth program is conducted outside the 

One-Stop system. As a consequence, conveying accountability for youth programs is a twostage process, including the procurement system and training. Although procurement offers 

LWIAs the theoretical advantage of an enforceable relationship based on the initial contract for 

services, few of the areas we visited can rigorously enforce provisions, due to normal delays in 

data availability, limited competitors for services, especially in rural environments, and 

longstanding collaborative relationships with the vendors. As a result, by default training, as in 

the adult program, has been the primary means for conveying performance goals to the youth 

providers. Interestingly, however, only about half of the sites we visited regularly provide any 

form of training to their youth contractors on performance accountability. 

In Chapter IV, we detailed the effects of the performance measurement system on the 

participants and services under WIA, describing the degree to which the customers who are 

being served under WIA, at least in the states and local areas we visited, are impacted by the 

performance measurement system, including both the original WIA measures and the new 

common measures. Overall, we found that programs are much more knowledgeable about the 

WIA performance system—particularly the original 17 measures—than in our earlier visits. 

Consequently we found many more instances where programs made changes to their 

management of program entry, service design and delivery and exit aimed at both improving 

their performance system outcomes and improving customer success. We also found more 

examples—particularly related to the earnings measures—where programs attempt to “game” the 

performance system and carry out actions or processes that either have no impact on customer 
V-4

success or even limit customer choice or customer access to timely services. Finally, we found 

that the implementation of the common measures has caused few changes to the management of 

participation, service design and delivery or exit among adult and dislocated worker programs, 

but are expected to cause major changes to these aspects of WIA youth programs. 

Additionally, we found widespread evidence that programs clearly attempt to manage both who

and when individuals become participants. Programs typically try to control when individuals 

become participants both by stretching out the services and interactions with staff that customers 

can have before they become participants, and by referring customers to partner services prior to 

enrollment. The primary purposes for delaying participation include changing the period 

measured for pre-program wages, giving individuals time to gain skills outside of WIA, and 

allowing time for more effective screening. Interestingly, we found that the common measures 

are expected to have almost no impact on how programs manage participation. 

In addition to managing when individuals become participants, we also found that programs also 

are somewhat selective in determining who becomes a participant. Among adult and dislocated 

worker programs, we found that many programs screen potential customers for their motivation 

and commitment, skills and qualifications related to being successful in completing training and 

finding employment, and pre-program wages. For youth programs, many areas similarly screen 

youth on factors such as motivation, ability of the program to effectively serve them, and the 

likelihood that youth will succeed in the program and achieve performance goals. 

Although we found these screening mechanisms to be widespread, we also found that few 

programs are particularly selective. Rather than always trying to exclude individuals, program 

staff said that they typically try to balance individuals they think will perform poorly with those 

they estimate will do better. Consequently, it does not appear that these LWIAs are making 

many decisions about who to serve purely for performance reasons, although on the few 

occasions programs exclude customers with high pre-program wages, this is an example of 

“gaming” that prevents these customers from accessing services from which they may benefit. 

The common measures are not expected to have much effect on adult and dislocated worker 

program management of who is served, but many respondents said that they expect them to have 

a significant effect on youth participation, forcing youth programs to be much more selective 

about who they serve. They attribute this primarily to the new Literacy and Numeracy Gains and 

Attainment of a Degree or Certificate measures, which they think will be too difficult for youth 

with significant barriers, such as out-of-school youth or disabled youth. They also expect to 

serve even fewer 14 and 15-year old youth due to both the Placement in Employment or 

Education measure and Attainment of a Degree or Certificate measure. 
V-5

According to respondents, the performance system has had a major impact on service design and 

delivery in all WIA Title IB program streams. For example, many WIA programs have 

increased the amount and intensity of their follow-up and case management services. The 

performance system has also had a strong impact on training services provided by adult and 

dislocated worker programs. For example, a number of local areas reported investing in higher 

cost training programs and encouraging OJT. Some adult and dislocated worker respondents 

also noted that the performance system has pushed them to beef up their placement services. A 

number of youth programs similarly reported making changes to beef up their services, 

especially in the area of follow-up services. As a consequence of the higher cost of many of 

these service changes, some programs reported being able to serve fewer participants. 

For adult and dislocated worker programs, these service delivery changes are expected to 

continue under the common measures with few changes. However, for WIA youth programs, a 

number of respondents said they expected the common measures to cause several significant 

programmatic changes. Most of these changes are expected to be related to improving basic 

skills programs and implementing the testing requirements of the Literacy and Numeracy Gains 

Measure. Respondents generally agreed that these service changes will result in more intensive 

and costly services, forcing programs to cut back on the number of youth they are able to serve. 

Finally, we found that respondents in all but one state have developed processes to manage exits 

to improve performance. Usually, these processes involve the collection of information from 

customers, exit decision-making by counselors based on local policies or guidance, and review 

of exit decisions by peers and supervisors. The most common practices and policies regarding 

when to exit customers involve delaying exits until customers are in stable placements or have 

achieved required credentials. 

Although it is clear that programs have made many changes to their management of 

participation, service design and delivery and exit that are aimed at ensuring success on the 

performance measures, it is likely too early to tell what impact these changes have had on 

program performance. This is because many of these changes have been implemented in only 

the last year or two. It is clear, though, that programs are highly cognizant of both the original 

measures and the newly implemented common measures, and work to the best of their ability to 

coordinate participation, service design and delivery, exits, and follow-up services to promote 

strong performance on these measures. Although it is likely too early to determine specifically 

how the common measures will impact these efforts, it seems clear that programs will continue 

to attempt to provide high-quality services to customers while promoting strong performance as 

captured by the performance measurement system.