Workforce Investment Act: Improvements Needed in Performance Measures to Provide a More Accurate Picture of WIA's Effectiveness
Published: 2002-02-01T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Congress passed the Workforce Investment Act (WIA) in 1998 to bring most federally funded employment and training services into a single, one one-stop center system. GAO assessed three programs that provide service through this system.
section_title: What GAO Found
paragraphs: States and localities have begun to implement the new performance measurement system for the three WIA-funded programs but report several challenges. States had to change the way they collected and reported performance data. They also faced challenges in implementing these measures due to their complexity and the resource demands created by new measures. Some developed new procedures to obtain access to sensitive records. The performance levels are of particular concern to state and local officials because failure to meet them can result in financial sanctions. As a result, states may be choosing to serve only those job seekers who are most likely to be successful. Even when fully implemented, performance measures may not provide a true picture of WIA-funded program performance because data are neither comparable across states nor timely. The measures include many of the indicators relevant to an employment and training program, such as getting and keeping jobs and increasing wages and skills. Although measures exist to gauge the performance of the three WIA-funded programs, there are no measures to gauge the performance of the one-stop system as a whole. At least 17 programs provide services through the one-stop system, and most have their own performance measures. Although these performance measures may be used for assessing outcomes for individual programs, they cannot be used to measure the success of the overall system.

section_title: Letter
paragraphs: The Congress passed the Workforce Investment Act (WIA) in 1998 to begin  unifying a fragmented employment and training system and to better serve  job seekers and employers. To create a more comprehensive workforce  investment system, WIA requires states and localities to bring together  most federally funded employment and training services into a single  system, called the one-stop center system. Seventeen programs across  four federal agencies—programs such as the Employment Service  (Wagner-Peyser) and Adult Education and Literacy—must provide  services through this one-stop system. Three of these programs, whose  funding is authorized by WIA under Title I to provide services to adults,  dislocated workers, and youth, replace those previously funded under the  Job Training Partnership Act (JTPA). These three new WIA programs,  authorized at about $3.7 billion in fiscal year 2001, have performance  measures established under WIA that states and localities must track in  order to demonstrate the effectiveness of the programs. These  performance measures gauge program results in the areas of job  placement, employment retention, and earnings change, as well as skill  attainment and customer satisfaction. States are held accountable by the  U.S. Department of Labor for their performance in these areas. If they fail  to meet their expected performance levels, they may suffer financial  sanctions; if they meet or exceed their levels, they may be eligible to  receive additional funds, called incentive grants.  In order to make  objective decisions about which states receive incentives and sanctions,  WIA seeks to ensure that states collect and report comparable  performance data on all participants.
In an effort to assess the effectiveness of WIA’s performance measures  and whether they will yield useful information that can clearly  demonstrate performance under WIA, we assessed (1) the progress states  and localities have made and the issues they have faced in implementing  performance measures for the three WIA-funded programs; (2) how useful  the WIA performance measures are in accurately gauging the performance  of the three WIA-funded programs; and (3) beyond gauging the  performance of the three WIA-funded programs, how well the  performance of the one-stop system is being measured.
Our review is based on a survey of WIA program administrators in all 50  states; visits to Florida, Indiana, Missouri, Oregon, and Pennsylvania,  where we interviewed state and local workforce investment officials;  interviews with Labor officials and with national associations representing  state and local workforce development officials; and a review of relevant  documents. We selected the five states for site visits based on a variety of  factors, including their experience using unemployment insurance (UI)  wage record data to measure outcomes, the status of their data collection  systems, and their progress in developing additional measures for state  and local use. Our review focused on measures for Title I Adult, Dislocated  Worker, and Older Youth programs. We did not review the performance  measures for younger youth (age 14-18). We conducted our work between  December 2000 and August 2001 in accordance with generally accepted  government auditing standards.
subsections: section_title: Background
paragraphs: Labor required states to implement major provisions of WIA Title I by July  1, 2000, although some states began implementing provisions of WIA as  early as July 1999. Services provided under WIA represent a marked  change from those provided under the previous program, allowing for a  greater array of services to the general public. WIA is designed to provide  for greater accountability over what existed previously: it established new  performance measures and a new requirement to use UI data to track and  report on the achievements of the three WIA-funded programs. WIA also  requires that many federal programs work together to provide  employment and training services through the one-stop system.
subsections: section_title: WIA-Funded Services Represent a Change from those Funded under JTPA
paragraphs: Program services provided under WIA represent a marked change from  those provided under JTPA. When WIA was enacted in 1998, it replaced  the JTPA programs for economically disadvantaged adults and youth and  for dislocated workers with three new programs—WIA Adult, Dislocated  Worker, and Youth—that provide a broader range of services to the  general public, no longer using income to determine eligibility for all  program services. The newly authorized WIA programs no longer focus  exclusively on training but provide for three tiers, or levels, of service for  adults and dislocated workers: core, intensive, and training. Core services  include basic services such as job searches and labor market information.  These activities may be self-service or require some staff assistance.  Intensive services include such activities as comprehensive assessment  and case management—activities that require greater staff involvement.  Training services include such activities as occupational skills or on-the- job training. These tiers of WIA-funded services are provided sequentially.  That is, in order to receive intensive services, job seekers must first  receive at least one core service; to receive training services, a job seeker  must first receive at least one core service and then at least one intensive  service. Key to moving from core to a higher level of services is that the  services are needed to help job seekers become self-sufficient. Labor’s  guidance provides for monitoring and tracking to begin when job seekers  receive core services that require significant staff assistance. Job seekers  who receive core services that are self-service in nature are not included  in the performance measures.
section_title: WIA Performance Measures Are Designed to Increase Accountability for Three WIA-Funded Programs
paragraphs: WIA is designed to provide for greater accountability than the  accountability provided for under JTPA. It does so by establishing new  performance measures and a new requirement to use UI data to track and  report on the achievements of the three WIA-funded programs. According  to Labor, performance data collected from the states in support of the  measures are intended to be comparable across states in order to maintain  objectivity in determining incentives and sanctions. They are also intended  to provide information to support Labor’s performance goals under the  Government Performance and Results Act (GPRA) and for program  evaluation. Some of the measures that relate to adults, dislocated workers,  and older youth are similar to those used under JTPA, including job  placement, job retention, and wage gains or replacement. Attainment of a  credential—a degree or certification of skills or training completed—and  customer satisfaction for both job seekers and employers are new under  WIA. (See table 1 for a complete list of the WIA performance measures and  appendix I for a more complete explanation of the performance measures  discussed in this report.)
In contrast to JTPA, for which data on outcomes were obtained through  follow-ups with job seekers, WIA requires states to use UI wage records to  track outcomes. According to Labor’s guidance, if a program participant  does not appear in the UI wage records, states may use supplemental data  sources, such as follow-ups with participants and employers, to track  entered employment, retention, and credential attainment. However, only  UI wage records may be used to calculate earnings change and  replacement.
Unlike JTPA, which established expected performance levels using a  computer model, WIA requires states to negotiate with Labor to establish  expected performance levels for each measure. States, in turn, must  negotiate performance levels with each local area. The law requires that  these negotiations take into account differences in economic conditions,  participant characteristics, and services provided. To derive equitable  performance levels, Labor and the states use historical data to develop  their estimates of expected performance levels. These estimates provide  the basis for negotiations.
WIA holds states accountable for achieving their performance levels by  tying those levels to financial sanctions and incentive funding. States that  meet their performance levels under WIA are eligible to receive incentive  grants that may generally range from $750,000 to $3 million. To be eligible  for an incentive grant, states must also meet levels established under the  Department of Education’s Vocational Education (Perkins Act) and Adult  Education and Literacy programs. States that do not meet their  performance levels under WIA are subject to sanctions.If a state fails to  meet its performance levels for 1 year, Labor provides technical  assistance, if requested. If a state fails to meet its performance levels for 2  consecutive years, it may be subject to up to a 5-percent reduction in its  annual WIA formula grant. Under JTPA, the most stringent sanction was  the possible reorganization of the local service delivery organization.
section_title: WIA Requires that Many Federal Programs Work Together to Provide Services through the One-Stop System
paragraphs: In addition to establishing the three new programs, WIA requires that  states use the one-stop center system to provide services for these and  many other employment and training programs. This system was  developed by states prior to WIA through One-Stop Planning and  Implementation Grants from Labor. About 17 programs funded through  four federal agencies are now required to provide services through the  one-stop center under WIA. Table 2 shows the programs that WIA  requires to provide services through the one-stop centers (termed  mandatory programs) and the related federal agency.
Department of Health and Human  Services  Department of Housing and Urban  Development (HUD)
Under WIA, employers are expected to play a key role in establishing  regional workforce development policies, deciding how services should be  provided in the one-stop, and overseeing one-stop operations. Employers,  who are encouraged to use the one-stop system to fill their job vacancies,  are also seen as key one-stop customers under WIA.
section_title: States and Localities Are Taking Action to Implement WIA Performance Measures but Face Challenges in Doing So
paragraphs: States and localities are taking action to implement performance measures  for the three WIA-funded programs, but they have confronted several  challenges in doing so. To implement these measures, states and localities  had to change the way they collected and reported performance data.  Most states we surveyed had to create new automated data systems to  collect and report WIA data. Many state systems, however, are still not  completely in place. The lack of final guidance from Labor on how to  report the data slowed the development of these systems. States and  localities also faced challenges in implementing the measures due to their  complexity and the resource demands they created, and some had to  develop new procedures to obtain UI wage records. In addition, states  faced a new negotiation process with Labor to set performance levels for  each measure. Many states believe these levels are too high because little  or no baseline data were used, and the negotiations did not sufficiently  account for differences in economic conditions and populations served.  Under WIA, performance levels are now tied to incentives and sanctions  so that states can be financially rewarded if they meet them or penalized if  they do not. States reported that the need to meet these performance  levels may lead local staff to focus WIA-funded services on job seekers  who are most likely to succeed in their job search or who are most able to  make wage gains.
subsections: section_title: Management Information Systems Are Still Being Developed to Collect and Report WIA Data
paragraphs: As part of implementing WIA performance measures, states had to develop  automated data systems to track the activities of individual WIA  participants and report on performance. Based on our survey, most states  developed a new automated data system, or management information  system (MIS), to collect and report WIA performance data at the state  level. The remaining states adapted their previous data collection systems  used under JTPA. However, 15 states, regardless of whether they were  developing a new system or adapting their existing system, reported that,  as of August 2001, they did not have their system completely in place. All  states expect to have completed their systems by July 2002. In some states,  local areas do not use the state MIS to collect local WIA performance  information. In these states, local areas must develop their own systems,  taking time and resources to do so.
The lack of timely reporting guidance slowed the development of the data  systems. Final guidance on how states must report their performance to  Labor was issued to the states in March 2001—8 months after the states  were required to implement major provisions of WIA and begin collecting  data. Lack of final guidance resulted in delays and costly program  changes as states and local areas developed and adjusted their final  systems. For example, one local area we visited decided to continue using  its old system and delayed the development of a new system pending final  guidance because it would be too costly and time-consuming to develop a  system that might need to be changed. All states, regardless of whether or  not they had implemented a new system, had to make changes in their  automated MIS systems to accommodate the final guidance.
section_title: Complex Performance Measures Stymied State and Local Efforts to Implement Them
paragraphs: States and localities reported that the complexity of WIA’s new  performance measures made them difficult and time-consuming to  implement. Many of the states that we surveyed commented that the  measures were hard to follow because the calculations for the measures  are complex and sometimes confusing, specifically who to include in the  measures, when to collect the data for the measures, and how to calculate  the measures.
Knowing who to include in the measures. It is difficult to know  whether or not a job seeker should be counted in the measure for a  program, even if he or she is served by the program. For example, a  participant in the adult program who is already employed must be  included in the retention and wage gain measures but cannot be  counted in the entered employment measure. Yet, for the dislocated  worker program, the entered employment measure can include those  who may be employed when they enter the program.
Knowing when to measure performance for participants. The  data for different performance measures can be collected in different  quarters of the program year. For example, for customer satisfaction,  data can be collected at two points in time, depending on how the  participant exited the program (see fig. 1). Data on entered  employment for participants in the adult program are collected in the  third quarter after exit; retention data are collected in the fifth quarter  after exit.  Data on earnings change and replacement are collected at  two points in time: pre-program earnings data can be collected at  registration, and post-program earnings data are collected in the fifth  quarter after exit. If a program participant does not appear in the UI  wage records, local staff can collect supplemental data to establish  employment for the participant, but this must be recorded within 30  days after the WIA participant is found missing from the wage records.  For entered employment, staff can collect supplemental data in the  fourth quarter after a participant leaves (or exits) the program, but for  retention, staff can collect it in the sixth quarter after exit.  Supplemental data cannot be used to measure earnings change and  replacement. Because the timing of data collection is complex and can  be confusing, one local area in Oregon developed a tool it calls the  “bean counter” to help local staff determine when to follow-up with  participants so their performance counts in the calculations (see fig. 2).
Knowing how to calculate the measures. In order to calculate the  measures, states must account for a variety of factors. The type and  combination of these factors determine the calculations that will be  used. For example, in calculating some of the measures for the adult  program, states must consider (1) whether the job seeker is employed  at registration, (2) whether he or she is employed at both the first and  third quarters after exit, and (3) whether the data source used to  confirm employment was UI records or supplemental data. This  information in various combinations results in 14 different ways that  adult participants can be grouped in order to calculate the measures.
In addition to noting the complexity of the measures, state and local  officials said that the new measures taxed their resources. States had to  develop procedures to collect data for the new customer satisfaction  measure in compliance with detailed guidance from Labor. The guidance  calls for states to conduct a telephone survey from a random sample large  enough to obtain 500 completed surveys from both participants and  employers. Because this guidance changed over time, states had to revise  their procedures accordingly. For example, a revision to the guidance  issued in October 2001 required states to maintain an up-to-date list of  participants’ names and addresses from which to sample—a requirement  that was originally voluntary.
One indication of states’ progress in implementing these measures may be  reflected in their ability to submit complete quarterly reports. Quarterly  reports require data on all 17 performance measures. For the quarterly  report that was due in May 2001, all states submitted their reports, but,  according to Labor, only 16 states were able to provide data for all 17  performance measures. For the quarterly report that was due in August  2001—more than 1 year after WIA implementation—all states submitted  their reports, but only 23 could provide data on employer customer  satisfaction. According to Labor, states could not fully report on  customer satisfaction because they have not yet fully implemented  procedures to measure it. One state had to compile the data manually  because its MIS was not fully operational.
section_title: States Adopt New Procedures to Access Sensitive Information
paragraphs: WIA’s new requirement that states use UI wage records to measure  outcomes has led states to adopt new procedures to access these and  other sensitive records. Unlike JTPA, which relied on surveys of  participants to collect information on employment and earnings, WIA  requires UI wage records to be used as the primary data source of  employment and wage information—and the only data source for some  measures, according to Labor’s guidance. To obtain employment and  earnings information, states match information collected on individual  WIA participants against state UI wage records. To access UI data from  the state agency that oversees the UI database, some states had to  establish data-sharing agreements. In Mississippi, for example, the agency  responsible for overseeing WIA—the Mississippi Development Authority  (MDA)—had to make arrangements with the agency that oversees the UI  data—the Mississippi Employment Security Commission—to have them  match the wage records and provide the results to MDA.  In addition, some  states may be more rigorous in protecting the confidentiality of UI records  through privacy laws, which may add obstacles to collecting performance  data. For example, Oregon law prohibits the release of WIA participants’  records without informed consent. Consequently, program providers had  to enter into an agreement that established a protocol for collecting and  sharing the data—one that developed safeguards to protect confidentiality.  In addition, the state had to develop a process to ensure that WIA  participants consented to the use of their protected records in this way.
section_title: States Believe Performance Levels Are Set Too High
paragraphs: All the states we visited believed that some of the established performance  levels for their measures were set too high for them to meet—either  because they were set in absence of historical or baseline data or because  negotiations did not sufficiently account for variations in economic  conditions or population served. States reported that limitations in  available baseline data made it difficult to set fair, realistic performance  levels. The new measures on credentials and customer satisfaction, for  instance, had no prior data available on which to set performance levels.  Where baseline data were available, such as for the wage-related  measures, the data were collected under JTPA, a program whose goals  were different from those of WIA. In addition, some states believe that  the performance levels did not account for variations in economic  conditions, such as the slow growth in new or existing businesses that  some areas have experienced. Performance levels also did not account for  the many economically disadvantaged or hard-to-serve individuals seeking  services in some local areas.
section_title: States Report that Performance Levels May Determine Who Receives WIA-Funded Services
paragraphs: Many states reported that the need to meet performance levels may be the  driving factor in deciding who receives WIA-funded services at the local  level. All the states we visited told us that local areas are not registering  many WIA participants, largely attributing the low number of WIA  participants to concerns by local staff about meeting performance levels.
Local staff are reluctant to provide WIA-funded services to job seekers  who may be less likely to get and keep a job. One state official described  how local areas were carefully screening potential participants and  holding meetings to decide whether to register them. As a result,  individuals who are eligible for and may benefit from WIA-funded services  may not be receiving services that are tracked under WIA.
Performance levels for the measures that track earnings change for adults  and earnings replacement for dislocated workers may be especially  problematic.  Several state officials reported that local staff were reluctant  to register already employed adults or dislocated workers.  Officials in one  state reported that some local areas had not yet registered any dislocated  workers. State and local officials explained that it would be hard to  increase the earnings of adults who are already employed or replace the  wages of dislocated workers, who are often laid off from high- paying, low- skilled jobs or from jobs that required skills that are now obsolete. In  addition, for dislocated workers, employers may provide severance pay or  workers might work overtime prior to a plant closure, increasing these  workers’ earnings before they are dislocated. As a result, many dislocated  workers who come to the one-stop center have earned high wages just  prior to being dislocated, making it hard to replace—let alone increase—  their earnings. If high wages are earned before dislocation and lower  wages are earned after job placement through WIA, the wage change will  be negative, depressing the wage replacement level. As a result, a local  area may not meet its performance level for this measure, discouraging  service to those who may need it.
A hypothetical example involving two workers dislocated at the same time  illustrates this point (see table 3). One worker is a sales clerk with limited  skills earning $25,000, the other a long-time factory worker with obsolete  skills earning $60,000. Both are laid off from work and go to their local  one-stop center seeking job placement assistance. The clerk is placed in a  new job as a receptionist paying $25,000. By calculating his wage  replacement from his salary as a clerk, the one-stop can claim a wage  replacement rate of 100 percent. The factory worker eventually gets a job  as a security guard earning $30,000, netting a wage replacement rate of 50  percent. As this example shows, a one-stop center can meet its  performance levels more easily by serving the clerk than by serving the  factory worker even though both job seekers may need the one-stop  system’s resources to find a job or enhance their skills.
Some states and Labor are making efforts to address this disincentive to  serve certain job seekers. Indiana instituted a policy allowing local areas  to adjust their dislocated worker wage replacement rate in light of the  significant dislocations they are facing. Texas uses a regression model to  establish local performance levels that adjust for differences in factors,  such as economic conditions and the characteristics of individuals served.  Without this policy, said a Texas official, WIA programs would have  registered fewer workers. Similarly, Michigan substantially reduced the  penalties to local areas for failing to meet performance levels and found  that the number of registered participants increased as a result of  instituting less threatening sanctions. WIA requires that states be allowed  to renegotiate their performance levels based on unanticipated  circumstances. Labor is currently developing criteria that states can use to  renegotiate their performance levels based on unanticipated  circumstances, such as changes in economic conditions due to plant  closings or shifts in unemployment for the current and future years. The  guidance is expected to be released soon.
section_title: Performance Measures May Not Accurately Assess Performance of the Three WIA-Funded Programs
paragraphs: Even when fully implemented, WIA performance measures may still not  provide an accurate picture of performance for the three WIA-funded  programs largely because data are neither comparable across states nor  timely. State and local officials generally supported many of the  performance measures as relevant indicators of the success of an  employment and training program. However, the performance data  collected and reported by states and localities are not comparable largely  because of the lack of clear guidance on when to collect and report  performance data and what constitutes a credential—the attainment of a  certified skill or degree. In addition, while UI wage records are one of the  best available sources of employment and earnings data, limitations in the  data may hinder the ability of states and local areas to meet their  performance levels and use the measures for short-term program  management.
subsections: section_title: Performance Measures Include Relevant Indicators, but Lack of Clear Guidance Affects Accuracy and Comparability of Performance Data
paragraphs: State and local officials in the states we visited generally support many of  the performance measures as relevant indicators of the success of an  employment and training program. For example, several officials cited the  wage-related measures, such as job placement, retention, and earnings  change, as important indicators of a successful employment and training  program. The measures are also generally consistent with the goal of WIA  to help individuals get and keep jobs and increase their wages and skills.  In addition, the states noted that the measures provide a good basis for  long-term evaluation.
However, the performance data collected and reported by states and  localities are not comparable—a critical component in creating a level  playing field from which states’ relative performance can be evaluated.  While there are various reasons that performance data are not  comparable, one of the chief reasons is the lack of clear guidance for  collecting and reporting performance data on participants. Labor has  provided detailed written guidance to states on who should be registered  under WIA and when this registration should occur, but the guidance is  open to interpretation in some areas. The lack of a uniform  understanding of when registration occurs and thus who should be  counted toward the measures raises questions about both the accuracy  and comparability of states’ performance data. For example, the guidance  tells states to register adults and dislocated workers who need significant  staff assistance designed to help with job seeking or acquiring  occupational skills, but the state can decide what constitutes significant  staff assistance. The guidance provides examples of when to register job  seekers, but it sometimes requires staff to make subtle and subjective  distinctions. For example, those who receive initial assessment of skill  levels and the need for supportive services are not to be registered; those  requiring comprehensive assessment or staff-assisted job search and  placement assistance, including career counseling, are to be registered. In  another example involving the classification of workshops, job seekers  who participate are to be registered in some cases, but not in others.
Labor has allowed states and local areas flexibility in implementing the  registration policy, and we found that local areas differed on when they  registered WIA job seekers. In one local area we visited, the one-stop  center registers most job seekers who come into the center, even if staff  assistance is minimal. At this center, a general orientation is sufficient for  the job seeker to be registered under WIA. In contrast, another center in  the same state registers only those job seekers who require significant  staff assistance and are likely to benefit from intensive services. Similar  disparities occurred in other states we visited. Labor has said there is little  consistency across states in registering participants and has convened a  work group to develop additional guidance on registration, but as yet, the  issues remain unresolved.
The lack of a definition for the credential measure is also leading to  performance data that are not accurate or comparable across states. Labor  allows the states and local areas to determine what constitutes a  credential and to develop a statewide list of approved credentials with  input from employers. Because states and, in many cases, local areas must  define what constitutes a credential, what is currently counted as a  credential differs within and across states. Some states may strictly define  credentials to include only diplomas from accredited institutions or use  only formal training completion criteria as defined by education partners.  Other states may expand their criteria to count a broad variety of  credentials, such as job readiness, on-the-job experience, and completion  of workshops. Labor officials note that states’ performance levels for the  credential measure are negotiated to take state and local definitions into  account, and the measure is intended to help local employers gauge the  readiness and skill level of job seekers. Nevertheless, given the broad  range of definitions states and localities employ, the outcomes on the  credential measure may be of limited value, even within a single state.
section_title: Challenges in Using UI Wage Record Data Also May Affect the Accuracy and Comparability of Performance Data
paragraphs: UI wage records are one of the best available data sources for tracking the  employment and earnings of individuals—a significant improvement over  the less objective self-reporting methods of JTPA—but the limitations of  the database pose challenges that need to be addressed. These challenges,  if unresolved, may hinder states’ ability to meet their performance levels.  As we have reported in prior work, one such limitation is that UI wage  records, while covering about 94 percent of workers, exclude certain  employment categories, such as self-employed persons, most independent  contractors, military personnel, federal government workers, and postal  workers. States, therefore, must develop alternative methods to track WIA  participants who are employed in these uncovered occupations.  Pennsylvania, for example, developed a partnership with other states in its  region to share the cost of purchasing the rights to federal civil service and  military personnel data. And Florida has developed agreements with the  Department of Defense, the Office of Personnel Management, and the U.S.  Postal Service to access employment and wage information on an annual  basis. Our survey data indicate that 33 states are using additional or  supplemental data to compensate for uncovered occupations, with only 27  of those using the supplemental data to count towards their performance  levels. Thus, at least 23 states have not used additional data to help them  meet their performance levels.
Another limitation is that state UI databases include only wage record  information on job seekers who get jobs within their state; they do not  track job seekers who find jobs in other states. States cannot readily  access UI wage records from other states to track outcomes under WIA,  making it difficult to track individuals who receive services in one state  but get a job in another. Over one-third of all of the states we surveyed  reported that an estimated 16 to 30 percent of cases are not being picked  up by their state’s UI wage record system. To fill in these gaps, seven  states have agreements with other states—often those that share a  common border—to exchange UI information. Indiana, for example,  established an agreement with Illinois to trade data. If data are missing on  particular participants, Illinois sends the cases to Indiana to see if the  Indiana UI wage records have information on the job seeker. The value of  these agreements, however, may be limited because job seekers may find  work in a state that does not have an agreement with the one in which  they received services.
Another way to obtain UI data on workers who are employed out of state  is through WRIS, a clearinghouse that makes UI wage records available to  states seeking employment and wage information on their WIA  participants. This information can provide outcome data on WIA  participants to help states meet their performance levels. While WRIS was  available for states to use by July 2001, only 7 out of the 50 states are  currently able and ready to participate, with 8 others in various stages of  completing the requirements for participation. Although many states have  shown an interest in a system such as WRIS, many are reluctant to  participate because Labor, while agreeing to cover all the costs of  operating WRIS for its first year, has not yet agreed to pay for subsequent  years. The estimated total cost of operating WRIS is $2 million annually,  but states have not been given a definitive answer about how much it  would cost them to participate after this first year if Labor does not  continue funding. Because of this uncertainty regarding future costs,  states are hesitant to commit to participation in WRIS. If not all states  participate, the value of WRIS will be diminished—even for participating  states—because no data will be available from nonparticipating states’ UI  wage records.
section_title: Lack of Timely Data Limits Use of Performance Measures for Short-Term Program Management
paragraphs: The lack of timely data, due to the time lag in obtaining UI wage records,  makes it difficult for state and local officials to use the performance  measures for short-term program management—because, for the wage- related measures, current available data on the measures will reflect  performance from the previous program year. While UI wage records are  the best available data source for documenting employment, the data  collection and reporting process is slow and time-consuming. Data are  generally collected from employers only once every quarter, and  employers have 30 days after the quarter ends to report the data. In many  states, employers—especially small businesses—are allowed to submit  data in paper format, which then must be converted to electronic media.  After data entry, information must be checked for errors and corrected. All  of these steps take time. As a result, WIA program administrators are  unable to get a timely picture of program performance. For example, we  asked states in our survey how quickly job placement outcome data would  be available to them from UI wage records. On the basis of our survey, we  found that for 30 states, the earliest time period that job placement data  would be available is 6 months or more after an individual entered  employment, with 15 states reporting that it may take 9 months or longer.  Similarly, for the employment retention measure, over half of states report  that obtaining this information could take a year or longer. (See fig. 3.)
The time delay in receiving UI wage record data makes it difficult for state  and local officials to use the performance measures to gauge the  effectiveness of their services. States report that not being able to get  performance results in the same program year is a problem: it makes it  difficult to manage programs and improve one-stop services. Labor reports  that the performance measures are not intended to be a management tool.  State and local officials, therefore, must develop alternative methods if  they want to assess the quality of their services so they can identify  problems and improve programs in a timely way. Labor has encouraged  these efforts, but, while some local areas are finding ways to collect data  to help them manage their programs, there is no cohesive effort at the  federal level to share strategies and promising approaches for the Adult  and Dislocated Worker Programs.
section_title: Existing Performance Measures Fail to Gauge Overall One- Stop Performance
paragraphs: Although there are performance measures for the three WIA-funded  programs and most of the programs required to provide services in the  one-stop, no measures exist to assess how well the overall one-stop  systems are working. The success of the one-stop system as a whole is not  captured by the program measures of individual one-stop partners.  Furthermore, combining the performance measures from mandatory  programs does not provide a comprehensive picture of one-stop  performance. Even when measures appear the same, comparing them is  difficult because of differences in definitions and calculations. Beyond  failing to provide a complete evaluation of one-stop performance, state  officials reported that the separate reporting requirements of the partner  programs have hampered coordination within their one-stop systems.  While WIA did not establish any comprehensive measures to assess the  overall one-stop system, it required that Labor take the lead in developing  optional measures to help states assess progress toward their workforce  investment goals.  Labor has made limited progress on such performance  measures, and only a few states have developed their own overall  measures.
subsections: section_title: Existing Performance Measures Fail to Capture Important One-Stop Features
paragraphs: The existing performance measures for participating one-stop programs  fail to capture important one-stop features. First, it is difficult to get an  unduplicated count of job seekers using the one-stop. While an individual  may have multiple outcome measures for the services received through  each of the programs at a one-stop, there is no single outcome measure for  multiple services. In addition, separate reporting systems for each of the  programs make it difficult to disaggregate data and track an individual’s  outcome for those receiving multiple services.
Second, other important aspects of one-stop performance are not included  within the existing measures. Customer satisfaction measures used in  support of WIA-funded programs and the Employment Service fail to  measure how job seekers and employers believe they are being served by  the one-stop system as a whole. Instead, these measures show satisfaction  with the individual programs. Employer satisfaction is important under  WIA because WIA created a more private-sector driven system. Capturing  customer satisfaction of the system as a whole would reflect whether job  seekers are successful at attaining the services they need to get jobs and  would assess whether employers are satisfied with job applicants sent to  them from different one-stop programs.
Finally, state and local officials expressed concern that a large portion of  one-stop participants are not included in performance measures. Many job  seekers use self-service and informational activities but they are not  tracked and counted in any program measures. While staff time and  resources are used to establish and maintain self-service resource rooms  and web sites, job seekers who use only these services will not be  included in any of the performance measures. Without any information on  individuals who use self-service, it will be difficult for Labor to show how  effectively one-stops are being used.
section_title: Multiple Measures Cannot Be Used to Measure the One-Stop System’s Performance
paragraphs: Performance measures for different programs often track similar  outcomes, as figure 4 shows. However, the measures cannot be combined  to obtain an overall view of one-stop performance. Although the same  terms are used in various performance measures, their definitions are not  identical. For example, while WIA older and younger youth programs  define youth as being between the ages of 14 and 21, the laws governing  Job Corps and HUD’s Youthbuild define youth as being between the ages  of 16 and 24. Similarly, the definition of veterans is different for the  Employment Service and Veterans’ Employment and Training Service  program. The differences in definitions mean that assessing the outcomes  for youth or veterans by combining the performance measures of  individual programs within the one-stop setting would be difficult.
Besides variations in definitions, there are also variations in how measures  are calculated in different programs. For example, while the entered  employment rate for WIA’s adult program is defined as the percentage of  workers who get a job by the end of the first quarter after exit, the entered  employment rate for the Employment Service is defined as the percentage  of workers who get a job or changed employers in the first or second  quarter after registration. As a result, performance data from these  separate programs cannot be combined to yield a single overall score to  assess various performance outcomes of the one-stop system. The Office  of Management and Budget (OMB) convened a work group representing  federal WIA partners to look at common definitions and measures across  programs, which may issue guidance to states and localities on WIA  performance measures and federal requirements.
section_title: Separate Measures May Impede Coordination
paragraphs: Beyond failing to provide an overall picture of one-stop performance, state  officials reported that separate performance measures impede the  cooperation of one-stop partners. As a result, even though WIA was  meant to establish a more coordinated workforce development system  through the use of one-stops, over one-third of states surveyed expressed  concern that individual program performance measures may impede this  process. Some states even believed that separate measures caused  competition among programs. They said that if staff did not understand  that a participant could be counted in more than one program, they might  not direct them to other one-stop programs. For example, one state  reported in its written comments to our survey that competition for  participants and duplication of services due to lack of coordination with  other programs would continue as long as each program is required to  meet its own performance and participation levels. Fourteen states  volunteered in their written comments to our survey that the federal  government should work to coordinate performance measures across  programs or develop systemwide measures. In addition, while states agree  that systemwide measures are needed, they caution against making any  additional measures mandatory, since states are still adapting to the  existing measures.
section_title: Limited Progress on Developing Performance Measures for One-Stop Systems
paragraphs: Although WIA did not establish one-stop measures, it does require that  Labor develop additional optional measures to assist states in assessing  progress toward their workforce investment goals, which Labor has  interpreted to include one-stop measures. Labor began developing  workforce development performance measures to capture overall one- stop use after one-stop systems were piloted. Since the passage of WIA,  Labor has continued its efforts to develop systemwide measures, but it has  made limited progress. Labor convened a working group in September  2001 to develop additional indicators of one-stop performance. Partner  representatives at this group included national workforce-related  organizations including the National Association of State Workforce  Agencies, National Governors’ Association, National Association of  Counties, U.S. Conference of Mayors, and representatives of states and  regional boards. This group is working to develop a menu of indicators  that will help provide a comprehensive picture of WIA system activity.  Such measures may include capturing information on self-service  customers and the cost of services at the one-stop.These measures,  while optional, would help provide information on overall one-stop use  across the country if all states report on at least some of the measures.  Labor plans to have guidelines for these optional indicators in place for  use in program year 2002, which begins on July 1, 2002. In order for states  to be able to implement them for the coming program year, Labor will  need to provide final guidance well before July 2002.
In addition to these national efforts, some states, on their own initiative,  have attempted to develop additional measures for one-stop systems, but  these efforts are not coordinated and do not allow for nationwide  assessment of the one-stop system. According to our survey, eight states  have created or are developing additional systemwide measures, but of  those, only three are reporting them to Labor. Pennsylvania, for example,  developed five measures specific to its one-stop system’s performance.  These indicators, intended to measure the overall effectiveness of the one- stop system, include median cycle time to fill a job, and the percentage of  employers and individuals using services through the one-stop. Florida, on  the other hand, has developed “tiers” of measures that focus on the  outcome of their workforce development programs. In the first tier, state- generated systemwide indicators measure many employment and training  programs together. The second tier clusters similar types of programs and  captures measures relevant to particular groups (e.g., continued education  status of youth in youth programs). The third tier captures all the federally  mandated measurements, as well as measures for the other tiers, such as  caseloads for specific programs. In this way, Florida has attempted to  measure the system overall as well as outcomes for individual programs.  Measures developed by other states include the number of people using  the resource rooms at one-stops and the increase over time in the number  of unemployed people getting a job. Despite these states’ efforts, the  absence of nationally established systemwide measures means that Labor  cannot ensure nationwide comparability.
section_title: Conclusions
paragraphs: WIA represents a fundamental shift in the way federally funded  employment and training services are provided to job seekers and in the  way WIA programs measure and monitor success. Despite obstacles, in  just over a year states have made good progress in implementing the new  requirements under WIA—developing new processes and designing new  systems. Labor, for its part, has been working to find ways to allow states  and localities greater flexibility to design their programs to meet local  needs and has been actively seeking opportunities for states to have input  into the process, particularly in the area of performance measurement. But  given the challenges states have faced in implementing the new  performance measurement system, more time is needed before the  measures can meaningfully gauge the success of the programs.
This new performance measurement system under WIA is a high-stakes  game—a state’s future funding and, therefore, its ability to serve its  citizens may depend upon how well it performs compared to how well it is  expected to perform. It should be no surprise that states and localities are  designing their systems and processes in ways that will enhance their  ability to meet their performance levels. Because states see the current  performance levels as too high for the current economy, states and  localities may choose not to serve those job seekers who may be helped by  their services, but who may not help in achieving their negotiated  performance levels. Unless the performance levels can be adjusted to truly  reflect differences in economic conditions and the population served, local  areas will continue to have a disincentive to serve some job seekers that  could be helped.
WIA’s requirement to use UI data to track outcomes is a step in the right  direction—it provides federal, state, and local government entities with an  objective means to evaluate program success. But it brings challenges that  need to be addressed, and states will need help to do so. Establishing the  means to routinely share data across state lines through WRIS and  developing ways to share promising approaches in the use of  supplemental data sources and in managing the assessment of short-term  program needs would go far in moderating these challenges. Without this  help and the cooperative efforts of states and localities toward this end,  developing a useful performance measurement system will take longer and  cost more.
In general, WIA’s performance measurement system captures some useful  information, but it may not capture all the right information. The measure  to track credentials has limited value because it lacks a standard definition  for what’s being measured. For other measures, the lack of clear  definitions for whom to track limits their usefulness in drawing  conclusions about program success at both the state and national levels.  Without clear definitions and processes, the measures will not provide the  Congress with a true picture of how well the programs are performing.  Furthermore, WIA performance measures gauge only WIA-funded  services; yet there is widespread agreement that measures are needed to  gauge the effectiveness of the entire one-stop system. The system’s narrow  focus on program outcomes for a limited number of participants misses a  key requirement of WIA to support the movement toward a coordinated  system. In fact, the measures may foster the opposite—a siloed approach  that encourages competition among programs and limits their  cooperation. Without global one-stop measures, the Congress will not be  able to assess how well states and localities are doing in meeting WIA’s  requirement to coordinate services. The lack of such measures may,  instead, send a signal to states that service coordination is a minor goal.
section_title: Recommendations for Executive Action
paragraphs: To give states and local areas more time to implement WIA performance  measures and establish baseline data needed to determine performance  levels, we recommend that the Department of Labor delay the application of financial sanctions for at least 1 year or until it  is judged that states have their data systems sufficiently in place to  successfully track WIA outcomes.
To eliminate possible disincentives to serve some job seekers and ensure  that states and local areas will not be unduly penalized for economic  downturns, we recommend that the Department of Labor expedite the release of guidance on revising negotiated performance  levels and allow states to immediately begin the process of re- negotiation.
To ensure uniformity in data collection and reporting so that performance  results are more accurate and comparable across states, we recommend  that the Department of Labor provide clearer guidance using objective criteria on who should or  should not be registered as a WIA participant for tracking purposes  and, once the guidance is released, work proactively with states to  implement it, and  issue guidance delineating a clear definition for what constitutes a  credential, and, once the guidance is released, ensure that states use it  to report on this indicator.
To help states address the challenges of using UI data to measure  outcomes, we recommend that the Department of Labor continue to fully fund the Wage Record Interchange System in order to  facilitate the sharing of UI data across state lines; and develop ways for states to share promising approaches in the use of  supplemental data sources in closing the data gaps for covered and  uncovered employment in UI; and develop ways for states to share promising approaches that help states  address the UI timeliness issue, providing methods to help states  monitor and improve their programs in a timely manner.
To help states measure one-stop performance, we recommend that the  Department of Labor ensure that the development of optional one-stop system measures is  completed in enough time for states to implement them at the  beginning of program year 2002.
section_title: Agency Comments
paragraphs: We provided a draft of this report to Labor for its review and comment.  Labor’s comments are in appendix II. We incorporated comments and  clarifications where appropriate.
Labor generally agreed with our findings and recommendations, noting  that they are consistent with information they have gathered from state  and local partners.  In its comments, Labor expressed concern that  negotiated performance levels may be determining who receives WIA- funded services, indicating that it will work with states and local areas to  address this issue.  Labor also commented on our finding regarding the  lack of clear guidance on certain policies, stressing the importance of state  and local flexibility in determining specific policies and practices to fit  local needs. While state and local flexibility is important, we continue to  be concerned that the lack of a uniform understanding of when  registration occurs and what constitutes a credential raises questions  about both the accuracy and the comparability of states’ performance  data.  We are pleased to note that Labor is in the process of reviewing this  issue.  Finally, Labor cites its efforts to collaborate with states and local  areas in developing a performance accountability system and increasing  partnerships.  We commend Labor for obtaining states’ input and  participation in developing such a system.
We are sending copies of this report to the Secretary of Labor, relevant  congressional committees, and others who are interested. Copies will also  be made available to others upon request. The report is also available on  GAO’s home page at http://www.gao.gov.
Please contact me on (202) 512-7215 if you or your staff have any  questions about this report. Other major contributors to this report are  listed in appendix III.
section_title: Appendix I: WIA Performance Measures for Adults, Dislocated Workers, and Older Youth
paragraphs: The following table includes descriptions of the performance measures  reviewed in this report. It does not include the calculations required to  attain the final value of the performance measures.
Definition  Of those who did not have a job when they registered for WIA, the percentage of adults  who got a job by the end of the 1st quarter after exit. This measure excludes participants  who are employed at the time of registration.  Of those who had a job in the 1st quarter after exit, the percentage of adults who have a job  in the 3rd  quarter after exit.  Of those who had a job in the 1st quarter after exit, the post-program earnings increases as  compared with pre-program earnings.  Of those adults who received WIA training services, the percentage who were employed in  the 1st quarter after exit and received a credential by the end of the 3rd  quarter after exit.
The percentage of dislocated workers who got a job by the end of the 1st quarter after exit.  This measure includes dislocated workers who are employed at the time of registration.  Of those who had a job in the 1st quarter after exit, the percentage of dislocated workers  who have a job in the 3rd  quarter after exit.  Of those who had a job in the 1st quarter after exit, the percentage of pre-program earnings  being earned post-program. Since it may be difficult to find dislocated workers jobs with  equivalent or better wages, this measure captures the percentage of earnings of the new  job in relation to the old.  Of those dislocated workers who received WIA training services, the percentage who were  employed in the 1st quarter after exit and received a credential by the end of the 3rd quarter  after exit.
Of those who are not employed at registration and who are not enrolled in post-secondary  education or advanced training in the 1st quarter after exit, the percentage of older youth  who have gotten a job by the end of the 1st quarter after exit. This measure also excludes  youth that move on to post-secondary education or advanced training and not  employment.  Of those who are employed in the 1st quarter after exit and who are not enrolled in post-  secondary education or advanced training in the 3rd  quarter after exit, the percentage of  older youth that are employed in the 3rd  quarter after exit.  Of those who had a job in the 1st  quarter after exit and who are not enrolled in post-  secondary education or advanced training, the post-program earnings increases as  compared with pre-program earnings.  The percentage of older youth who are in employment, post-secondary education, or  advanced training in the 1st quarter after exit and received a credential by the end of the 3rd  quarter after exit.
The average of three statewide survey questions rated 1-10: (1 being “very dissatisfied”  and 10 being “very satisfied”): was the employer satisfied with services  Definition did the service meet the expectations of the customer how well did the service compare to the ideal set of services  The average of three statewide survey questions rated 1-10: was the participant satisfied with services did the service meet the expectations of the customer how well did the service compare to the ideal set of services  A statewide telephone survey of a sample of 500 is conducted for all the WIA-funded programs.
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: Abbey Frank, Mikki Holmes, and Amanda Ahlstrand made significant  contributions to this report. In addition, James Wright assisted in the study design and the national survey; Jessica Botsford and Richard Burkard  provided legal support; and Patrick DiBattista and Barbara Alsip assisted  in the message and report development.
section_title: Related GAO Products
paragraphs: U.S. General Accounting Office. Workforce Investment Act: Better  Guidance Needed to Address Concerns Over New Requirements. GAO-02- 72. Washington, D.C.: 2001.
U.S. General Accounting Office. Veterans’ Employment and Training  Service: Proposed Performance Measurement System Improved, But  Further Changes Needed. GAO-01-580. Washington, D.C.: 2001.
U.S. General Accounting Office. Multiple Employment and Training  Programs: Overlapping Programs Indicate Need for Closer Examination  of Structure. GAO-01-71. Washington, D.C.: 2000.
U.S. General Accounting Office. Workforce Investment Act:  Implementation Status and the Integration of TANF Services. GAO/T- HEHS-00-145. Washington, D.C.: 2000.