Workforce Investment Act: Improvements Needed in Performance Measures to Provide a More Accurate Picture of WIA's Effectiveness
Released: 2002-02-01T00:00:00

Letter

Background

Labor required states to implement major provisions of WIA Title I by July 1, 2000, although some states began implementing provisions of WIA as early as July 1999. Services provided under WIA represent a marked change from those provided under the previous program, allowing for a greater array of services to the general public. WIA is designed to provide for greater accountability over what existed previously: it established new performance measures and a new requirement to use UI data to track and report on the achievements of the three WIA-funded programs. WIA also requires that many federal programs work together to provide employment and training services through the one-stop system.

WIA-Funded Services Represent a Change from those Funded under JTPA

Program services provided under WIA represent a marked change from those provided under JTPA. When WIA was enacted in 1998, it replaced the JTPA programs for economically disadvantaged adults and youth and for dislocated workers with three new programs—WIA Adult, Dislocated Worker, and Youth—that provide a broader range of services to the general public, no longer using income to determine eligibility for all program services. The newly authorized WIA programs no longer focus exclusively on training but provide for three tiers, or levels, of service for adults and dislocated workers: core, intensive, and training. Core services include basic services such as job searches and labor market information. These activities may be self-service or require some staff assistance. Intensive services include such activities as comprehensive assessment and case management—activities that require greater staff involvement. Training services include such activities as occupational skills or on-the- job training. These tiers of WIA-funded services are provided sequentially. That is, in order to receive intensive services, job seekers must first receive at least one core service; to receive training services, a job seeker must first receive at least one core service and then at least one intensive service. Key to moving from core to a higher level of services is that the services are needed to help job seekers become self-sufficient. Labor’s guidance provides for monitoring and tracking to begin when job seekers receive core services that require significant staff assistance. Job seekers who receive core services that are self-service in nature are not included in the performance measures.

WIA Performance Measures Are Designed to Increase Accountability for Three WIA-Funded Programs

WIA is designed to provide for greater accountability than the accountability provided for under JTPA. It does so by establishing new performance measures and a new requirement to use UI data to track and report on the achievements of the three WIA-funded programs. According to Labor, performance data collected from the states in support of the measures are intended to be comparable across states in order to maintain objectivity in determining incentives and sanctions. They are also intended to provide information to support Labor’s performance goals under the Government Performance and Results Act (GPRA) and for program evaluation. Some of the measures that relate to adults, dislocated workers, and older youth are similar to those used under JTPA, including job placement, job retention, and wage gains or replacement. Attainment of a credential—a degree or certification of skills or training completed—and customer satisfaction for both job seekers and employers are new under WIA. (See table 1 for a complete list of the WIA performance measures and appendix I for a more complete explanation of the performance measures discussed in this report.)

In contrast to JTPA, for which data on outcomes were obtained through follow-ups with job seekers, WIA requires states to use UI wage records to track outcomes. According to Labor’s guidance, if a program participant does not appear in the UI wage records, states may use supplemental data sources, such as follow-ups with participants and employers, to track entered employment, retention, and credential attainment. However, only UI wage records may be used to calculate earnings change and replacement.

Unlike JTPA, which established expected performance levels using a computer model, WIA requires states to negotiate with Labor to establish expected performance levels for each measure. States, in turn, must negotiate performance levels with each local area. The law requires that these negotiations take into account differences in economic conditions, participant characteristics, and services provided. To derive equitable performance levels, Labor and the states use historical data to develop their estimates of expected performance levels. These estimates provide the basis for negotiations.

WIA holds states accountable for achieving their performance levels by tying those levels to financial sanctions and incentive funding. States that meet their performance levels under WIA are eligible to receive incentive grants that may generally range from $750,000 to $3 million. To be eligible for an incentive grant, states must also meet levels established under the Department of Education’s Vocational Education (Perkins Act) and Adult Education and Literacy programs. States that do not meet their performance levels under WIA are subject to sanctions.If a state fails to meet its performance levels for 1 year, Labor provides technical assistance, if requested. If a state fails to meet its performance levels for 2 consecutive years, it may be subject to up to a 5-percent reduction in its annual WIA formula grant. Under JTPA, the most stringent sanction was the possible reorganization of the local service delivery organization.

WIA Requires that Many Federal Programs Work Together to Provide Services through the One-Stop System

In addition to establishing the three new programs, WIA requires that states use the one-stop center system to provide services for these and many other employment and training programs. This system was developed by states prior to WIA through One-Stop Planning and Implementation Grants from Labor. About 17 programs funded through four federal agencies are now required to provide services through the one-stop center under WIA. Table 2 shows the programs that WIA requires to provide services through the one-stop centers (termed mandatory programs) and the related federal agency.

Department of Health and Human Services Department of Housing and Urban Development (HUD)

Under WIA, employers are expected to play a key role in establishing regional workforce development policies, deciding how services should be provided in the one-stop, and overseeing one-stop operations. Employers, who are encouraged to use the one-stop system to fill their job vacancies, are also seen as key one-stop customers under WIA.

States and Localities Are Taking Action to Implement WIA Performance Measures but Face Challenges in Doing So

States and localities are taking action to implement performance measures for the three WIA-funded programs, but they have confronted several challenges in doing so. To implement these measures, states and localities had to change the way they collected and reported performance data. Most states we surveyed had to create new automated data systems to collect and report WIA data. Many state systems, however, are still not completely in place. The lack of final guidance from Labor on how to report the data slowed the development of these systems. States and localities also faced challenges in implementing the measures due to their complexity and the resource demands they created, and some had to develop new procedures to obtain UI wage records. In addition, states faced a new negotiation process with Labor to set performance levels for each measure. Many states believe these levels are too high because little or no baseline data were used, and the negotiations did not sufficiently account for differences in economic conditions and populations served. Under WIA, performance levels are now tied to incentives and sanctions so that states can be financially rewarded if they meet them or penalized if they do not. States reported that the need to meet these performance levels may lead local staff to focus WIA-funded services on job seekers who are most likely to succeed in their job search or who are most able to make wage gains.

Management Information Systems Are Still Being Developed to Collect and Report WIA Data

As part of implementing WIA performance measures, states had to develop automated data systems to track the activities of individual WIA participants and report on performance. Based on our survey, most states developed a new automated data system, or management information system (MIS), to collect and report WIA performance data at the state level. The remaining states adapted their previous data collection systems used under JTPA. However, 15 states, regardless of whether they were developing a new system or adapting their existing system, reported that, as of August 2001, they did not have their system completely in place. All states expect to have completed their systems by July 2002. In some states, local areas do not use the state MIS to collect local WIA performance information. In these states, local areas must develop their own systems, taking time and resources to do so.

The lack of timely reporting guidance slowed the development of the data systems. Final guidance on how states must report their performance to Labor was issued to the states in March 2001—8 months after the states were required to implement major provisions of WIA and begin collecting data. Lack of final guidance resulted in delays and costly program changes as states and local areas developed and adjusted their final systems. For example, one local area we visited decided to continue using its old system and delayed the development of a new system pending final guidance because it would be too costly and time-consuming to develop a system that might need to be changed. All states, regardless of whether or not they had implemented a new system, had to make changes in their automated MIS systems to accommodate the final guidance.

Complex Performance Measures Stymied State and Local Efforts to Implement Them

States and localities reported that the complexity of WIA’s new performance measures made them difficult and time-consuming to implement. Many of the states that we surveyed commented that the measures were hard to follow because the calculations for the measures are complex and sometimes confusing, specifically who to include in the measures, when to collect the data for the measures, and how to calculate the measures.

Knowing who to include in the measures. It is difficult to know whether or not a job seeker should be counted in the measure for a program, even if he or she is served by the program. For example, a participant in the adult program who is already employed must be included in the retention and wage gain measures but cannot be counted in the entered employment measure. Yet, for the dislocated worker program, the entered employment measure can include those who may be employed when they enter the program.

Knowing when to measure performance for participants. The data for different performance measures can be collected in different quarters of the program year. For example, for customer satisfaction, data can be collected at two points in time, depending on how the participant exited the program (see fig. 1). Data on entered employment for participants in the adult program are collected in the third quarter after exit; retention data are collected in the fifth quarter after exit. Data on earnings change and replacement are collected at two points in time: pre-program earnings data can be collected at registration, and post-program earnings data are collected in the fifth quarter after exit. If a program participant does not appear in the UI wage records, local staff can collect supplemental data to establish employment for the participant, but this must be recorded within 30 days after the WIA participant is found missing from the wage records. For entered employment, staff can collect supplemental data in the fourth quarter after a participant leaves (or exits) the program, but for retention, staff can collect it in the sixth quarter after exit. Supplemental data cannot be used to measure earnings change and replacement. Because the timing of data collection is complex and can be confusing, one local area in Oregon developed a tool it calls the “bean counter” to help local staff determine when to follow-up with participants so their performance counts in the calculations (see fig. 2).

Knowing how to calculate the measures. In order to calculate the measures, states must account for a variety of factors. The type and combination of these factors determine the calculations that will be used. For example, in calculating some of the measures for the adult program, states must consider (1) whether the job seeker is employed at registration, (2) whether he or she is employed at both the first and third quarters after exit, and (3) whether the data source used to confirm employment was UI records or supplemental data. This information in various combinations results in 14 different ways that adult participants can be grouped in order to calculate the measures.

In addition to noting the complexity of the measures, state and local officials said that the new measures taxed their resources. States had to develop procedures to collect data for the new customer satisfaction measure in compliance with detailed guidance from Labor. The guidance calls for states to conduct a telephone survey from a random sample large enough to obtain 500 completed surveys from both participants and employers. Because this guidance changed over time, states had to revise their procedures accordingly. For example, a revision to the guidance issued in October 2001 required states to maintain an up-to-date list of participants’ names and addresses from which to sample—a requirement that was originally voluntary.

One indication of states’ progress in implementing these measures may be reflected in their ability to submit complete quarterly reports. Quarterly reports require data on all 17 performance measures. For the quarterly report that was due in May 2001, all states submitted their reports, but, according to Labor, only 16 states were able to provide data for all 17 performance measures. For the quarterly report that was due in August 2001—more than 1 year after WIA implementation—all states submitted their reports, but only 23 could provide data on employer customer satisfaction. According to Labor, states could not fully report on customer satisfaction because they have not yet fully implemented procedures to measure it. One state had to compile the data manually because its MIS was not fully operational.

States Adopt New Procedures to Access Sensitive Information

WIA’s new requirement that states use UI wage records to measure outcomes has led states to adopt new procedures to access these and other sensitive records. Unlike JTPA, which relied on surveys of participants to collect information on employment and earnings, WIA requires UI wage records to be used as the primary data source of employment and wage information—and the only data source for some measures, according to Labor’s guidance. To obtain employment and earnings information, states match information collected on individual WIA participants against state UI wage records. To access UI data from the state agency that oversees the UI database, some states had to establish data-sharing agreements. In Mississippi, for example, the agency responsible for overseeing WIA—the Mississippi Development Authority (MDA)—had to make arrangements with the agency that oversees the UI data—the Mississippi Employment Security Commission—to have them match the wage records and provide the results to MDA. In addition, some states may be more rigorous in protecting the confidentiality of UI records through privacy laws, which may add obstacles to collecting performance data. For example, Oregon law prohibits the release of WIA participants’ records without informed consent. Consequently, program providers had to enter into an agreement that established a protocol for collecting and sharing the data—one that developed safeguards to protect confidentiality. In addition, the state had to develop a process to ensure that WIA participants consented to the use of their protected records in this way.

States Believe Performance Levels Are Set Too High

All the states we visited believed that some of the established performance levels for their measures were set too high for them to meet—either because they were set in absence of historical or baseline data or because negotiations did not sufficiently account for variations in economic conditions or population served. States reported that limitations in available baseline data made it difficult to set fair, realistic performance levels. The new measures on credentials and customer satisfaction, for instance, had no prior data available on which to set performance levels. Where baseline data were available, such as for the wage-related measures, the data were collected under JTPA, a program whose goals were different from those of WIA. In addition, some states believe that the performance levels did not account for variations in economic conditions, such as the slow growth in new or existing businesses that some areas have experienced. Performance levels also did not account for the many economically disadvantaged or hard-to-serve individuals seeking services in some local areas.

States Report that Performance Levels May Determine Who Receives WIA-Funded Services

Many states reported that the need to meet performance levels may be the driving factor in deciding who receives WIA-funded services at the local level. All the states we visited told us that local areas are not registering many WIA participants, largely attributing the low number of WIA participants to concerns by local staff about meeting performance levels.

Local staff are reluctant to provide WIA-funded services to job seekers who may be less likely to get and keep a job. One state official described how local areas were carefully screening potential participants and holding meetings to decide whether to register them. As a result, individuals who are eligible for and may benefit from WIA-funded services may not be receiving services that are tracked under WIA.

Performance levels for the measures that track earnings change for adults and earnings replacement for dislocated workers may be especially problematic. Several state officials reported that local staff were reluctant to register already employed adults or dislocated workers. Officials in one state reported that some local areas had not yet registered any dislocated workers. State and local officials explained that it would be hard to increase the earnings of adults who are already employed or replace the wages of dislocated workers, who are often laid off from high- paying, low- skilled jobs or from jobs that required skills that are now obsolete. In addition, for dislocated workers, employers may provide severance pay or workers might work overtime prior to a plant closure, increasing these workers’ earnings before they are dislocated. As a result, many dislocated workers who come to the one-stop center have earned high wages just prior to being dislocated, making it hard to replace—let alone increase— their earnings. If high wages are earned before dislocation and lower wages are earned after job placement through WIA, the wage change will be negative, depressing the wage replacement level. As a result, a local area may not meet its performance level for this measure, discouraging service to those who may need it.

A hypothetical example involving two workers dislocated at the same time illustrates this point (see table 3). One worker is a sales clerk with limited skills earning $25,000, the other a long-time factory worker with obsolete skills earning $60,000. Both are laid off from work and go to their local one-stop center seeking job placement assistance. The clerk is placed in a new job as a receptionist paying $25,000. By calculating his wage replacement from his salary as a clerk, the one-stop can claim a wage replacement rate of 100 percent. The factory worker eventually gets a job as a security guard earning $30,000, netting a wage replacement rate of 50 percent. As this example shows, a one-stop center can meet its performance levels more easily by serving the clerk than by serving the factory worker even though both job seekers may need the one-stop system’s resources to find a job or enhance their skills.

Some states and Labor are making efforts to address this disincentive to serve certain job seekers. Indiana instituted a policy allowing local areas to adjust their dislocated worker wage replacement rate in light of the significant dislocations they are facing. Texas uses a regression model to establish local performance levels that adjust for differences in factors, such as economic conditions and the characteristics of individuals served. Without this policy, said a Texas official, WIA programs would have registered fewer workers. Similarly, Michigan substantially reduced the penalties to local areas for failing to meet performance levels and found that the number of registered participants increased as a result of instituting less threatening sanctions. WIA requires that states be allowed to renegotiate their performance levels based on unanticipated circumstances. Labor is currently developing criteria that states can use to renegotiate their performance levels based on unanticipated circumstances, such as changes in economic conditions due to plant closings or shifts in unemployment for the current and future years. The guidance is expected to be released soon.

Performance Measures May Not Accurately Assess Performance of the Three WIA-Funded Programs

Even when fully implemented, WIA performance measures may still not provide an accurate picture of performance for the three WIA-funded programs largely because data are neither comparable across states nor timely. State and local officials generally supported many of the performance measures as relevant indicators of the success of an employment and training program. However, the performance data collected and reported by states and localities are not comparable largely because of the lack of clear guidance on when to collect and report performance data and what constitutes a credential—the attainment of a certified skill or degree. In addition, while UI wage records are one of the best available sources of employment and earnings data, limitations in the data may hinder the ability of states and local areas to meet their performance levels and use the measures for short-term program management.

Performance Measures Include Relevant Indicators, but Lack of Clear Guidance Affects Accuracy and Comparability of Performance Data

State and local officials in the states we visited generally support many of the performance measures as relevant indicators of the success of an employment and training program. For example, several officials cited the wage-related measures, such as job placement, retention, and earnings change, as important indicators of a successful employment and training program. The measures are also generally consistent with the goal of WIA to help individuals get and keep jobs and increase their wages and skills. In addition, the states noted that the measures provide a good basis for long-term evaluation.

However, the performance data collected and reported by states and localities are not comparable—a critical component in creating a level playing field from which states’ relative performance can be evaluated. While there are various reasons that performance data are not comparable, one of the chief reasons is the lack of clear guidance for collecting and reporting performance data on participants. Labor has provided detailed written guidance to states on who should be registered under WIA and when this registration should occur, but the guidance is open to interpretation in some areas. The lack of a uniform understanding of when registration occurs and thus who should be counted toward the measures raises questions about both the accuracy and comparability of states’ performance data. For example, the guidance tells states to register adults and dislocated workers who need significant staff assistance designed to help with job seeking or acquiring occupational skills, but the state can decide what constitutes significant staff assistance. The guidance provides examples of when to register job seekers, but it sometimes requires staff to make subtle and subjective distinctions. For example, those who receive initial assessment of skill levels and the need for supportive services are not to be registered; those requiring comprehensive assessment or staff-assisted job search and placement assistance, including career counseling, are to be registered. In another example involving the classification of workshops, job seekers who participate are to be registered in some cases, but not in others.

Labor has allowed states and local areas flexibility in implementing the registration policy, and we found that local areas differed on when they registered WIA job seekers. In one local area we visited, the one-stop center registers most job seekers who come into the center, even if staff assistance is minimal. At this center, a general orientation is sufficient for the job seeker to be registered under WIA. In contrast, another center in the same state registers only those job seekers who require significant staff assistance and are likely to benefit from intensive services. Similar disparities occurred in other states we visited. Labor has said there is little consistency across states in registering participants and has convened a work group to develop additional guidance on registration, but as yet, the issues remain unresolved.

The lack of a definition for the credential measure is also leading to performance data that are not accurate or comparable across states. Labor allows the states and local areas to determine what constitutes a credential and to develop a statewide list of approved credentials with input from employers. Because states and, in many cases, local areas must define what constitutes a credential, what is currently counted as a credential differs within and across states. Some states may strictly define credentials to include only diplomas from accredited institutions or use only formal training completion criteria as defined by education partners. Other states may expand their criteria to count a broad variety of credentials, such as job readiness, on-the-job experience, and completion of workshops. Labor officials note that states’ performance levels for the credential measure are negotiated to take state and local definitions into account, and the measure is intended to help local employers gauge the readiness and skill level of job seekers. Nevertheless, given the broad range of definitions states and localities employ, the outcomes on the credential measure may be of limited value, even within a single state.

Challenges in Using UI Wage Record Data Also May Affect the Accuracy and Comparability of Performance Data

UI wage records are one of the best available data sources for tracking the employment and earnings of individuals—a significant improvement over the less objective self-reporting methods of JTPA—but the limitations of the database pose challenges that need to be addressed. These challenges, if unresolved, may hinder states’ ability to meet their performance levels. As we have reported in prior work, one such limitation is that UI wage records, while covering about 94 percent of workers, exclude certain employment categories, such as self-employed persons, most independent contractors, military personnel, federal government workers, and postal workers. States, therefore, must develop alternative methods to track WIA participants who are employed in these uncovered occupations. Pennsylvania, for example, developed a partnership with other states in its region to share the cost of purchasing the rights to federal civil service and military personnel data. And Florida has developed agreements with the Department of Defense, the Office of Personnel Management, and the U.S. Postal Service to access employment and wage information on an annual basis. Our survey data indicate that 33 states are using additional or supplemental data to compensate for uncovered occupations, with only 27 of those using the supplemental data to count towards their performance levels. Thus, at least 23 states have not used additional data to help them meet their performance levels.

Another limitation is that state UI databases include only wage record information on job seekers who get jobs within their state; they do not track job seekers who find jobs in other states. States cannot readily access UI wage records from other states to track outcomes under WIA, making it difficult to track individuals who receive services in one state but get a job in another. Over one-third of all of the states we surveyed reported that an estimated 16 to 30 percent of cases are not being picked up by their state’s UI wage record system. To fill in these gaps, seven states have agreements with other states—often those that share a common border—to exchange UI information. Indiana, for example, established an agreement with Illinois to trade data. If data are missing on particular participants, Illinois sends the cases to Indiana to see if the Indiana UI wage records have information on the job seeker. The value of these agreements, however, may be limited because job seekers may find work in a state that does not have an agreement with the one in which they received services.

Another way to obtain UI data on workers who are employed out of state is through WRIS, a clearinghouse that makes UI wage records available to states seeking employment and wage information on their WIA participants. This information can provide outcome data on WIA participants to help states meet their performance levels. While WRIS was available for states to use by July 2001, only 7 out of the 50 states are currently able and ready to participate, with 8 others in various stages of completing the requirements for participation. Although many states have shown an interest in a system such as WRIS, many are reluctant to participate because Labor, while agreeing to cover all the costs of operating WRIS for its first year, has not yet agreed to pay for subsequent years. The estimated total cost of operating WRIS is $2 million annually, but states have not been given a definitive answer about how much it would cost them to participate after this first year if Labor does not continue funding. Because of this uncertainty regarding future costs, states are hesitant to commit to participation in WRIS. If not all states participate, the value of WRIS will be diminished—even for participating states—because no data will be available from nonparticipating states’ UI wage records.

Lack of Timely Data Limits Use of Performance Measures for Short-Term Program Management

The lack of timely data, due to the time lag in obtaining UI wage records, makes it difficult for state and local officials to use the performance measures for short-term program management—because, for the wage- related measures, current available data on the measures will reflect performance from the previous program year. While UI wage records are the best available data source for documenting employment, the data collection and reporting process is slow and time-consuming. Data are generally collected from employers only once every quarter, and employers have 30 days after the quarter ends to report the data. In many states, employers—especially small businesses—are allowed to submit data in paper format, which then must be converted to electronic media. After data entry, information must be checked for errors and corrected. All of these steps take time. As a result, WIA program administrators are unable to get a timely picture of program performance. For example, we asked states in our survey how quickly job placement outcome data would be available to them from UI wage records. On the basis of our survey, we found that for 30 states, the earliest time period that job placement data would be available is 6 months or more after an individual entered employment, with 15 states reporting that it may take 9 months or longer. Similarly, for the employment retention measure, over half of states report that obtaining this information could take a year or longer. (See fig. 3.)

The time delay in receiving UI wage record data makes it difficult for state and local officials to use the performance measures to gauge the effectiveness of their services. States report that not being able to get performance results in the same program year is a problem: it makes it difficult to manage programs and improve one-stop services. Labor reports that the performance measures are not intended to be a management tool. State and local officials, therefore, must develop alternative methods if they want to assess the quality of their services so they can identify problems and improve programs in a timely way. Labor has encouraged these efforts, but, while some local areas are finding ways to collect data to help them manage their programs, there is no cohesive effort at the federal level to share strategies and promising approaches for the Adult and Dislocated Worker Programs.

Existing Performance Measures Fail to Gauge Overall One- Stop Performance

Although there are performance measures for the three WIA-funded programs and most of the programs required to provide services in the one-stop, no measures exist to assess how well the overall one-stop systems are working. The success of the one-stop system as a whole is not captured by the program measures of individual one-stop partners. Furthermore, combining the performance measures from mandatory programs does not provide a comprehensive picture of one-stop performance. Even when measures appear the same, comparing them is difficult because of differences in definitions and calculations. Beyond failing to provide a complete evaluation of one-stop performance, state officials reported that the separate reporting requirements of the partner programs have hampered coordination within their one-stop systems. While WIA did not establish any comprehensive measures to assess the overall one-stop system, it required that Labor take the lead in developing optional measures to help states assess progress toward their workforce investment goals. Labor has made limited progress on such performance measures, and only a few states have developed their own overall measures.

Existing Performance Measures Fail to Capture Important One-Stop Features

The existing performance measures for participating one-stop programs fail to capture important one-stop features. First, it is difficult to get an unduplicated count of job seekers using the one-stop. While an individual may have multiple outcome measures for the services received through each of the programs at a one-stop, there is no single outcome measure for multiple services. In addition, separate reporting systems for each of the programs make it difficult to disaggregate data and track an individual’s outcome for those receiving multiple services.

Second, other important aspects of one-stop performance are not included within the existing measures. Customer satisfaction measures used in support of WIA-funded programs and the Employment Service fail to measure how job seekers and employers believe they are being served by the one-stop system as a whole. Instead, these measures show satisfaction with the individual programs. Employer satisfaction is important under WIA because WIA created a more private-sector driven system. Capturing customer satisfaction of the system as a whole would reflect whether job seekers are successful at attaining the services they need to get jobs and would assess whether employers are satisfied with job applicants sent to them from different one-stop programs.

Finally, state and local officials expressed concern that a large portion of one-stop participants are not included in performance measures. Many job seekers use self-service and informational activities but they are not tracked and counted in any program measures. While staff time and resources are used to establish and maintain self-service resource rooms and web sites, job seekers who use only these services will not be included in any of the performance measures. Without any information on individuals who use self-service, it will be difficult for Labor to show how effectively one-stops are being used.

Multiple Measures Cannot Be Used to Measure the One-Stop System’s Performance

Performance measures for different programs often track similar outcomes, as figure 4 shows. However, the measures cannot be combined to obtain an overall view of one-stop performance. Although the same terms are used in various performance measures, their definitions are not identical. For example, while WIA older and younger youth programs define youth as being between the ages of 14 and 21, the laws governing Job Corps and HUD’s Youthbuild define youth as being between the ages of 16 and 24. Similarly, the definition of veterans is different for the Employment Service and Veterans’ Employment and Training Service program. The differences in definitions mean that assessing the outcomes for youth or veterans by combining the performance measures of individual programs within the one-stop setting would be difficult.

Besides variations in definitions, there are also variations in how measures are calculated in different programs. For example, while the entered employment rate for WIA’s adult program is defined as the percentage of workers who get a job by the end of the first quarter after exit, the entered employment rate for the Employment Service is defined as the percentage of workers who get a job or changed employers in the first or second quarter after registration. As a result, performance data from these separate programs cannot be combined to yield a single overall score to assess various performance outcomes of the one-stop system. The Office of Management and Budget (OMB) convened a work group representing federal WIA partners to look at common definitions and measures across programs, which may issue guidance to states and localities on WIA performance measures and federal requirements.

Separate Measures May Impede Coordination

Beyond failing to provide an overall picture of one-stop performance, state officials reported that separate performance measures impede the cooperation of one-stop partners. As a result, even though WIA was meant to establish a more coordinated workforce development system through the use of one-stops, over one-third of states surveyed expressed concern that individual program performance measures may impede this process. Some states even believed that separate measures caused competition among programs. They said that if staff did not understand that a participant could be counted in more than one program, they might not direct them to other one-stop programs. For example, one state reported in its written comments to our survey that competition for participants and duplication of services due to lack of coordination with other programs would continue as long as each program is required to meet its own performance and participation levels. Fourteen states volunteered in their written comments to our survey that the federal government should work to coordinate performance measures across programs or develop systemwide measures. In addition, while states agree that systemwide measures are needed, they caution against making any additional measures mandatory, since states are still adapting to the existing measures.

Limited Progress on Developing Performance Measures for One-Stop Systems

Although WIA did not establish one-stop measures, it does require that Labor develop additional optional measures to assist states in assessing progress toward their workforce investment goals, which Labor has interpreted to include one-stop measures. Labor began developing workforce development performance measures to capture overall one- stop use after one-stop systems were piloted. Since the passage of WIA, Labor has continued its efforts to develop systemwide measures, but it has made limited progress. Labor convened a working group in September 2001 to develop additional indicators of one-stop performance. Partner representatives at this group included national workforce-related organizations including the National Association of State Workforce Agencies, National Governors’ Association, National Association of Counties, U.S. Conference of Mayors, and representatives of states and regional boards. This group is working to develop a menu of indicators that will help provide a comprehensive picture of WIA system activity. Such measures may include capturing information on self-service customers and the cost of services at the one-stop.These measures, while optional, would help provide information on overall one-stop use across the country if all states report on at least some of the measures. Labor plans to have guidelines for these optional indicators in place for use in program year 2002, which begins on July 1, 2002. In order for states to be able to implement them for the coming program year, Labor will need to provide final guidance well before July 2002.

In addition to these national efforts, some states, on their own initiative, have attempted to develop additional measures for one-stop systems, but these efforts are not coordinated and do not allow for nationwide assessment of the one-stop system. According to our survey, eight states have created or are developing additional systemwide measures, but of those, only three are reporting them to Labor. Pennsylvania, for example, developed five measures specific to its one-stop system’s performance. These indicators, intended to measure the overall effectiveness of the one- stop system, include median cycle time to fill a job, and the percentage of employers and individuals using services through the one-stop. Florida, on the other hand, has developed “tiers” of measures that focus on the outcome of their workforce development programs. In the first tier, state- generated systemwide indicators measure many employment and training programs together. The second tier clusters similar types of programs and captures measures relevant to particular groups (e.g., continued education status of youth in youth programs). The third tier captures all the federally mandated measurements, as well as measures for the other tiers, such as caseloads for specific programs. In this way, Florida has attempted to measure the system overall as well as outcomes for individual programs. Measures developed by other states include the number of people using the resource rooms at one-stops and the increase over time in the number of unemployed people getting a job. Despite these states’ efforts, the absence of nationally established systemwide measures means that Labor cannot ensure nationwide comparability.

Conclusions

WIA represents a fundamental shift in the way federally funded employment and training services are provided to job seekers and in the way WIA programs measure and monitor success. Despite obstacles, in just over a year states have made good progress in implementing the new requirements under WIA—developing new processes and designing new systems. Labor, for its part, has been working to find ways to allow states and localities greater flexibility to design their programs to meet local needs and has been actively seeking opportunities for states to have input into the process, particularly in the area of performance measurement. But given the challenges states have faced in implementing the new performance measurement system, more time is needed before the measures can meaningfully gauge the success of the programs.

This new performance measurement system under WIA is a high-stakes game—a state’s future funding and, therefore, its ability to serve its citizens may depend upon how well it performs compared to how well it is expected to perform. It should be no surprise that states and localities are designing their systems and processes in ways that will enhance their ability to meet their performance levels. Because states see the current performance levels as too high for the current economy, states and localities may choose not to serve those job seekers who may be helped by their services, but who may not help in achieving their negotiated performance levels. Unless the performance levels can be adjusted to truly reflect differences in economic conditions and the population served, local areas will continue to have a disincentive to serve some job seekers that could be helped.

WIA’s requirement to use UI data to track outcomes is a step in the right direction—it provides federal, state, and local government entities with an objective means to evaluate program success. But it brings challenges that need to be addressed, and states will need help to do so. Establishing the means to routinely share data across state lines through WRIS and developing ways to share promising approaches in the use of supplemental data sources and in managing the assessment of short-term program needs would go far in moderating these challenges. Without this help and the cooperative efforts of states and localities toward this end, developing a useful performance measurement system will take longer and cost more.

In general, WIA’s performance measurement system captures some useful information, but it may not capture all the right information. The measure to track credentials has limited value because it lacks a standard definition for what’s being measured. For other measures, the lack of clear definitions for whom to track limits their usefulness in drawing conclusions about program success at both the state and national levels. Without clear definitions and processes, the measures will not provide the Congress with a true picture of how well the programs are performing. Furthermore, WIA performance measures gauge only WIA-funded services; yet there is widespread agreement that measures are needed to gauge the effectiveness of the entire one-stop system. The system’s narrow focus on program outcomes for a limited number of participants misses a key requirement of WIA to support the movement toward a coordinated system. In fact, the measures may foster the opposite—a siloed approach that encourages competition among programs and limits their cooperation. Without global one-stop measures, the Congress will not be able to assess how well states and localities are doing in meeting WIA’s requirement to coordinate services. The lack of such measures may, instead, send a signal to states that service coordination is a minor goal.

Recommendations for Executive Action

To give states and local areas more time to implement WIA performance measures and establish baseline data needed to determine performance levels, we recommend that the Department of Labor delay the application of financial sanctions for at least 1 year or until it is judged that states have their data systems sufficiently in place to successfully track WIA outcomes.

To eliminate possible disincentives to serve some job seekers and ensure that states and local areas will not be unduly penalized for economic downturns, we recommend that the Department of Labor expedite the release of guidance on revising negotiated performance levels and allow states to immediately begin the process of re- negotiation.

To ensure uniformity in data collection and reporting so that performance results are more accurate and comparable across states, we recommend that the Department of Labor provide clearer guidance using objective criteria on who should or should not be registered as a WIA participant for tracking purposes and, once the guidance is released, work proactively with states to implement it, and issue guidance delineating a clear definition for what constitutes a credential, and, once the guidance is released, ensure that states use it to report on this indicator.

To help states address the challenges of using UI data to measure outcomes, we recommend that the Department of Labor continue to fully fund the Wage Record Interchange System in order to facilitate the sharing of UI data across state lines; and develop ways for states to share promising approaches in the use of supplemental data sources in closing the data gaps for covered and uncovered employment in UI; and develop ways for states to share promising approaches that help states address the UI timeliness issue, providing methods to help states monitor and improve their programs in a timely manner.

To help states measure one-stop performance, we recommend that the Department of Labor ensure that the development of optional one-stop system measures is completed in enough time for states to implement them at the beginning of program year 2002.

Agency Comments

We provided a draft of this report to Labor for its review and comment. Labor’s comments are in appendix II. We incorporated comments and clarifications where appropriate.

Labor generally agreed with our findings and recommendations, noting that they are consistent with information they have gathered from state and local partners. In its comments, Labor expressed concern that negotiated performance levels may be determining who receives WIA- funded services, indicating that it will work with states and local areas to address this issue. Labor also commented on our finding regarding the lack of clear guidance on certain policies, stressing the importance of state and local flexibility in determining specific policies and practices to fit local needs. While state and local flexibility is important, we continue to be concerned that the lack of a uniform understanding of when registration occurs and what constitutes a credential raises questions about both the accuracy and the comparability of states’ performance data. We are pleased to note that Labor is in the process of reviewing this issue. Finally, Labor cites its efforts to collaborate with states and local areas in developing a performance accountability system and increasing partnerships. We commend Labor for obtaining states’ input and participation in developing such a system.

We are sending copies of this report to the Secretary of Labor, relevant congressional committees, and others who are interested. Copies will also be made available to others upon request. The report is also available on GAO’s home page at http://www.gao.gov.

Please contact me on (202) 512-7215 if you or your staff have any questions about this report. Other major contributors to this report are listed in appendix III.

Appendix I: WIA Performance Measures for Adults, Dislocated Workers, and Older Youth

The following table includes descriptions of the performance measures reviewed in this report. It does not include the calculations required to attain the final value of the performance measures.

Definition Of those who did not have a job when they registered for WIA, the percentage of adults who got a job by the end of the 1st quarter after exit. This measure excludes participants who are employed at the time of registration. Of those who had a job in the 1st quarter after exit, the percentage of adults who have a job in the 3rd quarter after exit. Of those who had a job in the 1st quarter after exit, the post-program earnings increases as compared with pre-program earnings. Of those adults who received WIA training services, the percentage who were employed in the 1st quarter after exit and received a credential by the end of the 3rd quarter after exit.

The percentage of dislocated workers who got a job by the end of the 1st quarter after exit. This measure includes dislocated workers who are employed at the time of registration. Of those who had a job in the 1st quarter after exit, the percentage of dislocated workers who have a job in the 3rd quarter after exit. Of those who had a job in the 1st quarter after exit, the percentage of pre-program earnings being earned post-program. Since it may be difficult to find dislocated workers jobs with equivalent or better wages, this measure captures the percentage of earnings of the new job in relation to the old. Of those dislocated workers who received WIA training services, the percentage who were employed in the 1st quarter after exit and received a credential by the end of the 3rd quarter after exit.

Of those who are not employed at registration and who are not enrolled in post-secondary education or advanced training in the 1st quarter after exit, the percentage of older youth who have gotten a job by the end of the 1st quarter after exit. This measure also excludes youth that move on to post-secondary education or advanced training and not employment. Of those who are employed in the 1st quarter after exit and who are not enrolled in post- secondary education or advanced training in the 3rd quarter after exit, the percentage of older youth that are employed in the 3rd quarter after exit. Of those who had a job in the 1st quarter after exit and who are not enrolled in post- secondary education or advanced training, the post-program earnings increases as compared with pre-program earnings. The percentage of older youth who are in employment, post-secondary education, or advanced training in the 1st quarter after exit and received a credential by the end of the 3rd quarter after exit.

The average of three statewide survey questions rated 1-10: (1 being “very dissatisfied” and 10 being “very satisfied”): was the employer satisfied with services Definition did the service meet the expectations of the customer how well did the service compare to the ideal set of services The average of three statewide survey questions rated 1-10: was the participant satisfied with services did the service meet the expectations of the customer how well did the service compare to the ideal set of services A statewide telephone survey of a sample of 500 is conducted for all the WIA-funded programs.

Appendix III: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

Abbey Frank, Mikki Holmes, and Amanda Ahlstrand made significant contributions to this report. In addition, James Wright assisted in the study design and the national survey; Jessica Botsford and Richard Burkard provided legal support; and Patrick DiBattista and Barbara Alsip assisted in the message and report development.

Related GAO Products

U.S. General Accounting Office. Workforce Investment Act: Better Guidance Needed to Address Concerns Over New Requirements. GAO-02- 72. Washington, D.C.: 2001.

U.S. General Accounting Office. Veterans’ Employment and Training Service: Proposed Performance Measurement System Improved, But Further Changes Needed. GAO-01-580. Washington, D.C.: 2001.

U.S. General Accounting Office. Multiple Employment and Training Programs: Overlapping Programs Indicate Need for Closer Examination of Structure. GAO-01-71. Washington, D.C.: 2000.

U.S. General Accounting Office. Workforce Investment Act: Implementation Status and the Integration of TANF Services. GAO/T- HEHS-00-145. Washington, D.C.: 2000.