Workforce Investment Act: States and Local Areas Have Developed Strategies to Assess Performance, but Labor Could Do More to Help
Published: 2004-06-01T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: With rising federal deficits and greater competition for public resources, it is increasingly important for federal programs, such as the Workforce Investment Act (WIA) programs, to show results. This report examines (1) how useful WIA performance data are for gauging program performance; (2) what local areas are doing to manage their WIA performance and assess one-stops on a timely basis, and how states assist these efforts; and (3) the extent to which the Department of Labor is trying to improve WIA's performance measurement system and assess one-stop success.
section_title: What GAO Found
paragraphs: WIA performance data provide a long-term national picture of outcomes, but these data offer little information about current performance and represent a small portion of job seekers who received WIA services. Unemployment Insurance wage records--the primary data source for tracking WIA performance--provide reliable outcome information over time. But they have shortcomings, such as not including some categories of workers, and considerable time lags before data are available. Many states rely on alternative data sources to fill gaps in the wage records. However, the time between when a participant receives services and when their outcomes are reported to Labor can range from about 1 1/2 to 2 1/2 years or longer. In addition, states' annual reports reflect only a small portion of job seekers who receive WIA services because of restrictions in the law and policies of Labor. With assistance from states, many local areas collect interim outcome information from former participants or employers and use other interim indicators to track WIA performance levels long before wage record data are available. However, states and local areas would like more help from Labor in disseminating best practices on interim performance measures. In addition, these efforts tell them little about the performance of their overall one-stop systems. Many states and local areas rely on other indicators--job seeker measures, employer measures, program partnership measures, and family and community indicators to assess their one-stops. Labor has taken steps to improve WIA's performance system and assess onestops, but could do more. Although Labor is studying adjustment methods that could better take into account local differences when negotiating performance levels, it has not committed to using such a method nationally. Labor also has efforts to improve the quality of WIA's performance data and is developing a set of common measures for one-stop partner programs. Yet as part of the common measures, Labor plans to restrict the use of alternative data. Labor has also delayed plans to conduct an impact evaluation and will not meet its statutory requirement to do so by 2005.

section_title: Letter
paragraphs: As the nation faces rising budget deficits and greater competition for  federal resources to address 21st century challenges, it is becoming  increasingly important for federal programs to demonstrate that they are  meeting their long-range goals. Policy makers at all levels are seeking to  understand whether our current employment and training system is  making a difference in helping people get and keep jobs. Programs whose  services are provided through the Workforce Investment Act’s (WIA) one- stop centers constitute the largest portion of federal employment and  training funds that provide direct assistance, with the three programs  funded under WIA—Adults, Dislocated Workers, and Youth—totaling  about $3.3 billion in fiscal year 2004. With the passage of WIA in 1998,  lawmakers envisioned a consolidated new system for delivering most  federally funded employment and training programs—one that was more  efficient and effective than prior programs. To assess whether it is  accomplishing its goals, WIA established a rigorous performance  accountability structure for the programs directly funded by WIA—one  that emphasized outcomes in areas of job placement, retention, earnings,  and skill attainment, as well as customer satisfaction. From federal long- range goals to local program policies, it is critical to demonstrate results at  all levels with accurate outcome data, timely program management  information, and impact studies to assess effectiveness. But we and  others have raised questions about whether anyone has a clear picture of  what WIA-funded programs are achieving.
Because you were concerned whether WIA’s performance reporting  system allows for a useful and accurate assessment of program  achievements and to better understand how the performance system  allows for local assessment of program successes, we examined (1) how  useful WIA performance data are for gauging program performance,   (2) what local areas are doing to manage their WIA performance and  assess one-stop success on a timely basis and how states are assisting  these efforts, and (3) to what extent the Department of Labor is trying to  improve WIA’s performance measurement system and assess one-stop  success.
To learn more about what the data tell us about outcomes achieved under  WIA, we examined states’ annual reports, interviewed Labor officials, and  reviewed federal performance and reporting requirements. In addition, to  determine how states and local areas track interim progress toward  meeting their WIA performance levels that were negotiated with Labor and  to understand what they use to assess overall one-stop success, we  surveyed the 50 states, as well as all 568 local workforce investment areas.  We received responses from all 50 states and 463 local workforce  investment areas (81.5 percent). We also visited three states—Florida,  Michigan, and Utah—and at least two local areas or one-stops in each of  these states. We selected these states because they are geographically  diverse, have implemented additional performance measures or strategies  to assess one-stop success, and have developed integrated statewide data  systems. We supplemented our site visits with telephone interviews with  state and local officials in Pennsylvania and interviews with experts in the  area of workforce development performance measurement. Our review  focused primarily on the employment-based measures that are tracked or  partially tracked with the Unemployment Insurance wage records— entered employment rate, earnings change/replacement rate, employment  retention rate, employment and credential rate, and the younger youth  placement and retention rate. We conducted our work between April 2003  and April 2004 in accordance with generally accepted government auditing  standards.
subsections: section_title: Background
paragraphs: Labor required states to implement major provisions of WIA by July 1,  2000, although some states began implementing provisions of WIA as early  as July 1999. Services provided under WIA represent a marked change  from those provided under the previous program, allowing for a greater  array of services to the general public. WIA requires that many federal  programs provide employment and training services through one-stop  centers. WIA is designed to provide for greater accountability than under  previous law: it established new performance measures, a requirement to  use Unemployment Insurance (UI) wage data to track and report on  outcomes, and a requirement to conduct at least one multi-site control  group evaluation.
subsections: section_title: WIA-Funded Services
paragraphs: When WIA was enacted in 1998, it replaced the Job Training Partnership  Act (JTPA) programs for economically disadvantaged adults and youth  and for dislocated workers with three programs—WIA Adult, Dislocated  Worker, and Youth—that provide a broader range of services to the  general public, no longer using income to determine eligibility for all  program services. WIA programs provide for three tiers, or levels, of  service for adults and dislocated workers: core, intensive, and training.  Core services include basic services such as job searches and labor market  information. These activities may be self-service or require some staff  assistance. Intensive services include such activities as comprehensive  assessment and case management—activities that require greater staff  involvement. Training services include such activities as occupational  skills or on-the-job training. Labor’s guidance provides for monitoring and  tracking for the adult and dislocated worker programs to begin when job  seekers receive core services that require significant staff assistance. WIA  excludes job seekers who receive core services that are self-service and  informational in nature from being included in the performance measures.  WIA’s youth program does not have three tiers of services, but instead  requires that 10 youth services, referred to as program elements, be made  available to all eligible youth. All youth who are determined eligible and  receive WIA services are included in the performance measures.
section_title: WIA Performance Measures and a Required Evaluation Are Designed to Increase Accountability for Three WIA-Funded Programs
paragraphs: WIA is designed to provide for greater accountability than its predecessor  program by establishing new performance measures, a new requirement to  use UI wage data to track and report on outcomes, and a requirement for  Labor to conduct at least one multi-site control group evaluation.  According to Labor, performance data collected from the states in support  of the measures are intended to be comparable across states in order to  maintain objectivity in determining incentives and sanctions. The  performance measures also provide information to support Labor’s  performance goals under the Government Performance and Results Act  (GPRA), the budget formulation process using the Office of Management  and Budget’s (OMB) Program Assessment Rating Tool (PART), and for  program evaluation required under WIA.
In contrast to JTPA, for which data on outcomes were obtained through  follow-ups with job seekers, WIA requires states to use UI wage records to  track employment-related outcomes. Each state maintains UI wage  records to support the process of providing unemployment compensation  to unemployed workers. The records are compiled from data submitted to  the state each quarter by employers and primarily include information on  the total amount of income earned during that quarter by each of their  employees. Although UI wage records contain basic wage information for  about 94 percent of workers, certain employment categories are excluded,  such as self-employed persons, independent contractors, federal  employees, and military personnel. According to Labor’s guidance, if a  program participant does not appear in the UI wage records, states may  use supplemental data sources, such as follow-up with participants and  employers, or other administrative databases, such as U.S. Office of  Personnel Management or U.S. Department of Defense records, to track  most of the employment-related measures. However, only UI wage records  may be used to calculate earnings change and earnings replacement. (See  table 1 for a complete list of the WIA performance measures and data  sources used for tracking the measures.)
Unlike JTPA, which established expected performance levels using a  computer model, WIA requires states to negotiate with Labor to establish  expected performance levels for each measure. States, in turn, must  negotiate performance levels with each local area. The law requires that  these negotiations take into account differences in economic conditions,  participant characteristics, and services provided. To derive equitable  performance levels, Labor and the states primarily rely on historical data  to develop their estimates of expected performance levels. These  estimates provide the basis for negotiations.
WIA holds states accountable for achieving their performance levels by  tying those levels to financial sanctions and incentive funding. States that  meet their performance levels under WIA are eligible to receive incentive  grants that generally range from $750,000 to $3 million. States that do not  meet at least 80 percent of their WIA performance levels are subject to  sanctions. If a state fails to meet its performance levels for 1 year, Labor  provides technical assistance, if requested. If a state fails to meet its  performance levels for 2 consecutive years, it may be subject to up to a 5- percent reduction in its annual WIA formula grant. At the end of program  year 2001, four states received financial sanctions.
Labor determines incentive grants or sanctions based on the performance  data that states must submit each December in their annual reports.  States also submit quarterly performance reports, which are due 45 days  after the end of each quarter. In addition to the performance reports,  states submit their updates for the Workforce Investment Act  Standardized Record Data (WIASRD) every January. All three submissions  primarily represent participants who have exited the WIA programs within  the previous program year (July 1 – June 30).
WIA also requires Labor to conduct at least one multi-site control group  evaluation by the end of fiscal year 2005. WIA requires that evaluations  address the general effectiveness of programs and activities in relation to  costs and the impact of these services on the community and participants  involved.
section_title: WIA Requires That Many Federal Programs Work Together to Provide Services through the One- Stop System
paragraphs: WIA requires that states use the one-stop center system to provide  services for many employment and training programs. Seventeen  programs funded through four federal agencies are now required to  provide services through the one-stop center under WIA. Table 2 shows  the programs that WIA requires to provide services through the one-stop  centers (termed mandatory programs) and the related federal agencies.
Table 2. WIA’s Mandatory One-Stop Partner Programs and Related Federal  Agencies  Employment Service (Wagner-Peyser)
Veterans ‘employment and training programs  Senior Community Service Employment Program  Employment and training for migrant and seasonal    farm workers  Employment and training for Native Americans  Vocational Education (Perkins Act)
Department of Health and  Human Services  Department of Housing and  Urban Development (HUD)
Under WIA, employers are expected to play a key role in establishing  regional workforce development policies, deciding how services should be  provided in the one-stop, and overseeing one-stop operations. Employers,  who are encouraged to use the one-stop system to fill their job vacancies,  are also seen as key one-stop customers under WIA.
section_title: WIA Performance Data Provide a Long- Term National Picture of Outcomes, but Are Less Useful for Gauging Current Performance and Represent a Small Portion of WIA Participants
paragraphs: WIA performance data are useful for providing a long-term national  picture of program outcomes; however, these data are less useful for  providing information about current performance, and represent only a  small portion of job seekers that received WIA services. UI wage  recordsthe primary data source for tracking WIA performance—provide  a fairly consistent national view of WIA performance and allow for  tracking outcomes over time. At the same time, the UI wage records have  some shortcomings—they cannot be used to track job seekers who get  jobs in other states unless states share data; they do not cover certain  categories of workers, such as self-employed persons; and they are not  available on a timely basis. States are making progress in overcoming  some of these shortcomings by sharing wage data with other states and  supplementing information on participants not covered by the wage data.  Despite this progress, time lags and other factors affect the timing of  states’ reports on their annual performance to Labor and, subsequently,  Labor’s reports to Congress. Most of the outcomes data reported in a given  program year actually reflect participants who left the program during the  prior year, limiting usefulness for gauging current program performance.  In addition, the states’ annual reports reflect only a small portion of job  seekers who receive WIA services because, under the law and Labor’s  guidance, not all job seekers who utilize one-stop services are required to  be included in the performance reports.
subsections: section_title: WIA Annual Performance Data Are Useful for Measuring Outcomes Over the Long Term
paragraphs: WIA annual performance reportswhich provide a summary of states’  performance on the 17 core measuresare useful for providing a national  perspective of outcomes achieved over time. The information presented in  the annual reports compares states’ negotiated performance levels with  their actual performance levels. (See table 3 for an example of national  performance levels for WIA’s job placement ratecalled the entered  employment ratein program year 2002.) These reports provide Congress  with an annual picture of how well the WIA program is meeting its long- range goals to increase the employment, retention, and earnings of  participants. The WIA performance data are also useful to help Labor  assess quantitative, outcomes-oriented goals for its strategic plans, annual  performance plans, and annual performance reports required by the  Government Performance and Results Act (GPRA). In its annual  performance report for program year 2002, Labor used the WIA outcome  measures to assess its progress in meeting its strategic goals to increase  employment, earnings, and assistance to adults and increase the number  of youth in education or making a successful transition to work.
Most of the performance outcomes in the annual reports are measured  using UI wage records13 of the 17 WIA performance measures rely on  UI wage records as the primary data source for tracking employment  outcomes. (See table 4.) States maintain UI wage records to determine  whether unemployed workers qualify for unemployment compensation.  The records are compiled from data submitted to the state each quarter by  employers and primarily include information on the total amount of wages  paid to employees in the quarter. However, UI wage records for most  states do not include information on the number of hours an employee  worked during the quarter and when in the quarter the wages were earned.  For example, the UI wage records for most states would not show that one  employee may have worked 40 hours a week for the entire quarter and  another worker may have worked 35 hours a week for the last two weeks  of the quarter. The UI wage records would provide an overall snapshot of  the total amount of wages paid to both employees for the quarter.
The UI wage records provide a common yardstick for long-term  comparisons across states because they contain wage and employment  information on about 94 percent of the working population in the United  States, and all states collect and retain these data. In addition, UI wage  records can be used as a common data source to track employment  outcomes across multiple programs, such as vocational education and the  Temporary Assistance for Needy Families (TANF)  programs. Further,  researchers have found that wage record data are more objective and cost- effective than traditional survey information. For example, one state  estimated that the cost of doing participant surveys, as was done under  JTPA, was approximately $13.25 per participant compared with the cost of  automated record matching to UI wage records, which costs less than $.05  per participant. UI wage records make it easier to track longer-term  measures, such as those that assess earnings change, earnings  replacement, and employment retention 6 months after participants leave  the program. Without UI wage records, tracking these outcomes would  require contacting or surveying former participants, perhaps multiple  times, after they leave the program.
section_title: UI Wage Records Have Some Shortcomings, and Many States Rely on Supplemental Data to Report on Their Performance
paragraphs: UI wage records also have some shortcomings. State wage record  databases only include wage information on job seekers within their state;  they do not track job seekers who find jobs in other states. States cannot  readily gain access to UI wage records from other states, making it  difficult to track individuals who receive services in one state but get a job  in another. To help gain access to wage information in other states, Labor  established the Wage Record Interchange System (WRIS) a  clearinghouse that makes UI wage records available to states seeking  employment and wage information on their WIA participants, and states  are increasingly making use of this option. Nationwide, 38 states reported  that they currently participate in WRIS, an increase from the 15 states  that told us they were planning to or participating in WRIS in 2001. States  may also elect to establish their own agreements to share wage  information with other statesoften those that share a common border.  Seven of the 38 states reported that they maintain their own interstate  agreements with other states and they also participate in WRIS. One state  official we interviewed said the state maintains its own agreements in  addition to WRIS so that the state can get data more quickly than through  WRIS. According to a Labor official, states often retrieve wage record data  from other states within a matter of days using WRIS. However, the  process can take much longer up to a couple of weeksif participating  states take longer to respond to requests.
In addition, even though UI wage records contain information on about 94  percent of workers, they do not contain information on certain  employment categories of workers, such as self-employed persons, most  independent contractors, military personnel, federal government workers,  and postal workers. To compensate for the 6 percent of workers who are  not in the UI wage records, Labor allows states to report employment  outcomes using other data sourcesfor example, by contacting  participants after they leave the programto track WIA participants who  are employed in these uncovered occupations. We found that 39 states  reported relying on this supplemental information to report on  participants not covered by the wage data. Twenty-three states told us that  without the supplemental data, they would not have been able to show  that they met minimum performance levels on at least one measure, and  10 of these states said they would not have been able to show that they  met minimum performance levels on 10 of the measures in program year  2001. (See fig. 1.)
Labor also allows states to use other employment and administrative data  sources to track employees excluded from the UI wage records, such as  the U.S. Office of Personnel Management, the U.S. Postal Service, and the  U.S. Department of Defense. Eight states reported that they currently fill  gaps in coverage using other administrative and employment data sources.  Labor has recently established an agreement with the U.S. Office of  Personnel Management and is working on agreements with the U.S.  Department of Defense and the U.S. Postal Service to obtain employment  data through a clearinghouse similar to WRIS to help more states obtain  this outcome data. Labor plans to begin testing this new clearinghouse in  program year 2004. (See app. II for a detailed listing of states’ use of UI  wage records and other data sources for reporting on WIA outcomes.)
section_title: Time Lags Affect When Outcomes Are Reported and Limit Data’s Usefulness for Gauging Current Performance
paragraphs: UI wage records also suffer significant time delays between the time an  individual gets a job and the time it appears in the UI wage records. State  procedures for collecting and compiling wage information from employers  can be slow and time-consuming. Data are collected from employers only  once every quarter, and employers in most states have 30 days after the  quarter ends to report the data to the state. For example, the wage report  for the last calendar quarter of the year (ending on December 31) is due to  the state on January 31. After the state receives the wage report, the data  must be processed. Many employers report the data electronically, but  some employersespecially small businessesare allowed to submit data  in paper format, which then must be converted to electronic media. After  data entry, information must be checked for errors and corrected. All  these steps take time, which can delay the availability of the wage record  data for reporting on outcomes for several months. According to our  survey, 28 states estimated they get information on job placement within 4  months after participants exit the program, and 44 states have this  information within 6 months. (See fig. 2.)
The time lags in receiving wage data, together with the use of longer-term  outcome measures, affect when outcomes are reported and limit the data’s  usefulness for gauging current performance. All 13 of WIA’s employment- related outcomes are measured after participants leaveor exitthe  program, and some measures, such as those that assess wage changes and  employment retention, require a 6-month wait. To compensate for time  lags, Labor devised a reporting structure that reaches back to the prior  year to provide a complete year’s worth of outcome data on WIA  participants for the annual reports. For example, for the employment- based measures, participants who are reported on in the program year  2002 annual report, provided to Labor in December 2003, left WIA in the  four quarters between October 2001 and September 2002 and may have  received services much earlier. The amount of time between when  participants receive services and when their outcomes are reported to  Labor varies, but it is about 1½ years at a minimum. A hypothetical  example will illustrate this point by showing two participants that would  be included in the program year 2002 report. Sue registered in April 2001,  participated in the program for at least 6 months, and left between  October and December 2001, taking about 32 months from the time of  registration until her outcomes were reported. Joe, on the other hand, did  not register until July 2002, participated and left the program within 3  months, taking about 17 months from the time of registration until his  outcomes were reported. (See fig. 3.)
section_title: WIA Performance Data Represent a Small Proportion of the Population Provided Services under WIA, Making It Difficult to Know What the Program Is Achieving
paragraphs: WIA performance data represent a small proportion of the job seeker  population receiving services at one-stops, making it difficult to know  what the overall WIA program is achieving. Most one-stop customers who  participate in self-directed services and only receive limited staff  assistance, for example to conduct a job search, are not reflected in the  WIA performance reports. This group is estimated to be the largest portion  of customers served under WIA. For example, one of the local areas in our  study that tracks each one-stop customer told us that only about 5.5  percent of the individuals who walked into their one-stops in fiscal year  2003 were registered for WIA services. The current law excludes job  seekers who receive services that are self-service and informational in  nature. Labor’s guidance tells states to register adults and dislocated  workers who receive core services that require significant staff assistance  designed to help with job seeking or acquiring occupational skills, but  states have flexibility in deciding what constitutes significant staff  assistance. As a result of this flexibility, some local areas register a  smaller proportion of participants than others, and in an earlier report, we  said the local areas differed on when they registered WIA customers. In  our recent visits to 4 states, we found that states and localities still differ  on whom they tracksome local officials said they register job seekers  who received core services that required significant staff assistance, and  others said they do not register participants until they receive intensive  services. In addition, 21 of the 50 states we surveyed reported that they  have instituted their own policies to more specifically define when  registration should occur, suggesting that there is variation in interpreting  Labor’s guidance. Some experts told us that local workforce areas do not  get adequate credit for serving everyone, making it difficult to show what  is being achieved with available funding.
section_title: States and Local Areas Manage WIA Performance and Assess One-Stops by Collecting Timely Data and Making Use of a Range of Performance Information
paragraphs: With assistance from states, local areas manage WIA performance and  assess one-stop centers by collecting timely performance data and making  use of a variety of performance information. To understand how well they  are doing in meeting their performance levels, most local areas directly  contact former participants or employers to collect interim WIA  performance data that are not readily available from UI wage records.  States provide assistance to local areas in a variety of ways, ranging from  supporting their information technology (IT) systems to training local area  staff. While states and local areas must meet performance goals for WIA,  no similar goals exist for the overall one-stop systemthe service delivery  system required under WIA for most federally funded employment and  training services. Nonetheless, some states and many local areas have  developed a range of measures to help them assess how well the one-stop  is doing. Despite the progress states and local areas have made in  developing and using interim outcome information, states and local areas  told us they would like more help from Labor in collecting and  disseminating promising practices on interim ways to assess WIA  performance.
subsections: section_title: Many States Help Local Areas Monitor Their WIA Performance Levels
paragraphs: According to our survey, many states play an active role in helping local  areas monitor how well they are doing in meeting their performance  levels. The assistance they provide ranges from ensuring that local areas  have ready access to participants’ UI wage records to developing IT  systems and training local area staff on implementing WIA performance  measures. To ensure local areas have ready access to wage record  information on their participants, 23 states reported that they give local  areas some form of electronic access to UI wage data for their WIA  participants. Ten of these states give local areas direct online access to the  UI wage reporting system, making information available to local officials  as quickly as it is reported to the state; the others give local areas access  once information has been merged into the statewide WIA reporting  system. According to officials in Florida, having direct access to UI wage  data allows them to not only monitor performance levels but also develop  industry and wage profiles, tailor training programs to meet regional  needs, and obtain contact information for former participants, facilitating  follow-up with individuals that would not be found otherwise. When states  do not give local areas ready access to UI wage data, as might occur in  states with restrictive privacy laws, state officials usually provide local  areas with standard reports on their WIA progress, either for individual  WIA participants or, most often, aggregated across all WIA participants in  the local area.
In addition to helping provide timely information, almost all states are  supporting local areas’ IT efforts. According to our survey, 47 states have  established or are in the process of establishing statewide IT systems to  help local areas organize, track, and report WIA performance data. In  about three-fourths of these states, the statewide IT systems allow the  local areas to produce special reports that are tailored to local tracking  needs and can report information for the local areas to use at the one-stop  center, service provider, or case manager level. Although most local areas  reported that they use a statewide system to help meet federal reporting  requirements, half also use a locally developed IT system in combination  with a statewide IT system. Local officials we met with often commented  that they use a separate IT system because they do not find their state  systems useful for managing the day-to-day operations of a WIA program.  As a result of needing both a statewide and a local system, almost half of  local areas reported that at least some, if not all, of their one-stop staff  must enter the same WIA information into at least two IT systems.
Most states provide a range of other support services to local areas to help  them manage their WIA performance requirements and to understand  what implementation approaches work better than others in providing  one-stop services, according to our survey. States reported they most often  provide local areas with more specific written guidance or notices that  explain federal guidance on the WIA performance measures and  performance reports. In addition, about 90 percent of local areas told us  that their states conduct training, make presentations, and hold regular  meetings with local staff about WIA performance measures. To help local  areas better understand what implementation approaches work better  than others, several states have conducted special studies of the one-stop  system. Sixteen states told us they have recently conducted studies on  program implementation and processes; 11 states told us they have done  return-on-investment studies; and 4 states have done impact evaluations  that use control groups. Nationwide, half of the local areas believe that  having a strong relationship with their state greatly helps them achieve  their WIA levels.
section_title: To Help Them Meet Their WIA Performance Levels and Manage Their Programs, Most Local Areas Collect Outcome Information from a Variety of Sources and Actively Monitor Their Progress
paragraphs: Because UI wage data suffer from time delays, about three-fourths of local  areas collect outcome information from other sources to help them assess  whether they are meeting their WIA performance levels and to help them  manage their programs. Over 75 percent of local areas reported that they  directly follow up with participants after they leave the program,  collecting job placement or earnings information to help fill gaps until the  data are available from the UI wage records. Sometimes local officials will  also follow up with employers to verify employment or collect other  documentation, such as pay stubs or W-2 forms, as verification of  employment. If outcomes do not appear in UI wage records over time,  many local areas will report the findings from these other data sources in  their WIA performance reports to the state. Local officials in a rural area of  Pennsylvania told us that collecting this interim outcome data is important  to help them assess their progress in meeting their performance levels, so  much so that they provide small gift certificates to former participants  who periodically report back to WIA staff. According to these officials, this  strategy of obtaining follow-up data saves considerable staff time as well  as increases their performance levels by more completely capturing  information on participants.
Nearly all of the local areas reported on our survey that they track other  types of interim indicators to manage their WIA programsmost often the  number of registered WIA participants, services provided to WIA  participants, number of participants that have completed training, and  number of WIA exiters. Over half of these local areas report these data to  decision makers on at least a monthly basis. About 80 percent of local  areas track some kind of cost information, such as cost per participant or  cost per outcome, and 24 percent report this information at least monthly.  (See fig. 4.) Although these indicators may not be directly tracked and  reported under WIA, they are useful for helping local officials know the  number of participants that will be counted in their WIA measures.  Furthermore, in some cases, these interim indicators also help the local  areas predict their WIA performance outcomes. For example, one local  official told us that knowing the number of participants who complete  training helps him predict the number of participants who will find a job.  Overall, nearly half of local areas reported that this type of interim  information greatly helped them meet or exceed their performance levels.  Despite the progress states and local areas have made in developing and  using interim outcome information, nearly all states and local areas  reported they would like more help from Labor in collecting and  disseminating promising practices on interim indicators to assess WIA  performance.
Because meeting WIA performance levels may affect future funding, most  local areas hold service providers accountable and actively monitor their  WIA performance levels. Through our survey, we found that over 80  percent of local areas hold their service providers accountable by  incorporating negotiated performance levels in their contracts. In addition,  nearly 80 percent of local areas establish goals for the number of  participants who are registered in or exited from WIA. A lesser number of  local areas24 percentestablish pay for performance contracts, and 18  percent provide financial incentives to their service providers. (See table  5.) Officials from one local area that we visited told us they provide  monetary bonuses to providers that exceed their WIA goals and withhold  20 percent of their payments for those providers that do not reach their  WIA goals. In addition, over 80 percent of local areas nationwide reported  that having staff devoted to monitoring and managing WIA performance  greatly helps them achieve or exceed their levels.
Once final WIA performance information is available, local areas use this  information to assess program services over time and to guide future  program development. Most often local areas reported they use WIA  performance information to modify their programs. We found that about  two-thirds of local areas use performance information to a great extent to  help them identify areas for program improvement and adopt new  program approaches. Over half of local areas use their WIA performance  information to analyze trends over time and prepare strategic plans. (See  fig. 5.)
section_title: To Gauge One-Stop Performance, Many Local Areas Use Indicators in Addition to Those Required by WIA
paragraphs: While WIA requires that officials monitor outcomes for all job seekers who  receive staff-assisted core, intensive, and training services funded by WIA,  there is no requirement to track those who receive self-directed core  services, which may be the majority served under WIA. In addition, Labor  does not require that states and local areas measure the overall  performance of the one-stop system. Nonetheless, most states and local  areas have developed ways to assess the performance of their one-stops,  using four basic types of indicatorsjob seeker measures, employer  measures, program partnership measures, and family and community  indicators. (See fig. 6.)
Even without a federal requirement to do so, according to our survey,  almost 90 percent of local areas gather information on one-stop job  seekers, even if they are not registered and participating in any particular  federal program. Most often local areas reported that they require the one- stop centers to track and report the number of job seekers who visit the  one-stop in a single time period, usually through a paper and pencil or  computer log. We also found that 58 percent of the local areas are  collecting information on job seekers that repeatedly visit one-stop  centers, sometimes through electronic means. About 20 percent of all local  areas reported using electronic swipe cards to track job seekers in their  one-stop centers. These swipe cards, similar to membership or grocery  store discount cards, are issued to each job seeker using the one-stop and  contain unique identifying information that can be read each time the job  seeker accesses services. For example, according to local officials in  Philadelphia, they issue swipe cards to job seekers and scan these cards to  record both the services receivedsuch as using computers in the  resource room, attending orientation workshops, or talking with the case  managersand the date and time the services were provided. Using data  from this system, one-stop managers can assess traffic flow and schedule  staff accordingly, and may eventually be able to link participants and  services to outcomes achieved. Officials also told us they are using  demographic information from an analysis of swipe card data to target  marketing efforts and to develop services more strategically.
In addition to counting the number of job seekers who visit the one-stop  center, we found that local areas are tracking information on how many  program referrals they receive, how satisfied they are with services, and  what types of outcomes they achieve. Over half of local areas reported  that they survey job seekers who visit the one-stop to gauge their  satisfaction with services. For example, a one-stop center in Utah that we  visited not only uses a one-stop satisfaction survey, but officials also  periodically contact one-stop customers to ask how they liked the  services. According to our survey, some of the local areas said that having  job seeker satisfaction information was one of the best ways to assess the  one-stop system. Many local areas collect more in-depth information on all  one-stop job seekersover one-third collect demographic characteristics,  and over one-fourth monitor outcomes, such as whether job seekers got a  job and at what wages. (See fig. 7.)
Many local areas also track information on employers’ use of one-stops.  About 70 percent of local areas nationwide reported that they require one- stop centers to track some type of employer measure, such as the number  of employers that use one-stop services, how many hire one-stop  customers, and the type of services that employers use. To gauge  employer involvement, local areas most often require the one-stops to  count and report the number of employers that use one-stop services. Over  40 percent of local areas require one-stops to track the number of  employers that repeatedly use one-stop services. For example, a one-stop  center in Utah we visited tracks employers that repeatedly use one-stop  services and those that have not used services in a while. It uses this  information to reach out to employers who have not returned for services,  encouraging them to use one-stop services again. To understand how  employers view the one-stop services they received, 60 percent of local  areas reported they collect information on employer satisfaction. A  smaller numberabout 20 percent of local areastrack information on  market penetration, such as the number of employers in the labor market  that could potentially use one-stop services. For example, Philadelphia  officials told us they measure market penetration by comparing the  number of employers that use the one-stop center with the number of  employers in the community as a whole.  (See fig. 8.)
Most of the programs that provide services through the one-stop system  have their own performance measures, but as we have reported in the  past, these measures cannot be readily summed to obtain an overall  measure of one-stop performance. However, one-third of the local areas  told us that they combine in one report some of the key federal measures  for the various one-stop programsincluding wages at employment or  other earnings indicatorsand use this report to assess the one-stop  system as a whole. For example, Florida officials produce a reportcalled  the Red and Green reportthat assembles for each local area outcomes  on 22 measures from different one-stop programs, such as WIA, Wagner- Peyser, and TANF. Weaker program outcomes are identified in red and  stronger outcomes are identified in green. They use this report to assess  performance, diagnose weak spots, and predict long-term outcomes across  one-stop partners.
More often local areas have gone a step further and have identified  outcomes they consider to be key, developing common definitions for  these measures to be used across programs. Just over half of the local  areas reported in our survey that they track cross-cutting employment  measures, such as job placement, and a little less than half said they track  wages at placement and employment retention across programs. For  example, Utah developed a set of outcome, process, efficiency, and  activity measures to gauge the performance of all of their one-stops and to  ensure alignment with agency goals and objectives. These measures  include entered employment, earnings increase, and employment retention  across Wagner Peyser, WIA, TANF, Trade Adjustment Assistance, and  Food Stamp Employment and Training programs.
In addition to tracking outcomes for the various one-stop partners, some  local areas assess their one-stop systems by measuring the level of  coordination among one-stop partners, as well as the range and quality of  services they provide. Nearly 40 percent of local areas we surveyed said  that they use indicators, such as increased coordination among partners  and number of referrals partners made, to assess how well the overall one- stop system is operating. For example, one local area reported it is  developing a one-stop report card that will track the flow of customers  through the system and monitor each program’s contribution to the  services provided, including the results of program referrals. They will use  this report card to target areas that need attention. To ensure the one-stop  system is providing quality services, some local areas we visited also  conduct mystery shopper reviews wherein individuals posing as  employers or job seekers evaluate the quality of the services they receive.  Michigan conducts such mystery shopper visits of all their one-stops over  the course of a year to assess the quality of customer services, including  how courteous, professional, and knowledgeable one-stop staff are. The  state receives a comprehensive report of each visit and uses this  information to target technical assistance.
subsections: section_title: Family and Community Indicators
paragraphs: A few local areas look outside their one-stops to assess how well one-stop  services are meeting the needs of the family and the community. In their  written comments to our survey, several local areas told us that they  consider some type of community indicator, such as changes in the local  unemployment rate or increases in the average household income in the  local area, to be the best way to determine the overall effectiveness of  their one-stop system. Some local areas focus on indicators of family well- being, such as family self-sufficiencyor the ability of families to  financially support themselvesto assess whether their one-stop systems  are meeting family needs. One rural one-stop in Michigan even uses some  indicators that are not related to income. These local officials told us that  their indicators include a collection of family indicators, including whether  families are getting the child care they need and how well the children are  doing at home and at school, to understand how well the one-stop is  meeting the needs of the family.
section_title: Labor Has Taken Actions to Improve WIA’s Performance Measurement System and Assess One-Stops but Could Do More
paragraphs: Although Labor has taken steps to improve WIA’s performance  measurement system and assess one-stops, some of its efforts do not go  far enough. Labor has commissioned a study of adjustment methods that  would better take into account economic and demographic differences  when negotiating performance levels. However, even if an acceptable  model is developed, Labor has made no commitment to put a standard  adjustment method in place nationally. To improve the quality of WIA’s  performance data, Labor has initiated a data validation project. Labor is  taking a significant step toward measuring one-stop outcomes, but a  planned change may lead to restricting the use of supplemental data to fill  gaps in UI wage records. While Labor has plans to conduct impact studies,  the department will not meet WIA’s requirement to conduct an impact  study by 2005, and without such a study, little will be known about WIA’s  effectiveness.
subsections: section_title: Labor Has Commissioned a Study of Adjustment Methods That Could Be Used in Negotiating Performance Levels, but This Effort Is Limited
paragraphs: Labor has commissioned a study of adjustment methods that could be  used to set expected performance levels during the negotiations process,  but this effort does not go far enough. WIA requires that annual  negotiations to establish expected performance levels consider differences  in economic conditions, participant characteristics, and services  providedfactors that can have a significant effect on the performance  levels states and local areas are expected to achieve. However, many of  the state and local officials we interviewed said they did not think these  factors were adequately addressed in the negotiations process, and as a  result they think some of their performance levels were set too high for  the current economy. For example, some local officials said that their  negotiated performance levels on the earnings change and earnings  replacement measures were based on a stronger economy and did not  reflect recent increases in the unemployment rate. Nationwide, 22 states  reported that they are at risk of not meeting at least 80 percent of their  negotiated performance levels on one or more of the WIA measures for  program year 2002. (See fig. 9.) Further, 10 states reported that they are at  risk of receiving financial sanctions on one or more measures for program  year 2002.
To address states’ concerns, Labor has commissioned a study of  adjustment methods, such as the type of model used under JTPAone  that adjusted for factors beyond the control of local programs, such as  high unemployment or a high concentration of non-English-speaking  program applicants. The JTPA model assigned adjustment factors and  weights for each performance measure using a multiple regression  analysis, predicting how well a local area might do based on the relevant  factors. For example, the model would assign a lower expected  performance level to a local program serving extremely disadvantaged  participants in an economically depressed area and a higher expected  performance level to a local program serving job seekers who are nearly  ready to get a job in an area with good economic conditions. All states and  nearly all local areas we surveyed told us they would like Labor to use a  model that can adjust for varying economic and population factors.
Although Labor is studying adjustment methods, even if an acceptable  model is developed, it has made no commitment to implement such an  adjustment method nationally.
Some states currently use their own adjustment model or other methods in  the negotiation process to account for factors beyond the control of local  programs, but Labor has not yet taken steps to increase consistency across  states as it did under JTPA. According to our survey, we found that nine  states used a regression model or other method to a great extent to  establish their performance levels for negotiating their program year 2004  performance levels with Labor. Under JTPA, Labor allowed states  flexibility to develop their own adjustment procedures, but it established  standard parameters to govern the adjustment methods used by states.  These parameters addressed the procedures for adjusting performance  levels, the quality of data, and factors that could be used for adjustments.  For example, the procedures for adjusting performance levels were  required to be objective and equitable across all local areas. In addition,  Labor developed optional adjustment models that could be used by states  because it recognized that not all states and local areas have the expertise  and resources necessary to develop adjustment procedures. Without  standard parameters, the process will lack consistency, and some states  may be at a disadvantage in the process of negotiating their performance  levels.
section_title: Concerns Have Been Raised about the Quality of WIA’s Performance Data, and Labor Has Begun a Major Initiative to Address These Concerns
paragraphs: Issues have been raised about the quality of performance data that Labor  uses to assess program performance. As we mentioned previously, Labor  allows flexibility in determining which participants to track for reporting  purposes. This flexibility leads to variations in reporting, which raises  questions about both the accuracy and the comparability of states’  performance data. In addition, we recently reported that performance  data submitted by states in quarterly and annual reports were not  sufficiently reliable to determine outcomes for the WIA programs.  Furthermore, Labor’s Office of Inspector General has said that there is  little assurance that the states’ performance data for all WIA programs are  either accurate or complete because of inadequate oversight of data  collection and management at the federal, state, and local levels.
Labor has initiated a new data validation project to improve the quality of  the performance information collected and reported under WIA. Labor’s  data validation project includes developing procedures and accuracy  standards to help states validate that WIA performance and participant  data are correctly reported. For this project, Labor developed data  validation handbooks and software and required states to begin validating  program year 2002 data, which were reported to Labor on December 1,  2003. States are required to conduct two types of data validation: (1)  review samples of WIA participant files and (2) assess whether reporting  software accurately calculated the performance measures. Labor provided  software to help states generate the aggregate information required for  performance reports, such as performance outcomes. If states elect to use  Labor’s software, they are not required to validate the calculations. At the  time of our surveyDecember 2003 through February 2004we found  that 41 states had begun using Labor’s data validation software. Labor also  plans to hold states accountable for meeting accuracy standards,  beginning in the third year of validation. Once these accuracy standards  are in place, states failing to meet the standards may lose eligibility for  incentive awards or, in cases with significant deviations from the  standards, states may be sanctioned.
section_title: Labor Is Taking Steps to Measure One-Stop Outcomes, but a Planned Change May Lead to Restricting the Use of Supplemental Data to Fill Gaps in UI Wage Records
paragraphs: Labor is taking a significant step toward measuring outcomes across one- stop partners by developing definitions for a set of common performance  measures. The Office of Management and Budget established a set of  common measures to be applied to all federal employment and training  programs administered by Labor, Education, Health and Human Services,  Veterans Affairs, Interior, and Housing and Urban Development. (See table  6.) Labor has developed standard definitions for calculating these  measures across all of its Employment and Training Administration  programs. (See table 7.) This will allow Labor to sum outcomes across all  its programs to provide a more uniform picture of outcomes achieved.  According to a department official, Labor worked with other federal  agencies to get agreement on common data sources and common  language, where possible. For example, Labor is working on developing a  process, using WRIS, that would allow other federal programs to use UI  wage records to track outcomes. As part of the common measures, Labor  plans to require one-stops to track all participants who walk through the  door of a one-stop center and receive any one-stop service, regardless of  which program provides the service. According to Labor, tracking all one- stop job seekers will enable officials to obtain information about who is  served, what services are provided, which partner programs provided  services, and what outcomes are achieved. While these changes can  provide more information on job seekers, there is no provision for any  measure of employer involvement in the one-stops, and experts and state  and local officials we interviewed said that at least one measure is needed  to address employer usage.
While most of Labor’s policies for the common measures can advance  measurement across one-stop partners, Labor plans to rely almost entirely  on the UI wage records and discontinue the use of supplemental data for  filling gaps in UI wage records. Labor officials tell us that they are making  this change to address concerns about the quality of supplemental data  being collected. Under Labor’s current guidance, supplemental data must  be documented. However, the department has no systematic process in  place to monitor the accuracy of these supplemental data. If Labor elects  to replace the current definitions of the WIA entered employment rate and  earnings retention measure with the common measure definitions, this  restriction on the use of supplemental data could have a significant impact  on the ability of states and local areas to meet their negotiated  performance levels. In addition, Labor’s new data validation project could  help ensure the accuracy of supplemental data that is collected at the local  level.
section_title: Labor Will Not Meet WIA’s Requirement to Conduct an Impact Study by 2005, and Without Such a Study, Little Will Be Known about WIA’s Effectiveness
paragraphs: While Labor has plans to conduct impact studies, the department will not  meet WIA’s requirement to conduct at least one multi-site control group  evaluation by fiscal year 2005. This type of impact study is important  because outcome measures alone cannot show whether an outcome is a  direct result of program participation or whether it is a result of other  influences, such as the state of the local economy. Labor officials said they  did not initiate impact studies of WIA within the first few years after WIA  passed to allow states and local areas time to implement the considerable  changes that were required under WIA. According to officials, Labor had  planned to initiate an impact evaluation of the WIA adult and dislocated  worker programs in 2004, but this plan is currently on hold because Labor  is anticipating changes to these programs as a result of reauthorization.  Once WIA is reauthorized, Labor officials told us that they would likely  allow 2 or 3 years for changes to be implemented before initiating an  impact evaluation. The evaluation itself will take 5 to 6 years, but Labor  plans to issue interim reports on the findings once the study is under way.  Even though the House passed a reauthorization bill, the Workforce  Reinvestment and Adult Education Act of 2003 (HR1261), and the Senate  passed a bill, the Workforce Investment Act of 2003 (S1627), passage of a  final bill has stalled. Both bills propose changes to WIA, but most of the  basic one-stop service delivery and governance structure would stay the  same in both bills. Given that these changes will not likely affect the  fundamental service delivery and structure of WIA, it is unclear why Labor  has not proceeded with its evaluations of WIA as planned.
section_title: Conclusions
paragraphs: When WIA was implemented nearly 4 years ago, it fundamentally changed  the way federally funded employment and training services are provided  to job seekers, the way the system engages employers, and the way it  measures performance. Making this shift has taken time and some trial  and error as state and local policy makers and one-stop service providers  learned what type of service structure met local needs. Since  implementation, states and local areas have made great progress in  retooling their systems and in gathering all the data needed to report on  their performance to Labor. But, only recently are we getting a nationwide  glimpse of outcomes achieved under WIA.
The requirement to use UI wage data is a step in the right direction by  providing a reasonably consistent look at national program results over  time. Historically, there have been data quality issues with outcome data  collected directly from participants, as was done prior to WIA. The UI  wage data provide a level of credibility that other data sources do not  have. States have made progress in accessing data from other states. But  in order to meet their performance levels, some states must continue to  rely on other data sources to fill gaps. Out of concern for data  comparability, Labor is proposing to limit the use of data from other  sources. This decision, if applied to the WIA programs, will hinder the  ability of some states to demonstrate that they have met their expected  performance levels and may cause one-stops to focus their efforts on only  those occupations covered by the UI wage records. This policy also seems  overly restrictive, given that Labor is implementing data validation  procedures that could be used to ensure the accuracy and validity of  supplemental data. Even with the capability to use supplemental data,  some states and local areas have failed to demonstrate that they met their  negotiated performance levels for 2 years in a row and have suffered  financial sanctions—often citing local economic conditions as the cause.  The development of a method to systematically adjust for economic and  demographic factors outside the control of the local area in setting  expected performance levels could help mitigate these concerns.
While the use of UI wage records has improved the quality of the data that  are used to track outcomes under WIA, this information alone does little  for real-time program management. We found that state and local officials  have made significant strides in collecting their own data to assess  whether they are likely to meet their federally required performance  levels, manage their programs on a real-time basis, and track a broader  one-stop population than just registered WIA participants. In some ways,  the WIA performance measures based on the UI wage records and the  interim data collected at the state and local level provide a useful system  to cross check these data. However, not all states and local areas have  determined what interim information is necessary, nor have they had the  benefit of learning from their peers. Without some additional information  or the sharing of promising practices, these states and local areas will be  at a disadvantage in monitoring their progress and, perhaps, in meeting  their minimum performance levels. Further, Labor has also failed to meet  WIA’s requirement to conduct a systematic evaluation of WIA. Plans to do  an evaluation have been delayed until reauthorization is complete, even  though the proposed bills would retain most of the WIA service delivery  and governance structure. Delays in committing to an evaluation now may  be costly because policy makers will not be able to benefit from an  understanding of WIA’s effectiveness.
Without clear guidance from Labor, states and local areas continue to  struggle with determining who should be tracked in the WIA performance  measures. At the same time, even if states and localities had a common  understanding of whom to track and were consistently reporting on the  same categories of customers, they would only be reporting on a small  portion of overall one-stop customers. While a requirement to track all job  seekers who visit the one-stops may appear to be a major change, we  found that over half the local areas already collect information on job  seekers that repeatedly use one-stops, suggesting that some local areas are  already equipped to uniquely identify and track each job seeker. It may  take time and resources for local areas to fully develop the capability to  collect data on each job seeker, but this may be the best way to start  gauging the value of one-stops overall. As long as the law excludes  individuals who participate in self-service and informational services, it  will be difficult to understand the full reach of WIA.
section_title: Recommendations For Executive Action
paragraphs: To compensate for the impact of changes in the economy and to give  states and local areas an equal opportunity to meet their performance  levels, we recommend that the Secretary of Labor    continue to allow the use of supplemental data for reporting outcomes,  but develop more stringent guidance and monitoring of these data;    provide assistance to states and localities in developing and sharing  promising practices on interim indicators for assessing WIA’s  performance; and    develop an adjustment model or other systematic method to account for  different populations and local economic conditions when negotiating  performance levels.
To comply with statutory requirements and to help federal, state, and local  policy makers understand what services are most effective for improving  employment-related outcomes, we recommend that the Secretary of Labor  expedite efforts to design and implement an impact evaluation of WIA  services.
section_title: Matter for Congressional Consideration
paragraphs: We suggest that Congress may wish to consider requiring that information  be collected and reported on all WIA participants, including those who  only receive self-service and informational services, so that Congress may  have a better understanding of the full reach of WIA and the one-stop  system.
section_title: Agency Comments
paragraphs: We provided a draft of this report to Labor for review and comment. Labor  generally agreed with recommendations about continuing the use of  supplemental data, sharing promising practices on interim performance  indicators, and developing an adjustment model or other systematic  method for use in negotiating performance levels. In addition, Labor  agreed with our matter for Congressional consideration that information  be collected and reported on all WIA participants. However, Labor  disagreed with our recommendation to expedite efforts to design and  implement an impact evaluation of WIA services. We have incorporated  Labor’s comments in our report, as appropriate. A copy of Labor’s  response is in appendix III.
On our recommendation regarding the use of supplemental data for  reporting outcomes under WIA, Labor responded that it will continue to  allow supplemental wage data except when calculating results on the  common measures that are reported to the Office of Management and  Budget. Labor also told us that its ongoing data validation effort will  collect additional information that will help assess the quality of  supplemental wage data that states are reporting. We continue to believe  that when assessing state and local progress toward meeting WIA’s  expected performance levels, supplemental data will be essential to gather  a more complete picture of WIA outcomes.
On our recommendation to develop and share promising practices on  interim indicators for assessing WIA’s performance, Labor noted some of  the efforts currently under way to facilitate information exchange,  including state and local peer-to-peer alliances, Labor’s promising  practices web site, and a Performance Enhancement Project for states to  share ideas and promising practices. However, despite Labor’s ongoing  efforts to facilitate information exchange, nearly all states and local areas  reported on our survey that they would like more help from Labor in  collecting and disseminating information on promising practices on  interim indicators to assess WIA performance.
Regarding our recommendation to develop an adjustment or other  systematic method for use in negotiating performance levels, Labor agreed  with the importance of taking economic conditions and characteristics of  the population into account when setting performance expectations. Labor  noted the study it has commissioned on adjustment models that we cited  in our report and said the results of this study are not yet available. Labor  expressed concern that any systematic method for taking economic and  demographic factors into account must not diminish the role of the states  and local areas in setting strategic goals. Our recommendation for a  systematic approach would not replace any state and local efforts to  establish their own goals, but it could help make the national process for  setting goals more uniform and provide tools for states and local areas that  do not have the resources to develop their own adjustment procedures.
In response to our recommendation to expedite the design and  implementation of an impact evaluation of WIA services, Labor told us that  it believes the program consolidation changes proposed in the  reauthorization bill passed by the House are significant enough to delay  the multi-site evaluation required by WIA. However, we disagree that  proposed reauthorization changes would significantly affect the basic one- stop service delivery structure under WIA. It is now 4 years past the full  implementation of WIA and a well-designed evaluation would help inform  policymakers in the future. Waiting for the implementation of any changes  resulting from the current reauthorization cycle would likely delay the  start of an evaluation at least 2 years, thus not having results available  until after another reauthorization cycle has passed.
We are sending copies of this report to the Secretary of Labor, relevant  congressional committees, and others who are interested. Copies will also  be made available to others upon request. The report is also available on  GAO’s home page at http://www.gao.gov.
Please contact me on (202) 512-7215 if you or your staff have any  questions about this report. Other major contributors to this report are  listed in appendix III.
section_title: Appendix I: Objectives, Scope, And Methodology
paragraphs: We examined (1) how useful WIA performance data are in gauging  program performance, (2) what local areas are doing to manage their WIA  performance and assess one-stop success on a timely basis and how states  are assisting these efforts, and (3) to what extent Labor is trying to  improve WIA’s performance measurement system and assess one-stop  success. Our review focused primarily on the employment-based measures  that rely on UI wage records—entered employment rate, earnings  change/replacement rate, employment retention rate, employment and  credential rate, and the younger youth placement and retention rate. To  address these questions, we conducted two surveys—one of state WIA  officials and one of local area workforce officials; reviewed different types  of literature about WIA and the WIA performance measurement system;  interviewed experts and Department of Labor officials; interviewed state  and local WIA officials; and visited three states and two local areas or one- stops within each state. We supplemented our site visits with telephone  interviews with state and local officials in Pennsylvania  We provided a draft of this report to officials at the Department of Labor  for their review and incorporated their comments where appropriate. We  conducted our work from April 2003 through April 2004 in accordance  with generally accepted government auditing standards.
subsections: section_title: Research and Experts
paragraphs: To obtain further information on the area of WIA performance  management, we reviewed and analyzed numerous studies, reports, and  other literature, and we interviewed experts on WIA and workforce  development performance measurement. We reviewed a Department of  Labor study that discussed costs of data collection and found it  sufficiently reliable for the purpose of comparing costs of surveys and  automated record matching to UI wage records. We also interviewed  Department of Labor officials, as well as representatives of the National  Governors’ Association and the National Association of Workforce Boards.
section_title: Surveys
paragraphs: To determine how useful WIA performance data are in gauging program  performance and what states and local areas are doing to manage and  assess WIA programs and one-stop systems, we surveyed all 50 states and  the District of Columbia, as well as all existing local workforce  investment areas, using similar but not identical questionnaires. We  conducted both surveys via the Internet. We asked both groups to provide  information on issues related to the WIA performance measures, such as  state or local policies, the availability and use of UI data, WIA performance  levels; management practices; information technology systems, efforts to  monitor and manage their WIA programs and one-stop systems, factors  that adversely affected their ability to assess their one-stops systems, and  the types of technical assistance that would help with managing their one- stop systems’ performance.
We pre-tested the questionnaires used for each of the surveys at least three  times. Table 8 provides survey numbers and response rates for both  surveys.
Because these were not sample surveys, there are no sampling errors.  However, the practical difficulties of conducting any survey may introduce  errors, commonly referred to as nonsampling errors. For example,  difficulties in how a particular question is interpreted, in the sources of  information that are available to respondents, or how the data are entered  into a database can introduce unwanted variability into the survey results.  We took steps in the development of the questionnaires, the data  collection, and data analysis to minimize these nonsampling errors. For  example, as already noted, we pretested the questionnaires to ensure that  questions were clear and understandable. In that these were Web-based  surveys whereby respondents entered their responses directly into our  database, there was little possibility of data entry error. In addition, we  verified that the computer programs used to analyze the data were written  correctly.
section_title: Site Visits
paragraphs: We visited three states—Florida, Michigan, and Utah—and traveled to at  least two local areas or one-stop centers in each of these states. We  supplemented our site visits with telephone interviews with state and local  officials in Pennsylvania. (See table 9 for a list of the states and local areas  in our study.) Based on input from recognized experts and our literature  review, we selected these states because they are geographically diverse,  have experience in implementing additional performance measures to  assess one-stop success, and have developed integrated statewide data  systems.
In each state, we interviewed state officials responsible for monitoring  local areas’ WIA programs and analyzing and reporting on the state’s WIA  performance data, as well as other state WIA and IT officials and staff of  the state’s Workforce Investment Board. At the local areas, we interviewed  WIA officials and staff, including service providers, staff responsible for  performance management issues, IT staff, case managers and other  frontline staff, as well as staff of the local area Workforce Investment  Board. The state and local interviews were administered using a semi- structured interview guide that we developed through a review of relevant  literature and discussions with recognized experts on WIA performance  management.
Information that we gathered on our site visits represents only the  conditions present in the states and local areas at the time of our site  visits, from June through October 2003. We cannot comment on any  changes that may have occurred after our fieldwork was completed.  Furthermore, our fieldwork focused on in-depth analysis of only a few  selected states and local areas or sites. Based on our site visit information,  we cannot generalize our findings beyond the states and local areas or  sites we visited.
section_title: Appendix II: States’ Use of UI Wage Records and Other Data Sources for Reporting on Employment Outcomes Under WIA
section_title: Appendix III: Comments from the Department of Labor
section_title: Appendix IV: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: Carolyn S. Blocker and Cheri Harrington made significant contributions to  all phases of the effort. Stu Kaufman made significant contributions in the  design and administration of the surveys. In addition, Jessica Botsford  provided legal support; Avrum Ashery and Barbara Hills provided graphic  design assistance; and Elizabeth Curda, Patricia Dalton, Catherine Hurley,  and Shana Wallace also provided key technical assistance.
section_title: GAO Related Products
paragraphs: National Emergency Grants: Labor is Instituting Changes to Improve  Award Process, but Further Actions Are Required to Expedite Grant  Awards and Improve Data. GAO-04-496. Washington D.C.: April 16, 2004.
Workforce Investment Act: Labor Actions Can Help States Improve  Quality of Performance Outcome Data and Delivery of Youth Services.  GAO-04-308. Washington D.C.: February 23, 2004.
Workforce Training: Almost Half of States Fund Employment Placement  and Training and Employment through Employer Taxes and Most  Coordinate with Federally Funded Programs. GAO-04-282. Washington  D.C.: February 13, 2004.
Workforce Investment Act: One-Stop Centers Implemented Strategies to  Strengthen Services and Partnerships, but More Research and  Information Sharing Is Needed. GAO-03-725. Washington, D.C.: June 18,  2003.
Workforce Investment Act: Issues Related to Allocation Formulas for  Youth, Adults, and Dislocated Workers. GAO-03-636. Washington, D.C.:  April 25, 2003.
Workforce Training: Employed Worker Programs Focus on Business  Needs, but Revised Performance Measures Could Improve Access for  Some Workers. GAO-03-353. Washington, D.C.: February 14, 2003.
Older Workers: Employment Assistance Focuses on Subsidized Jobs and  Job Search, but Revised Performance Measures Could Improve Access to  Other Services. GAO-03-350. Washington, D.C.: January 24, 2003.
Workforce Investment Act: States’ Spending Is on Track, but Better  Guidance Would Improve Financial Reporting. GAO-03-239. Washington,  D.C.: November 22, 2002.
Workforce Investment Act: States and Localities Increasingly Coordinate  Services for TANF Clients, but Better Information Needed on Effective  Approaches. GAO-02-696. Washington, D.C.: July 3, 2002.
Workforce Investment Act: Youth Provisions Promote New Service  Strategies, but Additional Guidance Would Enhance Program  Development. GAO-02-413. Washington, D.C.: April 5, 2002.
Workforce Investment Act: Better Guidance and Revised Funding  Formula Would Enhance Dislocated Worker Program. GAO-02-274.  Washington, D.C.: February 11, 2002.
Workforce Investment Act: Improvements Needed in Performance  Measures to Provide a More Accurate Picture of WIA’s Effectiveness.  GAO-02-275. Washington, D.C.: February 1, 2002.
Workforce Investment Act: Better Guidance Needed to Address Concerns  Over New Requirements. GAO-02-72. Washington, D.C.: Oct. 4, 2001.
Workforce Investment Act: Implementation Status and the Integration of  TANF Services. GAO/T-HEHS-00-145. Washington, D.C.: June 29, 2000.