Workforce Investment Act: Labor and States Have Taken Actions to Improve Data Quality, but Additional Steps Are Needed
Published: 2005-11-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Federal programs carried out in partnership with states and localities continually balance the competing objectives of collecting uniform performance data with giving program implementers the flexibility they need. Our previous work identified limitations in the quality of performance data for the key employment and training program--the Workforce Investment Act (WIA). WIA relies on states and localities to work together to track and report on participant outcomes, and it changed the way outcomes are measured. Given the magnitude of changes and the impact such changes can have on data quality, we examined (1) the data quality issues that affected states' efforts to collect and report WIA performance data; (2) states' actions to address them; and (3) the actions the Department of Labor (Labor) is taking to address data quality issues, and the issues that remain.
section_title: What GAO Found
paragraphs: Three key issues--flexibility in federal guidance, major changes to states' information technology (IT) systems, and limited monitoring--compromised states' early efforts to collect and report WIA performance data. Labor's initial guidance allowed states and local areas flexibility in deciding which jobseekers to track and when jobseekers leave services and get counted in the measures. As a result, states and local areas have differed on whom they track and for how long. States took various approaches to implement IT systems for meeting WIA reporting requirements. Thirty-nine states reported to us that they made major modifications to their IT systems since WIA was first implemented in 2000. Thirteen of them said the changes resulted in problems affecting data quality, and 5 states are still trying to resolve these problems. In addition, oversight of WIA performance data was insufficient at all levels during early implementation. Almost all states have made efforts to improve the quality of WIA performance data--at least 40 states have controls in their IT systems that capture WIA performance data, such as edit checks or exception reports to help screen for errors or missing data. Forty-three states have taken actions to clarify Labor's guidance and help local areas determine who should be tracked in the performance measures. In addition, most states said they monitor local areas by assessing local procedures and policies. Labor recently began addressing data quality issues, however, some issues remain. In 2004, Labor addressed some data quality concerns by requiring states to validate their data and ensure the accuracy of their performance outcomes. Most states told us that Labor's requirements have increased awareness of data quality at the state and local level. However, Labor does not have methods in place to review states' validation efforts or hold states accountable for complying with its requirements. Labor issued guidance requiring states to implement common performance measures on July 1, 2005, which clarified some key data elements, but does not address all the issues. Labor has some federal monitoring processes in place but lacks a standard monitoring guide to address data quality.

section_title: Letter
paragraphs: Performance data are becoming increasingly significant in helping policy  makers and program managers assess progress of federal programs in  meeting their long-term goals and in helping to make a variety of  programmatic and budget decisions. Yet our previous work has identified  limitations in the ability of federal agencies to produce credible  performance data. In particular, federal programs that are carried out in  partnership with states and localities continually balance the competing  objectives of collecting uniform performance data at the national level  with giving states and localities the flexibility they need to implement  programs. The Workforce Investment Act (WIA) of 1998—the centerpiece  of the nation’s employment and training system—established three  programs that rely on states and localities to work together to track and  report on participant outcomes in areas of job placement, retention,  earnings, and skill attainment, as well as customer satisfaction. WIA,  implemented in July 2000, has resulted in a major shift from predecessor  programs, including the Job Training Partnership Act (JTPA) program, by  offering a broader array of services to the general public and no longer  using income to determine eligibility for all program services. WIA also  changed the way performance is measured, including establishing new  performance measures that assess outcomes over time, requiring the use  of Unemployment Insurance (UI) wage data to track outcomes, and  requiring states to negotiate expected performance levels with the  Department of Labor (Labor).
States are held accountable for achieving their performance levels through  financial incentives and sanctions. These changes have had profound  implications for the way WIA performance data are collected and  reported.
Given the magnitude of these changes, the potential impact such changes  can have on data quality, and the importance of having meaningful  performance data, we examined (1) the data quality issues that have  affected states’ efforts to collect and report WIA performance data,  (2) states’ actions to address them, and (3) the actions Labor is taking to  address data quality issues and the issues that remain.
To learn more about states’ experiences implementing data collection and  reporting system changes for WIA, their implementation of Labor’s data  validation requirements for WIA, and state and local efforts to address the  quality of WIA data, we conducted a web-based survey of workforce  officials in 50 states and received a 100 percent response rate. We did not  include the District of Columbia and U.S. territories in our survey. In  addition, we conducted site visits in California, New York, Texas, West  Virginia, and Wyoming, where we interviewed state officials and visited  two local areas in each state. We selected these states because they  represent a range of information technology (IT) systems—statewide  comprehensive systems versus local systems with a state reporting  function, include states with single and multiple workforce areas, and are  geographically diverse. We also collected information on the quality of  WIA data through interviews with Labor officials in headquarters and all  six regional offices, and nationally recognized experts, and reviewed  relevant research literature. Our work was conducted between June 2004  and September 2005 in accordance with generally accepted government  auditing standards. (For a complete description of our scope and  methodology, see app. I.)
subsections: section_title: Background
paragraphs: Labor required states to implement major provisions of WIA by July 1,  2000, although some states began implementing provisions of WIA as early  as July 1999.  WIA replaced the Job Training Partnership Act (JTPA)  program and requires that many federal programs provide employment  and training services through one-stop centers.  Services funded under  WIA represent a marked change from those provided under the previous  program, allowing for a greater array of services to the general public. WIA  is designed to provide for greater accountability than under previous law:  it established new performance measures and a requirement to use  Unemployment Insurance (UI) wage data to track and report on outcomes.
subsections: section_title: WIA-Funded Services Represent a Change from Those Funded under JTPA
paragraphs: Program services provided under WIA represent a marked change from  those provided under JTPA. When WIA was enacted in 1998, it replaced  the JTPA programs for economically disadvantaged adults and youth and  for dislocated workers with three new programs—Adult, Dislocated  Worker, and Youth—that provide a broad range of services to the general  public, no longer using income to determine eligibility for all program  services. The WIA adult and dislocated worker programs no longer focus  exclusively on training, but provide for three tiers, or levels, of service:  core, intensive, and training. Core services include basic services such as  help with job searches and providing labor market information. These  activities may either be self-service or require some staff assistance.  Intensive services include such activities as comprehensive assessment of  jobseekers’ skill levels and service needs and case management—activities  that typically require greater staff involvement. Training services include  such activities as occupational skills development or on-the-job training.  Labor’s guidance specifies that monitoring and tracking for the adult and  dislocated worker programs should begin when jobseekers receive core  services that require significant staff assistance. Jobseekers who receive  core services that are self-service or informational in nature are not  counted in the performance measures.
In addition to those services provided by the three WIA funded programs,  WIA also requires that states and local areas use the one-stop center  system to provide services for many other employment and training  programs. Seventeen categories of programs funded through four federal  agencies are now required to provide services through the one-stop center  under WIA. Table 1 shows the programs that WIA requires to provide  services through the one-stop centers (also known as mandatory  programs) and the federal agencies that administer these programs.
section_title: WIA Performance Measures Are Designed to Increase Accountability for Three WIA-Funded Programs
paragraphs: WIA is designed to provide for greater accountability than its predecessor  program by establishing new performance measures, a new requirement to  use UI wage data to track and report on outcomes, and a requirement for  Labor to conduct at least one multi-site control group evaluation.  According to Labor, performance data collected from the states in support  of the measures are intended to be comparable across states in order to  maintain objectivity in determining incentives and sanctions. The  performance measures also provide information to support Labor’s  performance goals under the Government Performance and Results Act  (GPRA), the budget formulation process using the Office of Management  and Budget’s (OMB) Program Assessment Rating Tool (PART), and for  program evaluation required under WIA.
In contrast to JTPA, under which data on outcomes were obtained through  follow-ups with job seekers, WIA requires states to use UI wage records to  track employment-related outcomes. Each state maintains UI wage  records to support the process of providing unemployment compensation  to unemployed workers. The records are compiled from data submitted to  the state each quarter by employers and primarily include information on  the total amount of income earned during that quarter by each of their  employees. Although UI wage records contain basic wage information for  about 94 percent of workers, certain employment categories are excluded,  such as self-employed persons, independent contractors, federal  employees, and military personnel. According to Labor’s guidance, if a  program participant does not appear in the UI wage records, states may  then use supplemental data sources, such as follow-up with participants  and employers, or other administrative databases, such as U.S. Office of  Personnel Management or U.S. Department of Defense records, to track  most of the employment-related measures. However, only UI wage records  may be used to calculate the earnings change and earnings replacement  performance measures. (See table 2 for a complete list of WIA  performance measures.)
Unlike JTPA, which established expected performance goals using a  computer model that took into account varying economic and  demographic factors, WIA requires states to negotiate with Labor to  establish expected performance levels for each measure. States, in turn,  must negotiate performance levels with each local area. The law requires  that these negotiations take into account differences in economic  conditions, participant characteristics, and services provided. To derive  equitable performance levels, Labor and the states use historical data to  develop their estimates of expected performance levels. These estimates  provide the basis for negotiations.
WIA holds states accountable for achieving their performance levels by  tying those levels to financial sanctions and incentive funding. States that  meet their performance levels under WIA are eligible to receive incentive  grants that generally range from $750,000 to $3 million. Nineteen states  were eligible to apply for incentive grants in program year 2003. States  that do not meet at least 80 percent of their WIA performance levels are  subject to sanctions. If a state fails to meet its performance levels for 1  year, Labor provides technical assistance, if requested. If a state fails to  meets its performance levels for 2 consecutive years, it may be subject to a  5 percent reduction in its annual WIA formula grant. No states received  financial sanctions in program year 2003.
Labor determines incentive grants or sanctions based on the performance  data submitted by states each October in their annual reports. States also  submit quarterly performance reports, which are due 45 days after the end  of each quarter. In addition to the performance reports, states submit  updates for their Workforce Investment Act Standardized Record Data  (WIASRD) in mid-October. WIASRD is a national database of individual  records containing characteristics, activities, and outcome information for  all enrolled participants who receive services or benefits under WIA. All  three submissions primarily represent participants who have exited the  WIA programs within the previous program year.
The process of collecting and reporting WIA data involves all three levels  of government. Participant data are typically collected by frontline staff in  local areas and entered into a state or local IT system. In some states, local  area staff may enter data directly into a statewide IT system; in other  states, local areas may use their own individualized IT system to enter  data, from which staff can extract and compile the necessary information  for state submission.
After the state receives data from local areas, this information is compiled  and formatted for various submissions to Labor, including the state’s  WIASRD file, quarterly report, and annual report. During the data  compilation process, state agencies administering WIA typically match  participant records to their state’s UI wage record system to obtain wage  records and employment status. In addition, states may use the Wage  Record Interchange System (WRIS) to match participant records to other  state’s UI wage records or use other databases such as that of the U.S.  Office of Personnel Management to fill gaps in the UI wage records. States  may also link participant records to partner programs’ IT systems to track  activities across programs or to determine outcomes such as attaining high  school diplomas, degrees, and certificates. For the quarterly and annual  report, states use software to calculate their performance measures. States  generate the required WIA performance reports and electronically submit  them to Labor’s regional offices using the Enterprise Business Support  System (see fig. 1).
section_title: Internal Controls to Ensure Data Quality
paragraphs: Internal controls comprise the plans, methods, and procedures an  organization uses to meet its missions, goals, and objectives. Internal  controls used by government agencies may include guidance that defines  the specific data to be collected and any documentation needed to support  the data and safeguards to ensure data are secure.
Some key aspects of internal controls for collecting and reporting data  include:    Guidance: Guidance should clearly and consistently define all data  elements required for reporting, and effectively communicate this  information to states and local areas. If definitions are vague or  inconsistent, then program staff may interpret them incorrectly, resulting  in more errors to the data. Additionally, any guidance and documentation  from the national office to states and local areas must be clear and free of  any conflicting or contradictory instructions. If reporting instructions are  misinterpreted by program staff, then the data may not be useful to assess  program performance.
Data entry procedures and edit check software: Data entry  procedures and edit check software can help ensure data entering the  designated reporting system are accurate and consistent. Written guides  establishing who is responsible for each step in data creation and  maintenance, and how data are transferred from initial to final formats can  ensure data are consistently reported. Additionally, using electronic data  management and processing software programs to conduct automated  checks on data values and integrity can limit errors when data are  reported at a later date.
Monitoring: Monitoring can ensure reported data are accurate and  complete. Common monitoring practices may include formal on-site  reviews of individual case files and source documentation at both the state  and local levels, and assessments of issued guidance to ensure that  information collected nationwide is consistent with existing policies and  in compliance with laws and regulations.
section_title: Several Key Issues Have Affected States’ Efforts to Ensure the Quality of WIA Performance Data
paragraphs: Three key issues—flexibility in federal guidance, major changes to states’  information technology (IT) systems and limited monitoring efforts—have  compromised states’ early efforts to collect and report WIA performance  data. The guidance available to states at the time of implementation  allowed flexibility in key definitions and contributed to inconsistency in  the way the data are collected and reported. The transition from JTPA to  WIA required states to make major changes to their IT systems and in  some cases, the transition led to problems with the data. States used a  variety of strategies to make the necessary system changes, some used the  software they had used to report under JTPA, and others developed new  software for WIA. More than three-fourths of the states told us that they  had made major modifications to their WIA IT systems since  implementation. One-third of these states reported that when these  modifications were made, they experienced significant problems that  affected the quality of the data. Lack of oversight at the local, state, and  federal levels made it difficult to ensure that early WIA performance data  were accurate.
subsections: section_title: Flexibility in Guidance from Labor Led to Inconsistency in the Way Data Are Collected and Reported
paragraphs: The guidance available to states at the time of implementation was open to  interpretation in key terms and contributed to inconsistency in the way  that data are collected and reported. Labor allowed states and local areas  flexibility in determining when to register a jobseeker in WIA and when  participants leave the program (see table 3).
Registration. When and who is registered affects all WIA performance  measures for adults and dislocated workers because performance data are  only collected for those job seekers who are registered under WIA—a  process that occurs when they begin receiving services that require  significant staff assistance. Labor has provided detailed written guidance  to states on who should be registered under WIA and when this  registration should occur, but the guidance is open to interpretation in  some areas. The guidance provides examples of when to register job  seekers, but it sometimes requires staff to make subtle and subjective  distinctions. For example, those who receive initial assessment of skill  levels and the need for supportive services are not to be registered; those  requiring comprehensive assessment or staff-assisted job search and  placement assistance are to be registered. In an earlier report, we found  that local areas differed on when they registered WIA jobseekers, raising  questions about both the accuracy and comparability of states’  performance data, and we recommended that Labor provide clearer  guidance. Inconsistencies in when states register participants could lead  some states to register fewer participants than others do, which could  affect the reported outcomes.
Exit. Determining when a participant leaves the program—or exits— affects nearly all WIA performance measures because jobseekers must  exit the program in order to be counted in the performance measures.  While Labor’s guidance explains when an exit occurs, it also has allowed  two different kinds of exits—the hard exit and the soft exit. A hard exit  occurs when a participant has a specific date of case closure, program  completion or known exit from WIA-funded or one-stop partner-funded  services. A soft exit occurs when a participant does not receive any   WIA-funded or partner-funded service for 90 days and is not scheduled for  future services except follow-up. Furthermore, Labor’s guidance on WIA  did not clearly specify which services are substantial enough to delay  exiting a participant, and local areas define these services differently. In a  recent review we found considerable variation in exit practices at the state  and local levels. For example, one local area defined exit as occurring  when participants are finished with their WIA services; another local area  defined exit when participants have found a new job and the wages for  their new job are considered acceptable (regardless of the number of days  that have passed since their last service).
In addition to allowing states the flexibility to define some performance  elements, the initial guidance failed to specify other key elements  necessary to ensure data quality. For example, the guidance did not  specify which source documentation was to be collected and maintained  to support entries into the IT system. In the absence of guidance, some  states continued to collect source documentation similar to that collected  under JTPA, other states moved to paperless systems and did not collect  and retain any source documentation. Without consistent source  documentation, there is no assurance that the data in the IT system are  accurate.
section_title: States Needed to Make Significant Changes to IT Systems That Initially Compromised Data Quality
paragraphs: The transition from JTPA to WIA required states to make significant  changes to their IT systems, and in some cases, problems during the  transition led to data errors. For example, several data elements required  in WIASRD—the file of individual exiters that states submit to Labor every  year—were similar to those collected under JTPA, but the data definitions  were slightly changed. This sometimes led to miscoded or missing data— especially for those participants who were carried over from JTPA into  WIA. In addition, new data sources were used to measure outcomes, and  the calculations for the measures were complex. Some states integrated  their IT systems so that the system that is used for WIA data collection is  used for tracking participation in other partner programs as well. These  changes required major modifications to the IT systems.
States used a variety of strategies to make the necessary system changes,  often facing challenges in fully implementing WIA’s requirements. For  example, 22 states reported that they used the same software they had  used under JTPA to report on WIA performance, but 15 of these states  later converted to different software for WIA. Twenty-six states used new  software for WIA at implementation, but almost one-third of them  replaced that system when it became clear that the new system was not  sufficient to meet WIA reporting requirements. The time needed to make  system changes varied across states. While nearly half of the states  reported that they were able to implement their IT system changes in   1 year or less, the other half reported that it took more than one year, and  as long as 3 years (see fig. 2). m. m. . .  . .
Thirty-nine states reported to us that they had made major modifications  to their WIA IT systems since implementation, such as converting to  Internet-based systems or adding new capabilities such as case  management tracking. Thirteen of these states reported that when they  made these modifications, they experienced significant problems that  affected the quality of the data, including lost data and difficulties in  combining or reconciling data from the multiple systems they had used.  While 8 of the states reported that these issues have been resolved, 5 told  us that they are still trying to resolve these data quality concerns. Some of  the remaining 11 states that did not report making major changes to their  IT systems since WIA implementation reported that they made minor  changes, such as adding or deleting data elements and adding reporting  capabilities.
In addition to collecting and reporting the performance data, IT systems  must also be able to calculate the performance measures. However, states  are not all using the same methodology to calculate these measures. The  calculations for the measures are complex and sometimes confusing. For  example, in calculating some of the measures for the adult program, states  must consider (1) whether the jobseeker is employed at registration,   (2) whether he or she is employed at both the first and third quarters after  exit, and (3) what data were used to confirm employment. This  information results in 14 different ways that adult participants can be  grouped together in order to calculate the measures. Labor does not  mandate which software package states must use to calculate their  performance measures, and at the 5 states we visited, each used a different  approach—commercially available software, software developed by the  state, or one of two different software packages developed under contract  with Labor. These software packages can use slightly different formulas to  calculate the measures and, as a result, produce differences in the  outcomes reported.
section_title: Monitoring of WIA Performance Data Was Limited at All Levels
paragraphs: Lack of oversight at the local, state, and federal levels made it difficult to  ensure that early WIA performance data are accurate or verifiable. During  the first year of WIA implementation, Labor’s Inspector General (IG) found  insufficient documentation of verification procedures at the state and local  levels. The same report questioned the lack of formal federal monitoring  to gauge the progress of state efforts to ensure the quality of the data.  Furthermore, the report noted that Labor and states lacked adequate  monitoring procedures and little was being done to monitor performance  data at the case file level. In a previous study, we reported that Labor did  not have a standard data monitoring guide in place, and regional  officials—who have primary responsibility for monitoring—followed  various oversight procedures. Table 4 summarizes WIA’s data quality  issues.
section_title: States Have Taken Steps to Improve the Quality of WIA Performance Data
paragraphs: States have made efforts to address data quality concerns and improve the  quality of WIA performance data. Most states have taken actions to clarify  Labor’s guidance to help local areas determine who should be tracked in  the performance measures. Almost all states reported on our survey that  they have controls for IT systems, such as edit checks or reports to help  screen for errors or missing data. In addition, most states reported to us  that they monitor local areas to ensure data quality and consistency by  assessing local procedures and policies.
subsections: section_title: States Have Taken Some Actions to Clarify Labor’s Guidance
paragraphs: States have taken some steps to provide additional clarity to help local  areas adhere to federal guidance. Over 40 states reported to us that they  provide guidance to help local areas determine which jobseekers should  be tracked—or registered—for WIA and when participants leave—or  exited—services, and therefore get counted in the performance measures.  For example, a West Virginia state official said the state developed a list of  staff-assisted services that should trigger registration under WIA. Most  states also provide technical assistance and training on registration and  exit policies (see fig. 3). Some states take other steps to help local areas  adhere to federal policies. For example, California state officials attempt  to prevent local areas from keeping participants enrolled in the program  once they have exited services by incorporating a capability in their IT  system that will automatically exit a person who has not had any service  for 150 days.
section_title: States Have Taken Steps to Reduce Errors in IT Systems
paragraphs: States have made efforts to reduce the errors in their WIA performance  data. Almost all states reported on our survey that they have controls for  IT systems, such as edit checks or reports to help screen for errors or  missing data. Forty-six states screen for missing values, and 44 states  screen for errors such as data logic inconsistencies (see fig. 4). For  example, if an individual is registered in the youth program, but the birth  date indicates that the person is 40 years old, this case would be flagged in  an error report checking for inconsistencies between these two data  elements. Some of the states we visited told us they allow local areas  flexibility in deciding who should enter data and how it gets done. In some  locations, a case manager who works with the participant may enter data,  and sometimes the case manager completes forms that are given to a data  entry specialist. Despite these differences, most states have implemented  edit checks and other controls in their IT systems to detect and control for  errors. For example, state officials we met with in West Virginia said that  the state created screen edits and drop-down menus to guide case  managers as they enter data. If a case manager does not enter the  necessary data, the system will not let the data entry process go forward  until the data are entered. State officials acknowledged that people  entering data can still make mistakes if they choose the wrong option on a  drop down menu, but they told us they try to minimize these mistakes by  conducting training sessions to acquaint staff with the right techniques.  States also address data entry errors by running error reports. In New  York, state officials told us that they produce error reports for each local  area to show where data are missing, meeting with local officials to  discuss these reports every 6 weeks.
Labor officials in most of Labor’s six regions told us that states have made  improvements to their IT systems since WIA was first implemented. For  example, Labor officials in one region said that they identified data quality  issues related to states’ IT systems in 10 of the 11 states in that region in  program year 2000 and found similar issues in only 5 states in the region  between program years 2002 and 2004. A Labor official in another region  told us that initial data collection efforts were poor because states were  largely focused on getting WIA up and running and had not developed  adequate IT system instructions. Now, most states have developed IT  system manuals with clear instructions. Some regional officials told us  that they provided technical assistance and closely monitored states that  had early problems with their IT systems.
section_title: Most States Conduct Monitoring of Key Data Elements at the Local Level
paragraphs: Most states told us they monitor local areas to ensure data quality and  consistency by assessing local procedures and policies. Thirty-eight states  reported to us that they monitor data collection at the local level. At least  33 states also reported to us that they conduct monitoring of local policies  and procedures on registrations and exits and data entry (see fig. 5). State  officials at the sites we visited generally said that they conduct annual  monitoring visits to local areas or one-stop centers, and some conduct  more frequent monitoring visits. Texas state officials we visited told us  that the state monitors each local area once a year that includes reviewing  participant files to assess eligibility decisions and ensure that outcomes  are documented. In New York, state officials said that they have  monitoring teams located in five regions across the state who visit the  local areas within their regions about once a month. Initially, these visits  focused on program compliance, but they have recently been expanded to  include data quality.
section_title: Labor Has Taken Steps to Improve WIA Data Quality, but Some Issues Remain
paragraphs: Labor recently began addressing data quality issues, however, some data  quality issues remain. In 2004, Labor addressed some data quality concerns  by implementing new data validation requirements that called for states to  review samples of participant files and provided software to help states  ensure that the performance measures are computed accurately. Most  states reported on our survey that Labor’s new requirements are having  positive effects on states’ and local areas’ attention to data quality.  However, Labor does not currently have methods in place to review states’  data validation efforts and hold states accountable to the data validation  requirements. Labor’s guidance requiring states to implement common  performance measures on July 1, 2005, clarified some key data elements  that had been problematic with regard to the WIA performance measures,  but it does not address all the issues. Further, Labor has some federal  monitoring processes in place but lacks a standard monitoring guide to  address data quality.
subsections: section_title: Labor’s Data Validation Requirements Address Some Data Quality Issues but Do Not Address All Concerns
paragraphs: To address data quality concerns, Labor required states to implement new  data validation procedures for WIA performance data in October 2004.  This process requires states to conduct two types of validation: (1) data  element validation—reviewing samples of WIA participant files, and   (2) report validation— assessing whether states’ software accurately  calculated performance outcomes. These requirements addressed a gap in  earlier guidance by providing instructions for collecting and retaining  source documentation to verify that the reported data are accurate. This  includes specifying which documentation is acceptable and what should  be maintained in participant files. For example, to document that a  participant is placed in post program employment, states must show that  the information was obtained from the UI wage records or Wage Record  Interchange System or other sources such as a pay stub, a 1099 form, or  telephone verification with employers.
Labor’s data validation process requires states to monitor local areas to  compare data elements that were reported to the state against source  documentation to verify that the data are accurate. Labor selected data  elements for validation based on factors such as feasibility and risk of  error. For example, self-reported data elements, such as race and  ethnicity, are not validated because it is not feasible to locate the  participant to verify these items. Data elements needing independent  documentation, such as the use of supplemental data sources to determine  employment, are assumed to be at higher risk of error than from using the  UI wage records. Labor provided software to help states select a sample of  files to be validated that includes participants from each group reported  on in the performance measures—adults, dislocated workers, older youth,  and younger youth.
States are required to conduct monitoring visits to the local areas selected  for validation and compare data elements for each participant in the  sample to source files to ensure accuracy, but Labor does not have a  standard process to verify that states did this correctly. State monitors  record whether each data element is supported by source documentation,  and therefore passes, or whether the documentation shows the element  was incorrect or was not supported with source documentation, and,  therefore, fails the element. States use Labor’s software to total error rates  for each population group and states submit these data to Labor.
To address inconsistencies in calculating the performance measures,  Labor’s report validation software verifies the accuracy of outcomes  reported by states. States can use Labor’s software in two ways: they can  use the software to compute the state’s performance measures or they can  use the software to check the calculations computed by their state’s  software to make sure that the measures were calculated accurately.  According to Labor, about 20 states are currently using its software to  compute their states performance measures.  The remainder of states use  their own or commercially available software to compute outcomes.   These states must submit validation reports to Labor to show any  differences between their calculations and the outcomes computed with  Labor’s software.
Since initiating data validation, Labor made a number of modifications to  its software, and states reported on our survey that they experienced some  challenges in using the software. Most states reported that they  experienced only minor difficulties or had no problems in using Labor’s  software for both data element validation and report validation (see fig. 6).  However, some states did report major difficulties. For example, seven  states reported that they initially had major difficulties with report  validation, such as resolving discrepancies or errors in Labor’s software.  States also reported concerns that they were not always informed when  Labor made updates to the software and did not always receive adequate  time to work with the software before the results were due to Labor. In  addition, some states reported on our survey that conducting data element  validation was time consuming. Half the states that were able to estimate  the time it took to complete data element validation said it took 60 days  and half said that it took more than 60 days.
The majority of states told us that Labor’s guidance, training, and technical  assistance on data validation were sufficient (see fig. 7).
section_title: Labor’s Data Validation Requirements May Be Having Some Positive Effects on States and Local Areas
paragraphs: It is too soon to fully assess whether Labor’s efforts have improved data  quality, however, at least 46 states reported on our survey that Labor’s new  requirements have helped increase awareness of data accuracy and  reliability at the state and local level (see fig. 8). A New York state official  told us that the federal requirements helped local staff better understand  the connection between the data that get entered and how these data  affect performance levels. In addition, over 30 states said that the new  requirements have helped them in their monitoring of outcomes and  eligibility. Some states and local areas we visited reported finding errors in  their data through the data validation process and have made  modifications to state and local procedures to enhance data quality as a  result. For example, a local area in California started doing monthly spot  checks of files to identify and correct errors on an ongoing basis. In New  York, a local area told us that it added a new staff person, developed new  forms and procedures, and centralized data entry to have more control  over data quality as a result of the federal data validation process. While  either centralized or decentralized data entry may be effective, experts in  WIA performance data told us that one of the most important factors to  avoid human error is for program managers and staff who enter data to  understand how the data are used.
section_title: Labor Currently Has No Method to Hold States Accountable for Complying with Data Validation Requirements
paragraphs: While Labor’s data validation requirements are having some positive  effects on states and local areas, Labor currently has no mechanism to  hold states accountable for complying with the data validation  requirements. Labor has plans to develop accuracy standards for report  validation and to hold states accountable to these standards in about   3 years. Initially, Labor planned to use program year 2003—July 1, 2003  until June 30, 2004—as a base year for developing accuracy standards on  report validation. However, as a result of reporting changes for the  common measures, Labor has postponed the development of these  standards until program year 2006, beginning July 1, 2006. At this time,  Labor does not have plans to develop accuracy standards for the data  element validation portion of its requirements. In addition, Labor does not  conduct its own review of a sample of WIA participant files verified by  states as part of data validation to ensure that states did this process  correctly. Table 5 provides a summary of data quality concerns and how  Labor’s data validation efforts affect these concerns.
section_title: Labor’s Recent Common Measures May Address Some Concerns, but Some Issues Remain
paragraphs: In response to an OMB initiative, Labor recently began requiring states to  implement common performance measures for WIA programs. OMB  established a set of common measures to be applied to most federally  funded job training programs that share similar goals.  Labor further  defined the common measures for all of its Employment and Training  Administration programs and required states to implement these measures  beginning July 1, 2005. In addition, Labor is replacing the definitions for  the WIA measures that are similar to the common measures with the new  definitions for common measures (see table 6).
Moving to the common measures may increase the comparability of  outcome information across programs and make it easier for states and  local areas to collect and report performance information across the full  range of programs that provide services in the one-stop system. Many  federal job training programs had performance measures that track similar  outcomes but have variations in the terms used and the way the measures  are calculated. For example, WIA’s adult program uses a different time  period to assess whether participants got a job than the Wagner-Peyser  funded Employment Service does. WIA’s adult program looks at whether  participants get a job by the end of the first quarter after exit, whereas the  Employment Service looks at whether participants get a job in the first or  second quarter after registration. Under common measures, both  programs use the same time period for this measure.
Labor’s new guidance for common measures requires states to collect a  count of all WIA participants who use one-stop centers. This can help  provide a more complete picture of the one-stop system, but it does not  clarify when participants should be registered for WIA and tracked in the  performance measures. Therefore, it raises questions about both the  accuracy and comparability of WIA’s outcomes for adults and dislocated  workers.  Under common measures, states are being required to begin  collecting and reporting a quarterly count of all jobseekers who receive  services at one-stop centers. To track these jobseekers, Labor suggested  that states collect a valid Social Security number, but allowed states to  exclude individuals who do not wish to disclose their Social Security  numbers. In addition, Labor is encouraging states to voluntarily report  performance information on all jobseekers that are counted in one-stops.  However, it is not clear how many states have the capability to track  jobseekers who receive only self-service and informational activities.
While 30 states reported on our survey that they have a state system to  track all jobseekers, some officials we visited told us they do not require  local areas to collect and report this information to the state. Given this,  implementing the new requirement may take time and early data collection  efforts may be incomplete.
Labor’s guidance on common measures provides for a clearer  understanding of when WIA participants should be exited from the  program than did earlier WIA guidance. First, the guidance provides a  more uniform definition of exit. In the past, local areas could use a hard  exit—when a participant has a known date of completion or exit from  services or a soft exit—when a participant has not received any services  for 90 days. Under the new guidance, only soft exits will be allowed and  states will no longer be able to report a hard exit. Second, Labor clarified  that some services are not substantial enough to keep a participant from  being exited from WIA. For example, if a case manager is only making  phone calls to the participant to see if he or she has a job or needs  additional services or income support payments, those phone calls are not  considered a service (see table 7). This new clarification may help prevent  local areas from keeping WIA participants enrolled long after they have  completed their last valid service. In a previous study, however, we  cautioned that rushed implementation of these reporting changes may not  allow states and local areas enough time to fully meet the requirements  and could negatively affect the data quality of the information reported.
section_title: Data Validation and Other Labor Efforts Address Some Monitoring Concerns, but Federal Monitoring Still Has Some Limitations
paragraphs: In addition to data validation, Labor has some limited federal monitoring  processes in place to oversee state and local performance data. Labor’s  regional offices—with primary responsibility for oversight—conduct a  limited review of WIA report data to review quarterly and annual WIA  performance reports. This generally involves identifying outliers or  missing data and comparing the data with data in previous reports. If  Labor regional officials identify basic problems with the data, they contact  states to reconcile concerns.
Labor’s headquarters implemented an electronic system to manage grant  oversight and track activities throughout the program year—called Grants  E-Management System (GEMS). This system provides automated tools for  conducting grant monitoring activities, including performing risk  assessments and generating reports. Labor developed the risk assessment  to help determine the programs and grant projects most in need of  monitoring. The risk assessment assigns a risk level to each state based on  past performance and other criteria. For example, for the WIA program, a  state may be considered at risk if it failed to meet its performance levels in  the prior year. However, regional officials can override the risk assessment  if they are aware of other information that may not be captured in GEMS.
Labor also implemented a core monitoring guide in spring 2005 to ensure  that certain basic parameters are being followed during monitoring visits  across all regions, but this guide does not provide for a standard analysis  of data quality issues. According to Labor officials, they are developing  program supplements for this guide that will address other issues specific  to various programs. One regional office developed an extensive  monitoring guide to review state and local guidance, procedures used for  data entry, IT systems, and other data quality factors. This guide has been  used since 2003 to review the eight states in its region. In addition, Labor  officials said that several regional offices are using this guide and they plan  to develop a similar guide that will be used across all regions.
section_title: Conclusions
paragraphs: WIA overhauled the way federally funded employment and training  services are provided to jobseekers and employers, and introduced  changes that significantly affected the way performance data are collected  and reported for WIA. Making this shift has taken a long time and some  trial and error on the part of Labor, states, and localities. The magnitude of  changes required considerable retooling of states’ IT systems, which had a  negative effect on the integrity of WIA performance data during the initial  years of implementation. Since then, states have made progress in  addressing challenges they faced in modifying or developing new IT  systems and have invested considerable effort establishing controls for IT  systems to minimize data errors.
In addition, Labor’s recent efforts to implement common performance  measures across many of the WIA partner programs and its revised WIA  reporting requirements have helped to address the concerns about when  participants complete services and should be tracked in the performance  measures. The new requirement for states to capture limited data on all  WIA participants is an important step to better determine the full reach of  WIA. However, this change still does not address the long-standing  challenge Labor has faced in clearly defining which participants should be  counted in the performance measures. Without clear guidance, the WIA  performance data will continue to be inconsistent, even if the other data  quality safeguards in place at the federal, state, and local level improve the  quality of each state’s and local area’ s data.
Labor’s implementation of new data validation requirements is a major  step toward addressing concerns about data quality resulting from the  limited guidance and monitoring of WIA performance data in the past. By  providing additional guidance and software to help states calculate the  performance measures in a more uniform manner and requiring states to  compare data reporting with participant case files, Labor has gone a long  way toward helping ensure the consistency and comparability of the data.  Most notably, these requirements have significantly raised awareness of  data quality at the state and local levels, which is an essential part of  ensuring data quality. However, more time is needed to fully assess the  impact these new requirements are having on data quality. In addition,  Labor does not currently review a sample of the participant files verified  by states, nor does it have a mechanism to hold states accountable for  meeting the data validation requirements. Further, Labor has not  developed a standard monitoring guide to more uniformly assess state and  local data collection and processing to ensure data quality. Without a  standard monitoring guide and a means to hold states accountable to the  data validation requirements, it will be difficult to assure decision makers  that the data is of sufficient quality for applying incentives and sanctions,  and making budget decisions.
section_title: Recommendations For Executive Action
paragraphs: To address the inconsistencies in determining when participants should be  registered and counted in the performance measures, we recommend that  the Secretary of Labor determine a standard point of registration and  monitor states to ensure that the policy is consistently applied.
To enhance the data validation requirements, we recommend that the  Secretary of Labor:    conduct its own review of the WIA participant files validated by states to  ensure that states did this correctly, and   ensure that steps are taken to hold states accountable to both the report  validation and data element validation requirements.
To address variations in federal monitoring practices, we recommend that  the Secretary of Labor develop a standard comprehensive monitoring tool  for WIA performance data that is used across all regions, including  monitoring the new guidelines for determining when participants end  services.
section_title: Agency Comments
paragraphs: We provided a draft of this report to Labor for review and comment. Labor  agreed with our findings and recommendations.  Labor agreed that the  lack of a standard point of registration and exit prevents comparisons  across states and leads to performance outcome information that is  arbitrary and inconsistent.  Labor also agreed that steps are needed to  increase the integrity of the data validation requirements and to improve  the completeness and consistency of oversight.   A copy of Labor’s  response is in appendix II.
In response to our recommendations, Labor noted that it plans to  implement a policy prior to the start of program year 2006 to clarify the  point of registration and exit. In addition, Labor plans to modify the  current data validation procedures to begin reviewing a sample of states’  validated files and plans to hold states accountable for data validation  results by program year 2006.  Further, Labor told us that it is taking steps  to develop a comprehensive monitoring guide for performance data and  plans to provide training on this new guide to help improve the  completeness and consistency of oversight.
We are sending copies of this report to the Secretary of Labor, relevant  congressional committees, and others who are interested. Copies will also  be made available to others upon request. The report is also available on  GAO’s home page at http://www.gao.gov.
If you or members of your staff have any questions about this report,  please contact me at (202) 512-7215. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report.  GAO staff who made major contributors to this report are  listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: We examined (1) the data quality issues that have affected states’ efforts to  collect and report Workforce Investment Act (WIA) performance data; (2)  states’ actions to address them; and (3) the actions the Department of  Labor (Labor) is taking to address data quality issues, and the issues that  remain. To learn more about states’ experiences implementing data  collection and reporting system changes for WIA, their implementation of  Labor’s data validation requirements for WIA, and state and local efforts to  address the quality of WIA data, we conducted a web-based survey of state  workforce officials and conducted site visits in five states, where we  interviewed state officials and visited two local areas or one-stop centers  in each state. We also collected information on the quality of WIA data  through interviews with Department of Labor officials in headquarters and  all six regional offices, nationally recognized experts, and reviewed  relevant research literature. Our work was conducted between June 2004  and September 2005 in accordance with generally accepted government  auditing standards.
subsections: section_title: Web-Based Survey
paragraphs: To determine the factors that affect the quality of WIA performance data,  we conducted a Web-based survey of state workforce officials. These  officials were identified using a GAO-maintained list of state WIA officials.  We e-mailed the contacts, and they confirmed that they were the  appropriate contact for our survey or identified and referred us to another  person at the state level. Survey topics included (1) the changes made to  data collection and reporting during the transition from the Job Training  Partnership Act to WIA, (2) the current status of WIA data collection and  reporting systems, (3) implementation of the U.S. Department of Labor’s  data validation requirements, and (4) state and local efforts to ensure the  accuracy and reliability of WIA data. The survey was conducted using a  self-administered electronic questionnaire posted on the Web. We  contacted respondents via e-mail announcing the survey, and sent follow- up e-mails to encourage responses. The survey data were collected  between February and May 2005. We received completed surveys from all  50 states (a 100 percent response rate). We did not include Washington,  D.C.,and U.S. territories in our survey.
We worked to develop the questionnaire with social science survey  specialists. Because these were not sample surveys, there are no sampling  errors. However, the practical difficulties of conducting any survey may  introduce errors, commonly referred to as nonsampling errors. For  example, differences in how a particular question is interpreted, in the  sources of information that are available to respondents, or how the data  are entered into a database can introduce unwanted variability into the  survey results. We took steps in the development of the questionnaires, the  data collection, and data analysis to minimize these nonsampling errors.  For example, prior to administering the survey, we pretested the content  and format of the questionnaire with several states to determine whether  (1) the survey questions were clear, (2) the terms used were precise, (3)  respondents were able to provide the information we were seeking, and  (4) the questions were unbiased. We made changes to the content and  format of the final questionnaire based on pretest results. In that these  were Web-based surveys whereby respondents entered their responses  directly into our database, possibility of data entry errors was greatly  reduced. We also performed computer analyses to identify inconsistencies  in responses and other indications of error. In addition, a second  independent analyst verified that the computer programs used to analyze  the data were written correctly.
section_title: Site Visits
paragraphs: We visited five states—California, New York, Texas, West Virginia, and  Wyoming,—and traveled to two local areas or one-stop centers in each of  these states. We selected these states because they represent a range of IT  systems—statewide comprehensive systems versus local systems with a  state reporting function, include single and multiple workforce areas, and  are geographically diverse. From within each state, we judgmentally  selected two local boards. In the case of our single workforce area state,  we visited two one-stop centers (see table 8).
In each state visited, we obtained general information about the state’s  implementation of WIA, an overview of the state’s WIA administrative  structure, the management information system and reporting processes in  place to meet the federal requirements, data quality practices at the state  and local levels, implementation of Labor’s data validation requirements.  We interviewed state officials responsible for local areas’ WIA programs  and analyzing and reporting on the state’s WIA performance data, as well  as other state WIA and information technology (IT) officials and staff of  the state’s Workforce Investment Board. At the local areas, we interviewed  WIA officials and staff, including service providers, staff responsible for  performance management issues, IT staff, case managers and other  frontline staff, as well as staff of the local area Workforce Investment  Board. The state and local interviews were administered using a semi- structured interview guide.
Information that we gathered on our site visits represents only the  conditions present in the states and local areas at the time of our site  visits, from August 2004 through March 2005. We cannot comment on any  changes that may have occurred after our fieldwork was completed.  Furthermore, our fieldwork focused on in-depth analysis of only a few  selected states and local areas or sites. On the basis of our site visit  information, we cannot generalize our findings beyond the states and local  areas or sites we visited.
section_title: Appendix II: Comments from the Department of Labor
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: Dianne Blank, Assistant Director  Laura Heald, Analyst-in-Charge In addition, the following staff made major contributions to this report:   Melinda Cordero, Vidhya Ananthakrishnan, and Leslie Sarapu served as  team members; Jennifer Miller assisted with early data collection.   Carolyn Boyce advised on design and methodology issues;   Susan Bernstein advised on report preparation; Jessica Botsford advised  on legal issues; Avrum Ashery and Robert Alarapon provided graphic  design assistance; and Bill Hutchinson and Daniele Schiffman verified our  findings.
section_title: GAO Related Products
paragraphs: Workforce Investment Act: Substantial Funds Are Used for Training, but  Little Is Known Nationally about Training Outcomes. GAO-05-650.  Washington, D.C.: June 29, 2005  Unemployment Insurance: Better Data Needed to Assess Reemployment  Services to Claimants. GAO-05-413. Washington, D.C.: June 24, 2005  Workforce Investment Act: Labor Should Consider Alternative  Approaches to Implement New Performance and Reporting  Requirements. GAO-05-539. Washington, D.C.: May 27, 2005  Workforce Investment Act: Employers Are Aware of, Using, and Satisfied  with One-Stop Services, but More Data Could Help Labor Better Address  Employers’ Needs. GAO-05-259. Washington, D.C.: February 18, 2005  Workforce Investment Act: States and Local Areas Have Developed  Strategies to Assess Performance, but Labor Could Do More to Help.  GAO-04-657. Washington, D.C.: June 1, 2004.
Workforce Investment Act: Labor Actions Can Help States Improve  Quality of Performance Outcome Data and Delivery of Youth Services.  GAO-04-308. Washington, D.C.: February 23, 2004.
Workforce Investment Act: One-Stop Centers Implemented Strategies to  Strengthen Services and Partnerships, but More Research and  Information Sharing Is Needed. GAO-03-725. Washington, D.C.: June 18,  2003.
Older Workers: Employment Assistance Focuses on Subsidized Jobs and  Job Search, but Revised Performance Measures Could Improve Access to  Other Services. GAO-03-350. Washington, D.C.: January 24, 2003  Workforce Investment Act: Youth Provisions Promote New Service  Strategies, but Additional Guidance Would Enhance Program  Development. GAO-02-413. Washington, D.C.: April 5, 2002.
Workforce Investment Act: Better Guidance and Revised Funding  Formula Would Enhance Dislocated Worker Program. GAO-02-274.  Washington, D.C.: February 11, 2002.
Workforce Investment Act: Improvements Needed in Performance  Measures to Provide a More Accurate Picture of WIA’s Effectiveness.  GAO-02-275. Washington, D.C.: February 1, 2002.