Workforce Investment Act: DOL Should Do More to Improve the Quality of Participant Data
Published: 2013-12-02T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Having reliable program data is important in effectively managing a program. However, there have been longstanding concerns about the quality of data on job seekers enrolled in the WIA Adult and Dislocated Worker Programs, which rely on states and local areas to track participants and the services they receive. Given these concerns and WIA's anticipated reauthorization, GAO was asked to examine the data on these WIA participants. This report addresses: (1) the factors that have affected the ability to report consistent and complete data on participants in the WIA Adult and Dislocated Worker Programs, and (2) actions that DOL has taken to improve the quality of these data. To conduct this work, GAO reviewed relevant federal laws, regulations, guidance, and documentation from DOL. GAO interviewed officials from DOL's national and regional offices and state and local workforce officials from a nongeneralizable sample of eight states. GAO also analyzed WIA data from program year 2011 to determine the number of, characteristics of, and services provided to WIA participants.
section_title: What GAO Found
paragraphs: Flexibility in the Department of Labor's (DOL) data reporting guidance and limitations in some state information systems continue to impair the quality of the data on participants in the Workforce Investment Act (WIA) Adult and Dislocated Worker Programs. The flexibility in the guidance stems from the inherent nature of WIA, which allows states and local areas to tailor program design and service delivery to their needs. As a result, DOL's guidance on collecting and reporting the data allows variation in how some WIA data elements are defined, collected, and reported. Specifically, an American Job Center--formerly known as a one-stop center--can choose to provide certain basic services exclusively through WIA programs, exclusively through a partner program, or through a blend of both WIA and partner programs. However, this flexibility involves variations in data reporting that have contributed to inconsistencies among states regarding when job seekers are counted as WIA participants. Moreover, some aspects of DOL's guidance are open to interpretation, leaving it to states to define variables such as type of training service received, further contributing to data inconsistencies. In addition, some state information systems used to collect and report WIA participant data have limitations that hamper the affected states' ability to report uniform and complete data. For example, data are incomplete to the extent that states may not have information systems that can track participants who access services online without significant staff assistance. Having inconsistent and incomplete data makes it difficult for DOL to compare data on program participants across states or to aggregate the data at a national level.
DOL engages in various oversight activities designed to ensure the accuracy of states' data on participants in the WIA Adult and Dislocated Worker Programs and has taken steps to improve data consistency across states. However, DOL does not consistently use the results of its oversight to identify and resolve systemic data issues nor has it evaluated the effect of oversight on the quality of WIA participant data. Specifically, DOL requires states to validate the data they collect and report on participants in the WIA Adult and Dislocated Worker Programs on an annual basis, but it does not strategically use the findings from this effort to identify systemic data issues or improve the quality of the data. Similarly, although DOL's regional offices review a sample of each state's WIA participant files every few years to assess states' compliance with data reporting and validation requirements, DOL officials said they have not analyzed the findings from the most recent reviews to identify nation-wide reporting issues. DOL has taken steps to improve the consistency of the data by providing general technical assistance to states and local areas and through standardizing the way DOL collects WIA data. For example, since 2007, two states have been piloting a unified reporting system developed by DOL that uses standardized data definitions and is integrated across certain American Job Center programs administered by DOL. However, DOL officials said they have no plans to evaluate the system before expanding it to other states. Without an evaluation, DOL does not know what impact the pilot has had on the quality of WIA participant data.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOL take steps to improve the consistency and completeness of data reported across states and to promote a continuous process for improving the data's quality. DOL officials did not agree or disagree with GAO's overall recommendations and detailed how data quality is being addressed primarily through existing efforts. However, GAO believes that the recommendations remain valid as discussed in the report.

section_title: Letter
paragraphs: Having reliable program data is an important factor in being able to  effectively manage and evaluate a program. Our previous work on the  Workforce Investment Act of 1998 (WIA), however, has raised questions  about the quality of data that states report to the U.S. Department of  Labor (DOL) on the number of individuals they serve. WIA, the  centerpiece of the nation’s employment and training system, established  three separately funded programs—Adult, Dislocated Worker, and  Youth—and created a comprehensive one-stop system, now known as  the American Job Center network, for the delivery of many federally  funded employment and training program services. At this writing, the  Congress is considering proposals to reauthorize WIA, which has been  due for reauthorization since 2003.
In our previous work, we found that the federal government’s efforts to  collect and report accurate, consistent WIA program performance data  from states were affected by (1) flexibility in federal guidance on collecting  and reporting the data, (2) major changes to states’ information systems  that resulted in lost data and other issues, and (3) limited monitoring by  the Department of Labor (DOL). We also found that DOL’s WIA data do  not include information on all participants in the Adult and Dislocated  Worker Programs and that the data were not comparable across states  and local areas. states were reporting inconsistent data on WIA participants in the Adult  and Dislocated Worker Programs because DOL’s guidance did not clarify  how certain participants should be counted. Given the longstanding  concerns about the quality of the data and in anticipation of WIA’s  reauthorization, you asked us to examine the data on WIA participants.  This report addresses the following questions:   Similarly, DOL’s Office of Inspector General found that  1.  What factors have affected the ability to report consistent and  complete data on participants in the WIA Adult and Dislocated Worker  Programs, and  2.  What actions has DOL taken to improve the quality of participant  data?
GAO-07-1051T; GAO-06-82.
Inspector General and representatives of two national workforce-related  organizations. To better understand the challenges faced by states in  reporting data on these WIA programs, we visited or telephoned DOL’s  Employment and Training Administration’s (ETA) six regional offices and  a nongeneralizable sample of eight states. Within each state, we visited  or contacted at least one comprehensive American Job Center—formerly  known as a one-stop center—or a local workforce board. We selected the  states to provide diversity on the basis of: (1) geographic location, (2)  total federal spending on the Adult and Dislocated Worker Programs in  program year 2010, (3) the extent of data issues identified by DOL’s data  contractor in the fourth quarter of program year 2010, (4) whether the  state reported participants who received only core “self-service,” and (5)  the number of local areas within the state.
We analyzed data for program year 2011 because they were the most recent full year of  WIASRD data available when we conducted our review. Program year 2011 ran from July  1 2011 through June 30 2012. estimates for the number of, characteristics of, and services provided to  participants whose information is recorded by DOL as having been  served by the WIA Adult or Dislocated Worker Program. However, we  did not find those data to be reliable for other purposes such as making  state-to-state comparisons because of variations in how states collect and  report participant data for the WIA Adult and Dislocated Worker  Programs. See appendix I for additional information on our scope and  methodology.
We conducted this performance audit from August 2012 through  November 2013 in accordance with generally accepted auditing  standards. Those standards require that we plan and perform the audit  work to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: WIA requires states and local areas to bring together a number of  federally funded employment and training programs into a comprehensive  workforce investment system, the American Job Center network. These  programs—including the Adult and Dislocated Worker Programs—are  known as mandatory partners, and must provide services through this  network (see tab. 1).
The WIA Adult and Dislocated Worker Programs are designed to provide  quality employment and training services to assist eligible individuals to  find and qualify for employment and to help employers find the skilled  workers they need. The Adult Program provides services to individuals  over the age of 18 who are job seekers, although states and local areas  must give priority of service to low-income individuals if funds are  determined to be limited. The Dislocated Worker Program provides  services to workers who have been or will be terminated or laid off from  employment. For fiscal year 2013, Congress appropriated over $1.9  billion for the Adult and Dislocated Worker Programs: $730 million for the  Adult Program and $1.2 billion for the Dislocated Worker Program.
DOL’s Employment and Training Administration administers the WIA  Adult and Dislocated Worker Programs and oversees their  implementation, which is carried out by states and local areas. Each state  must have one or more designated local workforce investment area, and  each local area must have at least one comprehensive American Job  Center where job seekers can receive core services and access other  programs and activities offered by the mandatory partners.  Although  each local area must have one comprehensive center, under WIA, the  mandatory partners have flexibility in the way they provide services  through the American Job Center network, and can co-locate services on  site or make referrals to external service providers or training, including to  local community colleges.
WIA provides three tiers, or levels, of service for adult and dislocated  workers: (1) core, (2) intensive, and (3) training.
Core services include basic services, such as job search or résumé-  building assistance, and may be accessed with or without staff  assistance.  Intensive services include activities such as staff-assisted  comprehensive assessment of a participant’s skill levels and case  management.
29 U.S.C. §§ 2831 and 2864(c)(2).
Training services include activities such as occupational skills or on- the-job training (see fig. 1).
Service at one level and a determination that a participant is unable to  obtain employment through that service are prerequisites for service at  the next level, although WIA does not specify the amount of time an  individual must spend or the number of attempts that must be made to  gain employment before moving to the next level. Job seekers who  receive only core services that are self-service and informational in nature  are not counted in the programs’ performance measures, but DOL  requires that states count such individuals as program participants. Self- service and informational activities can be accessed either at an  American Job Center or remotely, such as when a job seeker searches  for employment over an internet connection to an American Job Center  from a home computer.
As part of its oversight, DOL collects program data from states, which are  used to assess how well the Adult and Dislocated Worker Programs are  working. States must submit quarterly and annual performance reports to  DOL, in addition to uploading individual records on a quarterly basis to  DOL’s national WIASRD database. Specifically, states must submit  quarterly and supplemental monthly performance reports, a validated  annual performance report at the end of each year, and quarterly  WIASRD files for each reporting quarter for a program year. The  WIASRD files include demographic and characteristic information for  participants in the WIA Adult and Dislocated Worker Programs and  information about services received through these WIA programs, as well  as through some other partner programs. The process of collecting and  reporting WIA data involves all three levels of government. More  specifically, participant data are typically collected by staff at American  Job Centers and entered into a state or local information system. In some  states, local staff may enter data directly into a statewide information  system; in other states, local areas may use their own individualized  information systems to enter data from which they then must extract and  compile for submission to the state. After the data is submitted to the  state agency, it is compiled and formatted for the various submissions to  DOL, including for quarterly WIASRD record layout submissions. (see fig.  2).
section_title: DOL’s Guidance and Limitations in State Information Systems Have Resulted in Inconsistent and Incomplete National Data on WIA Participants
subsections: section_title: Flexibility in DOL’s Guidance Continues to Affect the Consistency of Data
paragraphs: DOL’s guidance to states, in the form of Training and Employment  Guidance Letters and Training and Employment Notices, details how  states should collect and report data on participants in the WIA Adult and  Dislocated Worker Programs. However, the flexibility in the guidance DOL  provides to states makes it difficult for DOL to provide consistent national  data on participants in these programs. The flexibility in the guidance  stems from the flexibility inherent in WIA, which allows states and local  areas to tailor service delivery to their needs. Consequently, DOL’s  guidance is designed to accommodate the different ways that states and  local areas can deliver services through the American Job Center  network, including decisions about whether to enroll participants in  partner programs. This flexibility, however, results in variations in how  states and local areas report participant data, which makes it challenging  for DOL to aggregate WIA data at the national level. In particular, it has  created inconsistencies among states in when a job seeker is counted as  a participant in the WIA Adult or Dislocated Worker Program. In addition,  DOL’s guidance is open to interpretation, allowing states to define and  report some variables differently, further contributing to inconsistencies in  the data that states report to DOL on these programs. For example, while  DOL requires states to report the type of training service provided to WIA  participants who receive such services, the agency’s guidance only lists  the six broad categories of training services states must report without  defining or describing them in detail. In accordance with government  internal control standards, management is responsible for developing  policies and procedures to achieve program objectives and clearly  communicating these policies and procedures to facilitate understanding  and consistent implementation.
The flexibility in DOL’s guidance on when states and local areas should  count individuals as participants in the WIA Adult and Dislocated Worker  Programs has resulted in inconsistencies in the data reported on program  participants. WIA allows for flexibility in the extent to which Adult and  Dislocated Worker Program services are integrated with those from other  programs. integrate WIA services with those from other partner programs so that job  seekers have access to a coordinated system of employment and training  services. As a result, states’ service delivery models differ in the extent  of integration between WIA services and other programs. For example,  state officials that we interviewed told us that they funded core services at  their American Job Centers exclusively through WIA, exclusively through  a partner program, or through a blend of both WIA and partner program  funds. According to officials from DOL’s national office, states and local  areas are best positioned to determine the mix of services that will meet  the needs of their job seekers. However, depending on which service  delivery model a state or local area selects, job seekers receiving the  same types of services may be counted as WIA participants at different  points in time, or they may never be counted under the WIA Adult or  Dislocated Worker Program.
Used in this context, integration can refer to the co-location of services at an American  Job Center, or to integrated funding to provide any given service. lower populations. For example, according to the state officials we  interviewed, Utah’s American Job Centers integrate WIA Adult Program  funding with funding for the Wagner-Peyser Program to provide core  services. As a result, every job seeker aged 18 years or older who  receives core services in Utah is counted as a participant in both  programs, as permitted under DOL’s guidance. In comparison, state  officials in California told us that the state typically funds core services  exclusively through the Wagner-Peyser Program and therefore counts all  individuals accessing core services from an American Job Center as  participants in the Wagner-Peyser Program, but not as participants in the  WIA Adult Program or the Dislocated Worker Program. Therefore, only  job seekers who meet the eligibility criteria for the WIA Adult or Dislocated  Worker Program and receive intensive or training services funded by  those programs are counted as WIA participants. Because of this  variability, the total counts of WIA Adult Program participants and WIA  Dislocated Worker Program participants represent different populations of  job seekers in different states, depending on the service delivery model  the state uses.
As a result of these differences, Utah ranked 3rd out of 53 states in the  total number of participants it served in the WIA Adult Program in  program year 2011, even though it ranked 36th in overall population.the same time, California ranked 27th in the total number of participants  served in the WIA Adult Program, even though it was the most populous  state (see fig. 3).
section_title: DOL Has Increased Its Oversight Efforts and Taken Steps to Improve Data Consistency, but Data Quality Issues Have Not Been Resolved DOL Performed Various Oversight Activities, but It Did Not Consistently Use the Results to Improve the Data
paragraphs: To improve the quality of data on participants in the WIA Adult or  Dislocated Worker Programs, DOL has enhanced its oversight efforts and  introduced new initiatives including having its data contractor produce  quarterly reports on data issues, requiring states to validate their WIA  data on an annual basis, and engaging its regional offices in periodic  reviews of case files from states. However, the agency has not  established a process to review the results of oversight to identify and  resolve systemic issues with the quality of participant data from the WIA  Adult and Dislocated Worker Programs. Our past work has shown that  the benefit of collecting performance information is only fully realized  when this information is actually used by managers to make decisions  oriented toward improving results.
DOL has a contract with SPRA to correct and analyze WIASRD data and provide data  files and reports on the accuracy of the data reported by states that are made available to  the public via DOL’s website. corrections SPRA makes to the state WIASRD data files may not be  accurate, and may result in incorrect data for a state.
The publicly available data files that SPRA produces are released  quarterly and include “data issues reports,” which identify issues or  anomalies in the WIA data submitted by each state to DOL. Officials from  DOL’s national office stated that the original intent of SPRA’s analysis  was to create the publicly available data files and not to report on data  anomalies or errors, even though SPRA has always identified issues with  the data while creating the publicly available files. However, since about  2010, when states started submitting quarterly WIASRD files instead of  annual ones, reviewing the quality of the data and issuing the quarterly  “data issues and anomalies report” has become standard practice under  SPRA’s contract, according to officials from DOL’s national office. The  agency’s regional offices are supposed to provide the states with the  published quarterly error reports and ask them to update any errors prior  to their next quarterly WIASRD submission.
DOL publishes SPRA’s data files and reports on its website, and its  regional offices are expected to share these reports with their states so  that the states can correct any data issues in their subsequent quarterly  submissions. However, officials from DOL’s Region 5 said that although  they receive these reports from DOL’s national office, they have not had  the chance to review them or comment on any of the errors for states in  their region. In addition, officials from DOL’s Region 1 said that although  these reports have always been available on DOL’s website, they only  recently began receiving the reports in a user friendly format and that they  only recently began providing feedback to SPRA on issues identified for  the states in their region. Our analysis of SPRA’s data issues reports  suggests that some states may not be using SPRA’s reports to improve  the quality of their data on WIA participants, which could be due to a lack  of awareness of these reports. For example, SPRA’s data issue reports  for the fourth quarters of both program years 2010 and 2011 identified  some of the same issues, such as errors in the dates of service reported  by the states. This suggests that the states that made these errors in  2010 may not have reviewed these reports and used them to correct the  data reported in the following year. In addition, while officials from four  states were aware of the SPRA reports, officials from four other states  told us they either were not aware of these reports or that they began  receiving these reports from the regions only recently, beginning with the  final report for program year 2011. When asked, officials from DOL’s  national office stated that they do not have a plan to systematically  identify or address recurring errors noted in SPRA’s reports. Government  internal control standards note that, for oversight and monitoring to be  effective, information should be recorded and communicated to  management and others within the entity and external stakeholders, and  this should be done within a time frame that enables them to carry out  their internal control and other responsibilities.
Officials added that they do not conduct formal oversight reviews or  audits of SPRA’s data analyses because they consider SPRA to be the  “data experts” and, therefore, do not know what kind of oversight they  could provide. According to government standards for internal controls,  agencies should ensure that ongoing monitoring occurs in the course of  normal operations, which would include monitoring and oversight of  contractors through regular management and supervisory activities. These officials also said they believe any major data issues or obstacles  would be uncovered by their internal data edit checks, which are run on  all WIASARD data submitted by the states prior to SPRA’s review of the  data.
GAO/AIMD-00-21.3.1. correcting the data. For example, officials from California explained that  SPRA changed some of the numbers entered by local areas for WIASRD  variable 325—Employment and Training Programs Related to Food  Stamps—to zeros because the numbers entered seemed too high.  According to the state officials, the numbers for that variable entered by  the local areas were correct. States, however, are generally not provided  an opportunity to review and verify SPRA’s changes before they are  made, as they only receive copies of the data issues reports after they  are published. Officials from both DOL’s national office and from some of  the state workforce agencies noted that not all issues identified by SPRA  represent actual errors and that some outliers on certain variables are  acceptable.
DOL requires each state to validate the data it collects and report on  participants in the WIA Adult and Dislocated Worker Programs on an  annual basis, but the findings from these validation efforts have not been  strategically used to identify systemic issues with or to improve the quality  of the data on WIA participants. Government standards for internal  controls state that for oversight and monitoring to be effective, information  should be communicated to management and others, along with the use  of this information for program assessment, so that it can be used to carry  out their internal control and other responsibilities.
Each year, states are required to review a sample of WIA participant  records to determine whether the source documents match the  information in the electronic records states use to collect and report data  to DOL in WIASRD. DOL requires states to validate the accuracy of the data they submit annually to ensure that decisions about WIA policy and  funding are made based on a true picture of the number of participants  and program outcomes. Although DOL has established a provisional 5  percent error rate threshold for states’ validation of the variables, the  agency does not have plans to tie the results of these validation efforts to  DOL’s financial awards or penalties because, according to officials from  DOL’s national office, the results of states’ validation efforts were never  intended to be used for enforcement purposes. Officials from DOL’s  national office explained that, in their opinion, DOL’s regional offices  should be using the states’ validation efforts as a management tool to  improve the quality of their data by identifying inaccurate or confusing  variables to target the technical assistance they provide to states and  local areas.
DOL, however, does not know what effect state data validation efforts  have had on the quality of participant data for the WIA Adult and  Dislocated Worker Programs. In addition, our interviews with regional and  state officials suggest that DOL’s regions and the states are not always  using the results of these data validation efforts to improve data quality or  target technical assistance. During our interviews with regional and state  officials, only those from Region 3 and Massachusetts described specific  efforts to use the results of the state data validation efforts to improve  data quality and direct technical assistance. Officials from DOL Region 3  noted that the region recently began requiring states to respond to  findings from states’ annual data validation efforts and to track error rates  found in each quarterly submission. Similarly, officials from  Massachusetts said that they use the results of their data validation  efforts to direct the technical assistance the state provides to local areas  and to improve the quality of the state’s WIA data. In contrast, officials  from one state said that, although they receive information about errors  associated with specific case files as they enter data from each file into  DOL’s data validation system, they do not know how to retrieve their  state-wide results from DOL’s database and they have not received any  reports from DOL documenting the nationwide results of the data  validation efforts. Officials from DOL’s national office said that they were  surprised by this, and that data element validation results are available to  states through DOL’s reporting system.
Moreover, our analyses of the results of DOL’s efforts to validate  WIASRD variables for program years 2010 and 2011 suggest that DOL’s  data validation efforts have not prevented high error rates on certain data  elements for select variables—nationally, error rates for certain variables  have remained well above the 5 percent threshold over both program  years. For example, in program year 2010, across all states, about 16  percent of the files for the Adult Program for which the “date of program  exit” variable was reviewed had errors, compared to around 14 percent in  program year 2011. Similarly, the nationwide error rate for “date of first  staff-assisted core service” was above 7 percent in both program years  for both the Adult and the Dislocated Worker Programs. When asked,  officials from DOL’s national office and two regional offices explained that  variables containing dates frequently have high error rates due to  discrepancies between the date reported and the date in the source  documentation. If a date, such as date of dislocation—the date a worker  lost his or her job—in the hard copy document differs from the date in the  electronic record, even by one day, the variable for that record “fails” the  validation check. Moreover, officials from DOL’s national office and three  regional offices stated that high error rates resulting from such  discrepancies do not necessarily reflect any serious issues with the  reported data—a participant would still be a dislocated worker whether,  for example, the date of dislocation was June 20th or June 21st.  Nonetheless, DOL requires these dates to match precisely and errors  noted in program year 2011 for these variables were still prevalent  although similar errors were noted the previous year.
DOL’s required annual data validation efforts are resource-intensive and  time-consuming both for DOL regions and states, according to officials  from DOL’s national office, two DOL regional offices, and five states.  However, DOL has not yet evaluated the process or determined its effect  on data quality. Specifically, an official from one state workforce agency  estimated the cost of its annual data validation efforts, including staff time,  travel, and other expenses, to be about $200,000. Officials in another  state explained that they would like DOL to reduce the required sample  size for the required validation of the data elements in WIASRD in order  to reduce the administrative burden on states. In addition, in 2011 one  DOL regional office convened a workgroup of representatives from four  states that analyzed the data validation procedures and provided  recommendations to DOL’s national office for improvements to reduce the  administrative burden on states. These recommendations included  considering using alternative sampling methods, revisiting the frequency  and precision requirements of data validation, and issuing guidance to  share “best practices” across states and local areas. However, as of  September 2013, DOL had not implemented the workgroup’s  recommendations, and the data validation process remains unchanged.  Officials from DOL’s national office acknowledged the trade-off between  monitoring data quality and minimizing the administrative burden on  states. They said that revising WIASRD’s edit checks to allow states more  flexibility may result in making the data validation process more efficient  by, for example, permitting states to report a range of dates, if  appropriate, for certain variables.
DOL’s regional offices review a sample of case files from states as part of  their oversight of the quality of data for the WIA Adult and Dislocated  Worker Programs, but they have not used the results of these reviews to  identify systemic issues with the quality of the data on WIA participants.  According to officials from DOL’s national office, its regional offices are  responsible for providing feedback to states based on these reviews, and  it is not the national office’s role to conduct any type of systemic review to  identify cross-state data issues. Government standards for internal  controls state that for oversight and monitoring to be effective, information  should be communicated to management and others, along with the use  of this information for program assessment, so that it can be used to carry  out their internal control and other responsibilities.
In response to a recommendation from a prior GAO report, DOL began to  require its regional offices to review a sample of case files to monitor  states’ annual data validation procedures. In addition, over the past few years, to address concerns about the nationwide consistency of  monitoring activities, DOL has issued additional guidance to its regional  offices on the process that should be used in reviewing the case files.
Officials from all six of DOL’s regional offices reported using these  materials when they design and conduct their reviews of the case files.  The review process begins when officials from DOL’s regions review a  state’s most recent data validation report and identify a subsample of  participant case files from the most recent review by the state. In addition  to checking the data in the electronic records by comparing them to the  source documentation, DOL’s regional staff assess whether the state  followed the proper procedures in conducting its annual data validation  efforts, according to officials from DOL’s national office. The process  concludes with a report outlining the DOL regional office’s findings,  including non-compliance with statutory or regulatory requirements for  collecting and reporting data on WIA participants. Many of the reports  also identify areas of concern, such as when states do not share their  annual data validation results with local area staff from American Job  Center partner programs, and identify promising practices observed  during the reviews. State officials have 45 days to respond to findings in  the region’s report and are also encouraged but are not required to  respond to areas of concern detailed in the report. In addition, officials  from DOL’s national office noted that states with high error rates on select  variables are encouraged to inform the regional offices of how they plan  to reduce their error rates in the future.
Officials from DOL’s national office said that, while they discuss the  results of these reviews with regional officials and provide state-specific  technical assistance as needed, they do not have a regular, formal  process for analyzing the findings from these reviews by their regional  offices, including determining whether similar findings and areas of  concern were identified across states. DOL officials explained that,  because the reviews are part of the regional offices’ oversight of the  states, they believe that the national office should not be involved in  monitoring the results of the reviews or the way in which they are  conducted. As a result, DOL does not have a systematic means of  determining the importance of the findings, their prevalence, or their likely  effect on the quality of the national data on participants in the WIA Adult  and Dislocated Worker Programs. This limits DOL’s ability to respond to  data issues that are systemic or widespread. Table 2 summarizes our  analysis of the most prevalent issues identified during the most recent  reviews for each of the 53 states and territories.
Over the past few years, DOL has issued additional guidance and  provided technical assistance to states and local areas, including training  and webinars, to clarify and explain the requirements for collecting and  reporting data for the WIA Adult and Dislocated Worker Programs.  However, some state officials said that DOL’s technical assistance is not  always timely, and that DOL could do more to facilitate the collection or  sharing of promising data collection and reporting practices across  regions and states.
For example, DOL has provided general technical assistance on data  reporting for the WIA Adult and Dislocated Worker Programs to states,  and officials from three of the eight states said that the assistance  provided by DOL’s regional offices was useful in helping them address  some of the challenges related to data reporting. In particular, officials  from DOL’s national office said that some regional offices issue quarterly  performance letters to states that include program year performance data  and any related analysis, in addition to hosting quarterly phone  conferences with state performance specialists to discuss performance  issues. DOL also sponsors the Workforce3one website, which contains a  variety of training and background materials related to data collection and  reporting for WIASRD. National and regional DOL data specialists also  said they hold biweekly meetings to discuss data issues. In addition, DOL  officials from each region described regular communication they have  with state officials to provide technical assistance in response to specific  data reporting issues, and officials from five of the six regional offices  described conference calls that they have hosted as opportunities for  states to discuss challenges related to the quality of their data on WIA  participants. Furthermore, DOL’s national and regional offices have  access to an internal data system, Infospace, which allows them to  retrieve and review publicly available WIASRD data by state and local  area.
Over the past few years, DOL has also issued a number of Training and  Employment Guidance Letters and Training and Employment Notices  related to data collection and reporting for WIA. In 2011, DOL developed  and hosted a series of webinars for states and local areas, including one  on data validation and data quality issues. The materials from this  session describe the approach DOL uses to monitor the data and explore  issues and findings across states. They also highlight how data validation  can be used to improve data systems and provide information on the  guidance regional offices use to review states’ case files. While the  webinar materials present information on some of the consistent findings  across states—such as incorrect source documentation and exit dates— DOL officials said that they have not provided any additional formal  technical assistance to address these issues on a national level because  of resource limitations. Moreover, they said it would be difficult for DOL to  provide such assistance without having first reviewed the results of its  own monitoring efforts. DOL’s webinar also noted that DOL would  establish a working group in the summer of 2011 to look into and possibly  revise the source documentation requirements for the annual data  validation process. However, as of the summer of 2013, DOL officials  said that this working group had not been established due to competing  priorities and resource constraints.
Officials from two states noted that DOL’s updates to its guidance for  collecting and reporting the data are often not provided far enough in  advance to be implemented by the time changes take effect. For  example, officials from California said that they often do not have enough  lead time to properly implement new data elements or guidance when it is  issued by DOL. According to state officials, the state generally has 1  month or less to change its automated system to meet deadlines, which is  not enough time. In addition, officials from Georgia said the data  validation reports they currently receive from DOL are not timely since  they are at least 3 months old by the time the state receives the reports.  Some state officials also told us that they would benefit from learning how  their peers are addressing challenges in reporting data on the WIA  programs. Officials from DOL’s national office, however, told us they do  not currently facilitate the collection or sharing of promising data collection  and reporting practices across regions or states, and have no plans to do  so—because of competing resources, the agency’s main focus is on  service delivery rather than data collection. Best practices state that high- performing organizations continually assess performance and efforts to  improve performance. In particular, managers can use performance  information to identify and share more effective processes and  approaches.
subsections: section_title: DOL Has Taken Steps to Improve Data Consistency, but It Has Not Evaluated These Efforts
paragraphs: In addition to its regular monitoring activities, DOL has taken specific  steps to improve the consistency of the WIA data collected by states and  local areas. For example, the agency has developed an integrated data  reporting system, which is being piloted in two states. However, DOL  has not evaluated the results of the pilot program to determine whether it  has had a positive effect on the quality of participant data for the WIA  Adult and Dislocated Worker Programs and, despite not having evaluated  its effectiveness, plans to expand the program to additional states.
To standardize and streamline reporting across several of DOL’s  workforce programs—the Wagner-Peyser Program, the WIA Adult,  Dislocated Worker and Youth Programs, Veterans Employment and  Training Service, National Emergency Grants, and Trade Adjustment  Assistance Programs—the agency has developed an integrated data  reporting system, WISPR. Two states, Pennsylvania and Texas, have  been piloting WISPR since 2007 to collect and report data on the WIA  Adult and Dislocated Worker Programs, and a third state—Utah—plans to  start piloting WISPR in the fall of 2013. Officials from DOL’s national  office and from Utah explained that one of WISPR’s key advantages over  the current separate reporting systems for each program is that WISPR  has standardized variables that include all the required variables for each  program. improvement over their current system because future changes in DOL’s  guidance for any of the workforce programs it administers—including the  WIA Adult and Dislocated Worker Programs—would be incorporated into  a single system, facilitating implementation of these changes.
Although DOL’s guidance for reporting data in WIASRD encourages states and local  areas to provide integrated services through multiple programs, each program has its own  reporting requirements, according to officials from DOL’s national office. As a result, it is  not possible to track individual job seekers who receive services from multiple programs  across the workforce system, or to determine the proportion of resources provided by  each program for a particular service, or to attribute participant outcomes to those  programs. the current record layout of WIASRD to match that of WISPR. DOL  officials told us they expect to implement the revised WIASRD record  layout in the fall of 2013. However, as of August 2013, they said it is not  clear whether or when nationwide implementation of WISPR will occur  because this depends on the resources available to upgrade both federal  and state information systems and the associated programming costs.  While WISPR appears to offer advantages over the current reporting  system that might make it a promising step forward, DOL does not  currently have plans to evaluate the results of the pilot program to  determine whether it has had a positive effect on the completeness and  consistency of participant data for the WIA Adult and Dislocated Worker  Programs before expanding the program to other states. DOL officials  cited the agency’s limited resources as the reason for not planning an  evaluation of the WISPR pilot program before expanding it to other states.  However, best practices note that evaluation can play a key role in  program planning, management, and oversight by providing feedback to  program managers, legislative and executive branch policy officials, and  the public. Further, when pilot programs are designed to produce  change—such as by allowing for more streamlined data collection and  reporting—assessing the impact is essential for knowing if the pilot is  meeting its goals. Without an evaluation of WISPR, DOL will not know if  this data system has resulted in the collection of more accurate WIA  participant data when compared to WIASRD.
Finally, DOL administers two grant programs that states can use to  improve their WIA information systems: the Workforce Innovation Fund  and the Workforce Data Quality Initiative.Fund is a competitive grant program that supports innovative approaches  to the design and delivery of employment and training services. Although  it is not targeted specifically at information systems, at least 3 of the 26  grants awarded by DOL have been used for local initiatives to integrate  their workforce data systems. For example, one local area we visited in  Chicago received a Workforce Innovation Fund grant that they plan to use  to integrate the different data systems used by the workforce programs in  their local area. Officials said that they hope the improvements will result  in more data-driven decisions about service delivery, and that all  employment and training programs in their local area will be able to share  information electronically. Another grant program, DOL’s Workforce Data  Quality Initiative, is also not specifically aimed at WIA data reporting but  may have positive incidental effects on WIA data quality, according to  DOL officials. The purpose of this initiative is to create a longitudinal  database to chart individuals’ progress through the education system and  beyond to the labor market. This effort would entail upgrades to state  information systems that may resolve some data reporting issues  currently attributed to limited technological capacities. At this time, it is too  early to know whether these grants will have a positive effect on the  quality of WIA participant data.
section_title: Conclusions
paragraphs: Collecting and reporting consistent and complete data is important for  program oversight and management and to evaluate the effectiveness of  program activities and services, but it can be difficult when federal  programs are carried out in partnership with states and local areas. DOL  has taken steps to improve the quality of the data on WIA’s Adult and  Dislocated Worker Programs. However, the flexibilities in DOL’s  guidance, reflective of those inherent in the programs’ authorizing statute,  which allows states flexibility in program design, along with limitations in  state information systems, present challenges to DOL in collecting and  reporting consistent and complete data on a  unique count nationwide of   participants in the WIA Adult and Dislocated Worker Programs. Without  such data, policymakers, program officials, and other stakeholders have  an incomplete picture of the number of adults and dislocated workers  served, their characteristics, and the type and level of services received.
In addition, while DOL engages in several types of oversight activities  designed to ensure the accuracy of states’ data on participants in the WIA  Adult and Dislocated Worker Programs, it does not consistently share the  results of its oversight activities with states and local areas. As a result,  states and local areas are not always aware of potential data quality  issues, and may miss opportunities to improve their data collection and  reporting. Moreover, since 2007, two states have been piloting a new  information system that tracks program participants across several of  DOL’s employment and training programs, but DOL does not plan to  evaluate its effects on the quality of the data collected on participants in  the WIA Adult or Dislocated Worker or other programs before it expands  the system to other states. Similarly, DOL does not regularly collect and  disseminate promising practices to states and local areas, which could  facilitate the adoption of steps other states and local areas have taken to  improve their data collection and reporting efforts. While it may not be  possible to achieve 100 percent precision and accuracy in the data  reported on participants in a large, complex system like the workforce  investment system, by not appropriately targeting their available  resources and facilitating sharing of promising practices among states to  continuously try to improve the quality of the data, DOL misses an  opportunity to identify and address longstanding, systemic issues.
section_title: Recommendations for Executive Action
paragraphs: 1.  To improve the consistency and completeness of national data on  participants in the WIA Adult and Dislocated Worker Programs, we  recommend that the Secretary of Labor take additional steps to  improve the uniformity of participant data reported by states. These  steps could include the following:  a.  providing additional guidance to states on data reporting, such as  how core and intensive services should be recorded for WIA  participants who receive these services through partner programs;  and  b.  conducting an evaluation or review of WISPR to determine if it has  resulted in more complete and consistent data collection and  reporting for participants in the WIA Adult and Dislocated Worker  Programs and placing a high priority on the implementation of  WISPR if it is shown to improve data consistency and  completeness.  2.  We also recommend that the Secretary of Labor promote a formal,  continuous process for improving the quality of data on participants in  the WIA Adult and Dislocated Worker Programs through such  measures as the following:  a.  consistently sharing the results of all oversight activities with  states and local areas, including findings from validation of  participant data;  b.  reviewing the methods used for data validation, such as its scope  and error rate threshold, to identify opportunities to increase  efficiencies and accountability in the process. This could include  implementing, if appropriate, recommendations from the Regions’   review of data validation procedures;  c.  evaluating data validation efforts to determine their effects on data  quality, particularly on systemic errors, and providing targeted  guidance and assistance to states and local areas to address  such errors;  d.  regularly monitoring Social Policy Research Associates’  corrections and analyses of state WIA participant data, sharing  this information with states, and coordinating with states to ensure  that any corrections are appropriate and accurate; and  e.  collecting and disseminating promising practices to states and  local areas on data collection and reporting on a regular basis.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to officials at DOL for their review and  comment. We received written comments from DOL, which are  reproduced in their entirety in appendix III. DOL officials did not state  whether they agreed or disagreed with our recommendations. These  officials acknowledged the importance of having reliable data to  effectively manage and evaluate the WIA Adult and Dislocated Worker  Programs; however, they commented that data reliability should be  balanced with the flexibility WIA gives to states and with DOL’s  responsibility to prioritize use of its limited resources. They stated that  WIA provides states and local areas with the flexibility to serve their  customers in the way that best suits their particular needs. DOL officials  also stated that the agency has invested significant resources in its  workforce performance accountability system, especially for WIA  programs. According to officials, the agency has a robust system in place  to ensure data quality and reliability and has recently made several  enhancements to the reporting system. In their comments, DOL officials  detailed various efforts they plan to take to address our  recommendations. Nonetheless, we believe that these efforts will not  sufficiently address the specific data quality issues we identified and  encourage DOL to take more targeted steps as outlined in our  recommendations.
In response to our first recommendation, DOL officials said that they  believe the agency’s current guidance is clear but that they will continue  to work with states to develop additional guidance, as necessary, such as  forthcoming guidance on how to avoid duplication of services when co- enrolling participants across multiple partner programs. However, it is  important that any additional guidance also specify when to count job  seekers as WIA participants if they also receive services funded by  partner programs. We also encourage DOL to develop additional  guidance for the WIASRD variables noted in our report that are open to  interpretation, such as “type of training,” to facilitate consistent reporting  on participants in these programs. DOL officials also noted that an  evaluation of WISPR is subject to the agency’s resource constraints,  adding that the purpose of WISPR was never explicitly to improve data  quality. As we stated in our report, however, WISPR seeks, in part, to  improve the consistency of WIA data by standardizing reporting across  the workforce system. As such, it has the potential to improve data  quality. Therefore, we encourage DOL to evaluate the system in order to  make an informed decision on how best to allocate finite agency  resources going forward.
In response to our second recommendation, DOL officials stated that they  consistently share the results of the agency’s oversight activities with  states but acknowledged that more could be done to analyze the results  of its activities to identify and share similar findings and areas of concern  across multiple states. DOL officials added that they will work with the  regional offices towards this goal. With regard to the agency’s data  validation methods, officials said they regularly review these methods and  solicit input from states on how to improve them. Specifically, DOL  pointed out that, as required by the Paperwork Reduction Act, the Office  of Management and Budget reviews DOL’s data validation process every  3 years and solicits public comment before approving the methodology  and authorizing data collection, and that their 2014 submission will reflect  state input. However, given the time-consuming nature of data validation,  we believe the agency should take additional actions to review its current  methods specifically with an eye toward making them more efficient and  holding states accountable for their data validation results.
With regard to evaluating the effectiveness of its data validation efforts,  DOL officials said that the agency plans to consider the regional data  validation workgroup’s findings and recommendations from 2011, explore  ways to streamline the process, and examine the effect of data validation  on error rates. We commend DOL’s plans, but to adequately address the  persistently high data error rates we found in our analysis, we believe it is  necessary to go beyond evaluating the effectiveness of its data validation  efforts and pinpoint the underlying cause of the errors so that they can be  addressed.
Officials also pointed out that the agency already monitors and shares the  analyses of state data conducted by its contractor, Social Policy  Research Associates (SPRA). They stated that SPRA’s corrections of  states’ data have been publicly available with the data set from the  inception of WIA. They also noted that since program year 2011 DOL has  provided SPRA’s analyses and corrections to the states (through ETA  Regional Offices) on a quarterly basis for states to either correct or  dispute. They noted that this is a formal and recurring process, and that  ETA Regional Offices have begun to analyze state date WIASRD data on  a regular basis as part of their annual review cycles. However, we found  that not all states are aware of or receive copies of SPRA’s reports, and  that some of the corrections SPRA makes to the state WIASRD data files  may not be accurate. Furthermore, as noted in our report, DOL officials  told us that they do not conduct formal oversight reviews or audits of  SPRA’s data analyses.
In addition, DOL officials reiterated that data collection and reporting are  topics that are included in workforce3one, a point we noted in our report.  However, we maintain that DOL could do more to facilitate the sharing of  information across states, such as creating a forum through which states  could learn how their peers are addressing challenges in data reporting  for participants in the WIA programs. Finally, DOL provided technical  comments, which we incorporated, as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, the Secretary of Labor, and other interested  parties. In addition, the report will be available at no charge on the GAO  website at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-7215 or moranr@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors to this report are listed in  appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: Our objectives were to determine: (1) what factors have affected the  ability to report consistent and complete data on participants in the  Workforce Investment Act (WIA) Adult and Dislocated Worker Programs,  and (2) what actions has the Department of Labor (DOL) taken to improve  the quality of participant data. To address our objectives, we reviewed  applicable laws and regulations, as well as DOL’s guidance to states for  collecting and reporting data on participants in the WIA Adult and  Dislocated Worker Programs. We also interviewed officials from DOL’s  Employment and Training Administration, its Office of Inspector General,  and its six regional offices. In addition, we visited or telephoned a  nongeneralizable sample of eight states. Within each state, we visited or  contacted at least one American Job Center—formerly known as a one- stop center—or a local workforce board. We also assessed the reliability  of program year 2011 data from the Workforce Investment Act  Standardized Record Data (WIASRD) system by testing the data  electronically and interviewing knowledgeable agency officials and DOL’s  data contractor. We found the data in appendix II to be sufficiently  reliable for the purposes of providing estimates of the number of,  characteristics of, and services provided to participants whose information  is recorded by DOL as having received services from either the WIA Adult  Program or the WIA Dislocated Worker Program. The data are not  reliable for other purposes, such as making state-to-state comparisons,  because of variations in how states collect and report data on participants  in the WIA Adult and Dislocated Worker Programs.
We conducted these interviews between September 2012 and June 2013. with DOL officials in Regions 1 (Boston), 3 (Atlanta), and 5 (Chicago);  state and local workforce officials in California, Georgia, Illinois,  Massachusetts, and Washington; and American Job Center officials in  Maryland. We conducted telephone interviews with DOL officials in  Regions 2 (Philadelphia), 4 (Dallas), and 6 (Sacramento), and with state  workforce officials in Maryland, South Dakota, and Utah. We  nonstatistically selected these states to provide diversity on the basis of:  (1) geographic location, (2) total federal spending on the Adult and  Dislocated Worker Programs in program year 2010, (3) the extent of data  issues identified in the fourth quarter of program year 2010, (4) whether  the state reported participants who only received core self-services, and  (5) the number of local areas within the state.
In each state, we obtained general information about the state’s and the  local area’s implementation of the WIA Adult and Dislocated Worker  Programs and on any challenges they may have encountered in  collecting and reporting data on program participants. We also asked  about actions DOL has taken to improve the quality of the data on  participants in the WIA Adult and Dislocated Worker Programs. We used  semi-structured interviews for our regional, state, and local interviews.  Because we interviewed officials from a nongeneralizable sample of eight  states and selected local areas, we cannot generalize our findings  beyond the data collected on those states and local areas.
To assess the reliability of DOL’s data in the WIASRD database for  participants in the WIA Adult and Dislocated Worker Programs, we (1)  reviewed existing documentation related to the data sources, including  reports issued by DOL’s Office of Inspector General; (2) electronically  tested the WIASRD data to identify potential problems with consistency,  completeness, or accuracy; and (3) interviewed DOL’s data contractor  and knowledgeable agency officials to obtain information about the data.
Our electronic testing consisted of identifying inconsistencies, outliers,  missing values, and other errors. More specifically, the electronic testing  included assessing the reliability of data collected on the characteristics  and the services participants in the WIA Adult and Dislocated Worker  Programs received in program year 2011. Prior to testing the data, we  combined 160 records that were overlapping or duplicative into 80 unique  records and removed 19 records that had missing or erroneous  participation dates. A few variables, including data on a participant’s  dislocation date—the date a worker lost his or her job, and the  occupational codes for participants who completed training, were found to  not be sufficiently reliable for our purposes and were not included in our  report. In addition, we analyzed the publicly available WIASRD data file  for program year 2011, which was produced for DOL by its data  contractor, Social Policy Research Associates. As part of our analysis, we  reviewed the steps the data contractor took to correct the data and, to the  extent possible, compared our data to the publicly available file. In light of  variations in how states collect and report participant data for the WIA  Adult and Dislocated Worker Programs and limitations in their information  systems, the actual number of participants in these programs is unknown.
However, we were able to estimate the number of, characteristics of, and  services provided to participants whose information is recorded by DOL  as having been served by the WIA Adult and Dislocated Worker  Programs using WIASRD data from program year 2011.
To describe and assess DOL’s oversight and monitoring efforts, we  reviewed technical assistance guides and material posted to  Workforce3One, including DOL’s Core Monitoring guides and Data  Validation Reporting System guidance. We also interviewed officials from  DOL’s Employment and Training Administration national office and from  all of DOL’s six regional offices. In addition, we obtained and reviewed  copies of DOL’s monitoring reports, including the results of DOL’s  program year 2010 and 2011 annual data validation efforts and the most  recent case file review for each state and territory. To analyze the results  of the annual data validation efforts, we calculated the average reported  error rate for each variable across states. We included in our analysis all  variables on characteristics and services. To analyze trends in the  results of the case file reviews, we reviewed the findings and areas of  concern identified in each review and categorized them to identify  common issues present in multiple states.
We conducted this performance audit from August 2012 through  November 2013 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit work to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
section_title: Appendix II: Data on Participants in the WIA Adult and Dislocated Worker Programs for Program Year 2011
paragraphs: We reviewed the data collected by the Department of Labor (DOL) in the  Workforce Investment Act Standardized Record Data (WIASRD) system  on the number of, characteristics of, and services provided to participants  in the WIA Adult and Dislocated Worker Programs in program year 2011.
We found the data to be sufficiently reliable for the purposes of providing  estimates of the number of, characteristics of, and services provided to,  participants whose information is recorded by DOL in WIASRD. These  estimates are presented in figures 4 through 9.
section_title: Appendix III: Comments from the Department of Labor
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Meeta Engle, Assistant Director;  Theodore Alexander; Jenn McDonald; and Brian Schwartz made key  contributions to this report. Also contributing to this report were Jessica  Botsford, David Chrisinger, Kathy Leslie, Mimi Nguyen, Carol Patey,  Rhiannon Patterson, Catherine Roark, Jerry Sandau, Walter Vance, and  Charles Youman.
section_title: Related GAO Products
paragraphs: Workforce Investment Act: Local Areas Face Challenges Helping  Employers Fill Some Types of Skilled Jobs. GAO-14-19. Washington  D.C.: December, 2013.
Workforce Investment Act: Additional Actions Would Further Improve the  Workforce System. GAO-07-105IT. Washington, D.C.: June 28, 2007.
Workforce Investment Act: Employers Found One-Stop Centers Useful in  Hiring Low-Skilled Workers; Performance Information Could Help Gauge  Employer Involvement. GAO-07-167. Washington, D.C.: December 22,  2006.
Workforce Investment Act: Labor and States Have Taken Actions to  Improve Data Quality, but Additional Steps Are Needed, GAO-06-82.  Washington, D.C.: Nov. 14, 2005.
Workforce Investment Act: Labor Should Consider Alternative  Approaches to Implement New Performance and Reporting  Requirements. GAO-05-539. Washington, D.C.: May 27, 2005.
Workforce Investment Act: Substantial Funds Are Used for Training, but  Little Is Known Nationally about Training Outcomes. GAO-05-650.  Washington, D.C.: June 29, 2005.
Workforce Investment Act: Labor Actions Can Help States Improve  Quality of Performance Outcome Data and Delivery of Youth Services.  GAO-04-308. Washington, D.C.: February 23, 2004.
Workforce Investment Act: States and Local Areas Have Developed  Strategies to Assess Performance, but Labor Could Do More to Help.  GAO-04-657. Washington, D.C.: June 1, 2004.
Workforce Investment Act: Improvements Needed in Performance  Measures to Provide a More Accurate Picture of WIA’s Effectiveness.  GAO-02-275. Washington, D.C.: February 1, 2002.