Workforce Investment Act: DOL Should Do More to Improve the Quality of Participant Data
Released: 2013-12-20T00:00:00

Letter

Background

WIA requires states and local areas to bring together a number of federally funded employment and training programs into a comprehensive workforce investment system, the American Job Center network. These programs—including the Adult and Dislocated Worker Programs—are known as mandatory partners, and must provide services through this network (see tab. 1).

The WIA Adult and Dislocated Worker Programs are designed to provide quality employment and training services to assist eligible individuals to find and qualify for employment and to help employers find the skilled workers they need. The Adult Program provides services to individuals over the age of 18 who are job seekers, although states and local areas must give priority of service to low-income individuals if funds are determined to be limited. The Dislocated Worker Program provides services to workers who have been or will be terminated or laid off from employment. For fiscal year 2013, Congress appropriated over $1.9 billion for the Adult and Dislocated Worker Programs: $730 million for the Adult Program and $1.2 billion for the Dislocated Worker Program.

DOL’s Employment and Training Administration administers the WIA Adult and Dislocated Worker Programs and oversees their implementation, which is carried out by states and local areas. Each state must have one or more designated local workforce investment area, and each local area must have at least one comprehensive American Job Center where job seekers can receive core services and access other programs and activities offered by the mandatory partners. Although each local area must have one comprehensive center, under WIA, the mandatory partners have flexibility in the way they provide services through the American Job Center network, and can co-locate services on site or make referrals to external service providers or training, including to local community colleges.

WIA provides three tiers, or levels, of service for adult and dislocated workers: (1) core, (2) intensive, and (3) training.

Core services include basic services, such as job search or résumé- building assistance, and may be accessed with or without staff assistance. Intensive services include activities such as staff-assisted comprehensive assessment of a participant’s skill levels and case management.

29 U.S.C. §§ 2831 and 2864(c)(2).

Training services include activities such as occupational skills or on- the-job training (see fig. 1).

Service at one level and a determination that a participant is unable to obtain employment through that service are prerequisites for service at the next level, although WIA does not specify the amount of time an individual must spend or the number of attempts that must be made to gain employment before moving to the next level. Job seekers who receive only core services that are self-service and informational in nature are not counted in the programs’ performance measures, but DOL requires that states count such individuals as program participants. Self- service and informational activities can be accessed either at an American Job Center or remotely, such as when a job seeker searches for employment over an internet connection to an American Job Center from a home computer.

As part of its oversight, DOL collects program data from states, which are used to assess how well the Adult and Dislocated Worker Programs are working. States must submit quarterly and annual performance reports to DOL, in addition to uploading individual records on a quarterly basis to DOL’s national WIASRD database. Specifically, states must submit quarterly and supplemental monthly performance reports, a validated annual performance report at the end of each year, and quarterly WIASRD files for each reporting quarter for a program year. The WIASRD files include demographic and characteristic information for participants in the WIA Adult and Dislocated Worker Programs and information about services received through these WIA programs, as well as through some other partner programs. The process of collecting and reporting WIA data involves all three levels of government. More specifically, participant data are typically collected by staff at American Job Centers and entered into a state or local information system. In some states, local staff may enter data directly into a statewide information system; in other states, local areas may use their own individualized information systems to enter data from which they then must extract and compile for submission to the state. After the data is submitted to the state agency, it is compiled and formatted for the various submissions to DOL, including for quarterly WIASRD record layout submissions. (see fig. 2).

DOL’s Guidance and Limitations in State Information Systems Have Resulted in Inconsistent and Incomplete National Data on WIA Participants

Flexibility in DOL’s Guidance Continues to Affect the Consistency of Data

DOL’s guidance to states, in the form of Training and Employment Guidance Letters and Training and Employment Notices, details how states should collect and report data on participants in the WIA Adult and Dislocated Worker Programs. However, the flexibility in the guidance DOL provides to states makes it difficult for DOL to provide consistent national data on participants in these programs. The flexibility in the guidance stems from the flexibility inherent in WIA, which allows states and local areas to tailor service delivery to their needs. Consequently, DOL’s guidance is designed to accommodate the different ways that states and local areas can deliver services through the American Job Center network, including decisions about whether to enroll participants in partner programs. This flexibility, however, results in variations in how states and local areas report participant data, which makes it challenging for DOL to aggregate WIA data at the national level. In particular, it has created inconsistencies among states in when a job seeker is counted as a participant in the WIA Adult or Dislocated Worker Program. In addition, DOL’s guidance is open to interpretation, allowing states to define and report some variables differently, further contributing to inconsistencies in the data that states report to DOL on these programs. For example, while DOL requires states to report the type of training service provided to WIA participants who receive such services, the agency’s guidance only lists the six broad categories of training services states must report without defining or describing them in detail. In accordance with government internal control standards, management is responsible for developing policies and procedures to achieve program objectives and clearly communicating these policies and procedures to facilitate understanding and consistent implementation.

The flexibility in DOL’s guidance on when states and local areas should count individuals as participants in the WIA Adult and Dislocated Worker Programs has resulted in inconsistencies in the data reported on program participants. WIA allows for flexibility in the extent to which Adult and Dislocated Worker Program services are integrated with those from other programs. integrate WIA services with those from other partner programs so that job seekers have access to a coordinated system of employment and training services. As a result, states’ service delivery models differ in the extent of integration between WIA services and other programs. For example, state officials that we interviewed told us that they funded core services at their American Job Centers exclusively through WIA, exclusively through a partner program, or through a blend of both WIA and partner program funds. According to officials from DOL’s national office, states and local areas are best positioned to determine the mix of services that will meet the needs of their job seekers. However, depending on which service delivery model a state or local area selects, job seekers receiving the same types of services may be counted as WIA participants at different points in time, or they may never be counted under the WIA Adult or Dislocated Worker Program.

Used in this context, integration can refer to the co-location of services at an American Job Center, or to integrated funding to provide any given service. lower populations. For example, according to the state officials we interviewed, Utah’s American Job Centers integrate WIA Adult Program funding with funding for the Wagner-Peyser Program to provide core services. As a result, every job seeker aged 18 years or older who receives core services in Utah is counted as a participant in both programs, as permitted under DOL’s guidance. In comparison, state officials in California told us that the state typically funds core services exclusively through the Wagner-Peyser Program and therefore counts all individuals accessing core services from an American Job Center as participants in the Wagner-Peyser Program, but not as participants in the WIA Adult Program or the Dislocated Worker Program. Therefore, only job seekers who meet the eligibility criteria for the WIA Adult or Dislocated Worker Program and receive intensive or training services funded by those programs are counted as WIA participants. Because of this variability, the total counts of WIA Adult Program participants and WIA Dislocated Worker Program participants represent different populations of job seekers in different states, depending on the service delivery model the state uses.

As a result of these differences, Utah ranked 3rd out of 53 states in the total number of participants it served in the WIA Adult Program in program year 2011, even though it ranked 36th in overall population.the same time, California ranked 27th in the total number of participants served in the WIA Adult Program, even though it was the most populous state (see fig. 3).

DOL Has Increased Its Oversight Efforts and Taken Steps to Improve Data Consistency, but Data Quality Issues Have Not Been Resolved DOL Performed Various Oversight Activities, but It Did Not Consistently Use the Results to Improve the Data

To improve the quality of data on participants in the WIA Adult or Dislocated Worker Programs, DOL has enhanced its oversight efforts and introduced new initiatives including having its data contractor produce quarterly reports on data issues, requiring states to validate their WIA data on an annual basis, and engaging its regional offices in periodic reviews of case files from states. However, the agency has not established a process to review the results of oversight to identify and resolve systemic issues with the quality of participant data from the WIA Adult and Dislocated Worker Programs. Our past work has shown that the benefit of collecting performance information is only fully realized when this information is actually used by managers to make decisions oriented toward improving results.

DOL has a contract with SPRA to correct and analyze WIASRD data and provide data files and reports on the accuracy of the data reported by states that are made available to the public via DOL’s website. corrections SPRA makes to the state WIASRD data files may not be accurate, and may result in incorrect data for a state.

The publicly available data files that SPRA produces are released quarterly and include “data issues reports,” which identify issues or anomalies in the WIA data submitted by each state to DOL. Officials from DOL’s national office stated that the original intent of SPRA’s analysis was to create the publicly available data files and not to report on data anomalies or errors, even though SPRA has always identified issues with the data while creating the publicly available files. However, since about 2010, when states started submitting quarterly WIASRD files instead of annual ones, reviewing the quality of the data and issuing the quarterly “data issues and anomalies report” has become standard practice under SPRA’s contract, according to officials from DOL’s national office. The agency’s regional offices are supposed to provide the states with the published quarterly error reports and ask them to update any errors prior to their next quarterly WIASRD submission.

DOL publishes SPRA’s data files and reports on its website, and its regional offices are expected to share these reports with their states so that the states can correct any data issues in their subsequent quarterly submissions. However, officials from DOL’s Region 5 said that although they receive these reports from DOL’s national office, they have not had the chance to review them or comment on any of the errors for states in their region. In addition, officials from DOL’s Region 1 said that although these reports have always been available on DOL’s website, they only recently began receiving the reports in a user friendly format and that they only recently began providing feedback to SPRA on issues identified for the states in their region. Our analysis of SPRA’s data issues reports suggests that some states may not be using SPRA’s reports to improve the quality of their data on WIA participants, which could be due to a lack of awareness of these reports. For example, SPRA’s data issue reports for the fourth quarters of both program years 2010 and 2011 identified some of the same issues, such as errors in the dates of service reported by the states. This suggests that the states that made these errors in 2010 may not have reviewed these reports and used them to correct the data reported in the following year. In addition, while officials from four states were aware of the SPRA reports, officials from four other states told us they either were not aware of these reports or that they began receiving these reports from the regions only recently, beginning with the final report for program year 2011. When asked, officials from DOL’s national office stated that they do not have a plan to systematically identify or address recurring errors noted in SPRA’s reports. Government internal control standards note that, for oversight and monitoring to be effective, information should be recorded and communicated to management and others within the entity and external stakeholders, and this should be done within a time frame that enables them to carry out their internal control and other responsibilities.

Officials added that they do not conduct formal oversight reviews or audits of SPRA’s data analyses because they consider SPRA to be the “data experts” and, therefore, do not know what kind of oversight they could provide. According to government standards for internal controls, agencies should ensure that ongoing monitoring occurs in the course of normal operations, which would include monitoring and oversight of contractors through regular management and supervisory activities. These officials also said they believe any major data issues or obstacles would be uncovered by their internal data edit checks, which are run on all WIASARD data submitted by the states prior to SPRA’s review of the data.

GAO/AIMD-00-21.3.1. correcting the data. For example, officials from California explained that SPRA changed some of the numbers entered by local areas for WIASRD variable 325—Employment and Training Programs Related to Food Stamps—to zeros because the numbers entered seemed too high. According to the state officials, the numbers for that variable entered by the local areas were correct. States, however, are generally not provided an opportunity to review and verify SPRA’s changes before they are made, as they only receive copies of the data issues reports after they are published. Officials from both DOL’s national office and from some of the state workforce agencies noted that not all issues identified by SPRA represent actual errors and that some outliers on certain variables are acceptable.

DOL requires each state to validate the data it collects and report on participants in the WIA Adult and Dislocated Worker Programs on an annual basis, but the findings from these validation efforts have not been strategically used to identify systemic issues with or to improve the quality of the data on WIA participants. Government standards for internal controls state that for oversight and monitoring to be effective, information should be communicated to management and others, along with the use of this information for program assessment, so that it can be used to carry out their internal control and other responsibilities.

Each year, states are required to review a sample of WIA participant records to determine whether the source documents match the information in the electronic records states use to collect and report data to DOL in WIASRD. DOL requires states to validate the accuracy of the data they submit annually to ensure that decisions about WIA policy and funding are made based on a true picture of the number of participants and program outcomes. Although DOL has established a provisional 5 percent error rate threshold for states’ validation of the variables, the agency does not have plans to tie the results of these validation efforts to DOL’s financial awards or penalties because, according to officials from DOL’s national office, the results of states’ validation efforts were never intended to be used for enforcement purposes. Officials from DOL’s national office explained that, in their opinion, DOL’s regional offices should be using the states’ validation efforts as a management tool to improve the quality of their data by identifying inaccurate or confusing variables to target the technical assistance they provide to states and local areas.

DOL, however, does not know what effect state data validation efforts have had on the quality of participant data for the WIA Adult and Dislocated Worker Programs. In addition, our interviews with regional and state officials suggest that DOL’s regions and the states are not always using the results of these data validation efforts to improve data quality or target technical assistance. During our interviews with regional and state officials, only those from Region 3 and Massachusetts described specific efforts to use the results of the state data validation efforts to improve data quality and direct technical assistance. Officials from DOL Region 3 noted that the region recently began requiring states to respond to findings from states’ annual data validation efforts and to track error rates found in each quarterly submission. Similarly, officials from Massachusetts said that they use the results of their data validation efforts to direct the technical assistance the state provides to local areas and to improve the quality of the state’s WIA data. In contrast, officials from one state said that, although they receive information about errors associated with specific case files as they enter data from each file into DOL’s data validation system, they do not know how to retrieve their state-wide results from DOL’s database and they have not received any reports from DOL documenting the nationwide results of the data validation efforts. Officials from DOL’s national office said that they were surprised by this, and that data element validation results are available to states through DOL’s reporting system.

Moreover, our analyses of the results of DOL’s efforts to validate WIASRD variables for program years 2010 and 2011 suggest that DOL’s data validation efforts have not prevented high error rates on certain data elements for select variables—nationally, error rates for certain variables have remained well above the 5 percent threshold over both program years. For example, in program year 2010, across all states, about 16 percent of the files for the Adult Program for which the “date of program exit” variable was reviewed had errors, compared to around 14 percent in program year 2011. Similarly, the nationwide error rate for “date of first staff-assisted core service” was above 7 percent in both program years for both the Adult and the Dislocated Worker Programs. When asked, officials from DOL’s national office and two regional offices explained that variables containing dates frequently have high error rates due to discrepancies between the date reported and the date in the source documentation. If a date, such as date of dislocation—the date a worker lost his or her job—in the hard copy document differs from the date in the electronic record, even by one day, the variable for that record “fails” the validation check. Moreover, officials from DOL’s national office and three regional offices stated that high error rates resulting from such discrepancies do not necessarily reflect any serious issues with the reported data—a participant would still be a dislocated worker whether, for example, the date of dislocation was June 20th or June 21st. Nonetheless, DOL requires these dates to match precisely and errors noted in program year 2011 for these variables were still prevalent although similar errors were noted the previous year.

DOL’s required annual data validation efforts are resource-intensive and time-consuming both for DOL regions and states, according to officials from DOL’s national office, two DOL regional offices, and five states. However, DOL has not yet evaluated the process or determined its effect on data quality. Specifically, an official from one state workforce agency estimated the cost of its annual data validation efforts, including staff time, travel, and other expenses, to be about $200,000. Officials in another state explained that they would like DOL to reduce the required sample size for the required validation of the data elements in WIASRD in order to reduce the administrative burden on states. In addition, in 2011 one DOL regional office convened a workgroup of representatives from four states that analyzed the data validation procedures and provided recommendations to DOL’s national office for improvements to reduce the administrative burden on states. These recommendations included considering using alternative sampling methods, revisiting the frequency and precision requirements of data validation, and issuing guidance to share “best practices” across states and local areas. However, as of September 2013, DOL had not implemented the workgroup’s recommendations, and the data validation process remains unchanged. Officials from DOL’s national office acknowledged the trade-off between monitoring data quality and minimizing the administrative burden on states. They said that revising WIASRD’s edit checks to allow states more flexibility may result in making the data validation process more efficient by, for example, permitting states to report a range of dates, if appropriate, for certain variables.

DOL’s regional offices review a sample of case files from states as part of their oversight of the quality of data for the WIA Adult and Dislocated Worker Programs, but they have not used the results of these reviews to identify systemic issues with the quality of the data on WIA participants. According to officials from DOL’s national office, its regional offices are responsible for providing feedback to states based on these reviews, and it is not the national office’s role to conduct any type of systemic review to identify cross-state data issues. Government standards for internal controls state that for oversight and monitoring to be effective, information should be communicated to management and others, along with the use of this information for program assessment, so that it can be used to carry out their internal control and other responsibilities.

In response to a recommendation from a prior GAO report, DOL began to require its regional offices to review a sample of case files to monitor states’ annual data validation procedures. In addition, over the past few years, to address concerns about the nationwide consistency of monitoring activities, DOL has issued additional guidance to its regional offices on the process that should be used in reviewing the case files.

Officials from all six of DOL’s regional offices reported using these materials when they design and conduct their reviews of the case files. The review process begins when officials from DOL’s regions review a state’s most recent data validation report and identify a subsample of participant case files from the most recent review by the state. In addition to checking the data in the electronic records by comparing them to the source documentation, DOL’s regional staff assess whether the state followed the proper procedures in conducting its annual data validation efforts, according to officials from DOL’s national office. The process concludes with a report outlining the DOL regional office’s findings, including non-compliance with statutory or regulatory requirements for collecting and reporting data on WIA participants. Many of the reports also identify areas of concern, such as when states do not share their annual data validation results with local area staff from American Job Center partner programs, and identify promising practices observed during the reviews. State officials have 45 days to respond to findings in the region’s report and are also encouraged but are not required to respond to areas of concern detailed in the report. In addition, officials from DOL’s national office noted that states with high error rates on select variables are encouraged to inform the regional offices of how they plan to reduce their error rates in the future.

Officials from DOL’s national office said that, while they discuss the results of these reviews with regional officials and provide state-specific technical assistance as needed, they do not have a regular, formal process for analyzing the findings from these reviews by their regional offices, including determining whether similar findings and areas of concern were identified across states. DOL officials explained that, because the reviews are part of the regional offices’ oversight of the states, they believe that the national office should not be involved in monitoring the results of the reviews or the way in which they are conducted. As a result, DOL does not have a systematic means of determining the importance of the findings, their prevalence, or their likely effect on the quality of the national data on participants in the WIA Adult and Dislocated Worker Programs. This limits DOL’s ability to respond to data issues that are systemic or widespread. Table 2 summarizes our analysis of the most prevalent issues identified during the most recent reviews for each of the 53 states and territories.

Over the past few years, DOL has issued additional guidance and provided technical assistance to states and local areas, including training and webinars, to clarify and explain the requirements for collecting and reporting data for the WIA Adult and Dislocated Worker Programs. However, some state officials said that DOL’s technical assistance is not always timely, and that DOL could do more to facilitate the collection or sharing of promising data collection and reporting practices across regions and states.

For example, DOL has provided general technical assistance on data reporting for the WIA Adult and Dislocated Worker Programs to states, and officials from three of the eight states said that the assistance provided by DOL’s regional offices was useful in helping them address some of the challenges related to data reporting. In particular, officials from DOL’s national office said that some regional offices issue quarterly performance letters to states that include program year performance data and any related analysis, in addition to hosting quarterly phone conferences with state performance specialists to discuss performance issues. DOL also sponsors the Workforce3one website, which contains a variety of training and background materials related to data collection and reporting for WIASRD. National and regional DOL data specialists also said they hold biweekly meetings to discuss data issues. In addition, DOL officials from each region described regular communication they have with state officials to provide technical assistance in response to specific data reporting issues, and officials from five of the six regional offices described conference calls that they have hosted as opportunities for states to discuss challenges related to the quality of their data on WIA participants. Furthermore, DOL’s national and regional offices have access to an internal data system, Infospace, which allows them to retrieve and review publicly available WIASRD data by state and local area.

Over the past few years, DOL has also issued a number of Training and Employment Guidance Letters and Training and Employment Notices related to data collection and reporting for WIA. In 2011, DOL developed and hosted a series of webinars for states and local areas, including one on data validation and data quality issues. The materials from this session describe the approach DOL uses to monitor the data and explore issues and findings across states. They also highlight how data validation can be used to improve data systems and provide information on the guidance regional offices use to review states’ case files. While the webinar materials present information on some of the consistent findings across states—such as incorrect source documentation and exit dates— DOL officials said that they have not provided any additional formal technical assistance to address these issues on a national level because of resource limitations. Moreover, they said it would be difficult for DOL to provide such assistance without having first reviewed the results of its own monitoring efforts. DOL’s webinar also noted that DOL would establish a working group in the summer of 2011 to look into and possibly revise the source documentation requirements for the annual data validation process. However, as of the summer of 2013, DOL officials said that this working group had not been established due to competing priorities and resource constraints.

Officials from two states noted that DOL’s updates to its guidance for collecting and reporting the data are often not provided far enough in advance to be implemented by the time changes take effect. For example, officials from California said that they often do not have enough lead time to properly implement new data elements or guidance when it is issued by DOL. According to state officials, the state generally has 1 month or less to change its automated system to meet deadlines, which is not enough time. In addition, officials from Georgia said the data validation reports they currently receive from DOL are not timely since they are at least 3 months old by the time the state receives the reports. Some state officials also told us that they would benefit from learning how their peers are addressing challenges in reporting data on the WIA programs. Officials from DOL’s national office, however, told us they do not currently facilitate the collection or sharing of promising data collection and reporting practices across regions or states, and have no plans to do so—because of competing resources, the agency’s main focus is on service delivery rather than data collection. Best practices state that high- performing organizations continually assess performance and efforts to improve performance. In particular, managers can use performance information to identify and share more effective processes and approaches.

DOL Has Taken Steps to Improve Data Consistency, but It Has Not Evaluated These Efforts

In addition to its regular monitoring activities, DOL has taken specific steps to improve the consistency of the WIA data collected by states and local areas. For example, the agency has developed an integrated data reporting system, which is being piloted in two states. However, DOL has not evaluated the results of the pilot program to determine whether it has had a positive effect on the quality of participant data for the WIA Adult and Dislocated Worker Programs and, despite not having evaluated its effectiveness, plans to expand the program to additional states.

To standardize and streamline reporting across several of DOL’s workforce programs—the Wagner-Peyser Program, the WIA Adult, Dislocated Worker and Youth Programs, Veterans Employment and Training Service, National Emergency Grants, and Trade Adjustment Assistance Programs—the agency has developed an integrated data reporting system, WISPR. Two states, Pennsylvania and Texas, have been piloting WISPR since 2007 to collect and report data on the WIA Adult and Dislocated Worker Programs, and a third state—Utah—plans to start piloting WISPR in the fall of 2013. Officials from DOL’s national office and from Utah explained that one of WISPR’s key advantages over the current separate reporting systems for each program is that WISPR has standardized variables that include all the required variables for each program. improvement over their current system because future changes in DOL’s guidance for any of the workforce programs it administers—including the WIA Adult and Dislocated Worker Programs—would be incorporated into a single system, facilitating implementation of these changes.

Although DOL’s guidance for reporting data in WIASRD encourages states and local areas to provide integrated services through multiple programs, each program has its own reporting requirements, according to officials from DOL’s national office. As a result, it is not possible to track individual job seekers who receive services from multiple programs across the workforce system, or to determine the proportion of resources provided by each program for a particular service, or to attribute participant outcomes to those programs. the current record layout of WIASRD to match that of WISPR. DOL officials told us they expect to implement the revised WIASRD record layout in the fall of 2013. However, as of August 2013, they said it is not clear whether or when nationwide implementation of WISPR will occur because this depends on the resources available to upgrade both federal and state information systems and the associated programming costs. While WISPR appears to offer advantages over the current reporting system that might make it a promising step forward, DOL does not currently have plans to evaluate the results of the pilot program to determine whether it has had a positive effect on the completeness and consistency of participant data for the WIA Adult and Dislocated Worker Programs before expanding the program to other states. DOL officials cited the agency’s limited resources as the reason for not planning an evaluation of the WISPR pilot program before expanding it to other states. However, best practices note that evaluation can play a key role in program planning, management, and oversight by providing feedback to program managers, legislative and executive branch policy officials, and the public. Further, when pilot programs are designed to produce change—such as by allowing for more streamlined data collection and reporting—assessing the impact is essential for knowing if the pilot is meeting its goals. Without an evaluation of WISPR, DOL will not know if this data system has resulted in the collection of more accurate WIA participant data when compared to WIASRD.

Finally, DOL administers two grant programs that states can use to improve their WIA information systems: the Workforce Innovation Fund and the Workforce Data Quality Initiative.Fund is a competitive grant program that supports innovative approaches to the design and delivery of employment and training services. Although it is not targeted specifically at information systems, at least 3 of the 26 grants awarded by DOL have been used for local initiatives to integrate their workforce data systems. For example, one local area we visited in Chicago received a Workforce Innovation Fund grant that they plan to use to integrate the different data systems used by the workforce programs in their local area. Officials said that they hope the improvements will result in more data-driven decisions about service delivery, and that all employment and training programs in their local area will be able to share information electronically. Another grant program, DOL’s Workforce Data Quality Initiative, is also not specifically aimed at WIA data reporting but may have positive incidental effects on WIA data quality, according to DOL officials. The purpose of this initiative is to create a longitudinal database to chart individuals’ progress through the education system and beyond to the labor market. This effort would entail upgrades to state information systems that may resolve some data reporting issues currently attributed to limited technological capacities. At this time, it is too early to know whether these grants will have a positive effect on the quality of WIA participant data.

Conclusions

Collecting and reporting consistent and complete data is important for program oversight and management and to evaluate the effectiveness of program activities and services, but it can be difficult when federal programs are carried out in partnership with states and local areas. DOL has taken steps to improve the quality of the data on WIA’s Adult and Dislocated Worker Programs. However, the flexibilities in DOL’s guidance, reflective of those inherent in the programs’ authorizing statute, which allows states flexibility in program design, along with limitations in state information systems, present challenges to DOL in collecting and reporting consistent and complete data on a unique count nationwide of participants in the WIA Adult and Dislocated Worker Programs. Without such data, policymakers, program officials, and other stakeholders have an incomplete picture of the number of adults and dislocated workers served, their characteristics, and the type and level of services received.

In addition, while DOL engages in several types of oversight activities designed to ensure the accuracy of states’ data on participants in the WIA Adult and Dislocated Worker Programs, it does not consistently share the results of its oversight activities with states and local areas. As a result, states and local areas are not always aware of potential data quality issues, and may miss opportunities to improve their data collection and reporting. Moreover, since 2007, two states have been piloting a new information system that tracks program participants across several of DOL’s employment and training programs, but DOL does not plan to evaluate its effects on the quality of the data collected on participants in the WIA Adult or Dislocated Worker or other programs before it expands the system to other states. Similarly, DOL does not regularly collect and disseminate promising practices to states and local areas, which could facilitate the adoption of steps other states and local areas have taken to improve their data collection and reporting efforts. While it may not be possible to achieve 100 percent precision and accuracy in the data reported on participants in a large, complex system like the workforce investment system, by not appropriately targeting their available resources and facilitating sharing of promising practices among states to continuously try to improve the quality of the data, DOL misses an opportunity to identify and address longstanding, systemic issues.

Recommendations for Executive Action

1. To improve the consistency and completeness of national data on participants in the WIA Adult and Dislocated Worker Programs, we recommend that the Secretary of Labor take additional steps to improve the uniformity of participant data reported by states. These steps could include the following: a. providing additional guidance to states on data reporting, such as how core and intensive services should be recorded for WIA participants who receive these services through partner programs; and b. conducting an evaluation or review of WISPR to determine if it has resulted in more complete and consistent data collection and reporting for participants in the WIA Adult and Dislocated Worker Programs and placing a high priority on the implementation of WISPR if it is shown to improve data consistency and completeness. 2. We also recommend that the Secretary of Labor promote a formal, continuous process for improving the quality of data on participants in the WIA Adult and Dislocated Worker Programs through such measures as the following: a. consistently sharing the results of all oversight activities with states and local areas, including findings from validation of participant data; b. reviewing the methods used for data validation, such as its scope and error rate threshold, to identify opportunities to increase efficiencies and accountability in the process. This could include implementing, if appropriate, recommendations from the Regions’ review of data validation procedures; c. evaluating data validation efforts to determine their effects on data quality, particularly on systemic errors, and providing targeted guidance and assistance to states and local areas to address such errors; d. regularly monitoring Social Policy Research Associates’ corrections and analyses of state WIA participant data, sharing this information with states, and coordinating with states to ensure that any corrections are appropriate and accurate; and e. collecting and disseminating promising practices to states and local areas on data collection and reporting on a regular basis.

Agency Comments and Our Evaluation

We provided a draft of this report to officials at DOL for their review and comment. We received written comments from DOL, which are reproduced in their entirety in appendix III. DOL officials did not state whether they agreed or disagreed with our recommendations. These officials acknowledged the importance of having reliable data to effectively manage and evaluate the WIA Adult and Dislocated Worker Programs; however, they commented that data reliability should be balanced with the flexibility WIA gives to states and with DOL’s responsibility to prioritize use of its limited resources. They stated that WIA provides states and local areas with the flexibility to serve their customers in the way that best suits their particular needs. DOL officials also stated that the agency has invested significant resources in its workforce performance accountability system, especially for WIA programs. According to officials, the agency has a robust system in place to ensure data quality and reliability and has recently made several enhancements to the reporting system. In their comments, DOL officials detailed various efforts they plan to take to address our recommendations. Nonetheless, we believe that these efforts will not sufficiently address the specific data quality issues we identified and encourage DOL to take more targeted steps as outlined in our recommendations.

In response to our first recommendation, DOL officials said that they believe the agency’s current guidance is clear but that they will continue to work with states to develop additional guidance, as necessary, such as forthcoming guidance on how to avoid duplication of services when co- enrolling participants across multiple partner programs. However, it is important that any additional guidance also specify when to count job seekers as WIA participants if they also receive services funded by partner programs. We also encourage DOL to develop additional guidance for the WIASRD variables noted in our report that are open to interpretation, such as “type of training,” to facilitate consistent reporting on participants in these programs. DOL officials also noted that an evaluation of WISPR is subject to the agency’s resource constraints, adding that the purpose of WISPR was never explicitly to improve data quality. As we stated in our report, however, WISPR seeks, in part, to improve the consistency of WIA data by standardizing reporting across the workforce system. As such, it has the potential to improve data quality. Therefore, we encourage DOL to evaluate the system in order to make an informed decision on how best to allocate finite agency resources going forward.

In response to our second recommendation, DOL officials stated that they consistently share the results of the agency’s oversight activities with states but acknowledged that more could be done to analyze the results of its activities to identify and share similar findings and areas of concern across multiple states. DOL officials added that they will work with the regional offices towards this goal. With regard to the agency’s data validation methods, officials said they regularly review these methods and solicit input from states on how to improve them. Specifically, DOL pointed out that, as required by the Paperwork Reduction Act, the Office of Management and Budget reviews DOL’s data validation process every 3 years and solicits public comment before approving the methodology and authorizing data collection, and that their 2014 submission will reflect state input. However, given the time-consuming nature of data validation, we believe the agency should take additional actions to review its current methods specifically with an eye toward making them more efficient and holding states accountable for their data validation results.

With regard to evaluating the effectiveness of its data validation efforts, DOL officials said that the agency plans to consider the regional data validation workgroup’s findings and recommendations from 2011, explore ways to streamline the process, and examine the effect of data validation on error rates. We commend DOL’s plans, but to adequately address the persistently high data error rates we found in our analysis, we believe it is necessary to go beyond evaluating the effectiveness of its data validation efforts and pinpoint the underlying cause of the errors so that they can be addressed.

Officials also pointed out that the agency already monitors and shares the analyses of state data conducted by its contractor, Social Policy Research Associates (SPRA). They stated that SPRA’s corrections of states’ data have been publicly available with the data set from the inception of WIA. They also noted that since program year 2011 DOL has provided SPRA’s analyses and corrections to the states (through ETA Regional Offices) on a quarterly basis for states to either correct or dispute. They noted that this is a formal and recurring process, and that ETA Regional Offices have begun to analyze state date WIASRD data on a regular basis as part of their annual review cycles. However, we found that not all states are aware of or receive copies of SPRA’s reports, and that some of the corrections SPRA makes to the state WIASRD data files may not be accurate. Furthermore, as noted in our report, DOL officials told us that they do not conduct formal oversight reviews or audits of SPRA’s data analyses.

In addition, DOL officials reiterated that data collection and reporting are topics that are included in workforce3one, a point we noted in our report. However, we maintain that DOL could do more to facilitate the sharing of information across states, such as creating a forum through which states could learn how their peers are addressing challenges in data reporting for participants in the WIA programs. Finally, DOL provided technical comments, which we incorporated, as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Labor, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-7215 or moranr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV.

Appendix I: Scope and Methodology

Our objectives were to determine: (1) what factors have affected the ability to report consistent and complete data on participants in the Workforce Investment Act (WIA) Adult and Dislocated Worker Programs, and (2) what actions has the Department of Labor (DOL) taken to improve the quality of participant data. To address our objectives, we reviewed applicable laws and regulations, as well as DOL’s guidance to states for collecting and reporting data on participants in the WIA Adult and Dislocated Worker Programs. We also interviewed officials from DOL’s Employment and Training Administration, its Office of Inspector General, and its six regional offices. In addition, we visited or telephoned a nongeneralizable sample of eight states. Within each state, we visited or contacted at least one American Job Center—formerly known as a one- stop center—or a local workforce board. We also assessed the reliability of program year 2011 data from the Workforce Investment Act Standardized Record Data (WIASRD) system by testing the data electronically and interviewing knowledgeable agency officials and DOL’s data contractor. We found the data in appendix II to be sufficiently reliable for the purposes of providing estimates of the number of, characteristics of, and services provided to participants whose information is recorded by DOL as having received services from either the WIA Adult Program or the WIA Dislocated Worker Program. The data are not reliable for other purposes, such as making state-to-state comparisons, because of variations in how states collect and report data on participants in the WIA Adult and Dislocated Worker Programs.

We conducted these interviews between September 2012 and June 2013. with DOL officials in Regions 1 (Boston), 3 (Atlanta), and 5 (Chicago); state and local workforce officials in California, Georgia, Illinois, Massachusetts, and Washington; and American Job Center officials in Maryland. We conducted telephone interviews with DOL officials in Regions 2 (Philadelphia), 4 (Dallas), and 6 (Sacramento), and with state workforce officials in Maryland, South Dakota, and Utah. We nonstatistically selected these states to provide diversity on the basis of: (1) geographic location, (2) total federal spending on the Adult and Dislocated Worker Programs in program year 2010, (3) the extent of data issues identified in the fourth quarter of program year 2010, (4) whether the state reported participants who only received core self-services, and (5) the number of local areas within the state.

In each state, we obtained general information about the state’s and the local area’s implementation of the WIA Adult and Dislocated Worker Programs and on any challenges they may have encountered in collecting and reporting data on program participants. We also asked about actions DOL has taken to improve the quality of the data on participants in the WIA Adult and Dislocated Worker Programs. We used semi-structured interviews for our regional, state, and local interviews. Because we interviewed officials from a nongeneralizable sample of eight states and selected local areas, we cannot generalize our findings beyond the data collected on those states and local areas.

To assess the reliability of DOL’s data in the WIASRD database for participants in the WIA Adult and Dislocated Worker Programs, we (1) reviewed existing documentation related to the data sources, including reports issued by DOL’s Office of Inspector General; (2) electronically tested the WIASRD data to identify potential problems with consistency, completeness, or accuracy; and (3) interviewed DOL’s data contractor and knowledgeable agency officials to obtain information about the data.

Our electronic testing consisted of identifying inconsistencies, outliers, missing values, and other errors. More specifically, the electronic testing included assessing the reliability of data collected on the characteristics and the services participants in the WIA Adult and Dislocated Worker Programs received in program year 2011. Prior to testing the data, we combined 160 records that were overlapping or duplicative into 80 unique records and removed 19 records that had missing or erroneous participation dates. A few variables, including data on a participant’s dislocation date—the date a worker lost his or her job, and the occupational codes for participants who completed training, were found to not be sufficiently reliable for our purposes and were not included in our report. In addition, we analyzed the publicly available WIASRD data file for program year 2011, which was produced for DOL by its data contractor, Social Policy Research Associates. As part of our analysis, we reviewed the steps the data contractor took to correct the data and, to the extent possible, compared our data to the publicly available file. In light of variations in how states collect and report participant data for the WIA Adult and Dislocated Worker Programs and limitations in their information systems, the actual number of participants in these programs is unknown.

However, we were able to estimate the number of, characteristics of, and services provided to participants whose information is recorded by DOL as having been served by the WIA Adult and Dislocated Worker Programs using WIASRD data from program year 2011.

To describe and assess DOL’s oversight and monitoring efforts, we reviewed technical assistance guides and material posted to Workforce3One, including DOL’s Core Monitoring guides and Data Validation Reporting System guidance. We also interviewed officials from DOL’s Employment and Training Administration national office and from all of DOL’s six regional offices. In addition, we obtained and reviewed copies of DOL’s monitoring reports, including the results of DOL’s program year 2010 and 2011 annual data validation efforts and the most recent case file review for each state and territory. To analyze the results of the annual data validation efforts, we calculated the average reported error rate for each variable across states. We included in our analysis all variables on characteristics and services. To analyze trends in the results of the case file reviews, we reviewed the findings and areas of concern identified in each review and categorized them to identify common issues present in multiple states.

We conducted this performance audit from August 2012 through November 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit work to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Data on Participants in the WIA Adult and Dislocated Worker Programs for Program Year 2011

We reviewed the data collected by the Department of Labor (DOL) in the Workforce Investment Act Standardized Record Data (WIASRD) system on the number of, characteristics of, and services provided to participants in the WIA Adult and Dislocated Worker Programs in program year 2011.

We found the data to be sufficiently reliable for the purposes of providing estimates of the number of, characteristics of, and services provided to, participants whose information is recorded by DOL in WIASRD. These estimates are presented in figures 4 through 9.

Appendix III: Comments from the Department of Labor

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Meeta Engle, Assistant Director; Theodore Alexander; Jenn McDonald; and Brian Schwartz made key contributions to this report. Also contributing to this report were Jessica Botsford, David Chrisinger, Kathy Leslie, Mimi Nguyen, Carol Patey, Rhiannon Patterson, Catherine Roark, Jerry Sandau, Walter Vance, and Charles Youman.

Related GAO Products

Workforce Investment Act: Local Areas Face Challenges Helping Employers Fill Some Types of Skilled Jobs. GAO-14-19. Washington D.C.: December, 2013.

Workforce Investment Act: Additional Actions Would Further Improve the Workforce System. GAO-07-105IT. Washington, D.C.: June 28, 2007.

Workforce Investment Act: Employers Found One-Stop Centers Useful in Hiring Low-Skilled Workers; Performance Information Could Help Gauge Employer Involvement. GAO-07-167. Washington, D.C.: December 22, 2006.

Workforce Investment Act: Labor and States Have Taken Actions to Improve Data Quality, but Additional Steps Are Needed, GAO-06-82. Washington, D.C.: Nov. 14, 2005.

Workforce Investment Act: Labor Should Consider Alternative Approaches to Implement New Performance and Reporting Requirements. GAO-05-539. Washington, D.C.: May 27, 2005.

Workforce Investment Act: Substantial Funds Are Used for Training, but Little Is Known Nationally about Training Outcomes. GAO-05-650. Washington, D.C.: June 29, 2005.

Workforce Investment Act: Labor Actions Can Help States Improve Quality of Performance Outcome Data and Delivery of Youth Services. GAO-04-308. Washington, D.C.: February 23, 2004.

Workforce Investment Act: States and Local Areas Have Developed Strategies to Assess Performance, but Labor Could Do More to Help. GAO-04-657. Washington, D.C.: June 1, 2004.

Workforce Investment Act: Improvements Needed in Performance Measures to Provide a More Accurate Picture of WIA’s Effectiveness. GAO-02-275. Washington, D.C.: February 1, 2002.