Trade Adjustment Assistance: Labor Should Take Action to Ensure Performance Data Are Complete, Accurate and Accessible
Released: 2006-05-25T00:00:00

Letter

Background

To assist workers who are laid off as a result of international trade, Congress passed the Trade Expansion Act of 1962 and created the Trade Adjustment Assistance program. Historically, the main benefits available through the program have been extended income support and training. Participants are generally entitled to income support, but the amount of funds available for training is limited by federal statute. Labor certifies groups of laid-off workers as potentially eligible for TAA benefits and services by investigating petitions that are filed on the workers’ behalf. Workers are eligible for TAA if they were laid off as a result of international trade and were involved in making a product or supplying component parts to or performing finishing work for directly affected firms. Workers served by the TAA program have generally been laid off from the manufacturing sector.

Congress has amended the TAA program a number of times since its inception. For example, in 1974 Congress eased program eligibility requirements, and in 1988 Congress added a requirement that workers be in training to receive income support. In 1993 Congress created a separate North American Free Trade Agreement Transitional Adjustment Assistance (NAFTA-TAA) program specifically for workers laid off because of trade with Canada or Mexico. The most recent amendments to the TAA program were included in the TAA Reform Act of 2002 (Pub. L. No. 107-210), which was signed into law in August 2002. The Reform Act consolidated the former TAA and NAFTA-TAA programs into a single TAA program, doubled the amount of funds available for training annually, expanded program eligibility to more workers, extended the time periods covered by the program, and added new benefits.

Services Available under the TAA Program

Under the current TAA program, eligible participants have access to a wider range of benefits and services than before, including: Training. Participants may receive up to 130 weeks of training, including 104 weeks of vocational training and 26 weeks of remedial training (e.g., English as a second language or literacy). On-the-job training is also available under TAA. Participants in TAA-approved training must attend training full-time.

Extended income support, or Trade Readjustment Allowances (TRA). Participants may receive up to 104 weeks of extended income support benefits after they exhaust the 26 weeks of UI benefits available in most states. This total includes 78 weeks while participants are completing vocational training and an additional 26 weeks, if necessary, while participants are completing remedial training. The amount of extended income support payments in a state is set by statute at the state’s UI benefit level.

During their first 26 weeks of extended income support, participants must be enrolled in training, have completed training, or have a waiver from this requirement; to qualify for more than 26 weeks of extended income support, participants must be enrolled in training. The TAA statute lists six reasons why a TAA participant may receive a waiver from the training requirement, including that the worker possesses marketable skills or that the approved training program is not immediately available. States must review participants’ waivers at least every 30 days and if necessary may continue to renew participants’ waivers each month throughout the initial 26 weeks of extended income support.

Job search and relocation benefits. Payments are available to help participants search for a job in a different geographical area and to relocate to a different area to take a job. Participants may receive up to a maximum of $1,250 to conduct a job search. The maximum relocation benefit includes 90 percent of the participant’s relocation expenses plus a lump sum payment of up to $1,250.

Health Coverage Tax Credit (HCTC). Eligible participants may receive a tax credit covering 65 percent of their health insurance premiums for certain health insurance plans. To be eligible for the credit, trade-affected workers must be either receiving extended income support payments, or they must be eligible for extended income support but are still receiving UI payments, or they must be recipients of benefits under the wage insurance program. As a result, trade-affected workers who are still receiving UI rather than extended income support may register for the HCTC only if they are in training, have completed training, or have a waiver from the training requirement.

Wage insurance. The wage insurance program—known as the Alternative TAA (ATAA) program—is a demonstration project designed for workers age 50 and older who forgo training, obtain reemployment within 26 weeks, but take a pay cut. Provided the participant’s annual earnings at his or her new job are $50,000 or less, the benefit reimburses 50 percent of the difference between the participant’s pre- and postlayoff earnings up to a maximum of $10,000 over 2 years.

Certification Process and Eligibility Requirements

The process of enrolling trade-affected workers in the TAA program begins when a petition for TAA assistance is filed with Labor on behalf of a group of laid-off workers. Petitions may be filed by entities including the employer experiencing the layoff, a group of at least three affected workers, a union, or the state or local workforce agency. The TAA statute lays out certain basic requirements that all certified petitions must meet, including that a significant proportion of workers employed by a company be laid off or threatened with layoff. In addition to meeting these basic requirements, a petition must demonstrate that the layoff is related to international trade. The law requires Labor to complete its investigation, and either certify or deny the petition, within 40 days after it has received it. When Labor has certified a petition, it notifies the relevant state, which has responsibility for contacting the workers covered by the petition, informing them of the benefits available to them, and telling them when and where to apply for benefits.

Workers generally receive services through a consolidated service delivery structure called the one-stop system, where they can access a broad range of services beyond TAA, including the Workforce Investment Act (WIA) Dislocated Worker program, the Wagner-Peyser Employment Service (ES) program, and services funded by the WIA National Emergency Grants. Training for trade-affected workers may be funded by TAA or by one of the WIA funding sources. Workers often meet one on one with a case manager who may assess worker’s skills and help decide what services they need. Because the TAA program has limited funds that can be used for case management and program administration, these case management services are often performed by ES or WIA Dislocated Worker program staff. When this occurs, participants are often co-enrolled in WIA or ES as well as TAA.

TAA Funding

About $750 million was appropriated for income support for trade-affected workers for fiscal year 2005, while another $259 million was appropriated for training, job search and relocation allowances, and administrative costs. Of the $259 million, $220 million is set aside for training, and Labor allocates 75 percent of it to states according to a formula that takes into account each state’s previous year allocations, accrued expenditures, and participant levels. Labor holds the remaining 25 percent of training funds in reserve, to distribute to states throughout the year according to need. To cover administrative costs associated with training under the TAA program, Labor allocates additional administrative funds to each state equal to 15 percent of its training allocation.

TAA Performance Reporting System

Labor is responsible for monitoring the performance of the TAA program. In fiscal year 1999, Labor introduced a new participant outcomes reporting system, the Trade Act Participant Report (TAPR), that was designed to collect national information on TAA program participants, services, and outcomes. States are required to submit TAPR reports to Labor each quarter, with data on individuals who exited the TAA program. The TAPR data submitted by states are used to calculate national and state outcomes on the TAA performance measures for each fiscal year, which include (1) the percentage of participants that found jobs after exiting the program (reemployment rate), (2) the percentage of those participants who were employed after exiting the program who were still employed 9 months later (retention rate), and (3) the earnings in their new jobs compared to prior earnings (wage replacement rate).

Labor’s guidance requires states to include in their TAPR submissions all TAA participants who exit the program, that is, stop receiving benefits or services. Under Labor’s guidance, a participant is defined as any individual who receives any TAA benefit or service, including extended income support payments, training, or job search and relocation allowances. According to this definition, participants would include those who, for example, received only extended income support and a waiver that allowed them to forgo training.

TAPR reports include data on each exiter’s characteristics, services received, and employment outcomes. Data on characteristics, for example, should include the worker’s date of birth, gender, ethnicity, educational level, and layoff date. Data on services received should include data on training (such as dates the participant entered and completed training, and the type of training received), on other TAA benefits received (such as extended income support, job search allowance, and relocation allowance), and on co-enrollment in WIA or other federal programs. Data on outcomes should include the date the worker exited the TAA or other federal program, whether the worker was employed in the first full quarter after exit, whether the worker was employed in the third full quarter after exit, and the worker’s earnings in these quarters. Where possible, outcome data are to be obtained from state UI wage records.

Labor uses the TAPR data to track TAA program outcomes against national goals. Unlike the WIA programs, however, TAA has no individual state performance goals, and states do not receive incentives or sanctions based on their performance levels, nor are they otherwise held accountable for their performance. At the national level, the TAA program has failed to meet at least one of its performance goals each year since 2001, the first year for which goals were set. Table 1 shows goals and outcomes for fiscal years 2004 and 2005.

In addition to submitting TAPR data, states also submit data to Labor on TAA services and expenditures each quarter through the Form 563. Form 563 includes counts of participants receiving TAA services, while TAPR includes individual-level data on former participants who have exited the program. States are required to submit each quarter’s Form 563 data about 1 month after the end of the quarter. Form 563 includes data on services such as the number of new training participants (by type of training— occupational, remedial, and on-the-job), the number of workers in training at the end of the quarter, the number of training waivers issued, and the number of recipients of job search and relocation allowances, and expenditures on extended income support.

Common Measures

In response to an Office of Management and Budget (OMB) initiative, Labor recently began requiring states to implement common performance measures for WIA programs. OMB established a set of common measures to be applied to most federally funded job training programs that share similar goals. Labor further defined the common measures for all of its Employment and Training Administration programs and required states to implement these measures beginning July 1, 2005. Because it operates on a fiscal year rather than a program year basis, Labor required the TAA program to implement the measures by October 1, 2005.

In addition to standardizing the performance measures, the common measures guidance also standardizes the definition of exiters across all programs. An exiter is defined as any participant who has not received a service funded by the program or funded by a partner program for 90 consecutive calendar days and is not scheduled for future services. The exit date is defined as the last date of service. For TAA participants, the exit date may be the training completion date, but if additional services are provided after training is completed, or if the participant is continuing to receive TRA, he or she would not be exited from the program. Some services are not significant enough to delay exiting, however. These include receiving UI benefits, some case management services, and postplacement follow-up.

Flow of Data for Collecting and Reporting TAA Data

The process of collecting and reporting TAA performance data involves all three levels of government. Participant forms and case files are generally collected and organized by frontline staff in local areas, usually at the one- stop. In some states, local staff may enter some of the information into an IT system that is either integrated with the state’s IT system or able to create an electronic file to transmit to the state. In other states, paper case files are physically transferred to state officials for data entry.

At the state level, TAA data are often maintained in more than one IT system. For example, benefit payment information is usually in the same IT system that houses Unemployment Insurance payment information. However, information on participant characteristics and services (including status of training and whether or not the individual has exited) resides in one or more other systems. In some states, this participant information remains as a paper case file until it is determined that the participant has exited, and it is time to include him or her in the TAPR submission.

To compile the TAPR submission, state agencies administering TAA typically match participant records to their state’s UI wage record system to determine whether these former participants are employed and, if so, the wages they are earning. In some states, staff must manually enter information obtained from the UI wage record system into the TAPR file, while other states have IT systems capable of automatically matching UI data with participants’ records. States may also use the Wage Record Interchange System (WRIS) to match participant records to other states’ UI wage records for participants who found jobs in other states. Some states may link participant records to other partner programs’ IT systems to track activities across programs or to determine if the participant has exited all programs. Once Labor receives the TAPR data, officials perform edit checks and calculate performance levels at the national and state level.

TAA Data Do Not Provide a Complete, Credible Picture of the Program’s Performance

TAA performance data are incomplete and may be inaccurate. States report that they are not including all TAA participants in their TAPR performance data, despite Labor’s requirement that all participants be included after they exit the program. In addition, some states may not have documentation to verify the accuracy of participants’ exit dates in TAPR and are not using all available data sources to determine TAA participants’ employment outcomes. Furthermore, 1 state in 5 is using manual rather than automated processes to compile TAPR data, and others have IT systems with limited capacity to control for errors. Having such IT systems could hinder states’ ability to ensure that the data are complete and accurate. However, many states are planning to make improvements to their TAA IT systems’ capabilities this year. Some state TAA officials said that resource constraints have made it difficult to ensure their data are complete and accurate.

TAA Performance Data Are Incomplete and May Be Inaccurate

Many states are not including all exiting participants in the TAPR submissions that Labor uses to calculate performance outcomes for TAA participants, such as the reemployment and retention rates. Participants who received training were most likely to be included in states’ TAPR data, but those who had training waivers and had not received training were least likely to be included. Only 23 of the 46 states we surveyed reported that they are including in their TAPR submissions to Labor all exiting participants, regardless of the type of benefit or service they received. Fourteen states reported that participants who received waivers but did not receive training were unlikely to be included in the TAPR (see fig. 1), and 3 states reported that they do not include any participants unless they receive training. This finding is consistent with a review by Pennsylvania’s state auditor that found that participants who received waivers from training were not included in their TAPR submissions.

Our review of the TAPR data states submitted to Labor during fiscal year 2005 confirms our survey results—some states appear to be excluding some of their participants in their TAPR data files. For example, 9 states only included in their TAPR submissions participants who received training. Another 12 states had TAPR submissions composed almost exclusively (97 to 99 percent) of participants who received training (see table 2). However, several states did include relatively more of the participants who had not received training. For example, for 6 states, under 60 percent of the participants reported in the TAPR had received training. We have no other reliable source of data to help us assess what proportion of participants nationwide actually receive training and, therefore, what the proportion in the TAPR should be. In a recent study that examined services and outcomes for five trade-related layoffs, however, we found that between 9 and 39 percent of potentially eligible TAA participants enrolled in training. Excluding certain participants from the TAPR could skew the TAA performance outcomes calculated by Labor because the outcomes may be disproportionately based on participants who received TAA-funded training.

Labor does not have a process in place to ensure that states are including in their TAPR submissions all exiting TAA participants. Labor’s regional offices may review whether states’ TAPR submissions are complete during their state monitoring visits. However, because Labor has not had a standard monitoring tool, there has been no assurance that the regional offices were consistently reviewing whether all exiting participants are reported in states’ TAPR data. Labor officials tell us that they are currently developing a core monitoring guide, but it is not clear if the guide will address this issue.

Despite the importance of accurately identifying exiters, the exit dates themselves may not be accurate because some states do not consistently obtain proper documentation to verify the dates. Accurate exit dates are critical to TAA performance data for two reasons. First, a participant’s exit date determines if the individual should be included in the state’s TAPR submission to Labor. Second, the timing of the date of exit determines when a participant’s employment outcomes will be assessed. Labor’s guidance requires that states have documentation for participants’ exit dates but does not specify the type of information that needs to be included in the documentation. For example, for participants who received training, it does not specify that the documentation should demonstrate that training was actually completed. Such documentation could include certificates of training completion, attendance records, or reports from training providers.

TAA officials in 4 of the 5 states we visited said they had a process for obtaining documentation to show that participants completed training, but it is not clear whether such processes are uniformly followed by states. Officials in 3 states said that they receive training certifications, either from participants or from trainers, that show that training was completed. In another state, a TAA official said that the state sends participants a follow-up survey after training to verify that the training was completed, but some participants do not return the survey. Officials in 1 of the 5 states we visited said they did not have a process for certifying or documenting that participants completed training.

A recent review in 4 other states by Labor’s Office of Inspector General (OIG) confirmed that states do not have effective processes for verifying exit dates. In its review of 150 TAA case files, the OIG found that there was no documentation in any of the reviewed files to verify that the participants had completed the program on the recorded date of exit. OIG reported that states often recorded an anticipated date of exit when participants first entered the program, but did not collect any further documentation to confirm that participants had completed the training, and if so, whether they had completed training on the originally recorded date. The OIG recommended that Labor ensure that states collect and record TAA participants’ actual date of exit, maintain the source documentation for such exit dates, and make the documentation readily available for review. According to an OIG official, Labor had not implemented these recommendations as of January 2006.

Some States Are Not Using All Available Data Sources to Determine Employment Outcomes for TAA Participants

Some states are not using all available data sources to determine TAA participants’ employment outcomes. Labor requires states to use UI wage records to determine the employment outcomes of participants reported in the TAPR. However, each state’s wage record database includes only wage data on workers within the state and does not have data on participants who found employment in another state.

To help track employment outcomes of TAA participants across state lines, states can obtain their employment and earnings information using other methods. Labor encourages states to use WRIS, a data clearinghouse that makes UI wage records available to participating states seeking information on TAA participants who may have found employment outside their state. Thirty-four of the 46 states we surveyed reported that they routinely use WRIS to obtain employment outcome data on former TAA participants (see fig. 2). Three states reported that they do not use WRIS but instead routinely use interstate agreements with individual states to obtain employment outcome data. Opting to use interstate agreements with individual states instead of using WRIS is likely to result in access to fewer states’ UI wage records than states would have if they used WRIS and may result in lower reported outcomes. Seven states use only their own states’ UI wage records to determine participants’ employment outcomes. State TAA officials cited several reasons for not using WRIS, including that it took too long to receive the needed information and it was not a priority for the state. Six states that do not currently use WRIS said that they plan to begin using this system in the future.

Nearly half of the 46 states are not routinely using other supplemental information sources even though it may be the only way to collect outcome information for certain participants. UI wage records, which cover about 94 percent of workers, do not include some categories of workers, such as self-employed persons, most independent contractors, military personnel, federal government workers, and postal workers. To document the employment status of these workers in the TAPR, states can use supplemental data, such as pay stubs and follow-up surveys sent to participants after they leave the program. Using supplemental data is likely to provide a more complete picture of participant outcomes because it helps states avoid inaccurately recording participants as unemployed in the TAPR. In an earlier report on WIA performance data, 23 of the 50 states told us they needed to use supplemental data in order to meet their expected performance levels for the reemployment measure under WIA.

Twenty-two states reported that they rarely if ever collect supplemental data to obtain outcome information on TAA participants (see fig. 3). State TAA officials said that they did not collect supplemental data because states’ TAA IT systems lacked the capacity to record supplemental data; they judged data collected through UI wage records and WRIS as sufficient, or collecting supplemental data was not required; and they lacked sufficient resources.

Some States’ TAA IT Systems Have Limited Capacity to Help Ensure Data Quality

Some states reported IT system limitations that could hinder the states’ ability to ensure their TAA data are complete and accurate.

GAO, Workforce Investment Act: States and Local Areas Have Developed Strategies to Assess Performance, but Labor Could Do More to Help, GAO-04-657 (Washington, D.C.: June 1, 2004). data on employment outcomes into their TAPR data rather than electronically transferring the data from the UI wage record file. Using manual rather than automated processes increases the opportunity for errors to be introduced into the data through data entry. Six states responding to our survey expressed concern that errors in data entry may be one of the main causes of incomplete or inaccurate TAA data.

One state’s process for manually compiling the TAPR illustrates opportunities to introduce errors into the data. In this state, staff at the state level enter data on TAA participants’ training contracts into a contract database. To compile the TAPR, they identify participants in the contract database whose training was scheduled to be completed during the quarter covered by the TAPR, and they enter data on those participants into a new spreadsheet. To identify employment outcomes for the TAPR, the staff look up the exiting TAA participants on printouts from the state’s UI wage record system and manually enter data on the participants’ employment status and wages into the spreadsheet. The data from the spreadsheet are then converted into the TAPR reporting format and sent to Labor.

Limited IT system capabilities. Many states’ IT systems for compiling TAA data do not have certain IT system capabilities, such as performing edit checks, that help a state report complete and accurate data to Labor. Only 15 of the 46 states we surveyed had TAA IT systems with each of three such capabilities: Performing edit checks to prevent data errors: Edit checks aid in identifying invalid data, such as an entry in a date field that is not a date.

Identifying dates TAA participants completed WIA-funded services: The ability to identify when TAA participants complete WIA-funded services can help ensure that TAA participants are not counted in the TAPR report while they are still receiving services under WIA. If participants are still receiving services, then it is too soon to assess their employment outcomes in the TAPR report.

Allowing staff to query the system to assess data reliability and completeness: Queries allow staff to pull certain information out of the system to answer questions, and without this capability, staff may not be able to properly assess data quality and diagnose data problems. For example, in one local area we visited, a TAA specialist who is responsible for reporting on numerous TAA participants described having great difficulty determining if training completion dates had been entered for participants as appropriate because the specialist could not query the system to get a list of participants and their training status.

More than half of the states told us they had plans to make at least one of these system improvements during the next year. For example, 17 states reported plans to improve their TAA IT systems’ capability to perform edit checks (see fig. 4).

Some states with electronic systems may have the capability to track TAA participants across other programs serving them, but several do not. Out of the 37 states that told us they had electronic systems to compile their TAPR data, 29 states said this same system captures program information for WIA programs, and similarly, 29 said the system captures information for Employment Services. Thirteen states said that these capabilities extended to all six programs and benefits that we examined, which includes, in addition to WIA and ES, Trade Readjustment Allowance, National Emergency Grants, Veterans Employment and Training Program, and UI. See appendix II for a complete listing of states’ systems linkages.

In addition, several states commented on our survey that they have planned enhancements to their TAA IT systems that may help coordinate across programs and increase the likelihood of capturing more outcomes: Improving coordination of data across programs: Six states reported planning changes, such as developing a single case management system for several programs that would allow more coordination of data across programs.

Transitioning from manual to electronic processes: Two states that have been using manual processes reported plans to develop electronic interfaces to capture needed data for the TAPR.

Adding capacity to record supplemental data: Two states reported that their IT system changes will enable them to begin recording supplemental data for use in determining TAA participants’ employment outcomes.

Some States Told Us Limited Resources Contributed to TAA Performance Data Problems

Some states reported that limited TAA administrative funds hindered their ability to ensure the quality of the TAA performance data they collect and maintain. To cover their TAA program’s administrative costs, states receive an allocation each year equal to 15 percent of their TAA training allocation. In fiscal year 2006, 9 states received less than $100,000 in TAA administrative funds, and another 10 states received between $100,000 and $300,000. These funds are used to cover all the administrative activities of the program, such as reviewing waivers and training plans, processing applications for job search or relocation allowances, and any associated data collection and reporting. Some states also use these funds for direct case management services to participants because they are the only TAA funds available to provide these services. However, we recently reported that state officials told us the TAA administrative funds were often insufficient to meet the case management needs of the program and they relied on other programs to provide those services. (For a complete listing of each state’s TAA training and administrative funds, see app. III.)

State and local TAA officials said that resource shortages contribute to difficulties in identifying exit dates, using supplemental data sources, and entering data in a timely manner. For example, one state official commented on our survey that TAA case managers often do not have enough time to follow up with participants to learn about their status after they have been sent to training. Another state official said that insufficient case management can delay the identification of participants exiting the program. Officials in 2 other states told us that supplemental data were too time-consuming and burdensome to collect, given the program’s current funding levels. Officials said that resource limitations also presented challenges in entering the data in a timely manner. An official in one local area we visited reported a tremendous backlog in entering TAA participant data into the IT system because there were just two staff to handle approximately 1,000 TAA cases. Similarly, in another local area, the office manager told us that TAA staff were spread too thinly, a condition that adversely affected the collection and entry of TAA data.

Labor Makes Some TAA Performance Data Available, but Its Usefulness Is Limited

Labor reports data on TAA petition and certification activity, program participation, and key performance measures, but this information may not be useful for gauging current program performance. The information may be helpful in providing a long-term national picture of program outcomes, but it represents past, rather than current, performance. UI wage records—the primary data source for tracking TAA performance— provide a fairly consistent national view of TAA performance and allow for tracking outcomes over time. At the same time, the UI wage records suffer from time delays and, together with the use of longer-term outcome measures, affect the timing of states’ performance reports to Labor and, subsequently, the information that Labor makes publicly available. Most of the outcome data reported in a given program year actually reflect participants who left the program up to 2 years earlier. In addition, Labor does not consistently report TAA data by state or industry or by services or benefits received—a step that would make the data more useful to policymakers. States responding to our survey reported that they would like additional information from Labor, such as how their TAA performance compares to the performance of other states and other federal employment and training programs.

Publicly Available TAA Data Do Not Provide a Clear Picture of Current Program Performance

Labor makes some TAA statistics available through postings on its Web site and through published reports, but they do not provide useful information on current performance. Labor provides some TAA activity and participant data by fiscal year including number of petitions received, certifications issued, and denials by state; distribution of certifications by industry; number of new participants receiving extended income support payments or training; and summary statistics on former TAA participants (such as race, education level, and benefits and services received).

In addition to reporting on TAA activity and participant data, Labor also reports on three key TAA performance measures. The TAPR data submitted by states are used to calculate national and state outcomes on the TAA performance measures—wage replacement, reemployment, and retention—for each fiscal year. In 2005, Labor made state-by-state TAA outcome information publicly available for the first time. According to Labor officials, making this information public represents an effort to emphasize performance, and they intend to post state-by-state outcome information on the Web site for all future fiscal years. Labor’s regional offices directly provide states with information on their TAA performance relative to the program’s national goals. Some regional offices also provide states with reports showing the performance of all states in the region, according to officials we interviewed.

However, the information Labor makes publicly available may not provide a clear picture of current TAA performance because, in addition to being incomplete and perhaps inaccurate, the data represent past performance and are not consistently reported by type of service, state, or industry.

Data represent past performance. Because TAA performance is measured using UI wage records and long-term performance measures such as employment retention, the most up-to-date TAA performance data currently available may represent performance from several years in the past.

Use of wage records: Using UI wage records to measure outcomes provides a common yardstick for making long-term comparisons across states because they contain wage and employment information on most workers. At the same time, these files suffer delays between the time an individual gets a job and when this information appears in wage records. State procedures for collecting and compiling wage information from employers can be slow and time-consuming. Data are collected from employers only once every quarter, and employers in most states have 30 days after the quarter ends to report the data to the state. For example, the wage report for the last calendar quarter of the year (ending on December 31) is due to the state on January 31. We previously reported that for the majority of states, the delay between the time an individual gets a job and the time this information appears in wage records is up to 4 months.

Design of measures: In addition to using a job placement measure, Labor also uses two longer-term measures to gauge TAA performance—an earnings measure and a job retention measure. These measures may be useful for assessing how well the program is meeting its long-range goals to increase the employment, retention, and earnings of participants. However, the use of these measures requires states to wait from one to three quarters after participants exit the TAA program before measuring the outcomes. For example, although states record whether participants entered employment in the first quarter after exit, two more quarters must elapse before employment retention is measured. Participants who exit the TAA program have their outcomes assessed in the first, second, and third quarters after exit. However, data to measure all outcomes are not available until the fifth quarter after exit, and the outcomes are not submitted to Labor until midway through the sixth quarter. Figure 5 illustrates the time it takes before a TAA participant would be included in performance outcome calculations.

Most States Find TAA Performance Data Useful, but Many Would Like Additional Information

While approximately one-third of the states found TAA performance information they currently receive from Labor to be greatly useful, some would like Labor to provide them with additional information to help manage their program. Nearly half of the 46 states we surveyed told us that they find the performance information they receive from Labor to be moderately useful (see fig. 7), and 8 states reported that Labor’s TAA performance information is of little or no use for program management.

Nearly half of the states we surveyed told us that they routinely develop information on their own performance beyond what they submit to Labor. For example, an official in 1 state reported that it calculates its own outcomes before receiving them from Labor in order to make managers and executives aware of the state’s performance, and it uses this information to engage state and local TAA staff in making program adjustments. In addition, approximately one-third of the states routinely develop information on their local areas’ performance. Labor does not provide analysis of local area performance to states because it does not collect this type of information in the TAPR.

While many states provide the performance information of their own state and that of other states to their local area TAA staff, few states provide information on their local areas’ performance to local TAA staff. Only 27 of the 46 states in our survey reported that they share information from Labor with local area staff on how their state’s performance compares to national TAA performance goals. In addition, only 7 of the 16 states that generate additional performance information for local areas reported that they share this information with local TAA staff. One expert we spoke with told us that regularly sharing performance information with local program staff enables them to understand how the data they collect are being used and the importance of complete and accurate data for producing reliable performance information. Our recent report on performance measurement also noted that frequent and routine communication of performance information helps program staff and stakeholders use such information to accomplish program goals as they pursue day-to-day activities. These practices could lead to better program management and produce more reliable performance data to assess TAA performance in the future.

States said that they would like to receive additional performance information from Labor to help manage their TAA program. Thirty-four states would like more information than they currently receive on their own state’s performance, and 39 states reported they would like information comparing their states’ TAA performance to their WIA Dislocated Worker performance (see fig. 8). According to one state official we spoke with, receiving additional TAA performance information that is displayed by type of service and by state would enable officials to respond more effectively to performance problems and to learn what strategies states with similar TAA populations are using to achieve different performance outcomes.

Labor Has Taken Steps to Improve the Quality of Performance Data, but Issues Remain

While it has limited authority to hold states accountable, Labor has taken steps to improve the quality of TAA data states submit, but these steps do not fully address all issues. Labor has no mechanism to sanction states for poor performance or poor-quality data because the law and current regulations do not provide one. However, Labor has begun an initiative that requires states to review a sample of their data for accuracy. It is too soon to fully assess whether Labor’s efforts have improved data quality, but most states reported on our survey that Labor’s new requirements have increased awareness of data quality at the state and local levels. States also report that they would like more opportunities to share lessons learned about issues related to data quality. Labor is requiring changes in some TAA performance measures to align them with measures for other federally funded job training programs. Many states reported that the changes are burdensome, and some states are experiencing delays in implementing the changes.

Data Validation Is Having Positive Effects but Does Not Address All Concerns

To address data quality concerns, Labor developed a process for states to use to validate the TAPR data they submitted to Labor. Starting with data submitted in fiscal year 2003, Labor required states to review a sample of participants’ records and compare what was reported for certain data elements to data in source files. State staff review the source files and record whether each data element is supported by source documentation and, therefore, passed data validation. If the source files show a data element was incorrect or was not supported with documentation, the data element fails. States use Labor’s software to calculate error rates, and they submit the results to Labor.

While it is too soon to assess whether Labor’s data validation efforts have improved data quality, many states said that the efforts are having a positive effect. Thirty-five states reported that efforts have improved the accuracy of the data. Thirty-seven of the 46 states told us they have helped increase the awareness of data quality at the state level, and 25 states told us they have improved awareness at the local level (see fig. 9).

Until recently, Labor has not had a standard process for ensuring that states performed data validation correctly. Labor officials tell us, however, that beginning in 2006, regional offices are conducting data validation compliance reviews of a subsample of validated records to ensure that the records were accurately validated and the files contained all required source documents.

While states report that Labor’s data validation requirements are having some positive effects, Labor’s data validation efforts do not address two key problems. First, guidance for data validation defined for the first time the type of source documents needed to validate TAPR data elements, including exit dates. However, the guidance does not specify that the source documents for training completion dates should show that participants actually completed training. Second, data validation does not provide for assessing whether TAPR submissions are complete. Because the data validation process only covers participant records included in states’ TAPR submissions for the year, it does not look beyond those records to determine whether all exiting participants were included.

Labor Has Taken Several Other Steps to Help Improve TAA Data, but States Want More Opportunities to Share Lessons Learned

In addition to implementing data validation, Labor has taken various actions to better instruct states and to provide tools for improving the data they submit to Labor.

Technical assistance and training: In 2005 and 2006, Labor brought together state TAA staff for training conferences on the new data requirements for implementing common measures. According to Labor officials, Labor’s regional offices periodically hold roundtables with states to discuss issues that sometimes include data quality. Labor provides technical assistance, as needed, to states through telephone calls and e-mails. According to Labor officials, Labor is planning to start holding quarterly conference calls with states about TAA issues, including data quality.

Guidance on data reporting: Labor issued guidance and instructions for TAA data reporting, such as instructions defining how “date of exit” is to be determined under common measures. In May 2005, Labor issued a guidance letter to states addressing several issues with data quality, such as the use of WRIS and supplemental data to determine employment outcomes. In general, states reported that the guidance and training they had received from Labor provided a clear understanding of certain data requirements, such as the requirements for data validation and for using UI wage records. States were somewhat less likely to say that Labor had provided a clear understanding of the documentation needed for the date of exit and how supplemental data could be used to document TAA employment outcomes.

Monitoring: Labor’s regional offices conduct monitoring visits to review states’ TAA programs. In the past, Labor did not have a standard protocol for these monitoring visits, and the monitoring did not always cover the quality of the TAA data being submitted by states. However, as of March 2006, Labor was developing a standard monitoring guide for its regional staff.

Pilot project on federal employment data: Labor collaborated with the Office of Personnel Management, the U.S. Postal Service, and the Department of Defense to create a pilot data exchange system to provide states access to wage record information on federal and military employment. The system that began operating in November 2003 can help states obtain more complete employment outcome data on participants who exited job training programs because it provides information on federal employment that is not available in state UI wage records. Many states are using the system to help determine employment outcomes for job training programs, such as those funded under the Workforce Investment Act. However, only 3 of the 46 states we surveyed reported that they were routinely using this system to obtain employment outcomes for the TAA program.

Despite Labor’s efforts to improve data quality, most states would like more help. Most states reported that they do not currently have opportunities to share lessons learned with other states on topics related to TAA data quality, such as how to use supplemental data, and they expressed interest in having such opportunities. For example, 29 states told us they do not currently have opportunities to share lessons learned on data validation, and 44 states told us more opportunities to do so would be helpful (see fig. 10).

Labor’s Common Measures May Address Some Concerns, but States Are Still Struggling to Implement Changes

In response to an OMB initiative, Labor made changes to some of the TAA performance measures and to TAA reporting requirements in order to implement common measures (see table 3). OMB established a set of common performance measures to be applied to most federally funded job training programs that share similar goals. Labor further defined the common measures for all of its Employment and Training Administration programs and required states to start reporting TAA data under the revised requirements in fiscal year 2006.

Moving to common measures may increase the comparability of outcome information across programs and make it easier for states and local areas to collect and report performance information across the full range of programs that provide services in a one-stop system. Prior to common measures, many federal job training programs had performance measures that tracked similar outcomes but had variation in the terms used and the way the measures were calculated. For example, the programs used different time periods to assess whether participants got jobs. Under common measures, the time period used to assess employment outcomes is uniform across all covered programs.

Implementation of common measures involved some changes in the data states collect for the TAPR: Standardized exit definitions: Labor’s guidance on common measures provides for a clearer understanding of when TAA participants should be exited from the program than did earlier TAA guidance. Under Labor’s guidance, states must wait 90 days after participants receive their last service or benefit—from TAA, WIA, or other related programs—to record them as exiters. Prior to this change, states could exit participants without waiting 90 days. Most states reported that the guidance and training they received from Labor provided a clear understanding of the definition of exit under common measures, but 7 states disagreed.

Coordination of exit dates: Under common measures, states are encouraged to establish a common exit date for each participant who is co-enrolled in more than one program. For example, if a participant receives services under TAA and under WIA, then the two programs should use the same exit date for the participant. Coordinating exit dates improves data quality by avoiding the problem of counting a participant as unemployed in the program’s performance measures when, in fact, the participant is still receiving services in another program and is not ready to be counted in the performance measures.

Changes in IT systems: A number of data fields were added or changed in the TAPR as part of the new common measures policy, requiring states to add or change data fields in their IT systems and to instruct staff on changes in data to be collected on participants and employment outcomes. Most states reported that the guidance and training they received from Labor provided a clear understanding of the changes needed in the TAPR to implement common measures; however, 7 states disagreed.

Although moving to common measures may ultimately make it easier for states to collect and report performance information across programs, most states reported that making changes to implement common measures had been a burden in terms of time and cost (see fig. 11), and often viewed coordinating exit dates as burdensome. States were nearly evenly divided in their views, however, on whether they had been given sufficient time by Labor to complete the changes. Nineteen states said they had not been given sufficient time, while 18 states said they had. Twenty- six states reported that they will have provided guidance to staff or changed data elements in their IT systems by the time the first quarterly TAPR is due in fiscal year 2006 (see fig. 12). Other states reported that they would have these changes completed sometime later in 2006, while some states said they could not estimate when they will complete the changes. Coordinating exit dates was the change that states considered the most burdensome. Seventeen states were unable to estimate when they would be able to coordinate exit dates across programs.

In a previous study, we cautioned that rushed implementation of reporting changes may not allow states and local areas enough time to fully meet the requirements and could negatively affect the data quality of the information reported.

Conclusions

Since the passage of the TAA Reform Act of 2002, the TAA program has evolved to become one of the most important means to help the workers affected by our nation’s trade policies rejoin our nation’s workforce. The program has seen substantial increases in the population it serves and in the funds available to serve them. Unfortunately, efforts to monitor the program’s performance have not kept pace with the program’s development. Four years after the passage of the reforms, we still do not know whether the program is achieving what lawmakers intended.

The TAA program has suffered a history of problems with its performance data that have undermined the data’s credibility and limited their usefulness. And while we see that Labor has taken some steps aimed at improving the performance data, the data remain suspect. They fail to capture outcomes for some of the program’s participants, and many participants are not included in the final outcomes at all. These failures may have contributed to the program’s poor performance in achieving its national goals. Labor lacks the authority to hold states accountable for their outcomes or for the quality of their data, and as a result, some states may not see the value of investing more effort to ensure their data are complete and accurate. In truth, officials tell us the funding to support their efforts is small, and it fluctuates from year to year, making such an investment difficult to sustain. But the success of the program is being judged by the outcomes the program achieves and whether or not it meets its goals. The current budgetary environment makes it risky not to take all necessary steps to ensure that the outcomes are an accurate and credible reflection of the program’s performance.

Labor has taken a major step toward improving the quality of its performance data through its new data validation requirements. States report that these requirements have significantly raised the awareness of data quality at the state and local levels–an essential component in any effort to improve the accuracy of the data. But these efforts do not fully address all issues. No steps have been taken to ensure that all participants are included in the TAA performance data or that exit dates are adequately documented. Monitoring can help address data issues, but Labor is just now developing a standard monitoring guide that would help ensure that key problems are identified during monitoring visits. Until these steps are complete, the data can not be verified and may remain incomplete. Providing opportunities for states to share lessons learned may make states more aware of effective approaches for ensuring data quality, and several states expressed an interest in more such opportunities.

Labor has recently improved the availability of TAA performance information by posting the information on its Web site and by making some state-by-state performance data available. However, the performance data are not as informative as they could be because they aggregate all participants and do not show the outcomes of participants based on the types of services they received. As a result, policymakers lack the information they need to understand program participation and performance and to assess future needs.

Recommendations for Executive Action

While Labor has taken steps to share information with states and to improve data quality, more work is needed.

To help ensure that TAA participant data reported by states are consistent, complete, and accurate, Labor should clarify through guidance and other communications with states that all participants who exit the program should be included in the TAPR and the documentation needed to verify the training completion date; ensure that the core monitoring guide currently under development for regional office site visits includes guidance for assessing whether states’ data collection processes for performance reporting capture all participants; and provide states with opportunities to share lessons learned with other states on issues that may affect data quality.

To make TAA performance information more useful for program management, Labor should provide this information by the type of services received by TAA participants.

Agency Comments

We provided a draft of this report to Labor for review and comment. In its comments, Labor did not disagree with our findings and recommendations and said the report will be helpful in its continuing efforts to improve the quality of TAA performance data. Labor noted that the issues raised in the report about administrative costs and the burden of new reporting requirements are compounded by having a workforce investment system that is duplicative in its service delivery design, resulting in separate record-keeping and reporting systems. Labor also identified a number of actions that it is taking to ensure that performance accountability is an expectation of the program. A copy of Labor’s response is in appendix IV.

As arranged with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution of this report until 30 days from the date of this report. At that time, we will send copies of this report to the Secretary of Labor, relevant congressional committees, and others who are interested. Copies will also be made available to others upon request. The report is also available on GAO’s home page at http://www.gao.gov.

If you or members of your staff have any questions about this report, please contact me at (202) 512-7215 or nilsens@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

We examined (1) whether the Trade Adjustment Assistance (TAA) performance data provide a credible picture of the program’s performance, (2) what TAA performance data Labor makes available to the public and states and the usefulness of the data for managing the program, and (3) what Labor is doing to address issues with the quality of TAA data submitted by states.

To learn more about the factors that affect TAA data quality and to learn what states are doing to ensure data quality, we conducted a Web-based survey of state TAA officials and conducted site visits in five states, where we interviewed state officials and visited local areas or one-stop centers. We also collected information on the quality of TAA data through interviews with Department of Labor officials in headquarters and all six regional offices, nationally recognized experts, and reviewed relevant literature. Our work was conducted between December 2004 and March 2006 in accordance with generally accepted government auditing standards.

To determine the factors that affect the quality of TAA performance data, we conducted a Web-based survey of workforce officials in the 46 states that were allocated TAA funds in fiscal year 2005, and we obtained a 100 percent response rate. These officials were identified using Labor’s list of state TAA officials. We e-mailed the contacts, and they confirmed that they were the appropriate contact for our survey or identified and referred us to another person at the state level. Survey topics included (1) the current status of TAA data collection and reporting systems, (2) implementation of the U.S. Department of Labor’s data validation requirements, (3) state and local efforts to ensure the quality of TAA data, and (4) the implementation of common measures. The survey was conducted using a self-administered electronic questionnaire posted on the Web. We contacted respondents via e-mail announcing the survey, and sent follow-up e-mails to encourage responses. The survey data were collected between November 2005 and January 2006. We received completed surveys from all 46 states that were allocated TAA funding in fiscal year 2005 (a 100 percent response rate). We did not include Washington, D.C., and U.S. territories in our survey.

Web-Based Survey

We worked to develop the questionnaire with social science survey specialists. Because this was not a sample survey, there is no sampling error. However, the practical difficulties of conducting any survey may introduce errors, commonly referred to as nonsampling errors. For example, differences in how a particular question is interpreted or in the sources of information that are available to respondents can introduce unwanted variability into the survey results. We took steps in the development of the questionnaire, the data collection, and data analysis to minimize these nonsampling errors. For example, prior to administering the survey, we pretested the content and format of the questionnaire with four states to determine whether (1) the survey questions were clear, (2) the terms used were precise, (3) respondents were able to provide the information we were seeking, and (4) the questions were unbiased. We made changes to the content and format of the final questionnaire based on pretest results. We also performed computer analyses to identify inconsistencies in responses and other indications of error. In addition, a second independent analyst verified that the computer programs used to analyze the data were written correctly.

Site Visits

We visited five states—California, Iowa, Ohio, Texas, and Virginia—and traveled to local areas or one-stop centers in each of these states. We selected these states because they represent different TAA data collection approaches (that is, states where data are entered into information technology systems at the local level and those where data are entered at the state level), received a relatively large share of TAA funds in fiscal year 2005, and are geographically dispersed. From within each state, we judgmentally selected local areas to visit (see table 4). In each state, we interviewed state TAA officials about their collection and use of TAA data, IT systems used to compile TAA performance data, and efforts to ensure the data are complete and accurate. Similarly, we interviewed local area officials about their collection and use of TAA data.

Information that we gathered on our site visits represents only the conditions present in the states and local areas at the time of our site visits, from January 2005 through October 2005. We cannot comment on any changes that may have occurred after our fieldwork was completed. Furthermore, we cannot generalize the findings from our site visits beyond the states and local areas we visited.

Appendix II: Program Information in State TAA IT Systems

In our survey, states were asked whether the IT system they use to compile data for the Trade Act Participant Report (TAPR) currently captures program information for certain other Labor programs or benefits.

Appendix III: State TAA Administrative Allocations, Fiscal Years 2005-2006

The TAA Reform Act authorizes up to $220 million per year for training under the TAA Program. Labor allocates 75 percent of the training funds to states according to a formula that takes into account each state’s previous year allocations, accrued expenditures, and participant levels. Labor holds the remaining 25 percent of training funds in reserve to distribute to states throughout the year according to need. To cover administrative costs associated with training under the TAA program, Labor allocates to each state additional administrative funds equal to 15 percent of its training allocation. Table 6 shows Labor’s initial 75 percent allocation for training and associated administrative expenses. States also receive an additional 15 percent of any reserve (25 percent) funding and job search/relocation allowances for program administration.

Appendix IV: Comments from the Department of Labor

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

Dianne Blank, Assistant Director Kathy Peyman, Analyst-in-Charge In addition, the following staff made major contributions to this report: Vidhya Ananthakrishnan, Melinda Cordero, Laura Heald, Adam Roye, and Leslie Sarapu served as team members; Amanda Miller and Carolyn Boyce advised on design and methodology issues; Rachael Valliere advised on report preparation; Jessica Botsford advised on legal issues; Lise Levie verified our findings.

GAO Related Products

Trade Adjustment Assistance: Most Workers in Five Layoffs Received Services, but Better Outreach Needed on New Benefits. GAO-06-43. Washington, D.C.: January 31, 2006.

Workforce Investment Act: Labor and States Have Taken Actions to Improve Data Quality, but Additional Steps Are Needed. GAO-06-82. Washington, D.C.: November 14, 2005.

Workforce Investment Act: Substantial Funds Are Used for Training, but Little Is Known Nationally about Training Outcomes. GAO-05-650. Washington, D.C.: June 29, 2005.

Unemployment Insurance: Better Data Needed to Assess Reemployment Services to Claimants. GAO-05-413. Washington, D.C.: June 24, 2005.

Workforce Investment Act: Labor Should Consider Alternative Approaches to Implement New Performance and Reporting Requirements. GAO-05-539. Washington, D.C.: May 27, 2005.

Trade Adjustment Assistance: Reforms Have Accelerated Training Enrollment, but Implementation Challenges Remain. GAO-04-1012. Washington, D.C.: September 22, 2004.

Workforce Investment Act: States and Local Areas Have Developed Strategies to Assess Performance, but Labor Could Do More to Help. GAO-04-657. Washington, D.C.: June 1, 2004.

National Emergency Grants: Labor Is Instituting Changes to Improve Award Process, but Further Actions Are Required to Expedite Grant Awards and Improve Data. GAO-04-496. Washington, D.C.: April 16, 2004.

Workforce Investment Act: Improvements Needed in Performance Measures to Provide a More Accurate Picture of WIA’s Effectiveness. GAO-02-275. Washington, D.C.: February 1, 2002.

Trade Adjustment Assistance: Experiences of Six Trade-Impacted Communities. GAO-01-838. Washington, D.C.: August 24, 2001.

Trade Adjustment Assistance: Trends, Outcomes, and Management Issues in Dislocated Worker Programs. GAO-01-59. Washington, D.C.: October 13, 2000.