Workforce Investment Act: Substantial Funds Are Used for Training, but Little Is Known Nationally about Training Outcomes
Released: 2005-06-29T00:00:00

Letter

Background

WIA specifies a different funding source for each of the act’s main client groups—youth, adults, and dislocated workers. Our report focuses on adults and dislocated workers. Once the Congress appropriates WIA funds, the amount of money that flows to states and local areas depends on a specific formula that takes into account unemployment for the adult and dislocated worker funding streams, the number of low-income individuals for the adult funding stream, and the number of long-term unemployed for the dislocated worker funding stream. Labor allots 100 percent of the adult funds and 80 percent of the dislocated worker funds to states. The Secretary of Labor retains 20 percent of the dislocated worker funds in a national reserve account to be used for National Emergency Grants, demonstrations, and technical assistance and allots the remaining funds to each of the 50 states, the District of Columbia, and Puerto Rico. In program year 2003, Labor allotted approximately $2 billion to states for adults and dislocated workers (see app. II for a listing of program year 2003 allotments by state). Upon receiving its allotments, each state can set aside no more than 15 percent to support statewide activities. These may include a variety of activities that benefit adults, youths, and dislocated workers statewide, such as providing assistance in the establishment and operation of one-stop centers, developing or operating state or local management information systems, and disseminating lists of organizations that can provide training. In addition, each state can set aside no more than 25 percent of its dislocated worker funds to provide rapid response services to workers affected by layoffs and plant closings. The funds set aside by the states to provide rapid response services are intended to help dislocated workers transition quickly to new employment. After states set aside funds for rapid response and for other statewide activities, they allocate the remainder of the funds—at least 60 percent—to their local workforce areas (see fig. 1).

Approximately 600 local workforce areas exist throughout the nation (see fig. 2). Each local area has a local workforce board that administers WIA activities within the local area, including selecting one-stop center operators, identifying eligible training providers, developing links with employers, and overseeing the use of funds for employment and training activities.

WIA was intended to meet both the needs of businesses for skilled workers and the training, education, and employment needs of individuals. The act allows training and employment programs to be designed and managed at the local level to meet the unique needs of local businesses and individuals. Another aspect of the act was to provide customers with easy access to the information and services they needed and empower those who need training to obtain the training they find most appropriate. One cornerstone of WIA was the one-stop concept where information about and access to a wide array of services would be available at a single location. At the one-stop center, customers can get information about job openings; receive job search and placement assistance; receive an assessment of their skill levels, aptitudes, and abilities; and obtain information on a full array of employment-related services, including information on local education and training providers. Through the one- stop centers, employers also have a single point of contact to provide information about current and future skills needed by their workers and to list job openings.

The services typically available at one-stop centers fall into the following categories: Core services. These include job search and placement assistance, the provision of labor market information, and preliminary assessment of skills and needs. Core services are available to all adults who come to a one-stop center, with no eligibility requirements imposed.

Intensive services. These include comprehensive assessments, case management, short-term prevocational services, work experience, and internships. Intensive services are available to qualified adults and dislocated workers who are unable to obtain or retain a job that leads to self-sufficiency.

Training services. These include occupational skills training, on-the- job training, customized training, and skill upgrading and retraining. Training services are available to qualified adults and dislocated workers who are unable to obtain or retain employment after receiving at least one intensive service.

Supportive services. These include services—such as transportation, child care, and housing—that are necessary to enable WIA participants to take part in WIA activities.

WIA requires the use of ITAs, which allow qualified individuals to purchase the training they determine best for themselves. Adults and dislocated workers use ITAs to purchase training services from eligible providers they select in consultation with case managers. Payments from ITAs may be made in a variety of ways, including the electronic transfer of funds through financial institutions, vouchers, or other appropriate methods. Payments may also be made incrementally. WIA requires that ITAs can only be used to purchase training from programs listed on an eligible training provider list (ETPL). Local boards, in partnership with the state, compile this list by identifying training providers and programs whose performance qualifies them to receive WIA funds to train adults and dislocated workers. Good information allows participants to make informed training choices. In this regard, WIA requires that local boards ensure that participants have access to performance information on training providers, including the percentage of individuals completing their training program, the percentage of individuals in the program who obtained jobs, and the wages earned by these individuals.

Under certain situations, however, local boards have the option of purchasing training without using ITAs. The three exceptions to using ITAs are if the activity is on-the-job training or customized training, if a local board determines an insufficient number of eligible providers exist in the area (such as in a rural area), or if a training provider has a demonstrated effectiveness in serving a special population that face multiple barriers to employment.

To assess whether it is accomplishing its goals, WIA established a performance measurement system for the programs directly funded by WIA—one that emphasized results in areas of job placement, retention, earnings, and skill attainment (see table 1). WIA requires states to use Unemployment Insurance wage records to track employment-related outcomes. States submit this information to Labor in annual reports submitted each December. States also submit quarterly performance reports, which are due 45 days after the end of each quarter. In addition to the performance reports, states submit their updates for WIASRD every January. WIA also requires Labor to conduct at least one multisite study to determine program results by the end of fiscal year 2005.

Local Boards Used an Estimated 40 Percent of Available WIA Funds to Train Program Participants

Local boards used an estimated 40 percent of the WIA funds they had available in program year 2003 to obtain training services for WIA participants. Nationally, local boards had approximately $2.4 billion in WIA funds that were available to serve adult participants during program year 2003 and used about $929 million for training activities, primarily occupational classroom training. The remaining funds pay for other program costs, including job search assistance, case management, and supportive services, as well as administrative costs. We estimate that 416,000 WIA participants received training during the year. However, because some individuals may have received more than one type of training, this count may include some individuals more than once. Of those trained in program year 2003, about 323,000 participants received occupational classroom training, of which about 85 percent was purchased through ITAs. Local boards also used the flexibility provided under WIA to offer a broad range of training-related activities aimed at increasing employability but not included in WIA’s definition of training.

An Estimated 40 Percent of Available WIA Funds Were Used for Training

Local boards nationwide used an estimated 40 percent of their WIA funds for training in program year 2003. During that year, local boards had about $2.4 billion in WIA funds available to serve adults and dislocated workers. Almost all local boards had funds from the WIA adult and dislocated worker funding streams; in addition, many boards had National Emergency Grants or funds from two state set-asides, the 15 percent set- aside for statewide activities and the 25 percent set-aside for rapid response. WIA permits local boards up to 2 years to spend each program’s funding. Accordingly, to get a national picture of available WIA funds at the local level, we defined available funds as the combined amount of program year 2003 funds and funds carried over from program year 2002. Of the approximate $2.4 billion in combined WIA funds that local boards had available, about $1.8 billion (75 percent) came from the program year 2003 allocation, while the rest consisted of funds carried over from 2002 (see fig. 3). Allocations from the WIA adult and WIA dislocated worker funding streams together constituted about 80 percent of the funds local boards could use.

Of the $2.4 billion available, local boards used approximately $929 million in program year 2003 to fund training activities, representing about 40 percent of the WIA funds that were available to serve adult participants in the program. The remaining funds pay for other program costs, including job search assistance, case management, and supportive services, as well as administrative costs. We found that local boards spent an estimated $724 million on training and obligated another $205 million. Obligations are funds local boards commit to pay for training, but for which services have not yet been provided and costs not yet incurred (see fig. 4).

Local boards used a slightly higher percentage of their WIA adult funds (43 percent) for training than their dislocated worker and state set-aside funds, both of which had 37 percent used for training (see fig. 5).

Of the WIA dollars local boards spent on training, an estimated 79 percent was for occupational classroom training. Boards used the remainder of the funds to pay for on-the-job training, customized training, and other types of training, including adult basic education and skill upgrading.

In addition to using WIA funding, many local boards also leveraged other sources of funding to help pay the costs of training for WIA participants. Some of these funding sources were federal programs, including Trade Adjustment Assistance (TAA), the H-1B skill grant program, and Temporary Assistance for Needy Families (TANF). For example, in program year 2003, one board we visited in Maryland enrolled 49 WIA participants in training funded by TAA. Other sources of funding came from state and local governments or private entities. For example, at one site we visited in Georgia, the local public school system paid for high school equivalency classes for WIA participants, including teacher salaries, testing, and books and materials. In addition, the local housing authority provided training on a variety of soft skills for 600 of its clients at the one- stop center.

WIA Participants Were Most Likely to Receive Occupational Classroom Training

Overall, we estimate that 416,000 WIA participants were enrolled in training during program year 2003 and that about 323,000 participants received occupational classroom training. In our survey, local boards reported the number of people enrolled in each category of training rather than the total receiving training. As a result, it is possible that the 416,000 includes some duplication of individuals who received more than one kind of training during that year (see table 2).

We estimate that more than three-quarters of the training that participants received (78 percent) was occupational classroom training (see fig. 6). On- the-job training and customized training each accounted for an additional 6 percent of the total training that occurred in program year 2003, while 10 percent of training included other activities, such as adult education, literacy classes, entrepreneurial training, and skill upgrading.

Approximately 85 percent of the occupational classroom training provided during program year 2003 was purchased by participants through ITAs. At some local boards, however, specific kinds of occupational classroom training were obtained without the use of ITAs. For example, one local board we visited in Iowa used WIA funds to pay for classes in typing skills at the local community college, but ITAs were not used to pay for this training because it was short-term training that did not result in a credential.

At the eight sites we visited, participants used ITAs to pay for training in a wide variety of occupations. WIA regulations require that participants select a training program directly linked to employment opportunities. Each of these local boards told us that nursing and other health care professions were in high demand locally, and accordingly, participants in all eight areas sought training in health care occupations. Other high- demand occupations at some of the local boards we visited included information technology, truck driving, manufacturing, and teaching. As local labor markets have changed, some boards have developed specially tailored programs designed to mirror shifts in labor demand. For example, one local area in California faced massive layoffs in the high-tech industry but had a dearth of qualified teachers in the local schools. As a result, the local board created a program to train dislocated high-tech workers to become teachers.

Local Boards Fund Training-Related Activities and Services

In addition to providing training activities, local boards used the flexibility provided under WIA to offer a broad range of intensive services, some of which are aimed at increasing job skills. These training-related activities, including work experience, internships, and computer skills training, are not captured in WIA’s definition of training and, therefore, not paid for with training dollars. Accordingly, neither the amount of funding spent on these activities nor the number of participants who benefit from them are identified in our statistics on training. Nevertheless, these activities can play a significant role in preparing WIA participants for successful employment.

Although many WIA participants do not need extensive training to obtain a job, some still need help improving a variety of skills that will further their chances of successfully searching for and retaining a job. Much like training, these activities are intended to increase employability. Many are short-term activities, such as computer lab training and other intensive skills workshops. For example, approximately one-half of local boards use WIA funds to offer participants computer lab training in software applications, basic keyboarding, and other computer skills. One board we visited in Georgia offers another type of short-term, intensive skills workshop through its Basic Industrial Maintenance program. During this 4-week training course, participants learn a variety of skills including the basics of construction, plumbing, and carpentry. Other training-related activities that are intended to increase skills but are not included in WIA training are internships and work experience opportunities. For example, one local board we visited in Iowa spent about $79,000 in program year 2003 to provide internships and work experience opportunities to 41 WIA participants.

Moreover, some boards we visited used WIA funds to pay for supportive services, such as child care and transportation, that enable participants to attend training. Like funding spent on training-related activities, the cost of these supportive services is not reflected in the amount of WIA funding that local boards spend on training. However, these services can represent a large investment of WIA dollars. Local boards have flexibility in whether and how they use WIA funds for services that support training. For example, one local board we visited in rural Iowa spent over $160,000 in program year 2003 on a wide array of supportive services for people in training, including child care, transportation, eye exams, and glasses. Because the area contains several correctional facilities, the board also used a portion of these WIA funds to purchase bicycles for ex-offenders who were attempting to reenter the workforce but had lost their driving privileges. Another local area we visited in California spent $87,000 in supportive services during program year 2003; in addition to paying for child care and transportation, these WIA funds paid for items including books, uniforms, and tools, as well as services such as fingerprinting and tuberculosis testing, which some training programs require. Not all boards provided WIA-funded supportive services to people in training, however. One local area we visited in Maryland did not use WIA funding to provide supportive services to its adult participants, although it referred those in need to other agencies for assistance.

Local Boards Manage ITAs but Have Faced Challenges in Implementing Them

Most local workforce boards have developed policies to manage the use of ITAs, but many boards have encountered challenges in trying to implement their use. Local boards often require participants to complete various skill assessments prior to entering training and gather additional information on the occupation for which they desire training. In addition, they generally limit the amount of money participants can spend on training using ITAs and how long participants can spend in training. Although the vast majority of local boards use ITAs, most also said they have faced challenges in managing their use. The challenge most frequently identified was lack of good performance data on training providers. Local boards in rural areas face a different challenge—lack of nearby training providers. Some boards have identified initiatives to mitigate the challenges they face.

Most Local Boards Have Policies to Manage ITAs

We estimate that most local boards established procedures to ensure that any training purchased using ITAs is warranted and placed spending limits on individual ITAs to control costs. WIA regulations require that participants must receive at least one intensive service, such as individual counseling and career planning, before enrolling in training. Many local boards also require participants in the adult and dislocated worker programs who want training to first complete specified activities to demonstrate their need for training. For example, local boards may require participants to complete skill assessments or attend specific workshops.

In addition, they may require participants to gather information on the occupation for which they want training or document their inability to find employment (see fig. 7).

More than 80 percent of the local boards require adults and dislocated workers to complete specified skill assessments or tests before being allowed to purchase training with ITAs. For example, three local boards in Georgia, Kansas, and Mississippi commented that participants are required to complete career assessments or occupational interest inventories prior to training. A local board in New York mentioned that staff are required to interview participants and determine whether they are in need of training and have the skills and qualifications to successfully participate in the training program. A local board we visited in Georgia required participants to demonstrate an aptitude and interest in an area before enrolling in training. Depending on how proactive the participant is, the process can take up to 6 months before the participant is enrolled in a training program.

Approximately 70 percent of the local boards require adults and dislocated workers to gather additional information about the occupation for which they want training. For example, a local board in Arizona noted that participants must interview three people working in their desired field; another board, in Washington, commented that participants are required to conduct informational interviews with employers in the occupation they wish to pursue. Similarly, three of the local boards that we visited required participants to perform specific tasks prior to entering training. For example, one local board we visited in Georgia required participants to gather specific information on training providers and then compose an essay explaining why they chose a particular course and provider. Another board we visited, in California, asks participants to research and disclose information on the training they want to pursue, including the occupation’s starting wage, whether this wage is sufficient to support them and their family, working conditions, available job openings, and education and skill requirements. Also, a local board we visited in Maryland requires participants seeking certain types of training, such as on the operation of a tractor-trailer, to obtain prehire letters that guarantee employment once training is completed.

About one-third of local boards required adults and dislocated workers to complete workshops prior to enrolling in training. For example, a West Virginia board noted that participants must demonstrate their soft skills or complete a soft skills program before entering occupational training. One of the local boards we visited in California requires that participants attend an orientation and a soft skills workshop prior to entering training. They also offer additional voluntary workshops, including those in which participants explore different vocations, complete applications, practice interviews, and perform self-assessments. Similarly, a local board we visited in Georgia requires participants to take a general orientation and résumé writing workshop before being eligible for ITAs.

Approximately 85 percent of local workforce boards limit the amount of money participants can spend on training using ITAs. An estimated 31 percent of the local boards limited ITAs to between $3,000 and $5,000 (see fig. 8). At local boards limiting ITAs, the amount of the caps ranged from $350 at one local board to $15,000 at three local boards. One of the local boards we visited in Maryland said that its ITA cap had changed four times since program year 2000. The cap started at $4,000, was then increased to $4500 because of inflation, and later rose to $5,500 because of the increased demand for, and cost of, information technology training. However, because of reduced WIA funding, the board later lowered the cap to $3,000.

Rather than having a single dollar limit on ITAs, two local boards reported having ITA caps that could vary by training program. Specifically, a local board in Hawaii limits an ITA for any particular training program to the cost of the least expensive provider among those who offer equivalent programs, while a local board we visited in Iowa limits an ITA for a particular training program to the highest cost of obtaining that training at a state public institution.

Most of the boards that impose dollar caps on ITAs expect the amount of the ITA to also cover the costs of books, tools, and uniforms. A number of local boards also expect the amount of the ITA to cover the costs for supportive services and other items, such as fees for licenses, certifications, tests, and physical exams (see fig. 9).

In addition to limiting the amount of money participants can spend on training, an estimated two-thirds of the local boards also limit how long participants can spend in training. The most frequent limit on training was 2 years. Many of these local boards, however, indicated that time limits could be waived depending on an individual’s circumstances. One of the local boards we visited in Iowa had no time limit for training using ITAs, but encouraged shorter-term training lasting one or two semesters, especially for dislocated workers, because of the limited period for collecting unemployment insurance.

The use of particular training providers varied among the local boards we visited. For example, in program year 2003, one local board we visited in California used 48 private schools as training providers for 305 participants and 3 community colleges for 42 participants. This local board also used 15 4-year colleges for 202 participants. The majority of these participants were former high-tech workers being trained to become teachers. A local board we visited in Georgia relied heavily on private, proprietary schools, using them for about 1,000 participants each year. The board believes these schools are more flexible than other training providers and offer a wide array of training courses. On the other hand, a local board we visited in Iowa used community colleges for 90 percent of the 246 ITAs issued in program year 2003. Local boards responding to our survey reported that 37 percent of the ITAs issued in program year 2003 were used at proprietary schools and 35 percent were used at community colleges. The remainder were used at various providers, including 4-year colleges, public vocational and technical schools, and community based- organizations.

In December 2004, Mathematica Policy Research issued an interim report concluding that the way ITAs are administered influences the likelihood of participants requesting counseling or receiving ITAs. The study also found that different approaches to administering ITAs appeared to have a limited effect on participants’ training choices. Labor funded the 3-year study to assess how different approaches to administering ITAs affect training choices, employment and earnings outcomes, returns on investment, and customer satisfaction. Eight sites were included in the study. These sites were located in or around Atlanta, Georgia; Bridgeport, Connecticut; Charlotte, North Carolina; North Cook County, Illinois; Jacksonville, Florida; and Phoenix, Arizona. Mathematica’s study results are not generalizable beyond these eight sites. A later report by Mathematica will present an analysis of how the ITA approaches affect additional outcomes, including training completion, customer satisfaction, and employment and earnings after training, as well as an analysis of the return on the investment in training.

Local Boards Face Challenges in Implementing ITAs

Most local boards faced some challenges in their efforts to implement ITAs, and local boards in rural areas face a unique challenge. The majority of local boards encountered as challenges the lack of performance data on training providers, the timing of training being offered, being able to get new training providers on the eligible training provider list (ETPL), and the ability to link the ITAs with economic development strategies (see table 3). A few local boards have found ways to mitigate some of these challenges.

Nearly two-thirds of the local workforce boards encountered the lack of performance data on training providers as a challenge. For example, a local board in Wisconsin commented that the lack of consistent data on training providers reduces the value of the ETPL to local boards. A local board in Missouri noted that one of the greatest challenges lies in not having reliable information regarding the quality and relevance of the training being offered by training providers. The board further stated that the state’s report card containing performance information on training providers in Missouri was incomplete or unavailable. Local boards we visited in California and Iowa said that a statewide report card on training provider performance did not exist. In lieu of a statewide report card, the California boards tracked training provider performance themselves, while the Iowa boards relied upon informal feedback about provider performance.

Approximately 60 percent of the local boards encountered the timing of the training offered by providers as a challenge. For example, two local boards we visited in Georgia and Iowa said that some participants are unable to attend training programs that are offered only during a regular academic schedule. The Iowa board explained that some participants who have to wait too long for a training program to begin may have their unemployment insurance benefits run out before the training can be completed. On the other hand, some local boards have found solutions to deal with this issue. For example, a local board in Washington commented that it purchased classroom group training to offer more flexibility as to when training will be offered and to satisfy the demand for particular training. Similarly, a local board in Massachusetts noted it persuaded local technical high schools to offer programs at night, thereby resulting in greater availability of training in trade-related fields that are in high labor demand. A local board we visited in Maryland developed close relationships with area community colleges that now schedule occupational training outside the regular academic calendar.

More than half the boards faced getting new providers on the ETPL as a challenge. This has been a long-standing concern. We reported in 2001 that according to training providers, the data collection burden resulting from participation in WIA can be significant and may discourage willingness to participate under WIA as training providers. Labor has heard these concerns from training providers and has approved waivers for 30 states. These waivers, in effect, give states additional time to address data collection challenges. However, getting training providers, particularly community colleges, to participate in the ETPL remains a concern for some local boards. For example, local boards in California, Indiana, Massachusetts, and Michigan noted that some providers, community colleges in particular, are reluctant to participate in the ETPL. A local board that we visited in California elaborated on this point, stating that community colleges in its area are operating at full capacity and do not need WIA dollars or participants and, therefore, are not interested in getting on the ETPL. Some local boards are finding ways to encourage providers to participate. For example, one local board in Massachusetts has been working collaboratively with other boards throughout the state to meet with key figures in the community college system to provide information, consultation, and feedback. Two local boards we visited in Iowa and Maryland have developed strong relationships with the 16 community colleges in their areas, each of which is on state lists. Another local board we visited in California conducts regional ETPL workshops with training providers and shares ideas with other local boards in the surrounding areas.

More than half of the local boards found linking ITAs with local economic and business development strategies to be a challenge. Several local boards provided different examples of why they found it difficult to provide participants with training in high-demand occupations in their area. The area around one local board we visited in California faced a nursing shortage, but nearby training for nurses was difficult to obtain. Some area community colleges were opting not to provide nursing training because they could not recoup the costs of operating them. A local board we visited in Georgia also said that keeping up with businesses’ needs is a challenge and noted that some information technology sector-based training courses are not always available. Other local boards identified some initiatives they are pursuing to strengthen links with economic development. For example: A local board in Michigan partners with a local technical training center to develop intensive, short-term certificate programs in high- skill, high-wage, and high-demand fields. The technical center is operated by a local community college but offers non-credit certificate programs responsive to business and community needs.

A local board in Ohio works closely with the local Chambers of Commerce and economic development partners to formulate training programs that are based on employer demands. Specifically, if the board hears that a group of employers have a skill need, then the board will develop the appropriate training program with a service provider.

A local board in Texas uses an industry cluster analysis report to focus attention on specific industries and then targets funds for training in these sectors. Training institutions must prepare industry-approved training curriculums in order to have programs approved by the local board and to have ITAs issued to participants for training.

A local board in Washington partnered with representatives from local education, economic development, and government to develop a shared blueprint for economic development and training around eight high-demand industry clusters.

A number of local boards representing rural locations mentioned that the requirement to use ITAs to purchase training from providers on the ETPL presented a different problem for them from their counterparts in urban areas. Local boards in Arizona, Colorado, Kansas, Montana, North Carolina, and Utah all mentioned that participants in rural areas have few nearby training providers from which to choose. Additionally, local boards in Kansas and Montana noted that being located in a rural area with limited providers makes dealing with ITAs burdensome. A local board that we visited in rural California applied for and received a waiver from using ITAs. The board directly contracts with a community college for a 2-year program to train participants to become registered nurses and an 18- month program to train participants to become licensed vocational nurses. Through April 2004, 73 participants have completed the nursing programs, and according to the board, all were employed in their respective fields.

Little Is Known about Outcomes of Those Being Trained because of Weaknesses in Data Collected

Little is known on a national level about the outcomes of those being trained. Certain aspects of WIASRD have been found to be incomplete and unverified. Additionally, data generally cannot be compared across states or local areas because of differences in data definitions. Labor is taking some steps that may address these concerns and plans to complete an evaluation that will measure the overall impact of the WIA program.

Concerns Remain That Data Collected Are Unreliable

Our analysis of program year 2003 WIASRD has shown that the database does not contain information for a large number of data elements. It is unclear if these values are missing because Unemployment Insurance wage records are not available or because they simply were not entered into the database by officials. This finding reaffirms issues that have been raised previously about the quality of data that Labor uses to assess program performance. Our 2004 report found that performance data submitted by states in quarterly and annual reports were not sufficiently reliable to determine outcomes for the WIA programs. Labor’s Office of Inspector General (OIG) raised the same concerns in 2002 by noting that because of inadequate oversight of data collection and management, little assurance exists that the states’ performance data for all WIA programs are either accurate or complete. OIG’s report found that of the 12 local areas examined, none had adequately documented procedures for validating participant performance data. Similarly, none of the four states examined had sufficient procedures to ensure the accuracy of their reported performance data. At the time, OIG recommended that states use a statistical sampling method for validating reported data.

Because of questions about the comparability of data elements, states’ performance data are of limited value for national comparisons, or even comparisons within a single state. Labor allows local areas to exercise some flexibility in determining how to collect and report certain performance data on participants. For example, while local areas collect data on a participant who leaves the program, they use different methods to determine when a person has officially exited from the program. WIASRD guidelines define participants as having exited the program on the last date they received WIA services. Labor allows two ways to define exit: (1) at the point of case closure or (2) when the participant has not received any services or training for more than 90 days and is not scheduled for future services. We found that local area definitions differ from this and from each other. For example, one local board we visited defines exit as occurring when participants are finished with their WIA services; another board defines exit when participants have found a new job and the wages for their new job are considered acceptable (regardless of the number of days that have passed since their last service). Wage and employment outcomes under these different circumstances would vary greatly, making it difficult to compare outcome data across local areas. In a prior report, we also noted the data are not comparable on what constitutes a credential—whether it is the attainment of a certified skill or of a degree. Labor allows states and local areas to determine what constitutes a credential.

Labor Is Taking Actions to Improve Data and Assess Impact of Program

Labor is taking some steps to address data quality concerns and improve the data used at the national level. This includes implementing a new project to validate the performance information collected and reported under WIA. The data validation initiative covers both the accuracy of reports submitted to Labor on program activity and performance outcomes and the accuracy of individual data elements. The report validation checks the accuracy of states’ software used to calculate the performance reports submitted to Labor. For example, if a state reports, for a particular time frame, that 100 adults found employment after they received services, the validation software searches through the state’s electronic records to ensure that 100 records are found that match these criteria. Data element validation, on the other hand, evaluates the accuracy of the participant data used to generate reports submitted to Labor. The process compares selected information from a sample of participant exit records with the original paperwork that contained this information. Data element validation results in an estimate of the error rate for each data element that has been selected for validation.

While Labor provides guidelines for the data validation, states are responsible for executing the initiative. Program year 2003 is the first year states have completed the process, and Labor plans to review state results and start setting acceptable standards for error rates. Eventually, Labor plans to hold states accountable for meeting accuracy standards. Once these accuracy standards are in place, states failing to meet the standards may lose eligibility for incentive awards or, in cases with significant deviations from the standards, may be sanctioned. Because this initiative is relatively new, it is too soon to tell if it will satisfactorily resolve all data quality problems associated with WIASRD outcome measures—but it is a step toward addressing these concerns.

Labor is also in the initial stages of developing a single, streamlined reporting and record-keeping system that will replace a series of databases, including WIASRD, and may address some concerns about data quality. The system, formally called the ETA Management Information and Longitudinal Evaluation (EMILE), would replace the current data collection and reporting requirements for 12 employment and training programs. The goal of the new system is to allow for consistent, comparable analysis across Labor’s employment and training programs, using the same definitions for specific measures. These definitions, known as common measures, are to be implemented July 1, 2005, well before EMILE is implemented. A feasibility analysis on the EMILE proposal, which will provide information on next steps, is in the planning stages and should be completed by December 2005, but it is unclear how soon EMILE will be implemented.

Labor has plans to meet the requirement that it conduct at least one multisite study to determine the general effectiveness of the WIA program and the specific impacts of WIA services on the community and participants involved. WIA requires Labor to conduct at least one multisite evaluation using a control group by the end of fiscal year 2005. However, as noted in a prior report, Labor will not meet this deadline. Labor is waiting for WIA reauthorization to begin the evaluation, and will likely wait until the second year after reauthorization to commission the study. Labor anticipates it will take at least 5 years to complete the first evaluation.

Concluding Observations

WIA represented a fundamental shift for workforce development because it attempted to significantly change how employment and training services were provided and because it provided considerable latitude to those implementing WIA at the state and local levels, yet little is known about the impacts of these changes. In program year 2003, local workforce boards used a substantial amount of WIA funds to train a large number of individuals. During these times of increasingly tight federal budgets, it is important to know what types of training are the most successful and how the outcomes from training compare with outcomes from other services. We previously recommended that Labor take actions to improve data quality and to conduct an impact evaluation of WIA services. While Labor is taking steps to address the accuracy of data contained in WIASRD, reliable information is not yet available on a national level as to the outcomes of those being trained. In addition, Labor will likely wait until 2007 to conduct the multisite impact evaluation required by WIA. Our findings reaffirm the need for a continued focus on resolving reported data quality issues and determining what services are the most successful.

Agency Comments

We provided a draft of this report to Labor for review and comment. In its comments, Labor acknowledged that the WIA reporting system currently has limited information on training expenditures and training outcomes, but noted that some of our information conflicts with their estimates of these activities. In particular, Labor states the Administration’s estimate of the amount of WIA funds spent on training is lower than the 40 percent that we estimate was used for training in program year 2003. In addition, Labor estimates adult training enrollments to be roughly 200,000. We agree with Labor that the WIA reporting system has limited reliable information. As a result, we went directly to the local workforce boards to obtain as complete a picture as possible on the extent to which WIA funds were used for training and how many adults were trained in program year 2003. Our information differs from Labor’s estimates in two ways. First, our report indicates that 40 percent of WIA funds were used for training in program year 2003. We define funds used for training to include funds spent as well as funds obligated. As shown in the report, of the approximately $929 million used for training in program year 2003, about $724 million was actually spent and another $205 million was obligated. Second, Labor’s estimate of 200,000 adults enrolled in training includes only those adults reported in WIASRD who exited from the program. Our estimate of 416,000 comes directly from the local workforce areas that provided the training and includes the total number of adults who received training in program year 2003, including those who had not exited from the program. We believe our estimates of the amount of WIA funds used for training and the number of adults trained represent a more complete and accurate picture than Labor’s estimates because we included all funds used for training in program year 2003, whether they were spent or obligated, and counted all adults who received training in program year 2003, not just those who exited the program.

Regarding our discussion on data quality, Labor stated that it appreciates that we recognize the steps taken to improve data quality through data validation and noted that it has provided states with the software, handbooks, user guides, and technical assistance necessary to develop reports and document validation. Labor commented that it plans to continue supporting data validation during the transition to common measures and is currently revising the software and related materials to match the new reporting requirements. In addition, Labor stated that in the coming months it will be publishing proposed revisions to the WIASRD in the Federal Register. Labor also has issued guidance standardizing the definition of “exit” for purposes of assessing program performance across all programs implementing common measures. Labor said that states will begin implementing the change on July 1, 2005, and believes that this will improve the comparability of WIA outcome data. We commend Labor’s efforts to improve data quality and acknowledge that these actions are a step in the right direction to having a reporting system that contains complete and accurate information.

Labor also provided technical comments that we incorporated where appropriate. Labor’s entire comments are reproduced in appendix III.

We will send copies of this report to the Secretary of Labor, relevant congressional committees, and other interested parties and will make copies available to others upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov.

A list of related GAO products is included at the end of this report. If you or your staffs have any questions about this report, please contact me at (202) 512-7215 or at nilsens@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other contacts and staff acknowledgments are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

We were asked to determine (1) to what extent Workforce Investment Act (WIA) funds have been used for training, (2) how local workforce investment boards have managed the use of Individual Training Accounts (ITA) and what challenges they have encountered, and (3) what is known at the national level about the outcomes of those receiving training. To address these issues, we conducted a Web-based survey of all local workforce investment boards in the 50 states, the District of Columbia, and Puerto Rico, and visited eight local boards located in 4 states. We conducted our work from June 2004 to May 2005 in accordance with generally accepted government auditing standards.

Web-Based Survey

To determine the extent to which program year 2003 WIA funds were used for training, how local workforce boards manage ITAs, and what challenges they have encountered, we conducted a Web-based survey of the local workforce investment boards for the 590 local workforce investment areas in existence in program year 2003. Program year 2003 was the most recent year for which complete data were available. The basis for our list of local workforce investment boards was the directory from the National Association of Workforce Boards (NAWB). To view the survey, go to www.gao.gov/cgi-bin/getrpt?GAO-05-807SP.

Prior to administering the survey, we pretested the content and format of the questionnaire with a number of local workforce investment boards to determine whether (1) the survey questions were clear, (2) the terms used were precise, (3) respondents were able to provide the financial and client data we were seeking, and (4) the questions were unbiased. We made changes to the content and format of the final questionnaire based on pretest results.

The surveys were conducted using self-administered electronic questionnaires posted on the Web. We received completed surveys from 428 boards (a 73 percent response rate).

We attempted to assess the reliability of the responses to survey questions that asked for quantitative data relating to WIA funds used for training and numbers of clients served. We included questions in the survey that asked whether or not the local board carried out certain practices or procedures to ensure that databases or data systems used to produce the financial and client information we asked for were in fact reliable. These questions asked (1) if there were written procedures that defined data elements or specified how data were collected; (2) if routine internal reviews of data were conducted to check for errors in completeness, accuracy, or reasonableness; (3) if periodic monitoring or audits of the data were conducted to check for errors in completeness, accuracy, or reasonableness; and (4) whether or not routine quality control procedures were in place such as data verification to source documents or computer edit checks. We asked these four questions separately for both the financial data and the client data obtained in the survey. If the local area responded that it had used at least two of the four practices or procedures to monitor the quality of the financial and participant data provided in their survey, we either accepted the responses or called for clarification, depending on which procedures were used. If the local board responded that three or four of these practices or procedures were not used for either financial or client data, we did not use those data. In cases where the local board responded that it was not sure whether these practices or procedures had been carried out or did not answer three or more of the four questions, we telephoned the board to try to determine whether or not it had actually carried out these practices or procedures. In cases where we determined that the data actually did meet our data reliability criteria, based on these telephone calls, we accepted the responses. On the basis of the criteria, we accepted the financial data for all 428 of the completed surveys and we accepted the participant data for 425 of the completed surveys. In the three cases where we did not accept the participant data, we did retain and use responses from other sections of the survey. In addition to including the questions on data reliability in the survey, we also checked the consistency of responses on a number of survey questions that asked for numeric data. On the basis of these consistency checks, some responses were dropped from our analysis.

We generated estimates to the population of 590 boards by treating the 428 responding boards as a simple random sample. We chose to treat the 428 responding boards as a simple random sample from the population of 590 based on an analysis of the differences between the responding boards and the nonresponding boards.

To determine if there were any significant differences between the responding boards and the nonresponding boards, we contacted officials in the 50 states, the District of Columbia, and Puerto Rico to obtain the number of unemployed individuals for each local area. We obtained this number for all 590 boards in the population. We used this information to determine whether sample-based estimates of this characteristic generated from the responding boards compared favorably with the known population values. The known population value of the number of unemployed individuals fell within the 95 percent confidence interval surrounding the sample-based estimate. On the basis of these results, and our assessment that the number of unemployed individuals is correlated with key items we were estimating, we concluded that treating the 428 responding boards as a simple random sample is not likely to introduce significant bias into estimates. Some of the 428 responding boards did not provide responses to all of the items in the survey. To improve our estimates, we employed a nearest neighbor hot deck imputation methodology to account for nonresponse of numerical items.

Because we decided to treat the respondents as a simple random sample of boards, our results are estimates of the population of 590 boards and thus are subject to sampling errors that are associated with samples of this size and type. Our confidence in the precision of the results from this sample is expressed in 95 percent confidence intervals, which are expected to include the actual results in 95 percent of samples of this type. We calculated confidence intervals based on methods that are appropriate for a simple random sample. All percentage estimates have margins of error of plus or minus 5 percentage points or less. All numerical estimates other than percentages have relative margins of error of plus or minus 15 percent or less, except for those shown in table 4. For example, an estimate of $1,000,000 with a relative margin of error of plus or minus 15 percent would have a 95 percent confidence interval of $850,000 to $1,150,000.

The practical difficulties of conducting any survey may introduce other kinds of errors, commonly referred to as nonsampling errors. For example, difficulties in how a particular question is interpreted, in the sources of information that are available to respondents, or in how the data are entered into a database or were analyzed can introduce unwanted variability into the survey results. We took steps in the development of the questionnaire, the data collection, and the data analysis to minimize these nonsampling errors. For example, social science survey specialists designed the questionnaire in collaboration with GAO staff with subject matter expertise. Then, the draft questionnaire was pretested with a number of local boards to ensure that the questions were relevant, clearly stated, and easy to comprehend. As mentioned above, we included additional questions to determine whether certain practices or procedures had been carried out on the financial and client data. When the data were analyzed, a second, independent analyst checked all computer programs.

Since this was a Web-based survey, respondents entered their answers directly into the electronic questionnaire. This eliminated the need to have the data keyed into a database, thus removing an additional source of error.

Site Visits

We used two criteria in selecting site visit locations. First, we stratified the 48 continental states and the District of Columbia into four categories according to amount of program year 2003 adult and dislocated worker formula funds to reflect states with varying funding sizes. The four categories were less than $10 million, $10 million to $25 million, over $25 million to $50 million, and over $50 million. We selected one from each category. Second, we chose geographically dispersed states to help illuminate regional differences in implementing ITAs. From within each state, we judgmentally selected two local boards to provide a mix of urban and rural areas (see table 5).

At each location visited, we obtained general information about the local workforce area and additional information on WIA funding, local training policies implemented, challenges encountered, innovative practices, and reliability of data systems. To increase our confidence in the reliability of the data we gathered from our survey, we interviewed each local board we visited about data monitoring and quality control procedures and policies with respect to their financial and participant databases. We also asked the local boards we visited to show us samples of records in their databases and to trace them to source documents. We generally found their data quality processes and procedures to be sufficiently reliable for the purposes of our report.

Assessment of WIASRD

To determine whether the Workforce Investment Act Standardized Record Data (WIASRD) might be a viable source of data for outcomes of training participants, we reviewed our prior reports about the reliability of the WIASRD data and a report by Labor’s Office of the Inspector General on WIA performance outcomes. We also performed electronic tests of the program year 2003 WIASRD data to check for missing values. The variables analyzed included employment after first, third, and fifth quarters after exit quarter, and type of credential attained. Missing data due to unemployment insurance wage data lags were taken under consideration. On the basis of our analysis, we determined that the WIASRD data elements pertinent to this report were not sufficiently reliable for our purposes. We have discussed the data reliability issues throughout the body of the report.

Appendix II: Summary of Adult and Dislocated Worker Formula Funds for Program Year 2003

Appendix III: Comments from the Department of Labor

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

Joan Mahagan, Assistant Director, and Wayne Sylvia, Analyst-in-Charge, managed all aspects of the assignment. Rebecca Woiwode made significant contributions to this report in all aspects of the work. In addition, Amanda Mackison and Matthew Saradjian assisted in the survey design, data collection, and report writing; Stuart Kaufman and Shana Wallace assisted in the design of the national survey and the assessment of the survey’s data reliability; James Ashley and Sidney Schwartz assisted in data projections of the national survey; Grant Mallie and Stefanie Bdzusek assisted in analyzing the national survey responses; Joan Vogel assisted in assessing the data reliability of the WIASRD data; Jessica Botsford and Richard Burkard provided legal support; and Corinna Nicolaou provided writing assistance.

Related GAO Products

Workforce Investment Act: Labor Should Consider Alternative Approaches to Implement New Performance and Reporting Requirements. GAO-05-539.Washington, D.C.: May 27, 2005.

Workforce Investment Act: Employers Are Aware of, Using, and Satisfied with One-Stop Services, but More Data Could Help Labor Better Address Employers’ Needs. GAO-05-259. Washington, D.C.: Feb. 18, 2005.

Public Community Colleges And Technical Schools: Most Schools Use Both Credit and Noncredit Programs for Workforce Development. GAO-05-04. Washington, D.C.: Oct. 18, 2004.

Trade Adjustment Assistance: Reforms Have Accelerated Training Enrollment, but Implementation Challenges Remain. GAO-04-1012. Washington, D.C.: Sept. 22, 2004.

Workforce Investment Act: States and Local Areas Have Developed Strategies to Assess Performance, but Labor Could Do More to Help. GAO-04-657. Washington, D.C.: June 1, 2004.

National Emergency Grants: Labor Is Instituting Changes to Improve Award Process, but Further Actions Are Required to Expedite Grant Awards and Improve Data. GAO-04-496. Washington, D.C.: April 16, 2004.

Workforce Investment Act: Labor Actions Can Help States Improve Quality of Performance Outcome Data and Delivery of Youth Services. GAO-04-308. Washington, D.C.: Feb. 23, 2004.

National Emergency Grants: Services to Dislocated Workers Hampered by Delays in Grant Awards, but Labor Is Initiating Actions to Improve Grant Award Process. GAO-04-222. Washington, D.C.: Nov. 14, 2003.

Workforce Investment Act: Exemplary One-Stops Devised Strategies to Strengthen Services, but Challenges Remain for Reauthorization. GAO-03-884T. Washington, D.C.: June 18, 2003.

Workforce Investment Act: One-Stop Centers Implemented Strategies to Strengthen Services and Partnerships, but More Research and Information Sharing Is Needed. GAO-03-725. Washington, D.C.: June 18, 2003.

Workforce Investment Act: Issues Related to Allocation Formulas for Youth, Adults, and Dislocated Workers. GAO-03-636. Washington, D.C.: April 25, 2003.

Workforce Investment Act: Interim Report on Status of Spending and States’ Available Funds. GAO-02-1074. Washington, D.C.: Sept. 5, 2002.

Workforce Investment Act: Better Guidance and Revised Funding Formula Would Enhance Dislocated Worker Program. GAO-02-274. Washington, D.C.: Feb. 11, 2002.

Workforce Investment Act: Better Guidance Needed to Address Concerns over New Requirements. GAO-02-72. Washington, D.C.: Oct. 4, 2001.