Workforce Investment Act: Strategies Needed to Improve Certain Training Outcome Data
Released: 2014-03-05T00:00:00

Letter

Background

WIA Programs and Administrative Structure

The WIA Adult and Dislocated Worker Programs provide employment services to a wide range of participants. The Adult Program serves all individuals age 18 and older, and the Dislocated Worker Program serves individuals who have been or will be terminated or laid off from employment, among others. The Adult Program prioritizes certain services for recipients of public assistance and other low-income individuals when program funds are limited. To enable individuals to participate, both programs may offer supportive services such as transportation, childcare, housing, and needs-related payments under certain circumstances. WIA requires that the Adult and Dislocated Worker Programs and other federally-funded employment and training programs provide services through one-stop centers—now called American Job Centers—so that jobseekers and employers can find assistance at a single location.

DOL’s Employment and Training Administration administers the Adult and Dislocated Worker Programs and oversees their implementation, which is carried out by states and local areas. At the state level, the WIA Adult and Dislocated Worker Programs are administered by state workforce agencies. Each state has one or more local workforce investment areas, each governed by a WIB. WIBs select the entities that will operate American Job Centers, which provide most WIA services, and oversee the American Job Center network. WIA provides substantial flexibility to states and WIBs to determine how services are provided.

Employment and Training Services Provided under WIA

WIA represented a fundamental shift from its predecessor program, the Job Training Partnership Act, by decreasing the focus on job training as the primary means to help adults and dislocated workers get a job. The Adult and Dislocated Worker Programs provide participants three types of services:

Core services include basic services such as job searches and labor market information, and may be accessed with or without staff assistance.

Intensive services include such activities as comprehensive assessment and case management, which require greater staff involvement. Intensive services are available to participants who are unable to obtain or retain employment after receiving at least one core service.

Training services include such activities as occupational skills or on- the-job training. To be eligible for training services, participants must: (1) be unable to obtain or retain employment after receiving at least one intensive service, (2) be in need of training, and (3) have the skills and qualifications to successfully complete the training program, among other requirements.

To assess participants’ skills and determine whether they need training, WIBs may require them to complete certain activities. We previously found that most WIBs required participants to complete skills assessments or gather information about the occupation for which they wanted training before entering a training program.

DOL requires that participants eligible for training select approved training providers in consultation with case managers, but participants ultimately choose the training programs in which they participate. that training be directly linked to in-demand occupations, which DOL interprets to include both currently-available jobs and occupations that are projected to grow in the future.

Funding, Performance Measures, and Data Quality

20 C.F.R. § 663.440(c). financial sanctions and incentive funding. In addition, WIA requires states to use Unemployment Insurance (UI) wage records to track employment-related outcomes.

In 2005, DOL began allowing states to request waivers to replace the WIA performance measures with a smaller set of common measures that focus on employment, retention, and earnings across multiple programs (see table 1). These measures do not include credential attainment and are calculated differently than the WIA measures.

As of December 2013, a total of 48 states and territories, plus the District of Columbia, had obtained this waiver and used the common measures for the Adult and Dislocated Worker Programs. Although states and territories are no longer subject to financial sanctions or incentive funding for the credential attainment measure, DOL still requires them to report the number of training participants who earn credentials.credential attainment improves workers’ labor market experience through In general, higher earnings, greater mobility, and enhanced job security, according to DOL and research studies.

We have previously raised concerns about the accuracy and comparability of DOL’s data on credential attainment because DOL’s guidance allowed states and local areas considerable flexibility in defining what constitutes a credential. In February 2002, we recommended that DOL more clearly define this term. Since that time, DOL issued guidance to clarify its definition of credential attainment. In addition, to verify that the data states report on credential attainment are accurate, DOL requires WIA programs to collect and retain documentation on participants’ credential attainment.

In addition to the performance measures, WIA requires states to report a wide array of data that includes whether training participants find employment that relates to their training. A WIA participant’s employment is considered to be related to the training received if in the new job the participant uses “a substantial portion of the skills taught in the training.” However, DOL does not require that states collect and retain documentation for employment related to training to verify the accuracy of data they report. In a September 2011 report, DOL’s Office of Inspector General raised questions about the quality of DOL’s data on employment related to training. Specifically, the Office of Inspector General found that these data were “incomplete and unreliable” based on its review of the data DOL maintains in its WIA database. The Office of Inspector General recommended, among other things, that DOL provide guidance on the best methodology for reporting such data as well as provide oversight to ensure that states develop or identify best practices for increasing the rate of training-related employment. DOL agreed with these recommendations.

Federal agencies that work in partnership with states and local areas to administer programs such as WIA must continually balance the competing objectives of collecting uniform performance data and giving program managers flexibility to meet local needs. Our prior work has found that federal agencies have considered key attributes of data quality for performance data, including:

Completeness—the extent to which enough of the required data elements are collected from a sufficient portion of the target population or sample.

Accuracy—the extent to which the data are free from significant errors.

Consistency—the extent to which data are collected using the same procedures and definitions across collectors and times.

Ease of use—how readily intended users can access data, aided by clear definitions, user-friendly software, and easy-to-use access procedures.

Congressional Proposals to Reauthorize WIA Programs

Congress is currently considering legislation to reauthorize the Workforce Investment Act of 1998, which has been due for reauthorization since the end of fiscal year 2003. The Supporting Knowledge and Investing in Lifelong Skills (SKILLS) Act (H.R. 803), passed by the House, would establish both credential attainment and training-related employment as performance measures. The Workforce Investment Act of 2013 (S. 1356), introduced in the Senate and reported out by the Committee on Health, Education, Labor, and Pensions, would establish credential attainment as a performance measure, but not training-related employment.

DOL’s Data Show That Most Training Participants Attained Credentials, but Data on Training-Related Employment Are Unreliable

Most Training Participants Attained Credentials

During the time-period from program year 2006 through program year 2011, the total number of participants in WIA’s Adult and Dislocated Specifically, in the Adult Worker Programs increased significantly.Program, the number of participants increased from about 625,000 to about 1.25 million, and in the Dislocated Worker Program, the number of participants increased from about 272,000 to about 761,000. During this same time-period, the number of participants who received training services also increased, but not as dramatically as the number of overall For example, the number of Dislocated Worker participants (see fig. 1).participants who received training increased from about 76,000 in program year 2006 to about 120,000 in program year 2011.

Since the number of participants who received training services did not increase at the same rate as the number of participants who entered into WIA’s Adult and Dislocated Worker Programs, the percentage of participants who received training generally declined, according to DOL reports (see fig. 2).

There are several reasons that may have contributed to the declining percentage of participants who received training. Beginning in December 2007, the significant increase in the overall number of participants is likely attributed to the downturn in the economy that led to a dramatic rise in unemployment and the subsequent infusion of additional funds to WIA programs from the American Recovery and Reinvestment Act of 2009 (Recovery Act). The Recovery Act provided $500 million for grants for the Adult Program and $1.25 billion for grants for the Dislocated Worker Program. By spring of 2009, DOL began allocating these funds to states to supplement existing WIA funds. DOL encouraged states to use Recovery Act funds to increase training in an effort to help Americans acquire new skills and return to work. DOL officials stated that the increase in demand for training services exceeded the increase in supply provided for by the Recovery Act, which may have led to shortages in training capacity that contributed to a relative decline in training. Another factor, according to DOL officials, is that some program participants had limited access to needs-based financial assistance and other supportive services, such as child care, which may have prevented them from entering WIA training services. Further, DOL noted that the preference of many program participants is for immediate employment rather than job training.

Similarly, the percentage of training participants who earned credentials has also generally declined from program year 2006 through program year 2011 (see fig. 3). For example, DOL’s data show the percentage of those who earned credentials in the Adult Program dropped from about 74 percent in program year 2006 to about 58 percent in program year 2011. Likewise, the percentage of those in the Dislocated Worker Program who earned credentials dropped from about 75 percent in program year 2006 to about 63 percent in program year 2011. Despite the decline in the percentage of training participants who earned credentials, DOL reported that the total number of participants attaining credentials increased during this time period. For example, the number of training participants in the Dislocated Worker Program who attained credentials increased from about 48,000 in program year 2006 to about 79,000 in program year 2011.

According to DOL officials, the percentage of training participants who earned credentials may have declined in part as a result of changes in the performance measures that states negotiated. In program year 2005, states began requesting waivers to replace the WIA performance measures with a set of common performance measures that do not include the employment and credential attainment measure for the Adult Officials in three of the six states we and Dislocated Worker Programs.reviewed said that after this request was approved, reporting data on credential attainment became a lower priority for them. Officials in one of these states also said they stopped collecting and reporting these data until DOL issued clarifying guidance in 2010 emphasizing the importance of credential attainment as a pathway to employment.

DOL’s data on credential attainment also show that participants in the Dislocated Worker Program typically have higher credential attainment rates than participants in the Adult Program. DOL officials explained that the Dislocated Worker Program has a higher funding level that supports more training, and that participants in this program generally have longer work histories and more advanced education and so are more likely to enter into training programs that lead to credentials. In contrast, participants in the Adult Program are more likely to require training focused on remedial education and job readiness which are less likely to result in credentials as defined by DOL.

Of the training participants who attained credentials during program year 2011—approximately 89,000 in the Adult Program and 84,000 in the Dislocated Worker Program— about two-thirds in each program earned an occupational credential, such as a welding certificate or clinical medical assistant certificate (see fig. 4). The next two most common types of credentials attained by training participants were an occupational skills license, such as a license in nursing, and an associate’s degree. The fact that training participants attained occupational credentials at higher rates than longer-term academic degrees is consistent with DOL’s recommendation that states shorten training duration in an effort to increase credential attainment. In our December 2013 report, we found that in program year 2011, of those in the Adult Program who entered training, 75 percent spent 1 year or less receiving training services, while 25 percent spent more time. Similarly, for the Dislocated Worker Program, 65 percent of training participants spent 1 year or less receiving training services. According to officials from one local workforce investment board we contacted, all training programs offered through their training providers must lead to a credential and must be completed in 12 months or less.

Little Is Known about the Extent to Which Participants Obtained Training-Related Employment

We found DOL’s data on training-related employment unreliable for our purposes based on our analysis of the data, an Office of Inspector General report, and data quality reports. We were not able to determine how many participants in the Adult or Dislocated Worker Programs obtained employment related to their training in program year 2011. For the Adult Program, we found that states reported data on 48 percent of training participants, but had missing data for the remaining 52 percent. For the Dislocated Worker Program, states reported data on 74 percent of training participants, but had missing data for the remaining 26 percent. Further, our analysis of the reported data showed wide variation among states regarding the percentage of participants who obtained training- related employment, raising questions about the data’s reliability (see table 2). Our findings are consistent with a September 2011 Office of Inspector General report, which found that DOL’s data on training-related employment were “incomplete and unreliable.” Specifically, the Inspector General reported that 5 of the 53 state workforce agencies it reviewed did not report any data and 12 state workforce agencies reported unreasonably high or abnormally low data on training-related employment. Further, DOL’s WIA data quality report for the third quarter of program year 2011 raised questions about training-related employment data for 26 states. For example, some states reported that none of their training participants secured training-related employment.

Selected States Cited Some Obstacles to Collecting Data on Credential Attainment and DOL and States Have Taken Steps to Address Challenges and Improve These Data

Manual Process of Collecting Data on Credentials Can Be Resource-Intensive

Workforce officials we interviewed in four of six states said that collecting data on credential attainment can be resource-intensive primarily because it requires manually tracking the information. Unlike employment-related outcomes—which states can generally obtain through the state’s UI wage record system—credentials are not generally recorded in a central, automated data system. As a result, case managers must manually collect this information from various sources including participants, training providers, and third-party credentialing organizations. DOL also requires documentation of credential attainment with a copy of a diploma, transcript, or other approved record in the participant’s case file. DOL monitors this requirement through its data validation process. The process of collecting and verifying a participant’s credential attainment generally entails one or more of the following steps: Contacting Participants. Workforce officials in most states we reviewed said they generally begin their efforts to determine credential attainment by attempting to contact training participants, though some are unresponsive or inaccessible. Several local officials noted that they use a variety of means, including phone, mail, email, and social media. Some training participants readily provide evidence of their credentials. For example, local officials in two states estimated that for about 70 percent of participants, credentials are fairly easy to verify. Other participants may be less responsive. Workforce officials in three states explained that participants who have already exited the program have little incentive to respond to their requests. Local officials in two states also noted that some participants relocate without providing updated contact information.

Contacting Training Providers. Training providers are another potential source of credential information, though in some cases they may decline to share such information. If case managers cannot reach a participant, they generally contact the training providers to determine whether a credential was earned. However, workforce officials from three states noted that training providers often declined to provide this information, citing student privacy rights such as those established by the Family Educational Rights and Privacy Act of 1974, as amended (FERPA).

Contacting Third-party Organizations. Third-party credentialing organizations represent an additional source of credential information. For some occupations, a license or certification is required before a person can be employed in that capacity, such as a licensed practical nurse. In these cases, third-party organizations, such as state regulatory bodies, issue credentials. Case managers can sometimes search licensing databases online to confirm credential attainment. Local officials from two states noted that such data are fairly easy to obtain because the information is generally centrally accessible. However, an official from another state said that third-party organizations do not always provide information on credential attainment before DOL’s reporting timeframes end.

Because case managers may not always be able to track down the documentation needed to verify credential attainment, the actual number of participants who attain credentials may be underreported to DOL. For example, officials in one state we contacted said they believe their credential attainment rate should be about 65 percent, but the rate they actually report is about half of that.

Despite such obstacles to verifying data on participant credentials, several workforce experts and officials noted the importance of collecting this information. Workforce experts from one national organization noted that credential attainment can demonstrate the value of the funds invested in training and show employers the value of workforce programs and their participants. Workforce experts from another national organization said that credential data could help officials determine which credentials are best aligned with good employment outcomes. Some employer groups also noted the value of credentials in some high- demand occupations, such as manufacturing and information technology. For example, representatives of employer groups in Illinois and Rhode Island said they value information technology and manufacturing credentials from certifying organizations because these programs prepare individuals to perform high-skill tasks.

To Address Challenges, DOL Has Stressed Both Credential Attainment and Reporting

In 2010, DOL provided guidance to states to increase the quantity and quality of credentials attained and to clarify the definition of credential for reporting purposes. During early WIA implementation, we reported that the definition of credential varied within and across states. For example, some states strictly defined credential as a diploma from an accredited institution, and other states broadly defined credential to include certificates of job readiness or completion of a workshop. DOL issued guidance in 2006 that provided additional clarification on which credentials to report, but, according to some workforce officials and experts, allowed for some interpretation.that defined “credential” as an umbrella term that can include a range of In 2010, DOL issued guidance postsecondary degrees, diplomas, licenses, certificates, and certifications. DOL also clarified that credentials must show attainment of measurable technical or occupational skills necessary to obtain employment or advance within an occupation. For this reason, DOL specified that credentials related to remedial training, such as work- readiness certificates, would not be counted for the purposes of credential attainment.

In addition to clarifying which credentials should be reported, the 2010 guidance also included strategies that state and local officials can use to increase the quantity and quality of credentials attained. It noted that the first step in increasing the quantity of credentials attained is to refer more participants to training. DOL’s guidance also encouraged officials to take steps to ensure that the training programs result in an industry-recognized credential and that participants complete these training programs. These steps include shortening the duration of training and providing supportive services that enable participants to succeed. Further, to improve the quality of credentials attained, DOL suggested that state and local agencies build the capacity of front-line staff to identify and assess valuable and appropriate credentials for participants.

DOL has also stressed the importance of credential attainment by measuring it through an agency-wide performance goal for its workforce development programs including the WIA Adult and Dislocated Worker Programs. DOL officials noted that credential attainment rates for these WIA programs are higher than the rates of some other DOL programs included in the agency-wide performance goal. DOL first began tracking credential attainment data for its agency-wide performance goal in 2010 when it set out to increase the number of training participants who attain credentials through any one of multiple federal workforce programs. Specifically, the goal was an increase of 10 percent, up to a total of 220,000 training participants earning credentials. In fiscal year 2013, DOL continued to assess credential attainment through this performance goal and sought to increase the percentage of training participants who earn credentials from 57 to 62 percent.attainment rate of 59.4 percent through the first two quarters of fiscal year 2013. DOL officials also said that DOL has established a new credential attainment goal; specifically, that by September 30, 2015, the percentage of training participants who attain credentials will increase by 10 percent from the level reported as of the end of fiscal year 2013.

DOL officials reported a credential In addition to issuing guidance and setting credential attainment goals, DOL also undertook a number of other related initiatives, including some that are specific to credentials and others that are more broadly designed, such as the Workforce Data Quality Initiative. See Table 3 for a description of DOL’s initiatives.

Selected States and Local Areas Have Taken Steps to Address Challenges by Enhancing Attainment and Reporting of Credentials

Some states have stressed the importance of credential attainment by implementing broad, statewide efforts. Similar to DOL’s efforts to enhance credential attainment by establishing annual goals, three of the six states we reviewed have either implemented statewide credential attainment goals or are working to do so:

Texas implemented an annual state performance measure on educational achievement that tracks credential attainment for multiple programs, including the WIA Adult and Dislocated Worker Programs. All WIBs in the state are held to this measure.

Washington has made credential attainment a state performance measure, but defines credential more broadly than DOL. For example, Washington recognizes a larger range of credentials, such as completion of on-the-job training.

Illinois re-implemented a credential attainment performance measure during program year 2012 and, according to state officials, is in the process of setting credential targets for program year 2014.

According to officials from some states, their efforts to emphasize credential attainment and reporting may have a positive impact on participants’ reported rate of credential attainment.

Moreover, officials in Alabama, Illinois, Kansas, and Rhode Island told us they targeted their training funds more narrowly on credential-yielding programs by only approving training providers with programs that resulted in credentials that met DOL’s definition. For one Chicago WIB, this strategy, along with its other efforts to streamline training options from 753 occupations to 40 in-demand occupations, reduced its number of training providers. This practice was one of many DOL suggested in its 2010 guidance as a means for states and local areas to improve the value of credentials for participants. Officials in nearly every state we interviewed reported that this guidance was helpful largely because it more clearly defined which credentials should be reported to DOL.

In addition, selected states and local areas have taken steps to ease the resource-intensive process of collecting data on credentials by enhancing communication with participants and working to overcome privacy issues with training providers. Workforce officials in three states told us that case managers seek to build rapport with participants early in the process so they are more likely to be responsive after their training program ends. Regarding training providers, officials in four of six states said they have made efforts to address privacy concerns. Local officials in Alabama, Kansas, and Texas, for example, told us that they ask participants to sign consent forms to allow training providers to share credential information with officials. In Washington, state officials access some credential data from the National Student Clearinghouse and from their state database of community and technical colleges. They said that student privacy rights are generally not a barrier to accessing credential data in Washington because students attending their community and technical colleges are notified that such information can be released to other entities unless the student opts out of sharing it. Washington state officials noted that they have been refining their process for collecting data on credential attainment for 15 years and now have a fairly sophisticated approach.

While these varied efforts to mitigate challenges may help reduce the resources required or improve the quality of reported data, workforce officials from three states and three experts we interviewed raised some additional considerations about measuring performance on credential attainment (see table 4). Establishing a performance measure on credential attainment may affect the type of training provided and which participants receive training. For example, neither work readiness training nor on-the-job training (OJT) leads to what DOL has defined as a credential for reporting purposes. However, these may be the most appropriate types of training for participants with basic skills or for particular industries, according to officials from two states we interviewed. Our December 2013 report found that participants in the WIA Adult and Dislocated Worker Programs often lacked the relevant qualifications and basic skills needed to participate in training that would meet the needs of employers seeking employees for in-demand occupations. In addition, work readiness certificates are generally valued by employers, according to several employer group representatives and local workforce officials we interviewed. Representatives from a few employer groups also noted that, in some cases, experience is more important than credentials. For example, local officials we interviewed in Illinois said that the vast majority of participants in OJT obtained jobs with the employers once their training was completed. The officials said OJT provided a good return on investment, despite the fact that these participants did not earn credentials. Currently, DOL’s credential attainment data do not include participants who completed these types of training programs. If credential attainment is established as a performance measure, it will be important to consider ways to address participants who are enrolled in certain types of training that do not lead to a credential, such as by excluding these participants from a credential attainment measure or considering if other measures, such as basic skill attainment, could capture the value of training provided to participants excluded from the measure.

States Face Continuing Challenges Reporting Data on Training- Related Employment, Despite Efforts by States and DOL to Improve Reporting

Reporting Data on Training-Related Employment Is Resource- Intensive and Involves Subjective Decisions

Workforce officials in most states we studied identified challenges reporting data on training-related employment that were greater than those for reporting data on credential attainment, including the high degree of resources required and the subjective nature of determining whether employment is linked to training. Similar to credential attainment, there is no definitive source for these data, so case managers must generally collect participants’ employment information from various sources, including participants, employers, and UI wage records. Then— in a step beyond what is required for reporting on credential attainment— they must piece this information together to determine whether participants’ employment is substantially related to their training. Also unlike reporting on credential attainment, DOL does not require that local WIA programs collect and retain documentation on training-related employment in the participants’ case files to verify the accuracy of data they report to states.

Officials in four of the six states we studied, as well as at DOL, said this data collection process often requires considerable time and effort. Further, officials from DOL and four states emphasized the need to consider the balance between the time required to collect outcome data and the time case managers spend serving participants, especially in an environment of reduced resources. In addition, officials in all six states said that making training-related employment determinations can be subjective. According to DOL’s reporting guidance, participants’ employment is related to their training if it uses “a substantial portion of the skills taught in training.” workforce experts from one national organization, said that one case manager’s interpretation of what constitutes a substantial portion of the skills obtained in training may differ from another’s. The training-related employment decision can be straightforward if the training and job are clearly connected. For example, if a participant received training to attain a commercial driver’s license and was subsequently hired as a driver by a trucking company, the case manager can easily determine that the participant’s employment is substantially related to the training received. In other cases, however, the decision may be more subjective. For example, officials in one state could not agree whether a participant who had received aviation instruction training had secured training-related employment in his position as an airframe and power plant mechanic. Some state officials thought the skills obtained were transferable, but others were unsure.

U.S. Department of Labor, Training and Employment Guidance Letter No. 17-09: Quarterly Submission of Workforce Investment Act Standardized Record Data (WIASRD) (March 10, 2010).

Collecting participants’ employment information and attempting to determine whether it was training-related generally entails several steps (see fig. 5).

Contacting participants. Several state and local officials we interviewed said that they generally begin the process of collecting data on training-related employment by attempting to contact participants, though some can be unresponsive or inaccessible, which workforce experts from one national organization noted as well. If case managers are successful, they ask participants for information such as the name of their employer and their job title. Some local areas also ask participants directly if their new jobs are—in the participants’ opinion—related to the training they received. In some cases, case managers make their training-related employment determinations based solely on information the participants provide about their employment.

Contacting employers. Some case managers contact employers to obtain participants’ employment information, though employers may not be responsive. If case managers could not reach a participant but know where the person works, they may contact the employer to obtain their job title and description. They may also contact an employer to verify the information provided by a participant. Case managers can use the employment information obtained from an employer, or from both the participant and employer, to determine if a participant’s job is training-related. However, workforce officials in Illinois and Texas said employers may not be responsive because they are concerned about employee privacy or about the amount of follow-up required.

Checking UI wage records. Some local workforce officials said that if they are unable to gather information about a participant’s employment from the participant or the employer, they check the UI wage records, which are generally not available until several months after a participant exits from WIA services. DOL officials and workforce experts at two national organizations said the UI wage records generally provide the name of the participant’s employer and a code associated with the employer’s industry, but specific information on the participant’s occupation is rarely included. In some cases, the industry code has a clear connection to the training received, making the case manager’s training-related employment determination straightforward. For example, if a participant who was trained as a nurse was hired by a hospital, the case manager can reasonably assume that the employment and training are related. However, some officials noted that the industry code is not always a good predictor of a participant’s occupation. For example, if the same participant was hired by the health unit of a manufacturing company, the industry code in the UI wage records would suggest that the person’s job was associated with manufacturing and not related to the nursing training.

Other steps to determine training-related employment. If successful in accessing a participant’s employment information, including job title, some workforce officials said case managers use DOL’s Occupational Information Network (O*NET), which provides an online tool to match job title occupational codes to the skills code associated with the participant’s training. This can help case managers decide whether the participant’s employment is training- related.

According to some local workforce officials, however, it can be difficult to find the precise occupational code that matches the participant’s new job. In addition, DOL officials told us that even attempting to match O*NET codes in this manner might not help case managers determine if participants’ employment is related to their training because the threshold of relatedness is still subjective, as mentioned previously. DOL officials also told us it is difficult to prescribe a standard definition for determining whether a job is related to training because it often requires some judgment on the part of local officials.

States Have Taken Steps to Address Challenges by Improving Reporting on Training-Related Employment

To improve reporting, some states have taken steps to increase their access to information about participants’ employment. Similar to their efforts to collect information on credential attainment, some local workforce officials told us that case managers seek to build rapport with participants. At the same time, officials we interviewed in four of six states have taken steps to increase access to employment-related information. In Illinois, Rhode Island, and Texas, local workforce officials ask participants to sign release forms authorizing employers to release employment information to officials. Local workforce staff in Kansas and Texas also said they obtain information on participants’ employment, including the names of their employers, and—unlike UI wage records— their job titles, by subscribing to an online payroll database called The Work Number. This service verifies employment via a database of national payroll data but does not include all employers, and local WIA programs must pay to subscribe.

Officials said they conduct this survey for internal use to collect and share data on training-related employment that is not otherwise available to program managers and state officials. Another official noted that a significant percentage of respondents who do not report their job as training-related find the training instrumental in getting the job or that the skills they acquired in training are useful on their job.

In addition, local workforce officials in four states we contacted said they have developed strategies to help reduce the subjectivity in determining whether a participant’s employment is related to their training. For example, local officials in Texas told us that staff may consult their American Job Center’s local business services office, which often has specific knowledge about what skills correspond with particular job titles. Officials in Texas and Alabama also said case managers may consult with their peers or supervisors to reach consensus about a training- related employment decision. Further, state workforce officials in Kansas told us that when employers post jobs on the state’s job bank, they are required to enter occupational codes from DOL’s online O*NET database. If a training participant gets one of these jobs, case managers can compare the employer-provided occupational code with the training codes supplied by training providers to help them determine whether the job is training-related. State and local officials in Washington also said managers routinely use O*NET when making their training-related employment decisions.

DOL Has Taken Limited Steps to Address Challenges and Improve Data on Training-Related Employment

While DOL has recently issued guidance aimed at increasing reporting rates for training-related employment data, it has taken limited steps to address states’ ongoing reporting challenges. As previously discussed, a September 2011 report by DOL’s Office of Inspector General raised questions about the quality of these data and made recommendations to address this issue. In response, DOL issued a notice to states in September 2013 that reiterated the requirement for states to report these data and stressed the data’s importance for program analysis and evaluation efforts pertaining to the value of investments in WIA training. The DOL notice also acknowledged that reporting on training-related employment is challenging largely because the information must be collected manually. According to the DOL notice, nearly every state that participated in conference calls on the topic indicated that there was considerable cost in conducting the necessary follow-up for reporting on training-related employment and that this was the primary reason the data were not well-reported. State and local officials we interviewed also noted that such data may be underreported because of the difficulty of following up with participants and employers. DOL also concluded that states with larger training caseloads had less complete reporting on training-related employment. In the notice, DOL also described a few practices most common among states with higher reporting rates for training-related employment. For example, DOL cited the practice of instituting a data system check to ensure that training-related employment data are recorded before closing a participant’s case file.

Use of crosswalk not necessarily a solution for determining training-related employment Another program DOL administers, Job Corps—a residential, educational, and career technical training program for disadvantaged youths—uses a crosswalk that links specific training codes and occupational codes to help staff determine training-related employment. However, in a September 2011 report, the DOL Inspector General found that this crosswalk included some matches that were either not related or poorly related. Moreover, DOL officials said that using a crosswalk for the WIA Adult and Dislocated Worker programs could make the training- employment link too restrictive and would require a considerable amount of resources to develop.

DOL has not identified and disseminated strategies for increasing access to employment-related data or helping to minimize the subjectivity of training-related determinations, but instead has focused exclusively on increasing data reporting rates. While DOL officials maintain that manual follow-up with participants is the best approach for obtaining employment related data, they also recognize that such data collection is resource- intensive. In addition, as officials in all six of our selected states also noted, DOL officials acknowledged that determining whether a participant’s employment is training-related can be a subjective decision. They noted challenges in defining training-related employment more precisely. For example, officials said some training is intended to develop broad, nonspecific skills that may help participants get jobs but are not associated with a specific occupation or industry. We recognize that utilizing professional judgment is inherent in certain tasks such as determining whether a participant’s employment is related to the training the participant received. However, minimizing the amount of subjective decision-making involved to the extent possible could help ensure better quality data on training-related employment. Reasonable approaches for improving the quality of performance data focus on aspects of completeness, accuracy, consistency, and ease of use. By identifying and sharing with states practices to increase access to employment- related data and reduce the subjectivity of some determinations, DOL could help states improve their reporting of data on training-related employment.

In addition to the strategies all six selected states use to mitigate reporting challenges, workforce officials from three states and workforce experts at two national organizations said some additional considerations should be taken into account in weighing a performance measure on training-related employment (see table 5). We previously noted concerns about the level of resources required and the subjectivity of determinations, both of which could affect the data’s completeness and consistency—key aspects of performance data quality. The state and local strategies we identified may help mitigate some reporting challenges. In addition, a participant’s successful placement in a training- related job depends on both the ability and decision to pursue such a position. Some local workforce officials we contacted said that it may take participants longer to find employment in their field of training than is allowed for reporting purposes. For example, workforce officials in Rhode Island said that due to the poor economy in their state, it is not uncommon for some participants to take 2 years or more to find a job. Additionally, workforce officials in three states we contacted said participants may decide to take a job unrelated to their training if it is the only job they can find or if they simply choose not to pursue a job in the field in which they were trained.

Conclusions

To ensure that public funds invested in WIA’s Adult and Dislocated Worker Programs are spent wisely, program managers and policymakers need performance data that are accessible, complete, accurate and consistent. The current common performance measures—employment, retention, and earnings—provide a basis for assessing the overall value of the services the programs provide, primarily using a standardized data source (state UI wage records). Beyond these, data on outcomes such as credential attainment and training-related employment can potentially provide information more specifically on the value of training services. However, as we have noted, collecting data on these outcomes can be more resource-intensive, in part because there is no single readily available source of data.

DOL has taken steps to elevate the importance of credential attainment and improve data quality for this outcome. We found credential attainment data reported to DOL to be reliable. In contrast, we found the data reported to DOL on training-related employment to be incomplete and inconsistent. While DOL has acknowledged challenges in collecting data on and determining training-related employment, it has taken only limited steps to address these challenges, focusing efforts exclusively on improving reporting rates. This effort alone will not improve the quality of the data being reported. Given the nature of the challenges we identified, there are no easy or complete solutions. However, we also identified strategies some states use that may help increase access to employment information and reduce the subjectivity of some training-related determinations. Sharing such strategies with other states, as well as identifying and communicating other approaches, could lead to incremental improvement in the quality of data reported. Without such action, the data states are required to report on training-related employment are likely to remain unusable.

Recommendation for Executive Action

To provide policymakers and program managers with better quality information to assess the value of training provided by WIA’s Adult and Dislocated Worker Programs, we recommend that the Secretary of Labor identify and share with states strategies for collecting and reporting data on training-related employment that could either increase access to employment information or reduce the subjectivity of determining when training is related to employment.

Agency Comments and Our Evaluation

We provided a draft of this report to the Secretary of Labor and selected draft sections to the Secretary of Education. DOL and Education provided technical comments, which we incorporated as appropriate, and DOL provided a written response (see app. II). DOL agreed with our recommendation and noted that having reliable data on training-related employment is important to effectively manage and evaluate the Adult and Dislocated Worker Programs. DOL also agreed that states can benefit from learning what other states are doing to address challenges regarding access to and subjectivity of these data. Toward this end, DOL noted that it plans to conduct additional conference calls with state officials to reiterate the importance of identifying training-related employment and continue to discuss and share best practices to improve these data. DOL noted that this sharing of best practices would supplement actions it has already taken to improve data on training- related employment. These actions include coding changes to the WIASRD to capture additional information, conference calls with state workforce officials to discuss reporting on training-related employment, and a work group considering adding more data elements to the UI wage records such as an occupational code.

As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Labor, the Secretary of Education, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or sherrilla@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who make key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives for this study on the Workforce Investment Act’s (WIA) Adult and Dislocated Worker Programs were to examine: 1) the extent to which training participants obtain credentials and secure training-related employment, 2) challenges states face reporting on credential attainment and what steps, if any, they and DOL are taking to address them, 3) challenges states face in reporting on training-related employment and what steps, if any, they and DOL are taking to address them.

To address these objectives, we reviewed relevant federal laws, regulations, and DOL’s guidance to states for reporting select data on participants in the WIA Adult and Dislocated Worker Programs. We interviewed DOL officials from the Employment and Training Administration and the Office of Inspector General as well as experts on workforce issues (see Selection of Experts, below). We also interviewed state and local workforce officials as well as organizations that provided an employer perspective from a nongeneralizeable sample of six states (see Selection of States and Local Areas, below). To assess the reliability of the data DOL collects on credential attainment and training-related employment, we analyzed data from the Workforce Investment Act Standardized Record Data (WIASRD) system for program year 2010 and program year 2011—the most recent years for which data were available—by testing the data electronically and interviewing knowledgeable agency officials. We found the data to be sufficiently reliable for providing estimates on: 1) participants who received training, 2) the minimum number of training participants who earned a credential, and 3) the types of credentials they earned. However, we determined that the data on training-related employment were not reliable for the purposes of our report (see Analysis of DOL’s Training Outcome Data below).

Selection of Experts

We interviewed experts on workforce issues representing six organizations. We identified experts by first reviewing relevant literature and asking officials from DOL for recommendations. We further developed the list by asking each expert we interviewed for additional names. Only experts who were mentioned more than once were selected. The results of these interviews are not generalizeable, but provided insights about the overall challenges states face in reporting on these outcomes and any efforts by states to overcome them.

Selection of States and Local Areas

We interviewed state and local workforce officials from a nongeneralizeable sample of six states. We conducted in-person interviews with officials in Alabama, Illinois, and Texas and telephone interviews with officials in Kansas, Rhode Island, and Washington (see table 4). These results are not generalizeable, but provided insights about the challenges they face in reporting data on credential attainment and training-related employment as well as any steps they are taking to address those challenges. We selected the states to reflect a mix of those that had leading-edge data practices (as identified by experts) and those with either incomplete data or relatively high rates of reporting errors (as identified by WIA data quality reports on credential attainment). We also factored geographic diversity into state selection. In each state, we interviewed state workforce officials and also local workforce officials from at least one workforce investment board and at least one American Job Center—formerly known as a one-stop center. We selected a non- generalizeable sample of local areas based on input from state workforce officials and, for states we visited, proximity to the state workforce agency. In addition, we interviewed at least one employer organization in each state. In selecting these entities, we considered states’ input on organizations that could provide us with an employer perspective on the value of credentials and certain types of training for various industries. They included statewide business associations, regional business associations, individual employers, and industry-specific representatives.

In each state, we obtained information about the state and local area’s process of collecting data on credentials and employment related to training as well as any challenges they may have encountered. We also asked state officials about DOL’s related guidance. We used a semi- structured interview guide for the state and local interviews.

Analysis of DOL’s Training Outcome Data

To assess the reliability of DOL’s data on training, credential attainment, and training-related employment in the WIASRD database for participants in the WIA Adult and Dislocated Worker programs, we: (1) reviewed documentation related to reporting these data, including DOL’s Office of Inspector General reports; (2) tested the data electronically to identify potential problems with consistency, completeness, or accuracy; and (3) interviewed knowledgeable DOL officials about the data. Our electronic testing consisted of identifying inconsistencies, outliers, and missing values. In addition, we analyzed the publicly-available WIASRD data file for program years 2010 and 2011, which was produced for DOL by its data contractor, Social Policy Research Associates. As part of our analysis, we reviewed the steps the data contractor took to address data errors and, to the extent possible, compared the data DOL provided for our analysis to the publicly-available file, and found only slight discrepancies. We found the data on training and credential attainment to be sufficiently reliable for reporting estimates of: (1) participants who received training, (2) the minimum number of training participants who earned a credential, and (3) the types of credentials they attained. We were not able to reliably make state-to-state comparisons because two states are piloting a new reporting format for DOL, and therefore would not have been compatible with the others. For the purposes of this report, we did not find the data on training-related employment reliable. We reached this conclusion based on our analysis of the data, an Office of Inspector General report, and DOL’s data quality reports. We were not able to determine how many training participants in the Adult or Dislocated Worker Programs obtained employment related to their training in program year 2011. For the Adult Program, we found that states reported data on 48 percent of training participants, but had missing data for the remaining 52 percent. For the Dislocated Worker Program, states reported data on 74 percent of training participants, but had missing data for the remaining 26 percent. Further, an analysis of the reported data showed wide variation among states regarding the percentage of participants who obtained training-related employment raising questions about the data’s reliability.

We conducted this performance audit from October 2012 to January 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit work to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Labor

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact:

Andrew Sherrill, (202) 512-7215 or sherrilla@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Laura Heald, Assistant Director, John Lack, Jeffrey G. Miller, and Kathryn O’Dea Lamas made key contributions to this report. Also contributing to this report were James Bennett, Jessica Botsford, David Chrisinger, Elizabeth Curda, Kathy Leslie, Carol Patey, Rhiannon Patterson, Jerry Sandau, and Walter Vance.

Related GAO Products

Workforce Investment Act: Local Areas Face Challenges Helping Employers Fill Some Types of Skilled Jobs. GAO-14-19. Washington, D.C.: December 2, 2013.

Workforce Investment Act: DOL Should Do More to Improve the Quality of Participant Data. GAO-14-4. Washington, D.C.: December 2, 2013.

Postsecondary Education: Many States Collect Graduates’ Employment Information, but Clearer Guidance on Student Privacy Requirements Is Needed. GAO-10-927. Washington, D.C.: September 27, 2010.

Workforce Investment Act: Employers Found One-Stop Centers Useful in Hiring Low-Skilled Workers; Performance Information Could Help Gauge Employer Involvement. GAO-07-167. Washington, D.C.: December 22, 2006.

Workforce Investment Act: Labor and States Have Taken Actions to Improve Data Quality, but Additional Steps Are Needed. GAO-06-82. Washington, D.C.: November 14, 2005.

Workforce Investment Act: Substantial Funds Are Used for Training, but Little Is Known Nationally about Training Outcomes. GAO-05-650. Washington, D.C.: June 29, 2005.

Workforce Investment Act: Labor Should Consider Alternative Approaches to Implement New Performance and Reporting Requirements. GAO-05-539. Washington, D.C.: May 27, 2005.

Workforce Investment Act: States and Local Areas Have Developed Strategies to Assess Performance, but Labor Could Do More to Help. GAO-04-657. Washington, D.C.: June 1, 2004.

Workforce Investment Act: Labor Actions Can Help States Improve Quality of Performance Outcome Data and Delivery of Youth Services. GAO-04-308. Washington, D.C.: February 23, 2004.

Workforce Investment Act: Improvements Needed in Performance Measures to Provide a More Accurate Picture of WIA’s Effectiveness. GAO-02-275. Washington, D.C.: February 1, 2002.