Background

		WIA Programs and Administrative Structure

The WIA Adult and Dislocated Worker Programs provide employment  services to a wide range of participants. The Adult Program serves all  individuals age 18 and older, and the Dislocated Worker Program serves  individuals who have been or will be terminated or laid off from  employment, among others. The Adult Program prioritizes certain  services for recipients of public assistance and other low-income  individuals when program funds are limited. To enable individuals to  participate, both programs may offer supportive services such as  transportation, childcare, housing, and needs-related payments under  certain circumstances. WIA requires that the Adult and Dislocated  Worker Programs and other federally-funded employment and training  programs provide services through one-stop centers—now called  American Job Centers—so that jobseekers and employers can find  assistance at a single location.
DOL’s Employment and Training Administration administers the Adult and  Dislocated Worker Programs and oversees their implementation, which is  carried out by states and local areas. At the state level, the WIA Adult and  Dislocated Worker Programs are administered by state workforce  agencies. Each state has one or more local workforce investment areas,  each governed by a WIB. WIBs select the entities that will operate  American Job Centers, which provide most WIA services, and oversee  the American Job Center network. WIA provides substantial flexibility to  states and WIBs to determine how services are provided.

		Employment and Training Services Provided under WIA

WIA represented a fundamental shift from its predecessor program, the  Job Training Partnership Act, by decreasing the focus on job training as  the primary means to help adults and dislocated workers get a job. The  Adult and Dislocated Worker Programs provide participants three types of  services:
Core services include basic services such as job searches and labor  market information, and may be accessed with or without staff  assistance.
Intensive services include such activities as comprehensive  assessment and case management, which require greater staff  involvement. Intensive services are available to participants who are  unable to obtain or retain employment after receiving at least one core  service.
Training services include such activities as occupational skills or on- the-job training. To be eligible for training services, participants must:  (1) be unable to obtain or retain employment after receiving at least  one intensive service, (2) be in need of training, and (3) have the skills  and qualifications to successfully complete the training program,  among other requirements.
To assess participants’ skills and determine whether they need training,  WIBs may require them to complete certain activities. We previously  found that most WIBs required participants to complete skills  assessments or gather information about the occupation for which they  wanted training before entering a training program.
DOL requires that participants eligible for training select approved training  providers in consultation with case managers, but participants ultimately  choose the training programs in which they participate. that training be directly linked to in-demand occupations, which DOL  interprets to include both currently-available jobs and occupations that are  projected to grow in the future.

		Funding, Performance Measures, and Data Quality

20 C.F.R. § 663.440(c). financial sanctions and incentive funding. In addition, WIA requires  states to use Unemployment Insurance (UI) wage records to track  employment-related outcomes.
In 2005, DOL began allowing states to request waivers to replace the  WIA performance measures with a smaller set of common measures that  focus on employment, retention, and earnings across multiple programs  (see table 1). These measures do not include credential attainment and  are calculated differently than the WIA measures.
As of December 2013, a total of 48 states and territories, plus the District  of Columbia, had obtained this waiver and used the common measures  for the Adult and Dislocated Worker Programs. Although states and  territories are no longer subject to financial sanctions or incentive funding  for the credential attainment measure, DOL still requires them to report  the number of training participants who earn credentials.credential attainment improves workers’ labor market experience through   In general,  higher earnings, greater mobility, and enhanced job security, according to  DOL and research studies.
We have previously raised concerns about the accuracy and  comparability of DOL’s data on credential attainment because DOL’s  guidance allowed states and local areas considerable flexibility in defining  what constitutes a credential.  In February 2002, we recommended that  DOL more clearly define this term. Since that time, DOL issued guidance  to clarify its definition of credential attainment. In addition, to verify that  the data states report on credential attainment are accurate, DOL  requires WIA programs to collect and retain documentation on  participants’ credential attainment.
In addition to the performance measures, WIA requires states to report a  wide array of data that includes whether training participants find  employment that relates to their training. A WIA participant’s  employment is considered to be related to the training received if in the  new job the participant uses “a substantial portion of the skills taught in  the training.” However, DOL does not require that states collect and  retain documentation for employment related to training to verify the  accuracy of data they report. In a September 2011 report, DOL’s Office of  Inspector General raised questions about the quality of DOL’s data on  employment related to training. Specifically, the Office of Inspector  General found that these data were “incomplete and unreliable” based on  its review of the data DOL maintains in its WIA database. The Office of  Inspector General recommended, among other things, that DOL provide  guidance on the best methodology for reporting such data as well as  provide oversight to ensure that states develop or identify best practices  for increasing the rate of training-related employment. DOL agreed with  these recommendations.
Federal agencies that work in partnership with states and local areas to  administer programs such as WIA must continually balance the  competing objectives of collecting uniform performance data and giving  program managers flexibility to meet local needs. Our prior work has  found that federal agencies have considered key attributes of data quality  for performance data, including:
Completeness—the extent to which enough of the required data  elements are collected from a sufficient portion of the target  population or sample.
Accuracy—the extent to which the data are free from significant  errors.
Consistency—the extent to which data are collected using the same  procedures and definitions across collectors and times.
Ease of use—how readily intended users can access data, aided by  clear definitions, user-friendly software, and easy-to-use access  procedures.

		Congressional Proposals to Reauthorize WIA Programs

Congress is currently considering legislation to reauthorize the Workforce  Investment Act of 1998, which has been due for reauthorization since the  end of fiscal year 2003. The Supporting Knowledge and Investing in  Lifelong Skills (SKILLS) Act (H.R. 803), passed by the House, would  establish both credential attainment and training-related employment as  performance measures. The Workforce Investment Act of 2013 (S. 1356),  introduced in the Senate and reported out by the Committee on Health,  Education, Labor, and Pensions, would establish credential attainment as  a performance measure, but not training-related employment.

	DOL’s Data Show That Most Training Participants Attained Credentials, but Data on Training-Related Employment Are Unreliable

		Most Training Participants Attained Credentials

During the time-period from program year 2006 through program year  2011, the total number of participants in WIA’s Adult and Dislocated   Specifically, in the Adult  Worker Programs increased significantly.Program, the number of participants increased from about 625,000 to  about 1.25 million, and in the Dislocated Worker Program, the number of  participants increased from about 272,000 to about 761,000. During this  same time-period, the number of participants who received training  services also increased, but not as dramatically as the number of overall   For example, the number of Dislocated Worker  participants (see fig. 1).participants who received training increased from about 76,000 in  program year 2006 to about 120,000 in program year 2011.
Since the number of participants who received training services did not  increase at the same rate as the number of participants who entered into  WIA’s Adult and Dislocated Worker Programs, the percentage of  participants who received training generally declined, according to DOL  reports (see fig. 2).
There are several reasons that may have contributed to the declining  percentage of participants who received training. Beginning in December  2007, the significant increase in the overall number of participants is likely  attributed to the downturn in the economy that led to a dramatic rise in  unemployment and the subsequent infusion of additional funds to WIA  programs from the American Recovery and Reinvestment Act of 2009  (Recovery Act). The Recovery Act provided $500 million for grants for  the Adult Program and $1.25 billion for grants for the Dislocated Worker  Program. By spring of 2009, DOL began allocating these funds to states  to supplement existing WIA funds. DOL encouraged states to use  Recovery Act funds to increase training in an effort to help Americans  acquire new skills and return to work. DOL officials stated that the  increase in demand for training services exceeded the increase in supply  provided for by the Recovery Act, which may have led to shortages in  training capacity that contributed to a relative decline in training. Another  factor, according to DOL officials, is that some program participants had  limited access to needs-based financial assistance and other supportive  services, such as child care, which may have prevented them from  entering WIA training services. Further, DOL noted that the preference of  many program participants is for immediate employment rather than job  training.
Similarly, the percentage of training participants who earned credentials  has also generally declined from program year 2006 through program  year 2011 (see fig. 3). For example, DOL’s data show the percentage of  those who earned credentials in the Adult Program dropped from about  74 percent in program year 2006 to about 58 percent in program year  2011. Likewise, the percentage of those in the Dislocated Worker  Program who earned credentials dropped from about 75 percent in  program year 2006 to about 63 percent in program year 2011. Despite  the decline in the percentage of training participants who earned  credentials, DOL reported that the total number of participants attaining  credentials increased during this time period. For example, the number of  training participants in the Dislocated Worker Program who attained  credentials increased from about 48,000 in program year 2006 to about  79,000 in program year 2011.
According to DOL officials, the percentage of training participants who  earned credentials may have declined in part as a result of changes in the  performance measures that states negotiated. In program year 2005,  states began requesting waivers to replace the WIA performance  measures with a set of common performance measures that do not  include the employment and credential attainment measure for the Adult   Officials in three of the six states we  and Dislocated Worker Programs.reviewed said that after this request was approved, reporting data on  credential attainment became a lower priority for them. Officials in one of  these states also said they stopped collecting and reporting these data  until DOL issued clarifying guidance in 2010 emphasizing the importance  of credential attainment as a pathway to employment.
DOL’s data on credential attainment also show that participants in the  Dislocated Worker Program typically have higher credential attainment  rates than participants in the Adult Program. DOL officials explained that  the Dislocated Worker Program has a higher funding level that supports  more training, and that participants in this program generally have longer  work histories and more advanced education and so are more likely to  enter into training programs that lead to credentials. In contrast,  participants in the Adult Program are more likely to require training  focused on remedial education and job readiness which are less likely to  result in credentials as defined by DOL.
Of the training participants who attained credentials during program year  2011—approximately 89,000 in the Adult Program and 84,000 in the  Dislocated Worker Program— about two-thirds in each program earned  an occupational credential, such as a welding certificate or clinical  medical assistant certificate (see fig. 4). The next two most common  types of credentials attained by training participants were an occupational  skills license, such as a license in nursing, and an associate’s degree.  The fact that training participants attained occupational credentials at  higher rates than longer-term academic degrees is consistent with DOL’s  recommendation that states shorten training duration in an effort to  increase credential attainment. In our December 2013 report, we found  that in program year 2011, of those in the Adult Program who entered  training, 75 percent spent 1 year or less receiving training services, while  25 percent spent more time. Similarly, for the Dislocated Worker  Program, 65 percent of training participants spent 1 year or less receiving  training services. According to officials from one local workforce  investment board we contacted, all training programs offered through  their training providers must lead to a credential and must be completed  in 12 months or less.

		Little Is Known about the Extent to Which Participants Obtained Training-Related Employment

We found DOL’s data on training-related employment unreliable for our  purposes based on our analysis of the data, an Office of Inspector  General report, and data quality reports. We were not able to determine  how many participants in the Adult or Dislocated Worker Programs  obtained employment related to their training in program year 2011. For  the Adult Program, we found that states reported data on 48 percent of  training participants, but had missing data for the remaining 52 percent.  For the Dislocated Worker Program, states reported data on 74 percent of  training participants, but had missing data for the remaining 26 percent.  Further, our analysis of the reported data showed wide variation among  states regarding the percentage of participants who obtained training- related employment, raising questions about the data’s reliability (see  table 2). Our findings are consistent with a September 2011 Office of  Inspector General report, which found that DOL’s data on training-related  employment were “incomplete and unreliable.” Specifically, the  Inspector General reported that 5 of the 53 state workforce agencies it  reviewed did not report any data and 12 state workforce agencies  reported unreasonably high or abnormally low data on training-related  employment. Further, DOL’s WIA data quality report for the third quarter  of program year 2011 raised questions about training-related employment  data for 26 states. For example, some states reported that none of their  training participants secured training-related employment.

	Selected States Cited Some Obstacles to Collecting Data on Credential Attainment and DOL and States Have Taken Steps to Address Challenges and Improve These Data

		Manual Process of Collecting Data on Credentials Can Be Resource-Intensive

Workforce officials we interviewed in four of six states said that collecting  data on credential attainment can be resource-intensive primarily  because it requires manually tracking the information. Unlike  employment-related outcomes—which states can generally obtain  through the state’s UI wage record system—credentials are not generally  recorded in a central, automated data system. As a result, case  managers must manually collect this information from various sources  including participants, training providers, and third-party credentialing  organizations. DOL also requires documentation of credential attainment  with a copy of a diploma, transcript, or other approved record in the  participant’s case file. DOL monitors this requirement through its data  validation process. The process of collecting and verifying a participant’s  credential attainment generally entails one or more of the following steps:  Contacting Participants. Workforce officials in most states we  reviewed said they generally begin their efforts to determine credential  attainment by attempting to contact training participants, though some  are unresponsive or inaccessible. Several local officials noted that  they use a variety of means, including phone, mail, email, and social  media. Some training participants readily provide evidence of their  credentials. For example, local officials in two states estimated that for  about 70 percent of participants, credentials are fairly easy to verify.  Other participants may be less responsive. Workforce officials in three  states explained that participants who have already exited the  program have little incentive to respond to their requests. Local  officials in two states also noted that some participants relocate  without providing updated contact information.
Contacting Training Providers. Training providers are another  potential source of credential information, though in some cases they  may decline to share such information. If case managers cannot  reach a participant, they generally contact the training providers to  determine whether a credential was earned. However, workforce  officials from three states noted that training providers often declined  to provide this information, citing student privacy rights such as those  established by the Family Educational Rights and Privacy Act of 1974,  as amended (FERPA).
Contacting Third-party Organizations. Third-party credentialing  organizations represent an additional source of credential  information. For some occupations, a license or certification is  required before a person can be employed in that capacity, such as a  licensed practical nurse. In these cases, third-party organizations,  such as state regulatory bodies, issue credentials. Case managers  can sometimes search licensing databases online to confirm  credential attainment. Local officials from two states noted that such  data are fairly easy to obtain because the information is generally  centrally accessible. However, an official from another state said that  third-party organizations do not always provide information on  credential attainment before DOL’s reporting timeframes end.
Because case managers may not always be able to track down the  documentation needed to verify credential attainment, the actual number  of participants who attain credentials may be underreported to DOL. For  example, officials in one state we contacted said they believe their  credential attainment rate should be about 65 percent, but the rate they  actually report is about half of that.
Despite such obstacles to verifying data on participant credentials,  several workforce experts and officials noted the importance of collecting  this information. Workforce experts from one national organization noted  that credential attainment can demonstrate the value of the funds  invested in training and show employers the value of workforce programs  and their participants. Workforce experts from another national  organization said that credential data could help officials determine which  credentials are best aligned with good employment outcomes. Some  employer groups also noted the value of credentials in some high- demand occupations, such as manufacturing and information technology.  For example, representatives of employer groups in Illinois and Rhode  Island said they value information technology and manufacturing  credentials from certifying organizations because these programs prepare  individuals to perform high-skill tasks.

		To Address Challenges, DOL Has Stressed Both Credential Attainment and Reporting

In 2010, DOL provided guidance to states to increase the quantity and  quality of credentials attained and to clarify the definition of credential for  reporting purposes. During early WIA implementation, we reported that  the definition of credential varied within and across states. For example,  some states strictly defined credential as a diploma from an accredited  institution, and other states broadly defined credential to include  certificates of job readiness or completion of a workshop. DOL issued  guidance in 2006 that provided additional clarification on which  credentials to report, but, according to some workforce officials and  experts, allowed for some interpretation.that defined “credential” as an umbrella term that can include a range of   In 2010, DOL issued guidance  postsecondary degrees, diplomas, licenses, certificates, and  certifications. DOL also clarified that credentials must show attainment  of measurable technical or occupational skills necessary to obtain  employment or advance within an occupation. For this reason, DOL  specified that credentials related to remedial training, such as work- readiness certificates, would not be counted for the purposes of credential  attainment.
In addition to clarifying which credentials should be reported, the 2010  guidance also included strategies that state and local officials can use to  increase the quantity and quality of credentials attained. It noted that the  first step in increasing the quantity of credentials attained is to refer more  participants to training. DOL’s guidance also encouraged officials to take  steps to ensure that the training programs result in an industry-recognized  credential and that participants complete these training programs. These  steps include shortening the duration of training and providing supportive  services that enable participants to succeed. Further, to improve the  quality of credentials attained, DOL suggested that state and local  agencies build the capacity of front-line staff to identify and assess  valuable and appropriate credentials for participants.
DOL has also stressed the importance of credential attainment by  measuring it through an agency-wide performance goal for its workforce  development programs including the WIA Adult and Dislocated Worker  Programs. DOL officials noted that credential attainment rates for these  WIA programs are higher than the rates of some other DOL programs  included in the agency-wide performance goal. DOL first began tracking  credential attainment data for its agency-wide performance goal in 2010  when it set out to increase the number of training participants who attain  credentials through any one of multiple federal workforce programs.  Specifically, the goal was an increase of 10 percent, up to a total of  220,000 training participants earning credentials. In fiscal year 2013, DOL  continued to assess credential attainment through this performance goal  and sought to increase the percentage of training participants who earn  credentials from 57 to 62 percent.attainment rate of 59.4 percent through the first two quarters of fiscal year  2013. DOL officials also said that DOL has established a new credential  attainment goal; specifically, that by September 30, 2015, the percentage  of training participants who attain credentials will increase by 10 percent  from the level reported as of the end of fiscal year 2013.
DOL officials reported a credential  In addition to issuing guidance and setting credential attainment goals,  DOL also undertook a number of other related initiatives, including some  that are specific to credentials and others that are more broadly designed,  such as the Workforce Data Quality Initiative. See Table 3 for a  description of DOL’s initiatives.

		Selected States and Local Areas Have Taken Steps to Address Challenges by Enhancing Attainment and Reporting of Credentials

Some states have stressed the importance of credential attainment by  implementing broad, statewide efforts. Similar to DOL’s efforts to enhance  credential attainment by establishing annual goals, three of the six states  we reviewed have either implemented statewide credential attainment  goals or are working to do so:
Texas implemented an annual state performance measure on  educational achievement that tracks credential attainment for multiple  programs, including the WIA Adult and Dislocated Worker Programs.  All WIBs in the state are held to this measure.
Washington has made credential attainment a state performance  measure, but defines credential more broadly than DOL. For example,  Washington recognizes a larger range of credentials, such as  completion of on-the-job training.
Illinois re-implemented a credential attainment performance measure  during program year 2012 and, according to state officials, is in the  process of setting credential targets for program year 2014.
According to officials from some states, their efforts to emphasize  credential attainment and reporting may have a positive impact on  participants’ reported rate of credential attainment.
Moreover, officials in Alabama, Illinois, Kansas, and Rhode Island told us  they targeted their training funds more narrowly on credential-yielding  programs by only approving training providers with programs that resulted  in credentials that met DOL’s definition. For one Chicago WIB, this  strategy, along with its other efforts to streamline training options from  753 occupations to 40 in-demand occupations, reduced its number of  training providers. This practice was one of many DOL suggested in its  2010 guidance as a means for states and local areas to improve the  value of credentials for participants. Officials in nearly every state we  interviewed reported that this guidance was helpful largely because it  more clearly defined which credentials should be reported to DOL.
In addition, selected states and local areas have taken steps to ease the  resource-intensive process of collecting data on credentials by enhancing  communication with participants and working to overcome privacy issues  with training providers. Workforce officials in three states told us that case  managers seek to build rapport with participants early in the process so  they are more likely to be responsive after their training program ends.  Regarding training providers, officials in four of six states said they have  made efforts to address privacy concerns. Local officials in Alabama,  Kansas, and Texas, for example, told us that they ask participants to sign  consent forms to allow training providers to share credential information  with officials. In Washington, state officials access some credential data  from the National Student Clearinghouse and from their state database of  community and technical colleges. They said that student privacy rights  are generally not a barrier to accessing credential data in Washington  because students attending their community and technical colleges are  notified that such information can be released to other entities unless the  student opts out of sharing it. Washington state officials noted that they  have been refining their process for collecting data on credential  attainment for 15 years and now have a fairly sophisticated approach.
While these varied efforts to mitigate challenges may help reduce the  resources required or improve the quality of reported data, workforce  officials from three states and three experts we interviewed raised some  additional considerations about measuring performance on credential  attainment (see table 4). Establishing a performance measure on  credential attainment may affect the type of training provided and which  participants receive training. For example, neither work readiness training  nor on-the-job training (OJT) leads to what DOL has defined as a  credential for reporting purposes. However, these may be the most  appropriate types of training for participants with basic skills or for  particular industries, according to officials from two states we  interviewed. Our December 2013 report found that participants in the WIA Adult and Dislocated Worker Programs often lacked the relevant  qualifications and basic skills needed to participate in training that would  meet the needs of employers seeking employees for in-demand  occupations. In addition, work readiness certificates are generally  valued by employers, according to several employer group  representatives and local workforce officials we interviewed.  Representatives from a few employer groups also noted that, in some  cases, experience is more important than credentials. For example, local  officials we interviewed in Illinois said that the vast majority of participants  in OJT obtained jobs with the employers once their training was  completed. The officials said OJT provided a good return on investment,  despite the fact that these participants did not earn credentials.  Currently, DOL’s credential attainment data do not include participants  who completed these types of training programs. If credential attainment  is established as a performance measure, it will be important to consider  ways to address participants who are enrolled in certain types of training  that do not lead to a credential, such as by excluding these participants  from a credential attainment measure or considering if other measures,  such as basic skill attainment, could capture the value of training provided  to participants excluded from the measure.

	States Face Continuing Challenges Reporting Data on Training- Related Employment, Despite Efforts by States and DOL to Improve Reporting

		Reporting Data on Training-Related Employment Is Resource- Intensive and Involves Subjective Decisions

Workforce officials in most states we studied identified challenges  reporting data on training-related employment that were greater than  those for reporting data on credential attainment, including the high  degree of resources required and the subjective nature of determining  whether employment is linked to training. Similar to credential attainment,  there is no definitive source for these data, so case managers must  generally collect participants’ employment information from various  sources, including participants, employers, and UI wage records. Then— in a step beyond what is required for reporting on credential attainment— they must piece this information together to determine whether  participants’ employment is substantially related to their training. Also  unlike reporting on credential attainment, DOL does not require that local  WIA programs collect and retain documentation on training-related  employment in the participants’ case files to verify the accuracy of data  they report to states.
Officials in four of the six states we studied, as well as at DOL, said this  data collection process often requires considerable time and effort.  Further, officials from DOL and four states emphasized the need to  consider the balance between the time required to collect outcome data  and the time case managers spend serving participants, especially in an  environment of reduced resources. In addition, officials in all six states  said that making training-related employment determinations can be  subjective. According to DOL’s reporting guidance, participants’  employment is related to their training if it uses “a substantial portion of  the skills taught in training.” workforce experts from one national organization, said that one case  manager’s interpretation of what constitutes a substantial portion of the  skills obtained in training may differ from another’s. The training-related  employment decision can be straightforward if the training and job are  clearly connected. For example, if a participant received training to attain  a commercial driver’s license and was subsequently hired as a driver by a  trucking company, the case manager can easily determine that the  participant’s employment is substantially related to the training received.  In other cases, however, the decision may be more subjective. For  example, officials in one state could not agree whether a participant who  had received aviation instruction training had secured training-related  employment in his position as an airframe and power plant mechanic.  Some state officials thought the skills obtained were transferable, but  others were unsure.
U.S. Department of Labor, Training and Employment Guidance Letter No. 17-09:  Quarterly Submission of Workforce Investment Act Standardized Record Data (WIASRD)  (March 10, 2010).
Collecting participants’ employment information and attempting to  determine whether it was training-related generally entails several steps  (see fig. 5).
Contacting participants. Several state and local officials we  interviewed said that they generally begin the process of collecting  data on training-related employment by attempting to contact  participants, though some can be unresponsive or inaccessible, which  workforce experts from one national organization noted as well. If  case managers are successful, they ask participants for information  such as the name of their employer and their job title. Some local  areas also ask participants directly if their new jobs are—in the  participants’ opinion—related to the training they received. In some  cases, case managers make their training-related employment  determinations based solely on information the participants provide  about their employment.
Contacting employers. Some case managers contact employers to  obtain participants’ employment information, though employers may  not be responsive. If case managers could not reach a participant but  know where the person works, they may contact the employer to  obtain their job title and description. They may also contact an  employer to verify the information provided by a participant. Case  managers can use the employment information obtained from an  employer, or from both the participant and employer, to determine if a  participant’s job is training-related. However, workforce officials in  Illinois and Texas said employers may not be responsive because  they are concerned about employee privacy or about the amount of  follow-up required.
Checking UI wage records. Some local workforce officials said that if  they are unable to gather information about a participant’s  employment from the participant or the employer, they check the UI  wage records, which are generally not available until several months  after a participant exits from WIA services. DOL officials and  workforce experts at two national organizations said the UI wage  records generally provide the name of the participant’s employer and  a code associated with the employer’s industry, but specific  information on the participant’s occupation is rarely included. In some  cases, the industry code has a clear connection to the training  received, making the case manager’s training-related employment  determination straightforward. For example, if a participant who was  trained as a nurse was hired by a hospital, the case manager can  reasonably assume that the employment and training are related.  However, some officials noted that the industry code is not always a  good predictor of a participant’s occupation. For example, if the same  participant was hired by the health unit of a manufacturing company,  the industry code in the UI wage records would suggest that the  person’s job was associated with manufacturing and not related to the  nursing training.
Other steps to determine training-related employment. If  successful in accessing a participant’s employment information,  including job title, some workforce officials said case managers use  DOL’s Occupational Information Network (O*NET), which provides an  online tool to match job title occupational codes to the skills code  associated with the participant’s training. This can help case  managers decide whether the participant’s employment is training- related.
According to some local workforce officials, however, it can be difficult  to find the precise occupational code that matches the participant’s  new job. In addition, DOL officials told us that even attempting to  match O*NET codes in this manner might not help case managers  determine if participants’ employment is related to their training  because the threshold of relatedness is still subjective, as mentioned  previously. DOL officials also told us it is difficult to prescribe a  standard definition for determining whether a job is related to training  because it often requires some judgment on the part of local officials.

		States Have Taken Steps to Address Challenges by Improving Reporting on Training-Related Employment

To improve reporting, some states have taken steps to increase their  access to information about participants’ employment. Similar to their  efforts to collect information on credential attainment, some local  workforce officials told us that case managers seek to build rapport with  participants. At the same time, officials we interviewed in four of six states  have taken steps to increase access to employment-related information.  In Illinois, Rhode Island, and Texas, local workforce officials ask  participants to sign release forms authorizing employers to release  employment information to officials. Local workforce staff in Kansas and  Texas also said they obtain information on participants’ employment,  including the names of their employers, and—unlike UI wage records— their job titles, by subscribing to an online payroll database called The  Work Number. This service verifies employment via a database of  national payroll data but does not include all employers, and local WIA  programs must pay to subscribe.
Officials said they conduct this survey for  internal use to collect and share data on  training-related employment that is not  otherwise available to program managers and  state officials. Another official noted that a  significant percentage of respondents who do  not report their job as training-related find the  training instrumental in getting the job or that  the skills they acquired in training are useful  on their job.
In addition, local workforce officials in four states we contacted said they  have developed strategies to help reduce the subjectivity in determining  whether a participant’s employment is related to their training. For  example, local officials in Texas told us that staff may consult their  American Job Center’s local business services office, which often has  specific knowledge about what skills correspond with particular job titles.  Officials in Texas and Alabama also said case managers may consult  with their peers or supervisors to reach consensus about a training- related employment decision. Further, state workforce officials in Kansas  told us that when employers post jobs on the state’s job bank, they are  required to enter occupational codes from DOL’s online O*NET database.  If a training participant gets one of these jobs, case managers can  compare the employer-provided occupational code with the training codes  supplied by training providers to help them determine whether the job is   training-related. State and local officials in Washington also said  managers routinely use O*NET when making their training-related  employment decisions.

		DOL Has Taken Limited Steps to Address Challenges and Improve Data on Training-Related Employment

While DOL has recently issued guidance aimed at increasing reporting  rates for training-related employment data, it has taken limited steps to  address states’ ongoing reporting challenges. As previously discussed, a  September 2011 report by DOL’s Office of Inspector General raised  questions about the quality of these data and made recommendations to  address this issue. In response, DOL issued a notice to states in  September 2013 that reiterated the requirement for states to report these  data and stressed the data’s importance for program analysis and  evaluation efforts pertaining to the value of investments in WIA training.  The DOL notice also acknowledged that reporting on training-related  employment is challenging largely because the information must be  collected manually. According to the DOL notice, nearly every state that  participated in conference calls on the topic indicated that there was  considerable cost in conducting the necessary follow-up for reporting on  training-related employment and that this was the primary reason the data  were not well-reported. State and local officials we interviewed also noted  that such data may be underreported because of the difficulty of following  up with participants and employers. DOL also concluded that states with  larger training caseloads had less complete reporting on training-related  employment. In the notice, DOL also described a few practices most  common among states with higher reporting rates for training-related  employment. For example, DOL cited the practice of instituting a data  system check to ensure that training-related employment data are  recorded before closing a participant’s case file.
Use of crosswalk not necessarily a  solution for determining training-related  employment  Another program DOL administers, Job  Corps—a residential, educational, and career  technical training program for disadvantaged  youths—uses a crosswalk that links specific  training codes and occupational codes to help  staff determine training-related employment.  However, in a September 2011 report, the  DOL Inspector General found that this  crosswalk included some matches that were  either not related or poorly related. Moreover,  DOL officials said that using a crosswalk for  the WIA Adult and Dislocated Worker  programs could make the training- employment link too restrictive and would  require a considerable amount of resources to  develop.
DOL has not identified and disseminated strategies for increasing access  to employment-related data or helping to minimize the subjectivity of  training-related determinations, but instead has focused exclusively on  increasing data reporting rates. While DOL officials maintain that manual  follow-up with participants is the best approach for obtaining employment  related data, they also recognize that such data collection is resource- intensive. In addition, as officials in all six of our selected states also  noted, DOL officials acknowledged that determining whether a  participant’s employment is training-related can be a subjective decision.  They noted challenges in defining training-related employment more  precisely. For example, officials said some training is intended to develop  broad, nonspecific skills that may help participants get jobs but are not  associated with a specific occupation or industry. We recognize that  utilizing professional judgment is inherent in certain tasks such as  determining whether a participant’s employment is related to the training  the participant received. However, minimizing the amount of subjective  decision-making involved to the extent possible could help ensure better  quality data on training-related employment. Reasonable approaches for  improving the quality of performance data focus on aspects of  completeness, accuracy, consistency, and ease of use. By identifying  and sharing with states practices to increase access to employment- related data and reduce the subjectivity of some determinations, DOL  could help states improve their reporting of data on training-related  employment.
In addition to the strategies all six selected states use to mitigate  reporting challenges, workforce officials from three states and workforce  experts at two national organizations said some additional considerations  should be taken into account in weighing a performance measure on  training-related employment (see table 5). We previously noted concerns  about the level of resources required and the subjectivity of  determinations, both of which could affect the data’s completeness and  consistency—key aspects of performance data quality. The state and  local strategies we identified may help mitigate some reporting  challenges. In addition, a participant’s successful placement in a training- related job depends on both the ability and decision to pursue such a  position. Some local workforce officials we contacted said that it may take  participants longer to find employment in their field of training than is  allowed for reporting purposes. For example, workforce officials in  Rhode Island said that due to the poor economy in their state, it is not  uncommon for some participants to take 2 years or more to find a job.  Additionally, workforce officials in three states we contacted said  participants may decide to take a job unrelated to their training if it is the  only job they can find or if they simply choose not to pursue a job in the  field in which they were trained.

	Conclusions

To ensure that public funds invested in WIA’s Adult and Dislocated  Worker Programs are spent wisely, program managers and policymakers  need performance data that are accessible, complete, accurate and  consistent. The current common performance measures—employment,  retention, and earnings—provide a basis for assessing the overall value  of the services the programs provide, primarily using a standardized data  source (state UI wage records). Beyond these, data on outcomes such as  credential attainment and training-related employment can potentially  provide information more specifically on the value of training services.  However, as we have noted, collecting data on these outcomes can be  more resource-intensive, in part because there is no single readily  available source of data.
DOL has taken steps to elevate the importance of credential attainment  and improve data quality for this outcome. We found credential attainment  data reported to DOL to be reliable. In contrast, we found the data  reported to DOL on training-related employment to be incomplete and  inconsistent. While DOL has acknowledged challenges in collecting data  on and determining training-related employment, it has taken only limited  steps to address these challenges, focusing efforts exclusively on  improving reporting rates. This effort alone will not improve the quality of  the data being reported. Given the nature of the challenges we identified,  there are no easy or complete solutions. However, we also identified  strategies some states use that may help increase access to employment  information and reduce the subjectivity of some training-related  determinations. Sharing such strategies with other states, as well as  identifying and communicating other approaches, could lead to  incremental improvement in the quality of data reported. Without such  action, the data states are required to report on training-related  employment are likely to remain unusable.

	Recommendation for Executive Action

To provide policymakers and program managers with better quality  information to assess the value of training provided by WIA’s Adult and  Dislocated Worker Programs, we recommend that the Secretary of Labor  identify and share with states strategies for collecting and reporting data  on training-related employment that could either increase access to  employment information or reduce the subjectivity of determining when  training is related to employment.

	Agency Comments and Our Evaluation

We provided a draft of this report to the Secretary of Labor and selected  draft sections to the Secretary of Education. DOL and Education provided  technical comments, which we incorporated as appropriate, and DOL  provided a written response (see app. II). DOL agreed with our  recommendation and noted that having reliable data on training-related  employment is important to effectively manage and evaluate the Adult  and Dislocated Worker Programs. DOL also agreed that states can  benefit from learning what other states are doing to address challenges  regarding access to and subjectivity of these data. Toward this end, DOL  noted that it plans to conduct additional conference calls with state  officials to reiterate the importance of identifying training-related  employment and continue to discuss and share best practices to improve  these data. DOL noted that this sharing of best practices would  supplement actions it has already taken to improve data on training- related employment. These actions include coding changes to the  WIASRD to capture additional information, conference calls with state  workforce officials to discuss reporting on training-related employment,  and a work group considering adding more data elements to the UI wage  records such as an occupational code.
As agreed with your office, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days from its  issue date. At that time, we will send copies of this report to the  appropriate congressional committees, the Secretary of Labor, the  Secretary of Education, and other interested parties. In addition, the  report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or sherrilla@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who make key contributions to this report  are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives for this study on the Workforce Investment Act’s (WIA)  Adult and Dislocated Worker Programs were to examine: 1) the extent to  which training participants obtain credentials and secure training-related  employment, 2) challenges states face reporting on credential attainment  and what steps, if any, they and DOL are taking to address them, 3)  challenges states face in reporting on training-related employment and  what steps, if any, they and DOL are taking to address them.
To address these objectives, we reviewed relevant federal laws,  regulations, and DOL’s guidance to states for reporting select data on  participants in the WIA Adult and Dislocated Worker Programs. We  interviewed DOL officials from the Employment and Training  Administration and the Office of Inspector General as well as experts on  workforce issues (see Selection of Experts, below). We also interviewed  state and local workforce officials as well as organizations that provided  an employer perspective from a nongeneralizeable sample of six states  (see Selection of States and Local Areas, below). To assess the reliability  of the data DOL collects on credential attainment and training-related  employment, we analyzed data from the Workforce Investment Act  Standardized Record Data (WIASRD) system for program year 2010 and  program year 2011—the most recent years for which data were  available—by testing the data electronically and interviewing  knowledgeable agency officials. We found the data to be sufficiently  reliable for providing estimates on: 1) participants who received training,  2) the minimum number of training participants who earned a credential,  and 3) the types of credentials they earned. However, we determined that  the data on training-related employment were not reliable for the  purposes of our report (see Analysis of DOL’s Training Outcome Data  below).

	Selection of Experts

We interviewed experts on workforce issues representing six  organizations. We identified experts by first reviewing relevant literature  and asking officials from DOL for recommendations. We further  developed the list by asking each expert we interviewed for additional  names. Only experts who were mentioned more than once were selected.  The results of these interviews are not generalizeable, but provided  insights about the overall challenges states face in reporting on these  outcomes and any efforts by states to overcome them.

	Selection of States and Local Areas

We interviewed state and local workforce officials from a  nongeneralizeable sample of six states. We conducted in-person  interviews with officials in Alabama, Illinois, and Texas and telephone  interviews with officials in Kansas, Rhode Island, and Washington (see  table 4). These results are not generalizeable, but provided insights about  the challenges they face in reporting data on credential attainment and  training-related employment as well as any steps they are taking to  address those challenges. We selected the states to reflect a mix of those  that had leading-edge data practices (as identified by experts) and those  with either incomplete data or relatively high rates of reporting errors (as  identified by WIA data quality reports on credential attainment). We also  factored geographic diversity into state selection. In each state, we  interviewed state workforce officials and also local workforce officials from  at least one workforce investment board and at least one American Job  Center—formerly known as a one-stop center. We selected a non- generalizeable sample of local areas based on input from state workforce  officials and, for states we visited, proximity to the state workforce  agency. In addition, we interviewed at least one employer organization in  each state. In selecting these entities, we considered states’ input on  organizations that could provide us with an employer perspective on the  value of credentials and certain types of training for various industries.  They included statewide business associations, regional business  associations, individual employers, and industry-specific representatives.
In each state, we obtained information about the state and local area’s  process of collecting data on credentials and employment related to  training as well as any challenges they may have encountered. We also  asked state officials about DOL’s related guidance. We used a semi- structured interview guide for the state and local interviews.

	Analysis of DOL’s Training Outcome Data

To assess the reliability of DOL’s data on training, credential attainment,  and training-related employment in the WIASRD database for participants  in the WIA Adult and Dislocated Worker programs, we: (1) reviewed  documentation related to reporting these data, including DOL’s Office of  Inspector General reports; (2) tested the data electronically to identify  potential problems with consistency, completeness, or accuracy; and (3)  interviewed knowledgeable DOL officials about the data. Our electronic  testing consisted of identifying inconsistencies, outliers, and missing  values. In addition, we analyzed the publicly-available WIASRD data file  for program years 2010 and 2011, which was produced for DOL by its  data contractor, Social Policy Research Associates. As part of our  analysis, we reviewed the steps the data contractor took to address data  errors and, to the extent possible, compared the data DOL provided for  our analysis to the publicly-available file, and found only slight  discrepancies. We found the data on training and credential attainment to  be sufficiently reliable for reporting estimates of: (1) participants who  received training, (2) the minimum number of training participants who  earned a credential, and (3) the types of credentials they attained. We  were not able to reliably make state-to-state comparisons because two  states are piloting a new reporting format for DOL, and therefore would  not have been compatible with the others. For the purposes of this report,  we did not find the data on training-related employment reliable. We  reached this conclusion based on our analysis of the data, an Office of  Inspector General report, and DOL’s data quality reports. We were not  able to determine how many training participants in the Adult or  Dislocated Worker Programs obtained employment related to their  training in program year 2011. For the Adult Program, we found that  states reported data on 48 percent of training participants, but had  missing data for the remaining 52 percent. For the Dislocated Worker  Program, states reported data on 74 percent of training participants, but  had missing data for the remaining 26 percent. Further, an analysis of the  reported data showed wide variation among states regarding the  percentage of participants who obtained training-related employment  raising questions about the data’s reliability.
We conducted this performance audit from October 2012 to January 2014  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit work to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Labor

Appendix III: GAO Contact and Staff Acknowledgments

	GAO Contact:

Andrew Sherrill, (202) 512-7215 or sherrilla@gao.gov.

	Staff Acknowledgments

In addition to the contact named above, Laura Heald, Assistant Director,  John Lack, Jeffrey G. Miller, and Kathryn O’Dea Lamas made key  contributions to this report. Also contributing to this report were James  Bennett, Jessica Botsford, David Chrisinger, Elizabeth Curda, Kathy  Leslie, Carol Patey, Rhiannon Patterson, Jerry Sandau, and Walter  Vance.

Related GAO Products

Workforce Investment Act: Local Areas Face Challenges Helping  Employers Fill Some Types of Skilled Jobs. GAO-14-19. Washington,  D.C.: December 2, 2013.
Workforce Investment Act: DOL Should Do More to Improve the Quality  of Participant Data. GAO-14-4. Washington, D.C.: December 2, 2013.
Postsecondary Education: Many States Collect Graduates’ Employment  Information, but Clearer Guidance on Student Privacy Requirements Is  Needed. GAO-10-927. Washington, D.C.: September 27, 2010.
Workforce Investment Act: Employers Found One-Stop Centers Useful in  Hiring Low-Skilled Workers; Performance Information Could Help Gauge  Employer Involvement. GAO-07-167. Washington, D.C.: December 22,  2006.
Workforce Investment Act: Labor and States Have Taken Actions to  Improve Data Quality, but Additional Steps Are Needed. GAO-06-82.  Washington, D.C.: November 14, 2005.
Workforce Investment Act: Substantial Funds Are Used for Training, but  Little Is Known Nationally about Training Outcomes. GAO-05-650.  Washington, D.C.: June 29, 2005.
Workforce Investment Act: Labor Should Consider Alternative  Approaches to Implement New Performance and Reporting  Requirements. GAO-05-539. Washington, D.C.: May 27, 2005.
Workforce Investment Act: States and Local Areas Have Developed  Strategies to Assess Performance, but Labor Could Do More to Help.  GAO-04-657. Washington, D.C.: June 1, 2004.
Workforce Investment Act: Labor Actions Can Help States Improve  Quality of Performance Outcome Data and Delivery of Youth Services.  GAO-04-308. Washington, D.C.: February 23, 2004.
Workforce Investment Act: Improvements Needed in Performance  Measures to Provide a More Accurate Picture of WIA’s Effectiveness.  GAO-02-275. Washington, D.C.: February 1, 2002.

Summary:
Why GAO Did This Study

As the economy recovers, some employers continue to face difficulty finding qualified workers. The WIA Adult and Dislocated Worker Programs provide services, including job training, which aims to help participants acquire skills and credentials employers need. Under WIA, states are required to report data on training participants who obtain credentials and on those who enter employment related to the training they receive. Given that a WIA reauthorization proposal would establish both of these outcomes as performance measures, GAO was asked to examine the capacity of states to report on these outcomes. This report addresses: 1) the extent to which training participants obtained credentials and training-related employment, 2) challenges states face in reporting data on credentials and what steps, if any, they and DOL are taking to address them, and 3) challenges states face in reporting data on training-related employment and what steps, if any, they and DOL are taking to address them. GAO interviewed DOL officials, workforce experts, and state and local officials and employer organizations from a nongeneralizeable sample of six states selected in part on the basis of geographic diversity. GAO also analyzed data on credential attainment and training-related employment for program years 2010 and 2011.

What GAO Found

Of the more than two million total participants in the Workforce Investment Act's (WIA) Adult and Dislocated Worker Programs, about 11 percent and 16 percent, respectively, received training in program year 2011, and about two-thirds of the training participants in each program attained a credential. Little is known, however, about how many participants got jobs related to their training. From program year 2006 through program year 2011, the percentages of training participants who earned a credential declined from about 74 percent to 58 percent for the Adult Program and from about 75 percent to 63 percent for the Dislocated Worker Program, according to data from the Department of Labor (DOL). Of those training participants who attained a credential in program year 2011, about 65 percent earned occupational credentials, such as a welding certificate, followed by lower percentages who earned occupational skill licenses and associate's degrees, among others. In contrast, GAO found training-related employment data unreliable primarily because a significant portion of the data was missing.
Workforce officials in four of six selected states cited some obstacles in reporting data on credential attainment, and both DOL and states are taking steps to address challenges. Officials in four states GAO contacted said reporting such data can be resource-intensive, largely because case managers must manually track this information from various sources, including participants, training providers, and third-party organizations. To improve credential attainment and reporting, DOL clarified which credentials should be reported and began measuring credential attainment through an agency-wide goal in 2010. Officials in five states have taken similar steps, such as setting goals and tracking credential attainment, and enhancing data exchange with training providers. Officials in most of the six states GAO contacted noted some obstacles to obtaining such data. For example, officials from several states cited student privacy laws as a barrier in verifying credentials with training providers. Officials in three states told us that they ask participants to sign consent forms allowing training providers to give credential information to local officials.
Workforce officials in most of the selected states identified even greater challenges reporting data on training-related employment, including the high degree of resources required and the subjective nature of determining whether employment is linked to training. DOL has taken only limited steps to address these challenges. To report such data, case managers seek participants' employment information from participants, employers, and wage records. Then they must piece it together to determine whether participants' employment is "substantially related" to their training. Officials in most of the six states described this process as resource-intensive and noted that making such determinations are subjective since one case manager's interpretation of "substantially related" may differ from another's. Given these challenges, officials in all six states have taken some steps to increase access to employment information or make decisions less subjective. DOL has recently stressed the importance of reporting data on training-related employment and shared a few practices with states to increase reporting rates; however, it has not identified and disseminated strategies to address the ongoing challenges states face regarding resource intensiveness and subjectivity, which could improve the quality of such data.

What GAO Recommends

GAO recommends that DOL identify and share with states strategies that may ease collection and improve the quality of training-related employment data. DOL agreed with GAO’s recommendation.