Veterans' Employment and Training Service: Labor Could Improve Information on Reemployment Services, Outcomes, and Program Impact
Released: 2007-05-24T00:00:00

Letter

Background

The nation’s veteran job seekers receive employment and training services from programs overseen by two agencies within Labor—the Veterans’ Employment and Training Service and the Employment and Training Administration. General employment services fall under the purview of ETA, which administers the Wagner-Peyser-funded Employment Service program, providing a national system of public employment services to all individuals seeking employment—including veterans. Thus, those veterans considered job ready and not in need of intensive services may be served by Employment Service staff and receive such services as assessment, counseling, job readiness evaluation, and placement. ETA carries out its Employment Service program through workforce agencies in each state.

In fiscal year 2006, the Employment Service program provided a total of about $716 million to states.

While ETA administers programs that serve the general population, including veterans, VETS administers the DVOP and LVER programs, which focus exclusively on serving veterans, often providing more intensive services than the Employment Service does. Like ETA, VETS carries out its responsibilities through a nationwide network that includes representation in each of Labor’s six regions and staff in each state. The Office of the Assistant Secretary for VETS administers the agency’s activities through regional administrators and state directors. The DVOP specialists and LVER staff, whose positions are funded by VETS, are part of states’ public employment services. In fiscal year 2006, the DVOP and LVER programs were funded at about $155 million. In the most recent program year—program year 2005, which spanned July 1, 2005, to June 30, 2006—the Employment Service, together with the DVOP and LVER programs, reported serving about 1.32 million veterans nationwide, of whom over 715,000 were served by DVOP specialists and LVER staff.

The Employment Service and the DVOP and LVER programs are mandatory partners in the one-stop system under WIA—where services are provided by a range of employment and training programs in a single location. Veterans, along with other eligible job seekers, may receive services from other mandatory one-stop partners, such as WIA-funded training or Trade Adjustment Assistance. Additionally, job seekers, including veterans, may use the one-stop centers’ computers and other resources without staff assistance, and in many places may access one- stop services online from home. Department of Veterans Affairs (VA) programs are not mandatory partners in the one-stop system, but do participate at some locations.

Jobs for Veterans Act Reforms

In 2002, the Jobs for Veterans Act amended Title 38 of the U.S. Code-- which governs the DVOP and LVER programs—and by doing so, introduced an array of reforms to the way employment and training services are provided to veterans. JVA sought to address concerns that the programs were overly prescriptive by providing states with enhanced flexibility to determine the best way to serve veteran job seekers. Among its reforms, JVA redefined the DVOP specialist and LVER staff roles but gave states flexibility in deciding their duties; established a single state grant and a new funding formula that allowed states to determine the mix of DVOP specialists and LVER staff; required a comprehensive performance accountability system consistent with WIA performance measures; required that veterans receive priority over other job seekers in all Labor job training programs, not just the Employment Service; and required that VETS include information in its annual report to Congress on employment services to veterans throughout the one-stop system.

JVA identified broad roles and responsibilities of DVOP specialists and LVER staff. For example, DVOP specialists are to focus on providing intensive services to eligible veterans, giving priority to disabled veterans and those with other barriers to employment. LVER staff are to focus on conducting outreach to employers to assist veterans in gaining employment, as well as facilitating employment, training, and placement services given to veterans. State workforce agencies receive a single veterans’ program grant to fund both programs; the amount each state receives is determined in part by the size of the veteran population within each state. State agencies then decide how to distribute the amount they receive between the two programs. Table 1 lists selected responsibilities of DVOP specialists and LVER staff as set forth in Labor guidance.

JVA also stipulated that veteran job seekers must receive priority over other job seekers in any job training program administered by Labor. Labor’s guidance requires states to explain how veterans will be given priority and how veterans’ services will be provided through the state’s one-stop system. For programs that target particular populations, such as seniors or low-income individuals, veterans’ priority is applied after any other mandatory eligibility provisions are met.

Changes to the DVOP and LVER Performance Information

Like other Labor employment and training programs, the DVOP and LVER programs have experienced changes both in the way outcomes are tracked and in the measures used to assess performance. Specifically, in 1998, WIA required that states use automated unemployment insurance wage records to track employment-related outcomes. Formerly, to obtain data on outcomes, states relied on a manual follow-up process using administrative records or contacts with job seekers. To conform to WIA, VETS moved from such a manual follow-up system to the new automated process in 2002.

The measures that Labor uses to assess performance in the DVOP and LVER programs have also changed over time, gradually reflecting more emphasis on outcome-based measures. Before passage of the JVA in 2002, for example, some of the measures used for the DVOP and LVER programs focused more on services received—such as the number of veterans in training or receiving counseling—than on outcomes achieved. In 2002, JVA required that Labor develop a comprehensive performance accountability system and required that the new system measure performance in a way that is consistent with WIA. In 2003, VETS adopted performance measures for the DVOP and LVER programs based on those then used in WIA. In 2005, in response to an Office of Management and Budget (OMB) initiative, Labor began requiring states to implement common performance measures for its employment and training programs, including the DVOP and LVER programs, the Employment Service, and WIA. OMB established a set of common measures to be applied to most federally funded job training programs that share similar goals. Labor further defined the common measures for all of its Employment and Training Administration programs, applying three measures to each of its adult programs (see table 2).

In applying the common measures to its programs, VETS also developed additional measures to emphasize outcomes for disabled veterans in the DVOP program and outcomes for recently separated veterans in the LVER program.

Data Reporting

Labor collects performance data for the DVOP and LVER programs on a quarterly basis from state workforce agencies. The state agencies use report formats developed by Labor to provide detailed tabulations of aggregate information on the characteristics of veteran participants, services, and outcomes for the two programs, including data showing states’ performance using the common measures. The state agencies provide this information to Labor in three separate reports: one for the DVOP program, one for the LVER program, and one representing an unduplicated count for both programs.

Furthermore, Labor collects additional information on veterans who participate in other Labor programs. For example, ETA collects performance data for the Employment Service on all participants on a quarterly basis from state workforce agencies, and these reports break out services and outcomes for veteran participants. States submit their quarterly reports for the Employment Service and the DVOP and LVER programs through the same Labor reporting system.

The Role of Impact Evaluations in Assessing Program Outcomes

Information on the services a program has provided and the outcomes obtained by program participants are necessary to assess program impacts. However, this information is not sufficient to measure program impacts—the outcomes may be due to other external factors such as local labor market conditions. While impact evaluations allow one to isolate a program’s effect on the outcomes of participants, there are several approaches to conducting such evaluations. The experimental method is often considered the most rigorous method for conducting impact evaluations. In the experimental method, participants are randomly assigned to two groups—one that receives a program service (or treatment) and one that does not (control group). The resulting outcome data on both groups are compared and the difference in outcomes between the groups is taken to demonstrate the programs impact. However, it is not always feasible to use the experimental method for assessing program impacts. Alternatively, researchers may use a quasi- experimental approach in which program participation is not randomly assigned. One approach, often called a comparison group study, compares outcome data for individuals who participated in the program with data on others who did not participate for various reasons. In a comparison group study, it is important to find ways to minimize, or statistically control for, any differences between the two groups. According to OMB, well-matched comparison group studies, under certain circumstances, can approach the rigor of the experimental method, and it recommends considering this method if random assignment is not feasible or appropriate.

Under WIA, Labor was required to conduct at least one impact evaluation of program services by 2005. In a 2004 report, we found that Labor had not yet begun such an evaluation, and recommended that the agency comply with this statutory requirement and help federal, state, and local policy makers understand what services are most effective for improving employment-related outcomes.

DVOP and LVER Performance Information Is Weakened by Several Factors

The DVOP and LVER programs’ performance information is weakened by several factors, including implementation challenges and frequent changes to performance reporting requirements. In July 2005, Labor implemented new performance measures, which provide information on some outcomes for veterans. However, not all performance measures have been fully implemented. Additionally, neither the performance measures nor the data reported to Labor reflect the full range of services that DVOP specialists and LVER staff provide to veteran job seekers. Furthermore, it is difficult to assess outcomes over time or across states because of frequent changes in states’ reporting requirements that prevent establishing reliable trend data.

The New Performance Measures Provide Information on Some Outcomes for Veterans, but Have Not Been Fully Implemented

In July 2005, the DVOP and LVER programs adopted the Office of Management and Budget’s common measures, along with other employment programs, including WIA and the Employment Service. Specifically, states implemented measures that track whether veterans obtain and keep jobs after receiving services through these programs, but they have not yet implemented a measure to track veterans’ earnings. States are held accountable for four separate measures in each program that focus on outcomes attained by veterans (see table 3). For the DVOP program, states are held accountable for employment and retention for all veterans served by the program, as well as for disabled veterans. For the LVER program, states are assessed on employment and retention for all veterans, as well as for recently separated veterans. Currently, all states collect and report data to Labor for calculating performance attainment and negotiating state goals for these eight measures.

However, states are not yet held accountable for an additional common measure—veterans’ average earnings—in either the DVOP or the LVER programs. Other employment and training programs, such as WIA and the Employment Service, include an average earnings measure for which states are accountable. For the DVOP and LVER programs, however, calculating the average earnings was not as straightforward as Labor had anticipated. A VETS official told us that the agency will calculate baseline data for average earnings during the current program year, but Labor will not establish goals and states will not be held accountable for their performance on this measure until the following year—program year 2007—at the earliest.

Furthermore, Labor has not adopted a system to give more weight to successful outcomes for veterans who have substantial barriers to employment, such as a disability. JVA required Labor to weight performance measures to provide special consideration to veterans requiring intensive services, as well as disabled veterans. Such a weighting system would compensate for the fact that veterans with barriers to employment may need more assistance than others in finding jobs. It would also provide an incentive for program staff to help veterans with severe barriers to employment. For example, if a veteran has a disability and requires intensive case management services, his or her successful outcomes would have a greater effect on a state’s overall performance than those of other veterans with fewer barriers. Following JVA’s enactment, Labor formed a work group to develop a weighting system for the DVOP and LVER performance measures. On the basis of the group’s work, the agency issued guidance to introduce the weighted measures to states in June 2003, with the expectation of implementing them soon after. However, after further review, a Labor official told us the agency did not implement the weights in order to give states time to fully implement other reporting changes. At this time, it is not clear whether Labor will implement this system in the future.

Performance Measures and the Data Reported to Labor Do Not Reflect the Full Range of Staff Services

Although DVOP specialists and LVER staff perform similar duties for all types of veterans in most states, the current performance measures hold the two programs accountable for different groups of veterans. JVA and Labor’s guidance outline the key responsibilities and target populations for DVOP specialists and LVER staff, but also allow for some flexibility in their roles and responsibilities. Both DVOP and LVER staff are expected to serve the general veteran population, but DVOP specialists are also expected to target their services toward veterans who have greater barriers to employment and need intensive case management, including disabled veterans. JVA specifies that LVER staff focus on conducting outreach to employers and assisting all veteran job seekers. In addition, Labor has recently added the expectation that LVER staff focus their responsibilities on assisting recently separated veterans. As a result of these expectations, Labor separately holds DVOP specialists accountable for the outcomes achieved by the disabled veterans they serve, and LVER staff for the outcomes of the recently separated veterans they serve.

In practice, however, both programs’ staff serve similar veteran populations. In program year 2005, for example, 14 percent of veterans served by the DVOP program were disabled and 21 percent were recently separated. For the LVER program, 10 percent of veterans served were disabled and 19 percent were recently separated (see table 4). States acknowledged this similarity in our survey as well. Over a third of states responded that DVOP and LVER staff are equally likely to serve disabled veterans, while about half of states responded that the two programs’ staff are equally likely to serve recently separated veterans.

In addition to finding similarity in populations served, we also found some similarity in activities carried out by DVOP and LVER staff. States reported that the three activities on which DVOP specialists spend the most time include providing intensive case management services, conducting an initial assessment or interview, and assisting with job search activities.

The top three activities LVER staff perform include conducting outreach to employers, assisting with job search activities, and conducting an initial assessment or interview.

This division of duties appears to reflect the different focuses of the two programs, as well as the flexibility under JVA for states to decide on staff duties. However, almost 85 percent of states responded that DVOP specialists conduct outreach to employers, a focus of the LVER program. Additionally, almost 60 percent of states responded that LVER staff provide intensive services, a primary focus of the DVOP program.

In our site visits, we found that this similarity in staff roles and target populations exists in part because some one-stop centers have only a single DVOP specialist or LVER staff on duty at any given time. In these particular one-stop centers, the same employee is responsible for serving all groups of veterans and carrying out job roles for both programs. Even in centers with more than one staff person, veterans tend to be served by whichever staff person is available at that time. Program staff in several centers told us that recently separated veterans were not specifically directed to LVER staff for services, nor were disabled veterans directed to DVOP staff. This sharing of duties may be due, in part, to changes in staffing levels. More than half of states reported a decrease in the number of full-time DVOP specialists or LVER staff over the last 2 years, and most attributed this decline to the size of their state grant for the programs.

Nevertheless, this similarity in roles and populations served causes the current performance measures to present an incomplete view of outcomes for disabled and recently separated veterans in the DVOP and LVER programs. The large numbers of disabled veterans served by the LVER program and recently separated veterans served by the DVOP program are not included in the set of measures that focus on the outcomes of those populations (see fig. 1).

Beyond the measures for disabled and recently separated veterans, having separate measures for the DVOP and LVER programs obscures the overall picture of outcomes to veterans, given the similarity between many of the program activities and the reality of how the programs operate. According to our survey, almost half the states would like Labor to consolidate the performance measures for the DVOP and LVER programs.

While the performance measures present an incomplete view of the outcomes for veterans, the data that states are required to report to Labor do not reflect the full range of staff services. Labor requires states to report a wide range of data for the DVOP and LVER programs, including information on veteran characteristics—such as age and disability status— and staff services provided—such as intensive services and referrals to other programs. However, Labor does not require data on employer outreach activities, despite JVA’s designation of employer outreach as a primary job responsibility of LVER staff. Consequently, Labor and states cannot formally monitor the extent to which staff perform this activity. Some states, however, collect these data for their own use. According to our survey, almost half of states currently collect employer-related information for the DVOP and LVER programs, and over 75 percent of states reported that it would be helpful to collect these data.

In addition, even though the data reported to Labor generally reflect services and outcomes for veterans, these data are aggregate tallies and do not show services provided to individual veterans. For example, each state’s quarterly reports reflect the sum of all services provided and do not show the number of services provided per veteran or per staff person. The current data are useful to provide an overall picture of the programs’ volume and operations. However, these data provide little information about services received by individual veterans or delivered by particular veteran staff.

Performance Data Are Not Comparable over Time or across States

In recent years, reporting requirements for the DVOP and LVER programs have undergone several significant changes. These changes have moved the performance accountability system closer to those of other employment and training programs. At the same time, the changes have resulted in a lack of reliable trend data. In July 2002, the DVOP and LVER programs changed from using administrative follow-up to determine veterans’ employment outcomes to obtaining information from Unemployment Insurance (UI) wage records. In doing so, Labor changed its method of calculating outcomes for veterans in the DVOP and LVER programs. Then, in July 2005, Labor applied the common measures to these two programs, refining and standardizing the application of UI wage records to determine outcomes. Under the old system, Labor calculated entered employment and employment retention rates based on the number of veterans who participated in the programs. However, under the new system, Labor calculated these rates based on how many veterans terminate services and exit the programs. Although these changes have standardized the performance measures across programs, they have also prevented Labor and states from developing consistent, comparable data over the past 5 years. As a result, Labor does not have reliable historic data for either program. Figure 2 illustrates the various changes to the DVOP and LVER programs’ performance reporting requirements.

Furthermore, the instability in data collection and reporting has left Labor unable to establish a national veterans’ entered employment standard, as required by JVA. Labor anticipates that it will need at least 3 years of stable data to establish the national standard. Once it is established, all states will be held accountable to the same minimum goal for veterans’ entered employment. However, it is unclear when Labor will have sufficient data to establish this standard because states continue to experience difficulty adjusting to the numerous changes. According to our survey, over 70 percent of states reported that frequent changes to performance reporting requirements have been either a great or very great challenge.

The data also vary somewhat state by state. For example, the application of wage records to calculate veteran outcomes across state lines is no longer consistent across states. The Wage Record Interchange System (WRIS) allows states to share UI wage records and account for job seekers who participate in one state’s employment programs but get jobs in another state. In recent years, all states but one participated in WRIS, which was operated by the nonprofit National Association of State Workforce Agencies. In July 2006, Labor assumed responsibility for administering WRIS. However, many states have withdrawn, in part because of a perceived conflict of interest between ETA’s role in enforcing federal law and the states’ role in protecting the confidentiality of their data. As of March 2007, only 30 states were participating in the program, and it is unknown if and when the other states will enter the data-sharing agreement. As a result, DVOP and LVER performance information in almost half the states will not include employment outcomes for veterans who found jobs outside the states in which they received services. In addition, other reasons contribute to data variation by state. Labor allows states flexibility in choosing data collection software, which has resulted in some states adapting more quickly than others to the recent changes, depending on their software capabilities. Several Labor officials told us that because of differences in software capabilities, some states’ data may be more reliable than others’.

Available Data Paint an Unclear Picture of Veterans’ Use of One- Stop Services

Labor’s data on veteran job seekers paint an unclear picture of their use of employment and training services in the one-stop system, despite the shared use of common performance measures across programs. Although many veterans use employment services other than those provided by the DVOP and LVER programs, key employment programs vary in how well their data on veteran participants are integrated or shared with other programs. As a result, many states may not know how many veterans use one-stop services. In addition, statutory differences in the way veterans are defined for purposes of program eligibility make it difficult to standardize data across employment programs. Moreover, Labor has no means of assessing whether priority of service for veterans has been implemented in various employment programs.

Veterans Receive Services from Multiple Programs in the One-Stop System

Many veteran job seekers receive employment services from the DVOP and LVER programs. However, some veterans—often the more job- ready—only use one-stop services aimed at the general population, such as the Employment Service and WIA programs. In addition, some veterans use services focused on other subsets of job seekers—such as TAA (see fig. 3). As a result, performance information on many veterans is collected and reported elsewhere in the one-stop system. In fact, 20 states reported that about half or fewer of veteran job seekers who access employment programs receive services from a DVOP specialist or LVER staff, according to our survey (see fig. 4). In addition, some veterans obtain services from more than one employment program in the one-stop system, all of which use the common measures to assess their performance.

Data on Veterans’ Services Are Well Integrated across Some but Not All Programs in the One-Stop System

Performance data on veteran job seekers are well integrated or shared across some key employment and training programs, but not others, despite the mutual use of common measures. As a result, many states may not know how many veterans they serve through the one-stop system. Data on veterans who access the Employment Service are completely integrated with data from the DVOP and LVER programs—they share the same reporting system, and DVOP and LVER data are a subset of Employment Service data. According to our survey, veteran job seekers in most states receive initial assistance from the Employment Service when they access the one-stop system. If they are subsequently referred to the DVOP and LVER programs, all of their information is housed in the same system and an unduplicated count of veterans served between these programs can be obtained. In addition, states are held accountable for meeting separate goals in the Employment Service for veterans and disabled veterans (see app. III). Labor considers these measures to reflect veterans’ outcomes for the entire one-stop system, as they constitute outcomes for all veterans who access the Employment Service, DVOP, and LVER programs. Furthermore, they are the best approximation of a total count of veterans who access the one-stop system that the current data will allow.

On the other hand, data on veterans served by other one-stop programs are not well integrated. States report data to Labor on WIA participants who exit the programs, including veterans, using the Workforce Investment Act Standardized Record Data (WIASRD) system. Although WIASRD contains sufficient information to produce separate veteran outcome data for WIA programs, states are not required to produce separate veteran reports and are not accountable for meeting veteran goals in those programs. In addition, fewer than half the states reported that they routinely match WIA and Employment Service records to attain an unduplicated count of veterans served by those programs. Consequently, veterans who access two different employment services may be counted twice in some cases. Data for TAA participants are reported to Labor in yet another data system, which does allow states to report on the veteran status of participants, but Labor officials told us the agency does not currently use veteran outcomes from that program for any purpose. VETS does not include the veteran outcome data from WIA or TAA in its annual report to Congress, and Labor officials told us they are exploring ways to better use the data.

In addition, data are not always collected on job seekers who use Employment Service or WIA resources without assistance from program staff. These self-assisted job seekers—including veterans—access services such as labor market or career information either in one-stop centers or on home computers, but do not receive active assistance from program staff. Historically, some states have collected information on these job seekers, while others have not. In our survey, 73 percent of states reported that they capture information on all veterans who receive self-assisted services through the Employment Service, while 82 percent of states reported doing so for all veterans who receive self-assisted WIA services. Labor has encouraged—but not mandated—states to collect information on this group of job seekers, but agency officials acknowledged that states continue to vary in how they report such data. Labor officials have expressed concern that requiring veterans who receive self-assisted services to register for the programs might discourage some of them from pursuing the services they need.

Different Veteran Definitions Complicate Efforts to Standardize Data and Implement Priority of Service

Labor and some state officials we surveyed reported that statutory differences in the definitions of veterans for various employment programs make it difficult to standardize data across programs. For the purposes of the DVOP and LVER programs, an eligible veteran is statutorily defined as an individual who served on active duty for more than 180 days. Labor also uses this definition for the Employment Service. WIA, on the other hand, does not specify a length of time in service for a person to be considered a veteran. Moreover, to qualify as a recently separated veteran in the DVOP and LVER programs, a veteran must have left active duty in the last 3 years. By contrast, WIA defines recently separated as having left active duty in the last 4 years (see table 5).

These inconsistent definitions have been difficult for Labor and states to reconcile with the concept of seamless service delivery and have caused some confusion for states as they implement priority of service throughout the one-stop system. While JVA requires that veterans receive priority over other job seekers in Labor-funded employment and training programs, it does not define a veteran for purposes of the priority requirement. Labor has interpreted JVA’s provisions to mean that while veterans are to receive preference in the programs after any other statutory eligibility requirements are met, each program must use its own statutory definition of a veteran in applying that preference. Labor officials told us that one state applied for a waiver in 2006 to use a single definition of veterans for all of its employment and training programs, but Labor’s Solicitor’s Office orally denied the request. In our survey, approximately half of all states reported that the conflicting veteran definitions in various employment programs complicate data entry, referrals to other programs, and the implementation of priority of service. In addition, about a third of the states claimed that the definitions created gaps in services for veteran clients as they moved among employment programs (see fig. 5). For example, if a veteran receives services from WIA and is subsequently referred to the DVOP program but is found ineligible, he or she may become discouraged and stop seeking services altogether.

Almost half of states shared their concerns about different definitions by providing additional comments in our survey, many of which cited the difficulty of providing priority of service under these circumstances. For example, one state responded that different definitions often lead to inappropriate referrals, resulting in poor customer service and frustration for program participants and service providers. Other states focused on the burden that competing definitions placed on data collection and reporting. For example, one state responded that the issue has made it difficult to integrate the state’s Employment Service and WIA data systems, because the different definitions could lead to invalidating the veteran numbers on reports for those programs. Another state cited the difficulty in assessing how many veterans were served by the state, highlighting the complexity of producing an unduplicated count of veterans served by different programs that do not share a single definition. States also cited challenges in dealing with other agencies that are not mandated partners in the one-stop system. For example, two states mentioned that some staff of other agencies’ programs may hesitate to refer participants to the DVOP and LVER programs because they are unsure about participant eligibility. An expert on veterans’ issues in the states concurred that the different eligibility criteria for veterans has been a problem for states and told us that a common veteran definition for employment and training programs would be an improvement.

Labor Has No Means of Evaluating Priority of Service for Veterans

Despite JVA’s mandate, Labor has not produced information on the extent to which veterans receive priority of service in all qualified employment and training programs. Specifically, JVA required Labor to evaluate and report on whether veterans are receiving priority of service and are fully served by its employment programs, as well as whether the representation of veterans in such programs is in proportion to their participation in the labor force. In its fiscal year 2005 report, Labor stated that the participation rate for veterans in its adult programs was approximately 8.4 percent—slightly higher than veterans’ participation rates in the U.S. workforce. In addition, the agency reported that outcomes for veterans served in these programs closely mirrored those of all job seekers in the programs. However, Labor has no method of gauging how—and how consistently—priority of service is actually applied. Labor officials told us that the highly devolved workforce development system makes it very difficult to evaluate priority of service, because different programs have multiple access points and diverse eligibility criteria that prevent Labor from applying a simple measurement technique to each.

States reported that implementing priority of service has been challenging, as has holding one-stop partner programs accountable for serving veterans. To supplement federal guidance on this issue, at least one state has developed its own guidance for implementing and measuring priority of service. Some Regional Directors of VETS told us they encouraged the use of that state’s guidance as a model for assessing priority of service for states in their own regions. We do not know when Labor will develop further guidance on the issue. However, in December 2006, Congress passed the Veterans Benefits, Health Care, and Information Technology Act of 2006, which included a requirement that Labor release regulations on implementing priority of service within 2 years. In addition, the agency has begun planning a study of priority of service in response to our prior report. According to Labor officials, the study will combine a survey of participants with a process evaluation and an analysis of outcomes. Labor does not yet know when the study will get under way.

Labor Has Taken Steps to Better Understand Veterans’ Outcomes, but the Programs’ Impact Remains Unknown

Labor has taken some steps to improve the quality of performance data and better understand veterans’ services and outcomes, but the overall impact of employment services for veterans is unknown. Labor has developed some processes to enhance data quality. For example, Labor’s ETA requires states to validate some data in key programs. Furthermore, Labor plans to implement an integrated data-reporting system that would allow Labor and states to track individual veterans’ progress through different programs in the one-stop system. Additionally, the new system would expand data collection by, for example, collecting more data on services to employers. However, states have raised concerns about the challenge of meeting the system’s planned implementation date, and the timeline for implementation remains unclear. Furthermore, while performance information helps assess whether individuals are achieving their intended outcomes—such as obtaining employment—it cannot measure whether the outcomes are a direct result of program participation, rather than external factors. To measure the effects of a program, it is necessary to conduct an impact evaluation that would seek to assess whether the program itself led to participant outcomes. Labor has sponsored research on services to veterans. However, it has not conducted an impact evaluation, as required under WIA, to assess the effectiveness of one-stop services. Such a study should include impacts for key participant groups, including veterans. We recommended in 2004 that Labor take steps to conduct such an evaluation, but there has been no action to date.

Labor Has Developed Some Processes to Enhance the Quality of Performance Data

Labor has taken some steps to improve the quality of performance data and enhance the understanding of veterans’ services and outcomes. To address data quality concerns, ETA has developed processes requiring states to validate certain data reported for participants in WIA and Wagner-Peyser-funded Employment Service programs. However, while these programs serve veterans, participant records are randomly selected in both programs from the total participant population and, therefore, may not include the records of veteran participants.

Both the WIA data validation process, developed in 2004, and the Employment Service process, developed in 2003, involve two types of data validation, although the WIA process is more intensive, according to Labor officials. Both processes involve (1) data element validation—comparing randomly sampled participant records to source files, and (2) report validation—assessing whether states’ software accurately calculated performance outcomes. While element validation in WIA is conducted on- site with hardcopy source documentation, the Employment Service data validation process is performed centrally and electronically, because Employment Service records are generally electronic. The Employment Service element validation process checks for duplicate or invalid entries in source files by, for example, checking for inconsistencies among various veteran-related fields, such as veteran status and disabled veteran. However, the Employment Service element validation process cannot check the underlying accuracy of the data, because there is no hard copy documentation to prove whether a participant is in fact a veteran.

Labor officials told us that the Employment Service data validation process has been helpful in raising awareness among states about the importance of data quality and that some states have come to see it as a useful tool. Additionally, states responding to our survey generally agreed that it has been effective—38 states, or about 75 percent, rated the Employment Service data validation process as effective in ensuring the accuracy of veteran job seekers’ information. For example, according to one respondent, review of the data validation results is used as a management tool, to highlight successes and to alert staff to weaknesses. Nevertheless, some states have expressed concerns about the data validation processes. Concerns about the process were also raised by state officials in all 3 of the states we visited. For example, officials in 2 of the 3 states noted that they had experienced difficulties adjusting to frequent changes in software before the results were due to Labor. On our survey, 2 states said that the sample size was too small to be meaningful, and 4 states expressed concerns about the fact that the process does not verify the accuracy of the data in source files. These concerns are similar to those we identified in a previous report that addressed the WIA process.

Additionally, Labor has taken steps to address data quality as a part of its routine monitoring and technical assistance. Specifically, beginning in 2004, ETA regional staff have incorporated a data quality component into compliance visits to state offices, which are generally conducted once or twice a year, according to Labor officials. Data validation is just one component of these compliance visits, which typically do not focus on veterans’ data as a separate issue. To support this effort, Labor officials told us that ETA has amended its monitoring guide for these visits to include a section on data validation. According to Labor officials, these visits have been useful in identifying problems and corrective actions. Moreover, ETA and VETS have recently collaborated on a few of these compliance visits. Labor officials said they believed this joint monitoring was beneficial, and expect those efforts to be a model for future joint visits.

There are several other forms of management reviews that generally focus on services to veterans but also offer a chance to review data. For example, VETS regional and state-based staff conduct site visits as part of their routine monitoring, which focus primarily on services to veterans but which can include reviewing performance information as well. Additionally, VETS has required a series of annual assessments—of the program for each state overall, and self-assessments by DVOP specialists, LVER staff, and one-stop managers—that address data issues to a limited extent. State directors use performance data to substantiate services described in the self-assessment. For example, according to one official we spoke with, to confirm a LVER staff’s claim of travel to several job fairs, the director can consult the one-stop’s travel log to substantiate whether the LVER staff actually made the trips.

Beyond the steps Labor has taken, state workforce agencies also perform functions that affect performance data on services to veterans. Most states responding to our survey reported that they have taken certain steps to ensure the accuracy and reliability of data for the Employment Service, DVOP, and LVER programs, such as having their systems perform automated checks for inconsistencies in data or for duplicate veteran files (see fig. 6).

Labor’s Proposed Integrated Data System Could Improve the Understanding of Services and Outcomes for Veterans

Since 2004, Labor has been planning to implement an integrated data reporting system that could greatly enhance the understanding of veterans’ services and outcomes. In 2004, Labor first proposed a single, streamlined reporting system, known as the ETA Management Information and Longitudinal Evaluation system (EMILE) that would have replaced reporting systems for several Labor programs. Labor substantially modified this system’s design in response to concerns raised by state and local agencies about the burden and cost of the new system, as well as the challenge of meeting the implementation deadline. The modified system, now called the Workforce Investment Streamlined Performance Reporting System (WISPR), was planned with a July 2007 implementation date.

WISPR has been designed to both integrate and expand data reporting. If implemented, the system would integrate data reporting by using standardized reporting requirements across the Employment Service, DVOP and LVER, WIA, and TAA programs, and ultimately replace their preexisting reporting systems with a single reporting structure. Additionally, it would rely on a standardized set of data elements and quarterly reports to provide data on participant characteristics and services provided, as well as performance outcomes based on the common measures. Its integrated design would, for the first time, allow Labor and states to track individual veterans’ progress through the one-stop system.

In addition, the system would expand data collection and reporting in two key areas: the services that LVER staff provide to employers, a key aspect of the LVER role on which Labor currently collects no data, and estimates of the population of veterans who access the one-stop system but ultimately receive limited or no services from one-stop staff.

As with EMILE, however, concerns have been raised about challenges in implementing the new system, and at present, the timeline for WISPR’s implementation remains unclear. Some of the comments received by OMB during the official comment period noted the challenge of a July 2007 implementation date, according to a Labor official. While states will have a 2-year period to consolidate reporting on the full range of programs, they are expected to begin collecting and reporting data in the new format immediately. As of December 2006, 39 entities, including state workforce agencies, local agencies and unions, had submitted comments reflecting their concerns about WISPR to the Office of Management and Budget (OMB). Of the 20 states that submitted comments, 14 noted that a July 2007 implementation date would represent a challenge. For example, some of them expressed the view that Labor had underestimated the time states would need to revise policy, reprogram systems, and retrain staff. In addition, some states expressed concerns about their ability to provide data on services to employers. Moreover, two states expressed the concern that meeting Labor's planned implementation date would have adverse consequences, such as compromised data quality or cost overruns. OMB’s official review will address the time needed to build the reporting system’s technical infrastructure, and will play a key role in deciding the system’s final implementation schedule, according to a Labor official. States and local areas will need enough time to fully meet the requirements of this expanded data collection.

Little Is Known about the Impact of One-Stop Services, including Those to Veterans

Although Labor has improved its outcome data on job seekers who participate in its programs, these data alone cannot measure whether outcomes are a direct result of program participation, rather than external factors. For example, local labor market conditions may affect an individual’s ability to find a job as much or more than participation in an employment and training program. To measure the effects of a program, it is necessary to conduct an impact evaluation that would seek to assess whether the program itself led to participant outcomes. Labor has not conducted an impact evaluation of one-stop services, including those to veterans. However, the department did sponsor a study, issued in 2003, that examined the relationship between services provided to certain groups of veterans and employment and earnings outcomes. This study employed a number of data sources and statistical techniques to learn more about how veterans were using one-stop services. However, while this study provided some useful information, it could not determine that these services caused positive outcomes for veteran job seekers. In addition, the study relied on data from 8 states and its findings could not be generalized to the national population of veteran job seekers.

Since the full implementation of WIA in 2000—in which the one-stop system became the required means to provide employment and training services, including those to veterans—Labor has not made evaluating the impact of those services a research priority. While WIA required one such evaluation by 2005, Labor has declined to fund one in prior budgets. In a 2004 report, we recommended that Labor comply with the requirements of WIA and conduct an impact evaluation of WIA services to better understand what services are most effective for improving employment- related outcomes. In response to our report, Labor cited the need for program stability and proposed delaying an impact evaluation of WIA until any changes that might be included in reauthorization legislation had been implemented. While efforts to reauthorize WIA began in 2003, they have stalled and it is not clear at this time when they will be complete. Furthermore, OMB has also found Labor’s evaluations of WIA services to be lacking. In response, in its 2008 budget proposal, Labor identified an assessment of WIA’s impact on employment, retention, and earnings outcomes for participants as an effort the agency would begin. According to Labor officials, the agency has not yet begun to design the study. Such a study should include impacts for key participant groups, including veterans. To do so would require a sufficient sample of veterans to allow such analysis.

Conclusions

At a time when the nation’s attention is focused on those who have served their country, it is vital that Congress and the Administration are able to make informed decisions about programs that help veterans find and keep jobs in the civilian labor market. Frequent changes in Labor’s performance accountability system have hampered Labor’s ability to produce consistent and meaningful performance information on veteran job seekers. States and local areas have had difficulty implementing the constant changes to performance information, which introduce error and make it difficult to identify trends that would give Congress a better idea of the programs’ achievements. While the anticipated transition to a new reporting system represents a promising advance in Labor’s ability to track the outcomes of veterans in the one-stop system, states will need time to effectively implement the changes to avoid compromising the potential benefits— such as improved data quality—of the system.

Furthermore, the current separate performance measures for the DVOP and LVER programs do not account for the considerable similarity in veteran populations served by DVOP specialists and LVER staff, and thus do not provide an accurate picture of outcomes for veterans served by these two programs. Using the existing measures, Labor also cannot ensure that performance outcomes give more weight to services for veterans with greater barriers to employment. In addition, different veteran definitions in other programs could make it difficult to analyze services to veterans throughout the one-stop system. Further, Labor cannot provide assurance that veterans are appropriately given service priority by programs in the one-stop system, or that services to veterans are truly effective. The federal government spends about $155 million each year on the DVOP and LVER programs alone, not counting the amounts spent on veterans who use other one-stop programs, but there is no information on whether these programs have an impact in helping this important population. Establishing a means to gauge the programs’ impact would require a considerable investment of time and money, but would contribute greatly to the understanding of whether current employment and training services are meeting veterans’ needs. Furthermore, we continue to urge Labor to meet WIA requirements and our 2004 recommendation to conduct an impact evaluation of one-stop services.

Recommendations for Executive Action

To provide a better picture of services and outcomes for veteran job seekers, improve program reporting, and facilitate priority of service, we recommend that the Secretary of Labor ensure that states are given adequate direction and sufficient time to implement ETA’s planned integrated data reporting system and make necessary changes; consolidate all performance measures for the DVOP and LVER programs, including those for disabled and recently separated veterans; comply with JVA’s requirement to implement a weighting system for the DVOP and LVER performance measures that takes into account the difficulty of serving veterans with particular barriers to employment; develop legislative proposals for appropriate changes to the definitions of veterans across employment and training programs to ensure consistency; and ensure that Labor moves forward with an impact evaluation for the one-stop system under WIA as we recommended in 2004, and that the evaluation’s sampling methodology includes veterans in sufficient numbers to allow analysis of the impact of services to veterans in the one-stop system, including those served by the DVOP and LVER programs.

Agency Comments and Our Evaluation

We provided a draft of this report to Labor for review and comment. In its comments, Labor generally concurred with our findings, conclusions, and recommendations and expressed appreciation that the report acknowledges the steps the agency has taken to improve the quality of performance data and better understand outcomes for veterans. Labor noted that it is considering adopting a different approach to measuring outcomes for the DVOP and LVER programs by program year 2008—one that may take into account the similar veteran populations served, as well as outreach to employers. As it develops this new approach, Labor reported that it will also introduce a system of weighted measures that will emphasize services to veterans with barriers to employment. These changes will coincide with the implementation of Labor’s proposed integrated data system, WISPR. Labor also noted that it would work with states and grantees to ensure a smooth transition to the new system. In addition, Labor stated that it intends to pursue a WIA impact evaluation, which will allow for analysis of services to sub-populations, including veterans. Labor reported that our recommendation to develop proposals for changing veteran definitions across employment and training programs must be evaluated with the input of other agencies. Labor also provided technical comments that we incorporated where appropriate. Labor’s comments are reproduced in full in appendix IV.

We will send copies of this report to the Secretary of Labor, relevant congressional committees, and other interested parties and will make copies available to others upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov.

A list of related GAO products is included at the end of this report. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or at nilsens@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other contacts and staff acknowledgments are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

The objectives of this report were to determine (1) the extent to which DVOP and LVER performance information reflects services and outcomes for veterans served by these programs, (2) the extent to which performance information on veterans served by other key programs is comprehensive and well integrated across programs in the one-stop system, and (3) what Labor is doing to improve the quality of performance data and better understand outcomes for veteran job seekers.

To address these objectives, we conducted a nationwide Web-based survey to state workforce administrators in the 50 states and the District of Columbia; conducted site visits to 3 states, during which we interviewed state and federal officials, one-stop managers, and program staff; interviewed Labor officials from both the Veterans’ Employment and Training Service (VETS) and the Employment and Training Administration (ETA); analyzed relevant performance data from ETA and VETS; and reviewed our previous work on attributes of successful performance measures.

We conducted our work in accordance with generally accepted government auditing standards between May 2006 and April 2007.

Survey

To obtain further information on our objectives, we surveyed state workforce administrators from November 15 to December 27, 2006. The survey addressed all three objectives and included questions about performance information for the DVOP and LVER programs, integration of data across employment programs serving veterans, and efforts to ensure data quality.

We developed the survey based on knowledge obtained during our preliminary research. This included a literature review and initial interviews with officials from the Department of Labor, the National Association of State Workforce Agencies (NASWA), and the state of New Hampshire, where we conducted our initial site visit. We then obtained a list of state workforce administrators from NASWA. We asked state administrators to provide information on the DVOP and LVER programs’ capacity, other programs within the one-stop system that serve veteran job seekers, performance measures and data; and challenges to managing the programs. To determine whether respondents would understand the questions as intended, we pretested the survey with state officials in 5 states. We then made changes to the questions based on comments we received during the pretests.

The survey was conducted using self-administered electronic Web-based questionnaires. We sent notification of the survey to the 50 states and the District of Columbia in November 2006 and followed up with e-mail messages and telephone calls as necessary during November and December. All 51 recipients submitted their responses by the end of December 2006, providing us with a response rate of 100 percent. We did not independently verify information obtained through the survey. During our data analysis we held three follow-up conversations to fill in gaps from incomplete survey information.

Because this survey was not a sample survey, there are no sampling errors. However, the practical difficulties of conducting any survey may introduce errors, commonly referred to as nonsampling errors. For example, difficulties in how a particular question is interpreted, in the sources of information that are available to respondents, or how the data are entered into a database can introduce unwanted variability into the survey results. We took steps during survey development, data collection, and data analysis to minimize these nonsampling errors. For example, we pretested the questionnaire to ensure that questions were clear and understandable. Since this was a Web-based survey in which respondents entered their responses directly into out database, there was little possibility of data entry error. During data analysis, a second, independent analyst checked all computer programming. Also, to the extent possible, we compared both closed and open ended survey responses with our site visit observations. While survey results are also subject to different types of systematic errors or bias, we do not have reason to believe that respondents falsely reported any information for this survey.

Site Visits

To obtain a detailed understanding of how veteran job seekers are served by the one-stop system and how their information is captured, we conducted visits to three states: New Hampshire, California, and Tennessee. We selected these states based on a range of selection criteria, including geographic dispersion, state size and veteran demographics, recent state performance in veterans’ programs, and recommendations by Labor and NASWA. Our site visits at the state level included interviews with state workforce agency officials and state directors of Veterans’ Employment and Training. We also chose two local one-stops in each state and met with local managers and veteran program staff (see table 6).

During each interview, we used standard interview protocols to obtain detailed and comparable information. In our interview with state workforce officials, we discussed the role of the state workforce agency in administering veterans’ employment and training programs, details about the programs serving veteran job seekers, views on the current performance accountability system, and information about data collection and validation. In our interviews with the state directors and their staff, we discussed their oversight roles and responsibilities, relationship with the state workforce agency, and views on the current performance accountability system and data collection. At the local one-stops, we discussed the coordination of veteran staff with other programs within the one-stop system, priority of service, and data collection and reporting. In each state, we also received a tutorial of the state’s data collection software. We conducted our site visits between July and November 2006.

Research and Experts

As part of our work, we interviewed officials of ETA and VETS, including all six Regional Administrators of VETS. We conducted these telephone interviews in the following locations: Boston, Atlanta, Dallas, Chicago, San Francisco, and Philadelphia. During each interview, we obtained information on regional differences in administering the DVOP and LVER programs, views on the current performance measures, and information on Labor’s monitoring role in each state.

We also analyzed performance data from the DVOP, LVER, and Employment Service programs and reviewed Labor’s guidance. In addition, we reviewed relevant literature, including our past work on attributes of successful performance measures. We also interviewed representatives of NASWA and two private-sector staffing agencies.

Appendix II: Summary of State Performance in the DVOP and LVER Programs for Benchmark Program Year 2005 and Negotiated Goals for Program Year 2006

baseline performance data by state for the DVOP and LVER programs from benchmark program year 2005 (July 1, 2005–June 30, 2006) and negotiated goals by state for the following year, program year 2006.

Labor and states did not negotiate goals for the DVOP or LVER programs for program year 2005, which was a baseline year for performance under the new common measures. Four performance measures contribute to each program’s performance. For the DVOP program, there is one set of measures for all veterans and one set for disabled veterans. For the LVER program, there is a set of measures for all veterans and another set for recently separated veterans. Each set of measures includes entered employment rate (EER): the number of participants who are employed in the first quarter after the exit quarter divided by the number of participants who exit during the quarter and employment retention rate (ERR): the number of participants who are employed in both the second and third quarters after the exit quarter divided by the number of adult participants who exit during the quarter.

These figures were provided by the Department of Labor. GAO has not verified the accuracy or reliability of these data.

Appendix III: Summary of State-Negotiated Goals and Performance for Veterans in the Employment Service, Program Year 2005

This table illustrates the negotiated goals and performance achieved by each state for program year 2005 for veterans in the Wagner-Peyser-funded Employment Service. It includes the entered employment and employment retention rates for all veterans and disabled veterans within the Employment Service, including those in the DVOP and LVER programs.

These figures were provided by the Department of Labor. GAO has not verified the accuracy or reliability of these data.

Appendix IV: Comments from the Department of Labor

Appendix V: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

Dianne Blank, Assistant Director Rebecca Woiwode, Analyst-in-Charge Chris Morehouse and Beth Faraguna made significant contributions to this report in all facets of the work. In addition, Walter Vance assisted in the design of the national survey; Gloria Hernandez-Saunders helped with data analysis; Meeta Engle lent subject matter expertise; Jessica Botsford and Richard Burkard provided legal support; and Charlie Willson provided writing assistance.

Related GAO Products

Trade Adjustment Assistance: Labor Should Take Action to Ensure Performance Data Are Complete, Accurate, and Accessible. GAO-06-496. Washington, D.C.: April 25, 2006.

Veterans’ Employment and Training Service: Greater Accountability and Other Labor Actions Needed to Better Serve Veterans. GAO-06-357T. Washington, D.C.: February 2, 2006.

Veterans’ Employment and Training Service: Labor Actions Needed to Improve Accountability and Help States Implement Reforms to Veterans’ Employment Services. GAO-06-176. Washington, D.C.: December 30, 2005.

Workforce Investment Act: Labor and States Have Taken Actions to Improve Data Quality, but Additional Steps Are Needed. GAO-06-82. Washington, D.C.: November 14, 2005.

Veterans’ Employment and Training Service: Preliminary Observations on Changes to Veterans’ Employment Programs. GAO-05-662T. Washington, D.C.: May 12, 2005.

Performance Measurement and Evaluation: Definitions and Relationships. GAO-05-739SP. Washington, D.C.: May 2005.

Workforce Investment Act: States and Local Areas Have Developed Strategies to Assess Performance, but Labor Could Do More to Help. GAO-04-657. Washington, D.C.: June 1, 2004.

Performance Reporting: Few Agencies Reported on the Completeness and Reliability of Performance Data. GAO-02-372. Washington, D.C.: April 26, 2002.

Veterans’ Employment and Training Service: Flexibility and Accountability Needed to Improve Service to Veterans. GAO-01-928. Washington, D.C.: September 12, 2001.

Veterans’ Employment and Training Service: Proposed Performance Measurement System Improved, but Further Changes Needed. GAO-01-580. Washington, D.C.: May 15, 2001.

Veterans’ Employment and Training Service: Strategic and Performance Plans Lack Vision and Clarity. GAO/T-HEHS-99-177. Washington, D.C.: July 29, 1999.

Veterans’ Employment and Training Service: Assessment of the Fiscal Year 1999 Performance Plan. GAO/HEHS-98-240R. Washington, D.C.: September 30, 1998.

Veterans’ Employment and Training: Services Provided by Labor Department Programs. GAO/HEHS-98-7. Washington, D.C.: October 17, 1997.

The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans. GAO/GGD-10.1.20. Washington, D.C.: April 1998.

Agencies’ Annual Performance Plans under the Results Act: An Assessment Guide to Facilitate Congressional Decisionmaking. GAO/GDD/AIMD-10.1.18. Washington, D.C.: February 1998.

Executive Guide: Effectively Implementing the Government Performance and Results Act. GAO/GGD-96-118. Washington, D.C.: June 1996.