Veterans' Employment and Training Service: Labor Could Improve Information on Reemployment Services, Outcomes, and Program Impact
Published: 2007-05-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 2002, Congress enacted the Jobs for Veterans Act (JVA), which modified two Department of Labor (Labor) programs that specifically target veteran job seekers: the Disabled Veterans' Outreach Program (DVOP) and the Local Veterans' Employment Representative (LVER) program. However, questions have been raised about the adequacy of performance information on services to veterans by these and other employment programs. In this report, GAO examined (1) the extent to which DVOP and LVER performance information reflects services and outcomes for veterans; (2) the extent to which performance information on veterans paints a clear picture of their use of one-stop services; and (3) what Labor is doing to improve the quality of performance data and better understand program impact and outcomes for veterans.
section_title: What GAO Found
paragraphs: Performance information for the DVOP and LVER programs provides some sense of services and outcomes for veterans, but is weakened by several factors. In July 2005, Labor adopted new performance measures for the programs, but not all have been fully implemented. For example, states are held accountable for helping veterans get and keep jobs, but are not yet held accountable for their average earnings once employed, as they are for other programs. Additionally, having separate performance measures for the DVOP and LVER programs fails to acknowledge the similarity of the populations they serve and duties they perform. Furthermore, it is difficult to assess outcomes over time or across states because of frequent changes in reporting requirements that prevent establishing reliable trend data. Labor's data on veteran job seekers paint an unclear picture of their use of other employment and training services in the one-stop system, despite the use of common performance measures across programs. Although many veterans use services other than those provided by the DVOP and LVER programs, key employment programs vary in how well their data on veteran job seekers are shared across programs, making it difficult to know how many veterans are served. In addition, statutory differences in the definitions of veterans hinder efforts to standardize data across employment programs. Moreover, Labor has no means of assessing whether priority of service for veterans has been implemented in various employment programs. Labor has taken some steps to improve the quality of performance data and better understand outcomes for veterans. For example, Labor requires states to validate key performance data. Labor has also planned an integrated data reporting system that would track individual veterans' progress through the one-stop system. However, states have raised concerns about the timelines and its current implementation date is unclear. Furthermore, while outcome information on veterans is helpful, it cannot measure whether the outcomes are due to the program or other factors. While Labor has sponsored research on services to veterans, it has not yet conducted the impact evaluation required by law to assess the effectiveness of one-stop services.

section_title: Letter
paragraphs: Approximately 700,000 veterans are unemployed in any given month, and  Approximately 700,000 veterans are unemployed in any given month, and  roughly 200,000 service members leave active duty and transition to  roughly 200,000 service members leave active duty and transition to  civilian life each year, according to the Department of Labor (Labor).  civilian life each year, according to the Department of Labor (Labor).  While their unemployment rate is similar to that of the general population,  While their unemployment rate is similar to that of the general population,  some veterans have special needs in transitioning to the civilian labor  some veterans have special needs in transitioning to the civilian labor  market. Labor has long provided assistance to veterans to help them  market. Labor has long provided assistance to veterans to help them  succeed in this endeavor. In 2002, Congress enacted the Jobs for Veterans  succeed in this endeavor. In 2002, Congress enacted the Jobs for Veterans  Act (JVA) to improve employment and training services for veterans by  Act (JVA) to improve employment and training services for veterans by  better integrating them into the comprehensive service delivery system, or  better integrating them into the comprehensive service delivery system, or  one-stop system, created under the Workforce Investment Act (WIA) of  one-stop system, created under the Workforce Investment Act (WIA) of  1998. JVA modified two Labor programs that specifically target veteran job  1998. JVA modified two Labor programs that specifically target veteran job  seekers and that are administered by the Veterans’ Employment and  seekers and that are administered by the Veterans’ Employment and  Training Service (VETS)—the Disabled Veterans’ Outreach Program  Training Service (VETS)—the Disabled Veterans’ Outreach Program  (DVOP) and the Local Veterans’ Employment Representative (LVER)  (DVOP) and the Local Veterans’ Employment Representative (LVER)  program—and consolidated the funding into a single grant to states. These  program—and consolidated the funding into a single grant to states. These  two programs were funded at about $155 million in fiscal year 2006, and  two programs were funded at about $155 million in fiscal year 2006, and  served over 715,000 veterans nationwide in the most recent program year.  served over 715,000 veterans nationwide in the most recent program year.  JVA addressed concerns raised by some that the programs were overly  JVA addressed concerns raised by some that the programs were overly  prescriptive and did not provide states the flexibility to determine the best  prescriptive and did not provide states the flexibility to determine the best  way to serve veteran job seekers. Instead, JVA identified broad roles and  way to serve veteran job seekers. Instead, JVA identified broad roles and  responsibilities of DVOP and LVER staff while giving states flexibility to  responsibilities of DVOP and LVER staff while giving states flexibility to  determine the number of staff for each program. In addition, the law  determine the number of staff for each program. In addition, the law  required the implementation of a comprehensive performance  required the implementation of a comprehensive performance  accountability system that included performance measures for the two  accountability system that included performance measures for the two  programs—consistent with those under WIA—and enhanced  programs—consistent with those under WIA—and enhanced  accountability for veteran services in the one-stop system. Moreover,  accountability for veteran services in the one-stop system. Moreover,  many veterans receive reemployment services from programs other than  many veterans receive reemployment services from programs other than  the DVOP and LVER, and JVA stipulated that veterans served by these  other programs be given preference over nonveterans, as long as they  meet the eligibility requirements. These programs—administered by a  separate office within Labor, the Employment and Training Administration  (ETA)—include the WIA Adult and Dislocated Worker programs, the  Wagner-Peyser-funded Employment Service, and other key employment  programs.
To implement JVA, VETS adopted a set of common performance measures  in July 2005, similar to those adopted by other employment and training  programs. Each quarter, states report a wide range of data to Labor on  demographics, services, and outcomes for veterans in the DVOP and LVER  programs. States are specifically held accountable for certain employment- related outcomes for all veterans—whether veteran participants get and  keep jobs—and separately, for the outcomes of some disabled and  recently separated veterans. JVA requires Labor to include information in  its annual report to Congress on whether veterans are receiving priority  and are being fully served by employment and training programs.  However, questions have been raised about whether available  performance information accurately reflects services and outcomes for  veterans. Moreover, outcome information alone cannot be used to  evaluate the effectiveness—or impact—of employment services for  veterans. Program outcomes may be attributable to factors other than a  program’s services, such as local labor market conditions. In view of these  concerns and the changes to the performance accountability system  resulting from the Jobs for Veterans Act, you requested that we review the  performance information collected on employment and training programs  that serve veterans. Specifically, we examined (1) the extent to which  DVOP and LVER performance information reflects services and outcomes  for veterans served by these programs; (2) the extent to which  performance information on veterans served by other key programs paints  a clear picture of veterans’ use of one-stop services and (3) what Labor is  doing to improve the quality of performance data and better understand  outcomes for veteran job seekers.
To determine the extent to which available performance information  reflects services and outcomes for veteran job seekers, we administered a  nationwide Web-based survey to state workforce administrators in all   50 states and the District of Columbia. We received responses from all   51 administrators. In addition, we analyzed performance data from the  relevant programs and reviewed Labor’s program guidance. To better  understand state and local variation in employment programs serving  veterans, we conducted site visits to three states: New Hampshire,  California, and Tennessee. We selected these states based on a range of  criteria, including geographic dispersion, state size and veteran  demographics, recent state performance in veterans’ programs, and  recommendations by Labor and the National Association of State  Workforce Agencies. Our site visits included interviews with state  workforce agency officials, the federal Director of Veterans’ Employment  and Training for each state, one-stop managers, and DVOP and LVER  program staff. Additionally, we interviewed Labor officials from both  VETS and ETA. As part of this effort, we conducted interviews with all six  Regional Administrators of Veterans’ Employment and Training to help  understand regional variation in the programs. Finally, we reviewed  literature on attributes of successful performance measures, including our  previous work. For a detailed discussion of our scope and methodology,  see appendix I. We conducted our review from May 2006 to April 2007 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: The nation’s veteran job seekers receive employment and training services  from programs overseen by two agencies within Labor—the Veterans’  Employment and Training Service and the Employment and Training  Administration. General employment services fall under the purview of  ETA, which administers the Wagner-Peyser-funded Employment Service  program, providing a national system of public employment services to all  individuals seeking employment—including veterans. Thus, those veterans  considered job ready and not in need of intensive services may be served  by Employment Service staff and receive such services as assessment,  counseling, job readiness evaluation, and placement. ETA carries out its  Employment Service program through workforce agencies in each state.
In fiscal year 2006, the Employment Service program provided a total of  about $716 million to states.
While ETA administers programs that serve the general population,  including veterans, VETS administers the DVOP and LVER programs,  which focus exclusively on serving veterans, often providing more  intensive services than the Employment Service does. Like ETA, VETS  carries out its responsibilities through a nationwide network that includes  representation in each of Labor’s six regions and staff in each state. The  Office of the Assistant Secretary for VETS administers the agency’s  activities through regional administrators and state directors. The DVOP  specialists and LVER staff, whose positions are funded by VETS, are part  of states’ public employment services. In fiscal year 2006, the DVOP and  LVER programs were funded at about $155 million. In the most recent  program year—program year 2005, which spanned July 1, 2005, to June 30,  2006—the Employment Service, together with the DVOP and LVER  programs, reported serving about 1.32 million veterans nationwide, of  whom over 715,000 were served by DVOP specialists and LVER staff.
The Employment Service and the DVOP and LVER programs are  mandatory partners in the one-stop system under WIA—where services  are provided by a range of employment and training programs in a single  location. Veterans, along with other eligible job seekers, may receive  services from other mandatory one-stop partners, such as WIA-funded  training or Trade Adjustment Assistance. Additionally, job seekers,  including veterans, may use the one-stop centers’ computers and other  resources without staff assistance, and in many places may access one- stop services online from home. Department of Veterans Affairs (VA)  programs are not mandatory partners in the one-stop system, but do  participate at some locations.
subsections: section_title: Jobs for Veterans Act Reforms
paragraphs: In 2002, the Jobs for Veterans Act amended Title 38 of the U.S. Code--  which governs the DVOP and LVER programs—and by doing so,  introduced an array of reforms to the way employment and training  services are provided to veterans. JVA sought to address concerns that  the programs were overly prescriptive by providing states with enhanced  flexibility to determine the best way to serve veteran job seekers. Among  its reforms, JVA  redefined the DVOP specialist and LVER staff roles but gave states  flexibility in deciding their duties;    established a single state grant and a new funding formula that  allowed states to determine the mix of DVOP specialists and LVER  staff;  required a comprehensive performance accountability system  consistent with WIA performance measures;  required that veterans receive priority over other job seekers in all  Labor job training programs, not just the Employment Service; and  required that VETS include information in its annual report to  Congress on employment services to veterans throughout the   one-stop system.
JVA identified broad roles and responsibilities of DVOP specialists and  LVER staff. For example, DVOP specialists are to focus on providing  intensive services to eligible veterans, giving priority to disabled veterans  and those with other barriers to employment. LVER staff are to focus on  conducting outreach to employers to assist veterans in gaining  employment, as well as facilitating employment, training, and placement  services given to veterans. State workforce agencies receive a single  veterans’ program grant to fund both programs; the amount each state  receives is determined in part by the size of the veteran population within  each state. State agencies then decide how to distribute the amount they  receive between the two programs. Table 1 lists selected responsibilities  of DVOP specialists and LVER staff as set forth in Labor guidance.
JVA also stipulated that veteran job seekers must receive priority over  other job seekers in any job training program administered by Labor.  Labor’s guidance requires states to explain how veterans will be given  priority and how veterans’ services will be provided through the state’s  one-stop system. For programs that target particular populations, such as  seniors or low-income individuals, veterans’ priority is applied after any  other mandatory eligibility provisions are met.
section_title: Changes to the DVOP and LVER Performance Information
paragraphs: Like other Labor employment and training programs, the DVOP and LVER  programs have experienced changes both in the way outcomes are tracked  and in the measures used to assess performance. Specifically, in 1998, WIA  required that states use automated unemployment insurance wage records  to track employment-related outcomes. Formerly, to obtain data on  outcomes, states relied on a manual follow-up process using  administrative records or contacts with job seekers. To conform to WIA,  VETS moved from such a manual follow-up system to the new automated  process in 2002.
The measures that Labor uses to assess performance in the DVOP and  LVER programs have also changed over time, gradually reflecting more  emphasis on outcome-based measures. Before passage of the JVA in 2002,  for example, some of the measures used for the DVOP and LVER programs  focused more on services received—such as the number of veterans in  training or receiving counseling—than on outcomes achieved. In 2002, JVA  required that Labor develop a comprehensive performance accountability  system and required that the new system measure performance in a way  that is consistent with WIA. In 2003, VETS adopted performance measures  for the DVOP and LVER programs based on those then used in WIA. In  2005, in response to an Office of Management and Budget (OMB) initiative,  Labor began requiring states to implement common performance  measures for its employment and training programs, including the DVOP  and LVER programs, the Employment Service, and WIA. OMB established  a set of common measures to be applied to most federally funded job  training programs that share similar goals. Labor further defined the  common measures for all of its Employment and Training Administration  programs, applying three measures to each of its adult programs (see   table 2).
In applying the common measures to its programs, VETS also developed  additional measures to emphasize outcomes for disabled veterans in the  DVOP program and outcomes for recently separated veterans in the LVER  program.
section_title: Data Reporting
paragraphs: Labor collects performance data for the DVOP and LVER programs on a  quarterly basis from state workforce agencies. The state agencies use  report formats developed by Labor to provide detailed tabulations of  aggregate information on the characteristics of veteran participants,  services, and outcomes for the two programs, including data showing  states’ performance using the common measures. The state agencies  provide this information to Labor in three separate reports: one for the  DVOP program, one for the LVER program, and one representing an  unduplicated count for both programs.
Furthermore, Labor collects additional information on veterans who  participate in other Labor programs. For example, ETA collects  performance data for the Employment Service on all participants on a  quarterly basis from state workforce agencies, and these reports break out  services and outcomes for veteran participants. States submit their  quarterly reports for the Employment Service and the DVOP and LVER  programs through the same Labor reporting system.
section_title: The Role of Impact Evaluations in Assessing Program Outcomes
paragraphs: Information on the services a program has provided and the outcomes  obtained by program participants are necessary to assess program  impacts. However, this information is not sufficient to measure program  impacts—the outcomes may be due to other external factors such as local  labor market conditions. While impact evaluations allow one to isolate a  program’s effect on the outcomes of participants, there are several  approaches to conducting such evaluations. The experimental method is  often considered the most rigorous method for conducting impact  evaluations. In the experimental method, participants are randomly  assigned to two groups—one that receives a program service (or  treatment) and one that does not (control group). The resulting outcome  data on both groups are compared and the difference in outcomes  between the groups is taken to demonstrate the programs impact.  However, it is not always feasible to use the experimental method for  assessing program impacts. Alternatively, researchers may use a quasi- experimental approach in which program participation is not randomly  assigned. One approach, often called a comparison group study, compares  outcome data for individuals who participated in the program with data on  others who did not participate for various reasons. In a comparison group  study, it is important to find ways to minimize, or statistically control for,  any differences between the two groups. According to OMB, well-matched  comparison group studies, under certain circumstances, can approach the  rigor of the experimental method, and it recommends considering this  method if random assignment is not feasible or appropriate.
Under WIA, Labor was required to conduct at least one impact evaluation  of program services by 2005. In a 2004 report, we found that Labor had not  yet begun such an evaluation, and recommended that the agency comply  with this statutory requirement and help federal, state, and local policy  makers understand what services are most effective for improving  employment-related outcomes.
section_title: DVOP and LVER Performance Information Is Weakened by Several Factors
paragraphs: The DVOP and LVER programs’ performance information is weakened by  several factors, including implementation challenges and frequent changes  to performance reporting requirements. In July 2005, Labor implemented  new performance measures, which provide information on some  outcomes for veterans. However, not all performance measures have been  fully implemented. Additionally, neither the performance measures nor the  data reported to Labor reflect the full range of services that DVOP  specialists and LVER staff provide to veteran job seekers. Furthermore, it  is difficult to assess outcomes over time or across states because of  frequent changes in states’ reporting requirements that prevent  establishing reliable trend data.
subsections: section_title: The New Performance Measures Provide Information on Some Outcomes for Veterans, but Have Not Been Fully Implemented
paragraphs: In July 2005, the DVOP and LVER programs adopted the Office of  Management and Budget’s common measures, along with other  employment programs, including WIA and the Employment Service.  Specifically, states implemented measures that track whether veterans  obtain and keep jobs after receiving services through these programs, but  they have not yet implemented a measure to track veterans’ earnings.  States are held accountable for four separate measures in each program  that focus on outcomes attained by veterans (see table 3). For the DVOP  program, states are held accountable for employment and retention for all  veterans served by the program, as well as for disabled veterans. For the  LVER program, states are assessed on employment and retention for all  veterans, as well as for recently separated veterans. Currently, all states  collect and report data to Labor for calculating performance attainment  and negotiating state goals for these eight measures.
However, states are not yet held accountable for an additional common  measure—veterans’ average earnings—in either the DVOP or the LVER  programs. Other employment and training programs, such as WIA and the  Employment Service, include an average earnings measure for which  states are accountable. For the DVOP and LVER programs, however,  calculating the average earnings was not as straightforward as Labor had  anticipated. A VETS official told us that the agency will calculate baseline  data for average earnings during the current program year, but Labor will  not establish goals and states will not be held accountable for their  performance on this measure until the following year—program year  2007—at the earliest.
Furthermore, Labor has not adopted a system to give more weight to  successful outcomes for veterans who have substantial barriers to  employment, such as a disability. JVA required Labor to weight  performance measures to provide special consideration to veterans  requiring intensive services, as well as disabled veterans. Such a weighting  system would compensate for the fact that veterans with barriers to  employment may need more assistance than others in finding jobs. It  would also provide an incentive for program staff to help veterans with  severe barriers to employment. For example, if a veteran has a disability  and requires intensive case management services, his or her successful  outcomes would have a greater effect on a state’s overall performance  than those of other veterans with fewer barriers. Following JVA’s  enactment, Labor formed a work group to develop a weighting system for  the DVOP and LVER performance measures. On the basis of the group’s  work, the agency issued guidance to introduce the weighted measures to  states in June 2003, with the expectation of implementing them soon after.  However, after further review, a Labor official told us the agency did not  implement the weights in order to give states time to fully implement other  reporting changes. At this time, it is not clear whether Labor will  implement this system in the future.
section_title: Performance Measures and the Data Reported to Labor Do Not Reflect the Full Range of Staff Services
paragraphs: Although DVOP specialists and LVER staff perform similar duties for all  types of veterans in most states, the current performance measures hold  the two programs accountable for different groups of veterans. JVA and  Labor’s guidance outline the key responsibilities and target populations for  DVOP specialists and LVER staff, but also allow for some flexibility in  their roles and responsibilities. Both DVOP and LVER staff are expected to  serve the general veteran population, but DVOP specialists are also  expected to target their services toward veterans who have greater  barriers to employment and need intensive case management, including  disabled veterans. JVA specifies that LVER staff focus on conducting  outreach to employers and assisting all veteran job seekers. In addition,  Labor has recently added the expectation that LVER staff focus their  responsibilities on assisting recently separated veterans. As a result of  these expectations, Labor separately holds DVOP specialists accountable  for the outcomes achieved by the disabled veterans they serve, and LVER  staff for the outcomes of the recently separated veterans they serve.
In practice, however, both programs’ staff serve similar veteran  populations. In program year 2005, for example, 14 percent of veterans  served by the DVOP program were disabled and 21 percent were recently  separated. For the LVER program, 10 percent of veterans served were  disabled and 19 percent were recently separated (see table 4). States  acknowledged this similarity in our survey as well. Over a third of states  responded that DVOP and LVER staff are equally likely to serve disabled  veterans, while about half of states responded that the two programs’ staff  are equally likely to serve recently separated veterans.
In addition to finding similarity in populations served, we also found some  similarity in activities carried out by DVOP and LVER staff. States reported  that the three activities on which DVOP specialists spend the most time  include    providing intensive case management services,    conducting an initial assessment or interview, and    assisting with job search activities.
The top three activities LVER staff perform include    conducting outreach to employers,    assisting with job search activities, and    conducting an initial assessment or interview.
This division of duties appears to reflect the different focuses of the two  programs, as well as the flexibility under JVA for states to decide on staff  duties. However, almost 85 percent of states responded that DVOP  specialists conduct outreach to employers, a focus of the LVER program.  Additionally, almost 60 percent of states responded that LVER staff  provide intensive services, a primary focus of the DVOP program.
In our site visits, we found that this similarity in staff roles and target  populations exists in part because some one-stop centers have only a  single DVOP specialist or LVER staff on duty at any given time. In these  particular one-stop centers, the same employee is responsible for serving  all groups of veterans and carrying out job roles for both programs. Even  in centers with more than one staff person, veterans tend to be served by  whichever staff person is available at that time. Program staff in several  centers told us that recently separated veterans were not specifically  directed to LVER staff for services, nor were disabled veterans directed to  DVOP staff. This sharing of duties may be due, in part, to changes in  staffing levels. More than half of states reported a decrease in the number  of full-time DVOP specialists or LVER staff over the last 2 years, and most  attributed this decline to the size of their state grant for the programs.
Nevertheless, this similarity in roles and populations served causes the  current performance measures to present an incomplete view of outcomes  for disabled and recently separated veterans in the DVOP and LVER  programs. The large numbers of disabled veterans served by the LVER  program and recently separated veterans served by the DVOP program are  not included in the set of measures that focus on the outcomes of those  populations (see fig. 1).
Beyond the measures for disabled and recently separated veterans, having  separate measures for the DVOP and LVER programs obscures the overall  picture of outcomes to veterans, given the similarity between many of the  program activities and the reality of how the programs operate. According  to our survey, almost half the states would like Labor to consolidate the  performance measures for the DVOP and LVER programs.
While the performance measures present an incomplete view of the  outcomes for veterans, the data that states are required to report to Labor  do not reflect the full range of staff services. Labor requires states to  report a wide range of data for the DVOP and LVER programs, including  information on veteran characteristics—such as age and disability status— and staff services provided—such as intensive services and referrals to  other programs. However, Labor does not require data on employer  outreach activities, despite JVA’s designation of employer outreach as a  primary job responsibility of LVER staff. Consequently, Labor and states  cannot formally monitor the extent to which staff perform this activity.  Some states, however, collect these data for their own use. According to  our survey, almost half of states currently collect employer-related  information for the DVOP and LVER programs, and over 75 percent of  states reported that it would be helpful to collect these data.
In addition, even though the data reported to Labor generally reflect  services and outcomes for veterans, these data are aggregate tallies and do  not show services provided to individual veterans. For example, each  state’s quarterly reports reflect the sum of all services provided and do not  show the number of services provided per veteran or per staff person. The  current data are useful to provide an overall picture of the programs’  volume and operations. However, these data provide little information  about services received by individual veterans or delivered by particular  veteran staff.
section_title: Performance Data Are Not Comparable over Time or across States
paragraphs: In recent years, reporting requirements for the DVOP and LVER programs  have undergone several significant changes. These changes have moved  the performance accountability system closer to those of other  employment and training programs. At the same time, the changes have  resulted in a lack of reliable trend data. In July 2002, the DVOP and LVER  programs changed from using administrative follow-up to determine  veterans’ employment outcomes to obtaining information from  Unemployment Insurance (UI) wage records. In doing so, Labor changed  its method of calculating outcomes for veterans in the DVOP and LVER  programs. Then, in July 2005, Labor applied the common measures to  these two programs, refining and standardizing the application of UI wage  records to determine outcomes. Under the old system, Labor calculated  entered employment and employment retention rates based on the number  of veterans who participated in the programs. However, under the new  system, Labor calculated these rates based on how many veterans  terminate services and exit the programs. Although these changes have  standardized the performance measures across programs, they have also  prevented Labor and states from developing consistent, comparable data  over the past 5 years. As a result, Labor does not have reliable historic data  for either program. Figure 2 illustrates the various changes to the DVOP  and LVER programs’ performance reporting requirements.
Furthermore, the instability in data collection and reporting has left Labor  unable to establish a national veterans’ entered employment standard, as  required by JVA. Labor anticipates that it will need at least 3 years of  stable data to establish the national standard. Once it is established, all  states will be held accountable to the same minimum goal for veterans’  entered employment. However, it is unclear when Labor will have  sufficient data to establish this standard because states continue to  experience difficulty adjusting to the numerous changes. According to our  survey, over 70 percent of states reported that frequent changes to  performance reporting requirements have been either a great or very great  challenge.
The data also vary somewhat state by state. For example, the application  of wage records to calculate veteran outcomes across state lines is no  longer consistent across states. The Wage Record Interchange System  (WRIS) allows states to share UI wage records and account for job seekers  who participate in one state’s employment programs but get jobs in  another state. In recent years, all states but one participated in WRIS,  which was operated by the nonprofit National Association of State  Workforce Agencies. In July 2006, Labor assumed responsibility for  administering WRIS. However, many states have withdrawn, in part  because of a perceived conflict of interest between ETA’s role in enforcing  federal law and the states’ role in protecting the confidentiality of their  data. As of March 2007, only 30 states were participating in the program,  and it is unknown if and when the other states will enter the data-sharing  agreement. As a result, DVOP and LVER performance information in  almost half the states will not include employment outcomes for veterans  who found jobs outside the states in which they received services. In  addition, other reasons contribute to data variation by state. Labor allows  states flexibility in choosing data collection software, which has resulted  in some states adapting more quickly than others to the recent changes,  depending on their software capabilities. Several Labor officials told us  that because of differences in software capabilities, some states’ data may  be more reliable than others’.
section_title: Available Data Paint an Unclear Picture of Veterans’ Use of One- Stop Services
paragraphs: Labor’s data on veteran job seekers paint an unclear picture of their use of  employment and training services in the one-stop system, despite the  shared use of common performance measures across programs. Although  many veterans use employment services other than those provided by the  DVOP and LVER programs, key employment programs vary in how well  their data on veteran participants are integrated or shared with other  programs. As a result, many states may not know how many veterans use  one-stop services. In addition, statutory differences in the way veterans  are defined for purposes of program eligibility make it difficult to  standardize data across employment programs. Moreover, Labor has no  means of assessing whether priority of service for veterans has been  implemented in various employment programs.
subsections: section_title: Veterans Receive Services from Multiple Programs in the One-Stop System
paragraphs: Many veteran job seekers receive employment services from the DVOP  and LVER programs. However, some veterans—often the more job- ready—only use one-stop services aimed at the general population, such  as the Employment Service and WIA programs. In addition, some veterans  use services focused on other subsets of job seekers—such as TAA (see  fig. 3). As a result, performance information on many veterans is collected  and reported elsewhere in the one-stop system. In fact, 20 states reported  that about half or fewer of veteran job seekers who access employment  programs receive services from a DVOP specialist or LVER staff,  according to our survey (see fig. 4). In addition, some veterans obtain  services from more than one employment program in the one-stop system,  all of which use the common measures to assess their performance.
section_title: Data on Veterans’ Services Are Well Integrated across Some but Not All Programs in the One-Stop System
paragraphs: Performance data on veteran job seekers are well integrated or shared  across some key employment and training programs, but not others,  despite the mutual use of common measures. As a result, many states may  not know how many veterans they serve through the one-stop system.  Data on veterans who access the Employment Service are completely  integrated with data from the DVOP and LVER programs—they share the  same reporting system, and DVOP and LVER data are a subset of  Employment Service data. According to our survey, veteran job seekers in  most states receive initial assistance from the Employment Service when  they access the one-stop system. If they are subsequently referred to the  DVOP and LVER programs, all of their information is housed in the same  system and an unduplicated count of veterans served between these  programs can be obtained. In addition, states are held accountable for  meeting separate goals in the Employment Service for veterans and  disabled veterans (see app. III). Labor considers these measures to reflect  veterans’ outcomes for the entire one-stop system, as they constitute  outcomes for all veterans who access the Employment Service, DVOP, and  LVER programs. Furthermore, they are the best approximation of a total  count of veterans who access the one-stop system that the current data  will allow.
On the other hand, data on veterans served by other one-stop programs  are not well integrated. States report data to Labor on WIA participants  who exit the programs, including veterans, using the Workforce  Investment Act Standardized Record Data (WIASRD) system. Although  WIASRD contains sufficient information to produce separate veteran  outcome data for WIA programs, states are not required to produce  separate veteran reports and are not accountable for meeting veteran  goals in those programs. In addition, fewer than half the states reported  that they routinely match WIA and Employment Service records to attain  an unduplicated count of veterans served by those programs.  Consequently, veterans who access two different employment services  may be counted twice in some cases. Data for TAA participants are  reported to Labor in yet another data system, which does allow states to  report on the veteran status of participants, but Labor officials told us the  agency does not currently use veteran outcomes from that program for  any purpose. VETS does not include the veteran outcome data from WIA  or TAA in its annual report to Congress, and Labor officials told us they  are exploring ways to better use the data.
In addition, data are not always collected on job seekers who use  Employment Service or WIA resources without assistance from program  staff. These self-assisted job seekers—including veterans—access services  such as labor market or career information either in one-stop centers or  on home computers, but do not receive active assistance from program  staff. Historically, some states have collected information on these job  seekers, while others have not. In our survey, 73 percent of states reported  that they capture information on all veterans who receive self-assisted  services through the Employment Service, while 82 percent of states  reported doing so for all veterans who receive self-assisted WIA services.  Labor has encouraged—but not mandated—states to collect information  on this group of job seekers, but agency officials acknowledged that states  continue to vary in how they report such data. Labor officials have  expressed concern that requiring veterans who receive self-assisted  services to register for the programs might discourage some of them from  pursuing the services they need.
section_title: Different Veteran Definitions Complicate Efforts to Standardize Data and Implement Priority of Service
paragraphs: Labor and some state officials we surveyed reported that statutory  differences in the definitions of veterans for various employment  programs make it difficult to standardize data across programs. For the  purposes of the DVOP and LVER programs, an eligible veteran is  statutorily defined as an individual who served on active duty for more  than 180 days. Labor also uses this definition for the Employment Service.  WIA, on the other hand, does not specify a length of time in service for a  person to be considered a veteran. Moreover, to qualify as a recently  separated veteran in the DVOP and LVER programs, a veteran must have  left active duty in the last 3 years. By contrast, WIA defines recently  separated as having left active duty in the last 4 years (see table 5).
These inconsistent definitions have been difficult for Labor and states to  reconcile with the concept of seamless service delivery and have caused  some confusion for states as they implement priority of service throughout  the one-stop system. While JVA requires that veterans receive priority over  other job seekers in Labor-funded employment and training programs, it  does not define a veteran for purposes of the priority requirement. Labor  has interpreted JVA’s provisions to mean that while veterans are to receive  preference in the programs after any other statutory eligibility  requirements are met, each program must use its own statutory definition  of a veteran in applying that preference. Labor officials told us that one  state applied for a waiver in 2006 to use a single definition of veterans for  all of its employment and training programs, but Labor’s Solicitor’s Office  orally denied the request. In our survey, approximately half of all states  reported that the conflicting veteran definitions in various employment  programs complicate data entry, referrals to other programs, and the  implementation of priority of service. In addition, about a third of the  states claimed that the definitions created gaps in services for veteran  clients as they moved among employment programs (see fig. 5). For  example, if a veteran receives services from WIA and is subsequently  referred to the DVOP program but is found ineligible, he or she may  become discouraged and stop seeking services altogether.
Almost half of states shared their concerns about different definitions by  providing additional comments in our survey, many of which cited the  difficulty of providing priority of service under these circumstances. For  example, one state responded that different definitions often lead to  inappropriate referrals, resulting in poor customer service and frustration  for program participants and service providers. Other states focused on  the burden that competing definitions placed on data collection and  reporting. For example, one state responded that the issue has made it  difficult to integrate the state’s Employment Service and WIA data  systems, because the different definitions could lead to invalidating the  veteran numbers on reports for those programs. Another state cited the  difficulty in assessing how many veterans were served by the state,  highlighting the complexity of producing an unduplicated count of  veterans served by different programs that do not share a single definition.  States also cited challenges in dealing with other agencies that are not  mandated partners in the one-stop system. For example, two states  mentioned that some staff of other agencies’ programs may hesitate to  refer participants to the DVOP and LVER programs because they are  unsure about participant eligibility. An expert on veterans’ issues in the  states concurred that the different eligibility criteria for veterans has been  a problem for states and told us that a common veteran definition for  employment and training programs would be an improvement.
section_title: Labor Has No Means of Evaluating Priority of Service for Veterans
paragraphs: Despite JVA’s mandate, Labor has not produced information on the extent  to which veterans receive priority of service in all qualified employment  and training programs. Specifically, JVA required Labor to evaluate and  report on whether veterans are receiving priority of service and are fully  served by its employment programs, as well as whether the representation  of veterans in such programs is in proportion to their participation in the  labor force. In its fiscal year 2005 report, Labor stated that the  participation rate for veterans in its adult programs was approximately   8.4 percent—slightly higher than veterans’ participation rates in the U.S.  workforce. In addition, the agency reported that outcomes for veterans  served in these programs closely mirrored those of all job seekers in the  programs. However, Labor has no method of gauging how—and how  consistently—priority of service is actually applied. Labor officials told us  that the highly devolved workforce development system makes it very  difficult to evaluate priority of service, because different programs have  multiple access points and diverse eligibility criteria that prevent Labor  from applying a simple measurement technique to each.
States reported that implementing priority of service has been challenging,  as has holding one-stop partner programs accountable for serving  veterans. To supplement federal guidance on this issue, at least one state  has developed its own guidance for implementing and measuring priority  of service. Some Regional Directors of VETS told us they encouraged the  use of that state’s guidance as a model for assessing priority of service for  states in their own regions. We do not know when Labor will develop  further guidance on the issue. However, in December 2006, Congress  passed the Veterans Benefits, Health Care, and Information Technology  Act of 2006, which included a requirement that Labor release regulations  on implementing priority of service within 2 years. In addition, the agency  has begun planning a study of priority of service in response to our prior  report. According to Labor officials, the study will combine a survey of  participants with a process evaluation and an analysis of outcomes. Labor  does not yet know when the study will get under way.
section_title: Labor Has Taken Steps to Better Understand Veterans’ Outcomes, but the Programs’ Impact Remains Unknown
paragraphs: Labor has taken some steps to improve the quality of performance data  and better understand veterans’ services and outcomes, but the overall  impact of employment services for veterans is unknown. Labor has  developed some processes to enhance data quality. For example, Labor’s  ETA requires states to validate some data in key programs. Furthermore,  Labor plans to implement an integrated data-reporting system that would  allow Labor and states to track individual veterans’ progress through  different programs in the one-stop system. Additionally, the new system  would expand data collection by, for example, collecting more data on  services to employers. However, states have raised concerns about the  challenge of meeting the system’s planned implementation date, and the  timeline for implementation remains unclear. Furthermore, while  performance information helps assess whether individuals are achieving  their intended outcomes—such as obtaining employment—it cannot  measure whether the outcomes are a direct result of program  participation, rather than external factors. To measure the effects of a  program, it is necessary to conduct an impact evaluation that would seek  to assess whether the program itself led to participant outcomes. Labor  has sponsored research on services to veterans. However, it has not  conducted an impact evaluation, as required under WIA, to assess the  effectiveness of one-stop services. Such a study should include impacts for  key participant groups, including veterans. We recommended in 2004 that  Labor take steps to conduct such an evaluation, but there has been no  action to date.
subsections: section_title: Labor Has Developed Some Processes to Enhance the Quality of Performance Data
paragraphs: Labor has taken some steps to improve the quality of performance data  and enhance the understanding of veterans’ services and outcomes. To  address data quality concerns, ETA has developed processes requiring  states to validate certain data reported for participants in WIA and  Wagner-Peyser-funded Employment Service programs. However, while  these programs serve veterans, participant records are randomly selected  in both programs from the total participant population and, therefore, may  not include the records of veteran participants.
Both the WIA data validation process, developed in 2004, and the  Employment Service process, developed in 2003, involve two types of data  validation, although the WIA process is more intensive, according to Labor  officials. Both processes involve (1) data element validation—comparing  randomly sampled participant records to source files, and (2) report  validation—assessing whether states’ software accurately calculated  performance outcomes. While element validation in WIA is conducted on- site with hardcopy source documentation, the Employment Service data  validation process is performed centrally and electronically, because  Employment Service records are generally electronic. The Employment  Service element validation process checks for duplicate or invalid entries  in source files by, for example, checking for inconsistencies among  various veteran-related fields, such as veteran status and disabled veteran.  However, the Employment Service element validation process cannot  check the underlying accuracy of the data, because there is no hard copy  documentation to prove whether a participant is in fact a veteran.
Labor officials told us that the Employment Service data validation  process has been helpful in raising awareness among states about the  importance of data quality and that some states have come to see it as a  useful tool. Additionally, states responding to our survey generally agreed  that it has been effective—38 states, or about 75 percent, rated the  Employment Service data validation process as effective in ensuring the  accuracy of veteran job seekers’ information. For example, according to  one respondent, review of the data validation results is used as a  management tool, to highlight successes and to alert staff to weaknesses.  Nevertheless, some states have expressed concerns about the data  validation processes. Concerns about the process were also raised by state  officials in all 3 of the states we visited. For example, officials in 2 of the   3 states noted that they had experienced difficulties adjusting to frequent  changes in software before the results were due to Labor. On our survey,   2 states said that the sample size was too small to be meaningful, and   4 states expressed concerns about the fact that the process does not verify  the accuracy of the data in source files. These concerns are similar to  those we identified in a previous report that addressed the WIA process.
Additionally, Labor has taken steps to address data quality as a part of its  routine monitoring and technical assistance. Specifically, beginning in  2004, ETA regional staff have incorporated a data quality component into  compliance visits to state offices, which are generally conducted once or  twice a year, according to Labor officials. Data validation is just one  component of these compliance visits, which typically do not focus on  veterans’ data as a separate issue. To support this effort, Labor officials  told us that ETA has amended its monitoring guide for these visits to  include a section on data validation. According to Labor officials, these  visits have been useful in identifying problems and corrective actions.  Moreover, ETA and VETS have recently collaborated on a few of these  compliance visits. Labor officials said they believed this joint monitoring  was beneficial, and expect those efforts to be a model for future joint  visits.
There are several other forms of management reviews that generally focus  on services to veterans but also offer a chance to review data. For  example, VETS regional and state-based staff conduct site visits as part of  their routine monitoring, which focus primarily on services to veterans but  which can include reviewing performance information as well.  Additionally, VETS has required a series of annual assessments—of the  program for each state overall, and self-assessments by DVOP specialists,  LVER staff, and one-stop managers—that address data issues to a limited  extent. State directors use performance data to substantiate services  described in the self-assessment. For example, according to one official  we spoke with, to confirm a LVER staff’s claim of travel to several job  fairs, the director can consult the one-stop’s travel log to substantiate  whether the LVER staff actually made the trips.
Beyond the steps Labor has taken, state workforce agencies also perform  functions that affect performance data on services to veterans. Most states  responding to our survey reported that they have taken certain steps to  ensure the accuracy and reliability of data for the Employment Service,  DVOP, and LVER programs, such as having their systems perform  automated checks for inconsistencies in data or for duplicate veteran files  (see fig. 6).
section_title: Labor’s Proposed Integrated Data System Could Improve the Understanding of Services and Outcomes for Veterans
paragraphs: Since 2004, Labor has been planning to implement an integrated data  reporting system that could greatly enhance the understanding of veterans’  services and outcomes. In 2004, Labor first proposed a single, streamlined  reporting system, known as the ETA Management Information and  Longitudinal Evaluation system (EMILE) that would have replaced  reporting systems for several Labor programs. Labor substantially  modified this system’s design in response to concerns raised by state and  local agencies about the burden and cost of the new system, as well as the  challenge of meeting the implementation deadline. The modified system,  now called the Workforce Investment Streamlined Performance Reporting  System (WISPR), was planned with a July 2007 implementation date.
WISPR has been designed to both integrate and expand data reporting.   If implemented, the system would integrate data reporting by using  standardized reporting requirements across the Employment Service,  DVOP and LVER, WIA, and TAA programs, and ultimately replace their  preexisting reporting systems with a single reporting structure.  Additionally, it would rely on a standardized set of data elements and  quarterly reports to provide data on participant characteristics and  services provided, as well as performance outcomes based on the common  measures. Its integrated design would, for the first time, allow Labor and  states to track individual veterans’ progress through the one-stop system.
In addition, the system would expand data collection and reporting in two  key areas: the services that LVER staff provide to employers, a key aspect  of the LVER role on which Labor currently collects no data, and estimates  of the population of veterans who access the one-stop system but  ultimately receive limited or no services from one-stop staff.
As with EMILE, however, concerns have been raised about challenges in  implementing the new system, and at present, the timeline for WISPR’s  implementation remains unclear. Some of the comments received by OMB  during the official comment period noted the challenge of a July 2007  implementation date, according to a Labor official. While states will have a  2-year period to consolidate reporting on the full range of programs, they  are expected to begin collecting and reporting data in the new format  immediately. As of December 2006, 39 entities, including state workforce  agencies, local agencies and unions, had submitted comments reflecting  their concerns about WISPR to the Office of Management and Budget  (OMB). Of the 20 states that submitted comments, 14 noted that a July  2007 implementation date would represent a challenge. For example, some  of them expressed the view that Labor had underestimated the time states  would need to revise policy, reprogram systems, and retrain staff. In  addition, some states expressed concerns about their ability to provide  data on services to employers. Moreover, two states expressed the  concern that meeting Labor's planned implementation date would have  adverse consequences, such as compromised data quality or cost  overruns. OMB’s official review will address the time needed to build the  reporting system’s technical infrastructure, and will play a key role in  deciding the system’s final implementation schedule, according to a Labor  official. States and local areas will need enough time to fully meet the  requirements of this expanded data collection.
section_title: Little Is Known about the Impact of One-Stop Services, including Those to Veterans
paragraphs: Although Labor has improved its outcome data on job seekers who  participate in its programs, these data alone cannot measure whether  outcomes are a direct result of program participation, rather than external  factors. For example, local labor market conditions may affect an  individual’s ability to find a job as much or more than participation in an  employment and training program. To measure the effects of a program, it  is necessary to conduct an impact evaluation that would seek to assess  whether the program itself led to participant outcomes. Labor has not  conducted an impact evaluation of one-stop services, including those to  veterans. However, the department did sponsor a study, issued in 2003,  that examined the relationship between services provided to certain  groups of veterans and employment and earnings outcomes. This study  employed a number of data sources and statistical techniques to learn  more about how veterans were using one-stop services. However, while  this study provided some useful information, it could not determine that  these services caused positive outcomes for veteran job seekers. In  addition, the study relied on data from 8 states and its findings could not  be generalized to the national population of veteran job seekers.
Since the full implementation of WIA in 2000—in which the one-stop  system became the required means to provide employment and training  services, including those to veterans—Labor has not made evaluating the  impact of those services a research priority. While WIA required one such  evaluation by 2005, Labor has declined to fund one in prior budgets. In a  2004 report, we recommended that Labor comply with the requirements of  WIA and conduct an impact evaluation of WIA services to better  understand what services are most effective for improving employment- related outcomes. In response to our report, Labor cited the need for  program stability and proposed delaying an impact evaluation of WIA until  any changes that might be included in reauthorization legislation had been  implemented. While efforts to reauthorize WIA began in 2003, they have  stalled and it is not clear at this time when they will be complete.  Furthermore, OMB has also found Labor’s evaluations of WIA services to  be lacking. In response, in its 2008 budget proposal, Labor identified an  assessment of WIA’s impact on employment, retention, and earnings  outcomes for participants as an effort the agency would begin. According  to Labor officials, the agency has not yet begun to design the study. Such a  study should include impacts for key participant groups, including  veterans. To do so would require a sufficient sample of veterans to allow  such analysis.
section_title: Conclusions
paragraphs: At a time when the nation’s attention is focused on those who have served  their country, it is vital that Congress and the Administration are able to  make informed decisions about programs that help veterans find and keep  jobs in the civilian labor market. Frequent changes in Labor’s performance  accountability system have hampered Labor’s ability to produce consistent  and meaningful performance information on veteran job seekers. States  and local areas have had difficulty implementing the constant changes to  performance information, which introduce error and make it difficult to  identify trends that would give Congress a better idea of the programs’  achievements. While the anticipated transition to a new reporting system  represents a promising advance in Labor’s ability to track the outcomes of  veterans in the one-stop system, states will need time to effectively  implement the changes to avoid compromising the potential benefits— such as improved data quality—of the system.
Furthermore, the current separate performance measures for the DVOP  and LVER programs do not account for the considerable similarity in  veteran populations served by DVOP specialists and LVER staff, and thus  do not provide an accurate picture of outcomes for veterans served by  these two programs. Using the existing measures, Labor also cannot  ensure that performance outcomes give more weight to services for  veterans with greater barriers to employment. In addition, different  veteran definitions in other programs could make it difficult to analyze  services to veterans throughout the one-stop system. Further, Labor  cannot provide assurance that veterans are appropriately given service  priority by programs in the one-stop system, or that services to veterans  are truly effective. The federal government spends about $155 million each  year on the DVOP and LVER programs alone, not counting the amounts  spent on veterans who use other one-stop programs, but there is no  information on whether these programs have an impact in helping this  important population. Establishing a means to gauge the programs’ impact  would require a considerable investment of time and money, but would  contribute greatly to the understanding of whether current employment  and training services are meeting veterans’ needs. Furthermore, we  continue to urge Labor to meet WIA requirements and our 2004  recommendation to conduct an impact evaluation of one-stop services.
section_title: Recommendations for Executive Action
paragraphs: To provide a better picture of services and outcomes for veteran job  seekers, improve program reporting, and facilitate priority of service, we  recommend that the Secretary of Labor    ensure that states are given adequate direction and sufficient time  to implement ETA’s planned integrated data reporting system and  make necessary changes;    consolidate all performance measures for the DVOP and LVER  programs, including those for disabled and recently separated  veterans;    comply with JVA’s requirement to implement a weighting system for  the DVOP and LVER performance measures that takes into account  the difficulty of serving veterans with particular barriers to  employment;    develop legislative proposals for appropriate changes to the  definitions of veterans across employment and training programs to  ensure consistency; and    ensure that Labor moves forward with an impact evaluation for the  one-stop system under WIA as we recommended in 2004, and that  the evaluation’s sampling methodology includes veterans in  sufficient numbers to allow analysis of the impact of services to  veterans in the one-stop system, including those served by the  DVOP and LVER programs.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to Labor for review and comment. In its  comments, Labor generally concurred with our findings, conclusions, and  recommendations and expressed appreciation that the report  acknowledges the steps the agency has taken to improve the quality of  performance data and better understand outcomes for veterans. Labor  noted that it is considering adopting a different approach to measuring  outcomes for the DVOP and LVER programs by program year 2008—one  that may take into account the similar veteran populations served, as well  as outreach to employers. As it develops this new approach, Labor  reported that it will also introduce a system of weighted measures that will  emphasize services to veterans with barriers to employment. These  changes will coincide with the implementation of Labor’s proposed  integrated data system, WISPR. Labor also noted that it would work with  states and grantees to ensure a smooth transition to the new system. In  addition, Labor stated that it intends to pursue a WIA impact evaluation,  which will allow for analysis of services to sub-populations, including  veterans. Labor reported that our recommendation to develop proposals  for changing veteran definitions across employment and training programs  must be evaluated with the input of other agencies. Labor also provided  technical comments that we incorporated where appropriate. Labor’s  comments are reproduced in full in appendix IV.
We will send copies of this report to the Secretary of Labor, relevant  congressional committees, and other interested parties and will make  copies available to others upon request. In addition, the report will be  available at no charge on GAO’s Web site at http://www.gao.gov.
A list of related GAO products is included at the end of this report. If you  or your staff have any questions about this report, please contact me at  (202) 512-7215 or at nilsens@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Other contacts and staff acknowledgments are listed in  appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this report were to determine (1) the extent to which  DVOP and LVER performance information reflects services and outcomes  for veterans served by these programs, (2) the extent to which  performance information on veterans served by other key programs is  comprehensive and well integrated across programs in the one-stop  system, and (3) what Labor is doing to improve the quality of performance  data and better understand outcomes for veteran job seekers.
To address these objectives, we    conducted a nationwide Web-based survey to state workforce  administrators in the 50 states and the District of Columbia;    conducted site visits to 3 states, during which we interviewed state  and federal officials, one-stop managers, and program staff;  interviewed Labor officials from both the Veterans’ Employment  and Training Service (VETS) and the Employment and Training  Administration (ETA);    analyzed relevant performance data from ETA and VETS; and  reviewed our previous work on attributes of successful  performance measures.
We conducted our work in accordance with generally accepted  government auditing standards between May 2006 and April 2007.
subsections: section_title: Survey
paragraphs: To obtain further information on our objectives, we surveyed state  workforce administrators from November 15 to December 27, 2006. The  survey addressed all three objectives and included questions about  performance information for the DVOP and LVER programs, integration of  data across employment programs serving veterans, and efforts to ensure  data quality.
We developed the survey based on knowledge obtained during our  preliminary research. This included a literature review and initial  interviews with officials from the Department of Labor, the National  Association of State Workforce Agencies (NASWA), and the state of New  Hampshire, where we conducted our initial site visit. We then obtained a  list of state workforce administrators from NASWA. We asked state  administrators to provide information on the DVOP and LVER programs’  capacity, other programs within the one-stop system that serve veteran job  seekers, performance measures and data; and challenges to managing the  programs. To determine whether respondents would understand the  questions as intended, we pretested the survey with state officials in   5 states. We then made changes to the questions based on comments we  received during the pretests.
The survey was conducted using self-administered electronic Web-based  questionnaires. We sent notification of the survey to the 50 states and the  District of Columbia in November 2006 and followed up with e-mail  messages and telephone calls as necessary during November and  December. All 51 recipients submitted their responses by the end of  December 2006, providing us with a response rate of 100 percent. We did  not independently verify information obtained through the survey. During  our data analysis we held three follow-up conversations to fill in gaps from  incomplete survey information.
Because this survey was not a sample survey, there are no sampling errors.  However, the practical difficulties of conducting any survey may introduce  errors, commonly referred to as nonsampling errors. For example,  difficulties in how a particular question is interpreted, in the sources of  information that are available to respondents, or how the data are entered  into a database can introduce unwanted variability into the survey results.  We took steps during survey development, data collection, and data  analysis to minimize these nonsampling errors. For example, we pretested  the questionnaire to ensure that questions were clear and understandable.  Since this was a Web-based survey in which respondents entered their  responses directly into out database, there was little possibility of data  entry error. During data analysis, a second, independent analyst checked  all computer programming. Also, to the extent possible, we compared both  closed and open ended survey responses with our site visit observations.  While survey results are also subject to different types of systematic errors  or bias, we do not have reason to believe that respondents falsely reported  any information for this survey.
section_title: Site Visits
paragraphs: To obtain a detailed understanding of how veteran job seekers are served  by the one-stop system and how their information is captured, we  conducted visits to three states: New Hampshire, California, and  Tennessee. We selected these states based on a range of selection criteria,  including geographic dispersion, state size and veteran demographics,  recent state performance in veterans’ programs, and recommendations by  Labor and NASWA. Our site visits at the state level included interviews  with state workforce agency officials and state directors of Veterans’  Employment and Training. We also chose two local one-stops in each state  and met with local managers and veteran program staff (see table 6).
During each interview, we used standard interview protocols to obtain  detailed and comparable information. In our interview with state  workforce officials, we discussed the role of the state workforce agency in  administering veterans’ employment and training programs, details about  the programs serving veteran job seekers, views on the current  performance accountability system, and information about data collection  and validation. In our interviews with the state directors and their staff, we  discussed their oversight roles and responsibilities, relationship with the  state workforce agency, and views on the current performance  accountability system and data collection. At the local one-stops, we  discussed the coordination of veteran staff with other programs within the  one-stop system, priority of service, and data collection and reporting. In  each state, we also received a tutorial of the state’s data collection  software. We conducted our site visits between July and November 2006.
section_title: Research and Experts
paragraphs: As part of our work, we interviewed officials of ETA and VETS, including  all six Regional Administrators of VETS. We conducted these telephone  interviews in the following locations: Boston, Atlanta, Dallas, Chicago, San  Francisco, and Philadelphia. During each interview, we obtained  information on regional differences in administering the DVOP and LVER  programs, views on the current performance measures, and information  on Labor’s monitoring role in each state.
We also analyzed performance data from the DVOP, LVER, and  Employment Service programs and reviewed Labor’s guidance. In  addition, we reviewed relevant literature, including our past work on  attributes of successful performance measures. We also interviewed  representatives of NASWA and two private-sector staffing agencies.
section_title: Appendix II: Summary of State Performance in the DVOP and LVER Programs for Benchmark Program Year 2005 and Negotiated Goals for Program Year 2006
paragraphs: baseline performance data by state for the DVOP and LVER  programs from benchmark program year 2005 (July 1, 2005–June 30,  2006) and    negotiated goals by state for the following year, program year 2006.
Labor and states did not negotiate goals for the DVOP or LVER programs  for program year 2005, which was a baseline year for performance under  the new common measures. Four performance measures contribute to  each program’s performance. For the DVOP program, there is one set of  measures for all veterans and one set for disabled veterans. For the LVER  program, there is a set of measures for all veterans and another set for  recently separated veterans. Each set of measures includes    entered employment rate (EER): the number of participants who  are employed in the first quarter after the exit quarter divided by the  number of participants who exit during the quarter and    employment retention rate (ERR): the number of participants who  are employed in both the second and third quarters after the exit  quarter divided by the number of adult participants who exit during  the quarter.
These figures were provided by the Department of Labor. GAO has not  verified the accuracy or reliability of these data.
section_title: Appendix III: Summary of State-Negotiated Goals and Performance for Veterans in the Employment Service, Program Year 2005
paragraphs: This table illustrates the negotiated goals and performance achieved by  each state for program year 2005 for veterans in the Wagner-Peyser-funded  Employment Service. It includes the entered employment and employment  retention rates for all veterans and disabled veterans within the  Employment Service, including those in the DVOP and LVER programs.
These figures were provided by the Department of Labor. GAO has not  verified the accuracy or reliability of these data.
section_title: Appendix IV: Comments from the Department of Labor
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: Dianne Blank, Assistant Director  Rebecca Woiwode, Analyst-in-Charge  Chris Morehouse and Beth Faraguna made significant contributions to this  report in all facets of the work. In addition, Walter Vance assisted in the  design of the national survey; Gloria Hernandez-Saunders helped with data  analysis; Meeta Engle lent subject matter expertise; Jessica Botsford and  Richard Burkard provided legal support; and Charlie Willson provided  writing assistance.
section_title: Related GAO Products
paragraphs: Trade Adjustment Assistance: Labor Should Take Action to Ensure  Performance Data Are Complete, Accurate, and Accessible. GAO-06-496.  Washington, D.C.: April 25, 2006.
Veterans’ Employment and Training Service: Greater Accountability  and Other Labor Actions Needed to Better Serve Veterans. GAO-06-357T.  Washington, D.C.: February 2, 2006.
Veterans’ Employment and Training Service: Labor Actions Needed to  Improve Accountability and Help States Implement Reforms to Veterans’  Employment Services. GAO-06-176. Washington, D.C.: December 30, 2005.
Workforce Investment Act: Labor and States Have Taken Actions to  Improve Data Quality, but Additional Steps Are Needed. GAO-06-82.  Washington, D.C.: November 14, 2005.
Veterans’ Employment and Training Service: Preliminary Observations  on Changes to Veterans’ Employment Programs. GAO-05-662T.  Washington, D.C.: May 12, 2005.
Performance Measurement and Evaluation: Definitions and  Relationships. GAO-05-739SP. Washington, D.C.: May 2005.
Workforce Investment Act: States and Local Areas Have Developed  Strategies to Assess Performance, but Labor Could Do More to Help.  GAO-04-657. Washington, D.C.: June 1, 2004.
Performance Reporting: Few Agencies Reported on the Completeness and  Reliability of Performance Data. GAO-02-372. Washington, D.C.: April 26,  2002.
Veterans’ Employment and Training Service: Flexibility and  Accountability Needed to Improve Service to Veterans. GAO-01-928.  Washington, D.C.: September 12, 2001.
Veterans’ Employment and Training Service: Proposed Performance  Measurement System Improved, but Further Changes Needed.   GAO-01-580. Washington, D.C.: May 15, 2001.
Veterans’ Employment and Training Service: Strategic and Performance  Plans Lack Vision and Clarity. GAO/T-HEHS-99-177. Washington, D.C.:  July 29, 1999.
Veterans’ Employment and Training Service: Assessment of the Fiscal  Year 1999 Performance Plan. GAO/HEHS-98-240R. Washington, D.C.:  September 30, 1998.
Veterans’ Employment and Training: Services Provided by Labor  Department Programs. GAO/HEHS-98-7. Washington, D.C.: October 17,  1997.
The Results Act: An Evaluator’s Guide to Assessing Agency Annual  Performance Plans. GAO/GGD-10.1.20. Washington, D.C.: April 1998.
Agencies’ Annual Performance Plans under the Results Act: An  Assessment Guide to Facilitate Congressional Decisionmaking.  GAO/GDD/AIMD-10.1.18. Washington, D.C.: February 1998.
Executive Guide: Effectively Implementing the Government Performance  and Results Act. GAO/GGD-96-118. Washington, D.C.: June 1996.