Veterans' Employment and Training Service: Proposed Performance Measurement System Improved, But Further Changes Needed
Released: 2001-05-15T00:00:00

Letter

Background

VETS administers national programs intended to (1) ensure that veterans receive priority in employment and training opportunities from the employment service; (2) assist veterans, reservists, and National Guard members in securing employment; and (3) protect veterans’ employment rights and benefits. The key elements of VETS’ services include enforcing veterans’ preference and reemployment rights and securing employment and training services. VETS’ programs are among those federal programs whose services have been affected by WIA and other legislative changes aimed at streamlining services and holding programs accountable for their results.

VETS’ Programs

VETS carries out its responsibilities through a nationwide network that includes representation in each of the Department of Labor’s 10 regions and staff in each state. The Office of the Assistant Secretary for Veterans’ Employment and Training administers VETS’ activities through regional administrators and a VETS director in each state. These federally paid VETS staff are the link between VETS and the states’ employment service system, which is overseen by Labor’s Employment and Training Administration (ETA). VETS funds two primary veterans’ employment assistance grants to states—the Disabled Veterans’ Outreach Program (DVOP) and the Local Veterans’ Employment Representatives (LVER). Fiscal year 2001 appropriation for VETS was about $183 million, including $81.6 million for DVOP specialists (DVOPS) and $77.2 million for LVER staff. These funds paid for 1,327 DVOP positions and 1,206 LVER positions.

The roles of the DVOPS and LVERs have been separately defined in two statutes. LVERs were first authorized under the original GI bill (the Servicemen’s Readjustment Act of 1944) and DVOP specialists were authorized by the Veterans’ Rehabilitation and Education Amendments of 1980. A key responsibility of a DVOP is to develop job and job training opportunities for veterans through contacts with employers, especially small- and medium-size private sector employers. LVERs are to provide program oversight of local employment service offices to ensure that veterans receive maximum employment and training opportunities from the entire local office staff. In addition, DVOPS and LVERs traditionally have provided services that include locating veterans who need services, networking in the community for employment and training programs, bringing together veterans looking for work and employers seeking to making referrals to support services, and providing case management for those veterans in need of more intensive services.

Increasingly, however, veterans are accessing services on their own, through tools such as internet-based job listings or resume writing software.

As part of the DVOP and LVER grant agreements, states must provide or ensure that veterans receive priority at every point where public employment and training services are available. The DVOP and LVER programs give priority to the needs of disabled veterans and veterans who served during the Vietnam era. States’ employment service systems are expected to give priority to veterans over nonveterans. Generally, this means that local employment offices are to offer or provide all services to veterans before offering or providing those services to nonveterans. To monitor the states’ programs, VETS has been using a set of measures that evaluates states’ performance in five dimensions: (1) veterans placed in training, (2) those receiving counseling, (3) those receiving services, (4) those entering employment, and (5) those obtaining federal contractor jobs. These measures primarily count the number of services that veterans receive and compare the totals with similar services provided to nonveterans. To ensure priority service to veterans, VETS expects levels of performance for services provided to veterans to be higher than levels for nonveterans. For example, veterans and other eligibles must be placed in or obtain employment at a rate 15 percent higher than that achieved by nonveterans. (See table 1 for VETS’ specific performance standards.)

To report on performance, VETS currently relies on the Employment and Training Administration’s 9002 system to aggregate data reported by states on veterans and nonveterans who register with state Employment Services (ES) offices, track the services provided to them (such as counseling or job referral), and gather information on their employment outcomes. The 9002 system also collects information such as the registrants’ employment status, level of education (e.g., high school, postsecondary degree/certificate), and basic demographic information, such as age and race.

Legislative and Regulatory Changes Affecting VETS

Over the past several years, the Congress has taken steps to streamline and integrate services provided by federally funded employment and training programs. WIA, which the Congress passed in 1998, requires states and localities to use a one-stop center structure to provide access to most employment and training services in a single location. WIA requires about 17 categories of programs, including VETS and ES programs, to provide services through the one-stop center. However, because DVOP and LVER staff can provide assistance only to veterans, and because their roles in one-stop centers are not specifically addressed in WIA, it is unclear how they will function with regard to one-stop centers. According to VETS officials, this lack of clarity has been addressed. Agreements made with each state on planned services to veterans now include provisions on how DVOPS and LVERs will be integrated into the one-stop delivery system.

In addition to changing the way services are provided, programs are now increasingly held accountable for their results. Through the Government Performance and Results Act of 1993 (GPRA), the Congress seeks to improve the efficiency, effectiveness, and public accountability of federal agencies as well as improve congressional decision making. GPRA does so, in part, by promoting a focus on what the program achieves rather than tracking program activities. GPRA outlines a series of steps in which agencies are required to identify their goals, measure performance, and report on the degree to which those goals were met. Executive branch agencies were required to submit the first of their strategic plans to the Office of Management and Budget and the Congress in September 1997. Although not required by GPRA, Labor’s component agencies, such as VETS, have prepared their own strategic and performance plans at the direction of the Secretary of Labor.

To address the goals of GPRA and in response to recommendations by us and other groups, such as the Congressional Commission on Servicemembers and Veterans Transition Assistance, VETS is currently developing a new system to measure the performance of its programs. Over the last several years, VETS conducted pilot programs in about eight states that tested some new performance measures and the use of new data to support these measures. VETS officials told us that they anticipate implementing their new performance measurement system in July 2001.

VETS’ Proposed Measures Hold Promise, But Some Concerns Remain

VETS’ proposed performance measures are a significant improvement over current measures, but certain aspects of these measures raise concerns that VETS may need to address. The proposed measures include an (1) entered-employment rate, (2) employment rate following staff- assisted services, (3) employment retention rate, and (4) increase in the number of federal contractor job openings listed. These measures are an improvement over current measures because they focus more on what the programs achieve and less on the number of services they provide, no longer use the level of services provided to nonveterans as the standard for services that must be provided to veterans, adjust expected state performance to economic conditions within establish two measures that are already collected for WIA-funded services and proposed for ES.

However, even with these improvements, the proposed measures continue to send a mixed message to staff about where to place their service priorities. In addition, the proposed measures include a redefined measure for tracking federal contractor job openings, but the measure is process- oriented and outside the scope of the work of DVOPS and LVERs.

Proposed Measures Are An Improvement Over the Current Ones

The proposed performance measures improve accountability because they place more emphasis on employment-related outcomes by eliminating process-oriented measures—measures that simply track services provided to veterans. Current process measures that VETS eliminated from the proposed performance system include the number of veterans referred to counseling, the number placed in training, and the number receiving certain other services, such as job referrals. As we noted in past reports, these process-oriented measures are activity- and volume-driven and focus efforts on the number of services provided, not on the outcomes veterans achieve. These measures offer states little incentive to provide services to those veterans who are only marginally prepared for work and who may need more intensive services requiring more staff time. The VETS’ proposal still includes one process-oriented measure that simply reflects the percentage increase in the number of federal contractor job openings listed with the public labor exchange but adds two outcome-oriented measures—job retention after 6 months and the employment rate following staff-assisted services. The VETS’ proposal also retains an outcome measure that is in the current system—the entered-employment rate. (See table 2.)

The proposed performance measures also improve the way VETS establishes the level of performance that states are expected to achieve. VETS no longer requires states to compare the level of services provided to veterans with those provided to nonveterans. In past reports, we have pointed out that the use of these relative standards results in states with poor levels of service to nonveterans being held to lower standards for service to veterans than states with better overall performance. For example, in program year 1999, Rhode Island reported an entered- employment rate of 5.49 percent for nonveterans. Because VETS requires states to ensure that they achieve an entered-employment rate for veterans that is 15 percent higher than that for nonveterans, Rhode Island’s 1999 expected performance level was 6.32 percent of registered veterans entering employment–a low level of performance. Under the proposed system, VETS will negotiate performance levels annually with each state based on that state’s past performance, using guidelines similar to those used for WIA. VETS will also be able to adjust these levels based on economic conditions within each state, such as the unemployment rate, the rate of job creation or loss, or other factors.

The proposed performance measures are also similar to those established under WIA, making it easier for service providers to achieve WIA’s goal of integrating and streamlining employment and training services. In the current environment, many of the programs that provide services through the one-stop centers have their own unique performance measures and program definitions, requiring multiple systems and multiple data collection efforts to track a single client. In the proposed system, VETS has made an effort to align its performance measures with those of WIA. In fact, two of the five proposed measures—entered-employment rate and employment retention—are nearly identical to WIA’s and to those proposed for ES. If VETS aligns the measures with those of WIA and ES, local offices will be more readily able to establish integrated data systems that will minimize the data collection burden on service providers and clients. (See app. I for a comparison of the WIA performance measures with those proposed for VETS and ES.)

Concerns Remain That VETS Should Address

While the proposed performance measures are an improvement over those currently in place, there are issues with these measures that VETS should address. First, a comparison of the performance measures with the strategic plan indicates that VETS is sending a mixed message to states about what services to provide and to whom. The strategic plan suggests that states focus their efforts on providing staff-assisted services to veterans, including case management. Yet, none of the proposed measures specifically gauges whether more staff-intensive services are helping veterans get jobs. VETS’ proposal includes a measure that tracks employment outcomes following staff-assisted services. However, this measure is broadly defined, and the list of staff-assisted services includes nearly all services provided to veterans. This makes the outcomes achieved for the staff-assisted measure nearly identical to those reported for the more general “entered-employment rate.” In addition, as VETS has defined it, staff-assisted services include many services that might not be considered “intensive,” such as referral to a job and job search activities. Because the definition is so broadly defined, a veteran who only attended a job search workshop would be counted the same as a veteran who received more intensive services, such as testing and employability planning. Both would be counted in the more general entered-employment rate measure, as well as the staff-assisted service measure. A stricter definition for staff-assisted services that includes only those services that are generally considered staff-intensive would allow VETS to more accurately assess the success of those services and help to clarify the goals of the program.

Second, VETS is sending a mixed message about which groups of veterans to target for services. As we noted in past reports and testimonies, VETS has inconsistently identified various “targeted” groups of veterans it plans to help. In its strategic plan, VETS identifies two broad veteran groups that should be targeted to receive special attention—(1) disabled veterans and (2) all veterans and other eligible persons. And consistent with this, VETS proposes that expected performance levels be negotiated separately for each of these same two groups. Yet, the strategic plan also suggests that, when providing services to all veterans, special attention should be given to meeting the needs of certain other target groups, some of which might require more intensive services to become employed. The groups targeted for special attention include (1) veterans who have significant barriers to employment, (2) veterans who served on active duty during a war (or campaign or expedition in which a campaign badge has been authorized), and (3) veterans recently separated from military service. In reviewing VETS’ proposed measures and the plan for negotiating performance levels, staff may be confused as to where they should place their service priorities. It is unclear what steps VETS will take to ensure that DVOPS and LVERs are provided ample opportunity and encouragement to focus attention on the portion of the “all veterans” group who may require more staff time to be successful in getting a job.

Last, VETS’ proposal also continues to include a performance measure related to federal contractor job openings listed with the state’s ES office. However, in its proposal, VETS has changed the measure. Under the current system, VETS tracks the number of Vietnam-era and special disabled veterans who were placed in jobs listed by federal contractors— an outcome measure. Now, under the proposed system, VETS will track the increase in the number of federal contractor jobs listed with the state’s ES office—a process-oriented measure. This new measure ultimately holds DVOPS and LVERs accountable for the number of federal contractors in a given state or local area, not for veteran placements with those contractors. The presence of federal contractors in a given state or local area is unpredictable and is determined by the federal agencies awarding contracts. Furthermore, according to state officials that we talked with, the federal contractor measure should be eliminated altogether because it is the responsibility of contractors to list their job openings. In addition, it is the Office of Federal Contract Compliance that is responsible for ensuring that all companies conducting business with the federal government list their jobs with state ES offices and take affirmative action to hire qualified veterans.

Proposed Data Source for New Measures Is An Improvement But Will Bring Some Challenges

The proposed data for the new measures will greatly improve the comparability and reliability of these measures, but this change will bring some challenges that VETS will need to address. Consistent with WIA and ES, VETS is proposing that all states use UI wage records to identify veterans who get jobs. UI wage records contain the earnings of each employee reported quarterly by employers to state UI agencies.Currently, the data VETS uses are not comparable across states, in part, because states use different data sources to report employment-related outcomes. Using a single, standardized source for collecting data will improve VETS’ ability to compare performance across states. UI wage records will also provide state officials with a better means to identify veterans who get jobs than does the traditional follow-up method of telephoning veterans and/or employers to verify employment. However, states cannot readily access wage records from other states, wage records do not cover certain types of employment, and these data are not available until 3 to 9 months after an individual gets a job.

Proposed Data Source Will Help To Ensure Comparability and Reliability Across States

Using a single data source will help to standardize the way in which states collect data on veterans, thereby making it easier to compare performance across states. Currently, states are using various data sources for performance-reporting purposes. While almost all of the states in our review used a combination of data sources to determine whether or not a veteran got a job, most of the states relied substantially on one data source, but that source differed among states. For example, in program year 1999 7 of the 15 states that we contacted relied to a large extent on wage record data to determine whether a veteran got a job or not; 7 others relied, for the most part, on telephone calls and letters to veterans and employers to determine a veteran’s employment status; and one state relied primarily on its new hire database for employment data.

In addition to making state data more comparable, we found evidence that states currently using wage records have been able to better identify those veterans who get jobs after receiving services. A recent study found that UI wage records more accurately identified how many veterans got jobs after receiving DVOP, LVER, or ES services. Using UI wage records, this study tracked veterans who registered with the Maryland Job Service during program year 1997 and found an entered-employment rate that ranged from 65 percent to 82 percent, depending on the way the study defined a registrant. In that same program year, Maryland reported to VETS an entered-employment rate of 31 percent, which was based on staff telephoning veterans and employers to verify employment. In addition, most states in our review that are now using UI wage records, either as their primary data source or to augment other data sources, reported higher employment rates in program year 1999 for veterans they served than that year’s national average of 30 percent. (See app. II for a list of all states and their respective entered-employment rates for program years 1996-1999.) By comparison, all but one of the states that relied either on manual follow-up or the new hire database reported an employment rate below the national average.

Another benefit of using UI wage records is that staff assisting veterans will be relying on data already available rather than collecting additional information from veterans or employers. Relying on these already reported data would require less staff time from DVOP, LVER, and ES staff, freeing them to focus more on providing job-related services to veterans. State officials told us that relying on manual follow-up, such as telephone calls, has been labor-intensive and has diverted staff attention away from providing appropriate assistance to veterans.

UI Data Presents Some Challenges

While UI wage records offer advantages over the current data collection system, some challenges need to be addressed. First, states should find ways to identify interstate job placements. Because the UI wage record system resides within each state, states generally do not have access to wage records from other states, making it difficult to track individuals who receive services in one state but get a job in another. Currently, there is no national system in place that facilitates data sharing among states. However, in response to WIA requirements, states are developing an interstate UI wage record information sharing system, known as the Wage Record Interchange System (WRIS). The system is designed to minimize the burden on state unemployment insurance programs in responding to requests for wage record data, to ensure the security of the transactions involving individual wage records, and to produce the results at a low cost per record. In addition, some states have entered into agreements with neighboring states to share wage information in support of WIA. These efforts should help VETS as well.

Second, states should find ways to identify those veterans finding jobs in categories not covered by UI wage records. UI wage records cover about 94 percent of wage and salary workers, but certain employment categories are not covered, such as self-employed persons, most independent contractors, military personnel, federal government workers, railroad employees, some part-time employees of nonprofit institutions, and employees of religious orders. Therefore, the UI system will not be able to track and count veterans who get these types of jobs. This is an issue for WIA as well, and states are beginning to assess the extent to which this issue will affect their ability to accurately determine the outcome of WIA- funded programs.

Other Measurement Issues Affect Comparability of States’ Performance Data

There are other issues not related to the use of UI wage records that VETS should consider as it finalizes its performance-reporting requirements. VETS’ proposed performance system does not standardize how states report veterans or nonveterans who use self-service activities, making it difficult to reliably assess nationwide performance. In an environment in which self-service is becoming more common, we found that states vary in whether they register veteran job seekers who access self-service tools, such as internet-based job listings or resume writing software. For example, some states allow job seekers greater access to job listings without requiring that they register, while others have more restrictions on who can access job lists. Table 3 shows how such differences can affect entered-employment rates. In this example, 100 veterans enter the employment service for assistance. In both cases, 40 veterans ultimately get jobs after receiving identical services. In one case, the placement rate is 40 percent and in the other, 50 percent—a 10 percentage point difference. This difference results from counting all job seekers in one case and only those requiring staff assistance in the other. As a result of the different ways states currently count veterans and report outcomes, the entered-employment rate measure is not consistently calculated across states, and nationwide comparisons are misleading.

VETS’ proposed performance system does not standardize how long a veteran or nonveteran remains registered after seeking services for performance-reporting purposes. We found that states differ in how long they keep veterans registered. This difference affects the calculation of the entered-employment rate (i.e., the number of veterans that get jobs), making performance comparisons across states less reliable. Many of the states we contacted count individuals as registered who have received a service in the last 6 months. However, two states only count those as registered who have received a service in the last 3 months, while two others count only those who received a service in the last 2 months. And in one state, anyone who has received a service from the state’s employment office since 1998 is counted as a registrant when determining the entered- employment rate. States with shorter registration periods may be able to report a higher entered-employment rate than states with longer registration periods.

Conclusions

VETS is improving its performance measurement system by proposing new measures that are more outcome-oriented than its current measures and by requiring that all states use wage record data to improve the comparability and reliability of reported program performance. While these changes move VETS a step closer to implementing an effective accountability system, they may not go far enough. VETS continues to send a mixed message to states about what services to provide and to whom. As presently defined, two of the proposed measures—the entered- employment rate and the employment rate following staff-assisted services—may provide nearly identical results, and neither helps VETS to monitor whether more intensive services are being provided to veterans or whether these services are successful. VETS also continues to inconsistently identify the groups of veterans that it wants states to help. In addition, VETS maintains a measure related to federal contractors—one that is beyond the control of DVOPS and LVERs.

Furthermore, in its proposed system, VETS allows states to decide which veterans to include in its performance reports. This results in data inconsistencies that make state-to-state comparisons unreliable. Without clear and consistent direction from VETS’ planning documents and performance measures, staff assisting veterans will be uncertain where to place their priorities. In addition, without stricter guidelines for how to count veterans, VETS will be unable to accurately assess program performance nationwide. Unless further modifications are made, VETS will be unable to fully determine whether its programs and services are fulfilling its mission.

Recommendations

In order to establish a more effective performance management system, we recommend that the Secretary of Labor direct VETS to do the following: Redefine staff-assisted services to include only those that may be considered staff intensive, such as case management, so that VETS will be able to evaluate the success of intensive staff-assisted services.

Clearly define target populations so that staff assisting veterans know where to place their priorities. If staff are to focus on assisting veterans who need more assistance, VETS should provide incentives and opportunities to do so through appropriate performance measures or negotiated levels of performance.

Eliminate the measure related to federal contractor jobs so that staff are not held accountable for the number of federal contractors in a state or local area or for the failure of contractors to list their jobs with ES offices.

Establish and communicate guidelines that standardize how to count veterans for performance-reporting purposes so that VETS will be able to assess program performance nationwide.

Agency Comments and Our Evaluation

We provided VETS with the opportunity to comment on a draft of this report. Formal comments from VETS appear in appendix III. In addition to the comments discussed below, VETS provided technical comments that we incorporated where appropriate.

VETS generally agreed with our findings and two of our recommendations but disagreed with the other two recommendations. VETS acknowledged that its current strategic plan (Nov. 2000) sends a mixed message to the states about which groups of veterans staff should target for special attention. VETS noted that it is revising its strategic and annual plans to reflect a more consistent message about what services to provide and to whom. VETS also explained that it is developing new performance standards specific to DVOP and LVER staff that will clarify the role they play in providing services to veterans. According to VETS officials, states will have the option of using these specific standards or developing their own. When developing these standards, VETS will need to ensure that the specific standards developed for DVOPS and LVERs are consistent with the message in the revised strategic plan and that together they provide a coherent strategy as to where staff should place their service priorities.

VETS disagreed with our recommendation for a revised definition of the performance measure related to staff-assisted services. VETS said that any veteran receiving staff-assisted services may require a multitude of the services cited in the definition—any one of which or combination thereof may require extensive staff time. We disagree that any one of these services necessarily requires extensive staff time. As noted in our report, a veteran may be counted as receiving staff-assisted services after receiving only a job referral or labor market information—services that by themselves would not involve extensive staff resources. Moreover, we continue to believe that the broadly defined staff-assisted service measure will likely not report outcomes substantially different from those reported for the more general entered-employment rate measure. As noted in our report, a stricter definition for staff-assisted services that includes only those services generally considered to be staff-intensive would allow VETS to more accurately assess outcomes associated with those services.

VETS disagreed with our recommendation to discontinue the measure related to jobs listed by federal contractors. However, VETS agreed to reconsider the suitability of this specific measure after public comments have been received. As we noted in our report, the presence of federal contractors in a given state or local area is determined by the federal agencies awarding contracts. In addition, state officials told us that it is the responsibility of the contractors, not DVOP and LVER staff, to list their job openings with employment services. Current law requires the Secretary of Labor to report annually to the Congress on the number of federal contractor positions listed and the number of veterans receiving job priority through this program. This information could be collected in absence of a specific performance measure.

With regard to our recommendation that VETS establish guidelines that standardize how states count veterans for performance-reporting purposes, VETS said that it will be working with ETA to determine how states can uniformly report veterans and nonveterans that use self-service activities. In addition, VETS noted that the revised ETA 9002 report will provide uniform instructions on how long individuals remain registered in the system.

We are sending copies of this report to the Honorable Elaine L. Chao, Secretary of Labor; appropriate congressional committees; and other interested parties. We will also make copies available to others upon request.

If you or your staff have questions about this report, please contact me on (202) 512-7215 or Dianne Blank on (202) 512-5654. Individuals making key contributions to this report include Elizabeth Morrison and Amanda Ahlstrand.

Appendix I: Comparison of VETS, ES, and WIA Performance Measures

Similar to Workforce Investment Act (WIA) programs, the Employment Service (ES) and the Veterans’ Employment and Training Service (VETS) are proposing that their programs use Unemployment Insurance wage records to report on performance measures. Each calendar quarter, employers submit wage record data to their state’s UI agency or some other state agency. The following table compares the proposed performance measures of VETS and ES and those used by WIA’s adult and dislocated worker programs.

ES proposed performance measures Entered-employment rate: The percentage of workers who got a job in the 1st or 2nd quarter after registration.

WIA performance measures (adult and dislocated worker programs) Entered-employment rate: The percentage of workers who got a job by the end of the 1st quarter after exit.

Employment retention rate: Of those who had a job in the 1st quarter after exit, the percentage of workers who have a job in the 3rd quarter after exit.

Employer customer satisfaction: Average of three survey questions on employers’ satisfaction with services received. Job seeker customer satisfaction: Average of three survey questions on job seekers’ satisfaction with services received.

Employer customer satisfaction: Average of three survey questions on employers’ satisfaction with services received. Job seeker customer satisfaction: Average of three survey questions on job seekers’ satisfaction with services received.

WIA performance measures (adult and dislocated worker programs) Earnings change (adults only): The difference between total post-program earnings (from the 2nd and 3rd quarters after exiting the WIA program) and the total pre-program earnings (from the 2nd and 3rd quarters prior to entering the WIA program) divided by the number of participants leaving the program. Earnings replacement rate (dislocated workers only): Total post-program earnings (in the 2nd and 3rd quarters after exit) divided by pre- dislocation earnings (in the 2nd and 3rd quarters prior to dislocation).

Staff-assisted services include: (a) referral to a job; (b) placement in training; (c) assessment services, including an assessment interview, testing, counseling and employability planning; (d) career guidance; (e) job search activities, including resume assistance, job search workshops, job finding clubs, specific labor market information and job search planning; (f) federal bonding program; (g) job development contacts; (h) tax credit eligibility determination; (i) referral to other services, including skills training, educational services and supportive services; and (j) any other service requiring expenditure of time. Application taking and/or registration services are not included as staff- assisted services.

Appendix II: States’ Entered-Employment Rates for Veterans in Program Year 1999

Appendix III: Comments From the Department of Labor