Vocational Rehabilitation: Better Measures and Monitoring Could Improve the Performance of the VR Program
Released: 2005-09-23T00:00:00

Letter

Background

Title I of the Rehabilitation Act of 1973 authorizes a federal-state VR program to provide services to persons with disabilities so that they may prepare for and engage in gainful employment. Education provided an estimated $2.6 billion in fiscal year 2005 in grants to the states and territories based on a formula that considers the state’s population and per capita income. Grants to individual states ranged from about $9 million to nearly $250 million. Four of the five territories received less than $3 million each. The act generally requires states to match federal funds at a ratio of 78.7 percent federal to 21.3 percent state dollars.

Each state and territory designates a single VR agency to administer the VR program, except where state law authorizes a separate agency to administer VR services for individuals who are blind. Education provides a single Title I grant to each state. States authorizing a separate blind VR agency decide how the grant will be apportioned between the general VR agency and the blind VR agency. Education tracks the performance of 80 state VR agencies—24 states have separate blind and general agencies; and 26 states, the District of Columbia, and five territories each have a single combined agency. The 80 state VR agencies are housed in various departments of state government, such as state departments of labor or education, or they may be free-standing agencies or commissions. State VR agencies also vary in their operations and locations. For example, some agencies provide services through several offices located throughout the state, while others provide services through one central location.

Education collects information about all individuals who exit each state VR agency’s program during a particular fiscal year, as reported by the 80 state VR agencies. The record for each individual exiting the program includes information such as whether or not each individual became employed, the weekly earnings and hours worked for individuals if they exited the VR program with employment, the types and costs of services they received, and demographic factors, such as impairment type, gender, age, race and ethnicity, public benefits, and income from work at the time of application. Education also collects summary information on agency expenditures in a number of categories from each state VR agency.

Education tracks individuals in terms of seven types of case closures, which can be collapsed into four categories, for individuals who exited without employment, during the application phase (including individuals who were found ineligible; individuals who could not be determined to be eligible or ineligible for various reasons such as they could not be located or contacted, they failed to cooperate or they refused services; and individuals who were found eligible, but were placed on a waiting list); exited without employment, with limited services (including individuals who were found eligible, but who left the program before an employment plan could be developed, or agreed to an employment plan, but left before receiving services under that plan); exited without employment, after receiving services under an employment exited with at least 90 days of employment, after receiving services under an employment plan.

Education considers several types of work activities to meet its definition of employment. First, Education counts as employment the paid work activity of an individual, who may or may not require ongoing support services, in an integrated work setting, that is, a setting typically found in the community where individuals both with and without disabilities interact. Second, Education counts self-employment as employment, whether the business is managed by the eligible individual or the state VR agency. Finally, Education considers certain types of unpaid work activity to be employment, such as homemakers whose work activity is keeping house for themselves or others in their households and unpaid workers in a family business or family farm.

While the total number of individuals exiting the VR program has increased slightly over the past several years, the number of individuals exiting with employment has remained relatively stable. (See fig. 1.)

State VR agencies that determine they will not be able to serve all eligible individuals who apply for services are required to state the order in which they will select individuals for services. Agencies using an order of selection process must develop criteria for ensuring that individuals with the most significant disabilities will be selected first for services. Thirty- nine of the 80 state VR agencies were using an order of selection process in fiscal year 2003. Beginning fiscal year 2004, 42 of the agencies are using an order process.

The Rehabilitation Act requires state VR agencies to enter into cooperative agreements with other entities that are part of the state’s workforce investment system. This workforce investment system includes a One-Stop system, which is required to provide a number of employment-related services to job seekers and employers at a single location. The act also requires state VR agencies to coordinate with public education officials to facilitate the transition of students with disabilities from school to work. Students with disabilities receive special education and related services from their school under an individualized education program (IEP). Beyond these required interactions, state VR agencies may also enter into third-party cooperative agreements with other state or local agencies to coordinate the services provided to their common program participants.

Education must abide by several statutes and executive branch directives to measure and monitor the performance of the VR program. The 1998 amendments to the Rehabilitation Act required that Education establish and publish evaluation standards and performance indicators for the VR program. The standards and indicators were supposed to include outcome and related measures of program performance that facilitate the accomplishment of the purpose and policy of the act. The act also gave Education the authority to reduce or suspend payments to state VR agencies that have performance falling below a certain level and fail to enter into a program improvement plan or substantially comply with the terms and conditions of such a plan. The act also directed Education to conduct annual review and periodic on-site monitoring of state VR agencies to determine, in part, whether they were complying with the standards and indicators. Education performs this monitoring function through the 10 regional offices of its Rehabilitation Services Administration (RSA).

In response to the 1998 amendments, Education established new performance measures in June 2000 that consisted of two standards for evaluating the performance of the state VR agencies, one relating to the agencies’ performance in assisting individuals in obtaining, maintaining, or regaining high-quality employment and the other relating to the agencies’ performance in ensuring that individuals from minority backgrounds have equal access to services. In addition, Education published performance indicators that establish what constitutes minimum compliance with these evaluation standards and required performance targets for each indicator. Six performance indicators were published for the employment standard, and one was published for the minority service standard. State VR agencies must meet or exceed performance targets in four of the six categories for the first standard and meet or exceed the performance target for the second standard in order to have passing performance. Table 1 provides details on these standards and indicators.

The Government Performance and Results Act of 1993 (GPRA) also requires federal executive branch agencies such as the Department of Education to set goals, measure their performance, and report on their accomplishments. Agencies are required to develop annual performance plans that use performance measurement to reinforce the connection between the long-term strategic goals outlined in their strategic plans and the day-to-day activities of their managers and staff. Among its performance goals for fiscal year 2005, Education is assessing its performance in assisting state VR agencies to achieve required performance targets on one performance target—1.2.

In 2002, the Office of Management and Budget (OMB) directed that the performance of a range of federal job training and employment programs be measured consistently to allow for the comparison of results across these programs. These common measures would be consistent with the common goals of these programs, that is, to improve participants’ employment and earnings and focus on measures of outcomes and efficiency. OMB identified the VR program as one of the federal programs that would be targeted for using the common measures.

Using its Program Assessment Rating Tool (PART), OMB assessed the effectiveness of the VR program in 2003 as part of its effort to hold federal agencies accountable for accomplishing results. The PART evaluation looks at four areas of assessment—program purpose and design, strategic planning, management, and results and accountability. Programs are rated in one of five categories: effective, moderately effective, adequate, ineffective or results not demonstrated. OMB will use the rating and relating findings to make decisions about budget and policy. OMB rated the effectiveness of the VR program as adequate and made recommendations to Education for improving program management and performance measures. As part of its assessment, OMB reviewed Education’s performance indicators 1.2, 1.3, and 1.5.

One-Third of Individuals Exited the VR Program Nationwide with Employment, although Employment Rates Varied Significantly among State VR Agencies

More than 217,000 individuals with disabilities exited the state VR programs with employment in fiscal year 2003 after receiving customized services. This group represents one-third of the 650,543 individuals who left the program nationwide in fiscal year 2003 after submitting an application for services. The most common reasons that the remaining two-thirds of the individuals left the program without a job were that the individual refused services, failed to cooperate, or could not be located or contacted. State VR agencies collectively purchased more than $1.3 billion in services for all individuals who exited the program in fiscal year 2003, two-thirds of which was used to provide services to individuals exiting with employment. Employment, earnings, and the amount of purchased services received while in the VR program varied significantly by individuals’ type of disability and other characteristics. In addition, the state VR agencies varied substantially in the employment rates they achieved, the characteristics of individuals they served, their frequency of providing certain services, and their service expenditures.

One-Third of Individuals Exited the VR Program Nationwide with Employment in Fiscal Year 2003

Of the more than 650,000 individuals exiting the VR program in fiscal year 2003, one-third (217,557) obtained a new job or maintained their existing job for at least 90 days after receiving customized services. (See fig. 2.) Most of these individuals (94 percent) exited the program with jobs that paid at least their state’s minimum wage, but about half of them worked less than 40 hours per week. Overall, individuals who exited the VR program with employment earned a median income of $271 per week, or the equivalent of $14,092 per year. In addition, 30 percent of these individuals (65,832) were already working when they applied to the program, and they increased their median earnings from $225 to $300 per week between program entry and exit. One state VR agency official noted that this figure may, however, underestimate the actual value of VR services extended to individuals working at both program entry and exit. For example, the VR program will pay for services such as vehicle modifications and repairs necessary to help some individuals maintain transportation to and from existing jobs, but these individuals typically do not experience any earnings increase between program entry and exit.

Overall, two-thirds (433,086) of individuals exited the VR program without employment at some point following their initial application to the program. Of those who exited without employment in fiscal year 2003, most did so because they refused services or failed to cooperate with their VR counselor (46 percent of the time) or could not be located or contacted (24 percent). (See fig. 3.) One state VR agency official told us that the VR program has historically closed a large number of cases because individuals cannot be located or contacted. However, she also noted that individuals with disabilities coming to the VR program are often a transient population with high rates of poverty and other multiple barriers—issues that can require more time and priority than notifying VR counselors that they have moved.

Of those who exited during the application phase in fiscal year 2003, the majority (105,955) left before an eligibility determination could be made, and relatively few (26,563) left because the VR program found them ineligible. Specifically, Education’s data show that 20 percent of individuals who exited during the application phase were found ineligible: 2 percent had disabilities deemed too significant to benefit from services, and 18 percent had no disabling condition, impediment to employment, or need for VR services.

Overall, the VR program invested nearly two-thirds ($872 million) of its $1.3 billion in purchased services on individuals who achieved or retained employment upon exiting the program in fiscal year 2003. State VR agencies also spent nearly $200 million on individuals who subsequently exited the program without employment because they failed to cooperate or refused services, and $112 million was spent on those whom state VR agencies were unable to locate or contact. As shown in table 2, individuals’ average length of time in the program, number of services, and cost of purchased services received varied by each of the four exit categories. However, the amounts of purchased services reported by state VR agencies do not reflect the total cost of services provided to individuals in the VR program. For example, state VR agencies do not report the cost of counselor time spent with each individual or the cost of services arranged for by the state VR agency but paid for by other sources. In addition, the amounts of purchased services do not reflect the amounts that individuals are required to pay for certain services at the majority of state VR agencies if they demonstrate the financial ability to do so.

Services received by individuals in the VR program varied both by whether they exited the program with employment after receiving services under an employment plan and whether they were employed at entry to the program. Regardless of whether they were working at application, individuals who received services under an employment plan but exited the VR program without employment received fewer job-related services, such as job search, job placement, or on-the-job supports than individuals who exited with employment. (See table 3.) In general, individuals not working when they applied to the VR program received more services than those who were already working when they applied to the program. However, individuals already working were more likely to receive diagnosis and medical treatment of their impairment as well as rehabilitation technology.

Individuals may also benefit from the VR program in important nonmonetary ways, aside from employment and earnings. For example, regardless of whether individuals leave the VR program with employment, they may increase their educational level, psychological or physical functioning, productivity in the work setting, independence for self or family members, or integration into the community as a result of receiving VR services. However, Education does not collect data on any nonmonetary benefits that individuals achieve through the VR program aside from educational gains and identification of homemakers and unpaid family workers. Moreover, it is difficult to measure the actual influence of the VR program in assisting individuals to obtain these benefits, employment in general, or any wage increases.

Rate of Employment, Earnings, and Purchased Services Varied by Individuals’ Impairments and Other Characteristics

Individuals exiting the VR program in fiscal year 2003 had a variety of primary impairments, and individuals with mental or psychosocial impairments (including depression, schizophrenia, and drug and alcohol abuse, among others) constituted the largest group. (See fig. 4.) About 40 percent of all individuals exiting the VR program in fiscal year 2003 also had secondary impairments.

Individuals with mental or psychosocial impairments collectively realized the lowest rate of employment (30 percent). In contrast, deaf and blind individuals, each collectively comprising 5 percent of the total VR population, achieved the highest rates of employment nationwide, at 63 percent and 52 percent respectively. (See fig. 5.) Officials at three state VR agencies we visited told us that the availability of state-funded supports and follow-along services were influential in placing and keeping individuals in jobs, especially for individuals with mental impairments.

Individuals with cognitive impairments (including mental retardation and specific learning disabilities) collectively achieved the lowest median rate of earnings, compared with those for all other impairment groups exiting the VR program with employment, as shown in figure 6.

Individuals exiting the VR program in fiscal year 2003 after having previously participated in the VR program did not necessarily gain employment after their repeat involvement. More than 96,000 individuals (15 percent) exited the VR program in fiscal year 2003 with a prior VR case closure during the previous 3 years. As figure 7 shows, nearly three- quarters of those who previously exited the program without employment failed to achieve employment after their repeat involvement with the VR program.

Individuals also received varying amounts of purchased services from state VR agencies, depending on their type of disability. For example, among those who exited in fiscal year 2003 with employment and did not require ongoing support services, blind or other visually impaired individuals received more in purchased service dollars ($2,889 at the median) than any other impairment group, while individuals with cognitive ($1,385) and mental impairments ($1,566) received the least. Officials at several state VR agencies for the blind told us that blind and visually impaired individuals generally require more expensive services, such as assistive technology, than individuals with other types of disabilities.

Individuals receiving Social Security disability benefits at application to the VR program comprised one-quarter (159,739) of the total VR population and collectively achieved a lower rate of employment nationwide than nonbeneficiaries in fiscal year 2003. Specifically, 29 percent of beneficiaries exited the VR program with employment in fiscal year 2003 as opposed to 36 percent of nonbeneficiaries. In addition, more Social Security disability beneficiaries required ongoing support services at their jobs, for earnings that were less than half that of nonbeneficiaries ($140 versus $300 per week at the median). Beneficiaries exiting with employment (47,142) also received a median $2,765 in purchased services during their time in the VR program, or nearly $1,000 more than nonbeneficiaries at the median. Separately, beneficiaries’ receipt of VR services did not necessarily reduce their dependence on Social Security disability benefits. For example, among individuals receiving SSDI benefits at their time of exit from the VR program with employment in fiscal year 2003, more than 82 percent of blind and visually impaired beneficiaries and 65 percent of nonblind beneficiaries did not have jobs that were paying enough to disqualify them from receiving continued SSDI benefits, if the work was sustained at that level.

Individuals who transitioned from special education services at their schools into the workplace (transitioning students) achieved about the same rate of employment as other individuals with impairments who were under age 22 when they applied to the VR program. (See table 4.) However, the latter group was in the VR program almost twice as long and received more than double the amount of purchased services than transitioning students, and enrolled in or completed a greater degree of postsecondary education than transitioning students. This might be related to the fact that more than 70 percent of the transitioning students had cognitive impairments (as opposed to 36 percent among their same-age counterparts), of which more than half were specific learning disabilities.

State VR Agencies Varied Significantly in the Rates of Employment They Achieved, the Populations They Served, and Their Service Expenditures

The 80 state VR agencies’ employment rates ranged from 20 to 74 percent in fiscal year 2003. (App. II lists all agencies’ employment and other exit rates.) Collectively, the separate state VR agencies for the blind, which exclusively serve blind or other visually impaired individuals, achieved an employment rate of 49 percent. This compared with a collective employment rate of 32 percent among the general agencies and 35 percent among the combined agencies. A study commissioned by Education found that a wide range of agency and external factors may help explain state VR agencies’ varied employment rates. Specifically, variables found to help facilitate higher employment rates included an agency’s emphasis on employment, access to ongoing support service programs, and an agency’s proportion of individuals with mental retardation, visual impairments, or existing employment at the time of application to the VR program. Conversely, poor labor market conditions, particularly high unemployment rates, were reported as being among the most influential hindrances to an agency’s performance.

The 39 state VR agencies using an order of selection process collectively achieved a slightly lower rate of employment in fiscal year 2003 than the group of 41 agencies not using an order of selection process. Specifically, the collective employment rate was 32 percent among agencies with orders and 35 percent among agencies without orders. In addition, both groups had a wide range of employment rates among individual agencies. Specifically, state VR agencies’ rates of employment ranged from 20 to 62 percent among agencies with orders and from 21 to 74 percent among agencies without orders. State VR agencies with orders are required to give priority acceptance to the category of applicants with “most significant” disabilities, and accept categories of applicants with “significant” and “nonsignificant” disabilities with decreasing priority. As a result, while we expected the group of agencies with orders to experience more individuals exiting during the application phase, the rate of individuals exiting during the application phase was 21 percent among agencies with orders as opposed to 19 percent among agencies without orders. Moreover, agencies with orders varied in how they administered their order of selection policies. For example, agencies with orders opened and closed their priority service categories at varying points during the year, depending on their available resources and population projections, thus affecting the proportion of individuals being accepted into their programs with most significant or significant disabilities. Further, agencies with orders varied in how they defined a most significant disability and how many service categories they expected to serve during a particular fiscal year. Specifically, according to the state plans submitted to Education by state VR agencies for fiscal year 2003, 2 of the 39 agencies with orders expected to serve only individuals with most significant disabilities. In contrast, 11 agencies had orders of selection in place for fiscal year 2003, which indicated they could not serve all eligible individuals, yet they expected to serve all eligible individuals who applied to their program during fiscal year 2003.

State VR agencies’ proportion of individuals in each impairment category varied significantly. (App. III shows the proportion of individuals in each impairment category by agency.) For example, the incidence of individuals with mental or psychosocial impairments at combined or general state VR agencies ranged from 1 percent to more than 50 percent. Education officials told us that one of the reasons for such variation may be state VR agencies maintaining third-party cooperative agreements with certain outside entities, such as state mental health institutions, which result in higher proportions of individuals in their VR population with corresponding impairments.

State VR agencies also varied in the proportion of Social Security disability beneficiaries they served, from 7 to 66 percent, and Education and state VR agency officials told us that this proportion can influence an agency’s overall employment rate and average hourly earnings among those exiting with employment. In addition, although beneficiaries’ employment rate nationwide was 29 percent, their collective employment rates at individual state VR agencies ranged from 9 to 68 percent. (See app. IV for a list of each agency’s employment rate.) Moreover, agency officials in two states told us that because their state legislatures passed Medicaid buy-in programs, they expected increasing numbers of Social Security disability beneficiaries to begin working or earning higher wages because they no longer feared the loss of important health care benefits, which many automatically received as disability beneficiaries.

State VR agencies also varied in the extent to which they worked closely with the One-Stop system in their states, although Education’s data did not completely capture this variation. Education’s data showed limited interaction overall between state VR agencies and One-Stop centers. For example, it showed One-Stop centers providing services to less than 1 percent of all individuals who exited the VR program nationwide in fiscal year 2003. In addition, Education’s data indicated that 3 percent of all individuals who exited the program in fiscal year 2003 were initially referred to the program by a One-Stop center. However, these data understate the extent of integration by state VR agencies in some states with One-Stop centers or WIA program partners. For example, all of Minnesota’s VR offices are co-located at One-Stop centers, and officials at each of the other state VR agencies that we visited told us they had at least some staff colocated at One-Stop centers on an itinerant basis. State VR agency officials in Minnesota cited a number of benefits to office colocation, such as additional services being more readily available, and credited the leadership both within their state parent agency at the Department of Employment and Economic Development and within local regions for the successful colocation of their VR offices. In contrast, officials at most of the other state VR agencies that we visited told us that their interaction with the One-Stop centers in their state was limited and that individuals in their VR programs did not receive very many, if any, services at the One-Stop centers, for a variety of reasons. They told us that these reasons included delayed or late implementation of One-Stop centers in their state, leasing and other legal problems relating to office colocation, lack of physical accessibility at some One-Stop centers, and the lack of specialized training or services available at One-Stop centers necessary to help individuals with disabilities to obtain or advance in employment.

While state VR agencies varied in the proportions they spent on different service categories, there were general trends across all agencies with respect to expenditures on certain services. Overall, agencies largely focused on assessment, guidance, counseling, and job placement, but did not focus as much on certain other services, such as postemployment services, transportation, or personal assistance services. (See app. V for a list of the percentages spent by each state VR agency on each service category.) Specifically, state VR agencies spent half of their collective total caseload “service budgets” in fiscal year 2003 on assessment, guidance, counseling, and placement. In contrast, they spent less than 1 percent on postemployment services for individuals who previously exited the VR program with employment but required additional services to maintain or advance in their existing jobs. State VR agency officials told us that VR counselors are generally not recognized, in terms of achieving an employment exit, for providing postemployment services. However, Education and several state VR agency officials told us that postemployment services are important for some individuals to maintain their employment. For example, services such as updated computer- assistive technology or software help to coordinate with an employer’s new computer system or ongoing mental health services help individuals with mental impairments to cope with new psychosocial issues that arise while on the job.

State VR agencies varied in the proportion of their total fiscal year 2003 budgets that they spent on providing services, from 43 to 95 percent. Their remaining expenditures were for administrative costs, which varied from 5 to 57 percent of their total budgets. (See app. VI for a list of each state VR agency’s administrative costs as a percentage of total expenditures.) The 24 blind state VR agencies collectively spent more of their total budgets on administrative costs than did the collective group of general agencies or combined agencies. This may be because of the cost of maintaining separate blind VR agencies that on average serve fewer individuals than the general or combined agencies. In addition, the group of blind agencies had the highest per capita cost overall for assisting individuals to achieve employment. Specifically, blind agencies spent an average of $42,392 for every person that exited one of their VR programs with employment in fiscal year 2003, compared with an average of $13,640 at the general agencies and $21,501 at the combined agencies. (See table 5.) For each state VR agency’s average total expenditure per person exiting its program with employment in fiscal year 2003, see appendix VII.

Officials at one separate state VR agency for the blind that we visited said their agency’s specialization allows them to understand and rehabilitate blind and visually impaired individuals better than combined VR agencies that do not have as specialized a focus on individuals with these impairments. However, blind and visually impaired individuals collectively achieved a 50 percent employment rate when served at blind state VR agencies, compared with a 56 percent employment rate when served at combined state VR agencies. Moreover, blind and visually impaired individuals exiting the VR program with employment in fiscal year 2003 received less in purchased services while in the program and slightly lower weekly earnings when they exited from blind state VR agencies as compared with combined agencies. (See table 6.)

Education’s VR Performance Measures Are Not Comprehensive, and Its Monitoring of State VR Agencies’ Performance Has Not Resulted in Timely Feedback

Education’s VR performance measures are not comprehensive in that they count only individuals exiting VR programs and fail to measure key populations. Moreover, the targets Education sets for performance do not take into consideration regional differences in VR populations or allow for comparisons across workforce programs. Education’s monitoring reports—state VR agencies’ primary source of feedback—are frequently late and based on data that are more than 2 years old. As a result, state VR agencies are not getting the kind of timely feedback they need to improve the efficiency and effectiveness of their programs. Education’s recent decision as part of its restructuring efforts to eliminate its regional VR offices, which had conducted most of the monitoring of state VR agencies, has made the details of the future monitoring process unclear.

VR Performance Measures Are Not Comprehensive, Do Not Take into Account State Variations, and Do Not Allow for Comparison with Other Workforce Programs

Education’s performance measures have a number of weaknesses, as they are not comprehensive, do not take into account demographic and economic variations among states, and do not incorporate common measures to allow for comparison of workforce programs across executive branch agencies. For example, Education cannot use the current performance measures to comprehensively evaluate the state VR agencies’ performance because the measures do not include data on the individuals still receiving services in the VR system, who made up nearly 40 percent of the state VR agencies’ service population in fiscal year 2003. Instead, Education’s performance measures reflect only the individuals who exit the program. Education does not track specific information on the services provided, costs, and related data until after individuals exit the program, although this information is generally collected by each state VR agency. As a result, Education cannot determine how well the program is accomplishing its purpose of assisting individuals with disabilities to maximize their employment, economic self-sufficiency, independence, and inclusion and integration into society. Additional measures could also add to the balance of performance measurement to ensure that the organization’s various priorities are covered and to prevent skewed incentives. In fact, a study commissioned by Education reported that because the VR employment rate includes only those who exited the VR program, some agencies avoid closing out certain cases in order to meet the performance target for that year.

The performance measures also do not isolate data on certain key populations of VR participants. For example, there are no separate performance targets for transitioning students or individuals receiving postemployment services. As a result, Education does not know the extent of these populations, the services provided to them, or the results they achieve. While Education focused on transitioning students in its fiscal year 2003 monitoring guide and held a national conference on transitioning students in June 2005, it does not use the data captured on this population in performance measures or evaluate the results and resources necessary to assist them. In contrast, according to Education officials, Education wants to move away from counting homemakers as a category of employment but continues to measure performance in this area, including homemakers in its employment outcome. In addition, Education does not capture individual-level data related to postemployment services, which some state VR agencies told us are important in helping former participants maintain their employment, despite the need for these measures to help assess the impact of these services.

Education has set uniform performance targets for state VR agencies to meet that do not take into account demographic and economic variations within states. For example, one performance measure intended to measure the quality of job placement compares the average wage of individuals exiting the VR program with employment to the average wage of the general population within a state overall, rather than to the population in similar types of jobs or industries. Recognizing that the VR population is different from the general population, Education set the performance target for general and combined state VR agencies as a 0.52:1 ratio to the average state wage. However Education’s performance target still does not capture variations in wages within states, which are outside the agencies’ control and can affect agencies’ ability to achieve performance levels. For example, some states, such as California, have wide variations in wages across the state because of high wages in certain areas that might skew the wage and make it more difficult to perform well on this indicator. In addition, a study commissioned by Education found that equaling or exceeding achievement on another performance measure intended to compare the number of individuals exiting VR services in the current year with that in the prior is “very difficult to achieve in times of declining resources and a poor labor market,” both of which may vary by region or state. Unlike the Department of Labor (Labor), which negotiates performance levels for its job training and employment programs under WIA, Education does not currently negotiate performance targets for performance measures with each state VR agency. The same study recommended that Education evaluate the degree to which adoption of alternatives to average state wage, such as entry-level wages, median wage or national mean wage, might improve this performance measure. Education has considered negotiating performance targets by state but has not implemented negotiations.

In contrast to other federal workforce programs, Education’s VR program has not yet adopted the OMB-required common measures that allow for comparison of these programs across agencies, but agency officials told us that they are working toward meeting this requirement. OMB’s PART review recommended that Education collect the necessary data to support new common measures among workforce programs. Labor created a set of common measures for job training and employment programs, which affect programs in Labor, Education, and several other agencies, and apply to programs serving adults and to those serving youth. Labor required most of its programs to implement the common measures by July 1, 2005, and is working with Education to come to an agreement on the measures and the data it will use. While Labor’s state programs generally have access to the data necessary to compute the common measures as they are collected by state labor departments, some state VR agencies have more difficulty obtaining the data because access sometimes requires establishment of data-sharing agreements. In addition, some states have privacy laws protecting the confidentiality of these data, which may further limit the ability of state VR agencies to collect them. Finally, some state VR agency officials expressed concerns that common measures will invite unfair comparisons between VR and other WIA partner programs. For example, they told us that these comparisons would not be fair or valid for state VR agencies because the populations they serve are unique and require a different mix of services or more resources than the populations served by general workforce programs.

Education has not reviewed or revised its performance measures as required under the Rehabilitation Act, although Education plans to evaluate them within the next year. Specifically, the act requires Education to review and, if necessary, revise the evaluation standards and performance measures every 3 years. However, Education has not done so since the measures were first regulated in fiscal year 2000. Although Education’s existing performance measures are generally consistent with the program’s purpose to promote the employment of individuals with disabilities, OMB’s PART review, Education’s own study, and many of the state VR agency officials that we contacted, have recommended that Education modify some VR performance measures, eliminate some measures, or add to its existing measures.

Education Does Not Effectively Monitor and Manage the Performance of State VR Agencies

In monitoring and managing the performance of state VR agencies, Education does not provide timely feedback, censure poorly performing agencies, or take full advantage of opportunities to promote the sharing of best practices among state VR agencies. Education provides feedback to state VR agencies through on-site monitoring visits and reports of annual reviews of performance, many of which are not issued on a timely basis. According to Education officials, as of July 2005 its regional offices had not issued 10 of the 80 monitoring reports expected for fiscal year 2003 and an additional 4 reports are still in process. Further, at least 32 of the 66 monitoring reports issued as of July 2005 were issued 6 months or more after the monitoring reviews. In an effort to address the timeliness and quality of its monitoring reports, Education discontinued its practice of having regional commissioners approve reports and now requires all reports to be approved and issued by Education headquarters staff. However, as of July 2005 Education had issued 7 out of the 80 state VR agency monitoring reports for its revised fiscal year 2004-2005 time frame.

Education’s process for monitoring state VR agencies has been impeded by the use of old data. As part of its performance assessment of state VR agencies, Education requires them to submit performance data by December 1, or 60 days after the end of the fiscal year. Until recently, Education took more than a year to identify and correct errors in the data and produce reliable data for its regional offices to use in monitoring. As a result, regional offices end up using 2-year-old data when conducting monitoring reviews and issuing reports. In addition, Education consistently processes its VR data too late to meet the deadline of November 15 that OMB established in implementing GPRA for executive branch agencies to issue annual reports on program performance for the previous fiscal year. OMB’s 2006 PART review recommended that Education improve program management using existing performance data and make these data available to the public in a timelier manner. Education has shown improvement by issuing fiscal year 2004 performance data 5 months after it collected the data, although it still has not met OMB’s deadline.

Education does not consistently censure poorly performing or reward high- performing state VR agencies. The Rehabilitation Act requires state VR agencies that fail to meet the performance targets for 1 year to develop a program improvement plan and allows Education to withhold funds from a poorly performing state VR agency—the most severe sanction prescribed by the Rehabilitation Act—in cases where agencies fail to meet the performance targets for 2 or more consecutive years. However, failing agencies do not always develop program improvement plans nor does Education follow up to ensure their completion. In addition, Education has never withheld funds from a poorly performing state VR agency despite the fact that some agencies failed the performance targets for 2 or more consecutive years, including two agencies in fiscal year 2003. Education could not tell us the extent to which agencies have or have not developed improvement plans because of failing targets. Further, while Education does not have authority to provide financial rewards for high- performing state VR agencies, it has not developed other means for rewarding high performance, even though other federal workforce and education programs have developed performance incentives. For example, Labor awards incentives and imposes sanctions on workforce programs based on negotiated performance goals as allowed under WIA.

Beyond the monitoring process, Education relies on the regional offices to promote the sharing of best practices among state VR agencies. Meetings organized by the regional office provide opportunities for state VR agencies in the region to share information on best practices. Education also uses its Web site to disseminate information on best practices. However, as of July 2005 Education’s Web site related to featured practices contained two practices submitted by state VR agencies, and the site is not frequently updated. Further, Education collects a large repository of data and the results of performance measures, but it does not review them to identify best practices among the state VR agencies. Education also hosts several national conferences, which provide a venue for state VR agency staff to share information with one another. In addition, Education hosts e-mail list services relating to issues such as the Social Security Administration’s Ticket to Work program, deafness, mental illness, and informational issuances.

Decision to Eliminate Regional Offices Has Made Details of the Future Monitoring Process Unclear

In early 2005, Education decided to restructure its VR program. As part of the restructuring, Education decided to eliminate its regional offices, which had conducted most of the monitoring of state VR agencies to date. According to Education officials, the restructuring was necessary to align the program in a manner to help meet the priorities of the administration and the Secretary. As part of its restructuring plan, Education included an interim monitoring plan that did not contain many details. Education has created a steering committee to look at how monitoring might be accomplished under the new structure and held a conference in August 2005 in order to create a blueprint for monitoring, but it has not completed a final monitoring plan and does not expect to have one in place until about one year after the conference.

In the absence of a final monitoring plan, VR agency officials in several states expressed concern that the elimination of the regional offices may make the monitoring process more difficult for a number of reasons. Specifically, they expressed concern that Education’s proposed reduction of its RSA staff will leave an insufficient number of staff to provide any significant feedback and to carry out the monitoring functions, the reduction in staff would result in a loss of institutional knowledge on the details of their particular program, and the elimination of the regional offices will make it difficult to have a knowledgeable single point of contact within Education.

When announcing its decision to restructure, Education stated that it plans to devote as many staff as necessary to monitor state VR agency performance. In addition, an Education official said that the plans are to continue its annual reviews of the agencies, but it will conduct on-site monitoring review visits less frequently than the annual reviews done in the past. Education also announced that it will assign a single point of contact to each state VR agency under its new structure.

Conclusions

Many people with disabilities face a number of barriers to entering or returning to the workforce, and ultimately achieving independence. The VR program was conceived to provide the comprehensive and intensive services needed to overcome these barriers, and indeed more than 200,000 individuals with disabilities were working in fiscal year 2003 after receiving VR services. However, twice as many individuals left the VR program in fiscal year 2003 without getting a job, and individual state VR agencies varied widely in the proportion of individuals who exited their programs with jobs as compared with those who did not.

Education designed its system of performance measures to focus on the group of individuals who exit the VR program with employment. In doing so, it has made evaluating state VR agencies’ performance with respect to the rest of the service population difficult. Little is known about the hundreds of thousands of individuals who received services but had not left the program by the end of fiscal year 2003 because Education does not require the state VR agencies to report detailed information about them. As a result, it is difficult for Education to assess performance in areas such as the timeliness, cost, and quality of the services provided to this population. In addition, this system fails to account for performance relating to special populations, such as transitioning students, that Education has encouraged the state VR agencies to serve. Further, its performance measures do not take into account demographic and economic variations among states. As a result, it is difficult for Education to assess whether or not state VR agencies are effectively and efficiently serving these individuals.

Whatever system of performance measures Education chooses to use, it will have little impact on changing the performance of state VR agencies unless Education provides timely and effective feedback to the agencies regarding their performance. Without timely information, state VR agencies may delay in undertaking necessary corrective action to improve performance. Further, in the absence of incentives for good performance or consequences for poor performance, state VR agencies may not find sufficient reasons for performing at the level that Education sets out for them. Yet Education’s provision of constructive feedback has been hindered because of untimely or incomplete monitoring reports. Until recently, the time Education needed to analyze performance data also contributed to delays in providing feedback. Although Education significantly reduced the time it needed to analyze fiscal year 2004 data, it has yet the meet OMB’s reporting requirements related to GPRA. Further, Education has missed opportunities for providing feedback by failing to require that some poorly performing agencies develop performance improvement plans.

While Education has begun planning for alterations to its monitoring process to provide better and timelier feedback to state VR agencies, there are too few details at this point to be able to assess whether the new process will achieve this intent. As it deliberates on these changes, it will be important for Education to consider the input of all stakeholders in the current monitoring process. In addition, it will be important to consider such issues as how frequently monitoring visits should be made and how much data the monitoring staff will need to conduct such reviews.

Finally, with the elimination of the regional offices, officials in each of the 80 separate VR agencies may find it more difficult to learn about best practices used by other agencies. Education will need to explore alternative means to share information about best practices in an efficient manner.

Recommendations for Executive Action

To improve Education’s performance measures and its monitoring of state VR agencies, we recommend that the Secretary of Education Reevaluate Education’s performance measures to determine whether they reflect the agency’s goals and values and ensure that sufficient data are collected to measure the performance of this program. As part of this evaluation, Education should consider: developing additional measures to evaluate performance relating to individuals who remain in the VR program as well as to certain special needs populations, such as transitioning students; revising performance measures to account for additional factors such as the economy and demographics of the states’ populations or adjusting performance targets for individual state VR agencies to address these issues, while also considering the costs and benefits associated with collecting additional data; and continuing its work to develop performance measures in line with OMB’s common measures.

Take steps to continue improving the timeliness of the performance data Education collects and analyzes to ensure that data are available for timely feedback to state VR agencies as well as to comply with the GPRA reporting requirements established by OMB.

Ensure that Education’s new plan for the monitoring of state VR agency performance addresses issues such as the timeliness of monitoring reports and the frequency of on-site visits that will be necessary to adequately gauge performance.

Ensure that it consistently applies its existing consequences for failure to meet required performance targets or consider developing new consequences that it will be willing to apply in such situations. Further, Education should consider whether developing incentives such as recognition for successful performance would provide a more useful tool for encouraging good performance.

Develop alternative means of disseminating best practices among state VR agencies in light of the elimination of the regional offices, such as a central repository of best practices information.

Agency Comments and Our Evaluation

We provided a draft of this report to Education for comment. In commenting on the report, Education indicated that it is in full agreement that better measures and monitoring could improve the performance of the VR program. However, Education did not specifically comment on each recommendation.

In addition, Education highlighted initiatives either planned or under way to improve the management of the VR program. Several of these initiatives addressed issues raised in our report for which we recommended changes. In particular, Education agreed with our findings on performance measures and indicated that the department is currently working to address these issues. Education also described the steps it is taking to implement a new system for monitoring state VR agencies. Further, the agency indicated that it will broaden the dissemination of the information it produces and will publicize the availability of its monitoring and analytic work products. Moreover, Education acknowledged that it will do more to highlight performance related to certain key populations such as transitioning students.

While acknowledging the importance of monitoring and feedback, Education pointed out that state VR agency performance is influenced by a number of factors that are beyond Education’s control. It also pointed out that the state VR agencies are aware of their own patterns of expenditures and outcomes and that the most fundamental opportunities for improvement are at the state level. We recognize the challenge Education faces in managing a program that is carried out through agencies that are not under its direct control. However, we believe that, by establishing better performance measures and implementing a better monitoring system that includes dissemination of comparative information about the performance of state VR agencies, Education will be in a better position to manage this decentralized program.

Education also expressed concern that the manner in which we calculated the employment rate for individuals exiting the VR program might be confusing because it differs from the way in which Education calculates this rate. We believe that it is important to report the outcomes of all the individuals who exited the VR program in fiscal year 2003 because more than 40 percent of those who applied for services left the program before receiving services and because a low percentage of the individuals who left before receiving services were found ineligible.

Education also provided technical comments, which we have incorporated as appropriate. Education’s comments are provided in appendix VIII.

Copies of this report are being sent to the Department of Education, appropriate congressional committees, and other interested parties. The report will also be made available at no charge on GAO’s Web site at http://www.gao.gov. If you or your staff have any questions, please contact me at (202) 512-7215. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IX.

Appendix I: Scope and Methodology

To determine the extent to which state vocational rehabilitation (VR) agencies assist individuals in achieving employment, we primarily used the Department of Education’s (Education) Rehabilitation Services Administration (RSA) RSA-911 case service dataset from fiscal year 2003, which was the most recently available in time for us to use in production of this report. We used Education’s RSA-2 expenditure dataset from fiscal year 2003 to compute service expenditures and to compute the information in appendix V, appendix VI, and appendix VII.

The RSA-911 is an individual-level dataset collected annually by Education from each of the 80 state VR agencies regarding every individual who exits the VR program during a particular fiscal year. The record for each individual includes information such as whether or not the individual exited the VR program with employment, the weekly earnings and hours worked if the individual exited with employment, the types of services received, and numerous demographic factors, such as disability type, gender, age, race and ethnicity, public benefits at program entry and exit, and any income from work at program entry. Counselors or other staff at each state VR agency typically input this information from each individual’s case file into their agency’s data system, which ultimately produces the RSA-911 file sent annually by that agency to Education. Education uses standardized, electronic testing to check each agency’s submitted data for errors and anomalies before publishing them and requires agencies to correct or verify data elements that are missing, “impossible,” or outside “reasonable” ranges.

We assessed the fiscal year 2003 RSA-911 dataset and determined that it was sufficiently reliable for our use. Specifically, we performed electronic testing on all 650,643 case records contained in this dataset to identify any missing data, errors, or anomalies. In addition, we interviewed key Education officials responsible for collecting, verifying, and publishing RSA-911 data to better understand their data reliability assessment process, as described above, in part. We also performed structured interviews with key officials who work with RSA-911 data at multiple state VR agencies to better understand the processes by which agencies collect, safeguard, and report these data to Education.

In order to more easily report some of our data findings, we recombined the categories for certain RSA-911 data elements, such as type of disability and type of exit or case closure. We then computed descriptive statistics, including frequencies and cross-tabulations, to produce the majority of data findings described in this report. Using these data, we calculated the employment rate in a different manner than Education. Education calculates the employment rate as the percentage of individuals who exited with employment after at least 90 days out of the group of individuals who received services under an employment plan, but does not include those individuals who exited without employment during the application phase or after limited services.

We also assessed the fiscal year 2003 RSA-2 dataset and determined that it was sufficiently reliable for our use. The RSA-2 contains aggregated agency expenditures from each of the 80 state VR agencies as reported in various categories, such as administration and different types of services. We interviewed key Education officials and learned that each state VR agency may use a different system for aggregating and reporting its RSA-2 data to Education. Therefore, we performed structured interviews with key officials at 15 state VR agencies in order to assess the processes, safeguards, and overall reliability in agency production and reporting of these data. We selected these 15 state VR agencies to interview because they collectively constituted more than 50 percent of all total expenditures by the VR program in fiscal year 2003 and included blind, general, and combined state VR agencies. Among these 15 agencies, we generally found that their processes and policies were sufficient to ensure data reliability for the purposes of this report.

To further determine the extent to which state VR agencies assist individuals in achieving employment as well as the extent to which Education monitors and measures performance to manage this decentralized VR program, we interviewed key program officials at Education’s headquarters and each regional office responsible for monitoring the state VR agencies. In addition, we reviewed relevant laws and regulations, Education policy documents relating to the VR program, and the 80 state VR agency’s fiscal year 2003 state plans. We also conducted site visits to the 9 state VR agencies in California, Maryland, Minnesota, New Mexico, Tennessee, and Virginia. We selected these sites to achieve a mix of state VR agency structures (including general, blind, and combined agencies), operations, performance, and geographic diversity. We conducted our review from August 2004 through September 2005 in accordance with generally accepted government auditing standards.

Appendix II: State VR Agency Exit and Employment Rates, Fiscal Year 2003

Appendix III: State VR Agency Population Proportions by Primary Impairment, Fiscal Year 2003

No impairment

Appendix IV: Social Security Beneficiaries’ Employment Rates by State VR Agency, Fiscal Year 2003

Appendix V: State VR Agency Percentages of Total Agency Service Budget Spent on Each Service Category, Fiscal Year 2003

Appendix VI: State VR Agency Total Administrative Costs as a Percentage of Total Expenditures, Fiscal Year 2003

Appendix VII: State VR Agency Average Total Expenditures in Fiscal Year 2003 Per Person Exiting with Employment in Fiscal Year 2003

Appendix VIII: Comments from the Department of Education

Appendix IX: GAO Contacts and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Michele Grgich, Assistant Director; Beverly Crawford; Shannon K. Groff ; and Megan Matselboba made key contributions to this report. Also, Elizabeth H. Curda, Wilfred B. Holloway, Jonathan McMurray, Luann Moy, Peter Rumble, Daniel A. Schwimer, and Susan B. Wallace provided technical assistance.