VA Disability Benefits: Improved Planning Practices Would Better Ensure Successful Appeals Reform
Released: 2018-03-22T00:00:00

Letter

Background

VA’s Disability Compensation Claims Process

VA’s process for deciding veterans’ eligibility for disability compensation begins when a veteran submits a claim to VA. The veteran submits his or her claim to one of VBA’s 56 regional offices, where staff members assist the veteran by gathering additional evidence, such as military and medical records, that is needed to evaluate the claim. Based on this evidence, VBA decides whether the veteran is entitled to compensation and, if so, how much.

A veteran dissatisfied with the initial claim decision can generally appeal within 1 year from the date of the notification letter sent by VBA. Under the current appeals process (now referred to by VA as the legacy process), an appeal begins with the veteran filing a Notice of Disagreement. VBA then re-examines the case and generally issues a Statement of the Case that represents its decision.

A veteran dissatisfied with VBA’s decision can file an appeal with the Board. In filing that appeal, the veteran can indicate whether a Board hearing is desired. Before the Board reviews the appeal, VBA prepares the file and certifies it as ready for Board review. If the veteran requests a hearing to present new evidence or arguments, the Board will hold a hearing by videoconference or at a local VBA regional office. The Board’s members, also known as Veterans Law Judges, review the evidence and either issue a decision to grant or deny the veteran’s appeal or refer (or remand) the appeal back to VBA for further work.

New Appeals Process

The 2017 Act made changes to VA’s legacy appeals process that will generally take effect no earlier than February 2019, which is approximately 18 months from the date of enactment. According to its appeals plan, VA intends to implement the Act by replacing the current appeals process with a process offering veterans who are dissatisfied with VBA’s decision on their claim one of five options: two of those options afford the veteran an opportunity for an additional review of VBA’s decision within VBA, and the other three options afford them the opportunity to bypass additional VBA review and appeal directly to the Board.

Under the new appeals process, the two VBA options will be: 1. Request higher-level review: The veteran asks VBA to review its initial decision based on the same evidence but with a higher-level official reviewing and issuing a new decision. 2. File supplemental claim: The veteran provides additional evidence and files a supplemental claim with VBA for a new decision on the claim.

The three Board options will be: 3. Request Board review of existing record: The veteran appeals to the Board and asks it to review only the existing record without a hearing. 4. Request Board review of additional evidence, without a hearing. 5. Request Board review of additional evidence, with a hearing.

VA’s Appeals Plan

The Act also requires VA to submit to the appropriate committees of Congress and GAO, within 90 days of the date of enactment, a comprehensive plan for (1) processing appeals under the legacy process until there are no more to process, (2) implementing the new appeals process, (3) processing of claims under the new appeals process in a timely manner, and (4) monitoring implementation of the new appeals process. In addition to these four broad elements, the Act lists 18 elements required to be included in the plan that relate to, among other things: staffing, IT, and other resources required to implement the plan; estimated timelines for hiring and training VA employees; and a description of risks associated with each element of the plan.

The Act also includes a provision for GAO to assess the plan within 90 days after VA submits it.

The Act also requires VA to provide progress reports to the appropriate committees of Congress and GAO at least once every 90 days (starting after VA submits its plan), until the date the Act’s legal changes to the appeals process generally go into effect and then at least once every 180 days after this date for 7 years.

Rapid Appeals Modernization Program (RAMP)

The Act also authorized VA to carry out a program to test any assumptions relied upon in developing its comprehensive plan and test the feasibility and advisability of any facet of the new appeals process. In its appeals plan, VA reported its decision to pilot test two of the five new options by allowing veterans with pending appeals in the legacy process (known as legacy appeals) to elect the VBA higher-level review or VBA supplemental claim options beginning in November 2017. This program, which VA refers to as RAMP, is intended to reduce legacy appeals by providing veterans with a chance for early resolution of their claims within VBA while the Board focuses on reducing its inventory of legacy appeals, according to VA. Participation in RAMP is voluntary, but veterans must withdraw their pending legacy appeal to participate, according to VA. Veterans dissatisfied with their RAMP decisions must wait until VA fully implements the new appeals process (in February 2019 at the earliest) before pursuing an appeal with the Board under the new process, according to VA officials.

VA’s Plan Addresses Most of the Act’s Required Elements for the New and Legacy Disability Appeals Processes

VA’s appeals plan addresses 17 of the Act’s 22 required elements, partially addresses 4 related to monitoring implementation and workforce planning, and does not address 1 element related to identifying total resources. For example, VA’s appeals plan addresses the required elements related to, among others, identifying legal authorities for hiring and removing employees, estimating timelines for hiring and training employees, and outlining the outreach VA expects to conduct. For the elements in the Act that VA’s appeals plan partially addresses or does not address, see table 1. For a detailed list of the 22 required elements in the Act, see appendix I.

When we provided VA with our preliminary assessment, VA officials said they disagreed and that their appeals plan addresses all 22 of the required elements. In general, they said that data are not available, and VA cannot yet forecast the information required by the Act until aspects of the new appeals process are tested or implemented. However, in discussing our assessment at a January 2018 House Committee on Veterans’ Affairs hearing, the Deputy Secretary of Veterans Affairs stated that the agency agreed with our assessment and will work with us to address these gaps in VA’s appeals plan.

Until VA’s appeals plan has complete information on all 22 of the required elements, Congress does not have the information it needs to fully conduct oversight of the plan and the agency’s efforts to implement and administer the new process while addressing legacy appeals. VA also needs information on resources, among other areas, to certify that the agency is prepared to carry out timely processing of appeals under the new and legacy appeals process. Further, as discussed below, addressing required elements through a more comprehensive plan and underlying analysis is consistent with sound planning practices and would better position VA to implement the new appeals process while attending to legacy appeals. For example, such an appeals plan would provide for carefully monitoring the new and legacy appeals processes against balanced goals and metrics, and clearly articulates resources, milestones and other information needed for effective program management.

VA’s Appeals Plan Reflects Certain Sound Planning Practices, but Could Improve on Others

VA’s appeals plan reflects certain sound planning practices, such as convening a working group on performance tracking; however, the plan could benefit from including important details related to three key planning areas: 1. articulating a balanced set of goals and related measures to monitor and assess the performance of the new appeals process, in conjunction with the legacy process; 2. developing a high-quality and reliable implementation schedule to manage key steps and activities of the project; and 3. assessing key risks in a comprehensive manner, including respective mitigation strategies, articulating clear criteria and an assessment plan for RAMP, and more fully testing or analyzing all appeals options.

VA’s Appeals Plan Indicates Steps to Assess Process Changes, but Should Also Include Goals and Measures to Provide Full Picture of Success

VA’s appeals plan reflects steps taken to track performance, but it could improve its planning practices related to monitoring and assessing performance on a range of key dimensions of success. Sound planning practices suggest that agencies develop overall goals tied to meaningful and balanced performance measures. These measures include a mix of outcome, output, and efficiency measures to ensure that an organization’s priorities—as well as government-wide priorities such as quality, timeliness, and cost of service—are addressed.

VA’s appeals plan reports that the agency convened a working group to design a process for tracking timeliness of both the legacy appeals and appeals within the new process. In supporting documentation that we requested, VA officials stated they are also determining the best way to measure veterans’ satisfaction with the new appeals process. VA’s appeals plan and supporting documentation also identify timeliness goals for the two VBA-only options and one of the three Board options. Nevertheless, its appeals plan does not articulate a set of goals and measures that cover all aspects of its new appeals process, such as accuracy of decisions and cost. The plan also does not provide details on the metrics the agency will develop, how it will assess if the new appeals process is an improvement over the legacy appeals process, and how it will monitor the allocation of resources between legacy and new appeals claims. More specifically:

VA’s reported timeliness measures are incomplete: VA’s appeals plan outlines timeliness goals for the two VBA options (average processing time of 125 days) and for the Board option that does not include new evidence or a hearing (average processing time of 365 days). However, VA’s plan does not establish timeliness goals for the other two Board options: Board review of additional evidence without a hearing and Board review of additional evidence with a hearing. In commenting on our findings, while VA officials indicated they expect the new process to be more efficient than the legacy process (and, therefore, more timely), they also said data to inform goal setting for all Board options will not be available until VA fully implements these options. Establishing timeliness goals for all options would provide a more complete picture of VA’s vision for the new appeals process, and help VA to develop concrete, objective, and observable performance measures to show progress in achieving that vision, as well as inform resource estimates.

VA’s reported measures lack adequate balance: Other than including certain timeliness goals, VA’s appeals plan does not articulate additional aspects of performance important for managing appeals, such as accuracy of decisions, veteran satisfaction with the process, or cost. We previously reported that VA officials said they wanted to also use veteran survey results, wait times, and inventories as sources of information to measure progress under the new appeals process. Further, VA’s fiscal year 2018 annual performance plan includes an overall customer satisfaction score for veterans’ benefits. However, these and other potential measures of success are not specified in VA’s appeals plan for monitoring the new appeals process as compared with legacy appeals. By not articulating a set of comprehensive and balanced goals and measures in its appeals plan, VA could be inadvertently creating skewed incentives by focusing on one area of program performance to the detriment of other areas (e.g., processing claims quickly but inaccurately).

In commenting on our findings, VA officials recognized the need to develop additional goals and measures. They indicated, for example, that they are developing and testing whether the existing quality assurance goal—92 percent accuracy—is appropriate for the new process. According to VA officials, once they have developed these other goals and measures, VA will communicate this information as part of the required progress reports to the appropriate committees of Congress and GAO. Moreover, at a January 2018 House Committee on Veterans’ Affairs hearing, the Deputy Secretary of Veterans Affairs acknowledged that their performance goals and measures are not yet complete and indicated that the agency will address these gaps in measuring performance.

VA’s plan does not reflect how it will establish baseline data: VA’s approach for evaluating the efficiency and effectiveness of the implementation of the new appeals process falls short of sound practices for using baseline data to assess performance. Our prior work has demonstrated that by developing and tracking a performance baseline for all measures, including those that demonstrate the effectiveness of a program, agencies can better evaluate progress made and whether or not goals are being achieved. However, VA’s appeals plan did not provide important details about what aspects of the new appeals process’ performance will be compared to what aspects of the legacy process’ performance.

In particular, section 5 of the Act lists a number of metrics VA is required to report periodically, including some that could be used as baseline measures. For example, VA is required to periodically publish on its website the average time that elapsed between the filing of an initial claim and the final resolution of the claim, for legacy appeals as well as appeals under the new system, which is consistent with our prior recommendation. However, VA’s appeals plan does not explain how or when the agency would collect and use these or other data about the legacy and new processes’ performance—such as accuracy, veteran satisfaction, and cost—to assess their relative performance.

As we had previously reported, VA’s business case for reform in some instances relied on unproven assumptions and limited analyses of its legacy process to identify root causes of performance problems. Specifically, VA determined that the open-ended nature of its legacy appeals process, whereby a veteran can submit additional evidence numerous times at any point, can cause additional cycles of re- adjudication, a process VA refers to as “churning.” According to VA, this re-adjudication can occur multiple times and can add years to the time needed to reach a final decision on an appeal. Without fully articulating a plan for collecting and using baseline and trend data, VA cannot determine the extent to which the new appeals process, which also allows for multiple appeal opportunities, will achieve final resolution of veterans’ appeals sooner, on average, than the legacy process.

In commenting on our assessment, VA indicated that it is working toward capturing the metrics listed in section 5 of the Act. VA officials also noted that reporting on the new appeals process will require IT system functionality that currently does not exist, but stated that efforts are underway to add this functionality.

VA’s plan does not explain how the agency will monitor processing of legacy versus new appeals: In addition, VA’s appeals plan does not fully articulate how the agency will monitor whether resources are being appropriately devoted to both the new and legacy appeals process and how it will track both sets of workloads. An appeals plan that does not specifically articulate how VA will manage the two processes in parallel exposes the agency to risk that veterans with appeals in the legacy process may experience significant delays or otherwise poor results relative to those in the new appeals process or vice versa. In commenting on our findings, VA officials noted that VA was not required under section 3 of the Act to provide a description of its plans to capture metrics listed in section 5. Even if not required by the Act, developing an approach for carefully monitoring the management of new and legacy appeals would help VA track progress being made and achievement of goals.

Until VA establishes complete and balanced goals and measures, identifies baseline data, and develops a plan for monitoring and assessing both the new and legacy processes, VA runs the risk of promoting skewed behaviors, or not fully understanding whether the new process is an improvement or whether veterans with appeals in the legacy process are experiencing poor results.

VA’s Appeals Plan Needs a Reliable Implementation Schedule to Manage the Project

VA’s appeals plan reflects certain aspects of sound planning practices related to managing the implementation of process change; however, other key components are not addressed. Sound planning practices for implementing process change suggest establishing a transition team. Consistent with such practices, VA’s appeals plan states that the agency convened an agency-wide governance structure to coordinate implementation of its new appeals process; it is comprised of senior-level employees with authority to make necessary decisions to keep the project on track. VA’s appeals plan also includes a copy of a master schedule. In its plan, VA asserts that the master schedule reflects timelines, interim goals and milestones, reporting requirements, and established deadlines, and that it will be used to guide implementation. VA’s appeals plan also reports that VA is consulting with project management professionals, who are using the master schedule, among other tools, to monitor implementation. In addition, VA made progress addressing some of the issues we previously identified by developing steps and timetables for updating training in anticipation of implementing the new appeals process.

However, VA’s master schedule for implementing reform is missing elements of a high-quality and reliable implementation schedule for key activities. We have previously reported that having a well-planned schedule is a fundamental management tool. Generally recognized sound practices from the Project Management Institute (PMI) and GAO call for organizations to employ an integrated and reliable master schedule that defines when work activities will occur, who will complete the work, how long they will take, how they are related to one another, and the constraints affecting the start and completion of work elements, as well as whether resources will be available when they are needed. Such a project management schedule not only provides a road map for systematic project execution, but also provides the means by which to gauge progress, identify and address potential problems, and promote accountability.

The master schedule VA provided in its appeals plan should have included other sound practices for project management related to a reliable schedule. Specifically:

Key activities and their duration are not included: VA’s master schedule does not capture specific Board-related activities, such as efforts to develop metrics, and the schedule and other project plans we reviewed do not go beyond February 2019. For example, the schedule does not indicate the period of time when VA expects to no longer be processing legacy appeals. Also, VA’s master schedule submitted with its November 2017 appeals plan did not include the Rapid Appeals Modernization Program (RAMP) activities, even though this pilot test is occurring at the same time VA is preparing for full implementation of appeals options at VBA and the Board. However, VA’s updated schedule that accompanied its comments on our draft report was updated to include RAMP.

When all key and necessary activities are not included, it raises questions about whether all activities are scheduled in the correct order, resources are properly allocated, or the estimated completion dates are reliable. In addition, if the schedule does not fully and accurately reflect VA’s efforts, it will not serve as an appropriate basis for analysis and may result in unreliable completion dates and delays.

Sequencing and linkages among activities are not identified: For the high-level activities VA’s appeals plan identifies, VA’s master schedule does not indicate whether there were sequencing or linkages among them, which is not consistent with sound scheduling practices. Sequencing and linkages would show, for example, if any of these activities or sub-activities must finish prior to the start of other activities, or the amount of time an activity could be delayed before the delay affects VA’s estimated implementation date. For example, VA cannot train new employees until after it hires them.

The activities VA identifies also do not appear supported by lower- level project schedules. Specifically, when we requested documentation to support VA’s high-level summary of activities and milestones, VA officials did not provide intermediate or more detailed schedules that reflected these practices. In particular, VA’s appeals plan lacks a complete schedule for IT modifications that clearly defines what is to be achieved and the time frames for achievement.

We previously recommended that VA develop a schedule for IT updates that explicitly addresses when and how process reform will be integrated into new systems and when these systems will be ready to support the new appeals process at its onset. For example, VA’s appeals plan references several required IT modifications that do not appear in its master schedule. Schedules that are defined at too high a level may disguise risk that is inherent in lower-level activities.

Interim goals are not reflected: VA officials stated that they have interim goals and milestones, though VA’s appeals plan and supporting documentation generally do not include this information.

Sound planning and redesign practices suggest closely monitoring implementation and developing project goals that include a mix of intermediate goals to be met at various stages. We previously made a recommendation that VA develop a more robust plan for closely monitoring implementation of process reform, including metrics and interim goals to help track progress, evaluate efficiency and effectiveness, and identify trouble spots—all of which are consistent with sound planning practices. Without interim goals and milestones, VA lacks information to support sequencing of activities and to track and ensure accountability for steady progress.

Resources are not assigned to all identified activities: The high- level summary schedule that VA provided us also lacks details regarding the assignment of resources for all activities. Specifically, while the plan identifies workgroups responsible for coordinating elements in the plan, such as regulations, training, and outreach, the schedule does not assign resources to the 40 listed activities. As discussed previously, VA’s appeals plan also does not provide information on the total resources required for this reform effort. Assigning resources to the listed activities, as well as providing other information, could provide a better indication of the estimated total resources required to implement the new appeals process and address legacy appeals.

In commenting on our findings, VA officials stated that the agency is developing lower-level project schedules for key activities—such as RAMP and IT requirements—and will provide these schedules as part of the required progress reports to the appropriate committees of Congress and GAO. VA officials also noted that future updates will include additional dependencies and risks, which VBA and the Board are still developing. Further, in discussing our findings at a January 2018 House Committee on Veterans’ Affairs hearing, the Deputy Secretary of Veterans Affairs reiterated VA’s commitment to developing more robust project plans, particularly for IT. Until VA has a robust integrated master schedule, supported by detailed project plans that adhere to sound practices, VA’s appeals plan does not provide reasonable assurance that decision makers have the essential program management information needed for this complex and important effort.

VA’s Plan Addresses Some but Not All Key Risks Related to the New Appeals Process

VA’s appeals plan includes an assessment of risks involved in implementing the new appeals system, but could more comprehensively reflect key risks posed by such a significant reform effort. VA’s appeals plan and supplemental materials include a “risk register” that describes risks associated with many elements of its plan, and the remaining level of risk after its planned response to these risks. VA’s appeals plan also states that senior leaders will receive regular updates of risks and mitigation strategies. However, because VA has not yet articulated a balanced set of performance goals and measures in its appeals plan, it is hindered in its ability to identify and assess risks.

Federal internal control standards state, and our previous work at VA and other agencies demonstrates, that establishing clear performance goals and objectives is a necessary pre-condition to effectively assessing risk. Having, for example, more complete timeliness goals, and goals and measures reflecting other areas of performance, would allow VA to better identify and target risks associated with managing two processes in parallel, including the potential that veterans with appeals in the legacy process may experience significant delays relative to those in the new appeals process.

Importantly, VA is missing an opportunity to fully benefit from RAMP by not testing and assessing other aspects of the new appeals process. The Act authorizes VA to test the feasibility and advisability of any facet of the new appeals process, and VA is taking a positive step to mitigate some risks by testing the two review options available within VBA (review of a claim by a higher-level official based on the same evidence and review of a supplemental claim with additional evidence) through RAMP. In November 2017, VA began RAMP by inviting 500 veterans whose appeals have been pending the longest to participate. According to VA officials, VA plans to continue offering RAMP to additional eligible veterans with pending legacy appeals each month until January 2019—a month before VA anticipates fully implementing the new appeals system. However, as designed, RAMP does not include features—consistent with a well-developed and documented pilot test program—that would provide VA with an opportunity to evaluate fully the soundness of new processes and practices on a smaller scale. Specifically:

VA’s plan does not clearly define success criteria for RAMP: VA’s appeals plan states that the agency will collect certain data from RAMP, such as the rate at which eligible veterans opt into the process, timeliness of claims processing, and individual employee productivity. VA also established an overall average processing time goal of 125 days for the two VBA options; however, the plan and supporting documentation did not clearly articulate whether RAMP reviews are expected to meet this timeliness goal. The plan also did not identify other success criteria for RAMP or the types of results expected before fully implementing the new appeals process. For example, VA’s plan does not articulate the expected number and type of subsequent appeals to the Board that result from RAMP.

In commenting on our findings, VA noted that its intent in implementing RAMP was to collect data and test aspects of the new process, and that RAMP was not an initiative in and of itself. However, developing performance measures and data gathering procedures and defining success criteria for a pilot test before proceeding to full implementation are sound practices for process redesign and pilot testing. In addition, because RAMP was not included in VA’s risk assessment, we asked VA if it had identified any risks or mitigation strategies specific to RAMP. In its supplemental materials, VA stated that the greatest risk to RAMP is a low participation rate among eligible veterans with legacy claims. VA also indicated that it would need 10 percent of eligible veterans to opt into RAMP to yield meaningful results. However, this threshold is not articulated in VA’s appeals plan as an explicit success criterion or objective. According to data provided by VA, as of January 22, 2018, 238 veterans opted in.

Of veterans with pending claims in RAMP, two-thirds chose the higher-level review option. VA also reported that 47 RAMP decisions have been made so far. As of yet, no appeals of RAMP decisions have been filed.

VA’s plan does not articulate how it will assess RAMP before proceeding with full implementation: Although VA’s appeals plan describes a “close-out” phase in which VA intends to assess the results of RAMP, it does not detail the conditions that would have to be met (or not met) to trigger changes. For example, VA’s plan does not explain when or how it might respond to low opt-in rates for RAMP—other than stating it will increase outreach to eligible veterans—or to unexpectedly high appeal rates to the Board resulting from RAMP decisions. Sound redesign and change management practices both suggest that pilot tests be rigorously monitored and evaluated, and that further roll-out occur only after an agency’s transition team takes any needed corrective action and determines that the new process is achieving previously identified success criteria. Without fully articulating its plan for deciding how and when to roll out changes more broadly, it is not clear whether VA would be prepared to fully implement a new appeals process that achieves its aim of better serving veterans.

RAMP does not test all aspects of the new appeals process: RAMP provides an opportunity to learn about experiences at VBA under the new system, such as the rate at which eligible veterans choose those options and the resources that will be required to process their appeals. However, RAMP was not designed to test how many veterans would choose to appeal directly to the Board and, therefore, it will not provide comparable information on the Board appeals options. Sound workforce planning practices suggest that agencies identify the total resources needed to manage the risk of implementing new processes and conduct scenario planning to determine those needs.

In addition, although we previously recommended VA conduct additional sensitivity analyses to inform projections of future appeals inventories, VA’s appeals plan does not reflect VA’s use or intended use of sensitivity analyses when projecting staffing needs for new appeals options at the Board. In commenting on our findings, VA officials said they do not plan to conduct additional sensitivity analyses to project future workloads until they have more information from RAMP to inform their assumptions. As a result, VA will lack data on scenarios in which veterans may overwhelmingly choose options available at the Board over those at VBA when the appeals plan is fully implemented. This presents a risk that VA’s early production projections and initial resource allocations may not be properly balanced between the Board and VBA. This, in turn, may result in an unexpectedly large number of appeals pending with the Board, and corresponding lengthy average wait and decision times for some, if not all, Board options.

Having information on the number of veterans who are likely to appeal to the Board is particularly critical, given that similar efforts to create additional review options at VBA did not achieve their goals of reducing the percentage of appeals that continue on to the Board. In 2001, VA established the Decision Review Officer (DRO) process—in which senior staff have the authority to overturn an initial disability claim decision without any new evidence—to resolve more appeals at the regional level and avoid long waits at the Board. However, we reported in 2011 that, although the DRO process helped some veterans get additional benefits at the regional office level, it did not accomplish the program’s primary goal of reducing the percentage of appeals continuing on to the Board.

In responding to our findings, VA officials reiterated their plans to increase outreach in the event of low opt-in rates for RAMP and indicated they recently began to send follow-up RAMP invitation letters. With respect to assessing all appeals options, VA officials stated that, while no legal bar prevents testing of the Board options, the Board is focused on reducing its inventory of pending appeals while RAMP provides early resolution of appeals within the new VBA-only options. Officials conceded that this approach means they cannot collect data on the rate at which veterans opt to appeal directly to the Board (e.g., bypassing additional VBA review) until the new process is fully implemented. However, they noted that they can collect some data on the rate at which veterans whose appeals go through RAMP file subsequent appeals to the Board, even though the Board will not begin processing those appeals until full implementation. At a January 2018 House Committee on Veterans’ Affairs hearing, the Deputy Secretary of Veterans Affairs stated that given our assessment, VA will adjust its approach to identify and mitigate risks associated with implementing all of the appeals options, including those at the Board.

Until VA pursues an approach that identifies and mitigates significant risks associated with implementing a new process, VA is taking a chance that untested aspects will not perform as desired. The Act provides VA authority to pilot aspects of the process and flexibility on the timing of implementing the new process, which could allow some additional time for VA to carefully measure performance under RAMP and determine whether any corrective actions are necessary. If VA does not take full advantage of this authority, it risks moving forward without knowing whether the new appeals process improves experiences for veterans, and potentially implementing a process that is more expensive or results in longer wait times than originally anticipated.

Conclusions

In implementing appeals reform after the enactment of the Veterans Appeals Improvement and Modernization Act of 2017, VA is undertaking a complex endeavor that has the potential to affect the lives of hundreds of thousands of veterans with service-connected disabilities. Such an endeavor demands a commensurate level of planning to be successful. While the Act required VA to submit its plan within 90 days of enactment, VA had proposed and began to plan for appeals reform much earlier, and had our March 2017 recommendations to guide its planning efforts from a foundation of sound practices.

VA’s November 2017 appeals plan is a positive step forward. Certain elements of the plan—such as establishing an agency-wide governance structure to oversee implementation and testing aspects of reform prior to full implementation—are notable gains since our March 2017 report. At the same time, the plan partially addresses or does not address five of the required elements called for by the Act, such as delineating the total resources required by VBA and the Board to implement and administer the new appeals process and address legacy appeals. The plan also is not fully responsive to our past recommendations and does not reflect a number of sound planning practices that are essential for gauging progress, establishing accountability, and linking resources to results.

One such key practice is articulating a desired “end state”—a vision for what successful implementation would look like for the new appeals process as well as the wind-down of the legacy process, such as accurate and timely processing of appeals while ensuring veteran satisfaction. Without establishing a complete and balanced set of goals and related performance measures to achieve this end state, and monitoring and assessing progress along the way, VA risks falling short of its overarching objective—to improve timeliness of appeals decisions for veterans overall. By not fully articulating how it plans to monitor workloads and devote resources to both the new and legacy processes, VA runs the risk of disadvantaging veterans with legacy appeals relative to those in the new process, or vice versa.

Just as important is establishing a robust integrated master schedule— rather than a high-level timeline—that is built upon and clearly reflects extensive detailed planning and includes all activities necessary to execute the program and interdependencies between these activities. Without such a road map, VA’s appeals plan does not provide reasonable assurance that decision makers have the essential information needed to manage this complex and important program.

We are encouraged that VA has taken some steps toward assessing risks, including establishing a risk register and implementing RAMP to collect information on the two VBA appeals options. However, unless VA assesses risks against a balanced set of goals and measures, VA may not be fully aware of risks that may impede successful implementation of appeals reform. Further, although VA will undoubtedly learn from the RAMP experience, it may not learn all that it should from its efforts without (1) establishing clear criteria for what success looks like (or the circumstances that would cause VA to consider making course corrections), and (2) building in time to take stock of the lessons learned before moving to full implementation.

VA’s plan places a lot of weight on RAMP to, among other efforts, mitigate risk and generate estimates of the resources needed for successful implementation after fiscal year 2018, even though RAMP does not fully test options for appealing to the Board that will be available to veterans after full implementation. Unless VA addresses key risks associated with fully implementing appeals reform—by either testing or conducting sensitivity analyses for all five appeals options, to better understand potential workloads at the Board—VA runs the risk of fully implementing the process without knowing if it is improving the process for veterans.

Recommendations for Executive Action

We are making the following four recommendations to VA: The Secretary of Veterans Affairs should address all of the required elements in the Act in VA’s appeals plan to Congress—including delineating resources required for all VBA and Board appeals options— using sensitivity analyses and RAMP results, where appropriate and needed. (Recommendation 1)

The Secretary of Veterans Affairs should clearly articulate in VA’s appeals plan how VA will monitor and assess the new appeals process compared to the legacy process, including specifying a balanced set of goals and measures—such as timeliness goals for all VBA appeals options and Board dockets, and measures of accuracy, veteran satisfaction, and cost—and related baseline data. (Recommendation 2)

The Secretary of Veterans Affairs should augment the master schedule for VA’s appeals plan to reflect all activities—such as modifications to IT systems—as well as assigned responsibilities, interdependencies, start and end dates for key activities for each workgroup, and resources, to establish accountability and reduce overall risk of implementation failures. (Recommendation 3)

The Secretary of Veterans Affairs should ensure that the appeals plan more fully addresses risk associated with appeals reform—for example, by assessing risks against a balanced set of goals and measures, articulating success criteria and an assessment plan for RAMP, and testing or conducting sensitivity analyses of all appeals options—prior to fully implementing the new appeals process. (Recommendation 4)

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Veterans Affairs (VA) for comment. In its comments, reproduced in appendix II, VA outlined its planned actions to address the recommendations. VA clarified in a subsequent communication that the agency agreed with all our recommendations. VA did not provide technical comments.

With respect to our first recommendation that VA address all the required elements in the Act in VA’s appeals plan (including delineating total resources), VA stated that both VBA and the Board will use existing resources to implement the new appeals process. VA also stated it plans to take additional steps to determine resource requirements for addressing workloads under both the legacy and new appeals process. For VBA, VA stated that it will continue to rely on RAMP to project resource requirements, while acknowledging the need to augment its analysis of RAMP data by adopting additional strategies to project resource requirements. VA did not describe these strategies, but stated that it will share them with Congress and GAO in the near future. Meanwhile, VA noted that its 2019 budget request includes 605 additional FTEs for VBA to process appeals, but did not indicate how it developed this budget request. For the Board, VA stated that it plans to develop better predictions regarding resource allocations among dockets by leveraging project management and other support within the agency. We will continue to monitor VA’s efforts to delineate needed resources, including how it uses the results of pilot tests and prediction analysis.

With respect to our second recommendation to articulate how it will monitor and assess the new appeals process compared to the legacy process, VA stated that it is working to develop a complete and balanced set of measures for the new appeals process, and timeliness goals for all appeals options. Further, VA indicated it will track performance for Board appeal options using an existing process. We are encouraged by VA’s proposed actions, which will provide a more complete picture of VA’s vision for the new appeals process. However, VA does not detail whether or how the agency will develop a baseline or compare performance of the new appeals process to the legacy process. Until VA develops a baseline and a plan for monitoring and assessing both the new and legacy processes—using a complete and balanced set of goals and measures— VA risks not fully understanding whether the new process is an improvement.

With respect to our third recommendation to augment its master implementation schedule to manage the project, VA provided an updated schedule with additional key activities and responsibilities, such as RAMP. Moreover, VA restated its plans to use an agency-wide governance structure to coordinate and track implementation of its new appeals process. We are encouraged by VA’s efforts to develop a more detailed implementation schedule. However, the updated schedule VA provided does not include major activities, such as integrated IT system testing, and completion dates for major activities, for example, adding functionality to VA’s primary claims processing system. In addition, VA provided an updated calendar for six major IT activities through the end of calendar year 2018. However, we continue to believe that VA will need to develop a longer term schedule that projects when processes will be integrated into new systems and when new systems will be ready to support the new appeals process. The schedule also does not indicate whether activities are interrelated, such that a delay in one activity could affect other activities and thereby affect VA’s estimated implementation date. This sound planning practice is especially important because VA stated the agency is concurrently executing many of the activities.

With respect of our fourth recommendation to more fully assess risks associated with appeals reform prior to its full implementation, VA stated that it will assess risks against a balanced set of goals it plans to select. Moreover, using its existing risk management process, VA stated it has identified additional risks and mitigation strategies after submitting its November 2017 plan. For example, VA states that it is addressing the continued low opt-in rate for RAMP, which is testing the new VBA-only options. VA is also acknowledging that delays in the development of IT required to implement the appeals process may prevent the agency from certifying readiness in January 2019. Importantly, VA states that the Board is exploring the development of a pilot program to identify needs and concerns related to full implementation—including all Board appeals options—and to make predictions about timeliness and productivity under the new appeals process. However, VA did not define success criteria for its current pilot test, RAMP, or clearly articulate how the agency will assess results of either RAMP or a new test of Board appeals options before proceeding to full implementation. Implementing our recommendation in a complete and timely manner is important because it would improve VA’s ability to identify and mitigate significant risks associated with implementing a new process.

We will continue to monitor the status of VA’s actions to address our recommendations and how they are implemented, to help ensure that VA is undertaking a level of planning appropriate to implementing a complex endeavor, and thereby improving VA’s chance of success.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, and other interested parties. In addition, the report is available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact Elizabeth Curda at (202) 512-7215 or curdae@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who contributed to this report are listed in appendix III.

Appendix I: Our Assessment of VA’s Appeals Plan Against Required Elements in the Act

To assess the extent to which VA’s appeals plan addresses the required elements in the Veterans Appeals Improvement and Modernization Act of 2017 (Act), we first identified and developed a checklist reflecting each required element for VA’s appeals plan (including sub-parts) under section 3(a) and (b) of the Act. To compare the required elements and their sub-parts against VA’s appeals plan and supplemental materials provided by VA, we developed decision rules for determining whether VA’s appeals plan addressed, partially addressed, or did not address each required element (see table 2). Specifically, we concluded that VA’s plan addressed (or partially addressed) a required element if the plan included information related to all (or some) subparts of the requirement. We focused on the plan as presented, rather than auditing the information VA relied on in developing the plan. For example, the Act’s section 3(b)(10) required VA’s plan to include a description of the modifications to the information technology (IT) systems required to carry out the new appeals system, including cost estimates and a timeline. We concluded that VA’s plan addressed all sub-parts of this element because it provided a description of required IT modifications, a reference to costs included in the Appeals Modernization IT budget, and a timeline. However, our determination that VA addressed this element should not be construed to necessarily mean that VA fully identified or described all IT requirements, or provided complete estimated costs and timelines associated with those requirements, or that the information in VA’s appeals plan comported with sound planning practices. This type of assessment was outside the scope of this objective.

Appendix II: Comments from the Department of Veterans Affairs

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Elizabeth H. Curda, (202) 512-7215 or curdae@gao.gov.

Staff Acknowledgments

In addition to the contact above, the following staff members made significant contributions to this report: Michele Grgich (Assistant Director), James Whitcomb (Analyst in Charge), and Rachael Chamberlin.

In addition, key support was provided by Susan Aschoff, Mark Bird, David Chrisinger, Daniel Concepcion, Clifton Douglas, Alex Galuten, Nisha Hazra, Melissa Jaynes, Benjamin Licht, Patricia McClure, Sheila McCoy, Lorin Obler, Gloria Proa, Almeta Spencer, James Sweetman, Walter Vance, and Greg Whitney.