VA Disability Benefits: Additional Planning Would Enhance Efforts to Improve the Timeliness of Appeals Decisions
Released: 2017-03-23T00:00:00

Letter

Background

VA Disability Compensation Benefit Claims Process

VA’s process for deciding veterans’ eligibility for disability compensation begins when a veteran submits a claim to VA. The claim is reviewed at one of VBA’s 56 regional offices where staff members assist the veteran by gathering any additional evidence, such as military and medical records, needed to evaluate the claim. Based on this evidence, and the results of any necessary medical examinations, VBA decides whether the veteran is entitled to compensation and, if so, how much. VBA assigns a rating of 0 to 100 percent disability in increments of 10 percentage points depending on the severity of the disability. This rating percentage then determines the monthly payment amount the veteran will receive. According to VA data, in many cases (74 percent), the veteran submitting a claim either is already a beneficiary but is seeking increased compensation, or the veteran was denied benefits previously and is claiming them again. In fiscal year 2015, VBA decided 1.4 million compensation claims and had an inventory of 363,000 claims at the end of the fiscal year. As previously noted, in fiscal year 2015, VA paid about $63.7 billion in disability compensation to about 4.1 million veterans.

VA Appeals Process

A veteran dissatisfied with VBA’s initial claim decision can generally appeal within one year from the date of VBA’s notification letter to the veteran. According to the Board, veterans appeal most often because they believe VBA: (1) incorrectly denied them compensation for service- connected disabilities, or (2) under-rated their service-connected disabilities. An appeal begins with the veteran filing a Notice of Disagreement (NOD). VBA then re-examines the case and generally issues a Statement of the Case (SOC) that represents its decision.

A veteran who is or remains dissatisfied with VBA’s decision can file an appeal with the Board. In filing that appeal, the veteran could indicate whether they would like a Board hearing. VBA prepares the claim file for Board review and certifies it as ready for review. If the veteran requests a hearing so they can present new evidence or arguments, the Board will generally hold a hearing either by video conference or at a local VBA regional office. The Board reports to the Office of the Secretary of Veterans Affairs, and is independent of VBA. The Board’s members, also known as Veterans Law Judges (VLJ), decide appeals and are supported by attorneys and administrative staff. After the appeal is docketed at the Board, a VLJ or panel of VLJs reviews the evidence and either (1) grants the claimed benefit, (2) denies the benefit, or (3) returns (remands) the claim to VBA for additional work on one or more issues pertinent to the claim and a new decision. According to VA, the Board remands an appeal to VBA in cases where consideration of new evidence, clarification of evidence, correction of procedural defect, or any other action it deems is essential to achieve a proper decision. If the veteran is unsatisfied with the Board’s final decision, the veteran can continue an appeal beyond VA to federal court. Such an appeal begins with the U.S. Court of Appeals for Veterans Claims, then may go to the U.S. Court of Appeals for the Federal Circuit, and finally to the U.S. Supreme Court. See figure 1 for a representation of the appeals process for VA disability compensation benefit decisions.

Inventories and Timeliness of VA Appeals

According to VA officials, the number of appeals filed has increased steadily as has the length of time needed for the agency to make a final decision. At the end of fiscal year 2015, according to VA data, VA had over 427,000 pending appeals, approximately 81,000 of which were at the Board. While the percentage of pending appeals awaiting decisions from the Board was less than a quarter of all pending appeals, the fiscal year 2015 inventory was almost double the 41,000 pending at the end of fiscal year 2011. About 20 percent of this growth occurred from fiscal year 2014 through 2015.

According to Board data, timeliness has worsened since fiscal year 2011 as well. From fiscal years 2011 through 2015, the average amount of time needed for the Board to make a final decision once the appeal is docketed increased from 240 to 270 days. In addition, the proportion of cases taking the longest to resolve (from when the Board receives the certified appeal to them making a final decision)—over 600 days— increased from 10 percent in fiscal year 2011 to 14 percent in fiscal year 2015 (see fig. 2). Given that the median time for the Board to decide an appeal was 145 days in fiscal year 2015 (compared to an average of 270 days), these data suggest that a relatively small number of appeals is driving up the Board’s reported average processing times. To illustrate, VA officials noted one case where a veteran appealed 27 times over the course of 25 years before the original appeal was concluded.

VA Has Proposed Increased Staffing, Process Reform, and Updated IT to Improve Appeals Timeliness, and Has Taken Action for All Three Approaches

VA has identified three broad approaches for addressing factors that it identified as having contributed to increased appeal inventories and reduced timeliness of appeals decisions, and has already taken action on all three fronts. Citing staffing levels that have not kept pace with workloads, VA secured additional Board staff for fiscal year 2017, and analyzed options for another hiring surge in fiscal 2018. Concerned that its appeals process contributes to delays in appeals decisions—because new evidence may be submitted at any juncture and because VA may be continually required to develop or obtain additional evidence—VA developed a legislative proposal for streamlining its appeals process, including new appeals options designed to accelerate decision-making. Finally, VA has put forth plans to modernize its current, outdated, and inefficient computer system.

VA Proposed Hiring More Staff to Address Increasing Workloads

VA has proposed increasing staff at the Board, as well as VBA, to manage its increasing inventory of appeals and to address related declines in the timeliness of appeals resolutions. VA officials stated that there is a direct and proportional correlation between the number of employees and the number of final appeals decisions, and that Board workloads especially have increased faster than the number of employees staffed to the Board. Specifically, officials have concluded that staff resources within the Board have not been sufficient to adjudicate the increasing number of appeals, ultimately lengthening appeals resolution times. According to VA, in fiscal year 2015 increases in staff (VLJs, attorneys, and support staff), as represented by full-time equivalents (FTEs), allowed the Board to make the highest number of decisions in nearly 30 years. However, despite Board staff increasing by 21 percent from fiscal years 2011 through 2015, officials said that this increase was not sufficient to address the growing inventory of pending appeals, which doubled during the same time period (see fig. 3).

Although the increase in Board staff brought about a record number of appeals decisions in fiscal year 2015, according to VA data we reviewed, each appeal took an average of about 3 months (97 days) longer to reach a final decision than in fiscal year 2012. Similarly, in fiscal year 2015 one Board FTE produced an average of 86 appeals decisions, down from 91 completed per FTE in fiscal year 2011. Growing workloads and the increased complexity of cases, according to Board officials, have contributed to these longer appeal resolution times. More specifically, officials said that claims have become more complicated due to not only the number and complexity of injuries and illnesses, but also to advances in medicine that have improved survival rates from catastrophic injuries, experienced by today’s veterans. VA officials estimated that if the number of FTEs and number of appeals decided per FTE stays steady or decreases, appeals resolution times will continue to lengthen. Specifically, as of October 2016, VA projected that if nothing else changes, and if the number of FTEs hold steady at the fiscal year 2017 number (922 FTEs for the Board and 1,495 for VBA), the inventory of appeals could exceed 1 million in fiscal year 2026, which would mean that veterans would wait an average of 8.5 years for a final appeals decision.

In light of this assessment, VA concluded that increasing the number of FTEs at the Board is a key step in mitigating the current and future pending inventory of appeals and ultimately improving appeals decision timeliness. In 2016, VA set a goal to decide the vast majority (90 percent) of appeals (including both those reviewed by VBA and the Board) within 1 year by 2021. As an initial step toward this goal, VA requested and received a funding amount that the agency asserted would allow it to fund an additional 242 FTEs for the Board in fiscal year 2017 (a 36 percent increase over the 680 FTEs funded in fiscal year 2016) for a total of 922 FTEs. VA also concluded, however, that this increase in staff will not be enough to reduce its appeals workload and decrease appeals processing time. Therefore, VA estimated the need for a subsequent hiring surge of up to 1,458 FTEs beginning in fiscal year 2018 to reduce the current pending appeals inventory.

To understand the need for and implications of a future hiring surge, VA modeled different staffing scenarios. Initially, VA compared how increasing staff in combination with and without proposed changes to the appeals process would achieve inventory reductions, and at what cost. VA determined that by combining staff increases with a new process, it could clear pending appeals faster and at a lower cost than if it hired additional staff under the current process. In response to congressional inquiries, in September 2016 VA also modeled the cost and impact on appeals inventories of four surge options beginning in fiscal year 2018 (in addition to planned hiring in fiscal year 2017). VA estimated, for example, that projected pending appeals in fiscal year 2017 (535,726) would be cleared in 10 years under option 2, compared to a 60 percent reduction over the same time period if there were no hiring surge. See table 1 for a comparison of the four options.

VA Proposed Reforming Current Appeals Process to Address Factors Contributing to Delays in Resolving Appeals

VA has proposed changes to its appeals process to address causes of delays in resolving appeals. The key challenge VA identified was the open-ended nature of its disability appeals process, whereby a veteran can submit additional evidence numerous times at any point during the VA appeals process, which can cause another cycle of re-adjudication. Specifically, when a veteran submits additional pertinent evidence after VA’s initial decision on their claim, VA is generally required to review the evidence, develop any other needed evidence, and issue another decision. This is the case regardless of whether the veteran submits the additional evidence to VBA or to the Board and, for appeals pending before the Board, the submission of additional evidence may result in a remand to VBA for further development. VA reported that in fiscal year 2015, the Board remanded about 46 percent of appeals to VBA for additional development. Of those remanded appeals, which may involve more than one issue, VA reported that about 60 percent of the reasons those appeals were returned to VBA were due to the open record that allows veterans to introduce new evidence at any point during the appeal. VA reported that in fiscal year 2015, it took VBA an additional 255 days on average to complete remand development and for the appeal to be re-docketed at the Board. VA also reported that in fiscal year 2015 it took the Board an average of 244 additional days to complete its subsequent review of the returned remands and decide the appeal. According to VA, this re-adjudication can occur multiple times and can add years to the time needed to reach a final decision on an appeal.

Board and VSO officials also identified factors within VBA’s initial claim process—and outside of the Board’s control—that cause delays in veterans receiving final decisions on their appeal. Specifically,

According to Board and VSO officials, VBA’s decision notification letters are unclear and confusing. In particular, the officials stated that these letters do not adequately explain why claims were denied and do not clearly identify the evidence a veteran needs to provide to fully support a claim on appeal. As a result, some veterans may appeal unnecessarily, or they may appeal without providing the evidence needed to support their claims.

VSO officials we interviewed said that some delays are attributable to errors in VBA’s initial decisions. They suggested that errors may have occurred because VBA rushed some decisions in its initiative to reduce its backlog of claims pending more than 125 days. Such errors can lead to Board remands and VBA re-work. For example, the Board may remand an appeal because VBA failed to meet the “duty to assist” responsibilities to a veteran. According to the Board, 41 percent of the reasons for the remands in fiscal year 2015 were due to a VBA error.

Board and VSO officials also cited delays in VBA’s transmittal of appeals to the Board as a possible cause for the delays in Board decisions. When a veteran files an application with VBA to appeal to the Board, VBA prepares the case file for transfer to the Board, certifies that the case file is complete and ready for Board review, and transmits the file to the Board. According to VA data on appeals decisions made by the Board in fiscal year 2015, it took on average 537 days to process the appeal from receipt to certification. Docketing appeals that had been certified to the Board added an average of 222 days to processing times for appeals decisions made in fiscal year 2015. VSOs (two of the four we interviewed) told us that they noticed these delays occurring as VA’s focus shifted to clearing the compensation benefit claims backlog.

To address process-related challenges, VA’s approach has been to develop a proposal to streamline the appeals process and to ask the Congress to make changes in the laws governing the process. In April 2016, VA issued a draft summary of a proposed streamlined appeals process that reflected collaboration with its stakeholders. This summary was accompanied by draft legislation for the Congress’ consideration. If enacted into law, the draft legislation would make process changes that VA identified as needed to streamline the appeals process. According to VA, key to the proposed process changes would be replacing the current appeals process, which begins in VBA, with a process giving a veteran four options—two in VBA and two in the Board. As presented in VA’s framework, these options would be:

Ask VBA to review its initial decision based on the same evidence.

Under this option, the veteran would not be able to submit new evidence or request a VBA hearing, and would not be subject to VA’s “duty to assist” requirement. A VBA official (at a level higher than the official who made the initial decision) would review the record supporting the initial decision, and issue a new decision.

The veteran could file a “supplemental claim” with VBA, asking VBA to review its initial decision, while providing additional evidence. Under this option, the veteran could also request a VBA hearing. Another VBA official (at the same level as the original VBA decision-maker) would review the revised record, including the additional evidence from the veteran, and issue a new decision.

The veteran could file a Notice of Disagreement directly with the Board, bypassing a VBA review. The veteran would have two options with the Board:

Ask the Board to review only the existing record without a hearing and then issue a decision.

Alternatively, ask the Board to review additional evidence, conduct a hearing before issuing its decision, or both.

See figure 4 for a representation of the options in VA’s proposed simplified appeals process.

VA officials anticipate that the proposed appeals process described above will expedite appeals in a number of ways, most notably:

For those appeals where no additional evidence is submitted, and no formal hearing is conducted (indicated as “VBA conducts local higher- level review” and “Board reviews record without new evidence or a hearing” in figure 4), the re-review of the original record could expedite a final appeals decision. In addition, VA’s “duty to assist” requirements would only apply to VBA for initial and supplemental claims. Unlike the current process, in which the Board may remand appeals to VBA to consider new evidence, the Board would only remand appeals under the new process in cases in which the Board found that VBA failed, in its initial or supplemental claim processing, to meet VA’s “duty to assist” the veteran. VA estimates that, once the new process is fully implemented, remands will steadily decrease and eventually occur in as few as 5 percent of appeals.

When the veteran appeals directly to the Board, VBA would no longer be required to review the record (including any additional evidence), prepare statements of its findings (i.e., prepare SOCs or SSOCs), and certify appeals as ready for Board review.

VA has estimated that as a result of these process changes—in combination with increased FTEs—the Board could complete cases faster, deciding many more appeals per FTE in fiscal year 2018 compared to fiscal year 2015. More specifically, VA estimated that the Board could complete an average of 180 appeals decisions per FTE without a hearing and 130 with a hearing, compared to the average of 86 total decisions per FTE in fiscal year 2015. We discuss VA’s estimates in more detail later in this report.

While VA’s proposal reflects VA’s intent to expedite appeals resolutions, it also contains various protections for veterans that are intended to address stakeholders’ concerns about fairness. Notably, such protections include the following: In contrast to the “one size fits all” process, the proposed reform allows the veteran to choose an option that best fits the circumstances of a veteran’s claim. As shown in figure 2 above, a veteran could choose to have VBA review the initial decision or they could appeal directly to the Board. Also, the veteran would have the option to have either VBA or the Board review the existing record, without having to submit new evidence and/or request a formal hearing. VA expects that these options could help the veteran obtain a faster decision from VBA or the Board. Per VA’s framework, under the new process, the veteran would have up to 1 year from VBA’s initial decision to choose an option. Further, if the veteran is unsuccessful in one appeal option, the veteran could, within 1 year, choose another option. However, according to VA, an appeal for a higher-level review by VBA without new evidence cannot directly follow a Board decision.

VBA would be required to provide more information in letters notifying veterans of decisions involving a denial of benefits, which could help veterans make more informed decisions on whether to appeal, which option to pursue, and what additional evidence (if any) to provide. The inclusion of additions to such notifications in VA’s proposed legislation addresses stakeholders’ concerns that veterans did not have enough information to decide whether they should appeal, or what additional evidence they needed to provide, thereby resulting in unnecessary appeals or delays in appeals.

A veteran who is not fully satisfied with the result of any lane would have 1 year to seek further review, while preserving an effective date for benefits based on the date the veteran filed the original claim with VBA. This would help ensure a veteran is not penalized for pursuing an appeal to the Board. For example, under VA’s proposal, a veteran denied benefits by the Board could choose to have VBA conduct another review, by filing a supplemental claim with additional evidence. In contrast, under current law, if a veteran appeals to the Board and is denied (and does not appeal to a federal court), the veteran must generally reopen the claim, or start over, by filing another claim with VBA. If the veteran is subsequently granted benefits, the benefits would generally be awarded from the date on which the new (not original) claim was filed, which could result in the veteran not receiving retroactive compensation payments.

VA Has Plans to Modernize Its Outdated Appeals IT System

VA has plans to modernize its current IT system, which it determined is antiquated and a source of delays in processing appeals. VA currently uses the Veterans Appeals Control and Locator System (VACOLS) to track and manage its appeals workload. VA identified a number of reasons why it believes VACOLS should be replaced, including that:

The system is based on outdated technology dating from the 1990s that VA determined would be difficult to modify to meet the changing needs of VA.

VA designed VACOLS around a paper-based VA claims process and as a result, VACOLS does not adequately support a fully electronic environment. According to VA, although VACOLS has been patched to some extent to handle paperless appeals, the Board relies on paper briefs to help manage its appeals workflow.

VACOLS’s lack of automation, integration with other VA systems, and error checks results in mistakes and lost productivity. According to VA, individual employees spend a significant amount of time correcting data entry errors that would be avoided if cases were automatically transferred to the Board. For instance, they said that after cases are transferred to the Board, a team of employees must manually review and correct most incoming cases due to issues with labeling, mismatched dates, and missing files. Via an internal study, VA determined that up to 88 percent of cases transferred to the Board had such errors. Additionally, VA notes that data entry errors can result in instances where paperless cases are mislabeled as paper- based. These cases will not show up as certified in VACOLS and the Board will erroneously wait for a paper case that will never arrive.

VACOLS is central to appeals processing, thus a system outage would halt the processing of appeals across VA, either paper or electronic, until VACOLS is repaired, according to VA.

VA expects its VACOLS replacement to improve the efficiency of its appeals decisions. Its planned replacement—called Caseflow—is intended to address the limitations of VACOLS and better support processing appeals in a paperless environment. According to VA, Caseflow is being developed in an agile process in which new functions are added to the system as they are completed. In fiscal year 2016, VA developed two initial deliverables. According to VA, the first is intended to automate and introduce consistency to the process of transferring appeals to the Board. The second introduced the ability for staff to access documentation from the Veterans Benefits Management System (VBMS)—VA’s system for processing claims—which VA believes will eventually allow users to review appeals more efficiently. As of February 2017, VA officials also noted the agency is in the process of developing additional components, including document review software for VLJs and attorneys, and a component to better track appeals that are remanded to VBA. According to officials, VA’s longer term plans include a broad roadmap for continuously adding improvements to Caseflow. For instance, VA has plans to build into the system the capability to generate performance metrics, using a component called Caseflow Dashboard. VA states that the dashboard will be able to draw on various VA data systems and provide information on bottlenecks in the appeals process, quantify improvements in the appeals process—including those attributable to improved IT systems—and track the reasons for and number of remands. While Caseflow improvements are being made, VA reported it plans to maintain VACOLS as a redundant resource until the new system is fully complete, at which point VACOLS will be retired.

VA Applied Several Sound Planning Practices in Its Efforts to Improve Its Appeals Processing Timeliness, but Fell Short in Other Areas

VA Weighed Several Factors When Determining Its Need for More Staff, but Did Not Fully Assess Risks or Document Its Workforce Plans

VA acted consistently with sound planning practices in determining its need for additional staff, but it did not fully consider risks and uncertainties in its approach. Sound practices for effective planning suggest that agencies should consider alternative solutions to a problem; assess the risk of unintended consequences; and use data to analyze the problem, including unknowns. Consistent with these concepts and more specific sound workforce planning practices, VA considered various hiring options, such as hiring staff under the current versus VA proposed process, and modeling appeals inventories under four hiring surge options. VA considered a number of factors when comparing the four hiring options including historical data on the volume and complexity of appeals, estimates of future growth in appeals, and the productivity of employees in estimating the number of Board staff needed to meet its timeliness goals. For instance, the Board reviewed past data on the productivity of new staff—which is generally lower for a period of time until individuals acclimate to their jobs—and factored this into the modeling assumptions used to project the number of Board staff needed.

More specifically, sound workforce planning practices suggest that agencies identify the resources needed to manage the risks of implementing new processes and conduct scenario planning to determine these needs. While VA considered a number of factors when analyzing hiring options, it initially made many assumptions using a single set of estimates instead of using a sensitivity analysis to consider a range of estimates. These assumptions could have significant implications for how accurately VA identifies needed resources. For example, in its scenario analysis VA assumed: (1) that an average of 50 percent of those veterans appealing will refile their appeal and go through two of the four appeals process options before being satisfied; and (2) that the Board will be able to decide 130 appeals per FTE, and do so within 3 years (1,095 days), for appeals with hearings and decide 180 appeals per FTE within 1 year for appeals without hearings. Because the Board did not consider alternate sets of assumptions, VA does not know the potential effect that variations in these key variables could have on staffing needs.

In response to discussions with us on its scenario analyses, VA recently conducted further analyses using alternative estimates for key factors, although the agency’s analyses fell short of the previously discussed sound practices for estimating outcomes based on assumptions. Specifically, VA calculated the effect on appeals inventories and timeliness if VA decided 20 percent fewer appeals, if VA decided more claims and thus had more appeals than expected, and if the breakdown of options that veterans selected for their appeals review is different than the 50/50 split VA projected. The 20 percent reduction in productivity alone could add 2.5 years to VA’s estimate of how long it would take to clear the appeals inventory under hiring surge option 4. However, VA ran a sensitivity analysis for only one of the four hiring surge options and did not analyze the compounded effect of different assumptions together. By not comprehensively conducting sensitivity analyses, VA is hampered in its ability to anticipate and plan for different contingencies, and risks being caught off guard and potentially hiring an inappropriate number of staff. Hiring too few staff could result in it taking longer to reduce the inventory of pending appeals, while hiring too many staff could result in higher expenditures than needed and exacerbate other challenges, such as ensuring sufficient office space, training, and other supports for newly hired staff, as discussed below. VA has acknowledged that some of its assumptions, and thus projections, are based on unknowns and that it will need to continuously rerun the models with updated data.

VA also identified strategies and resources needed for recruiting, hiring, and training staff in fiscal year 2017; however, aspects of VA’s workforce planning fall short of sound workforce planning practices that suggest having timely written plans with a systematic approach and detailed steps, time frames, and mitigation strategies to help identify where resources and investments should be targeted. As noted below, VA has identified strategies and taken some positive steps related to recruiting, hiring, and training staff in fiscal year 2017, although these plans sometimes lacked certain details specifically covered in sound workforce planning principles in time to inform ongoing efforts.

Recruitment and Hiring: Consistent with sound workforce planning practices, officials have worked to develop a center for excellence in hiring to coordinate workforce planning and develop strategies for recruiting and hiring staff quickly. However, the center was established in the last quarter of fiscal year 2016 to support hiring beginning early in fiscal year 2017. To date, the Board developed a project to recruit recent law school graduates and alumni in fiscal year 2017, according to Board officials. It also has formed a committee of over 90 volunteers to develop recruitment materials, identify opportunities, and make contact with law schools; developed a PowerPoint presentation for the visits; and conducted a few initial presentations at law schools. However, the agency had not yet worked out specific goals such as the number of presentations or resulting applications, average time taken to recruit, and skills needed in recruits, or identified metrics (other than hiring goals) against which it would measure the effectiveness of the recruitment efforts.

Also consistent with sound workforce planning practices, Board officials told us that they considered lessons learned from a 2013 hiring surge, although the agency did not provide documentation of these lessons learned. Having established a goal of hiring 25-52 new employees per month from October 2016 through April 2017, the Board subsequently faced challenges finding space for staff coming aboard in fiscal year 2017. Specifically, as of October 2016, the Board was reconfiguring its office space to accommodate the planned 242 new FTEs in fiscal year 2017, employing nearly all of its conference rooms, and developing a plan for using telework and office sharing to accommodate staff until the space is available for them, according to officials.

Training: As of February 2017, VA rolled out training for newly hired attorneys in fiscal year 2017, which includes 4 weeks of training and 8 additional weeks of one-on-one mentoring. VA also stated that its Office of Knowledge Management was expanded with additional staff resources to ensure training materials were up to date. However, in November 2016, officials reported that the Board was still in the process of updating various aspects of its training curriculum, such as how to support conducting work in a virtual environment, which is consistent with the agency’s plans to increase telework as a way to manage space restrictions for new staff. In its comments to this report, VA did not provide updates on this effort, as of February 2017.

As of October 2016, it was unclear how the Board’s 2017 recruiting, hiring, and training efforts will be adjusted to support the agency’s proposed hiring surges in 2018 or its proposed process reform. For example, the Board has not yet determined how it will meet the space needs for any additional growth associated with hiring surges proposed for fiscal year 2018, although more detailed planning in advance might have better prepared VA for bringing aboard 242 FTEs in fiscal year 2017. In addition, VA officials stated in February 2017 that draft training for the proposed new appeals process had been prepared based on statutory language, although these draft documents were not included in VA’s comments. Federal strategic planning guidance calls for an agency to have clear plans and goals, and regularly assess its human capital approaches through assessments, as well as through data-driven human capital management to improve its ability to maximize the value of human capital investments while managing related risks. Conversely, a lack of detailed workforce plans and mitigation strategies prior to proposed hiring surges in 2018, as well as potential process reform, further places VA at risk of not being ready to accommodate another quick and much larger increase in staff, or to train them in accordance with either the legacy or proposed reform process.

VA Collaborated with Stakeholders in Developing Its Proposed Streamlined Appeals Framework, but Lacks Strategies, Such as Pilot Testing, to Mitigate Risks

VA collaborated with key stakeholders in developing its proposed appeals process reform framework and related implementation plans, which is consistent with sound practices for business process redesign. Sound redesign practices suggest coordinating with stakeholders in developing and implementing plans to obtain and maintain buy-in from start to finish and to identify and address disagreements. In developing its proposal, Board and VBA officials engaged stakeholders from 11 organizations— including VSOs that represent veterans in appeals hearings before VBA and the Board—in discussions to design a streamlined appeals process. Officials we interviewed from three of four VSOs, all of whom participated in the discussions, noted that VA’s resulting process proposal addressed both the agency’s desire to expedite appeals resolutions and stakeholder desires that the new process be fair to veterans. For example, VA identified and prioritized key concerns and found areas of consensus with VSOs. VA officials stated that they plan to continue to discuss appeals process reform (among other topics) at regular meetings with stakeholders, during which they will have an opportunity to provide feedback on previously unforeseen issues. VA officials said that as process reform is implemented, the agency will invite local VSOs to training, and share training materials and provide briefings to them and other stakeholder groups.

While VA has achieved broad agreement internally and with VSOs on its proposed process reform, there are several unaddressed gaps in VA’s business case for implementation that introduce the risk of not producing the desired results, as follows.

To develop a business case for implementing process change, sound redesign practices suggest first mapping and analyzing the target process to understand the cause and cost of performance breakdowns, and assessing potential barriers, costs, and benefits of alternative processes. This, in turn, would inform the selection of a feasible alternative with a high return on investment, and the development of a business case that describes benefits, costs, and risks. However, due to IT limitations, VA lacks data to inform and confirm its understanding of the root causes of lengthy time frames. For example, VA lacks complete historical data on the extent to which submission of new evidence and multiple decisions and appeals occur, and thus cannot determine the impact of its current, open- ended process on appeals decision timeliness. To shed light on root causes, VA analyzed 10 appeals decisions that it found took a long time to adjudicate to illustrate extreme examples of cases being re- reviewed under VA’s open-ended process—referred to by VA as “churning.” However, VA cannot know the full extent to which churning might be occurring because, according to VA officials, the way data are stored made it difficult, if not impossible, to assemble a complete historical picture prior to December 2012. To help develop baseline data, VA analyzed the average number of decisions per appeals phase for several recent fiscal years, and, according to VA officials, they are still endeavoring to piece together additional historical baselines for performance.

Further, although it was appropriate for VA to develop its proposed reform in consultation with internal and external experts, the agency did not test alternatives using data-driven, cost-effective methods suggested by sound redesign practices.

Finally, as noted previously, in modeling staff resources needed under its proposed process reform, VA relied on assumptions—about the percent of veterans who will refile, will appeal to the Board, and will submit new evidence—that have direct implications for projections of appeals workloads, time frames, and cost. However, VA did not perform sufficient sensitivity analyses to help estimate a range of potential outcomes—analyses that might help VA understand the likelihood that the new process could be more costly and time- consuming in practice than anticipated, for example, if a higher percent of veterans file with the Board, submit new evidence, and request hearings than expected.

These gaps notwithstanding, VA made some progress planning for potential implementation of proposed process reform in a manner generally consistent with sound planning practices for process redesign and change management, although some important details are still absent. According to sound planning practices, implementation is the most difficult phase of business process redesign. An agency must manage human capital and technical issues as it turns an idea into reality and overcomes potential resistance to change. To ensure an orderly transition, sound planning practices suggest following a comprehensive implementation plan that includes several key activities, such as establishing a transition team and developing a comprehensive plan to manage implementation. Consistent with this, as of October 2016, the Board and VBA had identified general time frames and offices responsible for key implementation efforts. Based on its staff modeling efforts, VA also identified how many FTEs it expects it will devote to processing cases under the current process versus a new one, should it be implemented. Also, per sound planning practices, a comprehensive plan should address workforce training and redeployment issues (including working closely with employee unions to minimize potential adverse effects). Consistent with this, as of October 2016, VA had outlined general steps and time frames for training of staff and communicating with the unions.

While VA’s high-level implementation plan included many components suggested by sound practices, key details had yet to be addressed. In particular, VA’s general timetables and plans to date have not addressed in any detail how it will implement a new process while simultaneously working to reduce the appeals inventory under the current process. For example, the agency has not explained how or who will track timeliness of appeals of the old compared to the new process, and how decisions will be made to ensure the agency is devoting an appropriate share of resources to both processes. The lack of a detailed plan for managing this transition exposes the agency to risk that veterans whose appeals are pending under the old process may experience significant delays relative to those under a new process. The Board recognized the need to ensure fairness to veterans with appeals pending under the current process, and indicated that while legislation is pending that would authorize a new process, it will continue to develop plans for managing the two processes in parallel.

Sound practices for process redesign and change management also suggest having risk mitigation strategies—in particular, pilot testing—to ensure moving successfully to full implementation. Pilot testing provides agencies opportunities to evaluate the soundness of new processes in actual practice on a smaller scale, and to refine performance measures, collect and share implementation problems and solutions, correct problems with the new design and refine the process prior to full implementation, and build capacity among unit managers to lead change. Sound redesign and change management practices both suggest that pilot tests should be rigorously monitored and evaluated, and that further roll-out occur only after the agency’s transition team has taken any needed corrective action and determined that the new process is achieving previously identified success criteria. As noted above, pilot testing is not the only method of achieving these risk mitigation goals, but sound planning practices suggest pilot testing is an important, often necessary approach for ensuring successful implementation when undertaking significant institutional change.

Contrary to sound practices, VA officials stated they do not want to pilot proposed appeals reform, even though VA’s proposed reform can be considered complex. VA’s reform plans qualify as complex because in addition to implementing a new process, the agency must still manage a large inventory of appeals under the old process while hiring and training a large number of staff and implementing IT improvements. Occurring together, these efforts involve significant change and uncertainty and will require management oversight across a broad range of efforts. In addition, VA’s proposed process reform and other initiatives affect VBA’s regional offices spread across the country, as well as the centrally located Board, thereby further increasing complexity of implementation. VA officials also stated that the proposed process reform, which has been thoroughly vetted with stakeholders, has broad support, and noted their view that the risk of fully implementing change is outweighed by the cost of delay. VA’s rationale for not pilot testing centers on what they describe as widespread consensus that the current process is “fundamentally broken” and provides “inadequate service to veterans with a high percentage of wasted effort.” VA assumed that a pilot test authorized by Congress would include a sunset date with a default reversion to the current system, which they said would introduce uncertainty into the agency’s planning efforts and a reliance on subsequent, time consuming legislation before the conclusion of the pilot. VA officials stated that piloting with a sunset date would require the agency to expend additional resources and time to conduct parallel planning for reverting to the old system upon the sunset date. VA stated that pilot testing the new process for some veterans would be perceived as inequitable, despite VA having previously supported pilot testing a new appeals process. VA officials concluded that they have not identified any risk that would justify a pilot, and indicated that they plan to mitigate risk with a strong implementation plan.

While VA has made a compelling case for reforming the appeals process, as noted previously, VA’s business case for its proposed reform in some instances relies on unproven assumptions and limited analyses of its current process that introduces risk in VA’s plans for full implementation. Importantly, VA assumes that because the current framework is “fundamentally broken,” its proposed new framework will necessarily be a better option. However, VA made this decision lacking complete data on the root cause of lengthy appeals under the current process, and without analyzing barriers, costs and benefits of feasible alternatives using cost effective methods, such as computer simulations. VA correctly notes that pilot testing prior to full implementation would slow down an overhaul of the current system, thus countering the short-term net benefit that the agency expects to realize from such an overhaul. However, VA has not acknowledged that pilot testing the new process in a more limited fashion could greatly increase the probability of long-term success by decreasing the chance that a new system will experience unanticipated problems that are potentially more widespread and therefore costly to remedy. The inclusion of risk mitigation strategies such as pilot testing does not, as VA asserts, “imply that the status quo is not in dire need of sweeping reform” but rather balances the urgency of the current problem, the technical complexity of an overhaul, and the potential for unforeseeable complications.

In light of this and the previously discussed inconsistencies in following sound planning practices, pushing forward with full implementation without testing how process reform unfolds and interacts with other efforts in actuality, may lead VA to experience implementation challenges and setbacks that could undermine efficiencies and other outcomes resulting from planned reforms. In contrast, if VA were to pilot test the proposed appeals process reform, implementation problems encountered could be identified and resolved prior to full implementation. This could lead to smoother implementation and better outcomes overall. Further, resources that would otherwise be diverted to full implementation of process reform across the organization could be focused on its current inventory of appeals. VA will also have additional time and managerial capacity to recruit and train new staff and develop and implement a communication and outreach strategy in time for full implementation of the new process. Finally, if risk mitigation strategies demonstrated that process reform would be more costly and detrimental to time frames and workloads than predicted, a decision to modify or fix the process at that juncture would be made with more information and less impact on the agency overall.

Whether VA conducts pilot testing or not, VA has not yet developed a plan for closely monitoring implementation or developed a strategy for assessing the success of its proposed process reform. Sound planning and redesign practices suggest that the transition team develop metrics and data gathering procedures, define success criteria, measure performance carefully, and take corrective action of any pilot test before proceeding to full implementation. Sound practices also suggest the agency develop meaningful performance measures—generally a mix of outcome, output, and efficiency measures—tied to overall goals of the project, and that project goals include a mix of intermediate goals to be met at various stages during the implementation phase. That way, the agency can start to show a return on investment in the early stages of implementation. To date, VA has identified several broad metrics generally reflecting outcomes, output, and efficiency—such as veteran survey results, wait times, and inventories—that it plans to use to track and assess process improvements. VA also established separate timeliness measures for the Board and VBA that it will use in its annual performance reports. While these broad metrics and goals are appropriate, they fall short of sound practices for monitoring and assessing process change in several respects.

First, VA has not developed a dashboard or balanced scorecard, or otherwise identified how it will closely monitor progress, evaluate its efficiency and effectiveness, and identify trouble spots. For example, although VA has stated that it is developing a dashboard to measure performance under its proposed appeals process, VA has not yet indicated whether, how, and with what frequency it will monitor wait times and inventories under the new versus current processes. As a result, it is not clear how VA will determine whether veterans with appeals pending under the current process are receiving equitable treatment and not experiencing significant delays relative to those under the new process. It is also unclear the extent to which VA will systematically monitor staff productivity and IT processing, which may affect its ability to determine whether assumptions are being met to help pinpoint corrective action (e.g., whether staff need more training, VA’s communication and outreach efforts are working as expected, or process reform itself is achieving desired results). Further, VA has not established interim goals or criteria for success to help determine whether initial implementation is achieving intended results. Interim goals and criteria could include specific timeliness improvements for process steps and outcomes, such as average time for VBA or the Board to reach decisions under new appeals options. If VA pursues pilot testing, such goals or success criteria will help determine whether the new process is sufficiently successful to justify full implementation.

Second, although streamlining the current open-ended process was central to VA’s business case for its proposed process reform, as noted previously, VA currently lacks sufficient data to assess the extent to which process reform will improve on the open-ended nature of the current process. VA officials said that they plan to work to incorporate capabilities into Caseflow to piece together historical baselines for performance. VA also plans to develop new baseline and historical data on aspects of the appeals process that affect the timeliness of final decisions so that they can be compared to the new process. While these are positive steps, it remains to be determined how or whether VA will be able to measure the extent to which its proposed process—which would allow the veteran to appeal multiple times—is an improvement over “churning” associated with the old process.

Lastly, the new timeliness measures that VA plans to report to Congress and the public lack transparency on whether overall appeals resolution timeliness is improving from the veterans’ perspective. In its fiscal year 2015 performance report, VA stopped reporting its average appeals resolution time measure, which included appeals decisions made by both VBA and the Board. VA officials said they considered this measure inadequate because neither VBA nor the Board has full control over making improvements to performance under this measure. VA officials told us the measure does not appropriately provide insight into the appeals process because of the variety of appeals paths and wait times veterans experience. However, the combined measure would provide a basis for comparing timeliness under the old versus new process, and would provide historical perspectives on changes in timeliness from the point of view of a veteran who may file appeals with both VBA and the Board before his or her case is resolved. VA officials stated the agency will continue to track this measure internally, but they will not include it in VA’s annual performance reports. Instead, they plan to report on VBA and Board timeliness separately. VA also stated that it will not use this measure to evaluate success of the new process because it considers a timeliness measure covering both VBA and the Board to be inappropriate.

VA Has Taken Steps Consistent With Sound Planning Practices while Updating Its IT, but Lacks Clear Schedules for Implementing IT Updates

VA has generally planned the implementation of its Caseflow appeals system consistent with sound planning practices. Working with U.S. Digital Service at VA (DSVA)—the group tasked with developing Caseflow—VA outlined an approach that has a clear scope and purpose, which is to better process appeals in a paperless environment and improve automation and productivity. The actions consistent with sound IT planning practices include:

Setting goals and objectives: VA plans clearly lay out the need for replacing VACOLS and set forth how Caseflow will address the shortcomings of VA’s current IT system. Its plans also lay out a set of broad milestones in terms of the capabilities that will be added to Caseflow in the future and the ultimate retirement of VACOLS.

Identifying and mitigating potential risks: VA planning documents identify a number of risks (such as staffing shortfalls and technical delays) and strategies to mitigate them. In addition, VA is developing Caseflow in an agile process, which officials say will allow VA to continually add new capabilities and be responsive to changing agency needs. In addition, VA officials told us that rather than replace VACOLS at once, the various functions in VACOLS will be reproduced and tested in Caseflow iteratively, and each corresponding function in VACOLS will be left intact until there is reasonable assurance that there will be no impact to VA.

Measuring performance: VA plans to develop metrics for each new component of Caseflow that is implemented. For instance, VA has developed metrics for the two components that were developed in fiscal year 2016—electronic transfer of cases to the Board and a system to electronically access documents from VBMS—which specifically assess the performance and effect of those components. As mentioned earlier, VA also plans to create a Caseflow dashboard that will provide metrics on the effect of IT improvements on timeliness of the appeals process.

Identifying organizational roles and responsibilities: VA entered into a memorandum of understanding with DSVA and the Board that outlines priorities, and a working relationship for developing Caseflow. In addition, the memorandum states that DSVA requires all initiative partners within VA to have a single point of contact with the authority to make decisions on behalf of their component.

While VA’s plans for replacing VACOLS take steps to mitigate risks, they currently do not include consideration of the timing and implications of VA’s proposed reform efforts. Federal internal control standards state that program managers, in seeking to achieve program objectives, should define objectives clearly to enable the identification of risks. This includes clearly defining what is to be achieved and the time frames for achievement. Additionally, IT investment best practices stress the need for oversight regarding a project’s progress towards predefined schedule expectations. This oversight also includes systems to make corrections regarding schedule and performance slippages. Although VA has laid out the broad capabilities it would like to incorporate into Caseflow going forward, VA has not developed a schedule for completing Caseflow. Specifically, VA could not provide us with firm time frames for when different capabilities will be active in Caseflow. As the Caseflow effort lacks time frames, VA cannot ensure that the system will be completed in time to support the implementation of proposed reforms.

Further, VA’s lack of time frames for developing Caseflow may increase the risk of additional costs if the system cannot be developed as quickly as anticipated. Sound practices specific to project scheduling state that project planning is the basis for controlling and managing project performance, including managing the relationship between cost and time. In a prior GAO report on VBMS development—which was also developed in an agile process—we reported that the agency encountered some delays with its initial deployment of key functions of VBMS, and that its lack of a schedule made it difficult to hold program managers accountable for meeting time frames and demonstrating progress.

In addition, VA has not started planning and determining the changes that would be needed for Caseflow if and when appeals reforms are implemented. VA staff said that the agile approach they are using allows them to quickly respond to changing needs, and VA Office of Information and Technology officials told us that they will not begin planning for such changes until reform legislation is passed. As stated earlier, sound IT planning practices suggest that implementation plans include specific time frames and approaches needed to implement new systems, as well as the consideration of potential risks and mitigation strategies. As such, and given the absence of a schedule for completing Caseflow, VA further risks having an IT system that is not completed in a timely manner or, even if in place in time, falls short of meeting VA’s needs.

Conclusions

With an already large inventory of pending appeals—and expectations of further growth—VA has taken steps to bolster capacity and improve the efficiency and effectiveness of its disability appeals process. Specifically, VA hired and proposed hiring more staff, is moving forward with plans to upgrade its IT systems, and has proposed bold reforms to streamline its appeals process. In planning and executing these approaches, VA took several positive steps in line with sound planning practices—such as comparing different options for increasing future staffing resources, collaborating with external stakeholders to develop a streamlined process proposal, and outlining a vision for upgrading outdated IT systems.

Nonetheless, VA’s plans do not account for the significant challenges that remain. Above all, its proposal to implement appeals reform at the Board and across VBA’s regional offices is ambitious, and as a result, VA may be exposing itself to unforeseen risks and setbacks that could slow progress toward improving appeals decision timeliness. More specifically, VA has proposed implementing process reform while also hiring more staff and upgrading its IT, which are challenging efforts in their own right. Additionally, VA does not have any plans to pilot test its proposal—a sound and often necessary practice for experiencing, evaluating, and refining significant institutional change on a smaller scale prior to full implementation. At the same time, VA plans for hiring more staff and upgrading IT lack key details (for example, on how VA will train and find working space for new staff, or a schedule for when and how system changes might be integrated with the proposed streamlined process), exposing VA to risks of delays, inefficiencies, or other setbacks caused by not anticipating needs or a misalignment of efforts. VA also did not sufficiently apply sensitivity analysis when projecting staffing needs with or without process reform, which could affect the agency’s ability to mitigate any potential risks if assumptions are not met. Lastly, VA lacks a robust monitoring plan to help assure that unforeseen problems will be quickly and effectively addressed, and has not yet developed a strategy with appropriate interim goals for process reform, and overall goals for appeals process timeliness, to gauge whether the agency’s efforts are having the desired result and reflect an improvement over prior practices. Until VA incorporates these sound planning practices, the agency lacks reasonable assurance that its proposed reform will improve the overall efficiency of the appeals process and timeliness of disability appeals decisions.

Matter for Congressional Consideration

To improve VA’s ability to successfully implement appeals process reform, Congress should consider requiring that reforms of the VA disability appeals process be subject to a pilot test. To aid in the development of such a pilot test, Congress could require the Secretary of Veterans Affairs to propose options that would allow the agency the flexibility to test and refine the new process in a cost-effective and efficient manner, while ensuring pre-established interim goals and success criteria are being met prior to full implementation.

Recommendations for Executive Action

To further align efforts to address appeals workload and improve timeliness of decisions, and reduce the risk that efforts will not go as planned, we recommend that the Secretary of Veterans Affairs direct the Under Secretary for Benefits; the Chairman, Board of Veterans’ Appeals; and the Chief Information Officer, as appropriate, to: 1. Ensure development of a timely, detailed workforce plan for recruiting, hiring and training new hires. In particular, this plan should: (1) include detailed steps and timetables for updating training curriculum (such as preparing decisions in a virtual environment) and ensuring office space (such as telework guidance); and (2) incorporate risk mitigation strategies that consider how the timing of recruitment and training dovetails with uncertain time frames for implementing a new appeals process. 2. Develop a schedule for IT updates that explicitly addresses when and how any process reform will be integrated into new systems and when Caseflow will be ready to support a potential streamlined appeals process at its onset. 3. Conduct additional sensitivity analyses based on the assumptions used in projection models to more accurately estimate future appeals inventories and timeliness. In doing so, consider running additional analyses on how these factors, in conjunction with one another, may affect the timeliness and cost of deciding pending appeals. 4. Develop a more robust plan for closely monitoring implementation of process reform that includes metrics and interim goals to help track progress, evaluate efficiency and effectiveness, and identify trouble spots.

To better understand whether appeals process reform, in conjunction with other efforts, has improved timeliness, we recommend the Secretary of Veterans Affairs direct the Under Secretary for Benefits; the Chairman, Board of Veterans’ Appeals; and the Chief Information Officer, as appropriate to: 5. Develop a strategy for assessing process reform—relative to the current process—that ensures transparency in reporting to Congress and the public on the extent to which VA is improving veterans’ experiences with its disability appeals process.

Agency Comments and Our Evaluation

We provided a draft of the report to VA for its review. In written comments, VA disagreed with one of our recommendations and agreed in principle with the other five. We have reproduced VA’s comments in appendix I and have incorporated them—as well as technical comments provided by VA—into the report, as appropriate.

In its comments, VA agreed with us that improving the efficiency and effectiveness of its appeals process is an ambitious undertaking, and we commend VA for the many steps it has taken, including collaborating with stakeholders to develop the framework for a new process. We agree that obtaining the consensus of internal and external experts—including veterans service organizations—demonstrates important progress. We disagree, however, that such consensus negates the need for more detailed plans and robust risk mitigation strategies. While it is true that VA has made noteworthy progress developing an implementation plan to guide its efforts, we found the plan lacked important details, such as: how VA will monitor for interim success and trouble spots, including whether the agency has appropriately distributed resources among the new and old processes; how it will mitigate risk of implementation challenges or setbacks, and reduce their negative impact; and how it will measure whether the new process is improving overall appeals resolution timeliness from the veteran’s perspective.

VA officials also said that VA has extensive experience in organizational change management, but it is not clear how some of the practices VA used in past transformation efforts are applicable to appeals reform, and we are concerned that VA could not provide further information on what these practices include or how they are relevant. We believe implementing all of our recommendations will increase the likelihood that VA’s efforts to improve the efficiency and effectiveness of its appeals process will be successful. For the five recommendations that VA concurred with in principle, VA described planned actions to address them and stated that it also considered the actions complete and requested we close the recommendations. However, we believe VA still needs to take actions to address those recommendations, as noted more fully below.

VA disagreed with a draft recommendation that it incorporate pilot testing of its proposed appeals process into implementation plans and pursue necessary legislative authority. In its comments, VA noted that the appeals process is broken and that piloting a new process would result in further delays to veterans appealing their disability decisions. VA disagreed with GAO’s finding that it had proposed the new process without analyzing feasible alternatives, noting that the agency designed the new process based on the collective experience of internal and external experts, and that these experts reached consensus on a new design that will be beneficial to veterans, the agency, and taxpayers, among others. VA noted that it has carefully assessed risks, identified a number of risk mitigation strategies, modeled a number of different scenarios, and developed a detailed implementation plan.

When we reviewed these efforts, however, we found three primary shortcomings that need to be addressed. First, VA did not have the data it needs to fully understand the extent to which the current process has contributed to lengthy appeals time frames, which raises questions about whether the proposed process will address the root cause or causes of untimely appeals decisions. Specifically, VA lacks historical data on the extent to which the introduction of new evidence increases time frames. Second, VA’s list of potential risks and risk mitigation strategies did not always include steps for mitigating the identified risks. Third, we found that VA’s implementation plans lacked details on how it will carry out key aspects of appeals reform, including how it will monitor the timeliness of appeals decisions under the old process compared to the new appeals process, while also hiring additional staff and integrating changes into the Caseflow IT system as discussed below. In VA’s comments, it also stated that piloting a new appeals process “would raise constitutional issues and prompt litigation.” We acknowledge that changing an adjudicatory process for determinations of benefits may prompt litigation. However, VA has not clearly articulated why pilot testing as a category is unconstitutional or why pilot testing poses unique constitutional issues. Further, as noted in the report and in VA’s comments, VA previously supported H.R. 800 in the 114th Congress, which would have directed VA to conduct an opt-in pilot process where a veteran could present a limited amount of new evidence and the Board, to the extent practical, would decide cases within one year. While GAO did not take a position on that bill, or its specific approach to pilot testing, changes of this magnitude in such a complex program justify some form of pilot testing to ensure process reform is implemented successfully and ultimately achieves VA’s goals. As noted in the report, pilot testing is recognized to be a sound planning practice and an important, often necessary approach for ensuring successful implementation when undertaking significant institutional changes.

Until VA pilot tests its appeals reforms, it will lack data to properly plan for and overcome the challenges that will likely arise during implementation. For example, VA may encounter difficulties making needed process changes while simultaneously implementing other logistical requirements, such as hiring and training new staff and updating its IT system. By not pilot testing, VA is missing a valuable opportunity to refine its implementation strategy by first seeing how process reform will unfold on a smaller scale. We believe that the potentially negative consequences of delaying full implementation are far outweighed by the benefits that can be realized through piloting. For example, piloting could help avoid delays and expenses caused by the need to re-work the process after full scale implementation. In light of VA’s disagreement with our draft recommendation, we removed the recommendation and now pose a matter for congressional consideration. Specifically, to improve VA’s ability to successfully implement appeals process reform, Congress should consider requiring that reforms of the VA disability appeals process be subject to a pilot test. To aid in the development of such a pilot test, Congress could require the Secretary of Veterans Affairs to propose options that would allow the agency the flexibility to test and refine the new process in a cost-effective and efficient manner, while ensuring pre-established interim goals and success criteria are being met prior to full implementation.

VA concurred in principle with the draft recommendation that it finalize a detailed workforce plan that includes steps for training, support, and risk mitigation strategies. VA noted that, in addition to currently implementing a fiscal year 2017 workforce plan to hire additional staff, as discussed in the report, among other efforts it has recently launched new attorney training and continues to collaborate across the agency to identify space where new staff can be located. We have incorporated these updates into our report, as appropriate. In light of these efforts, and because future steps, such as developing training materials on the new appeals process, are contingent upon appeals reform legislation, VA stated it considers this recommendation complete and requested closure. While we recognize that VA has made progress and that certain actions, such as training on a new process, is contingent upon reform legislation, we disagree that the recommendation should be closed. As noted in our report, we found that VA’s final recruiting, hiring, and training plans lacked important details. For example, VA officials were still updating training curriculum that supports work conducted in a virtual environment and which is critical for managing space restrictions for new staff. Without a detailed workforce plan in place, VA cannot assess the success of its human capital approach, maximize its investments, or fully mitigate risks. More detailed workforce plans would help VA avoid the risks that staff will not be hired in time, not be properly trained, or not have the support necessary to process appeals. Waiting until legislation is enacted magnifies these risks. We believe additional action is needed to meet the intent of this recommendation; we also clarified the recommendation language to state that VA needs a more detailed plan.

VA concurred in principle with our recommendation that it develop a schedule for IT updates that lays out when and how any process reforms will be integrated into its Caseflow system. More specifically, VA noted that it will rely on the agile process to develop Caseflow—whereby new functions are continually added to the system as new user needs or policy changes arise—and does not plan to define schedules beyond 6 months. Given that Caseflow development related to the new appeals process is dependent on the enactment of new legislation, VA stated it considers this recommendation to be complete. While it is true that the agile process can help mitigate risks and avoid cost overruns and delays, we do not believe this approach precludes VA from taking additional steps to consider the scope of potential changes required by a new appeals process and have a broad plan in place to ensure that all aspects of the new process are adequately supported by Caseflow. We believe it is especially important for VA to have specific time frames for completing Caseflow considering the scope of the changes being proposed. Moreover, VA noted that components of Caseflow developed so far will not need to be significantly changed because of appeals reform legislation being enacted, but VA did not provide documentation to support this assertion. In light of these issues, we believe VA has not yet met the intent of this recommendation.

VA concurred in principle with our recommendation that it should conduct additional sensitivity analyses around the assumptions used in its models. VA noted that sensitivity analyses are valuable and that it has focused its efforts on risks its staff identified as most likely, such as variations in staffing and productivity, and the effect of remands. VA stated it would continue to analyze, update, and refine its modeling, and considers this recommendation to be complete. While we recognize the logic of focusing modeling resources on key variables, VA did not fully examine three of the four hiring surge options it proposed. Moreover, VA did not assess the compound effect that would result in changing multiple assumptions at once. Given the complexity of proposed changes and the number of variables beyond VA’s control, we believe that additional analyses are needed to identify potential risks that may warrant additional mitigation strategies. In addition, if VA goes forward with appeals process reform and begins to collect real-time data, these data could improve modeling accuracy and serve as a valuable management tool.

VA concurred in principle with our recommendation to develop a more robust plan for closely monitoring the implementation of its process reform, that includes metrics and interim goals to help VA track progress, evaluate efficiency and effectiveness, and pinpoint trouble spots. VA agreed that developing such a plan is valuable for monitoring the implementation of process reform, and should include metrics and interim goals. However, VA stated that it considers this recommendation complete, noting that preparing such a detailed plan depends on appeals reform legislation being enacted, and that it will incorporate specific goals and metrics as it moves towards implementation. While we recognize that VA cannot assume to know the exact provisions that may be included in future enacted legislation, nor can it predict when appeals reform might be enacted, we consider having a more robust monitoring plan to be essential to the successful implementation of a new appeals process. Moreover, the absence of such a plan raises questions as to how VA will ensure appropriate resources are devoted to managing appeals under the new versus old process, or intended results are achieved as the new process is implemented.

VA concurred in principle with our recommendation to develop a strategy to transparently report to Congress and the public on veterans’ experiences with the new appeals process. VA noted that it is already developing timeliness goals for three of the four appeal options in the proposed new process, as discussed in our report. VA said it also plans to measure success of the new process with results from customer satisfaction surveys and is developing a dashboard for internal performance monitoring. VA did not agree that measuring overall appeals resolution timeliness is an appropriate a measure and believes tracking time frames for each of the options separately is more appropriate. While we agree that metrics based on the different options could be valuable for VA, the Congress, and the public, we disagree that VA’s focus on measuring timeliness by option is in the best interest of the veteran. Because veterans may pursue more than one option under VBA, the Board or both, we believe that VA’s approach does not take into account the veteran’s perspective of how long it took for them to receive a final appeal decision. Metrics from the veterans’ overall perspective would complement, not replace, metrics for VBA, the Board, and each option. Further, because VA’s approach does not allow VA to compare the new process with the old or to determine whether the new process represents an improvement over the old process, we believe it does not promote transparency in reporting to the Congress and the public.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, the Under Secretary for Benefits, the Chairman of the Board of Veterans Appeals, and the VA’s Chief Information Officer. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or bertonid@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II.

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Michele Grgich (Assistant Director), Melissa Jaynes (Analyst-in-Charge), Daniel Concepcion, and Greg Whitney made key contributions to this report. Other key contributors to this report include James Bennett, Mark Bird, David Chrisinger, Clifton Douglas, Alex Galuten, Mitch Karpman, Sheila R. McCoy, Claudine Pauselli, Almeta Spencer, Eric Trout, Walter Vance, and Tom Williams.

Related GAO and Other Products

For the purpose of evaluating VA’s efforts to improve its appeal processing, we identified best practices and other criteria related to staffing, process reform, and IT upgrades identified in prior GAO products and other publications. These included government-wide internal control standards; key principles for effective strategic workforce planning; business process reengineering (or redesign) best practices, and information technology planning principles. We also reviewed additional guidance on project management.

Schedule Assessment Guide: Best Practices for Project Schedules. GAO-16-89G. Washington, D.C.: December 2015.

Veterans Benefits Management System: Ongoing Development and Implementation Can Be Improved; Goals are Needed to Promote Increased User Satisfaction. GAO-15-582. Washington, D.C.: September 1, 2015.

Standards for Internal Control in the Federal Government. GAO-14-704G. Washington, D.C.: September 2014.

Human Capital: Strategies to Help Agencies Meet Their Missions in an Era of Highly Constrained Resources. GAO-14-168. Washington, D.C.: May 7, 2014.

Managing For Results: Agencies Should More Fully Develop Priority Goals under the GPRA Modernization Act. GAO-13-174. Washington, D.C.: April 19, 2013.

GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs. GAO-09-3SP. Washington D.C.: March 2009.

Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government. GAO-04-546G. Washington, D.C.: March 2004.

Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity. GAO-04-394G. Washington, D.C.: March 2004.

Combating Terrorism: Evaluation of Selected Characteristics in National Strategies Related to Terrorism. GAO-04-408T. Washington, D.C.: February 3, 2004.

Human Capital: Key Principles for Effective Strategic Workforce Planning. GAO-04-39. Washington, D.C.: December 11, 2003.

Human Capital: A Self-Assessment Checklist for Agency Leaders. GAO/OCG-00-14G. Washington, D.C.: September 2000.

The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans. GAO/GGD-10.1.20. Washington, D.C.: April 1998.

Business Process Reengineering Assessment Guide. GAO/AIMD-10.1.15. Washington, D.C.: May 1997.

Executive Guide: Effectively Implementing the Government Performance and Results Act. GAO/GGD-96-118. Washington, D.C.: June 1996.

Blackburn, Simon, Sarah Ryerson, Leigh Weiss, Sarah Wilson, and Carter Wood, Insights into Organization: How Do I Implement Complex Change at Scale? Dallas, Texas: McKinsey & Company, May 2011.

George, Michael L, David Rowlands, Mark Price, and John Maxey. The Lean Six Sigma Pocket Toolbook: A Quick Reference Guide to Nearly 100 Tools for Improving Process Quality, Speed, and Complexity. New York: McGraw-Hill, 2005.

Office of Management and Budget. Circular No. A-11, Preparation, Submission, and Execution of the Budget. Washington, D.C.: July 1, 2016.

Project Management Institute, Inc. A Guide To The Project Management Body Of Knowledge (PMBOK Guide) Newtown Square, PA.: 2013.