VA Disability Benefits: Additional Planning Would Enhance Efforts to Improve the Timeliness of Appeals Decisions
Published: 2017-03-23T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: VA compensates veterans for disabling conditions incurred in or aggravated by military service. Veterans can appeal VBA's decisions on their compensation claims, first to VBA and then to the Board, a separate agency within VA. In fiscal year 2015, more than 427,000 appeals were pending and veterans waited over 3 years on average for decisions. Of this total, about 81,000 were pending at the Board and the average cumulative time veterans waited for a decision by the Board in 2015 was almost 5 years.
This report examines VA's approaches to address challenges it identified as contributing to lengthy appeals processing times, and the extent to which those approaches are consistent with sound planning practices.
GAO focused mainly on the Board, which experienced an increase in workload of about 20 percent from fiscal year 2014 to 2015. GAO reviewed VA's proposed plans and actions and compared them to sound practices relevant to workforce planning and implementing process redesign and new information technology identified in federal guidance, such as internal control standards, and prior GAO work. GAO also analyzed VA's data for fiscal years 2011-2015 (the most recent available) on appeals decision timeliness and workloads; reviewed relevant federal laws, regulations, and planning documents; and interviewed VA officials and veterans service organizations.
section_title: What GAO Found
paragraphs: The Department of Veterans Affairs' (VA) is taking steps to improve the timeliness of its benefit compensation appeals process, in which veterans who are dissatisfied with claims decisions by the Veterans Benefits Administration (VBA) can appeal first to VBA, and then to the Board of Veterans' Appeals (the Board). VA has taken actions related to increasing staff, reforming the process, and updating information technology (IT), which are consistent with relevant sound planning practices. However, gaps in planning exist, thereby reducing the agency's ability to ensure that these actions will improve the timeliness of disability appeals decisions.
Increase staff : VA determined that staff resources have not sufficiently kept pace with increased pending appeals, and concluded that additional staff are needed, particularly at the Board, to improve timeliness and reduce its appeals inventory. The Board received approval to hire more staff in fiscal year 2017, and expects to need an additional hiring surge beginning in fiscal year 2018. As of October 2016, officials estimated that if the agency does not take any action, such as increasing staff in 2018, veterans may have to wait an average of 8.5 years by fiscal year 2026 to have their appeals resolved.
Consistent with sound workforce planning practices, VA modeled different options for increasing staff levels to support its conclusion that staff increases in conjunction with process change would reduce the appeals inventory sooner. However, contrary to sound practices, VA often used fixed estimates for key variables in its models—such as staff productivity—rather than a range of estimates (sensitivity analysis) to understand the effect variation in these key variables could have on staffing needs. Also, VA's written workforce plans—which cover recruiting, hiring and training—did not include detailed steps, time frames, and mitigation strategies consistent with sound workforce planning practices. For example, while VA has established a center for excellence in hiring to focus on recruitment and hiring the agency has not finalized training or telework plans or otherwise mitigated space constraints that it encountered for hiring staff in fiscal year 2017. Without a timely, detailed workforce plan, VA risks delays in hiring and preparing staff to help manage workloads as soon as possible.
Reform process: VA determined that new evidence—which a veteran can submit at any point during his or her appeal—inefficiently causes an additional round of reviews, and thus delays appeals decisions, and in response it proposed legislation (not enacted) to streamline the process. Consistent with sound practices for process redesign, VA worked with veterans service organizations (VSO) and other key stakeholders in developing the proposal, and continued to update VSOs about the development of its implementation plans.
VA's proposed reform is promising, but there are several gaps in its implementation plans. In particular, VA plans to fully implement appeals process reform at the Board as well as at VBA regional offices across the country while it concurrently manages the existing appeals inventory, a hiring surge, and planned system changes discussed below. However, VA's plans run counter to sound redesign practices that suggest pilot testing the process changes in a more limited fashion before full implementation, in order to manage risks and help ensure successful implementation of significant institutional change. VA officials told GAO that pilot testing—which would require legislation to implement—will prolong a process that is fundamentally broken and delay urgently needed repairs. However, without pilot testing VA may experience challenges and setbacks on a broader scale, which could undermine planned efficiencies and other intended outcomes. In addition, VA has not sufficiently identified how it will monitor progress, evaluate efficiency and effectiveness, identify trouble spots, and otherwise know whether implementation of its proposed process change is on track and meeting expectations. The absence of a robust monitoring plan with success criteria is inconsistent with sound planning practices for redesign and places the agency at risk of not being able to quickly identify and address setbacks. In addition, the timeliness measures that VA currently plans to report to Congress and the public lack transparency because they focus on individual parts of the agency and pieces of the new process rather than overall appeals resolution time from the veterans' perspective. Without a strategy for assessing the proposed new process that includes comprehensive measures, VA, the public, and Congress cannot know the extent to which the proposed process represents an improvement over the old process.
Update technology: VA determined that the computer system supporting its appeals process is outdated, prone to failures, and does not adequately support electronic claims processing. VA proposed a new IT system to reduce delays in appeals to the Board, and better integrate data from other systems. Consistent with sound practices, VA clearly laid out the scope and purpose of IT upgrades, and identified risks and strategies to mitigate them. However, the agency's plan lacks details for how and when its new system will be implemented, as suggested by sound planning practices for implementing new technology. Without a detailed schedule, VA risks not having new systems aligned with potential changes in the appeals process when they are implemented.
section_title: What GAO Recommends
paragraphs: GAO is making five recommendations to VA and one matter for congressional consideration. VA should: apply sensitivity analyses when projecting staff needs, develop a more timely and detailed workforce plan, develop a robust plan for monitoring process reform, develop a strategy for assessing process reform, and create a schedule for IT improvements that takes into account plans for potential process reform. VA concurred in principle with the five recommendations, but believes it has met the intent of those recommendations and does not need to take additional action. GAO disagrees and—while recognizing VA's ongoing efforts—believes further action is needed on all five recommendations to improve VA's ability to successfully implement reforms, as discussed in the report.
VA disagreed with an additional draft recommendation that it incorporate pilot testing of its proposed appeals process into implementation plans and pursue necessary legislative authority. VA cited its perspective that the appeals process is broken and that piloting a new process would result in further delays to veterans appealing their disability decisions. GAO maintains that the benefits of pilot testing—which provides an opportunity to resolve implementation challenges and make refinements to the process on a smaller scale—outweigh the potentially negative consequences of delaying full implementation. Therefore, GAO removed the recommendation and added a matter for congressional consideration stating that Congress should consider requiring that appeals process reform be subject to a pilot test.

section_title: Letter
paragraphs: In fiscal year 2015, the Department of Veterans Affairs (VA) provided  about $63.7 billion in disability compensation payments to about 4.1  million veterans with disabling conditions that were incurred during or  aggravated by their military service. VA’s Veterans Benefits  Administration (VBA) determines veterans’ eligibility and payment  amounts based on medical records and examinations, military service  records, and other evidence. A veteran dissatisfied with VBA’s initial claim  decision may appeal at two VA levels—first to VBA, then to the Board of  Veterans’ Appeals (Board). In recent years, VA has reduced its high  inventory of initial disability claims to fewer than 100,000, but at the same  time, appeals inventories have grown. In fiscal year 2015, according to  VA data, the number of appeals pending at VBA and the Board was more  than 427,000 and veterans waited over 3 years on average for an  appeals decision made in that fiscal year by either VBA or the Board. VA  reported that appeals workloads at the Board almost doubled from 41,000  in fiscal year 2011 to 81,000 in fiscal year 2015, increasing about 20  percent from fiscal year 2014 to fiscal year 2015 and the average  cumulative time veterans waited for an appeals decision by the Board in  2015 was almost 5 years (1,789 days). GAO has designated VA disability  compensation and other federal disability programs as a high-risk area,  due in part to the challenges VA faces in deciding disability benefit claims  (including appeals) in a timely manner.
In light of VA’s growing appeals workload, and delays in deciding  appeals, we conducted this review under the Comptroller General’s  authority to conduct evaluations of significant current issues on his own  initiative. In this report, we examined: (1) the key challenges VA identified  that affect the timeliness of appeals decided by the Board, and what VA is  doing to address them; and (2) the extent to which VA’s proposed  approaches to address these challenges are consistent with sound  planning practices. Our review largely focused on the Board’s role in the  disability compensation appeals process, given recent increases in its  workloads, and that GAO has not conducted a review of the Board and its  handling of appeals in over 10 years.
To examine key timeliness challenges and VA’s efforts to address them,  we reviewed relevant agency reports, planning documents, budget  submissions, and federal laws and regulations. We also interviewed VA  officials representing the Board, VBA, and the Office of Information and  Technology. From these sources, we identified three approaches that VA  stressed as important to achieving significant improvements in appeals  resolution timeliness: (1) increasing staff; (2) streamlining the appeals  process; and (3) improving information technology (IT) support.
To provide additional perspective on workload challenges, we obtained  VA administrative data for fiscal years 2011 through 2015 on pending  appeals inventory, age of appeals determined in each year, number of  staff, quality measures (such as error rates and the types of errors  specific to appeals), and the extent to which veterans appeal cases  repeatedly. We selected a 5-year span for our analysis using data for the  most recent years available at the time of our request. We validated data  provided by VA by comparing what we received against a data extract we  collected from the Board’s Veterans Appeals Control and Locator System  (VACOLS)—which is used by VA to track the status of appeals. We  determined that these data were sufficiently reliable for our purposes. To  provide further insight on workload challenges, we also interviewed  representatives of four veterans service organizations (VSO)—the American Legion, Disabled American Veterans, National Veterans Legal  Services Program, and Paralyzed Veterans of America—each of which  has staff members who represent veterans before the Board, and had  participated in recent VA efforts to improve the appeals process.
To assess the extent to which VA’s approaches to address challenges  are consistent with sound planning practices, we identified best practices  and other criteria through a review of relevant literature, such as  government-wide internal control standards, and prior GAO reports. We  specifically selected prior GAO reports where we defined a number of  desirable characteristics of an effective, results-oriented plan, or  components of sound planning practices. We also reviewed additional  guidance pertinent to process change and project management.  Although there is no established set of requirements for all plans, we  incorporated relevant best practices and criteria into a data collection  instrument reflecting sound planning practices and used this instrument to  guide our evaluation of each of VA’s approaches. We further sought out  and reviewed sources identifying key practices and desirable  characteristics related to VA’s three key approaches (staffing, process  reform, and IT upgrades) to improving review processes and workloads  including strategic planning practices, transformation plans, technology  upgrades, and human capital improvements (mostly increasing staffing).  For example, for approaches related to staffing, we referred to reports  with key principles for effective strategic workforce planning; for process  reform, we used GAO guidance on business process reengineering (or  redesign); and for IT upgrades, we referred to GAO guidance and reports  on information technology planning. For a list of the sources of sound  practices and criteria we used to evaluate VA’s approaches, see related  products at the end of this report.
We also reviewed VA’s analysis assessing the effectiveness of two of the  proposed approaches (staffing increases and process reform) for  addressing workload challenges. Specifically, we reviewed data analysis  spreadsheets and related documentation (such as historical data and  assumptions), conducted interviews, and observed a demonstration of  how VA compiled the data for its projections. We did not assess the  feasibility of the options compared in its analysis.
We conducted this performance audit from February 2016 to March 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: VA Disability Compensation Benefit Claims Process
paragraphs: VA’s process for deciding veterans’ eligibility for disability compensation  begins when a veteran submits a claim to VA. The claim is reviewed at  one of VBA’s 56 regional offices where staff members assist the veteran  by gathering any additional evidence, such as military and medical  records, needed to evaluate the claim. Based on this evidence, and the  results of any necessary medical examinations, VBA decides whether the  veteran is entitled to compensation and, if so, how much. VBA assigns a  rating of 0 to 100 percent disability in increments of 10 percentage points  depending on the severity of the disability. This rating percentage then  determines the monthly payment amount the veteran will receive.  According to VA data, in many cases (74 percent), the veteran submitting  a claim either is already a beneficiary but is seeking increased  compensation, or the veteran was denied benefits previously and is  claiming them again. In fiscal year 2015, VBA decided 1.4 million  compensation claims and had an inventory of 363,000 claims at the end  of the fiscal year. As previously noted, in fiscal year 2015, VA paid about  $63.7 billion in disability compensation to about 4.1 million veterans.
section_title: VA Appeals Process
paragraphs: A veteran dissatisfied with VBA’s initial claim decision can generally  appeal within one year from the date of VBA’s notification letter to the  veteran. According to the Board, veterans appeal most often because  they believe VBA: (1) incorrectly denied them compensation for service- connected disabilities, or (2) under-rated their service-connected  disabilities. An appeal begins with the veteran filing a Notice of  Disagreement (NOD). VBA then re-examines the case and generally  issues a Statement of the Case (SOC) that represents its decision.
A veteran who is or remains dissatisfied with VBA’s decision can file an  appeal with the Board. In filing that appeal, the veteran could indicate  whether they would like a Board hearing. VBA prepares the claim file for  Board review and certifies it as ready for review. If the veteran requests a  hearing so they can present new evidence or arguments, the Board will  generally hold a hearing either by video conference or at a local VBA  regional office. The Board reports to the Office of the Secretary of  Veterans Affairs, and is independent of VBA. The Board’s members, also  known as Veterans Law Judges (VLJ), decide appeals and are supported  by attorneys and administrative staff. After the appeal is docketed at the  Board, a VLJ or panel of VLJs reviews the evidence and either (1) grants  the claimed benefit, (2) denies the benefit, or (3) returns (remands) the  claim to VBA for additional work on one or more issues pertinent to the  claim and a new decision. According to VA, the Board remands an appeal  to VBA in cases where consideration of new evidence, clarification of  evidence, correction of procedural defect, or any other action it deems is  essential to achieve a proper decision. If the veteran is unsatisfied with  the Board’s final decision, the veteran can continue an appeal beyond VA  to federal court. Such an appeal begins with the U.S. Court of Appeals for  Veterans Claims, then may go to the U.S. Court of Appeals for the  Federal Circuit, and finally to the U.S. Supreme Court. See figure 1 for a  representation of the appeals process for VA disability compensation  benefit decisions.
section_title: Inventories and Timeliness of VA Appeals
paragraphs: According to VA officials, the number of appeals filed has increased  steadily as has the length of time needed for the agency to make a final  decision. At the end of fiscal year 2015, according to VA data, VA had  over 427,000 pending appeals, approximately 81,000 of which were at  the Board. While the percentage of pending appeals awaiting decisions  from the Board was less than a quarter of all pending appeals, the fiscal  year 2015 inventory was almost double the 41,000 pending at the end of  fiscal year 2011. About 20 percent of this growth occurred from fiscal year  2014 through 2015.
According to Board data, timeliness has worsened since fiscal year 2011  as well. From fiscal years 2011 through 2015, the average amount of time  needed for the Board to make a final decision once the appeal is  docketed increased from 240 to 270 days. In addition, the proportion of  cases taking the longest to resolve (from when the Board receives the  certified appeal to them making a final decision)—over 600 days— increased from 10 percent in fiscal year 2011 to 14 percent in fiscal year  2015 (see fig. 2). Given that the median time for the Board to decide an  appeal was 145 days in fiscal year 2015 (compared to an average of 270  days), these data suggest that a relatively small number of appeals is  driving up the Board’s reported average processing times. To illustrate,  VA officials noted one case where a veteran appealed 27 times over the  course of 25 years before the original appeal was concluded.
section_title: VA Has Proposed Increased Staffing, Process Reform, and Updated IT to Improve Appeals Timeliness, and Has Taken Action for All Three Approaches
paragraphs: VA has identified three broad approaches for addressing factors that it  identified as having contributed to increased appeal inventories and  reduced timeliness of appeals decisions, and has already taken action on  all three fronts. Citing staffing levels that have not kept pace with  workloads, VA secured additional Board staff for fiscal year 2017, and  analyzed options for another hiring surge in fiscal 2018. Concerned that  its appeals process contributes to delays in appeals decisions—because  new evidence may be submitted at any juncture and because VA may be  continually required to develop or obtain additional evidence—VA  developed a legislative proposal for streamlining its appeals process,  including new appeals options designed to accelerate decision-making.  Finally, VA has put forth plans to modernize its current, outdated, and  inefficient computer system.
subsections: section_title: VA Proposed Hiring More Staff to Address Increasing Workloads
paragraphs: VA has proposed increasing staff at the Board, as well as VBA, to  manage its increasing inventory of appeals and to address related  declines in the timeliness of appeals resolutions. VA officials stated that  there is a direct and proportional correlation between the number of  employees and the number of final appeals decisions, and that Board  workloads especially have increased faster than the number of  employees staffed to the Board. Specifically, officials have concluded that  staff resources within the Board have not been sufficient to adjudicate the  increasing number of appeals, ultimately lengthening appeals resolution  times. According to VA, in fiscal year 2015 increases in staff (VLJs,  attorneys, and support staff), as represented by full-time equivalents  (FTEs), allowed the Board to make the highest number of decisions in  nearly 30 years. However, despite Board staff increasing by 21 percent  from fiscal years 2011 through 2015, officials said that this increase was  not sufficient to address the growing inventory of pending appeals, which  doubled during the same time period (see fig. 3).
Although the increase in Board staff brought about a record number of  appeals decisions in fiscal year 2015, according to VA data we reviewed,  each appeal took an average of about 3 months (97 days) longer to reach  a final decision than in fiscal year 2012. Similarly, in fiscal year 2015  one Board FTE produced an average of 86 appeals decisions, down from  91 completed per FTE in fiscal year 2011. Growing workloads and the  increased complexity of cases, according to Board officials, have  contributed to these longer appeal resolution times. More specifically,  officials said that claims have become more complicated due to not only  the number and complexity of injuries and illnesses, but also to advances  in medicine that have improved survival rates from catastrophic injuries,  experienced by today’s veterans. VA officials estimated that if the number  of FTEs and number of appeals decided per FTE stays steady or  decreases, appeals resolution times will continue to lengthen.  Specifically, as of October 2016, VA projected that if nothing else  changes, and if the number of FTEs hold steady at the fiscal year 2017  number (922 FTEs for the Board and 1,495 for VBA), the inventory of  appeals could exceed 1 million in fiscal year 2026, which would mean that  veterans would wait an average of 8.5 years for a final appeals decision.
In light of this assessment, VA concluded that increasing the number of  FTEs at the Board is a key step in mitigating the current and future  pending inventory of appeals and ultimately improving appeals decision  timeliness. In 2016, VA set a goal to decide the vast majority (90 percent)  of appeals (including both those reviewed by VBA and the Board) within 1  year by 2021. As an initial step toward this goal, VA requested and  received a funding amount that the agency asserted would allow it to fund  an additional 242 FTEs for the Board in fiscal year 2017 (a 36 percent  increase over the 680 FTEs funded in fiscal year 2016) for a total of 922  FTEs. VA also concluded, however, that this increase in staff will not be  enough to reduce its appeals workload and decrease appeals processing  time. Therefore, VA estimated the need for a subsequent hiring surge of  up to 1,458 FTEs beginning in fiscal year 2018 to reduce the current  pending appeals inventory.
To understand the need for and implications of a future hiring surge, VA  modeled different staffing scenarios. Initially, VA compared how  increasing staff in combination with and without proposed changes to the  appeals process would achieve inventory reductions, and at what cost.  VA determined that by combining staff increases with a new process, it  could clear pending appeals faster and at a lower cost than if it hired  additional staff under the current process. In response to congressional  inquiries, in September 2016 VA also modeled the cost and impact on  appeals inventories of four surge options beginning in fiscal year 2018 (in  addition to planned hiring in fiscal year 2017). VA estimated, for  example, that projected pending appeals in fiscal year 2017 (535,726)  would be cleared in 10 years under option 2, compared to a 60 percent  reduction over the same time period if there were no hiring surge. See  table 1 for a comparison of the four options.
section_title: VA Proposed Reforming Current Appeals Process to Address Factors Contributing to Delays in Resolving Appeals
paragraphs: VA has proposed changes to its appeals process to address causes of  delays in resolving appeals. The key challenge VA identified was the  open-ended nature of its disability appeals process, whereby a veteran  can submit additional evidence numerous times at any point during the  VA appeals process, which can cause another cycle of re-adjudication.  Specifically, when a veteran submits additional pertinent evidence after  VA’s initial decision on their claim, VA is generally required to review the  evidence, develop any other needed evidence, and issue another  decision. This is the case regardless of whether the veteran submits the  additional evidence to VBA or to the Board and, for appeals pending  before the Board, the submission of additional evidence may result in a  remand to VBA for further development. VA reported that in fiscal year  2015, the Board remanded about 46 percent of appeals to VBA for  additional development. Of those remanded appeals, which may involve  more than one issue, VA reported that about 60 percent of the reasons  those appeals were returned to VBA were due to the open record that  allows veterans to introduce new evidence at any point during the  appeal. VA reported that in fiscal year 2015, it took VBA an additional  255 days on average to complete remand development and for the  appeal to be re-docketed at the Board. VA also reported that in fiscal year  2015 it took the Board an average of 244 additional days to complete its  subsequent review of the returned remands and decide the appeal.  According to VA, this re-adjudication can occur multiple times and can  add years to the time needed to reach a final decision on an appeal.
Board and VSO officials also identified factors within VBA’s initial claim  process—and outside of the Board’s control—that cause delays in  veterans receiving final decisions on their appeal. Specifically,
According to Board and VSO officials, VBA’s decision notification  letters are unclear and confusing. In particular, the officials stated that  these letters do not adequately explain why claims were denied and  do not clearly identify the evidence a veteran needs to provide to fully  support a claim on appeal. As a result, some veterans may appeal  unnecessarily, or they may appeal without providing the evidence  needed to support their claims.
VSO officials we interviewed said that some delays are attributable to  errors in VBA’s initial decisions. They suggested that errors may have  occurred because VBA rushed some decisions in its initiative to  reduce its backlog of claims pending more than 125 days. Such errors  can lead to Board remands and VBA re-work. For example, the Board  may remand an appeal because VBA failed to meet the “duty to  assist” responsibilities to a veteran. According to the Board, 41  percent of the reasons for the remands in fiscal year 2015 were due to  a VBA error.
Board and VSO officials also cited delays in VBA’s transmittal of  appeals to the Board as a possible cause for the delays in Board  decisions. When a veteran files an application with VBA to appeal to  the Board, VBA prepares the case file for transfer to the Board,  certifies that the case file is complete and ready for Board review, and  transmits the file to the Board. According to VA data on appeals  decisions made by the Board in fiscal year 2015, it took on average  537 days to process the appeal from receipt to certification. Docketing  appeals that had been certified to the Board added an average of 222  days to processing times for appeals decisions made in fiscal year  2015. VSOs (two of the four we interviewed) told us that they noticed  these delays occurring as VA’s focus shifted to clearing the  compensation benefit claims backlog.
To address process-related challenges, VA’s approach has been to  develop a proposal to streamline the appeals process and to ask the  Congress to make changes in the laws governing the process. In April  2016, VA issued a draft summary of a proposed streamlined appeals  process that reflected collaboration with its stakeholders. This summary  was accompanied by draft legislation for the Congress’ consideration. If  enacted into law, the draft legislation would make process changes that  VA identified as needed to streamline the appeals process. According to  VA, key to the proposed process changes would be replacing the current  appeals process, which begins in VBA, with a process giving a veteran  four options—two in VBA and two in the Board. As presented in VA’s  framework, these options would be:
Ask VBA to review its initial decision based on the same evidence.
Under this option, the veteran would not be able to submit new  evidence or request a VBA hearing, and would not be subject to VA’s  “duty to assist” requirement. A VBA official (at a level higher than the  official who made the initial decision) would review the record  supporting the initial decision, and issue a new decision.
The veteran could file a “supplemental claim” with VBA, asking VBA to  review its initial decision, while providing additional evidence. Under  this option, the veteran could also request a VBA hearing. Another  VBA official (at the same level as the original VBA decision-maker)  would review the revised record, including the additional evidence  from the veteran, and issue a new decision.
The veteran could file a Notice of Disagreement directly with the  Board, bypassing a VBA review. The veteran would have two options  with the Board:
Ask the Board to review only the existing record without a hearing  and then issue a decision.
Alternatively, ask the Board to review additional evidence, conduct  a hearing before issuing its decision, or both.
See figure 4 for a representation of the options in VA’s proposed  simplified appeals process.
VA officials anticipate that the proposed appeals process described  above will expedite appeals in a number of ways, most notably:
For those appeals where no additional evidence is submitted, and no  formal hearing is conducted (indicated as “VBA conducts local higher- level review” and “Board reviews record without new evidence or a  hearing” in figure 4), the re-review of the original record could  expedite a final appeals decision. In addition, VA’s “duty to assist”  requirements would only apply to VBA for initial and supplemental  claims. Unlike the current process, in which the Board may remand  appeals to VBA to consider new evidence, the Board would only  remand appeals under the new process in cases in which the Board  found that VBA failed, in its initial or supplemental claim processing, to  meet VA’s “duty to assist” the veteran. VA estimates that, once the  new process is fully implemented, remands will steadily decrease and  eventually occur in as few as 5 percent of appeals.
When the veteran appeals directly to the Board, VBA would no longer  be required to review the record (including any additional evidence),  prepare statements of its findings (i.e., prepare SOCs or SSOCs), and  certify appeals as ready for Board review.
VA has estimated that as a result of these process changes—in  combination with increased FTEs—the Board could complete cases  faster, deciding many more appeals per FTE in fiscal year 2018  compared to fiscal year 2015. More specifically, VA estimated that the  Board could complete an average of 180 appeals decisions per FTE  without a hearing and 130 with a hearing, compared to the average of 86  total decisions per FTE in fiscal year 2015. We discuss VA’s estimates in  more detail later in this report.
While VA’s proposal reflects VA’s intent to expedite appeals resolutions, it  also contains various protections for veterans that are intended to  address stakeholders’ concerns about fairness. Notably, such protections  include the following:  In contrast to the “one size fits all” process, the proposed reform  allows the veteran to choose an option that best fits the circumstances  of a veteran’s claim. As shown in figure 2 above, a veteran could  choose to have VBA review the initial decision or they could appeal  directly to the Board. Also, the veteran would have the option to have  either VBA or the Board review the existing record, without having to  submit new evidence and/or request a formal hearing. VA expects  that these options could help the veteran obtain a faster decision from  VBA or the Board. Per VA’s framework, under the new process, the  veteran would have up to 1 year from VBA’s initial decision to choose  an option. Further, if the veteran is unsuccessful in one appeal option,  the veteran could, within 1 year, choose another option. However,  according to VA, an appeal for a higher-level review by VBA without  new evidence cannot directly follow a Board decision.
VBA would be required to provide more information in letters notifying  veterans of decisions involving a denial of benefits, which could help  veterans make more informed decisions on whether to appeal, which  option to pursue, and what additional evidence (if any) to provide.  The inclusion of additions to such notifications in VA’s proposed  legislation addresses stakeholders’ concerns that veterans did not  have enough information to decide whether they should appeal, or  what additional evidence they needed to provide, thereby resulting in  unnecessary appeals or delays in appeals.
A veteran who is not fully satisfied with the result of any lane would  have 1 year to seek further review, while preserving an effective date  for benefits based on the date the veteran filed the original claim with  VBA. This would help ensure a veteran is not penalized for pursuing  an appeal to the Board. For example, under VA’s proposal, a veteran  denied benefits by the Board could choose to have VBA conduct  another review, by filing a supplemental claim with additional  evidence. In contrast, under current law, if a veteran appeals to the  Board and is denied (and does not appeal to a federal court), the  veteran must generally reopen the claim, or start over, by filing  another claim with VBA. If the veteran is subsequently granted  benefits, the benefits would generally be awarded from the date on  which the new (not original) claim was filed, which could result in the  veteran not receiving retroactive compensation payments.
section_title: VA Has Plans to Modernize Its Outdated Appeals IT System
paragraphs: VA has plans to modernize its current IT system, which it determined is  antiquated and a source of delays in processing appeals. VA currently  uses the Veterans Appeals Control and Locator System (VACOLS) to  track and manage its appeals workload. VA identified a number of  reasons why it believes VACOLS should be replaced, including that:
The system is based on outdated technology dating from the 1990s  that VA determined would be difficult to modify to meet the changing  needs of VA.
VA designed VACOLS around a paper-based VA claims process and  as a result, VACOLS does not adequately support a fully electronic  environment. According to VA, although VACOLS has been patched  to some extent to handle paperless appeals, the Board relies on  paper briefs to help manage its appeals workflow.
VACOLS’s lack of automation, integration with other VA systems, and  error checks results in mistakes and lost productivity. According to  VA, individual employees spend a significant amount of time  correcting data entry errors that would be avoided if cases were  automatically transferred to the Board. For instance, they said that  after cases are transferred to the Board, a team of employees must  manually review and correct most incoming cases due to issues with  labeling, mismatched dates, and missing files. Via an internal study,  VA determined that up to 88 percent of cases transferred to the Board  had such errors. Additionally, VA notes that data entry errors can  result in instances where paperless cases are mislabeled as paper- based. These cases will not show up as certified in VACOLS and the  Board will erroneously wait for a paper case that will never arrive.
VACOLS is central to appeals processing, thus a system outage  would halt the processing of appeals across VA, either paper or  electronic, until VACOLS is repaired, according to VA.
VA expects its VACOLS replacement to improve the efficiency of its  appeals decisions. Its planned replacement—called Caseflow—is  intended to address the limitations of VACOLS and better support  processing appeals in a paperless environment. According to VA,  Caseflow is being developed in an agile process in which new functions  are added to the system as they are completed. In fiscal year 2016, VA  developed two initial deliverables. According to VA, the first is intended to  automate and introduce consistency to the process of transferring  appeals to the Board. The second introduced the ability for staff to access  documentation from the Veterans Benefits Management System  (VBMS)—VA’s system for processing claims—which VA believes will  eventually allow users to review appeals more efficiently. As of February  2017, VA officials also noted the agency is in the process of developing  additional components, including document review software for VLJs and  attorneys, and a component to better track appeals that are remanded to  VBA. According to officials, VA’s longer term plans include a broad  roadmap for continuously adding improvements to Caseflow. For  instance, VA has plans to build into the system the capability to generate  performance metrics, using a component called Caseflow Dashboard. VA  states that the dashboard will be able to draw on various VA data  systems and provide information on bottlenecks in the appeals process,  quantify improvements in the appeals process—including those  attributable to improved IT systems—and track the reasons for and  number of remands. While Caseflow improvements are being made, VA  reported it plans to maintain VACOLS as a redundant resource until the  new system is fully complete, at which point VACOLS will be retired.
section_title: VA Applied Several Sound Planning Practices in Its Efforts to Improve Its Appeals Processing Timeliness, but Fell Short in Other Areas
subsections: section_title: VA Weighed Several Factors When Determining Its Need for More Staff, but Did Not Fully Assess Risks or Document Its Workforce Plans
paragraphs: VA acted consistently with sound planning practices in determining its  need for additional staff, but it did not fully consider risks and  uncertainties in its approach. Sound practices for effective planning  suggest that agencies should consider alternative solutions to a problem;  assess the risk of unintended consequences; and use data to analyze the  problem, including unknowns. Consistent with these concepts and more  specific sound workforce planning practices, VA considered various hiring  options, such as hiring staff under the current versus VA proposed  process, and modeling appeals inventories under four hiring surge  options. VA considered a number of factors when comparing the four  hiring options including historical data on the volume and complexity of  appeals, estimates of future growth in appeals, and the productivity of  employees in estimating the number of Board staff needed to meet its  timeliness goals. For instance, the Board reviewed past data on the productivity of new staff—which is generally lower for a period of time  until individuals acclimate to their jobs—and factored this into the  modeling assumptions used to project the number of Board staff needed.
More specifically, sound workforce planning practices suggest that  agencies identify the resources needed to manage the risks of  implementing new processes and conduct scenario planning to determine  these needs. While VA considered a number of factors when analyzing  hiring options, it initially made many assumptions using a single set of  estimates instead of using a sensitivity analysis to consider a range of  estimates. These assumptions could have significant implications for  how accurately VA identifies needed resources. For example, in its  scenario analysis VA assumed: (1) that an average of 50 percent of those  veterans appealing will refile their appeal and go through two of the four  appeals process options before being satisfied; and (2) that the Board will  be able to decide 130 appeals per FTE, and do so within 3 years (1,095  days), for appeals with hearings and decide 180 appeals per FTE within 1  year for appeals without hearings. Because the Board did not consider  alternate sets of assumptions, VA does not know the potential effect that  variations in these key variables could have on staffing needs.
In response to discussions with us on its scenario analyses, VA recently  conducted further analyses using alternative estimates for key factors,  although the agency’s analyses fell short of the previously discussed  sound practices for estimating outcomes based on assumptions.  Specifically, VA calculated the effect on appeals inventories and  timeliness if VA decided 20 percent fewer appeals, if VA decided more  claims and thus had more appeals than expected, and if the breakdown  of options that veterans selected for their appeals review is different than  the 50/50 split VA projected. The 20 percent reduction in productivity  alone could add 2.5 years to VA’s estimate of how long it would take to  clear the appeals inventory under hiring surge option 4. However, VA ran  a sensitivity analysis for only one of the four hiring surge options and did  not analyze the compounded effect of different assumptions together. By  not comprehensively conducting sensitivity analyses, VA is hampered in  its ability to anticipate and plan for different contingencies, and risks being  caught off guard and potentially hiring an inappropriate number of staff.  Hiring too few staff could result in it taking longer to reduce the inventory  of pending appeals, while hiring too many staff could result in higher  expenditures than needed and exacerbate other challenges, such as  ensuring sufficient office space, training, and other supports for newly  hired staff, as discussed below. VA has acknowledged that some of its  assumptions, and thus projections, are based on unknowns and that it will  need to continuously rerun the models with updated data.
VA also identified strategies and resources needed for recruiting, hiring,  and training staff in fiscal year 2017; however, aspects of VA’s workforce  planning fall short of sound workforce planning practices that suggest  having timely written plans with a systematic approach and detailed  steps, time frames, and mitigation strategies to help identify where  resources and investments should be targeted. As noted below, VA has  identified strategies and taken some positive steps related to recruiting,  hiring, and training staff in fiscal year 2017, although these plans  sometimes lacked certain details specifically covered in sound workforce  planning principles in time to inform ongoing efforts.
Recruitment and Hiring: Consistent with sound workforce planning  practices, officials have worked to develop a center for excellence in  hiring to coordinate workforce planning and develop strategies for  recruiting and hiring staff quickly. However, the center was  established in the last quarter of fiscal year 2016 to support hiring  beginning early in fiscal year 2017. To date, the Board developed a  project to recruit recent law school graduates and alumni in fiscal year  2017, according to Board officials. It also has formed a committee of  over 90 volunteers to develop recruitment materials, identify  opportunities, and make contact with law schools; developed a  PowerPoint presentation for the visits; and conducted a few initial  presentations at law schools. However, the agency had not yet  worked out specific goals such as the number of presentations or  resulting applications, average time taken to recruit, and skills needed  in recruits, or identified metrics (other than hiring goals) against which  it would measure the effectiveness of the recruitment efforts.
Also consistent with sound workforce planning practices, Board  officials told us that they considered lessons learned from a 2013  hiring surge, although the agency did not provide documentation of  these lessons learned. Having established a goal of hiring 25-52  new employees per month from October 2016 through April 2017, the  Board subsequently faced challenges finding space for staff coming  aboard in fiscal year 2017. Specifically, as of October 2016, the  Board was reconfiguring its office space to accommodate the planned  242 new FTEs in fiscal year 2017, employing nearly all of its  conference rooms, and developing a plan for using telework and office  sharing to accommodate staff until the space is available for them,  according to officials.
Training: As of February 2017, VA rolled out training for newly hired  attorneys in fiscal year 2017, which includes 4 weeks of training and 8  additional weeks of one-on-one mentoring. VA also stated that its  Office of Knowledge Management was expanded with additional staff  resources to ensure training materials were up to date. However, in  November 2016, officials reported that the Board was still in the  process of updating various aspects of its training curriculum, such as  how to support conducting work in a virtual environment, which is  consistent with the agency’s plans to increase telework as a way to  manage space restrictions for new staff. In its comments to this report,  VA did not provide updates on this effort, as of February 2017.
As of October 2016, it was unclear how the Board’s 2017 recruiting,  hiring, and training efforts will be adjusted to support the agency’s  proposed hiring surges in 2018 or its proposed process reform. For  example, the Board has not yet determined how it will meet the space  needs for any additional growth associated with hiring surges proposed  for fiscal year 2018, although more detailed planning in advance might  have better prepared VA for bringing aboard 242 FTEs in fiscal year  2017. In addition, VA officials stated in February 2017 that draft training  for the proposed new appeals process had been prepared based on  statutory language, although these draft documents were not included in  VA’s comments. Federal strategic planning guidance calls for an agency  to have clear plans and goals, and regularly assess its human capital  approaches through assessments, as well as through data-driven human  capital management to improve its ability to maximize the value of human  capital investments while managing related risks. Conversely, a lack of  detailed workforce plans and mitigation strategies prior to proposed hiring  surges in 2018, as well as potential process reform, further places VA at  risk of not being ready to accommodate another quick and much larger  increase in staff, or to train them in accordance with either the legacy or  proposed reform process.
section_title: VA Collaborated with Stakeholders in Developing Its Proposed Streamlined Appeals Framework, but Lacks Strategies, Such as Pilot Testing, to Mitigate Risks
paragraphs: VA collaborated with key stakeholders in developing its proposed appeals  process reform framework and related implementation plans, which is  consistent with sound practices for business process redesign. Sound  redesign practices suggest coordinating with stakeholders in developing  and implementing plans to obtain and maintain buy-in from start to finish  and to identify and address disagreements. In developing its proposal,  Board and VBA officials engaged stakeholders from 11 organizations— including VSOs that represent veterans in appeals hearings before VBA  and the Board—in discussions to design a streamlined appeals process.  Officials we interviewed from three of four VSOs, all of whom participated  in the discussions, noted that VA’s resulting process proposal addressed  both the agency’s desire to expedite appeals resolutions and stakeholder  desires that the new process be fair to veterans. For example, VA  identified and prioritized key concerns and found areas of consensus with  VSOs. VA officials stated that they plan to continue to discuss appeals  process reform (among other topics) at regular meetings with  stakeholders, during which they will have an opportunity to provide  feedback on previously unforeseen issues. VA officials said that as  process reform is implemented, the agency will invite local VSOs to  training, and share training materials and provide briefings to them and  other stakeholder groups.
While VA has achieved broad agreement internally and with VSOs on its  proposed process reform, there are several unaddressed gaps in VA’s  business case for implementation that introduce the risk of not producing  the desired results, as follows.
To develop a business case for implementing process change, sound  redesign practices suggest first mapping and analyzing the target  process to understand the cause and cost of performance  breakdowns, and assessing potential barriers, costs, and benefits of  alternative processes. This, in turn, would inform the selection of a  feasible alternative with a high return on investment, and the  development of a business case that describes benefits, costs, and  risks. However, due to IT limitations, VA lacks data to inform and  confirm its understanding of the root causes of lengthy time frames.  For example, VA lacks complete historical data on the extent to which  submission of new evidence and multiple decisions and appeals  occur, and thus cannot determine the impact of its current, open- ended process on appeals decision timeliness. To shed light on root  causes, VA analyzed 10 appeals decisions that it found took a long  time to adjudicate to illustrate extreme examples of cases being re- reviewed under VA’s open-ended process—referred to by VA as  “churning.” However, VA cannot know the full extent to which  churning might be occurring because, according to VA officials, the  way data are stored made it difficult, if not impossible, to assemble a  complete historical picture prior to December 2012. To help develop  baseline data, VA analyzed the average number of decisions per  appeals phase for several recent fiscal years, and, according to VA  officials, they are still endeavoring to piece together additional  historical baselines for performance.
Further, although it was appropriate for VA to develop its proposed  reform in consultation with internal and external experts, the agency  did not test alternatives using data-driven, cost-effective methods  suggested by sound redesign practices.
Finally, as noted previously, in modeling staff resources needed under  its proposed process reform, VA relied on assumptions—about the  percent of veterans who will refile, will appeal to the Board, and will  submit new evidence—that have direct implications for projections of  appeals workloads, time frames, and cost. However, VA did not  perform sufficient sensitivity analyses to help estimate a range of  potential outcomes—analyses that might help VA understand the  likelihood that the new process could be more costly and time- consuming in practice than anticipated, for example, if a higher  percent of veterans file with the Board, submit new evidence, and  request hearings than expected.
These gaps notwithstanding, VA made some progress planning for  potential implementation of proposed process reform in a manner  generally consistent with sound planning practices for process redesign  and change management, although some important details are still  absent. According to sound planning practices, implementation is the  most difficult phase of business process redesign. An agency must  manage human capital and technical issues as it turns an idea into reality  and overcomes potential resistance to change. To ensure an orderly  transition, sound planning practices suggest following a comprehensive  implementation plan that includes several key activities, such as  establishing a transition team and developing a comprehensive plan to  manage implementation. Consistent with this, as of October 2016, the  Board and VBA had identified general time frames and offices  responsible for key implementation efforts. Based on its staff modeling  efforts, VA also identified how many FTEs it expects it will devote to  processing cases under the current process versus a new one, should it  be implemented. Also, per sound planning practices, a comprehensive  plan should address workforce training and redeployment issues  (including working closely with employee unions to minimize potential  adverse effects). Consistent with this, as of October 2016, VA had  outlined general steps and time frames for training of staff and  communicating with the unions.
While VA’s high-level implementation plan included many components  suggested by sound practices, key details had yet to be addressed. In  particular, VA’s general timetables and plans to date have not addressed  in any detail how it will implement a new process while simultaneously  working to reduce the appeals inventory under the current process. For  example, the agency has not explained how or who will track timeliness of  appeals of the old compared to the new process, and how decisions will  be made to ensure the agency is devoting an appropriate share of  resources to both processes. The lack of a detailed plan for managing  this transition exposes the agency to risk that veterans whose appeals  are pending under the old process may experience significant delays  relative to those under a new process. The Board recognized the need to  ensure fairness to veterans with appeals pending under the current  process, and indicated that while legislation is pending that would  authorize a new process, it will continue to develop plans for managing  the two processes in parallel.
Sound practices for process redesign and change management also  suggest having risk mitigation strategies—in particular, pilot testing—to  ensure moving successfully to full implementation. Pilot testing provides  agencies opportunities to evaluate the soundness of new processes in  actual practice on a smaller scale, and to refine performance measures,  collect and share implementation problems and solutions, correct  problems with the new design and refine the process prior to full  implementation, and build capacity among unit managers to lead change.  Sound redesign and change management practices both suggest that  pilot tests should be rigorously monitored and evaluated, and that further  roll-out occur only after the agency’s transition team has taken any  needed corrective action and determined that the new process is  achieving previously identified success criteria. As noted above, pilot  testing is not the only method of achieving these risk mitigation goals, but  sound planning practices suggest pilot testing is an important, often  necessary approach for ensuring successful implementation when  undertaking significant institutional change.
Contrary to sound practices, VA officials stated they do not want to pilot  proposed appeals reform, even though VA’s proposed reform can be  considered complex. VA’s reform plans qualify as complex because in  addition to implementing a new process, the agency must still manage a  large inventory of appeals under the old process while hiring and training  a large number of staff and implementing IT improvements. Occurring  together, these efforts involve significant change and uncertainty and will  require management oversight across a broad range of efforts. In  addition, VA’s proposed process reform and other initiatives affect VBA’s  regional offices spread across the country, as well as the centrally located  Board, thereby further increasing complexity of implementation. VA  officials also stated that the proposed process reform, which has been  thoroughly vetted with stakeholders, has broad support, and noted their  view that the risk of fully implementing change is outweighed by the cost  of delay. VA’s rationale for not pilot testing centers on what they describe  as widespread consensus that the current process is “fundamentally  broken” and provides “inadequate service to veterans with a high  percentage of wasted effort.” VA assumed that a pilot test authorized by  Congress would include a sunset date with a default reversion to the  current system, which they said would introduce uncertainty into the  agency’s planning efforts and a reliance on subsequent, time consuming  legislation before the conclusion of the pilot. VA officials stated that  piloting with a sunset date would require the agency to expend additional  resources and time to conduct parallel planning for reverting to the old  system upon the sunset date. VA stated that pilot testing the new process  for some veterans would be perceived as inequitable, despite VA having  previously supported pilot testing a new appeals process. VA officials  concluded that they have not identified any risk that would justify a pilot,  and indicated that they plan to mitigate risk with a strong implementation  plan.
While VA has made a compelling case for reforming the appeals process,  as noted previously, VA’s business case for its proposed reform in some  instances relies on unproven assumptions and limited analyses of its  current process that introduces risk in VA’s plans for full implementation.  Importantly, VA assumes that because the current framework is  “fundamentally broken,” its proposed new framework will necessarily be a  better option. However, VA made this decision lacking complete data on  the root cause of lengthy appeals under the current process, and without  analyzing barriers, costs and benefits of feasible alternatives using cost  effective methods, such as computer simulations. VA correctly notes that  pilot testing prior to full implementation would slow down an overhaul of  the current system, thus countering the short-term net benefit that the  agency expects to realize from such an overhaul. However, VA has not  acknowledged that pilot testing the new process in a more limited fashion  could greatly increase the probability of long-term success by decreasing  the chance that a new system will experience unanticipated problems that  are potentially more widespread and therefore costly to remedy. The  inclusion of risk mitigation strategies such as pilot testing does not, as VA  asserts, “imply that the status quo is not in dire need of sweeping reform”  but rather balances the urgency of the current problem, the technical  complexity of an overhaul, and the potential for unforeseeable  complications.
In light of this and the previously discussed inconsistencies in following  sound planning practices, pushing forward with full implementation  without testing how process reform unfolds and interacts with other efforts  in actuality, may lead VA to experience implementation challenges and  setbacks that could undermine efficiencies and other outcomes resulting  from planned reforms. In contrast, if VA were to pilot test the proposed  appeals process reform, implementation problems encountered could be  identified and resolved prior to full implementation. This could lead to  smoother implementation and better outcomes overall. Further, resources  that would otherwise be diverted to full implementation of process reform  across the organization could be focused on its current inventory of  appeals. VA will also have additional time and managerial capacity to  recruit and train new staff and develop and implement a communication  and outreach strategy in time for full implementation of the new process.  Finally, if risk mitigation strategies demonstrated that process reform  would be more costly and detrimental to time frames and workloads than  predicted, a decision to modify or fix the process at that juncture would be  made with more information and less impact on the agency overall.
Whether VA conducts pilot testing or not, VA has not yet developed a  plan for closely monitoring implementation or developed a strategy for  assessing the success of its proposed process reform. Sound planning  and redesign practices suggest that the transition team develop metrics  and data gathering procedures, define success criteria, measure  performance carefully, and take corrective action of any pilot test before  proceeding to full implementation. Sound practices also suggest the  agency develop meaningful performance measures—generally a mix of  outcome, output, and efficiency measures—tied to overall goals of the  project, and that project goals include a mix of intermediate goals to be  met at various stages during the implementation phase. That way, the  agency can start to show a return on investment in the early stages of  implementation. To date, VA has identified several broad metrics  generally reflecting outcomes, output, and efficiency—such as veteran  survey results, wait times, and inventories—that it plans to use to track  and assess process improvements. VA also established separate  timeliness measures for the Board and VBA that it will use in its annual  performance reports. While these broad metrics and goals are  appropriate, they fall short of sound practices for monitoring and  assessing process change in several respects.
First, VA has not developed a dashboard or balanced scorecard, or  otherwise identified how it will closely monitor progress, evaluate its  efficiency and effectiveness, and identify trouble spots. For example,  although VA has stated that it is developing a dashboard to measure  performance under its proposed appeals process, VA has not yet  indicated whether, how, and with what frequency it will monitor wait times  and inventories under the new versus current processes. As a result, it is  not clear how VA will determine whether veterans with appeals pending  under the current process are receiving equitable treatment and not  experiencing significant delays relative to those under the new process. It  is also unclear the extent to which VA will systematically monitor staff  productivity and IT processing, which may affect its ability to determine  whether assumptions are being met to help pinpoint corrective action  (e.g., whether staff need more training, VA’s communication and outreach  efforts are working as expected, or process reform itself is achieving  desired results). Further, VA has not established interim goals or criteria  for success to help determine whether initial implementation is achieving  intended results. Interim goals and criteria could include specific  timeliness improvements for process steps and outcomes, such as  average time for VBA or the Board to reach decisions under new appeals  options. If VA pursues pilot testing, such goals or success criteria will help determine whether the new process is sufficiently successful to justify full  implementation.
Second, although streamlining the current open-ended process was  central to VA’s business case for its proposed process reform, as noted  previously, VA currently lacks sufficient data to assess the extent to which  process reform will improve on the open-ended nature of the current  process. VA officials said that they plan to work to incorporate capabilities  into Caseflow to piece together historical baselines for performance. VA  also plans to develop new baseline and historical data on aspects of the  appeals process that affect the timeliness of final decisions so that they  can be compared to the new process. While these are positive steps, it  remains to be determined how or whether VA will be able to measure the  extent to which its proposed process—which would allow the veteran to  appeal multiple times—is an improvement over “churning” associated with  the old process.
Lastly, the new timeliness measures that VA plans to report to Congress  and the public lack transparency on whether overall appeals resolution  timeliness is improving from the veterans’ perspective. In its fiscal year  2015 performance report, VA stopped reporting its average appeals  resolution time measure, which included appeals decisions made by both  VBA and the Board. VA officials said they considered this measure  inadequate because neither VBA nor the Board has full control over  making improvements to performance under this measure. VA officials  told us the measure does not appropriately provide insight into the  appeals process because of the variety of appeals paths and wait times  veterans experience. However, the combined measure would provide a  basis for comparing timeliness under the old versus new process, and  would provide historical perspectives on changes in timeliness from the  point of view of a veteran who may file appeals with both VBA and the  Board before his or her case is resolved. VA officials stated the agency  will continue to track this measure internally, but they will not include it in  VA’s annual performance reports. Instead, they plan to report on VBA and  Board timeliness separately. VA also stated that it will not use this  measure to evaluate success of the new process because it considers a  timeliness measure covering both VBA and the Board to be inappropriate.
section_title: VA Has Taken Steps Consistent With Sound Planning Practices while Updating Its IT, but Lacks Clear Schedules for Implementing IT Updates
paragraphs: VA has generally planned the implementation of its Caseflow appeals  system consistent with sound planning practices. Working with U.S.  Digital Service at VA (DSVA)—the group tasked with developing  Caseflow—VA outlined an approach that has a clear scope and purpose,  which is to better process appeals in a paperless environment and  improve automation and productivity. The actions consistent with sound  IT planning practices include:
Setting goals and objectives: VA plans clearly lay out the need for  replacing VACOLS and set forth how Caseflow will address the  shortcomings of VA’s current IT system. Its plans also lay out a set of  broad milestones in terms of the capabilities that will be added to  Caseflow in the future and the ultimate retirement of VACOLS.
Identifying and mitigating potential risks: VA planning documents  identify a number of risks (such as staffing shortfalls and technical  delays) and strategies to mitigate them. In addition, VA is developing  Caseflow in an agile process, which officials say will allow VA to  continually add new capabilities and be responsive to changing  agency needs. In addition, VA officials told us that rather than replace  VACOLS at once, the various functions in VACOLS will be  reproduced and tested in Caseflow iteratively, and each  corresponding function in VACOLS will be left intact until there is  reasonable assurance that there will be no impact to VA.
Measuring performance: VA plans to develop metrics for each new  component of Caseflow that is implemented. For instance, VA has  developed metrics for the two components that were developed in  fiscal year 2016—electronic transfer of cases to the Board and a  system to electronically access documents from VBMS—which  specifically assess the performance and effect of those components.  As mentioned earlier, VA also plans to create a Caseflow dashboard  that will provide metrics on the effect of IT improvements on  timeliness of the appeals process.
Identifying organizational roles and responsibilities: VA entered  into a memorandum of understanding with DSVA and the Board that  outlines priorities, and a working relationship for developing Caseflow.  In addition, the memorandum states that DSVA requires all initiative  partners within VA to have a single point of contact with the authority  to make decisions on behalf of their component.
While VA’s plans for replacing VACOLS take steps to mitigate risks, they  currently do not include consideration of the timing and implications of  VA’s proposed reform efforts. Federal internal control standards state that  program managers, in seeking to achieve program objectives, should  define objectives clearly to enable the identification of risks. This includes  clearly defining what is to be achieved and the time frames for  achievement. Additionally, IT investment best practices stress the need  for oversight regarding a project’s progress towards predefined schedule  expectations. This oversight also includes systems to make corrections  regarding schedule and performance slippages. Although VA has laid out  the broad capabilities it would like to incorporate into Caseflow going  forward, VA has not developed a schedule for completing Caseflow.  Specifically, VA could not provide us with firm time frames for when  different capabilities will be active in Caseflow. As the Caseflow effort  lacks time frames, VA cannot ensure that the system will be completed in  time to support the implementation of proposed reforms.
Further, VA’s lack of time frames for developing Caseflow may increase  the risk of additional costs if the system cannot be developed as quickly  as anticipated. Sound practices specific to project scheduling state that  project planning is the basis for controlling and managing project  performance, including managing the relationship between cost and  time. In a prior GAO report on VBMS development—which was also  developed in an agile process—we reported that the agency encountered  some delays with its initial deployment of key functions of VBMS, and that  its lack of a schedule made it difficult to hold program managers  accountable for meeting time frames and demonstrating progress.
In addition, VA has not started planning and determining the changes that  would be needed for Caseflow if and when appeals reforms are  implemented. VA staff said that the agile approach they are using allows  them to quickly respond to changing needs, and VA Office of Information  and Technology officials told us that they will not begin planning for such  changes until reform legislation is passed. As stated earlier, sound IT  planning practices suggest that implementation plans include specific  time frames and approaches needed to implement new systems, as well  as the consideration of potential risks and mitigation strategies. As such,  and given the absence of a schedule for completing Caseflow, VA further  risks having an IT system that is not completed in a timely manner or,  even if in place in time, falls short of meeting VA’s needs.
section_title: Conclusions
paragraphs: With an already large inventory of pending appeals—and expectations of  further growth—VA has taken steps to bolster capacity and improve the  efficiency and effectiveness of its disability appeals process. Specifically,  VA hired and proposed hiring more staff, is moving forward with plans to  upgrade its IT systems, and has proposed bold reforms to streamline its  appeals process. In planning and executing these approaches, VA took  several positive steps in line with sound planning practices—such as  comparing different options for increasing future staffing resources,  collaborating with external stakeholders to develop a streamlined process  proposal, and outlining a vision for upgrading outdated IT systems.
Nonetheless, VA’s plans do not account for the significant challenges that  remain. Above all, its proposal to implement appeals reform at the Board  and across VBA’s regional offices is ambitious, and as a result, VA may  be exposing itself to unforeseen risks and setbacks that could slow  progress toward improving appeals decision timeliness. More specifically,  VA has proposed implementing process reform while also hiring more  staff and upgrading its IT, which are challenging efforts in their own right.  Additionally, VA does not have any plans to pilot test its proposal—a  sound and often necessary practice for experiencing, evaluating, and  refining significant institutional change on a smaller scale prior to full  implementation. At the same time, VA plans for hiring more staff and  upgrading IT lack key details (for example, on how VA will train and find  working space for new staff, or a schedule for when and how system  changes might be integrated with the proposed streamlined process),  exposing VA to risks of delays, inefficiencies, or other setbacks caused by  not anticipating needs or a misalignment of efforts. VA also did not  sufficiently apply sensitivity analysis when projecting staffing needs with  or without process reform, which could affect the agency’s ability to  mitigate any potential risks if assumptions are not met. Lastly, VA lacks a  robust monitoring plan to help assure that unforeseen problems will be  quickly and effectively addressed, and has not yet developed a strategy  with appropriate interim goals for process reform, and overall goals for  appeals process timeliness, to gauge whether the agency’s efforts are  having the desired result and reflect an improvement over prior practices.  Until VA incorporates these sound planning practices, the agency lacks  reasonable assurance that its proposed reform will improve the overall  efficiency of the appeals process and timeliness of disability appeals  decisions.
section_title: Matter for Congressional Consideration
paragraphs: To improve VA’s ability to successfully implement appeals process  reform, Congress should consider requiring that reforms of the VA  disability appeals process be subject to a pilot test. To aid in the  development of such a pilot test, Congress could require the Secretary of  Veterans Affairs to propose options that would allow the agency the  flexibility to test and refine the new process in a cost-effective and  efficient manner, while ensuring pre-established interim goals and  success criteria are being met prior to full implementation.
section_title: Recommendations for Executive Action
paragraphs: To further align efforts to address appeals workload and improve  timeliness of decisions, and reduce the risk that efforts will not go as  planned, we recommend that the Secretary of Veterans Affairs direct the  Under Secretary for Benefits; the Chairman, Board of Veterans’ Appeals;  and the Chief Information Officer, as appropriate, to:  1.  Ensure development of a timely, detailed workforce plan for recruiting,  hiring and training new hires. In particular, this plan should: (1) include  detailed steps and timetables for updating training curriculum (such as  preparing decisions in a virtual environment) and ensuring office  space (such as telework guidance); and (2) incorporate risk mitigation  strategies that consider how the timing of recruitment and training  dovetails with uncertain time frames for implementing a new appeals  process.  2.  Develop a schedule for IT updates that explicitly addresses when and  how any process reform will be integrated into new systems and when  Caseflow will be ready to support a potential streamlined appeals  process at its onset.  3.  Conduct additional sensitivity analyses based on the assumptions  used in projection models to more accurately estimate future appeals  inventories and timeliness. In doing so, consider running additional  analyses on how these factors, in conjunction with one another, may  affect the timeliness and cost of deciding pending appeals.  4.  Develop a more robust plan for closely monitoring implementation of  process reform that includes metrics and interim goals to help track  progress, evaluate efficiency and effectiveness, and identify trouble  spots.
To better understand whether appeals process reform, in conjunction with  other efforts, has improved timeliness, we recommend the Secretary of  Veterans Affairs direct the Under Secretary for Benefits; the Chairman,  Board of Veterans’ Appeals; and the Chief Information Officer, as  appropriate to:  5.  Develop a strategy for assessing process reform—relative to the  current process—that ensures transparency in reporting to Congress  and the public on the extent to which VA is improving veterans’  experiences with its disability appeals process.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of the report to VA for its review. In written  comments, VA disagreed with one of our recommendations and agreed in  principle with the other five. We have reproduced VA’s comments in  appendix I and have incorporated them—as well as technical comments  provided by VA—into the report, as appropriate.
In its comments, VA agreed with us that improving the efficiency and  effectiveness of its appeals process is an ambitious undertaking, and we  commend VA for the many steps it has taken, including collaborating with  stakeholders to develop the framework for a new process. We agree that  obtaining the consensus of internal and external experts—including  veterans service organizations—demonstrates important progress. We  disagree, however, that such consensus negates the need for more  detailed plans and robust risk mitigation strategies. While it is true that VA  has made noteworthy progress developing an implementation plan to  guide its efforts, we found the plan lacked important details, such as: how  VA will monitor for interim success and trouble spots, including whether  the agency has appropriately distributed resources among the new and  old processes; how it will mitigate risk of implementation challenges or  setbacks, and reduce their negative impact; and how it will measure  whether the new process is improving overall appeals resolution  timeliness from the veteran’s perspective.
VA officials also said that VA has extensive experience in organizational  change management, but it is not clear how some of the practices VA  used in past transformation efforts are applicable to appeals reform, and  we are concerned that VA could not provide further information on what  these practices include or how they are relevant. We believe  implementing all of our recommendations will increase the likelihood that  VA’s efforts to improve the efficiency and effectiveness of its appeals  process will be successful. For the five recommendations that VA  concurred with in principle, VA described planned actions to address  them and stated that it also considered the actions complete and  requested we close the recommendations. However, we believe VA still  needs to take actions to address those recommendations, as noted more  fully below.
VA disagreed with a draft recommendation that it incorporate pilot testing  of its proposed appeals process into implementation plans and pursue  necessary legislative authority. In its comments, VA noted that the  appeals process is broken and that piloting a new process would result in  further delays to veterans appealing their disability decisions. VA  disagreed with GAO’s finding that it had proposed the new process  without analyzing feasible alternatives, noting that the agency designed  the new process based on the collective experience of internal and  external experts, and that these experts reached consensus on a new  design that will be beneficial to veterans, the agency, and taxpayers,  among others. VA noted that it has carefully assessed risks, identified a  number of risk mitigation strategies, modeled a number of different  scenarios, and developed a detailed implementation plan.
When we reviewed these efforts, however, we found three primary  shortcomings that need to be addressed. First, VA did not have the data it  needs to fully understand the extent to which the current process has  contributed to lengthy appeals time frames, which raises questions about  whether the proposed process will address the root cause or causes of  untimely appeals decisions. Specifically, VA lacks historical data on the  extent to which the introduction of new evidence increases time frames.  Second, VA’s list of potential risks and risk mitigation strategies did not  always include steps for mitigating the identified risks. Third, we found  that VA’s implementation plans lacked details on how it will carry out key  aspects of appeals reform, including how it will monitor the timeliness of  appeals decisions under the old process compared to the new appeals  process, while also hiring additional staff and integrating changes into the  Caseflow IT system as discussed below. In VA’s comments, it also stated  that piloting a new appeals process “would raise constitutional issues and  prompt litigation.” We acknowledge that changing an adjudicatory process  for determinations of benefits may prompt litigation. However, VA has not  clearly articulated why pilot testing as a category is unconstitutional or  why pilot testing poses unique constitutional issues. Further, as noted in  the report and in VA’s comments, VA previously supported H.R. 800 in  the 114th Congress, which would have directed VA to conduct an opt-in  pilot process where a veteran could present a limited amount of new  evidence and the Board, to the extent practical, would decide cases  within one year. While GAO did not take a position on that bill, or its  specific approach to pilot testing, changes of this magnitude in such a  complex program justify some form of pilot testing to ensure process  reform is implemented successfully and ultimately achieves VA’s goals.  As noted in the report, pilot testing is recognized to be a sound planning  practice and an important, often necessary approach for ensuring  successful implementation when undertaking significant institutional  changes.
Until VA pilot tests its appeals reforms, it will lack data to properly plan for  and overcome the challenges that will likely arise during implementation.  For example, VA may encounter difficulties making needed process  changes while simultaneously implementing other logistical requirements,  such as hiring and training new staff and updating its IT system. By not  pilot testing, VA is missing a valuable opportunity to refine its  implementation strategy by first seeing how process reform will unfold on  a smaller scale. We believe that the potentially negative consequences of  delaying full implementation are far outweighed by the benefits that can  be realized through piloting. For example, piloting could help avoid delays  and expenses caused by the need to re-work the process after full scale  implementation. In light of VA’s disagreement with our draft  recommendation, we removed the recommendation and now pose a  matter for congressional consideration. Specifically, to improve VA’s  ability to successfully implement appeals process reform, Congress  should consider requiring that reforms of the VA disability appeals  process be subject to a pilot test. To aid in the development of such a  pilot test, Congress could require the Secretary of Veterans Affairs to  propose options that would allow the agency the flexibility to test and  refine the new process in a cost-effective and efficient manner, while  ensuring pre-established interim goals and success criteria are being met  prior to full implementation.
VA concurred in principle with the draft recommendation that it finalize a  detailed workforce plan that includes steps for training, support, and risk  mitigation strategies. VA noted that, in addition to currently implementing  a fiscal year 2017 workforce plan to hire additional staff, as discussed in  the report, among other efforts it has recently launched new attorney  training and continues to collaborate across the agency to identify space  where new staff can be located. We have incorporated these updates into  our report, as appropriate. In light of these efforts, and because future  steps, such as developing training materials on the new appeals process,  are contingent upon appeals reform legislation, VA stated it considers this  recommendation complete and requested closure. While we recognize  that VA has made progress and that certain actions, such as training on a  new process, is contingent upon reform legislation, we disagree that the  recommendation should be closed. As noted in our report, we found that  VA’s final recruiting, hiring, and training plans lacked important details.  For example, VA officials were still updating training curriculum that  supports work conducted in a virtual environment and which is critical for  managing space restrictions for new staff. Without a detailed workforce  plan in place, VA cannot assess the success of its human capital  approach, maximize its investments, or fully mitigate risks. More detailed  workforce plans would help VA avoid the risks that staff will not be hired  in time, not be properly trained, or not have the support necessary to  process appeals. Waiting until legislation is enacted magnifies these  risks. We believe additional action is needed to meet the intent of this  recommendation; we also clarified the recommendation language to state  that VA needs a more detailed plan.
VA concurred in principle with our recommendation that it develop a  schedule for IT updates that lays out when and how any process reforms  will be integrated into its Caseflow system. More specifically, VA noted  that it will rely on the agile process to develop Caseflow—whereby new  functions are continually added to the system as new user needs or policy  changes arise—and does not plan to define schedules beyond 6 months.  Given that Caseflow development related to the new appeals process is  dependent on the enactment of new legislation, VA stated it considers  this recommendation to be complete. While it is true that the agile  process can help mitigate risks and avoid cost overruns and delays, we  do not believe this approach precludes VA from taking additional steps to  consider the scope of potential changes required by a new appeals  process and have a broad plan in place to ensure that all aspects of the  new process are adequately supported by Caseflow. We believe it is  especially important for VA to have specific time frames for completing  Caseflow considering the scope of the changes being proposed.  Moreover, VA noted that components of Caseflow developed so far will  not need to be significantly changed because of appeals reform  legislation being enacted, but VA did not provide documentation to  support this assertion. In light of these issues, we believe VA has not yet  met the intent of this recommendation.
VA concurred in principle with our recommendation that it should conduct  additional sensitivity analyses around the assumptions used in its models.  VA noted that sensitivity analyses are valuable and that it has focused its  efforts on risks its staff identified as most likely, such as variations in  staffing and productivity, and the effect of remands. VA stated it would  continue to analyze, update, and refine its modeling, and considers this  recommendation to be complete. While we recognize the logic of focusing  modeling resources on key variables, VA did not fully examine three of  the four hiring surge options it proposed. Moreover, VA did not assess the  compound effect that would result in changing multiple assumptions at  once. Given the complexity of proposed changes and the number of  variables beyond VA’s control, we believe that additional analyses are  needed to identify potential risks that may warrant additional mitigation  strategies. In addition, if VA goes forward with appeals process reform  and begins to collect real-time data, these data could improve modeling  accuracy and serve as a valuable management tool.
VA concurred in principle with our recommendation to develop a more  robust plan for closely monitoring the implementation of its process  reform, that includes metrics and interim goals to help VA track progress,  evaluate efficiency and effectiveness, and pinpoint trouble spots. VA  agreed that developing such a plan is valuable for monitoring the  implementation of process reform, and should include metrics and interim  goals. However, VA stated that it considers this recommendation  complete, noting that preparing such a detailed plan depends on appeals  reform legislation being enacted, and that it will incorporate specific goals  and metrics as it moves towards implementation. While we recognize that  VA cannot assume to know the exact provisions that may be included in  future enacted legislation, nor can it predict when appeals reform might  be enacted, we consider having a more robust monitoring plan to be  essential to the successful implementation of a new appeals process.  Moreover, the absence of such a plan raises questions as to how VA will  ensure appropriate resources are devoted to managing appeals under the  new versus old process, or intended results are achieved as the new  process is implemented.
VA concurred in principle with our recommendation to develop a strategy  to transparently report to Congress and the public on veterans’  experiences with the new appeals process. VA noted that it is already  developing timeliness goals for three of the four appeal options in the  proposed new process, as discussed in our report. VA said it also plans  to measure success of the new process with results from customer  satisfaction surveys and is developing a dashboard for internal  performance monitoring. VA did not agree that measuring overall appeals  resolution timeliness is an appropriate a measure and believes tracking  time frames for each of the options separately is more appropriate. While  we agree that metrics based on the different options could be valuable for  VA, the Congress, and the public, we disagree that VA’s focus on  measuring timeliness by option is in the best interest of the veteran.  Because veterans may pursue more than one option under VBA, the  Board or both, we believe that VA’s approach does not take into account  the veteran’s perspective of how long it took for them to receive a final  appeal decision. Metrics from the veterans’ overall perspective would  complement, not replace, metrics for VBA, the Board, and each option.  Further, because VA’s approach does not allow VA to compare the new  process with the old or to determine whether the new process represents  an improvement over the old process, we believe it does not promote  transparency in reporting to the Congress and the public.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Veterans Affairs, the Under Secretary for  Benefits, the Chairman of the Board of Veterans Appeals, and the VA’s  Chief Information Officer. In addition, the report is available at no charge  on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or bertonid@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix II.
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Michele Grgich (Assistant  Director), Melissa Jaynes (Analyst-in-Charge), Daniel Concepcion, and  Greg Whitney made key contributions to this report. Other key  contributors to this report include James Bennett, Mark Bird, David  Chrisinger, Clifton Douglas, Alex Galuten, Mitch Karpman, Sheila R.  McCoy, Claudine Pauselli, Almeta Spencer, Eric Trout, Walter Vance,  and Tom Williams.
section_title: Related GAO and Other Products
paragraphs: For the purpose of evaluating VA’s efforts to improve its appeal  processing, we identified best practices and other criteria related to  staffing, process reform, and IT upgrades identified in prior GAO products  and other publications. These included government-wide internal control  standards; key principles for effective strategic workforce planning;  business process reengineering (or redesign) best practices, and  information technology planning principles. We also reviewed additional  guidance on project management.
Schedule Assessment Guide: Best Practices for Project Schedules.  GAO-16-89G. Washington, D.C.: December 2015.
Veterans Benefits Management System: Ongoing Development and  Implementation Can Be Improved; Goals are Needed to Promote  Increased User Satisfaction. GAO-15-582. Washington, D.C.: September  1, 2015.
Standards for Internal Control in the Federal Government. GAO-14-704G.  Washington, D.C.: September 2014.
Human Capital: Strategies to Help Agencies Meet Their Missions in an  Era of Highly Constrained Resources. GAO-14-168. Washington, D.C.:  May 7, 2014.
Managing For Results: Agencies Should More Fully Develop Priority  Goals under the GPRA Modernization Act. GAO-13-174. Washington,  D.C.: April 19, 2013.
GAO Cost Estimating and Assessment Guide: Best Practices for  Developing and Managing Capital Program Costs. GAO-09-3SP.  Washington D.C.: March 2009.
Human Capital: A Guide for Assessing Strategic Training and  Development Efforts in the Federal Government. GAO-04-546G.  Washington, D.C.: March 2004.
Information Technology Investment Management: A Framework for  Assessing and Improving Process Maturity. GAO-04-394G. Washington,  D.C.: March 2004.
Combating Terrorism: Evaluation of Selected Characteristics in National  Strategies Related to Terrorism. GAO-04-408T. Washington, D.C.:  February 3, 2004.
Human Capital: Key Principles for Effective Strategic Workforce Planning.  GAO-04-39. Washington, D.C.: December 11, 2003.
Human Capital: A Self-Assessment Checklist for Agency Leaders.  GAO/OCG-00-14G. Washington, D.C.: September 2000.
The Results Act: An Evaluator’s Guide to Assessing Agency Annual  Performance Plans. GAO/GGD-10.1.20. Washington, D.C.: April 1998.
Business Process Reengineering Assessment Guide.  GAO/AIMD-10.1.15. Washington, D.C.: May 1997.
Executive Guide: Effectively Implementing the Government Performance  and Results Act. GAO/GGD-96-118. Washington, D.C.: June 1996.
Blackburn, Simon, Sarah Ryerson, Leigh Weiss, Sarah Wilson, and  Carter Wood, Insights into Organization: How Do I Implement Complex  Change at Scale? Dallas, Texas: McKinsey & Company, May 2011.
George, Michael L, David Rowlands, Mark Price, and John Maxey. The  Lean Six Sigma Pocket Toolbook: A Quick Reference Guide to Nearly  100 Tools for Improving Process Quality, Speed, and Complexity. New  York: McGraw-Hill, 2005.
Office of Management and Budget. Circular No. A-11, Preparation,  Submission, and Execution of the Budget. Washington, D.C.: July 1,  2016.
Project Management Institute, Inc. A Guide To The Project Management  Body Of Knowledge (PMBOK Guide) Newtown Square, PA.: 2013.