Veterans Benefits Management System: Ongoing Efforts Can Be Improved; Goals Are Needed to Promote Increased User Satisfaction
Published: 2016-01-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: VBA pays disability benefits for conditions incurred or aggravated while in military service, and pension benefits for low-income veterans who are either elderly or have disabilities unrelated to military service. In fiscal year 2014, the department paid about $58 billion in disability compensation and about $5 billion in pension claims.
The disability claims process has been the subject of attention by Congress and others, due in part to long waits for processing claims and a large backlog of claims. To process disability and pension claims more efficiently, VA began development and implementation of an electronic, paperless system—VBMS—in 2009.
This statement summarizes GAO's September 2015 report ( GAO-15-582 ) on (1) VA's progress toward completing the development and implementation of VBMS and (2) the extent to which users report satisfaction with the system.
section_title: What GAO Found
paragraphs: As GAO reported in September 2015, the Veterans Benefits Administration (VBA) within the Department of Veterans Affairs (VA) has made progress in developing and implementing the Veterans Benefits Management System (VBMS), with deployment of the initial version of the system to all of its regional offices as of June 2013. Since then, VBA has continued developing and implementing additional system functionality and enhancements that support the electronic processing of disability compensation claims. As a result, 95 percent of records related to veterans' disability claims are electronic and reside in the system. However, VBMS is not yet able to fully support disability and pension claims, as well as appeals processing. Nevertheless, while the Under Secretary for Benefits stated in March 2013 that the development of VBMS was expected to be completed in 2015, implementation of functionality to fully support electronic claims processing has been delayed beyond 2015. In addition, VBA has not yet produced a plan that identifies when the system will be completed. Accordingly, holding VA management accountable for meeting a time frame and for demonstrating progress will be difficult.
As VA continues its efforts to complete development and implementation of VBMS, three areas could benefit from increased management attention.
Cost estimating: The program office does not have a reliable estimate of the cost for completing the system. Without such an estimate, VA management and the department's stakeholders have a limited view of the system's future resource needs, and the program risks not having sufficient funding to complete development and implementation of the system.
System availability: Although VBA has improved its performance regarding system availability to users, it has not established system response time goals. Without such goals, users do not have an expectation of the system response times they can anticipate and management does not have an indication of how well the system is performing relative to performance goals.
System defects : While the program has actively managed system defects, a recent system release included unresolved defects that impacted system performance and users' experiences. Continuing to deploy releases with large numbers of defects that reduce system functionality could adversely affect users' ability to process disability claims in an efficient manner.
VA has not conducted a customer satisfaction survey that would allow the department to compile data on how users view the system's performance, and ultimately, to develop goals for improving the system. GAO's survey of VBMS users found that a majority of them were satisfied with the system, but decision review officers were considerably less satisfied. Although the results of GAO's survey provide VBA with data about users' satisfaction with VBMS, the absence of user satisfaction goals limits the utility of survey results. Specifically, without having established goals to define user satisfaction, VBA does not have a basis for gauging the success of its efforts to promote satisfaction with the system, or for identifying areas where its efforts to complete development and implementation of the system might need attention.
section_title: What GAO Recommends
paragraphs: In its September 2015 report, GAO recommended that VA develop a plan with a time frame and a reliable cost estimate for completing VBMS, establish goals for system response time, minimize the incidence of high and medium severity system defects for future VBMS releases, assess user satisfaction, and establish satisfaction goals to promote improvement. VA concurred with GAO's recommendations.

section_title: Letter
paragraphs: I am pleased to be here to testify at today’s hearing on the Department of  Veterans Affairs’ (VA) efforts to develop and implement its Veterans  Benefits Management System (VBMS). As you know, VA’s disability  claims process has been a subject of attention for many years, due in part  to long waits for decisions and the large number of claims pending a  decision. In February 2010, the Secretary of Veterans Affairs committed  the department to eliminating the disability claims backlog and directed  the Veterans Benefits Administration (VBA) to process all incoming claims  within 125 days of their receipt and with at least 98 percent accuracy.  This was to be accomplished by the end of fiscal year 2015.
To help reduce the backlog and meet these claims processing goals, VBA  engaged in efforts to replace its paper-based claims process with  VBMS—a system intended to streamline the disability claims process by  providing claims processors with an electronic, paperless environment in  which to maintain, review, and make rating decisions for veterans’ claims.  The agency took an incremental approach to developing and  implementing VBMS and, as of June 2013, claims processors had begun  using an initial version of the system at all 56 regional offices. Since that  time, the agency has continued its efforts toward completing the system  and, through fiscal year 2015, had received approximately $1 billion in  funding for the initiative.
In September 2015, we issued a report documenting the results of a  study of VBMS that we undertook at this committee’s request. My  remarks today summarize key findings from that study, which (1)  assessed VA’s progress toward completing the development and  implementation of VBMS and (2) determined to what extent users  reported satisfaction with the system.
For the September 2015 report, we reviewed relevant program  documentation and interviewed appropriate VA officials. We also  administered a Web-based survey to a nationally representative stratified  random sample of VBMS users. More detailed information on our  objectives, scope, and methodology can be found in the issued report.
The work on which this statement is based was conducted in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.
subsections: section_title: Background
paragraphs: Throughout the disability compensation claims process, VBA staff have  various roles and responsibilities.
Claims assistants are primarily responsible for establishing the  electronic claims folders to determine whether the dispositions of the  claims and control actions have been appropriately identified.
Veteran service representatives are responsible for providing veterans  with explanations regarding the disability compensation benefits  programs and entitlement criteria. They also are to conduct  interviews, gather relevant evidence, adjudicate claims, authorize  payments, and input the data necessary to generate the awards and  notification letters to veterans describing the decisions and the  reasons for them.
Rating veterans service representatives are to make claims rating  decisions and analyze claims by applying VBA’s schedule for rating  disabilities (rating schedule) against claims submissions; they also are  to prepare rating decisions and the supporting justifications. Further,  they are to inform the veteran service representative, who then  notifies the claimant of the decision and the reasons for the decision.
Supervisory veteran service representatives are to ensure that the  quality and timeliness of service provided by VBA meets performance  indicator goals. They are also responsible for the cost-effective use of  resources to accomplish assigned outcomes.
Decision review officers are to examine claims decisions and perform  an array of duties to resolve issues raised by veterans and their  representatives. They may conduct a new review or complete a  review of a claim without deference to the original decision; they also  can revise that decision without new evidence or clear and obvious  evidence of errors in the original evaluation.
The disability compensation claims process starts when a veteran (or  other designated individual) submits a claim to VA in paper or electronic  form. If submitted electronically, a claim folder is created automatically.
When a paper claim is submitted, a claims assistant creates the  electronic folder. Specifically, when a regional office receives a new paper  claim, the receipt date is recorded electronically and the paper files (e.g.,  medical records and other supporting documents) are shipped to one of  four document conversion locations so that the supporting documents can  be scanned and converted into a digital image.
In the processing of both electronic and paper claims, a veteran service  representative reviews the information supporting the claim and helps  identify any additional evidence that is needed to evaluate the claim, such  as the veteran’s military service records, medical examinations, and  treatment records from medical facilities and private medical service  providers. Also, if necessary to provide support to substantiate the claim,  the department performs a medical examination on the veteran.
Once all of the supporting evidence has been gathered, a rating veterans  service representative evaluates the claim and determines whether the  veteran is eligible for benefits. If so, the rating veterans service  representative assigns a disability rating (expressed as a percentage). A  veteran who submits a claim with multiple disabilities receives a single  composite rating. If the veteran is due to receive compensation, an award  is prepared and the veteran is notified of the decision.
A veteran can reopen a claim for additional disability benefits if, for  example, he or she experiences a new or worsening service-connected  disability. If the veteran disagrees with the regional office’s decision on  the additional claim, a written notice of disagreement may be submitted to  the regional office to appeal the decision, and the veteran may request to  have the appeal processed at the regional office by a decision review  officer or through the Board of Veterans’ Appeals. Figure 1 presents a  simplified view of VA’s disability compensation claims process.
VBA began the transformation of its paper-intensive claims process to a  paperless environment in March 2009, and the effort became formally  established as the Veterans Benefits Management System program in  May 2010. VBA’s initial plans for VBMS emphasized the development of  a paperless claims platform to fully support the processing of disability  compensation and pension benefits, as well as appeals.
The program’s primary focus was to convert existing paper-based claims  folders into electronic claims folders (eFolders) to allow VBA staff to  access claims information and evidence in an electronic format. Beyond  the establishment of eFolders, VBMS is intended to streamline the entire  disability claims process, from establishment through award, by  automating rating decision recommendations, award and notification  processes, and communications between VBA and the veteran  throughout the claims life cycle. The system is also intended to assist in  eliminating the claims backlog and serve as the enabling technology for  quicker, more accurate, and integrated claims processing in the future.  Moreover, it is to replace many of the key outdated legacy systems— which are still in use today—for managing the claims process, including:
Share—used to establish claims; it records and updates basic  information about veterans and dependents.
Modern Award Processing-Development—used to manage the claims  development process, including the collection of data to support the  claims and tracking of them.
Rating Board Automation 2000—provides information about laws and  regulations pertaining to disabilities, which are used by rating  specialists in evaluating and rating disability claims.
Award—used to prepare and calculate the benefit award based on the  rating specialist’s determination of the claimant’s percentage of  disability. It is also used to authorize the claim for payment.
VBMS is to consist of three modules:
VBMS-Core is intended to provide the foundation for document  processing and storage during the claims development process,  including establishing claims; viewing and storing electronic  documents in the eFolder; and tracking evidence requested from  beneficiaries. The eFolder serves as a digital repository for all  documents related to a claim, such as the veteran’s military service  records, medical examinations, and treatment records from VA and  Department of Defense medical facilities, and from private medical  service providers. Unlike with paper files, this evidence can be  reviewed simultaneously by multiple VBA claims processors at any  location.
VBMS-Rating is to provide raters with Web-accessible tools, including  rules-based rating calculators and the capability for automated  decision recommendations. For example, the hearing loss calculator  is to automate decisions using objective audiology data and rules- based functionality to provide the rater with a suggested rating  decision. In addition, the module is expected to include stand-alone  evaluation builders—essentially interactive disability rating  schedules—for all parts of the human body. With this tool, the rater  uses a series of check boxes to identify the veteran’s symptoms and  the evaluation builder identifies the proper diagnostic code and the  level of compensation based on those symptoms.
VBMS-Awards is to provide an automated award and notification  process to improve award accuracy and reduce rework associated  with manual development of awards. This module is intended to  automate and standardize communications between VBA and the  veteran at the final stages of the claims process.
VBA is using an agile software development methodology to develop,  test, and deliver the system’s functionality to its users. An agile approach  allows subject matter experts to validate requirements, processes, and  system functionality in increments, and to deliver the functionality to users  in shorter cycles. Accordingly, the strategic road map that the VBMS  Program Management Office is using to guide the system development  effort indicated that releases of system functionality were to occur every 6  months. In a March 2013 Senate Veterans Affairs Committee hearing,  VA’s Under Secretary for Benefits stated that VBMS development was  expected to be completed in 2015.
section_title: Development and Implementation of VBMS Is Ongoing; Activities Can Benefit from Increased Management Attention
paragraphs: Our September 2015 report noted that, since completing rollout of the  initial version of VBMS at all regional offices in June 2013, VBA has  continued developing and implementing additional system functionality  and enhancements that support the electronic processing of disability  compensation claims. As a result, 95 percent of records related to  veterans’ disability claims are electronic and reside in the system.  However, while the Under Secretary for Benefits stated in March 2013  that the development of the system was expected to be completed in  2015, implementation of functionality to fully support electronic claims  processing was delayed until beyond 2015.
Specifically, even with the progress VBA has made toward developing  and implementing the system, the timeline for initial deployment of a  national workload management capability was delayed beyond the  originally planned date of September 2014 to October 2015, with  additional deployment to occur throughout fiscal year 2016. Efforts  undertaken thus far have addressed the strategic road map’s objective to  deliver a national workload management capability and have entailed  developing the technology and business processes needed to support the  National Work Queue, which is intended to handle new disability claims in  a centralized queue and assign claims to the next regional office with  available capacity.
The Program Management Office began work for the National Work  Queue in June 2014, and had intended to deploy the first phase of  functionality to users in September 2014. However, in late May 2015, the  Director of the office informed us that VBA had delayed the initial rollout  of the National Work Queue until October 2015 so that the department  could fully focus on meeting its goal to eliminate the claims backlog by the  end of September 2015. Following the initial rollout, the Program  Management Office intends to implement the National Work Queue at all  regional offices through fiscal year 2016.
Beyond this effort, VBMS program documentation identified additional  work to be performed after fiscal year 2015 to fully automate disability  claims processing. Specifically, the Program Management Office  identified the need to automate steps associated with a veteran’s request  for an increase in disability benefits, such as when an existing medical  condition worsens. In addition, the Director stated that the Program  Management Office intends to develop a capability to automatically  associate veterans’ correspondence when a new piece of evidence to  support a claim is received electronically or scanned into VBMS. The  office also plans to integrate VBMS with VA’s Integrated Disability  Evaluation System, which contains the results of veterans’ disability  medical examinations, as well as with external systems that contain  military service treatment records for veterans, including those at the  National Personnel Records Center.
Further, while VBMS was planned to support the processing of disability  compensation and pension benefits, VBA has not yet developed and  implemented end-to-end pension processing capabilities in the system.  Without such capabilities, the agency must continue to rely on three  legacy systems to process pension claims. Specifically, program officials  stated that both the Modern Award Processing-Development and Award  legacy systems contain functionality related to processing pensions and  will need to remain operational until VBMS can process pension claims.  In addition, the Share legacy system contains functionality that is still  needed throughout the claims process.
Program documentation indicates that the first phase of pension-related  functionality is expected to be introduced in December 2015. However,  VBA has not yet developed plans and schedules for retiring the legacy  systems and for fully developing and implementing their functionality in  VBMS.
VBA’s progress toward developing and implementing appeals processing  capabilities in VBMS also has been limited. Specifically, although the  information in a veteran’s eFolder is available to appeals staff for review,  the appeals process for disability claims is not managed using the new  system. According to VA’s fiscal year 2016 budget submission, the  department is pursuing a separate effort to manage end-to-end appeals  modernization, and has requested $19.1 million in fiscal year 2016 funds  to develop a system that will provide functionality not available in VBMS  or other VA systems. The Director of the Program Management Office  stated that VBA is currently analyzing commercial IT solutions that can  meet the business requirements for appeals, such as providing document  navigation capabilities. According to the Director, VBMS, nevertheless, is  expected to be part of the appeals modernization solution because  components of the system, such as the eFolder and certain workload  management functionality, are planned to continue supporting appeals  management.
In the Director’s view, the fact that VBMS requires additional development  beyond 2015 does not reflect a delay in completing the system’s  development. Instead, the additional time is a consequence of decisions  to enlarge the program’s scope over time. The Director stated that the  system’s original purpose had been to serve primarily as an electronic  document repository, and that the program has met this goal.
In addition, the Director said that, as the program’s mission has expanded  to support the department’s efforts to eliminate the disability claims  backlog, the office has had to re-prioritize, add, and defer system  requirements to accommodate broader departmental decisions and, in  some cases, regulatory changes. For example, the office was tasked with  developing functionality in VBMS to meet regulatory requirements for  processing disability claims using mandatory forms. Officials in the  office said they were made aware of this requirement well after system  planning for the March 2015 release had been completed, which had  introduced significant complexity to their development work.
Finally, VBA included in its strategic road map a number of objectives  related to VBMS that are planned to be addressed in fiscal year 2016.  Officials in the Program Management Office stated that they intend to  develop tactical plans that identify expected capabilities to be provided in  future releases.
Nevertheless, due to the department’s incremental approach to  developing and implementing VBMS, VBA has not yet produced a plan  that identifies when VBMS will be completed and can be expected to fully  support disability and pension claims processing and appeals. Thus, it will  be difficult for the department to hold its managers accountable for  meeting the time frame and for demonstrating progress. Accordingly, we  recommended that the department develop an updated plan for VBMS  that includes a schedule for when VBA intends to complete development  and implementation of the system, including capabilities that fully support  disability claims, pension claims, and appeals processing. VA agreed with  our recommendation.
subsections: section_title: VBMS Development and Implementation Have Been Hindered by Lack of a Reliable Cost Estimate
paragraphs: Consistent with our guidance on estimating program costs, an important  aspect of planning for IT projects, such as VBMS, involves developing a  reliable cost estimate to help managers evaluate a program’s affordability  and performance against its plans, and provide estimates of the funding  required to efficiently execute a program. In 2011, VBA submitted to the  Office of Management and Budget a life-cycle cost estimate for VBMS of  $934.8 million. This estimate was intended to capture costs for the  system’s development, deployment, sustainment, and general operating  expenses through the end of fiscal year 2018. However, as of July 2015,  the program’s actual costs had exceeded the 2011 life-cycle cost  estimate. Specifically, VBMS received approximately $1 billion in funding  through the end of fiscal year 2015 and the department has requested an  additional $290 million for the program in fiscal year 2016.
A significant concern is that the Program Management Office has not  reliably updated the VBMS life-cycle cost estimate to reflect the program’s  expanded scope and timelines for completion of the system. This is  largely attributable to the fact that the office has developed cost estimates  for 2-year project cycles that are used for VBMS milestone reviews under  the Office of Information and Technology’s Project Management  Accountability System.
When asked how the Program Management Office arrived at the cost  estimates reported in the milestone reviews, program officials stated that  they developed rough order of magnitude estimates for each business  need based on expert knowledge of the system, past development and  engineering experience, and lessons learned. However, while this  approach may have provided adequate information for VBA to prioritize  VBMS system requirements to be addressed in the next release, it has  not produced estimates that could serve as a basis for identifying the  system’s funding needs. Because it is typically derived from limited data  and in a short time, a rough order of magnitude analysis is not equivalent  to a budget-quality cost estimate and may limit an agency’s ability to  identify the funding necessary to efficiently execute a program.
In addition, the Program Management Office’s annual operating plan,  which is generally limited to high-level information about the program’s  organization, priorities, staffing, milestones, and performance measures  for fiscal year 2015, also shows estimated costs totaling $512 million for  VBMS development from fiscal years 2017 through 2020. However,  according to the Director of the Program Management Office, this  estimate was also developed using rough order of magnitude analysis.  Further, the estimate does not provide reliable information on life-cycle  costs because it does not include estimated IT sustainment and general  operating expenses.
Thus, even though the Program Management Office developed rough  order of magnitude cost estimates for VBMS, these estimates have not  been sufficiently reliable to effectively identify the program’s funding  needs. Instead, during the last 3 fiscal years, the Director has had to  request an additional $118 million in IT development funds to meet  program demands and to ensure support for ongoing development  contracts. Specifically, in May 2013, VA requested $13.3 million to  support additional work on VBMS. Then, during fiscal year 2014, VA  reprogrammed $73 million of unobligated IT sustainment funds to develop  functionality to transfer service treatment records from the Department of  Defense to VA, and to support development of VBMS-Core functionality.  In December 2014, the Program Management Office identified the need  for additional fiscal year 2015 funds for ongoing system development  contracts for VBMS-Core and VBMS-Awards, and, in late April 2015,  department leadership submitted a letter to Congress requesting  permission to reprogram $31.7 million to support work on these contracts,  the National Work Queue, and other VBMS efforts.
According to the Program Management Office Director, the need to  request additional funding does not represent additional risk to the  program, but is the result of VBMS’s success. The Director further noted  that, as the Program Management Office has identified opportunities to  increase functionality to improve the electronic claims process, their  funding needs have also increased. Nevertheless, evolution of the VBMS  program illustrates the importance of continuous planning, including cost  estimating, so that trade-offs between cost, schedule, and scope can be  effectively managed. Further, without a reliable estimate of the total costs  associated with completing work on VBMS, stakeholders will have a  limited view of VBMS’s future resource needs and the program is at risk  of not being able to secure appropriate funding to fully develop and  implement the system. Therefore, we recommended that VA develop an  updated plan for VBMS that includes the estimated cost to complete  development and implementation of the system. VA agreed with our  recommendation.
section_title: VBA Has Made Progress toward Improving VBMS Operation, but Does Not Have Key System Performance Goals
paragraphs: Our and other federal IT guidance recognize the importance of defining  program goals and related performance targets, and using such targets to  assess progress in achieving the goals. System performance and  response times have a large impact on whether staff successfully  complete work tasks. If systems are not responding at agreed-upon levels  for availability and performance, it can be difficult to ensure that staff will  complete tasks in a timely manner. This is especially important in the VBA  claims processing environment, where staff are evaluated on their ability  to process claims in a timely manner.
VBA reported that, since its initial rollout in January 2013, VBMS has  exceeded its 95 percent goal for availability. Specifically, the system was  available at a rate of 98.9 percent in fiscal year 2013 and 99.3 percent in  fiscal year 2014. Through May of fiscal year 2015, it was available for  99.98 percent of the time.
Nevertheless, while VBA has reported exceeding its availability goals for  VBMS, the system has also experienced periods of unavailability, many  times at a critical level affecting all users. Specifically, since January  2013, VBA reported 57 VBMS outages that totaled about 117 hours of  system unavailability. The system experienced about 18 hours of outages  in January 2014, which were almost entirely at the critical level and  affected all users. It reported experiencing only 2 system outages since  July 2014—a 30-minute critical outage in December 2014 and a 23- minute critical outage in May 2015.
In addition to system availability, VBA monitors system response times for  each of the VBMS modules using an application that measures the  amount of time taken for each transaction. From September 2013 through  April 2015, VBA reported a decrease in average response times for  VBMS-Core and VBMS-Rating. It attributed the decrease in response  times to continuous engineering improvements to system performance.  Program officials also explained that the difference in response times  between modules was due to the type of information that is being pulled  into each module from various other VBA systems. For example, both  VBMS-Core and VBMS-Rating require information from the VBA  corporate database, but VBMS-Core is populated with data from multiple  VBA systems in addition to the corporate database.
Program officials told us that specific goals for mean transaction response  times have not been established because they feel that adequate tools  are in place to monitor system performance and provide alerts if there are  response time issues. For example, VBMS performance is monitored in  real time by dedicated staff at a contractor’s facility, users have access to  a live chat feature where they can provide feedback on any issues they  are experiencing with the system, and the VBMS help desk offers another  avenue for users to provide feedback on the system’s performance. The  officials also noted that, because transaction response times have  decreased, which can be indicative of an improvement to system  performance, they are focusing their resources on adding additional  functionality instead of trying to get the system to achieve a specific  average transaction response time.
While VBA’s monitoring of VBMS’s performance is commendable and the  system’s performance and response times have improved over time, the  system is still in development and there is no guarantee that performance  will remain at current levels as the system evolves. Performance targets  and goals for VBMS response times would provide users with an  expectation of the system response times they should anticipate, and  management with an indication of how well the system is performing  relative to performance goals. To address this situation, we  recommended that the department establish goals for system response  time and use the goals as a basis for periodically reporting actual system  performance. VA agreed with this recommendation.
section_title: A Recent VBMS Release Included Unresolved Defects that Adversely Impacted System Implementation
paragraphs: A key element of successful system testing is appropriately identifying  and handling defects that are discovered during testing. Outstanding  defects can delay the release of functionality to end users, denying them  the benefit of features. Key aspects of a sound defect management  process include the planning, identification and classification, tracking,  and resolution of defects. Leading industry and government organizations  consider defect management and resolution to be among the primary  goals of testing.
The VBMS program has defect management policies in place and is  actively performing defect management activities. Specifically, in October  2012, the department developed the VBMS Program Management and  Technical Support Defect Management Plan, which describes the  program’s defect management process. The plan was updated in March  2015 and describes, among other things, the process for identifying,  classifying, tracking, and resolving VBMS defects. For example, it  provides criteria for assigning four different levels of severity for defects— critical, high, medium, and low.
According to the plan, critical severity defects are characterized by  complete system or subsystem failure, complete loss of functionality, and  compromised security or confidentiality. Critical defects also have  extensive user impact and do not have workarounds. High severity  defects can have major user impact, leading to significant loss of system  functionality. Medium severity defects can have moderate user impact  and lead to moderate loss of functionality. For high and medium severity  defects, workarounds could exist. Low severity defects lead to minor loss  of functionality with no workaround necessary. According to the Program  Management Office, high, medium, and low severity defects do not need  to be resolved prior to a system release.
The Program Management Office uses an automated tool to monitor and  track defects in the VBMS defect repository. It is used to produce a daily  defect management report that is shared with VBMS leadership, and to  provide the current status of all open defects identified in testing of a  forthcoming VBMS release or identified during production of a previous  release.
According to the defect management plan, defects can be resolved in a  number of different ways, and, once a defect has been fixed, tested, and  has passed testing, it is considered done or resolved. Defects that cannot  be attributed to an existing requirement are reclassified as a system  enhancement and considered resolved, as they do not affect a current  system release requirement. A defect is also considered resolved if it is  determined to work as designed, duplicate another defect, or if it is no  longer evident in the system.
From March 2014 through March 2015, the total number of VBMS defects  declined as release dates approached for four releases (7.0, 7.1, 8.0, and  8.1). Additionally, to the department’s credit, no critical defects remained  at the time of each of these releases.
However, even with the department’s efforts to resolve defects prior to a  VBMS release, defects that affected system functionality remained open  at the time of the releases. Specifically, of the 254 open defects at the  time of VBMS release 8.1, 76 were high severity, 99 were medium  severity, and 79 were low severity. Examples of medium and high level  defects that remained open at the time of VBMS release 8.1 included:
E-mail addresses for dependents only occasionally allowed special  characters (medium).
The intent to file for compensation/pension had an active status for a  deceased veteran (medium).
Creating a claim in legacy or VBMS would remove the Homeless,  POW, and/or Gulf War Registry Flash (high).
Disability name appeared incorrectly in Issue and Decision text for  amyotrophic lateral sclerosis (ALS) (high).
VBMS-Core did not recognize updated rating decisions from VBMS- Rating (high).
According to the Program Management Office, these defects were  communicated to users and an appropriate workaround for each was  established. Nevertheless, even with the workarounds, high and medium  severity open defects, which by definition impact system functionality,  degraded users’ experiences with the system. Continuing to deploy  system releases with defects that impact system functionality increases  the risk that these defects will diminish users’ ability to process disability  claims in an efficient manner. Accordingly, we recommended that VA  reduce the incidence of high and medium severity level defects that are  present at the time of future VBMS releases. The department agreed with  this recommendation.
section_title: VBA Had Not Conducted a Survey to Obtain Users’ Feedback or Established Related Goals; GAO Found that Satisfaction with the System Varied
paragraphs: Our September 2015 report noted that, in addition to having defined  program goals and related performance targets, leading practices identify  continuous customer feedback as a crucial element of IT project  success. Particularly for projects like VBMS, where development  activities are iterative, customer and end user perspectives and insights  can be solicited through various methods—user acceptance testing,  interviews, complaint programs, and satisfaction surveys—to validate or  raise questions about the project’s implementation.
Further, leading practices emphasize that periodic customer satisfaction  data should be proactively used to improve performance and demonstrate  the level of satisfaction the project is delivering. The Office of  Management and Budget has developed standards and guidelines in  survey research that are generally consistent with best practices and  call for statistically valid data collection efforts to be used in fulfilling  agencies’ customer service data collection. These leading practices also  stress the importance of centrally integrating all customer feedback data  in order to have more complete diagnostic information to guide  improvement efforts.
VA has used a variety of methods for obtaining customer and end user  feedback on the performance of VBMS. For example, the department  solicits end user involvement and feedback in the iterative system  development process based on user acceptance criteria. According to  the Senior Project Manager for VBMS Development within the Office of  Information and Technology, at the end of each development cycle and  before a new version of VBMS is deployed, end users are involved in  user acceptance testing and a final customer acceptance meeting.
The department also provides training to a subset of end users—known  as “superusers”—on the updated functionality introduced in a new version  of VBMS. These superusers are expected to train the remaining users in  the field on the new version’s features. The department tracks the overall  satisfaction level with training received after each VBMS major release.
However, this tracking is limited to superusers’ satisfaction with the  training, rather than with their satisfaction with the system.
Further, the department solicits customer feedback about the system  through interviews. For example, the Director of the Program  Management Office stated that the Under Secretary for Benefits hosts a  weekly phone call with bargaining unit employees as a “pulse check” on  VBA transformation activities, including VBMS. According to this official,  the VBA Office of Field Operations also offers an instant messaging chat  service to all regional office employees to solicit feedback about the latest  deployment of VBMS functionality.
Another method in which the department obtains customer input is  through a formal feedback process. For example, according to the  Director, VA provides national service desk support to assist users in  troubleshooting system issues and identifying system defects. In addition,  VBMS applications include a built-in feature that enables users to provide  feedback to the Program Management Office on problems with the  system. According to the Director, the feedback received by the office  also helps to identify user training issues.
Nevertheless, while VA has taken these various steps to obtain feedback  on the performance and implementation of VBMS, it has not established  goals to define user satisfaction that can be used as a basis for gauging  the success of its efforts to promote satisfaction with the system. Further,  while the efforts that have been taken to solicit users’ feedback provide  VBA with useful insights about particular problems, data are not centrally  compiled or sufficient for supporting overall conclusions about whether  customers are satisfied. In addition, VBA has not employed a customer  satisfaction survey of claims processing employees who use the system  on a daily basis to process disability claims. Such a survey could provide  a more comprehensive picture of overall customer satisfaction and help  identify areas where the system’s development and implementation  efforts might need additional attention.
According to the Director of the Program Management Office, VBA has  not used a survey to solicit feedback because of concern that such a  mechanism may negatively impact the efficiency of claims processors in  completing disability compensation claims on behalf of veterans. Further,  the Director believed that the office had the benefit of receiving ongoing  end user input on VBMS by virtue of the intensive testing cycles, as well  as several of the other mechanisms by which end users have provided  ongoing feedback. Nevertheless, without establishing user satisfaction  goals and collecting the comprehensive data that a statistically valid  survey can provide, the Program Management Office limits its ability to  obtain a comprehensive understanding of VBMS users’ satisfaction with  the system. Thus, VBA could miss opportunities to improve the efficiency  of its claims process by increasing satisfaction with VBMS. Therefore, we  recommended that VA develop and administer a statistically valid survey  of VBMS users to determine the effectiveness of steps taken to make  improvements in users’ satisfaction. The department agreed with this  recommendation.
subsections: section_title: Most Types of Users Reported Satisfaction with VBMS, but Decision Review Officers Were Generally Dissatisfied
paragraphs: In response to a statistical survey that we administered, most of the  VBMS users reported that they were satisfied with the system that had  been implemented at the time of the survey. These users (claims  assistants, veteran service representatives, supervisory veteran service  representatives, rating veterans service representatives, decision review  officers, and others) were satisfied with the three modules of VBMS.
Specifically, an estimated 59 percent of the claims processors were  satisfied with VBMS-Core; an estimated 63 percent were satisfied with  the Rating module, and an estimated 67 percent were satisfied with the  Awards module.
Nevertheless, while a majority of users were satisfied with the three  modules, decision review officers expressed considerably less  satisfaction than other users with VBMS-Core and VBMS-Rating.  Specifically, for VBMS-Core, an estimated 27 percent of decision review  officers were satisfied compared to an estimated 59 percent of all roles of  claims processors (including decision review officers) who were satisfied.  In addition, for VBMS-Rating, an estimated 38 percent of decision review  officers were satisfied, compared to an estimated 63 percent of all roles of  claims processors.
Decision review officers were considerably less satisfied with VBMS in  comparison to all roles of claims processors in additional areas. For  example, an estimated 26 percent of decision review officers viewed  VBMS-Core as an improvement over the previous legacy system or  systems for establishing claims and storing and reviewing electronic  documents related to a claim in an eFolder. In contrast, an estimated 58  percent of all users (including decision review officers) viewed the Core  module as an improvement.
In addition, an estimated 26 percent of decision review officers viewed  VBMS-Rating as an improvement over the previous systems with respect  to providing Web-accessible tools, including rules-based rating  calculators, to assist in making claims rating decisions. In contrast, an  estimated 55 percent of all roles of claims processors viewed the Rating  module as an improvement. For VBMS-Awards, an estimated 61 percent  of all roles viewed this module as an improvement over the previous  systems to automate the award and notification process.
Similarly, in considering the three modules, a majority of users (including  decision review officers) would have chosen VBMS over the legacy  system or systems. However, decision review officers indicated that they  were less likely to have chosen VBMS-Core and VBMS-Rating over  legacy systems. Specifically, an estimated 27 percent of decision review  officers would have chosen VBMS-Core compared to an estimated 60  percent of all roles of claims processors. In addition, an estimated 27  percent of decision review officers would have chosen VBMS-Rating  compared to 61 percent of all roles that would have chosen the system  over the legacy system or systems. For VBMS-Awards, an estimated 67  percent of all roles would have chosen this module over the previous  systems.
Decision review officers perform an array of duties to resolve claims  issues raised by veterans and their representatives. They may also  conduct a new review or complete a review of a claim without deference  to the original decision, and, in doing so, must click through all documents  included in the e-Folder. Survey comments from decision review officers  stated, for example, that reviews in the VBMS paperless environment  take longer because of the length of time spent loading, scrolling, and  viewing each document (particularly if the documents are large, such as a  service medical record file). Additionally, multiple decision review officers  commented that it is easier and faster to review documents in a paper file.  Although such comments provide illustrative examples of individual  decision review officers’ views and are not representative, according to  the Director of the Program Management Office, decision review officers’  relative dissatisfaction is not surprising because the system does not yet  include functionality that supports their work, which primarily relates to  appeals processing. To improve this situation, we recommended that VA  establish goals that define customer satisfaction with the system and  report on actual performance toward achieving the goals based on the  results of our survey of VBMS users and any future surveys VA conducts.  The department concurred with this recommendation.
In conclusion, while VA has made progress in developing and  implementing VBMS, additional capabilities to fully process disability  claims were delayed beyond when the system’s completion was originally  planned. Further, in the absence of a plan that identifies when and at  what cost the system can be expected to fully support disability  compensation and pension claims processing and appeals, holding VA  management accountable for meeting a schedule, while ensuring  sufficient program funding, will be difficult. Also, without goals for system  response times, users do not have an expectation of the response times  they can anticipate, and management lacks an indication of how well the  system is performing. Furthermore, continuing to deploy system releases  with defects that impact functionality increases the risk that these defects  will diminish users’ ability to process disability claims in an efficient  manner. Lastly, although the results of our survey provide VBA with  useful data about users’ satisfaction with VBMS (e.g., the majority of  users are satisfied), without having goals to define user satisfaction, VBA  does not have a basis for gauging the success of its efforts to improve the  system. As we stressed in our report, attention to these issues can  improve VA’s efforts to effectively complete the development and  implementation of VBMS. Fully addressing our recommendations, as VA  agreed to do, should help the department give appropriate attention to  these issues.
Chairman Miller, Ranking Member Brown, and Members of the  Committee, this concludes my prepared statement. I would be pleased to  respond to any questions that you may have.
section_title: Contacts and Staff Acknowledgments
paragraphs: For further information about this testimony, contact Valerie C. Melvin at  (202) 512-6304 or melvinv@gao.gov. Contact points for our offices of  Congressional Relations and Public Affairs are listed on the last page of  the testimony. Other key contributors to this testimony include Mark Bird  (assistant director), Kavita Daitnarayan, Kelly Dodson, Nancy Glover,  Brandon S. Pettis, and Eric Trout.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.