Veterans Benefits Management System: Ongoing Development and Implementation Can Be Improved; Goals Are Needed to Promote Increased User Satisfaction
Published: 2015-09-01T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: VBA pays disability benefits for disabling conditions incurred or aggravated while in military service, while pension benefits are for low-income veterans who are either elderly or have disabilities unrelated to military service. In fiscal year 2014, the department paid about $58 billion in disability compensation and about $5 billion in pension claims. The disability claims process has been the subject of attention by Congress and others, due in part, to long waits for processing claims and a large backlog of claims. To process disability and pension claims more efficiently, VA began implementation of an electronic, paperless system in 2009. GAO was asked to study VBMS. Specifically, GAO (1) assessed VA's progress toward completing the development and implementation of VBMS and (2) determined to what extent users report satisfaction with the system. To do so, GAO reviewed relevant program documentation, administered a survey to a stratified random sample of about 3,500 users, and interviewed appropriate VA officials.
section_title: What GAO Found
paragraphs: The Veterans Benefits Administration (VBA) within the Department of Veterans Affairs (VA) has made progress in developing and implementing the Veterans Benefits Management System (VBMS), with deployment of the system to all of its regional offices as of June 2013. While 95 percent of records related to veterans' disability claims are electronic and reside in the system, additional capabilities have not yet been completed, such as automation of the steps associated with a veteran's request for an increase in benefits. Further, VBA has not yet developed and implemented pension processing capabilities in VBMS, nor has it articulated when the system will support appeals processing. The VBMS program reported receiving funding of about $1 billion from fiscal years 2009 to 2015, at which time system completion was originally planned. Although development of the system is expected to continue beyond 2015, the incremental approach VA is using to develop and implement VBMS has not yet produced a plan that identifies when the system will be completed and can be expected to fully support disability and pension claims processing and appeals. Thus, it will be difficult for VA to hold its managers accountable for meeting its time frame and for demonstrating progress.
As VA continues its efforts to complete development and implementation of the system, three areas could benefit from increased management attention.
Cost estimating: The program office does not have a reliable estimate of the cost for completing the system. Without such an estimate, VA management and the department's stakeholders have a limited view of the system's future resource needs, and the program risks not having sufficient funding to complete development and implementation of the system.
System availability: Although VBA has improved its performance for ensuring the system is available to users, it has not established system response time goals. Without such goals, users do not have an expectation of the system response times they can anticipate and management does not have an indication of how well the system is performing relative to performance goals.
System defects: While the program has actively managed system defects, a recent system release included unresolved defects that impacted system performance and users' experiences. Continuing to deploy releases with large numbers of defects that reduce system functionality could adversely affect users' ability to process disability claims in an efficient manner.
While VBA has employed various methods to obtain VBMS users' feedback, it has neither established goals to define user satisfaction, nor conducted a survey of claims processing employees to obtain a more comprehensive picture of overall customer satisfaction. GAO's survey of VBMS users estimated that a majority report satisfaction with the system, but that one group of users who are responsible for examining claims decisions was considerably less satisfied. Although the results of GAO's survey provide VBA with useful data about users' satisfaction with the system, the absence of user satisfaction goals limits the utility of survey results. Specifically, without having established goals to define user satisfaction, VBA does not have a basis for gauging the success of its efforts to promote satisfaction with the system.
section_title: What GAO Recommends
paragraphs: GAO recommends that VA develop a plan for completing VBMS, establish goals for system response time, minimize the incidence of high and medium priority system defects for future VBMS releases, assess user satisfaction, and establish satisfaction goals to promote improvement. VA concurred with the recommendations and described actions it is planning to take in response, except for the first recommendation. GAO continues to believe development of a plan for completing the system is important.

section_title: Letter
paragraphs: The Department of Veterans Affairs’ (VA) disability compensation and  pension benefit programs provide monetary benefits to veterans.  Disability benefits are paid for disabling conditions incurred or aggravated  while in military service. In fiscal year 2014, the department paid about  $58 billion in disability compensation to nearly 3.9 million veterans and  their survivors. In addition, VA’s pension program pays benefits to low- income veterans who either are elderly or have disabilities unrelated to  their military service. The program is available to veterans who are age  65 and older or who have disabilities that are unrelated to their military  service; it is also available to their surviving spouses and dependent  children. In fiscal year 2014, VA paid approximately $5.2 billion in pension  benefits to about 308,000 veterans and 213,000 survivors.
The disability compensation claims process has been the subject of  attention by Congress, veterans’ service organizations, and others, due in  part, to long waits for decisions and the large number of claims pending a  decision. As such, in February 2010, the Secretary of Veterans Affairs  committed the department to eliminating the disability claims backlog by  the end of fiscal year 2015. In addition, the Secretary directed the  Veterans Benefits Administration (VBA) to, by the end of fiscal year 2015,  process all incoming claims within 125 days of their receipt and with at  least 98 percent accuracy.
To help reduce the backlog and meet its claims processing timeliness and  accuracy goals, VBA has engaged in efforts to replace its paper-based  claims process with an electronic paperless claims processing system  called the Veterans Benefits Management System (VBMS). This system  is intended to streamline the disability claims process by providing claims  processors with an electronic environment in which to maintain, review,  and make rating decisions for veteran’s claims. VBA is taking an  incremental approach to developing and implementing the system and, as  of June 2013, claims processors had begun using an initial version of the  system at all 56 regional offices.
At your request, we conducted a study of VBMS. Our specific objectives  were to (1) assess VA’s progress toward completing the development and  implementation of VBMS and (2) determine to what extent users report  satisfaction with the system.
To accomplish the objectives, we reviewed relevant program  documentation and interviewed appropriate VA officials. We also  administered a survey to VBMS users.
Specifically, to assess progress toward completing the development and  implementation of VBMS, we obtained and analyzed program  documentation, including plans that articulate the department’s goals and  expected benefits from use of the system; project plans and schedules for  iterative releases of different modules and generations of the system;  contracts and related contractor documentation; and change  management plans. We then compared program plans to VA’s policies for  incremental system development, as well as federal guidance and  information technology (IT) project management principles on program  planning and cost estimating.
Additionally, we obtained and analyzed data related to the system’s  availability and performance covering the time period since its initial  deployment in January 2013 through May 2015. We assessed the  reliability of the data by reviewing it for obvious errors and missing data;  corroborating the data with related documentation; and interviewing  responsible officials about their use of an automated tool that monitors  system performance. We determined the data to be sufficiently reliable for  the purposes of this report.
We compared system performance data with GAO and federal IT  guidance for defining program goals and related performance targets that  can be used to assess progress in achieving the goals.
In addition, we obtained the VBMS defect management plan, compared  the plan with key principles of sound defect management, and analyzed  the extent to which processes described in the plan were followed during  the development of recent system releases.
Further, we reviewed the department’s methods for soliciting end user  feedback on the performance of the system (e.g., VBMS training strategy,  operational management reviews, cost and performance reports, and  end-of-month reports). We compared these methods to leading practices  for obtaining customer feedback and collecting customer service data to  improve performance and demonstrate customer satisfaction of IT  projects.
To determine users’ satisfaction with VBMS, we administered a Web- based survey to a nationally representative stratified random sample of  VBA users. These users (collectively referred to as claims processors)  included claims assistants, veteran service representatives, supervisory  veteran service representatives, rating veterans service representatives,  decision review officers, and others.
We pretested the survey and obtained initial insight regarding claims  processors’ use of the system at two regional offices—in Philadelphia,  Pennsylvania, and Baltimore, Maryland. We selected these offices based  on their large size and location. We also observed and interviewed claims  processors who were using VBMS at the VA Appeals Management  Center in Washington, D.C., to gain additional insight into the use of the  system. We received a response rate of 60 percent and adjusted for  characteristics that were associated with survey response propensity  using standard weighting class adjustments defined by sampling strata.  As a result, we deemed the survey responses to be generalizable to all  claims processors using the system at the 56 VBA regional offices. We  supplemented our user observations with interviews of other VA officials  in the Office of Field Operations, VBMS Program Management Office  (PMO), and in VA’s Office of Information and Technology.
We conducted this performance audit from March 2014 through August  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. A more complete  discussion of our objectives, scope, and methodology is provided in  appendix I.
subsections: section_title: Background
paragraphs: VA pays monthly compensation to veterans with service-connected  disabilities (i.e., injuries or diseases incurred or aggravated while on  active military duty) according to the severity of the disability. It pays  additional compensation for certain dependent spouses, children, and  parents of veterans. The department processes service-connected  disability compensation benefits for about 3 million beneficiaries each  month. It reported completing the processing of approximately 1.3 million  disability rating claims for veterans and beneficiaries in fiscal year 2014— about 150,000 more than in fiscal year 2013. As of mid-June 2015, the  department reported a backlog of approximately 137,000 disability claims,  down from its peak of about 611,000 disability claims in March 2013.
VA has reported that the accuracy of claims decisions rose from a level of  83 percent in 2011 to 90 percent in 2014. However, even as VBA  reported providing historic numbers of veterans with decisions on their  claims in 2014, the number and complexity of claims continue to increase.  For example, VBA reported for 2014 that its employees made decisions  on 5.5 million medical issues within claims—a 101 percent increase in the  number of issues since 2009.
subsections: section_title: Overview of VBA’s Disability Compensation Claims Process
paragraphs: Throughout the disability compensation claims process, VBA staff have  various roles and responsibilities.
Claims assistants are primarily responsible for establishing the  electronic claims folders to determine whether the dispositions of the  claims and control actions have been appropriately identified.
Veteran service representatives are responsible for providing veterans  with explanations regarding the disability compensation benefits  programs and entitlement criteria. They also are to conduct  interviews, gather relevant evidence, adjudicate claims, authorize  payments, and input the data necessary to generate the awards and  notification letters to veterans describing the decisions and the  reasons for them.
Rating veterans service representatives are to make claims rating  decisions and analyze claims by applying VBA’s schedule for rating  disabilities (rating schedule) against claims submissions, and  preparing rating decisions and the supporting justifications. They also  are to inform the veteran service representative who then notifies the  claimant of the decision and the reasons for the decision.
Supervisory veteran service representatives are to ensure that the  quality and timeliness of service provided by VBA meets performance  indicator goals. They are also responsible for the cost-effective use of  resources to accomplish assigned outcomes.
Decision review officers are to examine claims decisions and perform  an array of duties to resolve issues raised by veterans and their  representatives. They may conduct a new review or complete a  review of a claim without deference to the original decision; they also  can revise that decision without new evidence or clear and obvious  evidence of errors in the original evaluation.
The disability compensation claims process starts when a veteran (or  other designated individual) submits a claim to VA, in paper or electronic  form. If submitted electronically, a claim folder is created automatically.
When a paper claim is submitted, a claims assistant creates the  electronic folder. Specifically, when a regional office receives a new paper  claim, the receipt date is recorded electronically and the paper files (e.g.,  medical records and other supporting documents) are shipped to one of  four document conversion locations so that the supporting documents can  be scanned and converted into a digital image.
In the processing of both electronic and paper claims, a veteran service  representative is to review the information supporting the claim and help  identify any additional evidence that is needed to evaluate the claim, such  as the veteran’s military service records, medical examinations, and  treatment records from medical facilities and private medical service  providers. Also, if necessary to provide support to substantiate the claim,  the department is to perform a medical examination on the veteran.
Once all of the supporting evidence has been gathered, a rating veterans  service representative evaluates the claim and determines whether the  veteran is eligible for benefits. If so, the rating veterans service  representative assigns a disability rating (expressed as a percentage). A  veteran who submits a claim with multiple disabilities receives a single  composite rating. If the veteran is due to receive compensation, an award  is prepared and the veteran is notified of the decision.
A veteran can reopen a claim for additional disability benefits if, for  example, the veteran experiences a new or worsening service-connected  disability. If the veteran disagrees with the regional office’s decision on  the additional claim, he or she may submit a written notice of  disagreement to the regional office to appeal the decision, and may  request to have the appeal processed at the regional office by a decision  review officer or through the Board of Veterans’ Appeals. Figure 1  presents a simplified view of VA’s disability compensation claims process.
section_title: Veterans Benefits Management System
paragraphs: VBA began the transformation of its paper-intensive claims process to a  paperless environment in March 2009. This effort became formally  established as the Veterans Benefits Management System program in  May 2010. VBA’s initial plans for VBMS emphasized the development of  a paperless claims platform to fully support the processing of disability  compensation and pension benefits, as well as appeals. The program’s  importance was further elevated in January 2013 when the Secretary of  Veterans Affairs identified the system as the enabling technology to meet  the goal to eliminate the disability claims backlog in fiscal year 2015 and  improve the efficiency and accuracy of all compensation claims  processing to 125 days and 98 percent, respectively. In a March 2013  Senate Veterans Affairs Committee hearing, VA’s Under Secretary for  Benefits stated that VBMS development was expected to be completed in  2015.
The program’s primary focus was to convert existing paper-based claims  folders into electronic claim folders (eFolders) to allow VBA employees  to access claims information and evidence in an electronic format.  Beyond the establishment of eFolders, VBMS is also intended to  streamline the entire disability claims process, from establishment  through award, by automating rating decision recommendations, award  and notification processes, and communications between VBA and the  veteran throughout the claims life cycle; assist in eliminating the claims  backlog and serve as the enabling technology for quicker, more accurate,  and integrated claims processing in the future; and replace many of the  key outdated legacy systems—which are still in use today—for managing  the claims process, including:
Share—used to establish claims; it records and updates basic  information about veterans and dependents.
Modern Award Processing–Development —used to manage the  claims development process, including the collection of data to  support the claims and the tracking of claims.
Rating Board Automation 2000—provides information about laws and  regulations pertaining to disabilities, which are used by rating  specialists in evaluating and rating disability claims.
Award—used to prepare and calculate the benefit award based on the  rating specialist’s determination of the claimant’s percentage of  disability. It is also used to authorize the claim for payment.
VBMS is to consist of three modules:
VBMS-Core is intended to provide the foundation for document  processing and storage during the claims development process,  including establishing claims; viewing and storing electronic  documents in the eFolder; and tracking evidence requested from  beneficiaries. The eFolder serves as a digital repository for all  documents related to a claim, such as the veteran’s military service  records, medical examinations, and treatment records from VA and  Department of Defense medical facilities, and from private medical  service providers. Unlike with paper files, this evidence can be  reviewed simultaneously by multiple VBA claims processors at any  location.
VBMS-Rating is to provide raters with Web-accessible tools, including  rules-based rating calculators and the capability for automated  decision recommendations. For example, the hearing loss calculator  is to automate decisions using objective audiology data and rules- based functionality to provide the rater with a suggested rating  decision. In addition, VBMS-Rating is expected to include stand-alone  evaluation builders—essentially interactive disability rating  schedules—for all parts of the human body. With this tool, the rater  uses a series of check boxes to identify the veteran’s symptoms, and  the evaluation builder identifies the proper diagnostic code and the  level of compensation based on those symptoms.
VBMS-Awards is to provide an automated award and notification  process to improve award accuracy and reduce rework associated  with manual development of awards. VBMS-Awards is intended to  automate and standardize communications between VBA and the  veteran at the final stages of the claims process.
subsections: section_title: VBMS Is Being Developed Incrementally and Was Initially Deployed In 2013
paragraphs: VBA is using an agile software development methodology to develop,  test, and deliver VBMS functionality to its users. An agile approach allows  subject matter experts to validate requirements, processes, and system  functionality in increments, and to deliver the functionality to users in  shorter cycles. To help guide its system development efforts, the VBMS  PMO has developed both a strategic road map that identifies the  program’s high-level objectives, timeline and intended outcomes, as well  as a regularly revised tactical road map that describes prospective  capabilities the PMO expects to develop and deploy for each system  release. Consistent with VA’s policy for incremental development, VBMS  is to be developed and implemented in a series of releases that are to  occur every 6 months. As shown in table 1, VBA plans to develop VBMS  over multiple years, with each year generally corresponding to a new  system generation.
From 2010 through January 2013, VBA planned, developed, and  deployed a foundational, Web-based version of VBMS to five pilot sites.  This phase included development of the eFolder capability and the ability  to establish, develop, and rate disability compensation claims in VBMS,  as well as a user interface with search capabilities. Generation 1  development was completed in January 2013; at that time, the system  was implemented at 18 regional offices.
From February 2013 through September 2013, the PMO continued to add  functionality to the system, including enhancement of data exchange  capabilities, correspondence tools, and rating functionality. By June 2013,  VBA had completed national rollout of the initial version of the system to  all 56 regional offices. The system was also made accessible to VA’s  Appeals Management Center, the Board of Veterans’ Appeals, VBA  National Call Centers, veterans service organizations, and all VA medical  centers that complete compensation exams. Subsequent to the  nationwide rollout of VBMS in June 2013, VBA has continued incremental  system development and enhancement of VBMS.
section_title: Multiple Organizations Have Responsibility for VBMS
paragraphs: The VBMS program is dependent on multiple organizations within the  department to meet its goals. It is jointly led by two program managers:  the VBMS PMO Director and a Program Manager in VA’s Office of  Information and Technology (OI&T).
Specifically, the VBMS PMO, which resides in VBA’s Office of Strategic  Planning, is responsible for all aspects of the program’s management,  including the coordination and direction of contract staff and VBA partners  in integrating the system’s components and managing program-level  dependencies, risks, and issues. The PMO has responsibility for  gathering and delivering system requirements, performing testing, and  providing training. The Director oversees all activities of the PMO and  reports to the Under Secretary for Benefits.
The OI&T program manager oversees the development and  implementation of VBMS and reports to the VA Chief Information Officer.  OI&T entered into an interagency agreement with the Department of the  Navy’s Space and Naval Warfare Systems Command (SPAWAR)  Systems Center Atlantic to lead the development of VBMS. SPAWAR  manages multiple contractors to develop the system and is providing  technical, information assurance, program management, testing, and data  integration services to support application development. VA and  SPAWAR work together to manage and develop the system. Specifically,  VBA subject matter experts and OI&T technical representatives are part  of the system development teams.
Further, VBA’s Office of Business Process Integration is to ensure that  strategic needs and requirements for business and data systems are  properly documented, integrated, and communicated. The office provides  internal coordination across the VBA lines of business (compensation,  pension, education, loan guaranty, etc.) and helps communicate the  system’s requirements to the OI&T at the department level. The Office of  Business Process Integration also manages VBA’s legacy claims  processing systems and their sustainment. As such, it is working with the  OI&T on long-term planning for the systems, which may include their  decommissioning.
In addition to these offices, VBMS is governed by the Transformation  Joint Executive Board. Jointly chaired by the VA CIO and the Under  Secretary for Benefits, this board is responsible for discussing and  addressing the program’s risks and other issues.
In addition, VBMS is to follow the OI&T’s Project Management  Accountability System, which is the department’s process for managing  IT projects. As part of this process, large IT programs are broken down  into multiple projects that are typically planned, developed, and  implemented in 2-year cycles. Each project is broken into several  increments, with each increment typically lasting 6 months or less. The  PMO is responsible for presenting a life-cycle cost estimate for the 2-year  project as part of each increment’s milestone review. Resources,  including staff and funding, are released once the increment has been  approved.
section_title: VBMS Has Been Provided with About $1 Billion in Funding
paragraphs: In 2008, VBA developed an initial, high-level life-cycle cost estimate of  $560 million for VBMS that reflected system development costs through  fiscal year 2012. The PMO revised the cost estimate in 2011, to include  costs for system development, sustainment, and general operating  expenses. This estimate showed that the department expected to spend  $934.8 million on VBMS through its life cycle.
In July 2012, we reported on the reliability of the cost estimate for the  VBMS program and noted that, while the 2011 cost estimate for the  system partially reflected key practices for developing a comprehensive  and well-documented estimate, it did not reflect key practices for  developing an accurate and credible estimate. We recommended that  any future life-cycle cost estimates for the VBMS program address the  detailed weaknesses that we identified using cost-estimating best  practices. VA’s Chief of Staff stated that the department concurred with  our recommendation and had efforts under way to improve its cost- estimating capabilities.
Officials from the PMO subsequently provided us information showing  that from fiscal years 2009 through 2015, the VBMS program had  received funding in the amount of approximately $1 billion— about $502  million for system development, $308 million for IT sustainment, and  $194 million for general operating expenses.
section_title: VA Has Made Progress toward Developing and Implementing VBMS, but Ongoing Activities Could Benefit from Increased Management Attention
paragraphs: VBA has developed and implemented capabilities for VBMS to support  disability claims processing. However, completion of additional  functionality to fully support processing disability claims will be delayed  beyond fiscal year 2015. Further, VBA’s plans for developing and  implementing capabilities to support the processing of pension benefits  and appeals are uncertain. As VBA continues to develop and implement  the system, three areas could benefit from increased management  attention. First, the PMO does not have a reliable estimate of the cost for  completing the system. Second, although VBA has improved VBMS’s  availability to users, it has not established goals for system response  times. Third, while the program has actively managed system defects, a  recent system release included multiple unresolved defects that adversely  impacted performance and users’ experiences.
subsections: section_title: VBA Has Made Important Progress toward Developing and Implementing VBMS to Support the Processing of Disability Claims; but Key Capabilities Are Delayed and Plans for Processing Pension Benefits and Appeals Are Uncertain
paragraphs: Since completing the implementation of VBMS at all regional offices in  June 2013, VBA has continued to make important progress toward  developing and implementing additional system functionality and  enhancements that support electronic processing of disability  compensation claims. As a result, 95 percent of records related to  veterans’ disability claims are electronic and reside in VBMS. However,  although the Under Secretary for Benefits stated in March 2013 that the  system’s development was expected to be completed in 2015,  implementation of functionality to fully support electronic claims  processing has been delayed beyond 2015. Additionally, federal guidance  and IT project management principles stress the importance of  continuous planning throughout the life of a program to serve as a basis  for managing trade-offs between cost, schedule, and scope.
During fiscal year 2014, VBA’s progress on the system included  developing and implementing capabilities that addressed the strategic  objectives for generation 3 of the strategic road map. For example:
Consistent with the objective to provide system users with more  complex automation capabilities, the PMO released an automation  feature that populates the VBMS rating calculator with a veteran’s  medical information for 37 of the 71 disability benefits questionnaires  that medical providers can use to submit medical information in  support of a disability claim.
Regarding the strategic objective to reduce dependency on legacy  systems, in April 2014, the PMO delivered the first iteration of the  VBMS-Awards module to all regional offices, providing claims  processors the ability to rely less on legacy systems, prepare an  award based on information from VBMS’s rating module, and  generate associated notification letters for veterans. The functionality  released during fiscal year 2014 also included the integration of  VBMS with VA’s legacy electronic records store, “Virtual VA,” and the  integration of additional correspondence letters.
To address the strategic objective to deliver a capability to accept  veterans’ electronic service treatment records, in December 2013, the  PMO implemented new functionality in VBMS to enable electronic  request and receipt of service treatment records from the Department  of Defense for veterans who had separated or been discharged from  military service after January 1, 2014.
VBA has continued its progress toward developing and implementing  VBMS during fiscal year 2015 with efforts to address generation 4 of the  strategic road map. Specifically, to address the strategic objective calling  for further reduced reliance on legacy systems, the PMO has made  enhancements to the system’s correspondence capabilities and letter  templates, and added several new tools to the rating module. Additionally,  the department has established plans and a schedule for retiring one  legacy system. According to an analysis by VBA’s Office of Business  Process Integration, fewer than 5 percent of disability cases are rated  using the legacy rating system, Rating Board Automation 2000, and the  functionality available in VBMS-Rating has largely eliminated the need for  claims processors to access the legacy rating system.
In February 2015, VBA conducted a 5-week pilot at four regional offices  to determine whether VBA’s Office of Business Process Integration can  proceed with plans to retire the legacy rating application. According to  VBMS program officials, the Office of Business Process Integration  expects to retire the legacy rating system by the end of September 2015.  According to the PMO Director, the office has also been working to  capture process improvements and use lessons learned from earlier  system development cycles to address the generation 4 strategic  objective to implement improved system development processes.
Even with the progress VBA has made toward developing and  implementing VBMS, the timeline for initial deployment of a national  workload management capability has been delayed beyond the originally  planned date of September 2014 to October 2015, with additional  deployment to occur throughout fiscal year 2016. VBMS generation 4  development efforts have included addressing the strategic objective that  calls for delivery of a national workload management capability. This  effort has entailed developing the technology and business processes  needed to support the national work queue, which is intended to handle  new disability claims in a centralized queue and assign claims to the next  regional office with available capacity.
The PMO began work for the national work queue in June 2014. The  office had intended to deploy the first phase of the work queue  functionality to users in September 2014. However, in late May 2015, the  PMO Director informed us that VBA had decided to delay the initial rollout  of the work queue to October 2015 so that the department can fully focus  on meeting its goal to eliminate the claims backlog by the end of  September 2015. Following the initial rollout, the PMO intends to  implement the work queue at all regional offices through fiscal year 2016.
VBMS program documentation also identifies additional work to be  performed after fiscal year 2015 to fully automate disability claims  processing. Specifically, the PMO has identified the need to automate  steps associated with a veteran’s request for an increase in disability  benefits, such as when an existing medical condition worsens. In addition,  according to the Director, the PMO intends to develop a capability to  automatically associate veterans’ correspondence when a new piece of  evidence to support a claim has been received electronically or scanned  into VBMS. The PMO also plans to integrate VBMS with VA’s Integrated  Disability Evaluation System, which contains the results of veterans’  disability medical examinations, as well as with external systems that  contain military service treatment records for veterans, including those at  the National Personnel Records Center.
Further, VBA has not yet developed and implemented end-to-end pension  processing capabilities in VBMS. Without such capabilities, VBA  continues to rely on three legacy systems to process pension claims.  Specifically, according to program officials, both the Modern Award  Processing–Development and Award legacy systems contain functionality  related to processing pensions and will need to remain operational until  VBMS can process these claims. In addition, they said that the Share  legacy system contains functionality that is still needed throughout the  claims process.
VBMS program officials stated that additional system analysis is needed  before they can develop plans for the retirement of Share. Program  documentation indicates that the first phase of pension-related  functionality is expected to be introduced in December 2015. However,  VBA has not yet developed plans and schedules for retiring the Modern  Award Processing-Development, Award, and Share systems and fully  developing and implementing the functionality of these legacy systems in  VBMS.
VBA’s progress toward developing and implementing appeals processing  capabilities in VBMS has also been limited. Specifically, although the  information in a veteran’s eFolder is available to appeals staff for review,  the appeals process for disability claims is not managed using the new  system. According to VA’s fiscal year 2016 budget submission, the  department is pursuing a separate effort to manage end-to-end appeals  modernization, and has requested $19.1 million in fiscal year 2016 funds  to develop a system that will provide functionality not available in VBMS  or other VA systems. The PMO Director stated that VBA is currently  analyzing commercial IT solutions that can meet the business  requirements for appeals, such as providing document navigation  capabilities. Nevertheless, the Director added that VBMS is expected to  be part of the appeals modernization solution because components of the  system, such as the VBMS eFolder and certain workload management  functionality, are planned to continue supporting appeals management.
According to the PMO Director, the fact that VBMS requires additional  development beyond 2015 does not reflect a delay in completing the  system’s development. Instead, the additional time is a consequence of  decisions to enlarge the VBMS program’s scope over time. This official  added that VBMS’s original purpose was to serve primarily as an  electronic document repository, and they have met this goal.
The PMO Director further stated that, as the program’s mission has  expanded to support the department’s efforts to eliminate the disability  claims backlog, the PMO has had to re-prioritize, add, and defer system  requirements to accommodate broader departmental decisions and, in  some cases, regulatory changes. For example, the PMO was tasked with  developing functionality in VBMS to meet regulatory requirements for  processing disability claims using mandatory forms. Officials from the  VBMS PMO explained that they were made aware of this requirement  well after system planning for the March 2015 release had been  completed, and it introduced significant complexity to their development  work.
Finally, VBA included in its strategic road map the strategic objectives  that are to be addressed in generation 5 of the system, which is planned  for fiscal year 2016. Further, officials from the VBMS PMO stated that  they intend to develop tactical plans that identify the expected capabilities  to be provided in the generation 5 releases. Nevertheless, due to the  department’s incremental approach to developing and implementing  VBMS, VBA has not yet produced a plan that identifies when the system  will be completed and can be expected to fully support disability and  pension claims processing and appeals. Thus, it will be difficult for VA to  hold its managers accountable for meeting its time frame and for  demonstrating progress.
section_title: Ongoing Progress of VBMS Development and Implementation May Be Hindered by Lack of a Reliable Cost Estimate
paragraphs: Consistent with our guidance on estimating program costs, an important  aspect of planning for IT projects such as VBMS involves developing a  reliable cost estimate to help managers evaluate a program’s affordability  and performance against its plans, and provide estimates of the funding  required to efficiently execute a program. Without this information,  programs are at risk of experiencing cost overruns, missed deadlines,  and performance shortfalls. Additionally, federal guidance and IT project  management principles stress the importance of continuous planning  throughout the life of a program to serve as a basis for managing trade- offs between cost, schedule, and scope.
In July 2012, we identified several weaknesses in VA’s policies related to  cost estimating and also found that the department’s 2011 cost estimate  for VBMS did not reflect key practices for developing an accurate and  credible estimate. At that time, we recommended that VA modify its  policies governing cost estimating to establish, among other things, a  requirement to prepare a full life-cycle cost estimate for a program (as  opposed to preparing an estimate for each program increment) and a  requirement for programs to prepare cost estimates using best practices.  In February 2013, VA’s Office of Corporate Analysis and Evaluation  released a cost estimating process guide, which states that VA program  offices are required to develop a life-cycle cost estimate to support senior  leaders’ decision making and the department’s planning and budgeting  processes. Further, this guide states that, at the request of VA leadership  or a program office, VA’s Office of Corporate Analysis and Evaluation is  to be available to develop independent cost estimates and review existing  program cost estimates to ensure they have been developed in  accordance with our cost-estimating guidance.
In 2011, VBA submitted a life-cycle cost estimate for VBMS of $934.8  million to the Office of Management and Budget. This estimate was  intended to capture costs for the system’s development, deployment,  sustainment, and general operating expenses through the end of fiscal  year 2018. However, as of July 2015, the program’s actual costs  exceeded the 2011 life-cycle cost estimate. Specifically, VBMS has  received approximately $1 billion in funding through the end of fiscal year  2015 and the department has requested an additional $290 million for the  program in fiscal year 2016.
The PMO has not reliably updated VBMS’s life-cycle cost estimate to  reflect the program’s expanded scope and timelines for completing the  system. This is largely attributable to the fact that the PMO has developed  cost estimates for 2-year project cycles that are used for VBMS milestone  reviews under OI&T’s Project Management Accountability System.
When asked how the PMO arrived at the cost estimates reported in the  milestone reviews, officials stated that they develop rough order of  magnitude estimates for each business need based on expert knowledge  of the system, past development and engineering experience, and  lessons learned. However, while this approach may have provided  adequate information for VBA to prioritize VBMS system requirements to  address in the next release, it has not produced estimates that could  serve as a basis for identifying VBMS’s funding needs. Because it is  typically derived from limited data and in a short time, a rough order of  magnitude analysis is not equivalent to a budget-quality cost estimate and  may limit an agency’s ability to identify the funding necessary to efficiently  execute a program.
In addition, the PMO’s annual operating plan, which is generally limited to  high-level information about the program’s organization, priorities,  staffing, milestones, and performance measures for fiscal year 2015, also  shows estimated costs totaling $512 million for VBMS development from  fiscal years 2017 through 2020. However, according to the PMO Director,  this estimate was also developed using rough order of magnitude  analysis. Further, the estimate does not provide reliable information on  life-cycle costs because it does not include estimated IT sustainment and  general operating expenses.
Even though the PMO has developed rough order of magnitude cost  estimates for VBMS, these estimates have not been sufficiently reliable to  effectively identify the program’s funding needs. Instead, during the last 3  fiscal years, the PMO Director has had to request an additional $118  million in IT development funds to meet program demands and to ensure  support for ongoing development contracts. Specifically, in May 2013,  VA requested $13.3 million to support additional work on VBMS. Then,  during fiscal year 2014, VA reprogrammed $73 million of unobligated IT  sustainment funds to develop functionality to transfer service treatment  records from DOD to VA, and to support development of VBMS-Core  functionality. In December 2014, the PMO identified the need for  additional fiscal year 2015 funds for ongoing system development  contracts for VBMS-Core and VBMS-Awards, and, in late April 2015, VA  leadership submitted a letter to Congress requesting permission to  reprogram $31.7 million to support work on these contracts, the National  Work Queue, and other VBMS efforts.
According to the PMO Director, the need to request additional funding  does not represent additional risk to the program, but is the result of  VBMS’s success. The Director further noted that, as the PMO has  identified opportunities to increase functionality to improve the electronic  claims process, their funding needs have also increased. We recognize  that as new capabilities are deployed, additional requirements may  surface. Nevertheless, evolution of the VBMS program illustrates the  importance of continuous planning, including cost estimating, so that  trade-offs between cost, schedule, and scope can be effectively  managed. Further, without a reliable estimate of the total costs associated  with completing work on VBMS, stakeholders will have a limited view of  VBMS’s future resource needs and the program is at risk of not being  able to secure appropriate funding to fully develop and implement the  system.
section_title: VBA Has Made Progress toward Improving VBMS Implementation, but Does Not Have Goals for System Response Times
paragraphs: GAO and federal IT guidance recognize the importance of defining  program goals and related performance targets and using such targets to  assess progress in achieving the goals. System performance and  response times have a large impact on whether staff successfully  complete work tasks. If systems are not responding at agreed-upon levels  for availability and performance, it can be difficult to ensure that staff will  complete tasks in a timely manner. This is especially important in the VBA  claims processing environment, where staff are evaluated on their ability  to process claims in a timely manner.
VBA’s availability goal for VBMS is 95 percent per month, operating on a  24 hour-a-day, 6-day-a-week schedule. According to the OI&T Program  Manager for VBMS, VBA defines availability as the extent to which the  system is operational, functional, and usable for completing business  requirements (e.g., processing claims). When an unscheduled system  outage occurs, the office identifies it as being in one of three categories:  critical, impacting all VBMS users across VA; moderate, impacting VBMS  users at a local or individual level; and serious, impacting VBMS users in  a certain regional area. The Program Manager stated that the department  does not consider the system to be unavailable if users are unable to  perform their duties due to non-system issues (such as local network  issues or the unavailability of systems that feed data into VBMS).
VBA has reported that, since its initial rollout in January 2013, the system  has exceeded the 95 percent goal for availability. Specifically, VBA  reported that the system was available at a rate of 98.9 percent in fiscal  year 2013 and 99.3 percent in fiscal year 2014. Through May of fiscal  year 2015, it was available for 99.98 percent of the time.
Nonetheless, while the department reported exceeding its availability  goals for VBMS, it has experienced periods of system unavailability,  many times at the critical level affecting all users, as reflected in figure 2.
Specifically, since January 2013, the department has reported 57 VBMS  outages that have totaled about 117 hours of system unavailability. VBA  experienced about 18 hours of VBMS outages in January 2014, which  were almost entirely at the critical level and affected all users. To the  department’s credit, it reported experiencing only 2 system outages since  July 2014—a 30-minute critical outage in December 2014 and a 23- minute critical outage in May 2015.
In addition to system availability, VBA monitors system response times for  each of the VBMS modules using an application that measures the  amount of time taken for each transaction. The PMO defines response  time as the time that elapses from when a user executes a transaction  (i.e., clicks a link or selects “Enter”) to when the resulting page fully loads  on the user’s screen. The PMO uses average page response times to  measure VBMS system performance, with lower response times  indicating optimal system performance and higher response times  indicating performance issues.
From September 2013 through April 2015, VBA reported a decrease in  average response times for VBMS-Core and VBMS-Rating (see table  2).
VBA attributed the decrease in response times to continuous engineering  improvements to system performance. Program officials also explained  that the difference in response times between modules was due to the  type of information that is being pulled into each module from various  other VBA systems. For example, both VBMS-Core and VBMS-Rating  require information from the VBA corporate database, but VBMS-Core is  populated with data from multiple VBA systems in addition to the  corporate database.
Program officials told us there are no specific goals for mean transaction  response times because they feel that there are adequate tools in place  to monitor system performance and provide alerts if there are response  time issues. For example, VBMS performance is monitored in real time by  dedicated staff at a contractor’s facility and users have access to a live  chat feature where they can provide feedback on any issues they are  experiencing with the system.
The VBMS help desk provides another avenue for users to provide  feedback on the system’s performance. Officials also noted that, because  transaction response times have decreased, which can be indicative of an  improvement to system performance, they are focusing their resources on  adding additional functionality instead of trying to get the system to  achieve a specific average transaction response time.
While VBA’s monitoring of VBMS’s performance is commendable and the  system’s performance and response time have improved over time, the  system is still in development and there is no guarantee that performance  will remain at current levels as the system evolves. Performance targets  and goals for VBMS response times would provide users with an  expectation of the system response times they should anticipate, and  management with an indication of how well the system is performing  relative to performance goals.
section_title: VBMS Program Has Actively Managed System Defects, but a Recent Release Included Unresolved Defects that Adversely Impacted System Implementation
paragraphs: A key element of successful system testing is appropriately identifying  and handling defects that are discovered during testing. Outstanding  defects can delay the release of functionality to end users, denying them  the benefit of features. Key aspects of a sound defect management  process include the planning, identification and classification, tracking,  and resolution of defects. Leading industry and government organizations  consider defect management and resolution to be among the primary  goals of testing.
The VBMS program has defect management policies in place and it is  actively performing defect management activities. Specifically, in October  2012, the department developed a VBMS Program Management and  Technical Support Defect Management Plan, Version 1.0, which  describes the program’s defect management process. The plan was  updated in March 2015 and describes, among other things, the process  for identifying, classifying, tracking, and resolving VBMS defects. For  example, it provides criteria for assigning four different severity levels for  defects—critical, high, medium, and low.
According to the plan, critical severity defects are characterized by  complete system or subsystem failure, complete loss of functionality, and  compromised security or confidentiality. Critical defects also have  extensive user impact and workarounds do not exist. High severity  defects can have major user impact, leading to significant loss of system  functionality. Medium severity defects can have moderate user impact  and lead to moderate loss of functionality. Low severity defects lead to  minor loss of functionality with no workaround necessary. For high and  medium severity defects, workarounds could exist. According to the PMO,  high, medium, and low defects do not need to be resolved prior to a  system release.
To monitor and track defects, the PMO uses an automated tool to  maintain the VBMS defect repository. This tool is used to produce a daily  defect management report that is shared with VBMS leadership. The  purpose of the daily defect management report is to provide the current  status of all open defects identified in testing of a forthcoming VBMS  release or identified during production of a previous release.
According to the defect management plan, defects can be resolved in a  number of different ways. Once a defect is fixed, tested, and has passed  testing, it is considered done or resolved. Defects that cannot be  attributed to an existing requirement are reclassified as a system  enhancement and considered resolved, as they do not affect a current  system release requirement. A defect is also considered resolved if it is  determined to work as designed, duplicate another defect, or if it is no  longer evident in the system.
From March 2014 through March 2015, the total number of VBMS defects  declined as release dates approached for releases 7.0, 7.1, 8.0, and 8.1.  Additionally, to the department’s credit, no critical defects remained at the  time of each of these releases. Specifically, prior to VBMS Release 7.0,  the total number of defects identified peaked at 226, while 97 remained at  the time of release. Prior to release 7.1, the total number of defects  identified peaked at 330, with 169 remaining at the time of release. Prior  to release 8.0, the total number of defects identified peaked at 330, with  161 remaining at the time of release. Prior to release 8.1, the total  number of defects remaining peaked at 421, with 254 remaining at the  time of release. Figure 3 shows the trend in the number of open total  defects for this time period.
Even with the department’s efforts to resolve defects prior to a VBMS  release, defects that affected system functionality remained open at the  time of the releases. Specifically, of the 254 open defects at the time of  VBMS release 8.1, 76 were high severity, 99 were medium severity, and  79 were low severity. Examples of defects that remained open at the time  of VBMS release 8.1 are described in table 3.
According to the PMO, these defects were communicated to users and an  appropriate workaround for each was established. Nevertheless, even  with workarounds, high- and medium-severity open defects, which by  definition impact system functionality, degraded users’ experiences with  the system. Continuing to deploy system releases with defects that  impact system functionality increases the risk that these defects will  diminish users’ ability to process disability claims in an efficient manner.
section_title: VBA’s Methods for Determining User’s Feedback on VBMS Have Not Included Establishing Goals or Conducting a Survey; GAO’s Survey Identified Varied Satisfaction Levels
paragraphs: While VBA has several methods to obtain VBMS users’ feedback, it has  neither established goals to define user satisfaction, nor conducted a  survey of claims processing employees to obtain a comprehensive picture  of overall customer satisfaction. Our survey of VBMS users found that a  majority reported satisfaction with the system, but decision review officers  were considerably less satisfied. Although the results of our survey  provide VBA with useful data about users’ satisfaction with the system,  the absence of user satisfaction goals limits the utility of survey results.
subsections: section_title: VBA Has Several Methods to Obtain Users’ Feedback on VBMS, but Has Not Established User Satisfaction Goals or Conducted a Survey
paragraphs: GAO and federal IT guidance recognize the importance of defining  program goals and related performance targets and using such targets to  assess progress in achieving the goals. Also, leading practices identify  continuous customer feedback as a crucial element of IT project success,  from project conception through sustainment. Particularly for IT projects  like VBMS, where development activities are iterative, customer (i.e., end  user) perspectives and insights can be solicited through various  methods—user acceptance testing, interviews, complaint programs, and  satisfaction surveys—to validate or raise questions about the project’s  implementation.
Further, leading practices emphasize that periodic customer satisfaction  data be proactively used to improve performance and demonstrate the  level of satisfaction the project is delivering. The Office of Management  and Budget has developed standards and guidelines in survey research  that are generally consistent with best practices and call for statistically  valid data collection efforts to be used in fulfilling agencies’ customer  service data collection. These leading practices also stress the  importance of centrally integrating all customer feedback data in order to  have more complete diagnostic information to guide improvement  efforts.
VA has used a variety of methods for obtaining customer or end user  feedback on the performance of VBMS. For example, the department  solicits end user involvement and feedback in the iterative system  development process based on user acceptance criteria. According to  the Senior Project Manager for VBMS Development within OI&T, at the  end of each development cycle and before a new version of VBMS is  deployed, end users are involved in user acceptance testing and a final  customer acceptance meeting.
VA also provides training to a subset of end users—known as  “superusers”—on the updated functionality introduced in a new version of  VBMS. These superusers are then expected to train the remaining users  in the field on the new version’s features. The department tracks the  overall satisfaction level with training received after each VBMS major  release. However, this tracking is limited to superusers’ satisfaction with  the training, rather than their satisfaction with the system.
VA also solicits customer feedback about the system through interviews.  For example, the PMO Director stated that the Under Secretary for  Benefits hosts a weekly phone call with bargaining unit employees as a  “pulse check” on VBA transformation activities, including VBMS.  According to this official, the VBA Office of Field Operations also offers an  instant messaging chat service to all regional office employees to solicit  feedback about the latest VBMS functionality deployment.
Another method in which the department obtains customer input is  through a formal feedback process. For example, according to the PMO  Director, VA provides national service desk support to assist users in  troubleshooting system issues and identifying system defects. In addition,  VBMS applications include a built-in feature that enables users to provide  feedback to the PMO on problems with the system. According to the  Director, the feedback received by the office also helps to identify user  training issues.
Nevertheless, while VA has taken steps to obtain feedback on the  performance and implementation of VBMS, it has not established goals to  define user satisfaction that can be used as a basis for gauging the  success of its efforts to promote satisfaction with the system. Further,  although the efforts that have been taken to solicit users’ feedback  provide VBA with useful insights about particular problems, data are not  centrally compiled or sufficient for supporting overall conclusions about  whether customers are satisfied. In addition, VBA has not employed a  customer satisfaction survey of claims processing employees who use  the system on a daily basis to process disability claims. Such a survey  could provide a more comprehensive picture of overall customer  satisfaction and help identify areas where VBMS development and  implementation efforts might need additional attention.
According to the PMO Director, VBA had not used a survey to solicit  feedback because of concern that such a mechanism may have  negatively impacted the efficiency of claims processors in completing  disability compensation claims on behalf of veterans. Further, the director  believed that the office had the benefit of receiving ongoing end user  input on VBMS by virtue of the intensive testing cycles, as well as several  of the other mechanisms noted previously by which end users have  provided ongoing feedback. Nevertheless, without establishing user  satisfaction goals and collecting the comprehensive data that a  statistically valid survey can provide, the PMO limits its ability to obtain a  comprehensive understanding of VBMS users’ levels of satisfaction with  the system. Thus, VBA could miss opportunities to improve the efficiency  of its claims process by increasing satisfaction with VBMS.
section_title: A Majority of VBMS Users, with the Exception of Decision Review Officers, Reported Satisfaction with the System
paragraphs: In response to the statistical survey that we administered, a majority of  VBMS users (i.e., VBA claims processors) were satisfied with the system  that had been implemented at the time of the survey. These users  (which represent claims assistants, veteran service representatives,  supervisory veteran service representatives, rating veterans service  representatives, decision review officers, and others) were satisfied with  the three modules of VBMS. Specifically, an estimated 59 percent of the  population of claims processors were satisfied with VBMS-Core; an  estimated 63 percent were satisfied with the Rating module, and an  estimated 67 percent were satisfied with the Awards module. Figure 4  depicts the estimated percentage of claims processors who were satisfied  with VBMS.
Although a majority of users were satisfied with the three modules,  decision review officers were considerably less satisfied than other users  with VBMS-Core and VBMS-Rating. Specifically, for VBMS-Core, an  estimated 27 percent of decision review officers were satisfied compared  to an estimated 59 percent of all roles of claims processors (including  decision review officers) who were satisfied. In addition, for VBMS-Rating,  an estimated 38 percent of decision review officers were satisfied,  compared to an estimated 63 percent of all roles of claims processors.
Figure 5 depicts the estimated satisfaction levels of decision review  officers in comparison to all roles.
Decision review officers were considerably less satisfied with VBMS in  comparison to all roles of claims processors in additional areas. For  example, an estimated 26 percent of decision review officers viewed  VBMS-Core as an improvement over the previous legacy system or  systems for establishing claims and storing and reviewing electronic  documents related to a claim in an eFolder. In contrast, an estimated 58  percent of all users (including decision review officers) viewed the Core  module as an improvement. In addition, an estimated 26 percent of  decision review officers viewed VBMS-Rating as an improvement over  the previous systems with respect to providing Web-accessible tools,  including rules-based rating calculators, to assist in making claims rating  decisions. In contrast, an estimated 55 percent of all roles of claims  processors viewed the Rating module as an improvement. For VBMS- Awards, an estimated 61 percent of all roles viewed this module as an  improvement over the previous systems to automate the award and  notification process. Figure 6 depicts the estimated percentage of  decision review officers, in comparison to all claims processors, who  viewed VBMS as an improvement over legacy systems.
Similarly, for the three modules, a majority of users (including decision  review officers) would have chosen VBMS over the legacy system or  systems. However, decision review officers indicated that they were less  likely to have chosen VBMS-Core and VBMS-Rating over legacy  systems. Specifically, an estimated 27 percent of decision review officers  would have chosen VBMS-Core compared to an estimated 60 percent of  all roles of claims processors. In addition, an estimated 27 percent of  decision review officers would have chosen VBMS-Rating compared to  61 percent of all roles who would have chosen the system over the legacy  system or systems. For VBMS-Awards, an estimated 67 percent of all  roles would have chosen this module over the previous systems. Figure 7  depicts the estimated percentage of decision review officers, in  comparison to all claims processors, who would have chosen VBMS  instead of the legacy systems to process claims.
Decision review officers perform an array of duties to resolve claims  issues raised by veterans and their representatives. They may also  conduct a new review or complete a review of a claim without deference  to the original decision and in doing so, must click through all documents  included in the e-Folder. Survey comments from decision review officers  stated, for example, that reviews in the VBMS paperless environment  take longer because of the length of time spent loading, scrolling, and  viewing each document (particularly if the documents are large, such as a  service medical record file). Additionally, multiple decision review officers  commented that it is easier and faster to review documents in a paper file.  Although such comments provide illustrative examples of individual  decision review officer’s views and are not representative, according to  the PMO Director, decision review officers’ relative dissatisfaction is not  surprising because the system does not yet include functionality that  supports their work, which primarily relates to appeals processing.
section_title: Conclusions
paragraphs: Even though VA has made progress toward completing the development  and implementation of VBMS, there is more work to be done and VBA  can improve management of its ongoing efforts. While 95 percent of  records related to veterans’ disability claims are electronic and reside in  VBMS, additional capabilities to fully process disability claims will be  delayed beyond fiscal year 2015, which is when completion was originally  planned. Further, VA’s incremental approach to developing and  implementing VBMS has not yet produced a plan that identifies when the  system will be completed and can be expected to fully support disability  compensation and pension claims processing and appeals. Thus, it will  be difficult for VA to hold its managers accountable for meeting its time  frame and for demonstrating progress.
VBA’s management of the system highlights areas that could benefit from  improvement as development and implementation of the system continue.  Specifically, without a reliable estimate of the total costs associated with  completing work on VBMS, the department’s stakeholders have only a  limited view of future resource needs, and the program risks not having  sufficient funding to complete development and implementation of the  system. Additionally, established goals for system response times would  provide users with an expectation of the response times they can  anticipate, and management with an indication of how well the system is  performing relative to performance goals. Furthermore, continuing to  deploy system releases with large numbers of defects that reduce system  functionality could adversely affect users’ ability to process disability  claims in an efficient manner.
Without user satisfaction goals and the data a customer satisfaction  survey could yield, VA may miss opportunities to collect important data on  how users view the system’s performance, and ultimately, to improve the  system. Our survey of VBMS users found that a majority of them were  satisfied with the system, but decision review officers were considerably  less satisfied. Although the results of our survey provide VBA with useful  data about users’ satisfaction with VBMS, the absence of user  satisfaction goals limits the utility of survey results. Specifically, without  having established goals to define user satisfaction, VBA does not have a  basis for gauging the success of its efforts to promote satisfaction with the  system or for identifying areas where its efforts to complete development  and implementation of the system might need attention.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Veterans Affairs direct the Under  Secretary for Benefits and the Chief Information Officer to take the  following five actions to improve VA’s efforts to effectively complete the  development and implementation of VBMS:
Develop an updated plan for VBMS that includes (1) a schedule for  when VBA intends to complete development and implementation of  the system, including capabilities that fully support disability claims,  pension claims, and appeals processing and (2) the estimated cost to  complete development and implementation of the system.
Establish goals for system response time and use the goals as a  basis for periodically reporting actual system performance.
Reduce the incidence of high- and medium-priority level defects that  are present at the time of future VBMS releases.
Develop and administer a statistically valid survey of VBMS users to  determine the effectiveness of steps taken to make improvements in  users’ satisfaction.
Establish goals that define customer satisfaction with VBMS and  report on actual performance toward achieving the goals based on the  results of GAO’s survey of VBMS users and any future surveys VA  conducts.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report (reprinted in  appendix III). In addition, VA provided technical comments, which we  incorporated, as appropriate.
In its comments, VA generally agreed with our conclusions. The  department also concurred with our recommendations and described  actions it is planning to take in response to four of our five  recommendations. Specifically, VA concurred with our recommendation  calling for an updated plan for VBMS. Although it described the  importance of VBMS to ensuring timely delivery of benefits to veterans  and recognized the need for continued investment in the system, the  department did not, however, identify actions to develop a VBMS plan  that includes a schedule for when VBA intends to complete development  and implementation of the system and the estimated cost of doing so. We  believe development of such a plan is an important action to help ensure  effective development and implementation of VBMS and to hold  managers accountable.
With regard to the remaining four recommendations, VA described  actions it is planning to take in response to each. For example, with  regard to our recommendation to establish goals for system response  time, the department stated that the VBMS program is participating in a  review of service-level agreements to establish metrics for the system’s  performance. Additionally, regarding our recommendation that it reduce  the incidence of high and medium priority level defects that are present in  future VBMS releases, the department reiterated its plans and procedures  for decreasing the defects in each release. With respect to our  recommendation to conduct a survey of VBMS users, the department  stated that the VBMS PMO is working with the Office of Field Operations  and labor partners regarding the distribution of a survey to measure  users’ satisfaction and expects to release a survey in March 2016.  Consistent with our final recommendation, the department is also  planning to establish customer satisfaction goals and report on actual  performance toward achieving the goals after all survey results are  received and analyzed with a target of July 2016 for completing this  action.
If VA develops the updated plan, including schedule and cost, for VBMS  as we recommended and follows through on the actions it described in  response to our remaining recommendations, the department will be  better positioned to effectively complete the development and  implementation of VBMS and to more effectively provide benefits and  services to our nation’s veterans.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to appropriate  congressional committees, the Secretary of Veterans Affairs, and other  interested parties. In addition, the report is available at no charge on the  GAO website at http://www.gao.gov.
If you or your staff have questions about this report, please contact me at  (202) 512-6304 or melvinv@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this study were to (1) assess the Department of  Veterans Affairs’ (VA) progress toward developing and implementing the  Veterans Benefits Management System (VBMS) and (2) determine to  what extent users report satisfaction with the system.
To assess the department’s progress toward developing and  implementing VBMS, we obtained and reviewed program plans and other  documentation that articulated the system’s goals and expected benefits.  These included the VA Strategic Plan to Eliminate the Compensation  Claims Backlog (2013), VBMS Strategic Roadmap, VBMS Tactical  Roadmap, VBMS Operating Plans, VBA’s 2011 life-cycle cost estimate  for the program, documentation supporting VBMS’s progression through  VA’s milestone review process, VA’s annual budget submissions, and  contracts and related contractor documentation. We compared program  plans and other documentation articulating the system’s goals, expected  benefits, and the system functionality expected to be delivered with  documents showing VBMS’s progress, such as monthly program  management reviews and cost and performance reports. We also  compared program plans to VA’s policies for incremental system  development, as well as federal guidance and IT project management  principles on program planning and cost estimating.
In addition, we obtained reports documenting the system’s availability and  performance and analyzed information in the reports to determine trends  in system availability and response time. We compared system  performance data with GAO and federal IT guidance for defining program  goals and related performance targets that can be used to assess  progress in achieving the goals. We assessed the reliability of the data  by reviewing it for obvious errors and missing data; corroborating the data  with related documentation; and interviewing responsible officials about  their use of an automated tool that monitors system performance. We  determined the data to be sufficiently reliable for the purposes of this  report.
We reviewed the VBMS Defect Management Plan and compared the plan  with key principles of sound defect management. We also compared the  processes described in the plan with actions taken to manage defects  identified for VBMS. We conducted analysis of critical defects identified  for VBMS releases 7.0, 7.1, 8.0, and 8.1 to determine whether they  remained open at the time of system release.
Further, we reviewed the department’s methods for soliciting end user  feedback on the performance of the system (e.g., VBMS training strategy,  operational management reviews, cost and performance reports, and end  of month reports). We compared these methods to leading practices for  obtaining customer feedback and collecting customer service data to  improve performance and demonstrate customer satisfaction of IT  projects.
To determine the extent that users report satisfaction with the system, we  conducted a Web-based survey of a nationally representative stratified  random sample of disability compensation claims processors. The survey  was administered during the time frame of September through November  2014. We developed survey questions with input from officials in the  VBMS PMO and VA’s Office of Information and Technology.
We pretested versions of the draft survey and observed claims  processors to gain initial insight regarding their use of the system in  VBA’s Baltimore, Maryland, and Philadelphia, Pennsylvania, regional  offices. We selected these offices based on their large size and location.  We revised the draft survey based on comments received during the  pretests. Once finalized, the claims processors included in the sample  were sent an e-mail that asked them to complete the survey, which was  available to them over the Web from September 30, 2014, to November  19, 2014. Following the initial request, we e-mailed weekly follow-up  requests to any nonrespondents.
Our eligible population for this survey consisted of all 10,622 disability  compensation claims processors employed by the agency as of July 31,  2014. To determine whether VBMS experiences differed based on  position and office workload, we designed a stratified random sample of  claims processors, with strata defined by position and office workload.  VBA provided us lists of these claims processors with information that  indicated their position and office. We determined average office  workload based on information published in VBA’s public weekly  workload reports. We divided the list of claims processors into 10 strata  based on their position at both smaller and larger workload regional  offices. Table 4 shows the size of each eligible population, the size of the  sample drawn from each eligible population, and the number of claims  processors in each stratum that responded to the survey.
We drew an independent random sample from each of these strata to  enable us to project survey results to all VBA claims processors, in  general, and to claims processors in each of the positions listed above,  within small and large workload offices. Of our total sample of 3,475  claims processors, 2,098 responded to the survey for an overall response  rate of 60 percent.
To produce estimates regarding the experiences and views of claims  processors from the survey responses of those included in our sample,  we analyzed the data with methods that are appropriate for a stratified  random sample using analysis weights. We weighted each response from  claims processors in each stratum to statistically account for all members  of that stratum. Because estimates are based on responses from a  sample, each estimate we report has a measurable precision or sampling  error. The sampling error or margin of error surrounding an estimate is  expressed as a number of percentage points higher or lower than that  estimate and the entire range that the sampling error covers is referred to  as a confidence interval. Confidence intervals are calculated based on a  certain confidence level. Confidence intervals for estimates we report  from this survey are based on a confidence level of 95 percent and are  calculated using methods appropriate for a stratified random sample.  Confidence intervals for percentage estimates in this report are never  wider than plus or minus 5 percentage points. At a 95 percent confidence  level, this means that in about 95 out of 100 instances, the sampling  procedures we used would be expected to produce a confidence interval  (in this case, a 10 percentage point range) containing the true population  value we estimate.
In addition to sampling error, estimates based on survey results are  subject to what is referred to as nonsampling error that can result from,  among other things, poorly designed survey questions and mistakes in  data entry or analysis or nonresponse. We took a number of steps in  developing the survey and in entering and analyzing the data to minimize  nonsampling error. For example, a social science survey specialist  collaborated with our subject matter experts in designing the survey. In  addition, it was reviewed by VBA officials and, as previously noted, was  pretested with a number of different types of claims processors in two  locations. Also, when we analyzed the data, an independent analyst  verified all computer programs. Because this was a Web-based survey,  respondents’ answers to survey questions were automatically entered  into an electronic file, eliminating the need to separately key responses  into a data file, further minimizing the potential for nonsampling error.
We also took steps to mitigate potential nonresponse error. For example,  we used follow-up e-mails to remind users to complete the survey in order  to reduce nonresponse. Further, we adjusted for characteristics that were  associated with survey response propensity using standard weighting  class adjustments defined by sampling strata. We assumed that  nonresponse adjusted data were missing at random, and therefore  concluded the respondent analyses using the nonresponse adjusted  weights were unbiased for the population of VBMS users sampled in the  survey.
We supplemented our analyses with interviews of VA officials that had  knowledge of the VBMS program, including officials in VA’s Office of  Information and Technology and the Veterans Benefits Administration’s  VBMS PMO and Office of Field Operations.
We conducted this performance audit from March 2014 through August  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Survey of Veterans Benefits Management System End User Experience
paragraphs: The questions we asked in our survey of Veterans Benefits Administration  (VBA) claims processors are shown here. Our survey was comprised of  closed- and open-ended questions. In this appendix, we include all survey  questions and aggregate results of responses to the closed-ended  questions. For a more detailed discussion of our survey methodology, see  appendix I.  1.  What is your current role in processing veterans’ disability  compensation claims?
If other, describe your role. (Open-ended response.)  2.  What responsibility (or responsibilities) do you have in this role  (Check all that apply.)
If other, describe your responsibility. (Open-ended response.)  3.  How many years have you worked within the VBA?
4.  The Department of Veterans Affairs’ (VA) records show that you  currently work at the (office name prepopulated) office. Is this  correct?
If no, enter the name of the office where you currently work. (If you work  in more than one office, enter the office where you spend most of your  time.) (Open-ended response.)  5.  How long have you been using the Veterans Benefits  Management System (VBMS)?
Response  Less than 6 months  95 percent  confidence  interval -   lower bound  6.9  95 percent  confidence  interval -  upper bound  9.8  6 months to less than   1 year  1 year to less than   2 years  2 years to less than   3 years  Not applicable, I do not use  VBMS (Skip to question 17.)
6.  In an average work week, how much time, if at all, do you spend  processing the following types of claims in VBMS? (Choose a  time period for each item listed.)
Response  6a. Fully electronic  claims  95 percent  confidence  interval -   lower bound  8.0  95 percent   confidence  interval -   upper bound  10.6  8 to less than 16  hours  16 to less than  24 hours  24 to less than  32 hours  32 to less than  40 hours  All week (40 or  more hours)
I do not process  these  6b. Fully paper- based claims  8 to less than 16  hours  16 to less than  24 hours  24 to less than  32 hours  32 to less than  40 hours  All week (40 or  more hours)
I do not process  these  6c. Hybrid claims  (that is, claims   that consist of both  paper and  electronic records)
8 to less than 16  hours  16 to less than  24 hours  8.7  24 to less than  32 hours  95 percent  confidence  interval -   lower bound  2.9  95 percent   confidence  interval -   upper bound  4.9  32 to less than  40 hours  All week (40 or  more hours)
7.  In addition to using VBMS, how much do you depend on each of  the following systems or applications to process disability  claims? (Choose a category for each system or application  listed.)
None or do not  use  95 percent  confidence  interval -  lower bound  49.0  95 percent  confidence  interval -  upper bound  52.9  7b. VETSNET:  Modern Award  Processing- Development  (MAP-D)
None or do not  use  7c. VETSNET:  Rating Board  Automation 2000  (RBA 2000)
Section 2—Veterans Benefits Management System-Core   (VBMS-Core)  8.  In the course of your work, do you use VBMS-Core?
Complete questions 8.1 and 8.2 for VBMS-Core. In question 8.3, you  will have a chance to comment on any of your answers after  responding to the questions.  8.1.  Do you agree or disagree with the following statements based  on your current experience using VBMS-Core? Select one  answer in each row.
Not  applicable or  no basis to  judge  8.1c. VBMS-Core  helps me be more  productive  compared to the  previous system(s)
Not  applicable or  no basis to  judge  8.1d. VBMS-Core  helps me be more  efficient compared  to the previous  system(s)
Not  applicable or  no basis to  judge  8.1f. VBMS-Core is  easier to use  compared to the  previous system(s)
Not  applicable or  no basis to  judge  8.1n. VBMS-Core  performance is  timely (e.g., minimal  time to load pages  and respond to  commands)
Response  8.1o. I believe  VBMS-Core helps to  reduce claims  processing times  compared to the  previous system(s)
Not  applicable or  no basis to  judge  8.1r. VBMS-Core is  an improvement  over the previous  system(s)
8.2.  While using VBMS-Core how often, if at all, do you experience  each of the following? Select one answer in each row.
Response  8.2a. The scanned  documents  displayed on the  screen within VBMS- Core are clearly  legible and correctly  oriented (i.e., not  upside-down)
8.3.  If you have any comments regarding your answers to  questions 8.1 and/or 8.2 concerning VBMS-Core, share them  here. (Open-ended response.)
Section 3—Veterans Benefits Management System-Rating   (VBMS-R)  9.  In the course of your work, do you use VBMS-R?
Complete questions 9.1 and 9.2 for VBMS-R. In question 9.3, you will  have a chance to comment on any of your answers after responding  to the questions.  9.1.  Do you agree or disagree with the following statements based  on your current experience using VBMS-R? Select one answer  in each row.
Not  applicable or  no basis to  judge  9.1c. VBMS-R helps  me be more  productive  compared to the  previous system(s)
Not  applicable or  no basis to  judge  9.1d. VBMS-R helps  me be more efficient  compared to the  previous system(s)
Not  applicable or  no basis to  judge  9.1f. VBMS-R is  easier to use  compared to the  previous system(s)
Not  applicable or  no basis to  judge  9.1n. VBMS-R  performance is  timely (e.g., minimal  time to load pages  and respond to  commands)
Not  applicable or  no basis to  judge  9.1o. I believe  VBMS-R helps to  reduce claims  processing times  compared to the  previous system(s)
Not  applicable or  no basis to  judge  9.1r. VBMS-R is an  improvement over  the previous  system(s)
9.2.  While using VBMS-R, how often, if at all, do you experience  each of the following? Select one answer in each row.
9.3.  If you have any comments regarding your answers to  questions 9.1 and/or 9.2 concerning VBMS-R, share them here.  (Open-ended response.)
Section 4—Veterans Benefits Management System-Awards   (VBMS-A)  10. In the course of your work, do you use VBMS-A?
Complete questions 10.1 and 10.2 for VBMS-A. In question 10.3, you  will have a chance to comment on any of your answers after  responding to the questions.  10.1. Do you agree or disagree with the following statements based  on your current experience using VBMS-A? Select one answer  in each row.
Not  applicable  or no basis  to judge  10.1c. VBMS-A helps  me be more  productive compared  to the previous  system(s)
Not  applicable  or no basis  to judge  10.1d. VBMS-A helps  me be more efficient  compared to the  previous system(s)
Not  applicable  or no basis  to judge  10.1f. VBMS-A is  easier to use  compared to the  previous system(s)
Not  applicable  or no basis  to judge  10.1n. VBMS-A  performance is timely  (e.g., minimal time to  load pages and  respond to  commands)
Not  applicable  or no basis  to judge  10.1o. I believe  VBMS-A helps to  reduce claims  processing times  compared to the  previous system(s)
Not  applicable  or no basis  to judge  10.1r. VBMS-A is an  improvement over the  previous system(s)
10.2. While using VBMS-A how often, if at all, do you experience  each of the following? Select one answer in each row.
Response  10.2a. VBMS-A gives  error messages that  clearly tell me how to  fix problems.
10.3. If you have any comments regarding your answers to  questions 10.1 and/or 10.2 concerning VBMS-A, please share them  here (Open-ended response.)
Section 5—VBMS Benefits and Challenges  11. How much, if at all, has VBMS improved your ability to do the  following? Select one answer in each row.
Not  applicable  or no basis  to judge  11d. Access claims  documentation on  demand (e.g., the  ability to electronically  view a Veteran’s  claim file at multiple  regional offices at the  same time)
Not  applicable  or no basis  to judge  11f. Other  improvement(s)
If other improvement(s), describe. (Open-ended response.)  12. Which of the following would you consider to be the single  greatest improvement in using VBMS?
Make more accurate rating  decisions  Use automated tools (e.g.,  standardized correspondence,  rating application evaluation  builder, rules-based calculators,  etc.) for quicker/more accurate  processing  Access claims documentation on  demand (e.g., the ability to  electronically view a Veteran’s  claim file at multiple regional offices  at the same time)
If other improvement, please describe the other VBMS improvement you  consider to be the greatest. (Open-ended response.)  13. How much of a challenge, if at all, has VBMS been to you in the  following areas? Select one answer in each row.
Response  13a. System  performance (e.g.,   slow response  times)
Not  applicable or  no basis to  judge  13b. System  access (e.g.,  system outages)
Response  13c. The need to  rely on previous  systems in addition  to VBMS (e.g.,  VETSNET  systems)
Not  applicable or  no basis to  judge  13d. Knowing  whether the  evidence required  to process claims is  complete in VBMS  (e.g., not knowing  whether all  documents have  been scanned and  added to the  veteran’s file)
Not  applicable or  no basis to  judge  13e. Other  challenge(s)
If other challenges(s), describe. (Open-ended response.)  14. Which of the following would you consider to be the single  greatest challenge in using VBMS?
Response  System performance (e.g., slow  response times)
System access (e.g., system  outages)
The need to rely on previous  systems in addition to VBMS  (e.g., VETSNET systems)
Knowing whether the evidence  required to process claims is  complete in VBMS (e.g., not  knowing whether all documents  have been scanned and added to  the veteran’s file)
If other challenge, describe the other VBMS challenge you consider to be  the greatest. (Open-ended response.)  15. If you could pick one change to be included in the next  release/update of VBMS, what would it be and why?   (Open-ended response.)  16. Please share any comments or suggestions for improvement you  have about the VBMS system. (Open-ended response.)
section_title: Appendix III: Comments from the Department of Veterans Affairs
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Mark T. Bird (Assistant Director),  Chris Businsky, Virginia Chanley, Heather A. Collins, Kavita Daitnarayan,  Kelly Dodson, Nancy Glover, Stuart Kaufman, Michael Little, Jamelyn  Payan, Scott Pettis, Martin L. Skorczynski, Roger M. Smith, Eric Trout,  Sonya Vartivarian, and Kevin Walsh made key contributions to this report.