Veterans Benefits Management System: Ongoing Development and Implementation Can Be Improved; Goals Are Needed to Promote Increased User Satisfaction
Released: 2015-09-15T00:00:00

Letter

Background

VA pays monthly compensation to veterans with service-connected disabilities (i.e., injuries or diseases incurred or aggravated while on active military duty) according to the severity of the disability. It pays additional compensation for certain dependent spouses, children, and parents of veterans. The department processes service-connected disability compensation benefits for about 3 million beneficiaries each month. It reported completing the processing of approximately 1.3 million disability rating claims for veterans and beneficiaries in fiscal year 2014— about 150,000 more than in fiscal year 2013. As of mid-June 2015, the department reported a backlog of approximately 137,000 disability claims, down from its peak of about 611,000 disability claims in March 2013.

VA has reported that the accuracy of claims decisions rose from a level of 83 percent in 2011 to 90 percent in 2014. However, even as VBA reported providing historic numbers of veterans with decisions on their claims in 2014, the number and complexity of claims continue to increase. For example, VBA reported for 2014 that its employees made decisions on 5.5 million medical issues within claims—a 101 percent increase in the number of issues since 2009.

Overview of VBA’s Disability Compensation Claims Process

Throughout the disability compensation claims process, VBA staff have various roles and responsibilities.

Claims assistants are primarily responsible for establishing the electronic claims folders to determine whether the dispositions of the claims and control actions have been appropriately identified.

Veteran service representatives are responsible for providing veterans with explanations regarding the disability compensation benefits programs and entitlement criteria. They also are to conduct interviews, gather relevant evidence, adjudicate claims, authorize payments, and input the data necessary to generate the awards and notification letters to veterans describing the decisions and the reasons for them.

Rating veterans service representatives are to make claims rating decisions and analyze claims by applying VBA’s schedule for rating disabilities (rating schedule) against claims submissions, and preparing rating decisions and the supporting justifications. They also are to inform the veteran service representative who then notifies the claimant of the decision and the reasons for the decision.

Supervisory veteran service representatives are to ensure that the quality and timeliness of service provided by VBA meets performance indicator goals. They are also responsible for the cost-effective use of resources to accomplish assigned outcomes.

Decision review officers are to examine claims decisions and perform an array of duties to resolve issues raised by veterans and their representatives. They may conduct a new review or complete a review of a claim without deference to the original decision; they also can revise that decision without new evidence or clear and obvious evidence of errors in the original evaluation.

The disability compensation claims process starts when a veteran (or other designated individual) submits a claim to VA, in paper or electronic form. If submitted electronically, a claim folder is created automatically.

When a paper claim is submitted, a claims assistant creates the electronic folder. Specifically, when a regional office receives a new paper claim, the receipt date is recorded electronically and the paper files (e.g., medical records and other supporting documents) are shipped to one of four document conversion locations so that the supporting documents can be scanned and converted into a digital image.

In the processing of both electronic and paper claims, a veteran service representative is to review the information supporting the claim and help identify any additional evidence that is needed to evaluate the claim, such as the veteran’s military service records, medical examinations, and treatment records from medical facilities and private medical service providers. Also, if necessary to provide support to substantiate the claim, the department is to perform a medical examination on the veteran.

Once all of the supporting evidence has been gathered, a rating veterans service representative evaluates the claim and determines whether the veteran is eligible for benefits. If so, the rating veterans service representative assigns a disability rating (expressed as a percentage). A veteran who submits a claim with multiple disabilities receives a single composite rating. If the veteran is due to receive compensation, an award is prepared and the veteran is notified of the decision.

A veteran can reopen a claim for additional disability benefits if, for example, the veteran experiences a new or worsening service-connected disability. If the veteran disagrees with the regional office’s decision on the additional claim, he or she may submit a written notice of disagreement to the regional office to appeal the decision, and may request to have the appeal processed at the regional office by a decision review officer or through the Board of Veterans’ Appeals. Figure 1 presents a simplified view of VA’s disability compensation claims process.

Veterans Benefits Management System

VBA began the transformation of its paper-intensive claims process to a paperless environment in March 2009. This effort became formally established as the Veterans Benefits Management System program in May 2010. VBA’s initial plans for VBMS emphasized the development of a paperless claims platform to fully support the processing of disability compensation and pension benefits, as well as appeals. The program’s importance was further elevated in January 2013 when the Secretary of Veterans Affairs identified the system as the enabling technology to meet the goal to eliminate the disability claims backlog in fiscal year 2015 and improve the efficiency and accuracy of all compensation claims processing to 125 days and 98 percent, respectively. In a March 2013 Senate Veterans Affairs Committee hearing, VA’s Under Secretary for Benefits stated that VBMS development was expected to be completed in 2015.

The program’s primary focus was to convert existing paper-based claims folders into electronic claim folders (eFolders) to allow VBA employees to access claims information and evidence in an electronic format. Beyond the establishment of eFolders, VBMS is also intended to streamline the entire disability claims process, from establishment through award, by automating rating decision recommendations, award and notification processes, and communications between VBA and the veteran throughout the claims life cycle; assist in eliminating the claims backlog and serve as the enabling technology for quicker, more accurate, and integrated claims processing in the future; and replace many of the key outdated legacy systems—which are still in use today—for managing the claims process, including:

Share—used to establish claims; it records and updates basic information about veterans and dependents.

Modern Award Processing–Development —used to manage the claims development process, including the collection of data to support the claims and the tracking of claims.

Rating Board Automation 2000—provides information about laws and regulations pertaining to disabilities, which are used by rating specialists in evaluating and rating disability claims.

Award—used to prepare and calculate the benefit award based on the rating specialist’s determination of the claimant’s percentage of disability. It is also used to authorize the claim for payment.

VBMS is to consist of three modules:

VBMS-Core is intended to provide the foundation for document processing and storage during the claims development process, including establishing claims; viewing and storing electronic documents in the eFolder; and tracking evidence requested from beneficiaries. The eFolder serves as a digital repository for all documents related to a claim, such as the veteran’s military service records, medical examinations, and treatment records from VA and Department of Defense medical facilities, and from private medical service providers. Unlike with paper files, this evidence can be reviewed simultaneously by multiple VBA claims processors at any location.

VBMS-Rating is to provide raters with Web-accessible tools, including rules-based rating calculators and the capability for automated decision recommendations. For example, the hearing loss calculator is to automate decisions using objective audiology data and rules- based functionality to provide the rater with a suggested rating decision. In addition, VBMS-Rating is expected to include stand-alone evaluation builders—essentially interactive disability rating schedules—for all parts of the human body. With this tool, the rater uses a series of check boxes to identify the veteran’s symptoms, and the evaluation builder identifies the proper diagnostic code and the level of compensation based on those symptoms.

VBMS-Awards is to provide an automated award and notification process to improve award accuracy and reduce rework associated with manual development of awards. VBMS-Awards is intended to automate and standardize communications between VBA and the veteran at the final stages of the claims process.

VBMS Is Being Developed Incrementally and Was Initially Deployed In 2013

VBA is using an agile software development methodology to develop, test, and deliver VBMS functionality to its users. An agile approach allows subject matter experts to validate requirements, processes, and system functionality in increments, and to deliver the functionality to users in shorter cycles. To help guide its system development efforts, the VBMS PMO has developed both a strategic road map that identifies the program’s high-level objectives, timeline and intended outcomes, as well as a regularly revised tactical road map that describes prospective capabilities the PMO expects to develop and deploy for each system release. Consistent with VA’s policy for incremental development, VBMS is to be developed and implemented in a series of releases that are to occur every 6 months. As shown in table 1, VBA plans to develop VBMS over multiple years, with each year generally corresponding to a new system generation.

From 2010 through January 2013, VBA planned, developed, and deployed a foundational, Web-based version of VBMS to five pilot sites. This phase included development of the eFolder capability and the ability to establish, develop, and rate disability compensation claims in VBMS, as well as a user interface with search capabilities. Generation 1 development was completed in January 2013; at that time, the system was implemented at 18 regional offices.

From February 2013 through September 2013, the PMO continued to add functionality to the system, including enhancement of data exchange capabilities, correspondence tools, and rating functionality. By June 2013, VBA had completed national rollout of the initial version of the system to all 56 regional offices. The system was also made accessible to VA’s Appeals Management Center, the Board of Veterans’ Appeals, VBA National Call Centers, veterans service organizations, and all VA medical centers that complete compensation exams. Subsequent to the nationwide rollout of VBMS in June 2013, VBA has continued incremental system development and enhancement of VBMS.

Multiple Organizations Have Responsibility for VBMS

The VBMS program is dependent on multiple organizations within the department to meet its goals. It is jointly led by two program managers: the VBMS PMO Director and a Program Manager in VA’s Office of Information and Technology (OI&T).

Specifically, the VBMS PMO, which resides in VBA’s Office of Strategic Planning, is responsible for all aspects of the program’s management, including the coordination and direction of contract staff and VBA partners in integrating the system’s components and managing program-level dependencies, risks, and issues. The PMO has responsibility for gathering and delivering system requirements, performing testing, and providing training. The Director oversees all activities of the PMO and reports to the Under Secretary for Benefits.

The OI&T program manager oversees the development and implementation of VBMS and reports to the VA Chief Information Officer. OI&T entered into an interagency agreement with the Department of the Navy’s Space and Naval Warfare Systems Command (SPAWAR) Systems Center Atlantic to lead the development of VBMS. SPAWAR manages multiple contractors to develop the system and is providing technical, information assurance, program management, testing, and data integration services to support application development. VA and SPAWAR work together to manage and develop the system. Specifically, VBA subject matter experts and OI&T technical representatives are part of the system development teams.

Further, VBA’s Office of Business Process Integration is to ensure that strategic needs and requirements for business and data systems are properly documented, integrated, and communicated. The office provides internal coordination across the VBA lines of business (compensation, pension, education, loan guaranty, etc.) and helps communicate the system’s requirements to the OI&T at the department level. The Office of Business Process Integration also manages VBA’s legacy claims processing systems and their sustainment. As such, it is working with the OI&T on long-term planning for the systems, which may include their decommissioning.

In addition to these offices, VBMS is governed by the Transformation Joint Executive Board. Jointly chaired by the VA CIO and the Under Secretary for Benefits, this board is responsible for discussing and addressing the program’s risks and other issues.

In addition, VBMS is to follow the OI&T’s Project Management Accountability System, which is the department’s process for managing IT projects. As part of this process, large IT programs are broken down into multiple projects that are typically planned, developed, and implemented in 2-year cycles. Each project is broken into several increments, with each increment typically lasting 6 months or less. The PMO is responsible for presenting a life-cycle cost estimate for the 2-year project as part of each increment’s milestone review. Resources, including staff and funding, are released once the increment has been approved.

VBMS Has Been Provided with About $1 Billion in Funding

In 2008, VBA developed an initial, high-level life-cycle cost estimate of $560 million for VBMS that reflected system development costs through fiscal year 2012. The PMO revised the cost estimate in 2011, to include costs for system development, sustainment, and general operating expenses. This estimate showed that the department expected to spend $934.8 million on VBMS through its life cycle.

In July 2012, we reported on the reliability of the cost estimate for the VBMS program and noted that, while the 2011 cost estimate for the system partially reflected key practices for developing a comprehensive and well-documented estimate, it did not reflect key practices for developing an accurate and credible estimate. We recommended that any future life-cycle cost estimates for the VBMS program address the detailed weaknesses that we identified using cost-estimating best practices. VA’s Chief of Staff stated that the department concurred with our recommendation and had efforts under way to improve its cost- estimating capabilities.

Officials from the PMO subsequently provided us information showing that from fiscal years 2009 through 2015, the VBMS program had received funding in the amount of approximately $1 billion— about $502 million for system development, $308 million for IT sustainment, and $194 million for general operating expenses.

VA Has Made Progress toward Developing and Implementing VBMS, but Ongoing Activities Could Benefit from Increased Management Attention

VBA has developed and implemented capabilities for VBMS to support disability claims processing. However, completion of additional functionality to fully support processing disability claims will be delayed beyond fiscal year 2015. Further, VBA’s plans for developing and implementing capabilities to support the processing of pension benefits and appeals are uncertain. As VBA continues to develop and implement the system, three areas could benefit from increased management attention. First, the PMO does not have a reliable estimate of the cost for completing the system. Second, although VBA has improved VBMS’s availability to users, it has not established goals for system response times. Third, while the program has actively managed system defects, a recent system release included multiple unresolved defects that adversely impacted performance and users’ experiences.

VBA Has Made Important Progress toward Developing and Implementing VBMS to Support the Processing of Disability Claims; but Key Capabilities Are Delayed and Plans for Processing Pension Benefits and Appeals Are Uncertain

Since completing the implementation of VBMS at all regional offices in June 2013, VBA has continued to make important progress toward developing and implementing additional system functionality and enhancements that support electronic processing of disability compensation claims. As a result, 95 percent of records related to veterans’ disability claims are electronic and reside in VBMS. However, although the Under Secretary for Benefits stated in March 2013 that the system’s development was expected to be completed in 2015, implementation of functionality to fully support electronic claims processing has been delayed beyond 2015. Additionally, federal guidance and IT project management principles stress the importance of continuous planning throughout the life of a program to serve as a basis for managing trade-offs between cost, schedule, and scope.

During fiscal year 2014, VBA’s progress on the system included developing and implementing capabilities that addressed the strategic objectives for generation 3 of the strategic road map. For example:

Consistent with the objective to provide system users with more complex automation capabilities, the PMO released an automation feature that populates the VBMS rating calculator with a veteran’s medical information for 37 of the 71 disability benefits questionnaires that medical providers can use to submit medical information in support of a disability claim.

Regarding the strategic objective to reduce dependency on legacy systems, in April 2014, the PMO delivered the first iteration of the VBMS-Awards module to all regional offices, providing claims processors the ability to rely less on legacy systems, prepare an award based on information from VBMS’s rating module, and generate associated notification letters for veterans. The functionality released during fiscal year 2014 also included the integration of VBMS with VA’s legacy electronic records store, “Virtual VA,” and the integration of additional correspondence letters.

To address the strategic objective to deliver a capability to accept veterans’ electronic service treatment records, in December 2013, the PMO implemented new functionality in VBMS to enable electronic request and receipt of service treatment records from the Department of Defense for veterans who had separated or been discharged from military service after January 1, 2014.

VBA has continued its progress toward developing and implementing VBMS during fiscal year 2015 with efforts to address generation 4 of the strategic road map. Specifically, to address the strategic objective calling for further reduced reliance on legacy systems, the PMO has made enhancements to the system’s correspondence capabilities and letter templates, and added several new tools to the rating module. Additionally, the department has established plans and a schedule for retiring one legacy system. According to an analysis by VBA’s Office of Business Process Integration, fewer than 5 percent of disability cases are rated using the legacy rating system, Rating Board Automation 2000, and the functionality available in VBMS-Rating has largely eliminated the need for claims processors to access the legacy rating system.

In February 2015, VBA conducted a 5-week pilot at four regional offices to determine whether VBA’s Office of Business Process Integration can proceed with plans to retire the legacy rating application. According to VBMS program officials, the Office of Business Process Integration expects to retire the legacy rating system by the end of September 2015. According to the PMO Director, the office has also been working to capture process improvements and use lessons learned from earlier system development cycles to address the generation 4 strategic objective to implement improved system development processes.

Even with the progress VBA has made toward developing and implementing VBMS, the timeline for initial deployment of a national workload management capability has been delayed beyond the originally planned date of September 2014 to October 2015, with additional deployment to occur throughout fiscal year 2016. VBMS generation 4 development efforts have included addressing the strategic objective that calls for delivery of a national workload management capability. This effort has entailed developing the technology and business processes needed to support the national work queue, which is intended to handle new disability claims in a centralized queue and assign claims to the next regional office with available capacity.

The PMO began work for the national work queue in June 2014. The office had intended to deploy the first phase of the work queue functionality to users in September 2014. However, in late May 2015, the PMO Director informed us that VBA had decided to delay the initial rollout of the work queue to October 2015 so that the department can fully focus on meeting its goal to eliminate the claims backlog by the end of September 2015. Following the initial rollout, the PMO intends to implement the work queue at all regional offices through fiscal year 2016.

VBMS program documentation also identifies additional work to be performed after fiscal year 2015 to fully automate disability claims processing. Specifically, the PMO has identified the need to automate steps associated with a veteran’s request for an increase in disability benefits, such as when an existing medical condition worsens. In addition, according to the Director, the PMO intends to develop a capability to automatically associate veterans’ correspondence when a new piece of evidence to support a claim has been received electronically or scanned into VBMS. The PMO also plans to integrate VBMS with VA’s Integrated Disability Evaluation System, which contains the results of veterans’ disability medical examinations, as well as with external systems that contain military service treatment records for veterans, including those at the National Personnel Records Center.

Further, VBA has not yet developed and implemented end-to-end pension processing capabilities in VBMS. Without such capabilities, VBA continues to rely on three legacy systems to process pension claims. Specifically, according to program officials, both the Modern Award Processing–Development and Award legacy systems contain functionality related to processing pensions and will need to remain operational until VBMS can process these claims. In addition, they said that the Share legacy system contains functionality that is still needed throughout the claims process.

VBMS program officials stated that additional system analysis is needed before they can develop plans for the retirement of Share. Program documentation indicates that the first phase of pension-related functionality is expected to be introduced in December 2015. However, VBA has not yet developed plans and schedules for retiring the Modern Award Processing-Development, Award, and Share systems and fully developing and implementing the functionality of these legacy systems in VBMS.

VBA’s progress toward developing and implementing appeals processing capabilities in VBMS has also been limited. Specifically, although the information in a veteran’s eFolder is available to appeals staff for review, the appeals process for disability claims is not managed using the new system. According to VA’s fiscal year 2016 budget submission, the department is pursuing a separate effort to manage end-to-end appeals modernization, and has requested $19.1 million in fiscal year 2016 funds to develop a system that will provide functionality not available in VBMS or other VA systems. The PMO Director stated that VBA is currently analyzing commercial IT solutions that can meet the business requirements for appeals, such as providing document navigation capabilities. Nevertheless, the Director added that VBMS is expected to be part of the appeals modernization solution because components of the system, such as the VBMS eFolder and certain workload management functionality, are planned to continue supporting appeals management.

According to the PMO Director, the fact that VBMS requires additional development beyond 2015 does not reflect a delay in completing the system’s development. Instead, the additional time is a consequence of decisions to enlarge the VBMS program’s scope over time. This official added that VBMS’s original purpose was to serve primarily as an electronic document repository, and they have met this goal.

The PMO Director further stated that, as the program’s mission has expanded to support the department’s efforts to eliminate the disability claims backlog, the PMO has had to re-prioritize, add, and defer system requirements to accommodate broader departmental decisions and, in some cases, regulatory changes. For example, the PMO was tasked with developing functionality in VBMS to meet regulatory requirements for processing disability claims using mandatory forms. Officials from the VBMS PMO explained that they were made aware of this requirement well after system planning for the March 2015 release had been completed, and it introduced significant complexity to their development work.

Finally, VBA included in its strategic road map the strategic objectives that are to be addressed in generation 5 of the system, which is planned for fiscal year 2016. Further, officials from the VBMS PMO stated that they intend to develop tactical plans that identify the expected capabilities to be provided in the generation 5 releases. Nevertheless, due to the department’s incremental approach to developing and implementing VBMS, VBA has not yet produced a plan that identifies when the system will be completed and can be expected to fully support disability and pension claims processing and appeals. Thus, it will be difficult for VA to hold its managers accountable for meeting its time frame and for demonstrating progress.

Ongoing Progress of VBMS Development and Implementation May Be Hindered by Lack of a Reliable Cost Estimate

Consistent with our guidance on estimating program costs, an important aspect of planning for IT projects such as VBMS involves developing a reliable cost estimate to help managers evaluate a program’s affordability and performance against its plans, and provide estimates of the funding required to efficiently execute a program. Without this information, programs are at risk of experiencing cost overruns, missed deadlines, and performance shortfalls. Additionally, federal guidance and IT project management principles stress the importance of continuous planning throughout the life of a program to serve as a basis for managing trade- offs between cost, schedule, and scope.

In July 2012, we identified several weaknesses in VA’s policies related to cost estimating and also found that the department’s 2011 cost estimate for VBMS did not reflect key practices for developing an accurate and credible estimate. At that time, we recommended that VA modify its policies governing cost estimating to establish, among other things, a requirement to prepare a full life-cycle cost estimate for a program (as opposed to preparing an estimate for each program increment) and a requirement for programs to prepare cost estimates using best practices. In February 2013, VA’s Office of Corporate Analysis and Evaluation released a cost estimating process guide, which states that VA program offices are required to develop a life-cycle cost estimate to support senior leaders’ decision making and the department’s planning and budgeting processes. Further, this guide states that, at the request of VA leadership or a program office, VA’s Office of Corporate Analysis and Evaluation is to be available to develop independent cost estimates and review existing program cost estimates to ensure they have been developed in accordance with our cost-estimating guidance.

In 2011, VBA submitted a life-cycle cost estimate for VBMS of $934.8 million to the Office of Management and Budget. This estimate was intended to capture costs for the system’s development, deployment, sustainment, and general operating expenses through the end of fiscal year 2018. However, as of July 2015, the program’s actual costs exceeded the 2011 life-cycle cost estimate. Specifically, VBMS has received approximately $1 billion in funding through the end of fiscal year 2015 and the department has requested an additional $290 million for the program in fiscal year 2016.

The PMO has not reliably updated VBMS’s life-cycle cost estimate to reflect the program’s expanded scope and timelines for completing the system. This is largely attributable to the fact that the PMO has developed cost estimates for 2-year project cycles that are used for VBMS milestone reviews under OI&T’s Project Management Accountability System.

When asked how the PMO arrived at the cost estimates reported in the milestone reviews, officials stated that they develop rough order of magnitude estimates for each business need based on expert knowledge of the system, past development and engineering experience, and lessons learned. However, while this approach may have provided adequate information for VBA to prioritize VBMS system requirements to address in the next release, it has not produced estimates that could serve as a basis for identifying VBMS’s funding needs. Because it is typically derived from limited data and in a short time, a rough order of magnitude analysis is not equivalent to a budget-quality cost estimate and may limit an agency’s ability to identify the funding necessary to efficiently execute a program.

In addition, the PMO’s annual operating plan, which is generally limited to high-level information about the program’s organization, priorities, staffing, milestones, and performance measures for fiscal year 2015, also shows estimated costs totaling $512 million for VBMS development from fiscal years 2017 through 2020. However, according to the PMO Director, this estimate was also developed using rough order of magnitude analysis. Further, the estimate does not provide reliable information on life-cycle costs because it does not include estimated IT sustainment and general operating expenses.

Even though the PMO has developed rough order of magnitude cost estimates for VBMS, these estimates have not been sufficiently reliable to effectively identify the program’s funding needs. Instead, during the last 3 fiscal years, the PMO Director has had to request an additional $118 million in IT development funds to meet program demands and to ensure support for ongoing development contracts. Specifically, in May 2013, VA requested $13.3 million to support additional work on VBMS. Then, during fiscal year 2014, VA reprogrammed $73 million of unobligated IT sustainment funds to develop functionality to transfer service treatment records from DOD to VA, and to support development of VBMS-Core functionality. In December 2014, the PMO identified the need for additional fiscal year 2015 funds for ongoing system development contracts for VBMS-Core and VBMS-Awards, and, in late April 2015, VA leadership submitted a letter to Congress requesting permission to reprogram $31.7 million to support work on these contracts, the National Work Queue, and other VBMS efforts.

According to the PMO Director, the need to request additional funding does not represent additional risk to the program, but is the result of VBMS’s success. The Director further noted that, as the PMO has identified opportunities to increase functionality to improve the electronic claims process, their funding needs have also increased. We recognize that as new capabilities are deployed, additional requirements may surface. Nevertheless, evolution of the VBMS program illustrates the importance of continuous planning, including cost estimating, so that trade-offs between cost, schedule, and scope can be effectively managed. Further, without a reliable estimate of the total costs associated with completing work on VBMS, stakeholders will have a limited view of VBMS’s future resource needs and the program is at risk of not being able to secure appropriate funding to fully develop and implement the system.

VBA Has Made Progress toward Improving VBMS Implementation, but Does Not Have Goals for System Response Times

GAO and federal IT guidance recognize the importance of defining program goals and related performance targets and using such targets to assess progress in achieving the goals. System performance and response times have a large impact on whether staff successfully complete work tasks. If systems are not responding at agreed-upon levels for availability and performance, it can be difficult to ensure that staff will complete tasks in a timely manner. This is especially important in the VBA claims processing environment, where staff are evaluated on their ability to process claims in a timely manner.

VBA’s availability goal for VBMS is 95 percent per month, operating on a 24 hour-a-day, 6-day-a-week schedule. According to the OI&T Program Manager for VBMS, VBA defines availability as the extent to which the system is operational, functional, and usable for completing business requirements (e.g., processing claims). When an unscheduled system outage occurs, the office identifies it as being in one of three categories: critical, impacting all VBMS users across VA; moderate, impacting VBMS users at a local or individual level; and serious, impacting VBMS users in a certain regional area. The Program Manager stated that the department does not consider the system to be unavailable if users are unable to perform their duties due to non-system issues (such as local network issues or the unavailability of systems that feed data into VBMS).

VBA has reported that, since its initial rollout in January 2013, the system has exceeded the 95 percent goal for availability. Specifically, VBA reported that the system was available at a rate of 98.9 percent in fiscal year 2013 and 99.3 percent in fiscal year 2014. Through May of fiscal year 2015, it was available for 99.98 percent of the time.

Nonetheless, while the department reported exceeding its availability goals for VBMS, it has experienced periods of system unavailability, many times at the critical level affecting all users, as reflected in figure 2.

Specifically, since January 2013, the department has reported 57 VBMS outages that have totaled about 117 hours of system unavailability. VBA experienced about 18 hours of VBMS outages in January 2014, which were almost entirely at the critical level and affected all users. To the department’s credit, it reported experiencing only 2 system outages since July 2014—a 30-minute critical outage in December 2014 and a 23- minute critical outage in May 2015.

In addition to system availability, VBA monitors system response times for each of the VBMS modules using an application that measures the amount of time taken for each transaction. The PMO defines response time as the time that elapses from when a user executes a transaction (i.e., clicks a link or selects “Enter”) to when the resulting page fully loads on the user’s screen. The PMO uses average page response times to measure VBMS system performance, with lower response times indicating optimal system performance and higher response times indicating performance issues.

From September 2013 through April 2015, VBA reported a decrease in average response times for VBMS-Core and VBMS-Rating (see table 2).

VBA attributed the decrease in response times to continuous engineering improvements to system performance. Program officials also explained that the difference in response times between modules was due to the type of information that is being pulled into each module from various other VBA systems. For example, both VBMS-Core and VBMS-Rating require information from the VBA corporate database, but VBMS-Core is populated with data from multiple VBA systems in addition to the corporate database.

Program officials told us there are no specific goals for mean transaction response times because they feel that there are adequate tools in place to monitor system performance and provide alerts if there are response time issues. For example, VBMS performance is monitored in real time by dedicated staff at a contractor’s facility and users have access to a live chat feature where they can provide feedback on any issues they are experiencing with the system.

The VBMS help desk provides another avenue for users to provide feedback on the system’s performance. Officials also noted that, because transaction response times have decreased, which can be indicative of an improvement to system performance, they are focusing their resources on adding additional functionality instead of trying to get the system to achieve a specific average transaction response time.

While VBA’s monitoring of VBMS’s performance is commendable and the system’s performance and response time have improved over time, the system is still in development and there is no guarantee that performance will remain at current levels as the system evolves. Performance targets and goals for VBMS response times would provide users with an expectation of the system response times they should anticipate, and management with an indication of how well the system is performing relative to performance goals.

VBMS Program Has Actively Managed System Defects, but a Recent Release Included Unresolved Defects that Adversely Impacted System Implementation

A key element of successful system testing is appropriately identifying and handling defects that are discovered during testing. Outstanding defects can delay the release of functionality to end users, denying them the benefit of features. Key aspects of a sound defect management process include the planning, identification and classification, tracking, and resolution of defects. Leading industry and government organizations consider defect management and resolution to be among the primary goals of testing.

The VBMS program has defect management policies in place and it is actively performing defect management activities. Specifically, in October 2012, the department developed a VBMS Program Management and Technical Support Defect Management Plan, Version 1.0, which describes the program’s defect management process. The plan was updated in March 2015 and describes, among other things, the process for identifying, classifying, tracking, and resolving VBMS defects. For example, it provides criteria for assigning four different severity levels for defects—critical, high, medium, and low.

According to the plan, critical severity defects are characterized by complete system or subsystem failure, complete loss of functionality, and compromised security or confidentiality. Critical defects also have extensive user impact and workarounds do not exist. High severity defects can have major user impact, leading to significant loss of system functionality. Medium severity defects can have moderate user impact and lead to moderate loss of functionality. Low severity defects lead to minor loss of functionality with no workaround necessary. For high and medium severity defects, workarounds could exist. According to the PMO, high, medium, and low defects do not need to be resolved prior to a system release.

To monitor and track defects, the PMO uses an automated tool to maintain the VBMS defect repository. This tool is used to produce a daily defect management report that is shared with VBMS leadership. The purpose of the daily defect management report is to provide the current status of all open defects identified in testing of a forthcoming VBMS release or identified during production of a previous release.

According to the defect management plan, defects can be resolved in a number of different ways. Once a defect is fixed, tested, and has passed testing, it is considered done or resolved. Defects that cannot be attributed to an existing requirement are reclassified as a system enhancement and considered resolved, as they do not affect a current system release requirement. A defect is also considered resolved if it is determined to work as designed, duplicate another defect, or if it is no longer evident in the system.

From March 2014 through March 2015, the total number of VBMS defects declined as release dates approached for releases 7.0, 7.1, 8.0, and 8.1. Additionally, to the department’s credit, no critical defects remained at the time of each of these releases. Specifically, prior to VBMS Release 7.0, the total number of defects identified peaked at 226, while 97 remained at the time of release. Prior to release 7.1, the total number of defects identified peaked at 330, with 169 remaining at the time of release. Prior to release 8.0, the total number of defects identified peaked at 330, with 161 remaining at the time of release. Prior to release 8.1, the total number of defects remaining peaked at 421, with 254 remaining at the time of release. Figure 3 shows the trend in the number of open total defects for this time period.

Even with the department’s efforts to resolve defects prior to a VBMS release, defects that affected system functionality remained open at the time of the releases. Specifically, of the 254 open defects at the time of VBMS release 8.1, 76 were high severity, 99 were medium severity, and 79 were low severity. Examples of defects that remained open at the time of VBMS release 8.1 are described in table 3.

According to the PMO, these defects were communicated to users and an appropriate workaround for each was established. Nevertheless, even with workarounds, high- and medium-severity open defects, which by definition impact system functionality, degraded users’ experiences with the system. Continuing to deploy system releases with defects that impact system functionality increases the risk that these defects will diminish users’ ability to process disability claims in an efficient manner.

VBA’s Methods for Determining User’s Feedback on VBMS Have Not Included Establishing Goals or Conducting a Survey; GAO’s Survey Identified Varied Satisfaction Levels

While VBA has several methods to obtain VBMS users’ feedback, it has neither established goals to define user satisfaction, nor conducted a survey of claims processing employees to obtain a comprehensive picture of overall customer satisfaction. Our survey of VBMS users found that a majority reported satisfaction with the system, but decision review officers were considerably less satisfied. Although the results of our survey provide VBA with useful data about users’ satisfaction with the system, the absence of user satisfaction goals limits the utility of survey results.

VBA Has Several Methods to Obtain Users’ Feedback on VBMS, but Has Not Established User Satisfaction Goals or Conducted a Survey

GAO and federal IT guidance recognize the importance of defining program goals and related performance targets and using such targets to assess progress in achieving the goals. Also, leading practices identify continuous customer feedback as a crucial element of IT project success, from project conception through sustainment. Particularly for IT projects like VBMS, where development activities are iterative, customer (i.e., end user) perspectives and insights can be solicited through various methods—user acceptance testing, interviews, complaint programs, and satisfaction surveys—to validate or raise questions about the project’s implementation.

Further, leading practices emphasize that periodic customer satisfaction data be proactively used to improve performance and demonstrate the level of satisfaction the project is delivering. The Office of Management and Budget has developed standards and guidelines in survey research that are generally consistent with best practices and call for statistically valid data collection efforts to be used in fulfilling agencies’ customer service data collection. These leading practices also stress the importance of centrally integrating all customer feedback data in order to have more complete diagnostic information to guide improvement efforts.

VA has used a variety of methods for obtaining customer or end user feedback on the performance of VBMS. For example, the department solicits end user involvement and feedback in the iterative system development process based on user acceptance criteria. According to the Senior Project Manager for VBMS Development within OI&T, at the end of each development cycle and before a new version of VBMS is deployed, end users are involved in user acceptance testing and a final customer acceptance meeting.

VA also provides training to a subset of end users—known as “superusers”—on the updated functionality introduced in a new version of VBMS. These superusers are then expected to train the remaining users in the field on the new version’s features. The department tracks the overall satisfaction level with training received after each VBMS major release. However, this tracking is limited to superusers’ satisfaction with the training, rather than their satisfaction with the system.

VA also solicits customer feedback about the system through interviews. For example, the PMO Director stated that the Under Secretary for Benefits hosts a weekly phone call with bargaining unit employees as a “pulse check” on VBA transformation activities, including VBMS. According to this official, the VBA Office of Field Operations also offers an instant messaging chat service to all regional office employees to solicit feedback about the latest VBMS functionality deployment.

Another method in which the department obtains customer input is through a formal feedback process. For example, according to the PMO Director, VA provides national service desk support to assist users in troubleshooting system issues and identifying system defects. In addition, VBMS applications include a built-in feature that enables users to provide feedback to the PMO on problems with the system. According to the Director, the feedback received by the office also helps to identify user training issues.

Nevertheless, while VA has taken steps to obtain feedback on the performance and implementation of VBMS, it has not established goals to define user satisfaction that can be used as a basis for gauging the success of its efforts to promote satisfaction with the system. Further, although the efforts that have been taken to solicit users’ feedback provide VBA with useful insights about particular problems, data are not centrally compiled or sufficient for supporting overall conclusions about whether customers are satisfied. In addition, VBA has not employed a customer satisfaction survey of claims processing employees who use the system on a daily basis to process disability claims. Such a survey could provide a more comprehensive picture of overall customer satisfaction and help identify areas where VBMS development and implementation efforts might need additional attention.

According to the PMO Director, VBA had not used a survey to solicit feedback because of concern that such a mechanism may have negatively impacted the efficiency of claims processors in completing disability compensation claims on behalf of veterans. Further, the director believed that the office had the benefit of receiving ongoing end user input on VBMS by virtue of the intensive testing cycles, as well as several of the other mechanisms noted previously by which end users have provided ongoing feedback. Nevertheless, without establishing user satisfaction goals and collecting the comprehensive data that a statistically valid survey can provide, the PMO limits its ability to obtain a comprehensive understanding of VBMS users’ levels of satisfaction with the system. Thus, VBA could miss opportunities to improve the efficiency of its claims process by increasing satisfaction with VBMS.

A Majority of VBMS Users, with the Exception of Decision Review Officers, Reported Satisfaction with the System

In response to the statistical survey that we administered, a majority of VBMS users (i.e., VBA claims processors) were satisfied with the system that had been implemented at the time of the survey. These users (which represent claims assistants, veteran service representatives, supervisory veteran service representatives, rating veterans service representatives, decision review officers, and others) were satisfied with the three modules of VBMS. Specifically, an estimated 59 percent of the population of claims processors were satisfied with VBMS-Core; an estimated 63 percent were satisfied with the Rating module, and an estimated 67 percent were satisfied with the Awards module. Figure 4 depicts the estimated percentage of claims processors who were satisfied with VBMS.

Although a majority of users were satisfied with the three modules, decision review officers were considerably less satisfied than other users with VBMS-Core and VBMS-Rating. Specifically, for VBMS-Core, an estimated 27 percent of decision review officers were satisfied compared to an estimated 59 percent of all roles of claims processors (including decision review officers) who were satisfied. In addition, for VBMS-Rating, an estimated 38 percent of decision review officers were satisfied, compared to an estimated 63 percent of all roles of claims processors.

Figure 5 depicts the estimated satisfaction levels of decision review officers in comparison to all roles.

Decision review officers were considerably less satisfied with VBMS in comparison to all roles of claims processors in additional areas. For example, an estimated 26 percent of decision review officers viewed VBMS-Core as an improvement over the previous legacy system or systems for establishing claims and storing and reviewing electronic documents related to a claim in an eFolder. In contrast, an estimated 58 percent of all users (including decision review officers) viewed the Core module as an improvement. In addition, an estimated 26 percent of decision review officers viewed VBMS-Rating as an improvement over the previous systems with respect to providing Web-accessible tools, including rules-based rating calculators, to assist in making claims rating decisions. In contrast, an estimated 55 percent of all roles of claims processors viewed the Rating module as an improvement. For VBMS- Awards, an estimated 61 percent of all roles viewed this module as an improvement over the previous systems to automate the award and notification process. Figure 6 depicts the estimated percentage of decision review officers, in comparison to all claims processors, who viewed VBMS as an improvement over legacy systems.

Similarly, for the three modules, a majority of users (including decision review officers) would have chosen VBMS over the legacy system or systems. However, decision review officers indicated that they were less likely to have chosen VBMS-Core and VBMS-Rating over legacy systems. Specifically, an estimated 27 percent of decision review officers would have chosen VBMS-Core compared to an estimated 60 percent of all roles of claims processors. In addition, an estimated 27 percent of decision review officers would have chosen VBMS-Rating compared to 61 percent of all roles who would have chosen the system over the legacy system or systems. For VBMS-Awards, an estimated 67 percent of all roles would have chosen this module over the previous systems. Figure 7 depicts the estimated percentage of decision review officers, in comparison to all claims processors, who would have chosen VBMS instead of the legacy systems to process claims.

Decision review officers perform an array of duties to resolve claims issues raised by veterans and their representatives. They may also conduct a new review or complete a review of a claim without deference to the original decision and in doing so, must click through all documents included in the e-Folder. Survey comments from decision review officers stated, for example, that reviews in the VBMS paperless environment take longer because of the length of time spent loading, scrolling, and viewing each document (particularly if the documents are large, such as a service medical record file). Additionally, multiple decision review officers commented that it is easier and faster to review documents in a paper file. Although such comments provide illustrative examples of individual decision review officer’s views and are not representative, according to the PMO Director, decision review officers’ relative dissatisfaction is not surprising because the system does not yet include functionality that supports their work, which primarily relates to appeals processing.

Conclusions

Even though VA has made progress toward completing the development and implementation of VBMS, there is more work to be done and VBA can improve management of its ongoing efforts. While 95 percent of records related to veterans’ disability claims are electronic and reside in VBMS, additional capabilities to fully process disability claims will be delayed beyond fiscal year 2015, which is when completion was originally planned. Further, VA’s incremental approach to developing and implementing VBMS has not yet produced a plan that identifies when the system will be completed and can be expected to fully support disability compensation and pension claims processing and appeals. Thus, it will be difficult for VA to hold its managers accountable for meeting its time frame and for demonstrating progress.

VBA’s management of the system highlights areas that could benefit from improvement as development and implementation of the system continue. Specifically, without a reliable estimate of the total costs associated with completing work on VBMS, the department’s stakeholders have only a limited view of future resource needs, and the program risks not having sufficient funding to complete development and implementation of the system. Additionally, established goals for system response times would provide users with an expectation of the response times they can anticipate, and management with an indication of how well the system is performing relative to performance goals. Furthermore, continuing to deploy system releases with large numbers of defects that reduce system functionality could adversely affect users’ ability to process disability claims in an efficient manner.

Without user satisfaction goals and the data a customer satisfaction survey could yield, VA may miss opportunities to collect important data on how users view the system’s performance, and ultimately, to improve the system. Our survey of VBMS users found that a majority of them were satisfied with the system, but decision review officers were considerably less satisfied. Although the results of our survey provide VBA with useful data about users’ satisfaction with VBMS, the absence of user satisfaction goals limits the utility of survey results. Specifically, without having established goals to define user satisfaction, VBA does not have a basis for gauging the success of its efforts to promote satisfaction with the system or for identifying areas where its efforts to complete development and implementation of the system might need attention.

Recommendations for Executive Action

We recommend that the Secretary of Veterans Affairs direct the Under Secretary for Benefits and the Chief Information Officer to take the following five actions to improve VA’s efforts to effectively complete the development and implementation of VBMS:

Develop an updated plan for VBMS that includes (1) a schedule for when VBA intends to complete development and implementation of the system, including capabilities that fully support disability claims, pension claims, and appeals processing and (2) the estimated cost to complete development and implementation of the system.

Establish goals for system response time and use the goals as a basis for periodically reporting actual system performance.

Reduce the incidence of high- and medium-priority level defects that are present at the time of future VBMS releases.

Develop and administer a statistically valid survey of VBMS users to determine the effectiveness of steps taken to make improvements in users’ satisfaction.

Establish goals that define customer satisfaction with VBMS and report on actual performance toward achieving the goals based on the results of GAO’s survey of VBMS users and any future surveys VA conducts.

Agency Comments and Our Evaluation

We received written comments on a draft of this report (reprinted in appendix III). In addition, VA provided technical comments, which we incorporated, as appropriate.

In its comments, VA generally agreed with our conclusions. The department also concurred with our recommendations and described actions it is planning to take in response to four of our five recommendations. Specifically, VA concurred with our recommendation calling for an updated plan for VBMS. Although it described the importance of VBMS to ensuring timely delivery of benefits to veterans and recognized the need for continued investment in the system, the department did not, however, identify actions to develop a VBMS plan that includes a schedule for when VBA intends to complete development and implementation of the system and the estimated cost of doing so. We believe development of such a plan is an important action to help ensure effective development and implementation of VBMS and to hold managers accountable.

With regard to the remaining four recommendations, VA described actions it is planning to take in response to each. For example, with regard to our recommendation to establish goals for system response time, the department stated that the VBMS program is participating in a review of service-level agreements to establish metrics for the system’s performance. Additionally, regarding our recommendation that it reduce the incidence of high and medium priority level defects that are present in future VBMS releases, the department reiterated its plans and procedures for decreasing the defects in each release. With respect to our recommendation to conduct a survey of VBMS users, the department stated that the VBMS PMO is working with the Office of Field Operations and labor partners regarding the distribution of a survey to measure users’ satisfaction and expects to release a survey in March 2016. Consistent with our final recommendation, the department is also planning to establish customer satisfaction goals and report on actual performance toward achieving the goals after all survey results are received and analyzed with a target of July 2016 for completing this action.

If VA develops the updated plan, including schedule and cost, for VBMS as we recommended and follows through on the actions it described in response to our remaining recommendations, the department will be better positioned to effectively complete the development and implementation of VBMS and to more effectively provide benefits and services to our nation’s veterans.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to appropriate congressional committees, the Secretary of Veterans Affairs, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have questions about this report, please contact me at (202) 512-6304 or melvinv@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

The objectives of this study were to (1) assess the Department of Veterans Affairs’ (VA) progress toward developing and implementing the Veterans Benefits Management System (VBMS) and (2) determine to what extent users report satisfaction with the system.

To assess the department’s progress toward developing and implementing VBMS, we obtained and reviewed program plans and other documentation that articulated the system’s goals and expected benefits. These included the VA Strategic Plan to Eliminate the Compensation Claims Backlog (2013), VBMS Strategic Roadmap, VBMS Tactical Roadmap, VBMS Operating Plans, VBA’s 2011 life-cycle cost estimate for the program, documentation supporting VBMS’s progression through VA’s milestone review process, VA’s annual budget submissions, and contracts and related contractor documentation. We compared program plans and other documentation articulating the system’s goals, expected benefits, and the system functionality expected to be delivered with documents showing VBMS’s progress, such as monthly program management reviews and cost and performance reports. We also compared program plans to VA’s policies for incremental system development, as well as federal guidance and IT project management principles on program planning and cost estimating.

In addition, we obtained reports documenting the system’s availability and performance and analyzed information in the reports to determine trends in system availability and response time. We compared system performance data with GAO and federal IT guidance for defining program goals and related performance targets that can be used to assess progress in achieving the goals. We assessed the reliability of the data by reviewing it for obvious errors and missing data; corroborating the data with related documentation; and interviewing responsible officials about their use of an automated tool that monitors system performance. We determined the data to be sufficiently reliable for the purposes of this report.

We reviewed the VBMS Defect Management Plan and compared the plan with key principles of sound defect management. We also compared the processes described in the plan with actions taken to manage defects identified for VBMS. We conducted analysis of critical defects identified for VBMS releases 7.0, 7.1, 8.0, and 8.1 to determine whether they remained open at the time of system release.

Further, we reviewed the department’s methods for soliciting end user feedback on the performance of the system (e.g., VBMS training strategy, operational management reviews, cost and performance reports, and end of month reports). We compared these methods to leading practices for obtaining customer feedback and collecting customer service data to improve performance and demonstrate customer satisfaction of IT projects.

To determine the extent that users report satisfaction with the system, we conducted a Web-based survey of a nationally representative stratified random sample of disability compensation claims processors. The survey was administered during the time frame of September through November 2014. We developed survey questions with input from officials in the VBMS PMO and VA’s Office of Information and Technology.

We pretested versions of the draft survey and observed claims processors to gain initial insight regarding their use of the system in VBA’s Baltimore, Maryland, and Philadelphia, Pennsylvania, regional offices. We selected these offices based on their large size and location. We revised the draft survey based on comments received during the pretests. Once finalized, the claims processors included in the sample were sent an e-mail that asked them to complete the survey, which was available to them over the Web from September 30, 2014, to November 19, 2014. Following the initial request, we e-mailed weekly follow-up requests to any nonrespondents.

Our eligible population for this survey consisted of all 10,622 disability compensation claims processors employed by the agency as of July 31, 2014. To determine whether VBMS experiences differed based on position and office workload, we designed a stratified random sample of claims processors, with strata defined by position and office workload. VBA provided us lists of these claims processors with information that indicated their position and office. We determined average office workload based on information published in VBA’s public weekly workload reports. We divided the list of claims processors into 10 strata based on their position at both smaller and larger workload regional offices. Table 4 shows the size of each eligible population, the size of the sample drawn from each eligible population, and the number of claims processors in each stratum that responded to the survey.

We drew an independent random sample from each of these strata to enable us to project survey results to all VBA claims processors, in general, and to claims processors in each of the positions listed above, within small and large workload offices. Of our total sample of 3,475 claims processors, 2,098 responded to the survey for an overall response rate of 60 percent.

To produce estimates regarding the experiences and views of claims processors from the survey responses of those included in our sample, we analyzed the data with methods that are appropriate for a stratified random sample using analysis weights. We weighted each response from claims processors in each stratum to statistically account for all members of that stratum. Because estimates are based on responses from a sample, each estimate we report has a measurable precision or sampling error. The sampling error or margin of error surrounding an estimate is expressed as a number of percentage points higher or lower than that estimate and the entire range that the sampling error covers is referred to as a confidence interval. Confidence intervals are calculated based on a certain confidence level. Confidence intervals for estimates we report from this survey are based on a confidence level of 95 percent and are calculated using methods appropriate for a stratified random sample. Confidence intervals for percentage estimates in this report are never wider than plus or minus 5 percentage points. At a 95 percent confidence level, this means that in about 95 out of 100 instances, the sampling procedures we used would be expected to produce a confidence interval (in this case, a 10 percentage point range) containing the true population value we estimate.

In addition to sampling error, estimates based on survey results are subject to what is referred to as nonsampling error that can result from, among other things, poorly designed survey questions and mistakes in data entry or analysis or nonresponse. We took a number of steps in developing the survey and in entering and analyzing the data to minimize nonsampling error. For example, a social science survey specialist collaborated with our subject matter experts in designing the survey. In addition, it was reviewed by VBA officials and, as previously noted, was pretested with a number of different types of claims processors in two locations. Also, when we analyzed the data, an independent analyst verified all computer programs. Because this was a Web-based survey, respondents’ answers to survey questions were automatically entered into an electronic file, eliminating the need to separately key responses into a data file, further minimizing the potential for nonsampling error.

We also took steps to mitigate potential nonresponse error. For example, we used follow-up e-mails to remind users to complete the survey in order to reduce nonresponse. Further, we adjusted for characteristics that were associated with survey response propensity using standard weighting class adjustments defined by sampling strata. We assumed that nonresponse adjusted data were missing at random, and therefore concluded the respondent analyses using the nonresponse adjusted weights were unbiased for the population of VBMS users sampled in the survey.

We supplemented our analyses with interviews of VA officials that had knowledge of the VBMS program, including officials in VA’s Office of Information and Technology and the Veterans Benefits Administration’s VBMS PMO and Office of Field Operations.

We conducted this performance audit from March 2014 through August 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Survey of Veterans Benefits Management System End User Experience

The questions we asked in our survey of Veterans Benefits Administration (VBA) claims processors are shown here. Our survey was comprised of closed- and open-ended questions. In this appendix, we include all survey questions and aggregate results of responses to the closed-ended questions. For a more detailed discussion of our survey methodology, see appendix I. 1. What is your current role in processing veterans’ disability compensation claims?

If other, describe your role. (Open-ended response.) 2. What responsibility (or responsibilities) do you have in this role (Check all that apply.)

If other, describe your responsibility. (Open-ended response.) 3. How many years have you worked within the VBA?

4. The Department of Veterans Affairs’ (VA) records show that you currently work at the (office name prepopulated) office. Is this correct?

If no, enter the name of the office where you currently work. (If you work in more than one office, enter the office where you spend most of your time.) (Open-ended response.) 5. How long have you been using the Veterans Benefits Management System (VBMS)?

Response Less than 6 months 95 percent confidence interval - lower bound 6.9 95 percent confidence interval - upper bound 9.8 6 months to less than 1 year 1 year to less than 2 years 2 years to less than 3 years Not applicable, I do not use VBMS (Skip to question 17.)

6. In an average work week, how much time, if at all, do you spend processing the following types of claims in VBMS? (Choose a time period for each item listed.)

Response 6a. Fully electronic claims 95 percent confidence interval - lower bound 8.0 95 percent confidence interval - upper bound 10.6 8 to less than 16 hours 16 to less than 24 hours 24 to less than 32 hours 32 to less than 40 hours All week (40 or more hours)

I do not process these 6b. Fully paper- based claims 8 to less than 16 hours 16 to less than 24 hours 24 to less than 32 hours 32 to less than 40 hours All week (40 or more hours)

I do not process these 6c. Hybrid claims (that is, claims that consist of both paper and electronic records)

8 to less than 16 hours 16 to less than 24 hours 8.7 24 to less than 32 hours 95 percent confidence interval - lower bound 2.9 95 percent confidence interval - upper bound 4.9 32 to less than 40 hours All week (40 or more hours)

7. In addition to using VBMS, how much do you depend on each of the following systems or applications to process disability claims? (Choose a category for each system or application listed.)

None or do not use 95 percent confidence interval - lower bound 49.0 95 percent confidence interval - upper bound 52.9 7b. VETSNET: Modern Award Processing- Development (MAP-D)

None or do not use 7c. VETSNET: Rating Board Automation 2000 (RBA 2000)

Section 2—Veterans Benefits Management System-Core (VBMS-Core) 8. In the course of your work, do you use VBMS-Core?

Complete questions 8.1 and 8.2 for VBMS-Core. In question 8.3, you will have a chance to comment on any of your answers after responding to the questions. 8.1. Do you agree or disagree with the following statements based on your current experience using VBMS-Core? Select one answer in each row.

Not applicable or no basis to judge 8.1c. VBMS-Core helps me be more productive compared to the previous system(s)

Not applicable or no basis to judge 8.1d. VBMS-Core helps me be more efficient compared to the previous system(s)

Not applicable or no basis to judge 8.1f. VBMS-Core is easier to use compared to the previous system(s)

Not applicable or no basis to judge 8.1n. VBMS-Core performance is timely (e.g., minimal time to load pages and respond to commands)

Response 8.1o. I believe VBMS-Core helps to reduce claims processing times compared to the previous system(s)

Not applicable or no basis to judge 8.1r. VBMS-Core is an improvement over the previous system(s)

8.2. While using VBMS-Core how often, if at all, do you experience each of the following? Select one answer in each row.

Response 8.2a. The scanned documents displayed on the screen within VBMS- Core are clearly legible and correctly oriented (i.e., not upside-down)

8.3. If you have any comments regarding your answers to questions 8.1 and/or 8.2 concerning VBMS-Core, share them here. (Open-ended response.)

Section 3—Veterans Benefits Management System-Rating (VBMS-R) 9. In the course of your work, do you use VBMS-R?

Complete questions 9.1 and 9.2 for VBMS-R. In question 9.3, you will have a chance to comment on any of your answers after responding to the questions. 9.1. Do you agree or disagree with the following statements based on your current experience using VBMS-R? Select one answer in each row.

Not applicable or no basis to judge 9.1c. VBMS-R helps me be more productive compared to the previous system(s)

Not applicable or no basis to judge 9.1d. VBMS-R helps me be more efficient compared to the previous system(s)

Not applicable or no basis to judge 9.1f. VBMS-R is easier to use compared to the previous system(s)

Not applicable or no basis to judge 9.1n. VBMS-R performance is timely (e.g., minimal time to load pages and respond to commands)

Not applicable or no basis to judge 9.1o. I believe VBMS-R helps to reduce claims processing times compared to the previous system(s)

Not applicable or no basis to judge 9.1r. VBMS-R is an improvement over the previous system(s)

9.2. While using VBMS-R, how often, if at all, do you experience each of the following? Select one answer in each row.

9.3. If you have any comments regarding your answers to questions 9.1 and/or 9.2 concerning VBMS-R, share them here. (Open-ended response.)

Section 4—Veterans Benefits Management System-Awards (VBMS-A) 10. In the course of your work, do you use VBMS-A?

Complete questions 10.1 and 10.2 for VBMS-A. In question 10.3, you will have a chance to comment on any of your answers after responding to the questions. 10.1. Do you agree or disagree with the following statements based on your current experience using VBMS-A? Select one answer in each row.

Not applicable or no basis to judge 10.1c. VBMS-A helps me be more productive compared to the previous system(s)

Not applicable or no basis to judge 10.1d. VBMS-A helps me be more efficient compared to the previous system(s)

Not applicable or no basis to judge 10.1f. VBMS-A is easier to use compared to the previous system(s)

Not applicable or no basis to judge 10.1n. VBMS-A performance is timely (e.g., minimal time to load pages and respond to commands)

Not applicable or no basis to judge 10.1o. I believe VBMS-A helps to reduce claims processing times compared to the previous system(s)

Not applicable or no basis to judge 10.1r. VBMS-A is an improvement over the previous system(s)

10.2. While using VBMS-A how often, if at all, do you experience each of the following? Select one answer in each row.

Response 10.2a. VBMS-A gives error messages that clearly tell me how to fix problems.

10.3. If you have any comments regarding your answers to questions 10.1 and/or 10.2 concerning VBMS-A, please share them here (Open-ended response.)

Section 5—VBMS Benefits and Challenges 11. How much, if at all, has VBMS improved your ability to do the following? Select one answer in each row.

Not applicable or no basis to judge 11d. Access claims documentation on demand (e.g., the ability to electronically view a Veteran’s claim file at multiple regional offices at the same time)

Not applicable or no basis to judge 11f. Other improvement(s)

If other improvement(s), describe. (Open-ended response.) 12. Which of the following would you consider to be the single greatest improvement in using VBMS?

Make more accurate rating decisions Use automated tools (e.g., standardized correspondence, rating application evaluation builder, rules-based calculators, etc.) for quicker/more accurate processing Access claims documentation on demand (e.g., the ability to electronically view a Veteran’s claim file at multiple regional offices at the same time)

If other improvement, please describe the other VBMS improvement you consider to be the greatest. (Open-ended response.) 13. How much of a challenge, if at all, has VBMS been to you in the following areas? Select one answer in each row.

Response 13a. System performance (e.g., slow response times)

Not applicable or no basis to judge 13b. System access (e.g., system outages)

Response 13c. The need to rely on previous systems in addition to VBMS (e.g., VETSNET systems)

Not applicable or no basis to judge 13d. Knowing whether the evidence required to process claims is complete in VBMS (e.g., not knowing whether all documents have been scanned and added to the veteran’s file)

Not applicable or no basis to judge 13e. Other challenge(s)

If other challenges(s), describe. (Open-ended response.) 14. Which of the following would you consider to be the single greatest challenge in using VBMS?

Response System performance (e.g., slow response times)

System access (e.g., system outages)

The need to rely on previous systems in addition to VBMS (e.g., VETSNET systems)

Knowing whether the evidence required to process claims is complete in VBMS (e.g., not knowing whether all documents have been scanned and added to the veteran’s file)

If other challenge, describe the other VBMS challenge you consider to be the greatest. (Open-ended response.) 15. If you could pick one change to be included in the next release/update of VBMS, what would it be and why? (Open-ended response.) 16. Please share any comments or suggestions for improvement you have about the VBMS system. (Open-ended response.)

Appendix III: Comments from the Department of Veterans Affairs

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Mark T. Bird (Assistant Director), Chris Businsky, Virginia Chanley, Heather A. Collins, Kavita Daitnarayan, Kelly Dodson, Nancy Glover, Stuart Kaufman, Michael Little, Jamelyn Payan, Scott Pettis, Martin L. Skorczynski, Roger M. Smith, Eric Trout, Sonya Vartivarian, and Kevin Walsh made key contributions to this report.