VETERANS 

BENEFITS 

MANAGEMENT 

SYSTEM

Ongoing Development 

and Implementation 

Can Be Improved; 

Goals Are Needed to 

Promote Increased 

User Satisfaction 

Report to the Chairman, Committee on 

Veterans' Affairs, House of 

Representatives

September 2015

GAO-15-582

United States Government Accountability Office
Highlights of GAO-15-582, a report to the 

Chairman, Committee on Veterans' Affairs, 

House of Representatives

September 2015

VETERANS BENEFITS MANAGEMENT SYSTEM

Ongoing Development and Implementation Can Be 

Improved; Goals Are Needed to Promote Increased

User Satisfaction

Why GAO Did This Study

VBA pays disability benefits for 

disabling conditions incurred or 

aggravated while in military service, 

while pension benefits are for lowincome veterans who are either elderly 

or have disabilities unrelated to military 

service. In fiscal year 2014, the 

department paid about $58 billion in 

disability compensation and about $5 

billion in pension claims. The disability 

claims process has been the subject of 

attention by Congress and others, due 

in part, to long waits for processing 

claims and a large backlog of claims. 

To process disability and pension 

claims more efficiently, VA began 

implementation of an electronic, 

paperless system in 2009. GAO was 

asked to study VBMS. Specifically, 

GAO (1) assessed VA’s progress 

toward completing the development 

and implementation of VBMS and (2) 

determined to what extent users report 

satisfaction with the system. To do so, 

GAO reviewed relevant program 

documentation, administered a survey

to a stratified random sample of about 

3,500 users, and interviewed 

appropriate VA officials.

What GAO Recommends

GAO recommends that VA develop a

plan for completing VBMS, establish 

goals for system response time, 

minimize the incidence of high and 

medium priority system defects for 

future VBMS releases, assess user 

satisfaction, and establish satisfaction

goals to promote improvement. VA 

concurred with the recommendations 

and described actions it is planning to 

take in response, except for the first 

recommendation. GAO continues to 

believe development of a plan for 

completing the system is important.

What GAO Found

The Veterans Benefits Administration (VBA) within the Department of Veterans 

Affairs (VA) has made progress in developing and implementing the Veterans 

Benefits Management System (VBMS), with deployment of the system to all of its 

regional offices as of June 2013. While 95 percent of records related to veterans’ 

disability claims are electronic and reside in the system, additional capabilities 

have not yet been completed, such as automation of the steps associated with a 

veteran’s request for an increase in benefits. Further, VBA has not yet developed 

and implemented pension processing capabilities in VBMS, nor has it articulated 

when the system will support appeals processing. The VBMS program reported 

receiving funding of about $1 billion from fiscal years 2009 to 2015, at which time 

system completion was originally planned. Although development of the system 

is expected to continue beyond 2015, the incremental approach VA is using to 

develop and implement VBMS has not yet produced a plan that identifies when 

the system will be completed and can be expected to fully support disability and 

pension claims processing and appeals. Thus, it will be difficult for VA to hold its 

managers accountable for meeting its time frame and for demonstrating 

progress.

As VA continues its efforts to complete development and implementation of the 

system, three areas could benefit from increased management attention. 

• Cost estimating: The program office does not have a reliable estimate of the 

cost for completing the system. Without such an estimate, VA management 

and the department’s stakeholders have a limited view of the system’s future 

resource needs, and the program risks not having sufficient funding to 

complete development and implementation of the system. 

• System availability: Although VBA has improved its performance for ensuring 

the system is available to users, it has not established system response time 

goals. Without such goals, users do not have an expectation of the system 

response times they can anticipate and management does not have an 

indication of how well the system is performing relative to performance goals. 

• System defects: While the program has actively managed system defects, a 

recent system release included unresolved defects that impacted system 

performance and users’ experiences. Continuing to deploy releases with 

large numbers of defects that reduce system functionality could adversely 

affect users’ ability to process disability claims in an efficient manner. 

While VBA has employed various methods to obtain VBMS users’ feedback, it 

has neither established goals to define user satisfaction, nor conducted a survey 

of claims processing employees to obtain a more comprehensive picture of 

overall customer satisfaction. GAO’s survey of VBMS users estimated that a 

majority report satisfaction with the system, but that one group of users who are 

responsible for examining claims decisions was considerably less satisfied. 

Although the results of GAO’s survey provide VBA with useful data about users’ 

satisfaction with the system, the absence of user satisfaction goals limits the 

utility of survey results. Specifically, without having established goals to define 

user satisfaction, VBA does not have a basis for gauging the success of its 

efforts to promote satisfaction with the system.

View GAO-15-582. For more information, 

contact Valerie C. Melvin at (202) 512-6304 or 

MelvinV@gao.gov.

United States Government Accountability Office
Letter 1

Background 5

VA Has Made Progress toward Developing and Implementing 

VBMS, but Ongoing Activities Could Benefit from Increased 

Management Attention 15

VBA’s Methods for Determining User’s Feedback on VBMS Have 

Not Included Establishing Goals or Conducting a Survey; 

GAO’s Survey Identified Varied Satisfaction Levels 32

Conclusions 40

Recommendations for Executive Action 41

Agency Comments and Our Evaluation 41

Appendix I Objectives, Scope, and Methodology 44

Appendix II Survey of Veterans Benefits Management System End 

User Experience 50

Appendix III Comments from the Department of Veterans Affairs 92

Appendix IV GAO Contact and Staff Acknowledgments 97

Tables

Table 1: Strategic Objectives for Veterans Benefits Management 

System (VBMS) Development (2010-2016) 11

Table 2: VA’s Reported Mean Transaction Response Times for 

Veterans Benefits Management System (VBMS) Modules 26

Table 3: Examples of Open Defects for Veterans Benefits 

Management System (VBMS) Release 8.1 31

Table 4: Eligible Population, Sample, and Respondents, by 

Stratum 47

Figures

Figure 1: Simplified View of VBA’s Disability Compensation 

Claims Process 8

Contents

Page i GAO-15-582 Veterans Benefits Management System
Figure 2: Reported Hours and Severity of Veterans Benefits 

Management System Outages by Month (January 2013-

May 2015) 25

Figure 3: Open Veterans Benefits Management System Defects 

(All Severity Levels) March 2014-March 2015 30

Figure 4: Estimated Percentage of All Roles of Claims Processors 

Who Were Overall Satisfied with VBMS, by Module 36

Figure 5: Estimated Percentage of All Roles of Claims Processors 

Compared to Decision Review Officers Who Were 

Satisfied with VBMS, by Module 37

Figure 6: Estimated Percentage of All Roles of Claims Processors 

Compared to Decision Review Officers Who Viewed 

VBMS as an Improvement over the Previous System(s), 

by Module 38

Figure 7: Estimated Percentage of All Roles of Claims Processors 

Compared to Decision Review Officers Who Would Have 

Chosen VBMS over the Previous System(s) to Process 

Claims, by Module 39

Page ii GAO-15-582 Veterans Benefits Management System
Abbreviations

IT information technology

OI&T Office of Information and Technology

PMO Program Management Office

SPAWAR Department of the Navy’s Space and Naval Warfare 

Systems Command

VA Department of Veterans Affairs

VBA Veterans Benefits Administration

VBMS Veterans Benefits Management System

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.

Page iii GAO-15-582 Veterans Benefits Management System
441 G St. N.W.

Washington, DC 20548

September 1, 2015

The Honorable Jeff Miller

Chairman

Committee on Veterans’ Affairs

House of Representatives

Dear Mr. Chairman:

The Department of Veterans Affairs’ (VA) disability compensation and 

pension benefit programs provide monetary benefits to veterans. 

Disability benefits are paid for disabling conditions incurred or aggravated 

while in military service. In fiscal year 2014, the department paid about 

$58 billion in disability compensation to nearly 3.9 million veterans and 

their survivors. In addition, VA’s pension program pays benefits to lowincome veterans who either are elderly or have disabilities unrelated to 

their military service. The program is available to veterans who are age 

65 and older or who have disabilities that are unrelated to their military 

service; it is also available to their surviving spouses and dependent 

children. In fiscal year 2014, VA paid approximately $5.2 billion in pension 

benefits to about 308,000 veterans and 213,000 survivors.

The disability compensation claims process has been the subject of 

attention by Congress, veterans’ service organizations, and others, due in 

part, to long waits for decisions and the large number of claims pending a 

decision. As such, in February 2010, the Secretary of Veterans Affairs 

committed the department to eliminating the disability claims backlog by 

the end of fiscal year 2015.1 In addition, the Secretary directed the 

Veterans Benefits Administration (VBA) to, by the end of fiscal year 2015, 

process all incoming claims within 125 days of their receipt and with at 

least 98 percent accuracy.2

To help reduce the backlog and meet its claims processing timeliness and 

accuracy goals, VBA has engaged in efforts to replace its paper-based 

claims process with an electronic paperless claims processing system 

1

VA defines a backlogged claim as one that has been awaiting a decision for more than 

125 days. 

2

In addition to the backlog claims, the department has an inventory of 403,032 disability

claims that have been pending processing for less than 125 days.

Letter

Page 1 GAO-15-582 Veterans Benefits Management System

 
called the Veterans Benefits Management System (VBMS). This system 

is intended to streamline the disability claims process by providing claims 

processors with an electronic environment in which to maintain, review, 

and make rating decisions for veteran’s claims. VBA is taking an 

incremental approach to developing and implementing the system and, as 

of June 2013, claims processors had begun using an initial version of the 

system at all 56 regional offices.

At your request, we conducted a study of VBMS. Our specific objectives 

were to (1) assess VA’s progress toward completing the development and 

implementation of VBMS and (2) determine to what extent users report 

satisfaction with the system.

To accomplish the objectives, we reviewed relevant program 

documentation and interviewed appropriate VA officials. We also 

administered a survey to VBMS users.

Specifically, to assess progress toward completing the development and 

implementation of VBMS, we obtained and analyzed program 

documentation, including plans that articulate the department’s goals and 

expected benefits from use of the system; project plans and schedules for 

iterative releases of different modules and generations of the system; 

contracts and related contractor documentation; and change 

management plans. We then compared program plans to VA’s policies for 

incremental system development,3 as well as federal guidance and 

information technology (IT) project management principles on program 

planning and cost estimating.4

Additionally, we obtained and analyzed data related to the system’s 

availability and performance covering the time period since its initial 

deployment in January 2013 through May 2015. We assessed the 

reliability of the data by reviewing it for obvious errors and missing data; 

corroborating the data with related documentation; and interviewing 

3

U.S. Department of Veterans Affairs, Office of Information and Technology, Project 

Management Accountability System (PMAS) Guide 5.0 (June 2014).

4

See, for example, OMB, Management of Federal Information Resources, Circular No. A130 (Washington, D.C.: November Nov. 28, 2000); GAO, Cost Estimating and 

Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, 

GAO-09-3SP (Washington, D.C.: March 2009); GAO, Schedule Assessment Guide: Best 

Practices for Project Schedules, GAO-12-120G (Washington, D.C.: May 2012). 

Page 2 GAO-15-582 Veterans Benefits Management System

 
responsible officials about their use of an automated tool that monitors 

system performance. We determined the data to be sufficiently reliable for 

the purposes of this report.

We compared system performance data with GAO and federal IT 

guidance for defining program goals and related performance targets that 

can be used to assess progress in achieving the goals.5

In addition, we obtained the VBMS defect management plan, compared 

the plan with key principles of sound defect management,6 and analyzed 

the extent to which processes described in the plan were followed during 

the development of recent system releases.

Further, we reviewed the department’s methods for soliciting end user 

feedback on the performance of the system (e.g., VBMS training strategy, 

operational management reviews, cost and performance reports, and 

end-of-month reports). We compared these methods to leading practices 

for obtaining customer feedback and collecting customer service data to 

5

GAO, Information Technology Investment Management: A Framework for Assessing and 

Improving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004); Executive 

Office of the President, Office of Management and Budget, Evaluating Information 

Technology Investments, A Practical Guide (November 1995).

6

The Project Management Institute, Inc., A Guide to the Project Management Body of 

Knowledge (PMBOK® Guide), Fifth Edition, (Newtown Square, Pa.: 2013), PMBOK is a 

trademark of the Project Management Institute, Inc.; Institute of Electrical and Electronics 

Engineers, Software and systems engineering — Software testing, ISO/TEC/IEEE Std 

29119 (New York, N.Y.: Sept. 1, 2013); Institute of Electrical and Electronics Engineers,

IEEE Standard for Software and System Test Documentation, IEEE Std 829-2008 (New 

York, N.Y.: July 10, 2008); Institute of Electrical and Electronics Engineers, IEEE Standard 

Classification for Software Anomalies, IEEE Std 1044-2009 (New York, N.Y.: Jan. 7, 

2010); Software Engineering Institute, CMMI® for Acquisition, Version 1.3 (Pittsburgh, Pa: 

November 2010); Software Engineering Institute, CMMI® for Development, Version 1.3

(Pittsburgh, Pa: November 2010); GAO, Year 2000 Computing Crisis: A Testing Guide, 

GAO/AIMD-10.1.21 (Washington, D.C.: November 1998); GAO Cost Estimating and 

Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, 

GAO-09-3SP (Washington, D.C.: March 2009); and GAO Schedule Assessment Guide: 

Best Practices for Project Schedules—Exposure Draft, GAO-12-120G (Washington, D.C.: 

May 2012).

Page 3 GAO-15-582 Veterans Benefits Management System

 
improve performance and demonstrate customer satisfaction of IT 

projects.7

To determine users’ satisfaction with VBMS, we administered a Webbased survey to a nationally representative stratified random sample of 

VBA users. These users (collectively referred to as claims processors) 

included claims assistants, veteran service representatives, supervisory 

veteran service representatives, rating veterans service representatives, 

decision review officers, and others.8

We pretested the survey and obtained initial insight regarding claims 

processors’ use of the system at two regional offices—in Philadelphia, 

Pennsylvania, and Baltimore, Maryland. We selected these offices based 

on their large size and location. We also observed and interviewed claims 

processors who were using VBMS at the VA Appeals Management 

Center in Washington, D.C., to gain additional insight into the use of the 

system. We received a response rate of 60 percent and adjusted for 

characteristics that were associated with survey response propensity 

using standard weighting class adjustments defined by sampling strata.9

As a result, we deemed the survey responses to be generalizable to all 

claims processors using the system at the 56 VBA regional offices. We 

7

GAO, Information Technology Investment Management: A Framework for Assessing and 

Improving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004); Carnegie 

Mellon Software Engineering Institute, Capability Maturity Model® Integration (CMMI) for 

Development, Version 1.3 (Pittsburgh, Pa.: November 2010); M.S. Garver and R.L. Cook, 

“Best Practice Customer Value and Satisfaction Cultures,” Mid-American Journal of 

Business, vol. 16, no. 1 (2001); M.S. Garver, “Modeling Best Practices for Government 

Agencies: Implementing Customer Satisfaction Programs” (Jan. 28, 2002); Best Practices, 

LLC, “Achieving World-Class Customer Service: An Integrated Approach” (copyright 1998-

2001); Federal Benchmarking Consortium, “Serving the American Public: Best Practices 

in Customer-Driven Strategic Planning” (February 1997); and Office of Management and 

Budget (OMB), Standards and Guidelines for Statistical Surveys (September 2006).

8

We randomly sampled 3,475 VBA-eligible claims processors to create estimates about 

the population of all claims processors. Confidence intervals for estimates we report from 

this survey are based on a confidence level of 95 percent and are calculated using 

methods appropriate for a stratified random sample. Confidence intervals for estimates in 

this report are never wider than plus or minus 5 percentage points. At a 95 percent 

confidence level, this means that in about 95 out of 100 instances, the sampling 

procedures we used would be expected to produce a confidence interval containing the 

true population value we estimate. See appendix I for more information.

9

We assumed that nonresponse adjusted data are missing at random and therefore 

concluded the respondent analyses using the nonresponse adjusted weights are unbiased 

for the population of VBMS users sampled in our survey. 

Page 4 GAO-15-582 Veterans Benefits Management System

 
supplemented our user observations with interviews of other VA officials 

in the Office of Field Operations, VBMS Program Management Office 

(PMO), and in VA’s Office of Information and Technology.

We conducted this performance audit from March 2014 through August

2015 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives. A more complete 

discussion of our objectives, scope, and methodology is provided in 

appendix I.

VA pays monthly compensation to veterans with service-connected 

disabilities (i.e., injuries or diseases incurred or aggravated while on 

active military duty) according to the severity of the disability.10 It pays 

additional compensation for certain dependent spouses, children, and 

parents of veterans. The department processes service-connected 

disability compensation benefits for about 3 million beneficiaries each 

month. It reported completing the processing of approximately 1.3 million 

disability rating claims for veterans and beneficiaries in fiscal year 2014—

about 150,000 more than in fiscal year 2013. As of mid-June 2015, the 

department reported a backlog of approximately 137,000 disability claims, 

down from its peak of about 611,000 disability claims in March 2013.

VA has reported that the accuracy of claims decisions rose from a level of 

83 percent in 2011 to 90 percent in 2014.11 However, even as VBA 

reported providing historic numbers of veterans with decisions on their 

claims in 2014, the number and complexity of claims continue to increase. 

For example, VBA reported for 2014 that its employees made decisions 

on 5.5 million medical issues within claims—a 101 percent increase in the 

number of issues since 2009.

1038 U.S.C. §§ 1101-1163. Section 1155 authorizes the VA Secretary to adopt and apply 

a schedule for rating disabilities that provides grades of disability upon which payments of 

compensation shall be based.

11Department of Veterans Affairs, Fact Sheet on Accelerating Access to Benefits.

Background

Page 5 GAO-15-582 Veterans Benefits Management System

 
Throughout the disability compensation claims process, VBA staff have 

various roles and responsibilities.

• Claims assistants are primarily responsible for establishing the 

electronic claims folders to determine whether the dispositions of the 

claims and control actions have been appropriately identified.

• Veteran service representatives are responsible for providing veterans 

with explanations regarding the disability compensation benefits 

programs and entitlement criteria. They also are to conduct 

interviews, gather relevant evidence, adjudicate claims, authorize 

payments, and input the data necessary to generate the awards and 

notification letters to veterans describing the decisions and the 

reasons for them.

• Rating veterans service representatives are to make claims rating 

decisions and analyze claims by applying VBA’s schedule for rating 

disabilities (rating schedule) against claims submissions, and 

preparing rating decisions and the supporting justifications. They also 

are to inform the veteran service representative who then notifies the

claimant of the decision and the reasons for the decision.

• Supervisory veteran service representatives are to ensure that the 

quality and timeliness of service provided by VBA meets performance 

indicator goals. They are also responsible for the cost-effective use of 

resources to accomplish assigned outcomes.

• Decision review officers are to examine claims decisions and perform 

an array of duties to resolve issues raised by veterans and their 

representatives. They may conduct a new review or complete a 

review of a claim without deference to the original decision; they also 

can revise that decision without new evidence or clear and obvious 

evidence of errors in the original evaluation.

The disability compensation claims process starts when a veteran (or 

other designated individual) submits a claim to VA, in paper or electronic 

form.12 If submitted electronically, a claim folder is created automatically. 

12Veterans and their beneficiaries can submit claims electronically through the eBenefits 

portal, a Web-based, online presence that combines data from the VBA and Department 

of Defense to provide veterans, active duty military, and their dependents with an alternate 

method to obtain assistance with a wide range of online benefits-related tools and 

information. Additionally, they can apply for benefits electronically using the Veterans 

Online Application.

Overview of VBA’s 

Disability Compensation 

Claims Process

Page 6 GAO-15-582 Veterans Benefits Management System

 
When a paper claim is submitted, a claims assistant creates the 

electronic folder. Specifically, when a regional office receives a new paper 

claim, the receipt date is recorded electronically and the paper files (e.g., 

medical records and other supporting documents) are shipped to one of 

four document conversion locations so that the supporting documents can 

be scanned and converted into a digital image.

In the processing of both electronic and paper claims, a veteran service 

representative is to review the information supporting the claim and help 

identify any additional evidence that is needed to evaluate the claim, such 

as the veteran’s military service records, medical examinations, and 

treatment records from medical facilities and private medical service 

providers. Also, if necessary to provide support to substantiate the claim, 

the department is to perform a medical examination on the veteran.

Once all of the supporting evidence has been gathered, a rating veterans 

service representative evaluates the claim and determines whether the 

veteran is eligible for benefits. If so, the rating veterans service 

representative assigns a disability rating (expressed as a percentage). A 

veteran who submits a claim with multiple disabilities receives a single 

composite rating. If the veteran is due to receive compensation, an award 

is prepared and the veteran is notified of the decision.

A veteran can reopen a claim for additional disability benefits if, for 

example, the veteran experiences a new or worsening service-connected 

disability. If the veteran disagrees with the regional office’s decision on 

the additional claim, he or she may submit a written notice of 

disagreement to the regional office to appeal the decision, and may 

request to have the appeal processed at the regional office by a decision 

review officer or through the Board of Veterans’ Appeals.13 Figure 1 

presents a simplified view of VA’s disability compensation claims process.

13The Board of Veterans’ Appeals makes final decisions on behalf of the VA Secretary on 

appeals from decisions of local VA offices. The Board reviews all appeals for entitlement 

to veterans’ benefits, including claims for service connection, increased disability ratings, 

total disability ratings, pension, insurance benefits, educational benefits, home loan 

guaranties, vocational rehabilitation, dependency and indemnity compensation, health 

care delivery, and fiduciary matters.

Page 7 GAO-15-582 Veterans Benefits Management System

 
Figure 1: Simplified View of VBA’s Disability Compensation Claims Process

VBA began the transformation of its paper-intensive claims process to a 

paperless environment in March 2009. This effort became formally 

established as the Veterans Benefits Management System program in 

May 2010. VBA’s initial plans for VBMS emphasized the development of 

a paperless claims platform to fully support the processing of disability 

compensation and pension benefits, as well as appeals. The program’s 

importance was further elevated in January 2013 when the Secretary of 

Veterans Affairs identified the system as the enabling technology to meet 

the goal to eliminate the disability claims backlog in fiscal year 2015 and 

improve the efficiency and accuracy of all compensation claims 

processing to 125 days and 98 percent, respectively. In a March 2013 

Senate Veterans Affairs Committee hearing, VA’s Under Secretary for 

Benefits stated that VBMS development was expected to be completed in 

2015.

The program’s primary focus was to convert existing paper-based claims 

folders into electronic claim folders (eFolders)14 to allow VBA employees 

14The eFolder is the electronic equivalent of a VBA paper claims folder. It contains all of 

the documents associated with a particular veteran and his or her claims.

Veterans Benefits 

Management System

Page 8 GAO-15-582 Veterans Benefits Management System

 
to access claims information and evidence in an electronic format. 

Beyond the establishment of eFolders, VBMS is also intended to 

streamline the entire disability claims process, from establishment 

through award, by automating rating decision recommendations, award 

and notification processes, and communications between VBA and the 

veteran throughout the claims life cycle; assist in eliminating the claims 

backlog and serve as the enabling technology for quicker, more accurate, 

and integrated claims processing in the future; and replace many of the 

key outdated legacy systems—which are still in use today—for managing 

the claims process, including:

• Share—used to establish claims; it records and updates basic 

information about veterans and dependents.

• Modern Award Processing–Development —used to manage the 

claims development process, including the collection of data to 

support the claims and the tracking of claims.

• Rating Board Automation 2000—provides information about laws and 

regulations pertaining to disabilities, which are used by rating 

specialists in evaluating and rating disability claims.

• Award—used to prepare and calculate the benefit award based on the 

rating specialist’s determination of the claimant’s percentage of 

disability. It is also used to authorize the claim for payment.

VBMS is to consist of three modules:

• VBMS-Core is intended to provide the foundation for document 

processing and storage during the claims development process, 

including establishing claims; viewing and storing electronic 

documents in the eFolder; and tracking evidence requested from 

beneficiaries. The eFolder serves as a digital repository for all 

documents related to a claim, such as the veteran’s military service 

records, medical examinations, and treatment records from VA and

Department of Defense medical facilities, and from private medical 

service providers. Unlike with paper files, this evidence can be 

reviewed simultaneously by multiple VBA claims processors at any 

location.

• VBMS-Rating is to provide raters with Web-accessible tools, including 

rules-based rating calculators and the capability for automated 

decision recommendations. For example, the hearing loss calculator 

is to automate decisions using objective audiology data and rulesbased functionality to provide the rater with a suggested rating 

decision. In addition, VBMS-Rating is expected to include stand-alone 

Page 9 GAO-15-582 Veterans Benefits Management System
evaluation builders—essentially interactive disability rating 

schedules—for all parts of the human body. With this tool, the rater 

uses a series of check boxes to identify the veteran’s symptoms, and 

the evaluation builder identifies the proper diagnostic code and the 

level of compensation based on those symptoms.

• VBMS-Awards is to provide an automated award and notification 

process to improve award accuracy and reduce rework associated 

with manual development of awards. VBMS-Awards is intended to 

automate and standardize communications between VBA and the 

veteran at the final stages of the claims process.

VBA is using an agile software development methodology to develop, 

test, and deliver VBMS functionality to its users. An agile approach allows 

subject matter experts to validate requirements, processes, and system 

functionality in increments, and to deliver the functionality to users in 

shorter cycles. To help guide its system development efforts, the VBMS 

PMO has developed both a strategic road map that identifies the 

program’s high-level objectives, timeline and intended outcomes, as well

as a regularly revised tactical road map that describes prospective 

capabilities the PMO expects to develop and deploy for each system 

release. Consistent with VA’s policy for incremental development, VBMS 

is to be developed and implemented in a series of releases that are to 

occur every 6 months. As shown in table 1, VBA plans to develop VBMS 

over multiple years, with each year generally corresponding to a new 

system generation.

VBMS Is Being Developed 

Incrementally and Was Initially 

Deployed In 2013

Page 10 GAO-15-582 Veterans Benefits Management System
Table 1: Strategic Objectives for Veterans Benefits Management System (VBMS) Development (2010-2016)

Generations of VBMS Strategic Objectives

Generation 1–Initiation From 2010 to January 2013, the program office’s strategic objectives for VBMS were to 

(1) conceptualize, pilot, and develop and deploy baseline system functionality; and (2) 

deliver a Web-based, electronic claims processing solution.

Generation 2– Deployment From February 2013 to September 2013, the program office’s strategic objectives for 

VBMS were to (1) build additional system capabilities, (2) leverage simple automation 

features, and (3) deploy the system to all regional offices.

Generation 3–Automation During fiscal year 2014, the strategic objectives for VBMS were to (1) provide more 

complex automation capabilities, (2) reduce dependency on legacy systems, and (3) 

deliver capability to accept veterans’ electronic service treatment records

Generation 4–National Workload 

Management

During fiscal year 2015, the strategic objectives for VBMS are (1) deliver the National 

Work Queue, (2) reduce reliance on legacy systems, and (3) institute processes and 

procedures within the program office to maximize business input for software delivery.

Generation 5–Integration Planned for fiscal year 2016, the program office has identified the following strategic 

objectives for VBMS: (1) continue to reduce the reliance on legacy systems, (2) 

integrate and enhance VBMS capabilities with Department of Defense systems, and 

(3) implement improvements to electronic communication and access to the eFolder. 

Source: VBMS program documentation. │ GAO-15-582

From 2010 through January 2013, VBA planned, developed, and 

deployed a foundational, Web-based version of VBMS to five pilot sites.

This phase included development of the eFolder capability and the ability 

to establish, develop, and rate disability compensation claims in VBMS, 

as well as a user interface with search capabilities. Generation 1 

development was completed in January 2013; at that time, the system 

was implemented at 18 regional offices.

From February 2013 through September 2013, the PMO continued to add 

functionality to the system, including enhancement of data exchange 

capabilities, correspondence tools, and rating functionality. By June 2013, 

VBA had completed national rollout of the initial version of the system to 

all 56 regional offices. The system was also made accessible to VA’s 

Appeals Management Center, the Board of Veterans’ Appeals, VBA 

National Call Centers, veterans service organizations, and all VA medical 

centers that complete compensation exams. Subsequent to the 

nationwide rollout of VBMS in June 2013, VBA has continued incremental 

system development and enhancement of VBMS.

The VBMS program is dependent on multiple organizations within the 

department to meet its goals. It is jointly led by two program managers: 

the VBMS PMO Director and a Program Manager in VA’s Office of 

Information and Technology (OI&T).

Multiple Organizations Have 

Responsibility for VBMS

Page 11 GAO-15-582 Veterans Benefits Management System
Specifically, the VBMS PMO, which resides in VBA’s Office of Strategic 

Planning, is responsible for all aspects of the program’s management, 

including the coordination and direction of contract staff and VBA partners 

in integrating the system’s components and managing program-level 

dependencies, risks, and issues.15 The PMO has responsibility for 

gathering and delivering system requirements, performing testing, and 

providing training. The Director oversees all activities of the PMO and 

reports to the Under Secretary for Benefits.

The OI&T program manager oversees the development and 

implementation of VBMS and reports to the VA Chief Information Officer. 

OI&T entered into an interagency agreement with the Department of the 

Navy’s Space and Naval Warfare Systems Command (SPAWAR) 

Systems Center Atlantic to lead the development of VBMS. SPAWAR 

manages multiple contractors to develop the system and is providing 

technical, information assurance, program management, testing, and data 

integration services to support application development. VA and 

SPAWAR work together to manage and develop the system. Specifically, 

VBA subject matter experts and OI&T technical representatives are part 

of the system development teams.

Further, VBA’s Office of Business Process Integration is to ensure that 

strategic needs and requirements for business and data systems are 

properly documented, integrated, and communicated. The office provides 

internal coordination across the VBA lines of business (compensation, 

pension, education, loan guaranty, etc.) and helps communicate the 

system’s requirements to the OI&T at the department level. The Office of 

Business Process Integration also manages VBA’s legacy claims 

processing systems and their sustainment. As such, it is working with the 

OI&T on long-term planning for the systems, which may include their 

decommissioning.

In addition to these offices, VBMS is governed by the Transformation 

Joint Executive Board. Jointly chaired by the VA CIO and the Under 

Secretary for Benefits, this board is responsible for discussing and 

addressing the program’s risks and other issues.

15VBA partners include stakeholder organizations such as the Department of Defense,

VA’s Veterans Health Administration, and veterans service organizations, among others.

Page 12 GAO-15-582 Veterans Benefits Management System

 
In addition, VBMS is to follow the OI&T’s Project Management 

Accountability System,16 which is the department’s process for managing 

IT projects. As part of this process, large IT programs are broken down 

into multiple projects that are typically planned, developed, and 

implemented in 2-year cycles. Each project is broken into several 

increments, with each increment typically lasting 6 months or less. The 

PMO is responsible for presenting a life-cycle cost estimate for the 2-year 

project as part of each increment’s milestone review. Resources, 

including staff and funding, are released once the increment has been 

approved.

In 2008, VBA developed an initial, high-level life-cycle cost estimate of 

$560 million for VBMS that reflected system development costs through 

fiscal year 2012. The PMO revised the cost estimate in 2011, to include 

costs for system development, sustainment, and general operating 

expenses. This estimate showed that the department expected to spend 

$934.8 million on VBMS through its life cycle.

In July 2012, we reported on the reliability of the cost estimate for the 

VBMS program17 and noted that, while the 2011 cost estimate for the 

system partially reflected key practices for developing a comprehensive 

and well-documented estimate,18 it did not reflect key practices for 

16VA implemented the Project Management Accountability System in June 2009 in an 

effort to improve the on-time delivery rate and strengthen accountability in its IT 

development efforts. According to VA documentation, PMAS adheres to eight major 

principles: incremental development, integrated teamwork across VA, accountability, 

resource management, transparency, senior leadership engagement, direct participation 

by the customer, and an emphasis on agile practices.

17GAO, Information Technology Cost Estimation: Agencies Need to Address Significant 

Weaknesses in Policies and Practice, GAO-12-629 (Washington, D.C.: July 11, 2012).

18GAO-09-3SP defines a comprehensive cost estimate as one that accounts for all 

possible costs associated with a program, is structured in sufficient detail to ensure that 

costs are neither omitted nor double counted, and documents all cost-influencing 

assumptions. Further, a cost estimate is well-documented when supporting documentation 

explains the process, sources, and methods used to create the estimate, contains the 

underlying data used to develop the estimate, and is adequately reviewed and approved 

by management.

VBMS Has Been Provided with 

About $1 Billion in Funding

Page 13 GAO-15-582 Veterans Benefits Management System

 
developing an accurate and credible estimate.19 We recommended that 

any future life-cycle cost estimates for the VBMS program address the 

detailed weaknesses that we identified using cost-estimating best 

practices. VA’s Chief of Staff stated that the department concurred with 

our recommendation and had efforts under way to improve its costestimating capabilities.

Officials from the PMO subsequently provided us information showing 

that from fiscal years 2009 through 2015,20 the VBMS program had 

received funding in the amount of approximately $1 billion— about $502 

million for system development, $308 million for IT sustainment,21 and 

$194 million for general operating expenses.

19GAO-09-3SP states that a cost estimate is accurate when it is not overly conservative or 

optimistic, is based on an assessment of the costs most likely to be incurred, and is 

regularly updated so that it always reflects the current status of the program. In addition, 

the estimate is credible when any limitations of the analysis are discussed (because of 

uncertainty or sensitivity surrounding data or assumptions), the estimate’s results are 

cross-checked, and an independent cost estimate is conducted by a group outside the 

acquiring organization to determine whether other estimating methods produce similar 

results.

20For fiscal year 2015, the program office provided information on funding that was current 

through April 15, 2015.

21According to VA documentation, IT sustainment consists of two categories: marginal 

and mandatory. Marginal sustainment refers to the portion of the costs that are required to 

deploy system functionality. Mandatory sustainment refers to the funding necessary to 

maintain the IT infrastructure and operation of the IT systems.

Page 14 GAO-15-582 Veterans Benefits Management System

 
VBA has developed and implemented capabilities for VBMS to support 

disability claims processing. However, completion of additional 

functionality to fully support processing disability claims will be delayed 

beyond fiscal year 2015. Further, VBA’s plans for developing and 

implementing capabilities to support the processing of pension benefits 

and appeals are uncertain. As VBA continues to develop and implement 

the system, three areas could benefit from increased management 

attention. First, the PMO does not have a reliable estimate of the cost for 

completing the system. Second, although VBA has improved VBMS’s 

availability to users, it has not established goals for system response 

times. Third, while the program has actively managed system defects, a 

recent system release included multiple unresolved defects that adversely 

impacted performance and users’ experiences.

Since completing the implementation of VBMS at all regional offices in 

June 2013, VBA has continued to make important progress toward 

developing and implementing additional system functionality and 

enhancements that support electronic processing of disability 

compensation claims. As a result, 95 percent of records related to 

veterans’ disability claims are electronic and reside in VBMS. However, 

although the Under Secretary for Benefits stated in March 2013 that the 

system’s development was expected to be completed in 2015, 

implementation of functionality to fully support electronic claims 

processing has been delayed beyond 2015. Additionally, federal guidance 

and IT project management principles stress the importance of 

continuous planning throughout the life of a program to serve as a basis 

for managing trade-offs between cost, schedule, and scope.22

During fiscal year 2014, VBA’s progress on the system included 

developing and implementing capabilities that addressed the strategic 

objectives for generation 3 of the strategic road map. For example:

• Consistent with the objective to provide system users with more 

complex automation capabilities, the PMO released an automation 

22See, for example, OMB, Management of Federal Information Resources, Circular No. A130 (Washington, D.C.: November 28, 2000); GAO, Cost Estimating and Assessment 

Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP

(Washington, D.C.: March 2009); GAO, Schedule Assessment Guide: Best Practices for 

Project Schedules—Exposure Draft, GAO-12-120G (Washington, D.C.: May 2012).

VA Has Made 

Progress toward 

Developing and 

Implementing VBMS, 

but Ongoing Activities 

Could Benefit from 

Increased 

Management 

Attention

VBA Has Made Important 

Progress toward 

Developing and 

Implementing VBMS to 

Support the Processing of 

Disability Claims; but Key 

Capabilities Are Delayed 

and Plans for Processing 

Pension Benefits and 

Appeals Are Uncertain

Page 15 GAO-15-582 Veterans Benefits Management System

 
feature that populates the VBMS rating calculator with a veteran’s 

medical information for 37 of the 71 disability benefits questionnaires 

that medical providers can use to submit medical information in 

support of a disability claim.23

• Regarding the strategic objective to reduce dependency on legacy 

systems, in April 2014, the PMO delivered the first iteration of the 

VBMS-Awards module to all regional offices, providing claims 

processors the ability to rely less on legacy systems, prepare an 

award based on information from VBMS’s rating module, and 

generate associated notification letters for veterans. The functionality 

released during fiscal year 2014 also included the integration of 

VBMS with VA’s legacy electronic records store, “Virtual VA,” and the 

integration of additional correspondence letters.

• To address the strategic objective to deliver a capability to accept 

veterans’ electronic service treatment records, in December 2013, the 

PMO implemented new functionality in VBMS to enable electronic 

request and receipt of service treatment records from the Department 

of Defense for veterans who had separated or been discharged from 

military service after January 1, 2014.24

VBA has continued its progress toward developing and implementing 

VBMS during fiscal year 2015 with efforts to address generation 4 of the 

strategic road map. Specifically, to address the strategic objective calling 

23According to VBA, the single largest category of work that delays claims processing is

medical examinations that contain insufficient information to rate the claim. In October 

2010, VBA developed disability benefits questionnaires to help address this issue. A 

questionnaire is a standardized, downloadable form for private or VA medical providers to 

submit medical information needed to support a disability claim. The disability benefits 

questionnaire replaces the traditional VA examination report and is mapped to the VA 

Schedule for Rating Disabilities. These forms can be mailed to VBA. The VBMS program 

office plans to develop functionality to automate the remaining 34 questionnaires in VBMS 

in future releases.

24The service treatment record is a key piece of evidence in the disability claims process, 

as it contains certified information on the medical and dental care received by service 

members during their military career. Previously, a veteran’s service treatment record was 

requested via mail or fax and, according to VBMS program office officials, this information 

could take several weeks to arrive at VBA, contributing to delays in processing a veteran’s 

claim. According to DOD Instruction 6040.45, the record must contain the veteran’s 

complete medical and dental records, and a certification form attesting that this 

information has been released. This information is required to be available in DOD’s 

Health Artifact and Image Management Solution system for transmission to VA in a 

complete package within 45 days of the veteran’s separation/retirement from the military.

Page 16 GAO-15-582 Veterans Benefits Management System

 
for further reduced reliance on legacy systems, the PMO has made 

enhancements to the system’s correspondence capabilities and letter 

templates, and added several new tools to the rating module. Additionally, 

the department has established plans and a schedule for retiring one 

legacy system. According to an analysis by VBA’s Office of Business 

Process Integration, fewer than 5 percent of disability cases are rated 

using the legacy rating system, Rating Board Automation 2000, and the 

functionality available in VBMS-Rating has largely eliminated the need for 

claims processors to access the legacy rating system.

In February 2015, VBA conducted a 5-week pilot at four regional offices 

to determine whether VBA’s Office of Business Process Integration can 

proceed with plans to retire the legacy rating application. According to 

VBMS program officials, the Office of Business Process Integration 

expects to retire the legacy rating system by the end of September 2015. 

According to the PMO Director, the office has also been working to 

capture process improvements and use lessons learned from earlier 

system development cycles to address the generation 4 strategic 

objective to implement improved system development processes.

Even with the progress VBA has made toward developing and 

implementing VBMS, the timeline for initial deployment of a national 

workload management capability has been delayed beyond the originally 

planned date of September 2014 to October 2015, with additional 

deployment to occur throughout fiscal year 2016. VBMS generation 4 

development efforts have included addressing the strategic objective that 

calls for delivery of a national workload management capability. This 

effort has entailed developing the technology and business processes 

needed to support the national work queue, which is intended to handle 

new disability claims in a centralized queue and assign claims to the next 

regional office with available capacity.25

The PMO began work for the national work queue in June 2014. The 

office had intended to deploy the first phase of the work queue 

25Traditionally, veterans have submitted disability claims—typically via mail—to their local 

regional office, where the claims are usually also processed. Under the previous paperbased model, claims folders were physically stored and processed at the regional office, 

and material was often mailed between the veteran, the regional office, and the closest VA 

medical facility. This paper-based business process is no longer necessary, now that 95 

percent of all disability claims are digital and all regional offices use VBMS.

Page 17 GAO-15-582 Veterans Benefits Management System

 
functionality to users in September 2014. However, in late May 2015, the 

PMO Director informed us that VBA had decided to delay the initial rollout 

of the work queue to October 2015 so that the department can fully focus 

on meeting its goal to eliminate the claims backlog by the end of 

September 2015. Following the initial rollout, the PMO intends to 

implement the work queue at all regional offices through fiscal year 2016.

VBMS program documentation also identifies additional work to be 

performed after fiscal year 2015 to fully automate disability claims 

processing.26 Specifically, the PMO has identified the need to automate 

steps associated with a veteran’s request for an increase in disability 

benefits, such as when an existing medical condition worsens. In addition, 

according to the Director, the PMO intends to develop a capability to 

automatically associate veterans’ correspondence when a new piece of 

evidence to support a claim has been received electronically or scanned 

into VBMS. The PMO also plans to integrate VBMS with VA’s Integrated 

Disability Evaluation System, which contains the results of veterans’

disability medical examinations,27 as well as with external systems that 

contain military service treatment records for veterans, including those at 

the National Personnel Records Center.

Further, VBA has not yet developed and implemented end-to-end pension 

processing capabilities in VBMS. Without such capabilities, VBA 

continues to rely on three legacy systems to process pension claims. 

Specifically, according to program officials, both the Modern Award 

Processing–Development and Award legacy systems contain functionality 

related to processing pensions and will need to remain operational until 

VBMS can process these claims. In addition, they said that the Share 

legacy system contains functionality that is still needed throughout the 

claims process.

26As of June 2015, claims processors are directed to establish all initial and supplemental 

compensation claims in VBMS, with several exclusions including pension claims, dual 

compensation and pension claims, sensitive cases, and claims where the claimant is not 

the veteran.

27Managed by both VA and DOD, the Integrated Disability Evaluation System provides a 

single set of disability medical examinations designed for determining a service member’s 

(1) fitness and ability to return to duty and (2) disability if the service member is inhibited 

from performing his or her assigned duties as a result of a service-connected injury or 

illness. DOD’s assessment of fitness for duty occurs concurrently with the VA disability 

determination process.

Page 18 GAO-15-582 Veterans Benefits Management System

 
VBMS program officials stated that additional system analysis is needed 

before they can develop plans for the retirement of Share.28 Program 

documentation indicates that the first phase of pension-related 

functionality is expected to be introduced in December 2015. However, 

VBA has not yet developed plans and schedules for retiring the Modern 

Award Processing-Development, Award, and Share systems and fully 

developing and implementing the functionality of these legacy systems in 

VBMS.

VBA’s progress toward developing and implementing appeals processing 

capabilities in VBMS has also been limited. Specifically, although the 

information in a veteran’s eFolder is available to appeals staff for review, 

the appeals process for disability claims is not managed using the new 

system. According to VA’s fiscal year 2016 budget submission, the 

department is pursuing a separate effort to manage end-to-end appeals 

modernization, and has requested $19.1 million in fiscal year 2016 funds 

to develop a system that will provide functionality not available in VBMS 

or other VA systems. The PMO Director stated that VBA is currently 

analyzing commercial IT solutions that can meet the business 

requirements for appeals, such as providing document navigation 

capabilities. Nevertheless, the Director added that VBMS is expected to 

be part of the appeals modernization solution because components of the 

system, such as the VBMS eFolder and certain workload management 

functionality, are planned to continue supporting appeals management.

According to the PMO Director, the fact that VBMS requires additional 

development beyond 2015 does not reflect a delay in completing the 

system’s development. Instead, the additional time is a consequence of 

decisions to enlarge the VBMS program’s scope over time. This official 

added that VBMS’s original purpose was to serve primarily as an 

electronic document repository, and they have met this goal.

The PMO Director further stated that, as the program’s mission has 

expanded to support the department’s efforts to eliminate the disability 

claims backlog, the PMO has had to re-prioritize, add, and defer system 

28These statements regarding the use of legacy systems are consistent with the results of 

our survey of VBMS claims processors. In addition to VBMS, an estimated 52 percent of 

users depend on Share, an estimated 37 percent depend on Modern Award Processing–

Development, and an estimated 13 percent depend on Award Processing “a great deal” in 

order to process claims.

Page 19 GAO-15-582 Veterans Benefits Management System

 
requirements to accommodate broader departmental decisions and, in 

some cases, regulatory changes. For example, the PMO was tasked with 

developing functionality in VBMS to meet regulatory requirements for 

processing disability claims using mandatory forms.29 Officials from the 

VBMS PMO explained that they were made aware of this requirement 

well after system planning for the March 2015 release had been 

completed, and it introduced significant complexity to their development 

work.

Finally, VBA included in its strategic road map the strategic objectives 

that are to be addressed in generation 5 of the system, which is planned 

for fiscal year 2016. Further, officials from the VBMS PMO stated that 

they intend to develop tactical plans that identify the expected capabilities 

to be provided in the generation 5 releases. Nevertheless, due to the 

department’s incremental approach to developing and implementing 

VBMS, VBA has not yet produced a plan that identifies when the system 

will be completed and can be expected to fully support disability and 

pension claims processing and appeals. Thus, it will be difficult for VA to 

hold its managers accountable for meeting its time frame and for 

demonstrating progress.

Consistent with our guidance on estimating program costs, an important 

aspect of planning for IT projects such as VBMS involves developing a 

reliable cost estimate to help managers evaluate a program’s affordability 

and performance against its plans, and provide estimates of the funding 

required to efficiently execute a program.30 Without this information, 

programs are at risk of experiencing cost overruns, missed deadlines, 

and performance shortfalls. Additionally, federal guidance and IT project 

management principles stress the importance of continuous planning 

29Department of Veterans Affairs, Standard Claims and Appeals Forms, Final Rule 79

Fed. Reg. 57660 (Sept. 25, 2014).The effective date for the Final Rule is March 2015.

30GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and 

Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009).

Ongoing Progress of 

VBMS Development and 

Implementation May Be 

Hindered by Lack of a 

Reliable Cost Estimate

Page 20 GAO-15-582 Veterans Benefits Management System

 
throughout the life of a program to serve as a basis for managing tradeoffs between cost, schedule, and scope.31

In July 2012, we identified several weaknesses in VA’s policies related to 

cost estimating and also found that the department’s 2011 cost estimate 

for VBMS did not reflect key practices for developing an accurate and 

credible estimate. At that time, we recommended that VA modify its 

policies governing cost estimating to establish, among other things, a 

requirement to prepare a full life-cycle cost estimate for a program (as 

opposed to preparing an estimate for each program increment) and a 

requirement for programs to prepare cost estimates using best practices. 

In February 2013, VA’s Office of Corporate Analysis and Evaluation 

released a cost estimating process guide, which states that VA program 

offices are required to develop a life-cycle cost estimate to support senior 

leaders’ decision making and the department’s planning and budgeting 

processes. Further, this guide states that, at the request of VA leadership 

or a program office, VA’s Office of Corporate Analysis and Evaluation is 

to be available to develop independent cost estimates and review existing 

program cost estimates to ensure they have been developed in 

accordance with our cost-estimating guidance.

In 2011, VBA submitted a life-cycle cost estimate for VBMS of $934.8 

million to the Office of Management and Budget. This estimate was 

intended to capture costs for the system’s development, deployment, 

sustainment, and general operating expenses through the end of fiscal 

year 2018. However, as of July 2015, the program’s actual costs 

exceeded the 2011 life-cycle cost estimate. Specifically, VBMS has 

received approximately $1 billion in funding through the end of fiscal year 

2015 and the department has requested an additional $290 million for the 

program in fiscal year 2016.32

31See, for example, OMB, Management of Federal Information Resources, Circular No. A130 (Washington, D.C.: November 28, 2000); GAO, Cost Estimating and Assessment 

Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP

(Washington, D.C.: March 2009); GAO, Schedule Assessment Guide: Best Practices for 

Project Schedules—Exposure Draft, GAO-12-120G (Washington, D.C.: May 2012).

32The $1 billion figure represents funding for VBMS IT development, sustainment, and 

general operating expenses for fiscal years 2009 through 2015. For fiscal year 2016, VA 

has requested $76 million for IT development, $177 million for sustainment, and $37 

million for general operating expenses. 

Page 21 GAO-15-582 Veterans Benefits Management System

 
The PMO has not reliably updated VBMS’s life-cycle cost estimate to 

reflect the program’s expanded scope and timelines for completing the 

system. This is largely attributable to the fact that the PMO has developed 

cost estimates for 2-year project cycles that are used for VBMS milestone 

reviews under OI&T’s Project Management Accountability System.

When asked how the PMO arrived at the cost estimates reported in the 

milestone reviews, officials stated that they develop rough order of 

magnitude estimates for each business need based on expert knowledge 

of the system, past development and engineering experience, and 

lessons learned. However, while this approach may have provided 

adequate information for VBA to prioritize VBMS system requirements to 

address in the next release, it has not produced estimates that could 

serve as a basis for identifying VBMS’s funding needs. Because it is 

typically derived from limited data and in a short time, a rough order of 

magnitude analysis is not equivalent to a budget-quality cost estimate and 

may limit an agency’s ability to identify the funding necessary to efficiently 

execute a program.

In addition, the PMO’s annual operating plan, which is generally limited to 

high-level information about the program’s organization, priorities, 

staffing, milestones, and performance measures for fiscal year 2015, also 

shows estimated costs totaling $512 million for VBMS development from 

fiscal years 2017 through 2020. However, according to the PMO Director, 

this estimate was also developed using rough order of magnitude 

analysis. Further, the estimate does not provide reliable information on 

life-cycle costs because it does not include estimated IT sustainment and 

general operating expenses.

Even though the PMO has developed rough order of magnitude cost 

estimates for VBMS, these estimates have not been sufficiently reliable to 

effectively identify the program’s funding needs. Instead, during the last 3 

fiscal years, the PMO Director has had to request an additional $118 

million in IT development funds to meet program demands and to ensure 

support for ongoing development contracts.33 Specifically, in May 2013, 

33In fiscal years 2013, 2014, and 2015, VA requested permission from Congress to 

reprogram OI&T funds that had not been used in the prior year, or had been identified for 

IT sustainment, to instead support IT development. In fiscal year 2014, the transfer of 

funds was also intended to allow IT development work to continue while the department 

waited for its final appropriation.

Page 22 GAO-15-582 Veterans Benefits Management System

 
VA requested $13.3 million to support additional work on VBMS. Then, 

during fiscal year 2014, VA reprogrammed $73 million of unobligated IT 

sustainment funds to develop functionality to transfer service treatment 

records from DOD to VA, and to support development of VBMS-Core 

functionality. In December 2014, the PMO identified the need for 

additional fiscal year 2015 funds for ongoing system development 

contracts for VBMS-Core and VBMS-Awards, and, in late April 2015, VA 

leadership submitted a letter to Congress requesting permission to 

reprogram $31.7 million to support work on these contracts, the National 

Work Queue, and other VBMS efforts.

According to the PMO Director, the need to request additional funding 

does not represent additional risk to the program, but is the result of 

VBMS’s success. The Director further noted that, as the PMO has 

identified opportunities to increase functionality to improve the electronic 

claims process, their funding needs have also increased. We recognize 

that as new capabilities are deployed, additional requirements may 

surface. Nevertheless, evolution of the VBMS program illustrates the 

importance of continuous planning, including cost estimating, so that 

trade-offs between cost, schedule, and scope can be effectively 

managed. Further, without a reliable estimate of the total costs associated 

with completing work on VBMS, stakeholders will have a limited view of 

VBMS’s future resource needs and the program is at risk of not being 

able to secure appropriate funding to fully develop and implement the 

system.

GAO and federal IT guidance recognize the importance of defining 

program goals and related performance targets and using such targets to 

assess progress in achieving the goals.34 System performance and 

response times have a large impact on whether staff successfully 

complete work tasks. If systems are not responding at agreed-upon levels 

for availability and performance, it can be difficult to ensure that staff will 

complete tasks in a timely manner. This is especially important in the VBA 

claims processing environment, where staff are evaluated on their ability 

to process claims in a timely manner.

34GAO, Information Technology Investment Management: A Framework for Assessing 

and Improving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004) and 

Executive Office of the President, Office of Management and Budget, Evaluating 

Information Technology Investments, A Practical Guide (November 1995).

VBA Has Made Progress 

toward Improving VBMS 

Implementation, but Does 

Not Have Goals for 

System Response Times

Page 23 GAO-15-582 Veterans Benefits Management System

 
VBA’s availability goal for VBMS is 95 percent per month, operating on a 

24 hour-a-day, 6-day-a-week schedule. According to the OI&T Program 

Manager for VBMS, VBA defines availability as the extent to which the 

system is operational, functional, and usable for completing business 

requirements (e.g., processing claims). When an unscheduled system 

outage occurs, the office identifies it as being in one of three categories: 

critical, impacting all VBMS users across VA; moderate, impacting VBMS 

users at a local or individual level; and serious, impacting VBMS users in 

a certain regional area. The Program Manager stated that the department 

does not consider the system to be unavailable if users are unable to 

perform their duties due to non-system issues (such as local network 

issues or the unavailability of systems that feed data into VBMS).

VBA has reported that, since its initial rollout in January 2013, the system 

has exceeded the 95 percent goal for availability. Specifically, VBA 

reported that the system was available at a rate of 98.9 percent in fiscal 

year 2013 and 99.3 percent in fiscal year 2014. Through May of fiscal 

year 2015, it was available for 99.98 percent of the time.

Nonetheless, while the department reported exceeding its availability 

goals for VBMS, it has experienced periods of system unavailability, 

many times at the critical level affecting all users, as reflected in figure 2.

Page 24 GAO-15-582 Veterans Benefits Management System
Figure 2: Reported Hours and Severity of Veterans Benefits Management System Outages by Month (January 2013-May 2015)

Specifically, since January 2013, the department has reported 57 VBMS 

outages that have totaled about 117 hours of system unavailability. VBA 

experienced about 18 hours of VBMS outages in January 2014, which 

were almost entirely at the critical level and affected all users. To the 

department’s credit, it reported experiencing only 2 system outages since 

July 2014—a 30-minute critical outage in December 2014 and a 23-

minute critical outage in May 2015.

In addition to system availability, VBA monitors system response times for 

each of the VBMS modules using an application that measures the 

amount of time taken for each transaction. The PMO defines response 

time as the time that elapses from when a user executes a transaction 

(i.e., clicks a link or selects “Enter”) to when the resulting page fully loads 

on the user’s screen. The PMO uses average page response times to 

measure VBMS system performance, with lower response times 

Page 25 GAO-15-582 Veterans Benefits Management System
indicating optimal system performance and higher response times 

indicating performance issues.

From September 2013 through April 2015, VBA reported a decrease in 

average response times for VBMS-Core and VBMS-Rating (see table 

2).35

Table 2: VA’s Reported Mean Transaction Response Times for Veterans Benefits 

Management System (VBMS) Modules 

Time Period

Mean response time 

for VBMS-Core

Mean response time 

for VBMS-Rating

Mean response time 

for VBMS-Awards 

September 2013 

through midFebruary 2014

6.25 seconds 4.91 seconds not applicable

Mid-February 

2014 through midJuly 2014

4.61 seconds 2.69 seconds not applicable

Mid-July 2014 

through 

December 2014

4.03 seconds 2.12 seconds 5.22 seconds

January 2015 

through April 2015

4.41 seconds 2.36 seconds 5.62 seconds

Source: VA-reported data. | GAO-15-582

VBA attributed the decrease in response times to continuous engineering 

improvements to system performance. Program officials also explained 

that the difference in response times between modules was due to the 

type of information that is being pulled into each module from various 

other VBA systems. For example, both VBMS-Core and VBMS-Rating 

require information from the VBA corporate database, but VBMS-Core is 

populated with data from multiple VBA systems in addition to the 

corporate database.

Program officials told us there are no specific goals for mean transaction 

response times because they feel that there are adequate tools in place 

to monitor system performance and provide alerts if there are response 

time issues. For example, VBMS performance is monitored in real time by 

dedicated staff at a contractor’s facility and users have access to a live 

35VBMS-Awards was deployed in April 2014.

Page 26 GAO-15-582 Veterans Benefits Management System

 
chat feature where they can provide feedback on any issues they are 

experiencing with the system.

The VBMS help desk provides another avenue for users to provide 

feedback on the system’s performance. Officials also noted that, because 

transaction response times have decreased, which can be indicative of an 

improvement to system performance, they are focusing their resources on 

adding additional functionality instead of trying to get the system to 

achieve a specific average transaction response time.

While VBA’s monitoring of VBMS’s performance is commendable and the 

system’s performance and response time have improved over time, the 

system is still in development and there is no guarantee that performance 

will remain at current levels as the system evolves. Performance targets 

and goals for VBMS response times would provide users with an 

expectation of the system response times they should anticipate, and 

management with an indication of how well the system is performing 

relative to performance goals.

Page 27 GAO-15-582 Veterans Benefits Management System
A key element of successful system testing is appropriately identifying 

and handling defects that are discovered during testing. Outstanding 

defects can delay the release of functionality to end users, denying them 

the benefit of features. Key aspects of a sound defect management 

process include the planning, identification and classification, tracking, 

and resolution of defects. Leading industry and government organizations 

consider defect management and resolution to be among the primary 

goals of testing.36

The VBMS program has defect management policies in place and it is 

actively performing defect management activities. Specifically, in October 

2012, the department developed a VBMS Program Management and 

Technical Support Defect Management Plan, Version 1.0, which 

describes the program’s defect management process. The plan was 

updated in March 2015 and describes, among other things, the process 

for identifying, classifying, tracking, and resolving VBMS defects. For 

example, it provides criteria for assigning four different severity levels for 

defects—critical, high, medium, and low.37

According to the plan, critical severity defects are characterized by 

complete system or subsystem failure, complete loss of functionality, and 

compromised security or confidentiality. Critical defects also have 

extensive user impact and workarounds do not exist. High severity 

defects can have major user impact, leading to significant loss of system 

36The Project Management Institute, Inc., A Guide to the Project Management Body of 

Knowledge (PMBOK® Guide), Fifth Edition, (Newtown Square, Pa.: 2013); Institute of 

Electrical and Electronics Engineers, Software and systems engineering — Software 

testing, ISO/TEC/IEEE Std 29119 (New York, N.Y.: Sept. 1, 2013); Institute of Electrical 

and Electronics Engineers, IEEE Standard for Software and System Test Documentation, 

IEEE Std 829-2008 (New York, N.Y.: July 10, 2008); Institute of Electrical and Electronics 

Engineers, IEEE Standard Classification for Software Anomalies, IEEE Std 1044-2009 

(New York, N.Y.: Jan. 7, 2010); Software Engineering Institute, CMMI® for Acquisition, 

Version 1.3 (Pittsburgh, Pa: November 2010); Software Engineering Institute, CMMI® for 

Development, Version 1.3 (Pittsburgh, Pa: November 2010); GAO, Year 2000 Computing 

Crisis: A Testing Guide, GAO/AIMD-10.1.21 (Washington, D.C.: November 1998); GAO 

Cost Estimating and Assessment Guide: Best Practices for Developing and Managing 

Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009); and GAO 

Schedule Assessment Guide: Best Practices for Project Schedules—Exposure Draft, 

GAO-12-120G (Washington, D.C.: May 2012).

37In addition to the defect severity level, these four defect priority-level assignments are

used to designate the immediacy of repair: (1) resolve immediately, (2) give high attention, 

(3) normal queue, and (4) low priority.

VBMS Program Has 

Actively Managed System 

Defects, but a Recent 

Release Included 

Unresolved Defects that 

Adversely Impacted 

System Implementation

Page 28 GAO-15-582 Veterans Benefits Management System

 
functionality. Medium severity defects can have moderate user impact 

and lead to moderate loss of functionality. Low severity defects lead to 

minor loss of functionality with no workaround necessary. For high and 

medium severity defects, workarounds could exist. According to the PMO, 

high, medium, and low defects do not need to be resolved prior to a 

system release.

To monitor and track defects, the PMO uses an automated tool to 

maintain the VBMS defect repository. This tool is used to produce a daily 

defect management report that is shared with VBMS leadership. The 

purpose of the daily defect management report is to provide the current 

status of all open defects identified in testing of a forthcoming VBMS 

release or identified during production of a previous release.38

According to the defect management plan, defects can be resolved in a 

number of different ways. Once a defect is fixed, tested, and has passed 

testing, it is considered done or resolved. Defects that cannot be 

attributed to an existing requirement are reclassified as a system 

enhancement and considered resolved, as they do not affect a current 

system release requirement. A defect is also considered resolved if it is

determined to work as designed, duplicate another defect, or if it is no 

longer evident in the system.

From March 2014 through March 2015, the total number of VBMS defects 

declined as release dates approached for releases 7.0, 7.1, 8.0, and 8.1. 

Additionally, to the department’s credit, no critical defects remained at the 

time of each of these releases. Specifically, prior to VBMS Release 7.0, 

the total number of defects identified peaked at 226, while 97 remained at 

the time of release. Prior to release 7.1, the total number of defects 

identified peaked at 330, with 169 remaining at the time of release. Prior 

to release 8.0, the total number of defects identified peaked at 330, with 

161 remaining at the time of release. Prior to release 8.1, the total 

number of defects remaining peaked at 421, with 254 remaining at the 

time of release. Figure 3 shows the trend in the number of open total 

defects for this time period.

38The daily defect management report consists of the following data: (1) total critical and 

high, priority one defects for resolution; (2) total number of critical defects; (3) total number 

of high, priority one defects; and (4) total defects for resolution. 

Page 29 GAO-15-582 Veterans Benefits Management System

 
Figure 3: Open Veterans Benefits Management System Defects (All Severity Levels) March 2014-March 2015

Even with the department’s efforts to resolve defects prior to a VBMS 

release, defects that affected system functionality remained open at the 

time of the releases. Specifically, of the 254 open defects at the time of 

VBMS release 8.1, 76 were high severity, 99 were medium severity, and 

79 were low severity. Examples of defects that remained open at the time 

of VBMS release 8.1 are described in table 3.

Page 30 GAO-15-582 Veterans Benefits Management System
Table 3: Examples of Open Defects for Veterans Benefits Management System 

(VBMS) Release 8.1

Severity Description

High The letter notifying the veteran of the claim decision was populated with an 

incorrect diagnosis.

High VBMS-Core did not recognize updated rating decisions from VBMS-Rating.

High There was a discrepancy between VBMS and Share (the legacy system) in 

the date of claim.

High The new mail indicator was malfunctioning on users’ queues.

High Disability name appears incorrectly in Issue and Decision text for ALS.

High Discrepancies in military service information displaying in the Beneficiary 

Identification Records Locator Subsystem (BIRLS)a and VBMS.

High Creating a claim in legacy or VBMS will remove the Homeless, POW, 

and/or Gulf War Registry Flash.

High When establishing new claims, the power of attorney box is not always 

checked to allow power of attorney access to the veteran’s eFolder.

Medium The intent to file for compensation/pension has an active status for a 

deceased veteran.

Medium E-mail addresses for dependents only occasionally allowed special 

characters.

Source: VBMS Release 8.1 Daily UAT Defect Management Report 3/20/2015 | GAO-15-582

a

BIRLS is an electronic information system administered and used by VBA to verify and confirm 

veteran information.

According to the PMO, these defects were communicated to users and an 

appropriate workaround for each was established. Nevertheless, even 

with workarounds, high- and medium-severity open defects, which by 

definition impact system functionality, degraded users’ experiences with 

the system. Continuing to deploy system releases with defects that 

impact system functionality increases the risk that these defects will 

diminish users’ ability to process disability claims in an efficient manner.

Page 31 GAO-15-582 Veterans Benefits Management System
While VBA has several methods to obtain VBMS users’ feedback, it has 

neither established goals to define user satisfaction, nor conducted a 

survey of claims processing employees to obtain a comprehensive picture 

of overall customer satisfaction. Our survey of VBMS users found that a 

majority reported satisfaction with the system, but decision review officers 

were considerably less satisfied. Although the results of our survey 

provide VBA with useful data about users’ satisfaction with the system, 

the absence of user satisfaction goals limits the utility of survey results.

GAO and federal IT guidance recognize the importance of defining 

program goals and related performance targets and using such targets to 

assess progress in achieving the goals.39 Also, leading practices identify 

continuous customer feedback as a crucial element of IT project success, 

from project conception through sustainment. Particularly for IT projects 

like VBMS, where development activities are iterative, customer (i.e., end 

user) perspectives and insights can be solicited through various 

methods—user acceptance testing, interviews, complaint programs, and 

satisfaction surveys—to validate or raise questions about the project’s 

implementation.

Further, leading practices emphasize that periodic customer satisfaction 

data be proactively used to improve performance and demonstrate the 

level of satisfaction the project is delivering. The Office of Management 

and Budget has developed standards and guidelines in survey research 

that are generally consistent with best practices40 and call for statistically 

39GAO, Information Technology Investment Management: A Framework for Assessing 

and Improving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004); and 

Executive Office of the President, Office of Management and Budget, Evaluating 

Information Technology Investments, A Practical Guide (November 1995).

40For example, the American Association of Public Opinion Research (AAPOR) best 

practices (http://www.aapor.org/best_practices1.htm), describe the manner in which to 

produce a quality survey when a need for information arises for which existing data 

appear to be insufficient. AAPOR describes features when selecting samples that 

represent the population to be studied, such as random selection. 

VBA’s Methods for 

Determining User’s 

Feedback on VBMS 

Have Not Included 

Establishing Goals or 

Conducting a Survey; 

GAO’s Survey 

Identified Varied 

Satisfaction Levels

VBA Has Several Methods 

to Obtain Users’ Feedback 

on VBMS, but Has Not 

Established User 

Satisfaction Goals or 

Conducted a Survey

Page 32 GAO-15-582 Veterans Benefits Management System

 
valid data collection efforts to be used in fulfilling agencies’ customer 

service data collection.41 These leading practices also stress the 

importance of centrally integrating all customer feedback data in order to 

have more complete diagnostic information to guide improvement 

efforts.42

VA has used a variety of methods for obtaining customer or end user 

feedback on the performance of VBMS. For example, the department 

solicits end user involvement and feedback in the iterative system 

development process based on user acceptance criteria.43 According to 

the Senior Project Manager for VBMS Development within OI&T, at the 

end of each development cycle and before a new version of VBMS is 

deployed, end users are involved in user acceptance testing and a final 

customer acceptance meeting.44

VA also provides training to a subset of end users—known as 

“superusers”—on the updated functionality introduced in a new version of 

VBMS. These superusers are then expected to train the remaining users 

in the field on the new version’s features. The department tracks the 

overall satisfaction level with training received after each VBMS major 

41OMB, Standards and Guidelines for Statistical Surveys (September 2006). In part, this 

guidance directs that agency survey designs use generally accepted statistical methods, 

such as probabilistic methods that can provide estimates of sampling error. Any use of 

nonprobability sampling methods must be justified statistically and be able to measure 

estimation error. According to the OMB standards, the size and design of the sample must 

reflect the level of detail needed in tabulations and other data products, and the precision 

required of key estimates.

42GAO, Information Technology Investment Management: A Framework for Assessing 

and Improving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004);

Carnegie Mellon Software Engineering Institute, Capability Maturity Model® Integration 

(CMMI) for Development, Version 1.3 (Pittsburgh, Pa.: November 2010); M.S. Garver and 

R.L. Cook, “Best Practice Customer Value and Satisfaction Cultures,” Mid-American 

Journal of Business, vol. 16, no. 1 (2001); M.S. Garver, “Modeling Best Practices for 

Government Agencies: Implementing Customer Satisfaction Programs” (Jan. 28, 2002); 

Best Practices, LLC, “Achieving World-Class Customer Service: An Integrated Approach” 

(copyright 1998-2001); Federal Benchmarking Consortium, “Serving the American Public: 

Best Practices in Customer-Driven Strategic Planning” (February 1997); and OMB, 

Evaluating Information Technology Investments, A Practical Guide (November 1995).

43User acceptance criteria are criteria that a deliverable must satisfy to be accepted by a 

user, customer, or other authorized entity.

44User acceptance testing is formal testing conducted to enable a user, customer, or other

authorized entity to determine whether to accept a deliverable.

Page 33 GAO-15-582 Veterans Benefits Management System

 
release. However, this tracking is limited to superusers’ satisfaction with 

the training, rather than their satisfaction with the system.45

VA also solicits customer feedback about the system through interviews. 

For example, the PMO Director stated that the Under Secretary for 

Benefits hosts a weekly phone call with bargaining unit employees as a 

“pulse check” on VBA transformation activities, including VBMS. 

According to this official, the VBA Office of Field Operations also offers an 

instant messaging chat service to all regional office employees to solicit 

feedback about the latest VBMS functionality deployment.

Another method in which the department obtains customer input is 

through a formal feedback process. For example, according to the PMO 

Director, VA provides national service desk support to assist users in 

troubleshooting system issues and identifying system defects. In addition, 

VBMS applications include a built-in feature that enables users to provide 

feedback to the PMO on problems with the system. According to the 

Director, the feedback received by the office also helps to identify user 

training issues.

Nevertheless, while VA has taken steps to obtain feedback on the 

performance and implementation of VBMS, it has not established goals to 

define user satisfaction that can be used as a basis for gauging the 

success of its efforts to promote satisfaction with the system. Further, 

although the efforts that have been taken to solicit users’ feedback 

provide VBA with useful insights about particular problems, data are not 

centrally compiled or sufficient for supporting overall conclusions about 

whether customers are satisfied. In addition, VBA has not employed a 

customer satisfaction survey of claims processing employees who use 

the system on a daily basis to process disability claims. Such a survey 

could provide a more comprehensive picture of overall customer 

satisfaction and help identify areas where VBMS development and 

implementation efforts might need additional attention.

According to the PMO Director, VBA had not used a survey to solicit 

feedback because of concern that such a mechanism may have 

negatively impacted the efficiency of claims processors in completing 

45As of March 31, 2015, VA reported it had exceeded its target goal of 90 percent with a 

94 percent satisfaction rate with VBMS superuser training. 

Page 34 GAO-15-582 Veterans Benefits Management System

 
disability compensation claims on behalf of veterans. Further, the director 

believed that the office had the benefit of receiving ongoing end user 

input on VBMS by virtue of the intensive testing cycles, as well as several 

of the other mechanisms noted previously by which end users have 

provided ongoing feedback. Nevertheless, without establishing user 

satisfaction goals and collecting the comprehensive data that a 

statistically valid survey can provide, the PMO limits its ability to obtain a 

comprehensive understanding of VBMS users’ levels of satisfaction with 

the system. Thus, VBA could miss opportunities to improve the efficiency 

of its claims process by increasing satisfaction with VBMS.

In response to the statistical survey that we administered, a majority of 

VBMS users (i.e., VBA claims processors) were satisfied with the system 

that had been implemented at the time of the survey.46 These users 

(which represent claims assistants, veteran service representatives, 

supervisory veteran service representatives, rating veterans service 

representatives, decision review officers, and others)47 were satisfied with 

the three modules of VBMS.48 Specifically, an estimated 59 percent of the 

population of claims processors were satisfied with VBMS-Core; an 

46We received a response rate of 60 percent. We adjusted for characteristics that were 

associated with survey response propensity using standard weighting class adjustments 

defined by sampling strata. We assumed that nonresponse adjusted data are missing at 

random and therefore conclude the respondent analyses using the nonresponse adjusted 

weights are unbiased for the population of VBMS users sampled in the survey and the 

responses to be generalizable to all VBA claims processors at 56 VA regional offices. 

Confidence intervals for estimates we report from this survey are based on a confidence 

level of 95 percent and are calculated using methods appropriate for a stratified random 

sample. Confidence intervals are never wider than plus or minus 5 percentage points. At a 

95 percent confidence level, this means that in about 95 out of 100 instances, the 

sampling procedures we used would be expected to produce a confidence interval 

containing the true population value we estimate.

47Some survey respondents identified themselves as “other” when selecting their role 

(e.g., Rating Quality Review Specialist). 

48Survey respondents were asked to rate their VBMS experience with various system 

usability statements and were given the option to select the following answer choices: 

“strongly agree,” “agree,” “neither agree nor disagree,” “disagree,” “strongly disagree,” and 

“not applicable or no basis to judge.” We defined satisfaction as a combination of the 

“strongly agree” and “agree” responses, and exclude those respondents who selected “not 

applicable or no basis to judge” for analysis of satisfaction, within the main report. For a 

more detailed explanation of our survey methodology, see app. I. For the full survey 

instrument, including estimates for those who selected “not applicable or no basis to 

judge,” see app. II.

A Majority of VBMS Users, 

with the Exception of 

Decision Review Officers, 

Reported Satisfaction

with the System

Page 35 GAO-15-582 Veterans Benefits Management System

 
estimated 63 percent were satisfied with the Rating module, and an 

estimated 67 percent were satisfied with the Awards module. Figure 4 

depicts the estimated percentage of claims processors who were satisfied 

with VBMS.

Figure 4: Estimated Percentage of All Roles of Claims Processors Who Were 

Overall Satisfied with VBMS, by Module

Although a majority of users were satisfied with the three modules, 

decision review officers were considerably less satisfied than other users 

with VBMS-Core and VBMS-Rating.49 Specifically, for VBMS-Core, an 

estimated 27 percent of decision review officers were satisfied compared 

to an estimated 59 percent of all roles of claims processors (including 

decision review officers) who were satisfied. In addition, for VBMS-Rating, 

an estimated 38 percent of decision review officers were satisfied, 

compared to an estimated 63 percent of all roles of claims processors. 

49Decision review officers do not typically use VBMS-Awards. Therefore, decision review 

officers were not compared to other users for that module. 

Page 36 GAO-15-582 Veterans Benefits Management System

 
Figure 5 depicts the estimated satisfaction levels of decision review 

officers in comparison to all roles.

Figure 5: Estimated Percentage of All Roles of Claims Processors Compared to 

Decision Review Officers Who Were Satisfied with VBMS, by Module

Decision review officers were considerably less satisfied with VBMS in 

comparison to all roles of claims processors in additional areas. For 

example, an estimated 26 percent of decision review officers viewed 

VBMS-Core as an improvement over the previous legacy system or 

systems for establishing claims and storing and reviewing electronic 

documents related to a claim in an eFolder. In contrast, an estimated 58 

percent of all users (including decision review officers) viewed the Core 

module as an improvement. In addition, an estimated 26 percent of 

decision review officers viewed VBMS-Rating as an improvement over 

the previous systems with respect to providing Web-accessible tools, 

including rules-based rating calculators, to assist in making claims rating 

decisions. In contrast, an estimated 55 percent of all roles of claims 

processors viewed the Rating module as an improvement. For VBMSPage 37 GAO-15-582 Veterans Benefits Management System
Awards, an estimated 61 percent of all roles viewed this module as an 

improvement over the previous systems to automate the award and 

notification process. Figure 6 depicts the estimated percentage of 

decision review officers, in comparison to all claims processors, who 

viewed VBMS as an improvement over legacy systems.

Figure 6: Estimated Percentage of All Roles of Claims Processors Compared to 

Decision Review Officers Who Viewed VBMS as an Improvement over the Previous 

System(s), by Module

Similarly, for the three modules, a majority of users (including decision 

review officers) would have chosen VBMS over the legacy system or 

systems. However, decision review officers indicated that they were less 

likely to have chosen VBMS-Core and VBMS-Rating over legacy 

systems. Specifically, an estimated 27 percent of decision review officers 

would have chosen VBMS-Core compared to an estimated 60 percent of 

all roles of claims processors. In addition, an estimated 27 percent of 

decision review officers would have chosen VBMS-Rating compared to 

61 percent of all roles who would have chosen the system over the legacy 

Page 38 GAO-15-582 Veterans Benefits Management System
system or systems. For VBMS-Awards, an estimated 67 percent of all 

roles would have chosen this module over the previous systems. Figure 7 

depicts the estimated percentage of decision review officers, in 

comparison to all claims processors, who would have chosen VBMS 

instead of the legacy systems to process claims.

Figure 7: Estimated Percentage of All Roles of Claims Processors Compared to 

Decision Review Officers Who Would Have Chosen VBMS over the Previous 

System(s) to Process Claims, by Module

Decision review officers perform an array of duties to resolve claims 

issues raised by veterans and their representatives. They may also 

conduct a new review or complete a review of a claim without deference 

to the original decision and in doing so, must click through all documents 

included in the e-Folder. Survey comments from decision review officers 

stated, for example, that reviews in the VBMS paperless environment 

take longer because of the length of time spent loading, scrolling, and 

viewing each document (particularly if the documents are large, such as a 

service medical record file). Additionally, multiple decision review officers 

Page 39 GAO-15-582 Veterans Benefits Management System
commented that it is easier and faster to review documents in a paper file. 

Although such comments provide illustrative examples of individual 

decision review officer’s views and are not representative, according to 

the PMO Director, decision review officers’ relative dissatisfaction is not 

surprising because the system does not yet include functionality that 

supports their work, which primarily relates to appeals processing.

Even though VA has made progress toward completing the development 

and implementation of VBMS, there is more work to be done and VBA 

can improve management of its ongoing efforts. While 95 percent of 

records related to veterans’ disability claims are electronic and reside in 

VBMS, additional capabilities to fully process disability claims will be 

delayed beyond fiscal year 2015, which is when completion was originally 

planned. Further, VA’s incremental approach to developing and 

implementing VBMS has not yet produced a plan that identifies when the 

system will be completed and can be expected to fully support disability

compensation and pension claims processing and appeals. Thus, it will 

be difficult for VA to hold its managers accountable for meeting its time

frame and for demonstrating progress.

VBA’s management of the system highlights areas that could benefit from 

improvement as development and implementation of the system continue. 

Specifically, without a reliable estimate of the total costs associated with 

completing work on VBMS, the department’s stakeholders have only a 

limited view of future resource needs, and the program risks not having 

sufficient funding to complete development and implementation of the 

system. Additionally, established goals for system response times would 

provide users with an expectation of the response times they can 

anticipate, and management with an indication of how well the system is 

performing relative to performance goals. Furthermore, continuing to 

deploy system releases with large numbers of defects that reduce system 

functionality could adversely affect users’ ability to process disability 

claims in an efficient manner.

Without user satisfaction goals and the data a customer satisfaction 

survey could yield, VA may miss opportunities to collect important data on 

how users view the system’s performance, and ultimately, to improve the 

system. Our survey of VBMS users found that a majority of them were 

satisfied with the system, but decision review officers were considerably 

less satisfied. Although the results of our survey provide VBA with useful 

data about users’ satisfaction with VBMS, the absence of user 

satisfaction goals limits the utility of survey results. Specifically, without 

Conclusions

Page 40 GAO-15-582 Veterans Benefits Management System
having established goals to define user satisfaction, VBA does not have a 

basis for gauging the success of its efforts to promote satisfaction with the 

system or for identifying areas where its efforts to complete development 

and implementation of the system might need attention.

We recommend that the Secretary of Veterans Affairs direct the Under 

Secretary for Benefits and the Chief Information Officer to take the 

following five actions to improve VA’s efforts to effectively complete the 

development and implementation of VBMS:

• Develop an updated plan for VBMS that includes (1) a schedule for 

when VBA intends to complete development and implementation of 

the system, including capabilities that fully support disability claims, 

pension claims, and appeals processing and (2) the estimated cost to 

complete development and implementation of the system.

• Establish goals for system response time and use the goals as a 

basis for periodically reporting actual system performance.

• Reduce the incidence of high- and medium-priority level defects that 

are present at the time of future VBMS releases.

• Develop and administer a statistically valid survey of VBMS users to 

determine the effectiveness of steps taken to make improvements in 

users’ satisfaction.

• Establish goals that define customer satisfaction with VBMS and 

report on actual performance toward achieving the goals based on the 

results of GAO’s survey of VBMS users and any future surveys VA 

conducts.

We received written comments on a draft of this report (reprinted in 

appendix III). In addition, VA provided technical comments, which we 

incorporated, as appropriate. 

In its comments, VA generally agreed with our conclusions. The 

department also concurred with our recommendations and described 

actions it is planning to take in response to four of our five 

recommendations. Specifically, VA concurred with our recommendation 

calling for an updated plan for VBMS. Although it described the 

importance of VBMS to ensuring timely delivery of benefits to veterans 

and recognized the need for continued investment in the system, the 

department did not, however, identify actions to develop a VBMS plan 

Recommendations for 

Executive Action

Agency Comments 

and Our Evaluation

Page 41 GAO-15-582 Veterans Benefits Management System
that includes a schedule for when VBA intends to complete development 

and implementation of the system and the estimated cost of doing so. We 

believe development of such a plan is an important action to help ensure 

effective development and implementation of VBMS and to hold 

managers accountable. 

With regard to the remaining four recommendations, VA described 

actions it is planning to take in response to each. For example, with 

regard to our recommendation to establish goals for system response 

time, the department stated that the VBMS program is participating in a 

review of service-level agreements to establish metrics for the system’s

performance. Additionally, regarding our recommendation that it reduce 

the incidence of high and medium priority level defects that are present in 

future VBMS releases, the department reiterated its plans and procedures 

for decreasing the defects in each release. With respect to our 

recommendation to conduct a survey of VBMS users, the department 

stated that the VBMS PMO is working with the Office of Field Operations 

and labor partners regarding the distribution of a survey to measure 

users’ satisfaction and expects to release a survey in March 2016. 

Consistent with our final recommendation, the department is also 

planning to establish customer satisfaction goals and report on actual 

performance toward achieving the goals after all survey results are 

received and analyzed with a target of July 2016 for completing this 

action.

If VA develops the updated plan, including schedule and cost, for VBMS 

as we recommended and follows through on the actions it described in 

response to our remaining recommendations, the department will be 

better positioned to effectively complete the development and 

implementation of VBMS and to more effectively provide benefits and 

services to our nation’s veterans. 

As agreed with your office, unless you publicly announce the contents of 

this report earlier, we plan no further distribution until 30 days from the 

report date. At that time, we will send copies of this report to appropriate 

congressional committees, the Secretary of Veterans Affairs, and other 

interested parties. In addition, the report is available at no charge on the 

GAO website at http://www.gao.gov.

Page 42 GAO-15-582 Veterans Benefits Management System
If you or your staff have questions about this report, please contact me at 

(202) 512-6304 or melvinv@gao.gov. Contact points for our Offices of 

Congressional Relations and Public Affairs may be found on the last page 

of this report. Key contributors to this report are listed in appendix IV.

Sincerely yours,

Valerie C. Melvin

Director, Information Management

and Technology Resources Issues

Page 43 GAO-15-582 Veterans Benefits Management System
Appendix I: Objectives, Scope, and 

Methodology

The objectives of this study were to (1) assess the Department of 

Veterans Affairs’ (VA) progress toward developing and implementing the 

Veterans Benefits Management System (VBMS) and (2) determine to 

what extent users report satisfaction with the system.

To assess the department’s progress toward developing and 

implementing VBMS, we obtained and reviewed program plans and other 

documentation that articulated the system’s goals and expected benefits. 

These included the VA Strategic Plan to Eliminate the Compensation 

Claims Backlog (2013), VBMS Strategic Roadmap, VBMS Tactical 

Roadmap, VBMS Operating Plans, VBA’s 2011 life-cycle cost estimate 

for the program, documentation supporting VBMS’s progression through 

VA’s milestone review process, VA’s annual budget submissions, and 

contracts and related contractor documentation. We compared program 

plans and other documentation articulating the system’s goals, expected 

benefits, and the system functionality expected to be delivered with 

documents showing VBMS’s progress, such as monthly program 

management reviews and cost and performance reports. We also 

compared program plans to VA’s policies for incremental system 

development,1 as well as federal guidance and IT project management 

principles on program planning and cost estimating.2

In addition, we obtained reports documenting the system’s availability and 

performance and analyzed information in the reports to determine trends 

in system availability and response time. We compared system 

performance data with GAO and federal IT guidance for defining program 

goals and related performance targets that can be used to assess 

progress in achieving the goals.3 We assessed the reliability of the data 

by reviewing it for obvious errors and missing data; corroborating the data 

1

U.S. Department of Veterans Affairs, Office of Information and Technology, Project 

Management Accountability System (PMAS) Guide 5.0 (June 2014).

2

See, for example, OMB, Management of Federal Information Resources, Circular No. A130 (Washington, D.C.: November 28, 2000); GAO, Cost Estimating and Assessment 

Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP

(Washington, D.C.: March 2009); GAO, Schedule Assessment Guide: Best Practices for 

Project Schedules, GAO-12-120G (Washington, D.C.: May 2012). 

3

GAO, Information Technology Investment Management: A Framework for Assessing and 

Improving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004); Executive 

Office of the President, Office of Management and Budget, Evaluating Information 

Technology Investments, A Practical Guide (November 1995).

Appendix I: Objectives, Scope, and 

Methodology

Page 44 GAO-15-582 Veterans Benefits Management System

 
Appendix I: Objectives, Scope, and 

Methodology

with related documentation; and interviewing responsible officials about 

their use of an automated tool that monitors system performance. We 

determined the data to be sufficiently reliable for the purposes of this 

report.

We reviewed the VBMS Defect Management Plan and compared the plan 

with key principles of sound defect management.4 We also compared the 

processes described in the plan with actions taken to manage defects 

identified for VBMS. We conducted analysis of critical defects identified 

for VBMS releases 7.0, 7.1, 8.0, and 8.1 to determine whether they 

remained open at the time of system release.

Further, we reviewed the department’s methods for soliciting end user 

feedback on the performance of the system (e.g., VBMS training strategy, 

operational management reviews, cost and performance reports, and end 

of month reports). We compared these methods to leading practices for 

obtaining customer feedback and collecting customer service data to 

4

The Project Management Institute, Inc., A Guide to the Project Management Body of 

Knowledge (PMBOK® Guide), Fifth Edition, (Newtown Square, Pa.: 2013); Institute of 

Electrical and Electronics Engineers, Software and systems engineering — Software 

testing, ISO/TEC/IEEE Std 29119 (New York, N.Y.: Sept. 1, 2013); Institute of Electrical 

and Electronics Engineers, IEEE Standard for Software and System Test Documentation, 

IEEE Std 829-2008 (New York, N.Y.: July 10, 2008); Institute of Electrical and Electronics 

Engineers, IEEE Standard Classification for Software Anomalies, IEEE Std 1044-2009 

(New York, N.Y.: Jan. 7, 2010); Software Engineering Institute, CMMI® for Acquisition, 

Version 1.3 (Pittsburgh, Pa: November 2010); Software Engineering Institute, CMMI® for

Development, Version 1.3 (Pittsburgh, Pa: November 2010); GAO, Year 2000 Computing 

Crisis: A Testing Guide, GAO/AIMD-10.1.21 (Washington, D.C.: November 1998); GAO 

Cost Estimating and Assessment Guide: Best Practices for Developing and Managing 

Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009); and GAO 

Schedule Assessment Guide: Best Practices for Project Schedules—Exposure Draft, 

GAO-12-120G (Washington, D.C.: May 2012).

Page 45 GAO-15-582 Veterans Benefits Management System

 
Appendix I: Objectives, Scope, and 

Methodology

improve performance and demonstrate customer satisfaction of IT 

projects.5

To determine the extent that users report satisfaction with the system, we 

conducted a Web-based survey of a nationally representative stratified 

random sample of disability compensation claims processors. The survey 

was administered during the time frame of September through November 

2014. We developed survey questions with input from officials in the 

VBMS PMO and VA’s Office of Information and Technology.

We pretested versions of the draft survey and observed claims 

processors to gain initial insight regarding their use of the system in 

VBA’s Baltimore, Maryland, and Philadelphia, Pennsylvania, regional 

offices. We selected these offices based on their large size and location. 

We revised the draft survey based on comments received during the 

pretests. Once finalized, the claims processors included in the sample 

were sent an e-mail that asked them to complete the survey, which was 

available to them over the Web from September 30, 2014, to November 

19, 2014. Following the initial request, we e-mailed weekly follow-up 

requests to any nonrespondents.

Our eligible population for this survey consisted of all 10,622 disability 

compensation claims processors employed by the agency as of July 31, 

2014. To determine whether VBMS experiences differed based on 

position and office workload, we designed a stratified random sample of 

claims processors, with strata defined by position and office workload. 

VBA provided us lists of these claims processors with information that 

indicated their position and office. We determined average office 

workload based on information published in VBA’s public weekly 

workload reports. We divided the list of claims processors into 10 strata 

based on their position at both smaller and larger workload regional 

5

GAO, Information Technology Investment Management: A Framework for Assessing and 

Improving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004); Carnegie 

Mellon Software Engineering Institute, Capability Maturity Model® Integration (CMMI) for 

Development, Version 1.3 (Pittsburgh, Pa.: November 2010); M.S. Garver and R.L. Cook, 

“Best Practice Customer Value and Satisfaction Cultures,” Mid-American Journal of 

Business, vol. 16, no. 1 (2001); M.S. Garver, “Modeling Best Practices for Government 

Agencies: Implementing Customer Satisfaction Programs” (Jan. 28, 2002); Best Practices, 

LLC, “Achieving World-Class Customer Service: An Integrated Approach” (copyright 1998-

2001); Federal Benchmarking Consortium, “Serving the American Public: Best Practices 

in Customer-Driven Strategic Planning” (February 1997); and Office of Management and 

Budget (OMB), Standards and Guidelines for Statistical Surveys (September 2006).

Page 46 GAO-15-582 Veterans Benefits Management System

 
Appendix I: Objectives, Scope, and 

Methodology

offices.6 Table 4 shows the size of each eligible population, the size of the 

sample drawn from each eligible population, and the number of claims 

processors in each stratum that responded to the survey.

Table 4: Eligible Population, Sample, and Respondents, by Stratum

Stratum

Claims

processor title

Average 

office 

workload

per user

Population 

size

Sample

size

Number of 

respondents

1 Claims assistants <136 548 317 179

2 Veteran service 

representatives 

<136 3,152 484 290

3 Supervisory 

veteran service 

representatives 

<136 543 322 175

4 Rating veterans 

service 

representatives 

<136 1,824 451 296

5 Decision review 

officers 

<136 417 288 208

6 Claims assistants >=136 339 255 123

7 Veteran service 

representatives

>=136 2,134 459 258

8 Supervisory 

veteran service 

representatives

>=136 366 268 132

9 Rating veterans 

service 

representatives

>=136 1,013 396 253

10 Decision review 

officers

>=136 286 235 184

Total 10,622 3,475 2,098

Source: Analysis of GAO survey data and information provided from VBA. I GAO-15-582

6

Average office workload is defined as the number of completed claims by fiscal year to 

date from October 1, 2013, to July 28, 2014, plus the number of pending claims as of July 

28, 2014, divided by the number of users in a regional office summed over all the user 

roles as of July 31, 2014. This is meant to be a rough proxy of workload in order to assure 

that VBMS users from both smaller and larger workload offices were included in the 

sample. Smaller average office workload is less than 136 and larger average office 

workload is greater than or equal to 136, which is equal to the mean/median.

Page 47 GAO-15-582 Veterans Benefits Management System

 
Appendix I: Objectives, Scope, and 

Methodology

We drew an independent random sample from each of these strata to 

enable us to project survey results to all VBA claims processors, in 

general, and to claims processors in each of the positions listed above, 

within small and large workload offices.7 Of our total sample of 3,475 

claims processors, 2,098 responded to the survey for an overall response 

rate of 60 percent.

To produce estimates regarding the experiences and views of claims 

processors from the survey responses of those included in our sample, 

we analyzed the data with methods that are appropriate for a stratified 

random sample using analysis weights. We weighted each response from 

claims processors in each stratum to statistically account for all members 

of that stratum. Because estimates are based on responses from a 

sample, each estimate we report has a measurable precision or sampling 

error. The sampling error or margin of error surrounding an estimate is 

expressed as a number of percentage points higher or lower than that 

estimate and the entire range that the sampling error covers is referred to 

as a confidence interval. Confidence intervals are calculated based on a 

certain confidence level. Confidence intervals for estimates we report 

from this survey are based on a confidence level of 95 percent and are 

calculated using methods appropriate for a stratified random sample. 

Confidence intervals for percentage estimates in this report are never 

wider than plus or minus 5 percentage points. At a 95 percent confidence 

level, this means that in about 95 out of 100 instances, the sampling 

procedures we used would be expected to produce a confidence interval 

(in this case, a 10 percentage point range) containing the true population 

value we estimate.

In addition to sampling error, estimates based on survey results are 

subject to what is referred to as nonsampling error that can result from, 

among other things, poorly designed survey questions and mistakes in 

data entry or analysis or nonresponse. We took a number of steps in 

developing the survey and in entering and analyzing the data to minimize 

nonsampling error. For example, a social science survey specialist 

collaborated with our subject matter experts in designing the survey. In 

addition, it was reviewed by VBA officials and, as previously noted, was 

pretested with a number of different types of claims processors in two 

7

We over sampled from each stratum to take into account sampled claims processors who 

may not respond to the survey in order to assure a sufficient number of respondents for 

the designed 95 percent level of confidence.

Page 48 GAO-15-582 Veterans Benefits Management System

 
Appendix I: Objectives, Scope, and 

Methodology

locations. Also, when we analyzed the data, an independent analyst 

verified all computer programs. Because this was a Web-based survey, 

respondents’ answers to survey questions were automatically entered 

into an electronic file, eliminating the need to separately key responses 

into a data file, further minimizing the potential for nonsampling error.

We also took steps to mitigate potential nonresponse error. For example, 

we used follow-up e-mails to remind users to complete the survey in order 

to reduce nonresponse. Further, we adjusted for characteristics that were 

associated with survey response propensity using standard weighting 

class adjustments defined by sampling strata. We assumed that 

nonresponse adjusted data were missing at random, and therefore 

concluded the respondent analyses using the nonresponse adjusted 

weights were unbiased for the population of VBMS users sampled in the 

survey.

We supplemented our analyses with interviews of VA officials that had 

knowledge of the VBMS program, including officials in VA’s Office of 

Information and Technology and the Veterans Benefits Administration’s

VBMS PMO and Office of Field Operations.

We conducted this performance audit from March 2014 through August

2015 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives.

Page 49 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

The questions we asked in our survey of Veterans Benefits Administration 

(VBA) claims processors are shown here. Our survey was comprised of 

closed- and open-ended questions. In this appendix, we include all survey 

questions and aggregate results of responses to the closed-ended 

questions. For a more detailed discussion of our survey methodology, see 

appendix I.

Section 1—Background Information

1. What is your current role in processing veterans’ disability 

compensation claims?

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound 

95 percent 

confidence

interval -

upper bound

Claims Assistant 8.2 7.8 8.5

Veterans Service Representative 42.7 41.2 44.2

Supervisory Veterans Service 

Representative

9.1 8.0 10.2

Rating Veterans Service 

Representative

26.8 26.3 27.2

Decision Review Officer 5.0 4.8 5.2

Other 8.3 7.1 9.4

If other, describe your role. (Open-ended response.)

2. What responsibility (or responsibilities) do you have in this role 

(Check all that apply.)

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Claims intake 14.5 13.0 15.9

Establishment 29.2 27.1 31.2

Development 53.9 52.0 55.8

Rating 34.3 33.6 35.0

Award 32.7 30.7 34.7

Authorization 19.9 18.0 21.8

Other 19.2 17.6 20.7

If other, describe your responsibility. (Open-ended response.)

Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Page 50 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

3. How many years have you worked within the VBA?

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

1 year or less 7.8 6.4 9.2

More than 1 year to less than 5 

years

24.4 22.4 26.5

5 years to less than 10 years 38.1 35.9 40.4

10 years or more 29.6 27.7 31.5

4. The Department of Veterans Affairs’ (VA) records show that you 

currently work at the (office name prepopulated) office. Is this 

correct?

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Yes 99.5 99.1 99.8

No 0.5 0.2 0.9

If no, enter the name of the office where you currently work. (If you work 

in more than one office, enter the office where you spend most of your 

time.) (Open-ended response.)

5. How long have you been using the Veterans Benefits 

Management System (VBMS)?

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Less than 6 months 8.4 6.9 9.8

6 months to less than 

1 year

18.4 16.5 20.2

1 year to less than 

2 years

50.9 48.6 53.2

2 years to less than 

3 years

18.1 16.4 19.7

3 years or more 3.6 2.8 4.5

Not applicable, I do not use 

VBMS (Skip to question 17.)

0.8 0.4 1.3

Page 51 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

6. In an average work week, how much time, if at all, do you spend 

processing the following types of claims in VBMS? (Choose a 

time period for each item listed.)

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

6a. Fully electronic 

claims

Less than 8 

hours

9.3 8.0 10.6

8 to less than 16 

hours

8.0 6.8 9.3

16 to less than 

24 hours

7.2 6.0 8.3

24 to less than 

32 hours

10.8 9.3 12.2

32 to less than 

40 hours

28.9 26.7 31.0

All week (40 or 

more hours)

27.8 25.7 30.0

I do not process 

these

8.1 7.1 9.2

6b. Fully paperbased claimsLess than 8 

hours

54.5 52.1 56.8

8 to less than 16 

hours

12.1 10.6 13.6

16 to less than 

24 hours

5.6 4.6 6.7

24 to less than 

32 hours

4.0 3.1 5.0

32 to less than 

40 hours

2.1 1.5 2.9

All week (40 or 

more hours)

4.1 3.2 5.1

I do not process 

these

17.7 16.0 19.4

6c. Hybrid claims 

(that is, claims 

that consist of both 

paper and 

electronic records)

Less than 8 

hours

49.4 47.0 51.7

8 to less than 16 

hours

15.6 13.8 17.3

16 to less than 

24 hours

7.4 6.1 8.7

Page 52 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

24 to less than 

32 hours

3.8 2.9 4.9

32 to less than 

40 hours

2.9 2.1 3.8

All week (40 or 

more hours)

5.3 4.3 6.4

I do not process 

these

15.7 14.1 17.2

7. In addition to using VBMS, how much do you depend on each of 

the following systems or applications to process disability 

claims? (Choose a category for each system or application 

listed.)

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

7a. Veterans 

Service Network 

(VETSNET): Award 

Processing

None or do not 

use

51.0 49.0 52.9

A little 23.4 21.3 25.6

A moderate 

amount

12.8 11.0 14.5

A great deal 12.8 11.0 14.6

7b. VETSNET: 

Modern Award 

ProcessingDevelopment 

(MAP-D)

None or do not 

use

6.9 5.9 8.0

A little 28.6 26.5 30.6

A moderate 

amount

27.6 25.5 29.7

A great deal 37.0 34.7 39.2

7c. VETSNET: 

Rating Board 

Automation 2000 

(RBA 2000)

None or do not 

use

79.8 78.1 81.4

A little 13.5 12.1 14.9

A moderate 

amount

3.7 2.9 4.6

Page 53 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

A great deal 3.1 2.3 3.9

7d. VETSNET: 

Share

None or do not 

use

4.7 3.8 5.7

A little 20.4 18.9 21.8

A moderate 

amount

22.7 20.8 24.7

A great deal 52.3 50.1 54.4

Section 2—Veterans Benefits Management System-Core 

(VBMS-Core)

8. In the course of your work, do you use VBMS-Core?

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

Yes 75.7 73.8 77.6

No 24.3 22.4 26.2

Complete questions 8.1 and 8.2 for VBMS-Core. In question 8.3, you 

will have a chance to comment on any of your answers after 

responding to the questions.

8.1. Do you agree or disagree with the following statements based 

on your current experience using VBMS-Core? Select one 

answer in each row.

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

8.1a. VBMS-Core 

meets my needs

Strongly 

agree

9.9 8.3 11.6

Agree 52.9 50.2 55.6

Neither 

agree nor 

disagree

16.2 14.2 18.2

Disagree 15.5 13.5 17.5

Page 54 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Strongly 

disagree

4.8 3.7 6.1

Not 

applicable or 

no basis to 

judge

0.7 0.3 1.4

8.1b. VBMS-Core 

does everything I 

would expect it to do

Strongly 

agree

5.1 3.9 6.4

Agree 26.5 24.1 28.9

Neither 

agree nor 

disagree

18.2 16.2 20.3

Disagree 36.6 33.9 39.2

Strongly 

disagree

12.4 10.5 14.2

Not 

applicable or 

no basis to 

judge

1.3 0.7 2.0

8.1c. VBMS-Core 

helps me be more 

productive 

compared to the 

previous system(s)

Strongly 

agree

10.2 8.5 11.9

Agree 33.8 31.2 36.4

Neither 

agree nor 

disagree

24.1 21.7 26.4

Disagree 19.4 17.2 21.5

Strongly 

disagree

9.8 8.2 11.4

Not 

applicable or 

no basis to 

judge

2.8 2.0 3.9

8.1d. VBMS-Core 

helps me be more 

efficient compared 

to the previous 

system(s)

Strongly 

agree

11.6 9.8 13.4

Agree 35.2 32.6 37.8

Page 55 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Neither 

agree nor 

disagree

21.5 19.3 23.6

Disagree 19.4 17.2 21.5

Strongly 

disagree

9.8 8.2 11.4

Not 

applicable or 

no basis to 

judge

2.6 1.7 3.7

8.1e. I feel 

comfortable using 

VBMS-Core

Strongly 

agree

24.7 22.3 27.1

Agree 52.9 50.2 55.6

Neither 

agree nor 

disagree

13.8 12.0 15.7

Disagree 5.1 4.0 6.3

Strongly 

disagree

2.9 2.1 4.0

Not 

applicable or 

no basis to 

judge

0.5 0.2 1.2

8.1f. VBMS-Core is 

easier to use 

compared to the 

previous system(s)

Strongly 

agree

14.0 12.1 16.0

Agree 29.2 26.7 31.7

Neither 

agree nor 

disagree

28.3 25.8 30.7

Disagree 18.3 16.2 20.4

Strongly 

disagree

8.0 6.6 9.4

Not 

applicable or 

no basis to 

judge

2.2 1.4 3.2

8.1g. I can use 

VBMS-Core 

successfully every 

time 

Strongly 

agree

8.9 7.4 10.4

Page 56 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Agree 29.2 26.7 31.6

Neither 

agree nor 

disagree

19.6 17.5 21.7

Disagree 33.0 30.4 35.6

Strongly 

disagree

8.9 7.4 10.5

Not 

applicable or 

no basis to 

judge

0.4 0.1 1.0

8.1h. Compared to 

the previous 

system(s), VBMSCore requires fewer 

steps to accomplish 

what I need to do

Strongly 

agree

9.7 8.0 11.3

Agree 26.9 24.5 29.4

Neither 

agree nor 

disagree

23.4 21.0 25.7

Disagree 25.3 22.9 27.6

Strongly 

disagree

12.2 10.5 13.9

Not 

applicable or 

no basis to 

judge

2.6 1.8 3.7

8.1i. I have been 

provided with 

effective training on 

how to use VBMSCore

Strongly 

agree

10.4 8.7 12.1

Agree 44.9 42.2 47.6

Neither 

agree nor 

disagree

18.4 16.3 20.5

Disagree 18.0 15.9 20.1

Strongly 

disagree

7.8 6.3 9.3

Page 57 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Not 

applicable or 

no basis to 

judge

0.4 0.1 1.0

8.1j. I learned to use 

VBMS-Core quickly

Strongly 

agree

14.9 13.0 16.9

Agree 48.7 46.0 51.4

Neither 

agree nor 

disagree

18.7 16.6 20.8

Disagree 14.0 12.1 15.9

Strongly 

disagree

3.3 2.4 4.4

Not 

applicable or 

no basis to 

judge

0.4 0.1 1.0

8.1k. I easily 

remember how to 

use VBMS-Core

Strongly 

agree

16.9 14.8 19.1

Agree 56.2 53.4 58.9

Neither 

agree nor 

disagree

16.9 14.8 18.9

Disagree 7.8 6.4 9.2

Strongly 

disagree

1.8 1.1 2.7

Not 

applicable or 

no basis to 

judge

0.5 0.1 1.1

8.1l. VA provides 

effective technical 

support 

(i.e., help desk) 

when I encounter 

problems with 

VBMS-Core

Strongly 

agree

5.8 4.6 7.1

Agree 32.9 30.3 35.5

Neither 

agree nor 

disagree

26.7 24.3 29.1

Disagree 17.6 15.5 19.7

Page 58 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Strongly 

disagree

9.8 8.2 11.5

Not 

applicable or 

no basis to 

judge

7.1 5.6 8.5

8.1m. As I do my 

work using VBMSCore, the 

organization of 

information on the 

screen is logical

Strongly 

agree

6.2 4.8 7.5

Agree 53.0 50.2 55.7

Neither 

agree nor 

disagree

20.3 18.1 22.5

Disagree 14.8 12.9 16.8

Strongly 

disagree

5.1 4.0 6.4

Not 

applicable or 

no basis to 

judge

0.6 0.3 1.3

8.1n. VBMS-Core 

performance is 

timely (e.g., minimal 

time to load pages 

and respond to 

commands)

Strongly 

agree

4.7 3.6 6.0

Agree 28.3 25.8 30.7

Neither 

agree nor 

disagree

19.4 17.2 21.6

Disagree 28.0 25.5 30.5

Strongly 

disagree

18.8 16.7 20.8

Not 

applicable or 

no basis to 

judge

0.8 0.4 1.6

Page 59 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

8.1o. I believe 

VBMS-Core helps to 

reduce claims 

processing times 

compared to the 

previous system(s)

Strongly 

agree

11.0 9.2 12.8

Agree 34.5 31.8 37.1

Neither 

agree nor 

disagree

23.5 21.2 25.8

Not 

applicable or 

no basis to 

judge

3.2 2.3 4.4

8.1p. I feel I need to 

have VBMS-Core to 

perform my job 

duties

Strongly 

agree

16.5 14.5 18.5

Agree 41.7 39.0 44.4

Neither 

agree nor 

disagree

20.5 18.3 22.7

Disagree 14.0 12.0 15.9

Strongly 

disagree

6.5 5.1 7.8

Not 

applicable or 

no basis to 

judge

0.9 0.4 1.7

8.1q. I see 

improvements being 

made to VBMS-Core 

from one release to 

the next

Strongly 

agree

14.3 12.4 16.3

Agree 57.6 54.9 60.3

Neither 

agree nor 

disagree

17.8 15.7 19.8

Disagree 6.4 5.1 7.8

Strongly 

disagree

2.4 1.7 3.3

Page 60 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Not 

applicable or 

no basis to 

judge

1.5 0.8 2.4

8.1r. VBMS-Core is 

an improvement 

over the previous 

system(s)

Strongly 

agree

14.8 12.7 16.7

Agree 42.0 39.3 44.7

Neither 

agree nor 

disagree

22.6 20.3 24.8

Disagree 11.5 9.8 13.3

Strongly 

disagree

6.7 5.4 8.0

Not 

applicable or 

no basis to 

judge

3.2 2.3 4.4

Not 

applicable or 

no basis to 

judge

2.4 1.6 3.5

8.1s. Currently, if I 

had to choose 

between VBMSCore or the previous 

system(s) to process 

claims, I would 

choose VBMS-Core

Strongly 

agree

20.3 18.1 22.6

Agree 38.2 35.5 40.8

Neither 

agree nor 

disagree

17.5 15.5 19.6

Disagree 12.6 10.8 14.4

Strongly 

disagree

9.7 8.1 11.3

Not 

applicable or 

no basis to 

judge

1.8 1.1 2.6

Page 61 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

8.1t. I would prefer 

to complete my 

claims processing 

tasks using only 

VBMS-Core

Strongly 

agree

22.9 20.5 25.2

Agree 38.3 35.7 41.0

Neither 

agree nor 

disagree

17.1 15.0 19.1

Disagree 12.5 10.7 14.3

Strongly 

disagree

8.2 6.7 9.7

Not 

applicable or 

no basis to 

judge

1.1 0.6 1.7

8.1u. Overall, I am 

satisfied with VBMSCore

Strongly 

agree

12.1 10.3 14.0

Agree 46.7 43.9 49.4

Neither 

agree nor 

disagree

18.4 16.3 20.5

Disagree 15.1 13.1 17.1

Strongly 

disagree

7.3 5.9 8.7

Not 

applicable or 

no basis to 

judge

0.4 0.1 0.8

Page 62 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

8.2. While using VBMS-Core how often, if at all, do you experience 

each of the following? Select one answer in each row.

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

8.2a. The scanned 

documents 

displayed on the 

screen within VBMSCore are clearly 

legible and correctly 

oriented (i.e., not 

upside-down)

Always or 

almost 

always

10.6 8.9 12.3

Most of the 

time

69.7 67.2 72.2

About half of 

the time

16.5 14.5 18.5

Rarely 2.4 1.6 3.4

Never 0.1 0.0 0.4

Not 

applicable or 

no basis to 

judge

0.7 0.4 1.3

8.2b. The scanned 

documents 

displayed on the 

screen within VBMSCore are 

labeled/categorized 

correctly

Always or 

almost 

always

4.1 3.1 5.3

Most of the 

time

50.5 47.8 53.2

About half of 

the time

34.8 32.2 37.4

Rarely 9.2 7.6 10.7

Never 0.6 0.3 1.2

Not 

applicable or 

no basis to 

judge

0.8 0.4 1.3

8.2c. The scanned 

documents within 

VBMS-Core have 

the correct receipt 

dates associated 

with them

Always or 

almost 

always

8.0 6.5 9.5

Page 63 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Most of the 

time

60.7 58.0 63.3

About half of 

the time

24.2 21.9 26.5

Rarely 5.1 4.0 6.4

Never 0.2 0.1 0.6

Not 

applicable or 

no basis to 

judge

1.8 1.1 2.5

8.2d. VBMS-Core 

gives error 

messages that 

clearly tell me how to 

fix problems

Always or 

almost 

always

3.3 2.5 4.4

Most of the 

time

17.4 15.3 19.4

About half of 

the time

19.1 17.0 21.3

Rarely 34.5 31.9 37.1

Never 16.3 14.2 18.4

Not 

applicable or 

no basis to 

judge

9.4 7.9 10.9

8.2e. I can recover 

from mistakes that I 

make quickly and 

easily when I use 

VBMS-Core

Always or 

almost 

always

8.4 6.9 10.0

Most of the 

time

39.6 36.9 42.2

About half of 

the time

25.6 23.2 28.0

Rarely 15.6 13.5 17.6

Never 3.7 2.7 4.9

Not 

applicable or 

no basis to 

judge

7.2 5.9 8.4

8.2f. VBMS-Core is 

reliable with very 

minimal downtime

Always or 

almost 

always

5.9 4.6 7.1

Page 64 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Most of the 

time

54.4 51.6 57.1

About half of 

the time

26.4 24.0 28.8

Rarely 8.3 6.8 9.8

Never 3.3 2.4 4.4

Not 

applicable or 

no basis to 

judge

1.8 1.1 2.7

8.3. If you have any comments regarding your answers to 

questions 8.1 and/or 8.2 concerning VBMS-Core, share them 

here. (Open-ended response.)

Section 3—Veterans Benefits Management System-Rating 

(VBMS-R)

9. In the course of your work, do you use VBMS-R?

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Yes 35.3 34.7 35.9

No 64.7 64.1 65.3

Complete questions 9.1 and 9.2 for VBMS-R. In question 9.3, you will 

have a chance to comment on any of your answers after responding 

to the questions.

Page 65 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

9.1. Do you agree or disagree with the following statements based 

on your current experience using VBMS-R? Select one answer 

in each row.

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

9.1a. VBMS-R 

meets my needs

Strongly 

agree

13.9 11.6 16.2

Agree 58.7 55.6 61.7

Neither agree 

nor disagree

13.4 11.3 15.4

Disagree 9.1 7.3 10.9

Strongly 

disagree

4.0 2.8 5.4

Not 

applicable or 

no basis to 

judge

1.0 0.5 1.8

9.1b. VBMS-R does 

everything I would 

expect it to do

Strongly 

agree

8.7 6.9 10.6

Agree 37.7 34.7 40.7

Neither agree 

nor disagree

17.2 14.9 19.5

Disagree 27.9 25.1 30.7

Strongly 

disagree

7.5 5.9 9.3

Not 

applicable or 

no basis to 

judge

1.0 0.4 1.8

9.1c. VBMS-R helps 

me be more 

productive 

compared to the 

previous system(s)

Strongly 

agree

12.3 10.1 14.5

Agree 37.3 34.3 40.4

Neither agree 

nor disagree

23.5 20.9 26.0

Disagree 15.9 13.8 18.1

Strongly 

disagree

8.8 7.4 10.3

Page 66 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Not 

applicable or 

no basis to 

judge

2.2 1.4 3.3

9.1d. VBMS-R helps 

me be more efficient 

compared to the 

previous system(s)

Strongly 

agree

12.6 10.4 14.8

Agree 41.0 37.9 44.1

Neither agree 

nor disagree

20.2 17.8 22.6

Disagree 15.3 13.2 17.4

Strongly 

disagree

8.9 7.3 10.4

Not 

applicable or 

no basis to 

judge

2.0 1.2 3.1

9.1e. I feel 

comfortable using 

VBMS-R

Strongly 

agree

23.0 20.3 25.7

Agree 55.9 52.9 59.0

Neither agree 

nor disagree

11.0 9.1 12.8

Disagree 7.1 5.6 8.9

Strongly 

disagree

2.1 1.3 3.2

Not 

applicable or 

no basis to 

judge

0.9 0.4 1.7

9.1f. VBMS-R is 

easier to use 

compared to the 

previous system(s)

Strongly 

agree

14.0 11.7 16.3

Agree 33.3 30.3 36.3

Neither agree 

nor disagree

25.3 22.7 28.0

Disagree 16.4 14.3 18.6

Strongly 

disagree

8.7 7.2 10.2

Page 67 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Not 

applicable or 

no basis to 

judge

2.2 1.4 3.3

9.1g. I can use 

VBMS-R 

successfully every 

time 

Strongly 

agree

10.0 8.0 11.9

Agree 38.4 35.4 41.4

Neither agree 

nor disagree

19.0 16.6 21.5

Disagree 25.3 22.6 28.0

Strongly 

disagree

6.6 5.1 8.3

Not 

applicable or 

no basis to 

judge

0.7 0.3 1.4

9.1h. Compared to 

the previous 

system(s), VBMS-R 

requires fewer steps 

to accomplish what I 

need to do

Strongly 

agree

11.0 8.9 13.2

Agree 29.9 27.0 32.8

Neither agree 

nor disagree

24.2 21.6 26.9

Disagree 21.3 18.8 23.7

Strongly 

disagree

11.5 9.8 13.3

Not 

applicable or 

no basis to 

judge

2.0 1.2 3.1

9.1i. I have been 

provided with 

effective training on 

how to use VBMS-R

Strongly 

agree

11.0 9.0 13.0

Agree 47.0 43.9 50.1

Neither agree 

nor disagree

18.2 15.9 20.6

Disagree 15.3 13.1 17.5

Page 68 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Strongly 

disagree

7.8 6.2 9.5

Not 

applicable or 

no basis to 

judge

0.7 0.3 1.5

9.1j. I learned to use 

VBMS-R quickly

Strongly 

agree

15.8 13.4 18.2

Agree 53.0 49.9 56.1

Neither agree 

nor disagree

16.5 14.2 18.7

Disagree 11.0 9.0 12.9

Strongly 

disagree

2.9 2.0 4.2

Not 

applicable or 

no basis to 

judge

0.9 0.4 1.7

9.1k. I easily 

remember how to 

use VBMS-R

Strongly 

agree

17.4 15.0 19.9

Agree 58.1 55.0 61.2

Neither agree 

nor disagree

15.1 12.9 17.3

Disagree 6.9 5.3 8.6

Strongly 

disagree

1.5 0.8 2.5

Not 

applicable or 

no basis to 

judge

1.1 0.5 1.9

9.1l. VA provides 

effective technical 

support 

(i.e., help desk) 

when I encounter 

problems with 

VBMS-R

Strongly 

agree

6.2 4.7 8.0

Agree 29.1 26.2 31.9

Neither agree 

nor disagree

27.1 24.3 29.8

Disagree 17.3 15.0 19.6

Page 69 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Strongly 

disagree

12.8 10.7 14.9

Not 

applicable or 

no basis to 

judge

7.5 5.9 9.3

9.1m. As I do my 

work using VBMS-R, 

the organization of 

information on the 

screen is logical

Strongly 

agree

10.4 8.4 12.4

Agree 57.2 54.1 60.3

Neither agree 

nor disagree

17.6 15.3 20.0

Disagree 9.6 7.8 11.3

Strongly 

disagree

4.2 3.0 5.7

Not 

applicable or 

no basis to 

judge

1.0 0.5 1.9

9.1n. VBMS-R 

performance is 

timely (e.g., minimal 

time to load pages 

and respond to 

commands)

Strongly 

agree

7.8 6.1 9.8

Agree 38.4 35.3 41.4

Neither agree 

nor disagree

19.0 16.5 21.4

Disagree 23.0 20.4 25.5

Strongly 

disagree

11.1 9.2 13.0

Not 

applicable or 

no basis to 

judge

0.9 0.4 1.7

9.1o. I believe 

VBMS-R helps to 

reduce claims 

processing times 

compared to the 

previous system(s)

Strongly 

agree

11.5 9.3 13.7

Agree 35.2 32.2 38.2

Page 70 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Neither agree 

nor disagree

26.4 23.7 29.1

Disagree 16.0 13.8 18.1

Strongly 

disagree

9.4 7.7 11.0

Not 

applicable or 

no basis to 

judge

1.6 0.9 2.6

9.1p. I feel I need to 

have VBMS-R to 

perform my job 

duties

Strongly 

agree

18.6 16.1 21.1

Agree 45.2 42.1 48.3

Neither agree 

nor disagree

18.4 16.0 20.7

Disagree 11.2 9.5 13.0

Strongly 

disagree

5.2 3.9 6.8

Not 

applicable or 

no basis to 

judge

1.4 0.7 2.3

9.1q. I see 

improvements being 

made to VBMS-R 

from one release to 

the next

Strongly 

agree

9.0 7.1 10.8

Agree 51.3 48.2 54.4

Neither agree 

nor disagree

24.1 21.5 26.7

Disagree 10.8 8.9 12.7

Strongly 

disagree

3.3 2.3 4.6

Not 

applicable or 

no basis to 

judge

1.6 0.9 2.6

9.1r. VBMS-R is an 

improvement over 

the previous 

system(s)

Strongly 

agree

13.0 10.8 15.2

Agree 41.0 37.9 44.1

Page 71 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Neither agree 

nor disagree

23.0 20.4 25.6

Disagree 13.2 11.2 15.1

Strongly 

disagree

7.5 6.0 9.4

Not 

applicable or 

no basis to 

judge

2.3 1.5 3.5

9.1s. Currently, if I 

had to choose 

between VBMS-R or 

the previous 

system(s) to process 

claims, I would 

choose VBMS-R

Strongly 

agree

19.4 16.8 22.0

Agree 40.4 37.3 43.5

Neither agree 

nor disagree

17.0 14.7 19.4

Disagree 12.4 10.6 14.2

Strongly 

disagree

8.7 7.2 10.2

Not 

applicable or 

no basis to 

judge

2.0 1.2 3.1

9.1t. I would prefer 

to complete my 

rating processing 

tasks using only 

VBMS-R

Strongly 

agree

19.7 17.1 22.3

Agree 39.9 36.8 42.9

Neither agree 

nor disagree

19.1 16.7 21.5

Disagree 11.3 9.6 13.1

Strongly 

disagree

8.2 6.7 9.6

Not 

applicable or 

no basis to 

judge

1.8 1.1 2.8

Page 72 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

9.1u. Overall, I am 

satisfied with VBMSR

Strongly 

agree

14.1 11.7 16.4

Agree 48.8 45.7 51.9

Neither agree 

nor disagree

16.9 14.6 19.2

Disagree 12.0 10.0 13.9

Strongly 

disagree

7.3 5.7 9.1

Not 

applicable or 

no basis to 

judge

1.0 0.4 1.8

9.2. While using VBMS-R, how often, if at all, do you experience 

each of the following? Select one answer in each row.

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

9.2a. VBMS-R gives 

error messages that 

clearly tell me how to 

fix problems

Always or 

almost 

always

4.7 3.4 6.4

Most of the 

time

25.2 22.4 27.9

About half 

of the time

20.8 18.3 23.3

Rarely 32.1 29.2 35.0

Never 11.4 9.4 13.4

Not 

applicable 

or no basis 

to judge

5.8 4.4 7.5

9.2b. I can recover 

from mistakes that I 

make quickly and 

easily when I use 

VBMS-R

Always or 

almost 

always

10.4 8.4 12.3

Page 73 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Most of the 

time

44.7 41.5 47.8

About half 

of the time

26.0 23.3 28.7

Rarely 12.4 10.5 14.3

Never 3.2 2.2 4.5

Not 

applicable 

or no basis 

to judge

3.4 2.3 4.7

9.2c. VBMS-R is 

reliable with very 

minimal downtime

Always or 

almost 

always

10.0 8.1 12.0

Most of the 

time

52.5 49.4 55.6

About half 

of the time

23.6 21.0 26.2

Rarely 8.0 6.4 9.6

Never 3.8 2.7 5.2

Not 

applicable 

or no basis 

to judge

2.0 1.2 3.1

9.3. If you have any comments regarding your answers to 

questions 9.1 and/or 9.2 concerning VBMS-R, share them here. 

(Open-ended response.)

Page 74 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Section 4—Veterans Benefits Management System-Awards 

(VBMS-A)

10. In the course of your work, do you use VBMS-A?

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

Yes 40.1 38.2 42.0

No 59.9 58.0 61.8

Complete questions 10.1 and 10.2 for VBMS-A. In question 10.3, you 

will have a chance to comment on any of your answers after 

responding to the questions.

10.1. Do you agree or disagree with the following statements based 

on your current experience using VBMS-A? Select one answer 

in each row.

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

10.1a. VBMS-A 

meets my needs

Strongly 

agree

12.8 10.0 15.6

Agree 55.7 51.4 59.9

Neither 

agree nor 

disagree

15.7 12.6 18.9

Disagree 12.9 10.0 15.9

Strongly 

disagree

1.7 0.8 3.2

Not 

applicable 

or no basis 

to judge

1.2 0.5 2.5

10.1b. VBMS-A does 

everything I would 

expect it to do

Strongly 

agree

8.4 6.2 11.0

Agree 34.0 30.0 38.0

Neither 

agree nor 

disagree

20.4 16.9 23.9

Disagree 29.3 25.3 33.2

Page 75 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

Strongly 

disagree

6.9 4.9 9.5

Not 

applicable 

or no basis 

to judge

1.0 0.4 2.2

10.1c. VBMS-A helps 

me be more 

productive compared 

to the previous 

system(s)

Strongly 

agree

16.0 12.9 19.1

Agree 43.7 39.5 48.0

Neither 

agree nor

disagree

21.8 18.3 25.3

Disagree 10.3 7.8 13.3

Strongly 

disagree

3.9 2.4 5.9

Not 

applicable 

or no basis 

to judge

4.3 2.8 6.3

10.1d. VBMS-A helps 

me be more efficient 

compared to the 

previous system(s)

Strongly 

agree

17.0 13.8 20.1

Agree 41.1 36.9 45.3

Neither 

agree nor 

disagree

22.2 18.7 25.7

Disagree 10.9 8.3 14.0

Strongly 

disagree

4.5 2.9 6.7

Not 

applicable 

or no basis 

to judge

4.3 2.8 6.4

10.1e. I feel 

comfortable using 

VBMS-A

Strongly 

agree

22.3 18.7 25.9

Agree 54.6 50.3 58.8

Page 76 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

Neither 

agree nor 

disagree

13.6 10.6 16.6

Disagree 7.5 5.4 10.2

Strongly 

disagree

1.7 0.8 3.1

Not 

applicable 

or no basis 

to judge

0.3 0.0 1.2

10.1f. VBMS-A is 

easier to use 

compared to the 

previous system(s)

Strongly 

agree

19.2 15.7 22.6

Agree 34.3 30.2 38.3

Neither 

agree nor 

disagree

26.1 22.3 29.8

Disagree 11.8 9.1 15.0

Strongly 

disagree

4.4 2.8 6.6

Not 

applicable 

or no basis 

to judge

4.2 2.7 6.3

10.1g. I can use 

VBMS-A successfully 

every time 

Strongly 

agree

9.1 6.8 11.8

Agree 31.6 27.7 35.5

Neither 

agree nor 

disagree

20.9 17.4 24.4

Disagree 30.0 26.0 33.9

Strongly 

disagree

7.9 5.7 10.7

Not 

applicable 

or no basis 

to judge

0.5 0.1 1.5

Page 77 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

10.1h. Compared to 

the previous 

system(s), VBMS-A 

requires fewer steps 

to accomplish what I

need to do

Strongly 

agree

17.5 14.2 20.7

Agree 38.4 34.2 42.5

Neither 

agree nor 

disagree

24.6 20.9 28.3

Disagree 10.5 8.0 13.4

Strongly 

disagree

4.4 2.8 6.6

Not 

applicable 

or no basis 

to judge

4.7 3.1 6.8

10.1i. I have been 

provided with 

effective training on 

how to use VBMS-A

Strongly 

agree

10.5 8.0 13.5

Agree 45.9 41.7 50.2

Neither 

agree nor 

disagree

17.2 14.0 20.3

Disagree 19.0 15.6 22.3

Strongly 

disagree

7.1 5.0 9.6

Not 

applicable 

or no basis 

to judge

0.3 0.0 1.3

10.1j. I learned to use 

VBMS-A quickly

Strongly 

agree

19.0 15.6 22.3

Agree 49.8 45.6 54.1

Neither 

agree nor 

disagree

18.3 15.0 21.7

Disagree 11.3 8.7 14.4

Strongly 

disagree

1.3 0.5 2.6

Page 78 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

Not 

applicable 

or no basis 

to judge

0.3 0.0 1.3

10.1k. I easily 

remember how to use 

VBMS-A

Strongly 

agree

20.0 16.5 23.4

Agree 54.5 50.2 58.8

Neither 

agree nor 

disagree

17.2 13.9 20.4

Disagree 6.0 4.2 8.4

Strongly 

disagree

1.5 0.6 3.0

Not 

applicable 

or no basis 

to judge

0.8 0.2 2.0

10.1l. VA provides 

effective technical 

support 

(i.e., help desk) when 

I encounter problems 

with VBMS-A

Strongly 

agree

8.1 5.9 10.8

Agree 27.6 23.8 31.3

Neither 

agree nor 

disagree

29.7 25.8 33.5

Disagree 15.5 12.3 18.7

Strongly 

disagree

4.5 2.8 6.7

Not 

applicable 

or no basis 

to judge

14.7 11.7 17.7

10.1m. As I do my 

work using VBMS-A, 

the organization of 

information on the 

screen is logical

Strongly 

agree

12.3 9.5 15.1

Agree 55.3 51.0 59.5

Neither 

agree nor 

disagree

18.9 15.6 22.3

Page 79 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

Disagree 10.7 8.2 13.7

Strongly 

disagree

2.2 1.2 3.9

Not 

applicable 

or no basis 

to judge

0.6 0.1 1.6

10.1n. VBMS-A 

performance is timely 

(e.g., minimal time to 

load pages and 

respond to 

commands)

Strongly 

agree

13.3 10.4 16.2

Agree 59.8 55.5 63.9

Neither 

agree nor 

disagree

14.3 11.4 17.3

Disagree 9.5 7.0 12.4

Strongly 

disagree

2.2 1.1 3.8

Not 

applicable 

or no basis 

to judge

1.0 0.4 2.3

10.1o. I believe 

VBMS-A helps to 

reduce claims 

processing times 

compared to the 

previous system(s)

Strongly 

agree

18.9 15.6 22.3

Agree 42.0 37.8 46.2

Neither 

agree nor 

disagree

23.8 20.1 27.4

Disagree 7.8 5.6 10.5

Strongly 

disagree

3.6 2.1 5.6

Not 

applicable 

or no basis 

to judge

4.0 2.5 6.0

10.1p. I feel I need to 

have VBMS-A to 

perform my job duties

Strongly 

agree

16.4 13.2 19.6

Page 80 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

Agree 42.3 38.1 46.6

Neither 

agree nor 

disagree

21.9 18.3 25.4

Disagree 13.6 10.6 16.6

Strongly 

disagree

4.2 2.6 6.3

Not 

applicable 

or no basis 

to judge

1.6 0.7 3.0

10.1q. I see 

improvements being 

made to VBMS-A 

from one release to 

the next

Strongly 

agree

14.5 11.5 17.6

Agree 47.2 43.0 51.5

Neither 

agree nor 

disagree

25.6 21.8 29.3

Disagree 5.9 4.0 8.4

Strongly 

disagree

1.4 0.6 2.7

Not 

applicable 

or no basis 

to judge

5.3 3.6 7.6

10.1r. VBMS-A is an 

improvement over the 

previous system(s)

Strongly 

agree

21.3 17.7 24.8

Agree 36.8 32.7 41.0

Neither 

agree nor 

disagree

24.5 20.7 28.2

Disagree 9.2 6.8 12.0

Strongly 

disagree

3.6 2.2 5.6

Not 

applicable 

or no basis 

to judge

4.6 2.9 6.8

Page 81 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

10.1s. Currently, if I 

had to choose 

between VBMS-A or 

the previous 

system(s) to process 

claims, I would 

choose VBMS-A

Strongly 

agree

25.4 21.6 29.1

Agree 39.2 35.0 43.3

Neither 

agree nor 

disagree

17.2 14.0 20.5

Disagree 10.2 7.7 13.1

Strongly 

disagree

5.1 3.4 7.4

Not 

applicable 

or no basis 

to judge

2.9 1.7 4.7

10.1t. I would prefer 

to complete my 

award processing 

tasks using only 

VBMS-A

Strongly 

agree

29.0 25.1 32.9

Agree 40.3 36.1 44.4

Neither 

agree nor 

disagree

16.0 12.8 19.1

Disagree 9.2 6.9 12.0

Strongly 

disagree

3.9 2.4 6.0

Not 

applicable 

or no basis 

to judge

1.6 0.7 3.0

10.1u. Overall, I am 

satisfied with VBMSA

Strongly 

agree

17.6 14.3 20.9

Agree 49.4 45.1 53.7

Neither 

agree nor 

disagree

19.6 16.1 23.0

Disagree 8.8 6.5 11.6

Page 82 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

Strongly 

disagree

4.2 2.6 6.3

Not 

applicable 

or no basis 

to judge

0.4 0.1 1.4

10.2. While using VBMS-A how often, if at all, do you experience 

each of the following? Select one answer in each row.

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

10.2a. VBMS-A gives 

error messages that 

clearly tell me how to 

fix problems.

Always or 

almost 

always

5.3 3.6 7.5

Most of the 

time

26.5 22.7 30.3

About half 

of the time

20.7 17.2 24.2

Rarely 26.6 22.8 30.4

Never 9.0 6.6 11.9

Not 

applicable 

or no basis 

to judge

11.9 9.3 14.5

10.2b. I can recover 

from mistakes that I 

make quickly and 

easily when I use 

VBMS-A

Always or 

almost 

always

12.5 9.7 15.3

Most of the 

time

48.3 44.1 52.6

About half 

of the time

19.9 16.5 23.3

Rarely 9.0 6.7 11.8

Never 2.0 1.0 3.6

Not 

applicable 

or no basis 

to judge

8.2 6.1 10.8

Page 83 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

10.2c. VBMS-A is 

reliable with very 

minimal downtime

Always or 

almost 

always

17.0 13.8 20.2

Most of the 

time

52.8 48.5 57.1

About half 

of the time

18.5 15.1 21.8

Rarely 4.4 2.8 6.5

Never 1.3 0.5 2.7

Not 

applicable 

or no basis 

to judge

6.1 4.2 8.4

10.3. If you have any comments regarding your answers to 

questions 10.1 and/or 10.2 concerning VBMS-A, please share them 

here (Open-ended response.)

Page 84 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Section 5—VBMS Benefits and Challenges

11. How much, if at all, has VBMS improved your ability to do the 

following? Select one answer in each row.

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

11a. Process claims 

faster

Very greatly 

improved

8.6 7.2 9.9

Greatly 

improved

23.3 21.3 25.3

Moderately 

improved

24.2 22.2 26.2

Slightly 

improved

14.6 13.0 16.3

Not 

improved

20.3 18.5 22.2

Not 

applicable 

or no basis 

to judge

9.0 7.8 10.3

11b. Make more 

accurate rating 

decisions

Very greatly 

improved

4.5 3.6 5.6

Greatly 

improved

13.8 12.2 15.3

Moderately 

improved

14.1 12.6 15.5

Slightly 

improved

8.2 7.1 9.3

Not 

improved

12.4 11.2 13.6

Not 

applicable 

or no basis 

to judge

47.1 45.1 49.0

11c. Use automated 

tools (e.g., 

standardized 

correspondence, 

rating application 

evaluation builder, 

rules-based 

calculators, etc.) for 

quicker/more 

accurate processing 

Very greatly 

improved

7.9 6.6 9.2

Page 85 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Greatly 

improved

25.0 23.0 27.1

Moderately 

improved

21.2 19.3 23.0

Slightly 

improved

13.9 12.2 15.5

Not 

improved

9.5 8.2 10.8

Not 

applicable 

or no basis 

to judge

22.6 20.7 24.5

11d. Access claims 

documentation on 

demand (e.g., the 

ability to electronically 

view a Veteran’s 

claim file at multiple 

regional offices at the 

same time)

Very greatly 

improved

21.4 19.4 23.3

Greatly 

improved

38.1 35.8 40.3

Moderately 

improved

17.6 15.8 19.3

Slightly 

improved

7.7 6.4 8.9

Not 

improved

5.8 4.8 6.9

Not 

applicable 

or no basis 

to judge

9.5 8.2 10.8

11e. 

Decrease/eliminate 

paper claims

Very greatly 

improved

31.5 29.3 33.7

Greatly 

improved

42.4 40.1 44.7

Moderately 

improved

12.6 11.0 14.1

Slightly 

improved

5.9 4.8 7.0

Not 

improved

2.7 2.0 3.6

Page 86 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Not 

applicable 

or no basis 

to judge

4.9 4.0 6.0

11f. Other 

improvement(s) 

Very greatly 

improved

3.7 2.8 4.8

Greatly 

improved

8.3 7.0 9.7

Moderately 

improved

10.1 8.6 11.6

Slightly 

improved

3.4 2.5 4.3

Not 

improved

12.8 11.2 14.3

Not 

applicable 

or no basis 

to judge

61.8 59.4 64.1

If other improvement(s), describe. (Open-ended response.)

12. Which of the following would you consider to be the single 

greatest improvement in using VBMS? 

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

Process claims faster 14.6 12.9 16.3

Make more accurate rating 

decisions

3.5 2.8 4.4

Use automated tools (e.g., 

standardized correspondence, 

rating application evaluation 

builder, rules-based calculators, 

etc.) for quicker/more accurate 

processing

14.0 12.5 15.6

Access claims documentation on 

demand (e.g., the ability to 

electronically view a Veteran’s 

claim file at multiple regional offices 

at the same time)

31.6 29.4 33.8

Decrease/eliminate paper claims 28.9 26.8 31.0

Other improvement 1.2 0.7 1.8

Page 87 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval -

lower bound

95 percent 

confidence

interval -

upper bound

I have not experienced any 

improvements

6.2 5.0 7.3

If other improvement, please describe the other VBMS improvement you 

consider to be the greatest. (Open-ended response.)

13. How much of a challenge, if at all, has VBMS been to you in the 

following areas? Select one answer in each row.

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound 

13a. System 

performance (e.g., 

slow response 

times)

Very great

challenge

14.2 12.7 15.8

Great 

challenge

20.9 19.0 22.8

Moderate 

challenge

33.2 31.0 35.4

Slight 

challenge

21.9 20.0 23.8

No challenge 7.7 6.5 9.0

Not 

applicable or 

no basis to 

judge

2.1 1.5 2.8

13b. System 

access (e.g., 

system outages)

Very great

challenge

6.8 5.7 7.9

Great 

challenge

12.8 11.3 14.4

Moderate 

challenge

31.3 29.1 33.5

Slight 

challenge

33.8 31.6 36.0

No challenge 13.0 11.4 14.6

Not 

applicable or 

no basis to 

judge

2.2 1.6 3.0

Page 88 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound 

13c. The need to 

rely on previous 

systems in addition 

to VBMS (e.g., 

VETSNET 

systems)

Very great

challenge

13.0 11.3 14.6

Great 

challenge

17.2 15.3 18.9

Moderate 

challenge

30.9 28.7 33.0

Slight 

challenge

22.7 20.8 24.6

No challenge 12.3 10.8 13.8

Not 

applicable or 

no basis to 

judge

4.0 3.2 5.0

13d. Knowing 

whether the 

evidence required 

to process claims is 

complete in VBMS 

(e.g., not knowing 

whether all 

documents have 

been scanned and 

added to the 

veteran’s file)

Very great

challenge

25.7 23.7 27.8

Great 

challenge

25.3 23.2 27.3

Moderate 

challenge

25.3 23.3 27.4

Slight 

challenge

14.7 13.1 16.4

No challenge 5.7 4.7 6.8

Not 

applicable or 

no basis to 

judge

3.2 2.5 4.0

13e. Other 

challenge(s)

Very great

challenge

9.3 7.8 10.9

Great 

challenge

6.3 4.9 7.6

Page 89 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound 

Moderate 

challenge

6.8 5.5 8.2

Slight 

challenge

3.0 2.1 4.0

No challenge 17.9 15.9 20.0

Not 

applicable or 

no basis to 

judge

56.7 54.1 59.4

If other challenges(s), describe. (Open-ended response.)

14. Which of the following would you consider to be the single 

greatest challenge in using VBMS?

Response

Estimated

percentage

95 percent 

confidence

interval - lower 

bound

95 percent 

confidence

interval -

upper bound

System performance (e.g., slow 

response times)

26.1 24.1 28.1

System access (e.g., system 

outages)

4.3 3.4 5.3

The need to rely on previous 

systems in addition to VBMS 

(e.g., VETSNET systems)

21.5 19.5 23.5

Knowing whether the evidence 

required to process claims is 

complete in VBMS (e.g., not 

knowing whether all documents 

have been scanned and added to 

the veteran’s file)

39.3 37.0 41.6

Other challenge 5.2 4.2 6.2

I have not experienced any 

challenges

3.7 2.9 4.6

If other challenge, describe the other VBMS challenge you consider to be 

the greatest. (Open-ended response.)

Page 90 GAO-15-582 Veterans Benefits Management System
Appendix II: Survey of Veterans Benefits 

Management System End User Experience

15. If you could pick one change to be included in the next 

release/update of VBMS, what would it be and why? 

(Open-ended response.)

16. Please share any comments or suggestions for improvement you 

have about the VBMS system. (Open-ended response.)

Page 91 GAO-15-582 Veterans Benefits Management System
Appendix III: Comments from the Department Appendix III: Comments from the of Veterans Affairs

Department of Veterans Affairs

Page 92 GAO-15-582 Veterans Benefits Management System
Appendix III: Comments from the Department 

of Veterans Affairs

Page 93 GAO-15-582 Veterans Benefits Management System
Appendix III: Comments from the Department 

of Veterans Affairs

Page 94 GAO-15-582 Veterans Benefits Management System
Appendix III: Comments from the Department 

of Veterans Affairs

Page 95 GAO-15-582 Veterans Benefits Management System
Appendix III: Comments from the Department 

of Veterans Affairs

Page 96 GAO-15-582 Veterans Benefits Management System
Appendix IV: GAO Contact and Staff 

Acknowledgments

Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov

In addition to the contact named above, Mark T. Bird (Assistant Director), 

Chris Businsky, Virginia Chanley, Heather A. Collins, Kavita Daitnarayan, 

Kelly Dodson, Nancy Glover, Stuart Kaufman, Michael Little, Jamelyn 

Payan, Scott Pettis, Martin L. Skorczynski, Roger M. Smith, Eric Trout, 

Sonya Vartivarian, and Kevin Walsh made key contributions to this report.

Appendix IV: GAO Contact and Staff 

Acknowledgments

GAO Contact

Staff 

Acknowledgments

(311512) Page 97 GAO-15-582 Veterans Benefits Management System
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (http://www.gao.gov). Each weekday 

afternoon, GAO posts on its website newly released reports, testimony, 

and correspondence. To have GAO e-mail you a list of newly posted 

products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. 

Listen to our Podcasts and read The Watchblog.

Visit GAO on the web at www.gao.gov.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Please Print on Recycled Paper.