VA HEALTH CARE

Additional Guidance, 

Training, and 

Oversight Needed to 

Improve Clinical 

Contract Monitoring

Report to Congressional Requesters

October 2013

GAO-14-54

United States Government Accountability Office
United States Government Accountability Office

Highlights of GAO-14-54, a report to 

congressional requesters

October 2013

VA HEALTH CARE

Additional Guidance, Training, and Oversight 

Needed to Improve Clinical Contract Monitoring

Why GAO Did This Study

VA must frequently contract with nonVA health care providers so that 

clinical providers are available to meet 

veterans’ health care needs. While 

recent studies have disclosed 

problems with VA’s development of 

contracts for clinical services, there 

has been little scrutiny of how VA 

monitors and evaluates the care 

contract providers give to veterans.

GAO was asked to review VA’s efforts 

to monitor clinical contractors working 

in VA facilities. This report examines 

the extent to which VA establishes 

complete performance requirements 

for contract providers, challenges VA 

staff encounter in monitoring contract 

providers’ performance, and the extent 

to which VA oversees VAMC staff 

responsible for monitoring contract 

providers.

GAO reviewed VA acquisition 

regulations and other guidance. In 

addition, GAO visited four VAMCs that 

varied in geographic location and 

selected a nongeneralizable sample of 

three types of clinical contracts from 

each of the four VAMCs to review. 

GAO discussed how VAMC and VISN 

staff monitor and oversee these 

contracts and reviewed contract 

monitoring documentation.

What GAO Recommends

GAO recommends that VA develop 

and disseminate standard templates 

that provide examples of performance 

requirements for clinical contracts, 

revise guidance for CORs to include 

workload information, modify COR 

training, and improve the monitoring 

and oversight of clinical contracts. VA 

concurred with GAO’s 

recommendations.

What GAO Found

All 12 contracts GAO reviewed from the four Department of Veterans Affairs’ 

(VA) medical centers (VAMC) visited contained performance requirements 

consistent with VA acquisition policy. However, the performance requirements 

lacked detail in six categories: type of provider or care; credentialing and 

privileging; clinical practice standards; medical record documentation; business 

processes; and access to care. GAO identified these categories from reviews of 

VA acquisition regulations, VA policies, and hospital accreditation standards; and 

VA officials verified that these six categories were an accurate reflection of 

performance requirements that should be in VA clinical contracts. GAO found, for 

example, one VAMC cardiothoracic contract that had detailed performance 

requirements while another VAMC’s cardiothoracic contract did not contain a 

statement describing the contract provider’s responsibilities for reporting and 

responding to adverse events and patient complaints. GAO also found that 

contracting officials lack tools, such as standard templates, that provide 

examples of the performance requirements that should be included in common 

types of clinical contracts. Such tools would help ensure consistency in 

requirements across contracts.

Contracting officer’s representatives (COR) cited two main challenges in 

monitoring contract providers’ performance—too little time to monitor clinical 

contractors’ performance effectively and inadequate training. Most of the 40 

CORs at the four VAMCs in GAO’s review said that their clinical contract 

monitoring duties were a collateral duty and that they had other primary 

responsibilities, such as serving as a business manager or administrative officer 

for a specialty clinic within the VAMC. GAO found that, on average, each of these 

40 CORs spent about 25 percent of their time monitoring an average of 12 

contracts. CORs said the demands of their primary positions at times prevented 

them from fully monitoring contract providers’ performance. Further, VA’s current 

guidance related to COR responsibilities does not include any information on 

how VAMCs are to determine the feasibility of whether a COR’s workload—

including both COR and primary position responsibilities—will allow them to carry 

out their tasks as CORs for monitoring contract provider performance. GAO also 

found that current VA COR training programs focus on contracts that buy goods, 

not clinical services, and include little information on monitoring responsibilities. 

CORs questioned the usefulness of the COR training VA uses to prepare them 

for monitoring clinical contracts.

VA Central Office conducts limited oversight of COR and contract monitoring 

activities. VA Central Office reviews of COR clinical contractor monitoring 

activities are limited to a small number of annual file reviews that focus on 

verifying the presence of required documentation only and do not assess the 

quality of CORs’ monitoring activities. Since implementing the program in March 

2013 these reviews have been conducted in 4 of 21 network contracting offices 

and as of August 2013 none of the 4 offices has received feedback on these 

reviews. Without a robust monitoring system, VA cannot ensure that all CORs in 

its VAMCs are properly monitoring, evaluating, and documenting the 

performance of contract providers caring for veterans. View GAO-14-54. For more information, 

contact Randall Williamson at (202) 512-7114 

or williamsonr@gao.gov.
Page i GAO-14-54 VA Clinical Contract Monitoring

Letter 1 

Background 6

Reviewed Contracts Included Performance Requirements, but 

Lacked Complete Detail Needed to Hold Contractors 

Accountable in Some Cases 10

Selected VA Clinical Contracts Did Not Always Include Clear and 

Measurable Performance Standards 16

CORs Reported that Heavy Workloads and Inadequate Training 

Make It Difficult to Effectively Monitor Contract Providers’ 

Performance 22

VA Central Office Oversight of VAMC Clinical Contract Monitoring 

is Limited 29

Conclusions 31

Recommendations for Executive Action 32

Agency Comments 33

Appendix I Objectives, Scope and Methodology 36

Appendix II Comments from the Department of Veterans Affairs 43

Appendix III GAO Contact and Staff Acknowledgments 46

Tables

Table 1: Level of Detail in Performance Requirements for 12 

Selected Contracts at Four VA Medical Centers (VAMC), 

by Contract Type and Performance Requirement Category 12

Table 2: Extent to Which Selected Contracts from Four VA Medical 

Centers (VAMC) Included Clearly Stated Performance 

Standards, by Contract Type and Performance 

Requirement Category 18

Table 3: Extent to Which Selected Contracts from Four VA Medical 

Centers (VAMC) Included Measurable Targets for 

Performance Standards, by Contract Type and 

Performance Requirement Category 20

Table 4: VA Requirements for Contracting Officer’s Representative 

(COR) Monitoring of Clinical Contractor Performance 23

Contents 
Page ii GAO-14-54 VA Clinical Contract Monitoring

Table 5: A Sample of Position Descriptions of Contracting Officer’s 

Representatives’ (COR) Primary Positions at the Four VA 

Medical Centers (VAMC) We Visited 24

Table 6: Contract Types Reviewed at Each Selected VA Medical 

Center (VAMC) 38

Table 7: Performance Requirement Categories and Required 

Components 40

Figures

Figure 1: Organization of the Veterans Health Administration 

(VHA) Procurement and Logistics Office 9

Figure 2: Performance Requirement Categories for VA Clinical 

Contracts 11

Abbreviations

CBOC community-based outpatient clinic

CO contracting officer

COR contracting officer’s representative

FAC Federal Acquisition Certification

FAR Federal Acquisition Regulation

OIG Office of the Inspector General

SAO Service Area Office

VA Department of Veterans Affairs

VAAR VA Acquisition Regulation

VAMC VA medical center

VHA Veterans Health Administration

VISN Veterans Integrated Service Network

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-14-54 VA Clinical Contract Monitoring

441 G St. N.W.

Washington, DC 20548

October 31, 2013

Congressional Requesters

The Department of Veterans Affairs (VA) employs the majority of the 

clinical providers that care for veterans in VA-operated facilities, such as 

VA medical centers (VAMC) and community-based outpatient clinics 

(CBOC).1

Previous studies have highlighted challenges VA has faced developing 

and administering its clinical contracts. In recent years, for example, the 

VA Office of the Inspector General (OIG) highlighted challenges VA faces 

in developing its clinical contracts and found systemic weaknesses in the 

process VA uses to award contracts.

However, in order to meet the needs of the veterans it serves 

and ensure that clinical providers are available to treat veterans, VA 

frequently contracts with non-VA health care providers to provide services 

in VA facilities. According to VA, every VAMC has at least one contract in 

place with a non-VA health care provider to help supplement the number 

of providers employed by VA facilities. These non-VA health care 

providers either provide services that the VAMC or CBOC do not currently 

offer or supplement the capacity of the VA facility by providing additional 

clinicians to treat veterans.

2

 

1

VA’s health care system includes 152 VAMCs. In addition to VAMCs, VA operates over 

800 CBOCs that are located in areas surrounding VAMCs and that provide primary care 

and some specialty care services that do not require a hospital stay. CBOCs help reduce 

the travel time of veterans seeking treatment for outpatient care.

These weaknesses were attributed 

to VA’s decentralized oversight of the initial stages of the contracting 

2

See Department of Veterans Affairs Office of Inspector General, Audit of Veterans 

Integrated Service Network Contracts, 10-01767-27 (Washington, D.C.: Dec. 1, 2011) and 

Department of Veterans Affairs Office of Inspector General, Audit of VA Electronic 

Contract Management System, 08-00921-181 (Washington, D.C.: July 30, 2009). In the 

2011 report the VA Office of the Inspector General (OIG) found problems with the 

oversight of Veterans Integrated Service Network (VISN) contracts and a lack of tools to 

effectively manage VISN contracting activities. In the 2009 report the VA OIG found that 

VA’s electronic Contract Management System is not used effectively and procurement 

information in the system is incomplete. In addition, the VA OIG found that VA cannot rely 

on the electronic Contract Management System to determine the total number of 

procurements accurately or the total estimated value of these procurements. As a result of 

the VA OIG’s findings, we did not include data from VA’s electronic Contract Management 

System, for example the number of clinical contracts across VA’s health care system, in 

this report’s findings.

 
 

Page 2 GAO-14-54 VA Clinical Contract Monitoring

process before a contract is awarded to a contractor. However, there 

have been no comprehensive reviews of VA’s efforts to monitor clinical 

contractors once a contract is awarded and contract providers begin 

caring for veterans in VA facilities. You asked us to review VA’s efforts to 

monitor clinical contractors working in VA facilities. In this report we 

examine: (1) the extent to which VA establishes complete performance 

requirements for clinical contractors; (2) the extent to which VA clinical 

contracts include clear and measurable performance standards for

assessing whether or not clinical contractors met the acceptable quality 

levels defined in selected contracts; (3) challenges VA staff encounter in 

monitoring clinical contractors’ performance; and (4) the extent to which 

VA Central Office provides oversight of VA staff responsible for 

monitoring clinical contracts.

To examine the extent to which VA establishes complete performance 

requirements for clinical contractors, we reviewed applicable VA 

Acquisition Regulations (VAAR), policies, and handbooks to determine 

what should be included in clinical contracts’ performance requirement 

statements.3 We also interviewed officials from VA Central Office involved 

in the oversight of acquisitions at VAMCs to discuss their roles in 

managing the staff responsible for the day-to-day oversight of clinical 

contractors—including the Medical Sharing Office.4 In addition, we 

conducted site visits to four VAMCs and reviewed how they monitor and 

oversee clinical contractors. Each of the four VAMCs was located in a 

different Veterans Integrated Service Network (VISN) and had several 

types of clinical contracts in place at their facility.5

 

3

VA established the VAAR to codify and publish uniform policies and procedures for VA’s 

acquisition of supplies and services, which supplement the Federal Acquisition 

Regulations (FAR) applicable to all executive branch agencies. The FAR and agency 

supplements are codified in title 48 of the Code of Federal Regulation. 

These four VAMCs 

were located in Lebanon, Pennsylvania; Nashville, Tennessee; 

Minneapolis, Minnesota; and Seattle, Washington. We also spoke with 

officials from the four network contracting offices responsible for 

administering and executing all contracts for these four VAMCs about 

how their staff manage these clinical contracts and how they interact with 

4

The Medical Sharing Office is a part of the Veterans Health Administration’s (VHA) 

Procurement and Logistics Office. 

5

VISNs oversee the day-to-day functions of VAMCs that are within their network. Each 

VAMC is assigned to a single VISN.
 

Page 3 GAO-14-54 VA Clinical Contract Monitoring

VAMC staff responsible for the daily monitoring of clinical contractors.6

Information obtained from these VAMCs and network contracting offices 

cannot be generalized to all VAMCs and network contracting offices. 

Finally, we reviewed three contracts from each of the four VAMCs we 

visited—including one contract for a CBOC, one contract for the services 

of a specialist, and one contract for the services of a temporary clinical 

provider.7

To analyze the performance requirements included in the 12 selected 

clinical contracts, we identified six performance requirement categories by 

reviewing: (1) the VAAR; (2) VA policy, guidance, and training 

documents;8 and (3) The Joint Commission’s hospital accreditation 

standards.9

 

6

There are 21 network contracting offices within VHA that report directly to VA Central 

Office and manage all the contracting activities of a single VISN and all the VAMCs 

assigned to that VISN.

We verified these categories with officials from the Veterans 

Health Administration (VHA) Procurement and Logistics Office to ensure 

they were an accurate reflection of performance requirements that should 

be included in VA clinical contracts. The six categories included in our 

analysis are: (1) type of provider or care; (2) credentialing and privileging; 

(3) clinical practice standards; (4) medical record documentation; 

7

CBOC contracts can be used to acquire all the services of a CBOC located in a 

community that is geographically separate from a VAMC—including the physical location 

of the clinic and all required clinical personnel, such as primary care and mental health 

providers. Specialty care contracts are used to acquire the services of clinical specialists 

that work within an existing VAMC, such as cardiologists and anesthesiologists. Both 

CBOC and specialty care contracts are typically executed for a 1-year base term with four 

1-year options. Temporary clinical provider contracts are used to acquire the services of 

clinical providers on a temporary basis and the terms of these contracts are typically 

1 year or less. VA refers to these temporary clinical providers as locum tenens.

8

See VA Acquisition Academy, Acquisition Planning Guide, Version 2, release 1 

(Washington, D.C.: undated); VA Veterans Health Administration, Quality of Medical 

Services Performed Within VA Facilities by Academic Affiliates Under Contract, Directive 

2009-040 (August 31, 2009); Veterans Health Administration, Veterans Health Care 

Service Standards, Directive 2006-041 (June 27, 2006); and Veterans Health 

Administration, VA Outpatient Scheduling Processes and Procedures, Directive 2010-027 

(June 9, 2010).

9

The Joint Commission, Hospital Accreditation Standards 2012: Standards, Elements of 

Performance, Scoring, Accreditation Policies (Oakbrook Terrace, IL: 2012); and VAAR 

§ 873.109(b). The Joint Commission is an independent organization that accredits and 

certifies health care organizations and programs in the United States.
 

Page 4 GAO-14-54 VA Clinical Contract Monitoring

(5) business processes; and (6) access to care.10

To examine the extent to which VA clinical contracts include clear and 

measurable performance standards for assessing whether or not clinical 

contractors met the acceptable quality levels defined in selected 

contracts, we reviewed the performance standards from the 12 selected 

contracts. To complete this review, we identified whether the contracts 

contained performance monitoring plans that included performance 

standards in five of the same six categories vital to VA operations we 

previously identified.

We analyzed the 12 

selected clinical contracts to determine if they included all, at least one, or 

none of the required components of these categories.

11 We analyzed performance standards in each 

category to determine if they included all, some, or none of the required 

components of the category. We also interviewed contracting officers 

(CO) and contract supervisors from the four network contracting offices 

that are responsible for administering and executing the contracts from 

the four VAMCs we visited regarding the use and development of 

performance standards within clinical contracts.12 Finally, we spoke with 

contracting officers’ representatives (COR) and medical directors 

responsible for overseeing the clinical contractors for the12 selected 

contracts about their experiences using performance standards within 

clinical contracts to assess contractor performance.13

To examine challenges VA staff encounter in monitoring clinical 

contractors’ performance, we held structured interviews with and 

administered a data collection instrument to the 40 CORs who oversaw 

 

10The access to care performance requirement category was used for CBOC contracts 

because VA policy requires CBOCs to comply with the same timeliness goals required of 

VAMCs.

11VA Directive 1663 requires all clinical contracts to have monitoring plans that ensure the 

appropriate quality assurance standards and data methods are in place, collection of 

these data is performed, and the performance of the clinical contractor is monitored. We 

did not include the business processes category in our assessment of performance 

standards because the administrative requirements covered by this category, such as 

invoicing procedures and time care submission, do not require performance standards for 

their validation.

12COs are authorized to enter into, administer, or terminate contracts and make related 

determinations and findings. 48 C.F.R. §§ 1.602-1(a), 801.602.

13CORs are appointed by the CO responsible for a clinical contract and are responsible 

for monitoring clinical contractor performance.
 

Page 5 GAO-14-54 VA Clinical Contract Monitoring

clinical contracts in the four VAMCs we visited. These structured 

interviews and the data collection instrument were designed to capture 

information about various aspects of CORs’ tasks and responsibilities—

including how many contracts CORs managed, whether or not serving as 

a COR was the individual’s primary position or a collateral duty, and 

experiences with COR training required by VA. Results from this data 

collection instrument cannot be generalized to all CORs throughout VA’s 

health care system, but provide important insights. In addition, we 

interviewed the CORs and medical directors responsible for monitoring 

the 12 contracts we reviewed about their experience monitoring these 

contracts. Finally, we reviewed VA’s COR training courses to assess their 

content and structure.

To examine the extent to which VA Central Office provides oversight of 

VA staff responsible for monitoring clinical contracts, we interviewed VA 

Central Office officials responsible for overseeing the acquisition process, 

including officials from the three regional Service Area Offices (SAO) and 

the VHA Procurement and Logistics Office, about their role in overseeing 

COs and VAMC-based CORs.14

We conducted this performance audit from October 2012 to October 2013 

in accordance with generally accepted government auditing standards.

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.

We also reviewed tools used by these 

offices to conduct oversight of COs and CORs, such as checklists and 

procedure guides. In addition, we spoke with COs and CORs responsible 

for managing and monitoring the 12 contracts we reviewed about their 

experience working with these VA Central Office officials. For additional 

details about the scope and methodology used in this report, see 

appendix I.

 

14Within VA, VHA is the organization responsible for providing health care to veterans at 

medical facilities across the country. SAOs are VHA’s regional contract oversight entities 

organized into three regions—East, West, and Central—that manage the contracting 

activities and acquisition personnel of six to eight VISNs each. SAOs report directly to the 

VHA Procurement and Logistics Office and do not report to VISN or VAMC directors.
 

Page 6 GAO-14-54 VA Clinical Contract Monitoring

Clinical contracts at VA are used to acquire the services of clinical 

personnel, such as physicians, pharmacists, and nurses. These contracts 

can be used to fill vacancies for clinicians in specialties that are difficult to 

recruit, supplement existing VAMC capacity by providing additional 

clinicians in high-volume areas where VA also manages a staff of its own 

employees, or fill critical staffing vacancies on a long- or short-term basis.

Clinical contracting at VA is governed by two sets of regulations—the 

FAR and VAAR—that contain general requirements for all VA 

acquisitions. The FAR contains government-wide regulations and 

establishes uniform policies and procedures used by all executive branch 

federal agencies for their acquisitions. VA supplements the FAR with the 

VAAR to establish uniform policies and procedures for all VA acquisitions 

of supplies and services. In addition to the VAAR, VA provides guidance 

to its acquisition workforce in the form of policy directives and handbooks.

Both acquisition and clinical staff at VA work together to plan, execute, 

and monitor clinical contracts at VA. On the acquisition side, COs are 

responsible for planning, awarding, and administering contracts on behalf 

of the federal government. Each CO is able to obligate federal funds up to 

a specified limit and a CO must formally approve all clinical contracts at 

VA. Common tasks of a CO include developing acquisition planning 

documents used to begin a clinical contract, conducting market research 

to determine pricing and availability for a clinical contract, and completing 

the formal competitive or non-competitive solicitation process for 

contracts. Each CO works within a network contracting office and is 

overseen by managers within that office who report directly to VA Central 

Office. There are 21 network contracting offices throughout VA’s health 

care system that manage all the contracting activities of a single VISN.15

Two types of VAMC staff have monitoring responsibilities for clinical 

contracts—CORs and medical directors. For each VA clinical contract, 

the CO responsible for the contract designates a COR at the VAMC to 

help develop the clinical contract and monitor the contract provider’s 

performance once the provider begins work. Common tasks delegated to 

the COR include providing input on the performance requirements for the 

 

15While network contracting offices manage the contracting activities of a single VISN, 

they are managed by VA Central Office regional contracting management entities, SAOs, 

and have no managerial link to VISN leadership.

Background 

Acquisition Workforce at 

VA
 

Page 7 GAO-14-54 VA Clinical Contract Monitoring

clinical contract, determining how the contract provider’s performance will 

be measured and monitoring performance once work has begun, 

validating the contract provider’s invoices to ensure their accuracy, 

managing contract modifications, and assisting the CO in resolving any 

issues that may arise with the contract provider. At VA, CORs are 

commonly administrative personnel responsible for managing the 

operations of a specialty care line at a VAMC—such as primary care, 

surgery, etc.—where the contractor will be working.

Medical directors in various specialty care lines often assist CORs in 

monitoring contract provider performance because CORs lack the 

expertise to evaluate a contract provider’s clinical abilities.16 Medical 

directors are responsible for overseeing the clinical care provided by all 

providers within a specialty care line, including VA-employed and contract 

clinicians. Medical directors use existing VA processes, including the 

credentialing and privileging process, to monitor contract providers.17 This 

clinical monitoring includes an initial assessment of the clinician’s 

competency during the first 90 days after the contract provider begins 

working at a VA facility and typically includes evaluations in several 

areas—such as patient care, clinical knowledge, and interpersonal 

communication skills.18

 

16For the purposes of our report, we use the term medical directors to mean the same as 

individuals commonly known as service chiefs, service line directors, or section chiefs.

Upon successful completion of the initial 

competency evaluation, medical directors monitor contract providers 

through ongoing evaluations of their performance according to the 

17During the credentialing process, VAMC staff collect and review information such as a 

provider’s professional training, malpractice history, peer references, and other 

components of professional background to determine whether providers have suitable 

abilities and experience for appointment to a VAMC’s medical staff. During the privileging 

process, VAMCs determine which health care services—known as clinical privileges—the 

provider should be allowed to provide. After a provider is hired, the credentialing and 

privileging processes are repeated at least every 2 years.

18These initial assessments of clinician competency are referred to as focused 

professional practice evaluations and typically occur for all VA clinical personnel who 

require clinical privileges at the time of their initial appointment to the medical staff. The 

process may include periodic chart review, direct observation, monitoring of diagnostic 

and treatment techniques, or discussion with other individuals involved in the care of 

patients.
 

Page 8 GAO-14-54 VA Clinical Contract Monitoring

specific performance standards used to evaluate all clinicians working at 

VA within their specialty.19

Medical directors provide the results of these initial and ongoing 

evaluations to the COR and work with the COR to resolve any issues that 

may arise as a result of a contract provider’s clinical performance. CORs 

are responsible for maintaining the official record of the contract 

provider’s performance and providing official performance assessments 

to the CO.

VA Central Office has primary responsibility for overseeing network 

contracting offices and manages clinical contracting activities through the 

VHA Procurement and Logistics Office. There are five primary offices 

within the VHA Procurement and Logistics Office that are responsible for 

overseeing various aspects of clinical contracting activities and report to 

VHA’s Deputy Chief Procurement Officer. (See fig. 1.)

 

19These ongoing evaluations of clinicians are referred to as ongoing professional practice 

evaluations and are regularly conducted at least every 6 months on all VA clinical 

personnel who require clinical privileges. These ongoing professional practice evaluations 

include a number of reviews of professional practice trends that impact the quality of care 

provided by a clinician and patient safety. 

VA Acquisition Oversight 

Structure 
 

Page 9 GAO-14-54 VA Clinical Contract Monitoring

Figure 1: Organization of the Veterans Health Administration (VHA) Procurement 

and Logistics Office

Medical Sharing Office. The Medical Sharing Office is responsible for 

providing guidance to network contracting offices regarding the content 

and structure of solicitations for clinical contracts and for reviewing 

several types of clinical contracts. The Medical Sharing Office reviews 

solicitations of all competitive clinical contracts valued at over $1.5 million, 

all non-competitive clinical contracts valued at over $500,000, and all 

organ transplant contracts.20

Procurement Operations Office. The Procurement Operations Office is 

responsible for providing ongoing guidance and monitoring of the COR 

population at VA. The Procurement Operations Office conducts reviews 

of COR files and publishes a COR newsletter.

All Medical Sharing Office reviews are 

conducted before a solicitation is issued to ensure that all the necessary 

provisions are in place prior to any competition or award.

Procurement Audit Office. The Procurement Audit Office is responsible 

for ensuring compliance with VA policies and procedures related to 

 

20The Medical Sharing Office does not review any contracts for nursing services. Nursing 

contracts are processed and reviewed by SAOs.
 

Page 10 GAO-14-54 VA Clinical Contract Monitoring

contracting. This office conducts internal compliance audits of contracts, 

including clinical contracts, once they are executed to ensure that all 

required documentation was included in the final contract and audits the 

activities of network contracting offices and SAOs to ensure their 

compliance with VA policies and regulations.

Procurement Policy Office. The Procurement Policy Office is 

responsible for providing guidance to VA’s acquisition workforce in 

network contracting offices and SAOs. This office produces and updates 

standard operating procedures for CORs and COs.

Service Area Offices. SAOs are the regional contract management 

entities created to oversee the activities of the 21 network contracting 

offices and the COs and supervisors that work within them. VHA created 

three SAOs—East, West, and Central—to manage the contracting 

activities of six to eight VISNs each. SAOs review solicitations for most 

clinical contracts during their initial stages to ensure that all necessary 

provisions are in place prior to any competition or award.

We found that all 12 contracts we reviewed from the four VAMCs we 

visited contained performance requirements consistent with VA 

acquisition policy. However, we found that 10 of the 12 contracts we

reviewed lacked detailed descriptions of contractors’ performance 

requirements in one or more of the six categories we assessed.21 We 

analyzed the content of these contracts using six performance 

requirement categories we established through reviews of the VAAR, VA 

policies, and the standards of the leading hospital accreditation 

organization. (See fig. 2)

 

21Results of our reviews were recorded as complete, partial, or incomplete based on 

whether the contract’s performance requirements covered all, at least one, or none of the 

specific components reviewed within each category.

Reviewed Contracts 

Included Performance 

Requirements, but 

Lacked Complete 

Detail Needed to Hold 

Contractors 

Accountable in Some 

Cases 
 

Page 11 GAO-14-54 VA Clinical Contract Monitoring

Figure 2: Performance Requirement Categories for VA Clinical Contracts

Note: We identified these performance requirement categories by reviewing the VA Acquisition 

Regulation (VAAR); VA policy, guidance, and training documents; and The Joint Commission’s 

hospital accreditation standards. Officials from the VHA Office of Procurement and Logistics verified 

these six performance requirement categories.

a

We added access to care as a performance requirement category for community-based outpatient 

clinic (CBOC) contracts because VA policy requires CBOCs to comply with the same timeliness goals 

required of VAMCs.

We also found that the level of detail in contract performance 

requirements contained in the 12 contracts we reviewed varied both by 

the type of contract—CBOC contract, specialty care contract, or 

temporary clinical provider contract—and by the six performance 

requirement categories we assessed. (See table 1.)
 

Page 12 GAO-14-54 VA Clinical Contract Monitoring

Table 1: Level of Detail in Performance Requirements for 12 Selected Contracts at Four VA Medical Centers (VAMC), by 

Contract Type and Performance Requirement Category

Performance requirement categorya

Contract type and location

Type of 

provider or 

careb

Credentialing 

and 

privilegingc

Clinical 

practice 

standardsd Medical record documentatione Business processesf Access to careg

Community-based outpatient clinic (CBOC)

Lebanon VAMC ● ● ● ● ● ●

Minneapolis VAMC ● ◓ ● ● ● ●

Nashville VAMC ● ● ● ● ● ●

Seattle VAMC ● ◓ ● ● ◓ ◓

Specialty care

Lebanon VAMC ● ◓ ● ● ● N/A

Minneapolis VAMC ● ◓ ◓ ● ● N/A

Nashville VAMC ● ◓ ◓ ◓ ● N/A

Seattle VAMC ● ◓ ● ● ● N/A

Temporary clinical provider

Lebanon VAMC ◓ ○ ◓ ◓ ● N/A

Minneapolis VAMC ● ◓ ◓ ◓ ● N/A

Nashville VAMC ● ◓ ◓ ◓ ● N/A

Seattle VAMC ● ◓ ◓ ○ ● N/A

Legend: ● = complete; ◓ = partial; ○ = incomplete; N/A = not applicable

Source: GAO analysis of VA information.

a

Each category was recorded as complete, partial, or incomplete, based on whether the contract 

contained performance requirement statements for all, at least one, or none of the components 

identified for each category.

b

Type of provider or care includes requirements for contractor personnel and qualifications, such as 

licensing, and a description of the specialty area in which the contractor will provide care.

c

Credentialing and privileging includes requirements for providers to submit required information for 

the credentialing and privileging process applicable to their specialty before providing care and 

renewal or review requirements.

d

Clinical practice standards includes requirements regarding compliance with VAMC medical staff 

bylaws and the provider’s response to adverse events or patient complaints.

e

Medical record documentation includes requirements for entering information into the veteran’s 

electronic medical record, such as referrals, consults, and a plan of care.

f

Business processes includes administrative requirements, such as invoicing procedures and time 

card submission.

g

Access to care applies only to CBOC contracts and includes requirements regarding compliance with 

VA’s timeliness and scheduling standards, CBOC hours of operation, and types of contract providers 

required to be on site. We added access to care as a performance requirement category for CBOC 

contracts because VA policy requires CBOCs to comply with the same timeliness goals required of 

VAMCs.
 

Page 13 GAO-14-54 VA Clinical Contract Monitoring

The CBOC contracts we reviewed were the most detailed, followed by 

specialty care contracts and temporary clinical provider contracts.

CBOC contracts. Contracts for CBOCs generally contained the most 

detailed performance requirements. For example, all four of the CBOC 

contracts we reviewed had complete performance requirements for three 

of the six applicable categories—the type of provider or care, clinical 

practice standards, and medical record documentation. However, two of 

the four CBOC contracts—at the Minneapolis and Seattle VAMCs—

lacked detail in one or more of the performance requirement categories 

we reviewed. For example, the Seattle VAMC CBOC contract was the 

only CBOC contract we reviewed that did not include a required set of 

operating hours for the contracted facility, which was a component we 

assessed within the access to care category.

Specialty care contracts. The amount of detail in performance 

requirements within specialty care contracts varied across our four 

selected VAMCs. Overall, performance requirements in these contracts

lacked specificity in key categories. For example, two of our selected 

specialty care contracts—for the Seattle and Minneapolis VAMCs—were 

for the services of cardiothoracic surgeons and had very similar results in 

our review, except for their descriptions of performance requirements 

within the clinical practice standards category. For this category, the 

Seattle VAMC cardiothoracic contract had detailed performance 

requirements, but the Minneapolis VAMC cardiothoracic contract did not 

contain a statement describing the provider’s responsibilities for the 

reporting of and response to adverse events and patient complaints, a 

component we assessed within the clinical practice standards category.22

 

22See Veterans Health Administration, Disclosure of Adverse Events to Patients,

Handbook 1004.08 (Oct. 2, 2012). According to VHA, adverse events are untoward 

incidents, diagnostic or therapeutic misadventures, iatrogenic injuries, or other 

occurrences of harm or potential harm directly associated with care or services provided 

within the jurisdiction of VA’s health care system.

In addition, the Nashville VAMC specialty care contract for the services of 

a psychiatrist had limited detail because it did not contain requirements 

for the clinical contractor’s privileges to be renewed at the beginning of 

each new contract term, and received a partial rating in the medical 

record documentation category because the contract did not contain 

performance requirements for the timely entry of information into VA’s 

electronic medical record.
 

Page 14 GAO-14-54 VA Clinical Contract Monitoring

Temporary clinical provider contracts. We found that temporary 

clinical provider contracts included the least detailed performance 

requirements among the three contract types we reviewed. For example, 

the Lebanon VAMC temporary clinical provider contract for a gynecologist 

was missing descriptions of requirements for providers to undergo VA’s 

credentialing and privileging process, comply with The Joint Commission 

requirements that apply to gynecology, and enter relevant data into 

veterans’ electronic medical records. These specific requirements are 

components of three separate categories we reviewed—credentialing and 

privileging, clinical practice standards, and medical record documentation.

In addition, the Seattle VAMC temporary clinical provider contract for a 

pharmacist was missing information on contractor requirements related to 

both the clinical practice standards and medical record documentation 

categories. According to VA officials, temporary clinical provider contracts 

are often written using a format that is designed to allow for shorter and 

faster contract award timeframes; however, this shorter format likely 

contributed to less complete performance requirement statements in the 

contracts we reviewed.

We also found the level of detail in contract performance requirements 

varied significantly by performance requirement category, with a majority 

of reviewed contracts missing some component of the credentialing and 

privileging performance requirement category.

Credentialing and privileging. This category included a review of three 

specific components of performance requirements: (1) requirements for 

all contract providers to initially undergo VA’s credentialing and privileging 

process; (2) the use of VA’s electronic credentialing system for that 

process; and (3) requirements for the renewal of privileges.23

 

23See VHA Handbook 1100.19. VHA’s electronic credentialing system must be used for 

credentialing all providers who are granted clinical privileges or credentialed for other 

reasons. According to VHA policy, the use of this system is necessary to reduce the 

potential for human error in the credentialing process. 

In our 

review, we found that 10 of the 12 contracts were missing performance 

requirements from one or more of these three components. Six of the 10 

contracts did not contain statements describing the rules for renewal of 

privileges, which is part of the credentialing and privileging process at VA. 

Privileges are required to be renewed every 2 years by VA policy. The 

term of a contract provider’s privileges may not extend beyond the term of 

the contract regardless of whether the VAMC intends to renew or extend 
 

Page 15 GAO-14-54 VA Clinical Contract Monitoring

the contract; this means that privileges need to be renewed more often 

than every 2 years for contract providers depending on the length of the 

contract.24

Type of provider or care. This performance requirement category had 

three components: (1) provider licensure requirements, (2) a description 

of the specialty area care was contracted for, and (3) a listing of key 

personnel providing services under the contract.

In addition, 5 of the 10 contracts did not contain statements 

requiring the use of VA’s electronic credentialing system for the 

credentialing process.

25

Business processes. This performance requirement category contained 

three components: (1) invoicing policies; (2) on- and off-duty hours and 

time card submission; and (3) personnel security responsibilities, such as 

background checks. Eleven of the 12 contracts included detailed 

performance requirements in this category, with the exception of the 

Seattle VAMC CBOC contract, which did not list on- and off-duty hours for 

staff, but included all other business process components.

Eleven of the 12 

contracts we reviewed included detailed performance requirements in this 

category, with the exception of the Lebanon VAMC temporary clinical 

provider contract, which did not include the name of the gynecologist 

selected for the contract or rules for the substitution of another provider if 

needed.

COs and CORs we interviewed told us that they lack available tools, such 

as standard templates, that provide examples for the types of 

performance requirements that should be included in the types of clinical 

contracts we selected for our review. Several COs and CORs we spoke 

with told us that they refer to previously awarded contracts as their source 

for developing performance requirements for future contracts. However, 

the previous contracts used as sources may not contain categories of 

performance requirements that adhere to VA policies and allow CORs to 

hold contractors accountable for their performance. In addition, several 

COs and CORs we interviewed said they would find it useful to have 

 

24See VHA Handbook 1100.19. 

25See Department of Veterans Affairs, Health Care Resources Contracting—Buying, Title 

38 U.S.C. 8153, VA Directive 1663 (Aug. 10, 2006). Directive 1663 states that contracts 

for full-time providers must identify the key personnel proposed to provide the service and 

their qualifications.
 

Page 16 GAO-14-54 VA Clinical Contract Monitoring

standard templates for clinical contracts. VA Central Office officials with 

whom we spoke said that there is a template for CBOC contracts that has 

been distributed among contracting officials and that templates for six 

common specialty care contracts, including cardiology and 

anesthesiology, are currently in development as of August 2013, and that 

VA’s progress in developing and implementing these templates is 

dependent upon VA Central Office and the VA OIG agreeing on the six 

specialties that will have templates developed.

Absent available tools, such as standard templates for common types of 

contracts—including those for CBOCs, specialty care, and temporary 

clinical providers—VA cannot reasonably ensure that critical requirements

for contract providers’ performance are consistently included in VA clinical 

contracts and are standardized across VAMCs. Without assurance that 

these critical performance requirements are included in clinical contracts, 

VA may not be able to hold clinical contractors accountable for providing 

the high-quality services VAMCs need to serve veterans.

Two of the 12 clinical contracts we reviewed did not include any 

performance standards and 7 contracts did not include performance 

standards in key categories. Additionally, the performance standards 

included within the 12 clinical contracts we reviewed were not always 

stated clearly and did not always include measurable targets. We found 

that VA did not provide guidance to COs and CORs on how to develop 

performance standards that are clear and measurable for determining 

whether or not clinical contractors met the acceptable quality levels 

defined in selected contracts. 

The FAR requires agencies to ensure that requirements for services are 

clearly defined and appropriate performance standards are developed so 

that the agency’s requirements can be understood by potential offerors 

and that performance in accordance with contract terms and conditions 

will meet the agency’s requirements.26

 

2648 C.F.R. § 37.503(a).

In addition, agencies, in order to 

successfully measure the performance of their operations, use 

performance standards that demonstrate results, are limited to a vital few 

Selected VA Clinical 

Contracts Did Not 

Always Include Clear 

and Measurable 

Performance 

Standards 
 

Page 17 GAO-14-54 VA Clinical Contract Monitoring

activities, and provide useful information for decision-making.27 To 

determine whether performance standards used by agencies meet these 

qualities, we previously found that performance standards should have 

several attributes—including being clearly stated and using measurable 

targets.28 We assessed each of the 12 selected contracts from the four 

VAMCs we visited to determine if the performance standards for five of 

the six categories we previously identified included these two attributes of 

successful performance standards.

29

Performance standards were not clearly stated. Two of the 12 clinical 

contracts we reviewed—the Lebanon VAMC CBOC and temporary 

clinical provider contracts—did not include any monitoring plan provisions 

defining the performance standards against which the VAMC would 

assess the contractor’s performance. Performance standards included in 

the monitoring plans of the remaining 10 clinical contracts we reviewed 

lacked clarity in what would be used to determine a clinical contractor’s 

performance. For example, the Nashville VAMC CBOC contract includes 

a performance standard in the clinical practice standards category that 

requires the contractor to meet a specific target for three of four outpatient 

satisfaction survey scores. However, VA’s outpatient satisfaction survey 

includes significantly more than four elements and the contract’s 

 

27See GAO, Executive Guide: Effectively Implementing the Government Performance and 

Results Act, GAO/GGD-96-118 (Washington, D.C.: June 1996). Prior GAO work had 

determined that these performance standards: (1) demonstrate results by showing an 

organization’s progress toward achieving an intended level of performance, (2) should be 

limited to core program activities to allow managers to make decisions without excess of 

data, and (3) should provide timely information in a format that helps managers make 

decisions to improve program performance.

28See: GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing Season 

Performance Measures, GAO-03-143 (Washington, D.C.: November 22, 2002). For the 

purposes of this report, we selected two attributes—clarity and measurability—to evaluate 

VA clinical contracts’ performance standards due to their applicability to performance 

standards for smaller scale projects. We previously found several other attributes of 

successful performance standards, including: (1) objectivity, (2) reliability, (3) core 

program activities, (4) limited overlap, (5) balance, (6) government-wide priorities, and 

(7) linked to agency priorities. We did not include these additional seven attributes in our 

assessment of each selected contract’s performance standards because they were 

designed to measure the success of broader agency-wide performance standards rather 

than those included in a single contract.

29We did not include assessments of the business processes category in our final results 

because these requirements are administrative and performance standards are not 

necessary to determine success in this area.
 

Page 18 GAO-14-54 VA Clinical Contract Monitoring

performance standard does not clearly specify which four survey 

elements will be used for this analysis. In another instance, the Seattle 

VAMC temporary clinical provider contract includes a performance 

standard for prescription dispensing that includes multiple standards that 

cover numerous tasks related to prescription processing and validation in 

inpatient and outpatient settings. By including multiple parts in a single 

performance standard, it is unclear what the COR will actually be 

measuring to determine the clinical contractor’s performance in this area. 

As table 2 shows, none of the 12 contracts we assessed included clearly 

stated performance standards in all five categories. 

Table 2: Extent to Which Selected Contracts from Four VA Medical Centers (VAMC) Included Clearly Stated Performance 

Standards, by Contract Type and Performance Requirement Category

Performance requirement categorya

Contract type and VAMC location

Type of 

provider or 

careb

Credentialing 

and 

privilegingc

Clinical 

practice 

standardsd Medical record documentatione Access to caref

Community-based outpatient clinic (CBOC)

Lebanon VAMC ⊗ ⊗ ⊗ ⊗ ⊗

Minneapolis VAMC ● ⊗ ● ● ●

Nashville VAMC ● ● ◓ ● ●

Seattle VAMC ● ⊗ ● ⊗ ●

Specialty care

Lebanon VAMC ◓ ● ◓ ● N/A

Minneapolis VAMC ● ◓ ◓ ⊗ N/A

Nashville VAMC ◓ ● ◓ ◓ N/A

Seattle VAMC ● ⊗ ⊗ ● N/A

Temporary clinical provider

Lebanon VAMC ⊗ ⊗ ⊗ ⊗ N/A

Minneapolis VAMC ⊗ ◓ ● ⊗ N/A

Nashville VAMC ◓ ⊗ ● ⊗ N/A

Seattle VAMC ◓ ⊗ ◓ ◓ N/A

Legend: ● = all performance standards were clearly stated; ◓ = some performance standards were 

clearly stated; ⊗ = no performance standards were included in the contract; N/A = not applicable

Source: GAO analysis of VA information.

Note: The business processes category includes only administrative requirements, such as invoicing 

procedures and time card submission. We did not assess performance standards in this category 

because these requirements do not require performance standards for their validation.

a

To assess if a performance standard was clear, we reviewed each performance standards included 

in a selected contract to determine if each performance standards clearly stated what the contractor 

was responsible for and included specific information about how the contractor would be assessed. 

Each category was scored on a scale of complete, partial, or incomplete, based on whether the 

performance standards included in the category met these criteria.
 

Page 19 GAO-14-54 VA Clinical Contract Monitoring

b

Type of provider or care includes performance standards that define specific assessments of clinical 

outcomes for the type of provider covered by the contract.

c

Credentialing and privileging includes performance standards related to the credentialing and 

privileging process.

d

Clinical practice standards includes performance standards related to general clinical practice at VA, 

including compliance with VAMC policies and response to adverse events and patient complaints.

e

Medical record documentation includes performance standards related to contract providers’ use of 

VA’s electronic medical record.

f

Access to care applies only to CBOC contracts and includes performance standards related to 

contract providers’ compliance with VA’s national timeliness goals, CBOC hours of operation, and 

types of clinical providers required to be onsite. We added access to care as a performance 

requirement category for CBOC contracts only because VA policy requires CBOCs to comply with the 

same timeliness goals required of VAMCs.

It is important to ensure that performance standards included in clinical 

contracts are clearly stated to minimize confusion during the monitoring of 

clinical contractors’ performance. Without clearly articulated performance 

standards, COs and CORs may not be able to effectively assess the 

performance of clinical contractors and VA may not be able to hold 

contractors accountable for poor performance.

Performance standards did not include measurable targets. We 

found that there were no categories that included fully measurable 

performance standards in the 12 contracts we assessed. As previously 

noted, 2 of the 12 contracts—the Lebanon VAMC CBOC and temporary 

clinical provider contracts—did not include any monitoring plan provisions 

defining the performance standards against which the VAMC would 

assess the contractor’s performance. Of the remaining 10 contracts with 

monitoring plans, only 1 contract—the Lebanon VAMC specialty 

contract—included fully measurable performance standards in all 

categories, and the other 9 contracts either did not include performance 

standards for a critical category or had performance standards that were 

only partially measurable in at least one category. (See table 3.) For 

example, the Nashville VAMC temporary clinical provider contract 

included a performance standard in the clinical practice standards 

category that stated that the contractor should have no more than two 

patient or staff complaints. However, this contract did not define how 

frequently this target would be measured and therefore it is not possible 

to determine whether this threshold applies to complaints on a monthly, 

quarterly, or annual basis. This limited information on how often the 

complaints would be reviewed and measured for the clinical contractor 

makes it difficult to determine what a successful performance outcome 

would be. In another instance, the Minneapolis VAMC specialty contract 

includes several performance standards in the credentialing and 

privileging category that indicate the clinical contractor is responsible for 
 

Page 20 GAO-14-54 VA Clinical Contract Monitoring

maintaining medical and clinical knowledge and engaging in practicedbased learning. However, this contract does not include information on 

how the clinical contractor’s performance in these areas would be 

assessed and does not specify a clear numerical target for determining 

whether or not these standards have been fulfilled. Finally, the Nashville 

VAMC specialty contract includes a performance standard in the medical 

record documentation category that states the clinical contractor should 

maintain documentation in VA’s electronic medical record system of 95 

percent, but does not include a description of how the COR would assess 

compliance with this performance standard.

Table 3: Extent to Which Selected Contracts from Four VA Medical Centers (VAMC) Included Measurable Targets for 

Performance Standards, by Contract Type and Performance Requirement Category

Performance requirement categorya

Contract type and VAMC location

Type of 

provider or 

careb

Credentialing 

and 

privilegingc

Clinical 

practice 

standardsd Medical record documentatione Access to caref

Community-based outpatient clinic (CBOC)

Lebanon VAMC ⊗ ⊗ ⊗ ⊗ ⊗

Minneapolis VAMC ● ⊗ ◓ ◓ ◓

Nashville VAMC ● ◓ ◓ ● ●

Seattle VAMC ◓ ⊗ ● ⊗ ●

Specialty care

Lebanon VAMC ● ● ● ● N/A

Minneapolis VAMC ● ◓ ◓ ⊗ N/A

Nashville VAMC ◓ ● ◓ ◓ N/A

Seattle VAMC ● ⊗ ⊗ ● N/A

Temporary clinical provider

Lebanon VAMC ⊗ ⊗ ⊗ ⊗ N/A

Minneapolis VAMC ⊗ ◓ ● ⊗ N/A

Nashville VAMC ◓ ⊗ ◓ ⊗ N/A

Seattle VAMC ◓ ⊗ ◓ ◓ N/A

Legend: ● = all performance standards were measurable; ◓ = some performance standards were 

measurable; ⊗ = no performance standards were included in the contract; N/A = not applicable

Source: GAO analysis of VA data.

Note: The business processes category includes only administrative requirements, such as invoicing 

procedures and time card submission. We did not assess performance standards in this category 

because these requirements do not require performance standards for their validation.

a

To assess if a performance standards was measurable, we reviewed each performance standard 

included in a selected contract to determine if each performance standard included a measurable 

target for the contractor, including a numerical target. Each category was scored on a scale of 

complete, partial, or incomplete, based on whether the performance standards included in the 

category met these criteria.
 

Page 21 GAO-14-54 VA Clinical Contract Monitoring

b

Type of provider or care includes performance standards that define specific assessments of clinical 

outcomes for the type of provider covered by the contract.

c

Credentialing and privileging includes performance standards related to the credentialing and 

privileging process.

d

Clinical practice standards includes performance standards related to general clinical practice at VA, 

including compliance with VAMC policies and response to adverse events and patient complaints.

e

Medical record documentation includes performance standards related to contract providers’ use of 

VA’s electronic medical record.

f

Access to care applies only to CBOC contracts and includes performance standards related to 

contract providers’ compliance with VA’s national timeliness goals, CBOC hours of operation, and 

types of clinical providers required to be onsite. We added access to care as a performance 

requirement category for CBOC contracts only because VA policy requires CBOCs to comply with the 

same timeliness goals required of VAMCs.

It is important to ensure that the performance standards included in 

clinical contracts be measurable in order to allow COs and CORs to 

confidently and effectively measure contractors’ performance.

VA has not provided detailed guidance to COs and CORs on how to 

develop performance standards that allow CORs to conduct meaningful 

performance monitoring. VA requires that each contract contain 

appropriate quality assurance standards—including a detailed description 

of the monitoring procedures that the CO and COR will use to evaluate 

the performance of clinical contractors and data collection that will be 

performed.30

Without ensuring that COs and CORs have access to detailed guidance 

on how to construct meaningful performance standards that are both 

clearly stated and measurable, VA cannot ensure that all clinical 

contractors will be monitored appropriately or that these performance 

assessments will be based on the most appropriate measurement of the 

care they provided to veterans in VA facilities.

However, this guidance does not include any information on 

what type of performance standards should be included in a clinical 

contract, does not include examples of effective performance standards, 

and does not define the elements of successful performance standards.

 

30See VA Directive 1663.
 

Page 22 GAO-14-54 VA Clinical Contract Monitoring

During our review of 12 clinical contracts at four VAMCs, CORs reported 

two challenges that may compromise VA’s monitoring of contractors’ 

performance—the heavy workload associated with the COR position and 

the lack of adequate training for CORs.

CORs at the four VAMCs we visited consistently reported facing 

significant challenges in effectively carrying out their COR responsibilities 

for monitoring clinical contractors. Most CORs at the four VAMCs we 

visited reported that they had other primary duties—including managing 

staff—that required them to approach their COR responsibilities as a 

collateral duty. This workload challenge may have led to CORs being 

unable to effectively monitor clinical contractors.

Current VA guidance requires VAMCs to provide CORs with the time to 

complete their responsibilities and ensure that contract compliance is 

managed by a knowledgeable COR. Specifically, VA’s standard operating 

procedure for CORs requires VAMCs to provide CORs with the time and 

resources necessary to complete required training and fulfill their duties 

as a COR. Further, the policy governing clinical contracting at VA states 

that the COR is responsible for ensuring contract compliance and 

specifies that the COR must have sufficient knowledge of the operation of 

the facility and the specific specialty requesting the contract.31 In addition, 

to monitor clinical contracts effectively CORs are required to perform a 

number of key functions according to VA’s standard operating procedure 

for CORs. This guidance requires CORs to submit documentation to the 

CO and carry out a number of responsibilities as part of their monitoring 

efforts. (See table 4.)

 

31See Veterans Health Administration, Standard Operating Procedure: Contracting Officer 

Technical Representative, (May 20, 2011). See also Department of Veterans Affairs 

Directive 1663, Health Care Resources Contracting – Buying Title 38 U.S.C. 8153, 

(August 10, 2006).

CORs Reported that 

Heavy Workloads and 

Inadequate Training 

Make It Difficult to 

Effectively Monitor

Contract Providers’ 

Performance 

CORs Reported Facing 

Heavy Workloads that May 

Compromise VAMC 

Monitoring of Clinical 

Contractors 
 

Page 23 GAO-14-54 VA Clinical Contract Monitoring

Table 4: VA Requirements for Contracting Officer’s Representative (COR) Monitoring of Clinical Contractor Performance

Monitoring task Description

Quarterly contract progress reports These reports contain the COR’s assessment of the contract provider’s performance 

related to patient satisfaction, access to care, timeliness of services, compliance with 

contract requirements and service quality, and compliance with reporting requirements.

Quarterly quality assurance reports These reports contain the COR’s documentation of their monitoring of performance 

standards included in each contract and any clinical quality assurance reviews that were 

conducted by VA clinical staff, such as medical directors.

Quarterly invoice audit reports These reports contain the COR’s certification that all of a contract provider’s invoices were 

verified against records of their actual work hours and any improper invoices were 

returned to the contract provider for correction.

Quarterly contract provider training reports These reports include copies of all required training records certifications, licenses, and 

records of changes in contractor personnel.

Annual past performance evaluations These evaluations are completed by the COR at the end of each contract term in order to 

record the COR’s evaluation of the contractor in several categories, including the quality 

of service provided, contract costs, timeliness of performance, and business relations.

Regularly assisting the contracting officer 

(CO)

CORs are required to assist COs in identifying and resolving issues with the contract 

provider—including identifying delays in contract delivery, analyzing contract provider 

claims and recommending settlement options to the CO, and determining when the CO 

should intervene and begin formal procedures for terminating a contract.

Source: GAO analysis of VA information.

The CORs responsible for monitoring clinical contracts at the VAMCs we 

visited were often serving in this role as a collateral duty and most had 

other primary duties that limited their ability to monitor clinical contracts. 

Thirty-seven of the 40 CORs (93 percent) that completed our data 

collection instrument, administered to all CORs with responsibility for 

clinical contracts at the four VAMCs we visited, stated that the COR 

position is a collateral duty to their primary position. Many of these CORs’ 

primary positions require them to manage staff, maintain budgets, and 

oversee other clinical providers. (See table 5.)
 

Page 24 GAO-14-54 VA Clinical Contract Monitoring

Table 5: A Sample of Position Descriptions of Contracting Officer’s Representatives’ (COR) Primary Positions at the Four VA 

Medical Centers (VAMC) We Visited

Primary position job title Position description

Business manager for a clinical specialty The business manager serves as the principle administrative advisor to the medical 

director of the specialty. This role requires the incumbent to develop and implement 

policies within the specialty and serve as the key staff member for planning, budgeting, 

and developing proposals for the utilization of resources and operations of the specialty. 

This individual’s responsibilities also include supervising staff and conducting workforce 

planning for the specialty.

Administrative officer for a clinical 

specialty

As an administrative officer, the individual is responsible for the management of all 

administrative staff and activities within a specialty. This role requires the incumbent to 

develop and implement policies for the specialty, advise the medical director and VAMC 

management on administrative issues, and develop the budget for the specialty. The 

administrative officer also serves as the primary financial analyst for the specialty, 

supervises the administrative staff working within the specialty, and assists in the 

management of clinical staff working in the specialty.

Medical director for a specialty The medical director is responsible for the management of all clinical and administrative 

staff working within the specialty, implementing policies for the specialty—including 

clinical procedures and requirements mandated by federal and state law, and advising 

VAMC leadership on issues related to the specialty. The medical director is also 

responsible for the specialty’s budget and monitoring expenses and for maintaining the 

specialty’s teaching program for students in applicable clinical training programs.

Source: GAO analysis of VA information.

Based on the results from our data collection instrument administered to 

40 CORs at the four VAMCs we visited, we found that the average COR 

spends about one-quarter of his or her time monitoring approximately 12 

contracts, according to estimates provided by CORs; however, some of 

these CORs were responsible for overseeing significantly more contracts. 

For example, we found that 6 of these 40 CORs managed nearly 190 of 

the 452 (41 percent) contracts in place at the four VAMCs we visited and 

told us they estimated spending at most 30 percent of their work time to 

their COR duties. These 6 CORs were each managing about 20 more 

contracts than the average, in our group of 40 CORs completing the data 

collection instrument, in less time and all 6 of these CORs had other more 

time-consuming primary duties, such as serving as an administrative 

officer for a clinical specialty or as a program manager.

In addition to CORs who responded to our data collection instrument, 

during our review of the 12 contracts for various clinical services in the 

four VAMCs we visited, we found that the CORs responsible for 

managing these contracts frequently did not have the time to effectively 

monitor the performance of contract providers. The majority of these 

CORs said that one of their greatest challenges is not having adequate 

time to fulfill their COR duties and responsibilities and that the demands 
 

Page 25 GAO-14-54 VA Clinical Contract Monitoring

of their primary positions prevented them from fully monitoring clinical 

contractors’ performance. Specifically, CORs for 8 of the 12 contracts 

reported that the demands of their primary positions have at times 

prevented them from fully monitoring contract providers’ performance. In 

addition, CORs for 6 of these 12 contracts stated that they could not 

complete certain elements of their COR responsibilities due to limited 

time and resources. For example, one COR stated that she did not 

adequately monitor costs for the 50 contracts she managed because the 

duties of her primary position demanded significant portions of her work 

day and caused frequent interruptions that did not allow her to focus on 

her duties as a COR. Another COR noted that she only had time to 

monitor contract issues that were a cause for concern or needed to be 

addressed immediately. As a result of this time pressure and the 

demands of their primary positions, CORs reported they did not complete 

several key monitoring functions, including: (1) submitting required 

reports, such as quarterly quality assurance reports; (2) documenting 

contract provider performance issues for annual evaluations; and 

(3) monitoring contract expenditures.

CORs managing 5 of our 12 selected contracts reported that to help 

address challenges presented by time constraints and the demands of 

their primary positions, they often worked extended hours in order to 

complete at least some of their COR duties and manage their day-to-day 

responsibilities. For example, one COR responsible for managing 1 of our 

selected contracts, along with 49 other contracts, said that she worked 

after hours to devote a block of time to her COR work that she could not 

devote during the normal work day due to the responsibilities of her 

primary position. In addition, a few CORs also stated that they believe 

COR duties should be a separate, stand-alone, full-time position rather 

than a collateral duty to ensure that the monitoring of clinical contractors 

and other COR duties receives full attention.

COs responsible for managing the 12 selected contracts echoed what 

CORs told us about the heavy workloads and monitoring challenges 

CORs face. However, none of the COs responsible for the 12 contracts 

we reviewed directly participate in the monitoring of contract providers’ 

performance, and all have delegated this responsibility directly to the 

COR for each clinical contract. Most of the 10 COs responsible for our 12 

selected clinical contracts told us that they also rely on the CORs to 

identify performance issues with contract providers. The majority of these 

COs recognized that CORs face competing demands and heavy 

workloads that may prevent them from effectively monitoring contract 

providers’ performance. Specifically, 7 of the 10 COs responsible for the 
 

Page 26 GAO-14-54 VA Clinical Contract Monitoring

12 selected contracts in our review reported that primary position duties 

and COR duties combined resulted in heavy workloads for CORs. As a 

result, these COs noted that CORs often do not have enough time to 

devote to their COR duties. Half of the COs responsible for the 12 

selected contracts expressed that the COR responsibility should be a fulltime, stand-alone position.

We found that VA’s current guidance related to COR responsibilities does 

not include any information on how VAMCs are to determine the 

feasibility of whether a COR’s workload—including both COR and primary 

position responsibilities—will allow them to carry out their tasks as CORs 

for monitoring contract provider performance. The COR standard 

operating procedure also does not provide any guidance for determining 

when COR duties should be assigned as a collateral duty or a full-time 

responsibility. Without clear guidance on how to determine a COR’s 

workload, VAMCs can unintentionally assign COR duties to a staff 

member who does not have the time available to properly monitor clinical 

contractors.

If CORs’ workloads prevent proper monitoring of clinical contracts, VA 

risks missing the opportunity to proactively identify and correct 

performance issues with contract providers and to recognize patient 

safety concerns potentially resulting from contract providers’ actions. By 

failing to identify performance concerns with contract providers, VA could 

unknowingly be receiving sub-standard service from these contractors, 

continue to receive services from these contract providers that do not 

meet the needs of the VAMCs, and risk patient safety problems when 

these contracts are extended for additional years.

CORs from the four VAMCs we visited noted weaknesses in VA’s COR 

training courses and our own analysis of these courses confirmed these 

limitations. Specifically, over half of the 40 CORs that completed our data 

collection instrument at the four VAMCs we visited responded that either 

their COR training did not prepare them for their role as a COR or were 

neutral on whether or not this training was helpful preparation. In addition, 

CORs for 8 of the 12 contracts we reviewed did not find the required COR 

training helpful or applicable to VA clinical contracting. Several of the 

CORs monitoring the 12 selected contracts and CORs managing other 

contracts at the four VAMCs we visited stated that VA’s required COR 

VA Training Does Not 

Adequately Prepare CORs 

to Monitor Contract 

Providers’ Performance
 

Page 27 GAO-14-54 VA Clinical Contract Monitoring

training is focused on Department of Defense contracting and is not 

tailored to CORs managing clinical contracts at VA.32

VA requires CORs to complete training courses to obtain the Federal 

Acquisition Certification (FAC) for CORs or FAC-COR.

For example, one 

COR stated that the training covered very broad areas of contracts and 

did not include specific information on which kinds of contracts need 

detailed quality assurance plans or information on how to manage a 

clinical contract rather than a supply contract. In addition, a few CORs 

stated that the instructors for their training courses had limited knowledge 

of clinical contracting.

33 These courses 

are developed and offered by VA’s Acquisition Academy and can be 

taken both in person and through instructor-led, web-based courses. 

These courses generally aim to provide individuals with the knowledge 

and tools to carry out the COR responsibilities, which include the 

monitoring of clinical contractors’ performance. Previously we found that 

well-designed training and development programs are linked to both 

agency goals and to the organizational, occupational, and individual skills 

and competencies needed for the agency to perform effectively.34

We reviewed the content of the 32-hour FAC-COR Level II course 

administered by the VA Acquisition Academy and found that this course 

has several limitations in preparing CORs to manage clinical contracts in 

VAMCs.

Focused on contracts that buy goods, not services. The primary 

examples used in the course do not include a discussion of clinical 

contracts at VA and instead walk students through the contracting 

process using examples such as replacing carpet, a large computer 

 

32The Defense Acquisition University develops most acquisition training courses, including 

those for CORs, and federal civilian agencies, such as VA, modify this training to meet 

their agency’s needs. 

33There are three levels of FAC-COR certifications, which directly correlate with the years 

of COR’s contracting experience. Specifically, the FAC-COR Level I certification is an 

8-hour training and does not require previous experience as a COR, the FAC-COR Level 

II certification is 40 hours of training (Level I combined with an additional 32 hours of 

training) and requires 1 year of previous experience serving as a COR, and the FAC-COR 

Level III certification is 60 hours of training and requires 2 years of previous experience 

serving as a COR. 

34GAO, Human Capital: A Guide for Assessing Strategic Training and Development 

Efforts in the Federal Government, GAO-04-546G (Washington, D.C.: March 2004).
 

Page 28 GAO-14-54 VA Clinical Contract Monitoring

equipment purchase, and soup contents and production. There were 

no examples focused on how to evaluate or measure the quality of 

services provided by a contract provider in a VAMC clinical setting. By 

focusing on the types of examples in the course content, CORs did 

not have the opportunity to learn and practice the skills necessary for 

developing and managing service contracts that do not focus on 

easily quantifiable elements and involve complex assessments of the 

performance of clinicians.

Included little information on monitoring responsibilities. The 

course content includes limited information for CORs on post-award 

monitoring responsibilities for clinical contracts and instead is heavily 

weighted to discussing the pre-award development of a contract. By 

failing to incorporate a robust discussion of how CORs should be 

monitoring contract providers in the course, VA has not provided 

these critical personnel with a solid foundation for how to conduct their 

responsibilities for monitoring the performance of contract providers 

serving in clinical roles at VAMCs.

To supplement the required FAC-COR Level II course, VA’s Medical 

Sharing Office recently developed and implemented an 8-hour training 

course for CORs managing clinical contracts. However, VA does not 

currently require this course be completed by all CORs managing clinical 

contracts.35

Without developing required training on clinical contract monitoring, VA 

cannot ensure that all CORs are receiving adequate information on how 

to carry out this critical contract management responsibility and cannot 

ensure that CORs are well trained in how to monitor contract providers. 

This training places VA at risk of having CORs miss performance 

weaknesses by clinical contractors that could affect patient safety, and 

extending clinical contracts for contract providers who do not help VAMCs 

This course covers primarily pre-award contract development 

responsibilities of CORs and does not include any significant information 

on the post-award monitoring responsibilities of CORs managing clinical 

contracts.

 

35In June 2013, the Chief of the Medical Sharing Office reported that VA had developed a 

proposal that makes this training course a requirement for all CORs of clinical contracts 

and submitted it to the agency’s labor relations partners for approval. However, there is no 

target date for completing this review and instituting this requirement.
 

Page 29 GAO-14-54 VA Clinical Contract Monitoring

effectively meet their clinical staffing needs or provide high-quality care to 

veterans.

VA has not established a robust method for overseeing the monitoring of 

clinical contractors by COs and CORs throughout VA’s health care 

system. Standards for internal control in the federal government state that 

agencies should design internal controls that assure ongoing monitoring 

occurs in the course of normal operations, is continually performed, and is 

ingrained in agency operations.36

VA’s primary oversight entity for health care contracting activities, the 

VHA Procurement and Logistics Office, has a limited role in overseeing 

the monitoring actions of COs and CORs once a contract has been 

approved and initiated at a VAMC. The VHA Procurement and Logistics 

Office conducts limited oversight of contracting activities throughout the 

VA health care system through its SAOs and Procurement Operations 

Office.37

Service Area Offices. According to officials from the three SAOs, the 

role of the three SAOs in clinical contract monitoring is limited to an 

audit of the records COs maintain in VA’s electronic Contract 

Management System. These reviews focus only on the completeness 

of COs’ electronic contracting files—including documentation that a 

COR with current training records was assigned to the contract. SAO 

electronic Contract Management System audits do not include any 

reviews of CORs’ monitoring of clinical contractors.

Procurement Operations Office. The Procurement Operations 

Office is the only VHA Procurement and Logistics Office entity 

responsible for overseeing the monitoring activities of CORs; 

however, the reviews conducted by this office are limited to a remote 

 

36See GAO, Internal Control: Standards for Internal Control in the Federal Government, 

GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999). 

37In June 2013, officials from the Medical Sharing Office reported that they are beginning 

to assess whether they can provide oversight to the post-award monitoring of COs and 

CORs; however, these officials noted that they do not currently have the necessary staff 

support to conduct post-award oversight.

VA Central Office 

Oversight of VAMC 

Clinical Contract 

Monitoring is Limited
 

Page 30 GAO-14-54 VA Clinical Contract Monitoring

electronic documentation review of a small sample of COR files.38

Officials from the Procurement Operations Office told us that to select 

COR files for these reviews, a Procurement Operations Office staff 

member aims to select 25 COR files for active contracts per network 

contracting office—at most 2.1 percent of clinical contracts in an 

average VISN if all 25 selected COR files are for clinical contracts.39

VA officials told us that, while the Procurement Operations Office sets 

a goal to review COR files from two network contracting offices each 

month, since implementing the program in March 2013 these reviews 

have been completed in only four network contracting offices and 

none of these four offices have received feedback on the outcomes of 

these reviews as of August 2013.40 Officials added that this limited 

review schedule is due to only one Procurement Operations Office 

staff member being assigned to complete these reviews. These 

reviews also have a narrow focus on the completeness of COR files 

because the Procurement Operations Office staff member reviewing 

the files relies on a checklist to verify the presence or absence of 

required documentation of COR monitoring activities and does not 

review the quality of information contained within CORs records.41

The limited review schedule and narrow focus on file completeness do 

not allow the Procurement Operations Office to comprehensively assess 

the monitoring activities of COs and CORs throughout VA’s health care 

system, as recommended by federal internal control standards. Without a 

 

38Because COR files are not maintained in VA’s electronic Contract Management System, 

the CORs for the contracts selected to be part of these reviews must send copies of their 

files by email to the Procurement Operations Office staff member conducting the review.

39Officials from the Procurement Operations Office told us that the actual number of files 

being reviewed has been typically around 21. COR files selected for these electronic 

documentation reviews may be for any active contract over $250,000 that originates in the 

network contracting office subject to the review. These contracts can include clinical 

contracts, supply contracts, construction contracts, and any other type of active contract.

40An official from the Procurement Operations Office told us the office has set a goal to 

complete a total of 8 to 10 of these electronic documentation reviews of COR files by the 

end of fiscal year 2013.

41The file reviews assess the presence of documentation in seven key areas: (1) COR 

training and delegation; (2) the contract and any modifications made to the contract; 

(3) records of inspections they have completed and any actions taken as a result of these 

inspections; (4) records of technical and financial reports—including copies of invoices 

and purchase orders; (5) copies of all required annual contractor performance reviews and 

security documents; (6) copies of all communications with the contractor and CO; and 

(7) verification that all contract providers have completed required VHA training.
 

Page 31 GAO-14-54 VA Clinical Contract Monitoring

robust monitoring system in place, VA cannot reasonably assure that all 

CORs in all VAMCs are maintaining the proper records of their efforts to 

monitor the activities of clinical contractors caring for veterans.

Clinical contracts serve an important role in helping to ensure that VA can 

provide health care to our nation’s veterans. Contract providers allow 

VAMCs and CBOCs to supplement the capacity of VA-employed 

providers and the ability to provide additional services at these VA 

facilities. However, VA must maintain robust monitoring of these contract 

providers to ensure they provide high quality care to veterans and fulfill 

the responsibilities of their contracts. We identified weaknesses in four 

areas that limit VA’s ability to effectively monitor clinical contractors and 

provide support to VAMC-based CORs and contracting officials 

responsible for conducting these critical reviews.

First, the lack of available tools such as templates for common types of 

clinical contracts requires COs and CORs to rely on less consistent 

sources for constructing critical performance requirements in clinical 

contracts. As a result, most clinical contracts we reviewed were 

insufficiently detailed in their descriptions of contract providers’ 

performance requirements and these performance requirements were not 

consistently applied throughout our sample of clinical contracts. Without 

making tools such as templates available to COs and CORs, VA cannot 

ensure that all clinical contracts include performance requirements in key 

categories necessary to VA’s operations and patient care, such as 

credentialing and privileging and medical record documentation.

Second, VA lacks guidance for COs and CORs that defines the types of 

performance standards and key elements of performance expected to be 

included in clinical contracts. Without access to this type of guidance, VA 

cannot ensure that contracting officials have the ability to establish clear 

and measurable performance standards. Consistency in performance 

measurement and attention to key elements—including clearly stating 

performance standards, and ensuring these performance standards can 

be accurately and consistently measured—is critical to VA’s efforts to 

ensure that all contract providers are monitored appropriately and can be 

held accountable for providing quality patient care.

Third, two significant challenges CORs face in the monitoring of clinical 

contracts—COR workload and training—if left unresolved, will likely have 

continuing negative impacts on the quality of the monitoring that CORs 

are able to perform. Better guidance to VAMCs on how to determine an 

Conclusions 
 

Page 32 GAO-14-54 VA Clinical Contract Monitoring

appropriate COR workload would allow CORs to fulfill key functions 

required to effectively monitor clinical contractors’ performance. Currently, 

VA guidance is silent on how to determine whether or not a potential COR 

has the time to take on the critical responsibilities of monitoring contract 

providers. With the current practice of assigning COR duties to VAMC 

staff as a collateral duty, it is critical that VA define how VAMC staff 

should allocate COR duties to ensure that contracts are monitored 

effectively. In addition, CORs are receiving insufficient training to prepare 

them for monitoring clinical contracts. Currently, CORs for clinical 

contracts cannot apply the current training to their duties because it 

focuses on purchasing supplies rather than on acquiring clinical care 

services. While VA has undertaken efforts to provide supplemental 

training to CORs on clinical contracts, this additional training fails to 

address how CORs should monitor clinical contracts and instead focuses 

on the development of contracts. Without ensuring that COR training 

addresses all of the critical monitoring functions CORs perform, VA 

cannot ensure that all CORs are approaching this role with adequate 

preparation and skills and VA risks failing to identify weaknesses in 

contract providers’ performance that could affect patient safety.

Finally, VA Central Office currently provides limited oversight of clinical 

contract monitoring activities. Without a robust monitoring program that 

includes involvement from VA Central Office, VA does not have 

reasonable assurance that all CORs in VAMCs throughout VA’s health 

care system are properly monitoring clinical contractors and is missing 

the opportunity to identify system-wide limitations in the monitoring of 

these critical contract providers.

To improve the monitoring and oversight of clinical contracts, we 

recommend that the Secretary of Veterans Affairs direct the Under 

Secretary for Health to take the following five actions:

Develop and disseminate tools, such as standard templates, for the 

most common types of clinical contracts in VA’s health care system. 

Such tools should include performance requirement statements 

covering key categories of VA health care policy and guidance—such 

as credentialing and privileging, provider qualifications, and 

expectations for compliance with critical VA policies and medical 

record documentation requirements.

Recommendations for 

Executive Action
 

Page 33 GAO-14-54 VA Clinical Contract Monitoring

Develop and issue guidance on the performance standards that could 

be included in common types of clinical contracts—including CBOC, 

specialty, and temporary clinical provider contracts—to ensure that 

these performance standards are clearly stated in the contracts and

have measurable targets for assessing contract provider performance.

Revise current standard operating procedures for CORs to provide 

guidance on the number of contracts, based on size and complexity, 

each COR should manage to ensure that all CORs maintain a 

workload that allows them to fulfill their duties as a COR and their 

primary position responsibilities.

Modify existing COR training to ensure it includes examples and 

discussion of how to develop and monitor service contracts—including 

contracts for the provision of clinical care in VAMCs.

Increase SAO and VA Central Office oversight of COs and CORs by 

ensuring that post-award contracting files are regularly reviewed for all 

network contracting offices.

VA provided written comments on a draft of this report, which we have 

reprinted in appendix II. In its comments, VA generally agreed with our 

conclusions, concurred with our five recommendations, and described the 

agency’s plans to implement each of our recommendations.

In its comments, VA stated that to address our first and second 

recommendations, VHA’s Medical Sharing Office will work with VHA’s 

Patient Care Services to update and complete templates for clinical 

contracts so that they include performance requirement statements 

covering key categories included in our review.

To address our third recommendation, VA noted that VHA’s Procurement 

Policy and Operations Offices will collaborate to revise existing COR 

standard operating procedures to include guidance on the number of 

contracts, based on size and complexity, that each COR should manage 

to ensure that all CORs maintain a workload that allows them to fulfill their 

duties as a COR and their primary position responsibilities.

To address our fourth recommendation, VA stated that VHA’s Medical 

Sharing Office will provide training to CORs that covers post-award COR 

monitoring responsibilities. We support VA’s efforts to provide this training 

content to CORs, especially those that monitor clinical contracts, but 

Agency Comments
 

Page 34 GAO-14-54 VA Clinical Contract Monitoring

encourage the agency to also modify the content of its existing required 

COR training to ensure that CORs have the adequate preparation and 

skills to effectively monitor contract providers.

Finally, to address our fifth recommendation, VA noted that VHA’s 

Procurement Policy and Operations offices will collaborate to revise 

existing COR standard operating procedures to include Service Area 

Office and CO oversight to ensure that regular reviews take place for 

post-award contract files from all network contracting offices.

We are sending copies of this report to the Secretary of Veterans Affairs, 

appropriate congressional committees, and other interested parties. In 

addition, the report is available at no charge on the GAO website at 

http://www.gao.gov. 

If you or your staffs have any questions about this report, please contact 

me at (202) 512-7114 or at williamsonr@gao.gov. Contact points for our 

Offices of Congressional Relations and Public Affairs may be found on 

the last page of this report. GAO staff who made major contributions to 

this report are listed in appendix III.

Randall B. Williamson

Director, Health Care
Page 35 GAO-14-54 VA Clinical Contract Monitoring

List of Requesters

The Honorable Patty Murray

Chairman

Committee on the Budget

United States Senate

The Honorable Bernard Sanders

Chairman

The Honorable Richard Burr

Ranking Member

Committee on Veterans’ Affairs

United States Senate

The Honorable Jeff Miller

Chairman

The Honorable Michael Michaud

Ranking Member

Committee on Veterans’ Affairs

House of Representatives

The Honorable Mike Coffman

Chairman

Subcommittee on Oversight and Investigations

Committee on Veterans’ Affairs

House of Representatives

The Honorable Bill Johnson

House of Representatives
Appendix I: Objectives, Scope and 

Methodology

Page 36 GAO-14-54 VA Clinical Contract Monitoring

This appendix describes the information and methods we used to 

examine: (1) the extent to which the Department of Veterans Affairs (VA) 

establishes complete performance requirements for clinical contractors; 

(2) the extent to which VA clinical contracts include clear and measurable 

performance standards for assessing whether or not clinical contractors 

met the acceptable quality levels defined in selected contracts; 

(3) challenges VA staff encounter in monitoring clinical contractors’ 

performance; and (4) the extent to which VA Central Office provides 

oversight of VA staff responsible for monitoring clinical contracts.

Specifically, we discuss our methods for selecting VA medical centers 

(VAMC) and network contracting offices to review; identifying appropriate 

VA Central Office Officials to interview; selecting clinical contracts for 

review and assessment;1 assessing the performance requirements 

included in our selected contracts; assessing the performance standards 

included in our selected contracts; administering a data collection 

instrument to contracting officer’s representatives (COR) at our selected 

VAMCs; and evaluating VA’s COR training.

We conducted four site visits to VAMCs to obtain the perspectives of 

VAMC officials responsible for monitoring the day-to-day activities of 

clinical contractors. To identify VAMCs for our site visits, we:

Selected VAMCs that were located in different Veterans Integrated 

Service Networks (VISN) to ensure that our selected VAMCs varied in 

their geographic locations and reported to different VISN management 

officials;

Ensured that our selected VAMCs had a variety of active clinical 

contracts in place to allow for a variety of COR perspectives on 

clinical contractor monitoring; and

Selected at least one VAMC from each of VA’s three Service Area 

Offices (SAO)—the regional contract management entities created to 

 

1

Clinical contracts at VA are used to acquire the services of clinical personnel, such as 

physicians, pharmacists, and nurses. These contracts can be used to fill vacancies for 

clinicians in specialties that are difficult to recruit, supplement existing VAMC capacity by 

providing additional clinicians in high-volume areas where VA also manages a staff of its 

own employees, or fill critical staffing vacancies on a long- or short-term basis.

Appendix I: Objectives, Scope and 

Methodology 

Site Selection 

Methodology and 

Interviews with VAMC and 

Network Contracting 

Office Officials
Appendix I: Objectives, Scope and 

Methodology

Page 37 GAO-14-54 VA Clinical Contract Monitoring

oversee the activities of the 21 network contracting offices and the 

contracting officers (CO) and supervisors that work within them.2

Using these criteria, we selected four VAMCs to visit during our field work 

located in Lebanon, Pennsylvania; Minneapolis, Minnesota; Nashville, 

Tennessee; and Seattle, Washington. During our site visits to these 

locations, we interviewed each VAMC’s leadership team; CORs 

responsible for managing all active clinical contracts; quality management 

staff; and the CORs and medical directors responsible for overseeing the 

clinical contractors for a select sample of clinical contracts. We spoke with 

these officials about a variety of topics—including COR training and 

workload, monitoring procedures for clinical contractors, and their role in 

developing performance requirements and performance standards for 

clinical contracts.

In addition, we spoke with officials from the four network contracting 

offices responsible for managing the contracting activities of our selected 

VAMCs to discuss how COs and management staff manage the 

monitoring of clinical contractors and interact with CORs at VAMCs.

Information obtained from our visits to selected VAMCs and interviews 

with selected network contracting offices cannot be generalized to all 

VAMCs and network contracting offices throughout VA’s health care 

system, but provide important insights.

We also interviewed VA Central Office officials responsible for developing 

policies and procedures for VA clinical contracting activities and 

overseeing the actions of CORs and COs throughout VA’s health care 

system. We spoke with the following offices within the Veterans Health 

Administration (VHA) Procurement and Logistics Office: (1) the Medical 

Sharing Office; (2) the Procurement Audit Office; (3) the Procurement 

Operations Office; and (4) all three SAOs.

 

2

VHA created three SAOs—East, West, and Central—to manage the contracting activities 

of six to eight VISNs each. SAOs report directly to the VHA Procurement and Logistics 

Office in VA Central Office.

Interviews with VA Central 

Office Officials
Appendix I: Objectives, Scope and 

Methodology

Page 38 GAO-14-54 VA Clinical Contract Monitoring

To assess the monitoring of clinical contracts at VA, we reviewed the 

contracts and accompanying COR documentation for a sample of clinical 

contracts from the four VAMCs we visited during our field work.

To select our sample of clinical contracts, we reviewed a list of all active 

clinical contracts for our selected VAMCs provided by the network 

contracting offices responsible for overseeing the contracting actions of 

the selected VAMCs. For each VAMC we visited, we selected three 

contracts to review in detail—one community-based outpatient clinic 

(CBOC) contract, one specialty care contract, and one temporary clinical 

provider contract.3

Table 6: Contract Types Reviewed at Each Selected VA Medical Center (VAMC)

(See table 6.)

Contract type Description

Community-based outpatient clinic 

(CBOC) contracts

These contracts can be used to acquire all the services of a CBOC located in a 

community that is geographically separate from a VAMC—including the physical location 

of the clinic and all required clinical personnel, such as primary care and mental health 

providers. CBOC contracts are typically executed for a 1-year base term with four 1-year 

options to extend the contract. 

Specialty contracts These contracts are used to acquire the services of clinical specialists that work within an 

existing VAMC, such as cardiologists and anesthesiologists. Specialty contracts are 

typically executed for a 1-year base term with four 1-year options to extend the contract. 

Temporary clinical provider contracts These contracts are used to acquire the services of clinical providers on a temporary basis 

and their terms are typically for 1 year or less.

Source: GAO.

To analyze the performance requirements included in our 12 selected 

clinical contracts, we identified six performance requirement categories 

for this analysis by reviewing: (1) VA Acquisition Regulation (VAAR);4

 

3

VA refers to these temporary clinical providers as locum tenens. These temporary clinical 

providers work in a VA facility on a temporary basis in order to fill critical staffing 

vacancies.

 

4

See 48 C.F.R. § 873.109(b), which provides that contracts for health care resources 

should, in most instances, specify qualifications or limitations, such as time limits for 

service delivery, medical certification or credentialing requirements, and small business or 

other socio-economic preference. 

Clinical Contract Selection 

and Analyses

Contract Selection 

Analyses of Performance 

Requirements 
Appendix I: Objectives, Scope and 

Methodology

Page 39 GAO-14-54 VA Clinical Contract Monitoring

(2) VA policy, guidance, and training documents;5 and (3) The Joint 

Commission’s hospital accreditation standards.6

During our assessment of the 12 selected contracts, we analyzed the 

performance requirements included in each clinical contract’s statement 

of work against these six performance requirement categories and 

determined whether or not performance requirements existed that 

matched each categories’ required components. (See table 7.) Results of 

our reviews were recorded as complete, partial, or incomplete based on 

whether the contract’s performance requirements covered all, some, or 

none of the specific components reviewed within each category.

We also verified these 

categories with officials from the Medical Sharing Office, Procurement 

Operations Office, and Procurement Audit Office to ensure they were an 

accurate reflection of performance requirements that should be included 

in VA clinical contracts.

 

5

See VA Acquisition Academy, Acquisition Planning Guide, Version 2, release 1 

(Washington, D.C.: undated); VA Veterans Health Administration, Quality of Medical 

Services Performed Within VA Facilities by Academic Affiliates Under Contract, Directive 

2009-040 (August 31, 2009); Veterans Health Administration, Veterans Health Care 

Service Standards, Directive 2006-041 (June 27, 2006); and Veterans Health 

Administration, VA Outpatient Scheduling Processes and Procedures, Directive 2010-027 

(June 9, 2010).

6

The Joint Commission, Hospital Accreditation Standards 2012: Standards, Elements of 

Performance, Scoring, Accreditation Policies (Oakbrook Terrace, IL: 2012. The Joint 

Commission is an independent organization that accredits and certifies health care 

organizations and programs in the United States.
Appendix I: Objectives, Scope and 

Methodology

Page 40 GAO-14-54 VA Clinical Contract Monitoring

Table 7: Performance Requirement Categories and Required Components

Performance requirement categorya Required components of performance requirements

Type of provider or care Licensure requirements for all clinicians serving under the contract

Description of the specialty area covered under the contract

A listing of key personnel serving under the contract and rules for substituting 

personnel if necessary

Credentialing and privileging Use of VA’s electronic credentialing and privileging system

Requirement for all personnel serving under the contract to be credentialed and 

privileged by VA staff at the initiation of the contract

Requirement for all personnel serving under the contract to have their privileges

renewed at the start of each new contract period

Clinical practice standards Compliance with VA medical staff bylaws and VAMC policies

Adverse event and patient complaint procedures

Compliance with applicable accreditation entity standards

Medical record documentation Entry of pertinent information into VA’s electronic medical record system.

Response to consult requests, referrals, and other communication in VA’s electronic 

medical record system

Rules for the timely entry of information into VA’s electronic medical record system

Business processes Invoicing policies and procedures

Specification of work hours and time card procedures

Personnel security requirements each clinical contractor must uphold

Access to careb Adherence to VA’s scheduling policies and timeliness goals

Specification of clinic hours of operation

Requirements for the types of providers that must be on site for the operation of the 

clinic

Source: GAO (analysis); VA (information); The Joint Commission (information).

a

We identified these performance requirement categories by reviewing the VA Acquisition Regulation 

(VAAR); VA policy, guidance, and training documents; and The Joint Commission’s hospital 

accreditation standards. Officials from the VHA Office of Procurement and Logistics verified these six 

performance requirement categories.

b

We added access to care as a performance requirement category for CBOC contracts because VA 

policy requires CBOCs to comply with the same timeliness goals required of VAMCs.

To analyze the performance standards included in our 12 selected clinical 

contracts, we reviewed whether each contract contained monitoring plan 

provisions that included performance standards in five of the same six 

categories we identified were vital to VA operations during our review of 

performance requirements. We did not include the business processes 

category in our analysis of performance standards because these 

requirements do not require performance standards for their validation 

and assessment. 

Analyses of Performance 

Standards 
Appendix I: Objectives, Scope and 

Methodology

Page 41 GAO-14-54 VA Clinical Contract Monitoring

For the 12 selected clinical contracts, we grouped performance standards 

contained in each contract’s monitoring plan under the five performance 

requirement categories used for this analysis—type of provider or care, 

credentialing and privileging, clinical practice standards, medical record 

documentation, and access to care (for CBOC contracts only).

We analyzed these performance standards using criteria previously 

identified by GAO. Specifically, that performance standards should have 

several attributes—including being clearly stated and using measurable 

targets.7 We assessed each of the 12 selected contracts from the four 

VAMCs we visited to determine if the performance standards included in 

their monitoring plans met these two attributes of successful performance 

standards. To determine whether performance standards were clearly 

stated, two analysts independently determined whether or not each 

performance standard had specified a desired outcome that was clearly 

stated and understandable. To determine whether performance standards 

used measurable targets, two analysts independently determined whether 

or not each performance standard had a numerical target and included 

information on the frequency and calculation method for this target. Each 

analyst entered results as complete, partial, or incomplete for each 

standard and entered a reason for their determination if the result was 

partial or incomplete. A third analyst reviewed these assessments and 

reconciled any differences that arose.

We administered a data collection instrument to the 40 CORs responsible 

for managing a clinical contract at the four VAMCs we visited to 

determine their perspectives on COR training and workload, and to gather 

information on whether or not they were serving as a COR as a collateral 

duty. The data collection instrument included questions on each CORs 

highest training level, assessment of VA COR training, amount of time 

 

7

See: GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing Season 

Performance Measures, GAO-03-143 (Washington, D.C.: November 22, 2002). For the 

purposes of this report, we selected two attributes— clarity and measurability—to evaluate 

VA clinical contracts’ performance standards due to their applicability to performance 

standards for smaller-scale projects. We previously found several other attributes of 

successful performance standards, including: (1) objectivity, (2) reliability, (3) core 

program activities, (4) limited overlap, (5) balance, (6) government-wide priorities, and 

(7) linkage. We did not include these additional seven attributes in our assessment of 

each selected contract’s performance standards because they were designed to measure 

the success of broader agency-wide performance standards rather than those included in 

a single contract. 

Data Collection Instrument 

Administered to CORs
Appendix I: Objectives, Scope and 

Methodology

Page 42 GAO-14-54 VA Clinical Contract Monitoring

devoted to COR duties, and primary position title and description. Each 

COR completed this data collection instrument in the presence of a GAO 

analyst during our field work at each of our four selected VAMCs.

To evaluate VA’s required COR training, three GAO analysts attended an 

offering of VA’s COR training in September 2012. These analysts 

reviewed the content of this training to determine if it met previously 

identified criteria for successful training—that well-designed training and 

development programs are linked to both agency goals and to the 

organizational, occupational, and individual skills and competencies 

needed for the agency to perform effectively.8

 

8

GAO, Human Capital: A Guide for Assessing Strategic Training and Development Efforts 

in the Federal Government, 

To determine if these 

criteria were met, we reviewed the content of this course to determine if it 

included information that was linked to VA’s agency goals and the 

organization, occupational, and individual skills needed for a COR 

monitoring clinical contractors in a VAMC.

GAO-04-546G (Washington, D.C.: March 2004).

Evaluation of VA COR 

Training
Appendix II: Comments from the Department 

of Veterans Affairs

Page 43 GAO-14-54 VA Clinical Contract Monitoring

Appendix II: Comments from the Department 

of Veterans Affairs
Appendix II: Comments from the Department 

of Veterans Affairs

Page 44 GAO-14-54 VA Clinical Contract Monitoring
Appendix II: Comments from the Department 

of Veterans Affairs

Page 45 GAO-14-54 VA Clinical Contract Monitoring
Appendix III: GAO Contact and Staff 

Acknowledgments

Page 46 GAO-14-54 VA Clinical Contract Monitoring

Randall B. Williamson, (202) 512-7114 or williamsonr@gao.gov

In addition to the contact named above, Marcia A. Mann, Assistant 

Director; A. Elizabeth Dobrenz; Pamela Dooley; Carolyn Garvey; 

Katherine Nicole Laubacher; Laurie Pachter; Said Sariolghalam; 

Jennifer Whitworth; and Malissa G. Winograd made key contributions 

to this report.

Appendix III: GAO Contact and Staff 

Acknowledgments 

GAO Contact

Staff 

Acknowledgments 

(291006)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (http://www.gao.gov). Each weekday 

afternoon, GAO posts on its website newly released reports, testimony, 

and correspondence. To have GAO e-mail you a list of newly posted 

products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube. 

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. 

Visit GAO on the web at www.gao.gov. 

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations 

Public Affairs

Please Print on Recycled Paper.