VA Health Care: Additional Guidance, Training, and Oversight Needed to Improve Clinical Contract Monitoring
Published: 2013-10-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: VA must frequently contract with non-VA health care providers so that clinical providers are available to meet veterans' health care needs. While recent studies have disclosed problems with VA's development of contracts for clinical services, there has been little scrutiny of how VA monitors and evaluates the care contract providers give to veterans.
GAO was asked to review VA's efforts to monitor clinical contractors working in VA facilities. This report examines the extent to which VA establishes complete performance requirements for contract providers, challenges VA staff encounter in monitoring contract providers' performance, and the extent to which VA oversees VAMC staff responsible for monitoring contract providers.
GAO reviewed VA acquisition regulations and other guidance. In addition, GAO visited four VAMCs that varied in geographic location and selected a nongeneralizable sample of three types of clinical contracts from each of the four VAMCs to review. GAO discussed how VAMC and VISN staff monitor and oversee these contracts and reviewed contract monitoring documentation.
section_title: What GAO Found
paragraphs: All 12 contracts GAO reviewed from the four Department of Veterans Affairs' (VA) medical centers (VAMC) visited contained performance requirements consistent with VA acquisition policy. However, the performance requirements lacked detail in six categories: type of provider or care; credentialing and privileging; clinical practice standards; medical record documentation; business processes; and access to care. GAO identified these categories from reviews of VA acquisition regulations, VA policies, and hospital accreditation standards; and VA officials verified that these six categories were an accurate reflection of performance requirements that should be in VA clinical contracts. GAO found, for example, one VAMC cardiothoracic contract that had detailed performance requirements while another VAMC's cardiothoracic contract did not contain a statement describing the contract provider's responsibilities for reporting and responding to adverse events and patient complaints. GAO also found that contracting officials lack tools, such as standard templates, that provide examples of the performance requirements that should be included in common types of clinical contracts. Such tools would help ensure consistency in requirements across contracts.
Contracting officer's representatives (COR) cited two main challenges in monitoring contract providers' performance--too little time to monitor clinical contractors' performance effectively and inadequate training. Most of the 40 CORs at the four VAMCs in GAO's review said that their clinical contract monitoring duties were a collateral duty and that they had other primary responsibilities, such as serving as a business manager or administrative officer for a specialty clinic within the VAMC. GAO found that, on average, each of these 40 CORs spent about 25 percent of their time monitoring an average of 12 contracts. CORs said the demands of their primary positions at times prevented them from fully monitoring contract providers' performance. Further, VA's current guidance related to COR responsibilities does not include any information on how VAMCs are to determine the feasibility of whether a COR's workload--including both COR and primary position responsibilities--will allow them to carry out their tasks as CORs for monitoring contract provider performance. GAO also found that current VA COR training programs focus on contracts that buy goods, not clinical services, and include little information on monitoring responsibilities. CORs questioned the usefulness of the COR training VA uses to prepare them for monitoring clinical contracts.
VA Central Office conducts limited oversight of COR and contract monitoring activities. VA Central Office reviews of COR clinical contractor monitoring activities are limited to a small number of annual file reviews that focus on verifying the presence of required documentation only and do not assess the quality of CORs' monitoring activities. Since implementing the program in March 2013 these reviews have been conducted in 4 of 21 network contracting offices and as of August 2013 none of the 4 offices has received feedback on these reviews. Without a robust monitoring system, VA cannot ensure that all CORs in its VAMCs are properly monitoring, evaluating, and documenting the performance of contract providers caring for veterans.
section_title: What GAO Recommends
paragraphs: GAO recommends that VA develop and disseminate standard templates that provide examples of performance requirements for clinical contracts, revise guidance for CORs to include workload information, modify COR training, and improve the monitoring and oversight of clinical contracts. VA concurred with GAO's recommendations.

section_title: Letter
paragraphs: The Department of Veterans Affairs (VA) employs the majority of the  clinical providers that care for veterans in VA-operated facilities, such as  VA medical centers (VAMC) and community-based outpatient clinics  (CBOC). However, in order to meet the needs of the veterans it serves  and ensure that clinical providers are available to treat veterans, VA  frequently contracts with non-VA health care providers to provide services  in VA facilities. According to VA, every VAMC has at least one contract in  place with a non-VA health care provider to help supplement the number  of providers employed by VA facilities. These non-VA health care  providers either provide services that the VAMC or CBOC do not currently  offer or supplement the capacity of the VA facility by providing additional  clinicians to treat veterans.
Previous studies have highlighted challenges VA has faced developing  and administering its clinical contracts. In recent years, for example, the  VA Office of the Inspector General (OIG) highlighted challenges VA faces  in developing its clinical contracts and found systemic weaknesses in the   These weaknesses were attributed  process VA uses to award contracts.to VA’s decentralized oversight of the initial stages of the contracting  process before a contract is awarded to a contractor. However, there  have been no comprehensive reviews of VA’s efforts to monitor clinical  contractors once a contract is awarded and contract providers begin  caring for veterans in VA facilities. You asked us to review VA’s efforts to  monitor clinical contractors working in VA facilities. In this report we  examine: (1) the extent to which VA establishes complete performance  requirements for clinical contractors; (2) the extent to which VA clinical  contracts include clear and measurable performance standards for  assessing whether or not clinical contractors met the acceptable quality  levels defined in selected contracts; (3) challenges VA staff encounter in  monitoring clinical contractors’ performance; and (4) the extent to which  VA Central Office provides oversight of VA staff responsible for  monitoring clinical contracts.
To examine the extent to which VA establishes complete performance  requirements for clinical contractors, we reviewed applicable VA  Acquisition Regulations (VAAR), policies, and handbooks to determine  what should be included in clinical contracts’ performance requirement  statements. We also interviewed officials from VA Central Office involved  in the oversight of acquisitions at VAMCs to discuss their roles in  managing the staff responsible for the day-to-day oversight of clinical  contractors—including the Medical Sharing Office. In addition, we  conducted site visits to four VAMCs and reviewed how they monitor and  oversee clinical contractors. Each of the four VAMCs was located in a  different Veterans Integrated Service Network (VISN) and had several  types of clinical contracts in place at their facility. These four VAMCs  were located in Lebanon, Pennsylvania; Nashville, Tennessee;  Minneapolis, Minnesota; and Seattle, Washington. We also spoke with  officials from the four network contracting offices responsible for  administering and executing all contracts for these four VAMCs about  how their staff manage these clinical contracts and how they interact with  VAMC staff responsible for the daily monitoring of clinical contractors.  Information obtained from these VAMCs and network contracting offices  cannot be generalized to all VAMCs and network contracting offices.  Finally, we reviewed three contracts from each of the four VAMCs we  visited—including one contract for a CBOC, one contract for the services  of a specialist, and one contract for the services of a temporary clinical  provider.
To analyze the performance requirements included in the 12 selected  clinical contracts, we identified six performance requirement categories by  reviewing: (1) the VAAR; (2) VA policy, guidance, and training  documents;  and (3) The Joint Commission’s hospital accreditation  standards. We verified these categories with officials from the Veterans  Health Administration (VHA) Procurement and Logistics Office to ensure  they were an accurate reflection of performance requirements that should  be included in VA clinical contracts. The six categories included in our  analysis are: (1) type of provider or care; (2) credentialing and privileging;  (3) clinical practice standards; (4) medical record documentation;   (5) business processes; and (6) access to care.selected clinical contracts to determine if they included all, at least one, or  none of the required components of these categories.
To examine the extent to which VA clinical contracts include clear and  measurable performance standards for assessing whether or not clinical  contractors met the acceptable quality levels defined in selected  contracts, we reviewed the performance standards from the 12 selected  contracts. To complete this review, we identified whether the contracts  contained performance monitoring plans that included performance  standards in five of the same six categories vital to VA operations we  previously identified. category to determine if they included all, some, or none of the required  components of the category. We also interviewed contracting officers  (CO) and contract supervisors from the four network contracting offices  that are responsible for administering and executing the contracts from  the four VAMCs we visited regarding the use and development of  performance standards within clinical contracts. Finally, we spoke with  contracting officers’ representatives (COR) and medical directors  responsible for overseeing the clinical contractors for the12 selected  contracts about their experiences using performance standards within  clinical contracts to assess contractor performance.
VA Directive 1663 requires all clinical contracts to have monitoring plans that ensure the  appropriate quality assurance standards and data methods are in place, collection of  these data is performed, and the performance of the clinical contractor is monitored. We  did not include the business processes category in our assessment of performance  standards because the administrative requirements covered by this category, such as  invoicing procedures and time care submission, do not require performance standards for  their validation. clinical contracts in the four VAMCs we visited. These structured  interviews and the data collection instrument were designed to capture  information about various aspects of CORs’ tasks and responsibilities— including how many contracts CORs managed, whether or not serving as  a COR was the individual’s primary position or a collateral duty, and  experiences with COR training required by VA. Results from this data  collection instrument cannot be generalized to all CORs throughout VA’s  health care system, but provide important insights. In addition, we  interviewed the CORs and medical directors responsible for monitoring  the 12 contracts we reviewed about their experience monitoring these  contracts. Finally, we reviewed VA’s COR training courses to assess their  content and structure.
To examine the extent to which VA Central Office provides oversight of  VA staff responsible for monitoring clinical contracts, we interviewed VA  Central Office officials responsible for overseeing the acquisition process,  including officials from the three regional Service Area Offices (SAO) and  the VHA Procurement and Logistics Office, about their role in overseeing   We also reviewed tools used by these  COs and VAMC-based CORs.offices to conduct oversight of COs and CORs, such as checklists and  procedure guides. In addition, we spoke with COs and CORs responsible  for managing and monitoring the 12 contracts we reviewed about their  experience working with these VA Central Office officials. For additional  details about the scope and methodology used in this report, see  appendix I.
We conducted this performance audit from October 2012 to October 2013  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Clinical contracts at VA are used to acquire the services of clinical  personnel, such as physicians, pharmacists, and nurses. These contracts  can be used to fill vacancies for clinicians in specialties that are difficult to  recruit, supplement existing VAMC capacity by providing additional  clinicians in high-volume areas where VA also manages a staff of its own  employees, or fill critical staffing vacancies on a long- or short-term basis.
Clinical contracting at VA is governed by two sets of regulations—the  FAR and VAAR—that contain general requirements for all VA  acquisitions. The FAR contains government-wide regulations and  establishes uniform policies and procedures used by all executive branch  federal agencies for their acquisitions. VA supplements the FAR with the  VAAR to establish uniform policies and procedures for all VA acquisitions  of supplies and services. In addition to the VAAR, VA provides guidance  to its acquisition workforce in the form of policy directives and handbooks.
subsections: section_title: Acquisition Workforce at VA
paragraphs: Both acquisition and clinical staff at VA work together to plan, execute,  and monitor clinical contracts at VA. On the acquisition side, COs are  responsible for planning, awarding, and administering contracts on behalf  of the federal government. Each CO is able to obligate federal funds up to  a specified limit and a CO must formally approve all clinical contracts at  VA. Common tasks of a CO include developing acquisition planning  documents used to begin a clinical contract, conducting market research  to determine pricing and availability for a clinical contract, and completing  the formal competitive or non-competitive solicitation process for  contracts. Each CO works within a network contracting office and is  overseen by managers within that office who report directly to VA Central  Office. There are 21 network contracting offices throughout VA’s health  care system that manage all the contracting activities of a single VISN.
Two types of VAMC staff have monitoring responsibilities for clinical  contracts—CORs and medical directors. For each VA clinical contract,  the CO responsible for the contract designates a COR at the VAMC to  help develop the clinical contract and monitor the contract provider’s  performance once the provider begins work. Common tasks delegated to  the COR include providing input on the performance requirements for the  clinical contract, determining how the contract provider’s performance will  be measured and monitoring performance once work has begun,  validating the contract provider’s invoices to ensure their accuracy,  managing contract modifications, and assisting the CO in resolving any  issues that may arise with the contract provider. At VA, CORs are  commonly administrative personnel responsible for managing the  operations of a specialty care line at a VAMC—such as primary care,  surgery, etc.—where the contractor will be working.
Medical directors in various specialty care lines often assist CORs in  monitoring contract provider performance because CORs lack the  expertise to evaluate a contract provider’s clinical abilities. Medical  directors are responsible for overseeing the clinical care provided by all  providers within a specialty care line, including VA-employed and contract  clinicians. Medical directors use existing VA processes, including the  credentialing and privileging process, to monitor contract providers. This  clinical monitoring includes an initial assessment of the clinician’s  competency during the first 90 days after the contract provider begins  working at a VA facility and typically includes evaluations in several  areas—such as patient care, clinical knowledge, and interpersonal  communication skills. Upon successful completion of the initial  competency evaluation, medical directors monitor contract providers  through ongoing evaluations of their performance according to the  specific performance standards used to evaluate all clinicians working at  VA within their specialty.
Medical directors provide the results of these initial and ongoing  evaluations to the COR and work with the COR to resolve any issues that  may arise as a result of a contract provider’s clinical performance. CORs  are responsible for maintaining the official record of the contract  provider’s performance and providing official performance assessments  to the CO.
section_title: VA Acquisition Oversight Structure
paragraphs: VA Central Office has primary responsibility for overseeing network  contracting offices and manages clinical contracting activities through the  VHA Procurement and Logistics Office. There are five primary offices  within the VHA Procurement and Logistics Office that are responsible for  overseeing various aspects of clinical contracting activities and report to  VHA’s Deputy Chief Procurement Officer. (See fig. 1.)
Medical Sharing Office. The Medical Sharing Office is responsible for  providing guidance to network contracting offices regarding the content  and structure of solicitations for clinical contracts and for reviewing  several types of clinical contracts. The Medical Sharing Office reviews  solicitations of all competitive clinical contracts valued at over $1.5 million,  all non-competitive clinical contracts valued at over $500,000, and all  organ transplant contracts. All Medical Sharing Office reviews are  conducted before a solicitation is issued to ensure that all the necessary  provisions are in place prior to any competition or award.
Procurement Operations Office. The Procurement Operations Office is  responsible for providing ongoing guidance and monitoring of the COR  population at VA. The Procurement Operations Office conducts reviews  of COR files and publishes a COR newsletter.
Procurement Audit Office. The Procurement Audit Office is responsible  for ensuring compliance with VA policies and procedures related to  contracting. This office conducts internal compliance audits of contracts,  including clinical contracts, once they are executed to ensure that all  required documentation was included in the final contract and audits the  activities of network contracting offices and SAOs to ensure their  compliance with VA policies and regulations.
Procurement Policy Office. The Procurement Policy Office is  responsible for providing guidance to VA’s acquisition workforce in  network contracting offices and SAOs. This office produces and updates  standard operating procedures for CORs and COs.
Service Area Offices. SAOs are the regional contract management  entities created to oversee the activities of the 21 network contracting  offices and the COs and supervisors that work within them. VHA created  three SAOs—East, West, and Central—to manage the contracting  activities of six to eight VISNs each. SAOs review solicitations for most  clinical contracts during their initial stages to ensure that all necessary  provisions are in place prior to any competition or award.
section_title: Reviewed Contracts Included Performance Requirements, but Lacked Complete Detail Needed to Hold Contractors Accountable in Some Cases
paragraphs: We found that all 12 contracts we reviewed from the four VAMCs we  visited contained performance requirements consistent with VA  acquisition policy. However, we found that 10 of the 12 contracts we  reviewed lacked detailed descriptions of contractors’ performance  requirements in one or more of the six categories we assessed. We  analyzed the content of these contracts using six performance  requirement categories we established through reviews of the VAAR, VA  policies, and the standards of the leading hospital accreditation  organization. (See fig. 2)
We also found that the level of detail in contract performance  requirements contained in the 12 contracts we reviewed varied both by  the type of contract—CBOC contract, specialty care contract, or  temporary clinical provider contract—and by the six performance  requirement categories we assessed. (See table 1.)
The CBOC contracts we reviewed were the most detailed, followed by  specialty care contracts and temporary clinical provider contracts.
CBOC contracts. Contracts for CBOCs generally contained the most  detailed performance requirements. For example, all four of the CBOC  contracts we reviewed had complete performance requirements for three  of the six applicable categories—the type of provider or care, clinical  practice standards, and medical record documentation. However, two of  the four CBOC contracts—at the Minneapolis and Seattle VAMCs— lacked detail in one or more of the performance requirement categories  we reviewed. For example, the Seattle VAMC CBOC contract was the  only CBOC contract we reviewed that did not include a required set of  operating hours for the contracted facility, which was a component we  assessed within the access to care category.
Specialty care contracts. The amount of detail in performance  requirements within specialty care contracts varied across our four  selected VAMCs. Overall, performance requirements in these contracts  lacked specificity in key categories. For example, two of our selected  specialty care contracts—for the Seattle and Minneapolis VAMCs—were  for the services of cardiothoracic surgeons and had very similar results in  our review, except for their descriptions of performance requirements  within the clinical practice standards category. For this category, the  Seattle VAMC cardiothoracic contract had detailed performance  requirements, but the Minneapolis VAMC cardiothoracic contract did not  contain a statement describing the provider’s responsibilities for the  reporting of and response to adverse events and patient complaints, a component we assessed within the clinical practice standards category.In addition, the Nashville VAMC specialty care contract for the services of  a psychiatrist had limited detail because it did not contain requirements  for the clinical contractor’s privileges to be renewed at the beginning of  each new contract term, and received a partial rating in the medical  record documentation category because the contract did not contain  performance requirements for the timely entry of information into VA’s  electronic medical record.
Temporary clinical provider contracts. We found that temporary  clinical provider contracts included the least detailed performance  requirements among the three contract types we reviewed. For example,  the Lebanon VAMC temporary clinical provider contract for a gynecologist  was missing descriptions of requirements for providers to undergo VA’s  credentialing and privileging process, comply with The Joint Commission  requirements that apply to gynecology, and enter relevant data into  veterans’ electronic medical records. These specific requirements are  components of three separate categories we reviewed—credentialing and  privileging, clinical practice standards, and medical record documentation.  In addition, the Seattle VAMC temporary clinical provider contract for a  pharmacist was missing information on contractor requirements related to  both the clinical practice standards and medical record documentation  categories. According to VA officials, temporary clinical provider contracts  are often written using a format that is designed to allow for shorter and  faster contract award timeframes; however, this shorter format likely  contributed to less complete performance requirement statements in the  contracts we reviewed.
We also found the level of detail in contract performance requirements  varied significantly by performance requirement category, with a majority  of reviewed contracts missing some component of the credentialing and  privileging performance requirement category.
Credentialing and privileging. This category included a review of three  specific components of performance requirements: (1) requirements for  all contract providers to initially undergo VA’s credentialing and privileging  process; (2) the use of VA’s electronic credentialing system for that   In our  process; and (3) requirements for the renewal of privileges.review, we found that 10 of the 12 contracts were missing performance  requirements from one or more of these three components. Six of the 10  contracts did not contain statements describing the rules for renewal of  privileges, which is part of the credentialing and privileging process at VA.  Privileges are required to be renewed every 2 years by VA policy. The  term of a contract provider’s privileges may not extend beyond the term of  the contract regardless of whether the VAMC intends to renew or extend  the contract; this means that privileges need to be renewed more often  than every 2 years for contract providers depending on the length of the  contract. In addition, 5 of the 10 contracts did not contain statements  requiring the use of VA’s electronic credentialing system for the  credentialing process.
Type of provider or care. This performance requirement category had  three components: (1) provider licensure requirements, (2) a description  of the specialty area care was contracted for, and (3) a listing of key  personnel providing services under the contract. contracts we reviewed included detailed performance requirements in this  category, with the exception of the Lebanon VAMC temporary clinical  provider contract, which did not include the name of the gynecologist  selected for the contract or rules for the substitution of another provider if  needed.
Business processes. This performance requirement category contained  three components: (1) invoicing policies; (2) on- and off-duty hours and  time card submission; and (3) personnel security responsibilities, such as  background checks. Eleven of the 12 contracts included detailed  performance requirements in this category, with the exception of the  Seattle VAMC CBOC contract, which did not list on- and off-duty hours for  staff, but included all other business process components.
See Department of Veterans Affairs, Health Care Resources Contracting—Buying, Title  38 U.S.C. 8153, VA Directive 1663 (Aug. 10, 2006). Directive 1663 states that contracts  for full-time providers must identify the key personnel proposed to provide the service and  their qualifications. standard templates for clinical contracts. VA Central Office officials with  whom we spoke said that there is a template for CBOC contracts that has  been distributed among contracting officials and that templates for six  common specialty care contracts, including cardiology and  anesthesiology, are currently in development as of August 2013, and that  VA’s progress in developing and implementing these templates is  dependent upon VA Central Office and the VA OIG agreeing on the six  specialties that will have templates developed.
Absent available tools, such as standard templates for common types of  contracts—including those for CBOCs, specialty care, and temporary  clinical providers—VA cannot reasonably ensure that critical requirements  for contract providers’ performance are consistently included in VA clinical  contracts and are standardized across VAMCs. Without assurance that  these critical performance requirements are included in clinical contracts,  VA may not be able to hold clinical contractors accountable for providing  the high-quality services VAMCs need to serve veterans.
section_title: Selected VA Clinical Contracts Did Not Always Include Clear and Measurable Performance Standards
paragraphs: Two of the 12 clinical contracts we reviewed did not include any  performance standards and 7 contracts did not include performance  standards in key categories. Additionally, the performance standards  included within the 12 clinical contracts we reviewed were not always  stated clearly and did not always include measurable targets. We found  that VA did not provide guidance to COs and CORs on how to develop  performance standards that are clear and measurable for determining  whether or not clinical contractors met the acceptable quality levels  defined in selected contracts.
The FAR requires agencies to ensure that requirements for services are  clearly defined and appropriate performance standards are developed so  that the agency’s requirements can be understood by potential offerors  and that performance in accordance with contract terms and conditions   In addition, agencies, in order to  will meet the agency’s requirements.successfully measure the performance of their operations, use  performance standards that demonstrate results, are limited to a vital few  activities, and provide useful information for decision-making. To  determine whether performance standards used by agencies meet these  qualities, we previously found that performance standards should have  several attributes—including being clearly stated and using measurable  targets. We assessed each of the 12 selected contracts from the four  VAMCs we visited to determine if the performance standards for five of  the six categories we previously identified included these two attributes of  successful performance standards.
Performance standards were not clearly stated. Two of the 12 clinical  contracts we reviewed—the Lebanon VAMC CBOC and temporary  clinical provider contracts—did not include any monitoring plan provisions  defining the performance standards against which the VAMC would  assess the contractor’s performance. Performance standards included in  the monitoring plans of the remaining 10 clinical contracts we reviewed  lacked clarity in what would be used to determine a clinical contractor’s  performance. For example, the Nashville VAMC CBOC contract includes  a performance standard in the clinical practice standards category that  requires the contractor to meet a specific target for three of four outpatient  satisfaction survey scores. However, VA’s outpatient satisfaction survey  includes significantly more than four elements and the contract’s  performance standard does not clearly specify which four survey  elements will be used for this analysis. In another instance, the Seattle  VAMC temporary clinical provider contract includes a performance  standard for prescription dispensing that includes multiple standards that  cover numerous tasks related to prescription processing and validation in  inpatient and outpatient settings. By including multiple parts in a single  performance standard, it is unclear what the COR will actually be  measuring to determine the clinical contractor’s performance in this area.  As table 2 shows, none of the 12 contracts we assessed included clearly  stated performance standards in all five categories.
It is important to ensure that performance standards included in clinical  contracts are clearly stated to minimize confusion during the monitoring of  clinical contractors’ performance. Without clearly articulated performance  standards, COs and CORs may not be able to effectively assess the  performance of clinical contractors and VA may not be able to hold  contractors accountable for poor performance.
Performance standards did not include measurable targets. We  found that there were no categories that included fully measurable  performance standards in the 12 contracts we assessed. As previously  noted, 2 of the 12 contracts—the Lebanon VAMC CBOC and temporary  clinical provider contracts—did not include any monitoring plan provisions  defining the performance standards against which the VAMC would  assess the contractor’s performance. Of the remaining 10 contracts with  monitoring plans, only 1 contract—the Lebanon VAMC specialty  contract—included fully measurable performance standards in all  categories, and the other 9 contracts either did not include performance  standards for a critical category or had performance standards that were  only partially measurable in at least one category. (See table 3.) For  example, the Nashville VAMC temporary clinical provider contract  included a performance standard in the clinical practice standards  category that stated that the contractor should have no more than two  patient or staff complaints. However, this contract did not define how  frequently this target would be measured and therefore it is not possible  to determine whether this threshold applies to complaints on a monthly,  quarterly, or annual basis. This limited information on how often the  complaints would be reviewed and measured for the clinical contractor  makes it difficult to determine what a successful performance outcome  would be. In another instance, the Minneapolis VAMC specialty contract  includes several performance standards in the credentialing and  privileging category that indicate the clinical contractor is responsible for  maintaining medical and clinical knowledge and engaging in practiced- based learning. However, this contract does not include information on  how the clinical contractor’s performance in these areas would be  assessed and does not specify a clear numerical target for determining  whether or not these standards have been fulfilled. Finally, the Nashville  VAMC specialty contract includes a performance standard in the medical  record documentation category that states the clinical contractor should  maintain documentation in VA’s electronic medical record system of 95  percent, but does not include a description of how the COR would assess  compliance with this performance standard.
It is important to ensure that the performance standards included in  clinical contracts be measurable in order to allow COs and CORs to  confidently and effectively measure contractors’ performance.
VA has not provided detailed guidance to COs and CORs on how to  develop performance standards that allow CORs to conduct meaningful  performance monitoring. VA requires that each contract contain  appropriate quality assurance standards—including a detailed description  of the monitoring procedures that the CO and COR will use to evaluate  the performance of clinical contractors and data collection that will be  performed. However, this guidance does not include any information on  what type of performance standards should be included in a clinical  contract, does not include examples of effective performance standards,  and does not define the elements of successful performance standards.
Without ensuring that COs and CORs have access to detailed guidance  on how to construct meaningful performance standards that are both  clearly stated and measurable, VA cannot ensure that all clinical  contractors will be monitored appropriately or that these performance  assessments will be based on the most appropriate measurement of the  care they provided to veterans in VA facilities.
section_title: CORs Reported that Heavy Workloads and Inadequate Training Make It Difficult to Effectively Monitor Contract Providers’ Performance
paragraphs: During our review of 12 clinical contracts at four VAMCs, CORs reported  two challenges that may compromise VA’s monitoring of contractors’  performance—the heavy workload associated with the COR position and  the lack of adequate training for CORs.
subsections: section_title: CORs Reported Facing Heavy Workloads that May Compromise VAMC Monitoring of Clinical Contractors
paragraphs: CORs at the four VAMCs we visited consistently reported facing  significant challenges in effectively carrying out their COR responsibilities  for monitoring clinical contractors. Most CORs at the four VAMCs we  visited reported that they had other primary duties—including managing  staff—that required them to approach their COR responsibilities as a  collateral duty. This workload challenge may have led to CORs being  unable to effectively monitor clinical contractors.
Current VA guidance requires VAMCs to provide CORs with the time to  complete their responsibilities and ensure that contract compliance is  managed by a knowledgeable COR. Specifically, VA’s standard operating  procedure for CORs requires VAMCs to provide CORs with the time and  resources necessary to complete required training and fulfill their duties  as a COR. Further, the policy governing clinical contracting at VA states  that the COR is responsible for ensuring contract compliance and  specifies that the COR must have sufficient knowledge of the operation of   In addition,  the facility and the specific specialty requesting the contract.to monitor clinical contracts effectively CORs are required to perform a  number of key functions according to VA’s standard operating procedure  for CORs. This guidance requires CORs to submit documentation to the  CO and carry out a number of responsibilities as part of their monitoring  efforts. (See table 4.)
The CORs responsible for monitoring clinical contracts at the VAMCs we  visited were often serving in this role as a collateral duty and most had  other primary duties that limited their ability to monitor clinical contracts.  Thirty-seven of the 40 CORs (93 percent) that completed our data  collection instrument, administered to all CORs with responsibility for  clinical contracts at the four VAMCs we visited, stated that the COR  position is a collateral duty to their primary position. Many of these CORs’  primary positions require them to manage staff, maintain budgets, and  oversee other clinical providers. (See table 5.)
Based on the results from our data collection instrument administered to  40 CORs at the four VAMCs we visited, we found that the average COR  spends about one-quarter of his or her time monitoring approximately 12  contracts, according to estimates provided by CORs; however, some of  these CORs were responsible for overseeing significantly more contracts.  For example, we found that 6 of these 40 CORs managed nearly 190 of  the 452 (41 percent) contracts in place at the four VAMCs we visited and  told us they estimated spending at most 30 percent of their work time to  their COR duties. These 6 CORs were each managing about 20 more  contracts than the average, in our group of 40 CORs completing the data  collection instrument, in less time and all 6 of these CORs had other more  time-consuming primary duties, such as serving as an administrative  officer for a clinical specialty or as a program manager.
In addition to CORs who responded to our data collection instrument,  during our review of the 12 contracts for various clinical services in the  four VAMCs we visited, we found that the CORs responsible for  managing these contracts frequently did not have the time to effectively  monitor the performance of contract providers. The majority of these  CORs said that one of their greatest challenges is not having adequate  time to fulfill their COR duties and responsibilities and that the demands  of their primary positions prevented them from fully monitoring clinical  contractors’ performance. Specifically, CORs for 8 of the 12 contracts  reported that the demands of their primary positions have at times  prevented them from fully monitoring contract providers’ performance. In  addition, CORs for 6 of these 12 contracts stated that they could not  complete certain elements of their COR responsibilities due to limited  time and resources. For example, one COR stated that she did not  adequately monitor costs for the 50 contracts she managed because the  duties of her primary position demanded significant portions of her work  day and caused frequent interruptions that did not allow her to focus on  her duties as a COR. Another COR noted that she only had time to  monitor contract issues that were a cause for concern or needed to be  addressed immediately. As a result of this time pressure and the  demands of their primary positions, CORs reported they did not complete  several key monitoring functions, including: (1) submitting required  reports, such as quarterly quality assurance reports; (2) documenting  contract provider performance issues for annual evaluations; and   (3) monitoring contract expenditures.
CORs managing 5 of our 12 selected contracts reported that to help  address challenges presented by time constraints and the demands of  their primary positions, they often worked extended hours in order to  complete at least some of their COR duties and manage their day-to-day  responsibilities. For example, one COR responsible for managing 1 of our  selected contracts, along with 49 other contracts, said that she worked  after hours to devote a block of time to her COR work that she could not  devote during the normal work day due to the responsibilities of her  primary position. In addition, a few CORs also stated that they believe  COR duties should be a separate, stand-alone, full-time position rather  than a collateral duty to ensure that the monitoring of clinical contractors  and other COR duties receives full attention.
COs responsible for managing the 12 selected contracts echoed what  CORs told us about the heavy workloads and monitoring challenges  CORs face. However, none of the COs responsible for the 12 contracts  we reviewed directly participate in the monitoring of contract providers’  performance, and all have delegated this responsibility directly to the  COR for each clinical contract. Most of the 10 COs responsible for our 12  selected clinical contracts told us that they also rely on the CORs to  identify performance issues with contract providers. The majority of these  COs recognized that CORs face competing demands and heavy  workloads that may prevent them from effectively monitoring contract  providers’ performance. Specifically, 7 of the 10 COs responsible for the  12 selected contracts in our review reported that primary position duties  and COR duties combined resulted in heavy workloads for CORs. As a  result, these COs noted that CORs often do not have enough time to  devote to their COR duties. Half of the COs responsible for the 12  selected contracts expressed that the COR responsibility should be a full- time, stand-alone position.
We found that VA’s current guidance related to COR responsibilities does  not include any information on how VAMCs are to determine the  feasibility of whether a COR’s workload—including both COR and primary  position responsibilities—will allow them to carry out their tasks as CORs  for monitoring contract provider performance. The COR standard  operating procedure also does not provide any guidance for determining  when COR duties should be assigned as a collateral duty or a full-time  responsibility. Without clear guidance on how to determine a COR’s  workload, VAMCs can unintentionally assign COR duties to a staff  member who does not have the time available to properly monitor clinical  contractors.
If CORs’ workloads prevent proper monitoring of clinical contracts, VA  risks missing the opportunity to proactively identify and correct  performance issues with contract providers and to recognize patient  safety concerns potentially resulting from contract providers’ actions. By  failing to identify performance concerns with contract providers, VA could  unknowingly be receiving sub-standard service from these contractors,  continue to receive services from these contract providers that do not  meet the needs of the VAMCs, and risk patient safety problems when  these contracts are extended for additional years.
section_title: VA Training Does Not Adequately Prepare CORs to Monitor Contract Providers’ Performance
paragraphs: CORs from the four VAMCs we visited noted weaknesses in VA’s COR  training courses and our own analysis of these courses confirmed these  limitations. Specifically, over half of the 40 CORs that completed our data  collection instrument at the four VAMCs we visited responded that either  their COR training did not prepare them for their role as a COR or were  neutral on whether or not this training was helpful preparation. In addition,  CORs for 8 of the 12 contracts we reviewed did not find the required COR  training helpful or applicable to VA clinical contracting. Several of the  CORs monitoring the 12 selected contracts and CORs managing other  contracts at the four VAMCs we visited stated that VA’s required COR  training is focused on Department of Defense contracting and is not  tailored to CORs managing clinical contracts at VA. For example, one  COR stated that the training covered very broad areas of contracts and  did not include specific information on which kinds of contracts need  detailed quality assurance plans or information on how to manage a  clinical contract rather than a supply contract. In addition, a few CORs  stated that the instructors for their training courses had limited knowledge  of clinical contracting.
VA requires CORs to complete training courses to obtain the Federal  Acquisition Certification (FAC) for CORs or FAC-COR. are developed and offered by VA’s Acquisition Academy and can be  taken both in person and through instructor-led, web-based courses.  These courses generally aim to provide individuals with the knowledge  and tools to carry out the COR responsibilities, which include the  monitoring of clinical contractors’ performance. Previously we found that  well-designed training and development programs are linked to both  agency goals and to the organizational, occupational, and individual skills  and competencies needed for the agency to perform effectively.
There are three levels of FAC-COR certifications, which directly correlate with the years  of COR’s contracting experience. Specifically, the FAC-COR Level I certification is an   8-hour training and does not require previous experience as a COR, the FAC-COR Level  II certification is 40 hours of training (Level I combined with an additional 32 hours of  training) and requires 1 year of previous experience serving as a COR, and the FAC-COR  Level III certification is 60 hours of training and requires 2 years of previous experience  serving as a COR. equipment purchase, and soup contents and production. There were  no examples focused on how to evaluate or measure the quality of  services provided by a contract provider in a VAMC clinical setting. By  focusing on the types of examples in the course content, CORs did  not have the opportunity to learn and practice the skills necessary for  developing and managing service contracts that do not focus on  easily quantifiable elements and involve complex assessments of the  performance of clinicians.
Included little information on monitoring responsibilities. The  course content includes limited information for CORs on post-award  monitoring responsibilities for clinical contracts and instead is heavily  weighted to discussing the pre-award development of a contract. By  failing to incorporate a robust discussion of how CORs should be  monitoring contract providers in the course, VA has not provided  these critical personnel with a solid foundation for how to conduct their  responsibilities for monitoring the performance of contract providers  serving in clinical roles at VAMCs.
To supplement the required FAC-COR Level II course, VA’s Medical  Sharing Office recently developed and implemented an 8-hour training  course for CORs managing clinical contracts. However, VA does not  currently require this course be completed by all CORs managing clinical  contracts. This course covers primarily pre-award contract development  responsibilities of CORs and does not include any significant information  on the post-award monitoring responsibilities of CORs managing clinical  contracts.
Without developing required training on clinical contract monitoring, VA  cannot ensure that all CORs are receiving adequate information on how  to carry out this critical contract management responsibility and cannot  ensure that CORs are well trained in how to monitor contract providers.  This training places VA at risk of having CORs miss performance  weaknesses by clinical contractors that could affect patient safety, and  extending clinical contracts for contract providers who do not help VAMCs  effectively meet their clinical staffing needs or provide high-quality care to  veterans.
section_title: VA Central Office Oversight of VAMC Clinical Contract Monitoring is Limited
paragraphs: VA has not established a robust method for overseeing the monitoring of  clinical contractors by COs and CORs throughout VA’s health care  system. Standards for internal control in the federal government state that  agencies should design internal controls that assure ongoing monitoring  occurs in the course of normal operations, is continually performed, and is  ingrained in agency operations.
VA’s primary oversight entity for health care contracting activities, the  VHA Procurement and Logistics Office, has a limited role in overseeing  the monitoring actions of COs and CORs once a contract has been  approved and initiated at a VAMC. The VHA Procurement and Logistics  Office conducts limited oversight of contracting activities throughout the  VA health care system through its SAOs and Procurement Operations  Office.
Service Area Offices. According to officials from the three SAOs, the  role of the three SAOs in clinical contract monitoring is limited to an  audit of the records COs maintain in VA’s electronic Contract  Management System. These reviews focus only on the completeness  of COs’ electronic contracting files—including documentation that a  COR with current training records was assigned to the contract. SAO  electronic Contract Management System audits do not include any  reviews of CORs’ monitoring of clinical contractors.
Procurement Operations Office. The Procurement Operations  Office is the only VHA Procurement and Logistics Office entity  responsible for overseeing the monitoring activities of CORs;  however, the reviews conducted by this office are limited to a remote  electronic documentation review of a small sample of COR files.  Officials from the Procurement Operations Office told us that to select  COR files for these reviews, a Procurement Operations Office staff  member aims to select 25 COR files for active contracts per network  contracting office—at most 2.1 percent of clinical contracts in an  average VISN if all 25 selected COR files are for clinical contracts.  VA officials told us that, while the Procurement Operations Office sets  a goal to review COR files from two network contracting offices each  month, since implementing the program in March 2013 these reviews  have been completed in only four network contracting offices and  none of these four offices have received feedback on the outcomes of  these reviews as of August 2013. Officials added that this limited  review schedule is due to only one Procurement Operations Office  staff member being assigned to complete these reviews. These  reviews also have a narrow focus on the completeness of COR files  because the Procurement Operations Office staff member reviewing  the files relies on a checklist to verify the presence or absence of  required documentation of COR monitoring activities and does not  review the quality of information contained within CORs records.
The limited review schedule and narrow focus on file completeness do  not allow the Procurement Operations Office to comprehensively assess  the monitoring activities of COs and CORs throughout VA’s health care  system, as recommended by federal internal control standards. Without a  robust monitoring system in place, VA cannot reasonably assure that all  CORs in all VAMCs are maintaining the proper records of their efforts to  monitor the activities of clinical contractors caring for veterans.
section_title: Conclusions
paragraphs: Clinical contracts serve an important role in helping to ensure that VA can  provide health care to our nation’s veterans. Contract providers allow  VAMCs and CBOCs to supplement the capacity of VA-employed  providers and the ability to provide additional services at these VA  facilities. However, VA must maintain robust monitoring of these contract  providers to ensure they provide high quality care to veterans and fulfill  the responsibilities of their contracts. We identified weaknesses in four  areas that limit VA’s ability to effectively monitor clinical contractors and  provide support to VAMC-based CORs and contracting officials  responsible for conducting these critical reviews.
First, the lack of available tools such as templates for common types of  clinical contracts requires COs and CORs to rely on less consistent  sources for constructing critical performance requirements in clinical  contracts. As a result, most clinical contracts we reviewed were  insufficiently detailed in their descriptions of contract providers’  performance requirements and these performance requirements were not  consistently applied throughout our sample of clinical contracts. Without  making tools such as templates available to COs and CORs, VA cannot  ensure that all clinical contracts include performance requirements in key  categories necessary to VA’s operations and patient care, such as  credentialing and privileging and medical record documentation.
Second, VA lacks guidance for COs and CORs that defines the types of  performance standards and key elements of performance expected to be  included in clinical contracts. Without access to this type of guidance, VA  cannot ensure that contracting officials have the ability to establish clear  and measurable performance standards. Consistency in performance  measurement and attention to key elements—including clearly stating  performance standards, and ensuring these performance standards can  be accurately and consistently measured—is critical to VA’s efforts to  ensure that all contract providers are monitored appropriately and can be  held accountable for providing quality patient care.
Third, two significant challenges CORs face in the monitoring of clinical  contracts—COR workload and training—if left unresolved, will likely have  continuing negative impacts on the quality of the monitoring that CORs  are able to perform. Better guidance to VAMCs on how to determine an  appropriate COR workload would allow CORs to fulfill key functions  required to effectively monitor clinical contractors’ performance. Currently,  VA guidance is silent on how to determine whether or not a potential COR  has the time to take on the critical responsibilities of monitoring contract  providers. With the current practice of assigning COR duties to VAMC  staff as a collateral duty, it is critical that VA define how VAMC staff  should allocate COR duties to ensure that contracts are monitored  effectively. In addition, CORs are receiving insufficient training to prepare  them for monitoring clinical contracts. Currently, CORs for clinical  contracts cannot apply the current training to their duties because it  focuses on purchasing supplies rather than on acquiring clinical care  services. While VA has undertaken efforts to provide supplemental  training to CORs on clinical contracts, this additional training fails to  address how CORs should monitor clinical contracts and instead focuses  on the development of contracts. Without ensuring that COR training  addresses all of the critical monitoring functions CORs perform, VA  cannot ensure that all CORs are approaching this role with adequate  preparation and skills and VA risks failing to identify weaknesses in  contract providers’ performance that could affect patient safety.
Finally, VA Central Office currently provides limited oversight of clinical  contract monitoring activities. Without a robust monitoring program that  includes involvement from VA Central Office, VA does not have  reasonable assurance that all CORs in VAMCs throughout VA’s health  care system are properly monitoring clinical contractors and is missing  the opportunity to identify system-wide limitations in the monitoring of  these critical contract providers.
section_title: Recommendations for Executive Action
paragraphs: To improve the monitoring and oversight of clinical contracts, we  recommend that the Secretary of Veterans Affairs direct the Under  Secretary for Health to take the following five actions:
Develop and disseminate tools, such as standard templates, for the  most common types of clinical contracts in VA’s health care system.  Such tools should include performance requirement statements  covering key categories of VA health care policy and guidance—such  as credentialing and privileging, provider qualifications, and  expectations for compliance with critical VA policies and medical  record documentation requirements.
Develop and issue guidance on the performance standards that could  be included in common types of clinical contracts—including CBOC,  specialty, and temporary clinical provider contracts—to ensure that  these performance standards are clearly stated in the contracts and  have measurable targets for assessing contract provider performance.
Revise current standard operating procedures for CORs to provide  guidance on the number of contracts, based on size and complexity,  each COR should manage to ensure that all CORs maintain a  workload that allows them to fulfill their duties as a COR and their  primary position responsibilities.
Modify existing COR training to ensure it includes examples and  discussion of how to develop and monitor service contracts—including  contracts for the provision of clinical care in VAMCs.
Increase SAO and VA Central Office oversight of COs and CORs by  ensuring that post-award contracting files are regularly reviewed for all  network contracting offices.
section_title: Agency Comments
paragraphs: VA provided written comments on a draft of this report, which we have  reprinted in appendix II. In its comments, VA generally agreed with our  conclusions, concurred with our five recommendations, and described the  agency’s plans to implement each of our recommendations.
In its comments, VA stated that to address our first and second  recommendations, VHA’s Medical Sharing Office will work with VHA’s  Patient Care Services to update and complete templates for clinical  contracts so that they include performance requirement statements  covering key categories included in our review.
To address our third recommendation, VA noted that VHA’s Procurement  Policy and Operations Offices will collaborate to revise existing COR  standard operating procedures to include guidance on the number of  contracts, based on size and complexity, that each COR should manage  to ensure that all CORs maintain a workload that allows them to fulfill their  duties as a COR and their primary position responsibilities.
To address our fourth recommendation, VA stated that VHA’s Medical  Sharing Office will provide training to CORs that covers post-award COR  monitoring responsibilities. We support VA’s efforts to provide this training  content to CORs, especially those that monitor clinical contracts, but  encourage the agency to also modify the content of its existing required  COR training to ensure that CORs have the adequate preparation and  skills to effectively monitor contract providers.
Finally, to address our fifth recommendation, VA noted that VHA’s  Procurement Policy and Operations offices will collaborate to revise  existing COR standard operating procedures to include Service Area  Office and CO oversight to ensure that regular reviews take place for  post-award contract files from all network contracting offices.
We are sending copies of this report to the Secretary of Veterans Affairs,  appropriate congressional committees, and other interested parties. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-7114 or at williamsonr@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope and Methodology
paragraphs: This appendix describes the information and methods we used to  examine: (1) the extent to which the Department of Veterans Affairs (VA)  establishes complete performance requirements for clinical contractors;  (2) the extent to which VA clinical contracts include clear and measurable  performance standards for assessing whether or not clinical contractors  met the acceptable quality levels defined in selected contracts;   (3) challenges VA staff encounter in monitoring clinical contractors’  performance; and (4) the extent to which VA Central Office provides  oversight of VA staff responsible for monitoring clinical contracts.
Specifically, we discuss our methods for selecting VA medical centers  (VAMC) and network contracting offices to review; identifying appropriate  VA Central Office Officials to interview; selecting clinical contracts for  review and assessment; assessing the performance requirements  included in our selected contracts; assessing the performance standards  included in our selected contracts; administering a data collection  instrument to contracting officer’s representatives (COR) at our selected  VAMCs; and evaluating VA’s COR training.
subsections: section_title: Site Selection Methodology and Interviews with VAMC and Network Contracting Office Officials
paragraphs: We conducted four site visits to VAMCs to obtain the perspectives of  VAMC officials responsible for monitoring the day-to-day activities of  clinical contractors. To identify VAMCs for our site visits, we:
Selected VAMCs that were located in different Veterans Integrated  Service Networks (VISN) to ensure that our selected VAMCs varied in  their geographic locations and reported to different VISN management  officials;
Ensured that our selected VAMCs had a variety of active clinical  contracts in place to allow for a variety of COR perspectives on  clinical contractor monitoring; and
Selected at least one VAMC from each of VA’s three Service Area  Offices (SAO)—the regional contract management entities created to  oversee the activities of the 21 network contracting offices and the  contracting officers (CO) and supervisors that work within them.
Using these criteria, we selected four VAMCs to visit during our field work  located in Lebanon, Pennsylvania; Minneapolis, Minnesota; Nashville,  Tennessee; and Seattle, Washington. During our site visits to these  locations, we interviewed each VAMC’s leadership team; CORs  responsible for managing all active clinical contracts; quality management  staff; and the CORs and medical directors responsible for overseeing the  clinical contractors for a select sample of clinical contracts. We spoke with  these officials about a variety of topics—including COR training and  workload, monitoring procedures for clinical contractors, and their role in  developing performance requirements and performance standards for  clinical contracts.
In addition, we spoke with officials from the four network contracting  offices responsible for managing the contracting activities of our selected  VAMCs to discuss how COs and management staff manage the  monitoring of clinical contractors and interact with CORs at VAMCs.
Information obtained from our visits to selected VAMCs and interviews  with selected network contracting offices cannot be generalized to all  VAMCs and network contracting offices throughout VA’s health care  system, but provide important insights.
section_title: Interviews with VA Central Office Officials
paragraphs: We also interviewed VA Central Office officials responsible for developing  policies and procedures for VA clinical contracting activities and  overseeing the actions of CORs and COs throughout VA’s health care  system. We spoke with the following offices within the Veterans Health  Administration (VHA) Procurement and Logistics Office: (1) the Medical  Sharing Office; (2) the Procurement Audit Office; (3) the Procurement  Operations Office; and (4) all three SAOs.
section_title: Clinical Contract Selection and Analyses
paragraphs: To assess the monitoring of clinical contracts at VA, we reviewed the  contracts and accompanying COR documentation for a sample of clinical  contracts from the four VAMCs we visited during our field work.
To select our sample of clinical contracts, we reviewed a list of all active  clinical contracts for our selected VAMCs provided by the network  contracting offices responsible for overseeing the contracting actions of  the selected VAMCs. For each VAMC we visited, we selected three  contracts to review in detail—one community-based outpatient clinic  (CBOC) contract, one specialty care contract, and one temporary clinical  provider contract. (See table 6.)
To analyze the performance requirements included in our 12 selected  clinical contracts, we identified six performance requirement categories  for this analysis by reviewing: (1) VA Acquisition Regulation (VAAR);   (2) VA policy, guidance, and training documents; and (3) The Joint  Commission’s hospital accreditation standards. We also verified these  categories with officials from the Medical Sharing Office, Procurement  Operations Office, and Procurement Audit Office to ensure they were an  accurate reflection of performance requirements that should be included  in VA clinical contracts.
During our assessment of the 12 selected contracts, we analyzed the  performance requirements included in each clinical contract’s statement  of work against these six performance requirement categories and  determined whether or not performance requirements existed that  matched each categories’ required components. (See table 7.) Results of  our reviews were recorded as complete, partial, or incomplete based on  whether the contract’s performance requirements covered all, some, or  none of the specific components reviewed within each category.
To analyze the performance standards included in our 12 selected clinical  contracts, we reviewed whether each contract contained monitoring plan  provisions that included performance standards in five of the same six  categories we identified were vital to VA operations during our review of  performance requirements. We did not include the business processes  category in our analysis of performance standards because these  requirements do not require performance standards for their validation  and assessment.
For the 12 selected clinical contracts, we grouped performance standards  contained in each contract’s monitoring plan under the five performance  requirement categories used for this analysis—type of provider or care,  credentialing and privileging, clinical practice standards, medical record  documentation, and access to care (for CBOC contracts only).
We analyzed these performance standards using criteria previously  identified by GAO. Specifically, that performance standards should have  several attributes—including being clearly stated and using measurable  targets. We assessed each of the 12 selected contracts from the four  VAMCs we visited to determine if the performance standards included in  their monitoring plans met these two attributes of successful performance  standards. To determine whether performance standards were clearly  stated, two analysts independently determined whether or not each  performance standard had specified a desired outcome that was clearly  stated and understandable. To determine whether performance standards  used measurable targets, two analysts independently determined whether  or not each performance standard had a numerical target and included  information on the frequency and calculation method for this target. Each  analyst entered results as complete, partial, or incomplete for each  standard and entered a reason for their determination if the result was  partial or incomplete. A third analyst reviewed these assessments and  reconciled any differences that arose.
section_title: Data Collection Instrument Administered to CORs
paragraphs: We administered a data collection instrument to the 40 CORs responsible  for managing a clinical contract at the four VAMCs we visited to  determine their perspectives on COR training and workload, and to gather  information on whether or not they were serving as a COR as a collateral  duty. The data collection instrument included questions on each CORs  highest training level, assessment of VA COR training, amount of time  devoted to COR duties, and primary position title and description. Each  COR completed this data collection instrument in the presence of a GAO  analyst during our field work at each of our four selected VAMCs.
section_title: Evaluation of VA COR Training
paragraphs: To evaluate VA’s required COR training, three GAO analysts attended an  offering of VA’s COR training in September 2012. These analysts  reviewed the content of this training to determine if it met previously  identified criteria for successful training—that well-designed training and  development programs are linked to both agency goals and to the  organizational, occupational, and individual skills and competencies   To determine if these  needed for the agency to perform effectively.criteria were met, we reviewed the content of this course to determine if it  included information that was linked to VA’s agency goals and the  organization, occupational, and individual skills needed for a COR  monitoring clinical contractors in a VAMC.
section_title: Appendix II: Comments from the Department of Veterans Affairs
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Marcia A. Mann, Assistant  Director; A. Elizabeth Dobrenz; Pamela Dooley; Carolyn Garvey;  Katherine Nicole Laubacher; Laurie Pachter; Said Sariolghalam;   Jennifer Whitworth; and Malissa G. Winograd made key contributions   to this report.