VA Health Care: Adequacy of Resident Supervision Is Not Assured, but Plans Could Improve Oversight
Published: 2003-07-02T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Veterans Affairs (VA) provides graduate medical education (GME) to as many as one-third of U.S. resident physicians, but oversight responsibilities spread across VA's organizational components and multiple affiliated hospitals and medical schools could allow supervision problems to go undetected or uncorrected. GAO was asked to examine VA's procedures for (1) monitoring VA medical centers' adherence to VA's requirements for resident supervision, (2) using evaluations of supervision by GME accrediting bodies and residents, and (3) using information about resident supervision drawn from VA's programs for monitoring the quality and outcomes of patient care.
section_title: What GAO Found
paragraphs: VA cannot assure that the resident physicians who provide care in its facilities receive adequate supervision because its procedures for monitoring supervision are insufficient. VA does not know whether medical centers have adopted VA's national requirements for supervision of residents' diagnosis, treatment, or discharge of patients. VA officials require a review of only one specific requirement that is intended to ensure availability of supervision when a supervising physician does not need to be in the operating or procedural suite while a resident performs a diagnostic or therapeutic procedure. Four of 11 network officials we interviewed had not conducted this review, and the requirement at one medical center in one of these four networks was less stringent than VA's national requirement. To obtain more complete information about adherence to its national supervision requirements, VA plans to have external peer reviewers examine documentation of supervision in patients' medical records. VA's plans for this review have not been finalized. For example, as of May 2003, VA had not decided whether reviewers would examine records from VA's new outpatients. Without records from new patients, reviewers will not be able to assess documentation of residents' supervision during a veteran's first outpatient visit. To improve its oversight, VA is making efforts to obtain information from accrediting bodies and residents about the quality of resident supervision. For example, VA has taken steps to obtain direct access to letters from accrediting bodies that contain evaluations of the GME programs in which its medical centers participate. To solicit feedback from residents, VA implemented a national survey, but was unable to send this survey to a representative sample of residents from each VA medical center because it does not have a complete central list of its residents. VA is taking action to obtain this information. In addition, VA uses information from its broader programs for monitoring the quality and outcomes of patient care, such as its patient safety and surgical quality improvement programs, to identify and correct problems with resident supervision. Information from these programs has served as the basis for corrective actions by VA officials.

section_title: Letter
paragraphs: The Veterans Health Administration of the Department of Veterans Affairs  (VA) is the largest single provider of graduate medical education (GME)  training sites in the United States, with as many as one-third of the nation’s  resident physicians receiving part or all of their training in VA health care  facilities. Residents are medical school graduates who receive supervised  training in a medical specialty (such as internal medicine or surgery) prior  to providing care without supervision. As a provider of GME, VA faces the  dual challenges of ensuring the safety and quality of the health care its  patients receive from residents while simultaneously providing residents  with appropriate educational opportunities. Supervision of residents by  qualified physicians is central to balancing these patient care and  educational goals, and responsibility for the care provided by a resident to  any patient belongs to the licensed physician who supervises that  resident. Through observation and direction, supervising physicians are to  impart knowledge and skills to residents while making sure that patients  receive appropriate, timely, and effective care.
Effective oversight is necessary if VA is to assure the adequacy of resident  supervision. Key components of oversight include procedures to assess  the supervision residents receive and to initiate corrective action when  there is a problem. Information from multiple, complementary sources can  be used to assess supervision; such information includes evidence of  whether residents receive required supervision, evaluations of the  adequacy of supervision by organizations that accredit GME programs and  by residents, and analyses of the quality and outcomes of care provided by  residents. In 1986 and 1992, we reported that VA headquarters officials had  not adequately overseen resident supervision and that the documentation  of resident supervision at some medical centers was inadequate. Although  documentation does not fully communicate the extent or quality of  supervision, it is an important record of whether a supervising physician  was involved in a patient’s care.
Responsibilities for resident supervision and its oversight are distributed  across multiple VA organizational components and are shared by VA’s  affiliated medical schools and teaching hospitals. VA headquarters  established national requirements for supervision of residents’ health care  activities—including diagnosis, treatment, and discharge of patients—and  for oversight of supervision. Responsibilities for implementing these  requirements are assigned to the administrators of its regional networks  of medical facilities, medical center managers, and supervising physicians.  Most residency training within VA medical centers is conducted through  GME programs run by medical schools or other teaching hospitals, which  are known as sponsoring institutions. GME accrediting bodies hold the  sponsoring institutions responsible for the quality of the GME program in  each medical specialty. As a result, VA medical centers share  responsibility for ensuring the adequacy of residents’ supervision with  these affiliated sponsoring institutions.
Concerned that overlapping authority for residents’ activities could allow  problems with resident supervision to go undetected or uncorrected, you  asked us to examine the adequacy of VA’s oversight of resident  supervision. In response to your request, we examined VA’s procedures  for (1) monitoring VA medical centers’ adherence to VA’s requirements for  resident supervision, (2) using evaluations of supervision by GME  accrediting bodies and residents, and (3) using information about resident  supervision drawn from VA’s programs for monitoring the quality and  outcomes of patient care.
To address these objectives, we examined VA’s policy for resident  supervision and reviewed relevant documents from VA headquarters  offices, networks, and medical centers. We analyzed annual reports on  residency training submitted to VA headquarters for the 2000/2001  academic year by VA’s regional networks and 114 of the approximately   130 VA medical centers that provide GME training. We interviewed VA  officials, as well as GME experts and officials of accrediting bodies,  medical associations, and other stakeholder groups. We also interviewed  GME managers from 11 of VA’s 21 regional networks and 11 medical  centers. The sample included one medical center from each sampled  network and was designed to cover a range in total number of residency  positions and number of medical specialties in which training occurred.  We reviewed information from three additional medical centers involved  in GME programs that, as of May 2002, had been placed on probationary  accreditation or for which accreditation was to be withdrawn. We visited  two of those medical centers. Our work covered VA’s oversight of resident  supervision and did not include an evaluation of the quality of care  provided by residents or the quality of the supervision provided to  residents. We conducted our work from September 2001 through June  2003 in accordance with generally accepted government auditing  standards. See appendix I for a more detailed discussion of our scope and  methodology.
subsections: section_title: Background
paragraphs: Education is one of VA’s four core missions, and in fiscal year 2002, VA  paid approximately $383 million to residents training at about 130 VA  health care facilities. For the 2002/2003 academic year, VA supported  almost 8,800 residency slots, about 9 percent of all residency training  positions in the United States. Moreover, because several residents  typically rotate through each slot, VA estimates that it provides graduate  medical training to more than 28,000 residents each year, or as many as  one-third of the nation’s residents. The number of residency slots VA  allocates to individual medical centers involved in GME ranges from less  than 1 to more than 200. Although about half of VA’s residency positions  are in primary care, VA supports GME in 45 recognized medical specialties  and subspecialties; individual medical centers provide training in from 1 to  more than 30 specialties.
VA headquarters officials have ultimate oversight responsibility for the  activities of residents within VA medical centers, and several different  headquarters offices have monitoring functions that relate to resident  supervision. VA’s Office of Academic Affiliations (OAA) has responsibility  for developing and overseeing policies for resident supervision,  monitoring VA’s GME activities, and allocating residency slots. Under the  Patient Safety Program VA implemented in January 2002, VA’s National  Center for Patient Safety collects and analyzes information from VA  medical centers about patient risk events and their causes. Medical  centers are required to report all patient safety events—including adverse  events and close calls—to the National Center for Patient Safety. In  addition, medical centers are required to determine the root causes of  patient safety incidents with severe or potentially severe outcomes and  develop plans to prevent them in the future. The success of this program  will depend on the extent to which VA is able to establish a culture in  which employees feel safe to make these reports. VA’s Office of Patient  Care Services establishes and monitors health care programs. For  example, its National Surgical Quality Improvement Program (NSQIP)  examines postoperative outcomes. Additional oversight of resident  supervision is provided by VA’s Office of Inspector General.
Because VA’s health care system is decentralized, responsibilities for  implementing VA’s national policy for resident supervision are assigned to  networks and medical centers. Network officials are to provide medical  centers with the resources necessary to ensure that residents are  supervised in accordance with VA’s national policy and are to evaluate the  strengths and weaknesses of medical centers’ GME activities. Medical  center directors are responsible for establishing facility policies for  resident supervision that fulfill the requirements of VA’s national policy,  and medical center chiefs of staff are responsible for the educational and  patient care activities of all residents within the facility. In addition, a  physician in each medical specialty is responsible for ensuring that the  residents training in that specialty are supervised as required.
VA medical centers typically also share responsibility for the oversight of  residents with affiliated institutions that sponsor GME programs. VA  participates in more than 1,900 distinct GME programs, 29 of which are  sponsored by VA medical centers. The rest are sponsored by about 120  medical schools and teaching hospitals with which VA medical centers are  affiliated. The majority of VA medical centers work with one GME  sponsoring institution, but individual VA medical centers participate in the  GME programs of up to four different sponsors. When a VA medical center  serves as a training site for residents, but is not the sponsoring institution,  it is known as a participating institution. GME accrediting bodies hold  sponsoring institutions responsible for all aspects of their educational  programs, including aspects conducted within participating institutions.  GME accrediting bodies do not separately accredit participating  institutions and do not evaluate the extent to which supervision that  occurs within participating institutions, such as VA medical centers, meets  requirements set by those participating institutions.
VA requires accreditation of each GME program through which its  residents obtain training. More than 98 percent of VA’s residency slots are  filled by residents in GME programs that are subject to accreditation  review by the Accreditation Council for Graduate Medical Education  (ACGME); the remaining slots are filled by residents in osteopathic  programs that are subject to accreditation review by the American  Osteopathic Association. GME accreditation status indicates an overall  assessment of the quality of an educational program in a particular  medical specialty. Accrediting bodies evaluate several aspects of each  GME program, including provisions for the supervision and safety of  residents, the adequacy of institutional resources, educational curriculum,  and the extent to which the program meets that specialty’s specific  training requirements. A program can be fully accredited, or a program can  be granted accreditation with notification of problems that must be  corrected. Accreditation can also be withdrawn. A program’s accreditation  status is made public, but to safeguard confidential information, specific  problems with the program or its training sites are described in letters sent  only to the sponsoring institution. Accrediting bodies have not been  sending these letters to participating institutions.
Accrediting bodies state that the quality of patient care must remain the  highest priority of GME programs. Health care organizations that provide  GME must ensure that qualified staff physicians supervise residents and  that the same standards for the quality and safety of patient care apply  when residents are involved in health care delivery as when they are not.  GME accrediting bodies require that supervising physicians adjust the  level of supervision to meet the educational goal of increasing residents’  competence by giving them appropriate opportunities to assume greater  independence in their patient-care activities, that is, allowing residents to  assume graduated responsibilities. The supervising physician relies on his  or her professional judgment and knowledge of the patient’s medical  condition and the resident’s level of mastery to determine the degree of  independence of the resident’s patient-care responsibilities.
VA’s national policy on resident supervision is detailed in a handbook that  establishes specific requirements for (1) the involvement of supervising  physicians in the care provided by residents who diagnose, treat, or  discharge patients and (2) the documentation of that involvement. These  specific requirements apply to four domains of residents’ clinical  activity—inpatient care, outpatient care, diagnostic and therapeutic  procedures, and consultations—and provide guidelines for putting into  practice GME accrediting bodies’ principles of resident supervision and  graduated levels of responsibility. (See table 1 for an example of VA’s  requirements for supervision in each of the four domains.) Experts on  GME told us that the requirements in VA’s handbook are reasonable and  appropriately consider the role of supervision in ensuring the quality of  patient care and of resident education. Some of these experts described it  as a best practice model.
section_title: VA Lacks Adequate Procedures to Monitor Implementation of Its Supervision Requirements
paragraphs: VA does not have adequate procedures to determine whether residents at  VA medical centers are supervised in accordance with its national  requirements. For example, VA does not check whether each medical  center involved in GME has adopted policies that are consistent with VA’s  requirements for resident supervision. To learn what medical centers and  networks do to monitor whether supervision is consistent with VA’s  national requirements, VA requires that medical centers and networks  submit annual reports on residency training. Medical centers’ reports filed  for the 2000/2001 academic year indicate that most medical centers review  some documentation of resident supervision, but few conduct  comprehensive reviews. To obtain more complete information about the  supervision residents receive, VA is planning to use external peer review  to assess adherence to its requirements for documenting resident  supervision. These plans have not been finalized. For example, as of May  2003, VA had not decided whether reviewers would examine records from  VA’s new outpatients.
subsections: section_title: VA Does Not Determine Whether VA Medical Centers’ Policies Are Consistent with Its National Requirements for Resident Supervision
paragraphs: VA does not know whether all its medical centers have adopted policies  that are consistent with the specific requirements in its resident  supervision handbook for the supervision of residents’ diagnosis,  treatment, and discharge of patients. The director of each medical center  involved in GME is to establish facility policies for resident supervision  that fulfill the requirements in VA’s handbook, but VA requires a review of  only one requirement involving the supervision of diagnostic and  therapeutic procedures—the medical centers’ requirements for the  minimal acceptable level of supervision for diagnostic and therapeutic  procedures. Specifically, in situations in which the supervising physician is  not in the operating or procedural suite, VA requires that the supervisor  must, at a minimum, be immediately available in the facility or campus to  provide direct supervision of the procedure if necessary. Network GME  managers are supposed to review and approve this requirement; they are  not required to report the results of their reviews to OAA. There is no  separate OAA review of any of the requirements in medical centers’  supervision policies.
We found that not all networks have completed the one required review  and that medical centers’ policies are not always consistent with VA’s  national policy. Of the 11 network GME managers we interviewed, 7 told  us that they had completed this required review of the minimal  requirements for supervision of procedures in medical center policies, but  4 told us that they had not. We found that the requirement of a medical  center in one of the four networks that had not conducted this review was  less stringent than the requirement in VA’s handbook for supervision of  diagnostic and therapeutic procedures. The written policy at this medical  center stated that the supervising physician can be immediately available  by telephone rather than requiring him or her to be immediately available  in the facility or on campus. One network GME manager who did review  this requirement for supervision of diagnostic and therapeutic procedures  told us that in 2002, he identified three medical centers that had written  requirements for supervision of these procedures that were less stringent  than the requirement in VA’s handbook and that he instructed each of  these facilities to change its policy to be consistent with VA’s national  requirement.
section_title: VA Headquarters Monitors Medical Center and Network Oversight of Resident Supervision through Annual Reports
paragraphs: To learn what medical centers and networks do to monitor whether  supervision is consistent with VA’s resident supervision handbook, VA has  required annual reports on residency training programs beginning with the  1999/2000 academic year. Medical center managers are to provide  narrative answers to specific open-ended questions about their monitoring  processes as well as about the problems they identified and actions they  took to address them for each of three areas of oversight. (See table 2.)  These medical center reports are channeled through VA’s networks to  OAA. Network officials are to review them and summarize the strengths  and weaknesses of the medical centers’ GME programs in network-level  annual reports, which are also submitted to OAA.
These annual reports can provide managers with limited, but useful,  information about the extent and quality of monitoring performed by  medical centers, including whether medical centers monitor  documentation or some other indication of supervision. Some medical  centers and networks provided little detail in response to the annual  reports’ open-ended questions. For example, not all medical centers  described which specific aspects of resident supervision they monitored.  OAA used open-ended questions in part to accommodate differences  among medical centers in the number and type of residents they train.
VA officials have used information from annual reports to monitor medical  center oversight of resident supervision. For example, one network GME  manager followed up on a problem identified through a medical center  annual report by requiring the medical center to submit an action plan for  improving supervision of ophthalmology residents by the beginning of the  2002/2003 academic year. An OAA official told us that analysis of these  annual reports not only helped identify areas of vulnerability with  residency programs, but also pointed to possible best practices.
section_title: Most VA Medical Centers Monitor Some Documentation of Resident Supervision, but Few Conduct Comprehensive Reviews
paragraphs: VA does not require its medical centers or networks to conduct systematic  reviews of the documentation of resident supervision, and medical  centers differ in the extent to which they monitor adherence to VA’s  requirements for supervision. More than three-fourths of medical centers’  annual reports included a description of an independent review of the  documentation of supervision of at least one aspect of care provided by  residents, but most medical centers did not describe reviews of all four  domains of residents’ health care activities. For each of three domains— inpatient care, outpatient care, and diagnostic and therapeutic  procedures—over half the medical centers described a process for an  independent review of at least one element of the documentation of  resident supervision, that is, a review by someone other than a physician  with related supervisory responsibilities (see table 3). For example, the  quality management office at one medical center reviews medical records  each month to determine whether documentation indicates that inpatients  were seen by supervising physicians within 24 hours of admission. As  shown in table 3, few medical centers, however, described such a process  for review of supervisory documentation when residents provide  consultations to patients’ primary physicians.
In addition, medical centers’ annual reports did not always include clear,  detailed descriptions of the documentation requirements they monitor.  Few specifically mentioned monitoring particular VA-wide requirements,  such as the requirement for documentation of supervisory involvement at  the time of each new outpatient’s first visit. In some instances, medical  centers described a less systematic review process or one that was used  for only some services provided by residents. For diagnostic and  therapeutic procedures, for example, some medical centers described  processes for reviewing only selected procedures, such as endoscopies or  major surgeries.
About half of the 91 medical centers that reported having an independent  review process indicated they found deficiencies with the documentation  of resident supervision, and all but one discussed actions they took to  correct these problems. For example, officials from one medical center  told us that they implemented a program to discipline individual  physicians who consistently do not meet the medical center’s  requirements for documenting supervision. The acting chief of staff there  told us that during the 2001/2002 academic year, three physicians had each  been suspended without pay for 1 day for not consistently meeting  documentation requirements and that there had been significant  improvement in the documentation of resident supervision since this  disciplinary program went into effect. This medical center has also  developed a strategy for linking contract physicians’ pay to their provision  and documentation of supervision.
Documentation reviews have proven useful in identifying inadequate  supervision. We identified three medical centers that described in their  annual reports finding evidence of inadequate resident supervision  through their documentation reviews. In their annual reports, two of these  three medical centers stated that there were no adverse patient events  involving resident supervision. The third did not state whether there had  been any adverse patient outcomes. In the first instance, the medical  center reported that its review of documentation indicated that some staff  physicians provided a “low level” of supervision to residents in the  inpatient surgical setting. Medical center officials responded by meeting  with those physicians and conducting a follow-up review to monitor the  level of supervision. In the second instance, the medical center reported  that its supervision of residents was generally satisfactory, but that it had  found through its documentation review one episode in which the  attending surgeon had left the city during a procedure that he was  supposed to be supervising. This medical center reported that the surgeon  was formally reprimanded. In the third instance, a medical center reported  that through its documentation review, it identified two specialties— urology and plastic surgery—for which it wanted to increase the number  of procedures performed with the staff physician physically present and  directly involved in the surgery. The medical center reported that its  management was working with the surgery service chief to achieve this  goal.
We also identified a few medical centers that described independent  processes for monitoring resident supervision that went beyond reviewing  documentation. One medical center, for example, reported that staff in its  intensive care unit are required to report to the nurse manager any  situation they observe in which the supervision of a resident was  inappropriate.
In addition to monitoring processes established by medical centers, five of  VA’s networks indicated in their 2000/2001 annual reports that they had a  networkwide process for assessing adherence to one or more VA  requirements for documentation of resident supervision. For example, two  networks stated that they monitor the documentation of supervising  physicians’ involvement in the care of inpatients within 24 hours of  admission and another network assesses documentation of the  supervision of high-risk procedures. Two other networks reported they are  developing networkwide monitoring processes.
section_title: VA’s Plans to Use External Peer Review to Monitor Documentation of Supervision Have Not Been Finalized
paragraphs: To obtain more complete information about the extent to which its  requirements for supervision are being followed, VA has begun to test its  plans to monitor adherence through external peer review of the  documentation of supervision. External peer reviewers would examine a  sample of medical records from each medical center involved in GME to  determine whether they include required documentation of supervision.  Although documentation does not provide full information about the  extent or quality of supervision, it can provide VA oversight officials with  important information about whether supervisors were involved in patient  care. We compared the instructions that external reviewers would follow  with the requirements for supervision in VA’s handbook and found that the  instructions would allow reviewers to assess adherence to most of VA’s  key documentation requirements in the four domains of residents’ health  care activities. For example, if a resident participated in the care of an  inpatient or an outpatient during the current academic year, the external  reviewer is to determine whether documentation of supervision in the  patient’s medical record met the requirements in VA’s national handbook.  Reviewers are also to assess documentation of the supervision of residents  who performed diagnostic or therapeutic procedures or provided  consultations to other physicians. Results from each medical center are to  be provided to that medical center, as well as to headquarters managers.
External peer review of documentation of supervision in medical records  will be facilitated by features of VA’s computerized patient record  system. For example, the system automatically records the date and time  of notes; it also has the capacity to require that notes written by a resident  be co-signed by the supervising physician, in which case the note is not  considered complete until the required co-signature has been entered. In  addition, supervising physicians with whom we spoke noted that  immediate and easy access to legible information facilitates supervisors’  review of residents’ activities.
VA is in the early stages of testing its procedures for external peer review  of the documentation of resident supervision, and a VA official told us that  this effort is a high priority. A pilot test of portions of the inpatient  assessment methodology was conducted from October 2001 through June  2002 on a sample of almost 10,000 medical records. That pilot test  indicated that the central database used to select the sample of medical  records does not include information about which patients were seen by  residents. As a result, reviewers were unable to select an appropriate  sample of medical records. Until this problem is resolved, VA cannot  implement its plans for external peer review of resident supervision. OAA  has worked with other headquarters offices to revise VA’s information  technology software to ensure that this database contains information  about whether patients’ physicians were residents. VA expects to  implement this revision to its software by July 2003. The pilot test did not  indicate any other obstacles to implementing the portion of the plan for  reviewing documentation of resident supervision in inpatient settings.  Pilot tests of methods for assessing documentation of outpatient care,  diagnostic and therapeutic procedures, and consultations will not begin  until patients seen by residents can be clearly identified through the  central database.
One unresolved issue that will affect the usefulness of the external review  of supervision documentation in the outpatient setting involves selection  of the sample of medical records. The two options under consideration are  relying on the main outpatient sample used for VA’s other external peer  reviews or developing a sample specifically for review of the  documentation of supervision. The main outpatient sample in any given  year includes only patients who have received primary health care from  VA in the past and excludes most new patients who began obtaining health  care through VA within the preceding year—a group that has greatly  expanded in recent years. Without a sample of records from new  patients, it will not be possible to assess adherence to VA’s requirement  for supervisory involvement during a veteran’s first outpatient visit. An  OAA official told us that developing an additional sample of outpatient  records for review of documentation of supervision, distinct from the main  outpatient sample used for other purposes, would add to the expense of  the review. As of May 2003, VA had not made a decision about which  sample to use.
section_title: VA Is Acting to Obtain Information about Supervision from Accrediting Bodies and Residents
paragraphs: VA is making efforts to obtain consistent access to information provided  by accrediting bodies and residents about the quality of resident  supervision in VA medical centers. VA has taken steps to gain direct  access to the letters accrediting bodies send to sponsoring institutions to  describe concerns about GME programs. VA headquarters also developed  a survey to obtain feedback from residents, but cannot send it to a random  sample of residents because VA does not have a complete list of its  residents. VA is improving its ability to obtain that information. According  to their annual reports for the 2000/2001 academic year, most VA medical  centers that provide GME have some procedure for obtaining feedback  from residents.
subsections: section_title: VA Is Taking Steps to Gain Access to Accreditation Reviews of Its Affiliates’ GME Programs
paragraphs: VA does not currently have direct access to accreditation letters that  contain reviews of the GME programs sponsored by VA medical centers’  affiliates. These letters document concerns about residents’ education or  clinical experience that the GME program must address to retain  accreditation. Timely access to the information in these letters can allow  medical centers to take corrective actions. Until early 2000, ACGME sent  copies of its accreditation letters to OAA, and OAA made VA support for  residency slots contingent on VA medical centers’ taking action to correct  identified problems. In 2000, however, ACGME adopted new policies to  safeguard confidential accreditation information. As a result, ACGME  stopped sending the letters to VA, instead sending these letters only to the  institution that sponsors the GME program.
Without direct access to ACGME accreditation letters, VA medical centers  are dependent on sponsoring institutions to inform them of concerns  about the GME programs in which VA participates, and we learned of one  instance in which a sponsoring institution did not do so when ACGME  notified it of problems. Officials from a medical center told us that the  sponsoring institution of a thoracic surgery program did not tell them that  ACGME had previously identified multiple problems with the program  until ACGME decided, in September 2002, to withdraw the program’s  accreditation. ACGME did not cite any problems with the VA rotation.  Nonetheless, unanticipated withdrawal of a program’s accreditation can  affect a medical center’s educational and patient care missions. In this  case, the VA medical center will lose one full-time advanced surgical  resident in July 2003 and had to hire a physician’s assistant to provide  some of the services that had been provided by the resident.
Most medical centers indicated in their 2000/2001 annual reports that their  GME sponsors had shared information from accreditation letters, and  these annual reports provided network and headquarters officials with  information about accrediting bodies’ concerns and medical centers’  corrective actions. Fifty-six medical centers stated that accrediting bodies  had identified concerns about VA rotations in 145 of the more than 1,900  GME programs in which VA is involved. Concerns about 17 of these  programs related to resident supervision. For example, according to one  medical center’s annual report, ACGME concluded that residents required  more direct supervision during certain oncology rotations. Medical centers  reported that they had taken corrective action in response in all but one  instance. In this case, the accrediting body expressed concern that the VA  medical center had provided inadequate supervision and teaching in its  physical medicine and rehabilitation rotation, but the medical center did  not describe a corrective action in its annual report.
We found that when OAA had direct access to ACGME accreditation  letters—through early 2000—it took action to ensure that VA medical  centers knew of and responded to ACGME concerns about VA rotations.
Our review of OAA’s correspondence about accreditation issues covering  a period from late 1998 through early 2000 indicated that ACGME  mentioned concerns that were specific to VA rotations in its letters about  17 GME programs. In 6 of these cases, ACGME cited a concern about the  adequacy of resident supervision. For example, ACGME determined that  ophthalmology residents at one VA medical center had not been given  clear information about lines of supervisory responsibility. On receipt of  these letters, OAA contacted the participating VA medical center. Three of  the medical centers submitted documents to substantiate a resolution to  the problem within 2 months of hearing from OAA. In the other three  cases, OAA asked VA’s chief consultant for the relevant medical specialty  (such as the Chief Consultant for Ophthalmology) to assess the situation.  In each case, the consultant reported to OAA that a resolution had been  achieved. For example, the consultant reported that the ophthalmology  program cited for unclear lines of supervision was preparing a written  document to clarify supervisory responsibilities.
OAA has taken steps to arrange for renewed direct access to ACGME  accreditation letters. As part of that effort, VA issued a revised policy on  confidential documents in July 2002 to make sure that accreditation  reviews would be treated confidentially. In February 2003, VA signed a  memorandum of understanding with ACGME that lays the foundation for  OAA to receive copies of accreditation letters. According to this  memorandum, VA must now obtain revised affiliation agreements between  VA medical centers and GME sponsors that authorize ACGME to provide  OAA with its accreditation letters. VA is taking steps to ensure that these  revised agreements will be in place by July 2004. OAA has come to a  similar agreement with the American Osteopathic Association.
As a further step to obtain information about, and monitor responses to,  GME issues—including accreditation concerns—OAA reissued a policy  requiring VA medical centers to establish an affiliation partnership council  and submit minutes of council meetings to OAA. The council is to include  representatives of the medical center and its academic affiliate or affiliates  and is to advise VA managers as they work to meet educational  accreditation requirements and correct deficiencies or resolve problems.
section_title: VA Is Improving Its Ability to Obtain Feedback from a Representative Group of Residents
paragraphs: A mechanism OAA uses to obtain standardized information about  residents’ views on the quality of their supervision and other aspects of  their training is its Learners’ Perceptions Survey, which was first  distributed in March 2001. The survey asks residents to indicate their  satisfaction with the supervision they received from VA faculty by rating  supervising physicians’ teaching ability, accessibility/availability, and  approachability/openness, as well as overall satisfaction with VA clinical  faculty. Residents are also asked to evaluate their satisfaction with the  degree of supervision and degree of autonomy they experienced.
In 2001 and 2002, VA headquarters could not send the survey to a random,  representative sample of residents from each of its medical centers  involved in GME because it did not have a complete list of its trainees.  OAA was able to obtain feedback from many residents who did receive the  survey and gave those results to medical centers and networks. OAA is  taking steps to capture each trainee’s name and address in its automated  and centrally accessible information system and expects to implement this  procedure in July 2003. Once VA has a full registry of its trainees, OAA  plans to send the survey to a representative sample of residents in  different medical specialties that will include residents from all VA  medical centers involved in GME.
Medical centers’ annual reports can provide network and headquarters  officials with additional information about concerns expressed by  residents and steps taken to address those concerns. According to the  annual reports for the 2000/2001 academic year, most VA medical centers  used VA’s nationwide Learners’ Perceptions Survey or another  mechanism, such as residents’ confidential evaluations obtained by  sponsoring institutions, to obtain feedback about supervision. About half  of the 109 medical centers whose annual reports indicate that they had a  process for obtaining residents’ feedback said that residents had concerns  about their VA rotations. None of these concerns, however, involved the  adequacy of supervision.
section_title: VA Uses Its Programs for Monitoring Patient Care to Identify and Correct Problems with Resident Supervision
paragraphs: VA headquarters, network, and medical center officials use information  from VA’s programs for monitoring the quality and outcomes of patient  care to identify and correct problems with resident supervision. VA’s  monitoring programs include its new Patient Safety Program and NSQIP.  Reviews of paid tort claims by VA’s Chief Patient Care Officer provide  another mechanism for identifying problems with resident supervision.  OAA monitors medical centers’ use of these programs through the annual  reports on residency training. In their annual reports for the 2000/2001  academic year, most medical centers indicated that they monitor patient  care information to determine whether resident supervision affected the  quality or outcomes of patient care.
The system for reporting adverse events and close calls established by  VA’s Patient Safety Program has the potential to capture information about  instances in which inadequate resident supervision contributed to  heightened risk of adverse health care outcomes. Based on analysis of the  17,000 reports of adverse events and close calls filed with VA’s National  Center for Patient Safety as of April 2002, its director estimated that  resident supervision was mentioned—in any context—in less than 0.1  percent of the incidents reported by VA medical centers and that  inadequate supervision was a causal factor in very few of those cases.
Analyses of postoperative outcomes recorded in the NSQIP database,  including mortality and morbidity, provide VA with a way to study the  effects of residents’ involvement in surgical procedures. NSQIP personnel  analyze nationwide data from major surgeries, provide site-specific reports  to medical centers and networks, and conduct site visits at medical  centers. A NSQIP official told us that these data are routinely examined  for signs that supervision of residents might be inadequate. For example,  NSQIP analysts review the data to ensure that residents are not  performing surgeries that are more advanced than would be appropriate  for their level of training. In addition to reviewing NSQIP reports,  headquarters officials who oversee VA’s surgical services monitor the  frequency with which supervising physicians are in the operating or  procedural suite when residents perform surgeries.
Medical center and network officials have used NSQIP reports to help  monitor resident supervision. For example, a team of experts selected by  NSQIP visited one medical center at its request in February 2002 to help it  evaluate the efficiency of its operating rooms. During its visit, the team  noted inadequate supervision of surgeries performed by urology  residents. The medical center corrected this problem by arranging for  urologists to spend more time at the medical center and ensuring that they  understood VA’s requirements for supervision. In another instance, a  network GME manager observed that NSQIP data indicated that  orthopedic surgery outcomes at a particular medical center were less  favorable than expected. After a site visit, network officials concluded that  the medical center could not support complex surgeries and determined  that continued training of orthopedic residents at that medical center  would require a decrease in the complexity of cases and greater  involvement by supervising physicians. When the sponsoring institution  decided that the medical center would not meet its training needs under  those conditions, VA officials chose to transfer patients with complex  surgical needs to VA’s tertiary hospitals in the network and shift its two  VA-funded residency slots in orthopedic surgery to a different VA medical  center.
Researchers using the NSQIP database have studied ways in which  participation in GME affects postoperative outcomes. To determine  whether residency training places surgical patients at risk for worse  outcomes, researchers using the NSQIP database compared risk-adjusted  mortality rates in VA’s teaching and nonteaching hospitals and found that  they did not differ, although the patients who underwent surgeries at  teaching hospitals had a higher prevalence of risk factors, underwent more  complex operations, and had longer operation times. Morbidity rates were  higher in teaching than nonteaching hospitals for some surgical specialties  that were studied. On the basis of their analyses, the authors suggested  that differences in morbidity rates could reflect incomplete adjustment for  risks, such as severity of illness, or the more complex systems of managing  and coordinating care that characterize teaching hospitals, and not  necessarily the involvement of residents. Another study begun in  September 2001 is designed to use the NSQIP database to clarify the  relationship between residents’ working conditions and surgical  outcomes, with data from 90 VA hospitals and 3 nonfederal hospitals in  which surgical residents are trained.
Tort claims also provide information that VA uses to identify problems  with resident supervision that affected patient care. Review of paid tort  claims by VA’s Chief Patient Care Services Officer resulted in clarification  of VA’s written requirements for resident supervision when patients are  admitted to inpatient units. In the specific case that led to this change, a  supervising physician did not come to the hospital during a weekend to  see a patient who had been admitted by a resident; the patient died on  Monday. At that time, the resident supervision policy of the VA hospital in  which the incident occurred did not specifically require supervising  physicians to come in on weekends. As a result of this case, in October  2001 an explicit reference to weekends and holidays was added to the  handbook’s requirement that each new inpatient be seen by the  supervising physician within 24 hours of admission.
OAA monitors incidents in which resident supervision contributed to  adverse events or patient risks through the annual reports it requires from  medical centers. In their 2000/2001 annual reports on residency training,  all but 11 of 114 medical centers indicated that they monitored patient  safety events associated with residents. They used a variety of processes  to collect this information, including root cause analyses and tort claim  reviews, as well as additional processes such as mortality and morbidity  conferences and reviews triggered by unexpected events, such as  readmission within 10 days of discharge from the medical center. Annual  reports indicated that reviews of at least 18 actual or potential adverse  patient outcomes at a total of 14 medical centers identified resident  supervision as a possible contributing factor or led medical center officials  to strengthen supervision to minimize the chance of future problems. For  example, one medical center established a requirement for greater  involvement by supervising physicians before a resident initiates  chemotherapy orders. Medical centers described taking corrective actions  in response to these reviews.
section_title: Conclusions
paragraphs: VA cannot assure that the residents who provide care in its facilities  receive adequate supervision because its current procedures for  monitoring supervision are insufficient. To oversee the supervision of its  residents, VA needs various types of information, including information  regarding supervising physicians’ adherence to VA’s requirements for  resident supervision, accrediting bodies’ and residents’ concerns about  supervision, and whether the quality or outcomes of patient care indicate  problems with supervision. Systematic monitoring of each of these types  of information would help ensure that problems with resident supervision  are detected and corrected by the various officials of VA medical centers  and affiliated institutions who have responsibilities for residents’  activities.
Although VA issued a handbook that established specific standards for  resident supervision, VA does not know what its medical centers’  supervision requirements are and does not ensure that its national  requirements are adopted at each medical center where residents train.  Moreover, VA does not know whether the supervision its residents receive  adheres to its national requirements. VA’s current plans for external peer  review of documentation have the potential to enhance its oversight  capability, but these plans have not been finalized. For example, as of May  2003, VA had not decided whether external reviewers would examine  documentation of supervision for VA’s new outpatients, who make up a  significant and growing number of VA’s patients. Including these new  outpatients in the external review could help ensure adequate supervision  of residents during a patient’s first visit to VA.
To further improve its oversight of resident supervision, VA will need to  complete its initiatives to obtain timely access to evaluations by  accrediting bodies and residents. VA will also need to continue to take  advantage of its programs for monitoring the quality and outcomes of  patient care. VA officials have generally acted to improve supervision  when faced with evidence of problems, and better access to information  will enhance their ability to monitor and improve resident supervision.
By strengthening its oversight capabilities, VA could help promote both  the quality of the health care in its facilities and the education its residents  receive. As the largest provider of residency training sites in the United  States, VA’s actions to enhance the quality of resident supervision and its  oversight will have benefits beyond the VA health care system.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Veterans Affairs direct the Under  Secretary for Health to take steps to improve VA’s oversight of the  supervision of residents by    ensuring that all VA medical centers that provide GME adopt and adhere  to the requirements for resident supervision established in VA’s handbook  and    ensuring that external peer review of documentation of resident  supervision includes examination of records from VA’s new outpatients.
section_title: Agency Comments
paragraphs: In written comments on a draft of this report, VA agreed with our findings  and our recommendations. VA said our report described many steps it has  already taken that would help assure systematic implementation of its  national resident supervision policies and adequate headquarters oversight  of resident supervision. In concurring with our recommendation to ensure  that all VA medical centers that provide GME adopt and adhere to  requirements for resident supervision established in its handbook, VA  indicated its intention to monitor compliance with policy requirements  and highlight those requirements in a memorandum to network officials.  In concurring with our recommendation to ensure that external peer  review of documentation of resident supervision includes examination of  records from its new outpatients, VA indicated that it would develop a  strategy to identify new outpatients who were seen by a resident. It stated  that it expects to draw its first sample of records from outpatients,  including new outpatients, in the second quarter of fiscal year 2004. VA  also reported that it completed a revision of its centralized patient  information database. This revision was necessary to allow selection of an  appropriate sample of inpatient records for external peer review. VA’s  comments are in appendix II.
We are sending copies of this report to the Secretary of Veterans Affairs,  appropriate congressional committees, and other interested parties. We  will also make copies available to others who are interested upon request.  In addition, the report will be available at no charge on the GAO Web site  at http://www.gao.gov.
If you or your staff have any questions, please contact me at (202)  512-7101. An additional contact and the names of other staff members who  made contributions to this report are listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: To do our work, we examined oversight of resident supervision at each of  the Department of Veterans Affairs (VA) Veterans Health Administration’s  three organization levels—headquarters, networks, and medical centers.  Our work covered VA’s oversight of resident supervision and did not  include an evaluation of the quality of care provided by residents or the  quality of the supervision provided to residents. To assess oversight by  VA’s headquarters officials, we reviewed documents and interviewed  officials from VA’s Office of Academic Affiliations (OAA), Office of Patient  Care Services, National Center for Patient Safety, Office of Quality and  Performance, and Office of Information. We analyzed VA’s plans to have  external peer reviewers examine documentation of supervision and  compared the instructions the reviewers are to be given with VA’s  requirements for supervision.
To assess oversight of resident supervision by network officials, we  analyzed each network’s annual report to OAA on resident supervision  covering the 2000/2001 academic year. These were the most recent annual  reports available at the time. We did not assess the accuracy of  information provided in these reports. We also interviewed network GME  managers (known as network academic affiliations officers) from a  sample of 11 of VA’s 21 regional networks of health care facilities and  analyzed documents they provided (see table 4). We used a stratified  random sampling strategy to ensure variation in the number of VA-funded  residency slots among the selected networks. Network 19 was included in  our sample prior to randomization because it is the only network that did  not summarize the information in its medical centers’ reports. Another  network was excluded from our sample because it had been formed by the  merger of two former networks in January 2002. Our results from these 11  networks cannot be generalized to other networks.
To assess oversight of resident supervision by medical center officials, we  reviewed and analyzed 2000/2001 academic year annual reports to OAA on  resident supervision. OAA provided us with 114 annual reports from the  approximately 130 VA medical centers that were involved in GME during  the 2000/2001 academic year after it removed identifying information, such  as the names of medical centers, affiliates, and specific individuals. These  were all the medical center annual reports for the 2000/2001 academic year  that OAA had received as of June 18, 2002. We did not assess the accuracy  of information in the annual reports. We also interviewed GME managers  at 11 VA medical centers (see table 5) and analyzed their 2000/2001  academic year annual reports on resident supervision (without redaction)  and other documents. We used a stratified random sampling strategy to  ensure that the medical centers we selected varied in the number of VA- funded residency slots they were allocated for the 2001/2002 academic  year. We also ensured that our sample included one medical center from  each of the networks we had sampled and that the medical centers  differed in the number of medical specialties in which their residents train.  We did not review a systematically selected sample of medical centers’  resident supervision policies. Our results from these 11 medical centers  cannot be generalized to other medical centers.
We also reviewed documentary and testimonial evidence from four  medical centers that participate in internal medicine or general surgery  GME programs that had received adverse accreditation decisions as of  May 2002. One of these—the Fresno VA Medical Center—was part of our  sample of medical centers. Of the others, we visited the medical centers in  West Haven, Connecticut and Gainesville, Florida and interviewed officials  of the medical center in Albuquerque, New Mexico. We also spoke to  officials of the institutions that sponsor these three GME programs.
To obtain additional information about GME and VA’s residency training,  we analyzed accreditation requirements of the Accreditation Council for  Graduate Medical Education, American Osteopathic Association, and Joint  Commission on Accreditation of Healthcare Organizations and  interviewed officials of those bodies. We also interviewed representatives  of professional associations that are involved in GME, including the  American Board of Medical Specialties, American College of Surgeons,  American Hospital Association, American Medical Association, American  Medical Student Association, Association of American Medical Colleges  and its Council of Deans, Association of Professors of Medicine,  Committee of Interns and Residents, and Council on Graduate Medical  Education, and we reviewed relevant documents issued by these groups.  We interviewed representatives of physicians who teach internal medicine,  ophthalmology, psychiatry, general surgery, orthopedic surgery, and  urology—specialties for which a large number of VA medical centers  provide residency slots. We also interviewed representatives of veterans’  service organizations. We reviewed published literature regarding the  quality of care provided by residents.
We conducted our work from September 2001 through June 2003 in  accordance with generally accepted government auditing standards.
section_title: Appendix II: Comments from the Department of Veterans Affairs
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the person named above, key contributors to this report  were Kristen J. Anderson, William D. Hadley, Martha Fisher, Krister  Friday, and Donald Morrison.