VA Health Care: Adequacy of Resident Supervision Is Not Assured, but Plans Could Improve Oversight
Released: 2003-07-02T00:00:00

Letter

Background

Education is one of VA’s four core missions, and in fiscal year 2002, VA paid approximately $383 million to residents training at about 130 VA health care facilities. For the 2002/2003 academic year, VA supported almost 8,800 residency slots, about 9 percent of all residency training positions in the United States. Moreover, because several residents typically rotate through each slot, VA estimates that it provides graduate medical training to more than 28,000 residents each year, or as many as one-third of the nation’s residents. The number of residency slots VA allocates to individual medical centers involved in GME ranges from less than 1 to more than 200. Although about half of VA’s residency positions are in primary care, VA supports GME in 45 recognized medical specialties and subspecialties; individual medical centers provide training in from 1 to more than 30 specialties.

VA headquarters officials have ultimate oversight responsibility for the activities of residents within VA medical centers, and several different headquarters offices have monitoring functions that relate to resident supervision. VA’s Office of Academic Affiliations (OAA) has responsibility for developing and overseeing policies for resident supervision, monitoring VA’s GME activities, and allocating residency slots. Under the Patient Safety Program VA implemented in January 2002, VA’s National Center for Patient Safety collects and analyzes information from VA medical centers about patient risk events and their causes. Medical centers are required to report all patient safety events—including adverse events and close calls—to the National Center for Patient Safety. In addition, medical centers are required to determine the root causes of patient safety incidents with severe or potentially severe outcomes and develop plans to prevent them in the future. The success of this program will depend on the extent to which VA is able to establish a culture in which employees feel safe to make these reports. VA’s Office of Patient Care Services establishes and monitors health care programs. For example, its National Surgical Quality Improvement Program (NSQIP) examines postoperative outcomes. Additional oversight of resident supervision is provided by VA’s Office of Inspector General.

Because VA’s health care system is decentralized, responsibilities for implementing VA’s national policy for resident supervision are assigned to networks and medical centers. Network officials are to provide medical centers with the resources necessary to ensure that residents are supervised in accordance with VA’s national policy and are to evaluate the strengths and weaknesses of medical centers’ GME activities. Medical center directors are responsible for establishing facility policies for resident supervision that fulfill the requirements of VA’s national policy, and medical center chiefs of staff are responsible for the educational and patient care activities of all residents within the facility. In addition, a physician in each medical specialty is responsible for ensuring that the residents training in that specialty are supervised as required.

VA medical centers typically also share responsibility for the oversight of residents with affiliated institutions that sponsor GME programs. VA participates in more than 1,900 distinct GME programs, 29 of which are sponsored by VA medical centers. The rest are sponsored by about 120 medical schools and teaching hospitals with which VA medical centers are affiliated. The majority of VA medical centers work with one GME sponsoring institution, but individual VA medical centers participate in the GME programs of up to four different sponsors. When a VA medical center serves as a training site for residents, but is not the sponsoring institution, it is known as a participating institution. GME accrediting bodies hold sponsoring institutions responsible for all aspects of their educational programs, including aspects conducted within participating institutions. GME accrediting bodies do not separately accredit participating institutions and do not evaluate the extent to which supervision that occurs within participating institutions, such as VA medical centers, meets requirements set by those participating institutions.

VA requires accreditation of each GME program through which its residents obtain training. More than 98 percent of VA’s residency slots are filled by residents in GME programs that are subject to accreditation review by the Accreditation Council for Graduate Medical Education (ACGME); the remaining slots are filled by residents in osteopathic programs that are subject to accreditation review by the American Osteopathic Association. GME accreditation status indicates an overall assessment of the quality of an educational program in a particular medical specialty. Accrediting bodies evaluate several aspects of each GME program, including provisions for the supervision and safety of residents, the adequacy of institutional resources, educational curriculum, and the extent to which the program meets that specialty’s specific training requirements. A program can be fully accredited, or a program can be granted accreditation with notification of problems that must be corrected. Accreditation can also be withdrawn. A program’s accreditation status is made public, but to safeguard confidential information, specific problems with the program or its training sites are described in letters sent only to the sponsoring institution. Accrediting bodies have not been sending these letters to participating institutions.

Accrediting bodies state that the quality of patient care must remain the highest priority of GME programs. Health care organizations that provide GME must ensure that qualified staff physicians supervise residents and that the same standards for the quality and safety of patient care apply when residents are involved in health care delivery as when they are not. GME accrediting bodies require that supervising physicians adjust the level of supervision to meet the educational goal of increasing residents’ competence by giving them appropriate opportunities to assume greater independence in their patient-care activities, that is, allowing residents to assume graduated responsibilities. The supervising physician relies on his or her professional judgment and knowledge of the patient’s medical condition and the resident’s level of mastery to determine the degree of independence of the resident’s patient-care responsibilities.

VA’s national policy on resident supervision is detailed in a handbook that establishes specific requirements for (1) the involvement of supervising physicians in the care provided by residents who diagnose, treat, or discharge patients and (2) the documentation of that involvement. These specific requirements apply to four domains of residents’ clinical activity—inpatient care, outpatient care, diagnostic and therapeutic procedures, and consultations—and provide guidelines for putting into practice GME accrediting bodies’ principles of resident supervision and graduated levels of responsibility. (See table 1 for an example of VA’s requirements for supervision in each of the four domains.) Experts on GME told us that the requirements in VA’s handbook are reasonable and appropriately consider the role of supervision in ensuring the quality of patient care and of resident education. Some of these experts described it as a best practice model.

VA Lacks Adequate Procedures to Monitor Implementation of Its Supervision Requirements

VA does not have adequate procedures to determine whether residents at VA medical centers are supervised in accordance with its national requirements. For example, VA does not check whether each medical center involved in GME has adopted policies that are consistent with VA’s requirements for resident supervision. To learn what medical centers and networks do to monitor whether supervision is consistent with VA’s national requirements, VA requires that medical centers and networks submit annual reports on residency training. Medical centers’ reports filed for the 2000/2001 academic year indicate that most medical centers review some documentation of resident supervision, but few conduct comprehensive reviews. To obtain more complete information about the supervision residents receive, VA is planning to use external peer review to assess adherence to its requirements for documenting resident supervision. These plans have not been finalized. For example, as of May 2003, VA had not decided whether reviewers would examine records from VA’s new outpatients.

VA Does Not Determine Whether VA Medical Centers’ Policies Are Consistent with Its National Requirements for Resident Supervision

VA does not know whether all its medical centers have adopted policies that are consistent with the specific requirements in its resident supervision handbook for the supervision of residents’ diagnosis, treatment, and discharge of patients. The director of each medical center involved in GME is to establish facility policies for resident supervision that fulfill the requirements in VA’s handbook, but VA requires a review of only one requirement involving the supervision of diagnostic and therapeutic procedures—the medical centers’ requirements for the minimal acceptable level of supervision for diagnostic and therapeutic procedures. Specifically, in situations in which the supervising physician is not in the operating or procedural suite, VA requires that the supervisor must, at a minimum, be immediately available in the facility or campus to provide direct supervision of the procedure if necessary. Network GME managers are supposed to review and approve this requirement; they are not required to report the results of their reviews to OAA. There is no separate OAA review of any of the requirements in medical centers’ supervision policies.

We found that not all networks have completed the one required review and that medical centers’ policies are not always consistent with VA’s national policy. Of the 11 network GME managers we interviewed, 7 told us that they had completed this required review of the minimal requirements for supervision of procedures in medical center policies, but 4 told us that they had not. We found that the requirement of a medical center in one of the four networks that had not conducted this review was less stringent than the requirement in VA’s handbook for supervision of diagnostic and therapeutic procedures. The written policy at this medical center stated that the supervising physician can be immediately available by telephone rather than requiring him or her to be immediately available in the facility or on campus. One network GME manager who did review this requirement for supervision of diagnostic and therapeutic procedures told us that in 2002, he identified three medical centers that had written requirements for supervision of these procedures that were less stringent than the requirement in VA’s handbook and that he instructed each of these facilities to change its policy to be consistent with VA’s national requirement.

VA Headquarters Monitors Medical Center and Network Oversight of Resident Supervision through Annual Reports

To learn what medical centers and networks do to monitor whether supervision is consistent with VA’s resident supervision handbook, VA has required annual reports on residency training programs beginning with the 1999/2000 academic year. Medical center managers are to provide narrative answers to specific open-ended questions about their monitoring processes as well as about the problems they identified and actions they took to address them for each of three areas of oversight. (See table 2.) These medical center reports are channeled through VA’s networks to OAA. Network officials are to review them and summarize the strengths and weaknesses of the medical centers’ GME programs in network-level annual reports, which are also submitted to OAA.

These annual reports can provide managers with limited, but useful, information about the extent and quality of monitoring performed by medical centers, including whether medical centers monitor documentation or some other indication of supervision. Some medical centers and networks provided little detail in response to the annual reports’ open-ended questions. For example, not all medical centers described which specific aspects of resident supervision they monitored. OAA used open-ended questions in part to accommodate differences among medical centers in the number and type of residents they train.

VA officials have used information from annual reports to monitor medical center oversight of resident supervision. For example, one network GME manager followed up on a problem identified through a medical center annual report by requiring the medical center to submit an action plan for improving supervision of ophthalmology residents by the beginning of the 2002/2003 academic year. An OAA official told us that analysis of these annual reports not only helped identify areas of vulnerability with residency programs, but also pointed to possible best practices.

Most VA Medical Centers Monitor Some Documentation of Resident Supervision, but Few Conduct Comprehensive Reviews

VA does not require its medical centers or networks to conduct systematic reviews of the documentation of resident supervision, and medical centers differ in the extent to which they monitor adherence to VA’s requirements for supervision. More than three-fourths of medical centers’ annual reports included a description of an independent review of the documentation of supervision of at least one aspect of care provided by residents, but most medical centers did not describe reviews of all four domains of residents’ health care activities. For each of three domains— inpatient care, outpatient care, and diagnostic and therapeutic procedures—over half the medical centers described a process for an independent review of at least one element of the documentation of resident supervision, that is, a review by someone other than a physician with related supervisory responsibilities (see table 3). For example, the quality management office at one medical center reviews medical records each month to determine whether documentation indicates that inpatients were seen by supervising physicians within 24 hours of admission. As shown in table 3, few medical centers, however, described such a process for review of supervisory documentation when residents provide consultations to patients’ primary physicians.

In addition, medical centers’ annual reports did not always include clear, detailed descriptions of the documentation requirements they monitor. Few specifically mentioned monitoring particular VA-wide requirements, such as the requirement for documentation of supervisory involvement at the time of each new outpatient’s first visit. In some instances, medical centers described a less systematic review process or one that was used for only some services provided by residents. For diagnostic and therapeutic procedures, for example, some medical centers described processes for reviewing only selected procedures, such as endoscopies or major surgeries.

About half of the 91 medical centers that reported having an independent review process indicated they found deficiencies with the documentation of resident supervision, and all but one discussed actions they took to correct these problems. For example, officials from one medical center told us that they implemented a program to discipline individual physicians who consistently do not meet the medical center’s requirements for documenting supervision. The acting chief of staff there told us that during the 2001/2002 academic year, three physicians had each been suspended without pay for 1 day for not consistently meeting documentation requirements and that there had been significant improvement in the documentation of resident supervision since this disciplinary program went into effect. This medical center has also developed a strategy for linking contract physicians’ pay to their provision and documentation of supervision.

Documentation reviews have proven useful in identifying inadequate supervision. We identified three medical centers that described in their annual reports finding evidence of inadequate resident supervision through their documentation reviews. In their annual reports, two of these three medical centers stated that there were no adverse patient events involving resident supervision. The third did not state whether there had been any adverse patient outcomes. In the first instance, the medical center reported that its review of documentation indicated that some staff physicians provided a “low level” of supervision to residents in the inpatient surgical setting. Medical center officials responded by meeting with those physicians and conducting a follow-up review to monitor the level of supervision. In the second instance, the medical center reported that its supervision of residents was generally satisfactory, but that it had found through its documentation review one episode in which the attending surgeon had left the city during a procedure that he was supposed to be supervising. This medical center reported that the surgeon was formally reprimanded. In the third instance, a medical center reported that through its documentation review, it identified two specialties— urology and plastic surgery—for which it wanted to increase the number of procedures performed with the staff physician physically present and directly involved in the surgery. The medical center reported that its management was working with the surgery service chief to achieve this goal.

We also identified a few medical centers that described independent processes for monitoring resident supervision that went beyond reviewing documentation. One medical center, for example, reported that staff in its intensive care unit are required to report to the nurse manager any situation they observe in which the supervision of a resident was inappropriate.

In addition to monitoring processes established by medical centers, five of VA’s networks indicated in their 2000/2001 annual reports that they had a networkwide process for assessing adherence to one or more VA requirements for documentation of resident supervision. For example, two networks stated that they monitor the documentation of supervising physicians’ involvement in the care of inpatients within 24 hours of admission and another network assesses documentation of the supervision of high-risk procedures. Two other networks reported they are developing networkwide monitoring processes.

VA’s Plans to Use External Peer Review to Monitor Documentation of Supervision Have Not Been Finalized

To obtain more complete information about the extent to which its requirements for supervision are being followed, VA has begun to test its plans to monitor adherence through external peer review of the documentation of supervision. External peer reviewers would examine a sample of medical records from each medical center involved in GME to determine whether they include required documentation of supervision. Although documentation does not provide full information about the extent or quality of supervision, it can provide VA oversight officials with important information about whether supervisors were involved in patient care. We compared the instructions that external reviewers would follow with the requirements for supervision in VA’s handbook and found that the instructions would allow reviewers to assess adherence to most of VA’s key documentation requirements in the four domains of residents’ health care activities. For example, if a resident participated in the care of an inpatient or an outpatient during the current academic year, the external reviewer is to determine whether documentation of supervision in the patient’s medical record met the requirements in VA’s national handbook. Reviewers are also to assess documentation of the supervision of residents who performed diagnostic or therapeutic procedures or provided consultations to other physicians. Results from each medical center are to be provided to that medical center, as well as to headquarters managers.

External peer review of documentation of supervision in medical records will be facilitated by features of VA’s computerized patient record system. For example, the system automatically records the date and time of notes; it also has the capacity to require that notes written by a resident be co-signed by the supervising physician, in which case the note is not considered complete until the required co-signature has been entered. In addition, supervising physicians with whom we spoke noted that immediate and easy access to legible information facilitates supervisors’ review of residents’ activities.

VA is in the early stages of testing its procedures for external peer review of the documentation of resident supervision, and a VA official told us that this effort is a high priority. A pilot test of portions of the inpatient assessment methodology was conducted from October 2001 through June 2002 on a sample of almost 10,000 medical records. That pilot test indicated that the central database used to select the sample of medical records does not include information about which patients were seen by residents. As a result, reviewers were unable to select an appropriate sample of medical records. Until this problem is resolved, VA cannot implement its plans for external peer review of resident supervision. OAA has worked with other headquarters offices to revise VA’s information technology software to ensure that this database contains information about whether patients’ physicians were residents. VA expects to implement this revision to its software by July 2003. The pilot test did not indicate any other obstacles to implementing the portion of the plan for reviewing documentation of resident supervision in inpatient settings. Pilot tests of methods for assessing documentation of outpatient care, diagnostic and therapeutic procedures, and consultations will not begin until patients seen by residents can be clearly identified through the central database.

One unresolved issue that will affect the usefulness of the external review of supervision documentation in the outpatient setting involves selection of the sample of medical records. The two options under consideration are relying on the main outpatient sample used for VA’s other external peer reviews or developing a sample specifically for review of the documentation of supervision. The main outpatient sample in any given year includes only patients who have received primary health care from VA in the past and excludes most new patients who began obtaining health care through VA within the preceding year—a group that has greatly expanded in recent years. Without a sample of records from new patients, it will not be possible to assess adherence to VA’s requirement for supervisory involvement during a veteran’s first outpatient visit. An OAA official told us that developing an additional sample of outpatient records for review of documentation of supervision, distinct from the main outpatient sample used for other purposes, would add to the expense of the review. As of May 2003, VA had not made a decision about which sample to use.

VA Is Acting to Obtain Information about Supervision from Accrediting Bodies and Residents

VA is making efforts to obtain consistent access to information provided by accrediting bodies and residents about the quality of resident supervision in VA medical centers. VA has taken steps to gain direct access to the letters accrediting bodies send to sponsoring institutions to describe concerns about GME programs. VA headquarters also developed a survey to obtain feedback from residents, but cannot send it to a random sample of residents because VA does not have a complete list of its residents. VA is improving its ability to obtain that information. According to their annual reports for the 2000/2001 academic year, most VA medical centers that provide GME have some procedure for obtaining feedback from residents.

VA Is Taking Steps to Gain Access to Accreditation Reviews of Its Affiliates’ GME Programs

VA does not currently have direct access to accreditation letters that contain reviews of the GME programs sponsored by VA medical centers’ affiliates. These letters document concerns about residents’ education or clinical experience that the GME program must address to retain accreditation. Timely access to the information in these letters can allow medical centers to take corrective actions. Until early 2000, ACGME sent copies of its accreditation letters to OAA, and OAA made VA support for residency slots contingent on VA medical centers’ taking action to correct identified problems. In 2000, however, ACGME adopted new policies to safeguard confidential accreditation information. As a result, ACGME stopped sending the letters to VA, instead sending these letters only to the institution that sponsors the GME program.

Without direct access to ACGME accreditation letters, VA medical centers are dependent on sponsoring institutions to inform them of concerns about the GME programs in which VA participates, and we learned of one instance in which a sponsoring institution did not do so when ACGME notified it of problems. Officials from a medical center told us that the sponsoring institution of a thoracic surgery program did not tell them that ACGME had previously identified multiple problems with the program until ACGME decided, in September 2002, to withdraw the program’s accreditation. ACGME did not cite any problems with the VA rotation. Nonetheless, unanticipated withdrawal of a program’s accreditation can affect a medical center’s educational and patient care missions. In this case, the VA medical center will lose one full-time advanced surgical resident in July 2003 and had to hire a physician’s assistant to provide some of the services that had been provided by the resident.

Most medical centers indicated in their 2000/2001 annual reports that their GME sponsors had shared information from accreditation letters, and these annual reports provided network and headquarters officials with information about accrediting bodies’ concerns and medical centers’ corrective actions. Fifty-six medical centers stated that accrediting bodies had identified concerns about VA rotations in 145 of the more than 1,900 GME programs in which VA is involved. Concerns about 17 of these programs related to resident supervision. For example, according to one medical center’s annual report, ACGME concluded that residents required more direct supervision during certain oncology rotations. Medical centers reported that they had taken corrective action in response in all but one instance. In this case, the accrediting body expressed concern that the VA medical center had provided inadequate supervision and teaching in its physical medicine and rehabilitation rotation, but the medical center did not describe a corrective action in its annual report.

We found that when OAA had direct access to ACGME accreditation letters—through early 2000—it took action to ensure that VA medical centers knew of and responded to ACGME concerns about VA rotations.

Our review of OAA’s correspondence about accreditation issues covering a period from late 1998 through early 2000 indicated that ACGME mentioned concerns that were specific to VA rotations in its letters about 17 GME programs. In 6 of these cases, ACGME cited a concern about the adequacy of resident supervision. For example, ACGME determined that ophthalmology residents at one VA medical center had not been given clear information about lines of supervisory responsibility. On receipt of these letters, OAA contacted the participating VA medical center. Three of the medical centers submitted documents to substantiate a resolution to the problem within 2 months of hearing from OAA. In the other three cases, OAA asked VA’s chief consultant for the relevant medical specialty (such as the Chief Consultant for Ophthalmology) to assess the situation. In each case, the consultant reported to OAA that a resolution had been achieved. For example, the consultant reported that the ophthalmology program cited for unclear lines of supervision was preparing a written document to clarify supervisory responsibilities.

OAA has taken steps to arrange for renewed direct access to ACGME accreditation letters. As part of that effort, VA issued a revised policy on confidential documents in July 2002 to make sure that accreditation reviews would be treated confidentially. In February 2003, VA signed a memorandum of understanding with ACGME that lays the foundation for OAA to receive copies of accreditation letters. According to this memorandum, VA must now obtain revised affiliation agreements between VA medical centers and GME sponsors that authorize ACGME to provide OAA with its accreditation letters. VA is taking steps to ensure that these revised agreements will be in place by July 2004. OAA has come to a similar agreement with the American Osteopathic Association.

As a further step to obtain information about, and monitor responses to, GME issues—including accreditation concerns—OAA reissued a policy requiring VA medical centers to establish an affiliation partnership council and submit minutes of council meetings to OAA. The council is to include representatives of the medical center and its academic affiliate or affiliates and is to advise VA managers as they work to meet educational accreditation requirements and correct deficiencies or resolve problems.

VA Is Improving Its Ability to Obtain Feedback from a Representative Group of Residents

A mechanism OAA uses to obtain standardized information about residents’ views on the quality of their supervision and other aspects of their training is its Learners’ Perceptions Survey, which was first distributed in March 2001. The survey asks residents to indicate their satisfaction with the supervision they received from VA faculty by rating supervising physicians’ teaching ability, accessibility/availability, and approachability/openness, as well as overall satisfaction with VA clinical faculty. Residents are also asked to evaluate their satisfaction with the degree of supervision and degree of autonomy they experienced.

In 2001 and 2002, VA headquarters could not send the survey to a random, representative sample of residents from each of its medical centers involved in GME because it did not have a complete list of its trainees. OAA was able to obtain feedback from many residents who did receive the survey and gave those results to medical centers and networks. OAA is taking steps to capture each trainee’s name and address in its automated and centrally accessible information system and expects to implement this procedure in July 2003. Once VA has a full registry of its trainees, OAA plans to send the survey to a representative sample of residents in different medical specialties that will include residents from all VA medical centers involved in GME.

Medical centers’ annual reports can provide network and headquarters officials with additional information about concerns expressed by residents and steps taken to address those concerns. According to the annual reports for the 2000/2001 academic year, most VA medical centers used VA’s nationwide Learners’ Perceptions Survey or another mechanism, such as residents’ confidential evaluations obtained by sponsoring institutions, to obtain feedback about supervision. About half of the 109 medical centers whose annual reports indicate that they had a process for obtaining residents’ feedback said that residents had concerns about their VA rotations. None of these concerns, however, involved the adequacy of supervision.

VA Uses Its Programs for Monitoring Patient Care to Identify and Correct Problems with Resident Supervision

VA headquarters, network, and medical center officials use information from VA’s programs for monitoring the quality and outcomes of patient care to identify and correct problems with resident supervision. VA’s monitoring programs include its new Patient Safety Program and NSQIP. Reviews of paid tort claims by VA’s Chief Patient Care Officer provide another mechanism for identifying problems with resident supervision. OAA monitors medical centers’ use of these programs through the annual reports on residency training. In their annual reports for the 2000/2001 academic year, most medical centers indicated that they monitor patient care information to determine whether resident supervision affected the quality or outcomes of patient care.

The system for reporting adverse events and close calls established by VA’s Patient Safety Program has the potential to capture information about instances in which inadequate resident supervision contributed to heightened risk of adverse health care outcomes. Based on analysis of the 17,000 reports of adverse events and close calls filed with VA’s National Center for Patient Safety as of April 2002, its director estimated that resident supervision was mentioned—in any context—in less than 0.1 percent of the incidents reported by VA medical centers and that inadequate supervision was a causal factor in very few of those cases.

Analyses of postoperative outcomes recorded in the NSQIP database, including mortality and morbidity, provide VA with a way to study the effects of residents’ involvement in surgical procedures. NSQIP personnel analyze nationwide data from major surgeries, provide site-specific reports to medical centers and networks, and conduct site visits at medical centers. A NSQIP official told us that these data are routinely examined for signs that supervision of residents might be inadequate. For example, NSQIP analysts review the data to ensure that residents are not performing surgeries that are more advanced than would be appropriate for their level of training. In addition to reviewing NSQIP reports, headquarters officials who oversee VA’s surgical services monitor the frequency with which supervising physicians are in the operating or procedural suite when residents perform surgeries.

Medical center and network officials have used NSQIP reports to help monitor resident supervision. For example, a team of experts selected by NSQIP visited one medical center at its request in February 2002 to help it evaluate the efficiency of its operating rooms. During its visit, the team noted inadequate supervision of surgeries performed by urology residents. The medical center corrected this problem by arranging for urologists to spend more time at the medical center and ensuring that they understood VA’s requirements for supervision. In another instance, a network GME manager observed that NSQIP data indicated that orthopedic surgery outcomes at a particular medical center were less favorable than expected. After a site visit, network officials concluded that the medical center could not support complex surgeries and determined that continued training of orthopedic residents at that medical center would require a decrease in the complexity of cases and greater involvement by supervising physicians. When the sponsoring institution decided that the medical center would not meet its training needs under those conditions, VA officials chose to transfer patients with complex surgical needs to VA’s tertiary hospitals in the network and shift its two VA-funded residency slots in orthopedic surgery to a different VA medical center.

Researchers using the NSQIP database have studied ways in which participation in GME affects postoperative outcomes. To determine whether residency training places surgical patients at risk for worse outcomes, researchers using the NSQIP database compared risk-adjusted mortality rates in VA’s teaching and nonteaching hospitals and found that they did not differ, although the patients who underwent surgeries at teaching hospitals had a higher prevalence of risk factors, underwent more complex operations, and had longer operation times. Morbidity rates were higher in teaching than nonteaching hospitals for some surgical specialties that were studied. On the basis of their analyses, the authors suggested that differences in morbidity rates could reflect incomplete adjustment for risks, such as severity of illness, or the more complex systems of managing and coordinating care that characterize teaching hospitals, and not necessarily the involvement of residents. Another study begun in September 2001 is designed to use the NSQIP database to clarify the relationship between residents’ working conditions and surgical outcomes, with data from 90 VA hospitals and 3 nonfederal hospitals in which surgical residents are trained.

Tort claims also provide information that VA uses to identify problems with resident supervision that affected patient care. Review of paid tort claims by VA’s Chief Patient Care Services Officer resulted in clarification of VA’s written requirements for resident supervision when patients are admitted to inpatient units. In the specific case that led to this change, a supervising physician did not come to the hospital during a weekend to see a patient who had been admitted by a resident; the patient died on Monday. At that time, the resident supervision policy of the VA hospital in which the incident occurred did not specifically require supervising physicians to come in on weekends. As a result of this case, in October 2001 an explicit reference to weekends and holidays was added to the handbook’s requirement that each new inpatient be seen by the supervising physician within 24 hours of admission.

OAA monitors incidents in which resident supervision contributed to adverse events or patient risks through the annual reports it requires from medical centers. In their 2000/2001 annual reports on residency training, all but 11 of 114 medical centers indicated that they monitored patient safety events associated with residents. They used a variety of processes to collect this information, including root cause analyses and tort claim reviews, as well as additional processes such as mortality and morbidity conferences and reviews triggered by unexpected events, such as readmission within 10 days of discharge from the medical center. Annual reports indicated that reviews of at least 18 actual or potential adverse patient outcomes at a total of 14 medical centers identified resident supervision as a possible contributing factor or led medical center officials to strengthen supervision to minimize the chance of future problems. For example, one medical center established a requirement for greater involvement by supervising physicians before a resident initiates chemotherapy orders. Medical centers described taking corrective actions in response to these reviews.

Conclusions

VA cannot assure that the residents who provide care in its facilities receive adequate supervision because its current procedures for monitoring supervision are insufficient. To oversee the supervision of its residents, VA needs various types of information, including information regarding supervising physicians’ adherence to VA’s requirements for resident supervision, accrediting bodies’ and residents’ concerns about supervision, and whether the quality or outcomes of patient care indicate problems with supervision. Systematic monitoring of each of these types of information would help ensure that problems with resident supervision are detected and corrected by the various officials of VA medical centers and affiliated institutions who have responsibilities for residents’ activities.

Although VA issued a handbook that established specific standards for resident supervision, VA does not know what its medical centers’ supervision requirements are and does not ensure that its national requirements are adopted at each medical center where residents train. Moreover, VA does not know whether the supervision its residents receive adheres to its national requirements. VA’s current plans for external peer review of documentation have the potential to enhance its oversight capability, but these plans have not been finalized. For example, as of May 2003, VA had not decided whether external reviewers would examine documentation of supervision for VA’s new outpatients, who make up a significant and growing number of VA’s patients. Including these new outpatients in the external review could help ensure adequate supervision of residents during a patient’s first visit to VA.

To further improve its oversight of resident supervision, VA will need to complete its initiatives to obtain timely access to evaluations by accrediting bodies and residents. VA will also need to continue to take advantage of its programs for monitoring the quality and outcomes of patient care. VA officials have generally acted to improve supervision when faced with evidence of problems, and better access to information will enhance their ability to monitor and improve resident supervision.

By strengthening its oversight capabilities, VA could help promote both the quality of the health care in its facilities and the education its residents receive. As the largest provider of residency training sites in the United States, VA’s actions to enhance the quality of resident supervision and its oversight will have benefits beyond the VA health care system.

Recommendations for Executive Action

We recommend that the Secretary of Veterans Affairs direct the Under Secretary for Health to take steps to improve VA’s oversight of the supervision of residents by ensuring that all VA medical centers that provide GME adopt and adhere to the requirements for resident supervision established in VA’s handbook and ensuring that external peer review of documentation of resident supervision includes examination of records from VA’s new outpatients.

Agency Comments

In written comments on a draft of this report, VA agreed with our findings and our recommendations. VA said our report described many steps it has already taken that would help assure systematic implementation of its national resident supervision policies and adequate headquarters oversight of resident supervision. In concurring with our recommendation to ensure that all VA medical centers that provide GME adopt and adhere to requirements for resident supervision established in its handbook, VA indicated its intention to monitor compliance with policy requirements and highlight those requirements in a memorandum to network officials. In concurring with our recommendation to ensure that external peer review of documentation of resident supervision includes examination of records from its new outpatients, VA indicated that it would develop a strategy to identify new outpatients who were seen by a resident. It stated that it expects to draw its first sample of records from outpatients, including new outpatients, in the second quarter of fiscal year 2004. VA also reported that it completed a revision of its centralized patient information database. This revision was necessary to allow selection of an appropriate sample of inpatient records for external peer review. VA’s comments are in appendix II.

We are sending copies of this report to the Secretary of Veterans Affairs, appropriate congressional committees, and other interested parties. We will also make copies available to others who are interested upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions, please contact me at (202) 512-7101. An additional contact and the names of other staff members who made contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

To do our work, we examined oversight of resident supervision at each of the Department of Veterans Affairs (VA) Veterans Health Administration’s three organization levels—headquarters, networks, and medical centers. Our work covered VA’s oversight of resident supervision and did not include an evaluation of the quality of care provided by residents or the quality of the supervision provided to residents. To assess oversight by VA’s headquarters officials, we reviewed documents and interviewed officials from VA’s Office of Academic Affiliations (OAA), Office of Patient Care Services, National Center for Patient Safety, Office of Quality and Performance, and Office of Information. We analyzed VA’s plans to have external peer reviewers examine documentation of supervision and compared the instructions the reviewers are to be given with VA’s requirements for supervision.

To assess oversight of resident supervision by network officials, we analyzed each network’s annual report to OAA on resident supervision covering the 2000/2001 academic year. These were the most recent annual reports available at the time. We did not assess the accuracy of information provided in these reports. We also interviewed network GME managers (known as network academic affiliations officers) from a sample of 11 of VA’s 21 regional networks of health care facilities and analyzed documents they provided (see table 4). We used a stratified random sampling strategy to ensure variation in the number of VA-funded residency slots among the selected networks. Network 19 was included in our sample prior to randomization because it is the only network that did not summarize the information in its medical centers’ reports. Another network was excluded from our sample because it had been formed by the merger of two former networks in January 2002. Our results from these 11 networks cannot be generalized to other networks.

To assess oversight of resident supervision by medical center officials, we reviewed and analyzed 2000/2001 academic year annual reports to OAA on resident supervision. OAA provided us with 114 annual reports from the approximately 130 VA medical centers that were involved in GME during the 2000/2001 academic year after it removed identifying information, such as the names of medical centers, affiliates, and specific individuals. These were all the medical center annual reports for the 2000/2001 academic year that OAA had received as of June 18, 2002. We did not assess the accuracy of information in the annual reports. We also interviewed GME managers at 11 VA medical centers (see table 5) and analyzed their 2000/2001 academic year annual reports on resident supervision (without redaction) and other documents. We used a stratified random sampling strategy to ensure that the medical centers we selected varied in the number of VA- funded residency slots they were allocated for the 2001/2002 academic year. We also ensured that our sample included one medical center from each of the networks we had sampled and that the medical centers differed in the number of medical specialties in which their residents train. We did not review a systematically selected sample of medical centers’ resident supervision policies. Our results from these 11 medical centers cannot be generalized to other medical centers.

We also reviewed documentary and testimonial evidence from four medical centers that participate in internal medicine or general surgery GME programs that had received adverse accreditation decisions as of May 2002. One of these—the Fresno VA Medical Center—was part of our sample of medical centers. Of the others, we visited the medical centers in West Haven, Connecticut and Gainesville, Florida and interviewed officials of the medical center in Albuquerque, New Mexico. We also spoke to officials of the institutions that sponsor these three GME programs.

To obtain additional information about GME and VA’s residency training, we analyzed accreditation requirements of the Accreditation Council for Graduate Medical Education, American Osteopathic Association, and Joint Commission on Accreditation of Healthcare Organizations and interviewed officials of those bodies. We also interviewed representatives of professional associations that are involved in GME, including the American Board of Medical Specialties, American College of Surgeons, American Hospital Association, American Medical Association, American Medical Student Association, Association of American Medical Colleges and its Council of Deans, Association of Professors of Medicine, Committee of Interns and Residents, and Council on Graduate Medical Education, and we reviewed relevant documents issued by these groups. We interviewed representatives of physicians who teach internal medicine, ophthalmology, psychiatry, general surgery, orthopedic surgery, and urology—specialties for which a large number of VA medical centers provide residency slots. We also interviewed representatives of veterans’ service organizations. We reviewed published literature regarding the quality of care provided by residents.

We conducted our work from September 2001 through June 2003 in accordance with generally accepted government auditing standards.

Appendix II: Comments from the Department of Veterans Affairs

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the person named above, key contributors to this report were Kristen J. Anderson, William D. Hadley, Martha Fisher, Krister Friday, and Donald Morrison.