Report to Congressional Requesters

United States General Accounting Office 

GAO 

July 2003 VA HEALTH CARE 

Adequacy of Resident 

Supervision Is Not 

Assured, but Plans 

Could Improve 

Oversight 

GAO-03-625 
VA cannot assure that the resident physicians who provide care in its 

facilities receive adequate supervision because its procedures for monitoring 

supervision are insufficient. VA does not know whether medical centers 

have adopted VA’s national requirements for supervision of residents’ 

diagnosis, treatment, or discharge of patients. VA officials require a review 

of only one specific requirement that is intended to ensure availability of 

supervision when a supervising physician does not need to be in the 

operating or procedural suite while a resident performs a diagnostic or 

therapeutic procedure. Four of 11 network officials we interviewed had not 

conducted this review, and the requirement at one medical center in one of 

these four networks was less stringent than VA’s national requirement. To 

obtain more complete information about adherence to its national 

supervision requirements, VA plans to have external peer reviewers examine 

documentation of supervision in patients’ medical records. VA’s plans for 

this review have not been finalized. For example, as of May 2003, VA had 

not decided whether reviewers would examine records from VA’s new 

outpatients. Without records from new patients, reviewers will not be able to 

assess documentation of residents’ supervision during a veteran’s first 

outpatient visit. 

To improve its oversight, VA is making efforts to obtain information from 

accrediting bodies and residents about the quality of resident supervision. 

For example, VA has taken steps to obtain direct access to letters from 

accrediting bodies that contain evaluations of the GME programs in which 

its medical centers participate. To solicit feedback from residents, VA 

implemented a national survey, but was unable to send this survey to a 

representative sample of residents from each VA medical center because it 

does not have a complete central list of its residents. VA is taking action to 

obtain this information. 

In addition, VA uses information from its broader programs for monitoring 

the quality and outcomes of patient care, such as its patient safety and 

surgical quality improvement programs, to identify and correct problems 

with resident supervision. Information from these programs has served as 

the basis for corrective actions by VA officials. 

The Department of Veterans Affairs 

(VA) provides graduate medical 

education (GME) to as many as 

one-third of U.S. resident 

physicians, but oversight 

responsibilities spread across VA’s 

organizational components and 

multiple affiliated hospitals and 

medical schools could allow 

supervision problems to go 

undetected or uncorrected. GAO 

was asked to examine VA’s 

procedures for (1) monitoring VA 

medical centers’ adherence to VA’s 

requirements for resident 

supervision, (2) using evaluations 

of supervision by GME accrediting 

bodies and residents, and (3) using 

information about resident 

supervision drawn from VA’s 

programs for monitoring the quality

and outcomes of patient care. 

GAO recommends that VA 

• ensure that VA medical centers 

that provide GME adopt and 

adhere to VA’s national 

requirements for resident 

supervision and 

• ensure that external peer 

review of documentation of 

resident supervision includes 

records from VA’s new 

outpatients. 

VA concurred with the 

recommendations. 

www.gao.gov/cgi-bin/getrpt?GAO-03-625.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Cynthia A. 

Bascetta at (202) 512-7101. 

Highlights of GAO-03-625, a report to 

congressional requesters 

July 2003 

VA HEALTH CARE 

Adequacy of Resident Supervision Is Not 

Assured, but Plans Could Improve 

Oversight 
Page i GAO-03-625 VA’s Oversight of Resident Supervision 

Letter 1

Results in Brief 3

Background 6

VA Lacks Adequate Procedures to Monitor Implementation of Its 

Supervision Requirements 10

VA Is Acting to Obtain Information about Supervision from 

Accrediting Bodies and Residents 20

VA Uses Its Programs for Monitoring Patient Care to Identify and 

Correct Problems with Resident Supervision 24

Conclusions 27

Recommendations for Executive Action 28

Agency Comments 28

Appendix I Scope and Methodology 30 

Appendix II Comments from the Department of Veterans 

Affairs 34 

Appendix III GAO Contact and Staff Acknowledgments 37 

GAO Contact 37

Staff Acknowledgments 37

Tables 

Table 1: Examples of Requirements from VA’s Resident 

Supervision Handbook Issued on October 25, 2001, by 

Domain of Residents’ Health Care Activities 10

Table 2: Examples of Questions about Monitoring Processes from 

the Annual Report on Residency Training Programs

Completed by Medical Centers and Networks 13

Table 3: Number of VA Medical Centers That Reported Monitoring 

Some Aspect of the Documentation of Resident 

Supervision, by Domain of Residents’ Health Care 

Activities 15

Table 4: VA Networks Included in Our Sample 31

Table 5: VA Medical Centers Included in Our Sample 32

Contents 
Page ii GAO-03-625 VA’s Oversight of Resident Supervision 

Abbreviations 

ACGME Accreditation Council for Graduate Medical Education 

GME graduate medical education 

NSQIP National Surgical Quality Improvement Program 

OAA Office of Academic Affiliations 

VA Department of Veterans Affairs 

This is a work of the U.S. Government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. It may contain copyrighted graphics, images or other materials. 

Permission from the copyright holder may be necessary should you wish to reproduce 

copyrighted materials separately from GAO’s product. 
Page 1 GAO-03-625 VA’s Oversight of Resident Supervision 

July 2, 2003 

The Honorable Bob Graham 

Ranking Minority Member 

Committee on Veterans’ Affairs 

United States Senate 

The Honorable John D. Rockefeller IV 

United States Senate 

The Veterans Health Administration of the Department of Veterans Affairs 

(VA) is the largest single provider of graduate medical education (GME) 

training sites in the United States, with as many as one-third of the nation’s 

resident physicians receiving part or all of their training in VA health care 

facilities. Residents are medical school graduates who receive supervised 

training in a medical specialty (such as internal medicine or surgery) prior 

to providing care without supervision.1

 As a provider of GME, VA faces the 

dual challenges of ensuring the safety and quality of the health care its 

patients receive from residents while simultaneously providing residents 

with appropriate educational opportunities. Supervision of residents by 

qualified physicians is central to balancing these patient care and 

educational goals, and responsibility for the care provided by a resident to 

any patient belongs to the licensed physician who supervises that 

resident.2

 Through observation and direction, supervising physicians are to 

impart knowledge and skills to residents while making sure that patients 

receive appropriate, timely, and effective care. 

Effective oversight is necessary if VA is to assure the adequacy of resident 

supervision. Key components of oversight include procedures to assess 

the supervision residents receive and to initiate corrective action when 

there is a problem. Information from multiple, complementary sources can 

be used to assess supervision; such information includes evidence of 

whether residents receive required supervision, evaluations of the 

 1

In this report, the term “residents” also refers to fellows, physicians who have already 

completed a residency and are obtaining additional training in an advanced specialty or 

subspecialty. 

2

VA requires that the supervisor be a licensed physician who has been credentialed and 

privileged as a member of the staff of the medical facility in which the care is provided. 

United States General Accounting Office

Washington, DC 20548 
Page 2 GAO-03-625 VA’s Oversight of Resident Supervision 

adequacy of supervision by organizations that accredit GME programs and 

by residents, and analyses of the quality and outcomes of care provided by 

residents. In 1986 and 1992, we reported that VA headquarters officials had 

not adequately overseen resident supervision and that the documentation 

of resident supervision at some medical centers was inadequate.3

 Although 

documentation does not fully communicate the extent or quality of 

supervision, it is an important record of whether a supervising physician 

was involved in a patient’s care. 

Responsibilities for resident supervision and its oversight are distributed 

across multiple VA organizational components and are shared by VA’s 

affiliated medical schools and teaching hospitals. VA headquarters 

established national requirements for supervision of residents’ health care 

activities—including diagnosis, treatment, and discharge of patients—and 

for oversight of supervision. Responsibilities for implementing these 

requirements are assigned to the administrators of its regional networks4

of medical facilities, medical center managers, and supervising physicians. 

Most residency training within VA medical centers is conducted through 

GME programs run by medical schools or other teaching hospitals, which 

are known as sponsoring institutions. GME accrediting bodies hold the 

sponsoring institutions responsible for the quality of the GME program in 

each medical specialty. As a result, VA medical centers share 

responsibility for ensuring the adequacy of residents’ supervision with 

these affiliated sponsoring institutions. 

Concerned that overlapping authority for residents’ activities could allow 

problems with resident supervision to go undetected or uncorrected, you 

asked us to examine the adequacy of VA’s oversight of resident 

supervision. In response to your request, we examined VA’s procedures 

for (1) monitoring VA medical centers’ adherence to VA’s requirements for 

resident supervision, (2) using evaluations of supervision by GME 

accrediting bodies and residents, and (3) using information about resident 

 3

See U.S. General Accounting Office, VA Hospitals: Surgical Residents Need Closer 

Supervision, GAO/HRD-86-15 (Washington, D.C.: Jan. 13, 1986) and VA Health Care: 

Medical Centers Are Not Correcting Identified Quality Assurance Problems, 

GAO/HRD-93-20 (Washington, D.C.: Dec. 30, 1992). 

4

VA health care facilities are organized into 21 regional networks, known as Veterans 

Integrated Service Networks, which are to coordinate the activities of and allocate funds to 

VA health care facilities. VA had 22 networks until January 2002, when it merged two of 

them. 
Page 3 GAO-03-625 VA’s Oversight of Resident Supervision 

supervision drawn from VA’s programs for monitoring the quality and 

outcomes of patient care. 

To address these objectives, we examined VA’s policy for resident 

supervision and reviewed relevant documents from VA headquarters 

offices, networks, and medical centers. We analyzed annual reports on 

residency training submitted to VA headquarters for the 2000/2001 

academic year by VA’s regional networks and 114 of the approximately 

130 VA medical centers that provide GME training.5

 We interviewed VA 

officials, as well as GME experts and officials of accrediting bodies, 

medical associations, and other stakeholder groups. We also interviewed 

GME managers from 11 of VA’s 21 regional networks and 11 medical 

centers. The sample included one medical center from each sampled 

network and was designed to cover a range in total number of residency 

positions and number of medical specialties in which training occurred. 

We reviewed information from three additional medical centers involved 

in GME programs that, as of May 2002, had been placed on probationary 

accreditation or for which accreditation was to be withdrawn.6

 We visited 

two of those medical centers. Our work covered VA’s oversight of resident 

supervision and did not include an evaluation of the quality of care 

provided by residents or the quality of the supervision provided to 

residents. We conducted our work from September 2001 through June 

2003 in accordance with generally accepted government auditing 

standards. See appendix I for a more detailed discussion of our scope and 

methodology. 

VA cannot assure that the residents who provide care in its facilities 

receive adequate supervision because its procedures for monitoring 

supervision are insufficient. VA does not know whether medical centers 

have adopted resident supervision policies that are consistent with VA’s 

national requirements for supervision of residents’ health care activities, 

such as diagnosis, treatment, and discharge of patients. VA officials 

require a review of only the requirement that is intended to ensure 

 5

The number of VA medical centers that provide GME varies slightly over time, in part 

because at facilities with only a few allocated residency slots, it is possible that no slot 

might be filled at a particular time. 

6

In addition to these three programs, one program at 1 of the 11 medical centers in our 

sample was on probationary accreditation. Of the four programs that had received adverse 

accreditation decisions, two are no longer in an adverse accreditation status. Reevaluation 

of the other two was not complete as of May 2003. 

Results in Brief 
Page 4 GAO-03-625 VA’s Oversight of Resident Supervision 

availability of supervision when a supervising physician does not need to 

be in the operating or procedural suite while a resident performs a 

diagnostic or therapeutic procedure. Network GME officials are to review 

this requirement in medical centers’ policies, and 4 of 11 network GME 

managers we interviewed had not conducted this review. Moreover, we 

found that the requirement at one medical center in one of these four 

networks was less stringent than VA’s national requirement. To learn 

which aspects of resident supervision medical centers and networks 

monitor, VA requires medical centers and networks to submit annual 

reports on residency training. Medical centers’ annual reports for the 

2000/2001 academic year indicate that most medical centers monitor some 

documentation of supervision, but few conduct comprehensive reviews. 

About half of the 91 medical centers that reported having a review process 

also reported finding inadequate documentation of supervision and then 

taking steps to improve it. To obtain more complete information about 

adherence to its national requirements for supervision, VA plans to have 

external peer reviewers examine the documentation of supervision in 

patients’ medical records. External peer review could allow assessment of 

whether most of VA’s key documentation requirements are being met. VA’s 

plans for this review, however, have not been finalized, and as of May 

2003, VA had not decided whether reviewers would examine records from 

VA’s new outpatients. Without a sample of records from new patients, 

reviewers would not be able to assess the required documentation of 

supervisory involvement during a veteran’s first outpatient visit. 

VA is making efforts to obtain information from accrediting bodies and 

residents about the quality of supervision provided to its residents. For 

example, VA has taken steps to gain copies of accreditation letters directly 

from GME accrediting bodies. These letters contain evaluations of the 

GME programs in which VA medical centers participate and are sent to the 

institutions that sponsor GME programs, but not to medical centers that 

participate in those programs. As a result, VA medical centers must 

generally rely on affiliated sponsoring institutions to inform them of 

problems identified by the accrediting body. We found that most VA 

medical center managers indicated that their affiliated sponsors had 

shared this information, and when problems with VA were identified, VA 

managers reported taking action to solve them. We also found, however, 

that one sponsoring institution did not provide a participating VA medical 

center with timely information about an impending withdrawal of 

accreditation for a GME program. To obtain standardized feedback from 

residents about their educational experiences, including the quality of 

their supervision, VA implemented a national survey in 2001. In 2001 and 

2002, VA could not send the survey to a representative sample of residents 
Page 5 GAO-03-625 VA’s Oversight of Resident Supervision 

from each VA medical center because it lacked a complete list of its 

residents. VA is taking action to obtain this information so that it can send 

the survey to a sample of residents at each medical center. Medical 

centers’ annual reports provide network and headquarters officials with 

additional information about concerns expressed by residents and steps 

taken to address those concerns. 

VA also uses information from its broader programs for monitoring the 

quality and outcomes of patient care, such as its patient safety and surgical 

quality improvement programs, to identify and correct problems with 

resident supervision. Although too new to evaluate, the patient safety 

program VA implemented in January 2002 established a process for 

determining the causes of events that led to or could have led to patient 

harm and for taking steps to eliminate or minimize identified risks, such as 

inadequate resident supervision. In addition, VA’s program for monitoring 

and improving surgical outcomes allows VA to examine residents’ 

performance of surgical procedures. Information generated by this 

program has prompted medical center and network officials to take steps 

to improve the supervision of surgical residents. For example, a team of 

experts from this program noted inadequate supervision of surgeries 

performed by urology residents at one medical center they visited in 2002. 

The medical center responded by arranging for urologists to spend more 

time at the medical center and ensuring that they understood VA’s 

supervision requirements. Tort claim review is another way VA monitors 

the possible role of resident supervision in problems with patient care. 

Review of paid tort claims led VA in 2001 to clarify its requirements for 

supervision in inpatient settings by adding an explicit reference to 

weekends and holidays to the requirement that each inpatient must be 

seen by the supervising physician within 24 hours of admission. 

To improve VA’s oversight of resident supervision and help ensure the 

quality of both health care and GME, we are making recommendations to 

the Secretary of Veterans Affairs to ensure that VA medical centers adopt 

and adhere to VA’s national requirements for resident supervision and to 

ensure that external peer review of documentation of resident supervision 

includes records from VA’s new outpatients. VA concurred with our 

recommendations. 
Page 6 GAO-03-625 VA’s Oversight of Resident Supervision 

Education is one of VA’s four core missions,7 and in fiscal year 2002, VA 

paid approximately $383 million to residents training at about 130 VA 

health care facilities.8

 For the 2002/2003 academic year, VA supported 

almost 8,800 residency slots, about 9 percent of all residency training 

positions in the United States. Moreover, because several residents 

typically rotate through each slot, VA estimates that it provides graduate 

medical training to more than 28,000 residents each year, or as many as 

one-third of the nation’s residents. The number of residency slots VA 

allocates to individual medical centers involved in GME ranges from less 

than 1 to more than 200.9

 Although about half of VA’s residency positions 

are in primary care, VA supports GME in 45 recognized medical specialties 

and subspecialties; individual medical centers provide training in from 1 to 

more than 30 specialties. 

VA headquarters officials have ultimate oversight responsibility for the 

activities of residents within VA medical centers, and several different 

headquarters offices have monitoring functions that relate to resident 

supervision. VA’s Office of Academic Affiliations (OAA) has responsibility 

for developing and overseeing policies for resident supervision, 

monitoring VA’s GME activities, and allocating residency slots. Under the 

Patient Safety Program VA implemented in January 2002, VA’s National 

Center for Patient Safety collects and analyzes information from VA 

medical centers about patient risk events and their causes. Medical 

centers are required to report all patient safety events—including adverse 

events and close calls10—to the National Center for Patient Safety. In 

addition, medical centers are required to determine the root causes of 

patient safety incidents with severe or potentially severe outcomes and 

develop plans to prevent them in the future. The success of this program 

will depend on the extent to which VA is able to establish a culture in 

 7

VA’s four core missions are patient care, education, research, and medical backup to the 

Department of Defense in the event of a national security emergency. 

8

These expenditures included stipends and benefits for residents training in accredited 

medical specialties and subspecialties and an additional 150 special fellows training in 

emerging, as yet nonaccredited fields of medicine, such as geriatric neurology and 

palliative care. 

9

Allocations of fractions of slots are possible because residents might obtain only a part of 

their training at a VA medical center. 

10Adverse events include adverse drug events and procedural errors or complications that 

are associated with care. Close calls are events or situations that could have resulted in an 

adverse event but did not, either by chance or through timely intervention. VA specifies 

that alternative procedures are to be used for reporting intentionally unsafe acts. 

Background 
Page 7 GAO-03-625 VA’s Oversight of Resident Supervision 

which employees feel safe to make these reports.11 VA’s Office of Patient 

Care Services establishes and monitors health care programs. For 

example, its National Surgical Quality Improvement Program (NSQIP) 

examines postoperative outcomes.12 Additional oversight of resident 

supervision is provided by VA’s Office of Inspector General.13

Because VA’s health care system is decentralized, responsibilities for 

implementing VA’s national policy for resident supervision are assigned to 

networks and medical centers. Network officials are to provide medical 

centers with the resources necessary to ensure that residents are 

supervised in accordance with VA’s national policy and are to evaluate the 

strengths and weaknesses of medical centers’ GME activities. Medical 

center directors are responsible for establishing facility policies for 

resident supervision that fulfill the requirements of VA’s national policy,14

and medical center chiefs of staff are responsible for the educational and 

patient care activities of all residents within the facility. In addition, a 

physician in each medical specialty is responsible for ensuring that the 

residents training in that specialty are supervised as required. 

VA medical centers typically also share responsibility for the oversight of 

residents with affiliated institutions that sponsor GME programs. VA 

participates in more than 1,900 distinct GME programs, 29 of which are 

sponsored by VA medical centers.15 The rest are sponsored by about 120 

medical schools and teaching hospitals with which VA medical centers are 

affiliated. The majority of VA medical centers work with one GME 

 11VA arranged for the National Aeronautics and Space Administration to provide an 

independent external system for reporting patient safety concerns. This system allows 

anyone who feels uncomfortable reporting an event to VA’s internal patient safety 

managers to file a voluntary, confidential report to an outside agency. Reports entered in 

this database are anonymous. Nationwide implementation of this second reporting system 

began in March 2002. 

12NSQIP is housed administratively in VA’s Office of Patient Care Services. It exercises its 

monitoring and advisory functions through the chief medical officers of VA’s networks. 

13In April 2003, VA’s Office of Inspector General reported that part-time physicians were 

not always present in the clinics where the residents they supervised provided care. See VA 

Office of Inspector General, Audit of the Veterans Health Administration’s Part-Time 

Physician Time and Attendance, 02-01339-85 (Washington, D.C.: April 2003). 

14Medical centers must adopt their own policies to ensure that local requirements, such as 

those established by affiliated GME sponsors, are included. 

15These 29 GME programs are sponsored by seven VA medical centers, each of which also 

participates in GME programs that are sponsored by affiliated institutions. 
Page 8 GAO-03-625 VA’s Oversight of Resident Supervision 

sponsoring institution, but individual VA medical centers participate in the 

GME programs of up to four different sponsors. When a VA medical center 

serves as a training site for residents, but is not the sponsoring institution, 

it is known as a participating institution. GME accrediting bodies hold 

sponsoring institutions responsible for all aspects of their educational 

programs, including aspects conducted within participating institutions. 

GME accrediting bodies do not separately accredit participating 

institutions and do not evaluate the extent to which supervision that 

occurs within participating institutions, such as VA medical centers, meets 

requirements set by those participating institutions. 

VA requires accreditation of each GME program through which its 

residents obtain training. More than 98 percent of VA’s residency slots are 

filled by residents in GME programs that are subject to accreditation 

review by the Accreditation Council for Graduate Medical Education 

(ACGME); the remaining slots are filled by residents in osteopathic 

programs that are subject to accreditation review by the American 

Osteopathic Association. GME accreditation status indicates an overall 

assessment of the quality of an educational program in a particular 

medical specialty. Accrediting bodies evaluate several aspects of each 

GME program, including provisions for the supervision and safety of 

residents, the adequacy of institutional resources, educational curriculum, 

and the extent to which the program meets that specialty’s specific 

training requirements. A program can be fully accredited, or a program can 

be granted accreditation with notification of problems that must be 

corrected. Accreditation can also be withdrawn. A program’s accreditation 

status is made public, but to safeguard confidential information,16 specific 

problems with the program or its training sites are described in letters sent 

only to the sponsoring institution. Accrediting bodies have not been 

sending these letters to participating institutions. 

Accrediting bodies state that the quality of patient care must remain the 

highest priority of GME programs. Health care organizations that provide 

GME must ensure that qualified staff physicians supervise residents and 

that the same standards for the quality and safety of patient care apply 

when residents are involved in health care delivery as when they are not. 

GME accrediting bodies require that supervising physicians adjust the 

level of supervision to meet the educational goal of increasing residents’ 

 16ACGME classifies certain records as confidential to foster candor by residency programs, 

residents, and others as they submit information during the accreditation process. 
Page 9 GAO-03-625 VA’s Oversight of Resident Supervision 

competence by giving them appropriate opportunities to assume greater 

independence in their patient-care activities, that is, allowing residents to 

assume graduated responsibilities. The supervising physician relies on his 

or her professional judgment and knowledge of the patient’s medical 

condition and the resident’s level of mastery to determine the degree of 

independence of the resident’s patient-care responsibilities. 

VA’s national policy on resident supervision is detailed in a handbook that 

establishes specific requirements for (1) the involvement of supervising 

physicians in the care provided by residents who diagnose, treat, or 

discharge patients and (2) the documentation of that involvement.17 These 

specific requirements apply to four domains of residents’ clinical 

activity—inpatient care, outpatient care, diagnostic and therapeutic 

procedures, and consultations—and provide guidelines for putting into 

practice GME accrediting bodies’ principles of resident supervision and 

graduated levels of responsibility. (See table 1 for an example of VA’s 

requirements for supervision in each of the four domains.) Experts on 

GME told us that the requirements in VA’s handbook are reasonable and 

appropriately consider the role of supervision in ensuring the quality of 

patient care and of resident education. Some of these experts described it 

as a best practice model. 

 17The most recent revision of the handbook was issued on October 25, 2001. 
Page 10 GAO-03-625 VA’s Oversight of Resident Supervision 

Table 1: Examples of Requirements from VA’s Resident Supervision Handbook 

Issued on October 25, 2001, by Domain of Residents’ Health Care Activities 

Domain 

Examples of requirements for supervision and its 

documentation 

Inpatient care The supervising physician must meet each new inpatient 

within 24 hours of admission (including weekends and 

holidays) and personally document that encounter in the 

patient’s medical record. Concurrence with, or 

modifications to, the resident’s diagnosis and treatment 

plan must be documented in the supervising physician’s 

progress note. 

Outpatient care The supervising physician must supervise the initial visit of 

each new patient to an outpatient clinic, either by seeing 

the patient or discussing the patient with the resident at 

that initial visit. Involvement of the supervising physician 

must be documented in the medical record. 

Diagnostic and therapeutic 

procedures 

When a resident is involved in the care of a patient who is 

to undergo an elective or scheduled procedure, the 

supervising physician is to write a preprocedural note that 

indicates the diagnosis and treatment plan. 

Consultations The supervising physician must meet with each patient 

who was seen by a resident for a consultation and 

document his or her personal evaluation in the patient’s 

medical record. 

Source: VA. 

VA does not have adequate procedures to determine whether residents at 

VA medical centers are supervised in accordance with its national 

requirements. For example, VA does not check whether each medical 

center involved in GME has adopted policies that are consistent with VA’s 

requirements for resident supervision. To learn what medical centers and 

networks do to monitor whether supervision is consistent with VA’s 

national requirements, VA requires that medical centers and networks 

submit annual reports on residency training. Medical centers’ reports filed 

for the 2000/2001 academic year indicate that most medical centers review 

some documentation of resident supervision, but few conduct 

comprehensive reviews. To obtain more complete information about the 

supervision residents receive, VA is planning to use external peer review 

to assess adherence to its requirements for documenting resident 

supervision. These plans have not been finalized. For example, as of May 

2003, VA had not decided whether reviewers would examine records from 

VA’s new outpatients. 

VA Lacks Adequate 

Procedures to 

Monitor 

Implementation of Its 

Supervision 

Requirements 
Page 11 GAO-03-625 VA’s Oversight of Resident Supervision 

VA does not know whether all its medical centers have adopted policies 

that are consistent with the specific requirements in its resident 

supervision handbook for the supervision of residents’ diagnosis, 

treatment, and discharge of patients. The director of each medical center 

involved in GME is to establish facility policies for resident supervision 

that fulfill the requirements in VA’s handbook, but VA requires a review of 

only one requirement involving the supervision of diagnostic and 

therapeutic procedures—the medical centers’ requirements for the 

minimal acceptable level of supervision for diagnostic and therapeutic 

procedures. Specifically, in situations in which the supervising physician is 

not in the operating or procedural suite, VA requires that the supervisor 

must, at a minimum, be immediately available in the facility or campus to 

provide direct supervision of the procedure if necessary.18 Network GME 

managers19 are supposed to review and approve this requirement; they are 

not required to report the results of their reviews to OAA. There is no 

separate OAA review of any of the requirements in medical centers’ 

supervision policies. 

We found that not all networks have completed the one required review 

and that medical centers’ policies are not always consistent with VA’s 

national policy. Of the 11 network GME managers we interviewed, 7 told 

us that they had completed this required review of the minimal 

requirements for supervision of procedures in medical center policies, but 

4 told us that they had not. We found that the requirement of a medical 

center in one of the four networks that had not conducted this review was 

less stringent than the requirement in VA’s handbook for supervision of 

diagnostic and therapeutic procedures. The written policy at this medical 

center stated that the supervising physician can be immediately available 

by telephone rather than requiring him or her to be immediately available 

in the facility or on campus.20 One network GME manager who did review 

this requirement for supervision of diagnostic and therapeutic procedures 

told us that in 2002, he identified three medical centers that had written 

 18VA’s requirements for the minimum level of supervision for diagnostic and therapeutic 

procedures do not apply to procedures performed in emergency situations, in which 

immediate action is necessary to save a patient’s life or prevent serious impairment of the 

patient’s health, or to procedures that are elements of routine and standard patient care, 

such as drainage of superficial abscesses. 

19These managers are known as network academic affiliations officers. 

20An official of this medical center told us in September 2002 that there had been no 

adverse patient outcomes associated with resident supervision during the preceding 2 

years. 

VA Does Not Determine 

Whether VA Medical 

Centers’ Policies Are 

Consistent with Its 

National Requirements for 

Resident Supervision 
Page 12 GAO-03-625 VA’s Oversight of Resident Supervision 

requirements for supervision of these procedures that were less stringent 

than the requirement in VA’s handbook and that he instructed each of 

these facilities to change its policy to be consistent with VA’s national 

requirement. 

To learn what medical centers and networks do to monitor whether 

supervision is consistent with VA’s resident supervision handbook, VA has 

required annual reports on residency training programs beginning with the 

1999/2000 academic year. Medical center managers are to provide 

narrative answers to specific open-ended questions about their monitoring 

processes as well as about the problems they identified and actions they 

took to address them for each of three areas of oversight. (See table 2.) 

These medical center reports are channeled through VA’s networks to 

OAA. Network officials are to review them and summarize the strengths 

and weaknesses of the medical centers’ GME programs in network-level 

annual reports, which are also submitted to OAA. 

VA Headquarters Monitors 

Medical Center and 

Network Oversight of 

Resident Supervision 

through Annual Reports 
Page 13 GAO-03-625 VA’s Oversight of Resident Supervision 

Table 2: Examples of Questions about Monitoring Processes from the Annual 

Report on Residency Training Programs Completed by Medical Centers and 

Networks 

Area of 

oversight 

Examples of questions to be 

completed by medical center 

managers 

Examples of questions to be 

completed by network 

managers 

Supervision 

requirements 

• Describe your process for 

reviewing and monitoring 

medical center data collected 

for assessing resident 

supervision in the following 

areas: (1) inpatient admission, 

continuing care, and discharge 

supervision; (2) outpatient visit 

supervision; and (3) supervision 

of diagnostic and therapeutic 

procedures and consultations.a

• Describe any network-level 

process for review of medical 

center data collected for 

assessing adherence to VA’s 

educational supervision 

requirements and the results of 

such review in the following 

areas: (1) inpatient admission, 

continuing care, and discharge 

supervision; (2) outpatient visit 

supervision; and (3) 

supervision of diagnostic and 

therapeutic procedures and 

consultations. 

Evaluations of 

resident 

supervision 

• Describe concerns of the 

accrediting bodies specific to 

VA clinical rotations.b

• Describe your process for 

obtaining and reviewing 

resident comments related to 

their VA clinical training 

experience.a

• Describe any network-level 

process for review of residents’ 

comments related to their VA 

clinical training experience and 

the results of such review. 

Patient care • Describe your process for 

reviewing and monitoring all 

incidents and risk eventsc with 

complications to ensure that the 

appropriate level of resident 

supervision occurred.a

• Describe any network-level 

review process for assessing 

incidents and risk eventsc

 to 

ensure that the appropriate 

level of resident supervision 

occurred and the results of that 

review. 

Source: VA. 

a

Medical centers are also asked to describe results of their reviews and action plans for correction or 

remediation of problems found. 

b

Medical centers are also asked to note each program’s accreditation status, summarize affiliate and 

VA responses to accrediting body concerns, and describe any corrective actions. 

c

Risk events include events that did result, or could have resulted, in an adverse outcome. 

These annual reports can provide managers with limited, but useful, 

information about the extent and quality of monitoring performed by 

medical centers, including whether medical centers monitor 

documentation or some other indication of supervision. Some medical 

centers and networks provided little detail in response to the annual 

reports’ open-ended questions. For example, not all medical centers 
Page 14 GAO-03-625 VA’s Oversight of Resident Supervision 

described which specific aspects of resident supervision they monitored. 

OAA used open-ended questions in part to accommodate differences 

among medical centers in the number and type of residents they train. 

VA officials have used information from annual reports to monitor medical 

center oversight of resident supervision. For example, one network GME 

manager followed up on a problem identified through a medical center 

annual report by requiring the medical center to submit an action plan for 

improving supervision of ophthalmology residents by the beginning of the 

2002/2003 academic year. An OAA official told us that analysis of these 

annual reports not only helped identify areas of vulnerability with 

residency programs, but also pointed to possible best practices. 

VA does not require its medical centers or networks to conduct systematic 

reviews of the documentation of resident supervision,21 and medical 

centers differ in the extent to which they monitor adherence to VA’s 

requirements for supervision. More than three-fourths of medical centers’ 

annual reports included a description of an independent review of the 

documentation of supervision of at least one aspect of care provided by 

residents, but most medical centers did not describe reviews of all four 

domains of residents’ health care activities.22 For each of three domains—

inpatient care, outpatient care, and diagnostic and therapeutic 

procedures—over half the medical centers described a process for an 

independent review of at least one element of the documentation of 

resident supervision, that is, a review by someone other than a physician 

with related supervisory responsibilities (see table 3). For example, the 

quality management office at one medical center reviews medical records 

each month to determine whether documentation indicates that inpatients 

were seen by supervising physicians within 24 hours of admission. As 

 21An OAA official told us that OAA does not require medical centers or networks to conduct 

comprehensive documentation reviews to avoid duplicating the cost and effort VA 

headquarters is expending to develop a plan for systemwide external peer review of 

supervision documentation. This plan will be addressed in the next section of this report. 

22OAA provided us with annual reports from 114 of the approximately 130 medical centers 

that were allocated VA-funded residency slots during the 2000/2001 academic year. These 

were all the medical center annual reports for the 2000/2001 academic year OAA had 

received as of June 18, 2002. Before giving these reports to us, OAA redacted them to 

remove identifying information such as the names of medical centers and sponsoring 

institutions. We analyzed these reports to determine whether the medical centers described 

a systematic, independent review of the documentation of resident supervision in a sample 

of medical records. 

Most VA Medical Centers 

Monitor Some 

Documentation of 

Resident Supervision, but 

Few Conduct 

Comprehensive Reviews 
Page 15 GAO-03-625 VA’s Oversight of Resident Supervision 

shown in table 3, few medical centers, however, described such a process 

for review of supervisory documentation when residents provide 

consultations to patients’ primary physicians. 

Table 3: Number of VA Medical Centers That Reported Monitoring Some Aspect of 

the Documentation of Resident Supervision, by Domain of Residents’ Health Care 

Activities 

Inpatient 

care 

Outpatient 

care

Diagnostic 

and 

therapeutic 

procedures Consultations

Explicit independent 

review process 

describeda

77 58 65 21

No explicit independent 

review process 

describedb

26 47 42 86

Blank, missing, or 

reported to be not 

applicable 11 9 7 7

Total 114 114 114 114

Source: VA. 

Notes: GAO analysis of VA medical center 2000/2001 academic year annual reports on resident 

supervision submitted to VA by June 18, 2002. We considered the review process to be independent 

if the description indicated that documentation is reviewed by someone other than a physician with 

related supervisory responsibilities. 

a

Includes all descriptions of systematic independent review processes of one or more aspects of 

documentation, as well as less systematic review processes and reviews of only some services 

provided by residents. 

b

Includes medical centers that stated that they had no process for that domain or for which the 

description included insufficient information to determine whether the process was independent and 

systematic. 

In addition, medical centers’ annual reports did not always include clear, 

detailed descriptions of the documentation requirements they monitor. 

Few specifically mentioned monitoring particular VA-wide requirements, 

such as the requirement for documentation of supervisory involvement at 

the time of each new outpatient’s first visit. In some instances, medical 

centers described a less systematic review process or one that was used 

for only some services provided by residents. For diagnostic and 

therapeutic procedures, for example, some medical centers described 
Page 16 GAO-03-625 VA’s Oversight of Resident Supervision 

processes for reviewing only selected procedures, such as endoscopies or 

major surgeries.23

About half of the 91 medical centers that reported having an independent 

review process indicated they found deficiencies with the documentation 

of resident supervision, and all but one discussed actions they took to 

correct these problems.24 For example, officials from one medical center 

told us that they implemented a program to discipline individual 

physicians who consistently do not meet the medical center’s 

requirements for documenting supervision. The acting chief of staff there 

told us that during the 2001/2002 academic year, three physicians had each 

been suspended without pay for 1 day for not consistently meeting 

documentation requirements and that there had been significant 

improvement in the documentation of resident supervision since this 

disciplinary program went into effect. This medical center has also 

developed a strategy for linking contract physicians’ pay to their provision 

and documentation of supervision.25

Documentation reviews have proven useful in identifying inadequate 

supervision. We identified three medical centers that described in their 

annual reports finding evidence of inadequate resident supervision 

through their documentation reviews. In their annual reports, two of these 

three medical centers stated that there were no adverse patient events 

involving resident supervision. The third did not state whether there had 

been any adverse patient outcomes. In the first instance, the medical 

center reported that its review of documentation indicated that some staff 

physicians provided a “low level” of supervision to residents in the 

inpatient surgical setting. Medical center officials responded by meeting 

with those physicians and conducting a follow-up review to monitor the 

 23Seven of the 11 medical center GME managers we interviewed told us that since 

preparing their 2000/2001 annual reports, their medical centers have implemented or are 

developing additional reviews of the documentation of supervision. For example, one 

medical center that had not reviewed documentation of resident supervision in inpatient 

settings during the 2000/2001 academic year began reviewing that documentation on a 

quarterly basis during the 2001/2002 academic year. 

24Insufficient documentation does not necessarily indicate a lack of supervision. For 

example, some medical centers reported that supervision was documented, but not in a 

way that met VA’s requirements, and others reported that interviews with staff indicated 

that appropriate supervision had occurred, although documentation was lacking. 

25In addition to employing salaried physicians, VA medical centers sometimes use contracts 

to obtain the services of medical specialists. 
Page 17 GAO-03-625 VA’s Oversight of Resident Supervision 

level of supervision. In the second instance, the medical center reported 

that its supervision of residents was generally satisfactory, but that it had 

found through its documentation review one episode in which the 

attending surgeon had left the city during a procedure that he was 

supposed to be supervising. This medical center reported that the surgeon 

was formally reprimanded. In the third instance, a medical center reported 

that through its documentation review, it identified two specialties—

urology and plastic surgery—for which it wanted to increase the number 

of procedures performed with the staff physician physically present and 

directly involved in the surgery. The medical center reported that its 

management was working with the surgery service chief to achieve this 

goal. 

We also identified a few medical centers that described independent 

processes for monitoring resident supervision that went beyond reviewing 

documentation. One medical center, for example, reported that staff in its 

intensive care unit are required to report to the nurse manager any 

situation they observe in which the supervision of a resident was 

inappropriate. 

In addition to monitoring processes established by medical centers, five of 

VA’s networks indicated in their 2000/2001 annual reports that they had a 

networkwide process for assessing adherence to one or more VA 

requirements for documentation of resident supervision. For example, two 

networks stated that they monitor the documentation of supervising 

physicians’ involvement in the care of inpatients within 24 hours of 

admission and another network assesses documentation of the 

supervision of high-risk procedures. Two other networks reported they are 

developing networkwide monitoring processes. 
Page 18 GAO-03-625 VA’s Oversight of Resident Supervision 

To obtain more complete information about the extent to which its 

requirements for supervision are being followed, VA has begun to test its 

plans to monitor adherence through external peer review of the 

documentation of supervision. External peer reviewers would examine a 

sample of medical records from each medical center involved in GME to 

determine whether they include required documentation of supervision.26

Although documentation does not provide full information about the 

extent or quality of supervision, it can provide VA oversight officials with 

important information about whether supervisors were involved in patient 

care. We compared the instructions that external reviewers would follow 

with the requirements for supervision in VA’s handbook and found that the 

instructions would allow reviewers to assess adherence to most of VA’s 

key documentation requirements in the four domains of residents’ health 

care activities. For example, if a resident participated in the care of an 

inpatient or an outpatient during the current academic year, the external 

reviewer is to determine whether documentation of supervision in the 

patient’s medical record met the requirements in VA’s national handbook. 

Reviewers are also to assess documentation of the supervision of residents 

who performed diagnostic or therapeutic procedures or provided 

consultations to other physicians. Results from each medical center are to 

be provided to that medical center, as well as to headquarters managers. 

External peer review of documentation of supervision in medical records 

will be facilitated by features of VA’s computerized patient record 

system.27 For example, the system automatically records the date and time 

of notes; it also has the capacity to require that notes written by a resident 

be co-signed by the supervising physician, in which case the note is not 

considered complete until the required co-signature has been entered. In 

addition, supervising physicians with whom we spoke noted that 

 26As part of its broader quality management process, VA began its External Peer Review 

Program in 1995. Through this program, trained reviewers from outside VA examine 

documentation from a sample of medical records from each medical center to determine 

whether specific health care activities, such as influenza immunization, have occurred. 

These data have allowed VA to monitor its progress in meeting specific health care 

objectives. 

27The core features of VA’s computerized patient record system, which was developed to 

support its health care mission, have been installed at all VA medical centers, although 

medical centers differ in the extent to which it is used. External reviewers will review 

either electronic or paper records, whichever are available. 

VA’s Plans to Use External 

Peer Review to Monitor 

Documentation of 

Supervision Have Not 

Been Finalized 
Page 19 GAO-03-625 VA’s Oversight of Resident Supervision 

immediate and easy access to legible information facilitates supervisors’ 

review of residents’ activities.28

VA is in the early stages of testing its procedures for external peer review 

of the documentation of resident supervision, and a VA official told us that 

this effort is a high priority. A pilot test of portions of the inpatient 

assessment methodology was conducted from October 2001 through June 

2002 on a sample of almost 10,000 medical records. That pilot test 

indicated that the central database used to select the sample of medical 

records does not include information about which patients were seen by 

residents. As a result, reviewers were unable to select an appropriate 

sample of medical records. Until this problem is resolved, VA cannot 

implement its plans for external peer review of resident supervision. OAA 

has worked with other headquarters offices to revise VA’s information 

technology software to ensure that this database contains information 

about whether patients’ physicians were residents. VA expects to 

implement this revision to its software by July 2003. The pilot test did not 

indicate any other obstacles to implementing the portion of the plan for 

reviewing documentation of resident supervision in inpatient settings. 

Pilot tests of methods for assessing documentation of outpatient care, 

diagnostic and therapeutic procedures, and consultations will not begin 

until patients seen by residents can be clearly identified through the 

central database. 

One unresolved issue that will affect the usefulness of the external review 

of supervision documentation in the outpatient setting involves selection 

of the sample of medical records. The two options under consideration are 

relying on the main outpatient sample used for VA’s other external peer 

reviews or developing a sample specifically for review of the 

documentation of supervision. The main outpatient sample in any given 

year includes only patients who have received primary health care from 

VA in the past and excludes most new patients who began obtaining health 

care through VA within the preceding year29—a group that has greatly 

 28The Association of American Medical College’s Joint Committee of the Group on Resident 

Affairs and Organization of Resident Representatives has reported that computerized 

medical records can enhance the safety of patient care in teaching hospitals. 

29VA told us that it excludes new patients from its main outpatient sample to facilitate 

comparison of its performance measures to those from the National Committee for Quality 

Assurance’s Health Plan Employer Data and Information Set, which collects data from 

private-sector patients who have been enrolled in a health plan for two consecutive years. 
Page 20 GAO-03-625 VA’s Oversight of Resident Supervision 

expanded in recent years.30 Without a sample of records from new 

patients, it will not be possible to assess adherence to VA’s requirement 

for supervisory involvement during a veteran’s first outpatient visit. An 

OAA official told us that developing an additional sample of outpatient 

records for review of documentation of supervision, distinct from the main 

outpatient sample used for other purposes, would add to the expense of 

the review. As of May 2003, VA had not made a decision about which 

sample to use. 

VA is making efforts to obtain consistent access to information provided 

by accrediting bodies and residents about the quality of resident 

supervision in VA medical centers. VA has taken steps to gain direct 

access to the letters accrediting bodies send to sponsoring institutions to 

describe concerns about GME programs. VA headquarters also developed 

a survey to obtain feedback from residents, but cannot send it to a random 

sample of residents because VA does not have a complete list of its 

residents. VA is improving its ability to obtain that information. According 

to their annual reports for the 2000/2001 academic year, most VA medical 

centers that provide GME have some procedure for obtaining feedback 

from residents. 

VA does not currently have direct access to accreditation letters that 

contain reviews of the GME programs sponsored by VA medical centers’ 

affiliates. These letters document concerns about residents’ education or 

clinical experience that the GME program must address to retain 

accreditation. Timely access to the information in these letters can allow 

medical centers to take corrective actions. Until early 2000, ACGME sent 

copies of its accreditation letters to OAA,31 and OAA made VA support for 

residency slots contingent on VA medical centers’ taking action to correct 

identified problems. In 2000, however, ACGME adopted new policies to 

safeguard confidential accreditation information. As a result, ACGME 

 30From fiscal year 1996 to fiscal year 2002, the number of patients who received health care 

from VA increased from about 2.9 million to 4.7 million. 

31During the time when OAA received copies of ACGME’s accreditation letters, OAA did 

not have direct access to accreditation letters from the American Osteopathic Association, 

which accredits a small number of the GME programs in which VA medical centers 

participate. 

VA Is Acting to Obtain 

Information about 

Supervision from 

Accrediting Bodies 

and Residents 

VA Is Taking Steps to Gain 

Access to Accreditation 

Reviews of Its Affiliates’ 

GME Programs 
Page 21 GAO-03-625 VA’s Oversight of Resident Supervision 

stopped sending the letters to VA, instead sending these letters only to the 

institution that sponsors the GME program. 

Without direct access to ACGME accreditation letters, VA medical centers 

are dependent on sponsoring institutions to inform them of concerns 

about the GME programs in which VA participates, and we learned of one 

instance in which a sponsoring institution did not do so when ACGME 

notified it of problems. Officials from a medical center told us that the 

sponsoring institution of a thoracic surgery program did not tell them that 

ACGME had previously identified multiple problems with the program 

until ACGME decided, in September 2002, to withdraw the program’s 

accreditation. ACGME did not cite any problems with the VA rotation. 

Nonetheless, unanticipated withdrawal of a program’s accreditation can 

affect a medical center’s educational and patient care missions. In this 

case, the VA medical center will lose one full-time advanced surgical 

resident in July 2003 and had to hire a physician’s assistant to provide 

some of the services that had been provided by the resident. 

Most medical centers indicated in their 2000/2001 annual reports that their 

GME sponsors had shared information from accreditation letters, and 

these annual reports provided network and headquarters officials with 

information about accrediting bodies’ concerns and medical centers’ 

corrective actions. Fifty-six medical centers stated that accrediting bodies 

had identified concerns about VA rotations in 145 of the more than 1,900 

GME programs in which VA is involved. Concerns about 17 of these 

programs related to resident supervision.32 For example, according to one 

medical center’s annual report, ACGME concluded that residents required 

more direct supervision during certain oncology rotations. Medical centers 

reported that they had taken corrective action in response in all but one 

instance. In this case, the accrediting body expressed concern that the VA 

medical center had provided inadequate supervision and teaching in its 

physical medicine and rehabilitation rotation, but the medical center did 

not describe a corrective action in its annual report. 

We found that when OAA had direct access to ACGME accreditation 

letters—through early 2000—it took action to ensure that VA medical 

centers knew of and responded to ACGME concerns about VA rotations. 

 32The annual reports indicated that most concerns noted by GME accrediting bodies did not 

involve resident supervision, but instead involved other problems, such as insufficient 

ancillary staff or inadequate rooms where residents can rest while they are on-call in the 

medical center. 
Page 22 GAO-03-625 VA’s Oversight of Resident Supervision 

Our review of OAA’s correspondence about accreditation issues covering 

a period from late 1998 through early 2000 indicated that ACGME 

mentioned concerns that were specific to VA rotations in its letters about 

17 GME programs. In 6 of these cases, ACGME cited a concern about the 

adequacy of resident supervision. For example, ACGME determined that 

ophthalmology residents at one VA medical center had not been given 

clear information about lines of supervisory responsibility. On receipt of 

these letters, OAA contacted the participating VA medical center. Three of 

the medical centers submitted documents to substantiate a resolution to 

the problem within 2 months of hearing from OAA. In the other three 

cases, OAA asked VA’s chief consultant for the relevant medical specialty 

(such as the Chief Consultant for Ophthalmology) to assess the situation. 

In each case, the consultant reported to OAA that a resolution had been 

achieved. For example, the consultant reported that the ophthalmology 

program cited for unclear lines of supervision was preparing a written 

document to clarify supervisory responsibilities. 

OAA has taken steps to arrange for renewed direct access to ACGME 

accreditation letters. As part of that effort, VA issued a revised policy on 

confidential documents in July 2002 to make sure that accreditation 

reviews would be treated confidentially. In February 2003, VA signed a 

memorandum of understanding with ACGME that lays the foundation for 

OAA to receive copies of accreditation letters. According to this 

memorandum, VA must now obtain revised affiliation agreements between 

VA medical centers and GME sponsors that authorize ACGME to provide 

OAA with its accreditation letters. VA is taking steps to ensure that these 

revised agreements will be in place by July 2004. OAA has come to a 

similar agreement with the American Osteopathic Association. 

As a further step to obtain information about, and monitor responses to, 

GME issues—including accreditation concerns—OAA reissued a policy 

requiring VA medical centers to establish an affiliation partnership council 

and submit minutes of council meetings to OAA.33 The council is to include 

representatives of the medical center and its academic affiliate or affiliates 

and is to advise VA managers as they work to meet educational 

accreditation requirements and correct deficiencies or resolve problems. 

 33By reissuing this policy, OAA reasserted its requirement for submission of minutes, which 

it had not consistently enforced in recent years. 
Page 23 GAO-03-625 VA’s Oversight of Resident Supervision 

A mechanism OAA uses to obtain standardized information about 

residents’ views on the quality of their supervision and other aspects of 

their training is its Learners’ Perceptions Survey, which was first 

distributed in March 2001.34 The survey asks residents to indicate their 

satisfaction with the supervision they received from VA faculty by rating 

supervising physicians’ teaching ability, accessibility/availability, and 

approachability/openness, as well as overall satisfaction with VA clinical 

faculty. Residents are also asked to evaluate their satisfaction with the 

degree of supervision and degree of autonomy they experienced. 

In 2001 and 2002, VA headquarters could not send the survey to a random, 

representative sample of residents from each of its medical centers 

involved in GME because it did not have a complete list of its trainees. 

OAA was able to obtain feedback from many residents who did receive the 

survey35 and gave those results to medical centers and networks. OAA is 

taking steps to capture each trainee’s name and address in its automated 

and centrally accessible information system and expects to implement this 

procedure in July 2003. Once VA has a full registry of its trainees, OAA 

plans to send the survey to a representative sample of residents in 

different medical specialties that will include residents from all VA 

medical centers involved in GME. 

Medical centers’ annual reports can provide network and headquarters 

officials with additional information about concerns expressed by 

residents and steps taken to address those concerns. According to the 

annual reports for the 2000/2001 academic year, most VA medical centers 

used VA’s nationwide Learners’ Perceptions Survey or another 

mechanism, such as residents’ confidential evaluations obtained by 

sponsoring institutions, to obtain feedback about supervision. About half 

of the 109 medical centers whose annual reports indicate that they had a 

process for obtaining residents’ feedback said that residents had concerns 

about their VA rotations. None of these concerns, however, involved the 

adequacy of supervision. 

 34VA’s Learners’ Perceptions Survey is designed to obtain information about the 

perceptions of all trainees who work within the VA system, including residents, student 

nurses, and psychology interns. Data from this survey are used to assess VA’s systemwide 

performance measure involving trainees’ ratings of their VA educational experience. In 

addition to GME, VA provides training in more than 40 associated health disciplines. 

35During 2001, surveys were sent to 3,338 residents and returned by 1,775. During 2002, 

surveys were sent to 6,084 residents and returned by 2,622. 

VA Is Improving Its Ability 

to Obtain Feedback from a 

Representative Group of 

Residents 
Page 24 GAO-03-625 VA’s Oversight of Resident Supervision 

VA headquarters, network, and medical center officials use information 

from VA’s programs for monitoring the quality and outcomes of patient 

care to identify and correct problems with resident supervision. VA’s 

monitoring programs include its new Patient Safety Program and NSQIP. 

Reviews of paid tort claims by VA’s Chief Patient Care Officer provide 

another mechanism for identifying problems with resident supervision. 

OAA monitors medical centers’ use of these programs through the annual 

reports on residency training. In their annual reports for the 2000/2001 

academic year, most medical centers indicated that they monitor patient 

care information to determine whether resident supervision affected the 

quality or outcomes of patient care. 

The system for reporting adverse events and close calls established by 

VA’s Patient Safety Program has the potential to capture information about 

instances in which inadequate resident supervision contributed to 

heightened risk of adverse health care outcomes. Based on analysis of the 

17,000 reports of adverse events and close calls filed with VA’s National 

Center for Patient Safety as of April 2002, its director estimated that 

resident supervision was mentioned—in any context—in less than 0.1 

percent of the incidents reported by VA medical centers and that 

inadequate supervision was a causal factor in very few of those cases.36

Analyses of postoperative outcomes recorded in the NSQIP database, 

including mortality and morbidity, provide VA with a way to study the 

effects of residents’ involvement in surgical procedures. NSQIP personnel 

analyze nationwide data from major surgeries, provide site-specific reports 

to medical centers and networks, and conduct site visits at medical 

centers.37 A NSQIP official told us that these data are routinely examined 

for signs that supervision of residents might be inadequate. For example, 

NSQIP analysts review the data to ensure that residents are not 

performing surgeries that are more advanced than would be appropriate 

for their level of training. In addition to reviewing NSQIP reports, 

headquarters officials who oversee VA’s surgical services monitor the 

 36We did not independently verify this estimate. 

37Each VA medical center that performs major surgeries receives an annual report that 

reports its mortality and morbidity outcomes, adjusted for risk factors, in comparison to 

VA’s other medical centers, along with suggestions for improvement. Networks also receive 

these reports. In addition, a team of experts visits medical centers with mortality rates that 

are consistently higher than expected to identify problems and recommend improvements. 

VA Uses Its Programs 

for Monitoring Patient 

Care to Identify and 

Correct Problems 

with Resident 

Supervision 
Page 25 GAO-03-625 VA’s Oversight of Resident Supervision 

frequency with which supervising physicians are in the operating or 

procedural suite when residents perform surgeries.38

Medical center and network officials have used NSQIP reports to help 

monitor resident supervision. For example, a team of experts selected by 

NSQIP visited one medical center at its request in February 2002 to help it 

evaluate the efficiency of its operating rooms. During its visit, the team 

noted inadequate supervision of surgeries performed by urology 

residents.39 The medical center corrected this problem by arranging for 

urologists to spend more time at the medical center and ensuring that they 

understood VA’s requirements for supervision. In another instance, a 

network GME manager observed that NSQIP data indicated that 

orthopedic surgery outcomes at a particular medical center were less 

favorable than expected. After a site visit, network officials concluded that 

the medical center could not support complex surgeries and determined 

that continued training of orthopedic residents at that medical center 

would require a decrease in the complexity of cases and greater 

involvement by supervising physicians. When the sponsoring institution 

decided that the medical center would not meet its training needs under 

those conditions, VA officials chose to transfer patients with complex 

surgical needs to VA’s tertiary hospitals in the network and shift its two 

VA-funded residency slots in orthopedic surgery to a different VA medical 

center. 

Researchers using the NSQIP database have studied ways in which 

participation in GME affects postoperative outcomes. To determine 

whether residency training places surgical patients at risk for worse 

outcomes, researchers using the NSQIP database40 compared risk-adjusted 

mortality rates in VA’s teaching and nonteaching hospitals and found that 

they did not differ, although the patients who underwent surgeries at 

teaching hospitals had a higher prevalence of risk factors, underwent more 

complex operations, and had longer operation times. Morbidity rates were 

 38The computer software used in VA medical centers for recording information about 

surgical procedures allows the generation of hospital reports that indicate the level of 

supervision provided for surgical procedures. Quarterly reports submitted to the Surgical 

Service at VA headquarters also include this information. 

39There was no evidence that any adverse patient safety events resulted from inadequate 

supervision of urology residents. 

40Shukri F. Khuri and others, “Comparison of Surgical Outcomes Between Teaching and 

Nonteaching Hospitals in the Department of Veterans Affairs,” Annals of Surgery, vol. 234, 

no. 3 (2001). 
Page 26 GAO-03-625 VA’s Oversight of Resident Supervision 

higher in teaching than nonteaching hospitals for some surgical specialties 

that were studied.41 On the basis of their analyses, the authors suggested 

that differences in morbidity rates could reflect incomplete adjustment for 

risks, such as severity of illness, or the more complex systems of managing 

and coordinating care that characterize teaching hospitals, and not 

necessarily the involvement of residents. Another study begun in 

September 2001 is designed to use the NSQIP database to clarify the 

relationship between residents’ working conditions and surgical 

outcomes, with data from 90 VA hospitals and 3 nonfederal hospitals in 

which surgical residents are trained. 

Tort claims also provide information that VA uses to identify problems 

with resident supervision that affected patient care. Review of paid tort 

claims by VA’s Chief Patient Care Services Officer resulted in clarification 

of VA’s written requirements for resident supervision when patients are 

admitted to inpatient units. In the specific case that led to this change, a 

supervising physician did not come to the hospital during a weekend to 

see a patient who had been admitted by a resident; the patient died on 

Monday. At that time, the resident supervision policy of the VA hospital in 

which the incident occurred did not specifically require supervising 

physicians to come in on weekends. As a result of this case, in October 

2001 an explicit reference to weekends and holidays was added to the 

handbook’s requirement that each new inpatient be seen by the 

supervising physician within 24 hours of admission. 

OAA monitors incidents in which resident supervision contributed to 

adverse events or patient risks through the annual reports it requires from 

medical centers. In their 2000/2001 annual reports on residency training, 

all but 11 of 114 medical centers indicated that they monitored patient 

safety events associated with residents.42 They used a variety of processes 

to collect this information, including root cause analyses and tort claim 

reviews, as well as additional processes such as mortality and morbidity 

conferences and reviews triggered by unexpected events, such as 

 41NSQIP defines morbidity as the occurrence of any one or more of 20 specific 

postoperative adverse events such as deep wound infection, pneumonia, or stroke within 

30 days of the operation. Morbidity rates were higher in teaching than nonteaching 

hospitals for general surgery, orthopedics, urology, and vascular surgery, but did not differ 

significantly for otolaryngology, neurosurgery, or thoracic surgery. 

42Medical centers that did not describe a process for monitoring patient safety events that 

involve residents either left the section on patient safety events blank or did not describe 

systematic review processes that are specific to incidents involving residents. 
Page 27 GAO-03-625 VA’s Oversight of Resident Supervision 

readmission within 10 days of discharge from the medical center. Annual 

reports indicated that reviews of at least 18 actual or potential adverse 

patient outcomes at a total of 14 medical centers identified resident 

supervision as a possible contributing factor or led medical center officials 

to strengthen supervision to minimize the chance of future problems. For 

example, one medical center established a requirement for greater 

involvement by supervising physicians before a resident initiates 

chemotherapy orders. Medical centers described taking corrective actions 

in response to these reviews. 

VA cannot assure that the residents who provide care in its facilities 

receive adequate supervision because its current procedures for 

monitoring supervision are insufficient. To oversee the supervision of its 

residents, VA needs various types of information, including information 

regarding supervising physicians’ adherence to VA’s requirements for 

resident supervision, accrediting bodies’ and residents’ concerns about 

supervision, and whether the quality or outcomes of patient care indicate 

problems with supervision. Systematic monitoring of each of these types 

of information would help ensure that problems with resident supervision 

are detected and corrected by the various officials of VA medical centers 

and affiliated institutions who have responsibilities for residents’ 

activities. 

Although VA issued a handbook that established specific standards for 

resident supervision, VA does not know what its medical centers’ 

supervision requirements are and does not ensure that its national 

requirements are adopted at each medical center where residents train. 

Moreover, VA does not know whether the supervision its residents receive 

adheres to its national requirements. VA’s current plans for external peer 

review of documentation have the potential to enhance its oversight 

capability, but these plans have not been finalized. For example, as of May 

2003, VA had not decided whether external reviewers would examine 

documentation of supervision for VA’s new outpatients, who make up a 

significant and growing number of VA’s patients. Including these new 

outpatients in the external review could help ensure adequate supervision 

of residents during a patient’s first visit to VA. 

To further improve its oversight of resident supervision, VA will need to 

complete its initiatives to obtain timely access to evaluations by 

accrediting bodies and residents. VA will also need to continue to take 

advantage of its programs for monitoring the quality and outcomes of 

patient care. VA officials have generally acted to improve supervision 

Conclusions 
Page 28 GAO-03-625 VA’s Oversight of Resident Supervision 

when faced with evidence of problems, and better access to information 

will enhance their ability to monitor and improve resident supervision. 

By strengthening its oversight capabilities, VA could help promote both 

the quality of the health care in its facilities and the education its residents 

receive. As the largest provider of residency training sites in the United 

States, VA’s actions to enhance the quality of resident supervision and its 

oversight will have benefits beyond the VA health care system. 

We recommend that the Secretary of Veterans Affairs direct the Under 

Secretary for Health to take steps to improve VA’s oversight of the 

supervision of residents by 

• ensuring that all VA medical centers that provide GME adopt and adhere 

to the requirements for resident supervision established in VA’s handbook 

and 

• ensuring that external peer review of documentation of resident 

supervision includes examination of records from VA’s new outpatients. 

In written comments on a draft of this report, VA agreed with our findings 

and our recommendations. VA said our report described many steps it has 

already taken that would help assure systematic implementation of its 

national resident supervision policies and adequate headquarters oversight 

of resident supervision. In concurring with our recommendation to ensure 

that all VA medical centers that provide GME adopt and adhere to 

requirements for resident supervision established in its handbook, VA 

indicated its intention to monitor compliance with policy requirements 

and highlight those requirements in a memorandum to network officials. 

In concurring with our recommendation to ensure that external peer 

review of documentation of resident supervision includes examination of 

records from its new outpatients, VA indicated that it would develop a 

strategy to identify new outpatients who were seen by a resident. It stated 

that it expects to draw its first sample of records from outpatients, 

including new outpatients, in the second quarter of fiscal year 2004. VA 

also reported that it completed a revision of its centralized patient 

information database. This revision was necessary to allow selection of an 

appropriate sample of inpatient records for external peer review. VA’s 

comments are in appendix II. 

Recommendations for 

Executive Action 

Agency Comments 
Page 29 GAO-03-625 VA’s Oversight of Resident Supervision 

We are sending copies of this report to the Secretary of Veterans Affairs, 

appropriate congressional committees, and other interested parties. We 

will also make copies available to others who are interested upon request. 

In addition, the report will be available at no charge on the GAO Web site 

at http://www.gao.gov. 

If you or your staff have any questions, please contact me at (202) 

512-7101. An additional contact and the names of other staff members who 

made contributions to this report are listed in appendix III. 

Cynthia A. Bascetta 

Director, Health Care—Veterans’ 

 Health and Benefits Issues 
Appendix I: Scope and Methodology 

Page 30 GAO-03-625 VA’s Oversight of Resident Supervision 

To do our work, we examined oversight of resident supervision at each of 

the Department of Veterans Affairs (VA) Veterans Health Administration’s 

three organization levels—headquarters, networks, and medical centers. 

Our work covered VA’s oversight of resident supervision and did not 

include an evaluation of the quality of care provided by residents or the 

quality of the supervision provided to residents. To assess oversight by 

VA’s headquarters officials, we reviewed documents and interviewed 

officials from VA’s Office of Academic Affiliations (OAA), Office of Patient 

Care Services, National Center for Patient Safety, Office of Quality and 

Performance, and Office of Information. We analyzed VA’s plans to have 

external peer reviewers examine documentation of supervision and 

compared the instructions the reviewers are to be given with VA’s 

requirements for supervision. 

To assess oversight of resident supervision by network officials, we 

analyzed each network’s annual report to OAA on resident supervision 

covering the 2000/2001 academic year.1

 These were the most recent annual 

reports available at the time. We did not assess the accuracy of 

information provided in these reports. We also interviewed network GME 

managers (known as network academic affiliations officers) from a 

sample of 11 of VA’s 21 regional networks of health care facilities and 

analyzed documents they provided (see table 4). We used a stratified 

random sampling strategy to ensure variation in the number of VA-funded 

residency slots among the selected networks.2

 Network 19 was included in 

our sample prior to randomization because it is the only network that did 

not summarize the information in its medical centers’ reports. Another 

network was excluded from our sample because it had been formed by the 

merger of two former networks in January 2002. Our results from these 11 

networks cannot be generalized to other networks. 

 1

These annual reports included separate reports from two networks that were merged in 

2002. 

2

Numbers of VA-funded residency slots were based on allocations for the 2001/2002 

academic year. 

Appendix I: Scope and Methodology 
Appendix I: Scope and Methodology 

Page 31 GAO-03-625 VA’s Oversight of Resident Supervision 

Table 4: VA Networks Included in Our Sample 

Network Number of VA-funded residency slotsa

 1 (Boston) 501.43

 3 (Bronx) 603.00

 6 (Durham) 348.30

10 (Cincinnati) 255.90

11 (Ann Arbor) 314.00

15 (Kansas City) 339.00

16 (Jackson) 671.95

18 (Phoenix) 305.70

19 (Denver) 230.00

21 (San Francisco) 383.02

22 (Long Beach) 729.50

Source: VA. 

a

The number of VA-funded residency slots allocated to networks during the 2001/2002 academic year 

ranged from 195.00 to 729.50. 

To assess oversight of resident supervision by medical center officials, we 

reviewed and analyzed 2000/2001 academic year annual reports to OAA on 

resident supervision. OAA provided us with 114 annual reports from the 

approximately 130 VA medical centers that were involved in GME during 

the 2000/2001 academic year after it removed identifying information, such 

as the names of medical centers, affiliates, and specific individuals. These 

were all the medical center annual reports for the 2000/2001 academic year 

that OAA had received as of June 18, 2002. We did not assess the accuracy 

of information in the annual reports. We also interviewed GME managers 

at 11 VA medical centers (see table 5) and analyzed their 2000/2001 

academic year annual reports on resident supervision (without redaction) 

and other documents. We used a stratified random sampling strategy to 

ensure that the medical centers we selected varied in the number of VAfunded residency slots they were allocated for the 2001/2002 academic 

year.3

 We also ensured that our sample included one medical center from 

each of the networks we had sampled and that the medical centers 

differed in the number of medical specialties in which their residents train. 

We did not review a systematically selected sample of medical centers’ 

 3

We excluded medical centers that received an allocation of 10 or fewer VA-funded 

residency slots or with fewer than three separate GME programs during the 2001/2002 

academic year from our sampling set, resulting in a possible set of 97 medical centers. 
Appendix I: Scope and Methodology 

Page 32 GAO-03-625 VA’s Oversight of Resident Supervision 

resident supervision policies. Our results from these 11 medical centers 

cannot be generalized to other medical centers. 

Table 5: VA Medical Centers Included in Our Sample 

VA medical center Network 

Number of 

VA-funded 

residency slotsa

Number of 

medical 

specialtiesb

White River Junction, Vt. 1 (Boston) 39.70 14

New York, N.Y. 3 (Bronx) 135.00 25

Hampton, Va. 6 (Durham) 45.00 7

Cleveland, Ohio 10 (Cincinnati) 112.40 23

Detroit, Mich. 11 (Ann Arbor) 79.00 26

St. Louis, Mo. 15 (Kansas City) 120.00 24

Biloxi, Miss. 16 (Jackson) 10.40 6

Tucson, Ariz. 18 (Phoenix) 93.01 21

Salt Lake City, Utah 19 (Denver) 110.50 25

Fresno, Calif. 21 (San Francisco) 42.00 4

Long Beach, Calif. 22 (Long Beach) 158.50 28

Source: VA. 

a

The number of VA-funded residency slots allocated to medical centers involved in GME for the 

2001/2002 academic year ranged from 0.60 to 218.00. 

b

The number of distinct medical specialties in which VA medical centers had residency slots during 

the 2001/2002 academic year ranged from 1 to 32. 

We also reviewed documentary and testimonial evidence from four 

medical centers that participate in internal medicine or general surgery 

GME programs that had received adverse accreditation decisions as of 

May 2002.4

 One of these—the Fresno VA Medical Center—was part of our 

sample of medical centers. Of the others, we visited the medical centers in 

West Haven, Connecticut and Gainesville, Florida and interviewed officials 

of the medical center in Albuquerque, New Mexico. We also spoke to 

officials of the institutions that sponsor these three GME programs. 

To obtain additional information about GME and VA’s residency training, 

we analyzed accreditation requirements of the Accreditation Council for 

Graduate Medical Education, American Osteopathic Association, and Joint 

Commission on Accreditation of Healthcare Organizations and 

interviewed officials of those bodies. We also interviewed representatives 

 4

Two of these programs are no longer under an adverse accreditation status. Reevaluation 

of the other two programs was not complete as of May 2003. 
Appendix I: Scope and Methodology 

Page 33 GAO-03-625 VA’s Oversight of Resident Supervision 

of professional associations that are involved in GME, including the 

American Board of Medical Specialties, American College of Surgeons, 

American Hospital Association, American Medical Association, American 

Medical Student Association, Association of American Medical Colleges 

and its Council of Deans, Association of Professors of Medicine, 

Committee of Interns and Residents, and Council on Graduate Medical 

Education, and we reviewed relevant documents issued by these groups. 

We interviewed representatives of physicians who teach internal medicine, 

ophthalmology, psychiatry, general surgery, orthopedic surgery, and 

urology—specialties for which a large number of VA medical centers 

provide residency slots. We also interviewed representatives of veterans’ 

service organizations. We reviewed published literature regarding the 

quality of care provided by residents. 

We conducted our work from September 2001 through June 2003 in 

accordance with generally accepted government auditing standards. 
Appendix II: Comments from the Department of Veterans Affairs 

Page 34 GAO-03-625 VA’s Oversight of Resident Supervision 

Appendix II: Comments from the Department 

of Veterans Affairs 
Appendix II: Comments from the Department of Veterans Affairs 

Page 35 GAO-03-625 VA’s Oversight of Resident Supervision 
Appendix II: Comments from the Department of Veterans Affairs 

Page 36 GAO-03-625 VA’s Oversight of Resident Supervision 
Appendix III: GAO Contact and Staff 

Acknowledgments 

Page 37 GAO-03-625 VA’s Oversight of Resident Supervision 

Helene F. Toiv, (202) 512-7162 

In addition to the person named above, key contributors to this report 

were Kristen J. Anderson, William D. Hadley, Martha Fisher, Krister 

Friday, and Donald Morrison. 

Appendix III: GAO Contact and Staff 

Acknowledgments 

GAO Contact 

Staff 

Acknowledgments 

(290096) 
The General Accounting Office, the audit, evaluation and investigative arm of 

Congress, exists to support Congress in meeting its constitutional responsibilities 

and to help improve the performance and accountability of the federal 

government for the American people. GAO examines the use of public funds; 

evaluates federal programs and policies; and provides analyses, 

recommendations, and other assistance to help Congress make informed 

oversight, policy, and funding decisions. GAO’s commitment to good government 

is reflected in its core values of accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older 

products. The Web site features a search engine to help you locate documents 

using key words and phrases. You can print these documents in their entirety, 

including charts and other graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 

daily. The list contains links to the full-text document files. To have GAO e-mail 

this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily 

E-mail alert for newly released products” under the GAO Reports heading. 

The first copy of each printed report is free. Additional copies are $2 each. A 

check or money order should be made out to the Superintendent of Documents. 

GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a 

single address are discounted 25 percent. Orders should be sent to: 

U.S. General Accounting Office 

441 G Street NW, Room LM 

Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800 

U.S. General Accounting Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 

GAO’s Mission 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Mail or Phone 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Public Affairs