United States Government Accountability Office

GAO Report to the Committee on Finance, 

U.S. Senate 

HOSPITAL QUALITY 

DATA 

HHS Should Specify 

Steps and Time Frame 

for Using Information 

Technology to Collect 

and Submit Data 

April 2007 

 

GAO-07-320 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

April 2007

HOSPITAL QUALITY DATA 

HHS Should Specify Steps and Time 

Frame for Using Information Technology 

to Collect and Submit Data 

Highlights of GAO-07-320, a report to the 

Committee on Finance, U.S. Senate 

Hospitals submit data in electronic

form on a series of quality 

measures to the Centers for 

Medicare & Medicaid Services 

(CMS) and receive scores on their 

performance. Increasingly, the 

clinical information from which

hospitals derive the quality data for 

CMS is stored in information 

technology (IT) systems. 

GAO was asked to examine 

(1) hospital processes to collect 

and submit quality data, (2) the 

extent to which IT facilitates

hospitals’ collection and 

submission of quality data, and 

(3) whether CMS has taken steps to 

promote the use of IT systems to 

facilitate the collection and 

submission of hospital quality data. 

GAO addressed these issues by 

conducting case studies of eight 

hospitals with varying levels of IT

development and interviewing

relevant officials at CMS and the 

Department of Health and Human

Services (HHS).

What GAO Recommends

GAO recommends that the

Secretary of HHS identify the 

specific steps the department plans

to take to promote the use of 

health IT for the collection and 

submission of data for CMS’s 

hospital quality measures and 

inform interested parties about 

those steps, the expected time 

frame, and associated milestones. 

In commenting on a draft of this 

report on behalf of HHS, CMS 

concurred with these 

recommendations. 

The eight case study hospitals used six steps to collect and submit quality 

data: (1) identify the patients, (2) locate information in their medical

records, (3) determine appropriate values for the data elements, (4) transmit 

the quality data to CMS, (5) ensure that the quality data have been accepted 

by CMS, and (6) supply copies of selected medical records to CMS to 

validate the data. Several factors account for the complexity of abstracting

all relevant information in a patient’s medical record, including the content 

and organization of the medical record, the scope of information and the 

clinical judgment required for the data elements, and frequent changes by

CMS in its data specifications. Due in part to these complexities, most of the 

case study hospitals relied on clinical staff to abstract the quality data.

Increases in the number of quality measures required by CMS led to

increased demands on clinical staff resources. Offsetting the demands 

placed on clinical staff were the benefits that case study hospitals reported 

finding in the quality data, such as providing feedback to clinicians and 

reports to hospital administrators. 

GAO’s case studies showed that existing IT systems can help hospitals 

gather some quality data but are far from enabling hospitals to automate the 

abstraction process. IT systems helped hospital staff to abstract information 

from patients’ medical records, in particular by improving accessibility to 

and legibility of the medical record. The limitations reported by officials in 

the case study hospitals included having a mix of paper and electronic 

records, which required staff to check multiple places to get the needed 

information; the prevalence of data recorded as unstructured narrative or 

text, which made locating the information time-consuming because it was 

not in a prescribed place in the record; and the inability of some IT systems 

to access related data stored in another IT system in the same hospital, 

which required staff to access each IT system separately to obtain related 

pieces of information. Hospital officials expected the scope and functionality 

of their IT systems to increase over time, but this process will occur over a 

period of years. 

CMS has sponsored studies and joined HHS initiatives to examine and 

promote the current and potential use of hospital IT systems to facilitate the 

collection and submission of quality data, but HHS lacks detailed plans, 

including milestones and a time frame against which to track its progress. 

CMS has joined efforts by HHS to promote the use of IT in health care,

including a Quality Workgroup charged with specifying how IT could 

capture, aggregate, and report inpatient and outpatient quality data. HHS 

plans to expand the use of health IT for quality data collection and 

submission through contracts with nongovernmental entities that currently 

address the use of health IT for a range of other purposes. However, HHS 

has identified no detailed plans, milestones, or time frames for either its 

broad effort to encourage IT in health care nationwide or its specific 

objective to promote the use of health IT for quality data collection. 

www.gao.gov/cgi-bin/getrpt?GAO-07-320.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Cynthia A. 

Bascetta, (202) 512-7101 or

BascettaC@gao.gov. 
Contents 

Letter 1

Results in Brief 5

Background 6

Hospitals Use Six Basic Steps to Collect and Submit Quality Data, 

Two of Which Involve Complex Abstraction by Hospital Staff 9 

Existing IT Systems Can Help Hospitals Gather Some Quality Data 

but Are Far from Enabling Automated Abstraction 22 

CMS Sponsored Studies and Joined Broader HHS Initiatives to 

Promote Use of IT for Quality Data Collection and Submission, 

but HHS Lacks Detailed Plans, Milestones, and Time Frame 27 

Conclusions 31

Recommendations for Executive Action 32 

Agency Comments and Our Evaluation 32 

Appendix I Medicare Quality Measures Required for Full 

Annual Payment Update 35 

Appendix II Data Elements Used to Calculate Hospital 

Performance on a Heart Attack Quality Measure 36 

Appendix III Tables on Eight Case Study Hospitals 38 

Appendix IV Scope and Methodology 45 

Appendix V Comments from the Centers for Medicare & 

Medicaid Services 50 

Appendix VI GAO Contact and Staff Acknowledgments 53 

 

 

Page i GAO-07-320 Hospital Quality Data
Tables 

Table 1: Case Study Hospital Characteristics 38 

Table 2: How Case Study Hospital Officials Described the Steps 

Taken to Complete Quality Data Collection and 

Submission 40

Table 3: Resources Used for Abstraction and Data Submission at 

Eight Case Study Hospitals 42 

Table 4: Electronic and Paper Records at Eight Case Study 

Hospitals 44

Figures 

Figure 1: Six Basic Steps for Hospitals Collecting and Submitting 

Quality Data 10 

Figure 2: Example of the Process for Locating and Assessing 

Clinical Information to Determine the Appropriate Value 

for One Data Element 18 

Figure 3: Data Elements Used to Calculate Hospital Performance 

on the Heart Attack Quality Measure That Asks Whether a 

Beta Blocker Was Given When the Patient Arrived at the 

Hospital 36

Abbreviations 

ACEI angiotensin-converting enzyme inhibitor 

AHIC American Health Information Community 

AHIMA American Health Information Management Association 

AHRQ Agency for Healthcare Research and Quality 

Alliance National Alliance for Health Information Technology 

AMI acute myocardial infarction 

APU Annual Payment Update 

ARB angiotensin receptor blocker 

Page ii GAO-07-320 Hospital Quality Data
CART CMS Abstraction & Reporting Tool 

CCHIT Certification Commission for Health Information 

 Technology 

CHI Consolidated Healthcare Informatics 

CMS Centers for Medicare & Medicaid Services 

CPOE computerized physician order entry 

DICOM Digital Imaging Communications in Medicine 

DRA Deficit Reduction Act of 2005 

FTE full-time equivalent 

H&P history and physical 

HCAHPS Hospital Consumer Assessment of Healthcare Providers 

 and Systems 

HHS Department of Health and Human Services 

HIMSS Healthcare Information and Management Systems Society 

HITSP Healthcare Information Technology Standards Panel 

ICD-9 International Classification of Diseases, Ninth Revision 

IFMC Iowa Foundation for Medical Care 

IT information technology 

JCAHO Joint Commission on Accreditation of Healthcare 

 Organizations 

LOINC Laboratory Logical Observation Identifier Name Codes 

LPN licensed practical nurse 

LVSD left ventricular systolic dysfunction 

MAR medication administration record 

MMA Medicare Prescription Drug, Improvement, and 

 Modernization Act 

MSA metropolitan statistical area 

NCDCP National Council on Prescription Drug Programs 

ONC Office of the National Coordinator for Health Information 

 Technology 

POS provider of services 

QIO quality improvement organization 

RN registered nurse 

SNOMED-CT Systematized Nomenclature of Medicine Clinical Terms 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately. 

Page iii GAO-07-320 Hospital Quality Data
United States Government Accountability Office

Washington, DC 20548 

April 25, 2007 

The Honorable Max Baucus 

Chairman 

The Honorable Charles E. Grassley 

Ranking Minority Member 

Committee on Finance 

United States Senate 

The Medicare Prescription Drug, Improvement, and Modernization Act of 

2003 (MMA) created a financial incentive for hospitals to submit to the 

Centers for Medicare & Medicaid Services (CMS) data that are used to 

calculate hospital performance on measures of the quality of care 

provided.1

 CMS established the Annual Payment Update (APU) program2

to implement that incentive. The APU program requires that participating 

hospitals submit these quality data3

 on a quarterly basis in order to avoid a 

reduction in their full Medicare payment update each fiscal year.4

 Although

the APU program was originally set to expire in 2007, the Deficit 

Reduction Act of 20055

 (DRA) made the APU program permanent. The act 

also raised the reduction6

 and required the Secretary of Health and Human 

Services (HHS) to increase the number of measures for which hospitals 

participating in the APU program would have to provide data in order to 

 1

See Pub. L. No. 108-173, § 501(b), 117 Stat. 2066, 2289-90. 

2

Throughout this report, we refer to CMS’s Reporting Hospital Quality Data for the Annual

Payment Update program as the “APU program.” 

3

Throughout this report, we refer to the data that hospitals submit to CMS that the agency

uses to calculate their performance on its quality measures as “quality data.” 

4

Most acute care hospitals (i.e., those paid under the Medicare inpatient prospective 

payment system) receive an annual payment update that increases the standardized 

payment amount that Medicare pays them per patient, based on projected increases in 

hospital operating expenses. For fiscal year 2007, 3,319 hospitals received their full 

payment update, about 95 percent of those eligible to participate in the APU program, and

the remaining 5 percent of eligible hospitals received a reduced annual payment update. 

CMS posts on a public Web site the performance scores that hospitals receive on quality 

measures derived from the data they submit. 

5

See Pub. L. No. 109-171, § 5001(a), 120 Stat. 4, 28-29. 

6

The magnitude of the reduction in the annual payment update for hospitals not submitting 

the quality data rose from 0.4 percentage points to 2 percentage points, starting in fiscal 

year 2007. 

Page 1 GAO-07-320 Hospital Quality Data
receive their full Medicare payment update.7

 CMS plans to continue 

expanding the number of required measures in future years.8

 Furthermore, 

DRA directed the Secretary to develop a plan to implement a value-based

purchasing program for Medicare that beginning in fiscal year 2009 would 

adjust payments to hospitals based on factors related to the quality of care

they provide. Such pay-for-performance programs are intended to 

strengthen the financial incentives for hospitals to invest in quality 

improvement efforts. 

Each quality measure consists of a set of standardized data elements, 

which define the specific data that hospitals need to submit to CMS. 

Hospitals determine a value for each data element of a measure for 

patients—Medicare and non-Medicare—who have a medical condition 

covered by the APU program, that is, heart attack, heart failure, 

pneumonia, or surgery. The values for the data elements consist of 

numerical data and other administrative and clinical information that are

obtained from the medical records of the patients.9

 For example, there are 

8 required quality measures for the heart attack condition, one of which is 

whether a beta blocker was given to the patient upon arrival at the 

hospital.10 This single measure, in turn, consists of 11 data elements, 

including administrative data elements, such as the patient’s date of arrival 

at the hospital, and clinical data elements, such as whether the patient 

received a beta blocker within 24 hours after hospital arrival (see app. II).

The values entered for data elements are used to calculate hospital 

performance on the 21 quality measures that are in effect as of fiscal 

 7

Initially, CMS designated 10 required quality measures under the APU program that

applied to patients treated for heart attacks, heart failure, or pneumonia. In accordance 

with DRA, the Secretary increased the number of required quality measures to 21. Nine of

the new measures related to the original three conditions, and 2 related to surgery, a new 

condition for the program. (See app. I for the list of measures.) 

8

CMS recently announced the addition of three surgery measures for fiscal year 2008. In 

addition, to receive their full annual update payment, hospitals will have to submit to CMS 

the responses provided by a random sample of their discharged patients on a specified 

survey instrument—the Hospital Consumer Assessment of Healthcare Providers and 

Systems (HCAHPS) survey—which is designed to obtain patient assessments of the care

they received. See 71 Fed. Reg. 67960, 68201-10 (Nov. 24, 2006). 

9

Although the term value is often associated with numerical data, we use it in this report to 

identify the information that hospitals submit to CMS for a given data element. Some data 

elements call for numerical values, and others call for nonnumerical values, such as Y or N 

for “yes” or “no.” 

10Beta blockers are medications that decrease the rate and force of heart contractions, 

which over time improves the heart’s pumping ability. 

Page 2 GAO-07-320 Hospital Quality Data
year 2007. For a hospital submitting data on all 21 measures, CMS receives 

values for a total of 73 unique data elements. For heart attack measures 

alone, the 8 measures utilize 35 of the 73 data elements. (Some data 

elements are used in more than 1 measure. See app. I for the number of 

data elements required for each measure.) Hospitals submit their quality

data electronically, over the Internet, to a clinical data warehouse 

operated by a CMS contractor. 

Increasingly, the information in patients’ medical records that provides the 

basis for hospital quality data submissions may be stored and accessed in 

electronic form in information technology (IT) systems. Currently, many 

hospitals record and store such clinical information on patients in a 

combination of paper and electronic systems. Over time, hospitals have 

added new health IT systems to expand the amount of information that is 

stored electronically. In 2005, the Secretary of HHS established the 

American Health Information Community (AHIC) to advance the adoption 

of electronic health records, after the President called in 2004 for the 

widespread adoption of interoperable electronic health records within 

10 years and appointed a National Coordinator for Health Information 

Technology to promote that goal. On August 12, 2005, CMS issued a 

regulation for the APU program that stated a goal of facilitating the use of 

health IT by hospitals to make it easier for them to collect the quality data

from the medical record and submit them to CMS.11 In the preamble to the 

regulation, CMS said that it intended to begin working toward modifying 

its requirements and mechanisms for accepting quality data to allow 

hospitals to transfer their data directly from hospital IT systems without 

having to first transfer the data into specially formatted files as is currently 

required. 

Because the vast majority of acute care hospitals treating Medicare 

patients choose to submit quality data each quarter to CMS, rather than 

accept a reduced annual payment update, you asked us to examine 

(1) how hospitals collect and submit quality data for the Medicare hospital 

quality measures, (2) the extent to which IT facilitates hospitals’ collection 

and submission of quality data for the Medicare hospital quality measures, 

and (3) whether CMS has taken steps to promote the development and use 

of IT systems that could facilitate the collection and submission of 

hospital quality data. 

 1170 Fed. Reg. 47278, 47420 (Aug. 12, 2005). 

Page 3 GAO-07-320 Hospital Quality Data
To assess how hospitals collect and submit quality data, we conducted 

case studies of eight individual acute care hospitals to obtain information 

about the processes they used to collect and submit the data.12 The 

hospitals varied on a number of standard hospital characteristics, 

including size, urban/rural location, and teaching status (see app. III, 

table 1). We visited each case study hospital, and we interviewed the 

individuals responsible for collecting and submitting the quality data to

CMS, managers of the hospital’s quality department, and hospital 

administrators. To assess the extent to which IT facilitates hospitals’ 

collection and submission of quality data, we selected the case study 

hospitals to include both hospitals with relatively well-developed IT 

systems that supported electronic patient records and hospitals with lessdeveloped levels of IT, based on screening interviews done at the time we 

selected the case study hospitals.13 During our site visits, we also 

interviewed IT staff involved in the process of collecting and submitting 

the quality data. To assess whether CMS has taken steps to promote the 

development and use of IT systems that could facilitate the collection and 

submission of hospital quality data, we reviewed relevant federal 

regulations, reports, and related documents and interviewed CMS officials 

and CMS contractors, as well as officials in HHS’s Office of the National 

Coordinator for Health Information Technology (ONC). Because our 

evidence is limited to the eight case studies, it does not offer a basis for 

relating any differences we observed among these particular hospitals to 

their differences on specific dimensions, such as size or teaching status.

Nor can we generalize from the group of eight as a whole to acute care 

hospitals across the country. Where appropriate, we obtained relevant

information about these hospitals from CMS documents and databases; 

however, most of our information for these case studies was reported by 

hospital officials.14 Furthermore, although we examined the processes 

 12All eight hospitals participated in the APU program and had their performance scores on 

the quality measures posted on CMS’s Hospital Compare Web site,

www.hospitalcompare.hhs.gov.

13Available research suggested that only a handful of hospitals had developed a high level 

of IT implementation for purposes of documenting patient care in electronic records, with 

the large majority having reached just the initial stages of this process. See D. Garets and 

M. Davis, “Electronic Medical Records vs. Electronic Health Records: Yes, There Is a

Difference” (Chicago, Ill.: HIMSS Analytics, LLC, updated Jan. 26, 2006). 

14Information obtained from CMS sources included the projected amount of Medicare 

payments that the hospitals would lose if they had not submitted quality data under the 

APU program and the number of patients for whom they submitted data to CMS. See 

app. IV. 

Page 4 GAO-07-320 Hospital Quality Data
hospitals used to collect and submit quality data and the role that IT plays

in that process, we did not examine general IT adoption in the hospital 

industry. We conducted our work from February 2006 to April 2007 in 

accordance with generally accepted government auditing standards. For a 

complete description of our methodology, see appendix IV.

The case study hospitals we visited used six steps to collect and submit 

quality data, two of which (steps 2 and 3) involved complex abstraction—

the process of reviewing and assessing all relevant pieces of information in 

a patient’s medical record to determine the appropriate value for each data 

element. Whether that patient information was recorded electronically, on

paper, or as a mix of both, the six steps were (1) identify the patients, 

(2) locate information in their medical records, (3) determine appropriate 

values for the data elements, (4) transmit the quality data to CMS, 

(5) ensure that the quality data have been accepted by CMS, and (6) supply 

copies of selected medical records to CMS to validate the data. Several 

factors account for the complexity of the abstraction process (steps 2 and 

3), including the content and organization of the medical record, the scope 

of information and clinical judgment required for the data elements, and 

frequent changes by CMS in its data specifications. Due in part to these 

complexities, most of our case study hospitals relied on clinical staff to 

abstract the quality data. Increases in the number of quality measures 

required by CMS led to increased demands on clinical staff resources. 

Offsetting the demands placed on clinical staff were the benefits that case 

study hospitals reported finding in the quality data. For example, all the 

hospitals reported having a process in place to track changes in their 

performance over time and provide feedback to clinicians and reports to 

hospital administrators and trustees. 

Results in Brief 

Our case studies showed that existing IT systems can help hospitals gather 

some quality data but are far from enabling hospitals to automate the 

abstraction process. IT systems helped hospital staff abstract information 

from patients’ medical records, in particular by improving accessibility to 

and legibility of the medical record and by enabling hospitals to 

incorporate CMS’s required data elements into the medical record. The 

limitations reported by officials in the case study hospitals included having

a mix of paper and electronic records, which required staff to check 

multiple places to get the needed information; the prevalence of data 

recorded as unstructured narrative or text, which made locating the 

information time-consuming because it was not in a prescribed place in 

the record; and the inability of some IT systems to access related data 

stored in another IT system in the same hospital, which required hospital 

Page 5 GAO-07-320 Hospital Quality Data
staff to access each IT system separately to obtain related pieces of 

information. While hospital officials expected the scope and functionality 

of their IT systems to increase over time, they projected that this process 

would occur incrementally over a period of years. 

CMS has sponsored studies and joined HHS initiatives to examine and 

promote the current and potential use of hospital IT systems to facilitate 

the collection and submission of quality data, but HHS lacks detailed 

plans, including milestones and a time frame against which to track its 

progress. CMS sponsored two studies that examined the use of hospital IT

systems for quality data collection and submission. Promoting the use of

health IT for quality data collection is also 1 of 14 objectives that HHS has 

identified in its broader effort to encourage the development and 

nationwide implementation of interoperable IT in health care. CMS has 

joined this broader effort by HHS, as well as the Quality Workgroup that 

AHIC created in August 2006 to specify how IT could capture, aggregate,

and report inpatient and outpatient quality data. Through its 

representation in AHIC and the Quality Workgroup, CMS has participated

in decisions about the specific focus areas to be examined through 

contracts with nongovernmental entities. These contracts currently 

address the use of health IT for a range of purposes, which may also 

include quality data collection and submission in the near future. 

However, HHS has identified no detailed plans, milestones, or time frames 

for either its broad effort to encourage IT in health care nationwide or its 

specific objective to promote the use of health IT for quality data 

collection. 

To support the expansion of quality measures for the APU program, we 

recommend that the Secretary of HHS identify the specific steps that the 

department plans to take to promote the use of health IT for the collection 

and submission of data for CMS’s hospital quality measures and inform 

interested parties on those steps and the expected time frame, including

milestones for completing them. In commenting on a draft of this report 

on behalf of HHS, CMS expressed its appreciation of our thorough analysis 

of the processes that hospitals use to report quality data and the role that

IT systems can play in that reporting, and it concurred with our two 

recommendations. 

The quality data submitted by hospitals are collected from the medical 

records of patients admitted to the hospital. Hospital patient medical 

records contain many different types of information, which are organized 

Background 

Page 6 GAO-07-320 Hospital Quality Data
into different sections. Frequently found examples of these sections 

include 

• the face sheet, which summarizes basic demographic and billing data,

including diagnostic codes; 

• history and physicals (H&P), which record both patient medical history 

and physician assessments; 

• physician orders, which show what medications, tests, and procedures 

were ordered by a physician; 

• medication administration records (MAR), which show that a specific 

medication was given to a patient, when it was given, and the dosage; 

• laboratory reports, radiology reports, and test results, such as an 

echocardiogram reading;

• progress notes, in which physicians, nurses, and other clinicians record 

information chronologically on patient status and response to treatments 

during the patient’s hospital stay; 

• operative reports for surgery patients; 

• physician and nursing notes for patients treated in the emergency 

department; and 

• discharge summaries, in which a physician summarizes the patient’s 

hospital stay and records prescriptions and instructions to be given to the 

patient at discharge. 

Hospitals have discretion to determine the structure of their patient 

medical records, as well as to set general policies stating what, where, and 

how specific information should be recorded by clinicians. To guide the 

hospital staff in the abstraction process—that is, in finding and properly 

assessing the information in the patient’s medical record needed to fill in 

the values for the data elements—CMS and the Joint Commission15 have 

 15The Joint Commission (previously the Joint Commission on Accreditation of Healthcare

Organizations or JCAHO) is a private, not-for-profit organization that accredits 

approximately 82 percent of hospitals that participate in Medicare. 

Page 7 GAO-07-320 Hospital Quality Data
jointly issued a Specifications Manual.16 It contains detailed specifications 

that define the data elements for which the hospital staff need to collect 

information and determine values and the correct interpretation of those 

data elements. The Joint Commission also requires hospitals to submit the 

same data that they submit to CMS for the APU program (and some 

additional data) to receive Joint Commission accreditation.

In many hospitals, information in a patient’s medical record is recorded 

and stored in a combination of paper and electronic systems. Patient 

medical records that clinicians record on paper may be stored in a folder

in the hospital’s medical record department and contain all the different 

forms, reports, and notes prepared by different individuals or by different 

departments during the patient’s stay. Depending on the length of the 

patient’s hospital stay and the complexity of the care, an individual patient 

medical record can amount to hundreds of pages.

17 For information stored 

electronically, clinicians may enter information directly into the electronic 

record themselves, as they do for paper records, or they may dictate their 

notes to be transcribed and added to the electronic record later. 

Information may also be recorded on paper and then scanned into the 

patient’s electronic record. For example, if a patient is transferred from 

another hospital, the paper documents from the transferring hospital may 

be scanned into the patient’s electronic record. 

The patient medical information that hospitals store electronically, rather

than on paper, typically resides in multiple health IT systems. One set of IT

systems usually handles administrative tasks such as patient registration 

and billing. Hospitals acquire other IT systems to record laboratory test 

results, to store digital radiological images, to process physician orders for 

medications, and to record notes written by physicians and nurses. 

Hospitals frequently build their health IT capabilities incrementally by

adding new health IT systems over time.18 If the systems that hospitals 

 16Current and past versions of the Specifications Manual for National Hospital Quality

Measures are available at www.qualitynet.org.

17Generally, the records for multiple hospital admissions for the same patient are stored 

together, creating an even more voluminous collection of documents in a single patient 

record. 

18For example, many hospitals have IT systems to record laboratory test results

electronically, but fewer have added IT systems to record radiology results electronically. 

D. Blumenthal et al., Health Information Technology in the United States: The 

Information Base for Progress (Princeton, N.J.: Robert Wood Johnson Foundation, 2006),

3:26. 

Page 8 GAO-07-320 Hospital Quality Data
purchase come from different companies, they are likely to be based on 

varying standards for how the information is stored and exchanged 

electronically. As a result, even in a single hospital, it can be difficult to 

access from one IT system clinical data stored in a different health IT 

system. 

One of the main objectives of ONC is to overcome the problem of multiple 

health IT systems, within and across health care providers, that store and 

exchange information according to varying standards. The mission of ONC 

is to promote the development and nationwide implementation of 

interoperable health IT in both the public and the private sectors in order 

to reduce medical errors, improve quality of care, and enhance the 

efficiency of health care.19 Health IT is interoperable when systems are 

able to exchange data accurately, effectively, securely, and consistently 

with different IT systems, software applications, and networks in such a 

way that the clinical or operational purposes and meaning of the data are 

preserved and unaltered. 

The case study hospitals we visited used six steps to collect and submit 

quality data, two of which involved complex abstraction—the process of 

reviewing and assessing all relevant pieces of information in a patient’s 

medical record to determine the appropriate value for each data element. 

Factors accounting for the complexity of the abstraction process included 

the content and organization of the medical record, the scope of 

information required for the data elements, and frequent changes by CMS 

in its data specifications. Due in part to these complexities, most of our 

case study hospitals relied on clinical staff to abstract the quality data.

Increases in the number of required quality measures led to increased 

demands on clinical staff resources. However, all case study hospitals 

reported finding benefits in the quality data that helped to offset the 

demands placed on clinical staff. 

We found that whether patient information was recorded electronically,

on paper, or as a mix of both, all the case study hospitals collected and

submitted their quality data by carrying out six sequential steps (see fig. 1). 

These steps started with identifying the patients for whom the hospitals 

needed to provide quality data to CMS and continued through the process 

Hospitals Use Six 

Basic Steps to Collect 

and Submit Quality 

Data, Two of Which 

Involve Complex 

Abstraction by 

Hospital Staff 

Hospitals Collect and 

Submit Quality Data by 

Completing Six Basic 

Steps 

 19Exec. Order No. 13335, 69 Fed. Reg. 24059 (Apr. 27, 2004). 

Page 9 GAO-07-320 Hospital Quality Data
of examining each patient’s medical record, one after the other, to find the 

information needed to determine the appropriate values for each of the

required data elements for that patient. Then, for each patient, those 

values were entered by computer into an electronic form or template 

listing each of the data elements for that condition. These forms were 

provided by the data vendor with which the hospital had contracted to 

transmit its quality data to CMS. The vendors also assisted the hospitals in

checking that the data were successfully received by CMS. Finally, the 

hospitals sent copies of the medical records of a selected sample of 

patients to a CMS contractor that used those records to validate the 

accuracy of the quality data submitted by the hospital. 

Figure 1: Six Basic Steps for Hospitals Collecting and Submitting Quality Data 

Note: Patient information may be obtained from either electronic or paper records. 

Specifically, the six steps, which are summarized for each case study 

hospital in appendix III, table 2, were as follows: 

Step 1: Identify patients—The first step was to identify the patients for 

whom the hospitals needed to submit quality data to CMS. Staff at three 

case study hospitals identified these patients using information on the 

patient’s principal diagnosis, or principal procedure in the case of surgery 

Source: GAO.

1. Identify 

the patients

for whom 

quality data

should be 

submitted.

6. Supply 

copies of 

selected

medical 

records to 

CMS for data

validation.

2. Locate 

needed 

information 

in the 

medical 

record.

3. Determine 

appropriate 

value for 

each data

element.

5. Review 

reports to 

ensure 

acceptance 

of data by 

CMS.

4. Transmit 

data to CMS.

http://

Page 10 GAO-07-320 Hospital Quality Data
patients, obtained from the hospital’s billing data.

20 Five case study 

hospitals had their data vendor use the hospital’s billing data to identify 

the eligible patients for them. Every month, all eight hospitals that we 

visited identified patients discharged in the prior month for whom quality 

data should be collected. The hospitals identified all patients 

retrospectively for quality data collection because hospitals have to wait 

until a patient is discharged to determine the principal diagnosis.21

CMS permits hospitals to reduce their data collection effort by providing 

quality data for a representative sample of patients when the total number 

of patients treated for a particular condition exceeds a certain threshold.22

Five case study hospitals drew samples for at least one condition. The data

vendor performed this task for four of those case study hospitals, and 

assisted the hospital in performing this task for the fifth hospital. 

Only one of the case study hospitals reported using nonbilling data 

sources to check the accuracy of the lists of patients selected for quality 

data collection that the hospitals drew from their billing data (see app. III, 

table 3). Several stated that they occasionally noted discrepancies, such as 

patients selected for heart attack measures who, upon review of their 

medical record, should not have had that as their principal diagnosis. 

However, the hospital officials we interviewed told us that discrepancies 

of this sort were likely to be minor. Officials at three hospitals noted that

hospitals generally have periodic routine audits conducted of the coding 

practices of their medical records departments, which would include the 

accuracy of the principal diagnoses and procedures. 

 20The CMS data specifications list the International Classification of Diseases, Ninth 

Revision (ICD-9) diagnostic codes that make a patient eligible for quality data collection. 

The other factor determining basic patient eligibility is age at the time of admission, 

derived from the patient’s admission date and date of birth, also available from billing data. 

For pneumonia patients, secondary diagnoses may also affect eligibility. 

21Medicare bases its payments for inpatient care on principal diagnosis, which it defines as

the condition established after study to be chiefly responsible for the admission. Principal 

diagnosis may also affect determination of the principal procedure, if several procedures

were performed. 

22CMS’s specific sampling requirements vary by medical condition. For example, hospitals

that have more than 78 heart attack patients in a given quarter can submit quality data for a 

random sample of those patients, as long as their sample includes a minimum of 78 patients

and applies a sampling rate of 20 percent up to a maximum required sample of 311.

Page 11 GAO-07-320 Hospital Quality Data
Step 2: Locate information in the medical record—Steps 2 and 3 were 

in practice closely linked in our case study hospitals. Abstractors23 at the

eight case study hospitals examined each selected patient’s medical 

record, looking for all of the discrete pieces of information that, taken

together, would determine what they would decide—in step 3—was the 

correct value for each of the data elements. For some data elements, there 

was a one-to-one correspondence between the piece of information in the 

medical record and the value to be entered. Typical examples included a 

patient’s date of birth and the name of a medication administered to the 

patient. For other data elements, the abstractors had to check for the 

presence or absence of multiple pieces of information in different parts of 

the medical record to determine the correct value for that data element. 

For example, to determine if the patient did, or did not, have a 

contraindication for aspirin, abstractors looked in different parts of the 

medical record for potential contraindications, such as the presence of 

internal bleeding, allergies, or prescriptions for certain other medications 

such as Coumadin.24

In order for abstractors to find information in the patient’s medical record, 

it had to be recorded properly by the clinicians providing the patient’s 

care. Officials at all eight case study hospitals described efforts designed 

to educate physicians and nurses about the specific data elements for 

which they needed to provide information in each patient’s medical 

record. The hospital officials were particularly concerned that the 

clinicians not undermine the hospital’s performance on the quality 

measures by inadequately documenting what they had done and the 

reasons why. For example, one heart failure measure tracks whether a 

patient received each of six specific instructions at the time of discharge, 

but unless information was explicitly recorded in a heart failure patient’s 

medical record for each of the six data elements, that patient was counted 

by CMS as one who had not received all pertinent discharge instructions 

 23Throughout this report, we use the term abstractor to indicate hospital staff who are 

trained to follow a detailed protocol in order to extract specified information in a

consistent fashion from the medical records of multiple patients. 

24Coumadin is a medication that acts as an anticoagulant. It is used to prevent and treat

harmful blood clots that increase the risk of heart attack and stroke. 

Page 12 GAO-07-320 Hospital Quality Data
and therefore did not meet that quality measure.25 This particular measure 

was cited by officials at several hospitals as one that required a higher 

level of documentation than had previously been the norm at their 

hospital. 

Step 3: Determine appropriate data element values—Once 

abstractors had located all the relevant pieces of information pertaining to

a given data element, they had to put those pieces together to arrive at the

appropriate value for the data element. The relevance of that information 

was defined by the detailed instructions provided by the hospitals’ 

vendors, as well as the Specifications Manual jointly issued by CMS and 

the Joint Commission that serves as the basis for the vendor instructions. 

The Specifications Manual sets out the decision rules for choosing among 

the allowable values for each data element. It also identifies which parts of 

the patient’s medical record may or may not provide the required 

information, and often lists specific terms or descriptions that, if recorded 

in the patient’s medical record, would indicate the appropriate value for a

given data element. In addition, the Specifications Manual provides 

abstractors with guidance on how to interpret conflicting information in 

the medical record, such as a note from one clinician that the patient is not 

a smoker and a note elsewhere in the record from another clinician that 

the patient does smoke. To help keep track of multiple pieces of 

information, many abstractors reported that they first filled in the data 

element values on a paper copy of the abstraction form provided by the 

data vendor. In this way, they could write notes in the margin to document 

how they came to their conclusions. 

Step 4: Transmit data to CMS—In order for the quality data to be 

accepted by the clinical data warehouse, they must pass a battery of edit 

checks that look for missing, invalid, or improperly formatted data 

element entries.26 All the case study hospitals contracted with data vendors 

 25These discharge instructions are supposed to cover recommended level of activity, diet, 

follow-up care after discharge, medications, weight monitoring, and what to do if 

symptoms worsen. The abstractor fills in a value for six separate data elements, one for 

each of the six specific instructions. The value is either a yes, written discharge 

instructions addressing the specified activity were provided, or no, instructions addressing 

the specified activity were not provided or unable to determine from medical record 

documentation. Anything less than a yes on all six data elements leads to a negative score 

on this quality measure for that patient. 

26This process is described in GAO, Hospital Quality Data: CMS Needs More Rigorous

Methods to Ensure Reliability of Publicly Released Data, GAO-06-54 (Washington, D.C.: 

Jan. 31, 2006), 14. 

Page 13 GAO-07-320 Hospital Quality Data
to submit their quality data to CMS. They did so, in part, because all of the 

hospitals submitted the same data to the Joint Commission, and it requires 

hospitals to submit their quality data through data vendors that meet the 

Joint Commission’s requirements. The additional cost to the hospitals to 

have the data vendors also submit their quality data to CMS was generally

minimal (see app. III, table 3). 

All of the case study hospitals submitted their data to the data vendor by

filling in values for the required data elements on an electronic version of 

the vendor’s abstraction form.27 Many abstractors did this for a batch of 

patient records at a time, working from paper copies of the form that they 

had filled in previously. Some abstractors entered the data online at the 

same time that they reviewed the patient’s medical records. In other cases, 

someone other than the abstractor who filled in the paper form used the 

completed form to enter the data on a computer. 

Step 5: Ensure data have been accepted by CMS—The case study 

hospitals varied in the extent to which they actively monitored the 

acceptance of their quality data into CMS’s clinical data warehouse. After 

the data vendors submitted the quality data electronically, they and the 

hospitals could download reports from the clinical data warehouse 

indicating whether the submitted data had passed the screening edits for 

proper formatting and valid entries. The hospitals could use these reports 

to detect data entry errors and make corrections prior to CMS’s data 

submission deadline. Three case study hospitals shared this task with their 

data vendors, three hospitals left it for their data vendors to handle, and 

two hospitals received and responded to reports on data edit checks 

produced by their data vendors, rather than reviewing the CMS reports. 

Approximately 2 months after hospitals submitted their quality data, CMS 

released reports to the hospitals showing their performance scores on the 

quality measures before posting the results on its public Web site. 

Step 6: Supply copies of selected medical records—CMS has put in 

place a data validation process to ensure the accuracy of hospital quality 

data submissions. It requires hospitals to supply a CMS contractor with 

paper copies of the complete medical record for five patients selected by

 27Abstractors at many hospitals entered the data through an online connection to the

vendor. Other hospitals submitted their quality data in the form of electronic files. 

Page 14 GAO-07-320 Hospital Quality Data
CMS each quarter.28 Officials at five hospitals noted that they check to

make sure that all parts of the medical records that they used to abstract 

the data originally are included in the package shipped to the CMS 

contractor. Most of the case study hospitals relied on CMS’s data 

validation to ensure the accuracy of their abstractions. However, two 

hospitals reported that they also routinely draw their own sample of cases, 

which are abstracted a second time by a different abstractor in the 

hospital, followed by a comparison of the two sets of results (see app. III, 

table 3). 

The description by hospital officials of the processes they used to collect

and submit quality data indicated that locating the relevant clinical 

information and determining appropriate values for the data elements 

(steps 2 and 3) were the most complex steps of the six identified, due to

several factors. These included the content and organization of the 

medical record, the scope of the information encompassed by the data 

elements, and frequent changes in data specifications. 

The first complicating factor related to the medical record was that the 

information abstractors needed to determine the correct data element 

values for a given patient was generally located in many different sections

of the patient’s medical record. These included documents completed for 

admission to the hospital, emergency department documents, laboratory 

and test results, operating room notes, medication administration records, 

nursing notes, and physician-generated documents such as history and 

physicals, progress notes and consults, orders for medications and tests, 

and discharge summaries. In addition, the abstractors may have had to 

look at documents that came from other providers if the patient was 

transferred to the hospital. Much of the clinical information needed was 

found in the sections of the medical record prepared by clinicians. Often 

the information in question, such as contraindications for aspirin or beta 

Two Most Complex Steps 

Were Locating Relevant 

Clinical Information and 

Determining Appropriate 

Values for Data Elements 

 28CMS draws a sample of five patient records from among those submitted by each hospital 

that provided data on six or more patients to the clinical data warehouse in a given quarter. 

CMS then tells each hospital which patients need to have their records copied, or printed

out in the case of electronic records, and shipped to its contractor. The contractor then

abstracts values for clinical data elements from those records, following the CMS/Joint 

Commission Specifications Manual, and the results are compared—data element by data

element—with the values originally abstracted and submitted by the hospital. If hospitals 

do not achieve a match of at least 80 percent of their data element values with those of the

CMS contractor, following the outcome of any appeals, the hospitals will not receive a full 

payment update from Medicare for the subsequent fiscal year. See GAO-06-54, 14–16. 

Page 15 GAO-07-320 Hospital Quality Data
blockers, could be found in any of a number of places in the medical 

record where clinicians made entries. As a result, abstractors frequently 

had to read through multiple parts of the record to find the information 

needed to determine the correct value for just one data element. At two 

case study hospitals, abstractors said that they routinely read each 

patient’s entire medical record. 

Experienced abstractors often knew where they were most likely to find 

particular pieces of information. They nevertheless also had to check for 

potentially contradictory information in different parts of the medical 

record. For example, as noted, patients may have provided varying 

responses about their smoking history to different clinicians. If any of 

these responses indicated that the patient had smoked cigarettes in the 

last 12 months, the patient was considered to be a smoker according to 

CMS’s data specifications. Another example concerns the possibility that a 

heart attack or heart failure patient may have had multiple 

echocardiogram results recorded in different parts of the medical record. 

Abstractors needed to find all such results in order to apply the rules 

stated in the Specifications Manual for identifying which result to use in 

deciding whether the patient had left ventricular systolic dysfunction 

(LVSD). This data element is used for the quality measure assessing 

whether an angiotensin-converting enzyme inhibitor (ACEI) or angiotensin 

receptor blocker (ARB) was prescribed for LVSD at discharge.29

The second factor was related to the scope of the information required for 

certain data elements. Some of the data elements that the abstractors had 

to fill in represented a composite of related data and clinical judgment 

applied by the abstractor, not just a single discrete piece of information. 

Such composite data elements typically were governed by complicated 

rules for determining the clinical appropriateness of a specific treatment 

for a given patient. For example, the data element for contraindications for 

both ACEIs and ARBs at discharge requires abstractors to check for the 

presence and assess the severity of any of a range of clinical conditions 

that would make the use of either ACEIs or ARBs inappropriate for that 

patient.30 (See fig. 2.) These conditions may appear at any time during the 

 29ACEIs and ARBs are two classes of drugs that have been shown in clinical trials to reduce

mortality and morbidity in patients with LVSD. 

30ACEIs or ARBs may be contraindicated if the patient is known to be allergic to such drugs 

or suffers from certain medical conditions, such as moderate to severe aortic stenosis or 

renal disease. 

Page 16 GAO-07-320 Hospital Quality Data
patient’s hospital stay and so could appear at any of several places in the

medical record. Abstractors must also look for evidence in the record 

from a physician31 linking a decision not to prescribe these drugs to one or 

more of those conditions. 

 31A nurse practitioner or physician assistant may also provide this documentation in the 

patient’s medical record. 

Page 17 GAO-07-320 Hospital Quality Data
Figure 2: Example of the Process for Locating and Assessing Clinical Information to Determine the Appropriate Value for One 

Data Element 

Sources: GAO, CMS.

Is there documentation of potential contraindications/reasons for not prescribing an angiotensin 

converting enzyme inhibitor (ACEI) at discharge AND potential contraindications/reasons for not 

prescribing an angiotensin receptor blocker (ARB) at discharge? 

(See Table 1.2 ACEIs and Table 1.7 ARBs)

 • ACEI allergy AND ARB allergy?

 • Moderate or severe aortic stenosis (This is a contraindication to both ACEIs and ARBs)?

 • Physician, nurse practitioner, or physician assistant documentation of BOTH a reason for 

 not prescribing ACEI at discharge AND a reason for not prescribing an ARB at discharge?

 • Reason documented by physician, nurse practitioner, or physician assistant for not 

 prescribing an ARB at discharge AND an ACEI allergy?

 • Reason documented by physician, nurse practitioner, or physician assistant for not 

 prescribing an ACEI at discharge AND an ARB allergy?

Table 1.2 (ACEIs):

(Continued)

Tarka

Teczem

Trandolapril

Uniretic

Vaseretic

Vasotec

Zestoretic

Zestril 

Version 1.05

Table 1.7 (ARBs):

(Continued) 

 Micardis

Olmesartan

 Tasosartan

Telmisartan

Teveten HCT

Valsartan

Verdia

Version 1.05

Conclusion: ACEI 

contraindicated but not 

ARB because there was

no indication in the 

patient’s medical record 

of an allergy to ARB, of 

aortic stenosis, or of 

another physician 

documented reason for 

not prescribing an ARB.

Patient medical record

Specifications Manual

Page 18 GAO-07-320 Hospital Quality Data
Note: In this illustrative case, adapted from CMS training materials, an abstractor would find that the 

patient was given an ACEI, Zestril, in the emergency department (see MAR,1/30), but because of its 

apparent effect on the patient’s pulse and blood pressure (see Progress Notes, 01/31), it was not 

continued during the hospital stay (see Progress Notes, 02/03) and no ACEI was prescribed at 

discharge (see Discharge Summary). However, there is no mention in the patient’s record of ARBs or 

aortic stenosis. The arrows point to some of the key pieces of information an abstractor would take 

note of in determining that the appropriate value for this data element was “N” for “no.” 

The third factor is the necessity abstractors at the case study hospitals 

faced to adjust to frequent changes in the data specifications set by CMS. 

Since CMS first released its detailed data specifications jointly with the 

Joint Commission in September 2004, it has issued seven new versions of 

the Specifications Manual.32 Therefore, from fall 2004 through summer 

2006, roughly every 3 months hospital abstractors have had to stop and 

take note of what had changed in the data specifications and revamp their 

quality data collection procedures accordingly. Some of these changes 

reflected modifications in the quality measures themselves, such as the 

addition of ARBs for treatment of LVSD. Other changes revised or 

expanded the guidance provided to abstractors, often in response to 

questions submitted by hospitals to CMS. CMS recently changed its 

schedule for issuing revisions to its data specifications from every 

3 months to every 6 months, but that change had not yet affected the 

interval between new revisions issued to hospitals at the time of our case 

study site visits. 

Clinical Staff Abstract 

Quality Data at Most 

Hospitals 

Case study hospitals typically used registered nurses (RN), often 

exclusively, to abstract quality data for the CMS quality measures (see 

app. III, table 3). One hospital relied on a highly experienced licensed 

practical nurse, and two case study hospitals used a mix of RNs and 

nonclinical staff. Officials at one hospital noted that RNs were familiar 

with both the nomenclature and the structure of the hospital’s medical 

records and they could more readily interact with the physicians and 

nurses providing the care about documentation issues. Even when using 

RNs, all but three of the case study hospitals had each abstractor focus on 

one or two medical conditions with which they had expertise. 

Four hospitals had tried using nonclinical staff, most often trained as 

medical record coders, to abstract the quality data. Officials at one of 

 32This tally of revisions only takes account of new versions of the Specifications Manual. 

Recently, the release of new versions has been followed by multiple addendums to the

revision, weeks or months later, to provide further modifications and clarifications. 

Page 19 GAO-07-320 Hospital Quality Data
these hospitals reported that this approach posed challenges. They said 

that it was difficult for nonclinical staff to learn all that they needed to

know to abstract quality data effectively, especially with the constant 

changes being made to the data specifications. At the second hospital, 

officials reported that using nonclinical staff for abstraction did not work 

at all and they switched to using clinically trained staff. At the third 

hospital, the chief clinician leading the quality team stated that the 

hospital’s nonclinical abstractors worked well enough when clinically 

trained colleagues were available to answer their questions. Officials at the 

fourth hospital cited no concerns about using staff who were not RNs to 

abstract quality data, but they subsequently hired an RN to abstract patient 

records for two of the four conditions. 

Case study hospitals drew on a mix of existing and new staff resources to 

handle the collection and submission of quality data to CMS. In two 

hospitals, new staff had been hired specifically to collect quality data for 

the Joint Commission and CMS. In other hospitals, quality data collection 

was assigned to staff already employed in the hospital’s quality 

management department or performing other functions. 

Adding Quality Measures 

Required a Proportionate 

Increase in Staff Resources 

All the case study hospitals found that, over time, they had to increase the 

amount of staff resources devoted to abstracting quality data for the CMS 

quality measures, most notably as the number of measures on which they 

were submitting data expanded. Officials at the case study hospitals 

generally reported that the amount of staff time required for abstraction 

increased proportionately with the number of conditions for which they 

reported quality data. The hospitals had all begun to report most recently 

on the surgical quality measures. They found that the staff hours needed 

for this new set of quality measures were directly related to the number of 

patient records to be abstracted and the number of data elements 

collected. In other words, they found no “economies of scale” as they 

expanded the scope of quality data abstraction. At the time of our site 

visits, four hospitals continued to draw on existing staff resources, while 

others had hired additional staff. Hospital officials estimated that the 

amount of staff resources devoted to abstracting data for the CMS quality 

Page 20 GAO-07-320 Hospital Quality Data
measures ranged from 0.7 to 2.5 full-time equivalents (FTE) (app. III, 

table 3).33

Hospitals Value and Use 

Quality Data 

Hospital officials reported that the demands that quality data collection 

and submission placed on their clinical staff resources were offset by the 

benefits that they derived from the resulting information on their clinical 

performance. Each one had a process for tracking changes in their 

performance over time. Based on those results, they provided feedback to

individual clinicians and reports to hospital administrators and trustees. 

Because they perceived feedback to clinicians to be much more effective 

when provided as soon as possible, several of the case study hospitals 

found ways to calculate their performance on the quality measures 

themselves, often on a monthly basis, rather than wait for CMS to report 

their results for the quarter. 

Officials at all eight case study hospitals pointed to specific changes they

had made in their internal procedures designed to improve their 

performance on one or more quality measures. Most of the case study 

hospitals developed “standing order sets” for particular diagnoses. Such 

order sets provide a mechanism for standardizing both the care provided 

and the documentation of that care, in such areas as prescribing beta 

blockers and aspirin on arrival and at discharge for heart attack patients. 

Another common example involved prompting physicians to administer 

pneumococcal vaccinations to pneumonia patients. However, at most of 

the case study hospitals, use of many standing order sets was optional for

physicians, and hospital officials reported widely varying rates of 

physician use, from close to 100 percent of physicians at one hospital 

using its order set for heart attack patients to just a few physicians using 

any order sets in another hospital. 

Case study hospitals also responded to the information generated from 

their quality data by adjusting their treatment protocols, especially for 

patients treated in their emergency departments. For example, five 

hospitals developed or elaborated on procedural checklists for emergency

department nurses treating pneumonia patients. The objective of these 

changes was to more quickly identify pneumonia patients when they 

 33These represent the FTEs devoted specifically to quality data collection and submission.

Hospital officials noted that additional FTEs were involved in analyzing the hospital’s 

performance on the quality measures and achieving improvements through changes in 

clinical process and educational efforts with the hospital’s clinicians. 

Page 21 GAO-07-320 Hospital Quality Data
arrived at the emergency department and then expeditiously perform 

required blood tests so that the patients would score positively for the 

quality measure on receiving antibiotics within 4 hours of arrival at the 

hospital. Three hospitals strengthened their procedures to identify 

smokers and make sure that they received appropriate counseling. 

Hospital officials noted that they provided quality of care data to entities 

other than CMS and the Joint Commission, such as state governments and 

private insurers, but for the most part they reported that the CMS quality 

measures had two advantages. First, the CMS quality measures enabled 

hospitals to benchmark their performance against the performances of 

virtually every other hospital in the country. Second, officials at two 

hospitals noted that the CMS measures were based on clinical information 

obtained from patient medical records and therefore had greater validity 

as measures of quality of care than measures based solely on 

administrative data.

34 Many hospital officials said that they wished that 

state governments and other entities collecting quality data would accept 

the CMS quality measures instead of requiring related quality data based 

on different definitions and patient populations. Hospital officials in two 

states reported some movement in that direction.

In the case studies, existing IT systems helped hospital abstractors to 

complete their work more quickly, but the limitations of those IT systems 

meant that trained staff still had to examine the entire patient medical 

record and manually abstract the quality data submitted to CMS. IT 

systems helped abstractors obtain information from patients’ medical 

records, in particular by improving their accessibility and legibility, and by 

enabling hospitals to incorporate CMS’s required data elements into those 

medical records. The challenges reported by hospital officials included 

having a mix of paper and electronic records, which required abstractors 

to check multiple places to get the needed information; the prevalence of 

unstructured data, which made locating the information time-consuming 

because it was not in a prescribed place in the record; and the presence of

multiple IT systems that did not share data, which required abstractors to 

separately access each IT system for related pieces of information that 

were in different parts of the medical record. While hospital officials 

expected the scope and functionality of their IT systems to increase over 

Existing IT Systems 

Can Help Hospitals 

Gather Some Quality 

Data but Are Far from 

Enabling Automated 

Abstraction 

 34For example, hospital officials identified several private insurers that assess quality based 

on patient outcomes derived from administrative data, such as hospital billing data.

Page 22 GAO-07-320 Hospital Quality Data
time, they projected that this would occur incrementally over a period of 

years.35

Existing IT Systems Help 

Abstractors Obtain 

Information from Medical 

Records but Have Notable 

Limitations 

Hospitals found that their existing IT systems could facilitate the 

collection of quality data, but that there were limits on the advantages that 

the systems could provide. IT systems, and the electronic records they 

support, offered hospitals two key benefits: (1) improving accessibility to 

and legibility of the medical record, and (2) facilitating the incorporation 

of CMS’s required data elements into the medical record. 

Many hospital abstractors noted that existing electronic records helped 

quality data collection by improving accessibility and legibility of patient 

records. In general, paper records were less accessible than electronic 

records because it took time to find them or to have them transported if 

hospitals had stored them in a remote location after the patients were 

discharged. Also, paper records were more likely to be missing or in use 

by someone else. However, in one case study hospital, an abstractor noted

difficulties in gaining access to a computer terminal to view electronic 

medical records. Many abstractors noted improvements in legibility as a 

fundamental benefit of electronic records. This advantage applied in 

particular to the many sections of the medical record that consisted of 

handwritten text, including history and physicals, progress notes, 

medication administration records, and discharge summaries. 

Some hospitals have used their existing IT systems to facilitate the 

abstraction of information by designing a number of discrete data fields 

that match CMS’s data elements. For example, two hospitals incorporated 

prompts for pneumococcal vaccination in their electronic medication 

ordering system. These prompts not only reminded physicians to order the 

vaccination (if the patient was not already vaccinated) but also helped to

insure documentation of the patient’s vaccination status. One hospital 

developed a special electronic discharge program for heart attack and

heart failure patients that had data elements for the quality measures built 

 35For example, one case study hospital began several years ago to use an IT system to 

record nursing notes. Hospital officials told us that they planned to initiate a pilot test of a

new component of that IT system that would provide computerized physician order entry 

(CPOE) beginning in October 2006. The officials reported that they would assess their 

experience with the pilot test in one hospital unit before deciding how quickly to expand it 

to the rest of the hospital. They said that they were planning ultimately to store all patient 

medical records in electronic form but there was no fixed timeline for that objective. The

timing would depend, they said, on the success of their CPOE pilot test. 

Page 23 GAO-07-320 Hospital Quality Data
into it. Another hospital built a prompt into its electronically generated

discharge instructions to instruct patients to measure their weight daily. 

This enabled the hospital to document more consistently one of the 

specific instructions that heart failure patients are supposed to receive on

discharge but that physicians and nurses tended to overlook in their 

documentation. 

The limitations that hospital officials reported in using existing IT systems 

to collect quality data stemmed from having a mix of paper and electronic 

systems; the prevalence of data recorded in IT systems as unstructured 

paragraphs of narrative or text, as opposed to discrete data fields reserved 

for specific pieces of information; and the inability of some IT systems to 

access related data stored on another IT system in the same hospital. 

Because all but one of the case study hospitals stored clinical records in a 

mix of paper and electronic systems, abstractors generally had to consult 

both paper and electronic records to obtain all needed information. What 

was recorded on paper and what was recorded electronically varied from 

hospital to hospital (see app. III, table 4). However, admissions and billing 

data were electronic at all the case study hospitals. Billing data include 

principal diagnosis and birth date, which are among the CMS-required data

elements. With regard to clinical data, all case study hospitals had test 

results, such as echocardiogram readings, in an electronic form. In 

contrast, nurse progress notes were least likely to be in electronic form at 

the case study hospitals. Moreover, it was not uncommon for a hospital to

have the same type of clinical documentation stored partly in electronic

form and partly on paper. For example, five of the eight case study 

hospitals had a mix of paper and electronic physician notes, reflecting the 

differing personal preferences of the physicians. Discharge summaries and 

medication administration records, on the other hand, tended to be either 

paper or electronic at a given hospital. 

Many of the data in existing IT systems were recorded in unstructured 

formats—that is, as paragraphs of narrative or other text, rather than in 

data fields designated to contain specific pieces of information—which 

created problems in locating the needed information. For example, 

physician notes and discharge summaries were often dictated and 

transcribed. Abstractors typically read through the entire electronic 

document to make sure that they had found all potentially relevant 

references, such as for possible contraindications for a beta blocker or an 

ACEI. By contrast, some of the data in existing IT systems were in 

structured data fields so that specific information could be found in a 

prescribed place in the record. One common example was a list of 

medication allergies, which abstractors used to quickly check for certain

Page 24 GAO-07-320 Hospital Quality Data
drug contraindications. However, officials at several hospitals said that 

developing and implementing structured data fields were labor intensive,

both in terms of programming and in terms of educating clinical staff in 

their use. That is why many of the data stored in electronic records at the 

case study hospitals remained in unstructured formats. 

Another limitation with existing IT systems was the inability of some 

systems to access related data stored on another IT system in the same 

hospital. This situation affected six of the eight case study hospitals to 

some degree. For example, one hospital had an IT system in the 

emergency department and an IT system on the inpatient floors, but the 

two systems were independent and the information in one was not linked 

to the information in the other. Abstractors had to access each IT system 

separately to obtain related pieces of information, which made abstraction 

more complicated and time-consuming. 

Existing IT systems helped hospital abstractors to complete their work 

more quickly, but the limitations of those IT systems meant that, for the 

most part, the nature of their work remained the same. Existing IT systems 

enabled abstractors at several hospitals to more quickly locate the clinical 

information needed to determine the appropriate values for at least some 

of the data elements that the hospitals submitted to CMS. Where hospitals

designed a discrete data field in their IT systems to match a specific CMS 

data element, abstractors could simply transcribe that value into the data

vendor’s abstraction form. However, in all the case study hospitals there 

remained a large number of data elements for which there was no discrete 

data field in a patient’s electronic record that could provide the required 

value for that data element. As a result, trained staff still had to examine 

the medical record as a whole and manually abstract the quality data

submitted to CMS, whether the information in the medical record was 

recorded electronically or on paper.36

 36Some of the data vendors captured values for certain data elements from the hospital’s

billing data, such as the patient’s birth date and discharge status, and entered those values 

in the abstraction form that it provided to the hospital for that patient. The abstractors 

were supposed to make sure those entries were consistent with the information found in

the patient’s medical record. 

Page 25 GAO-07-320 Hospital Quality Data
All the case study hospitals were working to expand the scope and 

functionality of their IT systems, but this expansion was generally 

projected to occur incrementally over a period of years. Hospital officials 

noted that with wider use of IT systems, the advantages of these 

systems—including accessibility, legibility, and the use of discrete data 

fields—would apply to a larger proportion of the clinical records that 

abstractors have to search. As the case study hospitals continue to bring 

more of their clinical documentation into IT systems, and to link separate 

systems within their hospital so that data in one system can be accessed 

from another, it should reduce the time required to collect quality data. 

Full Automation of Quality 

Data Collection Is Not 

Imminent 

However, most officials at the case study hospitals viewed full-scale 

automation of quality data collection and submission through 

implementation of IT systems as, at best, a long-term prospect. They 

pointed to a number of challenges that hospitals would have to overcome

before they could use IT systems to achieve full-scale automation of 

quality data collection and submission. Primary among these were 

overcoming physician reluctance to use IT systems to record clinical

information and the intrinsic complexity of the quality data required by

CMS. One hospital with unusually extensive IT systems had initiated a 

pilot project to see how close it could get to fully automating quality data 

collection for patients with heart failure. Drawing to the maximum extent 

on the data that were amenable to programming, which excluded 

unstructured physician notes, the hospital found that it could complete 

data collection for approximately 10 percent of cases without additional 

manual abstraction. Reflecting on this effort, the hospital official leading 

this project noted that at least some of the data elements required for 

heart failure patients represented “clinical judgment calls.” An official at 

another hospital observed that someone had to apply CMS’s complex 

decision rules to determine the appropriate value for the data elements. If

a hospital wanted to eliminate the need for an abstractor, who currently

makes those decisions retrospectively after weighing multiple pieces of 

information in the patient’s medical record, the same complex decisions 

would have to be made by the patient’s physician at the time of treatment. 

The official suggested that it was preferable not to ask physicians to take 

on that additional task when they should be focused on making

appropriate treatment decisions. 

Another barrier to automated quality data collection mentioned by several 

hospital officials was the frequency of change in the data specifications. As 

noted above, hospitals had to invest considerable staff resources for 

programming and staff education to develop structured data fields for the 

clinical information required for the data elements. Officials at one 

Page 26 GAO-07-320 Hospital Quality Data
hospital stated that it would be difficult to justify that investment without 

knowing how long the data specifications underlying that structured data 

field would remain valid.

CMS has sponsored studies and joined HHS initiatives to examine and 

promote the current and potential use of hospital IT systems to facilitate 

the collection and submission of quality data, but HHS lacks detailed 

plans, including milestones and a time frame against which to track its 

progress. CMS sponsored two studies that examined the use of hospital IT

systems for quality data collection and submission. Promoting the use of

health IT for quality data collection is also 1 of 14 objectives that HHS has 

identified in its broader effort to encourage the development and 

nationwide implementation of interoperable IT in health care. CMS has 

joined this broader effort by HHS, as well as the Quality Workgroup that 

AHIC created in August 2006 to specify how IT could capture, aggregate,

and report inpatient and outpatient quality data. Through its 

representation in AHIC and the Quality Workgroup, CMS has participated

in decisions about the specific focus areas to be examined through 

contracts with nongovernmental entities. These contracts currently 

address the use of health IT for a range of purposes, which may also 

include quality data collection and submission in the near future. 

However, HHS has identified no detailed plans, milestones, or time frames 

for either its broad effort to encourage IT in health care nationwide or its 

specific objective to promote the use of health IT for quality data 

collection. 

Over the past several years, CMS sponsored two studies to examine the 

current and potential capacity of hospital IT systems to facilitate quality 

data collection and submission. These studies identified challenges to 

using existing hospital IT systems for quality data collection and 

submission, including gaps and inconsistencies in applicable data 

standards, as well as in the content of clinical information recorded in 

existing IT systems. Data standards create a uniform vocabulary for 

electronically recorded information by providing common definitions and 

coding conventions for a specified set of medical terms. Currently, an 

array of different standards apply to different aspects of patient care, 

including drug ordering, digital imaging, clinical laboratory results, and 

overall clinical terminology relating to anatomy, problems, and 

CMS Sponsored 

Studies and Joined 

Broader HHS 

Initiatives to Promote 

Use of IT for Quality 

Data Collection and 

Submission, but HHS 

Lacks Detailed Plans, 

Milestones, and Time 

Frame 

CMS Sponsored Studies 

Examining Use of IT 

Systems for Collection and 

Submission of Quality Data 

Page 27 GAO-07-320 Hospital Quality Data
procedures.37 The studies also found that existing IT systems did not 

record much of the specific clinical information needed to determine the

appropriate data element values that hospitals submit to CMS. To achieve

CMS’s goal of enabling hospitals to transmit quality data directly from 

their own IT systems to CMS’s nationwide clinical database, the sets of 

data in the two systems should conform to a common set of data 

standards and capture all the data necessary for quality measures.38 A key 

element in the effort to create this congruence is the further development 

and implementation of data standards. 

In the first study, completed in March 2005, CMS contracted with the 

Colorado Foundation for Medical Care to test the potential for directly 

downloading values for data elements for CMS’s hospital quality measures 

using patient data from electronic medical records in three hospitals and 

one hospital system.39 The study found that numerous factors impeded this 

process under current conditions, including the lack of certain key types 

of information in the hospitals’ IT systems, such as emergency department

data, prearrival data, transfer information, and information on medication 

contraindications. The study also noted that hospitals differed in how they 

coded their data, and that even when they had implemented data 

standards, the hospitals had used different versions of the standards or 

applied them in different ways.40 For example, the study found wide 

variation in the way that the hospitals recorded drug names and laboratory 

results in their IT systems, as none of the hospitals had implemented the 

existing data standards in those areas. 

 37The following standards apply in these areas: the National Council on Prescription Drug

Programs (NCDCP) for drug ordering, Digital Imaging Communications in Medicine

(DICOM) for radiological and other images, Laboratory Logical Observation Identifier 

Name Codes (LOINC) for clinical laboratory results, and Systematized Nomenclature of 

Medicine Clinical Terms (SNOMED-CT) for clinical terminology. 

38This congruence is one component within the broader initiative announced by the

President to promote the adoption of interoperable electronic health records. 

39They were MedStar Health hospital system in Baltimore, New York Presbyterian Hospital 

in New York, Vanderbilt University Medical Center in Nashville, and Wishard Memorial 

Hospital in Indianapolis. All had volunteered to participate in a demonstration project

called Connecting for Health sponsored by the independent, nonprofit eHealth Initiative. 

See Colorado Foundation for Medical Care, Analysis of Data from the “Healthcare

Collaborative Network” (HCN) Project, CMS Special Study SS-CO-08, Final Report

(Denver, Colo., Mar. 31, 2005). 

40Most notably, the hospitals used the messaging standard for transmitting clinical and 

administrative data—HL7—in different ways, including their coding for such data fields as 

admission source and discharge disposition.

Page 28 GAO-07-320 Hospital Quality Data
In the second study, which was conducted by the Iowa Foundation for 

Medical Care and completed in February 2006, CMS examined the 

potential to expand its current data specifications for heart attack, heart 

failure, pneumonia, and surgical measures to incorporate the standards 

adopted by the federal Consolidated Healthcare Informatics (CHI) 

initiative.41 Unlike the first study, which focused on actual patient data in

existing IT systems, this study focused on the relationship of current data 

standards to the data specifications for CMS’s quality data. It found that 

there were inconsistencies in the way that corresponding data elements 

were defined in the CMS/Joint Commission Specifications Manual and in 

the CHI standards that precluded applying those standards to all of CMS’s 

data elements. Moreover, it found that some of the data elements are not 

addressed in the CHI standards. These results suggested to CMS officials 

that the data standards needed to undergo further development before 

they could support greater use of health IT to facilitate quality data 

collection and submission. 

CMS Has Joined HHS’s

Efforts to Promote Greater 

Use of Health IT for 

Quality Data Collection 

and Submission, but HHS 

Lacks Detailed Plans, 

Milestones, and a Time 

Frame to Track Progress 

CMS has joined efforts by HHS to promote greater use of health IT in 

general and, more recently, in facilitating the use of health IT for quality 

data collection and submission. The overall goal of HHS’s efforts in this 

area, working through AHIC and ONC, is to encourage the development 

and nationwide implementation of interoperable health IT in both the 

public and the private sectors. To guide those efforts, ONC has developed

a strategic framework that outlines its goals, objectives, and high-level 

strategies. One of the 14 objectives involves the collection of quality 

information.42

 41The results of the Hospital Data Collection Consolidated Healthcare Informatics 

Adaptation Project were summarized in an internal CMS memo dated March 9, 2006. The 

CHI initiative is a collaborative agreement among federal agencies to adopt a common set 

of health information interoperability standards encompassing a wide range of clinical 

domains, including the data standards referred to in footnote 37. It is a component of the

Federal Health Architecture, which is a partnership of approximately 20 federal agencies 

that use health IT. 

42See GAO, Health Information Technology: HHS Is Continuing Efforts to Define Its 

National Strategy, GAO-06-1071T (Washington, D.C.: Sept. 1, 2006), 17–18. Other objectives

that are in the strategic framework, and that ONC has initiated specific activities to 

address, include encouraging widespread adoption of data standards, promoting consumer

use of personal health information, and expanding health information support in disasters

and crises. 

Page 29 GAO-07-320 Hospital Quality Data
CMS, through its participation in AHIC, has taken part in the selection of

specific focus areas for ONC to pursue in its initial activities to promote 

health IT. Those activities have largely taken place through a series of 

contracts with a number of nongovernmental entities. ONC has sought 

through these contracts to address issues affecting wider use of health IT, 

including standards harmonization, the certification of IT systems, and the 

development of a Nationwide Health Information Network. For example, 

the initial work on standards harmonization, conducted under contract to

ONC by the Healthcare Information Technology Standards Panel (HITSP), 

focused on three targeted areas: biosurveillance,43 sharing laboratory 

results across institutions, and patient registration and medication history. 

Meanwhile, the Certification Commission for Health Information 

Technology (CCHIT) has worked under a separate contract with ONC to 

develop and apply certification criteria for electronic health record 

products used in physician offices, with some initial work on certification

of electronic health record products for inpatient care as well.44

CMS is also represented on the Quality Workgroup that AHIC created in

August 2006 as a first step in promoting the use of health IT for quality 

data collection and submission. One of seven workgroups appointed by 

AHIC, the Quality Workgroup received a specific charge to specify how 

health IT should capture, aggregate, and report inpatient as well as 

outpatient quality data. It plans to address this charge by adding activities 

related to using IT for quality data collection to the work performed by 

HITSP and CCHIT addressing other objectives under their ongoing ONC 

contracts. Members of the Quality Workgroup, along with AHIC itself, 

have recently begun to consider the specific focus areas to include in the 

directions given to HITSP and CCHIT for their activities during the coming 

year.45 Early discussions among AHIC members indicated that they would 

try to select focus areas that built on the work already completed by 

 43Biosurveillance generally refers to the automated monitoring of information sources of

potential value in detecting an emerging epidemic, whether naturally occurring or the

result of bioterrorism. 

44CCHIT is a voluntary, private-sector organization set up in July 2004 by three leading 

health IT industry associations—the American Health Information Management 

Association (AHIMA), the Healthcare Information and Management Systems Society 

(HIMSS), and the National Alliance for Health Information Technology (Alliance)—to 

certify health IT products. 

45As discussed at the AHIC Meeting, Washington, D.C., October 31, 2006. These discussions 

resulted in a set of recommendations that the workgroup presented at AHIC’s Meeting on

March 13, 2007. 

Page 30 GAO-07-320 Hospital Quality Data
ONC’s contractors and that targeted specific improvements in quality data

collection that could also support other priorities for IT development that 

AHIC had identified.46 The focus areas that AHIC selects will, over time, 

influence the decisions that HHS makes regarding the resources it will 

allocate and the specific steps it will take to overcome the limitations of 

existing IT systems for quality data collection and submission. 

In a previous report and subsequent testimony, we noted that ONC’s 

overall approach lacked detailed plans and milestones to ensure that the 

goals articulated in its strategic framework were met. We pointed out that

without setting milestones and tracking progress toward completing them, 

HHS cannot tell if the necessary steps are in place to provide the building 

blocks for achieving its overall objectives.47 HHS concurred with our 

recommendation that it establish detailed plans and milestones for each 

phase of its health IT strategic framework, but it has not yet released any 

such plans, milestones, or a time frame for completion. Moreover, HHS 

has not announced any detailed plans or milestones or a time frame 

relating to the efforts of the Quality Workgroup to promote the use of 

health IT to capture, aggregate, and report inpatient and outpatient quality 

data. Without such plans, it will be difficult to assess how much the focus 

areas AHIC selects in the near term on its contracted activities will 

contribute to enabling the Quality Workgroup to fulfill its charge in a 

timely way. 

There is widespread agreement on the importance of hospital quality data. 

The Congress made the APU program permanent to provide a financial 

incentive for hospitals to submit quality data to CMS and directed the 

Secretary of HHS to increase the number of measures for which hospitals 

would have to provide data. In addition, the hospitals we visited reported 

finding value in the quality data they collected and submitted to CMS to 

improve care. 

Conclusions 

Collecting quality data is a complex and labor-intensive process. Hospital 

officials told us that as the number of quality measures required by CMS 

increased, the number of clinically trained staff required to collect and

 46AHIC has identified priority areas involving consumer empowerment, biosurveillance,

electronic health records, and chronic care. 

47GAO, Health Information Technology: HHS Is Taking Steps to Develop a National

Strategy, GAO-05-628 (Washington, D.C.: May 27, 2005), 3; GAO-06-1071T, 18. 

Page 31 GAO-07-320 Hospital Quality Data
submit quality data increased proportionately. They also told us that 

increased use of IT facilitates the collection and submission of quality data

and thereby lessens the demand for greater staff resources. The degree to

which existing IT systems can facilitate data collection is, however, 

constrained by limitations such as the prevalence of data recorded as 

unstructured narrative or text. Overcoming these limitations would 

enhance the potential of IT systems to ease the demand on hospital 

resources. 

Promoting the use of health IT for quality data collection is 1 of 14 

objectives that HHS has identified in its broader effort to encourage the

development and nationwide implementation of interoperable IT in health 

care. The extent to which HHS can overcome the limitations of existing IT

systems and make progress on this objective will depend in part on where 

this objective falls on the list of priorities for the broader effort. To date,

HHS has identified no detailed plans, milestones, or time frames for either 

the broad effort or the specific objective on promoting the use of health IT 

for collecting quality data. Without such plans, HHS cannot track its 

progress in promoting the use of health IT for collecting quality data, 

making it less likely that HHS will achieve that objective in a timely way. 

Our analysis indicates that unless activities to facilitate greater use of IT 

for quality data collection and submission proceed promptly, hospitals 

may have difficulty collecting and submitting quality data required for an 

expanded APU program. 

To support the expansion of quality measures for the APU program, we 

recommend that the Secretary of HHS take the following actions: 

• identify the specific steps that the department plans to take to promote the 

use of health IT for the collection and submission of data for CMS’s 

hospital quality measures; and 

• inform interested parties about those steps and the expected time frame, 

including milestones for completing them. 

In commenting on a draft of this report on behalf of HHS, CMS expressed 

its appreciation of our thorough analysis of the processes that hospitals 

use to report quality data and the role that IT systems can play in that 

reporting, and it concurred with our two recommendations. (CMS’s 

comments appear in app. V.) With respect to the recommendations, CMS 

stated that it will continue to participate in relevant HHS studies and 

Recommendations for 

Executive Action 

Agency Comments 

and Our Evaluation 

Page 32 GAO-07-320 Hospital Quality Data
workgroups, and, as appropriate, it will inform interested parties regarding 

progress in the implementation of health IT for the collection and 

submission of hospital quality data as specific steps, including time frames 

and milestones, are identified. In addition, as health IT is implemented, 

CMS anticipates that a formal plan will be developed that includes training 

for providers in the use of health IT for reporting quality data. CMS also 

provided technical comments that we incorporated where appropriate.

CMS made two additional comments relating to the information provided 

on our case study hospitals and our discussion of patients excluded from 

the hospital performance assessments. CMS suggested that we describe 

the level of health IT adoption in the case study hospitals in table 1 of

appendix III; this information was already provided in table 4 of 

appendix III. CMS suggested that we highlight the application of patient 

exclusions in adapting health IT for quality data collection and submission. 

We chose not to because our analysis showed that the degree of challenge 

depended on the nature of the information required for a given data 

element. Exclusions based on billing data, such as discharge status, pose 

much less difficulty than other exclusions, such as checking for 

contraindications to ACEIs and ARBs for LVSD, which require a wide 

range of clinical information. 

CMS noted that the AHIC Quality Workgroup had presented its initial set 

of recommendations at AHIC’s most recent meeting on March 13, 2007, 

and provided a copy of those recommendations as an appendix to its 

comments. The agency characterized these recommendations as first 

steps, with initial timelines, to address the complex issues that affect 

implementation of health IT for quality data collection and submission. 

Specifically with reference to collecting quality data from hospitals as well 

as physicians, the Quality Workgroup recommended the appointment of 

an expert panel that would designate a set of quality measures to have 

priority for standardization of their data elements, which, in turn, would 

enable automation of their collection and submission using electronic 

health records and health information exchange. The first 

recommendations from the expert panel are due June 5, 2007. The work of 

the expert panel is intended to guide subsequent efforts by HITSP to fill 

identified gaps in related data standards and by CCHIT to develop criteria 

for certifying electronic health record products. In addition, the Quality 

Workgroup recommended that CMS and the Agency for Healthcare 

Research and Quality (AHRQ) both work to bring together the developers 

of health quality measures and health IT vendors, so that development of 

future health IT systems would take greater account of the data 

Page 33 GAO-07-320 Hospital Quality Data
requirements of emerging quality measures. AHIC approved these 

recommendations from the Quality Workgroup at its March 13 meeting.

We also sent to each of the eight case study hospitals sections from the 

appendixes pertaining to that hospital. We asked each hospital to check 

that the section accurately described its processes for collecting and 

submitting quality data as well as related information on its characteristics 

and resources. Officials from four of the eight hospitals responded and 

provided technical comments that we incorporated where appropriate.

As arranged with your offices, unless you publicly announce its contents 

earlier, we plan no further distribution of this report until 30 days after its 

issue date. At that time, we will send copies of this report to the Secretary 

of HHS, the Administrator of CMS, and other interested parties. We will 

also make copies available to others on request. In addition, the report will 

be available at no charge on GAO’s Web site at http://www.gao.gov. 

If you or your staffs have any questions about this report, please contact 

me at (202) 512-7101 or BascettaC@gao.gov. Contact points for our Offices 

of Congressional Relations and Public Affairs may be found on the last 

page of this report. GAO staff who made major contributions to this report 

are listed in appendix VI.

Cynthia A. Bascetta 

Director, Health Care 

Page 34 GAO-07-320 Hospital Quality Data
Appendix I: Medicare Quality Measures Appendix I: Medicare Quality Measures Required for Full Annual Payment Update 

Required for Full Annual Payment Update 

Condition Quality measure

Number of required

data elements

Heart attack Aspirin at hospital arrival

a

11

Aspirin prescribed at dischargea7

Angiotensin-converting enzyme inhibitor or angiotensin receptor blocker for left 

ventricular systolic dysfunctiona

9

Beta blocker at hospital arrival

a

11

Beta blocker prescribed at dischargea7

Thrombolytic agent received within 30 minutes of hospital arrival 13

Percutaneous coronary intervention received within 120 minutes of hospital arrival 16

Adult smoking cessation advice/counseling 7

Heart failure Left ventricular function assessmenta7

Angiotensin-converting enzyme inhibitor or angiotensin receptor blocker for left 

ventricular systolic dysfunctiona

10

Discharge instructions 12

Adult smoking cessation advice/counseling 8

Pneumonia Initial antibiotic received within 4 hours of hospital arrival

a

16

Oxygenation assessmenta11

Pneumococcal vaccination statusa8

Blood culture performed before first antibiotic received in hospital 19

Adult smoking cessation advice/counseling 9

Appropriate initial antibiotic selection 24

Influenza vaccination status 9

Surgery Prophylactic antibiotic received within 1 hour prior to surgical incision 14

Prophylactic antibiotics discontinued within 24 hours after surgery end time 17

Sources: Federal Register, CMS, GAO (analysis). 

Notes: The 21 measures are listed in 71 Fed. Reg. 47870, 48033-48034, 48045 (Aug. 18, 2006), and

we analyzed the Specifications Manual for National Hospital Quality Measures, version 2.1a, to 

calculate the number of required data elements for each. This set of quality measures is effective for 

discharges from July 2006 on. The Centers for Medicare & Medicaid Services (CMS) uses 73

different data elements to calculate hospital performance on the 21 measures required for the APU

program. The total number of unique data elements is less than the sum of the data elements used to 

calculate each measure because some data elements are included in the calculation of more than

one quality measure. In addition, CMS obtains from hospitals approximately 20 other data elements 

on each patient, including demographic and billing data. 

a

One of the 10 original quality measures. 

Page 35 GAO-07-320 Hospital Quality Data
Appendix II: Da

H

Qua

ta Elements Used to Calculate 

ospital Performance on a Heart Attack

lity Measure 

Page 36 GAO-07-320

Appendix II: Data Elements Used to 

Calculate Hospital Performance on a Heart 

Attack Quality Measure 

Figure 3: Data Elements Used to Calculate Hospital Performance on the Heart 

Attack Quality Measure That Asks Whether a Beta Blocker Was Given When the 

Patient Arrived at the Hospital 

Notes: The boxes represent data elements and the circles and rounded rectangles represent values 

for those elements. In addition to the seven data elements shown in the figure (including arrival date 

and discharge date that appear in the same box), an eighth data element, comfort measures only, is 

first applied for this quality measure, as well as all the other heart attack, heart failure, and pneumonia 

quality measures, to screen out terminal patients receiving palliative care. Three other data 

elements—principal diagnosis, admission date, and birthdate—are used to initially identify the 

patients for whom the heart failure quality measures apply in a given quarter. 

Source: GAO.

Included codesa

= Hospital score

Number of patients who received a beta blocker

Number of patients for whom a beta blocker was appropriate

=

Yes

Yes + No

>1

Admission source?

Transfer from another emergency department?

Compute duration of hospital stay in days from difference 

between the “Arrival Date” and “Discharge Date” data elements

Contraindication to beta blocker on arrival?

Beta blocker received within 24 hours after hospital arrival?

Not in measure 

populationc

Excluded codesb

No

Not in measure 

population

Yes

No

Not in measure 

population

Yes

Yes No

Not in measure 

population

Not in measure 

population

Excluded codese

=0

Included codesd

Final patient count

of Yes’s and No’s

Discharge

status?

=1

Hospital Quality Data
Appendix II: Data Elements Used to Calculate 

Hospital Performance on a Heart Attack

Quality Measure 

a

Included codes consist of eight different values for admission source that represent patients who

were admitted from any source other than those listed in footnote b, including physician referral, 

skilled nursing facility, and the hospital’s emergency room. 

b

Excluded codes consist of three different values for admission source that represent patients who

were transferred to this hospital from another acute care hospital, from a critical access hospital, or 

within the same hospital with a separate claim. 

c

Patients may be excluded from the population used to calculate a hospital’s performance for a variety 

of reasons, including inappropriateness of beta blockers for their treatment—for example, if they have 

a contraindication for their use—or prior treatment in another acute care facility.

d

Included codes consist of 13 different values for discharge status that represent patients who were 

discharged to any setting other than those listed in footnote e, including home care, skilled nursing 

facility, and hospice. 

e

Excluded codes consist of five different values for discharge status that represent patients who were 

discharged to another acute care hospital or federal health care facility, left against medical advice, or 

died. 

Page 37 GAO-07-320 Hospital Quality Data
Appendix III: Tables on Eight Case Study 

Hospitals 

Table 1: Case Study Hospital Characteristics 

Case study hospital 

A B C D E F G H

Number of 

beds 

300–349 500+ 50–99 500+ 100–149 500+ 150–199 500+

Urban/rural Urban Urban Rural Urban Suburban Urban Suburban Urban

Major 

teaching 

Yes Yes No Yes No Yes No Yes

Member of 

multihospital 

system 

Yes Yes Yes No No No No No

Joint 

Commission 

accredited 

Yes Yes Yes Yes Yes Yes Yes Yes

Vendor 

submits 

quality data 

Yes Yes Yes Yes Yes Yes Yes Yes

Patients 

identified for 

data 

collection 

how often 

Monthly Monthly Weekly Monthly Monthly Monthly Monthly Monthly

Abstraction 

tool used

Vendor’s Vendor’s Vendor’s CARTaVendor’s Vendor’s Vendor’s Vendor’s

Conditions 

reported on 

Heart attack, 

heart failure, 

pneumonia, 

surgery 

Heart attack, 

heart failure, 

pneumonia, 

surgery 

Heart attack, 

heart failure, 

pneumonia, 

surgery 

Heart attack, 

heart failure, 

pneumonia, 

surgery 

Heart attack, 

heart failure, 

pneumonia, 

surgery 

Heart attack, 

heart failure, 

pneumonia, 

surgery 

Heart attack, 

heart failure, 

pneumonia, 

surgery 

Heart attack, 

heart failure, 

pneumonia, 

surgery 

Entities that 

receive 

Annual 

Payment 

Update 

(APU) 

program data 

CMS, Joint 

Commission 

CMS, Joint 

Commission, 

vendor 

database, 

private 

insurers 

CMS, Joint 

Commission 

CMS, Joint 

Commission 

CMS, Joint 

Commission, 

vendor 

database 

CMS, Joint 

Commission 

CMS, Joint 

Commission 

CMS, Joint 

Commission 

Entities that 

receive 

different 

quality data 

LeapfrogbLeapfrog, 

state health 

department, 

private 

insurers 

Private 

insurer 

Leapfrog, 

private 

insurer 

Private 

insurer 

Leapfrog, 

private 

insurer 

State health 

department, 

private 

insurers 

Private 

insurer 

Appendix III: Tables on Eight Case Study 

Hospitals 

Page 38 GAO-07-320 Hospital Quality Data
Appendix III: Tables on Eight Case Study 

Hospitals 

Case study hospital 

A B C D E F G H

Amount of 

projected 

reduction in 

fiscal year 

2006 

Medicare 

payments if 

quality data 

not 

submittedc

$139,000 $608,000 $33,000 $449,000 $57,000 $430,000 $93,000 $123,000 

Amount of 

projected 

reduction in 

fiscal year 

2007 

Medicare 

payments if 

quality data 

not 

submittedc

$801,000 $3,250,000 $161,000 $2,298,000 $283,000 $2,451,000 $503,000 $608,000

Sources: American Hospital Association, GAO, Centers for Medicare & Medicaid Services (CMS).

a

CART, which stands for the CMS Abstraction and Reporting Tool, was developed by CMS and made 

available to hospitals at no charge for collecting and submitting quality data. 

b

The Leapfrog Group is a consortium of large private and public health care purchasers that publicly 

recognizes hospitals that have implemented certain specific quality and safety practices, such as

computerized physician order entry. 

c

The projected reduction in fiscal year 2006 and fiscal year 2007 Medicare payments (rounded to the 

nearest $1,000) represents the amount that the hospital’s revenue from Medicare would have 

decreased for that fiscal year had the hospital not submitted quality data under the Annual Payment 

Update program. These estimates are based on information on the number and case mix of Medicare 

patients served by these hospitals during the previous period. This is the information that was 

available to hospital administrators from CMS at the beginning of the fiscal year. The actual reduction 

would ultimately depend on the number and case mix of the Medicare patients that the hospital 

actually treated during the course of that fiscal year. The projected reduction for fiscal year 2007 was 

substantially larger because that was the first year in which the higher rate of reduction mandated by

the Deficit Reduction Act of 2005—from 0.4 percentage points to 2.0 percentage points—took effect.

Page 39 GAO-07-320 Hospital Quality Data
Appendix III: Tables on Eight Case Study 

Hospitals 

Table 2: How Case Study Hospital Officials Described the Steps Taken to Complete Quality Data Collection and Submission 

Case study hospital 

A B C D

1. Identify patientsaVendor prepares list of 

patients to abstract, 

sampling heart failure, 

pneumonia, and surgery 

Vendor prepares list of patients 

based on diagnosis codes, and 

draws samples for heart failure, 

pneumonia, and surgery 

Vendor prepares list of 

patients to abstract 

based on billing data, no 

sampling 

Hospital IT department 

identifies patients based 

on billing data, no 

sampling 

2. Locate 

information in the 

medical record

Abstractor searches 

through emergency 

room and inpatient 

electronic and paper 

records, checking 

multiple forms and 

screens where relevant 

information could be

found 

Abstractor starts search with 

electronic discharge summary, 

then other electronic records 

and paper documents 

Abstractor searches 

through different 

components of paper 

record, including 

printouts from electronic 

records 

Abstractor clicks through 

various electronic screens 

representing different 

types of records, plus 

some scanned 

documents, for example, 

from other providers 

3. Determine 

appropriate data 

element values

Some demographic data 

prepopulated; abstractor 

notes ambiguous or 

conflicting information 

on paper abstraction 

form 

Some demographic data 

prepopulated; other data 

elements written on paper 

abstraction form 

Some demographic data 

prepopulated; other data 

elements entered 

directly into vendor’s 

online abstraction tool 

Some demographic data 

prepopulated; most 

abstractors fill in data 

elements on paper 

abstraction form 

4. Transmit data to 

CMS 

Data elements copied 

from paper abstraction 

form to vendor’s online 

form 

Data elements copied from 

paper abstraction form to 

vendor’s online form 

Data elements entered

directly into vendor’s 

online abstraction tool 

Data elements copied 

from paper abstraction 

form to vendor’s electronic 

form; data manager 

checks data and uploads 

file to vendor 

5. Ensure data have 

been accepted

by CMS 

Performed by vendor Hospital staff reviews 

error reports from clinical 

data warehouse and corrects 

errors 

Performed by vendor Hospital staff reviews 

error reports from vendor 

6. Supply copies of 

selected medical 

records 

Hospital copies and 

ships requested patient 

records 

Hospital copies, checks 

completeness of, and ships 

requested patient records 

Hospital copies, checks 

completeness of, and 

ships requested patient 

records 

Hospital copies and ships 

requested patient records 

Page 40 GAO-07-320 Hospital Quality Data
Appendix III: Tables on Eight Case Study 

Hospitals 

Case study hospital 

E F G H

Hospital prepares list of patients 

from billing data, no sampling

Hospital provides billing data to 

vendor; vendor draws samples

and generates list of patients to 

abstract 

Hospital creates list from billing 

data; vendor provides 

instructions to draw sample of

pneumonia cases 

Hospital submits billing data to

vendor, which identifies eligible

patients and draws samples 

Abstractor works through paper 

records, such as face sheet, 

emergency room treatment 

forms, progress notes, and 

discharge summary 

Abstractor starts with 

electronic records (for heart 

attack and heart failure)—first 

structured records (discharge)

and then free text—and then

examines paper records if 

needed; paper records 

searched for pneumonia and 

surgery 

Abstractor starts searching 

through paper records, then 

looks for additional information

in electronic records (e.g., for 

echocardiogram results) 

Abstractor searches through 

both electronic and paper 

records 

Data elements entered into 

computerized abstraction form 

Some demographic data 

prepopulated; abstractors 

fill out abstraction form, 

some on paper and some 

online 

Some demographic data 

prepopulated; other data 

elements written on paper 

abstraction form 

Some demographic data 

prepopulated; other data 

elements written on paper 

abstraction form 

Completed abstraction forms 

sent on disk to vendor; will 

change soon to completion of

forms online 

For pneumonia and surgery, 

abstractor enters data online, 

for heart attack and heart 

failure, hospital scans paper 

abstraction forms and sends 

electronic file to vendor, which 

submits data to CMS 

Data elements copied from 

paper abstraction form to 

vendor’s online form 

Data elements copied from 

paper abstraction form to 

vendor’s online form 

Hospital reviews error reports 

from vendor and clinical 

warehouse 

Performed by vendor Hospital receives error report 

from vendor and clinical data 

warehouse and makes 

corrections 

Hospital reviews error reports 

from vendor and makes 

corrections; vendor deals with

clinical data warehouse 

Hospital copies and ships 

requested patient records 

Hospital copies and ships 

requested patient records; 

before shipping hospital flags

relevant information 

Hospital copies, checks 

completeness of, and ships 

requested patient records 

Hospital copies, checks 

completeness of, and ships 

requested patient records 

Source: GAO.

Note: Information summarized from hospital case study interviews.

a

The identifying patients step included both determining all the patients who met the CMS criteria for 

inclusion and the application of the CMS sampling procedures, if applicable. CMS only permitted 

hospitals to sample patients for a given condition in a given quarter if the number of eligible patients 

met a certain threshold. Otherwise, the hospital was required to abstract quality data for all patients 

who met the inclusion criteria for any one of the four conditions. Hospitals could also choose not to

sample, even if it were permitted under the CMS sampling procedures. 

Page 41 GAO-07-320 Hospital Quality Data
Appendix III: Tables on Eight Case Study 

Hospitals 

Table 3: Resources Used for Abstraction and Data Submission at Eight Case Study Hospitals 

Case study hospital 

A B C D E F G H

Qualifications of 

abstractors 

Medical

record coders 

and a Master 

of Public 

Health 

Registered 

nurse (RN) 

and 

nonclinical 

All RN All RN Licensed 

practical 

nurse (LPN) 

Medical

records coder 

and RN with 

physician 

support 

RN and 

LPNa

All RN 

Number of 

abstractors 

3 3 3 9 2 3 3 4

Estimated full 

time equivalents 

for abstraction 

of data 

elements 

0.7 <2.0 <1.5 2.5 1.2 1.3 1.2 2.0

Estimated time 

to abstract one

chart 

60 minutes 

(average) 

10 to 15 

minutes 

20 minutes 

(average) 

3 to 120 

minutes 

5 to 60 

minutes 

5 to 60 

minutes 

10 to 30 

minutes 

10 to 90 

minutes 

Average 

number of heart 

attack, heart 

failure, and 

pneumonia 

charts 

abstracted per 

quarterb

222 399 86 686 118 252 190 202

Average 

number of 

surgery charts 

abstracted per 

quarter 

94b218c6c553b105b186b82c61c

Data vendor 

costs for CMS 

quality data 

services per 

year 

$8,200 $5,000 to

$7,500 

$3,600 $3,500 $560 $1,800 $12,450 $13,000 

Checks for 

accuracy of 

case selection 

against another 

data source

Some 

discrepancies 

observed 

A few 

discrepancies 

observed 

Relies on 

vendor 

processes 

and audits 

of medical

records 

coding 

Checks only 

that data 

were 

submitted to 

CMS for all 

patients on 

original list 

to be 

abstracted 

None Some

discrepancies 

observed 

None Checks

patient lists 

for heart 

attack 

patients 

against 

medical

records 

Page 42 GAO-07-320 Hospital Quality Data
Appendix III: Tables on Eight Case Study 

Hospitals 

Case study hospital 

A B C D E F G H

Checks for 

accuracy of data 

abstraction

Hospital

reabstracts 

5 percent of 

cases each 

quarter 

None beyond 

reviews by 

CMS 

contractor 

Hospital

redoes 5 to 

10 cases 

per measure 

set every 

quarter 

Only for 

cases where 

quality 

standard not 

met 

Only for 

cases where 

quality 

standard not 

met 

Only for 

cases where 

quality 

standard not 

met 

Not 

routinely, 

only for 

startup in 

new 

condition 

None 

beyond 

reviews by 

CMS 

contractor 

Sources: GAO, CMS. 

a

The LPN was abstracting cases for one condition temporarily until an RN could be hired to perform 

the work. 

b

Based on submissions to the clinical warehouse for four quarters of discharges from April 2005 

through March 2006. 

c

Based on submissions to the clinical warehouse for one quarter of discharges from January through 

March 2006. 

Page 43 GAO-07-320 Hospital Quality Data
Appendix III: Tables on Eight Case Study 

Hospitals 

Table 4: Electronic and Paper Records at Eight Case Study Hospitals

Case study hospital 

A B C D E F G H

Admissions E E E E E E E E

Billing E E E E E E E E

Emergency department E&P E&P P E P E P P

Medication administration E E P E P P E P

Physician orders including 

prescriptions 

E&P E&P P E P E P E

Nursing notes P P P E P P E P

Laboratory and test results E E E E E E E E 

Physician notes P E&P P E E&P E&P E&P E&P

Discharge summaries and 

instructions 

P E P E P E&P E E

Operating room P E&P E&P E P E&P E E

Source: GAO.

Note: E = electronic, P = paper. 

Page 44 GAO-07-320 Hospital Quality Data
Appendix IV: Scope and Methodology Appendix IV: Scope and Methodology 

To examine how hospitals collect and submit quality data, and to 

determine the extent to which information technology (IT) facilitates 

those processes, we conducted case studies of eight individual acute care 

hospitals that collect and submit quality data to the Centers for Medicare 

& Medicaid Services (CMS). We chose this approach to obtain an in-depth

understanding of these processes as they are currently experienced at the 

hospital level. For background information on the requirements that the 

hospitals had to satisfy, we reviewed CMS documents relevant to the 

Annual Payment Update (APU) program. In particular, we examined 

multiple revisions of the Specifications Manual for National Hospital 

Quality Measures, which is issued jointly by CMS and the Joint 

Commission (formerly the Joint Commission on Accreditation of 

Healthcare Organizations). 

We structured our selection of hospitals for the eight case studies to 

provide a contrast of hospitals with highly sophisticated IT systems and 

hospitals with an average level of IT capability. We excluded critical 

access hospitals from this selection process because they are not included

in the APU program.1

 The selected hospitals varied on several hospital 

characteristics, including urban/rural location, size, teaching status, and 

membership in a system that linked multiple hospitals through shared 

ownership or other formal arrangements. (See app. III, table 1.) 

To select four hospitals with highly sophisticated IT systems, we relied on

recommendations from interviews with a number of experts in the field of

health IT, as well as on a recent review of the research literature on the 

costs and benefits of health IT2

 and other published articles. Three of the 

four hospitals we chose were among those where much of the published 

research has taken place. They were all early adopters of health IT, and 

each had implemented internally developed IT systems. The fourth 

hospital had more recently acquired and adapted a commercially 

developed system. This hospital was distinguished by the extent to which

it had replaced its paper medical records with an integrated system of 

 1

Some critical access hospitals submit quality data to CMS voluntarily, but this does not

affect their Medicare payments. 

2

P.G. Shekelle, S.C. Morton, and E.B. Keeler, Costs and Benefits of Health Information 

Technology, Evidence Report/Technology Assessment No. 132 (prepared by the Southern

California Evidence-based Practice Center under Contract No. 290-02-0003), Agency for

Healthcare Research and Quality Publication No. 06-E006 (Rockville, Md., April 2006).

Page 45 GAO-07-320 Hospital Quality Data
Appendix IV: Scope and Methodology 

electronic patient records. Each of these four case study hospitals was 

located in a different metropolitan area. 

We selected the four hospitals with less sophisticated IT systems from the 

geographic vicinity of the four hospitals already chosen, thus providing 

two case study hospitals from each of four metropolitan areas. We decided

that one should be a rural hospital, using the Medicare definition of rural, 

which is located outside of a Metropolitan Statistical Area (MSA). To 

determine from which of the four metropolitan areas we should select a 

neighboring rural hospital, we analyzed data on Medicare-approved 

hospitals drawn from CMS’s Provider of Services (POS) file. We identified 

the rural hospitals located within 150 miles of each of the first four 

hospitals. From among those four sets of rural hospitals, we chose the set 

with the largest number of acute care hospitals as the set from which to 

choose our rural case study hospital. For each of the remaining three 

metropolitan areas, we used the hospitals listed in the POS file as shortterm acute care hospitals located in the same MSAs as the three sets from 

which to choose our remaining three hospitals. We excluded hospitals 

located in a different state from the first hospital selected for that 

metropolitan area, so that all of the hospitals under consideration for that 

area would come under the jurisdiction of the same Quality Improvement 

Organization (QIO).3

To select the second case study hospital from among those available in or 

near each of the four metropolitan areas, we applied a procedure designed 

to produce a straightforward and unbiased selection. We began by 

recording the total number of cases for which each of these hospitals had 

reported results on CMS’s Web site for heart attack, heart failure, and 

pneumonia quality measures. We obtained this information from the Web 

site itself, running reports for each hospital that showed, for each quality 

measure, the number of cases that the hospital’s quality performance 

score was based on. Since some quality measures apply only to certain 

patients, we recorded the largest number of cases listed for any of the

quality measures reported for a given condition. Next we summed the 

cases for the three conditions and rank ordered the hospitals in each of 

the three MSAs, and the rural hospitals in the fourth metropolitan area,

 3

QIOs are independent organizations that work under contract to CMS to monitor quality of

care for the Medicare program within a given state and help providers to improve their

clinical practices. CMS has assigned primary responsibility to the QIOs to inform hospitals 

about the APU program’s requirements and to provide technical assistance to hospitals in 

meeting those requirements. 

Page 46 GAO-07-320 Hospital Quality Data
Appendix IV: Scope and Methodology 

from most to least total cases submitted. We then made a preliminary 

selection by taking the hospital with the median value in each of those 

lists.4

 By selecting the hospital with the median number of cases reported, 

we attempted to minimize the chances of picking a hospital that would 

represent an outlier compared to other hospitals in the selection pool.5

Before selecting the final four case study hospitals, we checked to make 

sure that the hospitals did not happen to have an unusually high level of IT

capabilities with respect to electronic patient records. To do this, we 

contacted each of the selected hospitals and obtained a description of its 

current IT systems. We compared this description to the stages of 

electronic medical record implementation laid out by the Healthcare 

Information and Management Systems Society (HIMSS).6

 The HIMSS 

model identifies eight stages based on the scope and sophistication of 

clinical functions implemented through a hospital’s system of electronic 

medical records. According to HIMSS, the large majority of hospitals in the 

United States are at the lower three stages. Based on the descriptions of 

these stages, we determined that none of the prospectively selected 

hospitals had IT systems that exceeded the third stage. 

We collected information about the processes used to collect and submit 

quality data from each of the eight case study hospitals through on-site 

interviews with hospital abstractors, quality managers, IT staff, and 

hospital administrators. We told these officials that neither they personally 

nor their hospitals would be identified by name in our report. The site 

visits took place between mid-July and early September 2006 and ranged 

in duration from 3 to 8 hours. Our data collection at each hospital was 

guided by a protocol that specified a series of topics to cover in our 

interviews. These topics included a description of the processes used at 

each hospital and the financial and staff resources devoted to quality data

collection and submission. We pretested the protocol at two hospitals not 

included in our set of eight case study hospitals.

 4

Any ties on median values, which were possible if the list had an even number of hospitals,

were resolved by implementing a decision rule that alternated between taking the higher

number of cases for the first instance, the lower number for the second instance, and so on. 

5

For example, a few hospitals on these lists had submitted results for only one condition. 

6

D. Garets and M. Davis, “Electronic Medical Records vs. Electronic Health Records: Yes,

There Is a Difference” (Chicago, Ill.: HIMSS Analytics, LLC, updated Jan. 26, 2006). 

Page 47 GAO-07-320 Hospital Quality Data
Appendix IV: Scope and Methodology 

As part of the protocol, we asked abstractors at each hospital to explain in

detail how they found the information needed to determine the 

appropriate values for each of the data elements required for two specific 

quality measures: (1) angiotensin-converting enzyme inhibitor (ACEI) or 

antiotensin receptor blocker (ARB) for left ventricular systolic 

dysfunction (LVSD) for heart failure patients and (2) initial antibiotic 

received within 4 hours of hospital arrival for pneumonia patients. We 

selected these measures because they covered a number of different types 

of data elements, including those involving administration of medications, 

determining contraindications, date and time variables, and making 

clinical assessments such as whether a patient had LVSD. 

To determine the extent to which IT facilitated these processes at the eight

case study hospitals, we included several topics on IT systems in our site 

visit protocol. We asked about any IT systems used by the abstractors in 

locating relevant clinical information in patient medical records and the 

specific advantages and limitations they encountered in using those 

systems. We also asked hospital officials to assess the potential for IT 

systems to provide higher levels of assistance for quality data collection 

and submission over time. If separate IT staff were involved in the 

hospital’s quality data collection and submission process, we included 

them in the interviews. 

Where possible, we supplemented the information provided through 

interviews with direct observation of the processes used by hospitals to 

collect and submit quality data. We asked the case study hospitals to show 

us how they performed these processes, and five of the eight hospitals 

arranged for us to observe the collection of quality data for all or part of a 

patient record. We observed abstractors accessing clinical information 

from both paper and electronic records. 

We also obtained pertinent information about the case study hospitals 

from CMS documents and contractors. The estimated amount of dollars 

that the case study hospitals would have lost had they not submitted 

quality data to CMS, presented in appendix III, table 1, was calculated 

from data provided in documents made available to all hospitals at the 

Page 48 GAO-07-320 Hospital Quality Data
Appendix IV: Scope and Methodology 

start of each of the fiscal years.7 Information on the average number of 

patient charts abstracted quarterly by each case study hospital, shown in 

appendix III, table 3, was drawn from a table showing the number of 

patients for whom quality data were submitted to CMS’s clinical data

warehouse. We obtained that table from the Iowa Foundation for Medical 

Care (IFMC), which is the CMS contractor that operates the clinical data 

warehouse. The IFMC table provided this information for all hospitals 

submitting quality data for discharges that occurred from April 2005 

through March 2006. These were the most recent data available. 

The evidence that we obtained from our eight case study hospitals is 

specific to those hospitals. In particular, it does not offer a basis for 

relating any differences we observed among these individual hospitals to 

their differences on specific dimensions, such as size or teaching status.

Nor can we generalize from the group of eight as a whole to acute care 

hospitals across the country. Furthermore, although we examined the 

processes hospitals used to collect and submit quality data and the role 

that IT plays in that process, we did not examine general IT adoption in 

the hospital industry. 

To obtain information on whether CMS has taken steps to promote the 

development of IT systems to facilitate quality data collection and 

submission, we interviewed CMS officials as well as CMS contractors and 

reviewed documents including reports on related studies funded by CMS. 

We also interviewed officials at the Office of the National Coordinator for 

Health Information Technology (ONC) regarding the plans and activities of 

the American Health Information Community (AHIC) quality workgroup. 

In addition, we downloaded relevant documents from the AHIC Web site, 

including meeting agendas, prepared presentations, and meeting minutes 

for both AHIC as a whole and its Quality Workgroup. 

We conducted our work from February 2006 to April 2007 in accordance 

with generally accepted government auditing standards. 

 7

These included the final rule for Inpatient Prospective Payment System updates for fiscal 

years 2006 and 2007, 70 Fed. Reg. 47507 (Aug. 12, 2005) and 71 Fed. Reg. 48166 (Aug. 18, 

2006), and the “Impact file for IPPS FY 2006 Final Rule” and “Impact file for IPPS FY 2007 

Final Rule” downloaded from

http://www.cms.hhs.gov/AcuteInpatientPPS/FFD/list.asp#TopOfPage on October 12, 2006. 

Page 49 GAO-07-320 Hospital Quality Data
Appendix V: Comments from the Centers for Appendix V Medicare & Medicaid Services : Comments from the Centers for 

Medicare & Medicaid Services 

Page 50 GAO-07-320 Hospital Quality Data
Appendix V: Comments from the Centers for 

Medicare & Medicaid Services 

Page 51 GAO-07-320 Hospital Quality Data
Appendix V: Comments from the Centers for 

Medicare & Medicaid Services 

Page 52 GAO-07-320 Hospital Quality Data
Appendix VI: GAO Contact and Staff Acknowledgments 

Page 53 GAO-07-320

Appendix VI: GAO Contact and Staff 

Acknowledgments 

Cynthia A. Bascetta, (202) 512-7101 or BascettaC@gao.gov 

In addition to the contact named above, Linda T. Kohn, Assistant Director; 

Mohammad S. Khan; Eric A. Peterson; Roseanne Price; Jessica C. Smith; 

and Teresa F. Tucker made key contributions to this report. 

Hospital Quality Data

GAO Contact 

Acknowledgments 

(290520) 
GAO’s Mission The Government Accountability Office, the audit, evaluation and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To

have GAO e-mail you a list of newly posted products every afternoon, go 

to www.gao.gov and select “Subscribe to Updates.” 

The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, D.C. 20548 

Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Mail or Phone 

To Report Fraud,

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Public Affairs 

PRINTED ON RECYCLED PAPER